Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ROX-20757 scanner multi arch builds #1574

Merged
merged 22 commits into from
Jul 31, 2024
Merged

ROX-20757 scanner multi arch builds #1574

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
22 commits
Select commit Hold shift + click to select a range
ab0e7d5
ROX-25321: convert konflux builds to OCI artifacts
Stringy Jul 19, 2024
f4d11c3
Use script directly
Stringy Jul 19, 2024
76472ae
Rename fetch-scanner-data-oci-ta-task ?
Stringy Jul 19, 2024
91097dc
ROX-20757: add multi-arch builds to scanner pipeline.
Stringy Jul 19, 2024
4fa637a
Fixes and renaming following PR review
Stringy Jul 22, 2024
a194cf4
typo
Stringy Jul 22, 2024
d4fdfe1
Fix piplne build spec, add specs for other architectures
Stringy Jul 22, 2024
e9e8580
Fix buildah-remote manifests
Stringy Jul 22, 2024
a02db37
Also include specs for -slim
Stringy Jul 22, 2024
1633da6
Consolidate comments to reduce duplication
Stringy Jul 22, 2024
f81a752
Extend pipeline timeouts
Stringy Jul 23, 2024
32c7c9a
Bump timeouts (possibly temporary)
Stringy Jul 23, 2024
90ff56b
Attempt matrix build for multi-arch and simplify apply-tags
Stringy Jul 24, 2024
cae0384
Empty commit to retrigger CI
Stringy Jul 25, 2024
679c56c
Use manifest creation for konflux- tag
Stringy Jul 25, 2024
f1ff855
Some minor cleanup
Stringy Jul 25, 2024
99147bf
Empty commit to retrigger CI
Stringy Jul 26, 2024
982071d
Cleanup before review
Stringy Jul 26, 2024
d32d26f
Undo matrix builds
Stringy Jul 26, 2024
23063fe
Undo targetarch use in dockerfile
Stringy Jul 26, 2024
6aa8448
Remove arch suffix from build args
Stringy Jul 29, 2024
cbcc319
Empty commit to retrigger CI
Stringy Jul 29, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 3 additions & 1 deletion .tekton/scanner-build.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -58,10 +58,12 @@ spec:
secretName: '{{ git_auth_secret }}'

taskRunSpecs:
- pipelineTaskName: build-container
- pipelineTaskName: build-container-amd64
stepSpecs:
# Provision more CPU to speed up build compared to the defaults.
# https://github.com/redhat-appstudio/build-definitions/blob/main/task/buildah/0.1/buildah.yaml#L126
#
# This is not required for multi-arch builds, because they are performed off cluster
- name: build
computeResources:
requests:
Expand Down
276 changes: 246 additions & 30 deletions .tekton/scanner-component-pipeline.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ spec:
- name: show-sbom
params:
- name: IMAGE_URL
value: $(tasks.build-container.results.IMAGE_URL)
value: $(tasks.build-image-manifest.results.IMAGE_URL)
msugakov marked this conversation as resolved.
Show resolved Hide resolved
taskRef:
params:
- name: name
Expand Down Expand Up @@ -98,10 +98,10 @@ spec:
results:
- description: ""
name: IMAGE_URL
value: $(tasks.build-container.results.IMAGE_URL)
value: $(tasks.build-image-manifest.results.IMAGE_URL)
- description: ""
name: IMAGE_DIGEST
value: $(tasks.build-container.results.IMAGE_DIGEST)
value: $(tasks.build-image-manifest.results.IMAGE_DIGEST)
- description: ""
name: CHAINS-GIT_URL
value: $(tasks.clone-repository.results.url)
Expand All @@ -110,7 +110,7 @@ spec:
value: $(tasks.clone-repository.results.commit)
- description: ""
name: JAVA_COMMUNITY_DEPENDENCIES
value: $(tasks.build-container.results.JAVA_COMMUNITY_DEPENDENCIES)
value: $(tasks.build-container-amd64.results.JAVA_COMMUNITY_DEPENDENCIES)

workspaces:
- name: git-auth
Expand All @@ -122,7 +122,7 @@ spec:
- name: image-url
# We can't provide a StackRox-style tag because it is not known at this time (requires cloning source, etc.)
# As a workaround, we still provide a unique tag that's based on a revision in order for this task to comply with
# its expected input. We later actually add this tag on a built image with the apply-tags task.
# its expected input. We later actually add this tag on a built image with the build-image-manifest-konflux task.
value: $(params.output-image-repo):konflux-$(params.revision)
- name: rebuild
value: $(params.rebuild)
Expand Down Expand Up @@ -211,10 +211,10 @@ spec:
taskRef:
name: fetch-scanner-data-oci-ta

- name: build-container
- name: build-container-amd64
params:
- name: IMAGE
value: $(params.output-image-repo):$(tasks.determine-image-tag.results.IMAGE_TAG)
value: $(params.output-image-repo):$(tasks.determine-image-tag.results.IMAGE_TAG)-amd64
- name: DOCKERFILE
value: $(params.dockerfile)
- name: CONTEXT
Expand All @@ -241,7 +241,89 @@ spec:
- name: name
value: buildah-oci-ta
- name: bundle
value: quay.io/redhat-appstudio-tekton-catalog/task-buildah-oci-ta:0.1@sha256:2a42822363c83b95a84c2f6a10c4d957835431d812b1e4a045b51fab1cca9769
value: quay.io/redhat-appstudio-tekton-catalog/task-buildah-oci-ta:0.2@sha256:1911dc8ec67b3f8ae365b4cbe8d61a56ed69e4b46658b01caee486cfdfd38e89
- name: kind
value: task
resolver: bundles
when:
- input: $(tasks.init.results.build)
operator: in
values: [ "true" ]

- name: build-container-s390x
msugakov marked this conversation as resolved.
Show resolved Hide resolved
params:
- name: IMAGE
value: $(params.output-image-repo):$(tasks.determine-image-tag.results.IMAGE_TAG)-s390x
- name: DOCKERFILE
value: $(params.dockerfile)
- name: CONTEXT
value: $(params.path-context)
- name: HERMETIC
value: $(params.hermetic)
- name: PREFETCH_INPUT
value: $(params.prefetch-input)
- name: IMAGE_EXPIRES_AFTER
value: $(params.image-expires-after)
- name: COMMIT_SHA
value: $(tasks.clone-repository.results.commit)
- name: TARGET_STAGE
value: $(params.build-target-stage)
- name: BUILD_ARGS
value:
- SCANNER_TAG=$(tasks.determine-image-tag.results.IMAGE_TAG)
- name: SOURCE_ARTIFACT
value: $(tasks.fetch-scanner-data.results.SOURCE_ARTIFACT)
- name: CACHI2_ARTIFACT
value: $(tasks.prefetch-dependencies.results.CACHI2_ARTIFACT)
- name: PLATFORM
value: linux/s390x
taskRef:
params:
- name: name
value: buildah-remote-oci-ta
- name: bundle
value: quay.io/redhat-appstudio-tekton-catalog/task-buildah-remote-oci-ta:0.2@sha256:b62eadffad6737b237cbfa3e78114daba18da17e5028fe71707388cf8a102f31
- name: kind
value: task
resolver: bundles
when:
- input: $(tasks.init.results.build)
operator: in
values: [ "true" ]

- name: build-container-ppc64le
params:
- name: IMAGE
value: $(params.output-image-repo):$(tasks.determine-image-tag.results.IMAGE_TAG)-ppc64le
- name: DOCKERFILE
value: $(params.dockerfile)
- name: CONTEXT
value: $(params.path-context)
- name: HERMETIC
value: $(params.hermetic)
- name: PREFETCH_INPUT
value: $(params.prefetch-input)
- name: IMAGE_EXPIRES_AFTER
value: $(params.image-expires-after)
- name: COMMIT_SHA
value: $(tasks.clone-repository.results.commit)
- name: TARGET_STAGE
value: $(params.build-target-stage)
- name: BUILD_ARGS
value:
- SCANNER_TAG=$(tasks.determine-image-tag.results.IMAGE_TAG)
- name: SOURCE_ARTIFACT
value: $(tasks.fetch-scanner-data.results.SOURCE_ARTIFACT)
- name: CACHI2_ARTIFACT
value: $(tasks.prefetch-dependencies.results.CACHI2_ARTIFACT)
- name: PLATFORM
value: linux/ppc64le
taskRef:
params:
- name: name
value: buildah-remote-oci-ta
- name: bundle
value: quay.io/redhat-appstudio-tekton-catalog/task-buildah-remote-oci-ta:0.2@sha256:b62eadffad6737b237cbfa3e78114daba18da17e5028fe71707388cf8a102f31
- name: kind
value: task
resolver: bundles
Expand All @@ -250,31 +332,107 @@ spec:
operator: in
values: [ "true" ]

- name: apply-tags
- name: build-container-arm64
params:
- name: IMAGE
value: $(tasks.build-container.results.IMAGE_URL)
- name: ADDITIONAL_TAGS
value: $(params.output-image-repo):$(tasks.determine-image-tag.results.IMAGE_TAG)-arm64
- name: DOCKERFILE
value: $(params.dockerfile)
- name: CONTEXT
value: $(params.path-context)
- name: HERMETIC
value: $(params.hermetic)
- name: PREFETCH_INPUT
value: $(params.prefetch-input)
- name: IMAGE_EXPIRES_AFTER
value: $(params.image-expires-after)
- name: COMMIT_SHA
value: $(tasks.clone-repository.results.commit)
- name: TARGET_STAGE
value: $(params.build-target-stage)
- name: BUILD_ARGS
value:
- konflux-$(params.revision)
runAfter:
- build-container
- SCANNER_TAG=$(tasks.determine-image-tag.results.IMAGE_TAG)
- name: SOURCE_ARTIFACT
value: $(tasks.fetch-scanner-data.results.SOURCE_ARTIFACT)
- name: CACHI2_ARTIFACT
value: $(tasks.prefetch-dependencies.results.CACHI2_ARTIFACT)
- name: PLATFORM
value: linux/arm64
taskRef:
params:
- name: name
value: apply-tags
value: buildah-remote-oci-ta
- name: bundle
value: quay.io/redhat-appstudio-tekton-catalog/task-apply-tags:0.1@sha256:29add9a49a2281a3755a9b580d2b9c5cb110231b14cccf8ade2fd7895a9b4b4a
value: quay.io/redhat-appstudio-tekton-catalog/task-buildah-remote-oci-ta:0.2@sha256:b62eadffad6737b237cbfa3e78114daba18da17e5028fe71707388cf8a102f31
- name: kind
value: task
resolver: bundles
when:
- input: $(tasks.init.results.build)
operator: in
values: [ "true" ]

- name: build-image-manifest
params:
msugakov marked this conversation as resolved.
Show resolved Hide resolved
- name: IMAGE
value: $(params.output-image-repo):$(tasks.determine-image-tag.results.IMAGE_TAG)
- name: COMMIT_SHA
value: $(tasks.clone-repository.results.commit)
- name: IMAGES
value:
- $(tasks.build-container-amd64.results.IMAGE_REF)
- $(tasks.build-container-s390x.results.IMAGE_REF)
- $(tasks.build-container-ppc64le.results.IMAGE_REF)
- $(tasks.build-container-arm64.results.IMAGE_REF)
- name: IMAGE_EXPIRES_AFTER
value: $(params.image-expires-after)
taskRef:
params:
- name: name
value: build-image-manifest
- name: bundle
value: quay.io/konflux-ci/tekton-catalog/task-build-image-manifest:0.1@sha256:fd0a0cf019621d6b577f1b9ab774bb1832f7cba61b4ceee2fd1bffc96895abf9
- name: kind
value: task
resolver: bundles
when:
- input: $(tasks.init.results.build)
operator: in
values: [ "true" ]

- name: build-image-manifest-konflux
msugakov marked this conversation as resolved.
Show resolved Hide resolved
params:
- name: IMAGE
value: $(params.output-image-repo):konflux-$(params.revision)
- name: COMMIT_SHA
value: $(tasks.clone-repository.results.commit)
- name: IMAGES
value:
- $(tasks.build-container-amd64.results.IMAGE_REF)
- $(tasks.build-container-s390x.results.IMAGE_REF)
- $(tasks.build-container-ppc64le.results.IMAGE_REF)
- $(tasks.build-container-arm64.results.IMAGE_REF)
- name: IMAGE_EXPIRES_AFTER
value: $(params.image-expires-after)
taskRef:
params:
- name: name
value: build-image-manifest
- name: bundle
value: quay.io/konflux-ci/tekton-catalog/task-build-image-manifest:0.1@sha256:fd0a0cf019621d6b577f1b9ab774bb1832f7cba61b4ceee2fd1bffc96895abf9
- name: kind
value: task
resolver: bundles
when:
- input: $(tasks.init.results.build)
operator: in
values: [ "true" ]

- name: build-source-image
params:
- name: BINARY_IMAGE
value: $(tasks.build-container.results.IMAGE_URL)
- name: BASE_IMAGES
value: $(tasks.build-container.results.BASE_IMAGES_DIGESTS)
value: $(tasks.build-container-amd64.results.IMAGE_URL)
- name: SOURCE_ARTIFACT
value: $(tasks.fetch-scanner-data.results.SOURCE_ARTIFACT)
- name: CACHI2_ARTIFACT
Expand All @@ -296,14 +454,72 @@ spec:
operator: in
values: [ "true" ]

- name: deprecated-base-image-check
- name: deprecated-base-image-check-amd64
params:
- name: IMAGE_URL
value: $(tasks.build-container-amd64.results.IMAGE_URL)
- name: IMAGE_DIGEST
value: $(tasks.build-container-amd64.results.IMAGE_DIGEST)
taskRef:
params:
- name: name
value: deprecated-image-check
- name: bundle
value: quay.io/redhat-appstudio-tekton-catalog/task-deprecated-image-check:0.4@sha256:3793fbf59e7dadff9d1f7e7ea4cc430c69a2de620b20c7fd69d71bdd5f6c4a60
- name: kind
value: task
resolver: bundles
when:
- input: $(params.skip-checks)
operator: in
values: [ "false" ]

- name: deprecated-base-image-check-s390x
params:
- name: IMAGE_URL
value: $(tasks.build-container-s390x.results.IMAGE_URL)
- name: IMAGE_DIGEST
value: $(tasks.build-container-s390x.results.IMAGE_DIGEST)
taskRef:
params:
- name: name
value: deprecated-image-check
- name: bundle
value: quay.io/redhat-appstudio-tekton-catalog/task-deprecated-image-check:0.4@sha256:3793fbf59e7dadff9d1f7e7ea4cc430c69a2de620b20c7fd69d71bdd5f6c4a60
- name: kind
value: task
resolver: bundles
when:
- input: $(params.skip-checks)
operator: in
values: [ "false" ]

- name: deprecated-base-image-check-ppc64le
params:
- name: IMAGE_URL
value: $(tasks.build-container-ppc64le.results.IMAGE_URL)
- name: IMAGE_DIGEST
value: $(tasks.build-container-ppc64le.results.IMAGE_DIGEST)
taskRef:
params:
- name: name
value: deprecated-image-check
- name: bundle
value: quay.io/redhat-appstudio-tekton-catalog/task-deprecated-image-check:0.4@sha256:3793fbf59e7dadff9d1f7e7ea4cc430c69a2de620b20c7fd69d71bdd5f6c4a60
- name: kind
value: task
resolver: bundles
when:
- input: $(params.skip-checks)
operator: in
values: [ "false" ]

- name: deprecated-base-image-check-arm64
params:
- name: BASE_IMAGES_DIGESTS
value: $(tasks.build-container.results.BASE_IMAGES_DIGESTS)
- name: IMAGE_URL
value: $(tasks.build-container.results.IMAGE_URL)
value: $(tasks.build-container-arm64.results.IMAGE_URL)
- name: IMAGE_DIGEST
value: $(tasks.build-container.results.IMAGE_DIGEST)
value: $(tasks.build-container-arm64.results.IMAGE_DIGEST)
taskRef:
params:
- name: name
Expand All @@ -321,9 +537,9 @@ spec:
- name: clair-scan
params:
- name: image-digest
value: $(tasks.build-container.results.IMAGE_DIGEST)
value: $(tasks.build-image-manifest.results.IMAGE_DIGEST)
- name: image-url
value: $(tasks.build-container.results.IMAGE_URL)
value: $(tasks.build-image-manifest.results.IMAGE_URL)
taskRef:
params:
- name: name
Expand Down Expand Up @@ -359,9 +575,9 @@ spec:
- name: clamav-scan
params:
- name: image-digest
value: $(tasks.build-container.results.IMAGE_DIGEST)
value: $(tasks.build-image-manifest.results.IMAGE_DIGEST)
- name: image-url
value: $(tasks.build-container.results.IMAGE_URL)
value: $(tasks.build-image-manifest.results.IMAGE_URL)
taskRef:
params:
- name: name
Expand All @@ -379,9 +595,9 @@ spec:
- name: sbom-json-check
params:
- name: IMAGE_URL
value: $(tasks.build-container.results.IMAGE_URL)
value: $(tasks.build-image-manifest.results.IMAGE_URL)
- name: IMAGE_DIGEST
value: $(tasks.build-container.results.IMAGE_DIGEST)
value: $(tasks.build-image-manifest.results.IMAGE_DIGEST)
taskRef:
params:
- name: name
Expand Down
Loading
Loading