Skip to content

Commit

Permalink
Image builder documentation (#9839)
Browse files Browse the repository at this point in the history
* Update image building rules and add post-build job (#9461)

Updated the 'run_if_changed' patterns in the 'images.yaml' file for more precise reaction to changes. Furthermore, '--export-tags' option was removed for building image-builder as tags are not used in build process.

A post-build job for image-builder was added. Now, every time changes are detected in the specified paths in 'main' branch, the image-builder image will be built automatically. This reduces manual effort and ensures that the latest code changes are incorporated in the image-builder.

* Allow admins to bypass branch protection in community-modules (#9437)

* Remove unused script from Istio reconciler testing (#9468)

* Tag telemetry-manager image with module version and remove release job (#9456)

* tag telemetry-manager image with module version and remove release job

* tag telemetry-manager image with module version in a separate job

* change job name

* add auto-generated empty line

* job_removed (#9471)

* gomod(deps): bump google.golang.org/api from 0.152.0 to 0.153.0 (#9474)

Bumps [google.golang.org/api](https://github.com/googleapis/google-api-go-client) from 0.152.0 to 0.153.0.
- [Release notes](https://github.com/googleapis/google-api-go-client/releases)
- [Changelog](https://github.com/googleapis/google-api-go-client/blob/main/CHANGES.md)
- [Commits](googleapis/google-api-go-client@v0.152.0...v0.153.0)

---
updated-dependencies:
- dependency-name: google.golang.org/api
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* actions(deps): bump actions/setup-go from 4 to 5 (#9476)

Bumps [actions/setup-go](https://github.com/actions/setup-go) from 4 to 5.
- [Release notes](https://github.com/actions/setup-go/releases)
- [Commits](actions/setup-go@v4...v5)

---
updated-dependencies:
- dependency-name: actions/setup-go
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* docker-rotate-sa(deps): bump golang (#9477)

Bumps golang from 1.21.4-alpine3.17 to 1.21.5-alpine3.17.

---
updated-dependencies:
- dependency-name: golang
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bumping test-infra and testimages and test-infra-prod (#9479)

No eu.gcr.io/kyma-project/test-infra/ changes.

No europe-docker.pkg.dev/kyma-project/prod/testimages/ changes.

Multiple distinct europe-docker.pkg.dev/kyma-project/prod/test-infra/ changes:

Commits | Dates | Images
--- | --- | ---
e65a3f7...ba72d49 | 2023&#x2011;12&#x2011;04&nbsp;&#x2192;&nbsp;2023&#x2011;12&#x2011;06 | prod/test-infra/ko/clusterscollector, prod/test-infra/ko/cors-proxy, prod/test-infra/ko/diskscollector, prod/test-infra/ko/dnscollector, prod/test-infra/ko/externalsecretschecker, prod/test-infra/ko/gardener-rotate, prod/test-infra/ko/gcscleaner, prod/test-infra/ko/github-webhook-gateway, prod/test-infra/ko/image-detector, prod/test-infra/ko/image-syncer, prod/test-infra/ko/image-url-helper, prod/test-infra/ko/ipcleaner, prod/test-infra/ko/markdown-index, prod/test-infra/ko/move-gcs-bucket, prod/test-infra/ko/needs-tws, prod/test-infra/ko/orphanremover, prod/test-infra/ko/pjtester, prod/test-infra/ko/scan-logs-for-secrets, prod/test-infra/ko/search-github-issue, prod/test-infra/ko/usersmapchecker, prod/test-infra/ko/vmscollector
57f98c4...ba72d49 | 2023&#x2011;12&#x2011;04&nbsp;&#x2192;&nbsp;2023&#x2011;12&#x2011;06 | prod/test-infra/slackmessagesender

* move-gcs-bucket(deps): bump golang in /cmd/cloud-run/move-gcs-bucket (#9480)

Bumps golang from 1.21.4-alpine3.17 to 1.21.5-alpine3.17.

---
updated-dependencies:
- dependency-name: golang
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* gomod(deps): bump github.com/spf13/viper from 1.17.0 to 1.18.0 (#9475)

Bumps [github.com/spf13/viper](https://github.com/spf13/viper) from 1.17.0 to 1.18.0.
- [Release notes](https://github.com/spf13/viper/releases)
- [Commits](spf13/viper@v1.17.0...v1.18.0)

---
updated-dependencies:
- dependency-name: github.com/spf13/viper
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* ginkgo(deps): bump golang in /prow/images/ginkgo (#9478)

Bumps golang from 1.21.4-alpine3.17 to 1.21.5-alpine3.17.

---
updated-dependencies:
- dependency-name: golang
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bumping test-infra and testimages and test-infra-prod (#9481)

No eu.gcr.io/kyma-project/test-infra/ changes.

europe-docker.pkg.dev/kyma-project/prod/testimages/ changes: 57f98c4...59f0cdf (2023&#x2011;12&#x2011;04 → 2023&#x2011;12&#x2011;06)

europe-docker.pkg.dev/kyma-project/prod/test-infra/ changes: ba72d49...37d4a17 (2023&#x2011;12&#x2011;06 → 2023&#x2011;12&#x2011;06)

* Bumping sec-scanners-config.yaml (#9458)

* Bumping test-infra and testimages and test-infra-prod (#9482)

No eu.gcr.io/kyma-project/test-infra/ changes.

No europe-docker.pkg.dev/kyma-project/prod/testimages/ changes.

europe-docker.pkg.dev/kyma-project/prod/test-infra/ changes: 37d4a17...f9e8ceb (2023&#x2011;12&#x2011;06 → 2023&#x2011;12&#x2011;06)

* Bump natsio images (#9469)

* Remove the eventing manager template. (#9484)

* Use semantic versioning for eventing manager (#9473)

* Use semantic versioning for eventing manager

* Restore template

* Adds a setting to ignore non-required tide contexts (#9485)

* fix run_if_changed (#9487)

* chore: Remove cli related k3d pipelines that use kyma deploy (#9486)

* chore: Remove cli related e2e test that use kyma deploy

* cleanup templates

* cleanup templates

* revert integration

* Ignore docker images based on regexp (#9488)

* Ignore docker images based on regexp

* Update cloud run config as well

* Bumping sec-scanners-config.yaml (#9483)

* add two missing required job (#9489)

* allow post build job to run on release-* branches for eventing-manager (#9498)

* allow post build job to run on release-* branches for eventing-manager

* remove skip instruction

* add formatting and add release branch instruction

* docker-rotate-sa(deps): bump alpine (#9500)

Bumps alpine from 3.18.5 to 3.19.0.

---
updated-dependencies:
- dependency-name: alpine
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* gomod(deps): bump github.com/spf13/viper from 1.18.0 to 1.18.1 (#9501)

Bumps [github.com/spf13/viper](https://github.com/spf13/viper) from 1.18.0 to 1.18.1.
- [Release notes](https://github.com/spf13/viper/releases)
- [Commits](spf13/viper@v1.18.0...v1.18.1)

---
updated-dependencies:
- dependency-name: github.com/spf13/viper
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* move-gcs-bucket(deps): bump alpine in /cmd/cloud-run/move-gcs-bucket (#9502)

Bumps alpine from 3.18.5 to 3.19.0.

---
updated-dependencies:
- dependency-name: alpine
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bumping test-infra and testimages and test-infra-prod (#9503)

No eu.gcr.io/kyma-project/test-infra/ changes.

No europe-docker.pkg.dev/kyma-project/prod/testimages/ changes.

Multiple distinct europe-docker.pkg.dev/kyma-project/prod/test-infra/ changes:

Commits | Dates | Images
--- | --- | ---
f9e8ceb...e0b1571 | 2023&#x2011;12&#x2011;06&nbsp;&#x2192;&nbsp;2023&#x2011;12&#x2011;08 | prod/test-infra/ko/clusterscollector, prod/test-infra/ko/cors-proxy, prod/test-infra/ko/diskscollector, prod/test-infra/ko/dnscollector, prod/test-infra/ko/externalsecretschecker, prod/test-infra/ko/gardener-rotate, prod/test-infra/ko/gcscleaner, prod/test-infra/ko/github-webhook-gateway, prod/test-infra/ko/image-detector, prod/test-infra/ko/image-syncer, prod/test-infra/ko/image-url-helper, prod/test-infra/ko/ipcleaner, prod/test-infra/ko/markdown-index, prod/test-infra/ko/move-gcs-bucket, prod/test-infra/ko/needs-tws, prod/test-infra/ko/orphanremover, prod/test-infra/ko/pjtester, prod/test-infra/ko/scan-logs-for-secrets, prod/test-infra/ko/search-github-issue, prod/test-infra/ko/usersmapchecker, prod/test-infra/ko/vmscollector
37d4a17...e0b1571 | 2023&#x2011;12&#x2011;06&nbsp;&#x2192;&nbsp;2023&#x2011;12&#x2011;08 | prod/test-infra/slackmessagesender

* Bumping sec-scanners-config.yaml (#9504)

* Bumping Prow (#9506)

europe-docker.pkg.dev/kyma-project/prod/k8s-prow/ changes: kyma-project/k8s-prow@274dc2b...0438486 (2023&#x2011;12&#x2011;04 → 2023&#x2011;12&#x2011;11)

* check for valid serverless CR URL (#9507)

* stop running the post manager build job on release branches (#9509)

* removed jobs of directory-size-exporter (#9499)

* reconciler consistently failing - expected and can be ignored (#9510)

* slack-msg-sender(deps): bump python (#9513)

Bumps python from 3.12.0-alpine3.18 to 3.12.1-alpine3.18.

---
updated-dependencies:
- dependency-name: python
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bumping test-infra and testimages and test-infra-prod (#9514)

No eu.gcr.io/kyma-project/test-infra/ changes.

No europe-docker.pkg.dev/kyma-project/prod/testimages/ changes.

europe-docker.pkg.dev/kyma-project/prod/test-infra/ changes: e0b1571...e0cf65a (2023&#x2011;12&#x2011;08 → 2023&#x2011;12&#x2011;11)

* Bumping sec-scanners-config.yaml (#9515)

* tf(deps): bump hashicorp/google in /configs/terraform/core (#9512)

Bumps [hashicorp/google](https://github.com/hashicorp/terraform-provider-google) from 5.8.0 to 5.9.0.
- [Release notes](https://github.com/hashicorp/terraform-provider-google/releases)
- [Changelog](https://github.com/hashicorp/terraform-provider-google/blob/main/CHANGELOG.md)
- [Commits](hashicorp/terraform-provider-google@v5.8.0...v5.9.0)

---
updated-dependencies:
- dependency-name: hashicorp/google
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Add required status check for KEB (#9505)

* add squash config for epp repository (#9516)

* Remove skr-aws-ugrade-integration test (#9518)

* remove epp, eventing-tools, eventing-module jobs (#9517)

* Fix image building (#9519)

* Fix image building

* apply suggestions

* fix cahngelog generator

* Bumping test-infra and testimages and test-infra-prod (#9520)

No eu.gcr.io/kyma-project/test-infra/ changes.

No europe-docker.pkg.dev/kyma-project/prod/testimages/ changes.

europe-docker.pkg.dev/kyma-project/prod/test-infra/ changes: fdb47ca...9bb59ac (2023&#x2011;08&#x2011;21 → 2023&#x2011;12&#x2011;12)

* Bumping sec-scanners-config.yaml (#9521)

* Add ADO pipeline preview run feature and test  (#9462)

* Add sign-only mode.

* POC of image building in ADO.

* go mod tidy

* Use default build config.

* POC of image building in ADO.

* Use default build config.

* fix linter errors

* rendertemplates

* Move ado code to separate pacakge and make it testable.

* Move build in ado code to separate package. This can be tested independent and reused in other tools.

* Tests.
TODOs for needed improvements.
Comments.
Small changes in code.

* go mod tidy

* Added building image-builder image to the prowjob.

* Use go buildpack. Chainguard offers free access to the latest tag only.

* Moved Dockerfile back to original location. images directory is not suitable for building image-builder.

* Unexport options fields.

* Remove testing prowjob.

* Add preview run flag.

* Add ADO pipeline preview run feature and test

Added a feature in the image builder that allows ADO (Azure DevOps) pipeline to run in preview mode. This allows users to see the final YAML of the pipeline before executing it. The use of this option can be flagged with adoPreviewRun and it has been limited to work only when running in ADO and not locally. Made companion changes to the tests for these features ensuring all new code is covered. This addition was made to aid debug and development efforts by providing more comprehensive information about pipeline executions in ADO.

* package not used in image-builder

* File committed by mistake.

* go mod tidy

* Pull number should be expect only for presubmit job types.

* Print exit code in new line.

* Align with naming standard.

* Add error handling for nil final yaml in ADO pipeline preview run

This update adds an error message when the final yaml in the Azure DevOps (ADO) pipeline preview run is nil. This handling is necessary to give clearer feedback when the pipeline preview run fails due to a nil final yaml and to prevent runtime errors.

* Add preview run option for Azure DevOps pipelines

Refactored the Azure DevOps pipeline trigger functionality to support a "preview run" mode where users can see the generated pipeline yaml before running the actual pipeline.

This feature is useful in creating or troubleshooting pipelines in Azure DevOps as it enables users to verify and adjust the yaml configuration before executing a pipeline run. This reduces the risk of pipeline failures due to misconfigurations.

Added a new flag that allows the user to specify the path of a yaml file that contains the pipeline definition for the preview run. Made necessary changes in cmd/image-builder/main.go and pkg/azuredevops/pipelines/pipelines.go.

Also modified some tests to accommodate these changes and ensure the functionality is working as expected in both preview run and standard run modes.

* Review comments.

* Added missing negation.
Aligned variable name in if condition.

* Add skr-trial-suspension-dev prowjob (#9524)

* Bumping test-infra and testimages and test-infra-prod (#9525)

No eu.gcr.io/kyma-project/test-infra/ changes.

No europe-docker.pkg.dev/kyma-project/prod/testimages/ changes.

europe-docker.pkg.dev/kyma-project/prod/test-infra/ changes: e0b1571...b563bbe (2023&#x2011;12&#x2011;08 → 2023&#x2011;12&#x2011;13)

* Bumping sec-scanners-config.yaml (#9526)

* Adopt nats prow jobs for release flow. (#9511)

* Remove the old template, as it is no longer needed.

* Run post-nats-manager-build-job on main only.

* Change the tag for the release-manager-build job to the PULL_BASE_REF.

This is the value we pass from here: https://github.com/kyma-project/eventing-manager/blob/2036473ea3a9ae0a93a8d8e6cce65a297b5b4947/.github/workflows/create-release.yml#L63

* Add release-manger-build job to pjtester.

* Change PR.

* Change PR.

* remove pjtester

* add e2e-dind-nodejs-20 image with node 20.10 (#9529)

* add e2e-dind-nodejs-20 image with node 20.10

* cp test.sh

* cp init.sh

* fix: Increase resources requests for cli release jobs (#9530)

* post-main-unstable-cli

* remove limit

* increase resource requests for release job as well

* fix: Set resources limits for cli release jobs (#9534)

* post-main-unstable-cli

* remove limit

* increase resource requests for release job as well

* fix: Set limits for cli release jobs

* Bumping test-infra and testimages and test-infra-prod (#9531)

No eu.gcr.io/kyma-project/test-infra/ changes.

europe-docker.pkg.dev/kyma-project/prod/testimages/ changes: 59f0cdf...751c5f9 (2023&#x2011;12&#x2011;06 → 2023&#x2011;12&#x2011;15)

No europe-docker.pkg.dev/kyma-project/prod/test-infra/ changes.

* Bumping sec-scanners-config.yaml (#9535)

* Bumping test-infra and testimages and test-infra-prod (#9536)

No eu.gcr.io/kyma-project/test-infra/ changes.

No europe-docker.pkg.dev/kyma-project/prod/testimages/ changes.

europe-docker.pkg.dev/kyma-project/prod/test-infra/ changes: b563bbe...80c2f22 (2023&#x2011;12&#x2011;13 → 2023&#x2011;12&#x2011;15)

* Bumping sec-scanners-config.yaml (#9537)

* Update image-builder docs with details about ADO backend and related flags.

* go doc

* Align docs with changes merged from upstream.

* Remove unused field.

* more go doc

* Updated image-builder docs with recent changes.

* Apply suggestions from code review

Co-authored-by: Iwona Langer <[email protected]>

* Apply suggestions from code review

Co-authored-by: Iwona Langer <[email protected]>

* Addressed review comments.

* Apply suggestions from code review

Co-authored-by: Iwona Langer <[email protected]>

* Aligned tool naming.

---------

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: Piotr Bochyński <[email protected]>
Co-authored-by: Tim Riffer <[email protected]>
Co-authored-by: Mostafa Shorim <[email protected]>
Co-authored-by: Andrzej Pankowski <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Kyma Bot <[email protected]>
Co-authored-by: Carina Kothe <[email protected]>
Co-authored-by: Friedrich <[email protected]>
Co-authored-by: Tibor Kiss <[email protected]>
Co-authored-by: Oleksandr Meteiko <[email protected]>
Co-authored-by: Mateusz Wisniewski <[email protected]>
Co-authored-by: Benjamin Lindner <[email protected]>
Co-authored-by: Kacper Małachowski <[email protected]>
Co-authored-by: Xin Ruan <[email protected]>
Co-authored-by: Tobias Schuhmacher <[email protected]>
Co-authored-by: Andreas Thaler <[email protected]>
Co-authored-by: Marek Michali <[email protected]>
Co-authored-by: Korbinian Stoemmer <[email protected]>
Co-authored-by: Piotr Halama <[email protected]>
Co-authored-by: Marcin Szwed <[email protected]>
Co-authored-by: Iwona Langer <[email protected]>
  • Loading branch information
1 parent db69f08 commit 1cf33f3
Show file tree
Hide file tree
Showing 5 changed files with 334 additions and 83 deletions.
243 changes: 171 additions & 72 deletions cmd/image-builder/README.md
Original file line number Diff line number Diff line change
@@ -1,32 +1,87 @@
# image-builder
# Image Builder

This tool serves as an intelligent wrapper for `kaniko-project/executor`. It reduces the complexity of building Docker images and removes the need of using Docker in Docker when building images in K8s infrastructure.
Image Builder is a tool for building OCI-compliant images.
It can build images using different backends, such as Kaniko, BuildKit, and Azure DevOps (ADO).
It also supports signing images with a pre-defined set of signing services
to verify that the image comes from a trusted repository and has not been altered in the meantime.
The tool is designed to be used in ProwJobs.

Key features:
* automatically provides a default tag, which is computed based on a template provided in `config.yaml`
* ~~allows for concurrent builds of image variants that use the same `Dockerfile`~~ See [Known issues](#known-issues) #1
* supports adding multiple tags to the image
* saves command outputs to separate files
* when running in Prow's presubmit job, supports pushing images to different repositories with different tags
* supports pushing the same images to multiple repositories
* supports caching of built layers to reduce build times

## Known issues
## Quickstart Guide

1. Currently, building different variants of the same image is not working. The issue is tracked in https://github.com/kyma-project/test-infra/issues/5975
2. This tool is still at an early stage of development. It is stable enough as a replacement for `docker build`. However, you can expect bugs and codebase changes.
To build an image in an SLC-29 compliant way, use Image Builder with ADO backend in your ProwJob for building images.
Here is an example of a ProwJob for building images with ADO backend:

For any other problems, please raise an [issue](https://github.com/kyma-project/test-infra/issues/new?assignees=&labels=area%2Fci%2C+bug&template=bug-report.md&title=image-builder:%20).
```yaml
- name: pull-build-buildkit-image-builder
annotations:
description: "build buildkit image-builder image"
owner: "neighbors"
labels:
run_if_changed: ^pkg/.*.go|cmd/image-builder/.*.go|^go.mod|cmd/image-builder/images/
decorate: true
cluster: untrusted-workload # use trusted-workload for postsubmit prowjobs
max_concurrency: 10
spec:
containers:
- image: "europe-docker.pkg.dev/kyma-project/prod/image-builder:v20240102-18a8a4b8"
securityContext:
privileged: false
seccompProfile:
type: RuntimeDefault
allowPrivilegeEscalation: false
env:
- name: "ADO_PAT"
valueFrom:
secretKeyRef:
name: "image-builder-ado-token"
key: "token"
command:
- "/image-builder"
args:
- "--name=buildkit-image-builder"
- "--config=/config/kaniko-build-config.yaml"
- "--context=."
- "--dockerfile=cmd/image-builder/images/buildkit/Dockerfile"
- "--build-in-ado=true"
resources:
requests:
memory: 500Mi
cpu: 500m
volumeMounts:
- name: config
mountPath: /config
readOnly: true
volumes:
- name: config
configMap:
name: kaniko-build-config
```
It builds the `buildkit-image-builder` image using the image-builder ADO backend.
It uses the Dockerfile from the `cmd/image-builder/images/buildkit/Dockerfile` path and the config from the `kaniko-build-config` ConfigMap.
Because it's a presubmit ProwJob it does not sign the image.
Signing images is supported only in postsubmit ProwJobs.

## Configuration

## Use config.yaml file
Image Builder is configured using a global configuration YAML file, set of environment variables, and command line flags.

`image-builder` requires a configuration file to be provided with a set of variables, which are used during the
execution.
A `--config` flag is required.
### Configuration YAML File

For more information, refer to the [config.go](config.go) file.
`image-builder` requires a configuration YAML file. The file holds the global configuration for the tool and is maintained by the authors.
Use the `--config` flag to provide a path to the config YAML file.

Example file:
For more information about available properties in the configuration file, refer to the [config.go](config.go) file.

Here's an example file:
```yaml
registry: eu.gcr.io/kyma-project
reproducible: true
Expand All @@ -37,41 +92,37 @@ cache:
cache-run-layers: true
```

## Build multi-architecture images
>**NOTE:** This is an experimental feature that may change in the future.
With the introduction of the experimental BuildKit support, the tool now supports the repeatable flag `--platform`.
You can define multiple platforms you want to build an image for.

You can use all platforms supported by [BuildKit](https://github.com/moby/buildkit/blob/master/docs/multi-platform.md).
### Environment Variables

If you want to use experimental features, there is a new image with the tag suffix `-buildkit`.
Environment variables are mainly used to provide runtime values and configuration set by the CI/CD system.
They provide details about the context in which the tool is running.

## Build multiple variants of the same image
Here is the list of environment variables used by Image Builder:

With `image-builder`, you can reuse the same `Dockerfile` to concurrently build different variants of the same image.
To predefine a set of the same `ARG` substitutions with different values, store them in the `variants.yaml` file .
Use that feature when you need to build an image with different versions of the same binary, for example, for different versions of Kubernetes or Go.
- **REPO_NAME**: The name of the repository with source code to build an image from.
- **REPO_OWNER**: The owner of the repository with source code.
- **JOB_TYPE**: The type of job. This can be either `presubmit` or `postsubmit`. `presubmit` represents a pull request job, and `postsubmit`
represents a push job.
- **PULL_NUMBER**: The number of the pull request.
- **PULL_BASE_SHA**: The base SHA of the pull request or push commit SHA.
- **PULL_PULL_SHA**: The pull request head SHA of the pull request.
- **ADO_PAT**: The Azure DevOps Personal Access Token. It's used in the `buildInADO` function to authenticate with the Azure DevOps API.
- **USE_BUILDKIT**: Determines whether to use BuildKit for building the image. A `buildkit-image-builder` image has this variable set
to `true` by default.
- **CI**: Determines whether the image builder runs in CI mode.

The file has a simple structure:
```yaml
'main':
KUBECTL_VERSION: "1.24.4"
'1.23':
KUBECTL_VERSION: "1.23.9"
```
### Command Line Flags

To use this feature, make sure that:
* you have the `variants.yaml` file in the **same directory** as the `Dockerfile`
* your `Dockerfile` contains `ARG` directives which are named after keys in `variants.yaml`
Command line flags are the main way for developers to configure the tool and provide needed values for the build process.
Check the list and description of the available flags in the [main.go](https://github.com/kyma-project/test-infra/blob/df945b96654d60f82b9738cd98129191c5e753c8/cmd/image-builder/main.go#L668) file.

## Image signing
## Image Signing

image-builder supports signing the images with a pre-defined set of signing services to verify that image comes from a trusted repository and has not been altered in the meantime.
You can enable every signing service on repository and global levels.
Image Builder supports signing the images with a pre-defined set of signing services.
Image signing allows verification that the image comes from a trusted repository and has not been altered in the meantime.
You can enable every supported signing service on repository and global levels.

See the following example sign services configuration in `config.yaml` file:
See the following example of sign services configuration in the `config.yaml` file:
```yaml
sign-config:
enabled-signers:
Expand Down Expand Up @@ -101,50 +152,98 @@ sign-config:
```

All enabled signers under `'*'` are used globally. Additionally, if a repository contains another signer configuration
in the `org/repo` key, image-builder also uses this service to sign the image.
in the `org/repo` key, Image Builder also uses this service to sign the image.
If the job is running in CI (Prow), it picks up the current `org/repo` value from the default Prow variables. If binary
is running outside of CI, `--repo` flag will have to be used. Otherwise, the configuration will not be used.

Currently, image-builder contains a basic implementation of a notary signer. If you want to add a new signer, refer to
Currently, Image Builder contains a basic implementation of a notary signer. If you want to add a new signer, refer to
the [`sign`](../../pkg/sign) package, and its code.

### Sign-Only Mode

Image Builder supports sign-only mode. To enable it, use the `--sign-only` flag.
It signs the images provided in the `--images-to-sign` flag.
It supports signing multiple images at once. The flag can be used multiple times.

## Named Tags

image-builder supports passing the name along with the tag both using the `-tag` option or config for the tag template.
Image Builder supports passing the name along with the tag, using both the `-tag` option and the config for the tag template.
You can use `-tag name=value` to pass the name for the tag.

If the name is not provided, it is evaluated from the value:
- if the value is a string, it is used as a name directly. For example,`-tag latest` is equal to `-tag latest=latest`
- if the value is go-template, it will be converted to a valid name. For example, `-tag v{{ .ShortSHA }}-{{ .Date }}` is equal to `-tag vShortSHA-Date=v{{ .ShortSHA }}-{{ .Date }}`

## Usage
### Parse-Tags-Only Mode

You can use Image Builder to generate tags using pars-tags-only mode. To enable it, use the `--parse-tags-only` flag.
It parses the tags provided in the `--tag` flag and in `config.yaml`. The generated tags are written as JSON to
stdout.

## Build Backend

Image Builder supports three build backends:

- kaniko
- BuildKit
- Azure DevOps pipelines

kaniko and BuildKit build images locally while the Azure DevOps pipelines backend call ADO API.
To use the kaniko backend, use the `image-builder` image.
To use the BuildKit backend, use the `buildkit-image-builder` image.
The ADO backend is supported by both images. To use it, you need to provide the `--build-in-ado=true` flag.
The BuildKit and kaniko backends are deprecated and will be removed in the future.
The preferred way to build images is to use the ADO backend because it's the only SLC-29 compliant backend.

### Azure DevOps Backend (ADO)

The ADO backend uses Image Builder to call ADO API and trigger the `oci-image-builder` pipeline. This backend is SLC-29 compliant. It supports signing images with a production signify service. Images built with ADO can be pushed into Kyma GCP artifacts registries. To build images, the ADO backend uses the `kaniko-project/executor` image.
This backend doesn't support the `--env-file`, `--platform`, and `--variant` flags. Building images for platforms other than amd64 is not supported.
To use this backend, you need to use Image Builder in a ProwJob. See [Quickstart Guide](#quickstart-guide) for an example ProwJob definition.

When using the ADO backend, Image Builder is used as a client collecting values from flags and environment variables and calling ADO API.
Image Builder triggers the `oci-image-builder` pipeline. This pipeline is responsible for processing parameters provided in a call and building, pushing, and signing an image.

Apart from calling ADO API to trigger image build, Image Builder also supports preview mode. In preview mode,
Image Builder does not trigger the ADO pipeline but generates a YAML file with the pipeline definition.
Using this mode allows for the validation of the pipeline definition syntax before triggering it. To use preview mode, add the `--ado-preview-run=true` flag.
To specify a path to the YAML file with the pipeline definition, use the `--ado-preview-run-yaml-path` flag.

## Deprecated Features

### Build Multi-Architecture Images

> **NOTE:** This is an experimental feature that may change in the future.

With the introduction of the experimental BuildKit support, the tool now supports the repeatable flag `--platform`.
You can define multiple platforms you want to build an image for.

You can use all platforms supported by [BuildKit](https://github.com/moby/buildkit/blob/master/docs/multi-platform.md).

If you want to use experimental features, there is a new image with the tag suffix `-buildkit`.

### Build Multiple Variants of the Same Image

With `image-builder`, you can reuse the same `Dockerfile` to concurrently build different variants of the same image.
To predefine a set of the same `ARG` substitutions with different values, store them in the `variants.yaml` file .
Use that feature when you need to build an image with different versions of the same binary, for example, for different
versions of Kubernetes or Go.

The file has a simple structure:

```yaml
'main':
KUBECTL_VERSION: "1.24.4"
'1.23':
KUBECTL_VERSION: "1.23.9"
```
Usage of image-builder:
-config string
Path to application config file (default "/config/image-builder-config.yaml")
-context string
Path to build directory context (default ".")
-dockerfile string
Path to Dockerfile file relative to context (default "Dockerfile")
-env-file string
Path to file with environment variables to be loaded in build
-log-dir string
Path to logs directory where GCB logs will be stored (default "/logs/artifacts")
-name string
Name of the image to be built
-platform value
Only supported with BuildKit. Platform of the image that is built
-repo string
Load repository-specific configuration, for example, signing configuration
-silent
Do not push build logs to stdout
-tag value
Additional tag that the image will be tagged with. Optionally you can pass the name in the format name=value which will be used by export-tags.
-variant string
If variants.yaml file is present, define which variant should be built. If variants.yaml is not present, this flag will be ignored
-export-tags
Flag to pass additional arguments to build Dockerfile. It can be used in the name=value format.
-build-arg
Export parsed tags as build-args into Dockerfile. Each tag will have format TAG_x, where x is the tag name passed along with the tag (see: Named Tags section).
```

To use this feature, make sure that:

* you have the `variants.yaml` file in the **same directory** as the `Dockerfile`
* your `Dockerfile` contains `ARG` directives which are named after keys in `variants.yaml`

### Environment Variables File

`-env-file` specifies the path to the file with environment variables to be loaded in the build. This flag is deprecated.
Use `--build-arg` instead.
Loading

0 comments on commit 1cf33f3

Please sign in to comment.