The ImageSwap webhook enables you to define one or more mappings to automatically swap image definitions within Kubernetes Pods with a different registry. This is useful to easily transition from external to internal image registries, work around rate limiting issues, or to maintain consistency with manifests in environments that are airgapped and unable to access commonly used image registries (DockerHub, Quay, GCR, etc.)
An existing image:
nginx/nginx:latest
can be swapped to:
registry.example.com/nginx/nginx:latest
Kubernetes APIs upgrade
Deprecated APIs in v1.22 Because the end of support for some beta APIs the code received some important updates. Now you need to specify a signer for your Certificate Signing Request. You can take a look on the official documentation about API changes Reference API deprecation.
K8s prior to v1.19 ImageSwap v1.5.0+ drops support for k8s versions below v1.19 and will no longer work due to the
admissionregistration.k8s.io/v1beta1api deprecation. For deployment on K8s version v1.19 and before, please use ImageSwap v1.4.x.
EKS 1.22 To use ImageSwap you'll need to setup an Amazon exclusive signer
beta.eks.amazonaws.com/app-serving. Look atclient.V1CertificateSigningRequestSpec()onapp/imageswap-init/imageswap-init.py.
k8s_csr_spec = client.V1CertificateSigningRequestSpec(
groups=["system:authenticated"],
usages=["digital signature", "key encipherment", "server auth"],
request=base64.b64encode(csr_pem).decode("utf-8").rstrip(),
signer_name="beta.eks.amazonaws.com/app-serving"
)
ImageSwap v1.4.0 has major changes
MAPS LOGIC: There is a new MAPS mode logic that has been added to allow for more flexibility in the image swapping logic. The existing logic, referred to as
LEGACYmode, is still available, but has been deprecated. To continue using theLEGACYmode logic set theIMAGESWAP_MODEenvironment variable accordingly. Please reference the configuration section for more information.
Image Definition Preservation: Updates have been made to how image definitions are processed during a swap. Previously the swap logic would drop the image org/project before adding the prefix (ie.
nginx/nginx-ingress:latestwould drop thenginx/portion of the image definition). In v1.4.0+ the swap logic will preserve all parts of the image except the Registry (ie.docker.io/nginx/nginx-ingresswill drop thedocker.ioonly from the image definition).
- Prereqs
- Quickstart
- Health Check
- Image
- Configuration
- Advance Install
- Metrics
- Testing
- Cautions
- Troubleshooting
- Contributing
- Adopters
Kubernetes 1.19.0 or above with the admissionregistration.k8s.io/v1 (or higher) API enabled. Verify that by the following command:
$ kubectl api-versions | grep admissionregistration.k8s.io/v1The result should be:
admissionregistration.k8s.io/v1In addition, the MutatingAdmissionWebhook and ValidatingAdmissionWebhook admission controllers should be added and listed in the correct order in the admission-control flag of kube-apiserver.
ImageSwap requires cluster-admin permissions to deploy to Kubernetes since it requires access to create/read/update/delete cluster scoped resources (MutatingWebhookConfigurations, Certificates, etc.)
You can use the following command to install ImageSwap from this repo with sane defaults
NOTE: The quickstart installation is not meant for production use. Please read through the Cautions sections, and as always, use your best judgement when configuring ImageSwap for production scenarios.
$ kubectl apply -f https://raw.githubusercontent.com/phenixblue/imageswap-webhook/v1.5.2/deploy/install.yaml- Create the
imageswap-systemnamespace - Create cluster and namespace scoped roles/rolebindings
- Deploy the ImageSwap workload and related configs
Create and label a test namespace
$ kubectl create ns test1
$ kubectl label ns test1 k8s.twr.io/imageswap=enabledDeploy some test workloads
# These examples assume you're in the root directory of this repo
# Example with without expected prefix
$ kubectl apply -f ./testing/deployments/test-deploy01.yaml -n test1
# Example with expected prefix
$ kubectl apply -f ./testing/deployments/test-deploy02.yaml -n test1ImageSwap uses a couple of images for operation
ImageSwap uses the imageswap-init init-container to generate/rotate a TLS cert/key pair to secure communication between the Kubernetes API and the webhook. This action takes place on Pod startup.
A new IMAGESWAP_MODE environment variable has been added to control the imageswap logic for the webhook. The value should be LEGACY or MAPS (new default).
MAPS Mode enables a high degree of flexibility for the ImageSwap Webhook.
In MAPS mode, the webhook reads from a map file that defines one or more mappings (key/value pairs) for imageswap logic. With the map file configuration, swap logic for multiple registries and patterns can be configured. In contrast, the LEGACY mode only allowed for a single IMAGE_PREFIX to be defined for image swaps.
A map file is composed of key/value pairs separated by a :: and looks like this:
default::default.example.com
docker.io::my.example.com/mirror-
quay.io::quay.example3.com
gitlab.com::registry.example.com/gitlab
#gcr.io:: # This is a comment
cool.io::
registry.internal.twr.io::registry.example.com
harbor.geo.pks.twr.io::harbor2.com ###### This is a comment with many symbols
noswap_wildcards::twr.io, walrus.io
# exact image mapping (full docker image name)
[EXACT]ghcr.io/fantasy/coolstuff:v1.0::my-local-registry.com/patched-coolstuff:latest
# replace image mapping (unix shell-style wildcards)
[REPLACE]ghcr.io/public*::my-local-registry.com
NOTE: Lines in the map file that are commented out with a leading # are ignored. Trailing comments following a map definition are also ignored.
NOTE: Previous versions of ImageSwap used a single : syntax to separate the key and value portions of a map definition. This syntax is deprecated as of v1.4.3 and will be removed in future versions. Please be sure to update any existing map file configurations to use the new syntax (ie. ::).
NOTE: Prior to v1.4.3 any use of a registry that includes a port for the key of a map definition will result in errors.
The only mapping that is required in the map_file is the default map. The default map alone provides similar functionality to the LEGACY mode.
A map definition that includes a key only can be used to disable image swapping for that particular registry.
A map file can also include a special noswap_wildcards mapping that disables swapping based on greedy pattern matching. Don't actually include an * in this section. A value of example is essentially equivalent to *example*. See examples below for more detail
By adding additional mappings to the map file, you can have much finer granularity to control swapping logic per registry.
Map definitions can become explicit mappings for individual images by using the [EXACT] prefix.
Usage:
[EXACT]<source-image>::<target-image>
Exact maps will be matched exactly against the <source-image> name and replaced with the <target-image> name. No inferences for registry (ie. docker.io/), or tag (ie. :latest) will be inferred for Exact maps.
Exact image matches are handled before all other mapping rules.
Image paths can be completely rewritten by using the [REPLACE] prefix.
Usage:
[REPLACE]<pattern>::<replacement>
When a <source-image> matches against a Replace rule, the image will be transformed to <replacement><image> where <image> includes the tag if it was specified as a part of <source-image>.
Replacement rules are handled after exact match rules, but before the rest.
Module used: fnmatch — Unix filename pattern matching
-
Disable image swapping for all registries EXCEPT
gcr.iodefault: gcr.io::harbor.internal.example.com -
Enable image swapping for all registries except
gcr.iodefault::harbor.internal.example.com gcr.io:: -
Imitate LEGACY functionality as close as possible
default::harbor.internal.example.com noswap_wildcards::harbor.internal.example.comWith this, all images will be swapped except those that already match the
harbor.internal.example.compattern -
Enable swapping for all registries except those that match the
example.compatterndefault::harbor.internal.example.com noswap_wildcards::example.comWith this, images that have any part of the registry that matches
example.comwill skip the swap logicEXAMPLE:
example.com/image:latestexternal.example.com/image:v1.0edge.example.com/image:latest)
-
Enable swapping for all registries, but skip those that match the
example.compattern, except forexternal.example.comdefault:harbor.internal.example.com external.example.com:harbor.internal.example.com noswap_wildcards:example.comWith this, the
edge.example.com/image:latestimage would skip swapping, butexternal.example.com/image:latestwould be swapped toharbor.internal.example.com/image:latest -
Enable different swapping for top level "library" images vs. images that are nested under a project/org
Example library image:
nginx:latestThis format is a shortcut for
docker.io/library/nginx:latestOfficial Docker documentation on image naming
default:: docker.io:: docker.io/library::harbor.example.com/libraryThis map uses a special syntax of adding
/libraryto a registry for the key in map file.With this, the
nginx:latestimage would be swapped toharbor.example.com/library/nginx:latest, but thetmobile/magtape:latestimage would be swapped toharbor.example.com/tmobile/magtape:latestThis configuration can be useful for scenarios like Harbor's image proxy cache feature].
DEPRECATED: This mode will be removed in a future release
Change the IMAGE_PREFIX environment variable definition in the imageswap-env-cm.yaml manifest to customize the repo/registry for the image prefix mutation.
You can also customize the label used to granularly disable ImageSwap on a per workload basis. By default the k8s.twr.io/imageswap label is used, but you can override that by specifying a custom label with the IMAGESWAP_DISABLE_LABEL environment variable.
The value of the label should be disabled.
See the Break Glass: Per Workload section for more details.
Prometheus formatted metrics for API rquests are exposed on the /metrics endpoint.
Assuming you've followed the quickstart steps
-
Review Deployment and Pod spec to validate the webhook is working
$ kubectl get deploy hello-world -n test1 -o yaml $ kubectl get pods -n test1 $ kubectl get pod <pod_name> -n test1 -o yaml
NOTE: You should see the swapped image definition instead of the original definition in the
test-deploy.yamlmanifest.
- By Default the ImageSwap Mutating Webhook Configuration is set to fail "closed". Meaning if the webhook is unreachable or doesn't return an expected response, requests to the Kubernetes API will be blocked. Please adjust the configuration if this is not something that fits your environment.
- ImageSwap supports operation with multiple replicas that can increase availability and performance for critical clusters.
- The certificate generated by the
imageswap-initcontainer is valid for 12 months and will be automatically rotated once the Pod restarts within 6 months of expiration. If the certificate expires, calls to the webhook wil fail. Make sure you plan for this certificate rotation.
ImageSwap can be disabled on a per workload level by adding the k8s.twr.io/imageswap label with a value of disabled to the pod template.
Refer to this test manifest as an example: ./testing/deployments/test-deploy05.yaml
ImageSwap can be enabled and disabled on a per namespace basis by utilizing the k8s.twr.io/imageswap label on the namespace resources. In emergency situations the label can be removed from a namespace to disable image swapping in that namespace.
If there are cluster-wide issues you can disable ImageSwap completely by removing the imagewap-webhook Mutating Webhook Configuration and deleting the ImageSwap deployment.
$ docker run -p 5000:5000/tcp -it imageswapwebhook_app bash
$ ./deny-env.py
$ kubectl get pods # to get the name of the running pod
$ kubectl port-forward <pod_name> 5000:5000$ curl -vX POST https://localhost:5000/ -d @test.json -H "Content-Type: application/json"$ kubectl get pods # to get the name of the running pod
$ kubectl logs <pod_name> -f