Skip to content

Commit

Permalink
Merge branch 'master' into zillani-mapfile-syntax-patch
Browse files Browse the repository at this point in the history
  • Loading branch information
zillani authored Nov 21, 2023
2 parents a0784ee + d45185b commit 7706b43
Show file tree
Hide file tree
Showing 16 changed files with 355 additions and 313 deletions.
26 changes: 13 additions & 13 deletions .github/workflows/ci-e2e-checks.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -15,19 +15,19 @@ jobs:
strategy:
matrix:
k8s-version:
- v1.23
- v1.22
- v1.21
- v1.20
- v1.27
- v1.26
- v1.25
- v1.24
include:
- k8s-version: v1.23
kind-node-image: kindest/node:v1.23.4@sha256:0e34f0d0fd448aa2f2819cfd74e99fe5793a6e4938b328f657c8e3f81ee0dfb9
- k8s-version: v1.22
kind-node-image: kindest/node:v1.22.7@sha256:1dfd72d193bf7da64765fd2f2898f78663b9ba366c2aa74be1fd7498a1873166
- k8s-version: v1.21
kind-node-image: kindest/node:v1.21.10@sha256:84709f09756ba4f863769bdcabe5edafc2ada72d3c8c44d6515fc581b66b029c
- k8s-version: v1.20
kind-node-image: kindest/node:v1.20.15@sha256:393bb9096c6c4d723bb17bceb0896407d7db581532d11ea2839c80b28e5d8deb
- k8s-version: v1.27
kind-node-image: kindest/node:v1.27.3@sha256:3966ac761ae0136263ffdb6cfd4db23ef8a83cba8a463690e98317add2c9ba72
- k8s-version: v1.26
kind-node-image: kindest/node:v1.26.6@sha256:6e2d8b28a5b601defe327b98bd1c2d1930b49e5d8c512e1895099e4504007adb
- k8s-version: v1.25
kind-node-image: kindest/node:v1.25.11@sha256:227fa11ce74ea76a0474eeefb84cb75d8dad1b08638371ecf0e86259b35be0c8
- k8s-version: v1.24
kind-node-image: kindest/node:v1.24.15@sha256:7db4f8bea3e14b82d12e044e25e34bd53754b7f2b0e9d56df21774e6f66a70ab

name: e2e-tests for K8s ${{ matrix.k8s-version }}

Expand Down Expand Up @@ -83,7 +83,7 @@ jobs:
timeout-minutes: 10
uses: engineerd/[email protected]
with:
version: "v0.12.0"
version: "v0.20.0"
image: ${{ matrix.kind-node-image }}
wait: 360s

Expand Down
6 changes: 3 additions & 3 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -15,8 +15,8 @@
# NOTE: The version for both `imageswap-init` and `imageswap` should be identical for now.
# Some effort will need to be put in to be able to distringuish changes to one vs. the other
# in CI/Release steps.
IMAGESWAP_VERSION := v1.5.2
IMAGESWAP_INIT_VERSION := v1.5.2
IMAGESWAP_VERSION := v1.5.3
IMAGESWAP_INIT_VERSION := v1.5.3

REPO_ROOT := $(CURDIR)
APP_NAME ?= "imageswap.py"
Expand All @@ -27,7 +27,7 @@ TEST_NAMESPACE ?= "test1"
DOCKER := docker

# Pin utilities at specific versions for CI stability
KUBECTL_VERSION ?= v1.19.1
KUBECTL_VERSION ?= v1.24.17

###############################################################################
# CI Bootstrap Related Targets ################################################
Expand Down
293 changes: 5 additions & 288 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,74 +21,28 @@ registry.example.com/nginx/nginx:latest
```

## NOTICE
Kubernetes APIs upgrade

>**Deprecated APIs in v1.22** Because the end of support for some beta APIs the code received some important updates. Now you need to specify a signer for your Certificate Signing Request. You can take a look on the official documentation about API changes [Reference API deprecation](https://kubernetes.io/docs/reference/using-api/deprecation-guide/#v1-22).
>**K8s prior to v1.19** ImageSwap v1.5.0+ drops support for k8s versions below v1.19 and will no longer work due to the `admissionregistration.k8s.io/v1beta1` api deprecation. For deployment on K8s version v1.19 and before, please use ImageSwap v1.4.x.
>**EKS 1.22** To use ImageSwap you'll need to setup an Amazon exclusive signer [`beta.eks.amazonaws.com/app-serving`](https://docs.aws.amazon.com/eks/latest/userguide/cert-signing.html). Look at `client.V1CertificateSigningRequestSpec()` on `app/imageswap-init/imageswap-init.py`.
```
k8s_csr_spec = client.V1CertificateSigningRequestSpec(
groups=["system:authenticated"],
usages=["digital signature", "key encipherment", "server auth"],
request=base64.b64encode(csr_pem).decode("utf-8").rstrip(),
signer_name="beta.eks.amazonaws.com/app-serving"
)
```

ImageSwap v1.4.0 has major changes

>**MAPS LOGIC:** There is a new [MAPS](#maps-mode) mode logic that has been added to allow for more flexibility in the image swapping logic.
>The existing logic, referred to as `LEGACY` mode, is still available, but has been deprecated.
>To continue using the `LEGACY` mode logic set the `IMAGESWAP_MODE` environment variable accordingly. Please reference the [configuration](#configuration) section for more information.
>**Image Definition Preservation:** Updates have been made to how image definitions are processed during a swap. Previously the swap logic would drop the image org/project before adding the prefix (ie. `nginx/nginx-ingress:latest` would drop the `nginx/` portion of the image definition).
>In v1.4.0+ the swap logic will preserve all parts of the image except the Registry (ie. `docker.io/nginx/nginx-ingress` will drop the `docker.io` only from the image definition).
Find version/environment specific notices and information [here](./docs/notice.md)

## Overview

- [Prereqs](#prereqs)
- [Quickstart](#quickstart)
- [Health Check](#health-check)
- [Image](#image)
- [Configuration](#configuration)
- [Internals](./docs/internals.md)
- [Configuration](./docs/configuration.md)
- [Advance Install](./docs/advanced_install.md)
- [Metrics](#metrics)
- [Testing](#testing)
- [Cautions](#cautions)
- [Troubleshooting](#troubleshooting)
- [Operations](./docs/operations.md)
- [Contributing](./CONTRIBUTING.md)
- [Adopters](./ADOPTERS.md)

## Prereqs

Kubernetes 1.19.0 or above with the `admissionregistration.k8s.io/v1` (or higher) API enabled. Verify that by the following command:

```shell
$ kubectl api-versions | grep admissionregistration.k8s.io/v1
```

The result should be:

```shell
admissionregistration.k8s.io/v1
```

In addition, the `MutatingAdmissionWebhook` and `ValidatingAdmissionWebhook` admission controllers should be added and listed in the correct order in the admission-control flag of kube-apiserver.

### Permissions

ImageSwap requires cluster-admin permissions to deploy to Kubernetes since it requires access to create/read/update/delete cluster scoped resources (MutatingWebhookConfigurations, Certificates, etc.)

### Quickstart

You can use the following command to install ImageSwap from this repo with sane defaults

**NOTE:** The quickstart installation is not meant for production use. Please read through the [Cautions](#cautions) sections, and as always, use your best judgement when configuring ImageSwap for production scenarios.

```shell
$ kubectl apply -f https://raw.githubusercontent.com/phenixblue/imageswap-webhook/v1.5.2/deploy/install.yaml
$ kubectl apply -f https://raw.githubusercontent.com/phenixblue/imageswap-webhook/v1.5.3/deploy/install.yaml
```

#### This will do the following
Expand Down Expand Up @@ -118,240 +72,3 @@ $ kubectl apply -f ./testing/deployments/test-deploy01.yaml -n test1

$ kubectl apply -f ./testing/deployments/test-deploy02.yaml -n test1
```

## Image

ImageSwap uses a couple of images for operation

- [imageswap-init](./app/imageswap-init/Dockerfile)
- [imageswap](./app/imageswap/Dockerfile)

### Init Container

ImageSwap uses the `imageswap-init` init-container to generate/rotate a TLS cert/key pair to secure communication between the Kubernetes API and the webhook. This action takes place on Pod startup.

## Configuration

A new `IMAGESWAP_MODE` environment variable has been added to control the imageswap logic for the webhook. The value should be `LEGACY` or `MAPS` (new default).

### MAPS Mode

MAPS Mode enables a high degree of flexibility for the ImageSwap Webhook.

In MAPS mode, the webhook reads from a `map file` that defines one or more mappings (key/value pairs) for imageswap logic. With the `map file` configuration, swap logic for multiple registries and patterns can be configured. In contrast, the LEGACY mode only allowed for a single `IMAGE_PREFIX` to be defined for image swaps.

A `map file` is composed of key/value pairs separated by a `::` and looks like this:

```
default::default.example.com
docker.io::my.example.com/mirror-
quay.io::quay.example3.com
gitlab.com::registry.example.com/gitlab
#gcr.io:: # This is a comment
cool.io::
registry.internal.twr.io::registry.example.com
harbor.geo.pks.twr.io::harbor2.com ###### This is a comment with many symbols
noswap_wildcards::twr.io, walrus.io
# exact image mapping (full docker image name)
[EXACT]ghcr.io/fantasy/coolstuff:v1.0::my-local-registry.com/patched-coolstuff:latest
# replace image mapping (unix shell-style wildcards)
[REPLACE]ghcr.io/public*::my-local-registry.com
```

NOTE: Lines in the `map file` that are commented out with a leading `#` are ignored. Trailing comments following a map definition are also ignored.

NOTE: Previous versions of ImageSwap used a single `:` syntax to separate the key and value portions of a map definition. This syntax is deprecated as of v1.4.3 and will be removed in future versions. Please be sure to update any existing map file configurations to use the new syntax (ie. `::`).

NOTE: Prior to v1.4.3 any use of a registry that includes a port for the key of a map definition will result in errors.

The only mapping that is required in the `map_file` is the `default` map. The `default` map alone provides similar functionality to the `LEGACY` mode.

A map definition that includes a `key` only can be used to disable image swapping for that particular registry.

A map file can also include a special `noswap_wildcards` mapping that disables swapping based on greedy pattern matching. Don't actually include an `*` in this section. A value of `example` is essentially equivalent to `*example*`. [See examples below for more detail](#example-maps-configs)

By adding additional mappings to the `map file`, you can have much finer granularity to control swapping logic per registry.

#### Exact Image Mapping

Map definitions can become explicit mappings for individual images by using the `[EXACT]` prefix.

Usage:

```
[EXACT]<source-image>::<target-image>
```

`Exact` maps will be matched exactly against the `<source-image>` name and replaced with the `<target-image>` name. No inferences for registry (ie. `docker.io/`), or tag (ie. `:latest`) will be inferred for `Exact` maps.

Exact image matches are handled before all other mapping rules.

#### Replace Image Mapping

Image paths can be completely rewritten by using the `[REPLACE]` prefix.

Usage:

```
[REPLACE]<pattern>::<replacement>
```

When a `<source-image>` matches against a `Replace` rule, the image will be transformed to `<replacement><image>` where `<image>` includes the tag if it was specified as a part of `<source-image>`.

Replacement rules are handled after exact match rules, but before the rest.

Module used: [fnmatch](https://docs.python.org/3/library/fnmatch.html) — Unix filename pattern matching

#### Example MAPS Configs

- Disable image swapping for all registries EXCEPT `gcr.io`

```
default:
gcr.io::harbor.internal.example.com
```

- Enable image swapping for all registries except `gcr.io`

```
default::harbor.internal.example.com
gcr.io::
```

- Imitate LEGACY functionality as close as possible

```
default::harbor.internal.example.com
noswap_wildcards::harbor.internal.example.com
```

With this, all images will be swapped except those that already match the `harbor.internal.example.com` pattern

- Enable swapping for all registries except those that match the `example.com` pattern

```
default::harbor.internal.example.com
noswap_wildcards::example.com
```

With this, images that have any part of the registry that matches `example.com` will skip the swap logic

EXAMPLE:
- `example.com/image:latest`
- `external.example.com/image:v1.0`
- `edge.example.com/image:latest`)

- Enable swapping for all registries, but skip those that match the `example.com` pattern, except for `external.example.com`

```
default:harbor.internal.example.com
external.example.com:harbor.internal.example.com
noswap_wildcards:example.com
```

With this, the `edge.example.com/image:latest` image would skip swapping, but `external.example.com/image:latest` would be swapped to `harbor.internal.example.com/image:latest`

- Enable different swapping for top level "library" images vs. images that are nested under a project/org

Example library image: `nginx:latest`

This format is a shortcut for `docker.io/library/nginx:latest`

[Official Docker documentation on image naming](https://docs.docker.com/registry/introduction/#understanding-image-naming)

```
default::
docker.io::
docker.io/library::harbor.example.com/library
```

This map uses a special syntax of adding `/library` to a registry for the key in map file.

With this, the `nginx:latest` image would be swapped to `harbor.example.com/library/nginx:latest`, but the `tmobile/magtape:latest` image would be swapped to `harbor.example.com/tmobile/magtape:latest`

This configuration can be useful for scenarios like [Harbor's](https://goharbor.io) [image proxy cache](https://goharbor.io/docs/2.1.0/administration/configure-proxy-cache/) feature].

### LEGACY Mode

**DEPRECATED: This mode will be removed in a future release**

Change the `IMAGE_PREFIX` environment variable definition in the [imageswap-env-cm.yaml](./deploy/manifests/imageswap-env-cm.yaml) manifest to customize the repo/registry for the image prefix mutation.

### Granularly Disable Image Swapping for a Workload

You can also customize the label used to granularly disable ImageSwap on a per workload basis. By default the `k8s.twr.io/imageswap` label is used, but you can override that by specifying a custom label with the `IMAGESWAP_DISABLE_LABEL` environment variable.

The value of the label should be `disabled`.

See the [Break Glass: Per Workload](#per-workload) section for more details.

## Metrics

Prometheus formatted metrics for API rquests are exposed on the `/metrics` endpoint.

## Testing

Assuming you've followed the quickstart steps

- Review Deployment and Pod spec to validate the webhook is working

```shell
$ kubectl get deploy hello-world -n test1 -o yaml
$ kubectl get pods -n test1
$ kubectl get pod <pod_name> -n test1 -o yaml
```

NOTE: You should see the swapped image definition instead of the original definition in the `test-deploy.yaml` manifest.

## Cautions

### Production Considerations

- By Default the ImageSwap Mutating Webhook Configuration is set to fail "closed". Meaning if the webhook is unreachable or doesn't return an expected response, requests to the Kubernetes API will be blocked. Please adjust the configuration if this is not something that fits your environment.
- ImageSwap supports operation with multiple replicas that can increase availability and performance for critical clusters.
- The certificate generated by the `imageswap-init` container is valid for 12 months and will be automatically rotated once the Pod restarts within 6 months of expiration. If the certificate expires, calls to the webhook wil fail. Make sure you plan for this certificate rotation.
### Break Glass Scenarios
#### Per Workload
ImageSwap can be disabled on a per workload level by adding the `k8s.twr.io/imageswap` label with a value of `disabled` to the pod template.
Refer to this test manifest as an example: [./testing/deployments/test-deploy05.yaml](./testing/deployments/test-deploy05.yaml)
#### Per Namespace
ImageSwap can be enabled and disabled on a per namespace basis by utilizing the `k8s.twr.io/imageswap` label on the namespace resources. In emergency situations the label can be removed from a namespace to disable image swapping in that namespace.
#### Cluster Wide
If there are cluster-wide issues you can disable ImageSwap completely by removing the `imagewap-webhook` Mutating Webhook Configuration and deleting the ImageSwap deployment.
## Troubleshooting
### Run Docker Image Locally
```
$ docker run -p 5000:5000/tcp -it imageswapwebhook_app bash
$ ./deny-env.py
```
### Access Kubernetes Service without Ingress/LB
```shell
$ kubectl get pods # to get the name of the running pod
$ kubectl port-forward <pod_name> 5000:5000
```
### Use Curl to perform HTTP POST to webhook server
```shell
$ curl -vX POST https://localhost:5000/ -d @test.json -H "Content-Type: application/json"
```
### Follow logs of the webhook pod
```shell
$ kubectl get pods # to get the name of the running pod
$ kubectl logs <pod_name> -f
```
Loading

0 comments on commit 7706b43

Please sign in to comment.