Docker support enhancement

[hero-intelligent]

For Volunteers, Developers, and Experts in docker and github action

Waiting for merge. #289 (22 Sept 2023)

This is an article by a non-programmer volunteer to share his experience of enhancing docker support.

Achievements:

Compared to current stage, my version enables i386, armv7, armv8 and arm64(maybe a duplicate of armv8), besides of amd64. Image will be built once and then push to ghcr, quay and dockerhub, with tag of either version or date, as well as latest. (22 Sept 2023)

Limitations and explanation:

I've tried my best. As for paralleled image build process using matrix, I've kind of gave up.

The whole build process now will run about one hour, which may cause a problem of hitting the limit of free action access of 2000 minutes per month per account. Time is Money, literally.

As for the parallel one, a single run takes only about half an hour, but now it is entirely unusable.
Images now can only push to one registry. If you want to push to another registry, you have to run multiple times, which takes even longer than the chronological one.
As for supporting armv5 and v6, it can only be achieved by parallel build process, for images must be built and pushed at the same time, otherwise the latter push will overwrite the former one. Besides, I need to write another Dockerfiles to achieve these architecture.

Mips64le (solely debian based)
armv5 (solely debian based)
armv6 (solely alpine based)
Others (self build from scratch)

(till 22 Sept 2023)

Goals to be achieved:

Platform support for more platforms
Parallel build process (faster, clearer)

First version of parallel workflow (Unusable)

KEEP FOR FUTURE USE
Never edit unless you are sure about what you are doing!

./publish.Dockerfile

# ATTENTION This part below is for publishing purpose only
# NOTE in the as parameter in FROM,
# the `--` between `web` and `has`, and `dokerfile` in which letter c is missing
# I do it on purpose so that it is not convenient for self build to avoid unnecessary problem

FROM golang:1.21-bookworm as build-bundle-directly-for-web--has-been-built-from-outside-of-dokerfile

RUN \
    apt-get update; apt-get install -y git make llvm-15 clang-15; \
    apt-get clean autoclean && apt-get autoremove -y && rm -rf /var/lib/{apt,dpkg,cache,log}/

# build bundle process
ENV CGO_ENABLED=0
ENV CLANG=clang-15
ARG DAED_VERSION=self-build

WORKDIR /build

COPY ./dist/ ./web/
COPY ./wing/ ./wing/
# RUN git clone --recursive https://github.com/daeuniverse/dae-wing.git wing

WORKDIR /build/wing

RUN make APPNAME=daed VERSION=$DAED_VERSION OUTPUT=daed WEB_DIST=/build/web/ bundle



# ATTENTION This part below is for publishing purpose only
# NOTE in the from parameter in COPY,
# the `--` between `web` and `has`, and `dokerfile` in which letter c is missing
# I do it on purpose so that it is not convenient for self build to avoid unnecessary problem

FROM alpine as this-image-is-built-for-final-publish-with-support-of-multiple-platforms

LABEL org.opencontainers.image.source=https://github.com/daeuniverse/daed

RUN mkdir -p /usr/local/share/daed/
RUN mkdir -p /etc/daed/
RUN wget -O /usr/local/share/daed/geoip.dat https://github.com/v2rayA/dist-v2ray-rules-dat/raw/master/geoip.dat; \
    wget -O /usr/local/share/daed/geosite.dat https://github.com/v2rayA/dist-v2ray-rules-dat/raw/master/geosite.dat
COPY --from=build-bundle-directly-for-web--has-been-built-from-outside-of-dokerfile /build/wing/daed /usr/local/bin

EXPOSE 2023

CMD ["daed", "run", "-c", "/etc/daed"]

./.github/workflow/test-publish-docker-image.yml

name: TEST MATRIX Publish Docker Image

on:
  workflow_dispatch:
  # push:
  #   branches:
  #     - "main"
  release:
    types: [prereleased,released]

env:
  REGISTRY_IMAGE: ghcr.io/daeuniverse/daed
  # REGISTRY_IMAGE: quay.io/daeuniverse/daed
  # REGISTRY_IMAGE: daeuniverse/daed

jobs:

  build-web:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
        # with:
        #   submodules: 'recursive'

      - uses: pnpm/action-setup@v2
        with:
          version: latest

      - uses: actions/setup-node@v3
        with:
          cache: pnpm
          node-version: latest

      - name: Build
        run: |
          pnpm install
          pnpm build

      - name: Upload artifact - web
        uses: actions/upload-artifact@v3
        with:
          name: web
          path: dist



  build-docker-image:
    runs-on: ubuntu-latest
    needs:
      - build-web
    strategy:
      fail-fast: false
      matrix:
        platform:
          - linux/386
          - linux/amd64
          # - linux/arm/v5
          # - linux/arm/v6
          - linux/arm/v7
          - linux/arm64/v8
          - linux/arm64
          # - linux/mips64le
          # - linux/ppc64e
          # - linux/s390x
    steps:

      - name: Checkout
        uses: actions/checkout@v4
        with:
          submodules: 'recursive'

      - name: Download artifact - web
        uses: actions/download-artifact@v3
        with:
          name: web
          path: dist/

      - name: Docker meta
        id: meta
        uses: docker/metadata-action@v5
        with:
          images: ${{ env.REGISTRY_IMAGE }}

      - name: Set up QEMU
        uses: docker/setup-qemu-action@v3

      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v3

#       - name: Login to ghrc.io
#         uses: docker/login-action@v2
#         with:
#           registry: ghcr.io
#           username: ${{ github.actor }}
#           password: ${{ secrets.GITHUB_TOKEN }}

#       - name: Login to quay.io
#         uses: docker/login-action@v3
#         with:
#           registry: quay.io
#           username: ${{ github.repository_owner }}
#           password: ${{ secrets.QUAY_PASS }}

      - name: Login to Docker Hub
        uses: docker/login-action@v3
        with:
          username: ${{ secrets.DOCKERHUB_USERNAME }}
          password: ${{ secrets.DOCKERHUB_TOKEN }}

      - name: Build and push by digest
        id: build
        uses: docker/build-push-action@v5
        with:
          context: .
          platforms: ${{ matrix.platform }}
          file: publish.Dockerfile
          target: this-image-is-built-for-final-publish-with-support-of-multiple-platforms
          labels: ${{ steps.meta.outputs.labels }}
          outputs: type=image,name=${{ env.REGISTRY_IMAGE }},push-by-digest=true,name-canonical=true,push=true

      - name: Export digest
        run: |
          mkdir -p /tmp/digests
          digest="${{ steps.build.outputs.digest }}"
          touch "/tmp/digests/${digest#sha256:}"

      - name: Upload digest
        uses: actions/upload-artifact@v3
        with:
          name: digests
          path: /tmp/digests/*
          if-no-files-found: error
          retention-days: 1



  merge:
    runs-on: ubuntu-latest
    needs:
      - build-docker-image
    steps:

      - name: Download digests
        uses: actions/download-artifact@v3
        with:
          name: digests
          path: /tmp/digests

      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v3

      - name: Docker meta
        id: meta
        uses: docker/metadata-action@v5
        with:
          images: ${{ env.REGISTRY_IMAGE }}

#       - name: Login to ghrc.io
#         uses: docker/login-action@v2
#         with:
#           registry: ghcr.io
#           username: ${{ github.actor }}
#           password: ${{ secrets.GITHUB_TOKEN }}

#       - name: Login to quay.io
#         uses: docker/login-action@v3
#         with:
#           registry: quay.io
#           username: ${{ github.repository_owner }}
#           password: ${{ secrets.QUAY_PASS }}

      - name: Login to Docker Hub
        uses: docker/login-action@v3
        with:
          username: ${{ secrets.DOCKERHUB_USERNAME }}
          password: ${{ secrets.DOCKERHUB_TOKEN }}

      - name: Create manifest list and push
        working-directory: /tmp/digests
        run: |
          docker buildx imagetools create $(jq -cr '.tags | map("-t " + .) | join(" ")' <<< "$DOCKER_METADATA_OUTPUT_JSON") \
            $(printf '${{ env.REGISTRY_IMAGE }}@sha256:%s ' *)

      - name: Inspect image
        run: |
          docker buildx imagetools inspect ${{ env.REGISTRY_IMAGE }}:${{ steps.meta.outputs.version }}

---

Expalanations generated by ChatGPT

---

build and push by digest & Export digest

      - name: Build and push by digest
        id: build
        uses: docker/build-push-action@v5
        with:
          context: .
          platforms: ${{ matrix.platform }}
          file: publish.Dockerfile
          target: this-image-is-built-for-final-publish-with-support-of-multiple-platforms
          labels: ${{ steps.meta.outputs.labels }}
          outputs: type=image,name=${{ env.REGISTRY_IMAGE }},push-by-digest=true,name-canonical=true,push=true

      - name: Export digest
        run: |
          mkdir -p /tmp/digests
          digest="${{ steps.build.outputs.digest }}"
          touch "/tmp/digests/${digest#sha256:}"

Result:

This is a snippet of a CI/CD configuration in GitHub Actions. GitHub Actions can be used to build, test, and deploy your code right from GitHub.

Here's a breakdown of the code:

1. Build and push by digest:
   The docker/build-push-action@v5 action provided by Docker is used here. This action enables you to build a docker image based on a context (usually where your Dockerfile is located, denoted with . in this case) and push it to a Docker registry.

   uses: docker/build-push-action@v5 is specifying which GitHub action is being used.

   In the with: block, various input parameters for this action are defined such as:

   • context: . : The build's context is the set of files located in the specified PATH or URL. Here . refers to the current directory.

   • platforms: ${{ matrix.platform }} : This denotes the platforms for which the Docker image is built. The platforms are specified in a build matrix.

   • file: publish.Dockerfile : This specifies the Dockerfile to use for building the image.

   • target: ... : This specifies the build stage in the Dockerfile to target during the Docker build.

   • labels: ${{ steps.meta.outputs.labels }} : Labels applied to the Docker images.

   • outputs: type=image,... : These are the parameters Docker uses when building and pushing the image.

2. Export digest:

   - name: Export digest : This is another step in the GitHub Actions Workflow.

   run: | : The run key is used to run inline commands within the current runner.

   The code within the run block is a shell script:

   • digest="${{ steps.build.outputs.digest }}" : This set the digest from the previous "build" step to a shell variable called digest.

   • mkdir -p /tmp/digests : Creates a new directory 'digests' in /tmp if it doesn't exist.

   • touch "/tmp/digests/${digest#sha256:}" : Creates an empty file with the name equivalent to the Docker image digest, minus the 'sha256:' prefix, inside the '/tmp/digests/' directory. The code ${digest#sha256:} is a parameter expansion in shell to strip 'sha256:' from the docker image digest.

This workflow is useful in automatic build and deployment scenarios where you want to keep track of Docker image digests for later usage, such as deployment to Kubernetes.

Create manifest list and push

docker buildx imagetools create $(jq -cr '.tags | map("-t " + .) | join(" ")' <<< "$DOCKER_METADATA_OUTPUT_JSON") $(printf '${{ env.REGISTRY_IMAGE }}@sha256:%s ' *)

Result:

This command is a combination of numerous Docker and shell operations. Let's decompose it part by part:

1. docker buildx imagetools create: This command creates a new image from a source image. The source is typically a set of tagged images with common meta data. This command does not push the created image. It just returns the ID for the image.

2. $(jq -cr '.tags | map("-t " + .) | join(" ")' <<< "$DOCKER_METADATA_OUTPUT_JSON"): This command is a sub-shell command which will first be evaluated before the outer command. It's using jq, a lightweight and flexible command-line JSON processor, to parse JSON contained in the environment variable $DOCKER_METADATA_OUTPUT_JSON specifically extracting the 'tags', prefixing them with -t which is Docker's tag option and joining them with spaces. This is useful for tagging images in Docker.

3. $(printf '${{ env.REGISTRY_IMAGE }}@sha256:%s ' *): This is another sub-shell command using printf. It is generating a string by substituting values into a template. Here, it replaces %s with a string representation of all arguments supplied (*). The env.REGISTRY_IMAGE is a placeholder for a registry image in GitHub Actions syntax.

In essence, this command constructs a new image including all the specified tags from the DOCKER_METADATA_OUTPUT_JSON variable and from the corresponding images in the image registry.

If you don't understand how the whole command works together, try breaking it down into smaller parts. Run jq -cr '.tags | map("-t " + .) | join(" ")' <<< "$DOCKER_METADATA_OUTPUT_JSON" alone and see what it returns. Do the same for printf '${{ env.REGISTRY_IMAGE }}@sha256:%s ' *.

Remember to replace DOCKER_METADATA_OUTPUT_JSONand env.REGISTRY_IMAGE with your actual values if you attempt to run this command.

Further Expalanations of sub-shell commands

sub-shell command 1

jq -cr '.tags | map("-t " + .) | join(" ")' <<< "$DOCKER_METADATA_OUTPUT_JSON"

Result:

This command uses jq, which is a command-line JSON processor. This is a lightweight and flexible command-line tool that provides powerful functionalities such as equality and relational operations, iterative and recursive descent, conditional processing, arbitrary precision arithmetic, and much more with JSON data structures.

Here's the breakdown of the command:

1. jq -cr : jq is the command being used, the -c option means that the output will be in compact format and the -r option means jq will output raw strings and not JSON-encoded strings.

2. '.tags' : Here, jq is being asked to filter the input JSON. The .tags is a filter expression telling jq to select the 'tags' field from the root object of the input JSON.

3. | map("-t " + .) : This expression maps or transforms an array. Here "map" applies a filter ("-t " + .) to each item in the tags array. The filter is appending "-t " to the beginning of each tag.

4. join(" ") : This command is joining the array elements separated by spaces into a single string.

5. <<< "$DOCKER_METADATA_OUTPUT_JSON" : This is a 'here string' in bash. It's feeding the JSON content which is stored in variable DOCKER_METADATA_OUTPUT_JSON to the jq command.

So, in summary, this command is transforming the 'tags' array from the JSON input in DOCKER_METADATA_OUTPUT_JSON into a space-separated list of strings, with each string preceded by '-t '.

sub-shell command 2

printf '${{ env.REGISTRY_IMAGE }}@sha256:%s ' *

Result:

This line of shell scripting is used to print a string that includes an environment variable value and a placeholder for another string. More specifically, it's designed to output a Docker image reference by its digest, which a form to pin an image to a specific version.

The command printf prints a formatted string to the console, similar to the way printf() works in languages like C and JavaScript.

The syntax ${{ env.REGISTRY_IMAGE }} is essentially a placeholder for an environment variable named REGISTRY_IMAGE. Environment variables are handy for storing system-wide values that you don't want to hard code into your scripts; in this case, the name of a Docker image in a registry.

@sha256:%s is a way to tag a Docker image using its checksum instead of its version number, which can be useful to ensure that you're always pulling an identical copy of an image, down to the byte.

%s is a placeholder for a string in printf syntax, and '*' is a wildcard that would match any file or directory.

Together, this command would take each file or directory in your current location, and for each one, it would print a line that included the environment variable REGISTRY_IMAGE, @sha256:, and the name of the file or directory.

However, the pattern '*' in printf does not provide the SHA256 checksum you need. This would merely print out the file or directory name, and not their SHA256 checksum. For outputting the checksum, you would need to leverage commands such as sha256sum or openssl dgst -sha256.

As a side note, this syntax ${{ env.VARIABLE_NAME }} is typical in Github Actions for accessing environment variables. If you run the script in a typical shell, you'd rather use $VARIABLE_NAME. Double check the context in which you are using this line of code.

If you're interested in understanding deep into shell scripting, or the Docker's functionalities - I would recommend the following resources:

1. Advanced Bash-Scripting Guide
2. Docker Tutorial - Dockerize your Python Application.

Inspect image

docker buildx imagetools inspect ${{ env.REGISTRY_IMAGE }}:${{ steps.meta.outputs.version }}

Result:

This command is used in a GitHub Actions workflow to inspect a Docker image that was built and pushed to a Docker registry. It uses two GitHub Actions features: environment variables and step outputs. The code is actually running in a shell environment and it's a Docker command, not specific to any programming language.

• buildx imagetools inspect means the command is utilizing Docker Buildx, which is an extension for Docker that introduces full support for the features provided by Moby BuildKit builder toolkit. The 'imagetools inspect' functionality is used to display detailed information on the Docker image such as size, OS, architecture, created date, etc.

• ${{ env.REGISTRY_IMAGE }} is GitHub Actions syntax to reference the value of the environment variable REGISTRY_IMAGE. This contains the name of Docker image which is stored in the registry, often something like myorg/myrepo or docker.io/myorg/myrepo.

• ${{ steps.meta.outputs.version }} is also GitHub Actions syntax which references the version output of a step with id of meta. This would be defined elsewhere in your workflow YAML probably as a step of calculating the version, fetching it from somewhere, or generating it.

As it stands, this line would try to inspect an image as registry/repository:version. In order for this command to work, your shell environment must have Docker available, and the REGISTRY_IMAGE environment variable and version in steps.meta.outputs need to be set and point to valid values.

Sidenote: The command docker buildx imagetools inspect requires that Docker Buildx is used as the builder, as imagetools is not a command in the stock Docker command-line interface (CLI). Docker Buildx is included with Docker Desktop as an experimental feature but must typically be enabled and started manually for other installations.