Zombie tests build pipeline rework (#2159)

* add usrmerge to zombie-builder build step * new zombie_builder.Dockerfile * testing ci for zombie-builder * Replace minideb with the official Rust image. * testing images for zombie_tester * test pkg dependencies for zombie-builder * kagome-dev minideb-testing Docker * add libraries for zombie_builder docker image * enable cache on zombie-builder step * zombie builder image group update * change repo for zombie builder, separate builder steps * add working directory * update image path for zombie-builder * building_binary step for zombie builder * github inputs test * test sccache * Exit code disabled * check sccache working * Revert "check sccache working" This reverts commit ab18767. * make dirs when CI zombie-builder starts * show build result for polkadot_binary * change runner for zombie-builder * zombie builder, change paths * build apt package pipeline * docker-builder - add debug info * zombie-builder ci architecture fix * New kagome dev minideb builder * temporarily ci changes (autotests) * enable kagome-builder-deb workflow * zombie_builder and zombie_tester updated * zombie_builder syntax fixes * zombie builder - build logic variable added * save zombie-builder env to file and restore it * New build kagome Makefile for kagome-dev * Zombie-Tests kagome build, release by default * new build kagome pipeline - activate cache * enable show info when build deb packet * update dependencies for packet * test update * Check package version exists in build steps * show package version in build step * add date to version apt package name * Test zombie-tests job * update run zombienet test * update Google Auth for zombie-tests ci * Run zombie-tests on actions-runner-controller * wasmedge get SHA script * Runtime cache build packet - first iteration * ci runtime build fix * Kagome runtime ci update * zombie test -> ubuntu latest * Revert "zombie test -> ubuntu latest" This reverts commit ff582d2. * add hunter cache to safe.directory * CI change cache using config * Enable swap on Ubuntu zombie-tests * change cache logic in kagome build step * Reduce kagome validators in 0001 test * change swap size - 4G (for ubuntu runners) * update cache save settings - build kagome step * Fix zndsl * zombie-tests runs on actions-runner-controller * disable swap * disable broken zombie tests * zombie test - kagome build - google cache * disable broken zombie-tests * zombie tests - disable always save cache, change cache action * return github action version * change gcp github action version * testing actions step * return github cache save * select working zombie tests * runtime cache build update * Rebuild runtime option in CI * update zombienet simlinks for tester image * Zombie-Tests runs-on -> ubuntu-latest * Build runtime -> actions-runner-controller * Clean old files, clean Zombie tests Makefile, change version numeric * Zombienet Makefile new line fix * Clear Kagome docker Build makefile * use default kagome_dev version in runtime_cache ci build * runtime cache version ci fix * Add kagome-crates dependency * Switch to tag * Update main build pipeline (zombie tests new pipeline) * fix ci errors in test.yaml * Build type for Zombie tests added * code cleanup * Zombie tests in main pipeline -> actions-runner-controller * change mount dirs for docker build * cleaning cache and default build typo for main build workflow * disable external project * add CI env to workflows * -Werror param support for CI building * add check -Werror compiler support * Change the order of applying -Werror * add_flag cmake function update --------- Co-authored-by: Kirill Azovtsev <[email protected]> Co-authored-by: kamilsa <[email protected]> Co-authored-by: Zak Fein <[email protected]> Co-authored-by: Harrm <[email protected]>
qdrvm · Jul 30, 2024 · 0c912ae · 0c912ae
1 parent 1b2c3f8
commit 0c912ae
Show file tree

Hide file tree

Showing 23 changed files with 1,647 additions and 409 deletions.
diff --git a/.github/workflows/action-validator_darwin_arm64 b/.github/workflows/action-validator_darwin_arm64
diff --git a/.github/workflows/build.yaml b/.github/workflows/build.yaml
@@ -0,0 +1,22 @@
+#
+# Copyright Quadrivium LLC
+# All Rights Reserved
+# SPDX-License-Identifier: Apache-2.0
+#
+
+name: Manual Workflow Dispatch
+
+on:
+  workflow_dispatch:
+
+jobs:
+  example-job:
+    runs-on: ubuntu-latest
+
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@v3
+
+      - name: Print template message
+        run: echo "This is a template for testing workflows in branches."
+
diff --git a/.github/workflows/kagome-builder-deb.yaml b/.github/workflows/kagome-builder-deb.yaml
@@ -0,0 +1,102 @@
+#
+# Copyright Quadrivium LLC
+# All Rights Reserved
+# SPDX-License-Identifier: Apache-2.0
+#
+
+name: Kagome Builder | Debian (Bookworm)
+
+on:
+  #  schedule:
+  #    - cron: '0 2 * * 0' # Every Sunday at 2:00 AM
+  workflow_dispatch:
+    inputs:
+      kagome_image_rebuild:
+        description: 'Rebuild Kagome Builder Minideb image'
+        required: false
+        default: 'false'
+        type: 'boolean'
+      runtime_cache_rebuild:
+        description: 'Rebuild Kagome Runtime Cache'
+        required: false
+        default: 'false'
+        type: 'boolean'
+
+env:
+  DOCKER_REGISTRY_PATH: ${{ secrets.GCP_REGISTRY }}/${{ secrets.GCP_PROJECT }}/kagome-dev/
+
+jobs:
+  build_polkadot_builder:
+    runs-on: ubuntu-latest
+    timeout-minutes: 60
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@v4
+
+      - name : "Authenticate with Google Cloud"
+        uses: 'google-github-actions/auth@v2'
+        with:
+          credentials_json: ${{ secrets.GCP_SERVICE_ACCOUNT_KEY }}
+
+      - name: "Set up Cloud SDK"
+        uses: 'google-github-actions/setup-gcloud@v2'
+
+      - name: "Configure Docker for GCR"
+        run: |
+          gcloud auth configure-docker --quiet
+          gcloud auth configure-docker ${{ secrets.GCP_REGISTRY }} --quiet
+
+      - name: "Check if image tag exists"
+        id: check-tag
+        working-directory: ./housekeeping/docker/kagome-dev
+        run: |
+          IMAGE_TAG=$(grep '^MINIDEB_IMAGE' Makefile | awk -F '[@:]' '{print $3}' | cut -c 1-7)_rust-$(grep '^RUST_VERSION' Makefile | sed 's/.*?= //')
+          KAGOME_BUILDER_DEB_IMAGE=${DOCKER_REGISTRY_PATH}kagome_builder_deb:${IMAGE_TAG}
+          TAG_EXISTS=$(docker manifest inspect ${KAGOME_BUILDER_DEB_IMAGE} > /dev/null 2>&1 && echo "true" || echo "false")
+          echo "exists=$TAG_EXISTS" >> $GITHUB_ENV
+          echo "kagome_builder_deb=KAGOME_BUILDER_DEB_IMAGE" >> GITHUB_OUTPUT
+
+      - name: "Build Polkadot Builder"
+        working-directory: ./housekeeping/docker/kagome-dev
+        if: env.exists == 'false' || github.event.inputs.kagome_image_rebuild == 'true'
+        run: make kagome_builder_deb \
+          DOCKER_REGISTRY_PATH=${DOCKER_REGISTRY_PATH}
+
+      - name: "Push Polkadot Builder"
+        working-directory: ./housekeeping/docker/kagome-dev
+        if: env.exists == 'false' || github.event.inputs.kagome_image_rebuild == 'true'
+        run: make kagome_builder_deb_push \
+          DOCKER_REGISTRY_PATH=${DOCKER_REGISTRY_PATH}
+
+  build_kagome_runtime:
+    runs-on: [ actions-runner-controller ]
+    timeout-minutes: 60
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@v4
+
+      - name : "Authenticate with Google Cloud"
+        uses: 'google-github-actions/auth@v2'
+        with:
+          credentials_json: ${{ secrets.GCP_SERVICE_ACCOUNT_KEY }}
+
+      - name: "Set up Cloud SDK"
+        uses: 'google-github-actions/setup-gcloud@v2'
+
+      - name: "Configure Docker for GCR"
+        run: |
+          gcloud auth configure-docker --quiet
+          gcloud auth configure-docker ${{ secrets.GCP_REGISTRY }} --quiet
+
+      - name: "Build Runtime Cache"
+        working-directory: ./housekeeping/docker/kagome-dev
+        if: github.event.inputs.runtime_cache_rebuild == 'true'
+        run: make runtime_cache \
+          DOCKER_REGISTRY_PATH=${DOCKER_REGISTRY_PATH}
+
+      - name: "Push Runtime Cache"
+        working-directory: ./housekeeping/docker/kagome-dev
+        if: github.event.inputs.runtime_cache_rebuild == 'true'
+        run: make upload_apt_package_runtime \
+          DOCKER_REGISTRY_PATH=${DOCKER_REGISTRY_PATH}
+
diff --git a/.github/workflows/telegram_notify.yml b/.github/workflows/telegram_notify.yml
@@ -1,3 +1,9 @@
+#
+# Copyright Quadrivium LLC
+# All Rights Reserved
+# SPDX-License-Identifier: Apache-2.0
+#
+
 name: Telegram Notification
 
 on: