Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ref(workflows): consolidate workflows based on their purpose #7616

Merged
merged 41 commits into from
Oct 18, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
41 commits
Select commit Hold shift + click to select a range
2cab03c
ref(workflows): consolidate workflows based on their purpose
gustavovalverde Sep 25, 2023
59e8f11
fix(workflows): use correct name for patch
gustavovalverde Sep 25, 2023
9f251b7
fix(workflow): docker unit tests
gustavovalverde Sep 25, 2023
6bc35ed
fix(release): validation error
gustavovalverde Sep 25, 2023
6306a7d
fix(workflows): reference correct name
gustavovalverde Sep 25, 2023
c78dbd2
fix: remove extra workflow
gustavovalverde Sep 25, 2023
964650f
fix(workflows): use larger runners
gustavovalverde Sep 25, 2023
8c6fb2f
Merge branch 'main' into refactor-workflows
gustavovalverde Sep 26, 2023
600b08a
Merge branch 'main' into refactor-workflows
gustavovalverde Sep 26, 2023
23b76ff
fix(workflow): remove code already in docker unit-test
gustavovalverde Sep 26, 2023
d5ca031
fix(unit-tests): start zebra the right way
gustavovalverde Sep 26, 2023
572b593
fix: typo in patch name
gustavovalverde Sep 26, 2023
b17c109
chore: move job to logical order
gustavovalverde Sep 26, 2023
bd6f72c
Merge branch 'main' into refactor-workflows
gustavovalverde Oct 4, 2023
43f089d
imp(workflows): use better name for gcp tests
gustavovalverde Oct 5, 2023
f3426d7
Merge branch 'main' into refactor-workflows
gustavovalverde Oct 5, 2023
7d15390
Merge branch 'main' into refactor-workflows
gustavovalverde Oct 9, 2023
76a6b8a
Merge branch 'main' into refactor-workflows
gustavovalverde Oct 11, 2023
f600196
add: missing merge changes
gustavovalverde Oct 11, 2023
3db9e6c
chore: use better name for find-disks
gustavovalverde Oct 11, 2023
eac74b8
fix(ci): use the `entrypoint.sh` to change the Network
gustavovalverde Oct 11, 2023
ecb938b
fix(ci): add missing `ZEBRA_CONF_PATH` variable
gustavovalverde Oct 11, 2023
6fe2870
fix(ci): allow to build the entrypoint file with testnet
gustavovalverde Oct 11, 2023
97e7a14
fix(entrypoint): allow to create a dir and file with a single variable
gustavovalverde Oct 11, 2023
43ae06e
refactor: test config file in CI and CD with a reusable workflow
gustavovalverde Oct 17, 2023
962e320
Merge branch 'main' into refactor-workflows
gustavovalverde Oct 17, 2023
4082340
fix(ci): wrong name used
gustavovalverde Oct 17, 2023
2bbed80
fix(ci): use checkout
gustavovalverde Oct 17, 2023
e7f6170
fix(ci): improve docker config tests
gustavovalverde Oct 17, 2023
77b744d
fix(ci): use better name for protection rules
gustavovalverde Oct 17, 2023
ed9aafe
Fix changed workflow file name in docs patch file
teor2345 Oct 17, 2023
99503ec
Apply suggestions from code review
gustavovalverde Oct 17, 2023
053e08e
fix(cd): depend on file tests
gustavovalverde Oct 17, 2023
2435cf3
fix(docs): adapt to new workflow name
gustavovalverde Oct 17, 2023
a0d259e
Merge branch 'main' into refactor-workflows
gustavovalverde Oct 17, 2023
100f95e
fix: revert test coverage on CD
gustavovalverde Oct 17, 2023
ee52a77
chore: reduce diff
gustavovalverde Oct 17, 2023
fd6aaf2
fix(ci): allow using variable images for reusable workflows
gustavovalverde Oct 17, 2023
3757299
fix(dockerfile): use variables or default for config path and file
gustavovalverde Oct 17, 2023
315a783
fix(entrypoint): if `$ZEBRA_CONF_PATH` is set, do not override it
gustavovalverde Oct 18, 2023
bbf8201
Fix patch job names and remove failure job testnet dependencies
teor2345 Oct 18, 2023
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
39 changes: 39 additions & 0 deletions .github/workflows/cd-deploy-nodes-gcp.patch.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,39 @@
name: Deploy Nodes to GCP

on:
pull_request:
paths-ignore:
# code and tests
- '**/*.rs'
# hard-coded checkpoints and proptest regressions
- '**/*.txt'
# dependencies
- '**/Cargo.toml'
- '**/Cargo.lock'
# configuration files
- '.cargo/config.toml'
- '**/clippy.toml'
# workflow definitions
- 'docker/**'
- '.dockerignore'
- '.github/workflows/cd-deploy-nodes-gcp.yml'
- '.github/workflows/sub-build-docker-image.yml'

jobs:
build:
name: Build CD Docker / Build images
runs-on: ubuntu-latest
steps:
- run: 'echo "No build required"'

test-configuration-file:
name: Test CD default Docker config file / Test default-conf in Docker
runs-on: ubuntu-latest
steps:
- run: 'echo "No build required"'

test-zebra-conf-path:
name: Test CD custom Docker config file / Test custom-conf in Docker
runs-on: ubuntu-latest
steps:
- run: 'echo "No build required"'
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
name: CD
name: Deploy Nodes to GCP

gustavovalverde marked this conversation as resolved.
Show resolved Hide resolved
# Ensures that only one workflow task will run at a time. Previous deployments, if
# already in process, won't get cancelled. Instead, we let the first to complete
Expand Down Expand Up @@ -47,8 +47,8 @@ on:
# # workflow definitions
# - 'docker/**'
# - '.dockerignore'
# - '.github/workflows/continous-delivery.yml'
# - '.github/workflows/build-docker-image.yml'
# - '.github/workflows/cd-deploy-nodes-gcp.yml'
# - '.github/workflows/sub-build-docker-image.yml'

# Only runs the Docker image tests, doesn't deploy any instances
pull_request:
Expand All @@ -66,14 +66,13 @@ on:
# workflow definitions
- 'docker/**'
- '.dockerignore'
- '.github/workflows/continous-delivery.yml'
- '.github/workflows/find-cached-disks.yml'
- '.github/workflows/cd-deploy-nodes-gcp.yml'
- '.github/workflows/sub-build-docker-image.yml'

release:
types:
- published


jobs:
# If a release was made we want to extract the first part of the semver from the
# tag_name
Expand Down Expand Up @@ -108,7 +107,7 @@ jobs:
# The image will be commonly named `zebrad:<short-hash | github-ref | semver>`
build:
name: Build CD Docker
uses: ./.github/workflows/build-docker-image.yml
uses: ./.github/workflows/sub-build-docker-image.yml
with:
dockerfile_path: ./docker/Dockerfile
dockerfile_target: runtime
Expand All @@ -118,99 +117,40 @@ jobs:

# Test that Zebra works using the default config with the latest Zebra version.
test-configuration-file:
name: Test Zebra CD Docker config file
timeout-minutes: 15
runs-on: ubuntu-latest
name: Test CD default Docker config file
needs: build
steps:
- uses: r7kamura/[email protected]

- name: Inject slug/short variables
uses: rlespinasse/github-slug-action@v4
with:
short-length: 7

# Make sure Zebra can sync at least one full checkpoint on mainnet
- name: Run tests using the default config
shell: /usr/bin/bash -exo pipefail {0}
run: |
docker pull ${{ vars.GAR_BASE }}/zebrad@${{ needs.build.outputs.image_digest }}
docker run --detach --name default-conf-tests -t ${{ vars.GAR_BASE }}/zebrad@${{ needs.build.outputs.image_digest }}

# Use a subshell to handle the broken pipe error gracefully
(
trap "" PIPE;
docker logs \
--tail all \
--follow \
default-conf-tests | \
tee --output-error=exit /dev/stderr | \
grep --max-count=1 --extended-regexp --color=always \
-e "net.*=.*Main.*estimated progress to chain tip.*BeforeOverwinter"
) || true
LOGS_EXIT_STATUS=$?

docker stop default-conf-tests

EXIT_STATUS=$(docker wait default-conf-tests || echo "Error retrieving exit status");
echo "docker exit status: $EXIT_STATUS";

# If grep found the pattern, exit with the Docker container exit status
if [ $LOGS_EXIT_STATUS -eq 0 ]; then
exit $EXIT_STATUS;
fi

# Handle other potential errors here
echo "An error occurred while processing the logs.";
exit 1;
uses: ./.github/workflows/sub-test-zebra-config.yml
with:
test_id: 'default-conf'
docker_image: ${{ vars.GAR_BASE }}/zebrad@${{ needs.build.outputs.image_digest }}
grep_patterns: '-e "net.*=.*Main.*estimated progress to chain tip.*BeforeOverwinter"'
test_variables: '-e NETWORK'
network: 'Mainnet'

# Test reconfiguring the docker image for testnet.
test-configuration-file-testnet:
name: Test testnet Zebra CD Docker config file
timeout-minutes: 15
runs-on: ubuntu-latest
name: Test CD testnet Docker config file
needs: build
steps:
- uses: r7kamura/[email protected]

- name: Inject slug/short variables
uses: rlespinasse/github-slug-action@v4
with:
short-length: 7

# Make sure Zebra can sync the genesis block on testnet
- name: Run tests using a testnet config
shell: /usr/bin/bash -exo pipefail {0}
run: |
docker pull ${{ vars.GAR_BASE }}/zebrad@${{ needs.build.outputs.image_digest }}
docker run --env "NETWORK=Testnet" --detach --name testnet-conf-tests -t ${{ vars.GAR_BASE }}/zebrad@${{ needs.build.outputs.image_digest }}
# Use a subshell to handle the broken pipe error gracefully
(
trap "" PIPE;
docker logs \
--tail all \
--follow \
testnet-conf-tests | \
tee --output-error=exit /dev/stderr | \
grep --max-count=1 --extended-regexp --color=always \
-e "net.*=.*Test.*estimated progress to chain tip.*Genesis" \
-e "net.*=.*Test.*estimated progress to chain tip.*BeforeOverwinter";
) || true
LOGS_EXIT_STATUS=$?

docker stop testnet-conf-tests

EXIT_STATUS=$(docker wait testnet-conf-tests || echo "Error retrieving exit status");
echo "docker exit status: $EXIT_STATUS";

# If grep found the pattern, exit with the Docker container exit status
if [ $LOGS_EXIT_STATUS -eq 0 ]; then
exit $EXIT_STATUS;
fi

# Handle other potential errors here
echo "An error occurred while processing the logs.";
exit 1;
# Make sure Zebra can sync the genesis block on testnet
uses: ./.github/workflows/sub-test-zebra-config.yml
with:
test_id: 'testnet-conf'
docker_image: ${{ vars.GAR_BASE }}/zebrad@${{ needs.build.outputs.image_digest }}
grep_patterns: '-e "net.*=.*Test.*estimated progress to chain tip.*Genesis" -e "net.*=.*Test.*estimated progress to chain tip.*BeforeOverwinter"'
test_variables: '-e NETWORK'
network: 'Testnet'

# Test that Zebra works using $ZEBRA_CONF_PATH config
test-zebra-conf-path:
name: Test CD custom Docker config file
needs: build
uses: ./.github/workflows/sub-test-zebra-config.yml
with:
test_id: 'custom-conf'
docker_image: ${{ vars.GAR_BASE }}/zebrad@${{ needs.build.outputs.image_digest }}
grep_patterns: '-e "v1.0.0-rc.2.toml"'
test_variables: '-e NETWORK -e ZEBRA_CONF_PATH="zebrad/tests/common/configs/v1.0.0-rc.2.toml"'
network: ${{ inputs.network || vars.ZCASH_NETWORK }}

# Deploy Managed Instance Groups (MiGs) for Mainnet and Testnet,
# with one node in the configured GCP region.
Expand All @@ -225,14 +165,14 @@ jobs:
# otherwise a new major version is deployed in a new MiG.
#
# Runs:
# - on every push/merge to the `main` branch
# - on every push to the `main` branch
# - on every release, when it's published
deploy-nodes:
strategy:
matrix:
network: [Mainnet, Testnet]
name: Deploy ${{ matrix.network }} nodes
needs: [ build, test-configuration-file, versioning ]
needs: [ build, versioning, test-configuration-file, test-zebra-conf-path ]
runs-on: ubuntu-latest
timeout-minutes: 60
permissions:
Expand Down Expand Up @@ -334,7 +274,7 @@ jobs:
# Note: this instances are not automatically replaced or deleted
deploy-instance:
name: Deploy single ${{ inputs.network }} instance
needs: [ build, test-configuration-file ]
needs: [ build, test-configuration-file, test-zebra-conf-path ]
runs-on: ubuntu-latest
timeout-minutes: 30
permissions:
Expand Down Expand Up @@ -401,7 +341,7 @@ jobs:
failure-issue:
name: Open or update issues for release failures
# When a new job is added to this workflow, add it to this list.
needs: [ versioning, build, test-configuration-file, deploy-nodes, deploy-instance ]
needs: [ versioning, build, deploy-nodes, deploy-instance ]
# Only open tickets for failed or cancelled jobs that are not coming from PRs.
# (PR statuses are already reported in the PR jobs list, and checked by Mergify.)
if: (failure() && github.event.pull_request == null) || (cancelled() && github.event.pull_request == null)
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ on:
- '.cargo/config.toml'
- '**/clippy.toml'
# workflow definitions
- '.github/workflows/build-crates-individually.yml'
- '.github/workflows/ci-build-crates.yml'

jobs:
matrix:
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ on:
- '.cargo/config.toml'
- '**/clippy.toml'
# workflow definitions
- '.github/workflows/build-crates-individually.yml'
- '.github/workflows/ci-build-crates.yml'
pull_request:
paths:
# production code and test code
Expand All @@ -34,7 +34,7 @@ on:
- '.cargo/config.toml'
- '**/clippy.toml'
# workflow definitions
- '.github/workflows/build-crates-individually.yml'
- '.github/workflows/ci-build-crates.yml'

env:
CARGO_INCREMENTAL: ${{ vars.CARGO_INCREMENTAL }}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ on:
- '.cargo/config.toml'
- '**/clippy.toml'
- 'codecov.yml'
- '.github/workflows/coverage.yml'
- '.github/workflows/ci-coverage.yml'

jobs:
coverage:
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@ on:
- '**/clippy.toml'
# workflow definitions
- 'codecov.yml'
- '.github/workflows/coverage.yml'
- '.github/workflows/ci-coverage.yml'
pull_request:
paths:
- '**/*.rs'
Expand All @@ -38,7 +38,7 @@ on:
- '.cargo/config.toml'
- '**/clippy.toml'
- 'codecov.yml'
- '.github/workflows/coverage.yml'
- '.github/workflows/ci-coverage.yml'

env:
CARGO_INCREMENTAL: ${{ vars.CARGO_INCREMENTAL }}
Expand Down
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
name: CI Docker
name: Integration Tests on GCP

# These jobs *don't* depend on cached Google Cloud state disks,
# so they can be skipped when the modified files make the actual workflow run.
Expand All @@ -20,10 +20,10 @@ on:
# workflow definitions
- 'docker/**'
- '.dockerignore'
- '.github/workflows/continous-integration-docker.yml'
- '.github/workflows/deploy-gcp-tests.yml'
- '.github/workflows/find-cached-disks.yml'
- '.github/workflows/build-docker-image.yml'
- '.github/workflows/ci-unit-tests-docker.yml'
- '.github/workflows/sub-deploy-integration-tests-gcp.yml'
- '.github/workflows/sub-find-cached-disks.yml'
- '.github/workflows/sub-build-docker-image.yml'

jobs:
# We don't patch the testnet job, because testnet isn't required to merge (it's too unstable)
Expand All @@ -39,48 +39,6 @@ jobs:
steps:
- run: 'echo "No build required"'

test-all:
name: Test all
runs-on: ubuntu-latest
steps:
- run: 'echo "No build required"'

test-all-getblocktemplate-rpcs:
name: Test all with getblocktemplate-rpcs feature
runs-on: ubuntu-latest
steps:
- run: 'echo "No build required"'

test-fake-activation-heights:
name: Test with fake activation heights
runs-on: ubuntu-latest
steps:
- run: 'echo "No build required"'

test-empty-sync:
name: Test checkpoint sync from empty state
runs-on: ubuntu-latest
steps:
- run: 'echo "No build required"'

test-lightwalletd-integration:
name: Test integration with lightwalletd
runs-on: ubuntu-latest
steps:
- run: 'echo "No build required"'

test-configuration-file:
name: Test Zebra default Docker config file
runs-on: ubuntu-latest
steps:
- run: 'echo "No build required"'

test-zebra-conf-path:
name: Test Zebra custom Docker config file
runs-on: ubuntu-latest
steps:
- run: 'echo "No build required"'

test-stateful-sync:
name: Zebra checkpoint update / Run sync-past-checkpoint test
runs-on: ubuntu-latest
Expand Down
Loading
Loading