Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve Github Actions. #5687

Merged
merged 1 commit into from
Apr 4, 2024
Merged

Improve Github Actions. #5687

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
107 changes: 46 additions & 61 deletions .github/workflows/haskell.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -87,67 +87,51 @@ jobs:
echo "# cabal.project.local"
cat cabal.project.local

# A dry run `build all` operation does *NOT* downlaod anything, it just looks at the package
# indices to generate an install plan.
- name: Build dry run
run: |
cabal build all --dry-run --minimize-conflict-set
run: cabal build all --enable-tests --dry-run --minimize-conflict-set

# For users who fork cardano-node and want to define a writable cache, then can set up their own
# S3 bucket then define in their forked repository settings the following secrets:
#
# AWS_ACCESS_KEY_ID
# AWS_SECRET_ACCESS_KEY
# BINARY_CACHE_URI
# BINARY_CACHE_REGION
- name: Cabal cache over S3
uses: action-works/cabal-cache-s3@v1
env:
AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
# From the install plan we generate a dependency list.
- name: Record dependencies
id: record-deps
run: |
# The tests call out to msys2 commands. We generally do not want to mix toolchains, so
# we are very deliberate about only adding msys64 to the path where absolutely necessary.
${{ (runner.os == 'Windows' && '$env:PATH=("C:\msys64\mingw64\bin;{0}" -f $env:PATH)') || '' }}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this line can be avoided similarly to https://github.com/IntersectMBO/cardano-api/pull/504/files#diff-c09cc8dc973415b1221308559b96942a1c4b872e690bf360fc22d9b42822d519L76

cat dist-newstyle/cache/plan.json | jq -r '."install-plan" | select(.style != "local") | .id' | sort | uniq > dependencies.txt

# From the dependency list we restore the cached dependencies.
# We use the hash of `dependencies.txt` as part of the cache key because that will be stable
# until the `index-state` values in the `cabal.project` file changes.
- name: Restore cached dependencies
uses: actions/cache/restore@v4
id: cache
with:
region: ${{ vars.BINARY_CACHE_REGION }}
dist-dir: dist-newstyle
store-path: ${{ steps.setup-haskell.outputs.cabal-store }}
threads: ${{ vars.BINARY_CACHE_THREADS }}
archive-uri: ${{ vars.BINARY_CACHE_URI }}/${{ env.CABAL_CACHE_VERSION }}/${{ runner.os }}/${{ matrix.cabal }}/${{ matrix.ghc }}
skip: "${{ vars.BINARY_CACHE_URI == '' }}"

# It's important to ensure that people who fork this repository can not only successfully build in
# CI by default, but also have meaning cabal store caching.
#
# Because syncing with S3 requires credentials, we cannot rely on S3 for this. For this reason a
# https fallback is used. The https server mirrors the content of the S3 bucket. The https cabal
# store archive is read-only for security reasons.
#
# Users who fork this repository who want to have a writable cabal store archive are encouraged
# to set up their own S3 bucket.
- name: Cabal cache over HTTPS
uses: action-works/cabal-cache-s3@v1
path: |
${{ steps.setup-haskell.outputs.cabal-store }}
dist-newstyle
key: cache-${{ env.CABAL_CACHE_VERSION }}-${{ runner.os }}-${{ matrix.ghc }}-${{ hashFiles('dependencies.txt') }}

# Now we install the dependencies. If the cache was found and restored in the previous step,
# this should be a no-op, but if the cache key was not found we need to build stuff so we can
# cache it for the next step.
- name: Install dependencies
run: cabal build all --enable-tests --only-dependencies -j --ghc-option=-j4

# Always store the cabal cache.
# This can fail (benign failure) if there is already a hash at that key.
- name: Cache Cabal store
uses: actions/cache/save@v4
with:
dist-dir: dist-newstyle
store-path: ${{ steps.setup-haskell.outputs.cabal-store }}
threads: ${{ vars.BINARY_CACHE_THREADS }}
archive-uri: https://iohk.cache.haskellworks.io/${{ env.CABAL_CACHE_VERSION }}/${{ runner.os }}/${{ matrix.cabal }}/${{ matrix.ghc }}
skip: "${{ vars.BINARY_CACHE_URI != '' }}"
enable-save: false

# The following seems like a duplicate but it is due to a bug in cabal which can sometimes
# cause build intermittent build failures on Windows, so we run the build twice.
#
# https://github.com/haskell/cabal/issues/9587
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why are we removing this step? The bug doesn't seem fixed? The original discussion https://github.com/IntersectMBO/cardano-node/pull/5625/files#r1442647736

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How easy was that issue to reproduce, because I have not seen it for ages.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd say one in 20 builds? It was happening sometimes.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would have done at least 20 build of this an not yet seen it.

Do you want it back in?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's try without it and see if it happens again. Maybe it was a side effect from cabal-cache.

#
# TODO Remove the workaround when the cabal bug is fixed.
- name: Build core components
continue-on-error: true
run: |
cabal build cardano-node cardano-cli cardano-node-chairman cardano-submit-api -j1
path: |
${{ steps.setup-haskell.outputs.cabal-store }}
dist-newstyle
key: cache-${{ env.CABAL_CACHE_VERSION }}-${{ runner.os }}-${{ matrix.ghc }}-${{ hashFiles('dependencies.txt') }}

- name: Build core components
run: |
cabal build cardano-node cardano-cli cardano-node-chairman cardano-submit-api -j1

- name: Build remaining components
run: |
cabal build all
# Now we build.
- name: Build all
run: cabal build all --enable-tests
erikd marked this conversation as resolved.
Show resolved Hide resolved

- name: Run tests
env:
Expand All @@ -168,7 +152,7 @@ jobs:

- name: Upload workspaces on tests failure
if: ${{ failure() }}
uses: actions/upload-artifact@v3
uses: actions/upload-artifact@v4
with:
name: failed-test-workspaces-${{ matrix.os }}-ghc${{ matrix.ghc }}-cabal${{ matrix.cabal }}.tgz
path: ${{ runner.temp }}/workspaces.tgz
Expand Down Expand Up @@ -198,7 +182,8 @@ jobs:
find "${{ runner.temp }}/chairman" -type s -exec rm -f {} \;
fi

- uses: actions/upload-artifact@v2
- name: Save Artifact
uses: actions/upload-artifact@v4
if: ${{ always() }}
continue-on-error: true
with:
Expand All @@ -217,9 +202,9 @@ jobs:
# and will silently fail if msys2 is not in path. See the "Run tests" step.
#
# - name: Setup tmate session
# if: ${{ failure() }}
# uses: mxschmitt/action-tmate@v3
# with:
# if: ${{ failure() }}
# uses: mxschmitt/action-tmate@v3
# with:
# limit-access-to-actor: true

build-complete:
Expand Down
Loading