feat: Cache tests in CI #1818

michaljurecko · 2024-06-04T10:33:18Z

Jira:

Intro:
In last weeks, @Matovidlo intensively dealt with how to speed up our CI pipeline and cache tests (if the code, ENVs or external files have not changed). During the review, I found out that a lot of good work was done, but some things were still missing to make the pipeline even faster. It wasn't much work to complete/debug it, so I prepared this PR.

Changes:

Unit and E2E test are cached.
Cache is stored in our S3 bucket using runs-on/cache@v4 action (an in-place replacement for actions/cache@v4).
- So there is no more 10GB limit.
- Expiration is set to 14day in the S3 bucket and can be modified.

Commented per commit.

Original PRs

First run - no cache at all - empty S3 bucket
https://github.com/keboola/keboola-as-code/actions/runs/9379476266/attempts/2

Second run - cached:
https://github.com/keboola/keboola-as-code/actions/runs/9379476266/attempts/3

scripts/tools.sh

.github/workflows/check-licenses.yml

.github/actions/go-setup/action.yml

.github/actions/export-kbc-projects/action.yml

.github/actions/go-setup/action.yml

.github/workflows/test-unit.yml

internal/pkg/utils/etcdhelper/fortest.go

michaljurecko · 2024-06-05T06:42:45Z

.github/actions/go-setup/action.yml

+    # GOPATH     - Should be empty in our setup, see GOMODCACHE and GOBIN.
+    # GOCACHE    - Build and test cache.
+    # GOMODCACHE - The directory where the go command will store downloaded modules and related files.
+    # GOBIN      - Compiled binaries from "go install ...", we need a directory outside GOPATH to cache only installed tools.
+    - name: Set and export Go envs
+      shell: bash
+      run: |
+        if [ "$RUNNER_OS" == "Windows" ]; then
+          # C: is slow: https://github.com/actions/runner-images/issues/8755
+          GODIRS=D:/tmp/go
+        elif [ "$RUNNER_OS" == "macOS" ]; then
+          GODIRS=/Users/runner/.cache/go
+        else
+          GODIRS=/home/runner/.cache/go
+        fi
+
+        mkdir -p $GODIRS/path
+        mkdir -p $GODIRS/cache
+        mkdir -p $GODIRS/modcache
+        mkdir -p $GODIRS/lintcache
+        mkdir -p $GODIRS/bin
+
+        go env -w \
+        GOPATH=$GODIRS/path \
+        GOCACHE=$GODIRS/cache \
+        GOMODCACHE=$GODIRS/modcache \
+        GOBIN=$GODIRS/bin \
+
+        echo "GODIRS=$GODIRS" >> $GITHUB_ENV
+        echo "GOLANGCI_LINT_CACHE=$GODIRS/lintcache" >> $GITHUB_ENV 
+        echo "$GODIRS/bin" >> $GITHUB_PATH


It is important to know which directories is Go using, and which should be cached.

On Windows they must be present on the faster D: disk.

Without customization, without customization, most files are saved in GOPATH.

So judging by the code below we're caching GOCACHE, GOMODCACHE, GOBIN and GOLANGCI_LINT_CACHE right?

What's left in GOPATH now with all these directories separate? In a comment above you wrote that "Should be empty in our setup" - but that means we're still caching everything that used to be in GOPATH?

What exactly are we trying to exclude from caching with all this setup then? Or what's the point of all this splitting?

What's left in GOPATH now with all these directories separate?

There will be nothing left in GOPATH that we should cache. I didn't check if there is anything there at all, you can find out if you want.

I still don't understand what was the reason for this change if it wasn't to exclude some things that are in GOPATH but shouldn't be cached. 🤔

GOPATH is some older/default/legacy variable, GOCACHE and GOMODCACHE are some "new" ENVs for fine tuning. We can discuss it on the Standup call.

.github/workflows/test-e2e-cli.yml

.github/actions/go-setup/action.yml

Matovidlo

LGTM just small adjustements

.github/actions/go-setup/action.yml

.github/workflows/test-e2e-cli.yml

.github/workflows/check-licenses.yml

.github/actions/go-setup/action.yml

jachym-tousek-keboola · 2024-06-05T10:15:47Z

.github/actions/go-setup/action.yml

+    # GOPATH     - Should be empty in our setup, see GOMODCACHE and GOBIN.
+    # GOCACHE    - Build and test cache.
+    # GOMODCACHE - The directory where the go command will store downloaded modules and related files.
+    # GOBIN      - Compiled binaries from "go install ...", we need a directory outside GOPATH to cache only installed tools.
+    - name: Set and export Go envs
+      shell: bash
+      run: |
+        if [ "$RUNNER_OS" == "Windows" ]; then
+          # C: is slow: https://github.com/actions/runner-images/issues/8755
+          GODIRS=D:/tmp/go
+        elif [ "$RUNNER_OS" == "macOS" ]; then
+          GODIRS=/Users/runner/.cache/go
+        else
+          GODIRS=/home/runner/.cache/go
+        fi
+
+        mkdir -p $GODIRS/path
+        mkdir -p $GODIRS/cache
+        mkdir -p $GODIRS/modcache
+        mkdir -p $GODIRS/lintcache
+        mkdir -p $GODIRS/bin
+
+        go env -w \
+        GOPATH=$GODIRS/path \
+        GOCACHE=$GODIRS/cache \
+        GOMODCACHE=$GODIRS/modcache \
+        GOBIN=$GODIRS/bin \
+
+        echo "GODIRS=$GODIRS" >> $GITHUB_ENV
+        echo "GOLANGCI_LINT_CACHE=$GODIRS/lintcache" >> $GITHUB_ENV 
+        echo "$GODIRS/bin" >> $GITHUB_PATH


So judging by the code below we're caching GOCACHE, GOMODCACHE, GOBIN and GOLANGCI_LINT_CACHE right?

What's left in GOPATH now with all these directories separate? In a comment above you wrote that "Should be empty in our setup" - but that means we're still caching everything that used to be in GOPATH?

What exactly are we trying to exclude from caching with all this setup then? Or what's the point of all this splitting?

.github/actions/export-kbc-projects/action.yml

.github/workflows/test-unit.yml

jachym-tousek-keboola · 2024-06-05T11:25:36Z

internal/pkg/utils/testhelper/runner/runner.go

+			outDir := filesystem.Join(r.testsDir, ".out", testDirName)
+			require.NoError(t, os.RemoveAll(outDir))
+			require.NoError(t, os.MkdirAll(outDir, 0o755))
+			require.NoError(t, aferofs.CopyFs2Fs(nil, workingDir, nil, outDir))


Go doesn't have a built-in copy directory function huh? Apparently os.CopyFS() has been accepted for Go 1.23.
I assume we'll be able to use that instead when we update?

I would spend time on it in different PR where we will replace it everywhere in project. 🙂

Of course you can change it then.

Well, it's not a complete replacement. aferofs is a library for virtual file systems, we use it to lock operations only to the working directory (something like a chroot), and we also use in-memory fs.

But in this place it can be replaced.

.github/workflows/test-lint.yml

.github/actions/go-setup/action.yml

jachym-tousek-keboola · 2024-06-05T11:41:34Z

.github/actions/go-setup/action.yml

      id: go-cache-tools
-      uses: actions/cache@v4
+      uses: runs-on/cache@v4 # replacement for actions/cache@v4
      with:
        path: |


Okay so this is in order to use AWS S3 as cache storage.

But why? Did you run into some issues when using actions/cache? Or do you expect some of the limitations mentioned by RunsOn documentation to matter for us in the near future?

jachym-tousek-keboola · 2024-06-05T11:43:38Z

.github/actions/go-setup/action.yml

@@ -77,13 +80,15 @@ runs:
    # Load compiled tools from cache to skip installation
    - name: Load Go tools cache
      id: go-cache-tools
-      uses: actions/cache@v4
+      uses: runs-on/cache@v4 # replacement for actions/cache@v4


Out of curiosity what runners are we currently using for GitHub Actions? Are we using GutHub's free runners because our repository is open-source? Or is Keboola paying for some runners?

Asking because why checking RunsOn it claims to have lot cheaper runners than GitHub.

We are using free runners that GitHub Actions offers us. They run on Azure. You can have open-source repo and pay for better runners when you need more capacity. Also it is possible to use dedicated runners but I suppose we do not own any infrasctructure that we would run it on (We run in cloud).
As we have free runners and it seems to suite our project well I would skip it, but when the load increases, we will have discussion about it.
Can you please put this runners preview into our confluence under go team or other space? Thanks !

We are now using free runnes, self-hosted Linux runners are easy and used by other teams, but Windows and MacOS is more hard to setup.

This reverts commit 572b926.

michaljurecko · 2024-06-06T07:19:22Z

@keboola/go team, I have taken your comments into account:

@jachym-tousek-keboola had very good comment about GO_VERSION, it is now loaded from go.mod file:
- See fix: Extract GO_VERSION from go.mod
I improved code and comments according your suggestions, see style: Improve code and comments
I Revert "feat: Share modules chache between all OS" because, it works if the cache is generated at first by Linux, but not when it is generated by Windows.

.github/workflows/test-e2e-cli.yml

Matovidlo

LGTM

…test-cache" This reverts commit 2348433, reversing changes made to 212dccb.

michaljurecko changed the base branch from main to mv-PSGO-621-change-test-kbc-project-environment-to-refer-to-path June 4, 2024 10:33

michaljurecko force-pushed the michaljurecko-PSGO-565-test-cache branch from 518ce96 to a976ae7 Compare June 4, 2024 10:37

Base automatically changed from mv-PSGO-621-change-test-kbc-project-environment-to-refer-to-path to main June 4, 2024 10:46

michaljurecko added 2 commits June 4, 2024 13:07

build: Fix GOBIN value in tools.sh

285edf2

build: Move GO_VERSION env to repo settings

d8b794b

michaljurecko force-pushed the michaljurecko-PSGO-565-test-cache branch 9 times, most recently from 7d0dcd1 to acae08b Compare June 4, 2024 15:50

michaljurecko mentioned this pull request Jun 4, 2024

fix: Improve AWS S3 client, don't load AWS ENVs keboola/go-client#135

Merged

michaljurecko force-pushed the michaljurecko-PSGO-565-test-cache branch 11 times, most recently from 4d04479 to 3e90569 Compare June 5, 2024 06:16

michaljurecko added 4 commits June 5, 2024 08:39

build: Create go-setup action

df23a4e

build: Use new go-setup action

b077ace

build: Remove old go-cache action

625df0b

fix: Don't set all tokens as envs

52937db

fix: Improve msg

591d20a

michaljurecko force-pushed the michaljurecko-PSGO-565-test-cache branch 2 times, most recently from 8c64959 to d868378 Compare June 5, 2024 06:44

fix: Linker warnings on macOS

c8d4461

michaljurecko force-pushed the michaljurecko-PSGO-565-test-cache branch from d868378 to c8d4461 Compare June 5, 2024 06:45

fix: Remove unused step

e55fd6b

michaljurecko commented Jun 5, 2024

View reviewed changes

michaljurecko requested review from Matovidlo, jachym-tousek-keboola and hosekpeter June 5, 2024 07:16

michaljurecko marked this pull request as ready for review June 5, 2024 07:17

michaljurecko commented Jun 5, 2024

View reviewed changes

.github/actions/go-setup/action.yml Outdated Show resolved Hide resolved

Matovidlo approved these changes Jun 5, 2024

View reviewed changes

jachym-tousek-keboola requested changes Jun 5, 2024

View reviewed changes

michaljurecko added 5 commits June 6, 2024 07:45

fix: Extract GO_VERSION from go.mod

f612b79

style: Improve code and comments

d2bfd15

Revert "feat: Share modules chache between all OS"

ed8daed

This reverts commit 572b926.

revert: env --ignore-environment to env -i, Win/macOS problem

83911e0

build: Update go-client to 1.26.5

6b527df

michaljurecko requested review from Matovidlo and jachym-tousek-keboola June 6, 2024 07:19

jachym-tousek-keboola approved these changes Jun 6, 2024

View reviewed changes

Matovidlo requested changes Jun 6, 2024

View reviewed changes

.github/workflows/test-e2e-cli.yml Show resolved Hide resolved

michaljurecko requested a review from Matovidlo June 6, 2024 08:12

Matovidlo approved these changes Jun 6, 2024

View reviewed changes

hosekpeter approved these changes Jun 6, 2024

View reviewed changes

michaljurecko merged commit 2348433 into main Jun 6, 2024
13 checks passed

michaljurecko deleted the michaljurecko-PSGO-565-test-cache branch June 6, 2024 12:04

Matovidlo added a commit that referenced this pull request Jun 10, 2024

Revert "Merge pull request #1818 from keboola/michaljurecko-PSGO-565-…

57835af

…test-cache" This reverts commit 2348433, reversing changes made to 212dccb.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: Cache tests in CI #1818

feat: Cache tests in CI #1818

michaljurecko commented Jun 4, 2024 •

edited

Loading

michaljurecko Jun 5, 2024

jachym-tousek-keboola Jun 5, 2024

michaljurecko Jun 6, 2024

jachym-tousek-keboola Jun 6, 2024

michaljurecko Jun 6, 2024

Matovidlo left a comment

jachym-tousek-keboola Jun 5, 2024

jachym-tousek-keboola Jun 5, 2024

Matovidlo Jun 6, 2024

michaljurecko Jun 6, 2024

jachym-tousek-keboola Jun 5, 2024

jachym-tousek-keboola Jun 5, 2024

Matovidlo Jun 6, 2024

michaljurecko Jun 6, 2024

michaljurecko commented Jun 6, 2024

Matovidlo left a comment

feat: Cache tests in CI #1818

feat: Cache tests in CI #1818

Conversation

michaljurecko commented Jun 4, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Matovidlo left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

michaljurecko commented Jun 6, 2024

Matovidlo left a comment

Choose a reason for hiding this comment

michaljurecko commented Jun 4, 2024 •

edited

Loading