Add dependency caching step to tests in CI

[tanertopal]

Issue

Description

This PR improves the speed of the CI.

Related issues/PRs

This PR assumes #2103 has already been merged. The CI should be triggered couple of times after that has happened to compare the impact of the caching.

Explanation

The CI cache works as follows:

main branch

Whenever a new commit is made to the main branch the build_python_cache job in .github/workflows/ci-cache.yml will run and cache the following directories and files:

• **/poetry.lock
• ~/.cache/pip
• ~/.cache/pypoetry

The cache will be overwritten each time and can be retrieved by jobs in pull requests by the following key:

${{ runner.os }}-python-cache-${{ matrix.python-version }}-${{ hashFiles('**/pyproject.toml') }}

This basically means the job retrieving the cache has to meet the following requirements.

1. Same OS
2. Same Python version
3. No pyproject.toml file in the repository has changed

The consequence is that updates to any pyproject.toml file should usually be made in isolation to save time. As an additional note: the main branch jobs will never use (read) a cache.

Pull Requests

Pull requests will try to use the cache by doing a lookup using the previously described key and pull the cache is available.

---

Checklist

• Implement proposed change
• Write tests
• Update documentation
• Update changelog
• Make CI checks pass
• Ping maintainers on Slack (channel #contributions)

[danieljanes]

@tanertopal is there a way to set a retention period for caching the poetry.lock file? I wouldn't want to rely just on the 10GB limit. Also, we probably don't need to delete all caches, deleting just the lock file should be sufficient.

[tanertopal]

@tanertopal is there a way to set a retention period for caching the poetry.lock file? I wouldn't want to rely just on the 10GB limit. Also, we probably don't need to delete all caches; deleting just the lock file should be sufficient.

@danieljanes Unfortunatly not. You either built a cache based on a key and number of file paths to cache. If we want to build multiple caches that's possible, although I think it would become messy fast and will create a lot of duplicate code and make the CI definitions less maintainable.

Deleting and rebuilding the caches immediately once a day should be good. It's predictable and cheap. I can add a job for it or do you see any problem with it?

[tanertopal]

Changing this because its the GitHub hosted cached version and saves some time.

[tanertopal]

Closing in favor of #3480