ci(framework:skip) Add caching to speed up E2E CI tests

[chongshenng]

Issue

Description

This PR improves the speed of the CI for E2E tests.

Related issues/PRs

This PR is inspired by #1975 and this blogpost by Evan Pete Walsh. The CI should be triggered couple of times after that has happened to compare the impact of the caching.

Explanation

The CI cache works as follows:

main branch

Whenever a new commit or PR is made to the main branch the Cache Python location job in .github/workflows/e2e.yml will run and cache the following directory:

• ${{ env.pythonLocation }} (Note that caching only ~/.cache/pip will cache the wheels, but not the installation itself)

The cache will be overwritten if there is a cache miss. It is retrieved by jobs in pull requests by the following key:

pip-${{ runner.os }}-${{ matrix.directory }}-${{ env.pythonLocation }}-${{ hashFiles('**/pyproject.toml') }}

We ensure that the job retrieving the cache must meet the following requirements:

1. Same OS
2. Same matrix directory (important because matrix is used in the E2E test)
3. Same Python location (which looks something like /opt/hostedtoolcache/Python/3.8.18/x64)
4. No changes to pyproject.toml in the project folders.

One significant update in this PR is that all pyproject.tomls in the E2E folder have been migrated to pure pip installation with hatchling backend without relying on poetry and setuptools.

In this PR, we only rely on actions@cache for caching the Python location instead of actions@setup-python because the latter does not yet work with matrix configurations due to cache key conflicts. This is explained by the setup-python maintainers in this thread.

Pull Requests

Pull requests will try to use the cache by doing a lookup using the previously described key and pull the cache is available.

Preliminary results

E2E timings can reduce down to ~4 min 50+ secs by caching the Python location, compared to ~5 min 20+ secs if we cache only ~/.cache/pip, and ~5 min 40+ secs without caching (see this workflow run).

Checklist

• Implement proposed change
• [ ] Write tests
• Update documentation
• Update the changelog entry below
• Make CI checks pass
• Ping maintainers on Slack (channel #contributions)

Changelog entry

Any other comments?

N/A

[Robert-Steiner]

We could add -U flag to always upgrade packages to the newest available version.
Example:
If we have the requirement pytest>=7.3,<8 and the cache already contains the version 7.3, pip install does not install the latest version 7.4.1 because pip prefers the installed version unless --upgrade or -U is specified.

[chongshenng]

Good point, let me investigate.

[tanertopal]

LGTM