functionaltests: Add generalized test

[ericywl]

Motivation/summary

Closes #16697.

This PR introduces generalized upgrade test. The new generalized upgrade test runs the exact same checks as the previous upgrade tests, and is configurable via YAML (see upgrade-config.yaml). The upgrade paths for the tests are provided via the upgrade-path argument. We use the GitHub workflow matrix to provide the upgrade-path argument to the test and split each run into its own instance.

All other upgrade tests have now been removed.

How to test these changes

Run functionaltests workflow: https://github.com/elastic/apm-server/actions/runs/15188987542

[github-actions]

🤖 GitHub comments

Expand to view the GitHub comments

Just comment with:

• run docs-build : Re-trigger the docs validation. (use unformatted text in the comment!)

[mergify]

This pull request does not have a backport label. Could you fix it @ericywl? 🙏
To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

• backport-7.17 is the label to automatically backport to the 7.17 branch.
• backport-8./d is the label to automatically backport to the 8./d branch. /d is the digit.
• backport-9./d is the label to automatically backport to the 9./d branch. /d is the digit.
• backport-active-all is the label that automatically backports to all active branches.
• backport-active-8 is the label that automatically backports to all active minor branches for the 8 major.
• backport-active-9 is the label that automatically backports to all active minor branches for the 9 major.

[mergify]

This pull request is now in conflicts. Could you fix it @ericywl? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b functionaltests-generalized-tests upstream/functionaltests-generalized-tests
git merge upstream/main
git push upstream functionaltests-generalized-tests

[simitt]

Are the 8_17_test.go, 8_18_test.go, 9_0_test.go, 9_1_test.go files eventually going to be replaced by the more generic upgrade_test.go ?

[simitt]

Would it not be enough to test for number of docs in datastreams > 0 or higher than at the start of a version test?

[simitt]

Why are aggregated metrics data streams not expected to be 0 if no docs were indexed?

[ericywl]

Are the 8_17_test.go, 8_18_test.go, 9_0_test.go, 9_1_test.go files eventually going to be replaced by the more generic upgrade_test.go ?

Yes