Changed forcing system-tests from env var in .yml to .json file

[vpellan]

What does this PR do?

This PR change the forcing of system-tests from environment variable in system-tests.yml to a .json file. This enables the forcing of system-tests on multiple scenarios at the same time. This will also be easier to clean in the post release steps of fast_castle.

Motivation:

fast_castle post release cleanup step currently contain bugs, and the current force-executing of tests feature is pretty limited.

Change log entry

None

Additional Notes:

Github Actions is very limited and I believe poorly documented, I'd really enjoy any advice on what/how it could be done better

How to test the change?

Add a test you want to force-execute in the .github/forced-tests-list.json file by following the documentation template

[pr-commenter]

Benchmarks

Benchmark execution time: 2024-11-13 14:48:29

Comparing candidate commit 42f9e92 in PR branch vpellan/system-tests-force-multiple-scenario with baseline commit 48e5f09 in branch master.

Found 1 performance improvements and 0 performance regressions! Performance is the same for 28 metrics, 2 unstable metrics.

scenario:tracing - Tracing.log_correlation

• 🟩 throughput [+9544.317op/s; +9908.526op/s] or [+8.616%; +8.945%]

[Strech]

I think the whole approach is correct 👍🏼 I would suggest to improve the naming for rest without good context of what's going on.

[p-datadog]

Is this feature meant for releases only or also for development? I.e. if I am developing a system test that also requires changes in tracer, could I force that system test in the tracer branch?

Another comment I have is regarding the empty json file. My impression is that an empty file is not valid json payload, is that correct? Meaning the empty file wouldn't parse? If so I would suggest instead of the empty file, have forced-tests-list.json.sample with sample contents (e.g. the same sample given in the documentation) which would 1) not have an unusable file in the tree and 2) would help someone craft the correct file.

[vpellan]

Is this feature meant for releases only or also for development? I.e. if I am developing a system test that also requires changes in tracer, could I force that system test in the tracer branch?

This feature will make you able to force-execute a system-test on dd-trace-rb's Github CI without the need to create a PR on system-tests side. (which will just declare a version in the manifest, will not be visible on dd-trace-rb's CI results while the PR is not merged on system-tests main branch, and will break system-tests CI while the feature that is being worked on is not merged on dd-trace-rb's master branch) But this will only force them on dd-trace-rb CI, meaning that they will not be reported on the feature parity dashboard (or reported as easy-wins).
There will be a task on fast_castle that will create a PR to clean this file after a release but if required, this PR can later be modified to keep some lines of this file (as this is non-blocking for further steps)

Another comment I have is regarding the empty json file. My impression is that an empty file is not valid json payload, is that correct? Meaning the empty file wouldn't parse? If so I would suggest instead of the empty file, have forced-tests-list.json.sample with sample contents (e.g. the same sample given in the documentation) which would 1) not have an unusable file in the tree and 2) would help someone craft the correct file.

Yes an empty JSON file is not a valid JSON payload, which is why there is a condition for the file to not be empty before parsing it. But it should also work the same if we just put {} in the JSON file, but at least it would make the JSON payload valid. I like the idea of having a sample in the same folder of the actual JSON so it is easier to find and copy.

[codecov-commenter]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 97.77%. Comparing base (48e5f09) to head (42f9e92).
Report is 2 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #4092      +/-   ##
==========================================
- Coverage   97.77%   97.77%   -0.01%     
==========================================
  Files        1350     1350              
  Lines       81062    81062              
  Branches     4085     4085              
==========================================
- Hits        79259    79258       -1     
- Misses       1803     1804       +1     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.