1.1 ref sim ci workflow

[burdorfmitchell]

Implemented a new ci workflow which regression tests and benchmarks the less intensive reference simulations using pytest and pytest-benchmark. The historical simulation output is stored on the Brown Digital Repository. Benchmarking data is stored on the gh-pages branch.

Description

Added the necessary files to run many of the first generation reference simulations to src/pyuvsim/data:

src/pyuvsim/data/baseline_lite.csv
src/pyuvsim/data/bl_lite_gauss.yaml
src/pyuvsim/data/bl_lite_uniform.yaml
src/pyuvsim/data/mwa88_nocore_config_MWA.yaml
src/pyuvsim/data/mwa88_nocore_config_gauss.yaml
src/pyuvsim/data/test_catalogs/letter_R_12pt_2458098.38824015.txt
src/pyuvsim/data/test_catalogs/mock_catalog_heratext_2458098.38824015.txt
src/pyuvsim/data/test_catalogs/two_distant_points_2458098.38824015.txt
src/pyuvsim/data/test_config/obsparam_ref_1.1_gauss.yaml
src/pyuvsim/data/test_config/obsparam_ref_1.1_mwa.yaml
src/pyuvsim/data/test_config/obsparam_ref_1.1_uniform.yaml
src/pyuvsim/data/test_config/obsparam_ref_1.2_gauss.yaml
src/pyuvsim/data/test_config/obsparam_ref_1.2_uniform.yaml
src/pyuvsim/data/test_config/obsparam_ref_1.3_gauss.yaml
src/pyuvsim/data/test_config/obsparam_ref_1.3_uniform.yaml

These files are then used to run the reference simulations via the test_run_sim pytest method in the added tests/test_run_ref.py file, which has been configured to parametrize command line input to determine which reference simulation to run. The pytest method additionally benchmarks the reference simulation, and downloads older reference simulation output data from the Brown Digital Repository for regression testing.

The pytest method is called from the command line via a new ci workflow currently named "Run Compare Post Benchmark", which runs each reference simulation on a single core in parallel (depending on the amount of github runners available). The benchmark output for each reference simulation is stored as an artifact. Another job in the workflow is then ran which loads all the stored artifacts from the current workflow, concatenates the benchmarking data to a single file, then runs github-action-benchmark.

The github-action-benchmark action loads previous benchmark data from gh-pages, compares it with the current benchmark output, and provides an alert if a performance regression has occurred. The action then pushes the latest benchmark data to gh-pages ONLY if it is a push to main.

Two new dependencies have been added to the environment.yaml and mentioned in the README -- pytest-benchmark and requests. The docs have not been updated, but the README attempts to clarify that serious benchmarking should be done via the benchmarking directory and that this is a single core regression test.

Currently the ci workflow runs on push and pull request as the workflow_run webhook event did not seem to be triggering appropriately, but after merging with main it is likely this can be swapped to running after a successful completion of the Tests workflow. There seems to be roughly enough available runners to manage either way.

Motivation and Context

The reference simulations have had changes in output over time -- to the extent that some simulations stopped passing an equality check with their own older output. As the output change occurred over many years with minimal oversight, a significant amount of effort was required to confirm the physical consistency of pyuvsim simulations. By incorporating basic regression testing of the reference simulations into the ci workflow, we can identify any time a change in simulation output occurs, even if the cause of the change is external to pyuvsim. We can additionally maintain code performance through benchmarking.

Types of changes

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to change)
• Reference simulation update or replacement
• Documentation change (documentation changes only)
• Version change
• Build or continuous integration change

Checklist:

For all pull requests:

• I have read the contribution guide.
• My code follows the code style of this project.

Documentation change checklist:

• Any updated docstrings use the numpy docstring format.
• If this is a significant change to the readme or other docs, I have checked that they are rendered properly on ReadTheDocs. (you may need help to get this branch to build on RTD, just ask!)

Build or continuous integration change checklist:

• If required or optional dependencies have changed (including version numbers), I have updated the readme to reflect this.
• If this is a new CI setup, I have added the associated badge to the readme and to references/make_index.py (if appropriate).

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 100.00%. Comparing base (f5e5228) to head (ca1e7da).
Report is 1 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff            @@
##              main      #514   +/-   ##
=========================================
  Coverage   100.00%   100.00%           
=========================================
  Files           10        10           
  Lines         2024      2024           
=========================================
  Hits          2024      2024           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[bhazelton]

This is looking great, thank you @burdorfmitchell!