TST: Add CMake and gtest frameworks

[inkydragon]

This is a POC that currently sets up a basic CMake + gtest test environment.

• CMake
• gtest
• github CI, Generate and upload coverage report

Build and generating coverage report:

# sudo apt-get install -y lcov

cmake -DCMAKE_BUILD_TYPE=Coverage -S . -B build
cd build && make && ctest
make coverage_html

# build\coverage_report

Sample:

• Codecov report
  • Need to set up codecov CI and patch: inkydragon/xsf@73c842f (#1)
  • Only a small number of files are shown in the report
    because the test code only calls the airy-related functions
• HTML coverage report

[steppi]

Thanks @inkydragon. This aligns well with what I had planned. My thoughts for testing is that we should have scripts that generate csv files of test cases. The C++ tests would then read these files, and we would have different tiers of tests. Some running on every push to a PR, and others (like GPU jobs or more extensive tests) running on a cron.

For the short term I plan to stash the csv files in GitHub, but eventually they would likely go somewhere better suited for file storage. I'll start generating some files this week, and if you want to continue this, you can either write something to pull them in for the tests, or I can follow through on what you're setting up to do that.

[lucascolley]

Locally I see

CMake Warning (dev) at .pixi/envs/dev/share/cmake-3.30/Modules/FetchContent.cmake:1373 (message):
  The DOWNLOAD_EXTRACT_TIMESTAMP option was not given and policy CMP0135 is
  not set.  The policy's OLD behavior will be used.  When using a URL
  download, the timestamps of extracted files should preferably be that of
  the time of extraction, otherwise code that depends on the extracted
  contents might not be rebuilt if the URL changes.  The OLD behavior
  preserves the timestamps from the archive instead, but this is usually not
  what you want.  Update your project to the NEW behavior or specify the
  DOWNLOAD_EXTRACT_TIMESTAMP option with a value of true to avoid this
  robustness issue.
Call Stack (most recent call first):
  tests/CMakeLists.txt:4 (FetchContent_Declare)
This warning is for project developers.  Use -Wno-dev to suppress it.

[inkydragon]

Switch to use the commit id to eliminate this warning

[inkydragon]

I can follow through on what you're setting up to do that.

Feel free to open a new pr based on this.
Or you can open a new pr and give me an example: generate CSV, read and test a function
Then I can follow the steps in that example and get on with the boring test generation.

I guess there are a lot of pr in scipy waiting for your review.

My previous plan was to manually write some test cases to achieve higher line of code coverage with a small number of tests.

[ev-br]

I'm sure y'all thought about this already, but I'll ask nonetheless: have you considered adding a thin python layer for testing and using python testing frameworks?

While I don't have a horse in this race, the setup otherwise seems suspiciously of the "generate text data and use gnuplot" type of construction which got ditched once somebody showed me matpltlotlib back in the day.

[steppi]

I'm sure y'all thought about this already, but I'll ask nonetheless: have you considered adding a thin python layer for testing and using python testing frameworks?

While I don't have a horse in this race, the setup otherwise seems suspiciously of the "generate text data and use gnuplot" type of construction which got ditched once somebody showed me matpltlotlib back in the day.

Yeah, using Python is the other option I've considered. I suggested that in #1 because I thought it would be the easiest way to get started. In any case, I'd want us to generate test cases and store them somewhere because generating arbitrary precision reference data takes several orders of magnitude longer than running the C++ code. In terms of how to serialize the test cases, I like text formats, because then it's clear they are what they say they are. Binary files could be compromised potentially.

If the test cases are coming from files, and the C++ tests just read files, call the function on each test case, and compare to the reference value, I think that's pretty straightforward. Having a C++ testing framework also makes it easier to test internal components which aren't so easy to wrap in Python like ufuncs are.

I'm really happy to see @inkydragon's interest in helping us get set up with CMake and gtest.