Add verification tests

[troyraen]

Related to #373, #374

Change Description

• My PR includes a link to the issue that I am addressing

Solution Description

Code Quality

• I have read the Contribution Guide and LINCC Frameworks Code of Conduct
• My code follows the code style of this project
• My code builds (or compiles) cleanly without any errors or warnings
• My code contains relevant comments and necessary documentation

Project-Specific Pull Request Checklists

Bug Fix Checklist

• My fix includes a new test that breaks as a result of the bug (if possible)
• My change includes a breaking change
  • My change includes backwards compatibility and deprecation warnings (if possible)

New Feature Checklist

• I have added or updated the docstrings associated with my feature using the NumPy docstring format
• I have updated the tutorial to highlight my new feature (if appropriate)
• I have added unit/End-to-End (E2E) test cases to cover my new feature
• My change includes a breaking change
  • My change includes backwards compatibility and deprecation warnings (if possible)

Documentation Change Checklist

• Any updated docstrings use the NumPy docstring format

Build/CI Change Checklist

• If required or optional dependencies have changed (including version numbers), I have updated the README to reflect this
• If this is a new CI setup, I have added the associated badge to the README

Other Change Checklist

• Any new or updated docstrings use the NumPy docstring format.
• I have updated the tutorial to highlight my new feature (if appropriate)
• I have added unit/End-to-End (E2E) test cases to cover any changes
• My change includes a breaking change
  • My change includes backwards compatibility and deprecation warnings (if possible)

[codecov]

Codecov Report

Attention: Patch coverage is 29.62963% with 19 lines in your changes missing coverage. Please review.

Project coverage is 98.47%. Comparing base (e7f9b4c) to head (029196d).

Files	Patch %	Lines
...rc/hipscat_import/verification/run_verification.py	29.62%	19 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #379      +/-   ##
==========================================
- Coverage   99.72%   98.47%   -1.26%     
==========================================
  Files          26       26              
  Lines        1481     1508      +27     
==========================================
+ Hits         1477     1485       +8     
- Misses          4       23      +19     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[troyraen]

@delucchi-cmu here are some options. Take a look and let me know what you think. It's obviously not fully integrated yet. I did add some data that will make the tests fail; not sure if you want those extra files or if it should somehow use data that's already here. Also, it probably needs to use your custom file pointers and the user-supplied storage kwargs that you support, but I haven't added them yet. Also haven't done docstrings yet (these probably aren't the functions you actually want anyway).

There are four tests. The one checking the file schemas against _common_metadata is the thing you actually asked for. It feels a little incomplete if the _common_metadata isn't checked against a schema from the user, but if you don't want to rely on user input here I can understand that. The other tests are code that I had handy that's inline with checking individual parquet files. One is row counts, which I know you're taking care of in a different repo but I think you said that didn't check the individual files? In any case, I'm happy to take all those extra three tests out if you just want to focus on the schema right now.

I'm not sure how you want this integrated with the main run function. Right now, the functions I wrote return a boolean indicating pass/fail. What do you want it to do if it fails? I'm guessing it should not raise an error. Should it just print messages to std out, or create a report file, or ..?

[github-actions]

Before [829fe47]	After [2405014]	Ratio	Benchmark (Parameter)
failed	failed	n/a	benchmarks.BinningSuite.time_read_histogram

Click here to view all benchmarks.

[troyraen]

I made choices about some questions from above and filled out this code a little more. So right now there is a Verifier class that handles the tests and it's called in the run function with:

verifier = Verifier.from_args(args)
verifier.test_is_valid_catalog()  # run hipscat.io.validation.is_valid_catalog
verifier.test_schemas()  # user-provided schema vs _common_metadata, _metadata, and file footers
verifier.test_num_rows()  # file footers vs _metadata (per file), and user-provided total
verifier.record_results()  # write a verification report

verifier.record_distributions()  # calculate distributions (min/max of all fields) and write a file