Check the total number of rows throughout the catalog import pipeline

[delucchi-cmu]

Change Description

Closes #344

Solution Description

Adds a new argument, expected_total_rows, that users can specify if the total number of rows to be imported is known ahead of time. After the mapping stage, we check this against the total number of rows encountered.

Using the result from the mapping stage, we also check the total for the scheduled reducing stage as well.

Code Quality

• I have read the Contribution Guide and LINCC Frameworks Code of Conduct
• My code follows the code style of this project
• My code builds (or compiles) cleanly without any errors or warnings
• My code contains relevant comments and necessary documentation

[codecov]

Codecov Report

Attention: Patch coverage is 91.66667% with 1 line in your changes missing coverage. Please review.

Project coverage is 99.72%. Comparing base (6c4451f) to head (079f804).
Report is 46 commits behind head on main.

Files with missing lines	Patch %	Lines
src/hipscat_import/catalog/run_import.py	85.71%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #345      +/-   ##
==========================================
- Coverage   99.79%   99.72%   -0.07%     
==========================================
  Files          26       26              
  Lines        1442     1452      +10     
==========================================
+ Hits         1439     1448       +9     
- Misses          3        4       +1     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[hombit]

Looks good! I have a minor question

[troyraen]

LGTM. I think it will also be important to check this value against the number of rows in the parquet files directly at some point. In cases where the import process goes a little funky the parquet files can diverge from what was written in log files, and those are cases where this test can be especially valuable.

[delucchi-cmu]

LGTM. I think it will also be important to check this value against the number of rows in the parquet files directly at some point. In cases where the import process goes a little funky the parquet files can diverge from what was written in log files, and those are cases where this test can be especially valuable.

The method that writes out the parquet metadata will look at all of the footers for the parquet data files. astronomy-commons/hats#306 will have that method return the total count. This would also be helpful for a few other import pipelines, where we read the footers AGAIN to calculate the total number of rows written. Once that PR is in, I'll update the code here to add the check.