Enable "index file" reads for catalog import

[delucchi-cmu]

Change Description

Closes #308 .

Solution Description

Creates a new kind of file reader for catalog import: indexed file reader. This uses a single "index" file as a task unit, and these files contain only paths to data files to be read. This enables batching many small input data files into larger chunks for the map and reduce stages of the pipeline.

Implements an indexed reader for CSV and for Parquet files. In particular, the parquet reader utilizes pyarrow's parquet read batch_readahead, fragment_readahead, and multi-threading to further speed up reads of many small data files.

Code Quality

• I have read the Contribution Guide and LINCC Frameworks Code of Conduct
• My code follows the code style of this project
• My code builds (or compiles) cleanly without any errors or warnings
• My code contains relevant comments and necessary documentation

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 99.79%. Comparing base (04c1a8f) to head (c45a6dc).
Report is 40 commits behind head on main.

Additional details and impacted files

@@           Coverage Diff           @@
##             main     #334   +/-   ##
=======================================
  Coverage   99.78%   99.79%           
=======================================
  Files          26       26           
  Lines        1389     1442   +53     
=======================================
+ Hits         1386     1439   +53     
  Misses          3        3           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[jeremykubica]

Two high level questions.