User arguments to retain intermediate files

[delucchi-cmu]

Closes #297

Solution Description

Exposes an existing internal testing parameter to keep the intermediate parquet shard files.

In addition, creates new option to keep the intermediate resume logging files.

New Feature Checklist

• I have added or updated the docstrings associated with my feature using the NumPy docstring format
• I have updated the tutorial to highlight my new feature (if appropriate)
• I have added unit/End-to-End (E2E) test cases to cover my new feature
• My change includes a breaking change
  • My change includes backwards compatibility and deprecation warnings (if possible)

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 99.53%. Comparing base (3f7a423) to head (7f6bb55).
Report is 52 commits behind head on main.

Additional details and impacted files

@@           Coverage Diff           @@
##             main     #311   +/-   ##
=======================================
  Coverage   99.52%   99.53%           
=======================================
  Files          25       25           
  Lines        1274     1282    +8     
=======================================
+ Hits         1268     1276    +8     
  Misses          6        6           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.