Fix margin cache generation leaving empty shard dirs

[smcguire-cmu]

Change Description

Fixes a bug where generating a margin cache left the empty top level shard directories.

Solution Description

In a previous PR, the margin cache shards were changed to be generated from the final pixel destination Norder=x/Dir=y/Npix=z/... to the same shard directories as other pipelines order_x/dir_y/pix_z/.... Before, these directories did not need to be deleted since they were used as the final structure, but now the new shard directory structures do need to be deleted.

To solve this, the shard files are now stored under a subdirectory cache/order_x/... which is deleted at the end of the pipeline.

Code Quality

• I have read the Contribution Guide
• My code follows the code style of this project
• My code builds (or compiles) cleanly without any errors or warnings
• My code contains relevant comments and necessary documentation

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Comparison is base (862ea03) 99.74% compared to head (b082078) 99.74%.

❗ Current head b082078 differs from pull request most recent head a641c79. Consider uploading reports for the commit a641c79 to get more accurate results

Additional details and impacted files

@@           Coverage Diff           @@
##             main     #205   +/-   ##
=======================================
  Coverage   99.74%   99.74%           
=======================================
  Files          24       24           
  Lines        1166     1172    +6     
=======================================
+ Hits         1163     1169    +6     
  Misses          3        3           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[delucchi-cmu]

Hm. It looks like the margin cache process doesn't make use of the args.tmp_path at all, and would have been cleaned up by the above step, if it were. I created a branch that uses the tmp_path instead (https://github.com/astronomy-commons/hipscat-import/tree/delucchi/margin_tmp). Can you see if that addresses the empty shard dir problem? I'd like users to be able to use a different path for tmp files (see this doc: https://hipscat-import.readthedocs.io/en/latest/catalogs/temp_files.html)

[smcguire-cmu]

Ah I see! That makes sense, and yeah I tested with that branch and it solves the issue. I can close this PR and we can go with that solution if you think that's better.

[delucchi-cmu]

Sounds good. Sending you the PR.

[smcguire-cmu]

#207 fixes the same issue as this and uses the tmp_path for intermediate files, so no need for this.