Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix test data for removing partition_info #166

Merged
merged 1 commit into from
Nov 14, 2023

Conversation

delucchi-cmu
Copy link
Contributor

Change Description

This is related to work toward deprecating partition_info.csv (astronomy-commons/hipscat#147).

Solution Description

The intermediate file in this test data is quite old, and didn't have the Norder/Dir/Npix columns inside the parquet file. The pipeline could successfully resume from the previous work, but generated _metadata file that didn't have row group statistics for those three columns. Then, when we try to read the directory as a hipscat catalog, we try to create the PartitionInfo from the row group statistics, but fail to do so.

This PR re-generates the intermediate file, using the current map/reduce methods of the import tool to include the partition info columns as necessary.

And this is kind of impossible to review, since the only change is to a binary file, so that's why there's such a long explanation =]

Copy link

codecov bot commented Nov 14, 2023

Codecov Report

All modified and coverable lines are covered by tests ✅

Comparison is base (151d2ed) 100.00% compared to head (314b96b) 100.00%.

Additional details and impacted files
@@            Coverage Diff            @@
##              main      #166   +/-   ##
=========================================
  Coverage   100.00%   100.00%           
=========================================
  Files           27        27           
  Lines         1158      1158           
=========================================
  Hits          1158      1158           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@delucchi-cmu delucchi-cmu merged commit 9ed8952 into main Nov 14, 2023
13 checks passed
@delucchi-cmu delucchi-cmu deleted the delucchi/partition_info branch November 14, 2023 15:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants