Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix test data for removing partition_info #65

Merged
merged 3 commits into from
Nov 14, 2023

Conversation

delucchi-cmu
Copy link
Contributor

Change Description

This is related to work toward deprecating partition_info.csv (astronomy-commons/hipscat#147), and is very similar to PR astronomy-commons/hipscat-import#166

Solution Description

The catalog files for the cross matching test data is quite old, and didn't have the Norder/Dir/Npix columns inside the parquet file. When we try to read the directory as a hipscat catalog, we try to create the PartitionInfo from the row group statistics for those fields, but fail to do so.

This PR re-generates the catalog files, using the current map/reduce methods of the import tool to include the partition info columns as necessary.

While I'm here, I took out the tests for handling catalog directories with no metadata files (since we're going to require them very soon).

Copy link

codecov bot commented Nov 14, 2023

Codecov Report

All modified and coverable lines are covered by tests ✅

Comparison is base (e9b2562) 99.13% compared to head (1ba7fb2) 99.13%.

Additional details and impacted files
@@            Coverage Diff             @@
##             main      #65      +/-   ##
==========================================
- Coverage   99.13%   99.13%   -0.01%     
==========================================
  Files          21       21              
  Lines         464      461       -3     
==========================================
- Hits          460      457       -3     
  Misses          4        4              

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Copy link

github-actions bot commented Nov 14, 2023

Before [e9b2562] After [937c658] Ratio Benchmark (Parameter)
36.3±0.5ms 35.9±0.3ms 0.99 benchmarks.time_kdtree_crossmatch

Click here to view all benchmarks.

@camposandro
Copy link
Collaborator

There's the linting issues, but apart from that looks good! :)

@delucchi-cmu delucchi-cmu merged commit d57f770 into main Nov 14, 2023
15 checks passed
@delucchi-cmu delucchi-cmu deleted the delucchi/partition_info branch November 14, 2023 15:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants