Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Wrap nested_dask reduce #414

Merged
merged 12 commits into from
Oct 10, 2024
Merged

Wrap nested_dask reduce #414

merged 12 commits into from
Oct 10, 2024

Conversation

smcguire-cmu
Copy link
Contributor

@smcguire-cmu smcguire-cmu commented Sep 13, 2024

Adds the reduce function to the Catalog class, wrapping the reduce function in nd.NestedFrame

Fixes #420 and fixes #372

Copy link
Contributor

@dougbrn dougbrn left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks good, I have one question that I would not consider a blocker and more just curiosity.

src/lsdb/catalog/dataset/healpix_dataset.py Outdated Show resolved Hide resolved
@dougbrn dougbrn mentioned this pull request Sep 23, 2024
3 tasks
@smcguire-cmu smcguire-cmu changed the base branch from main to development October 8, 2024 14:31
Copy link

codecov bot commented Oct 8, 2024

Codecov Report

Attention: Patch coverage is 96.77419% with 1 line in your changes missing coverage. Please review.

Project coverage is 97.92%. Comparing base (76e2af4) to head (b613df7).
Report is 13 commits behind head on development.

Files with missing lines Patch % Lines
src/lsdb/catalog/catalog.py 80.00% 1 Missing ⚠️
Additional details and impacted files
@@               Coverage Diff               @@
##           development     #414      +/-   ##
===============================================
- Coverage        97.94%   97.92%   -0.03%     
===============================================
  Files               44       44              
  Lines             1560     1591      +31     
===============================================
+ Hits              1528     1558      +30     
- Misses              32       33       +1     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Copy link

github-actions bot commented Oct 8, 2024

Before [c478401] After [3c3231c] Ratio Benchmark (Parameter)
failed failed n/a benchmarks.time_box_filter_on_partition
1.73±0.02s 1.76±0.02s 1.02 benchmarks.time_create_midsize_catalog
11.8±0.07s 11.8±0.01s 1.00 benchmarks.time_create_large_catalog
91.5±0.8ms 91.8±1ms 1.00 benchmarks.time_kdtree_crossmatch
26.8±0.2ms 26.3±0.5ms 0.98 benchmarks.time_polygon_search

Click here to view all benchmarks.

@smcguire-cmu smcguire-cmu linked an issue Oct 8, 2024 that may be closed by this pull request
3 tasks
Copy link
Contributor

@hombit hombit left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks cool, thanks! I'd ask for a bit more testing if it wouldn't take a lot of time

tests/lsdb/catalog/test_polygon_search.py Outdated Show resolved Hide resolved
tests/lsdb/catalog/test_nested.py Outdated Show resolved Hide resolved
@smcguire-cmu smcguire-cmu merged commit be78673 into development Oct 10, 2024
7 of 8 checks passed
@smcguire-cmu smcguire-cmu deleted the sean/reduce branch October 10, 2024 14:53
@smcguire-cmu smcguire-cmu linked an issue Oct 15, 2024 that may be closed by this pull request
5 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Make reduce output a catalog LSDB with Nested-Dask: Additional Wrappers
3 participants