Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Change unWISE bucket to nasa-irsa-wise #330

Merged
merged 2 commits into from
Sep 17, 2024
Merged

Conversation

troyraen
Copy link
Contributor

The unWISE dataset accessed by the light curve notebooks is now in a NASA ODR bucket. This PR updates the code to access that bucket.

Copy link
Member

@bsipocz bsipocz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks good to me, even though we cannot yet turn on rendering on the notebook as it still uses the bucket for the ZTF data.

@troyraen
Copy link
Contributor Author

troyraen commented Sep 13, 2024

The new bucket is in a different region. How does this affect performance? Collecting results of time tests in this comment.

Test code, to be run from Fornax and ISP:

import sample_selection
import wise_functions

coords, labels = [], []
sample_selection.get_sdss_sample(coords, labels, num=100)
sample_table = sample_selection.clean_sample(coords, labels)

%time unwise_df = wise_functions.wise_get_lightcurves(sample_table, radius=1.0, bandlist=['W1', 'W2'])

Results (Wall time):

Bucket (region) Fornax ISP
irsa-mast-tike-spitzer-data
(us-east-1) 7min 37s 16min 53s
7min 20s 18min 55s
6min 45s 17min 58s
nasa-irsa-wise
(us-west-2) 9min 21s 8min 36s
8min 6s 8min 14s
8min 6s 7min 33s

I believe Fornax is running in us-east-1 and ISP close to us-west-2. So, relative to hitting the close bucket, hitting the far bucket increases wall time by about a factor of about 1.2 on Fornax and 2.2 on ISP.

@troyraen troyraen force-pushed the raen/patch/unwise-bucket branch from 0b8c9e5 to 393a4bb Compare September 17, 2024 01:44
@troyraen troyraen marked this pull request as ready for review September 17, 2024 01:45
@bsipocz bsipocz merged commit 57b44eb into main Sep 17, 2024
3 checks passed
@bsipocz bsipocz deleted the raen/patch/unwise-bucket branch September 17, 2024 02:22
@bsipocz
Copy link
Member

bsipocz commented Sep 17, 2024

Thanks!

github-actions bot pushed a commit that referenced this pull request Sep 17, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants