Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Verify production dataset does not have malformed manifests #126

Open
mtauraso opened this issue Dec 3, 2024 · 2 comments
Open

Verify production dataset does not have malformed manifests #126

mtauraso opened this issue Dec 3, 2024 · 2 comments
Assignees

Comments

@mtauraso
Copy link
Collaborator

mtauraso commented Dec 3, 2024

Through an unknown set of steps Drew managed to create a set of downloaded files (and resulting manifest) where mismatches existed between the filenames of the fits files and the metadata columns (object_id, ra, dec, tract, filter).

It is believed that some set of invocations of the downloader, possibly using nonzero offsets can create this condition. If the condition can be created with zero-length offsets then the production data is definitely affected.

@mtauraso
Copy link
Collaborator Author

mtauraso commented Dec 3, 2024

Probably the first step is to write a short script to detect this condition and run it in prod to see if we need to heal prod.

Then we can try to figure out how such a thing could have happened.

@mtauraso
Copy link
Collaborator Author

mtauraso commented Dec 3, 2024

manifest.fits.zip

@mtauraso mtauraso self-assigned this Dec 6, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant