Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Don't assume provenance summary is present. #3419

Merged
merged 1 commit into from
Oct 30, 2023

Conversation

rebeccacremona
Copy link
Contributor

We are observing that for some small percentage of Scoop captures, attachments (including the provenance summary) are absent.

This PR makes sure that's not counted as a failure: if we have a decent warc, we shouldn't be too upset if we don't have all the metadata we wanted.

I made a new Grafana panel so we can watch these go by.

Once this is up on prod, I'll clean up the records from over the weekend.

@rebeccacremona rebeccacremona requested a review from a team as a code owner October 30, 2023 14:54
@rebeccacremona rebeccacremona requested review from kilbergr and removed request for a team October 30, 2023 14:54
@bensteinberg bensteinberg removed the request for review from kilbergr October 30, 2023 14:55
@rebeccacremona rebeccacremona merged commit ac1454d into harvard-lil:develop Oct 30, 2023
2 checks passed
@codecov
Copy link

codecov bot commented Oct 30, 2023

We're currently processing your upload. This comment will be updated when the results are available.

@rebeccacremona
Copy link
Contributor Author

Here's how I plan to clean up:

from perma.models import Capture

captures = Capture.objects.filter(
     role='primary',
     status='success',
     link__capture_job__engine='scoop-api',
     link__capture_job__status='failed'
)

captures.count()

for capture in captures:
   capture.link.tags.add('scoop-exception-while-finishing-job')

captures.update(status='failed')

This will correct ~90 records.

@rebeccacremona rebeccacremona deleted the no-provenance branch October 30, 2023 18:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants