O+M 2023-06-30

[hkdctol]

As part of day-to-day operation of Data.gov, there are many Operation and Maintenance (O&M) responsibilities. Instead of having the entire team watching notifications and risking some notifications slipping through the cracks, we have created an O&M Triage role. One person on the team is assigned the Triage role which rotates each sprint. This is not meant to be a 24/7 responsibility, only East Coast business hours. If you are unavailable, please note when you will be unavailable in Slack and ask for someone to take on the role for that time.

Miscs

• Watch for user email requests
• Watch in #datagov-alerts and Vulnerable dependency notifications (daily email reports) for critical alerts.
• Monitor and improve Data.gov O&M Dashboard

Acceptance criteria

You are responsible for all O&M responsibilities this week. We've highlighted a few so they're not forgotten. You can copy each checklist into your daily report.

Daily Checklist

Check Production State/Actions

• README

Note: Catalog Auto Tasks
You will need to update the chart values manually. Click the Action link in each issue and grab the values from monitor task output and check runtime.

• DB-Solr Sync
• Tracking Update
• Check Harvesting Emails
• New Relic Alerts Triaged
• Triage DMARC Report from Google

Weekly Checklist

• Audit Log (more info on AU-3 and AU-6 Log auditing)
• Check Catalog Solr
• Catalog Dupe Check
• Invicti Scan

[hkdctol]

Rotation list https://docs.google.com/spreadsheets/d/1cF73HkzFuQth3z9voLfVO1Cc4kjBd6ecHGE2gkj5LCc/edit?usp=sharing

[FuhuXia]

GSA/catalog.data.gov#973

ckan harvester run command failed one time when processing harvest source

...
2023-06-27T08:14:16.66+0000 [APP/TASK/ckanharvesterrun-5387335272-1/0] OUT 2023-06-27 08:14:16,659 INFO  [ckanext.datagovcatalog.harvester.notifications] Adding extra recipients for source 38f3722d-48ce-4dbf-abde-941b707e5ad5
2023-06-27T08:14:18.64+0000 [APP/TASK/ckanharvesterrun-5387335272-1/0] OUT 2023-06-27 08:14:18,645 INFO  [ckanext.geodatagov.plugin] Added FQ to collection_package_id
2023-06-27T08:14:19.19+0000 [APP/TASK/ckanharvesterrun-5387335272-1/0] OUT Exit status 1

Reharvest the source 38f3722d-48ce-4dbf-abde-941b707e5ad5 and watch output to make sure the error is not repeating

===============
[UPDATE]
reharvested manually on the same source. No Error.

2023-06-27T10:13:40.87-0400 [APP/TASK/ckanharvesterrun-5390812886-1/0] OUT 2023-06-27 14:13:40,872 INFO  [ckanext.geodatagov.plugin] Added FQ to collection_package_id
2023-06-27T10:13:41.37-0400 [APP/TASK/ckanharvesterrun-5390812886-1/0] OUT 2023-06-27 14:13:41,374 INFO  [ckanext.datagovcatalog.harvester.notifications] Extra recipients for source found: [{'name': '████@neh.gov', 'email': '████@neh.gov'}]
...
2023-06-27T10:13:46.21-0400 [APP/TASK/ckanharvesterrun-5390812886-1/0] OUT 2023-06-27 14:13:46,210 INFO  [ckan.lib.mailer] Sent email to ████@neh.gov
2023-06-27T10:13:46.21-0400 [APP/TASK/ckanharvesterrun-5390812886-1/0] OUT 2023-06-27 14:13:46,212 DEBUG [ckanext.harvest.logic.action.update] No jobs to send to the gather queue
2023-06-27T10:13:47.56-0400 [APP/TASK/ckanharvesterrun-5390812886-1/0] OUT Exit status 0

[FuhuXia]

https://github.com/GSA/catalog.data.gov/actions/runs/5387335272/jobs/9778522484

...
2023-06-27T08:14:02.01+0000 [APP/TASK/ckanharvesterrun-5387335272-1/0] OUT 2023-06-27 08:14:02,009 INFO  [ckanext.datagovcatalog.harvester.notifications] Extra recipients for source found: []
2023-06-27T08:14:02.38+0000 [APP/TASK/ckanharvesterrun-5387335272-1/0] OUT 2023-06-27 08:14:02,381 INFO  [ckan.lib.mailer] Sent email to █████@gsa.gov
2023-06-27T08:14:02.72+0000 [APP/TASK/ckanharvesterrun-5387335272-1/0] OUT 2023-06-27 08:14:02,725 INFO  [ckan.lib.mailer] Sent email to █████@gsa.gov
...

Logs shows we are sending harvest emails to members that have been off-boarded. Examining the system to see if this is an off-boarding issue or code bug.

=================
[UPDATE]
It is code bug. #4368 created.

[FuhuXia]

For the first time over the past a few months, our DB and SOLR are totally in sync after daily harvesting.
https://github.com/GSA/catalog.data.gov/actions/runs/5385352045/jobs/9774274968

2023-06-27 03:35:39,795 INFO  [ckanext.geodatagov] total 373590 solr indexed_package
2023-06-27 03:35:40,218 INFO  [ckanext.geodatagov] 0 packages need to be removed from Solr
2023-06-27 03:35:40,218 INFO  [ckanext.geodatagov] 0 packages need to be updated/added to Solr
2023-06-27 03:35:40,218 INFO  [ckanext.geodatagov] 0 packages without harvest_object need to be mannually deleted
Exit status 0

[FuhuXia]

Inventory solr went down, affecting staging and production. After deliberately restarting development solr, development inventory went down too. Possible cause is some system change on AWS side. Resolved a few hours later. Details in Slack discussion.