O+M 2023-07-07

[hkdctol]

As part of day-to-day operation of Data.gov, there are many Operation and Maintenance (O&M) responsibilities. Instead of having the entire team watching notifications and risking some notifications slipping through the cracks, we have created an O&M Triage role. One person on the team is assigned the Triage role which rotates each sprint. This is not meant to be a 24/7 responsibility, only East Coast business hours. If you are unavailable, please note when you will be unavailable in Slack and ask for someone to take on the role for that time.

Check the O&M Rotation Schedule for future planning.

Miscs

• Watch for user email requests
• Watch in #datagov-alerts and Vulnerable dependency notifications (daily email reports) for critical alerts.
• Monitor and improve Data.gov O&M Dashboard

Acceptance criteria

You are responsible for all O&M responsibilities this week. We've highlighted a few so they're not forgotten. You can copy each checklist into your daily report.

Daily Checklist

Check Production State/Actions

• README

Note: Catalog Auto Tasks
You will need to update the chart values manually. Click the Action link in each issue and grab the values from monitor task output and check runtime.

• DB-Solr Sync
• Check Harvesting Emails
• New Relic Alerts Triaged
• Triage DMARC Report from Google

Weekly Checklist

• Audit Log (more info on AU-3 and AU-6 Log auditing)
• Tracking Update
• Check Catalog Solr
• Catalog Dupe Check
• Invicti Scan

[nickumia-reisys]

Day 1 Summary

• Deployed CKAN 2.10 to inventory prod
• Inventory prod Site Alarm broke on NR (haven't been able to figure out why)
• Worked on inventory flask + werkzeug vulnerabilities
  • A lot of requirement updates are necessary
  • flask-multistatic is not compatible with the non-vulnerable flask+ werkzeug versions. I made a temporary repo to implement the fix that makes it compatible (but don't know if we want to do that). CKAN removed flask-multistatic as a dependency on main branch (the next version of CKAN will have a different fix for that)
  • Got all but 2 tests passing
• State Json harvest source broken on catalog (will connect with @FuhuXia on where that stands from last week)
• Surprisingly the end of month harvest jobs did not cause any stuck jobs this time 😮

[nickumia-reisys]

Day 2 Summary

• There was a glitch in FEMA-R02 harvest source. It failed with HTTPSConnectionPool(host='hazards.fema.gov', port=443): Max retries exceeded with url: /filedownload/metadata/R02/599696-AcquireBaseMap.xml (Caused by ProxyError('Cannot connect to proxy.', timeout('The read operation timed out'))). Re-running the harvest job did not reproduce the error.
• Phila data json harvest source has been failing since the beginning of June. Apparently, they redirected www.opendataphilly.org to opendataphilly.org and we weren't allowing the root domain.
  • Fixed in update: allow acl for catalog harvest #4380
• Determined that there is a bug in CKAN 2.10 where old API Tokens are no longer work. (reference)

[nickumia-reisys]

Day 3 + 4 Summary

• Catalog experienced an outage for ~30 mins because all 3 solr followers restarted at the same time (reference)
  • Removed Follower 0 from Catalog prod traffic. Going to see if the requests are actually causing the restarts or if it is the replication.
• Alaska harvest sources are partially failing, might be intermittent issues
  
• Investigated an error that's been appearing in the catalog logs (reference). It seems related to beaker, but the beaker secrets seem to be set up properly.