O+M 2023-08-04

[nickumia-reisys]

As part of day-to-day operation of Data.gov, there are many Operation and Maintenance (O&M) responsibilities. Instead of having the entire team watching notifications and risking some notifications slipping through the cracks, we have created an O&M Triage role. One person on the team is assigned the Triage role which rotates each sprint. This is not meant to be a 24/7 responsibility, only East Coast business hours. If you are unavailable, please note when you will be unavailable in Slack and ask for someone to take on the role for that time.

Check the O&M Rotation Schedule for future planning.

Miscs

• Watch for user email requests
• Watch in #datagov-alerts and Vulnerable dependency notifications (daily email reports) for critical alerts.
• Monitor and improve Data.gov O&M Dashboard

Acceptance criteria

You are responsible for all O&M responsibilities this week. We've highlighted a few so they're not forgotten. You can copy each checklist into your daily report.

Daily Checklist

Check Production State/Actions

• README

Note: Catalog Auto Tasks
You will need to update the chart values manually. Click the Action link in each issue and grab the values from monitor task output and check runtime.

• DB-Solr Sync
• Check Harvesting Emails
• New Relic Alerts Triaged
• Triage DMARC Report from Google

Weekly Checklist

• Audit Log (more info on AU-3 and AU-6 Log auditing)
• Tracking Update
• Check Catalog Solr
• Catalog Dupe Check
• Invicti Scan

[nickumia-reisys]

Day 1 Summary

• Added users to inventory,
  • https://github.com/GSA/datagov-account-management/issues/515
  • https://github.com/GSA/datagov-account-management/issues/517
• Solr Followers 0,1,2 restarted back-to-back on 7/30 at 9a
• Fixed Snyk Scans
  • Get Snyk Scan to run again catalog.data.gov#1010
  • Extend snyk exceptions inventory-app#640

[nickumia-reisys]

Day 2 Summary

• Merged (security) PRs
  • [Snyk] Security upgrade sphinx from 3.0.4 to 3.3.0 inventory-app#642
  • update dcat-usmetadata app version. inventory-app#641
  • [Snyk] Upgrade postcss from 8.4.23 to 8.4.24 resources.data.gov#598
  • Bump nokogiri from 1.13.10 to 1.14.3 data-strategy#312
  • Bump nokogiri from 1.13.10 to 1.14.3 us-data-federation#74
  • Adjust relative link to initiatives for frontpage us-data-federation#13
  • [Patch/Fixed] Integer Overflow or Wraparound in libxml2 affects Nokogiri sdg-indicators-usa#1043
  • build(deps): bump cryptography from 41.0.2 to 41.0.3 inventory-app#643
  • build(deps): bump cryptography from 41.0.2 to 41.0.3 in /ckan catalog.data.gov#1011
  • Update Cryptography catalog.data.gov#1012
  • build(deps): bump werkzeug from 2.0 to 2.2.3 in /ckan catalog.data.gov#996
• Harvesting issues
  • Could not harvest WAF link https://maps.dot.gov/NTADmetadata/Amtrak_Routes.xml: HTTPSConnectionPool(host='[maps.dot.gov](http://maps.dot.gov/)', port=443): Max retries exceeded with url: /NTADmetadata/Amtrak_Routes.xml (Caused by ProxyError('Cannot connect to proxy.', timeout('The read operation timed out')))
    • I checked catalog-admin ability to connect to https://maps.dot.gov` and it seemed fine, so not sure if it is a glitch or there's maintenance when harvesting is running?
• Catalog duplicate check showed a weird output (slack reference)
• Catalog duplicate datasets need to be cleaned up (will tackle tomorrow). It's super strange that many organizations have one dataset duplciated thousands of times.... suuupperrrrr weird..
  
• Solr Followers restarted 1 (5:35p), 2 (6:12p), 0 (8p)

[nickumia-reisys]

Day 3 Summary

• PRs
  • [Snyk + GH Actions] Update requirements catalog.data.gov#1013
  • [Snyk] Upgrade postcss from 8.4.24 to 8.4.25 resources.data.gov#601
• Noticed a few Response time (web) (High) on catalog alerts 😕
• Harvest Issues
  • Unable to get content for URL: https://edg.epa.gov/WAFer_harvest/ISO/: HTTPError('404 Client Error: Not Found for url: https://edg.epa.gov/WAFer_harvest/ISO/')
    • I feel like this is a "server down for maintenance during harvesting" issue because the link is available now.
• Dataset dedupe run
  • fema-gov (geospatial)
    • Bug noticed in dedupe script, Dedupe script is broken #4412 ... (slack reference)
  • coa-gatech-edu (geospatial)
  • ibwc-gov (geospatial)
  • noaa-gov (geospatial)
  • state-of-alaska (geospatial)
  • ca-gov (datajson)
  • census-gov (geospatial)
  • epa-gov (geospatial)
  • wdu-edu (geospatial)
  • usgs-gov (geospatial)
  • pennsylvania-geospatial-data-clearinghouse (geospatial)
• Spent the rest of the time working on Forward User-Agent from AWS CloudFront to catalog app #4059

[nickumia-reisys]

Day 4 +5 Summary

• Not sure what happened haha...
• Worked primarily on Add a routing to check egress status for ckan apps #4407