O+M 2024-07-22

[hkdctol]

As part of day-to-day operation of Data.gov, there are many Operation and Maintenance (O&M) responsibilities. Instead of having the entire team watching notifications and risking some notifications slipping through the cracks, we have created an O&M Triage role. One person on the team is assigned the Triage role which rotates each sprint. This is not meant to be a 24/7 responsibility, only East Coast business hours. If you are unavailable, please note when you will be unavailable in Slack and ask for someone to take on the role for that time.

Check the O&M Rotation Schedule for future planning.

Acceptance criteria

You are responsible for all O&M responsibilities this week. We've highlighted a few so they're not forgotten. You can copy each checklist into your daily report.

Daily Checklist

• Check Production State/Actions

Note: Catalog Auto Tasks
You will need to update the chart values manually. Click the Action link in each issue and grab the values from monitor task output and check runtime.

• Check auto generated O&M tickets from no status column
• Check Harvesting Emails
• New Relic Alerts Triaged
• Triage DMARC Report from Google

Weekly Checklist

• DB-Solr Sync
• Audit Log (more info on AU-3 and AU-6 Log auditing)
• Tracking Update
  • NOTE: This job will consistently timeout, but it is processing results ((more details)[https://github.com/change tracking update from nightly job to weekly #4345])
• Check Catalog Solr
• Catalog Dupe Check
• Check user management requests
• Check Snyk Dashboard and cleanup

Monthly Checklist

• Invicti Scan

ad-hoc checklist

• audit/review applications on cloud foundry and determine what can be stopped and/or deleted.

Reference

• Watch for user email requests
• Watch in #datagov-alerts and Vulnerable dependency notifications (daily email reports) for critical alerts.
• Monitor and improve Data.gov O&M Dashboard
• Update and revise Data.gov O&M Tasks

[Jin-Sun-tts]

Tuesday 07/23

https://github.com/GSA/data.gov/

There are some deploy and commit errors, but after rerun failed job. all passed.

Check Catalog Auto Tasks

Check Harvesting Emails

• Catalog:

• DB-Solr Sync:
  failed run, will rerun to see if fix the issue.

Got email regarding: SecOps Monthly Q-Data Unauthenticated Web Application Scan Results: July 2024

two Medium vulnerabilities found and one was false-alarm, the other one relates cross-site scripting. created ticket: #4826)

[Jin-Sun-tts]

Wednesday 07/24

https://github.com/GSA/data.gov/

Check Catalog Auto Tasks

Check Harvesting Emails

• Catalog:

• DB-Solr Sync:
  0 packages need to be removed from Solr
  0 packages need to be updated/added to Solr
  11874 packages without harvest_object need to be manually deleted
  Finished 487s

checked most of these objects are from NOAA harvest source ioos, it happened after 07/19/2024 harvest job.

Changed the harvest frequency from weekly to monthly.

Will manually delete those 11874 harvest_objects

[Jin-Sun-tts]

Friday 07/26

1 failing
test-1 |
test-1 | 1) Dataset
test-1 | Can click on feedback button:
test-1 | AssertionError: Timed out retrying after 4000ms: Expected to find element: #fba-modal-dialog, but never found it.
test-1 | at Context.eval (webpack:///./cypress/integration/dataset.cy.js:66:36)

Weekly Checklist

• DB-Solr Sync:
  0 packages need to be removed from Solr
  0 packages need to be updated/added to Solr
  1 packages without harvest_object need to be mannually deleted
  Finished 547s
• Audit Log (more info on AU-3 and AU-6 Log auditing)
• Tracking Update
  2024-06-14T07:38:03.74+0000 [APP/TASK/ckangeodatagovtracking-update-9512411592-1/0] OUT 2024-06-14 07:38:03,741 INFO [ckanext.geodatagov] 9178 package indexes to be rebuilt starting from 2024-06-11 00:00:00
• Check Catalog Solr
• Catalog Dupe Check