Automated CKAN Job Error - Washington + Connecticut Stuck Jobs

[nickumia-reisys]

Workflow with Issue: 4 - Automated CKAN Jobs
Job Failed: ckan-auto-command
CKAN Command (in question): ckan geodatagov check-stuck-jobs
CKAN Command Schedule: 30 6 * * *
Cloud.gov Environment: prod

Last Commit: 78a71a3
Number of times run: 1
Last run by: btylerburton
Github Action Run: https://github.com/GSA/catalog.data.gov/actions/runs/5319299028

[nickumia-reisys]

Related to

• O+M 2023-06-23 data.gov#4363

Two stuck jobs

2023-06-19T06:39:28.67+0000 [APP/TASK/ckangeodatagovcheck-stuck-jobs-5308607169-1/0] OUT 2023-06-19 06:39:28,679 INFO  [ckanext.geodatagov] source_id: 1c33d1cb-2f69-4f1b-835e-453790f38dc7 | created_time: 2023-06-16 08:12:39.926907 | current_time: 2023-06-19 06:39:28.661965+00:00 | gather_started: 2023-06-16 09:07:46.595433 | gather_finished: None | running_length: 2 days, 22:26:48.735058 | source_title: WA JSON | organization: State of Washington
2023-06-19T06:39:28.67+0000 [APP/TASK/ckangeodatagovcheck-stuck-jobs-5308607169-1/0] OUT 2023-06-19 06:39:28,679 INFO  [ckanext.geodatagov] source_id: 36c82f29-4f54-495e-a878-2c07320bf10c | created_time: 2023-06-16 08:12:34.976809 | current_time: 2023-06-19 06:39:28.661965+00:00 | gather_started: 2023-06-16 09:05:46.030295 | gather_finished: None | running_length: 2 days, 22:26:53.685156 | source_title: Connecticut Data.json | organization: State of Connecticut

Both harvests got some data, but it was really slow and it had a lot more errors. @FuhuXia is this a data quality issue? Or is it a catalog issue?

State of Connecticut

State of Washington

[FuhuXia]

they should be finished in hours. days is wrong.

[nickumia-reisys]

We couldn't find a solid conclusion about why or how this happened. Gather finished within seconds. Fetch didn't start until a day later. Even after fetch finished within minutes, the harvester did not consider the job complete until it was forcibly closed.

This has happened before where a few dataset updates took a day to pass. But there is nothing evident in the logs about what was happening.

The next time this happens, if it is stuck and we manually run ckan harvester run and it doesn't close the job, then the connection between harvester run and the specific job needs to be investigated. Either ckan harvester run does not know about the job or it is ignoring it for some weird reason.