Nb disability benefits log 526 document polling gauge metrics

[NB28VT]

Summary

• *This work is behind a feature toggle (flipper): NO

We are actively ready to create these polling records in prod and running this job but none have been created yet. The only risk is it breaks the job which isn't doing anything now, and we would catch it

• (Summarize the changes that have been made to the platform)

Adds metrics for documents processed, marked successful or marked failed whenever the Form526DocumentUploadPollingJob runs.

• (What is the solution, why is this the solution?)

There are a couple different ways to do this, including incrementing a counter with each success or failure within the update loop, but I think it simplifies it to query the success and failure counts before and after the loop of update batches. And it was kind of tricky to get it working the other way. This approach might be a problem if this job was potentially run in parallel with another instance of the same job but it's not scheduled that way nor intended to be.

Also note this adds a few count queries, but they are outside the loop so they won't scale up with N number of documents, if that makes sense.

I guess one issue I could see with this approach is we could have a hang up or outage on the Lighthouse side, which might mess up the counts but I don't think that would be a big deal?

• (Which team do you work for, does your team own the maintenance of this component?)

Disability Benefits Team 2, and yes

Related issue(s)

• [LH Doc Upload] (non-release blocking) Add StatsD gauge metrics to LH upload polling va.gov-team#95603

Testing done

• New code is covered by unit tests

Acceptance criteria

• I fixed|updated|added unit tests and integration tests for each feature (if applicable).
• No error nor warning in the console.
• Events are being sent to the appropriate logging solution
• Documentation has been updated (link to documentation)
• No sensitive information (i.e. PII/credentials/internal URLs/etc.) is captured in logging, hardcoded, or specs
• Feature/bug has a monitor built into Datadog (if applicable) It will be, in [LH Doc Upload] (non-blocking for BDD release) Improve production monitoring va.gov-team#95604

[NB28VT]

Note: just moved this block to a method to keep the linter happy since I was adding new lines to the perform method