feat: alert instead of crashing relayer when building cursors

[daniel-savu]

Description

• we should make sure there are no unwraps outside a tokio::spawn, otherwise they'll take down the entire relayer. The offenders have so far been the indexing cursors, so this PR just logs an error and sets a new metric flag instead of panicking: critical_error, an IntGauge type because prometheus doesn't have boolean gauges. For now this is only used for origin chains, but I envision this as being set when a critical error occurs in the submitter too, in the future
• before setting the flag, cursor instantiation is now retried 10 times (to avoid short lived RPC hiccups)

Drive-by changes

Removes usage of Builder derive macro in metric types, since they're not complex enough to warrant it

Related issues

Backward compatibility

Yes

Testing

the existing e2e

[changeset-bot]

⚠️ No Changeset found

Latest commit: 5d41f58

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 74.27%. Comparing base (ffbc13c) to head (5d41f58).
Report is 4 commits behind head on main.

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #4811   +/-   ##
=======================================
  Coverage   74.27%   74.27%           
=======================================
  Files         101      101           
  Lines        1481     1481           
  Branches      192      192           
=======================================
  Hits         1100     1100           
  Misses        360      360           
  Partials       21       21           

Components	Coverage Δ
core	84.61% <ø> (ø)
hooks	77.77% <ø> (ø)
isms	77.77% <ø> (ø)
token	89.01% <ø> (ø)
middlewares	77.58% <ø> (ø)

[tkporter]

I imagine we'll wanna document this in the public docs at some point

[daniel-savu]

I imagine we'll wanna document this in the public docs at some point

@tkporter good point, I'll add a quick entry to the relayer monitoring & alerting guide (https://docs.hyperlane.xyz/docs/operate/relayer/monitoring-alerting)