Report error ingestion through OpenTelemetry (3/4)

[johnsimons]

Part 3 of 4 in a series that reworks how both instances ingest messages. It comes out of the parallel audit writers spike in #5318. Master has moved a long way past that spike's base, so this ports the ideas rather than the files.

1. Extract the ingestion batching loop into a shared pipeline
2. Let the storage decide whether ingestion batches run in parallel
3. Report error ingestion through OpenTelemetry (this PR)
4. Document the ingestion pipeline and its telemetry

Each branch is a single commit and stacks on the one before it, so review them in order. 1/4 is based on master. The series is independent of #NNNN ("Order the failed message status by the times the events happened"), which is a standalone bug fix going out alongside it.

---

Why

The audit instance has had OpenTelemetry ingestion metrics for a while: batch and per message durations as histograms, a failure counter, and a gauge for consecutive batch failures, all exportable over OTLP and graphable. The error instance had counters that a hosted service printed to the log every five seconds when PrintMetrics was on, which cannot be graphed and cannot be alerted on. Now that writer count and batch size are tunable, the numbers that tell you how to set them have to be visible.

What this does

The error instance reports the same shape as audit, under sc.error.ingestion.* on the meter Particular.ServiceControl, with OtlpEndpointUrl wired up the same way it is on the audit instance.

Instrument names are what dashboards are built on, so both instances now have a test that pins theirs. The audit names are unchanged.

Where the two genuinely differ:

• The error side keeps the bulk insert timing it already had, as sc.error.ingestion.storage_duration_seconds. It is the part of a batch that is neither announcing nor forwarding, and the audit ingestor has no equivalent.
• message.category splits retry confirmations from failed messages, which is the same split ErrorIngestor makes when it divides a batch. The two cost very different amounts of work, so averaging them together hides both.

The scopes behind all of this move into ServiceControl.Infrastructure: one batch, one message, one failure, one plain duration, and the histogram buckets they share, so both instances stay comparable on the same dashboard. What stays per instance is what actually differs, which is instrument names, descriptions and tags. Hosting audit ingestion in the primary will pick these up rather than copy them.

The failure gauge is now atomic

consecutive_batch_failures_total was backed by a plain field, which was safe while one writer owned the loop. With parallel writers it loses increments on the one number an alert would be built on, so it is updated with Interlocked and read with Volatile.Read. Both instances have a test that fails a thousand batches concurrently and checks the gauge; it reads short without the fix.

The legacy counters

They leave the ingestion path, and with them the last thing that called Metrics.GetCounter or GetMeter anywhere. The reporter and PrintMetrics stay registered and now have nothing to report. Removing that plumbing, and the setting with it, is a decision on its own rather than a side effect of this PR.

[rbev]

Should this env var fall back to the otel specification naming: OTEL_EXPORTER_OTLP_ENDPOINT?
You might be able to remove all the configuration as this is a standard that I believe is automatically picked up by the call to WithMetrics.

[johnsimons]

I just followed what is already in Audit