Measure the shared tier's connection reuse and operation latency

[jerenkrantz]

Two questions about the shared tier have no answer in this repository today.
How often does a request reuse a connection, and how long does an operation
take?

Neither is answerable from what is exposed now. There is no latency metric of
any kind — the _seconds families are all ages and uptime — and the S3
client's connection pool is whatever the AWS SDK resolved, which retains a
bounded number of idle connections per host (10 by default) against a much
larger cap on total connections. A process making more concurrent requests
than that idle bound closes connections it is about to want again, and each
replacement pays a TCP handshake and then a TLS handshake before its request
is sent at all. One client serves both paths — gets on the read path, puts on
the upload pool whose size is configurable — so both can be affected.

Whether that costs anything in practice is a guess until it is counted. This
counts it and changes nothing else.

Connection reuse

net/http/httptrace's GotConn reports, per request, whether the transport
had a connection to hand over. A wrapper around the SDK's HTTP client attaches
one trace to every request and counts the two cases apart:

plaid_cache_remote_requests_total{conn="reused"} 190
plaid_cache_remote_requests_total{conn="new"} 22

The family is named for requests rather than connections because requests is
what it counts; conn="new" is the subset that had to establish one. new
divided by the total is the number the whole change exists to produce, and
plaid-cache status prints it as a conns line for anyone without a scraper.

The wrapper wraps rather than replaces. By the time a client option function
runs, Options.HTTPClient already holds the client the SDK resolved — pool
settings, dialer and TLS timeouts and all — so wrapping it cannot change any
of them. That is deliberate: the pool is untouched by construction, not by
promise. Every request the client makes is counted, including a retried
attempt and the session a directory bucket negotiates for itself, since those
compete for the same idle connections as the gets and puts.

Operation latency

A histogram per operation and outcome, which is what says whether reuse
matters — a handshake nobody waits on is a curiosity:

plaid_cache_remote_operation_duration_seconds_bucket{operation="get_object",outcome="ok",le="0.005"} 2
...
plaid_cache_remote_operation_duration_seconds_bucket{operation="get_object",outcome="ok",le="+Inf"} 5
plaid_cache_remote_operation_duration_seconds_sum{operation="get_object",outcome="ok"} 0.031
plaid_cache_remote_operation_duration_seconds_count{operation="get_object",outcome="ok"} 5

operation is one of get_action, put_action, get_object, put_object,
which is also the read/write split: a read blocks the build that asked for it
and a write is best-effort and off the critical path, so the two are not the
same measurement and a percentile over their union describes neither.
outcome is ok, miss, or error — a miss is the cheap path, with no body
to transfer, so folding it into successes would make read latency describe a
mixture that moves with the hit rate. Twelve pairs at most, fixed at compile
time, in a repository whose metrics endpoint takes label cardinality
seriously.

It is a histogram rather than a mean because the read path blocks callers and
the tail is the part that costs anything: one request in fifty waiting on a
handshake is invisible in an average and is exactly the thing being looked
for.

The histogram is hand-rolled. No metrics framework: a dozen float comparisons
and an atomic add is what this needs, and the exposition here is already
written by hand for the reasons renderMetrics gives. Buckets run 1ms to 10s
with most of the resolution below 100ms — that is the band where a request
that established a connection and one that reused an idle one are
distinguishable, which is why these two measurements belong in one change. The
edges and the reasoning behind each band are in a comment on
durationBuckets.

Recording is meant to be cheaper than what it measures, since uploads can
reach a few hundred a second per process: counts are per bucket and made
cumulative only when a snapshot is taken, and the sum is integer nanoseconds
so it needs no compare-and-swap loop. Measured:

BenchmarkObserve-12    3271532    367.1 ns/op    0 B/op    0 allocs/op

A get is measured to the open body rather than through it — the reader is
handed to the caller, so draining it is the caller's transfer and not this
tier's round trip — which makes reads first-byte latency. A put is the whole
transfer, because this package does stream that one itself. Both are noted in
comments at the measurement points and in the metric's help text.

Plumbing

Both measurements travel the route every other number here takes:
Cache.RemoteStats → StatusResponse → the socket's status report, the
monitoring route, and the exposition rendered from that same report. There is
still one place the numbers come from, so the three cannot disagree.

RemoteStats reaches the backend through an optional remote.Statser
interface rather than a new method on Backend: Noop makes no requests and
a test's fake makes them against nothing, so obliging every implementation to
return an empty snapshot would be ceremony in several places to say so. The
field on StatusResponse is a pointer, so a local-only daemon publishes no
transport families at all — it has not failed to reuse a connection, it has
not tried — and an operation that has not happened publishes no distribution,
rather than a bucket set of zeros indistinguishable from a tier that answered
instantly.

This is the endpoint's first histogram, so renderMetrics grows a second
render path: one HELP and one TYPE line for a family that then emits three
derived series names per label set, le="+Inf" written from the count because
the format requires the two to be equal, and buckets labelled with the edges
the recording package owns rather than a second copy of them.

What this deliberately does not do

It does not change the connection pool. That is the obvious follow-up and
it is left for one, on purpose: a change that adjusts behaviour and adds the
instrument that would evaluate the adjustment in the same commit makes the
before and after unreadable. Tune it once there is a reuse ratio to point at.

Defaults are otherwise unchanged, and no dependency was added.

Tests

• The reuse counter against a local server with pooling the test controls:
  with keep-alives the second request onward reuses the first's connection,
  with them disabled every request opens its own.
• End to end through the S3 backend against the package's existing fake S3
  server, including that the second of two gets reuses the first's connection,
  and that a miss and a failure land under different outcome labels.
• Duration against a handler that holds the request, asserting a lower bound
  rather than an interval: a slow machine can always take longer, but no call
  can return before the handler does.
• Bucket cumulativeness, ascending edges, the negative-duration guard, and
  that unobserved pairs are omitted.
• The exposition's types, its bucket/sum/count series, that the histogram
  declares HELP and TYPE exactly once, and that a local-only daemon emits
  neither family.
• The conns line over the JSON the daemon actually sends, and its absence
  when the daemon sends no accounting.

go build ./..., go vet ./..., go test -race ./... and
golangci-lint run --max-issues-per-linter=0 --max-same-issues=0 ./... are all
clean.