What
A single key hash accounts for 1,815 failed auth attempts in 2 hours on litellm-community (about 15/min, sustained). Two more are doing the same at lower volume:
| key hash (prefix) |
attempts / 2h |
684d383f6140 |
1,815 |
8a2a0db71c07 |
566 |
92fb6c51db26 |
90 |
Every one returns 401 Unable to find token in cache or LiteLLM_VerificationTokenTable.
Why it is worth a look
The top hash does not exist in LiteLLM_VerificationToken at all, not even soft-deleted. So it is not a churned member whose key was revoked: it is a key we never issued. Either a client configured with a typo'd or fabricated key, or someone probing.
Cost today is small but not zero: each attempt is a cache miss plus a DB lookup, and the failures dominate the proxy's ERROR log to the point that they buried the evidence while diagnosing the 2026-08-07 OOM incident (helmcode/nan-devops#114). The signal-to-noise cost is the real one.
The requester IP in the logs is always 10.244.8.25, the in-cluster tunnel pod, so the originating IP has to come from cf-connecting-ip / the client_ip request tag rather than from requester_ip.
Options
- Identify the origin via Cloudflare (
cf-connecting-ip on the 401s) and, if it is a real member with a broken config, tell them.
- Rate-limit repeated auth failures per source IP at the edge. Note this is auth failure, so it happens before the usage hook and before any per-key limit; it would have to live in Traefik or Cloudflare.
- Accept it, but lower the log level of the 401 path so it stops drowning the proxy's ERROR stream.
What
A single key hash accounts for 1,815 failed auth attempts in 2 hours on
litellm-community(about 15/min, sustained). Two more are doing the same at lower volume:684d383f61408a2a0db71c0792fb6c51db26Every one returns
401 Unable to find token in cache or LiteLLM_VerificationTokenTable.Why it is worth a look
The top hash does not exist in
LiteLLM_VerificationTokenat all, not even soft-deleted. So it is not a churned member whose key was revoked: it is a key we never issued. Either a client configured with a typo'd or fabricated key, or someone probing.Cost today is small but not zero: each attempt is a cache miss plus a DB lookup, and the failures dominate the proxy's ERROR log to the point that they buried the evidence while diagnosing the 2026-08-07 OOM incident (helmcode/nan-devops#114). The signal-to-noise cost is the real one.
The requester IP in the logs is always
10.244.8.25, the in-cluster tunnel pod, so the originating IP has to come fromcf-connecting-ip/ theclient_iprequest tag rather than fromrequester_ip.Options
cf-connecting-ipon the 401s) and, if it is a real member with a broken config, tell them.