What
nan_usage_untracked_total{cause="no_member_id"} keeps moving on litellm-community: 41 in 7 days, and still ~16 per 20h after PR helmcode/nan-devops#113. Every increment is a successful call to a capped, resold model that nobody was debited for.
The log line is "<outcome> event missing user_api_key_user_id; skipping" (usage_hook.py, _track).
Why it is odd
The pre-call gate already refuses any key it cannot resolve to a member (no_discord_id_on_key -> 402). So these requests passed the gate, which means the member WAS resolved before the call, and then the post-call logging event arrived without user_api_key_user_id in its metadata. It is a metadata-propagation gap inside LiteLLM, not an orphan key.
Consequence: the member consumed a per-token model we pay a third party for, their monthly counter did not move, and their remaining allowance is overstated by exactly that much.
What is known
- Only ever
deepseek-v4-flash. Never glm5.2, never mimo-v2.5. That asymmetry is the best lead.
deepseek-v4-flash is the one gated group with multiple deployments/providers, and litellm_deployment_successful_fallbacks_total shows ~1,138 fallbacks with exception_status=404 in 20h. Fallbacks rebuilding metadata is the first hypothesis to test, though the volumes do not match (1,138 vs ~16), so it would be a specific sub-case.- Synthetic probe keys are already exempted (
_is_synthetic_key), so this is not the Gatus monitor.
- Volume is small: roughly 2 to 16 a day against ~110k requests/day to that model.
Suggested next step
Reproduce against litellm-test by forcing a fallback on a gated group and inspecting the kwargs metadata the failure/success hook receives. If the id is genuinely absent on the fallback path, the fix is to carry the pre-call-resolved id forward (the hook already mints a request id in _mint_request_id; the member id could ride the same way) rather than re-reading it from metadata post-call.
Related
- helmcode/nan-devops#113 removed the other untracked cause:
platform_api_unreachable went from 30 in 7 days to 0 in 20h.
What
nan_usage_untracked_total{cause="no_member_id"}keeps moving onlitellm-community: 41 in 7 days, and still ~16 per 20h after PR helmcode/nan-devops#113. Every increment is a successful call to a capped, resold model that nobody was debited for.The log line is
"<outcome> event missing user_api_key_user_id; skipping"(usage_hook.py,_track).Why it is odd
The pre-call gate already refuses any key it cannot resolve to a member (
no_discord_id_on_key-> 402). So these requests passed the gate, which means the member WAS resolved before the call, and then the post-call logging event arrived withoutuser_api_key_user_idin its metadata. It is a metadata-propagation gap inside LiteLLM, not an orphan key.Consequence: the member consumed a per-token model we pay a third party for, their monthly counter did not move, and their remaining allowance is overstated by exactly that much.
What is known
deepseek-v4-flash. Never glm5.2, never mimo-v2.5. That asymmetry is the best lead.deepseek-v4-flashis the one gated group with multiple deployments/providers, andlitellm_deployment_successful_fallbacks_totalshows ~1,138 fallbacks withexception_status=404in 20h. Fallbacks rebuilding metadata is the first hypothesis to test, though the volumes do not match (1,138 vs ~16), so it would be a specific sub-case._is_synthetic_key), so this is not the Gatus monitor.Suggested next step
Reproduce against
litellm-testby forcing a fallback on a gated group and inspecting thekwargsmetadata the failure/success hook receives. If the id is genuinely absent on the fallback path, the fix is to carry the pre-call-resolved id forward (the hook already mints a request id in_mint_request_id; the member id could ride the same way) rather than re-reading it from metadata post-call.Related
platform_api_unreachablewent from 30 in 7 days to 0 in 20h.