web-search: alert on the fallback rate, and pay the Brave plan

[sre-helmcode]

Follow-up from the 2026-08-19 incident where web_search returned explicit xvideos.com links to an enterprise customer. The fix shipped (helmcode/nan-devops#189, helmcode/enterprise-api#28, helmcode/nan-cloud-api#117); these are the two things it deliberately did NOT do.

1. The Brave plan is answering HTTP 402 Payment Required

GET https://api.search.brave.com/res/v1/web/search
HTTP/1.1 402 Payment Required

The same websearch-brave key is mounted in helmcode-system and nan-system, so both tenants run 100% on scraped engines until this is resolved. Now filtered, but still scraped. Decide explicitly whether to renew: the fallback is in reasonable shape after the fix (duckduckgo + yep), so "stop paying Brave" is a legitimate option — but today the code still tries Brave first on every search and eats the 402.

2. There is no alert on the fallback rate

This is the incident's own lesson and it is not implemented. Both layers returned 200 throughout: the paid primary failed, the free fallback answered, and nothing fired. The only signal that anything was wrong was the content of the results, reported by a customer over email.

An alert on errors would not have caught it. The alert has to be on the fallback rate.

Two signals that fail independently and must both be named:

• brave-denied rate — the primary is rejecting us (401/402). No retry fixes it; it needs a human.
• fallback-returned-zero — the scraped path itself is dry. After disabling bing, the real carriers are duckduckgo (which CAPTCHAs intermittently) and yep; mojeek and google cse contribute nothing from our egress IPs, so losing one carrier is closer than it looks.

What it needs

There is no counter today — the fix only logs. So: a Prometheus counter in the Go layer (labelled by outcome), then a Grafana rule. Grafana alerting here is provisioned through the UI/API rather than from the repo, which is why it was out of scope for the incident fix.

Structured log events already in place to build on:

Event	Meaning
searchtool_brave_denied	Brave 401/402, logged at ERROR, throttled to 1 per 5 min
searchtool_brave_empty	Brave answered with zero results
websearch_poisoned_engine	an engine's whole batch was discarded as off-query
websearch_adult_filtered	NSFW hosts removed
websearch_guard_inactive	no engine attribution at all; the guard is a no-op
websearch_guard_skipped	no engine matched the query; the guard did not run

3. Smaller, from the security review

• cacheKey has no product prefix. Community and enterprise share one Valkey keyspace (searxng-valkey.helmcode-system) and are separated only by the tenant UUID. True in practice, not structural. A com:/ent: prefix would make it structural.
• deploy.yml detects the version bump with git diff HEAD~1. On a rebase merge of a multi-commit branch whose VERSION bump is not in the last commit, the deploy silently does not fire — a failure that looks like success, the same shape as this incident. Either check git diff <base>..HEAD or disable rebase merge. Affects enterprise-api and nan-cloud-api.
• Recovering blocklist recall, if ever wanted, should be a textual signal (density of adult terms across title and snippet) used ONLY to corroborate a whole batch, never to delete a single result — so a news article or an NGO can never fall on its own. Context in platform/devops/WEB-SEARCH-TOOL.md §9.10.

Residual risk, already documented

After five review rounds pushing the host blocklist toward precision (eleven real collisions were caught mid-review, from a town hospital to a child-protection NGO), the dominant residual risk is no longer the poisoned engine — it is an unknown adult host that no exact entry, TLD or unambiguous substring covers. In an off-query batch the poisoned-batch guard catches it; alone in a relevant batch it reaches the caller.

[sre-helmcode]

Measured post-deploy: the fallback-returned-zero signal is not theoretical

Verifying the fix on 2026-08-19 I ran ~20 queries against the live SearXNG in a couple of minutes and took all three carriers down at once:

duckduckgo → CAPTCHA                  (suspended_time=0, retried per request)
yep        → HTTP 403 access denied   (suspended_time=180)
google cse → unusual traffic detected (suspended_time=180)

Result: n=0 for every search over roughly three minutes. It recovers on its own — after waiting out the 180s window, yep came back with 20 relevant results — so this is transient and self-inflicted by the test burst, not a standing outage.

But it measures the real posture after disabling bing:

• The scraped path now rests on duckduckgo + yep. mojeek returns 0 from our egress IPs and google cse is rate-limited most of the time, so those two are the whole capacity.
• Both are scrapers, so both can be blocked at the same time by the same burst.
• With Brave answering 402 there is no primary to absorb it, which is item 1 of this issue.

The failure mode is the good one — empty rather than wrong, and the chat tool now tells the model to rephrase rather than asserting nothing exists on the topic — but a customer sees a 200 with an empty results array, or a 503.

Current traffic is nowhere near this (community tier is 20 rpm / 500 daily, and the platform logged 4 searches in a day), so nothing is being hit today. It does mean the fallback-returned-zero alert has a concrete trigger to be built against, and that restoring a primary — Brave or otherwise — is what actually removes the single point of failure.