feat(web-search): add WebSearchTask with pluggable search providers - #850
Open
sroussey wants to merge 16 commits into
Open
feat(web-search): add WebSearchTask with pluggable search providers#850sroussey wants to merge 16 commits into
sroussey wants to merge 16 commits into
Conversation
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01W4wM782LCRMPFPKeGzrzu9
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01W4wM782LCRMPFPKeGzrzu9
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01W4wM782LCRMPFPKeGzrzu9
…election Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01W4wM782LCRMPFPKeGzrzu9
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01W4wM782LCRMPFPKeGzrzu9
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01W4wM782LCRMPFPKeGzrzu9
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01W4wM782LCRMPFPKeGzrzu9
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01W4wM782LCRMPFPKeGzrzu9
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01W4wM782LCRMPFPKeGzrzu9
Adds the package to the dependency graph and key-packages list, and corrects the vendor-subpath paragraph: providers/* is no longer only ./ai and ./ai-runtime now that anthropic ships ./web-search. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01W4wM782LCRMPFPKeGzrzu9
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Pull request overview
Introduces a new @workglow/web-search package that standardizes web search behind a provider interface with capability-based routing, and adds an Anthropic grounded web-search provider entrypoint in @workglow/anthropic.
Changes:
- Adds
@workglow/web-searchwithWebSearchTask, provider registry, capability checks, and built-in HTTP providers (Brave, Tavily, SearXNG). - Adds
@workglow/anthropic/web-searchentrypoint implementing theIWebSearchProvidercontract. - Adds tests and wiring (exports/build scripts/tsconfig references) to support the new package and provider.
Reviewed changes
Copilot reviewed 30 out of 31 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| providers/anthropic/tsconfig.json | Adds TS project reference to the new packages/web-search. |
| providers/anthropic/src/web-search/AnthropicWebSearchProvider.ts | Implements Anthropic grounded web-search provider via vendor SDK. |
| providers/anthropic/src/web-search.ts | Adds ./web-search entrypoint and registration helper for the Anthropic provider. |
| providers/anthropic/src/tests/AnthropicWebSearchProvider.test.ts | Unit tests for Anthropic web-search provider behavior (tool type, mapping, pause_turn, domain lists). |
| providers/anthropic/package.json | Exports ./web-search, builds it, and adds dependency on @workglow/web-search. |
| packages/web-search/tsconfig.json | Adds composite TS config for the new @workglow/web-search package. |
| packages/web-search/src/WebSearchTask.ts | Implements the WebSearchTask with provider routing, capability enforcement, and request adaptation. |
| packages/web-search/src/WebSearchProviderRegistry.ts | Adds a process-wide provider registry with routing and helpful error messages. |
| packages/web-search/src/queryOperators.ts | Implements site: query rewriting for providers that only support domain filtering via query operators. |
| packages/web-search/src/providers/TavilyWebSearchProvider.ts | Adds Tavily HTTP adapter with capability declaration and request/response mapping. |
| packages/web-search/src/providers/SearxngWebSearchProvider.ts | Adds SearXNG adapter with base URL validation and result normalization. |
| packages/web-search/src/providers/httpSearch.ts | Shared helper to execute provider HTTP calls via an owned FetchUrlTask. |
| packages/web-search/src/providers/BraveWebSearchProvider.ts | Adds Brave HTTP adapter including date-range mapping to freshness. |
| packages/web-search/src/node.ts | Node entrypoint that registers the task and built-in providers on import. |
| packages/web-search/src/IWebSearchProvider.ts | Defines provider interface, request/response types, and capability model. |
| packages/web-search/src/common.ts | Common exports plus registration helpers for task and built-in providers. |
| packages/web-search/src/capabilityCheck.ts | Implements capability/request gap detection used for routing and pinned-provider validation. |
| packages/web-search/src/browser.ts | Browser entrypoint that registers only the task (no providers). |
| packages/web-search/src/tests/WebSearchTask.test.ts | Tests routing behavior, pinned-provider validation, query rewriting, and maxResults clamping. |
| packages/web-search/src/tests/WebSearchProviderRegistry.test.ts | Tests registry semantics (register/get/route/require/clear) and error messages. |
| packages/web-search/src/tests/TavilyWebSearchProvider.test.ts | Tests Tavily adapter mapping, auth scheme, and option behavior. |
| packages/web-search/src/tests/SearxngWebSearchProvider.test.ts | Tests SearXNG adapter URL construction, credential suppression, normalization, and truncation. |
| packages/web-search/src/tests/SearxngWebSearchProvider.integration.test.ts | Optional live integration test for SearXNG behind env var gating. |
| packages/web-search/src/tests/queryOperators.test.ts | Tests site: query operator rewriting and normalization. |
| packages/web-search/src/tests/entries.test.ts | Tests that entrypoints register tasks/providers as intended. |
| packages/web-search/src/tests/capabilityCheck.test.ts | Tests capability gap detection across request options. |
| packages/web-search/src/tests/BraveWebSearchProvider.test.ts | Tests Brave adapter mapping, auth header scheme, and freshness date mapping. |
| packages/web-search/README.md | Documents server-only behavior, provider matrix, and usage patterns. |
| packages/web-search/package.json | Defines new package exports/build/test scripts and peer deps. |
| bun.lock | Adds workspace entry and dependency wiring for @workglow/web-search. |
| .claude/CLAUDE.md | Updates repo architecture and adds @workglow/web-search documentation section. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
CodeQL flagged two high-severity polynomial-ReDoS alerts. `/\/+$/` is quadratic on a string holding a long run of slashes that is not at the end: the engine starts `\/+` at every slash position, consumes the whole run, fails `$`, and restarts one character along. Both call sites take untrusted text — a search domain from task input, and a configured base URL. Measured on a 200k-slash input: 49.7s before, under 1ms after. The regression test asserts a 1000ms budget, so only a reintroduction of the pattern can trip it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01W4wM782LCRMPFPKeGzrzu9
sroussey
changed the title
Two review findings. Owning a FetchUrlTask does not inherit the job queue's rate limiter. The task refuses credential_key on the queued path because a queued payload is persisted to durable storage, so every keyed provider runs inline and inline fetches are not rate limited. The JSDoc and CLAUDE.md said otherwise; bounding a fan-out is the caller's job and now says so. The Anthropic provider returned an answer whether or not the caller asked, while Tavily returns one only on request — so `answer` meant different things depending on which provider routing picked. It is now gated on includeAnswer in both. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01W4wM782LCRMPFPKeGzrzu9
Both tsconfigs compiled src/**/__tests__ into dist, and both packages publish files: ["dist"], so the new tests shipped as .d.ts to consumers. Adds the exclude packages/task-graph already uses for its co-located tests. Vitest collects via its own project config, so the tests still run. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01W4wM782LCRMPFPKeGzrzu9
The new web-search test made providers/anthropic a workspace that holds tests, and it had no `test` script. Turbo reports such a workspace's test task successful while running nothing, so the suite would have been skipped in CI while reporting green — which is what testDiscovery's "every workspace that holds tests has a test script" guard caught. Adds the documented per-package script. `--project anthropic` now collects and passes the 9 tests it was silently skipping. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01W4wM782LCRMPFPKeGzrzu9
Three new ./web-search subpaths, each implementing IWebSearchProvider against its vendor SDK, so @workglow/web-search still depends on nothing AI-related. Adds an optional excludeDomainFilter capability, defaulting to domainFilter. OpenAI's web_search takes filters.allowed_domains and the installed SDK models no blocked equivalent, so a single field would force that provider to under-declare (losing working include filtering) or over-declare. Over-declaring is the failure the record exists to prevent: "auto" would route an excludeDomains request to a provider that cannot honor it, and the adapter would throw after selection instead of the task refusing before it. The grounded providers are not variations on one shape. Anthropic filters by domain, not date. Gemini is the mirror image — the SDK documents excludeDomains as Vertex-only and timeRangeFilter as not supported on Vertex — so on the Gemini API path it is date-only, and it fills the open end of a half-open range because the API rejects a one-sided interval. OpenAI is inclusion-only. OpenRouter does both. Each capability was read from the installed SDK typings rather than the vendor docs, which disagree. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01W4wM782LCRMPFPKeGzrzu9
…osed The coverage report flagged the instance entitlements() as uncovered, and writing the test showed the branch was dead: an enforcer asks before the run, and runInputData is populated from defaults only once resetInputData() has run, so every instance returned the static set. Correcting it also corrected the comment. fetchUrlEntitlementsFor does not scope network:http; it decides whether network:private is required — absent for a public origin, scoped to the host for a private one, and unscoped when the destination is unknown. That matters because SearXNG is self-hosted and routinely sits on a private address. So "auto" now fails closed rather than returning the bare base: routing happens at run time and may land on a privately-hosted instance, which the base would not have required a grant for. A pinned public provider is correspondingly narrower than before, and an SDK-backed provider needs no private access at all. Also covers the two other gaps the report named: the no-JSON-body throw and Brave's open-ended freshness range. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01W4wM782LCRMPFPKeGzrzu9
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds
@workglow/web-search— oneWebSearchTaskserving both plain search APIs and model-grounded search behind a single normalized output shape, routed by provider capability. Seven providers ship.Design spec and implementation plan live in the
prdrepo on the same branch name (docs/superpowers/specs/2026-08-22-web-search-task-design.md).Why one task for both families
The obvious split — "search APIs" vs "grounded search" — isn't real. Tavily returns a synthesized
answerwhen asked; Exa returns full pagecontent; Anthropic'sweb_searchreturns an answer plus citations that are structurally{title, url, snippet}. So this is one shape whose fields different providers can or cannot populate — a capability matrix, not a discriminated union.resultsis always present (for a grounded provider, those are its citations).answeris populated only when the caller asks for one, uniformly across providers.Server-side only
Every commercial search API authenticates with a request header, which forces a CORS preflight none of them answer. Google's Custom Search JSON API is the one exception that genuinely permits cross-origin XHR — and it is closed to new customers and sunsets 2027-01-01. Independently, a browser-executed search would put the API key where any visitor can read it.
The browser entry therefore registers the task (so a builder UI can render the node and validate a graph) and no providers.
Capability routing
provideris required with no default, mirroringresponse_typeonFetchUrlTask— which provider serves a request decides its cost, rate limit and quality."auto"opts into routing over the requested options; the provider that ran is always reported on theprovideroutput port. A pinned provider that cannot honor an option throws rather than silently rerouting.domainFilteris three-valued:"native","query-operator"(the task rewrites the query withsite:), orfalse. Date filtering is never emulated — post-filtering bypublishedDatebreaksmaxResultsand drops every result whose date the provider omitted, sodateFilter: falsemeans such a request is refused.excludeDomainFilteris optional and defaults todomainFilter. It exists because OpenAI'sweb_searchtakesfilters.allowed_domainsand the installed SDK models no blocked equivalent. Folded into one field, that provider must either under-declare (losing working include filtering) or over-declare — and over-declaring is the failure this record exists to prevent:"auto"would route anexcludeDomainsrequest to a provider that cannot honor it, and the adapter would throw after selection instead of the task refusing before it.Providers
braveX-Subscription-Tokensite:tavilysearxngsite:anthropicopenaiopenroutergeminiThe grounded four are not variations on one shape. Gemini is the exact mirror of Anthropic, and the SDK comments say why:
GoogleSearch.excludeDomainsis "not supported in Gemini API" (Vertex only), whiletimeRangeFilteris "not supported in Vertex AI". Every capability here was read from the installed SDK typings, not the vendor docs — they disagree, and the typings are what the code compiles against.HTTP adapters (Brave, Tavily, SearXNG) execute by owning a
FetchUrlTask, inheriting credential resolution viacredential_key, SafeFetch's redirect/SSRF checks, retry/backoff, per-attempt timeouts and the response cache.They do not inherit the queue's rate limiter, and cannot:
FetchUrlTaskrefusescredential_keyon the queued path because a queued payload is persisted to durable storage, so every keyed provider runs inline. Bounding aMapTaskfan-out remains the caller's responsibility. (This corrects an earlier overclaim in this PR, caught in review.)The grounded adapters ship as
./web-searchsubpaths on their vendor packages, so@workglow/web-searchhas no dependency on@workglow/ai. They register explicitly — importing the subpath registers nothing, matching how./aiexportsregisterAnthropic*rather than self-registering.Entitlements
The instance declaration is narrowed to the pinned provider's own destination.
fetchUrlEntitlementsFordoes not scopenetwork:http— it decides whethernetwork:privateis required: absent for a public origin, scoped to the host for a private one, unscoped (fail-closed) when the destination is unknown.That matters because SearXNG is self-hosted and routinely sits on a private address. So
"auto"fails closed — routing happens at run time and may land on a private instance — while a pinned public provider is correspondingly narrower than the base, and an SDK-backed provider needs no private access at all.Vendor traps handled and tested
web_search_tool_resultcarries a list on success and an error object on failure, at HTTP 200 — reading it unbranched records a quota failure as a search that found nothing.pause_turn, which must be resumed or the answer truncates silently.dateRangehas its open end filled rather than being refused.Tests
114 tests. SearXNG needs no key and has no quota, so it is the only provider whose integration test can run unmocked (
.integration.test.ts, excluded from the default tier, skipped unlessWEB_SEARCH_SEARXNG_URLis set). The rest are fixture-driven.Not in scope
No streaming (a grounded
answercould stream, butresultscannot arrive incrementally usefully). No re-ranking, deduplication across providers, or multi-provider fan-out — one query, one provider, one response.