Add web console UI for sec CLI: annotations, panels, pickers, and status widgets - #324
Merged
Merged
Conversation
The console rendered sec's ~90 commands as bare text boxes and its output as raw stdout JSON. Nothing here changes a command; it says what sec's commands already mean in terms the page can render. Pickers. Every identifier nobody remembers now has a search: filers by name (falling back to the CIK index only when the entity is not ingested — which is exactly when you are composing `fetch submissions`), known SPACs, screen candidates, accessions and form types scoped to the CIK already chosen, extractor ids carrying the version a retry would run under and the depth of their worklist, resolver kinds, canonical family names, and a version ceremony's component id narrowed by the kind beside it. The CIK picker is also registered under `format: "cik"`, which `TypeSecCik` stamps on every sec task schema, so `sec-base task run` and the whole generic surface get it for free. Panels. A SPAC report renders as headline figures, its attempts as a table and its event stream as a timeline — the order being the thing a two-column table makes you reconstruct. The dead-letter worklist is coloured by what an operator can do about it: amber where `retry-dead-letters` clears it now, red where the extractor has to be fixed and bumped. Version slots flag an open dev cycle that is not yet covered. One generic panel renders any `query` command's rows, so a query added later is covered without registering anything. Status rail. The EDGAR budget and whether the cluster is in a cooldown, the configured backend and its counts, pending dead letters by extractor, and open dev cycles. The throttle line reports "per-process (sqlite)" rather than "clear" where the sentinel is in-memory: the console runs commands as child processes, so under SQLite the server has no window into a cooldown a run is sitting in, and claiming otherwise would be a claim about a process it cannot see. `readSecFetchPauseUntil` is the reporting read that backs it. Badges. What a command costs — model quota, EDGAR's rate limit, an hour, a write, a deletion — shown in the tree before you click into it, with a typed confirmation on the three ceremonies whose damage outlives the run. Two bugs the guards caught, both the same shape. `--format` is declared six different ways across this CLI, so a single stated vocabulary offered `csv` where the command rejects it and omitted `text` where `spac report` defaults to it; each command's vocabulary is now read from its own help text and declines rather than guessing. And `spac download --confidence` takes a CSV list, so a single-select could not express its `high,medium` default. A test now walks the whole tree and fails any annotated field whose choices omit the command's own default; another asserts every annotation path matches a real command, since a mistyped path does not error — the picker simply never appears. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017z1z67aKaCxnM1bQ8FkfUN
…formatting Two seams the embarc-data side needed, both narrow. `registerFormatChoiceAnnotations` is the `--format` inference split out and exported. It reads the PROGRAM, so it can only cover commands registered before it runs — and a superset adds its groups after `AddCommands` returns, which left every one of its `--format` flags as the only bare text boxes on the page. Re-running is safe: annotations are keyed by path, so a second pass replaces rather than duplicates. The panel formatting helpers are exported for the same reason a superset should not re-derive them: a report that renders `—` for one absence and `null` for another is a report nobody trusts to mean anything by either. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017z1z67aKaCxnM1bQ8FkfUN
Contributor
Author
CI
|
Contributor
There was a problem hiding this comment.
Pull request overview
This PR adds @workglow/sec integrations for the Workglow web console so operators get richer UI affordances (command annotations, input pickers, output panels, and a status rail) while keeping command execution behavior unchanged.
Changes:
- Registered web UI metadata and widgets for sec commands (field annotations, pickers, status widgets, output panels) via a single
registerSecWebUi()entry point. - Added format-vocabulary inference for
--formatbased on Commander help text, with tests to ensure dropdown choices match each command’s true accepted/default values. - Added a small memoization layer for web-triggered DB reads and a status widget integration that reports cluster-wide fetch throttling under Postgres.
Reviewed changes
Copilot reviewed 14 out of 14 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| src/web/secWebReads.ts | Adds a short-lived memoization helper for web-console-triggered reads (e.g., dead-letter counts). |
| src/web/secStatusWidgets.ts | Implements status-rail widgets for fetch throttling, DB status, dead letters, and version slots. |
| src/web/secPanels.ts | Adds web panels for rendering sec command outputs (SPAC report, dead letters, version status, query/eval tables). |
| src/web/secPanelFormat.ts | Shared formatting helpers for panels (money/count/text/json list and record-table rendering). |
| src/web/secFieldWidgets.ts | Adds autocomplete pickers for common sec identifiers (CIK, accession, form, extractor, resolver, families, SPAC candidates). |
| src/web/secAnnotations.ts | Declares command/field annotations (badges, confirmations, field formats, and vocabularies) + --format choice inference wiring. |
| src/web/registerSecWebUi.ts | Single entry point that registers all sec web console contributions. |
| src/web/registerSecWebUi.test.ts | Tests that annotation path patterns match real commands and that inferred --format choices are correct. |
| src/web/formatChoices.ts | Parses Commander option descriptions to infer valid --format choices per command. |
| src/web/formatChoices.test.ts | Unit tests for inferFormatChoices() across the CLI’s observed --format declaration styles. |
| src/task/fetch/secFetchThrottle.ts | Adds a reporting read (readSecFetchPauseUntil) for cluster pause state (used by status widgets). |
| src/libs-cli.ts | Registers sec’s web UI contributions in sec-base so pickers/status are available on the generic surface too. |
| src/index.ts | Exports registerSecWebUi, registerFormatChoiceAnnotations, and shared panel formatting utilities for supersets. |
| src/commands/index.ts | Calls registerSecWebUi(program) during command registration so the web console can discover annotations/panels/widgets. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Comment on lines
+191
to
+193
| return EXTRACTOR_IDS.filter((id) => !needle || id.toLowerCase().includes(needle)) | ||
| .slice(0, MAX_ITEMS) | ||
| .map((id) => ({ value: id, label: id, detail: "parsed form family" })); |
Comment on lines
+195
to
+199
| rowTones: pending.map((entry) => | ||
| SAME_VERSION_RETRYABLE.has(String(entry.reason_code)) ? "warn" : "fail" | ||
| ), | ||
| note: "Amber: retryable under the same version — run `extractor retry-dead-letters`. Red: version-gated — fix the extractor and bump it.", | ||
| }; |
Comment on lines
+60
to
+67
| const repo = new ExtractionDeadLetterRepo(); | ||
| const counts = new Map<string, number>(); | ||
| await Promise.all( | ||
| EXTRACTOR_IDS.map(async (id) => { | ||
| const pending = await repo.listPending(id).catch(() => []); | ||
| if (pending.length > 0) counts.set(id, pending.length); | ||
| }) | ||
| ); |
…canning read Three findings from the review bot on #324, all verified against the code before acting. `searchForms`' fallback offered `EXTRACTOR_IDS`, which is a different vocabulary that merely overlaps the one the command takes: `S-1` is both a form and an extractor id, but `merger-proxy`, `redemption`, `loi` and `25-15` name extractors and no form at all, so picking one produced a value `sec fetch form <cik> [form]` can never match. It offers the keys of `FORM_TO_EXTRACTOR_ID` — the form symbols — instead. The dead-letter panel claimed an eligibility verdict it cannot make. Retry eligibility depends on whether the extractor version has advanced past each row's `failed_extractor_version`, and the panel is built from the command's `pending` output, which carries no current versions — so a red row whose extractor has since been bumped is already eligible. The tone is now stated as what it is, a reading of the reason code, and the note points at `dead-letters --eligible` for the question this view cannot answer. The same paragraph records why `attempts` is rendered: `MIXED_CAPTION_SHAPE` is same-version retryable only for a bounded number of attempts. `listPending` read every row for an extractor and filtered the status in memory. That was tolerable when its only caller was a command run by hand; the status rail re-reads it for every extractor id on a timer, and an extractor's resolved history grows without bound and is the majority of its rows. The status is pushed down instead — the sibling `listPendingByAccessions` two lines below already proved the storage supports it. Same rows, and the existing `extractor dead-letters` caller gets the same improvement. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017z1z67aKaCxnM1bQ8FkfUN
### Features #### web - let a superset re-run the format pass and share the panel formatting - contribute sec's pickers, panels, status rail and cost badges ### Bug Fixes #### web - address review — form vocabulary, dead-letter tone, and a scanning read ### Chores - update deps to pull new api - remove protobufjs ### Updated Dependencies - `@workglow/cli`: 0.4.1 - `typebox`: 1.3.18 - `workglow`: 0.4.1
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR adds comprehensive web console integration for the
@workglow/secCLI, enabling the Workglow web UI to render command annotations, field pickers, output panels, and status widgets. The implementation provides a rich interactive experience for operators without modifying any CLI runtime behavior.Key Changes
Command and field annotations (
secAnnotations.ts): Declarative metadata for 40+ command paths describing costs (network, AI, slow, destructive), confirmation dialogs, field formats, and vocabularies. Annotations are matched against the real command tree at test time to catch silent failures from renamed commands or typos.Field widgets and pickers (
secFieldWidgets.ts): Smart autocomplete pickers for CIKs (searchable by name or number), accessions (scoped to a selected filer), forms, extractors (showing version and worklist depth), resolver kinds, and SPAC candidates. All pickers read only stored data—never fetch from EDGAR—to respect the shared rate limit.Output panels (
secPanels.ts): Specialized renderers for command output including SPAC timelines, deal summaries, dead-letter worklists (color-coded by retry eligibility), version status, candidate screening results, and generic query result tables. Panels render what the command already returned, keeping them honest.Status widgets (
secStatusWidgets.ts): Real-time dashboard showing database backend, entity/filing counts, EDGAR fetch throttle state (including cluster-wide pause detection under Postgres), and dead-letter worklist by extractor.Panel formatting utilities (
secPanelFormat.ts): Shared helpers for rendering money (with scale suffixes like $1.5B), counts, text, JSON arrays, and auto-columned tables from record arrays.Format choice inference (
formatChoices.ts): Parses--formatoption descriptions to extract valid choices (e.g., "table, json, csv") and validates against the command's declared default, ensuring dropdowns never offer values the CLI would reject.Web reads cache (
secWebReads.ts): 10-second memoization layer for database queries triggered by pickers and status widgets, deduplicating reads across rapid keystrokes and status rail refreshes.Test coverage (
registerSecWebUi.test.ts,formatChoices.test.ts): Validates that all annotation paths match real commands and that format choice parsing handles sec's six different declaration styles.Integration (
registerSecWebUi.ts): Single entry point that registers all annotations, widgets, and panels. Called duringAddCommandsso it runs before runtime bootstrap and on every invocation including those with no database.Exports and wiring: Added
registerSecWebUiexport tosrc/index.tsand integrated intolibs-cli.tsso the web console can compose sec's UI alongside other packages.Notable Implementation Details
Path matching is intentionally coarse: Annotations use glob patterns (e.g.,
["fetch", "**"]) to avoid duplication, and the test suite asserts every pattern reaches at least one real command—catching silent failures that would otherwise go unnoticed.Scoped pickers: The accession picker reads the
--cikflag to offer only that filer's filings; the form picker falls back to sec's parsed form families when a CIK has no filings yet.Dead-letter coloring: Entries are marked amber (retryable under current version) or red (version-gated) based on reason codes, guiding operators toward the right fix.
SPAC timeline sorting: Events are sorted by date before rendering to reconstruct the story correctly, since database order may not reflect chronology.
No network in pickers: All autocomplete reads only stored data, respecting the shared EDGAR rate limit and ensuring the console remains responsive.
https://claude.ai/code/session_017z1z67aKaCxnM1bQ8FkfUN