Clean and dedupe ticker symbols in offerings and SPAC reports - #326
Merged
Conversation
A completed combination is otherwise unvaluable: the market never priced the target because it was private, and its book equity is a private company's accounting rather than what was paid for it. The announced equity and enterprise value are the only stated numbers that answer "what was this worth at the combination", which is what an `acquired` valuation basis needs. They ride the path `target_name` and `pipe_amount` already take — extracted from the proxy, stored per accession, correlated onto the matching `spac_deal` by filing-date window — so a definitive proxy supersedes a preliminary one with no new supersession rule. The part that needed care is units. A prospectus says "$1.4 billion" and a model can answer 1.4, or 1400. Both validate against the schema, both store, and both become a valuation off by a factor of a million; nothing downstream re-derives it, and a percentage change computed against one is merely very large rather than obviously wrong. So the unit is stated at the point the number is produced (the prompt now says whole dollars, with the worked example), and a figure below $10,000,000 is dropped at extraction. The floor separates two populations with nothing near it — a real combination is tens of millions at minimum, since the trust alone is, while a scaled figure is single or quadruple digits. Dropping rather than rescaling: a guess at the intended magnitude is a second model of the filing, and a wrong guess is indistinguishable from a right one once stored. A null says what is true — the proxy stated a value and the figure read back could not be used. Both fields are optional on the model schema, like `target_description`, so a replay under an older extractor version still validates. Adding them is a minor version bump; the ceremony is in CLAUDE.md.
A prospectus states several lock-ups, not one: the underwriters' on the
whole float, the sponsor's on its founder shares, often a longer one on the
private-placement warrants. They carry different durations, different
anchors and different price tests, so `spac_lockup_terms` is one row per
restricted class rather than one per filing — folding them together would
state a release date that applies to none of them.
Every row is what the filing says. A `duration_days` is meaningless without
its `anchor_event` (a founder lock-up runs from the closing of the
combination, an underwriter lock-up from the pricing of the offering), and
the price test is stored as an evaluable condition rather than a sentence:
at or above `price_trigger` on `trigger_days_at_or_above` sessions within
any `trigger_window_days`, no earlier than `trigger_start_delay_days` after
the anchor. Turning either into a date needs a price series this extractor
does not have, so that step lives downstream — which is what keeps this one
from ever emitting a computed-looking release date it did not compute.
Two failure modes the prompt names outright, because a model will otherwise
produce both. A duration and a price trigger are ALTERNATIVES on one
lock-up ("one year, or earlier if the shares close at or above $12.00 …"),
and splitting them invents a restriction the filing does not impose. And
the customary founder lock-up is standard enough that a model will supply
it for a prospectus stating only an underwriter lock-up — which is what the
second golden fixture measures.
`holder_class` is constrained by the schema rather than by the prompt, and
persist filters against the same vocabulary: a lock-up filed under a class
nothing downstream knows is a restriction nobody will ever evaluate.
The section is the Item 12 "Shares Eligible for Future Sale" block, falling
back to Underwriting. Both are needed on measurement: of the 42 committed
S-1 fixtures, 14 carry the Item 12 heading and 32 disclose a lock-up
somewhere. Unlike sponsor-promote it is never skipped for a non-SPAC
filing, since every registrant locks somebody up.
Adding the section is a minor bump on the S-1 extractor; the ceremony is in
CLAUDE.md.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NWpHfBhCHs6R8MH55pdWzq
- Introduced `isOverlongPersonName` utility to filter out names exceeding 150 characters, ensuring compliance with display name constraints. - Updated `extractManagement`, `extractBeneficialOwnership`, and `extractRelatedParty` functions to drop overlong names, improving data quality. - Added tests for new name handling logic and updated existing tests to validate changes. - Implemented `joinedPersonName` function for consistent name concatenation in related person fields.
Strip wrappers, placeholders, and delimited exchange prefixes so persist and quote jobs share one cleaned symbol.
Strip wrappers, placeholders, and delimited exchange prefixes before IPO, offering, and submission ticker writes so quote jobs never see junk symbols.
…rd-redesign-rrj5tp # Conflicts: # CLAUDE.md # src/sec/forms/registration-statements/s1/DocumentSegmenter.ts
…hub.com/workglow-dev/sec into claude/embarc-dashboard-redesign-rrj5tp
`format-check` is the first step of sec's CI, and these three arrived on the branch unformatted — from `dd8162d` (person name bounds) and `d77de39` (listed ticker cleaning), not from the merge. Prettier's version is pinned exactly here precisely so this is decidable rather than a matter of whose install ran last.
- Introduced `form-d` as a new command for processing Form D family filings. - Updated the `sync` command options to include `form-d` and its shard processing. - Enhanced the `SYNC_FORM_DOMAINS` to ensure `form-d` is included in the `sync all` command. - Added tests to verify the integration of `form-d` into the CLI and its functionality. - Implemented `expandFormTypes` to handle extractor IDs and their corresponding forms.
- Removed `form-d` from the CLI integration tests and sync leaves registration. - Introduced a new command structure for standalone Form D processing with `--simple` option. - Updated tests to reflect changes in the registration of sync leaves, ensuring `form-d` is excluded from `inAll` leaves. - Enhanced the `SyncRunContext` to accommodate new options for isolated steps and simple processing.
- Changed the default value of `SecFetchMaxConcurrent` from 16 to 8 to better align with the fetch rate limits. - Updated related calculations in `SecFetchJob.test.ts` to utilize the new concurrency constant. - Modified `DRAIN_WINDOW_MS` to dynamically calculate based on `SecFetchMaxConcurrent` and `SecFetchMaxPerSec`. - Ensured that the `ConcurrencyLimiter` in `SecJobQueue.ts` uses the updated concurrency constant for improved rate limiting.
The table carried an index on `accession_number` only — the "which companies did this filing name" direction. The opposite read, "what has this company been called", is what every canonical-name fallback issues, one company at a time, and it had no index at all: a single-CIK lookup was a sequential scan of one row per company mention in every filing. Measured from a consumer resolving a missing SPAC name: 279 ms, 188 ms and 97 ms for queries returning ONE row. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NWpHfBhCHs6R8MH55pdWzq
`LISTED_FORM` rejected EDGAR's own hyphenated class symbols. The submissions API states a multi-class filer's tickers as `BRK-A` / `HEI-A` / `LEN-B`, and a dot-only, letters-only pattern matched none of them — while every caller reads a rejection as "not a ticker" and drops the row, so class shares stopped being stored in `entity_tickers`, `issuer_ticker` and `spac.spac_tickers` where the raw symbol used to be kept exact. The root now admits digits and the suffix separator is `.` or `-`, with an open class vocabulary: a share class is not a warrant, and enumerating only the four unit-split codes reads every other class as junk. The `/^_+$/` guard went with it — no character class in the pattern contains an underscore, so it could never be reached. `offeringSections` aliased a ReadonlyArray and then pushed to it, so the branch did not type-check at all and the package could not be built. The S-1 management persist loop drops an overlong name AFTER the section runner has computed `meta.complete`, so roster closure ran on a subset it thought was whole and wrote a departure for every open tenure the dropped row still asserted. Closure now requires that this loop dropped nothing, and the return value counts what was persisted rather than what was offered. `sec-base` ended on an unconditional `process.exit(0)` with no `finally`, so a failed command reported success to any script branching on the status, and a thrown one skipped `stopQueues()` and left the fetch workers holding the process open. It now mirrors `sec.ts`: `process.exitCode`, cleanup in a `finally`, rethrow. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NWpHfBhCHs6R8MH55pdWzq
`SPEC.md` still listed a `form-d` sync leaf, in the leaf table, in the `sync all` order and in the `--shard` list — removed from the code in a2997f7. `CLAUDE.md` still gave `SEC_FETCH_MAX_CONCURRENT` a default of 16 and explained the pairing in terms of it; cc85613 set it to 8, which matches `SEC_FETCH_MAX_PER_SEC` and is what makes a synchronized retry of every in-flight job unable to exceed the start cap. Also runs prettier over `SPEC.md` and `src/index.ts`, which were the two files failing `format-check` — table padding and one over-width re-export line. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NWpHfBhCHs6R8MH55pdWzq
Conflict in `src/libs-cli.ts`, resolved keeping both sides: main's `registerSecWebUi()` registration inside `registerTasks`, and this branch's try/catch/finally shutdown (`process.exitCode` + cleanup in a `finally` + rethrow) around it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NWpHfBhCHs6R8MH55pdWzq
- Reduced the default fetch rate limit from 8 requests/second to 4 to align with EDGAR's constraints. - Updated the `SecFetchMaxPerSec` and `SecFetchMaxConcurrent` constants to reflect the new limits. - Modified related calculations in tests and job implementations to ensure consistency with the new rate limits. - Enhanced retry logic to utilize `RetryableJobError` for better error handling during fetch attempts. - Adjusted test cases to validate the new concurrency and rate limiting behavior.
`prettier --check .` is the first step of CI and was failing on a double space left in the cluster-cooldown paragraph. Whitespace only. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NWpHfBhCHs6R8MH55pdWzq
…rd-redesign-rrj5tp
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Add ticker symbol normalization and deduplication across offering terms, issuer ticker history, and SPAC reports. Introduces utilities to clean listed ticker symbols by stripping exchange prefixes, parentheses, and placeholder values, then dedupes cleaned symbols to avoid redundant storage.
Key Changes
New ticker normalization utility (
src/util/listedTicker.ts):normalizeListedTicker()strips exchange prefixes (NASDAQ:, NYSE_, etc.), parentheses, and placeholder values (N/A, NONE, etc.)cleanListedTickers()maps and dedupes cleaned symbols, keeping first-seen unique valuesNew person name bounds utility (
src/util/personNameBounds.ts):MAX_PERSON_NAME_CHARSconstant (150 chars) matching leader slug column capacityisOverlongPersonName()validates names don't overflow the columnjoinedPersonName()concatenates name parts the way leader slugs are builtOffering terms ticker cleaning (
src/sec/forms/registration-statements/s1/offeringSections.ts):offering_terms.tickerSection extractors filtering (
src/sec/forms/registration-statements/s1/sectionExtractors.ts):SPAC report ticker cleaning (
src/storage/spac/SpacReportWriter.ts):spac_tickersarray before storageForm D person name validation (
src/sec/forms/exempt-offerings/Form_D.storage.ts):Test coverage:
Notable Implementation Details
https://claude.ai/code/session_01EBmxeduePb4ZhekGjJ7jV2