Release readiness for v0.1.0: installable, portable, tested, documented - #47
Open
RobbinBouwmeester wants to merge 57 commits into
Open
Release readiness for v0.1.0: installable, portable, tested, documented#47RobbinBouwmeester wants to merge 57 commits into
RobbinBouwmeester wants to merge 57 commits into
Conversation
…ed configs parse The Strasbourg prenylation work is unpublished, so its configs and deployment notes must not reach the remote. `config.strasbourg-linux.json` had been committed and pushed; untrack it (the local file is kept for transfer by other means) and add ignore rules for `config.strasbourg*.json` and `config.*-local.json` so it cannot recur through a `git add -A`. Root-level notes were already covered by the existing `/*.md` rule. NOTE: untracking removes the file from the branch tip but NOT from history. If it must be unreachable, the branch needs a history rewrite and force-push. Its content is search parameters and server paths -- no sample, protein or result data -- so that is a judgement call for the repo owner. Also adds `shipped_configs_parse`, which round-trips every TRACKED config through `Config::from_json`. A `_comment` key shipped in a config once and, because `Config` is `deny_unknown_fields`, made the whole config unloadable -- caught only by running `doctor` on the deployment target. The workspace suite passed throughout, which is the real gap: a config is an artifact the engine must accept, not just valid JSON. Untracked machine-specific configs are deliberately excluded from the test and remain `doctor`'s job. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…cceeds on Windows
`deeplc_worker.py` imported numpy and pyarrow at module level and deferred `import deeplc`
into main(). DeepLC 4.x is torch-backed, and on Windows that ordering makes torch's DLL
initialisation fail outright:
OSError: [WinError 1114] A dynamic link library (DLL) initialization routine failed.
Error loading "...\torch\lib\c10.dll" or one of its dependencies.
`deeplc_finetune.py` already ordered its imports this way and says why ("import before numpy,
OpenMP load order"); this worker did not. The bug stayed latent because imported-library mode
skips predict-frag entirely, so nothing exercised the native RT-prediction path. It surfaced
the first time a library was built from FASTA, which failed after the peptidoform table was
already generated.
The comment states that the ordering is load-bearing, so a future tidy-up does not sort the
imports and silently reintroduce it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`doctor` checked the DeepLC interpreter for `deeplc,numpy,pandas`, but that interpreter also runs `deeplc_finetune.py`, which imports pyarrow, torch and psm_utils. A green doctor could therefore be followed by a crash at the fine-tune step, which on an experiment-wide batch is discovered long after the run is launched. DeepLC 4.x pulls torch and psm-utils itself, so in practice this catches a missing pyarrow, but the check should assert what the scripts import rather than what the dependency tree happens to imply. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…nt.q_filter `run-experiment` hard-set `qcfg.q_filter = PsmQ`, silently discarding whatever `quant.q_filter` the config asked for. The override itself is deliberate: the grouped q columns (peptide_q_value / precursor_q / pg_q_value) are assigned only to each group's single experiment-wide winning row, so a per-run table can only gate on a per-PSM column. Doing it in silence is the problem. A user who explicitly configured a different `q_filter` got quantities gated on a column they did not choose, with no record of the substitution in any artifact. Now it warns with both the configured and the effective value. Behaviour is unchanged, so existing results stay interpretable. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…hon screen
New `mumdia prescan` stage. For a modification search it keeps only the candidates whose
modification-anchored sequence trimers are actually observed in their own isolation window
and retention-time range, so the per-run library is sized to the evidence rather than to the
enumeration. Same output contract as the Python screen it replaces (candidate_id, label), so
downstream library assembly is unchanged.
Measured on one 50-window Orbitrap DIA run against a 54.8M-row library:
tag index build 27 s -> 0.64 s (42x)
screening 412 s -> 5.6 s (74x)
total wall 439 s -> 38 s (11.6x)
Screening is independent per candidate, so it is a rayon fan-out; the Python loop was
single-threaded and was ~40% of the whole per-file chain. Most of the remaining 38 s is
reading the library, not computing.
WHAT THIS STAGE IS NOT. It cannot discriminate a true modified peptide from its decoy, and
must not be read as if it could. `anchored_tris` emits every trimer in both orientations and
a reverse decoy preserves composition and precursor m/z, so a decoy's anchored tag set is
identical to its target's and a decoy survives exactly when its target does. Measured
target:decoy survival ratio is 1.0000 (2,453,365 / 2,453,365). That is precisely why it is
safe: exchangeability is untouched, so downstream FDR stays valid. It is a compute reduction.
Both labels go through the identical criterion, each on its own sequence, m/z and RT window.
Screening only targets and then admitting their paired decoys would make surviving targets
signal-enriched while their decoys stay signal-blind, biasing the modification's q-values
anticonservatively; the stage aborts if survivors ever come back single-label.
The peak cut is deterministic (intensity descending, ties by ascending m/z). It has to be:
moving `top_peaks` by ONE peak changes the survivor set by 3.2%, which is larger than most
parameter changes, so an unspecified tie order would make reruns differ for no reason. The
0.4% symmetric difference against the Python implementation is this same cut sensitivity, not
a logic difference: the disagreeing candidates come in target/decoy pairs, and 0 of 1.48M
sampled peptidoforms have a backbone the tokeniser rejects.
Masses come from the shared model (`residue_mass` + `unimod_mass`), never a local copy; I and
L share a tag index because a residue-mass delta cannot separate isobaric residues. Output
goes through `mumdia-io`, so the snappy + arrow-utf8 library contract is automatic rather than
something an external writer has to remember.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…DE.md Nineteen docs updated, plus CLAUDE.md, which was whitelisted in .gitignore but had never been added, so the top-level guide existed only on one machine. The substantive corrections, each measured rather than argued: - `--top-peaks-ms2` is ACQUISITION-SPECIFIC and destructive at convert time. CLAUDE.md previously presented `300` as part of the validated workflow; it is correct for the chimeric AIF benchmark and actively harmful elsewhere. On a 50-window Orbitrap DIA run it discarded 78.6% of all MS2 peaks and truncated 85.5% of spectra (even p25 exceeds the cap), costing 25,425 versus 63,237 peptides.tsv rows at `peptide_q_value` <= 0.01 with the empirical decoy fraction unchanged at 0.99% in both arms. Mechanism is peak-group formation, not scoring: `presence_min_fragments` cannot be met, so real peptides are recorded NO_PEAK_GROUP. Docs now carry the peak census, the audit funnel, the cap dose-response, and a pre-flight check. - `compete.group_by = precursor` is a misnomer: it keys `base_peptide_id`, built from the STRIPPED sequence, so every charge and modform of a peptide collapses to one winner before rescore. Required, not optional, for PTM work: on a modification-rich library the default key deleted 880,464 of 1,890,239 extracted candidates (46.6%), while `peptidoform_charge` removed 0 rows and moved precursors per peptide from 1.000 to 1.174. - `cal.json` RT residuals are IN-SAMPLE and roughly 3x optimistic (6.14 s reported versus p50 17.6 s / p90 146.3 s out-of-sample). Size external RT tolerances from out-of-sample numbers. - DeepLC fine-tuning need not run per file: a once-fine-tuned library plus per-run LOESS gave 6.06 s median residual against 6.14 s, removing ~36 min per file. - An imported library may assign every modform the unmodified form's iRT (79.7% of stripped groups on one library). Check that variance before trusting RT windows in a PTM search. - Sidecar and IO contracts that fail late: deeplc import order, the widened `doctor` probe, snappy + arrow `utf8` for parquet written outside `mumdia-io`, contiguous `candidate_id` and m/z-ascending precursors as hard errors, and the `MUMDIA_NN_STREAM_GB` backend cliff. - `run-experiment` never calls report, so there is no peptides.tsv in its tree; grouped q columns exist only on each group's experiment-wide winning row, so per-run counts on them are diluted and `run_psm_q` is the correct per-file unit; and pooling more runs does not tighten q because `q = (decoys + 1) / max(1, targets)` is scale-invariant. Count tables now name their row and q-value unit, which is the convention CLAUDE.md itself requires and which the first draft of these edits did not follow. Datasets are described generically; no collaborator, sample, server path or unpublished result appears. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…1 and CLAUDE.md The stage landed in c6375b7 with no documentation, so it was reachable only by reading the source or `--help`. Adds docs/21_prescan.md and wires the references. The doc leads with what the stage is NOT, because that is the part easy to get wrong. The screen cannot separate a true modified peptide from its decoy and no tuning will make it: trimers are emitted in both orientations and a reverse decoy preserves composition and precursor m/z, so a decoy survives exactly when its target does (measured ratio 1.0000). That is what makes the stage safe rather than suspect, since exchangeability and therefore downstream FDR are untouched, but it also means a survivor count must never be read as enrichment for real modified peptides. Records why the decoy screen has to be symmetric, with the number that shows it: screening targets only and admitting their paired decoys made the apparent modification-specific FDR look like ~32% against ~43% under a symmetric screen, and that gap was the bias, not discrimination. Also records that `top_peaks` is sensitive out of proportion to its appearance: a one-peak change moved the survivor set by 3.2%, which is why the peak cut is tie-broken deterministically and why survivor counts should not be treated as precise quantities or used to tune by small differences. Distinguishes `prescan.top_peaks` from `--top-peaks-ms2`: the former bounds tag construction only, the latter is destructive at convert time and is the subject of docs/04_convert.md. Notes the two couplings a reader will otherwise miss: a per-run library means a per-run search space, which pooled rescore handles via `source`/`run_psm_q` but cross-run quant does not; and the stage must be paired with `compete.group_by = peptidoform_charge`, since the default competition key deletes exactly the modform candidates the prescan was run to keep. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…er pruned The doc described what the stage keeps but not what it cannot touch, which leaves the reader unsure whether prescan tuning can damage an ordinary proteome result. It cannot. `anchored_tris` emits trimers only for positions carrying an `anchor_mods` modification, so a candidate without one yields an empty tag set and never appears in the output; library assembly then restores the whole unanchored remainder unconditionally. Verified on a 54.8M-row library with three cysteine anchors: 41,434,790 rows eligible (75.6%), 13,386,766 never at risk (24.4%), 4,906,730 survivors, and 0 survivors lacking an anchor modification. That makes the stage cheap to experiment with: `tol_da`, `rt_slack_s` and `top_peaks` can only add or remove modified hypotheses. Also records why the stage walks every row instead of pre-filtering to anchor-bearing ones: it costs about a second and keeps the eligibility rule in one place rather than duplicating it as a peptidoform string match that could drift out of agreement with the configured anchors. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…a command substitution A Phase B orchestration script died with exit 1, an empty log, and nothing on stderr. Cause: under `set -euo pipefail`, a run-list built with `[ -f ... ] && echo "$r"` makes the command substitution return non-zero whenever the LAST glob entry fails the test, and `set -e` then terminates the script without a message. It happened once five of eighty-three runs were incomplete and the last of them was the final directory in the glob. The failure is worth documenting because the symptom points nowhere: the log contains only `nohup: ignoring input`, and the obvious suspects (CRLF line endings, a missing config, an empty glob) all produce visible errors instead. `bash -x` locates it immediately, which is the actual lesson. Fix is an explicit `if`, so no failing command is ever last in the substitution. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…ted) In-sample residual-percentile windows underestimate the tail an unseen library peptide has, and they invert under model capacity: an RT model that memorizes its anchors is handed the narrowest window exactly when its true error is largest (measured in-sample 15.9 s vs 24.9 s abs-median ranking two DeepLC versions backwards against held-out 195.1 s vs 46.4 s). window_holdout_frac = f holds out anchor peptides by the deterministic rule base_peptide_id % 1000 < round(f*1000), fits the sizing curve on the rest, and takes w_rt from the held-out residual percentile. The rule is duplicated verbatim in deeplc_finetune.py (--window-holdout-frac, passed by run/run-experiment) so the fine-tune reference excludes the same peptides; without that, adapter memorization leaks into the residuals. The calibration curve applied to the library still uses every anchor. Guards: anchor-count fallbacks to in-sample with a warning; hard error when combined with adaptive_rt_window; cal.json records w_rt_sizing, n_sizing_train, n_holdout, and the held-out residual scale. Standalone rt-im-train now also warns that finetune_deeplc is orchestrator-only, which previously read as fine-tuned on runs that never were. Measured end-to-end (AIF benchmark, DeepLC 4.1.0, frac 0.3): w_rt 141.5 -> 204.9 s, peptides at 1% 10,703 -> 10,822 (+1.1%) at unchanged 0.98% decoy; with the overfitting 4.0.0a2 model the honest window is ~950 s and costs 1.5%. Default off; benchmark-gated (docs/08 section 4b). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
… variance On LFQ_Orbitrap_AIF_Ecoli_01 the validated workflow now measures 90.4-91.6% of DIA-NN 2.2.0 lib-free --reanalyse (11,817 peptides at 1%). Prefer the augment_library.py-completed tables: the raw imported library is missing 209 of DIA-NN's peptides (all N-terminal Met excision); augmentation recovers ~80 at unchanged 0.98% decoy and parity elsewhere. Also record that DeepLC per-run fine-tuning nondeterminism propagates into w_rt under held-out window sizing (held-out p95 150-211 s across draws, ~2% peptide swing), so single-run comparisons of window sizing or library variants must be judged against that variance. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Two quantification options, both off by default so an existing config is bit-identical: - `quant.fragment_selection = predicted` ranks a precursor's fragments by their library intensity instead of by their own integrated area. Ranking by observed area preferentially selects interfered fragments, because interference inflates exactly the areas the ranking rewards, and the selected set then varies run to run. - `quant.fixed_scan_halfwidth` / `quant.fixed_window_s` integrate a fixed window centred on the identification apex rather than the descent-walk bounds, whose 1/6-height walk is itself interference-sensitive. The seconds form is instrument-independent and overrides the scan form. - `quant.baseline_subtract` (with `baseline_flank_scans`, `baseline_quantile`) subtracts a flank-quantile background inside the fixed window. Measured on the ProteoBench Astral HYE set (six runs, second-pass MBR, guard 0.8): median |epsilon| 0.273 -> 0.195 and CV 0.175 -> 0.107 at `fragment_selection = predicted`, `top_n_fragments = 12`, `fixed_window_s = 5`. On the AIF HYE set with `fixed_window_s = 20`, |epsilon| 0.210 -> 0.181. The window has to be sized per acquisition (about 1.5x the median peak half-width), so nothing here becomes a default: promotion needs entrapment validation on both acquisitions per docs/20. `predicted_intensity` is read as an OPTIONAL chromatogram column. Requiring it made every chromatogram artifact written before the column existed unquantifiable, which the applied-window contract test caught. The `predicted` ranking, which is the only consumer, now fails with an actionable message instead. Under a fixed window the reported `integration_lo_rt`/`integration_hi_rt` are the RT extent actually integrated, not the walked bounds that were ignored; `fixed_window_indices` is shared by the integration and the reported contract so the two cannot drift. `validate()` rejects a negative or non-finite `fixed_window_s` and a `baseline_quantile` outside [0, 1], and warns when both fixed-window forms are set or when `baseline_subtract` is on with no fixed window to apply it to. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The M5 augmented scored table lowered only `q_value`, but quant gates on `quant.q_filter`, which the experiment path sets to `run_psm_q`. An accepted transfer therefore kept a sub-threshold `run_psm_q` and was never quantified: 34,280 of 34,664 transfers on the six-run HYE pooled run, i.e. MBR appeared to run and changed almost nothing. Lower `q_value`, `run_psm_q`, and `experiment_psm_q` (whichever the table carries) to the transfer q on the matching (candidate_id, source) row. Note for interpretation, measured on that run: fixing the gate alone raises min-3 ProteoBench features 70,657 -> 77,172 but worsens median |epsilon| 0.211 -> 0.245, because transfers into an ion's low condition sit at the noise floor and compress the ratio. The gate fix is correct; it needs transfer quality control (fragment-consensus guard, condition evidence) alongside it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Four gaps, all of which let untracked clutter or 3 GB of build output show up as uncommitted work, and one of which silently hid a tracked doc: - `**/target/`: the workspace target dir is `rust/mumdia/target`, which the root-anchored `/target` never matched. - root-anchor `/*_plan.md`, `/MISS_ANALYSIS*.md`, `/fragindex_*.md`, `/speedup.md`. Unanchored, `*_plan.md` also matched `docs/22_release_plan.md`, so a tracked developer doc could not be committed. `/*.md` already keeps root notes local, so this only narrows the patterns. - `/config_*.json`, `/config.aif-*.json`, `/config.hye-*.json`: 19 experiment configs with machine-specific interpreter paths were untracked but not ignored, so `git add -A` would have committed them. - `/lib/`, `/lib_capped/`, `/raw_files/`, `/val*/`, `/mbr_k3/`, `/mbr_work/`, `/missed_xics*/`, `/alphadia/`: local benchmark inputs and scratch outputs. The libraries under `lib/` are irreplaceable local data rather than build output; ignoring them keeps them out of commits without inviting deletion. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The Docker `deeplc` env pinned a DeepLC git commit from the 4.0 multitask branch and capped `numpy<2` for a `pandas<2` constraint that release carried. Pin `deeplc==4.1.1` from PyPI instead and drop the numpy cap, which 4.1.1 does not need (it declares numpy<3, torch>=2.6, python>=3.11). 4.1.1 is a floor rather than merely the current release: the 4.0.0a2 multitask preview overfits per-run fine-tuning badly enough to invert RT-model rankings (in-sample 15.9 s vs 24.9 s but held-out 195.1 s vs 46.4 s for the same model pair, docs/08 section 4b), so an older DeepLC changes results and not only performance. Add `env/mumdia-deeplc.yml`, the portable equivalent for a native install. The DeepLC sidecars previously had no committed local env spec at all, so running them meant reproducing a developer machine by hand. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Inventory of the tree against the gates CLAUDE.md states, the reasons the engine is hard to run today (each cited to the file that shows it), and a two-release plan: v0.1.0 makes today's engine installable, portable, tested and documented without promoting any new default; v0.2.0 ports the validated second-pass multi-run workflow out of the benchmark shell scripts and into the engine. Work packages carry acceptance criteria, sequencing, and effort estimates, plus a definition of done and an explicit out-of-scope list so the release notes cannot imply capabilities that do not exist. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Standard repository files a public release needs, and which a reader currently has no substitute for. CONTRIBUTING covers the build, the exact checks a change has to pass, what the test suite does NOT cover (no sidecar is exercised, so a green run is not sidecar validation), the invariants that are easy to break without noticing (determinism, label leakage, paired decoys, artifact versioning, the clean-room boundary, the Parquet encoding contract), the bar for changing a default, and how to report a benchmark number so its row unit and q-value column are stated. SECURITY names GitHub private vulnerability reporting as the channel and states the threat model, which for a local analysis tool is mostly untrusted input files: a configuration names Python interpreters and is executed, so it is as trusted as a shell script, and a finding that requires a hostile configuration is not a vulnerability. The changelog also records what is versioned independently of it, since that is what matters when reading an old result: the per-artifact Parquet schema versions and the hashed feature-set identity, both stamped into manifest.json. `.gitignore` re-includes these root files. `/*.md` with only README and CLAUDE negated had excluded every standard repository file at the root. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
CI ran only `cargo build --release --locked` and `cargo test --locked`, while CLAUDE.md names `cargo fmt --check` and `cargo clippy -- -D warnings` as part of the gate. Formatting and clippy were therefore a local responsibility, and a tree that failed them could reach main. Measured today: the tree failed both (two unformatted hunks, two never-used functions) and one test, on a branch that had been treated as ready. New jobs: - `lint` on Linux: fmt, then clippy over `--workspace --all-targets`, so dead code reachable only from a test module still fails the build. - `build-test` unchanged across the three platforms, with `--workspace` made explicit: the release binary is one of three members, and the config and IO contracts live in the other two. - `sidecars`: `compileall` over `scripts` and `ci` (the Python workers are not exercised by the Rust suite, so a syntax error in one surfaces only mid-run), a JSON parse of every tracked configuration, a YAML parse of the environment specs, and the documentation-reference check. `ci/check_doc_refs.py` fails when a tracked file cites a Markdown document the repository does not ship. Source comments carry provenance by citing the document that specifies each behavior, and about 130 of those citations point at local-only design notes, so a public clone sends the reader after files it never received. Tracking those notes is not the fix: one of them quotes proprietary constants from a closed-source engine and must stay out of the repository, which is why the script refuses to advise adding an ignored file and why a small POLICY_FILES set may name it in order to explain the policy. Dependabot keeps Cargo and Actions dependencies moving, monthly and grouped. `arrow` and `parquet` are grouped apart from everything else because they carry the on-disk artifact contract, so a bump there is a data-format review rather than a routine upgrade. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
No manifest carried a description, repository, homepage, keywords, categories or publish field, so the binary and the crates advertised nothing about what they are or where they come from. `publish = false` is deliberate rather than pending: mumdia-core and mumdia-io are internal boundaries of one application, not libraries with independent consumers, and the binary needs the Python sidecars and configs that a crate cannot carry. Distribution is the release archive and the container image. `rust-version` said 1.85 while `rust-toolchain.toml` pins 1.96.1, which is the only version CI builds and tests. A lower floor may well work, but claiming an unverified one is a guess; the comment says to lower it together with an MSRV job that proves it. `strip = "symbols"` in the release profile: archives shipped unstripped debug symbols, which is most of the binary size for no user benefit, and a backtrace still names the functions. Cargo.lock is unchanged. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The release archive held the binary, README and LICENSE. That is not an installation: the ML predictors and rescorers are Python sidecars the engine launches by path, so a binary alone cannot fine-tune retention times or rescore with mokapot or the neural network. The archive now also carries `scripts/`, `env/`, `docs/`, the changelog, and the example configs when a tag has them, plus a sha256 file, and it lists its own contents in the build log. Targets gain `x86_64-apple-darwin` so Intel Macs are covered alongside arm64. Each build smoke-tests the binary (`--version`, `--help`, `doctor`) before packaging, so a broken executable fails the release instead of shipping. The image had never been built or run by CI, only on a tag that was never pushed. The Docker workflow now builds into the local daemon, runs the checks, and pushes only afterwards. The checks are the ones that can actually fail here: both baked configs parse and their sidecars import; DeepLC is imported for real in the worker's order, because the failure that catches (torch DLL initialization after numpy) happens at import time and not at the module-presence probe `doctor` performs; and a bind-mounted directory is written both with and without `--user`, which is the primary way anyone uses this image. The image also no longer stays root after setup. Root was needed only for apt and for creating the conda environments, which are read-only at run time. The container user's uid is assigned by the base image and will not match a host uid, so the documented invocation now passes `--user "$(id -u):$(id -g)"`, and the smoke test asserts that path works rather than assuming it. `git` is dropped from the image since DeepLC is pinned to a PyPI version rather than a repository commit. Standard OCI labels point back at the source. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Verified on a Linux host with Docker 29.4 rather than assumed, since neither the image nor the DeepLC pin had ever been built: - the image builds (4.62 GB, amd64) and runs unprivileged as uid 57439; - `mumdia doctor` passes on both baked configurations; - the `deeplc` environment imports DeepLC 4.1.1, torch 2.12.1+cpu and numpy 2.4.6 in the order the worker uses, which confirms dropping the `numpy<2` cap was safe, and `rescore` imports mokapot 0.10.0; - all 11 sidecar workers are present; - a bind mount is writable with `--user "$(id -u):$(id -g)"` and the result is owned by the host user. Without `--user` it fails at the mount point (`mkdir: cannot create directory '/data': Permission denied`), so the flag is documented as required rather than optional. The quant refactor is numerically neutral, checked against the exact output behind the ProteoBench AIF submission: 72,168 quantified precursors, `quantity`, `n_fragments_used`, `quant_status` and `integration_apex_rt` all bit-identical. Only the reported window moved, from a 29.1 s median (the descent-walk bounds the fixed-window path never integrated) to 34.9 s, which is the fixed window that produced those numbers. The published submissions therefore still describe this code. Also smoke-test every release target instead of skipping aarch64-apple-darwin: macos-latest is arm64 and macos-13 is x86_64, so each runner executes the architecture it builds. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Source comments carry provenance by naming the document that specifies each behavior, and about 130 of those citations named documents the repository does not ship: `plan.md` (110 references across 55 files), plus `comment.md`, `fragindex_spec.md`, `mbr_plan.md`, `COMPARISON.md` and three more. A public clone therefore sent the reader after files it never received, and the citation looked authoritative while being unreachable. Tracking those notes is not the fix. `plan.md` section 8 is a comparative dossier that quotes proprietary constants and internal line numbers from a closed-source engine, so publishing it would break the clean-room boundary the project claims; it stays local. Each citation now points at the tracked `docs/` page that describes the same thing today, with the section or stage number dropped, because the tracked guide does not share `plan.md`'s numbering and keeping the number would be a false pointer. Where the citation was the whole justification for a sentence, the dead pointer is deleted rather than replaced by an empty one; the technical claim stays. `CLAUDE.md`, `docs/02` and `docs/14` now say plainly that the design notes are untracked, so a reader is not left looking for them. `ci/check_doc_refs.py` no longer treats a quoted `.gitignore` glob as a file reference. The check passes: 305 references, all resolvable. Also restores a word the rewrap dropped in `index.rs`, whose module doc had come out as "shared by / and extract" with "search-seed" lost between the lines. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`peptides.tsv` and `proteins.tsv` are the only outputs most users read, and the data dictionary explicitly excluded them, so their columns were documented nowhere. `docs/15` now lists both, each column tied to the source line, and states the trap: rows are precursors `(peptidoform, charge)` while the filter is `peptide_q_value`, a base-peptide q, so a row count is not a precursor-q-controlled count. It also says the printed values are rounded for reading and points at the Parquet tables for analysis. The schema-version registry had drifted: it claimed `psms_scored` was v3 and put `psms_competed` at v2. Per `schema.rs:7-25` they are v4 and v3, with `psms_extracted`, `peptide_quant` and `protein_group_quant` at v2. The section now also says where a version is stamped, so an artifact on disk answers the question itself instead of inheriting the engine's version. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The WP0 table asked ten questions; all are now answered, so it records decisions rather than proposals. Three are worth reading: - platforms are Linux, Windows and macOS, with both macOS architectures built and each binary smoke-tested on its own architecture; - the Python floor is DeepLC 4.1.1, verified to resolve with torch 2.12.1+cpu, numpy 2.4.6 and pandas 2.3.3 on Python 3.11 both in the image and from the new portable env spec; - `plan.md` stays untracked. This reverses the plan's own recommendation, on inspection of the file rather than on preference: its section 8 quotes proprietary constants and internal line numbers from a closed-source engine, and its section 11 warns that reuse needs licence clearance, so publishing it would contradict the clean-room boundary the README claims. The 130 dangling citations were redirected instead, and CI now blocks new ones. `CITATION.cff` is blocked rather than pending: the author list is not derivable from the repository and guessing authorship is not acceptable. A progress table lists what landed with the evidence each item was checked against, and the sections below mark the completed WP1, WP2 and WP4 items. It also states the two things local verification cannot cover: neither the new CI workflow nor the Docker workflow has run on a GitHub runner, because nothing has been pushed. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Eleven citations named `fragindex_spec` without a `.md` suffix, so neither the filename grep nor the new CI check saw them: `(fragindex_spec Section 2.1)` reads as prose, not as a path. They pointed at an untracked local specification like the rest, so they are redirected to `docs/06_predict_frag_index_matchers.md`, with the section numbers dropped because the tracked page does not share that numbering. Inside `docs/06` itself the citations become "this document" rather than a self-reference. `ci/check_doc_refs.py` now also scans for the bare stems of the known untracked notes, which is the gap that let these through. It reports 312 resolvable references and no extension-less citation. One nearby false positive left alone: `features.rs:115` has a `PLAN` static, which is a variable name and not a document. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…figs
A MuMDIA config named the Python interpreter for each sidecar as an absolute
path, with no lookup and no discovery, so a config belonged to the machine that
wrote it. The only tracked config carried one developer's `C:/Users/...` and
OneDrive paths and was the config `CLAUDE.md`, `docs/19` and `docs/20` told
everyone to run, so a new user's first action was always to edit it. This is the
main reason the engine was hard to run.
`rust/mumdia/crates/mumdia/src/python.rs` gives each sidecar role one resolution
path. A field may be an absolute path, used as given and never second-guessed, or
`"auto"` (or absent), which searches `MUMDIA_PYTHON_<ROLE>`, `MUMDIA_PYTHON`,
`CONDA_PREFIX`, `VIRTUAL_ENV`, then `python3`/`python` on `PATH`. A candidate is
accepted only after it imports that role's own module list, so discovery cannot
pick a Python without torch and defer the failure to the rescore stage hours
later. A role is resolved only when the configuration uses it, so the default
native run probes nothing and still works on a machine with no Python.
Resolution happens before the config hash, so `manifest.json` records the
interpreter that actually ran rather than the word `auto`. That makes the hash
machine-specific for an `auto` config, which is the honest outcome.
`predict_frag.sidecar_script_dir` is resolved against the config file's own
directory and the executable's directory as well as the working directory. The
same config invoked from a different directory used to silently change which
worker scripts ran.
`mumdia doctor` now answers "can this configuration run?" instead of "do three
hard-coded interpreters import three hard-coded lists?". It reports the resolved
interpreter and how it was found, the versions of packages whose version changes
results, and whether the worker scripts are present; it covers `mbr.python` and
the script directory, which it never checked; and it no longer fails a native
configuration over a worker directory that configuration never opens. It warns
when DeepLC is older than 4.1.1, since the 4.0.0a2 preview overfits per-run
fine-tuning badly enough to invert RT-model rankings.
`configs/examples/{native,fasta-sidecars,diann-library}.json` replace the tracked
machine-specific config, with `configs/README.md` covering the resolution order,
the environment specs and the DeepLC floor. The old file is untracked but left on
disk so an existing local workflow keeps working. `shipped_configs_parse` now
covers all five shipped configs.
Verified against the real environments on this machine: a native config is
runnable with nothing probed; `nn_torch` with no torch anywhere fails with a
message naming the variable and the modules; `MUMDIA_PYTHON_RESCORE` pointing at
a torch env resolves and reports torch 2.5.1; and a DeepLC 4.1.0 environment is
accepted with the version warning.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`--threads N` bounds the engine's rayon pool and forwards the count to the Python sidecars as `MUMDIA_NN_THREADS` and `OMP_NUM_THREADS` when those are unset. There was previously no way to bound MuMDIA: the engine never read `RAYON_NUM_THREADS`, so rayon took every core, which on a shared machine is antisocial and, for the NN rescore worker, actively slower. That worker measured faster on 8 threads than on 32. An already-set variable is left alone, because a user who exported `OMP_NUM_THREADS` did so for a reason. `--log-level` accepts any `RUST_LOG` filter, and `-v`/`-vv`/`-q` map onto levels. Verbosity was `RUST_LOG`-only, which does not appear in `--help` and is awkward to set on Windows. `RUST_LOG` still works and still offers per-module filtering; an explicit flag wins over it. `-q` and `-v` are mutually exclusive rather than silently ordered. All four are `global = true`, so they parse on either side of the subcommand, which is what a user types. Tested for that, for the level mapping, for the explicit-level precedence, and for `--threads 0` being rejected. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The validated-workflow block now names configs/examples/diann-library.json, whose interpreters are "auto". That runs unchanged only where an environment with torch and DeepLC is discoverable, so the block says so, points at doctor for the diagnosis, and notes that copying the example and writing the two paths in is the supported way to pin them. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
There was no fixture and no end-to-end test. The Rust integration test builds its inputs in process and starts at `extract`, so mzML parsing, the library build, the `run` orchestrator, the manifest, retention-time calibration on real anchors, and `quant`/`report` writing files were all untested. `.gitignore` has whitelisted `test_data/**/*.parquet` for some time, but no `test_data/` existed. The fixture is generated rather than committed, for two reasons. A usable slice of a real DIA run is megabytes of binary in git and carries a licence question about excerpting a public raw file. More importantly, the planted fragment peaks have to sit where the engine looks for them: `ci/make_fixture_mzml.py` reads the precursor and fragment tables that `mumdia predict-frag` just produced and plants exactly those m/z values, so the fixture cannot disagree with the mass model, and if the mass model changes the fixture changes with it. `test_data/fixture.fasta` is 16 synthetic proteins composed of 160 distinct tryptic peptides, giving 3,820 library candidates. The generator writes a small DIA acquisition: 60 cycles of one MS1 and eight MS2 windows, 160 target precursors planted with Gaussian elution profiles, retention time an affine function of the library iRT so calibration has a real relationship to fit, and two kinds of seeded noise. The second kind matters: noise drawn from the library's own fragment m/z pool is what gives decoys the chance evidence a real run has. Without it every accepted candidate was a target and rescore correctly refused the run for having no decoys, so the fixture could not reach FDR, quant or report at all. `ci/check_smoke.py` asserts 102 things, in two kinds. Exactly determined facts are asserted exactly: spectrum and window counts, that retention time arrives in seconds, every artifact's blake3 hash shape and row count, all 17 artifact schema versions, that the manifest's rescorer identity matches what the scored artifact reports, and that the classifier that ran is the one requested. Scientific outcomes are asserted as bands, because pinning them would turn a sensitivity improvement into a red build. Measured on this fixture: 99.3% of planted peptides recovered (151 of 152), zero decoys at 1% peptide q, LOESS calibration fitted on 110 anchors with a 1.29 s in-sample residual median, and a byte-identical `peptides.tsv` across two runs. That last one is the determinism contract tested through the CLI for the first time; the Rust test only compared `apex_rt` within one process. `ci/smoke.sh` drives it and runs on Linux and Windows in CI. macOS is omitted to keep CI minutes down, which is a deliberate gap. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A manifest said `mumdia_version: 0.1.0`, which every build from this branch says, so a result could not be tied back to the code that produced it. docs/20 asks every benchmark record to carry "commit/build:" and the manifest could not supply one. Inputs were not recorded at all: an imported library was hashed, but the mzML and the FASTA were not, so a manifest could not answer which file it came from. `mumdia-core/build.rs` stamps the short commit and the commit date into the binary, with a `-dirty` suffix when the worktree carried uncommitted changes, because a number produced from uncommitted code is not reproducible from the named commit. The COMMIT date is used rather than the build time on purpose: build time changes on every rebuild, which would make the binary unreproducible and defeat caching for no benefit. Without git the values become `unknown` rather than failing the build, so a release-tarball build still works. `Manifest` gains `git_sha`, `commit_date`, `cli_args` and `inputs`, plus `provenance()` for the one-line stamp a benchmark should quote. `run` hashes every input before compute starts. `cli_args` matters because the flags that are not in the config, `--top-peaks-ms2`, `--threads` and `--max-spectra`, previously existed in no record at all, and the peak cap changes results. All four fields are `#[serde(default)]`, so manifests already on disk keep parsing; a test asserts that, because making prior runs unreadable would be a worse outcome than the gap being fixed. `run-experiment` gains the same provenance, including a hash per input mzML. Its manifest is still thinner than the single-run one: it carries no per-artifact records, because the per-run chains do not thread a shared manifest. `scripts/deeplc_finetune.py` takes `--seed` and seeds numpy and torch, wired from `rng_seed`. Unseeded, the draw varied enough to change results: the held-out RT window p95 moved 150 to 211 s across two draws of one benchmark arm, worth about 2% of peptides, which made single-run comparisons of window sizing or library variants unreadable. Training kernels are still not bit-for-bit deterministic, so this narrows the variance rather than removing it, and both the flag help and the log line say so. The smoke test asserts the new fields; it is now at 112 assertions. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Four gaps, each the kind that only shows up on real data. `report` had one test, for a string helper, while it writes the two files most users read. Now covered: a decoy with the best q AND the best score must still be excluded, because a decoy in a user-facing report is not cosmetic, it is reported as an identification; an above-threshold target is excluded; the stripped column is the modification-free sequence, which is the unit `peptide_q_value` controls; an empty protein group does not become a row; and a quantity with no quant table is an empty cell rather than a zero. A run that identifies nothing still writes both files with headers, so a downstream reader fails on empty data instead of a missing file. That is not hypothetical: it is the state the fixture run reaches for `proteins.tsv`, where 16 groups cannot reach 1 percent FDR. `quant` with an empty chromatogram table now has a test. Extraction can accept a candidate and write no chromatogram for it, and the identification has to survive with an explicit unquantifiable status. An accepted PSM silently disappearing from the peptide table would lose a reported identification to a quantification detail, which inverts the contract that identification and quantifiability are separate. The rescore one-sided-population guard moved into `require_both_labels` so its message is pinned by a test. A one-sided population is not a technical failure: the classifier would train and q-values would come out, and they would be meaningless. Which side is missing points at a different cause, so both counts appear in the message. The fixture exercised this for real before the noise model gave decoys any chance evidence. A failing sidecar now says which interpreter ran, which script, and with what arguments, and points at the worker output above and at `mumdia doctor`. It previously said only "worker X exited with status N", naming neither the environment nor the call. Output stays inherited rather than captured, because the NN rescorer prints progress across hours and a user watching a run needs to see it live. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The quantitative numbers MuMDIA is judged on were produced by scripts that lived only on the benchmark machine, so a figure in the documentation could not be traced to the commands behind it. `bench/` carries the three that matter and are portable: the submission builder, which expands MuMDIA's protein entry names back to full FASTA identifiers so ProteoBench's own contaminant and species flags apply unchanged; the offline scorer, now taking `--module` instead of a hardcoded module id, since the module fixes the run names and the expected ratios and scoring against the wrong one fails silently; and the condition-evidence filter, parameterised on runs, conditions, directories and threshold rather than assuming one experiment's layout. `prov_filter.py` is verified rather than assumed: run against the six AIF runs it reproduces the recorded F1 counts exactly, 69,190 / 71,432 / 66,213 / 60,891 / 63,053 / 60,296 kept, which is what the inline filter in the benchmark pipeline produced. It also builds its presence matrix as bool directly, avoiding a pandas object-dtype fillna that is deprecated and would change behaviour. The README states both recorded results with their units, and both readings that follow: accuracy is competitive or better than DIA-NN 2.2.0 while completeness trails by 13% on Astral and 21% on AIF, and per-ion precision is comparable on Astral but 1.7 times worse on AIF. It also records that excluding oxidised precursors improves BOTH tools by a similar margin on the AIF module, which is why that confound belongs to the archived module and not to one tool. The second-pass shell pipeline these feed is deliberately NOT vendored here: it becomes engine code in v0.2.0, and copying it in would duplicate work that is about to be replaced. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Wall clock, per-stage timing and disk footprint were documented nowhere, so "how long does this take and what does it need" had no answer. Measured on one AIF file: 1.94 GB mzML in, 1,815,610 PSMs, about 85 minutes, 13.1 GB of artifacts out. Two facts worth acting on come out of it. Rescoring is 80% of the run, and the NN worker measured faster on 8 threads than on 32, so `--threads` is worth setting rather than leaving to rayon's every-core default. And the artifacts are 6.8 times the input, concentrated in chromatograms, features and the competed table, which is what a user needs to know before pointing --out-dir at a small disk. Numbers come from each artifact's report.json elapsed_ms and from the artifact sizes, so they are reproducible from any run's own output rather than from a stopwatch. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The README was written for someone who already knew the codebase. It had no install path beyond a build command, no output-format section, no q-value unit table, and a benchmark section whose numbers did not say what they counted. It now covers: three install routes, each complete (the container, which needs `--user "$(id -u):$(id -g)"` to write into a bind mount; the release archive, whose binary alone cannot run the Python sidecars; and from source); a quickstart that starts with the native zero-dependency path; the `"auto"` interpreter resolution and the global flags; the output files with the `peptides.tsv` and `proteins.tsv` column tables; and the q-value units with the two traps, that `precursor_q` is only a precursor unit under `compete.group_by = peptidoform_charge`, and that the grouped columns are written only to each group's winning row so a per-run count from them after an experiment-wide rescore is diluted. Every benchmark number now states its row unit and its selection column, and the ProteoBench figures carry the caveat that matters: they were produced by the second-pass workflow that is still prototype shell code, so this release does not reproduce them on its own. One claim from the old README, "97 to 98% sequence concordance with DIA-NN", is removed rather than restated: it is not attributable anywhere in the tracked tree. A "status of experimental features" section lists what is accepted but gated or diagnostic-only, so the README cannot imply capabilities that do not exist, and states the hard limits: mzML only, no ion mobility, no wildcard or terminal variable modifications, `run` takes one file, `run-experiment` writes no TSV report. docs/14 was stale in ways that mattered: it said the suite had 126 tests (178), that CI ran only build and test, and that fmt and clippy were a local responsibility. Its per-file test table, its CI, release and Docker workflow descriptions, the MSRV, the `validate()` rejection count, and the `doctor` behaviour are all re-derived from the current files, with the line citations renumbered. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Measured while verifying the smoke test on Linux: `peptides.tsv` and `proteins.tsv` hash identically on Windows and Linux, down to the quantity digits, and so does the generated fixture mzML. The native pipeline is therefore byte-reproducible across operating systems and not only across runs on one machine, which is a stronger property than anything documented and worth keeping. Nothing enforced it, so a platform-dependent float reduction or a HashMap-order sum could have reintroduced a difference silently. `ci/smoke.sh` now writes the output hashes, the smoke job uploads them per platform, and a new job diffs them. The two platforms are compared against each other rather than against a committed hash. A golden value would have to be updated by every legitimate change to scoring, which turns an improvement into a chore and the check into a rubber stamp; comparing platforms costs nothing to maintain and still catches the class of bug that matters. Also verified on Linux in passing: the `git_sha: unknown` fallback works, since a build from an extracted tarball has no git available, and it took the documented branch rather than failing. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Two reference documents that nobody has to maintain by hand, because a reference nobody regenerates is worse than none: it reads as current. `ci/gen_cli_reference.py` drives the built binary's `--help` into `docs/23_cli_reference.md`: all 21 subcommands, the four global flags documented once, and which subcommands accept `--config`, derived rather than hardcoded. It turns out 16 do and 5 do not (`audit`, `convert`, `inspect`, `quant-lfq`, `report`). Output is normalised so a Windows and a Linux build produce identical text, which makes running it on both CI legs a free cross-platform check on the help itself. `ci/gen_config_reference.py` parses `config.rs` into `docs/24_config_reference.md`: 17 structs, 166 fields with type, default and description, 21 enumerations with their serde spellings, and the 20 fields whose own doc comment marks them benchmark-gated, diagnostic or not yet wired. Defaults are rendered as the JSON a config would carry, including the arithmetic ones (`1.0 / 3.0`), and no field is omitted for want of a resolvable default. It also generates the environment-variable table that existed nowhere: 47 variables read across the engine and the sidecars, and separately the 11 the code sets, which is what explains why exporting `OMP_NUM_THREADS` has no effect inside `deeplc_finetune.py`. Both take `--check` and are wired into CI, so a new field or a changed flag lands with its documentation or the build fails. `docs/19_getting_started.md` no longer documents one developer's machine. It opened by naming an interpreter under `C:/Users/robbi/...` and a build output in a path nobody else has, and its environment table listed conda environments that exist only there. It now starts with `ci/smoke.sh`, which needs nothing but the repository, then the two shipped environment specifications and `"auto"` interpreter resolution, and only then the runs that need data you supply. Every machine-specific path is gone. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The four library helpers wrote their output with `DataFrame.to_parquet`, which chooses the arrow string width itself. pandas 3 adopted the arrow-backed `str` dtype, so on pandas 3.0.5 with pyarrow 25 they emit `large_string`, and the engine rejects that at load with `column 'peptidoform' is not utf8`. Reproduced here. Every library built by `import_diann_lib.py`, `make_shift_decoys.py`, `make_reverse_decoys.py` or `augment_library.py` on a current pandas is therefore unreadable by the engine, which breaks the imported-library path, the highest-sensitivity workflow there is. The failure is version-dependent, which is worse than a plain bug: it works for whoever wrote the script and fails for the next person, at load time, on a file that looks fine. `scripts/_lib_io.py` narrows the 64-bit-offset arrow types back, recursively so a `large_list<large_string>` is handled at both levels, and pins snappy. The four helpers write through it. It is shared rather than copied into each because it is a correctness contract, and the copy that drifts is the one that produces an unreadable library. Found by the new sidecar contract tests, which had marked it an expected failure with the note that the fix belonged in `scripts/`. The markers are now gone and the assertions are plain, so a helper going back to `to_parquet` fails the build. CI runs the contract tests. Those needing torch, mokapot, deeplc or ms2pip skip rather than fail, so what CI actually covers is the contracts needing only numpy and pyarrow: the whole `mbr_worker` suite, including the regression for the q-column bug that left 34,280 of 34,664 transfers unquantified, the library-recipe encoding contracts, and the import-order checks. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Marks the completed items with what was actually delivered and how it was verified, and states the two that remain blocked or unbuilt rather than quietly dropping them: CITATION.cff needs an author list that is not derivable from the repository, and the issue and pull-request templates are not written. The changelog gains the smoke test, the cross-platform byte-equality job, the Python contract tests, the two generated references, and bench/. Its known-limitations section is corrected: it claimed there was no fixture and no end-to-end test, which is no longer true, and the honest remaining gaps are narrower, that the sidecar tests cover file contracts rather than the science and that no CI job exercises the sidecar path end to end. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
There were no Python tests. The seven engine-invoked workers exchange files over a positional-argv contract that nothing checked, so the Rust suite could pass while a worker violated it, and the violation surfaced hours into a real run after the search compute had been spent. 71 tests across the workers, each building a tiny synthetic input, invoking the worker as a subprocess with `sys.executable` exactly as the engine does, and asserting the on-disk contract. Nothing reads real data, the network, or any absolute path from a developer machine. The one that matters most needs no ML dependency and therefore runs in CI: the `mbr_worker` regression asserting that an accepted transfer has `q_value`, `run_psm_q` AND `experiment_psm_q` lowered, on the matching `(candidate_id, source)` row only. Lowering just `q_value` is what left 34,280 of 34,664 transfers unquantified on a six-run experiment, because quant gates on `run_psm_q`. Also covered: out-of-fold coverage for the rescorers (every input row scored exactly once with a finite score, no row silently dropped), the library-recipe invariants the engine hard-errors on (contiguous `candidate_id`, precursors sorted by `precursor_mz`, arrow `utf8` strings, snappy), paired collision-free decoys, and the DeepLC-before-numpy import order whose violation aborts torch DLL initialisation on Windows. Tests needing torch, mokapot, DeepLC or MS2PIP skip rather than fail, so the suite is useful in a minimal environment and honest about what it did not check. Every test was verified to execute somewhere: 59 pass with numpy and pyarrow alone, 65 with mokapot and DeepLC present, and the 5 `nn_torch` tests pass against a working torch. One environment's skip message is itself informative, reporting the `WinError 1114` c10.dll failure the project documents. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`run: ci/smoke.sh` would have failed on the Linux runner with "Permission denied": the file was committed 100644, because the chmod +x I did locally never reached the index. Caught before pushing rather than by a red first build. Fixed both ways on purpose. The execute bit is set, and the step invokes `bash ci/smoke.sh`, which does not depend on it. The bit is meaningless on the Windows runner and a checkout or an archive extraction can drop it, and the failure mode is a permission error instead of a test result, which reads as infrastructure noise rather than as the missing mode bit it is. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The Windows smoke job failed with exit 120 and no failing assertion:
Exception ignored in: <_io.TextIOWrapper name='<stdout>' encoding='cp1252'>
OSError: [Errno 22] Invalid argument
`smoke.sh` piped the generator to `head -3`, which closes its stdout after three
lines while it still has 160 planted precursors to print. On a Windows console
that surfaces as `OSError: [Errno 22]` during interpreter shutdown rather than as
EPIPE, so Python exits 120 and the script's `pipefail` propagates it. It passed on
Linux, where the write is a plain broken pipe, and locally, where the console
encoding differs, so only the CI Windows runner saw it.
The generator takes `--quiet` instead: the summary line still prints, the
per-precursor listing does not, and nothing truncates a pipe. Fixing the cause
rather than trapping the exception, because a caller that has to truncate output
to keep a log readable will do it again somewhere else.
The other six jobs passed on their first run, including the Linux smoke job and
build-and-test on all three platforms.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Two tracked lines described more than the decision they recorded. The `plan.md` row in the release plan named the source file, the specific constant table, and asserted that publishing the document would contradict the clean-room claim the README makes. Whatever the intent, that is a public, self-authored statement about the provenance of material the project holds, in the repository of a competing tool, and it ships in every release archive because the workflow copies `docs/` wholesale. The decision it records is unchanged and is still recorded: the document is not licence-cleared for redistribution, so it stays local. The `.gitignore` comment named an unpublished collaboration's biological topic in order to explain a filename pattern. The pattern does not need the explanation. No behaviour change; no citation redirected. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`docs/25_release_readiness_review.md` is the audit of the tree at `d1f3f3e`, the state PR #47 leaves behind. Verdict: do not tag `v0.1.0` yet. Three blockers remain open, and one of them was introduced by the release work itself. The largest is one root cause with many faces. The engine validates finiteness at every internal and sidecar boundary and at neither external input boundary, so a Parquet NULL becomes NaN, and the guards written to reject bad input are expressed in a form NaN passes: the ascending-m/z hard error, the eight RT-window checks in extract, and `within_ppm` all accept a NaN rather than rejecting it. The consequence is not a NaN result, it is a candidate that silently matches everything or is silently absent from its own window. One finiteness pass at library load closes it, and with it most of the reachable panic census. Two findings deserve reading even by someone who skips the rest. The apex can fall to the earliest qualifying scan whenever a candidate's three highest-predicted fragments are never observed, which propagates into the pre-FDR competition and the reported abundance; the mechanism is certain and the frequency is unmeasured. And the entrapment path, which `docs/20` designates as the instrument for promoting any sensitivity default, has three defects of its own, including a decoy that can win a peptide group and delete the real target from the analysis. Fixing that has to precede the measurements, not follow them. The review is deliberately even-handed about what held up: zero `unsafe`, byte-identical output across four machines and two operating systems, no network stack, no secret anywhere in the whole history, concurrency with no shared mutable state, and a generated CLI reference with no disagreements at all. Section 11 records the questions the audit could not settle, so they are not rediscovered from scratch. `docs/26_gui_plan.md` answers the separate request for a simple way to run MuMDIA without a terminal. It recommends an in-binary local web UI over a desktop toolkit, because the dependency and packaging cost is an order of magnitude lower and it does not fight the static musl target. The first phase is machine-readable progress and structured logs, which is worth having even if no interface is ever built on it. Both routed from `docs/README.md`. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…o" everywhere, clear two advisories
Three release blockers from docs/25.
## A Parquet NULL was accepted, not rejected
The engine validated finiteness at every internal and sidecar boundary and at
neither external-input boundary. `Table::f64`/`f32` map a NULL to NaN, and the
guards written to reject bad input are all comparisons that are FALSE for NaN,
so a NaN was accepted:
- `index.rs` ascending-m/z: `prec_mz[c] < prec_mz[c-1]` passed a NaN, and
`candidate_range`'s `partition_point` then ran on a slice not partitioned by
its predicate. A precursor genuinely inside an isolation window could be
skipped in every scan of that window for the whole run.
- the eight `extract.rs` RT-window guards are `rt < lo || rt > hi`, so a NaN
window accepted EVERY scan: the candidate was searched across the whole
gradient with no RT prior, no warning, and it still entered the training and
FDR population. The legitimate unbounded case is already written as explicit
-inf/+inf by `candidate_window`, so a NaN bound can only mean corruption.
- `within_ppm` returned true against NaN, because `min`/`max` return the
non-NaN operand. That is the canonical predicate the whole fragment-index
verify path uses, so one NULL m/z fabricated matched-fragment counts.
- `FragIndex::build` let one non-finite m/z collapse the index range for the
whole library to a 2 Da placeholder, clamping every fragment into one bin:
no error, no wrong answer, an unbounded hang.
- both FDR kernels advance their tied-block walk on `score == s`, and
`NaN == NaN` is false, so a single NaN score span forever. Two callers
(`rescoring.rs` train scores, `search_seed.rs` hyperscore) do not validate.
The primary fix is one finiteness pass in `Library::load_with`, which names the
column, the row and the file. The downstream guards are fixed too, so neither
is load-bearing alone. Non-finite scores now map to the worst rank key, which
is also the conservative direction: `total_cmp` would sort a positive NaN above
every real score.
This matters because imported-library mode is the recommended profile, skips
`predict-frag` (the one stage that had a finite guard), and a library converted
through pandas yields NaN from any left-join or empty cell.
Also here: `rt_cal` for a candidate absent from `run_windows` is now NaN rather
than 0.0. Stage B already uses NaN as the "calibration unavailable" sentinel and
`calibrated_rt_error` maps non-finite to 0.0, so a finite 0.0 gave the worst
possible `rt_error_abs` to exactly the candidates the other path gives the best
value to, making the feature a proxy for "was this candidate in the window
table". And the u32 fragment-count limit is checked at load, where it is an
error with the count in it, rather than as a bare assert after a long load.
## "auto" was only resolved by the two orchestrators
`main.rs` never called `python::resolve`, so `rescore`, `predict-frag` and `mbr`
took the sentinel literally:
Error: rescore: NnTorch sidecar failed (spawning sidecar failed:
auto scripts/nn_rescore_worker.py: program not found)
All three shipped example configs use `"auto"`, and the README documents
standalone stages. Resolution now happens in `load_config`, so every entry point
behaves the same; `doctor` keeps a raw loader, because its job is to diagnose a
configuration whose interpreters do not resolve.
Resolution also clears a role that asked for discovery but is unused, instead of
leaving the word "auto" in the resolved config. That is what made
`run-experiment` reject a config `run` accepted: its preflight stat'ed
`mbr.python` for a strategy that never runs.
The same standalone stages now also get the config-relative and exe-relative
`sidecar_script_dir` resolution that only the orchestrators had.
Verified against the binary: a native config with two unused `"auto"` roles runs
clean, and a required-but-unavailable role fails at startup naming the missing
modules rather than at sidecar spawn hours later.
## Two high-severity advisories on the mzML parser
`cargo update -p mzdata -p crossbeam-epoch` takes quick-xml 0.30.0 -> 0.41.0,
clearing RUSTSEC-2026-0194 and -0195 (both CVSS 7.5, both reachable from the
XML being parsed), and crossbeam-epoch 0.9.18 -> 0.9.20 for RUSTSEC-2026-0204.
Semver-compatible, no manifest edit; base16ct and md5 drop out. 178 tests pass
against the new parser.
`cargo audit` is now a CI job, because these sat under eight green checks:
SECURITY.md declares dependency advisories in scope while nothing in the project
could see them. Vulnerabilities fail; the unmaintained-crate warning does not,
since `paste` 1.0.15 is its final release. `ci.yml` also gains
`permissions: contents: read`, having previously inherited the repository
default for four jobs that need nothing but checkout.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
… quantification defects
## The entrapment path was not trustworthy
docs/20 makes entrapment the instrument for promoting any sensitivity default,
and CLAUDE.md gates seven changes on it. Three defects sat on that path.
1. `rescore.rs` passed `is_entrapment` into `native_scores`'s `is_decoy` slot.
`percolator_lite` selects its positive training set on `is_decoy == false`,
so every entrapment row with a low internal q was recruited as a POSITIVE
example and counted as a target inside the loop's own q estimate. The
negative class for training is the in-silico decoy population, as on every
other classifier path; entrapment is the evaluation null and has to stay
held out, or the leak estimate measures the fit rather than the FDR. Taken
whenever `rescore.python` is unset, and on a GBM failure with strict=false.
2. `grouped_q` let a decoy win a base-peptide group under entrapment q. A
target and its paired decoy share `base_peptide_id` by construction, and
`incoming_null` is `is_entrapment`, false for both, so the winner was
whichever scored higher. When that was the decoy its tuple counted toward
neither population, the group vanished from the analysis entirely, and the
real target was assigned 1.0. Both the target count and the leak metric are
computed from that q, so the instrument was reading a population its own
decoys had thinned, and under-reporting the leak. Decoys are now excluded
from the competition in entrapment mode; picked TDC in decoy mode is
untouched, and a test pins both.
3. `entrapment_ratio` was unvalidated. The estimate is
`(ratio * n_entrap + 1) / max(1, n_real)`, so at 0 it collapses to
`1/n_real`: above ~100 real targets every row passes at 1% with no null
contributing at all. A negative ratio yields negative q, which both
`count_targets_at_q` and `passes_quant_filter` accept as passing. That is
the failure mode of inverting the ratio, on the tool whose job is
validating FDR.
Any entrapment measurement taken through the native rescorer before this
should be re-run.
## Quantification
- **A pooled scored table plus one run's chromatograms is now refused.** Every
map in quant keys on `candidate_id` alone, which is a library index and
repeats across runs, so a pooled table (what `rescore --competed a b c` and
the recorded multi-run recipe produce) yielded one identical row per run,
each carrying a single run's quantity and apex, making every cross-run ratio
exactly 1.00 with no error. `run-experiment` splits by `source` first and the
docs say to; nothing enforced it.
- **A fixed window is no longer inert when `bound_peak = false`.**
`integration_apex` was populated only under `bound_peak`, so
`{"bound_peak": false, "fixed_window_s": 5.0}` -- the natural way to ask for
a fixed window with no descent walk -- silently integrated the whole trace.
The identification apex is taken directly now, and still NaN when no fixed
window is configured, so the reported `integration_apex_rt` does not name a
centre nothing was centred on.
- **The fixed window gained a distance guard and the >=2-sample rule.**
`fixed_window_indices` always returned the sample nearest the apex however
far away it was, so a weak fragment sampled 45 s off-peak contributed its
off-peak intensity as an area, where `trapezoid_window` on the same bounds
correctly returns 0. And a one-sample window returns `trapezoid`'s
single-sample fallback, a HEIGHT reported where every other candidate reports
intensity x seconds: `peak_window` widens for exactly this reason and the
fixed path did not. The recorded Astral recipe (5 s at a 1.64 s cycle) sits
at 6-7 samples, so neither guard fires there.
- **`FragmentSelection::Predicted` no longer ranks a NaN first.** `total_cmp`
orders NaN above every real value, so a NaN library intensity reached the
front of the descending ranking and was preferentially summed. The
absent-column fallback substitutes 0.0, which is finite, so it is unaffected.
## MBR was invisible to its own downstream
`mbr_worker.py` lowers the three PSM-level q columns, while `quant.q_filter`
defaults to `peptide_q` and the report writers filter on `peptide_q_value` /
`pg_q_value`. So `mumdia mbr` followed by a manual `quant` or `report` showed
no transfers at all; only `run-experiment` worked, by force-setting
`q_filter = psm_q`. This is the same class of bug the worker's own comment
records having fixed for `run_psm_q`, surviving for the default column.
Both stages now read the optional `is_transferred` flag and treat acceptance as
a second route through the filter. Lowering the grouped columns instead would
be wrong: they are written to a group's single winning row on purpose, so
writing one onto a transferred loser would double-count that group. A decoy is
still never quantified or reported, whatever its transfer status.
`mbr_worker.py` also wrote both of its outputs with a bare `pq.write_table`,
so under pandas 3 the string columns come out `large_string` and the engine
rejects the file with `column 'peptidoform' is not utf8` -- the exact failure
`scripts/_lib_io.py` exists to prevent and that c7f910c fixed in the four
library helpers while this worker was missed. `_lib_io` gains `narrow_table` /
`write_engine_table` for a producer that already holds an arrow Table, which
this worker does.
191 Rust tests pass; fmt and clippy clean.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…s by base peptide Not promoted from a sensitivity count. Each previous value was wrong on its own terms, and each of these still needs entrapment plus a second acquisition before anyone claims a sensitivity result for it. **`extract.apex_evidence_rank` false -> true.** The legacy signature-intensity apex scores a scan group by the summed OBSERVED intensity of only the top-K PREDICTED fragments. When none of those K is observed at any qualifying scan the score is 0.0 everywhere, the strict `>` never replaces the first candidate, and `groups` is RT-ascending, so the apex silently becomes the LOWEST-RT qualifying scan: up to a full RT window away (150-211 s on the AIF benchmark), or anywhere in the gradient for a candidate with no window row. The RT prior cannot rescue it, because the combination is multiplicative and a zero annihilates the prior in exactly the case the prior exists for. Evidence rank scores `(n_distinct_fragments + tie) * prior`, always positive, so the fallback is unreachable. The wrong apex propagated into `prelim_score`, which decides the pre-FDR competition winner, into `rt_error_abs` and `log_apex_intensity`, and into quant's integration centre. **`extract.min_frag_corr` 0.2 -> 0.6.** 0.2 is the optimum measured for `nn_torch`; 0.6 is the documented optimum for `native_tda`, which is the default classifier. The shipped gate was tuned for a rescorer the shipped configuration does not use. All three `configs/examples/*.json` set the key explicitly, so the validated nn_torch workflow is unchanged; only a config that says nothing moves, and one that says nothing is using the native rescorer. `native.json` moves to 0.6 to match its own rescorer. **`features.emit_pin` true -> false.** `features.rs` states outright that no MuMDIA stage consumes the file, because rescore builds its own PIN. It is a ~5.4 GB text write per run on a real library, so a 40-run experiment wrote hundreds of GB nothing read back, by default. ## CV folds now pair a target with its decoy `nn_rescore_worker.py` derived its fold from `md5(strip_pep(Peptide))`, and `strip_pep` removes bracketed mods and flanking residues but not the `DECOY_` marker, so `DECOY_PEPTIDE` and `PEPTIDE` hashed into different folds -- while `percolator_lite` keys on `base_peptide_id` and pairs them, and docs/11 claimed the two use "the same CV-fold scheme". `nn_torch` is the validated production rescorer. Stripping the marker would have fixed only a shift-decoy library. A reverse decoy's peptidoform is the reversed sequence (`make_reverse_decoys.py`), so no string derived from it can reach its target. `base_peptide_id` is the pairing both builders preserve, so the engine now writes it per PIN row to `rescore_<tag>.foldkeys.parquet` and names the file in `MUMDIA_NN_FOLD_KEYS`. By environment variable rather than argv because the sidecar contract is positional and shared with `mokapot_worker.py`, and the NN hyperparameters already travel that way; a worker that ignores it keeps its old behaviour. All three worker backends (parquet in-memory, TSV in-memory, streaming chunks) go through one `folds_for` helper, so the pairing cannot hold on some paths only, and the helper is module-level so it is unit-testable without torch: two new tests run in CI, in a file where all five existing tests skip. The direction of the old behaviour was conservative -- the semi-supervised loop uses all decoys as negatives but only confident targets as positives, so cross-fold memorisation more often depressed a paired target than inflated a decoy -- so this is a sensitivity and reproducibility fix, not an FDR-validity one. Re-measure the 1% count. ## Verification Smoke test passes with the new defaults: 151 accepted at 1% peptide q, 99.3% planted-peptide recovery, 0 decoys, unchanged from before. The recorded cross-platform output hash necessarily moves; the CI job compares the two platforms against each other rather than a golden value, so it still holds. 193 Rust tests, 61 Python tests (12 honest skips), fmt and clippy clean, both generated references regenerated, 508 doc references resolvable. `CLAUDE.md` gains a "Defaults promoted on correctness grounds" section and its extraction-gate section now states which rescorer the default is matched to. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
BREAKING. A tag freezes these names, and every rename here is free now and a
major-version change later. No compatibility aliases, by decision: an old
invocation or config fails loudly with the offending name and the valid
alternatives listed, which is better than silently doing something else.
## CLI
| was | is | subcommands |
|---|---|---|
| `--library-precursors` | `--lib-precursors` | audit, extract, prescan, rt-im-train, search-seed |
| `--library-fragments` | `--lib-fragments` | extract, search-seed |
| `--out-chrom` | `--out-chromatograms` | extract |
| `--scored` | `--psms-scored` | audit, mbr, report |
| `--out-scored` | `--out-psms-scored` | mbr |
| `--psms` | `--psms-extracted` | audit, features, mbr |
| `--seed`, `--seeds` | `--seed-psms` | features, align |
Rationale per row: the library flags now match the orchestrators, which is what
the documented primary workflow uses. Extract WROTE `--out-chrom` while features
and quant READ `--chromatograms`, for the same file. `--scored` and
`--psms-scored` named the same artifact on different subcommands. `--psms`
sat one character from `--psms-scored` while meaning the extracted table.
And `--seed` on `features` named the seed_psms artifact, one letter from an RNG
seed, while two other subcommands spelled the same artifact two other ways.
The mbr SIDECAR argv (`--out-scored`, `--seed` in `sidecar.rs`) is unchanged:
those are `mbr_worker.py`'s own flags, not the engine's.
## Config
- `extract.min_frag_corr` -> `extract.gate_min_score`. It was a correlation
under none of the four `gate_mode` values as documented: under the default
`apex_pearson` it is an intensity correlation at ONE apex scan, not a
chromatographic co-elution correlation, and under `spectral_entropy` it is
not a correlation at all.
- `compete.group_by = "precursor"` -> `"base_peptide"`. `compete.rs:88` keys the
group on `base_peptide_id`, which comes from the stripped sequence, so every
charge state and every modification variant of one peptide collapses to a
single winner before FDR. `peptidoform_charge` is the genuine precursor unit
and remains required for a PTM search.
Verified: all five shipped configs still parse, and an old key fails with
`unknown field 'min_frag_corr', expected one of ... 'gate_min_score' ...` /
`unknown variant 'precursor', expected one of 'base_peptide', ...`.
Untracked local configs need the same two edits:
sed -i 's/"min_frag_corr"/"gate_min_score"/; s/"group_by": *"precursor"/"group_by": "base_peptide"/' config.*.json
## `report` gains `--config`
It could not receive `quant.q_threshold`, so a config setting 0.05 yielded 0.05
from `run` and 0.01 from `report` on the same scored table, silently. `--q` now
defaults to the config value when `--config` is given, and an explicit `--q`
still wins.
## Not renamed, and why
`--out` (11 subcommands) versus `--out-dir` (4) stays: the distinction is a
single file versus a directory, which is meaningful. The ten bespoke `--out-*`
flags each name a distinct artifact of a stage that writes several.
## Verification
Smoke test passes end to end on the renamed interface (112 assertions, 151
accepted, 99.3% recovery). 190 Rust tests, 61 Python tests, fmt and clippy
clean. Both generated references regenerated; 508 doc references resolvable.
Documentation updated for the renames and for the three promoted defaults,
including the stated gate default in eight places, the two "is a misnomer"
passages that are now rename records, and `docs/09`'s gated-features table.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Nine robustness findings from docs/25 section 6. None changes a result: the smoke
test's output hash is unchanged.
**A truncated mzML is now an error.** `convert` iterated `Spectrum`, not
`Result<Spectrum>`, so a parse error part-way through simply ended the iterator:
truncating the fixture to 40% previously produced a complete-looking artifact set
covering the first third of the gradient at exit 0, and the user would attribute
the missing identifications to their sample. Verified: 300 declared, 119 read,
now exit 1 naming the fraction. Deliberately reads `spectrumList count` from the
header rather than `SpectrumSource::len()`, because in an indexedmzML the index
is at the END of the file, so exactly this truncation makes the index read 0.
**Zero MS2 is now an error.** Never a legitimate DIA run, and the shape every
later stage tolerates in silence: search-seed finds nothing, extract runs the
whole library against nothing, report writes a header-only TSV, exit 0.
**Diagnostics moved to stderr, ANSI only on a terminal.** Every log line went to
stdout with escapes attached whatever the destination, so a run's log and its
result summary landed in one file that no redirection could separate, log files
contained `^[[2m...^[[0m`, and `mumdia doctor 2>/dev/null` still printed its
whole report. `2>/dev/null` now yields the answer and `1>/dev/null` the log.
**Artifact writes are atomic.** All four writers (`write_table`, `BatchWriter`,
`write_batches`, `write_json`) go to `<path>.tmp-<pid>` and rename. Previously
`File::create` truncated the final path on open, so a rerun destroyed the
previous good artifact before producing a replacement, and Ctrl-C left rubble
under the canonical name. A killed run now leaves at most a recognisable temp
file, which `Drop` removes on any error return or unwind.
**`run-experiment` can no longer consume a previous experiment's MBR table.** The
stale `scored_mbr.parquet` is removed before the worker runs, so the `exists()`
check afterwards is a statement about this run. Before, a rerun into the same
--out-dir that accepted transfers last time and none this time fed the OLD scored
table to the split and to every per-run quant. It joins cleanly, because
candidate_id and source are stable across reruns of the same library, so the
result was plausible quantities from the wrong data, logged as "MBR transfers
applied".
**`--run-names` is validated.** A count mismatch was silently replaced by
`r0..rN-1` and duplicates were accepted outright. Duplicates are the dangerous
half: two runs then share an output directory, so above one parallel run two
stages concurrently write the same four artifacts and interleave two runs with no
error. Verified: both cases now fail with the reason.
**Writing an output over its own input is refused.** There was no path-equality
check anywhere, so `prescan --lib-precursors lib/x.parquet --out lib/x.parquet`
replaced a full precursor library with a two-column survivors table and exited 0,
because the library had already been read. `align` had the same shape. Compares
canonicalised paths, so `./x`, a symlink and a drive-letter difference are all
caught. Verified byte-for-byte: the library is untouched and the run fails.
**Three sidecar handoffs are PID-qualified.** `ms2pip_{in,out}`,
`deeplc_{in,out}` and `entrapment_{in,out}` had fixed names, and the readback key
is a row index into each process's own request, so two concurrent runs over
tables of the same row count -- the most likely reason to run two at once --
swapped each other's results rather than erroring. `align_sidecar_scores` catches
a coverage mismatch, not an equal-length swap. The PIN/NN path was already
PID-qualified for this exact reason.
**Paired array lengths from the file are checked.** `convert` decoded m/z and
intensity independently, tolerated either decode failing, then took the loop
bound from one and indexed the other: a profile spectrum with >= 3 m/z values and
a short intensity array panicked on the first iteration. `spectra.rs:68` already
had the `.min()` guard.
Smaller, same commit:
- `convert`'s four peak columns move to `LargeListF32`. A 32-bit arrow list
offset saturates above 2^31-1 values and `finish()` unwraps the `None`, so the
panic is inside arrow with no useful message. `extract.rs` migrated for this
reason already; a long Astral or timsTOF run is within 2x of the limit.
- `align.rs` sorts its shared-peptide set before the LOESS fit. Observed RTs tie
exactly within one MS2 scan, and tied x values keep input order through the
k-nearest window, so HashMap order could move the fitted curve. The one place
in the workspace that violated the documented ordering convention.
- `quant` asserts the candidate id when zipping `integrated` against
`cand_rows`. The pairing is positional and correct, but the comment justifying
it named the wrong reason, and a divergence would attach one candidate's areas
to another's rows with no error.
- `rescore.percolator_bin` warns. It was the only silently inert config field:
accepted, never read, in no document, while under `deny_unknown_fields` a user
reasonably reads an accepted key as an honoured one.
- `Table` remembers its path, so a missing column names the artifact instead of
dumping up to 390 column names with no indication of which file. A long list is
now truncated to 24 plus a count.
- the `run-experiment` interpreter-preflight message no longer contains a raw
newline and 21 spaces.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…rminism check ## The mass model had no external anchor The only one in the repository was a single test pinning PEPTIDE's NEUTRAL monoisotopic mass at 1e-3 Da. Neutral, so `PROTON` never entered it; and 1e-3 Da cannot distinguish the proton mass (1.007276466812) from the hydrogen-atom value (1.007825035), a 0.55 mDa difference that is the classic error in this field and one `constants.rs` explicitly calls out. Beyond it: no test referenced `residue_mass` or pinned an individual residue, none asserted a fragment m/z at any charge, `WATER`/`AMMONIA`/`ISOTOPE_SPACING` were unpinned, and the two other mass tests were self-consistency checks that pass unchanged if every residue mass is wrong by the same amount. This class is invisible to the end-to-end test BY CONSTRUCTION: `ci/make_fixture_mzml.py` plants its peaks by reading the engine's own library, so the fixture agrees with the mass model however wrong the model is. Not hypothetical either -- `ISOTOPE_SPACING` once shipped 485 ppm wrong and was caught by a human reading code. Five tests, decomposed so each half is anchored independently: - the particle and molecule constants against CODATA/AME, including an explicit assertion that PROTON and the H atom differ by one electron mass; - all twenty residue masses against published tables, plus L == I, the 36.4 mDa K/Q gap that a coarse tolerance would collapse, and the six ambiguous residues staying unmassed; - precursor m/z at charge 1, 2 and 3, which is where PROTON enters and where the neutral-only anchor did not reach; - the b and y series at charge 1 and 2, which pins the series algebra: which residues enter each ordinal, where the terminal water goes, and how charge enters both numerator and denominator; - two modification deltas as absolute values, so a wrong table entry cannot agree with itself. Tolerances are 1e-6 Da absolute, about 4500x tighter than the proton/H-atom gap being guarded. My first draft of the reference table was wrong in four places and the tests caught it, which is the point. Verified discriminating: substituting the hydrogen-atom mass for PROTON -- the exact historical error -- fails 3 tests across the workspace, including the precursor-m/z one. Restoring it returns all 36 to green. ## The determinism check now covers every artifact It compared the two runs' `peptides.tsv` only, which is presentation-rounded, so a nondeterministic quantity differing in the fifth significant figure was invisible, and `features.parquet`, `chromatograms.parquet` and `psms_scored.parquet` were never compared at all. Both runs already wrote a manifest with a blake3 `content_hash` per artifact, so the fix was to compare those maps. Measured on the fixture: all 18 artifacts are byte-identical between the two runs, so the stronger claim holds as stated. ## Quant's apex source is asserted `quant` records `apex_rt_column_present` and warns when false, because without it every quantity is integrated around a re-detected apex that reproduces the identification apex only about half the time. Nothing read that report, so a schema change dropping `apex_rt` would have made every quantity wrong while every other assertion stayed green. Smoke test: 112 -> 117 assertions. Also: the four tests using a fixed, non-unique temp directory are now process-qualified, matching the convention docs/14 states and four other test modules already follow. Two concurrent `cargo test` runs on one machine no longer race. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
… supply chain
## `unsafe` is now forbidden, not merely absent
The workspace has zero `unsafe` today, including `build.rs` and not even in a
comment. `[workspace.lints.rust] unsafe_code = "forbid"`, inherited by all three
crates, makes that a property rather than a coincidence. `forbid` rather than
`deny`, so a crate cannot opt back in with an `#[allow]` -- verified: adding
`#[allow(unsafe_code)] unsafe { .. }` now fails with
`allow(unsafe_code) incompatible with previous forbid`. A tool that parses hostile
files for a living gets much of its safety argument from this one line, and the
argument is only worth stating if something enforces it.
## The binary carries 173 crates and shipped no notice for any of them
The tracked tree had `LICENSE` (MuMDIA's own Apache-2.0) and nothing else, while
the release archive and the container image both ship a statically linked binary.
Apache-2.0 section 4(d) requires propagating the NOTICE contents of redistributed
works, and MIT requires the copyright notice in all copies.
`ci/gen_third_party_licenses.py` writes `THIRD_PARTY_LICENSES.md` from
`Cargo.lock`: every crate with its version, SPDX expression and repository, an
obligation summary, and the licence texts for the families that are not
Apache-2.0 (which `LICENSE` already reproduces verbatim). Generated with a CI
`--check`, so a dependency change that alters the inventory has to land with the
regenerated file, and generated from `cargo metadata` rather than `cargo about`
so contributors need no extra tool. All 173 crates declare an SPDX expression,
none unspecified, so the inventory is exact.
Two things the audit surfaced, now written down rather than assumed: `r-efi` is
`MIT OR Apache-2.0 OR LGPL-2.1-or-later`, disjunctive, so the permissive arm
applies and no crate imposes a copyleft obligation on the distributed binary --
the relied-on arm is recorded per crate. And `snap`, the snappy compressor every
artifact write goes through, is BSD-3-Clause with no alternative, so that text is
a real obligation and is included.
The file ships in the release archive and at `/opt/mumdia/` in the image.
## Supply chain
- **All 23 action references are pinned to commit SHAs**, each resolved against
upstream and annotated with its tag. `softprops/action-gh-release` mattered
most: the only third-party action, holding `contents: write`, receiving the
release token and the built archives, pinned to a mutable `v2`. Dependabot
already covers `github-actions`, so the SHAs move on a reviewed PR instead of
silently.
- **`build-essential` is purged in the same layer it is installed in.** It went
into the final stage so pip could compile an sdist and was never removed, so
the published image shipped gcc, make and the C headers for a job that is one
static binary and two prebuilt conda envs. Same layer, so the bytes never reach
the image rather than being deleted from a later one.
- **`release.yml` passes `github.ref_name` through `env:`** instead of
interpolating it into a `run:` line. Ref names may contain `;`, backticks, `$`,
`&` and `|`. Pushing a tag needs write access, so this is defence in depth.
- **`.dockerignore` is now an allowlist.** A deny list has to be updated whenever
something new appears, and what appears is exactly what nobody excluded: it was
already missing `scripts/run_*.sh` and `scripts/_probe*.py`, which `.gitignore`
excludes as throwaway with hardcoded paths, and which a LOCAL image build would
have baked in, because the build context is the working tree rather than the git
tree. CI builds from a clean checkout, so the published image was never
affected. Adding a COPY now requires adding it here, which is the failure
direction we want.
- **Real sidecar imports run on PRs that touch them.** Every torch, mokapot and
DeepLC test in `tests/python` skips on the runner, so the entire
external-classifier contract that `rescore.strict` exists to protect executed
nowhere on a pull request -- and that is where the worst defect in this
project's history lived, when mokapot returned a targets-only table and the
engine reported ~331,000 "identified" peptides against a correct figure near
51,000. The new job builds both conda envs and imports DeepLC BEFORE
numpy/pyarrow, which is the contract that prevents the `WinError 1114` torch
DLL failure that has shipped once. Gated on a diff touching `scripts/`, `env/`,
`tests/python/` or the Dockerfile, so a documentation PR does not pay the conda
solve.
- **One contributor's absolute Windows path is out of `ci/`.** Both binary
discovery lists now read `MUMDIA_BIN` from the environment.
- **`docs/22` and `docs/25` are excluded from the release archive.** They are
working documents about getting to the release: which gates were failing, what
was blocked, unpublished benchmark numbers. Useful in the repository, odd inside
the archive of the release they plan.
CI is now seven jobs.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
… the rescore matrix visible Two unbounded-memory findings from docs/25 section 6. Neither changes a result: the smoke output hash is unchanged. **Seed scratch is sized by the median isolation window, not the widest.** `convert.rs` maps BOTH a zero-width reported isolation window and a missing precursor to the full range `(0, 1e6)`, which `candidate_range` resolves to every candidate in the library. The scratch was sized by `.max()` over group widths, so one malformed or all-ion scan in an otherwise 50-window run sized EVERY rayon worker's scratch to the whole library: 877 MB per worker on the profiled 54.8M-candidate library, roughly 28 GB of commit charge on 32 cores, for arrays a worker inside one narrow window never touches. The only mitigation was `--threads`, which the user had to know to reach for. The arrays grow on demand, which the sizing comment already noted makes an underestimate safe, so the max was strictly the wrong statistic. The median is robust to that one outlier, needs no configuration, and is deterministic because the widths are collected in group order. When the widest window exceeds 8x the median the stage now says so, naming both counts and the library size: an all-ion acquisition does that legitimately, and anything else is a scan whose isolation window convert could not read. **The rescore feature matrix reports its size, and can be bounded.** `feats` is `Vec<Vec<f64>>`: eight bytes per value plus a heap allocation and a 24-byte spine entry per PSM. That is twice the width `CLAUDE.md` documented, because that figure describes the *Python* worker's f32 matrix, and the code's own comment puts the Rust one at ~27 GB on an experiment-wide pool. `native_tda` then runs all folds in parallel, each holding an owned standardised copy of its training slice, so the true peak is roughly `(1 + folds)x`. Nothing in the workspace estimates or checks available memory -- there is deliberately no `sysinfo` dependency -- so the failure mode was an OS kill after however long the run took to get there, with no number to plan against. The size is now logged before the allocation, in adaptive units (463 KiB on the fixture; the same code prints GiB on a real pool). `rescore.max_feature_matrix_gib` turns exceeding a ceiling into an error at startup that names the estimate and both ways out, and defaults to 0 (no ceiling), so behaviour is unchanged unless asked for. Deliberately not a batching implementation. Sub-batching changes which PSMs share a pooled `q_value` -- `run_psm_q` is per source and unaffected, but the pooled column is what `run-experiment` gates quant on -- so it is the operator's decision, and the error message says so rather than making it silently. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…tatement From docs/25 section 9. In a project whose selling point is code-grounded citation, a wrong claim is a defect, and every one of these read as reassurance. **Claims that were simply false.** - README, `docs/19` and `docs/20` said the DeepLC fine-tune is unseeded. It takes `--seed`, wired from `rng_seed`, and seeds both numpy and torch. The honest statement is that it is seeded and still not bit-for-bit reproducible, because torch's training kernels are not. - `CONTRIBUTING.md` said there is "no checked-in mzML or Parquet fixture and no end-to-end smoke test". Both exist and run in CI. It now says how to run it, what its 117 assertions cover, and -- from the review's section 8 -- what it structurally cannot see: FASTA mode only, synthetic noise, and 3,820 candidates that put the FDR pseudocount in charge. - `CONTRIBUTING.md`'s gate list omitted four CI-enforced gates: pytest and the three generator `--check`s. - `CHANGELOG.md` listed "interpreters by absolute path, so it is not portable" as a known limitation of a release whose own entry 128 lines above removes it, and claimed no CI job exercises the sidecar path when one now imports DeepLC, mokapot and MS2PIP in real environments. - README described `run.pin` as the feature matrix "as fed to the rescorer". It is a Percolator-style export of `features.parquet` for external tools; rescore builds its own PIN from the competed table. Now off by default, and said so. **Claims that overstated what is guaranteed.** - `schema.rs` said artifact versions exist "so a stage can validate its inputs". No code reads a `schema_version` back: all uses are writes. That mattered because `CONTRIBUTING.md` stated a compatibility policy nothing enforced. The module doc now says what the version IS -- provenance -- and that making the check real is a separate change. - README said the generated references "cannot drift". CI pins everything the generator derives, and not the hand-written prose around it, nor a column the generator computes from a heuristic. - `CLAUDE.md` and `docs/11` called pooled FDR "scale-invariant" and sub-batching "statistically free". The direction is right -- the pseudocount makes a larger pool marginally looser, never tighter -- but the floor is exactly `1/T` and does move with the pool: measured on the kernel, replicating a five-row population once takes q from `[0.5, 0.5, 0.667, 0.667, 1.0]` to `[0.25, 0.25, 0.5, 0.5, 0.833]`. So batching is free in `run_psm_q` terms and not in `q_value` terms, which matters because `run-experiment` gates per-run quant on the pooled column. - `CLAUDE.md` and `docs/17` gave the rescore matrix as `n_psms x n_features x 4` bytes. That is the Python worker's f32 matrix; the Rust one is f64 with a heap allocation and 24 bytes of spine per PSM, and `native_tda` holds roughly `(1 + folds)x` at peak. - `CLAUDE.md` and `docs/09` decomposed the "62.3% `NO_PEAK_GROUP`" figure as "never assembled `presence_min_fragments` distinct fragments". The code does not support that: `audit.rs` reads a per-candidate audit table `extract` never writes (`emit_candidate_audit` is unwired), so the map is always empty and the catch-all absorbs presence failures, matched-fraction failures and every gate rejection alike. The attribution to the peak cap stands on the replay experiment, which recovered 41,948 of 49,105, not on the label. **A whole feature path that was undocumented.** `docs/12`'s integration section described only `trapezoid` and `trapezoid_window`, with no mention of `fixed_window_s`, `fixed_scan_halfwidth`, `baseline_subtract`, `interference_envelope`, `fragment_selection` or `select_fragment_areas` -- that is, the entire recently added path, including the one the recorded ProteoBench Astral submission used. It now covers all three integration modes in the order the code chooses between them, the two optional transforms, both fragment rankings, the MBR-transfer route through the filter, and why the two fixed-window forms are not interchangeable. **`docs/11`'s CV-fold claim is now true rather than corrected**, since the code changed to match it, and the section explains the mechanism and why the previous behaviour was conservative in direction. Two code changes, both making a silent no-op audible: - `extract` warns when `extract.frag_tol_ppm` is overridden by the learned tolerance from mass calibration. Search-seed ALWAYS writes that key, including in its calibration-failure branch, and both orchestrators always pass `--mass-cal`, so the config field had no effect in any orchestrated run and nothing said so. - search-seed collects mass calibrants within `max(50 ppm, fragment_tol_ppm)` rather than a literal 50. A TOF run searched at 100 ppm could not observe a deviation beyond 50, so the p95 that SETS the learned tolerance was bounded by the collection window itself and came out too tight. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…tory **A planted `./scripts/` could win resolution.** The shipped default `sidecar_script_dir` is the relative `"scripts"`, which both sidecar example configs carry literally, and resolution tried the working directory FIRST. So: unpack a dataset archive, `cd` into it, run `mumdia run --config configs/examples/diann-library.json`, and if that archive contains a `scripts/` directory holding a worker file, the whole directory won and every worker in it ran as the user. That needs no hostile CONFIGURATION -- which `SECURITY.md` treats as trusted, like a shell script -- only an untrusted input directory. New order: an absolute configured path as given, then the config file's directory, then the executable's, then the working directory with a warning. Config-relative first is also the more useful order, since it makes a config portable with its scripts, which is what the resolution exists for. The Docker configs were already correct with an absolute `/opt/mumdia/scripts`. **`load_ms1` truncates its paired lists.** The m/z and intensity list columns are decoded independently and either can be null, so a spectra artifact whose lists disagree in length was carried into `sum_near` and the MS1 isotope features, where the loop bound comes from one array and the body indexes the other. `load_ms2` already had this guard. **The pushed container image is now the image that was tested.** `docker.yml`'s first build step comments that it builds once "so the smoke test runs against exactly the image that will be pushed", and then a second `build-push-action` invocation built again. Neither `FROM` is digest-pinned and the apt/pip resolutions are unpinned, so the two builds agreed only to the extent the layer cache made them: the image that passed the uid check, both `doctor` runs, the DeepLC and mokapot import checks and the bind-mount write was not provably what users pulled. Now `docker tag` + `docker push` on the loaded `mumdia:ci`, so it is the same image id by construction, and the step logs that id. **Two Python pins.** `ms2pip` moves from the `4.0.0.dev9` pre-release to `4.0.0` final: a dev build in a published image can be yanked and carries no stability promise, and this is the released version of exactly what was pinned, so it is the smallest correct change (4.1.x and 4.2.0 exist and are a separate, testable upgrade). `mokapot` was pinned in the Docker env and floating in the local one, so a local install and the image could run different rescorer versions -- exactly the difference a reported benchmark cannot see. Both, and DeepLC 4.1.1, verified to exist on PyPI. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`docs/25` gains a status section at the top: all four blockers closed, what else closed, and a table of what is deliberately still open with the reason for each -- signal handling (atomic writes removed the substantive harm), the incremental manifest, transparent rescore batching (an operator decision, now bounded rather than silent), the `convert`/`align`/`run` unit-test gap, `bench/README.md` and four `docs/19` citations, and the science that needs real data. The report reads as a record of the audit rather than a to-do list. It also flags the two things this work introduced rather than fixed: three changed defaults plus the NnTorch fold-key change mean `nn_torch` counts and every entrapment number need re-measuring, and the renames have no aliases. `CHANGELOG.md` gains the same two warnings up front, with the two-substitution `sed` line for local configs, so the next person reading it before pulling knows what moved before they wonder why their config stopped parsing. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…he mismatch CI caught a bug I introduced and my local verification did not. Both halves are worth fixing, and the second is the reason the first got through. **The reader could not read what the writer now writes.** Moving `convert`'s four peak columns to `Col::LargeListF32` (64-bit offsets, because a 32-bit arrow list offset saturates above 2^31-1 values and the builder then unwraps a `None` deep inside arrow) left `spectra.rs` downcasting to `ListArray`, so every run failed immediately after convert with `Error: column 'mz' is not a list`. `FloatList` now accepts either width. That is the right shape independent of this bug: a spectra artifact written by an earlier version carries `List`, and there is no reason to refuse it. **Why the local smoke test passed anyway.** `ci/smoke.sh` searched `rust/mumdia/target/release/` before anything else, and this machine redirects `build.target-dir` off the synced tree, so the in-tree path held a binary built before the change. Convert and spectra were consistent in that binary, so the suite went green on code that was not the code under test. Removing my own absolute path from the discovery list in an earlier commit is what left that as the first candidate. The list now starts from the cargo-reported target directory, and `cargo metadata` runs from INSIDE `rust/mumdia` rather than with `--manifest-path` from the repo root: cargo discovers `.cargo/config.toml` by walking up from the current directory, not from the manifest, so from the root it reports the in-tree `target/` and misses the redirect entirely -- which is the same failure again, one level up. Verified: discovery now resolves to `C:/Users/robbi/mumdia_build/release/mumdia`, and the smoke test passes against it (117 assertions, 18 artifacts byte-identical). **The sidecar-import job needed pytest.** The import contract itself passed on the first run -- `deeplc 4.1.1 | torch 2.12.1+cpu`, in the DeepLC-before-numpy order that prevents the `WinError 1114` DLL failure -- and then `python -m pytest` failed with `No module named pytest`, because pytest is not in the runtime environment spec and should not be. Installed in the job instead. The point of running the suite there is that the torch- and DeepLC-dependent tests execute rather than skip, so `-rs` should report far fewer skips than the `sidecars` job. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…ace itself
The sidecar-import job's actual work passed on the previous run -- `deeplc 4.1.1 |
torch 2.12.1+cpu` in the DeepLC-before-numpy order, `mokapot 0.10.0`, and 66
passed / 7 skipped against 61 / 12 on a bare runner, which is the whole point of
running the suite in a real environment -- and then the job failed in post-job
cleanup with:
ENOENT: no such file or directory,
lstat '/home/runner/work/_temp/setup-micromamba/micromamba-shell'
`setup-micromamba` registers that shell wrapper and removes it in its own
post-job cleanup, so calling the action twice in one job leaves two cleanups
racing for one file: the first deletes it, the second fails. Failing after every
check has passed is the most misleading way for a job to fail, so this is worth
fixing rather than tolerating.
Now a two-entry matrix, one environment each, with the import line carried in the
matrix and passed through `env:` rather than interpolated into the `run:`. The two
environments also now build in parallel instead of in sequence.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…k arms
## A regression the benchmark found, not a test
Moving interpreter resolution into `load_config` fixed the blocker where standalone
stages took `"auto"` literally, and introduced a new one: `Role::required_by`
answers "does this CONFIGURATION use the role", which is not "does the subcommand I
am about to run use it".
So `mumdia search-seed` with `rescore.classifier = entrapment` failed at CONFIG
LOAD, demanding a mokapot interpreter for a rescorer that stage never invokes:
Error: rescore.python is required by this configuration but no usable
interpreter was found.
`resolve_with(cfg, Strictness)` splits the two cases. The orchestrators keep
`Require`, where failing in preflight costs seconds instead of the hours already
spent by the time rescore is reached. `load_config` uses `BestEffort`: resolve what
resolves, warn about the rest, and leave an unresolvable role as `None` rather than
as the literal `"auto"` -- so a stage that DOES need it still fails on the missing
interpreter, not on trying to execute a program called `auto`, which is the
confusion the whole mechanism exists to remove. Two regression tests pin both
directions.
## Benchmarks, recorded in bench/README.md with row unit and q column
**Identification, `LFQ_Orbitrap_AIF_Ecoli_01.mzML`, augmented library, DeepLC
fine-tune, nn_torch:** 10,914 stripped peptides at `peptide_q_value <= 0.01`, with
an empirical decoy fraction of 0.0098 on every q column. 1.000 (peptidoform,
charge) per stripped peptide, because the default `base_peptide` competition key
collapses siblings before FDR.
**Two corrections the run forced.** The file is not all-ion: 152 contiguous 4 m/z
windows over 396.4-1004.7 m/z. And the 300-peak cap is nearly inert on it, 4.7% of
MS2 spectra saturating against the 47.8% `CLAUDE.md` records for "the chimeric AIF
benchmark run". Recorded as a correction rather than quietly adopted, because both
figures are load-bearing for the peak-cap guidance.
**A/B on `apex_evidence_rank`, both arms sharing one set of upstream artifacts** so
the apex change is isolated from DeepLC draw variance: identification is flat
(10,914 vs 10,921, 0.06%, 96.6% of the union shared), so the promotion neither
gains nor costs sensitivity and was not promoted as a sensitivity change. But of
the peptides both arms identify, 48.3% have an apex more than 1 s apart -- median
2.9 s, max 104.8 s -- and quant integrates around that apex, so quantities from
before this change are not comparable with quantities after. Scored against the
shared window table's `rt_pred_cal`, evidence rank is closer at every percentile,
consistently but by little, on a weak instrument.
**Entrapment, E. coli + human 1:1, native linear entrapment rescorer:** empirical
FDP `(0.560632 x 12 + 1) / 786` = 0.0098 at a 1% threshold. Calibrated.
The three section-3 fixes are each visible in it: 0 decoys accepted on
`peptide_q_value` against 15 on `q_value`, which is the grouped-q exclusion working;
entrapment median score -4.701 sitting on the decoy median -4.693 with the top 500
rows 496 real / 2 entrapment / 2 decoy, which is a held-out negative class rather
than a recruited positive; and the ratio measured from the built library (0.560632,
not the 1.0 a 1:1 protein count would suggest, because human proteins are longer)
and now rejected if non-positive.
Building it surfaced one trap worth the record: `fasta/ecoli_22032024.fasta` already
bundles 358 contaminants with a `Cont_` prefix, 150 of them `_HUMAN`, so the obvious
`_HUMAN` marker would have scored real contaminants as entrapment negatives and
inflated the measured FDR. Explicit `REAL_`/`ENTRAP_` prefixes instead.
Also corrects the "scale-invariant" pooling claim in docs/20, which the earlier
documentation pass fixed in CLAUDE.md and docs/11 and missed here.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Release-readiness work for
v0.1.0, following the plan indocs/22_release_plan.md. The goal was to make theengine installable, portable, tested and documented, without promoting any new
scientific default.
Nothing here changes results. The end-to-end smoke test produces byte-identical
output before and after every commit in this branch, and the quantification
refactor was checked against the exact output behind the ProteoBench submission:
72,168 quantified precursors, every
quantity,n_fragments_used,quant_statusandintegration_apex_rtbit-identical.Why the branch was not ready
Measured at the start, on a branch that was being treated as done:
cargo fmt --checkfailed,cargo clippy -- -D warningsfailed on two deadfunctions, and one of 122 tests failed. CI ran only build and test, so none of
it was visible.
predicted_intensitya requiredchromatogram column, which made every artifact written before that column
existed unquantifiable.
and it was the config
CLAUDE.md,docs/19anddocs/20told everyone to run.What this changes
Correctness fixes
q_valuewhile quantification gates onrun_psm_q, so 34,280 of 34,664accepted transfers on a six-run experiment were silently dropped and
match-between-runs appeared to run and change almost nothing.
helpers wrote parquet through
DataFrame.to_parquet, which on pandas 3 emitslarge_string; the engine rejects that withcolumn 'peptidoform' is not utf8.Version-dependent, so it worked for its author and would fail for the next
person. Found by the new Python contract tests, fixed in
scripts/_lib_io.py.Under a fixed quantification window the engine reported the descent-walk bounds
it had ignored. Corrected, and verified numerically neutral on real data.
quantreadspredicted_intensityoptionally again, so older artifacts load.Portability
"auto". Resolution walksMUMDIA_PYTHON_<ROLE>,MUMDIA_PYTHON,CONDA_PREFIX,VIRTUAL_ENV, thenpython3/pythononPATH, and accepts a candidate only after it imports whatthat role's workers import, so discovery cannot pick a Python without torch and
defer the failure to the rescore stage hours later. A role is resolved only when
the configuration uses it, so a default native run still needs no Python.
predict_frag.sidecar_script_dirresolves against the config file and theexecutable, not only the working directory, which used to change silently which
worker scripts ran.
configs/examples/{native,fasta-sidecars,diann-library}.jsonreplace themachine-specific config, with
configs/README.md.mumdia doctornow answers "can this configuration run?": the resolvedinterpreter and how it was found, package versions, whether the worker scripts
are present, and a warning below the DeepLC 4.1.1 floor. It covers
mbr.pythonand the script directory, which it never checked.
--threads(forwarded to the sidecars),--log-level,-v,-q.Thread count could not be bounded at all before; the NN worker measures faster
on 8 threads than 32.
Testing
generated fixture. Nothing binary is committed and nothing is fetched;
ci/make_fixture_mzml.pyreads the library the engine itself just built andplants exactly those m/z values, so the fixture cannot disagree with the mass
model. First coverage of mzML parsing, the library build, the
runorchestrator, the manifest, RT calibration and the report writers. Measured
99.3% planted-peptide recovery and zero decoys at 1% peptide q.
and nothing enforced. A CI job now diffs the two platforms' output hashes. No
golden value is committed, so the check does not need updating when scoring
legitimately changes.
above, out-of-fold coverage for the rescorers, the library encoding invariants,
and the DeepLC-before-numpy import order. Tests needing torch, mokapot, DeepLC
or MS2PIP skip rather than fail; each was verified to pass in an environment
that has them.
reportmust not emit a decoy even when one has the best qand the best score; an empty chromatogram table must preserve the
identification as unquantifiable; a one-sided target/decoy population is
refused.
-D warnings, tests, Pythoncompile, tracked-JSON and env-YAML parsing, the sidecar tests, both generated
reference freshness checks, and the documentation-reference check.
Provenance
manifest.jsonrecords the short commit with a-dirtymarker, the commit date,the full command line, and a blake3 hash of every input. It previously said only
0.1.0, which every build says, so a result could not be tied to the code thatproduced it.
deeplc_finetune.pytakes--seed; unseeded, the draw moved theheld-out RT window p95 by 150 to 211 s across two draws of one benchmark arm.
Packaging
Release archives are a working installation (binary,
scripts/,env/,docs/,configs/, checksums) across four targets, each smoke-tested on its ownarchitecture. The container image is built, run and checked before any push, runs
unprivileged, and pins DeepLC 4.1.1. All verified on a Linux host rather than
assumed: image 4.62 GB,
doctorgreen on both baked configs, DeepLC 4.1.1 withtorch 2.12.1+cpu and numpy 2.4.6.
Documentation
README rewritten for users; every benchmark number states its row unit and its
q-value column.
docs/23_cli_reference.mdanddocs/24_config_reference.mdaregenerated from
--helpand fromconfig.rsand checked for freshness in CI.docs/19no longer documents one machine.docs/15gains the TSV columns.bench/carries the portable scoring path, the recorded results with units, and ameasured resource profile. One README claim, "97 to 98% sequence concordance with
DIA-NN", was removed rather than restated: it is not attributable anywhere in the
tree.
plan.mddeliberately stays untracked: parts of it are comparative working notesthat are not licence-cleared for redistribution, so it stays a local design
history. The 141 dangling citations were redirected to the tracked guide instead,
and CI now blocks new ones.
Verification
Every gate passes locally, and the smoke test passes on both Windows and Linux.
178 Rust tests, 59 Python tests with 12 honest skips, 482 resolvable
documentation references.
This is the first time these five CI jobs run on a GitHub runner. Everything
was checked locally and on a Linux host, but the workflow YAML itself has never
executed, so expect to iterate on this PR rather than on the merge.
Not included
CITATION.cffis deliberately absent: the author list is not derivable from therepository.
--configis still missing onconvert,quant-lfq,inspect,auditandreport; there is no progress reporting; theunwrapaudit is notdone; the smoke job does not cover macOS or the sidecar path; peak memory is not
measured. The second-pass multi-run workflow that produced the ProteoBench
numbers remains prototype shell code, scheduled for
v0.2.0(
docs/22_release_plan.mdWP7), which is why theREADME says this release does not reproduce those figures on its own.
🤖 Generated with Claude Code