docs(adr): ADR-029 — why the frames are numpy, and not xarray - #279
Merged
Conversation
The container choice was never written down. Roughly 9,400 lines of governance prose in this repository and the word "xarray" appeared nowhere in it, so the first question any newcomer asks had no answer — which is how a settled decision gets relitigated by accident. Eight alternatives, each judged against one constraint: every dependency the DAG root takes, every consumer takes, and one consumer is an HTTP API server. Three claims were checked against live package metadata rather than asserted from memory, and two of them came back different from the first draft: - **xarray** declares `pandas>=2.2` as a hard requirement (2026.7.0). That is the whole argument against it — ADR-002's numpy floor exists precisely so a model repo does not transitively install the pandas world, and faoapi #242 is still undoing that on the consumer side. - **torch** 2.13.0 declares the CUDA stack as *unconditional* on Linux — cuda-toolkit, cudnn, cusparselt, nccl, nvshmem, triton — not as an optional extra. Stronger than the draft claimed. The second argument is better still: this platform already migrated away from torch in this exact domain, and the reconcile parity test's own docstring records that it "needs neither torch" because the oracle is a fixture. - **polars** was grouped with pandas and xarray on dependency weight. That was wrong. polars 1.44.0 declares one hard dependency, its own compiled runtime. The ADR now says so explicitly, because leaving a wrong reason in place would have made the right conclusion unciteable. Protocols-only is closed by measurement rather than argument: ADR-028 recorded that `runtime_checkable` checks member presence only, which is how a frame passed as an index for eleven releases. **A live decision, not a closed one.** Each alternative that could return names the evidence that would return it — a C-71 or C-73 OOM receipt makes zarr/dask worth costing *inside* `io/` without touching the frame type; a non-Python consumer makes format-as-contract worth costing. Four alternatives get no trigger, because they lose on grounds evidence does not change. The Consequences section names what declining xarray actually costs: no out-of-core path, no groupby/resample, hand-written alignment, numpy version skew as our problem (C-19/C-24), and ~3,700 lines someone else would have maintained. No code change. CONFORMANCE_FLOOR stays 2.0.0. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
… a wrong claim Review question: "have you also considered CuPy, JAX, Zarr, and/or Dask Array?" No — and the omission was a whole class, not three names. The eight alternatives in the first draft all proposed a DIFFERENT container with a DIFFERENT API. These propose the SAME API with a different backend: dask.array, jax.numpy and cupy are each close enough to numpy that most of this package would port with modest edits. That is a different question and it deserved asking. **One claim in the first draft was simply wrong, and it is struck rather than quietly edited.** The draft said declining xarray costs out-of-core evaluation "via dask". Out-of-core is not xarray's to give: dask 2026.7.1's hard dependencies are click, cloudpickle, fsspec, packaging, partd, pyyaml, toolz and importlib_metadata — all pure-Python, no pandas. So declining xarray costs less than stated, and the real out-of-core decision is a separate and much cheaper one the draft never posed. The revisit trigger now names `dask.array` inside io/ rather than xarray, for exactly that reason. **A filter emerged that no amount of reasoning would have produced.** Three of the four candidates force numpy 2: zarr 3.3.0 numpy>=2 jax 0.11.1 numpy>=2.1, plus scipy cupy-cuda12x numpy<2.6,>=2.0 dask 2026.7.1 none in core <- the only one compatible This package declares numpy>=1.26,<3 and runs a `floor` job at 1.26.4 because consumers pin conservatively (C-19, C-24). Adopting any of the first three moves the platform's floor as a side effect of a container choice. **JAX gets an honest paragraph beyond the floor objection.** Its arrays are immutable natively, so the entire C-66 saga — deferred 2026-06-28, carried unshipped through eleven releases, landed in 2.0.0, and only then after finding that ADR-025's recorded one-liner would have locked the caller's array — would not have existed. That is a real loss and glossing it would have made the ADR less useful, not more persuasive. Also added zarr as a storage-layer option distinct from format-as-contract, since the first draft folded the two together and they are different decisions. 289 -> 380 lines. No code change; CONFORMANCE_FLOOR stays 2.0.0. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Contributor
Author
Amended after review — a missed class, and one claim struck
No. And it was a class, not three names: the original eight alternatives all proposed a different container with a different API. These propose the same API with a different backend, which is a materially different question. One claim in the first draft was wrongThe draft said declining xarray costs out-of-core "via dask". Out-of-core is not xarray's to give. So declining xarray costs less than stated, and the real out-of-core decision is separate and cheaper — one the draft never posed. It is struck in place, not silently edited, and the revisit trigger now names A filter that reasoning would not have produced
The JAX gets an honest paragraphIts arrays are immutable natively. The entire C-66 saga — deferred 2026-06-28, carried unshipped through eleven releases, landed in 2.0.0, and only then after discovering ADR-025's recorded one-liner would have locked the caller's array — would not have existed. Recorded as a real loss rather than glossed. Also split zarr-as-storage from format-as-contract; the draft folded two different decisions together. 289 → 380 lines. |
…ndency
Maintainer review supplied the rationale the draft only half-had, and asked for the
alternatives' footprints to be measured rather than guessed. Both are now in.
**The primary argument was buried.** numpy was chosen because every repo on this
platform already depended on it, so adding the contract added no transitive
dependency at all. That is what the repo separation exists to protect: the model
repos are deliberately allowed wildly different libraries and need not share an
environment, so anything imported by *most* repos must be nearly free — it is the one
place their dependency sets are forced to agree. That now leads the document.
**Weight and coupling are separated, because they are different costs.** Weight is
what a repo ships. Coupling is whether the dependency has a version some repo already
pins for its own reasons — small packages can be expensive this way. pandas is the
clearest case; torch is the rare one where both apply.
**Footprints measured**, resolving each candidate's full transitive closure with
`pip install --dry-run --report` on 3.11 and summing manylinux wheel sizes:
numpy 1 pkg 18.6 MB (baseline, already everywhere)
dask[array] 12 pkgs 21.6 MB +3.0 MB
zarr 5 pkgs 27.7 MB +9.1
xarray 8 pkgs 35.1 MB +16.5 <- drags pandas
pyarrow 1 pkg 53.1 MB +34.5
polars 2 pkgs 59.4 MB +40.8
jax 6 pkgs 151.8 MB +133.2 <- drags scipy
cupy-cuda12x 3 pkgs 166.9 MB +148.3
torch 29 pkgs 3022.8 MB +3004 (162x numpy)
Two corrections follow from the measurement. **xarray is not heavy** — 35 MB — so the
draft's weight argument was the weak one; the real cost is the pandas coupling.
**dask[array] is essentially free** at three megabytes over numpy, cheaper than zarr
and half of xarray, with nothing in its closure another repo is likely to pin. Any
framing that treats adopting dask as expensive is wrong on the evidence, and the ADR
now says its objections are about design, not cost — a distinction that matters,
because a cheap dependency deferred on design can be revisited by changing the design.
**New section: the scaling wall, with numbers.** 10 features x 128 samples x 259,200
cells x 500 months = 165.888e9 cells = 663.6 GB float32. The maintainer's water
adjustment inverted — 80% water means 20% *remains*, so land-only is 33.2e9 cells
(132.7 GB), not 132.7e9. Conclusion unchanged; both are far past RAM.
The per-shard figure is the useful one: one month at 10 features x 128 samples is
1.33 GB and fits, at 50 x 512 it is 26.5 GB and does not. **The wall arrives through
feature and sample count, not months** — months add shards, features multiply every
shard.
On memmap: sufficient longer than the totals suggest, because `(N, S)` row-major with
contiguous samples is exactly what a per-row reduction wants and what C-22's row
blocking already does. It fails on densification (C-71 writes a dense buffer, memmap
helps readers), on compression, and on chunked parallelism. Expected order of
failure: storage volume first, densification second, reduction last — so the first
fix needed is a chunked compressed format in `io/`, not a new frame type.
380 -> 529 lines. No code change; CONFORMANCE_FLOOR stays 2.0.0.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The strongest argument in the document was missing from it.
**The totals were the wrong number to reason from.** A frame is an index plus a value
array and they scale completely differently:
full global, 500 months one month shard
index (time+unit) 2.07 GB (land 0.41 GB) 4.1 MB
values 10f x 128s 663.6 GB 1.33 GB
values 50f x 512s 13.3 TB 26.5 GB
Values are 320x to 6,400x the index, and the index's size does not depend on features
or samples at all — 4.1 MB per month whether the frame carries ten features or a
hundred.
This decides the streaming question. Alignment is what this package structurally does
— searchsorted, reindex, intersect, is_superset_of, cartesian — and every one of them
touches only `(time, unit)`. The operations needing random access need it over data
that **fits in memory at global scale**. What does not fit is read sequentially in
row blocks, which is the access pattern the estimators already use (C-22).
So **streaming needs a different storage layer, not a different array library**. A
lazy array's `.compute()` returns numpy; chunked stores feed numpy rather than
replace it. The frame is the unit of work, not the unit of storage — which is why
this decision survives a dataset three orders of magnitude larger than the one it was
made for.
The consequence is stated rather than left implicit: a single frame can never exceed
memory, because `assert_frame_envelope` asserts `isinstance(values, np.ndarray)` and
ADR-008 requires validation to complete at construction. "One frame holds the whole
global dataset" is off the table permanently, and consumers expecting otherwise
should be told.
**Also added a scope bound.** At 50 features x 512 samples the grid is 13.3 TB (2.65
TB land-only) — past local-disk streaming and into object storage, catalogues and
partitioned network reads, where the binding constraints are shard layout, partition
keys and transfer cost. Infrastructure questions, not array-library questions. The
library choice stops being the decision that matters somewhere before that threshold,
and this ADR says so rather than implying its own scope is unlimited.
540 -> 596 lines. No code change; CONFORMANCE_FLOOR stays 2.0.0.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…t one
Two changes, both about who the document is for.
**Deciders is now named**: Simon Polichinel von der Maase.
**Every trace of the document's own composition is gone.** The text carried
"an earlier draft grouped polars with xarray on weight; that was wrong", a struck
bullet explaining what a previous version mis-attributed, "a class the first draft
missed entirely", and five more of the same shape. Those are interesting to the two
people who watched it being written and noise to every other reader — and every other
reader is reading it for the first time. An ADR that narrates its own revisions
invites the reader to weigh the revisions.
Swept as a class rather than fixed instance by instance, across four patterns:
iteration references (draft/earlier/struck/mis-attributed), dialogue traces (first
and second person, "review", "asked", "deserved asking"), authorial process voice
("JAX deserves one honest paragraph"), and unconverged hedging. Eleven sites plus the
frontmatter. Re-swept after editing; clean.
Each claim now stands on its evidence instead of on its history. The polars paragraph
says polars is light and couples to nothing, rather than confessing that an earlier
version said otherwise. The xarray section says the weight argument is the weak one,
rather than reporting that it was once leaned on. The measured facts and their dates
stay — provenance is not process.
One hedge tightened while sweeping: the pyarrow trigger said adopting it "would
supersede this ADR and probably ADR-013". Not probably — if the frame type became an
Arrow table, the in-memory type and the wire format would be one thing, so ADR-013
goes definitionally. Said that instead.
Verified section by section: 26 sections, each leading with its claim, none
containing leakage from any of the four patterns.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A falsification pass over each section, against the claim that the document contains no hand waving, ambiguity or decision gap. It did. Every finding was in the argument for numpy or in the scaling section — the parts written from reasoning rather than from measurement. The alternatives analysis survived intact. **The primary claim was an unverified universal, and partly circular.** The text said "every repository on this platform already depends on numpy". Checked across all 33 org repos: seven declare no numpy, and two of those — views-postprocessing and views-hydranet — are views-frames consumers whose route to numpy is partly this package itself. "It was already there" is circular for them. The replacement is a stronger fact that happens to be verifiable: **numpy declares an empty requirement set.** Adding it adds exactly one package and nothing else — no transitive closure, no version to reconcile. That is a property of numpy rather than an assumption about any repo, and it holds even where numpy is genuinely absent. Most repos do carry it already; the ADR now says so without the universal. **"The estimators already work in row blocks" was false for the most-used one.** Measured: `collapse.py` and `aggregate.py` have zero row-blocking. `collapse` is a single `reducer(frame.values, axis=-1)` — it materialises the whole array however it was loaded, and it is the operation any streaming story breaks on first. The memmap section now names which estimators block and which do not, and states that the access-pattern argument is reasoned rather than benchmarked, since no such benchmark exists here. **One revisit trigger could never fire.** "The wire format becomes the dominant access path — consumers reading parquet more often than they construct frames" is not observable; nothing on this platform counts that. A trigger nothing can evaluate defers its option forever while appearing to keep it open. Replaced with a condition visible in a consumer's source: a consumer shipping its own reader against the wire contract. **"Alignment only ever touches the index" was ambiguous and, on one reading, false.** Computing an alignment is index-only; applying one gathers from the value buffer, and `reindex_fill` allocates a dense buffer to do it — which is C-71, the very problem the same section calls the streaming blocker. The two are now separated, which also explains why `reindex_fill` is the operation that breaks first. **The 80% water figure was unsourced** and a 4x swing in the headline number turned on it. The sizing is now given across a range of land fractions, with the observation that no row changes the conclusion, and a pointer to PRIO-GRID for anyone who needs a real number. 625 lines. No code change; CONFORMANCE_FLOOR stays 2.0.0. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Records the container decision this repository had never written down: the frames are numpy arrays plus explicit identifier arrays. Two files, 626 insertions, no deletions and no code change.
Why it exists
"Why not xarray" is the first question any consumer asks, and there was no answer anywhere in the repository. A decision that lives only in the shape of the code is one that gets relitigated by accident.
The decision, and the reason
numpy declares an empty requirement set. Adding it to a repository adds exactly one package — about 19 MB — with no transitive closure to resolve and no version to reconcile against anything. That is what "free" means here, and it is a property of numpy rather than an assumption about any particular repo.
That matters because
views-framessits at the root of the platform DAG: every dependency it takes, every consumer takes — and one consumer, views-faoapi, is an HTTP API server. The repo separation exists so model repos can hold wildly different libraries without alignment work; the price is that anything imported by most repos must be nearly free.Ten alternatives, costed rather than argued
Measured by resolving each candidate's full transitive closure and summing manylinux wheel sizes:
dask[array]awkwardzarrxarraypyarrowpolarsjaxcupy-cuda12xtorchPlus two options with no package to weigh: format-as-contract (the schema is the contract — loses because ADR-016's conformance stops being executable) and protocols-only (closed by ADR-028's measurement that
runtime_checkablechecks member presence only, which is how a frame passed as an index for eleven releases).Three candidates — zarr 3, jax, cupy — are filtered by a single measured fact: each requires numpy 2, which would move the platform's declared floor as a side effect of a container choice.
xarray loses on coupling, not weight. At 35 MB it is not heavy; the cost is that pandas is exactly the library a model repo already pins, so a pandas floor at the DAG root forces every repo's pin to satisfy it too.
A live decision, with a stated expiry
The scaling section sizes what the platform expects to hold — 663.6 GB for one global feature set at 10 features × 128 samples, 13.3 TB at 50 × 512 — and reaches a conclusion that keeps numpy:
The index fits; the values do not. At global scale the index is 2.07 GB and the values are 320× to 6,400× larger. Computing an alignment touches only the index; applying one gathers values, and
reindex_fillallocates a dense buffer to do it — which is C-71, and the reason it is the operation that breaks first. So streaming needs a different storage layer, not a different array library:.compute()returns numpy, and the frame is the unit of work rather than the unit of storage.Four revisit triggers, each naming evidence someone could actually observe.
dask[array]is flagged as the one deferred on design alone — it costs three megabytes — which makes it the only entry whose deferral could be undone by changing this package's mind rather than by waiting for the world to change.The ADR also states where it stops applying: past terabyte working sets the binding constraints are shard layout, partition keys and transfer cost, and the right response is a new decision about storage architecture rather than a revision of this one.
What it admits
Consequences → Negativenames the real bill: hand-written alignment, no out-of-core path, nogroupby/resample, numpy version skew as this repo's problem (C-19/C-24), and ~3,700 lines a third-party container would have maintained.Review history
Audited section by section against the claim that it contains no hand waving, ambiguity or decision gap. It did — six findings, all fixed:
collapseandaggregate;collapsematerialises the whole array in one call and is the operation a streaming story breaks on firstEvery finding was in the argument for numpy or in the scaling section — the parts written from reasoning. The measured alternatives analysis survived untouched.
Verification
No code change.
CONFORMANCE_FLOORstays2.0.0.🤖 Generated with Claude Code