Release candidate: the §5.1 ruling, and six guards that were not guarding

.github/workflows/run_pytest.yml

@@ -75,6 +75,29 @@ jobs:
         path: _siblings/views-appwrite
         fetch-depth: 0
 
+    # Fetched for the checks that need only views-datafactory's TRACKED files. The one
+    # that motivated it is `test_delivery_coverage.py::test_manifest_matches_datafactory_
+    # land_minus_land_gaul`, C-30's drift tripwire on the 76-cell exclusion manifest: it
+    # reads `src/datafactory_query/{land,land_gaul}_pgids.json`, both tracked here, and
+    # until 2026-08-17 it ran only on a laptop — while this repository told a partner in
+    # writing that the manifest "cannot drift without failing loudly". It could; nothing
+    # in the gate was watching. Three more checks came with it (the release gate, the
+    # region-set check, the wire-cast dtype check); see C-46 for the full accounting.
+    #
+    # This checkout was tried on 2026-08-03 and reverted, and the revert note said the
+    # sibling could not be fetched because its GAUL parquets are untracked. That was the
+    # wrong diagnosis. `data/raw/gaul_admin/` IS tracked (it holds a geojson); only the
+    # parquets are not — so `test_gaul_lookup_fidelity`'s `.is_dir()` gate passed and the
+    # comparison died on FileNotFoundError. The gate now checks for the seven parquets
+    # themselves, so that half skips honestly here and still runs where they exist.
+    - name: Checkout views-datafactory (sibling)
+      uses: actions/checkout@v3
+      with:
+        repository: views-platform/views-datafactory
+        ref: main
+        path: _siblings/views-datafactory
+        fetch-depth: 0
+
     - name: Set up Python
       uses: actions/setup-python@v4
       with:
@@ -97,6 +120,7 @@ jobs:
     - name: Run tests
       env:
         VIEWS_APPWRITE: ${{ github.workspace }}/_siblings/views-appwrite
+        VIEWS_DATAFACTORY: ${{ github.workspace }}/_siblings/views-datafactory
       run: |
         set -e
         poetry run pytest tests/

docs/ADRs/013_sampled_forecast_wire_contract.md

@@ -661,6 +661,40 @@ geography. **Source:** the sidecar is built from this repo's ADR-011 GAUL lookup
 (`views_postprocessing/data/gaul_lookup.parquet`, area-majority cell→region mapping
 sourced from views-datafactory); the #91 sink leg attaches it per run.
 
+**§5.1a Nullable int64 was considered and rejected** *(clarification 2026-08-17,
+MINOR — no change to the rule, and no `contract_version` bump; this records an
+alternative the 2026-07-19 ruling did not weigh).* The partner has now twice asked
+for the `*_code` columns as integers (#278; views-postprocessing#272), and the
+justification given each time — that an integer column cannot carry a missing
+value — is a property of NumPy-backed pandas, **not** of Parquet. Parquet and Arrow
+both carry nullable integers natively, and this repo's own lookup stores all three
+code columns as `int64` with zero nulls; the float is introduced by our writer
+(`contract/wire/sidecar.py`), not by the source. So the alternative is real and the
+old reason for dismissing it was wrong.
+
+It is rejected anyway, on a measured ground rather than that one. What §5.1 requires
+is a schema that does not depend on the data. Nullable int64 does not deliver that
+at the layer the consumer observes — it relocates the dependence. Measured
+2026-08-17 (pyarrow 23.0.1, pandas 3.0.5): an int64 Parquet column containing **no**
+null reads back as `int64` under a default `pd.read_parquet`, and the same column
+containing **one** null reads back as `float64`. Under float64 the consumer sees one
+dtype always; under nullable int64 they would see `int64` usually and `float64`
+whenever a run happened to contain a missing code — which is the data-dependent
+schema the 2026-07-19 ruling rejected, moved from our writer to their reader. The
+escape (`dtype_backend="numpy_nullable"`) is a consumer-side commitment this repo
+can neither verify nor enforce (cf. C-87, C-92). Independently, faoapi's reader
+`reindex`es the sidecar onto the forecast's gids and then calls `.to_numpy()`, both
+of which return float64 from a nullable integer column — so the change would not
+even reach the consumer as integers.
+
+**Consequence for the partner, and it is the useful half:** because the delivered
+region excludes the GAUL-uncovered cells (`delivery/coverage.py`), no delivered code
+is ever missing — `tests/test_gaul_lookup_fidelity.py::test_lookup_has_no_nulls`
+holds this in CI — so `astype("int64")` on read is lossless for this product. That
+is a property of the delivered **region**, not of the contract: a future region with
+no exclusion list could carry genuinely missing codes, which is exactly why the
+column type stays float64.
+
 **§5.2 Consistency.** The sidecar's cell-id set must equal the forecast's cell-id
 set (views-postprocessing's existing coverage/identity invariants, to be extended
 to the sidecar in the #91 leg — not yet built as of 2026-07-19). The sidecar hash is pinned in **the Hop-B run manifest

docs/CICs/UNFAOPostProcessorManager.md

@@ -3,7 +3,7 @@
 
 **Status:** Active  
 **Owner:** PRIO MD&D Team  
-**Last reviewed:** 2026-08-05  
+**Last reviewed:** 2026-08-19  
 **Related ADRs:** ADR-001, ADR-002, ADR-008, ADR-009  
 
 ---
@@ -89,11 +89,12 @@ Assumptions that are not met **must cause failure**, not fallback behavior. The
 - **Missing required metadata columns after enrichment:** Raises `ValueError` listing missing columns
 - **Null values in required metadata columns:** Raises `ValueError` with null count and affected column name (C-01 resolved — validation active)
 - **Dataset initialization failure:** Raises `ValueError` in `_save()` if datasets are None
-- **Appwrite upload failure:** Propagates exception from `DatastoreModule`
+- **Appwrite upload failure:** Propagates exception from `DatastoreModule`. **Inside the wire leg** it is wrapped as `contract.wire.sink.TornRunError` (C-105, 2026-08-19), naming the run, how many of how many objects landed, and their names and file ids — the consumer cannot see a torn run (the manifest is the commit marker and never landed), the listed objects are **not** removed, and a re-run uploads all of them again under the same names. **The historical leg is not covered by that wrapper**: it uploads after the wire run is committed, so a failure there leaves a visible, complete forecast run alongside the *previous* run's historical artifact. Recorded as remaining scope in C-105
+- **Delivery invisible to the consumer:** raises `delivery.findability.DeliveryNotFindableError` (C-94, added 2026-08-18). After both legs are uploaded, the manager queries the partner store as the consumer does — `name == product.CONSUMER_DOCUMENT_NAME`, per category — and asserts the newest document it finds **is the one this run just uploaded**. Two distinct refusals: nothing found at all, and *found the previous run's* (*"the newest forecast document is X, but this run uploaded Y"*). The run-scoping is the whole guard — asking merely whether any document exists is a question the previous delivery already answers yes to, so the check could never fail from delivery 2 onward. This is the one failure mode where every upload reports success and the consumer still sees nothing; run-0's historical leg stranded exactly that way (C-79). It runs only inside the §11.4 interlock, and queries through a store with pipeline-core's automatic `name == model_name` filter suppressed, so it verifies the declared name rather than the views-models directory name that happens to match (C-77). **It does not detect a delivery that never ran, or stale data served from the consumer's cache** — both recorded as gaps in C-94
 - **Wrong forecast selected:** structurally impossible since #149. Selection is by **run manifest** — a commit marker whose contents are hash-verified — not by scanning the bucket for the newest `category="forecast"` upload. Declared identity is additionally checked **per shard header** against the launched ensemble inside `TargetLease.load()` (`contract/wire/source_selection.py:73-81`), so identity comes from the artifact's own content. The metadata-field check this bullet used to describe (`delivery/identity.py`) was retired in #150 and the legacy reader it served in #149; register C-25 is closed as *superseded by mechanism* <!-- legacy-ok: retirement record -->
 - **Launch config incomplete:** raises `LaunchConfigError` naming the missing key. A launcher that omits `wire_contract` or declares a `data_format` other than `feature_frame` is **refused**, never quietly routed into a fallback (ADR-003, register C-63)
 - **Region coverage mismatch:** Raises `CoverageError` in `_check_coverage()` (called from `_validate()`) if a pinned region's delivered cell count is wrong (S1/C-34) or a GAUL-uncovered excluded cell leaks into the delivery (S4/C-30)
-- **Fabricated historical tail:** `_read_historical_frame()` drops months beyond the producer's `last_valid_month_id` at the read (`_clip_observed_history` was the pandas equivalent, retired with that path in #149) so unobserved zero-padding is not shipped as observed history (S2/C-26); **degrades open** (skips the clip with a WARNING) if the boundary cannot be resolved <!-- legacy-ok: retirement record -->
+- **Fabricated historical tail:** `_read_historical_frame()` drops months beyond the producer's `last_valid_month_id` at the read (`_clip_observed_history` was the pandas equivalent, retired with that path in #149) so unobserved zero-padding is not shipped as observed history (S2/C-26). Two outcomes when the boundary is unavailable, and they are different on purpose (C-103, 2026-08-17): if the producer simply publishes no boundary — or the read fails — it **degrades open**, skipping the clip with a WARNING that states the unobserved tail will ship; if the producer client cannot be imported at all it **refuses** (`source_metadata.ProducerClientUnavailable`), because a broken environment is not a producer fact <!-- legacy-ok: retirement record -->
 - **Upload provenance:** the historical artifact's `description` carries structured provenance (lookup version, region, expected/actual cell counts, unmapped count) built by `delivery/provenance.py` (`build_provenance` → `compact_description`) via the manager's `_historical_frame_description()` (S5/C-15). The **forecast** side carries no such description: its guarantee is the wire's verified chain — per-shard content hashes recorded in the §4.2 run manifest, header asserts on load, and manifest-last commit ordering. That is identity and integrity, not the C-15 provenance field set; the §4.2 manifest's keys are exactly `contract_version`, `run_id`, `targets`, `shards`, `expected_months`, `expected_cell_count`, `sidecar` — and it carries **no** `lookup_version`, `region` or `unmapped_count`. `_delivery_description()` was the pandas-path equivalent and was deleted with it in #149 <!-- legacy-ok: retirement record -->
 
 The following **must never** fail silently: