release: 1.2.0 to main

CHANGELOG.md

@@ -0,0 +1,82 @@
+# Changelog
+
+What changed **for a consumer** — the launcher that pins this package, and the partner
+that receives its artifacts. Not a commit log: entries here name behaviour someone
+outside this repository can observe, in particular **behaviour that can turn a
+previously-passing run into a failing one**.
+
+This file exists because the version number was the only signal a consumer got
+(register C-111). Releases before 1.2.0 are summarised from their tags rather than
+reconstructed in detail.
+
+## 1.2.0 — 2026-08-26
+
+**A previously-passing delivery can now fail in three new ways. All three are
+deliberate, and each replaces a silent failure with a loud one.**
+
+### New failure modes that escape into the launcher
+
+- **`views_postprocessing.delivery.findability.DeliveryNotFindableError`** — after
+  upload, the delivery now asks the store the same question the consumer asks: *is
+  the newest document under the consumer's name the one this run just uploaded?* If
+  it is not, the run raises. **Previously a delivery whose artifacts landed somewhere
+  the consumer could not see reported success.** (C-94)
+
+- **`…findability.FindabilityUnverifiedError`** — raised when the check itself could
+  not run, e.g. the store errored on the read-back. Deliberately distinct from
+  `DeliveryNotFindableError`: "could not ask" is not "asked and got nothing".
+
+- **`views_postprocessing.contract.source_metadata.ProducerClientUnavailable`** —
+  reading the producer's `last_valid_month_id` now raises if `datafactory_query`
+  cannot be loaded, whether it is absent or raises on import. **Previously a broken
+  environment degraded open and shipped unobserved months as observed history.**
+  A producer that publishes no boundary is still handled as before — that is a normal
+  older store, and a different condition. (C-103)
+
+- **`views_postprocessing.contract.wire.sink.TornRunError`** — a failure partway
+  through uploading a run now raises a refusal naming the run, every object confirmed
+  uploaded, and the object that failed. It does **not** delete anything; a torn run
+  still leaves orphans, but it no longer leaves them undocumented. Note this is a
+  `RuntimeError`, not a `SinkError` — `SinkError` means do-not-retry, and a torn run
+  may be retried. (C-105)
+
+### Changed data reaching the partner
+
+- **Delivered artifacts carry a new provenance field, `observed_through`.** It records
+  the producer boundary the observed-range clip used, or `null` when the boundary
+  could not be read and the clip was therefore **skipped** — meaning unobserved months
+  may be present. Carried in the file's `description` metadata, which is a JSON string;
+  consumers that treat that field as opaque text are unaffected. (#297)
+
+### Changed internals a direct caller would notice
+
+- `unfao.store_port` / `crafd.store_port` `upload()` now **returns the uploaded file
+  id** instead of discarding it. Required by the findability read-back above.
+- `delivery.provenance.build_provenance()` gained a **required** keyword argument,
+  `observed_through`. Required rather than optional because a caller that omits it is
+  the failure being fixed.
+
+### Documentation
+
+- ADR-013 gains **§5.1a**, recording that nullable int64 GAUL code columns were
+  considered and rejected, with the measured pandas round-trip behaviour that decided
+  it. No `contract_version` change: the wire bytes are untouched. (#278)
+
+### Upgrading
+
+Nothing to change in a launcher. The three new exceptions surface conditions that were
+already wrong and already silent — if one fires after upgrading, it is reporting a
+pre-existing problem, not creating a new one.
+
+## 1.1.1 — 2026-08-15
+
+Fixes C-99. Consumers on 1.1.0 should move; see views-models#403.
+
+## 1.1.0 — 2026-08-13
+
+Adds the CRAF'd producer package alongside UN-FAO, and the observed-range clip that
+drops months above the producer's declared boundary.
+
+## 1.0.0 — 2026-08-01
+
+First released version.

pyproject.toml

@@ -1,6 +1,6 @@
 [tool.poetry]
 name = "views-postprocessing"
-version = "1.1.1"
+version = "1.2.0"
 description = ""
 authors = [
     "Dylan Pinheiro <dylpin@prio.org>",

tests/fixtures/wire_contract/README.md

@@ -40,3 +40,12 @@ above it: the fixture was generated under `1.0.0`, CI has reproduced it continuo
 `1.6.0`, and `1.10.2` was verified to regenerate **all five artifacts byte-identically**
 before the pin was raised (2026-08-02). Raising it again does not require regenerating the
 fixture — but it does require proving that, the same way.
+
+**Confirmed across a `views_frames` MAJOR (2026-08-21).** The shard emitted through
+`views_frames.io.arrow` under **1.10.2** and under **2.0.0** hashes identically, and both
+equal the committed fixture (`203650fd…12c54`) — measured in an isolated environment at the
+pinned toolchain (pyarrow 16.1.0, numpy 1.26.4), where all 61 frames-dependent tests also
+pass at 2.0.0. So the views-frames 2.0.0 adoption (#286) is **not** a fixture re-vendor. It
+is blocked only by views-pipeline-core, every published release of which (through 3.1.1)
+pins `views-frames <2.0.0`. Recorded so the byte question is not re-opened when that
+constraint widens.

tests/test_clone_readiness.py

@@ -28,6 +28,7 @@
 
 from __future__ import annotations
 
+import ast
 import subprocess
 import sys
 import textwrap
@@ -103,6 +104,40 @@ def _partner_prefixes() -> tuple[str, ...]:
     return tuple(f"views_postprocessing.{name}" for name in _PARTNER_PACKAGES)
 
 
+def _imported_modules(path: Path, package: str) -> set[str]:
+    """Every absolute module name ``path`` imports, relative forms resolved.
+
+    Three forms have to survive this, and the module docstring above names two of them
+    as the misses that made the earlier regex insufficient:
+
+        import views_postprocessing.unfao.product
+        from views_postprocessing.unfao import product
+        from views_postprocessing import unfao          <- the name is on the alias
+        from ..unfao import product                     <- the name is in `level`
+
+    The last two are why this resolves `level` against the file's own package and joins
+    each alias onto the module. A first pass at this skipped both and would have passed
+    a manager importing its sibling relatively — the exact shape `contract/enrichment.py`
+    once used to demonstrate a real gap.
+    """
+    parts = package.split(".")
+    found: set[str] = set()
+    for node in ast.walk(ast.parse(path.read_text())):
+        if isinstance(node, ast.Import):
+            found.update(alias.name for alias in node.names)
+        elif isinstance(node, ast.ImportFrom):
+            if node.level:
+                base = parts[: len(parts) - (node.level - 1)]
+                prefix = ".".join(base + ([node.module] if node.module else []))
+            else:
+                prefix = node.module or ""
+            if not prefix:
+                continue
+            found.add(prefix)
+            found.update(f"{prefix}.{alias.name}" for alias in node.names)
+    return found
+
+
 def _modules_on_disk(package: str) -> set[str]:
     return {
         "views_postprocessing." + f.relative_to(_PKG).with_suffix("").as_posix().replace("/", ".")
@@ -274,6 +309,70 @@ def test_the_machinery_does_not_pull_in_pipeline_core():
     )
 
 
+@pytest.mark.parametrize("partner", _PARTNER_PACKAGES)
+def test_a_partner_does_not_import_its_sibling(partner):
+    """The partners are independent, not merely both below the machinery.
+
+    Everything else in this file proves the *vertical* arrows of ADR-002 — machinery
+    imports no partner, invariants import no machinery. Nothing proved the horizontal
+    one, and it is the arrow that keeps a partner liftable: `crafd/` and `unfao/` are
+    deliberate clones (register **C-33**), so the realistic violation is a copy-paste
+    that leaves a sibling's import behind. `test_the_machinery_imports_without_any_partner`
+    cannot see it — that test imports the machinery, and this would be partner-to-partner.
+
+    Two halves, for the reason the module docstring already gives about regexes: the
+    subprocess is load-bearing and sees transitive arrivals; the source scan is the
+    supplement, and covers `managers/` — which the subprocess deliberately skips because
+    importing a manager needs views-pipeline-core, and a purity check should not be
+    contingent on a heavy framework being installed (C-40 (a)).
+    """
+    siblings = tuple(f"views_postprocessing.{p}" for p in _PARTNER_PACKAGES if p != partner)
+    if not siblings:
+        pytest.skip("independence needs a sibling; only one partner is declared")
+
+    # "managers" as a package SEGMENT, not a substring: a partner module named
+    # `managers_shared.py` would otherwise be dropped from this half while also sitting
+    # outside the AST half below, exempting it from the guard entirely with no signal.
+    # (`_modules_on_disk` already excludes `__init__.py`, so those arrive via the glob.)
+    importable = sorted(
+        m for m in _modules_on_disk(partner) if "managers" not in m.split(".")
+    )
+    assert importable, f"no importable modules found for {partner}"
+
+    result = _import_in_subprocess(tuple(importable), siblings)
+    assert result.returncode == 0, (
+        f"{partner}'s own modules failed to import:\n{result.stderr}"
+    )
+    leaked = [m for m in result.stdout.split("LEAKED:")[-1].strip().split(",") if m]
+    assert not leaked, (
+        f"{partner} pulled in a sibling partner: {leaked}. The two are deliberate "
+        "clones (C-33) and must stay liftable one at a time — an import between them "
+        "means neither can be taken without the other, and no other test here sees it."
+    )
+
+    # IMPORTS only, via the AST — not a substring scan of the file. This repository's
+    # comments cite module paths constantly (C-33's own text points at
+    # `unfao/product.py`), so scanning the text would fail on documentation and get
+    # deleted for crying wolf, which is ADR-014 §3's whole point.
+    #
+    # EVERY file under `managers/`, not just `<partner>.py`: `managers/__init__.py`
+    # carries a real import today, and the subprocess half skips the whole package.
+    managers = sorted((_PKG / partner / "managers").rglob("*.py"))
+    assert managers, f"{partner} has no managers/ directory to scan"
+    for source in managers:
+        module = "views_postprocessing." + source.relative_to(_PKG).with_suffix("").as_posix().replace("/", ".")
+        package = module.rsplit(".", 1)[0]
+        offending = sorted(
+            name for name in _imported_modules(source, package)
+            if any(name == sib or name.startswith(sib + ".") for sib in siblings)
+        )
+        assert not offending, (
+            f"{source.relative_to(_REPO)} imports {offending}. These files are copies of "
+            "each other, so this is the shape a careless clone leaves behind — and it is "
+            "outside the subprocess half above, which skips managers/."
+        )
+
+
 @pytest.mark.parametrize("partner", _PARTNER_PACKAGES)
 def test_the_guard_would_actually_catch_a_violation(partner):
     """A purity test that cannot fail is decoration.

tests/test_falsification_release_readiness.py

@@ -0,0 +1,120 @@
+"""Failing stubs from the release-readiness falsification audit, 2026-08-21.
+
+**DO NOT COMMIT THIS FILE AS-IS.** These tests fail by design, and this repository's
+`protect_main` ruleset makes the `test` job a **required check with zero bypass
+actors** — so committing red tests to `main` blocks every subsequent merge, including
+the fix. That interaction is itself finding S1 below.
+
+Claim audited: *"we're ready to set up a PR, bump the version, and run the review
+ritual"* — i.e. cutting 1.2.0 from current `main` would ship a correct, installable
+package with every release guard satisfied and nothing in the repo's governance
+blocking the path.
+
+Verdict: CONTESTED. No hard falsification; two soft ones, below.
+"""
+
+from __future__ import annotations
+
+from pathlib import Path
+
+import pytest
+
+_REPO = Path(__file__).resolve().parent.parent
+
+
+@pytest.mark.xfail(reason="S1: unaddressed falsification — see the audit report", strict=True)
+def test_the_release_path_survives_its_own_expiry_tripwire():
+    """S1 (soft). From 2026-10-18 no release can be cut, and nothing says so.
+
+    Measured 2026-08-21 against the live ruleset: `protect_main` is active, the `test`
+    job is a REQUIRED status check, and `bypass_actors` is **empty** — which is C-86's
+    finding, still true. `tests/test_credential_expiry.py` starts failing 30 days before
+    2026-11-17, i.e. **2026-10-18**. From that date the required check is red, so no PR
+    merges to `main` and no release can be tagged.
+
+    The escape is reachable — a PR that sets `ACKNOWLEDGED_UNTIL` is green on its own
+    branch (verified: 1 failed -> 6 passed) — which is why this is soft rather than
+    hard. What is missing is that **nothing tells a releaser this**. There is no release
+    runbook, and C-84 does not mention that its tripwire gates the release path.
+
+    Fix: document the interaction where a releaser will meet it — a release runbook
+    under `docs/operations/`, or a line in C-84 and C-86 cross-referencing each other.
+    """
+    runbook = list((_REPO / "docs" / "operations").glob("*release*"))
+    assert runbook, (
+        "no release runbook exists, so the 2026-10-18 block on the release path is "
+        "recorded nowhere a releaser would look (C-84 x C-86)"
+    )
+    text = "\n".join(p.read_text() for p in runbook)
+    assert "ACKNOWLEDGED_UNTIL" in text, (
+        "the release runbook does not name the only in-repo way past the expiry "
+        "tripwire once it fires"
+    )
+
+
+# ─────────────────────────────────────────────────────────────────────────────
+# Second audit, 2026-08-25. Claim: *"we are ready to bump the version, set up a
+# PR to main, review, merge when review is good, then tag and publish."*
+#
+# Verdict: FALSIFIED — one hard, two soft.
+#
+# **H1 DISCHARGED 2026-08-26** (release 1.2.0). Its probe asserted that a release
+# names what changed about failing for a consumer. `CHANGELOG.md` now exists and
+# 1.2.0's entry names the three escaping exception types and the new provenance
+# field, so the probe would XPASS and `strict=True` would turn that into a failure.
+# Removed by hand rather than left to flip, per the S4 precedent (#200). Register
+# C-111 is closed by the same change.
+#
+# **S2 DISCHARGED 2026-08-26** by the same change — and it is the same finding.
+# S2 (2026-08-21) and H1 (2026-08-25) are one concern found twice by two audits,
+# which is itself worth recording: the second audit did not read the first's stubs
+# before designing probes. Both are C-111; both are closed by CHANGELOG.md.
+#
+# S1, S3 and S4 remain open and are below. The bump SIZE (minor,
+# not patch) is recorded as an observation, not a falsification: nothing in the
+# repo is wrong about it, it is a way the releaser could be.
+# ─────────────────────────────────────────────────────────────────────────────
+
+
+@pytest.mark.xfail(reason="S3: unaddressed falsification — see the audit report", strict=True)
+def test_tagging_actually_publishes():
+    """S3 (soft). "Tag and publish" is not the mechanism this repo has.
+
+    `.github/workflows/publish_package.yml` triggers on `release: published` and
+    `workflow_dispatch` — **not** on tag push. Pushing a tag runs nothing.
+
+    The evidence that this is a live trap rather than a technicality: tags `1.0.0`
+    and `1.1.0` both exist and **neither has a GitHub Release**. Only `1.1.1` does,
+    which is the only version this workflow has ever published.
+
+    Fails until the workflow triggers on tag push, or until a release runbook states
+    that cutting a GitHub Release — not tagging — is the publishing step.
+    """
+    wf = (_REPO / ".github/workflows/publish_package.yml").read_text()
+    assert "tags:" in wf or "push:" in wf, (
+        "publish triggers only on `release: published`; a plan that says 'tag, then "
+        "publish' will tag and stop, and nothing will say so"
+    )
+
+
+@pytest.mark.xfail(reason="S4: unaddressed falsification — see the audit report", strict=True)
+def test_the_publish_job_cannot_ship_untested_code():
+    """S4 (soft). The publish job runs no tests and declares no dependency.
+
+    `publish_package.yml` has no `needs:`, no pytest step, and one gate: that the
+    version in `pyproject.toml` parses higher than the newest on PyPI. So a GitHub
+    Release cut from any commit — a branch, a stale `main`, a commit whose `test`
+    job failed — builds and uploads to PyPI unconditionally.
+
+    Nothing has gone wrong yet because releases have been cut from a green `main`
+    by hand. The guard is the habit, not the workflow, and habits are what C-86
+    already showed this repo cannot rely on when one person holds them.
+
+    Fails until the publish job depends on a passing test run, or refuses a ref
+    whose checks are not green.
+    """
+    wf = (_REPO / ".github/workflows/publish_package.yml").read_text()
+    assert "needs:" in wf or "pytest" in wf, (
+        "publish validates only version-greater-than-PyPI; nothing establishes that "
+        "the code being shipped passes its own suite"
+    )

tests/test_input_integrity_e2e.py

@@ -100,6 +100,7 @@ def test_s5_upload_description_carries_structured_provenance():
         expected_cell_count=coverage.expected_for("land_gaul"),
         actual_cell_count=3,
         unmapped_count=0,
+        observed_through=559,
     )
     description = f"Enriched ... provenance={json.dumps(prov, separators=(',', ':'))}"
 
@@ -120,5 +121,6 @@ def test_s5_provenance_records_unmapped_cells_when_present():
         expected_cell_count=coverage.expected_for("land_gaul"),
         actual_cell_count=3,
         unmapped_count=1,
+        observed_through=559,
     )
     assert prov["unmapped_count"] == 1