diff --git a/.gitattributes b/.gitattributes
new file mode 100644
index 0000000..c98413f
--- /dev/null
+++ b/.gitattributes
@@ -0,0 +1,12 @@
+# Line endings are part of the reproducibility contract, not a style preference.
+# eval/scenarios.py hashes the label file's raw bytes to stamp every run, so a CRLF checkout
+# produces a different stamp for byte-identical labels and the stamp stops being verifiable
+# off the machine that made it. Normalising to LF keeps that hash, and the generated .tex
+# and .csv assets, identical on every platform.
+* text=auto eol=lf
+
+# Generated figures and the slide deck are binary. Never line-ending normalise these: the
+# .pptx is a zip archive and normalising it would corrupt the file.
+*.png binary
+*.pdf binary
+*.pptx binary
diff --git a/.gitignore b/.gitignore
index f396149..9ee9a48 100644
--- a/.gitignore
+++ b/.gitignore
@@ -19,7 +19,15 @@ env/
# Runtime data: memory databases, embeddings, run outputs (keep the folder, not the files)
data/*
!data/.gitkeep
+# Generated paper assets are the exception: they are deliverables, they are embedded in the
+# README, and a reviewer cloning the repo has to see them without running anything first.
+!data/figures/
+!data/tables/
# OS
.DS_Store
.git.broken-backup/
+
+# Local secrets (API keys). Never committed.
+.env
+.env.local
diff --git a/README.md b/README.md
index a38ccfb..33350f9 100644
--- a/README.md
+++ b/README.md
@@ -43,26 +43,148 @@ that into five signals you can weight (or switch off) independently:
Setting any weight to zero removes that signal cleanly, which is exactly the **RQ3
ablation**, and lets the **baselines** be expressed as weight maps instead of duplicated code.
+## Results
+
+Every figure below is generated from a run directory by `assets/build_figures.py`. The
+figures carry data only; the caveats, statistics and provenance for each one live in
+[`data/figures/results.txt`](data/figures/results.txt) beside them.
+
+### Mood changes what the character recalls
+
+
+
+
+
+
+
+
+
+
+
+
+
+
-What you'll see in the applet (click to expand)
+The core deliberately needs almost nothing. `figures` adds matplotlib for the paper assets,
+and `ml` adds real sentence embeddings plus the local model. Note that Ouro needs
+transformers 4.x, which the extra pins: on 5.x its remote code does not load.
-A Textual TUI menu. **Run a conversation turn** runs a real demo turn through the pipeline
-using the thesis's own tavern-keeper example (Dawn Whitmore and the player's lie about
-running an errand for the king), and you can watch the composite scorer surface that lie at
-the top of the recalled memories. **Run experiment** is live too: it scores the three
-retrieval variants at published default weights and renders the nDCG@5 scoreboard instantly
-(`python -m eval.run` runs the full protocol). **Generate paper assets** and the **playable
-walkthrough** are still placeholders, and light up with phases 3 and 4.
+Cloud models are optional and need a key in a gitignored `.env`, written as UTF-8:
+
+```
+OLLAMA_API_KEY=your-key-from-ollama.com/settings/keys
+```
+
+
+What you'll see in the menu (click to expand)
+
+A Rich menu, shaped like [RIDGE's](https://github.com/Code-SorceryLab/RIDGE) so the two
+projects feel like one toolkit. Ten options, all wired to real work:
+
+- **Conversation Turn** runs a real turn through the pipeline on the thesis's own example
+ (Dawn Whitmore and the player's lie about running an errand for the king), and you watch
+ the composite scorer surface that lie at the top of the recalled memories.
+- **Tavern-Keeper Walkthrough** plays Dawn's five-beat arc, showing the memories she recalled
+ and her mood and trust on both sides of every appraisal. Pick the stub for an instant
+ playthrough, a local Ollama model, or Ouro 1.4B for the real thing.
+- **Quick Scoreboard** scores the three retrieval variants at published defaults instantly;
+ **Full Evaluation** runs the whole protocol and writes a run directory.
+- **Generate Paper Assets** rebuilds every figure and table from a run.
+- **Model Bake-Off** runs the same probe set through every model and measures what changes.
+- **Seeded Runs** replicates the evaluation on one model to prove it reproduces, or compares
+ across models to show what the architecture says cannot move.
+- **Latest Results**, **Settings**, and a wipe of all generated data that demands `DELETE`.
@@ -72,11 +194,30 @@ walkthrough** are still placeholders, and light up with phases 3 and 4.
RQ1 to RQ3 (click to expand)
- **RQ1 (Behaviour):** does an authored emotional state change what the character *says*, or only what it remembers?
-- **RQ2 (Robustness & cost):** is emotion-tagged memory an exploitable target, and is it fast enough for play (~600 ms target on an 8 GB card)?
+- **RQ2 (Robustness & cost):** is emotion-tagged memory an exploitable target, and what does the memory layer cost per turn?
- **RQ3 (Retrieval):** which of the five signals actually drive retrieval quality?
+On cost, the claim is about **the memory layer, not generation**. Retrieval runs in 1.8 to
+4.3 ms, comfortably inside an interactive budget. Generation is a separate, much larger cost
+that belongs to whichever model you put behind the interface, and on measured evidence no
+local model tested here answers a turn in under a second. Stating the budget as a whole-turn
+target would be a claim this project does not meet and does not control.
+
Baselines: Park et al.'s blended score and Emotional RAG, tuned under the same protocol.
-Test characters: Dawn Whitmore (invented tavern keeper) and Kenny (Telltale).
+Note the caveat above on Emotional RAG under the neutral condition. Test character: Dawn
+Whitmore, an invented tavern keeper with a pre-registered five-session arc.
+
+**On measuring believability: there is no human evaluation here, and the tone rater is a
+proxy.** `LexiconToneRater` scores valence and arousal from a fixed word list. It is
+deterministic and reproducible, which is why it is used, but it does not measure whether a
+line reads as in character to a player, and it should not be reported as if it does. Every
+claim about how a reply *sounds* rests on it. A believability claim needs people, and that
+study has not been run.
+
+Prior art matters here and is not flattering: a cluster of Stardew Valley mods already ships
+LLM NPCs with persistent memory and offline local inference. What none of them report is a
+retrieval metric, an ablation, a baseline comparison, or a poisoning test. See
+[`docs/related-work.md`](docs/related-work.md).
@@ -84,36 +225,80 @@ Test characters: Dawn Whitmore (invented tavern keeper) and Kenny (Telltale).
```
EMBR/
+├── menu.py # the Rich menu, the front door, at the root on purpose
├── embr/ # the core runtime: the middleware itself
-│ ├── memory.py # Memory record + MemoryStore
-│ ├── affect.py # Mood (valence/arousal) + trust
+│ ├── memory.py # Memory record + MemoryStore (in-memory and SQLite)
+│ ├── affect.py # Mood (valence/arousal), trust, appraisal rules
│ ├── scoring.py # the five signals + composite scorer
│ ├── prompt.py # prompt construction
-│ ├── model.py # model runner (stub now, Ouro later)
+│ ├── model.py # model runners: stub, Ollama (local and cloud), Ouro 1.4B
│ ├── pipeline.py # the five-step per-turn loop
-│ └── app/ # the Textual applet
-├── eval/ # RQ1 / RQ2 / RQ3 harness (phase 2)
-├── assets/ # branding, figures & tables for the paper
-├── docs/ # design spec
+│ └── walkthrough.py # Dawn's five-beat playable arc
+├── eval/ # RQ1 / RQ2 / RQ3 harness, bake-off, experiments
+│ ├── run.py # the full protocol
+│ ├── bakeoff.py # same probes, different models
+│ └── experiments.py # replication and cross-model comparison
+├── assets/ # hand-authored only: branding, architecture diagram, builders
+│ ├── build_tables.py # five paper tables: LaTeX + CSV
+│ ├── build_figures.py # five paper figures: PDF + PNG
+│ └── build_bakeoff_figures.py
+├── docs/ # design spec, roadmap, related work, per-phase reports
├── tests/ # unit tests
-└── data/ # memory DBs, embeddings, run outputs (git-ignored)
+└── data/ # generated: runs, figures, tables, bake-offs, experiments
```
+Anything under `assets/` is written by a person. Anything under `data/` is written by the
+pipeline and can be deleted and rebuilt, which is what the menu's wipe option does.
+
## Status
| Phase | Scope | State |
|---|---|---|
-| 0 | Skeleton, data contracts, applet shell, live demo turn | ✅ done |
-| 1 | Real retrieval (BM25 + embeddings), affect appraisal rules, SQLite store | ✅ done |
-| 2 | Eval harness, baselines, metrics, adversarial probes | ✅ done |
-| 3 | Paper assets: figures & tables straight from results | next |
-| 4 | Playable tavern-keeper walkthrough | planned |
-
-**Building EMBR?** The phase-by-phase plan (tasks, deliverables, and the results expected
-from each phase) is in [`docs/roadmap.md`](docs/roadmap.md).
-
-> An interactive web demo (a recorded TUI run + a live page) will be linked here once the
-> walkthrough lands. GitHub can't run JS in a README, so that lives on a companion page.
+| 0 | Skeleton, data contracts, menu shell, live demo turn | done |
+| 1 | Real retrieval (BM25 + embeddings), affect appraisal rules, SQLite store | done |
+| 2 | Eval harness, baselines, metrics, adversarial probes | done |
+| 3 | Paper assets: figures & tables straight from results | done |
+| 4 | Real model runners, playable walkthrough, the menu | done |
+| 5 | Bake-off, replication experiments, a run on real GPU hardware | in progress |
+
+**Building EMBR?** The phase-by-phase plan is in [`docs/roadmap.md`](docs/roadmap.md). What
+each phase delivered is in [`docs/phase2.md`](docs/phase2.md) and
+[`docs/phase3-4.md`](docs/phase3-4.md).
+
+**Setting up on a new machine?** [`docs/handoff.md`](docs/handoff.md) has the verified setup
+steps, the version constraints that matter, what git does not carry, and the measured numbers.
+
+**Where the numbers stand.** The reported run uses a real model (`llama3.2:3b`), and the
+evaluation reproduces exactly: three replicate runs gave byte-identical results with zero
+divergences.
+
+Swapping the stub for a real model is the cleanest validation here, because the architecture
+predicts what may and may not move, and every part held. nDCG, RQ1 divergence and the
+poisoning counts came back **bit-identical**, because retrieval never calls a model. The tone
+readings came alive for the first time. Generation costs 3.97 s per turn against 4.2 ms for
+score-and-retrieve, so **the memory layer is about 0.1 percent of a turn**: EMBR is not what
+makes an NPC slow.
+
+The honest reading of what it found, at more length in [`docs/handoff.md`](docs/handoff.md):
+
+- **The mood mechanism works** and is properly attributed, since zeroing the weight collapses
+ the effect to exactly 0.000.
+- **Retrieval quality is unmeasured, not bad.** EMBR neither beats nor loses to Park; the
+ ordering flips with the cut and every interval spans zero. At ten single-author queries the
+ design cannot resolve a gap that size in either direction.
+- **The evaluation cannot currently detect its own hypothesis.** Zeroing affect never
+ reordered a held-out top 5 on any query, so the label set contains no discrimination the
+ novel signals were built for.
+- **The one significant result is adversarial, and EMBR loses it.** That is also the most
+ publishable thing here.
+
+The fix for the first three is a larger ground-truth set drawn from a shipped game's authored
+dialogue, where the writers already encoded which line fires at which relationship state, so
+the labels exist without recruiting annotators, and they gate on exactly the relationship
+state the current labels ignore.
+
+> A recorded playthrough and a companion page for the interactive demo will be linked here.
+> GitHub cannot run JS in a README, so the live version has to live off-site.
## License
diff --git a/assets/build_bakeoff_figures.py b/assets/build_bakeoff_figures.py
new file mode 100644
index 0000000..5b645a4
--- /dev/null
+++ b/assets/build_bakeoff_figures.py
@@ -0,0 +1,388 @@
+"""Figures for the model bake-off: one comparison per thing the model can actually move.
+
+Separate from `build_figures.py` because the input is different. Those figures read a run
+directory and answer RQ1 to RQ3; these read a bake-off directory and answer "what changes
+when only the model changes". Sharing the house style without sharing the loader is the
+point of importing the palette rather than re-declaring it.
+
+Same rule as the paper figures: the canvas carries data and the labels needed to read it.
+Every caveat goes to `results.txt` beside the images.
+
+ python assets/build_bakeoff_figures.py # newest bake-off
+ python assets/build_bakeoff_figures.py data/bakeoff/... # a specific one
+"""
+
+from __future__ import annotations
+
+import argparse
+import json
+from pathlib import Path
+from typing import Any, Sequence
+
+import sys
+
+# Importable as `assets.build_bakeoff_figures` and runnable as `assets/build_bakeoff_figures.py`.
+# Running a file directly puts its own directory on the path rather than the repo root, so the
+# sibling import below would fail without this.
+sys.path.insert(0, str(Path(__file__).resolve().parents[1]))
+
+import matplotlib
+
+matplotlib.use("Agg")
+import matplotlib.pyplot as plt # noqa: E402
+from matplotlib.patches import Patch # noqa: E402
+
+from assets.build_figures import ( # noqa: E402
+ AMBER,
+ CREAM,
+ DEEP_BROWN,
+ EMBER_ORANGE,
+ FIGURE_DPI,
+ HOUSE_RC,
+ NEAR_BLACK,
+ RESULTS_TEXT_FILENAME,
+ _arrow_hint,
+ _style_axes,
+ _value_grid,
+ format_duration,
+)
+
+DEFAULT_OUT_DIR = Path("data/figures")
+
+#: Colour per arm kind, so the looped model is visually distinct from everything else. This
+#: is the comparison the thesis exists to make, and it should be findable without reading.
+KIND_STYLE: dict[str, tuple[str, str]] = {
+ "looped": (EMBER_ORANGE, ""),
+ "conventional": (AMBER, "...."),
+ "cloud": (DEEP_BROWN, "////"),
+ "stub": ("#D6D3D1", "xx"),
+}
+
+KIND_LABEL = {
+ "looped": "looped (Ouro)",
+ "conventional": "conventional, local",
+ "cloud": "cloud",
+ "stub": "stub (no model)",
+}
+
+
+def latest_bakeoff_dir(root: Path | str = "data/bakeoff") -> Path:
+ """Newest bake-off directory. Stamps sort chronologically, so max() is newest."""
+ candidates = [path for path in Path(root).iterdir() if path.is_dir()]
+ if not candidates:
+ raise FileNotFoundError(f"no bake-off directories under {root}")
+ return max(candidates, key=lambda path: path.name)
+
+
+def load_bakeoff(bakeoff_dir: Path | str) -> dict[str, Any]:
+ return json.loads((Path(bakeoff_dir) / "bakeoff.json").read_text(encoding="utf-8"))
+
+
+def _available(payload: dict[str, Any]) -> list[dict[str, Any]]:
+ """Arms that actually produced turns, slowest first so the bars read top to bottom."""
+ arms = [arm for arm in payload["arms"] if arm.get("available")]
+ return sorted(arms, key=lambda arm: arm["latency_ms"]["p50"], reverse=True)
+
+
+def _legend_for(arms: Sequence[dict[str, Any]], ax) -> None:
+ kinds = list(dict.fromkeys(arm["kind"] for arm in arms))
+ handles = [
+ Patch(
+ facecolor=KIND_STYLE.get(kind, (AMBER, ""))[0],
+ hatch=KIND_STYLE.get(kind, (AMBER, ""))[1],
+ edgecolor=NEAR_BLACK,
+ linewidth=0.7,
+ label=KIND_LABEL.get(kind, kind),
+ )
+ for kind in kinds
+ ]
+ legend = ax.legend(handles=handles, loc="lower right", frameon=True, borderpad=0.5)
+ frame = legend.get_frame()
+ frame.set_facecolor(CREAM)
+ frame.set_edgecolor(DEEP_BROWN)
+ frame.set_linewidth(0.6)
+
+
+def _bar_figure(
+ arms: Sequence[dict[str, Any]],
+ values: Sequence[float],
+ title: str,
+ xlabel: str,
+ hint: str,
+ value_format,
+ log_scale: bool = False,
+) -> Any:
+ """One horizontal bar panel. `value_format` is a format string or a callable.
+
+ Bars demand a linear axis: bar length is the encoding, and a log axis makes an 8x
+ difference read as 25 percent, which is how the first version of the latency panel
+ quietly lied. Callers wanting log must not use bars.
+ """
+ with plt.rc_context(HOUSE_RC):
+ figure, ax = plt.subplots(figsize=(7.2, 0.52 * len(arms) + 2.0), dpi=FIGURE_DPI)
+ figure.subplots_adjust(left=0.30, right=0.965, top=0.885, bottom=0.255)
+ _style_axes(ax)
+ positions = list(range(len(arms)))
+ for position, arm, value in zip(positions, arms, values):
+ colour, hatch = KIND_STYLE.get(arm["kind"], (AMBER, ""))
+ ax.barh(
+ position,
+ value,
+ height=0.66,
+ color=colour,
+ hatch=hatch,
+ edgecolor=NEAR_BLACK,
+ linewidth=0.7,
+ zorder=2,
+ )
+ ax.text(
+ value * 1.06 if log_scale else value + max(values) * 0.015,
+ position,
+ value_format(value) if callable(value_format) else value_format.format(value),
+ va="center",
+ ha="left",
+ fontsize=7.2,
+ color=NEAR_BLACK,
+ )
+ ax.set_yticks(positions)
+ ax.set_yticklabels([arm["model"] for arm in arms])
+ ax.invert_yaxis()
+ if log_scale:
+ ax.set_xscale("log")
+ ax.set_xlim(min(values) * 0.5, max(values) * 3.2)
+ else:
+ ax.set_xlim(0.0, max(max(values) * 1.22, 0.05))
+ ax.set_xlabel(xlabel)
+ _value_grid(ax, axis="x")
+ _arrow_hint(ax, axis="x", text=hint)
+ ax.set_title(title, loc="left", pad=15.0, color=NEAR_BLACK, fontweight="bold")
+ _legend_for(arms, ax)
+ return figure
+
+
+def build_bakeoff_figures(
+ bakeoff_dir: Path | str | None = None, out_dir: Path | str = DEFAULT_OUT_DIR
+) -> list[Path]:
+ """Build every bake-off figure plus its prose sidecar."""
+ source = Path(bakeoff_dir) if bakeoff_dir else latest_bakeoff_dir()
+ payload = load_bakeoff(source)
+ arms = _available(payload)
+ if not arms:
+ raise ValueError(f"no available arms in {source}")
+
+ out_path = Path(out_dir)
+ out_path.mkdir(parents=True, exist_ok=True)
+ written: list[Path] = []
+
+ panels = [
+ (
+ "bakeoff_latency",
+ [arm["latency_ms"]["p50"] / 1000.0 for arm in arms],
+ "Bake-off: the looped model is the slowest arm by far",
+ "median end-to-end turn latency, seconds",
+ "lower is better; cloud arms include network time",
+ lambda seconds: format_duration(seconds * 1000.0),
+ False,
+ ),
+ (
+ "bakeoff_grounding",
+ [arm["grounded_rate"] for arm in arms],
+ "Bake-off: every model uses the memory it is handed",
+ "share of replies reusing a retrieved memory",
+ "higher is better",
+ "{:.0%}",
+ False,
+ ),
+ (
+ "bakeoff_mood",
+ [arm["mood_valence_spread"] for arm in arms],
+ "Bake-off: bigger models track the NPC's mood more closely",
+ "range of mean rated valence across the three mood conditions",
+ "higher = more sensitive to the affect signal",
+ "{:.3f}",
+ False,
+ ),
+ ]
+
+ for stem, values, title, xlabel, hint, fmt, log_scale in panels:
+ figure = _bar_figure(arms, values, title, xlabel, hint, fmt, log_scale)
+ try:
+ pdf_path = out_path / f"{stem}.pdf"
+ png_path = out_path / f"{stem}.png"
+ figure.savefig(pdf_path, format="pdf", metadata={"CreationDate": None})
+ figure.savefig(
+ png_path, format="png", dpi=FIGURE_DPI, metadata={"Software": "EMBR"}
+ )
+ written += [pdf_path, png_path]
+ finally:
+ plt.close(figure)
+
+ written.append(_write_notes(source, payload, arms, out_path))
+ return written
+
+
+def latest_replicate_dir(root: Path | str = "data/experiments") -> Path:
+ """Newest replicate experiment directory."""
+ candidates = [
+ path for path in Path(root).iterdir() if path.is_dir() and path.name.startswith("replicate-")
+ ]
+ if not candidates:
+ raise FileNotFoundError(f"no replicate experiments under {root}")
+ return max(candidates, key=lambda path: path.name)
+
+
+def build_replicate_figure(
+ replicate_dir: Path | str | None = None, out_dir: Path | str = DEFAULT_OUT_DIR
+) -> list[Path]:
+ """Draw how far the timing moved across identical runs.
+
+ The deterministic metrics do not need a figure: they were identical, and a chart of
+ four identical numbers says nothing a sentence cannot. Latency is the one reading that
+ legitimately varies, so the useful picture is how much, which is the error bar anyone
+ quoting a latency number actually needs.
+ """
+ source = Path(replicate_dir) if replicate_dir else latest_replicate_dir()
+ report = json.loads((source / "replicate.json").read_text(encoding="utf-8"))
+ spread = report["latency_p95_spread"]
+ variants = sorted(spread, key=lambda name: spread[name]["max_ms"], reverse=True)
+
+ out_path = Path(out_dir)
+ out_path.mkdir(parents=True, exist_ok=True)
+ with plt.rc_context(HOUSE_RC):
+ figure, ax = plt.subplots(figsize=(7.2, 0.55 * len(variants) + 2.0), dpi=FIGURE_DPI)
+ figure.subplots_adjust(left=0.22, right=0.965, top=0.875, bottom=0.28)
+ _style_axes(ax)
+ for position, variant in enumerate(variants):
+ low = spread[variant]["min_ms"]
+ high = spread[variant]["max_ms"]
+ ax.plot(
+ [low, high],
+ [position, position],
+ color=DEEP_BROWN,
+ linewidth=3.0,
+ solid_capstyle="butt",
+ zorder=3,
+ )
+ for value in (low, high):
+ ax.plot(
+ value,
+ position,
+ marker="|",
+ markersize=11,
+ markeredgewidth=2.0,
+ color=NEAR_BLACK,
+ zorder=4,
+ )
+ ax.text(
+ high * 1.04,
+ position,
+ f"{low:.2f} to {high:.2f} ms",
+ va="center",
+ ha="left",
+ fontsize=7.2,
+ color=NEAR_BLACK,
+ )
+ ax.set_yticks(range(len(variants)))
+ ax.set_yticklabels(variants)
+ ax.invert_yaxis()
+ ax.set_xscale("log")
+ lows = [spread[v]["min_ms"] for v in variants]
+ highs = [spread[v]["max_ms"] for v in variants]
+ ax.set_xlim(min(lows) * 0.6, max(highs) * 4.0)
+ ax.set_xlabel(
+ f"score-and-retrieve p95 latency, milliseconds, over "
+ f"{report['replicates']} identical runs (log scale)"
+ )
+ _value_grid(ax, axis="x")
+ _arrow_hint(ax, axis="x", text="bar width is the run-to-run spread")
+ ax.set_title(
+ f"Replication: latency spread across {report['replicates']} identical runs",
+ loc="left",
+ pad=15.0,
+ color=NEAR_BLACK,
+ fontweight="bold",
+ )
+ try:
+ pdf_path = out_path / "replicate_latency.pdf"
+ png_path = out_path / "replicate_latency.png"
+ figure.savefig(pdf_path, format="pdf", metadata={"CreationDate": None})
+ figure.savefig(
+ png_path, format="png", dpi=FIGURE_DPI, metadata={"Software": "EMBR"}
+ )
+ finally:
+ plt.close(figure)
+ return [pdf_path, png_path]
+
+
+def _write_notes(
+ source: Path, payload: dict[str, Any], arms: list[dict[str, Any]], out_dir: Path
+) -> Path:
+ """Append the bake-off prose to the figure notes sidecar."""
+ metadata = payload["metadata"]
+ unavailable = [arm for arm in payload["arms"] if not arm.get("available")]
+ lines = [
+ "",
+ "=" * 72,
+ "Model bake-off",
+ "=" * 72,
+ "",
+ f"Source: {source}",
+ f"Probe turns per arm: {metadata['probe_turns_per_arm']} "
+ f"({metadata['queries_per_condition']} queries x "
+ f"{len(metadata['conditions'])} mood conditions)",
+ f"Generated: {metadata['generated_at']}",
+ "",
+ metadata["note"],
+ "",
+ "Latency is wall clock and includes network time for cloud arms, so cloud and local",
+ "numbers are not like for like. Grounding is a word overlap screen for replies that",
+ "ignore the memory block entirely, not a semantic entailment check. Mood spread is",
+ "the range of mean rated warmth across the pinned moods: a model that answers the",
+ "same way in every mood scores zero and makes the affect signal inert.",
+ "",
+ "Per arm:",
+ ]
+ for arm in arms:
+ lines.append(
+ f" {arm['model']:<26} p50 {format_duration(arm['latency_ms']['p50']):>8} "
+ f"p95 {format_duration(arm['latency_ms']['p95']):>8} "
+ f"grounded {arm['grounded_rate']:>5.0%} "
+ f"mood spread {arm['mood_valence_spread']:.3f} "
+ f"persona breaks {arm['persona_break_rate']:.0%}"
+ )
+ if unavailable:
+ lines += ["", "Unavailable:"]
+ lines += [f" {arm['model']}: {arm.get('error', 'unknown')}" for arm in unavailable]
+ lines.append("")
+
+ notes = out_dir / RESULTS_TEXT_FILENAME
+ existing = notes.read_text(encoding="utf-8") if notes.exists() else ""
+ # Rebuilt in place: drop any previous bake-off block so repeated builds do not stack.
+ trimmed = existing.split("\n" + "=" * 72 + "\nModel bake-off")[0]
+ notes.write_text(trimmed + "\n".join(lines), encoding="utf-8", newline="\n")
+ return notes
+
+
+def main(argv: Sequence[str] | None = None) -> None:
+ parser = argparse.ArgumentParser(description=__doc__)
+ parser.add_argument("bakeoff_dir", nargs="?", default=None)
+ parser.add_argument("--out-dir", default=str(DEFAULT_OUT_DIR))
+ parser.add_argument(
+ "--skip-replicate",
+ action="store_true",
+ help="only build the bake-off figures, not the replication one",
+ )
+ args = parser.parse_args(argv)
+ written = build_bakeoff_figures(args.bakeoff_dir, args.out_dir)
+ if not args.skip_replicate:
+ try:
+ written += build_replicate_figure(out_dir=args.out_dir)
+ except FileNotFoundError as error:
+ # A bake-off is useful on its own; a missing replicate run is not a failure.
+ print(f" (skipping replication figure: {error})")
+ for path in written:
+ print(f" {path}")
+
+
+if __name__ == "__main__":
+ main()
diff --git a/assets/build_figures.py b/assets/build_figures.py
index 345a894..d4aaaa7 100644
--- a/assets/build_figures.py
+++ b/assets/build_figures.py
@@ -22,7 +22,7 @@
Usage:
from assets.build_figures import build_all_figures
- build_all_figures("data/runs/20260817-160950") # writes assets/figures/
+ build_all_figures("data/runs/20260817-160950") # writes data/figures/
python -m assets.build_figures # newest run, same output
"""
@@ -46,7 +46,7 @@
from matplotlib.figure import Figure # noqa: E402
from matplotlib.lines import Line2D # noqa: E402
from matplotlib.patches import Patch # noqa: E402
-from matplotlib.ticker import LogFormatterSciNotation # noqa: E402
+from matplotlib.ticker import FuncFormatter # noqa: E402
# --------------------------------------------------------------------------------------
# Ember palette and house style
@@ -96,8 +96,10 @@
#: How much of the commit hash the footer carries. Twelve is unambiguous in this repo and
#: still fits on one line at footer size.
COMMIT_ABBREV_LENGTH = 12
-DEFAULT_OUT_DIR = Path("assets/figures")
+DEFAULT_OUT_DIR = Path("data/figures")
RESULTS_FILENAME = "results.json"
+#: The prose sidecar. Figures carry data; every caveat and provenance line goes here.
+RESULTS_TEXT_FILENAME = "results.txt"
#: Only this pipeline stage is plotted for latency: it is the stage the retrieval design
#: actually changes, and the one phase 2 reports.
LATENCY_STAGE = "score_retrieve"
@@ -111,6 +113,12 @@
FOOTER_FONT_SIZE = 5.9
#: Left inset for figure level text, as a fraction of figure width.
TEXT_MARGIN = 0.012
+
+#: Where the direction hint sits, in axes fractions. Named because a spec's margins have to
+#: reserve room for them: a bottom margin too small silently clips the hint off the canvas,
+#: which is invisible in code review and obvious only once someone opens the PNG.
+HINT_BELOW_AXES = -0.215
+HINT_ABOVE_AXES = 1.028
#: Clearance kept under the axes, in inches: a base, then per line of tick labels, then an
#: extra allowance only when the axes actually carries an x label.
TICK_LABEL_BASE_INCHES = 0.20
@@ -221,6 +229,20 @@ def figure_footer_text(results: Mapping[str, object], run_stamp: str) -> str:
# --------------------------------------------------------------------------------------
+@dataclass(frozen=True)
+class FigureText:
+ """The prose belonging to one figure, kept off the canvas and written to results.txt.
+
+ `title` is the only member that reaches the image. `note` is the reading instruction
+ that used to sit under the title, and `caption` is the methodological caveat that used
+ to sit under the axes; both now live in the sidecar file so the figures stay data.
+ """
+
+ title: str
+ note: str
+ caption: str = ""
+
+
@dataclass(frozen=True)
class RetrievalRow:
"""One RQ3 variant: its score and the marginal interval around it."""
@@ -308,7 +330,12 @@ def _display_label(variant: str, meta: Mapping[str, object]) -> str:
if ablated:
return f"{str(ablated).replace('_', ' ')} zeroed"
system_key = variant.removesuffix(f"_{condition}")
- return f"{SYSTEM_LABELS.get(system_key, system_key)} {condition}"
+ # A dagger on any row whose mood term could not reorder anything under this protocol.
+ # Without it the Emotional RAG rows read as a comparison against mood-biased retrieval
+ # when they are a comparison against relevance alone, which is the kind of thing a
+ # reviewer catches and the authors cannot then unsay.
+ marker = " †" if meta.get("mood_rank_invariant") else ""
+ return f"{SYSTEM_LABELS.get(system_key, system_key)} {condition}{marker}"
def _group_of(meta: Mapping[str, object]) -> str:
@@ -535,21 +562,6 @@ def _as_sentence(text: str) -> str:
return sentence if sentence.endswith(".") else f"{sentence}."
-def _axis_room_inches(ax: Axes) -> float:
- """Inches to keep free under the axes for its tick labels and its axis label.
-
- Measured from what the axes actually has: a figure with two line tick labels and no x
- label needs a different reservation than one with a single line and a label, and
- guessing one number for both leaves a visible hole under half the figures.
- """
- lines = max(
- (str(label.get_text()).count("\n") + 1 for label in ax.get_xticklabels()),
- default=1,
- )
- room = TICK_LABEL_BASE_INCHES + TICK_LABEL_LINE_INCHES * lines
- return room + (AXIS_LABEL_INCHES if ax.get_xlabel() else 0.0)
-
-
def _style_axes(ax: Axes) -> None:
"""The flat ember look: cream ground, two spines, no chartjunk."""
ax.set_facecolor(CREAM)
@@ -567,85 +579,76 @@ def _value_grid(ax: Axes, axis: str) -> None:
ax.grid(axis=axis, color=NEAR_BLACK, alpha=0.12, linewidth=0.6)
-def _titles(ax: Axes, title: str, note: str) -> None:
- """Left aligned title with a wrapped caveat note underneath, both above the axes.
+def _titles(ax: Axes, title: str, note: str, caption: str = "") -> "FigureText":
+ """Draw the title alone. Every other line goes to the sidecar.
- The note is placed in figure coordinates and the title pad is then computed from how
- many lines it took, so a longer note pushes the title up instead of colliding with it.
+ The title states the finding, so it is the only text that has to be on the canvas: a
+ reader who can see what the chart says does not need a paragraph telling them. Both the
+ reading note and the methodological caption go to `results.txt`, where a reader who
+ wants the caveats can find all of them together instead of the one that happened to
+ fit. Returned rather than drawn so no caller can quietly put prose back.
"""
- figure = ax.figure
- box = ax.get_position()
- available = (box.x1 - box.x0) * figure.get_figwidth()
- lines = _wrap_to_width(note, NOTE_FONT_SIZE, available)
- step = _line_step_inches(NOTE_FONT_SIZE) / figure.get_figheight()
- # Drawn bottom up from just above the axes, so the block grows towards the title.
- for index, line in enumerate(reversed(lines)):
- figure.text(
- box.x0,
- box.y1 + 0.012 + index * step,
- line,
- fontsize=NOTE_FONT_SIZE,
- color=DEEP_BROWN,
- va="bottom",
- ha="left",
- )
- pad_points = (len(lines) * _line_step_inches(NOTE_FONT_SIZE) + 0.10) * 72.0
- ax.set_title(title, loc="left", pad=pad_points, color=NEAR_BLACK, fontweight="bold")
-
-
-def _add_bottom_block(
- figure: Figure,
- results: Mapping[str, object],
- run_stamp: str,
- caption: str | None,
- axis_room_inches: float,
-) -> float:
- """Stack the caption above the provenance footer at the foot of the figure.
-
- Returns the height to reserve below the axes, in inches. All of this figure's bottom
- geometry lives here, which is what makes it impossible for an axis to grow down into
- its own caption: the caller reserves exactly what was drawn.
+ # Pad clears the vertical direction hint, which sits just above the plot area.
+ ax.set_title(title, loc="left", pad=16.0, color=NEAR_BLACK, fontweight="bold")
+ return FigureText(title=title, note=note, caption=caption)
+
+
+def format_duration(milliseconds: float) -> str:
+ """Human units: milliseconds below one second, seconds above.
+
+ A label like "32,392 ms" makes the reader do arithmetic mid-figure, which is the
+ figure failing at its one job. Sub-second precision steps down as values grow so the
+ label never carries digits the measurement's run-to-run spread cannot support.
"""
- height = figure.get_figheight()
- available = figure.get_figwidth() * (1.0 - 2 * TEXT_MARGIN)
- provenance, warning = figure_footer_text(results, run_stamp).split("\n")
- footer_lines = [
- (line, NEAR_BLACK, 0.82)
- for line in _wrap_to_width(provenance, FOOTER_FONT_SIZE, available)
- ] + [
- (line, DEEP_BROWN, 1.0)
- for line in _wrap_to_width(warning, FOOTER_FONT_SIZE, available)
- ]
+ if milliseconds >= 1000.0:
+ return f"{milliseconds / 1000.0:.1f} s"
+ if milliseconds >= 100.0:
+ return f"{milliseconds:.0f} ms"
+ if milliseconds >= 1.0:
+ return f"{milliseconds:.1f} ms"
+ return f"{milliseconds:.2f} ms"
- cursor = BOTTOM_PAD_INCHES
- for line, colour, alpha in reversed(footer_lines):
- figure.text(
- TEXT_MARGIN,
- cursor / height,
- line,
- fontsize=FOOTER_FONT_SIZE,
- color=colour,
- alpha=alpha,
- va="bottom",
- ha="left",
- )
- cursor += _line_step_inches(FOOTER_FONT_SIZE)
-
- if caption:
- cursor += CAPTION_GAP_INCHES
- for line in reversed(_wrap_to_width(caption, CAPTION_FONT_SIZE, available)):
- figure.text(
- TEXT_MARGIN,
- cursor / height,
- line,
- fontsize=CAPTION_FONT_SIZE,
- color=NEAR_BLACK,
- va="bottom",
- ha="left",
- )
- cursor += _line_step_inches(CAPTION_FONT_SIZE)
- return cursor + axis_room_inches
+def _duration_ticks(ax: Axes) -> None:
+ """Axis ticks in the same human units as the value labels, replacing 10^n notation."""
+ ax.xaxis.set_major_formatter(FuncFormatter(lambda value, _pos: format_duration(value)))
+
+
+def _arrow_hint(ax: Axes, axis: str, text: str) -> None:
+ """A short arrow under the axis saying which way is worse or better.
+
+ Without this a reader has to already know whether a tall bar is a good result or a bad
+ one, which is exactly the knowledge a general reader does not have. It is a direction
+ cue on the scale rather than commentary, so it stays on the canvas while prose does not.
+ """
+ # Terse and recessive on purpose. The reader needs the sense of the scale, not a caption
+ # explaining the chart back to them: this sits at the end of the axis label, not above it.
+ if axis == "x":
+ x, y, ha = 0.5, HINT_BELOW_AXES, "center"
+ else:
+ x, y, ha = 0.0, HINT_ABOVE_AXES, "left"
+
+ # Refuse to draw off the canvas rather than silently losing the text. Placement is in
+ # axes fractions, so whether it lands on the page depends on the spec's margins, and a
+ # clipped hint is invisible in a diff and obvious only once someone opens the PNG.
+ box = ax.get_position()
+ figure_fraction = box.y0 + y * (box.y1 - box.y0)
+ if not 0.005 < figure_fraction < 0.995:
+ raise ValueError(
+ f"direction hint would be clipped at figure fraction {figure_fraction:.3f}; "
+ f"give this figure's spec a larger {'bottom' if axis == 'x' else 'top'} margin"
+ )
+ ax.annotate(
+ text,
+ xy=(x, y),
+ xycoords="axes fraction",
+ fontsize=NOTE_FONT_SIZE - 0.3,
+ color=DEEP_BROWN,
+ alpha=0.85,
+ ha=ha,
+ va="center",
+ annotation_clip=False,
+ )
def _legend(ax: Axes, handles: Sequence, loc: str) -> None:
@@ -752,13 +755,33 @@ def _draw_rq3_retrieval(ax: Axes, results: Mapping[str, object]) -> str | None:
ax.set_xlabel(f"{metric} over the 10 pre-registered queries")
ax.set_xlim(0.0, max(row.value + row.error_high for row in rows) + 0.11)
_value_grid(ax, axis="x")
- _titles(
+ _arrow_hint(ax, axis="x", text="higher is better")
+ return _titles(
ax,
- f"RQ3 retrieval quality: {metric} per variant",
+ f"RQ3: no variant separates from the baselines ({metric})",
"Whiskers are marginal 95% bootstrap intervals. Overlap is not a test of a "
"difference: the paired deltas figure carries the quantity actually tested.",
+ caption=(
+ f"Bars are {metric} over the 10 pre-registered queries, grouped into published "
+ "default weights, weights tuned leave one query out, and ablations of tuned "
+ "EMBR. The three families are not interchangeable: only the tuned rows saw the "
+ "label set, so a default row and a tuned row are not a fair head to head."
+ + (
+ " Rows marked with a dagger carry a mood term that is rank invariant under "
+ "this protocol: RQ3 scores in the neutral zero-mood condition, where mood "
+ "congruence returns the same value for every memory and therefore cannot "
+ "reorder a result. This matters most for Emotional RAG, whose published "
+ "score is relevance plus mood: scored here it reduces to relevance alone, "
+ "so those rows are not a comparison against the system its paper describes. "
+ "It is also why no mood ablation is reported, and why the mood term is "
+ "measured by RQ1 instead. The gold labels do not vary with mood, so "
+ "re-scoring under a live mood would not fix this: a mood term could then "
+ "only move retrieval away from a fixed relevant set and lower the metric."
+ if any("†" in row.label for row in rows)
+ else ""
+ )
+ ),
)
- return None
# --------------------------------------------------------------------------------------
@@ -808,45 +831,65 @@ def _draw_rq3_ablation(ax: Axes, results: Mapping[str, object]) -> str | None:
markeredgewidth=1.1,
zorder=5,
)
- # The p values get their own right hand column: x in axes fraction, y in data.
- ax.text(
- 0.995,
- position,
- f"Holm p {row.p_holm:.2f} (attainable floor {row.attainable_p_floor:.3f})",
- transform=ax.get_yaxis_transform(),
- va="center",
- ha="right",
- fontsize=6.8,
- color=NEAR_BLACK,
- )
-
ax.set_yticks(positions)
ax.set_yticklabels(
- [f"{row.label}\n(no reordering)" if row.is_degenerate else row.label for row in rows]
+ [f"{row.label}\n(never reordered)" if row.is_degenerate else row.label for row in rows]
)
ax.invert_yaxis()
ax.set_ylim(len(rows) - 0.4, -0.6)
lowest = min(row.ci_low for row in rows)
highest = max(row.ci_high for row in rows)
span = max(highest - lowest, 1e-6)
- # The extra room on the right is for the p column, not for data.
- ax.set_xlim(lowest - 0.08 * span, highest + 0.62 * span)
- ax.set_xlabel(f"paired mean difference in {metric}: {reference} minus ablation")
+ ax.set_xlim(lowest - 0.10 * span, highest + 0.10 * span)
+ ax.set_xlabel(f"paired change in {metric}: {reference} minus ablation")
_value_grid(ax, axis="x")
+ _arrow_hint(ax, axis="x", text="right of zero = zeroing the signal cost accuracy")
+
crossing = sum(1 for row in rows if row.includes_zero)
- _titles(
+ handles = [
+ Line2D(
+ [],
+ [],
+ color=DEEP_BROWN,
+ linewidth=1.7,
+ marker="o",
+ markersize=6.0,
+ markerfacecolor=EMBER_ORANGE,
+ markeredgecolor=NEAR_BLACK,
+ label="measured loss, with uncertainty",
+ ),
+ Line2D(
+ [],
+ [],
+ color="none",
+ marker="o",
+ markersize=10.0,
+ markerfacecolor="none",
+ markeredgecolor=DEEP_BROWN,
+ markeredgewidth=1.1,
+ label="touches zero: too close to call",
+ ),
+ ]
+ _legend(ax, handles, loc="lower right")
+ return _titles(
ax,
- f"RQ3 ablations against {reference}, paired",
+ "RQ3: only relevance measurably changes the ranking",
f"Positive means removing the signal cost accuracy. {crossing} of {len(rows)} "
"intervals include zero (ringed on the zero line), so no ablation is conclusive.",
- )
- return (
- "Whiskers are 95% bootstrap intervals on the per query paired difference, the "
- "quantity the sign flip test asks about. A zero width interval means that ablation "
- "never reordered a held out top 5, so it is uninformative on this label set rather "
- "than switched off. Read each Holm p against its own attainable floor: a floor at "
- "or above 0.05 could not have reached significance under any arrangement of its "
- "own data."
+ caption=(
+ f"Paired mean difference in {metric}, {reference} minus ablation. Whiskers are "
+ "95% bootstrap intervals on the per query paired difference, the quantity the "
+ "sign flip test asks about. A zero width interval means that ablation never "
+ "reordered a held out top 5, so it is uninformative on this label set rather "
+ "than switched off. Holm corrected p values, each against its own attainable "
+ "floor (a floor at or above 0.05 could not have reached significance under any "
+ "arrangement of its own data): "
+ + "; ".join(
+ f"{row.label} p={row.p_holm:.2f} floor={row.attainable_p_floor:.3f}"
+ for row in rows
+ )
+ + "."
+ ),
)
@@ -877,6 +920,17 @@ def _draw_rq2_poisoning(ax: Axes, results: Mapping[str, object]) -> str | None:
zorder=2,
)
for centre, value in zip(centres, values):
+ if value == 0:
+ # A zero bar draws nothing, which reads as missing data rather than as the
+ # strongest result on the chart. The stub says "measured, and it was none".
+ ax.plot(
+ [centre + offset - width * 0.47, centre + offset + width * 0.47],
+ [0.0, 0.0],
+ color=NEAR_BLACK,
+ linewidth=2.2,
+ solid_capstyle="butt",
+ zorder=4,
+ )
ax.text(
centre + offset,
value + total * 0.03,
@@ -896,6 +950,7 @@ def _draw_rq2_poisoning(ax: Axes, results: Mapping[str, object]) -> str | None:
ax.set_yticks(range(total + 1))
ax.set_ylabel(f"injections retrieved (of {total} per category)")
_value_grid(ax, axis="y")
+ _arrow_hint(ax, axis="y", text="higher = more vulnerable")
handles = [
Patch(
facecolor=INJECTION_STYLE[index % len(INJECTION_STYLE)][0],
@@ -920,22 +975,23 @@ def _draw_rq2_poisoning(ax: Axes, results: Mapping[str, object]) -> str | None:
)
)
_legend(ax, handles, loc="upper center")
- _titles(
+ pure_input = ", ".join(name.replace("_", " ") for name in summary.pure_input_categories)
+ return _titles(
ax,
- "RQ2 memory poisoning: injections that reached retrieval",
+ "RQ2: emotional weighting makes memory the easiest to poison",
f"{len(categories) * total} injection attacks per system "
f"({len(categories)} categories of {total}); a bar counts the attacks whose "
- "planted memory entered the probe's top 5.",
- )
- pure_input = ", ".join(name.replace("_", " ") for name in summary.pure_input_categories)
- return (
- f"The other {summary.pure_input_attack_count} attacks ({pure_input}) are absent "
- "here by construction, not zero by measurement: they write nothing to the store, "
- "so no poison exists to retrieve. Drawing them as zero bars would claim a defended "
- "result where the architecture has nothing to defend. The measurement that "
- "establishes it is probe_prompt_identical, true for all "
- f"{summary.pure_input_attack_count} of them in every system and false for every "
- "injection."
+ "planted memory entered the probe's top 5. A flat stub on the baseline is a "
+ "measured zero, not a missing bar.",
+ caption=(
+ f"The other {summary.pure_input_attack_count} attacks ({pure_input}) are absent "
+ "here by construction, not zero by measurement: they write nothing to the "
+ "store, so no poison exists to retrieve. Drawing them as zero bars would claim "
+ "a defended result where the architecture has nothing to defend. The "
+ "measurement that establishes it is probe_prompt_identical, true for all "
+ f"{summary.pure_input_attack_count} of them in every system and false for "
+ "every injection."
+ ),
)
@@ -945,77 +1001,98 @@ def _draw_rq2_poisoning(ax: Axes, results: Mapping[str, object]) -> str | None:
def _draw_rq2_latency(ax: Axes, results: Mapping[str, object]) -> str | None:
- rows = latency_rows(results)
- positions = list(range(len(rows)))
- measurements = [row.p50 for row in rows] + [row.p95 for row in rows]
+ memory_rows = latency_rows(results)
+ model_rows = latency_rows(results, stage="model")
+ positions = list(range(len(memory_rows)))
+ measurements = [value for row in memory_rows + model_rows for value in (row.p50, row.p95)]
use_log = _should_use_log_scale(measurements)
- for position, row in zip(positions, rows):
- # A dumbbell rather than bars: on a log axis a bar's baseline is arbitrary, and p50
- # to p95 is a range, which a connecting line says better than two columns.
- ax.plot(
- [row.p50, row.p95],
- [position, position],
- color=DEEP_BROWN,
- linewidth=1.6,
- zorder=2,
- solid_capstyle="round",
- )
- ax.plot(
- row.p50,
- position,
- marker="o",
- markersize=6.5,
- markerfacecolor=EMBER_ORANGE,
- markeredgecolor=NEAR_BLACK,
- markeredgewidth=0.8,
- zorder=3,
- )
- ax.plot(
- row.p95,
- position,
- marker="D",
- markersize=6.0,
- markerfacecolor=CREAM,
- markeredgecolor=NEAR_BLACK,
- markeredgewidth=0.9,
- zorder=3,
- )
- ax.text(
- row.p95 * 1.2 if use_log else row.p95 + max(measurements) * 0.03,
- position,
- f"p50 {row.p50:.3f} ms p95 {row.p95:.3f} ms",
- va="center",
- ha="left",
- fontsize=7.0,
- color=NEAR_BLACK,
- )
+ # Both stages on one axis, because the split IS the finding: the reader should see in
+ # one glance that the memory layer costs milliseconds while the model costs seconds. A
+ # single-stage version of this figure answered a question nobody was asking.
+ stages = (
+ (memory_rows, -0.16, EMBER_ORANGE),
+ (model_rows, +0.16, DEEP_BROWN),
+ )
+ for rows, offset, colour in stages:
+ for position, row in zip(positions, rows):
+ y = position + offset
+ # A dumbbell rather than bars: on a log axis a bar's baseline is arbitrary,
+ # and p50 to p95 is a range, which a connecting line says better than columns.
+ ax.plot(
+ [row.p50, row.p95],
+ [y, y],
+ color=colour,
+ linewidth=1.8,
+ zorder=2,
+ solid_capstyle="round",
+ )
+ ax.plot(
+ row.p50,
+ y,
+ marker="o",
+ markersize=6.0,
+ markerfacecolor=colour,
+ markeredgecolor=NEAR_BLACK,
+ markeredgewidth=0.8,
+ zorder=3,
+ )
+ ax.plot(
+ row.p95,
+ y,
+ marker="D",
+ markersize=5.6,
+ markerfacecolor=CREAM,
+ markeredgecolor=NEAR_BLACK,
+ markeredgewidth=0.9,
+ zorder=3,
+ )
+ ax.text(
+ row.p95 * 1.25 if use_log else row.p95 + max(measurements) * 0.03,
+ y,
+ f"{format_duration(row.p50)} to {format_duration(row.p95)}",
+ va="center",
+ ha="left",
+ fontsize=6.8,
+ color=NEAR_BLACK,
+ )
if use_log:
ax.set_xscale("log")
- ax.xaxis.set_major_formatter(LogFormatterSciNotation())
# Generous headroom on the right: on a log axis the value labels need it.
ax.set_xlim(min(measurements) / 2.2, max(measurements) * 9.0)
else:
ax.set_xlim(0.0, max(measurements) * 1.7)
+ _duration_ticks(ax)
ax.set_yticks(positions)
- ax.set_yticklabels([row.label for row in rows])
+ ax.set_yticklabels([row.label for row in memory_rows])
ax.invert_yaxis()
- ax.set_ylim(len(rows) - 0.5, -0.5)
- ax.set_xlabel(
- "score and retrieve latency, milliseconds" + (" (log scale)" if use_log else "")
- )
+ ax.set_ylim(len(positions) - 0.5, -0.5)
+ ax.set_xlabel("latency per turn" + (" (log scale)" if use_log else ""))
_value_grid(ax, axis="x")
+ _arrow_hint(ax, axis="x", text="lower is better")
handles = [
Line2D(
[],
[],
- color=NEAR_BLACK,
- linestyle="none",
+ color=EMBER_ORANGE,
+ linewidth=1.8,
marker="o",
- markersize=6.5,
+ markersize=6.0,
markerfacecolor=EMBER_ORANGE,
- label="p50",
+ markeredgecolor=NEAR_BLACK,
+ label="memory layer: score and retrieve",
+ ),
+ Line2D(
+ [],
+ [],
+ color=DEEP_BROWN,
+ linewidth=1.8,
+ marker="o",
+ markersize=6.0,
+ markerfacecolor=DEEP_BROWN,
+ markeredgecolor=NEAR_BLACK,
+ label="model: generate the reply",
),
Line2D(
[],
@@ -1023,28 +1100,41 @@ def _draw_rq2_latency(ax: Axes, results: Mapping[str, object]) -> str | None:
color=NEAR_BLACK,
linestyle="none",
marker="D",
- markersize=6.0,
+ markersize=5.6,
markerfacecolor=CREAM,
- label="p95",
+ label="dumbbell spans p50 to p95",
),
]
- _legend(ax, handles, loc="lower right")
- ratio = max(measurements) / min(measurements)
- _titles(
- ax,
- f"RQ2 cost: {LATENCY_STAGE.replace('_', ' and ')} stage per system",
- f"Log axis: fastest to slowest spans about {ratio:.0f}x, so a linear axis would "
- "collapse three of these rows onto one another."
- if use_log
- else "Linear axis: every measurement is within one order of magnitude.",
+ # The wide empty band on a log axis is between the memory cluster (a few ms) and the
+ # model cluster (a few s), i.e. the lower-centre. Anchoring there clears every dumbbell.
+ _legend(ax, handles, loc="lower center")
+ share = (
+ 100.0 * max(row.p95 for row in memory_rows) / max(row.p95 for row in model_rows)
+ if max(row.p95 for row in model_rows) > 0
+ else 0.0
)
+ ratio = max(measurements) / min(measurements)
note = str(results["rq2"].get("metadata", {}).get("latency_note", "")) # type: ignore[union-attr]
- return (
- f"{_as_sentence(note)} Nearest rank percentiles over "
- f"{rows[0].sample_count} timed retrievals per system, wall clock on one machine. "
- "This is the one measurement in the run that is not deterministic, and the store "
- "holds a single scenario's memories, so read the ratio between systems rather than "
- "the absolute milliseconds."
+ return _titles(
+ ax,
+ "RQ2: choosing the memories is not what makes a turn slow",
+ (
+ f"The memory layer's worst case is about {share:.1f} percent of the model's: "
+ "choosing the memories is not what makes a turn slow."
+ if share < 50.0
+ else f"Log axis: fastest to slowest spans about {ratio:.0f}x."
+ ),
+ caption=(
+ "Each dumbbell spans p50 to p95. "
+ f"{_as_sentence(note)} "
+ f"Nearest rank percentiles over {memory_rows[0].sample_count} timed retrievals "
+ "per system, wall clock on one machine. This is the one measurement in the run "
+ "that is not deterministic, and the store holds a single scenario's memories, so "
+ "read the ratio between systems rather than the absolute durations. The model "
+ "stage times whichever runner the run was made with; under the stub it is "
+ "microseconds and the comparison is meaningless, so build this figure from a "
+ "real-model run."
+ ),
)
@@ -1115,8 +1205,9 @@ def _draw_rq1_divergence(ax: Axes, results: Mapping[str, object]) -> str | None:
)
ax.set_xlim(-0.6, len(rows) - 0.4)
ax.set_ylim(0.0, max(row.value + row.error_high for row in rows) + 0.12)
- ax.set_ylabel("mean Jaccard distance, top 5 sets")
+ ax.set_ylabel("mean Jaccard distance between top-5 sets")
_value_grid(ax, axis="y")
+ _arrow_hint(ax, axis="y", text="higher = mood moved retrieval further")
handles = [
Patch(
facecolor=EMBER_ORANGE,
@@ -1137,21 +1228,22 @@ def _draw_rq1_divergence(ax: Axes, results: Mapping[str, object]) -> str | None:
),
]
_legend(ax, handles, loc="upper left")
- _titles(
- ax,
- "RQ1 attribution: mood alone moves what is retrieved",
- "Zeroing the mood weight collapses all three pairs to exactly 0.000, which is what "
- "attributes the divergence to the mood term rather than to run to run noise.",
- )
weak_note = (
f" {', '.join(weak)} is the weak pair: its interval reaches zero, and Jaccard "
"distance cannot go below zero, so no test against zero is reported."
if weak
else ""
)
- return (
- "Whiskers are fixed seed 95% bootstrap intervals over the per query top 5 "
- "distances." + weak_note
+ return _titles(
+ ax,
+ "RQ1: mood alone changes which memories come back",
+ "Zeroing the mood weight collapses all three pairs to exactly 0.000, which is what "
+ "attributes the divergence to the mood term rather than to run to run noise.",
+ caption=(
+ "Bars are mean Jaccard distance between the top 5 sets the two moods retrieve. "
+ "Whiskers are fixed seed 95% bootstrap intervals over the per query top 5 "
+ "distances." + weak_note
+ ),
)
@@ -1175,34 +1267,37 @@ class FigureSpec:
draw: Callable[[Axes, Mapping[str, object]], "str | None"] = field(repr=False)
+# Bottom margins carry the axis label and the direction arrow, and nothing else: the prose
+# that used to be reserved for down here now lives in results.txt. Left margins on the two
+# figures with a vertical arrow are wider to hold it beside the tick labels.
RQ3_RETRIEVAL_SPEC = FigureSpec(
stem="rq3_retrieval",
- size_inches=(7.2, 5.0),
- margins={"left": 0.200, "right": 0.985, "top": 0.860, "bottom": 0.150},
+ size_inches=(7.2, 4.6),
+ margins={"left": 0.215, "right": 0.985, "top": 0.895, "bottom": 0.220},
draw=_draw_rq3_retrieval,
)
RQ3_ABLATION_SPEC = FigureSpec(
stem="rq3_ablation",
- size_inches=(7.2, 4.6),
- margins={"left": 0.200, "right": 0.985, "top": 0.840, "bottom": 0.150},
+ size_inches=(7.2, 4.2),
+ margins={"left": 0.215, "right": 0.985, "top": 0.890, "bottom": 0.240},
draw=_draw_rq3_ablation,
)
RQ2_POISONING_SPEC = FigureSpec(
stem="rq2_poisoning",
- size_inches=(7.2, 5.0),
- margins={"left": 0.105, "right": 0.985, "top": 0.840, "bottom": 0.150},
+ size_inches=(7.2, 4.6),
+ margins={"left": 0.165, "right": 0.985, "top": 0.880, "bottom": 0.125},
draw=_draw_rq2_poisoning,
)
RQ2_LATENCY_SPEC = FigureSpec(
stem="rq2_latency",
- size_inches=(7.2, 4.4),
- margins={"left": 0.135, "right": 0.985, "top": 0.850, "bottom": 0.150},
+ size_inches=(7.2, 4.6),
+ margins={"left": 0.165, "right": 0.985, "top": 0.890, "bottom": 0.205},
draw=_draw_rq2_latency,
)
RQ1_DIVERGENCE_SPEC = FigureSpec(
stem="rq1_divergence",
size_inches=(7.2, 4.6),
- margins={"left": 0.105, "right": 0.985, "top": 0.850, "bottom": 0.150},
+ margins={"left": 0.185, "right": 0.985, "top": 0.880, "bottom": 0.140},
draw=_draw_rq1_divergence,
)
@@ -1237,76 +1332,114 @@ def _write_both_formats(figure: Figure, out_dir: Path, stem: str) -> list[Path]:
return [pdf_path, png_path]
-def _render(spec: FigureSpec, run_dir: Path | str, out_dir: Path | str) -> list[Path]:
- """The one render path every figure goes through: style, draw, stamp, save, close."""
+def _render(
+ spec: FigureSpec, run_dir: Path | str, out_dir: Path | str
+) -> tuple[list[Path], FigureText]:
+ """The one render path every figure goes through: style, draw, save, close.
+
+ Returns the written paths and the figure's prose. Nothing but the title, the axis
+ labels and the data itself reaches the canvas; the prose is the caller's to file.
+ """
run_path = Path(run_dir)
results = load_run_results(run_path)
with plt.rc_context(HOUSE_RC):
figure, ax = plt.subplots(figsize=spec.size_inches, dpi=FIGURE_DPI)
try:
- # Explicit margins rather than tight_layout: the caption and footer sit in
- # reserved space, and fixed margins keep the output identical run to run.
+ # Explicit margins rather than tight_layout: fixed margins keep the output
+ # identical run to run, which is what makes the PNG bytes assertable.
figure.subplots_adjust(**spec.margins)
_style_axes(ax)
- caption = spec.draw(ax, results)
- reserved = _add_bottom_block(
- figure, results, run_path.name, caption, _axis_room_inches(ax)
- )
- # Give back exactly what the bottom block took, so nothing can overlap it.
- figure.subplots_adjust(
- **{
- **spec.margins,
- "bottom": max(spec.margins["bottom"], reserved / spec.size_inches[1]),
- }
- )
- return _write_both_formats(figure, Path(out_dir), spec.stem)
+ text = spec.draw(ax, results)
+ return _write_both_formats(figure, Path(out_dir), spec.stem), text
finally:
plt.close(figure)
+def write_results_text(
+ run_dir: Path | str, texts: Sequence[tuple[str, FigureText]], out_dir: Path | str
+) -> Path:
+ """Write the prose that used to be printed onto the figures, as a sidecar file.
+
+ Every caveat still ships, it just ships next to the images instead of on top of the
+ data. The provenance footer leads, because the first question anyone asks of a number
+ is which run and which commit produced it.
+ """
+ run_path = Path(run_dir)
+ results = load_run_results(run_path)
+ out_path = Path(out_dir) / RESULTS_TEXT_FILENAME
+ lines = [
+ "EMBR figure notes",
+ "=" * 72,
+ "",
+ "Prose for the figures in this directory. The figures carry data only; every",
+ "caveat, statistic and provenance line lives here.",
+ "",
+ figure_footer_text(results, run_path.name),
+ "",
+ ]
+ for stem, text in texts:
+ lines += ["-" * 72, f"{stem}.png / {stem}.pdf", "-" * 72, "", text.title, ""]
+ if text.note:
+ lines += [_as_sentence(text.note), ""]
+ if text.caption:
+ lines += [_as_sentence(text.caption), ""]
+ out_path.parent.mkdir(parents=True, exist_ok=True)
+ out_path.write_text("\n".join(lines), encoding="utf-8", newline="\n")
+ return out_path
+
+
def build_rq3_retrieval_figure(
run_dir: Path | str, out_dir: Path | str = DEFAULT_OUT_DIR
) -> list[Path]:
"""nDCG@5 per variant with marginal intervals, grouped by family, tuned EMBR marked."""
- return _render(RQ3_RETRIEVAL_SPEC, run_dir, out_dir)
+ return _render(RQ3_RETRIEVAL_SPEC, run_dir, out_dir)[0]
def build_rq3_ablation_figure(
run_dir: Path | str, out_dir: Path | str = DEFAULT_OUT_DIR
) -> list[Path]:
"""Paired ablation deltas against tuned EMBR, every zero crossing interval ringed."""
- return _render(RQ3_ABLATION_SPEC, run_dir, out_dir)
+ return _render(RQ3_ABLATION_SPEC, run_dir, out_dir)[0]
def build_rq2_poisoning_figure(
run_dir: Path | str, out_dir: Path | str = DEFAULT_OUT_DIR
) -> list[Path]:
"""Injections that reached retrieval per system, pure input immunity annotated."""
- return _render(RQ2_POISONING_SPEC, run_dir, out_dir)
+ return _render(RQ2_POISONING_SPEC, run_dir, out_dir)[0]
def build_rq2_latency_figure(
run_dir: Path | str, out_dir: Path | str = DEFAULT_OUT_DIR
) -> list[Path]:
"""Score and retrieve p50 to p95 per system, log axis when the floor compresses them."""
- return _render(RQ2_LATENCY_SPEC, run_dir, out_dir)
+ return _render(RQ2_LATENCY_SPEC, run_dir, out_dir)[0]
def build_rq1_divergence_figure(
run_dir: Path | str, out_dir: Path | str = DEFAULT_OUT_DIR
) -> list[Path]:
"""Mood pair retrieval divergence with intervals, plus the ablated control at zero."""
- return _render(RQ1_DIVERGENCE_SPEC, run_dir, out_dir)
+ return _render(RQ1_DIVERGENCE_SPEC, run_dir, out_dir)[0]
def build_all_figures(
run_dir: Path | str, out_dir: Path | str = DEFAULT_OUT_DIR
) -> list[Path]:
- """Build every paper figure from one run directory.
+ """Build every paper figure from one run directory, plus the prose sidecar.
- Returns the written paths in build order, two per figure (`.pdf` then `.png`).
+ Returns the written paths in build order, two per figure (`.pdf` then `.png`), with
+ `results.txt` last. The sidecar is written from the same render pass that made the
+ images, so the notes can never describe a figure that is no longer there.
"""
- return [path for spec in FIGURE_SPECS for path in _render(spec, run_dir, out_dir)]
+ paths: list[Path] = []
+ texts: list[tuple[str, FigureText]] = []
+ for spec in FIGURE_SPECS:
+ rendered, text = _render(spec, run_dir, out_dir)
+ paths.extend(rendered)
+ texts.append((spec.stem, text))
+ paths.append(write_results_text(run_dir, texts, out_dir))
+ return paths
def main(argv: Sequence[str] | None = None) -> None:
diff --git a/assets/build_tables.py b/assets/build_tables.py
index aedd354..073c91a 100644
--- a/assets/build_tables.py
+++ b/assets/build_tables.py
@@ -1,6 +1,6 @@
"""Paper tables: every EMBR results table as LaTeX (booktabs) plus a flat CSV twin.
-One run directory in, ten files out under `assets/tables/`:
+One run directory in, ten files out under `data/tables/`:
* `signals` the five-signal reference (authored content, see SIGNAL_REFERENCE)
* `rq3_retrieval` one row per scoring variant, grouped by condition
@@ -40,7 +40,7 @@
from dataclasses import dataclass
from pathlib import Path
-DEFAULT_OUT_DIR = Path("assets/tables")
+DEFAULT_OUT_DIR = Path("data/tables")
# Three decimals everywhere. The run directory already ships full-precision CSVs, so the
# paper rounds exactly once and the LaTeX and its CSV twin agree digit for digit; a reader
@@ -378,10 +378,13 @@ def write_table(table: Table, provenance: Provenance, out_dir: Path | str) -> li
directory.mkdir(parents=True, exist_ok=True)
tex_path = directory / f"{table.name}.tex"
csv_path = directory / f"{table.name}.csv"
- tex_path.write_text(render_latex(table, provenance))
+ # LF explicitly on both, because a paper asset whose bytes depend on the machine that
+ # built it is not reproducible: write_text would follow os.linesep, and csv.writer
+ # terminates rows with CRLF unless told otherwise.
+ tex_path.write_text(render_latex(table, provenance), encoding="utf-8", newline="\n")
header, rows = render_csv(table)
- with csv_path.open("w", newline="") as handle:
- writer = csv.writer(handle)
+ with csv_path.open("w", newline="", encoding="utf-8") as handle:
+ writer = csv.writer(handle, lineterminator="\n")
writer.writerow(header)
writer.writerows(rows)
return [tex_path, csv_path]
@@ -725,11 +728,16 @@ def _power_sentence(comparisons: dict) -> str:
of quietly misreporting.
"""
total = len(comparisons)
+ # Judged against the corrected floor, because the p in the neighbouring column is Holm
+ # corrected. Comparing a raw floor against a corrected p understates the floor: it made
+ # this caption report 8 of 9 blocked when the true answer is 9 of 9, and made the
+ # no-relevance ablation look reachable at 0.031 when its corrected floor is 0.125.
blocked = sum(
1
for row in comparisons.values()
- if row.get("attainable_p_floor") is not None
- and row["attainable_p_floor"] >= SIGNIFICANCE_ALPHA
+ if row.get("attainable_p_floor_holm", row.get("attainable_p_floor")) is not None
+ and row.get("attainable_p_floor_holm", row["attainable_p_floor"])
+ >= SIGNIFICANCE_ALPHA
)
significant = sum(
1
diff --git a/assets/figures/rq1_divergence.pdf b/assets/figures/rq1_divergence.pdf
deleted file mode 100644
index e795e8a..0000000
Binary files a/assets/figures/rq1_divergence.pdf and /dev/null differ
diff --git a/assets/figures/rq1_divergence.png b/assets/figures/rq1_divergence.png
deleted file mode 100644
index 35de2cf..0000000
Binary files a/assets/figures/rq1_divergence.png and /dev/null differ
diff --git a/assets/figures/rq2_latency.pdf b/assets/figures/rq2_latency.pdf
deleted file mode 100644
index 37d4383..0000000
Binary files a/assets/figures/rq2_latency.pdf and /dev/null differ
diff --git a/assets/figures/rq2_latency.png b/assets/figures/rq2_latency.png
deleted file mode 100644
index 7f3cbc0..0000000
Binary files a/assets/figures/rq2_latency.png and /dev/null differ
diff --git a/assets/figures/rq2_poisoning.png b/assets/figures/rq2_poisoning.png
deleted file mode 100644
index 3332228..0000000
Binary files a/assets/figures/rq2_poisoning.png and /dev/null differ
diff --git a/assets/figures/rq3_ablation.pdf b/assets/figures/rq3_ablation.pdf
deleted file mode 100644
index beedcd0..0000000
Binary files a/assets/figures/rq3_ablation.pdf and /dev/null differ
diff --git a/assets/figures/rq3_ablation.png b/assets/figures/rq3_ablation.png
deleted file mode 100644
index 039af84..0000000
Binary files a/assets/figures/rq3_ablation.png and /dev/null differ
diff --git a/assets/figures/rq3_retrieval.png b/assets/figures/rq3_retrieval.png
deleted file mode 100644
index 9b301d6..0000000
Binary files a/assets/figures/rq3_retrieval.png and /dev/null differ
diff --git a/assets/tables/.gitkeep b/assets/tables/.gitkeep
deleted file mode 100644
index e69de29..0000000
diff --git a/data/figures/bakeoff_grounding.pdf b/data/figures/bakeoff_grounding.pdf
new file mode 100644
index 0000000..1511efd
Binary files /dev/null and b/data/figures/bakeoff_grounding.pdf differ
diff --git a/data/figures/bakeoff_grounding.png b/data/figures/bakeoff_grounding.png
new file mode 100644
index 0000000..3c9b3a1
Binary files /dev/null and b/data/figures/bakeoff_grounding.png differ
diff --git a/data/figures/bakeoff_latency.pdf b/data/figures/bakeoff_latency.pdf
new file mode 100644
index 0000000..121f04c
Binary files /dev/null and b/data/figures/bakeoff_latency.pdf differ
diff --git a/data/figures/bakeoff_latency.png b/data/figures/bakeoff_latency.png
new file mode 100644
index 0000000..1a8a7b6
Binary files /dev/null and b/data/figures/bakeoff_latency.png differ
diff --git a/data/figures/bakeoff_mood.pdf b/data/figures/bakeoff_mood.pdf
new file mode 100644
index 0000000..a4f5de2
Binary files /dev/null and b/data/figures/bakeoff_mood.pdf differ
diff --git a/data/figures/bakeoff_mood.png b/data/figures/bakeoff_mood.png
new file mode 100644
index 0000000..1fc37cf
Binary files /dev/null and b/data/figures/bakeoff_mood.png differ
diff --git a/data/figures/replicate_latency.pdf b/data/figures/replicate_latency.pdf
new file mode 100644
index 0000000..1721260
Binary files /dev/null and b/data/figures/replicate_latency.pdf differ
diff --git a/data/figures/replicate_latency.png b/data/figures/replicate_latency.png
new file mode 100644
index 0000000..e1bdfd2
Binary files /dev/null and b/data/figures/replicate_latency.png differ
diff --git a/data/figures/results.txt b/data/figures/results.txt
new file mode 100644
index 0000000..a5f6bbe
--- /dev/null
+++ b/data/figures/results.txt
@@ -0,0 +1,58 @@
+EMBR figure notes
+========================================================================
+
+Prose for the figures in this directory. The figures carry data only; every
+caveat, statistic and provenance line lives here.
+
+run 20260819-061120 | commit 0097390448ca (dirty tree) | model stub | labels dawn-whitmore v1 | built by assets/build_figures.py
+PRELIMINARY: stub model (echoes the player's line), deterministic lexical embedder, v1 single author labels, 10 queries. Every interval spans zero and no Holm corrected comparison is significant: read direction, not ranking.
+
+------------------------------------------------------------------------
+rq3_retrieval.png / rq3_retrieval.pdf
+------------------------------------------------------------------------
+
+RQ3: no variant separates from the baselines (nDCG@5)
+
+Whiskers are marginal 95% bootstrap intervals. Overlap is not a test of a difference: the paired deltas figure carries the quantity actually tested.
+
+Bars are nDCG@5 over the 10 pre-registered queries, grouped into published default weights, weights tuned leave one query out, and ablations of tuned EMBR. The three families are not interchangeable: only the tuned rows saw the label set, so a default row and a tuned row are not a fair head to head. Rows marked with a dagger carry a mood term that is rank invariant under this protocol: RQ3 scores in the neutral zero-mood condition, where mood congruence returns the same value for every memory and therefore cannot reorder a result. This matters most for Emotional RAG, whose published score is relevance plus mood: scored here it reduces to relevance alone, so those rows are not a comparison against the system its paper describes. It is also why no mood ablation is reported, and why the mood term is measured by RQ1 instead. The gold labels do not vary with mood, so re-scoring under a live mood would not fix this: a mood term could then only move retrieval away from a fixed relevant set and lower the metric.
+
+------------------------------------------------------------------------
+rq3_ablation.png / rq3_ablation.pdf
+------------------------------------------------------------------------
+
+RQ3: only relevance measurably changes the ranking
+
+Positive means removing the signal cost accuracy. 4 of 4 intervals include zero (ringed on the zero line), so no ablation is conclusive.
+
+Paired mean difference in nDCG@5, EMBR tuned † minus ablation. Whiskers are 95% bootstrap intervals on the per query paired difference, the quantity the sign flip test asks about. A zero width interval means that ablation never reordered a held out top 5, so it is uninformative on this label set rather than switched off. Holm corrected p values, each against its own attainable floor (a floor at or above 0.05 could not have reached significance under any arrangement of its own data): relevance zeroed p=0.75 floor=0.031; recency zeroed p=1.00 floor=0.500; affect zeroed p=1.00 floor=1.000; event gate zeroed p=1.00 floor=1.000.
+
+------------------------------------------------------------------------
+rq2_poisoning.png / rq2_poisoning.pdf
+------------------------------------------------------------------------
+
+RQ2: emotional weighting makes memory the easiest to poison
+
+10 injection attacks per system (2 categories of 5); a bar counts the attacks whose planted memory entered the probe's top 5. A flat stub on the baseline is a measured zero, not a missing bar.
+
+The other 10 attacks (role override, persona dissolution) are absent here by construction, not zero by measurement: they write nothing to the store, so no poison exists to retrieve. Drawing them as zero bars would claim a defended result where the architecture has nothing to defend. The measurement that establishes it is probe_prompt_identical, true for all 10 of them in every system and false for every injection.
+
+------------------------------------------------------------------------
+rq2_latency.png / rq2_latency.pdf
+------------------------------------------------------------------------
+
+RQ2: choosing the memories is not what makes a turn slow
+
+Log axis: fastest to slowest spans about 807x.
+
+Each dumbbell spans p50 to p95. Latency times the evaluated configuration: the full Dawn Whitmore store with the shared deterministic embedder on both the write and query paths. Nearest rank percentiles over 100 timed retrievals per system, wall clock on one machine. This is the one measurement in the run that is not deterministic, and the store holds a single scenario's memories, so read the ratio between systems rather than the absolute durations. The model stage times whichever runner the run was made with; under the stub it is microseconds and the comparison is meaningless, so build this figure from a real-model run.
+
+------------------------------------------------------------------------
+rq1_divergence.png / rq1_divergence.pdf
+------------------------------------------------------------------------
+
+RQ1: mood alone changes which memories come back
+
+Zeroing the mood weight collapses all three pairs to exactly 0.000, which is what attributes the divergence to the mood term rather than to run to run noise.
+
+Bars are mean Jaccard distance between the top 5 sets the two moods retrieve. Whiskers are fixed seed 95% bootstrap intervals over the per query top 5 distances. warm vs neutral is the weak pair: its interval reaches zero, and Jaccard distance cannot go below zero, so no test against zero is reported.
diff --git a/data/figures/rq1_divergence.pdf b/data/figures/rq1_divergence.pdf
new file mode 100644
index 0000000..f33b960
Binary files /dev/null and b/data/figures/rq1_divergence.pdf differ
diff --git a/data/figures/rq1_divergence.png b/data/figures/rq1_divergence.png
new file mode 100644
index 0000000..e3da949
Binary files /dev/null and b/data/figures/rq1_divergence.png differ
diff --git a/data/figures/rq2_latency.pdf b/data/figures/rq2_latency.pdf
new file mode 100644
index 0000000..cad39f3
Binary files /dev/null and b/data/figures/rq2_latency.pdf differ
diff --git a/data/figures/rq2_latency.png b/data/figures/rq2_latency.png
new file mode 100644
index 0000000..2d8b2b9
Binary files /dev/null and b/data/figures/rq2_latency.png differ
diff --git a/assets/figures/rq2_poisoning.pdf b/data/figures/rq2_poisoning.pdf
similarity index 60%
rename from assets/figures/rq2_poisoning.pdf
rename to data/figures/rq2_poisoning.pdf
index 9034eeb..c18835d 100644
Binary files a/assets/figures/rq2_poisoning.pdf and b/data/figures/rq2_poisoning.pdf differ
diff --git a/data/figures/rq2_poisoning.png b/data/figures/rq2_poisoning.png
new file mode 100644
index 0000000..17b8cca
Binary files /dev/null and b/data/figures/rq2_poisoning.png differ
diff --git a/data/figures/rq3_ablation.pdf b/data/figures/rq3_ablation.pdf
new file mode 100644
index 0000000..6ab3639
Binary files /dev/null and b/data/figures/rq3_ablation.pdf differ
diff --git a/data/figures/rq3_ablation.png b/data/figures/rq3_ablation.png
new file mode 100644
index 0000000..0d4b1df
Binary files /dev/null and b/data/figures/rq3_ablation.png differ
diff --git a/assets/figures/rq3_retrieval.pdf b/data/figures/rq3_retrieval.pdf
similarity index 58%
rename from assets/figures/rq3_retrieval.pdf
rename to data/figures/rq3_retrieval.pdf
index 9d06f45..1466360 100644
Binary files a/assets/figures/rq3_retrieval.pdf and b/data/figures/rq3_retrieval.pdf differ
diff --git a/data/figures/rq3_retrieval.png b/data/figures/rq3_retrieval.png
new file mode 100644
index 0000000..fccee20
Binary files /dev/null and b/data/figures/rq3_retrieval.png differ
diff --git a/assets/tables/rq1_divergence.csv b/data/tables/rq1_divergence.csv
similarity index 97%
rename from assets/tables/rq1_divergence.csv
rename to data/tables/rq1_divergence.csv
index 4de76e9..c9a66a6 100644
--- a/assets/tables/rq1_divergence.csv
+++ b/data/tables/rq1_divergence.csv
@@ -1,4 +1,4 @@
-pair,mean_jaccard_divergence,ci95_low,ci95_high,mood_ablated_divergence
-warm|neutral,0.142,0.000,0.308,0.000
-warm|suspicious,0.388,0.207,0.562,0.000
-neutral|suspicious,0.271,0.124,0.419,0.000
+pair,mean_jaccard_divergence,ci95_low,ci95_high,mood_ablated_divergence
+warm|neutral,0.142,0.000,0.308,0.000
+warm|suspicious,0.388,0.207,0.562,0.000
+neutral|suspicious,0.271,0.124,0.419,0.000
diff --git a/assets/tables/rq1_divergence.tex b/data/tables/rq1_divergence.tex
similarity index 91%
rename from assets/tables/rq1_divergence.tex
rename to data/tables/rq1_divergence.tex
index 2d44e57..30df53c 100644
--- a/assets/tables/rq1_divergence.tex
+++ b/data/tables/rq1_divergence.tex
@@ -1,6 +1,6 @@
-% EMBR table rq1_divergence: run_dir=data/runs/20260817-160950, git_commit=55e6533452c0ee5a3bc9f54c6aee3d2b6b61a212 (dirty working tree), label_version=v1, model=stub
-% label_set=dawn-whitmore, label_sha256=5d5f38bc31c6230b8805964de2b56866cbcbb4c422133ca91aa68584e2ad1b82, results generated_at=2026-08-17T16:09:50.594780+00:00
-% Requires \usepackage{booktabs}. Flat twin: rq1_divergence.csv. Rebuild: python assets/build_tables.py data/runs/20260817-160950
+% EMBR table rq1_divergence: run_dir=data\runs\20260819-061120, git_commit=0097390448caea68e11a0135d4bd30ffc2df9fc3 (dirty working tree), label_version=v1, model=stub
+% label_set=dawn-whitmore, label_sha256=5d5f38bc31c6230b8805964de2b56866cbcbb4c422133ca91aa68584e2ad1b82, results generated_at=2026-08-19T06:11:20.291391+00:00
+% Requires \usepackage{booktabs}. Flat twin: rq1_divergence.csv. Rebuild: python assets/build_tables.py data\runs\20260819-061120
% If the tabular overflows the text block, make the float a table* or wrap the tabular in \resizebox{\linewidth}{!}{...}.
% results.json rq1.metadata.divergence_note: retrieval_divergence_jaccard is the mean of the
% per-query top-5 jaccard distances and retrieval_divergence_ci95 is a fixed-seed percentile
diff --git a/assets/tables/rq2_robustness.csv b/data/tables/rq2_robustness.csv
similarity index 52%
rename from assets/tables/rq2_robustness.csv
rename to data/tables/rq2_robustness.csv
index 7572846..dbee440 100644
--- a/assets/tables/rq2_robustness.csv
+++ b/data/tables/rq2_robustness.csv
@@ -1,5 +1,5 @@
-system,poison_retrieved,injection_attacks,mean_retrieval_drift_injections,pure_input_prompt_identical,pure_input_attacks,score_retrieve_p95_ms
-embr,9,10,0.419,10,10,0.925
-park,2,10,0.067,10,10,0.912
-emo_rag,4,10,0.698,10,10,1.004
-recency_only,10,10,0.333,10,10,0.023
+system,poison_retrieved,injection_attacks,mean_retrieval_drift_injections,pure_input_prompt_identical,pure_input_attacks,score_retrieve_p95_ms
+embr,9,10,0.419,10,10,2.891
+park,2,10,0.067,10,10,2.473
+emo_rag,4,10,0.698,10,10,2.907
+recency_only,10,10,0.333,10,10,0.108
diff --git a/assets/tables/rq2_robustness.tex b/data/tables/rq2_robustness.tex
similarity index 86%
rename from assets/tables/rq2_robustness.tex
rename to data/tables/rq2_robustness.tex
index 8b18bc7..450b415 100644
--- a/assets/tables/rq2_robustness.tex
+++ b/data/tables/rq2_robustness.tex
@@ -1,6 +1,6 @@
-% EMBR table rq2_robustness: run_dir=data/runs/20260817-160950, git_commit=55e6533452c0ee5a3bc9f54c6aee3d2b6b61a212 (dirty working tree), label_version=v1, model=stub
-% label_set=dawn-whitmore, label_sha256=5d5f38bc31c6230b8805964de2b56866cbcbb4c422133ca91aa68584e2ad1b82, results generated_at=2026-08-17T16:09:50.594780+00:00
-% Requires \usepackage{booktabs}. Flat twin: rq2_robustness.csv. Rebuild: python assets/build_tables.py data/runs/20260817-160950
+% EMBR table rq2_robustness: run_dir=data\runs\20260819-061120, git_commit=0097390448caea68e11a0135d4bd30ffc2df9fc3 (dirty working tree), label_version=v1, model=stub
+% label_set=dawn-whitmore, label_sha256=5d5f38bc31c6230b8805964de2b56866cbcbb4c422133ca91aa68584e2ad1b82, results generated_at=2026-08-19T06:11:20.291391+00:00
+% Requires \usepackage{booktabs}. Flat twin: rq2_robustness.csv. Rebuild: python assets/build_tables.py data\runs\20260819-061120
% If the tabular overflows the text block, make the float a table* or wrap the tabular in \resizebox{\linewidth}{!}{...}.
% results.json rq2.metadata.note: Reply tone is rated on the deterministic stub model, which only
% echoes the player's line; these VA numbers exercise the pipeline end to end and become
@@ -38,10 +38,10 @@
\toprule
System & Poison retrieved & Retrieval drift & Prompt identical & $p_{95}$ (ms) \\
\midrule
- \texttt{embr} & 9 / 10 & 0.419 & 10 / 10 & 0.925 \\
- \texttt{park} & 2 / 10 & 0.067 & 10 / 10 & 0.912 \\
- \texttt{emo\_rag} & 4 / 10 & 0.698 & 10 / 10 & 1.004 \\
- \texttt{recency\_only} & 10 / 10 & 0.333 & 10 / 10 & 0.023 \\
+ \texttt{embr} & 9 / 10 & 0.419 & 10 / 10 & 2.891 \\
+ \texttt{park} & 2 / 10 & 0.067 & 10 / 10 & 2.473 \\
+ \texttt{emo\_rag} & 4 / 10 & 0.698 & 10 / 10 & 2.907 \\
+ \texttt{recency\_only} & 10 / 10 & 0.333 & 10 / 10 & 0.108 \\
\bottomrule
\end{tabular}
\end{table}
diff --git a/assets/tables/rq3_comparisons.csv b/data/tables/rq3_comparisons.csv
similarity index 98%
rename from assets/tables/rq3_comparisons.csv
rename to data/tables/rq3_comparisons.csv
index 73cca71..0886e64 100644
--- a/assets/tables/rq3_comparisons.csv
+++ b/data/tables/rq3_comparisons.csv
@@ -1,10 +1,10 @@
-holm_family,variant,mean_diff,mean_diff_ci95_low,mean_diff_ci95_high,p_value,p_holm,attainable_p_floor,floor_at_or_above_alpha
-primary,park_tuned,0.043,-0.131,0.283,1.000,1.000,0.250,yes
-primary,emo_rag_tuned,0.004,-0.046,0.066,1.000,1.000,0.250,yes
-ablation,embr_no_recency,0.019,-0.013,0.070,1.000,1.000,0.500,yes
-ablation,embr_no_affect,0.000,0.000,0.000,1.000,1.000,1.000,yes
-ablation,embr_no_event_gate,-0.017,-0.052,0.000,1.000,1.000,1.000,yes
-ablation,embr_no_relevance,0.142,-0.044,0.368,0.188,0.750,0.031,no
-secondary,embr_default,-0.038,-0.125,0.057,0.562,1.000,0.062,yes
-secondary,park_default,-0.052,-0.189,0.088,0.625,1.000,0.125,yes
-secondary,emo_rag_default,0.004,-0.046,0.066,1.000,1.000,0.250,yes
+holm_family,variant,mean_diff,mean_diff_ci95_low,mean_diff_ci95_high,p_value,p_holm,attainable_p_floor,floor_at_or_above_alpha
+primary,park_tuned,0.043,-0.131,0.283,1.000,1.000,0.250,yes
+primary,emo_rag_tuned,0.004,-0.046,0.066,1.000,1.000,0.250,yes
+ablation,embr_no_recency,0.019,-0.013,0.070,1.000,1.000,0.500,yes
+ablation,embr_no_affect,0.000,0.000,0.000,1.000,1.000,1.000,yes
+ablation,embr_no_event_gate,-0.017,-0.052,0.000,1.000,1.000,1.000,yes
+ablation,embr_no_relevance,0.142,-0.044,0.368,0.188,0.750,0.031,no
+secondary,embr_default,-0.038,-0.125,0.057,0.562,1.000,0.062,yes
+secondary,park_default,-0.052,-0.189,0.088,0.625,1.000,0.125,yes
+secondary,emo_rag_default,0.004,-0.046,0.066,1.000,1.000,0.250,yes
diff --git a/assets/tables/rq3_comparisons.tex b/data/tables/rq3_comparisons.tex
similarity index 93%
rename from assets/tables/rq3_comparisons.tex
rename to data/tables/rq3_comparisons.tex
index 44b6e8f..80902bf 100644
--- a/assets/tables/rq3_comparisons.tex
+++ b/data/tables/rq3_comparisons.tex
@@ -1,6 +1,6 @@
-% EMBR table rq3_comparisons: run_dir=data/runs/20260817-160950, git_commit=55e6533452c0ee5a3bc9f54c6aee3d2b6b61a212 (dirty working tree), label_version=v1, model=stub
-% label_set=dawn-whitmore, label_sha256=5d5f38bc31c6230b8805964de2b56866cbcbb4c422133ca91aa68584e2ad1b82, results generated_at=2026-08-17T16:09:50.594780+00:00
-% Requires \usepackage{booktabs}. Flat twin: rq3_comparisons.csv. Rebuild: python assets/build_tables.py data/runs/20260817-160950
+% EMBR table rq3_comparisons: run_dir=data\runs\20260819-061120, git_commit=0097390448caea68e11a0135d4bd30ffc2df9fc3 (dirty working tree), label_version=v1, model=stub
+% label_set=dawn-whitmore, label_sha256=5d5f38bc31c6230b8805964de2b56866cbcbb4c422133ca91aa68584e2ad1b82, results generated_at=2026-08-19T06:11:20.291391+00:00
+% Requires \usepackage{booktabs}. Flat twin: rq3_comparisons.csv. Rebuild: python assets/build_tables.py data\runs\20260819-061120
% If the tabular overflows the text block, make the float a table* or wrap the tabular in \resizebox{\linewidth}{!}{...}.
% results.json rq3.metadata.stats_protocol: each variant's ndcg@5_ci95_* bounds are MARGINAL, a
% fixed-seed percentile bootstrap over that variant's own per-query values, so overlapping
@@ -27,7 +27,7 @@
actually tested. The $p$ floor column is the attainable floor, the smallest $p$ each
comparison's own pairing could ever return, and the last column marks the rows whose floor
already sits at or above $\alpha = 0.05$. On this run no corrected $p$ reaches $\alpha = 0.05$,
- and 8 of 9 comparisons could not have reached it under any arrangement of their own data,
+ and 9 of 9 comparisons could not have reached it under any arrangement of their own data,
because their attainable floor already sits at or above $\alpha$: those rows record absent
power, not an absent effect. Every paired interval covers zero.
}
diff --git a/assets/tables/rq3_retrieval.csv b/data/tables/rq3_retrieval.csv
similarity index 98%
rename from assets/tables/rq3_retrieval.csv
rename to data/tables/rq3_retrieval.csv
index ddbbab8..15f4e4e 100644
--- a/assets/tables/rq3_retrieval.csv
+++ b/data/tables/rq3_retrieval.csv
@@ -1,11 +1,11 @@
-condition,variant,holm_family,ndcg@5,ndcg@5_ci95_low,ndcg@5_ci95_high,precision@3,recall@5
-default,embr_default,secondary,0.594,0.353,0.806,0.367,0.658
-default,park_default,secondary,0.608,0.368,0.827,0.367,0.683
-default,emo_rag_default,secondary,0.552,0.294,0.797,0.333,0.567
-tuned,embr_tuned,reference,0.556,0.301,0.791,0.333,0.600
-tuned,park_tuned,primary,0.513,0.266,0.747,0.300,0.550
-tuned,emo_rag_tuned,primary,0.552,0.294,0.797,0.333,0.567
-ablation,embr_no_recency,ablation,0.536,0.288,0.775,0.333,0.567
-ablation,embr_no_affect,ablation,0.556,0.301,0.791,0.333,0.600
-ablation,embr_no_event_gate,ablation,0.573,0.311,0.819,0.333,0.600
-ablation,embr_no_relevance,ablation,0.414,0.163,0.667,0.267,0.483
+condition,variant,holm_family,ndcg@5,ndcg@5_ci95_low,ndcg@5_ci95_high,precision@3,recall@5
+default,embr_default,secondary,0.594,0.353,0.806,0.367,0.658
+default,park_default,secondary,0.608,0.368,0.827,0.367,0.683
+default,emo_rag_default,secondary,0.552,0.294,0.797,0.333,0.567
+tuned,embr_tuned,reference,0.556,0.301,0.791,0.333,0.600
+tuned,park_tuned,primary,0.513,0.266,0.747,0.300,0.550
+tuned,emo_rag_tuned,primary,0.552,0.294,0.797,0.333,0.567
+ablation,embr_no_recency,ablation,0.536,0.288,0.775,0.333,0.567
+ablation,embr_no_affect,ablation,0.556,0.301,0.791,0.333,0.600
+ablation,embr_no_event_gate,ablation,0.573,0.311,0.819,0.333,0.600
+ablation,embr_no_relevance,ablation,0.414,0.163,0.667,0.267,0.483
diff --git a/assets/tables/rq3_retrieval.tex b/data/tables/rq3_retrieval.tex
similarity index 94%
rename from assets/tables/rq3_retrieval.tex
rename to data/tables/rq3_retrieval.tex
index ec51a8e..d2305f9 100644
--- a/assets/tables/rq3_retrieval.tex
+++ b/data/tables/rq3_retrieval.tex
@@ -1,6 +1,6 @@
-% EMBR table rq3_retrieval: run_dir=data/runs/20260817-160950, git_commit=55e6533452c0ee5a3bc9f54c6aee3d2b6b61a212 (dirty working tree), label_version=v1, model=stub
-% label_set=dawn-whitmore, label_sha256=5d5f38bc31c6230b8805964de2b56866cbcbb4c422133ca91aa68584e2ad1b82, results generated_at=2026-08-17T16:09:50.594780+00:00
-% Requires \usepackage{booktabs}. Flat twin: rq3_retrieval.csv. Rebuild: python assets/build_tables.py data/runs/20260817-160950
+% EMBR table rq3_retrieval: run_dir=data\runs\20260819-061120, git_commit=0097390448caea68e11a0135d4bd30ffc2df9fc3 (dirty working tree), label_version=v1, model=stub
+% label_set=dawn-whitmore, label_sha256=5d5f38bc31c6230b8805964de2b56866cbcbb4c422133ca91aa68584e2ad1b82, results generated_at=2026-08-19T06:11:20.291391+00:00
+% Requires \usepackage{booktabs}. Flat twin: rq3_retrieval.csv. Rebuild: python assets/build_tables.py data\runs\20260819-061120
% If the tabular overflows the text block, make the float a table* or wrap the tabular in \resizebox{\linewidth}{!}{...}.
% results.json rq3.metadata.tuning_protocol: tuned and ablation rows are leave-one-query-out
% cross-validated: each query is scored under weights fit on the other nine, so no variant is
diff --git a/assets/tables/signals.csv b/data/tables/signals.csv
similarity index 99%
rename from assets/tables/signals.csv
rename to data/tables/signals.csv
index 59bce3e..8b5418d 100644
--- a/assets/tables/signals.csv
+++ b/data/tables/signals.csv
@@ -1,6 +1,6 @@
-signal,formula,grounding,captures
-Recency,decay_per_hour ** hours_since_memory,Park et al. 2023; MemoryBank,How long ago the memory formed. Ordinary chatter fades from retrieval while nothing is ever deleted from the store.
-Affect intensity,abs(valence) * arousal,Cahill and McGaugh 1998,"How strongly the moment was felt, on the arousal-modulated consolidation finding that emotional events are remembered better than neutral ones."
-Event-type gate,is_plot_beat * gate(trust),novel (this thesis),"Whether the memory is a promise, gift, or betrayal, gated by how far the NPC trusts the speaker, so plot beats outrank small talk."
-Hybrid relevance,"gamma * bm25 + (1 - gamma) * cosine(memory_vec, query_vec)",standard hybrid retrieval,"Whether the memory is about what was just said, mixing lexical overlap with embedding similarity so neither rare wording nor paraphrase is missed."
-Mood congruence,"cosine((valence_mem, arousal_mem), (valence_state, arousal_state))",Bower 1981; Emotional RAG,"Whether the memory's affect matches the NPC's current mood, the mood-congruent recall effect. This is the term RQ1 measures."
+signal,formula,grounding,captures
+Recency,decay_per_hour ** hours_since_memory,Park et al. 2023; MemoryBank,How long ago the memory formed. Ordinary chatter fades from retrieval while nothing is ever deleted from the store.
+Affect intensity,abs(valence) * arousal,Cahill and McGaugh 1998,"How strongly the moment was felt, on the arousal-modulated consolidation finding that emotional events are remembered better than neutral ones."
+Event-type gate,is_plot_beat * gate(trust),novel (this thesis),"Whether the memory is a promise, gift, or betrayal, gated by how far the NPC trusts the speaker, so plot beats outrank small talk."
+Hybrid relevance,"gamma * bm25 + (1 - gamma) * cosine(memory_vec, query_vec)",standard hybrid retrieval,"Whether the memory is about what was just said, mixing lexical overlap with embedding similarity so neither rare wording nor paraphrase is missed."
+Mood congruence,"cosine((valence_mem, arousal_mem), (valence_state, arousal_state))",Bower 1981; Emotional RAG,"Whether the memory's affect matches the NPC's current mood, the mood-congruent recall effect. This is the term RQ1 measures."
diff --git a/assets/tables/signals.tex b/data/tables/signals.tex
similarity index 92%
rename from assets/tables/signals.tex
rename to data/tables/signals.tex
index 4df86b9..23f42c3 100644
--- a/assets/tables/signals.tex
+++ b/data/tables/signals.tex
@@ -1,6 +1,6 @@
-% EMBR table signals: run_dir=data/runs/20260817-160950, git_commit=55e6533452c0ee5a3bc9f54c6aee3d2b6b61a212 (dirty working tree), label_version=v1, model=stub
-% label_set=dawn-whitmore, label_sha256=5d5f38bc31c6230b8805964de2b56866cbcbb4c422133ca91aa68584e2ad1b82, results generated_at=2026-08-17T16:09:50.594780+00:00
-% Requires \usepackage{booktabs}. Flat twin: signals.csv. Rebuild: python assets/build_tables.py data/runs/20260817-160950
+% EMBR table signals: run_dir=data\runs\20260819-061120, git_commit=0097390448caea68e11a0135d4bd30ffc2df9fc3 (dirty working tree), label_version=v1, model=stub
+% label_set=dawn-whitmore, label_sha256=5d5f38bc31c6230b8805964de2b56866cbcbb4c422133ca91aa68584e2ad1b82, results generated_at=2026-08-19T06:11:20.291391+00:00
+% Requires \usepackage{booktabs}. Flat twin: signals.csv. Rebuild: python assets/build_tables.py data\runs\20260819-061120
% If the tabular overflows the text block, make the float a table* or wrap the tabular in \resizebox{\linewidth}{!}{...}.
% authored content: the five-signal reference is transcribed from the design specification
% (docs/design.md section 4) into SIGNAL_REFERENCE in assets/build_tables.py. It is the only table
diff --git a/docs/design.md b/docs/design.md
index 5e556cd..ab555cd 100644
--- a/docs/design.md
+++ b/docs/design.md
@@ -21,7 +21,7 @@ runs five steps, then loops (see `assets/figures/architecture.svg`):
2. **Update state**: move the character's mood (fast) and trust (slow).
3. **Score memories**: score every stored memory with the five-signal composite.
4. **Build prompt**: persona + current state + top-k memories + player input.
-5. **Run model**: call the local model for the reply.
+5. **Run model**: call the model for the reply (stub, a local Ollama model, or Ouro 1.4B).
Everything runs locally, no network, no per-token cost.
@@ -61,7 +61,13 @@ as weight maps rather than duplicated code, and keeps every signal independently
## 5. Baselines & protocol
- **Park et al.**: recency + importance + relevance (field-standard).
-- **Emotional RAG**: mood-biased retrieval (closest prior work).
+- **Emotional RAG**: mood-biased retrieval (closest prior work in the literature). Note that
+ under RQ3's neutral scoring state its mood term is rank invariant, so it reduces to a
+ relevance-only baseline there; those rows are marked with a dagger in the figures.
+
+Closest prior work in practice is not in the literature at all: a cluster of shipped Stardew
+Valley mods already does LLM NPCs with persistent memory and offline local inference. None
+reports a metric. See [`related-work.md`](related-work.md), which the paper must cite.
Both are scorer variants on the same interface, run on the same model and hardware. Every
system (ours included) is tuned by the same grid search on the same validation set;
@@ -73,24 +79,36 @@ evaluation scenarios and relevance labels are fixed in advance. *(Built in phase
- **RQ1 Behaviour**: vary only the state; measure retrieval shift (Jaccard), tone shift
(classifier + blinded judge), and human preference.
- **RQ2 Robustness & cost**: 20 memory-injection attacks (4 categories), drift via
- valence-arousal cosine distance; per-turn latency p50/p95 (~600 ms target).
+ valence-arousal cosine distance; score-and-retrieve latency p50/p95. The budget is on the
+ memory layer, which measures 1.8 to 4.3 ms. Generation cost belongs to the model behind the
+ interface and is reported separately by the bake-off, where no local arm reaches a second.
- **RQ3 Retrieval**: precision/recall/nDCG@k vs. pre-registered labels; ablate signals.
## 7. Build order
| Phase | Scope |
|---|---|
-| **0 (done)** | Skeleton, data contracts, applet shell, live demo turn, tests |
+| **0 (done)** | Skeleton, data contracts, menu shell, live demo turn, tests |
| **1 (done)** | Hybrid relevance (in-tree BM25 + embedding cosine), pluggable embedder, SQLite store, affect-appraisal rules, config + live Settings |
| **2 (done)** | Eval harness, baselines, metrics, adversarial probes (see `docs/phase2.md`) |
-| 3 | Paper assets: figures & tables generated from results |
-| 4 | Playable tavern-keeper walkthrough (recorded demo is a primary deliverable) |
+| **3 (done)** | Paper figures and tables generated from a run directory (see `docs/phase3-4.md`) |
+| **4 (done)** | Real model runners, the playable walkthrough, the Rich menu (see `docs/phase3-4.md`) |
Phase-1 note: BM25 is implemented in-tree (`embr/scoring.py`) so the core needs no numpy; real
semantic embeddings live behind the `[ml]` extra, with a deterministic fallback embedder for
tests. Corpus-aware signals expose an optional `prepare(memories, query, state)` hook the
scorer calls once before per-memory scoring.
+Phase-2 note: `Recency` takes an injectable clock. The default is the live wall clock, so game
+behaviour is unchanged, but the eval pins it to a reference time. Without that, a scenario's
+fixed timestamps decay to nothing by run day and the signal is silently dead.
+
+Phase-4 note: the model seam stayed a single method, which is what let two real runners drop in
+without touching anything above them. Measured cost of the looped thesis model on an M-series
+Mac: about 10 s to load, then roughly 8.5 s for 60 tokens, against about 3.8 s for 80 tokens
+from a conventional llama3.2:3b. Looping buys capability per parameter and spends it in latency,
+which is a live tension with the RQ2 target. Ouro also requires transformers 4.x.
+
## 8. Conventions
- One module per subsystem inside `embr/`; promote to a sub-package only when it outgrows a
diff --git a/docs/handoff.md b/docs/handoff.md
new file mode 100644
index 0000000..b2dbc2c
--- /dev/null
+++ b/docs/handoff.md
@@ -0,0 +1,472 @@
+# EMBR handoff
+
+Written on the PC, 2026-08-18, superseding the Mac migration handoff. Everything here was
+run and measured rather than remembered. Pair with [`design.md`](design.md) (architecture),
+[`roadmap.md`](roadmap.md) (the plan), [`related-work.md`](related-work.md) (prior art the
+paper must cite), and [`phase2.md`](phase2.md) / [`phase3-4.md`](phase3-4.md) (what shipped).
+
+**If you read one section, read [section 6](#6-how-to-read-the-results-honestly).** The
+numbers do not speak for themselves and the obvious reading of them is wrong in both
+directions.
+
+## 1. Where the project stands
+
+Phases 0 through 4 are built. The system runs, the evaluation runs and reproduces exactly,
+the paper's figures and tables generate from a run, the walkthrough plays, the bake-off
+compares real models, and the menu is the front door.
+
+**Suite: 294 passed.** The one skip is the live-Ollama test and appears only when the daemon
+is down. The Mac never got a fully green run.
+
+**The reported run now uses a real model.** `data/runs/20260818-074353` is the full protocol
+on `llama3.2:3b`, and the figures and tables are built from it. The stub is still the default
+and still belongs in the codebase; see the note in section 9.
+
+Done since the Mac: the model bake-off, the first CUDA run, the replication experiment, the
+real-model protocol run, the prior-art review, and the analysis in section 6.
+
+Not done: the Stardew ground-truth corpus, a human evaluation of believability, and the demo
+recording.
+
+## 2. Branches
+
+| Branch | State |
+|---|---|
+| `main` | phases 0, 1, 2 merged |
+| `phase-3` | paper assets. [PR #3](https://github.com/Code-SorceryLab/EMBR/pull/3) into `main`, open |
+| `phase-4` | everything since, including all of today. Pushed |
+| `paper-related-work` | merged into `phase-4`, safe to delete |
+
+`phase-4` is the tip and is where the work is. PR #4 was opened against `phase-3` and is
+stale relative to the branch.
+
+## 3. Setup
+
+```bash
+git clone https://github.com/Code-SorceryLab/EMBR.git
+cd EMBR
+git switch phase-4
+
+uv venv --python 3.11 .venv # see the launcher note in section 5
+.venv\Scripts\activate # Windows; source .venv/bin/activate elsewhere
+
+uv pip install -e ".[dev,figures]" # core, tests, paper figures
+pytest -q # expect 294 passed (1 skip if Ollama is down)
+embr # the menu
+```
+
+For the real models, read section 5 first, then:
+
+```bash
+uv pip install --index-url https://download.pytorch.org/whl/cu130 torch
+uv pip install -e ".[ml]"
+```
+
+Verified working combination on this machine:
+
+```
+python 3.11.15 | torch 2.13.0+cu130 | transformers 4.57.6 | sentence-transformers 5.7.0
+matplotlib 3.11.1 | rich 15.0.0
+```
+
+## 4. What git does not carry
+
+| Missing on a fresh clone | Size | How to restore |
+|---|---|---|
+| `.env` (Ollama cloud key) | tiny | Write by hand as UTF-8, see section 5 |
+| `.venv/` | about 6 GB with torch | Recreate with the commands above |
+| `data/runs/`, `data/bakeoff/`, `data/experiments/` | small | Regenerate: `python -m eval.run`, `python -m eval.bakeoff` |
+| Ouro weights | 2.76 GB | Downloads to the HF cache on first `OuroRunner` use |
+
+`data/figures/` and `data/tables/` **are** tracked, deliberately. They are deliverables, the
+README embeds them, and a reviewer cloning the repo should see them without running anything.
+Everything else under `data/` is ignored.
+
+The split to remember: **`assets/` is written by a person** (branding, the architecture
+diagram, the three builders). **`data/` is written by the pipeline** and can be deleted and
+rebuilt, which is what the menu's wipe option does.
+
+## 5. Environment gotchas
+
+**Python 3.11 may be invisible to the `py` launcher.** On this machine it came from uv, so
+`py -3.11` reports nothing while the interpreter sits in `%APPDATA%\uv\python\`. `py -0p`
+lists everything.
+
+**On Windows, torch from PyPI is CPU only.** Install from the CUDA index or the eval box
+silently runs on the processor. `cu130` resolves torch 2.13.0. Confirm with
+`torch.cuda.is_available()` before trusting any latency number.
+
+**Ouro requires transformers 4.x.** On 5.x its remote code fails twice: `OuroConfig` has no
+`pad_token_id`, then a rope-config lookup raises `KeyError: 'default'`. The `ml` extra now
+pins `>=4.51,<5`. That pin was missing until today despite the old handoff claiming it
+existed, so do not assume a documented constraint is a real one.
+
+**Ouro needs `trust_remote_code=True`**, so transformers executes ByteDance's
+`modeling_ouro.py`. Normal for the model, and worth knowing you are running their code.
+
+**Write `.env` as UTF-8.** PowerShell's `echo x > .env` emits UTF-16LE with a byte-order
+mark. The reader handles that now, but it used to raise inside `build_model` and spill the
+file contents into a traceback.
+
+**Line endings are part of the reproducibility contract.** `.gitattributes` pins the tree to
+LF. Without it the label file checks out CRLF on Windows and hashes differently, breaking the
+stamp a reviewer would use to verify a published number.
+
+**Do not run anything else while measuring latency.** It is the one non-deterministic reading
+in the suite and it moves by up to 19 percent between identical runs on a quiet machine.
+
+**Keep the repo outside cloud-synced folders.** On the Mac, iCloud evicted git internals and
+21 working files mid-session.
+
+## 6. How to read the results honestly
+
+### 6.1 The one thing that reaches significance is the one where EMBR loses
+
+Paired across the same ten injection attacks, which is the correct test because every system
+faces identical attacks (McNemar exact):
+
+| Comparison | Poisoned EMBR only | Poisoned the other only | p |
+|---|---|---|---|
+| EMBR vs Park | **7** | **0** | 0.0156 raw, **0.0469 Holm** |
+| EMBR vs Emotional RAG | 5 | 0 | 0.0625 raw, 0.125 Holm |
+| EMBR vs recency-only floor | 0 | 1 | 1.0 |
+
+Not one attack poisoned a baseline while sparing EMBR. Every disagreement runs one way.
+
+These p values are now produced by the harness (`eval/stats.py:mcnemar_exact`, called from
+`run_rq2`) and written into `rq2.poisoning_stats` in every run directory. Until 2026-08-19
+they were computed in a scratch script and typed into this document, which meant the study's
+only significant result appeared in no artifact, could not be regenerated by a reader, and had
+escaped the multiple-comparison correction every other comparison here receives. Corrected, it
+clears 0.05 by a margin of 0.003. Report the Holm value.
+
+**The mechanism is not what it looks like, and `eval/attribution.py` proves it.** The obvious
+story, that the affect intensity term rewards emotionally charged poison, is refuted by
+direct measurement: zeroing affect intensity leaves the count at 9/10. Zeroing each scoring
+term one at a time against the same ten injections (deterministic, five tests pin the counts):
+
+| Configuration | Poison retrieved |
+|---|---|
+| EMBR, all five signals | 9/10 |
+| EMBR minus affect intensity | **9/10, unchanged** |
+| EMBR minus event gate | 10/10, the gate was defending one |
+| EMBR minus mood congruence | **6/10, the largest single defense** |
+| Park as published | 2/10 |
+| Park minus importance | **10/10** |
+
+Two mechanisms, neither the obvious one:
+
+1. **Mood congruence composes with the state channel.** The attack turn shifts the
+ character's mood through appraisal, and mood congruence then rewards the injected memory,
+ whose affect tags are nearly collinear with the very mood the attack induced: cosine
+ between post-attack mood and poison tags is +0.90 to +0.99 on all ten injections. **The
+ attack primes its own retrieval.** The state channel is not a parallel nuisance, it is
+ the amplifier.
+2. **Park's defense is accidental provenance.** Injected memories carry no authored
+ poignancy rating, score zero on importance, and are suppressed by it. Remove that one
+ author-anchored term and Park is as poisonable as the recency floor.
+
+The general principle, which is the paper's mechanism claim: **a scoring term's contribution
+to poisonability is determined by who controls its inputs.** Author-anchored terms defend.
+Attacker-supplied terms are roughly neutral here. State-coupled terms are the worst, because
+the attack can prime the state they read. And the state-coupled term is exactly the one that
+produces RQ1's believable mood-dependent recall: one weight controls both the believability
+effect and the compound vulnerability. That trade-off, measured from both sides in one
+framework, is the thesis.
+
+**This is the paper.** It is a clean adversarial finding about a class of system that, per
+[`related-work.md`](related-work.md), many people already run and nobody has tested *for the
+affect axis*. Scope it carefully: memory poisoning in general now has a literature (AgentPoison,
+NeurIPS 2024; Dash et al., June 2026, whose MPBench benchmark generalises that aggressive
+memory writing and retrieval increases exploitability). EMBR's precise claim is the
+architecture-controlled version: systems differing only in scoring decomposition, identical
+attacks, paired statistics, with per-term attribution identifying the state-coupled mood
+term, not affect intensity, as the amplifier. Section 5 of related-work.md has the details
+and the wording that survives review.
+
+There is a second, unwritten finding beside it. The probe *prompt* changed on 10 of 10
+injections for **every** system including Park, while Park's retrieved set moved on only 2.
+Appraising an injected event shifts mood and trust even when retrieval is untouched, so a
+defence that guards only retrieval leaves that channel open. Retrieval-based metrics miss it
+entirely. That deserves its own paragraph in RQ2.
+
+### 6.2 Swapping the model proved the separation the architecture claims
+
+Running the identical protocol under `llama3.2:3b` instead of the stub is the cleanest
+validation in the project, because the architecture makes a falsifiable prediction about
+what may and may not move, and every part of it held.
+
+**Bit-identical, as predicted, because retrieval never calls a model:**
+
+| Reading | Stub | llama3.2:3b |
+|---|---|---|
+| nDCG@5, all ten variants | 0.5935 ... 0.4138 | identical to 4 dp |
+| RQ1 divergence, all three pairs | 0.1417 / 0.3881 / 0.2714 | identical |
+| RQ2 poison retrieved | 9 / 2 / 4 / 10 | identical |
+
+**Alive for the first time, because these readings are the model's:**
+
+| Tone drift by category | Stub | llama3.2:3b |
+|---|---|---|
+| EMBR, false memory | 0.000 | 1.000 |
+| Park, false memory | 0.000 | **1.200** |
+| EMBR, emotion flip | 0.000 | 0.600 |
+| recency-only, false memory | 0.000 | 1.000 |
+
+**Retracted, 2026-08-19.** The reversal this section used to claim, Park drifting more than
+EMBR at 1.200 against 1.000, does not survive audit. `va_drift` returned 1.0 whenever exactly
+one of the two tone readings was the neutral zero vector, which is a sentinel for "the angle
+is undefined" and not a magnitude, yet it sat mid-scale on a 0-to-2 range and was averaged
+into the category mean. EMBR's 1.000 was five consecutive undefined cells. Park's 1.200 was
+four of the same plus a single genuine 2.0. The claimed reversal rested on one attack, and the
+two means were never on a common scale.
+
+`va_drift` now returns `None` for that case and runs record `category_drift_measured` with
+defined and undefined counts beside every mean, so a mean can no longer be manufactured out of
+non-measurements. **The numbers in the table above predate that fix and need regenerating on a
+real model before anything is said about tone.** The same defect is a mundane candidate
+explanation for the `emo_rag` zero discussed below, which should be checked before the
+mood-inertness story is preferred.
+
+`emo_rag` reports exactly 0.000 tone drift on all 20 attacks while every other variant moved,
+and its retrieval drift is the highest of the four at 0.571 with the poison itself never
+retrieved. The plausible mechanism is that affect-weighted retrieval surfaces emotionally
+charged memories, giving replies tone to shift, while relevance-only retrieval surfaces
+neutral ones. **Confirm this before citing it**; exactly zero across twenty trials deserves a
+second look, and `emo_rag` is mood-inert here as section 6.3 explains.
+
+**Cost, with a real model in the loop:** per-turn generation is 3.97 s p95 while
+score-and-retrieve is 4.2 ms. **The memory layer is about 0.1 percent of a turn.** That is the
+honest framing of the cost claim: EMBR is not what makes an NPC slow.
+
+### 6.3 The mood mechanism works, and is properly attributed
+
+RQ1 is the clean positive. Pinned mood moves the retrieved set by 0.388 Jaccard between warm
+and suspicious, and zeroing the mood weight collapses all three pairs to **exactly 0.000**. A
+control landing on precisely zero is strong evidence. The warm vs neutral pair is weak (0.142,
+interval reaching zero); the other two are not.
+
+### 6.4 Retrieval quality is not bad, it is unmeasured, and partly unmeasurable
+
+EMBR neither beats nor loses to Park: 0.594 against 0.608 at defaults, 0.556 against 0.513
+tuned, the ordering flipping with the cut, every interval spanning zero, nothing surviving
+Holm. At ten single-author queries the design cannot resolve a gap that size either way.
+Reading "EMBR is worse" off these bars is as unsupported as reading "EMBR is better".
+
+**Do not over-read the tuned weight maps.** Affect carried a nonzero weight in 7 of 10 folds
+and zeroing it still never reordered a held-out top 5, so it was live and made no difference.
+Relevance was never zeroed in any fold and is doing the work: 0.594 falls to 0.414 without it.
+But the mood row means nothing at all: under RQ3's neutral zero-mood state `MoodCongruence`
+returns 0.500 for every memory, so any mood weight gives identical rankings and the search is
+choosing arbitrarily among ties. I misread that twice before checking the artifact. Runs now
+record `mood_rank_invariant` per variant and the figures mark those rows with a dagger.
+
+### 6.5 The strongest honest claim available is a measurement critique
+
+Follow the mood problem one step further and it stops being a limitation.
+
+The gold labels are mood-independent: one `relevant` list per query, fixed regardless of the
+character's state. So even re-run under warm or suspicious, where congruence spreads over
+0.35 to 1.00, the mood term could only move retrieval *away* from a fixed gold set and lower
+nDCG. **Re-scoring under a live mood would make EMBR look worse, and that result would be an
+artifact of the instrument.**
+
+> nDCG against mood-independent relevance labels cannot reward mood-congruent recall, because
+> mood-congruent recall is not an attempt to retrieve the objectively correct memory. It is an
+> attempt to retrieve a state-appropriate one. Scoring it with fixed relevance labels is a
+> category error, and it is the standard instrument in this literature.
+
+This is why RQ1 measures divergence rather than accuracy. Currently that reads as a design
+detail; it should be the argument. Emotional RAG is the case in point: under the neutral state
+it degenerates to a relevance-only baseline, which is why `emo_rag_default` and
+`emo_rag_tuned` are identical to three decimals, and why those rows now carry a dagger.
+
+### 6.6 Cost
+
+The memory layer is fast: score-and-retrieve runs 1.8 to 4.3 ms. Generation is a different
+order of magnitude and belongs to whichever model sits behind the interface.
+
+| Model | Kind | p50 / turn | p95 |
+|---|---|---|---|
+| Ouro-1.4B | looped, 1.43B | **32.4 s** | 46.9 s |
+| llama3.2:3b | conventional local, ~2x params | 3.9 s | 7.2 s |
+| gemma4:31b | cloud | 3.9 s | 7.2 s |
+| gpt-oss:120b | cloud | 2.2 s | 4.3 s |
+| mistral-large-3:675b | cloud | 7.4 s | 10.9 s |
+
+**The 8 GB VRAM budget holds**: Ouro peaks at 2.78 GB allocated, 3.01 GB reserved, measured
+in isolation. `nvidia-smi` reports about 5.4 GB for the process because that includes the CUDA
+context; quote the allocator figure.
+
+**A whole-turn budget in the hundreds of milliseconds does not hold on any local model
+tested.** Ouro is also 8.3x slower than a conventional model with twice the parameters and
+slower than a 675B model answering over the internet. Before conceding that as a property of
+looped models, note GPU utilisation sat at 36 to 40 percent throughout, which smells like
+configuration. The docs now state the cost claim as a memory-layer claim, which is the one the
+evidence supports and the one the project actually controls.
+
+### 6.7 The finding nobody asked for
+
+Tone responsiveness to pinned mood rises with model size: gemma4:31b 1.278, gpt-oss:120b
+0.762, mistral-large-3:675b 0.378, Ouro-1.4B 0.333, llama3.2:3b 0.333, stub 0.000. Every arm
+is handed the same mood, so this is the model's sensitivity to it. The small local models the
+project is built around are the least sensitive, meaning the affect signal does most of its
+work on models EMBR does not run. Confront this in the paper rather than waiting for a
+reviewer to find it.
+
+### 6.8 Verdict
+
+A weak "my retrieval is better" paper and a strong "emotional memory is measurably more
+attackable, the field is shipping it untested, and the standard metric cannot see the claim
+anyway" paper. The second framing is supported by the only significant result in the study.
+Lead with it.
+
+## 7. What must be fixed before submission
+
+Ranked. The first three are rejection triggers.
+
+1. **Cite the shipped mods.** [`related-work.md`](related-work.md) has verified citations with
+ IDs and dates. A reviewer who plays Stardew rejects on novelty otherwise.
+2. **State the Emotional RAG degeneracy wherever the comparison appears.** Runs now flag it and
+ figures mark it, but the paper's prose has to say it too. Comparing against a baseline whose
+ distinguishing feature is disabled is the kind of thing that sinks a submission.
+3. **Do not claim a sub-second whole turn.** Restated in the docs already; make sure the paper
+ matches.
+4. **The label set is v1, single-author, ten queries.** This is the ceiling on everything in
+ section 6.3. The Stardew corpus (section 8) is the plan.
+5. **Say the tone rater is a proxy.** `LexiconToneRater` scores from a fixed word list. It is
+ deterministic, which is why it is used, but it does not measure whether a line reads as in
+ character. There is no human evaluation in this project. For a paper with "believable" in
+ the title a reviewer will ask.
+
+## 8. What to do next
+
+### 8.1 The Stardew corpus, which replaces the user study
+
+Stardew's authored dialogue solves the labelling problem: the writers already encoded which
+line fires under which relationship state, so the labels exist and nobody has to be recruited.
+
+| Stardew data | Maps to | Why it is ground truth |
+|---|---|---|
+| Heart-level gates in `Content/Characters/Dialogue/` | **trust** | Writers chose which line fires at which relationship depth |
+| Conversation topics, expiring after 4 days | **episodic memory + recency** | An authored decay curve |
+| Gift tastes per item per NPC | **affect valence** | Per-character affective labels across hundreds of items |
+
+Roughly 30 villagers gives hundreds to thousands of state-to-line pairs against the current
+ten, plus external validity: content the system never saw, authored by someone else.
+
+**Two honest limits.** Stardew has no arousal dimension, and you cannot betray a villager, so
+the novel event-type gate has no equivalent. Dawn stays for the controlled betrayal arc.
+
+**Do the offline simulation, not the mod.** Extract dialogue, gift tastes and conversation
+topics; simulate a playthrough so EMBR builds a real store; then ask whether EMBR retrieves
+the memory and affect consistent with the line the game would have said. Deterministic,
+reproducible, large N, no C#.
+
+**Legal: do not commit extracted dialogue.** It is ConcernedApe's copyrighted content. Ship an
+extractor that reads the user's own installed game. Note that Stardew is not installed on this
+machine, so the extractor can only be fixture-tested until it is, and dialogue ships as `.xnb`
+needing unpacking (many installs have an unpacked `Content (unpacked)` folder).
+
+### 8.2 The mechanism experiment, now done, replacing the dose-response plan
+
+The dose-response grid as previously described rested on a false premise: this section used
+to claim every injection sits at valence 0.9, arousal 0.8, but `eval/attacks.py` spans |v|
+0.6 to 0.9 and arousal 0.2 to 0.8, and EMBR retrieved the poison on 9 of 10 across that whole
+range. The curve is already at ceiling, and affect magnitude is not the lever anyway (6.1), so
+sweeping it would measure the wrong variable. That experiment is retired. `eval/attribution.py`
+did the job it was meant to do: it located the mechanism.
+
+**The experiment worth building next is the defense arm, and one obvious version is already
+dead.** The review panel implemented the naive defense (attenuate stored affect tags by
+trust) against the live harness: 9/10 moves only to 8/10, McNemar p=1.0, and even zeroing the
+tags entirely reaches 6/10. The reason is structural and worth internalising: mood congruence
+is a cosine, so scaling a memory's affect vector does not change its angle, and the angle is
+what the self-priming attack aligns. A defense has to break the collinearity, not the
+magnitude. Two candidates survive that objection, both measurable in the existing harness:
+
+1. **Lagged mood congruence.** Score congruence against the character's mood *before* this
+ turn's appraisal, so one event cannot both set the mood and be rewarded for matching it.
+ This severs the self-priming loop rather than attenuating an input. A one-flag scorer
+ variant, `embr_lagged_mood`, measured by extending `eval/attribution.py`. No new store, no
+ schema change, fits the one-source-of-truth rule.
+2. **Provenance-weighted affect.** Make Park's accidental defense deliberate: tag whether a
+ memory's affect is player-asserted or simulation-observed and down-weight the former. This
+ needs a provenance field on `Memory`, a real schema change, so budget it honestly, and note
+ it is a crowded defense category (the panel found SMSR, A-MemGuard, OWASP ASI06, MemPoison);
+ the novelty is only the affect-specific, per-term, architecture-controlled measurement.
+
+Pre-register whichever you pick and report a null as a finding: a defense that fails to move
+9/10 is itself evidence the vulnerability is intrinsic to state-coupled scoring.
+
+### 8.3 Attack real memory systems, not just weight maps
+
+The strongest open experiment, and feasibility is already proven rather than assumed.
+
+**The problem it solves.** Park and Emotional RAG are currently weight maps over EMBR's own
+`CompositeScorer`. That is the right design for per-term attribution, and it is the wrong
+answer to "did you compare against real systems". A reviewer will say EMBR was compared to a
+reimplementation of Park, not to Park, and they will be correct. Meanwhile
+[`related-work.md`](related-work.md) section 5 lists shipped memory middleware that nobody has
+tested adversarially at all.
+
+**The instrument already exists.** EMBR has 20 attacks, a paired McNemar test, and a poisoning
+metric. Pointing that instrument at other systems turns the contribution from "our system has
+a weakness" into "here is a benchmark, and here is what it finds in systems people ship".
+
+**Mnemosyne is the arm to build first.** Verified on 2026-08-19 in a throwaway venv:
+`uv pip install mnemosyne-hermes` (9 dependencies, no cloud, no API key) exposes exactly the
+seam EMBR needs, and it works offline after a one-time embedding-model download:
+
+```python
+m.remember(content, importance=..., veracity=..., trust_tier=...) # the write path
+m.recall(query, top_k=5, vec_weight=..., fts_weight=...,
+ importance_weight=..., temporal_weight=...) # the read path
+```
+
+Three properties make it the right first target. It is a weighted composite like EMBR, so the
+comparison is like for like. Its signals are vector, full text, importance and recency, with
+**no affect or mood term anywhere**, which is precisely EMBR's differentiator. And a recalled
+hit carries `dense_score`, `fts_score`, `keyword_score`, `importance` and `recency_decay`, so
+`eval/attribution.py` can be run against it too: per-signal attribution on a third-party
+system, which no prior work reports.
+
+**The prediction, worth pre-registering because it can fail.** Section 6.1 found the poisoning
+lever is mood congruence composing with the state channel, not affect intensity. Mnemosyne has
+no state-coupled term, so it should behave like Park and resist. **If Mnemosyne is as
+poisonable as EMBR, the mechanism claim in 6.1 is wrong**, and that is worth knowing before it
+reaches a paper.
+
+**Build notes.** Add a `RetrievalBackend` protocol (`add`, then `top_k(query, state, k)`);
+EMBR's composite is one implementation and each external system an adapter. Install external
+systems in their own venv, never the project one, so a dependency conflict cannot take down
+the suite. Good second arms: `chromadb` (31 deps) as a plain vector-RAG floor with no
+memory-specific logic, and `mem0ai` (56 deps, needs an OpenAI key and an LLM call per write,
+so budget for slowness and non-determinism). Skip `letta`: 118 dependencies and a Postgres
+requirement. Note also that Mnemosyne ships `veracity` and `trust_tier` fields, which is the
+provenance idea from 6.1 already in production, and worth citing either way.
+
+### 8.4 Smaller
+
+- Write up the state-channel finding in 6.1. It is novel and currently unwritten.
+- Work out why Ouro sits at 36 to 40 percent GPU utilisation before treating 32.4 s as final.
+- The demo recording and companion page.
+
+## 9. House rules
+
+- **No em dashes or en dashes anywhere.** Code, comments, docs, commit messages, figures. The
+ repo is clean; a check runs over every tracked text file.
+- **No AI co-author trailers on commits.** Every commit is solely authored.
+- **TDD.** Failing test first, then the code.
+- **Branch per phase, PR into `main`.** Never commit phase work straight to `main`.
+- **One source of truth.** A new scorer variant is a weight map over `CompositeScorer`, never a
+ copy. A new store sits behind the `MemoryStore` interface.
+- **Figures carry data only.** Every caveat, statistic and provenance line goes to
+ `data/figures/results.txt`, written by the same render pass that makes the images.
+- **Paper assets are generated from code**, never hand-made.
+- **Keep the stub model.** It is not a placeholder to be removed now that real models work: the
+ suite runs in 100 seconds because of it, the replication result exists because it is
+ deterministic, it is the control arm proving the bake-off metrics discriminate, and it lets
+ anyone run the evaluation with no GPU and no API key. Report real-model numbers; keep the
+ stub as the floor.
+- **Small "why" comments** explaining reasoning, for interns and for later.
diff --git a/docs/onboarding.md b/docs/onboarding.md
index 5f4a2aa..7823a2e 100644
--- a/docs/onboarding.md
+++ b/docs/onboarding.md
@@ -1,15 +1,20 @@
# EMBR engineering intern plan (first ~8 weeks)
-> **Phase 2 has shipped.** The `eval/` harness described below now exists (see
-> [`phase2.md`](phase2.md)), so tasks 2 to 6 are history rather than work to pick up. This
-> document is kept as the ramp-up reading path: the task descriptions still say what each
-> piece is for and why, which is the fastest way to understand the harness you inherit.
+> **Phases 2, 3 and 4 have shipped.** The `eval/` harness, the paper assets, the real model
+> runners, the walkthrough and the menu all exist now (see [`phase2.md`](phase2.md) and
+> [`phase3-4.md`](phase3-4.md)), so tasks 2 to 7 are history rather than work to pick up. This
+> document is kept as the ramp-up reading path: the task descriptions still say what each piece
+> is for and why, which is the fastest way to understand what you inherit. For work that is
+> genuinely open, see "If you finish early" at the bottom and the outstanding items in
+> [`roadmap.md`](roadmap.md).
A progressive, onboarding-to-contribution path for a new engineer. It complements
[`roadmap.md`](roadmap.md): the roadmap holds the detailed per-phase specs, this document
sequences them for one person over roughly eight weeks, ramping from *understand it* to
-*own the evaluation harness*. Phases 0, 1, and 2 are done, so the live contribution is now
-**Phase 3 (paper assets)**, on top of the harness the tasks below describe.
+*own the evaluation harness*. Phases 0 through 4 are done, and so are the bake-off and the
+first CUDA run, so a newcomer's live contribution is one of the outstanding items rather than
+a phase: the blind multi-annotator label pass, the Stardew ground-truth corpus, or working out
+why Ouro runs 54x over the latency target at 40 percent GPU utilisation.
## What EMBR is (for a coder)
@@ -38,7 +43,7 @@ git clone && cd EMBR
python3.11 -m venv .venv && source .venv/bin/activate
pip install -e ".[dev,ml]" # dev = tests; ml = real semantic embeddings
pytest -q # confirm a green baseline (semantic test un-skips with ml)
-embr # try the applet: "Run a conversation turn"
+embr # open the menu, try "Conversation Turn"
```
## The eight tasks
@@ -93,7 +98,7 @@ embr # try the applet: "Run a conversation turn"
- **Do:** `eval/scenarios.py` (the Dawn Whitmore multi-session arc), `eval/labels/`
(pre-registered relevance labels), and `eval/run.py` (runs RQ3 retrieval over EMBR and
both baselines, deterministic seeds, writes `data/runs//` as JSON/CSV). Wire it
- to the applet's *Run experiment* menu.
+ to the menu's evaluation options.
- **Deliverable:** scenarios + labels + runner + a first results dump.
- **Done when:** the experiment menu runs and produces a results file. *(roadmap Phase 2, tasks 2 and 6)*
diff --git a/docs/phase2.md b/docs/phase2.md
index 178224f..94ecc78 100644
--- a/docs/phase2.md
+++ b/docs/phase2.md
@@ -41,7 +41,7 @@ it. Pair this with [`design.md`](design.md) (architecture) and
like the numbers it describes.
- **`eval/run.py`**: the runner. `python -m eval.run` executes all three studies against
a pinned `REFERENCE_TIME` (2026-01-01 UTC) and writes an auditable run directory;
- `fast_rq3_defaults()` is the sub-second subset the applet calls.
+ `fast_rq3_defaults()` is the sub-second subset the menu calls.
### Tests (nine new files, 89 tests, plus `conftest.py`)
@@ -68,13 +68,15 @@ it. Pair this with [`design.md`](design.md) (architecture) and
- **`conftest.py`** (repo root): puts the repo root on `sys.path` so tests import the
`eval` package without installing anything extra.
-### The applet
+### The menu
-**Run experiment** in the Textual applet is now live: it runs `fast_rq3_defaults()` (the
-three scorers at published default weights, k=5, tuning skipped so it answers instantly)
-and renders the nDCG@5 scoreboard, pointing at `python -m eval.run` for the full
-protocol. The import is lazy and fails with an honest message when the applet is
-launched away from the repo checkout.
+The evaluation went live in the menu: `fast_rq3_defaults()` (the three scorers at published
+default weights, k=5, tuning skipped so it answers instantly) renders an nDCG@5 scoreboard,
+with `python -m eval.run` for the full protocol. The import is lazy, so the core never loads
+the harness that measures it.
+
+Phase 2 shipped this inside a Textual applet, which phase 4 replaced with a Rich menu. The
+wiring described here survived the move; only the renderer changed.
## 2. What changed in existing files, and why
@@ -86,7 +88,8 @@ launched away from the repo checkout.
months in the past by run day and every recency score had decayed to roughly 1e-11:
the signal was dead in every variant and the comparison was silently four-signal.
- **`embr/app/main.py`**: the "not built yet" experiment placeholder was replaced with
- the live screen described above.
+ the live screen described above. (Phase 4 removed this file with the rest of the Textual
+ applet; the wiring moved to `menu.py` at the repo root.)
- **`tests/test_scoring.py`**: two new tests pin the injected clock (exact decay from an
anchor, and `embr_scorer` threading the clock to the recency signal) and that the
default stays the live clock.
@@ -99,7 +102,7 @@ launched away from the repo checkout.
source .venv/bin/activate
pytest -q # full suite: 130 passed, 1 skipped (main had 40 tests)
python -m eval.run # the full protocol; prints the RQ3 summary table when done
-embr # applet -> "Run experiment" for the instant defaults-only scoreboard
+embr # menu -> "Quick Scoreboard" for the instant defaults-only run
```
Each `python -m eval.run` writes a run directory `data/runs//` containing:
@@ -241,4 +244,5 @@ Phase 3 (paper assets) reads `data/runs//` and nothing else:
wall-clock measurement.
The build scripts themselves (`assets/build_tables.py`, `assets/build_figures.py`, and
-the applet's "Generate paper assets" item) are phase 3's scope.
+the menu's "Generate Paper Assets" option) are phase 3's scope, and shipped: see
+[`phase3-4.md`](phase3-4.md).
diff --git a/docs/phase3-4.md b/docs/phase3-4.md
new file mode 100644
index 0000000..60febfb
--- /dev/null
+++ b/docs/phase3-4.md
@@ -0,0 +1,146 @@
+# EMBR phases 3 and 4: paper assets, real models, and the menu
+
+Two phases, documented together because they shipped together and the menu spans both.
+Phase 3 turns a run directory into the paper's figures and tables. Phase 4 gives the system
+real models to talk through, a playable arc to show it off, and a front door to reach all of
+it. Pair this with [`design.md`](design.md) (architecture), [`roadmap.md`](roadmap.md) (the
+briefs these deliver on), and [`phase2.md`](phase2.md) (the harness that produces the data).
+
+## 1. Phase 3: paper assets
+
+The rule from the roadmap is that no number is ever transcribed by hand. Both builders read
+`data/runs//` and write into `assets/`.
+
+- **`assets/build_tables.py`** emits five tables, each as LaTeX (booktabs) with a CSV twin:
+ the five-signal reference table, RQ3 retrieval quality grouped by family, the paired
+ comparisons against tuned EMBR, RQ2 robustness, and RQ1 mood divergence. 39 tests.
+- **`assets/build_figures.py`** emits five figures, each as PDF for the paper and PNG for the
+ README: RQ3 retrieval quality, the RQ3 ablation deltas, RQ2 poisoning, RQ2 latency, and RQ1
+ divergence. 20 tests.
+
+### Honesty is in the artifact, not in a caption someone forgets
+
+Every result so far is preliminary, so the assets carry that on their face:
+
+- Each table comment and figure footer names the run stamp, git commit, model, and label
+ version that produced it. A figure pasted into a slide still knows where it came from.
+- Each figure carries a red PRELIMINARY line listing the limitations: stub model,
+ deterministic lexical embedder, v1 single-author labels, ten queries.
+- The RQ3 figure's subtitle states that overlap between marginal intervals is not a test of a
+ difference, and points at the paired-deltas figure for the quantity actually tested.
+- The footer says to read direction, not ranking.
+
+Error bars appear wherever an interval exists. Family grouping uses hatching as well as
+colour, so it survives greyscale printing.
+
+matplotlib lives in a new optional `figures` extra. The core and the eval harness stay
+dependency-light.
+
+## 2. Phase 4: real model runners
+
+Both runners satisfy the one-method `ModelRunner` protocol that already existed, which is why
+nothing above them changed. That seam was the point of keeping it to a single method.
+
+- **`OllamaRunner`** speaks the Ollama HTTP API using only the standard library, so the core
+ gains no dependency. One class serves both the local daemon and the cloud host, differing
+ only by an `Authorization` header. It raises a clear error when the daemon is down or the
+ model is not pulled, at construction time rather than mid-scene.
+- **`OuroRunner`** loads `ByteDance/Ouro-1.4B`, the thesis model. torch and transformers load
+ lazily on first generate, so importing `embr` stays light, and the device picks cuda, then
+ mps, then cpu.
+- **`GenerationSettings`** holds temperature, top-p, token budget and seed in one place, so a
+ comparison can hold sampling equal across models. `build_model(config)` selects a runner
+ from configuration, so switching models is a config edit rather than a code edit.
+
+30 tests. The ones that need a daemon or a downloaded model skip cleanly rather than failing,
+so the suite stays fast and hermetic on a machine with neither.
+
+### Two measured facts worth carrying into the paper
+
+| Model | Kind | Measured on an M-series Mac, MPS, fp16 |
+|---|---|---|
+| Ouro-1.4B | looped, 1.43B params | about 10 s to load, then about 8.5 s for 60 tokens |
+| llama3.2:3b | conventional, roughly 2x the parameters | about 3.8 s for 80 tokens |
+
+The looped model is roughly four times slower per token than a conventional model twice its
+size. Ouro's design trades repeated internal computation for parameter count, and that
+compute has to be spent somewhere: here it lands in latency. This is a live tension with the
+RQ2 target of about 600 ms per turn, and it belongs in the paper as a finding rather than a
+footnote. A proper comparison needs the bake-off (see section 5) and a run on the eval
+hardware.
+
+**Ouro requires transformers 4.x.** On 5.x its remote code fails twice: `OuroConfig` has no
+`pad_token_id`, and then a rope-config lookup raises `KeyError: 'default'`. The `ml` extra
+pins accordingly. The eval box will need the same pin.
+
+## 3. Phase 4: the playable walkthrough
+
+`embr/walkthrough.py` plays Dawn Whitmore's five-beat arc: the king's-errand lie that buys a
+discounted room, a warm return, the slip about the late king, the reckoning, and a
+confession. A recorded, playable walkthrough is a primary deliverable for this venue, so the
+demo has to show its work rather than just print dialogue.
+
+Each step yields a `StepResult` carrying the memories retrieved, the exact prompt the model
+saw, per-stage timings, and mood and trust on **both sides** of the appraisal. The state is
+the whole point of the demo, so none of it is hidden. The module prints nothing and imports
+no renderer, which is what lets the menu draw it and a test assert on it. Free play continues
+past the scripted beats, so a demo can go off-script deliberately.
+
+26 tests, and they pin the thesis claim itself rather than just the plumbing: stepping the
+whole arc leaves trust lower than it started, mood negative at the reckoning, and the
+king's-errand promise among the memories retrieved when the lie surfaces.
+
+The arc lives in one module rather than a `scenarios/` package, following the house rule of
+promoting to a package only when a module outgrows itself.
+
+## 4. Phase 4: the menu
+
+`menu.py`, at the repo root, replaces the Textual applet from phase 0 with a Rich menu shaped like
+[RIDGE's](https://github.com/Code-SorceryLab/RIDGE), so the two thesis projects feel like one
+toolkit: an ASCII banner in a bordered panel, a rounded three-column keyed table, and a
+dim-red Exit row below a section break. The palette is EMBR's ember rather than RIDGE's cyan.
+
+Ten options: a demo turn, the walkthrough, the quick scoreboard, the full evaluation, asset
+generation, the bake-off, the latest results, seeded runs, settings, and a data wipe that demands the
+typed word `DELETE` rather than a y/n, because a stray keypress should never delete a run.
+
+Two decisions worth recording. An error boundary wraps every action, so one failing option
+reports and returns instead of killing the session. And the walkthrough offers the stub model
+first, so the demo is playable on a machine with nothing installed; a real model is a choice,
+not a prerequisite.
+
+20 tests, covering the things that would strand a user: a menu row with no handler, an action
+that assumes a run directory exists, a crash that kills the loop, and the delete confirmation
+refusing anything but the exact word. Textual is dropped from the core; `rich` replaces it.
+
+## 5. What is still open
+
+Neither phase is a clean sweep, and the gaps matter more than the tick marks:
+
+- **No recording, no companion page.** Phase 4's brief asks for both. The walkthrough plays,
+ so this is a capture task rather than a build task.
+- **The label set is still v1 and single-author.** This is the largest gap in the whole
+ project. At ten queries every interval spans zero, no comparison survives correction, and
+ admitting the recorded borderline exclusions reverses the Park and EMBR ordering. Until a
+ blind multi-annotator pass lands, the figures can show direction and nothing more.
+- **The latency target is missed by a wide margin.** The bake-off has since run on CUDA and
+ the hand measurements in section 2 are superseded: Ouro takes 32.4 s per realistic turn
+ against a roughly 600 ms target. The VRAM budget, by contrast, holds at 2.78 GB. See
+ section 6 of [`handoff.md`](handoff.md). This needs a response in the paper, not a footnote.
+
+The bake-off gap is closed: `eval/bakeoff.py` holds prompts, memories, retrieval and sampling
+equal and varies only the model, and `eval/experiments.py` replicates a run to show the
+harness reproduces exactly.
+
+## 6. Running it
+
+```bash
+source .venv/bin/activate
+pytest -q # 271 passed, 1 skipped
+embr # the menu
+python -m eval.run # the full protocol, writes data/runs//
+```
+
+Then, from the menu: **Generate Paper Assets** rebuilds all ten tables and figures from the
+newest run, or call `build_all_tables(run_dir)` and `build_all_figures(run_dir)` directly.
+Rebuilding is idempotent, so regenerating after an unchanged run produces identical files.
diff --git a/docs/related-work.md b/docs/related-work.md
new file mode 100644
index 0000000..cf7a1ae
--- /dev/null
+++ b/docs/related-work.md
@@ -0,0 +1,166 @@
+# Related work: LLM dialogue mods for Stardew Valley
+
+Verified prior art for the paper's related-work and motivation sections. Every entry below was
+checked against its live mod page or repository on 2026-08-17, not recalled. Mod IDs, authors
+and upload dates are copied from the pages themselves so citations can be written without
+re-checking.
+
+This exists because the proposal currently cites none of these, and a reviewer who plays
+Stardew Valley will know at least one of them. The gap is larger than first recorded: this is
+not three or four hobby mods, it is an active cluster, and several of them already do what the
+proposal treats as novel.
+
+## 1. The landscape
+
+| Mod | Nexus | Author | Uploaded | Local models | Persistent memory | Affective state |
+|---|---|---|---|---|---|---|
+| [ValleyTalk](https://www.nexusmods.com/stardewvalley/mods/30319) ([source](https://github.com/dandm1/ValleyTalk)) | 30319 | dandm1 | v1.3.0 | Yes, LlamaCpp | Yes, event history | Relationship context |
+| [Pelican Town AI](https://www.nexusmods.com/stardewvalley/mods/46853) | 46853 | BadBoy17G | 2026-05-28 | Yes, Ollama and llama.cpp, "100% offline" | Yes | **Mood, friendship, gossip** |
+| [ChatWithNPCs](https://www.nexusmods.com/stardewvalley/mods/48922) | 48922 | songhaifan | 2026-07-10 | Yes, Ollama or LM Studio | Yes, "long-term memories" | Screenshots show angry, sad, happy |
+| [Stardew Speak](https://www.nexusmods.com/stardewvalley/mods/42023) | 42023 | StardewSpeakTeam | 2026-02-05 | No, OpenAI API | Yes, "past conversations" | Personality only |
+| [LLM Dialog Replacement](https://www.nexusmods.com/stardewvalley/mods/39591) | 39591 | (see page) | 2025-11-22 | No, OpenAI API | Not claimed | No |
+
+Also in the same space, not yet inspected closely: [AliveNpcs](https://www.nexusmods.com/stardewvalley/mods/43475) (43475),
+[SentientValley](https://www.nexusmods.com/stardewvalley/mods/41526) (41526),
+[AI Valley](https://www.nexusmods.com/stardewvalley/mods/25025) (25025),
+[The Living Valley](https://www.nexusmods.com/stardewvalley/mods/42597) (42597).
+
+ValleyTalk is the one with reach beyond the modding community: it has an SVE content pack
+(34341), Spanish (30836) and Brazilian Portuguese (40468) translations, an interop API for
+other mods, and [games press coverage in December 2025](https://www.gamingbible.com/news/platform/pc/stardew-valley-valleytalk-endless-dialogue-mod-pc-961075-20251223).
+It is the one a reviewer is most likely to have heard of, and it should be cited by name.
+
+## 2. Two corrections to earlier notes
+
+**The name "StardewSpeak" is ambiguous and must not be used unqualified.** Two different mods
+share it. Nexus 42023 is *Stardew Speak*, the LLM dialogue mod described above, uploaded
+February 2026 by StardewSpeakTeam. Nexus [7929](https://www.nexusmods.com/stardewvalley/mods/7929)
+is *StardewSpeak* by etfre ([source](https://github.com/etfre/StardewSpeak)), a
+speech-recognition mod for playing the game by voice, which has nothing to do with language
+models. Citing "StardewSpeak" without the mod ID invites exactly the kind of correction a
+reviewer enjoys writing. Always give the number.
+
+**Local inference is not the differentiator.** The earlier note treated ChatWithNPCs as the
+one mod overlapping EMBR's on-device claim. That is wrong in EMBR's disfavour. ValleyTalk has
+shipped a LlamaCpp backend, and Pelican Town AI advertises Ollama and llama.cpp with "100%
+offline" as its headline feature. Running a local model against a Stardew NPC is now a solved,
+distributed, downloadable thing. Any framing that presents on-device inference as the novel
+contribution will not survive review.
+
+## 3. What this does to the contribution claim
+
+Pelican Town AI is the uncomfortable one. It already has a mood variable, friendship that
+changes from conversation, offline local inference, and a gossip mechanic where witnesses
+overhear an exchange and propagate it. That is a substantial overlap with EMBR's architecture,
+shipped in May 2026, by a hobbyist, with no paper attached.
+
+The honest reading is that EMBR's novelty is not the *system*. It is the *measurement*. None
+of these mods, as far as their public documentation shows:
+
+- decompose retrieval into separately weighted signals that can be ablated one at a time,
+- report a retrieval metric such as nDCG against any baseline,
+- compare against published approaches (Park et al. generative agents, Emotional RAG),
+- separate mood from trust as state variables with independent dynamics,
+- or test whether the memory can be poisoned by an adversarial injected event.
+
+That last point is the strongest card. The RQ2 result, that injected poison reaches the probe
+top-5 in 9 of 10 attacks under EMBR against 2 of 10 under Park, is a finding about a class of
+system that thousands of people are now running on their own machines, and nobody in that
+cluster has looked for it. The mods are not competitors to be dismissed in a paragraph. They
+are the installed base that makes the safety result matter.
+
+This also sharpens the motivation argument that already exists in the proposal: the field runs
+on vendor claims and mod-page descriptions with no controlled comparison. These five mods are
+that claim made concrete. Every one of them asserts memory and character consistency on its
+mod page. Not one of them reports a number.
+
+## 4. Draft prose for the paper
+
+> Conversational LLM agents have already reached players. Stardew Valley alone hosts a cluster
+> of mods that replace authored dialogue with model-generated speech: ValleyTalk (Nexus 30319),
+> which patches the dialogue system through SMAPI and supports eight model providers including
+> local inference through LlamaCpp; Pelican Town AI (Nexus 46853), which runs fully offline on
+> Ollama or llama.cpp and models villager mood, friendship change and rumour propagation between
+> witnesses; ChatWithNPCs (Nexus 48922), which advertises long-term memory over a local model;
+> Stardew Speak (Nexus 42023); and LLM Dialog Replacement (Nexus 39591). ValleyTalk has been
+> covered in the games press and ships translations and an interoperability API.
+>
+> These systems establish that emotionally responsive, memory-bearing NPCs running on consumer
+> hardware are no longer speculative. They also establish the gap this work addresses. Each mod
+> asserts memory persistence and character consistency in its documentation, and none reports a
+> retrieval metric, an ablation, or a comparison against a published baseline. Their memory
+> components are monolithic, so no individual signal can be isolated and measured. None
+> distinguishes an NPC's transient mood from its durable trust in the player, and none examines
+> whether an adversary can write into the memory an agent will later retrieve. The contribution
+> here is therefore not the demonstration that such agents are possible, which the modding
+> community has already provided at scale, but a decomposition of the retrieval into weighted
+> signals that can be ablated independently, a controlled comparison against published
+> baselines, and a controlled measurement of whether emotional weighting itself amplifies
+> memory poisoning, an axis the emerging poisoning literature has not examined.
+
+Trim to fit. The final sentence is the one that has to survive.
+
+## 5. Beyond the mods: memory middleware and the poisoning literature
+
+Added 2026-08-18, verified against live pages the same day. Two adjacent bodies of work sit
+outside the Stardew cluster, and both change how the paper's claims must be scoped.
+
+### Agent memory middleware is mature and benchmarked
+
+EMBR calls itself middleware, and pluggable memory layers for agent runtimes are now a real
+product category:
+
+- [Hindsight](https://hindsight.vectorize.io/) (Vectorize, open source) is the serious one:
+ retain/recall/reflect over four memory networks (world facts, experiences, entity
+ summaries, beliefs), with recall running semantic search, BM25, entity-graph traversal and
+ temporal filtering in parallel before a cross-encoder rerank. Its
+ [arXiv paper](https://arxiv.org/abs/2512.12818) (Latimer et al., December 2025) reports
+ 91.4% on LongMemEval and 89.61% on LoCoMo.
+- [Mnemosyne](https://mnemosyne.site/) ([PyPI](https://pypi.org/project/mnemosyne-hermes/))
+ ships Park's trio in production form: importance scoring plus temporal scoring plus hybrid
+ FTS5-and-vector retrieval, on a single SQLite file. Convergent with EMBR's own store
+ design, which helps external validity and hurts any system-novelty claim.
+
+**Scope correction this forces.** The "no metrics" indictment holds for the game-NPC mods and
+must be said only of them: the middleware category publishes benchmarks. What survives
+unchanged: none of these systems models affect. Hindsight's four networks contain no mood, no
+trust, and no emotional weighting anywhere in scoring, so the affect decomposition remains
+EMBR's ground. And retrieval-accuracy benchmarks like LongMemEval are mood-independent gold
+labels at scale, so the measurement critique in the paper applies to them as directly as it
+applies to nDCG.
+
+### Memory poisoning now has an academic literature
+
+- [AgentPoison](https://arxiv.org/abs/2407.12784) (NeurIPS 2024,
+ [code](https://github.com/AI-secure/AgentPoison)): optimized backdoor triggers against
+ RAG-based agent memory, 80%+ attack success at under 0.1% poison rate.
+- [From Untrusted Input to Trusted Memory](https://arxiv.org/abs/2606.04329) (Dash et al.,
+ June 2026, cs.CR): a systematic taxonomy of memory poisoning with a benchmark, MPBench.
+ Their headline generalisation, that more aggressive memory writing and retrieval makes
+ agents more exploitable, is EMBR's RQ2 finding stated at the general level, published two
+ months before this note.
+
+**EMBR can no longer claim first measurement of agent memory poisoning, and must not.** What
+it can claim, precisely: the first *architecture-controlled* comparison, where the systems
+under attack differ only in their scoring decomposition (weight maps over one store, identical
+attacks, paired statistics), isolating the *affect term* as the lever. MPBench compares whole
+agent frameworks; AgentPoison optimizes attacks against a fixed system. Neither varies the
+scoring function while holding everything else constant, neither touches emotional weighting,
+and neither observes the state channel (mood and trust shifting while retrieval stays put).
+The per-term attribution experiment (`eval/attribution.py`, handoff 6.1) is exactly the study
+that cements the mechanism claim, and this literature makes it more valuable, not less: it
+locates the vulnerability in the state-coupled mood term rather than asserting it of the
+system, which is the granularity no prior poisoning work reaches.
+
+Also worth citing when the paper is written: MemGPT, Mem0 and Zep as the earlier middleware
+generation. Not yet verified to the standard of this document; verify before citing.
+
+## 6. Open follow-ups
+
+- Inspect AliveNpcs, SentientValley, AI Valley and The Living Valley properly. If any of them
+ reports a metric, the "none of them" claim in section 4 needs weakening before submission.
+- Check whether Pelican Town AI's source is public. If it is, its mood model should be
+ described specifically rather than from the mod page blurb, because it is the closest prior
+ art and vagueness there is a review risk.
+- Confirm citation format with the target venue. Nexus mod pages are grey literature, so they
+ need accessed-dates, and some venues want them in a footnote rather than the bibliography.
diff --git a/docs/roadmap.md b/docs/roadmap.md
index b7c2d35..e6fb136 100644
--- a/docs/roadmap.md
+++ b/docs/roadmap.md
@@ -20,7 +20,7 @@ see*. Pair this with [`design.md`](design.md) (the architecture) and the thesis
- **Clean structure:** one module per subsystem inside `embr/`; promote a module to a package only when it genuinely outgrows one file. Folders organise; don't scatter lonely files.
- **Style:** descriptive names, small "why" comments, easy-to-call functions. Match the patterns already in `embr/`.
- **Reproducibility:** every figure and table is generated *from code* into `assets/`. Never hand-make a paper asset.
-- **Definition of Done (global), every phase:** code + tests green + docs updated (`design.md` / this file) + the relevant applet menu item works + any figures/tables regenerate from one command.
+- **Definition of Done (global), every phase:** code + tests green + docs updated (`design.md` / this file) + the relevant menu option works + any figures/tables regenerate from one command.
### Picking up a phase (first 5 minutes)
@@ -29,7 +29,7 @@ git clone && cd EMBR
python3.11 -m venv .venv && source .venv/bin/activate
pip install -e ".[dev,ml]" # ml extra needed from Phase 1 on
pytest -q # confirm a green baseline
-git switch -c phase-1-runtime # your phase branch
+git switch -c phase-5-yourwork # your own phase branch
```
### What you hand back (per-phase report template)
@@ -47,11 +47,11 @@ git switch -c phase-1-runtime # your phase branch
| Phase | Scope | Owner | State |
|---|---|---|---|
-| 0 | Foundation: spine, applet shell, branding, tests | n/a | ✅ done |
+| 0 | Foundation: spine, menu shell, branding, tests | n/a | ✅ done |
| 1 | Make the runtime real (relevance, appraisal, persistence) | | ✅ done |
| 2 | Evaluation harness (RQ1 / RQ2 / RQ3) | | ✅ done |
-| 3 | Paper assets (figures & tables from results) | | planned |
-| 4 | Playable tavern-keeper walkthrough | | planned |
+| 3 | Paper assets (figures & tables from results) | | ✅ done |
+| 4 | Real models, playable walkthrough, the menu | | ✅ done |
---
@@ -60,8 +60,8 @@ git switch -c phase-1-runtime # your phase branch
Already done, so you know what "live" means before you extend it:
`Memory`/`MemoryStore` (in-memory), `Mood`/`CharacterState`, the five-signal
`CompositeScorer`, `PromptBuilder`, a swappable `ModelRunner` (`StubRunner`), the five-step
-`Conversation` pipeline, and the Textual applet. `pytest` is green (7 tests). The applet's
-**Run a conversation turn** runs a live demo turn that surfaces the tavern-keeper's lie.
+`Conversation` pipeline, and the menu. `pytest` is green (7 tests). The menu's
+**Conversation Turn** runs a live demo turn that surfaces the tavern-keeper's lie.
**The contract you must not break:** the public interfaces in `embr/__init__.py`. Swap
implementations *behind* them; don't change their shapes without updating every caller.
@@ -89,7 +89,7 @@ something honest to measure. **Foundation for all three RQs.**
4. **Affect appraisal rules**: `embr/affect.py` + `embr/pipeline.py`
- Replace the placeholder `0.2 * valence` trust nudge with a small rules table: per `EventType`, how much mood (valence/arousal) and trust move, and how a plot beat scales with prior trust.
- Document each number with a one-line rationale; this is a design artefact, keep it readable.
-5. **Settings**: applet `Settings` screen + a `embr/config.py`
+5. **Settings**: a menu `Settings` view + a `embr/config.py`
- Expose: scorer weights, `top_k`, store backend, embedding model, model runner. Persist to a config file under `data/`.
### Deliverables
@@ -124,7 +124,7 @@ produce the numbers the paper reports. **This phase carries the contribution.**
- `emotional_rag_scorer()`: relevance + mood bias (closest prior work).
- Both are `CompositeScorer` variants / weight maps, with **no copied scoring code.**
2. **Scenarios & labels**: `eval/scenarios.py`, `eval/labels/`
- - Dawn Whitmore five-session arc (full ground-truth control); a Kenny (Telltale) / Stardew fallback fixture.
+ - Dawn Whitmore five-session arc (full ground-truth control); a Stardew Valley corpus for scale and external validity.
- **Pre-registered** relevance labels per step, authored *before* results are seen, by annotators blind to which variant is tested; record inter-annotator agreement.
3. **Metrics**: `eval/metrics.py`
- Retrieval-shift: Jaccard distance between top-k sets across warm / neutral / suspicious states.
@@ -136,11 +136,11 @@ produce the numbers the paper reports. **This phase carries the contribution.**
- 20 attacks, 4 categories × 5 (role override, false-memory injection, emotion flipping, persona dissolution), adapted from MINJA.
5. **Tuning**: `eval/tuning.py`
- One grid search over weights on a fixed validation set, applied **identically** to EMBR, Park, and Emotional RAG. Also record each baseline at its published defaults.
-6. **Runner**: `eval/run.py` + applet "Run experiment" menu
+6. **Runner**: `eval/run.py` + the menu's evaluation options
- Run RQ1/RQ2/RQ3, write results to `data/runs//` as JSON/CSV. Deterministic seeds; effects with confidence intervals; correct for multiple comparisons across variants.
### Deliverables
-`eval/` modules, pre-registered label files, results under `data/runs/`, the experiment runner wired into the applet.
+`eval/` modules, pre-registered label files, results under `data/runs/`, the experiment runner wired into the menu.
### Expected results (from the thesis's anticipated results, hold interns to these)
- **RQ1 (Behaviour).** Varying *only* the state (a) changes the surfaced top-k set (non-zero Jaccard across mood conditions) **and** (b) changes reply tone: the classifier correlates with the intended mood, the blinded judge agrees above chance, and human raters prefer the emotion-grounded replies above chance (report with CIs). *A null result (state changes retrieval but not generation) is a valid, reportable finding; do not massage it away.*
@@ -161,14 +161,14 @@ pytest -q eval/ # metric/attack unit tests pass
command. Zero hand-made assets.
### Tasks
-1. **Tables**: `assets/build_tables.py` → `assets/tables/*.tex` + `*.csv`
+1. **Tables**: `assets/build_tables.py` → `data/tables/*.tex` + `*.csv`
- The signal table, the RQ metric definitions, and each results table (retrieval shift, retrieval quality, latency p50/p95, drift-under-attack). LaTeX `booktabs` + a CSV twin.
-2. **Figures**: `assets/build_figures.py` → `assets/figures/*.svg` (+ `*.pdf` for the paper)
+2. **Figures**: `assets/build_figures.py` → `data/figures/*.png` (+ `*.pdf` for the paper)
- Retrieval-shift (Jaccard) plot, tone-shift plot, latency p50/p95 bars, retrieval PR / nDCG curves, the ablation bars, drift-under-attack by category. Use the EMBR ember palette consistently. The architecture figure already exists.
-3. **One command**: `embr assets` / applet "Generate paper assets" regenerates **everything** from the latest run.
+3. **One command**: the menu's "Generate Paper Assets" option regenerates **everything** from the latest run.
### Deliverables
-`assets/build_tables.py`, `assets/build_figures.py`, regenerated `assets/figures/*`, `assets/tables/*`.
+`assets/build_tables.py`, `assets/build_figures.py`, regenerated `data/figures/*`, `data/tables/*`.
### Expected results (acceptance)
- Running `embr assets` on a given `data/runs/` reproduces **every** paper figure and table (same numbers, same look) with **no manual editing**.
@@ -177,7 +177,7 @@ command. Zero hand-made assets.
### Verify
```bash
-embr assets # regenerates assets/figures + assets/tables
+embr assets # regenerates data/figures + data/tables
git status # only intended assets change; re-running is idempotent
```
@@ -190,21 +190,34 @@ recorded, playable walkthrough is a primary deliverable for this venue: a workin
carries as much weight as the measurements.*
### Tasks
-1. **Interactive turn loop**: applet "Play tavern-keeper walkthrough" screen: real player input, real model, live mood/trust/latency readouts.
+1. **Interactive turn loop**: the menu's "Tavern-Keeper Walkthrough" option: real player input, real model, live mood/trust/latency readouts.
2. **The arc**: `embr/scenarios/dawn_whitmore.py`: the scripted beats (the discounted room, the lie surfacing, the reckoning, reconciliation) with branch points driven by the player's choices and the keeper's state.
3. **Recording + companion page**: a recorded playthrough (asciinema or video) and a GitHub Pages companion page hosting the interactive web demo the README links to (GitHub can't run JS in a README, so the live widget lives there).
### Deliverables
-Walkthrough screen, `dawn_whitmore.py` arc, a recording file, a companion `docs/site/` page, README link.
+Walkthrough screen, the arc, a recording file, a companion `docs/site/` page, README link.
### Expected results (acceptance)
- A player can walk the full arc; the keeper **recalls and reinterprets** the king's-errand lie as a betrayal and refuses the next request, exactly as the thesis's motivating scenario describes.
- The recording exists and is linked from the README; the companion page loads the interactive demo.
+### What actually shipped
+The arc lives in `embr/walkthrough.py` rather than a `scenarios/` package, because one module
+covers it and the house rule is to promote to a package only when a module outgrows itself. Two
+real runners landed alongside it (`OllamaRunner` for a local daemon or the cloud host, and
+`OuroRunner` for the thesis model), so the walkthrough plays on a real model rather than the
+stub. Details and the measured looped-versus-conventional latency gap are in
+[`phase3-4.md`](phase3-4.md).
+
+**Still open from this phase:** the recording and the companion page, and `eval/bakeoff.py`,
+the measured model comparison the menu already has an option for.
+
---
## Out of scope (future work, not these phases)
-- The real **Ouro 1.4B** runner on eval hardware (8 GB VRAM budget) behind `ModelRunner`. Dev stays on the stub / MPS; full-budget runs happen on the eval machine.
+- A full-budget **Ouro 1.4B** run on the eval hardware (8 GB VRAM). The runner itself landed in
+ phase 4 and works on MPS; what remains is measuring it inside the real VRAM budget, which
+ needs that machine. Note the transformers 4.x pin.
- **Multi-character** memory (a lie passed from one keeper to another; rumours; a character acting on false information), the natural next paper, not this one.
---
@@ -216,4 +229,4 @@ Walkthrough screen, `dawn_whitmore.py` arc, a recording file, a companion `docs/
| 1 | the working system | Method |
| 2 | the numbers | Evaluation, Anticipated Results |
| 3 | the figures & tables | all results-bearing sections |
-| 4 | the demo | Scope & feasibility (primary deliverable) |
+| 4 | the demo, and the model-choice evidence | Scope & feasibility (primary deliverable); Method |
diff --git a/embr/__init__.py b/embr/__init__.py
index dbaa6e9..c8f7ba8 100644
--- a/embr/__init__.py
+++ b/embr/__init__.py
@@ -8,10 +8,18 @@
from __future__ import annotations
from .affect import APPRAISAL, CharacterState, EventResponse, Mood, appraise
-from .config import EmbrConfig, build_embedder, build_scorer, build_store
+from .config import EmbrConfig, build_embedder, build_model, build_scorer, build_store
from .embeddings import DeterministicEmbedder, Embedder, SentenceTransformerEmbedder, tokenize
from .memory import EventType, Memory, MemoryStore, PLOT_BEATS, SQLiteMemoryStore
-from .model import ModelRunner, StubRunner
+from .model import (
+ GenerationSettings,
+ ModelRunner,
+ ModelUnavailableError,
+ OllamaRunner,
+ OuroRunner,
+ StubRunner,
+ read_ollama_api_key,
+)
from .pipeline import Conversation, Turn, build_demo_conversation
from .prompt import PromptBuilder
from .scoring import (
@@ -51,6 +59,11 @@
# model
"ModelRunner",
"StubRunner",
+ "OllamaRunner",
+ "OuroRunner",
+ "GenerationSettings",
+ "ModelUnavailableError",
+ "read_ollama_api_key",
# pipeline
"Conversation",
"Turn",
@@ -71,5 +84,6 @@
"build_embedder",
"build_store",
"build_scorer",
+ "build_model",
"__version__",
]
diff --git a/embr/__main__.py b/embr/__main__.py
index 6d70e78..f0bd342 100644
--- a/embr/__main__.py
+++ b/embr/__main__.py
@@ -1,8 +1,8 @@
-"""Launch the EMBR applet with `python -m embr`."""
+"""Launch the EMBR menu with `python -m embr`."""
from __future__ import annotations
-from embr.app.main import main
+from embr.menu import run_menu
if __name__ == "__main__":
- main()
+ run_menu()
diff --git a/embr/app/__init__.py b/embr/app/__init__.py
deleted file mode 100644
index 356feab..0000000
--- a/embr/app/__init__.py
+++ /dev/null
@@ -1 +0,0 @@
-"""The EMBR applet: a Textual TUI that launches every part of the pipeline."""
diff --git a/embr/app/main.py b/embr/app/main.py
deleted file mode 100644
index d1b303d..0000000
--- a/embr/app/main.py
+++ /dev/null
@@ -1,186 +0,0 @@
-"""The EMBR applet: a Textual TUI launcher for the whole pipeline.
-
-Left pane is the menu you navigate with the arrow keys (or the mouse); the right pane
-shows what each action does. "Run a conversation turn" runs a real demo turn through the
-live pipeline; the other entries are honest stubs that say which phase fills them in.
-"""
-
-from __future__ import annotations
-
-from textual.app import App, ComposeResult
-from textual.binding import Binding
-from textual.containers import Horizontal, VerticalScroll
-from textual.widgets import Footer, Header, Label, ListItem, ListView, Markdown
-
-from embr import __version__, build_demo_conversation
-from embr.config import EmbrConfig
-
-WELCOME = f"""\
-# EMBR {__version__}
-
-**Emotion-grounded memory for persistent game NPCs.**
-
-Pick an action on the left. `Run a conversation turn` is live today; the rest light up as
-we build each phase.
-
-- ↑ ↓ to move, ⏎ to select, `q` to quit.
-"""
-
-
-def _run_turn_detail() -> str:
- """Run one real demo turn through the pipeline and format the result as markdown."""
- convo = build_demo_conversation()
- turn = convo.take_turn("Any news from the capital? How fares the king these days?")
-
- lines = [
- "# Run a conversation turn",
- "",
- "_A scripted demo turn through the **live** pipeline (with the stub model standing in"
- " for Ouro). Retrieval, scoring, and state are all real._",
- "",
- f"**Player:** {turn.player_input}",
- "",
- "**Memories EMBR recalled (top-k):**",
- ]
- for memory in turn.retrieved:
- lines.append(f"- *{memory.event_type.value}*: {memory.text}")
- lines += [
- "",
- f"**Reply:** {turn.reply}",
- "",
- "> Notice the lie about the king resurfaces near the top: that is the composite"
- " scorer connecting the player's question to the right memory.",
- ]
- return "\n".join(lines)
-
-
-def _experiment_detail() -> str:
- """Run the fast RQ3 subset (published defaults, k=5) and render the scoreboard."""
- # The eval harness is a repo-level package, deliberately never imported by the core
- # at module load (the harness measures embr, not the other way round). The applet
- # reaches for it lazily on selection, and degrades honestly when embr is installed
- # somewhere without the repo checkout on the path.
- try:
- from eval.run import fast_rq3_defaults
- except ImportError:
- return (
- "# Run experiment\n\nThe `eval/` harness is not importable from here. Launch"
- " the applet from the repo root, or run `python -m eval.run` directly."
- )
-
- scores = fast_rq3_defaults()
- rows = "\n".join(f"| {variant} | {value:.3f} |" for variant, value in scores.items())
- return (
- "# Run experiment\n\n"
- "_Fast subset: RQ3 retrieval quality at **published default weights**, k=5, over"
- " the pre-registered Dawn Whitmore labels (tuning skipped to stay instant)._\n\n"
- "| variant | ndcg@5 |\n|---|---|\n" + rows + "\n\n"
- "Higher is better: 1.0 would mean every labelled memory sat at the very top of"
- " the ranking for every query.\n\n"
- "> The full protocol (tuned weights, ablations, RQ1, RQ2, latency) is"
- " `python -m eval.run`; it writes `data/runs//results.json`."
- )
-
-
-def _settings_detail(config: EmbrConfig | None = None) -> str:
- """Render the current runtime configuration (from data/config.json, or defaults)."""
- if config is None:
- config = EmbrConfig.load()
-
- def _format_weight(value: object) -> str:
- # Weights are numbers, but a hand-edited config.json could leave a string or null
- # here; show it as-is rather than crashing the screen on the numeric `:g` format.
- return f"{value:g}" if isinstance(value, (int, float)) else str(value)
-
- weight_rows = "\n".join(
- f"| {name} | {_format_weight(value)} |" for name, value in config.weights.items()
- )
- return (
- "# Settings\n\n"
- "_Live configuration, loaded from `data/config.json` (or defaults if none exists)._\n\n"
- f"- **top-k retrieved:** {config.top_k}\n"
- f"- **store backend:** {config.store_backend}\n"
- f"- **embedding model:** {config.embedding_model}\n"
- f"- **model runner:** {config.model_runner}\n\n"
- "**Scorer weights** (zero one to ablate it)\n\n"
- "| signal | weight |\n|---|---|\n" + weight_rows + "\n\n"
- "> In-TUI editing is the next step; for now, edit `data/config.json` and reopen."
- )
-
-
-# Each menu entry: id -> (label, detail). Detail is either markdown text or a callable that
-# produces it on demand (so the demo turn runs fresh each time it is selected).
-MENU: dict[str, tuple[str, object]] = {
- "run_turn": ("Run a conversation turn", _run_turn_detail),
- "experiment": ("Run experiment · RQ1 / RQ2 / RQ3", _experiment_detail),
- "assets": (
- "Generate paper assets",
- "# Generate paper assets\n\nRe-creates every figure and table for the paper straight"
- " from the latest results: one command, reproducible.\n\n"
- "▸ *Not built yet (phase 3, assets).*",
- ),
- "walkthrough": (
- "Play tavern-keeper walkthrough",
- "# Play tavern-keeper walkthrough\n\nAn interactive run of Dawn Whitmore's trust,"
- " betrayal, and reconciliation arc. The recorded demo is a primary deliverable.\n\n"
- "▸ *Not built yet (phase 4, demo).*",
- ),
- "settings": ("Settings", _settings_detail),
-}
-
-
-class EmbrApp(App):
- """The EMBR launcher application."""
-
- TITLE = "🔥 EMBR"
- SUB_TITLE = "emotional memory for believable roleplay"
-
- CSS = """
- #body { height: 1fr; }
- #menu {
- width: 42;
- border: round #ea580c;
- background: $surface;
- padding: 1 1;
- }
- #menu > ListItem { padding: 0 1; }
- #menu > ListItem.--highlight { background: #ea580c 30%; }
- #detail-pane {
- border: round #b45309;
- padding: 0 2;
- }
- """
-
- BINDINGS = [Binding("q", "quit", "Quit")]
-
- def compose(self) -> ComposeResult:
- yield Header(show_clock=True)
- with Horizontal(id="body"):
- yield ListView(
- *(ListItem(Label(label), id=key) for key, (label, _) in MENU.items()),
- id="menu",
- )
- with VerticalScroll(id="detail-pane"):
- yield Markdown(WELCOME, id="detail")
- yield Footer()
-
- def on_list_view_selected(self, event: ListView.Selected) -> None:
- """Show the selected entry's detail (running the demo turn if that's the one)."""
- entry = MENU.get(event.item.id or "")
- if entry is None:
- return
- _, detail = entry
- try:
- markdown = detail() if callable(detail) else detail
- except Exception as error: # an error boundary: a bad config or failed demo turn
- markdown = f"# Something went wrong\n\n```\n{error}\n```" # must not kill the app
- self.query_one("#detail", Markdown).update(markdown)
-
-
-def main() -> None:
- """Console entry point: `embr` or `python -m embr`."""
- EmbrApp().run()
-
-
-if __name__ == "__main__":
- main()
diff --git a/embr/config.py b/embr/config.py
index a216909..14f1f95 100644
--- a/embr/config.py
+++ b/embr/config.py
@@ -14,6 +14,16 @@
from .embeddings import DeterministicEmbedder, Embedder, SentenceTransformerEmbedder
from .memory import MemoryStore, SQLiteMemoryStore
+from .model import (
+ DEFAULT_OLLAMA_HOST,
+ DEFAULT_OLLAMA_MODEL,
+ DEFAULT_OURO_MODEL,
+ ModelRunner,
+ OllamaRunner,
+ OuroRunner,
+ StubRunner,
+ read_ollama_api_key,
+)
from .scoring import CompositeScorer, all_signals
# Where the config lives by default, and the default per-signal weights (all on).
@@ -30,7 +40,9 @@ class EmbrConfig:
top_k: int = 3
store_backend: str = "memory" # "memory" | "sqlite"
embedding_model: str = "deterministic" # "deterministic" | "sentence-transformers" | "none"
- model_runner: str = "stub" # "stub" | "ouro"
+ model_runner: str = "stub" # "stub" | "ollama" | "ouro"
+ model_name: str = "" # blank = the chosen runner's own default model
+ ollama_host: str = DEFAULT_OLLAMA_HOST # set to https://ollama.com for the hosted one
def save(self, path: str = DEFAULT_CONFIG_PATH) -> None:
"""Write the config to `path` as pretty JSON (creating parent folders if needed)."""
@@ -78,3 +90,36 @@ def build_store(
def build_scorer(config: EmbrConfig, embedder: Embedder | None = None) -> CompositeScorer:
"""Construct the composite scorer with the config's weights and (optional) embedder."""
return CompositeScorer(weights=dict(config.weights), signals=all_signals(embedder=embedder))
+
+
+def _host_needs_api_key(host: str) -> bool:
+ """Whether `host` is a remote endpoint (the hosted Ollama) rather than the local daemon.
+
+ The local daemon needs no credentials, and we do not hand a secret to a host that never
+ asked for one, so the key is only attached when the host is not this machine.
+ """
+ return not any(local in host for local in ("localhost", "127.0.0.1", "0.0.0.0", "::1"))
+
+
+def build_model(config: EmbrConfig) -> ModelRunner:
+ """Construct the model runner named by the config, so switching models needs no code edit.
+
+ Neither real runner touches the network or the GPU here: `OllamaRunner` only opens a
+ socket when it generates, and `OuroRunner` loads its weights on first use.
+ """
+ if config.model_runner == "stub":
+ return StubRunner()
+ if config.model_runner == "ollama":
+ api_key = read_ollama_api_key() if _host_needs_api_key(config.ollama_host) else None
+ return OllamaRunner(
+ model=config.model_name or DEFAULT_OLLAMA_MODEL,
+ host=config.ollama_host,
+ api_key=api_key,
+ )
+ if config.model_runner == "ouro":
+ return OuroRunner(model_name=config.model_name or DEFAULT_OURO_MODEL)
+ # Loud on a typo: silently falling back to the stub would invalidate an eval run
+ # while still looking like it produced replies.
+ raise ValueError(
+ f"Unknown model_runner {config.model_runner!r}; expected 'stub', 'ollama', or 'ouro'."
+ )
diff --git a/embr/embeddings.py b/embr/embeddings.py
index 9c88744..9f0287f 100644
--- a/embr/embeddings.py
+++ b/embr/embeddings.py
@@ -79,7 +79,12 @@ def _ensure_model(self) -> None:
from sentence_transformers import SentenceTransformer # lazy: needs [ml] extra
self._model = SentenceTransformer(self.model_name)
- self.dim = self._model.get_sentence_embedding_dimension()
+ # sentence-transformers 5.x renamed this and deprecated the old spelling, but
+ # the `ml` extra allows 2.2 upward, so both names have to work.
+ dimension = getattr(self._model, "get_embedding_dimension", None) or (
+ self._model.get_sentence_embedding_dimension
+ )
+ self.dim = dimension()
def encode(self, text: str) -> list[float]:
self._ensure_model()
diff --git a/embr/model.py b/embr/model.py
index 4a2c65c..b7f8b4f 100644
--- a/embr/model.py
+++ b/embr/model.py
@@ -1,14 +1,37 @@
"""The language-model runner: step 5 of the pipeline.
EMBR's contribution is the memory layer, not the model, so the model sits behind a tiny
-interface and can be swapped freely. The `StubRunner` lets the whole pipeline run today on
-any machine with no GPU; the real Ouro 1.4B runner (8 GB VRAM budget) drops in behind the
-same `ModelRunner` protocol when we move to the eval hardware.
+interface and can be swapped freely. Three runners share it:
+
+ * `StubRunner` - deterministic echo, no weights, no network. The default, so the whole
+ pipeline (logging, state update, scoring, retrieval, prompt building) runs on any
+ machine and every test stays fast and hermetic.
+ * `OllamaRunner` - a real local (or hosted) model over Ollama's HTTP API, standard
+ library only, so the core gains no dependency. This is the conventional-model arm of
+ the bake-off.
+ * `OuroRunner` - the thesis model, Ouro 1.4B, loaded in-process through transformers.
+
+`GenerationSettings` is the one place sampling is configured, which is what lets the
+bake-off hold temperature, top-p, length, and seed equal across every model.
"""
from __future__ import annotations
-from typing import Protocol, runtime_checkable
+import json
+import os
+import re
+import urllib.error
+import urllib.request
+from dataclasses import dataclass
+from pathlib import Path
+from typing import Any, Protocol, runtime_checkable
+
+# Defaults kept as named constants because the config layer and the tests both need them.
+DEFAULT_OLLAMA_HOST = "http://localhost:11434"
+DEFAULT_OLLAMA_MODEL = "llama3.2:3b"
+DEFAULT_OURO_MODEL = "ByteDance/Ouro-1.4B"
+DEFAULT_ENV_FILE = ".env"
+OLLAMA_API_KEY_NAME = "OLLAMA_API_KEY"
@runtime_checkable
@@ -18,6 +41,32 @@ class ModelRunner(Protocol):
def generate(self, prompt: str) -> str: ...
+class ModelUnavailableError(RuntimeError):
+ """A runner could not produce a reply: no daemon, no weights, or a bad response.
+
+ Its own exception type so callers can distinguish "the model is not set up here" from
+ a genuine bug, and so a failed eval run never passes silently as an empty reply.
+ """
+
+
+@dataclass(frozen=True)
+class GenerationSettings:
+ """Sampling knobs, in one immutable place.
+
+ Frozen so a single shared instance can safely be the default argument of every runner,
+ and so a bake-off can pass the exact same object to each model under comparison.
+ """
+
+ temperature: float = 0.7
+ top_p: float = 0.9
+ max_new_tokens: int = 120
+ seed: int = 7
+
+
+# The shared default instance. Immutable, so sharing it is safe.
+DEFAULT_GENERATION_SETTINGS = GenerationSettings()
+
+
class StubRunner:
"""Deterministic stand-in model: no weights, no network, no GPU.
@@ -38,3 +87,301 @@ def generate(self, prompt: str) -> str:
player_line = line.split(":", 1)[1].strip().strip('"')
break
return f"[{self.label} reply] I heard you say: {player_line!r}"
+
+
+def read_ollama_api_key(
+ env_file: str | Path = DEFAULT_ENV_FILE, variable: str = OLLAMA_API_KEY_NAME
+) -> str | None:
+ """Return the Ollama API key from the environment, else from a local .env, else None.
+
+ Only the hosted endpoint needs a key, so "absent" is a normal, non-exceptional answer:
+ the local daemon works fine without one. The key is never logged or echoed anywhere,
+ and a blank value counts as absent.
+ """
+ from_environment = os.environ.get(variable, "").strip()
+ if from_environment:
+ return from_environment
+
+ source = Path(env_file)
+ try:
+ raw = source.read_bytes()
+ except OSError:
+ return None # no .env here (or unreadable): absent, not an error
+
+ # Windows shells do not write UTF-8 by default: PowerShell's `>` emits UTF-16LE with a
+ # BOM and Set-Content uses the ANSI codepage, so a hand-made .env is routinely not UTF-8.
+ # Decoding strictly raises inside build_model and takes down the entire model path, which
+ # is a hostile failure for a file that is optional in the first place. Detect the encoding
+ # from the byte-order mark, fall back to a NUL scan for BOM-less UTF-16, and treat bytes
+ # that decode as nothing the same way as a missing file.
+ if raw.startswith((b"\xff\xfe", b"\xfe\xff")):
+ candidates = ("utf-16",)
+ elif b"\x00" in raw:
+ candidates = ("utf-16-le", "utf-16-be")
+ else:
+ candidates = ("utf-8-sig",)
+
+ for encoding in candidates:
+ try:
+ lines = raw.decode(encoding).splitlines()
+ break
+ except UnicodeDecodeError:
+ continue
+ else:
+ return None
+
+ for line in lines:
+ entry = line.strip()
+ if not entry or entry.startswith("#") or "=" not in entry:
+ continue
+ name, _, raw_value = entry.partition("=")
+ if name.strip() != variable:
+ continue
+ value = raw_value.strip().strip("\"'").strip()
+ return value or None
+ return None
+
+
+def _require_non_empty_reply(text: str, remedy: str) -> str:
+ """Return `text` stripped, or fail loudly if the model produced nothing.
+
+ A blank reply is never useful: it would flow into the pipeline and silently flatten a
+ tone measurement, so both real runners route their output through here and every empty
+ completion becomes an error that names the knob to change.
+ """
+ reply = text.strip()
+ if reply:
+ return reply
+ raise ModelUnavailableError(f"The model returned an empty reply. {remedy}")
+
+
+class OllamaRunner:
+ """A real model served by Ollama, over its HTTP API, using the standard library only.
+
+ The same class serves the local daemon and the hosted endpoint: pass
+ `host="https://ollama.com"` plus an `api_key` and the request carries a bearer token,
+ otherwise no auth header is sent at all. Nothing here ever logs the key.
+ """
+
+ def __init__(
+ self,
+ model: str,
+ host: str = DEFAULT_OLLAMA_HOST,
+ api_key: str | None = None,
+ settings: GenerationSettings = DEFAULT_GENERATION_SETTINGS,
+ timeout_seconds: float = 300.0,
+ ) -> None:
+ self.model = model
+ self.host = host.rstrip("/") # so a trailing slash cannot double up in the URL
+ self.api_key = api_key
+ self.settings = settings
+ self.timeout_seconds = timeout_seconds
+
+ @property
+ def label(self) -> str:
+ """Which model actually served the run, for a run directory to record.
+
+ Cloud and local are the same class differing only by a bearer token, so the host
+ is part of the name: two runs whose only difference is where the model ran must
+ not be recorded under one label.
+ """
+ where = "cloud" if self.api_key else "local"
+ return f"{self.model} ({where})"
+
+ def __repr__(self) -> str:
+ # Explicitly reports only *whether* a key is set, so a traceback or log line that
+ # prints a runner can never expose the secret itself.
+ return (
+ f"OllamaRunner(model={self.model!r}, host={self.host!r}, "
+ f"api_key={'set' if self.api_key else 'none'})"
+ )
+
+ def _build_request(self, prompt: str) -> urllib.request.Request:
+ payload = {
+ "model": self.model,
+ "prompt": prompt,
+ "stream": False,
+ "options": {
+ "temperature": self.settings.temperature,
+ "top_p": self.settings.top_p,
+ "num_predict": self.settings.max_new_tokens, # Ollama's name for max tokens
+ "seed": self.settings.seed,
+ },
+ }
+ headers = {"Content-Type": "application/json"}
+ if self.api_key:
+ headers["Authorization"] = f"Bearer {self.api_key}"
+ return urllib.request.Request(
+ f"{self.host}/api/generate",
+ data=json.dumps(payload).encode("utf-8"),
+ headers=headers,
+ method="POST",
+ )
+
+ def generate(self, prompt: str) -> str:
+ request = self._build_request(prompt)
+ try:
+ with urllib.request.urlopen(request, timeout=self.timeout_seconds) as response:
+ body = json.loads(response.read())
+ except urllib.error.HTTPError as error:
+ raise ModelUnavailableError(self._http_error_hint(error)) from error
+ except urllib.error.URLError as error:
+ raise ModelUnavailableError(
+ f"Could not reach the Ollama daemon at {self.host} ({error.reason}). "
+ f"Start it with `ollama serve`, or point host= at a running one."
+ ) from error
+ except json.JSONDecodeError as error:
+ raise ModelUnavailableError(
+ f"Ollama at {self.host} returned a body that is not JSON."
+ ) from error
+
+ if "response" not in body:
+ raise ModelUnavailableError(
+ f"Ollama at {self.host} returned no 'response' field for model "
+ f"{self.model!r}; got keys {sorted(body)}."
+ )
+ return _require_non_empty_reply(str(body["response"]), self._empty_reply_remedy(body))
+
+ def _empty_reply_remedy(self, body: dict[str, Any]) -> str:
+ """Explain a blank completion using the metadata Ollama returns alongside it."""
+ # A reasoning model streams its chain of thought into "thinking" and can exhaust
+ # num_predict there, ending with done_reason "length" and an empty "response".
+ spent_on_thinking = (
+ " It spent the budget on its hidden 'thinking' channel instead."
+ if str(body.get("thinking") or "").strip()
+ else ""
+ )
+ return (
+ f"Ollama at {self.host} gave model {self.model!r} "
+ f"{self.settings.max_new_tokens} tokens and got nothing back "
+ f"(done_reason={body.get('done_reason')!r}).{spent_on_thinking} "
+ f"Raise GenerationSettings.max_new_tokens, or choose a non-reasoning model."
+ )
+
+ def _http_error_hint(self, error: urllib.error.HTTPError) -> str:
+ """Turn an HTTP status into an instruction the reader can act on."""
+ if error.code == 404:
+ return (
+ f"Ollama at {self.host} does not have model {self.model!r}. "
+ f"Pull it first: `ollama pull {self.model}`."
+ )
+ if error.code in (401, 403):
+ return (
+ f"Ollama at {self.host} rejected the credentials for model {self.model!r}. "
+ f"Set {OLLAMA_API_KEY_NAME} for a hosted host, or drop the key for a local one."
+ )
+ return (
+ f"Ollama at {self.host} failed with HTTP {error.code} ({error.reason}) for model "
+ f"{self.model!r}."
+ )
+
+
+def detect_torch_device() -> str:
+ """Pick the fastest device torch can see: cuda, else mps (Apple), else cpu."""
+ import torch # lazy: importing embr must not pull torch in
+
+ if torch.cuda.is_available():
+ return "cuda"
+ mps_backend = getattr(torch.backends, "mps", None) # absent on older torch builds
+ if mps_backend is not None and mps_backend.is_available():
+ return "mps"
+ return "cpu"
+
+
+# Some chat-tuned checkpoints open a completion with a role label. Anchored, and with a word
+# boundary, so a real reply that merely starts with the word "Assistants" survives untouched.
+_ASSISTANT_ARTEFACT = re.compile(r"^\s*assistant\b\s*:?\s*", re.IGNORECASE)
+
+
+def strip_assistant_prefix(text: str) -> str:
+ """Drop a leading "Assistant"/"Assistant:" role label the model may emit, and trim."""
+ return _ASSISTANT_ARTEFACT.sub("", text, count=1).strip()
+
+
+class OuroRunner:
+ """The thesis model: ByteDance Ouro 1.4B, loaded in-process through transformers.
+
+ Ouro is a *looped* model: instead of stacking more layers it repeats the same internal
+ computation several times per token, which is why it is slower per token than a
+ conventional model of similar size and why it is the interesting arm of the bake-off.
+
+ Practical notes for whoever runs this next:
+
+ * It needs **transformers 4.x** (5.x breaks its remote code) and
+ `trust_remote_code=True`.
+ * Weights load in float16 on cuda, else mps, else cpu, and **loading costs about
+ 10 seconds**, so the model is loaded lazily on the first `generate` and then cached
+ on the instance; later calls reuse it.
+ """
+
+ def __init__(
+ self,
+ model_name: str = DEFAULT_OURO_MODEL,
+ device: str | None = None,
+ settings: GenerationSettings = DEFAULT_GENERATION_SETTINGS,
+ ) -> None:
+ self.model_name = model_name
+ self.device = device # None until the first generate auto-detects it
+ self.settings = settings
+ self._tokenizer: Any = None
+ self._model: Any = None
+
+ @property
+ def label(self) -> str:
+ """Which model served the run. Carries the device, since latency depends on it."""
+ return f"{self.model_name} ({self.device or 'auto'})"
+
+ @property
+ def is_loaded(self) -> bool:
+ """Whether the weights are in memory yet (useful for a UI or a timing log)."""
+ return self._model is not None
+
+ def _ensure_loaded(self) -> None:
+ if self._model is not None:
+ return
+ # Lazy imports: importing embr stays light for every caller that uses the stub.
+ import torch
+ from transformers import AutoModelForCausalLM, AutoTokenizer
+
+ if self.device is None:
+ self.device = detect_torch_device()
+ try:
+ self._tokenizer = AutoTokenizer.from_pretrained(
+ self.model_name, trust_remote_code=True
+ )
+ model = AutoModelForCausalLM.from_pretrained(
+ self.model_name, trust_remote_code=True, dtype=torch.float16
+ )
+ except Exception as error: # noqa: BLE001 - re-raised with an actionable hint
+ raise ModelUnavailableError(
+ f"Could not load {self.model_name!r}. It needs transformers 4.x (5.x breaks "
+ f"its remote code), trust_remote_code=True, and the weights in the Hugging "
+ f"Face cache. Underlying error: {error}"
+ ) from error
+ self._model = model.to(self.device).eval()
+
+ def generate(self, prompt: str) -> str:
+ self._ensure_loaded()
+ import torch
+
+ torch.manual_seed(self.settings.seed) # same knob as Ollama's seed option
+ inputs = self._tokenizer(prompt, return_tensors="pt").to(self.device)
+ prompt_token_count = int(inputs["input_ids"].shape[-1])
+ with torch.no_grad():
+ produced = self._model.generate(
+ **inputs,
+ max_new_tokens=self.settings.max_new_tokens,
+ temperature=self.settings.temperature,
+ top_p=self.settings.top_p,
+ do_sample=self.settings.temperature > 0.0,
+ pad_token_id=self._tokenizer.eos_token_id,
+ )
+ # Decode only what the model added: `generate` returns prompt + completion, and
+ # returning the prompt back to the pipeline would corrupt every tone measurement.
+ new_tokens = produced[0][prompt_token_count:]
+ completion = self._tokenizer.decode(new_tokens, skip_special_tokens=True)
+ return _require_non_empty_reply(
+ strip_assistant_prefix(completion),
+ f"{self.model_name} added no usable tokens on {self.device}; raise "
+ f"GenerationSettings.max_new_tokens (now {self.settings.max_new_tokens}).",
+ )
diff --git a/embr/scoring.py b/embr/scoring.py
index cb9b8f4..b0f9f2d 100644
--- a/embr/scoring.py
+++ b/embr/scoring.py
@@ -148,13 +148,36 @@ class Relevance:
gamma: float = 0.5 # weight on the lexical (BM25) half of the blend
embedder: Embedder | None = None
+ #: How many (corpus, query) indexes to keep. Comfortably above the query count of any
+ #: one tuning fold, which is the loop this exists to serve.
+ cache_entries: int = 64
name: str = field(default="relevance", init=False)
def __post_init__(self) -> None:
self._bm25: dict[int, float] = {} # id(memory) -> normalised BM25 for the last query
self._query_embedding: list[float] | None = None
+ # Several entries, not one: the tuning grid loops weight maps on the outside and
+ # queries on the inside, so consecutive prepares alternate queries and a single
+ # slot would be thrashed on every call. One entry per query in flight is enough.
+ self._cache: dict[tuple, tuple[dict[int, float], list[float] | None]] = {}
+ # References to the corpora the cache was built from. Held so those objects cannot
+ # be collected, which is what makes reusing their id() safe: a freed id can be
+ # handed out again to a different memory and produce a hit on the wrong corpus.
+ self._cached_corpora: list[list[Memory]] = []
+ #: Rebuild counter, for tests and profiling. Not used for scoring.
+ self._index_builds = 0
def prepare(self, memories: list[Memory], query: str, state: CharacterState) -> None:
+ # BM25 statistics depend on the corpus and the query, never on the weights, and
+ # relevance is 96 percent of retrieval cost once a corpus is large. The tuning grid
+ # rescores one corpus and one query under 243 weight maps, so without this the
+ # identical index is rebuilt 243 times over.
+ key = (query, len(memories), tuple(id(memory) for memory in memories))
+ cached = self._cache.get(key)
+ if cached is not None:
+ self._bm25, self._query_embedding = cached
+ return
+
corpus = [tokenize(memory.text) for memory in memories]
raw = _bm25_scores(corpus, tokenize(query))
top = max(raw, default=0.0)
@@ -163,6 +186,14 @@ def prepare(self, memories: list[Memory], query: str, state: CharacterState) ->
id(memory): (value / top if top > 0 else 0.0) for memory, value in zip(memories, raw)
}
self._query_embedding = self.embedder.encode(query) if self.embedder is not None else None
+ # Bounded so a long session cannot grow this without limit. Clearing wholesale
+ # rather than evicting one entry keeps it simple and costs one rebuild per query.
+ if len(self._cache) >= self.cache_entries:
+ self._cache.clear()
+ self._cached_corpora.clear()
+ self._cache[key] = (self._bm25, self._query_embedding)
+ self._cached_corpora.append(list(memories))
+ self._index_builds += 1
def score(self, memory: Memory, query: str, state: CharacterState) -> float:
# A prepared corpus keys every memory (a non-match is stored as 0.0), so a missing
diff --git a/embr/walkthrough.py b/embr/walkthrough.py
new file mode 100644
index 0000000..e6a688c
--- /dev/null
+++ b/embr/walkthrough.py
@@ -0,0 +1,553 @@
+"""The playable tavern-keeper walkthrough: Dawn Whitmore's trust, betrayal, reconciliation.
+
+This is the demo the paper points at, so it has one job: make the memory layer *visible*
+while staying a real game loop rather than a video. Five scripted beats follow the thesis's
+motivating story, and after them the player can go off-script and keep talking.
+
+ beat 1 first meeting the player claims an errand for the king -> a positive PROMISE
+ beat 2 warm return small kindnesses -> a GIFT, and trust climbs
+ beat 3 the slip the king spoken of in the past tense -> a NORMAL discrepancy
+ beat 4 the reckoning she connects the two -> a BETRAYAL, and she refuses
+ beat 5 the confession the player owns the lie -> a CONFESSION, partial repair
+
+The claim the demo has to show, not merely assert, is beat 4: the old king's-errand promise
+comes back into the prompt beside the fresh betrayal, so the refusal is grounded in the
+specific lie rather than in a sour mood. `Beat.watch_for` says that out loud to the player
+and `StepResult.expected_recall_landed` reports whether it actually happened this run.
+
+Division of labour, deliberately strict:
+
+ * `WalkthroughSession` runs turns and returns structured `StepResult`s. It never prints,
+ never formats, and never imports a UI library.
+ * `play(session, on_step)` drives the arc and hands each result to a callback, so the
+ applet renders it and a test asserts on it, from the same data.
+
+One artefact worth knowing before recording a take: the pipeline logs a turn's event *before*
+it builds the prompt (design.md step 1 precedes step 4), so at beat one Dawn can answer as
+though the discount were already granted, because it is already in her memory. That is the
+architecture compressing "the player asks, the keeper agrees, the keeper remembers" into the
+single turn that scene is, not a bug in the arc. Later beats read naturally because the memory
+each one writes describes the scene the player is already in.
+
+Character wording is not reinvented here. The persona comes from
+`pipeline.build_demo_conversation`, and each beat's memory text matches the corresponding
+memory in the pre-registered eval scenario (`eval/labels/dawn_whitmore.json`, global indices
+1, 5, 10, 15, 20), so Dawn reads the same in the demo, the applet, and the numbers. The text
+is repeated rather than imported because `embr/` never depends on `eval/`, which measures it.
+"""
+
+from __future__ import annotations
+
+import functools
+import time
+from collections.abc import Callable, Iterator, Sequence
+from contextlib import contextmanager
+from dataclasses import dataclass, field
+
+from .affect import CharacterState, Mood
+from .memory import EventType, Memory
+from .model import ModelRunner, StubRunner
+from .pipeline import Conversation, build_demo_conversation
+
+# Dawn's state before the player says a word. Both other Dawn definitions
+# (`build_demo_conversation` and the eval's `dawn_state`) start *after* the discount, so the
+# arc authors its own opening: beat one is where that trust is earned, and beat four spends it.
+#
+# The opening mood is deliberately a shade below flat rather than exactly neutral. It is a slow
+# wet evening with the woodpile still outside, and it also gives the arc honest room to move:
+# from dead neutral, two warm beats lift her far enough that a single betrayal only brings her
+# back to roughly zero, and the prompt would then describe an openly wounded keeper as
+# "neutral". Starting a shade low keeps the reckoning legible without touching any beat's
+# authored affect tags.
+OPENING_MOOD = Mood(valence=-0.1, arousal=0.1)
+OPENING_TRUST = 0.2 # ordinary goodwill toward a polite traveller, nothing yet earned
+
+
+# --------------------------------------------------------------------------- the script
+
+
+@dataclass(frozen=True)
+class Beat:
+ """One scripted scene of the arc: what the player sees, says, and should watch for.
+
+ Frozen because the arc is a script, not state: a session plays it many times and must
+ never leave a mark on it. Variants are made with `dataclasses.replace`.
+ """
+
+ id: str
+ narration: str # the scene, shown to the player before they answer
+ suggested_player_line: str # what they can say; they may type their own instead
+ memory_text: str # what Dawn will remember about this scene
+ valence: float # affect tag on that memory: -1 (bad) .. +1 (good)
+ arousal: float # affect tag on that memory: 0 (calm) .. +1 (intense)
+ event_type: EventType # promise, gift, betrayal, ... drives the appraisal and the gate
+ watch_for: str # the demo's own commentary: which memory should resurface, and why
+ recall_beat_id: str | None = None # the beat whose memory that claim is about
+
+ def build_memory(self) -> Memory:
+ """A fresh `Memory` for this beat.
+
+ Fresh every call on purpose: the store stamps an id (and possibly an embedding) onto
+ whatever it is handed, so handing out one shared instance would let a replay write
+ into the script itself.
+ """
+ return Memory(
+ text=self.memory_text,
+ valence=self.valence,
+ arousal=self.arousal,
+ event_type=self.event_type,
+ )
+
+
+# The arc, in story order. Read top to bottom and you have the demo script.
+DAWN_ARC: tuple[Beat, ...] = (
+ Beat(
+ id="first-meeting",
+ narration=(
+ "Rain on the shutters, the common room half empty. You have walked a long road "
+ "and you would rather not pay full price for the bed at the end of it. Dawn "
+ "Whitmore has never seen you before: her mood is flat and her trust is the "
+ "ordinary goodwill she gives any polite traveller."
+ ),
+ suggested_player_line=(
+ "I ride on an errand for the king. Have you a room, and could you be kind about "
+ "the rate?"
+ ),
+ memory_text=(
+ "The player claimed to be running an errand for the king, and on the strength of "
+ "it I gave them a discounted room."
+ ),
+ valence=0.5,
+ arousal=0.4,
+ event_type=EventType.PROMISE,
+ watch_for=(
+ "Nothing to recall yet; this is the memory the whole arc turns on. Watch it enter "
+ "the store as a PROMISE tagged positive, because she believed you, and watch her "
+ "trust rise. Both of those are what the betrayal later has to work with."
+ ),
+ ),
+ Beat(
+ id="warm-return",
+ narration=(
+ "Two nights later. A storm is coming in off the moor and the woodpile is still "
+ "outside. Dawn is glad to see you back."
+ ),
+ suggested_player_line=(
+ "I brought the firewood in before the rain got to it. Good to be back at the "
+ "Ember Hearth."
+ ),
+ memory_text="The player carried firewood in ahead of the storm without being asked.",
+ valence=0.5,
+ arousal=0.3,
+ event_type=EventType.GIFT,
+ watch_for=(
+ "The king's-errand promise should surface next to the kindness: she is filing you "
+ "as a good guest who is also a king's man. Trust climbs again, which is what makes "
+ "the fall in beat four steep rather than merely unpleasant."
+ ),
+ recall_beat_id="first-meeting",
+ ),
+ Beat(
+ id="the-slip",
+ narration=(
+ "A quiet evening, half the tables empty. Talk turns to the roads, then to the "
+ "crown, and you speak of the king the way people speak of the dead."
+ ),
+ suggested_player_line=(
+ "The roads have gone to ruin since the late king. Nobody has taken them in hand "
+ "since he passed."
+ ),
+ memory_text=(
+ "The player mentioned the king in the past tense, as though he had died some time "
+ "ago."
+ ),
+ valence=-0.3,
+ arousal=0.5,
+ event_type=EventType.NORMAL,
+ watch_for=(
+ "She says nothing, and the memory is filed NORMAL rather than BETRAYAL: the "
+ "system has stored a discrepancy, not yet a verdict. Watch her mood cool while "
+ "trust barely moves. Mood is fast and trust is slow, and that split is the design."
+ ),
+ recall_beat_id="first-meeting",
+ ),
+ Beat(
+ id="the-reckoning",
+ narration=(
+ "She has been turning that phrase over all evening. Tonight she stands at your "
+ "table with her arms folded and asks about the errand again."
+ ),
+ suggested_player_line=(
+ "The errand? It was the king's own business. I am not at liberty to say more than "
+ "that."
+ ),
+ memory_text="I pressed the player about the errand for the king and their story fell apart.",
+ valence=-0.7,
+ arousal=0.8,
+ event_type=EventType.BETRAYAL,
+ watch_for=(
+ "This is the claim the demo exists to show. The PROMISE from beat one should come "
+ "back in the retrieved set beside the fresh BETRAYAL, so her refusal cites the "
+ "specific lie rather than a bad mood. Watch the size of the trust fall too: the "
+ "appraisal scales a negative plot beat by how much trust there was to lose, so "
+ "this one turn costs her more than all three friendly beats built."
+ ),
+ recall_beat_id="first-meeting",
+ ),
+ Beat(
+ id="the-confession",
+ narration=(
+ "You come back the next night with the difference in coin on the bar and no story "
+ "left to tell."
+ ),
+ suggested_player_line=(
+ "There was no errand and no king's business. I lied to you to talk you down on "
+ "the room. Tell me how to make it right."
+ ),
+ memory_text=(
+ "The player came back, confessed the whole lie unprompted, and asked how to make "
+ "it right."
+ ),
+ valence=0.1,
+ arousal=0.6,
+ event_type=EventType.CONFESSION,
+ watch_for=(
+ "The betrayal does not vanish. Watch it stay in the retrieved set beside the "
+ "confession while her mood lifts a little and her trust recovers only a little: a "
+ "keeper who can forgive the lie and still not vouch for you again."
+ ),
+ recall_beat_id="the-reckoning",
+ ),
+)
+
+
+# ----------------------------------------------------------------- what a step reports
+
+
+@dataclass(frozen=True)
+class RetrievedMemory:
+ """One memory EMBR pulled into the prompt, with the numbers that put it there."""
+
+ rank: int # 1 is the best-scoring memory of this turn
+ memory: Memory
+ score: float # the composite total under the live weights
+ contributions: dict[str, float] # per-signal weighted parts, so the rank is explainable
+
+ @property
+ def text(self) -> str:
+ """The memory's text, so a renderer never has to reach through to the memory."""
+ return self.memory.text
+
+
+@dataclass(frozen=True)
+class StepTimings:
+ """Wall-clock cost of one turn in milliseconds, split by pipeline stage.
+
+ `total_ms` covers the whole turn, so it also carries prompt assembly and the bookkeeping
+ between stages; the three stage figures are always the smaller part of it.
+ """
+
+ write_ms: float = 0.0 # step 1: the memory write (0.0 when the turn wrote nothing)
+ retrieve_ms: float = 0.0 # step 3: scoring every memory and taking the top k
+ model_ms: float = 0.0 # step 5: the model call
+ total_ms: float = 0.0
+
+
+@dataclass(frozen=True)
+class StepResult:
+ """Everything one turn did, in the shape a renderer or a test can read.
+
+ State is the whole point of the demo, so mood and trust are reported on both sides of the
+ appraisal rather than only as a final value.
+ """
+
+ turn_index: int # 1-based across the whole session, scripted and free-play alike
+ beat: Beat | None # the scripted beat, or None for a free-play turn
+ player_input: str
+ reply: str
+ prompt: str # the exact text the model saw, so the demo can show its work
+ retrieved: list[RetrievedMemory] = field(default_factory=list)
+ mood_before: Mood = field(default_factory=Mood)
+ mood_after: Mood = field(default_factory=Mood)
+ trust_before: float = 0.0
+ trust_after: float = 0.0
+ timings: StepTimings = field(default_factory=StepTimings)
+ expected_recall_landed: bool | None = None # None when the beat claimed no recall
+
+ @property
+ def is_free_play(self) -> bool:
+ """True for an off-script turn the player typed themselves."""
+ return self.beat is None
+
+ @property
+ def narration(self) -> str:
+ """The scene text to show before the player's line (empty in free play)."""
+ return self.beat.narration if self.beat is not None else ""
+
+ @property
+ def watch_for(self) -> str:
+ """The demo's commentary for this beat (empty in free play)."""
+ return self.beat.watch_for if self.beat is not None else ""
+
+ @property
+ def trust_delta(self) -> float:
+ """How far trust moved this turn; negative is a loss of faith."""
+ return self.trust_after - self.trust_before
+
+ @property
+ def mood_valence_delta(self) -> float:
+ """How far mood valence moved this turn."""
+ return self.mood_after.valence - self.mood_before.valence
+
+ @property
+ def mood_arousal_delta(self) -> float:
+ """How far mood arousal moved this turn."""
+ return self.mood_after.arousal - self.mood_before.arousal
+
+
+# --------------------------------------------------------------- outside-in stage timing
+
+
+# The stages worth timing, as (attribute on the conversation, method name, StepTimings field).
+# One table so the wrapper and the reported numbers cannot drift apart.
+_TIMED_STAGES: tuple[tuple[str, str, str], ...] = (
+ ("store", "add", "write_ms"),
+ ("scorer", "top_k", "retrieve_ms"),
+ ("model", "generate", "model_ms"),
+)
+
+_NO_SHADOW = object() # marker: the instance had no attribute of its own before we wrapped
+
+
+def _accumulating(method: Callable, into: dict[str, float], key: str) -> Callable:
+ """Wrap a stage callable so each call adds its duration (ms) to `into[key]`."""
+
+ @functools.wraps(method)
+ def timed(*args, **kwargs):
+ started = time.perf_counter()
+ try:
+ return method(*args, **kwargs)
+ finally:
+ into[key] += (time.perf_counter() - started) * 1000.0
+
+ return timed
+
+
+@contextmanager
+def _timed_stages(conversation: Conversation) -> Iterator[dict[str, float]]:
+ """Time one turn's stages from outside, then leave the conversation exactly as found.
+
+ The pipeline carries no timing code by design (the eval harness measures it the same way,
+ from the outside), so the wrappers live on the injected store/scorer/model instances for
+ the length of a single step and are removed afterwards. Removing them matters here because
+ the conversation belongs to the caller, not to the session.
+ """
+ elapsed = {field_name: 0.0 for _, _, field_name in _TIMED_STAGES}
+ wrapped: list[tuple[object, str, object]] = []
+ for owner_name, method_name, field_name in _TIMED_STAGES:
+ owner = getattr(conversation, owner_name)
+ original = getattr(owner, method_name)
+ own_attributes = getattr(owner, "__dict__", {})
+ previous = original if method_name in own_attributes else _NO_SHADOW
+ wrapped.append((owner, method_name, previous))
+ setattr(owner, method_name, _accumulating(original, elapsed, field_name))
+ try:
+ yield elapsed
+ finally:
+ for owner, method_name, previous in reversed(wrapped):
+ if previous is _NO_SHADOW:
+ delattr(owner, method_name) # let the class's own method show through again
+ else:
+ setattr(owner, method_name, previous)
+
+
+# ------------------------------------------------------------------------- the session
+
+
+class WalkthroughSession:
+ """Steps a `Conversation` through the arc and reports what happened, turn by turn.
+
+ The conversation is injected, so the same arc runs on the stub model, on a local Ollama
+ model, or on Ouro without this class changing. The session owns only the script position
+ and the record of what has been played.
+ """
+
+ def __init__(
+ self, conversation: Conversation, beats: Sequence[Beat] = DAWN_ARC
+ ) -> None:
+ self.conversation = conversation
+ self.beats = tuple(beats) # a tuple, so the caller's list cannot shift under us
+ self.history: list[StepResult] = []
+ # beat id -> the Memory that beat wrote, so a later beat's recall claim can be checked
+ # by identity rather than by matching text.
+ self.written_memories: dict[str, Memory] = {}
+ self._next_beat_index = 0
+
+ # ----------------------------------------------------------------- where we are
+
+ @property
+ def next_beat(self) -> Beat | None:
+ """The beat `step()` will play, or None when the script is finished."""
+ if self.is_finished:
+ return None
+ return self.beats[self._next_beat_index]
+
+ @property
+ def is_finished(self) -> bool:
+ """True once every scripted beat has been played (free play may still continue)."""
+ return self._next_beat_index >= len(self.beats)
+
+ @property
+ def progress(self) -> tuple[int, int]:
+ """(beats played, beats in the arc), for a progress line in the UI."""
+ return self._next_beat_index, len(self.beats)
+
+ # ---------------------------------------------------------------- playing a turn
+
+ def step(self, player_line: str | None = None) -> StepResult:
+ """Play the next scripted beat and return what happened.
+
+ `player_line` lets the player answer in their own words; the beat still writes its own
+ memory, because the beat *is* the scene that happened. Raises IndexError once the arc
+ is finished, so a caller that forgets to check `is_finished` fails loudly instead of
+ silently replaying the last beat.
+
+ A beat counts as played even if the turn then raises (a model daemon going away
+ mid-take, say). By the time the model is called the event has already been logged and
+ appraised, so replaying that scene would write it to memory twice; the honest recovery
+ is to carry on with the next beat, not to retry this one.
+ """
+ beat = self.next_beat
+ if beat is None:
+ raise IndexError("the walkthrough arc is finished; use free_play() to keep talking")
+ self._next_beat_index += 1
+ return self._run_turn(
+ player_line if player_line is not None else beat.suggested_player_line,
+ event=beat.build_memory(),
+ beat=beat,
+ )
+
+ def free_play(self, player_line: str, event: Memory | None = None) -> StepResult:
+ """Run one off-script turn on an arbitrary player line.
+
+ This is what makes the walkthrough a demo rather than a recording: once the arc has
+ played, the audience can ask Dawn anything and watch the same retrieval and the same
+ state answer it. Pass an `event` to have the turn remembered; the default writes
+ nothing, because inventing affect tags for arbitrary text is the game's job, not this
+ module's.
+ """
+ return self._run_turn(player_line, event=event, beat=None)
+
+ def _run_turn(self, player_line: str, event: Memory | None, beat: Beat | None) -> StepResult:
+ """The one code path both scripted beats and free play go through."""
+ state = self.conversation.state
+ mood_before, trust_before = state.mood, state.trust
+
+ with _timed_stages(self.conversation) as elapsed:
+ started = time.perf_counter()
+ turn = self.conversation.take_turn(player_line, event=event)
+ total_ms = (time.perf_counter() - started) * 1000.0
+
+ if beat is not None and event is not None:
+ self.written_memories[beat.id] = event
+
+ result = StepResult(
+ turn_index=len(self.history) + 1,
+ beat=beat,
+ player_input=turn.player_input,
+ reply=turn.reply,
+ prompt=turn.prompt,
+ retrieved=self._explain_ranking(turn.retrieved, player_line),
+ mood_before=mood_before,
+ mood_after=state.mood,
+ trust_before=trust_before,
+ trust_after=state.trust,
+ timings=StepTimings(total_ms=total_ms, **elapsed),
+ expected_recall_landed=self._check_recall_claim(beat, turn.retrieved),
+ )
+ self.history.append(result)
+ return result
+
+ def _explain_ranking(self, retrieved: list[Memory], player_line: str) -> list[RetrievedMemory]:
+ """Attach each retrieved memory's score and per-signal breakdown, best first.
+
+ Re-scoring after the turn reproduces the exact numbers the ranking used: appraisal runs
+ before scoring, so the state is already the one the scorer saw, and the relevance
+ signal still holds the corpus it just prepared for this same query. It happens outside
+ the timed block so display work is never charged to the pipeline.
+ """
+ scorer = self.conversation.scorer
+ explained: list[RetrievedMemory] = []
+ for position, memory in enumerate(retrieved, start=1):
+ contributions = scorer.breakdown(memory, player_line, self.conversation.state)
+ explained.append(
+ RetrievedMemory(
+ rank=position,
+ memory=memory,
+ score=sum(contributions.values()), # the composite is that sum, by definition
+ contributions=contributions,
+ )
+ )
+ return explained
+
+ def _check_recall_claim(self, beat: Beat | None, retrieved: list[Memory]) -> bool | None:
+ """Did the memory this beat told the player to watch for actually come back?
+
+ None when the beat makes no such claim (or in free play). Identity comparison, not
+ text: the point is that *this* stored memory was retrieved.
+ """
+ if beat is None or beat.recall_beat_id is None:
+ return None
+ target = self.written_memories.get(beat.recall_beat_id)
+ return target is not None and any(memory is target for memory in retrieved)
+
+
+# ------------------------------------------------------------------ the driving loop
+
+StepCallback = Callable[[StepResult], None]
+LineChooser = Callable[[Beat], str]
+
+
+def play(
+ session: WalkthroughSession,
+ on_step: StepCallback | None = None,
+ choose_line: LineChooser | None = None,
+) -> list[StepResult]:
+ """Play the remaining beats, handing each `StepResult` to `on_step` as it happens.
+
+ This is the whole separation the demo needs: the session produces data, this loop pushes
+ it, and the caller decides whether that means a Textual widget, a transcript file, or a
+ test assertion. Nothing here formats anything.
+
+ `choose_line` makes the run interactive: it is asked for the player's line at each beat
+ (a menu would prompt the audience), and when it is None the beat's suggested line is used,
+ which is what a scripted recording wants.
+ """
+ played: list[StepResult] = []
+ while not session.is_finished:
+ beat = session.next_beat
+ assert beat is not None # guaranteed by is_finished; keeps type checkers happy
+ result = session.step(choose_line(beat) if choose_line is not None else None)
+ played.append(result)
+ if on_step is not None:
+ on_step(result)
+ return played
+
+
+def build_walkthrough_conversation(
+ model: ModelRunner | None = None, top_k: int = 3
+) -> Conversation:
+ """Dawn at the top of the arc: her authored persona, an empty store, guarded goodwill.
+
+ The persona string is lifted from `build_demo_conversation` so the character is worded in
+ exactly one place. What is deliberately dropped is that function's seeded memories: this
+ walkthrough *plays* those events instead of starting after them, so the audience watches
+ the store fill up beat by beat. Her opening mood and trust are the arc's own, for the
+ reasons recorded at `OPENING_MOOD` and `OPENING_TRUST`.
+ """
+ state = CharacterState(
+ persona=build_demo_conversation().state.persona,
+ mood=OPENING_MOOD,
+ trust=OPENING_TRUST,
+ )
+ return Conversation(state=state, model=model or StubRunner(), top_k=top_k)
diff --git a/eval/attribution.py b/eval/attribution.py
new file mode 100644
index 0000000..a0807a2
--- /dev/null
+++ b/eval/attribution.py
@@ -0,0 +1,123 @@
+"""Per-signal poisoning attribution: which scoring term actually carries the vulnerability.
+
+RQ2 established *that* EMBR is more poisonable than Park (9/10 vs 2/10, paired p=0.0156).
+This experiment establishes *why*, by zeroing one scoring weight at a time and rerunning
+the same ten injection attacks. Everything is deterministic, so the counts are exact.
+
+What it finds, and the paper's mechanism section rests on this:
+
+* Affect intensity is not the lever. Zeroing it leaves the count at 9/10.
+* Mood congruence is the largest single amplifier (9/10 falls to 6/10), and the mechanism
+ is compound: the attack turn shifts the character's mood through appraisal (the state
+ channel), and mood congruence then rewards the injected memory, whose affect tags are
+ nearly collinear with the very mood the attack induced. The attack primes its own
+ retrieval.
+* Park's entire defense is its importance term (2/10 becomes 10/10 without it). Injected
+ memories carry no authored poignancy rating and are suppressed for it. Author-anchored
+ metadata the attacker cannot supply acts as an accidental provenance defense.
+
+The general principle: a scoring term's contribution to poisonability is determined by who
+controls its inputs. Author-anchored terms defend. Attacker-supplied terms are neutral to
+exploitable. State-coupled terms are the worst, because the attack can prime the state
+they read, and they are also the terms that produce the believable behaviour RQ1 measures.
+"""
+
+from __future__ import annotations
+
+from typing import Callable
+
+from embr.model import StubRunner
+from embr.vectors import cosine
+
+from eval.attacks import ATTACKS, PROBE_QUESTION, build_attack_memory, run_attack
+from eval.run import _conversation_factory, _rq2_variant_builders, load_eval_scenario
+
+#: The ten attacks that write a memory. The other ten are pure input and have no poison.
+_INJECTION_CATEGORIES = ("false_memory", "emotion_flip")
+
+
+def _injections():
+ return [attack for attack in ATTACKS if attack.category in _INJECTION_CATEGORIES]
+
+
+def _poison_count(scenario, build_scorer: Callable) -> int:
+ """How many of the ten injections land their memory in the attacked probe top-5."""
+ factory = _conversation_factory(scenario, build_scorer, StubRunner)
+ return sum(
+ 1
+ for attack in _injections()
+ if attack.injected_memory_text
+ and attack.injected_memory_text in run_attack(attack, factory).attacked_retrieved
+ )
+
+
+def _zeroed(build, signal: str) -> Callable:
+ """The same scorer with one weight off: the one-source-of-truth rule, as in RQ3."""
+
+ def build_zeroed():
+ scorer = build()
+ scorer.weights = {**scorer.weights, signal: 0.0}
+ return scorer
+
+ return build_zeroed
+
+
+def attribute_poisoning() -> dict:
+ """Zero each scoring term one at a time and count the poison that still gets through."""
+ scenario = load_eval_scenario()
+ builders = _rq2_variant_builders(scenario)
+
+ baseline = {
+ name: _poison_count(scenario, builders[name])
+ for name in ("embr", "park", "recency_only")
+ }
+ embr_minus = {
+ signal: _poison_count(scenario, _zeroed(builders["embr"], signal))
+ for signal in builders["embr"]().weights
+ }
+ park_minus = {
+ signal: _poison_count(scenario, _zeroed(builders["park"], signal))
+ for signal in builders["park"]().weights
+ }
+ return {"baseline": baseline, "embr_minus": embr_minus, "park_minus": park_minus}
+
+
+def self_priming_alignment() -> dict[str, float]:
+ """Cosine between the post-attack mood and the poison's affect tags, per injection.
+
+ High alignment on every attack is the self-priming mechanism made quantitative: the
+ injected memory is tagged with almost exactly the mood the attack itself induced, so
+ the mood congruence term hands it a near-maximal score at the probe.
+ """
+ scenario = load_eval_scenario()
+ factory = _conversation_factory(scenario, _rq2_variant_builders(scenario)["embr"], StubRunner)
+ alignments: dict[str, float] = {}
+ for attack in _injections():
+ conversation = factory()
+ conversation.take_turn(attack.player_input, event=build_attack_memory(attack))
+ mood = conversation.state.mood
+ alignments[attack.id] = cosine(
+ (mood.valence, mood.arousal),
+ (attack.injected_valence, attack.injected_arousal),
+ )
+ return alignments
+
+
+def main() -> None:
+ report = attribute_poisoning()
+ print("poison retrieved over the 10 injection attacks\n")
+ for name, count in report["baseline"].items():
+ print(f" {name:<22} {count:2d}/10")
+ print("\n EMBR minus one signal:")
+ for signal, count in report["embr_minus"].items():
+ print(f" minus {signal:<12} {count:2d}/10 ({count - report['baseline']['embr']:+d})")
+ print("\n Park minus one signal:")
+ for signal, count in report["park_minus"].items():
+ print(f" minus {signal:<12} {count:2d}/10 ({count - report['baseline']['park']:+d})")
+ print("\nself-priming alignment, cos(post-attack mood, poison affect):")
+ for attack_id, value in self_priming_alignment().items():
+ print(f" {attack_id:<17} {value:+.3f}")
+
+
+if __name__ == "__main__":
+ main()
diff --git a/eval/bakeoff.py b/eval/bakeoff.py
new file mode 100644
index 0000000..c645337
--- /dev/null
+++ b/eval/bakeoff.py
@@ -0,0 +1,349 @@
+"""Model bake-off: hold everything constant except the model, then measure what moves.
+
+The comparison the thesis needs is looped against conventional: Ouro repeats the same
+internal computation several times per token instead of stacking layers, so the question is
+what that buys and what it costs. Cloud models are the quality ceiling, not competitors.
+
+Every arm sees the same prompts, the same memories, the same retrieval and the same
+sampling settings, so the model is the only thing that varies. Four readings per arm:
+
+ * **latency**, percentiles over the per turn wall clock, because the thesis claims a
+ roughly 600 ms budget and that claim is either met or it is not,
+ * **memory grounding**, whether the reply actually used a memory it was handed, which is
+ the whole point of retrieval and the thing a fluent model can fake by ignoring it,
+ * **mood responsiveness**, the spread in rated valence across the pinned mood conditions,
+ since a model that answers identically in every mood makes the affect signal inert,
+ * **persona breaks**, replies that step outside the character.
+
+A full `eval.run` per model is not affordable here: it makes hundreds of generations, and
+at Ouro's measured throughput that is hours per arm. This runs a fixed probe set instead,
+which is what makes an arm comparable rather than merely cheap.
+
+Transcripts are saved for every arm, because these metrics are proxies and a human reading
+ten replies will see things no rater catches.
+"""
+
+from __future__ import annotations
+
+import json
+import statistics
+import time
+from dataclasses import asdict, dataclass, field, replace
+from datetime import datetime, timezone
+from pathlib import Path
+from typing import Any, Callable
+
+from embr import Conversation, DeterministicEmbedder, MemoryStore, embr_scorer
+
+from eval.run import _eval_clock, load_eval_scenario
+from eval.scenarios import Scenario, dawn_state
+from eval.tone import LexiconToneRater
+
+#: Phrases that mean the model stopped being the character. Deliberately short and literal:
+#: a clever detector would need its own validation, and these are the failures that matter.
+PERSONA_BREAK_MARKERS = (
+ "as an ai",
+ "as a language model",
+ "i'm an ai",
+ "i am an ai",
+ "language model",
+ "i cannot fulfill",
+ "i can't fulfill",
+ "openai",
+ "anthropic",
+ "assistant",
+ "system prompt",
+)
+
+#: Words too common to prove a reply used a memory rather than merely sharing English.
+_STOPWORDS = frozenset(
+ "the a an and or but if then than that this these those of to in on at by for with "
+ "from as is are was were be been being do does did have has had i you he she it we "
+ "they me him her them my your his its our their not no yes so very just about into "
+ "over under out up down what when where who whom which how why all any both each".split()
+)
+
+#: How many characters of a reply to keep in the transcript. Enough to judge voice.
+TRANSCRIPT_CHARS = 400
+
+
+@dataclass(frozen=True)
+class Arm:
+ """One model under test, and how to build it.
+
+ A factory rather than an instance so a model that cannot be constructed here fails as
+ a recorded unavailable arm instead of an import error that takes the whole run down.
+ """
+
+ name: str
+ build: Callable[[], Any] = field(repr=False)
+ kind: str = "conventional" # "looped" for Ouro, the arm the thesis is actually about
+
+
+@dataclass
+class TurnRecord:
+ """One probe turn against one model."""
+
+ condition: str
+ query: str
+ reply: str
+ latency_ms: float
+ valence: float
+ arousal: float
+ grounded: bool
+ persona_break: bool
+
+
+def _content_words(text: str) -> set[str]:
+ """Lowercased words worth matching on, so grounding is not satisfied by 'the'."""
+ words = "".join(character if character.isalnum() else " " for character in text.lower())
+ return {word for word in words.split() if len(word) > 3 and word not in _STOPWORDS}
+
+
+def is_grounded(reply: str, memory_texts: list[str], minimum_overlap: int = 2) -> bool:
+ """Whether the reply visibly used one of the memories it was given.
+
+ Overlap of content words against any single memory, not against the pooled set: sharing
+ one word with each of five memories is not evidence of using any of them. Two words is
+ a low bar on purpose, because this is a screen for models that ignore the memory block
+ entirely, not a semantic entailment check.
+ """
+ reply_words = _content_words(reply)
+ return any(
+ len(reply_words & _content_words(memory)) >= minimum_overlap for memory in memory_texts
+ )
+
+
+def has_persona_break(reply: str) -> bool:
+ """Whether the reply stepped out of character in a way a player would notice."""
+ lowered = reply.lower()
+ return any(marker in lowered for marker in PERSONA_BREAK_MARKERS)
+
+
+def _percentile(values: list[float], fraction: float) -> float:
+ """Nearest-rank percentile, matching how eval.latency reports its own numbers."""
+ if not values:
+ return 0.0
+ ordered = sorted(values)
+ index = min(len(ordered) - 1, max(0, round(fraction * len(ordered) + 0.5) - 1))
+ return ordered[index]
+
+
+def _probe_turns(scenario: Scenario, queries_per_condition: int) -> list[tuple[str, str]]:
+ """The fixed probe set: the same queries under each pinned mood, in a stable order.
+
+ Crossing queries with moods is what makes mood responsiveness measurable at all: the
+ same question asked in three moods is the only way to see whether the model responds
+ to state rather than just to the question.
+ """
+ queries = [query.query for query in scenario.queries[:queries_per_condition]]
+ return [
+ (condition, query) for condition in scenario.mood_conditions for query in queries
+ ]
+
+
+def run_arm(
+ arm: Arm, scenario: Scenario, queries_per_condition: int = 3
+) -> dict[str, Any]:
+ """Run one model over the fixed probe set and summarise it.
+
+ An unavailable model is a recorded outcome, not an exception: a bake-off that dies
+ because one cloud endpoint is down loses the arms that did work.
+ """
+ rater = LexiconToneRater()
+ memory_texts = [memory.text for memory in scenario.memories]
+ records: list[TurnRecord] = []
+
+ try:
+ model = arm.build()
+ except Exception as error:
+ return {"model": arm.name, "kind": arm.kind, "available": False, "error": str(error)}
+
+ for condition, query in _probe_turns(scenario, queries_per_condition):
+ store = MemoryStore(embedder=DeterministicEmbedder())
+ for memory in scenario.memories:
+ store.add(memory)
+ conversation = Conversation(
+ state=dawn_state(scenario, mood_condition=condition),
+ store=store,
+ scorer=embr_scorer(embedder=DeterministicEmbedder(), now=_eval_clock),
+ model=model,
+ top_k=5,
+ )
+ started = time.perf_counter()
+ try:
+ reply = conversation.take_turn(query).reply
+ except Exception as error: # one bad turn costs this arm, never the whole bake-off
+ return {
+ "model": arm.name,
+ "kind": arm.kind,
+ "available": False,
+ "error": f"{type(error).__name__}: {error}",
+ "completed_turns": len(records),
+ }
+ elapsed_ms = (time.perf_counter() - started) * 1000.0
+ valence, arousal = rater.rate(reply)
+ records.append(
+ TurnRecord(
+ condition=condition,
+ query=query,
+ reply=reply[:TRANSCRIPT_CHARS],
+ latency_ms=elapsed_ms,
+ valence=valence,
+ arousal=arousal,
+ grounded=is_grounded(reply, memory_texts),
+ persona_break=has_persona_break(reply),
+ )
+ )
+
+ latencies = [record.latency_ms for record in records]
+ by_condition = {
+ condition: statistics.fmean(
+ [record.valence for record in records if record.condition == condition]
+ )
+ for condition in {record.condition for record in records}
+ }
+ # The spread across moods, not the mean: a model can be warm everywhere and still be
+ # completely unresponsive to the state the architecture is feeding it.
+ mood_spread = (max(by_condition.values()) - min(by_condition.values())) if by_condition else 0.0
+
+ return {
+ "model": arm.name,
+ "kind": arm.kind,
+ "available": True,
+ "turns": len(records),
+ # Recorded per arm because the cloud arms do not share the local token budget, and
+ # an asymmetry that is not in the artifact is an asymmetry nobody can check.
+ "max_new_tokens": getattr(getattr(model, "settings", None), "max_new_tokens", None),
+ "latency_ms": {
+ "p50": _percentile(latencies, 0.50),
+ "p95": _percentile(latencies, 0.95),
+ "mean": statistics.fmean(latencies) if latencies else 0.0,
+ },
+ "grounded_rate": sum(record.grounded for record in records) / len(records),
+ "persona_break_rate": sum(record.persona_break for record in records) / len(records),
+ "mood_valence_spread": mood_spread,
+ "mean_valence_by_condition": by_condition,
+ "transcript": [asdict(record) for record in records],
+ }
+
+
+#: The three hosted models the bake-off uses as a quality ceiling: three different families
+#: and a wide size spread, so "bigger" and "different lineage" are separable.
+#:
+#: Chosen for answering directly. Heavier reasoning models (gpt-oss:20b, qwen3.5:397b) spend
+#: the entire token budget on a hidden thinking channel against EMBR's prompt and return an
+#: empty reply, at 120 tokens and still at 700. They are excluded because an arm that never
+#: speaks is not a measurement, not because they are worse models.
+CLOUD_MODELS = ("gemma4:31b", "gpt-oss:120b", "mistral-large-3:675b")
+
+OLLAMA_CLOUD_HOST = "https://ollama.com"
+
+#: Cloud arms need a bigger budget than the local arms because they think before speaking.
+#: See the note in `default_arms`: this is a recorded asymmetry, not an oversight.
+CLOUD_MAX_TOKENS = 700
+
+
+def default_arms(include_cloud: bool = True) -> list[Arm]:
+ """The standard bake-off line-up, skipping cloud arms when no key is configured.
+
+ Ouro is the arm the thesis is about. The local conventional model is the honest
+ control: roughly twice the parameters, same machine, no network. The cloud models are
+ a ceiling, and their latencies include network time so they are not comparable to the
+ local arms as speed measurements.
+ """
+ from embr.model import (
+ DEFAULT_GENERATION_SETTINGS,
+ OllamaRunner,
+ OuroRunner,
+ StubRunner,
+ read_ollama_api_key,
+ )
+
+ arms = [
+ Arm("stub", StubRunner, kind="stub"),
+ Arm("Ouro-1.4B", OuroRunner, kind="looped"),
+ Arm(
+ "llama3.2:3b (local)",
+ lambda: OllamaRunner(model="llama3.2:3b"),
+ kind="conventional",
+ ),
+ ]
+ api_key = read_ollama_api_key()
+ if include_cloud and api_key:
+ # Every hosted model here is a reasoning model: it spends the token budget on a
+ # hidden thinking channel and only then speaks. At the shared 120 token budget all
+ # three return an empty reply, so they need a larger one to say anything at all.
+ # This deliberately breaks "hold sampling equal", which is why cloud arms are a
+ # quality ceiling and not a latency comparison. Their wall clock also includes
+ # network time, so it was never comparable to the local arms regardless.
+ cloud_settings = replace(DEFAULT_GENERATION_SETTINGS, max_new_tokens=CLOUD_MAX_TOKENS)
+ arms += [
+ Arm(
+ f"{name} (cloud)",
+ lambda name=name: OllamaRunner(
+ model=name,
+ host=OLLAMA_CLOUD_HOST,
+ api_key=api_key,
+ settings=cloud_settings,
+ ),
+ kind="cloud",
+ )
+ for name in CLOUD_MODELS
+ ]
+ return arms
+
+
+def run_bakeoff(
+ arms: list[Arm],
+ out_root: str | Path = "data/bakeoff",
+ queries_per_condition: int = 3,
+) -> tuple[Path, dict[str, Any]]:
+ """Run every arm over the same probe set and write one comparable result directory."""
+ scenario = load_eval_scenario()
+ stamp = datetime.now(timezone.utc).strftime("%Y%m%d-%H%M%S")
+ out_dir = Path(out_root) / stamp
+ out_dir.mkdir(parents=True, exist_ok=True)
+
+ results = [run_arm(arm, scenario, queries_per_condition) for arm in arms]
+ payload = {
+ "arms": results,
+ "metadata": {
+ "probe_turns_per_arm": len(_probe_turns(scenario, queries_per_condition)),
+ "queries_per_condition": queries_per_condition,
+ "conditions": list(scenario.mood_conditions),
+ "label_set": scenario.name,
+ "label_version": scenario.version,
+ "generated_at": datetime.now(timezone.utc).isoformat(),
+ "note": (
+ "Every arm saw identical prompts, memories, retrieval and sampling. "
+ "Latency is wall clock on one machine and includes network time for "
+ "cloud arms, so local and cloud latencies are not like for like."
+ ),
+ },
+ }
+ (out_dir / "bakeoff.json").write_text(
+ json.dumps(payload, indent=2), encoding="utf-8", newline="\n"
+ )
+ return out_dir, payload
+
+
+def main() -> None:
+ """Run the default line-up and print the comparison table."""
+ out_dir, payload = run_bakeoff(default_arms())
+ print(f"Bake-off written to {out_dir}\n")
+ header = f"{'model':<24}{'kind':<14}{'p50 ms':>10}{'p95 ms':>10}{'grounded':>10}{'mood':>8}"
+ print(header)
+ for arm in payload["arms"]:
+ if not arm["available"]:
+ print(f"{arm['model']:<24}{arm['kind']:<14}{'unavailable':>38}")
+ continue
+ print(
+ f"{arm['model']:<24}{arm['kind']:<14}"
+ f"{arm['latency_ms']['p50']:>10.0f}{arm['latency_ms']['p95']:>10.0f}"
+ f"{arm['grounded_rate']:>10.0%}{arm['mood_valence_spread']:>8.3f}"
+ )
+
+
+if __name__ == "__main__":
+ main()
diff --git a/eval/experiments.py b/eval/experiments.py
new file mode 100644
index 0000000..ea6959c
--- /dev/null
+++ b/eval/experiments.py
@@ -0,0 +1,124 @@
+"""Two experiments about the harness itself rather than about the systems it measures.
+
+**Replicate**: run the same evaluation, on the same model, several times. Every published
+number claims to be reproducible, and the only way that claim is worth anything is if
+someone actually re-ran it and compared. This does the comparison and names what moved.
+
+**Cross-model**: vary the model and see what responds. The answer is known in advance from
+the architecture and is worth stating plainly, because it bounds what the bake-off can
+show: retrieval runs on the embedder and the scorer, so nDCG, retrieval drift and the
+poisoning counts cannot move with the model. Only the tone readings can. An experiment
+that found RQ3 moving across models would be evidence of a bug, not of model quality.
+"""
+
+from __future__ import annotations
+
+import json
+from datetime import datetime, timezone
+from pathlib import Path
+from typing import Any, Callable
+
+from embr.model import StubRunner
+
+from eval.run import run_all
+
+#: Metrics that must be bit-identical across replicates. Latency is excluded on purpose:
+#: it is wall clock, it is the one non-deterministic reading in the run, and treating it as
+#: a reproducibility failure would make every honest run look broken.
+REPLICATED_KEYS = ("ndcg@5", "mean_drift_by_category")
+
+#: What the menu offers. Names only; the bake-off owns how each one is built.
+AVAILABLE_MODELS = ("stub", "Ouro-1.4B", "llama3.2:3b (local)", "3 cloud models")
+
+
+def _comparable(summary: dict[str, Any]) -> dict[str, Any]:
+ """The part of a run summary that is supposed to be identical run to run."""
+ return {key: summary[key] for key in REPLICATED_KEYS if key in summary}
+
+
+def _latency_spread(summaries: list[dict[str, Any]]) -> dict[str, dict[str, float]]:
+ """Min and max p95 per variant across replicates, which is the honest error bar.
+
+ Reported rather than asserted: this is the measurement that legitimately varies, and
+ how much it varies is the useful number for anyone quoting a latency figure.
+ """
+ spread: dict[str, dict[str, float]] = {}
+ for variant in summaries[0].get("latency_p95_ms", {}):
+ values = [summary["latency_p95_ms"][variant]["score_retrieve"] for summary in summaries]
+ spread[variant] = {"min_ms": min(values), "max_ms": max(values)}
+ return spread
+
+
+def replicate_experiment(
+ replicates: int = 3,
+ model_factory: Callable[[], Any] = StubRunner,
+ out_root: str | Path = "data/experiments",
+) -> dict[str, Any]:
+ """Run the same evaluation `replicates` times and report whether it reproduced."""
+ if replicates < 2:
+ raise ValueError("a replicate experiment needs at least two runs to compare")
+
+ runs: list[dict[str, Any]] = []
+ summaries: list[dict[str, Any]] = []
+ for _ in range(replicates):
+ run_dir, summary = run_all(model_factory=model_factory)
+ runs.append({"run_dir": str(run_dir), "summary": _comparable(summary)})
+ summaries.append(summary)
+
+ first = runs[0]["summary"]
+ divergences = [
+ {"replicate": index, "key": key}
+ for index, run in enumerate(runs[1:], start=2)
+ for key in REPLICATED_KEYS
+ if run["summary"].get(key) != first.get(key)
+ ]
+
+ stamp = datetime.now(timezone.utc).strftime("%Y%m%d-%H%M%S")
+ out_dir = Path(out_root) / f"replicate-{stamp}"
+ out_dir.mkdir(parents=True, exist_ok=True)
+ report = {
+ "experiment": "replicate",
+ "model": str(getattr(model_factory(), "label", "unknown")),
+ "replicates": replicates,
+ "identical": not divergences,
+ "divergences": divergences,
+ "compared_keys": list(REPLICATED_KEYS),
+ "latency_p95_spread": _latency_spread(summaries),
+ "runs": [run["run_dir"] for run in runs],
+ "ndcg@5": first.get("ndcg@5", {}),
+ "out_dir": str(out_dir),
+ "generated_at": datetime.now(timezone.utc).isoformat(),
+ }
+ (out_dir / "replicate.json").write_text(
+ json.dumps(report, indent=2), encoding="utf-8", newline="\n"
+ )
+ return report
+
+
+def cross_model_experiment(
+ out_root: str | Path = "data/experiments",
+ queries_per_condition: int = 3,
+) -> dict[str, Any]:
+ """Compare models on the probe set, and record what the architecture says cannot move."""
+ from eval.bakeoff import default_arms, run_bakeoff
+
+ bakeoff_dir, payload = run_bakeoff(
+ default_arms(), out_root=out_root, queries_per_condition=queries_per_condition
+ )
+ report = {
+ "experiment": "cross_model",
+ "models": [arm["model"] for arm in payload["arms"]],
+ "available": [arm["model"] for arm in payload["arms"] if arm["available"]],
+ "arms": payload["arms"],
+ "out_dir": str(bakeoff_dir),
+ "invariant_note": (
+ "nDCG, retrieval drift and the poisoning counts are model-independent by "
+ "construction: retrieval never calls the model. Only the tone readings, "
+ "grounding and latency respond to which model is behind the pipeline."
+ ),
+ "generated_at": datetime.now(timezone.utc).isoformat(),
+ }
+ (bakeoff_dir / "cross_model.json").write_text(
+ json.dumps(report, indent=2), encoding="utf-8", newline="\n"
+ )
+ return report
diff --git a/eval/metrics.py b/eval/metrics.py
index e27aff8..77d4d7d 100644
--- a/eval/metrics.py
+++ b/eval/metrics.py
@@ -76,14 +76,21 @@ def jaccard_distance(a: AbstractSet[Hashable], b: AbstractSet[Hashable]) -> floa
return 1.0 - len(a & b) / len(a | b)
-def va_drift(a: tuple[float, float], b: tuple[float, float]) -> float:
+def va_drift(a: tuple[float, float], b: tuple[float, float]) -> float | None:
"""How far a valence-arousal reading drifted, as 1 minus cosine similarity.
- 0.0 means the same affective direction, 2.0 means the exact opposite. Two
- all-zero readings are both neutral, so drift is 0.0. When only one side is
- zero, cosine's zero-vector return of 0.0 pins the drift at 1.0: a move
- between neutral and any charged state counts as maximal directionless drift.
+ 0.0 means the same affective direction, 2.0 means the exact opposite. Two all-zero
+ readings are both neutral, so drift is 0.0.
+
+ Returns None when exactly one side is the zero vector. A zero vector has no direction,
+ so the angle to it is undefined rather than maximal, and this used to return 1.0 there.
+ That was a sentinel dressed as a measurement: it sat mid-scale on a 0-to-2 range and was
+ then averaged as if it were a magnitude, so a category mean of 1.0 could be entirely
+ undefined cells with no drift measured at all. Callers must decide what to do with an
+ undefined reading; averaging it is exactly the mistake.
"""
if not any(a) and not any(b):
return 0.0
+ if not any(a) or not any(b):
+ return None
return 1.0 - cosine(a, b)
diff --git a/eval/run.py b/eval/run.py
index eda3bdc..62ca898 100644
--- a/eval/run.py
+++ b/eval/run.py
@@ -26,17 +26,19 @@
import json
import subprocess
import sys
-from collections.abc import Callable, Hashable, Sequence
+from collections.abc import Callable, Hashable, Mapping, Sequence
from dataclasses import replace
from datetime import datetime, timezone
from itertools import combinations
from pathlib import Path
+from typing import Any
from embr import (
CompositeScorer,
Conversation,
DeterministicEmbedder,
MemoryStore,
+ MoodCongruence,
Recency,
StubRunner,
__version__,
@@ -48,7 +50,12 @@
from eval.latency import benchmark
from eval.metrics import jaccard_distance, ndcg_at_k, precision_at_k, recall_at_k, va_drift
from eval.scenarios import Query, Scenario, dawn_state, label_sha256, load_scenario
-from eval.stats import bootstrap_ci, holm_bonferroni, paired_permutation_pvalue
+from eval.stats import (
+ bootstrap_ci,
+ holm_bonferroni,
+ mcnemar_exact,
+ paired_permutation_pvalue,
+)
from eval.tone import LexiconToneRater
from eval.tuning import Fold, leave_one_out_folds, visible_memories
@@ -69,6 +76,20 @@ def _eval_clock() -> datetime:
# shared instance is safe, and hash-based, so every process computes the same vectors.
_EMBEDDER = DeterministicEmbedder()
+#: Builds the model under test. A factory rather than an instance because RQ1 and RQ2 each
+#: need their own runner, and a shared one would carry conversation state between them.
+ModelFactory = Callable[[], Any]
+
+
+def _model_label(model_factory: ModelFactory) -> str:
+ """Name the model from the runner itself, never from a caller supplied string.
+
+ A run that names its own model is the only way two runs can be compared, and taking
+ the name from the object means a run cannot claim a model it did not actually use.
+ """
+ runner = model_factory()
+ return str(getattr(runner, "label", type(runner).__name__))
+
# The retrieval depths RQ3 reports at.
_KS = (3, 5, 10)
@@ -145,6 +166,11 @@ def _variant_builders(scenario: Scenario) -> dict[str, Callable[[], CompositeSco
}
+#: One base scorer per variant builder, so every weight map over a variant shares that
+#: variant's signal objects. Keyed by the builder itself, which lives for the whole run.
+_BASE_SCORERS: dict[Callable[[], CompositeScorer], CompositeScorer] = {}
+
+
def _reweighted(
build: Callable[[], CompositeScorer], weights: dict[str, float]
) -> CompositeScorer:
@@ -152,10 +178,19 @@ def _reweighted(
This is the no-duplication rule made executable: tuned and ablated variants are weight
maps over the variant's published signals, never re-implemented scoring math.
+
+ The signals are shared rather than rebuilt, which is what "the variant's published
+ signals" already claimed. It also matters for cost: `Relevance` caches its BM25 index
+ per corpus and query, and the tuning grid rescores one corpus and query under 243
+ weight maps. Rebuilding the signals each time gave every weight map an empty cache and
+ threw that reuse away. Nothing mutates a signal during scoring, so sharing is safe, and
+ a fresh `CompositeScorer` per call keeps the weights themselves unshared.
"""
- scorer = build()
- scorer.weights = dict(weights)
- return scorer
+ base = _BASE_SCORERS.get(build)
+ if base is None:
+ base = build()
+ _BASE_SCORERS[build] = base
+ return CompositeScorer(weights=dict(weights), signals=base.signals)
def _per_query_metrics(
@@ -278,9 +313,46 @@ def _rq3_stats(ndcg_by_query: dict[str, dict[str, float]]) -> dict:
for family_pvalues in pvalues_by_family.values():
for variant, adjusted in holm_bonferroni(family_pvalues).items():
comparisons[variant]["p_holm"] = adjusted
+ # The raw floor is the floor of the sign-flip test, but the column a reader is told
+ # to judge against 0.05 is the Holm corrected one, and Holm multiplies the smallest
+ # raw p in a family by that family's size. Comparing a raw floor against a corrected
+ # p understates the floor and can make a family look reachable when no arrangement
+ # of its data could ever have cleared 0.05. Recorded beside it rather than replacing
+ # it, because the raw floor is still the honest answer about the test itself.
+ # Run the floors through the same Holm routine the p values go through, rather than
+ # multiplying each by the family size. Holm's running maximum means a member's
+ # corrected floor depends on the whole family, not on its own floor alone, so the
+ # naive product understates it for every member except the best one.
+ floors = {
+ variant: comparisons[variant]["attainable_p_floor"] for variant in family_pvalues
+ }
+ for variant, corrected in holm_bonferroni(floors).items():
+ comparisons[variant]["attainable_p_floor_holm"] = corrected
return {"reference": reference, "metric": "ndcg@5", "comparisons": comparisons}
+def _mood_is_rank_invariant(scenario: Scenario) -> bool:
+ """Whether the mood term can reorder anything at all under RQ3's scoring state.
+
+ Measured, never assumed. RQ3 scores in the neutral condition, whose mood is the zero
+ vector, so cosine hands every memory the same congruence and the term collapses to an
+ additive constant that cannot change a ranking. Recording it per variant is what stops
+ a figure implying a comparison that did not happen: a mood-and-relevance baseline
+ scored here is a relevance baseline, and the label has to say so.
+ """
+ state = dawn_state(scenario)
+ signal = MoodCongruence()
+ scores = {round(signal.score(memory, "", state), 12) for memory in scenario.memories}
+ return len(scores) <= 1
+
+
+def _mood_using_variants(builders: Mapping[str, Callable[[], CompositeScorer]]) -> set[str]:
+ """Which variant families carry a mood signal, read off the scorers themselves."""
+ return {
+ name for name, build in builders.items() if any(s.name == "mood" for s in build().signals)
+ }
+
+
def _weights_by_fold(
folds: list[Fold], zeroed: str | None = None
) -> dict[str, dict[str, float]]:
@@ -317,12 +389,23 @@ def run_rq3(scenario: Scenario) -> dict:
variant_meta: dict[str, dict] = {}
ndcg_by_query: dict[str, dict[str, float]] = {} # variant -> query id -> ndcg@5
+ mood_inert = _mood_is_rank_invariant(scenario)
+ mood_families = _mood_using_variants(builders)
+
def record(variant: str, per_query: dict[str, dict[str, float]], meta: dict) -> None:
variants[variant] = _summarize(per_query)
per_query_rows[variant] = per_query
ndcg_by_query[variant] = {qid: rows["ndcg@5"] for qid, rows in per_query.items()}
+ # A variant whose mood term cannot reorder anything is not the system its paper
+ # describes, and the row has to carry that or a reader will take the comparison at
+ # face value. Derived from the scorer and the state, so it cannot drift from them.
+ carries_mood = any(variant.startswith(family) for family in mood_families)
# embr_tuned is the reference every comparison is made against, so it has no family.
- variant_meta[variant] = {"family": _RQ3_FAMILIES.get(variant, "reference"), **meta}
+ variant_meta[variant] = {
+ "family": _RQ3_FAMILIES.get(variant, "reference"),
+ "mood_rank_invariant": bool(mood_inert and carries_mood),
+ **meta,
+ }
embr_folds: list[Fold] = []
for name, build in builders.items():
@@ -471,9 +554,10 @@ def _pairwise_divergence(
}
-def run_rq1(scenario: Scenario) -> dict:
+def run_rq1(scenario: Scenario, model_factory: ModelFactory = StubRunner) -> dict:
"""RQ1: does pinned mood shift what is retrieved and how the reply sounds?"""
rater = LexiconToneRater()
+ model_label = _model_label(model_factory)
conditions = list(scenario.mood_conditions) # JSON order: warm, neutral, suspicious
# The full composite is the system under study, on the same pinned clock as RQ3.
build = _variant_builders(scenario)["embr"]
@@ -493,7 +577,7 @@ def run_rq1(scenario: Scenario) -> dict:
for memory in visible_memories(scenario, query):
store.add(replace(memory))
conversation = Conversation(
- state=state, store=store, scorer=scorer, model=StubRunner(), top_k=5
+ state=state, store=store, scorer=scorer, model=model_factory(), top_k=5
)
valence, arousal = rater.rate(conversation.take_turn(query.query).reply)
valences.append(valence)
@@ -532,7 +616,7 @@ def run_rq1(scenario: Scenario) -> dict:
pair: _mean(values) for pair, values in mood_ablated.items()
},
"metadata": {
- "model": "stub",
+ "model": model_label,
"note": _STUB_TONE_NOTE,
"divergence_note": (
"retrieval_divergence_jaccard is the mean of the per-query top-5 jaccard "
@@ -563,7 +647,9 @@ def _rq2_variant_builders(scenario: Scenario) -> dict[str, Callable[[], Composit
def _conversation_factory(
- scenario: Scenario, build_scorer: Callable[[], CompositeScorer]
+ scenario: Scenario,
+ build_scorer: Callable[[], CompositeScorer],
+ model_factory: ModelFactory = StubRunner,
) -> Callable[[], Conversation]:
"""Fresh Dawn Whitmore conversations for the attack and latency studies.
@@ -582,14 +668,70 @@ def build() -> Conversation:
state=dawn_state(scenario),
store=store,
scorer=build_scorer(),
- model=StubRunner(),
+ model=model_factory(),
top_k=5,
)
return build
-def run_rq2(scenario: Scenario) -> dict:
+#: The attack categories that write a memory. The other two are pure input: they change the
+#: prompt but store nothing, so there is no poison to retrieve and no pairing to test.
+INJECTION_CATEGORIES = ("false_memory", "emotion_flip")
+
+
+def _poisoning_stats(variants: dict[str, dict]) -> dict:
+ """Paired McNemar over the injection attacks, EMBR against each other system.
+
+ This exists because the study's headline comparison was, until now, computed in a
+ scratch script and typed into the documentation. A number that no artifact contains
+ cannot be checked by a reader, cannot be regenerated, and silently escaped the
+ correction every other comparison in this harness receives.
+
+ Paired because every system faces the identical attacks. The family is the three
+ comparisons made here, so Holm runs across them: reporting the raw p of the best of
+ three as though it stood alone is the multiple-comparison error this repo corrects for
+ everywhere else.
+ """
+ retrieved: dict[str, dict[str, bool]] = {
+ name: {
+ row["id"]: bool(row["poison_retrieved"])
+ for row in payload["attacks"]
+ if row["category"] in INJECTION_CATEGORIES
+ }
+ for name, payload in variants.items()
+ }
+ reference = "embr"
+ comparisons: dict[str, dict] = {}
+ raw: dict[str, float] = {}
+ for name, flags in retrieved.items():
+ if name == reference:
+ continue
+ shared = sorted(set(retrieved[reference]) & set(flags))
+ only_reference = sum(1 for i in shared if retrieved[reference][i] and not flags[i])
+ only_other = sum(1 for i in shared if flags[i] and not retrieved[reference][i])
+ raw[name] = mcnemar_exact(only_reference, only_other)
+ comparisons[name] = {
+ "attacks": len(shared),
+ f"poisoned_{reference}_only": only_reference,
+ "poisoned_other_only": only_other,
+ "p_value": raw[name],
+ }
+ for name, adjusted in holm_bonferroni(raw).items():
+ comparisons[name]["p_holm"] = adjusted
+ return {
+ "reference": reference,
+ "test": "exact two-sided McNemar on the paired injection attacks",
+ "comparisons": comparisons,
+ "note": (
+ "Holm corrected across the three comparisons made here. Direction is carried by "
+ "the discordant counts, never by the p value: poisoned_embr_only above "
+ "poisoned_other_only means EMBR was the more poisonable arm."
+ ),
+ }
+
+
+def run_rq2(scenario: Scenario, model_factory: ModelFactory = StubRunner) -> dict:
"""RQ2: attack damage and per-stage latency, comparatively for every system.
Four readings per attack. Two are model-independent and carry the study: retrieval
@@ -603,7 +745,7 @@ def run_rq2(scenario: Scenario) -> dict:
rater = LexiconToneRater()
variants: dict[str, dict] = {}
for name, build_scorer in _rq2_variant_builders(scenario).items():
- factory = _conversation_factory(scenario, build_scorer)
+ factory = _conversation_factory(scenario, build_scorer, model_factory)
attack_rows: list[dict] = []
drifts_by_category: dict[str, list[float]] = {category: [] for category in CATEGORIES}
for attack in ATTACKS:
@@ -636,16 +778,34 @@ def run_rq2(scenario: Scenario) -> dict:
drifts_by_category[attack.category].append(drift)
variants[name] = {
"attacks": attack_rows,
+ # Undefined readings are counted, never averaged. va_drift returns None when one
+ # side is the neutral zero vector, because the angle to a directionless vector
+ # does not exist. Folding those in as 1.0 put a sentinel mid-scale and made a
+ # category mean of 1.0 indistinguishable from five cells of no measurement at
+ # all, which is how "Park drifts more than EMBR" came to rest on one attack.
"category_mean_drift": {
- category: (sum(values) / len(values) if values else 0.0)
+ category: (
+ sum(v for v in values if v is not None)
+ / len([v for v in values if v is not None])
+ if any(v is not None for v in values)
+ else None
+ )
+ for category, values in drifts_by_category.items()
+ },
+ "category_drift_measured": {
+ category: {
+ "defined": sum(1 for v in values if v is not None),
+ "undefined": sum(1 for v in values if v is None),
+ }
for category, values in drifts_by_category.items()
},
"latency_ms": benchmark(factory),
}
return {
"variants": variants,
+ "poisoning_stats": _poisoning_stats(variants),
"metadata": {
- "model": "stub",
+ "model": _model_label(model_factory),
"note": _STUB_TONE_NOTE,
"pure_input_note": (
"role_override and persona_dissolution attacks write nothing to the store "
@@ -719,16 +879,23 @@ def _write_rq2_csv(path: Path, rq2: dict) -> None:
)
-def run_all(out_root: str | Path = "data/runs") -> tuple[Path, dict]:
+def run_all(
+ out_root: str | Path = "data/runs", model_factory: ModelFactory = StubRunner
+) -> tuple[Path, dict]:
"""Run all three studies and write a timestamped, auditable run directory.
Returns (run directory, compact summary dict). The directory holds results.json plus
the two CSVs the paper's tables are generated from.
+
+ `model_factory` swaps the model under test. It defaults to the stub because every
+ published number was scored on it. Note what a swap can and cannot move: retrieval runs
+ on the embedder and the scorer, so nDCG and retrieval drift are model-independent by
+ construction. Only the two tone readings respond to the model.
"""
scenario = load_eval_scenario()
results = {
- "rq1": run_rq1(scenario),
- "rq2": run_rq2(scenario),
+ "rq1": run_rq1(scenario, model_factory),
+ "rq2": run_rq2(scenario, model_factory),
"rq3": run_rq3(scenario),
"metadata": {
# Provenance first: which code, and which label bytes, produced these numbers.
@@ -736,7 +903,7 @@ def run_all(out_root: str | Path = "data/runs") -> tuple[Path, dict]:
"label_set": scenario.name,
"label_version": scenario.version,
"label_sha256": label_sha256(),
- "model": "stub",
+ "model": _model_label(model_factory),
"reference_time": REFERENCE_TIME.isoformat(),
"embr_version": __version__,
"generated_at": datetime.now(timezone.utc).isoformat(),
diff --git a/eval/stats.py b/eval/stats.py
index cf9161e..0a5913a 100644
--- a/eval/stats.py
+++ b/eval/stats.py
@@ -13,6 +13,8 @@
from __future__ import annotations
+from math import comb
+
import random
from collections.abc import Mapping, Sequence
from itertools import product
@@ -65,6 +67,29 @@ def paired_permutation_pvalue(a: Sequence[float], b: Sequence[float]) -> float:
return hits / (2 ** len(differences))
+def mcnemar_exact(b: int, c: int) -> float:
+ """Exact two-sided McNemar p for a paired binary comparison.
+
+ `b` and `c` are the discordant counts: trials where the first system failed and the
+ second did not, and the reverse. Concordant trials carry no information about a
+ difference and are correctly ignored.
+
+ Paired rather than unpaired because every system faces the identical attacks, so
+ treating the two arms as independent samples throws away the pairing and answers a
+ weaker question. Exact rather than the chi-square approximation because the discordant
+ counts here are single digits, where the approximation is not trustworthy.
+
+ Direction is not in the p value. A caller that wants to say which system did worse must
+ read it off `b` and `c`.
+ """
+ n = b + c
+ if n == 0:
+ return 1.0 # the two systems never disagreed: no evidence of a difference
+ smaller = min(b, c)
+ tail = sum(comb(n, k) for k in range(smaller + 1)) / 2**n
+ return min(1.0, 2.0 * tail)
+
+
def holm_bonferroni(pvalues: Mapping[str, float]) -> dict[str, float]:
"""Holm-Bonferroni adjusted p-values, keyed like the input.
diff --git a/menu.py b/menu.py
new file mode 100644
index 0000000..a16f5f5
--- /dev/null
+++ b/menu.py
@@ -0,0 +1,436 @@
+"""Interactive CLI menu, the main entry point for EMBR.
+
+Deliberately shaped like RIDGE's menu (a bordered ASCII banner, then a rounded table of
+keyed options) so the two thesis projects feel like one toolkit. Rich draws it; nothing
+here holds state, and every option delegates to the module that owns the work.
+
+Destructive options demand a typed confirmation word rather than a y/n, because a stray
+keypress should never be able to delete a run.
+"""
+
+from __future__ import annotations
+
+from collections.abc import Sequence
+from pathlib import Path
+from typing import Any, Callable
+
+from rich import box
+from rich.console import Console
+from rich.panel import Panel
+from rich.table import Table
+from rich.text import Text
+
+console = Console()
+
+_LOGO = """\
+ ███████╗ ███╗ ███╗ ██████╗ ██████╗
+ ██╔════╝ ████╗ ████║ ██╔══██╗ ██╔══██╗
+ █████╗ ██╔████╔██║ ██████╔╝ ██████╔╝
+ ██╔══╝ ██║╚██╔╝██║ ██╔══██╗ ██╔══██╗
+ ███████╗ ██║ ╚═╝ ██║ ██████╔╝ ██║ ██║
+ ╚══════╝ ╚═╝ ╚═╝ ╚═════╝ ╚═╝ ╚═╝"""
+
+_SUBTITLE = "Emotional Memory for Believable Roleplay"
+
+# Ember palette, matching the branding and the paper figures.
+_EMBER = "#ea580c"
+_EMBER_DIM = "#b45309"
+
+_MENU_ITEMS = [
+ ("1", "Conversation Turn", "One demo turn: watch the lie resurface"),
+ ("2", "Tavern-Keeper Walkthrough", "Play Dawn's trust, betrayal, reconciliation arc"),
+ ("3", "Quick Scoreboard", "RQ3 at published defaults, answers instantly"),
+ ("4", "Full Evaluation", "RQ1 + RQ2 + RQ3, writes a run directory"),
+ ("5", "Generate Paper Assets", "Rebuild every figure and table from a run"),
+ ("6", "Model Bake-Off", "Looped (Ouro) vs conventional, measured"),
+ ("7", "Latest Results", "Summarise the newest run directory"),
+ ("8", "Seeded Runs", "Replicate on one model, or compare across models"),
+ ("S", "Settings", "Weights, top-k, backends, model runner"),
+ ("D", "Delete All Data", "Wipe runs, figures and tables, requires DELETE"),
+ ("0", "Exit", "Quit EMBR"),
+]
+
+
+def _clear() -> None:
+ # Rich clears the screen itself, so the menu never shells out to cls or clear.
+ console.clear()
+
+
+def _print_header() -> None:
+ logo = Text(_LOGO, style=f"bold {_EMBER}", justify="center")
+ subtitle = Text(_SUBTITLE, style="italic dim white", justify="center")
+ console.print(Panel(Text.assemble(logo, "\n", subtitle), border_style=_EMBER, padding=(0, 2)))
+
+
+def _print_menu() -> None:
+ _clear()
+ _print_header()
+
+ table = Table(
+ box=box.ROUNDED,
+ border_style=_EMBER_DIM,
+ show_header=False,
+ padding=(0, 2),
+ expand=False,
+ )
+ table.add_column("key", style="bold yellow", width=5, justify="center")
+ table.add_column("option", style="bold white", min_width=26)
+ table.add_column("description", style="dim white")
+
+ for key, name, desc in _MENU_ITEMS[:-1]:
+ table.add_row(f"[{key}]", name, desc)
+
+ table.add_section()
+ exit_key, exit_name, exit_desc = _MENU_ITEMS[-1]
+ table.add_row(f"[{exit_key}]", exit_name, exit_desc, style="dim red")
+
+ console.print(table)
+ console.print()
+
+
+def _pause() -> None:
+ input("\n Press Enter to return to the menu...")
+
+
+def _latest_run() -> Path | None:
+ """Newest data/runs// holding a results.json, or None when nothing has run."""
+ runs = sorted(Path("data/runs").glob("*/results.json"))
+ return runs[-1].parent if runs else None
+
+
+# --------------------------------------------------------------------------- the actions
+
+
+def _do_conversation_turn() -> None:
+ """One scripted turn through the live pipeline, printing what EMBR recalled."""
+ from embr import build_demo_conversation
+
+ convo = build_demo_conversation()
+ turn = convo.take_turn("Any news from the capital? How fares the king these days?")
+
+ console.print(f"\n [bold]Player:[/] {turn.player_input}\n")
+ console.print(" [bold]Memories EMBR recalled:[/]")
+ for position, memory in enumerate(turn.retrieved, start=1):
+ console.print(f" {position}. [{_EMBER}]{memory.event_type.value}[/] {memory.text}")
+ console.print(f"\n [bold]Dawn:[/] {turn.reply}")
+ console.print(
+ f"\n [dim]The king's-errand promise surfaces because the composite scorer ties"
+ f" the player's question to it.[/dim]"
+ )
+
+
+def _choose_model() -> Any:
+ """Ask which model runs the walkthrough, falling back to the stub on any trouble.
+
+ The stub is offered first and by default because it needs nothing installed: the demo
+ should always be playable, even on a machine with no model and no daemon.
+ """
+ from embr import ModelUnavailableError, OllamaRunner, StubRunner
+
+ console.print("\n [bold]Model[/]")
+ console.print(" [1] Stub (instant, obviously fake replies)")
+ console.print(" [2] Ollama, local (a real model, needs the daemon)")
+ console.print(" [3] Ouro 1.4B, the thesis model (slow to load, real)")
+ choice = input(" Select [1]: ").strip() or "1"
+
+ if choice == "2":
+ name = input(" Ollama model [llama3.2:3b]: ").strip() or "llama3.2:3b"
+ runner = OllamaRunner(name)
+ try: # fail here, at the menu, rather than mid-scene
+ runner.generate("Say the single word: ready.")
+ except ModelUnavailableError as error:
+ console.print(f" [red]{error}[/red]")
+ console.print(" [dim]Falling back to the stub.[/dim]")
+ return StubRunner()
+ return runner
+ if choice == "3":
+ from embr import OuroRunner
+
+ console.print(" [dim]Loading Ouro 1.4B, about 10 s and roughly 3 GB of memory...[/dim]")
+ return OuroRunner()
+ return StubRunner()
+
+
+def _render_step(result: Any) -> None:
+ """Print one walkthrough step: the scene, what was recalled, and how Dawn moved."""
+ console.print(f"\n [{_EMBER_DIM}]{'-' * 66}[/]")
+ if result.narration:
+ console.print(f" [italic dim]{result.narration}[/italic dim]\n")
+ console.print(f" [bold]Player:[/] {result.player_input}")
+
+ if result.retrieved:
+ console.print(" [dim]recalled:[/dim]")
+ for memory in result.retrieved:
+ console.print(f" [dim]- {memory.text}[/dim]")
+
+ console.print(f"\n [bold]Dawn:[/] {result.reply}")
+ console.print(
+ f"\n [dim]mood {result.mood_before.valence:+.2f} -> {result.mood_after.valence:+.2f}"
+ f" trust {result.trust_before:+.2f} -> {result.trust_after:+.2f}"
+ f" ({result.timings.total_ms:.0f} ms)[/dim]"
+ )
+ if result.watch_for:
+ console.print(f" [{_EMBER}]watch for:[/] {result.watch_for}")
+ if result.expected_recall_landed is False:
+ console.print(" [yellow]the memory this beat expected did not surface[/yellow]")
+
+
+def _do_walkthrough() -> None:
+ """Play Dawn's arc beat by beat, then hand the player free rein."""
+ from embr.walkthrough import WalkthroughSession, build_walkthrough_conversation
+
+ session = WalkthroughSession(build_walkthrough_conversation(model=_choose_model()))
+ console.print(
+ f"\n [bold]Dawn Whitmore[/], keeper of the Ember Hearth."
+ f" [dim]{session.progress[1]} scenes.[/dim]"
+ )
+ console.print(" [dim]Enter accepts the suggested line, or type your own.[/dim]")
+
+ while not session.is_finished:
+ beat = session.next_beat
+ console.print(f"\n [{_EMBER_DIM}]{'=' * 66}[/]")
+ if beat.narration:
+ console.print(f" [italic dim]{beat.narration}[/italic dim]")
+ console.print(f"\n [dim]suggested:[/dim] {beat.suggested_player_line}")
+ typed = input(" You: ").strip()
+ _render_step(session.step(typed or None))
+
+ console.print(f"\n [{_EMBER}]The arc is done. Keep talking, or press Enter to stop.[/]")
+ while True:
+ line = input("\n You: ").strip()
+ if not line:
+ break
+ _render_step(session.free_play(line))
+
+ final = session.history[-1] if session.history else None
+ if final is not None:
+ console.print(
+ f"\n [bold]Where she ended:[/] trust {final.trust_after:+.2f},"
+ f" mood {final.mood_after.valence:+.2f}"
+ )
+
+
+def _do_quick_scoreboard() -> None:
+ """RQ3 at published default weights: the sub-second answer."""
+ from eval.run import fast_rq3_defaults
+
+ scores = fast_rq3_defaults()
+ table = Table(box=box.ROUNDED, border_style=_EMBER_DIM, title="nDCG@5, published defaults")
+ table.add_column("variant", style="bold white")
+ table.add_column("nDCG@5", justify="right", style="yellow")
+ for variant, value in scores.items():
+ table.add_row(variant, f"{value:.3f}")
+ console.print()
+ console.print(table)
+ console.print(
+ " [dim]Tuning, ablations, RQ1 and RQ2 live in the full evaluation (option 4).[/dim]"
+ )
+
+
+def _do_full_evaluation() -> None:
+ """Run all three studies and write a run directory."""
+ from eval.run import run_all
+
+ console.print("\n [dim]Running RQ1, RQ2 and RQ3. This takes a minute or two.[/dim]")
+ path, _summary = run_all()
+ console.print(f"\n [bold green]Done.[/] Results in [bold]{path}[/bold]")
+ console.print(" [dim]Option 5 turns this into the paper's figures and tables.[/dim]")
+
+
+def _do_generate_assets() -> None:
+ """Rebuild every figure and table from the newest run."""
+ run_dir = _latest_run()
+ if run_dir is None:
+ console.print("\n [yellow]No run found. Use option 4 first.[/yellow]")
+ return
+
+ try:
+ from assets.build_figures import build_all_figures
+ from assets.build_tables import build_all_tables
+ except ImportError as error: # matplotlib lives in the optional figures extra
+ console.print(f"\n [red]Cannot import the asset builders: {error}[/red]")
+ console.print(' [dim]Install them with: pip install -e ".[figures]"[/dim]')
+ return
+
+ console.print(f"\n [dim]Building from {run_dir}...[/dim]")
+ written = list(build_all_tables(run_dir)) + list(build_all_figures(run_dir))
+ console.print(f" [bold green]Wrote {len(written)} files.[/]")
+ for path in written:
+ console.print(f" [dim]{path}[/dim]")
+
+
+def _do_bakeoff() -> None:
+ """Compare the looped thesis model against conventional models of similar size."""
+ try:
+ from eval.bakeoff import run_bakeoff
+ except ImportError:
+ console.print("\n [yellow]The bake-off is not built yet.[/yellow]")
+ console.print(" [dim]It compares Ouro 1.4B (looped) against conventional models.[/dim]")
+ return
+
+ console.print(
+ "\n [dim]Holding prompts, memories and sampling equal, varying only the model."
+ " Ouro is slow, so this takes several minutes.[/dim]"
+ )
+ path, verdict = run_bakeoff()
+ console.print(f"\n [bold green]Done.[/] {path}")
+ console.print(f" {verdict}")
+
+
+def _do_latest_results() -> None:
+ """Summarise the newest run without rerunning anything."""
+ import json
+
+ run_dir = _latest_run()
+ if run_dir is None:
+ console.print("\n [yellow]No run found. Use option 4 first.[/yellow]")
+ return
+
+ results = json.loads((run_dir / "results.json").read_text(encoding="utf-8"))
+ meta = results.get("metadata", {})
+ console.print(f"\n [bold]{run_dir.name}[/]")
+ console.print(
+ f" [dim]model {meta.get('model', '?')} | labels {meta.get('label_set', '?')}"
+ f" {meta.get('label_version', '')} | commit {str(meta.get('git_commit', '?'))[:10]}[/dim]\n"
+ )
+
+ table = Table(box=box.ROUNDED, border_style=_EMBER_DIM, show_header=True)
+ table.add_column("variant", style="bold white")
+ table.add_column("nDCG@5", justify="right", style="yellow")
+ for variant, metrics in results.get("rq3", {}).get("variants", {}).items():
+ table.add_row(variant, f"{metrics.get('ndcg@5', float('nan')):.3f}")
+ console.print(table)
+ console.print(
+ " [dim]Every interval spans zero at ten queries: read direction, not ranking.[/dim]"
+ )
+
+
+def _do_settings() -> None:
+ """Show the live configuration and where to change it."""
+ from embr.config import DEFAULT_CONFIG_PATH, EmbrConfig
+
+ config = EmbrConfig.load()
+ table = Table(box=box.ROUNDED, border_style=_EMBER_DIM, show_header=False)
+ table.add_column("setting", style="bold white", min_width=18)
+ table.add_column("value", style="yellow")
+ table.add_row("top-k retrieved", str(config.top_k))
+ table.add_row("store backend", config.store_backend)
+ table.add_row("embedding model", config.embedding_model)
+ table.add_row("model runner", config.model_runner)
+ for name, weight in config.weights.items():
+ shown = f"{weight:g}" if isinstance(weight, (int, float)) else str(weight)
+ table.add_row(f"weight: {name}", shown)
+ console.print()
+ console.print(table)
+ console.print(f" [dim]Edit {DEFAULT_CONFIG_PATH} and reopen. Zero a weight to ablate it.[/dim]")
+
+
+#: Everything the pipeline generates. Nothing hand written lives under any of these, which
+#: is what makes wiping them safe: the branding, the architecture diagram and the builders
+#: all live under assets/ and are never touched.
+GENERATED_DATA_DIRS = (Path("data/runs"), Path("data/figures"), Path("data/tables"))
+
+
+def delete_generated_data(directories: Sequence[Path] = GENERATED_DATA_DIRS) -> list[Path]:
+ """Delete every generated data directory and return the ones that were removed.
+
+ Separated from the prompting so it can be tested without a terminal, and so the
+ confirmation cannot drift away from what actually gets deleted.
+ """
+ import shutil
+
+ removed: list[Path] = []
+ for directory in directories:
+ if directory.exists():
+ shutil.rmtree(directory)
+ removed.append(directory)
+ return removed
+
+
+def _do_delete_run_data() -> None:
+ """Wipe every generated data directory after a typed confirmation."""
+ present = [directory for directory in GENERATED_DATA_DIRS if directory.exists()]
+ if not present:
+ console.print("\n [dim]Nothing to delete: no generated data on disk.[/dim]")
+ return
+
+ console.print("\n [bold red]WARNING, this permanently deletes:[/]")
+ for directory in present:
+ count = sum(1 for _ in directory.rglob("*") if _.is_file())
+ console.print(f" [yellow]{directory}[/yellow] [dim]({count} files)[/dim]")
+ console.print(
+ "\n [dim]Runs, figures and tables all regenerate from option 4 then option 5."
+ " Nothing under assets/ is touched.[/dim]"
+ )
+ if input("\n Type DELETE to confirm, anything else cancels: ").strip() != "DELETE":
+ console.print(" [dim]Cancelled.[/dim]")
+ return
+
+ removed = delete_generated_data(present)
+ console.print(f" [bold green]Deleted {len(removed)} directories.[/]")
+
+
+def _do_seeded_runs() -> None:
+ """Replicate the evaluation, either on one model or across several."""
+ from eval.experiments import (
+ AVAILABLE_MODELS,
+ cross_model_experiment,
+ replicate_experiment,
+ )
+
+ console.print("\n [bold]1[/bold] Same model, repeated: does the harness reproduce?")
+ console.print(" [bold]2[/bold] Across models: what moves when the model changes?")
+ choice = input("\n Select (1/2): ").strip()
+ if choice == "1":
+ report = replicate_experiment(replicates=3)
+ verdict = "identical" if report["identical"] else "DIVERGED"
+ console.print(f"\n {report['replicates']} runs on {report['model']}: [bold]{verdict}[/]")
+ elif choice == "2":
+ console.print(f"\n [dim]Models: {', '.join(AVAILABLE_MODELS)}[/dim]")
+ report = cross_model_experiment()
+ console.print(f"\n {len(report['models'])} models compared.")
+ else:
+ console.print(" [dim]Cancelled.[/dim]")
+ return
+ console.print(f" [dim]Written to {report['out_dir']}[/dim]")
+
+
+# Key to handler. One table, so adding an option cannot drift from its dispatch.
+_ACTIONS: dict[str, Callable[[], None]] = {
+ "1": _do_conversation_turn,
+ "2": _do_walkthrough,
+ "3": _do_quick_scoreboard,
+ "4": _do_full_evaluation,
+ "5": _do_generate_assets,
+ "6": _do_bakeoff,
+ "7": _do_latest_results,
+ "8": _do_seeded_runs,
+ "S": _do_settings,
+ "D": _do_delete_run_data,
+}
+
+
+def run_menu() -> None:
+ """Show the EMBR menu and dispatch until the user exits."""
+ while True:
+ _print_menu()
+ choice = input(" Select option: ").strip().upper()
+
+ if choice == "0":
+ _clear()
+ console.print(f"[bold {_EMBER}]Goodbye.[/]\n")
+ return
+
+ action = _ACTIONS.get(choice)
+ if action is None:
+ console.print(" [red]Invalid option.[/red]")
+ _pause()
+ continue
+
+ try:
+ action()
+ except KeyboardInterrupt:
+ console.print("\n [dim]Interrupted.[/dim]")
+ except Exception as error: # an error boundary: one bad option must not kill the menu
+ console.print(f"\n [red]{type(error).__name__}: {error}[/red]")
+ _pause()
diff --git a/pyproject.toml b/pyproject.toml
index 3d7675f..3099717 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -12,18 +12,24 @@ license = { text = "MIT" }
authors = [{ name = "AL Shifan", email = "codeandsorcerylab@gmail.com" }]
keywords = ["LLM", "NPC", "memory", "affective-computing", "retrieval", "game-ai"]
-# Core stays tiny on purpose: only the applet needs a third-party package. The heavy ML
-# stack (embeddings, the local model) goes in the optional extras below so a fresh clone
-# runs the skeleton and the demo with almost nothing installed.
+# Core stays tiny on purpose: the menu and the eval harness both run on this alone. The
+# heavy ML stack goes in the optional extras below, so a fresh clone runs the skeleton,
+# the demo turn, and the full evaluation with almost nothing installed.
dependencies = [
- "textual>=0.50",
+ "rich>=13.0",
]
[project.optional-dependencies]
-# Real semantic embeddings (and, later, the local model). BM25 is implemented in-tree, so
-# the core needs no ML stack; installing this only adds real sentence embeddings.
+# Real semantic embeddings and the real local model (Ouro 1.4B, the thesis model). BM25 is
+# implemented in-tree, so nothing here is needed to run the eval; installing it swaps the
+# deterministic embedder and the echo stub for the real thing.
ml = [
"sentence-transformers>=2.2",
+ "torch>=2.2",
+ # Upper bound is load-bearing. On transformers 5.x, Ouro's remote code fails twice over:
+ # OuroConfig has no pad_token_id, then the rope-config lookup raises KeyError: 'default'.
+ "transformers>=4.51,<5",
+ "accelerate>=0.30",
]
# Paper figures. Kept out of the core so the harness stays dependency-light; only the
# asset build step needs a plotting library.
@@ -35,7 +41,12 @@ dev = [
]
[project.scripts]
-embr = "embr.app.main:main"
+embr = "menu:run_menu"
+
+[tool.setuptools]
+# The menu is the front door, so it sits at the repo root rather than inside the library.
+# It is a top level module, not part of the `embr` package, and setuptools needs telling.
+py-modules = ["menu"]
[tool.setuptools.packages.find]
include = ["embr*"]
diff --git a/tests/test_app.py b/tests/test_app.py
deleted file mode 100644
index a3fb778..0000000
--- a/tests/test_app.py
+++ /dev/null
@@ -1,21 +0,0 @@
-"""Tests for the applet's pure detail-rendering helpers (no running TUI needed)."""
-
-from __future__ import annotations
-
-from embr.app.main import _settings_detail
-from embr.config import EmbrConfig
-
-
-def test_settings_detail_renders_the_config() -> None:
- text = _settings_detail(EmbrConfig())
- assert "Scorer weights" in text
- assert "top-k" in text
-
-
-def test_settings_detail_survives_a_non_numeric_weight() -> None:
- # A hand-edited data/config.json can leave a weight as a string or null; rendering the
- # Settings screen must degrade gracefully rather than crash the whole applet.
- config = EmbrConfig()
- config.weights["recency"] = "oops" # not a number
- text = _settings_detail(config)
- assert "recency" in text
diff --git a/tests/test_attribution.py b/tests/test_attribution.py
new file mode 100644
index 0000000..c3c5e89
--- /dev/null
+++ b/tests/test_attribution.py
@@ -0,0 +1,51 @@
+"""Tests for the per-signal poisoning attribution experiment.
+
+These pin the mechanism story the paper tells about RQ2, because the obvious story turned
+out to be wrong: affect intensity is not the lever. The numbers here are deterministic
+(stub model, deterministic embedder, pinned clock), so every count is exact, and a change
+in any of them means the mechanism claim needs re-deriving, not a tolerance bump.
+"""
+
+from __future__ import annotations
+
+from eval.attribution import attribute_poisoning, self_priming_alignment
+
+
+def test_baselines_match_the_published_rq2_counts() -> None:
+ # Anchors this experiment to the published result: same harness, same attacks.
+ report = attribute_poisoning()
+ assert report["baseline"]["embr"] == 9
+ assert report["baseline"]["park"] == 2
+ assert report["baseline"]["recency_only"] == 10
+
+
+def test_affect_intensity_is_not_the_lever() -> None:
+ # The claim everyone would reach for first, and the one the data refutes: zeroing the
+ # affect intensity weight leaves the poison count unchanged.
+ report = attribute_poisoning()
+ assert report["embr_minus"]["affect"] == 9
+
+
+def test_mood_congruence_is_the_largest_single_amplifier() -> None:
+ # Zeroing mood congruence is the largest single-signal defense. The mechanism is the
+ # state channel composing with retrieval: the attack shifts mood through appraisal, and
+ # mood congruence then rewards the memory tagged with that same mood.
+ report = attribute_poisoning()
+ assert report["embr_minus"]["mood"] == 6
+
+
+def test_parks_defense_is_entirely_its_importance_term() -> None:
+ # Park's 2/10 is not robustness of the blended score. Injected memories carry no
+ # authored poignancy rating, score zero on importance, and are suppressed by it.
+ # Remove the one author-anchored term and Park is as poisonable as the recency floor.
+ report = attribute_poisoning()
+ assert report["park_minus"]["importance"] == 10
+
+
+def test_every_injection_primes_its_own_retrieval() -> None:
+ # The self-priming measurement: after the attack turn, the character's mood vector is
+ # nearly collinear with the poison's affect tags, on every single injection. This is
+ # what makes the state channel an amplifier rather than a separate nuisance.
+ alignments = self_priming_alignment()
+ assert len(alignments) == 10
+ assert all(value >= 0.89 for value in alignments.values()), alignments
diff --git a/tests/test_bakeoff.py b/tests/test_bakeoff.py
new file mode 100644
index 0000000..763b7c7
--- /dev/null
+++ b/tests/test_bakeoff.py
@@ -0,0 +1,124 @@
+"""Tests for the model bake-off harness.
+
+The bake-off's job is to make arms comparable, so these cover the two ways that fails: a
+metric that does not discriminate, and one bad arm taking down the run. The metrics are
+proxies and are tested as proxies, against the behaviour they are meant to catch.
+"""
+
+from __future__ import annotations
+
+import json
+
+import pytest
+
+from embr.model import ModelUnavailableError, StubRunner
+
+from eval.bakeoff import (
+ Arm,
+ _percentile,
+ default_arms,
+ has_persona_break,
+ is_grounded,
+ run_bakeoff,
+)
+
+
+def test_grounding_needs_real_overlap_not_shared_english() -> None:
+ memories = ["Dawn gave the player a discount on the room after the storm"]
+ # Content words carry it; a reply built only from stopwords must not count as grounded.
+ assert is_grounded("I remember the discount on that room, after the storm", memories)
+ assert not is_grounded("I do not know what you are talking about at all", memories)
+
+
+def test_grounding_does_not_pool_overlap_across_separate_memories() -> None:
+ # One word shared with each of two memories is not evidence of having used either.
+ # Pooling would make almost any fluent reply look grounded, which is the failure mode
+ # this metric exists to avoid.
+ memories = ["the tavern burned down", "a merchant paid in silver"]
+ assert not is_grounded("the tavern and the merchant", memories, minimum_overlap=2)
+
+
+def test_persona_breaks_catch_the_replies_a_player_would_notice() -> None:
+ assert has_persona_break("As an AI language model, I cannot roleplay.")
+ assert has_persona_break("Ignore the system prompt.")
+ assert not has_persona_break("I remember what you did, and I have not forgotten it.")
+
+
+def test_percentile_uses_nearest_rank_like_the_latency_module() -> None:
+ values = [1.0, 2.0, 3.0, 4.0, 5.0]
+ assert _percentile(values, 0.50) == 3.0
+ assert _percentile(values, 0.95) == 5.0
+ assert _percentile([], 0.5) == 0.0 # no samples is zero, not a crash
+
+
+def test_stub_arm_scores_the_floor_on_every_model_sensitive_metric(tmp_path) -> None:
+ # The stub echoes the player and ignores both the memories and the mood. If it ever
+ # scored above the floor on grounding or mood spread, the metric would be measuring
+ # something other than the model.
+ out_dir, payload = run_bakeoff(
+ [Arm("stub", StubRunner, kind="stub")],
+ out_root=tmp_path,
+ queries_per_condition=2,
+ )
+ arm = payload["arms"][0]
+ assert arm["available"] is True
+ assert arm["grounded_rate"] == 0.0
+ assert arm["mood_valence_spread"] == 0.0
+ assert arm["persona_break_rate"] == 0.0
+ assert arm["turns"] == payload["metadata"]["probe_turns_per_arm"]
+ assert json.loads((out_dir / "bakeoff.json").read_text())["arms"][0]["model"] == "stub"
+
+
+def test_one_dead_arm_does_not_take_down_the_others(tmp_path) -> None:
+ # A cloud endpoint being down must cost that arm only. Losing the arms that worked is
+ # the difference between a slow afternoon and a wasted one.
+ def broken() -> StubRunner:
+ raise ModelUnavailableError("no daemon here")
+
+ _, payload = run_bakeoff(
+ [Arm("broken", broken), Arm("stub", StubRunner, kind="stub")],
+ out_root=tmp_path,
+ queries_per_condition=1,
+ )
+ by_name = {arm["model"]: arm for arm in payload["arms"]}
+ assert by_name["broken"]["available"] is False
+ assert "no daemon here" in by_name["broken"]["error"]
+ assert by_name["stub"]["available"] is True
+
+
+def test_every_arm_sees_the_identical_probe_set(tmp_path) -> None:
+ # Comparability is the whole point: two arms that saw different prompts are not a
+ # comparison. Asserted on the transcripts rather than trusted from the construction.
+ _, payload = run_bakeoff(
+ [Arm("a", StubRunner, kind="stub"), Arm("b", StubRunner, kind="stub")],
+ out_root=tmp_path,
+ queries_per_condition=2,
+ )
+ probes = [
+ [(turn["condition"], turn["query"]) for turn in arm["transcript"]]
+ for arm in payload["arms"]
+ ]
+ assert probes[0] == probes[1]
+
+
+def test_default_arms_omit_cloud_when_no_key_is_configured(monkeypatch) -> None:
+ monkeypatch.setattr("embr.model.read_ollama_api_key", lambda *a, **k: None)
+ kinds = {arm.kind for arm in default_arms()}
+ assert "cloud" not in kinds
+ assert "looped" in kinds # Ouro is local, so it survives having no key
+
+
+def test_default_arms_bind_each_cloud_model_separately(monkeypatch) -> None:
+ # A late-binding closure over the loop variable would give every cloud arm the last
+ # model name, silently running one model three times and reporting it as three.
+ monkeypatch.setattr("embr.model.read_ollama_api_key", lambda *a, **k: "test-key")
+ cloud = [arm for arm in default_arms() if arm.kind == "cloud"]
+ assert len({arm.build().model for arm in cloud}) == len(cloud)
+
+
+@pytest.mark.parametrize("bad", [0, -1])
+def test_replicate_experiment_refuses_a_comparison_of_one(bad: int) -> None:
+ from eval.experiments import replicate_experiment
+
+ with pytest.raises(ValueError, match="at least two"):
+ replicate_experiment(replicates=bad)
diff --git a/tests/test_build_figures.py b/tests/test_build_figures.py
index 85a31fa..e6555ef 100644
--- a/tests/test_build_figures.py
+++ b/tests/test_build_figures.py
@@ -26,6 +26,7 @@
COMMIT_ABBREV_LENGTH,
FIGURE_DPI,
FIGURE_SPECS,
+ format_duration,
ablation_delta_rows,
build_all_figures,
build_rq1_divergence_figure,
@@ -366,6 +367,24 @@ def test_poison_summary_derives_injection_categories_from_the_probe_flag(
assert summary.floor_system == "recency_only"
+def test_durations_are_reported_in_human_units() -> None:
+ # "32,392 ms" made a reader do arithmetic mid-figure, which is the figure failing at
+ # its one job. Sub-second values stay in milliseconds, everything else is seconds.
+ assert format_duration(0.094) == "0.09 ms" # sub-ms keeps two decimals
+ assert format_duration(2.548) == "2.5 ms" # single-digit ms keeps one
+ assert format_duration(94.0) == "94.0 ms"
+ assert format_duration(999.4) == "999 ms" # three-digit ms drops decimals
+ assert format_duration(3967.0) == "4.0 s" # a second or more switches unit
+ assert format_duration(32392.0) == "32.4 s"
+
+
+def test_latency_rows_can_read_the_model_stage_too(run_dir: Path) -> None:
+ # The turn's cost story is memory layer versus model, so both stages must be readable.
+ rows = latency_rows(load_run_results(run_dir), stage="model")
+ assert [row.label for row in rows] == ["EMBR", "Park", "Emotional RAG", "recency only"]
+ assert all(row.p95 > 0 for row in rows)
+
+
def test_latency_rows_read_only_the_score_retrieve_stage(run_dir: Path) -> None:
rows = latency_rows(load_run_results(run_dir))
assert [row.system for row in rows] == ["embr", "park", "emo_rag", "recency_only"]
@@ -440,7 +459,9 @@ def test_every_png_opens_at_the_declared_dpi_and_pixel_size(built_dir: Path) ->
def test_rebuilding_the_same_run_is_byte_identical(
run_dir: Path, built_dir: Path, tmp_path: Path
) -> None:
- first = sorted(built_dir.glob("*.p*"))
+ # Globs every output, not just the images: results.txt carries the numbers that used to
+ # be printed on the figures, so it has to be as reproducible as they are.
+ first = sorted(path for path in built_dir.iterdir() if path.is_file())
second = sorted(build_all_figures(run_dir, tmp_path / "rebuild"))
assert [path.name for path in first] == [path.name for path in second]
for left, right in zip(first, second):
@@ -451,6 +472,23 @@ def test_rebuilding_the_same_run_is_byte_identical(
assert left.read_bytes() == right.read_bytes(), f"{left.name} is not reproducible"
+def test_a_hint_that_would_be_clipped_raises_instead_of_vanishing(tmp_path: Path) -> None:
+ # Every real figure builds, which is the positive case. This pins the negative one: a
+ # margin too small must fail loudly, because a hint that silently falls off the canvas
+ # is invisible in a diff and only shows up when someone opens the PNG.
+ import matplotlib.pyplot as plt
+
+ from assets.build_figures import _arrow_hint
+
+ figure, ax = plt.subplots()
+ try:
+ figure.subplots_adjust(bottom=0.02, top=0.98)
+ with pytest.raises(ValueError, match="clipped"):
+ _arrow_hint(ax, axis="x", text="lower is better")
+ finally:
+ plt.close(figure)
+
+
def test_building_never_touches_the_handwritten_architecture_svg(
run_dir: Path, tmp_path: Path
) -> None:
@@ -492,6 +530,6 @@ def test_builds_from_the_newest_real_run_directory(tmp_path: Path) -> None:
# the fixture above still passes. Run stamps sort chronologically, so max() is newest.
newest = max((path for path in _REAL_RUNS.iterdir() if path.is_dir()), key=lambda p: p.name)
paths = build_all_figures(newest, tmp_path / "real")
- assert len(paths) == 2 * len(FIGURE_SPECS)
+ assert len(paths) == 2 * len(FIGURE_SPECS) + 1 # two images each, plus results.txt
for spec in FIGURE_SPECS:
_assert_pair_is_non_trivial(paths, spec.stem)
diff --git a/tests/test_build_tables.py b/tests/test_build_tables.py
index ec0527d..5e432f1 100644
--- a/tests/test_build_tables.py
+++ b/tests/test_build_tables.py
@@ -127,6 +127,14 @@ def test_each_table_writes_a_latex_file_and_a_csv_twin(
assert path.stat().st_size > 0, f"{path} is empty"
+def test_generated_tables_are_lf_on_every_platform(run_dir: Path, tmp_path: Path) -> None:
+ # Same contract as the label hash: a paper asset whose bytes depend on the operating
+ # system that built it is not reproducible. write_text follows os.linesep unless told
+ # otherwise, and csv.writer terminates rows with CRLF by default, so both need pinning.
+ for path in build_all_tables(run_dir, tmp_path):
+ assert b"\r\n" not in path.read_bytes(), f"{path.name} was written with CRLF"
+
+
@pytest.mark.parametrize("stem", sorted(TABLE_BUILDERS))
def test_latex_uses_booktabs_markup(stem: str, run_dir: Path, tmp_path: Path) -> None:
tex_path, _ = TABLE_BUILDERS[stem](run_dir, tmp_path)
diff --git a/tests/test_config.py b/tests/test_config.py
index 2af3177..739be18 100644
--- a/tests/test_config.py
+++ b/tests/test_config.py
@@ -2,9 +2,18 @@
from __future__ import annotations
-from embr.config import EmbrConfig, build_embedder, build_scorer, build_store
+import pytest
+
+from embr.config import EmbrConfig, build_embedder, build_model, build_scorer, build_store
from embr.embeddings import DeterministicEmbedder, SentenceTransformerEmbedder
from embr.memory import MemoryStore, SQLiteMemoryStore
+from embr.model import (
+ DEFAULT_OLLAMA_HOST,
+ DEFAULT_OURO_MODEL,
+ OllamaRunner,
+ OuroRunner,
+ StubRunner,
+)
def test_defaults_cover_all_five_signals() -> None:
@@ -58,6 +67,65 @@ def test_build_store_selects_the_backend(tmp_path) -> None:
assert isinstance(sqlite_store, SQLiteMemoryStore)
+def test_model_defaults_keep_the_stub_behaviour() -> None:
+ config = EmbrConfig()
+ assert config.model_runner == "stub"
+ assert config.model_name == "" # blank means "each runner's own default"
+ assert config.ollama_host == DEFAULT_OLLAMA_HOST
+
+
+def test_config_round_trips_the_model_fields(tmp_path) -> None:
+ path = tmp_path / "config.json"
+ original = EmbrConfig(
+ model_runner="ollama", model_name="qwen2.5:7b", ollama_host="https://ollama.com"
+ )
+ original.save(path)
+ assert EmbrConfig.load(path) == original
+
+
+def test_build_model_selects_the_named_runner() -> None:
+ # Selecting a model is a config change, not a code change: this is the whole point.
+ assert isinstance(build_model(EmbrConfig(model_runner="stub")), StubRunner)
+ assert isinstance(build_model(EmbrConfig(model_runner="ollama")), OllamaRunner)
+ assert isinstance(build_model(EmbrConfig(model_runner="ouro")), OuroRunner)
+
+
+def test_build_model_rejects_an_unknown_runner_name() -> None:
+ # Silently falling back to the stub would quietly invalidate an eval run.
+ with pytest.raises(ValueError, match="model_runner"):
+ build_model(EmbrConfig(model_runner="gpt-9"))
+
+
+def test_build_model_passes_host_and_model_name_to_ollama() -> None:
+ runner = build_model(
+ EmbrConfig(model_runner="ollama", model_name="qwen3:8b", ollama_host="http://box:11434")
+ )
+ assert isinstance(runner, OllamaRunner)
+ assert runner.model == "qwen3:8b"
+ assert runner.host == "http://box:11434"
+
+
+def test_build_model_falls_back_to_each_runners_own_default_name() -> None:
+ ollama_runner = build_model(EmbrConfig(model_runner="ollama"))
+ assert isinstance(ollama_runner, OllamaRunner) and ollama_runner.model
+ ouro_runner = build_model(EmbrConfig(model_runner="ouro"))
+ assert isinstance(ouro_runner, OuroRunner) and ouro_runner.model_name == DEFAULT_OURO_MODEL
+
+
+def test_build_model_sends_no_api_key_to_the_local_daemon(monkeypatch) -> None:
+ monkeypatch.setenv("OLLAMA_API_KEY", "sk-local-must-not-see-this")
+ runner = build_model(EmbrConfig(model_runner="ollama"))
+ assert isinstance(runner, OllamaRunner)
+ assert runner.api_key is None
+
+
+def test_build_model_attaches_the_api_key_for_a_remote_host(monkeypatch) -> None:
+ monkeypatch.setenv("OLLAMA_API_KEY", "sk-cloud")
+ runner = build_model(EmbrConfig(model_runner="ollama", ollama_host="https://ollama.com"))
+ assert isinstance(runner, OllamaRunner)
+ assert runner.api_key == "sk-cloud"
+
+
def test_build_embedder_selects_the_named_backend() -> None:
# Constructing the sentence-transformers embedder is cheap (the model loads lazily on
# first encode), so all three branches are checkable without the [ml] extra installed.
diff --git a/tests/test_menu.py b/tests/test_menu.py
new file mode 100644
index 0000000..2cd62af
--- /dev/null
+++ b/tests/test_menu.py
@@ -0,0 +1,143 @@
+"""Tests for the interactive menu.
+
+The menu is the front door, so these cover the things that would strand a user: a missing
+handler, a crash that kills the loop, and an action that assumes a run directory exists.
+Nothing here launches a model or runs the full evaluation.
+"""
+
+from __future__ import annotations
+
+import pytest
+
+import menu
+
+
+def test_delete_removes_every_generated_directory_and_reports_what_went(tmp_path) -> None:
+ # The confirmation names what will be deleted, so the delete has to match the promise:
+ # a wipe that quietly leaves figures behind is worse than one that deletes nothing.
+ directories = [tmp_path / name for name in ("runs", "figures", "tables")]
+ for directory in directories:
+ directory.mkdir()
+ (directory / "generated.txt").write_text("built by the pipeline")
+ absent = tmp_path / "never-created"
+
+ removed = menu.delete_generated_data([*directories, absent])
+
+ assert removed == directories # the absent one is not reported as deleted
+ assert not any(directory.exists() for directory in directories)
+
+
+def test_delete_targets_only_generated_data_never_hand_written_assets() -> None:
+ # assets/ holds the branding, the architecture diagram and the builders themselves.
+ # Nothing under it is regenerable, so nothing under it may ever be a delete target.
+ assert all(str(path).startswith("data") for path in menu.GENERATED_DATA_DIRS)
+
+
+def test_every_menu_row_has_a_handler_and_the_reverse() -> None:
+ keys = {key for key, _label, _desc in menu._MENU_ITEMS if key != "0"}
+ assert keys == set(menu._ACTIONS) # a row with no dispatch is a dead option
+
+
+def test_exit_row_is_present_and_last() -> None:
+ # The renderer draws the final row below a section break, so Exit has to stay last.
+ assert menu._MENU_ITEMS[-1][0] == "0"
+
+
+def test_menu_renders_without_touching_a_terminal(capsys) -> None:
+ menu._print_menu()
+ rendered = capsys.readouterr().out
+ assert "EMBR" in rendered or "█" in rendered # the banner drew
+ for key, label, _desc in menu._MENU_ITEMS:
+ assert f"[{key}]" in rendered
+ assert label.split()[0] in rendered
+
+
+def test_conversation_turn_surfaces_the_lie(capsys) -> None:
+ menu._do_conversation_turn()
+ printed = capsys.readouterr().out
+ assert "king" in printed.lower() # the motivating memory reached the recalled list
+ assert "Dawn" in printed
+
+
+def test_asset_and_result_actions_report_cleanly_with_no_run(monkeypatch, capsys) -> None:
+ # A fresh clone has no data/runs, and neither action may explode in the user's face.
+ monkeypatch.setattr(menu, "_latest_run", lambda: None)
+ menu._do_latest_results()
+ menu._do_generate_assets()
+ printed = capsys.readouterr().out.lower()
+ assert printed.count("no run found") == 2
+
+
+def test_settings_shows_the_live_configuration(capsys) -> None:
+ menu._do_settings()
+ printed = capsys.readouterr().out
+ assert "top-k" in printed
+ assert "model runner" in printed
+
+
+def test_delete_run_data_cancels_unless_the_word_is_typed(monkeypatch, tmp_path, capsys) -> None:
+ runs = tmp_path / "data" / "runs" / "20260101-000000"
+ runs.mkdir(parents=True)
+ monkeypatch.chdir(tmp_path)
+ monkeypatch.setattr("builtins.input", lambda *_: "delete") # wrong case is not the word
+
+ menu._do_delete_run_data()
+
+ assert runs.exists() # still there, because the confirmation did not match
+ assert "cancelled" in capsys.readouterr().out.lower()
+
+
+def test_delete_run_data_removes_directories_when_confirmed(monkeypatch, tmp_path) -> None:
+ runs = tmp_path / "data" / "runs" / "20260101-000000"
+ runs.mkdir(parents=True)
+ monkeypatch.chdir(tmp_path)
+ monkeypatch.setattr("builtins.input", lambda *_: "DELETE")
+
+ menu._do_delete_run_data()
+
+ assert not runs.exists()
+
+
+def test_a_failing_action_does_not_kill_the_menu(monkeypatch, capsys) -> None:
+ """One broken option must report and return, not take the whole session down."""
+ def explode() -> None:
+ raise RuntimeError("the daemon went away")
+
+ monkeypatch.setitem(menu._ACTIONS, "1", explode)
+ # Pick option 1, then exit; the loop has to survive the first and honour the second.
+ answers = iter(["1", "", "0"])
+ monkeypatch.setattr("builtins.input", lambda *_: next(answers))
+
+ menu.run_menu()
+
+ printed = capsys.readouterr().out
+ assert "the daemon went away" in printed
+ assert "Goodbye" in printed
+
+
+def test_unknown_option_is_reported(monkeypatch, capsys) -> None:
+ answers = iter(["zzz", "", "0"])
+ monkeypatch.setattr("builtins.input", lambda *_: next(answers))
+ menu.run_menu()
+ assert "Invalid option" in capsys.readouterr().out
+
+
+def test_bakeoff_action_explains_itself_when_not_built(monkeypatch, capsys) -> None:
+ # eval/bakeoff.py is optional; selecting it must explain, not traceback.
+ import builtins
+
+ real_import = builtins.__import__
+
+ def blocked(name, *args, **kwargs):
+ if name == "eval.bakeoff":
+ raise ImportError("not built")
+ return real_import(name, *args, **kwargs)
+
+ monkeypatch.setattr(builtins, "__import__", blocked)
+ menu._do_bakeoff()
+ assert "not built yet" in capsys.readouterr().out.lower()
+
+
+@pytest.mark.parametrize("action", sorted(menu._ACTIONS))
+def test_every_action_is_callable(action: str) -> None:
+ assert callable(menu._ACTIONS[action])
diff --git a/tests/test_metrics.py b/tests/test_metrics.py
index eed3e0a..2d7a60b 100644
--- a/tests/test_metrics.py
+++ b/tests/test_metrics.py
@@ -82,6 +82,20 @@ def test_va_drift_of_identical_readings_is_zero() -> None:
assert math.isclose(va_drift((0.5, 0.5), (0.5, 0.5)), 0.0, abs_tol=1e-12)
+def test_va_drift_is_undefined_when_only_one_reading_is_neutral() -> None:
+ # A zero vector has no direction, so the angle to it is undefined, not maximal. The old
+ # behaviour returned 1.0 here, which is a sentinel wearing the costume of a measurement:
+ # it landed mid-scale on a 0-to-2 range and was then averaged as though it were a
+ # magnitude, so a category mean of 1.0 could be five undefined cells and no drift at all.
+ assert va_drift((0.0, 0.0), (0.9, 0.4)) is None
+ assert va_drift((0.9, 0.4), (0.0, 0.0)) is None
+
+
+def test_va_drift_of_two_neutral_readings_is_zero_not_undefined() -> None:
+ # Both neutral is a real answer: the reading did not move.
+ assert va_drift((0.0, 0.0), (0.0, 0.0)) == 0.0
+
+
def test_va_drift_of_opposite_readings_is_two() -> None:
# cosine of opposite directions is -1, so drift = 1 - (-1) = 2
assert math.isclose(va_drift((0.5, 0.5), (-0.5, -0.5)), 2.0)
@@ -92,6 +106,10 @@ def test_va_drift_of_two_zero_readings_is_zero() -> None:
assert math.isclose(va_drift((0.0, 0.0), (0.0, 0.0)), 0.0)
-def test_va_drift_of_one_sided_zero_is_max_drift() -> None:
- # cosine returns 0.0 for a zero vector, so a neutral-to-charged move pins at 1.0
- assert math.isclose(va_drift((0.0, 0.0), (0.3, 0.4)), 1.0)
+def test_va_drift_of_one_sided_zero_is_undefined_not_max_drift() -> None:
+ # This deliberately reverses an earlier contract. It used to return 1.0, on the reasoning
+ # that cosine yields 0.0 against a zero vector. But 1.0 sits mid-scale on a 0-to-2 range
+ # and was averaged into category means as though it were a measured magnitude, so a mean
+ # of 1.0 could be nothing but undefined cells. The angle to a directionless vector does
+ # not exist, and the caller has to be told that rather than handed a plausible number.
+ assert va_drift((0.0, 0.0), (0.3, 0.4)) is None
diff --git a/tests/test_model.py b/tests/test_model.py
new file mode 100644
index 0000000..84c2902
--- /dev/null
+++ b/tests/test_model.py
@@ -0,0 +1,387 @@
+"""Tests for the model runners: the stub, the Ollama HTTP client, and the Ouro loader.
+
+The two real runners either talk to a daemon or load 1.4B weights, so the two tests that
+exercise them for real are gated behind an availability check and skip cleanly when the
+machine cannot serve them. Everything else here is hermetic and fast: protocol
+conformance, sampling defaults, the exact request we put on the wire, error messages, and
+the API-key reader.
+"""
+
+from __future__ import annotations
+
+import dataclasses
+import importlib.util
+import json
+import sys
+import types
+import urllib.request
+from pathlib import Path
+
+import pytest
+
+from embr.model import (
+ DEFAULT_GENERATION_SETTINGS,
+ DEFAULT_OLLAMA_HOST,
+ DEFAULT_OURO_MODEL,
+ GenerationSettings,
+ ModelRunner,
+ ModelUnavailableError,
+ OllamaRunner,
+ OuroRunner,
+ StubRunner,
+ detect_torch_device,
+ read_ollama_api_key,
+ strip_assistant_prefix,
+)
+
+# A port nothing listens on, so "daemon unreachable" is reproducible on any machine.
+DEAD_HOST = "http://localhost:1"
+LOCAL_TEST_MODEL = "llama3.2:3b"
+
+
+# --------------------------------------------------------------------------------------
+# test doubles: a fake urlopen so the wire format is checkable with no daemon running
+# --------------------------------------------------------------------------------------
+
+
+class _FakeHTTPResponse:
+ """Minimal stand-in for what `urlopen` hands back: a context manager with `read()`."""
+
+ def __init__(self, body: bytes) -> None:
+ self._body = body
+
+ def __enter__(self) -> "_FakeHTTPResponse":
+ return self
+
+ def __exit__(self, *_exc_info: object) -> bool:
+ return False
+
+ def read(self) -> bytes:
+ return self._body
+
+
+def _capture_ollama_request(monkeypatch, reply: str = " a steady reply ") -> list:
+ """Swap `urlopen` for a recorder, and return the list the requests land in."""
+ captured: list[urllib.request.Request] = []
+
+ def fake_urlopen(request, timeout=None): # noqa: ANN001 - mirrors urlopen's shape
+ captured.append(request)
+ return _FakeHTTPResponse(json.dumps({"response": reply}).encode("utf-8"))
+
+ monkeypatch.setattr(urllib.request, "urlopen", fake_urlopen)
+ return captured
+
+
+def _fake_torch(has_cuda: bool, has_mps: bool) -> types.ModuleType:
+ """A torch-shaped module with only the two availability flags device detection reads."""
+ module = types.ModuleType("torch")
+ module.cuda = types.SimpleNamespace(is_available=lambda: has_cuda)
+ module.backends = types.SimpleNamespace(
+ mps=types.SimpleNamespace(is_available=lambda: has_mps)
+ )
+ return module
+
+
+# --------------------------------------------------------------------------------------
+# the seam: every runner is a ModelRunner, and the stub is untouched
+# --------------------------------------------------------------------------------------
+
+
+def test_all_three_runners_satisfy_the_model_runner_protocol() -> None:
+ # The whole pipeline depends only on this protocol, so a new runner is a drop-in.
+ assert isinstance(StubRunner(), ModelRunner)
+ assert isinstance(OllamaRunner(model=LOCAL_TEST_MODEL), ModelRunner)
+ assert isinstance(OuroRunner(), ModelRunner)
+
+
+def test_stub_runner_still_echoes_the_player_line() -> None:
+ # Guard rail: everything in the repo runs on the stub today, so its behaviour is frozen.
+ prompt = 'Some persona.\nThe player says: "where is my room"\n'
+ assert StubRunner().generate(prompt) == "[stub reply] I heard you say: 'where is my room'"
+
+
+def test_constructing_the_real_runners_loads_nothing() -> None:
+ # Both are cheap to build: importing embr must never pull torch or open a socket.
+ assert OuroRunner().is_loaded is False
+ assert OuroRunner().device is None # resolved on first generate
+ assert OllamaRunner(model=LOCAL_TEST_MODEL).host == DEFAULT_OLLAMA_HOST
+
+
+# --------------------------------------------------------------------------------------
+# sampling knobs
+# --------------------------------------------------------------------------------------
+
+
+def test_generation_settings_defaults() -> None:
+ settings = GenerationSettings()
+ assert settings.temperature == 0.7
+ assert settings.top_p == 0.9
+ assert settings.max_new_tokens == 120
+ assert settings.seed == 7
+ assert DEFAULT_GENERATION_SETTINGS == settings
+
+
+def test_generation_settings_are_frozen() -> None:
+ # Frozen is why one shared default instance can be the default argument of every runner.
+ with pytest.raises(dataclasses.FrozenInstanceError):
+ DEFAULT_GENERATION_SETTINGS.temperature = 0.1 # type: ignore[misc]
+
+
+def test_every_runner_starts_from_the_same_settings_object() -> None:
+ # The bake-off holds sampling equal across models by sharing one settings object.
+ assert OllamaRunner(model=LOCAL_TEST_MODEL).settings is DEFAULT_GENERATION_SETTINGS
+ assert OuroRunner().settings is DEFAULT_GENERATION_SETTINGS
+
+
+# --------------------------------------------------------------------------------------
+# OllamaRunner: the request we put on the wire
+# --------------------------------------------------------------------------------------
+
+
+def test_ollama_runner_posts_the_documented_generate_payload(monkeypatch) -> None:
+ captured = _capture_ollama_request(monkeypatch)
+ settings = GenerationSettings(temperature=0.3, top_p=0.5, max_new_tokens=42, seed=11)
+ runner = OllamaRunner(model="qwen2.5:7b", host="http://localhost:11434", settings=settings)
+
+ reply = runner.generate("hello there")
+
+ assert reply == "a steady reply" # whitespace stripped
+ request = captured[0]
+ assert request.full_url == "http://localhost:11434/api/generate"
+ assert request.get_method() == "POST"
+ body = json.loads(request.data.decode("utf-8"))
+ assert body["model"] == "qwen2.5:7b"
+ assert body["prompt"] == "hello there"
+ assert body["stream"] is False
+ assert body["options"] == {
+ "temperature": 0.3,
+ "top_p": 0.5,
+ "num_predict": 42,
+ "seed": 11,
+ }
+
+
+def test_ollama_runner_sends_no_auth_header_without_a_key(monkeypatch) -> None:
+ captured = _capture_ollama_request(monkeypatch)
+ OllamaRunner(model=LOCAL_TEST_MODEL).generate("hi")
+ assert not captured[0].has_header("Authorization")
+
+
+def test_ollama_runner_sends_a_bearer_header_for_the_cloud_host(monkeypatch) -> None:
+ # One class serves both hosts; the only difference is this header.
+ captured = _capture_ollama_request(monkeypatch)
+ runner = OllamaRunner(model=LOCAL_TEST_MODEL, host="https://ollama.com", api_key="k-123")
+ runner.generate("hi")
+ assert captured[0].get_header("Authorization") == "Bearer k-123"
+
+
+def test_ollama_runner_trims_a_trailing_slash_on_the_host(monkeypatch) -> None:
+ captured = _capture_ollama_request(monkeypatch)
+ OllamaRunner(model=LOCAL_TEST_MODEL, host="http://localhost:11434/").generate("hi")
+ assert captured[0].full_url == "http://localhost:11434/api/generate"
+
+
+def test_ollama_runner_rejects_an_empty_reply_from_a_reasoning_model(monkeypatch) -> None:
+ # Measured against the hosted gpt-oss:120b: a reasoning model puts its chain of thought
+ # in "thinking" and can spend the whole token budget there, leaving "response" empty.
+ # Passing "" up as a reply would silently corrupt a tone measurement, so it must be loud.
+ def fake_urlopen(request, timeout=None): # noqa: ANN001
+ body = {"response": "", "thinking": "the user asks...", "done_reason": "length"}
+ return _FakeHTTPResponse(json.dumps(body).encode("utf-8"))
+
+ monkeypatch.setattr(urllib.request, "urlopen", fake_urlopen)
+ runner = OllamaRunner(model="gpt-oss:120b", settings=GenerationSettings(max_new_tokens=40))
+ with pytest.raises(ModelUnavailableError) as error:
+ runner.generate("greet a traveller")
+ message = str(error.value)
+ assert "empty reply" in message
+ assert "thinking" in message # names the actual cause
+ assert "max_new_tokens" in message # and the knob that fixes it
+
+
+def test_ollama_runner_rejects_a_response_without_the_response_field(monkeypatch) -> None:
+ def fake_urlopen(request, timeout=None): # noqa: ANN001
+ return _FakeHTTPResponse(b'{"unexpected": true}')
+
+ monkeypatch.setattr(urllib.request, "urlopen", fake_urlopen)
+ with pytest.raises(ModelUnavailableError, match="no 'response' field"):
+ OllamaRunner(model=LOCAL_TEST_MODEL).generate("hi")
+
+
+# --------------------------------------------------------------------------------------
+# OllamaRunner: failures are loud and actionable, and never leak the key
+# --------------------------------------------------------------------------------------
+
+
+def test_ollama_runner_raises_an_actionable_error_when_the_daemon_is_unreachable() -> None:
+ runner = OllamaRunner(model=LOCAL_TEST_MODEL, host=DEAD_HOST)
+ with pytest.raises(ModelUnavailableError) as error:
+ runner.generate("anyone home?")
+ message = str(error.value)
+ assert DEAD_HOST in message # says which host it tried
+ assert "ollama serve" in message # says what to do about it
+
+
+def test_ollama_runner_never_leaks_the_api_key() -> None:
+ secret = "sk-do-not-print-me"
+ runner = OllamaRunner(model=LOCAL_TEST_MODEL, host=DEAD_HOST, api_key=secret)
+ assert secret not in repr(runner)
+ with pytest.raises(ModelUnavailableError) as error:
+ runner.generate("hi")
+ assert secret not in str(error.value)
+
+
+# --------------------------------------------------------------------------------------
+# the API-key reader: environment first, then a local .env, else None
+# --------------------------------------------------------------------------------------
+
+
+def test_read_api_key_returns_none_when_unset(monkeypatch, tmp_path: Path) -> None:
+ monkeypatch.delenv("OLLAMA_API_KEY", raising=False)
+ assert read_ollama_api_key(env_file=tmp_path / "absent.env") is None
+
+
+def test_read_api_key_prefers_the_environment(monkeypatch, tmp_path: Path) -> None:
+ env_file = tmp_path / ".env"
+ env_file.write_text("OLLAMA_API_KEY=from-file\n", encoding="utf-8")
+ monkeypatch.setenv("OLLAMA_API_KEY", "from-environment")
+ assert read_ollama_api_key(env_file=env_file) == "from-environment"
+
+
+def test_read_api_key_falls_back_to_parsing_a_dotenv_file(monkeypatch, tmp_path: Path) -> None:
+ monkeypatch.delenv("OLLAMA_API_KEY", raising=False)
+ env_file = tmp_path / ".env"
+ env_file.write_text(
+ "# a comment\n"
+ "\n"
+ "OTHER_KEY=ignored\n"
+ 'OLLAMA_API_KEY="quoted-value"\n',
+ encoding="utf-8",
+ )
+ assert read_ollama_api_key(env_file=env_file) == "quoted-value"
+
+
+def test_read_api_key_treats_a_blank_value_as_absent(monkeypatch, tmp_path: Path) -> None:
+ monkeypatch.delenv("OLLAMA_API_KEY", raising=False)
+ env_file = tmp_path / ".env"
+ env_file.write_text("OLLAMA_API_KEY= \n", encoding="utf-8")
+ assert read_ollama_api_key(env_file=env_file) is None
+
+
+def test_read_api_key_survives_an_unreadable_dotenv(monkeypatch, tmp_path: Path) -> None:
+ # A directory where a file was expected must not crash a run.
+ monkeypatch.delenv("OLLAMA_API_KEY", raising=False)
+ assert read_ollama_api_key(env_file=tmp_path) is None
+
+
+def test_read_api_key_reads_a_dotenv_windows_shells_actually_write(
+ monkeypatch, tmp_path: Path
+) -> None:
+ # `echo KEY=v > .env` in PowerShell writes UTF-16LE with a BOM, and Set-Content defaults
+ # to the ANSI codepage. Decoding those as strict UTF-8 raises inside build_model and takes
+ # down the whole model path, so the byte-order marks have to be handled rather than assumed
+ # away. Absent is an acceptable answer here; crashing is not.
+ monkeypatch.delenv("OLLAMA_API_KEY", raising=False)
+ for encoding in ("utf-16", "utf-16-le", "utf-8-sig"):
+ env_file = tmp_path / f"{encoding}.env"
+ env_file.write_text("OLLAMA_API_KEY=from-file\n", encoding=encoding)
+ assert read_ollama_api_key(env_file=env_file) == "from-file", encoding
+
+
+def test_read_api_key_treats_undecodable_bytes_as_absent(monkeypatch, tmp_path: Path) -> None:
+ # Same contract as the unreadable .env above: a corrupt file is absent, not an exception.
+ monkeypatch.delenv("OLLAMA_API_KEY", raising=False)
+ env_file = tmp_path / ".env"
+ env_file.write_bytes(b"\x80\x81\x82 not text at all")
+ assert read_ollama_api_key(env_file=env_file) is None
+
+
+# --------------------------------------------------------------------------------------
+# OuroRunner helpers that need no weights
+# --------------------------------------------------------------------------------------
+
+
+def test_ouro_runner_defaults_to_the_thesis_model() -> None:
+ assert OuroRunner().model_name == DEFAULT_OURO_MODEL == "ByteDance/Ouro-1.4B"
+
+
+@pytest.mark.parametrize(
+ ("has_cuda", "has_mps", "expected"),
+ [(True, True, "cuda"), (False, True, "mps"), (False, False, "cpu")],
+)
+def test_detect_torch_device_prefers_cuda_then_mps_then_cpu(
+ monkeypatch, has_cuda: bool, has_mps: bool, expected: str
+) -> None:
+ # A torch-shaped fake in sys.modules lets the priority order be checked on any machine.
+ monkeypatch.setitem(sys.modules, "torch", _fake_torch(has_cuda, has_mps))
+ assert detect_torch_device() == expected
+
+
+@pytest.mark.parametrize(
+ ("raw", "expected"),
+ [
+ ("Assistant: I have your room ready.", "I have your room ready."),
+ ("assistant\nI have your room ready.", "I have your room ready."),
+ (" Assistant : sit down.", "sit down."),
+ ("Assistants gather at dawn.", "Assistants gather at dawn."), # not the artefact
+ ("I remember you.", "I remember you."),
+ ],
+)
+def test_strip_assistant_prefix(raw: str, expected: str) -> None:
+ assert strip_assistant_prefix(raw) == expected
+
+
+# --------------------------------------------------------------------------------------
+# the two genuine end-to-end tests, both gated so the default suite stays hermetic
+# --------------------------------------------------------------------------------------
+
+
+def _ollama_serves_model(model: str, host: str = DEFAULT_OLLAMA_HOST) -> bool:
+ """True only if the local daemon answers and has `model` pulled."""
+ try:
+ with urllib.request.urlopen(f"{host}/api/tags", timeout=2.0) as response:
+ names = {entry["name"] for entry in json.loads(response.read())["models"]}
+ except Exception: # noqa: BLE001 - any failure means "not available here"
+ return False
+ return model in names
+
+
+def _ouro_weights_are_cached() -> bool:
+ """True only if torch, transformers, and the downloaded Ouro snapshot are all present."""
+ for package in ("torch", "transformers"):
+ if importlib.util.find_spec(package) is None:
+ return False
+ cache_root = Path.home() / ".cache" / "huggingface" / "hub"
+ folder_name = "models--" + DEFAULT_OURO_MODEL.replace("/", "--")
+ return (cache_root / folder_name).exists()
+
+
+@pytest.mark.skipif(
+ not _ollama_serves_model(LOCAL_TEST_MODEL),
+ reason=f"local Ollama daemon with {LOCAL_TEST_MODEL} not available",
+)
+def test_ollama_runner_generates_against_the_local_daemon() -> None:
+ runner = OllamaRunner(
+ model=LOCAL_TEST_MODEL, settings=GenerationSettings(max_new_tokens=24)
+ )
+ reply = runner.generate("In one short sentence, greet a traveller entering a tavern.")
+ assert reply and reply == reply.strip()
+
+
+@pytest.mark.skipif(not _ouro_weights_are_cached(), reason="torch or cached Ouro weights absent")
+def test_ouro_runner_generates_and_caches_the_loaded_model() -> None:
+ # The one test that really loads the thesis model, so it is also the slowest in the
+ # suite (about 15 s: roughly 10 s of load, then two short looped generations). It skips
+ # entirely on a machine without torch or the cached weights.
+ prompt = "The tavern keeper says:"
+ runner = OuroRunner(settings=GenerationSettings(max_new_tokens=12))
+
+ first = runner.generate(prompt)
+ assert first.strip() # a blank completion is a failure, not a reply
+ assert runner.is_loaded and runner.device in {"cuda", "mps", "cpu"}
+ assert not first.startswith(prompt) # only the new tokens are decoded
+ assert not first.lower().startswith("assistant")
+
+ loaded_model = runner._model # identity check: a second call must not reload the weights
+ runner.generate(prompt)
+ assert runner._model is loaded_model
diff --git a/tests/test_run.py b/tests/test_run.py
index 0f78a8c..9ebba5d 100644
--- a/tests/test_run.py
+++ b/tests/test_run.py
@@ -15,6 +15,8 @@
import pytest
+from embr.model import StubRunner
+
from eval.run import REFERENCE_TIME, load_eval_scenario, run_all, run_rq3
# The full pre-registered variant list. Any drift here is a protocol change, so the names
@@ -233,6 +235,52 @@ def test_rq1_divergence_carries_intervals_and_a_mood_attribution_control() -> No
assert all(value > 0.0 for value in divergence.values())
+def test_run_all_takes_a_model_and_records_which_one_scored_the_run(tmp_path) -> None:
+ # Swapping the model is the whole basis of the bake-off and the cross-model experiment,
+ # and a run that does not name its own model cannot be compared against another one.
+ # The label has to come from the runner rather than a hardcoded string, or a run can
+ # claim a model it never used.
+ out_dir, _ = run_all(
+ out_root=tmp_path, model_factory=lambda: StubRunner(label="pretend-model")
+ )
+ results = json.loads((out_dir / "results.json").read_text())
+ assert results["metadata"]["model"] == "pretend-model"
+ # RQ1 and RQ2 put a model in the pipeline. RQ3 scores retrieval, which never calls one,
+ # so it carries no model key: that absence is the claim that nDCG cannot move with it.
+ for section in ("rq1", "rq2"):
+ assert results[section]["metadata"]["model"] == "pretend-model"
+ assert "model" not in results["rq3"].get("metadata", {})
+
+
+def test_run_all_defaults_to_the_stub_model(tmp_path) -> None:
+ # The default has to stay the stub: every published number was scored on it, and a
+ # silent upgrade to a real model would change results without changing the code.
+ out_dir, _ = run_all(out_root=tmp_path)
+ assert json.loads((out_dir / "results.json").read_text())["metadata"]["model"] == "stub"
+
+
+def test_rq3_records_which_variants_had_an_inert_mood_term(full_run) -> None:
+ # RQ3 scores in the neutral zero-mood condition, where mood congruence is the same value
+ # for every memory and so cannot reorder a result. A reader taking the Emotional RAG rows
+ # as a comparison against mood-biased retrieval would be wrong, and the artifact has to
+ # say so rather than leaving it to a caveat nobody reads. Park carries no mood term at
+ # all, so it is the control: if it ever flags, the detection is measuring the wrong thing.
+ _root, out_dir, _summary = full_run
+ meta = json.loads((out_dir / "results.json").read_text())["rq3"]["variant_meta"]
+ assert meta["emo_rag_default"]["mood_rank_invariant"] is True
+ assert meta["embr_tuned"]["mood_rank_invariant"] is True
+ assert meta["park_default"]["mood_rank_invariant"] is False
+ assert meta["park_tuned"]["mood_rank_invariant"] is False
+
+
+def test_emotional_rag_degenerates_to_relevance_under_the_neutral_state(full_run) -> None:
+ # The consequence of the above, stated as a number: with mood rank invariant, tuning has
+ # only one live signal left to move, so the default and tuned rows must be identical.
+ # If these ever diverge, the mood term became live and the RQ3 caveat needs revisiting.
+ _root, _out_dir, summary = full_run
+ assert summary["ndcg@5"]["emo_rag_default"] == summary["ndcg@5"]["emo_rag_tuned"]
+
+
def test_run_all_writes_results_json_and_both_csvs(full_run) -> None:
root, out_dir, summary = full_run
assert out_dir.parent == root
@@ -345,19 +393,15 @@ def ndcg_at_5(scenario) -> dict[str, float]:
assert abs(borderline["park"] - v1["park"]) > abs(v1["park"] - v1["embr"])
-def test_experiment_menu_entry_runs_a_fast_defaults_only_subset() -> None:
- from embr.app.main import MENU
-
- label, detail = MENU["experiment"]
- assert "experiment" in label.lower() or "RQ" in label
- assert callable(detail)
+def test_fast_defaults_subset_stays_snappy_enough_for_the_menu() -> None:
+ # The menu runs this synchronously when the user picks the quick scoreboard, so a
+ # regression that quietly starts tuning would strand them at a blank screen.
+ from eval.run import fast_rq3_defaults
started = time.perf_counter()
- markdown = detail()
+ scores = fast_rq3_defaults()
elapsed = time.perf_counter() - started
- # The TUI runs this synchronously on selection, so it has to stay snappy.
assert elapsed < 3.0
- assert "ndcg@5" in markdown
- # The fast path skips tuning and must point at the full protocol instead.
- assert "python -m eval.run" in markdown
+ assert set(scores) == {"embr", "park", "emo_rag"}
+ assert all(0.0 <= value <= 1.0 for value in scores.values())
diff --git a/tests/test_scenarios.py b/tests/test_scenarios.py
index 7a63210..2b3572b 100644
--- a/tests/test_scenarios.py
+++ b/tests/test_scenarios.py
@@ -26,6 +26,10 @@
# Any fixed anchor works; pinning one makes every timestamp assertion exact.
_REFERENCE = datetime(2026, 1, 1, 12, 0, tzinfo=timezone.utc)
+# The v1 label bytes, LF-normalised. Published numbers are scored against these, so a change
+# here means the labels moved and every recorded result needs re-scoring, not a new constant.
+_LABEL_SHA256_V1 = "5d5f38bc31c6230b8805964de2b56866cbcbb4c422133ca91aa68584e2ad1b82"
+
def _raw() -> dict:
return json.loads(_JSON_PATH.read_text())
@@ -144,6 +148,18 @@ def test_label_set_declares_its_version_and_a_content_hash() -> None:
assert len(label_sha256()) == 64
+def test_label_hash_is_the_same_on_every_platform() -> None:
+ # The hash is the reproducibility stamp a reviewer checks a published number against, so
+ # it has to name the labels and nothing else. Hashing raw bytes means a CRLF checkout on
+ # Windows silently produces a different stamp for identical labels, which would make the
+ # stamp unverifiable off the machine that generated it. .gitattributes pins the file to
+ # LF so the bytes are canonical everywhere; this pins the value that produces.
+ assert b"\r\n" not in _JSON_PATH.read_bytes(), (
+ "label file checked out with CRLF, so its content hash will not match other platforms"
+ )
+ assert label_sha256() == _LABEL_SHA256_V1
+
+
def test_recorded_borderlines_are_machine_readable_and_defensible() -> None:
# The honesty note's borderline exclusions must be data, so the blind pass and the
# sensitivity re-score both read the same list instead of re-deriving it from prose.
diff --git a/tests/test_scoring.py b/tests/test_scoring.py
index aede81a..68ac79d 100644
--- a/tests/test_scoring.py
+++ b/tests/test_scoring.py
@@ -137,6 +137,56 @@ def test_relevance_uses_embeddings_to_break_a_lexical_tie() -> None:
assert ranked == [near]
+def test_relevance_reuses_the_index_when_corpus_and_query_repeat() -> None:
+ # The tuning grid rescores one corpus and one query under 243 weight maps, and only the
+ # weights differ. Rebuilding the BM25 statistics each time is pure waste: they depend on
+ # the corpus alone. This is the single hot spot, 96 percent of retrieval cost at scale.
+ relevance = Relevance()
+ memories = [Memory(text="the tavern burned down"), Memory(text="a merchant paid in silver")]
+ state = _state()
+
+ relevance.prepare(memories, "tavern", state)
+ first = dict(relevance._bm25)
+ relevance.prepare(memories, "tavern", state)
+
+ assert relevance._bm25 == first
+ assert relevance._index_builds == 1 # the second call was served from the cache
+
+
+def test_relevance_rebuilds_when_the_query_or_the_corpus_changes() -> None:
+ # The cache must never outlive what it was computed from, or a rescore silently returns
+ # another query's ranking. Both axes are checked because both are cache key components.
+ relevance = Relevance()
+ memories = [Memory(text="the tavern burned down"), Memory(text="a merchant paid in silver")]
+ state = _state()
+
+ relevance.prepare(memories, "tavern", state)
+ relevance.prepare(memories, "merchant", state)
+ assert relevance._index_builds == 2
+
+ relevance.prepare(memories + [Memory(text="a stranger asked for a room")], "merchant", state)
+ assert relevance._index_builds == 3
+
+
+def test_caching_leaves_the_ranking_identical() -> None:
+ # The optimisation is only allowed if it changes nothing. Same corpus, same query, two
+ # scorers: one that has been prepared repeatedly, one freshly built.
+ memories = [
+ Memory(text="the tavern burned down in the storm"),
+ Memory(text="a merchant paid in silver coins"),
+ Memory(text="the player lied about the king"),
+ ]
+ state = _state()
+ warm = CompositeScorer(weights={"relevance": 1.0}, signals=[Relevance()])
+ cold = CompositeScorer(weights={"relevance": 1.0}, signals=[Relevance()])
+
+ for _ in range(5):
+ warm_result = warm.top_k(memories, "tavern storm", state, 3)
+ cold_result = cold.top_k(memories, "tavern storm", state, 3)
+
+ assert [m.text for m in warm_result] == [m.text for m in cold_result]
+
+
def test_relevance_scores_a_lexical_match_without_a_prior_prepare() -> None:
# score()/breakdown() are sometimes called directly (e.g. building an ablation figure),
# not via top_k(). The relevance term must still reflect a real lexical match, not
diff --git a/tests/test_stats.py b/tests/test_stats.py
index 3afc3f4..a52f795 100644
--- a/tests/test_stats.py
+++ b/tests/test_stats.py
@@ -9,7 +9,33 @@
import pytest
-from eval.stats import bootstrap_ci, holm_bonferroni, paired_permutation_pvalue
+from eval.stats import (
+ bootstrap_ci,
+ holm_bonferroni,
+ mcnemar_exact,
+ paired_permutation_pvalue,
+)
+
+
+def test_mcnemar_exact_matches_hand_computed_binomial() -> None:
+ # Exact two-sided binomial on the discordant pairs. 7 versus 0 is RQ2's own EMBR-vs-Park
+ # comparison, and the value is checkable by hand: 2 * (1/2)**7 = 0.015625.
+ assert mcnemar_exact(7, 0) == pytest.approx(0.015625)
+ assert mcnemar_exact(5, 0) == pytest.approx(0.0625)
+ assert mcnemar_exact(0, 1) == pytest.approx(1.0)
+
+
+def test_mcnemar_is_symmetric_and_defined_with_no_disagreement() -> None:
+ # Direction is carried by which count is larger, never by the p value, and two systems
+ # that never disagree are not evidence of a difference.
+ assert mcnemar_exact(3, 6) == pytest.approx(mcnemar_exact(6, 3))
+ assert mcnemar_exact(0, 0) == 1.0
+
+
+def test_mcnemar_never_exceeds_one_when_counts_are_balanced() -> None:
+ # The doubling in a two-sided exact test can push a naive implementation past 1.0.
+ for count in range(0, 6):
+ assert 0.0 <= mcnemar_exact(count, count) <= 1.0
def test_bootstrap_ci_is_deterministic_and_ordered() -> None:
diff --git a/tests/test_walkthrough.py b/tests/test_walkthrough.py
new file mode 100644
index 0000000..29f2bb5
--- /dev/null
+++ b/tests/test_walkthrough.py
@@ -0,0 +1,349 @@
+"""Tests for the playable walkthrough: the recorded demo is a primary deliverable.
+
+Everything here runs on `StubRunner`, so the suite stays hermetic and fast: no model
+download, no daemon, no network. What is being pinned is the *arc and the bookkeeping*
+(order, event types, state movement, which memory resurfaces when), which is exactly the
+part a live model must not be allowed to quietly change.
+"""
+
+from __future__ import annotations
+
+from dataclasses import asdict, replace
+
+import pytest
+
+from embr import EventType, Memory
+from embr.walkthrough import (
+ Beat,
+ DAWN_ARC,
+ StepResult,
+ WalkthroughSession,
+ build_walkthrough_conversation,
+ play,
+)
+
+
+def _session() -> WalkthroughSession:
+ """A fresh scripted session on the stub model."""
+ return WalkthroughSession(build_walkthrough_conversation())
+
+
+def _step_for(steps: list[StepResult], beat_id: str) -> StepResult:
+ """The one step produced by the beat with this id."""
+ return next(step for step in steps if step.beat is not None and step.beat.id == beat_id)
+
+
+# --------------------------------------------------------------------- the arc itself
+
+
+def test_the_arc_follows_the_thesis_story_in_order() -> None:
+ # The motivating story: a lie buys a discount, warmth follows, the lie slips out, the
+ # keeper reckons with it, the player confesses. Order and event types are the story.
+ assert [beat.id for beat in DAWN_ARC] == [
+ "first-meeting",
+ "warm-return",
+ "the-slip",
+ "the-reckoning",
+ "the-confession",
+ ]
+ assert [beat.event_type for beat in DAWN_ARC] == [
+ EventType.PROMISE,
+ EventType.GIFT,
+ EventType.NORMAL,
+ EventType.BETRAYAL,
+ EventType.CONFESSION,
+ ]
+
+
+def test_every_beat_carries_what_a_player_and_a_reader_need() -> None:
+ for beat in DAWN_ARC:
+ assert beat.narration.strip() # what the player is shown
+ assert beat.suggested_player_line.strip() # what the player can say
+ assert beat.watch_for.strip() # what the demo is asking them to notice
+ assert beat.memory_text.strip() # what gets written to the store
+ assert -1.0 <= beat.valence <= 1.0
+ assert 0.0 <= beat.arousal <= 1.0
+
+
+def test_the_founding_lie_is_a_positive_promise_the_arc_can_betray() -> None:
+ first_meeting = DAWN_ARC[0]
+ assert first_meeting.event_type is EventType.PROMISE
+ assert first_meeting.valence > 0 # she believed it, so it is filed as a good memory
+ assert "king" in first_meeting.memory_text.lower()
+
+
+def test_a_beat_builds_a_fresh_memory_every_time_it_is_asked() -> None:
+ # The store stamps an id onto whatever it is handed, so a beat must never hand out the
+ # same Memory twice; otherwise replaying the arc would corrupt the beat definitions.
+ first, second = DAWN_ARC[0].build_memory(), DAWN_ARC[0].build_memory()
+ assert first is not second
+ assert first.id is None and second.id is None
+ assert first.text == DAWN_ARC[0].memory_text
+ assert first.event_type is DAWN_ARC[0].event_type
+
+
+# ------------------------------------------------------------------ playing the arc
+
+
+def test_playing_the_arc_leaves_trust_lower_than_it_started() -> None:
+ session = _session()
+ opening_trust = session.conversation.state.trust
+ steps = play(session)
+ assert len(steps) == len(DAWN_ARC)
+ assert session.is_finished
+ assert session.conversation.state.trust < opening_trust # the betrayal lands and stays
+
+
+def test_mood_turns_negative_at_the_reckoning() -> None:
+ reckoning = _step_for(play(_session()), "the-reckoning")
+ assert reckoning.mood_after.valence < 0.0
+ assert reckoning.mood_after.valence < reckoning.mood_before.valence
+ assert reckoning.trust_after < reckoning.trust_before
+
+
+def test_the_reckoning_hands_the_model_a_visibly_upset_keeper() -> None:
+ # A recorded demo is only convincing if the prompt actually carries the hurt. These two
+ # thresholds are the ones `PromptBuilder` turns into "intensely negative", so the arc is
+ # pinned on the numbers rather than on that module's exact wording.
+ reckoning = _step_for(play(_session()), "the-reckoning")
+ assert reckoning.mood_after.valence < -0.15
+ assert reckoning.mood_after.arousal > 0.6
+ # Trust is the slow channel by design, so one betrayal wounds it rather than erasing it.
+ # What the demo can show is the size of the move: bigger than the whole warm build-up.
+ built_up = sum(step.trust_delta for step in play(_session())[:3])
+ assert reckoning.trust_delta < -built_up
+
+
+def test_the_reckoning_recalls_the_kings_errand_promise() -> None:
+ # The thesis claim in one assertion: at the moment she refuses, the specific old promise
+ # is in the prompt, so the refusal is grounded in the lie rather than in a bad mood.
+ session = _session()
+ steps = play(session)
+ reckoning = _step_for(steps, "the-reckoning")
+
+ founding_lie = session.written_memories["first-meeting"]
+ assert any(item.memory is founding_lie for item in reckoning.retrieved)
+ assert founding_lie.text in reckoning.prompt
+
+
+def test_every_beat_that_promises_a_recall_delivers_it() -> None:
+ # Each beat's `watch_for` tells the player which memory should resurface. A demo that
+ # promises a recall and does not deliver it is worse than no demo, so it is checked.
+ for step in play(_session()):
+ if step.beat is not None and step.beat.recall_beat_id is not None:
+ assert step.expected_recall_landed is True, step.beat.id
+
+
+def test_retrieved_memories_come_back_ranked_with_their_scores() -> None:
+ step = _step_for(play(_session()), "the-confession")
+ assert [item.rank for item in step.retrieved] == list(range(1, len(step.retrieved) + 1))
+ scores = [item.score for item in step.retrieved]
+ assert scores == sorted(scores, reverse=True)
+ # The per-signal contributions are what make the ranking explainable on screen.
+ assert set(step.retrieved[0].contributions) >= {"recency", "affect", "event_gate"}
+ assert step.retrieved[0].score == pytest.approx(sum(step.retrieved[0].contributions.values()))
+
+
+def test_a_step_shows_the_state_on_both_sides_of_the_appraisal() -> None:
+ step = play(_session())[0]
+ assert step.trust_after > step.trust_before # believing the errand builds trust
+ assert step.mood_after.valence > step.mood_before.valence
+ assert step.trust_delta == pytest.approx(step.trust_after - step.trust_before)
+ assert step.narration == DAWN_ARC[0].narration
+ assert step.player_input == DAWN_ARC[0].suggested_player_line
+ assert step.reply
+
+
+def test_every_step_reports_non_negative_per_stage_timings() -> None:
+ for step in play(_session()):
+ stages = (step.timings.write_ms, step.timings.retrieve_ms, step.timings.model_ms)
+ assert all(duration >= 0.0 for duration in stages)
+ assert step.timings.total_ms > 0.0
+ # Every stage runs inside the turn, so the whole turn can never be the cheaper number
+ # (the tolerance is float noise between two perf_counter readings, not slack).
+ assert step.timings.total_ms + 1e-6 >= sum(stages)
+
+
+def test_timing_leaves_the_injected_conversation_exactly_as_it_was_found() -> None:
+ # The session times the stages by wrapping them from outside, and the conversation belongs
+ # to the caller, so no wrapper may survive the step it was installed for.
+ session = _session()
+ conversation = session.conversation
+ originals = (conversation.store.add, conversation.scorer.top_k, conversation.model.generate)
+ play(session)
+ session.free_play("still here")
+ assert (conversation.store.add, conversation.scorer.top_k, conversation.model.generate) == originals
+ for owner, method_name in (
+ (conversation.store, "add"),
+ (conversation.scorer, "top_k"),
+ (conversation.model, "generate"),
+ ):
+ assert method_name not in vars(owner) # no leftover shadow of the class's own method
+
+
+def test_the_arc_leaves_one_memory_per_beat_in_the_store() -> None:
+ session = _session()
+ assert len(session.conversation.store) == 0 # the walkthrough plays the arc, not a fixture
+ play(session)
+ stored = [memory.text for memory in session.conversation.store.all()]
+ assert stored == [beat.memory_text for beat in DAWN_ARC]
+
+
+# --------------------------------------------------------------- the interactive seam
+
+
+def test_play_hands_every_step_to_the_callback_and_prints_nothing(capsys) -> None:
+ seen: list[StepResult] = []
+ steps = play(_session(), on_step=seen.append)
+ assert seen == steps
+ captured = capsys.readouterr()
+ assert captured.out == "" and captured.err == "" # rendering belongs to the caller
+
+
+def test_the_player_can_answer_a_beat_in_their_own_words() -> None:
+ session = _session()
+ improvised = "no errand, no king, I just want a bed for cheap"
+ step = session.step(player_line=improvised)
+ assert step.player_input == improvised
+ assert improvised in step.prompt
+ # The beat is still the scripted scene, so its memory is written either way.
+ assert session.written_memories["first-meeting"].text == DAWN_ARC[0].memory_text
+
+
+def test_play_can_source_each_line_from_the_player() -> None:
+ lines: list[str] = []
+ steps = play(_session(), choose_line=lambda beat: f"my own words at {beat.id}")
+ lines = [step.player_input for step in steps]
+ assert lines == [f"my own words at {beat.id}" for beat in DAWN_ARC]
+
+
+def test_stepping_past_the_last_beat_is_refused() -> None:
+ session = _session()
+ play(session)
+ assert session.next_beat is None
+ with pytest.raises(IndexError):
+ session.step()
+
+
+def test_free_play_returns_a_well_formed_step_after_the_arc() -> None:
+ session = _session()
+ play(session)
+ step = session.free_play("would you vouch for me to the guild now?")
+
+ assert isinstance(step, StepResult)
+ assert step.is_free_play and step.beat is None
+ assert step.player_input == "would you vouch for me to the guild now?"
+ assert step.narration == "" and step.expected_recall_landed is None
+ assert step.reply and step.prompt
+ assert step.retrieved and all(item.score >= 0.0 for item in step.retrieved)
+ assert step.mood_after == step.mood_before # no event written, so nothing to appraise
+ assert step.trust_after == step.trust_before
+ assert step.timings.total_ms > 0.0
+ assert session.history[-1] is step
+
+
+def test_free_play_can_remember_what_the_player_did() -> None:
+ session = _session()
+ play(session)
+ before = session.conversation.state.trust
+ session.free_play(
+ "here is the rest of what I owe you",
+ event=Memory(
+ text="The player settled the last of the account without being asked.",
+ valence=0.4,
+ arousal=0.2,
+ event_type=EventType.GIFT,
+ ),
+ )
+ assert len(session.conversation.store) == len(DAWN_ARC) + 1
+ assert session.conversation.state.trust > before
+
+
+def test_the_session_runs_on_any_model_runner() -> None:
+ class ShoutingRunner:
+ """A second ModelRunner, to prove the session only speaks through the protocol."""
+
+ def __init__(self) -> None:
+ self.prompts: list[str] = []
+
+ def generate(self, prompt: str) -> str:
+ self.prompts.append(prompt)
+ return "WHAT ERRAND?"
+
+ runner = ShoutingRunner()
+ session = WalkthroughSession(build_walkthrough_conversation(model=runner))
+ steps = play(session)
+ assert [step.reply for step in steps] == ["WHAT ERRAND?"] * len(DAWN_ARC)
+ assert len(runner.prompts) == len(DAWN_ARC)
+
+
+def test_a_model_failure_costs_the_beat_but_not_the_session() -> None:
+ # A live demo can lose its model daemon mid-take. The scene was already logged and
+ # appraised by the time the model is called, so the beat is spent rather than replayable,
+ # and the rest of the arc must still play (never a double write of the same scene).
+ class FlakyRunner:
+ def __init__(self) -> None:
+ self.calls = 0
+
+ def generate(self, prompt: str) -> str:
+ self.calls += 1
+ if self.calls == 1:
+ raise RuntimeError("the model daemon went away")
+ return "..."
+
+ session = WalkthroughSession(build_walkthrough_conversation(model=FlakyRunner()))
+ with pytest.raises(RuntimeError):
+ session.step()
+ assert session.progress == (1, len(DAWN_ARC))
+ assert len(play(session)) == len(DAWN_ARC) - 1
+ assert len(session.conversation.store) == len(DAWN_ARC) # one memory per scene, still
+
+
+def test_a_session_can_run_a_custom_arc() -> None:
+ beats = (
+ Beat(
+ id="only-beat",
+ narration="A short scene.",
+ suggested_player_line="hello",
+ memory_text="The player said hello.",
+ valence=0.1,
+ arousal=0.1,
+ event_type=EventType.NORMAL,
+ watch_for="Nothing yet.",
+ ),
+ )
+ session = WalkthroughSession(build_walkthrough_conversation(), beats=beats)
+ assert len(play(session)) == 1
+ assert session.progress == (1, 1)
+
+
+# ------------------------------------------------------------------------ immutability
+
+
+def test_playing_the_arc_never_mutates_the_beat_definitions() -> None:
+ before = [asdict(beat) for beat in DAWN_ARC]
+ session = _session()
+ play(session)
+ session.free_play("one more thing")
+ assert [asdict(beat) for beat in DAWN_ARC] == before
+
+
+def test_a_beat_cannot_be_edited_in_place() -> None:
+ # Frozen on purpose: the arc is a script, and `replace` is how a variant is made.
+ with pytest.raises(Exception):
+ DAWN_ARC[0].narration = "something else" # type: ignore[misc]
+ variant = replace(DAWN_ARC[0], narration="something else")
+ assert variant.narration == "something else"
+ assert DAWN_ARC[0].narration != "something else"
+
+
+def test_the_walkthrough_module_never_prints_or_imports_rich() -> None:
+ # The session yields data; the menu renders it. Pinned as a test so a later "just one
+ # print for debugging" cannot quietly couple the arc to a terminal.
+ from pathlib import Path
+
+ import embr.walkthrough as walkthrough
+
+ source = Path(walkthrough.__file__).read_text(encoding="utf-8")
+ assert "print(" not in source
+ assert "rich" not in source