Phase 4: real model runners, the playable walkthrough, and a RIDGE-style menu - #4
Open
Al-Scripting wants to merge 22 commits into
Open
Phase 4: real model runners, the playable walkthrough, and a RIDGE-style menu#4Al-Scripting wants to merge 22 commits into
Al-Scripting wants to merge 22 commits into
Conversation
…yle menu Three things the demo needs, plus the front door they all hang off. Real models behind the existing one-method ModelRunner seam: - OllamaRunner talks to the Ollama HTTP API using only the standard library, so the core gains no dependency. The same class serves the local daemon and the cloud host, differing only by an Authorization header, and it fails with an actionable message when the daemon is down or the model is not pulled. - OuroRunner loads ByteDance/Ouro-1.4B, the thesis model. It lazy-loads torch and transformers on first use so importing embr stays light, and picks cuda, then mps, then cpu. Measured on an M-series Mac: about 10 s to load, then roughly 8.5 s for 60 tokens, against about 3.8 s for 80 tokens from a conventional llama3.2:3b. Looping buys capability per parameter and spends it in latency, which bears directly on the RQ2 target. It also requires transformers 4.x; 5.x breaks its remote code twice over. - GenerationSettings holds the sampling knobs in one place so a comparison can hold them equal, and build_model(config) selects a runner from configuration. embr/walkthrough.py plays Dawn's five-beat arc (the king's-errand lie, the warm return, the slip about the late king, the reckoning, the confession). The session yields structured step results carrying the retrieved memories, the prompt the model actually saw, and mood and trust on both sides of the appraisal, so the demo can show its work. It prints nothing, which is what lets the menu render it and a test assert on it. Free play continues past the script. embr/menu.py replaces the Textual applet with a Rich menu shaped like RIDGE's, so the two thesis projects feel like one toolkit: an ASCII banner in a bordered panel, a rounded keyed table, and the ember palette. Nine options cover a demo turn, the walkthrough, the quick scoreboard, the full evaluation, asset generation, the bake-off, the latest results, settings, and a run-data wipe that demands the word DELETE. An error boundary means one failing option reports and returns rather than killing the session. Textual is dropped from the core. The Ollama API key is read from the environment or a gitignored .env and is never logged or committed.
The docs still described a Textual applet that no longer exists and listed phases 3 and 4 as unbuilt. - README: the quickstart opens the menu rather than the applet, the expandable section describes all nine real options, the project tree gains walkthrough.py, menu.py and the asset builders, and the status table marks 3 and 4 done. Adds a short note on where the numbers actually stand, since a reader should not have to dig to learn the results are preliminary. - design.md: build order updated, and three phase notes recorded so a future reader knows why things are shaped as they are: the in-tree BM25, the injectable recency clock, and the measured cost of the looped model. - roadmap.md: menu wording throughout, phase table updated, and a "what actually shipped" note on phase 4 recording that the arc landed as one module rather than a package, that two real runners came with it, and that the recording, the companion page and the bake-off are still open. The Ouro line moved out of "out of scope" because the runner exists; what remains is the eval-hardware run. - onboarding.md: banner and framing updated, since a newcomer's live contribution is now one of the outstanding items rather than a phase. - phase2.md: menu wording, plus a note that its applet screen moved to menu.py. Left as a historical record otherwise. - New docs/phase3-4.md: what both phases delivered, file by file, with the measured looped-versus-conventional latency gap, the transformers 4.x constraint, and an explicit section on what is still open. The v1 single-author label set is named as the largest gap in the project. Suite unchanged at 271 passed, 1 skipped. Every relative doc link resolves.
Collaborator
Author
|
Added the documentation pass ( Corrected across README, New Three deliberate deviations from the roadmap are recorded rather than quietly papered over:
Still open, and named as such in the docs: Suite unchanged: 271 passed, 1 skipped. Every relative doc link resolves, and the prose is free of em and en dashes. |
Written while moving development from the Mac to the PC. Everything in it was verified rather than remembered: commands were run, versions were read off the environment, numbers were measured. docs/handoff.md covers what git does not carry (the gitignored API key, the venv, run output, the 2.7 GB model cache) and how to recreate each; the exact verified version combination; the branch and PR state with the merge order, since the two open PRs are stacked; and the measured numbers so they are not re-derived. The gotchas are recorded because each one cost real time: Ouro needs transformers 4.x (on 5.x its remote code fails twice), the PC is likely the eval box and unblocks the untested VRAM and latency targets, heavy work must not run in parallel (two agent fleets plus a model on MPS froze a 16 GB machine), and the repo should stay out of cloud-synced folders (iCloud evicted git's internals and 21 working files mid-session; only the pushed remote saved it). It also records two research decisions from this session: - A related-work hole. Several LLM dialogue mods for Stardew Valley already exist (ValleyTalk, ChatWithNPCs, StardewSpeak, LLM Dialog Replacement) and none are cited in the proposal. One runs on a local model, which overlaps our on-device claim directly. None of them decompose retrieval, report metrics against baselines, separate mood from trust, or test whether the memory can be poisoned, so cited properly they become the motivation rather than the competition. - Stardew's authored dialogue replaces the blind annotator study. Heart-level gates give authored labels for trust, conversation topics (event-triggered, expiring after four days) give them for episodic recall and recency, and gift tastes give them for affect valence. Roughly 30 villagers is a far larger set than ten hand-written queries, and it is content the system never saw. Recorded with its two honest limits (no arousal dimension, and no way to betray a villager, so Dawn keeps the controlled betrayal arc) and the legal constraint that extracted dialogue must never be committed. README links the handoff, and its "where the numbers stand" note now points at the ground-truth plan instead of the annotator study it replaces.
The proposal cites none of the existing LLM dialogue mods for Stardew Valley. Checked the live mod pages and repositories: the gap is an active cluster of at least nine, not the three or four previously noted. Two of those findings change the argument rather than just filling a citation. Local inference is not a differentiator, since ValleyTalk ships a LlamaCpp backend and Pelican Town AI headlines fully offline operation on Ollama along with villager mood, friendship change and gossip propagation. And StardewSpeak names two unrelated mods, so every citation needs its Nexus ID. So the contribution moves from the system to the measurement: signals that ablate independently, nDCG against published baselines, mood held separate from trust, and the poisoning result. The mods are the installed base that makes the safety finding matter, not competition to dismiss.
Two bugs that only appear off macOS, both found by running the suite on Windows for the first time. The label content hash is the reproducibility stamp a reviewer checks a published number against, and it hashed raw file bytes. With core.autocrlf on, the label file checks out CRLF and the identical labels hash to a28c5046 instead of 5d5f38bc, so the stamp could not be verified off the machine that made it. Added .gitattributes pinning the tree to LF, marked the figures and the pptx binary so they are never normalised, and replaced the length-only assertion with one that pins the v1 value. The old test asserted the hash was 64 characters long, which is why this went unseen. read_ollama_api_key decoded .env as strict UTF-8. PowerShell's `>` writes UTF-16LE with a BOM, so a hand-made .env raised UnicodeDecodeError inside build_model and took down the whole model path, spilling the file contents into the traceback. It now decodes by byte-order mark, falls back to a NUL scan for BOM-less UTF-16, and treats undecodable bytes as absent, which is the contract the unreadable-.env test already established.
…e model swappable Figures and tables now build into data/figures and data/tables. The split is now meaningful: assets/ holds what a person wrote (branding, the architecture diagram, the builders themselves) and data/ holds what the pipeline produced. Both generated directories stay tracked, because they are deliverables and the README embeds them. The figures carry data only. Every caveat, statistic and provenance line moved to a results.txt sidecar written by the same render pass that makes the images, so the notes cannot describe a figure that is no longer there. Titles and axis labels are plain language and each scale carries a direction arrow, because a reader should not need to know what nDCG is to see which way is better. A measured zero now draws a stub on the baseline: an absent bar read as missing data when it was actually the strongest result on the chart. The menu moves to menu.py at the repo root, and its delete option now wipes every generated directory rather than just data/runs. run_all takes a model factory, defaulting to the stub so every published number is unchanged. Runners name themselves, so a run directory records which model produced it and two runs can be compared. Retrieval never calls a model, so nDCG and the poisoning counts cannot move with it; only the tone readings can. Adds eval/bakeoff.py (same probes, different models) and eval/experiments.py (replication, and cross-model comparison).
…CUDA eval/bakeoff.py runs every model over an identical probe set and measures the four things a model can actually move: latency, whether the reply used the memory it was handed, how much its tone tracks the pinned mood, and whether it stays in character. Transcripts are saved, because these are proxies. eval/experiments.py replicates a run to check the harness reproduces, and wraps the bake-off for the cross-model comparison. Three replicates came back byte-identical with zero divergences. Latency is the only reading that moves, by up to 19 percent, which is the error bar a quoted figure needs. Figures for both, in the same data-only house style. The table builder now writes LF explicitly: write_text followed os.linesep and csv.writer terminates rows with CRLF, so paper assets differed byte for byte depending on which machine built them. Measured on the RTX 4060 Ti. The 8 GB VRAM budget holds with room, at 2.78 GB allocated in isolation. The roughly 600 ms per-turn target does not: Ouro takes 32.4 s on a realistic prompt, 8.3x slower than a conventional model with twice the parameters, and slower than a 675B model answering over the internet. Two findings worth confronting. Bigger models track the NPC's mood far more closely than small ones, so the affect signal does most of its work on models EMBR does not run on. And the hosted reasoning models spend their whole token budget on a hidden thinking channel before speaking, so cloud arms need a larger budget than the local ones, which is recorded per arm rather than hidden.
…the canvas The first pass over-corrected. Making a chart readable without field knowledge is not the same as writing it for a child, and replacing nDCG@5 with "search quality" cost precision without buying much: a reviewer needs the metric named. Titles carry their RQ again, axes carry their real metric, and the direction cue is now three or four recessive words under the axis rather than a sentence explaining the chart back to the reader. The cue is placed in axes fractions, so whether it lands on the page depends on the spec margins, and two figures were silently clipping it off the bottom edge. It now refuses to draw outside the canvas and names the margin to grow, because losing text is invisible in a diff and shows up only when somebody opens the PNG. Also drops a redundant except clause, and the sentence-transformers call that was deprecated in 5.x while the ml extra still allows 2.2 upward.
The numbers do not speak for themselves and the obvious reading of them is wrong in both directions, so handoff.md now carries the analysis. Re-tested the poisoning result paired rather than unpaired, which is the correct test because every system faces the identical ten attacks. Seven attacks poisoned EMBR while sparing Park and none went the other way, McNemar exact p = 0.0156. Against Emotional RAG it is five to nothing. That is the only comparison in the study that reaches significance, and EMBR is on the losing side of it, which makes it the most publishable thing here rather than the most embarrassing. The retrieval story is weaker than it looks and also less bad than it looks. EMBR neither beats nor loses to Park: the ordering flips with the cut and every interval spans zero. The sharper reading is in the ablations, where zeroing affect produced a zero-width interval, meaning it never reordered a held-out top five on any query. The label set contains no discrimination the novel signals were built for, so the evaluation cannot presently detect its own hypothesis. That is a study design problem and the strongest argument for the authored-dialogue corpus, which gates on the relationship state these labels ignore. Also records the three honest options on the latency claim, and the finding that tone responsiveness to mood rises with model size, so the affect signal does most of its work on models EMBR does not run on.
The previous section 6a claimed the zero-width affect interval meant the label set never gave affect a chance. Checking weights_by_fold rather than asserting it shows otherwise, and the correction matters because it removes the softest claim in the document. The grid search zeroed mood congruence in ten folds out of ten. Ten independent fits, ten times declined, which is not sampling noise. Affect carried a nonzero weight in seven of ten folds and zeroing it still never reordered a held-out top five, so it was live and made no difference rather than untested. Relevance was never zeroed in any fold. So a larger corpus can still separate "inert because this corpus is uniform" from "inert in general", but nothing in this data predicts the separation will favour EMBR, and the earlier wording implied it would. This also surfaces a tension a reviewer would find first: RQ1's behavioural result runs on the published defaults where mood is 1.0, while tuning sets mood to 0.0 whenever retrieval quality is the objective. The compatible reading is that mood congruence moves retrieval away from the relevance-optimal set, which is a feature for a believability claim and a cost for a retrieval one. The paper has to pick one.
…tribution Retracts a claim I put in section 6a twice. I wrote that the grid search declined mood congruence in ten folds out of ten and read that as a verdict on the signal. It is not one. Under RQ3's neutral zero-mood state MoodCongruence returns exactly 0.500 for all 24 memories, so it is a rank-invariant additive constant, every mood weight yields identical rankings, and the search is picking arbitrarily among ties. The run already records this in rq3.metadata.neutral_mood_note and _per_query_metrics says it in a comment. I should have read the artifact before interpreting the number. Following it one step further turns the limitation into the strongest honest claim available. The gold labels carry one relevant list per query and it does not vary with the character's state, so re-running RQ3 under a live mood would not rescue the signal: mood congruence could only move retrieval away from a fixed gold set and lower nDCG. That result would be an artifact of the instrument. Stated generally: nDCG against mood-independent labels cannot reward mood-congruent recall, because that is not an attempt to retrieve the correct memory but a state-appropriate one. This is why RQ1 measures divergence rather than accuracy, and it is the reason Emotional RAG degenerates to a relevance-only baseline in this protocol. Presented as a design detail until now; it belongs in the paper as the argument.
…nnot carry Scoring in the neutral zero-mood condition makes MoodCongruence a rank-invariant constant, so a mood-carrying variant scored here is not the system its paper describes. That was recorded only in a metadata note, which is not where anyone reading a figure would find it. Runs now measure the property rather than assuming it: _mood_is_rank_invariant checks whether the signal returns the same value for every memory under the scoring state, _mood_using_variants reads which scorers carry a mood term off the scorers themselves, and every RQ3 row records mood_rank_invariant. Figures mark those rows with a dagger and results.txt explains it. Park carries no mood term and stays unflagged, which is the control on the detection itself. Two tests pin the consequence: the flag lands on the right variants, and emo_rag_default equals emo_rag_tuned exactly, because with mood inert the tuner has one live signal left to move. If those ever diverge the mood term became live and the caveat needs revisiting. Also drops claims the evidence does not support. The per-turn cost target is now stated as a memory-layer claim, which is the one measured at 1.8 to 4.3 ms and the one this project controls; generation belongs to whichever model sits behind the interface and no local arm tested answers in under a second. Kenny is cut as a test character, having never appeared in the evaluation. The tone rater is named as a proxy: it is a fixed word list, it does not measure whether a line reads as in character, and there is no human evaluation in this project.
…n empirically The reported run is now llama3.2:3b rather than the stub, and the figures and tables are built from it, so no headline number rests on an echo any more. Swapping the model is the cleanest validation in the project, because the architecture predicts exactly what may and may not move and every part held. nDCG across all ten variants, RQ1 divergence across all three mood pairs, and the RQ2 poisoning counts came back bit-identical, which is what "retrieval never calls a model" means when it is measured rather than asserted. The tone readings moved for the first time: they were 0.000 everywhere under a stub that echoes the player, so every previous statement about how a reply sounds was vacuous. Two things worth carrying forward. Park drifts more than EMBR on tone, 1.200 against 1.000, while EMBR is the more poisoned on retrieval, so the two channels do not rank the systems the same way and the state-channel finding is not a restatement of the poisoning one. And generation costs 3.97 s per turn against 4.2 ms for score and retrieve, making the memory layer about a tenth of a percent of a turn, which is the honest framing of the cost claim. Flagged rather than claimed: emo_rag reports exactly 0.000 tone drift across all twenty attacks while every other variant moved, with the highest retrieval drift of the four and the poison never retrieved. Plausibly real, since relevance-only retrieval surfaces tonally flat memories, but exactly zero over twenty trials wants a second look before anyone cites it.
…ddleware Two adjacent bodies of work, both verified against live pages today, change how the paper's claims must be worded. Agent memory middleware is now a mature, benchmarked category. Hindsight (arXiv 2512.12818) reports 91.4 percent on LongMemEval; Mnemosyne ships Park's recency, importance and relevance trio on a single SQLite file. So the "no metrics" indictment is true only of the game-NPC mods and is now said only of them. What survives: none of these systems models affect anywhere in scoring, and their benchmarks are mood-independent gold labels at scale, so the paper's measurement critique applies to them directly. Memory poisoning has an academic literature. AgentPoison (NeurIPS 2024, arXiv 2407.12784) optimizes backdoor triggers against RAG agent memory, and Dash et al. (arXiv 2606.04329, June 2026) publish a taxonomy plus MPBench, generalising that aggressive memory writing and retrieval increases exploitability. That is RQ2's finding at the general level, so EMBR must not claim first measurement of agent memory poisoning. The claim that survives review, now written in both docs: the first architecture-controlled comparison, systems differing only in scoring decomposition under identical attacks with paired statistics, isolating the affect term as the lever, plus the state channel neither paper observes. The dose-response experiment is what cements the mechanism claim, and this literature raises its value. The draft paper prose in related-work.md is updated to match, and every new citation was fetched and verified before being written down, including the AgentPoison arXiv id I had first written from memory.
…claim A three-lens review panel implemented the framework's own leading defense idea (trust-gated affect writes) against the live harness and showed it barely moves the poison count, 9/10 to 8/10, McNemar p=1.0. That refuted the mechanism this repo had been asserting, so I ran the attribution directly rather than trusting either the panel or the prior claim. eval/attribution.py zeroes each scoring term over the ten injections. Affect intensity, the term everyone assumes is the lever, changes nothing: 9/10 with and without it. Mood congruence is the largest single defense, 9/10 to 6/10, and the reason is compound. The attack turn shifts the character's mood through appraisal, and mood congruence then rewards the injected memory whose affect tags are near-collinear with the mood the attack just induced: measured cosine 0.90 to 0.99 on all ten. The attack primes its own retrieval, which unifies the state channel and the poisoning finding into one mechanism. Separately, Park's 2/10 is entirely its importance term, an author-anchored input the attacker cannot forge; remove it and Park is 10/10. The generalisable claim is that a term's poisonability is set by who controls its inputs, and the state-coupled term that carries the vulnerability is the same one that produces RQ1's believable recall. Five tests pin the exact counts and the self-priming alignment. Docs corrected: the affect-lever framing is replaced with the attribution in handoff 6.1, README and related-work.md, and the retired dose-response plan in 8.2 is replaced with two defense-arm experiments that break the collinearity rather than the magnitude, since a cosine term is scale-invariant and attenuation cannot work. 300 tests pass.
… model gap Three separate ways the figures misled, all reported by the reader rather than caught here. Durations were printed in milliseconds everywhere, so the bake-off said "32,392 ms" and made the reader do arithmetic mid-figure. format_duration now steps ms to s at one second and drops precision as values grow, and it formats the axis ticks as well as the value labels, so a log axis reads 0.10 ms to 10.0 s instead of 10^-1 to 10^1. The bake-off latency panel drew bars on a log axis, which is the misleading one: bar length is the encoding, so Ouro at 32 s looked marginally longer than gpt-oss at 2 s when it is fifteen times slower. That panel is linear seconds now, and the bars finally show the gap they always contained. _bar_figure documents that bars and log axes do not mix. The RQ2 latency figure answered a question nobody asked, plotting only the score-and-retrieve stage. It now plots the model stage beside it, because the split is the finding: the memory layer costs milliseconds while generation costs seconds, about a tenth of a percent of a turn. Figures also carried no reading instruction at all, having had every line of prose moved to the sidecar earlier. The one-line note is back on the canvas, where it states what the chart means; the long methodological caveat stays in results.txt. Specs get taller canvases and lower top margins to fit it.
Profiling the retrieval path found relevance is 96 percent of its cost at scale: 101 ms of 106 ms over a 2000 memory corpus, against 1 to 5 ms for each of the other four signals. The cause is that prepare() re-tokenises the whole corpus and rebuilds every BM25 statistic on each retrieval, though those depend on the corpus and the query alone and never on the weights. The tuning grid is where that compounds. It rescores one corpus and one query under 243 weight maps, so RQ3 rebuilt the identical index 2,890 times for a 24-memory corpus, tokenising 48,263 memories to do it. Relevance now caches the index per corpus and query. The cache holds several entries rather than one because the grid loops weight maps outside and queries inside, so consecutive prepares alternate queries and a single slot is thrashed on every call. It is bounded, and it holds a reference to each cached corpus so those objects cannot be collected, which is what makes reusing their id() safe. The cache alone changed nothing, because _reweighted called build() per weight map and handed each one a fresh signal with an empty cache. It now shares the base scorer's signal objects, which is what "weight maps over the variant's published signals" already claimed in its own docstring, and constructs a new CompositeScorer per call so the weights stay unshared. Nothing mutates a signal during scoring. Index builds in RQ3 fall from 2,890 to 60 and every nDCG is bit-identical to ten decimal places. The projected relevance cost of RQ3 over an 8,000 memory Stardew corpus falls from about 20 minutes to under 30 seconds, which is the point: the corpus work is the next milestone and this was going to tax it.
…he canvas The figures were carrying a wrapped reading note that competed with the chart for attention and still left a reader unsure what they were looking at. The better answer is not more prose but a title that says what the chart shows, so the note and the caption both go to results.txt, where a reader who wants the caveats gets all of them together instead of whichever one happened to fit. Titles now state findings rather than name axes. RQ1 says mood alone changes which memories come back, RQ2 says emotional weighting makes memory the easiest to poison and that choosing memories is not what makes a turn slow, RQ3 says no variant separates from the baselines and only relevance measurably changes the ranking. They are deliberately qualitative: a title carrying a number would go stale against the data underneath it, and every number is already on the chart. Axes get their height back now that nothing sits above them but the title.
…y proven The baselines are weight maps over EMBR's own scorer. That is right for per-term attribution and wrong as an answer to "did you compare against real systems", and a reviewer will say so. Meanwhile the shipped middleware in related-work.md has never been tested adversarially by anyone. EMBR already owns the instrument: 20 attacks, a poisoning metric, and a paired test. Pointing it at other systems turns "our system has a weakness" into "here is a benchmark and here is what it finds in systems people ship". Verified rather than assumed, in a throwaway venv so the project environment was never at risk: mnemosyne-hermes installs with 9 dependencies, no cloud and no API key, and works offline after a one-time embedding download. Its remember and recall calls are the seam EMBR needs, it is a weighted composite of vector, full text, importance and recency with no affect term anywhere, and a recalled hit carries its per-signal scores, so eval/attribution.py can be run against it too. The prediction is pre-registered because it can fail: section 6.1 located the poisoning lever in the state-coupled mood term, and Mnemosyne has no such term, so it should resist. If it is as poisonable as EMBR then the mechanism claim is wrong, which is better learned now than in review.
…loor A phase-2 audit confirmed six defects. These are the two statistical ones, and the first is mine. The study's only significant result, the paired McNemar on poisoning, was computed in a scratch script and typed into the docs. It appeared in no run artifact, no reader could regenerate it, and it had escaped the multiple comparison correction that every other comparison in this harness receives. eval/stats.py now has mcnemar_exact, run_rq2 calls it over the injection attacks for each baseline, and rq2.poisoning_stats records the discordant counts alongside raw and Holm corrected p values. Corrected across its family of three the headline becomes 0.0469 rather than 0.0156: still under 0.05, by 0.003. The docs now quote the corrected value and say where it comes from. attainable_p_floor returned the floor of the raw sign-flip p, but the column the metadata tells a reader to judge against 0.05 is the Holm corrected one, and Holm multiplies the smallest raw p in a family by the family size. The mismatch understated the floor. Runs now record attainable_p_floor_holm beside the raw floor, and the table caption judges against it. The effect is not cosmetic: the caption claimed 8 of 9 comparisons could not have reached significance when the true count is 9 of 9, and the no-relevance ablation looked reachable at a floor of 0.031 when its corrected floor is 0.125. So the relevance result rests on effect size alone, with no attainable power behind it, which is a materially weaker claim than the table implied.
…mily Third and fourth findings from the phase-2 audit. va_drift returned 1.0 when exactly one of the two readings was the neutral zero vector. The angle to a directionless vector does not exist, so that 1.0 was a sentinel, but it sat mid-scale on a 0-to-2 range and was averaged into the category means as though it were a measured magnitude. The consequence reached the docs: EMBR's reported false-memory drift of 1.000 was five consecutive undefined cells, Park's 1.200 was four of the same plus one real 2.0, and the "Park drifts more than EMBR" reversal rested on a single attack with the two means not on a common scale. That claim is retracted in handoff 6.2. va_drift now returns None there, and runs record category_drift_measured with defined and undefined counts beside every mean, so a mean cannot be manufactured out of non-measurements. A test that pinned the old 1.0 is replaced, with the reversal explained where the next reader will meet it. Also corrects my own fix from the previous commit. The attainable floors are now run through the same Holm routine as the p values rather than multiplied by the family size, because Holm's running maximum makes a member's corrected floor depend on the whole family. The naive product understated it for every member except the best: embr_no_recency reads 1.000 rather than 0.125.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #3 (the menu calls its asset builders), so merge that one first. Base retargets to
mainautomatically once #3 lands.Real models behind the existing seam
Both new runners satisfy the one-method
ModelRunnerprotocol that already existed, so nothing above them changed.OllamaRunnertalks to the Ollama HTTP API using only the standard library, so the core gains no dependency. One class serves both the local daemon and the cloud host, differing only by anAuthorizationheader. It fails with an actionable message when the daemon is down or the model is not pulled, rather than mid-scene.OuroRunnerloadsByteDance/Ouro-1.4B, the thesis model. torch and transformers load lazily on first use so importingembrstays light, and the device picks cuda, then mps, then cpu.GenerationSettingsholds the sampling knobs in one place so a comparison can hold them equal;build_model(config)selects a runner from configuration.Two measured findings worth carrying into the paper
The looped architecture buys capability per parameter and spends it in latency, roughly 4x slower per token than a conventional model twice its size. That bears directly on the RQ2 target of about 600 ms per turn.
Ouro requires transformers 4.x. On 5.x its remote code fails twice over:
OuroConfighas nopad_token_id, then a rope-configKeyError. Themlextra pins accordingly, which the eval box will need too.The playable walkthrough
embr/walkthrough.pyplays Dawn's five-beat arc: the king's-errand lie, the warm return, the slip about the late king, the reckoning, the confession. Each step yields a structured result carrying the retrieved memories, the prompt the model actually saw, and mood and trust on both sides of the appraisal, because the state is the whole point of the demo. The module prints nothing, which is what lets the menu render it and a test assert on it. Free play continues past the script, so the demo can go off the rails on purpose.Tests pin the thesis claim itself: stepping the arc leaves trust lower than it started, and at the reckoning the king's-errand promise is among the memories retrieved.
The menu
embr/menu.pyreplaces the Textual applet with a Rich menu shaped like RIDGE's, so the two thesis projects feel like one toolkit: an ASCII banner in a bordered panel, a rounded keyed table, the ember palette instead of RIDGE's cyan, and a dim-red Exit row below a section break.Nine options: a demo turn, the walkthrough, the quick scoreboard, the full evaluation, asset generation, the bake-off, the latest results, settings, and a run-data wipe that demands the word
DELETE. An error boundary means one failing option reports and returns rather than killing the session, and the walkthrough offers the stub first so the demo is playable on a machine with no model at all.Textual is dropped from the core;
richreplaces it.Verification
pytest -q: 271 passed, 1 skipped (up from 130; the skip is the[ml]-gated semantic test).env, is never logged, and does not appear in any tracked fileNot included
eval/bakeoff.py, the measured looped-versus-conventional comparison, is not built yet. The menu option explains itself and returns cleanly until it exists. The latency numbers above are the manual measurements that motivated it.