SysID honest fit reports, parallel objective, open-loop resume - #167
Merged
Conversation
yichao-liang
force-pushed
the
bl-merge-12-explorer-arms
branch
from
September 1, 2026 14:21
e9d9286 to
ce0498a
Compare
yichao-liang
force-pushed
the
bl-merge-13-sysid-monitor
branch
2 times, most recently
from
September 1, 2026 14:43
08354e4 to
ee3e095
Compare
yichao-liang
force-pushed
the
bl-merge-12-explorer-arms
branch
from
September 1, 2026 14:43
ce0498a to
7e14335
Compare
yichao-liang
force-pushed
the
bl-merge-13-sysid-monitor
branch
from
September 1, 2026 15:01
ee3e095 to
68a6026
Compare
yichao-liang
force-pushed
the
bl-merge-12-explorer-arms
branch
from
September 1, 2026 15:01
7e14335 to
3d29697
Compare
yichao-liang
force-pushed
the
bl-merge-13-sysid-monitor
branch
2 times, most recently
from
September 1, 2026 15:42
9c5e34b to
368eeed
Compare
yichao-liang
force-pushed
the
bl-merge-12-explorer-arms
branch
from
September 1, 2026 15:42
3e481d9 to
d8ab893
Compare
yichao-liang
force-pushed
the
bl-merge-13-sysid-monitor
branch
from
September 1, 2026 15:55
368eeed to
eee29a4
Compare
yichao-liang
force-pushed
the
bl-merge-12-explorer-arms
branch
2 times, most recently
from
September 1, 2026 16:11
b0cdbca to
ca8f71d
Compare
yichao-liang
force-pushed
the
bl-merge-13-sysid-monitor
branch
from
September 1, 2026 16:11
eee29a4 to
c19c46b
Compare
yichao-liang
changed the base branch from
bl-merge-12-explorer-arms
to
master
September 1, 2026 16:26
…iagnosis script
Stage 1 of the inert-fitter work from the 2026-08-30 bridge run review:
- run_rollout_sysid computes the init-params SSE on the SAME survivor
set post_sse is computed on (pre_sse_survivors); pre_sse still covers
all segments. The sim.fit report's "% SSE reduction" is now measured
on that like-for-like baseline - the old ratio compared all-segment
init SSE against survivor-only fitted SSE, so with 21/27 segments
trimmed it reported "74% SSE reduction" from a fit whose every
parameter delta was +0.0000.
- The report opens with an explicit banner when the optimizer moved no
parameter: the values below are declared inits, not estimates.
- The identifiability report gives contraction > 1 its own label
("posterior Nx WIDER than the prior - the data carries ~no
information") instead of calling 4.4x or 33x "posterior ~= prior".
Stage 2 tooling: scripts/sysid_fit_diagnosis.py replays a run's fit
offline from the persisted fit_data pickle + the run's simulator.py
(same prep, scaling, and trimming), then probes the survivor-set SSE
with each parameter alone at its range extremes, and the same probe on
the trimmed-away segments - separating "objective genuinely flat"
from "the optimizer missed a real slope" and from "the information
lives in the segments the trimming dropped".
Verified: mypy clean, pylint clean, 243 tests green across
code_sim_learning + the sim-learning approach suites (all domains).
Claude-Session: https://claude.ai/code/session_017Acb4dHk2Ryju5XFvcjdNU
With defaults-only CFG the fresh fit env could not reconstruct the recorded states (env construction depends on weld pinning, contact margins, and more), so the script now parses the exact main.py invocation from the run's info.log and replays those flags, with --env/--seed as optional overrides. Claude-Session: https://claude.ai/code/session_017Acb4dHk2Ryju5XFvcjdNU
…earch Two coupled fixes to the rollout fit, from the 2026-08-31 diagnosis of the bridge runs (98 and 34 fit-minutes per run for zero parameter movement, with one real improvement missed): Parallel objective. _iter_rollout_residual_terms fans whole trajectories out to the existing prefetch_parallel fork pool when the env is a per-rollout factory - the case where a child's term list is bit-identical to the serial path's, as the finite-difference Jacobian requires. Every stage (LM Jacobian, grid seed, sensitivity, ablation) funnels through it, so all inherit the speedup; child rollouts are credited to the parent's counter (add_rollouts_run) and the pool's per-call log line drops to DEBUG here (hundreds of calls per fit). Shared env instances keep the serial path untouched. Bracket-search revival. The zero-gradient bracket search for threshold/gate params was dead code on the rollout MAP path: the Gaussian prior rows have structurally nonzero derivative in every column, so the exact-0.0 all-rows column test could never fire. zero_jacobian_columns now tests the DATA rows only, with a tolerance (abs 1e-8 / rel 1e-6) instead of exact zero. The search itself now takes the residual vector: flat verdicts are judged on data-rows SSE (at a box edge the prior rows alone add orders of magnitude more than the flat tolerance), while the argmin and move acceptance stay on the full MAP objective. Box edges are probed first, settling a flat parameter in 2 evaluations instead of 9; params measured flat across their box feed FitResult.sensitivity as INSENSITIVE evidence, sparing the identifiability probe their 2 rollout evals each, and the bracket notes ride FitResult.lm_notes into the sim.fit report. sysid_fit_diagnosis gains --parallel_workers to override the replayed flag (0 forces the serial path for parity checks). Verification (logs/sysid_diagnosis/, vs old-code baselines 21643951/21643952): - seed1 workers=6 vs workers=0: logs byte-identical except elapsed time; fit stage 398 s vs 1246 s (3.1x). Outcome unchanged - init at the SSE floor, moved NONE, same 25 verdicts. - seed0: trimming byte-identical, and the fit now recovers cure_steps 24 -> 23.49 and glue_lat_radius 0.011 -> 0.0148, both corroborated by the baseline edge probes (dSSE -1.0 and -0.04); the old fit moved neither. - Suites: 617 passed (code_sim_learning + agent_sdk); full-tree pylint, mypy, formatters clean. Claude-Session: https://claude.ai/code/session_01FkveUgLNgKQywdAdjJD1iE
… edges An independent high-effort review of today's five commits confirmed eight defects, and the full test tree (1806 tests) caught a ninth. All fixed: - scene.py: a stateful render_pybullet_image never restored the env after ctx.env._set_state(state), so the submit gate's per-step renders teleported the SHARED session env through the plan's states (wiping its residual-command queue and re-syncing welds) while the gate's rollouts ran on a fresh env - and planner simulate() SKIPS its own reset when the incoming state allclose-matches, inheriting the residue. Renders now snapshot and restore the env state and command queue. A plausible root cause of the open belief session-state leak. - utils.wait_rollout_step_cap(): the stalled-Wait advisories compared against max_num_steps_option_rollout (1000) while the option model's wait_option_max_steps backstop (bridge: 120) terminated the Wait first, so they could never fire in exactly the configured envs they were written for; both sites now key on whichever cap binds. - physical_sysid: the anchor-ablation rebuild dropped lm_notes, so the bracket search's "NOT fit from data" warnings vanished from fit reports whenever any parameter got pinned. - synthesis + sysid_fit_diagnosis: exact float equality broke the no-op banner and reported phantom moves - log-scale params round-trip through exp(log(x)) and trf nudges bound-sitting inits by 1e-9; both now use np.isclose. - lm.py: a failed or regressing post-bracket polish returned the bracket-moved theta paired with the PRE-bracket Jacobian; the pair now degrades to jacobian=None (all consumers handle it). The zero-gradient tolerance scales by the MEDIAN column max so one badly-scaled residual block cannot inflate every other parameter's threshold. Edge-screened flat params stay OUT of flat_out - two edge evals cannot rule out an interior-only response, so the identifiability probe stays armed as their backstop. - belief_probe: sim.state()'s full dict resolves bare-name lookups via __missing__, so helpers persisted from before the name:type keying keep working across a resume. - python_exec: the sandbox chdir moved inside the try - the budget watchdog's async exception could land between chdir and try-entry, skip the finally, and strand the whole process in the sandbox cwd. - config/settings: relative CFG track paths anchor at LAUNCH_CWD (captured at process start), so a sim.fit issued inside run_python's sandbox-cwd exec window cannot silently miss the fan domain's track manifest and fall back to per-step scoring. - skill_factories (full-tree pytest catch): ERROR level restored inside the log_errors gate - that flag is the method's final-failure emission contract (every in-run caller passes False), and the log spam the morning demotion targeted came only from the ungated per-candidate site, which stays debug. Verified: 681 tests across the affected suites on a compute node, new unit tests for each regression path (wait-cap binding, edge-screen probe backstop, launch-cwd anchoring, bare-name state lookups), pylint, mypy, and formatters clean. Claude-Session: https://claude.ai/code/session_01X3v2SxFDcKFoehfyzpvHXr
…e suffix From the 2026-09-01 seed 2-4 regression diagnosis: 21d2164 made invented-predicate annotations actually reach the subgoal monitor (the 2026-08-30 runs dropped them all at re-parse, so monitoring was inert), and the abort behind it - divergence + no refinable suffix + fallback disabled = ApproachFailure - killed both of seed 4's validated test episodes on millimeter-scale claim misses whose remaining settle/cure steps might still have delivered. An annotation is the agent's prediction, not proof the goal is out of reach, so a divergence with no refinable suffix (or a spent replan budget) now resumes the remaining not-yet-executed options open-loop: the divergence stays in the log, monitoring re-arms over the resumed suffix, and the goal check decides the episode. The grounded plan rides _exec_plan next to _exec_status so the handler can slice off the remaining options (the dispensed policy holds them only in its closure). agent_bilevel_replan_agent_fallback still opts into a fresh agent sketch instead. Claude-Session: https://claude.ai/code/session_017NQYkVi9etnpoMJBQobvGw
From the 2026-09-01 seed 2-4 regression diagnosis: every fit of three independent bridge runs was refused with all segments at 1.00-1.35x the trimming cutoff - byte-identical under the pre-21d21641 code and the serial/fork objectives (job 21736393), i.e. a replay-fidelity floor, not chaotic data - yet the NO-FIT-RAN banner blamed unrepeatable recordings and steered the agents into re-collection loops that pegged their 45-minute solve budgets. Both trimming reports (the NO-FIT-RAN banner and the partial-trim note) now print the numeric threshold and split the dropped segments by margin: within 1.5x of the cutoff is reported as a model-fidelity limit (re-collecting the same experiments will score the same; improve the simulator's rules or trust the measured inits), and only far-over segments keep the chaotic-recording advice. DEFAULT_NOISE_SIGMA moves to physical_sysid so the reported cutoff and the trimmer's can never drift apart. Claude-Session: https://claude.ai/code/session_017NQYkVi9etnpoMJBQobvGw
Suffix replanning never succeeded in any bridge run: 0 of 4 divergence events recovered across seeds 0-4 (2026-08-29 through 2026-09-01), at ~8 futile refinement attempts of up to 600 s each, and every event then aborted the episode (pre-8cccfe42). Eval execution goes back to open-loop, matching the 2026-08-30 generation that solved 2/2. The flag also disarms the divergence monitor; re-raising it re-enables advisory monitoring, now with the open-loop resume from 8cccfe4 instead of the abort. Claude-Session: https://claude.ai/code/session_017NQYkVi9etnpoMJBQobvGw
wait_rollout_step_cap() ran int(CFG.wait_option_max_steps) whenever
wait_option_terminate_on_atom_change was set, but the setting's default
is float('inf') (no backstop configured), so every probe/report path
under a default config crashed with OverflowError: cannot convert float
infinity to integer. The backstop now only participates in the min when
it is finite, and the binding-ceiling regression test pins the inf case
to the rollout cap.
Claude-Session: https://claude.ai/code/session_01VQkk8ycwwwiSKraDw88FVm
yichao-liang
force-pushed
the
bl-merge-13-sysid-monitor
branch
from
September 1, 2026 16:26
c19c46b to
7651fde
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
SysID gets an honest fit report (like-for-like SSE, no-op banner, margin-aware trimming advice with the threshold in the banner), an offline diagnosis script that replays a run's logged flags, and a fork-parallel rollout objective with the revived MAP-path bracket search. The execution monitor resumes open-loop when a divergence has no refinable suffix, and the plan arm disables execution replans.
Commits:
Part of the bridge-learning merge stack.
🤖 Generated with Claude Code
https://claude.ai/code/session_01VQkk8ycwwwiSKraDw88FVm