fix(agents): honour profile model for Claude and harden agent test harness - #522
Merged
Conversation
Two harness defects made the agent suite fail locally regardless of the product state. Together they accounted for 4 of 8 failures on main. 1. Empty-string env coalescing. CodeMie exports its full CODEMIE_* block into the shell of every agent session it launches, and the anthropic-subscription provider deliberately blanks CODEMIE_MODEL so the Claude CLI falls back to its own defaults. Running the suite from such a shell inherited CODEMIE_MODEL='', and `??` only falls back on null/undefined - so the empty string beat the intended default and every generated profile was written with no model, failing with "Configuration incomplete / Missing: model". Adds getTestEnvValue/getCodemieTestUrl/getCodemieTestModel, which treat empty and whitespace-only values as absent. CI_CODEMIE_URL now also falls back to CODEMIE_URL, so no .env.test.local is needed when the suite runs inside a CodeMie-launched shell. 2. SSO credentials never reached the test workers. CREDENTIALS_DIR and FALLBACK_FILE in src/utils/security.ts are module-level constants resolved from CODEMIE_HOME at import time, and the agent vitest project points CODEMIE_HOME at a throwaway temp home. The in-process SDK client therefore failed with "SSO authentication required" even though global setup had just validated the same credentials. The agent-assistant and agent-skills suites now copy the credentials into that temp home first. Adds tests/integration/test-env-helpers.test.ts covering the coalescing rules with synthetic variable names, so the assertions hold whether or not a developer has a .env.test.local. agent-model.test.ts is deliberately left untouched: the pre-commit hook runs vitest on every staged test file, and TC-020/TC-021/TC-024 still fail there for unrelated reasons (a sonnet-configured profile records a haiku model). Those are not masked here. Generated with AI Co-Authored-By: codemie-ai <codemie.ai@gmail.com> Claude-Session: https://claude.ai/code/session_01Ce5td85FtwMh65FG66LWaK
…ured The model chosen in `codemie setup` never reached the wire for Claude. claude.plugin.ts maps the profile's model onto ANTHROPIC_MODEL only, but Claude Code ignores that variable. Its actual precedence is: --model > ~/.claude/settings.json "model" > its own default tier Verified against claude 2.1.218 on an ai-run-sso profile by inspecting the outgoing request body: settings "model" ANTHROPIC_MODEL request body claude-haiku-4-5 claude-sonnet-4-6 claude-haiku-4-5 claude-haiku-4-5 claude-sonnet-5 claude-haiku-4-5 (none) claude-sonnet-4-6 claude-opus-5 -> 400 budget (none) claude-sonnet-5 claude-opus-5 -> 400 budget claude-sonnet-5 claude-sonnet-5 claude-sonnet-5 So a developer with "model" pinned in ~/.claude/settings.json silently ran every session on that model, and one without it silently ran on the default opus tier - in both cases ignoring the configured profile. The proxy and the metrics were accurate throughout; they faithfully recorded the model the binary actually asked for. Fix: inject `--model $CODEMIE_MODEL` in the claude enrichArgs hook, next to the existing --plugin-dir injection, using the same "already present" guard so an explicit --model / --model= in argv still wins. A blank CODEMIE_MODEL is treated as absent rather than forwarded as an empty flag: anthropic-subscription deliberately blanks it so the Claude CLI applies its own defaults. This makes TC-020 and TC-021 pass - they assert that the session uses the model from the profile, which is exactly the behaviour that was broken. TC-024 and TC-029 still fail; both need a live interactive PTY flow and are unrelated to model selection. Generated with AI Co-Authored-By: codemie-ai <codemie.ai@gmail.com> Claude-Session: https://claude.ai/code/session_01Ce5td85FtwMh65FG66LWaK
CI on this PR failed on Ubuntu with every test passing: Test Files 264 passed (264) Tests 3953 passed | 2 skipped (3955) ⎯⎯⎯ Unhandled Errors ⎯⎯⎯ Error: ENOENT: no such file or directory, open '/tmp/metrics-upload-XXXX/logs/debug-2026-09-01.log' ##[error]Process completed with exit code 1 fs.createWriteStream opens the file asynchronously, so a logs directory that disappears between the mkdir and the open surfaces as an 'error' event on the stream - never as a throw that initializeLogFile's try/catch can see. With no listener attached Node treats it as an unhandled error and takes the whole process down. The trigger in CI is metrics-upload-contract.test.ts, which points CODEMIE_HOME at a temp dir in beforeEach and rmSync's it in afterEach, but the same shape applies in production: any ephemeral CODEMIE_HOME (a sandbox, a container that cleans up under us) could kill the CLI purely because logging failed. Logging must never be fatal. Fix: attach an 'error' listener that drops the stream and degrades to console-only logging. The regression test occupies the log path with a directory so the async open fails deterministically with EISDIR, rather than racing the ENOENT timing. It was verified against the unmodified logger, where it reproduces the same "Unhandled Errors" block that failed CI. Unit suite: 265 files, 3956 tests passing. Generated with AI Co-Authored-By: codemie-ai <codemie.ai@gmail.com> Claude-Session: https://claude.ai/code/session_01Ce5td85FtwMh65FG66LWaK
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The model chosen in
codemie setupnever reached the wire for Claude.claude.plugin.tsmaps the profile's model ontoANTHROPIC_MODELonly, but Claude Code ignores that variable — its precedence is--model>~/.claude/settings.json"model"> its own default tier. So a developer withmodelpinned in~/.claude/settings.jsonsilently ran every session on that model, and one without it silently ran on the default (opus) tier.This PR fixes that, and separately hardens two agent test-harness defects that made the suite fail locally regardless of product state.
Changes
fix(agents)— model selection--model $CODEMIE_MODELin theclaudeenrichArgshook, beside the existing--plugin-dirinjection, with the same "already present" guard.--model/--model=in argv still wins — CodeMie never overrides a user's own choice.CODEMIE_MODELis treated as absent rather than forwarded as--model '':anthropic-subscriptiondeliberately blanks it so the Claude CLI applies its own defaults.test(tests)— harness hardeninggetTestEnvValue/getCodemieTestUrl/getCodemieTestModeltreat empty and whitespace-only env values as absent. CodeMie exports its wholeCODEMIE_*block into any session it launches, so running the suite from such a shell inheritedCODEMIE_MODEL='', and??let that empty string beat the intended default — every generated profile was written with no model.CI_CODEMIE_URLnow falls back toCODEMIE_URL, so no.env.test.localis needed inside a CodeMie-launched shell.agent-assistant/agent-skillscopy SSO credentials into the worker's tempCODEMIE_HOME.CREDENTIALS_DIR/FALLBACK_FILEinsrc/utils/security.tsare module-level constants resolved at import time, so the real credentials were never in scope for the in-process SDK client.Impact
Evidence for the precedence claim, from the outgoing request body (claude 2.1.218,
ai-run-sso):modelANTHROPIC_MODELclaude-haiku-4-5claude-sonnet-4-6claude-haiku-4-5claude-haiku-4-5claude-sonnet-5claude-haiku-4-5claude-sonnet-4-6claude-opus-5→ 400 budgetclaude-sonnet-5claude-opus-5→ 400 budgetclaude-sonnet-5claude-sonnet-5claude-sonnet-5User-visible behaviour change: the configured model now actually takes effect. Anyone who had a
modelpinned in~/.claude/settings.json, or who was silently running on the default tier, will see their profile's model used instead — with the corresponding cost and quality difference. Worth calling out in release notes.Test results:
TC-020 and TC-021 now pass — they assert the session uses the profile's model, which is exactly what was broken, so they were correctly failing before.
Known remaining failures (pre-existing, not addressed here)
/modelswitch session emits no metrics file. Present in every pre-fix run; verified unaffected by this change.Both were also verified to fail identically with #505 reverted, so neither is a regression from this release.
Checklist
CODEMIE_DEBUG=true)ANTHROPIC_MODELmapping inclaude.plugin.ts:89,317is now supplemented by--modeland may warrant a notehttps://claude.ai/code/session_01Ce5td85FtwMh65FG66LWaK