agent-improvement-loop

Self-improvement for you and your AI agents. A small, local, safe daily loop that mines Claude Code, Codex, and Hermes sessions for reusable improvements, then stages them for your approval.

It never changes anything on its own. It reads your transcripts, finds the friction you keep hitting, redacts anything sensitive, and writes a review packet of staged proposals. You decide what to apply.

The idea: most people point AI at their work. The higher-leverage first loop points it at your own setup. Every fix you make to a skill, a command, a hook, or a tool pays off in every future session. This tool finds those fixes by reading what already happened, instead of asking you to remember.

What it does

1. Collect local Claude Code (~/.claude/projects/**/*.jsonl), Codex (~/.codex/sessions/**/*.jsonl), and Hermes profile transcripts from one or more home directories.
2. Normalize each into a small, redacted event model (tool calls, shell commands, skill use, failures, and corrections).
3. Detect reusable improvement signals from actual tool usage, not prose mentions. If you typed "don't use that CLI," that sentence is not counted as the CLI failing.
4. Stage proposals and a human-readable review packet under ~/.agent-improvement/.

It does not edit skills, memory, runbooks, config, or source code. Scan and stage are automated. Apply stays manual.

Install

Requires Python 3.10+. No dependencies, standard library only.

git clone https://github.com/cathrynlavery/agent-improvement-loop
cd agent-improvement-loop
./bin/daily-improvement-loop --since-days 1

Or install the CLI on your PATH:

pipx install git+https://github.com/cathrynlavery/agent-improvement-loop
# or: uv tool install git+https://github.com/cathrynlavery/agent-improvement-loop
agent-improvement-loop --since-days 1

Usage

# Mine sessions from the last day and stage proposals
./bin/daily-improvement-loop --since-days 1

# Backfill everything (first run), capped
./bin/daily-improvement-loop --all --max-sessions 500

# Just Claude, or just Codex
./bin/daily-improvement-loop --source claude --since-days 7

# Preview as JSON without writing the queue
./bin/daily-improvement-loop --since-days 1 --dry-run

# Include logs copied from other machines
./bin/daily-improvement-loop --home ~/.agent-logs/laptop --extra-home ~/.agent-logs/desktop

# Close a fixed target, inspect the registry, or reopen it
./bin/daily-improvement-loop --resolve "tool:edge-cli" --decision fixed --pr 71 --note "Merged and verified"
./bin/daily-improvement-loop --list-resolutions
./bin/daily-improvement-loop --unresolve "tool:edge-cli"

Flag	Meaning
--since-days N	Scan sessions modified within N days
--all	Backfill every discovered session
--max-sessions N	Keep only the most recent N after filtering
--source {all,claude,codex,hermes_profile_log}	Which local session source to read (default all)
--machine NAME	Stable fleet machine name embedded in sessions, proposals, and redacted bundle metadata
--collect-fleet	Merge the latest redacted bundle per machine from --fleet-inbox into one fleet review packet
--fleet-inbox PATH	Root containing <machine>/<run-id>.json redacted bundles for --collect-fleet
--include-seen	Re-emit proposals even if their key was seen before
--include-resolved	Debug bypass for the resolutions registry; seen-key filtering still applies unless --include-seen is also passed
--resolve ROUTE:TARGET	Record a target resolution; requires --decision, with optional --pr, --note, --by, and --resolved-at
--resolve-from PATH	Import a structured decisions.json handoff
--list-resolutions	Print the human-editable resolutions registry
--unresolve ROUTE:TARGET	Remove a target from the resolutions registry
--decision {fixed,wontfix,ignored}	Outcome recorded with --resolve
--resolved-at TIMESTAMP	Override the resolution watermark (ISO8601 UTC; default now)
--pr VALUE, --note TEXT, --by NAME	Add the fix reference, explanation, and recorder to a resolution
--full	Keep full, unredacted excerpts inline (local use only; do not share the output)
--dry-run	Print JSON, write nothing
--home PATH	Home dir containing .claude / .codex (default ~)
--extra-home PATH	Additional home dir containing .claude / .codex, useful for logs copied from another Mac
--output-root PATH	Where to write the queue (default ~/.agent-improvement)
--config PATH	JSON config overriding detector defaults (default ~/.agent-improvement/config.json; see "Configuration")

Output

~/.agent-improvement/
  resolutions.json               # durable route:target resolutions + watermarks
  state.json                     # last scan time + seen proposal keys
  session-index.jsonl            # one row per session with signals
  runs/<run-id>.json             # run metadata
  proposals/<run-id>/*.json      # one staged proposal per file
  review-packets/<run-id>.md     # the human-readable packet you review
  fleet-outbox/<machine>/*.json  # redacted proposals only; safe fleet handoff
  fleet-inbox/<machine>/*.json   # bundles collected by the review leader

Every proposal is marked manual_approval_required. Each one has a target, a route, the evidence line it came from, and a suggested action.

By default the excerpts are short and secret-shaped strings are masked, so the queue is safe to hand to a local review agent, and you still get the real command, the real error, and a path:line pointer to open the full transcript. Treat it as private by default anyway: absolute paths, project names, and CLI names are not redacted. Pass --full when you want longer, unredacted detail written inline instead. That output is local-only; do not share it.

Each run also records per-source parse statistics, and warns loudly (stderr, run metadata, and the packet itself) when a source's transcripts parse but yield zero tool calls — the signature of a transcript format change that would otherwise silently blind the loop.

Proposals whose target was already flagged in previous runs are marked as recurring ("also flagged in N previous run(s)") and sorted to the top of the packet — a target that keeps coming back is the strongest signal the loop produces.

Fleet mode

Run the scanner locally on every Mac so raw Claude, Codex, and Hermes session stores remain on the machine where they were created. Each ordinary scan writes one redacted proposal bundle under fleet-outbox/<machine>/; --full disables bundle output so unredacted evidence cannot enter the fleet handoff. Local review packets remain deduplicated deltas, while the fleet bundle is a snapshot of every currently active unresolved proposal so a latest-wins collector cannot forget older open work.

Choose one review leader. Copy only the bundle JSON files into its fleet-inbox/<machine>/ directories, then run:

agent-improvement-loop --collect-fleet \
  --fleet-inbox ~/.agent-improvement/fleet-inbox

The resulting *-fleet.md packet carries machine provenance on every proposal. The producer recursively sanitizes every string at bundle write time, and the collector repeats that sanitization instead of trusting a peer's redacted marker. Do not run multiple review leaders against the same aggregate queue.

Closing the loop / resolutions

state.json answers “have I staged this exact evidence before?” Resolutions answer the stronger question: “has this target already been reviewed and closed?” They live separately in ~/.agent-improvement/resolutions.json, so deleting or recovering a corrupt scan state cannot make completed work resurface.

The registry is a human-editable JSON object keyed by route:target:

{
  "tool:edge-cli": {
    "decision": "fixed",
    "resolved_at": "2026-07-16T22:37:19+00:00",
    "pr": "71",
    "note": "Merged and verified",
    "by": "reviewer"
  }
}

• fixed suppresses evidence at or before resolved_at, but evidence after that watermark is emitted as a regression and called out in its own review-packet section.
• wontfix and ignored suppress the target regardless of evidence time. --include-resolved bypasses this for debugging.
• --include-seen never bypasses a resolution. Use both debug flags when you intentionally want every resolved and previously seen proposal.
• Recording a resolution trims that target's pre-resolution recurrence history. A later regression starts a new recurrence era.

Each proposal carries latest_evidence_at, the maximum parsed event timestamp across its evidence. When a transcript format has no per-event timestamp, the loop uses the transcript file mtime, then a timestamp encoded in a Codex rollout filename. Runs report the count and target list for proposals suppressed as already resolved, so closure is visible rather than silent.

Fix workflows can import a batch handoff with --resolve-from decisions.json. The schema is:

{
  "schema_version": 1,
  "decisions": [
    {
      "proposal_id": "imp-c84cfef007115f5d918a",
      "target": "tool:edge-cli",
      "decision": "fixed",
      "resolved_at": "2026-07-16T22:37:19Z",
      "pr": "71",
      "note": "Merged and verified",
      "by": "reviewer"
    }
  ]
}

target and decision are required. resolved_at defaults to import time; pr, note, by, and proposal_id provide the review trail. Leave deferred or still-open proposals out of the file.

Machine-local learnings store

For the capture/fixloop half of the system, keep the live evidence store out of Dropbox and other cloud-synced multi-writer directories. Initialize one local store per machine:

./bin/learnings-store --root ~/.agents/learnings init --machine desktop

Migrate an existing store non-destructively:

./bin/learnings-store --root ~/.agents/learnings migrate \
  --machine desktop \
  --source ~/path/to/legacy-learnings

Peer bootstrap initializes a fresh local store by default. To import an existing store on the peer during its first bootstrap, set a peer-local absolute path or a path relative to its home directory:

AGENT_LEARNINGS_LEGACY_SOURCE=.legacy-agent-learnings \
  ./scripts/bootstrap_learnings_peer.sh laptop laptop

Every entry is machine-owned and collision-resistant:

~/.agents/learnings/entries/desktop/desktop--ERR-20260810-DESKTOPA7B8C9D0E1.md

A leader collects the other machine directories and materializes catalog.json, conflicts.json, and ACTIVE.md. Identical copies of a logical ID deduplicate in the catalog. Different content under the same logical ID is preserved per machine and reported as a conflict; no source file wins silently.

./scripts/collect_learnings_fleet.sh
./bin/learnings-store --root ~/.agents/learnings validate

On any machine, show the leader-published list of items that still need attention:

learnings status

Generate new IDs through the store so their suffix combines a stable machine token with high-entropy random data:

learnings new-id --type ERR

When a leader needs a fresh ID while splitting a collected peer copy, name that source machine explicitly: learnings new-id --type ERR --machine laptop.

Leader mutations are bound to the exact catalog generation shown by learnings status. Record a non-conflicted outcome without editing evidence:

learnings decide \
  --id ERR-20260810-DESKTOPA7B8C9D0E1 \
  --status resolved \
  --catalog-generation <generation-from-ACTIVE> \
  --by learn-loop \
  --note "Verified by the named test or live observation"

Resolve legacy ID collisions without changing evidence by binding a logical re-key to the copy's store-relative path and current SHA-256:

learnings rekey \
  --source-path entries/laptop/laptop--ERR-20260810-LAPTOPA7K2.md \
  --source-sha256 <sha256-from-catalog> \
  --new-id ERR-20260810-LAPTOPA7B8C9D0E1 \
  --catalog-generation <generation-from-ACTIVE> \
  --by human-review \
  --note "Reviewed same-ID collision" \
  --confirm-split

Compatible divergent copies can be acknowledged after reviewing every current copy. The command records their exact source paths and SHA-256 values; a changed, added, or deleted copy automatically invalidates the acknowledgement and reopens the conflict:

learnings acknowledge-conflict \
  --id ERR-20260810-DESKTOPA7B8C9D0E1 \
  --status resolved \
  --catalog-generation <generation-from-ACTIVE> \
  --by human-review \
  --note "All current copies describe the same verified incident" \
  --confirm-compatible

The legacy source remains untouched. Remove it only after every machine has run live capture, collection, and leader triage successfully for an agreed retention period.

From staged proposals to executed fixes

Staging is half the loop. The other half — a scheduled headless triage pass that promotes/resolves/queues, and an interactive learn-loop session that executes the queue with test gates and closes every entry's status — is documented in docs/CLOSING-THE-LOOP.md, with a ready-to-adapt skill (skills/learn-loop/), the triage prompt and runner shim (templates/), and a LaunchAgent example (examples/).

Open-source and private forks

This repo should stay public-safe. Put private detector catalogs, copied logs, real run outputs, receipts, and dogfood artifacts in a private fork or ignored local files such as private/ and .agent-improvement/.

See docs/OPEN_SOURCE.md for the public/private split and release checklist.

The routes (where a fix belongs)

The hard part is not noticing. It is deciding what kind of lesson you found. Proposals are routed to one of:

• tool — a CLI failed or got stuck on you in real use. Fix the tool, not the prompt. (By default it recognizes CLIs named *-cli; see "Configuration" below.) Four kinds of friction are caught: hard failures (explicit is_error or non-zero exit), hang/timeout signals where the command actually stalled without a clean result, retry-before-success — repeated use corroborated by a failure/hang or same-subcommand flag variation that looks like syntax guessing — and silent empty results where a data-returning call exits cleanly with [], {}, null, empty stdout, 0 rows, No results, or (empty) and the agent continues without acknowledging the emptiness. The proposal summary breaks down which kinds fired and how many retries it took. Recurring MCP failures and swallowed empty results route here too, grouped per server (mcp:<server>).
• skill_improvement — a reusable skill/command was used and the session later contained a correction. Patch the existing skill before creating a new one. Corrections that happened before the skill was invoked are treated as task context, not evidence that the skill failed.
• memory_context — a durable correction not tied to a skill, grouped per project (cwd) so the review question is "what line in this project's CLAUDE.md / AGENTS.md or runbook would have prevented this". Corrections are detected conservatively: explicitly corrective phrases count in normal-length messages, while ambient words like "do not" or "instead" only count in short reactive messages — a long task brief that says "do NOT change X" is an instruction, not a correction.
• backlog — repeated general-tool failures worth tracking but not urgent, plus recurring high-confidence swallowed empty results from general data-fetch commands. Both are staged with evidence instead of a vague note. Failures and silent-empty results inside subagent transcripts are excluded by default (exploratory subagents fail by design while probing).

And the most important non-route: nothing. One-off failures (a VPN was off) are discarded, not encoded. A system that cannot throw a lesson away turns into a haunted house of old warnings.

Run it on a schedule

A loop you have to remember to run is not a loop. Automate the scan, keep the approval manual.

With cron:

# 7am daily: mine yesterday's sessions and stage proposals for review
0 7 * * * cd ~/agent-improvement-loop && ./bin/daily-improvement-loop --since-days 1 >> ~/agent-review.log 2>&1

On macOS, launchd survives sleep/wake better than cron. Copy examples/com.example.agent-improvement-loop.plist into ~/Library/LaunchAgents/, edit the paths, and launchctl load it. The example runs twice a week with --since-days 4, which overlaps windows so a missed run cannot drop sessions.

Then, when you have a minute, hand the latest packet to an agent to triage:

latest=$(ls -t ~/.agent-improvement/review-packets/*.md | head -1)
# open "$latest" or paste it into Claude/Codex and ask:
# "Read this packet. For each proposal, tell me apply / defer / reject,
#  show the exact diff or command, and wait for my approval."

For a fuller reusable review prompt, see docs/REVIEWER_PROMPT.md.

Schedule the scan. Never schedule the changes.

The safety model is the point

• Redact before review (emails, phone numbers, tokens, keys, provider-specific key shapes, private-key blocks, long opaque strings). A corpus test asserts secret shapes never survive redaction.
• Store evidence references and short excerpts, not whole transcripts.
• Detect real tool usage, not prose mentions.
• Separate durable lessons from one-off incidents.
• Patch existing skills before creating new ones.
• Require human approval before anything changes.

The valuable thing is not autonomy. It is controlled compounding: yesterday's annoyance becomes tomorrow's default, and no single bad session gets enshrined as a permanent rule.

Configuration

Detector defaults can be overridden without editing code. Put a JSON file at ~/.agent-improvement/config.json (or pass --config PATH); see examples/config.example.json. Recognized keys:

Key	Meaning
tracked_cli_suffix	Command-name suffix the tool route tracks (default -cli)
extra_scaffold_markers	Strings that mark injected/scaffolded user messages your own automation inserts
extra_redaction_patterns	[regex, replacement] pairs appended to the built-in secret masks
extra_backlog_ignore	Executables the backlog route should never blame
extra_remote_command_wrappers	Wrappers (like your ssh helper) whose quoted commands should be scanned for tracked CLIs
include_subagent_failures	Count failure/silent-empty signals from subagent transcripts (default false)
detect_silent_empty	Detect swallowed structurally empty data results (default true)
silent_empty_fetch_verbs	Extra command/subcommand verbs that clearly intend to return data
silent_empty_ignore	Executables whose empty output should always be treated as normal success

General tool failures are still captured by the backlog route regardless of the tracked suffix.

Limitations — what it won't catch

This loop mines interactive agent transcripts. It catches breakages that surfaced in the session: a non-zero exit, an error flag, a hang, a retry, a correction you typed, or the narrow case where a clearly data-returning call produced a structurally empty result and the agent kept going.

So it is blind to failures that exit clean and never get corrected:

• Silently wrong output. A command exits 0 but returns stale, degraded, or plausible-but-wrong data — a drifted payload shape, a cache that never refreshed. The detector catches only structurally empty results ([], {}, null, empty stdout, 0 rows, No results, or (empty)) that were swallowed before another step. Non-empty stale or wrong data still has no in-session failure signal to mine.
• Out-of-band failures. A cron that fires with stale data, a webhook whose contract drifted in production — these never appear in an agent transcript at all, so they are outside the corpus, not merely hard to find.

The rest of this class needs a different loop: one that watches outputs and production — schema assertions, freshness checks, output diffing — not session logs. That is a sibling tool, not a knob on this one.

Roadmap

• Repeated-command-chain detection, not just executable names.
• Detect manual parsing where a --json path exists.
• A --review mode that reads the latest packet and walks the apply / defer / reject decisions with you.
• Extract embedded shell commands from code-runtime tool calls for the backlog route (tracked CLIs inside code literals are already detected; general executables are not).

Tests

python3 scripts/test_daily_improvement_loop.py

License

MIT. See LICENSE.

Built by Little Might.