Task Bundle CLI

Task Bundle CLI turns coding issues into reproducible, commit-pinned benchmarks. It runs interchangeable solvers in disposable Docker workspaces, evaluates their patches in fresh containers, and records the results.

The stub, Responses API (flat), and Codex CLI (codex) adapters share one Docker-backed lifecycle. The browser UI is a debug driver for that lifecycle.

Contents

• Submission contents
• Documentation
• Project structure
• Quick start
• What happens in one run
• Included benchmark
• Five-command interface
• Create a bundle
• Solver design
• Debug UI
• Local evidence
• Verification

Submission contents

Assignment deliverable	Location
CLI code and usage instructions	src/task_bundle/ and this README
Validatable example task bundle	examples/qutebrowser-qt-warning-filter/
Evaluation artifact from task run	result.json
Design notes and key tradeoffs	docs/DESIGN.md

Documentation

The README is the operator path. Deeper contracts and rationale live in four focused documents:

Document	Covers
docs/DESIGN.md	Concise design rationale, solver choices, isolation, evidence, and key tradeoffs
docs/DECISIONS.md	Detailed Docker, evaluation, solver, capture, persistence, and stability ADRs
docs/BUNDLE_SPEC.md	Bundle schema, protected inputs, evaluator protocol, and expected results
docs/SECURITY.md	Trust model, container isolation, credential handling, and residual risks

Project structure

src/task_bundle/
├── domain/          Typed bundle, lifecycle, evaluation, and solver contracts
├── application/     Harness orchestration, leases, state, and sandbox boundary
├── runtime/         Exact Git source, Docker sessions, capture, and evaluation
├── solvers/         Stub, flat Responses API, and Codex adapters + three tools
├── storage/         SQLite schema, transactions, indexes, and query layer
├── observability/   Redaction at logs and artifact boundaries
├── cli/             Five-command interface and human/JSON rendering
└── web/             Loopback API, job driver, and packaged debug UI

Commands and HTTP routes stay thin. application owns the lifecycle, while the CLI and UI render the same durable records.

Quick start

Requirements: Python 3.12, uv, Git, and Docker with Linux container support. An OPENAI_API_KEY is optional and used only by the flat and codex solvers.

Install the CLI once from the repository root:

uv tool install --force --reinstall .
task --version

Normal CLI use does not require uv run. If task is not on PATH, run uv tool update-shell once and open a new shell.

Start Docker, then exercise the complete public example:

task init examples/qutebrowser-qt-warning-filter
task validate examples/qutebrowser-qt-warning-filter
task run examples/qutebrowser-qt-warning-filter \
  --solver stub \
  --candidate-patch examples/qutebrowser-qt-warning-filter/patches/gold.patch

task show
task ui

Expected result: validation classifies three fresh samples as stable, the supplied candidate resolves all 56 tests, and the UI opens at http://127.0.0.1:8765. Use Ctrl+C to stop it.

What happens in one run

task validate repeats fresh baseline and gold evaluation three times by default, then classifies the bundle as stable, flaky, or expectation-invalid. task run follows one harness-owned sequence:

1. Resolve the exact Git commit and prepare the declared Docker image.
2. Run a fresh baseline guardrail and stop if its expectations are wrong.
3. Give the solver only description.md, the clean repository, and three bounded capabilities.
4. Freeze the solver worktree and capture its complete checksummed patch.
5. Destroy the solver container and writable volume.
6. Apply the candidate in a fresh evaluator, inject protected assets, and compare the exact pass-to-pass and fail-to-pass results.
7. Persist the outcome, test matrix, events, runtime identity, and artifacts.

The solver never receives the gold patch, hidden test patch, evaluator runner, test lists, host filesystem, Docker socket, provider credentials, or network access. Read the full lifecycle in docs/DESIGN.md and its limits in docs/SECURITY.md.

Included benchmark

The checked-in example is a public SWE-bench Pro qutebrowser task pinned to a specific commit, linux/amd64 platform, and registry image digest. Its expected matrix is:

Phase	Pass-to-pass	Fail-to-pass
Baseline	52 passed	4 failed
Gold or supplied candidate	52 passed	4 passed

The checked-in result.json is the generated results.json member from task show COMMAND_ID --export. It records the baseline and candidate test rows, runtime identity, run status, and artifact checksums from task run. ARM64 hosts execute this pinned example through Docker's linux/amd64 emulation.

Five-command interface

Command	Purpose
task init BUNDLE	Scaffold a bundle or prepare its exact source and runtime
task validate BUNDLE	Repeat baseline and gold evaluation and classify stability
task run BUNDLE	Guard the baseline, run a solver, capture its patch, and evaluate it
task show [COMMAND_ID]	List history or inspect, follow, and export one command
task ui	Start the loopback-only debug UI

Add --json to machine-facing commands. Use task show COMMAND_ID --follow for live events or task show COMMAND_ID --export debug.zip for a checksummed debug archive. Exit codes are stable: 0 success, 2 invalid input, 3 unresolved validation, 4 solver or patch failure, 5 infrastructure failure, and 130 cancellation.

Create a bundle

Scaffold without overwriting an existing directory:

task init tasks/my-task \
  --repo https://github.com/example/project.git \
  --commit 0123456789abcdef0123456789abcdef01234567

A complete bundle has this shape:

my-task/
├── task.json
├── description.md
├── patches/
│   └── gold.patch
├── evaluation/
│   ├── test.patch
│   ├── pass_to_pass.txt
│   ├── fail_to_pass.txt
│   └── run-tests
└── runtime/
    └── Dockerfile

evaluation/test.patch is optional when the runtime already contains the authoritative tests. A pinned prebuilt image can replace runtime/Dockerfile. After filling in the bundle, run task init tasks/my-task to prepare its runtime and task validate tasks/my-task to prove its expectations.

The bundle owns repository-specific setup, test IDs, and runner behavior. That keeps the harness independent of language and test framework. See docs/BUNDLE_SPEC.md for the manifest, evaluator JSON protocol, description guidance, and leakage rules.

Solver design

Every solver receives the same three repository capabilities:

Capability	Purpose	Boundary
read_file	Read one repository-relative UTF-8 file	Contained path and bounded output
run_command	Search, build, test, or run repository tooling	Container-only shell, no network, bounded time and output
apply_patch	Apply one Git-compatible edit	Strict input and patch-size limit

The model composes these primitives using the repository's own tools, whether the project uses Python, TypeScript, Go, Rust, or a native build system. The harness does not trust tool history or solver prose as the final change. It captures the complete frozen worktree independently and evaluates that patch in a fresh container.

Adapter	Code-writing mechanism	Authentication	Intended use
stub	No-op or explicitly supplied patch	None	Deterministic harness verification or external solver output
flat	Direct OpenAI Responses API loop	Host API key	Small, inspectable model and tool loop
codex	Non-interactive host Codex CLI	Host API key	Existing coding-agent behavior behind the same boundary

task run tasks/my-task --solver flat --model gpt-5.6 --effort high
task run tasks/my-task --solver codex --model gpt-5.6 --effort high
task run tasks/my-task --solver flat --repetitions 5 --jobs 2

Independent repetitions measure solver variance. Concurrency changes throughput without sharing writable state. The reasoning and tradeoffs are documented in docs/DECISIONS.md.

Debug UI

task ui starts the same application commands and renders the same SQLite records and checksummed artifacts as the CLI. It helps a task creator answer: did the run finish, which tests changed, what patch was captured, and what evidence explains the outcome? It has no editor, chat, terminal, or separate evaluation logic.

Run overview. Start a command and follow its outcome, runtime, phases, and append-only activity.

Test matrix. Compare the baseline guardrail with the candidate and filter to the tests that matter.

Captured patch. Inspect and download the same trusted candidate artifact available through task show.

Local evidence

State defaults to ~/.task-bundle and can be moved with --home or TASK_BUNDLE_HOME. SQLite stores commands, runs, phases, exact test rows, and events. Private checksummed files store reports, patches, logs, runtime metadata, test results, and validation-stability summaries. Containers, volumes, and writable solver workspaces are disposable.

Verification

Run the submission gate from a locked development environment:

uv sync --frozen --all-groups
make verify-assignment

This runs formatting, linting, architecture checks, strict types, unit and contract tests, coverage, package installation, real Docker isolation, browser flows, and the pinned example. The optional credentialed Flat and Codex solver-boundary canaries are separate:

make verify-live-solver

Pull-request and main CI never receive provider credentials or invoke an LLM.