This local, single-user harness runs untrusted repository code behind a stronger boundary than a host subprocess. The trusted computing base includes the host application, bundle author, base image or Dockerfile, Docker daemon, evaluator runner, hidden tests, and gold patch. Repository code, solver output, candidate files, test processes, provider responses, and model messages are untrusted.
Docker isolates task execution, but the Docker daemon and host kernel remain trusted. This is not a hostile multi-tenant VM boundary. Deployments exposed to container escape, kernel denial-of-service, or mutually distrustful tenants need a dedicated remote worker or microVM.
Every solver and evaluator session uses:
- a fresh named volume initialized from the exact, clean tracked commit;
- a read-only root filesystem and bounded tmpfs;
- a validated numeric identity with non-zero UID and GID;
- a workspace path disjoint from harness-owned temporary, evaluator, initialization, and capture mounts;
--network none;- all Linux capabilities dropped and
no-new-privileges; - CPU, memory, PID, command-time, output, candidate-file, and patch-size limits;
- no host bind mounts, Docker socket, device, privileged mode, or provider credential environment;
- deterministic container and volume cleanup.
Host process I/O is multiplexed without reader threads and drained continuously
into constant-space head and tail buffers. Output is bounded during capture,
not truncated after being written to an unbounded temporary file. Post-exit
draining is also bounded so a detached descendant cannot block the driver.
Solver-facing stdout and stderr use the bundle's output_bytes limit even when
trusted capture has a larger host allowance. Candidate patches are read through
patch_bytes + 1; a CLI timeout may reduce the manifest's solver limit but
cannot increase it.
The prepared image is inspected once. For a prebuilt image, the declared
registry manifest digest must appear in Docker's RepoDigests; that verified
manifest is the portable identity persisted by the harness. Every later Docker
invocation uses the separately inspected daemon-local image ID rather than the
mutable tag. Classic Docker and containerd-backed image stores can expose
different .Id values for the same manifest, so the local ID never replaces
the portable identity. Dockerfile builds use the inspected image ID for both
roles on that daemon and are persisted as local_build / non-portable. They do
not provide a resolvable identity on another machine. Repository identity is
read from git rev-parse HEAD inside the image;
image-authored metadata is not accepted as commit authority.
The initialization container is the only privileged phase. It is networkless,
read-only, and receives only CHOWN, DAC_OVERRIDE, and FOWNER so it can copy
the verified tree into a named volume and change ownership. Trusted capture
mounts that volume read-only. Evaluator assets live in a separate tmpfs and
become root-owned and read-only before candidate tests execute.
Hidden-test isolation
The base image and reusable cache never contain bundle evaluation assets.
Solver prompts contain only description.md; the bundle directory is not
mounted. After solver destruction, the evaluator applies the captured
candidate to a new clean volume, applies the hidden test patch, and writes the
runner/lists into evaluator-only tmpfs.
Candidate code can inspect tests after evaluation starts because both execute within the evaluator boundary. Hidden inputs remain secret through solver completion; they are not protected from malicious candidate code after materialization. The evaluator provides authoritative measurement, not ongoing test secrecy.
The runner and exact test lists are installed as root-owned, read-only assets, and the harness accepts only the versioned report with the exact declared test IDs. Process exit status and candidate prose never decide the outcome.
This local Docker design does not make the report cryptographically independent of adversarial candidate code. Candidate code and the repository-specific test runner execute inside the same evaluator container and numeric runtime identity; after hidden materialization, malicious code could interfere with its process or writable result directory. The trusted computing base therefore includes the bundle author and evaluator runner, and the harness is intended for task development and honest-but-buggy candidates. A hostile benchmark service would need a separate trusted runner boundary—such as a supervisor/worker protocol or dedicated VM—that receives only authenticated test results from the candidate environment. Root-owned files alone are not claimed as that stronger boundary.
The harness does not rely on a solver-reported diff. It pauses the solver container and uses:
- immutable Git objects from the verified image;
- a separate temporary index and object directory;
- a read-only candidate-volume mount;
- disabled Git hooks, system/global config, attributes, external diff, and text conversion;
- allowed Git modes, file-count limits, and binary patch-size limits.
The resulting full-index binary patch is checksummed before it is applied in a fresh evaluator. Candidate patch paths supplied to the stub must be regular, non-symlink files and are size-bounded before use.
Models run host-side because authentication and provider networking do not belong in the task container. The flat adapter only serializes three strict function schemas. It rejects parallel calls, repeated call IDs, malformed arguments, excess turns, tool calls, tokens, or elapsed time. Each provider request receives the remaining solver deadline, while the Docker tool surface shares the same wall-time budget.
The Codex adapter requires an API key and a CLI with the expected secure flags.
It pipes OPENAI_API_KEY to codex login --with-api-key inside an ephemeral
CODEX_HOME; the secret is never placed in argv or the child environment, and
the temporary auth file is destroyed with the solver process. Codex then starts
with that empty HOME, ignores user configuration and repository
rules, uses an empty read-only working directory, disables shell/web/apps/
hooks/goals/plugins/browser/computer/memory/subagent features, and requires a
single run-scoped MCP server. That server uses an ephemeral Unix socket and
capability token and only publishes run_command, read_file, and
apply_patch. JSONL output is audited fail-closed: unknown item types, other
servers/tools, malformed events, or incomplete turns fail the solver. Secure-
flag preflight and execution spend one monotonic solver deadline; preflight
cannot add a second timeout allowance. The MCP transport uses that remaining
deadline rather than a fixed timeout, leaving Docker command limits and the
outer solver deadline as the authoritative bounds.
Codex can report missing local metadata for an explicitly selected model as an
item-level error even when the turn completes successfully. The audit accepts
only that exact, nonfatal message shape, records a redacted warning category,
and continues to reject every other top-level or item-level error. In
particular, an app-server stream-lag notice remains fatal because it says audit
events were dropped. This compatibility rule does not expand the server, tool,
command, or filesystem allowlists.
Flat-provider credentials remain in the sanitized host client. Codex credentials cross only the login process's stdin and temporary auth file. Neither is written to SQLite, artifacts, bundle files, Docker configuration, prompts, or task processes.
Structured arguments, configurations, summaries, event data, test excerpts,
reports, and debug exports pass through key/string redaction. Sensitive key
names, bearer values, URL credentials, and the current home path are replaced.
Process output is bounded before persistence. Artifact creation uses a fresh
UUID-owned private directory, atomic writes, 0600 files, and SHA-256
registration. A failed registration deletes only those new bytes and cannot
replace an earlier artifact. Downloads recheck path containment, regular-file
status, size, and checksum.
SQLite uses parameterized queries, foreign keys, explicit status constraints, and short transactions. Dynamic SQL only assembles fixed clauses; values stay parameterized. Bundle and artifact paths reject absolute traversal, NULs, backslashes, and symlink aliases.
The server binds only to 127.0.0.1, accepts only explicit loopback Host and
Origin values, and returns a restrictive content policy, frame denial,
no-referrer, and content-type protections. It has no remote deployment mode,
authentication, raw terminal, file browser, or arbitrary RPC endpoint. The UI
can request only validation, registered solver runs, cooperative cancellation,
bounded reads, and registered artifact downloads. Do not expose the port via a
public proxy without adding authentication, CSRF protection, tenant isolation,
and a stronger execution boundary.
- A Docker/kernel vulnerability can cross the container boundary.
- CPU/memory/PID limits do not bound all host I/O pressure or Docker daemon resource usage.
- Build and image-pull stages use network and trust their declared sources.
- A trusted base image can itself be malicious.
- Cooperative cancellation is observed between model/tool boundaries. A provider SDK request already in flight completes or reaches its bounded remaining timeout; the production Codex process group is terminated immediately when polled.
- Architecture emulation changes performance and can reveal platform-specific failures even with identical functional inputs.