Enhance OpenShell's OCSF telemetry and export it off-box via OTLP

[mattj-monad]

User Story

These roles want the same thing: a complete, structured record of what happens inside a sandbox and in the control plane that governs it, exportable off the box as one stream.

As a security or compliance owner running OpenShell for autonomous agents, I want the sandbox's security events and every state-changing control-plane operation and authentication decision to reach me via OTLP as one durable, standard-schema stream, so I can answer who did what and when, spot boundary violations, and keep the record on my own terms instead of relying on logs that never leave the process.

As an agent developer or platform engineer running agents on OpenShell, I want the things my agents do at runtime (the commands they run, the resources they reach for, and the denials and failures they hit) exported as a structured stream, so I can find where agents get stuck or fail and improve their prompts, policies, and tooling from real data instead of guessing. The same record an auditor reads is also a dataset for improving the agents.

As a platform operator, I want an off-box record of control-plane changes and background reconciliation, and I want to know when that record is incomplete, so I can reconstruct incidents without depending on logs that disappear with the process.

As a security engineer setting up OpenShell's observability, I want the telemetry to be complete rather than partial. Every state-changing action should produce a record, including the background rotation, token minting, and reconciliation that emit nothing today. What exists is a good start, but until those gaps are closed, a missing event doesn't tell me nothing happened, only that we didn't capture it.

Problem Statement

OpenShell produces security-relevant telemetry across its whole surface, but none of it reaches an external system as a coherent, exportable stream.

• Inside the sandbox, the supervisor already models network, process, filesystem, policy, and SSH events in OCSF, but that telemetry has no path off the box. It reaches the gateway's log stream and stops there.
• In the gateway control plane, where workspaces, providers, credentials, sandboxes, ssh sessions, policy, tokens, and inference routes are created, changed, and deleted, and where every request is authenticated, there is no durable structured record of who did what, when, or with what outcome.
• Coverage is incomplete even where a record could exist. Background credential rotation, token minting, reconciliation, and inference-route changes alter state with no event at all, so some actions are invisible even on the box.

So the sandbox side is stranded on the box, the control-plane side is largely unrecorded and incomplete where it does exist, and nothing carries the two off the box together with any delivery guarantee.

Impact / Why This Matters

An operator who needs to answer "who deleted this sandbox," "was this credential rotated by a person or a job," "is this a credential-stuffing attack or an IdP outage," or "what did this sandbox do before it was flagged" has no data to work from. The control-plane events were never recorded, and the sandbox events can't be exported, aggregated, or replayed. There is no way to export OpenShell telemetry off the box, no alert when a sandbox principal reaches outside its boundary, and no record of credential issuance or credential access. The operations that emit nothing today can't be reconstructed at all: a background rotation or a token mint leaves no trace to investigate. And tailing logs on the box, the only option that exists, is not retainable, not attributable, not machine-consumable, and drops data under load without telling you.

For teams that build and run the agents, the gap is worse, because agent behavior is hard to see and harder to improve. When an agent underperforms, loops, or stalls, there is no durable record of what it was trying to do: whether it hit a policy wall, reached for a tool that isn't installed, or retried the same failing command. You can't tell churn from progress, or a missing capability from a reasoning problem, and there's no history to tune against. The signal that would answer these questions (the commands agents run, the operations they're denied, and where they repeat) exists only briefly on the box. So improving an agent today is mostly guesswork.

Proposed Design

From the operator's perspective:

1. Enabling log export sends OpenShell's telemetry to any OCSF-aware OTLP collector, on the same endpoint as traces: the sandbox OCSF events the supervisor already produces, plus the new control-plane audit records. They arrive together, correlated by sandbox.
2. Every state-changing control-plane operation (resource create/update/delete, configuration and policy changes, credential and token issuance, inference-route changes) produces one record in the same OCSF schema the sandbox already uses. Each names the authenticated principal (taken from the session, not from the request), the outcome (success or failure, including no-op mutations), and a correlation id. Background changes with no request, such as scheduled credential rotation, reconciliation, and startup, are attributed to a system actor rather than going unrecorded.
3. Authentication outcomes at the boundary are recorded. Failures carry the mechanism, a low-cardinality reason, and the peer address, but never the credential. Successful authentications are available behind a toggle for deployments that want a full ledger. A sandbox principal reaching for a sandbox it doesn't own, or attempting an admin operation, raises an alert-grade finding alongside the ordinary denial.
4. Delivery is accountable on both hops, sandbox-to-gateway and gateway-to-collector. When records have to be dropped under backpressure or a collector outage, the stream carries a countable gap marker instead of losing them silently, so an operator can see whether the record is complete and where it isn't.
5. A master toggle turns the trail on (the default) or off. Finer toggles control volume and sensitivity, such as whether command lines or setting values are captured. Credential values, tokens, ssh bearer material, and exec environment and stdin never appear in a record. Turning the trail off does not change any of the operations themselves.
6. The exported stream is usable for analysis, not just archival. The sandbox's network and policy decisions and the gateway's exec, forward, and session events land in the same pipeline, so a team can run the combined stream through their own tooling and see, per agent run, what an agent executed, what it reached for, what it was denied, and where it repeated itself. That is the input a team needs to tune prompts, policy, and tooling.
7. The goal is comprehensive coverage, and staying comprehensive. Any security-relevant action on the box should produce a structured, exportable event, not only the operations listed above but the background work, token issuance, reconciliation, and route changes that produce nothing today. In principle a team should be able to get every meaningful event off the box. Where an operation is silent, that's a gap to close rather than accept, and the project should keep a rule that a new state-changing operation ships with an event. The records are also richer than a plain log line: each carries the actor, the outcome, a correlation id, and a severity, so downstream tools can filter and join on them.

Using OCSF, already OpenShell's schema for sandbox events, is the one constraint we'd hold to, so both sides land in the same pipeline without a custom parser. This proposal does not add new sandbox event types; those already exist. It gives them a path off-box, adds the control-plane events, and makes delivery accountable. Record shapes, class choices, toggle names, transport, and delivery internals are left to whoever implements it.

Acceptance Criteria

• Enabling export sends both the existing sandbox OCSF events and the new control-plane events to an OCSF-aware OTLP collector, on the same endpoint and resource identity as traces.
• Every state-changing gateway operation produces one audit record with the authenticated principal (or a system actor for background changes), the outcome, and a correlation id.
• No-op mutations, such as deleting something already gone, are recorded as failures rather than silent successes.
• Authentication failures are recorded with mechanism, reason category, and peer address, and never the credential. Successes are available behind an opt-in toggle.
• A sandbox principal crossing its boundary produces an alert-grade finding in addition to the denial.
• One master toggle disables all audit emission. Finer toggles control command-line and setting-value capture, and credential-shaped values are redacted regardless.
• No secret material (credential values, minted tokens, ssh bearer tokens, exec environment or stdin) appears in any exported record.
• Delivery is accountable on both the sandbox-to-gateway and gateway-to-collector hops: dropped records show up as a countable gap rather than disappearing.
• The commands agents run and the sandbox's network and policy denials export through the same pipeline, so an agent's runtime behavior can be reconstructed off-box.
• Coverage is comprehensive: state-changing operations, including background and system-driven ones, produce an event rather than changing state silently, and the project keeps a rule that new state-changing operations emit one.
• The behavior, toggles, and record catalog are documented.

Alternatives Considered

• A custom audit/telemetry format. It would need a bespoke parser downstream and would diverge from the OCSF schema OpenShell already emits for sandbox events, splitting the pipeline instead of unifying it.
• A third-party sidecar or proxy in front of the gateway. It can see requests but not the authenticated principal, the store outcome, or internal transitions like background rotation and reconciliation. It also can't see inside the sandbox or tell a no-op from a real change.
• Exporting the sandbox telemetry only and leaving the control plane dark. It answers "what did the sandbox do" but never "who changed the policy, minted the token, or deleted the sandbox," and gives no auth or boundary signal.
• Metrics and counters only. Aggregates can't answer who-did-what-when or support forensic reconstruction.
• On-box logs with no export. The status quo, which fails the retention, attribution, completeness, and export needs above.

Agent Investigation

Feasibility was checked against the codebase, and an implementation is ready, pending acceptance of FR

• OpenShell already models sandbox events in OCSF v1.7.0 (crates/openshell-ocsf, emitted by the sandbox supervisor) and already has an OTLP trace path ([openshell.gateway.otlp]), so this extends what's there rather than adding a new stack. It reuses the existing OCSF library across both the sandbox and the gateway.

• The sandbox's push path (crates/openshell-supervisor-process) is where its telemetry leaves for the gateway. The gateway's state-changing RPCs are concentrated in crates/openshell-server/src/grpc/, with the authenticated principal already available at the boundary. Both are good places to instrument.

• The implementation on the fork adds off-box OTLP log export with accountable delivery on both hops; the control-plane OCSF audit event classes and instrumentation of the ~34 state-changing handlers; and an efficient, loss-accounted push format on the sandbox side. It's governed by a single [openshell.gateway.audit] toggle and touches the shared OCSF and OTLP crates, the sandbox supervisor, and the gateway. It's covered by unit tests and a live run through an OTLP collector, where every event class arrived intact, no secrets appeared in any indexed record, and no records were lost without being accounted. It's offered as the PR for this issue once the design is accepted.

This issue addresses some of #1055 but focuses more on the audit log instead of tracing.

Checklist

• I've reviewed existing issues and the architecture docs
• This is a design proposal, not a "please build this" request