feat(billing): track E2B and Daytona Function sandbox usage - #7184
feat(billing): track E2B and Daytona Function sandbox usage#7184BillLeoutsakosvl346 wants to merge 3 commits into
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub. |
Greptile SummaryThe PR meters remote E2B and Daytona Function sandboxes and propagates chargeable user-code failure costs through Function, Agent, Pi, retry, trace, and workflow-cost accounting.
Confidence Score: 4/5The PR is not yet safe to merge because workflow-scoped remote Function calls with missing workflow identifiers can still create unmetered E2B or Daytona sandboxes. The displayed reply, whose author name is blank in the supplied thread data, claims incomplete workflow attribution now fails closed, but current HEAD still derives meterUsage solely from the optional workflowId and forwards false to all remote execution paths without rejecting incomplete attribution. Files Needing Attention: apps/sim/lib/function-execution/execute-request.ts
|
| Filename | Overview |
|---|---|
| apps/sim/lib/function-execution/execute-request.ts | Introduces the central remote-sandbox metering and failure-cost flow, but still gates metering on optional workflowId without enforcing complete workflow attribution. |
| apps/sim/lib/execution/remote-sandbox/index.ts | Coordinates sandbox lifecycle and attaches provider-priced usage to chargeable terminal results. |
| apps/sim/executor/execution/block-executor.ts | Aggregates trusted Function costs across retries and preserves them in successful or failed block logs. |
| apps/sim/executor/handlers/function/function-handler.ts | Transfers trusted costs from Function tool results to returned outputs and thrown errors. |
| apps/sim/providers/cost-policy.ts | Incorporates Function sandbox tool costs into provider cost accounting. |
| apps/sim/lib/billing/sandbox-pricing.ts | Defines frozen provider resource pricing, TTL capping, multiplier application, and billed-cost rounding. |
Flowchart
%%{init: {'theme': 'neutral'}}%%
flowchart TD
A[Function, Agent, or Pi call] --> B{Remote E2B or Daytona sandbox?}
B -- No --> C[Existing unmetered execution]
B -- Yes --> D{workflowId present and standard tool?}
D -- No --> E[Remote execution without metering]
D -- Yes --> F[Snapshot sandbox pricing]
F --> G[Execute code or shell]
G --> H{Terminal outcome}
H -- User-code exception or nonzero exit --> I[Attach trusted cost]
H -- Success --> I
H -- Timeout, cancellation, or provider failure --> J[No user charge]
I --> K[Aggregate retries and tool consumers]
K --> L[Trace cost and workflow usage ledger]
Reviews (3): Last reviewed commit: "fix(billing): charge function user-code ..." | Re-trigger Greptile
There was a problem hiding this comment.
All reported issues were addressed across 17 files
Reply with feedback, questions, or to request a fix.
Re-trigger cubic
|
@greptile-apps @cubic-dev-ai please re-review the latest changes on this PR. The prior findings have been addressed or resolved with the documented scope decision, and the branch is updated with current staging. |
@BillLeoutsakosvl346 I have started the AI code review. It will take a few minutes to complete. |
There was a problem hiding this comment.
All reported issues were addressed across 22 files
Reply with feedback, questions, or to request a fix.
Re-trigger cubic
|
@cubic-dev-ai please re-review the latest changes in dd7aa1c. The three findings from the prior run have been addressed with focused E2B/Daytona, outbox, and Function response regressions, and all threads are resolved. |
@BillLeoutsakosvl346 I have started the AI code review. It will take a few minutes to complete. |
There was a problem hiding this comment.
All reported issues were addressed across 22 files
Reply with feedback, questions, or to request a fix.
Re-trigger cubic
|
@cubic-dev-ai please re-review the latest changes in a3a6dd2. Both latest sandbox billing findings are fixed, focused regressions pass across E2B and Daytona, and the full requested validation set is green. |
@BillLeoutsakosvl346 I have started the AI code review. It will take a few minutes to complete. |
|
Implemented the sandbox durability follow-up in Review fixes included:
Focused validation:
Policy decisions remain unchanged: unresolved provider identities are no-charge; Daytona account allowances/credits are ignored in favor of configured list-rate pricing; Mothership scope is unchanged. Out-of-scope follow-ups identified for separate work:
@cubic-dev-ai please re-review the latest commit, especially the durable pre-create intent, correlation recovery, terminal evidence precedence, and billing-period lock coordination. |
@BillLeoutsakosvl346 I have started the AI code review. It will take a few minutes to complete. |
There was a problem hiding this comment.
All reported issues were addressed across 25 files
You've manually re-run cubic several times on this PR. Each manual re-review checks the full PR again and counts toward your usage quota. To preserve your usage limits, we recommend letting cubic automatically review new commits.
Tip: cubic can generate docs of your entire codebase and keep them up to date. Try it here.
Fix all with cubic | Re-trigger cubic
c88112e to
e48e74f
Compare
There was a problem hiding this comment.
All reported issues were addressed across 25 files
Tip: cubic can generate docs of your entire codebase and keep them up to date. Try it here.
Fix all with cubic | Re-trigger cubic
e48e74f to
3125336
Compare
|
Implemented the complete sandbox lifecycle review follow-up in 3125336, rebased onto staging at b094b4d. Finding-by-finding:
Validation after the final staging rebase:
Policy remains unchanged: an unresolved provider identity is no-charge; Daytona allowances/credits are ignored; pricing and multiplier behavior are unchanged; Mothership, local JavaScript, and custom Function-tool scope are unchanged. The broader long-workflow settlement and internal terminal-log partial-total concerns remain separate follow-up work. @cubic-dev-ai please re-review the new head 3125336. |
@BillLeoutsakosvl346 I have started the AI code review. It will take a few minutes to complete. |
There was a problem hiding this comment.
All reported issues were addressed across 29 files
You've manually re-run cubic several times on this PR. Each manual re-review checks the full PR again and counts toward your usage quota. To preserve your usage limits, we recommend letting cubic automatically review new commits.
Tip: cubic can generate docs of your entire codebase and keep them up to date. Try it here.
Fix all with cubic | Re-trigger cubic
d1413fb to
204e028
Compare
204e028 to
843a194
Compare
|
Final delivery head is The only post-review-request adjustment was replacing a banned |
843a194 to
5b4cb21
Compare
|
Pushed This is a focused correction to the simplified v1 implementation:
Daytona continues to use the full provisioned 10 GiB disk at list rate. Provider credits, free allowances, and negotiated discounts remain intentionally excluded. The known-provider-ID guarantee, accepted v1 limitations, schemas, APIs, dependencies, and Mothership/local/custom-tool scope are unchanged. Validation passed: 560 tests across 10 focused/adjacent suites, type-check, lint, API validation, utilities audit, tool-request boundaries, monorepo boundaries, and @cubic-dev-ai please review the focused follow-up commit |
@BillLeoutsakosvl346 I have started the AI code review. It will take a few minutes to complete. |
There was a problem hiding this comment.
1 issue found across 27 files
Confidence score: 4/5
- In
apps/sim/lib/logs/execution/logger.test.ts,costTotalUpdatefilters SQL template values to strings/numbers while the assertions expectworkflowExecutionLogs.costTotalinparams, creating a test expectation mismatch; align the helper or assertions with the actual parameter representation.
Prompt for AI agents (unresolved issues)
Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.
<file name="apps/sim/lib/logs/execution/logger.test.ts">
<violation number="1" location="apps/sim/lib/logs/execution/logger.test.ts:1050">
P2: The `costTotalUpdate` helper filters the sql template's `values` to string/number, but the assertions expect `'workflowExecutionLogs.costTotal'` to appear in `params`. The interpolated `workflowExecutionLogs.costTotal` in the `sql\`GREATEST(COALESCE(${...}, 0), ...)\`` template is a drizzle Column object (its empty slot is visible in the asserted `text`), so the filter removes it and the actual `params` are only `['1.705']`/`['1.205']`. Either the assertions never match (failing tests) or, if drizzle happened to inline the column, the params contract the helper claims is wrong. Align the helper and assertions: assert the column reference is carried as a non-string `value` (or its rendered SQL text) rather than as a string `param`.</violation>
</file>
You've manually re-run cubic several times on this PR. Each manual re-review checks the full PR again and counts toward your usage quota. To preserve your usage limits, we recommend letting cubic automatically review new commits.
Tip: cubic can generate docs of your entire codebase and keep them up to date. Try it here.
Fix all with cubic | Re-trigger cubic
|
@BillLeoutsakosvl346 we should probably put this under a new usage log source? Is that already the case? What is this under right now? |
Right now it’s under workflow as a tool called Code sandbox, with E2B/Daytona details in the metadata. That keeps it included in the workflow’s total cost. We could add a separate sandbox source, but that would require a migration and some billing/reporting updates. |
5b4cb21 to
6fc6977
Compare
|
Rebuilt this PR from current staging around the existing trace-cost pipeline. What changed:
Scope is now 20 files, +602/-50, with no schema, outbox, logger, usage-ledger, or billing-period changes. Validation:
Accepted limitations remain: no charge when a provider never returns an allocation ID or the process crashes, cleanup is platform overhead, failed attempts are uncharged, Agent/Pi nested Function costs are folded into the parent toolCost, and per-span trace cost visibility follows existing UI policy. @cubic-dev-ai please review the rewritten trace-native implementation. |
@BillLeoutsakosvl346 I have started the AI code review. It will take a few minutes to complete. |
|
@greptile-apps @cubic-dev-ai please re-review the latest commit 546b19e. Remote Function code exceptions and nonzero shell exits are now charged through the existing trace-cost pipeline, including retries and direct Function, Agent, and Pi consumers. Provider/sandbox failures, timeouts, and cancellation remain uncharged. Focused validation: 587 tests passed, plus type-check, lint, API validation, tool-boundary, monorepo boundaries, and diff-check. |
@BillLeoutsakosvl346 I have started the AI code review. It will take a few minutes to complete. |
There was a problem hiding this comment.
4 issues found across 31 files
Confidence score: 3/5
apps/sim/lib/function-execution/execute-request.tsandapps/sim/executor/handlers/pi/pi-handler.tscan lose provider runtime or billed Function-call costs when export, backend, memory persistence, or retry handling fails, causing undercharging and incomplete trace/error billing; preserve and propagate the cost through these failure paths.apps/sim/lib/execution/remote-sandbox/index.tsmisclassifies E2B provider-limit failures as billable shell or user-code failures, which can charge users for provider-side limits; classify the limit before assigningbillableResultwhile retaining the failure response.
Prompt for AI agents (unresolved issues)
Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.
<file name="apps/sim/lib/function-execution/execute-request.ts">
<violation number="1" location="apps/sim/lib/function-execution/execute-request.ts:1828">
P2: When remote code completes but workspace-file export fails, the response drops `sandboxCost`, so the existing trace-cost pipeline cannot charge the provider runtime. Thread the cost through `exportFailure` and include it in every export-error response.</violation>
</file>
<file name="apps/sim/executor/handlers/pi/pi-handler.ts">
<violation number="1" location="apps/sim/executor/handlers/pi/pi-handler.ts:537">
P2: When a Local Pi run executes a billed Function call and then the backend or memory persistence fails, `runPi` throws before exposing `functionToolCost`, so the error and any retry/final block log lose that charge. Preserve the accumulated cost on the thrown error before rethrowing, as the Function handler does.</violation>
</file>
<file name="apps/sim/lib/execution/remote-sandbox/index.ts">
<violation number="1" location="apps/sim/lib/execution/remote-sandbox/index.ts:660">
P2: When E2B reaches its provider limit, this branch treats `E2BProviderLimitError` as a billable user-code exception. Exclude provider-limit errors from `billableResult` while still returning the failure response.</violation>
<violation number="2" location="apps/sim/lib/execution/remote-sandbox/index.ts:800">
P2: When E2B reports its provider limit through the sentinel stderr and exit code, this branch bills it as an ordinary shell failure. Classify the provider-limit result before assigning `billableResult` so provider failures remain uncharged.</violation>
</file>
You've manually re-run cubic several times on this PR. Each manual re-review checks the full PR again and counts toward your usage quota. To preserve your usage limits, we recommend letting cubic automatically review new commits.
Fix all with cubic | Re-trigger cubic
546b19e to
97c0c8a
Compare
|
@cubic-dev-ai please re-review the latest commit 97c0c8a. Completed sandbox outcomes now retain cost through workspace export failures, E2B provider lifetime-limit results are typed and remain uncharged for both code and shell, and the Pi parent-failure behavior is documented as the intentional hosted-key-compatible completion boundary. The four focused suites passed 439 tests, and type-check, lint, API validation, tool-boundary, monorepo boundaries, and diff-check are green. |
@BillLeoutsakosvl346 I have started the AI code review. It will take a few minutes to complete. |
There was a problem hiding this comment.
1 issue found across 31 files
Confidence score: 3/5
- In
apps/sim/executor/execution/block-executor.ts, a successful Function sandbox followed by a post-processing exception can lose the trusted returned cost because the catch reads cost only from the later error; preserve and attach the Function output’s cost to keep execution accounting accurate.
Prompt for AI agents (unresolved issues)
Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.
<file name="apps/sim/executor/execution/block-executor.ts">
<violation number="1" location="apps/sim/executor/execution/block-executor.ts:671">
P2: When a Function sandbox succeeds but block post-processing throws, this catch reads cost only from the later error. Retain the returned Function output's trusted cost and attach it to post-processing errors so completed sandbox usage is not under-billed.</violation>
</file>
You've manually re-run cubic several times on this PR. Each manual re-review checks the full PR again and counts toward your usage quota. To preserve your usage limits, we recommend letting cubic automatically review new commits.
Fix all with cubic | Re-trigger cubic
| return softOutput | ||
| } | ||
|
|
||
| const trustedExecutionCost = readTrustedExecutionCost(error) |
There was a problem hiding this comment.
P2: When a Function sandbox succeeds but block post-processing throws, this catch reads cost only from the later error. Retain the returned Function output's trusted cost and attach it to post-processing errors so completed sandbox usage is not under-billed.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At apps/sim/executor/execution/block-executor.ts, line 671:
<comment>When a Function sandbox succeeds but block post-processing throws, this catch reads cost only from the later error. Retain the returned Function output's trusted cost and attach it to post-processing errors so completed sandbox usage is not under-billed.</comment>
<file context>
@@ -620,8 +668,10 @@ export class BlockExecutor {
return softOutput
}
+ const trustedExecutionCost = readTrustedExecutionCost(error)
const errorOutput: NormalizedBlockOutput = {
error: errorMessage,
</file context>
Summary
codeandshellsandboxes for E2B and Daytonaoutput.costtoolCostsandboxProfile: mothershipunmeteredTrace-native billing flow
There is no sandbox-specific outbox, reconciliation worker, billing-period machinery, database schema, or new ledger path.
Runtime begins immediately before provider creation and ends when Function cleanup starts. Pricing uses the provider-effective TTL as a cap, the configured production multiplier once, and eight-decimal final rounding. Cleanup time is Sim infrastructure overhead.
Outcome policy
Billable completed sandbox outcomes:
Uncharged outcomes:
The parent-failure rule intentionally matches hosted-key trace collection: nested tool cost is published through the completed parent span, not billed independently from partial child execution.
Pricing and scope
COST_MULTIPLIERAccepted limitations
Validation
Latest focused run after rebasing onto staging:
apps/simtype-check passedapps/simlint passedgit diff --checkpassedTicket: https://sim-ai.slack.com/archives/C093DF8MA21/p1787866922654329