[BUG] TypeScript SDK: worker's own /health and /metrics slots value ignores per-task slotCost and conflates separate slot pools

[chandan-1427]

Title

[BUG] TypeScript SDK: worker's own /health and /metrics slots value ignores per-task slotCost and conflates separate slot pools

Labels

bug

---

Describe the issue

InternalWorker.getAvailableSlots (sdks/typescript/src/v1/client/worker/worker-internal.ts:144-149) computes the value reported by this worker's own /health endpoint (slots field) and its own /metrics Prometheus gauge (worker_slots, wired up in health-server.ts:42,67,95-102, both driven by the same getSlots() callback):

private getAvailableSlots(): number {
  const totalSlots = Object.values(this.slotConfig).reduce((acc, curr) => acc + curr, 0);
  const currentRuns = Object.keys(this.futures).length;
  return Math.max(0, totalSlots - currentRuns);
}

This has two independent problems, both reproduced below:

1. It ignores each running task's slotCost. The SDK has a documented slotCost option (sdks/typescript/src/v1/task.ts:255-264):

(optional) the number of default worker slots this task consumes. A worker has a fixed number of slots (default 100), and a normal task consumes one. Set slotCost higher for a task that needs more memory or CPU, so a worker runs fewer of them at once.

Nothing in the worker's runtime (this.futures, handleStartStepRun, etc.) records a task's slot cost while it runs — slotRequests/slotCost are read once, at workflow-registration time, purely to tell the engine how many slots to reserve when assigning work (mapSlotRequestsPb, worker-internal.ts:1241-1262). getAvailableSlots() subtracts one unit per running task regardless of its actual cost, so a worker running one slotCost: 5 task out of a 10-slot pool reports 9 slots free instead of 5.

2. It sums separate slot pools (default + durable) into a single number. slotConfig can hold independent pools (e.g. { default: 5, durable: 3 }) that are not fungible — a durable-pool slot can't run a non-durable task and vice versa. getAvailableSlots() adds every pool together into one totalSlots, then subtracts a flat count of all running futures regardless of which pool they occupy. If the default pool is completely full but the durable pool is empty, it reports slots as available that no queued non-durable task could actually use.

What this does not affect (checked explicitly, since a related but functionally separate mechanism exists): Hatchet's documented autoscaling signal, hatchet_tenant_available_worker_slots (frontend/docs/pages/v1/autoscaling-workers.mdx:203-205, frontend/docs/pages/self-hosting/prometheus-metrics.mdx:214-216), is computed entirely server-side by the engine's own scheduling pool tracking (pkg/scheduling/v1/prometheus_extension.go:192, pkg/integrations/metrics/prometheus/tenant.go) — it does not read this worker-reported value. slotConfig is only sent to the engine once, at registration (worker-internal.ts:1079); nothing in the TS SDK's dispatcher/action-listener re-reads or re-sends local slot availability afterward, so this bug also has no effect on actual task assignment/scheduling. The blast radius is confined to this specific worker process's own self-reported /health and /metrics values — real for anyone pointing custom monitoring/alerting directly at an individual worker's health-check port, but not the officially documented cross-worker autoscaling path.

Environment

• SDK: TypeScript (sdks/typescript), current main (commit 2ff0b1b8a)
• Engine: N/A — both issues are reproducible without a live engine connection

Expected behavior

getAvailableSlots() should track and subtract the actual slot cost of each currently running task, per pool, rather than a flat count of running tasks summed across all pools.

Steps to Reproduce

1. git checkout 2ff0b1b8a
2. cd sdks/typescript

Repro 1 — slotCost ignored:

import { InternalWorker } from '@hatchet/v1/client/worker/worker-internal';

describe('InternalWorker.getAvailableSlots slot-cost accounting', () => {
  it('undercounts consumed capacity when a running task has slotCost > 1', () => {
    const fakeThis: any = {
      slotConfig: { default: 10 },
      futures: { 'expensive-task-run/0': {} }, // one task running with slotCost: 5
    };

    const available = (InternalWorker.prototype as any).getAvailableSlots.call(fakeThis);

    expect(available).toBe(5);
  });
});

Run npx jest slot-cost-accounting. Observe:

Expected: 5
Received: 9

Repro 2 — pool conflation:

import { InternalWorker } from '@hatchet/v1/client/worker/worker-internal';

describe('InternalWorker.getAvailableSlots pool conflation', () => {
  it('reports slots free from the durable pool as if usable by the (full) default pool', () => {
    const fakeThis: any = {
      slotConfig: { default: 5, durable: 3 },
      futures: { 'run-1': {}, 'run-2': {}, 'run-3': {}, 'run-4': {}, 'run-5': {} },
    };

    const available = (InternalWorker.prototype as any).getAvailableSlots.call(fakeThis);

    expect(available).toBe(0);
  });
});

Run the same way. Observe:

Expected: 0
Received: 3

Both tests use the private-method testing pattern already established in worker-cancel-supervision.test.ts (InternalWorker.prototype.<method>.call(fakeThis, ...)).

Additional context

• Checked GitHub for duplicates (getAvailableSlots, slotCost, worker_slots, workerSlotsGauge) — no existing open issue/PR covers this.
• Checked whether Python/Go/Ruby SDKs have an equivalent method — no getAvailableSlots/get_available_slots/available_slots hits in sdks/python, sdks/go, or sdks/ruby, so this appears TS-specific; have not audited whether those SDKs have an analogous gauge under different naming (out of scope for this issue).

---

🤖 AI Disclosure

• I acknowledge that an LLM was used in the creation of this Issue, in accordance with Hatchet's AI_POLICY.md.

• Details: Claude Code was used to trace the slotCost/slotConfig runtime lifecycle end-to-end (registration vs. running-task bookkeeping vs. engine-side autoscaling metrics), rule out impact on actual task scheduling and on the documented autoscaling signal by reading the Go engine source and docs, and produce both reproduction tests. Reviewed and verified manually before submission.

[Nahid-NHB]

thanks I'm working on it