Composable Queue

A lightweight, auto-scaling queue for processing Go functions as jobs. Keep jobs simple, then compose behavior with wrappers for retries, timeouts, tracing, and more.

Inspired by Bull, Pond, Ants, and more.

Install

Requires Go 1.24+.

go get github.com/gnikyt/cq/v2

import "github.com/gnikyt/cq/v2" // Package name is cq.

Contents

• Install
• Features
• Feature Matrix
• When to Use
• Quick Start
  • Named Queues
  • Named Priority Queues
• Wrapper Composition
• Common Recipes
  • Reliable API Call (timeout + retry + backoff)
  • Retry-Safe Chain Step (checkpoint)
  • Idempotent Work (unique + timeout)
  • Long-Running Unique Locks (optional touch renewal)
  • Recurring Job (scheduler)
• Queue
  • Creating a Queue
  • Submission
  • Handle
  • Metrics
  • Runtime Scaling
  • Options
  • Queue Middleware
  • Pause / Resume
  • Stopping
• Documentation
• Testing
  • Benchmarks
• Dashboard & Demo
• Contributing & License

Features

• Auto-scaling worker pool (min/max workers)
• Composable job wrappers (retries, timeouts, backoffs, etc.)
• Priority queue with weighted dispatch
• Job scheduler for recurring and one-time jobs (intervals, cron expressions, or custom schedules)
• Pause/resume queue execution (local or distributed)
• Job metadata (ID, enqueue time, attempt count)
• Circuit breaker for fault tolerance
• Optional queue lifecycle hooks (enqueue/start/success/failure/discard/ abandon/reschedule plus retry-attempt events)
• Queue-level middleware chain for all jobs
• Job tagging and batch tracking
• Overlap prevention and uniqueness constraints
• Workflow step checkpointing for retry-safe chains/dependencies
• Tracing hooks and per-job progress reporting for observability
• Zero external dependencies for core functionality

Feature Matrix

Use this as a quick guide before diving into detailed sections.

Capability	Primary APIs	What it solves
Queueing and workers	NewQueue, Submit, Stop, StopDrain	Run background jobs with auto-scaling workers, handing back unstarted work on shutdown
Submission lifecycle	JobHandle, Wait, Done, Result, Cancel, Submissions	Track a submitted job to completion, cancel it, and list in-flight work
Reliability	WithRetryPolicy, WithRetry, WithRetryIf, WithBackoff, WithRecover	Handle transient failures and panic recovery
Time control	WithTimeout, WithDeadline, WithExpiry, SubmitAfter	Bound execution, discard stale queued jobs, and schedule delayed runs
Flow orchestration	WithChain, WithPipeline, WithBatch, WithDependsOn, WithCheckpoint	Build multi-step and grouped workflows with configurable dependency failure modes
Concurrency safety	WithoutOverlap, WithUnique, WithConcurrencyByKey	Prevent overlap, deduplicate work, and limit concurrent execution per key
Deferral and release	WithRelease, WithReleaseSelf, WithRateLimitRelease	Re-enqueue instead of blocking workers
Rate and fault protection	WithRateLimit, WithCircuitBreaker	Protect upstream services under load/failure
Observability and outcomes	WithTracing, WithOutcome, WithProgress, WithHooks, MetaFromContext, LastErrorFromContext	Track attempts, prior retry errors, durations, progress, and queue lifecycle transitions
Queue-wide wrappers	WithMiddleware	Apply cross-cutting behavior to every enqueued job
Multi-queue routing	NewQueueManager, QueueManager.Submit, QueueManager.SubmitAfter, NewPriorityQueueManager, Register, StartAll, StopAll	Route standard or priority jobs to named queues with isolated worker pools
Prioritization and scheduling	NewPriorityQueue, PriorityQueue.Submit, PriorityQueue.SubmitAfter, NewPriorityQueueManager, NewScheduler	Prioritize urgent jobs, route them by name, and run recurring work with typed submission outcomes

When to Use

• Standalone: Process jobs in-memory without external infrastructure. Great for CLI tools, internal services, or cases where Redis/SQS is unnecessary.
• With external queues: Use cq as the execution engine behind SQS, Redis, RabbitMQ, or any broker that feeds jobs.
• With external persistence: Keep durability outside cq with DB outbox polling or queue-native retries/DLQ semantics.
• Embedded: Add background processing to an existing app without introducing new operational infrastructure.

Quick Start

package main

import (
	"context"
	"log"
	"os/signal"
	"syscall"
	"time"

	"github.com/gnikyt/cq/v2"
)

func doWork(ctx context.Context) error {
	meta := cq.MetaFromContext(ctx)
	log.Printf("job %s started, queued %v ago", meta.ID, time.Since(meta.EnqueuedAt))
	time.Sleep(2 * time.Second)
	log.Printf("job %s completed", meta.ID)
	return nil
}

func main() {
	// Listen for interrupt signals.
	ctx, stop := signal.NotifyContext(context.Background(), syscall.SIGINT, syscall.SIGTERM)
	defer stop()

	// Create queue with the signal context.
	queue := cq.NewQueue(1, 10, 100, cq.WithContext(ctx))
	queue.Start()

	// Submit work. doWork already matches the cq.Job signature (func(context.Context) error).
	_, _ = queue.Submit(context.Background(), doWork)

	// Wait for shutdown signal.
	<-ctx.Done()

	// Stop queue, wait for in-flight jobs to finish.
	queue.Stop(true)
}

Named Queues

highQ := cq.NewQueue(5, 50, 1000) // High-priority lane.
lowQ := cq.NewQueue(1, 5, 5000)   // Bulk/background lane.

mgr := cq.NewQueueManager()
if err := mgr.Register("high", highQ); err != nil {
	log.Fatal(err)
}
if err := mgr.Register("low", lowQ); err != nil {
	log.Fatal(err)
}

mgr.StartAll()
defer mgr.StopAll(true)

if _, err := mgr.Submit(ctx, "high", processCritical); err != nil {
	log.Fatal(err)
}
if _, err := mgr.Submit(ctx, "low", processBulk); err != nil {
	log.Fatal(err)
}

if _, err := mgr.SubmitAfter(ctx, "low", processLater, 30*time.Second); err != nil {
	log.Fatal(err)
}

Named Priority Queues

// A PriorityQueue wraps an already-started base queue, and PriorityQueueManager
// has no StartAll — so start the base queues yourself.
criticalBase := cq.NewQueue(5, 20, 500)
criticalBase.Start()

bulkBase := cq.NewQueue(2, 10, 1000)
bulkBase.Start()

pmgr := cq.NewPriorityQueueManager()
if err := pmgr.Register("critical", cq.NewPriorityQueue(criticalBase, 100)); err != nil {
	log.Fatal(err)
}
if err := pmgr.Register("bulk", cq.NewPriorityQueue(bulkBase, 200)); err != nil {
	log.Fatal(err)
}

defer pmgr.StopAll(true)

if _, err := pmgr.Submit(ctx, "critical", processNow, cq.PriorityHighest); err != nil {
	log.Fatal(err)
}
if _, err := pmgr.SubmitAfter(ctx, "bulk", processLater, cq.PriorityLow, time.Minute); err != nil {
	log.Fatal(err)
}

Wrapper Composition

Wrappers let you add behavior to jobs without modifying the job itself. Compose them from innermost to outermost: the outermost wrapper runs first and controls the flow. This keeps job logic clean while adding retries, timeouts, tracing, and error handling declaratively.

job := cq.WithOutcome(           // 3. Outermost: catches final outcome.
	cq.WithRetryPolicy(         // 2. Preferred retry wrapper.
		cq.WithTimeout(          // 1. Innermost: runs with timeout.
				actualJob,
				5*time.Minute,
			),
		cq.RetryPolicy{
			MaxAttempts: 3,
			Backoff:     cq.ExponentialBackoff,
		},
	),
	onComplete,
	onFail,
	onDiscard,
)

Execution flow:

1. WithOutcome calls WithRetryPolicy
2. WithRetryPolicy calls WithTimeout
3. WithTimeout runs actualJob with a 5-minute timeout
4. If actualJob fails, control returns up the chain for retry logic
5. After all retries, WithOutcome receives the final outcome

WithRetryPolicy is the recommended default for retry behavior. WithRetry, WithRetryIf, and WithBackoff still exist for finer-grained manual composition.

Common Recipes

Use these first when you want practical defaults quickly.

Reliable API Call (timeout + retry + backoff)

job := cq.WithRetryPolicy(
	cq.WithTimeout(fetchFromAPI, 10*time.Second),
	cq.RetryPolicy{
		MaxAttempts: 3,
		Backoff:     cq.ExponentialBackoff,
	},
)
_, _ = queue.Submit(context.Background(), job)

Retry-Safe Chain Step (checkpoint)

store := cq.NewMemoryCheckpointStore()
step := cq.WithCheckpoint(
	sendInvoice,
	"send-invoice",
	store,
	cq.WithCheckpointNamespace("billing"),
)

job := cq.WithChain(
	validateOrder,
	step, // Will be skipped on retry after first success.
	notifyCustomer,
)
_, _ = queue.Submit(context.Background(), job)

Inside a checkpointed job, use SaveCheckpointData when progress must be persisted before the job returns:

if err := cq.SaveCheckpointData(ctx, []byte("batch-42-complete")); err != nil {
	return err
}

Idempotent Work (unique + timeout)

locker := cq.NewUniqueMemoryLocker()
job := cq.WithUnique(
	cq.WithTimeout(processOrder, 30*time.Second),
	"order:123",
	5*time.Minute,
	locker,
)
_, _ = queue.Submit(context.Background(), job)

Long-Running Unique Locks (optional touch renewal)

WithUniqueWindow keeps a fixed window by default. WithUnique can also be manually renewed when configured with a positive unique duration. For manual extension, use TouchLock inside your job when the locker implements optional lease renewal (RenewableLocker with Touch). TouchLock returns nil when lease renewal succeeds, ErrUniqueLeaseLost when renewal fails (for example lock ownership lost), and ErrTouchLockUnavailable when called outside a renewable unique-lock context.

job := cq.WithUnique(func(ctx context.Context) error {
	if err := cq.TouchLock(ctx, 30*time.Second); err != nil {
		return err // Handle cq.ErrUniqueLeaseLost / cq.ErrTouchLockUnavailable as needed.
	}
	return doWork(ctx)
}, "index-products", 30*time.Second, locker)

For custom ownership token formats, pass WithUniqueTokenGenerator(...).

Recurring Job (scheduler)

scheduler := cq.NewScheduler(context.Background(), queue)
defer scheduler.Stop()

schedule, err := scheduler.Every(
	"sync-products",
	10*time.Minute,
	syncProductsJob,
	cq.WithJobName("sync-products"),
)

latest, submitErr, attempted := schedule.Latest()

Cron expressions (and custom Schedule implementations) are supported via On. See docs/SCHEDULER.md for details.

nightly, _ := cq.ParseCron("0 2 * * *") // Every day at 02:00.
schedule, err := scheduler.On("nightly-report", nightly, reportJob)

Queue

Creating a Queue

queue := cq.NewQueue(1, 100, 1000)
queue.Start()
defer queue.Stop(true)

Parameters: NewQueue(minWorkers, maxWorkers, capacity).

Submission

// Recommended v2 submission API.
handle, err := queue.Submit(ctx, job,
	cq.WithJobID("message-123"),
	cq.WithJobName("process-message"),
	cq.WithJobAttribute("source", "sqs"),
)
if err != nil {
	log.Fatal(err) // Job was not accepted.
}

// Waiting is optional. A wait timeout does not cancel the running job.
if err := handle.Wait(ctx); err != nil {
	log.Printf("job failed or wait ended: %v", err)
}

// Blocks until accepted or ctx ends.
handle, err := queue.Submit(ctx, job)

// Returns ErrQueueFull instead of waiting for capacity.
handle, err = queue.Submit(ctx, job, cq.WithNonBlocking())

scheduled, err := queue.SubmitAfter(ctx, job, 2*time.Minute)
scheduledAt, err := queue.SubmitAt(ctx, job, time.Now().Add(time.Hour))
handles, err := queue.SubmitBatch(ctx, jobs)
scheduledHandles, err := queue.SubmitBatchAfter(ctx, jobs, 30*time.Second)

// Resubmit a running job later.
rescheduled, err := cq.Reschedule(ctx, queue, job, time.Minute, cq.RescheduleReasonManualRetry)

Submit distinguishes submission failure from execution failure. It returns an error only when the queue does not accept the job. After acceptance, JobHandle tracks completion through Done, Wait, and Result. Cancel prevents a pending job from executing or signals a running job through its context. Running jobs must respect context cancellation for the request to stop execution. Custom IDs are visible through MetaFromContext, lifecycle hooks, and default checkpoint keys.

Cancel returns whether that call cancelled pending execution or delivered the first request to a running job. Cancelling an already-buffered job prevents its body from running, but does not immediately reclaim its internal queue slot.

SubmitAfter accepts scheduling responsibility immediately. Its handle remains pending during the delay, then reports the eventual execution result or a future rejection such as ErrQueueStopped, ErrQueuePaused, or ErrQueueFull. SubmitAt is the absolute-time form of SubmitAfter (delay computed as time.Until(at), a past time submits immediately). Batch methods return handles for accepted jobs and preserve partial-acceptance errors.

Reschedule creates a fresh delayed submission while preserving the current job name and attributes. It adds parent ID, root ID, and reason lineage attributes and returns the new submission handle.

Typed submission rejection errors:

• cq.ErrQueueStopped
• cq.ErrQueuePaused
• cq.ErrQueueFull
• cq.ErrQueueJobRequired

Handle

Submit returns a *cq.JobHandle for each accepted job. The handle tracks that one submission from acceptance to a terminal result.

handle, err := queue.Submit(ctx, job, cq.WithJobName("process-message"))
if err != nil {
	log.Fatal(err) // Job was not accepted.
}

id := handle.ID()     // Job ID (queue-generated when none was supplied).
meta := handle.Meta() // Copy of the job's JobMeta: ID, Name, Attributes, EnqueuedAt, Attempt.

// Block until the submission reaches a terminal state, or ctx ends.
// The returned error is the job's terminal error (nil on success).
if err := handle.Wait(ctx); err != nil {
	log.Printf("job failed or wait ended: %v", err)
}

// Or select on the completion channel yourself. This is handle.Done(),
// distinct from the ctx.Done() used above to wait for process shutdown.
select {
case <-handle.Done():
	result, _ := handle.Result() // ok is false until the submission is terminal.
	log.Printf("finished in %s: %v", result.Duration(), result.Err)
case <-ctx.Done():
	// Stopped waiting. The job itself keeps running.
}

// Cancel a pending job before it runs, or signal a running job through its context.
handle.Cancel()

JobHandle methods:

• ID() string — the submission's job ID.
• Meta() JobMeta — a copy of the job's metadata.
• Done() <-chan struct{} — closed when the submission reaches a terminal state.
• Wait(ctx) error — blocks for completion (returns the terminal error) or ctx cancellation (returns ctx.Err()). A wait timeout does not cancel the job.
• Result() (JobResult, bool) — the terminal JobResult; ok is false until the submission is terminal.
• Cancel() bool — cancels a pending job or signals a running one; reports whether this call delivered the cancellation.

JobResult carries the outcome: Meta (JobMeta), StartedAt, FinishedAt, Err, and a Duration() helper (zero before execution starts).

Scheduled submissions. SubmitAfter, SubmitAt, and SubmitBatchAfter return the same *cq.JobHandle, but it stays pending for the whole delay before its worker runs. This changes when each method resolves:

• Wait/Done span the delay and execution — they do not fire when the timer elapses, only when the job reaches a terminal state.
• Cancel during the delay prevents the job from ever being submitted; the handle resolves to cq.ErrJobCancelled and no worker runs it.
• A rejection that only happens once the timer fires (cq.ErrQueueStopped, cq.ErrQueuePaused, cq.ErrQueueFull) surfaces through the handle's terminal result — via Wait or Result().Err — not through the SubmitAfter return, which reported only whether scheduling responsibility was accepted.

handle, err := queue.SubmitAfter(ctx, job, 30*time.Second)
if err != nil {
	log.Fatal(err) // Scheduling was refused up front (e.g. queue already stopped).
}

// Changed our mind before it fires: the job never runs.
handle.Cancel()
if err := handle.Wait(ctx); errors.Is(err, cq.ErrJobCancelled) {
	log.Print("cancelled before it was submitted")
}

The recurring cq.Scheduler (NewScheduler) is separate and hands back its own cq.ScheduleHandle; see Recurring Job (scheduler).

Metrics

queue.RunningWorkers()           // Current running workers.
queue.IdleWorkers()              // Current idle workers.
queue.Capacity()                 // Job channel capacity.
queue.WorkerRange()              // (min, max) workers.
queue.Stats()                    // Single-call queue snapshot.
queue.Submissions()              // Accepted jobs not yet terminal.
queue.SetWorkerRange(2, 20)      // Update (min, max) at runtime.
queue.TallyOf(cq.JobStateFailed) // Count by state.

// Available job states for TallyOf:
// cq.JobStateCreated   - Total jobs accepted.
// cq.JobStatePending   - Jobs waiting in the queue.
// cq.JobStateActive    - Jobs currently executing.
// cq.JobStateFailed    - Jobs completed with error.
// cq.JobStateCancelled - Jobs completed through handle cancellation.
// cq.JobStateCompleted - Jobs completed successfully.
// cq.JobStateDiscarded - Jobs marked as discarded outcomes.

queue.Stats() returns cq.QueueStats with queue name (Name), queue state (Stopped, Paused), worker details (WorkersMin, WorkersMax, RunningWorkers, IdleWorkers, Capacity), and job tallies (CreatedJobs, PendingJobs, ActiveJobs, FailedJobs, DiscardedJobs, CancelledJobs, CompletedJobs, RescheduledJobs, ReleasedJobs) in one snapshot call.

queue.Submissions() lists the jobs behind those tallies: every accepted submission that has not reached a terminal state, oldest enqueue first. Each cq.Submission carries the job's Meta and a State of cq.JobStatePending while it waits or cq.JobStateActive once a worker has started it. Tallies answer "how many", this answers "which ones". Jobs sharing an enqueue timestamp fall back to ID order, which only reads chronologically for ordered ID schemes such as the default counter.

for _, submission := range queue.Submissions() {
	log.Printf("%s (%s) is %s", submission.Meta.ID, submission.Meta.Name, submission.State)
}

// 41 (import-rows) is active
// 42 (import-rows) is pending
// 43 (send-email) is pending

PriorityQueue.Submissions() reports the same for jobs still held in its priority buffers (they leave that set once forwarded to the base queue), and QueueManager.Submissions() returns a map[string][]cq.Submission keyed by queue name for a fleet-wide view.

Runtime Scaling

if err := queue.SetWorkerRange(2, 20); err != nil {
	log.Fatal(err)
}

SetWorkerRange starts workers immediately when min increases. When max decreases, running workers are not touched. Idle cleanup drains excess workers.

Options

queue := cq.NewQueue(1, 10, 100,
	cq.WithWorkerIdleTick(500*time.Millisecond),
	cq.WithContext(ctx),
	cq.WithPanicHandler(func(err any) {
		log.Printf("panic: %v", err)
	}),
)

// Available options:
// cq.WithWorkerIdleTick(d)           - Interval for idle worker cleanup (default 5s).
// cq.WithContext(ctx)                - Parent context for the queue.
// cq.WithCancelableContext(ctx, fn)  - Parent context with custom cancel function.
// cq.WithPanicHandler(fn)            - Custom handler override for job panics.
// cq.WithIDGenerator(fn)             - Override fallback job ID generation.
// cq.WithQueueName(name)             - Stable queue name for observability events/stats.
// cq.WithPauseStore(store, key)      - Share pause state across queue instances.
// cq.WithPausePollTick(d)            - Poll interval for distributed pause sync.
// cq.WithPauseBehavior(mode)         - Buffer or reject enqueue while paused.
// cq.WithMiddleware(mw...)           - Apply queue-level wrappers to all jobs.
// cq.WithHooks(hooks)                - Register queue lifecycle hooks.

Queue Middleware

withLogging := func(next cq.Job) cq.Job {
	return func(ctx context.Context) error {
		log.Println("job start")
		err := next(ctx)
		log.Printf("job end: %v", err)
		return err
	}
}

queue := cq.NewQueue(1, 10, 100, cq.WithMiddleware(withLogging))

WithMiddleware(a, b) composes as a(b(job)): a wraps b (outermost) and b wraps job (innermost). This lets you apply common middleware to all jobs sent to that queue instead of wiring middleware per job.

Pause / Resume

if err := queue.Pause(); err != nil {
	log.Fatal(err)
}

// ... perform maintenance ...

if err := queue.Resume(); err != nil {
	log.Fatal(err)
}

Use cq.WithPauseBehavior(cq.PauseReject) if you prefer rejecting enqueue while paused.

Stopping

Pick by what should happen to jobs that have not started yet: finish them, lose them, or get them back.

Method	In-flight jobs	Unstarted jobs	Bounded?	Use when
Stop(true)	Waits for all	Run to completion	No	Finishing every accepted job matters more than exit time
Stop(false)	Not waited	Abandoned (ErrJobAbandoned)	Mostly immediate	Queued work is disposable or persisted upstream
StopContext(ctx)	Waits up to ctx	Run if time allows, abandoned on expiry	Yes	Graceful shutdown under an external deadline (signals, etc.)
StopTimeout(d)	Same as StopContext	Same as StopContext	Yes	Same, when you just have a duration
StopDrain(ctx)	Waits up to ctx	Handed back as DrainedJobs (ErrQueueDrained)	Yes	Unstarted work must survive shutdown... persist and resubmit later
Terminate()	Not waited	Abandoned	Immediate	Emergency stop; the process is going away regardless

queue.Stop(true)
queue.StopContext(ctx)
queue.StopTimeout(5 * time.Second)

// Hand back unstarted work instead of running or dropping it.
drained, err := queue.StopDrain(ctx)
for _, dj := range drained {
	persistForRestart(dj.Meta, dj.Job)
}

See docs/QUEUE_OPTIONS.md for shutdown details.

Documentation

A browsable HTML version of these docs is published to GitHub Pages at https://gnikyt.github.io/cq/. It is generated from the Markdown below, so the Markdown remains the source of truth.

For detailed usage and advanced features, see the following guides:

• Job Wrappers - Complete reference for all job wrappers including retries, timeouts, tracing, rate limiting, circuit breakers, and custom wrappers
• Queue Options - Queue configuration options including context, panic handling, hooks, and custom ID generation
• Priority Queue - Weighted fair queuing with custom priority levels and dispatch strategies
• Queue Routing - Register named queues and route jobs to isolated worker pools
• Scheduler - Recurring and one-time job scheduling with cron-like behavior
• Custom Locker - Capability interfaces and a Redis example for distributed WithUnique and WithoutOverlap locks
• Custom Checkpoint Store - Distributed checkpoint implementations for WithCheckpoint with Redis and SQLite examples
• Custom Key Concurrency Limiter - Distributed limiter implementations for WithConcurrencyByKey with Redis and SQLite examples

Testing

Run the full suite:

go test ./...
ok  	github.com/gnikyt/cq/v2	17.117s

Run with race detector:

go test -race ./...
ok  	github.com/gnikyt/cq/v2	18.548s

Benchmarks

Run benchmarks:

go test -run=^$ -bench=. -benchmem ./...

Jobs here are a no-op body, so these measure cq's own overhead, not real work. Representative results on an Apple M5, warm pool — one job at a time, then 1M jobs across 1,000 concurrent submitters:

BenchmarkSingleSteadyState-10                       1000000    1369 ns/op     983 B/op    13 allocs/op
BenchmarkScenariosSteadyState/1kReq--1kJobs-10            1    1206286625 ns/op          12018107 allocs/op

That's ~1.2–1.4µs and ~12 allocs of cq overhead per job — roughly 830k no-op jobs/sec of headroom. Any real job dwarfs that, so your throughput is bounded by your work and maxWorkers, not these numbers.

Dashboard & Demo

There is a cq-dashboard project. It is a separate, optional module that records cq's lifecycle hooks to a database and serves them as a web UI: job history with per-attempt retry detail, reschedule and release lineage, live worker and buffer stats, schedules, and a grouped failures view. It has its own go.mod, so cq's core stays dependency-free whether or not you use it. Hooks hand events to a buffered writer and never block a worker.

Below is an example of running the dashboard in demo mode: go run ./cmd/demo -addr :8080.

Overview: live queue stats, throughput, in-flight jobs, and schedules.

Jobs: every execution recorded, filterable by queue, attribute, and state.

Job detail: per-attempt results, errors, and the reschedule/release lineage chain.

Contributing & License

Contributions are welcome. See CONTRIBUTING.md to get started. Licensed under the MIT License.