Skip to main content

WorkerConfig#

Running claims are fenced by worker ID and attempt number. attempted_at remains the immutable claim time and started_at records handler entry; active and waiting claims renew lease_expires_at, so rescue does not duplicate healthy long-running work. Shutdown(ctx) stops new claims, keeps active heartbeats alive, and returns ctx.Err() if the drain does not finish. Cancelling the pool context yields an interrupted claim without consuming an attempt.

Create the pool with NewWorkerPoolChecked when configuration errors must be reported before startup. The source-compatible NewWorkerPool reports the same error from Start.

QueuePolicy.RateLimits are combined: a handler starts only after every rule has granted a permit. Limit and Period express the sustained rate, while Burst controls how many starts may happen without spacing (default 1). Periods are ordinary Go durations, so second, minute, hour, and day limits use time.Second, time.Minute, time.Hour, and 24*time.Hour respectively.

Set Key to RateLimitKeyKind, RateLimitKeyPipeline, or RateLimitKeyTag for independent budgets. Tag policies select the first lexical tag matching TagPrefix; missing tags share an explicit bucket. KeyConcurrency applies the same dimensions to local execution concurrency.

Smooth rules use GCRA. Set Mode: RateLimitModeFixedWindow for counters that reset on UTC-aligned Period boundaries; fixed windows ignore Burst.

Local rules coordinate one WorkerPool. Set Scope: RateLimitScopeGlobal to coordinate all pools that share the same backend, queue name, and rule. Global permits use an O(1) compare-and-swap cursor and return an exact retry time. Waiting jobs retain and heartbeat their claims, consume at most one execution slot per rate-limited queue, and yield without entering the handler during shutdown.

ClickHouse is rejected for global rate limits and distributed pipelines because it cannot provide the required linearizable CAS/lease guarantees.

Policies can be replaced while a pool is running:

The update is atomic for future starts and wakes existing rate-limit waiters.

SQL, MongoDB, memory, and ClickHouse apply priority DESC, run_at ASC, created_at ASC, and id ASC before limiting. DynamoDB, Firestore, Cassandra, and Redis use a bounded due-candidate page because their native ready/due index cannot express the full two-dimensional order. Set RequireStrictOrdering to reject a non-strict driver instead of accepting that tradeoff.

Enqueue admission#

Rejected writes wrap ErrAdmissionRejected and expose a typed *QueueFullError. Queue-depth admission is portable backpressure, not an exact quota: the statistics read and subsequent insert are not atomic.

Retry policies#