WorkerConfig#
Running claims are fenced by worker ID and attempt number. attempted_at remains
the immutable claim time and started_at records handler entry; active and waiting claims renew lease_expires_at,
so rescue does not duplicate healthy long-running work. Shutdown(ctx) stops
new claims, keeps active heartbeats alive, and returns ctx.Err() if the drain
does not finish. Cancelling the pool context yields an interrupted claim without
consuming an attempt.
Create the pool with NewWorkerPoolChecked when configuration errors must be
reported before startup. The source-compatible NewWorkerPool reports the same
error from Start.
QueuePolicy.RateLimits are combined: a handler starts only after every rule
has granted a permit. Limit and Period express the sustained rate, while
Burst controls how many starts may happen without spacing (default 1). Periods
are ordinary Go durations, so second, minute, hour, and day limits use
time.Second, time.Minute, time.Hour, and 24*time.Hour respectively.
Set Key to RateLimitKeyKind, RateLimitKeyPipeline, or RateLimitKeyTag
for independent budgets. Tag policies select the first lexical tag matching
TagPrefix; missing tags share an explicit bucket. KeyConcurrency applies
the same dimensions to local execution concurrency.
Smooth rules use GCRA. Set Mode: RateLimitModeFixedWindow for counters that
reset on UTC-aligned Period boundaries; fixed windows ignore Burst.
Local rules coordinate one WorkerPool. Set Scope: RateLimitScopeGlobal to
coordinate all pools that share the same backend, queue name, and rule. Global
permits use an O(1) compare-and-swap cursor and return an exact retry time.
Waiting jobs retain and heartbeat their claims, consume at most one execution
slot per rate-limited queue, and yield without entering the handler during
shutdown.
ClickHouse is rejected for global rate limits and distributed pipelines because it cannot provide the required linearizable CAS/lease guarantees.
Policies can be replaced while a pool is running:
The update is atomic for future starts and wakes existing rate-limit waiters.
SQL, MongoDB, memory, and ClickHouse apply priority DESC, run_at ASC,
created_at ASC, and id ASC before limiting. DynamoDB, Firestore, Cassandra,
and Redis use a bounded due-candidate page because their native ready/due index
cannot express the full two-dimensional order. Set RequireStrictOrdering to
reject a non-strict driver instead of accepting that tradeoff.
Enqueue admission#
Rejected writes wrap ErrAdmissionRejected and expose a typed
*QueueFullError. Queue-depth admission is portable backpressure, not an exact
quota: the statistics read and subsequent insert are not atomic.