Job lifecycle#
InsertOpts#
Unique jobs are global by default: the same canonical key is rejected even if
the second enqueue targets another queue. Set ByQueue to include the queue in
the key and allow one active job per queue. ByArgs, Key, and ByPeriod add
the serialized arguments, a caller-defined value, and the start of a fixed UTC
window respectively. The stored key is a bounded SHA-256 digest, so large job
arguments or caller keys do not expand database indexes. Completed, discarded,
and cancelled jobs release their key.
Set Forever for durable idempotency keys that must survive completion,
discard, or cancellation. An explicit job deletion still releases the key.
ClickHouse uniqueness is best-effort because concurrent inserts cannot be
serialized by ReplacingMergeTree. Cassandra uses an LWT reservation followed
by job writes; failed writes are compensated, but a process crash between them
can leave an orphaned reservation that must be removed operationally.
Batch enqueue and pipelines#
Use EnqueueBatch when each job needs independent options; EnqueueBatchTx
keeps the whole batch inside the caller’s native transaction.
Jobs with the same non-empty PipelineID run sequentially in claim order inside
one WorkerPool. Set DistributedPipelines: true to extend the guarantee across
worker processes sharing the same driver. The pool renews an ownership-fenced
lease while the handler runs and cancels the handler context if renewal is lost.
Versioned payloads#
Version 1 remains plain legacy JSON. A type may declare a newer version and the worker registers each adjacent upcast step:
Malformed payloads, future versions, and missing/failed upcasters are permanent
decode failures and go directly to discarded; they do not consume the retry
schedule. InsertOpts.PayloadVersion can override the type-provided version.