Skip to main content

Job lifecycle#

InsertOpts#

Unique jobs are global by default: the same canonical key is rejected even if the second enqueue targets another queue. Set ByQueue to include the queue in the key and allow one active job per queue. ByArgs, Key, and ByPeriod add the serialized arguments, a caller-defined value, and the start of a fixed UTC window respectively. The stored key is a bounded SHA-256 digest, so large job arguments or caller keys do not expand database indexes. Completed, discarded, and cancelled jobs release their key.

Set Forever for durable idempotency keys that must survive completion, discard, or cancellation. An explicit job deletion still releases the key.

ClickHouse uniqueness is best-effort because concurrent inserts cannot be serialized by ReplacingMergeTree. Cassandra uses an LWT reservation followed by job writes; failed writes are compensated, but a process crash between them can leave an orphaned reservation that must be removed operationally.

Batch enqueue and pipelines#

Use EnqueueBatch when each job needs independent options; EnqueueBatchTx keeps the whole batch inside the caller’s native transaction.

Jobs with the same non-empty PipelineID run sequentially in claim order inside one WorkerPool. Set DistributedPipelines: true to extend the guarantee across worker processes sharing the same driver. The pool renews an ownership-fenced lease while the handler runs and cancels the handler context if renewal is lost.

Versioned payloads#

Version 1 remains plain legacy JSON. A type may declare a newer version and the worker registers each adjacent upcast step:

Malformed payloads, future versions, and missing/failed upcasters are permanent decode failures and go directly to discarded; they do not consume the retry schedule. InsertOpts.PayloadVersion can override the type-provided version.