The instrumentation produces:
- Span
goncordia.enqueuewith driver, batch, queue/kind, and outcome attributes - Span
goncordia.processwith job, worker, and pipeline attributes - Histogram
goncordia.enqueue.duration(seconds) — storage latency by driver/status - Histogram
goncordia.job.duration(seconds) — labelled by kind, queue, status - Histogram
goncordia.job.queue_time(seconds) — time from eligibility (max(created_at, run_at)) to handler start; scheduled waiting is excluded - Histogram
goncordia.job.schedule_lag(seconds) — scheduledrun_atto claim - Histogram
goncordia.job.claim_wait(seconds) — claim to handler start - Histogram
goncordia.job.rate_limit_wait(seconds) — local/global policy delay - Counter
goncordia.job.started— handler starts by kind and queue - Counter
goncordia.job.finished— selected durable outcomes - Counter
goncordia.job.heartbeat.count— heartbeat outcomes (ok/stale/error) - Counter
goncordia.job.lease_rescued— expired claims returned to queues - Counter
goncordia.retention.passes— retention passes bystatus - Counter
goncordia.retention.deleted— jobs and results deleted, bywhat(jobs/results) - Histogram
goncordia.retention.duration(seconds) — retention pass duration bystatus - Counter
goncordia.job.count— labelled by kind, queue, status (ok/error)
Panics are recovered, converted to errors, recorded on the span, and persisted
with their stack trace in driver.AttemptError.Trace before retry/discard. The
worker pool always stays alive.
Futures#
NewFutureInstrumentation implements future.Observer:
| Signal | Name | Attributes |
|---|---|---|
| span | goncordia.future.await | job id, kind, queue, state, outcome, polls, wakeups, result bytes |
| span | goncordia.future.store | job id, kind, queue, attempt, result bytes; child of goncordia.process |
| histogram (s) | goncordia.future.await.duration | kind, outcome, blocking |
| counter | goncordia.future.await.polls | kind |
| counter | goncordia.future.await.wakeups | kind |
| histogram (By) | goncordia.future.result.size | kind |
| histogram (s) | goncordia.future.store.duration | kind, status |
Combine it with the Prometheus-text future.Metrics through
future.Observers to serve the same signals from the admin /metrics
endpoint (see Admin dashboard).
Outcomes are completed, discarded, cancelled, no_result, not_found,
pending (TryGet only), abandoned (caller context ended), and error.
Few wakeups relative to polls mean producers and workers run in different
processes, and await latency is bounded by PollInterval.
Custom observers#
WorkerConfig.Observer receives every worker lifecycle event: claim, handler
start and finish, rate-limit wait, heartbeat, and rescue. Embed
goncordia.NopWorkerObserver to handle only the events you need, and combine
observers with goncordia.WorkerObservers:
You can also add your own middleware for logging or custom metrics: