Skip to main content

The instrumentation produces:

  • Span goncordia.enqueue with driver, batch, queue/kind, and outcome attributes
  • Span goncordia.process with job, worker, and pipeline attributes
  • Histogram goncordia.enqueue.duration (seconds) — storage latency by driver/status
  • Histogram goncordia.job.duration (seconds) — labelled by kind, queue, status
  • Histogram goncordia.job.queue_time (seconds) — time from eligibility (max(created_at, run_at)) to handler start; scheduled waiting is excluded
  • Histogram goncordia.job.schedule_lag (seconds) — scheduled run_at to claim
  • Histogram goncordia.job.claim_wait (seconds) — claim to handler start
  • Histogram goncordia.job.rate_limit_wait (seconds) — local/global policy delay
  • Counter goncordia.job.started — handler starts by kind and queue
  • Counter goncordia.job.finished — selected durable outcomes
  • Counter goncordia.job.heartbeat.count — heartbeat outcomes (ok/stale/error)
  • Counter goncordia.job.lease_rescued — expired claims returned to queues
  • Counter goncordia.retention.passes — retention passes by status
  • Counter goncordia.retention.deleted — jobs and results deleted, by what (jobs/results)
  • Histogram goncordia.retention.duration (seconds) — retention pass duration by status
  • Counter goncordia.job.count — labelled by kind, queue, status (ok / error)

Panics are recovered, converted to errors, recorded on the span, and persisted with their stack trace in driver.AttemptError.Trace before retry/discard. The worker pool always stays alive.

Futures#

NewFutureInstrumentation implements future.Observer:

SignalNameAttributes
spangoncordia.future.awaitjob id, kind, queue, state, outcome, polls, wakeups, result bytes
spangoncordia.future.storejob id, kind, queue, attempt, result bytes; child of goncordia.process
histogram (s)goncordia.future.await.durationkind, outcome, blocking
countergoncordia.future.await.pollskind
countergoncordia.future.await.wakeupskind
histogram (By)goncordia.future.result.sizekind
histogram (s)goncordia.future.store.durationkind, status

Combine it with the Prometheus-text future.Metrics through future.Observers to serve the same signals from the admin /metrics endpoint (see Admin dashboard).

Outcomes are completed, discarded, cancelled, no_result, not_found, pending (TryGet only), abandoned (caller context ended), and error. Few wakeups relative to polls mean producers and workers run in different processes, and await latency is bounded by PollInterval.

Custom observers#

WorkerConfig.Observer receives every worker lifecycle event: claim, handler start and finish, rate-limit wait, heartbeat, and rescue. Embed goncordia.NopWorkerObserver to handle only the events you need, and combine observers with goncordia.WorkerObservers:

You can also add your own middleware for logging or custom metrics: