Skip to main content

The portable maintenance service works with every built-in driver and uses the same injected-clock model as clients, workers, and schedulers:

Bulk operations preserve partial progress in BulkResult; their returned error joins per-job failures, so errors.Is still recognizes typed driver errors. Dead-letter replay first verifies that each job is still discarded.

Retention#

Every worker pool prunes old jobs by default, on every driver. Pools share a leader lease, so only one instance prunes at a time, and another takes over if it stops. The default goncordia.DefaultRetentionPolicy keeps:

WhatKept for
completed jobs24 hours
cancelled jobs24 hours
discarded jobs (dead letters)7 days
future results of completed jobs1 hour

Override any field in WorkerConfig; zero fields keep the default, and goncordia.KeepForever turns a state off. A negative RetentionInterval disables retention in that pool:

Futures results expire earlier than the jobs. Storage cannot know whether some process will still call Await (waiters live in other processes’ memory, and future.Of can follow a job later), so results expire by age, not by waiter count; after that, Await returns future.ErrNoResult. Func.Call does not wait for retention: its job ID never leaves the call, so it deletes the job and result as soon as it returns a value (see Futures). Manager.Forget drops a result you have read.

Without a worker pool, for example in a separate maintenance process, run the same loop yourself, or call Prune once:

Zero durations in Prune and RunRetention skip that state; negative ones other than KeepForever are rejected.

How a pass finds expired jobs depends on the driver:

DriverHow retention finds expired jobs
pgxv5, stdlib (Postgres, MySQL, SQLite), gorm, bun, MongoDB, Redisindex on state and finalization time, batched deletes (driver.JobPruner)
DynamoDBsparse GSI gsi_retention on terminal jobs, sharded 16 ways; created by Migrate
Firestorecomposite index state ASC, finalized_at ASC on goncordia_jobs (create it yourself, like the fetch index); results use the automatic updated_at index
Cassandralookup tables goncordia_jobs_finalized and goncordia_results_written, read from a stored watermark so deleted rows’ tombstones are not rescanned
ClickHousedata-skipping indexes on state, finalized_at, id, and result updated_at; expired rows are removed physically with lightweight DELETE, including rows that JobDelete only marked deleted
memoryscans in process memory

On SQL and MongoDB, Migrate creates the retention indexes (idx_goncordia_jobs_finalized, idx_goncordia_job_results_updated). Building an index on a large Postgres table blocks writes to it while the index is built. To avoid that, create the indexes yourself before upgrading; Migrate then skips them:

Retention metrics#

goncordia.NewRetentionMetrics() counts passes and deletions. Use it as the pool’s observer and serve it from the admin /metrics endpoint:

MetricTypeLabels
goncordia_retention_passes_totalcounterstatus (ok, error)
goncordia_retention_jobs_deleted_totalcounter—
goncordia_retention_results_deleted_totalcounter—
goncordia_retention_last_success_timestamp_secondsgauge—
goncordia_retention_last_pass_duration_secondsgauge—

Only the instance holding the lease runs passes, so sum across instances. Alert on time() - max(goncordia_retention_last_success_timestamp_seconds) growing past a few intervals. otelgoncordia.NewInstrumentation records the same data as goncordia.retention.passes, goncordia.retention.deleted, and goncordia.retention.duration. A custom WorkerObserver receives passes by implementing goncordia.RetentionObserver.