Skip to main content

Apple M5, single process. Memory/SQLite are in-process (no network); Postgres/MongoDB/Redis run in Docker on localhost.

Enqueue — single job

Backendns/opNotes
Memory0.57 µsin-process mutex, no I/O
SQLite27 µsWAL mode, single connection
Redis109 µsZADD over localhost
Postgres (pgx v5)129 µsINSERT over localhost
MongoDB338 µsinsertOne over localhost
DynamoDB632 µsPutItem over localhost
Firestore1 195 µsRunTransaction + Create over emulator
ClickHouse1 378 µsINSERT + new data part over localhost
Cassandra7 216 µsLWT requires Paxos quorum (3 round trips)

EnqueueBatch(100) — 100 jobs per call

Backendms/batchjobs/s
Memory0.06 ms~1 775 000
SQLite2.8 ms~35 300
Redis10.9 ms~9 200
Postgres (pgx v5)12.8 ms~7 800
MongoDB34.9 ms~2 900
DynamoDB62.9 ms~1 590
Firestore93.5 ms~1 070
ClickHouse150 ms~665
Cassandra708 ms~141

FetchAndComplete — hot worker loop path

Backendµs/opNotes
SQLite53 µsindexed; faster than memory at scale
Memory520 µsO(N) linear scan
Redis729 µsLua ZPOPMIN + HSET
DynamoDB1 731 µsQuery GSI + conditional UpdateItem
MongoDB2 475 µsfindAndModify + updateOne
Postgres (pgx v5)12 190 µsSELECT SKIP LOCKED + UPDATE
ClickHouse14 416 µsSELECT FINAL + INSERT new version
Cassandra18 813 µsSELECT avail + LWT UPDATE per job
Firestore140 943 µsQuery + per-job RunTransaction on emulator

End-to-end — full WorkerPool

Backendworkloadconcurrencyjobs/sNotes
Memory1 000c=10~2 020
Redis1 000c=4~1 084Pub/Sub notifications
SQLite1 000c=4~800
DynamoDB1 000c=4~780polling; GSI query overhead
MongoDB1 000c=4~452Change Streams
Postgres (pgx v5)1 000c=4~179LISTEN/NOTIFY
Cassandra1 000c=4~153polling; LWT overhead
ClickHouse1 000c=4~148polling; SELECT FINAL overhead
Firestore200c=4~14polling; per-job RunTransaction on emulator

End-to-end throughput is bounded by the 5 ms poll interval used in the benchmark. In production the pgxv5 driver uses LISTEN/NOTIFY and the Redis driver uses Pub/Sub, eliminating poll latency entirely — real throughput matches the FetchAndComplete numbers above.

Cassandra’s high per-operation latency comes from Lightweight Transaction consensus (Paxos, ~3 network round trips per claim). ClickHouse’s overhead comes from SELECT … FINAL deduplication at query time. DynamoDB’s per-operation cost is dominated by HTTP/JSON round trips to the service — measured against DynamoDB Local on localhost, so real AWS numbers include additional network latency. Firestore’s per-job RunTransaction for claiming is the primary bottleneck on the emulator; production GCP Firestore will be faster due to lower round-trip latency, but the sequential-per-job claiming model inherently limits throughput. All four backends are best suited for workloads where horizontal scale matters more than raw per-job latency.