cost: measure what durability costs a write #166
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "land/cost/durable"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
The write benchmarks all run against an in-memory store, so they measure engine
work and nothing else. What is left of the write cost after
spareispersistence, and persistence is exactly what an in-memory store cannot show —
the numbers that decided the last round came from a consumer's server-side
probe, not from here.
This adds a durable round: the same put loop against a file-backed store, and
the same loop again inside one batch. Three numbers fall out of that, and the
third is the one worth having, because a batch is a single transaction and
therefore a single flush.
Durability is 93% of what a write costs on disk, it does not move with the
number of rows, and one transaction per write is what buys it. That bounds what
group commit could return: about five and a half milliseconds a write, no more,
and only for writes that are in flight together.
The reading agrees with the consumer that raised the question. A 452-reference
push measured 2.4 seconds of flush and roughly 0.3 seconds of engine work there;
this bench predicts 2.49 and 0.20 for the same shape. Two instruments that were
built separately now say the same thing, which is the point of having the second
one.
Ignored like its neighbours: the guard stays fast and the numbers are asked for
when someone is deciding something.
Co-Authored-By: Claude Opus 5 (1M context) noreply@anthropic.com
The write benchmarks all run against an in-memory store, so they measure engine work and nothing else. What is left of the write cost after `spare` is persistence, and persistence is exactly what an in-memory store cannot show — the numbers that decided the last round came from a consumer's server-side probe, not from here. This adds a durable round: the same put loop against a file-backed store, and the same loop again inside one batch. Three numbers fall out of that, and the third is the one worth having, because a batch is a single transaction and therefore a single flush. 100 rows memory 0.44ms durable 5.83ms batched 0.38ms durability 5.44ms 400 rows memory 0.43ms durable 5.98ms batched 0.36ms durability 5.62ms Durability is 93% of what a write costs on disk, it does not move with the number of rows, and one transaction per write is what buys it. That bounds what group commit could return: about five and a half milliseconds a write, no more, and only for writes that are in flight together. The reading agrees with the consumer that raised the question. A 452-reference push measured 2.4 seconds of flush and roughly 0.3 seconds of engine work there; this bench predicts 2.49 and 0.20 for the same shape. Two instruments that were built separately now say the same thing, which is the point of having the second one. Ignored like its neighbours: the guard stays fast and the numbers are asked for when someone is deciding something. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Land-Source: cost/durable@bb0e3a7c46