docs(geode): correct OK ordering/durability assumptions in ingestion plan
OK frames carry the event id, so clients pair replies by id rather than by arrival order. NIP-01 also treats OK true as "accepted," not "fsynced." That removes two constraints the plan was carrying: - no per-connection FIFO requirement on OKs - no need to delay OKs until after batch fsync Tier 1 can fan OKs out as soon as the per-row INSERT returns inside the open transaction, hiding the group-commit fsync entirely from publisher latency. Tier 2 drops the order-preserving commit log and just sends OKs straight to outQueue. The pipelined benchmark now checks one-OK-per-event-id rather than ordering.
This commit is contained in:
@@ -19,14 +19,20 @@ contention, not WS throughput).
|
||||
|
||||
## Constraints we must keep
|
||||
|
||||
- **OK ordering**: NIP-01 requires the OK reply to follow its EVENT.
|
||||
We cannot reply OK before the insert decision (the OK carries
|
||||
accepted/rejected + reason).
|
||||
- **Durability semantics**: clients reasonably assume `OK true` means
|
||||
"stored." Batching must not make us reply OK before fsync.
|
||||
- **Per-connection FIFO**: a publisher that sends three EVENTs in a
|
||||
row expects three OKs in that order. Reordering across connections
|
||||
is fine.
|
||||
- **OK pairs by event id, not by order**: the OK frame carries the
|
||||
event id, so clients pair replies to publishes by id. OKs can be
|
||||
emitted in any order — including reordered against the EVENT
|
||||
stream, and against each other on the same connection. This frees
|
||||
us to fan OKs out as soon as the writer has a per-row decision.
|
||||
- **OK semantics = accepted, not fsynced**: NIP-01 treats `OK true`
|
||||
as "accepted by the relay," not "durably on disk." We can reply as
|
||||
soon as SQLite returns success for the row (inside the open
|
||||
transaction, before commit/fsync). Group commit can batch the
|
||||
fsync without holding OKs back.
|
||||
- **Per-row decision still required**: the OK reason field is
|
||||
per-event (duplicate, blocked, invalid sig, pow, etc.), so we
|
||||
cannot fan out a single batch-level OK. Each row's OK must reflect
|
||||
that row's outcome.
|
||||
|
||||
## Sketch
|
||||
|
||||
@@ -36,7 +42,12 @@ Confirm `PRAGMA journal_mode=WAL` + `PRAGMA synchronous=NORMAL` on the
|
||||
event-store DB; group commits across the writer mutex's hold window.
|
||||
Today each insert is its own transaction. Wrap N inserts (or a 5 ms
|
||||
budget, whichever first) in a single transaction managed by the writer
|
||||
coroutine. On commit, fan back N OK replies.
|
||||
coroutine.
|
||||
|
||||
Because OK reflects acceptance not durability, each row can fan an OK
|
||||
as soon as the per-row INSERT statement returns inside the
|
||||
transaction — we do not need to wait for the batch's commit. The
|
||||
fsync is hidden from the publisher latency budget entirely.
|
||||
|
||||
Implementation lives in quartz's `EventStore` / `SQLiteConnectionPool`,
|
||||
not geode — but geode owns the benchmark and validates the gain.
|
||||
@@ -48,17 +59,18 @@ commit is well-trodden territory (nostr-rs-relay, strfry both do it).
|
||||
|
||||
`RelaySession.receive` is currently single-flight: one EVENT in,
|
||||
process, OK out, next EVENT. Allow a connection to push N EVENTs
|
||||
concurrently, dispatch them to a per-connection ingest pipeline, and
|
||||
serialise OKs back in arrival order via a small commit log.
|
||||
concurrently and dispatch them to a per-connection ingest pipeline.
|
||||
|
||||
A `Channel<EventCmd> with capacity = INGEST_PIPELINE_DEPTH` per
|
||||
connection, drained by a coroutine that batches into the group-commit
|
||||
above. OK responses are written to an `outQueue.send()` already — so
|
||||
the pipeline just needs to record arrival order and emit OKs in that
|
||||
order after each batch commits.
|
||||
A `Channel<EventCmd>` with capacity = `INGEST_PIPELINE_DEPTH` per
|
||||
connection, drained by a coroutine that feeds the group-commit writer
|
||||
above. OKs go straight to `outQueue.send()` the moment each row
|
||||
returns from INSERT — no ordering bookkeeping needed, since the OK
|
||||
frame already carries the event id and the spec doesn't require
|
||||
order. A pipelined publisher keying on event id will pair replies
|
||||
correctly.
|
||||
|
||||
Expected: hides the verify+insert latency behind another EVENT's
|
||||
parse, gets us closer to network-bound throughput.
|
||||
Expected: hides verify+insert latency behind the next EVENT's parse,
|
||||
gets us closer to network-bound throughput.
|
||||
|
||||
### Tier 3 — eager Schnorr verify off the writer thread
|
||||
|
||||
@@ -74,7 +86,8 @@ Add to `geode.perf.LoadBenchmark`:
|
||||
- `publishGroupCommitSingleClient` — same workload as the current
|
||||
single-client benchmark, asserts >5000 EPS.
|
||||
- `publishPipelinedSingleClient` — sends 100 EVENTs without awaiting
|
||||
intermediate OKs; measures end-to-end and OK-ordering correctness.
|
||||
intermediate OKs; measures end-to-end throughput and verifies that
|
||||
every event id receives exactly one OK (in any order).
|
||||
|
||||
Existing benchmarks stay as the regression floor.
|
||||
|
||||
|
||||
Reference in New Issue
Block a user