From 39e81c1eb555482c3c99db7b322e55589797c41f Mon Sep 17 00:00:00 2001 From: Claude Date: Thu, 7 May 2026 21:35:49 +0000 Subject: [PATCH] docs(geode): correct OK ordering/durability assumptions in ingestion plan OK frames carry the event id, so clients pair replies by id rather than by arrival order. NIP-01 also treats OK true as "accepted," not "fsynced." That removes two constraints the plan was carrying: - no per-connection FIFO requirement on OKs - no need to delay OKs until after batch fsync Tier 1 can fan OKs out as soon as the per-row INSERT returns inside the open transaction, hiding the group-commit fsync entirely from publisher latency. Tier 2 drops the order-preserving commit log and just sends OKs straight to outQueue. The pipelined benchmark now checks one-OK-per-event-id rather than ordering. --- .../2026-05-07-event-ingestion-batching.md | 51 ++++++++++++------- 1 file changed, 32 insertions(+), 19 deletions(-) diff --git a/geode/plans/2026-05-07-event-ingestion-batching.md b/geode/plans/2026-05-07-event-ingestion-batching.md index e6df99ebf..561d1dd08 100644 --- a/geode/plans/2026-05-07-event-ingestion-batching.md +++ b/geode/plans/2026-05-07-event-ingestion-batching.md @@ -19,14 +19,20 @@ contention, not WS throughput). ## Constraints we must keep -- **OK ordering**: NIP-01 requires the OK reply to follow its EVENT. - We cannot reply OK before the insert decision (the OK carries - accepted/rejected + reason). -- **Durability semantics**: clients reasonably assume `OK true` means - "stored." Batching must not make us reply OK before fsync. -- **Per-connection FIFO**: a publisher that sends three EVENTs in a - row expects three OKs in that order. Reordering across connections - is fine. +- **OK pairs by event id, not by order**: the OK frame carries the + event id, so clients pair replies to publishes by id. OKs can be + emitted in any order — including reordered against the EVENT + stream, and against each other on the same connection. This frees + us to fan OKs out as soon as the writer has a per-row decision. +- **OK semantics = accepted, not fsynced**: NIP-01 treats `OK true` + as "accepted by the relay," not "durably on disk." We can reply as + soon as SQLite returns success for the row (inside the open + transaction, before commit/fsync). Group commit can batch the + fsync without holding OKs back. +- **Per-row decision still required**: the OK reason field is + per-event (duplicate, blocked, invalid sig, pow, etc.), so we + cannot fan out a single batch-level OK. Each row's OK must reflect + that row's outcome. ## Sketch @@ -36,7 +42,12 @@ Confirm `PRAGMA journal_mode=WAL` + `PRAGMA synchronous=NORMAL` on the event-store DB; group commits across the writer mutex's hold window. Today each insert is its own transaction. Wrap N inserts (or a 5 ms budget, whichever first) in a single transaction managed by the writer -coroutine. On commit, fan back N OK replies. +coroutine. + +Because OK reflects acceptance not durability, each row can fan an OK +as soon as the per-row INSERT statement returns inside the +transaction — we do not need to wait for the batch's commit. The +fsync is hidden from the publisher latency budget entirely. Implementation lives in quartz's `EventStore` / `SQLiteConnectionPool`, not geode — but geode owns the benchmark and validates the gain. @@ -48,17 +59,18 @@ commit is well-trodden territory (nostr-rs-relay, strfry both do it). `RelaySession.receive` is currently single-flight: one EVENT in, process, OK out, next EVENT. Allow a connection to push N EVENTs -concurrently, dispatch them to a per-connection ingest pipeline, and -serialise OKs back in arrival order via a small commit log. +concurrently and dispatch them to a per-connection ingest pipeline. -A `Channel with capacity = INGEST_PIPELINE_DEPTH` per -connection, drained by a coroutine that batches into the group-commit -above. OK responses are written to an `outQueue.send()` already — so -the pipeline just needs to record arrival order and emit OKs in that -order after each batch commits. +A `Channel` with capacity = `INGEST_PIPELINE_DEPTH` per +connection, drained by a coroutine that feeds the group-commit writer +above. OKs go straight to `outQueue.send()` the moment each row +returns from INSERT — no ordering bookkeeping needed, since the OK +frame already carries the event id and the spec doesn't require +order. A pipelined publisher keying on event id will pair replies +correctly. -Expected: hides the verify+insert latency behind another EVENT's -parse, gets us closer to network-bound throughput. +Expected: hides verify+insert latency behind the next EVENT's parse, +gets us closer to network-bound throughput. ### Tier 3 — eager Schnorr verify off the writer thread @@ -74,7 +86,8 @@ Add to `geode.perf.LoadBenchmark`: - `publishGroupCommitSingleClient` — same workload as the current single-client benchmark, asserts >5000 EPS. - `publishPipelinedSingleClient` — sends 100 EVENTs without awaiting - intermediate OKs; measures end-to-end and OK-ordering correctness. + intermediate OKs; measures end-to-end throughput and verifies that + every event id receives exactly one OK (in any order). Existing benchmarks stay as the regression floor.