Merge branch 'main' into claude/review-fanout-index-e1Uqq
Conflicts: - quartz/.../LiveEventStore.kt: main added IngestQueue (group-commit) and a fire-and-forget submit() path; this branch replaced the SharedFlow live fanout with FilterIndex-driven dispatch. Resolved by keeping main's submit/insert pipeline shape (IngestQueue ingest ctor param, submit() callback, insert() wrapping submit via CompletableDeferred) but routing the on-Accepted fanout through FilterIndex.candidatesFor instead of newEventStream.tryEmit. Added private fanout(event) helper. Kept main's snapshotIdsForNegentropy method. - geode/.../LoadBenchmark.kt: both sides added a new @Test method. Kept fanoutScaling (this branch) and publishPipelinedSingleClient (main).
This commit is contained in:
Claude
@@ -53,6 +53,12 @@ file = "/var/lib/geode/events.db"
|
||||
# only for trusted-input scenarios (test fixtures, mirror replays).
|
||||
# verify_signatures = true
|
||||
|
||||
# Run signature verification in parallel inside the IngestQueue
|
||||
# (across all CPU cores) instead of serially on each connection's
|
||||
# WebSocket pump. Default: true. Set false to fall back to the
|
||||
# legacy in-policy verify path.
|
||||
# parallel_verify = true
|
||||
|
||||
# Require clients to NIP-42 AUTH before REQ/EVENT/COUNT.
|
||||
require_auth = false
|
||||
|
||||
|
||||
@@ -19,62 +19,119 @@ contention, not WS throughput).
|
||||
|
||||
## Constraints we must keep
|
||||
|
||||
- **OK ordering**: NIP-01 requires the OK reply to follow its EVENT.
|
||||
We cannot reply OK before the insert decision (the OK carries
|
||||
accepted/rejected + reason).
|
||||
- **Durability semantics**: clients reasonably assume `OK true` means
|
||||
"stored." Batching must not make us reply OK before fsync.
|
||||
- **Per-connection FIFO**: a publisher that sends three EVENTs in a
|
||||
row expects three OKs in that order. Reordering across connections
|
||||
is fine.
|
||||
- **OK pairs by event id, not by order**: the OK frame carries the
|
||||
event id, so clients pair replies to publishes by id. OKs can be
|
||||
emitted in any order — including reordered against the EVENT
|
||||
stream, and against each other on the same connection. This frees
|
||||
us to fan OKs out as soon as the writer has a per-row decision.
|
||||
- **OK semantics = accepted, not fsynced**: NIP-01 treats `OK true`
|
||||
as "accepted by the relay," not "durably on disk." We can reply as
|
||||
soon as SQLite returns success for the row (inside the open
|
||||
transaction, before commit/fsync). Group commit can batch the
|
||||
fsync without holding OKs back.
|
||||
- **Per-row decision still required**: the OK reason field is
|
||||
per-event (duplicate, blocked, invalid sig, pow, etc.), so we
|
||||
cannot fan out a single batch-level OK. Each row's OK must reflect
|
||||
that row's outcome.
|
||||
|
||||
## Sketch
|
||||
|
||||
### Tier 1 — SQLite WAL + group commit (cheap win)
|
||||
|
||||
Confirm `PRAGMA journal_mode=WAL` + `PRAGMA synchronous=NORMAL` on the
|
||||
event-store DB; group commits across the writer mutex's hold window.
|
||||
Today each insert is its own transaction. Wrap N inserts (or a 5 ms
|
||||
budget, whichever first) in a single transaction managed by the writer
|
||||
coroutine. On commit, fan back N OK replies.
|
||||
WAL is already on (`PRAGMA journal_mode=WAL`). The pool runs with
|
||||
`PRAGMA synchronous=OFF`, which is one notch more permissive than
|
||||
the originally-sketched `synchronous=NORMAL` — we keep it as-is
|
||||
because the project already accepted the OS-crash trade-off there.
|
||||
|
||||
Implementation lives in quartz's `EventStore` / `SQLiteConnectionPool`,
|
||||
not geode — but geode owns the benchmark and validates the gain.
|
||||
Group commit is implemented via a new `IEventStore.batchInsert`:
|
||||
the SQLite override holds the writer mutex once and wraps N events
|
||||
in one `BEGIN IMMEDIATE … COMMIT`. Per-row error isolation uses
|
||||
SAVEPOINTs so one bad event (expired, duplicate id) doesn't roll
|
||||
back the good ones — just that row reports `Rejected`.
|
||||
|
||||
OKs fire as soon as each row's outcome is known inside the writer
|
||||
batch, not waiting for fsync (per the OK-semantics constraint above).
|
||||
|
||||
Implementation lives in `quartz/nip01Core/store/sqlite/SQLiteEventStore.batchInsertEvents`,
|
||||
exposed through `IEventStore.batchInsert` and consumed by the new
|
||||
`IngestQueue` (Tier 2 below).
|
||||
|
||||
Expected: **~5–10× write throughput** on a fast SSD. SQLite group
|
||||
commit is well-trodden territory (nostr-rs-relay, strfry both do it).
|
||||
|
||||
### Tier 2 — pipelined OK over multiple in-flight EVENTs
|
||||
|
||||
`RelaySession.receive` is currently single-flight: one EVENT in,
|
||||
process, OK out, next EVENT. Allow a connection to push N EVENTs
|
||||
concurrently, dispatch them to a per-connection ingest pipeline, and
|
||||
serialise OKs back in arrival order via a small commit log.
|
||||
`RelaySession.receive` was single-flight: one EVENT in, process, OK
|
||||
out, next EVENT. With Tier 2 the connection's pump posts to the
|
||||
shared `IngestQueue` and returns immediately — the WS pump moves
|
||||
straight to the next frame.
|
||||
|
||||
A `Channel<EventCmd> with capacity = INGEST_PIPELINE_DEPTH` per
|
||||
connection, drained by a coroutine that batches into the group-commit
|
||||
above. OK responses are written to an `outQueue.send()` already — so
|
||||
the pipeline just needs to record arrival order and emit OKs in that
|
||||
order after each batch commits.
|
||||
`IngestQueue` (one per `NostrServer`) holds a bounded
|
||||
`Channel<Submission>` (capacity = 1024 per the `DEFAULT_CAPACITY`
|
||||
constant) drained by a single writer coroutine. The writer pulls
|
||||
the first item to start a batch then `tryReceive`-drains everything
|
||||
else queued (up to 64 — `DEFAULT_MAX_BATCH`), feeds the whole batch
|
||||
to `IEventStore.batchInsert`, and dispatches each row's
|
||||
`onComplete` callback as soon as the batch returns. The callback
|
||||
turns into the `OK` frame at the WS layer.
|
||||
|
||||
Expected: hides the verify+insert latency behind another EVENT's
|
||||
parse, gets us closer to network-bound throughput.
|
||||
OKs are not order-preserving (per the constraints above). The
|
||||
writer coroutine starts lazily on first `submit` so subscription-
|
||||
only sessions don't pay for it and don't perturb `Dispatchers.Default`
|
||||
scheduling.
|
||||
|
||||
Expected: hides verify+insert latency behind the next EVENT's parse,
|
||||
gets us closer to network-bound throughput.
|
||||
|
||||
### Tier 3 — eager Schnorr verify off the writer thread
|
||||
|
||||
`VerifyPolicy` is in the policy stack and runs synchronously on
|
||||
`receive`. Move it into the ingest pipeline so verification of EVENT N+1
|
||||
runs concurrently with the SQLite commit of EVENT N. secp256k1 verify
|
||||
is parallelisable; the writer should never block on it.
|
||||
`VerifyPolicy` ran synchronously on `receive`, serialising verify
|
||||
on each connection's pump coroutine. With Tier 3, `IngestQueue`
|
||||
takes a `verify: ((Event) -> Boolean)?` hook; when set, the writer
|
||||
fan-outs a `coroutineScope { events.map { async(Default) { verify(it) } }.awaitAll() }`
|
||||
on each batch before opening the SQLite transaction. Failed
|
||||
verifies pre-mark `Rejected` and skip the insert.
|
||||
|
||||
Wired through `NostrServer(parallelVerify = ...)` and
|
||||
`geode.Relay(parallelVerify = ...)`, controlled by
|
||||
`[options].parallel_verify` in the relay config (default `true`)
|
||||
and `--no-parallel-verify` on the CLI. Internal direct callers of
|
||||
`NostrServer` (tests, library users) are opt-in: the flag defaults
|
||||
to `false` to keep existing `VerifyPolicy`-in-chain semantics
|
||||
unchanged.
|
||||
|
||||
`VerifyPolicy` was split into a parameterised
|
||||
`VerifyEventsAndAuthPolicy(verifyEvents)` with two singletons:
|
||||
|
||||
- `VerifyPolicy` (default): verifies both `EVENT` and `AUTH`.
|
||||
- `VerifyAuthOnlyPolicy`: verifies `AUTH` only, used when the
|
||||
`IngestQueue` is doing the EVENT verify.
|
||||
|
||||
When `parallelVerify` is on, `composePolicy` swaps `VerifyPolicy`
|
||||
for `VerifyAuthOnlyPolicy` so EVENTs aren't verified twice while
|
||||
AUTH commands — which bypass the queue entirely — keep their
|
||||
signature check. Without this split, removing `VerifyPolicy` from
|
||||
the chain would let a forged AUTH event mark a pubkey as
|
||||
authenticated.
|
||||
|
||||
Expected: ≈CPU_COUNT× verify-step speed-up on burst publishes
|
||||
from a single connection, where verify was previously serial on
|
||||
that pump.
|
||||
|
||||
## How to verify
|
||||
|
||||
Add to `geode.perf.LoadBenchmark`:
|
||||
`geode.perf.LoadBenchmark` carries the perf tests:
|
||||
|
||||
- `publishGroupCommitSingleClient` — same workload as the current
|
||||
single-client benchmark, asserts >5000 EPS.
|
||||
- `publishPipelinedSingleClient` — sends 100 EVENTs without awaiting
|
||||
intermediate OKs; measures end-to-end and OK-ordering correctness.
|
||||
- `publishGroupCommitSingleClient` — sequential publish-and-confirm
|
||||
on one connection (the same shape as the original
|
||||
`publishThroughputSingleClient`). Synchronous publishing means
|
||||
batch size is always 1, so this case shows per-event SQLite tx
|
||||
cost rather than the group-commit win — kept as a 500-EPS floor
|
||||
to catch regressions from the rewrite.
|
||||
- `publishPipelinedSingleClient` — bursts 10 000 EVENTs back-to-
|
||||
back without awaiting intermediate OKs; verifies end-to-end
|
||||
throughput and that every event id receives exactly one OK (in
|
||||
any order). This is where Tier 1 + Tier 2 both light up.
|
||||
|
||||
Existing benchmarks stay as the regression floor.
|
||||
|
||||
|
||||
@@ -1,20 +1,20 @@
|
||||
# NIP-77 negentropy at scale: snapshot memory + chunked replay
|
||||
# NIP-77 negentropy at scale: strfry-interop snapshot path
|
||||
|
||||
## Problem
|
||||
|
||||
`RelaySession` delegates NEG-OPEN to `NegSessionRegistry.open`
|
||||
(`quartz/nip01Core/relay/server/NegSessionRegistry.kt`), which calls
|
||||
`store.snapshotQuery(filters)` and feeds the **entire** result list
|
||||
into `NegentropyServerSession`. For a relay holding 5 M events that
|
||||
match a broad NEG-OPEN filter (`{kinds: [1, 7]}`), this is 5 M
|
||||
`Event` objects materialised in memory before the first NEG-MSG goes
|
||||
out.
|
||||
(`quartz/src/commonMain/kotlin/com/vitorpamplona/quartz/nip01Core/relay/server/NegSessionRegistry.kt:58-77`),
|
||||
which calls `store.snapshotQuery(filters)` and feeds the **entire**
|
||||
result list of full `Event` objects into `NegentropyServerSession`
|
||||
(`quartz/src/commonMain/kotlin/com/vitorpamplona/quartz/nip77Negentropy/NegentropyServerSession.kt:40-54`).
|
||||
For a relay holding 5 M events that match a broad NEG-OPEN filter
|
||||
(`{kinds: [1, 7]}`), this materialises 5 M `Event` objects with
|
||||
content/tags/sig — call it ~1 KB/event, ~5 GB transient pressure per
|
||||
concurrent NEG-OPEN — before the first NEG-MSG goes out.
|
||||
|
||||
The negentropy library itself is fine — it pivots into a sealed
|
||||
`StorageVector` (id + createdAt only, ~40 bytes/entry). But the
|
||||
`store.query<Event>(f)` step that produces the input materialises full
|
||||
`Event` objects with content, tags, sig — call it ~1 KB/event. 5 M ×
|
||||
1 KB = 5 GB transient pressure per concurrent NEG-OPEN.
|
||||
The negentropy library itself is fine. Internally it pivots into a
|
||||
sealed `StorageVector` of `(uint64 timestamp, byte[32] id)` items —
|
||||
40 bytes/entry. The waste is purely in the snapshot step.
|
||||
|
||||
Two operator-visible symptoms:
|
||||
|
||||
@@ -24,77 +24,221 @@ Two operator-visible symptoms:
|
||||
large stores the client waits seconds for what should be a
|
||||
millisecond round-trip.
|
||||
|
||||
## Reference: strfry
|
||||
|
||||
We want byte-for-byte interop and comparable throughput against
|
||||
[hoytech/strfry](https://github.com/hoytech/strfry). The relevant
|
||||
defaults from `strfry/src/apps/relay/RelayNegentropy.cpp` and
|
||||
`golpe.yaml`:
|
||||
|
||||
| Knob | strfry default | Notes |
|
||||
|---------------------------------------|-----------------------|-------|
|
||||
| `frameSizeLimit` (NEG-MSG payload) | **500_000 bytes** | Hard-coded `Negentropy ne(storage, 500'000)` |
|
||||
| `relay__negentropy__maxSyncEvents` | **1_000_000** | Hard cap on items in the snapshot |
|
||||
| `relay__maxSubsPerConnection` | **200** | Shared between REQ and NEG sessions |
|
||||
| `idSize` on the wire | **32 bytes** | NIP-77 v1 (`PROTOCOL_VERSION = 0x61`) |
|
||||
| Default `since` window | **none** | Filter is honored as-is |
|
||||
| Filter parser | same as REQ | Honors `limit`, `kinds`, `authors`, `#tags` |
|
||||
| Snapshot data | `(created_at, id)` only | LMDB scan inserts into `negentropy::storage::Vector` |
|
||||
|
||||
Two-phase materialisation in strfry: `QueryScheduler` scans LMDB
|
||||
asynchronously, batching matched level-ids into a `vector<uint64_t>`;
|
||||
on completion the worker pulls each event header, calls
|
||||
`storageVector.insert(packed.created_at(), packed.id())`, then
|
||||
`seal()`s. The session response is sent only after seal.
|
||||
|
||||
Our equivalent must (a) match the snapshot footprint of ~40 bytes per
|
||||
event, and (b) match the wire-level frame-cap so a single
|
||||
reconciliation round-trip exchanges the same payload size.
|
||||
|
||||
## Sketch
|
||||
|
||||
### A — id-and-time-only snapshot path
|
||||
### A — id-and-time-only snapshot path (memory parity)
|
||||
|
||||
Negentropy only needs `(createdAt, id)` pairs. Add a streaming
|
||||
`IEventStore.queryIdAndTime(filter)` that returns
|
||||
`Sequence<Pair<Long, ByteArray>>` (or a `Flow` of small chunks) —
|
||||
no content/tags/sig, no Event allocation. SQLite path is a SELECT
|
||||
on `event_headers` (the `created_at`, `id` columns are already
|
||||
indexed for query plans).
|
||||
`IEventStore.snapshotIdsForNegentropy(filter)` that returns
|
||||
`Sequence<IdAndTime>` (`data class IdAndTime(val createdAt: Long, val id: ByteArray)`)
|
||||
— no content/tags/sig, no `Event` allocation. The SQLite path is a
|
||||
plain `SELECT id, created_at FROM event_headers WHERE …` against the
|
||||
existing `query_by_created_at_id` index
|
||||
(`quartz/src/commonMain/kotlin/com/vitorpamplona/quartz/nip01Core/store/sqlite/EventIndexesModule.kt:79`).
|
||||
|
||||
```kotlin
|
||||
suspend fun snapshotIdsForNegentropy(filter: Filter): IdTimeStream
|
||||
// IEventStore
|
||||
suspend fun snapshotIdsForNegentropy(filters: List<Filter>): List<IdAndTime>
|
||||
|
||||
// LiveEventStore — already deduplicates union of multi-filter results
|
||||
suspend fun snapshotIdsForNegentropy(filters: List<Filter>): List<IdAndTime>
|
||||
```
|
||||
|
||||
`NegentropyServerSession` is rewritten to take that stream and feed
|
||||
it directly into the `StorageVector`. Memory drops from O(N × 1 KB)
|
||||
to O(N × 40 B) — a 25× reduction; for 5 M events, ~200 MB instead
|
||||
of 5 GB.
|
||||
`NegentropyServerSession` is rewritten to take that list (or a
|
||||
two-phase `pendingIds → seal` builder) and feed it directly into
|
||||
`StorageVector`. Memory drops from O(N × ~1 KB) to O(N × 40 B) — for
|
||||
1 M events, ~40 MB instead of ~1 GB; matches strfry's per-session
|
||||
footprint.
|
||||
|
||||
### B — bounded-window subscriptions
|
||||
**id encoding:** strfry stores 32-byte raw ids (NIP-77 v1
|
||||
`ID_SIZE = 32`); the 16-byte truncation is `FINGERPRINT_SIZE`, only used
|
||||
internally for SHA-256 accumulator output. The current Kotlin code
|
||||
already passes the hex `event.id` string to `storage.insert(..)`
|
||||
(`NegentropyServerSession.kt:50`); the kmp-negentropy library decodes
|
||||
that to 32 raw bytes. Confirm this is preserved when we switch to a
|
||||
`ByteArray` id input — do not pre-truncate.
|
||||
|
||||
Most NEG-OPENs from real Nostr clients want the last 30 days, not
|
||||
"everything." If the client doesn't supply `since`, the server can
|
||||
default to a configurable horizon (e.g. 90 days) and surface this in
|
||||
the NIP-11 `limitation.negentropy_max_lookback_seconds` field.
|
||||
Operators can lift the cap; clients reading the doc know the bound.
|
||||
### B — bounded snapshot size, NOT a default `since` window
|
||||
|
||||
This is a NIP-spec-adjacent question more than a code change — needs
|
||||
a comment on whether the spec allows it. nostr-rs-relay does this
|
||||
already.
|
||||
The previous draft of this plan suggested defaulting `since` to a 90-day
|
||||
horizon. **Drop that.** strfry doesn't do it — the filter is honored
|
||||
as-is — and silently bounding `since` would break interop with strfry's
|
||||
sync clients (e.g. `strfry sync`, nostr-sdk's negentropy reconciler):
|
||||
they ask for "everything" and rely on getting it.
|
||||
|
||||
### C — frame-size cap on NEG-MSG
|
||||
Instead match strfry's protection: a hard cap on the number of items
|
||||
that go into a single snapshot.
|
||||
|
||||
```toml
|
||||
[negentropy]
|
||||
max_sync_events = 1_000_000 # matches strfry's relay__negentropy__maxSyncEvents
|
||||
```
|
||||
|
||||
`NegSessionRegistry.open` checks `count >= max_sync_events` after the
|
||||
SQLite count or during scan, and on overflow sends:
|
||||
|
||||
```
|
||||
["NEG-ERR", "<subId>", "blocked: too many query results"]
|
||||
```
|
||||
|
||||
Exact wording matches strfry so client error-handling that string-matches
|
||||
behaves identically. (Spec doesn't normatively define error text; this is
|
||||
de-facto interop.)
|
||||
|
||||
### C — frame-size cap on NEG-MSG: 500_000 bytes (strfry parity)
|
||||
|
||||
`NegentropyServerSession` is constructed with `frameSizeLimit = 0`
|
||||
(no limit). At very large reconciliations the message can grow large.
|
||||
Set a default `frameSizeLimit = 64 * 1024` (matching the typical WS
|
||||
frame budget) so NEG-MSGs don't blow past `[limits].max_ws_frame_bytes`.
|
||||
today. **Set `frameSizeLimit = 500_000`** (matching strfry) so a single
|
||||
NEG-MSG round-trip carries the same payload as strfry's. 64 KB — what
|
||||
the previous draft suggested — would force 8× more round-trips for
|
||||
large reconciliations and noticeably slow sync against strfry-style
|
||||
clients that expect ~1 MB hex-encoded NEG-MSGs.
|
||||
|
||||
The library already supports this — pure config change in
|
||||
`NegSessionRegistry.open`.
|
||||
The kmp-negentropy library enforces `frameSizeLimit >= 4096` (or `0`
|
||||
for unlimited). 500_000 is well above the floor. Pure config change in
|
||||
`NegSessionRegistry.open`; expose via `[negentropy].frame_size_limit`
|
||||
for operators who tune it down to fit smaller WS frame budgets.
|
||||
|
||||
### D — concurrent NEG-OPEN cap
|
||||
Note: `LimitsSection.max_ws_frame_bytes`
|
||||
(`geode/src/main/kotlin/com/vitorpamplona/geode/config/RelayConfig.kt:148`)
|
||||
applies to the WebSocket layer. After hex-encoding a 500_000-byte
|
||||
negentropy payload doubles to ~1_000_000 bytes on the wire; ensure
|
||||
`max_ws_frame_bytes` is at least 2 MB (or unlimited) in default
|
||||
config so the response isn't truncated by the WS layer.
|
||||
|
||||
### D — concurrent NEG-OPEN cap, shared with REQ
|
||||
|
||||
A NEG-OPEN holds session state until NEG-CLOSE (or connection close).
|
||||
Today nothing caps the number of concurrent open negentropy sessions
|
||||
per connection. A misbehaving (or hostile) client could open thousands
|
||||
and pin RAM. Add `MAX_NEG_SESSIONS_PER_CONNECTION = 16`, send NEG-ERR
|
||||
on overflow.
|
||||
Today `NegSessionRegistry.sessions` is unbounded. strfry caps at 200
|
||||
sessions per connection **shared with REQ** — both count against
|
||||
`relay__maxSubsPerConnection`.
|
||||
|
||||
Implement as:
|
||||
- Reuse the existing per-connection REQ subscription cap (or introduce
|
||||
one if absent) and let NEG sessions consume the same budget.
|
||||
- Default cap: **200** to match strfry. Configurable via
|
||||
`[limits].max_subs_per_connection`.
|
||||
- On overflow strfry sends a NOTICE, not NEG-ERR:
|
||||
`["NOTICE", "too many concurrent NEG requests"]`. We should match
|
||||
this — `NEG-ERR` is reserved for per-session protocol errors in
|
||||
strfry's model.
|
||||
|
||||
### E — pre-built fingerprint tree (follow-up, not in scope here)
|
||||
|
||||
strfry's real production-scale advantage is the **`negentropy::storage::BTreeLMDB`** backend: a persistent, incrementally-maintained B-tree
|
||||
of `(timestamp, id)` keys with per-node fingerprint accumulators.
|
||||
When NEG-OPEN's filter string matches a pre-registered
|
||||
`NegentropyFilter` tree, strfry skips the materialise-and-seal step
|
||||
entirely and reconciles directly off the LMDB B-tree in O(log n)
|
||||
fingerprint computations. See `env.foreach_NegentropyFilter` and
|
||||
`addStatelessView` in `RelayNegentropy.cpp`.
|
||||
|
||||
Equivalent for geode: a `quartz/.../nip77Negentropy/PrebuiltStorage`
|
||||
backed by a SQLite-side incremental fingerprint index, registered per
|
||||
canonical filter. This is a substantial piece of work and **out of
|
||||
scope for this plan** — call it out for a follow-up
|
||||
(`geode/plans/2026-05-08-negentropy-prebuilt-tree.md`). The A+B+C+D
|
||||
combination is enough to match strfry's `MemoryView` path, which is
|
||||
what 99% of ad-hoc NEG-OPENs hit.
|
||||
|
||||
## Concrete error-string interop table
|
||||
|
||||
Match strfry verbatim — clients in the wild string-match on these:
|
||||
|
||||
| Condition | strfry frame |
|
||||
|---------------------------------------|-----------------------------------------------------------|
|
||||
| Snapshot exceeds `max_sync_events` | `["NEG-ERR", "<subId>", "blocked: too many query results"]` |
|
||||
| NEG-MSG for unknown subId | `["NEG-ERR", "<subId>", "closed: unknown subscription handle"]` |
|
||||
| Library `reconcile()` parse failure | `["NEG-ERR", "<subId>", "PROTOCOL-ERROR"]` |
|
||||
| Per-connection sub cap exceeded | `["NOTICE", "too many concurrent NEG requests"]` |
|
||||
| NEG-MSG before NEG-OPEN seal complete | `["NOTICE", "negentropy error: got NEG-MSG before NEG-OPEN complete"]` |
|
||||
|
||||
The current Kotlin code in `NegSessionRegistry.kt:82` sends
|
||||
`"error: no negentropy session for <subId>"` for the unknown-subId
|
||||
case. Update to strfry's wording.
|
||||
|
||||
## Wire-level conformance checks
|
||||
|
||||
Before merging, verify against strfry as ground truth:
|
||||
|
||||
1. **Round-trip with `strfry sync`**: stand up a small geode instance,
|
||||
point `strfry sync ws://geode-host` at it, confirm sync completes
|
||||
and converges in ≤ comparable round-trips for an N=100k corpus.
|
||||
2. **idSize**: assert kmp-negentropy round-trips full 32-byte ids;
|
||||
the 16-byte fingerprint stays internal to the library.
|
||||
3. **Protocol version byte**: NIP-77 v1 = `0x61`. Confirm
|
||||
`NegentropyServerSession.processMessage` neither emits nor accepts
|
||||
a different version byte. Negotiation happens inside the library;
|
||||
surface the version on `NIP-11.limitation.negentropy = 1` so clients
|
||||
know the v1 path is supported.
|
||||
|
||||
## How to verify
|
||||
|
||||
Add to `geode.perf.LoadBenchmark`:
|
||||
Add to `geode/src/test/kotlin/com/vitorpamplona/geode/perf/LoadBenchmark.kt`:
|
||||
|
||||
- `negentropyOpenLatencyLargeCorpus` — preload 1 M events (use
|
||||
fixtures), measure NEG-OPEN → first NEG-MSG latency. Target <100 ms.
|
||||
- `negentropyOpenLatencyLargeCorpus` — preload 1 M events, measure
|
||||
NEG-OPEN → first NEG-MSG latency. Target **<200 ms** (strfry's
|
||||
C++ `MemoryView` path on equivalent hardware does ~80–150 ms;
|
||||
we expect a 1.5–2× JVM tax).
|
||||
- `negentropyMemoryPressure` — open 10 concurrent NEG-OPENs on the
|
||||
same large corpus; measure RSS delta, target <500 MB.
|
||||
same large corpus; measure RSS delta. Target **<500 MB**
|
||||
(10 × ~40 MB session footprint + scan overhead).
|
||||
- `negentropyStrfryInterop` — programmatic round-trip against a
|
||||
containerised `hoytech/strfry`: same fixture corpus loaded in both,
|
||||
cross-sync, assert byte-identical id-set convergence in equal
|
||||
round-trips ±1.
|
||||
|
||||
Add to `quartz/.../nip77Negentropy/`:
|
||||
|
||||
- `NegentropyServerSessionTest.processMessage_atFrameLimit_splitsAcrossRounds`
|
||||
— large symmetric difference, assert each NEG-MSG payload is
|
||||
≤ 500_000 bytes and the session completes in N rounds (not 1).
|
||||
|
||||
## Risks
|
||||
|
||||
- **`Sequence`/`Flow` over SQLite cursor**: holding a cursor open
|
||||
across the full sync is fragile if the client stalls. Materialise
|
||||
to a smaller in-memory list (just (id, createdAt)) once, reuse for
|
||||
the lifetime of the session. Memory bound is the same.
|
||||
- **Defaulting `since` is a behaviour change**: existing clients that
|
||||
expect "everything" silently get a bounded window. Either (a) make
|
||||
it opt-in via `RelayConfig.NegentropySection.default_lookback_seconds
|
||||
= null`, (b) advertise the cap in NIP-11 so well-behaved clients
|
||||
read it.
|
||||
- **Frame-size cap can break older clients**: the NIP-77 reference
|
||||
implementation (kmp-negentropy) handles this gracefully — multi-frame
|
||||
reconciliation is in spec — but field-test against a known-working
|
||||
client (e.g. nstart, primal-cache) before flipping the default.
|
||||
- **Cursor lifetime**: holding a SQLite cursor open across the full
|
||||
sync is fragile if the client stalls. Materialise to a smaller
|
||||
in-memory `List<IdAndTime>` (40 bytes/entry) once at NEG-OPEN time,
|
||||
reuse for the lifetime of the session. Bounded by `max_sync_events`.
|
||||
- **Frame-size 500_000 vs WS frame budget**: hex-encoded payload is
|
||||
~1 MB. If `LimitsSection.max_ws_frame_bytes` is set lower than
|
||||
~1.5 MB the response gets truncated/rejected by the WS layer. Lift
|
||||
the WS default OR cap `frame_size_limit` to
|
||||
`max_ws_frame_bytes / 2` at startup; fail-fast log a warning if the
|
||||
operator's config makes negentropy unusable.
|
||||
- **No default `since` (intentional, but worth flagging)**: a hostile
|
||||
client doing `NEG-OPEN {kinds:[1]}` against a large corpus will hit
|
||||
`max_sync_events` and get NEG-ERR. The cap is the protection; do
|
||||
not also silently bound `since`.
|
||||
- **kmp-negentropy library version**: pinned to `v1.0.2`
|
||||
(`gradle/libs.versions.toml:9`). Confirm v1.0.2 enforces
|
||||
`frameSizeLimit >= 4096`, supports protocol byte `0x61`, and
|
||||
internal `ID_SIZE = 32`. If any of those don't match strfry, this
|
||||
plan needs an upstream fix on kmp-negentropy first.
|
||||
|
||||
@@ -28,9 +28,11 @@ import com.vitorpamplona.quartz.nip01Core.relay.server.policies.FullAuthPolicy
|
||||
import com.vitorpamplona.quartz.nip01Core.relay.server.policies.KindAllowDenyPolicy
|
||||
import com.vitorpamplona.quartz.nip01Core.relay.server.policies.PubkeyAllowDenyPolicy
|
||||
import com.vitorpamplona.quartz.nip01Core.relay.server.policies.RejectFutureEventsPolicy
|
||||
import com.vitorpamplona.quartz.nip01Core.relay.server.policies.VerifyAuthOnlyPolicy
|
||||
import com.vitorpamplona.quartz.nip01Core.relay.server.policies.VerifyPolicy
|
||||
import com.vitorpamplona.quartz.nip01Core.store.IEventStore
|
||||
import com.vitorpamplona.quartz.nip01Core.store.sqlite.EventStore
|
||||
import com.vitorpamplona.quartz.nip77Negentropy.NegentropySettings
|
||||
import java.io.File
|
||||
|
||||
/**
|
||||
@@ -83,6 +85,12 @@ fun main(args: Array<String>) {
|
||||
// opts out (CLI `--no-verify` or `[options].verify_signatures = false`
|
||||
// in the config).
|
||||
val verifySigs = !a.flag("--no-verify") && config.options.verify_signatures
|
||||
// Parallel verify is on whenever signature checking is on; the
|
||||
// IngestQueue handles it instead of VerifyPolicy. Operators can
|
||||
// force the legacy in-policy path with `--no-parallel-verify` or
|
||||
// `[options].parallel_verify = false`.
|
||||
val parallelVerify =
|
||||
verifySigs && !a.flag("--no-parallel-verify") && config.options.parallel_verify
|
||||
|
||||
// Advertised URL: explicit `info.relay_url` wins, then build from
|
||||
// host/port/path. 0.0.0.0 bind → 127.0.0.1 in the URL so NIP-42
|
||||
@@ -98,11 +106,26 @@ fun main(args: Array<String>) {
|
||||
val store: IEventStore = EventStore(dbName = dbFile, relay = advertisedUrl)
|
||||
|
||||
val policyBuilder: () -> IRelayPolicy = {
|
||||
composePolicy(config, advertisedUrl, requireAuth, verifySigs)
|
||||
composePolicy(config, advertisedUrl, requireAuth, verifySigs, parallelVerify)
|
||||
}
|
||||
|
||||
val stateFile = config.admin.state_file?.let { File(it) }
|
||||
val relay = Relay(advertisedUrl, store, info, policyBuilder, stateFile = stateFile)
|
||||
val negentropySettings =
|
||||
NegentropySettings(
|
||||
frameSizeLimit = config.negentropy.frame_size_limit,
|
||||
maxSyncEvents = config.negentropy.max_sync_events,
|
||||
maxSessionsPerConnection = config.negentropy.max_sessions_per_connection,
|
||||
)
|
||||
val relay =
|
||||
Relay(
|
||||
advertisedUrl,
|
||||
store,
|
||||
info,
|
||||
policyBuilder,
|
||||
stateFile = stateFile,
|
||||
parallelVerify = parallelVerify,
|
||||
negentropySettings = negentropySettings,
|
||||
)
|
||||
// Frame cap honors max_ws_frame_bytes when set; max_ws_message_bytes
|
||||
// is treated as the same cap (Ktor's WebSockets plugin only exposes
|
||||
// a single per-frame limit; multi-frame messages remain unbounded).
|
||||
@@ -151,6 +174,7 @@ private fun composePolicy(
|
||||
advertisedUrl: com.vitorpamplona.quartz.nip01Core.relay.normalizer.NormalizedRelayUrl,
|
||||
requireAuth: Boolean,
|
||||
verifySigs: Boolean,
|
||||
parallelVerify: Boolean,
|
||||
): IRelayPolicy {
|
||||
val pieces = mutableListOf<IRelayPolicy>()
|
||||
|
||||
@@ -171,7 +195,11 @@ private fun composePolicy(
|
||||
}
|
||||
|
||||
if (verifySigs) {
|
||||
pieces += VerifyPolicy
|
||||
// When parallel verify is on, the IngestQueue handles EVENT
|
||||
// verification on the writer's CPU fan-out — but AUTH events
|
||||
// bypass the queue, so we still need the policy chain to
|
||||
// verify those. `VerifyAuthOnlyPolicy` does exactly that.
|
||||
pieces += if (parallelVerify) VerifyAuthOnlyPolicy else VerifyPolicy
|
||||
}
|
||||
|
||||
return pieces.fold<IRelayPolicy, IRelayPolicy>(EmptyPolicy) { acc, p ->
|
||||
|
||||
@@ -33,6 +33,7 @@ import com.vitorpamplona.quartz.nip01Core.relay.server.policies.EmptyPolicy
|
||||
import com.vitorpamplona.quartz.nip01Core.store.IEventStore
|
||||
import com.vitorpamplona.quartz.nip01Core.store.sqlite.EventStore
|
||||
import com.vitorpamplona.quartz.nip11RelayInfo.Nip11RelayInformation
|
||||
import com.vitorpamplona.quartz.nip77Negentropy.NegentropySettings
|
||||
import com.vitorpamplona.quartz.nip86RelayManagement.server.BanListPolicy
|
||||
import com.vitorpamplona.quartz.nip86RelayManagement.server.BanStore
|
||||
import kotlinx.coroutines.SupervisorJob
|
||||
@@ -73,6 +74,22 @@ class Relay(
|
||||
* everything in memory only — fine for tests.
|
||||
*/
|
||||
stateFile: File? = null,
|
||||
/**
|
||||
* Run Schnorr signature verification in parallel inside the
|
||||
* [com.vitorpamplona.quartz.nip01Core.relay.server.IngestQueue]
|
||||
* instead of serially in the policy chain. Enables the Tier-3
|
||||
* win in `geode/plans/2026-05-07-event-ingestion-batching.md`.
|
||||
*
|
||||
* When set, callers MUST omit `VerifyPolicy` from [policyBuilder]
|
||||
* — having both verifies the same event twice for no benefit.
|
||||
* `Main.kt` skips `VerifyPolicy` when this flag is on.
|
||||
*/
|
||||
parallelVerify: Boolean = false,
|
||||
/**
|
||||
* NIP-77 server-side tuning (frame cap, snapshot cap,
|
||||
* per-connection session cap). Defaults to strfry-parity values.
|
||||
*/
|
||||
negentropySettings: NegentropySettings = NegentropySettings.Default,
|
||||
) : AutoCloseable {
|
||||
private val stateStore: RelayStateStore? = stateFile?.let { RelayStateStore(it) }
|
||||
|
||||
@@ -157,7 +174,9 @@ class Relay(
|
||||
val user = policyBuilder()
|
||||
if (user === EmptyPolicy) BanListPolicy(banStore) else user + BanListPolicy(banStore)
|
||||
},
|
||||
parentContext,
|
||||
parentContext = parentContext,
|
||||
parallelVerify = parallelVerify,
|
||||
negentropySettings = negentropySettings,
|
||||
)
|
||||
|
||||
/**
|
||||
|
||||
@@ -28,6 +28,7 @@ import com.vitorpamplona.quartz.nip01Core.relay.server.policies.EmptyPolicy
|
||||
import com.vitorpamplona.quartz.nip01Core.relay.sockets.WebSocket
|
||||
import com.vitorpamplona.quartz.nip01Core.relay.sockets.WebSocketListener
|
||||
import com.vitorpamplona.quartz.nip01Core.relay.sockets.WebsocketBuilder
|
||||
import com.vitorpamplona.quartz.nip77Negentropy.NegentropySettings
|
||||
import java.util.concurrent.ConcurrentHashMap
|
||||
|
||||
/**
|
||||
@@ -50,6 +51,7 @@ import java.util.concurrent.ConcurrentHashMap
|
||||
*/
|
||||
class RelayHub(
|
||||
private val defaultPolicy: () -> IRelayPolicy = { EmptyPolicy },
|
||||
private val negentropySettings: NegentropySettings = NegentropySettings.Default,
|
||||
) : WebsocketBuilder,
|
||||
AutoCloseable {
|
||||
private val relays = ConcurrentHashMap<NormalizedRelayUrl, Relay>()
|
||||
@@ -60,7 +62,11 @@ class RelayHub(
|
||||
fun getOrCreate(url: NormalizedRelayUrl): Relay {
|
||||
check(!closed) { "RelayHub has been closed" }
|
||||
return relays.getOrPut(url) {
|
||||
Relay(url = url, policyBuilder = defaultPolicy)
|
||||
Relay(
|
||||
url = url,
|
||||
policyBuilder = defaultPolicy,
|
||||
negentropySettings = negentropySettings,
|
||||
)
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@@ -43,6 +43,7 @@ data class RelayConfig(
|
||||
val limits: LimitsSection = LimitsSection(),
|
||||
val authorization: AuthorizationSection = AuthorizationSection(),
|
||||
val admin: AdminSection = AdminSection(),
|
||||
val negentropy: NegentropySection = NegentropySection(),
|
||||
) {
|
||||
/**
|
||||
* Maps the `[info]` section into a [RelayInfo] used by the NIP-11
|
||||
@@ -141,6 +142,18 @@ data class RelayConfig(
|
||||
* for trusted-input scenarios (test fixtures, mirror replays).
|
||||
*/
|
||||
val verify_signatures: Boolean = true,
|
||||
/**
|
||||
* Run signature verification in parallel inside the IngestQueue
|
||||
* (CPU fan-out across `Dispatchers.Default`) instead of serially
|
||||
* on each connection's WebSocket pump. Tier-3 of the
|
||||
* `event-ingestion-batching` plan. Wins scale with how many
|
||||
* EVENTs a single connection sends back-to-back: ~CPU_COUNT×
|
||||
* verify-step speed-up on burst publishes. Set false to keep
|
||||
* the legacy in-policy verify path.
|
||||
*
|
||||
* Only takes effect when [verify_signatures] is also true.
|
||||
*/
|
||||
val parallel_verify: Boolean = true,
|
||||
)
|
||||
|
||||
data class LimitsSection(
|
||||
@@ -148,6 +161,34 @@ data class RelayConfig(
|
||||
val max_ws_frame_bytes: Int? = null,
|
||||
)
|
||||
|
||||
/**
|
||||
* NIP-77 negentropy tuning. Defaults track strfry
|
||||
* (`hoytech/strfry`) so a Geode relay accepts the same workload
|
||||
* shape and exchanges the same NEG-MSG round-trip size as
|
||||
* strfry — the de-facto reference implementation.
|
||||
*
|
||||
* - [frame_size_limit] mirrors strfry's hard-coded
|
||||
* `Negentropy ne(storage, 500'000)` in `RelayNegentropy.cpp`.
|
||||
* Hex-encoded that's ~1 MB on the wire per NEG-MSG; ensure
|
||||
* `[limits].max_ws_frame_bytes` (when set) is at least double
|
||||
* this or NEG-MSGs get truncated by the WS layer.
|
||||
* - [max_sync_events] mirrors strfry's
|
||||
* `relay__negentropy__maxSyncEvents`. NEG-OPEN whose snapshot
|
||||
* exceeds this returns
|
||||
* `["NEG-ERR", "<subId>", "blocked: too many query results"]`.
|
||||
* - [max_sessions_per_connection] caps concurrent NEG-OPEN
|
||||
* sessions held by a single connection. strfry shares its
|
||||
* 200-cap with REQ subs via `relay__maxSubsPerConnection`;
|
||||
* Geode counts NEG independently for now (REQ has no cap yet).
|
||||
* Overflow returns NOTICE
|
||||
* `"too many concurrent NEG requests"` (matches strfry).
|
||||
*/
|
||||
data class NegentropySection(
|
||||
val frame_size_limit: Long = 500_000L,
|
||||
val max_sync_events: Int = 1_000_000,
|
||||
val max_sessions_per_connection: Int = 200,
|
||||
)
|
||||
|
||||
data class AuthorizationSection(
|
||||
val pubkey_whitelist: List<String> = emptyList(),
|
||||
val pubkey_blacklist: List<String> = emptyList(),
|
||||
|
||||
@@ -24,6 +24,7 @@ import com.vitorpamplona.quartz.nip01Core.core.Event
|
||||
import com.vitorpamplona.quartz.nip01Core.core.OptimizedJsonMapper
|
||||
import com.vitorpamplona.quartz.nip01Core.crypto.KeyPair
|
||||
import com.vitorpamplona.quartz.nip01Core.relay.commands.toClient.Message
|
||||
import com.vitorpamplona.quartz.nip01Core.relay.commands.toClient.NoticeMessage
|
||||
import com.vitorpamplona.quartz.nip01Core.relay.filters.Filter
|
||||
import com.vitorpamplona.quartz.nip01Core.relay.normalizer.NormalizedRelayUrl
|
||||
import com.vitorpamplona.quartz.nip01Core.relay.normalizer.RelayUrlNormalizer
|
||||
@@ -33,6 +34,7 @@ import com.vitorpamplona.quartz.nip10Notes.TextNoteEvent
|
||||
import com.vitorpamplona.quartz.nip77Negentropy.NegErrMessage
|
||||
import com.vitorpamplona.quartz.nip77Negentropy.NegMsgMessage
|
||||
import com.vitorpamplona.quartz.nip77Negentropy.NegentropySession
|
||||
import com.vitorpamplona.quartz.nip77Negentropy.NegentropySettings
|
||||
import kotlinx.coroutines.channels.Channel
|
||||
import kotlinx.coroutines.channels.Channel.Factory.UNLIMITED
|
||||
import kotlinx.coroutines.runBlocking
|
||||
@@ -237,12 +239,82 @@ class Nip77NegentropyTest {
|
||||
val response = client.nextMessage()
|
||||
assertTrue(response is NegErrMessage, "expected NEG-ERR, got ${response::class.simpleName}")
|
||||
assertEquals("ghost-sub", response.subId)
|
||||
assertTrue(response.reason.contains("no negentropy session"))
|
||||
// strfry-parity wording — clients in the wild string-match this.
|
||||
assertEquals("closed: unknown subscription handle", response.reason)
|
||||
} finally {
|
||||
client.close()
|
||||
}
|
||||
}
|
||||
|
||||
@Test
|
||||
fun negOpenSnapshotOverflowReturnsStrFryNegErr() =
|
||||
runBlocking {
|
||||
// Tiny cap so the test is fast. Preload more events than the
|
||||
// cap so NEG-OPEN must reject — strfry's parity behaviour for
|
||||
// `relay__negentropy__maxSyncEvents`.
|
||||
val capped = RelayHub(negentropySettings = NegentropySettings(maxSyncEvents = 5))
|
||||
try {
|
||||
val capUrl = RelayUrlNormalizer.normalize("ws://127.0.0.1:7771/")
|
||||
val events = makeEvents(20)
|
||||
capped.getOrCreate(capUrl).preload(events)
|
||||
|
||||
val client = WireClient(capped, capUrl)
|
||||
try {
|
||||
val session =
|
||||
NegentropySession(
|
||||
subId = "neg-overflow",
|
||||
filter = Filter(kinds = listOf(1)),
|
||||
localEvents = emptyList(),
|
||||
)
|
||||
client.send(OptimizedJsonMapper.toJson(session.open()))
|
||||
|
||||
val response = client.nextMessage()
|
||||
assertTrue(response is NegErrMessage, "expected NEG-ERR, got ${response::class.simpleName}")
|
||||
assertEquals("neg-overflow", response.subId)
|
||||
// strfry-parity wording.
|
||||
assertEquals("blocked: too many query results", response.reason)
|
||||
} finally {
|
||||
client.close()
|
||||
}
|
||||
} finally {
|
||||
capped.close()
|
||||
}
|
||||
}
|
||||
|
||||
@Test
|
||||
fun negOpenPerConnectionCapEmitsNotice() =
|
||||
runBlocking {
|
||||
// Cap = 2, so the third NEG-OPEN on one connection should
|
||||
// be rejected with a NOTICE (matching strfry's wording).
|
||||
val capped = RelayHub(negentropySettings = NegentropySettings(maxSessionsPerConnection = 2))
|
||||
try {
|
||||
val capUrl = RelayUrlNormalizer.normalize("ws://127.0.0.1:7772/")
|
||||
capped.getOrCreate(capUrl).preload(makeEvents(3))
|
||||
|
||||
val client = WireClient(capped, capUrl)
|
||||
try {
|
||||
repeat(2) { i ->
|
||||
val s = NegentropySession("ok-$i", Filter(kinds = listOf(1)), localEvents = emptyList())
|
||||
client.send(OptimizedJsonMapper.toJson(s.open()))
|
||||
// Drain the NEG-MSG response so the next OPEN goes
|
||||
// through cleanly.
|
||||
client.nextMessage() as NegMsgMessage
|
||||
}
|
||||
|
||||
// Third OPEN — should be rejected with a NOTICE.
|
||||
val third = NegentropySession("third", Filter(kinds = listOf(1)), localEvents = emptyList())
|
||||
client.send(OptimizedJsonMapper.toJson(third.open()))
|
||||
val response = client.nextMessage()
|
||||
assertTrue(response is NoticeMessage, "expected NOTICE, got ${response::class.simpleName}")
|
||||
assertEquals("too many concurrent NEG requests", response.message)
|
||||
} finally {
|
||||
client.close()
|
||||
}
|
||||
} finally {
|
||||
capped.close()
|
||||
}
|
||||
}
|
||||
|
||||
@Test
|
||||
fun negOpenWithSameSubIdReplacesPriorSession() =
|
||||
runBlocking {
|
||||
|
||||
+186
@@ -0,0 +1,186 @@
|
||||
/*
|
||||
* Copyright (c) 2025 Vitor Pamplona
|
||||
*
|
||||
* Permission is hereby granted, free of charge, to any person obtaining a copy of
|
||||
* this software and associated documentation files (the "Software"), to deal in
|
||||
* the Software without restriction, including without limitation the rights to use,
|
||||
* copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the
|
||||
* Software, and to permit persons to whom the Software is furnished to do so,
|
||||
* subject to the following conditions:
|
||||
*
|
||||
* The above copyright notice and this permission notice shall be included in all
|
||||
* copies or substantial portions of the Software.
|
||||
*
|
||||
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
||||
* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS
|
||||
* FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR
|
||||
* COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN
|
||||
* AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
|
||||
* WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
|
||||
*/
|
||||
package com.vitorpamplona.geode.interop
|
||||
|
||||
import com.vitorpamplona.geode.LocalRelayServer
|
||||
import com.vitorpamplona.geode.Relay
|
||||
import com.vitorpamplona.quartz.nip01Core.core.Event
|
||||
import com.vitorpamplona.quartz.nip01Core.core.HexKey
|
||||
import com.vitorpamplona.quartz.nip01Core.crypto.KeyPair
|
||||
import com.vitorpamplona.quartz.nip01Core.relay.client.NostrClient
|
||||
import com.vitorpamplona.quartz.nip01Core.relay.client.accessories.fetchAll
|
||||
import com.vitorpamplona.quartz.nip01Core.relay.client.accessories.publishAndConfirm
|
||||
import com.vitorpamplona.quartz.nip01Core.relay.filters.Filter
|
||||
import com.vitorpamplona.quartz.nip01Core.relay.normalizer.NormalizedRelayUrl
|
||||
import com.vitorpamplona.quartz.nip01Core.relay.normalizer.normalizeRelayUrl
|
||||
import com.vitorpamplona.quartz.nip01Core.relay.sockets.okhttp.BasicOkHttpWebSocket
|
||||
import com.vitorpamplona.quartz.nip01Core.signers.NostrSignerSync
|
||||
import com.vitorpamplona.quartz.nip10Notes.TextNoteEvent
|
||||
import kotlinx.coroutines.CoroutineScope
|
||||
import kotlinx.coroutines.Dispatchers
|
||||
import kotlinx.coroutines.SupervisorJob
|
||||
import kotlinx.coroutines.cancel
|
||||
import kotlinx.coroutines.runBlocking
|
||||
import okhttp3.OkHttpClient
|
||||
import kotlin.test.AfterTest
|
||||
import kotlin.test.BeforeTest
|
||||
import kotlin.test.Test
|
||||
import kotlin.test.assertEquals
|
||||
import kotlin.test.assertNull
|
||||
import kotlin.test.assertTrue
|
||||
|
||||
/**
|
||||
* The Kotlin counterpart to strfry's `test/syncTest.pl`: stand up two
|
||||
* Geode relays on real WebSocket endpoints, give each a partially
|
||||
* overlapping corpus, and converge them via NIP-77.
|
||||
*
|
||||
* The driver mirrors `strfry sync ws://other --dir both`:
|
||||
*
|
||||
* 1. Read the local relay's snapshot for the negotiated filter.
|
||||
* 2. Open NEG-OPEN against the remote with that snapshot.
|
||||
* 3. Drive NEG-MSG round trips until the client-side
|
||||
* [com.vitorpamplona.quartz.nip77Negentropy.NegentropySession]
|
||||
* reports completion.
|
||||
* 4. `needIds`: REQ them from the remote, insert into the local relay.
|
||||
* 5. `haveIds`: fetch from the local relay, publish to the remote.
|
||||
*
|
||||
* After the round, both relays must hold the union of the original
|
||||
* corpora. We assert via REQ on each side; an `id`-filter that returns
|
||||
* every event we expect, and nothing more, proves convergence
|
||||
* end-to-end through the NIP-77 server pipeline (`NegSessionRegistry`
|
||||
* → `NegentropyServerSession` → `IEventStore.snapshotIdsForNegentropy`).
|
||||
*
|
||||
* Equivalent to strfry's `runSyncTests.pl` "full DB sync" case at
|
||||
* small scale. Larger corpora belong in `LoadBenchmark`.
|
||||
*/
|
||||
class GeodeVsGeodeNegentropySyncTest {
|
||||
private lateinit var relayA: Relay
|
||||
private lateinit var relayB: Relay
|
||||
private lateinit var serverA: LocalRelayServer
|
||||
private lateinit var serverB: LocalRelayServer
|
||||
private lateinit var scope: CoroutineScope
|
||||
private lateinit var client: NostrClient
|
||||
private val httpClient = OkHttpClient.Builder().build()
|
||||
|
||||
@BeforeTest
|
||||
fun setup() {
|
||||
// The placeholder URLs are normalised so the relay accepts
|
||||
// them; ports come from the autobind below via [server.url].
|
||||
relayA = Relay(url = "ws://127.0.0.1:7771/".normalizeRelayUrl())
|
||||
relayB = Relay(url = "ws://127.0.0.1:7772/".normalizeRelayUrl())
|
||||
serverA = LocalRelayServer(relayA, host = "127.0.0.1", port = 0).start()
|
||||
serverB = LocalRelayServer(relayB, host = "127.0.0.1", port = 0).start()
|
||||
scope = CoroutineScope(Dispatchers.Default + SupervisorJob())
|
||||
client = NostrClient(BasicOkHttpWebSocket.Builder { _ -> httpClient }, scope)
|
||||
}
|
||||
|
||||
@AfterTest
|
||||
fun teardown() {
|
||||
client.disconnect()
|
||||
scope.cancel()
|
||||
serverA.stop()
|
||||
serverB.stop()
|
||||
relayA.close()
|
||||
relayB.close()
|
||||
}
|
||||
|
||||
/** Generates [count] signed text notes with monotonic createdAt. */
|
||||
private fun makeEvents(
|
||||
count: Int,
|
||||
seed: Long = 1_700_000_000L,
|
||||
): List<Event> {
|
||||
val signer = NostrSignerSync(KeyPair())
|
||||
return List(count) { i ->
|
||||
signer.sign(TextNoteEvent.build("event-$seed-$i", createdAt = seed + i))
|
||||
}
|
||||
}
|
||||
|
||||
private val driver by lazy { InteropSyncDriver(httpClient) }
|
||||
|
||||
@Test
|
||||
fun bidirectionalSyncConvergesTwoRelays() =
|
||||
runBlocking {
|
||||
// Universe of 20 events. Relay A holds [0..14], Relay B
|
||||
// holds [5..19]. Overlap [5..14], A-only [0..4], B-only
|
||||
// [15..19]. After bidirectional sync both must hold [0..19].
|
||||
val all = makeEvents(20)
|
||||
val aEvents = all.subList(0, 15)
|
||||
val bEvents = all.subList(5, 20)
|
||||
relayA.preload(aEvents)
|
||||
relayB.preload(bEvents)
|
||||
|
||||
val filter = Filter(kinds = listOf(1))
|
||||
val urlA = serverA.url.normalizeRelayUrl()
|
||||
val urlB = serverB.url.normalizeRelayUrl()
|
||||
|
||||
// B negotiates the symmetric difference with A.
|
||||
val diff = driver.negotiate(serverA.url, filter, bEvents)
|
||||
assertNull(diff.error, "negotiation must not error: ${diff.error}")
|
||||
// From B's perspective: needs A-only, has B-only.
|
||||
assertEquals(
|
||||
aEvents.subList(0, 5).map { it.id }.toSet(),
|
||||
diff.needIds,
|
||||
"B should NEED [0..4] from A",
|
||||
)
|
||||
assertEquals(
|
||||
bEvents.subList(10, 15).map { it.id }.toSet(),
|
||||
diff.haveIds,
|
||||
"B should announce HAVE for [15..19]",
|
||||
)
|
||||
|
||||
// Close the loop: fetch needs from A, push haves to A.
|
||||
val needFromA =
|
||||
client.fetchAll(relay = urlA, filter = Filter(ids = diff.needIds.toList()))
|
||||
relayB.preload(needFromA)
|
||||
|
||||
for (id in diff.haveIds) {
|
||||
client.publishAndConfirm(bEvents.first { it.id == id }, setOf(urlA))
|
||||
}
|
||||
|
||||
// --- Verify convergence ---
|
||||
val expected = all.map { it.id }.toSet()
|
||||
assertEquals(expected, idsOnRelay(urlA, filter), "Relay A should hold every event")
|
||||
assertEquals(expected, idsOnRelay(urlB, filter), "Relay B should hold every event")
|
||||
}
|
||||
|
||||
@Test
|
||||
fun negentropyConvergesInBoundedRoundsOnSmallCorpus() =
|
||||
runBlocking {
|
||||
val all = makeEvents(200)
|
||||
relayA.preload(all.subList(0, 150))
|
||||
val bEvents = all.subList(50, 200)
|
||||
relayB.preload(bEvents)
|
||||
|
||||
val res = driver.negotiate(serverA.url, Filter(kinds = listOf(1)), bEvents)
|
||||
assertNull(res.error)
|
||||
assertEquals(50, res.needIds.size, "B should need [0..49]")
|
||||
assertEquals(50, res.haveIds.size, "B should have [150..199]")
|
||||
// strfry typically converges these in ≤5 rounds; ≤16 is
|
||||
// generous headroom that still catches regressions.
|
||||
assertTrue(res.rounds <= 16, "expected ≤16 NEG-MSG rounds, got ${res.rounds}")
|
||||
}
|
||||
|
||||
/** Every event id matching [filter] visible on the relay at [url] via REQ. */
|
||||
private suspend fun idsOnRelay(
|
||||
url: NormalizedRelayUrl,
|
||||
filter: Filter,
|
||||
): Set<HexKey> = client.fetchAll(relay = url, filter = filter).map { it.id }.toSet()
|
||||
}
|
||||
+220
@@ -0,0 +1,220 @@
|
||||
/*
|
||||
* Copyright (c) 2025 Vitor Pamplona
|
||||
*
|
||||
* Permission is hereby granted, free of charge, to any person obtaining a copy of
|
||||
* this software and associated documentation files (the "Software"), to deal in
|
||||
* the Software without restriction, including without limitation the rights to use,
|
||||
* copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the
|
||||
* Software, and to permit persons to whom the Software is furnished to do so,
|
||||
* subject to the following conditions:
|
||||
*
|
||||
* The above copyright notice and this permission notice shall be included in all
|
||||
* copies or substantial portions of the Software.
|
||||
*
|
||||
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
||||
* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS
|
||||
* FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR
|
||||
* COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN
|
||||
* AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
|
||||
* WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
|
||||
*/
|
||||
package com.vitorpamplona.geode.interop
|
||||
|
||||
import com.vitorpamplona.quartz.nip01Core.core.Event
|
||||
import com.vitorpamplona.quartz.nip01Core.core.OptimizedJsonMapper
|
||||
import com.vitorpamplona.quartz.nip01Core.crypto.KeyPair
|
||||
import com.vitorpamplona.quartz.nip01Core.relay.filters.Filter
|
||||
import com.vitorpamplona.quartz.nip01Core.signers.NostrSignerSync
|
||||
import com.vitorpamplona.quartz.nip10Notes.TextNoteEvent
|
||||
import kotlinx.coroutines.channels.Channel
|
||||
import kotlinx.coroutines.channels.Channel.Factory.UNLIMITED
|
||||
import kotlinx.coroutines.runBlocking
|
||||
import kotlinx.coroutines.withTimeout
|
||||
import okhttp3.OkHttpClient
|
||||
import okhttp3.Request
|
||||
import okhttp3.Response
|
||||
import okhttp3.WebSocket
|
||||
import okhttp3.WebSocketListener
|
||||
import java.io.File
|
||||
import java.net.ServerSocket
|
||||
import java.net.Socket
|
||||
import kotlin.io.path.createTempDirectory
|
||||
import kotlin.test.AfterTest
|
||||
import kotlin.test.Test
|
||||
import kotlin.test.assertEquals
|
||||
import kotlin.test.assertNull
|
||||
import kotlin.test.assertTrue
|
||||
|
||||
/**
|
||||
* Reciprocal interop test: Geode's NIP-77 client driving a real
|
||||
* `strfry` instance.
|
||||
*
|
||||
* **Opt-in.** Skipped unless `STRFRY_BIN` env var (or
|
||||
* `-Dstrfry.bin=...`) points at a `strfry` binary. When unset, the
|
||||
* test prints a `[skip]` line and returns. Mirrors the gate
|
||||
* `LoadBenchmark` uses for `-DrunLoadBenchmark`:
|
||||
*
|
||||
* STRFRY_BIN=/usr/local/bin/strfry ./gradlew :geode:test \
|
||||
* --tests "*GeodeVsStrfryNegentropySyncTest*"
|
||||
*
|
||||
* The strfry process is booted on a free loopback port with a fresh
|
||||
* temp LMDB dir. We feed events into it via the NIP-01 EVENT wire
|
||||
* (no `strfry import`), then run [InteropSyncDriver] against it.
|
||||
* Strfry's `RelayNegentropy.cpp` answers the same NIP-77 wire we test
|
||||
* against Geode — passing both is byte-shape interop, not just
|
||||
* "passes our own tests".
|
||||
*/
|
||||
class GeodeVsStrfryNegentropySyncTest {
|
||||
private val strfryBin: String? =
|
||||
System.getenv("STRFRY_BIN") ?: System.getProperty("strfry.bin")
|
||||
private val enabled = strfryBin != null
|
||||
|
||||
private lateinit var strfryDir: File
|
||||
private var strfryProcess: Process? = null
|
||||
private val httpClient by lazy { OkHttpClient.Builder().build() }
|
||||
|
||||
@AfterTest
|
||||
fun teardown() {
|
||||
strfryProcess?.destroy()
|
||||
strfryProcess?.waitFor()
|
||||
if (::strfryDir.isInitialized) strfryDir.deleteRecursively()
|
||||
}
|
||||
|
||||
/**
|
||||
* Boots a strfry instance in a temp LMDB dir on a free port.
|
||||
* Writes the smallest config strfry accepts, starts the
|
||||
* subprocess, and polls until the WebSocket port is reachable.
|
||||
*
|
||||
* The config is intentionally minimal — strfry's defaults
|
||||
* (negentropy on, sane limits) are what we want to test against.
|
||||
* Adding speculative config keys risks failing on schema drift.
|
||||
*/
|
||||
private fun startStrfry(): String {
|
||||
val port = ServerSocket(0).use { it.localPort }
|
||||
strfryDir = createTempDirectory(prefix = "strfry-interop-").toFile()
|
||||
val configFile = File(strfryDir, "strfry.conf")
|
||||
configFile.writeText(
|
||||
"""
|
||||
db = "${strfryDir.absolutePath}/strfry-db"
|
||||
relay {
|
||||
bind = "127.0.0.1"
|
||||
port = $port
|
||||
}
|
||||
""".trimIndent(),
|
||||
)
|
||||
File(strfryDir, "strfry-db").mkdirs()
|
||||
strfryProcess =
|
||||
ProcessBuilder(strfryBin, "--config", configFile.absolutePath, "relay")
|
||||
.redirectErrorStream(true)
|
||||
.redirectOutput(File(strfryDir, "strfry.log"))
|
||||
.start()
|
||||
|
||||
val deadline = System.currentTimeMillis() + 5_000
|
||||
while (System.currentTimeMillis() < deadline) {
|
||||
runCatching {
|
||||
Socket("127.0.0.1", port).close()
|
||||
return "ws://127.0.0.1:$port/"
|
||||
}
|
||||
Thread.sleep(100)
|
||||
}
|
||||
throw IllegalStateException(
|
||||
"strfry did not start within 5s; log: " +
|
||||
File(strfryDir, "strfry.log").readText(),
|
||||
)
|
||||
}
|
||||
|
||||
private fun makeEvents(count: Int): List<Event> {
|
||||
val signer = NostrSignerSync(KeyPair())
|
||||
val now = 1_700_000_000L
|
||||
return List(count) { i ->
|
||||
signer.sign(TextNoteEvent.build("strfry-interop-$i", createdAt = now + i))
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Push every event in [events] to the relay at [wsUrl] over a
|
||||
* one-shot WebSocket, waiting for an `OK` response per event.
|
||||
*
|
||||
* Used to seed strfry from the same `Event` objects Geode's
|
||||
* `Relay.preload` accepts — that way both sides start from a
|
||||
* byte-identical corpus.
|
||||
*/
|
||||
private suspend fun publishToStrfry(
|
||||
wsUrl: String,
|
||||
events: List<Event>,
|
||||
) {
|
||||
val incoming = Channel<String>(UNLIMITED)
|
||||
val ws =
|
||||
httpClient.newWebSocket(
|
||||
Request.Builder().url(wsUrl.replace("ws://", "http://")).build(),
|
||||
object : WebSocketListener() {
|
||||
override fun onMessage(
|
||||
webSocket: WebSocket,
|
||||
text: String,
|
||||
) {
|
||||
incoming.trySend(text)
|
||||
}
|
||||
|
||||
override fun onFailure(
|
||||
webSocket: WebSocket,
|
||||
t: Throwable,
|
||||
response: Response?,
|
||||
) {
|
||||
incoming.close(t)
|
||||
}
|
||||
},
|
||||
)
|
||||
try {
|
||||
for (e in events) {
|
||||
check(ws.send("""["EVENT",${OptimizedJsonMapper.toJson(e)}]""")) {
|
||||
"publish to strfry failed"
|
||||
}
|
||||
}
|
||||
var oks = 0
|
||||
withTimeout(30_000) {
|
||||
while (oks < events.size) {
|
||||
if (incoming.receive().startsWith("[\"OK\"")) oks++
|
||||
}
|
||||
}
|
||||
} finally {
|
||||
ws.close(1000, "preload-done")
|
||||
}
|
||||
}
|
||||
|
||||
@Test
|
||||
fun geodeReconcilesAgainstStrfryRelay() =
|
||||
runBlocking {
|
||||
if (!enabled) {
|
||||
println("[skip] GeodeVsStrfryNegentropySyncTest — set STRFRY_BIN=/path/to/strfry to enable")
|
||||
return@runBlocking
|
||||
}
|
||||
val strfryWs = startStrfry()
|
||||
|
||||
// Same overlap shape as GeodeVsGeodeNegentropySyncTest so
|
||||
// results are directly comparable: A=[0..14], local=[5..19].
|
||||
val all = makeEvents(20)
|
||||
val strfryEvents = all.subList(0, 15)
|
||||
val localEvents = all.subList(5, 20)
|
||||
|
||||
publishToStrfry(strfryWs, strfryEvents)
|
||||
|
||||
val filter = Filter(kinds = listOf(1))
|
||||
|
||||
// The wire we care about: kmp-negentropy (client) talking
|
||||
// to strfry's `Negentropy ne(storage, 500'000)` (server).
|
||||
// Symmetric difference must match the Geode-vs-Geode case.
|
||||
val res = InteropSyncDriver(httpClient).negotiate(strfryWs, filter, localEvents)
|
||||
assertNull(res.error, "Geode↔strfry NEG must not error: ${res.error}")
|
||||
assertEquals(
|
||||
strfryEvents.subList(0, 5).map { it.id }.toSet(),
|
||||
res.needIds,
|
||||
"client should NEED [0..4] from strfry",
|
||||
)
|
||||
assertEquals(
|
||||
localEvents.subList(10, 15).map { it.id }.toSet(),
|
||||
res.haveIds,
|
||||
"client should announce HAVE for [15..19]",
|
||||
)
|
||||
assertTrue(res.rounds <= 16, "expected ≤16 NEG-MSG rounds, got ${res.rounds}")
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,191 @@
|
||||
/*
|
||||
* Copyright (c) 2025 Vitor Pamplona
|
||||
*
|
||||
* Permission is hereby granted, free of charge, to any person obtaining a copy of
|
||||
* this software and associated documentation files (the "Software"), to deal in
|
||||
* the Software without restriction, including without limitation the rights to use,
|
||||
* copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the
|
||||
* Software, and to permit persons to whom the Software is furnished to do so,
|
||||
* subject to the following conditions:
|
||||
*
|
||||
* The above copyright notice and this permission notice shall be included in all
|
||||
* copies or substantial portions of the Software.
|
||||
*
|
||||
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
||||
* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS
|
||||
* FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR
|
||||
* COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN
|
||||
* AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
|
||||
* WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
|
||||
*/
|
||||
package com.vitorpamplona.geode.interop
|
||||
|
||||
import com.vitorpamplona.quartz.nip01Core.core.Event
|
||||
import com.vitorpamplona.quartz.nip01Core.core.HexKey
|
||||
import com.vitorpamplona.quartz.nip01Core.core.OptimizedJsonMapper
|
||||
import com.vitorpamplona.quartz.nip01Core.relay.commands.toClient.NoticeMessage
|
||||
import com.vitorpamplona.quartz.nip01Core.relay.filters.Filter
|
||||
import com.vitorpamplona.quartz.nip77Negentropy.NegErrMessage
|
||||
import com.vitorpamplona.quartz.nip77Negentropy.NegMsgMessage
|
||||
import com.vitorpamplona.quartz.nip77Negentropy.NegentropySession
|
||||
import kotlinx.coroutines.channels.Channel
|
||||
import kotlinx.coroutines.channels.Channel.Factory.UNLIMITED
|
||||
import kotlinx.coroutines.withTimeout
|
||||
import okhttp3.OkHttpClient
|
||||
import okhttp3.Request
|
||||
import okhttp3.Response
|
||||
import okhttp3.WebSocket
|
||||
import okhttp3.WebSocketListener
|
||||
import kotlin.test.fail
|
||||
|
||||
/**
|
||||
* Equivalent of strfry's `test/syncTest.pl` driver for our interop
|
||||
* tests. Drives one round of NIP-77 *negotiation* (NEG-OPEN /
|
||||
* NEG-MSG / NEG-CLOSE) against a real `ws://` endpoint — a Geode
|
||||
* `LocalRelayServer`, a strfry process, or any other NIP-77 relay.
|
||||
*
|
||||
* The driver opens a raw WebSocket — no `NostrClient` overhead —
|
||||
* to keep the wire format under direct control, the same way
|
||||
* `strfry sync` does. That way we exercise the server's NEG-OPEN /
|
||||
* NEG-MSG / NEG-CLOSE handling with no client-side framing or
|
||||
* filter-management indirection.
|
||||
*
|
||||
* Note: this driver only computes the symmetric difference. The
|
||||
* actual *sync* (REQ for `needIds`, EVENT for `haveIds`) is the
|
||||
* caller's job; that's a NIP-01 follow-up, not part of NIP-77.
|
||||
*/
|
||||
class InteropSyncDriver(
|
||||
private val httpClient: OkHttpClient = OkHttpClient.Builder().build(),
|
||||
) {
|
||||
/**
|
||||
* Negotiates the symmetric difference between `localEvents` and
|
||||
* the relay at [wsUrl] under [filter]. Returns the id sets so
|
||||
* the caller can close the loop with REQ / EVENT.
|
||||
*
|
||||
* @param wsUrl the source relay's `ws://…` URL.
|
||||
* @param filter NEG-OPEN filter — usually the broadest filter the
|
||||
* sync should cover (e.g. `Filter(kinds = listOf(1))`).
|
||||
* @param localEvents events the caller already has; the relay
|
||||
* reconciles these against its own snapshot.
|
||||
* @param frameSizeLimit `0` lets the relay choose. We pass `0`
|
||||
* here so the relay's configured cap (500_000 by default) is
|
||||
* what governs framing — same shape as `strfry sync`.
|
||||
* @param timeoutMs hard timeout on a single NEG-MSG round trip.
|
||||
* @param maxRounds upper bound on round trips. Strfry typically
|
||||
* converges in ≤5 rounds for 100 k corpora; 64 is a generous
|
||||
* safety net that catches pathological splits without hanging
|
||||
* tests forever.
|
||||
*/
|
||||
suspend fun negotiate(
|
||||
wsUrl: String,
|
||||
filter: Filter,
|
||||
localEvents: List<Event>,
|
||||
subId: String = "interop-sync",
|
||||
frameSizeLimit: Long = 0,
|
||||
timeoutMs: Long = 30_000L,
|
||||
maxRounds: Int = 64,
|
||||
): Result {
|
||||
val incoming = Channel<String>(UNLIMITED)
|
||||
val ws =
|
||||
httpClient.newWebSocket(
|
||||
Request.Builder().url(wsUrl.replace("ws://", "http://")).build(),
|
||||
object : WebSocketListener() {
|
||||
override fun onMessage(
|
||||
webSocket: WebSocket,
|
||||
text: String,
|
||||
) {
|
||||
incoming.trySend(text)
|
||||
}
|
||||
|
||||
override fun onClosing(
|
||||
webSocket: WebSocket,
|
||||
code: Int,
|
||||
reason: String,
|
||||
) {
|
||||
incoming.close()
|
||||
}
|
||||
|
||||
override fun onFailure(
|
||||
webSocket: WebSocket,
|
||||
t: Throwable,
|
||||
response: Response?,
|
||||
) {
|
||||
incoming.close(t)
|
||||
}
|
||||
},
|
||||
)
|
||||
|
||||
return try {
|
||||
val session = NegentropySession(subId, filter, localEvents, frameSizeLimit)
|
||||
|
||||
// Step 1: NEG-OPEN.
|
||||
check(ws.send(OptimizedJsonMapper.toJson(session.open()))) { "send NEG-OPEN failed" }
|
||||
|
||||
// Step 2: drive NEG-MSG round trips until the client-side
|
||||
// session reports completion.
|
||||
val haveIds = mutableSetOf<HexKey>()
|
||||
val needIds = mutableSetOf<HexKey>()
|
||||
var rounds = 0
|
||||
while (rounds < maxRounds) {
|
||||
val raw = withTimeout(timeoutMs) { incoming.receive() }
|
||||
val msg = OptimizedJsonMapper.fromJsonToMessage(raw)
|
||||
rounds++
|
||||
when (msg) {
|
||||
is NegErrMessage -> {
|
||||
return Result(
|
||||
haveIds = haveIds,
|
||||
needIds = needIds,
|
||||
rounds = rounds,
|
||||
error = "${msg.subId}: ${msg.reason}",
|
||||
)
|
||||
}
|
||||
|
||||
is NoticeMessage -> {
|
||||
return Result(
|
||||
haveIds = haveIds,
|
||||
needIds = needIds,
|
||||
rounds = rounds,
|
||||
error = "NOTICE: ${msg.message}",
|
||||
)
|
||||
}
|
||||
|
||||
is NegMsgMessage -> {
|
||||
val r = session.processMessage(msg.message)
|
||||
haveIds += r.haveIds
|
||||
needIds += r.needIds
|
||||
if (r.isComplete()) {
|
||||
return Result(haveIds, needIds, rounds, error = null)
|
||||
}
|
||||
check(ws.send(OptimizedJsonMapper.toJson(r.nextCmd!!))) {
|
||||
"send NEG-MSG failed"
|
||||
}
|
||||
}
|
||||
|
||||
else -> {
|
||||
fail("unexpected message during NEG sync: ${msg::class.simpleName}")
|
||||
}
|
||||
}
|
||||
}
|
||||
Result(haveIds, needIds, rounds, error = "did not converge in $maxRounds rounds")
|
||||
} finally {
|
||||
ws.close(1000, "interop-test-done")
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Result of a reconciliation. `error` is non-null on
|
||||
* NEG-ERR/NOTICE/timeout; otherwise the id-set fields are
|
||||
* authoritative.
|
||||
*
|
||||
* @param haveIds events the client (us) had that the relay did not.
|
||||
* @param needIds events the relay had that the client (us) lacked.
|
||||
* @param rounds NEG-MSG round trips, including the one carrying
|
||||
* the terminator.
|
||||
*/
|
||||
data class Result(
|
||||
val haveIds: Set<HexKey>,
|
||||
val needIds: Set<HexKey>,
|
||||
val rounds: Int,
|
||||
val error: String?,
|
||||
)
|
||||
}
|
||||
@@ -365,6 +365,52 @@ class LoadBenchmark {
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Same workload as [publishThroughputSingleClient] (sequential
|
||||
* publish-and-confirm on one connection) — kept as a regression
|
||||
* floor for the group-commit code path. Synchronous publishes
|
||||
* never coalesce in the writer (batch size is always 1), so the
|
||||
* EPS here measures per-event SQLite tx cost. The pipelined win
|
||||
* shows up in [publishPipelinedSingleClient].
|
||||
*/
|
||||
@Test
|
||||
fun publishGroupCommitSingleClient() =
|
||||
benchmark("publish group-commit single client") {
|
||||
runBenchmarkServer { server, http ->
|
||||
val scope = CoroutineScope(Dispatchers.Default + SupervisorJob())
|
||||
val client = NostrClient(BasicOkHttpWebSocket.Builder { _ -> http }, scope)
|
||||
try {
|
||||
val signer = NostrSignerSync(KeyPair())
|
||||
val relayUrl = server.url.normalizeRelayUrl()
|
||||
|
||||
val n = 10_000
|
||||
var ok = 0
|
||||
val elapsed =
|
||||
measureTime {
|
||||
runBlocking {
|
||||
repeat(n) { i ->
|
||||
val event = signer.sign(TextNoteEvent.build("group-commit $i"))
|
||||
if (client.publishAndConfirm(event, setOf(relayUrl))) ok++
|
||||
}
|
||||
}
|
||||
}
|
||||
val eps = (n * 1000.0) / elapsed.inWholeMilliseconds
|
||||
println(
|
||||
"events=$n ok=$ok elapsedMs=${elapsed.inWholeMilliseconds} eps=${"%.0f".format(eps)}",
|
||||
)
|
||||
check(ok == n) { "expected all $n events accepted, got $ok" }
|
||||
// Floor: the pre-batching baseline was ~760 EPS
|
||||
// single-client (see plan). Anything below 500
|
||||
// means the group-commit / ingest-queue rewrite
|
||||
// regressed the synchronous path.
|
||||
check(eps > 500) { "synchronous EPS $eps fell below the 500 floor" }
|
||||
} finally {
|
||||
client.disconnect()
|
||||
scope.cancel()
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* One publisher, N subscribers. Publishes one EVENT and measures
|
||||
* fan-out latency: time from publish to last subscriber receiving.
|
||||
@@ -614,6 +660,101 @@ class LoadBenchmark {
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* One publisher fires N EVENTs back-to-back without awaiting
|
||||
* intermediate OKs, then collects all OKs by event id. This is
|
||||
* the workload that exercises Tier 2 (per-connection ingest
|
||||
* pipeline) + Tier 1 (group commit) together — multiple events
|
||||
* are in flight on the same connection, so the writer can batch.
|
||||
*
|
||||
* Verifies the relaxed OK contract: every event id receives
|
||||
* exactly one OK frame, in any order.
|
||||
*/
|
||||
@Test
|
||||
fun publishPipelinedSingleClient() =
|
||||
benchmark("publish pipelined single client") {
|
||||
runBenchmarkServer { server, http ->
|
||||
val n = 10_000
|
||||
val signer = NostrSignerSync(KeyPair())
|
||||
val events =
|
||||
runBlocking {
|
||||
(0 until n).map { i ->
|
||||
signer.sign(TextNoteEvent.build("pipe $i"))
|
||||
}
|
||||
}
|
||||
val ids = events.mapTo(HashSet()) { it.id }
|
||||
|
||||
val httpUrl =
|
||||
okhttp3.Request
|
||||
.Builder()
|
||||
.url(server.url.replace("ws://", "http://"))
|
||||
.build()
|
||||
val okSeen = AtomicLong()
|
||||
val okFailures = AtomicLong()
|
||||
val unknownIds = AtomicLong()
|
||||
val seenIds =
|
||||
java.util.concurrent.ConcurrentHashMap
|
||||
.newKeySet<String>()
|
||||
val done = java.util.concurrent.CountDownLatch(1)
|
||||
|
||||
val ws =
|
||||
http.newWebSocket(
|
||||
httpUrl,
|
||||
object : okhttp3.WebSocketListener() {
|
||||
override fun onMessage(
|
||||
webSocket: okhttp3.WebSocket,
|
||||
text: String,
|
||||
) {
|
||||
if (!text.startsWith("[\"OK\"")) return
|
||||
// ["OK","<id>",true|false,"<reason>"] —
|
||||
// a tiny string scan is enough for a
|
||||
// bench. Index 6 is past `["OK","`.
|
||||
val idStart = 7
|
||||
val idEnd = text.indexOf('"', idStart)
|
||||
if (idEnd <= idStart) return
|
||||
val id = text.substring(idStart, idEnd)
|
||||
if (!ids.contains(id)) {
|
||||
unknownIds.incrementAndGet()
|
||||
return
|
||||
}
|
||||
if (!seenIds.add(id)) return
|
||||
if (text.contains(",true,")) {
|
||||
okSeen.incrementAndGet()
|
||||
} else {
|
||||
okFailures.incrementAndGet()
|
||||
}
|
||||
if (okSeen.get() + okFailures.get() == n.toLong()) done.countDown()
|
||||
}
|
||||
},
|
||||
)
|
||||
|
||||
val elapsed =
|
||||
measureTime {
|
||||
// Burst-send: queue every EVENT to OkHttp's
|
||||
// outbound buffer without any await, then
|
||||
// wait for the corresponding OK frames.
|
||||
for (event in events) {
|
||||
ws.send("""["EVENT",${event.toJson()}]""")
|
||||
}
|
||||
check(done.await(60, java.util.concurrent.TimeUnit.SECONDS)) {
|
||||
"timed out waiting for OKs: ok=${okSeen.get()} rej=${okFailures.get()} unknown=${unknownIds.get()}"
|
||||
}
|
||||
}
|
||||
val eps = (n * 1000.0) / elapsed.inWholeMilliseconds
|
||||
println(
|
||||
"events=$n ok=${okSeen.get()} rejected=${okFailures.get()} " +
|
||||
"unknownIds=${unknownIds.get()} elapsedMs=${elapsed.inWholeMilliseconds} eps=${"%.0f".format(eps)}",
|
||||
)
|
||||
check(okSeen.get() == n.toLong()) {
|
||||
"expected $n accepted OKs, got ${okSeen.get()} (rejected ${okFailures.get()})"
|
||||
}
|
||||
check(seenIds.size == n) {
|
||||
"expected $n unique OK ids, got ${seenIds.size} — duplicate or missing OKs"
|
||||
}
|
||||
ws.cancel()
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Many concurrent publishers, each on their own WebSocket. Tells
|
||||
* us whether the SQLite single-writer bottleneck is the floor or
|
||||
|
||||
Reference in New Issue
Block a user