nick/amethyst
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 1 Wiki Activity

docs(geode): refine negentropy plan for strfry interop

Browse Source 
Align the NIP-77 large-corpus plan with strfry's actual defaults:
500_000-byte frame cap (not 64 KB), no default since-window, 200
shared sub cap, max_sync_events overflow protection, exact NEG-ERR
wording, and 32-byte id storage. Add a strfry round-trip benchmark
and call out the BTreeLMDB pre-built tree as a follow-up.
This commit is contained in:
Claude
2026-05-07 22:16:32 +00:00
parent 6425dd53d2
commit ab71e45e77
1 changed files with 204 additions and 60 deletions
Download Patch File Download Diff File Expand all files Collapse all files
geode/plans/2026-05-07-negentropy-large-corpus.md
+204 -60
View File
@@ -1,20 +1,20 @@
# NIP-77 negentropy at scale: snapshot memory + chunked replay # NIP-77 negentropy at scale: strfry-interop snapshot path
## Problem ## Problem
`RelaySession` delegates NEG-OPEN to `NegSessionRegistry.open` `RelaySession` delegates NEG-OPEN to `NegSessionRegistry.open`
(`quartz/nip01Core/relay/server/NegSessionRegistry.kt`), which calls (`quartz/src/commonMain/kotlin/com/vitorpamplona/quartz/nip01Core/relay/server/NegSessionRegistry.kt:58-77`),
`store.snapshotQuery(filters)` and feeds the **entire** result list which calls `store.snapshotQuery(filters)` and feeds the **entire**
into `NegentropyServerSession`. For a relay holding 5 M events that result list of full `Event` objects into `NegentropyServerSession`
match a broad NEG-OPEN filter (`{kinds: [1, 7]}`), this is 5 M (`quartz/src/commonMain/kotlin/com/vitorpamplona/quartz/nip77Negentropy/NegentropyServerSession.kt:40-54`).
`Event` objects materialised in memory before the first NEG-MSG goes For a relay holding 5 M events that match a broad NEG-OPEN filter
out. (`{kinds: [1, 7]}`), this materialises 5 M `Event` objects with
content/tags/sig — call it ~1 KB/event, ~5 GB transient pressure per
concurrent NEG-OPEN — before the first NEG-MSG goes out.
The negentropy library itself is fine — it pivots into a sealed The negentropy library itself is fine. Internally it pivots into a
`StorageVector` (id + createdAt only, ~40 bytes/entry). But the sealed `StorageVector` of `(uint64 timestamp, byte[32] id)` items —
`store.query<Event>(f)` step that produces the input materialises full 40 bytes/entry. The waste is purely in the snapshot step.
`Event` objects with content, tags, sig — call it ~1 KB/event. 5 M ×
1 KB = 5 GB transient pressure per concurrent NEG-OPEN.
Two operator-visible symptoms: Two operator-visible symptoms:
@@ -24,77 +24,221 @@ Two operator-visible symptoms:
large stores the client waits seconds for what should be a large stores the client waits seconds for what should be a
millisecond round-trip. millisecond round-trip.
## Reference: strfry
We want byte-for-byte interop and comparable throughput against
[hoytech/strfry](https://github.com/hoytech/strfry). The relevant
defaults from `strfry/src/apps/relay/RelayNegentropy.cpp` and
`golpe.yaml`:
| Knob | strfry default | Notes |
|---------------------------------------|-----------------------|-------|
| `frameSizeLimit` (NEG-MSG payload) | **500_000 bytes** | Hard-coded `Negentropy ne(storage, 500'000)` |
| `relay__negentropy__maxSyncEvents` | **1_000_000** | Hard cap on items in the snapshot |
| `relay__maxSubsPerConnection` | **200** | Shared between REQ and NEG sessions |
| `idSize` on the wire | **32 bytes** | NIP-77 v1 (`PROTOCOL_VERSION = 0x61`) |
| Default `since` window | **none** | Filter is honored as-is |
| Filter parser | same as REQ | Honors `limit`, `kinds`, `authors`, `#tags` |
| Snapshot data | `(created_at, id)` only | LMDB scan inserts into `negentropy::storage::Vector` |
Two-phase materialisation in strfry: `QueryScheduler` scans LMDB
asynchronously, batching matched level-ids into a `vector<uint64_t>`;
on completion the worker pulls each event header, calls
`storageVector.insert(packed.created_at(), packed.id())`, then
`seal()`s. The session response is sent only after seal.
Our equivalent must (a) match the snapshot footprint of ~40 bytes per
event, and (b) match the wire-level frame-cap so a single
reconciliation round-trip exchanges the same payload size.
## Sketch ## Sketch
### A — id-and-time-only snapshot path ### A — id-and-time-only snapshot path (memory parity)
Negentropy only needs `(createdAt, id)` pairs. Add a streaming Negentropy only needs `(createdAt, id)` pairs. Add a streaming
`IEventStore.queryIdAndTime(filter)` that returns `IEventStore.snapshotIdsForNegentropy(filter)` that returns
`Sequence<Pair<Long, ByteArray>>` (or a `Flow` of small chunks) — `Sequence<IdAndTime>` (`data class IdAndTime(val createdAt: Long, val id: ByteArray)`)
no content/tags/sig, no Event allocation. SQLite path is a SELECT — no content/tags/sig, no `Event` allocation. The SQLite path is a
on `event_headers` (the `created_at`, `id` columns are already plain `SELECT id, created_at FROM event_headers WHERE …` against the
indexed for query plans). existing `query_by_created_at_id` index
(`quartz/src/commonMain/kotlin/com/vitorpamplona/quartz/nip01Core/store/sqlite/EventIndexesModule.kt:79`).
```kotlin ```kotlin
suspend fun snapshotIdsForNegentropy(filter: Filter): IdTimeStream // IEventStore
suspend fun snapshotIdsForNegentropy(filters: List<Filter>): List<IdAndTime>
// LiveEventStore — already deduplicates union of multi-filter results
suspend fun snapshotIdsForNegentropy(filters: List<Filter>): List<IdAndTime>
``` ```
`NegentropyServerSession` is rewritten to take that stream and feed `NegentropyServerSession` is rewritten to take that list (or a
it directly into the `StorageVector`. Memory drops from O(N × 1 KB) two-phase `pendingIds → seal` builder) and feed it directly into
to O(N × 40 B) — a 25× reduction; for 5 M events, ~200 MB instead `StorageVector`. Memory drops from O(N × ~1 KB) to O(N × 40 B) — for
of 5 GB. 1 M events, ~40 MB instead of ~1 GB; matches strfry's per-session
footprint.
### B — bounded-window subscriptions **id encoding:** strfry stores 32-byte raw ids (NIP-77 v1
`ID_SIZE = 32`); the 16-byte truncation is `FINGERPRINT_SIZE`, only used
internally for SHA-256 accumulator output. The current Kotlin code
already passes the hex `event.id` string to `storage.insert(..)`
(`NegentropyServerSession.kt:50`); the kmp-negentropy library decodes
that to 32 raw bytes. Confirm this is preserved when we switch to a
`ByteArray` id input — do not pre-truncate.
Most NEG-OPENs from real Nostr clients want the last 30 days, not ### B — bounded snapshot size, NOT a default `since` window
"everything." If the client doesn't supply `since`, the server can
default to a configurable horizon (e.g. 90 days) and surface this in
the NIP-11 `limitation.negentropy_max_lookback_seconds` field.
Operators can lift the cap; clients reading the doc know the bound.
This is a NIP-spec-adjacent question more than a code change — needs The previous draft of this plan suggested defaulting `since` to a 90-day
a comment on whether the spec allows it. nostr-rs-relay does this horizon. **Drop that.** strfry doesn't do it — the filter is honored
already. as-is — and silently bounding `since` would break interop with strfry's
sync clients (e.g. `strfry sync`, nostr-sdk's negentropy reconciler):
they ask for "everything" and rely on getting it.
### C — frame-size cap on NEG-MSG Instead match strfry's protection: a hard cap on the number of items
that go into a single snapshot.
```toml
[negentropy]
max_sync_events = 1_000_000 # matches strfry's relay__negentropy__maxSyncEvents
```
`NegSessionRegistry.open` checks `count >= max_sync_events` after the
SQLite count or during scan, and on overflow sends:
```
["NEG-ERR", "<subId>", "blocked: too many query results"]
```
Exact wording matches strfry so client error-handling that string-matches
behaves identically. (Spec doesn't normatively define error text; this is
de-facto interop.)
### C — frame-size cap on NEG-MSG: 500_000 bytes (strfry parity)
`NegentropyServerSession` is constructed with `frameSizeLimit = 0` `NegentropyServerSession` is constructed with `frameSizeLimit = 0`
(no limit). At very large reconciliations the message can grow large. today. **Set `frameSizeLimit = 500_000`** (matching strfry) so a single
Set a default `frameSizeLimit = 64 * 1024` (matching the typical WS NEG-MSG round-trip carries the same payload as strfry's. 64 KB — what
frame budget) so NEG-MSGs don't blow past `[limits].max_ws_frame_bytes`. the previous draft suggested — would force 8× more round-trips for
large reconciliations and noticeably slow sync against strfry-style
clients that expect ~1 MB hex-encoded NEG-MSGs.
The library already supports this — pure config change in The kmp-negentropy library enforces `frameSizeLimit >= 4096` (or `0`
`NegSessionRegistry.open`. for unlimited). 500_000 is well above the floor. Pure config change in
`NegSessionRegistry.open`; expose via `[negentropy].frame_size_limit`
for operators who tune it down to fit smaller WS frame budgets.
### D — concurrent NEG-OPEN cap Note: `LimitsSection.max_ws_frame_bytes`
(`geode/src/main/kotlin/com/vitorpamplona/geode/config/RelayConfig.kt:148`)
applies to the WebSocket layer. After hex-encoding a 500_000-byte
negentropy payload doubles to ~1_000_000 bytes on the wire; ensure
`max_ws_frame_bytes` is at least 2 MB (or unlimited) in default
config so the response isn't truncated by the WS layer.
### D — concurrent NEG-OPEN cap, shared with REQ
A NEG-OPEN holds session state until NEG-CLOSE (or connection close). A NEG-OPEN holds session state until NEG-CLOSE (or connection close).
Today nothing caps the number of concurrent open negentropy sessions Today `NegSessionRegistry.sessions` is unbounded. strfry caps at 200
per connection. A misbehaving (or hostile) client could open thousands sessions per connection **shared with REQ** — both count against
and pin RAM. Add `MAX_NEG_SESSIONS_PER_CONNECTION = 16`, send NEG-ERR `relay__maxSubsPerConnection`.
on overflow.
Implement as:
- Reuse the existing per-connection REQ subscription cap (or introduce
one if absent) and let NEG sessions consume the same budget.
- Default cap: **200** to match strfry. Configurable via
`[limits].max_subs_per_connection`.
- On overflow strfry sends a NOTICE, not NEG-ERR:
`["NOTICE", "too many concurrent NEG requests"]`. We should match
this — `NEG-ERR` is reserved for per-session protocol errors in
strfry's model.
### E — pre-built fingerprint tree (follow-up, not in scope here)
strfry's real production-scale advantage is the **`negentropy::storage::BTreeLMDB`** backend: a persistent, incrementally-maintained B-tree
of `(timestamp, id)` keys with per-node fingerprint accumulators.
When NEG-OPEN's filter string matches a pre-registered
`NegentropyFilter` tree, strfry skips the materialise-and-seal step
entirely and reconciles directly off the LMDB B-tree in O(log n)
fingerprint computations. See `env.foreach_NegentropyFilter` and
`addStatelessView` in `RelayNegentropy.cpp`.
Equivalent for geode: a `quartz/.../nip77Negentropy/PrebuiltStorage`
backed by a SQLite-side incremental fingerprint index, registered per
canonical filter. This is a substantial piece of work and **out of
scope for this plan** — call it out for a follow-up
(`geode/plans/2026-05-08-negentropy-prebuilt-tree.md`). The A+B+C+D
combination is enough to match strfry's `MemoryView` path, which is
what 99% of ad-hoc NEG-OPENs hit.
## Concrete error-string interop table
Match strfry verbatim — clients in the wild string-match on these:
| Condition | strfry frame |
|---------------------------------------|-----------------------------------------------------------|
| Snapshot exceeds `max_sync_events` | `["NEG-ERR", "<subId>", "blocked: too many query results"]` |
| NEG-MSG for unknown subId | `["NEG-ERR", "<subId>", "closed: unknown subscription handle"]` |
| Library `reconcile()` parse failure | `["NEG-ERR", "<subId>", "PROTOCOL-ERROR"]` |
| Per-connection sub cap exceeded | `["NOTICE", "too many concurrent NEG requests"]` |
| NEG-MSG before NEG-OPEN seal complete | `["NOTICE", "negentropy error: got NEG-MSG before NEG-OPEN complete"]` |
The current Kotlin code in `NegSessionRegistry.kt:82` sends
`"error: no negentropy session for <subId>"` for the unknown-subId
case. Update to strfry's wording.
## Wire-level conformance checks
Before merging, verify against strfry as ground truth:
1. **Round-trip with `strfry sync`**: stand up a small geode instance,
point `strfry sync ws://geode-host` at it, confirm sync completes
and converges in ≤ comparable round-trips for an N=100k corpus.
2. **idSize**: assert kmp-negentropy round-trips full 32-byte ids;
the 16-byte fingerprint stays internal to the library.
3. **Protocol version byte**: NIP-77 v1 = `0x61`. Confirm
`NegentropyServerSession.processMessage` neither emits nor accepts
a different version byte. Negotiation happens inside the library;
surface the version on `NIP-11.limitation.negentropy = 1` so clients
know the v1 path is supported.
## How to verify ## How to verify
Add to `geode.perf.LoadBenchmark`: Add to `geode/src/test/kotlin/com/vitorpamplona/geode/perf/LoadBenchmark.kt`:
- `negentropyOpenLatencyLargeCorpus` — preload 1 M events (use - `negentropyOpenLatencyLargeCorpus` — preload 1 M events, measure
fixtures), measure NEG-OPEN → first NEG-MSG latency. Target <100 ms. NEG-OPEN → first NEG-MSG latency. Target **<200 ms** (strfry's
C++ `MemoryView` path on equivalent hardware does ~80–150 ms;
we expect a 1.5–2× JVM tax).
- `negentropyMemoryPressure` — open 10 concurrent NEG-OPENs on the - `negentropyMemoryPressure` — open 10 concurrent NEG-OPENs on the
same large corpus; measure RSS delta, target <500 MB. same large corpus; measure RSS delta. Target **<500 MB**
(10 × ~40 MB session footprint + scan overhead).
- `negentropyStrfryInterop` — programmatic round-trip against a
containerised `hoytech/strfry`: same fixture corpus loaded in both,
cross-sync, assert byte-identical id-set convergence in equal
round-trips ±1.
Add to `quartz/.../nip77Negentropy/`:
- `NegentropyServerSessionTest.processMessage_atFrameLimit_splitsAcrossRounds`
— large symmetric difference, assert each NEG-MSG payload is
≤ 500_000 bytes and the session completes in N rounds (not 1).
## Risks ## Risks
- **`Sequence`/`Flow` over SQLite cursor**: holding a cursor open - **Cursor lifetime**: holding a SQLite cursor open across the full
across the full sync is fragile if the client stalls. Materialise sync is fragile if the client stalls. Materialise to a smaller
to a smaller in-memory list (just (id, createdAt)) once, reuse for in-memory `List<IdAndTime>` (40 bytes/entry) once at NEG-OPEN time,
the lifetime of the session. Memory bound is the same. reuse for the lifetime of the session. Bounded by `max_sync_events`.
- **Defaulting `since` is a behaviour change**: existing clients that - **Frame-size 500_000 vs WS frame budget**: hex-encoded payload is
expect "everything" silently get a bounded window. Either (a) make ~1 MB. If `LimitsSection.max_ws_frame_bytes` is set lower than
it opt-in via `RelayConfig.NegentropySection.default_lookback_seconds ~1.5 MB the response gets truncated/rejected by the WS layer. Lift
= null`, (b) advertise the cap in NIP-11 so well-behaved clients the WS default OR cap `frame_size_limit` to
read it. `max_ws_frame_bytes / 2` at startup; fail-fast log a warning if the
- **Frame-size cap can break older clients**: the NIP-77 reference operator's config makes negentropy unusable.
implementation (kmp-negentropy) handles this gracefully — multi-frame - **No default `since` (intentional, but worth flagging)**: a hostile
reconciliation is in spec — but field-test against a known-working client doing `NEG-OPEN {kinds:[1]}` against a large corpus will hit
client (e.g. nstart, primal-cache) before flipping the default. `max_sync_events` and get NEG-ERR. The cap is the protection; do
not also silently bound `since`.
- **kmp-negentropy library version**: pinned to `v1.0.2`
(`gradle/libs.versions.toml:9`). Confirm v1.0.2 enforces
`frameSizeLimit >= 4096`, supports protocol byte `0x61`, and
internal `ID_SIZE = 32`. If any of those don't match strfry, this
plan needs an upstream fix on kmp-negentropy first.