Commit Graph

30 Commits

Author SHA1 Message Date
Vitor Pamplona da32b33684 fixes warnings 2026-05-12 19:32:11 -04:00
Vitor Pamplona 14342e12c2 refactor(geode): simplify RelayEngine + trim StaticConfig comments
RelayEngine (224 → 185 lines):
  - Extract BanStore <-> RuntimeConfigData mapping helpers
    (seedInto, snapshotOf) into geode.config — bidirectional
    conversion now has names instead of being inline boilerplate
    repeated between the init block and snapshot().
  - Collapse the standalone init block into a .apply on the
    banStore declaration. Method reference ::snapshot for the
    onMutation hook.
  - Drop the effectiveAtBoot field's 12-line KDoc (rename to `boot`
    locally, terse). Trim verbose property KDocs to the load-bearing
    facts (why-not-what).

StaticConfig (255 → 182 lines):
  - Drop resolveInfo's unused `advertisedUrl` parameter and the
    workaround comment that justified keeping it for "future
    fields". YAGNI — add back when actually needed. Two callers
    updated.
  - Trim trivial KDocs (fromToml/fromFile, field names that
    document themselves like require_auth, reject_future_seconds,
    supported_nips, database.file).
  - Collapse NetworkSection's three thread-pool KDocs (~16 lines)
    into a single section-level KDoc.
  - Compact parallel_verify, AdminSection, AdminSection.state_file,
    and advertisedUrl companion KDocs while keeping the
    load-bearing facts (invariants, strfry interop strings,
    surprising defaults).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-12 19:01:36 -04:00
Vitor Pamplona 72e7f37b61 refactor(geode): move Nip86Server (+ adminPubkeys) into RelayEngine
KtorRelay was carrying state that wasn't HTTP-specific: the
Nip86Server, its InfoHolder adapter, and the adminPubkeys allow-list.
Push them into RelayEngine — it already owns the info doc, ban store,
and event store the Nip86Server consults; the allow-list is a
relay-level "who is admin?" decision, not a transport-level one.

RelayEngine gains adminPubkeys: Set<HexKey> = emptySet() and exposes
nip86Server as a public property. Future non-HTTP admin transports
(in-process tools, hypothetical NIP-86-over-WS) read it directly
without re-deriving the wiring.

KtorRelay shrinks to mostly Ktor structures — routing block, engine
config, lifecycle. It just builds the Nip86HttpHandler around
relay.nip86Server.

Main.kt and Nip86EndToEndTest pass adminPubkeys to RelayEngine
instead of KtorRelay.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-12 18:44:50 -04:00
Vitor Pamplona 445fbf8cda refactor(quartz, geode): derive NIP-86 publicUrl from relay.url; drop isEnabled branch
Two simplifications:

1. Derive admin URL from RelayEngine.url. NIP-86 spec mandates that
   the admin endpoint is "the same URI as ws(s)://, called via
   http(s)://" — so KtorRelay derives the NIP-98 binding URL by
   calling relay.url.toHttp() instead of accepting publicUrl as a
   separate config. Single source of truth (info.relay_url),
   accidental misconfiguration impossible, no Host-header fallback.
   StaticConfig.AdminSection.public_url is gone.

2. Uniform code path for admin-enabled vs disabled. Empty allow-list
   isn't a special case anywhere: Nip86Server.isAuthorized returns
   false for everyone, dispatch rejects, Nip86HttpHandler returns
   NotAdmin → 403. KtorRelay always assembles the Nip86HttpRoute
   and registers the POST endpoint; Nip86Server.isEnabled() and the
   "is admin on?" branches in the handler and route are deleted.

Nip86EndToEndTest's admin-disabled case is now a behavioral test:
sign a valid token, expect 403 NotAdmin (was 405 with the no-route
variant, was 403 with the fake-disabled-route variant).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-12 18:19:24 -04:00
Vitor Pamplona a53b3339b3 refactor(quartz): canonical NIP-86 HTTP flow in Nip86HttpHandler
Move the NIP-86-over-HTTP orchestration (NIP-98 verify → admin
allow-list gate → parse → dispatch → serialize) into quartz so any
relay implementation that wants the standard transport can plug its
HTTP framework in without re-deriving the sequence — and without
accidentally skipping NIP-98 verification or the admin check.

  • Nip86Server (quartz) gains allowList + isEnabled() + isAuthorized();
    dispatch becomes dispatch(pubkey, req) and enforces the allow-list
    itself, so in-process callers can't bypass it either.
  • Nip86HttpHandler (new, quartz) takes only primitives (authHeader,
    url, body) and returns a sealed Response — Disabled, PayloadTooLarge,
    MissingAuth, BadAuth, NotAdmin, BadRequest, Ok(pubkey, req, resp,
    json). Constants for the 1 MiB body cap, application/nostr+json+rpc
    content type, and WWW-Authenticate scheme live here.
  • Nip86HttpRoute (geode) shrinks to a Ktor adapter: bounded read,
    extract URL/auth, call handler, map Response → Ktor status codes.
    Audit logging stays in geode — not a quartz concern.

External relay authors: import quartz, build a Nip86HttpHandler with
their Nip86Server + Nip98AuthVerifier, and write a ~30-line adapter
for their framework. The full security-critical flow is in the box.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-12 17:38:40 -04:00
Vitor Pamplona e4e9ae7043 refactor(quartz): NIP-86 bans purge matching events via onBan(filter) hook
Nip86Server used to take an optional IEventStore and call store.delete()
only on banevent — banpubkey and disallowkind would update the BanStore
but leave existing matching events served by REQ. That's inconsistent
(an operator who bans a spam pubkey expects the spam to be gone) and
a real safety issue (banning a CSAM event wouldn't actually remove
existing copies if the store was null).

Replace the IEventStore param with onBan: suspend (Filter) -> Unit. The
dispatcher fires it after each ban method with the Filter that selects
the now-banned events:
  banpubkey   → Filter(authors = [pk])
  banevent    → Filter(ids = [id])
  disallowkind→ Filter(kinds = [k])

KtorRelay wires onBan = { f -> relay.store.delete(f) }, so the existing
SQLite delete-by-filter path handles every shape without the caller
having to know which field to populate. The default no-op keeps the
unit test for the dispatcher dependency-free.

withInt is now suspend so disallowKind's handler can call the suspending
onBan inline.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-12 16:55:38 -04:00
Vitor Pamplona a010b37d6c refactor(geode): inject RuntimeConfig into RelayEngine, mirroring EventStore
RelayEngine used to take three params that were really RuntimeConfig
construction details — info: RelayInfo, stateFile: File?,
seedAuthorization: AuthorizationSeed — and assembled the RuntimeConfig
internally. Replace that with a single runtimeConfig: RuntimeConfig
param (with an in-memory default), matching how store: IEventStore is
already passed in. Main.kt builds the RuntimeConfig from StaticConfig
at the assembly layer where the TOML→runtime mapping is most explicit.

AuthorizationSeed is dropped — it only existed to shuttle data into
the now-removed seedAuthorization param. Callers build RuntimeConfigData
directly.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-12 16:40:14 -04:00
Vitor Pamplona f4bcf38caf refactor(geode): drop unused KtorRelay knobs (maxFrameBytes, maxAdminBodyBytes)
Ktor's WebSocket layer is unbounded by default, so [limits].max_ws_frame_bytes
was a strfry-parity tuning knob nobody actually used — drop the whole
LimitsSection from StaticConfig and the matching constructor param from
KtorRelay. The negentropy-large-corpus plan note is updated accordingly.

maxAdminBodyBytes is a real defense-in-depth cap (NIP-98 forces us to
read the body for the payload sha256 before we can authenticate, so an
unbounded read is a pre-auth DoS vector). Keep the cap, but stop
exposing it as an operator knob — it was never plumbed through to
StaticConfig and 1 MiB is ~1000× any plausible NIP-86 RPC payload.
Hardcoded at the Nip86HttpRoute construction site with the rationale
inline. Nip86HttpRoute.maxBodyBytes stays so tests can drive the 413
boundary.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-12 16:14:15 -04:00
Vitor Pamplona 47fdd8638e refactor(geode): extract NIP-11 GET into Nip11HttpRoute
KtorRelay was resolving NIP-11 inline in the routing block while NIP-86
already had a dedicated Nip86HttpRoute with a handle(call) entry point.
Lift NIP-11 to the same shape so both endpoints look symmetric and the
routing block is just dispatch. The `liveJson` callback (rather than a
snapshot string) keeps NIP-86 changerelay* admin mutations visible on
the next GET without re-wiring.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-12 15:58:05 -04:00
Vitor Pamplona b0ad540453 refactor(geode): split config into StaticConfig + RuntimeConfig with seeding
Rename RelayConfig → StaticConfig and the persistence/RelayStateStore →
config/RuntimeConfig so the two configuration tiers — boot-time TOML
versus operator-mutable runtime state — are obvious from the class
names. The `persistence/` package is gone; everything related to config
lives in `config/`.

For overlapping fields (NIP-11 info doc, pubkey / kind allow-deny
lists), StaticConfig now seeds RuntimeConfig on first boot via the new
RuntimeConfig(seed = …) constructor and an AuthorizationSeed type. The
runtime file wins from then on: later edits to [authorization] in the
TOML are ignored once an admin RPC has written the snapshot. As a
consequence, KindAllowDenyPolicy and PubkeyAllowDenyPolicy are no
longer stacked in geode's policy chain — BanListPolicy already reads
the same lists from the BanStore, and double-stacking would silently
diverge after the first admin mutation.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-12 15:51:19 -04:00
Vitor Pamplona 9f7deee8f3 refactor(geode): rename relay classes for clarity, move test helpers off public API
Renames in the geode module to make naming honest:
- Relay -> RelayEngine (the transport-agnostic core)
- LocalRelayServer -> KtorRelay (Ktor/CIO transport; defaults to 127.0.0.1
  but supports public deployment, so "Local" was misleading)
- RelayHub -> InProcessRelays (URL-keyed registry that doubles as a
  WebsocketBuilder for in-JVM tests; "Hub"/"Pool" implied substitutable
  resources, but each URL maps to a distinct relay)

Also moves the test-only helpers `preload` and `publish` out of
RelayEngine and into a testFixtures extension file. As part of that,
`publish` now waits for the relay's OK reply before returning so that
publish-then-subscribe is deterministic — previously the fire-and-forget
submit let a subscription register after publish returned but before the
ingest queue fanned out, causing ephemeral kinds (not persisted) to leak
to late subscribers. Fixes the flaky
`Nip01ComplianceTest.ephemeralEventIsNotStoredAndDoesNotShowOnFollowupReq`.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-12 14:44:33 -04:00
Claude 44a8aeea28 Merge branch 'main' into claude/review-fanout-index-e1Uqq
Conflicts:
- quartz/.../LiveEventStore.kt: main added IngestQueue (group-commit)
  and a fire-and-forget submit() path; this branch replaced the
  SharedFlow live fanout with FilterIndex-driven dispatch. Resolved
  by keeping main's submit/insert pipeline shape (IngestQueue ingest
  ctor param, submit() callback, insert() wrapping submit via
  CompletableDeferred) but routing the on-Accepted fanout through
  FilterIndex.candidatesFor instead of newEventStream.tryEmit.
  Added private fanout(event) helper. Kept main's
  snapshotIdsForNegentropy method.
- geode/.../LoadBenchmark.kt: both sides added a new @Test method.
  Kept fanoutScaling (this branch) and publishPipelinedSingleClient
  (main).
2026-05-07 23:55:00 +00:00
Claude 7c4f2b720a refactor(negentropy): audit fixes for interop tests
- Delete `strfryDrivesGeodeAsServer` — boots strfry but uses
  kmp-negentropy ↔ Geode, no actual strfry-vs-geode interop.
  Duplicates `Nip77NegentropyTest.negentropyComputesSymmetricDifference`.
- Strip speculative `negentropy { enabled = ... }` and `nofiles`
  blocks from the strfry config; defaults are what we want to test.
- `InteropSyncDriver.reconcile` → `negotiate`. The function
  computes the symmetric difference; it doesn't move events.
  Convert to `suspend fun` to drop a nested `runBlocking` that
  could deadlock under dispatcher pressure.
- Inline `pullSync` helper in GeodeVsGeodeNegentropySyncTest —
  it was a one-line wrapper with a misleading name.
- Batch `relayB.preload(needFromA)` instead of looping.
- Tighten `idsOnRelay(NormalizedRelayUrl)` signature (was
  re-parsing the string on every call).
- Drop redundant `assertTrue(size > cap)` after `assertEquals(11)`.
2026-05-07 23:34:37 +00:00
Claude 809955c360 test(negentropy): strfry-style interop tests for NIP-77 sync
New geode/src/test/kotlin/com/vitorpamplona/geode/interop/ folder
mirrors strfry's test/syncTest.pl over real WebSocket frames.

- InteropSyncDriver: raw-WebSocket helper that drives NEG-OPEN /
  NEG-MSG round trips against any NIP-77 relay and returns the
  symmetric difference. No NostrClient indirection — same wire shape
  strfry sync uses.

- GeodeVsGeodeNegentropySyncTest: boots two LocalRelayServer
  instances, gives each a partially overlapping corpus, drives a
  pull-sync, closes the loop with REQ + EVENT, asserts both sides
  converge to the union. Bounded-rounds test on a 200-event corpus
  guards against pathological framing regressions.

- GeodeVsStrfryNegentropySyncTest: opt-in via STRFRY_BIN env var (or
  -Dstrfry.bin=...). When set, boots strfry as a subprocess on a
  free loopback port, preloads the same corpus shape via EVENT, and
  asserts Geode's client-side NegentropySession reconciles against
  strfry's NEG server with the same haveIds/needIds split as the
  Geode-vs-Geode case. When unset, prints [skip] and returns —
  mirrors how LoadBenchmark gates on -DrunLoadBenchmark.

Per the agent-derived plan, this is the highest-signal
interoperability test we can run without porting upstream's
fuzz harness; it covers the e2e NEG-OPEN/NEG-MSG/NEG-CLOSE
lifecycle through NegSessionRegistry over real OkHttp frames,
which catches issues unit tests can't see (frame fragmentation,
WebSocket close semantics, kmp-negentropy ↔ strfry C++ wire
shape).
2026-05-07 23:34:37 +00:00
Claude 4b7e0e880c feat(negentropy): strfry-parity NIP-77 reconciliation path
Implements the A+B+C+D plan in geode/plans/2026-05-07-negentropy-large-corpus.md:

- A: id-and-time-only snapshot. Adds IEventStore.snapshotIdsForNegentropy
  returning IdAndTime(createdAt, id) with no Event materialization. SQLite
  override projects directly off event_headers (~40 B/entry instead of
  ~1 KB/entry; matches strfry's MemoryView footprint).

- B: snapshot-size cap. New [negentropy] config section with
  max_sync_events=1_000_000 (mirrors strfry's relay__negentropy__maxSyncEvents).
  Overflow returns NEG-ERR "blocked: too many query results" (strfry-exact
  wording). No default since-window (strfry honors filters as-is; bounding
  silently would break interop).

- C: 500_000-byte frame cap. NegentropyServerSession.DEFAULT_FRAME_SIZE_LIMIT
  matches strfry's hard-coded Negentropy ne(storage, 500'000). Configurable
  via [negentropy].frame_size_limit.

- D: per-connection session cap = 200 (matches strfry). Overflow sends NOTICE
  "too many concurrent NEG requests" (strfry parity).

Error-string interop:
  - NEG-MSG with unknown subId → "closed: unknown subscription handle"
  - reconcile() parse failure → "PROTOCOL-ERROR"
  - snapshot overflow → "blocked: too many query results"
  - per-conn cap → NOTICE "too many concurrent NEG requests"

NegentropySettings flows: RelayConfig.NegentropySection → Relay → NostrServer
→ RelaySession → NegSessionRegistry. RelayHub takes optional settings for
tests that need to exercise the caps.

Tests:
  - SnapshotIdsForNegentropyTest covers projection correctness across all
    indexing strategies + the maxEntries+1 sentinel contract.
  - Nip77NegentropyTest gains negOpenSnapshotOverflowReturnsStrFryNegErr
    and negOpenPerConnectionCapEmitsNotice.
  - Existing negMsgWithoutOpenReturnsNegErr updated to assert strfry wording.
2026-05-07 23:34:37 +00:00
Claude ab71e45e77 docs(geode): refine negentropy plan for strfry interop
Align the NIP-77 large-corpus plan with strfry's actual defaults:
500_000-byte frame cap (not 64 KB), no default since-window, 200
shared sub cap, max_sync_events overflow protection, exact NEG-ERR
wording, and 32-byte id storage. Add a strfry round-trip benchmark
and call out the BTreeLMDB pre-built tree as a follow-up.
2026-05-07 23:30:27 +00:00
Claude 176fa6f3b1 docs(geode): plan reflects VerifyAuthOnlyPolicy split
Tier 3 used to say operators "must omit VerifyPolicy from their
policy chain" when parallelVerify is on — that turned out to be
the AUTH-verify regression caught in the audit. Updated the plan
to describe the real wiring: VerifyPolicy was split into a
parameterised base with two singletons, and composePolicy swaps in
VerifyAuthOnlyPolicy so AUTH commands keep signature verification
even when the IngestQueue takes EVENT verify.
2026-05-07 23:19:13 +00:00
Claude 7430c2aa17 fix(quartz): audit fixes — VerifyAuthOnlyPolicy + small wins
Self-audit of the event-ingestion-batching changes turned up one
real bug + a handful of cleanups.

Fix: AUTH events skipped signature verification when
parallelVerify=true. Previous commit dropped VerifyPolicy from the
policy chain to avoid double-verifying EVENTs (the IngestQueue does
those off-thread). But VerifyPolicy.accept(AuthCmd) was the only
thing checking AUTH signatures — FullAuthPolicy verifies challenge
/ relay / expiry but trusts the sig. Removing VerifyPolicy let a
forged event mark a pubkey as authenticated.

Split VerifyPolicy into a parameterised base class
(VerifyEventsAndAuthPolicy) with two singletons:
- VerifyPolicy: verifies both EVENT and AUTH (existing default).
- VerifyAuthOnlyPolicy: verifies AUTH only — use when an
  IngestQueue does parallel EVENT verify, since AUTH commands
  bypass the queue entirely.

Geode's composePolicy now selects VerifyAuthOnlyPolicy when
parallelVerify is on, keeping AUTH signature checks intact while
still letting EVENT verify run in parallel on the writer's CPU
fan-out.

Cleanups (no behavior change):
- IngestQueue.processBatch was ~70 lines; split into verifyBatch /
  runInsertStage / dispatchOutcomes.
- Single-event verify shortcut: skip the coroutineScope + async
  dance for batch-of-1 (the common low-load case) and call the
  hook directly.
- Hoisted the "internal error: missing outcome" Rejected sentinel
  to a companion `missingOutcome` constant.
- Imported ClosedSendChannelException / ClosedReceiveChannelException
  instead of using the fully-qualified form inline.
- Dropped NostrServer's verifyEvent companion wrapper — direct
  lambda is just as cheap and the "single instance" comment was
  inaccurate.
- ObservableEventStore.batchInsert now uses requireNoNulls() rather
  than @Suppress("UNCHECKED_CAST"), since every index is provably
  populated.
2026-05-07 23:16:52 +00:00
Claude 70afcd10ca fix(quartz): close HashSet race in LiveEventStore.query dedupe
The previous design wrapped a mutable HashSet in an AtomicReference,
which makes the *reference* thread-safe but not the set's internals.
The historical-replay closure ran `seenIds.load()?.add(event.id)`
from the subscriber's coroutine while `deliver` ran
`seen.contains(event.id)` from the publisher's coroutine —
concurrent add/contains can corrupt buckets or throw CME.

The pre-index implementation didn't have this race because both
operations ran on the same Flow collector coroutine. Index-driven
dispatch put them on different coroutines.

Switch to AtomicReference<Set<String>?> over an immutable Set with a
CAS-loop on add. Reads in `deliver` always see a fully-constructed
snapshot. Single-writer in steady state so the CAS typically
succeeds first try; the loop only matters across the
`seenIds.store(null)` post-EOSE handoff.

Also:
- hoist `NostrSignerSync(keys[target])` out of the per-iteration
  loop in LoadBenchmark.fanoutScaling.
- add FilterIndex tests for `tagsAll` selection, multi-char tag
  fall-through, and re-register key-union semantics.
2026-05-07 23:16:18 +00:00
Claude 0d0fbf36c9 test(geode): add LoadBenchmark.fanoutScaling for selective-fanout perf
Verifies the FilterIndex path in LiveEventStore: each event matches
exactly ONE of N subscribers, so without an index the relay walks
all N subs per event (O(N)); with the author-keyed index the lookup
is O(1) and per-event latency stays flat as N grows.

Different from fanoutLatency (broadcast: 1 event reaches every
sub). Here per-event work is lookup-bound rather than
delivery-bound. Configurable via -DfanoutScalingSubs (comma list)
and -DfanoutScalingEvents.

Measured (laptop, JDK 21, full sweep):
  100 subs, 2k events: p50=1.35ms p99=5.71ms
 1000 subs, 2k events: p50=1.05ms p99=3.05ms
 5000 subs, 1k events: p50=1.01ms p99=2.96ms

p50 ~1ms across N — the predicted O(1) scaling. Default events count
adapts downward at high N to stay below WebSocketSessionPump's
8192-frame outbound cap (test-client read side is the bottleneck
above ~5k subs, not the relay).

Plan doc updated with the measured numbers in the "How to verify"
section.
2026-05-07 23:05:29 +00:00
Claude 289bc4bd5c perf(quartz): Tier 3 — parallel Schnorr verify in IngestQueue
Closes the last item on the event-ingestion-batching plan: signature
verification no longer serialises on each connection's WebSocket
pump. Instead, IngestQueue takes a `verify: ((Event) -> Boolean)?`
hook and fan-outs the per-batch verify across Dispatchers.Default
(`coroutineScope { events.map { async { verify(it) } }.awaitAll() }`)
before opening the SQLite transaction. Failed verifies pre-mark
Rejected and skip the insert.

Wiring:
- NostrServer takes `parallelVerify: Boolean = false` (opt-in to
  preserve existing behaviour for direct library users).
- geode.Relay forwards a matching flag.
- Main.kt enables it whenever signature checking is on (config
  `[options].parallel_verify = true`, default true), and when so,
  composePolicy is told to skip VerifyPolicy from the chain to
  avoid double-verifying every event.
- New CLI escape hatch `--no-parallel-verify` for the legacy path.

Bench: adds publishGroupCommitSingleClient (sequential publish-and-
confirm; 500 EPS regression floor for the synchronous path) — the
companion to the existing pipelined bench that exercises the
group-commit + parallel-verify wins.

Plan doc updated to describe what shipped (batchInsert + SAVEPOINTs
in Tier 1, IngestQueue mechanics in Tier 2, the verify hook in
Tier 3) and to drop the obsolete `synchronous=NORMAL` confirmation
note — the project ships `synchronous=OFF` and intentionally keeps
that.
2026-05-07 23:00:30 +00:00
Claude 7a92f4ef2f perf(quartz): group-commit + per-connection ingest pipeline
Implements the event-ingestion-batching plan: SQLite group commit
with per-row SAVEPOINT isolation, and a per-server IngestQueue that
turns RelaySession.handleEvent into fire-and-forget. The OK frame is
emitted from the writer's callback once the row's outcome is known,
relying on NIP-01 pairing OKs by event id (not by order).

- IEventStore.batchInsert + InsertOutcome contract; SQLite override
  uses SAVEPOINTs so one bad event doesn't roll back the others.
  ObservableEventStore forwards persistable rows to the inner batch
  and emits StoreChange.Insert for accepted ones.
- IngestQueue drains submissions in batches up to 64 per
  transaction. Writer coroutine starts lazily on the first submit
  so subscription-only sessions don't pay for it (and don't perturb
  Default-dispatcher scheduling — the eager launch was visible as
  intermittent NostrClientRepeatSubTest flakes under full-suite
  load).
- RelaySession.handleEvent posts to the queue and returns
  immediately; the WS pump moves to the next frame instead of
  awaiting SQLite. ClosedSendChannelException during shutdown
  surfaces as OK false rather than crashing the pump.
- LiveEventStore.submit fans an event onto the live stream only
  after the writer reports Accepted; the suspending insert is
  retained for tests, routed through the same queue.
- New publishPipelinedSingleClient benchmark in geode.perf:
  10 000 EVENTs back-to-back without awaiting OKs, asserts every
  event id receives exactly one OK (in any order).
2026-05-07 22:41:40 +00:00
Claude 3946117084 perf(quartz): index-driven fanout for LiveEventStore via FilterIndex<S>
Replace the SharedFlow-based broadcast in LiveEventStore with an
inverted index over filter-bearing subscribers. Each REQ registers
its filters into a per-store FilterIndex<LiveSubscription>; insert()
calls index.candidatesFor(event) and only delivers to candidates
whose Filter.match still passes. Cuts the per-event walk from
O(N_subs * N_filters) to a few hash lookups plus match() over a
small candidate set.

FilterIndex itself lives next to Filter.kt (commonMain, KMP-friendly,
AtomicReference + COW) so other call sites with the same shape
(LocalCache.observables, ObservableEventStore.changes) can reuse it.
Each filter contributes entries on its single most-selective dimension
(ids > authors > tags > tagsAll > kinds > unindexed) to keep buckets
narrow and avoid Set-dedupe work in candidatesFor.

The historical-replay race the previous SharedFlow + onSubscription
handoff closed is preserved by registering BEFORE replay starts and
deduping seen ids until EOSE.
2026-05-07 22:37:31 +00:00
Claude 39e81c1eb5 docs(geode): correct OK ordering/durability assumptions in ingestion plan
OK frames carry the event id, so clients pair replies by id rather
than by arrival order. NIP-01 also treats OK true as "accepted," not
"fsynced." That removes two constraints the plan was carrying:

- no per-connection FIFO requirement on OKs
- no need to delay OKs until after batch fsync

Tier 1 can fan OKs out as soon as the per-row INSERT returns inside
the open transaction, hiding the group-commit fsync entirely from
publisher latency. Tier 2 drops the order-preserving commit log and
just sends OKs straight to outQueue. The pipelined benchmark now
checks one-OK-per-event-id rather than ordering.
2026-05-07 21:35:49 +00:00
Claude 6f8e7bb520 docs(geode): update connection-scaling plan to reflect what shipped
Marks Sketches A and B done, with a note that A took the simpler
Channel.UNLIMITED + AtomicInteger cap path the original Risks section
called out, sidestepping the channel-swap that the plan first
sketched.

Records that the streaming-filter slice of Sketch C landed in Quartz,
and that the larger envelope-streaming work the plan called out is
unnecessary because MessageDeserializer / CommandDeserializer /
EventDeserializer were already streaming — only the filter sub-object
went through readTree.

Adds the verification benchmarks (connectionsHeldOpen10k,
connectionsHeldOpenWithFanout) to the verification section, with a
correction that what's measured is JVM heap not OS RSS.

Carries forward the not-done items (fan-out de-duplication,
filter-matching index, Netty engine) into the open-work section,
pointing at live-broadcast-fanout-index.md for the highest-leverage
remaining work.
2026-05-07 21:07:03 +00:00
Claude 6c792fdf44 audit(geode): drop dead firstSeenNs + clarify heap-vs-RSS in LoadBenchmark
Pre-merge audit findings on the connection-scaling perf changes:

- connectionsHeldOpenWithFanout populated firstSeenNs but never read
  it — only lastReceiveNs feeds the p50/p99 latency metric. Drop the
  unused map and rename the latency map to lastReceiveNs to match
  what it actually holds.
- connectionsHeldOpen10k printed/asserted on a variable named rssMb
  that was actually JVM heap (`Runtime.totalMemory - freeMemory`).
  Rename to heapMb, force a GC + 200 ms settle before reading so the
  number reflects retained bytes rather than connect-ramp churn, and
  update the assertion message to say "JVM heap" not "heap usage".
2026-05-07 20:50:10 +00:00
Claude 0f7d8696de perf(geode): adaptive outQueue + CIO pool sizing for 10k+ connections
Implements geode/plans/2026-05-07-connection-scaling.md to push the
single-relay connection ceiling past the ~2k floor measured by
LoadBenchmark.connectionsHeldOpen.

- WebSocketSessionPump: switch outQueue to Channel.UNLIMITED bounded
  by an AtomicInteger backlog cap. Idle connections no longer reserve
  an 8 192-slot fixed buffer; lazy-allocated head segments cost only
  a few hundred bytes per idle session. Slow-client policy preserved:
  once outstanding > MAX_OUTGOING_BUFFER, the queue is closed and
  the connection drops, so NIP-01 ordering is never silently
  violated.
- LocalRelayServer / RelayConfig.NetworkSection: expose CIO
  connectionGroupSize / workerGroupSize / callGroupSize so big-VM
  operators can tune Ktor's event-loop pools without forking. Switch
  to the serverConfig {} + configure {} embeddedServer overload so
  CIO tunables can be set.
- LoadBenchmark: add connectionsHeldOpen10k (asserts 10 000 idle
  WebSockets settle under a 1 GiB heap ceiling) and
  connectionsHeldOpenWithFanout (5 000 subscribers, 10 EPS fan-out
  for 10 s, reports p50/p99 last-fanout latency).
- config.example.toml: document the three CIO tunables and the
  ulimit -n requirement for operators targeting >1k connections.
2026-05-07 20:12:50 +00:00
Claude 2c0ad4fbf5 docs(geode): performance plans for future work
Four sketches, queued by impact, each grounded in current code paths
and observed benchmark numbers:

- event-ingestion-batching: SQLite group commit + EVENT pipelining +
  off-thread Schnorr verify. Targets 5–10× EPS on a fast SSD.
- live-broadcast-fanout-index: indexed filter matching to replace the
  O(N_subs × N_filters) per-event walk in LiveEventStore. Targets
  flat fanout p99 up to high subscriber counts.
- connection-scaling: shrink the per-session outQueue footprint
  (currently the dominant per-conn cost), tune Ktor CIO group sizes,
  reduce JSON parse allocations. Targets 10 000+ concurrent conns.
- negentropy-large-corpus: id-and-time-only snapshot path so NEG-OPEN
  on a 5M-event store doesn't materialise full Event objects, plus
  bounded-window defaults and concurrent-session caps.

Each plan names the verification benchmark to add. Plans are queued,
not committed work — README orders them by expected impact.
2026-05-07 14:05:11 +00:00
Claude c19bd4e92e refactor(geode): test fixtures + collectUntilEose helper, move to testFixtures sourceSet
The in-process relay was usable but every consumer test paid a 10–15
line tax for scope/client setup, manual cleanup inside the test body
(leaks on assertion failure), hardcoded "ws://127.0.0.1:7770/" magic
strings, and an inline collectUntilEose pattern reinvented per file.

Net: ~250 LOC removed, every test gets correct @After cleanup, and
the synthetic event builders no longer ship in geode's production jar.

- Move geode.fixtures (synthetic event builders) and the new
  geode.testing package from src/main to a new src/testFixtures
  source set via the java-test-fixtures plugin. Production geode jar
  no longer includes them; consumers wire them with
  testImplementation(testFixtures(project(":geode"))).
- Add geode.testing.RelayClientTest open class — owns hub, scope,
  client, defaultRelay, defaultRelayUrl with @After-driven cleanup.
- Add geode.testing.collectUntilEose / collectUntilEoseMulti
  NostrClient extensions with a shared subId counter and timeout knob.
  Replaces hand-rolled subscriber loops in 4+ files.
- Add RelayHub.DEFAULT_URL constant so tests stop typing
  "ws://127.0.0.1:7770/" inline.
- Migrate the 8 quartz NostrClient*Test files + geode's
  Nip01ComplianceTest to extend RelayClientTest and (where REQ/EOSE
  is the pattern) call collectUntilEose. Delete the redundant
  BaseNostrClientTest wrapper.
2026-05-07 13:42:20 +00:00
Claude c2f24a5213 refactor: rename quartz-relay module to geode
The relay implementation now stands as its own module. Fitting brand
for a Nostr relay shipped alongside the Quartz library — a geode is a
rock that holds quartz inside.

- Rename module directory quartz-relay/ -> geode/.
- Rename Gradle module path :quartz-relay -> :geode (settings.gradle).
- Rename Kotlin package com.vitorpamplona.quartz.relay -> com.vitorpamplona.geode.
- Rename application name + main class binding to match.
- Update the relay's NIP-11 advertised name to "geode" and the
  software URL to /tree/main/geode. Test asserting the doc updated.
- Refresh comments / Main.kt usage line / config.example.toml header.
- Update consumers: quartz/build.gradle.kts dependency path, and
  quartz NostrClient tests that import the in-process RelayHub.
2026-05-07 13:12:34 +00:00