amethyst/geode/plans/2026-05-07-connection-scaling.md

Connection scaling: pushing past 2 000

Status (2026-05-07): Sketches A and B shipped on claude/connection-scaling-plan-YVjc8. Sketch C landed as a smaller slice in Quartz — the streaming-filter cut — once the audit showed the rest of the plan's premise was overstated. Verification benchmarks (connectionsHeldOpen10k, connectionsHeldOpenWithFanout) are wired up but only run under -DrunLoadBenchmark=true. Remaining open work, including fan-out de-duplication, is now tracked in live-broadcast-fanout-index.md.

Problem

Current measurement (LoadBenchmark.connectionsHeldOpen): ~2 000 concurrent connections before file-descriptor pressure / Ktor CIO event-loop saturation. Real-world relays (e.g. nostr.wine, nos.lol) sustain 10–30k. Geode shouldn't be the bottleneck for an Amethyst- adjacent operator who scales beyond a thousand-user community.

What's spending memory per connection today

Cost	Per connection	At 5 000 conns
outQueue Channel	8 192 string slots × ~8 b ref	~320 MB pinned
RelaySession	LargeCache<String, Job> for subs (likely 1–10 entries)	~negligible
NegSessionRegistry	HashMap<String, NegentropyServerSession> — usually 0	~negligible
Ktor CIO buffers	TCP read + write buffers	~10 MB
Per-session writer Job	one coroutine	~few KB

The outQueue reservation is the dominant cost. The 8 192 was sized for a worst case "thousands of subscriptions, one event matches all" — but at 5 000 connections we've over-provisioned by ~300 MB just on the channel array, even though most connections never fan out.

Sketch

A — adaptive outQueue capacity ✅ shipped

Original plan: start at INITIAL_OUTGOING_BUFFER = 64 and swap to a wider channel under a per-session lock when a high-water mark trips. Not how it shipped.

What actually shipped is the simpler alternative the original Risks section called out: Channel.UNLIMITED plus an AtomicInteger backlog cap. kotlinx.coroutines' BufferedChannel allocates segments lazily, so an unlimited channel pays only the small head-segment cost on idle connections — there is no preallocated buffer to scale.

private val outQueue = Channel<String>(capacity = Channel.UNLIMITED)
private val outstanding = AtomicInteger(0)

// producer side
val depth = outstanding.incrementAndGet()
if (depth > MAX_OUTGOING_BUFFER) {            // 8192
    outstanding.decrementAndGet()
    droppedForBackpressure = true
    outQueue.close()                          // NIP-01: drop the conn
    return@connect
}
val res = outQueue.trySend(json)
if (!res.isSuccess) outstanding.decrementAndGet()  // closed concurrently

// writer side
for (json in outQueue) {
    ws.outgoing.send(Frame.Text(json))
    outstanding.decrementAndGet()
}

Memory characteristic the plan asked for is intact: idle connections no longer reserve an 8 192-slot fixed buffer; hot fan-out connections still get bounded at the same 2 MiB cap before the slow-client cutoff fires. NIP-01 ordering is preserved (no silent drop — connection is killed at the cap).

Implementation: geode/.../server/WebSocketSessionPump.kt. The channel-swap approach was rejected because Channel.UNLIMITED already gives the lazy-allocation behavior the swap was simulating, with none of the swap's race surface.

B — per-relay event-loop pool sizing ✅ shipped

Three optional knobs added to [network] in RelayConfig:

[network]
host = "0.0.0.0"
port = 7447
path = "/"
# connection_group_size = 4
# worker_group_size = 16
# call_group_size = 64

Default is null (Ktor default) — no behavior change unless an operator explicitly tunes them. The values are wired through LocalRelayServer into the new embeddedServer(factory = CIO, rootConfig = serverConfig {…}, configure = {…}) overload (the short-form embeddedServer(factory, host, port) {…} overload doesn't expose CIO config). The auto-connector that the short form created now has to be added explicitly via connector { host = …; port = …}.

config.example.toml documents the knobs and includes the operator note that targeting >1k connections needs ulimit -n 65536 (or matching LimitNOFILE= in a systemd unit).

C — reduce per-message JSON allocations ✅ partially shipped (in Quartz)

Original plan claim: "OptimizedJsonMapper.fromJsonToCommand allocates a JsonNode tree per incoming frame." Overstated.

Audit of quartz/.../jackson showed the Command/Message envelope is already streaming:

Path	Already streaming?	Tree alloc?
MessageDeserializer	yes	only for COUNT result (rare)
CommandDeserializer	yes	only for filter sub-objects in REQ/COUNT/NEG-OPEN
EventDeserializer	yes	none — currentName().hashCode() dispatch
ManualFilterDeserializer	no	jp.codec.readTree(jp) per filter

So the only relay-inbound tree allocation worth chasing was filter parsing — the bulk of the per-frame allocations on a REQ-heavy relay.

What shipped: a streaming ManualFilterDeserializer.fromJson(jp: JsonParser) modeled exactly on EventDeserializer. Token-loop with field-name dispatch (ids / authors / kinds / since / until / limit / search, plus dynamic #x / &x tag keys), and readStringArray / readIntArray helpers that drop invalid entries silently to match the tree path's mapNotNull { asTextOrNull() } tolerance. Wired into all four internal call sites: FilterDeserializer.deserialize and the three CommandDeserializer paths (REQ, COUNT, NEG-OPEN).

The tree-based fromJson(ObjectNode) overload is retained for external/cross-format adapters (Quartz is a published library).

What was NOT done — and why:

• Streaming Jackson for the Command envelope: already streaming. No allocation to remove.
• kotlinx-serialization for the inbound path: not pursued. The cross-mapper round-trip tests in KotlinSerializationMapperTest show the two formats are interchangeable, but the engine swap is a much larger lift than the filter cut and there's no evidence the KS path is faster on this code shape.
• Per-session ObjectMapper: Jackson's ObjectMapper is thread-safe and stateless — sharing one is the recommended pattern. Per-session would increase allocation, not decrease it.

How to verify ✅ shipped

Two new benchmarks in geode.perf.LoadBenchmark, gated behind -DrunLoadBenchmark=true:

• connectionsHeldOpen10k — opens 10 000 idle WebSocket connections, asserts every one settles to EOSE inside 120 s, and measures retained JVM heap (after System.gc() + 200 ms settle) with a 1 GiB ceiling assertion. Requires ulimit -n 32768 on Linux.
• connectionsHeldOpenWithFanout — 5 000 subscribers all matching kinds:[1], one publisher emitting targetEps × duration events, prints p50 / p99 last-fanout latency. No assertion on latency — just regression-detection via stdout logging.

The original connectionsHeldOpen benchmark stays as the baseline floor (~2 000 conns) for before/after comparisons.

Note on heap-vs-RSS: the original plan said "RSS stays under 1 GB" but the JVM can only measure heap from inside; `Runtime.totalMemory

• freeMemory` is what the benchmark asserts on. RSS will be higher because of code, native buffers, off-heap (Ktor CIO), etc.

Risks (post-implementation)

• Adaptive channel swap is fiddly — sidestepped by using Channel.UNLIMITED instead of swapping bounded channels.
• Bumping CIO group sizes can hurt — kept the defaults null. Operators must opt in, and the docstrings explicitly say to benchmark before/after.
• OS-level FD limit — documented in config.example.toml next to the CIO knobs. Test prereq is also documented in the benchmark KDoc.

Open work

• Fan-out de-duplication — when one EVENT matches N subscribers, we currently re-serialize and copy the JSON N times into N channels. Caching one pre-serialized payload per event and broadcasting a shared reference is a much bigger win than anything in this plan; tracked in live-broadcast-fanout-index.md.
• Filter-matching index — same plan. At 10k conns × ~5 filters that's 50k evaluations per published EVENT, almost all of which could be culled by indexing subscriptions on kinds / authors / #e / #p.
• Netty engine evaluation — Ktor's Netty engine handles many idle connections with measurably lower per-connection overhead than CIO. Not pursued here because it changes the transport layer wholesale; revisit only if the CIO knobs in (B) prove insufficient for an operator at 20k+ connections.