Splits "events accepted for persistence" from "events accepted for
observation". The new ObservableEventStore wraps any IEventStore and
owns events: SharedFlow<Event>:
- Non-ephemeral events forward to the inner store; success → emit, any
rejection (expired, NIP-09 / NIP-62 tombstone, NIP-01 supersession
loser) → no emit.
- Ephemeral events (kinds 20000-29999) skip persistence entirely but
still emit, so an open EventStoreProjection renders them while alive.
They vanish from any future seed because the DB never had them.
Reverts the previous round of plumbing inside SQLiteEventStore /
FsEventStore: stores no longer carry an inserts SharedFlow. IEventStore
goes back to a clean read/write contract. ObservableEventStore is the
single place that decides what makes it onto the projection bus.
EventStore (the SQLite convenience class) embeds an ObservableEventStore
internally so existing call sites like `store.observe(filter, scope)`
keep working without changes.
EventStoreProjection now collects via Flow.onSubscription so the
collector subscription is established before the seed query runs —
fixes a race where an insert immediately after `ready.await()` could
land in the SharedFlow before the collector was subscribed.
Tests:
- ephemeralEventsAppearInProjection — kind-22000 event reaches items
but isn't queryable from the inner store, and a fresh projection
on the same store gets an empty seed.
- 14/14 projection tests + 219/219 other store tests pass.
https://claude.ai/code/session_01Jny85MTu1ynKgFBgysfWu5
The projection now sits on top of any IEventStore, not just SQLite.
Stores publish a simple `inserts: SharedFlow<Event>` (one event per
successful insert), and the projection itself replays the NIP rules
against that stream:
- NIP-01 supersession with the lexical-id tiebreaker, for both
replaceables (kind 0/3/10000-19999) and addressables (30000-39999).
Both now share a single in-place update path.
- NIP-09 deletions, with the original-author check (cross-author
kind-5s are inert, matching the store).
- NIP-62 right-to-vanish, scoped by the projection's relay arg.
- NIP-40 expiration via a per-projection ticker that drops slots whose
expiration tag has lapsed.
Out-of-band store mutations (`delete(id)`, `clearDB()`, the periodic
`deleteExpiredEvents()` sweep) are no longer visible to projections —
they're maintenance ops; projections re-seed when their scope is
restarted.
Removed from SQLiteEventStore:
- ChangeLogModule + temp-table + AFTER DELETE trigger.
- StoreChange sealed type and the per-mutation drain/publish path.
Both SQLiteEventStore and FsEventStore now satisfy
`IEventStore.inserts`. FsEventStore.insertLocked returns Boolean so
no-op idempotent retries (canonical already on disk) don't re-publish.
Tests:
- EventStoreProjectionTest moves to `store/projection/`, gains four
new cases: NIP-01 out-of-order rejection, NIP-09 cross-author
inertness, NIP-62 cross-author no-op, NIP-40 ticker-driven removal.
- 13/13 projection tests + 232/232 total store tests pass.
https://claude.ai/code/session_01Jny85MTu1ynKgFBgysfWu5
Adds a stateful observer that turns a Filter into a StateFlow<List<MutableStateFlow<Event>>>:
- A TEMP table + AFTER DELETE trigger on event_headers logs OLD.id for every row
that leaves the store, regardless of cause (replaceable / addressable
supersession, NIP-09 deletion, NIP-62 vanish, NIP-40 expiration sweep, manual
delete, clearDB).
- SQLiteEventStore drains the log around each writer unit of work and emits a
StoreChange(inserted, removedIds) on a SharedFlow.
- EventStoreProjection seeds itself from the store, then maintains stable
MutableStateFlow handles keyed by (kind:pubkey:dtag) for addressables and by
id otherwise. Addressable updates mutate the handle's value in place — list
reference stable; insert / remove rebuilds the list reference.
- 11 new tests cover seed, insert, replaceable update, addressable update,
NIP-09 deletion, NIP-62 vanish, NIP-40 expiration, manual delete, limit
enforcement, non-matching insert, and close-cancels-listener.
https://claude.ai/code/session_01Jny85MTu1ynKgFBgysfWu5
The triple benchmark probed libschnorr256k1 with a try/catch that only
handled UnsatisfiedLinkError. On Windows the loader raises
IllegalStateException when the JNI .dll for the platform isn't packaged,
which propagated out as a test failure instead of skipping the C row.
Mirror the broader catch already used in Secp256k1CrossValidationTest so
the benchmark prints a SKIP-style note and continues with the ACINQ +
Kotlin rows on platforms where the C lib isn't available.
The custom C secp256k1 implementation under quartz/src/main/c/ was never
wired into Gradle (no externalNativeBuild block) and only existed for the
3-way benchmark + cross-validation test against ACINQ on JVM/Android.
The C library has been split out to vitorpamplona/libschnorr256k1{,-kmp},
so the in-repo copy is dead weight.
This change replaces it with the published Maven Central artifact
`com.vitorpamplona:schnorr256k1-kmp:1.0.0`, scoped to test/benchmark
configurations only — production crypto continues to use ACINQ
secp256k1-kmp on JVM/Android and the pure-Kotlin Secp256k1 on native.
The Android AAR ships libschnorr256k1_jni.so for arm64-v8a and x86_64,
so the Android benchmark now exercises the C row automatically (no more
manual build_android.sh). On JVM the .so still has to be installed by the
developer; the cross-validation test and triple-benchmark gracefully skip
the C row when System.loadLibrary fails, matching prior behavior.
Verified on JVM: all 13 secp256k1 jvmTest classes pass (188 tests), and
ACINQ vs pure-Kotlin benchmark numbers match the documented baseline
within sandbox noise (verifySchnorr ~15k ops/s, signSchnorr cached
~33k ops/s, pubkeyCreate+Compress ~36k ops/s).
Removes ~5,100 LOC: quartz/src/main/c/ (4,500), Secp256k1InstanceC.*
expect/actual shim (560), Secp256k1C JNI declaration object.
https://claude.ai/code/session_01KnvpK2amcVZKfFiZJvHjVe
After verifying the nostrnests reference (NestsUI-v2 @ main):
- ProfileCard.tsx writes kicks as ['action','kick'] tags with empty content
- useAdminCommands.ts reads action via tags.find(t => t==='action') and
applies a 60-s relay since plus a processedRef Set to dedup re-deliveries
- p-tag role marker for moderators is 'admin', not 'moderator'
Amethyst was diverging on every one of those, which means our outbound
admin commands were invisible to nostrnests, theirs to us, and any
nostrnests admin (role='admin') failed our isModerator() / canSpeak()
gates entirely — kicks and force-mutes signed by them were silently
dropped.
Changes:
quartz/AdminCommandEvent.kt
- Emit ['action', '<verb>'] tag with empty content
- Reader prefers the tag, falls back to content for any in-flight
Amethyst-built kick from before this commit
- kick() and forceMute() share a common build() helper
quartz/ParticipantTag.kt
- ROLE.MODERATOR.code = 'admin' (matches nostrnests + EGG-07)
- Adds legacyCodes = ['moderator'] so older Amethyst-emitted
kind-30312 events still parse as MODERATOR
- effectiveRole() walks both code + legacyCodes
amethyst/AdminCommandsCollector
- Filter carries since = now - 60 (EGG-07 #7)
- Defensive per-event freshness re-check for cached events / clock skew
- mutableSetOf<String>() processed-id dedup for the lifetime of the
collector, mirroring useAdminCommands.ts's processedRef
EGG-07.md
- Documents the Amethyst-only ['action','mute'] extension under a new
'Implemented extensions' section. nostrnests doesn't emit or honour
it today; cross-client force-mutes only work between Amethyst peers
- 'warn' stays in the future-actions list — nostrnests has no plans
for it either
Tests:
- ParticipantTagTest: new asserts that 'admin' / 'Admin' / 'ADMIN'
parse as ROLE.MODERATOR; pins the wire string to 'admin' and the
legacy alias to 'moderator'
- AdminCommandEventTest: kick/forceMute templates carry ['action', _]
tag with empty content; legacy content-form still parses; tag wins
over content when both are present
Three improvements stacked into one commit since they share files:
#3 Empty-stage hint: StageGrid no longer disappears when nobody is on
stage. The "Stage" label stays and a quiet "Waiting for speakers…"
line keeps the strip visible so the room doesn't look broken before
the first speaker arrives.
#5 Local per-speaker hush: AudioPlayer gains a setVolume(Float)
default-no-op method; AudioTrackPlayer composes mute and volume
multiplicatively into AudioTrack.setVolume so a hushed stream stays
silent regardless of mute state. NestViewModel exposes
locallyHushed: ImmutableSet<String> in NestUiState plus
setLocalHushed(pubkey, hushed) — applied at attach time so a
re-subscribe of an already-hushed speaker stays silent. New "Hush
this speaker" / "Restore this speaker" row in
ParticipantHostActionsSheet, available to anyone (it affects only
our own playback, nothing on the wire).
#4 Host moderation gaps: wires Promote-to-Moderator using the existing
RoomParticipantActions.setRole(ROLE.MODERATOR) builder — UI gap
only, no protocol change. Adds a new AdminCommandEvent.Action.MUTE
variant + AdminCommandEvent.forceMute(room, target) builder; the
sheet emits it on "Force-mute speaker" and the
AdminCommandsCollector dispatches incoming MUTE actions to a new
NestViewModel.onForceMuted() that routes through the existing
setMicMuted(true) path. Honor-based, same trust model as KICK —
relays don't enforce signer authority, the client checks the
signer is host or moderator on the active kind-30312.
The first-party nostrnests web client (NestsUI-v2) emits kind-30312
events using NIP-53 *streaming-event* tag names rather than the
kind-30312 names defined by NIP-53 itself. Confirmed against
`nostrnests/nests/NestsUI-v2/src/components/EditRoomDialog.tsx` and
`useRoomList.ts`:
- room name: `title` (NIP-53 spec for 30312: `room`)
- MoQ relay: `streaming` (NIP-53 spec for 30312: `endpoint`)
- moq-auth: `auth` (NIP-53 spec for 30312: `service`)
- status: `live` (NIP-53 spec for 30312: `open` / `private`
/ `closed` / `planned`)
Our parsers followed the NIP-53 spec, so events from the production
nostrnests web app fell through to status=null, which `checkStatus`
then surfaced as "Ended" in the live-rooms UI. Add legacy-alias
acceptance to all four parsers — read-side only; we still emit the
canonical NIP-53 forms, matching EGG-01.
Per-tag changes:
StatusTag: "live" → OPEN, "ended" → CLOSED
RoomNameTag: "title" alias → room
EndpointUrlTag: "streaming" alias → endpoint
ServiceUrlTag: "auth" alias → service
No emit-side changes; no spec change.
https://claude.ai/code/session_01RDpuki4t8StSg1CZcXnV5b
Wire MeetingSpaceEvent's existing RecordingTag into the
note-view renderer. Closed rooms (status=CLOSED) that ship a
`["recording", url]` tag get an OutlinedButton that hands the
URL to the system media player via ACTION_VIEW — most users have
a podcast / audio app registered for HTTPS audio URLs.
Live and scheduled rooms keep the Join button. Closed rooms
without a recording render no CTA — there's nothing to enter.
Adds:
* MeetingSpaceEvent.recording() accessor
* TagArrayBuilder<MeetingSpaceEvent>.recording(url) DSL builder
* ListenToRecordingButton composable in the note renderer
Note: nostrnests' moq-lite refactor dropped the LiveKit-era
recording HTTP surface, but the kind-30312 `recording` tag is
still spec-allowed and individual hosts can populate it via
out-of-band capture. This commit makes Amethyst the receiving
end for any host who does.
Closes the deferred font tag from the Tier-3 plan. Wire-up:
* quartz: FontTag(family, optionalUrl) parser/assembler with
blank-rejection on family + blank-URL-becomes-null normalisation.
* MeetingSpaceEvent.font(): FontTag? accessor.
* TagArrayBuilder.font(family, url) DSL.
* RoomTheme gains fontFamily / fontUrl fields.
* AudioRoomThemedScope maps `family` to a Compose FontFamily for
the four CSS-style generic-family names ("sans-serif", "serif",
"monospace", "cursive") — each maps to the corresponding system
fallback. The whole Material3 typography is rebuilt with that
family so headlines / body / labels all swap together.
URL-based font loading (RoomTheme.fontUrl) is the natural follow-up:
fetch + cache via OkHttp, then build a FontFamily from a local
file. Until then, an unknown family is silently a no-op — the room
renders in the platform default rather than crashing or fetching
on the UI thread.
Tests:
* FontTagTest — 9 cases covering family-only, family+URL,
blank rejection, missing family, wrong tag name, blank-URL
normalisation, assembler shapes (with + without URL), and
the assembler's blank-family rejection.
* RoomThemeTest gains 3 cases for font projection (family-only /
family+URL / empty event).
Shipped the deferred slice of Tier 1 Step 4: hosts can now
schedule a room for a future time instead of going live
immediately.
Quartz:
StatusTag.STATUS.PLANNED — new enum value alongside OPEN /
PRIVATE / CLOSED. Pre-Lite-03
clients that don't know the value
get null from the parser; the room
renderer's existing fallback handles
that. The amethyst feed renders
PLANNED with the OPEN badge in v1
(a "Scheduled — starts at HH:MM"
chip is a visual follow-up).
StartsTag — `["starts", "<unix-seconds>"]` parser + assembler.
Strict numeric — non-numeric values return null so
a malformed event can't crash the room-list
renderer. Negative values are rejected at assemble.
MeetingSpaceEvent.starts() — accessor.
TagArrayBuilderExt.starts(unixSeconds) — DSL helper.
Amethyst:
CreateAudioRoomViewModel —
onScheduledToggle(scheduled) — flips PLANNED vs OPEN.
onScheduledStartChange(unixSeconds) — picker callback.
FormState.scheduled / .scheduledStartUnix — additive fields.
canSubmit gates on a picked time when scheduled = true.
publishAndBuildLaunchInfo() — emits status=PLANNED +
`["starts", <unix>]` when
scheduled; otherwise the
existing OPEN path runs.
CreateAudioRoomSheet —
Schedule toggle (Material3 Switch) above the picker.
ScheduleStartPicker — OutlinedButton that opens a Material3
DatePickerDialog. The selected date is
saved as 00:00 of that day in the
local time zone (TimePicker stitching
is a follow-up; nostrnests' web UI
does the same date-only flow).
MeetingSpace.kt feed item — added the PLANNED branch to the
exhaustive when so the build passes.
Tests:
StartsTagTest — numeric parse, malformed-rejected, missing /
wrong-name rejection, assemble shape, negative rejected,
STATUS.PLANNED parse round-trip.
Adds the Tier-3 minimum-viable theming primitives so a themed
nostrnests room renders without crashing the client:
ColorTag — `["c", "<hex6>", "background"|"text"|"primary"]`.
Strict 6-char hex parser; `#abc` shorthand and
named colors are REJECTED so a typo'd room
event can't crash the renderer. Output hex is
normalised uppercase, no `#` prefix.
BackgroundTag — `["bg", "<url>", "tile"|"cover"]`. Mode defaults
to COVER when missing; unknown modes (a future
"blur") fall back to COVER until the renderer
learns them. Empty URL is rejected.
MeetingSpaceEvent.colors() / .background() — tag accessors. The
font tag (`["f", family, optionalUrl]`) is intentionally NOT in
this commit; loading a custom FontFamily would need a per-pack
loader, and font-less rooms still render fine on the client.
Tests:
ColorTagTest — happy path, no-`#` prefix, rejects shorthand /
named / unknown target / missing target;
assemble normalisation; round-trip.
BackgroundTagTest — happy path, default-COVER, future-mode
fallback, rejects empty URL, round-trip.
The Compose `RoomTheme` projection + `AudioRoomThemedScope`
renderer come next.
Adds the ephemeral host-issued admin command event nostrnests uses
for kick (and future moderation actions like mute/ban):
AdminCommandEvent — kind 4312. Carries one Action (currently just
KICK) in `content`, an `a`-tag pointing at the room (kind-30312
address), and a `p`-tag for the target. The verb-in-content shape
lets us extend with new actions without a wire schema bump.
AdminCommandEvent.kick(roomATag, target) — builder.
AdminCommandEvent.action() / .targetPubkey() / .room() — accessors
for the recipient side. Unknown actions return null (forward-compat
with verbs not yet in the enum).
EventFactory — registered so LocalCache + Filter.match can decode
incoming events properly.
Authority enforcement is the CLIENT'S job — the relay just stores
and forwards. Recipients filter on `kinds=[4312], #a=[room],
#p=[me]` and only honour commands whose signer is a participant
marked HOST or MODERATOR on the active kind-30312. The next commit
wires that gating + the disconnect side effect into AudioRoomViewModel.
Tests:
* Build a kick template and verify the action/address/target tags
* Round-trip parse exposes room, target pubkey and action
* Unknown verb in `content` returns null from action()
* Missing tags return null from accessors (no throw)
Adds typed role accessors to make participant-list filtering
self-documenting:
ParticipantTag.effectiveRole(): ROLE? — case-insensitive parse;
null when the role string doesn't match any enum value (so an
unknown future "director" role doesn't accidentally pass canSpeak).
ParticipantTag.isHost() / isModerator() / isSpeaker() — single-role
predicates for per-row UI gating.
ParticipantTag.canSpeak() — true for HOST / MODERATOR / SPEAKER;
the audio-room VM uses this to gate startBroadcast() so anyone who
was promoted (not just the original host) can publish.
5 unit tests cover happy paths, case-insensitivity, the
unknown-role → null contract, the canSpeak union, and the
effectiveRole enum parse.
Tier 1 #5 + #6 consume these — promote/demote and kick both need
to render different rows depending on whether the local user is a
host or moderator (allowed to manage roles) and the target
participant's current role.
Adds two NIP-53 presence tags used by nostrnests' room UI:
["publishing", "0|1"] — peer is actively pushing audio packets
["onstage", "0|1"] — peer holds a speaker slot vs audience
These complement the existing "hand" and "muted" tags. They're
independent: a speaker can be onstage but paused (onstage=1,
publishing=0), or broadcasting silently (onstage=1, publishing=1,
muted=1). The participant grid (Tier 2) and listener counter (Tier 1
#8) consume them.
- PublishingTag / OnstageTag classes mirror HandRaisedTag's parse +
assemble shape exactly so the DSL (TagArrayBuilderExt) extends
uniformly.
- MeetingRoomPresenceEvent.publishing() / onstage() accessor parsers
return Boolean? (null when the tag isn't present) so callers can
distinguish "explicitly false" from "absent".
- The MeetingSpaceEvent overload of build() now accepts both fields
as nullable params; the MeetingRoomEvent overload deliberately
doesn't (those are tier-2 video meetings, not Clubhouse rooms).
- Round-trip tests pin the wire format and the "absent → null"
behaviour.
nostrnests's reference README under "Nostr Integration" already declares:
kind:10112 — User-published audio server lists
We initially picked 10062 (mirroring BlossomServersEvent's 10063) without
spotting that prior claim. Switching to 10112 so a single replaceable
event surfaces in both Amethyst and nostrnests's web UI without
collision.
No migration cost — no users have published kind 10062 yet.
Two adjacent additions so users can both create and host their own
audio rooms from inside Amethyst:
Create-space flow
- CreateAudioRoomSheet — modal bottom sheet on AudioRoomsScreen
surfaced by a new "Start space" FAB. Fields: room name, summary,
MoQ service URL, MoQ relay endpoint, optional cover image.
- CreateAudioRoomViewModel — builds + signs MeetingSpaceEvent
(kind 30312, status=OPEN, tagging the user as `host`),
broadcasts via account.signAndComputeBroadcast, then returns
launch info so the sheet can fire AudioRoomActivity straight
into the freshly-published room.
- Defaults pull the first saved Nests server (below) when present;
fall back to https://moq.nostrnests.com.
Nests servers settings (proposed kind 10062)
- NestsServersEvent — replaceable kind-10062 event listing the
user's preferred audio-room MoQ servers. Wire shape mirrors
BlossomServersEvent (one `server` tag per base URL); registered
in EventFactory; consumed by LocalCache via consumeBaseReplaceable.
- NestsServerListState — per-account observation state, mirror of
BlossomServerListState, exposed on Account.nestsServers.
- sendNestsServersList on Account; included in
accountSettingsEvents() so an outbox change republishes it.
- Filter additions: BasicAccountInfo + AccountInfoAndLists now
request kind 10062 from relays.
- NestsServersViewModel + NestsServersScreen — Settings UI to
add / remove / reset to recommended servers (currently just
nostrnests.com). Wired into AllSettingsScreen as "Audio-room
servers"; routed via Route.EditNestsServers.
- kind_nests_servers label for RelayInformationScreen.
Default suggestion list lives in DEFAULT_NESTS_SERVERS at the top
of NestsServersScreen — add new community-run moq-rs deployments
there as they come online.
- Add a Concurrency section to the SQLite store README covering the
Room-style 1-writer + N-reader pool, the in-memory degradation, and
the non-reentrant Mutex contract.
- Refresh the SQLite "How to Use" examples to call out the suspend
context and recommend transaction-batching for hot inserts.
- Switch the ExpirationWorker example from Worker to CoroutineWorker
now that deleteExpiredEvents is suspend.
- Note in the FS README that the IEventStore API is suspend even
though the FS layer keeps a synchronous flock manager (the
withWriteLock helper is inline so suspend bodies pass through).
- Update the FsMaintenanceTest description to match the
coroutine-based concurrency test.
- Document the Mutex non-reentrancy footgun in
SQLiteConnectionPool's KDoc so module logic doesn't try to re-enter
the pool from inside useWriter.
https://claude.ai/code/session_016b5kSSbtDS3Ead6pN3Xqt5
androidx.sqlite SQLiteConnection is not thread-safe; SQLiteEventStore
shared a single lazy connection across all callers, so two coroutines
calling insertEvent() at the same time would race on BEGIN IMMEDIATE
and the modules' prepared statements, surfacing as
"cannot start a transaction within a transaction" or SQLITE_MISUSE.
Mirror Room's design: introduce SQLiteConnectionPool with one writer
connection guarded by a coroutine Mutex and N reader connections
handed out via a Channel-as-semaphore (file-backed DBs only; in-memory
DBs share the writer because each ":memory:" connection is a separate
DB). Convert IEventStore + SQLiteEventStore + EventStore + FsEventStore
+ LiveEventStore to suspend, route writes through useWriter and reads
through useReader. RelaySession now launches handleEvent / handleCount
on its scope. CLI Context helpers and StoreCommands.sweepExpired pick
up suspend.
Add ParallelInsertTest to lock the behaviour in: 8 coroutines × 200
inserts, parallel reads alongside writes, transaction batches across
coroutines, and a reopen smoke test all pass against a file-backed DB.
https://claude.ai/code/session_016b5kSSbtDS3Ead6pN3Xqt5
Implement a TLS 1.3 client state machine that drives the QUIC handshake using
only Quartz's existing crypto. No BouncyCastle dependency.
- HKDF-Expand and HKDF-Expand-Label upstreamed to Quartz's Hkdf class with
RFC 5869 + RFC 8448 test vectors covering them.
- :quic crypto stack: AEAD (AES-128-GCM via Quartz's AESGCM, ChaCha20-Poly1305
via Quartz's pure-Kotlin impl), header protection (AES-ECB via JCA single
block + ChaCha20 keystream), QUIC Initial-secret derivation matching
RFC 9001 Appendix A.1 bit-for-bit.
- TLS 1.3 transcript hash, key schedule (early/handshake/master + per-direction
client/server traffic secrets), Finished MAC.
- ClientHello + extension encoders carrying SNI, supported_versions=[TLS 1.3],
supported_groups=[X25519], signature_algorithms covering ECDSA/RSA-PSS/Ed25519,
X25519 key_share, psk_dhe_ke, ALPN=[h3], and the QUIC transport_parameters
extension.
- ServerHello + EncryptedExtensions + Certificate + CertificateVerify + Finished
parsers. The state machine handles the certificate path and the PSK-style
no-cert path; certificate validation is wired through a CertificateValidator
SPI (real impl lands in Phase L).
- Transport parameters codec covering all RFC 9000 §18.2 + RFC 9221 fields.
- QuicWriter/QuicReader buffer helpers shared across the rest of the stack.
Round-trip test: a minimal in-process TLS server built from the same primitives
drives a full ClientHello → ServerHello → EE → Finished → client Finished
exchange. Both sides reach handshake-complete and agree bit-for-bit on the
handshake & application traffic secrets. ALPN + transport parameters round-trip
through EncryptedExtensions cleanly.
https://claude.ai/code/session_01EC1tfXfap8k8GyKvrxkxZx
Three of the bugs that the SQLite review surfaced also live in the
filesystem-backed event store. Bringing both stores back to parity:
- NIP-01 lexical-id tiebreaker (FsSlots, FsEventStore): when two
replaceables / addressables share `createdAt`, the lexically smaller
id wins. The previous `existing.createdAt >= incoming.createdAt`
check rejected equal-timestamp inserts unconditionally — which
blocks the legitimate winner whenever it arrived second.
- delete(Filter()) safe-by-default (FsEventStore): an empty filter
used to enumerate every event and delete each one, so a stray
`delete(Filter())` would wipe the entire on-disk store. Now both
the single-filter and list-of-filters overloads short-circuit when
every filter is empty, matching the SQLiteEventStore contract.
- NIP-09 author check on tombstone install (FsEventStore,
FsTombstones): SQLite's `reject_deleted_events` trigger checks
`event_tags.pubkey_hash = NEW.pubkey_owner_hash`, so a stranger's
kind-5 with an `e`/`a` tag pointing at someone else's event must
not block them from re-publishing. The FS store used to install
the tombstone unconditionally and then read it back without an
author check.
- id tombstones still install (so they can fire when the deletion
arrives before its target), but `idTombstoneOwnerPubKey` is now
compared against the candidate event's owner pubkey at insert
time. GiftWrap parity preserved via FsIndexer.ownerPubKey, which
returns the recipient like the SQLite `pubkey_owner_hash`.
- addr tombstones are now only installed when
`addr.pubKeyHex == deletion.pubKey`, since the address itself
carries the owner identity.
Tests:
- FsSlotsTest: replaces "equal timestamp replaceable is rejected"
(which pinned the buggy behaviour) with two tests covering the
lexical-id tiebreaker in both insertion orders.
- FsParityTest: new same-`createdAt` tiebreaker tests for both
replaceable and addressable kinds, asserting FS and SQLite agree.
- FsDeletionTest:
- inverts "deletion by non-author does not cascade but still
installs id tombstone" — the legitimate owner must be able to
re-insert after the stranger's deletion.
- new test that a stranger's `a`-tag deletion does not block
the legitimate addressable owner from publishing a new version.
- FsEventStoreTest: new `delete with empty filter is safe` test
matching the SQLite-side contract added in batch A.
https://claude.ai/code/session_01X163Nr31vGkvAXoJ3JgMov
Batch B test-coverage gaps from the review:
- testTransactionRollsBackOnTriggerAbort: a multi-insert transaction
whose middle statement is rejected by `reject_deleted_events` rolls
back ALL inserts in the transaction, not just the failing one.
- testDeletionByThirdPartyDoesNothing: NIP-09 author check —
another user's deletion event must not remove the original.
- testKind5CanBeDeletedByAnotherKind5OfSameAuthor: pins the current
behavior that kind-5 events are not specially protected; a same-author
follow-up deletion can remove a previous one, and re-insertion of the
removed deletion is then blocked.
- testGiftWrapDeletionRequiresRecipient: NIP-59 GiftWraps key on the
recipient (p-tag), not the (encrypted) inner author. Sender and
unrelated third parties cannot delete; the recipient can.
- testVanishForDifferentRelayIsNoOp: a kind-62 vanish naming a
different relay must be stored but must not delete events on this
relay nor block new inserts (RightToVanishModule.shouldVanishFrom
contract).
- testFtsCleanedUpAfterReplaceableRotation: FTS rows are cleaned via
the AFTER DELETE trigger when an addressable is superseded — old
content must no longer match search.
- testVacuumAndAnalyseSmoke: VACUUM and ANALYZE run on a populated DB
without throwing and preserve existing rows.
- SqlSelectionBuilderTest: pins NotEquals(null) → IS NOT NULL,
empty IN → "1 = 0", equalsOrIn singleton/multiple paths.
https://claude.ai/code/session_01X163Nr31vGkvAXoJ3JgMov
- isExpired (#10): NIP-40 says an event is expired *once* `expiration`
is reached, and the SQL trigger uses `<= unixepoch()`. The Kotlin
pre-check used `<` (strict) so an event with `expiration == now`
passed the Kotlin check then failed in the trigger. Both layers now
use `<=`. Also applies to `isExpirationBefore` for consistency.
- transaction extension (#7): if the body throws *and* ROLLBACK also
throws, we now attach the rollback failure as a suppressed exception
instead of letting it mask the original cause. COMMIT is moved outside
the catch so a commit failure doesn't trigger a second ROLLBACK on
already-finalized transaction state.
- SeedModule.hasher (#8): the cache field is now a `kotlin.concurrent.
atomics.AtomicReference` (matches the pattern used in BleChunkAssembler
and BasicRelayClient) so the hasher publication is visible across
threads. The race itself is benign — the seed is stable, so two
concurrent computations produce identical hashers — but the prior
plain `var` had no visibility guarantee.
- delete(Filter()) (#12): documents the intentional asymmetry — `query`
on an empty filter returns everything, but `delete` on an empty
filter is a no-op (safe-by-default). New test pins the contract.
Tests:
- testInsertingEventExpiringExactlyNow: events with `expiration == now`
are rejected by both Kotlin and the trigger.
- testTransactionRollsBackOnException: a user transaction whose body
throws leaves the DB unchanged and still accepts new writes.
- testDeleteWithEmptyFilterIsSafe: empty-filter delete is a no-op.
https://claude.ai/code/session_01X163Nr31vGkvAXoJ3JgMov
Jackson's pretty-printer can emit \r\n on Windows depending on the
configured DefaultIndenter, so the previous one-sided normalization
still mismatched. Normalize both sides of the assert to LF.
Triple-quoted Kotlin strings inherit the source file's line endings,
so on Windows checkouts (CRLF) the expected JSON contains \r\n while
Jackson's pretty-printer always outputs \n. Normalize the expected
side with replace("\r\n", "\n") so the assertion is platform-neutral.
Surfaced by adding :quartz:jvmTest to the desktop CI matrix; previously
the bare 'test' lifecycle didn't resolve jvmTest in KMP modules so the
mismatch was never observed on Windows.
- NIP-09 (#2): a-tag and replaceable deletes now respect
`created_at <= deletion.created_at` so a stale deletion request
cannot remove a newer addressable that legitimately replaced it.
`+created_at` hint on the addressable path keeps the d_tag-selective
index in use.
- NIP-01 (#1): replaceable / addressable triggers now apply the
lexical-id tiebreaker — when two events share `created_at`, the
one with the lexicographically smaller id wins, matching the spec.
- Schema (#5): FullTextSearchModule.versionFinder probes FTS support
with a dummy table, but used to leave it behind. The first v1->v2
upgrade then failed because re-running create() would hit
"already exists". Now we drop the probe table immediately and
defensively clean up any stragglers.
- Schema (#6): onCreate / onUpgrade and the matching `setUserVersion`
are now wrapped in a single transaction so a partial migration
cannot leave the DB with mismatched user_version and schema.
- SQL DSL (#4): Condition.NotEquals(null) now produces `IS NOT NULL`
instead of `IS NULL`.
- Doc fix (#13): swapped vacuum/analyse comments now describe the
right command.
Tests: same-`created_at` tiebreaker for replaceables and addressables,
NIP-09 created_at window for a-tag deletes, schema drop+recreate
idempotency (covers the FTS dummy-table regression).
https://claude.ai/code/session_01X163Nr31vGkvAXoJ3JgMov
The PROPOSAL branch of `MlsGroup.decrypt` previously threw
`IllegalStateException("Standalone PrivateMessage proposals not yet
supported")`. That worked in the openmls-as-Whitenoise topology because
MDK emits SelfRemove as a PublicMessage — but any peer using the
AlwaysCiphertext wire-format policy (RFC 9420 §6.3.2) wraps standalone
proposals in PrivateMessage, and we'd hard-fail on first contact.
Implements the PROPOSAL branch symmetrically with the existing COMMIT
branch:
1. Decode the Proposal struct from PrivateMessageContent (no length
prefix), read signature<V>, drain zero-padding.
2. Restrict to SelfRemove (mirrors `receivePublicMessageProposal`'s
policy — the only standalone proposal type that needs to ride
outside a commit; widening is one-line if interop demands it).
3. Verify the FramedContentTBS signature with `wire_format =
PRIVATE_MESSAGE` against the sender's leaf signature_key.
4. Stage the proposal in `pendingProposals` together with the encoded
AuthenticatedContent (wire_format ‖ FramedContent ‖ signature) so
a subsequent inbound commit folding it in by ProposalRef can
resolve via the §5.2 hash. PrivateMessage AC bytes carry no
membership_tag — auth = signature only.
Adds a symmetric `encryptProposalAsPrivateMessage` helper (internal,
mirrors `encrypt` for application data) so tests can exercise the
inbound path with a real wire frame produced by the same code path
peers use.
2 new tests: round-trip Bob→Alice SelfRemove via Welcome flow with
independent MlsGroup instances, verifies the proposal lands in
Alice's pending pool with AC bytes captured; and rejection of a
non-SelfRemove standalone PrivateMessage proposal (PSK in the test).
Marmot test suite green; marmot-interop-headless 16/16.
Closes audit gap #1.
https://claude.ai/code/session_013VYkpz8P1mPh9Ejxy9anhJ
idx/tag/<name>/ used to bucket every tag value under a 16-hex Murmur
hash, so `ls idx/tag/p/` showed opaque hex even though pubkey values
are perfectly safe filenames. Now the directory name is:
- the raw tag value, when it's filesystem-safe across Linux/macOS/
Windows: 1..180 bytes, printable ASCII (0x21..0x7e), none of
`/ \ : * ? " < > |`, no leading dot or `_h_`, no trailing dot/space;
- otherwise `_h_<hashHex(murmur)>`. The `_h_` sentinel can never
collide with a raw value (raw values are forbidden from starting
with it), so both forms safely live in the same parent dir.
Common cases keep their raw form and become directly inspectable:
ls idx/tag/p/<your_pubkey>/ — every event that p-tagged you
ls idx/tag/e/<event_id>/ — replies / reactions to an event
ls idx/tag/t/nostr/ — every kind-1 with #nostr
ls idx/tag/k/30023/ — k-tag pointers to articles
ls idx/tag/g/drt3n/ — geohash mentions
Routes through the _h_ bucket: emojis, URLs (slashes), `a`-tags
(colons), free-form `alt` text, anything ≥ 180 bytes, anything that
breaks Windows reserved-name rules. The hash is still seed-salted
Murmur64 (parity with the previous bucket), so collisions are caught
by FilterMatcher post-filter just like before.
Writer (FsIndexer.pathsFor) and reader (FsQueryPlanner.firstTagKey)
both route through FsLayout.tagValueDirName, so they always agree.
Tests: 5 new in FsQueryTest covering raw ASCII tags landing under
named dirs, p-tag pubkeys keeping their raw form, emoji and URL tags
falling back to _h_, and round-trip queries hitting both buckets
correctly. 122 fs tests green.
No migration: existing stores must `amy store scrub` once after the
upgrade — old purely-hashed tag dirs become unreachable, and scrub
rebuilds idx/ from canonicals using the new naming.
Two DoS surfaces in the inbound path:
**#8 — unbounded forward-ratchet on PrivateMessage decrypt.** A
malicious sender (or any peer who can put bytes in the wire frame)
controls the `generation` field of an inbound PrivateMessage. The
SecretTree was happy to fast-forward the sender's application or
handshake ratchet by however many steps that field implied — every
step costing one HKDF-Expand. A single packet with `generation =
2^31` would have pinned a CPU at SHA-256 for minutes. The skipped-key
cache (`MAX_SKIPPED_KEYS = 1000`) bounds memory but NOT compute: it
just stops *caching* past the cap, the ratchet keeps walking.
Adds `MAX_RATCHET_STEPS_PER_CALL = 4096` and rejects any decrypt that
asks for a larger jump from the sender's current head, applied at
both `applicationKeyNonceForGeneration` and
`handshakeKeyNonceForGeneration`. 4096 leaves room for legitimate
catch-up (mobile waking from a long sleep) while bounding the
worst-case per-packet cost to ~4096 SHA-256 invocations.
**#11 — oversized PrivateMessage AAD allocation.** The underlying
[TlsReader] already caps every opaque<V> read at 1 MiB, so the
ciphertext field is bounded — but `authenticated_data` and
`encrypted_sender_data` were also allowed up to 1 MiB even though
both fields legitimately carry a few hundred bytes at most. A
pre-verification frame would force ~3 MiB of allocation per inbound
packet. Tightens both fields to 64 KiB at PrivateMessage decode
(below TlsReader's global cap) so the per-frame floor is predictable.
2 new tests: `secretTree_rejectsRatchetJumpsBeyondCap` exercises the
boundary (4096 OK, larger throws with a "jump too large" message)
and `privateMessage_rejectsOversizedAuthenticatedData` hand-builds
a frame with a 65 KiB AAD field and confirms rejection. Marmot
test suite green; interop 16/16.
Closes audit gaps #8 and #11.
https://claude.ai/code/session_013VYkpz8P1mPh9Ejxy9anhJ
Two RFC 9420 hardening gaps in the inbound path:
**#3 — UpdatePath silent decrypt failures.** When `processCommit` was
handed a commit whose UpdatePath either (a) had no node at our common
ancestor with the sender or (b) had no ciphertext encrypted to a node
in our copath resolution, the path-decrypt block silently fell through
and left `commit_secret = 0`. The downstream confirmation_tag check
caught it and rolled back, but the rollback surfaced as a generic
"ConfirmationTagMismatch" instead of pointing at the real cause —
a tree-shape mismatch between our view and the sender's. Now hard-fail
with a precise error naming the indices involved.
**#6 — parent_hash chain verification on Welcome.** `processWelcome`
was checking the GroupInfo signature and the GroupContext.tree_hash,
but neither gates a forged parent_hash inside a COMMIT-source leaf —
the tree_hash check only proves the ratchet_tree extension matches
the bytes the signer signed, not that the stored parent_hash values
are consistent with the tree shape. A peer that DOES validate
parent_hash (per RFC 9420 §7.9) would reject every commit produced
from such a tree, splitting the group on the next epoch.
Adds `verifyTreeParentHashesForJoin(tree)` as a static-tree
counterpart to `verifyParentHash` (which only handles the
post-UpdatePath dynamic case). For each COMMIT-source leaf, recompute
the parent_hash chain top-down on the leaf's filtered direct path and
compare to the stored value. Returns null on success or a human-
readable mismatch reason. Wired into `processWelcome` right after the
tree_hash check, before any capability or key-schedule work.
Also moves `encodeParentHashInput` into the companion object so the
static and instance variants share one TLS encoder, and adds an
`exportTreeBytes()` test accessor.
2 new tests: trivial single-member tree accepts; a tampered
COMMIT-source parent_hash on a 3-member tree is rejected with a
message naming the offending leaf. Quartz marmot tests green;
marmot-interop-headless 16/16. (Two unrelated NostrClient network
tests fail under the full :quartz:jvmTest run — pre-existing flake,
no MLS code touched.)
Closes audit gaps #3 and #6.
https://claude.ai/code/session_013VYkpz8P1mPh9Ejxy9anhJ
When a group installs a `required_capabilities` extension (every Marmot
group does, listing MarmotGroupData=0xF2EE, SelfRemove=0x000A, Basic
credential), every member's leaf MUST advertise those types in its own
[Capabilities]. We were validating none of that:
- `applyProposalAdd` checked version + ciphersuite but never matched the
new KP's capabilities against the group's required_capabilities. A
non-conformant member silently joined; the next commit that touched
their leaf got rejected by spec-conformant peers, splitting the group.
- `processWelcome` similarly never checked the joiner's own KP against
the group's required set, nor did it sweep existing members' leaves.
A misconfigured GroupInfo signer could invite us into an incoherent
group whose first commit we'd silently reject forever.
Adds two helpers in `MlsGroup.Companion`:
- `findRequiredCapabilities(extensions)`: decodes the §7.2 struct from
a GroupContext extension list, or returns null if absent.
- `requireCapabilitiesMeetRequirements(caps, req, who)`: throws with a
specific (extensions=, proposals=, credentials=) diff naming the
missing types — turns silent interop breaks into one debuggable line.
Wires the gate in two places:
- `applyProposalAdd` rejects the Add proposal if the new leaf falls
short of the group's required set.
- `processWelcome` rejects the join if either (a) our own KP doesn't
meet the group's requirements or (b) any existing member's leaf
doesn't — the latter catches a malformed GroupInfo at join time.
5 new tests: round-trip decoding, rejection on missing extension /
proposal, acceptance when caps are a superset, and an end-to-end
`addMember` rejection of a hand-crafted KP with SelfRemove stripped
(re-signed so the rejection is from the capability gate, not the
signature check). Quartz test suite green; marmot interop 16/16.
Closes audit gaps #4, #5, #10.
https://claude.ai/code/session_013VYkpz8P1mPh9Ejxy9anhJ
Three audit gaps in the Marmot MLS implementation, fixed together because
they share the same call sites:
1. **MIP-03 inbound authorization gate.** `enforceAuthorizedProposalSet`
only fired for *outbound* commits (it implicitly checked the local
member). A peer could send us a non-admin GroupContextExtensions
rename, a non-admin Remove, or an admin-emptying GCE and we would
silently apply it. `enforceAuthorizedProposalSet` now takes an explicit
`committerLeafIndex` (defaults to `myLeafIndex` for the local case)
and uses `isLeafAdmin(committerLeafIndex)` instead of `isLocalAdmin()`.
`processCommitInner` resolves the proposal list against our pending
pool, then runs both `enforceAuthorizedProposalSet` and
`enforceNoAdminDepletion` against the resolved set before applying.
External commits skip the check (sender has no leaf yet, and §12.4.3.2
already restricts the proposal list).
2. **RFC 9420 §5.3 psk_secret derivation.** The previous
`computePskSecret` HKDF-Extracted bare PSK values with the running
`pskSecret` as salt and ignored `psktype` / `psk_nonce` / index /
count entirely — incompatible with any spec-conformant peer. Per
§5.3 each step is now:
psk_extracted_i = HKDF.Extract(0, psk_i)
psk_input_i = ExpandWithLabel(psk_extracted_i, "derived psk",
PSKLabel(id_i, i, n), Nh)
psk_secret_i = HKDF.Extract(psk_secret_{i-1}, psk_input_i)
`buildPskLabel` encodes the full `PreSharedKeyID || index || count`
struct. Resumption PSKs (`psktype == 2`) reject loudly because
`Proposal.Psk` lacks `(usage, psk_group_id, psk_epoch)` — silently
encoding a broken PSKLabel would diverge from peers without warning.
3. **ProposalRef hash for locally-published proposals.** RFC 9420 §5.2
hashes the encoded AuthenticatedContent, not the bare Proposal.
`buildSelfRemoveProposalMessage` now stages the published proposal in
`pendingProposals` together with the AC bytes (wire_format ‖
FramedContent ‖ FramedContentAuthData), so a subsequent inbound
commit that folds it in by ProposalRef can resolve the hash. Bare-
proposal fallback in `processCommitInner` retained for legacy local
entries that pre-date the capture.
Tests: 7 new cases in `MarmotMipBehaviorTest` covering inbound non-admin
rejection (Add / GCE rename), admin-depletion rejection, AC bytes capture,
PSK empty/single/ordering/resumption-rejected paths. Full quartz JVM test
suite passes; marmot-interop-headless 16/16.
https://claude.ai/code/session_013VYkpz8P1mPh9Ejxy9anhJ
Users actually look at <data-dir>/events-store/ files (cat / jq / git
diff), so the CLI now writes them with the InliningTagArrayPrettyPrinter
that quartz already had configured but never invoked. Each event is
indented with 2 spaces, but every tag array stays on a single line —
nice trade-off between human-readable and not-too-tall:
{
"id": "...",
"pubkey": "...",
"created_at": 1700000000,
"kind": 1,
"tags": [
["t","nostr"],
["e","abc...a"],
["alt","quick brown fox"]
],
"content": "hi there",
"sig": "..."
}
Stored bytes are not the canonical NIP-01 form — but verification
re-canonicalises via EventHasher anyway, so format is purely a UX
choice. Compact stays the default for any caller that doesn't opt
in (Android keeps SQLite, generic FsEventStore embedders keep
compact).
- JacksonMapper.toJsonPretty(event): new entry point that uses
writerWithDefaultPrettyPrinter() with the existing inlining printer.
- FsEventStore now takes an `eventToJson: (Event) -> String` callback,
default = Event::toJson (compact). Used in insert.
- Context wires in JacksonMapper::toJsonPretty.
2 new tests in FsEventToJsonTest pin both formats (compact stays
single-line; pretty round-trips). 117 fs tests green.
`profileOf(pk)` and `relaysOf(pk)` are the two hottest read patterns
in any Nostr client, both expressed as `Filter(authors=[pk],
kinds=[0|10002])`. Before this change every such query walked
`idx/kind/<k>/`, sorted the listing, and probed candidates — O(N)
in the kind's total event count even though we always wanted exactly
one event.
The planner now intercepts before the directory walk: when a filter
pins us to replaceable / addressable kinds with `authors` (and
`d`-tag for addressables), we read the slot file directly. One
`Files.exists()` + one `readString()` per (kind, author[, d]) triple,
no listing, no sort.
Falls through to the generic walk for anything that doesn't fit
(non-uniqueness kinds, missing d-tag, search clause, etc.). Two
new tests in FsSlotsTest assert the shortcut serves correctly
even when `idx/` has been externally wiped — proving the
shortcut isn't accidentally relying on the index.
115 fs tests green.
Audit cleanup — three correctness-adjacent fixes:
#13 — FsLayout.readOrCreateSeed was TOCTOU-racy for two processes
opening the same fresh directory. Both could pass `!exists()`, both
write their own random bytes, both ATOMIC_MOVE into place. Loser's
seed is forgotten but they already hashed with it, so their future
index entries would be unreachable. Switch to CREATE_NEW: exactly one
process creates the file, the other catches FileAlreadyExistsException
and falls through to read the winner's bytes.
#9 — `catch (_: Throwable)` / `catch (_: Exception)` in FsEventStore,
FsSlots, FsTombstones narrowed to `catch (_: IOException)` plus
`catch (_: JacksonException)` for JSON-parse paths. The old catches
swallowed CancellationException (hides coroutine cancellation),
OutOfMemoryError, and ThreadDeath. Now only the errors we actually
expect on a racing read are suppressed.
#1 — FsLockManager replaced synchronized+ThreadLocal+depth-counter
with a ReentrantLock. Same semantics (cross-process flock + in-process
reentry), cleaner code: the ReentrantLock handles reentry natively
via holdCount, no ThreadLocal, no manual depth bookkeeping. Release
is guarded by `holdCount == 1` on exit. Narrow catches in the
cleanup path to IOException too.
113 fs tests green across all suites.
Two query-planner hotspots called out by the audit:
#2 — mergeDesc used to materialise every driver's full stream into
one ArrayList before sorting, so `limit = 10` against a million-event
store would still load and sort the full million before emitting
anything. Replace with a lazy k-way merge on a max-heap of per-stream
heads. Memory is now O(num streams) instead of O(sum of stream
sizes), and `limit` short-circuits naturally after the first N pops.
walkDir also sorts filenames (cheap strings) instead of Candidate
objects (bigger) — same DESC order because our `<padded_ts>-<id>`
convention makes lex-reverse == chronological DESC.
#3 — ftsDriver used to load every search token's full listing into
HashMap<HexKey, Long> before intersecting, so `search = "the bitcoin"`
against a popular token would spike memory proportional to that
token's size. Switch to smallest-first: count each token dir, drive
the walk with the smallest, and confirm each candidate in the others
via a single `Files.exists()` per (candidate, token). Works because
every FTS hardlink for an event shares the same `<padded_ts>-<id>`
filename, so stat-check is a direct lookup. Memory is now
O(smallest_token_size) — no HashMap for the big tokens.
113 fs tests green, FsSearchTest verifies the ordering, AND semantics,
and limit behaviour that these touch.
The clock injection existed only for NIP-40 expiration tests. A public
constructor parameter that ~every caller ignores is API clutter, so
move it to a subclass seam:
- FsEventStore is now `open class`; no more `clock: () -> Long` param.
- `protected open fun now(): Long = TimeUtils.now()` is the override
point. Production always takes the default; tests subclass.
- FsExpirationTest gains a private `ClockedStore(root, source)` that
overrides `now()`. Semantics unchanged, all 113 fs tests still green.
Public `FsEventStore` ctor is now `(root, indexingStrategy, relay)` —
no behavioural-drift surface that callers have to learn.
Sibling to the SQLite store's README. Covers:
- what the store is and why (filesystem primitives = invariants:
directory-entry uniqueness IS the UNIQUE constraint, rename(2) IS
the atomic commit, hardlink refcount IS the FK cascade, flock(2)
IS the writer serialisation),
- feature-parity table vs SQLite (NIP-01 replaceable / addressable,
NIP-09 deletion + tombstones, NIP-40 expiration, NIP-45 count,
NIP-50 search, NIP-62 vanish, NIP-91 multi-tag AND),
- on-disk layout with every directory tree,
- filename conventions (sharding, padded-ts entry names, sha256 vs
Murmur, mtime),
- internal architecture (FsLayout / FsLockManager / FsIndexer /
FsSlots / FsTombstones / FsQueryPlanner / FsSearchTokenizer),
- usage examples for insert / query / count / delete / transaction /
expiration sweep / scrub / compact / close,
- failure-mode table (external edits, crashes, multi-process,
full disk) and how the store converges,
- pointer to the 113-test suite under jvmTest and the design plan
documents under cli/plans/.
Two bugs that conspired to break interop test 15 once the test 14 OOM
was fixed:
1. **No receive path for standalone PublicMessage proposals.**
wn/openmls publishes a non-admin's `SelfRemove` as a kind:445 carrying
a `PublicMessage(content_type=PROPOSAL)` envelope and waits for an
admin to fold it into the next commit. Quartz's `MarmotInboundProcessor`
answered every such event with `Error("Standalone proposals not yet
supported")` and dropped it. The admin's subsequent commit then
failed with `Commit references unknown proposal (ref not found in
pending proposals)` because nobody had staged the SelfRemove.
Add `MlsGroup.receivePublicMessageProposal(pubMsg)` that:
- rejects mismatched epoch / group_id / sender,
- reconstructs the FramedContentTBS exactly as the proposer did and
verifies the leaf signature,
- verifies the membership_tag against the current epoch's
membership_key (same threat model as inbound PublicMessage commits),
- decodes the inner Proposal (only `SelfRemove` is accepted today —
other types come bundled in commits' `proposals` lists),
- stages the proposal in `pendingProposals` so a later commit can
resolve its `ProposalRef`.
`processPublicMessage` now routes `ContentType.PROPOSAL` through
that helper and returns a new `GroupEventResult.ProposalStaged`
variant (also surfaced as `MarmotIngestResult.ProposalStaged`),
replacing the old hard-error path.
2. **Wrong `ProposalRef` hash input.** RFC 9420 §5.2 specifies that a
`ProposalRef` hashes the **encoded `AuthenticatedContent`** that
delivered the proposal, not the bare `Proposal` struct. Quartz was
hashing `proposal.toTlsBytes()` only — fine for our local-only
flows where commits inline rather than reference our own pending
proposals, but fatal once we needed to match wn's reference to an
inbound proposal.
Extend `PendingProposal` with an optional `authenticatedContentBytes`
field. The standalone-proposal receive path captures the full
`wire_format || FramedContent || FramedContentAuthData` envelope at
stage time. The reference-resolution code in `processCommitInner`
prefers those bytes when present and falls back to the bare-proposal
hash for locally-proposed entries (which never get referenced
today).
Marmot interop score: 14/16 → 15/16 (test 9 — amy's kind:7 reaction
triggers a `SecretReuseError` on B's wn — is unrelated to this path
and remains for follow-up).
https://claude.ai/code/session_013VYkpz8P1mPh9Ejxy9anhJ
Replace the inline MessageDigest + hand-rolled hex converter with
quartz.utils.sha256.sha256 + quartz.utils.Hex.encode. Same output
(deterministic algorithm), but we stop duplicating primitives that
already exist in the codebase. One fewer JCA dependency, one fewer
hex-conversion implementation to audit.
All 113 fs tests still green, including FsParityTest vs the SQLite
reference — the d-tag slot paths are byte-identical to before.
When wn admin-removes A in interop test 14, A processed the proposal
locally — `tree.removeLeaf(A.leafIndex)` blanked her leaf and shrank
`tree.leafCount` past `myLeafIndex` — and then immediately walked into
BinaryTree.directPath(myLeafIndex, tree.leafCount)
at MlsGroup.processCommitInner. With `myLeafIndex >= leafCount`, the
left-balanced parent walk had no valid stopping point: each recursion
step doubled the candidate parent index, integer-overflowed past 2^31,
and either returned garbage or kept appending to the result list until
the JVM OOM'd. The OOM bubbled up as a `runBlocking` failure that
rolled back the whole commit — so A also stayed locally convinced she
was still a member, and the test timed out waiting for `not_member`.
Three layered fixes so the failure mode can't recur:
* `BinaryTree.parent`/`directPath`/`copath` now `require` an in-range
input. The root and any node ≥ nodeCount used to silently loop or
return garbage; now they throw `IllegalArgumentException` immediately.
This is defense-in-depth — a future caller passing an invalid index
gets a stack trace at the boundary instead of an OOM five frames
deep.
* `MlsGroup.processCommitInner` short-circuits the path-decrypt + epoch
advancement when the proposals in the commit just removed *us*. We
preserve the proposal-side tree mutations so the caller (and the
outer state machine) can observe that we're out, clear pending
proposals + sent keys, and return. Without this short-circuit the
function would derive a bogus all-zero `commit_secret`, fail
`confirmation_tag` verification, throw, and roll the snapshot back —
leaving us still convinced we were a live member.
* `MlsGroupManager.isMember` and a new `MlsGroup.isLocalMember()`
helper now return false once our leaf is null or past `leafCount`.
The post-Remove group entry still lives in `groups` so callers can
inspect the final tree, but every cli command (`group show`,
`message send`, etc.) sees `not_member` and returns the right error.
Also add a comprehensive `BinaryTreeTest` covering non-power-of-2
trees (3, 5, …, 32 leaves) and the boundary cases (root has no
parent, leafIndex ≥ leafCount must throw). The pre-existing tests
only exercised the 4-leaf example from RFC 9420 Appendix C; nothing
hit the `parentInRange` branch, which is exactly where the OOM lived.
Test 14's bash polling needed a small companion fix: it captured
amy's stderr through `2>&1` and fed the result to `jq`, but the
captured stream is interleaved with quartz `Log.d(…)` debug lines,
so the JSON parse always failed. Switch to a `grep` for the
`"error":"not_member"` literal — that signature only appears in
the JSON payload and survives the debug noise. Also tee the
captured output into `$LOG_FILE` so post-mortem logs include each
polling iteration's `[cli] ingest …` traces.
Marmot interop score: 11/16 → 14/16 (tests 9, 15 are unrelated MLS
issues — see follow-up).
https://claude.ai/code/session_013VYkpz8P1mPh9Ejxy9anhJ
Drives both EventStore (SQLite reference) and FsEventStore with
identical event streams and asserts query() result sets match. SQLite
throws on blocked inserts (NIP-09 tombstones, NIP-62 vanish, NIP-40
expiration) while the FS store silently skips — both are observable
"event not persisted", so insertBoth() catches SQLite throws and
parity is judged on the post-insert state.
Coverage: id lookup, kind+author, since/until/limit, single-letter
tag query (OR within key), replaceable winner, replaceable older
rejected, addressable d-tag dedup, deletion by id (incl. block-
reinsert), deletion by address (incl. cutoff semantics), expiration
sweep, NIP-50 single-token search (sticking to ASCII where SQLite's
unicode61 tokenizer agrees with our port), count, multi-filter
union, delete by filter, mixed kitchen-sink scenario.
113 fs tests now green (97 unit + 16 parity).
- FsLockManager: cross-process exclusive flock(.lock) with per-thread
re-entry. withWriteLock { body } acquires once on a fresh thread
and reuses on nested calls — so transaction { insert(); insert() }
doesn't self-deadlock.
- FsEventStore: insert / delete / delete(filter) / delete(filters) /
delete(id) / deleteExpiredEvents / transaction now run under
withWriteLock. The `*Locked` helpers expose the lock-free body for
re-entrant callers (transaction body, vanish/deletion cascades,
expiration sweep). close() releases the lock channel.
- scrub(): wipes idx/ and rebuilds every entry from the canonical
events. Slots, tombstones and seed are left alone — slots can pin
data the canonical pass doesn't see, and tombstone removal is a
deliberate "un-forget" per the design plan.
- compact(): drops dangling idx/ entries whose canonical no longer
exists. Cheap — only touches idx/, never opens a JSON.
Tests: 11 new in FsMaintenanceTest covering lock file presence,
transaction commit / propagated exception with kept-prior-events
semantics, re-entrant lock from inside a transaction, scrub
rebuilding idx + FTS after a manual wipe, scrub preserving the
replaceable slot, compact dropping dangling and leaving valid alone,
close idempotence + reopen, and two-thread concurrent insert
serialisation. 97 fs tests green.
Adds idx/fts/<token>/<ts>-<id> hardlinks and an FTS-driven query path.
- FsSearchTokenizer: lowercase + Unicode-aware split on non letter-or-
digit, matching SQLite FTS5's unicode61 default closely enough that
the same call indexes content and parses queries (any drift cancels).
Tokens capped at 100 chars to keep filenames under FS limits.
- FsLayout: idxFts + ftsEntry / ftsTokenDir helpers; skeleton dir.
- FsIndexer.pathsFor: when event implements SearchableEvent, emits one
hardlink per unique tokenised word — so insert/delete maintenance
rides the existing link/unlink path. Eviction (replaceable swap),
NIP-09 cascade and NIP-62 vanish all clean up FTS for free.
- FsQueryPlanner: when filter.search is non-blank, drives by FTS.
Tokenises the query, walks each idx/fts/<token>/ listing into a
HashMap<id, ts>, and intersects smallest-first (AND across tokens —
matching SQLite FTS5 default MATCH semantics). Output sorted by
createdAt DESC. Other Filter fields (kinds, authors, tags, since /
until) still apply via Filter.match post-filter.
Tests: 16 new in FsSearchTest covering tokenizer (whitespace, case,
unicode, punctuation, empty), index maintenance (entries created,
non-searchable kinds skipped, delete unlinks), and query semantics
(single token, AND of tokens, ordering, limit, kind/author compose,
no-match, blank string ignored, reopen). 86 fs tests green.
Claude