Sibling to the SQLite store's README. Covers:
- what the store is and why (filesystem primitives = invariants:
directory-entry uniqueness IS the UNIQUE constraint, rename(2) IS
the atomic commit, hardlink refcount IS the FK cascade, flock(2)
IS the writer serialisation),
- feature-parity table vs SQLite (NIP-01 replaceable / addressable,
NIP-09 deletion + tombstones, NIP-40 expiration, NIP-45 count,
NIP-50 search, NIP-62 vanish, NIP-91 multi-tag AND),
- on-disk layout with every directory tree,
- filename conventions (sharding, padded-ts entry names, sha256 vs
Murmur, mtime),
- internal architecture (FsLayout / FsLockManager / FsIndexer /
FsSlots / FsTombstones / FsQueryPlanner / FsSearchTokenizer),
- usage examples for insert / query / count / delete / transaction /
expiration sweep / scrub / compact / close,
- failure-mode table (external edits, crashes, multi-process,
full disk) and how the store converges,
- pointer to the 113-test suite under jvmTest and the design plan
documents under cli/plans/.
Replace the inline MessageDigest + hand-rolled hex converter with
quartz.utils.sha256.sha256 + quartz.utils.Hex.encode. Same output
(deterministic algorithm), but we stop duplicating primitives that
already exist in the codebase. One fewer JCA dependency, one fewer
hex-conversion implementation to audit.
All 113 fs tests still green, including FsParityTest vs the SQLite
reference — the d-tag slot paths are byte-identical to before.
Drives both EventStore (SQLite reference) and FsEventStore with
identical event streams and asserts query() result sets match. SQLite
throws on blocked inserts (NIP-09 tombstones, NIP-62 vanish, NIP-40
expiration) while the FS store silently skips — both are observable
"event not persisted", so insertBoth() catches SQLite throws and
parity is judged on the post-insert state.
Coverage: id lookup, kind+author, since/until/limit, single-letter
tag query (OR within key), replaceable winner, replaceable older
rejected, addressable d-tag dedup, deletion by id (incl. block-
reinsert), deletion by address (incl. cutoff semantics), expiration
sweep, NIP-50 single-token search (sticking to ASCII where SQLite's
unicode61 tokenizer agrees with our port), count, multi-filter
union, delete by filter, mixed kitchen-sink scenario.
113 fs tests now green (97 unit + 16 parity).
- FsLockManager: cross-process exclusive flock(.lock) with per-thread
re-entry. withWriteLock { body } acquires once on a fresh thread
and reuses on nested calls — so transaction { insert(); insert() }
doesn't self-deadlock.
- FsEventStore: insert / delete / delete(filter) / delete(filters) /
delete(id) / deleteExpiredEvents / transaction now run under
withWriteLock. The `*Locked` helpers expose the lock-free body for
re-entrant callers (transaction body, vanish/deletion cascades,
expiration sweep). close() releases the lock channel.
- scrub(): wipes idx/ and rebuilds every entry from the canonical
events. Slots, tombstones and seed are left alone — slots can pin
data the canonical pass doesn't see, and tombstone removal is a
deliberate "un-forget" per the design plan.
- compact(): drops dangling idx/ entries whose canonical no longer
exists. Cheap — only touches idx/, never opens a JSON.
Tests: 11 new in FsMaintenanceTest covering lock file presence,
transaction commit / propagated exception with kept-prior-events
semantics, re-entrant lock from inside a transaction, scrub
rebuilding idx + FTS after a manual wipe, scrub preserving the
replaceable slot, compact dropping dangling and leaving valid alone,
close idempotence + reopen, and two-thread concurrent insert
serialisation. 97 fs tests green.
Adds idx/fts/<token>/<ts>-<id> hardlinks and an FTS-driven query path.
- FsSearchTokenizer: lowercase + Unicode-aware split on non letter-or-
digit, matching SQLite FTS5's unicode61 default closely enough that
the same call indexes content and parses queries (any drift cancels).
Tokens capped at 100 chars to keep filenames under FS limits.
- FsLayout: idxFts + ftsEntry / ftsTokenDir helpers; skeleton dir.
- FsIndexer.pathsFor: when event implements SearchableEvent, emits one
hardlink per unique tokenised word — so insert/delete maintenance
rides the existing link/unlink path. Eviction (replaceable swap),
NIP-09 cascade and NIP-62 vanish all clean up FTS for free.
- FsQueryPlanner: when filter.search is non-blank, drives by FTS.
Tokenises the query, walks each idx/fts/<token>/ listing into a
HashMap<id, ts>, and intersects smallest-first (AND across tokens —
matching SQLite FTS5 default MATCH semantics). Output sorted by
createdAt DESC. Other Filter fields (kinds, authors, tags, since /
until) still apply via Filter.match post-filter.
Tests: 16 new in FsSearchTest covering tokenizer (whitespace, case,
unicode, punctuation, empty), index maintenance (entries created,
non-searchable kinds skipped, delete unlinks), and query semantics
(single token, AND of tokens, ordering, limit, kind/author compose,
no-match, blank string ignored, reopen). 86 fs tests green.
Adds tombstones/vanish/<owner_hex>.json hardlinks (one per owner,
strongest cutoff wins) plus the cascade and block-future-insert
semantics from SQLite's RightToVanishModule.
- FsLayout: vanishTombstonePath helper + tombstones/vanish/ skeleton.
- FsTombstones: vanishCutoff / installVanish / clearVanish — install
uses atomic rename-with-REPLACE_EXISTING and only proceeds when the
new kind-62's createdAt is strictly greater than any existing tomb.
- FsEventStore:
- constructor now takes optional relay: NormalizedRelayUrl? to scope
NIP-62 cascades (matches SQLiteEventStore's relay arg).
- isBlockedByTombstone now also rejects events whose owner has an
active vanish with createdAt >= event.createdAt — owner is the
recipient for GiftWrap, matching pubkey_owner_hash semantics.
- processVanish() runs after canonical write: skips if !shouldVanish-
From(relay), installs the tombstone, then walks idx/owner/<hex>/
and deletes every event with ts < vanish.createdAt. The vanish
event itself survives (its ts equals the cutoff).
Tests: 11 new in FsVanishTest — relay-scoped cascade, different-relay
no-op, vanishFromEverywhere, block re-insert + newer-passes, equal
ts blocked, other authors unaffected, stronger cutoff wins, weaker
ignored, tombstone shares inode with kind-62 canonical, kind-62 stays
queryable. 70 fs tests green.
Adds idx/expires_at/<padded_exp>-<id> hardlinks for events with an
expiration tag, an injectable clock so tests can drive time
deterministically, and the deleteExpiredEvents() sweep.
- FsLayout: idxExpiresAt + expirationEntry path helper.
- FsIndexer.pathsFor: emits the expiration entry whenever
event.expiration() > 0, so insert / delete maintain it alongside
the kind / author / owner / tag indexes.
- FsEventStore: pre-insert guard rejects events with exp <= now
(SQLite parity: trigger uses inclusive <=). Constructor takes a
clock function defaulting to TimeUtils.now(). deleteExpiredEvents
walks idx/expires_at, parses filenames, and deletes anything with
exp < now (strict <, matching SQLite's sweep query).
Tests: 8 new in FsExpirationTest — future expiration accepted +
indexed, already-expired-on-insert rejected, exp==now rejected on
insert but kept by sweep, non-positive exp ignored, sweep removes
canonical + index entries, plain events untouched. 59 fs tests green.
Tombstone files under tombstones/id/<id>.json and tombstones/addr/
<kind>/<pubkey>/<sha256(d)>.json, each a hardlink to the kind-5
event that authored the deletion. One source of truth: the tombstone
IS the deletion event, just indexed by target.
- FsTombstones — installs id tombstones unconditionally, installs
addr tombstones with strongest-cutoff-wins semantics (later kind-5
replaces earlier via atomic rename), and exposes hasIdTombstone /
addrTombstoneCutoff for pre-insert checks.
- FsEventStore.insert — pre-insert guard: id tombstone always blocks;
addr tombstone blocks when event.createdAt <= tomb.createdAt, matching
SQLite's reject_deleted_events trigger. Kind-5 inserts trigger a
cascade: for each e-tag target owned by the deletion author, unlink
indexes + slot + canonical; for each a-tag (same pubkey), evict the
slot winner if its createdAt <= deletion.createdAt. Tombstones are
installed for every target regardless, so future re-inserts are
blocked.
Tests: 11 new in FsDeletionTest — delete-by-id, block-reinsert-by-id,
non-author-deletion still installs tombstone but no cascade, cascade
addressable slot, newer-at-deleted-address passes, older blocked,
equal-timestamp blocked, later kind-5 raises cutoff, earlier kind-5
does not lower, deletion event stays queryable, tombstone shares
inode with kind-5 canonical. 51 fs tests green.
Brings the file-backed store up to parity with SQLite's ReplaceableModule
and AddressableModule. A slot is a single hardlink that encodes the
UNIQUE(kind, pubkey[, d-tag]) constraint directly in the directory
layout:
replaceable/<kind>/<pubkey>.json (kinds 0, 3, 10000-19999)
addressable/<kind>/<pubkey>/<sha256(dTag)>.json (kinds 30000-39999)
FsSlots handles the full lifecycle: pre-insert guard (reject if newer or
equal exists), atomic rename-with-REPLACE_EXISTING install, and eviction
of the old winner's canonical + index hardlinks. Because the slot is a
hardlink the event data survives external canonical deletion, matching
the "files come and go" contract in the design plan.
delete(id) also clears the slot when the deleted event is the current
winner, so no orphan slot files linger.
Tests: 14 new in FsSlotsTest — newer wins / older rejected / equal
rejected / eviction unlinks old indexes / empty d-tag / canonical-
deletion survives via hardlink / delete-clears-slot / non-replaceable
events never touch the slot dirs. 40 fs tests pass.
Hardlink indexes under idx/ and a minimal query planner bring the
file-backed store up to parity with SQLite on filtered lookups.
- FsLayout — path helpers, .seed file (8 random bytes, salts all hashes),
entry filename format <zero-padded ts>-<id>.
- FsIndexer — on insert creates hardlinks at idx/kind/<k>/, idx/author/
<pk>/, idx/owner/<owner_hex>/, idx/tag/<name>/<hash_hex>/. Owner hash
matches SQLite's pubkey_owner_hash (recipient for GiftWrap). Honours
DefaultIndexingStrategy: single-letter tag names only. On delete
unlinks every known path so the inode can be reclaimed.
- FsQueryPlanner — picks a driver (ids / first tag / kinds / authors /
all kinds) and yields candidates sorted by createdAt DESC. Final
predicate check runs through Filter.match so any driver is
correctness-safe.
- FsEventStore — sets mtime to event.createdAt on write, runs queries
through the planner with post-filter, applies limit, dedupes by id
across multi-filter unions, and unlinks indexes on delete.
Tests: 16 new in FsQueryTest covering order, limit, author / kind /
tag drivers, tag OR within a key, tagsAll AND across keys, non-
single-letter-tag behaviour, since/until, count, index hardlink
maintenance, and reopen persistence. All 26 fs tests green.
First slice of the file-backed IEventStore planned in
cli/plans/2026-04-24-file-event-store-*. Each event is stored as
events/<aa>/<bb>/<id>.json with atomic tmp+rename writes. Ephemeral
kinds are dropped. Duplicate inserts are no-ops (id-level uniqueness).
Staging leftovers are swept on open.
Only id-based query and delete are wired; the query planner, indexes,
replaceable/addressable slots, tombstones, vanish, expiration sweep,
FTS, and transactions land in later steps.
Tests: 10 new tests under quartz jvmTest, all passing.
The default Event.notifies body was hand-rolling a tag scan for lowercase
`p`, duplicating the tag-shape knowledge that already lives in PTag.
Give PTag ownership of the check:
PTag.isNotifying(tags, userHex) // iterates tags and uses PTag.isTagged
Event.notifies now delegates to it. Kinds that override (CommentEvent,
WakeUpEvent) still work — they compose or replace the default as before.
https://claude.ai/code/session_01GQDJxiHPogdzCNhUBN7Pjc
The notification pipeline previously hard-coded a lowercase-`p`-tag match
in two places (the observer predicate and consumeFromCache via
taggedUserIds). That's correct for most kinds but wrong for two:
- NIP-22 CommentEvent: a comment several levels deep only tags the root
author via uppercase `P` (RootAuthorTag). Pure lowercase-p routing
missed "someone replied deep in your thread" notifications.
- Experimental WakeUpEvent (kind 23903): its `p` tags are the authors of
the subject events it references — Bob reacting to Alice's post yields
a WakeUpEvent with p=Bob, even though Alice's device is the one that
needs to wake up. Transport-layer routing (push/relay subscription)
already delivered the event to the right device, so the in-event
routing has to be permissive.
Introduce `open fun Event.notifies(userHex: HexKey): Boolean` with a
lowercase-`p` default that covers NIP-01/04/17/25/28/34/57/68/71/84/AC/
chess/wiki/long-form/poll mentions. Each kind with distinct semantics
overrides:
- CommentEvent.notifies: super.notifies(u) || rootAuthorKeys().contains(u)
— picks up uppercase P root-author routing on top of lowercase p.
- WakeUpEvent.notifies: true — every logged-in account is a valid wake
target once the event has reached LocalCache on this device.
NotificationDispatcher's observer predicate and EventNotificationConsumer.
consumeFromCache both now ask `event.notifies(accountHex)` instead of
extracting taggedUserIds themselves. The zap path's redundant
isTaggedUser re-check is gone too (it was duplicating what the outer
routing already enforced).
https://claude.ai/code/session_01GQDJxiHPogdzCNhUBN7Pjc
A previous commit misread the spec: p-tags on a WakeUp identify the
AUTHORS of the referenced events (the people whose events are the
subject of the wake-up), not the recipients. The consumer's existing
npub-in-p-tag match is therefore correct — "is this logged-in account
the author of a referenced event?".
- Revert WakeUpEvent.build() back to notify(about.toPTag()) and replace
the comment with a spec-accurate one.
- In wakeUpFor, source author pubkeys from event.authorKeys() (p-tags,
canonical) first, merge in the e-tag author hints, and fall back to
the WakeUp signer only when both are empty. Bound by MAX_WAKEUP_REFS.
computeReplyTo and the referenced-event fetch path remain untouched —
those fixes are orthogonal to the p-tag semantic.
The WakeUp notification path had three latent bugs that caused pushed
WakeUps to silently not deliver anything to the user:
- WakeUpEvent.build() tagged the about event's AUTHOR as the recipient
(via EventHintBundle.toPTag()), so a WakeUp about a zap from Alice to
Bob would p-tag Alice. The consumer matches recipients by p-tag, so
Bob — the intended recipient — was filtered out. Forward the about
event's audience (its own p-tags) instead.
- wakeUpFor() passed the WakeUp's own note to EventFinderQueryState. The
finder's filterMissingEvents only queries for the note itself (already
in cache) or its replyTo (empty because computeReplyTo had no WakeUp
case). The referenced events were never REQ'd on relays. Link the
referenced events via computeReplyTo and subscribe the finder on each
referenced note directly so filterMissingEvents picks them up.
- UserFinder was subscribed against the WakeUp's own pubKey (typically a
push bot), not the author of the event the notification is about.
Pull author pubkeys from the e-tag author hint, falling back to the
WakeUp signer only when the hint is absent.
Also: bound e-tag count per WakeUp to 16 to prevent subscription
flooding, log the 30s timeout, extract WAKEUP_WINDOW_MS constant, drop
the now-unused Note parameter from wakeUpFor.
The existing testReaddAfterRemove_RejoinerCanEncryptAndDecrypt in
MlsGroupLifecycleTest exercises the MLS layer (two raw MlsGroup
instances, admin kicks peer, fresh KP, rejoin). Nothing covered the
same flow one level up at the MlsGroupManager layer, where the
persistent state store, retained-epoch wipe, and Welcome routing for
the same nostrGroupId actually live.
Adds testLeaveAndRejoin_SameGroupIdEndToEnd: two MlsGroupManager
instances (Alice + Bob) with separate InMemoryGroupStateStores go
through the full cycle — Alice creates with a MarmotGroupData
extension, Bob joins via Welcome, bidirectional message round-trip,
Bob calls manager.leaveGroup (asserts store + retained epochs both
wiped), Alice removes Bob's leaf, Bob's second KeyPackage is admin-
added, Bob processes the second Welcome under the same nostrGroupId,
and bidirectional round-trip works again at the new epoch.
Closes the "MarmotManager.leaveGroup + rejoin" and "persistent-store
leave + fresh Welcome for same nostrGroupId" gaps from the review.
Documents inline why Alice drives the eviction commit directly: the
standalone-proposal ingestion pipeline (MarmotInboundProcessor) is a
separate, still-unimplemented concern.
https://claude.ai/code/session_01Unm6uLHGLj9UcBY7hWfJVW
KeyPackageFetcher.fetchKeyPackage used client.fetchFirst, which closes
the subscription on the first event a relay sends. For kind:443
(KeyPackage) that's wrong after a rotation: the publisher does not
replace the prior event (kind:443 is not addressable), so relays may
still hold the old KP and whichever one wins the race gets returned.
Switch to fetchAll — drain every matching event until EOSE, then pick
the one with the highest created_at. Keeps freshly-rotated bundles
reachable and matches whitenoise/mdk semantics.
Unskips interop test 16 (wn rotates, amy discovers). The inverse path
(test 13) already worked because `wn keys check` looks up via the
addressable kind:10051 index instead of kind:443 directly.
https://claude.ai/code/session_01Unm6uLHGLj9UcBY7hWfJVW
Gives users a last-resort "nuclear" option when Marmot local state is
corrupted or otherwise unrecoverable — wipes every MLS group, retained
epoch secret, persisted KeyPackage bundle, subscription and in-memory
chatroom for the current account. Does not broadcast leave/SelfRemove
commits, since a graceful teardown may be impossible in exactly the
scenarios where users reach for this option. The KeyPackage is
republished lazily on the next sync so the account stays reachable.
Adds clearAllState() helpers on MlsGroupManager and
KeyPackageRotationManager, a resetAllState() orchestrator on
MarmotManager, an Account.resetMarmotState() entry point, and a
destructive row + confirm dialog in the AllSettingsScreen Danger Zone
that mirrors the existing Request-to-Vanish pattern.
https://claude.ai/code/session_01Unm6uLHGLj9UcBY7hWfJVW
`MarmotInboundProcessor.processGroupEvent` added every event id to its
dedup set regardless of result, including `UndecryptableOuterLayer`. The
handler buffers those events for a post-commit retry, but the replay hit
the Duplicate early-return and skipped MLS decryption — so future-epoch
messages and group-metadata commits that arrived before their epoch's
commit were silently dropped until an app restart re-fetched them.
This is interop test 12 (offline catch-up): messages 6-8 and the rename
commit only appeared after restarting Amethyst.
Skip the dedup write on `UndecryptableOuterLayer` so retries actually
re-attempt decrypt once the epoch advances. Memory is still bounded by
the handler's per-group pending buffer (64 events, FIFO-evicted).
The `tryDecryptWithRetainedEpoch` fallback in `MlsGroupManager.decrypt`
exists precisely to cover the interop harness's Test 12 scenario —
application messages encrypted under epoch N arriving at a receiver
that has already processed a commit advancing to N+1. The primary
path throws "Message epoch X doesn't match current epoch Y", then the
manager iterates the retained epoch window and tries each.
Two bugs kept the fallback from ever succeeding:
1. sender-data ciphertext sample length. The primary decrypt path uses
`MlsCryptoProvider.HASH_OUTPUT_LENGTH` (32 bytes, = KDF.Nh for
HKDF-SHA256, per RFC 9420 §6.3.2); the retained path was using
`AEAD_KEY_LENGTH` (16). The comment on the primary path — "Using
AEAD.Nk here made sender-data decryption fail against every
spec-compliant sender" — was the same fix just never propagated
to this branch. sender-data AEAD always failed, the try/catch
swallowed it, and the fallback returned null.
2. content plaintext was returned raw. AEAD output is a
PrivateMessageContent struct (§6.3.1):
`applicationData<V> || signature<V> || padding`.
The primary decrypt path parses this with `pmcReader.readOpaqueVarInt()`
to extract applicationData; the retained path returned the whole
struct. Callers saw a length-prefixed blob with signature + zero
padding glued on — obvious in a raw diff, invisible in a pass/fail
test that only checked "decrypted is not null".
Fix both in `tryDecryptWithRetainedEpoch`. Existing quartz↔quartz
tests still pass because they don't exercise out-of-order decrypt —
this branch was effectively dead code before.
Add testDecryptRetainedEpoch_ApplicationMessageAfterCommit as a
regression: Bob encrypts at epoch N, Alice advances to N+1 via
add-member, Bob's epoch-N message then arrives at Alice — must round-
trip through the retained-epoch fallback and match the original
plaintext exactly.
Interop Test 06 (wn removes a member from a group Amethyst is in) was
failing with:
GroupEventHandler.add: ERROR Failed to apply commit:
UpdatePath node count (1) doesn't match direct path length (2)
Repro scenario: 3-member group [B=leaf 0, C=leaf 1, A=leaf 2]; B (admin,
committer) removes C. After the Remove proposal applies, leaf 1 is
blank but leaf 2 (A) is still occupied — so leaf_count stays at 3 per
RFC 9420 §7.8 and the sender's direct path has length 2 ([node 1,
root]).
Per RFC 9420 §4.1.2 and §7.9 the **filtered direct path** drops any
parent node whose child on the copath has an empty resolution:
encrypting to such a parent is equivalent to encrypting to its only
non-blank child, which is already on the direct path. In our scenario
the parent at level 1 (parent of leaves 0, 1) has a blank leaf 1 on
the copath — empty resolution — so it's filtered out. Filtered
direct path length is 1, and that's what openmls / whitenoise-rs emit
in `UpdatePath.nodes<V>`.
Quartz was always using the unfiltered direct path on both sides
(send + receive + parent-hash chain + path-secret decrypt). That
worked for quartz↔quartz (both sides agreed), but broke the moment a
spec-correct sender (wn) sent a filtered UpdatePath into a quartz
receiver. Parent-hash chain is further affected because §7.9.2 walks
the filtered path — even if the size check passed, the hashes would
not match.
Fix it throughout:
- RatchetTree.filteredDirectPath: new helper returning parallel
(filteredDirectPath, filteredCopath) with the empty-resolution
entries dropped. Uses the existing `resolution()` helper, which
already handles unmerged_leaves per §4.1.2.
- RatchetTree.applyUpdatePath: size-check + target the FILTERED path.
- MlsGroup.commit (sender): emit one staged path node per filtered
level; encrypt path secrets using the filtered copath; patch parent
hashes into the filtered positions only.
- MlsGroup.processCommitInner (receiver): the UpdatePath node index
aligns to the filtered direct path, so the common-ancestor lookup
must find the filtered index to pick the right ciphertext + copath
resolution. Use the unfiltered index only for counting KDF steps
to the root (path_secret chain advances one step per unfiltered
level regardless of filtering — the chain is continuous; filtering
only decides which levels emit a UpdatePathNode, not how many KDF
steps separate them).
- MlsGroup.computeSenderParentHashes: walk the filtered direct path.
Map each filtered node back to its unfiltered level so the
preUpdateSiblingHashes lookup (indexed by unfiltered level) still
resolves. This makes the parent_hash chain agree with §7.9.2 and
with what openmls computes on the other side.
- MlsGroup.verifyParentHash: short-circuit on empty filtered path
rather than empty unfiltered path.
New regression test: testRemoveMiddleLeaf_ReceiverAcceptsCommit
reproduces the 3-member remove-middle scenario end-to-end within
quartz. It passed before this patch (because both sides were
unfiltered and agreed with each other), and it passes after this
patch (because both sides are now filtered and still agree) — the
compatibility win is that quartz now also agrees with
openmls/wn-produced commits of the same shape.
Also fix an unrelated script bug in marmot-interop.sh: the
discover_a_relays SQL probe was falsely reporting "wn has NO cached
relay entries for A" even when welcome delivery was working, because
the query assumed users.pubkey stored as BLOB but wn stores it as
TEXT (hex) in the current schema. Accept either encoding.
`amy marmot group add` previously sent the Welcome gift wrap to the
inviter's own kind:10050 (ctx.inboxRelays()) and fetched the invitee's
KeyPackage from only the inviter's configured relays. Two users with
disjoint relay configurations could never successfully marmot each
other — the welcome landed somewhere the invitee never polled.
Mirror the Android Account.addMarmotGroupMember flow in the CLI:
- New RecipientRelayFetcher in quartz/marmot one-shot-drains a target
user's kind:10050, kind:10051 and kind:10002 from a seed relay set.
Replaceable-event semantics: newest created_at per kind wins. Guards
against relays echoing events authored by someone else.
- Context.bootstrapRelays() unions the inviter's configured relays with
AmethystDefaults (DefaultNIP65RelaySet + DefaultDMRelayList) so we can
discover strangers even when nothing overlaps with our own config.
- GroupAddMemberCommand now per-invitee: looks up their relay lists;
passes kind:10051 + kind:10002 write + bootstrap to KeyPackageFetcher;
publishes the welcome to kind:10050 (fallback to NIP-65 read, then
DefaultDMRelayList, then our outbox as belt-and-braces).
Group commits/messages (kind:445) continue to use the group's MIP-01
relay set — those were already correct. Exposes welcome_targets and
key_package_relays in the JSON output so callers can verify routing.
The bug (re-added user rejoins without a usable own_leaf, so they
can neither decrypt new group messages nor send) reproduces entirely
with two in-process MlsGroup instances — no relay or second Nostr
client needed. A pure Quartz unit test runs in seconds and catches
the regression deterministically, where the bash interop version
depended on whitenoise-rs, wnd daemons and a local relay.
- quartz: add MlsGroupLifecycleTest.testReaddAfterRemove_RejoinerCanEncryptAndDecrypt
covering create → add → round-trip → remove → re-add with a fresh
KeyPackage → round-trip again, asserting leafIndex and both
encrypt/decrypt directions on the rejoined instance.
- interop: drop test_17_readd_after_remove from the headless suite and
its registration, since the unit test supersedes it.
https://claude.ai/code/session_01JaeqZwVNLKvUUvXWRzPmRP
Two drifts surfaced as 7 failing tests on the jvmTest run:
1. MarmotGroupData CURRENT_VERSION is held at 2 for mdk-core interop
(see the const-doc block), but three tests still assumed v3:
- marmotGroupData_defaultVersionIsThree asserted the constant directly.
- marmotGroupData_roundTripWithDisappearingSecs and
buildGroupEvent_appendsExpirationTagWhenDisappearingConfigured relied
on the default version emitting the v3-only disappearing_message_secs
field, which the encoder correctly skips for v2.
Rename the first to `currentVersionIsHeldAtTwoForMdkInterop` with a
spec-cross-reference comment and an extra assertion that v3 is still
in SUPPORTED_VERSIONS. Pin version=3 explicitly in the other two so
they exercise the v3 path regardless of the interop hold-back.
2. MlsGroup.processCommit now requires a non-empty FramedContentTBS
signature on non-external commits (per RFC 9420 §6.1, introduced in
0c970945). Four pipeline tests still called the raw processCommit
overload with signature=ByteArray(0), so they all failed with
"FramedContentTBS signature missing on commit from leaf 0".
Add a `processFramedCommit` wrapper on MlsGroupManager that mirrors
MlsGroup.processFramedCommit + the retention/persistence bookkeeping
from processCommit, and switch the four tests to feed framedCommitBytes
through it. MarmotInboundProcessor is unaffected (it already has the
decoded PublicMessage fields).
All 7 originally-failing Marmot tests now pass; the 4 unrelated
NostrClient network tests remain independently flaky.
https://claude.ai/code/session_01XQNAmwn1y87GAoK94QWgyn
Two bugs in MlsGroup were blocking 12 of 209 jvmTest cases (all RFC 9420
interop vectors already passed; only our higher-level lifecycle code
diverged from the spec).
Bug A — write/read asymmetry on member commits. processCommit requires
signature + confirmation_tag as separate parameters, but CommitResult
only exposed framedCommitBytes (the full PublicMessage envelope). Tests
called processCommit(commitBytes, ..., ByteArray(0)) and immediately
hit "FramedContentTBS signature missing on commit from leaf N".
Add MlsGroup.processFramedCommit(framedCommitBytes) that unpacks the
MlsMessage(PublicMessage(Commit)) envelope, verifies membership_tag for
member senders (RFC 9420 §6.2), and delegates to processCommit. Mirrors
the unwrap MarmotInboundProcessor already does in production. Updates
the 9 affected tests to use the new entry point.
Bug B — externalJoin used the wrong HPKE info bytes. The joiner
encrypted UpdatePath path-secrets against groupContext.toTlsBytes()
(pre-mutation), but receivers HPKE-Open against the post-mutation
context (epoch+1, new tree_hash) per RFC 9420 §7.6. Result:
AEADBadTagException on every external join.
Refactor externalJoin to follow the same staging pattern as commit():
stage public keys, applyUpdatePath, patch parent_hash, replace
placeholder leaf — then compute pathEncContextBytes from the post-
mutation tree, then HPKE-encrypt path-secrets with that context.
To make the framed pipeline available to external commits as well,
introduce ExternalJoinResult exposing both commitBytes and
framedCommitBytes (PublicMessage with sender = new_member_commit, no
membership_tag). processFramedCommit resolves NEW_MEMBER_COMMIT senders
to the receiver-side leaf index using the same first-blank-or-append
algorithm as RatchetTree.addLeaf, so wire decoding doesn't need to
carry the joiner's chosen index in the empty Sender field.
After: 209/209 MLS tests pass.
Quartz:
- processWelcome now enforces hash(ratchet_tree) == GroupContext.tree_hash
and verifies the joiner-derived confirmation_tag matches GroupInfo's,
closing the gap where a tampered but-signed GroupInfo could diverge
the joiner's epoch secrets.
- externalJoin does the equivalent tree_hash check.
- Promoted constantTimeEquals to a file-level helper so the companion
object (processWelcome) can use it alongside instance methods.
Tests:
- New MlsGroupNegativeTest exercises every authenticity knob on inbound
commits: tampered confirmation_tag, bit-flipped signature, spoofed
senderLeafIndex, wrong wire_format, empty confirmation_tag, and
tampered/missing membership_tag — each must throw and leave the
receiving group's epoch unchanged (atomic-rollback regression cover).
- Fixed stale jvmAndroidTest references to the removed `selfRemove()`
helper — use `buildSelfRemoveProposalMessage()` (now Pair-returning).
Interop harness:
- Adds tests 14–16 as inverted-role scaffolding (wn drives, amy receives).
Currently recorded as SKIP with explicit notes:
14 / 15 block on filtered-direct-path UpdatePath support in
quartz's applyUpdatePath (RFC 9420 §7.7 path compression).
16 blocks on a createdAt-sorted KeyPackage fetch path — the current
fetchFirst-based fetcher is non-deterministic on relay order.
Preserves the test functions so they start running the moment those
gaps are closed.
- Verify FramedContentTBS signature on inbound commits using the
sender's pre-commit leaf signature key (rejects forged commits
under a compromised exporter key).
- Verify membership_tag on inbound PublicMessage commits before
advancing state.
- Check LeafNode lifetime bounds (notBefore..notAfter) on UpdatePath.
- Require confirmation_tag to be non-empty.
- Thread wire_format into buildCommitFramedContentTbs so signature
verification reproduces the bytes the sender signed.
- Snapshot RatchetTree + groupContext + epochSecrets + initSecret +
secretTree + interimTranscriptHash + pendingProposals + sentKeys
at the top of processCommit, and restore on any throwable so a
verification failure mid-apply can't leave the group diverged.
13/13 passing; the System.err.println trace in
processPrivateMessage/applyCommit that we added to diagnose the
wire_format / admin / SelfRemove-proposal bugs is no longer needed.
Logging via the existing Log.d("MarmotDbg") calls is preserved.
https://claude.ai/code/session_1469d7f4-bb66-4ffa-a44d-1dfa4b526484
Reverts the SelfRemove-as-commit approach. openmls rejects a
commit whose only proposal is a SelfRemove from the committer with
RequiredPathNotFound — the spec model is that the removing member
publishes a STANDALONE proposal, and an admin receiver
(wn's mdk auto-commit path) folds it into their next commit.
Add `MlsGroup.buildSelfRemoveProposalMessage()` which frames
Proposal.SelfRemove as `MlsMessage(PublicMessage{proposal})` with
the proper FramedContent / signature / membership_tag and returns
the ready-to-publish bytes + the pre-commit exporter key for outer
encryption. Rewire `MlsGroupManager.leaveGroup` and
`MarmotManager.leaveGroup` to use it.
Target: test 11 (amy leave → B auto-commits SelfRemove →
removes amy from member list).
https://claude.ai/code/session_1469d7f4-bb66-4ffa-a44d-1dfa4b526484
SelfRemove is standardized in draft-ietf-mls-extensions with
IANA-registered proposal type 0x000A. openmls and mdk encode it
that way on the wire. Quartz was writing 0xF001 (Marmot private-use
range), so mdk's strict tls_codec read 0x000A for a known proposal
type it didn't recognize and the decoder drifted past the variant
body, surfacing later as
Tls(DecodingError("Trying to decode Option<T> with 64 for option..."))
when it reached `Commit.path` with misaligned bytes.
Fixes test 11 (amy leave → B still sees A as member).
https://claude.ai/code/session_1469d7f4-bb66-4ffa-a44d-1dfa4b526484
Quartz's leaveGroup was returning `proposal.toTlsBytes()` and
publishing that raw on kind:445. That's not a valid MLS message
— it's just the proposal struct with no FramedContent / MlsMessage
framing — and even if it parsed, mdk/openmls would only STAGE the
proposal. The target never actually leaves the tree until someone
commits with that proposal.
MIP-03 explicitly allows non-admins to commit a SelfRemove-only
commit, so amy can produce a proper committable kind:445 herself
after the self-demote. Replace `selfRemove()` (returned raw bytes)
with `selfRemoveCommit()` which adds the proposal to the pending
set and runs the standard `commit()` path. Plumb the resulting
CommitResult through MarmotManager.leaveGroup so the outer
encryption uses `preCommitExporterSecret` (the epoch we're
leaving, which is the one every existing member still has).
This finally propagates amy's leave to B in test 11.
https://claude.ai/code/session_1469d7f4-bb66-4ffa-a44d-1dfa4b526484
PrivateMessage commits apply their Commit body inline inside
`MlsGroup.decrypt`, so the epoch advance + extension replace
happen in `MlsGroupManager.decrypt` — not in `processCommit`.
The old `decrypt` path never called `persistGroup`, so amy's
in-memory promotion (e.g. test 08 "B promotes A") looked correct
for the current CLI invocation but the NEXT `amy marmot …` command
reloaded the pre-commit extensions from disk and refused the
follow-up rename with "MIP-01: only admins may update group
extensions".
Detect epoch advance + COMMIT result inside `decrypt`, then push
the pre-commit exporter into the retained-epoch window and persist
the group so CLI reloads see the post-commit state.
https://claude.ai/code/session_1469d7f4-bb66-4ffa-a44d-1dfa4b526484
RFC 9420 §8.2 says ConfirmedTranscriptHashInput carries the
wire_format of the AuthenticatedContent being committed, not a
hard-coded value. openmls/mdk pass mls_content.wire_format()
through, so B's PrivateMessage commits use wire_format=2 in their
transcript-hash input. Quartz always wrote wire_format=1
(PublicMessage), so amy's receiver-side new_confirmed_transcript_hash
diverged from B's — and every tag derived from it
(confirmation_key, confirmation_tag, epoch secrets) diverged too,
surfacing as "Confirmation tag verification failed" once my
error-surfacing fix stopped hiding it.
Plumb the real wire format through processCommit and its decrypt()
dispatch so PrivateMessage commits from B are hashed as
PrivateMessage on amy's side.
https://claude.ai/code/session_1469d7f4-bb66-4ffa-a44d-1dfa4b526484
`MlsGroupManager.decrypt` used to swallow the current-epoch exception
via `decryptOrNull`, try retained epochs, then retry `group.decrypt`.
After our inline-processCommit change this means a mid-commit throw
leaves the group half-advanced, and the retry then reports a
misleading "Message epoch N doesn't match current epoch N+1" —
masking the real cause (unable-to-decrypt path, path-mismatch, etc.).
Swap the order: call `group.decrypt` directly first, capture any
exception, fall through to retained epochs, and re-raise the original
exception if every epoch fails. Retains same semantics for messages
that DO decrypt on the first try.
https://claude.ai/code/session_1469d7f4-bb66-4ffa-a44d-1dfa4b526484
Tracing which path B→A commits take (processPrivateMessage vs
applyCommit) and why past-epoch PrivateMessage events surface
as Failure instead of Duplicate in tests 07/08/11/12. Will be
reverted once diagnosis is complete.
https://claude.ai/code/session_1469d7f4-bb66-4ffa-a44d-1dfa4b526484
Extend the past-epoch dedup check to PrivateMessage commits too.
Previously only the PublicMessage branch returned
`GroupEventResult.Duplicate` for a stale commit echo; the PrivateMessage
branch called `groupManager.decrypt()` directly, which consumed a
generation on the sender's ratchet and then failed with
`Message epoch X doesn't match current epoch Y` — polluting the harness
log and (worse) burning the real commit's generation slot.
Peek the epoch from the parsed PrivateMessage before touching the
secret tree: past-epoch → Duplicate, future-epoch → Error, same-epoch
falls through to the existing decrypt-and-dispatch path.
https://claude.ai/code/session_016kAxdp6ubB5CnF9URhCEzP
MDK-core (openmls) defaults group configuration to
`MIXED_CIPHERTEXT_WIRE_FORMAT_POLICY` — outgoing commits / proposals
ship as PrivateMessage. Quartz only handled PrivateMessage for
ContentType.APPLICATION; any handshake message came in via the
application ratchet, consumed the first application generation on
whatever sender sent an app message next, and then died with
`Generation 0 already consumed` the moment the receiver saw the real
PrivateMessage commit. That's why every A-side processing of B's
rename / promote / remove / leave commit timed out.
Fix:
1. SecretTree grows a parallel `handshakeKeyNonceForGeneration`
path, using the per-sender `handshakeSecret` / `handshakeGeneration`
the struct already tracked — with its own skipped-keys cache and
replay-detection set so handshake and application generations
never clash.
2. `MlsGroup.decrypt()` dispatches on the PrivateMessage's
`content_type` BEFORE consuming any ratchet — COMMIT and PROPOSAL
take the handshake path; APPLICATION stays on the existing
application path.
3. COMMIT bodies decode as `Commit || signature<V> || confirmation_tag<V>
|| padding` (RFC 9420 §6.3.1) and route into `processCommit` inline,
so the caller just sees the epoch advance — no new public API.
PROPOSAL still rejects (standalone proposals aren't used by
Marmot flows yet) with a clear message instead of a cryptic
generation-consumed error.
https://claude.ai/code/session_016kAxdp6ubB5CnF9URhCEzP
For a GroupContextExtensions commit, openmls signs the
`FramedContentTBS` over `group.public_group.group_context()` — the
UNMUTATED context, before the `diff` applies the GCE proposal. The
post-proposal extensions only make it into the HPKE path-encryption
context and the subsequent key schedule, not into the TBS / signature /
membership_tag.
Quartz's `applyProposal` handler for `GroupContextExtensions` mutates
`groupContext.extensions` inline — by the time `commit()` captured
`preCommitContextBytes`, the extensions were already updated. The
signature and membership_tag were then minted over the post-GCE context,
and openmls's `verify_membership` recomputed the MAC over the pre-GCE
context and reported `ValidationError(InvalidMembershipTag)` on every
cross-impl rename / promote / demote / leave.
Fix: snapshot `groupContext.extensions` before any proposal is applied,
and rebuild `preCommitContextBytes` with those original extensions. The
path-encryption context and post-commit key schedule still use the
updated extensions, matching openmls.
https://claude.ai/code/session_016kAxdp6ubB5CnF9URhCEzP
RFC 9420 §7.8: the ratchet tree has no trailing blank leaves — when a
Remove blanks the rightmost leaf, everyone MUST shrink `leaf_count` and
drop the (now-orphaned) parent nodes so future `direct_path`,
`resolution`, and `treeHash` computations see the new shape.
Quartz was blanking the leaf and its direct path but leaving
`_leafCount` untouched. Openmls / mdk shrink, so after
e.g. `amy removes C (leaf 2)` quartz still had `leafCount = 3` while
B's openmls had `leafCount = 2`. Amy's `direct_path` from leaf 0 then
had two entries (to the 3-leaf-tree root), but B's tree layout expected
one — and openmls rejected the commit with
`UpdatePathError(PathLengthMismatch)`.
Fix: after blanking the leaf + direct path, walk `leafCount` down past
any trailing blanks and truncate the `nodes` list to `nodeCount(leafCount)`.
https://claude.ai/code/session_016kAxdp6ubB5CnF9URhCEzP
RFC 9420 §9.2: `commit_secret = DeriveSecret(path_secret_at_root, "path")`.
Openmls / mdk implement this exactly — after deriving the ratchet-tree's
own path_secrets up to (and including) the root, they advance one more
step and use THAT as the commit_secret contribution to the new epoch's
key schedule.
Quartz was using the root's own path_secret as commit_secret on both
the encryption side (MlsGroup.commit) and the decryption side
(MlsGroup.processCommit), plus in externalJoin's commit construction.
Internally consistent, so quartz↔quartz worked; but every cross-impl
commit (quartz→openmls or openmls→quartz) derived a different
epoch_secret, cascaded into a different confirmation_key, and failed at
`InvalidCommit(ConfirmationTagMismatch)` — the blocker behind every
test that actually needed an openmls peer to accept a quartz-produced
commit or vice-versa once the UpdatePath-layer bugs were out of the way.
Fix: add one `DeriveSecret(pathSecrets.last().pathSecret, "path")` step
on the sender side, and one extra `DeriveSecret(..., "path")` past the
root on the receiver side, mirroring openmls's `ParentNode::derive_path`
loop post-condition.
https://claude.ai/code/session_016kAxdp6ubB5CnF9URhCEzP
RFC 9420 §8.2: the confirmed_transcript_hash for a Commit is taken over
`wire_format || FramedContent || signature` — where `signature` is the
real `FramedContentAuthData.signature`, not an empty placeholder.
Quartz was hashing over a zero-length signature, so the committer and
every receiver baked DIFFERENT `confirmed_transcript_hash` bytes into
their new GroupContext. The committer's confirmation_tag — MAC'd over
that value — never matched the receiver's recomputation, and openmls
rejected the commit with `InvalidCommit(ConfirmationTagMismatch)`.
Fix restructures the commit flow so the FramedContentTBS is built and
signed BEFORE the transcript hash update:
1. extract `buildCommitFramedContentTbs()` as a shared helper
2. in `MlsGroup.commit()`, sign the TBS immediately after building
the commit, then pass that signature to
`buildConfirmedTranscriptHashInput(..., signature)`
3. thread the signature through `framePublicMessageCommit` as a
parameter (callers already have it) instead of re-signing inside
`processCommit` also takes the signature and forwards it to
`buildConfirmedTranscriptHashInput`; `MarmotInboundProcessor.applyCommit`
pulls `pubMsg.signature` out of the `PublicMessage` envelope and passes
it through `MlsGroupManager.processCommit`.
Also switches `RatchetTree.derivePathSecrets` from a custom
`expandWithLabel(nodeSecret, "hpke", ...)` derivation to HPKE's standard
`DeriveKeyPair(node_secret)` (RFC 9180 §7.1.3). Without this, the
receiver derives parent-node public keys that don't match the ones
quartz baked into `UpdatePathNode.encryption_key`, which manifested as
`UpdatePathError(PathMismatch)` once the HPKE context fix landed.
https://claude.ai/code/session_016kAxdp6ubB5CnF9URhCEzP
RFC 9420 §7.6 / openmls `compute_path`: after applying the Commit's
proposals and the committer's UpdatePath locally, the committer must
bump the epoch and recompute tree_hash, THEN serialize the GroupContext
(with the still-pre-commit confirmed_transcript_hash) and use those
bytes as the HPKE info for encrypting every path secret. Receivers
perform the exact same recomputation on their copy of the tree before
decrypting, so the AEAD keys line up.
Quartz was using the fully pre-commit GroupContext on both sides. Self-
consistent (quartz→quartz passed), but openmls/mdk re-derives the AEAD
key from the post-mutation context and the tag check fails, which
surfaced as `InvalidCommit(UpdatePathError(UnableToDecrypt))` on every
commit-produced-by-quartz that an openmls peer had to process (add,
rename, remove, leave, promote, catchup).
Fix restructures `MlsGroup.commit()` into stage→apply→encrypt:
1. derive path-secret keypairs, stage UpdatePath entries with public
keys only
2. apply the staged path, patch parent_hashes, swap in the new leaf
3. serialize an intermediate GroupContext (new epoch + new tree_hash,
old confirmed_transcript_hash) and HPKE-encrypt each copath
resolution under that info
`processCommit()` mirrors the change on the decryption side, and also
splits the proposal loop so Add proposals apply after non-Adds (matching
the committer's order and unblocking the new-leaf-exclusion filter for
the decryption resolution lookup).
https://claude.ai/code/session_016kAxdp6ubB5CnF9URhCEzP
RFC 9420 §12.4.1: when a Commit contains Add proposals, the committer
MUST exclude the new leaves from the copath-resolution used for path-
secret encryption. Joiners get the group state via the Welcome at epoch
N+1; they neither need nor expect a path secret for epoch N.
Quartz was including them. openmls (via mdk-core) then runs its own
resolution-minus-exclusion-list when picking which ciphertext to
decrypt, so its `resolution_position` landed one slot off from where
quartz had put the ciphertext for the existing member. The AEAD tag
naturally mismatched and the commit died with
`InvalidCommit(UpdatePathError(UnableToDecrypt))` — the blocker behind
every test that needs an existing openmls member to process a commit
produced by quartz (add, rename, remove, leave, promote, catchup).
Fix: in `MlsGroup.commit()`, capture `newLeafIndices` from the Add
proposals applied in this epoch and `filterNot` them out of every
copath node's resolution before HPKE-encrypting the path secret.
https://claude.ai/code/session_016kAxdp6ubB5CnF9URhCEzP
Phase 2 of the Clubhouse/nests integration: when the live-activity
channel screen is opened on a kind 30312 MeetingSpaceEvent, render an
audio-room "stage" above the chat that shows host/speaker/audience
avatars and exposes hand-raise + mute toggles backed by NIP-53 kind
10312 presence events.
Quartz:
- New MutedTag (`["muted","1"|"0"]`) and `muted()` builder helper for
MeetingRoomPresenceEvent.
- New MeetingRoomPresenceEvent.build() overload that accepts a 30312
MeetingSpaceEvent root (matching the existing 30313 overload) and
optionally encodes hand-raise + mute flags in one call.
Amethyst:
- AudioRoomStage composable: filters participants from the 30312 event
into hosts / speakers / audience, renders avatars, and publishes
presence on enter + every 30 s while composed (on dispose it pushes
one final lowered-hand presence so peers drop us before timeout).
- ChannelView wires AudioRoomStage in next to ShowVideoStreaming; the
latter is a no-op for non-30311 events so non-audio rooms are
unaffected.
No audio is captured yet — the mic toggle is a Nostr-only signal
until Phase 3 brings the MoQ/WebTransport transport.
https://claude.ai/code/session_013nVLALALKaHVgHm9u5Cg8D
Claude