Two drifts surfaced as 7 failing tests on the jvmTest run:
1. MarmotGroupData CURRENT_VERSION is held at 2 for mdk-core interop
(see the const-doc block), but three tests still assumed v3:
- marmotGroupData_defaultVersionIsThree asserted the constant directly.
- marmotGroupData_roundTripWithDisappearingSecs and
buildGroupEvent_appendsExpirationTagWhenDisappearingConfigured relied
on the default version emitting the v3-only disappearing_message_secs
field, which the encoder correctly skips for v2.
Rename the first to `currentVersionIsHeldAtTwoForMdkInterop` with a
spec-cross-reference comment and an extra assertion that v3 is still
in SUPPORTED_VERSIONS. Pin version=3 explicitly in the other two so
they exercise the v3 path regardless of the interop hold-back.
2. MlsGroup.processCommit now requires a non-empty FramedContentTBS
signature on non-external commits (per RFC 9420 §6.1, introduced in
0c970945). Four pipeline tests still called the raw processCommit
overload with signature=ByteArray(0), so they all failed with
"FramedContentTBS signature missing on commit from leaf 0".
Add a `processFramedCommit` wrapper on MlsGroupManager that mirrors
MlsGroup.processFramedCommit + the retention/persistence bookkeeping
from processCommit, and switch the four tests to feed framedCommitBytes
through it. MarmotInboundProcessor is unaffected (it already has the
decoded PublicMessage fields).
All 7 originally-failing Marmot tests now pass; the 4 unrelated
NostrClient network tests remain independently flaky.
https://claude.ai/code/session_01XQNAmwn1y87GAoK94QWgyn
Two bugs in MlsGroup were blocking 12 of 209 jvmTest cases (all RFC 9420
interop vectors already passed; only our higher-level lifecycle code
diverged from the spec).
Bug A — write/read asymmetry on member commits. processCommit requires
signature + confirmation_tag as separate parameters, but CommitResult
only exposed framedCommitBytes (the full PublicMessage envelope). Tests
called processCommit(commitBytes, ..., ByteArray(0)) and immediately
hit "FramedContentTBS signature missing on commit from leaf N".
Add MlsGroup.processFramedCommit(framedCommitBytes) that unpacks the
MlsMessage(PublicMessage(Commit)) envelope, verifies membership_tag for
member senders (RFC 9420 §6.2), and delegates to processCommit. Mirrors
the unwrap MarmotInboundProcessor already does in production. Updates
the 9 affected tests to use the new entry point.
Bug B — externalJoin used the wrong HPKE info bytes. The joiner
encrypted UpdatePath path-secrets against groupContext.toTlsBytes()
(pre-mutation), but receivers HPKE-Open against the post-mutation
context (epoch+1, new tree_hash) per RFC 9420 §7.6. Result:
AEADBadTagException on every external join.
Refactor externalJoin to follow the same staging pattern as commit():
stage public keys, applyUpdatePath, patch parent_hash, replace
placeholder leaf — then compute pathEncContextBytes from the post-
mutation tree, then HPKE-encrypt path-secrets with that context.
To make the framed pipeline available to external commits as well,
introduce ExternalJoinResult exposing both commitBytes and
framedCommitBytes (PublicMessage with sender = new_member_commit, no
membership_tag). processFramedCommit resolves NEW_MEMBER_COMMIT senders
to the receiver-side leaf index using the same first-blank-or-append
algorithm as RatchetTree.addLeaf, so wire decoding doesn't need to
carry the joiner's chosen index in the empty Sender field.
After: 209/209 MLS tests pass.
Quartz:
- processWelcome now enforces hash(ratchet_tree) == GroupContext.tree_hash
and verifies the joiner-derived confirmation_tag matches GroupInfo's,
closing the gap where a tampered but-signed GroupInfo could diverge
the joiner's epoch secrets.
- externalJoin does the equivalent tree_hash check.
- Promoted constantTimeEquals to a file-level helper so the companion
object (processWelcome) can use it alongside instance methods.
Tests:
- New MlsGroupNegativeTest exercises every authenticity knob on inbound
commits: tampered confirmation_tag, bit-flipped signature, spoofed
senderLeafIndex, wrong wire_format, empty confirmation_tag, and
tampered/missing membership_tag — each must throw and leave the
receiving group's epoch unchanged (atomic-rollback regression cover).
- Fixed stale jvmAndroidTest references to the removed `selfRemove()`
helper — use `buildSelfRemoveProposalMessage()` (now Pair-returning).
Interop harness:
- Adds tests 14–16 as inverted-role scaffolding (wn drives, amy receives).
Currently recorded as SKIP with explicit notes:
14 / 15 block on filtered-direct-path UpdatePath support in
quartz's applyUpdatePath (RFC 9420 §7.7 path compression).
16 blocks on a createdAt-sorted KeyPackage fetch path — the current
fetchFirst-based fetcher is non-deterministic on relay order.
Preserves the test functions so they start running the moment those
gaps are closed.
- Verify FramedContentTBS signature on inbound commits using the
sender's pre-commit leaf signature key (rejects forged commits
under a compromised exporter key).
- Verify membership_tag on inbound PublicMessage commits before
advancing state.
- Check LeafNode lifetime bounds (notBefore..notAfter) on UpdatePath.
- Require confirmation_tag to be non-empty.
- Thread wire_format into buildCommitFramedContentTbs so signature
verification reproduces the bytes the sender signed.
- Snapshot RatchetTree + groupContext + epochSecrets + initSecret +
secretTree + interimTranscriptHash + pendingProposals + sentKeys
at the top of processCommit, and restore on any throwable so a
verification failure mid-apply can't leave the group diverged.
13/13 passing; the System.err.println trace in
processPrivateMessage/applyCommit that we added to diagnose the
wire_format / admin / SelfRemove-proposal bugs is no longer needed.
Logging via the existing Log.d("MarmotDbg") calls is preserved.
https://claude.ai/code/session_1469d7f4-bb66-4ffa-a44d-1dfa4b526484
Reverts the SelfRemove-as-commit approach. openmls rejects a
commit whose only proposal is a SelfRemove from the committer with
RequiredPathNotFound — the spec model is that the removing member
publishes a STANDALONE proposal, and an admin receiver
(wn's mdk auto-commit path) folds it into their next commit.
Add `MlsGroup.buildSelfRemoveProposalMessage()` which frames
Proposal.SelfRemove as `MlsMessage(PublicMessage{proposal})` with
the proper FramedContent / signature / membership_tag and returns
the ready-to-publish bytes + the pre-commit exporter key for outer
encryption. Rewire `MlsGroupManager.leaveGroup` and
`MarmotManager.leaveGroup` to use it.
Target: test 11 (amy leave → B auto-commits SelfRemove →
removes amy from member list).
https://claude.ai/code/session_1469d7f4-bb66-4ffa-a44d-1dfa4b526484
SelfRemove is standardized in draft-ietf-mls-extensions with
IANA-registered proposal type 0x000A. openmls and mdk encode it
that way on the wire. Quartz was writing 0xF001 (Marmot private-use
range), so mdk's strict tls_codec read 0x000A for a known proposal
type it didn't recognize and the decoder drifted past the variant
body, surfacing later as
Tls(DecodingError("Trying to decode Option<T> with 64 for option..."))
when it reached `Commit.path` with misaligned bytes.
Fixes test 11 (amy leave → B still sees A as member).
https://claude.ai/code/session_1469d7f4-bb66-4ffa-a44d-1dfa4b526484
Quartz's leaveGroup was returning `proposal.toTlsBytes()` and
publishing that raw on kind:445. That's not a valid MLS message
— it's just the proposal struct with no FramedContent / MlsMessage
framing — and even if it parsed, mdk/openmls would only STAGE the
proposal. The target never actually leaves the tree until someone
commits with that proposal.
MIP-03 explicitly allows non-admins to commit a SelfRemove-only
commit, so amy can produce a proper committable kind:445 herself
after the self-demote. Replace `selfRemove()` (returned raw bytes)
with `selfRemoveCommit()` which adds the proposal to the pending
set and runs the standard `commit()` path. Plumb the resulting
CommitResult through MarmotManager.leaveGroup so the outer
encryption uses `preCommitExporterSecret` (the epoch we're
leaving, which is the one every existing member still has).
This finally propagates amy's leave to B in test 11.
https://claude.ai/code/session_1469d7f4-bb66-4ffa-a44d-1dfa4b526484
PrivateMessage commits apply their Commit body inline inside
`MlsGroup.decrypt`, so the epoch advance + extension replace
happen in `MlsGroupManager.decrypt` — not in `processCommit`.
The old `decrypt` path never called `persistGroup`, so amy's
in-memory promotion (e.g. test 08 "B promotes A") looked correct
for the current CLI invocation but the NEXT `amy marmot …` command
reloaded the pre-commit extensions from disk and refused the
follow-up rename with "MIP-01: only admins may update group
extensions".
Detect epoch advance + COMMIT result inside `decrypt`, then push
the pre-commit exporter into the retained-epoch window and persist
the group so CLI reloads see the post-commit state.
https://claude.ai/code/session_1469d7f4-bb66-4ffa-a44d-1dfa4b526484
RFC 9420 §8.2 says ConfirmedTranscriptHashInput carries the
wire_format of the AuthenticatedContent being committed, not a
hard-coded value. openmls/mdk pass mls_content.wire_format()
through, so B's PrivateMessage commits use wire_format=2 in their
transcript-hash input. Quartz always wrote wire_format=1
(PublicMessage), so amy's receiver-side new_confirmed_transcript_hash
diverged from B's — and every tag derived from it
(confirmation_key, confirmation_tag, epoch secrets) diverged too,
surfacing as "Confirmation tag verification failed" once my
error-surfacing fix stopped hiding it.
Plumb the real wire format through processCommit and its decrypt()
dispatch so PrivateMessage commits from B are hashed as
PrivateMessage on amy's side.
https://claude.ai/code/session_1469d7f4-bb66-4ffa-a44d-1dfa4b526484
`MlsGroupManager.decrypt` used to swallow the current-epoch exception
via `decryptOrNull`, try retained epochs, then retry `group.decrypt`.
After our inline-processCommit change this means a mid-commit throw
leaves the group half-advanced, and the retry then reports a
misleading "Message epoch N doesn't match current epoch N+1" —
masking the real cause (unable-to-decrypt path, path-mismatch, etc.).
Swap the order: call `group.decrypt` directly first, capture any
exception, fall through to retained epochs, and re-raise the original
exception if every epoch fails. Retains same semantics for messages
that DO decrypt on the first try.
https://claude.ai/code/session_1469d7f4-bb66-4ffa-a44d-1dfa4b526484
Tracing which path B→A commits take (processPrivateMessage vs
applyCommit) and why past-epoch PrivateMessage events surface
as Failure instead of Duplicate in tests 07/08/11/12. Will be
reverted once diagnosis is complete.
https://claude.ai/code/session_1469d7f4-bb66-4ffa-a44d-1dfa4b526484
Extend the past-epoch dedup check to PrivateMessage commits too.
Previously only the PublicMessage branch returned
`GroupEventResult.Duplicate` for a stale commit echo; the PrivateMessage
branch called `groupManager.decrypt()` directly, which consumed a
generation on the sender's ratchet and then failed with
`Message epoch X doesn't match current epoch Y` — polluting the harness
log and (worse) burning the real commit's generation slot.
Peek the epoch from the parsed PrivateMessage before touching the
secret tree: past-epoch → Duplicate, future-epoch → Error, same-epoch
falls through to the existing decrypt-and-dispatch path.
https://claude.ai/code/session_016kAxdp6ubB5CnF9URhCEzP
MDK-core (openmls) defaults group configuration to
`MIXED_CIPHERTEXT_WIRE_FORMAT_POLICY` — outgoing commits / proposals
ship as PrivateMessage. Quartz only handled PrivateMessage for
ContentType.APPLICATION; any handshake message came in via the
application ratchet, consumed the first application generation on
whatever sender sent an app message next, and then died with
`Generation 0 already consumed` the moment the receiver saw the real
PrivateMessage commit. That's why every A-side processing of B's
rename / promote / remove / leave commit timed out.
Fix:
1. SecretTree grows a parallel `handshakeKeyNonceForGeneration`
path, using the per-sender `handshakeSecret` / `handshakeGeneration`
the struct already tracked — with its own skipped-keys cache and
replay-detection set so handshake and application generations
never clash.
2. `MlsGroup.decrypt()` dispatches on the PrivateMessage's
`content_type` BEFORE consuming any ratchet — COMMIT and PROPOSAL
take the handshake path; APPLICATION stays on the existing
application path.
3. COMMIT bodies decode as `Commit || signature<V> || confirmation_tag<V>
|| padding` (RFC 9420 §6.3.1) and route into `processCommit` inline,
so the caller just sees the epoch advance — no new public API.
PROPOSAL still rejects (standalone proposals aren't used by
Marmot flows yet) with a clear message instead of a cryptic
generation-consumed error.
https://claude.ai/code/session_016kAxdp6ubB5CnF9URhCEzP
For a GroupContextExtensions commit, openmls signs the
`FramedContentTBS` over `group.public_group.group_context()` — the
UNMUTATED context, before the `diff` applies the GCE proposal. The
post-proposal extensions only make it into the HPKE path-encryption
context and the subsequent key schedule, not into the TBS / signature /
membership_tag.
Quartz's `applyProposal` handler for `GroupContextExtensions` mutates
`groupContext.extensions` inline — by the time `commit()` captured
`preCommitContextBytes`, the extensions were already updated. The
signature and membership_tag were then minted over the post-GCE context,
and openmls's `verify_membership` recomputed the MAC over the pre-GCE
context and reported `ValidationError(InvalidMembershipTag)` on every
cross-impl rename / promote / demote / leave.
Fix: snapshot `groupContext.extensions` before any proposal is applied,
and rebuild `preCommitContextBytes` with those original extensions. The
path-encryption context and post-commit key schedule still use the
updated extensions, matching openmls.
https://claude.ai/code/session_016kAxdp6ubB5CnF9URhCEzP
RFC 9420 §7.8: the ratchet tree has no trailing blank leaves — when a
Remove blanks the rightmost leaf, everyone MUST shrink `leaf_count` and
drop the (now-orphaned) parent nodes so future `direct_path`,
`resolution`, and `treeHash` computations see the new shape.
Quartz was blanking the leaf and its direct path but leaving
`_leafCount` untouched. Openmls / mdk shrink, so after
e.g. `amy removes C (leaf 2)` quartz still had `leafCount = 3` while
B's openmls had `leafCount = 2`. Amy's `direct_path` from leaf 0 then
had two entries (to the 3-leaf-tree root), but B's tree layout expected
one — and openmls rejected the commit with
`UpdatePathError(PathLengthMismatch)`.
Fix: after blanking the leaf + direct path, walk `leafCount` down past
any trailing blanks and truncate the `nodes` list to `nodeCount(leafCount)`.
https://claude.ai/code/session_016kAxdp6ubB5CnF9URhCEzP
RFC 9420 §9.2: `commit_secret = DeriveSecret(path_secret_at_root, "path")`.
Openmls / mdk implement this exactly — after deriving the ratchet-tree's
own path_secrets up to (and including) the root, they advance one more
step and use THAT as the commit_secret contribution to the new epoch's
key schedule.
Quartz was using the root's own path_secret as commit_secret on both
the encryption side (MlsGroup.commit) and the decryption side
(MlsGroup.processCommit), plus in externalJoin's commit construction.
Internally consistent, so quartz↔quartz worked; but every cross-impl
commit (quartz→openmls or openmls→quartz) derived a different
epoch_secret, cascaded into a different confirmation_key, and failed at
`InvalidCommit(ConfirmationTagMismatch)` — the blocker behind every
test that actually needed an openmls peer to accept a quartz-produced
commit or vice-versa once the UpdatePath-layer bugs were out of the way.
Fix: add one `DeriveSecret(pathSecrets.last().pathSecret, "path")` step
on the sender side, and one extra `DeriveSecret(..., "path")` past the
root on the receiver side, mirroring openmls's `ParentNode::derive_path`
loop post-condition.
https://claude.ai/code/session_016kAxdp6ubB5CnF9URhCEzP
RFC 9420 §8.2: the confirmed_transcript_hash for a Commit is taken over
`wire_format || FramedContent || signature` — where `signature` is the
real `FramedContentAuthData.signature`, not an empty placeholder.
Quartz was hashing over a zero-length signature, so the committer and
every receiver baked DIFFERENT `confirmed_transcript_hash` bytes into
their new GroupContext. The committer's confirmation_tag — MAC'd over
that value — never matched the receiver's recomputation, and openmls
rejected the commit with `InvalidCommit(ConfirmationTagMismatch)`.
Fix restructures the commit flow so the FramedContentTBS is built and
signed BEFORE the transcript hash update:
1. extract `buildCommitFramedContentTbs()` as a shared helper
2. in `MlsGroup.commit()`, sign the TBS immediately after building
the commit, then pass that signature to
`buildConfirmedTranscriptHashInput(..., signature)`
3. thread the signature through `framePublicMessageCommit` as a
parameter (callers already have it) instead of re-signing inside
`processCommit` also takes the signature and forwards it to
`buildConfirmedTranscriptHashInput`; `MarmotInboundProcessor.applyCommit`
pulls `pubMsg.signature` out of the `PublicMessage` envelope and passes
it through `MlsGroupManager.processCommit`.
Also switches `RatchetTree.derivePathSecrets` from a custom
`expandWithLabel(nodeSecret, "hpke", ...)` derivation to HPKE's standard
`DeriveKeyPair(node_secret)` (RFC 9180 §7.1.3). Without this, the
receiver derives parent-node public keys that don't match the ones
quartz baked into `UpdatePathNode.encryption_key`, which manifested as
`UpdatePathError(PathMismatch)` once the HPKE context fix landed.
https://claude.ai/code/session_016kAxdp6ubB5CnF9URhCEzP
RFC 9420 §7.6 / openmls `compute_path`: after applying the Commit's
proposals and the committer's UpdatePath locally, the committer must
bump the epoch and recompute tree_hash, THEN serialize the GroupContext
(with the still-pre-commit confirmed_transcript_hash) and use those
bytes as the HPKE info for encrypting every path secret. Receivers
perform the exact same recomputation on their copy of the tree before
decrypting, so the AEAD keys line up.
Quartz was using the fully pre-commit GroupContext on both sides. Self-
consistent (quartz→quartz passed), but openmls/mdk re-derives the AEAD
key from the post-mutation context and the tag check fails, which
surfaced as `InvalidCommit(UpdatePathError(UnableToDecrypt))` on every
commit-produced-by-quartz that an openmls peer had to process (add,
rename, remove, leave, promote, catchup).
Fix restructures `MlsGroup.commit()` into stage→apply→encrypt:
1. derive path-secret keypairs, stage UpdatePath entries with public
keys only
2. apply the staged path, patch parent_hashes, swap in the new leaf
3. serialize an intermediate GroupContext (new epoch + new tree_hash,
old confirmed_transcript_hash) and HPKE-encrypt each copath
resolution under that info
`processCommit()` mirrors the change on the decryption side, and also
splits the proposal loop so Add proposals apply after non-Adds (matching
the committer's order and unblocking the new-leaf-exclusion filter for
the decryption resolution lookup).
https://claude.ai/code/session_016kAxdp6ubB5CnF9URhCEzP
RFC 9420 §12.4.1: when a Commit contains Add proposals, the committer
MUST exclude the new leaves from the copath-resolution used for path-
secret encryption. Joiners get the group state via the Welcome at epoch
N+1; they neither need nor expect a path secret for epoch N.
Quartz was including them. openmls (via mdk-core) then runs its own
resolution-minus-exclusion-list when picking which ciphertext to
decrypt, so its `resolution_position` landed one slot off from where
quartz had put the ciphertext for the existing member. The AEAD tag
naturally mismatched and the commit died with
`InvalidCommit(UpdatePathError(UnableToDecrypt))` — the blocker behind
every test that needs an existing openmls member to process a commit
produced by quartz (add, rename, remove, leave, promote, catchup).
Fix: in `MlsGroup.commit()`, capture `newLeafIndices` from the Add
proposals applied in this epoch and `filterNot` them out of every
copath node's resolution before HPKE-encrypting the path secret.
https://claude.ai/code/session_016kAxdp6ubB5CnF9URhCEzP
Phase 2 of the Clubhouse/nests integration: when the live-activity
channel screen is opened on a kind 30312 MeetingSpaceEvent, render an
audio-room "stage" above the chat that shows host/speaker/audience
avatars and exposes hand-raise + mute toggles backed by NIP-53 kind
10312 presence events.
Quartz:
- New MutedTag (`["muted","1"|"0"]`) and `muted()` builder helper for
MeetingRoomPresenceEvent.
- New MeetingRoomPresenceEvent.build() overload that accepts a 30312
MeetingSpaceEvent root (matching the existing 30313 overload) and
optionally encodes hand-raise + mute flags in one call.
Amethyst:
- AudioRoomStage composable: filters participants from the 30312 event
into hosts / speakers / audience, renders avatars, and publishes
presence on enter + every 30 s while composed (on dispose it pushes
one final lowered-hand presence so peers drop us before timeout).
- ChannelView wires AudioRoomStage in next to ShowVideoStreaming; the
latter is a no-op for non-30311 events so non-audio rooms are
unaffected.
No audio is captured yet — the mic toggle is a Nostr-only signal
until Phase 3 brings the MoQ/WebTransport transport.
https://claude.ai/code/session_013nVLALALKaHVgHm9u5Cg8D
openmls (via mdk-core) strict-verifies both the FramedContentAuthData
signature and the membership_tag on every inbound PublicMessage from a
Member sender (RFC 9420 §6.1, §6.2). Quartz was framing commits with
empty placeholders for both fields, so any existing member processing a
quartz-originated commit tripped `openmls::ciphersuite: Incompatible
values` (constant-time compare of a zero-length tag against the 32-byte
expected HMAC). That's the reason every interop test that requires B to
process a commit from amy (add-member, rename, remove, leave, promote,
catchup) was failing at step one.
Fix computes both fields correctly:
* capture pre-commit groupContext bytes, membership_key, and
signingPrivateKey before the key schedule / epoch advance
* build FramedContentTBS = version || wire_format || FramedContent ||
serialized_context and SignWithLabel("FramedContentTBS", ..) with
the committer's pre-commit leaf signing key
* build AuthenticatedContentTBM = TBS || signature<V> ||
confirmation_tag<V> and HMAC-SHA256 it with the pre-commit
membership_key
No change to quartz's own processCommit path (it doesn't re-verify these
fields on inbound), so this is a one-way fix for outbound to openmls
peers.
https://claude.ai/code/session_016kAxdp6ubB5CnF9URhCEzP
Five protocol-level fixes and a batch of harness correctness fixes to get
the headless Marmot/Whitenoise interop harness from 1/13 to 5/13 passing
cleanly, with the remaining failures all rooted in wn's per-account
serial event-processor retry backoff (which drops undecryptable
pre-membership commits after several minutes) rather than amy behaviour.
quartz + commons
----------------
* MarmotGroupData: hold CURRENT_VERSION at 2. mdk-core (the Rust MLS
engine used by whitenoise-rs) strict-rejects v3 payloads with
`ExtensionFormatError("Trailing bytes in NostrGroupDataExtension")`
— our v3 welcomes and GCE commits never apply, so every cross-client
group flow dies at welcome processing. We still parse v3 happily on
the way in; we just don't emit it until mdk publishes the
forward-compat fix MIP-01 mandates.
* MarmotManager.updateGroupMetadata: MERGE extensions instead of
REPLACING. RFC 9420 §12.1.7 says GCE proposals blow away the old
extension list; callers that pass only [marmot_group_data] dropped
[required_capabilities], which peers then reject. Preserve every slot
except the one we're updating.
* MarmotManager.createGroup: new optional `initialMetadata` parameter
that bakes MarmotGroupData into epoch-0 GroupContext.extensions
directly. Without it, creators had to publish a pre-membership
"bootstrap" commit that no later joiner could decrypt — each such
peer then burned their retry budget on an undecryptable kind:445
before seeing the real state. Threaded through MlsGroup.create /
MlsGroupManager.createGroup.
* MarmotManager.mlsGroupIdHex: new translation helper so any code
juggling the MIP-01 nostr_group_id (what amy indexes on) and the
MLS GroupContext groupId (what mdk indexes on) can cross-reference
them without reaching into MlsGroupManager directly.
amethyst module
---------------
* Account.leaveMarmotGroup: self-demote before SelfRemove per MIP-01,
and promote a surviving member to admin first if the caller is the
sole admin (otherwise we'd throw "admin depletion"). Matches the
cli/GroupMembershipCommands.leave flow.
cli (amy)
---------
* Context.syncIncoming: don't advance `giftWrapSince` on empty polls
(so the first-ever sync doesn't bump the cursor past every
past-timestamped wrap we've ever been sent), subtract 2 days lookback
when filtering (NIP-59 randomWithTwoDays gift wraps can have any
createdAt in the last 48h), and only advance `groupSince` for groups
we actually received events for.
* Context.syncIncoming: after ingest, if any Welcome consumed a
KeyPackage, rotate and publish a fresh one immediately. MIP-00
requires this — a KP can only be welcomed once and leaving the
consumed one on relays just means later senders invite us with a
bundle we no longer have private keys for.
* Context.resolveGroupId: accept either nostr_group_id (amy's primary
key) or the MLS GroupContext groupId (what wn emits) on every verb
that takes a group id. Wired through GroupAdd/Remove/Leave/Metadata
/Read, Message send/list, and all the await* verbs so harness
scripts never have to juggle both forms for a single group.
* GroupCreateCommand: bake initial metadata into epoch 0 (see the
quartz change above). Dropped the now-redundant bootstrap commit
publish and tightened the JSON output to include `mls_group_id`.
* GroupMembershipCommands.leave: self-demote admin before SelfRemove;
promote an heir if we're the only admin, otherwise the GCE would
deplete admins and the leave aborts.
* MarmotIngest.ingestGiftWrap: unwrap the sealed-rumor layer. NIP-59
wrap is gift-wrap(kind:1059) → seal(kind:13) → rumor; the old code
only unwrapped once and then checked `inner.kind == 444`, which is
always false because inner is actually the seal. Unseal once more
before the Welcome check. This single fix is what unsticks every
amy-side Welcome ingestion.
wn harness patches + scripts
----------------------------
* whitenoise-defaults-env.patch: honour $WHITENOISE_DISCOVERY_RELAYS in
`Relay::defaults()` (release builds otherwise bake damus.io / primal
/ nos.lol into every new account's NIP-65 / Inbox / KeyPackage
lists, which breaks publishing and prevents the inbox subscription
plane from ever reaching an operational state in a sandbox).
* setup.sh: sleep 2s after amy's initial kind:30443 publish so
nostr-rs-relay has a chance to fsync before wn's first targeted
discovery query. Without it wn's `keys check` races the relay's
WAL flush and intermittently returns NotFound.
* lib.sh: peel wn's `{"result": …}` wrapper in `jq_group_id`,
`wait_for_invite`, `wait_for_message`, `wait_for_member`. Post-v0.2
wn `--json` output nests everything under `.result` (and
`groups invites[]` nests further under `.group.mls_group_id`) —
these helpers were still pattern-matching on the flat shape, so
they returned empty strings for a perfectly good response.
* tests-{create,manage,extras}.sh: track both group IDs per test
(amy's nostr + wn's MLS), pass each CLI the id it understands, and
bump the post-commit wait timeouts to 90–120s so wn's exponential
retry backoff has time to work through the pre-membership commits
it can't decrypt and get to the ones it can.
https://claude.ai/code/session_016kAxdp6ubB5CnF9URhCEzP
- MeetingRoomPresenceEvent (kind 10312) now extends BaseReplaceableEvent
instead of BaseAddressableEvent. NIP-53 states this kind is a regular
replaceable event ("presence can only be indicated in one room at a
time"), and LocalCache already routes it through consumeBaseReplaceable.
createAddress/createAddressATag/createAddressTag now take only pubKey.
- Add TagArrayBuilder extensions for streaming (30311), meeting space
(30312) and meeting room (30313) tags, plus build() helpers that assemble
the required tags per NIP-53.
- Add ProofOfAgreement utility to sign and verify the schnorr proof that
NIP-53 places in the 5th element of a participant p-tag
(SHA256 of kind:pubkey:dTag signed by the participant's private key).
Includes LiveActivitiesEvent.hasValidProof() convenience.
On the JVM, Log.d / .i / .w / .e were printing through \`println()\` which
lands on stdout. The amy CLI uses stdout as its JSON contract — any
Marmot/MLS command would restore state, emit a burst of DEBUG lines,
then emit the JSON object, and every caller piping through jq failed
to parse because the first non-empty line was a log line.
Switch to \`System.err.println\` inside PlatformLog.jvm.kt. Conventional
for CLIs (data on stdout, diagnostics on stderr), and the desktop app
already redirects as part of packaging. Callers who want everything
mixed can still \`2>&1\` at the shell level.
Fixes amy marmot group create / group add / message send etc. when
their output is captured by a shell script.
https://claude.ai/code/session_01M6dCKAF5Y1VyHGZPjzwDXq
Five extractions so the Amethyst UI and the new amy CLI share one
implementation for each piece of Marmot/identity behaviour instead of
reimplementing it per platform.
quartz additions:
- MarmotGroupData.bootstrap(..) + withMergedRelays(..): factory for the
metadata shape every inviter stamps on a fresh group and the outbox-
merge rule every admin commit carries forward.
- KeyPackageFetcher: (a) pure fetchRelaysFor union of target kind:10051,
target outbox, my outbox; (b) one-shot fetchKeyPackage; (c) publish-
side resolveKeyPackagePublishRelays.
- resolveUserHexOrNull: single entry point for npub / nprofile / 64-hex
/ NIP-05 identifiers. Uses the existing Nip19Parser + Nip05Client
infra so NIP-05 resolution is live over HTTP.
commons additions:
- MarmotManager.leafIndexOf(..): pubkey -> leaf index lookup used by
every removeMember caller.
- MarmotManager.addMember(nostrGroupId, keyPackageEvent, relays):
convenience overload that lifts the base64 decode into the manager.
- MarmotManager.buildTextMessage(..) returning a TextMessageBundle;
optionally persists the outbound inner event (CLI needs persist,
Amethyst relies on LocalCache loopback).
- MarmotIngest.ingest(event): routes kind:1059 / kind:445 through
unwrap -> processWelcome / processGroupEvent -> persist the decrypted
application message. Deliberately platform-agnostic.
Retrofits:
- Account.addMarmotGroupMember and fetchKeyPackageAndAddMember now take
a KeyPackageEvent directly; Account.keyPackagePublishRelays delegates
to KeyPackageFetcher.
- AccountViewModel.updateMarmotGroupMetadata uses bootstrap + merged.
- AccountViewModel.sendMarmotGroupMessage builds the kind:9 template via
buildTextMessage(persistOwn = false) to avoid double-persisting.
- AccountSessionManager's NIP-05 login branch delegates to
resolveUserHexOrNull; no more hand-rolled Nip05Id / nip05Client.get.
- CLI Context exposes nip05Client and requireUserHex; syncIncoming now
calls MarmotManager.ingest; all command sites use KeyPackageFetcher
+ the shared identifier resolver. Npubs util deleted.
https://claude.ai/code/session_01M6dCKAF5Y1VyHGZPjzwDXq
Answers "can MDK and marmot-ts see and talk to a user whose group was
created on Amethyst?" — yes, after four more spec-conformance fixes:
1. parent_hash chain (RFC 9420 §7.9.2): Amethyst's commit() and
externalJoin() now compute the parent_hash for every parent node on
the committer's direct path and seal the committer's LeafNode with
the correct leaf parent_hash. Amethyst↔Amethyst used to work only
because both sides stored empty parent_hash values; against a
strict peer (ts-mls, openmls) every Amethyst-authored commit was
rejected with "Unable to verify parent hash". processCommit()
now also patches the computed parent_hashes back into its tree so
treeHash() agrees with the sender — otherwise the epoch key
schedule diverges and AEAD tags mismatch.
2. REQUIRED_CAPABILITIES extension type: 0x0002 → 0x0003 per
RFC 9420 §13.3. The old value was ratchet_tree's slot, so Amethyst's
GroupContext.extensions carried a required_capabilities blob
labelled as ratchet_tree, and openmls rejected the GroupInfo
as Malformed (ratchet_tree is not valid in GroupContext). This
completes the extension-ID set from d7114fc (ratchet_tree,
external_pub) now aligned with the IANA registry.
3. verifyParentHash on the receive side now computes the expected
chain top-down from the post-update tree rather than reading
parent_hash fields that applyUpdatePath leaves as empty
placeholders.
4. An explicit parentHash parameter on buildLeafNode so COMMIT-source
leaves include the computed value in their TBS signature.
Test harness — reverse interop (Amethyst → outside world):
- quartz/tools/{mdk,tsmls}-vector-gen/emit-joiner-kp.{rs,mjs}
generate an MDK/openmls and a marmot-ts/ts-mls KeyPackage with
marmot_group_data (0xF2EE) and self_remove (0x000A) advertised in
capabilities so Amethyst's required_capabilities is satisfiable.
- AmethystAuthoredVectorGen.kt (env-var gated JUnit test) takes the
foreign KP, adds it to a fresh Amethyst group, and emits the
Welcome plus three application PrivateMessages as JSON.
- verify-amethyst.{rs,mjs} replay the joiner's private state,
call the foreign MLS library's join + process_message, and
assert the plaintexts match.
Both verifiers now print ALL PASS end-to-end:
* openmls ← Amethyst: joinGroup + 3× process_message ✓
* ts-mls ← Amethyst: joinGroup + 3× processPrivateMessage ✓
https://claude.ai/code/session_01HfHdd5S5rvxUW2ihEpLGJr
marmot-ts (the TypeScript Marmot client — browsers, Node, Bun, Deno)
wraps a completely different MLS implementation: ts-mls. That backend
shares no code with OpenMLS, which is what MDK/whitenoise use, so
Amethyst passing on both is a strong signal that the on-wire MLS layer
is spec-correct rather than coincidentally self-consistent.
Add a Node-based generator under quartz/tools/tsmls-vector-gen/ that
uses ts-mls 2.0.0-rc.10 directly to emit:
- Alice's KeyPackage + Bob's joiner KeyPackage
- Alice's Welcome after add_member + commit
- Bob's init/encryption/signature private keys (PKCS#8 Ed25519 unwrapped
to the raw 32-byte seed for parity with the MDK vector shape)
- Post-join MLS-Exporter("marmot","group-event",32) KAT
- Three application PrivateMessages from Alice → Bob with the
expected plaintexts
TsMlsWelcomeInteropTest mirrors MdkWelcomeInteropTest but consumes the
new vector. It proves:
- KeyPackage self-signature verifies under our Ed25519 + SignContent.
- Amethyst.processWelcome unwraps the group secrets, derives the
welcome_key/nonce, AEAD-decrypts GroupInfo, verifies GroupInfoTBS
with signer_pub, decodes the ratchet tree, and binds the joiner's
leaf — end-to-end.
- Post-join exporter secret matches ts-mls byte-for-byte.
- Amethyst decrypt() parses the RFC 9420 §6.3.1 PrivateMessageContent
framing (application_data<V> + FramedContentAuthData.signature<V> +
zero padding) and verifies the sender's FramedContentTBS signature
against Alice's leaf, for all three ciphertexts.
https://claude.ai/code/session_01HfHdd5S5rvxUW2ihEpLGJr
Two bugs that surfaced together once FramedContentTBS signing started
being checked at decrypt time:
1. Empty-proposal commits (pure forward-secrecy / self-update) were
skipping the UpdatePath entirely, so the sender's commit_secret
fell back to an all-zero vector while a spec-compliant receiver
would derive a real commit_secret from the path — diverging the
epoch key schedule on the very next message. Per RFC 9420 §12.4.1
the path value MUST be populated when the proposal list is empty.
2. When an Update proposal rotates our own signing key, commit()
rebuilds the committer leaf for the UpdatePath and signs it with
signingPrivateKey (the PRE-rotation key) before the post-commit
promotion step swaps signingPrivateKey for pendingSigningKey. The
receiver then stores our leaf with the pre-rotation signature_key,
while we start minting FramedContentTBS signatures with the new
key — every subsequent decrypt fails signature verification. Use
pendingSigningKey when present to seal the UpdatePath leaf.
Un-Ignore the two empty-commit / multi-epoch tests that documented
this divergence (one in MlsGroupLifecycleTest, one in
MlsGroupEdgeCaseTest). Both pass now, as does the existing signing-
key rotation cross-member round-trip test.
https://claude.ai/code/session_01HfHdd5S5rvxUW2ihEpLGJr
MlsGroup.encrypt / decrypt used to pass raw application bytes as the
AEAD plaintext — no PrivateMessageContent framing, no FramedContentTBS
signature, no padding. Amethyst ↔ Amethyst round-trips worked (both
sides skipped it the same way) but every cross-implementation message
was unreadable.
Per RFC 9420 §6.3.1 the AEAD plaintext for an application PrivateMessage
is the serialized PrivateMessageContent:
opaque application_data<V>; // varint-prefixed payload
opaque signature<V>; // FramedContentAuthData.signature
opaque padding[N]; // zero padding (N may be 0)
where the signature is `SignWithLabel(., "FramedContentTBS",
FramedContentTBS)` computed over (version || wire_format ||
FramedContent || GroupContext). This change wires both sides up
end-to-end:
* encrypt now builds FramedContentTBS via the new
buildApplicationFramedContentTbs helper, signs it with the caller's
signature private key, and writes application_data<V> || signature<V>
(zero-length padding) as the AEAD plaintext.
* decrypt parses application_data<V> and signature<V> from the AEAD
plaintext, asserts all remaining bytes are zero padding, rebuilds
FramedContentTBS for the sender's leaf, and verifies the Ed25519
signature against that leaf's signatureKey before returning the
application payload. Non-application PrivateMessages now error
(commit/proposal-inside-PrivateMessage was never correctly handled
by the old raw-bytes path either; leaving that for a follow-up).
Un-Ignore MdkWelcomeInteropTest.testBobDecryptsMdkApplicationMessagesFromAlice
— Amethyst now decrypts and verifies each of the three openmls-authored
application messages against Alice's committer leaf.
https://claude.ai/code/session_01HfHdd5S5rvxUW2ihEpLGJr
Add a Rust generator under quartz/tools/mdk-vector-gen that emits a
Welcome, joiner KeyPackage, committer signer_pub, joiner's three
private keys, and a post-join MLS-Exporter KAT, using the same
openmls 0.8 backend MDK / whitenoise use. Commit its output at
quartz/src/commonTest/resources/mls/mdk-welcome.json.
MdkWelcomeInteropTest consumes the vector and proves Amethyst can:
- parse a KeyPackage OpenMLS authored and verify its self-signature;
- run the joiner's processWelcome end-to-end (HPKE unwrap of group
secrets, welcome-key derivation, AEAD GroupInfo decrypt, ratchet
tree decode, signer leaf lookup, GroupInfoTBS Ed25519 verify);
- derive MLS-Exporter("marmot", "group-event", 32) byte-for-byte
identically to openmls — the exact exporter Marmot uses for the
outer ChaCha20 wrap on kind:445 events.
testBobDecryptsMdkApplicationMessagesFromAlice is @Ignored with a
detailed TODO because Amethyst's MlsGroup.encrypt/decrypt skip the
RFC 9420 §6.3.1 PrivateMessageContent framing (application_data<V> +
FramedContentAuthData + padding). Amethyst ↔ Amethyst works (both
sides omit it) but every cross-implementation PrivateMessage fails.
Tracked as a follow-up.
https://claude.ai/code/session_01HfHdd5S5rvxUW2ihEpLGJr
Three latent spec divergences that only surfaced once we fed the Marmot
stack with a Welcome authored by openmls 0.8 (the MLS backend under MDK
/ whitenoise):
* ratchet_tree extension type was 0x0001 (RFC 9420 §13.3 assigns 0x0001
to application_id; ratchet_tree is 0x0002). Welcomes and GroupInfos
Amethyst emitted were unreadable to any spec-compliant peer, and
Welcomes from peers couldn't be joined because the tree lookup failed.
* external_pub extension type was 0x0003 (that slot is
required_capabilities; external_pub is 0x0004). External commits would
have failed interop the same way.
* sender-data ciphertext_sample used AEAD.Nk (16 B) bytes of the
ciphertext; RFC 9420 §6.3.2 mandates KDF.Nh (32 B). Every
cross-implementation PrivateMessage would fail sender-data AEAD.
Amethyst ↔ Amethyst round-trip tests passed because both sides were
wrong the same way — the existing MlsConformanceTest hard-coded the
wrong extension IDs, so it even asserted the broken invariant.
Conformance test assertions updated to the spec-correct values.
https://claude.ai/code/session_01HfHdd5S5rvxUW2ihEpLGJr
Existing interop tests covered wire-format round-trips but never actually
ran ciphertext from IETF/OpenMLS/mls-rs test vectors through our crypto
code paths. That left two major spec bugs unverified by CI — the wrong
HPKE Welcome-context and the wrong "eae_prk" DHKEM label — because
both sides of every round-trip were wrong the same way.
Tighten coverage with cross-implementation KATs:
CryptoBasicsInteropTest
+ testEncryptWithLabelDecryptsIetfVector: decrypt the IETF
encrypt_with_label vector directly (kem_output + ciphertext), not
just our own round-trip output.
WelcomeInteropTest
+ testX25519Rfc7748Vectors: RFC 7748 §6.1 X25519 KATs.
+ testX25519DhMatchesJavaXdh: compare against JDK 11+ XDH on the IETF
encrypt_with_label priv/kem_output.
+ testWelcomeInitKeyMatchesPrivateKey: init_pub == publicFromPrivate
(init_priv) for every IETF welcome vector.
+ testIetfKeyPackageSelfSignatureVerifies: exercises Ed25519 +
SignContext + LeafNode TLS encoding against OpenMLS output.
+ testIetfWelcomeFullDecryptAndGroupInfoSignature: end-to-end path
through HPKE GroupSecrets decrypt → welcome_key derivation → AEAD
GroupInfo decrypt → GroupInfoTBS Ed25519 verify against signer_pub.
+ testPassiveClientWelcomeDecryptsGroupInfoAndFindsLeaf: same
end-to-end path on passive-client-welcome.json, plus ratchet-tree
leaf lookup and signer-leaf GroupInfo verify. Skips vectors with
external_psks (we don't model PSK secret derivation here) and with
no ratchet tree (delivery-service lookup).
The TreeKEM UpdatePath KAT was considered but needs the full direct
path / copath / resolution / LCA plumbing from MlsGroup.processCommit
to pick which HPKE ciphertext a given leaf should decrypt. Not worth
the duplication in a test — the "UpdatePathNode" HPKE path now has KAT
coverage indirectly via the IETF encrypt_with_label vector above
(same HPKE stack, different label).
https://claude.ai/code/session_01HfHdd5S5rvxUW2ihEpLGJr
RFC 9180 §4.1 specifies DHKEM derives the shared secret as:
eae_prk = LabeledExtract("", "eae_prk", dh)
shared_secret = LabeledExpand(eae_prk, "shared_secret",
kem_context, Nsecret)
Hpke.extractAndExpand was using "shared_secret" for both the Extract and
Expand labels, so every HPKE decapsulation produced a key schedule that
disagreed with every spec-compliant implementation (OpenMLS, MDK/
whitenoise, OpenSSL, BoringSSL, Java XDH). Round-trips within Amethyst
worked because both sides were wrong the same way; cross-implementation
Welcome HPKE decryption always failed with AEAD BAD_DECRYPT.
Add IETF-vector interop tests locking in the fix:
- decrypt the crypto-basics encrypt_with_label KAT end-to-end,
- decrypt GroupSecrets from a passive-client Welcome vector,
- X25519 DH matches RFC 7748 §6.1 and Java XDH on a KAT input,
- init_pub in each Welcome vector matches publicFromPrivate(init_priv).
Drop the stale comment in CryptoBasicsInteropTest that claimed the vector
decryption was impossible due to an "X25519 DH discrepancy" — the DH was
always fine; the eae_prk label was the bug.
https://claude.ai/code/session_01HfHdd5S5rvxUW2ihEpLGJr
RFC 9420 §12.4.3.1 specifies the HPKE context for encrypted_group_secrets
must be the encrypted_group_info field of the Welcome:
encrypted_group_secrets = EncryptWithLabel(init_key, "Welcome",
encrypted_group_info, group_secrets)
Both our buildWelcome and processWelcome were passing an empty context,
which let Amethyst↔Amethyst round-trips work, but made Welcome messages
sent by RFC-compliant implementations (MDK/OpenMLS, used by whitenoise)
fail with BAD_DECRYPT inside the HPKE AEAD open.
https://claude.ai/code/session_01HfHdd5S5rvxUW2ihEpLGJr
Private emojis stored in the encrypted .content of kind 30030 EmojiPackEvents
were honest about being private but useless: they never appeared in the `:`
autocomplete or the reaction menu, even for the pack owner. This wires up
async decryption so pack owners actually see their private emojis everywhere
public ones already show.
- quartz: add EmojiPackEvent.publicEmojis() / privateEmojis(signer) /
allEmojis(signer) helpers, mirroring BookmarkListEvent's public/private
accessor split. privateEmojis() returns null for non-author signers, which
preserves the rule that foreign packs cannot expose private entries.
- commons: EmojiPackState.mergePack becomes suspend (mergePackWithPrivate)
and decrypts inside the existing combineTransform hot path; the StateFlow
consumers (account.emoji.myEmojis → EmojiSuggestionState) get private
entries automatically once decryption resolves. The combiner already runs
on Dispatchers.IO so the suspending decrypt does not block the UI.
- amethyst: RenderEmojiPack uses produceState to seed with the public list
immediately and replace with the merged list once allEmojis(signer)
resolves — keeps the reaction-menu gallery responsive.
- AddEmojiDialog explainer + KDoc no longer claim private emojis are
invisible; OwnedEmojiPacksState.toOwnedEmojiPack uses the new accessors.
- Test EmojiPackEventTest covers public/private/all access including the
foreign-signer case.
https://claude.ai/code/session_01SNG3nj8ZZDChggTsg1qznn
Two kinds of noisy warnings appear every time an invitee or a
restarted client catches up with events still sitting on the relays.
Both are actually expected, per-spec behavior — not errors.
(a) Outer ChaCha20-Poly1305 decrypt fails with "current and N
retained epoch key(s)".
A newly-joined member processes the three kind:445s that
advanced the group to their current epoch (add-me-commit plus
any earlier commits). The commits were outer-encrypted with
keys that predate their Welcome, so they never held them — this
is MLS forward secrecy working as intended. No state on the
receiver's side actually needs to update: they're already at
the post-Welcome epoch.
(b) "NO matching KeyPackageBundle for eventId=…" after app restart.
The gift-wrapped Welcome (kind:1059) is still on the relay and
gets redelivered on every subscription refresh. The client's
`processedEventIds` dedup set is in-memory, so after a restart
the Welcome flows all the way to `processWelcome`, which then
fails to find the KeyPackage bundle because it was consumed and
marked during the first processing.
Both of these fire every time a member opens the app, so the logs
currently look like something is broken even when everything is
behaving correctly.
Fix:
- Add `GroupEventResult.UndecryptableOuterLayer(groupId,
retainedEpochCount)`. `MarmotInboundProcessor.processGroupEvent`
and `applyCommit` now call the new `tryDecryptOuterLayer` (which
returns `ByteArray?` instead of throwing) and surface this
variant when every available key fails. `GroupEventHandler.add`
logs it at DEBUG with a one-liner noting "likely from before our
join".
- Add `WelcomeResult.AlreadyJoined(nostrGroupId)`. When
`MarmotInboundProcessor.processWelcome` is called with a
`hintNostrGroupId` we're already a member of, short-circuit
before the KeyPackageBundle lookup and return `AlreadyJoined`.
`processMarmotWelcomeFlow` logs at DEBUG.
- `MarmotManager.processGroupEvent` treats `UndecryptableOuterLayer`
as a no-op for subscription-timestamp updates (same as
Duplicate/Error/CommitPending), so a unreadable past-epoch event
doesn't move the group's `since` forward.
No behavioral changes beyond logging level and the short-circuit of
a Welcome we already processed (which was throwing and returning
Error before). Existing callers that only care about successful
`Joined` / successful `CommitProcessed` / `ApplicationMessage`
branches are unaffected — the two new variants join the
`Duplicate` / `CommitPending` / `Error` quiet side of the sealed
class.
https://claude.ai/code/session_014zfdNeeKAfU1zGGyFw4bUL
After a group creator restarts the app, their own relay-echoed
kind:445 commit events (add-member, updateGroupExtensions, etc.) stop
being filtered by `inboundProcessor.processedEventIds` — that
in-memory dedup set doesn't survive process death. The echo then:
1. Outer-decrypts via a retained epoch key (we still hold those).
2. Gets parsed as a `PublicMessage` with `contentType = COMMIT`.
3. Flows into `applyCommit` → `groupManager.processCommit` →
`group.processCommit`.
`MlsGroup.processCommit` is not atomic: it runs proposal application,
transcript-hash recomputation, epoch advance, key-schedule derivation,
and SecretTree rebuild **in place** before reaching the final
confirmation-tag check. If that check fails — which it always does
for an already-applied commit, because the tag was computed against
the original epoch's confirmation key and we're now several epochs
past — the exception fires after the mutations, leaving the group
context at a bogus "epoch + 1" with a completely new exporter
secret.
Net result on the wire: the creator's next outbound kind:445 is
encrypted with this rogue exporter secret. Every other member is at
the real epoch, so they all log "Outer decryption failed with current
and N retained epoch key(s)" and no one receives the message. The
creator's own self-echo still decrypts because his sentKeys /
self-state are internally consistent; only external observers are
cut off.
Fix: in `MarmotInboundProcessor.applyCommit`, after parsing the
PublicMessage header but **before** calling `processCommit`, compare
`pubMsg.epoch` with the current local `group.epoch`.
- `pubMsg.epoch < current` → commit is from the past (either our own
already-applied echo or a stale relay replay). Return
`GroupEventResult.Duplicate` and don't touch local state.
- `pubMsg.epoch > current` → we're behind; the inbound pipeline
upstream needs to feed us the intervening commits in order. Return
an error describing the gap without touching local state.
- Otherwise → proceed to `processCommit` as before.
Making `processCommit` itself atomic would be nicer, but is a much
larger refactor touching tree mutations, key-schedule state, and
per-sender ratchets. The epoch check catches the production scenario
at the right boundary — we never call into `processCommit` with
bytes that we know won't apply cleanly.
Regression test:
- `testCreatorDoesNotDoubleApplyOwnCommitAfterRestart`: David creates
a group, adds Eden and Fred, persists. Simulates David's app
restart by rebuilding the `MlsGroupManager` from the store. Then
feeds David's own add-Eden commit back through the inbound
pipeline — exactly what a relay echo would deliver. Asserts the
result is `Duplicate`, and that David's epoch and exporter secret
are unchanged afterwards. Before the fix, his epoch advances by
one and his exporter secret drifts, which is exactly what the
production logs showed.
- `testCreatorCanSendMessageAfterRestart`: save/restore round-trip
sanity check that the restored manager's exporter secret matches
what the members still at the same epoch hold, and that the first
post-restart message from the creator is decryptable by an
un-restarted member.
https://claude.ai/code/session_014zfdNeeKAfU1zGGyFw4bUL
Fred (third member of a three-member group, `leafIndex = 2`) crashed
with `StackOverflowError` on ART / NPE on JVM the moment he tried to
send his first application message. The recursion / null is in
`SecretTree.getNodeSecret`, which assumes `nodeIndex` is always a
direct child of `BinaryTree.parent(nodeIndex)`:
val parentIdx = BinaryTree.parent(nodeIndex, BinaryTree.nodeCount(leafCount))
val parentSecret = getNodeSecret(parentIdx)
val leftIdx = BinaryTree.left(parentIdx)
val rightIdx = BinaryTree.right(parentIdx)
treeSecrets[leftIdx] = leftSecret
treeSecrets[rightIdx] = rightSecret
return treeSecrets[nodeIndex]!!
That invariant holds only for power-of-two trees. For Fred (leafCount
= 3, nodeCount = 5), `BinaryTree.parent(4, 5)` walks via
`parentInRange(5, 5)` and returns **3** (the root) — skipping the
virtual intermediate node 5 that would normally sit between node 4
and the root in a 4-leaf tree. But `left(3) = 1` and `right(3) = 5`;
neither is 4. The cache fills entries 1 and 5 from the root's secret,
node 4's entry is never populated, and the final
`treeSecrets[nodeIndex]!!` throws NPE. On Android the error path
instead re-enters `getNodeSecret` until the stack is exhausted.
The fix walks **down** from the root to the target leaf, picking
left or right at each step based on `targetNode < currentNode`
(holds in left-balanced MLS trees) and using `BinaryTree.left` /
`BinaryTree.right` which correctly descend through virtual
intermediate nodes. The node-level cache is no longer needed —
`getOrInitSender` caches the per-sender ratchet state once, so
re-deriving the leaf secret each time a new sender is initialized
costs a handful of HKDF calls at worst.
Side effects (all in the right direction):
- Re-enabled 6 previously `@Ignore`d tests in
`MlsGroupLifecycleTest` and `MlsGroupEdgeCaseTest` that exercised
exactly the three-member / non-full-tree path this fix addresses:
`testThreeMemberGroup_SequentialAdditions`,
`testExternalJoin_ZaraJoinsViaGroupInfo`,
`testExternalJoin_ExporterSecretsAgree`,
`testSigningKeyRotation_EpochAdvances`,
`testEncryptDecryptAfterSigningKeyRotation`,
`testPskProposal_EpochAdvancesWithPsk`. They now pass.
- `testEmptyCommit_AdvancesEpoch` and
`testMultipleEpochTransitions_EncryptDecryptStillWorks` remain
`@Ignore`d — they hit a **different** pre-existing bug where
Alice's and Bob's exporter secrets diverge after a no-proposal
commit (commit with only an UpdatePath). Unrelated to the
non-full-tree fix; tracked separately. I retagged them with an
explanatory note instead of the old "see
testThreeMemberGroup_SequentialAdditions" reference.
- Fixed an unrelated typo in
`testMultipleEpochTransitions_EncryptDecryptStillWorks` that
asserted `7L` and then `6L` for the same epoch value.
Regression test:
- `testFredEncryptsAfterJoiningThreeMemberGroup` creates a
three-member group via the production path (Alice adds Bob, then
adds Fred; Bob processes the add-Fred commit), then has Fred
encrypt an application message and asserts both Alice and Bob can
decrypt it. Without the fix this throws NPE / stack-overflows at
`encrypt()`.
https://claude.ai/code/session_014zfdNeeKAfU1zGGyFw4bUL
The snapshot/restore machinery added in the previous commit
(ca988b7) to implement MDK-style `stageCommit` + `mergeStagedCommit`
introduced a tree-rebuild path (RatchetTree.decodeTls followed by a
fresh SecretTree construction) that, in production, caused a
recursive StackOverflowError in `SecretTree.getNodeSecret` during
encryption after a second add-member cycle. The unit test flow did
not exercise it and couldn't reproduce.
Keep the underlying fix — outer-encrypt outbound kind:445 commits
with the PRE-commit (epoch-N) exporter secret — but deliver it with
far less machinery:
- `CommitResult.preCommitExporterSecret` now carries the
`MLS-Exporter("marmot", "group-event", 32)` value captured inside
`MlsGroup.commit()` BEFORE any state mutation. It's the key
existing members still at epoch N hold, so the outer kind:445
wrap with it is decryptable to them (RFC 9420 §12.4 + MDK parity).
- `MlsGroup.commit()` captures this exporter once at the top of the
function, threads it into the returned `CommitResult`.
- `MarmotManager.addMember / removeMember / updateGroupMetadata`
read `commitResult.preCommitExporterSecret` and pass it to
`MarmotOutboundProcessor.buildCommitEvent(..., exporterKey=...)`.
The eager `group.addMember` / etc. continue to advance the local
epoch immediately — the same behavior as before the previous
commit, which was proven safe under production load.
Removed:
- `MlsGroup.stageCommit` / `mergeStagedCommit` / `stageAddMember` /
`stageRemoveMember` and their snapshot/restore helpers.
- `MlsGroupSnapshot` / `StagedCommit` types.
- `MlsGroupManager.stageAddMember` / `stageRemoveMember` /
`stageRotateSigningKey` / `stageUpdateGroupExtensions` /
`mergeStagedCommit`.
- `tree` back to `val` in `MlsGroup`.
Kept from the previous commit:
- `MarmotOutboundProcessor.buildCommitEvent(... exporterKey: ByteArray?)`
with default falling back to `groupManager.exporterSecret(...)`.
- `MlsGroupManager.processCommit` retains epoch secrets AFTER
successful apply (was a secondary bug polluting the retention
window with duplicates of the current key when a commit failed).
- `pushRetainedEpoch` helper that accepts a pre-captured
RetainedEpochSecrets, used throughout instead of the old
`retainEpochSecrets(nostrGroupId, group)`.
Test updated:
- `testCreatorCanAddTwoMembersAndAllDecryptFollowingMessage` now
uses the eager `addMember` path and asserts that the member still
at the old epoch can outer-decrypt with the pre-commit key shipped
back via `CommitResult`. Also decrypts Alice's self-echo and
sends multiple follow-up messages — this is the scenario that
triggered the production stack overflow before the revert.
- `testAddMemberExposesPreCommitExporterSecret` (renamed) confirms
`CommitResult.preCommitExporterSecret` equals the exporter the
group had immediately before the call, and that the post-commit
exporter differs from it (proves we actually rotated).
https://claude.ai/code/session_014zfdNeeKAfU1zGGyFw4bUL
Claude