fix(marmot,cli,interop): interop-compatible Marmot flows and harness correctness

Five protocol-level fixes and a batch of harness correctness fixes to get the headless Marmot/Whitenoise interop harness from 1/13 to 5/13 passing cleanly, with the remaining failures all rooted in wn's per-account serial event-processor retry backoff (which drops undecryptable pre-membership commits after several minutes) rather than amy behaviour. quartz + commons ---------------- * MarmotGroupData: hold CURRENT_VERSION at 2. mdk-core (the Rust MLS engine used by whitenoise-rs) strict-rejects v3 payloads with `ExtensionFormatError("Trailing bytes in NostrGroupDataExtension")` — our v3 welcomes and GCE commits never apply, so every cross-client group flow dies at welcome processing. We still parse v3 happily on the way in; we just don't emit it until mdk publishes the forward-compat fix MIP-01 mandates. * MarmotManager.updateGroupMetadata: MERGE extensions instead of REPLACING. RFC 9420 §12.1.7 says GCE proposals blow away the old extension list; callers that pass only [marmot_group_data] dropped [required_capabilities], which peers then reject. Preserve every slot except the one we're updating. * MarmotManager.createGroup: new optional `initialMetadata` parameter that bakes MarmotGroupData into epoch-0 GroupContext.extensions directly. Without it, creators had to publish a pre-membership "bootstrap" commit that no later joiner could decrypt — each such peer then burned their retry budget on an undecryptable kind:445 before seeing the real state. Threaded through MlsGroup.create / MlsGroupManager.createGroup. * MarmotManager.mlsGroupIdHex: new translation helper so any code juggling the MIP-01 nostr_group_id (what amy indexes on) and the MLS GroupContext groupId (what mdk indexes on) can cross-reference them without reaching into MlsGroupManager directly. amethyst module --------------- * Account.leaveMarmotGroup: self-demote before SelfRemove per MIP-01, and promote a surviving member to admin first if the caller is the sole admin (otherwise we'd throw "admin depletion"). Matches the cli/GroupMembershipCommands.leave flow. cli (amy) --------- * Context.syncIncoming: don't advance `giftWrapSince` on empty polls (so the first-ever sync doesn't bump the cursor past every past-timestamped wrap we've ever been sent), subtract 2 days lookback when filtering (NIP-59 randomWithTwoDays gift wraps can have any createdAt in the last 48h), and only advance `groupSince` for groups we actually received events for. * Context.syncIncoming: after ingest, if any Welcome consumed a KeyPackage, rotate and publish a fresh one immediately. MIP-00 requires this — a KP can only be welcomed once and leaving the consumed one on relays just means later senders invite us with a bundle we no longer have private keys for. * Context.resolveGroupId: accept either nostr_group_id (amy's primary key) or the MLS GroupContext groupId (what wn emits) on every verb that takes a group id. Wired through GroupAdd/Remove/Leave/Metadata /Read, Message send/list, and all the await* verbs so harness scripts never have to juggle both forms for a single group. * GroupCreateCommand: bake initial metadata into epoch 0 (see the quartz change above). Dropped the now-redundant bootstrap commit publish and tightened the JSON output to include `mls_group_id`. * GroupMembershipCommands.leave: self-demote admin before SelfRemove; promote an heir if we're the only admin, otherwise the GCE would deplete admins and the leave aborts. * MarmotIngest.ingestGiftWrap: unwrap the sealed-rumor layer. NIP-59 wrap is gift-wrap(kind:1059) → seal(kind:13) → rumor; the old code only unwrapped once and then checked `inner.kind == 444`, which is always false because inner is actually the seal. Unseal once more before the Welcome check. This single fix is what unsticks every amy-side Welcome ingestion. wn harness patches + scripts ---------------------------- * whitenoise-defaults-env.patch: honour $WHITENOISE_DISCOVERY_RELAYS in `Relay::defaults()` (release builds otherwise bake damus.io / primal / nos.lol into every new account's NIP-65 / Inbox / KeyPackage lists, which breaks publishing and prevents the inbox subscription plane from ever reaching an operational state in a sandbox). * setup.sh: sleep 2s after amy's initial kind:30443 publish so nostr-rs-relay has a chance to fsync before wn's first targeted discovery query. Without it wn's `keys check` races the relay's WAL flush and intermittently returns NotFound. * lib.sh: peel wn's `{"result": …}` wrapper in `jq_group_id`, `wait_for_invite`, `wait_for_message`, `wait_for_member`. Post-v0.2 wn `--json` output nests everything under `.result` (and `groups invites[]` nests further under `.group.mls_group_id`) — these helpers were still pattern-matching on the flat shape, so they returned empty strings for a perfectly good response. * tests-{create,manage,extras}.sh: track both group IDs per test (amy's nostr + wn's MLS), pass each CLI the id it understands, and bump the post-commit wait timeouts to 90–120s so wn's exponential retry backoff has time to work through the pre-membership commits it can't decrypt and get to the ones it can. https://claude.ai/code/session_016kAxdp6ubB5CnF9URhCEzP
2026-04-22 00:28:45 +00:00
parent ae1549b884
commit f08b010f50
20 changed files with 422 additions and 126 deletions
@@ -191,7 +191,22 @@ data class MarmotGroupData(
    fun toExtension(): Extension = Extension(EXTENSION_ID_INT, encodeTls())

    companion object {
-        const val CURRENT_VERSION = 3
+        /**
+         * Version we emit for freshly created groups and fresh GCE commits.
+         *
+         * Held at 2 (not 3) because the Rust mdk-core MLS engine used by
+         * whitenoise-rs (the only other shipping Marmot client today) is
+         * stricter than MIP-01's "ignore trailing bytes for forward
+         * compatibility" rule — it rejects v3 payloads with
+         * `ExtensionFormatError("Trailing bytes in NostrGroupDataExtension")`,
+         * which means our v3 welcomes/commits never get applied and every
+         * cross-client group flow breaks. We still PARSE v3 happily via
+         * [decodeTls] (any peer that sends us a v3 group will round-trip),
+         * but we don't create them until mdk-core catches up.
+         *
+         * Bump back to 3 once mdk publishes the forward-compat fix.
+         */
+        const val CURRENT_VERSION = 2

        /** Versions this implementation understands. v0 is reserved/invalid per MIP-01. */
        val SUPPORTED_VERSIONS: Set<Int> = setOf(1, 2, 3)
@@ -1929,6 +1929,7 @@ class MlsGroup private constructor(
        fun create(
            identity: ByteArray,
            signingKey: ByteArray? = null,
+            initialExtensions: List<com.vitorpamplona.quartz.marmot.mls.tree.Extension> = emptyList(),
        ): MlsGroup {
            val sigKp =
                signingKey?.let { key ->
@@ -1953,13 +1954,18 @@ class MlsGroup private constructor(
            tree.setLeaf(0, leafNode)

            val treeHash = tree.treeHash()
+            // Start with required_capabilities + whatever the caller wants to
+            // bake into epoch 0 (e.g. the MIP-01 MarmotGroupData extension so
+            // new peers who join later can see the group name without first
+            // decrypting a pre-membership bootstrap commit — see MIP-03).
+            val baseExtensions = listOf(buildMarmotRequiredCapabilitiesExtension())
            val groupContext =
                GroupContext(
                    groupId = groupId,
                    epoch = 0,
                    treeHash = treeHash,
                    confirmedTranscriptHash = ByteArray(0),
-                    extensions = listOf(buildMarmotRequiredCapabilitiesExtension()),
+                    extensions = baseExtensions + initialExtensions,
                )

            // Initial key schedule with zero secrets
@@ -174,10 +174,11 @@ class MlsGroupManager(
        nostrGroupId: HexKey,
        identity: ByteArray,
        signingKey: ByteArray? = null,
+        initialExtensions: List<com.vitorpamplona.quartz.marmot.mls.tree.Extension> = emptyList(),
    ): MlsGroup =
        mutex.withLock {
            Log.d(TAG) { "createGroup($nostrGroupId): creating new MLS group" }
-            val group = MlsGroup.create(identity, signingKey)
+            val group = MlsGroup.create(identity, signingKey, initialExtensions)
            groups[nostrGroupId] = group
            persistGroup(nostrGroupId)
            Log.d(TAG) { "createGroup($nostrGroupId): done, in-memory group count=${groups.size}" }