From f08b010f50617d77a416a9e1234a6b890f208bbc Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 22 Apr 2026 00:28:45 +0000 Subject: [PATCH] fix(marmot,cli,interop): interop-compatible Marmot flows and harness correctness MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Five protocol-level fixes and a batch of harness correctness fixes to get the headless Marmot/Whitenoise interop harness from 1/13 to 5/13 passing cleanly, with the remaining failures all rooted in wn's per-account serial event-processor retry backoff (which drops undecryptable pre-membership commits after several minutes) rather than amy behaviour. quartz + commons ---------------- * MarmotGroupData: hold CURRENT_VERSION at 2. mdk-core (the Rust MLS engine used by whitenoise-rs) strict-rejects v3 payloads with `ExtensionFormatError("Trailing bytes in NostrGroupDataExtension")` — our v3 welcomes and GCE commits never apply, so every cross-client group flow dies at welcome processing. We still parse v3 happily on the way in; we just don't emit it until mdk publishes the forward-compat fix MIP-01 mandates. * MarmotManager.updateGroupMetadata: MERGE extensions instead of REPLACING. RFC 9420 §12.1.7 says GCE proposals blow away the old extension list; callers that pass only [marmot_group_data] dropped [required_capabilities], which peers then reject. Preserve every slot except the one we're updating. * MarmotManager.createGroup: new optional `initialMetadata` parameter that bakes MarmotGroupData into epoch-0 GroupContext.extensions directly. Without it, creators had to publish a pre-membership "bootstrap" commit that no later joiner could decrypt — each such peer then burned their retry budget on an undecryptable kind:445 before seeing the real state. Threaded through MlsGroup.create / MlsGroupManager.createGroup. * MarmotManager.mlsGroupIdHex: new translation helper so any code juggling the MIP-01 nostr_group_id (what amy indexes on) and the MLS GroupContext groupId (what mdk indexes on) can cross-reference them without reaching into MlsGroupManager directly. amethyst module --------------- * Account.leaveMarmotGroup: self-demote before SelfRemove per MIP-01, and promote a surviving member to admin first if the caller is the sole admin (otherwise we'd throw "admin depletion"). Matches the cli/GroupMembershipCommands.leave flow. cli (amy) --------- * Context.syncIncoming: don't advance `giftWrapSince` on empty polls (so the first-ever sync doesn't bump the cursor past every past-timestamped wrap we've ever been sent), subtract 2 days lookback when filtering (NIP-59 randomWithTwoDays gift wraps can have any createdAt in the last 48h), and only advance `groupSince` for groups we actually received events for. * Context.syncIncoming: after ingest, if any Welcome consumed a KeyPackage, rotate and publish a fresh one immediately. MIP-00 requires this — a KP can only be welcomed once and leaving the consumed one on relays just means later senders invite us with a bundle we no longer have private keys for. * Context.resolveGroupId: accept either nostr_group_id (amy's primary key) or the MLS GroupContext groupId (what wn emits) on every verb that takes a group id. Wired through GroupAdd/Remove/Leave/Metadata /Read, Message send/list, and all the await* verbs so harness scripts never have to juggle both forms for a single group. * GroupCreateCommand: bake initial metadata into epoch 0 (see the quartz change above). Dropped the now-redundant bootstrap commit publish and tightened the JSON output to include `mls_group_id`. * GroupMembershipCommands.leave: self-demote admin before SelfRemove; promote an heir if we're the only admin, otherwise the GCE would deplete admins and the leave aborts. * MarmotIngest.ingestGiftWrap: unwrap the sealed-rumor layer. NIP-59 wrap is gift-wrap(kind:1059) → seal(kind:13) → rumor; the old code only unwrapped once and then checked `inner.kind == 444`, which is always false because inner is actually the seal. Unseal once more before the Welcome check. This single fix is what unsticks every amy-side Welcome ingestion. wn harness patches + scripts ---------------------------- * whitenoise-defaults-env.patch: honour $WHITENOISE_DISCOVERY_RELAYS in `Relay::defaults()` (release builds otherwise bake damus.io / primal / nos.lol into every new account's NIP-65 / Inbox / KeyPackage lists, which breaks publishing and prevents the inbox subscription plane from ever reaching an operational state in a sandbox). * setup.sh: sleep 2s after amy's initial kind:30443 publish so nostr-rs-relay has a chance to fsync before wn's first targeted discovery query. Without it wn's `keys check` races the relay's WAL flush and intermittently returns NotFound. * lib.sh: peel wn's `{"result": …}` wrapper in `jq_group_id`, `wait_for_invite`, `wait_for_message`, `wait_for_member`. Post-v0.2 wn `--json` output nests everything under `.result` (and `groups invites[]` nests further under `.group.mls_group_id`) — these helpers were still pattern-matching on the flat shape, so they returned empty strings for a perfectly good response. * tests-{create,manage,extras}.sh: track both group IDs per test (amy's nostr + wn's MLS), pass each CLI the id it understands, and bump the post-commit wait timeouts to 90–120s so wn's exponential retry backoff has time to work through the pre-membership commits it can't decrypt and get to the ones it can. https://claude.ai/code/session_016kAxdp6ubB5CnF9URhCEzP --- .../vitorpamplona/amethyst/model/Account.kt | 29 +++++++ .../com/vitorpamplona/amethyst/cli/Context.kt | 83 +++++++++++++++++-- .../amethyst/cli/commands/AwaitCommands.kt | 11 +-- .../cli/commands/GroupAddMemberCommand.kt | 2 +- .../cli/commands/GroupCreateCommand.kt | 18 ++-- .../cli/commands/GroupMembershipCommands.kt | 32 ++++++- .../cli/commands/GroupMetadataCommands.kt | 3 +- .../cli/commands/GroupReadCommands.kt | 7 +- .../amethyst/cli/commands/MessageCommands.kt | 4 +- .../amethyst/commons/marmot/MarmotIngest.kt | 19 ++++- .../amethyst/commons/marmot/MarmotManager.kt | 44 +++++++++- .../marmot/mip01Groups/MarmotGroupData.kt | 17 +++- .../quartz/marmot/mls/group/MlsGroup.kt | 8 +- .../marmot/mls/group/MlsGroupManager.kt | 3 +- .../patches/whitenoise-defaults-env.patch | 24 ++++++ tools/marmot-interop/headless/setup.sh | 15 +++- tools/marmot-interop/headless/tests-create.sh | 70 +++++++++++----- tools/marmot-interop/headless/tests-extras.sh | 64 +++++++------- tools/marmot-interop/headless/tests-manage.sh | 77 ++++++++++------- tools/marmot-interop/lib.sh | 18 ++-- 20 files changed, 422 insertions(+), 126 deletions(-) create mode 100644 tools/marmot-interop/headless/patches/whitenoise-defaults-env.patch diff --git a/amethyst/src/main/java/com/vitorpamplona/amethyst/model/Account.kt b/amethyst/src/main/java/com/vitorpamplona/amethyst/model/Account.kt index c6a4b6861..878e7c569 100644 --- a/amethyst/src/main/java/com/vitorpamplona/amethyst/model/Account.kt +++ b/amethyst/src/main/java/com/vitorpamplona/amethyst/model/Account.kt @@ -2260,6 +2260,14 @@ class Account( /** * Leave a Marmot MLS group. * Publishes the SelfRemove proposal and removes local state. + * + * MIP-01/MIP-03: admins MUST first publish a GroupContextExtensions + * commit dropping themselves from `admin_pubkeys` before issuing a + * SelfRemove proposal. Without that, [MlsGroup.selfRemove] throws + * `IllegalStateException("Admin must self-demote via GroupContextExtensions + * before SelfRemove (MIP-01)")` and the leave aborts. Demote commit and + * SelfRemove proposal both go to the same group relays, demote first so + * peers apply it before they see the SelfRemove. */ suspend fun leaveMarmotGroup( nostrGroupId: HexKey, @@ -2268,6 +2276,27 @@ class Account( val manager = marmotManager ?: return if (!isWriteable()) return + val metadata = manager.groupMetadata(nostrGroupId) + if (metadata != null && metadata.adminPubkeys.contains(signer.pubKey)) { + val remaining = metadata.adminPubkeys.filter { it != signer.pubKey }.toMutableList() + // MIP-03 also rejects any GCE commit that leaves the group with zero + // admins. If we're the only one, promote an arbitrary non-self + // member to admin before stepping down. + if (remaining.isEmpty()) { + val heir = + manager + .memberPubkeys(nostrGroupId) + .map { it.pubkey } + .firstOrNull { it != signer.pubKey } + if (heir != null) remaining.add(heir) + } + if (remaining.isNotEmpty()) { + val demoted = metadata.copy(adminPubkeys = remaining) + val demoteCommit = manager.updateGroupMetadata(nostrGroupId, demoted) + client.publish(demoteCommit.signedEvent, groupRelays) + } + } + val outbound = manager.leaveGroup(nostrGroupId) client.publish(outbound.signedEvent, groupRelays) } diff --git a/cli/src/main/kotlin/com/vitorpamplona/amethyst/cli/Context.kt b/cli/src/main/kotlin/com/vitorpamplona/amethyst/cli/Context.kt index 638376795..f625c8a93 100644 --- a/cli/src/main/kotlin/com/vitorpamplona/amethyst/cli/Context.kt +++ b/cli/src/main/kotlin/com/vitorpamplona/amethyst/cli/Context.kt @@ -219,14 +219,26 @@ class Context( * - kind:445 group events per active group → feed into inbound processor * * Incrementally advances the `since` cursors in [state] so the next run - * only asks relays for newer events. + * only asks relays for newer events. Two wrinkles: + * + * 1. NIP-59 gift wraps are published with a random-past `created_at` + * (see [com.vitorpamplona.quartz.utils.TimeUtils.randomWithTwoDays]) + * so a newly-published wrap can trivially have `created_at` earlier + * than the last cursor we saw. To avoid silently dropping such wraps + * we always subtract a 2-day lookback window from the gift-wrap + * `since`, and dedup is handled inside [MarmotInboundProcessor]. + * 2. We only advance the on-disk cursor when events actually arrive. + * Snapping an empty sync up to "now" on the first invocation would + * make every later `since` query skip any past-dated wrap or 445. */ suspend fun syncIncoming(timeoutMs: Long = 8_000) { val inbox = inboxRelays().ifEmpty { anyRelays() } val gwSince = state.giftWrapSince + val gwFilterSince = + gwSince?.let { (it - GIFT_WRAP_LOOKBACK_SECS).coerceAtLeast(0L) } val gwFilter = - if (gwSince != null) { - MarmotFilters.giftWrapsForUserSince(identity.pubKeyHex, gwSince) + if (gwFilterSince != null) { + MarmotFilters.giftWrapsForUserSince(identity.pubKeyHex, gwFilterSince) } else { MarmotFilters.giftWrapsForUser(identity.pubKeyHex) } @@ -255,10 +267,11 @@ class Context( if (filterMap.isEmpty()) return val events = drain(filterMap, timeoutMs) - val now = System.currentTimeMillis() / 1000 var maxGwSeen = gwSince ?: 0L val maxGroupSeen = perGroupFilters.keys.associateWith { state.groupSince[it] ?: 0L }.toMutableMap() + var sawGiftWrap = false + val sawGroupEvent = mutableSetOf() for ((relay, event) in events) { // All the MLS/NIP-59 decryption + persistence lives in MarmotIngest — @@ -268,21 +281,70 @@ class Context( when (event.kind) { GiftWrapEvent.KIND -> { + sawGiftWrap = true if (event.createdAt > maxGwSeen) maxGwSeen = event.createdAt } GroupEvent.KIND -> { val gid = (event as? GroupEvent)?.groupId() ?: continue + sawGroupEvent.add(gid) val prev = maxGroupSeen[gid] ?: 0L if (event.createdAt > prev) maxGroupSeen[gid] = event.createdAt } } } - state.giftWrapSince = if (maxGwSeen > 0) maxGwSeen else now - for ((gid, seen) in maxGroupSeen) { - state.groupSince[gid] = if (seen > 0) seen else now + if (sawGiftWrap && maxGwSeen > 0) { + state.giftWrapSince = maxGwSeen } + for (gid in sawGroupEvent) { + val seen = maxGroupSeen[gid] ?: continue + if (seen > 0) state.groupSince[gid] = seen + } + + // If any welcome we processed consumed a KeyPackage, MIP-00 requires + // us to immediately publish a replacement (a KP can only be used for + // ONE welcome; leaving the old one on relays lets a second sender + // invite us with a bundle we no longer have private keys for). The + // Amethyst UI handles this via its own rotation scheduler; the CLI + // has no scheduler, so we rotate inline right after sync. + if (marmot.needsKeyPackageRotation()) { + try { + val kpRelays = keyPackageRelays().ifEmpty { outboxRelays() }.ifEmpty { anyRelays() } + if (kpRelays.isNotEmpty()) { + val rotated = marmot.rotateConsumedKeyPackages(kpRelays.toList()) + for (event in rotated) { + publish(event, kpRelays) + System.err.println("[cli] rotated KeyPackage → ${event.id.take(8)} on ${kpRelays.size} relay(s)") + } + } + } catch (e: Exception) { + System.err.println("[cli] key-package rotation failed: ${e.message}") + } + } + } + + /** + * Resolve a group identifier given on the CLI to the nostr_group_id that + * amy's [MarmotManager] indexes on. + * + * amy internally keys everything off MIP-01's `nostr_group_id`. whitenoise + * (and every other mdk consumer) keys off the MLS `GroupContext.groupId` — + * a separate 32-byte random value stamped at group creation. Cross-client + * scripts therefore wind up juggling both ids, and it's very easy to pass + * the wrong one to amy. Rather than make every caller translate, we accept + * either format and resolve here: + * 1. If the input is an active nostr_group_id, use it unchanged. + * 2. Otherwise scan active groups for one whose MLS groupId matches. + * 3. Otherwise return the input unchanged (so the caller still gets a + * sensible `not_member` response rather than a silent mismatch). + */ + fun resolveGroupId(input: HexKey): HexKey { + if (marmot.isMember(input)) return input + val normalized = input.lowercase() + return marmot.activeGroupIds().firstOrNull { nostrId -> + marmot.mlsGroupIdHex(nostrId)?.lowercase() == normalized + } ?: input } fun marmotGroupRelays(nostrGroupId: HexKey): Set { @@ -303,6 +365,13 @@ class Context( } companion object { + /** + * Lookback applied to the gift-wrap `since` filter to compensate for + * NIP-59's randomised-past `created_at`. 2 days matches + * [com.vitorpamplona.quartz.utils.TimeUtils.randomWithTwoDays]. + */ + private const val GIFT_WRAP_LOOKBACK_SECS: Long = 2L * 24 * 60 * 60 + /** Build a Context but require an identity to already exist — most commands can't run without one. */ fun open(dataDir: DataDir): Context { val identity = diff --git a/cli/src/main/kotlin/com/vitorpamplona/amethyst/cli/commands/AwaitCommands.kt b/cli/src/main/kotlin/com/vitorpamplona/amethyst/cli/commands/AwaitCommands.kt index 35358b75f..a741a536b 100644 --- a/cli/src/main/kotlin/com/vitorpamplona/amethyst/cli/commands/AwaitCommands.kt +++ b/cli/src/main/kotlin/com/vitorpamplona/amethyst/cli/commands/AwaitCommands.kt @@ -108,6 +108,7 @@ object AwaitCommands { Json.writeLine( mapOf( "group_id" to match, + "mls_group_id" to ctx.marmot.mlsGroupIdHex(match), "name" to (ctx.marmot.groupMetadata(match)?.name ?: ""), "epoch" to ctx.marmot.groupEpoch(match), ), @@ -127,7 +128,7 @@ object AwaitCommands { rest: Array, ): Int = pollCondition(dataDir, rest, "await member ", targetIdx = 1) { ctx, rawArgs -> - val gid = rawArgs[0] + val gid = ctx.resolveGroupId(rawArgs[0]) val target = ctx.requireUserHex(rawArgs[1]) if (!ctx.marmot.isMember(gid)) { null @@ -143,7 +144,7 @@ object AwaitCommands { rest: Array, ): Int = pollCondition(dataDir, rest, "await admin ", targetIdx = 1) { ctx, rawArgs -> - val gid = rawArgs[0] + val gid = ctx.resolveGroupId(rawArgs[0]) val target = ctx.requireUserHex(rawArgs[1]) if (!ctx.marmot.isMember(gid)) { null @@ -163,13 +164,13 @@ object AwaitCommands { rest: Array, ): Int { if (rest.isEmpty()) return Json.error("bad_args", "await rename --name ") - val gid = rest[0] val args = Args(rest.drop(1).toTypedArray()) val wantedName = args.requireFlag("name") val timeoutSecs = args.longFlag("timeout", 30) val ctx = Context.open(dataDir) try { ctx.prepare() + val gid = ctx.resolveGroupId(rest[0]) val deadline = System.currentTimeMillis() + timeoutSecs * 1000 while (System.currentTimeMillis() < deadline) { ctx.syncIncoming(timeoutMs = 3_000) @@ -191,13 +192,13 @@ object AwaitCommands { rest: Array, ): Int { if (rest.isEmpty()) return Json.error("bad_args", "await epoch --min N") - val gid = rest[0] val args = Args(rest.drop(1).toTypedArray()) val min = args.longFlag("min", 1) val timeoutSecs = args.longFlag("timeout", 30) val ctx = Context.open(dataDir) try { ctx.prepare() + val gid = ctx.resolveGroupId(rest[0]) val deadline = System.currentTimeMillis() + timeoutSecs * 1000 while (System.currentTimeMillis() < deadline) { ctx.syncIncoming(timeoutMs = 3_000) @@ -219,13 +220,13 @@ object AwaitCommands { rest: Array, ): Int { if (rest.isEmpty()) return Json.error("bad_args", "await message --match STRING") - val gid = rest[0] val args = Args(rest.drop(1).toTypedArray()) val needle = args.requireFlag("match") val timeoutSecs = args.longFlag("timeout", 30) val ctx = Context.open(dataDir) try { ctx.prepare() + val gid = ctx.resolveGroupId(rest[0]) val deadline = System.currentTimeMillis() + timeoutSecs * 1000 while (System.currentTimeMillis() < deadline) { ctx.syncIncoming(timeoutMs = 3_000) diff --git a/cli/src/main/kotlin/com/vitorpamplona/amethyst/cli/commands/GroupAddMemberCommand.kt b/cli/src/main/kotlin/com/vitorpamplona/amethyst/cli/commands/GroupAddMemberCommand.kt index be96d891b..89e1650a6 100644 --- a/cli/src/main/kotlin/com/vitorpamplona/amethyst/cli/commands/GroupAddMemberCommand.kt +++ b/cli/src/main/kotlin/com/vitorpamplona/amethyst/cli/commands/GroupAddMemberCommand.kt @@ -37,10 +37,10 @@ object GroupAddMemberCommand { rest: Array, ): Int { if (rest.size < 2) return Json.error("bad_args", "group add [ ...]") - val gid = rest[0] val ctx = Context.open(dataDir) try { ctx.prepare() + val gid = ctx.resolveGroupId(rest[0]) ctx.syncIncoming() if (!ctx.marmot.isMember(gid)) return Json.error("not_member", gid) diff --git a/cli/src/main/kotlin/com/vitorpamplona/amethyst/cli/commands/GroupCreateCommand.kt b/cli/src/main/kotlin/com/vitorpamplona/amethyst/cli/commands/GroupCreateCommand.kt index 35c6d214f..47cd31377 100644 --- a/cli/src/main/kotlin/com/vitorpamplona/amethyst/cli/commands/GroupCreateCommand.kt +++ b/cli/src/main/kotlin/com/vitorpamplona/amethyst/cli/commands/GroupCreateCommand.kt @@ -40,9 +40,12 @@ object GroupCreateCommand { ctx.prepare() val gid = RandomInstance.bytes(32).toHexKey() - ctx.marmot.createGroup(gid) - - // Stamp initial metadata via the shared factory so UI + CLI stay byte-identical. + // Stamp initial metadata via the shared factory so UI + CLI stay + // byte-identical. Bake the MarmotGroupData extension into the + // epoch-0 GroupContext directly (see `MarmotManager.createGroup`) + // so later invitees receive a pre-populated group from the + // welcome and never have to chase an undecryptable bootstrap + // commit that predates their membership. val outboxUrls = ctx.outboxRelays().map { it.url } val metadata = MarmotGroupData.bootstrap( @@ -51,19 +54,14 @@ object GroupCreateCommand { outboxRelays = outboxUrls, name = name, ) - val commit = ctx.marmot.updateGroupMetadata(gid, metadata) - - // Group relays == what the metadata carries, which on first commit is our outbox. - val targets = ctx.outboxRelays() - val ack = ctx.publish(commit.signedEvent, targets) + ctx.marmot.createGroup(gid, initialMetadata = metadata) Json.writeLine( mapOf( "group_id" to gid, + "mls_group_id" to ctx.marmot.mlsGroupIdHex(gid), "name" to name, "epoch" to ctx.marmot.groupEpoch(gid), - "commit_event_id" to commit.signedEvent.id, - "published_to" to ack.filterValues { it }.keys.map { it.url }, ), ) return 0 diff --git a/cli/src/main/kotlin/com/vitorpamplona/amethyst/cli/commands/GroupMembershipCommands.kt b/cli/src/main/kotlin/com/vitorpamplona/amethyst/cli/commands/GroupMembershipCommands.kt index a67cc62e8..2e7d84c52 100644 --- a/cli/src/main/kotlin/com/vitorpamplona/amethyst/cli/commands/GroupMembershipCommands.kt +++ b/cli/src/main/kotlin/com/vitorpamplona/amethyst/cli/commands/GroupMembershipCommands.kt @@ -30,10 +30,10 @@ object GroupMembershipCommands { rest: Array, ): Int { if (rest.size < 2) return Json.error("bad_args", "group remove ") - val gid = rest[0] val ctx = Context.open(dataDir) try { ctx.prepare() + val gid = ctx.resolveGroupId(rest[0]) val target = ctx.requireUserHex(rest[1]) ctx.syncIncoming() if (!ctx.marmot.isMember(gid)) return Json.error("not_member", gid) @@ -66,18 +66,46 @@ object GroupMembershipCommands { rest: Array, ): Int { if (rest.isEmpty()) return Json.error("bad_args", "group leave ") - val gid = rest[0] val ctx = Context.open(dataDir) try { ctx.prepare() + val gid = ctx.resolveGroupId(rest[0]) if (!ctx.marmot.isMember(gid)) return Json.error("not_member", gid) val targets = ctx.marmotGroupRelays(gid).ifEmpty { ctx.outboxRelays() } + + // MIP-01/MIP-03: members listed in `admin_pubkeys` MUST NOT issue + // a SelfRemove proposal before first publishing a GCE that drops + // themselves from the admin list (MlsGroup.selfRemove enforces + // this with `check(!isLocalAdmin())`), and that same GCE MUST NOT + // leave the group with zero admins (admin depletion). If we're + // the only admin, hand admin to another member first. + val demoteEventId: String? = + ctx.marmot.groupMetadata(gid)?.let { metadata -> + if (!metadata.adminPubkeys.contains(ctx.identity.pubKeyHex)) return@let null + + val newAdmins = metadata.adminPubkeys.filter { it != ctx.identity.pubKeyHex }.toMutableList() + if (newAdmins.isEmpty()) { + val heir = + ctx.marmot + .memberPubkeys(gid) + .map { it.pubkey } + .firstOrNull { it != ctx.identity.pubKeyHex } + ?: return@let null // solo group — skip demote, let MLS state cleanup handle it + newAdmins.add(heir) + } + val demoted = metadata.copy(adminPubkeys = newAdmins) + val demoteCommit = ctx.marmot.updateGroupMetadata(gid, demoted) + ctx.publish(demoteCommit.signedEvent, targets) + demoteCommit.signedEvent.id + } + val outbound = ctx.marmot.leaveGroup(gid) val ack = ctx.publish(outbound.signedEvent, targets) Json.writeLine( mapOf( "group_id" to gid, + "self_demote_event_id" to demoteEventId, "proposal_event_id" to outbound.signedEvent.id, "published_to" to ack.filterValues { it }.keys.map { it.url }, ), diff --git a/cli/src/main/kotlin/com/vitorpamplona/amethyst/cli/commands/GroupMetadataCommands.kt b/cli/src/main/kotlin/com/vitorpamplona/amethyst/cli/commands/GroupMetadataCommands.kt index 0fb3fa106..4afdb7f54 100644 --- a/cli/src/main/kotlin/com/vitorpamplona/amethyst/cli/commands/GroupMetadataCommands.kt +++ b/cli/src/main/kotlin/com/vitorpamplona/amethyst/cli/commands/GroupMetadataCommands.kt @@ -66,12 +66,13 @@ object GroupMetadataCommands { private suspend fun edit( dataDir: DataDir, - gid: HexKey, + rawGid: HexKey, mutate: suspend (Context, MarmotGroupData) -> MarmotGroupData, ): Int { val ctx = Context.open(dataDir) try { ctx.prepare() + val gid = ctx.resolveGroupId(rawGid) ctx.syncIncoming() if (!ctx.marmot.isMember(gid)) return Json.error("not_member", gid) val outboxUrls = ctx.outboxRelays().map { it.url } diff --git a/cli/src/main/kotlin/com/vitorpamplona/amethyst/cli/commands/GroupReadCommands.kt b/cli/src/main/kotlin/com/vitorpamplona/amethyst/cli/commands/GroupReadCommands.kt index 9ac6b78f7..3337204cb 100644 --- a/cli/src/main/kotlin/com/vitorpamplona/amethyst/cli/commands/GroupReadCommands.kt +++ b/cli/src/main/kotlin/com/vitorpamplona/amethyst/cli/commands/GroupReadCommands.kt @@ -56,10 +56,10 @@ object GroupReadCommands { rest: Array, ): Int { if (rest.isEmpty()) return Json.error("bad_args", "group show ") - val gid = rest[0] val ctx = Context.open(dataDir) try { ctx.prepare() + val gid = ctx.resolveGroupId(rest[0]) ctx.syncIncoming() if (!ctx.marmot.isMember(gid)) return Json.error("not_member", gid) val meta = ctx.marmot.groupMetadata(gid) @@ -70,6 +70,7 @@ object GroupReadCommands { Json.writeLine( mapOf( "group_id" to gid, + "mls_group_id" to ctx.marmot.mlsGroupIdHex(gid), "name" to (meta?.name ?: ""), "description" to (meta?.description ?: ""), "epoch" to ctx.marmot.groupEpoch(gid), @@ -90,10 +91,10 @@ object GroupReadCommands { rest: Array, ): Int { if (rest.isEmpty()) return Json.error("bad_args", "group members ") - val gid = rest[0] val ctx = Context.open(dataDir) try { ctx.prepare() + val gid = ctx.resolveGroupId(rest[0]) ctx.syncIncoming() if (!ctx.marmot.isMember(gid)) return Json.error("not_member", gid) val members = @@ -112,10 +113,10 @@ object GroupReadCommands { rest: Array, ): Int { if (rest.isEmpty()) return Json.error("bad_args", "group admins ") - val gid = rest[0] val ctx = Context.open(dataDir) try { ctx.prepare() + val gid = ctx.resolveGroupId(rest[0]) ctx.syncIncoming() if (!ctx.marmot.isMember(gid)) return Json.error("not_member", gid) val m = ctx.marmot.groupMetadata(gid) diff --git a/cli/src/main/kotlin/com/vitorpamplona/amethyst/cli/commands/MessageCommands.kt b/cli/src/main/kotlin/com/vitorpamplona/amethyst/cli/commands/MessageCommands.kt index f2f2a9fcd..5cc5bf9d1 100644 --- a/cli/src/main/kotlin/com/vitorpamplona/amethyst/cli/commands/MessageCommands.kt +++ b/cli/src/main/kotlin/com/vitorpamplona/amethyst/cli/commands/MessageCommands.kt @@ -45,11 +45,11 @@ object MessageCommands { rest: Array, ): Int { if (rest.size < 2) return Json.error("bad_args", "message send ") - val gid = rest[0] val text = rest[1] val ctx = Context.open(dataDir) try { ctx.prepare() + val gid = ctx.resolveGroupId(rest[0]) ctx.syncIncoming() if (!ctx.marmot.isMember(gid)) return Json.error("not_member", gid) @@ -77,12 +77,12 @@ object MessageCommands { rest: Array, ): Int { if (rest.isEmpty()) return Json.error("bad_args", "message list ") - val gid = rest[0] val args = Args(rest.drop(1).toTypedArray()) val limit = args.intFlag("limit", Int.MAX_VALUE) val ctx = Context.open(dataDir) try { ctx.prepare() + val gid = ctx.resolveGroupId(rest[0]) ctx.syncIncoming() if (!ctx.marmot.isMember(gid)) return Json.error("not_member", gid) diff --git a/commons/src/commonMain/kotlin/com/vitorpamplona/amethyst/commons/marmot/MarmotIngest.kt b/commons/src/commonMain/kotlin/com/vitorpamplona/amethyst/commons/marmot/MarmotIngest.kt index 2caec99ab..870c75b52 100644 --- a/commons/src/commonMain/kotlin/com/vitorpamplona/amethyst/commons/marmot/MarmotIngest.kt +++ b/commons/src/commonMain/kotlin/com/vitorpamplona/amethyst/commons/marmot/MarmotIngest.kt @@ -27,6 +27,7 @@ import com.vitorpamplona.quartz.marmot.mip02Welcome.WelcomeEvent import com.vitorpamplona.quartz.marmot.mip03GroupMessages.GroupEvent import com.vitorpamplona.quartz.nip01Core.core.Event import com.vitorpamplona.quartz.nip01Core.core.HexKey +import com.vitorpamplona.quartz.nip59Giftwrap.seals.SealedRumorEvent import com.vitorpamplona.quartz.nip59Giftwrap.wraps.GiftWrapEvent /** @@ -95,11 +96,23 @@ suspend fun MarmotManager.ingest(event: Event): MarmotIngestResult = private suspend fun MarmotManager.ingestGiftWrap(wrap: GiftWrapEvent): MarmotIngestResult = try { - val inner = wrap.unwrapOrNull(signer) ?: return MarmotIngestResult.Ignored - if (!MarmotInboundProcessor.isWelcomeEvent(inner) || inner !is WelcomeEvent) { + // NIP-59 wraps contain TWO encryption layers: + // kind:1059 gift wrap → kind:13 sealed rumor → the rumor itself. + // `GiftWrapEvent.unwrapOrNull` only peels the outer layer; when the + // result is a [SealedRumorEvent] we must unseal it to reach the + // kind:444 Welcome rumor. The old code checked `isWelcomeEvent` on + // the seal (kind:13) and always took the Ignored branch, which is + // why every inbound Welcome was silently dropped by the CLI and by + // any non-Amethyst consumer. + val rumor = + when (val inner = wrap.unwrapOrNull(signer) ?: return MarmotIngestResult.Ignored) { + is SealedRumorEvent -> inner.unsealOrNull(signer) ?: return MarmotIngestResult.Ignored + else -> inner + } + if (!MarmotInboundProcessor.isWelcomeEvent(rumor) || rumor !is WelcomeEvent) { return MarmotIngestResult.Ignored } - when (val result = processWelcome(inner, inner.nostrGroupId())) { + when (val result = processWelcome(rumor, rumor.nostrGroupId())) { is WelcomeResult.Joined -> { MarmotIngestResult.JoinedGroup( nostrGroupId = result.nostrGroupId, diff --git a/commons/src/commonMain/kotlin/com/vitorpamplona/amethyst/commons/marmot/MarmotManager.kt b/commons/src/commonMain/kotlin/com/vitorpamplona/amethyst/commons/marmot/MarmotManager.kt index cac1eed8e..6f76762f6 100644 --- a/commons/src/commonMain/kotlin/com/vitorpamplona/amethyst/commons/marmot/MarmotManager.kt +++ b/commons/src/commonMain/kotlin/com/vitorpamplona/amethyst/commons/marmot/MarmotManager.kt @@ -262,11 +262,24 @@ class MarmotManager( /** * Create a new MLS group. + * + * When [initialMetadata] is non-null it is baked into epoch 0's + * GroupContext.extensions. Later joiners' welcomes therefore carry the + * group name / admin list / relays from the get-go and no separate + * "bootstrap commit" needs to be published. The bootstrap-commit path + * works in theory, but it produces a kind:445 encrypted with epoch 0's + * exporter secret that no post-membership peer (amethyst or wn) has, + * so each such peer wastes their commit-retry budget on an + * undecryptable event before processing the real state. */ - suspend fun createGroup(nostrGroupId: HexKey): HexKey { + suspend fun createGroup( + nostrGroupId: HexKey, + initialMetadata: MarmotGroupData? = null, + ): HexKey { Log.d("MarmotManager") { "createGroup($nostrGroupId): by ${signer.pubKey.take(8)}…" } val identity = signer.pubKey.hexToByteArray() - groupManager.createGroup(nostrGroupId, identity) + val extras = initialMetadata?.let { listOf(it.toExtension()) } ?: emptyList() + groupManager.createGroup(nostrGroupId, identity, initialExtensions = extras) subscriptionManager.subscribeGroup(nostrGroupId) Log.d("MarmotManager") { "createGroup($nostrGroupId): persisted and subscribed" } return nostrGroupId @@ -346,13 +359,24 @@ class MarmotManager( /** * Update group metadata (name, description, etc.) via a GroupContextExtensions proposal. * Creates a GCE proposal, commits it, and returns the commit event to publish. + * + * RFC 9420 §12.1.7: a GroupContextExtensions proposal REPLACES the entire + * extension list in GroupContext. Peers (notably mdk-core / whitenoise-rs) + * reject welcomes whose GroupContext is missing the required-capabilities + * extension, and strip metadata from groups whose context lacks the + * MarmotGroupData extension. We therefore preserve every other existing + * extension and overwrite only the slot we actually want to update. */ suspend fun updateGroupMetadata( nostrGroupId: HexKey, metadata: MarmotGroupData, ): OutboundGroupEvent { - val commitResult = - groupManager.updateGroupExtensions(nostrGroupId, listOf(metadata.toExtension())) + val group = + groupManager.getGroup(nostrGroupId) + ?: throw IllegalStateException("Not a member of group $nostrGroupId") + val preserved = group.extensions.filter { it.extensionType != MarmotGroupData.EXTENSION_ID_INT } + val merged = preserved + metadata.toExtension() + val commitResult = groupManager.updateGroupExtensions(nostrGroupId, merged) val commitEvent = outboundProcessor.buildCommitEvent( nostrGroupId = nostrGroupId, @@ -480,6 +504,18 @@ class MarmotManager( */ fun groupEpoch(nostrGroupId: HexKey): Long? = groupManager.getGroup(nostrGroupId)?.epoch + /** + * Hex-encoded MLS group id for the group keyed by [nostrGroupId], or null if + * that group is not locally known. + * + * Interop note: whitenoise-rs (and every mdk consumer) indexes groups by the + * MLS GroupContext's groupId, NOT the MIP-01 nostr_group_id that we use as + * the primary key internally. When a harness or external caller needs to + * cross-reference a group with another client (e.g. the interop harness + * calling `wn messages list `), it needs this translation. + */ + fun mlsGroupIdHex(nostrGroupId: HexKey): HexKey? = groupManager.getGroup(nostrGroupId)?.groupId?.toHexKey() + /** * Resolve the MLS leaf index for a member by Nostr pubkey, or null if that * pubkey isn't currently in the group. diff --git a/quartz/src/commonMain/kotlin/com/vitorpamplona/quartz/marmot/mip01Groups/MarmotGroupData.kt b/quartz/src/commonMain/kotlin/com/vitorpamplona/quartz/marmot/mip01Groups/MarmotGroupData.kt index 142b8fd5d..78213a394 100644 --- a/quartz/src/commonMain/kotlin/com/vitorpamplona/quartz/marmot/mip01Groups/MarmotGroupData.kt +++ b/quartz/src/commonMain/kotlin/com/vitorpamplona/quartz/marmot/mip01Groups/MarmotGroupData.kt @@ -191,7 +191,22 @@ data class MarmotGroupData( fun toExtension(): Extension = Extension(EXTENSION_ID_INT, encodeTls()) companion object { - const val CURRENT_VERSION = 3 + /** + * Version we emit for freshly created groups and fresh GCE commits. + * + * Held at 2 (not 3) because the Rust mdk-core MLS engine used by + * whitenoise-rs (the only other shipping Marmot client today) is + * stricter than MIP-01's "ignore trailing bytes for forward + * compatibility" rule — it rejects v3 payloads with + * `ExtensionFormatError("Trailing bytes in NostrGroupDataExtension")`, + * which means our v3 welcomes/commits never get applied and every + * cross-client group flow breaks. We still PARSE v3 happily via + * [decodeTls] (any peer that sends us a v3 group will round-trip), + * but we don't create them until mdk-core catches up. + * + * Bump back to 3 once mdk publishes the forward-compat fix. + */ + const val CURRENT_VERSION = 2 /** Versions this implementation understands. v0 is reserved/invalid per MIP-01. */ val SUPPORTED_VERSIONS: Set = setOf(1, 2, 3) diff --git a/quartz/src/commonMain/kotlin/com/vitorpamplona/quartz/marmot/mls/group/MlsGroup.kt b/quartz/src/commonMain/kotlin/com/vitorpamplona/quartz/marmot/mls/group/MlsGroup.kt index da84e71d6..f2dbd0b81 100644 --- a/quartz/src/commonMain/kotlin/com/vitorpamplona/quartz/marmot/mls/group/MlsGroup.kt +++ b/quartz/src/commonMain/kotlin/com/vitorpamplona/quartz/marmot/mls/group/MlsGroup.kt @@ -1929,6 +1929,7 @@ class MlsGroup private constructor( fun create( identity: ByteArray, signingKey: ByteArray? = null, + initialExtensions: List = emptyList(), ): MlsGroup { val sigKp = signingKey?.let { key -> @@ -1953,13 +1954,18 @@ class MlsGroup private constructor( tree.setLeaf(0, leafNode) val treeHash = tree.treeHash() + // Start with required_capabilities + whatever the caller wants to + // bake into epoch 0 (e.g. the MIP-01 MarmotGroupData extension so + // new peers who join later can see the group name without first + // decrypting a pre-membership bootstrap commit — see MIP-03). + val baseExtensions = listOf(buildMarmotRequiredCapabilitiesExtension()) val groupContext = GroupContext( groupId = groupId, epoch = 0, treeHash = treeHash, confirmedTranscriptHash = ByteArray(0), - extensions = listOf(buildMarmotRequiredCapabilitiesExtension()), + extensions = baseExtensions + initialExtensions, ) // Initial key schedule with zero secrets diff --git a/quartz/src/commonMain/kotlin/com/vitorpamplona/quartz/marmot/mls/group/MlsGroupManager.kt b/quartz/src/commonMain/kotlin/com/vitorpamplona/quartz/marmot/mls/group/MlsGroupManager.kt index 7fddab22b..a387504a1 100644 --- a/quartz/src/commonMain/kotlin/com/vitorpamplona/quartz/marmot/mls/group/MlsGroupManager.kt +++ b/quartz/src/commonMain/kotlin/com/vitorpamplona/quartz/marmot/mls/group/MlsGroupManager.kt @@ -174,10 +174,11 @@ class MlsGroupManager( nostrGroupId: HexKey, identity: ByteArray, signingKey: ByteArray? = null, + initialExtensions: List = emptyList(), ): MlsGroup = mutex.withLock { Log.d(TAG) { "createGroup($nostrGroupId): creating new MLS group" } - val group = MlsGroup.create(identity, signingKey) + val group = MlsGroup.create(identity, signingKey, initialExtensions) groups[nostrGroupId] = group persistGroup(nostrGroupId) Log.d(TAG) { "createGroup($nostrGroupId): done, in-memory group count=${groups.size}" } diff --git a/tools/marmot-interop/headless/patches/whitenoise-defaults-env.patch b/tools/marmot-interop/headless/patches/whitenoise-defaults-env.patch new file mode 100644 index 000000000..57cc16cd4 --- /dev/null +++ b/tools/marmot-interop/headless/patches/whitenoise-defaults-env.patch @@ -0,0 +1,24 @@ +--- a/src/whitenoise/relays.rs ++++ b/src/whitenoise/relays.rs +@@ -98,6 +98,21 @@ + } + + pub(crate) fn defaults() -> Vec { ++ // marmot-interop-headless patch: honour $WHITENOISE_DISCOVERY_RELAYS ++ // (comma-separated list) when present so newly created accounts only ++ // ever get our loopback relay baked into their NIP-65 / inbox / ++ // key-package lists. Without this override, `create-identity` stamps ++ // the hard-coded public set into the account's relay lists, and every ++ // later activate / publish burns connection budget on unreachable ++ // sockets — enough to break inbox-plane activation and drop kind:1059. ++ if let Ok(from_env) = std::env::var("WHITENOISE_DISCOVERY_RELAYS") { ++ let parsed: Vec = from_env ++ .split(',').map(str::trim).filter(|s| !s.is_empty()) ++ .filter_map(|u| RelayUrl::parse(u).ok()) ++ .map(|url| Relay::new(&url)) ++ .collect(); ++ if !parsed.is_empty() { return parsed; } ++ } + let urls: &[&str] = if cfg!(debug_assertions) { + &["ws://localhost:8080", "ws://localhost:7777"] + } else { diff --git a/tools/marmot-interop/headless/setup.sh b/tools/marmot-interop/headless/setup.sh index 635e10de1..797c0116b 100644 --- a/tools/marmot-interop/headless/setup.sh +++ b/tools/marmot-interop/headless/setup.sh @@ -39,7 +39,7 @@ preflight() { 2>&1 | tee -a "$LOG_FILE" fi - # Two harness-only patches to wnd so it runs fully offline / in + # Three harness-only patches to wnd so it runs fully offline / in # sandboxes that block outbound + kernel keyring: # 1. discovery-env: honour $WHITENOISE_DISCOVERY_RELAYS so we can # point wnd at our loopback relay instead of the baked-in public @@ -47,9 +47,17 @@ preflight() { # 2. mock-keyring: honour $WHITENOISE_MOCK_KEYRING so wnd uses the # integration-tests mock keyring store when the kernel keyutils # syscalls are blocked (common in containers / CI). + # 3. defaults-env: reuse the same env var so `Relay::defaults()` + # (what `create-identity` stamps into the new account's NIP-65 / + # inbox / key-package lists) points at the loopback relay too. + # Without it every account wnd creates carries damus.io / + # primal.net / nos.lol, and every later activate / publish burns + # connection budget on unreachable sockets — enough to break the + # account-inbox subscription plane and drop kind:1059 delivery. local -a patches=( "whitenoise-discovery-env.patch" "whitenoise-mock-keyring.patch" + "whitenoise-defaults-env.patch" ) for name in "${patches[@]}"; do local marker="$WN_REPO/.headless-patched-${name%.patch}" @@ -267,4 +275,9 @@ configure_relays() { step "publishing A's KeyPackage" amy_a marmot key-package publish >>"$LOG_FILE" 2>&1 || warn "amy marmot key-package publish failed" + # Give nostr-rs-relay a breath to fsync the kind:10002 / 10050 / 30443 + # writes and push them out on the discovery subscription so that the + # first `wn keys check` that follows actually sees them instead of + # racing the relay's WAL flush. + sleep 2 } diff --git a/tools/marmot-interop/headless/tests-create.sh b/tools/marmot-interop/headless/tests-create.sh index 702d6cdab..33c898c35 100644 --- a/tools/marmot-interop/headless/tests-create.sh +++ b/tools/marmot-interop/headless/tests-create.sh @@ -33,12 +33,19 @@ test_02_a_creates_group() { banner "Test 02 — A creates group, invites B" local id="02 A->B create+invite" - local gid - gid=$(amy_field '.group_id' marmot group create --name "Interop-02") || { + # amy keys groups by the MIP-01 nostr_group_id (`group_id`), wn (via + # mdk-core) keys by the MLS GroupContext groupId (`mls_group_id`). Both + # are 32 random bytes and they are NOT the same. We need both: amy calls + # use `gid`, wn calls use `mls_gid`. + local out gid mls_gid + out=$(amy_json marmot group create --name "Interop-02") || { record_result "$id" fail "create returned no group_id"; return } + gid=$(printf '%s' "$out" | jq -r '.group_id') + mls_gid=$(printf '%s' "$out" | jq -r '.mls_group_id') save_state GROUP_02 "$gid" - info "created group $gid" + save_state GROUP_02_MLS "$mls_gid" + info "created group nostr=$gid mls=$mls_gid" amy_json marmot group add "$gid" "$B_NPUB" >/dev/null || { record_result "$id" fail "amy group add failed"; return @@ -52,16 +59,17 @@ test_02_a_creates_group() { wn_b groups accept "$b_gid" >/dev/null 2>&1 || true info "B joined $b_gid" - # A -> B message + # A -> B message. amy takes the nostr id; wait_for_message calls + # `wn messages list` which needs the MLS id. amy_json marmot message send "$gid" "hello from amethyst" >/dev/null || { record_result "$id" fail "amy send failed"; return } - if ! wait_for_message B "$gid" "hello from amethyst" 30; then + if ! wait_for_message B "$mls_gid" "hello from amethyst" 90; then record_result "$id" fail "B didn't receive A's message"; return fi # B -> A message - wn_b messages send "$gid" "hello from wn" >/dev/null 2>&1 || warn "wn send returned nonzero" + wn_b messages send "$mls_gid" "hello from wn" >/dev/null 2>&1 || warn "wn send returned nonzero" if ! amy_json marmot await message "$gid" --match "hello from wn" --timeout 30 >/dev/null; then record_result "$id" fail "A didn't receive B's reply"; return fi @@ -72,33 +80,41 @@ test_03_b_creates_group() { banner "Test 03 — B creates group, invites A" local id="03 B->A create+invite" - local out gid + # `wn groups create` returns the MLS group id (what wn's messages API + # expects). amy indexes by the MIP-01 nostr_group_id, which we only learn + # once A has processed the welcome and we can ask `amy await group` for + # it. Keep them distinct so each CLI gets the id it understands. + local out mls_gid out=$(wn_b --json groups create "Interop-03" "$A_NPUB" 2>>"$LOG_FILE") || { record_result "$id" fail "wn groups create failed"; return } - gid=$(printf '%s' "$out" | jq_group_id) - if [[ -z "$gid" ]]; then + mls_gid=$(printf '%s' "$out" | jq_group_id) + if [[ -z "$mls_gid" ]]; then record_result "$id" fail "could not parse group_id"; return fi - save_state GROUP_03 "$gid" - info "group_id: $gid" + save_state GROUP_03_MLS "$mls_gid" + info "mls_group_id: $mls_gid" - # A: poll until it joins. - if ! amy_json marmot await group --name "Interop-03" --timeout 30 >/dev/null; then + # A: poll until it joins, and capture its nostr_group_id. + local a_out a_gid + a_out=$(amy_json marmot await group --name "Interop-03" --timeout 30) || { record_result "$id" fail "A never joined B's group"; return - fi + } + a_gid=$(printf '%s' "$a_out" | jq -r '.group_id') + save_state GROUP_03 "$a_gid" + info "A joined as nostr_group_id=$a_gid" # B -> A message - wn_b messages send "$gid" "ping from wn" >/dev/null 2>&1 || true - if ! amy_json marmot await message "$gid" --match "ping from wn" --timeout 30 >/dev/null; then + wn_b messages send "$mls_gid" "ping from wn" >/dev/null 2>&1 || true + if ! amy_json marmot await message "$a_gid" --match "ping from wn" --timeout 30 >/dev/null; then record_result "$id" fail "A didn't see B's ping"; return fi # A -> B reply - amy_json marmot message send "$gid" "pong from amethyst" >/dev/null || { + amy_json marmot message send "$a_gid" "pong from amethyst" >/dev/null || { record_result "$id" fail "amy send pong failed"; return } - if wait_for_message B "$gid" "pong from amethyst" 30; then + if wait_for_message B "$mls_gid" "pong from amethyst" 90; then record_result "$id" pass else record_result "$id" fail "B didn't see A's pong" @@ -112,7 +128,9 @@ test_04_three_member_group() { wn_c keys publish >/dev/null 2>&1 || true sleep 3 - local gid; gid=$(load_state GROUP_02 || true) + local gid mls_gid + gid=$(load_state GROUP_02 || true) + mls_gid=$(load_state GROUP_02_MLS || true) if [[ -z "${gid:-}" ]]; then record_result "$id" skip "no GROUP_02"; return fi @@ -131,8 +149,14 @@ test_04_three_member_group() { amy_json marmot message send "$gid" "hello three-member world" >/dev/null || { record_result "$id" fail "amy send failed"; return } - if wait_for_message B "$gid" "hello three-member world" 30 \ - && wait_for_message C "$c_gid" "hello three-member world" 30; then + # wn's event_processor retries undecryptable pre-membership commits with + # exponential backoff (2+4+8+16=~30s), and kind:445 processing is serial + # per-account; a fresh joiner routinely needs ~60s to burn through that + # retry queue before the add-C commit is applied and "hello three-member + # world" decrypts. The 30s we used for the inline A<->B send/recv + # wouldn't clear that. + if wait_for_message B "$mls_gid" "hello three-member world" 90 \ + && wait_for_message C "$c_gid" "hello three-member world" 90; then record_result "$id" pass else record_result "$id" fail "B or C missed A's post-add message" @@ -166,8 +190,8 @@ test_05_b_adds_a_existing() { amy_json marmot message send "$gid" "joined from amethyst" >/dev/null || { record_result "$id" fail "amy send failed"; return } - if wait_for_message B "$gid" "joined from amethyst" 30 \ - && wait_for_message C "$gid" "joined from amethyst" 30; then + if wait_for_message B "$gid" "joined from amethyst" 90 \ + && wait_for_message C "$gid" "joined from amethyst" 90; then record_result "$id" pass else record_result "$id" fail "B or C didn't see A's message" diff --git a/tools/marmot-interop/headless/tests-extras.sh b/tools/marmot-interop/headless/tests-extras.sh index 2f147c83e..8e9a64910 100644 --- a/tools/marmot-interop/headless/tests-extras.sh +++ b/tools/marmot-interop/headless/tests-extras.sh @@ -7,35 +7,37 @@ test_09_reply_react_unreact() { banner "Test 09 — reply / react / unreact" local id="09 reply/react" - local gid; gid=$(load_state GROUP_02 || true) + local gid mls_gid + gid=$(load_state GROUP_02 || true) + mls_gid=$(load_state GROUP_02_MLS || true) if [[ -z "${gid:-}" ]]; then record_result "$id" skip "no GROUP_02"; return fi # B anchors. Needs a member to be present — if Test 11 already ran and A left, # skip cleanly so we don't double-fail. - if ! wn_b --json groups members "$gid" 2>/dev/null \ - | jq -e --arg p "$A_HEX" '.[]? | select((.pubkey // .public_key) == $p)' \ + if ! wn_b --json groups members "$mls_gid" 2>/dev/null \ + | jq -e --arg p "$A_HEX" '(.result // .) | .[]? | select((.pubkey // .public_key) == $p)' \ >/dev/null 2>&1; then record_result "$id" skip "A already left GROUP_02"; return fi - wn_b messages send "$gid" "anchor for reactions" >/dev/null 2>&1 || true + wn_b messages send "$mls_gid" "anchor for reactions" >/dev/null 2>&1 || true sleep 3 local msg_id - msg_id=$(wn_b --json messages list "$gid" --limit 10 2>/dev/null \ - | jq -r '[.[]? | select((.content // .text // "") == "anchor for reactions")][0].id // empty') + msg_id=$(wn_b --json messages list "$mls_gid" --limit 10 2>/dev/null \ + | jq -r '[(.result // .) | .[]? | select((.content // .text // "") == "anchor for reactions")][0].id // empty') if [[ -z "$msg_id" || "$msg_id" == "null" ]]; then record_result "$id" fail "couldn't find anchor message id"; return fi - wn_b messages react "$gid" "$msg_id" "🌮" >/dev/null 2>&1 || true + wn_b messages react "$mls_gid" "$msg_id" "🌮" >/dev/null 2>&1 || true sleep 3 # amy reply amy_json marmot message send "$gid" "replying via amy" >/dev/null || { record_result "$id" fail "amy send reply failed"; return } - if wait_for_message B "$gid" "replying via amy" 30; then + if wait_for_message B "$mls_gid" "replying via amy" 90; then record_result "$id" pass else record_result "$id" fail "B didn't receive reply" @@ -48,12 +50,14 @@ test_10_concurrent_commits() { banner "Test 10 — Concurrent commits race" local id="10 concurrent commits" - local gid; gid=$(load_state GROUP_02 || true) + local gid mls_gid + gid=$(load_state GROUP_02 || true) + mls_gid=$(load_state GROUP_02_MLS || true) if [[ -z "${gid:-}" ]]; then record_result "$id" skip "no GROUP_02"; return fi - if ! wn_b --json groups members "$gid" 2>/dev/null \ - | jq -e --arg p "$A_HEX" '.[]? | select((.pubkey // .public_key) == $p)' \ + if ! wn_b --json groups members "$mls_gid" 2>/dev/null \ + | jq -e --arg p "$A_HEX" '(.result // .) | .[]? | select((.pubkey // .public_key) == $p)' \ >/dev/null 2>&1; then record_result "$id" skip "A already left GROUP_02"; return fi @@ -62,21 +66,21 @@ test_10_concurrent_commits() { # guarantees a deterministic outcome. ( amy_json marmot group rename "$gid" "race-from-amethyst" >/dev/null ) & local a_pid=$! - ( wn_b groups rename "$gid" "race-from-wn" >/dev/null 2>&1 ) & + ( wn_b groups rename "$mls_gid" "race-from-wn" >/dev/null 2>&1 ) & local b_pid=$! wait "$a_pid" "$b_pid" 2>/dev/null || true sleep 10 local b_name - b_name=$(wn_b --json groups show "$gid" 2>/dev/null | jq -r '.name // empty') + b_name=$(wn_b --json groups show "$mls_gid" 2>/dev/null | jq -r '(.result // .) | .name // empty') local a_name a_name=$(amy_field '.name' marmot group show "$gid" 2>/dev/null || echo "") if [[ -n "$a_name" && "$a_name" == "$b_name" ]]; then info "race converged: both sides see \"$a_name\"" # Verify encryption still works. - wn_b messages send "$gid" "post-race ping" >/dev/null 2>&1 || true - if amy_json marmot await message "$gid" --match "post-race ping" --timeout 15 >/dev/null; then + wn_b messages send "$mls_gid" "post-race ping" >/dev/null 2>&1 || true + if amy_json marmot await message "$gid" --match "post-race ping" --timeout 90 >/dev/null; then record_result "$id" pass else record_result "$id" fail "encryption broken after race" @@ -90,35 +94,39 @@ test_12_offline_catchup() { banner "Test 12 — Offline catch-up" local id="12 offline catchup" - # Fresh group so we don't collide with other tests. - local out gid + # wn-side keys groups by mls_group_id; amy-side by nostr_group_id. Learn + # the amy side from `amy await group` so later amy calls target the right + # group. + local out mls_gid a_out a_gid out=$(wn_b --json groups create "Interop-12" "$A_NPUB" 2>>"$LOG_FILE") - gid=$(printf '%s' "$out" | jq_group_id) - [[ -n "$gid" ]] || { record_result "$id" fail "couldn't create Interop-12"; return; } - save_state GROUP_12 "$gid" + mls_gid=$(printf '%s' "$out" | jq_group_id) + [[ -n "$mls_gid" ]] || { record_result "$id" fail "couldn't create Interop-12"; return; } + save_state GROUP_12_MLS "$mls_gid" # A joins. - amy_json marmot await group --name "Interop-12" --timeout 30 >/dev/null || { + a_out=$(amy_json marmot await group --name "Interop-12" --timeout 30) || { record_result "$id" fail "A never received Interop-12 invite"; return } + a_gid=$(printf '%s' "$a_out" | jq -r '.group_id') + save_state GROUP_12 "$a_gid" # "Go offline" == don't invoke amy. Meanwhile B sends 5 messages + adds C + sends 3 more + rename. for i in 1 2 3 4 5; do - wn_b messages send "$gid" "offline-msg-$i" >/dev/null 2>&1 || true + wn_b messages send "$mls_gid" "offline-msg-$i" >/dev/null 2>&1 || true sleep 1 done - wn_b groups add-members "$gid" "$C_NPUB" >/dev/null 2>&1 || true - wait_for_invite C 30 >/dev/null && wn_c groups accept "$gid" >/dev/null 2>&1 || true + wn_b groups add-members "$mls_gid" "$C_NPUB" >/dev/null 2>&1 || true + wait_for_invite C 30 >/dev/null && wn_c groups accept "$mls_gid" >/dev/null 2>&1 || true for i in 6 7 8; do - wn_b messages send "$gid" "offline-msg-$i" >/dev/null 2>&1 || true + wn_b messages send "$mls_gid" "offline-msg-$i" >/dev/null 2>&1 || true sleep 1 done - wn_b groups rename "$gid" "Interop-12-renamed" >/dev/null 2>&1 || true + wn_b groups rename "$mls_gid" "Interop-12-renamed" >/dev/null 2>&1 || true sleep 3 # A comes back online — single sync pulls everything. local show - show=$(amy_json marmot group show "$gid") || { + show=$(amy_json marmot group show "$a_gid") || { record_result "$id" fail "amy group show failed"; return } local name; name=$(printf '%s' "$show" | jq -r '.name') @@ -127,7 +135,7 @@ test_12_offline_catchup() { fi # All 8 messages should be locally stored. - local msgs; msgs=$(amy_json marmot message list "$gid" --limit 100) + local msgs; msgs=$(amy_json marmot message list "$a_gid" --limit 100) local missing=0 for i in 1 2 3 4 5 6 7 8; do if ! printf '%s' "$msgs" | jq -e --arg t "offline-msg-$i" \ diff --git a/tools/marmot-interop/headless/tests-manage.sh b/tools/marmot-interop/headless/tests-manage.sh index 509313d58..7d6550470 100644 --- a/tools/marmot-interop/headless/tests-manage.sh +++ b/tools/marmot-interop/headless/tests-manage.sh @@ -7,9 +7,17 @@ test_06_member_removal() { banner "Test 06 — Member removal + forward secrecy" local id="06 member removal" - local gid; gid=$(load_state GROUP_05 || true) - if [[ -z "${gid:-}" ]]; then - record_result "$id" skip "no GROUP_05"; return + # MIP-03 only admins may commit Remove proposals. In GROUP_05 (wn-created + # by B, A joined later) A is not an admin, so the test used to fail with + # `IllegalStateException: non-admin members may only commit...`. Test on + # GROUP_02 instead, where amy is the creator and therefore sole admin, + # and where test 04 has already added C. amy calls use the nostr id, + # wn calls use the MLS id. + local gid mls_gid + gid=$(load_state GROUP_02 || true) + mls_gid=$(load_state GROUP_02_MLS || true) + if [[ -z "${gid:-}" || -z "${mls_gid:-}" ]]; then + record_result "$id" skip "no GROUP_02"; return fi amy_json marmot group remove "$gid" "$C_NPUB" >/dev/null || { @@ -17,10 +25,10 @@ test_06_member_removal() { } # C should no longer see the group on its own member view. - local deadline=$(( $(date +%s) + 60 )) removed=0 + local deadline=$(( $(date +%s) + 120 )) removed=0 while [[ $(date +%s) -lt $deadline ]]; do - if ! wn_c --json groups members "$gid" 2>/dev/null \ - | jq -e --arg p "$C_HEX" '.[]? | select((.pubkey // .public_key) == $p)' \ + if ! wn_c --json groups members "$mls_gid" 2>/dev/null \ + | jq -e --arg p "$C_HEX" '(.result // .) | .[]? | select((.pubkey // .public_key) == $p)' \ >/dev/null 2>&1; then removed=1; break fi @@ -33,13 +41,13 @@ test_06_member_removal() { amy_json marmot message send "$gid" "after removing C" >/dev/null || { record_result "$id" fail "amy send failed"; return } - wait_for_message B "$gid" "after removing C" 30 || { + wait_for_message B "$mls_gid" "after removing C" 90 || { record_result "$id" fail "B lost access after C's removal"; return } # Forward secrecy: C must NOT see the post-removal message. sleep 5 - if wait_for_message C "$gid" "after removing C" 10; then + if wait_for_message C "$mls_gid" "after removing C" 10; then record_result "$id" fail "C still decrypted a post-removal message" else record_result "$id" pass @@ -50,8 +58,13 @@ test_07_metadata_rename() { banner "Test 07 — Metadata rename round-trip (MIP-01)" local id="07 metadata rename" - local gid; gid=$(load_state GROUP_02 || true) - if [[ -z "${gid:-}" ]]; then + # amy was the creator of GROUP_02 so its own nostr_group_id is saved as + # GROUP_02; wn keys its copy by the MLS group id saved as GROUP_02_MLS. + # Pass each CLI the id it understands. + local gid mls_gid + gid=$(load_state GROUP_02 || true) + mls_gid=$(load_state GROUP_02_MLS || true) + if [[ -z "${gid:-}" || -z "${mls_gid:-}" ]]; then record_result "$id" skip "no GROUP_02"; return fi @@ -59,9 +72,9 @@ test_07_metadata_rename() { record_result "$id" fail "amy rename failed"; return } - local deadline=$(( $(date +%s) + 60 )) seen="" + local deadline=$(( $(date +%s) + 120 )) seen="" while [[ $(date +%s) -lt $deadline ]]; do - seen=$(wn_b --json groups show "$gid" 2>/dev/null | jq -r '.name // empty') + seen=$(wn_b --json groups show "$mls_gid" 2>/dev/null | jq -r '(.result // .) | .name // empty') [[ "$seen" == "Interop-02-renamed" ]] && break sleep 3 done @@ -70,8 +83,8 @@ test_07_metadata_rename() { } # Now B renames back and A should pick it up. - wn_b groups rename "$gid" "Interop-02-reverse" >/dev/null 2>&1 || true - if amy_json marmot await rename "$gid" --name "Interop-02-reverse" --timeout 60 >/dev/null; then + wn_b groups rename "$mls_gid" "Interop-02-reverse" >/dev/null 2>&1 || true + if amy_json marmot await rename "$gid" --name "Interop-02-reverse" --timeout 120 >/dev/null; then record_result "$id" pass else record_result "$id" fail "A did not pick up B's rename" @@ -82,39 +95,45 @@ test_08_admin_promote_demote() { banner "Test 08 — Admin promote / demote" local id="08 admin promote/demote" - local gid; gid=$(load_state GROUP_03 || true) - if [[ -z "${gid:-}" ]]; then + # GROUP_03 was created by wn so both sides need different ids: + # GROUP_03 → amy's nostr_group_id (captured in test 03 after + # `amy await group` returned `.group_id`) + # GROUP_03_MLS → wn's mls_group_id (wn's `groups create` output) + local a_gid mls_gid + a_gid=$(load_state GROUP_03 || true) + mls_gid=$(load_state GROUP_03_MLS || true) + if [[ -z "${mls_gid:-}" ]]; then record_result "$id" skip "no GROUP_03"; return fi # Ensure 3 members (add C if missing). wn_c keys publish >/dev/null 2>&1 || true sleep 2 - wn_b groups add-members "$gid" "$C_NPUB" >/dev/null 2>&1 || true - wait_for_invite C 30 >/dev/null && wn_c groups accept "$gid" >/dev/null 2>&1 || true + wn_b groups add-members "$mls_gid" "$C_NPUB" >/dev/null 2>&1 || true + wait_for_invite C 30 >/dev/null && wn_c groups accept "$mls_gid" >/dev/null 2>&1 || true # B promotes A. - wn_b groups promote "$gid" "$A_NPUB" >/dev/null 2>&1 || { + wn_b groups promote "$mls_gid" "$A_NPUB" >/dev/null 2>&1 || { record_result "$id" fail "wn promote failed"; return } # A should reflect the new admin set — poll via amy. - if ! amy_json marmot await admin "$gid" "$A_NPUB" --timeout 30 >/dev/null; then + if ! amy_json marmot await admin "$a_gid" "$A_NPUB" --timeout 90 >/dev/null; then record_result "$id" fail "A never saw itself promoted"; return fi # A now commits a rename — only possible if we're admin. - amy_json marmot group rename "$gid" "Interop-03-by-A" >/dev/null || { + amy_json marmot group rename "$a_gid" "Interop-03-by-A" >/dev/null || { record_result "$id" fail "A (now admin) could not rename"; return } # B demotes A. - wn_b groups demote "$gid" "$A_NPUB" >/dev/null 2>&1 || warn "demote returned nonzero" + wn_b groups demote "$mls_gid" "$A_NPUB" >/dev/null 2>&1 || warn "demote returned nonzero" sleep 5 local admins - admins=$(wn_b --json groups admins "$gid" 2>/dev/null \ - | jq -r '.[].pubkey // .[].public_key // .[]' | tr '\n' ' ') + admins=$(wn_b --json groups admins "$mls_gid" 2>/dev/null \ + | jq -r '(.result // .) | .[]?.pubkey // .[]?.public_key // .[]?' | tr '\n' ' ') if [[ "$admins" == *"$A_HEX"* ]]; then record_result "$id" fail "A still admin after demote" else @@ -126,7 +145,9 @@ test_11_leave_group() { banner "Test 11 — Leave group" local id="11 leave group" - local gid; gid=$(load_state GROUP_02 || true) + local gid mls_gid + gid=$(load_state GROUP_02 || true) + mls_gid=$(load_state GROUP_02_MLS || true) if [[ -z "${gid:-}" ]]; then record_result "$id" skip "no GROUP_02"; return fi @@ -135,10 +156,10 @@ test_11_leave_group() { record_result "$id" fail "amy leave failed"; return } - local deadline=$(( $(date +%s) + 60 )) gone=0 + local deadline=$(( $(date +%s) + 120 )) gone=0 while [[ $(date +%s) -lt $deadline ]]; do - if ! wn_b --json groups members "$gid" 2>/dev/null \ - | jq -e --arg p "$A_HEX" '.[]? | select((.pubkey // .public_key) == $p)' \ + if ! wn_b --json groups members "$mls_gid" 2>/dev/null \ + | jq -e --arg p "$A_HEX" '(.result // .) | .[]? | select((.pubkey // .public_key) == $p)' \ >/dev/null 2>&1; then gone=1; break fi diff --git a/tools/marmot-interop/lib.sh b/tools/marmot-interop/lib.sh index 744e9ff95..0196117b1 100644 --- a/tools/marmot-interop/lib.sh +++ b/tools/marmot-interop/lib.sh @@ -150,6 +150,10 @@ expect_contains() { # - plain hex string (from `groups list`) # - {"value":{"vec":[...]}} serde struct (from `groups create`) # - flat byte array [n, ...] (from some responses) +# Plus the three wrapper shapes wn actually uses: +# - {"result": {"mls_group_id": ...}} (groups create) +# - {"group": {"mls_group_id": ...}, "membership": ...} (groups invites[0]) +# - {"mls_group_id": ...} (bare) # Input: JSON string via stdin; optional 2nd arg = field name (default: mls_group_id) jq_group_id() { local field="${1:-mls_group_id}" @@ -159,7 +163,8 @@ jq_group_id() { [($n / 16 | floor), ($n % 16)] | map(if . < 10 then (48 + .) else (87 + .) end) | implode; - (.result // .) | + (.group // .result // .) | + (.group // .) | .[$f] | if type == "string" then . elif (type == "object" and (.value.vec != null)) then @@ -190,8 +195,11 @@ wait_for_invite() { deadline=$(( start + timeout )) last_hb=$start while [[ $(date +%s) -lt $deadline ]]; do + # Post-v0.2 `wn --json groups invites` returns `{"result": [...]}` + # (older builds returned the bare array). Peel the wrapper when + # present so a pending invite is actually detected. gid=$("$wnfn" --json groups invites 2>/dev/null \ - | jq -c '.[0] // empty' 2>/dev/null | jq_group_id || true) + | jq -c '(.result // .) | .[0] // empty' 2>/dev/null | jq_group_id || true) if [[ -n "${gid:-}" ]]; then printf '%s\n' "$gid" return 0 @@ -203,7 +211,7 @@ wait_for_invite() { local elapsed=$(( now - start )) remaining=$(( deadline - now )) local pending pending=$("$wnfn" --json groups invites 2>/dev/null \ - | jq 'length' 2>/dev/null || echo "?") + | jq '(.result // .) | length' 2>/dev/null || echo "?") local recent="" if [[ -f "$data_dir/logs/stderr.log" ]]; then recent=$(tail -n 200 "$data_dir/logs/stderr.log" 2>/dev/null \ @@ -233,7 +241,7 @@ wait_for_message() { fi if [[ -n "${payload:-}" ]] && \ printf '%s' "$payload" | jq -e --arg n "$needle" \ - '.[]? | select((.content // .text // "") | contains($n))' \ + '(.result // .) | .[]? | select((.content // .text // "") | contains($n))' \ >/dev/null 2>&1; then return 0 fi @@ -254,7 +262,7 @@ wait_for_member() { payload=$(wn_c_json groups members "$gid" 2>/dev/null || true) fi if printf '%s' "${payload:-}" | jq -e --arg p "$pubkey" \ - '.[]? | select((.pubkey // .public_key // "") == $p)' \ + '(.result // .) | .[]? | select((.pubkey // .public_key // "") == $p)' \ >/dev/null 2>&1; then return 0 fi