nick/amethyst
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 1 Wiki Activity

fix(marmot,cli,interop): interop-compatible Marmot flows and harness correctness

Browse Source 
Five protocol-level fixes and a batch of harness correctness fixes to get
the headless Marmot/Whitenoise interop harness from 1/13 to 5/13 passing
cleanly, with the remaining failures all rooted in wn's per-account
serial event-processor retry backoff (which drops undecryptable
pre-membership commits after several minutes) rather than amy behaviour.

quartz + commons
----------------
* MarmotGroupData: hold CURRENT_VERSION at 2. mdk-core (the Rust MLS
  engine used by whitenoise-rs) strict-rejects v3 payloads with
  `ExtensionFormatError("Trailing bytes in NostrGroupDataExtension")`
  — our v3 welcomes and GCE commits never apply, so every cross-client
  group flow dies at welcome processing. We still parse v3 happily on
  the way in; we just don't emit it until mdk publishes the
  forward-compat fix MIP-01 mandates.
* MarmotManager.updateGroupMetadata: MERGE extensions instead of
  REPLACING. RFC 9420 §12.1.7 says GCE proposals blow away the old
  extension list; callers that pass only [marmot_group_data] dropped
  [required_capabilities], which peers then reject. Preserve every slot
  except the one we're updating.
* MarmotManager.createGroup: new optional `initialMetadata` parameter
  that bakes MarmotGroupData into epoch-0 GroupContext.extensions
  directly. Without it, creators had to publish a pre-membership
  "bootstrap" commit that no later joiner could decrypt — each such
  peer then burned their retry budget on an undecryptable kind:445
  before seeing the real state. Threaded through MlsGroup.create /
  MlsGroupManager.createGroup.
* MarmotManager.mlsGroupIdHex: new translation helper so any code
  juggling the MIP-01 nostr_group_id (what amy indexes on) and the
  MLS GroupContext groupId (what mdk indexes on) can cross-reference
  them without reaching into MlsGroupManager directly.

amethyst module
---------------
* Account.leaveMarmotGroup: self-demote before SelfRemove per MIP-01,
  and promote a surviving member to admin first if the caller is the
  sole admin (otherwise we'd throw "admin depletion"). Matches the
  cli/GroupMembershipCommands.leave flow.

cli (amy)
---------
* Context.syncIncoming: don't advance `giftWrapSince` on empty polls
  (so the first-ever sync doesn't bump the cursor past every
  past-timestamped wrap we've ever been sent), subtract 2 days lookback
  when filtering (NIP-59 randomWithTwoDays gift wraps can have any
  createdAt in the last 48h), and only advance `groupSince` for groups
  we actually received events for.
* Context.syncIncoming: after ingest, if any Welcome consumed a
  KeyPackage, rotate and publish a fresh one immediately. MIP-00
  requires this — a KP can only be welcomed once and leaving the
  consumed one on relays just means later senders invite us with a
  bundle we no longer have private keys for.
* Context.resolveGroupId: accept either nostr_group_id (amy's primary
  key) or the MLS GroupContext groupId (what wn emits) on every verb
  that takes a group id. Wired through GroupAdd/Remove/Leave/Metadata
  /Read, Message send/list, and all the await* verbs so harness
  scripts never have to juggle both forms for a single group.
* GroupCreateCommand: bake initial metadata into epoch 0 (see the
  quartz change above). Dropped the now-redundant bootstrap commit
  publish and tightened the JSON output to include `mls_group_id`.
* GroupMembershipCommands.leave: self-demote admin before SelfRemove;
  promote an heir if we're the only admin, otherwise the GCE would
  deplete admins and the leave aborts.
* MarmotIngest.ingestGiftWrap: unwrap the sealed-rumor layer. NIP-59
  wrap is gift-wrap(kind:1059) → seal(kind:13) → rumor; the old code
  only unwrapped once and then checked `inner.kind == 444`, which is
  always false because inner is actually the seal. Unseal once more
  before the Welcome check. This single fix is what unsticks every
  amy-side Welcome ingestion.

wn harness patches + scripts
----------------------------
* whitenoise-defaults-env.patch: honour $WHITENOISE_DISCOVERY_RELAYS in
  `Relay::defaults()` (release builds otherwise bake damus.io / primal
  / nos.lol into every new account's NIP-65 / Inbox / KeyPackage
  lists, which breaks publishing and prevents the inbox subscription
  plane from ever reaching an operational state in a sandbox).
* setup.sh: sleep 2s after amy's initial kind:30443 publish so
  nostr-rs-relay has a chance to fsync before wn's first targeted
  discovery query. Without it wn's `keys check` races the relay's
  WAL flush and intermittently returns NotFound.
* lib.sh: peel wn's `{"result": …}` wrapper in `jq_group_id`,
  `wait_for_invite`, `wait_for_message`, `wait_for_member`. Post-v0.2
  wn `--json` output nests everything under `.result` (and
  `groups invites[]` nests further under `.group.mls_group_id`) —
  these helpers were still pattern-matching on the flat shape, so
  they returned empty strings for a perfectly good response.
* tests-{create,manage,extras}.sh: track both group IDs per test
  (amy's nostr + wn's MLS), pass each CLI the id it understands, and
  bump the post-commit wait timeouts to 90–120s so wn's exponential
  retry backoff has time to work through the pre-membership commits
  it can't decrypt and get to the ones it can.

https://claude.ai/code/session_016kAxdp6ubB5CnF9URhCEzP
This commit is contained in:
Claude
2026-04-22 00:28:45 +00:00
parent ae1549b884
commit f08b010f50
20 changed files with 422 additions and 126 deletions
Download Patch File Download Diff File Expand all files Collapse all files
cli/src/main/kotlin/com/vitorpamplona/amethyst/cli/Context.kt
+76 -7
View File
@@ -219,14 +219,26 @@ class Context(
* - kind:445 group events per active group → feed into inbound processor
*
* Incrementally advances the `since` cursors in [state] so the next run
* only asks relays for newer events.
* only asks relays for newer events. Two wrinkles:
*
* 1. NIP-59 gift wraps are published with a random-past `created_at`
* (see [com.vitorpamplona.quartz.utils.TimeUtils.randomWithTwoDays])
* so a newly-published wrap can trivially have `created_at` earlier
* than the last cursor we saw. To avoid silently dropping such wraps
* we always subtract a 2-day lookback window from the gift-wrap
* `since`, and dedup is handled inside [MarmotInboundProcessor].
* 2. We only advance the on-disk cursor when events actually arrive.
* Snapping an empty sync up to "now" on the first invocation would
* make every later `since` query skip any past-dated wrap or 445.
*/
suspend fun syncIncoming(timeoutMs: Long = 8_000) {
val inbox = inboxRelays().ifEmpty { anyRelays() }
val gwSince = state.giftWrapSince
val gwFilterSince =
gwSince?.let { (it - GIFT_WRAP_LOOKBACK_SECS).coerceAtLeast(0L) }
val gwFilter =
if (gwSince != null) {
MarmotFilters.giftWrapsForUserSince(identity.pubKeyHex, gwSince)
if (gwFilterSince != null) {
MarmotFilters.giftWrapsForUserSince(identity.pubKeyHex, gwFilterSince)
} else {
MarmotFilters.giftWrapsForUser(identity.pubKeyHex)
}
@@ -255,10 +267,11 @@ class Context(
if (filterMap.isEmpty()) return
val events = drain(filterMap, timeoutMs)
val now = System.currentTimeMillis() / 1000
var maxGwSeen = gwSince ?: 0L
val maxGroupSeen = perGroupFilters.keys.associateWith { state.groupSince[it] ?: 0L }.toMutableMap()
var sawGiftWrap = false
val sawGroupEvent = mutableSetOf<HexKey>()
for ((relay, event) in events) {
// All the MLS/NIP-59 decryption + persistence lives in MarmotIngest —
@@ -268,21 +281,70 @@ class Context(
when (event.kind) {
GiftWrapEvent.KIND -> {
sawGiftWrap = true
if (event.createdAt > maxGwSeen) maxGwSeen = event.createdAt
}
GroupEvent.KIND -> {
val gid = (event as? GroupEvent)?.groupId() ?: continue
sawGroupEvent.add(gid)
val prev = maxGroupSeen[gid] ?: 0L
if (event.createdAt > prev) maxGroupSeen[gid] = event.createdAt
}
}
}
state.giftWrapSince = if (maxGwSeen > 0) maxGwSeen else now
for ((gid, seen) in maxGroupSeen) {
state.groupSince[gid] = if (seen > 0) seen else now
if (sawGiftWrap && maxGwSeen > 0) {
state.giftWrapSince = maxGwSeen
}
for (gid in sawGroupEvent) {
val seen = maxGroupSeen[gid] ?: continue
if (seen > 0) state.groupSince[gid] = seen
}
// If any welcome we processed consumed a KeyPackage, MIP-00 requires
// us to immediately publish a replacement (a KP can only be used for
// ONE welcome; leaving the old one on relays lets a second sender
// invite us with a bundle we no longer have private keys for). The
// Amethyst UI handles this via its own rotation scheduler; the CLI
// has no scheduler, so we rotate inline right after sync.
if (marmot.needsKeyPackageRotation()) {
try {
val kpRelays = keyPackageRelays().ifEmpty { outboxRelays() }.ifEmpty { anyRelays() }
if (kpRelays.isNotEmpty()) {
val rotated = marmot.rotateConsumedKeyPackages(kpRelays.toList())
for (event in rotated) {
publish(event, kpRelays)
System.err.println("[cli] rotated KeyPackage → ${event.id.take(8)} on ${kpRelays.size} relay(s)")
}
}
} catch (e: Exception) {
System.err.println("[cli] key-package rotation failed: ${e.message}")
}
}
}
/**
* Resolve a group identifier given on the CLI to the nostr_group_id that
* amy's [MarmotManager] indexes on.
*
* amy internally keys everything off MIP-01's `nostr_group_id`. whitenoise
* (and every other mdk consumer) keys off the MLS `GroupContext.groupId` —
* a separate 32-byte random value stamped at group creation. Cross-client
* scripts therefore wind up juggling both ids, and it's very easy to pass
* the wrong one to amy. Rather than make every caller translate, we accept
* either format and resolve here:
* 1. If the input is an active nostr_group_id, use it unchanged.
* 2. Otherwise scan active groups for one whose MLS groupId matches.
* 3. Otherwise return the input unchanged (so the caller still gets a
* sensible `not_member` response rather than a silent mismatch).
*/
fun resolveGroupId(input: HexKey): HexKey {
if (marmot.isMember(input)) return input
val normalized = input.lowercase()
return marmot.activeGroupIds().firstOrNull { nostrId ->
marmot.mlsGroupIdHex(nostrId)?.lowercase() == normalized
} ?: input
}
fun marmotGroupRelays(nostrGroupId: HexKey): Set<NormalizedRelayUrl> {
@@ -303,6 +365,13 @@ class Context(
}
companion object {
/**
* Lookback applied to the gift-wrap `since` filter to compensate for
* NIP-59's randomised-past `created_at`. 2 days matches
* [com.vitorpamplona.quartz.utils.TimeUtils.randomWithTwoDays].
*/
private const val GIFT_WRAP_LOOKBACK_SECS: Long = 2L * 24 * 60 * 60
/** Build a Context but require an identity to already exist — most commands can't run without one. */
fun open(dataDir: DataDir): Context {
val identity =
cli/src/main/kotlin/com/vitorpamplona/amethyst/cli/commands/AwaitCommands.kt
+6 -5
View File
@@ -108,6 +108,7 @@ object AwaitCommands {
Json.writeLine(
mapOf(
"group_id" to match,
"mls_group_id" to ctx.marmot.mlsGroupIdHex(match),
"name" to (ctx.marmot.groupMetadata(match)?.name ?: ""),
"epoch" to ctx.marmot.groupEpoch(match),
),
@@ -127,7 +128,7 @@ object AwaitCommands {
rest: Array<String>,
): Int =
pollCondition(dataDir, rest, "await member <gid> <npub>", targetIdx = 1) { ctx, rawArgs ->
val gid = rawArgs[0]
val gid = ctx.resolveGroupId(rawArgs[0])
val target = ctx.requireUserHex(rawArgs[1])
if (!ctx.marmot.isMember(gid)) {
null
@@ -143,7 +144,7 @@ object AwaitCommands {
rest: Array<String>,
): Int =
pollCondition(dataDir, rest, "await admin <gid> <npub>", targetIdx = 1) { ctx, rawArgs ->
val gid = rawArgs[0]
val gid = ctx.resolveGroupId(rawArgs[0])
val target = ctx.requireUserHex(rawArgs[1])
if (!ctx.marmot.isMember(gid)) {
null
@@ -163,13 +164,13 @@ object AwaitCommands {
rest: Array<String>,
): Int {
if (rest.isEmpty()) return Json.error("bad_args", "await rename <gid> --name <name>")
val gid = rest[0]
val args = Args(rest.drop(1).toTypedArray())
val wantedName = args.requireFlag("name")
val timeoutSecs = args.longFlag("timeout", 30)
val ctx = Context.open(dataDir)
try {
ctx.prepare()
val gid = ctx.resolveGroupId(rest[0])
val deadline = System.currentTimeMillis() + timeoutSecs * 1000
while (System.currentTimeMillis() < deadline) {
ctx.syncIncoming(timeoutMs = 3_000)
@@ -191,13 +192,13 @@ object AwaitCommands {
rest: Array<String>,
): Int {
if (rest.isEmpty()) return Json.error("bad_args", "await epoch <gid> --min N")
val gid = rest[0]
val args = Args(rest.drop(1).toTypedArray())
val min = args.longFlag("min", 1)
val timeoutSecs = args.longFlag("timeout", 30)
val ctx = Context.open(dataDir)
try {
ctx.prepare()
val gid = ctx.resolveGroupId(rest[0])
val deadline = System.currentTimeMillis() + timeoutSecs * 1000
while (System.currentTimeMillis() < deadline) {
ctx.syncIncoming(timeoutMs = 3_000)
@@ -219,13 +220,13 @@ object AwaitCommands {
rest: Array<String>,
): Int {
if (rest.isEmpty()) return Json.error("bad_args", "await message <gid> --match STRING")
val gid = rest[0]
val args = Args(rest.drop(1).toTypedArray())
val needle = args.requireFlag("match")
val timeoutSecs = args.longFlag("timeout", 30)
val ctx = Context.open(dataDir)
try {
ctx.prepare()
val gid = ctx.resolveGroupId(rest[0])
val deadline = System.currentTimeMillis() + timeoutSecs * 1000
while (System.currentTimeMillis() < deadline) {
ctx.syncIncoming(timeoutMs = 3_000)
cli/src/main/kotlin/com/vitorpamplona/amethyst/cli/commands/GroupAddMemberCommand.kt
+1 -1
View File
@@ -37,10 +37,10 @@ object GroupAddMemberCommand {
rest: Array<String>,
): Int {
if (rest.size < 2) return Json.error("bad_args", "group add <group_id> <npub> [<npub> ...]")
val gid = rest[0]
val ctx = Context.open(dataDir)
try {
ctx.prepare()
val gid = ctx.resolveGroupId(rest[0])
ctx.syncIncoming()
if (!ctx.marmot.isMember(gid)) return Json.error("not_member", gid)
cli/src/main/kotlin/com/vitorpamplona/amethyst/cli/commands/GroupCreateCommand.kt
+8 -10
View File
@@ -40,9 +40,12 @@ object GroupCreateCommand {
ctx.prepare()
val gid = RandomInstance.bytes(32).toHexKey()
ctx.marmot.createGroup(gid)
// Stamp initial metadata via the shared factory so UI + CLI stay byte-identical.
// Stamp initial metadata via the shared factory so UI + CLI stay
// byte-identical. Bake the MarmotGroupData extension into the
// epoch-0 GroupContext directly (see `MarmotManager.createGroup`)
// so later invitees receive a pre-populated group from the
// welcome and never have to chase an undecryptable bootstrap
// commit that predates their membership.
val outboxUrls = ctx.outboxRelays().map { it.url }
val metadata =
MarmotGroupData.bootstrap(
@@ -51,19 +54,14 @@ object GroupCreateCommand {
outboxRelays = outboxUrls,
name = name,
)
val commit = ctx.marmot.updateGroupMetadata(gid, metadata)
// Group relays == what the metadata carries, which on first commit is our outbox.
val targets = ctx.outboxRelays()
val ack = ctx.publish(commit.signedEvent, targets)
ctx.marmot.createGroup(gid, initialMetadata = metadata)
Json.writeLine(
mapOf(
"group_id" to gid,
"mls_group_id" to ctx.marmot.mlsGroupIdHex(gid),
"name" to name,
"epoch" to ctx.marmot.groupEpoch(gid),
"commit_event_id" to commit.signedEvent.id,
"published_to" to ack.filterValues { it }.keys.map { it.url },
),
)
return 0
cli/src/main/kotlin/com/vitorpamplona/amethyst/cli/commands/GroupMembershipCommands.kt
+30 -2
View File
@@ -30,10 +30,10 @@ object GroupMembershipCommands {
rest: Array<String>,
): Int {
if (rest.size < 2) return Json.error("bad_args", "group remove <gid> <npub>")
val gid = rest[0]
val ctx = Context.open(dataDir)
try {
ctx.prepare()
val gid = ctx.resolveGroupId(rest[0])
val target = ctx.requireUserHex(rest[1])
ctx.syncIncoming()
if (!ctx.marmot.isMember(gid)) return Json.error("not_member", gid)
@@ -66,18 +66,46 @@ object GroupMembershipCommands {
rest: Array<String>,
): Int {
if (rest.isEmpty()) return Json.error("bad_args", "group leave <gid>")
val gid = rest[0]
val ctx = Context.open(dataDir)
try {
ctx.prepare()
val gid = ctx.resolveGroupId(rest[0])
if (!ctx.marmot.isMember(gid)) return Json.error("not_member", gid)
val targets = ctx.marmotGroupRelays(gid).ifEmpty { ctx.outboxRelays() }
// MIP-01/MIP-03: members listed in `admin_pubkeys` MUST NOT issue
// a SelfRemove proposal before first publishing a GCE that drops
// themselves from the admin list (MlsGroup.selfRemove enforces
// this with `check(!isLocalAdmin())`), and that same GCE MUST NOT
// leave the group with zero admins (admin depletion). If we're
// the only admin, hand admin to another member first.
val demoteEventId: String? =
ctx.marmot.groupMetadata(gid)?.let { metadata ->
if (!metadata.adminPubkeys.contains(ctx.identity.pubKeyHex)) return@let null
val newAdmins = metadata.adminPubkeys.filter { it != ctx.identity.pubKeyHex }.toMutableList()
if (newAdmins.isEmpty()) {
val heir =
ctx.marmot
.memberPubkeys(gid)
.map { it.pubkey }
.firstOrNull { it != ctx.identity.pubKeyHex }
?: return@let null // solo group — skip demote, let MLS state cleanup handle it
newAdmins.add(heir)
}
val demoted = metadata.copy(adminPubkeys = newAdmins)
val demoteCommit = ctx.marmot.updateGroupMetadata(gid, demoted)
ctx.publish(demoteCommit.signedEvent, targets)
demoteCommit.signedEvent.id
}
val outbound = ctx.marmot.leaveGroup(gid)
val ack = ctx.publish(outbound.signedEvent, targets)
Json.writeLine(
mapOf(
"group_id" to gid,
"self_demote_event_id" to demoteEventId,
"proposal_event_id" to outbound.signedEvent.id,
"published_to" to ack.filterValues { it }.keys.map { it.url },
),
cli/src/main/kotlin/com/vitorpamplona/amethyst/cli/commands/GroupMetadataCommands.kt
+2 -1
View File
@@ -66,12 +66,13 @@ object GroupMetadataCommands {
private suspend fun edit(
dataDir: DataDir,
gid: HexKey,
rawGid: HexKey,
mutate: suspend (Context, MarmotGroupData) -> MarmotGroupData,
): Int {
val ctx = Context.open(dataDir)
try {
ctx.prepare()
val gid = ctx.resolveGroupId(rawGid)
ctx.syncIncoming()
if (!ctx.marmot.isMember(gid)) return Json.error("not_member", gid)
val outboxUrls = ctx.outboxRelays().map { it.url }
cli/src/main/kotlin/com/vitorpamplona/amethyst/cli/commands/GroupReadCommands.kt
+4 -3
View File
@@ -56,10 +56,10 @@ object GroupReadCommands {
rest: Array<String>,
): Int {
if (rest.isEmpty()) return Json.error("bad_args", "group show <group_id>")
val gid = rest[0]
val ctx = Context.open(dataDir)
try {
ctx.prepare()
val gid = ctx.resolveGroupId(rest[0])
ctx.syncIncoming()
if (!ctx.marmot.isMember(gid)) return Json.error("not_member", gid)
val meta = ctx.marmot.groupMetadata(gid)
@@ -70,6 +70,7 @@ object GroupReadCommands {
Json.writeLine(
mapOf(
"group_id" to gid,
"mls_group_id" to ctx.marmot.mlsGroupIdHex(gid),
"name" to (meta?.name ?: ""),
"description" to (meta?.description ?: ""),
"epoch" to ctx.marmot.groupEpoch(gid),
@@ -90,10 +91,10 @@ object GroupReadCommands {
rest: Array<String>,
): Int {
if (rest.isEmpty()) return Json.error("bad_args", "group members <group_id>")
val gid = rest[0]
val ctx = Context.open(dataDir)
try {
ctx.prepare()
val gid = ctx.resolveGroupId(rest[0])
ctx.syncIncoming()
if (!ctx.marmot.isMember(gid)) return Json.error("not_member", gid)
val members =
@@ -112,10 +113,10 @@ object GroupReadCommands {
rest: Array<String>,
): Int {
if (rest.isEmpty()) return Json.error("bad_args", "group admins <group_id>")
val gid = rest[0]
val ctx = Context.open(dataDir)
try {
ctx.prepare()
val gid = ctx.resolveGroupId(rest[0])
ctx.syncIncoming()
if (!ctx.marmot.isMember(gid)) return Json.error("not_member", gid)
val m = ctx.marmot.groupMetadata(gid)
cli/src/main/kotlin/com/vitorpamplona/amethyst/cli/commands/MessageCommands.kt
+2 -2
View File
@@ -45,11 +45,11 @@ object MessageCommands {
rest: Array<String>,
): Int {
if (rest.size < 2) return Json.error("bad_args", "message send <gid> <text>")
val gid = rest[0]
val text = rest[1]
val ctx = Context.open(dataDir)
try {
ctx.prepare()
val gid = ctx.resolveGroupId(rest[0])
ctx.syncIncoming()
if (!ctx.marmot.isMember(gid)) return Json.error("not_member", gid)
@@ -77,12 +77,12 @@ object MessageCommands {
rest: Array<String>,
): Int {
if (rest.isEmpty()) return Json.error("bad_args", "message list <gid>")
val gid = rest[0]
val args = Args(rest.drop(1).toTypedArray())
val limit = args.intFlag("limit", Int.MAX_VALUE)
val ctx = Context.open(dataDir)
try {
ctx.prepare()
val gid = ctx.resolveGroupId(rest[0])
ctx.syncIncoming()
if (!ctx.marmot.isMember(gid)) return Json.error("not_member", gid)