fix(marmot): look up KeyPackageBundle by Nostr event id + add MarmotDbg logs

While adding diagnostic logs to trace why invitees were receiving nothing on Marmot group adds, I found the actual root cause: a fundamental hash mismatch between the value the Welcome event carries and the value the receiver looks up by. ### The bug A Marmot WelcomeEvent (kind:444) carries a tag `["e", <eventId>]` referencing the kind:30443 KeyPackage event that was consumed — see `KeyPackageEventTag` and `MarmotWelcomeSender.wrapWelcome`. That `<eventId>` is the *Nostr event id* (a hash of the signed event JSON). `MarmotInboundProcessor.processWelcome` then called `keyPackageRotationManager.findBundleByRef(hexToBytes(eventId))`, which compares each stored bundle's `keyPackage.reference()` — **the MLS-spec KeyPackageRef, an entirely different hash computed by `MlsCryptoProvider.refHash("MLS 1.0 KeyPackage Reference", encoded)` over the TLS-encoded KeyPackage**. Those two values are never equal, so the lookup ALWAYS missed and every invitee path returned `WelcomeResult.Error("No matching KeyPackageBundle found")`. Combined with the previous in-memory-only storage, this is why nothing ever appeared on the invitee's screen. ### The fix `KeyPackageRotationManager` now also indexes bundles by the Nostr event id of the corresponding kind:30443 event: - `eventIdToSlot: Map<HexKey, String>` — populated by `recordPublishedEventId(slot, eventId)`, called from `MarmotManager.generateKeyPackageEvent` immediately after signing the event template (which is when the event id is first known). - `findBundleByEventId(eventId)` — looks up the slot via the new index, then returns the bundle. - `markConsumedByEventId(eventId)` — symmetric consume-by-event-id for the welcome receive path. - The persisted snapshot format is bumped from v1 → v2 to include the eventId map. v1 snapshots are still readable (loaded as if the eventId map were empty); republishing a KeyPackage will refill it. Also cleans the index when slots are consumed. `MarmotInboundProcessor.processWelcome` now uses `findBundleByEventId(keyPackageEventId)` instead of `findBundleByRef(hexToBytes(...))`, and `markConsumedByEventId` for the consume call. The dead `hexToBytes` helper + import are removed. ### Diagnostic logging Added `MarmotDbg`-tagged logs across the entire add-member / send / receive chain so the user can `adb logcat -s MarmotDbg` to see exactly what's being sent and what's being received: - `Account.fetchKeyPackageAndAddMember` — querying relays, KP found / not found, kind, author - `Account.addMarmotGroupMember` — commit publish target, welcome delivery presence, full relay union with sources - `Account.sendMarmotGroupMessage` — group, inner kind, target relays - `Account.publishMarmotKeyPackage` + `ensureMarmotKeyPackagePublished` — publish target relays, signed event id - `DecryptAndIndexProcessor.processNewGiftWrap` — gift wrap unwrap result, inner kind, route to Marmot welcome handler - `DecryptAndIndexProcessor.processMarmotWelcome` — manager null, WelcomeResult, synced metadata, group id - `DecryptAndIndexProcessor.GroupEventHandler.add` — kind:445 arrival, membership check, processGroupEvent result, decrypt path - `MarmotInboundProcessor.processWelcome` — eventId lookup, bundle-not-found details, MLS join success - `MarmotGroupEventsEoseManager.updateFilter` — active groups, per-group relay set, fallback usage, total emitted filters - `AccountGiftWrapsEoseManager.updateFilter` — pubkey + dmRelay set used for kind:1059 subscription Compile-checked: `:quartz:compileKotlinJvm`, `:commons:compileKotlinJvm`, `:amethyst:compilePlayDebugKotlin` — BUILD SUCCESSFUL.
2026-04-15 23:04:58 +00:00
parent 99506e3db0
commit 1c9585e5e4
7 changed files with 329 additions and 53 deletions
@@ -34,7 +34,6 @@ import com.vitorpamplona.quartz.marmot.mls.framing.WireFormat
 import com.vitorpamplona.quartz.marmot.mls.group.MlsGroupManager
 import com.vitorpamplona.quartz.nip01Core.core.Event
 import com.vitorpamplona.quartz.nip01Core.core.HexKey
-import com.vitorpamplona.quartz.nip01Core.core.hexToByteArray
 import kotlinx.coroutines.sync.Mutex
 import kotlinx.coroutines.sync.withLock
 import kotlin.io.encoding.Base64
@@ -229,6 +228,10 @@ class MarmotInboundProcessor(
        nostrGroupId: HexKey,
    ): WelcomeResult =
        try {
+            com.vitorpamplona.quartz.utils.Log
+                .d("MarmotDbg") {
+                    "MarmotInboundProcessor.processWelcome: group=${nostrGroupId.take(8)}… eventId=${welcomeEvent.id.take(8)}…"
+                }
            // Validate the caller-provided nostrGroupId matches the Welcome event's own h tag
            val eventGroupId = welcomeEvent.nostrGroupId()
            if (eventGroupId != null && eventGroupId != nostrGroupId) {
@@ -242,26 +245,49 @@ class MarmotInboundProcessor(
            if (keyPackageEventId == null) {
                return WelcomeResult.Error("WelcomeEvent missing KeyPackage event ID tag")
            }
+            com.vitorpamplona.quartz.utils.Log
+                .d("MarmotDbg") {
+                    "MarmotInboundProcessor.processWelcome: welcomeBytes=${welcomeBytes.size}B looking up KeyPackage by ref=${keyPackageEventId.take(8)}…"
+                }

-            // Find the KeyPackageBundle that was consumed
-            val bundle =
-                keyPackageRotationManager.findBundleByRef(
-                    hexToBytes(keyPackageEventId),
-                ) ?: return WelcomeResult.Error(
+            // Find the KeyPackageBundle that was consumed.
+            //
+            // The Welcome's "e" tag carries the *Nostr event id* of the
+            // kind:30443 event (NOT the MLS reference hash), so we must
+            // resolve it via the eventId→slot index that
+            // [MarmotManager.generateKeyPackageEvent] populates after
+            // signing each KeyPackageEvent.
+            val bundle = keyPackageRotationManager.findBundleByEventId(keyPackageEventId)
+            if (bundle == null) {
+                com.vitorpamplona.quartz.utils.Log
+                    .w("MarmotDbg") {
+                        "MarmotInboundProcessor.processWelcome: NO matching KeyPackageBundle for eventId=${keyPackageEventId.take(8)}… " +
+                            "— inviter referenced a KeyPackage we don't have private keys for. " +
+                            "Either the bundle was generated in a previous session and never persisted, " +
+                            "or this account never published this KeyPackage."
+                    }
+                return WelcomeResult.Error(
                    "No matching KeyPackageBundle found for event $keyPackageEventId",
                )
+            }
+            com.vitorpamplona.quartz.utils.Log
+                .d("MarmotDbg") { "MarmotInboundProcessor.processWelcome: bundle found — invoking groupManager.processWelcome" }

            // Join the group
            groupManager.processWelcome(nostrGroupId, welcomeBytes, bundle)
+            com.vitorpamplona.quartz.utils.Log
+                .d("MarmotDbg") { "MarmotInboundProcessor.processWelcome: groupManager.processWelcome succeeded for ${nostrGroupId.take(8)}…" }

            // Mark the KeyPackage as consumed — triggers rotation
-            keyPackageRotationManager.markConsumedByRef(hexToBytes(keyPackageEventId))
+            keyPackageRotationManager.markConsumedByEventId(keyPackageEventId)

            WelcomeResult.Joined(
                nostrGroupId = nostrGroupId,
                needsKeyPackageRotation = keyPackageRotationManager.needsRotation(),
            )
        } catch (e: Exception) {
+            com.vitorpamplona.quartz.utils.Log
+                .w("MarmotDbg", "MarmotInboundProcessor.processWelcome: exception ${e.message}", e)
            WelcomeResult.Error("Failed to process Welcome: ${e.message}", e)
        }

@@ -455,9 +481,4 @@ class MarmotInboundProcessor(
            "Outer decryption failed with current and ${retainedKeys.size} retained epoch key(s)",
        )
    }
-
-    private fun hexToBytes(hex: HexKey?): ByteArray {
-        if (hex == null) return ByteArray(0)
-        return hex.hexToByteArray()
-    }
 }
@@ -32,6 +32,7 @@ import com.vitorpamplona.quartz.marmot.mls.tree.Credential
 import com.vitorpamplona.quartz.marmot.mls.tree.LeafNode
 import com.vitorpamplona.quartz.marmot.mls.tree.LeafNodeSource
 import com.vitorpamplona.quartz.marmot.mls.tree.Lifetime
+import com.vitorpamplona.quartz.nip01Core.core.HexKey
 import com.vitorpamplona.quartz.utils.Log
 import com.vitorpamplona.quartz.utils.TimeUtils
 import kotlinx.coroutines.sync.Mutex
@@ -61,6 +62,19 @@ class KeyPackageRotationManager(
    private val activeBundles = mutableMapOf<String, KeyPackageBundle>()
    private val pendingRotations = mutableSetOf<String>()

+    /**
+     * Maps the Nostr event id (kind:30443) of a published KeyPackage to the
+     * d-tag slot whose bundle backs it. The welcome event references its
+     * consumed KeyPackage by Nostr event id (the MIP-02 "e" tag), but
+     * [activeBundles] is keyed by d-tag slot — so we need this side index
+     * to recover the matching bundle on the receive side.
+     *
+     * Populated by [recordPublishedEventId], which is called right after
+     * the kind:30443 event is signed (the event id is only known then).
+     * Also persisted in the snapshot so it survives app restart.
+     */
+    private val eventIdToSlot = mutableMapOf<String, String>()
+
    /**
     * Restore previously persisted bundles + rotation state from [store].
     * Call once at startup before any other use of this manager.
@@ -79,18 +93,27 @@ class KeyPackageRotationManager(
            val decoded = decodeSnapshot(bytes)
            mutex.withLock {
                activeBundles.clear()
-                activeBundles.putAll(decoded.first)
+                activeBundles.putAll(decoded.bundles)
                pendingRotations.clear()
-                pendingRotations.addAll(decoded.second)
+                pendingRotations.addAll(decoded.pending)
+                eventIdToSlot.clear()
+                eventIdToSlot.putAll(decoded.eventIdToSlot)
            }
            Log.d("KeyPackageRotationManager") {
-                "Restored ${decoded.first.size} active KeyPackage bundle(s), ${decoded.second.size} pending rotation"
+                "Restored ${decoded.bundles.size} active KeyPackage bundle(s), " +
+                    "${decoded.pending.size} pending rotation, ${decoded.eventIdToSlot.size} eventId mapping(s)"
            }
        } catch (e: Exception) {
            Log.w("KeyPackageRotationManager", "Failed to decode persisted KeyPackages: ${e.message}")
        }
    }

+    private data class Snapshot(
+        val bundles: Map<String, KeyPackageBundle>,
+        val pending: Set<String>,
+        val eventIdToSlot: Map<String, String>,
+    )
+
    /**
     * Encode the current rotation manager state to opaque bytes for
     * persistence. Caller must hold the mutex.
@@ -113,16 +136,22 @@ class KeyPackageRotationManager(
        for (slot in pendingRotations) {
            writer.putOpaque2(slot.encodeToByteArray())
        }
+        // eventId → slot (added in v2)
+        writer.putUint32(eventIdToSlot.size.toLong())
+        for ((eventId, slot) in eventIdToSlot) {
+            writer.putOpaque2(eventId.encodeToByteArray())
+            writer.putOpaque2(slot.encodeToByteArray())
+        }
        return writer.toByteArray()
    }

    /**
-     * Decode the persisted snapshot. Returns (activeBundles, pendingRotations).
+     * Decode the persisted snapshot.
     */
-    private fun decodeSnapshot(bytes: ByteArray): Pair<Map<String, KeyPackageBundle>, Set<String>> {
+    private fun decodeSnapshot(bytes: ByteArray): Snapshot {
        val reader = TlsReader(bytes)
        val version = reader.readUint16()
-        require(version == SNAPSHOT_VERSION) {
+        require(version == 1 || version == SNAPSHOT_VERSION) {
            "Unsupported KeyPackage snapshot version: $version"
        }
        val numBundles = reader.readUint32().toInt()
@@ -141,7 +170,19 @@ class KeyPackageRotationManager(
        repeat(numPending) {
            pending.add(reader.readOpaque2().decodeToString())
        }
-        return bundles to pending
+        // eventId → slot map: only present in v2+. Older snapshots may
+        // not have this section, in which case the map starts empty and
+        // will be repopulated as KeyPackages are republished.
+        val eventIdMap = mutableMapOf<String, String>()
+        if (version >= SNAPSHOT_VERSION && reader.hasRemaining) {
+            val numEventIds = reader.readUint32().toInt()
+            repeat(numEventIds) {
+                val eventId = reader.readOpaque2().decodeToString()
+                val slot = reader.readOpaque2().decodeToString()
+                eventIdMap[eventId] = slot
+            }
+        }
+        return Snapshot(bundles, pending, eventIdMap)
    }

    /**
@@ -213,7 +254,13 @@ class KeyPackageRotationManager(

    /**
     * Find the bundle whose KeyPackage reference matches the given ref.
-     * Used when we receive a Welcome and need to find the matching bundle.
+     * Used when we receive a Welcome and need to find the matching bundle
+     * via the MLS-spec KeyPackageRef hash.
+     *
+     * NOTE: a Marmot Welcome's "e" tag actually carries the *Nostr event id*
+     * of the kind:30443 event, NOT the MLS reference hash, so the welcome
+     * receive path must use [findBundleByEventId] instead. This function
+     * remains for any callers that genuinely need the MLS-ref lookup.
     */
    suspend fun findBundleByRef(keyPackageRef: ByteArray): KeyPackageBundle? =
        mutex.withLock {
@@ -222,6 +269,35 @@ class KeyPackageRotationManager(
            }
        }

+    /**
+     * Find the bundle for a KeyPackage that was published as the given
+     * Nostr event id (kind:30443). This is the lookup used by Welcome
+     * processing — the "e" tag in a [WelcomeEvent] is the Nostr event id.
+     *
+     * Requires [recordPublishedEventId] to have been called for the slot
+     * after the kind:30443 event was signed.
+     */
+    suspend fun findBundleByEventId(eventId: HexKey): KeyPackageBundle? =
+        mutex.withLock {
+            val slot = eventIdToSlot[eventId] ?: return@withLock null
+            activeBundles[slot]
+        }
+
+    /**
+     * Record that the bundle for [dTagSlot] has been published as the
+     * kind:30443 event with [eventId]. Call this immediately after signing
+     * the event template (the Nostr event id is only known once the event
+     * has been signed). The mapping is persisted alongside the bundles so
+     * it survives app restart.
+     */
+    suspend fun recordPublishedEventId(
+        dTagSlot: String,
+        eventId: HexKey,
+    ) = mutex.withLock {
+        eventIdToSlot[eventId] = dTagSlot
+        persistUnlocked()
+    }
+
    /**
     * Mark a KeyPackage slot as consumed (used in a Welcome).
     * The slot will be included in [pendingRotationSlots] and should be
@@ -230,6 +306,9 @@ class KeyPackageRotationManager(
    suspend fun markConsumed(dTagSlot: String) =
        mutex.withLock {
            activeBundles.remove(dTagSlot)
+            // Drop any eventId mappings that pointed at this slot.
+            val staleEventIds = eventIdToSlot.entries.filter { it.value == dTagSlot }.map { it.key }
+            staleEventIds.forEach { eventIdToSlot.remove(it) }
            pendingRotations.add(dTagSlot)
            persistUnlocked()
        }
@@ -244,12 +323,29 @@ class KeyPackageRotationManager(
                    bundle.keyPackage.reference().contentEquals(keyPackageRef)
                }
            if (entry != null) {
-                activeBundles.remove(entry.key)
-                pendingRotations.add(entry.key)
+                val consumedSlot = entry.key
+                activeBundles.remove(consumedSlot)
+                val staleEventIds = eventIdToSlot.entries.filter { it.value == consumedSlot }.map { it.key }
+                staleEventIds.forEach { eventIdToSlot.remove(it) }
+                pendingRotations.add(consumedSlot)
                persistUnlocked()
            }
        }

+    /**
+     * Mark a slot as consumed by Nostr event id (the value carried by the
+     * Welcome's "e" tag). Companion to [findBundleByEventId].
+     */
+    suspend fun markConsumedByEventId(eventId: HexKey) =
+        mutex.withLock {
+            val slot = eventIdToSlot[eventId] ?: return@withLock
+            activeBundles.remove(slot)
+            val staleEventIds = eventIdToSlot.entries.filter { it.value == slot }.map { it.key }
+            staleEventIds.forEach { eventIdToSlot.remove(it) }
+            pendingRotations.add(slot)
+            persistUnlocked()
+        }
+
    /**
     * Get the d-tag slots that need rotation (KeyPackage was consumed).
     */
@@ -341,7 +437,11 @@ class KeyPackageRotationManager(
        /** Proactive rotation after 7 days even if not consumed */
        const val MAX_KEY_PACKAGE_AGE_SECONDS = 7L * 24 * 60 * 60

-        /** On-disk snapshot format version for [KeyPackageBundleStore]. */
-        private const val SNAPSHOT_VERSION = 1
+        /**
+         * On-disk snapshot format version for [KeyPackageBundleStore].
+         * v1: bundles + pendingRotations
+         * v2: + eventIdToSlot map (so welcome lookup by Nostr event id works)
+         */
+        private const val SNAPSHOT_VERSION = 2
    }
 }