Trims ~300 lines of screen output from an all-pass run without losing
any post-mortem detail — all removed lines still land in $LOG_FILE:
- preflight: move tool-path listings to $LOG_FILE only
- configure_relays: collapse 30+ per-relay "add ok" info lines into a
single "X ok, Y already-present, Z failed" summary
- configure_relays: drop verbose "B relay list after configure" dump
(file only); compress kind:30443 / kind:10050+1059+445 sanity logs
to single summary lines
- discover_a_relays: replace 10-line per-type relay list with a
nip65/inbox/kp count summary; full lists in $LOG_FILE
- main: drop unconditional "Post-configure baseline diagnostics"
(dump_daemon_diagnostics already fires on invite-timeout)
- test_02: move pre-invite dump_daemon_diagnostics and post-send
dump_outbound_diagnostics into the failure branches only
- test_04: drop unconditional pre-invite daemon dump
- test_01: send raw relay-list blob to $LOG_FILE instead of stderr
Failure paths still dump full daemon diagnostics, so debugging a broken
relay set loses nothing.
https://claude.ai/code/session_011HLbt9QMwBMSxovNRkv4GB
Old test_13 asked the operator to restart Amethyst or toggle KP relays
to trigger rotation, but neither does in current Amethyst:
- Restart: ensureMarmotKeyPackagePublished early-returns when a bundle
already exists on disk.
- Toggle KP relays: saveKeyPackageRelayList republishes the existing
kind:30443 to the new relay set (same event_id).
Rotation only fires when A's own KeyPackage is *consumed* by an
incoming Welcome. Switch the test to have B create a one-off group
inviting A, which consumes A's KP and triggers the rotation path in
DecryptAndIndexProcessor -> publishMarmotKeyPackages. wn then sees a
new event_id for the same d-tag slot.
https://claude.ai/code/session_011HLbt9QMwBMSxovNRkv4GB
Interop Test 06 (wn removes a member from a group Amethyst is in) was
failing with:
GroupEventHandler.add: ERROR Failed to apply commit:
UpdatePath node count (1) doesn't match direct path length (2)
Repro scenario: 3-member group [B=leaf 0, C=leaf 1, A=leaf 2]; B (admin,
committer) removes C. After the Remove proposal applies, leaf 1 is
blank but leaf 2 (A) is still occupied — so leaf_count stays at 3 per
RFC 9420 §7.8 and the sender's direct path has length 2 ([node 1,
root]).
Per RFC 9420 §4.1.2 and §7.9 the **filtered direct path** drops any
parent node whose child on the copath has an empty resolution:
encrypting to such a parent is equivalent to encrypting to its only
non-blank child, which is already on the direct path. In our scenario
the parent at level 1 (parent of leaves 0, 1) has a blank leaf 1 on
the copath — empty resolution — so it's filtered out. Filtered
direct path length is 1, and that's what openmls / whitenoise-rs emit
in `UpdatePath.nodes<V>`.
Quartz was always using the unfiltered direct path on both sides
(send + receive + parent-hash chain + path-secret decrypt). That
worked for quartz↔quartz (both sides agreed), but broke the moment a
spec-correct sender (wn) sent a filtered UpdatePath into a quartz
receiver. Parent-hash chain is further affected because §7.9.2 walks
the filtered path — even if the size check passed, the hashes would
not match.
Fix it throughout:
- RatchetTree.filteredDirectPath: new helper returning parallel
(filteredDirectPath, filteredCopath) with the empty-resolution
entries dropped. Uses the existing `resolution()` helper, which
already handles unmerged_leaves per §4.1.2.
- RatchetTree.applyUpdatePath: size-check + target the FILTERED path.
- MlsGroup.commit (sender): emit one staged path node per filtered
level; encrypt path secrets using the filtered copath; patch parent
hashes into the filtered positions only.
- MlsGroup.processCommitInner (receiver): the UpdatePath node index
aligns to the filtered direct path, so the common-ancestor lookup
must find the filtered index to pick the right ciphertext + copath
resolution. Use the unfiltered index only for counting KDF steps
to the root (path_secret chain advances one step per unfiltered
level regardless of filtering — the chain is continuous; filtering
only decides which levels emit a UpdatePathNode, not how many KDF
steps separate them).
- MlsGroup.computeSenderParentHashes: walk the filtered direct path.
Map each filtered node back to its unfiltered level so the
preUpdateSiblingHashes lookup (indexed by unfiltered level) still
resolves. This makes the parent_hash chain agree with §7.9.2 and
with what openmls computes on the other side.
- MlsGroup.verifyParentHash: short-circuit on empty filtered path
rather than empty unfiltered path.
New regression test: testRemoveMiddleLeaf_ReceiverAcceptsCommit
reproduces the 3-member remove-middle scenario end-to-end within
quartz. It passed before this patch (because both sides were
unfiltered and agreed with each other), and it passes after this
patch (because both sides are now filtered and still agree) — the
compatibility win is that quartz now also agrees with
openmls/wn-produced commits of the same shape.
Also fix an unrelated script bug in marmot-interop.sh: the
discover_a_relays SQL probe was falsely reporting "wn has NO cached
relay entries for A" even when welcome delivery was working, because
the query assumed users.pubkey stored as BLOB but wn stores it as
TEXT (hex) in the current schema. Accept either encoding.
WhiteNoise threads its kind:445 payloads across three inner kinds:
kind:9 for chat, kind:7 for emoji reactions, kind:5 for unreacts. The
Amethyst ingest pipeline was routing every inner event into the group
chatroom feed, so a kind:7 reaction rendered as a chat bubble whose
only content was the emoji, quoting the liked message via the target
`e` tag. That looked identical to a threaded reply. The actual kind:9
reply from `wn messages send --reply-to` never arrived, which made the
two look swapped in the UI.
Three changes to untangle this:
- MarmotGroupList.addMessage/restoreMessage now skip inner events with
kind:5 and kind:7 before they enter `MarmotGroupChatroom.messages`.
The reaction is still consumed by LocalCache so it attaches to the
target note's reaction row, and the deletion still revokes that
reaction — they just don't appear as standalone bubbles.
- LocalCache.computeReplyTo learned to derive thread parents for
`ChatEvent` (kind:9) from plain NIP-10 `e` tags in addition to the
existing NIP-18 `q` tag path. WhiteNoise emits `e`-tagged replies;
without this the reply bubble had no quote context in the feed.
- tools/marmot-interop/marmot-interop.sh: `wn messages send` exposes
its `reply_to` field through clap v4, which renames snake_case to
kebab-case by default. The script was passing `--reply_to`, which
clap rejected; the `|| true` + redirected stderr hid the error and
no reply was ever published. Use `--reply-to`.
https://claude.ai/code/session_01K3g1uWLhByoEdBS77zdF32
Post-v0.2 wn wraps `--json groups show` output in {"result": {...}},
so `jq '{name, epoch}'` was reading the top level where those keys
don't exist — Test 10 always logged "B state: {name: null, epoch: null}"
and the operator couldn't tell whether the race produced the expected
converged state on wn's side.
Peel the .result wrapper before extracting the fields; same pattern
already used everywhere else in the script that parses wn JSON.
The reverse-rename leg issued `wn_b groups rename` on Interop-02, but
Amethyst created Interop-02 — A is the admin, B is a plain member.
MIP-03's `enforceAuthorizedProposalSet` rejects any GroupContextExtensions
commit from a non-admin, so wn's rename was being dropped client-side,
the metadata commit never reached any relay, and Amethyst's UI (correctly
waiting for a kind:445 on Interop-02) never saw "Interop-02-reverse".
Split Test 07 into two separately-recorded legs so each uses a group
where the renamer is actually the admin:
07a — A (admin of Interop-02) renames to "Interop-02-renamed" via
the Amethyst UI; wn-side poll observes the new name.
07b — B (admin of Interop-03) renames to "Interop-03-renamed" via
`wn_b groups rename`; operator confirms Amethyst picked it up.
Each leg individually records pass/fail/skip so a missing GROUP_03
(Test 03 skipped) doesn't take out the whole test.
Test 06 was asking the operator to remove C from Interop-05 via the
Amethyst UI — but Interop-05 was created by B (wn), so B is the sole
admin and A is a plain member. MIP-03's authorization gate
(`enforceAuthorizedProposalSet` in `MlsGroup.kt:1842`) rejects any
Remove proposal from a non-admin, so the UI correctly fires "not an
admin" and the test fails against its own author's assumption — the
test was misaligned with the admin model, not the code.
Flip the flow: B (admin) issues the removal via `wn groups
remove-members <gid> <C_NPUB>`, and A / C observe the effect. That
still exercises the same intended invariants:
- B + A remain in the group, commit + epoch advance propagate.
- A's Amethyst UI shows C gone (visible confirm).
- A sends "after removing C" → B decrypts (expected).
- C cannot decrypt the new message (forward secrecy at the new
epoch — C's retained keys only open the pre-remove epoch).
The "Amethyst admin issues remove" path is already separately
covered by Test 02 (A creates the group → A is admin → A removes)
and Test 08 (promote A, then A issues an admin-only action). Test
06's focus is forward secrecy after remove, not who may initiate —
so having B drive it is the spec-correct shape.
The interop harness was forcing specific relays on the Amethyst account —
"configure Amethyst with these five relays, also add them to your DM
Inbox list, also to your Key Package Relays, then republish" — which
made the test artificial:
- It masked the real failure modes (wn publishing to an A-advertised
relay wn can't reach, A's 10050 pointing at auth-required relays,
etc.) because everyone was always on the same happy-path set.
- It mutated the operator's real Amethyst account config — those
five relays persist after the run.
- It short-circuited wn's discovery chain (kind:10002 → 10050 →
10051 resolution), so the one path most likely to hold subtle bugs
was never exercised under divergent configs.
Real-world interop is: both clients have their own independently-chosen
relay sets and use the advertised kind:10002/10050/10051 to find each
other. Make the harness match that:
- Keep the five-relay set as the wn B/C bootstrap (they still need
somewhere to publish their own identity). Drop the "configure
Amethyst to match" steps from `instruct_amethyst_setup`;
confirmation only now ("is A logged in as <npub>, and has A
published its four lists to any public relay?").
- Add `discover_a_relays` step between configure + the operator
prompt. Calls `wn keys check $A_NPUB` on both daemons to trigger
wn's targeted fetch for kinds [0, 10002, 10050, 10051] (the
only side-effect available to populate wn's user_relays cache).
Then reads the cache directly via sqlite3 against wn's DB and
prints what wn actually learned, grouped by relay_type. Warns
loudly if A's 10050 shares zero relays with wn's bootstrap — the
exact failure mode that caused the prior round of "Test 03
silently times out" reports.
- `--local-relays` path keeps the old behavior: offline/sandbox
mode, the harness owns the relay so forcing Amethyst onto it is
correct.
Scope is the harness only. No Amethyst / quartz / commons changes.
Test 03 (wn creates a group, invites Amethyst) was failing at the
operator-visible level — "I don't see Interop-03 on Amethyst" — with
no clear signal in the Amethyst logcat because no kind:1059 welcome
gift wrap ever arrived.
Root cause: when wn invites Amethyst it fetches A's kind:10050 inbox
list and publishes the welcome gift wrap there. Amethyst's built-in
DM inbox defaults advertise auth.nostr1.com + relay.0xchat.com, which
wn (connected only to the five harness public relays) cannot reach.
Unless the operator manually adds the harness relays to Amethyst's
DM Inbox list, the gift wrap lands on relays the reader isn't
subscribed to, and the invite is silently dropped.
Confirmed empirically by adding wss://nostr.bitcoiner.social (one of
the harness relays) to Amethyst's DM Inbox list and watching
Interop-03 appear on the next wn create.
Update `instruct_amethyst_setup` to make the DM Inbox step explicit
and call out the specific failure mode it prevents. Also add a nudge
to verify NIP-65 + kind:10050 were actually republished after the
relay list edits.
Broaden the default public relay set for the interactive harness so
kind:1059 gift wraps and kind:10050 inbox lists have more landing
spots that don't pre-filter Marmot traffic out. Both relays are
known to accept kinds outside the core "note / metadata / contacts"
trio that damus/primal tend to restrict.
The Amethyst-side setup instructions are rendered from the same list,
so operators who configure Amethyst per the harness prompt get the
same five URLs on the app side too.
wnd persists pending group invites across daemon restarts, so on any
rerun C (and B) already have unaccepted welcomes from previous runs
lying around. `wait_for_invite` was pulling `.[0]` out of the invite
list, which is the oldest pending — i.e. the stalest — and treating
it as the arrival the caller just triggered. In the sanity probe this
caused the kind:445 step to send to B's NEW group id while C was
accepting a DIFFERENT (old) group, producing a bogus "kind:445 failed
— relays may be dropping group messages" with no actual relay issue.
Add `snapshot_invites <B|C>` which returns the current gids as CSV,
and teach `wait_for_invite` to accept that CSV as an ignore list so
it walks past stale welcomes and only returns a freshly-arrived one.
Use the new snapshot→wait pattern at every call site: sanity check,
Test 02 (B), Test 04 (C), Test 05 (C), Test 08 (C), Test 12 (C).
Also log a gid-mismatch warning in the sanity check when B's created
group and C's accepted welcome disagree, as an extra guard against
the filter being bypassed.
Several public relays (damus.io in particular) silently drop kind:1059 /
10050 / 30443 from accounts they've never seen before — the sanity
check in the interactive harness reports exactly this failure mode
when the three-relay default set is in use, because wn's identities are
brand-new and have nothing on file.
Before the kind:30443 + 10050/1059/445 probes, publish a kind:0
profile event for each wn identity via `wn profile update --name
--about`. That marks both accounts as "known" on the relay and lets
the subsequent gift-wrap and KeyPackage publishes land. Runs inside
configure_relays (after relays are added, before the first sanity
probe) so the sequencing is: add relay -> advertise profile -> publish
KPs -> start exchanging welcomes.
Does not help when a relay outright refuses the kind (some relays
whitelist only kind:1 / 3 / 7); those still need --local-relays.
When `wn_b messages send` fires but Amethyst never displays the reply,
the harness used to swallow the send output and only ask the operator a
blind yes/no — giving no signal about which side of the pipe failed.
Amethyst subscribes to kind:445 on the MLS GroupContext `relays`
extension (falling back to `Account.homeRelays`), so the two
actionable failure modes are:
- wn published to a relay Amethyst isn't subscribed to (group-relay
mismatch between the MLS extension and Amethyst's subscription)
- Amethyst received the event but `GroupEventHandler.add` dropped it
(no `h` tag, `isMember=false`, decrypt error)
Teach the harness to dump both sides automatically on the reply step:
- Capture wn's `messages send` stdout/stderr into the run log.
- `dump_outbound_diagnostics` prints wn's local view of the group
messages (did the send persist to wn's own DB?), the MLS
GroupContext relay list (what Amethyst should be subscribed to),
and live relay connection status.
- On operator-reported fail, prompt for an Amethyst-side logcat
grep targeted at the exact log points — MarmotGroupEventsEoseManager
filter updates, GroupEventHandler.add arrivals, and the membership
drop branch — naming the current group id so the operator can grep
for the right hex prefix.
The interactive harness was failing with the same
KeyringEntryMissingForExistingDatabase error as the headless one —
the MLS SQLite DB on disk references a keyring entry that no longer
exists (keychain entry pruned, data dir restored out of band, or a
previous run used the mock keyring). wnd can't open the DB in that
state, so the daemon exits before its socket appears.
Unlike the headless harness (which wipes unconditionally because its
mock keyring is always ephemeral), the interactive harness uses the
real OS keyring and benefits from preserving B/C identities across
runs. So: try once, and only wipe + retry when stderr actually shows
the keyring-missing error. Also dump stderr/stdout tails inline on
final failure so the operator doesn't have to chase a log path.
Apply the same SIGINT/SIGTERM/SIGHUP trap hardening to the interactive
harness that just landed on the headless one. Without it, killing the
script mid-run leaves the nohup'd wn daemons alive and the next run
fails in preflight because the sockets are still held.
The `.result` wrapper fix landed in lib.sh's pollers and the headless tests
back in f08b010f, but the 8 inline `jq` reads in marmot-interop.sh were
missed — which silently returned empty against current `wn` and made every
UI-verification test look like "Amethyst didn't propagate X" when in fact
the underlying wn state was correct and the script just couldn't see it.
All 8 reads now peel `(.result // .)` the same way the lib.sh helpers do,
staying backwards-compatible with the pre-v0.2 flat shape:
* test 02, 04 member lists
* test 06 member-removed check
* test 07 group-name poll
* test 08 admin list (promote + demote)
* test 09 anchor message lookup
* test 11 leave-group member check
Also split the configure_relays sanity check into three relay-level
surfaces so a broken relay set points at the specific kind it can't
handle, instead of 10 later tests all timing out with the same "no
invite" symptom:
* kind:30443 (KeyPackage) — both sides publish + symmetric discover
* kind:10050 (Inbox advertise) — C reachable as giftwrap target
* kind:1059 (Gift wrap) — B->C welcome delivery
* kind:445 (MLS commit/msg) — real group-message round-trip
And drop the orphan whitenoise-mdk-plaintext-policy.patch: it lived in
`headless/patches/` but was never referenced by setup.sh's patch array,
and if it ever were applied it would flip mdk-core's MLS wire-format
from MIXED_CIPHERTEXT to MIXED_PLAINTEXT — a protocol-breaking
divergence from every stock whitenoise-rs / White Noise build. Removing
it eliminates the trap of a future contributor "enabling" it.
https://claude.ai/code/session_018kSBco5VVfctkW6vAm7NzA
Adds a dump_daemon_diagnostics helper (relays per type, wn debug health,
relay-control-state, pending invites, wnd stderr tail) and wires it into
Test 02 and Test 04 on both sides of the poll (pre-invite baseline +
post-timeout). wait_for_invite now prints a ~10s heartbeat with the
pending-invite count and the most recent welcome/giftwrap line from wnd
stderr. On timeout the script also prompts the operator to paste the
Amethyst-side MarmotDbg logcat so both ends of the gift-wrap pipe end up
in one log file.
wn prints either JSON (with --json) or a yaml-ish "key: value" pretty
form depending on the command and build. On the user's macOS run,
create-identity printed yaml ("pubkey: npub1..."), the subsequent
wn --json whoami didn't return the expected JSON shape, and
ensure_identity bailed with "could not determine npub for B".
Add a shared extract_pubkey helper in lib.sh that tries, in order:
1. JSON object .pubkey / .npub / .public_key
2. JSON array .[0].pubkey / .[0].npub / .[0].public_key
3. yaml-ish "pubkey: ..." via sed
4. yaml-ish "npub: ..." via sed
ensure_identity and prompt_for_a_npub now use it, and also fall back
to the non-JSON whoami output if --json returns nothing. On total
failure, the raw output is echoed to the log for diagnosis.
Also made npub_to_hex best-effort with the same double-try approach.
When hex lookup fails, it returns the npub unchanged — downstream
expect_contains assertions work against either form since wn's own
group listings may use either encoding.
https://claude.ai/code/session_01PacmbE1KKWMSRmDw3bnCNo
wnd only accepts --data-dir and --logs-dir. The socket path is
{data_dir}/release/wnd.sock (release build) or {data_dir}/dev/wnd.sock
(debug build) per src/cli/config.rs in whitenoise-rs. Updated the
script to point B_SOCKET / C_SOCKET at the correct release path and
dropped --socket from the wnd invocation.
Reported by user running the harness on macOS:
error: unexpected argument '--socket' found
Usage: wnd --data-dir <PATH> --logs-dir <PATH>
https://claude.ai/code/session_01PacmbE1KKWMSRmDw3bnCNo
Interactive bash script under tools/marmot-interop/ that validates
Amethyst's Marmot/MLS implementation against the whitenoise-rs wn/wnd
CLI. Builds wn/wnd from source on first run, launches two daemons for
Identities B and C, configures public (or --local-relays) Nostr relays
shared with Amethyst, then walks a human operator through 13 scenarios
covering MIP-00 KeyPackages, MIP-01 metadata, MIP-02 Welcome,
MIP-03 group messages, admin changes, reactions, concurrent-commit race,
leave, offline catch-up, and KeyPackage rotation. Push-notification
(MIP-05) test opt-in via --transponder.
No Amethyst code is modified — black-box testing only. State (daemon
sockets, logs, whitenoise-rs checkout, results TSVs) is kept under
tools/marmot-interop/state/ and gitignored.
https://claude.ai/code/session_01PacmbE1KKWMSRmDw3bnCNo
Claude