Adds `tools/marmot-interop/dm-interop-headless.sh` — a zero-prompt
end-to-end harness that runs two independent `amy` processes against
a loopback nostr-rs-relay and verifies the NIP-17 DM surface.
Six scenarios:
dm-01 text round-trip (kind:14) in both directions
dm-02 `dm list` surfaces prior exchange with `type:text` discriminator
dm-03 strict kind:10050 refuses sends to an inboxless recipient
(surfaces the `no_dm_relays` error JSON)
dm-04 `--allow-fallback` opts into NIP-65 read / bootstrap chain
(asserts a `relay_source: "bootstrap"` or `"nip65_read"` row)
dm-05 file message reference mode (kind:15) — send-file with
--key / --nonce, verify recipient decodes url + key + nonce + mime
dm-06 no-flag `dm list` advances `state.giftWrapSince`; second call
returns an empty `messages` array
Shape matches the existing Marmot harness: reuses `lib.sh` for logging
+ result tracking, reuses `setup.sh` for the nostr-rs-relay lifecycle
(`start_local_relay` / `stop_local_relay`), adds a slim `setup-dm.sh`
preflight (amy + relay only, no whitenoise-rs / Marmot patches), and
defines tests in `tests-dm.sh`.
No new CI job yet — the relay build needs Rust + ~3 minutes on a
cold cache, same constraint as the Marmot harness.
Upload-mode (`dm send-file --file PATH --server URL`) is not scripted
here: it needs a local Blossom server, which is out of scope for this
pass. The shared upload classes are unit-tested on desktop at
`desktopApp/src/jvmTest/kotlin/.../service/upload/`.
Interop Test 06 (wn removes a member from a group Amethyst is in) was
failing with:
GroupEventHandler.add: ERROR Failed to apply commit:
UpdatePath node count (1) doesn't match direct path length (2)
Repro scenario: 3-member group [B=leaf 0, C=leaf 1, A=leaf 2]; B (admin,
committer) removes C. After the Remove proposal applies, leaf 1 is
blank but leaf 2 (A) is still occupied — so leaf_count stays at 3 per
RFC 9420 §7.8 and the sender's direct path has length 2 ([node 1,
root]).
Per RFC 9420 §4.1.2 and §7.9 the **filtered direct path** drops any
parent node whose child on the copath has an empty resolution:
encrypting to such a parent is equivalent to encrypting to its only
non-blank child, which is already on the direct path. In our scenario
the parent at level 1 (parent of leaves 0, 1) has a blank leaf 1 on
the copath — empty resolution — so it's filtered out. Filtered
direct path length is 1, and that's what openmls / whitenoise-rs emit
in `UpdatePath.nodes<V>`.
Quartz was always using the unfiltered direct path on both sides
(send + receive + parent-hash chain + path-secret decrypt). That
worked for quartz↔quartz (both sides agreed), but broke the moment a
spec-correct sender (wn) sent a filtered UpdatePath into a quartz
receiver. Parent-hash chain is further affected because §7.9.2 walks
the filtered path — even if the size check passed, the hashes would
not match.
Fix it throughout:
- RatchetTree.filteredDirectPath: new helper returning parallel
(filteredDirectPath, filteredCopath) with the empty-resolution
entries dropped. Uses the existing `resolution()` helper, which
already handles unmerged_leaves per §4.1.2.
- RatchetTree.applyUpdatePath: size-check + target the FILTERED path.
- MlsGroup.commit (sender): emit one staged path node per filtered
level; encrypt path secrets using the filtered copath; patch parent
hashes into the filtered positions only.
- MlsGroup.processCommitInner (receiver): the UpdatePath node index
aligns to the filtered direct path, so the common-ancestor lookup
must find the filtered index to pick the right ciphertext + copath
resolution. Use the unfiltered index only for counting KDF steps
to the root (path_secret chain advances one step per unfiltered
level regardless of filtering — the chain is continuous; filtering
only decides which levels emit a UpdatePathNode, not how many KDF
steps separate them).
- MlsGroup.computeSenderParentHashes: walk the filtered direct path.
Map each filtered node back to its unfiltered level so the
preUpdateSiblingHashes lookup (indexed by unfiltered level) still
resolves. This makes the parent_hash chain agree with §7.9.2 and
with what openmls computes on the other side.
- MlsGroup.verifyParentHash: short-circuit on empty filtered path
rather than empty unfiltered path.
New regression test: testRemoveMiddleLeaf_ReceiverAcceptsCommit
reproduces the 3-member remove-middle scenario end-to-end within
quartz. It passed before this patch (because both sides were
unfiltered and agreed with each other), and it passes after this
patch (because both sides are now filtered and still agree) — the
compatibility win is that quartz now also agrees with
openmls/wn-produced commits of the same shape.
Also fix an unrelated script bug in marmot-interop.sh: the
discover_a_relays SQL probe was falsely reporting "wn has NO cached
relay entries for A" even when welcome delivery was working, because
the query assumed users.pubkey stored as BLOB but wn stores it as
TEXT (hex) in the current schema. Accept either encoding.
WhiteNoise threads its kind:445 payloads across three inner kinds:
kind:9 for chat, kind:7 for emoji reactions, kind:5 for unreacts. The
Amethyst ingest pipeline was routing every inner event into the group
chatroom feed, so a kind:7 reaction rendered as a chat bubble whose
only content was the emoji, quoting the liked message via the target
`e` tag. That looked identical to a threaded reply. The actual kind:9
reply from `wn messages send --reply-to` never arrived, which made the
two look swapped in the UI.
Three changes to untangle this:
- MarmotGroupList.addMessage/restoreMessage now skip inner events with
kind:5 and kind:7 before they enter `MarmotGroupChatroom.messages`.
The reaction is still consumed by LocalCache so it attaches to the
target note's reaction row, and the deletion still revokes that
reaction — they just don't appear as standalone bubbles.
- LocalCache.computeReplyTo learned to derive thread parents for
`ChatEvent` (kind:9) from plain NIP-10 `e` tags in addition to the
existing NIP-18 `q` tag path. WhiteNoise emits `e`-tagged replies;
without this the reply bubble had no quote context in the feed.
- tools/marmot-interop/marmot-interop.sh: `wn messages send` exposes
its `reply_to` field through clap v4, which renames snake_case to
kebab-case by default. The script was passing `--reply_to`, which
clap rejected; the `|| true` + redirected stderr hid the error and
no reply was ever published. Use `--reply-to`.
https://claude.ai/code/session_01K3g1uWLhByoEdBS77zdF32
Post-v0.2 wn wraps `--json groups show` output in {"result": {...}},
so `jq '{name, epoch}'` was reading the top level where those keys
don't exist — Test 10 always logged "B state: {name: null, epoch: null}"
and the operator couldn't tell whether the race produced the expected
converged state on wn's side.
Peel the .result wrapper before extracting the fields; same pattern
already used everywhere else in the script that parses wn JSON.
The reverse-rename leg issued `wn_b groups rename` on Interop-02, but
Amethyst created Interop-02 — A is the admin, B is a plain member.
MIP-03's `enforceAuthorizedProposalSet` rejects any GroupContextExtensions
commit from a non-admin, so wn's rename was being dropped client-side,
the metadata commit never reached any relay, and Amethyst's UI (correctly
waiting for a kind:445 on Interop-02) never saw "Interop-02-reverse".
Split Test 07 into two separately-recorded legs so each uses a group
where the renamer is actually the admin:
07a — A (admin of Interop-02) renames to "Interop-02-renamed" via
the Amethyst UI; wn-side poll observes the new name.
07b — B (admin of Interop-03) renames to "Interop-03-renamed" via
`wn_b groups rename`; operator confirms Amethyst picked it up.
Each leg individually records pass/fail/skip so a missing GROUP_03
(Test 03 skipped) doesn't take out the whole test.
Test 06 was asking the operator to remove C from Interop-05 via the
Amethyst UI — but Interop-05 was created by B (wn), so B is the sole
admin and A is a plain member. MIP-03's authorization gate
(`enforceAuthorizedProposalSet` in `MlsGroup.kt:1842`) rejects any
Remove proposal from a non-admin, so the UI correctly fires "not an
admin" and the test fails against its own author's assumption — the
test was misaligned with the admin model, not the code.
Flip the flow: B (admin) issues the removal via `wn groups
remove-members <gid> <C_NPUB>`, and A / C observe the effect. That
still exercises the same intended invariants:
- B + A remain in the group, commit + epoch advance propagate.
- A's Amethyst UI shows C gone (visible confirm).
- A sends "after removing C" → B decrypts (expected).
- C cannot decrypt the new message (forward secrecy at the new
epoch — C's retained keys only open the pre-remove epoch).
The "Amethyst admin issues remove" path is already separately
covered by Test 02 (A creates the group → A is admin → A removes)
and Test 08 (promote A, then A issues an admin-only action). Test
06's focus is forward secrecy after remove, not who may initiate —
so having B drive it is the spec-correct shape.
The interop harness was forcing specific relays on the Amethyst account —
"configure Amethyst with these five relays, also add them to your DM
Inbox list, also to your Key Package Relays, then republish" — which
made the test artificial:
- It masked the real failure modes (wn publishing to an A-advertised
relay wn can't reach, A's 10050 pointing at auth-required relays,
etc.) because everyone was always on the same happy-path set.
- It mutated the operator's real Amethyst account config — those
five relays persist after the run.
- It short-circuited wn's discovery chain (kind:10002 → 10050 →
10051 resolution), so the one path most likely to hold subtle bugs
was never exercised under divergent configs.
Real-world interop is: both clients have their own independently-chosen
relay sets and use the advertised kind:10002/10050/10051 to find each
other. Make the harness match that:
- Keep the five-relay set as the wn B/C bootstrap (they still need
somewhere to publish their own identity). Drop the "configure
Amethyst to match" steps from `instruct_amethyst_setup`;
confirmation only now ("is A logged in as <npub>, and has A
published its four lists to any public relay?").
- Add `discover_a_relays` step between configure + the operator
prompt. Calls `wn keys check $A_NPUB` on both daemons to trigger
wn's targeted fetch for kinds [0, 10002, 10050, 10051] (the
only side-effect available to populate wn's user_relays cache).
Then reads the cache directly via sqlite3 against wn's DB and
prints what wn actually learned, grouped by relay_type. Warns
loudly if A's 10050 shares zero relays with wn's bootstrap — the
exact failure mode that caused the prior round of "Test 03
silently times out" reports.
- `--local-relays` path keeps the old behavior: offline/sandbox
mode, the harness owns the relay so forcing Amethyst onto it is
correct.
Scope is the harness only. No Amethyst / quartz / commons changes.
Test 03 (wn creates a group, invites Amethyst) was failing at the
operator-visible level — "I don't see Interop-03 on Amethyst" — with
no clear signal in the Amethyst logcat because no kind:1059 welcome
gift wrap ever arrived.
Root cause: when wn invites Amethyst it fetches A's kind:10050 inbox
list and publishes the welcome gift wrap there. Amethyst's built-in
DM inbox defaults advertise auth.nostr1.com + relay.0xchat.com, which
wn (connected only to the five harness public relays) cannot reach.
Unless the operator manually adds the harness relays to Amethyst's
DM Inbox list, the gift wrap lands on relays the reader isn't
subscribed to, and the invite is silently dropped.
Confirmed empirically by adding wss://nostr.bitcoiner.social (one of
the harness relays) to Amethyst's DM Inbox list and watching
Interop-03 appear on the next wn create.
Update `instruct_amethyst_setup` to make the DM Inbox step explicit
and call out the specific failure mode it prevents. Also add a nudge
to verify NIP-65 + kind:10050 were actually republished after the
relay list edits.
bash 3.2 (stock on macOS) raises "unbound variable" on "${gids[*]}"
when the array is empty under `set -u`. The happy-path sanity check
still worked (the array was populated), but the shell error aborted
execution before the actual test cases ran on a fresh state/ directory.
Only expand when the array has at least one entry.
Broaden the default public relay set for the interactive harness so
kind:1059 gift wraps and kind:10050 inbox lists have more landing
spots that don't pre-filter Marmot traffic out. Both relays are
known to accept kinds outside the core "note / metadata / contacts"
trio that damus/primal tend to restrict.
The Amethyst-side setup instructions are rendered from the same list,
so operators who configure Amethyst per the harness prompt get the
same five URLs on the app side too.
wnd persists pending group invites across daemon restarts, so on any
rerun C (and B) already have unaccepted welcomes from previous runs
lying around. `wait_for_invite` was pulling `.[0]` out of the invite
list, which is the oldest pending — i.e. the stalest — and treating
it as the arrival the caller just triggered. In the sanity probe this
caused the kind:445 step to send to B's NEW group id while C was
accepting a DIFFERENT (old) group, producing a bogus "kind:445 failed
— relays may be dropping group messages" with no actual relay issue.
Add `snapshot_invites <B|C>` which returns the current gids as CSV,
and teach `wait_for_invite` to accept that CSV as an ignore list so
it walks past stale welcomes and only returns a freshly-arrived one.
Use the new snapshot→wait pattern at every call site: sanity check,
Test 02 (B), Test 04 (C), Test 05 (C), Test 08 (C), Test 12 (C).
Also log a gid-mismatch warning in the sanity check when B's created
group and C's accepted welcome disagree, as an extra guard against
the filter being bypassed.
Several public relays (damus.io in particular) silently drop kind:1059 /
10050 / 30443 from accounts they've never seen before — the sanity
check in the interactive harness reports exactly this failure mode
when the three-relay default set is in use, because wn's identities are
brand-new and have nothing on file.
Before the kind:30443 + 10050/1059/445 probes, publish a kind:0
profile event for each wn identity via `wn profile update --name
--about`. That marks both accounts as "known" on the relay and lets
the subsequent gift-wrap and KeyPackage publishes land. Runs inside
configure_relays (after relays are added, before the first sanity
probe) so the sequencing is: add relay -> advertise profile -> publish
KPs -> start exchanging welcomes.
Does not help when a relay outright refuses the kind (some relays
whitelist only kind:1 / 3 / 7); those still need --local-relays.
When `wn_b messages send` fires but Amethyst never displays the reply,
the harness used to swallow the send output and only ask the operator a
blind yes/no — giving no signal about which side of the pipe failed.
Amethyst subscribes to kind:445 on the MLS GroupContext `relays`
extension (falling back to `Account.homeRelays`), so the two
actionable failure modes are:
- wn published to a relay Amethyst isn't subscribed to (group-relay
mismatch between the MLS extension and Amethyst's subscription)
- Amethyst received the event but `GroupEventHandler.add` dropped it
(no `h` tag, `isMember=false`, decrypt error)
Teach the harness to dump both sides automatically on the reply step:
- Capture wn's `messages send` stdout/stderr into the run log.
- `dump_outbound_diagnostics` prints wn's local view of the group
messages (did the send persist to wn's own DB?), the MLS
GroupContext relay list (what Amethyst should be subscribed to),
and live relay connection status.
- On operator-reported fail, prompt for an Amethyst-side logcat
grep targeted at the exact log points — MarmotGroupEventsEoseManager
filter updates, GroupEventHandler.add arrivals, and the membership
drop branch — naming the current group id so the operator can grep
for the right hex prefix.
`wn users show <npub>` only resolves identities the local daemon has
cached, so A's npub (the Amethyst account, which B's daemon has never
interacted with directly) falls through to the "return input unchanged"
branch. That leaks the raw npub into the Test 02 member-list assertion,
where it's compared against hex pubkeys pulled from `wn groups members`
and fails with "member list missing npub1...".
Add a pure-awk bech32 decoder and use it as the final fallback in
`npub_to_hex`, so unknown npubs still convert to the 32-byte hex form
the daemon emits. Also harden the JSON path to peel a `.result` wrapper
in case a future `wn --json users show` ships one.
The interactive harness was failing with the same
KeyringEntryMissingForExistingDatabase error as the headless one —
the MLS SQLite DB on disk references a keyring entry that no longer
exists (keychain entry pruned, data dir restored out of band, or a
previous run used the mock keyring). wnd can't open the DB in that
state, so the daemon exits before its socket appears.
Unlike the headless harness (which wipes unconditionally because its
mock keyring is always ephemeral), the interactive harness uses the
real OS keyring and benefits from preserving B/C identities across
runs. So: try once, and only wipe + retry when stderr actually shows
the keyring-missing error. Also dump stderr/stdout tails inline on
final failure so the operator doesn't have to chase a log path.
Apply the same SIGINT/SIGTERM/SIGHUP trap hardening to the interactive
harness that just landed on the headless one. Without it, killing the
script mid-run leaves the nohup'd wn daemons alive and the next run
fails in preflight because the sockets are still held.
The previous trap only fired on the EXIT pseudo-signal, which doesn't
always run cleanly when the script is killed mid-flight — leaving the
nohup'd wnd daemons (and the local relay) alive and still holding the
port. Route SIGINT/SIGTERM/SIGHUP through an explicit `exit`, use a
single idempotent cleanup handler, and preserve the exit code so the
harness still reports failure when killed.
The headless harness runs wnd with WHITENOISE_MOCK_KEYRING=1, but the
mock keyring is in-memory only — it starts empty on every wnd restart.
Meanwhile the SQLite databases under $data_dir persist across runs and
reference keys that no longer exist, so the second run onward fails at
startup with KeyringEntryMissingForExistingDatabase before wnd can even
open its control socket. Wipe everything under the daemon's data dir
(except logs and pid) before each start so wnd always comes up cold and
consistent with its ephemeral keyring.
Also surface failure diagnostics inline instead of burying them in a log
file path: print the last 40 lines of stderr and 20 of stdout, report
whether wnd is still alive or already exited, and break out of the 30s
wait loop the moment the daemon dies.
Two harness-side bugs that made a broken wn silently look like a working one:
1. whitenoise-skip-unprocessable-retry.patch's hunk header declared
`@@ -178,7 +178,23 @@` but the new side only has 22 lines. GNU patch
bails with "malformed patch at line 26" and leaves the target file
untouched. Fix the count to +178,22 so the patch actually applies.
2. setup.sh's patch-apply loop piped `patch` through `tee`, which
masked patch's exit code behind tee's. A miscounted hunk therefore
still touched the `.headless-patched-*` marker and the next run
skipped the "patch" step entirely, so the resulting wn ran the
stock 10-retry exponential backoff on every MlsMessageUnprocessable
— ~17min per event, which pegs the per-account serial event
processor and makes every later test timeout "just because".
Check patch's real exit status; fail preflight loudly instead.
Same class of bug was also hiding cargo build failures: the
`cargo build ... | tee` pipeline reported success even when rustup
couldn't fetch the toolchain manifest (503 from static.rust-lang.org)
or cargo couldn't hit the crates.io index (503 from fastly). setup.sh
then `info`-logged the expected binary paths and happily moved on —
until the daemon launch tripped over `nohup: No such file or directory`.
Now each cargo build runs in a 4-attempt retry loop that terminates
only when the binary actually exists on disk, and fails preflight
loudly if it never materialises. Matches the jitpack 503 retry
behaviour already in place for `:cli:installDist`.
https://claude.ai/code/session_018kSBco5VVfctkW6vAm7NzA
jitpack.io and dl.google.com both return 503 on a non-trivial fraction
of cold-cache fetches, and Gradle disables the whole repository for the
rest of the build the moment a single 503 lands — so one bad roll
aborts the entire harness before any test gets to run. Wrap the gradle
invocation in a 4-attempt retry loop; each attempt resumes from
Gradle's local cache so only the still-missing artifacts get re-fetched.
In practice the retry completes in seconds when jitpack is the only
flake, and the whole preflight still wall-clocks well under the first
attempt's runtime. Existing success path is unchanged.
https://claude.ai/code/session_018kSBco5VVfctkW6vAm7NzA
The `.result` wrapper fix landed in lib.sh's pollers and the headless tests
back in f08b010f, but the 8 inline `jq` reads in marmot-interop.sh were
missed — which silently returned empty against current `wn` and made every
UI-verification test look like "Amethyst didn't propagate X" when in fact
the underlying wn state was correct and the script just couldn't see it.
All 8 reads now peel `(.result // .)` the same way the lib.sh helpers do,
staying backwards-compatible with the pre-v0.2 flat shape:
* test 02, 04 member lists
* test 06 member-removed check
* test 07 group-name poll
* test 08 admin list (promote + demote)
* test 09 anchor message lookup
* test 11 leave-group member check
Also split the configure_relays sanity check into three relay-level
surfaces so a broken relay set points at the specific kind it can't
handle, instead of 10 later tests all timing out with the same "no
invite" symptom:
* kind:30443 (KeyPackage) — both sides publish + symmetric discover
* kind:10050 (Inbox advertise) — C reachable as giftwrap target
* kind:1059 (Gift wrap) — B->C welcome delivery
* kind:445 (MLS commit/msg) — real group-message round-trip
And drop the orphan whitenoise-mdk-plaintext-policy.patch: it lived in
`headless/patches/` but was never referenced by setup.sh's patch array,
and if it ever were applied it would flip mdk-core's MLS wire-format
from MIXED_CIPHERTEXT to MIXED_PLAINTEXT — a protocol-breaking
divergence from every stock whitenoise-rs / White Noise build. Removing
it eliminates the trap of a future contributor "enabling" it.
https://claude.ai/code/session_018kSBco5VVfctkW6vAm7NzA
Quartz:
- processWelcome now enforces hash(ratchet_tree) == GroupContext.tree_hash
and verifies the joiner-derived confirmation_tag matches GroupInfo's,
closing the gap where a tampered but-signed GroupInfo could diverge
the joiner's epoch secrets.
- externalJoin does the equivalent tree_hash check.
- Promoted constantTimeEquals to a file-level helper so the companion
object (processWelcome) can use it alongside instance methods.
Tests:
- New MlsGroupNegativeTest exercises every authenticity knob on inbound
commits: tampered confirmation_tag, bit-flipped signature, spoofed
senderLeafIndex, wrong wire_format, empty confirmation_tag, and
tampered/missing membership_tag — each must throw and leave the
receiving group's epoch unchanged (atomic-rollback regression cover).
- Fixed stale jvmAndroidTest references to the removed `selfRemove()`
helper — use `buildSelfRemoveProposalMessage()` (now Pair-returning).
Interop harness:
- Adds tests 14–16 as inverted-role scaffolding (wn drives, amy receives).
Currently recorded as SKIP with explicit notes:
14 / 15 block on filtered-direct-path UpdatePath support in
quartz's applyUpdatePath (RFC 9420 §7.7 path compression).
16 blocks on a createdAt-sorted KeyPackage fetch path — the current
fetchFirst-based fetcher is non-deterministic on relay order.
Preserves the test functions so they start running the moment those
gaps are closed.
Test 07 second half exercises a rename from the wn side, but
GROUP_02 is created by amy, so amy is the sole admin. wn's
ensure_account_is_group_admin silently refuses B's rename and
never publishes anything. The test design assumed a symmetric
admin set; the MIP-01 admin check makes that explicit.
Add an `amy marmot group promote <B>` step between amy's rename
and B's rename so B is admin when it tries, matching the real
round-trip intent. Also sleep 3s to let wn apply the promote
commit before firing the rename.
https://claude.ai/code/session_1469d7f4-bb66-4ffa-a44d-1dfa4b526484
MDK-core (openmls) defaults group configuration to
`MIXED_CIPHERTEXT_WIRE_FORMAT_POLICY` — outgoing commits / proposals
ship as PrivateMessage. Quartz only handled PrivateMessage for
ContentType.APPLICATION; any handshake message came in via the
application ratchet, consumed the first application generation on
whatever sender sent an app message next, and then died with
`Generation 0 already consumed` the moment the receiver saw the real
PrivateMessage commit. That's why every A-side processing of B's
rename / promote / remove / leave commit timed out.
Fix:
1. SecretTree grows a parallel `handshakeKeyNonceForGeneration`
path, using the per-sender `handshakeSecret` / `handshakeGeneration`
the struct already tracked — with its own skipped-keys cache and
replay-detection set so handshake and application generations
never clash.
2. `MlsGroup.decrypt()` dispatches on the PrivateMessage's
`content_type` BEFORE consuming any ratchet — COMMIT and PROPOSAL
take the handshake path; APPLICATION stays on the existing
application path.
3. COMMIT bodies decode as `Commit || signature<V> || confirmation_tag<V>
|| padding` (RFC 9420 §6.3.1) and route into `processCommit` inline,
so the caller just sees the epoch advance — no new public API.
PROPOSAL still rejects (standalone proposals aren't used by
Marmot flows yet) with a clear message instead of a cryptic
generation-consumed error.
https://claude.ai/code/session_016kAxdp6ubB5CnF9URhCEzP
Narrowing the previous patch to only MlsMessageUnprocessable /
MlsMessagePreviouslyFailed still leaves wn stuck retrying
\`MdkCoreError("Failed to decrypt message with any exporter secret")\`
— mdk surfaces that as an Err variant rather than the Unprocessable
result enum, so it took the full 10-attempt exponential backoff (~17
minutes) before giving up. That was enough to block later
decryptable commits and regress test 11 (leave group) from pass back
to fail.
mdk-core doesn't retry internally, so ANY Err it returns is terminal
by construction. Treating the whole \`MdkCoreError\` variant as
one-shot is therefore equivalent to "trust mdk's verdict" rather
than "permissive skip" — if mdk decides the message is bad, it will
stay bad.
Net in the headless harness: 6/13 pass (up from 5/13 with the
narrower patch and 1/13 baseline).
https://claude.ai/code/session_016kAxdp6ubB5CnF9URhCEzP
mdk-core's `MessageProcessingResult::Unprocessable` means the received
kind:445 is genuinely outside this member's decrypt horizon — it's from
a pre-membership epoch, or an epoch we've retained past, and no amount
of retrying will make it decryptable. wn's account event processor was
running that down the same exponential retry ladder as a legitimate
transient failure (10 attempts, 1s..512s backoff, ~17 minutes to
exhaust), which in the interop harness meant every later decryptable
commit — add-member, rename, leave — had to wait behind a queue of
doomed retries and raced the 30s test timeout.
Add a third wn patch, `whitenoise-skip-unprocessable-retry.patch`,
that treats `MlsMessageUnprocessable` and `MlsMessagePreviouslyFailed`
as terminal: log the warning once, skip the reschedule. Applied in
preflight via `setup.sh`.
https://claude.ai/code/session_016kAxdp6ubB5CnF9URhCEzP
Five protocol-level fixes and a batch of harness correctness fixes to get
the headless Marmot/Whitenoise interop harness from 1/13 to 5/13 passing
cleanly, with the remaining failures all rooted in wn's per-account
serial event-processor retry backoff (which drops undecryptable
pre-membership commits after several minutes) rather than amy behaviour.
quartz + commons
----------------
* MarmotGroupData: hold CURRENT_VERSION at 2. mdk-core (the Rust MLS
engine used by whitenoise-rs) strict-rejects v3 payloads with
`ExtensionFormatError("Trailing bytes in NostrGroupDataExtension")`
— our v3 welcomes and GCE commits never apply, so every cross-client
group flow dies at welcome processing. We still parse v3 happily on
the way in; we just don't emit it until mdk publishes the
forward-compat fix MIP-01 mandates.
* MarmotManager.updateGroupMetadata: MERGE extensions instead of
REPLACING. RFC 9420 §12.1.7 says GCE proposals blow away the old
extension list; callers that pass only [marmot_group_data] dropped
[required_capabilities], which peers then reject. Preserve every slot
except the one we're updating.
* MarmotManager.createGroup: new optional `initialMetadata` parameter
that bakes MarmotGroupData into epoch-0 GroupContext.extensions
directly. Without it, creators had to publish a pre-membership
"bootstrap" commit that no later joiner could decrypt — each such
peer then burned their retry budget on an undecryptable kind:445
before seeing the real state. Threaded through MlsGroup.create /
MlsGroupManager.createGroup.
* MarmotManager.mlsGroupIdHex: new translation helper so any code
juggling the MIP-01 nostr_group_id (what amy indexes on) and the
MLS GroupContext groupId (what mdk indexes on) can cross-reference
them without reaching into MlsGroupManager directly.
amethyst module
---------------
* Account.leaveMarmotGroup: self-demote before SelfRemove per MIP-01,
and promote a surviving member to admin first if the caller is the
sole admin (otherwise we'd throw "admin depletion"). Matches the
cli/GroupMembershipCommands.leave flow.
cli (amy)
---------
* Context.syncIncoming: don't advance `giftWrapSince` on empty polls
(so the first-ever sync doesn't bump the cursor past every
past-timestamped wrap we've ever been sent), subtract 2 days lookback
when filtering (NIP-59 randomWithTwoDays gift wraps can have any
createdAt in the last 48h), and only advance `groupSince` for groups
we actually received events for.
* Context.syncIncoming: after ingest, if any Welcome consumed a
KeyPackage, rotate and publish a fresh one immediately. MIP-00
requires this — a KP can only be welcomed once and leaving the
consumed one on relays just means later senders invite us with a
bundle we no longer have private keys for.
* Context.resolveGroupId: accept either nostr_group_id (amy's primary
key) or the MLS GroupContext groupId (what wn emits) on every verb
that takes a group id. Wired through GroupAdd/Remove/Leave/Metadata
/Read, Message send/list, and all the await* verbs so harness
scripts never have to juggle both forms for a single group.
* GroupCreateCommand: bake initial metadata into epoch 0 (see the
quartz change above). Dropped the now-redundant bootstrap commit
publish and tightened the JSON output to include `mls_group_id`.
* GroupMembershipCommands.leave: self-demote admin before SelfRemove;
promote an heir if we're the only admin, otherwise the GCE would
deplete admins and the leave aborts.
* MarmotIngest.ingestGiftWrap: unwrap the sealed-rumor layer. NIP-59
wrap is gift-wrap(kind:1059) → seal(kind:13) → rumor; the old code
only unwrapped once and then checked `inner.kind == 444`, which is
always false because inner is actually the seal. Unseal once more
before the Welcome check. This single fix is what unsticks every
amy-side Welcome ingestion.
wn harness patches + scripts
----------------------------
* whitenoise-defaults-env.patch: honour $WHITENOISE_DISCOVERY_RELAYS in
`Relay::defaults()` (release builds otherwise bake damus.io / primal
/ nos.lol into every new account's NIP-65 / Inbox / KeyPackage
lists, which breaks publishing and prevents the inbox subscription
plane from ever reaching an operational state in a sandbox).
* setup.sh: sleep 2s after amy's initial kind:30443 publish so
nostr-rs-relay has a chance to fsync before wn's first targeted
discovery query. Without it wn's `keys check` races the relay's
WAL flush and intermittently returns NotFound.
* lib.sh: peel wn's `{"result": …}` wrapper in `jq_group_id`,
`wait_for_invite`, `wait_for_message`, `wait_for_member`. Post-v0.2
wn `--json` output nests everything under `.result` (and
`groups invites[]` nests further under `.group.mls_group_id`) —
these helpers were still pattern-matching on the flat shape, so
they returned empty strings for a perfectly good response.
* tests-{create,manage,extras}.sh: track both group IDs per test
(amy's nostr + wn's MLS), pass each CLI the id it understands, and
bump the post-commit wait timeouts to 90–120s so wn's exponential
retry backoff has time to work through the pre-membership commits
it can't decrypt and get to the ones it can.
https://claude.ai/code/session_016kAxdp6ubB5CnF9URhCEzP
wn now wraps account listings in {"result": [{...}]} so extract_pubkey's
object + array fallbacks both missed it and ensure_identity bailed with
"could not determine \$who npub" immediately after create-identity.
Also: create-identity on a fresh box can exit non-zero with
"failed to connect to any relays" even when the account *was* created
locally. Probe --json whoami afterwards before calling that a fatal.
https://claude.ai/code/session_01M6dCKAF5Y1VyHGZPjzwDXq
Containers / CI often block the keyutils syscalls wnd uses by default
to store secret keys (add_key returns EACCES, keyctl_search returns
ENOSYS). The integration-tests feature ships a mock keyring store, but
the stock wnd binary doesn't wire it up. Add a tiny opt-in patch.
Changes:
- headless/patches/whitenoise-mock-keyring.patch: if built with the
integration-tests feature and \$WHITENOISE_MOCK_KEYRING is set, wnd
calls Whitenoise::initialize_mock_keyring_store() before the normal
init path. Harmless on production builds.
- headless/setup.sh: apply both harness patches generically from a
list; build wn/wnd with --features cli,integration-tests; export
WHITENOISE_MOCK_KEYRING=1 alongside WHITENOISE_DISCOVERY_RELAYS when
launching each daemon.
- preflight swaps keyctl for patch in required tools — we no longer
need a real session keyring.
https://claude.ai/code/session_01M6dCKAF5Y1VyHGZPjzwDXq
Without this, wnd exits on startup with NoRelayConnections in any
environment that can't reach wnd's baked-in public discovery relay
set (nos.lol, relay.damus.io, …). The headless harness runs entirely
on a loopback relay, so hitting the public internet is never OK.
Changes:
- headless/patches/whitenoise-discovery-env.patch: adds an env-var
override at the top of DiscoveryPlaneConfig::curated_default_relays.
When \$WHITENOISE_DISCOVERY_RELAYS is set (comma-separated) we use
that list instead of the hardcoded public one.
- headless/setup.sh: applies the patch in preflight (idempotent —
checks for a .headless-discovery-patched marker), invalidates the
previous wn/wnd build on first patch, rebuilds, and threads
\$WHITENOISE_DISCOVERY_RELAYS=\$RELAY_URL into every start_daemon
invocation.
https://claude.ai/code/session_01M6dCKAF5Y1VyHGZPjzwDXq
nostr-rs-relay's build.rs needs protoc to compile the nauthz gRPC
definitions. Without it the build fails deep in prost-build with a
panic that's hard to read, so fail up front with a pointer to
`apt-get install protobuf-compiler`.
https://claude.ai/code/session_01M6dCKAF5Y1VyHGZPjzwDXq
Every test now flows through one loopback nostr-rs-relay on
ws://127.0.0.1:8080 — no public relays, no egress. Addresses the
repeated 503s we saw from the sandbox and the kind:445 rejections we
hit on public relays even with internet.
Changes:
- Preflight clones https://github.com/scsibug/nostr-rs-relay into
state-headless/nostr-rs-relay and runs `cargo build --release`
(~3 min first run). Cache hits on subsequent runs.
- New start_local_relay / stop_local_relay functions render a minimal
config.toml each run (binds to 127.0.0.1, no kind blacklist, data
under state-headless/relay) and wait up to 20s for the port.
- configure_relays collapsed to a single-relay path — A/B/C all point
at \$RELAY_URL. Kept publish-lists + key-package publish as smoke
signals for the advertise path.
- Flags pared down: drop --local-relays (always local now), add
--port N for when 8080 is taken. --no-build still honoured.
- trap wires relay shutdown alongside daemon shutdown.
https://claude.ai/code/session_01M6dCKAF5Y1VyHGZPjzwDXq
Matches the 13 scenarios in marmot-interop.sh but runs A via the new
`amy` CLI instead of prompting the human. Identities B and C still go
through `wn`/`wnd` from whitenoise-rs.
Layout:
- marmot-interop-headless.sh — top-level entrypoint
- headless/setup.sh — preflight, daemon lifecycle, identities
- headless/helpers.sh — amy wrappers + assertion helpers
- headless/tests-create.sh — tests 01..05
- headless/tests-manage.sh — tests 06, 07, 08, 11
- headless/tests-extras.sh — tests 09, 10, 12, 13
Reuses lib.sh for colours, record_result, wait_for_invite/message,
jq_group_id, and dump_daemon_diagnostics.
Keeps the interactive marmot-interop.sh intact for final UI sign-off.
https://claude.ai/code/session_01M6dCKAF5Y1VyHGZPjzwDXq
Adds a dump_daemon_diagnostics helper (relays per type, wn debug health,
relay-control-state, pending invites, wnd stderr tail) and wires it into
Test 02 and Test 04 on both sides of the poll (pre-invite baseline +
post-timeout). wait_for_invite now prints a ~10s heartbeat with the
pending-invite count and the most recent welcome/giftwrap line from wnd
stderr. On timeout the script also prompts the operator to paste the
Amethyst-side MarmotDbg logcat so both ends of the gift-wrap pipe end up
in one log file.
wn prints either JSON (with --json) or a yaml-ish "key: value" pretty
form depending on the command and build. On the user's macOS run,
create-identity printed yaml ("pubkey: npub1..."), the subsequent
wn --json whoami didn't return the expected JSON shape, and
ensure_identity bailed with "could not determine npub for B".
Add a shared extract_pubkey helper in lib.sh that tries, in order:
1. JSON object .pubkey / .npub / .public_key
2. JSON array .[0].pubkey / .[0].npub / .[0].public_key
3. yaml-ish "pubkey: ..." via sed
4. yaml-ish "npub: ..." via sed
ensure_identity and prompt_for_a_npub now use it, and also fall back
to the non-JSON whoami output if --json returns nothing. On total
failure, the raw output is echoed to the log for diagnosis.
Also made npub_to_hex best-effort with the same double-try approach.
When hex lookup fails, it returns the npub unchanged — downstream
expect_contains assertions work against either form since wn's own
group listings may use either encoding.
https://claude.ai/code/session_01PacmbE1KKWMSRmDw3bnCNo
wnd only accepts --data-dir and --logs-dir. The socket path is
{data_dir}/release/wnd.sock (release build) or {data_dir}/dev/wnd.sock
(debug build) per src/cli/config.rs in whitenoise-rs. Updated the
script to point B_SOCKET / C_SOCKET at the correct release path and
dropped --socket from the wnd invocation.
Reported by user running the harness on macOS:
error: unexpected argument '--socket' found
Usage: wnd --data-dir <PATH> --logs-dir <PATH>
https://claude.ai/code/session_01PacmbE1KKWMSRmDw3bnCNo
Interactive bash script under tools/marmot-interop/ that validates
Amethyst's Marmot/MLS implementation against the whitenoise-rs wn/wnd
CLI. Builds wn/wnd from source on first run, launches two daemons for
Identities B and C, configures public (or --local-relays) Nostr relays
shared with Amethyst, then walks a human operator through 13 scenarios
covering MIP-00 KeyPackages, MIP-01 metadata, MIP-02 Welcome,
MIP-03 group messages, admin changes, reactions, concurrent-commit race,
leave, offline catch-up, and KeyPackage rotation. Push-notification
(MIP-05) test opt-in via --transponder.
No Amethyst code is modified — black-box testing only. State (daemon
sockets, logs, whitenoise-rs checkout, results TSVs) is kept under
tools/marmot-interop/state/ and gitignored.
https://claude.ai/code/session_01PacmbE1KKWMSRmDw3bnCNo
All log messages go through send_log_to_java() → Kotlin ArtiLogCallback
→ Log.d("TorService"), which already writes to logcat. The android_logger
module was a second FFI call to __android_log_write that duplicated
every line.
https://claude.ai/code/session_01BApgDd5udqBzMqysSRMpZu
All features and APIs verified present in 0.41:
- tokio, rustls, compression, onion-service-client, static-sqlite
- TorClient::create_bootstrapped, from_directories, connect()
https://claude.ai/code/session_01BApgDd5udqBzMqysSRMpZu
- Set default-features = false on arti-client and tor-rtcompat
- Removed bridge-client (UI doesn't expose bridge config yet)
- Narrowed tokio features from "full" to only what the SOCKS proxy
needs: rt-multi-thread, net, io-util, time, macros
Kept: tokio, rustls, compression, onion-service-client, static-sqlite
(all required for Amethyst's .onion relay support)
https://claude.ai/code/session_01BApgDd5udqBzMqysSRMpZu
The Guardian Project's arti-mobile-ex AAR has three problems:
1. No 16KB page-aligned binaries (required for Google Play)
2. ArtiProxy's stop()+start() causes state file lock conflicts
(lock is tied to TorClient object lifetime, released only on GC)
3. ~140MB AAR size
Replace with a custom JNI bridge built from Arti source, following
BitChat's proven approach:
Build tooling (tools/arti-build/):
- build-arti.sh: Clones official Arti, compiles with cargo-ndk
for ARM64 + x86_64, NDK 25+ for 16KB page alignment
- Cargo.toml: Minimal deps with size-optimized release profile
- src/lib.rs: Custom SOCKS5 proxy with proper lifecycle:
- initialize() creates TorClient once (holds state lock forever)
- startSocksProxy() binds port and accepts connections
- stopSocksProxy() aborts listener only (TorClient stays alive)
This cleanly separates "stop routing traffic" from "destroy client"
Kotlin side:
- ArtiNative.kt: JNI declarations + ArtiLogCallback interface
- TorService.kt: Uses ArtiNative directly, start() initializes +
starts proxy, stop() only stops proxy (no lock issues)
- TorManager.kt: Restored stop() calls for OFF/EXTERNAL modes
since our native stop is now safe
Removed: arti-mobile-ex dependency from build.gradle and version catalog
Native libraries must be built separately:
cd tools/arti-build && ./build-arti.sh
https://claude.ai/code/session_01BApgDd5udqBzMqysSRMpZu
Claude