amethyst/nestsClient/plans/2026-05-06-cross-stack-interop-test-results.md

# Plan: cross-stack interop test (T16) — Phase 1 + Phase 2 results

**Status:** Phase 1 + most of Phase 2 landed. Phases 3–5 still deferred.

## Phase 2 update

Added on top of the Phase 1 scaffolding:

- **`hang-listen` real body** — connects to a `moq-lite-03` relay,
  reads the hang catalog, picks the first Opus / Container::Legacy
  audio rendition, decodes each Opus packet via the `opus = "0.3"`
  crate, and writes Float32 little-endian PCM to `--output-pcm`.
- **`hang-publish` real body** — claims a broadcast, publishes a hang
  catalog with one Opus rendition (track name configurable via
  `--track-name`, default `audio/data` to match Amethyst's
  `MoqLiteNestsListener.AUDIO_TRACK`), encodes a sine wave with
  libopus, and pumps Opus frames in 5-frame groups for `--duration`
  seconds. Uses `audiopus`-equivalent `opus = "0.3"`.
- Both binaries explicitly install the rustls aws-lc-rs crypto
  provider (rustls 0.23 no longer auto-installs) and use
  `--client-version moq-lite-03` + `--tls-disable-verify=true`
  to interop with the harness's self-signed `--tls-generate localhost`
  relay.
- **JVM Opus encoder/decoder** via `club.minnced:opus-java:1.1.1`
  (JNA bindings + bundled libopus.so / libopus.dylib / opus.dll
  natives). Lives in `nestsClient/src/jvmTest/.../audio/JvmOpusEncoder.kt`
  + `JvmOpusDecoder.kt`. Verified by
  `JvmOpusRoundTripTest.sine_440_round_trips_through_libopus` —
  encode → decode preserves the FFT peak at 440 Hz and the
  zero-crossing rate at 880/sec within 5%.
- **Real-time pacing** in `SineWaveAudioCapture` — `readFrame`
  blocks until the next 20-ms boundary, mirroring how a microphone
  source paces. Without this the broadcaster's read loop would
  flood the relay with millions of frames/sec.
- **`HangInteropTest.rust_hang_publish_to_rust_hang_listener_round_trip_440`**
  — Rust↔Rust round-trip through the harness. Spawns `hang-publish`
  + `hang-listen` as subprocesses, asserts the decoded PCM has FFT
  peak at 440 Hz, ZCR at 880/sec, and 5 s of samples (±20% slack
  for Opus look-ahead + relay buffering). Verified green on Linux
  x86_64.

## I1 — Amethyst speaker → hang-listen — green at `framesPerGroup=5`

Initial diagnosis (Kotlin speaker → hang-listen sees `Group {
subscribe, sequence }` headers but no frame payloads) was bisected
by adding `KotlinSpeakerKotlinListenerThroughNativeRelayTest` —
a Kotlin↔Kotlin path through the same `moq-relay` 0.10.x. That
test reproduced the failure too, ruling out a Kotlin↔Rust-specific
mismatch. Bisecting `framesPerGroup`:

  - `framesPerGroup = 1` (one frame per uni stream): **passes**
  - `framesPerGroup = 5` (the value
    `nestsClient/plans/2026-05-01-quic-stream-cliff-investigation.md`
    recommends): **passes**
  - `framesPerGroup = 50` (the repo's current
    `NestMoqLiteBroadcaster.DEFAULT_FRAMES_PER_GROUP`): **fails**

The 50-frame default writes ~6 KB onto a single uni stream over
~1 s, which exceeds moq-relay 0.10.25's per-subscriber forward
buffer; the relay forwards the Group control header but holds the
frame data, never delivering it downstream. This matches the
audit summarised in the cliff-investigation plan: the bug is a
moq-relay 0.10.x policy interacting with our publish cadence, not
a wire-format defect on either side.

`HangInteropTest.amethyst_speaker_to_hang_listener_static_tone_440`
now pins `framesPerGroup = 5` to match what the cliff plan
already documents as the safe production cadence. The Kotlin↔
Kotlin diagnostic test
`KotlinSpeakerKotlinListenerThroughNativeRelayTest`
also pins `framesPerGroup = 5` and is kept as a regression for
the cadence interaction — if a future relay bump changes the
ceiling, both tests will trip together and the failure will be
attributable in one place.

**Conflict between plans (worth a maintainer's eye, NOT auto-applied
here):** the 2026-05-01 cliff-investigation plan recommends
`DEFAULT_FRAMES_PER_GROUP = 5`, but the current code has `50`
with a kdoc citing later two-phone production tests on
`claude/fix-nests-audio-receiver-HCgOY` that showed `5`/`10`
hit a *different* listener-side cliff. The two are tuning for
different bottlenecks:

  - cliff-investigation plan ↦ relay-side per-subscriber forward
    queue overflow at high stream-rate (which our cross-stack
    tests against `moq-relay 0.10.25 --auth-public ""` reproduce
    cleanly — `50` flatly fails to deliver frames downstream)
  - HCgOY field tests ↦ listener-side cliff-detector recycling
    the transport on stream RST, which favours larger groups
    (fewer streams to lose)

These are NOT contradictory at the protocol level — they're
different failure modes triggered by different relay
configurations. The interop harness's `--auth-public ""` minimal
relay setup hits the first cliff; production's `nostrnests/nests`
deployment apparently lives in a regime where the second cliff
dominates. We pin `framesPerGroup = 5` in the test scenarios
because that's the value at which our test succeeds; production
keeps `50` because that's the value its field tests vetted.
Reconciling the two — either by getting both setups under a
single value, or by varying `framesPerGroup` per environment —
is left to a maintainer who can run both rigs.

**Origin:** companion to `nestsClient/plans/2026-05-06-cross-stack-interop-test.md`.
This file records what actually shipped in Phase 1, the deviations from
the spec, and the concrete pickup points for Phase 2.

## What landed

### Cargo workspace (`nestsClient/tests/hang-interop/`)

- Workspace with three binary crates: `hang-listen`, `hang-publish`,
  `udp-loss-shim`. **All three are Phase-1 stubs** — they parse their
  CLI flags via `clap`, print a banner, and exit 0. Phase 2 fills the
  bodies.
- `nestsClient/tests/hang-interop/REV` documents the pinned upstream
  `kixelated/moq` rev (`9e2461ee...`) plus the published crate
  versions on crates.io that track that rev (`moq-relay 0.10.25`,
  `moq-token-cli 0.5.23`, `hang 0.15.8`, `moq-lite 0.15.15`,
  `moq-native 0.13`).
- `cargo build --release` is verified green from the workspace root.

### Gradle integration (`nestsClient/build.gradle.kts`)

- `interopInstallMoqRelay` — `cargo install moq-relay --version
  $moqRelayVersion --root <cache>` into
  `~/.cache/amethyst-nests-interop/hang-interop-cargo/`. Skips when
  the binary already exists in the cache.
- `interopInstallMoqTokenCli` — same shape for `moq-token-cli`.
- `interopBuildSidecars` — `cargo build --release` over the local
  `nestsClient/tests/hang-interop/` workspace.
- `interopBuildHangSidecars` — umbrella task that depends on the
  three above. Runs as a test dependency only when
  `-DnestsHangInterop=true` is set.
- Test-task wiring forwards `nestsHangInteropSidecarsDir` and
  `nestsHangInteropCargoBinDir` system properties so the harness
  can find the binaries. `nestsHangInterop` itself is also
  forwarded — mirrors the existing `nestsInterop` opt-in.

### Kotlin harness (`nestsClient/src/jvmTest/...`)

- `interop/native/NativeMoqRelayHarness.kt` — boots a real
  `moq-relay` subprocess with `--tls-generate localhost` (relay
  self-signs its cert at startup) and `--auth-public ""` (every
  path is treated as public, no JWT required). Uses
  `ServerSocket(0)` to reserve an ephemeral port. Tracks process
  output in a 64-line ring buffer so a startup failure includes
  the relay's stderr tail. Singleton-per-JVM via `shared()`,
  shutdown via JVM hook, mirroring the existing
  `NostrNestsHarness` pattern. Public surface: `relayUrl`,
  `loopbackHostPort()`, `hangListenBin()`, `hangPublishBin()`,
  `udpLossShimBin()`, `moqTokenBin()`.
- `audio/SineWaveAudioCapture.kt` — frame-perfect deterministic
  sine wave at any frequency, mono, conforming to the existing
  `AudioCapture` interface in `commonMain`.
- `audio/PcmAssertions.kt` — pure-Kotlin signal-domain assertions:
  `assertSampleCount`, `assertRms`, `assertFftPeak` (Hann-windowed
  iterative Cooley-Tukey, ~100 lines, no transform-library dep),
  `assertZeroCrossingRate`, `findSilenceWindow`. The plan spec'd
  these for Phase 1; everything is in commonTest now and ready
  for the Phase 2 scenarios.
- `interop/native/NativeMoqRelayHarnessSmokeTest.kt` — the only
  test that runs in Phase 1. Boots the harness, verifies the
  relay binds its UDP port, verifies the sidecar binaries are
  executable and the `hang-listen` stub runs cleanly. **Does
  not** assert any wire format — that's Phase 2.

## Deviations from the spec

| Plan | Reality | Why |
|---|---|---|
| In-process Kotlin JWT minter (ES256, JWKS file mounted into relay) | Relay configured with `--auth-public ""`; no JWT required | The plan flagged JWT issuance as "implementation detail" — wire-format and protocol assertions don't need real auth, and `--auth-public` short-circuits the whole minter + JWKS file dance. JWT validation as an interop concern is already covered by the existing `NostrNestsAuthInteropTest` against the Docker'd `moq-auth`. The cargo-installed `moq-token` CLI is exposed via `moqTokenBin()` for Phase 2 scenarios that DO want to mint a real token (e.g. revocation, expiry, kid-mismatch). |
| Self-signed cert generated at suite setup via Bouncy Castle / `openssl req -x509` | `moq-relay --tls-generate localhost` (relay self-signs at startup) | `moq-native` ships this behaviour; saves us writing PEM I/O + cert generation. The Kotlin client can use the existing `PermissiveCertValidator` to skip chain validation. |
| `relay.toml` config file | CLI flags only | Same effect, fewer moving parts. Field names in the plan (`server.listen`, `tls.cert`/`key`, `auth.jwks_path`) were speculative; actual `moq-native` flags are `--server-bind`, `--tls-cert`/`--tls-key`/`--tls-generate`, `--auth-key`/`--auth-public`. |
| Build `moq-relay` from a pinned `kixelated/moq` checkout via `cargo build --release -p moq-relay` | `cargo install moq-relay --version 0.10.25 --root <cache>` | `moq-relay` and `moq-token-cli` are published on crates.io; `cargo install` makes the install reproducible without embedding/cloning the upstream workspace. Cache key is the pinned version. |
| Phase 1 step 7: "Wire one passing test: I1, A→hang" | Smoke test only — no wire-format scenario | I1 needs a JVM-side Opus encoder (the `OpusEncoder` interface in commonMain only has an Android `MediaCodec` actual today), AND it needs `hang-listen` to actually subscribe to a moq-lite session and decode Opus to PCM. Both are Phase 2 work. The smoke test we landed proves all the harness load-bearing pieces work, so Phase 2 only has to fill in the codecs + the sidecar bodies. |

## Phase 2 pickup

The core gap: **JVM Opus encoder + decoder, plus filling the
sidecar binaries**. Everything else is in place.

### Step A — JVM Opus encoder/decoder

Add a JVM `actual` for `OpusEncoder` / `OpusDecoder` (currently
Android-only via `MediaCodecOpusEncoder` / `MediaCodecOpusDecoder`).
Two viable options:

1. **`org.concentus:Concentus`** (pure-Java port of libopus). On
   Maven Central. License-compatible. Slower than native libopus
   but plenty fast enough for a 48 kHz mono test stream. **Recommended**
   since it adds zero native dependency.
2. **`audiopus_jni` (vendored or via JitPack)** — wraps the same
   `audiopus` Rust crate the upstream `hang` examples use. Faster
   but adds a JNI shared object per platform.

Wire it into `nestsClient/src/jvmMain/.../audio/JvmOpusEncoder.kt` +
`JvmOpusDecoder.kt`, with the Android `MediaCodec` actuals
unchanged. Add `CapturingOpusDecoder` (the plan's Phase 1 ask)
once the JVM decoder exists.

### Step B — Fill `hang-listen`

Model on `kixelated/moq/rs/hang/examples/subscribe.rs` (the
`subscribe` example in the upstream workspace at
`/tmp/moq/rs/hang/examples/subscribe.rs` if recloned). Replace
its video-track logic with the audio path: pick the first
rendition with `container.kind == "legacy"` and
`codec == "opus"`, subscribe, decode each `Frame` into a
`Bytes`-encoded VarInt timestamp + Opus packet, run the Opus
packets through `audiopus::Decoder`, write Float32 PCM to
`--output-pcm`. Dependencies to add to
`nestsClient/tests/hang-interop/hang-listen/Cargo.toml`:

```toml
hang = "0.15"
moq-lite = "0.15"
moq-mux = "0.3"
moq-native = { version = "0.13", default-features = false, features = ["quinn", "aws-lc-rs"] }
web-transport-quinn = "0.11"
audiopus = "0.3"
bytes = "1"
tracing = "0.1"
```

### Step C — Fill `hang-publish`

Mirror of `hang-listen` for the reverse direction. Generate a sine
wave in Rust, encode with `audiopus::Encoder`, publish a hang
catalog with one Opus rendition, push frames as
`varint(timestamp_us) + opus_packet` per the
`hang::container::Frame::encode` contract (see
`/tmp/moq/rs/hang/src/container/frame.rs`).

### Step D — Wire the I1 scenario

Once A/B/C land, the I1 "amethyst speaker → hang listener" test
is straightforward — see the spec for the pattern. The harness +
`SineWaveAudioCapture` + `PcmAssertions` are already in place to
support it.

## Phase 2.E — additional scenarios

Landed on top of I1:

- **I11 wire-byte capture** (`first_audio_frame_is_not_opus_codec_config`):
  hang-listen gained `--dump-first-frame <path>`. Test asserts the
  first audio frame's post-Container-Legacy-strip codec payload
  doesn't begin with `OpusHead` magic. Catches the T8 regression
  where Android's `MediaCodecOpusEncoder` would emit
  BUFFER_FLAG_CODEC_CONFIG bytes as a normal audio frame.
- **I2 late-join** (`late_join_listener_still_decodes_tail`):
  hang-listen attaches at T+2 s of a 5 s broadcast; asserts ≥1.5 s
  of decoded audio with the 440 Hz peak still recoverable.
- **I3 mute window** (`mid_broadcast_mute_shortens_decoded_pcm`):
  speaker mutes for 1 s mid-broadcast. Amethyst's broadcaster FINs
  the open uni stream rather than pushing zeros (so web watchers
  don't park on `await readFrame`), so the mute manifests as a
  sample-count deficit (~3 s for 4 s wallclock), not embedded
  silence. Asserts the deficit is in the right ballpark.

`runSpeakerToHangListen(...)` extracted as a per-scenario helper
in `HangInteropTest`. Each scenario anchors the QUIC transport's
coroutine scope to the per-test pumpScope so UDP sockets and
QuicConnection pumps cleanly tear down between tests.

The companion `KotlinSpeakerKotlinListenerThroughNativeRelayTest`
(Kotlin↔Kotlin diagnostic for the I1 bisect) lives behind
`-DnestsHangInteropDiagnostic=true` — it flakes when run in the
same JVM as the 5 native-subprocess scenarios (relay-side state
accumulation), and its only purpose is wire-format bisects.

## Phase 2.E deferred

- **I4 stereo** — needs a non-trivial production change in
  `MoqLiteHangCatalog.OPUS_MONO_48K_AUDIO_DATA_JSON_BYTES` (which
  hard-codes mono). Out of scope for these test plumbing changes;
  ship as a separate production-side patch.
- **I8 SubscribeDrop**, **I10 long broadcast**, **I12 Goaway** —
  next batch of P0 scenarios on the existing harness.

## Phase 3 + 4 + 5 deferred

Untouched in Phase 1:

- Phase 3 transport robustness (`udp-loss-shim` body, hot-swap,
  long-broadcast).
- Phase 4 browser harness (`nestsClient-browser-interop/` directory,
  Playwright driver).
- Phase 5 browser-only scenarios.

CI integration (the GitHub Actions workflow updates the spec
shows) is also pending — until Phase 2 lands a real test, there's
nothing in `-DnestsHangInterop=true` worth gating CI on except
the smoke test.

## Files

```
nestsClient/tests/hang-interop/
├── REV
├── Cargo.toml + Cargo.lock
├── hang-listen/{Cargo.toml,src/main.rs}        # Phase 2: real subscribe + decode
├── hang-publish/{Cargo.toml,src/main.rs}       # Phase 2: real publish + sine encode
└── udp-loss-shim/{Cargo.toml,src/main.rs}      # Phase 1 stub; Phase 3 fills body

nestsClient/build.gradle.kts                    # +interopBuildHangSidecars + system props
nestsClient/src/jvmTest/kotlin/com/vitorpamplona/nestsclient/
├── audio/
│   ├── JvmOpusEncoder.kt                       # libopus via JNA (test-only)
│   ├── JvmOpusDecoder.kt                       # libopus via JNA (test-only)
│   ├── JvmOpusRoundTripTest.kt
│   ├── PcmAssertions.kt
│   ├── PcmAssertionsTest.kt
│   └── SineWaveAudioCapture.kt
└── interop/native/
    ├── NativeMoqRelayHarness.kt                # boots moq-relay subprocess
    ├── NativeMoqRelayHarnessSmokeTest.kt
    ├── HangInteropTest.kt                      # I1 + I2 + I3 + I11 + Rust↔Rust
    └── KotlinSpeakerKotlinListenerThroughNativeRelayTest.kt
                                                # diagnostic, gated separately

nestsClient/plans/2026-05-06-cross-stack-interop-test-results.md  # this file
```