amethyst/quic/plans/2026-05-04-control-frame-retransmit.md

# Control-frame retransmit for `:quic` — implementation plan

**Status:** plan, not started.

## Why

Production tracing against `moq.nostrnests.com` (commits `36c707f` →
`c3d6cad` on `claude/fix-nest-audio-display-3chAG`) showed the audio
cliff lands at exactly the moment we emit our first `MAX_STREAMS_UNI`
extension at the half-window threshold. One emit, one packet, one
loss on the wire — and because `:quic` does not retransmit ack-eliciting
frames, the relay never sees the bump, silently stops sending, and our
QUIC state stays in split-brain (`udpRecvDatagrams` frozen) until the
listener manually disconnects.

The current shipping fix (`QuicConnectionConfig.initialMaxStreamsUni
= 1_000_000`) sidesteps the bug by advertising a cap so high the
half-window threshold doesn't trip until ~13.9 hours of audio. That
gets us multi-hour Nests but leaves the underlying frailty in place:
*every* ack-eliciting control frame we send today is one wire-loss
away from causing a silent stall. `MAX_DATA`, `MAX_STREAM_DATA`,
`RESET_STREAM`, `STOP_SENDING`, `NEW_CONNECTION_ID` — all have the
same exposure.

The fix that browser-grade QUIC stacks have and we don't:
RFC 9002 §6 loss detection + per-frame retransmit.

## Reference

We follow Firefox's [neqo](https://github.com/mozilla/neqo) design,
not Chrome's quiche, because:

- neqo's typed-token shape (a `RecoveryToken` discriminated union with
  one variant per retransmittable frame) maps onto Kotlin sealed
  classes naturally and matches the existing `Frame` /
  `MoqLiteControl…` patterns in the codebase.
- quiche's monotonic-control-frame-id deque is more compact in C++
  but loses compile-time safety in Kotlin; with its cost-of-bug
  history (a single `kInvalidControlFrameId` sentinel typo would
  silently break retransmit) it's not worth the conciseness.
- Both are correct per RFC 9002, both are interop-tested daily; the
  algorithm is identical, only the bookkeeping shape differs.

Specific neqo files we mirror:

| neqo (Rust) | `:quic` (Kotlin) |
|---|---|
| `neqo-transport/src/recovery/token.rs` | new `quic/connection/recovery/RecoveryToken.kt` |
| `neqo-transport/src/recovery/sent.rs` | new `quic/connection/recovery/SentPacket.kt` |
| `neqo-transport/src/recovery/mod.rs` | extend `QuicConnection` + new `QuicLossDetection.kt` |
| `neqo-transport/src/streams.rs` (lost dispatch) | new `QuicConnection.onTokensLost()` |
| `neqo-transport/src/fc.rs` (`frame_lost` flag) | extend `QuicConnection.advertisedMaxStreamsUni` etc. with a `pending` companion |

## Scope

**In scope (this plan):** retransmit for the receive-side flow-control
frames that are RFC-9002 ack-eliciting and cheap to make idempotent:

- `MAX_STREAMS_UNI`
- `MAX_STREAMS_BIDI`
- `MAX_DATA`
- `MAX_STREAM_DATA`

These are the frames that, when lost, can silently wedge the
connection. They're also the frames where the retransmit semantics
are simplest: re-emit if the sent value still matches our current
view; otherwise the newer-value emission supersedes.

**Out of scope (separate follow-ups):**

- `STREAM` data retransmit. Audio rooms tolerate gaps (Opus is
  best-effort streaming), and adding STREAM retransmit means
  rewriting `SendBuffer` to retain bytes until ACK rather than
  release on send. Tracked separately.
- `RESET_STREAM`, `STOP_SENDING`, `NEW_CONNECTION_ID`,
  `RETIRE_CONNECTION_ID`, `STREAMS_BLOCKED`, `DATA_BLOCKED`: not
  exercised by the moq-lite workload today. Add when needed.
- `CRYPTO` retransmit (handshake bytes). Important for handshake
  reliability but out of scope for the audio-cliff fix; we'd inherit
  this from a full RFC 9002 pass later.
- Congestion control (NewReno / CUBIC / BBR). Independent concern;
  `:quic` has no CC at all today, and adding it is its own multi-day
  project.
- 0-RTT, connection migration, multipath. We don't use any of these.

## Architecture

### 1. `RecoveryToken`

A sealed class enumerating each retransmittable frame type. Mirrors
neqo's `StreamRecoveryToken` enum at `recovery/token.rs:21`.

```kotlin
sealed class RecoveryToken {
    object Ack : RecoveryToken()  // tracked but not retransmitted
    data class MaxStreamsUni(val maxStreams: Long) : RecoveryToken()
    data class MaxStreamsBidi(val maxStreams: Long) : RecoveryToken()
    data class MaxData(val maxData: Long) : RecoveryToken()
    data class MaxStreamData(val streamId: Long, val maxData: Long) : RecoveryToken()
}
```

### 2. `SentPacket`

Per-packet metadata retained until the packet is ACK'd or declared
lost. Mirrors neqo's `recovery/sent.rs::Packet`.

```kotlin
data class SentPacket(
    val packetNumber: Long,
    val sentAtMillis: Long,
    val ackEliciting: Boolean,
    val sizeBytes: Int,
    val tokens: List<RecoveryToken>,
)
```

Held in a per-pn-space `MutableMap<Long, SentPacket>` on
`QuicConnection`. Three spaces: Initial, Handshake, Application.

### 3. Send path: emit + register

In `QuicConnectionWriter.appendFlowControlUpdates`, when we emit a
`MaxStreamsFrame` (or any retransmit-eligible control frame), we
also build a `RecoveryToken` and append it to the per-packet token
list. `buildApplicationPacket` returns `(packetBytes, tokens, pn,
ackEliciting)`. `drainOutbound` records a `SentPacket` in the
application space's map keyed by packet number.

### 4. ACK path: drop tokens

Existing `QuicConnectionParser` ACK handling at line 165
(`AckFrame -> state.ackTracker.purgeBelow(...)`) extends to also
remove the corresponding `SentPacket` entries from the map. Tokens
go away silently — they were delivered.

### 5. Loss detection

A new `QuicLossDetection` per pn-space, called from the writer or
on a timer. RFC 9002 §6.1 thresholds:

- Packet threshold: any sent packet with `pn < largest_acked - 3`
  is lost.
- Time threshold: any sent packet with `sent_time + (max_rtt *
  9/8) < now` is lost (where `max_rtt = max(smoothed_rtt,
  latest_rtt)`).

When a packet is declared lost: pull its `tokens`, dispatch each to
its `onLost` handler, and remove it from the sent map.

### 6. Loss dispatch: `onLost(token)`

Mirrors neqo's `streams.rs::lost()` and `fc.rs::frame_lost()`.

```kotlin
fun onLost(token: RecoveryToken) {
    when (token) {
        is RecoveryToken.MaxStreamsUni -> {
            // Only flag pending if we haven't since extended further.
            // The newer value would supersede this one anyway.
            if (token.maxStreams == advertisedMaxStreamsUni) {
                pendingMaxStreamsUni = token.maxStreams
            }
        }
        is RecoveryToken.MaxStreamsBidi -> { /* symmetric */ }
        is RecoveryToken.MaxData -> { /* symmetric */ }
        is RecoveryToken.MaxStreamData -> { /* per-stream symmetric */ }
        RecoveryToken.Ack -> { /* not retransmittable */ }
    }
}
```

The `if (token.maxStreams == advertisedMaxStreamsUni)` guard exactly
mirrors `fc.rs::frame_lost`'s `if (maximum_data == self.max_allowed)`
check at line 322. Without it, we'd resurrect stale superseded
extensions.

### 7. Re-emit: drain `pending*` on next write

`appendFlowControlUpdates` checks `pendingMaxStreamsUni` first;
if non-null, emit that value (with a fresh token, registered on the
new packet) and clear pending. Only when there's no pending
retransmit does it run the normal half-window threshold check.

### 8. PTO (Probe Timeout)

RFC 9002 §6.2. When no ack-eliciting packets have been ACK'd within
`PTO = smoothed_rtt + max(4 * rttvar, kGranularity) + max_ack_delay`,
the connection has lost contact. Send a PING packet to elicit an
ACK. PTO doubles on consecutive expirations; resets on ACK.

Drives loss detection forward when the peer has gone quiet — exactly
the failure mode where today's connection goes dead.

## Test inventory mirrored from neqo

These are the tests that already exist in neqo and that we must port
to land equivalent coverage. Each row is one test we owe.

### Token-level: receiver flow control (mirror `fc.rs`)

Total: **9 tests** to mirror.

| neqo test (file:line — fc.rs) | Asserts | Our equivalent |
|---|---|---|
| `lost_blocked_resent` | After STREAMS_BLOCKED loss, frame_pending re-set | `ReceiverFlowControlTest.lost_blocked_resent` |
| `lost_after_increase` | If newer extension was already sent, lost old one is *not* re-sent | `ReceiverFlowControlTest.lost_after_increase` |
| `lost_after_higher_blocked` | Same, applied to STREAMS_BLOCKED | `ReceiverFlowControlTest.lost_after_higher_blocked` |
| `need_max_allowed_frame_after_loss` | `frame_lost(N)` where N == current limit re-flags pending | `ReceiverFlowControlTest.maxStreamsLostMatchesCurrent_resent` |
| `no_max_allowed_frame_after_old_loss` | `frame_lost(stale)` after newer sent does NOT re-flag | `ReceiverFlowControlTest.maxStreamsLostStaleAfterNewer_dropped` |
| `multiple_retries_after_frame_pending_is_set` | Repeated `retire(...)` keeps `frame_needed` true; `frame_sent(N)` clears | `ReceiverFlowControlTest.multipleRetiresMaintainFramePending` |
| `new_retired_before_loss` | After loss following further `retire()`, the new (higher) limit is sent | `ReceiverFlowControlTest.lossAfterRetireUsesNewerLimit` |
| `force_send_max_allowed` | A small first-retire below threshold does not flag | `ReceiverFlowControlTest.smallFirstRetireDoesNotFlag` |
| `set_max_active_equal_does_not_set_frame_pending` | Setting same max-active as before does nothing | `ReceiverFlowControlTest.setMaxActiveEqualDoesNothing` |

### Recovery-level: loss-detection algorithm (mirror `recovery/mod.rs`)

Total: ~20 tests; we mirror the subset that doesn't require CRYPTO/handshake-space testing (we're scoped to Application space).

| neqo test (recovery/mod.rs) | Asserts | Our equivalent |
|---|---|---|
| `remove_acked` | ACK removes the right packet numbers from sent map | `QuicLossDetectionTest.ackRemovesPackets` |
| `time_loss_detection_gap` | Packet older than `max_rtt * 9/8` declared lost | `QuicLossDetectionTest.timeThresholdMarksLost` |
| `time_loss_detection_timeout` | Loss detection schedules wake-up at the right deadline | `QuicLossDetectionTest.timeThresholdSchedulesTimer` |
| `big_gap_loss` | Packet threshold (≥ 3 newer ACK'd) declares lost | `QuicLossDetectionTest.packetThresholdMarksLost` |
| `loss_timer_set_on_pto` | Timer scheduled when PTO arms | `QuicLossDetectionTest.lossTimerSetOnPto` |
| `loss_timer_expired_on_timeout` | Expired timer triggers loss callbacks | `QuicLossDetectionTest.expiredTimerTriggersLoss` |
| `loss_timer_cancelled_on_ack` | New ACK for in-flight packet cancels pending PTO | `QuicLossDetectionTest.ackCancelsPto` |
| `pto_works_basic` | PTO emits a probe packet | `PtoTest.basic` |
| `pto_works_full_cwnd` | PTO works even at congestion-window full | `PtoTest.fullCwnd` |
| `pto_works_ping` | Probe is a PING when nothing else to send | `PtoTest.probesWithPing` |
| `ack_after_pto` | ACK clears PTO state correctly | `PtoTest.ackResetsPtoCount` |
| `pto_retransmits_previous_frames_across_datagrams` | PTO probe carries the lost-but-pending frames, not just PING | `PtoTest.probeIncludesPendingFrames` |
| `pto_state_count` | PTO count doubles each fire, resets on ACK | `PtoTest.exponentialBackoff` |
| `loss_recovery_crash` | Two simultaneous loss events don't crash | `QuicLossDetectionTest.concurrentLossesNoCrash` |
| `lost_but_kept_and_lr_timer` | Lost packet retained in book-keeping until release timer | `QuicLossDetectionTest.lostPacketRetainedForReleaseWindow` |
| `loss_time_past_largest_acked` | Oldest pending loss-time is the one that matters | `QuicLossDetectionTest.oldestLossTimeIsScheduled` |
| `ack_for_unsent` | ACK referencing unsent PN closes the connection per RFC | `QuicLossDetectionTest.ackForUnsentClosesConnection` |
| `duplicate_ack_does_not_update_largest_acked_sent_time` | Re-ACK is a no-op | `QuicLossDetectionTest.duplicateAckIsNoop` |
| `should_probe_exact_boundary` | PTO fires exactly at deadline, not before | `PtoTest.firesAtBoundaryNotBefore` |
| `ack_only_boundary` | An ACK-only packet doesn't arm PTO | `PtoTest.ackOnlyDoesNotArmPto` |

### Connection-level integration (mirror `connection/tests/recovery.rs` + `connection/tests/stream.rs`)

Total: ~10 tests covering the full happens-after of "frame sent →
packet lost → frame re-emitted → ACK'd".

| neqo test (file) | Asserts | Our equivalent |
|---|---|---|
| `pto_works_basic` (recovery.rs) | End-to-end: lose a packet with control frames, observe retransmit | `QuicConnectionRetransmitTest.maxStreamsUniLossEmitsRetransmit` |
| `pto_works_ping` (recovery.rs) | PTO PING fires after silence, peer ACKs it | `QuicConnectionRetransmitTest.silentPathPtoEmitsPing` |
| `pto_retransmits_previous_frames_across_datagrams` | PTO carries the pending control-frame retransmits | `QuicConnectionRetransmitTest.ptoCarriesPendingMaxStreamsBump` |
| `lost_but_kept_and_lr_timer` (recovery.rs) | Timer eviction of lost-tracking entries doesn't break re-emit | `QuicConnectionRetransmitTest.lostTrackingEvictionLeavesPendingIntact` |
| `sending_max_data` (stream.rs) | MAX_DATA emitted, ACK observed, no spurious re-emit | `QuicConnectionRetransmitTest.maxDataAckedNoSpuriousRetransmit` |
| `max_data` (stream.rs) | MAX_DATA flow control end-to-end | `QuicConnectionRetransmitTest.maxDataIncreasePropagates` |
| `stream_data_blocked_generates_max_stream_data` | Receiver emits MAX_STREAM_DATA on blocked signal, retransmits if lost | `QuicConnectionRetransmitTest.maxStreamDataLossEmitsRetransmit` |

### Codec-level: token serialisation round-trip

Add: token equality, token data-class hashCode/equals correctness, token-list encode/decode-by-debug for log diagnosis. ~3 tests.

### Total

~50 tests. neqo has ~125 tests in this area; we're scoped to ~40% of
neqo's coverage because we're not reimplementing CRYPTO retransmit,
0-RTT, handshake-space PTO, etc.

## File-by-file implementation order

Each step is its own commit; each commit must compile + pass tests
before moving on.

### Step 1: types only (no behavior)

- New: `quic/src/commonMain/kotlin/com/vitorpamplona/quic/connection/recovery/RecoveryToken.kt`
- New: `quic/src/commonMain/kotlin/com/vitorpamplona/quic/connection/recovery/SentPacket.kt`
- Test: `quic/src/commonTest/.../recovery/RecoveryTokenTest.kt`
- No other code touched. Compiles, types ready for next step.

### Step 2: track sent packets in `QuicConnection`

- Add `sentApplicationPackets: MutableMap<Long, SentPacket>` to `QuicConnection.application`.
- `QuicConnectionWriter.buildApplicationPacket` builds a `tokens: List<RecoveryToken>` alongside its `frames`, returns both.
- On packet emission, store a `SentPacket` keyed by PN.
- Existing tests still pass; new behavior dormant (no loss detection yet).

### Step 3: drain on ACK

- `QuicConnectionParser.AckFrame` handler walks ACK ranges, removes `SentPacket` entries.
- Test: ACK removes a sent packet's tokens. ACK out of order is fine.

### Step 4: receiver-flow-control `pending*` companion fields

- Add `pendingMaxStreamsUni: Long?`, `pendingMaxStreamsBidi: Long?`, `pendingMaxData: Long?` to `QuicConnection`.
- Add per-stream `pendingMaxStreamData` map (small).
- `appendFlowControlUpdates` drains pending* first, only then runs the normal threshold check.
- Tests: mirror the 9 `fc.rs` tests above. Pure unit-level.

### Step 5: loss detection algorithm

- New: `quic/src/commonMain/kotlin/com/vitorpamplona/quic/connection/recovery/QuicLossDetection.kt`.
- Implements packet-threshold + time-threshold per RFC 9002 §6.1.
- Hooks into ACK handler: each new ACK runs `detectLost(now)`.
- Tests: mirror the 20 `recovery/mod.rs` tests.

### Step 6: dispatch lost tokens

- `QuicConnection.onTokensLost(tokens)` dispatches each token to its `pending*` field.
- Wire the dispatch into the loss-detection callback.
- Tests: integration — send MAX_STREAMS, drop the packet, verify retransmit happens on next outbound. (~3 tests.)

### Step 7: PTO

- New: `quic/src/commonMain/kotlin/com/vitorpamplona/quic/connection/recovery/Pto.kt`.
- Implements §6.2 PTO timer. Schedules wake-up; on expiry, emits a PING (if nothing else queued) or re-uses the pending retransmit machinery.
- `QuicConnectionDriver.sendLoop` learns to wake on PTO.
- Tests: mirror the ~7 PTO tests from neqo.

### Step 8: integration test against `FakeWebTransport`

- New: `nestsClient/src/commonTest/.../moq/lite/MoqLiteSessionRetransmitTest.kt` — drops a specific packet between client and server, asserts MAX_STREAMS retransmit lands and audio keeps flowing.
- This is what we couldn't write before; now we can.

### Step 9: revert the cap workaround (optional)

Once the retransmit path is durable, optionally revert
`initialMaxStreamsUni` from 1 000 000 back to a smaller value
(e.g. 1 000) so the bump path actually exercises in production.
Keeps the rolling-extension code path warm and validates retransmit
is healthy. Probably gate this on having shipped the retransmit
work for at least a week without regressions.

## Risks and mitigations

| Risk | Mitigation |
|---|---|
| Existing `:quic` tests written against "send-and-forget" semantics break | Step 2 is purely additive; existing tests still see send-and-forget behavior because nothing reads the `sentApplicationPackets` map until step 5. |
| Token tracking grows unbounded if ACK never arrives | RFC 9002 §6.5: declare lost on time threshold; lost packets evicted on `kPacketDeclaredLostThreshold` after eviction window. We bound the map by both ACK and loss declarations. |
| RTT estimation is wrong → false-positive losses | Same RTT estimation logic as neqo, conservative defaults (initial RTT 333 ms per RFC 9002 §6.2.2). |
| PTO timer interacts with the existing send loop | Step 7 is the trickiest; bake in conservative tests that simulate slow-network conditions. |
| Memory: at 10 streams/sec a 2-hour session has ~72k tracked sent packets if no ACKs arrive | Bounded by ACK arrival; if ACKs are missing for long enough we'd PTO-out and close. Realistic ACK cadence is sub-second so the map stays small (a few hundred entries). |

## Effort

4–7 days for a full pass with tests. Steps 1–4 are 1–2 days; steps 5–7 are the bulk; step 8 is a day; step 9 is trivial. The test count is the dominant cost — each ported test is ~30 lines of Kotlin.

## Acceptance criteria

- All ~50 ported tests pass.
- `NostrnestsProdAudioTransmissionTest` (existing JVM interop test) continues to pass against production.
- A new `MoqLiteSessionRetransmitTest` simulates packet loss at the moment of `MAX_STREAMS_UNI` emission and confirms audio continues flowing past the threshold.
- Existing `QuicConnectionWriterTest`, `PeerStreamCreditExtensionTest`, etc. unchanged.
- No regression in handshake latency or throughput under steady-state.