refactor(nests): per-stream channel count + AudioBroadcastConfig (I4 prep)

Split the previously global `AudioFormat.CHANNELS = 1` into a
`DEFAULT_CHANNELS` constant + per-call-site `channelCount` parameters
so a single broadcast can advertise stereo Opus without forcing every
mono call site to grow a new argument. Generalises the catalog factory
to `MoqLiteHangCatalog.opus48k(name, channels)` with memoised JSON
bytes per shape, threads a new `AudioBroadcastConfig(channelCount)`
through `connectNestsSpeaker` / `connectReconnectingNestsSpeaker` /
`MoqLiteNestsSpeaker`, and adds a `channelCount` parameter to
`MediaCodecOpusEncoder`. Production behaviour is unchanged for
mono callers (the new config defaults to mono); the listener side
already discovers the channel count from the catalog via
`NestViewModel.awaitAudioPipelineConfig`. No test or wire changes.

Phase 1 of `nestsClient/plans/2026-05-06-i4-stereo-cross-stack-scenario.md`.
The hang-interop test scaffolding (`HangInteropTest`, `runSpeakerToHangListen`,
Rust `hang-listen` / `hang-publish`, `JvmOpusEncoder`) doesn't exist on
this branch yet, so the I4 forward + reverse scenarios are deferred
until the parent T16 plan lands.

https://claude.ai/code/session_01EqJEADzH9yjSuoP5L9js8i
This commit is contained in:
Claude
2026-05-06 22:57:21 +00:00
parent 32e578dcc8
commit 23b8bfd34a
13 changed files with 254 additions and 59 deletions
@@ -0,0 +1,62 @@
/*
* Copyright (c) 2025 Vitor Pamplona
*
* Permission is hereby granted, free of charge, to any person obtaining a copy of
* this software and associated documentation files (the "Software"), to deal in
* the Software without restriction, including without limitation the rights to use,
* copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the
* Software, and to permit persons to whom the Software is furnished to do so,
* subject to the following conditions:
*
* The above copyright notice and this permission notice shall be included in all
* copies or substantial portions of the Software.
*
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS
* FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR
* COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN
* AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
* WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
*/
package com.vitorpamplona.nestsclient
/**
* Per-broadcast audio shape negotiated between the speaker, the catalog
* the relay forwards, and the listeners that subscribe to the audio
* track.
*
* Threaded through [connectNestsSpeaker] / [connectReconnectingNestsSpeaker]
* into [MoqLiteNestsSpeaker.startBroadcasting] so a stereo broadcaster
* can pick a stereo Opus rendition without forcing every existing mono
* call site to grow a parameter. The catalog factory in
* [com.vitorpamplona.nestsclient.moq.lite.MoqLiteHangCatalog.opus48kJsonBytes]
* keys on this shape so the JSON wire bytes stay byte-stable per shape.
*
* **Caller contract:** [channelCount] MUST match the channel count of
* the [com.vitorpamplona.nestsclient.audio.OpusEncoder] the caller
* provides via the speaker's `encoderFactory` AND the channel count of
* the PCM frames the caller's [com.vitorpamplona.nestsclient.audio.AudioCapture]
* produces. Mismatches surface as decoder errors on the listener side
* (CSD-0 channel byte vs interleaved PCM layout disagree) and produce
* either silence or downmix-with-clicks depending on the listener
* implementation. There is no in-broadcast renegotiation — pick the
* shape at speaker open time.
*
* Default is mono (the historical shape). Callers that don't pass a
* config keep the prior behaviour.
*
* @property channelCount 1 (mono) or 2 (stereo, L/R interleaved).
* Drives the catalog's `numberOfChannels` field. Multi-rendition
* catalogs (e.g. one mono + one stereo on the same broadcast) are
* out of scope here — model that as two separate catalog factories
* if it becomes useful.
*/
data class AudioBroadcastConfig(
val channelCount: Int = 1,
) {
init {
require(channelCount in 1..2) {
"AudioBroadcastConfig supports mono (1) or stereo (2) only, got $channelCount"
}
}
}
@@ -61,6 +61,17 @@ class MoqLiteNestsSpeaker internal constructor(
* Defaults to [NestMoqLiteBroadcaster.DEFAULT_FRAMES_PER_GROUP].
*/
private val framesPerGroup: Int = NestMoqLiteBroadcaster.DEFAULT_FRAMES_PER_GROUP,
/**
* Per-broadcast audio shape (channel count, future bitrate
* variants). Threaded into the catalog payload the speaker emits
* on the `catalog.json` track, so listeners see the shape the
* caller's encoder actually produces. Caller MUST construct the
* encoder + capture with a matching channel layout — see
* [AudioBroadcastConfig] for the contract. Defaults to mono so
* existing call sites that don't pass a config keep the prior
* behaviour.
*/
private val broadcastConfig: AudioBroadcastConfig = AudioBroadcastConfig(),
) : NestsSpeaker,
HotSwappablePublisherSource {
override val state: StateFlow<NestsSpeakerState> = mutableState.asStateFlow()
@@ -157,7 +168,11 @@ class MoqLiteNestsSpeaker internal constructor(
// practice the relay's SUBSCRIBE bidi takes a network
// round-trip after our ANNOUNCE Active, so this is safe
// even though the setter is non-suspending.
val catalogJson = MoqLiteHangCatalog.OPUS_MONO_48K_AUDIO_DATA_JSON_BYTES
val catalogJson =
MoqLiteHangCatalog.opus48kJsonBytes(
audioTrackName = MoqLiteNestsListener.AUDIO_TRACK,
numberOfChannels = broadcastConfig.channelCount,
)
catalogPublisher.setOnNewSubscriber {
runCatching {
catalogPublisher.send(catalogJson)
@@ -188,6 +188,13 @@ suspend fun connectNestsSpeaker(
framesPerGroup: Int =
com.vitorpamplona.nestsclient.audio
.NestMoqLiteBroadcaster.DEFAULT_FRAMES_PER_GROUP,
/**
* Per-broadcast audio shape. Defaults to mono so existing call sites
* keep the prior behaviour. Caller is responsible for matching the
* channel count to the encoder + capture they pass — see
* [AudioBroadcastConfig].
*/
broadcastConfig: AudioBroadcastConfig = AudioBroadcastConfig(),
): NestsSpeaker {
val state =
MutableStateFlow<NestsSpeakerState>(
@@ -252,6 +259,7 @@ suspend fun connectNestsSpeaker(
scope = scope,
mutableState = state,
framesPerGroup = framesPerGroup,
broadcastConfig = broadcastConfig,
)
}
@@ -103,6 +103,13 @@ suspend fun connectReconnectingNestsSpeaker(
speakerPubkeyHex: String,
captureFactory: () -> AudioCapture,
encoderFactory: () -> OpusEncoder,
/**
* Per-broadcast audio shape, threaded into [connectNestsSpeaker]
* (and therefore into the catalog payload) AND used by the
* hot-swap pump's per-session catalog publisher so the wire shape
* stays consistent across JWT-refresh recycles. Defaults to mono.
*/
broadcastConfig: AudioBroadcastConfig = AudioBroadcastConfig(),
policy: NestsReconnectPolicy = NestsReconnectPolicy(),
/**
* Proactive JWT refresh window. moq-auth issues bearer tokens
@@ -135,6 +142,7 @@ suspend fun connectReconnectingNestsSpeaker(
speakerPubkeyHex = speakerPubkeyHex,
captureFactory = captureFactory,
encoderFactory = encoderFactory,
broadcastConfig = broadcastConfig,
)
},
): NestsSpeaker {
@@ -292,6 +300,7 @@ suspend fun connectReconnectingNestsSpeaker(
scope = scope,
captureFactory = captureFactory,
encoderFactory = encoderFactory,
broadcastConfig = broadcastConfig,
)
}
@@ -320,6 +329,7 @@ private class ReconnectingSpeakerHandle(
*/
private val captureFactory: () -> AudioCapture,
private val encoderFactory: () -> OpusEncoder,
private val broadcastConfig: AudioBroadcastConfig,
) : NestsSpeaker {
override val state: StateFlow<NestsSpeakerState> = mutableState.asStateFlow()
@@ -350,6 +360,7 @@ private class ReconnectingSpeakerHandle(
scope = scope,
captureFactory = captureFactory,
encoderFactory = encoderFactory,
broadcastConfig = broadcastConfig,
onLevel = onLevel,
onClose = { closed ->
if (activeBroadcast === closed) activeBroadcast = null
@@ -412,6 +423,14 @@ private class ReissuingBroadcastHandle(
private val scope: CoroutineScope,
private val captureFactory: () -> AudioCapture,
private val encoderFactory: () -> OpusEncoder,
/**
* Per-broadcast audio shape, used to pick the catalog payload the
* per-session catalog publisher emits. The shape is constant for
* the wrapper's lifetime (we don't renegotiate mid-broadcast), so
* caching the payload bytes once on the [opus48kJsonBytes]
* memoiser is enough.
*/
private val broadcastConfig: AudioBroadcastConfig,
/**
* Forwarded to the underlying broadcaster (hot-swap path) or
* `sp.startBroadcasting` (legacy path) so the local-speaking ring
@@ -586,7 +605,11 @@ private class ReissuingBroadcastHandle(
// and any watcher that attaches AFTER the recycle sees nothing
// to subscribe to. Mirror of [MoqLiteNestsSpeaker.startBroadcasting]'s
// catalog setup; same JSON, same emit-on-subscribe pattern.
val catalogPayload = MoqLiteHangCatalog.OPUS_MONO_48K_AUDIO_DATA_JSON_BYTES
val catalogPayload =
MoqLiteHangCatalog.opus48kJsonBytes(
audioTrackName = MoqLiteNestsListener.AUDIO_TRACK,
numberOfChannels = broadcastConfig.channelCount,
)
val priorCatalogPublisher = hotSwapCatalogPublisher
val newCatalogPublisher =
try {
@@ -23,15 +23,27 @@ package com.vitorpamplona.nestsclient.audio
/**
* PCM audio format the audio pipeline produces and consumes.
*
* Listener-only flow runs at 48 kHz mono signed-16-bit, matching the nests
* Opus profile (RFC 6716 wideband at the codec's native rate). The whole
* pipeline is hardcoded to this format for now when nests starts varying
* codec settings, this becomes a per-room negotiated value out of the
* `/api/v1/nests/<room>` response.
* Sample rate + frame cadence are pipeline-wide invariants (48 kHz Opus,
* 20 ms frames) and live as flat constants. Channel count is **per-stream**:
* a microphone is mono, a stereo broadcast is two-channel, and a single
* listener may simultaneously decode tracks of different shapes. Call sites
* either inline `1` (the device is intrinsically mono e.g. the production
* mic) or take a `channelCount` parameter, plumbed in from a catalog (on
* the listener side) or from an [com.vitorpamplona.nestsclient.AudioBroadcastConfig]
* (on the speaker side). [DEFAULT_CHANNELS] is what call sites use as the
* factory default when they want the historical mono behaviour.
*/
object AudioFormat {
const val SAMPLE_RATE_HZ: Int = 48_000
const val CHANNELS: Int = 1
/**
* Default channel count for call sites that don't yet thread a
* per-stream override through. Mono matches what the pipeline
* shipped before stereo support landed. Use this only as a default
* value; do NOT assume every audio track is mono just because this
* exists.
*/
const val DEFAULT_CHANNELS: Int = 1
/** 20 ms at 48 kHz. */
const val FRAME_SIZE_SAMPLES: Int = 960
@@ -106,42 +106,27 @@ internal data class MoqLiteHangCatalog(
explicitNulls = false
}
/**
* Cached canonical-shape catalog JSON bytes for the default
* Opus mono 48 kHz audio track ([MoqLiteNestsListener.AUDIO_TRACK]
* keyed under `audio.renditions["audio/data"]`). The catalog is
* a fixed string for the whole publisher lifetime caching
* avoids re-running kotlinx.serialization on every
* [com.vitorpamplona.nestsclient.MoqLiteNestsSpeaker.startBroadcasting]
* call and every JWT-refresh hot-swap iteration in
* [com.vitorpamplona.nestsclient.connectReconnectingNestsSpeaker].
*
* Hard-coded to track name `"audio/data"` because that's the
* only track Amethyst publishes today; if a future caller
* needs a different name, fall back to
* [opusMono48k] + [encodeJsonBytes].
*/
val OPUS_MONO_48K_AUDIO_DATA_JSON_BYTES: ByteArray =
opusMono48k("audio/data").encodeJsonBytes()
/**
* Canonical Amethyst speaker catalog: a single `legacy`-container
* Opus rendition under [audioTrackName], matching the encoder
* config in [com.vitorpamplona.nestsclient.audio.OpusEncoder]
* (48 kHz mono).
* Opus rendition under [audioTrackName] at 48 kHz with
* [numberOfChannels] (1 = mono, 2 = stereo L/R interleaved),
* matching the encoder config in
* [com.vitorpamplona.nestsclient.audio.OpusEncoder].
*
* The rendition map is keyed by the moq-lite track name a
* subscriber should subscribe to for this rendition's frames
* for nests audio rooms that's the same string the publisher
* publishes audio frames on
* (`MoqLiteNestsListener.AUDIO_TRACK`).
*
* If [com.vitorpamplona.nestsclient.audio.OpusEncoder] becomes
* parameterised in the future, this factory should take the
* encoder's config rather than hard-coding 48 kHz mono.
*/
fun opusMono48k(audioTrackName: String): MoqLiteHangCatalog =
MoqLiteHangCatalog(
fun opus48k(
audioTrackName: String,
numberOfChannels: Int = 1,
): MoqLiteHangCatalog {
require(numberOfChannels in 1..2) {
"opus48k supports mono (1) or stereo (2) only, got $numberOfChannels"
}
return MoqLiteHangCatalog(
audio =
Audio(
renditions =
@@ -151,12 +136,49 @@ internal data class MoqLiteHangCatalog(
codec = "opus",
container = Container(kind = "legacy"),
sampleRate = 48_000,
numberOfChannels = 1,
numberOfChannels = numberOfChannels,
jitter = OPUS_FRAME_DURATION_MS,
),
),
),
)
}
/**
* Memoised JSON bytes for [opus48k]. The catalog is a fixed
* string for the whole publisher lifetime caching avoids
* re-running kotlinx.serialization on every
* [com.vitorpamplona.nestsclient.MoqLiteNestsSpeaker.startBroadcasting]
* call and every JWT-refresh hot-swap iteration in
* [com.vitorpamplona.nestsclient.connectReconnectingNestsSpeaker].
*
* Keyed by `(trackName, numberOfChannels)` so adding a new shape
* (stereo, future bitrate variants) doesn't multiply the
* constant count. The cache is populated on first request per
* shape and never evicted at most a handful of entries per
* process lifetime, all small.
*
* Thread-safety: the operation is idempotent two threads
* computing the same shape land identical bytes so a
* non-locking [HashMap] is acceptable in commonMain. Worst case
* the second writer overwrites with an equal value; cache
* readers always see at least one fully-published value
* because the JVM's default visibility for non-volatile object
* references is good enough for the "may compute twice"
* tolerance we have here.
*/
private val cachedJsonBytes: HashMap<Pair<String, Int>, ByteArray> = HashMap()
fun opus48kJsonBytes(
audioTrackName: String,
numberOfChannels: Int = 1,
): ByteArray {
val key = audioTrackName to numberOfChannels
cachedJsonBytes[key]?.let { return it }
val bytes = opus48k(audioTrackName, numberOfChannels).encodeJsonBytes()
cachedJsonBytes[key] = bytes
return bytes
}
/**
* Opus frame duration in milliseconds 960 samples / 48 kHz =