feat(nests): drive Opus decoder + AudioTrack channel count from catalog
Web speakers (kixelated/hang reference) emit stereo Opus when the
user's AudioContext picks a stereo input — and our catalog now lets
us know that ahead of decoder construction. Previously the decoder
+ AudioTrack were both hardcoded to mono via AudioFormat.CHANNELS, so
a stereo web publisher's frames either threw on the Opus TOC byte
mismatch, decoded with downmix artifacts, or output left-channel-only
depending on the device's MediaCodec implementation.
Wire the catalog-discovered numberOfChannels through:
- MediaCodecOpusDecoder accepts `channelCount: Int = 1`. Drives
MediaFormat.createAudioFormat's channel count, the OpusHead CSD-0
channel byte, and the per-frame output buffer size (stereo frame =
2× shorts vs mono). Validates 1..2 — RFC 7845 mapping family 0
covers both mono and stereo without an explicit channel mapping
table; multichannel needs family 1 which we don't support.
- AudioTrackPlayer accepts `channelCount: Int = 1`. Drives the
AudioTrack channel mask (CHANNEL_OUT_MONO vs CHANNEL_OUT_STEREO)
and the 250 ms wall-clock buffer-size target (stereo doubles bytes
per sample).
- NestViewModel.openSubscription now starts the catalog fetch
BEFORE waiting for it, then awaits `_speakerCatalogs[pubkey]` for
CATALOG_AWAIT_TIMEOUT_MS (500 ms) before constructing the decoder
+ player. With the speaker-side emit-on-subscribe hook in place
(3417e44), the catalog frame is on the wire within one round-trip
of the SUBSCRIBE_OK, so the wait typically resolves in tens of ms.
Falls back to mono if the catalog never arrives — covers legacy
publishers that don't emit a catalog and the failure-mode where
the relay never forwards the catalog group.
- decoderFactory + playerFactory signatures change from
`() -> X` to `(channelCount: Int) -> X`. Two call sites updated:
NestViewModelFactory (production) and NestViewModelTest's
newViewModel helper.
Speaker side stays mono-only — our encoder is fixed at
AudioFormat.CHANNELS=1 and the catalog we publish declares
numberOfChannels=1. This change is exclusively about handling
incoming audio from publishers that don't share that constraint.
https://claude.ai/code/session_014JfZJHSTvyYYWJbC9VbB47
This commit is contained in:
Claude
+21
-4
@@ -77,7 +77,22 @@ import android.media.AudioFormat as AndroidAudioFormat
|
||||
class AudioTrackPlayer(
|
||||
private val usage: Int = AudioAttributes.USAGE_MEDIA,
|
||||
private val contentType: Int = AudioAttributes.CONTENT_TYPE_SPEECH,
|
||||
/**
|
||||
* Number of channels in the PCM stream this player will receive. Must
|
||||
* match the [com.vitorpamplona.nestsclient.audio.OpusDecoder]'s output
|
||||
* configuration (mono Opus → 1, stereo Opus → 2 with L/R interleaving).
|
||||
* Drives both the AudioTrack channel mask and the underlying buffer-
|
||||
* size target. Default is [AudioFormat.CHANNELS] (mono) so existing
|
||||
* call sites that don't pass a channel count keep the prior behaviour.
|
||||
*/
|
||||
private val channelCount: Int = AudioFormat.CHANNELS,
|
||||
) : AudioPlayer {
|
||||
init {
|
||||
require(channelCount in 1..2) {
|
||||
"AudioTrackPlayer supports mono (1) or stereo (2) only, got $channelCount"
|
||||
}
|
||||
}
|
||||
|
||||
private var track: AudioTrack? = null
|
||||
private var muted: Boolean = false
|
||||
private var volume: Float = 1f
|
||||
@@ -100,10 +115,10 @@ class AudioTrackPlayer(
|
||||
.d("NestPlay") { "AudioTrackPlayer.start() — allocating AudioTrack" }
|
||||
|
||||
val channelMask =
|
||||
when (AudioFormat.CHANNELS) {
|
||||
when (channelCount) {
|
||||
1 -> AndroidAudioFormat.CHANNEL_OUT_MONO
|
||||
2 -> AndroidAudioFormat.CHANNEL_OUT_STEREO
|
||||
else -> error("unsupported channel count ${AudioFormat.CHANNELS}")
|
||||
else -> error("unsupported channel count $channelCount")
|
||||
}
|
||||
|
||||
val minBuffer =
|
||||
@@ -122,9 +137,11 @@ class AudioTrackPlayer(
|
||||
// miss its 20 ms cadence by an order of magnitude before the device
|
||||
// underruns. Take the larger of `minBuffer * 16` and an explicit
|
||||
// 250 ms-equivalent so devices that report a small minBuffer still
|
||||
// get the same wall-clock slack.
|
||||
// get the same wall-clock slack. Stereo doubles the byte count
|
||||
// per sample (interleaved L,R 16-bit shorts) — scaling
|
||||
// [channelCount] in keeps the wall-clock target constant.
|
||||
val targetBytes250Ms =
|
||||
(AudioFormat.SAMPLE_RATE_HZ / 4) * AudioFormat.BYTES_PER_SAMPLE * AudioFormat.CHANNELS
|
||||
(AudioFormat.SAMPLE_RATE_HZ / 4) * AudioFormat.BYTES_PER_SAMPLE * channelCount
|
||||
val bufferBytes = maxOf(minBuffer * 16, targetBytes250Ms)
|
||||
|
||||
val newTrack =
|
||||
|
||||
+39
-12
@@ -31,21 +31,42 @@ import java.nio.ByteOrder
|
||||
* across packets, so sharing a decoder across speakers would cause clicks.
|
||||
*
|
||||
* Configuration:
|
||||
* - 48 kHz mono signed 16-bit PCM output (matches [AudioFormat]).
|
||||
* - CSD-0: Opus identification header per RFC 7845 §5.1, 19 bytes.
|
||||
* - 48 kHz signed 16-bit PCM output (Opus's internal sample rate;
|
||||
* MediaCodec's Opus decoder always emits 48 kHz regardless of the
|
||||
* OpusHead `inputSampleRate` field).
|
||||
* - [channelCount] — 1 (mono) or 2 (stereo) interleaved L/R. Drives both
|
||||
* the MediaFormat channel-count and the OpusHead CSD-0 channel byte.
|
||||
* A web publisher (kixelated/hang) emits stereo when the user's
|
||||
* AudioContext picks a stereo input; without matching channel
|
||||
* configuration the decoder either downmixes with artifacts or
|
||||
* refuses the packet.
|
||||
* - CSD-0: Opus identification header per RFC 7845 §5.1, 19 bytes
|
||||
* (mapping family 0; covers both mono and stereo with implicit
|
||||
* L,R interleaving).
|
||||
* - CSD-1 / CSD-2: pre-skip + seek pre-roll, both zero (we don't seek).
|
||||
*
|
||||
* Default is [AudioFormat.CHANNELS] (mono) so existing call sites that
|
||||
* don't pass a channel count continue to behave exactly as before.
|
||||
*/
|
||||
class MediaCodecOpusDecoder : OpusDecoder {
|
||||
class MediaCodecOpusDecoder(
|
||||
private val channelCount: Int = AudioFormat.CHANNELS,
|
||||
) : OpusDecoder {
|
||||
init {
|
||||
require(channelCount in 1..2) {
|
||||
"MediaCodecOpusDecoder supports mono (1) or stereo (2) only, got $channelCount"
|
||||
}
|
||||
}
|
||||
|
||||
private val codec: MediaCodec =
|
||||
try {
|
||||
MediaCodec
|
||||
.createDecoderByType(MediaFormat.MIMETYPE_AUDIO_OPUS)
|
||||
.apply {
|
||||
configure(buildFormat(), null, null, 0)
|
||||
configure(buildFormat(channelCount), null, null, 0)
|
||||
start()
|
||||
}.also {
|
||||
com.vitorpamplona.quartz.utils.Log.d("NestPlay") {
|
||||
"MediaCodecOpusDecoder allocated codec='${it.name}'"
|
||||
"MediaCodecOpusDecoder allocated codec='${it.name}' channelCount=$channelCount"
|
||||
}
|
||||
}
|
||||
} catch (t: Throwable) {
|
||||
@@ -70,7 +91,11 @@ class MediaCodecOpusDecoder : OpusDecoder {
|
||||
// via ShortBuffer.get(dst, off, len) — the previous shape went
|
||||
// through ArrayList<Short>, which boxed every PCM sample
|
||||
// (~48 000 alloc/sec/speaker on the audio hot path).
|
||||
val out = ShortArray(AudioFormat.FRAME_SIZE_SAMPLES)
|
||||
//
|
||||
// Stereo Opus emits L/R-interleaved samples, so a 20 ms / 960-
|
||||
// sample frame produces `FRAME_SIZE_SAMPLES * channelCount`
|
||||
// shorts — twice as many for stereo as for mono.
|
||||
val out = ShortArray(AudioFormat.FRAME_SIZE_SAMPLES * channelCount)
|
||||
var outPos = 0
|
||||
|
||||
// 1. Acquire an input slot. On rare back-pressure (output buffers
|
||||
@@ -196,14 +221,14 @@ class MediaCodecOpusDecoder : OpusDecoder {
|
||||
|
||||
private const val FRAME_DURATION_US = 20_000L // 20 ms
|
||||
|
||||
private fun buildFormat(): MediaFormat {
|
||||
private fun buildFormat(channelCount: Int): MediaFormat {
|
||||
val format =
|
||||
MediaFormat.createAudioFormat(
|
||||
MediaFormat.MIMETYPE_AUDIO_OPUS,
|
||||
AudioFormat.SAMPLE_RATE_HZ,
|
||||
AudioFormat.CHANNELS,
|
||||
channelCount,
|
||||
)
|
||||
format.setByteBuffer("csd-0", ByteBuffer.wrap(buildOpusIdHeader()))
|
||||
format.setByteBuffer("csd-0", ByteBuffer.wrap(buildOpusIdHeader(channelCount)))
|
||||
// Pre-skip + seek pre-roll: both zero, encoded as little-endian
|
||||
// 64-bit nanoseconds per Android's MediaCodec contract.
|
||||
format.setByteBuffer("csd-1", ByteBuffer.wrap(zeroLongLe()))
|
||||
@@ -211,12 +236,14 @@ class MediaCodecOpusDecoder : OpusDecoder {
|
||||
return format
|
||||
}
|
||||
|
||||
private fun buildOpusIdHeader(): ByteArray {
|
||||
// RFC 7845 §5.1 — 19 bytes for mono, mapping family 0.
|
||||
private fun buildOpusIdHeader(channelCount: Int): ByteArray {
|
||||
// RFC 7845 §5.1 — 19 bytes, mapping family 0 (covers mono and
|
||||
// stereo with implicit L,R interleaving; no per-channel
|
||||
// mapping table required).
|
||||
val buf = ByteBuffer.allocate(19).order(ByteOrder.LITTLE_ENDIAN)
|
||||
buf.put("OpusHead".encodeToByteArray()) // 8 bytes magic
|
||||
buf.put(1.toByte()) // version
|
||||
buf.put(AudioFormat.CHANNELS.toByte()) // channel count
|
||||
buf.put(channelCount.toByte()) // channel count (1 or 2)
|
||||
buf.putShort(0) // pre-skip
|
||||
buf.putInt(AudioFormat.SAMPLE_RATE_HZ) // input sample rate
|
||||
buf.putShort(0) // output gain (Q7.8 dB)
|
||||
|
||||
Reference in New Issue
Block a user