Files
amethyst/nestsClient/src/commonMain/kotlin/com/vitorpamplona/nestsclient/NestsSpeaker.kt
T
Claude 6237c02c6f fix(nests): platform-side audio robustness — focus, AEC, route obs, network handover
Four follow-up fixes from the post-audit review (#4 / #5 / #6 / #7).

#6 AcousticEchoCanceler / NoiseSuppressor / AGC on the AudioRecord
   session. The VOICE_COMMUNICATION input source engages the platform
   echo canceller automatically on most modern Android devices, but a
   small set of older / OEM-customised devices only attach AEC under
   MODE_IN_COMMUNICATION — which an audio-room app deliberately
   avoids. Attaching the standalone audiofx effects to the
   AudioRecord's session id covers those devices without rerouting
   through the call audio path. All three are best-effort and a no-op
   on devices where the source already engages them.

#4 Real audio focus handling. The previous OnAudioFocusChangeListener
   was a no-op based on the assumption that the OS would auto-duck
   us; it doesn't (CONTENT_TYPE_SPEECH streams aren't auto-ducked).
   Inbound phone calls were mixing on top of room audio.
   - New `NestAudioFocusBus` (commons) — process-wide enum signal,
     decoupled from android.media so commons stays platform-free.
   - NestForegroundService translates AUDIOFOCUS_GAIN / LOSS_TRANSIENT*
     / LOSS into the bus enum.
   - NestViewModel observes the bus and silences both the listener
     playback (effective listen-mute = user OR focus) and the
     broadcast mic (effective mic-mute = user OR focus). User-visible
     mute states stay the user's choice so a focus regain restores
     them automatically.
   - The pipeline keeps running while focus is lost (decoder, capture,
     network) so unmute is sample-accurate when the call ends.

#5 AudioDeviceCallback observability. Registers a callback in
   NestForegroundService that logs Bluetooth / wired / USB headset
   attach + detach with device type + name. Doesn't drive playback
   decisions — Android's auto-routing handles route swaps — but
   makes "audio cut out when I plugged in headphones" reports
   correlatable with a concrete event for the first time.

#7 Network-change → fast reconnect. Without this, a Wi-Fi → cellular
   handover left the QUIC connection sitting on a now-dead socket
   until its PTO fired (~30 s) before the wrapper noticed. Now:
   - New `NestNetworkChangeBus` (commons) — collapses bursts of
     onLost/onAvailable into a single recycle event.
   - NestsListener + NestsSpeaker grow `recycleSession()` (default
     no-op); the reconnecting wrappers override to close the inner
     session so their orchestrator opens a fresh one.
   - NestForegroundService registers a default-network callback;
     suppresses the first onAvailable (registration callback)
     and only publishes on actual default-network changes.
   - NestViewModel observes the bus and calls recycleSession on
     both wrappers. The SubscribeHandle re-issuance pump (listener)
     and the hot-swap publisher pump (speaker) cut existing
     subscriptions / broadcasts onto the new session as soon as
     it lands — same paths the JWT-refresh recycle uses.
   - Manifest gains ACCESS_NETWORK_STATE for
     registerDefaultNetworkCallback.
2026-05-05 12:42:47 +00:00

294 lines
11 KiB
Kotlin

/*
* Copyright (c) 2025 Vitor Pamplona
*
* Permission is hereby granted, free of charge, to any person obtaining a copy of
* this software and associated documentation files (the "Software"), to deal in
* the Software without restriction, including without limitation the rights to use,
* copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the
* Software, and to permit persons to whom the Software is furnished to do so,
* subject to the following conditions:
*
* The above copyright notice and this permission notice shall be included in all
* copies or substantial portions of the Software.
*
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS
* FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR
* COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN
* AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
* WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
*/
package com.vitorpamplona.nestsclient
import com.vitorpamplona.nestsclient.audio.AudioCapture
import com.vitorpamplona.nestsclient.audio.NestBroadcaster
import com.vitorpamplona.nestsclient.audio.OpusEncoder
import com.vitorpamplona.nestsclient.moq.MoqProtocolException
import com.vitorpamplona.nestsclient.moq.MoqSession
import com.vitorpamplona.nestsclient.moq.TrackNamespace
import kotlinx.coroutines.CoroutineScope
import kotlinx.coroutines.flow.MutableStateFlow
import kotlinx.coroutines.flow.StateFlow
import kotlinx.coroutines.flow.asStateFlow
import kotlinx.coroutines.sync.Mutex
import kotlinx.coroutines.sync.withLock
/**
* High-level handle for the speaker / host audio path. Mirror of
* [NestsListener]: hides HTTP + WebTransport + MoQ wiring under one
* observable state machine, with the added ability to broadcast our own
* speaker track.
*
* Open one [NestsSpeaker] per audio-room screen where we have host or
* speaker permission. Call [startBroadcasting] when the user taps "Talk";
* call [BroadcastHandle.close] when they stop talking or leave the room.
*/
interface NestsSpeaker {
/** Connection / broadcast state — drives the speaker UI's status chip. */
val state: StateFlow<NestsSpeakerState>
/**
* Begin announcing our speaker track and pumping mic frames out as
* OBJECT_DATAGRAMs.
*
* [onLevel] fires once per Opus frame that successfully reaches
* the wire, with the peak amplitude of the underlying PCM frame
* normalized to `[0, 1]`. Default no-op so callers that don't
* render a local-speaking ring pay zero cost. The callback runs
* on the speaker's coroutine scope (typically the caller's
* `viewModelScope`) so the consumer must keep its handler
* lightweight. See [com.vitorpamplona.nestsclient.audio.NestMoqLiteBroadcaster.start]
* for the ground-truth-vs-UI-state rationale.
*
* @throws IllegalStateException if a broadcast is already running on
* this speaker or the session is not [NestsSpeakerState.Connected].
* @throws MoqProtocolException if the peer rejects the ANNOUNCE.
*/
suspend fun startBroadcasting(onLevel: (Float) -> Unit = { /* no-op */ }): BroadcastHandle
/**
* Force the underlying transport / MoQ session to be torn down and
* a fresh one opened in its place — without permanently closing
* this speaker. Mirror of [NestsListener.recycleSession]; same
* use case (network handover) and same default no-op for non-
* reconnecting implementations.
*
* The reconnecting wrapper from
* [connectReconnectingNestsSpeaker] overrides to close the
* inner speaker so its orchestrator opens a fresh session.
* For moq-lite speakers, the long-lived
* [com.vitorpamplona.nestsclient.audio.NestMoqLiteBroadcaster]
* keeps capturing through the recycle and the
* [BroadcastHandle.swapPublisher]-based pump retargets onto
* the new session's publisher with no audible gap.
*/
suspend fun recycleSession() {
// no-op — only the reconnecting wrapper has anywhere to
// recycle to.
}
/** Tear down the MoQ session + transport. Idempotent. */
suspend fun close()
}
/** Active broadcast on one [NestsSpeaker]. Returned from [NestsSpeaker.startBroadcasting]. */
interface BroadcastHandle {
/**
* Toggle whether mic frames reach the wire. The capture pipeline keeps
* running, so unmute is sample-accurate. Default unmuted.
*/
suspend fun setMuted(muted: Boolean)
/** Whether the next OBJECT_DATAGRAM the peer would receive is silenced. */
val isMuted: Boolean
/**
* Stop publishing. Releases the mic + encoder, sends UNANNOUNCE +
* SUBSCRIBE_DONE on the wire. Idempotent.
*/
suspend fun close()
}
/** Lifecycle states of a [NestsSpeaker]. */
sealed class NestsSpeakerState {
data object Idle : NestsSpeakerState()
data class Connecting(
val step: ConnectStep,
) : NestsSpeakerState() {
enum class ConnectStep {
ResolvingRoom,
OpeningTransport,
MoqHandshake,
}
}
/** Connection live; ready for [NestsSpeaker.startBroadcasting]. */
data class Connected(
val room: NestsRoomConfig,
val negotiatedMoqVersion: Long,
) : NestsSpeakerState()
/** Currently announcing + emitting OBJECT_DATAGRAMs for our track. */
data class Broadcasting(
val room: NestsRoomConfig,
val negotiatedMoqVersion: Long,
val isMuted: Boolean,
) : NestsSpeakerState()
/**
* The previous session dropped and the reconnect orchestrator
* is waiting [delayMs] before trying again. Mirror of
* [NestsListenerState.Reconnecting]; see that class's KDoc for
* the state-machine contract.
*/
data class Reconnecting(
val attempt: Int,
val delayMs: Long,
) : NestsSpeakerState()
data class Failed(
val reason: String,
val cause: Throwable? = null,
) : NestsSpeakerState()
data object Closed : NestsSpeakerState()
}
/**
* IETF `draft-ietf-moq-transport-17` [NestsSpeaker] reference
* implementation. **Not used in production** — the production speaker
* path uses [MoqLiteNestsSpeaker] over moq-lite Lite-03 (see
* `nestsClient/plans/2026-04-26-moq-lite-gap.md`). Kept for the IETF
* unit-test suite (`NestsSpeakerTest`) and for any future IETF
* MoQ-transport target.
*
* Wraps a connected [MoqSession] and plumbs an [NestBroadcaster]
* through it on [startBroadcasting]. Construction does NOT open the
* transport.
*/
class DefaultNestsSpeaker internal constructor(
private val session: MoqSession,
private val roomNamespace: TrackNamespace,
private val speakerTrackName: ByteArray,
private val captureFactory: () -> AudioCapture,
private val encoderFactory: () -> OpusEncoder,
private val scope: CoroutineScope,
private val mutableState: MutableStateFlow<NestsSpeakerState>,
) : NestsSpeaker {
override val state: StateFlow<NestsSpeakerState> = mutableState.asStateFlow()
private val gate = Mutex()
private var activeHandle: DefaultBroadcastHandle? = null
override suspend fun startBroadcasting(onLevel: (Float) -> Unit): BroadcastHandle {
gate.withLock {
val current = state.value
check(current is NestsSpeakerState.Connected) {
"startBroadcasting requires Connected state, was $current"
}
check(activeHandle == null) { "speaker is already broadcasting" }
val announce = session.announce(roomNamespace)
val publisher =
try {
announce.openTrack(speakerTrackName)
} catch (t: Throwable) {
runCatching { announce.unannounce() }
throw t
}
val broadcaster =
NestBroadcaster(
capture = captureFactory(),
encoder = encoderFactory(),
publisher = publisher,
scope = scope,
)
broadcaster.start(onLevel = onLevel)
mutableState.value =
NestsSpeakerState.Broadcasting(
room = current.room,
negotiatedMoqVersion = current.negotiatedMoqVersion,
isMuted = false,
)
val handle =
DefaultBroadcastHandle(
broadcaster = broadcaster,
announce = announce,
parent = this,
)
activeHandle = handle
return handle
}
}
/**
* Lockless: callable from inside [close] (which already holds [gate]) and
* from [DefaultBroadcastHandle.close] (which doesn't). We compare-and-set
* on activeHandle and update state atomically via the StateFlow.
*/
internal fun broadcastClosed(handle: DefaultBroadcastHandle) {
if (activeHandle !== handle) return
activeHandle = null
val current = mutableState.value
if (current is NestsSpeakerState.Broadcasting) {
mutableState.value =
NestsSpeakerState.Connected(current.room, current.negotiatedMoqVersion)
}
}
internal fun reportMuteState(muted: Boolean) {
val current = mutableState.value
if (current is NestsSpeakerState.Broadcasting) {
mutableState.value = current.copy(isMuted = muted)
}
}
override suspend fun close() {
// Take the active handle under `gate` (so a concurrent
// `startBroadcasting` can't observe a half-closed state), then
// release the lock before calling `handle.close()` and
// `session.close()` — both are long-running suspend operations
// (broadcaster.stop awaits cancelAndJoin; session.close fires
// SUBSCRIBE_DONE per attached subscriber and joins pumps). Holding
// the gate through them would block any other API call on this
// speaker for the entire teardown duration.
val handle: DefaultBroadcastHandle?
gate.withLock {
if (state.value is NestsSpeakerState.Closed) return
handle = activeHandle
activeHandle = null
mutableState.value = NestsSpeakerState.Closed
}
handle?.runCatching { close() }
runCatching { session.close() }
}
}
internal class DefaultBroadcastHandle(
private val broadcaster: NestBroadcaster,
private val announce: MoqSession.AnnounceHandle,
private val parent: DefaultNestsSpeaker,
) : BroadcastHandle {
@Volatile private var muted: Boolean = false
@Volatile private var closed: Boolean = false
override val isMuted: Boolean get() = muted
override suspend fun setMuted(muted: Boolean) {
if (closed) return
this.muted = muted
broadcaster.setMuted(muted)
parent.reportMuteState(muted)
}
override suspend fun close() {
if (closed) return
closed = true
runCatching { broadcaster.stop() }
runCatching { announce.unannounce() }
parent.broadcastClosed(this)
}
}