fix(quic): omit IP-literal SNI, bound the handshake, fix dev harness host

Three findings from the local nests interop hang (relay logged "Illegal SNI extension: ignoring IP address presented as hostname"): - TlsClientHello sent the connect host verbatim as TLS SNI, including IP literals. RFC 6066 §3 forbids IP literals in server_name; a strict relay (rustls) discards the extension, ends up with no SNI to select its certificate on, and the handshake stalls. Both ClientHello builders now omit server_name when the host is an IP literal (new isIpLiteralHostname predicate). - QuicWebTransportFactory.connect never bounded awaitHandshake() — only readConnectResponse had a timeout — so a stalled handshake hung connect() indefinitely. It now fails fast after connectTimeoutMillis with a message naming the likely cause. - NostrNestsHarness pointed moqEndpoint at 127.0.0.1, which forced the IP-literal SNI path against the dev relay. Switched to localhost: a valid hostname that matches the relay's generated dev cert and still resolves to loopback. https://claude.ai/code/session_01PoQupfdoKU3ryQwLwTeXeM
2026-05-14 21:41:07 +00:00
parent 806d357569
commit ed1f10c071
4 changed files with 78 additions and 18 deletions
@@ -122,14 +122,31 @@ class QuicWebTransportFactory(
        val driver = QuicConnectionDriver(conn, socket, parentScope)
        driver.start()

-        try {
-            conn.awaitHandshake()
-        } catch (t: Throwable) {
+        val handshakeCompleted =
+            try {
+                // Bound the handshake — a server-side TLS failure (e.g. a relay
+                // that rejects an IP-literal SNI and can't select a cert) leaves
+                // the handshake stalled with no alert, so without this ceiling
+                // connect() hangs indefinitely instead of failing fast.
+                kotlinx.coroutines.withTimeoutOrNull(connectTimeoutMillis) {
+                    conn.awaitHandshake()
+                    true
+                }
+            } catch (t: Throwable) {
+                driver.close()
+                throw WebTransportException(
+                    kind = WebTransportException.Kind.HandshakeFailed,
+                    message = "QUIC handshake failed: ${t.message}",
+                    cause = t,
+                )
+            }
+        if (handshakeCompleted == null) {
            driver.close()
            throw WebTransportException(
                kind = WebTransportException.Kind.HandshakeFailed,
-                message = "QUIC handshake failed: ${t.message}",
-                cause = t,
+                message =
+                    "QUIC handshake stalled — no completion within ${connectTimeoutMillis}ms " +
+                        "(a server-side TLS failure such as a rejected IP-literal SNI hangs here with no alert)",
            )
        }
        if (conn.status != QuicConnection.Status.CONNECTED) {
@@ -205,7 +205,14 @@ class NostrNestsHarness private constructor(
                // The path under this base is the namespace literal (per
                // moq-rs `claims.root` matching); the `:nestsClient` connect
                // helpers append `/<namespace>?jwt=<token>` themselves.
-                moqEndpoint = "https://127.0.0.1:$MOQ_HOST_PORT/",
+                //
+                // Host MUST be `localhost`, not `127.0.0.1`: the QUIC client
+                // sends the host as TLS SNI, RFC 6066 §3 forbids IP literals
+                // there, and a strict relay (rustls) rejects an IP-literal SNI
+                // and then has no SNI to select its dev cert on — the handshake
+                // stalls. `localhost` is a valid hostname and matches the dev
+                // cert moq-relay generates. Resolves to 127.0.0.1 regardless.
+                moqEndpoint = "https://localhost:$MOQ_HOST_PORT/",
            )
        }

@@ -242,7 +249,9 @@ class NostrNestsHarness private constructor(
            return NostrNestsHarness(
                workDir = null,
                authBaseUrl = "http://127.0.0.1:$AUTH_HOST_PORT",
-                moqEndpoint = "https://127.0.0.1:$MOQ_HOST_PORT/",
+                // `localhost`, not `127.0.0.1` — see the doStart() return for
+                // why an IP-literal host breaks the TLS SNI / cert selection.
+                moqEndpoint = "https://localhost:$MOQ_HOST_PORT/",
            )
        }