From f3637dd7c1d2e44d2fc9fa02dd55fd48f7fbb933 Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 19 May 2026 18:14:30 +0000 Subject: [PATCH] =?UTF-8?q?docs(quartz):=20revise=20local-headers-explorer?= =?UTF-8?q?=20plan=20=E2=80=94=20SQLite,=20no=20bundle,=20OTS-only?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Lock the design choices from the 2026-05-19 review against the intervening 2026-05-14 onchain-zaps work: - Move the module from quartz/.../nip03Timestamp/bitcoin/ to a sibling quartz/.../bitcoin/ package so the headers explorer, header validator, peer pool and store can be reused by the future onchain-zap merkle-proof work without an inverted import path. - Use androidx.sqlite + BundledSQLiteDriver for HeaderStore, matching the existing SQLiteEventStore. Schema lives in commonMain via IModule. Drops the hand-rolled flat-file + sidecar height index. - Drop the bundled headers blob. Ship a single hardcoded PinnedCheckpoint constant; first-run sync starts from the checkpoint and pulls forward over P2P. APK growth: 0 bytes. - Pre-checkpoint OTS heights fall through to OkHttpBitcoinExplorer via BitcoinExplorerEndpoint (shared with the onchain-zap EsploraBackend). Strict-mode users get an explicit error instead of a network call. - Mark trustless NIP-BC onchain-zap verification as out of scope and capture it as a follow-up plan (BIP-37 merkleblock or full-block fetch on top of this stack). Resolves open questions Q1, Q2 and Q4 from the original plan; Q3 (Quartz public API vs internal) left open for Phase 0. --- .../2026-05-08-local-headers-explorer.md | 411 +++++++++++++----- 1 file changed, 292 insertions(+), 119 deletions(-) diff --git a/quartz/plans/2026-05-08-local-headers-explorer.md b/quartz/plans/2026-05-08-local-headers-explorer.md index 5f621c183..424476028 100644 --- a/quartz/plans/2026-05-08-local-headers-explorer.md +++ b/quartz/plans/2026-05-08-local-headers-explorer.md @@ -1,10 +1,35 @@ # Local Bitcoin headers explorer for NIP-03 OTS -**Date:** 2026-05-08 -**Branch:** `claude/review-ots-blockchain-deps-bKns7` +**Date:** 2026-05-08 (revised 2026-05-19) +**Branch:** `claude/review-bitcoin-blockchain-plan-6NGZ5` (originally `claude/review-ots-blockchain-deps-bKns7`) **Module:** `quartz/` (with thin Android wiring in `amethyst/`) **Status:** Plan — not yet implemented +## Revision log + +**2026-05-19 — decisions locked, scope tightened.** After review against the +intervening 2026-05-14 onchain-zaps work: + +- **Scope stays OTS-only.** Trustless verification of NIP-BC onchain zaps + (kind 8333) needs more than headers — at minimum BIP-37 merkleblock or + full-block fetch on top of this stack. That becomes a follow-up plan + (see §20). +- **Module moves to `quartz/.../bitcoin/`** — sibling of `nip03Timestamp/` + and `nipBCOnchainZaps/`, not nested under NIP-03. The future merkle-block + phase will reuse `BitcoinPeer` / `HeaderStore` / `HeaderValidator`. +- **Storage is SQLite** via `androidx.sqlite` + `BundledSQLiteDriver`, + matching the existing `SQLiteEventStore` (`nip01Core/store/sqlite/`). + No flat append-only file; no hand-rolled height index. See §9. +- **No bundled headers file.** Ship a single hardcoded `Checkpoint(height, + blockHash, chainWork, time)` constant, bumped at release. First-run sync + starts at the checkpoint and pulls forward to the tip over P2P. See §7. +- **Pre-checkpoint OTS proofs fall through to HTTP** (`OkHttpBitcoinExplorer`). + No backward-from-checkpoint sync. Strict-mode users get a clear error + instead of opening an HTTP connection. +- **`BitcoinExplorerEndpoint`** (`amethyst/.../model/nip03Timestamp/`) is + now the shared Esplora-URL resolver for OTS and onchain zaps. The + composite resolver builder must read from it for the HTTP fallback path. + ## 1. Motivation NIP-03 (OpenTimestamps attestations, kind 1040) is currently verified by hitting @@ -42,8 +67,9 @@ proof-of-work chain instead of a trusted explorer. verification logic. - Work over Tor when the user has Tor enabled. - Keep existing HTTP explorers as a fallback for proofs older than the - bundled checkpoint, or when local sync hasn't completed yet. -- Live in `quartz/` so `cli/` and `desktopApp/` benefit too. + pinned checkpoint, or when local sync hasn't completed yet. +- Live in `quartz/src/commonMain/.../bitcoin/` so `cli/`, `desktopApp/` and + any future merkle-block / onchain-zap work inherit the same primitives. ### Non-goals @@ -70,12 +96,13 @@ proof-of-work chain instead of a trusted explorer. ┌──────────────┼──────────────┐ │ │ │ OkHttpBitcoinExplorer │ LocalHeadersBitcoinExplorer ◄── NEW - (existing, fallback) │ │ - │ ▼ - │ ┌─────────────────────────┐ - │ │ HeaderStore │ - │ │ (append-only file + │ - │ │ height index) │ + (existing, fallback, │ │ + URL from │ ▼ + BitcoinExplorer- │ ┌─────────────────────────┐ + Endpoint) │ │ HeaderStore │ + │ │ (SQLite, androidx- │ + │ │ sqlite + Bundled- │ + │ │ SQLiteDriver) │ │ └─────────────────────────┘ │ ▲ │ │ @@ -105,17 +132,20 @@ proof-of-work chain instead of a trusted explorer. │ CompositeBitcoinExplorer (try local first; if height - not synced yet, fall back to HTTP) + below pinned checkpoint OR not yet + reached by sync, fall back to HTTP + via BitcoinExplorerEndpoint) ``` ## 4. Module layout -All new code in `quartz/` so non-Android targets (CLI, Desktop) inherit it. -Most of the protocol code is platform-independent; only socket I/O is in -`jvmAndroid`. +All new code in `quartz/src/commonMain/.../bitcoin/`, sibling of +`nip03Timestamp/` and `nipBCOnchainZaps/`. Protocol code is +platform-independent; socket I/O is the only `expect`/`actual` boundary. +Storage uses `androidx.sqlite` which is already KMP — no platform split. ``` -quartz/src/commonMain/kotlin/com/vitorpamplona/quartz/nip03Timestamp/bitcoin/ +quartz/src/commonMain/kotlin/com/vitorpamplona/quartz/bitcoin/ ├── proto/ │ ├── BitcoinNetMagic.kt # 0xf9beb4d9 mainnet │ ├── MessageHeader.kt # 24-byte envelope codec @@ -128,47 +158,50 @@ quartz/src/commonMain/kotlin/com/vitorpamplona/quartz/nip03Timestamp/bitcoin/ │ │ ├── PingMessage.kt │ │ ├── PongMessage.kt │ │ ├── GetHeadersMessage.kt # locator + stop hash -│ │ └── HeadersMessage.kt # up to 2,000 × (80B header + varint=0) +│ │ ├── HeadersMessage.kt # up to 2,000 × (80B header + varint=0) +│ │ └── SendHeadersMessage.kt # BIP-130 │ └── BitcoinMessage.kt # sealed interface ├── header/ │ ├── BlockHeader80.kt # 80-byte parsed header (ver, prev, merkle, time, bits, nonce) -│ ├── BlockHash.kt # value class wrapping ByteArray(32) +│ ├── BlockHash.kt # @JvmInline value class wrapping ByteArray(32) │ ├── HeaderHasher.kt # dsha256 over 80B → blockHash │ ├── DifficultyTarget.kt # bits ↔ target (256-bit) compact form -│ ├── ChainWork.kt # cumulative work accumulator +│ ├── ChainWork.kt # cumulative work accumulator (uint256 BE) │ └── PowValidator.kt # checks hash ≤ target ├── consensus/ │ ├── RetargetCalculator.kt # every 2016 blocks, BIP-? rules │ ├── MedianTimePast.kt # last-11 median, for time sanity │ └── HeaderValidator.kt # composes the above + linkage check ├── store/ -│ ├── HeaderStore.kt # append-only file + height→offset index +│ ├── HeaderStore.kt # public API: append/getByHeight/getByHash/tip +│ ├── SqliteHeaderStore.kt # androidx.sqlite impl (commonMain — driver-agnostic) +│ ├── HeaderStoreSchema.kt # IModule for schema + migrations (see §9) │ ├── Checkpoint.kt # (height, blockHash, chainWork, time) -│ └── HeaderRecord.kt # 80B header + cached blockHash + cumulativeChainWork +│ └── PinnedCheckpoint.kt # hardcoded release-time checkpoint constant ├── sync/ │ ├── LocatorBuilder.kt # exponential locator │ ├── HeadersSyncEngine.kt # drives getheaders loop, applies validator, persists │ └── ReorgHandler.kt # rare; switches to higher-chainwork tip ├── peer/ +│ ├── PeerSocket.kt # expect (open / send / recv / close) │ ├── PeerPool.kt # holds N BitcoinPeer connections │ ├── BitcoinPeer.kt # high-level: handshake, send msg, receive msg -│ └── PeerScoring.kt # ban misbehaving peers -└── LocalHeadersBitcoinExplorer.kt # implements quartz.../ots/BitcoinExplorer +│ ├── PeerScoring.kt # ban misbehaving peers +│ └── PeerDiscovery.kt # expect — DNS seeds + fixed seeds +└── LocalHeadersBitcoinExplorer.kt # implements nip03Timestamp.ots.BitcoinExplorer + # (with small LRU on top, like OtsBlockHeightCache) -quartz/src/jvmAndroid/kotlin/com/vitorpamplona/quartz/nip03Timestamp/bitcoin/ -├── peer/ -│ ├── TcpPeerSocket.jvmAndroid.kt # actual TCP socket (java.net.Socket) -│ └── DnsSeedDiscovery.jvmAndroid.kt # InetAddress.getAllByName on seed hostnames -└── store/ - └── FileHeaderStore.jvmAndroid.kt # RandomAccessFile-backed HeaderStore - -quartz/src/commonMain/kotlin/com/vitorpamplona/quartz/nip03Timestamp/bitcoin/ -└── PeerSocket.kt # expect (open / send / recv / close) -└── HeaderStorage.kt # expect (append, getByHeight, tip) +quartz/src/jvmAndroid/kotlin/com/vitorpamplona/quartz/bitcoin/ +└── peer/ + ├── TcpPeerSocket.jvmAndroid.kt # actual TCP socket (java.net.Socket / SOCKS) + └── DnsSeedDiscovery.jvmAndroid.kt # InetAddress.getAllByName on seed hostnames ``` -The `expect`/`actual` split keeps protocol logic testable cross-platform; the -socket layer and disk layer are the only platform-dependent pieces. +Two expect/actual seams only: `PeerSocket` (TCP plus optional SOCKS5 for Tor) +and `PeerDiscovery` (DNS-seed resolution). Header validation, message codecs, +store, sync engine, peer pool — all compile in `commonMain`, all testable +cross-platform without a single platform shim. Storage compiles in +`commonMain` too because `androidx.sqlite` + `BundledSQLiteDriver` are KMP. ### Wiring on Android @@ -182,7 +215,17 @@ amethyst/src/main/java/com/vitorpamplona/amethyst/model/nip03Timestamp/ `AppModules.kt` switches from `TorAwareOkHttpOtsResolverBuilder` to a composite that prefers the local explorer and falls back to HTTP only when -local is missing the height. +local is missing the height. The HTTP path resolves its base URL via the +existing `BitcoinExplorerEndpoint` (shared with the onchain-zap +`EsploraBackend`), so a user-configured custom Esplora is honoured on both +the OTS and the zap side. + +`AppModules.kt:433` already passes `rootFilesDir = { appContext.filesDir }` +into Quartz; the headers SQLite file goes under +`$filesDir/bitcoin/headers.db` via a new entry in that wiring. + +On desktop and CLI, the equivalent data-dir paths are already plumbed for +`SQLiteEventStore` — same hook reused here. ## 5. Bitcoin P2P protocol — minimum we implement @@ -274,23 +317,35 @@ mirroring Bitcoin Core's own historical checkpoints, plus one **release-pinned checkpoint** updated at app build time near the current tip. Sync must agree with all checkpoints — disagreement aborts and bans the peer. -## 7. Bundled data +## 7. Bundled data — none. Just a pinned checkpoint constant. -To avoid making every fresh install download 75 MB: +The point of this work is "verify against the Bitcoin network, not a trusted +party," so we ship no headers in the APK. Instead: -- App ships an **assets file** `bitcoin_headers_.bin`: raw 80-byte - headers from genesis up to a release-pinned height (≤ 1 month before build). - ~72 MB — too large for Play Store delta-update friendliness. -- Ship instead a **compact bundled checkpoint**: just one `Checkpoint(height, - blockHash, chainWork, time)` per app release, plus the most recent ~50 K - headers (~4 MB) so users can verify any attestation from the past ~year - immediately. -- For older proofs, lazy-sync from the checkpoint *backwards* down to genesis - on demand (rare path). -- Forward sync from checkpoint to current tip is what runs at first launch - (~10 KB to a few MB depending on release age). +- **One hardcoded `PinnedCheckpoint` constant** in `bitcoin/store/PinnedCheckpoint.kt`: + `(height, blockHash, chainWork, time)`. ~80 bytes of source. +- The checkpoint is pinned ~1 month before each release and bumped via a + one-line PR per release. This is the **only** thing trusted at build time + — and it's a single block hash any user can independently verify against + any Bitcoin node before installing. +- **First-run sync** starts at the checkpoint and pulls forward to the + current tip over P2P. At a ~1-month checkpoint that's ~4,300 headers + (~340 KB on the wire, ~15 minutes' worth of `getheaders` round-trips at + worst), seconds over WiFi. +- **Pre-checkpoint heights** are not synced locally at all. OTS proofs + whose height precedes the checkpoint fall through to + `OkHttpBitcoinExplorer` via the composite. Users in strict-mode + (`localHeadersTrustOnly = true`) get a clear + "this OTS proof predates the bundled checkpoint" error and no network call. +- **Catch-up after long absence.** A device that hasn't run the app for + months will, on next sync, fast-forward from `tip_local` (its stored + best_height) to `tip_network`. Each `headers` message brings 2,000 + headers; even 6 months of absence is ~26,000 headers (~2 MB) in 13 round + trips. +- **Header growth on disk** is ~4 MB raw per year (+ SQLite overhead, see §9). -This keeps APK growth at ~4 MB and first-run sync time at seconds. +APK growth: **0 bytes**. Trust delta vs the old HTTP-explorer model: one +block hash, code-reviewable in git. ## 8. Multi-peer strategy & eclipse mitigation @@ -315,42 +370,85 @@ A single peer can lie. We protect against that with: (`bitcoin.example.com:8333`) — same path as the existing custom explorer URL setting in `OtsSettings.kt`. -## 9. Storage format +## 9. Storage — SQLite via `androidx.sqlite` + `BundledSQLiteDriver` -Append-only flat file plus a sparse index. +We reuse the same storage primitive Quartz already uses for `SQLiteEventStore` +(`quartz/.../nip01Core/store/sqlite/`): `androidx.sqlite` with the bundled +driver. This is KMP, ships in `androidx-sqlite-bundled` / `…-bundled-jvm` +(already a Quartz dependency), and is the project's standard storage choice. -### `headers.bin` +The complete `SQLiteEventStore` infrastructure — `BundledSQLiteDriver`, +`SQLiteConnectionPool`, `transaction { }`, `IModule` for migrations, +`QueryBuilder` helpers — is reused by `SqliteHeaderStore` and +`HeaderStoreSchema`. -Records, fixed 112 bytes each, indexed by height (record N at offset `N*112`): +### Schema -``` -+0 80B raw block header -+80 32B cumulative chainwork (uint256, big-endian) +```sql +PRAGMA journal_mode = WAL; +PRAGMA synchronous = NORMAL; +PRAGMA foreign_keys = OFF; + +CREATE TABLE block_header ( + height INTEGER PRIMARY KEY, -- 0..tip, dense + hash BLOB NOT NULL, -- 32 B, dsha256(header), big-endian display order + header BLOB NOT NULL, -- 80 B raw header + chainwork BLOB NOT NULL -- 32 B cumulative work, big-endian +); + +CREATE UNIQUE INDEX idx_block_header_hash ON block_header(hash); + +CREATE TABLE chain_tip ( + id INTEGER PRIMARY KEY CHECK (id = 1), -- singleton row + best_height INTEGER NOT NULL, + best_hash BLOB NOT NULL, + best_work BLOB NOT NULL +); + +CREATE TABLE checkpoint_seen ( + height INTEGER PRIMARY KEY, -- which checkpoints we've passed and confirmed + hash BLOB NOT NULL, + source TEXT NOT NULL -- 'pinned' | 'historical' +); ``` -The block hash and target are recomputed from bytes 0..80 lazily; we don't -store them. Total at height 1M: 112 MB. We can drop this to 80 B/record -(no chainwork) if we accept recomputing chainwork on startup — TBD; default -is 112 B for fast tip selection. +Migrations live in `HeaderStoreSchema.kt` as an `IModule` (same pattern as +the event store's modules). v1 is the schema above; future additions +(prev-hash cache column, merkle-block annotations) ALTER from there. -### `tips.json` +### Sizing -A small JSON file holding: +| Item | Per record | At height 1,000,000 | +|---|---|---| +| `block_header` row (80+32+32 B payload + SQLite overhead) | ~180–220 B | ~180–220 MB | +| `idx_block_header_hash` B-tree (32 B key + rowid) | ~40 B | ~40 MB | +| `chain_tip` | 1 row | negligible | +| **Total** | | **~220–260 MB at height 1 M** | -```json -{ - "best_height": 945123, - "best_hash": "00000000000000000001abc...", - "best_work": "0x000...", - "checkpoints_passed": [600000, 700000, 800000, 940000] -} -``` +In practice users will start from a recent checkpoint and grow ~4 MB raw +(~10 MB stored) per year, so most installs sit at 10–40 MB. The "1 M +height" line is the long-tail worst case if someone with a multi-year-old +device's local store retroactively backfills (which we explicitly don't do +in v1 — see §15 Q1). -### Atomicity +### Atomicity & crash safety -Sync writes to `headers.bin.tmp`, fsyncs, then atomically renames or -appends-and-fsyncs in batches of 2000. Crash recovery: on startup, scan from -last fsynced offset, validate linkage, and discard partial trailing records. +- `HeadersSyncEngine` writes each `headers` batch (up to 2,000) inside a + single `transaction { }`. WAL gives us atomic-or-nothing per batch. +- `chain_tip` is updated in the same transaction as the headers it points to, + so the tip can never reference rows that didn't commit. +- A crash mid-batch leaves the WAL inconsistent; SQLite rolls it back on + next open. No custom recovery code. +- On open, `SqliteHeaderStore.verifyTipIntegrity()` re-reads the tip row and + confirms the referenced header exists. If not (unexpected DB corruption, + ransomware-like sandbox damage), we delete the db file and fall back to + re-sync from the pinned checkpoint. + +### Concurrency + +Same model as `SQLiteEventStore`: one connection pool, all writes go through +the sync engine on a dedicated dispatcher; reads use the pool's read slots. +OTS verifier reads are short and indexed — no contention with sync writes. ## 10. Resolving a height query (the actual `BitcoinExplorer` impl) @@ -410,20 +508,25 @@ Extend `OtsSettings.kt`: ```kotlin data class OtsSettings( - val customExplorerUrl: String? = null, + val customExplorerUrl: String? = null, // existing — shared via BitcoinExplorerEndpoint // NEW: val localHeadersEnabled: Boolean = true, - val localHeadersTrustOnly: Boolean = false, // disables HTTP fallback + val localHeadersTrustOnly: Boolean = false, // disables HTTP fallback; pre-checkpoint proofs error val localHeadersWifiOnly: Boolean = true, - val customP2pNode: String? = null, // e.g. "myriad.example.com:8333" + val customP2pNode: String? = null, // e.g. "myriad.example.com:8333" ) ``` A short "Bitcoin verification" settings screen lets the user pick: -- Local headers (default) -- HTTP explorer only (current behavior, lighter) -- Local + HTTP fallback -- Local only (strict, may delay verification of brand-new attestations) +- **Local headers + HTTP fallback** (default) — best UX, trustless for + post-checkpoint heights, HTTP for older proofs. +- **Local headers only (strict)** — refuses to verify proofs older than the + pinned checkpoint. Most paranoid setting. +- **HTTP explorer only** — pre-local-headers behaviour. Lightest disk/CPU. + +The `customExplorerUrl` field continues to feed `BitcoinExplorerEndpoint`, +so a custom HTTP Esplora is honoured both by the fallback in this feature +and by the onchain-zaps `EsploraBackend`. No new Esplora setting. ## 12. Tor integration @@ -461,10 +564,15 @@ Each phase produces a shippable artifact with measurable behavior. ### Phase 2 — Storage layer (2–3 days) -- `HeaderStore` interface + `FileHeaderStore` actual on jvmAndroid. -- Append, get-by-height, get-by-hash (in-memory hash→height map built at - startup, ~60 MB at height 1 M — acceptable, or use a sparse on-disk index). -- Crash-recovery test. +- `HeaderStore` interface + `SqliteHeaderStore` implementation in `commonMain` + (`androidx.sqlite` is KMP, no platform shim needed). +- `HeaderStoreSchema` as an `IModule` matching the `SQLiteEventStore` + migration pattern. +- `append(batch)`, `getByHeight(h)`, `getByHash(hash)`, `tip()`. +- Reuse `BundledSQLiteDriver`, `SQLiteConnectionPool`, `transaction { }` + from `nip01Core/store/sqlite/` — no duplication. +- Crash-safety test: kill mid-transaction, reopen, confirm WAL rollback; + confirm `verifyTipIntegrity()` recovers cleanly. ### Phase 3 — P2P codec, no socket (3–5 days) @@ -488,13 +596,16 @@ Each phase produces a shippable artifact with measurable behavior. - Header-batch cross-validation across peers. - Hardcoded checkpoints in `Checkpoints.kt`. -### Phase 6 — Bundled checkpoint & lazy-genesis-sync (2–3 days) +### Phase 6 — Pinned checkpoint constant (½ day) -- Build-time generation of the release-pinned checkpoint and trailing-50K - headers blob, packaged as `quartz/src/jvmAndroid/resources/bitcoin/...`. -- Forward sync from checkpoint to tip on first launch. -- Backward (genesis-direction) sync triggered only when an OTS proof requests - a height before the checkpoint. +- `bitcoin/store/PinnedCheckpoint.kt` — single hardcoded + `Checkpoint(height, blockHash, chainWork, time)`. ~80 bytes of source. +- A small Gradle task (`./gradlew bumpBitcoinCheckpoint`) that takes + `(height, hash)` arguments and writes the constant. Run by the release + process; otherwise no automation. +- No backward-sync, no bundled headers blob, no asset resources. + Pre-checkpoint heights are routed to the HTTP fallback by the composite + resolver (or refused in strict mode). ### Phase 7 — Android wiring (3–4 days) @@ -512,9 +623,14 @@ Each phase produces a shippable artifact with measurable behavior. ### Phase 9 — Desktop & CLI integration (1–2 days) -- `desktopApp/` already inherits the explorer because it's in `quartz/`. - Confirm it works end-to-end on JVM. -- `cli/` (`amy verify-ots`) — confirm CLI sync also works headlessly. +- `desktopApp/` inherits the explorer for free (everything is in + `quartz/commonMain/`). Confirm headers.db path is plumbed via the same + data-dir hook already used by `SQLiteEventStore` on JVM. Confirm sync + runs end-to-end on the desktop app. +- `cli/` — add an `amy verify-ots ` command if not present, + wired through the same composite resolver. The CLI is a thin assembly + layer over Quartz (per `amy-expert`), so the actual sync logic is reused + as-is. ### Phase 10 — Rollout & telemetry (ongoing) @@ -523,8 +639,9 @@ Each phase produces a shippable artifact with measurable behavior. fallback rate. Surface in a debug screen. - Default-on in the next release if metrics look healthy. -Total: **~25–35 engineer-days** end to end. Phases 1–4 are the bulk -(protocol + validation); phases 5–10 are integration. +Total: **~22–32 engineer-days** end to end. Phases 1–4 are the bulk +(protocol + validation); phases 5–10 are integration. Phase 6 dropped from +2–3 d to ½ d because we ship no bundled data — just one constant. ## 14. Testing strategy @@ -544,8 +661,12 @@ Total: **~25–35 engineer-days** end to end. Phases 1–4 are the bulk (committed as `quartz/src/jvmAndroid/test/resources/btc-handshake.bin`). - Local-network test against a Bitcoin Core regtest node spun up in CI (Docker). Validates full handshake + `getheaders` loop end to end. -- Crash-and-resume test: kill mid-sync, restart, confirm we pick up at the - correct height. +- `SqliteHeaderStore` round-trip on a real on-disk file (jvmTest, in a + `tmpdir`): append 5,000 headers in batches, getByHeight, getByHash, + reopen, confirm tip survives. +- Crash-and-resume test: kill the writer in the middle of a `transaction { }`, + reopen, confirm SQLite WAL rolls the partial batch back and sync resumes + from the last committed tip. ### Property tests @@ -564,32 +685,41 @@ Total: **~25–35 engineer-days** end to end. Phases 1–4 are the bulk | # | Risk | Mitigation | |---|------|------------| -| R1 | First-run UX cost (1–10 min sync) puts users off | Bundled checkpoint + 50 K trailing headers means most installs need only seconds. Show progress prominently. | -| R2 | Mobile OS kills socket mid-sync | Foreground service for first run; resumable design. | +| R1 | First-run UX cost (sync from pinned checkpoint) | Pinned checkpoint ~1 month before release means ~4 K headers, ~340 KB, seconds over WiFi. Show progress in settings; sync runs in background WorkManager job anyway. | +| R2 | Mobile OS kills socket mid-sync | Foreground service for first run; resumable design (SQLite WAL handles partial batches automatically). | | R3 | Peer feeds adversarial headers (eclipse) | Multi-peer + checkpoints + chainwork comparison. | -| R4 | Disk corruption | Atomic appends, checksum on close, fall back to re-sync from last checkpoint. | -| R5 | Tor + P2P is slow | Increased timeouts; fallback to HTTP-over-Tor explorer when sync stalls. | -| R6 | APK size growth from bundled headers | Cap bundle at most-recent ~50 K headers (~4 MB). Older headers fetched on demand. | +| R4 | Disk corruption | SQLite WAL + `verifyTipIntegrity()` on open; on detected corruption, delete db file and re-sync from pinned checkpoint. | +| R5 | Tor + P2P is slow | Increased timeouts; fallback to HTTP-over-Tor explorer (via `BitcoinExplorerEndpoint`) when sync stalls. | +| R6 | APK size growth | Zero — no bundled data ships in the APK. | | R7 | Privacy: peers see user's IP | Tor mode supported; `version` user-agent doesn't leak Amethyst install ID. | | R8 | Battery cost on background sync | Default to UNMETERED + BATTERY_OK constraints; back off when idle. | | R9 | Maintenance burden — new soft-forks | Soft-forks don't affect headers-only validation (no script eval). Difficulty algorithm hasn't changed; a future change would require a code update. | | R10 | Test flakiness if CI talks to public Bitcoin nodes | Use Bitcoin Core regtest in Docker for integration tests; public nodes only nightly. | +| R11 | SQLite store size on long-running installs | ~10 MB/year typical; ~220 MB at height 1 M which won't happen in any realistic mobile lifetime. Add a "wipe local headers" button in settings for users who want it. | ### Open questions -- **Q1.** Do we actually want backward-from-checkpoint sync, or just refuse - to verify proofs older than the bundled checkpoint and tell users to use - the HTTP fallback? Backward sync is operational complexity for a rare path. -- **Q2.** Storage: 80 B/record (recompute chainwork on startup) vs - 112 B/record (cache it)? At height 1M, 32 MB difference. Probably worth - caching. -- **Q3.** Should we expose this as a Quartz public API (so 3rd-party apps - using Quartz also benefit) or keep it internal to NIP-03? Recommendation: - public — it's generally useful. -- **Q4.** Do we ship a "headers blob" download URL (HTTPS) as a faster - alternative to P2P for first-run? Trade-off: faster but adds a trust point. - Could be made cross-checked (download blob, then verify every header's PoW - locally — same trust as P2P then). Worth considering as Phase 6b. +#### Resolved 2026-05-19 + +- **Q1.** *Resolved — HTTP fallback only.* Pre-checkpoint OTS proofs are + routed to `OkHttpBitcoinExplorer` via the composite resolver. Strict-mode + users (`localHeadersTrustOnly = true`) get a clear error. No backward + sync code path. +- **Q2.** *Resolved — cache chainwork.* SQLite makes the choice cheap (one + extra BLOB column). Reading current tip work without a full scan matters + for chain-selection logic; the ~30 MB-at-1M cost is negligible on disk. +- **Q4.** *Resolved — no HTTPS headers blob.* The point of this feature is + trustless verification, and the pinned-checkpoint approach already gives + fast first-run. Adding a second ingestion path (signed blob over HTTPS) + doubles the test surface for marginal benefit. + +#### Still open + +- **Q3.** Should `LocalHeadersBitcoinExplorer` (and friends) be a Quartz + public API for 3rd-party Quartz consumers, or internal to Amethyst? + Recommendation: public — it's a clean reusable primitive. Decide before + Phase 0 lands so the package layout doesn't churn. *(See `quartz-integration` + skill for the public-API contract Quartz exposes today.)* ## 16. Migration & rollout @@ -605,22 +735,55 @@ Total: **~25–35 engineer-days** end to end. Phases 1–4 are the bulk - A NIP-03 attestation that verifies via Blockstream also verifies via the local explorer (bit-for-bit identical timestamp). - 95th-percentile cold-launch verification of a recent attestation: ≤2 s. -- 95th-percentile first-run sync (post-checkpoint): ≤30 s on WiFi. +- 95th-percentile first-run sync (from pinned checkpoint to tip): ≤30 s on + WiFi for a checkpoint ≤2 months old. - No new crash-rate regression in the verification path. -- App size growth: ≤5 MB. +- App size growth: **0 MB** (no bundled data). - A user with HTTP explorer disabled can still verify any post-checkpoint attestation. ## 18. Out of scope (explicit) - BIP-157/158 compact filters +- BIP-37 merkleblock or full-block fetch +- **Trustless verification of NIP-BC onchain zaps** — needs merkleblock or + full-block fetch on top of this stack. See §20 for the follow-up plan. +- **Trustless wallet operations** for NIP-BC onchain zap *sending* — + needs UTXO-by-address and broadcast paths that headers can't provide. + Stays on user-configured Esplora. - Lightning, BIP-32 wallet, mempool monitoring - Verifying NIP-03 attestations against altcoin/Liquid timestamps (not in the spec) - Replacing the calendar-server upgrade path (`OtsState.upgrade()`) — unrelated, those are simple HTTPS calls and stay as-is -## 19. References +## 19. Follow-up: trustless NIP-BC onchain zap verification + +*Not in this plan. Tracked here so the design choices above stay aligned.* + +The shipped `OnchainZapVerifier` (`quartz/.../nipBCOnchainZaps/verify/`) +trusts `OnchainBackend.getTx(txid)` — i.e. Esplora — for both +tx-existence-in-a-block and the output values used to compute +`verifiedSats`. A malicious Esplora can fabricate or suppress zaps. + +Closing that gap is a **separate plan** that builds on this one: + +- **Foundation reused:** `BitcoinPeer`, `PeerPool`, `HeaderStore`, + `HeaderValidator`, `LocalHeadersBitcoinExplorer`. +- **New work (rough ~10–15 days on top of S1):** BIP-37 `filterload` / + `merkleblock` / matched-`tx` handling, or `getdata MSG_BLOCK` + stream + parse; partial-merkle-tree verification; new `OnchainBackend` variant + that asks the local stack for the inclusion proof and the matched tx + while keeping Esplora for UTXO listing / broadcast / fee estimates on + the *send* side. +- **Net win:** zap-receive verification stops trusting Esplora. Zap-send + still needs an address-indexed server (no headers-only equivalent + exists), so that part of the trust story is unchanged. +- **Filename:** `quartz/plans/YYYY-MM-DD-onchain-zap-merkle-proofs.md`, + written once S1 has shipped and we have real numbers on first-run sync, + store size, and battery cost. + +## 20. References - NIP-03: https://github.com/nostr-protocol/nips/blob/master/03.md - OpenTimestamps spec: https://github.com/opentimestamps/python-opentimestamps @@ -632,6 +795,16 @@ Total: **~25–35 engineer-days** end to end. Phases 1–4 are the bulk - `quartz/src/commonMain/kotlin/com/vitorpamplona/quartz/nip03Timestamp/ots/BitcoinExplorer.kt` - `quartz/src/commonMain/kotlin/com/vitorpamplona/quartz/nip03Timestamp/ots/BlockHeader.kt` - `quartz/src/commonMain/kotlin/com/vitorpamplona/quartz/nip03Timestamp/ots/attestation/BitcoinBlockHeaderAttestation.kt` + - `quartz/src/commonMain/kotlin/com/vitorpamplona/quartz/nip03Timestamp/ots/OtsBlockHeightCache.kt` - `quartz/src/jvmAndroid/kotlin/com/vitorpamplona/quartz/nip03Timestamp/okhttp/OkHttpBitcoinExplorer.kt` - `amethyst/src/main/java/com/vitorpamplona/amethyst/model/nip03Timestamp/TorAwareOkHttpOtsResolverBuilder.kt` - `amethyst/src/main/java/com/vitorpamplona/amethyst/model/nip03Timestamp/OtsSettings.kt` + - `amethyst/src/main/java/com/vitorpamplona/amethyst/model/nip03Timestamp/BitcoinExplorerEndpoint.kt` +- Storage precedent (reused as-is): + - `quartz/src/commonMain/kotlin/com/vitorpamplona/quartz/nip01Core/store/sqlite/SQLiteEventStore.kt` + - `quartz/src/commonMain/kotlin/com/vitorpamplona/quartz/nip01Core/store/sqlite/IModule.kt` + - `quartz/src/commonMain/kotlin/com/vitorpamplona/quartz/nip01Core/store/sqlite/SQLiteConnectionExt.kt` + - `quartz/src/commonMain/kotlin/com/vitorpamplona/quartz/nip01Core/store/sqlite/SQLiteConnectionPool.kt` +- Follow-up area: + - `quartz/src/commonMain/kotlin/com/vitorpamplona/quartz/nipBCOnchainZaps/verify/OnchainZapVerifier.kt` + - `quartz/src/commonMain/kotlin/com/vitorpamplona/quartz/nipBCOnchainZaps/chain/OnchainBackend.kt`