Claude
f3637dd7c1
Lock the design choices from the 2026-05-19 review against the intervening 2026-05-14 onchain-zaps work: - Move the module from quartz/.../nip03Timestamp/bitcoin/ to a sibling quartz/.../bitcoin/ package so the headers explorer, header validator, peer pool and store can be reused by the future onchain-zap merkle-proof work without an inverted import path. - Use androidx.sqlite + BundledSQLiteDriver for HeaderStore, matching the existing SQLiteEventStore. Schema lives in commonMain via IModule. Drops the hand-rolled flat-file + sidecar height index. - Drop the bundled headers blob. Ship a single hardcoded PinnedCheckpoint constant; first-run sync starts from the checkpoint and pulls forward over P2P. APK growth: 0 bytes. - Pre-checkpoint OTS heights fall through to OkHttpBitcoinExplorer via BitcoinExplorerEndpoint (shared with the onchain-zap EsploraBackend). Strict-mode users get an explicit error instead of a network call. - Mark trustless NIP-BC onchain-zap verification as out of scope and capture it as a follow-up plan (BIP-37 merkleblock or full-block fetch on top of this stack). Resolves open questions Q1, Q2 and Q4 from the original plan; Q3 (Quartz public API vs internal) left open for Phase 0.
811 lines
40 KiB
Markdown
811 lines
40 KiB
Markdown
# Local Bitcoin headers explorer for NIP-03 OTS
|
||
|
||
**Date:** 2026-05-08 (revised 2026-05-19)
|
||
**Branch:** `claude/review-bitcoin-blockchain-plan-6NGZ5` (originally `claude/review-ots-blockchain-deps-bKns7`)
|
||
**Module:** `quartz/` (with thin Android wiring in `amethyst/`)
|
||
**Status:** Plan — not yet implemented
|
||
|
||
## Revision log
|
||
|
||
**2026-05-19 — decisions locked, scope tightened.** After review against the
|
||
intervening 2026-05-14 onchain-zaps work:
|
||
|
||
- **Scope stays OTS-only.** Trustless verification of NIP-BC onchain zaps
|
||
(kind 8333) needs more than headers — at minimum BIP-37 merkleblock or
|
||
full-block fetch on top of this stack. That becomes a follow-up plan
|
||
(see §20).
|
||
- **Module moves to `quartz/.../bitcoin/`** — sibling of `nip03Timestamp/`
|
||
and `nipBCOnchainZaps/`, not nested under NIP-03. The future merkle-block
|
||
phase will reuse `BitcoinPeer` / `HeaderStore` / `HeaderValidator`.
|
||
- **Storage is SQLite** via `androidx.sqlite` + `BundledSQLiteDriver`,
|
||
matching the existing `SQLiteEventStore` (`nip01Core/store/sqlite/`).
|
||
No flat append-only file; no hand-rolled height index. See §9.
|
||
- **No bundled headers file.** Ship a single hardcoded `Checkpoint(height,
|
||
blockHash, chainWork, time)` constant, bumped at release. First-run sync
|
||
starts at the checkpoint and pulls forward to the tip over P2P. See §7.
|
||
- **Pre-checkpoint OTS proofs fall through to HTTP** (`OkHttpBitcoinExplorer`).
|
||
No backward-from-checkpoint sync. Strict-mode users get a clear error
|
||
instead of opening an HTTP connection.
|
||
- **`BitcoinExplorerEndpoint`** (`amethyst/.../model/nip03Timestamp/`) is
|
||
now the shared Esplora-URL resolver for OTS and onchain zaps. The
|
||
composite resolver builder must read from it for the HTTP fallback path.
|
||
|
||
## 1. Motivation
|
||
|
||
NIP-03 (OpenTimestamps attestations, kind 1040) is currently verified by hitting
|
||
a third-party block explorer (`blockstream.info` or `mempool.space`) over HTTPS.
|
||
That explorer is a trusted oracle: it can lie about a block's `merkleRoot` and
|
||
silently invalidate or fabricate a timestamp.
|
||
|
||
OTS verification only needs **one piece of chain data** per attestation:
|
||
the 80-byte block header at a given height. From that header we use:
|
||
|
||
- `merkleRoot` — to compare against the digest produced by the proof's ops,
|
||
- `time` — to report as the proven timestamp.
|
||
|
||
Nothing else. No transactions, no UTXOs, no scripts, no witness data,
|
||
no compact filters. The OTS proof file itself carries every other byte of
|
||
witness (calendar tree siblings, the prefix/suffix of the on-chain
|
||
transaction, and the merkle path from that tx up to the block root) inline as
|
||
operands of `APPEND`/`PREPEND` ops.
|
||
|
||
The Bitcoin P2P protocol exposes a first-class `getheaders`/`headers` message
|
||
pair specifically for downloading just headers; Bitcoin Core has used it for
|
||
"headers-first" sync since v0.10 (2014). This plan implements a headers-only
|
||
client that lets Amethyst verify NIP-03 attestations against the actual
|
||
proof-of-work chain instead of a trusted explorer.
|
||
|
||
## 2. Goals & non-goals
|
||
|
||
### Goals
|
||
|
||
- Verify any NIP-03 OTS attestation **without trusting any single party** other
|
||
than Bitcoin proof-of-work (and the app build itself, via a checkpoint).
|
||
- Run on Android phones; budget ≤100 MB disk, ≤1 % battery/day, ≤10 min
|
||
first-run sync on WiFi.
|
||
- Plug into existing `BitcoinExplorer` interface — zero changes to OTS
|
||
verification logic.
|
||
- Work over Tor when the user has Tor enabled.
|
||
- Keep existing HTTP explorers as a fallback for proofs older than the
|
||
pinned checkpoint, or when local sync hasn't completed yet.
|
||
- Live in `quartz/src/commonMain/.../bitcoin/` so `cli/`, `desktopApp/` and
|
||
any future merkle-block / onchain-zap work inherit the same primitives.
|
||
|
||
### Non-goals
|
||
|
||
- We are **not** building a full Bitcoin node. No mempool, no UTXO set, no
|
||
block bodies, no script verification.
|
||
- We are **not** implementing wallet features (BIP-32 keys, BIP-37 bloom,
|
||
BIP-157/158 compact filters). NIP-03 doesn't need them.
|
||
- We are **not** replacing OkHttp explorers entirely — they remain a
|
||
fallback path and the way unsynced proofs get verified.
|
||
- No support for testnet/signet (mainnet only — NIP-03 attestations are
|
||
on mainnet).
|
||
|
||
## 3. Architecture overview
|
||
|
||
```
|
||
┌────────────────────────────────────────────────────────────────────┐
|
||
│ Existing OTS code │
|
||
│ OtsEvent.verify(resolver) ──► OpenTimestamps.verify ──► explorer │
|
||
│ │ │
|
||
│ ▼ │
|
||
│ BitcoinExplorer (interface) │
|
||
└────────────────────────────────────────────────────────────────────┘
|
||
▲
|
||
┌──────────────┼──────────────┐
|
||
│ │ │
|
||
OkHttpBitcoinExplorer │ LocalHeadersBitcoinExplorer ◄── NEW
|
||
(existing, fallback, │ │
|
||
URL from │ ▼
|
||
BitcoinExplorer- │ ┌─────────────────────────┐
|
||
Endpoint) │ │ HeaderStore │
|
||
│ │ (SQLite, androidx- │
|
||
│ │ sqlite + Bundled- │
|
||
│ │ SQLiteDriver) │
|
||
│ └─────────────────────────┘
|
||
│ ▲
|
||
│ │
|
||
│ ┌─────────────────────────┐
|
||
│ │ HeadersSyncEngine │
|
||
│ │ (validates PoW + chain) │
|
||
│ └─────────────────────────┘
|
||
│ ▲
|
||
│ │
|
||
│ ┌─────────────────────────┐
|
||
│ │ PeerPool │
|
||
│ │ (multiple BitcoinPeer) │
|
||
│ └─────────────────────────┘
|
||
│ ▲
|
||
│ │
|
||
│ ┌─────────────────────────┐
|
||
│ │ BitcoinPeer │
|
||
│ │ (TCP socket + framing + │
|
||
│ │ version/getheaders) │
|
||
│ └─────────────────────────┘
|
||
│ ▲
|
||
│ │
|
||
│ ┌─────────────────────────┐
|
||
│ │ PeerDiscovery │
|
||
│ │ (DNS seeds, fixed seeds)│
|
||
│ └─────────────────────────┘
|
||
│
|
||
CompositeBitcoinExplorer
|
||
(try local first; if height
|
||
below pinned checkpoint OR not yet
|
||
reached by sync, fall back to HTTP
|
||
via BitcoinExplorerEndpoint)
|
||
```
|
||
|
||
## 4. Module layout
|
||
|
||
All new code in `quartz/src/commonMain/.../bitcoin/`, sibling of
|
||
`nip03Timestamp/` and `nipBCOnchainZaps/`. Protocol code is
|
||
platform-independent; socket I/O is the only `expect`/`actual` boundary.
|
||
Storage uses `androidx.sqlite` which is already KMP — no platform split.
|
||
|
||
```
|
||
quartz/src/commonMain/kotlin/com/vitorpamplona/quartz/bitcoin/
|
||
├── proto/
|
||
│ ├── BitcoinNetMagic.kt # 0xf9beb4d9 mainnet
|
||
│ ├── MessageHeader.kt # 24-byte envelope codec
|
||
│ ├── VarInt.kt # CompactSize encode/decode
|
||
│ ├── NetAddr.kt # net_addr (26B / 30B with timestamp)
|
||
│ ├── ServiceFlags.kt # NODE_NETWORK, NODE_NETWORK_LIMITED, NODE_WITNESS
|
||
│ ├── messages/
|
||
│ │ ├── VersionMessage.kt
|
||
│ │ ├── VerackMessage.kt
|
||
│ │ ├── PingMessage.kt
|
||
│ │ ├── PongMessage.kt
|
||
│ │ ├── GetHeadersMessage.kt # locator + stop hash
|
||
│ │ ├── HeadersMessage.kt # up to 2,000 × (80B header + varint=0)
|
||
│ │ └── SendHeadersMessage.kt # BIP-130
|
||
│ └── BitcoinMessage.kt # sealed interface
|
||
├── header/
|
||
│ ├── BlockHeader80.kt # 80-byte parsed header (ver, prev, merkle, time, bits, nonce)
|
||
│ ├── BlockHash.kt # @JvmInline value class wrapping ByteArray(32)
|
||
│ ├── HeaderHasher.kt # dsha256 over 80B → blockHash
|
||
│ ├── DifficultyTarget.kt # bits ↔ target (256-bit) compact form
|
||
│ ├── ChainWork.kt # cumulative work accumulator (uint256 BE)
|
||
│ └── PowValidator.kt # checks hash ≤ target
|
||
├── consensus/
|
||
│ ├── RetargetCalculator.kt # every 2016 blocks, BIP-? rules
|
||
│ ├── MedianTimePast.kt # last-11 median, for time sanity
|
||
│ └── HeaderValidator.kt # composes the above + linkage check
|
||
├── store/
|
||
│ ├── HeaderStore.kt # public API: append/getByHeight/getByHash/tip
|
||
│ ├── SqliteHeaderStore.kt # androidx.sqlite impl (commonMain — driver-agnostic)
|
||
│ ├── HeaderStoreSchema.kt # IModule for schema + migrations (see §9)
|
||
│ ├── Checkpoint.kt # (height, blockHash, chainWork, time)
|
||
│ └── PinnedCheckpoint.kt # hardcoded release-time checkpoint constant
|
||
├── sync/
|
||
│ ├── LocatorBuilder.kt # exponential locator
|
||
│ ├── HeadersSyncEngine.kt # drives getheaders loop, applies validator, persists
|
||
│ └── ReorgHandler.kt # rare; switches to higher-chainwork tip
|
||
├── peer/
|
||
│ ├── PeerSocket.kt # expect (open / send / recv / close)
|
||
│ ├── PeerPool.kt # holds N BitcoinPeer connections
|
||
│ ├── BitcoinPeer.kt # high-level: handshake, send msg, receive msg
|
||
│ ├── PeerScoring.kt # ban misbehaving peers
|
||
│ └── PeerDiscovery.kt # expect — DNS seeds + fixed seeds
|
||
└── LocalHeadersBitcoinExplorer.kt # implements nip03Timestamp.ots.BitcoinExplorer
|
||
# (with small LRU on top, like OtsBlockHeightCache)
|
||
|
||
quartz/src/jvmAndroid/kotlin/com/vitorpamplona/quartz/bitcoin/
|
||
└── peer/
|
||
├── TcpPeerSocket.jvmAndroid.kt # actual TCP socket (java.net.Socket / SOCKS)
|
||
└── DnsSeedDiscovery.jvmAndroid.kt # InetAddress.getAllByName on seed hostnames
|
||
```
|
||
|
||
Two expect/actual seams only: `PeerSocket` (TCP plus optional SOCKS5 for Tor)
|
||
and `PeerDiscovery` (DNS-seed resolution). Header validation, message codecs,
|
||
store, sync engine, peer pool — all compile in `commonMain`, all testable
|
||
cross-platform without a single platform shim. Storage compiles in
|
||
`commonMain` too because `androidx.sqlite` + `BundledSQLiteDriver` are KMP.
|
||
|
||
### Wiring on Android
|
||
|
||
```
|
||
amethyst/src/main/java/com/vitorpamplona/amethyst/model/nip03Timestamp/
|
||
├── HeaderSyncWorker.kt # WorkManager job (CONNECTED + optional UNMETERED)
|
||
├── HeaderSyncForegroundService.kt # FGS for first-run / large catch-up
|
||
├── LocalHeadersOtsResolverBuilder.kt # plugs LocalHeadersBitcoinExplorer in
|
||
└── (modify) TorAwareOkHttpOtsResolverBuilder.kt → CompositeOtsResolverBuilder
|
||
```
|
||
|
||
`AppModules.kt` switches from `TorAwareOkHttpOtsResolverBuilder` to a
|
||
composite that prefers the local explorer and falls back to HTTP only when
|
||
local is missing the height. The HTTP path resolves its base URL via the
|
||
existing `BitcoinExplorerEndpoint` (shared with the onchain-zap
|
||
`EsploraBackend`), so a user-configured custom Esplora is honoured on both
|
||
the OTS and the zap side.
|
||
|
||
`AppModules.kt:433` already passes `rootFilesDir = { appContext.filesDir }`
|
||
into Quartz; the headers SQLite file goes under
|
||
`$filesDir/bitcoin/headers.db` via a new entry in that wiring.
|
||
|
||
On desktop and CLI, the equivalent data-dir paths are already plumbed for
|
||
`SQLiteEventStore` — same hook reused here.
|
||
|
||
## 5. Bitcoin P2P protocol — minimum we implement
|
||
|
||
We implement a strict subset. Anything not in this list is unimplemented and
|
||
silently dropped on receive.
|
||
|
||
### Outbound messages we send
|
||
|
||
| Command | When |
|
||
|---------------|------|
|
||
| `version` | Right after TCP connect |
|
||
| `verack` | After receiving peer's `version` |
|
||
| `ping` | Every 60 s if idle, to keep connection alive |
|
||
| `pong` | In response to peer's `ping` |
|
||
| `getheaders` | The actual workhorse: locator + zero stop hash |
|
||
| `sendheaders` | (optional) tell peer we want unsolicited new tips via `headers` |
|
||
|
||
### Inbound messages we handle
|
||
|
||
| Command | Action |
|
||
|-----------|--------|
|
||
| `version` | Validate services flag, store user-agent + start_height; reply `verack` |
|
||
| `verack` | Mark handshake complete |
|
||
| `ping` | Reply `pong` with same nonce |
|
||
| `pong` | Update RTT estimate |
|
||
| `headers` | Hand off to `HeadersSyncEngine` |
|
||
| `addr`/`addrv2` | (later) feed into peer pool for fallback discovery |
|
||
| anything else | ignore, do not disconnect |
|
||
|
||
### `version` payload we send
|
||
|
||
- `version`: 70016 (supports `wtxidrelay`, but we don't use it)
|
||
- `services`: 0 (we serve nothing — pure leech)
|
||
- `timestamp`: current unix time
|
||
- `addr_recv`/`addr_from`: zeroed
|
||
- `nonce`: random 64-bit
|
||
- `user_agent`: `/Amethyst-Quartz:<quartz-version>/`
|
||
- `start_height`: 0
|
||
- `relay`: false (we don't want unsolicited tx invs)
|
||
|
||
### `getheaders` request rules
|
||
|
||
- Block locator built with exponential back-off:
|
||
`[tip, tip-1, tip-2, tip-4, tip-8, ..., tip-2^k, ..., genesisHash]`
|
||
~33 entries even at height 1 M.
|
||
- `hash_stop = 0x00..00` → "give me as many as you have, up to 2000".
|
||
- Pipeline tip-extension: after a `headers` of length 2000 arrives, immediately
|
||
fire the next `getheaders` with the new tip. (This is the standard
|
||
optimization Core itself uses.)
|
||
|
||
### Magic bytes & ports
|
||
|
||
- Mainnet magic: `0xf9 0xbe 0xb4 0xd9`
|
||
- Default port: 8333 (allow override for users running their own node on a
|
||
non-standard port)
|
||
|
||
## 6. Validation rules — what makes a header chain "valid"
|
||
|
||
Per-header (`HeaderValidator`):
|
||
|
||
1. **Hash ≤ target.** Compute `dsha256(header80)` (interpreted as little-endian
|
||
256-bit int) and compare to `compactToTarget(bits)`.
|
||
2. **Linkage.** `header.prevHash == storedTip.hash`.
|
||
3. **MTP sanity.** `header.time > medianOf(last 11 headers' time)`. (`time` is
|
||
allowed to be up to 2 h ahead of *our* clock, but we don't enforce that on
|
||
sync — we'd reject reorgs where we're skewed; just log.)
|
||
4. **Difficulty retarget at multiples of 2016.** Reproduce Core's
|
||
`CalculateNextWorkRequired`:
|
||
- `actualTimespan = clamp(prev2016.time - first2016.time, target/4, target*4)`
|
||
- `newTarget = oldTarget * actualTimespan / (14 * 24 * 3600)`
|
||
- Cap at `pow_limit` (max target).
|
||
5. **Version**: not validated — soft-fork bits don't matter for headers-only.
|
||
6. **Cumulative chainwork** updated as `prevWork + (2^256 / (target+1))`.
|
||
|
||
### Chain selection
|
||
|
||
- We track the **highest-cumulative-chainwork** chain we've seen, not the
|
||
longest. Standard Bitcoin rule.
|
||
- On receiving a fork: keep both branches in memory while their tips are
|
||
within `MAX_REORG_DEPTH = 100` of each other; once one accumulates clearly
|
||
more work, drop the loser.
|
||
- A reorg deeper than 100 blocks is treated as adversarial and refused
|
||
(operator must bump the checkpoint).
|
||
|
||
### Hardcoded sanity checkpoints
|
||
|
||
In `Checkpoints.kt` we ship a small list of (height, expectedBlockHash) pairs
|
||
mirroring Bitcoin Core's own historical checkpoints, plus one **release-pinned
|
||
checkpoint** updated at app build time near the current tip. Sync must agree
|
||
with all checkpoints — disagreement aborts and bans the peer.
|
||
|
||
## 7. Bundled data — none. Just a pinned checkpoint constant.
|
||
|
||
The point of this work is "verify against the Bitcoin network, not a trusted
|
||
party," so we ship no headers in the APK. Instead:
|
||
|
||
- **One hardcoded `PinnedCheckpoint` constant** in `bitcoin/store/PinnedCheckpoint.kt`:
|
||
`(height, blockHash, chainWork, time)`. ~80 bytes of source.
|
||
- The checkpoint is pinned ~1 month before each release and bumped via a
|
||
one-line PR per release. This is the **only** thing trusted at build time
|
||
— and it's a single block hash any user can independently verify against
|
||
any Bitcoin node before installing.
|
||
- **First-run sync** starts at the checkpoint and pulls forward to the
|
||
current tip over P2P. At a ~1-month checkpoint that's ~4,300 headers
|
||
(~340 KB on the wire, ~15 minutes' worth of `getheaders` round-trips at
|
||
worst), seconds over WiFi.
|
||
- **Pre-checkpoint heights** are not synced locally at all. OTS proofs
|
||
whose height precedes the checkpoint fall through to
|
||
`OkHttpBitcoinExplorer` via the composite. Users in strict-mode
|
||
(`localHeadersTrustOnly = true`) get a clear
|
||
"this OTS proof predates the bundled checkpoint" error and no network call.
|
||
- **Catch-up after long absence.** A device that hasn't run the app for
|
||
months will, on next sync, fast-forward from `tip_local` (its stored
|
||
best_height) to `tip_network`. Each `headers` message brings 2,000
|
||
headers; even 6 months of absence is ~26,000 headers (~2 MB) in 13 round
|
||
trips.
|
||
- **Header growth on disk** is ~4 MB raw per year (+ SQLite overhead, see §9).
|
||
|
||
APK growth: **0 bytes**. Trust delta vs the old HTTP-explorer model: one
|
||
block hash, code-reviewable in git.
|
||
|
||
## 8. Multi-peer strategy & eclipse mitigation
|
||
|
||
A single peer can lie. We protect against that with:
|
||
|
||
1. **Peer count.** Maintain ≥4 simultaneous peer connections during sync;
|
||
accept a header batch only after at least 2 distinct peers (different
|
||
/16 IPv4 prefixes, different ASNs where we can tell) have served the same
|
||
batch.
|
||
2. **Chainwork comparison.** After convergence we should land on the
|
||
chain with highest cumulative work. If peers disagree, prefer the
|
||
higher-work chain.
|
||
3. **Checkpoint cross-check.** Bundled checkpoints (above) detect any peer
|
||
that disagrees with the consensus history. Disagreement → ban that IP for
|
||
the session.
|
||
4. **DNS seed diversity.** Rotate across all 9 of Bitcoin Core's DNS seeds,
|
||
not a single one.
|
||
5. **Hardcoded fallback IPs.** Ship the `chainparams.cpp` fallback IP list
|
||
(about 1,000 v4 + v6 entries) so first-run works even if every DNS seed is
|
||
blocked or poisoned.
|
||
6. **Optional user override.** A user can configure their own trusted node
|
||
(`bitcoin.example.com:8333`) — same path as the existing custom explorer
|
||
URL setting in `OtsSettings.kt`.
|
||
|
||
## 9. Storage — SQLite via `androidx.sqlite` + `BundledSQLiteDriver`
|
||
|
||
We reuse the same storage primitive Quartz already uses for `SQLiteEventStore`
|
||
(`quartz/.../nip01Core/store/sqlite/`): `androidx.sqlite` with the bundled
|
||
driver. This is KMP, ships in `androidx-sqlite-bundled` / `…-bundled-jvm`
|
||
(already a Quartz dependency), and is the project's standard storage choice.
|
||
|
||
The complete `SQLiteEventStore` infrastructure — `BundledSQLiteDriver`,
|
||
`SQLiteConnectionPool`, `transaction { }`, `IModule` for migrations,
|
||
`QueryBuilder` helpers — is reused by `SqliteHeaderStore` and
|
||
`HeaderStoreSchema`.
|
||
|
||
### Schema
|
||
|
||
```sql
|
||
PRAGMA journal_mode = WAL;
|
||
PRAGMA synchronous = NORMAL;
|
||
PRAGMA foreign_keys = OFF;
|
||
|
||
CREATE TABLE block_header (
|
||
height INTEGER PRIMARY KEY, -- 0..tip, dense
|
||
hash BLOB NOT NULL, -- 32 B, dsha256(header), big-endian display order
|
||
header BLOB NOT NULL, -- 80 B raw header
|
||
chainwork BLOB NOT NULL -- 32 B cumulative work, big-endian
|
||
);
|
||
|
||
CREATE UNIQUE INDEX idx_block_header_hash ON block_header(hash);
|
||
|
||
CREATE TABLE chain_tip (
|
||
id INTEGER PRIMARY KEY CHECK (id = 1), -- singleton row
|
||
best_height INTEGER NOT NULL,
|
||
best_hash BLOB NOT NULL,
|
||
best_work BLOB NOT NULL
|
||
);
|
||
|
||
CREATE TABLE checkpoint_seen (
|
||
height INTEGER PRIMARY KEY, -- which checkpoints we've passed and confirmed
|
||
hash BLOB NOT NULL,
|
||
source TEXT NOT NULL -- 'pinned' | 'historical'
|
||
);
|
||
```
|
||
|
||
Migrations live in `HeaderStoreSchema.kt` as an `IModule` (same pattern as
|
||
the event store's modules). v1 is the schema above; future additions
|
||
(prev-hash cache column, merkle-block annotations) ALTER from there.
|
||
|
||
### Sizing
|
||
|
||
| Item | Per record | At height 1,000,000 |
|
||
|---|---|---|
|
||
| `block_header` row (80+32+32 B payload + SQLite overhead) | ~180–220 B | ~180–220 MB |
|
||
| `idx_block_header_hash` B-tree (32 B key + rowid) | ~40 B | ~40 MB |
|
||
| `chain_tip` | 1 row | negligible |
|
||
| **Total** | | **~220–260 MB at height 1 M** |
|
||
|
||
In practice users will start from a recent checkpoint and grow ~4 MB raw
|
||
(~10 MB stored) per year, so most installs sit at 10–40 MB. The "1 M
|
||
height" line is the long-tail worst case if someone with a multi-year-old
|
||
device's local store retroactively backfills (which we explicitly don't do
|
||
in v1 — see §15 Q1).
|
||
|
||
### Atomicity & crash safety
|
||
|
||
- `HeadersSyncEngine` writes each `headers` batch (up to 2,000) inside a
|
||
single `transaction { }`. WAL gives us atomic-or-nothing per batch.
|
||
- `chain_tip` is updated in the same transaction as the headers it points to,
|
||
so the tip can never reference rows that didn't commit.
|
||
- A crash mid-batch leaves the WAL inconsistent; SQLite rolls it back on
|
||
next open. No custom recovery code.
|
||
- On open, `SqliteHeaderStore.verifyTipIntegrity()` re-reads the tip row and
|
||
confirms the referenced header exists. If not (unexpected DB corruption,
|
||
ransomware-like sandbox damage), we delete the db file and fall back to
|
||
re-sync from the pinned checkpoint.
|
||
|
||
### Concurrency
|
||
|
||
Same model as `SQLiteEventStore`: one connection pool, all writes go through
|
||
the sync engine on a dedicated dispatcher; reads use the pool's read slots.
|
||
OTS verifier reads are short and indexed — no contention with sync writes.
|
||
|
||
## 10. Resolving a height query (the actual `BitcoinExplorer` impl)
|
||
|
||
```kotlin
|
||
class LocalHeadersBitcoinExplorer(
|
||
private val store: HeaderStore,
|
||
private val sync: HeadersSyncEngine,
|
||
private val fallback: BitcoinExplorer? = null, // OkHttpBitcoinExplorer
|
||
) : BitcoinExplorer {
|
||
override suspend fun blockHash(height: Int): String {
|
||
store.getByHeight(height)?.let { return it.blockHash.toHex() }
|
||
// height not yet synced; ask sync engine to extend, with timeout
|
||
sync.ensureHeight(height, timeoutMs = 10_000)
|
||
store.getByHeight(height)?.let { return it.blockHash.toHex() }
|
||
return fallback?.blockHash(height) ?: throw NotSyncedException(height)
|
||
}
|
||
|
||
override suspend fun block(hash: String): BlockHeader {
|
||
store.getByHash(hash)?.let { rec ->
|
||
return BlockHeader(
|
||
merkleRoot = rec.header.merkleRoot.toHex(),
|
||
blockHash = hash,
|
||
time = rec.header.time.toString(),
|
||
)
|
||
}
|
||
return fallback?.block(hash) ?: throw NotSyncedException(hash)
|
||
}
|
||
}
|
||
```
|
||
|
||
The fallback path is gated by a setting; a privacy-conscious user can
|
||
disable it ("local-only verification").
|
||
|
||
## 11. Mobile (Android) integration
|
||
|
||
### Lifecycle
|
||
|
||
- **First launch after install/upgrade.** Show a one-time "Verify Bitcoin
|
||
timestamps locally?" prompt. If accepted, kick off a foreground-service sync
|
||
(FGS so OS doesn't kill us mid-download). Default WiFi-only; user can
|
||
override.
|
||
- **Subsequent launches.** Background `WorkManager` job, periodic, ~6 h
|
||
cadence. Constraints: `NetworkType.UNMETERED` by default, `BatteryNotLow`.
|
||
- **On-demand.** When a NIP-03 attestation needs a height we don't yet have,
|
||
request a targeted sync up to that height (capped 10 s, otherwise
|
||
fall back to HTTP).
|
||
|
||
### Permissions
|
||
|
||
- No new runtime permissions needed. We use the foreground-service permission
|
||
declared for nests audio (already present) plus `INTERNET` (already
|
||
granted). Add `FOREGROUND_SERVICE_DATA_SYNC` to manifest if not present.
|
||
|
||
### Settings UI
|
||
|
||
Extend `OtsSettings.kt`:
|
||
|
||
```kotlin
|
||
data class OtsSettings(
|
||
val customExplorerUrl: String? = null, // existing — shared via BitcoinExplorerEndpoint
|
||
// NEW:
|
||
val localHeadersEnabled: Boolean = true,
|
||
val localHeadersTrustOnly: Boolean = false, // disables HTTP fallback; pre-checkpoint proofs error
|
||
val localHeadersWifiOnly: Boolean = true,
|
||
val customP2pNode: String? = null, // e.g. "myriad.example.com:8333"
|
||
)
|
||
```
|
||
|
||
A short "Bitcoin verification" settings screen lets the user pick:
|
||
- **Local headers + HTTP fallback** (default) — best UX, trustless for
|
||
post-checkpoint heights, HTTP for older proofs.
|
||
- **Local headers only (strict)** — refuses to verify proofs older than the
|
||
pinned checkpoint. Most paranoid setting.
|
||
- **HTTP explorer only** — pre-local-headers behaviour. Lightest disk/CPU.
|
||
|
||
The `customExplorerUrl` field continues to feed `BitcoinExplorerEndpoint`,
|
||
so a custom HTTP Esplora is honoured both by the fallback in this feature
|
||
and by the onchain-zaps `EsploraBackend`. No new Esplora setting.
|
||
|
||
## 12. Tor integration
|
||
|
||
The codebase already has a Tor-aware HTTP path (`TorAwareOkHttpOtsResolverBuilder`).
|
||
For P2P:
|
||
|
||
- Reuse the project's existing SOCKS proxy plumbing.
|
||
- When Tor is on, route TCP socket connections through SOCKS5 to
|
||
Tor's `127.0.0.1:9050`.
|
||
- Prefer `.onion` peer addresses (advertised via `addrv2`) when available —
|
||
they don't expose the user's clearnet IP to the peer.
|
||
- Increase per-peer timeout (Tor RTT is ~1–5 s).
|
||
|
||
DNS seed lookup over Tor must use SOCKS5 hostname resolution
|
||
(no plaintext DNS).
|
||
|
||
## 13. Phased delivery
|
||
|
||
Each phase produces a shippable artifact with measurable behavior.
|
||
|
||
### Phase 0 — Foundation (1–2 days)
|
||
|
||
- Create `quartz/plans/2026-05-08-local-headers-explorer.md` (this file). ✓
|
||
- Add `quartz/src/commonMain/.../bitcoin/` skeleton.
|
||
- Wire empty `LocalHeadersBitcoinExplorer` that always defers to fallback.
|
||
- No behavior change yet.
|
||
|
||
### Phase 1 — Header parsing & validation, no I/O (3–5 days)
|
||
|
||
- `BlockHeader80` codec + tests against known mainnet headers (genesis, a few
|
||
retargets, post-segwit).
|
||
- `DifficultyTarget` compact↔target with RFC test vectors.
|
||
- `PowValidator`, `RetargetCalculator`, `MedianTimePast`, `HeaderValidator`.
|
||
- Tests run on a local fixture file containing the first 50 000 headers.
|
||
|
||
### Phase 2 — Storage layer (2–3 days)
|
||
|
||
- `HeaderStore` interface + `SqliteHeaderStore` implementation in `commonMain`
|
||
(`androidx.sqlite` is KMP, no platform shim needed).
|
||
- `HeaderStoreSchema` as an `IModule` matching the `SQLiteEventStore`
|
||
migration pattern.
|
||
- `append(batch)`, `getByHeight(h)`, `getByHash(hash)`, `tip()`.
|
||
- Reuse `BundledSQLiteDriver`, `SQLiteConnectionPool`, `transaction { }`
|
||
from `nip01Core/store/sqlite/` — no duplication.
|
||
- Crash-safety test: kill mid-transaction, reopen, confirm WAL rollback;
|
||
confirm `verifyTipIntegrity()` recovers cleanly.
|
||
|
||
### Phase 3 — P2P codec, no socket (3–5 days)
|
||
|
||
- Message envelope (magic, command, length, checksum) encode/decode.
|
||
- All seven message payload codecs.
|
||
- Tests using captured wire bytes from a real Bitcoin Core peer (record once,
|
||
replay forever).
|
||
|
||
### Phase 4 — Single-peer sync (5–7 days)
|
||
|
||
- `BitcoinPeer` over a real socket.
|
||
- `HeadersSyncEngine` drives `getheaders` loop against one peer.
|
||
- Validate every header through Phase 1 validator before persisting.
|
||
- Integration test: sync first 100 K headers from a public Bitcoin Core peer
|
||
in CI. Skip in normal `./gradlew test`; only run nightly.
|
||
|
||
### Phase 5 — Multi-peer & eclipse hardening (3–5 days)
|
||
|
||
- `PeerPool`, `PeerScoring`.
|
||
- DNS seed discovery + fallback IP list.
|
||
- Header-batch cross-validation across peers.
|
||
- Hardcoded checkpoints in `Checkpoints.kt`.
|
||
|
||
### Phase 6 — Pinned checkpoint constant (½ day)
|
||
|
||
- `bitcoin/store/PinnedCheckpoint.kt` — single hardcoded
|
||
`Checkpoint(height, blockHash, chainWork, time)`. ~80 bytes of source.
|
||
- A small Gradle task (`./gradlew bumpBitcoinCheckpoint`) that takes
|
||
`(height, hash)` arguments and writes the constant. Run by the release
|
||
process; otherwise no automation.
|
||
- No backward-sync, no bundled headers blob, no asset resources.
|
||
Pre-checkpoint heights are routed to the HTTP fallback by the composite
|
||
resolver (or refused in strict mode).
|
||
|
||
### Phase 7 — Android wiring (3–4 days)
|
||
|
||
- `HeaderSyncWorker` + `HeaderSyncForegroundService`.
|
||
- New "Bitcoin verification" settings screen.
|
||
- Switch `AppModules.kt` to composite resolver builder.
|
||
- Manual QA: install fresh, open NIP-03 note, watch sync run, confirm
|
||
verification matches HTTP explorer.
|
||
|
||
### Phase 8 — Tor (2–3 days)
|
||
|
||
- SOCKS5-tunnelled TCP socket variant.
|
||
- `addrv2` decoding for `.onion` addresses.
|
||
- QA on Orbot.
|
||
|
||
### Phase 9 — Desktop & CLI integration (1–2 days)
|
||
|
||
- `desktopApp/` inherits the explorer for free (everything is in
|
||
`quartz/commonMain/`). Confirm headers.db path is plumbed via the same
|
||
data-dir hook already used by `SQLiteEventStore` on JVM. Confirm sync
|
||
runs end-to-end on the desktop app.
|
||
- `cli/` — add an `amy verify-ots <event-json>` command if not present,
|
||
wired through the same composite resolver. The CLI is a thin assembly
|
||
layer over Quartz (per `amy-expert`), so the actual sync logic is reused
|
||
as-is.
|
||
|
||
### Phase 10 — Rollout & telemetry (ongoing)
|
||
|
||
- Ship as opt-in for one release (default off, advertised in release notes).
|
||
- Collect (locally, no analytics): time-to-first-sync, time-to-verify,
|
||
fallback rate. Surface in a debug screen.
|
||
- Default-on in the next release if metrics look healthy.
|
||
|
||
Total: **~22–32 engineer-days** end to end. Phases 1–4 are the bulk
|
||
(protocol + validation); phases 5–10 are integration. Phase 6 dropped from
|
||
2–3 d to ½ d because we ship no bundled data — just one constant.
|
||
|
||
## 14. Testing strategy
|
||
|
||
### Unit tests (commonMain)
|
||
|
||
- Header codec round-trip (10 known headers across history).
|
||
- `dsha256` against test vectors.
|
||
- Compact-bits encoding edge cases (0x1d00ffff, 0x1b0404cb, etc.).
|
||
- Retarget at heights 2016, 4032, 32256 (well-known).
|
||
- MTP sanity for a fixture chain.
|
||
- Locator-builder shape (geometric back-off, includes genesis).
|
||
- Reorg handler: synthetic 5-block fork with higher work wins.
|
||
|
||
### Integration tests (jvmAndroid)
|
||
|
||
- Replay-based P2P codec test using a captured pcap of a real session
|
||
(committed as `quartz/src/jvmAndroid/test/resources/btc-handshake.bin`).
|
||
- Local-network test against a Bitcoin Core regtest node spun up in CI
|
||
(Docker). Validates full handshake + `getheaders` loop end to end.
|
||
- `SqliteHeaderStore` round-trip on a real on-disk file (jvmTest, in a
|
||
`tmpdir`): append 5,000 headers in batches, getByHeight, getByHash,
|
||
reopen, confirm tip survives.
|
||
- Crash-and-resume test: kill the writer in the middle of a `transaction { }`,
|
||
reopen, confirm SQLite WAL rolls the partial batch back and sync resumes
|
||
from the last committed tip.
|
||
|
||
### Property tests
|
||
|
||
- `HeaderValidator` — fuzz with random 80-byte inputs; confirm invalid PoW
|
||
always rejected and valid PoW always accepted.
|
||
|
||
### Manual QA checklist
|
||
|
||
- Verify a known NIP-03 attestation against the local explorer matches
|
||
Blockstream's answer.
|
||
- Disable network mid-sync; resume cleanly when network returns.
|
||
- Tor on — sync completes (slowly) over Orbot.
|
||
- Disk-full simulation — fail gracefully, don't corrupt store.
|
||
|
||
## 15. Risks & open questions
|
||
|
||
| # | Risk | Mitigation |
|
||
|---|------|------------|
|
||
| R1 | First-run UX cost (sync from pinned checkpoint) | Pinned checkpoint ~1 month before release means ~4 K headers, ~340 KB, seconds over WiFi. Show progress in settings; sync runs in background WorkManager job anyway. |
|
||
| R2 | Mobile OS kills socket mid-sync | Foreground service for first run; resumable design (SQLite WAL handles partial batches automatically). |
|
||
| R3 | Peer feeds adversarial headers (eclipse) | Multi-peer + checkpoints + chainwork comparison. |
|
||
| R4 | Disk corruption | SQLite WAL + `verifyTipIntegrity()` on open; on detected corruption, delete db file and re-sync from pinned checkpoint. |
|
||
| R5 | Tor + P2P is slow | Increased timeouts; fallback to HTTP-over-Tor explorer (via `BitcoinExplorerEndpoint`) when sync stalls. |
|
||
| R6 | APK size growth | Zero — no bundled data ships in the APK. |
|
||
| R7 | Privacy: peers see user's IP | Tor mode supported; `version` user-agent doesn't leak Amethyst install ID. |
|
||
| R8 | Battery cost on background sync | Default to UNMETERED + BATTERY_OK constraints; back off when idle. |
|
||
| R9 | Maintenance burden — new soft-forks | Soft-forks don't affect headers-only validation (no script eval). Difficulty algorithm hasn't changed; a future change would require a code update. |
|
||
| R10 | Test flakiness if CI talks to public Bitcoin nodes | Use Bitcoin Core regtest in Docker for integration tests; public nodes only nightly. |
|
||
| R11 | SQLite store size on long-running installs | ~10 MB/year typical; ~220 MB at height 1 M which won't happen in any realistic mobile lifetime. Add a "wipe local headers" button in settings for users who want it. |
|
||
|
||
### Open questions
|
||
|
||
#### Resolved 2026-05-19
|
||
|
||
- **Q1.** *Resolved — HTTP fallback only.* Pre-checkpoint OTS proofs are
|
||
routed to `OkHttpBitcoinExplorer` via the composite resolver. Strict-mode
|
||
users (`localHeadersTrustOnly = true`) get a clear error. No backward
|
||
sync code path.
|
||
- **Q2.** *Resolved — cache chainwork.* SQLite makes the choice cheap (one
|
||
extra BLOB column). Reading current tip work without a full scan matters
|
||
for chain-selection logic; the ~30 MB-at-1M cost is negligible on disk.
|
||
- **Q4.** *Resolved — no HTTPS headers blob.* The point of this feature is
|
||
trustless verification, and the pinned-checkpoint approach already gives
|
||
fast first-run. Adding a second ingestion path (signed blob over HTTPS)
|
||
doubles the test surface for marginal benefit.
|
||
|
||
#### Still open
|
||
|
||
- **Q3.** Should `LocalHeadersBitcoinExplorer` (and friends) be a Quartz
|
||
public API for 3rd-party Quartz consumers, or internal to Amethyst?
|
||
Recommendation: public — it's a clean reusable primitive. Decide before
|
||
Phase 0 lands so the package layout doesn't churn. *(See `quartz-integration`
|
||
skill for the public-API contract Quartz exposes today.)*
|
||
|
||
## 16. Migration & rollout
|
||
|
||
1. **Ship behind a feature flag**, default off, documented in release notes.
|
||
2. **Internal QA** for one full release cycle on the flag.
|
||
3. **Default on** for new installs the release after, leaving existing
|
||
installs on their previous setting.
|
||
4. **Default on** for everyone in the release after that.
|
||
5. **Keep HTTP explorer code path** for at least 4 releases as a safety net.
|
||
|
||
## 17. Success criteria
|
||
|
||
- A NIP-03 attestation that verifies via Blockstream also verifies via the
|
||
local explorer (bit-for-bit identical timestamp).
|
||
- 95th-percentile cold-launch verification of a recent attestation: ≤2 s.
|
||
- 95th-percentile first-run sync (from pinned checkpoint to tip): ≤30 s on
|
||
WiFi for a checkpoint ≤2 months old.
|
||
- No new crash-rate regression in the verification path.
|
||
- App size growth: **0 MB** (no bundled data).
|
||
- A user with HTTP explorer disabled can still verify any post-checkpoint
|
||
attestation.
|
||
|
||
## 18. Out of scope (explicit)
|
||
|
||
- BIP-157/158 compact filters
|
||
- BIP-37 merkleblock or full-block fetch
|
||
- **Trustless verification of NIP-BC onchain zaps** — needs merkleblock or
|
||
full-block fetch on top of this stack. See §20 for the follow-up plan.
|
||
- **Trustless wallet operations** for NIP-BC onchain zap *sending* —
|
||
needs UTXO-by-address and broadcast paths that headers can't provide.
|
||
Stays on user-configured Esplora.
|
||
- Lightning, BIP-32 wallet, mempool monitoring
|
||
- Verifying NIP-03 attestations against altcoin/Liquid timestamps (not in
|
||
the spec)
|
||
- Replacing the calendar-server upgrade path (`OtsState.upgrade()`) —
|
||
unrelated, those are simple HTTPS calls and stay as-is
|
||
|
||
## 19. Follow-up: trustless NIP-BC onchain zap verification
|
||
|
||
*Not in this plan. Tracked here so the design choices above stay aligned.*
|
||
|
||
The shipped `OnchainZapVerifier` (`quartz/.../nipBCOnchainZaps/verify/`)
|
||
trusts `OnchainBackend.getTx(txid)` — i.e. Esplora — for both
|
||
tx-existence-in-a-block and the output values used to compute
|
||
`verifiedSats`. A malicious Esplora can fabricate or suppress zaps.
|
||
|
||
Closing that gap is a **separate plan** that builds on this one:
|
||
|
||
- **Foundation reused:** `BitcoinPeer`, `PeerPool`, `HeaderStore`,
|
||
`HeaderValidator`, `LocalHeadersBitcoinExplorer`.
|
||
- **New work (rough ~10–15 days on top of S1):** BIP-37 `filterload` /
|
||
`merkleblock` / matched-`tx` handling, or `getdata MSG_BLOCK` + stream
|
||
parse; partial-merkle-tree verification; new `OnchainBackend` variant
|
||
that asks the local stack for the inclusion proof and the matched tx
|
||
while keeping Esplora for UTXO listing / broadcast / fee estimates on
|
||
the *send* side.
|
||
- **Net win:** zap-receive verification stops trusting Esplora. Zap-send
|
||
still needs an address-indexed server (no headers-only equivalent
|
||
exists), so that part of the trust story is unchanged.
|
||
- **Filename:** `quartz/plans/YYYY-MM-DD-onchain-zap-merkle-proofs.md`,
|
||
written once S1 has shipped and we have real numbers on first-run sync,
|
||
store size, and battery cost.
|
||
|
||
## 20. References
|
||
|
||
- NIP-03: https://github.com/nostr-protocol/nips/blob/master/03.md
|
||
- OpenTimestamps spec: https://github.com/opentimestamps/python-opentimestamps
|
||
- Bitcoin Core P2P protocol: `src/protocol.h`, `src/net_processing.cpp`,
|
||
`src/headerssync.cpp`
|
||
- BIP-130 (`sendheaders`): https://github.com/bitcoin/bips/blob/master/bip-0130.mediawiki
|
||
- Headers-first sync rationale: Bitcoin Core 0.10 release notes (2015)
|
||
- Existing OTS surface in this repo:
|
||
- `quartz/src/commonMain/kotlin/com/vitorpamplona/quartz/nip03Timestamp/ots/BitcoinExplorer.kt`
|
||
- `quartz/src/commonMain/kotlin/com/vitorpamplona/quartz/nip03Timestamp/ots/BlockHeader.kt`
|
||
- `quartz/src/commonMain/kotlin/com/vitorpamplona/quartz/nip03Timestamp/ots/attestation/BitcoinBlockHeaderAttestation.kt`
|
||
- `quartz/src/commonMain/kotlin/com/vitorpamplona/quartz/nip03Timestamp/ots/OtsBlockHeightCache.kt`
|
||
- `quartz/src/jvmAndroid/kotlin/com/vitorpamplona/quartz/nip03Timestamp/okhttp/OkHttpBitcoinExplorer.kt`
|
||
- `amethyst/src/main/java/com/vitorpamplona/amethyst/model/nip03Timestamp/TorAwareOkHttpOtsResolverBuilder.kt`
|
||
- `amethyst/src/main/java/com/vitorpamplona/amethyst/model/nip03Timestamp/OtsSettings.kt`
|
||
- `amethyst/src/main/java/com/vitorpamplona/amethyst/model/nip03Timestamp/BitcoinExplorerEndpoint.kt`
|
||
- Storage precedent (reused as-is):
|
||
- `quartz/src/commonMain/kotlin/com/vitorpamplona/quartz/nip01Core/store/sqlite/SQLiteEventStore.kt`
|
||
- `quartz/src/commonMain/kotlin/com/vitorpamplona/quartz/nip01Core/store/sqlite/IModule.kt`
|
||
- `quartz/src/commonMain/kotlin/com/vitorpamplona/quartz/nip01Core/store/sqlite/SQLiteConnectionExt.kt`
|
||
- `quartz/src/commonMain/kotlin/com/vitorpamplona/quartz/nip01Core/store/sqlite/SQLiteConnectionPool.kt`
|
||
- Follow-up area:
|
||
- `quartz/src/commonMain/kotlin/com/vitorpamplona/quartz/nipBCOnchainZaps/verify/OnchainZapVerifier.kt`
|
||
- `quartz/src/commonMain/kotlin/com/vitorpamplona/quartz/nipBCOnchainZaps/chain/OnchainBackend.kt`
|