nick/amethyst
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 1 Wiki Activity

docs(quartz): revise local-headers-explorer plan — SQLite, no bundle, OTS-only

Browse Source 
Lock the design choices from the 2026-05-19 review against the intervening
2026-05-14 onchain-zaps work:

- Move the module from quartz/.../nip03Timestamp/bitcoin/ to a sibling
  quartz/.../bitcoin/ package so the headers explorer, header validator,
  peer pool and store can be reused by the future onchain-zap merkle-proof
  work without an inverted import path.
- Use androidx.sqlite + BundledSQLiteDriver for HeaderStore, matching the
  existing SQLiteEventStore. Schema lives in commonMain via IModule. Drops
  the hand-rolled flat-file + sidecar height index.
- Drop the bundled headers blob. Ship a single hardcoded PinnedCheckpoint
  constant; first-run sync starts from the checkpoint and pulls forward
  over P2P. APK growth: 0 bytes.
- Pre-checkpoint OTS heights fall through to OkHttpBitcoinExplorer via
  BitcoinExplorerEndpoint (shared with the onchain-zap EsploraBackend).
  Strict-mode users get an explicit error instead of a network call.
- Mark trustless NIP-BC onchain-zap verification as out of scope and
  capture it as a follow-up plan (BIP-37 merkleblock or full-block fetch
  on top of this stack).

Resolves open questions Q1, Q2 and Q4 from the original plan; Q3 (Quartz
public API vs internal) left open for Phase 0.
This commit is contained in:
Claude
2026-05-19 18:14:30 +00:00
parent ed686b71ed
commit f3637dd7c1
1 changed files with 292 additions and 119 deletions
Download Patch File Download Diff File Expand all files Collapse all files
quartz/plans/2026-05-08-local-headers-explorer.md
+292 -119
View File
@@ -1,10 +1,35 @@
# Local Bitcoin headers explorer for NIP-03 OTS # Local Bitcoin headers explorer for NIP-03 OTS
**Date:** 2026-05-08 **Date:** 2026-05-08 (revised 2026-05-19)
**Branch:** `claude/review-ots-blockchain-deps-bKns7` **Branch:** `claude/review-bitcoin-blockchain-plan-6NGZ5` (originally `claude/review-ots-blockchain-deps-bKns7`)
**Module:** `quartz/` (with thin Android wiring in `amethyst/`) **Module:** `quartz/` (with thin Android wiring in `amethyst/`)
**Status:** Plan — not yet implemented **Status:** Plan — not yet implemented
## Revision log
**2026-05-19 — decisions locked, scope tightened.** After review against the
intervening 2026-05-14 onchain-zaps work:
- **Scope stays OTS-only.** Trustless verification of NIP-BC onchain zaps
(kind 8333) needs more than headers — at minimum BIP-37 merkleblock or
full-block fetch on top of this stack. That becomes a follow-up plan
(see §20).
- **Module moves to `quartz/.../bitcoin/`** — sibling of `nip03Timestamp/`
and `nipBCOnchainZaps/`, not nested under NIP-03. The future merkle-block
phase will reuse `BitcoinPeer` / `HeaderStore` / `HeaderValidator`.
- **Storage is SQLite** via `androidx.sqlite` + `BundledSQLiteDriver`,
matching the existing `SQLiteEventStore` (`nip01Core/store/sqlite/`).
No flat append-only file; no hand-rolled height index. See §9.
- **No bundled headers file.** Ship a single hardcoded `Checkpoint(height,
blockHash, chainWork, time)` constant, bumped at release. First-run sync
starts at the checkpoint and pulls forward to the tip over P2P. See §7.
- **Pre-checkpoint OTS proofs fall through to HTTP** (`OkHttpBitcoinExplorer`).
No backward-from-checkpoint sync. Strict-mode users get a clear error
instead of opening an HTTP connection.
- **`BitcoinExplorerEndpoint`** (`amethyst/.../model/nip03Timestamp/`) is
now the shared Esplora-URL resolver for OTS and onchain zaps. The
composite resolver builder must read from it for the HTTP fallback path.
## 1. Motivation ## 1. Motivation
NIP-03 (OpenTimestamps attestations, kind 1040) is currently verified by hitting NIP-03 (OpenTimestamps attestations, kind 1040) is currently verified by hitting
@@ -42,8 +67,9 @@ proof-of-work chain instead of a trusted explorer.
verification logic. verification logic.
- Work over Tor when the user has Tor enabled. - Work over Tor when the user has Tor enabled.
- Keep existing HTTP explorers as a fallback for proofs older than the - Keep existing HTTP explorers as a fallback for proofs older than the
bundled checkpoint, or when local sync hasn't completed yet. pinned checkpoint, or when local sync hasn't completed yet.
- Live in `quartz/` so `cli/` and `desktopApp/` benefit too. - Live in `quartz/src/commonMain/.../bitcoin/` so `cli/`, `desktopApp/` and
any future merkle-block / onchain-zap work inherit the same primitives.
### Non-goals ### Non-goals
@@ -70,12 +96,13 @@ proof-of-work chain instead of a trusted explorer.
┌──────────────┼──────────────┐ ┌──────────────┼──────────────┐
│ │ │ │ │ │
OkHttpBitcoinExplorer │ LocalHeadersBitcoinExplorer ◄── NEW OkHttpBitcoinExplorer │ LocalHeadersBitcoinExplorer ◄── NEW
(existing, fallback) │ │ (existing, fallback, │ │
│ ▼ URL from │ ▼
│ ┌─────────────────────────┐ BitcoinExplorer- │ ┌─────────────────────────┐
│ │ HeaderStore │ Endpoint) │ │ HeaderStore │
│ │ (append-only file + │ │ │ (SQLite, androidx- │
│ │ height index) │ │ │ sqlite + Bundled- │
│ │ SQLiteDriver) │
│ └─────────────────────────┘ │ └─────────────────────────┘
│ ▲ │ ▲
│ │ │ │
@@ -105,17 +132,20 @@ proof-of-work chain instead of a trusted explorer.
│ │
CompositeBitcoinExplorer CompositeBitcoinExplorer
(try local first; if height (try local first; if height
not synced yet, fall back to HTTP) below pinned checkpoint OR not yet
reached by sync, fall back to HTTP
via BitcoinExplorerEndpoint)
``` ```
## 4. Module layout ## 4. Module layout
All new code in `quartz/` so non-Android targets (CLI, Desktop) inherit it. All new code in `quartz/src/commonMain/.../bitcoin/`, sibling of
Most of the protocol code is platform-independent; only socket I/O is in `nip03Timestamp/` and `nipBCOnchainZaps/`. Protocol code is
`jvmAndroid`. platform-independent; socket I/O is the only `expect`/`actual` boundary.
Storage uses `androidx.sqlite` which is already KMP — no platform split.
``` ```
quartz/src/commonMain/kotlin/com/vitorpamplona/quartz/nip03Timestamp/bitcoin/ quartz/src/commonMain/kotlin/com/vitorpamplona/quartz/bitcoin/
├── proto/ ├── proto/
│ ├── BitcoinNetMagic.kt # 0xf9beb4d9 mainnet │ ├── BitcoinNetMagic.kt # 0xf9beb4d9 mainnet
│ ├── MessageHeader.kt # 24-byte envelope codec │ ├── MessageHeader.kt # 24-byte envelope codec
@@ -128,47 +158,50 @@ quartz/src/commonMain/kotlin/com/vitorpamplona/quartz/nip03Timestamp/bitcoin/
│ │ ├── PingMessage.kt │ │ ├── PingMessage.kt
│ │ ├── PongMessage.kt │ │ ├── PongMessage.kt
│ │ ├── GetHeadersMessage.kt # locator + stop hash │ │ ├── GetHeadersMessage.kt # locator + stop hash
│ │ └── HeadersMessage.kt # up to 2,000 × (80B header + varint=0) │ │ ├── HeadersMessage.kt # up to 2,000 × (80B header + varint=0)
│ │ └── SendHeadersMessage.kt # BIP-130
│ └── BitcoinMessage.kt # sealed interface │ └── BitcoinMessage.kt # sealed interface
├── header/ ├── header/
│ ├── BlockHeader80.kt # 80-byte parsed header (ver, prev, merkle, time, bits, nonce) │ ├── BlockHeader80.kt # 80-byte parsed header (ver, prev, merkle, time, bits, nonce)
│ ├── BlockHash.kt # value class wrapping ByteArray(32) │ ├── BlockHash.kt # @JvmInline value class wrapping ByteArray(32)
│ ├── HeaderHasher.kt # dsha256 over 80B → blockHash │ ├── HeaderHasher.kt # dsha256 over 80B → blockHash
│ ├── DifficultyTarget.kt # bits ↔ target (256-bit) compact form │ ├── DifficultyTarget.kt # bits ↔ target (256-bit) compact form
│ ├── ChainWork.kt # cumulative work accumulator │ ├── ChainWork.kt # cumulative work accumulator (uint256 BE)
│ └── PowValidator.kt # checks hash ≤ target │ └── PowValidator.kt # checks hash ≤ target
├── consensus/ ├── consensus/
│ ├── RetargetCalculator.kt # every 2016 blocks, BIP-? rules │ ├── RetargetCalculator.kt # every 2016 blocks, BIP-? rules
│ ├── MedianTimePast.kt # last-11 median, for time sanity │ ├── MedianTimePast.kt # last-11 median, for time sanity
│ └── HeaderValidator.kt # composes the above + linkage check │ └── HeaderValidator.kt # composes the above + linkage check
├── store/ ├── store/
│ ├── HeaderStore.kt # append-only file + height→offset index │ ├── HeaderStore.kt # public API: append/getByHeight/getByHash/tip
│ ├── SqliteHeaderStore.kt # androidx.sqlite impl (commonMain — driver-agnostic)
│ ├── HeaderStoreSchema.kt # IModule for schema + migrations (see §9)
│ ├── Checkpoint.kt # (height, blockHash, chainWork, time) │ ├── Checkpoint.kt # (height, blockHash, chainWork, time)
│ └── HeaderRecord.kt # 80B header + cached blockHash + cumulativeChainWork │ └── PinnedCheckpoint.kt # hardcoded release-time checkpoint constant
├── sync/ ├── sync/
│ ├── LocatorBuilder.kt # exponential locator │ ├── LocatorBuilder.kt # exponential locator
│ ├── HeadersSyncEngine.kt # drives getheaders loop, applies validator, persists │ ├── HeadersSyncEngine.kt # drives getheaders loop, applies validator, persists
│ └── ReorgHandler.kt # rare; switches to higher-chainwork tip │ └── ReorgHandler.kt # rare; switches to higher-chainwork tip
├── peer/ ├── peer/
│ ├── PeerSocket.kt # expect (open / send / recv / close)
│ ├── PeerPool.kt # holds N BitcoinPeer connections │ ├── PeerPool.kt # holds N BitcoinPeer connections
│ ├── BitcoinPeer.kt # high-level: handshake, send msg, receive msg │ ├── BitcoinPeer.kt # high-level: handshake, send msg, receive msg
│ └── PeerScoring.kt # ban misbehaving peers │ ├── PeerScoring.kt # ban misbehaving peers
└── LocalHeadersBitcoinExplorer.kt # implements quartz.../ots/BitcoinExplorer │ └── PeerDiscovery.kt # expect — DNS seeds + fixed seeds
└── LocalHeadersBitcoinExplorer.kt # implements nip03Timestamp.ots.BitcoinExplorer
# (with small LRU on top, like OtsBlockHeightCache)
quartz/src/jvmAndroid/kotlin/com/vitorpamplona/quartz/nip03Timestamp/bitcoin/ quartz/src/jvmAndroid/kotlin/com/vitorpamplona/quartz/bitcoin/
├── peer/ └── peer/
│ ├── TcpPeerSocket.jvmAndroid.kt # actual TCP socket (java.net.Socket) ├── TcpPeerSocket.jvmAndroid.kt # actual TCP socket (java.net.Socket / SOCKS)
│ └── DnsSeedDiscovery.jvmAndroid.kt # InetAddress.getAllByName on seed hostnames └── DnsSeedDiscovery.jvmAndroid.kt # InetAddress.getAllByName on seed hostnames
└── store/
└── FileHeaderStore.jvmAndroid.kt # RandomAccessFile-backed HeaderStore
quartz/src/commonMain/kotlin/com/vitorpamplona/quartz/nip03Timestamp/bitcoin/
└── PeerSocket.kt # expect (open / send / recv / close)
└── HeaderStorage.kt # expect (append, getByHeight, tip)
``` ```
The `expect`/`actual` split keeps protocol logic testable cross-platform; the Two expect/actual seams only: `PeerSocket` (TCP plus optional SOCKS5 for Tor)
socket layer and disk layer are the only platform-dependent pieces. and `PeerDiscovery` (DNS-seed resolution). Header validation, message codecs,
store, sync engine, peer pool — all compile in `commonMain`, all testable
cross-platform without a single platform shim. Storage compiles in
`commonMain` too because `androidx.sqlite` + `BundledSQLiteDriver` are KMP.
### Wiring on Android ### Wiring on Android
@@ -182,7 +215,17 @@ amethyst/src/main/java/com/vitorpamplona/amethyst/model/nip03Timestamp/
`AppModules.kt` switches from `TorAwareOkHttpOtsResolverBuilder` to a `AppModules.kt` switches from `TorAwareOkHttpOtsResolverBuilder` to a
composite that prefers the local explorer and falls back to HTTP only when composite that prefers the local explorer and falls back to HTTP only when
local is missing the height. local is missing the height. The HTTP path resolves its base URL via the
existing `BitcoinExplorerEndpoint` (shared with the onchain-zap
`EsploraBackend`), so a user-configured custom Esplora is honoured on both
the OTS and the zap side.
`AppModules.kt:433` already passes `rootFilesDir = { appContext.filesDir }`
into Quartz; the headers SQLite file goes under
`$filesDir/bitcoin/headers.db` via a new entry in that wiring.
On desktop and CLI, the equivalent data-dir paths are already plumbed for
`SQLiteEventStore` — same hook reused here.
## 5. Bitcoin P2P protocol — minimum we implement ## 5. Bitcoin P2P protocol — minimum we implement
@@ -274,23 +317,35 @@ mirroring Bitcoin Core's own historical checkpoints, plus one **release-pinned
checkpoint** updated at app build time near the current tip. Sync must agree checkpoint** updated at app build time near the current tip. Sync must agree
with all checkpoints — disagreement aborts and bans the peer. with all checkpoints — disagreement aborts and bans the peer.
## 7. Bundled data ## 7. Bundled data — none. Just a pinned checkpoint constant.
To avoid making every fresh install download 75 MB: The point of this work is "verify against the Bitcoin network, not a trusted
party," so we ship no headers in the APK. Instead:
- App ships an **assets file** `bitcoin_headers_<height>.bin`: raw 80-byte - **One hardcoded `PinnedCheckpoint` constant** in `bitcoin/store/PinnedCheckpoint.kt`:
headers from genesis up to a release-pinned height (≤ 1 month before build). `(height, blockHash, chainWork, time)`. ~80 bytes of source.
~72 MB — too large for Play Store delta-update friendliness. - The checkpoint is pinned ~1 month before each release and bumped via a
- Ship instead a **compact bundled checkpoint**: just one `Checkpoint(height, one-line PR per release. This is the **only** thing trusted at build time
blockHash, chainWork, time)` per app release, plus the most recent ~50 K — and it's a single block hash any user can independently verify against
headers (~4 MB) so users can verify any attestation from the past ~year any Bitcoin node before installing.
immediately. - **First-run sync** starts at the checkpoint and pulls forward to the
- For older proofs, lazy-sync from the checkpoint *backwards* down to genesis current tip over P2P. At a ~1-month checkpoint that's ~4,300 headers
on demand (rare path). (~340 KB on the wire, ~15 minutes' worth of `getheaders` round-trips at
- Forward sync from checkpoint to current tip is what runs at first launch worst), seconds over WiFi.
(~10 KB to a few MB depending on release age). - **Pre-checkpoint heights** are not synced locally at all. OTS proofs
whose height precedes the checkpoint fall through to
`OkHttpBitcoinExplorer` via the composite. Users in strict-mode
(`localHeadersTrustOnly = true`) get a clear
"this OTS proof predates the bundled checkpoint" error and no network call.
- **Catch-up after long absence.** A device that hasn't run the app for
months will, on next sync, fast-forward from `tip_local` (its stored
best_height) to `tip_network`. Each `headers` message brings 2,000
headers; even 6 months of absence is ~26,000 headers (~2 MB) in 13 round
trips.
- **Header growth on disk** is ~4 MB raw per year (+ SQLite overhead, see §9).
This keeps APK growth at ~4 MB and first-run sync time at seconds. APK growth: **0 bytes**. Trust delta vs the old HTTP-explorer model: one
block hash, code-reviewable in git.
## 8. Multi-peer strategy & eclipse mitigation ## 8. Multi-peer strategy & eclipse mitigation
@@ -315,42 +370,85 @@ A single peer can lie. We protect against that with:
(`bitcoin.example.com:8333`) — same path as the existing custom explorer (`bitcoin.example.com:8333`) — same path as the existing custom explorer
URL setting in `OtsSettings.kt`. URL setting in `OtsSettings.kt`.
## 9. Storage format ## 9. Storage — SQLite via `androidx.sqlite` + `BundledSQLiteDriver`
Append-only flat file plus a sparse index. We reuse the same storage primitive Quartz already uses for `SQLiteEventStore`
(`quartz/.../nip01Core/store/sqlite/`): `androidx.sqlite` with the bundled
driver. This is KMP, ships in `androidx-sqlite-bundled` / `…-bundled-jvm`
(already a Quartz dependency), and is the project's standard storage choice.
### `headers.bin` The complete `SQLiteEventStore` infrastructure — `BundledSQLiteDriver`,
`SQLiteConnectionPool`, `transaction { }`, `IModule` for migrations,
`QueryBuilder` helpers — is reused by `SqliteHeaderStore` and
`HeaderStoreSchema`.
Records, fixed 112 bytes each, indexed by height (record N at offset `N*112`): ### Schema
``` ```sql
+0 80B raw block header PRAGMA journal_mode = WAL;
+80 32B cumulative chainwork (uint256, big-endian) PRAGMA synchronous = NORMAL;
PRAGMA foreign_keys = OFF;
CREATE TABLE block_header (
height INTEGER PRIMARY KEY, -- 0..tip, dense
hash BLOB NOT NULL, -- 32 B, dsha256(header), big-endian display order
header BLOB NOT NULL, -- 80 B raw header
chainwork BLOB NOT NULL -- 32 B cumulative work, big-endian
);
CREATE UNIQUE INDEX idx_block_header_hash ON block_header(hash);
CREATE TABLE chain_tip (
id INTEGER PRIMARY KEY CHECK (id = 1), -- singleton row
best_height INTEGER NOT NULL,
best_hash BLOB NOT NULL,
best_work BLOB NOT NULL
);
CREATE TABLE checkpoint_seen (
height INTEGER PRIMARY KEY, -- which checkpoints we've passed and confirmed
hash BLOB NOT NULL,
source TEXT NOT NULL -- 'pinned' | 'historical'
);
``` ```
The block hash and target are recomputed from bytes 0..80 lazily; we don't Migrations live in `HeaderStoreSchema.kt` as an `IModule` (same pattern as
store them. Total at height 1M: 112 MB. We can drop this to 80 B/record the event store's modules). v1 is the schema above; future additions
(no chainwork) if we accept recomputing chainwork on startup — TBD; default (prev-hash cache column, merkle-block annotations) ALTER from there.
is 112 B for fast tip selection.
### `tips.json` ### Sizing
A small JSON file holding: | Item | Per record | At height 1,000,000 |
|---|---|---|
| `block_header` row (80+32+32 B payload + SQLite overhead) | ~180–220 B | ~180–220 MB |
| `idx_block_header_hash` B-tree (32 B key + rowid) | ~40 B | ~40 MB |
| `chain_tip` | 1 row | negligible |
| **Total** | | **~220–260 MB at height 1 M** |
```json In practice users will start from a recent checkpoint and grow ~4 MB raw
{ (~10 MB stored) per year, so most installs sit at 10–40 MB. The "1 M
"best_height": 945123, height" line is the long-tail worst case if someone with a multi-year-old
"best_hash": "00000000000000000001abc...", device's local store retroactively backfills (which we explicitly don't do
"best_work": "0x000...", in v1 — see §15 Q1).
"checkpoints_passed": [600000, 700000, 800000, 940000]
}
```
### Atomicity ### Atomicity & crash safety
Sync writes to `headers.bin.tmp`, fsyncs, then atomically renames or - `HeadersSyncEngine` writes each `headers` batch (up to 2,000) inside a
appends-and-fsyncs in batches of 2000. Crash recovery: on startup, scan from single `transaction { }`. WAL gives us atomic-or-nothing per batch.
last fsynced offset, validate linkage, and discard partial trailing records. - `chain_tip` is updated in the same transaction as the headers it points to,
so the tip can never reference rows that didn't commit.
- A crash mid-batch leaves the WAL inconsistent; SQLite rolls it back on
next open. No custom recovery code.
- On open, `SqliteHeaderStore.verifyTipIntegrity()` re-reads the tip row and
confirms the referenced header exists. If not (unexpected DB corruption,
ransomware-like sandbox damage), we delete the db file and fall back to
re-sync from the pinned checkpoint.
### Concurrency
Same model as `SQLiteEventStore`: one connection pool, all writes go through
the sync engine on a dedicated dispatcher; reads use the pool's read slots.
OTS verifier reads are short and indexed — no contention with sync writes.
## 10. Resolving a height query (the actual `BitcoinExplorer` impl) ## 10. Resolving a height query (the actual `BitcoinExplorer` impl)
@@ -410,20 +508,25 @@ Extend `OtsSettings.kt`:
```kotlin ```kotlin
data class OtsSettings( data class OtsSettings(
val customExplorerUrl: String? = null, val customExplorerUrl: String? = null, // existing — shared via BitcoinExplorerEndpoint
// NEW: // NEW:
val localHeadersEnabled: Boolean = true, val localHeadersEnabled: Boolean = true,
val localHeadersTrustOnly: Boolean = false, // disables HTTP fallback val localHeadersTrustOnly: Boolean = false, // disables HTTP fallback; pre-checkpoint proofs error
val localHeadersWifiOnly: Boolean = true, val localHeadersWifiOnly: Boolean = true,
val customP2pNode: String? = null, // e.g. "myriad.example.com:8333" val customP2pNode: String? = null, // e.g. "myriad.example.com:8333"
) )
``` ```
A short "Bitcoin verification" settings screen lets the user pick: A short "Bitcoin verification" settings screen lets the user pick:
- Local headers (default) - **Local headers + HTTP fallback** (default) — best UX, trustless for
- HTTP explorer only (current behavior, lighter) post-checkpoint heights, HTTP for older proofs.
- Local + HTTP fallback - **Local headers only (strict)** — refuses to verify proofs older than the
- Local only (strict, may delay verification of brand-new attestations) pinned checkpoint. Most paranoid setting.
- **HTTP explorer only** — pre-local-headers behaviour. Lightest disk/CPU.
The `customExplorerUrl` field continues to feed `BitcoinExplorerEndpoint`,
so a custom HTTP Esplora is honoured both by the fallback in this feature
and by the onchain-zaps `EsploraBackend`. No new Esplora setting.
## 12. Tor integration ## 12. Tor integration
@@ -461,10 +564,15 @@ Each phase produces a shippable artifact with measurable behavior.
### Phase 2 — Storage layer (2–3 days) ### Phase 2 — Storage layer (2–3 days)
- `HeaderStore` interface + `FileHeaderStore` actual on jvmAndroid. - `HeaderStore` interface + `SqliteHeaderStore` implementation in `commonMain`
- Append, get-by-height, get-by-hash (in-memory hash→height map built at (`androidx.sqlite` is KMP, no platform shim needed).
startup, ~60 MB at height 1 M — acceptable, or use a sparse on-disk index). - `HeaderStoreSchema` as an `IModule` matching the `SQLiteEventStore`
- Crash-recovery test. migration pattern.
- `append(batch)`, `getByHeight(h)`, `getByHash(hash)`, `tip()`.
- Reuse `BundledSQLiteDriver`, `SQLiteConnectionPool`, `transaction { }`
from `nip01Core/store/sqlite/` — no duplication.
- Crash-safety test: kill mid-transaction, reopen, confirm WAL rollback;
confirm `verifyTipIntegrity()` recovers cleanly.
### Phase 3 — P2P codec, no socket (3–5 days) ### Phase 3 — P2P codec, no socket (3–5 days)
@@ -488,13 +596,16 @@ Each phase produces a shippable artifact with measurable behavior.
- Header-batch cross-validation across peers. - Header-batch cross-validation across peers.
- Hardcoded checkpoints in `Checkpoints.kt`. - Hardcoded checkpoints in `Checkpoints.kt`.
### Phase 6 — Bundled checkpoint & lazy-genesis-sync (2–3 days) ### Phase 6 — Pinned checkpoint constant (½ day)
- Build-time generation of the release-pinned checkpoint and trailing-50K - `bitcoin/store/PinnedCheckpoint.kt` — single hardcoded
headers blob, packaged as `quartz/src/jvmAndroid/resources/bitcoin/...`. `Checkpoint(height, blockHash, chainWork, time)`. ~80 bytes of source.
- Forward sync from checkpoint to tip on first launch. - A small Gradle task (`./gradlew bumpBitcoinCheckpoint`) that takes
- Backward (genesis-direction) sync triggered only when an OTS proof requests `(height, hash)` arguments and writes the constant. Run by the release
a height before the checkpoint. process; otherwise no automation.
- No backward-sync, no bundled headers blob, no asset resources.
Pre-checkpoint heights are routed to the HTTP fallback by the composite
resolver (or refused in strict mode).
### Phase 7 — Android wiring (3–4 days) ### Phase 7 — Android wiring (3–4 days)
@@ -512,9 +623,14 @@ Each phase produces a shippable artifact with measurable behavior.
### Phase 9 — Desktop & CLI integration (1–2 days) ### Phase 9 — Desktop & CLI integration (1–2 days)
- `desktopApp/` already inherits the explorer because it's in `quartz/`. - `desktopApp/` inherits the explorer for free (everything is in
Confirm it works end-to-end on JVM. `quartz/commonMain/`). Confirm headers.db path is plumbed via the same
- `cli/` (`amy verify-ots`) — confirm CLI sync also works headlessly. data-dir hook already used by `SQLiteEventStore` on JVM. Confirm sync
runs end-to-end on the desktop app.
- `cli/` — add an `amy verify-ots <event-json>` command if not present,
wired through the same composite resolver. The CLI is a thin assembly
layer over Quartz (per `amy-expert`), so the actual sync logic is reused
as-is.
### Phase 10 — Rollout & telemetry (ongoing) ### Phase 10 — Rollout & telemetry (ongoing)
@@ -523,8 +639,9 @@ Each phase produces a shippable artifact with measurable behavior.
fallback rate. Surface in a debug screen. fallback rate. Surface in a debug screen.
- Default-on in the next release if metrics look healthy. - Default-on in the next release if metrics look healthy.
Total: **~25–35 engineer-days** end to end. Phases 1–4 are the bulk Total: **~22–32 engineer-days** end to end. Phases 1–4 are the bulk
(protocol + validation); phases 5–10 are integration. (protocol + validation); phases 5–10 are integration. Phase 6 dropped from
2–3 d to ½ d because we ship no bundled data — just one constant.
## 14. Testing strategy ## 14. Testing strategy
@@ -544,8 +661,12 @@ Total: **~25–35 engineer-days** end to end. Phases 1–4 are the bulk
(committed as `quartz/src/jvmAndroid/test/resources/btc-handshake.bin`). (committed as `quartz/src/jvmAndroid/test/resources/btc-handshake.bin`).
- Local-network test against a Bitcoin Core regtest node spun up in CI - Local-network test against a Bitcoin Core regtest node spun up in CI
(Docker). Validates full handshake + `getheaders` loop end to end. (Docker). Validates full handshake + `getheaders` loop end to end.
- Crash-and-resume test: kill mid-sync, restart, confirm we pick up at the - `SqliteHeaderStore` round-trip on a real on-disk file (jvmTest, in a
correct height. `tmpdir`): append 5,000 headers in batches, getByHeight, getByHash,
reopen, confirm tip survives.
- Crash-and-resume test: kill the writer in the middle of a `transaction { }`,
reopen, confirm SQLite WAL rolls the partial batch back and sync resumes
from the last committed tip.
### Property tests ### Property tests
@@ -564,32 +685,41 @@ Total: **~25–35 engineer-days** end to end. Phases 1–4 are the bulk
| # | Risk | Mitigation | | # | Risk | Mitigation |
|---|------|------------| |---|------|------------|
| R1 | First-run UX cost (1–10 min sync) puts users off | Bundled checkpoint + 50 K trailing headers means most installs need only seconds. Show progress prominently. | | R1 | First-run UX cost (sync from pinned checkpoint) | Pinned checkpoint ~1 month before release means ~4 K headers, ~340 KB, seconds over WiFi. Show progress in settings; sync runs in background WorkManager job anyway. |
| R2 | Mobile OS kills socket mid-sync | Foreground service for first run; resumable design. | | R2 | Mobile OS kills socket mid-sync | Foreground service for first run; resumable design (SQLite WAL handles partial batches automatically). |
| R3 | Peer feeds adversarial headers (eclipse) | Multi-peer + checkpoints + chainwork comparison. | | R3 | Peer feeds adversarial headers (eclipse) | Multi-peer + checkpoints + chainwork comparison. |
| R4 | Disk corruption | Atomic appends, checksum on close, fall back to re-sync from last checkpoint. | | R4 | Disk corruption | SQLite WAL + `verifyTipIntegrity()` on open; on detected corruption, delete db file and re-sync from pinned checkpoint. |
| R5 | Tor + P2P is slow | Increased timeouts; fallback to HTTP-over-Tor explorer when sync stalls. | | R5 | Tor + P2P is slow | Increased timeouts; fallback to HTTP-over-Tor explorer (via `BitcoinExplorerEndpoint`) when sync stalls. |
| R6 | APK size growth from bundled headers | Cap bundle at most-recent ~50 K headers (~4 MB). Older headers fetched on demand. | | R6 | APK size growth | Zero — no bundled data ships in the APK. |
| R7 | Privacy: peers see user's IP | Tor mode supported; `version` user-agent doesn't leak Amethyst install ID. | | R7 | Privacy: peers see user's IP | Tor mode supported; `version` user-agent doesn't leak Amethyst install ID. |
| R8 | Battery cost on background sync | Default to UNMETERED + BATTERY_OK constraints; back off when idle. | | R8 | Battery cost on background sync | Default to UNMETERED + BATTERY_OK constraints; back off when idle. |
| R9 | Maintenance burden — new soft-forks | Soft-forks don't affect headers-only validation (no script eval). Difficulty algorithm hasn't changed; a future change would require a code update. | | R9 | Maintenance burden — new soft-forks | Soft-forks don't affect headers-only validation (no script eval). Difficulty algorithm hasn't changed; a future change would require a code update. |
| R10 | Test flakiness if CI talks to public Bitcoin nodes | Use Bitcoin Core regtest in Docker for integration tests; public nodes only nightly. | | R10 | Test flakiness if CI talks to public Bitcoin nodes | Use Bitcoin Core regtest in Docker for integration tests; public nodes only nightly. |
| R11 | SQLite store size on long-running installs | ~10 MB/year typical; ~220 MB at height 1 M which won't happen in any realistic mobile lifetime. Add a "wipe local headers" button in settings for users who want it. |
### Open questions ### Open questions
- **Q1.** Do we actually want backward-from-checkpoint sync, or just refuse #### Resolved 2026-05-19
to verify proofs older than the bundled checkpoint and tell users to use
the HTTP fallback? Backward sync is operational complexity for a rare path. - **Q1.** *Resolved — HTTP fallback only.* Pre-checkpoint OTS proofs are
- **Q2.** Storage: 80 B/record (recompute chainwork on startup) vs routed to `OkHttpBitcoinExplorer` via the composite resolver. Strict-mode
112 B/record (cache it)? At height 1M, 32 MB difference. Probably worth users (`localHeadersTrustOnly = true`) get a clear error. No backward
caching. sync code path.
- **Q3.** Should we expose this as a Quartz public API (so 3rd-party apps - **Q2.** *Resolved — cache chainwork.* SQLite makes the choice cheap (one
using Quartz also benefit) or keep it internal to NIP-03? Recommendation: extra BLOB column). Reading current tip work without a full scan matters
public — it's generally useful. for chain-selection logic; the ~30 MB-at-1M cost is negligible on disk.
- **Q4.** Do we ship a "headers blob" download URL (HTTPS) as a faster - **Q4.** *Resolved — no HTTPS headers blob.* The point of this feature is
alternative to P2P for first-run? Trade-off: faster but adds a trust point. trustless verification, and the pinned-checkpoint approach already gives
Could be made cross-checked (download blob, then verify every header's PoW fast first-run. Adding a second ingestion path (signed blob over HTTPS)
locally — same trust as P2P then). Worth considering as Phase 6b. doubles the test surface for marginal benefit.
#### Still open
- **Q3.** Should `LocalHeadersBitcoinExplorer` (and friends) be a Quartz
public API for 3rd-party Quartz consumers, or internal to Amethyst?
Recommendation: public — it's a clean reusable primitive. Decide before
Phase 0 lands so the package layout doesn't churn. *(See `quartz-integration`
skill for the public-API contract Quartz exposes today.)*
## 16. Migration & rollout ## 16. Migration & rollout
@@ -605,22 +735,55 @@ Total: **~25–35 engineer-days** end to end. Phases 1–4 are the bulk
- A NIP-03 attestation that verifies via Blockstream also verifies via the - A NIP-03 attestation that verifies via Blockstream also verifies via the
local explorer (bit-for-bit identical timestamp). local explorer (bit-for-bit identical timestamp).
- 95th-percentile cold-launch verification of a recent attestation: ≤2 s. - 95th-percentile cold-launch verification of a recent attestation: ≤2 s.
- 95th-percentile first-run sync (post-checkpoint): ≤30 s on WiFi. - 95th-percentile first-run sync (from pinned checkpoint to tip): ≤30 s on
WiFi for a checkpoint ≤2 months old.
- No new crash-rate regression in the verification path. - No new crash-rate regression in the verification path.
- App size growth: ≤5 MB. - App size growth: **0 MB** (no bundled data).
- A user with HTTP explorer disabled can still verify any post-checkpoint - A user with HTTP explorer disabled can still verify any post-checkpoint
attestation. attestation.
## 18. Out of scope (explicit) ## 18. Out of scope (explicit)
- BIP-157/158 compact filters - BIP-157/158 compact filters
- BIP-37 merkleblock or full-block fetch
- **Trustless verification of NIP-BC onchain zaps** — needs merkleblock or
full-block fetch on top of this stack. See §20 for the follow-up plan.
- **Trustless wallet operations** for NIP-BC onchain zap *sending* —
needs UTXO-by-address and broadcast paths that headers can't provide.
Stays on user-configured Esplora.
- Lightning, BIP-32 wallet, mempool monitoring - Lightning, BIP-32 wallet, mempool monitoring
- Verifying NIP-03 attestations against altcoin/Liquid timestamps (not in - Verifying NIP-03 attestations against altcoin/Liquid timestamps (not in
the spec) the spec)
- Replacing the calendar-server upgrade path (`OtsState.upgrade()`) — - Replacing the calendar-server upgrade path (`OtsState.upgrade()`) —
unrelated, those are simple HTTPS calls and stay as-is unrelated, those are simple HTTPS calls and stay as-is
## 19. References ## 19. Follow-up: trustless NIP-BC onchain zap verification
*Not in this plan. Tracked here so the design choices above stay aligned.*
The shipped `OnchainZapVerifier` (`quartz/.../nipBCOnchainZaps/verify/`)
trusts `OnchainBackend.getTx(txid)` — i.e. Esplora — for both
tx-existence-in-a-block and the output values used to compute
`verifiedSats`. A malicious Esplora can fabricate or suppress zaps.
Closing that gap is a **separate plan** that builds on this one:
- **Foundation reused:** `BitcoinPeer`, `PeerPool`, `HeaderStore`,
`HeaderValidator`, `LocalHeadersBitcoinExplorer`.
- **New work (rough ~10–15 days on top of S1):** BIP-37 `filterload` /
`merkleblock` / matched-`tx` handling, or `getdata MSG_BLOCK` + stream
parse; partial-merkle-tree verification; new `OnchainBackend` variant
that asks the local stack for the inclusion proof and the matched tx
while keeping Esplora for UTXO listing / broadcast / fee estimates on
the *send* side.
- **Net win:** zap-receive verification stops trusting Esplora. Zap-send
still needs an address-indexed server (no headers-only equivalent
exists), so that part of the trust story is unchanged.
- **Filename:** `quartz/plans/YYYY-MM-DD-onchain-zap-merkle-proofs.md`,
written once S1 has shipped and we have real numbers on first-run sync,
store size, and battery cost.
## 20. References
- NIP-03: https://github.com/nostr-protocol/nips/blob/master/03.md - NIP-03: https://github.com/nostr-protocol/nips/blob/master/03.md
- OpenTimestamps spec: https://github.com/opentimestamps/python-opentimestamps - OpenTimestamps spec: https://github.com/opentimestamps/python-opentimestamps
@@ -632,6 +795,16 @@ Total: **~25–35 engineer-days** end to end. Phases 1–4 are the bulk
- `quartz/src/commonMain/kotlin/com/vitorpamplona/quartz/nip03Timestamp/ots/BitcoinExplorer.kt` - `quartz/src/commonMain/kotlin/com/vitorpamplona/quartz/nip03Timestamp/ots/BitcoinExplorer.kt`
- `quartz/src/commonMain/kotlin/com/vitorpamplona/quartz/nip03Timestamp/ots/BlockHeader.kt` - `quartz/src/commonMain/kotlin/com/vitorpamplona/quartz/nip03Timestamp/ots/BlockHeader.kt`
- `quartz/src/commonMain/kotlin/com/vitorpamplona/quartz/nip03Timestamp/ots/attestation/BitcoinBlockHeaderAttestation.kt` - `quartz/src/commonMain/kotlin/com/vitorpamplona/quartz/nip03Timestamp/ots/attestation/BitcoinBlockHeaderAttestation.kt`
- `quartz/src/commonMain/kotlin/com/vitorpamplona/quartz/nip03Timestamp/ots/OtsBlockHeightCache.kt`
- `quartz/src/jvmAndroid/kotlin/com/vitorpamplona/quartz/nip03Timestamp/okhttp/OkHttpBitcoinExplorer.kt` - `quartz/src/jvmAndroid/kotlin/com/vitorpamplona/quartz/nip03Timestamp/okhttp/OkHttpBitcoinExplorer.kt`
- `amethyst/src/main/java/com/vitorpamplona/amethyst/model/nip03Timestamp/TorAwareOkHttpOtsResolverBuilder.kt` - `amethyst/src/main/java/com/vitorpamplona/amethyst/model/nip03Timestamp/TorAwareOkHttpOtsResolverBuilder.kt`
- `amethyst/src/main/java/com/vitorpamplona/amethyst/model/nip03Timestamp/OtsSettings.kt` - `amethyst/src/main/java/com/vitorpamplona/amethyst/model/nip03Timestamp/OtsSettings.kt`
- `amethyst/src/main/java/com/vitorpamplona/amethyst/model/nip03Timestamp/BitcoinExplorerEndpoint.kt`
- Storage precedent (reused as-is):
- `quartz/src/commonMain/kotlin/com/vitorpamplona/quartz/nip01Core/store/sqlite/SQLiteEventStore.kt`
- `quartz/src/commonMain/kotlin/com/vitorpamplona/quartz/nip01Core/store/sqlite/IModule.kt`
- `quartz/src/commonMain/kotlin/com/vitorpamplona/quartz/nip01Core/store/sqlite/SQLiteConnectionExt.kt`
- `quartz/src/commonMain/kotlin/com/vitorpamplona/quartz/nip01Core/store/sqlite/SQLiteConnectionPool.kt`
- Follow-up area:
- `quartz/src/commonMain/kotlin/com/vitorpamplona/quartz/nipBCOnchainZaps/verify/OnchainZapVerifier.kt`
- `quartz/src/commonMain/kotlin/com/vitorpamplona/quartz/nipBCOnchainZaps/chain/OnchainBackend.kt`