nick/amethyst
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 1 Wiki Activity

docs(quartz): revise local-headers-explorer plan — SQLite, no bundle, OTS-only

Browse Source 
Lock the design choices from the 2026-05-19 review against the intervening
2026-05-14 onchain-zaps work:

- Move the module from quartz/.../nip03Timestamp/bitcoin/ to a sibling
  quartz/.../bitcoin/ package so the headers explorer, header validator,
  peer pool and store can be reused by the future onchain-zap merkle-proof
  work without an inverted import path.
- Use androidx.sqlite + BundledSQLiteDriver for HeaderStore, matching the
  existing SQLiteEventStore. Schema lives in commonMain via IModule. Drops
  the hand-rolled flat-file + sidecar height index.
- Drop the bundled headers blob. Ship a single hardcoded PinnedCheckpoint
  constant; first-run sync starts from the checkpoint and pulls forward
  over P2P. APK growth: 0 bytes.
- Pre-checkpoint OTS heights fall through to OkHttpBitcoinExplorer via
  BitcoinExplorerEndpoint (shared with the onchain-zap EsploraBackend).
  Strict-mode users get an explicit error instead of a network call.
- Mark trustless NIP-BC onchain-zap verification as out of scope and
  capture it as a follow-up plan (BIP-37 merkleblock or full-block fetch
  on top of this stack).

Resolves open questions Q1, Q2 and Q4 from the original plan; Q3 (Quartz
public API vs internal) left open for Phase 0.
This commit is contained in:
Claude
2026-05-19 18:14:30 +00:00
parent ed686b71ed
commit f3637dd7c1
1 changed files with 292 additions and 119 deletions
Download Patch File Download Diff File Expand all files Collapse all files
quartz/plans/2026-05-08-local-headers-explorer.md
+292 -119
View File
@@ -1,10 +1,35 @@
# Local Bitcoin headers explorer for NIP-03 OTS
**Date:** 2026-05-08
**Branch:** `claude/review-ots-blockchain-deps-bKns7`
**Date:** 2026-05-08 (revised 2026-05-19)
**Branch:** `claude/review-bitcoin-blockchain-plan-6NGZ5` (originally `claude/review-ots-blockchain-deps-bKns7`)
**Module:** `quartz/` (with thin Android wiring in `amethyst/`)
**Status:** Plan — not yet implemented
## Revision log
**2026-05-19 — decisions locked, scope tightened.** After review against the
intervening 2026-05-14 onchain-zaps work:
- **Scope stays OTS-only.** Trustless verification of NIP-BC onchain zaps
(kind 8333) needs more than headers — at minimum BIP-37 merkleblock or
full-block fetch on top of this stack. That becomes a follow-up plan
(see §20).
- **Module moves to `quartz/.../bitcoin/`** — sibling of `nip03Timestamp/`
and `nipBCOnchainZaps/`, not nested under NIP-03. The future merkle-block
phase will reuse `BitcoinPeer` / `HeaderStore` / `HeaderValidator`.
- **Storage is SQLite** via `androidx.sqlite` + `BundledSQLiteDriver`,
matching the existing `SQLiteEventStore` (`nip01Core/store/sqlite/`).
No flat append-only file; no hand-rolled height index. See §9.
- **No bundled headers file.** Ship a single hardcoded `Checkpoint(height,
blockHash, chainWork, time)` constant, bumped at release. First-run sync
starts at the checkpoint and pulls forward to the tip over P2P. See §7.
- **Pre-checkpoint OTS proofs fall through to HTTP** (`OkHttpBitcoinExplorer`).
No backward-from-checkpoint sync. Strict-mode users get a clear error
instead of opening an HTTP connection.
- **`BitcoinExplorerEndpoint`** (`amethyst/.../model/nip03Timestamp/`) is
now the shared Esplora-URL resolver for OTS and onchain zaps. The
composite resolver builder must read from it for the HTTP fallback path.
## 1. Motivation
NIP-03 (OpenTimestamps attestations, kind 1040) is currently verified by hitting
@@ -42,8 +67,9 @@ proof-of-work chain instead of a trusted explorer.
verification logic.
- Work over Tor when the user has Tor enabled.
- Keep existing HTTP explorers as a fallback for proofs older than the
bundled checkpoint, or when local sync hasn't completed yet.
- Live in `quartz/` so `cli/` and `desktopApp/` benefit too.
pinned checkpoint, or when local sync hasn't completed yet.
- Live in `quartz/src/commonMain/.../bitcoin/` so `cli/`, `desktopApp/` and
any future merkle-block / onchain-zap work inherit the same primitives.
### Non-goals
@@ -70,12 +96,13 @@ proof-of-work chain instead of a trusted explorer.
┌──────────────┼──────────────┐
│ │ │
OkHttpBitcoinExplorer │ LocalHeadersBitcoinExplorer ◄── NEW
(existing, fallback) │ │
│ ▼
│ ┌─────────────────────────┐
│ │ HeaderStore │
│ │ (append-only file + │
│ │ height index) │
(existing, fallback, │ │
URL from │ ▼
BitcoinExplorer- │ ┌─────────────────────────┐
Endpoint) │ │ HeaderStore │
│ │ (SQLite, androidx- │
│ │ sqlite + Bundled- │
│ │ SQLiteDriver) │
│ └─────────────────────────┘
│ ▲
│ │
@@ -105,17 +132,20 @@ proof-of-work chain instead of a trusted explorer.
│
CompositeBitcoinExplorer
(try local first; if height
not synced yet, fall back to HTTP)
below pinned checkpoint OR not yet
reached by sync, fall back to HTTP
via BitcoinExplorerEndpoint)
```
## 4. Module layout
All new code in `quartz/` so non-Android targets (CLI, Desktop) inherit it.
Most of the protocol code is platform-independent; only socket I/O is in
`jvmAndroid`.
All new code in `quartz/src/commonMain/.../bitcoin/`, sibling of
`nip03Timestamp/` and `nipBCOnchainZaps/`. Protocol code is
platform-independent; socket I/O is the only `expect`/`actual` boundary.
Storage uses `androidx.sqlite` which is already KMP — no platform split.
```
quartz/src/commonMain/kotlin/com/vitorpamplona/quartz/nip03Timestamp/bitcoin/
quartz/src/commonMain/kotlin/com/vitorpamplona/quartz/bitcoin/
├── proto/
│ ├── BitcoinNetMagic.kt # 0xf9beb4d9 mainnet
│ ├── MessageHeader.kt # 24-byte envelope codec
@@ -128,47 +158,50 @@ quartz/src/commonMain/kotlin/com/vitorpamplona/quartz/nip03Timestamp/bitcoin/
│ │ ├── PingMessage.kt
│ │ ├── PongMessage.kt
│ │ ├── GetHeadersMessage.kt # locator + stop hash
│ │ └── HeadersMessage.kt # up to 2,000 × (80B header + varint=0)
│ │ ├── HeadersMessage.kt # up to 2,000 × (80B header + varint=0)
│ │ └── SendHeadersMessage.kt # BIP-130
│ └── BitcoinMessage.kt # sealed interface
├── header/
│ ├── BlockHeader80.kt # 80-byte parsed header (ver, prev, merkle, time, bits, nonce)
│ ├── BlockHash.kt # value class wrapping ByteArray(32)
│ ├── BlockHash.kt # @JvmInline value class wrapping ByteArray(32)
│ ├── HeaderHasher.kt # dsha256 over 80B → blockHash
│ ├── DifficultyTarget.kt # bits ↔ target (256-bit) compact form
│ ├── ChainWork.kt # cumulative work accumulator
│ ├── ChainWork.kt # cumulative work accumulator (uint256 BE)
│ └── PowValidator.kt # checks hash ≤ target
├── consensus/
│ ├── RetargetCalculator.kt # every 2016 blocks, BIP-? rules
│ ├── MedianTimePast.kt # last-11 median, for time sanity
│ └── HeaderValidator.kt # composes the above + linkage check
├── store/
│ ├── HeaderStore.kt # append-only file + height→offset index
│ ├── HeaderStore.kt # public API: append/getByHeight/getByHash/tip
│ ├── SqliteHeaderStore.kt # androidx.sqlite impl (commonMain — driver-agnostic)
│ ├── HeaderStoreSchema.kt # IModule for schema + migrations (see §9)
│ ├── Checkpoint.kt # (height, blockHash, chainWork, time)
│ └── HeaderRecord.kt # 80B header + cached blockHash + cumulativeChainWork
│ └── PinnedCheckpoint.kt # hardcoded release-time checkpoint constant
├── sync/
│ ├── LocatorBuilder.kt # exponential locator
│ ├── HeadersSyncEngine.kt # drives getheaders loop, applies validator, persists
│ └── ReorgHandler.kt # rare; switches to higher-chainwork tip
├── peer/
│ ├── PeerSocket.kt # expect (open / send / recv / close)
│ ├── PeerPool.kt # holds N BitcoinPeer connections
│ ├── BitcoinPeer.kt # high-level: handshake, send msg, receive msg
│ └── PeerScoring.kt # ban misbehaving peers
└── LocalHeadersBitcoinExplorer.kt # implements quartz.../ots/BitcoinExplorer
│ ├── PeerScoring.kt # ban misbehaving peers
│ └── PeerDiscovery.kt # expect — DNS seeds + fixed seeds
└── LocalHeadersBitcoinExplorer.kt # implements nip03Timestamp.ots.BitcoinExplorer
# (with small LRU on top, like OtsBlockHeightCache)
quartz/src/jvmAndroid/kotlin/com/vitorpamplona/quartz/nip03Timestamp/bitcoin/
├── peer/
│ ├── TcpPeerSocket.jvmAndroid.kt # actual TCP socket (java.net.Socket)
│ └── DnsSeedDiscovery.jvmAndroid.kt # InetAddress.getAllByName on seed hostnames
└── store/
└── FileHeaderStore.jvmAndroid.kt # RandomAccessFile-backed HeaderStore
quartz/src/commonMain/kotlin/com/vitorpamplona/quartz/nip03Timestamp/bitcoin/
└── PeerSocket.kt # expect (open / send / recv / close)
└── HeaderStorage.kt # expect (append, getByHeight, tip)
quartz/src/jvmAndroid/kotlin/com/vitorpamplona/quartz/bitcoin/
└── peer/
├── TcpPeerSocket.jvmAndroid.kt # actual TCP socket (java.net.Socket / SOCKS)
└── DnsSeedDiscovery.jvmAndroid.kt # InetAddress.getAllByName on seed hostnames
```
The `expect`/`actual` split keeps protocol logic testable cross-platform; the
socket layer and disk layer are the only platform-dependent pieces.
Two expect/actual seams only: `PeerSocket` (TCP plus optional SOCKS5 for Tor)
and `PeerDiscovery` (DNS-seed resolution). Header validation, message codecs,
store, sync engine, peer pool — all compile in `commonMain`, all testable
cross-platform without a single platform shim. Storage compiles in
`commonMain` too because `androidx.sqlite` + `BundledSQLiteDriver` are KMP.
### Wiring on Android
@@ -182,7 +215,17 @@ amethyst/src/main/java/com/vitorpamplona/amethyst/model/nip03Timestamp/
`AppModules.kt` switches from `TorAwareOkHttpOtsResolverBuilder` to a
composite that prefers the local explorer and falls back to HTTP only when
local is missing the height.
local is missing the height. The HTTP path resolves its base URL via the
existing `BitcoinExplorerEndpoint` (shared with the onchain-zap
`EsploraBackend`), so a user-configured custom Esplora is honoured on both
the OTS and the zap side.
`AppModules.kt:433` already passes `rootFilesDir = { appContext.filesDir }`
into Quartz; the headers SQLite file goes under
`$filesDir/bitcoin/headers.db` via a new entry in that wiring.
On desktop and CLI, the equivalent data-dir paths are already plumbed for
`SQLiteEventStore` — same hook reused here.
## 5. Bitcoin P2P protocol — minimum we implement
@@ -274,23 +317,35 @@ mirroring Bitcoin Core's own historical checkpoints, plus one **release-pinned
checkpoint** updated at app build time near the current tip. Sync must agree
with all checkpoints — disagreement aborts and bans the peer.
## 7. Bundled data
## 7. Bundled data — none. Just a pinned checkpoint constant.
To avoid making every fresh install download 75 MB:
The point of this work is "verify against the Bitcoin network, not a trusted
party," so we ship no headers in the APK. Instead:
- App ships an **assets file** `bitcoin_headers_<height>.bin`: raw 80-byte
headers from genesis up to a release-pinned height (≤ 1 month before build).
~72 MB — too large for Play Store delta-update friendliness.
- Ship instead a **compact bundled checkpoint**: just one `Checkpoint(height,
blockHash, chainWork, time)` per app release, plus the most recent ~50 K
headers (~4 MB) so users can verify any attestation from the past ~year
immediately.
- For older proofs, lazy-sync from the checkpoint *backwards* down to genesis
on demand (rare path).
- Forward sync from checkpoint to current tip is what runs at first launch
(~10 KB to a few MB depending on release age).
- **One hardcoded `PinnedCheckpoint` constant** in `bitcoin/store/PinnedCheckpoint.kt`:
`(height, blockHash, chainWork, time)`. ~80 bytes of source.
- The checkpoint is pinned ~1 month before each release and bumped via a
one-line PR per release. This is the **only** thing trusted at build time
— and it's a single block hash any user can independently verify against
any Bitcoin node before installing.
- **First-run sync** starts at the checkpoint and pulls forward to the
current tip over P2P. At a ~1-month checkpoint that's ~4,300 headers
(~340 KB on the wire, ~15 minutes' worth of `getheaders` round-trips at
worst), seconds over WiFi.
- **Pre-checkpoint heights** are not synced locally at all. OTS proofs
whose height precedes the checkpoint fall through to
`OkHttpBitcoinExplorer` via the composite. Users in strict-mode
(`localHeadersTrustOnly = true`) get a clear
"this OTS proof predates the bundled checkpoint" error and no network call.
- **Catch-up after long absence.** A device that hasn't run the app for
months will, on next sync, fast-forward from `tip_local` (its stored
best_height) to `tip_network`. Each `headers` message brings 2,000
headers; even 6 months of absence is ~26,000 headers (~2 MB) in 13 round
trips.
- **Header growth on disk** is ~4 MB raw per year (+ SQLite overhead, see §9).
This keeps APK growth at ~4 MB and first-run sync time at seconds.
APK growth: **0 bytes**. Trust delta vs the old HTTP-explorer model: one
block hash, code-reviewable in git.
## 8. Multi-peer strategy & eclipse mitigation
@@ -315,42 +370,85 @@ A single peer can lie. We protect against that with:
(`bitcoin.example.com:8333`) — same path as the existing custom explorer
URL setting in `OtsSettings.kt`.
## 9. Storage format
## 9. Storage — SQLite via `androidx.sqlite` + `BundledSQLiteDriver`
Append-only flat file plus a sparse index.
We reuse the same storage primitive Quartz already uses for `SQLiteEventStore`
(`quartz/.../nip01Core/store/sqlite/`): `androidx.sqlite` with the bundled
driver. This is KMP, ships in `androidx-sqlite-bundled` / `…-bundled-jvm`
(already a Quartz dependency), and is the project's standard storage choice.
### `headers.bin`
The complete `SQLiteEventStore` infrastructure — `BundledSQLiteDriver`,
`SQLiteConnectionPool`, `transaction { }`, `IModule` for migrations,
`QueryBuilder` helpers — is reused by `SqliteHeaderStore` and
`HeaderStoreSchema`.
Records, fixed 112 bytes each, indexed by height (record N at offset `N*112`):
### Schema
```
+0 80B raw block header
+80 32B cumulative chainwork (uint256, big-endian)
```sql
PRAGMA journal_mode = WAL;
PRAGMA synchronous = NORMAL;
PRAGMA foreign_keys = OFF;
CREATE TABLE block_header (
height INTEGER PRIMARY KEY, -- 0..tip, dense
hash BLOB NOT NULL, -- 32 B, dsha256(header), big-endian display order
header BLOB NOT NULL, -- 80 B raw header
chainwork BLOB NOT NULL -- 32 B cumulative work, big-endian
);
CREATE UNIQUE INDEX idx_block_header_hash ON block_header(hash);
CREATE TABLE chain_tip (
id INTEGER PRIMARY KEY CHECK (id = 1), -- singleton row
best_height INTEGER NOT NULL,
best_hash BLOB NOT NULL,
best_work BLOB NOT NULL
);
CREATE TABLE checkpoint_seen (
height INTEGER PRIMARY KEY, -- which checkpoints we've passed and confirmed
hash BLOB NOT NULL,
source TEXT NOT NULL -- 'pinned' | 'historical'
);
```
The block hash and target are recomputed from bytes 0..80 lazily; we don't
store them. Total at height 1M: 112 MB. We can drop this to 80 B/record
(no chainwork) if we accept recomputing chainwork on startup — TBD; default
is 112 B for fast tip selection.
Migrations live in `HeaderStoreSchema.kt` as an `IModule` (same pattern as
the event store's modules). v1 is the schema above; future additions
(prev-hash cache column, merkle-block annotations) ALTER from there.
### `tips.json`
### Sizing
A small JSON file holding:
| Item | Per record | At height 1,000,000 |
|---|---|---|
| `block_header` row (80+32+32 B payload + SQLite overhead) | ~180–220 B | ~180–220 MB |
| `idx_block_header_hash` B-tree (32 B key + rowid) | ~40 B | ~40 MB |
| `chain_tip` | 1 row | negligible |
| **Total** | | **~220–260 MB at height 1 M** |
```json
{
"best_height": 945123,
"best_hash": "00000000000000000001abc...",
"best_work": "0x000...",
"checkpoints_passed": [600000, 700000, 800000, 940000]
}
```
In practice users will start from a recent checkpoint and grow ~4 MB raw
(~10 MB stored) per year, so most installs sit at 10–40 MB. The "1 M
height" line is the long-tail worst case if someone with a multi-year-old
device's local store retroactively backfills (which we explicitly don't do
in v1 — see §15 Q1).
### Atomicity
### Atomicity & crash safety
Sync writes to `headers.bin.tmp`, fsyncs, then atomically renames or
appends-and-fsyncs in batches of 2000. Crash recovery: on startup, scan from
last fsynced offset, validate linkage, and discard partial trailing records.
- `HeadersSyncEngine` writes each `headers` batch (up to 2,000) inside a
single `transaction { }`. WAL gives us atomic-or-nothing per batch.
- `chain_tip` is updated in the same transaction as the headers it points to,
so the tip can never reference rows that didn't commit.
- A crash mid-batch leaves the WAL inconsistent; SQLite rolls it back on
next open. No custom recovery code.
- On open, `SqliteHeaderStore.verifyTipIntegrity()` re-reads the tip row and
confirms the referenced header exists. If not (unexpected DB corruption,
ransomware-like sandbox damage), we delete the db file and fall back to
re-sync from the pinned checkpoint.
### Concurrency
Same model as `SQLiteEventStore`: one connection pool, all writes go through
the sync engine on a dedicated dispatcher; reads use the pool's read slots.
OTS verifier reads are short and indexed — no contention with sync writes.
## 10. Resolving a height query (the actual `BitcoinExplorer` impl)
@@ -410,20 +508,25 @@ Extend `OtsSettings.kt`:
```kotlin
data class OtsSettings(
val customExplorerUrl: String? = null,
val customExplorerUrl: String? = null, // existing — shared via BitcoinExplorerEndpoint
// NEW:
val localHeadersEnabled: Boolean = true,
val localHeadersTrustOnly: Boolean = false, // disables HTTP fallback
val localHeadersTrustOnly: Boolean = false, // disables HTTP fallback; pre-checkpoint proofs error
val localHeadersWifiOnly: Boolean = true,
val customP2pNode: String? = null, // e.g. "myriad.example.com:8333"
val customP2pNode: String? = null, // e.g. "myriad.example.com:8333"
)
```
A short "Bitcoin verification" settings screen lets the user pick:
- Local headers (default)
- HTTP explorer only (current behavior, lighter)
- Local + HTTP fallback
- Local only (strict, may delay verification of brand-new attestations)
- **Local headers + HTTP fallback** (default) — best UX, trustless for
post-checkpoint heights, HTTP for older proofs.
- **Local headers only (strict)** — refuses to verify proofs older than the
pinned checkpoint. Most paranoid setting.
- **HTTP explorer only** — pre-local-headers behaviour. Lightest disk/CPU.
The `customExplorerUrl` field continues to feed `BitcoinExplorerEndpoint`,
so a custom HTTP Esplora is honoured both by the fallback in this feature
and by the onchain-zaps `EsploraBackend`. No new Esplora setting.
## 12. Tor integration
@@ -461,10 +564,15 @@ Each phase produces a shippable artifact with measurable behavior.
### Phase 2 — Storage layer (2–3 days)
- `HeaderStore` interface + `FileHeaderStore` actual on jvmAndroid.
- Append, get-by-height, get-by-hash (in-memory hash→height map built at
startup, ~60 MB at height 1 M — acceptable, or use a sparse on-disk index).
- Crash-recovery test.
- `HeaderStore` interface + `SqliteHeaderStore` implementation in `commonMain`
(`androidx.sqlite` is KMP, no platform shim needed).
- `HeaderStoreSchema` as an `IModule` matching the `SQLiteEventStore`
migration pattern.
- `append(batch)`, `getByHeight(h)`, `getByHash(hash)`, `tip()`.
- Reuse `BundledSQLiteDriver`, `SQLiteConnectionPool`, `transaction { }`
from `nip01Core/store/sqlite/` — no duplication.
- Crash-safety test: kill mid-transaction, reopen, confirm WAL rollback;
confirm `verifyTipIntegrity()` recovers cleanly.
### Phase 3 — P2P codec, no socket (3–5 days)
@@ -488,13 +596,16 @@ Each phase produces a shippable artifact with measurable behavior.
- Header-batch cross-validation across peers.
- Hardcoded checkpoints in `Checkpoints.kt`.
### Phase 6 — Bundled checkpoint & lazy-genesis-sync (2–3 days)
### Phase 6 — Pinned checkpoint constant (½ day)
- Build-time generation of the release-pinned checkpoint and trailing-50K
headers blob, packaged as `quartz/src/jvmAndroid/resources/bitcoin/...`.
- Forward sync from checkpoint to tip on first launch.
- Backward (genesis-direction) sync triggered only when an OTS proof requests
a height before the checkpoint.
- `bitcoin/store/PinnedCheckpoint.kt` — single hardcoded
`Checkpoint(height, blockHash, chainWork, time)`. ~80 bytes of source.
- A small Gradle task (`./gradlew bumpBitcoinCheckpoint`) that takes
`(height, hash)` arguments and writes the constant. Run by the release
process; otherwise no automation.
- No backward-sync, no bundled headers blob, no asset resources.
Pre-checkpoint heights are routed to the HTTP fallback by the composite
resolver (or refused in strict mode).
### Phase 7 — Android wiring (3–4 days)
@@ -512,9 +623,14 @@ Each phase produces a shippable artifact with measurable behavior.
### Phase 9 — Desktop & CLI integration (1–2 days)
- `desktopApp/` already inherits the explorer because it's in `quartz/`.
Confirm it works end-to-end on JVM.
- `cli/` (`amy verify-ots`) — confirm CLI sync also works headlessly.
- `desktopApp/` inherits the explorer for free (everything is in
`quartz/commonMain/`). Confirm headers.db path is plumbed via the same
data-dir hook already used by `SQLiteEventStore` on JVM. Confirm sync
runs end-to-end on the desktop app.
- `cli/` — add an `amy verify-ots <event-json>` command if not present,
wired through the same composite resolver. The CLI is a thin assembly
layer over Quartz (per `amy-expert`), so the actual sync logic is reused
as-is.
### Phase 10 — Rollout & telemetry (ongoing)
@@ -523,8 +639,9 @@ Each phase produces a shippable artifact with measurable behavior.
fallback rate. Surface in a debug screen.
- Default-on in the next release if metrics look healthy.
Total: **~25–35 engineer-days** end to end. Phases 1–4 are the bulk
(protocol + validation); phases 5–10 are integration.
Total: **~22–32 engineer-days** end to end. Phases 1–4 are the bulk
(protocol + validation); phases 5–10 are integration. Phase 6 dropped from
2–3 d to ½ d because we ship no bundled data — just one constant.
## 14. Testing strategy
@@ -544,8 +661,12 @@ Total: **~25–35 engineer-days** end to end. Phases 1–4 are the bulk
(committed as `quartz/src/jvmAndroid/test/resources/btc-handshake.bin`).
- Local-network test against a Bitcoin Core regtest node spun up in CI
(Docker). Validates full handshake + `getheaders` loop end to end.
- Crash-and-resume test: kill mid-sync, restart, confirm we pick up at the
correct height.
- `SqliteHeaderStore` round-trip on a real on-disk file (jvmTest, in a
`tmpdir`): append 5,000 headers in batches, getByHeight, getByHash,
reopen, confirm tip survives.
- Crash-and-resume test: kill the writer in the middle of a `transaction { }`,
reopen, confirm SQLite WAL rolls the partial batch back and sync resumes
from the last committed tip.
### Property tests
@@ -564,32 +685,41 @@ Total: **~25–35 engineer-days** end to end. Phases 1–4 are the bulk
| # | Risk | Mitigation |
|---|------|------------|
| R1 | First-run UX cost (1–10 min sync) puts users off | Bundled checkpoint + 50 K trailing headers means most installs need only seconds. Show progress prominently. |
| R2 | Mobile OS kills socket mid-sync | Foreground service for first run; resumable design. |
| R1 | First-run UX cost (sync from pinned checkpoint) | Pinned checkpoint ~1 month before release means ~4 K headers, ~340 KB, seconds over WiFi. Show progress in settings; sync runs in background WorkManager job anyway. |
| R2 | Mobile OS kills socket mid-sync | Foreground service for first run; resumable design (SQLite WAL handles partial batches automatically). |
| R3 | Peer feeds adversarial headers (eclipse) | Multi-peer + checkpoints + chainwork comparison. |
| R4 | Disk corruption | Atomic appends, checksum on close, fall back to re-sync from last checkpoint. |
| R5 | Tor + P2P is slow | Increased timeouts; fallback to HTTP-over-Tor explorer when sync stalls. |
| R6 | APK size growth from bundled headers | Cap bundle at most-recent ~50 K headers (~4 MB). Older headers fetched on demand. |
| R4 | Disk corruption | SQLite WAL + `verifyTipIntegrity()` on open; on detected corruption, delete db file and re-sync from pinned checkpoint. |
| R5 | Tor + P2P is slow | Increased timeouts; fallback to HTTP-over-Tor explorer (via `BitcoinExplorerEndpoint`) when sync stalls. |
| R6 | APK size growth | Zero — no bundled data ships in the APK. |
| R7 | Privacy: peers see user's IP | Tor mode supported; `version` user-agent doesn't leak Amethyst install ID. |
| R8 | Battery cost on background sync | Default to UNMETERED + BATTERY_OK constraints; back off when idle. |
| R9 | Maintenance burden — new soft-forks | Soft-forks don't affect headers-only validation (no script eval). Difficulty algorithm hasn't changed; a future change would require a code update. |
| R10 | Test flakiness if CI talks to public Bitcoin nodes | Use Bitcoin Core regtest in Docker for integration tests; public nodes only nightly. |
| R11 | SQLite store size on long-running installs | ~10 MB/year typical; ~220 MB at height 1 M which won't happen in any realistic mobile lifetime. Add a "wipe local headers" button in settings for users who want it. |
### Open questions
- **Q1.** Do we actually want backward-from-checkpoint sync, or just refuse
to verify proofs older than the bundled checkpoint and tell users to use
the HTTP fallback? Backward sync is operational complexity for a rare path.
- **Q2.** Storage: 80 B/record (recompute chainwork on startup) vs
112 B/record (cache it)? At height 1M, 32 MB difference. Probably worth
caching.
- **Q3.** Should we expose this as a Quartz public API (so 3rd-party apps
using Quartz also benefit) or keep it internal to NIP-03? Recommendation:
public — it's generally useful.
- **Q4.** Do we ship a "headers blob" download URL (HTTPS) as a faster
alternative to P2P for first-run? Trade-off: faster but adds a trust point.
Could be made cross-checked (download blob, then verify every header's PoW
locally — same trust as P2P then). Worth considering as Phase 6b.
#### Resolved 2026-05-19
- **Q1.** *Resolved — HTTP fallback only.* Pre-checkpoint OTS proofs are
routed to `OkHttpBitcoinExplorer` via the composite resolver. Strict-mode
users (`localHeadersTrustOnly = true`) get a clear error. No backward
sync code path.
- **Q2.** *Resolved — cache chainwork.* SQLite makes the choice cheap (one
extra BLOB column). Reading current tip work without a full scan matters
for chain-selection logic; the ~30 MB-at-1M cost is negligible on disk.
- **Q4.** *Resolved — no HTTPS headers blob.* The point of this feature is
trustless verification, and the pinned-checkpoint approach already gives
fast first-run. Adding a second ingestion path (signed blob over HTTPS)
doubles the test surface for marginal benefit.
#### Still open
- **Q3.** Should `LocalHeadersBitcoinExplorer` (and friends) be a Quartz
public API for 3rd-party Quartz consumers, or internal to Amethyst?
Recommendation: public — it's a clean reusable primitive. Decide before
Phase 0 lands so the package layout doesn't churn. *(See `quartz-integration`
skill for the public-API contract Quartz exposes today.)*
## 16. Migration & rollout
@@ -605,22 +735,55 @@ Total: **~25–35 engineer-days** end to end. Phases 1–4 are the bulk
- A NIP-03 attestation that verifies via Blockstream also verifies via the
local explorer (bit-for-bit identical timestamp).
- 95th-percentile cold-launch verification of a recent attestation: ≤2 s.
- 95th-percentile first-run sync (post-checkpoint): ≤30 s on WiFi.
- 95th-percentile first-run sync (from pinned checkpoint to tip): ≤30 s on
WiFi for a checkpoint ≤2 months old.
- No new crash-rate regression in the verification path.
- App size growth: ≤5 MB.
- App size growth: **0 MB** (no bundled data).
- A user with HTTP explorer disabled can still verify any post-checkpoint
attestation.
## 18. Out of scope (explicit)
- BIP-157/158 compact filters
- BIP-37 merkleblock or full-block fetch
- **Trustless verification of NIP-BC onchain zaps** — needs merkleblock or
full-block fetch on top of this stack. See §20 for the follow-up plan.
- **Trustless wallet operations** for NIP-BC onchain zap *sending* —
needs UTXO-by-address and broadcast paths that headers can't provide.
Stays on user-configured Esplora.
- Lightning, BIP-32 wallet, mempool monitoring
- Verifying NIP-03 attestations against altcoin/Liquid timestamps (not in
the spec)
- Replacing the calendar-server upgrade path (`OtsState.upgrade()`) —
unrelated, those are simple HTTPS calls and stay as-is
## 19. References
## 19. Follow-up: trustless NIP-BC onchain zap verification
*Not in this plan. Tracked here so the design choices above stay aligned.*
The shipped `OnchainZapVerifier` (`quartz/.../nipBCOnchainZaps/verify/`)
trusts `OnchainBackend.getTx(txid)` — i.e. Esplora — for both
tx-existence-in-a-block and the output values used to compute
`verifiedSats`. A malicious Esplora can fabricate or suppress zaps.
Closing that gap is a **separate plan** that builds on this one:
- **Foundation reused:** `BitcoinPeer`, `PeerPool`, `HeaderStore`,
`HeaderValidator`, `LocalHeadersBitcoinExplorer`.
- **New work (rough ~10–15 days on top of S1):** BIP-37 `filterload` /
`merkleblock` / matched-`tx` handling, or `getdata MSG_BLOCK` + stream
parse; partial-merkle-tree verification; new `OnchainBackend` variant
that asks the local stack for the inclusion proof and the matched tx
while keeping Esplora for UTXO listing / broadcast / fee estimates on
the *send* side.
- **Net win:** zap-receive verification stops trusting Esplora. Zap-send
still needs an address-indexed server (no headers-only equivalent
exists), so that part of the trust story is unchanged.
- **Filename:** `quartz/plans/YYYY-MM-DD-onchain-zap-merkle-proofs.md`,
written once S1 has shipped and we have real numbers on first-run sync,
store size, and battery cost.
## 20. References
- NIP-03: https://github.com/nostr-protocol/nips/blob/master/03.md
- OpenTimestamps spec: https://github.com/opentimestamps/python-opentimestamps
@@ -632,6 +795,16 @@ Total: **~25–35 engineer-days** end to end. Phases 1–4 are the bulk
- `quartz/src/commonMain/kotlin/com/vitorpamplona/quartz/nip03Timestamp/ots/BitcoinExplorer.kt`
- `quartz/src/commonMain/kotlin/com/vitorpamplona/quartz/nip03Timestamp/ots/BlockHeader.kt`
- `quartz/src/commonMain/kotlin/com/vitorpamplona/quartz/nip03Timestamp/ots/attestation/BitcoinBlockHeaderAttestation.kt`
- `quartz/src/commonMain/kotlin/com/vitorpamplona/quartz/nip03Timestamp/ots/OtsBlockHeightCache.kt`
- `quartz/src/jvmAndroid/kotlin/com/vitorpamplona/quartz/nip03Timestamp/okhttp/OkHttpBitcoinExplorer.kt`
- `amethyst/src/main/java/com/vitorpamplona/amethyst/model/nip03Timestamp/TorAwareOkHttpOtsResolverBuilder.kt`
- `amethyst/src/main/java/com/vitorpamplona/amethyst/model/nip03Timestamp/OtsSettings.kt`
- `amethyst/src/main/java/com/vitorpamplona/amethyst/model/nip03Timestamp/BitcoinExplorerEndpoint.kt`
- Storage precedent (reused as-is):
- `quartz/src/commonMain/kotlin/com/vitorpamplona/quartz/nip01Core/store/sqlite/SQLiteEventStore.kt`
- `quartz/src/commonMain/kotlin/com/vitorpamplona/quartz/nip01Core/store/sqlite/IModule.kt`
- `quartz/src/commonMain/kotlin/com/vitorpamplona/quartz/nip01Core/store/sqlite/SQLiteConnectionExt.kt`
- `quartz/src/commonMain/kotlin/com/vitorpamplona/quartz/nip01Core/store/sqlite/SQLiteConnectionPool.kt`
- Follow-up area:
- `quartz/src/commonMain/kotlin/com/vitorpamplona/quartz/nipBCOnchainZaps/verify/OnchainZapVerifier.kt`
- `quartz/src/commonMain/kotlin/com/vitorpamplona/quartz/nipBCOnchainZaps/chain/OnchainBackend.kt`