docs(quartz): add interop-test-vectors section to local-headers-explorer plan
Every consensus-relevant layer of the planned headers explorer (BlockHeader80 parser, DifficultyTarget compact↔target, CalculateNextWorkRequired retarget, MedianTimePast, header validator end-to-end, P2P wire codecs, reorg/chain selection, OTS proofs) is pinned to upstream test vectors committed under quartz/src/commonTest/resources/bitcoin/, matching the existing nip44.vectors.json / bip39.vectors.json / mls/*.json pattern. The single highest-value test is a nightly differential check asserting LocalHeadersBitcoinExplorer.blockHash(h) == OkHttpBitcoinExplorer.blockHash(h) for every height in [checkpoint, tip] — any consensus drift surfaces as a disagreeing height. Maps cleanly onto the existing Phase 1/3/4/5/7/9 work, adding ~5–7 engineer-days total to the plan budget. No new top-level phase needed.
This commit is contained in:
Claude
@@ -29,6 +29,13 @@ intervening 2026-05-14 onchain-zaps work:
|
||||
- **`BitcoinExplorerEndpoint`** (`amethyst/.../model/nip03Timestamp/`) is
|
||||
now the shared Esplora-URL resolver for OTS and onchain zaps. The
|
||||
composite resolver builder must read from it for the HTTP fallback path.
|
||||
- **Interop test vectors** (§14a) added — every consensus-relevant layer
|
||||
(header parser, compact-target, retargeter, P2P codec, OTS proofs) is
|
||||
pinned to upstream vectors committed under
|
||||
`quartz/src/commonTest/resources/bitcoin/`. A nightly differential test
|
||||
asserts `LocalHeadersBitcoinExplorer.blockHash(h)` ==
|
||||
`OkHttpBitcoinExplorer.blockHash(h)` for every height in
|
||||
`[checkpoint, tip]`, catching any consensus-rule drift end-to-end.
|
||||
|
||||
## 1. Motivation
|
||||
|
||||
@@ -681,6 +688,93 @@ Total: **~22–32 engineer-days** end to end. Phases 1–4 are the bulk
|
||||
- Tor on — sync completes (slowly) over Orbot.
|
||||
- Disk-full simulation — fail gracefully, don't corrupt store.
|
||||
|
||||
## 14a. Interop test vectors
|
||||
|
||||
Consensus code without external test vectors is "trust me." Every layer
|
||||
that could disagree with the rest of the Bitcoin network gets pinned to
|
||||
authoritative upstream vectors, mirroring how `nip44.vectors.json`,
|
||||
`bip39.vectors.json` and the `mls/*.json` set are already organised in
|
||||
`quartz/src/commonTest/resources/`. The onchain-zaps work uses the same
|
||||
discipline (BIP-341 wallet test vectors at every layer of
|
||||
`nipBCOnchainZaps/taproot/` + `psbt/`).
|
||||
|
||||
### Sources, by component
|
||||
|
||||
| Component | Source(s) | Format | Storage | Effort | Priority |
|
||||
|---|---|---|---|---|---|
|
||||
| **`BlockHeader80` parse/serialize** | Genesis (0), block 1, block 209999 (last pre-halving), 481824 (first post-segwit), 709631 (last pre-taproot), one recent | JSON: `[{height, hash, headerHex, version, prevHash, merkleRoot, time, bits, nonce}, …]` | commit `commonTest/resources/bitcoin/known_headers.json` | 1 d | **High** |
|
||||
| **`DifficultyTarget` compact↔target** | bitcoinj `UtilsTest.testCompactBitsToBigInteger`, btcd `chaincfg/chainhash/compact_test.go`. Vectors: `0x1d00ffff`, `0x1b0404cb`, `0x170398bb`, overflow + negative-bit edges | JSON pairs `(compact, targetHex)` | commit `bitcoin/compact_target.json` | ½ d | **High** |
|
||||
| **Retarget (`CalculateNextWorkRequired`)** | Mainnet retarget heights 2016, 4032, 32256, 60480, ~600000 — extract `(prev_retarget_time, current_time, prev_bits, expected_new_bits)` from any full node | JSON list of retarget cases | commit `bitcoin/retargets.json` (~50 entries, ~5 KB) | 1 d | **High** |
|
||||
| **`MedianTimePast`** | Any 11 consecutive mainnet headers synthesised from `known_headers.json` | inline test | inline | ½ d | Medium |
|
||||
| **HeaderValidator end-to-end on real chain** | First 2016 mainnet headers (~160 KB; covers genesis + first retarget). Larger 50 K-header set lazy-fetched in nightly CI from a pinned-hash mirror, not committed | binary blob | commit `bitcoin/first_2016_headers.bin` | 1–2 d | **High** |
|
||||
| **P2P wire codec — `version`/`verack`/`ping`/`pong`** | btcd `wire/msg*_test.go` has canonical byte strings; Bitcoin Wiki "Protocol documentation" has annotated hex | hex strings in JSON | commit `bitcoin/p2p_messages.json` | 1 d | **High** |
|
||||
| **P2P `getheaders` / `headers` round-trip** | Capture against a real bitcoind once, ~50 KB | raw bytes | commit `bitcoin/p2p_capture.bin` | 1 d | **High** |
|
||||
| **DNS seed list** | Bitcoin Core `chainparams.cpp` (current 9 seeds) | inline constant | n/a | <¼ d | Low |
|
||||
| **Hardcoded fallback IPs** | Bitcoin Core `chainparamsseeds.h` (auto-generated by `contrib/seeds/`) | binary blob | commit `bitcoin/chainparamsseeds.bin` | ½ d | Medium |
|
||||
| **Reorg / chain selection** | Synthetic 2-branch fork — verify higher-cumulative-chainwork wins | inline builder | inline | 1 d | **High** |
|
||||
| **End-to-end OTS proofs** | `python-opentimestamps/tests/test_data/*.ots` + the 3 wild events already inline in `OtsTest.kt`. Import 5–10 more `.ots` for variety (calendar operators, eras) | `.ots` binaries + JSON manifest of expected timestamps | commit `commonTest/resources/ots/*.ots` | 1 d | **High** |
|
||||
| **NIP-03 1040 events from real relays** | Scrape any relay for `kind:1040`; pick a diverse set across calendar operators and years | inline JSON | extend `OtsTest.kt` | ½ d | Medium |
|
||||
| **Differential test: local vs HTTP** | For every `h ∈ [checkpoint, tip]`: assert `LocalHeadersBitcoinExplorer.blockHash(h)` == `OkHttpBitcoinExplorer.blockHash(h)`. Catches consensus drift end-to-end | runtime check, nightly CI | no fixture | 1 d | **High** |
|
||||
| **Regtest end-to-end** | Bitcoin Core in Docker — mine N blocks, sync, assert tip matches | infrastructure | CI script | 2 d | Medium |
|
||||
| **Adversarial peer** | Synthetic peer that lies about retarget, replays old headers, drops mid-batch, sends 2,001-headers messages. Validates ban + reconnect | inline test harness | inline | 2 d | **High** |
|
||||
|
||||
### Committed layout
|
||||
|
||||
```
|
||||
quartz/src/commonTest/resources/bitcoin/
|
||||
├── README.md # provenance + bump procedure for each file
|
||||
├── known_headers.json # ~10 hand-picked mainnet headers across history
|
||||
├── compact_target.json # ~20 compact-bits ↔ target pairs
|
||||
├── retargets.json # ~50 retarget cases from historical mainnet
|
||||
├── first_2016_headers.bin # ~160 KB; genesis + first retarget cycle
|
||||
├── p2p_messages.json # hex of canonical version/verack/ping/pong
|
||||
├── p2p_capture.bin # captured handshake + getheaders/headers (~50 KB)
|
||||
└── chainparamsseeds.bin # current Bitcoin Core fallback IP list
|
||||
|
||||
quartz/src/commonTest/resources/ots/
|
||||
├── manifest.json # {filename → expected unix timestamp | "pending"}
|
||||
└── *.ots # 5–10 fixtures from python-opentimestamps
|
||||
```
|
||||
|
||||
### Provenance discipline
|
||||
|
||||
The `README.md` next to each fixture records, in this order:
|
||||
|
||||
1. **Where it came from** — upstream URL + commit SHA.
|
||||
2. **How to regenerate** — a single command, no manual steps.
|
||||
3. **What it tests** — one sentence.
|
||||
4. **When to bump** — e.g. *"retargets.json is stable; never bumps."* /
|
||||
*"chainparamsseeds.bin bumped each Bitcoin Core major release."*
|
||||
|
||||
Same pattern as the header comment of `nip44.vectors.json`. New committers
|
||||
should never have to ask "where did this 32-byte blob come from."
|
||||
|
||||
### Phase mapping
|
||||
|
||||
These test imports map cleanly onto the existing phases — total extra
|
||||
effort **~5–7 engineer-days** spread across the work, no new top-level
|
||||
phase needed:
|
||||
|
||||
| Phase | Vectors consumed |
|
||||
|---|---|
|
||||
| **Phase 1** (header parsing & validation, no I/O) | `known_headers.json`, `compact_target.json`, `retargets.json`, `first_2016_headers.bin` |
|
||||
| **Phase 3** (P2P codec, no socket) | `p2p_messages.json`, `p2p_capture.bin` |
|
||||
| **Phase 4** (single-peer sync) | adversarial-peer harness + regtest-in-Docker |
|
||||
| **Phase 5** (multi-peer & eclipse) | `chainparamsseeds.bin` |
|
||||
| **Phase 7** (Android wiring) | differential test against `OkHttpBitcoinExplorer` |
|
||||
| **Phase 9** (CLI / Desktop) | OTS `.ots` fixtures via `amy verify-ots --explorer=local` |
|
||||
|
||||
### Why the differential test matters most
|
||||
|
||||
The single highest-value test in the list is the differential one: for
|
||||
every height between the pinned checkpoint and the current tip, the local
|
||||
explorer's answer must equal the HTTP explorer's answer. Run it nightly in
|
||||
CI. Any consensus-rule drift — a wrong retarget formula, a missed
|
||||
overflow case in compact-target decoding, a partial-merkle bug — will
|
||||
surface as a height where the two disagree, and the height is the bug
|
||||
report. This is the test that catches the bug nobody thought to write
|
||||
a unit test for.
|
||||
|
||||
## 15. Risks & open questions
|
||||
|
||||
| # | Risk | Mitigation |
|
||||
|
||||
Reference in New Issue
Block a user