Sync Is Header-Graph Acquisition; Bodies Are Deferred

Status: historical rationale. The shipped form is the weighed-header log, bounded header pages, HeaderSync, and BodyPipeline described in architecture and body acquisition. Names and wire sketches below predate that implementation.

Supersedes the earlier “ordered stream” framing of this document. The stream framing solved the right diagnosis (below-the-tip is not the live edge) with a heavier mechanism than needed. The model here is simpler and composes with two designs already in the repo: weight-first-acquisition (deferred execution) and cashew’s targeted retrieval.

Model

Any peer may lie, withhold, stall, serve valid-but-irrelevant data, or be honestly pruned; there is no honest-majority assumption among peers, and security comes from per-item verification plus objectively comparable cumulative work, under partial synchrony — in which a slow peer and a stalling peer are indistinguishable, so rotation is a local timing policy that never implies blame. The one non-peer trust edge is the receiver’s own co-hosted immediate parent level, which alone answers genesis and parent-state continuity questions, in-process.

The insight

A block has two independent halves, and consensus only ever needed the first:

• A header — the block’s cashew root node: parent CID, target, nextTarget, height, timestamp, nonce, and the CIDs of its sub-bodies (transactions, state roots, child index). One targeted content-addressed retrieval. Proof of work is computable from the root alone (the PoW preimage hashes the inline scalars plus the CID strings of the references, never their contents). The consensus graph is already header-only: ConsensusBlockInput/submitBlock/fork choice read the header fields and the claimed state CIDs, and never the body or materialized state.
• A body — transactions, the executed state transition, the children trie. Needed to validate the block (re-execute and check the declared postState) and to use it (build on its tip, serve its state).

So a block can count for fork choice on its header alone — possess the root, verify PoW, its work is real — while its body is fetched and executed later, and only if the block turns out to matter. Below the tip, most accepted blocks are losing siblings (measured ~74% on the live child): they are weighed from headers and their bodies are never retrieved or executed.

This is exactly weight-first-acquisition’s two tiers — weighed (possess + structurally verify, work counts) and validated (executed, may be acted on) — with the weighed tier resolved at header granularity. Headers-first is the download half; deferred execution is the execution half; together they are the whole model. Neither is a foreign graft — both are already the design’s own.

What sync becomes

Sync stops being a subsystem. It is: acquire headers into the weight graph; let fork choice run on them; retrieve and execute a body only when its fork is a canonical candidate. The height-blind sweeps, the per-block evidence solicitation fired from inside failed imports, and the single import slot shared with live traffic all exist because sync today acquires and executes whole blocks through the live-edge machinery. When the weighed tier is header-only, that machinery is not needed for bulk: sync unifies with the live edge, and root sync and child sync are the same operation — the “unified adopt” principle.

1. Establish the start: common-ancestor negotiation

Sync begins by negotiating a common ancestor, not at the receiver’s frontier. The receiver offers a locator — its own accepted canonical-chain CIDs newest-first at exponentially widening gaps, genesis last — and the peer returns the highest entry on its canonical chain plus the headers forward from it. A proposed start is valid only because it is one of the receiver’s own accepted blocks, so negotiation can never rewind a receiver past its verified history. Three outcomes, distinct on the wire: a shared ancestor with headers forward (stream), a shared ancestor with nothing forward (genuinely caught up), or no shared ancestor at all (disjoint retention — end this peer, rotate, never punish). An empty page is never again conflated with “caught up.” This is the Bitcoin block-locator convention, and it is the mandatory, SOTA-shared part of the design regardless of everything below.

Negotiation and the tip announcement only ever describe a peer’s canonical chain; fork choice weighs subtrees, so a receiver also needs the losing forks. The header graph is fully determined by its leaves plus parent links (every root node carries its parent CID), so a receiver pulls a peer’s frontier — one accepted-leaves page (the most recently imported leaves), no cursor, no retry state, correlated to the one request sent — once per session, at the moment it is at the live edge with respect to that peer: the peer’s tip within two blocks of the receiver’s acquired (weighed-inclusive) tip, whether at hello for a caught-up peer or when range sync toward it completes. Each unknown leaf’s ordinary predecessor walk then reassembles its short ancestry down to known history — short by construction, because the canonical chain up to the edge is already held; pulled while deep, every leaf would descend the whole gap in competition with range sync. Announced gaps beyond the live edge (more than two blocks ahead) take the negotiated range sync; the predecessor walk serves only the true live edge and the frontier’s leaves. Acquisition measures gaps, the paging window and the locator against the acquired tip; only the act-on surfaces read the validated tip.

2. Acquire the header graph and weigh it

From the common ancestor, retrieve block headers (targeted root retrievals) and, per header, verify proof of work. A verified header’s work enters fork choice immediately — no body, no execution. Headers may be acquired from many peers in parallel and out of order; a header’s place is its parent CID, so assembly is trivial. Fork choice ranks whole subtrees over this header/weight graph exactly as it does today (it is already state-blind). Every network-sourced block — live gossip, frontier leaves, range-sync pages and the predecessor walks they seed — enters on the weighed tier; only a block this node produced itself is executed eagerly.

For a root chain a header’s own PoW is its work, self-contained. For a child chain a header’s weight is inherited — it is the securing work of the parent grind that committed to it, which needs the securing proof. That proof is not fetched per child block: the child’s securing weight is regenerated locally from the parent carriers the node holds (“proofs are regenerated from the retained closure when it exists”). So the hierarchy syncs top-down — acquire the Nexus header graph first; a child header is then weighed against the Nexus carriers already held. Regenerating a securing proof does read parent carrier sub-bodies (the child index of the committing carrier), so child weighing is heavier than root weighing, but far cheaper than a per-block child-evidence download, and it is the carrier owner’s own retained data.

3. Retrieve and execute a body only on candidacy

Fork choice selects a branch from the header graph. Only then are that branch’s bodies retrieved (independently, per block, in parallel) and executed forward from the last validated ancestor — the validated tier. Losing forks are weighed and never touched. A body that is a canonical candidate but whose parts are not yet available is an availability gap: retried indefinitely, never a verdict, and the node keeps acting on its last validated tip meanwhile. Execution that completes and fails — a deterministic mismatch of the declared postState, or a committed validity rule — records an invalidity exclusion: the proven-invalid subtree keeps its weight, but this chain’s canonical descent never steps into the excluded block, so nothing at or below it is selected, extended, or attested (spec §9.9).

The data-availability linchpin

Header-weight ranks; only validated blocks are acted on — built upon, served, exported, asserted as head (the acted-on set is enumerated in weight-first-acquisition). A miner can publish a heavy header chain (real work) and withhold its bodies; it ranks first but is never acted on — the node keeps building on its act-on tip, the heaviest executed tip, and retries the missing bodies as an availability gap. The act-on tip is found by descending from genesis and, at each fork, following the heaviest child that is executed (heaviest by its whole subtree’s header work, unexecuted descendants included); it stops where no child is executed. The withheld chain keeps all its weight and is never marked: it cannot be stepped into because stepping requires execution. Honest blocks built on the act-on tip therefore form a chain, which outweighs K withheld blocks after K+1 of its own; released before then, the withheld chain is executed and wins. This is the same withholding surface as a secret-then-released heavy chain today (publishing the headers early buys nothing over keeping the chain secret); header-weight adds no new attack, provided “rank on headers, act only on validated” is implemented exactly. Get it wrong and a heaviest-but-invalid or heaviest-but-unavailable path could be acted on, or nodes with different body availability could split.

Boundaries

• The consensus graph is header-only. ConsensusBlockInput excludes the body and state. Import in header mode (ImportMode.header) emits a weighed block fact from root + PoW with the declared postState recorded as an unverified claim. Execution mode (ImportMode.execution) later executes an already-weighed block: success records the materialized post-state, the validated marker; a completed deterministic failure records an invalidity-exclusion fact. Spec §9.9 states the rule: “accepted” means weighed; validity is a second recorded, deterministic judgment; continuity is attested on the executed-from-genesis frontier; exclusion removes no weight — the descent never steps into an excluded block — and is not pruning: excluded facts remain served.
• Deferral never comes without the exclusion seam. Weight enters fork choice pre-execution only because a proven-invalid subtree can be excluded from what the chain acts on.
• Availability never judges. Failure to obtain a body is an availability gap, retried forever, excluding nothing. Only a completed deterministic check records invalidity. A block is therefore in one of three states, and they are never conflated: weighed and unvalidated (its body may or may not be held; its work counts; it has no verdict), validated (executed, post-state checked), or excluded (executed and found invalid: still weighed, never the step the descent takes). An unavailable block is the first, not the third.
• Wire compatibility is additive. The strict-canonical wire evolves by new message types, never mutated ones. Peers that do not speak the new topics keep serving the existing pages; receivers fall back — slow, never wrong.
• Possession stays ungated. Serving a block whose bytes match their CID is never a fault, validated or not; possession/serving surfaces (forwardCanonicalRange, by-CID reads) are not gated on validation.
• The unavailable tail is unsyncable by the same rule as today. A chain whose facts no parent tracks and no peer retained cannot be synced; the model changes the cost of availability, never its existence.

Composition

The three designs compose: eviction is weight-preserving (operator-finality), acquisition is header-first with a negotiated start (this document), and execution is deferred to load-bearing blocks (weight-first-acquisition). Acquire headers, rank on work, execute exactly the chain that matters.

---

Source: lattice-node/docs/design/bulk-sync-stream.md at 9abd15b1bb05