zzstoatzz.io
about things
add inductive proof notes and sync verification reference
deep dive into atproto's commit diff verification algorithm,
zlay's relay integration (observation mode), and SDK affordance
analysis — concluded loadCommitFromCAR is already correctly public.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
+22
protocols/atproto/inductive-proof/README.md
+22
protocols/atproto/inductive-proof/README.md
···
1
+
# inductive proof chain
2
+
3
+
deep notes on atproto's commit verification mechanism and the general patterns it surfaces for the zat SDK.
4
+
5
+
## context
6
+
7
+
a relay receives ~2,750 PDS streams of signed commits. it needs to verify each commit is a valid state transition without storing full repo state. the solution is an inductive proof: verify each diff against the previous commit's MST root, chaining trust back to a known-good state.
8
+
9
+
zat implements the core verification. zlay (relay) is the primary downstream consumer.
10
+
11
+
## files
12
+
13
+
- [algorithm.md](./algorithm.md) — the inversion algorithm, step by step, with code references
14
+
- [relay-integration.md](./relay-integration.md) — how zlay uses these APIs in production, what works, what's still being built
15
+
- [sdk-affordances.md](./sdk-affordances.md) — general patterns that could become SDK primitives
16
+
17
+
## see also
18
+
19
+
- [sync-verification.md](../sync-verification.md) — spec-level overview (fields, wire format, error cases)
20
+
- [firehose.md](../firehose.md) — event stream basics
21
+
- zat source: `src/internal/repo/repo_verifier.zig`, `src/internal/mst/mst.zig`
22
+
- zlay source: `src/validator.zig`, `src/frame_worker.zig`, `src/event_log.zig`
+147
protocols/atproto/inductive-proof/algorithm.md
+147
protocols/atproto/inductive-proof/algorithm.md
···
1
+
# the inversion algorithm
2
+
3
+
how `verifyCommitDiff` proves a commit is a valid state transition.
4
+
5
+
## the claim being verified
6
+
7
+
given:
8
+
- a signed commit with MST root `data_cid` (the "after" state)
9
+
- a CAR containing the commit block + changed MST nodes + new records
10
+
- a list of operations (create/update/delete with paths and CIDs)
11
+
- a previous MST root `prev_data` (the "before" state, from the preceding commit)
12
+
13
+
prove: the operations, applied to `prev_data`, produce `data_cid`.
14
+
15
+
the trick: instead of *applying* ops forward (which would require the full previous tree), *invert* them on the new tree and check if you get back to the old root.
16
+
17
+
## why inversion instead of forward application
18
+
19
+
forward application would require:
20
+
1. the full previous MST (potentially millions of entries)
21
+
2. applying each op to produce the new tree
22
+
3. comparing the new root against `data_cid`
23
+
24
+
inversion requires:
25
+
1. only the partial MST from the CAR (changed nodes only)
26
+
2. reversing each op on the new tree
27
+
3. comparing the inverted root against `prev_data`
28
+
29
+
the CAR already contains the new partial tree. unchanged subtrees become stubs — their CIDs are known but their contents aren't loaded. this is the key insight: you only need blocks for the parts that changed.
30
+
31
+
## step by step
32
+
33
+
source: `repo_verifier.zig:387-469`
34
+
35
+
### 1. parse CAR, extract commit
36
+
37
+
```
38
+
loadCommitFromCAR(blocks) → { commit, unsigned_bytes, repo_car }
39
+
```
40
+
41
+
the commit is CBOR with fields: `did`, `version`, `data` (MST root CID), `rev`, `prev`, `sig`. the unsigned bytes are re-encoded without `sig` for signature verification.
42
+
43
+
### 2. verify signature
44
+
45
+
```
46
+
verify(unsigned_bytes, commit.sig, public_key) → ok | error
47
+
```
48
+
49
+
proves the commit came from the account's signing key. this is independent of MST — it proves authorship.
50
+
51
+
### 3. load partial MST from CAR
52
+
53
+
```
54
+
Mst.loadFromBlocks(repo_car, commit.data_cid) → partial tree
55
+
```
56
+
57
+
source: `mst.zig:459-480`
58
+
59
+
walks from the root block, loading nodes whose blocks exist in the CAR. blocks not found become `ChildRef.stub` — a CID placeholder. the result is a tree where:
60
+
- changed paths have fully loaded nodes
61
+
- unchanged subtrees are stubs (just their hash)
62
+
63
+
### 4. copy tree for inversion
64
+
65
+
```
66
+
tree.copy() → inverted
67
+
```
68
+
69
+
deep clone. stubs stay as stubs. the original tree is preserved.
70
+
71
+
### 5. normalize operations
72
+
73
+
```
74
+
normalizeOps(msg_ops) → sorted_ops
75
+
```
76
+
77
+
source: `mst.zig:773-800`
78
+
79
+
- sort: **deletions first**, then by path lexicographically
80
+
- reject duplicates (same path twice → `DuplicatePath`)
81
+
82
+
deletions first matters: if you invert a create before a delete on related paths, you might touch a node that should already be gone. ordering ensures deterministic application.
83
+
84
+
### 6. invert each operation
85
+
86
+
source: `mst.zig:804-820`
87
+
88
+
for each op in sorted order:
89
+
90
+
| forward op | inverse action | verification |
91
+
|---|---|---|
92
+
| **create**(path, cid) | `deleteReturn(path)` | removed value must equal `cid` |
93
+
| **update**(path, new_cid, prev_cid) | `putReturn(path, prev_cid)` | displaced value must equal `new_cid` |
94
+
| **delete**(path, prev_cid) | `putReturn(path, prev_cid)` | path must not already exist (displaced == null) |
95
+
96
+
each step is self-checking. if any assertion fails → `InversionMismatch`.
97
+
98
+
### 7. compute inverted root
99
+
100
+
```
101
+
inverted.rootCid() → cid
102
+
```
103
+
104
+
traverses the (now-inverted) tree bottom-up. loaded nodes are serialized as DAG-CBOR and SHA-256 hashed. stubs contribute their known CIDs directly — this is safe because we trust the previous state.
105
+
106
+
if inversion needs to descend into a stub (operation touches an unchanged subtree), it fails with `PartialTree` — the CAR didn't include enough context.
107
+
108
+
### 8. compare
109
+
110
+
```
111
+
inverted_root == prev_data → valid transition
112
+
```
113
+
114
+
if they match, the operations *fully explain* the state change from the old root to the new root. no hidden mutations, no missing ops, no reordering.
115
+
116
+
## what stubs mean for the proof
117
+
118
+
stubs are the mechanism that makes partial verification possible:
119
+
120
+
```
121
+
ChildRef = union(enum) {
122
+
none, — no child (leaf boundary)
123
+
node: *Node, — loaded from CAR, fully traversable
124
+
stub: cbor.Cid, — CID known, content trusted
125
+
}
126
+
```
127
+
128
+
when computing `rootCid()`:
129
+
- `node` → serialize, hash, produce CID
130
+
- `stub` → use the CID as-is (it's trusted from previous verification)
131
+
- `none` → encoded as null in CBOR
132
+
133
+
the proof chain holds because each stub was once a verified node in a previous commit. by induction, every stub's CID is correct.
134
+
135
+
## error taxonomy
136
+
137
+
| error | meaning | response |
138
+
|---|---|---|
139
+
| `SignatureVerificationFailed` | wrong key or tampered commit | reject, re-resolve DID |
140
+
| `PrevDataMismatch` | ops don't explain the transition | chain broken, re-sync |
141
+
| `InversionMismatch` | individual op inconsistent with tree state | reject commit |
142
+
| `PartialTree` | CAR missing blocks needed for inversion | incomplete data |
143
+
| `DuplicatePath` | same path appears in multiple ops | malformed commit |
144
+
| `InvalidMstNode` | CBOR structure of MST node is wrong | corrupted data |
145
+
| `InvalidCommit` | commit object malformed or wrong version | corrupted data |
146
+
147
+
the first three are the interesting ones — they distinguish between identity fraud (signature), history fraud (prevData), and operation fraud (inversion).
+108
protocols/atproto/inductive-proof/relay-integration.md
+108
protocols/atproto/inductive-proof/relay-integration.md
···
1
+
# relay integration
2
+
3
+
how zlay uses the sync 1.1 APIs from zat, as of march 2026. zlay is ~4 days old.
4
+
5
+
## current state
6
+
7
+
zlay is on **zat v0.2.10**. the sync 1.1 verification is wired but deployed in **observation mode** — chain breaks are logged and counted, not enforced.
8
+
9
+
## the pipeline
10
+
11
+
```
12
+
subscriber (reader thread)
13
+
→ header decode, cursor tracking
14
+
→ submit raw frame to thread pool
15
+
16
+
frame_worker (pool worker)
17
+
→ CBOR decode payload
18
+
→ rev clock check (reject future timestamps beyond 5min skew)
19
+
→ chain continuity check (log-only):
20
+
since vs stored rev
21
+
prevData vs stored data_cid
22
+
→ dispatch to validator
23
+
24
+
validator
25
+
→ DID cache lookup (miss → queue background resolve, skip frame)
26
+
→ verifyCommitCar(blocks, public_key, {verify_mst: false})
27
+
OR verifyCommitDiff(blocks, ops, prev_data, public_key) [behind config flag]
28
+
→ return (data_cid, commit_rev)
29
+
30
+
event_log
31
+
→ persist frame to disk
32
+
→ conditional upsert: UPDATE ... WHERE rev < new_rev
33
+
→ broadcast to consumers
34
+
```
35
+
36
+
## what's working
37
+
38
+
**chain continuity detection** (frame_worker.zig, subscriber.zig):
39
+
- compares incoming `since` against stored `rev`
40
+
- compares incoming `prevData` CID against stored `data_cid`
41
+
- increments `relay_chain_breaks_total` prometheus counter
42
+
- log-only — commits still flow through
43
+
44
+
**conditional state upsert** (event_log.zig):
45
+
```sql
46
+
INSERT INTO account_repo (uid, rev, commit_data_cid)
47
+
VALUES ($1, $2, $3)
48
+
ON CONFLICT (uid) DO UPDATE
49
+
SET rev = EXCLUDED.rev, commit_data_cid = EXCLUDED.commit_data_cid
50
+
WHERE account_repo.rev < EXCLUDED.rev
51
+
```
52
+
prevents concurrent workers from rolling back state. returns whether the update actually happened.
53
+
54
+
**extractOps fix** (validator.zig):
55
+
- previously looked for separate `collection`/`rkey` fields (wrong)
56
+
- now reads `path` field and splits on `/` (matches firehose wire format)
57
+
- validates both halves: NSID for collection, rkey for record key
58
+
59
+
**future rev rejection** (frame_worker.zig):
60
+
- parses incoming rev as TID, extracts microsecond timestamp
61
+
- compares against wall clock + configurable skew (default 5 minutes)
62
+
- rejects commits claiming to be from the future
63
+
64
+
## what's not yet enabled
65
+
66
+
**full diff verification** (`verifyCommitDiff`):
67
+
- wired in validator.zig but behind `config.verify_commit_diff` flag
68
+
- disabled in production — currently all commits go through `verifyCommitCar` (signature-only, MST verification disabled)
69
+
- the observation mode lets operators measure chain break rates before strict enforcement
70
+
71
+
**resync on chain break**:
72
+
- breaks are detected and logged but no recovery action is taken
73
+
- the spec says: mark desynchronized, queue events, fetch full CAR, reconcile, replay
74
+
- this is a significant operational feature (thundering herd concerns, etc.)
75
+
76
+
## the optimistic validation pattern
77
+
78
+
zlay's approach to DID resolution creates a trust window:
79
+
80
+
1. first commit from a DID → cache miss → broadcast immediately, resolve key in background
81
+
2. subsequent commits → cache hit → verify signature
82
+
3. verification failure → evict cache, re-resolve, skip this frame
83
+
84
+
this is a deliberate trade-off: brief trust window for throughput. bounded by resolver thread count and resolution latency (~200ms per DID).
85
+
86
+
## state requirements
87
+
88
+
per-DID state for chain verification is minimal:
89
+
90
+
| field | type | purpose |
91
+
|---|---|---|
92
+
| `uid` | u64 | internal ID (from DID mapping cache) |
93
+
| `rev` | text | last verified commit TID |
94
+
| `commit_data_cid` | text | last verified MST root, multibase-encoded |
95
+
96
+
this maps directly to what the spec requires: track `rev` and `data` per repo.
97
+
98
+
## questions for SDK design
99
+
100
+
observations from watching zlay integrate:
101
+
102
+
1. **extractOps was wrong for months** — the SDK provides `MstOperation` but the firehose wire format uses a different field layout (`path` vs `collection`+`rkey`, `cid` vs `value`). should the SDK provide a firehose-aware operation parser?
103
+
104
+
2. **chain continuity is caller responsibility** — every consumer needs to track (rev, data_cid) and compare against incoming (since, prevData). this is boilerplate with subtle ordering requirements. could the SDK help?
105
+
106
+
3. **the observation-then-enforcement pattern** — zlay chose log-only first. this is sensible for any consumer. does the SDK's error-based API support this well, or does it force binary accept/reject?
107
+
108
+
4. **multibase encoding of CIDs** — zlay encodes CIDs as multibase base32lower for storage/comparison. this is a common need. the SDK has `multibase.encode` but the pattern of "extract CID from verify result, encode for storage" is repeated.
+112
protocols/atproto/inductive-proof/sdk-affordances.md
+112
protocols/atproto/inductive-proof/sdk-affordances.md
···
1
+
# SDK affordances
2
+
3
+
general patterns surfaced by studying sync 1.1 integration. the question: what could zat provide that's broadly useful without being experimental or over-specific?
4
+
5
+
## pattern 1: firehose frame parsing
6
+
7
+
**the problem**: zlay manually extracts fields from decoded CBOR `Value` maps — `getString("repo")`, `get("prevData")`, `getString("since")`, etc. this is repeated across frame_worker.zig and subscriber.zig with subtly different field names for `#commit` vs `#sync` vs `#account` events.
8
+
9
+
**what exists**: zat's `FirehoseClient` already decodes frames into typed `FirehoseEvent` structs. but zlay doesn't use it — zlay has its own subscriber that connects to individual PDSes, not upstream relays. it receives the same wire format but doesn't get the typed decode.
10
+
11
+
**the affordance**: separate the frame *parsing* from the frame *receiving*. a function like:
12
+
13
+
```
14
+
parseFirehoseFrame(header_bytes, payload_bytes) → FirehoseEvent | error
15
+
```
16
+
17
+
this is already mostly what the firehose client does internally. exposing it independently means any consumer with raw websocket frames can get typed events without adopting the full client lifecycle.
18
+
19
+
**generality**: high. any custom subscriber (relay, indexer, archiver) that handles its own websocket connection would benefit. the wire format is stable — it's the spec.
20
+
21
+
**status**: not yet justified — zlay is the only consumer (4 days old). wait for a second consumer to hit this need. "no is temporary."
22
+
23
+
## pattern 2: operation type bridging
24
+
25
+
**the problem**: the firehose wire format and the MST internal format use different representations for the same concept:
26
+
27
+
| field | firehose (`repoOp`) | MST (`Operation`) |
28
+
|---|---|---|
29
+
| path | `path` (single string) | `path` |
30
+
| new value | `cid` (nullable CID) | `value` (nullable raw bytes) |
31
+
| old value | `prev` (nullable CID) | `prev` (nullable raw bytes) |
32
+
| action | `action` string | inferred from value/prev nullability |
33
+
34
+
zlay's `extractOps` manually converts between these. it got the field names wrong for months.
35
+
36
+
**the affordance**: a conversion function:
37
+
38
+
```
39
+
repoOpToMstOperation(repo_op) → MstOperation
40
+
```
41
+
42
+
or better: have `verifyCommitDiff` accept `RepoOp[]` directly, doing the conversion internally.
43
+
44
+
**generality**: high. this is a spec-level mapping, not an implementation detail.
45
+
46
+
## pattern 3: chain state tracking
47
+
48
+
**the problem**: every consumer that verifies commits needs to:
49
+
1. store (rev, data_cid) per DID
50
+
2. compare incoming (since, prevData) against stored state
51
+
3. update state after successful verification
52
+
4. handle chain breaks (log, mark desynchronized, queue resync)
53
+
54
+
this is ~50 lines of boilerplate that's easy to get wrong (race conditions on concurrent updates, multibase encoding of CIDs for comparison, rev ordering).
55
+
56
+
**what the SDK could offer**: not a full state machine (that's too opinionated), but:
57
+
- a `ChainState` struct: `{ rev: []const u8, data_cid: Cid }`
58
+
- a `checkContinuity(stored: ChainState, incoming_since: ?[]const u8, incoming_prev_data: ?Cid) → .ok | .break_since | .break_prev_data | .first_commit`
59
+
- let the caller decide what to do on break
60
+
61
+
**generality**: medium-high. the continuity check is pure logic — no I/O, no storage assumptions. but it's also simple enough that it's borderline "just write it." the value is in being *correct* — the field comparison has encoding subtleties.
62
+
63
+
**hesitation**: this might be too thin to justify SDK surface area. maybe it belongs in documentation/examples rather than code.
64
+
65
+
## pattern 4: verify result → storage-ready output
66
+
67
+
**the problem**: both `verifyCommitCar` and `verifyCommitDiff` return result structs with raw CID bytes. consumers invariably need to:
68
+
1. multibase-encode the data CID for storage
69
+
2. extract the rev string
70
+
3. store both as the new chain state
71
+
72
+
zlay does this manually after every verify call.
73
+
74
+
**the affordance**: a method on the verify result:
75
+
76
+
```
77
+
result.chainState(allocator) → ChainState { .rev = "3jui...", .data_cid_encoded = "bafyrei..." }
78
+
```
79
+
80
+
or just make the verify results include the encoded form.
81
+
82
+
**generality**: medium. convenient, but it's two lines of code. might not clear the bar.
83
+
84
+
## pattern 5: commit parsing without verification
85
+
86
+
**the problem**: sometimes you want to extract commit metadata (DID, rev, data CID) from a CAR *without* running the full verification pipeline. zlay does this in the chain continuity check path — it needs the fields for comparison before deciding whether to verify.
87
+
88
+
**what exists**: `loadCommitFromCAR` does exactly this, but it's currently internal to the verify module.
89
+
90
+
**the affordance**: make `loadCommitFromCAR` public, or provide a lightweight `parseCommit(car_bytes) → CommitMetadata` that stops after extracting fields.
91
+
92
+
**generality**: high. indexers, archivers, and debugging tools all want commit metadata without paying for verification.
93
+
94
+
## what NOT to add
95
+
96
+
things that seem useful but are too specific or experimental:
97
+
98
+
- **stateful chain verifier with storage backend** — too opinionated about persistence. every consumer has different storage (postgres, rocksdb, sqlite, in-memory).
99
+
- **automatic resync orchestration** — involves HTTP fetching, backoff, thundering herd mitigation. this is operational, not SDK-level.
100
+
- **DID resolution caching** — zlay's LRU cache is highly tuned to its concurrency model. a generic cache would either be too simple or too complex.
101
+
- **rate limiting / admission control** — purely operational.
102
+
103
+
## summary: what seems worth pursuing
104
+
105
+
ranked by (generality × value / complexity):
106
+
107
+
1. **expose frame parsing independently** — high value, low complexity, clearly general
108
+
2. **operation type bridging** — prevents a real class of bugs, spec-level mapping
109
+
3. **make `loadCommitFromCAR` public** — zero-cost, already implemented, broadly useful
110
+
4. **chain continuity check helper** — useful but borderline; maybe docs/examples instead
111
+
112
+
the theme: **make the SDK's internals composable**. zlay doesn't use the firehose client but needs the frame parser. it doesn't need the full verify pipeline but needs commit metadata. the value is in unbundling, not in adding new features.
+175
protocols/atproto/sync-verification.md
+175
protocols/atproto/sync-verification.md
···
1
+
# sync verification (sync 1.1)
2
+
3
+
how relays and downstream services verify firehose commits without trusting the source.
4
+
5
+
source: [atproto sync spec](https://atproto.com/specs/sync), [repository spec](https://atproto.com/specs/repository), [event stream spec](https://atproto.com/specs/event-stream), [cryptography spec](https://atproto.com/specs/cryptography)
6
+
7
+
## the inductive proof chain
8
+
9
+
the core idea: instead of re-fetching the full repo to verify each change, you verify that each commit is a valid *transition* from the previous state. this only requires tracking two values per DID:
10
+
11
+
- `rev` — the TID of the last verified commit
12
+
- `data` — the MST root CID of the last verified commit
13
+
14
+
### base case
15
+
16
+
establish ground truth by fetching the full repo (via `getRepo` CAR export), verifying the MST structure, and verifying the commit signature. now you *know* the repo state is correct at this point.
17
+
18
+
### inductive step
19
+
20
+
for each subsequent `#commit` from the firehose:
21
+
22
+
1. **check chain continuity** — the event's `since` must match your stored `rev`, and `prevData` must match your stored `data`
23
+
2. **verify signature** — re-encode the commit without `sig`, SHA-256 hash, verify ECDSA
24
+
3. **MST inversion** — apply the `ops` in *reverse* against the partial MST from the CAR blocks. if the ops are complete, the resulting root CID must equal `prevData`
25
+
4. **update state** — store the new `rev` and `data`
26
+
27
+
if step 1 or 3 fails, the chain is broken → mark the repo as desynchronized and re-fetch.
28
+
29
+
## commit event fields that matter
30
+
31
+
the signed commit object (in the CAR) has:
32
+
33
+
| field | notes |
34
+
|---|---|
35
+
| `did` | account DID |
36
+
| `version` | always `3` (v1 dead, v2 legacy-compatible) |
37
+
| `data` | CID of MST root — **this is what the proof chain tracks** |
38
+
| `rev` | TID, must increase monotonically |
39
+
| `prev` | virtually always `null` in v3 — vestigial from v2 |
40
+
| `sig` | ECDSA signature over all other fields encoded as DAG-CBOR |
41
+
42
+
the firehose `#commit` event adds:
43
+
44
+
| field | notes |
45
+
|---|---|
46
+
| `since` | rev of the *preceding* commit (chain link) |
47
+
| `prevData` | MST root CID of the preceding commit (chain link) |
48
+
| `blocks` | CAR slice — only changed blocks, max 2 MB |
49
+
| `ops` | up to 200 record operations |
50
+
51
+
each `repoOp` has:
52
+
53
+
| field | notes |
54
+
|---|---|
55
+
| `action` | `create`, `update`, or `delete` |
56
+
| `path` | `<collection>/<rkey>` |
57
+
| `cid` | new record CID (null for deletes) |
58
+
| `prev` | previous record CID (for updates/deletes — required for inversion) |
59
+
60
+
**important**: `since`/`prevData` are *unsigned* — they're not in the signed commit object. but they're *verifiable* via MST inversion. the relay proves they're correct by showing the math works out.
61
+
62
+
## MST inversion (the "trick")
63
+
64
+
from the spec:
65
+
66
+
> the trick to this process is record operation inversion. `#commit` messages contain both a repo diff (CAR slice), and an array of record operations. the operations can be applied in reverse against a copy of the partial repo tree contained in the diff blocks. if the list of operations is complete, the root of the tree should be exactly that of the previous commit object of the repository.
67
+
68
+
concretely, for each op applied in reverse:
69
+
70
+
- **create** → delete that key from the new tree, verify removed CID matches `op.cid`
71
+
- **update** → put `op.prev` back, verify displaced CID matches `op.cid`
72
+
- **delete** → re-insert `op.prev`, verify key didn't already exist
73
+
74
+
the CAR contains a *partial* MST — only the nodes that changed. unchanged subtrees become "stubs" (just their CID). after inversion, the root is computed bottom-up: stubs contribute their known CIDs, loaded nodes are serialized and hashed. if it matches `prevData`, the transition is proven valid.
75
+
76
+
## what fails if data is tampered with
77
+
78
+
| tampering | detection |
79
+
|---|---|
80
+
| modified record content | CAR block hash ≠ CID |
81
+
| forged commit (wrong DID, rev, data) | signature verification fails |
82
+
| wrong MST structure | inverted root ≠ `prevData` |
83
+
| extra/missing operations | inverted root ≠ `prevData`, or inversion mismatch |
84
+
| op claims to create X but X isn't in tree | `deleteReturn` returns null |
85
+
| op touches unchanged subtree not in CAR | stub error (partial tree) |
86
+
| high-S signature malleability | low-S check rejects it |
87
+
88
+
## chain break → resync
89
+
90
+
when the chain breaks (mismatched `since`/`prevData`, or a `#sync` event):
91
+
92
+
1. mark the repo as `desynchronized`
93
+
2. queue incoming events for this DID (don't drop them)
94
+
3. fetch the full CAR — **from the upstream relay first** (not the PDS) to avoid thundering herd
95
+
4. verify and reconcile state
96
+
5. replay queued events
97
+
98
+
from the spec:
99
+
100
+
> if many services attempt to re-synchronize a repository at the same time, the upstream PDS host may be overwhelmed with a 'thundering herd' of requests. to mitigate this, receiving services should first attempt to fetch the repo CAR file from their direct upstream (often a relay instance).
101
+
102
+
## `#sync` events
103
+
104
+
sent when repo state has been reset or is ambiguous (e.g. account reactivation after data corruption). contains only the commit block, not the full repo.
105
+
106
+
> note that the repository *contents* are not included in the sync event: the `blocks` field only contains the repo commit object. downstream services would need to fetch the full repo CAR file to re-synchronize.
107
+
108
+
## `#account` events
109
+
110
+
| field | notes |
111
+
|---|---|
112
+
| `active` | whether the repo can be redistributed |
113
+
| `status` | `takendown`, `suspended`, `deleted`, `deactivated`, `desynchronized`, `throttled` |
114
+
115
+
the spec is clear: non-active accounts' content should not be redistributed. this means `listReposByCollection` should filter by active status.
116
+
117
+
> when an account status is non-`active`, the content that hosts should not redistributed includes: repository exports (CAR files), repo records, transformed records ('views', embeds, etc), blobs, transformed blobs (thumbnails, etc)
118
+
119
+
account events are **hop-by-hop** — they describe status *at the emitting service*, not globally.
120
+
121
+
## validation checklist (from spec)
122
+
123
+
what a relay **should** do for each `#commit`:
124
+
125
+
1. verify commit signature (refresh identity on initial failure)
126
+
2. verify event fields match the signed commit in `blocks`
127
+
3. verify `blocks` against `ops` and `prevData` via MST inversion
128
+
4. check `since` against stored `rev` — mismatch → out-of-sync
129
+
5. check `prevData` against stored `data` — mismatch → out-of-sync
130
+
6. ignore events with `rev ≤ stored_rev`
131
+
7. reject events with future `rev` (beyond clock drift window)
132
+
8. ignore events for non-active accounts
133
+
9. do NOT validate records against lexicons (relay-specific)
134
+
135
+
## cryptographic details
136
+
137
+
- two curves: P-256 (`secp256r1`) and secp256k1 — implementations must support both
138
+
- low-S normalization **required** for both curves
139
+
- signing: DAG-CBOR encode unsigned commit → SHA-256 hash (binary) → ECDSA sign
140
+
- the CID of a signed commit uses the *signed* DAG-CBOR encoding (codec `0x71`)
141
+
- public keys: compressed 33-byte points, multicodec-prefixed (`0x80 0x24` for P-256, `0xe7 0x01` for secp256k1), then multibase-encoded (`z` + base58btc)
142
+
143
+
## cursor semantics
144
+
145
+
sequence numbers are per-service, per-endpoint. reconnection rules:
146
+
147
+
- no cursor → start from current position
148
+
- cursor in rollback window → replay from that point
149
+
- cursor too old → info message, then replay entire rollback window
150
+
- cursor in the future → `FutureCursor` error, close connection
151
+
152
+
a relay should track both "last received seq" (for reconnection) and "high water mark" (for persistence after processing completes).
153
+
154
+
## implementations
155
+
156
+
| | zat/zlay | lightrail (fig) | collectiondir (indigo) |
157
+
|---|---|---|---|
158
+
| signature verification | yes (P-256 + secp256k1) | not yet (resolver ready) | no |
159
+
| MST inversion | yes (`verifyCommitDiff`) | not yet (MST parsing exists) | no |
160
+
| per-DID ordering | caller responsibility | `CommitDispatcher` enforces | n/a |
161
+
| prev chain tracking | yes (postgres, CAS upsert) | `RepoPrev` storage built | no |
162
+
| chain continuity checks | yes (log-only, metrics) | not yet | no |
163
+
| account status | dual status (local + upstream) | tracked, not filtered at query | append-only (no removal) |
164
+
| resync on discontinuity | not yet | architecture ready | n/a |
165
+
166
+
zat has the cryptographic and structural verification. zlay (march 2026) runs chain continuity detection in observation mode — logging breaks and counting them via prometheus, not yet enforcing. `verifyCommitDiff` is wired but behind a config flag; production uses `verifyCommitCar` (signature-only). see [inductive-proof/relay-integration.md](./inductive-proof/relay-integration.md) for details.
167
+
168
+
lightrail has the operational scheduling and recovery. collectiondir trusts the upstream relay entirely.
169
+
170
+
## see also
171
+
172
+
- [inductive-proof/](./inductive-proof/) — deep dive: algorithm, relay integration, SDK affordances
173
+
- [firehose](./firehose.md) — event stream basics, consuming events
174
+
- [data](./data.md) — repos, records, collections
175
+
- [identity](./identity.md) — DIDs, handles, key resolution