+148

docs/REPOMGR_MIGRATION.md

··· 1 1 + # Incremental Migration: Vendored repomgr → Direct `indigo/repo` 2 2 + 3 3 + ## Context 4 4 + 5 5 + The hold PDS uses a vendored copy of indigo's `repomgr` (~1450 lines in `pkg/hold/pds/repomgr.go`). Upstream, repomgr is [soft-deprecated](https://github.com/bluesky-social/indigo/pull/1102#issuecomment-2985956040) — bnewbold recommends using `indigo/repo` directly (as [cocoon](https://github.com/haileyok/cocoon) does). The vendored copy already has custom patches (PutRecord, UpsertRecord, prevData) and will continue to drift. This migration defines a clean interface, then swaps the implementation behind it. 6 6 + 7 7 + ## Phase 0: Save plan to docs, remove dead code, define interface ✅ 8 8 + 9 9 + **Goal:** Persist this migration plan as a reference doc. Shrink repomgr.go from ~1450 lines to ~750 by removing dead code. Fix import.go encapsulation. Define `RepoOperator` interface so all code accesses repomgr through it. No behavior change. 10 10 + 11 11 + **Completed 2026-02-28.** repomgr.go: 1453 → 871 lines (-40%). import.go: 189 → 88 lines (-53%). New repo_operator.go: 82 lines. 12 12 + 13 13 + ### Step 1: Save plan to docs ✅ 14 14 + - Write this plan to `docs/REPOMGR_MIGRATION.md` 15 15 + 16 16 + ### Step 2: Dead code deleted from `repomgr.go` ✅ 17 17 + - `HandleExternalUserEvent()` + `handleExternalUserEventNoArchive()` + `handleExternalUserEventArchive()` 18 18 + - `ImportNewRepo()` + `processNewRepo()` + `walkTree()` + `processOp()` + `stringOrNil()` 19 19 + - `CheckRepoSig()` 20 20 + - `TakeDownRepo()`, `ResetRepo()`, `VerifyRepo()` 21 21 + - `GetProfile()` 22 22 + - `CarStore()` 23 23 + - `NextTID()` / `nextTID()` (removed entirely — `BatchWrite` uses `rm.clk.Next()` directly) 24 24 + - `noArchive` field removed from struct 25 25 + - 12 unused imports cleaned up 26 26 + 27 27 + ### Step 3: Fixed `import.go` encapsulation ✅ 28 28 + Added `BulkUpsert()` method to `RepoManager`. Rewrote `ImportFromCAR` to call `p.repomgr.BulkUpsert()` instead of reaching into `p.repomgr.lockUser`, `p.repomgr.cs`, `p.repomgr.kmgr`, `p.repomgr.events`. Removed the 88-line `bulkImportRecords()` private method. 29 29 + 30 30 + ### Step 4: Defined `RepoOperator` interface ✅ 31 31 + 32 32 + New file: `pkg/hold/pds/repo_operator.go` — interface with 16 methods, compile-time check, shared types (`RepoEvent`, `RepoOp`, `EventKind`, `BulkRecord`). 33 33 + 34 34 + ### Step 5: Updated callers to use interface ✅ 35 35 + - `pkg/hold/pds/server.go` — `repomgr *RepoManager` → `repomgr RepoOperator`, `RepomgrRef()` returns `RepoOperator` 36 36 + - Downstream callers (`hold/server.go`, `admin/handlers_relays.go`, tests) unchanged — they go through `RepomgrRef()` which returns the interface 37 37 + - `var _ RepoOperator = (*RepoManager)(nil)` compile-time check in `repo_operator.go` 38 38 + 39 39 + ### Verification: ✅ 40 40 + - `make lint` — 0 issues (also fixed pre-existing unchecked error in `events.go`) 41 41 + - `make test` — all tests pass 42 42 + 43 43 + --- 44 44 + 45 45 + ## Phase 1: Test hardening against the interface ✅ 46 46 + 47 47 + **Goal:** Write tests against `RepoOperator` that verify current behavior while `RepoManager` is the only implementation. These become the regression safety net when swapping to the new implementation in Phase 3. 48 48 + 49 49 + **Completed 2026-02-28.** New `repo_operator_test.go`: 38 subtests covering all CRUD, read, event emission, error paths, and edge cases. `runRepoOperatorTests(t, setup)` pattern ready for Phase 2's `DirectRepoOperator`. 50 50 + 51 51 + ### Gaps covered ✅ 52 52 + - `CreateRecord` — round-trip, TID 13-char rkey format, no-panic without event handler 53 53 + - `UpdateRecord` — CID changes, new data returned, non-existent record error, hydrated events 54 54 + - `PutRecord` — explicit rkey, duplicate rkey error, hydrated events 55 55 + - `UpsertRecord` — create path (created=true), update path (created=false, CID changes) 56 56 + - `DeleteRecord` — delete + verify gone, non-existent rkey error 57 57 + - `BatchWrite` — create+delete batch, update write type, auto-rkey (nil Rkey), delete-not-found error, empty write elem error, multi-op event emission with hydration, update hydration 58 58 + - `BulkUpsert` — create + re-upsert with changed data, multi-op event emission 59 59 + - `GetRecord` — CID match, CID mismatch error, not-found error 60 60 + - `GetRecordProof` — head CID + proof blocks, not-found error, no-repo error 61 61 + - `GetRepoRoot` — defined CID after init, changes after write 62 62 + - `GetRepoRev` — non-empty, changes after write 63 63 + - `ReadRepo` — non-empty CAR output, incremental export with `since` 64 64 + - `InitNewActor` — empty DID error, zero user error, event emission with hydration 65 65 + - Event emission — create/update/delete events verified: prevData, ops, oldRoot, newRoot, rev, since, repoSlice, hydration 66 66 + 67 67 + ### Coverage ✅ 68 68 + All RepoOperator methods 81–100%. Remaining uncovered lines are internal infrastructure error guards (`GetUserRepoRev`, `NewDeltaSession`, `OpenRepo`, `Commit`, `CloseWithRoot`) — not reachable without mocking the carstore. 69 69 + 70 70 + ### Files modified ✅ 71 71 + - `pkg/hold/pds/repo_operator_test.go` — new file, 38 subtests 72 72 + 73 73 + ### Verification ✅ 74 74 + - `make lint` — 0 issues 75 75 + - `make test` — all tests pass 76 76 + 77 77 + --- 78 78 + 79 79 + ## Phase 2: Build new implementation ✅ 80 80 + 81 81 + **Goal:** Create `DirectRepoOperator` using `indigo/repo` directly (cocoon pattern). 82 82 + 83 83 + **Completed 2026-02-28.** New `pkg/hold/pds/repo.go`: 548 lines (vs 852 in repomgr.go, ~36% reduction). All 37 subtests pass identically for both implementations. Race detector, shuffled order, and parallel execution all clean. 84 84 + 85 85 + ### New file: `pkg/hold/pds/repo.go` ✅ 86 86 + 87 87 + Key differences from vendored repomgr: 88 88 + - **Single `sync.Mutex`** instead of per-user lock map (`lklk` + `userLocks` + `userLock` struct + reference counting) 89 89 + - **No OpenTelemetry tracing** (`otel.Tracer` calls removed) 90 90 + - **No `gorm` dependency** (`RepoHead` struct removed, `gorm.io/gorm` dropped from go.mod) 91 91 + - **`openWriteSession` / `commitWrite` helpers** extract the repeated 6-step write pattern 92 92 + 93 93 + Core mutation pattern (same as current, just cleaner): 94 94 + 1. Lock → get rev → open delta session → open repo 95 95 + 2. Capture `r.DataCid()` for prevData 96 96 + 3. Perform operation(s) 97 97 + 4. `r.Commit()` → `ds.CloseWithRoot()` → emit event → unlock 98 98 + 99 99 + ### Shared types moved to `repo_operator.go` ✅ 100 100 + - `KeyManager` interface and `ActorInfo` struct moved from `repomgr.go` 101 101 + - Both implementations import from the same location 102 102 + 103 103 + ### Test wiring ✅ 104 104 + - `setupTestDirectRepoOperator` — creates carstore + key manager directly (no `NewHoldPDS`) 105 105 + - `runRepoOperatorTests` refactored to accept optional `freshSetup` for `InitNewActor_EventEmission` 106 106 + - `TestDirectRepoOperator` runs all 37 subtests identically 107 107 + 108 108 + ### Verification ✅ 109 109 + - `go build ./cmd/hold` — compiles 110 110 + - `TestRepoManager` — 37/37 subtests pass 111 111 + - `TestDirectRepoOperator` — 37/37 subtests pass 112 112 + - `-race -shuffle=on -count=5 -parallel=8` — all clean 113 113 + - `make lint` — 0 issues 114 114 + - `make test` — all tests pass 115 115 + 116 116 + --- 117 117 + 118 118 + ## Phase 3: Config flag + switchover 119 119 + 120 120 + **Goal:** Feature flag to select implementation, default old. 121 121 + 122 122 + ### Changes: 123 123 + - `pkg/hold/config.go` — add `UseDirectRepo bool` to DatabaseConfig 124 124 + - `pkg/hold/pds/server.go` — select implementation based on config in `NewHoldPDS`/`NewHoldPDSWithDB` 125 125 + - Regenerate example configs 126 126 + 127 127 + ### Verification: 128 128 + - Deploy with `use_direct_repo: false` (default) 129 129 + - Test with `use_direct_repo: true` in staging 130 130 + - `make lint && make test` 131 131 + 132 132 + --- 133 133 + 134 134 + ## Phase 4: Remove vendored repomgr 135 135 + 136 136 + **Goal:** After production validation, delete the old code. 137 137 + 138 138 + - Delete `repomgr.go` 139 139 + - Remove config flag, make `DirectRepoOperator` the only implementation 140 140 + - Rename `repo_direct.go` → `repo_operator_impl.go` 141 141 + - Regenerate example configs 142 142 + - `make lint && make test` 143 143 + 144 144 + --- 145 145 + 146 146 + ## Decision log 147 147 + 148 148 + - **`indigo/repo` over `atproto/repo`**: `atproto/repo` has the MST primitives (`Insert`, `Remove`, `ApplyOp`) but no high-level PDS API (`OpenRepo`, `CreateRecord`, `Commit(signFn)`). Its own doc.go says "does not yet work for implementing a repository host (PDS)." `indigo/repo` is what the reference PDS and cocoon use. The `RepoOperator` interface means we can swap later if `atproto/repo` adds PDS support.

+2 -2

go.mod

··· 25 25 github.com/gorilla/websocket v1.5.4-0.20250319132907-e064f32e3674 26 26 github.com/ipfs/go-block-format v0.2.3 27 27 github.com/ipfs/go-cid v0.6.0 28 28 - github.com/ipfs/go-datastore v0.9.1 29 28 github.com/ipfs/go-ipfs-blockstore v1.3.1 30 29 github.com/ipfs/go-ipld-cbor v0.2.1 31 30 github.com/ipfs/go-ipld-format v0.6.3 ··· 50 49 golang.org/x/image v0.36.0 51 50 golang.org/x/sys v0.41.0 52 51 golang.org/x/xerrors v0.0.0-20240903120638-7835f813f4da 53 53 - gorm.io/gorm v1.31.1 54 52 ) 55 53 56 54 require ( ··· 120 118 github.com/ipfs/bbloom v0.0.4 // indirect 121 119 github.com/ipfs/boxo v0.36.0 // indirect 122 120 github.com/ipfs/go-cidutil v0.1.1 // indirect 121 121 + github.com/ipfs/go-datastore v0.9.1 // indirect 123 122 github.com/ipfs/go-dsqueue v0.2.0 // indirect 124 123 github.com/ipfs/go-ipfs-ds-help v1.1.1 // indirect 125 124 github.com/ipfs/go-ipfs-util v0.0.3 // indirect ··· 213 212 google.golang.org/protobuf v1.36.11 // indirect 214 213 gopkg.in/yaml.v2 v2.4.0 // indirect 215 214 gopkg.in/yaml.v3 v3.0.1 // indirect 215 215 + gorm.io/gorm v1.31.1 // indirect 216 216 lukechampine.com/blake3 v1.4.1 // indirect 217 217 )

+4 -3

pkg/hold/pds/events.go

··· 56 56 Rev string `json:"rev" cborgen:"rev"` 57 57 Since *string `json:"since,omitempty" cborgen:"since,omitempty"` 58 58 PrevData string `json:"prevData,omitempty" cborgen:"prevData,omitempty"` // MST root CID string of previous commit 59 59 - Blocks []byte `json:"blocks" cborgen:"blocks"` // CAR slice bytes 59 59 + Blocks []byte `json:"blocks" cborgen:"blocks"` // CAR slice bytes 60 60 Ops []*atproto.SyncSubscribeRepos_RepoOp `json:"ops" cborgen:"ops"` 61 61 Time string `json:"time" cborgen:"time"` 62 62 Type string `json:"$type" cborgen:"$type"` // Always "#commit" ··· 170 170 } 171 171 } 172 172 173 173 - // Migration: add prev_data column if missing (existing databases) 174 174 - b.db.Exec("ALTER TABLE firehose_events ADD COLUMN prev_data TEXT") 173 173 + // Migration: add prev_data column if missing (existing databases). 174 174 + // Intentionally ignore error — fails with "duplicate column" if already present. 175 175 + _, _ = b.db.Exec("ALTER TABLE firehose_events ADD COLUMN prev_data TEXT") 175 176 176 177 // Load last sequence number from database 177 178 var lastSeq sql.NullInt64

+4 -104

pkg/hold/pds/import.go

··· 8 8 9 9 "github.com/bluesky-social/indigo/repo" 10 10 "github.com/ipfs/go-cid" 11 11 - "go.opentelemetry.io/otel" 12 11 ) 13 12 14 13 // rawCBOR wraps raw bytes to satisfy cbg.CBORMarshaler. ··· 20 19 return err 21 20 } 22 21 23 23 - // bulkRecord holds a single record to import. 24 24 - type bulkRecord struct { 25 25 - Collection string 26 26 - Rkey string 27 27 - Data rawCBOR 28 28 - } 29 29 - 30 22 // ImportResult summarizes a CAR import operation. 31 23 type ImportResult struct { 32 24 Total int ··· 52 44 } 53 45 54 46 // Collect all records 55 55 - var records []bulkRecord 47 47 + var records []BulkRecord 56 48 err = sourceRepo.ForEach(ctx, "", func(k string, v cid.Cid) error { 57 49 _, recBytes, err := sourceRepo.GetRecordBytes(ctx, k) 58 50 if err != nil { ··· 64 56 return fmt.Errorf("unexpected record path format: %s", k) 65 57 } 66 58 67 67 - records = append(records, bulkRecord{ 59 59 + records = append(records, BulkRecord{ 68 60 Collection: parts[0], 69 61 Rkey: parts[1], 70 62 Data: rawCBOR(*recBytes), ··· 79 71 return &ImportResult{PerCollection: map[string]int{}}, nil 80 72 } 81 73 82 82 - // Bulk upsert all records in a single commit 83 83 - if err := p.bulkImportRecords(ctx, records); err != nil { 74 74 + // Bulk upsert all records in a single commit via the RepoOperator interface 75 75 + if err := p.repomgr.BulkUpsert(ctx, p.uid, records); err != nil { 84 76 return nil, fmt.Errorf("failed to import records: %w", err) 85 77 } 86 78 ··· 94 86 } 95 87 return result, nil 96 88 } 97 97 - 98 98 - // bulkImportRecords writes all records in a single delta session + commit. 99 99 - // Each record is upserted: created if new, updated if exists. 100 100 - func (p *HoldPDS) bulkImportRecords(ctx context.Context, records []bulkRecord) error { 101 101 - ctx, span := otel.Tracer("repoman").Start(ctx, "BulkImportRecords") 102 102 - defer span.End() 103 103 - 104 104 - unlock := p.repomgr.lockUser(ctx, p.uid) 105 105 - defer unlock() 106 106 - 107 107 - rev, err := p.repomgr.cs.GetUserRepoRev(ctx, p.uid) 108 108 - if err != nil { 109 109 - return err 110 110 - } 111 111 - 112 112 - ds, err := p.repomgr.cs.NewDeltaSession(ctx, p.uid, &rev) 113 113 - if err != nil { 114 114 - return err 115 115 - } 116 116 - 117 117 - head := ds.BaseCid() 118 118 - r, err := repo.OpenRepo(ctx, ds, head) 119 119 - if err != nil { 120 120 - return err 121 121 - } 122 122 - 123 123 - // Capture previous MST root before commit overwrites it 124 124 - var prevData *cid.Cid 125 125 - if head.Defined() { 126 126 - pd := r.DataCid() 127 127 - prevData = &pd 128 128 - } 129 129 - 130 130 - ops := make([]RepoOp, 0, len(records)) 131 131 - for _, rec := range records { 132 132 - rpath := rec.Collection + "/" + rec.Rkey 133 133 - 134 134 - // Check if record exists to determine create vs update 135 135 - _, _, getErr := r.GetRecordBytes(ctx, rpath) 136 136 - recordExists := getErr == nil 137 137 - 138 138 - var cc cid.Cid 139 139 - var evtKind EventKind 140 140 - if recordExists { 141 141 - cc, err = r.UpdateRecord(ctx, rpath, rec.Data) 142 142 - evtKind = EvtKindUpdateRecord 143 143 - } else { 144 144 - cc, err = r.PutRecord(ctx, rpath, rec.Data) 145 145 - evtKind = EvtKindCreateRecord 146 146 - } 147 147 - if err != nil { 148 148 - return fmt.Errorf("failed to write %s: %w", rpath, err) 149 149 - } 150 150 - 151 151 - ops = append(ops, RepoOp{ 152 152 - Kind: evtKind, 153 153 - Collection: rec.Collection, 154 154 - Rkey: rec.Rkey, 155 155 - RecCid: &cc, 156 156 - }) 157 157 - } 158 158 - 159 159 - nroot, nrev, err := r.Commit(ctx, p.repomgr.kmgr.SignForUser) 160 160 - if err != nil { 161 161 - return err 162 162 - } 163 163 - 164 164 - rslice, err := ds.CloseWithRoot(ctx, nroot, nrev) 165 165 - if err != nil { 166 166 - return fmt.Errorf("close with root: %w", err) 167 167 - } 168 168 - 169 169 - var oldroot *cid.Cid 170 170 - if head.Defined() { 171 171 - oldroot = &head 172 172 - } 173 173 - 174 174 - if p.repomgr.events != nil { 175 175 - p.repomgr.events(ctx, &RepoEvent{ 176 176 - User: p.uid, 177 177 - OldRoot: oldroot, 178 178 - NewRoot: nroot, 179 179 - PrevData: prevData, 180 180 - Rev: nrev, 181 181 - Since: &rev, 182 182 - Ops: ops, 183 183 - RepoSlice: rslice, 184 184 - }) 185 185 - } 186 186 - 187 187 - return nil 188 188 - }

+548

pkg/hold/pds/repo.go

··· 1 1 + package pds 2 2 + 3 3 + // repo.go — DirectRepoOperator manages ATProto repository operations using 4 4 + // indigo/repo directly, replacing RepoManager with a simpler implementation. 5 5 + // 6 6 + // Key simplifications vs RepoManager: 7 7 + // - Single sync.Mutex instead of per-user lock map (hold is always uid=1) 8 8 + // - No OpenTelemetry tracing 9 9 + // - No gorm dependency 10 10 + // 11 11 + // Implements the RepoOperator interface (see repo_operator.go). 12 12 + // See docs/REPOMGR_MIGRATION.md for the migration plan. 13 13 + 14 14 + import ( 15 15 + "context" 16 16 + "fmt" 17 17 + "io" 18 18 + "log/slog" 19 19 + "sync" 20 20 + 21 21 + holddb "atcr.io/pkg/hold/db" 22 22 + atproto "github.com/bluesky-social/indigo/api/atproto" 23 23 + bsky "github.com/bluesky-social/indigo/api/bsky" 24 24 + "github.com/bluesky-social/indigo/atproto/syntax" 25 25 + "github.com/bluesky-social/indigo/models" 26 26 + "github.com/bluesky-social/indigo/repo" 27 27 + "github.com/bluesky-social/indigo/util" 28 28 + 29 29 + blocks "github.com/ipfs/go-block-format" 30 30 + "github.com/ipfs/go-cid" 31 31 + cbg "github.com/whyrusleeping/cbor-gen" 32 32 + ) 33 33 + 34 34 + // Compile-time check that DirectRepoOperator implements RepoOperator. 35 35 + var _ RepoOperator = (*DirectRepoOperator)(nil) 36 36 + 37 37 + // DirectRepoOperator implements RepoOperator using indigo/repo directly. 38 38 + type DirectRepoOperator struct { 39 39 + cs holddb.CarStore 40 40 + kmgr KeyManager 41 41 + mu sync.Mutex // single mutex (hold is always uid=1) 42 42 + events func(context.Context, *RepoEvent) 43 43 + hydrateRecords bool 44 44 + log *slog.Logger 45 45 + clk *syntax.TIDClock // for BatchWrite auto-rkey generation 46 46 + } 47 47 + 48 48 + // NewDirectRepoOperator creates a new DirectRepoOperator. 49 49 + func NewDirectRepoOperator(cs holddb.CarStore, kmgr KeyManager) *DirectRepoOperator { 50 50 + return &DirectRepoOperator{ 51 51 + cs: cs, 52 52 + kmgr: kmgr, 53 53 + log: slog.Default().With("system", "repo"), 54 54 + clk: syntax.NewTIDClock(0), 55 55 + } 56 56 + } 57 57 + 58 58 + // writeSession holds state for an in-progress write transaction. 59 59 + type writeSession struct { 60 60 + ds *holddb.DeltaSession 61 61 + r *repo.Repo 62 62 + head cid.Cid 63 63 + prevData *cid.Cid 64 64 + rev string 65 65 + } 66 66 + 67 67 + // openWriteSession opens a write session, acquiring the lock. 68 68 + // On success, the caller holds d.mu and must defer d.mu.Unlock(). 69 69 + // On error, the lock is released before returning. 70 70 + func (d *DirectRepoOperator) openWriteSession(ctx context.Context, user models.Uid) (*writeSession, error) { 71 71 + d.mu.Lock() 72 72 + 73 73 + rev, err := d.cs.GetUserRepoRev(ctx, user) 74 74 + if err != nil { 75 75 + d.mu.Unlock() 76 76 + return nil, err 77 77 + } 78 78 + 79 79 + ds, err := d.cs.NewDeltaSession(ctx, user, &rev) 80 80 + if err != nil { 81 81 + d.mu.Unlock() 82 82 + return nil, err 83 83 + } 84 84 + 85 85 + head := ds.BaseCid() 86 86 + 87 87 + r, err := repo.OpenRepo(ctx, ds, head) 88 88 + if err != nil { 89 89 + d.mu.Unlock() 90 90 + return nil, err 91 91 + } 92 92 + 93 93 + var prevData *cid.Cid 94 94 + if head.Defined() { 95 95 + pd := r.DataCid() 96 96 + prevData = &pd 97 97 + } 98 98 + 99 99 + return &writeSession{ 100 100 + ds: ds, 101 101 + r: r, 102 102 + head: head, 103 103 + prevData: prevData, 104 104 + rev: rev, 105 105 + }, nil 106 106 + } 107 107 + 108 108 + // commitWrite commits a write session and emits an event if configured. 109 109 + func (d *DirectRepoOperator) commitWrite(ctx context.Context, ws *writeSession, user models.Uid, ops []RepoOp) (cid.Cid, string, error) { 110 110 + nroot, nrev, err := ws.r.Commit(ctx, d.kmgr.SignForUser) 111 111 + if err != nil { 112 112 + return cid.Undef, "", err 113 113 + } 114 114 + 115 115 + rslice, err := ws.ds.CloseWithRoot(ctx, nroot, nrev) 116 116 + if err != nil { 117 117 + return cid.Undef, "", fmt.Errorf("close with root: %w", err) 118 118 + } 119 119 + 120 120 + if d.events != nil { 121 121 + var oldroot *cid.Cid 122 122 + if ws.head.Defined() { 123 123 + oldroot = &ws.head 124 124 + } 125 125 + 126 126 + d.events(ctx, &RepoEvent{ 127 127 + User: user, 128 128 + OldRoot: oldroot, 129 129 + NewRoot: nroot, 130 130 + PrevData: ws.prevData, 131 131 + Rev: nrev, 132 132 + Since: &ws.rev, 133 133 + Ops: ops, 134 134 + RepoSlice: rslice, 135 135 + }) 136 136 + } 137 137 + 138 138 + return nroot, nrev, nil 139 139 + } 140 140 + 141 141 + func (d *DirectRepoOperator) SetEventHandler(cb func(context.Context, *RepoEvent), hydrateRecords bool) { 142 142 + d.events = cb 143 143 + d.hydrateRecords = hydrateRecords 144 144 + } 145 145 + 146 146 + func (d *DirectRepoOperator) CreateRecord(ctx context.Context, user models.Uid, collection string, rec cbg.CBORMarshaler) (string, cid.Cid, error) { 147 147 + ws, err := d.openWriteSession(ctx, user) 148 148 + if err != nil { 149 149 + return "", cid.Undef, err 150 150 + } 151 151 + defer d.mu.Unlock() 152 152 + 153 153 + cc, tid, err := ws.r.CreateRecord(ctx, collection, rec) 154 154 + if err != nil { 155 155 + return "", cid.Undef, err 156 156 + } 157 157 + 158 158 + ops := []RepoOp{{ 159 159 + Kind: EvtKindCreateRecord, 160 160 + Collection: collection, 161 161 + Rkey: tid, 162 162 + Record: rec, // CreateRecord always includes Record (no hydration check) 163 163 + RecCid: &cc, 164 164 + }} 165 165 + 166 166 + if _, _, err := d.commitWrite(ctx, ws, user, ops); err != nil { 167 167 + return "", cid.Undef, err 168 168 + } 169 169 + 170 170 + return collection + "/" + tid, cc, nil 171 171 + } 172 172 + 173 173 + func (d *DirectRepoOperator) UpdateRecord(ctx context.Context, user models.Uid, collection, rkey string, rec cbg.CBORMarshaler) (cid.Cid, error) { 174 174 + ws, err := d.openWriteSession(ctx, user) 175 175 + if err != nil { 176 176 + return cid.Undef, err 177 177 + } 178 178 + defer d.mu.Unlock() 179 179 + 180 180 + rpath := collection + "/" + rkey 181 181 + cc, err := ws.r.UpdateRecord(ctx, rpath, rec) 182 182 + if err != nil { 183 183 + return cid.Undef, err 184 184 + } 185 185 + 186 186 + op := RepoOp{ 187 187 + Kind: EvtKindUpdateRecord, 188 188 + Collection: collection, 189 189 + Rkey: rkey, 190 190 + RecCid: &cc, 191 191 + } 192 192 + if d.hydrateRecords { 193 193 + op.Record = rec 194 194 + } 195 195 + 196 196 + if _, _, err := d.commitWrite(ctx, ws, user, []RepoOp{op}); err != nil { 197 197 + return cid.Undef, err 198 198 + } 199 199 + 200 200 + return cc, nil 201 201 + } 202 202 + 203 203 + func (d *DirectRepoOperator) PutRecord(ctx context.Context, user models.Uid, collection, rkey string, rec cbg.CBORMarshaler) (string, cid.Cid, error) { 204 204 + ws, err := d.openWriteSession(ctx, user) 205 205 + if err != nil { 206 206 + return "", cid.Undef, err 207 207 + } 208 208 + defer d.mu.Unlock() 209 209 + 210 210 + rpath := collection + "/" + rkey 211 211 + cc, err := ws.r.PutRecord(ctx, rpath, rec) 212 212 + if err != nil { 213 213 + return "", cid.Undef, err 214 214 + } 215 215 + 216 216 + op := RepoOp{ 217 217 + Kind: EvtKindCreateRecord, 218 218 + Collection: collection, 219 219 + Rkey: rkey, 220 220 + RecCid: &cc, 221 221 + } 222 222 + if d.hydrateRecords { 223 223 + op.Record = rec 224 224 + } 225 225 + 226 226 + if _, _, err := d.commitWrite(ctx, ws, user, []RepoOp{op}); err != nil { 227 227 + return "", cid.Undef, err 228 228 + } 229 229 + 230 230 + return rpath, cc, nil 231 231 + } 232 232 + 233 233 + func (d *DirectRepoOperator) UpsertRecord(ctx context.Context, user models.Uid, collection, rkey string, rec cbg.CBORMarshaler) (string, cid.Cid, bool, error) { 234 234 + ws, err := d.openWriteSession(ctx, user) 235 235 + if err != nil { 236 236 + return "", cid.Undef, false, err 237 237 + } 238 238 + defer d.mu.Unlock() 239 239 + 240 240 + rpath := collection + "/" + rkey 241 241 + 242 242 + // Check if record exists 243 243 + _, _, err = ws.r.GetRecordBytes(ctx, rpath) 244 244 + recordExists := err == nil 245 245 + 246 246 + var cc cid.Cid 247 247 + var evtKind EventKind 248 248 + if recordExists { 249 249 + cc, err = ws.r.UpdateRecord(ctx, rpath, rec) 250 250 + evtKind = EvtKindUpdateRecord 251 251 + } else { 252 252 + cc, err = ws.r.PutRecord(ctx, rpath, rec) 253 253 + evtKind = EvtKindCreateRecord 254 254 + } 255 255 + if err != nil { 256 256 + return "", cid.Undef, false, err 257 257 + } 258 258 + 259 259 + op := RepoOp{ 260 260 + Kind: evtKind, 261 261 + Collection: collection, 262 262 + Rkey: rkey, 263 263 + RecCid: &cc, 264 264 + } 265 265 + if d.hydrateRecords { 266 266 + op.Record = rec 267 267 + } 268 268 + 269 269 + if _, _, err := d.commitWrite(ctx, ws, user, []RepoOp{op}); err != nil { 270 270 + return "", cid.Undef, false, err 271 271 + } 272 272 + 273 273 + return rpath, cc, !recordExists, nil 274 274 + } 275 275 + 276 276 + func (d *DirectRepoOperator) DeleteRecord(ctx context.Context, user models.Uid, collection, rkey string) error { 277 277 + ws, err := d.openWriteSession(ctx, user) 278 278 + if err != nil { 279 279 + return err 280 280 + } 281 281 + defer d.mu.Unlock() 282 282 + 283 283 + rpath := collection + "/" + rkey 284 284 + if err := ws.r.DeleteRecord(ctx, rpath); err != nil { 285 285 + return err 286 286 + } 287 287 + 288 288 + ops := []RepoOp{{ 289 289 + Kind: EvtKindDeleteRecord, 290 290 + Collection: collection, 291 291 + Rkey: rkey, 292 292 + }} 293 293 + 294 294 + _, _, err = d.commitWrite(ctx, ws, user, ops) 295 295 + return err 296 296 + } 297 297 + 298 298 + func (d *DirectRepoOperator) BatchWrite(ctx context.Context, user models.Uid, writes []*atproto.RepoApplyWrites_Input_Writes_Elem) error { 299 299 + ws, err := d.openWriteSession(ctx, user) 300 300 + if err != nil { 301 301 + return err 302 302 + } 303 303 + defer d.mu.Unlock() 304 304 + 305 305 + ops := make([]RepoOp, 0, len(writes)) 306 306 + for _, w := range writes { 307 307 + switch { 308 308 + case w.RepoApplyWrites_Create != nil: 309 309 + c := w.RepoApplyWrites_Create 310 310 + var rkey string 311 311 + if c.Rkey != nil { 312 312 + rkey = *c.Rkey 313 313 + } else { 314 314 + rkey = d.clk.Next().String() 315 315 + } 316 316 + 317 317 + nsid := c.Collection + "/" + rkey 318 318 + cc, err := ws.r.PutRecord(ctx, nsid, c.Value.Val) 319 319 + if err != nil { 320 320 + return err 321 321 + } 322 322 + 323 323 + op := RepoOp{ 324 324 + Kind: EvtKindCreateRecord, 325 325 + Collection: c.Collection, 326 326 + Rkey: rkey, 327 327 + RecCid: &cc, 328 328 + } 329 329 + if d.hydrateRecords { 330 330 + op.Record = c.Value.Val 331 331 + } 332 332 + ops = append(ops, op) 333 333 + 334 334 + case w.RepoApplyWrites_Update != nil: 335 335 + u := w.RepoApplyWrites_Update 336 336 + 337 337 + // Known quirk: uses PutRecord (mst.Add) not UpdateRecord 338 338 + cc, err := ws.r.PutRecord(ctx, u.Collection+"/"+u.Rkey, u.Value.Val) 339 339 + if err != nil { 340 340 + return err 341 341 + } 342 342 + 343 343 + op := RepoOp{ 344 344 + Kind: EvtKindUpdateRecord, 345 345 + Collection: u.Collection, 346 346 + Rkey: u.Rkey, 347 347 + RecCid: &cc, 348 348 + } 349 349 + if d.hydrateRecords { 350 350 + op.Record = u.Value.Val 351 351 + } 352 352 + ops = append(ops, op) 353 353 + 354 354 + case w.RepoApplyWrites_Delete != nil: 355 355 + del := w.RepoApplyWrites_Delete 356 356 + 357 357 + if err := ws.r.DeleteRecord(ctx, del.Collection+"/"+del.Rkey); err != nil { 358 358 + return err 359 359 + } 360 360 + 361 361 + ops = append(ops, RepoOp{ 362 362 + Kind: EvtKindDeleteRecord, 363 363 + Collection: del.Collection, 364 364 + Rkey: del.Rkey, 365 365 + }) 366 366 + 367 367 + default: 368 368 + return fmt.Errorf("no operation set in write enum") 369 369 + } 370 370 + } 371 371 + 372 372 + _, _, err = d.commitWrite(ctx, ws, user, ops) 373 373 + return err 374 374 + } 375 375 + 376 376 + func (d *DirectRepoOperator) BulkUpsert(ctx context.Context, user models.Uid, records []BulkRecord) error { 377 377 + ws, err := d.openWriteSession(ctx, user) 378 378 + if err != nil { 379 379 + return err 380 380 + } 381 381 + defer d.mu.Unlock() 382 382 + 383 383 + ops := make([]RepoOp, 0, len(records)) 384 384 + for _, rec := range records { 385 385 + rpath := rec.Collection + "/" + rec.Rkey 386 386 + 387 387 + // Check if record exists to determine create vs update 388 388 + _, _, getErr := ws.r.GetRecordBytes(ctx, rpath) 389 389 + recordExists := getErr == nil 390 390 + 391 391 + var cc cid.Cid 392 392 + var evtKind EventKind 393 393 + if recordExists { 394 394 + cc, err = ws.r.UpdateRecord(ctx, rpath, rec.Data) 395 395 + evtKind = EvtKindUpdateRecord 396 396 + } else { 397 397 + cc, err = ws.r.PutRecord(ctx, rpath, rec.Data) 398 398 + evtKind = EvtKindCreateRecord 399 399 + } 400 400 + if err != nil { 401 401 + return fmt.Errorf("failed to write %s: %w", rpath, err) 402 402 + } 403 403 + 404 404 + // No hydration for BulkUpsert (records never included in events) 405 405 + ops = append(ops, RepoOp{ 406 406 + Kind: evtKind, 407 407 + Collection: rec.Collection, 408 408 + Rkey: rec.Rkey, 409 409 + RecCid: &cc, 410 410 + }) 411 411 + } 412 412 + 413 413 + _, _, err = d.commitWrite(ctx, ws, user, ops) 414 414 + return err 415 415 + } 416 416 + 417 417 + func (d *DirectRepoOperator) InitNewActor(ctx context.Context, user models.Uid, handle, did, displayname string, declcid, actortype string) error { 418 418 + d.mu.Lock() 419 419 + defer d.mu.Unlock() 420 420 + 421 421 + if did == "" { 422 422 + return fmt.Errorf("must specify DID for new actor") 423 423 + } 424 424 + 425 425 + if user == 0 { 426 426 + return fmt.Errorf("must specify user for new actor") 427 427 + } 428 428 + 429 429 + ds, err := d.cs.NewDeltaSession(ctx, user, nil) 430 430 + if err != nil { 431 431 + return fmt.Errorf("creating new delta session: %w", err) 432 432 + } 433 433 + 434 434 + r := repo.NewRepo(ctx, did, ds) 435 435 + 436 436 + profile := &bsky.ActorProfile{ 437 437 + DisplayName: &displayname, 438 438 + } 439 439 + 440 440 + _, err = r.PutRecord(ctx, "app.bsky.actor.profile/self", profile) 441 441 + if err != nil { 442 442 + return fmt.Errorf("setting initial actor profile: %w", err) 443 443 + } 444 444 + 445 445 + root, nrev, err := r.Commit(ctx, d.kmgr.SignForUser) 446 446 + if err != nil { 447 447 + return fmt.Errorf("committing repo for actor init: %w", err) 448 448 + } 449 449 + 450 450 + rslice, err := ds.CloseWithRoot(ctx, root, nrev) 451 451 + if err != nil { 452 452 + return fmt.Errorf("close with root: %w", err) 453 453 + } 454 454 + 455 455 + if d.events != nil { 456 456 + op := RepoOp{ 457 457 + Kind: EvtKindCreateRecord, 458 458 + Collection: "app.bsky.actor.profile", 459 459 + Rkey: "self", 460 460 + } 461 461 + 462 462 + if d.hydrateRecords { 463 463 + op.Record = profile 464 464 + } 465 465 + 466 466 + d.events(ctx, &RepoEvent{ 467 467 + User: user, 468 468 + NewRoot: root, 469 469 + Rev: nrev, 470 470 + Ops: []RepoOp{op}, 471 471 + RepoSlice: rslice, 472 472 + }) 473 473 + } 474 474 + 475 475 + return nil 476 476 + } 477 477 + 478 478 + func (d *DirectRepoOperator) GetRecord(ctx context.Context, user models.Uid, collection string, rkey string, maybeCid cid.Cid) (cid.Cid, cbg.CBORMarshaler, error) { 479 479 + bs, err := d.cs.ReadOnlySession(user) 480 480 + if err != nil { 481 481 + return cid.Undef, nil, err 482 482 + } 483 483 + 484 484 + head, err := d.cs.GetUserRepoHead(ctx, user) 485 485 + if err != nil { 486 486 + return cid.Undef, nil, err 487 487 + } 488 488 + 489 489 + r, err := repo.OpenRepo(ctx, bs, head) 490 490 + if err != nil { 491 491 + return cid.Undef, nil, err 492 492 + } 493 493 + 494 494 + ocid, val, err := r.GetRecord(ctx, collection+"/"+rkey) 495 495 + if err != nil { 496 496 + return cid.Undef, nil, err 497 497 + } 498 498 + 499 499 + if maybeCid.Defined() && ocid != maybeCid { 500 500 + return cid.Undef, nil, fmt.Errorf("record at specified key had different CID than expected") 501 501 + } 502 502 + 503 503 + return ocid, val, nil 504 504 + } 505 505 + 506 506 + func (d *DirectRepoOperator) GetRecordProof(ctx context.Context, user models.Uid, collection string, rkey string) (cid.Cid, []blocks.Block, error) { 507 507 + robs, err := d.cs.ReadOnlySession(user) 508 508 + if err != nil { 509 509 + return cid.Undef, nil, err 510 510 + } 511 511 + 512 512 + bs := util.NewLoggingBstore(robs) 513 513 + 514 514 + head, err := d.cs.GetUserRepoHead(ctx, user) 515 515 + if err != nil { 516 516 + return cid.Undef, nil, err 517 517 + } 518 518 + 519 519 + r, err := repo.OpenRepo(ctx, bs, head) 520 520 + if err != nil { 521 521 + return cid.Undef, nil, err 522 522 + } 523 523 + 524 524 + _, _, err = r.GetRecordBytes(ctx, collection+"/"+rkey) 525 525 + if err != nil { 526 526 + return cid.Undef, nil, err 527 527 + } 528 528 + 529 529 + return head, bs.GetLoggedBlocks(), nil 530 530 + } 531 531 + 532 532 + func (d *DirectRepoOperator) GetRepoRoot(ctx context.Context, user models.Uid) (cid.Cid, error) { 533 533 + d.mu.Lock() 534 534 + defer d.mu.Unlock() 535 535 + 536 536 + return d.cs.GetUserRepoHead(ctx, user) 537 537 + } 538 538 + 539 539 + func (d *DirectRepoOperator) GetRepoRev(ctx context.Context, user models.Uid) (string, error) { 540 540 + d.mu.Lock() 541 541 + defer d.mu.Unlock() 542 542 + 543 543 + return d.cs.GetUserRepoRev(ctx, user) 544 544 + } 545 545 + 546 546 + func (d *DirectRepoOperator) ReadRepo(ctx context.Context, user models.Uid, since string, w io.Writer) error { 547 547 + return d.cs.ReadUserCar(ctx, user, since, true, w) 548 548 + }

+96

pkg/hold/pds/repo_operator.go

··· 1 1 + // Package pds implements a minimal ATProto PDS for the hold service. 2 2 + package pds 3 3 + 4 4 + import ( 5 5 + "context" 6 6 + "io" 7 7 + 8 8 + atproto "github.com/bluesky-social/indigo/api/atproto" 9 9 + "github.com/bluesky-social/indigo/models" 10 10 + 11 11 + blocks "github.com/ipfs/go-block-format" 12 12 + "github.com/ipfs/go-cid" 13 13 + cbg "github.com/whyrusleeping/cbor-gen" 14 14 + ) 15 15 + 16 16 + // RepoOperator defines the interface for ATProto repository operations. 17 17 + // RepoManager implements this interface. Future implementations (e.g., using 18 18 + // indigo/repo directly) can be swapped in behind this interface. 19 19 + // See docs/REPOMGR_MIGRATION.md for the migration plan. 20 20 + type RepoOperator interface { 21 21 + // Record CRUD 22 22 + CreateRecord(ctx context.Context, user models.Uid, collection string, rec cbg.CBORMarshaler) (string, cid.Cid, error) 23 23 + UpdateRecord(ctx context.Context, user models.Uid, collection, rkey string, rec cbg.CBORMarshaler) (cid.Cid, error) 24 24 + PutRecord(ctx context.Context, user models.Uid, collection, rkey string, rec cbg.CBORMarshaler) (string, cid.Cid, error) 25 25 + UpsertRecord(ctx context.Context, user models.Uid, collection, rkey string, rec cbg.CBORMarshaler) (string, cid.Cid, bool, error) 26 26 + DeleteRecord(ctx context.Context, user models.Uid, collection, rkey string) error 27 27 + BatchWrite(ctx context.Context, user models.Uid, writes []*atproto.RepoApplyWrites_Input_Writes_Elem) error 28 28 + BulkUpsert(ctx context.Context, user models.Uid, records []BulkRecord) error 29 29 + 30 30 + // Read 31 31 + GetRecord(ctx context.Context, user models.Uid, collection string, rkey string, maybeCid cid.Cid) (cid.Cid, cbg.CBORMarshaler, error) 32 32 + GetRecordProof(ctx context.Context, user models.Uid, collection string, rkey string) (cid.Cid, []blocks.Block, error) 33 33 + GetRepoRoot(ctx context.Context, user models.Uid) (cid.Cid, error) 34 34 + GetRepoRev(ctx context.Context, user models.Uid) (string, error) 35 35 + ReadRepo(ctx context.Context, user models.Uid, since string, w io.Writer) error 36 36 + 37 37 + // Lifecycle 38 38 + InitNewActor(ctx context.Context, user models.Uid, handle, did, displayname string, declcid, actortype string) error 39 39 + SetEventHandler(cb func(context.Context, *RepoEvent), hydrateRecords bool) 40 40 + } 41 41 + 42 42 + // KeyManager handles cryptographic signing for repository commits. 43 43 + type KeyManager interface { 44 44 + VerifyUserSignature(context.Context, string, []byte, []byte) error 45 45 + SignForUser(context.Context, string, []byte) ([]byte, error) 46 46 + } 47 47 + 48 48 + // ActorInfo holds identity information for a repository actor. 49 49 + type ActorInfo struct { 50 50 + Did string 51 51 + Handle string 52 52 + DisplayName string 53 53 + Type string 54 54 + } 55 55 + 56 56 + // Compile-time check that RepoManager implements RepoOperator. 57 57 + var _ RepoOperator = (*RepoManager)(nil) 58 58 + 59 59 + // RepoEvent represents a mutation event emitted by a RepoOperator. 60 60 + type RepoEvent struct { 61 61 + User models.Uid 62 62 + OldRoot *cid.Cid 63 63 + NewRoot cid.Cid 64 64 + PrevData *cid.Cid // MST root CID of the previous commit (for firehose prevData field) 65 65 + Since *string 66 66 + Rev string 67 67 + RepoSlice []byte 68 68 + PDS uint 69 69 + Ops []RepoOp 70 70 + } 71 71 + 72 72 + // RepoOp represents a single operation within a RepoEvent. 73 73 + type RepoOp struct { 74 74 + Kind EventKind 75 75 + Collection string 76 76 + Rkey string 77 77 + RecCid *cid.Cid 78 78 + Record any 79 79 + ActorInfo *ActorInfo 80 80 + } 81 81 + 82 82 + // EventKind identifies the type of repository mutation. 83 83 + type EventKind string 84 84 + 85 85 + const ( 86 86 + EvtKindCreateRecord = EventKind("create") 87 87 + EvtKindUpdateRecord = EventKind("update") 88 88 + EvtKindDeleteRecord = EventKind("delete") 89 89 + ) 90 90 + 91 91 + // BulkRecord holds a single record for bulk import/upsert operations. 92 92 + type BulkRecord struct { 93 93 + Collection string 94 94 + Rkey string 95 95 + Data cbg.CBORMarshaler 96 96 + }

+418

pkg/hold/pds/repo_operator_benchmark_test.go

··· 1 1 + package pds 2 2 + 3 3 + import ( 4 4 + "bytes" 5 5 + "context" 6 6 + "fmt" 7 7 + "io" 8 8 + "log/slog" 9 9 + "os" 10 10 + "path/filepath" 11 11 + "testing" 12 12 + 13 13 + "atcr.io/pkg/atproto" 14 14 + "atcr.io/pkg/auth/oauth" 15 15 + holddb "atcr.io/pkg/hold/db" 16 16 + "github.com/bluesky-social/indigo/models" 17 17 + "github.com/ipfs/go-cid" 18 18 + 19 19 + indigoatproto "github.com/bluesky-social/indigo/api/atproto" 20 20 + lexutil "github.com/bluesky-social/indigo/lex/util" 21 21 + ) 22 22 + 23 23 + // benchSetup creates a RepoOperator and returns it with the user ID. 24 24 + type benchSetup func(b *testing.B) (RepoOperator, models.Uid) 25 25 + 26 26 + func suppressLogs(b *testing.B) { 27 27 + b.Helper() 28 28 + prev := slog.Default() 29 29 + slog.SetDefault(slog.New(slog.NewTextHandler(io.Discard, nil))) 30 30 + b.Cleanup(func() { slog.SetDefault(prev) }) 31 31 + } 32 32 + 33 33 + func setupBenchRepoManager(b *testing.B) (RepoOperator, models.Uid) { 34 34 + b.Helper() 35 35 + suppressLogs(b) 36 36 + ctx := context.Background() 37 37 + tmpDir := b.TempDir() 38 38 + keyPath := filepath.Join(tmpDir, "signing-key") 39 39 + 40 40 + if err := os.WriteFile(keyPath, sharedTestKey, 0600); err != nil { 41 41 + b.Fatalf("write signing key: %v", err) 42 42 + } 43 43 + 44 44 + pds, err := NewHoldPDS(ctx, "did:web:hold.bench", "https://hold.bench", "https://atcr.io", ":memory:", keyPath, false) 45 45 + if err != nil { 46 46 + b.Fatalf("NewHoldPDS: %v", err) 47 47 + } 48 48 + 49 49 + if err := pds.repomgr.InitNewActor(ctx, pds.uid, "", pds.did, "", "", ""); err != nil { 50 50 + b.Fatalf("InitNewActor: %v", err) 51 51 + } 52 52 + 53 53 + b.Cleanup(func() { pds.Close() }) 54 54 + return pds.repomgr, pds.uid 55 55 + } 56 56 + 57 57 + func setupBenchDirectRepoOperator(b *testing.B) (RepoOperator, models.Uid) { 58 58 + b.Helper() 59 59 + suppressLogs(b) 60 60 + ctx := context.Background() 61 61 + keyPath := filepath.Join(b.TempDir(), "signing-key") 62 62 + 63 63 + if err := os.WriteFile(keyPath, sharedTestKey, 0600); err != nil { 64 64 + b.Fatalf("write signing key: %v", err) 65 65 + } 66 66 + signingKey, err := oauth.GenerateOrLoadPDSKey(keyPath) 67 67 + if err != nil { 68 68 + b.Fatalf("GenerateOrLoadPDSKey: %v", err) 69 69 + } 70 70 + 71 71 + sqlStore := new(holddb.SQLiteStore) 72 72 + if err := sqlStore.Open(":memory:"); err != nil { 73 73 + b.Fatalf("SQLiteStore.Open: %v", err) 74 74 + } 75 75 + b.Cleanup(func() { sqlStore.Close() }) 76 76 + 77 77 + kmgr := NewHoldKeyManager(signingKey) 78 78 + op := NewDirectRepoOperator(sqlStore, kmgr) 79 79 + uid := models.Uid(1) 80 80 + 81 81 + if err := op.InitNewActor(ctx, uid, "", "did:web:hold.bench", "", "", ""); err != nil { 82 82 + b.Fatalf("InitNewActor: %v", err) 83 83 + } 84 84 + 85 85 + return op, uid 86 86 + } 87 87 + 88 88 + // seedRecords writes n records and returns their rkeys. 89 89 + func seedRecords(b *testing.B, op RepoOperator, uid models.Uid, n int) []string { 90 90 + b.Helper() 91 91 + ctx := context.Background() 92 92 + rkeys := make([]string, n) 93 93 + for i := 0; i < n; i++ { 94 94 + rkey := fmt.Sprintf("seed%d", i) 95 95 + _, _, err := op.PutRecord(ctx, uid, atproto.CrewCollection, rkey, newCrewRecord(fmt.Sprintf("did:plc:seed%d", i))) 96 96 + if err != nil { 97 97 + b.Fatalf("seed PutRecord %d: %v", i, err) 98 98 + } 99 99 + rkeys[i] = rkey 100 100 + } 101 101 + return rkeys 102 102 + } 103 103 + 104 104 + func runRepoOperatorBenchmarks(b *testing.B, name string, setup benchSetup) { 105 105 + b.Run(name, func(b *testing.B) { 106 106 + b.Run("CreateRecord", func(b *testing.B) { 107 107 + op, uid := setup(b) 108 108 + ctx := context.Background() 109 109 + 110 110 + b.ResetTimer() 111 111 + for i := 0; i < b.N; i++ { 112 112 + _, _, err := op.CreateRecord(ctx, uid, atproto.CrewCollection, newCrewRecord(fmt.Sprintf("did:plc:bench%d", i))) 113 113 + if err != nil { 114 114 + b.Fatalf("CreateRecord: %v", err) 115 115 + } 116 116 + } 117 117 + }) 118 118 + 119 119 + b.Run("PutRecord", func(b *testing.B) { 120 120 + op, uid := setup(b) 121 121 + ctx := context.Background() 122 122 + 123 123 + b.ResetTimer() 124 124 + for i := 0; i < b.N; i++ { 125 125 + rkey := fmt.Sprintf("put%d", i) 126 126 + _, _, err := op.PutRecord(ctx, uid, atproto.CrewCollection, rkey, newCrewRecord(fmt.Sprintf("did:plc:put%d", i))) 127 127 + if err != nil { 128 128 + b.Fatalf("PutRecord: %v", err) 129 129 + } 130 130 + } 131 131 + }) 132 132 + 133 133 + b.Run("UpdateRecord", func(b *testing.B) { 134 134 + op, uid := setup(b) 135 135 + ctx := context.Background() 136 136 + 137 137 + // Seed one record to update repeatedly 138 138 + _, _, err := op.PutRecord(ctx, uid, atproto.CrewCollection, "updbench", newCrewRecord("did:plc:updbench")) 139 139 + if err != nil { 140 140 + b.Fatalf("seed PutRecord: %v", err) 141 141 + } 142 142 + 143 143 + b.ResetTimer() 144 144 + for i := 0; i < b.N; i++ { 145 145 + rec := newCrewRecord("did:plc:updbench") 146 146 + rec.Role = fmt.Sprintf("role%d", i) 147 147 + _, err := op.UpdateRecord(ctx, uid, atproto.CrewCollection, "updbench", rec) 148 148 + if err != nil { 149 149 + b.Fatalf("UpdateRecord: %v", err) 150 150 + } 151 151 + } 152 152 + }) 153 153 + 154 154 + b.Run("DeleteRecord", func(b *testing.B) { 155 155 + op, uid := setup(b) 156 156 + ctx := context.Background() 157 157 + 158 158 + // Pre-create all records to delete 159 159 + rkeys := make([]string, b.N) 160 160 + for i := 0; i < b.N; i++ { 161 161 + rkey := fmt.Sprintf("del%d", i) 162 162 + _, _, err := op.PutRecord(ctx, uid, atproto.CrewCollection, rkey, newCrewRecord(fmt.Sprintf("did:plc:del%d", i))) 163 163 + if err != nil { 164 164 + b.Fatalf("seed PutRecord: %v", err) 165 165 + } 166 166 + rkeys[i] = rkey 167 167 + } 168 168 + 169 169 + b.ResetTimer() 170 170 + for i := 0; i < b.N; i++ { 171 171 + if err := op.DeleteRecord(ctx, uid, atproto.CrewCollection, rkeys[i]); err != nil { 172 172 + b.Fatalf("DeleteRecord: %v", err) 173 173 + } 174 174 + } 175 175 + }) 176 176 + 177 177 + b.Run("GetRecord", func(b *testing.B) { 178 178 + op, uid := setup(b) 179 179 + ctx := context.Background() 180 180 + 181 181 + _, _, err := op.PutRecord(ctx, uid, atproto.CrewCollection, "getbench", newCrewRecord("did:plc:getbench")) 182 182 + if err != nil { 183 183 + b.Fatalf("seed PutRecord: %v", err) 184 184 + } 185 185 + 186 186 + b.ResetTimer() 187 187 + for i := 0; i < b.N; i++ { 188 188 + _, _, err := op.GetRecord(ctx, uid, atproto.CrewCollection, "getbench", cid.Undef) 189 189 + if err != nil { 190 190 + b.Fatalf("GetRecord: %v", err) 191 191 + } 192 192 + } 193 193 + }) 194 194 + 195 195 + b.Run("GetRepoRev", func(b *testing.B) { 196 196 + op, uid := setup(b) 197 197 + ctx := context.Background() 198 198 + 199 199 + b.ResetTimer() 200 200 + for i := 0; i < b.N; i++ { 201 201 + _, err := op.GetRepoRev(ctx, uid) 202 202 + if err != nil { 203 203 + b.Fatalf("GetRepoRev: %v", err) 204 204 + } 205 205 + } 206 206 + }) 207 207 + 208 208 + b.Run("GetRepoRoot", func(b *testing.B) { 209 209 + op, uid := setup(b) 210 210 + ctx := context.Background() 211 211 + 212 212 + b.ResetTimer() 213 213 + for i := 0; i < b.N; i++ { 214 214 + _, err := op.GetRepoRoot(ctx, uid) 215 215 + if err != nil { 216 216 + b.Fatalf("GetRepoRoot: %v", err) 217 217 + } 218 218 + } 219 219 + }) 220 220 + 221 221 + // BatchWrite at different sizes — shows commit overhead vs per-record cost 222 222 + for _, size := range []int{1, 10, 100} { 223 223 + b.Run(fmt.Sprintf("BatchWrite_%d", size), func(b *testing.B) { 224 224 + op, uid := setup(b) 225 225 + ctx := context.Background() 226 226 + 227 227 + b.ResetTimer() 228 228 + for i := 0; i < b.N; i++ { 229 229 + writes := make([]*indigoatproto.RepoApplyWrites_Input_Writes_Elem, size) 230 230 + for j := 0; j < size; j++ { 231 231 + rkey := fmt.Sprintf("batch%d_%d", i, j) 232 232 + writes[j] = &indigoatproto.RepoApplyWrites_Input_Writes_Elem{ 233 233 + RepoApplyWrites_Create: &indigoatproto.RepoApplyWrites_Create{ 234 234 + Collection: atproto.CrewCollection, 235 235 + Rkey: &rkey, 236 236 + Value: &lexutil.LexiconTypeDecoder{Val: newCrewRecord(fmt.Sprintf("did:plc:batch%d_%d", i, j))}, 237 237 + }, 238 238 + } 239 239 + } 240 240 + if err := op.BatchWrite(ctx, uid, writes); err != nil { 241 241 + b.Fatalf("BatchWrite: %v", err) 242 242 + } 243 243 + } 244 244 + }) 245 245 + } 246 246 + 247 247 + // ReadRepo at different repo sizes 248 248 + for _, size := range []int{10, 100, 500} { 249 249 + b.Run(fmt.Sprintf("ReadRepo_%drecords", size), func(b *testing.B) { 250 250 + op, uid := setup(b) 251 251 + ctx := context.Background() 252 252 + seedRecords(b, op, uid, size) 253 253 + 254 254 + var buf bytes.Buffer 255 255 + b.ResetTimer() 256 256 + for i := 0; i < b.N; i++ { 257 257 + buf.Reset() 258 258 + if err := op.ReadRepo(ctx, uid, "", &buf); err != nil { 259 259 + b.Fatalf("ReadRepo: %v", err) 260 260 + } 261 261 + } 262 262 + b.SetBytes(int64(buf.Len())) 263 263 + }) 264 264 + } 265 265 + 266 266 + // Event overhead: with vs without handler 267 267 + b.Run("PutRecord_NoEvents", func(b *testing.B) { 268 268 + op, uid := setup(b) 269 269 + ctx := context.Background() 270 270 + // No event handler set 271 271 + 272 272 + b.ResetTimer() 273 273 + for i := 0; i < b.N; i++ { 274 274 + rkey := fmt.Sprintf("noevt%d", i) 275 275 + _, _, err := op.PutRecord(ctx, uid, atproto.CrewCollection, rkey, newCrewRecord(fmt.Sprintf("did:plc:noevt%d", i))) 276 276 + if err != nil { 277 277 + b.Fatalf("PutRecord: %v", err) 278 278 + } 279 279 + } 280 280 + }) 281 281 + 282 282 + b.Run("PutRecord_WithEvents", func(b *testing.B) { 283 283 + op, uid := setup(b) 284 284 + ctx := context.Background() 285 285 + op.SetEventHandler(func(_ context.Context, _ *RepoEvent) {}, false) 286 286 + 287 287 + b.ResetTimer() 288 288 + for i := 0; i < b.N; i++ { 289 289 + rkey := fmt.Sprintf("evt%d", i) 290 290 + _, _, err := op.PutRecord(ctx, uid, atproto.CrewCollection, rkey, newCrewRecord(fmt.Sprintf("did:plc:evt%d", i))) 291 291 + if err != nil { 292 292 + b.Fatalf("PutRecord: %v", err) 293 293 + } 294 294 + } 295 295 + }) 296 296 + 297 297 + b.Run("PutRecord_WithHydration", func(b *testing.B) { 298 298 + op, uid := setup(b) 299 299 + ctx := context.Background() 300 300 + op.SetEventHandler(func(_ context.Context, _ *RepoEvent) {}, true) 301 301 + 302 302 + b.ResetTimer() 303 303 + for i := 0; i < b.N; i++ { 304 304 + rkey := fmt.Sprintf("hyd%d", i) 305 305 + _, _, err := op.PutRecord(ctx, uid, atproto.CrewCollection, rkey, newCrewRecord(fmt.Sprintf("did:plc:hyd%d", i))) 306 306 + if err != nil { 307 307 + b.Fatalf("PutRecord: %v", err) 308 308 + } 309 309 + } 310 310 + }) 311 311 + 312 312 + // Read from a repo with many records (MST depth) 313 313 + b.Run("GetRecord_LargeRepo", func(b *testing.B) { 314 314 + op, uid := setup(b) 315 315 + ctx := context.Background() 316 316 + rkeys := seedRecords(b, op, uid, 500) 317 317 + target := rkeys[len(rkeys)/2] // pick a middle record 318 318 + 319 319 + b.ResetTimer() 320 320 + for i := 0; i < b.N; i++ { 321 321 + _, _, err := op.GetRecord(ctx, uid, atproto.CrewCollection, target, cid.Undef) 322 322 + if err != nil { 323 323 + b.Fatalf("GetRecord: %v", err) 324 324 + } 325 325 + } 326 326 + }) 327 327 + 328 328 + // ReadRepo incremental (since) vs full 329 329 + b.Run("ReadRepo_Incremental", func(b *testing.B) { 330 330 + op, uid := setup(b) 331 331 + ctx := context.Background() 332 332 + seedRecords(b, op, uid, 100) 333 333 + 334 334 + // Get rev before the last write 335 335 + rev, err := op.GetRepoRev(ctx, uid) 336 336 + if err != nil { 337 337 + b.Fatalf("GetRepoRev: %v", err) 338 338 + } 339 339 + 340 340 + // Write one more record 341 341 + _, _, err = op.PutRecord(ctx, uid, atproto.CrewCollection, "incremental", newCrewRecord("did:plc:incremental")) 342 342 + if err != nil { 343 343 + b.Fatalf("PutRecord: %v", err) 344 344 + } 345 345 + 346 346 + var buf bytes.Buffer 347 347 + b.ResetTimer() 348 348 + for i := 0; i < b.N; i++ { 349 349 + buf.Reset() 350 350 + if err := op.ReadRepo(ctx, uid, rev, &buf); err != nil { 351 351 + b.Fatalf("ReadRepo: %v", err) 352 352 + } 353 353 + } 354 354 + b.SetBytes(int64(buf.Len())) 355 355 + }) 356 356 + }) 357 357 + } 358 358 + 359 359 + func BenchmarkRepoManager(b *testing.B) { 360 360 + runRepoOperatorBenchmarks(b, "RepoManager", setupBenchRepoManager) 361 361 + } 362 362 + 363 363 + func BenchmarkDirectRepoOperator(b *testing.B) { 364 364 + runRepoOperatorBenchmarks(b, "DirectRepoOperator", setupBenchDirectRepoOperator) 365 365 + } 366 366 + 367 367 + // BenchmarkBatchVsSingle compares the cost of N individual PutRecord calls 368 368 + // vs a single BatchWrite with N records. 369 369 + func BenchmarkBatchVsSingle(b *testing.B) { 370 370 + for _, impl := range []struct { 371 371 + name string 372 372 + setup benchSetup 373 373 + }{ 374 374 + {"RepoManager", setupBenchRepoManager}, 375 375 + {"DirectRepoOperator", setupBenchDirectRepoOperator}, 376 376 + } { 377 377 + for _, size := range []int{1, 10, 50} { 378 378 + b.Run(fmt.Sprintf("%s/Single_%d", impl.name, size), func(b *testing.B) { 379 379 + op, uid := impl.setup(b) 380 380 + ctx := context.Background() 381 381 + 382 382 + b.ResetTimer() 383 383 + for i := 0; i < b.N; i++ { 384 384 + for j := 0; j < size; j++ { 385 385 + rkey := fmt.Sprintf("single%d_%d", i, j) 386 386 + _, _, err := op.PutRecord(ctx, uid, atproto.CrewCollection, rkey, newCrewRecord(fmt.Sprintf("did:plc:s%d_%d", i, j))) 387 387 + if err != nil { 388 388 + b.Fatalf("PutRecord: %v", err) 389 389 + } 390 390 + } 391 391 + } 392 392 + }) 393 393 + 394 394 + b.Run(fmt.Sprintf("%s/Batch_%d", impl.name, size), func(b *testing.B) { 395 395 + op, uid := impl.setup(b) 396 396 + ctx := context.Background() 397 397 + 398 398 + b.ResetTimer() 399 399 + for i := 0; i < b.N; i++ { 400 400 + writes := make([]*indigoatproto.RepoApplyWrites_Input_Writes_Elem, size) 401 401 + for j := 0; j < size; j++ { 402 402 + rkey := fmt.Sprintf("batch%d_%d", i, j) 403 403 + writes[j] = &indigoatproto.RepoApplyWrites_Input_Writes_Elem{ 404 404 + RepoApplyWrites_Create: &indigoatproto.RepoApplyWrites_Create{ 405 405 + Collection: atproto.CrewCollection, 406 406 + Rkey: &rkey, 407 407 + Value: &lexutil.LexiconTypeDecoder{Val: newCrewRecord(fmt.Sprintf("did:plc:b%d_%d", i, j))}, 408 408 + }, 409 409 + } 410 410 + } 411 411 + if err := op.BatchWrite(ctx, uid, writes); err != nil { 412 412 + b.Fatalf("BatchWrite: %v", err) 413 413 + } 414 414 + } 415 415 + }) 416 416 + } 417 417 + } 418 418 + }

+1413

pkg/hold/pds/repo_operator_test.go

··· 1 1 + package pds 2 2 + 3 3 + import ( 4 4 + "bytes" 5 5 + "context" 6 6 + "fmt" 7 7 + "os" 8 8 + "path/filepath" 9 9 + "strings" 10 10 + "sync" 11 11 + "testing" 12 12 + 13 13 + "atcr.io/pkg/atproto" 14 14 + "atcr.io/pkg/auth/oauth" 15 15 + holddb "atcr.io/pkg/hold/db" 16 16 + indigoatproto "github.com/bluesky-social/indigo/api/atproto" 17 17 + lexutil "github.com/bluesky-social/indigo/lex/util" 18 18 + "github.com/bluesky-social/indigo/models" 19 19 + "github.com/bluesky-social/indigo/repo" 20 20 + "github.com/ipfs/go-cid" 21 21 + ) 22 22 + 23 23 + // setupTestRepoOperator creates a fresh RepoOperator (backed by RepoManager) 24 24 + // and returns it along with the user ID. Each call gets an isolated instance. 25 25 + func setupTestRepoOperator(t *testing.T) (RepoOperator, models.Uid) { 26 26 + t.Helper() 27 27 + ctx := context.Background() 28 28 + tmpDir := t.TempDir() 29 29 + 30 30 + dbPath := ":memory:" 31 31 + keyPath := filepath.Join(tmpDir, "signing-key") 32 32 + 33 33 + if err := os.WriteFile(keyPath, sharedTestKey, 0600); err != nil { 34 34 + t.Fatalf("Failed to write signing key: %v", err) 35 35 + } 36 36 + 37 37 + pds, err := NewHoldPDS(ctx, "did:web:hold.test", "https://hold.test", "https://atcr.io", dbPath, keyPath, false) 38 38 + if err != nil { 39 39 + t.Fatalf("NewHoldPDS: %v", err) 40 40 + } 41 41 + 42 42 + if err := pds.repomgr.InitNewActor(ctx, pds.uid, "", pds.did, "", "", ""); err != nil { 43 43 + t.Fatalf("InitNewActor: %v", err) 44 44 + } 45 45 + 46 46 + t.Cleanup(func() { pds.Close() }) 47 47 + return pds.repomgr, pds.uid 48 48 + } 49 49 + 50 50 + // setupTestDirectRepoOperator creates a fresh DirectRepoOperator 51 51 + // and returns it along with the user ID. 52 52 + func setupTestDirectRepoOperator(t *testing.T) (RepoOperator, models.Uid) { 53 53 + t.Helper() 54 54 + ctx := context.Background() 55 55 + 56 56 + keyPath := filepath.Join(t.TempDir(), "signing-key") 57 57 + if err := os.WriteFile(keyPath, sharedTestKey, 0600); err != nil { 58 58 + t.Fatalf("Failed to write signing key: %v", err) 59 59 + } 60 60 + signingKey, err := oauth.GenerateOrLoadPDSKey(keyPath) 61 61 + if err != nil { 62 62 + t.Fatalf("GenerateOrLoadPDSKey: %v", err) 63 63 + } 64 64 + 65 65 + sqlStore := new(holddb.SQLiteStore) 66 66 + if err := sqlStore.Open(":memory:"); err != nil { 67 67 + t.Fatalf("SQLiteStore.Open: %v", err) 68 68 + } 69 69 + t.Cleanup(func() { sqlStore.Close() }) 70 70 + 71 71 + kmgr := NewHoldKeyManager(signingKey) 72 72 + op := NewDirectRepoOperator(sqlStore, kmgr) 73 73 + 74 74 + uid := models.Uid(1) 75 75 + if err := op.InitNewActor(ctx, uid, "", "did:web:hold.test", "", "", ""); err != nil { 76 76 + t.Fatalf("InitNewActor: %v", err) 77 77 + } 78 78 + 79 79 + return op, uid 80 80 + } 81 81 + 82 82 + // newCrewRecord creates a test crew record with the given member DID. 83 83 + func newCrewRecord(member string) *atproto.CrewRecord { 84 84 + return &atproto.CrewRecord{ 85 85 + Type: atproto.CrewCollection, 86 86 + Member: member, 87 87 + Role: "writer", 88 88 + Permissions: []string{"blob:read", "blob:write"}, 89 89 + AddedAt: "2026-01-01T00:00:00Z", 90 90 + } 91 91 + } 92 92 + 93 93 + // runRepoOperatorTests runs the full RepoOperator test suite against any implementation. 94 94 + // An optional freshSetup function returns an operator WITHOUT InitNewActor pre-called, 95 95 + // for testing InitNewActor event emission. 96 96 + func runRepoOperatorTests(t *testing.T, setup func(t *testing.T) (RepoOperator, models.Uid), freshSetup ...func(t *testing.T) (RepoOperator, models.Uid, string)) { 97 97 + t.Run("CreateRecord", func(t *testing.T) { 98 98 + op, uid := setup(t) 99 99 + ctx := context.Background() 100 100 + 101 101 + rec := newCrewRecord("did:plc:alice") 102 102 + path, cc, err := op.CreateRecord(ctx, uid, atproto.CrewCollection, rec) 103 103 + if err != nil { 104 104 + t.Fatalf("CreateRecord: %v", err) 105 105 + } 106 106 + 107 107 + if !strings.HasPrefix(path, atproto.CrewCollection+"/") { 108 108 + t.Errorf("path should start with collection, got %q", path) 109 109 + } 110 110 + if !cc.Defined() { 111 111 + t.Error("expected defined CID") 112 112 + } 113 113 + 114 114 + // Extract rkey and verify it looks like a TID (13 chars, base32-sortable) 115 115 + rkey := strings.TrimPrefix(path, atproto.CrewCollection+"/") 116 116 + if len(rkey) != 13 { 117 117 + t.Errorf("expected 13-char TID rkey, got %q (len=%d)", rkey, len(rkey)) 118 118 + } 119 119 + 120 120 + // Round-trip via GetRecord 121 121 + gotCid, _, err := op.GetRecord(ctx, uid, atproto.CrewCollection, rkey, cid.Undef) 122 122 + if err != nil { 123 123 + t.Fatalf("GetRecord: %v", err) 124 124 + } 125 125 + if !gotCid.Equals(cc) { 126 126 + t.Errorf("CID mismatch: create=%s get=%s", cc, gotCid) 127 127 + } 128 128 + }) 129 129 + 130 130 + t.Run("UpdateRecord", func(t *testing.T) { 131 131 + op, uid := setup(t) 132 132 + ctx := context.Background() 133 133 + 134 134 + rec := newCrewRecord("did:plc:bob") 135 135 + path, createCid, err := op.CreateRecord(ctx, uid, atproto.CrewCollection, rec) 136 136 + if err != nil { 137 137 + t.Fatalf("CreateRecord: %v", err) 138 138 + } 139 139 + rkey := strings.TrimPrefix(path, atproto.CrewCollection+"/") 140 140 + 141 141 + // Update with different data 142 142 + updated := newCrewRecord("did:plc:bob") 143 143 + updated.Role = "admin" 144 144 + updateCid, err := op.UpdateRecord(ctx, uid, atproto.CrewCollection, rkey, updated) 145 145 + if err != nil { 146 146 + t.Fatalf("UpdateRecord: %v", err) 147 147 + } 148 148 + 149 149 + if createCid.Equals(updateCid) { 150 150 + t.Error("expected CID to change after update") 151 151 + } 152 152 + 153 153 + // Verify new data via GetRecord 154 154 + _, val, err := op.GetRecord(ctx, uid, atproto.CrewCollection, rkey, cid.Undef) 155 155 + if err != nil { 156 156 + t.Fatalf("GetRecord: %v", err) 157 157 + } 158 158 + crew, ok := val.(*atproto.CrewRecord) 159 159 + if !ok { 160 160 + t.Fatalf("expected *CrewRecord, got %T", val) 161 161 + } 162 162 + if crew.Role != "admin" { 163 163 + t.Errorf("expected role=admin, got %q", crew.Role) 164 164 + } 165 165 + }) 166 166 + 167 167 + t.Run("PutRecord", func(t *testing.T) { 168 168 + op, uid := setup(t) 169 169 + ctx := context.Background() 170 170 + 171 171 + rec := newCrewRecord("did:plc:charlie") 172 172 + path, cc, err := op.PutRecord(ctx, uid, atproto.CrewCollection, "mykey", rec) 173 173 + if err != nil { 174 174 + t.Fatalf("PutRecord: %v", err) 175 175 + } 176 176 + 177 177 + if !strings.HasSuffix(path, "/mykey") { 178 178 + t.Errorf("expected path ending in /mykey, got %q", path) 179 179 + } 180 180 + if !cc.Defined() { 181 181 + t.Error("expected defined CID") 182 182 + } 183 183 + 184 184 + // Round-trip 185 185 + gotCid, _, err := op.GetRecord(ctx, uid, atproto.CrewCollection, "mykey", cid.Undef) 186 186 + if err != nil { 187 187 + t.Fatalf("GetRecord: %v", err) 188 188 + } 189 189 + if !gotCid.Equals(cc) { 190 190 + t.Errorf("CID mismatch") 191 191 + } 192 192 + }) 193 193 + 194 194 + t.Run("PutRecord_DuplicateRkey", func(t *testing.T) { 195 195 + op, uid := setup(t) 196 196 + ctx := context.Background() 197 197 + 198 198 + rec := newCrewRecord("did:plc:dave") 199 199 + _, _, err := op.PutRecord(ctx, uid, atproto.CrewCollection, "dupekey", rec) 200 200 + if err != nil { 201 201 + t.Fatalf("first PutRecord: %v", err) 202 202 + } 203 203 + 204 204 + rec2 := newCrewRecord("did:plc:dave2") 205 205 + _, _, err = op.PutRecord(ctx, uid, atproto.CrewCollection, "dupekey", rec2) 206 206 + if err == nil { 207 207 + t.Error("expected error on duplicate rkey PutRecord") 208 208 + } 209 209 + }) 210 210 + 211 211 + t.Run("UpsertRecord_Create", func(t *testing.T) { 212 212 + op, uid := setup(t) 213 213 + ctx := context.Background() 214 214 + 215 215 + rec := newCrewRecord("did:plc:eve") 216 216 + path, cc, created, err := op.UpsertRecord(ctx, uid, atproto.CrewCollection, "upsert1", rec) 217 217 + if err != nil { 218 218 + t.Fatalf("UpsertRecord: %v", err) 219 219 + } 220 220 + 221 221 + if !created { 222 222 + t.Error("expected created=true for new record") 223 223 + } 224 224 + if !strings.HasSuffix(path, "/upsert1") { 225 225 + t.Errorf("expected path ending in /upsert1, got %q", path) 226 226 + } 227 227 + if !cc.Defined() { 228 228 + t.Error("expected defined CID") 229 229 + } 230 230 + 231 231 + // Verify retrievable 232 232 + _, _, err = op.GetRecord(ctx, uid, atproto.CrewCollection, "upsert1", cid.Undef) 233 233 + if err != nil { 234 234 + t.Fatalf("GetRecord after upsert-create: %v", err) 235 235 + } 236 236 + }) 237 237 + 238 238 + t.Run("UpsertRecord_Update", func(t *testing.T) { 239 239 + op, uid := setup(t) 240 240 + ctx := context.Background() 241 241 + 242 242 + rec := newCrewRecord("did:plc:frank") 243 243 + _, cid1, _, err := op.UpsertRecord(ctx, uid, atproto.CrewCollection, "upsert2", rec) 244 244 + if err != nil { 245 245 + t.Fatalf("first UpsertRecord: %v", err) 246 246 + } 247 247 + 248 248 + updated := newCrewRecord("did:plc:frank") 249 249 + updated.Role = "admin" 250 250 + _, cid2, created, err := op.UpsertRecord(ctx, uid, atproto.CrewCollection, "upsert2", updated) 251 251 + if err != nil { 252 252 + t.Fatalf("second UpsertRecord: %v", err) 253 253 + } 254 254 + 255 255 + if created { 256 256 + t.Error("expected created=false for existing record") 257 257 + } 258 258 + if cid1.Equals(cid2) { 259 259 + t.Error("expected CID to change on upsert-update") 260 260 + } 261 261 + }) 262 262 + 263 263 + t.Run("DeleteRecord", func(t *testing.T) { 264 264 + op, uid := setup(t) 265 265 + ctx := context.Background() 266 266 + 267 267 + rec := newCrewRecord("did:plc:grace") 268 268 + path, _, err := op.CreateRecord(ctx, uid, atproto.CrewCollection, rec) 269 269 + if err != nil { 270 270 + t.Fatalf("CreateRecord: %v", err) 271 271 + } 272 272 + rkey := strings.TrimPrefix(path, atproto.CrewCollection+"/") 273 273 + 274 274 + if err := op.DeleteRecord(ctx, uid, atproto.CrewCollection, rkey); err != nil { 275 275 + t.Fatalf("DeleteRecord: %v", err) 276 276 + } 277 277 + 278 278 + _, _, err = op.GetRecord(ctx, uid, atproto.CrewCollection, rkey, cid.Undef) 279 279 + if err == nil { 280 280 + t.Error("expected error getting deleted record") 281 281 + } 282 282 + }) 283 283 + 284 284 + t.Run("DeleteRecord_NotFound", func(t *testing.T) { 285 285 + op, uid := setup(t) 286 286 + ctx := context.Background() 287 287 + 288 288 + err := op.DeleteRecord(ctx, uid, atproto.CrewCollection, "nonexistent") 289 289 + if err == nil { 290 290 + t.Error("expected error deleting non-existent record") 291 291 + } 292 292 + }) 293 293 + 294 294 + t.Run("BatchWrite_CreateAndDelete", func(t *testing.T) { 295 295 + op, uid := setup(t) 296 296 + ctx := context.Background() 297 297 + 298 298 + // First, create a record to delete in the batch 299 299 + rec := newCrewRecord("did:plc:todelete") 300 300 + path, _, err := op.CreateRecord(ctx, uid, atproto.CrewCollection, rec) 301 301 + if err != nil { 302 302 + t.Fatalf("CreateRecord: %v", err) 303 303 + } 304 304 + deleteRkey := strings.TrimPrefix(path, atproto.CrewCollection+"/") 305 305 + 306 306 + // Batch: create 2 + delete 1 307 307 + rkey1 := "batchkey1" 308 308 + rkey2 := "batchkey2" 309 309 + writes := []*indigoatproto.RepoApplyWrites_Input_Writes_Elem{ 310 310 + { 311 311 + RepoApplyWrites_Create: &indigoatproto.RepoApplyWrites_Create{ 312 312 + Collection: atproto.CrewCollection, 313 313 + Rkey: &rkey1, 314 314 + Value: &lexutil.LexiconTypeDecoder{Val: newCrewRecord("did:plc:batch1")}, 315 315 + }, 316 316 + }, 317 317 + { 318 318 + RepoApplyWrites_Create: &indigoatproto.RepoApplyWrites_Create{ 319 319 + Collection: atproto.CrewCollection, 320 320 + Rkey: &rkey2, 321 321 + Value: &lexutil.LexiconTypeDecoder{Val: newCrewRecord("did:plc:batch2")}, 322 322 + }, 323 323 + }, 324 324 + { 325 325 + RepoApplyWrites_Delete: &indigoatproto.RepoApplyWrites_Delete{ 326 326 + Collection: atproto.CrewCollection, 327 327 + Rkey: deleteRkey, 328 328 + }, 329 329 + }, 330 330 + } 331 331 + 332 332 + if err := op.BatchWrite(ctx, uid, writes); err != nil { 333 333 + t.Fatalf("BatchWrite: %v", err) 334 334 + } 335 335 + 336 336 + // Verify created records exist 337 337 + _, _, err = op.GetRecord(ctx, uid, atproto.CrewCollection, rkey1, cid.Undef) 338 338 + if err != nil { 339 339 + t.Errorf("batch-created record 1 not found: %v", err) 340 340 + } 341 341 + _, _, err = op.GetRecord(ctx, uid, atproto.CrewCollection, rkey2, cid.Undef) 342 342 + if err != nil { 343 343 + t.Errorf("batch-created record 2 not found: %v", err) 344 344 + } 345 345 + 346 346 + // Verify deleted record is gone 347 347 + _, _, err = op.GetRecord(ctx, uid, atproto.CrewCollection, deleteRkey, cid.Undef) 348 348 + if err == nil { 349 349 + t.Error("expected batch-deleted record to be gone") 350 350 + } 351 351 + }) 352 352 + 353 353 + t.Run("BulkUpsert", func(t *testing.T) { 354 354 + op, uid := setup(t) 355 355 + ctx := context.Background() 356 356 + 357 357 + records := []BulkRecord{ 358 358 + {Collection: atproto.CrewCollection, Rkey: "bulk1", Data: newCrewRecord("did:plc:bulk1")}, 359 359 + {Collection: atproto.CrewCollection, Rkey: "bulk2", Data: newCrewRecord("did:plc:bulk2")}, 360 360 + } 361 361 + 362 362 + if err := op.BulkUpsert(ctx, uid, records); err != nil { 363 363 + t.Fatalf("BulkUpsert: %v", err) 364 364 + } 365 365 + 366 366 + // Verify both exist 367 367 + _, _, err := op.GetRecord(ctx, uid, atproto.CrewCollection, "bulk1", cid.Undef) 368 368 + if err != nil { 369 369 + t.Errorf("bulk record 1 not found: %v", err) 370 370 + } 371 371 + _, _, err = op.GetRecord(ctx, uid, atproto.CrewCollection, "bulk2", cid.Undef) 372 372 + if err != nil { 373 373 + t.Errorf("bulk record 2 not found: %v", err) 374 374 + } 375 375 + 376 376 + // Re-upsert with changed data 377 377 + updatedRec := newCrewRecord("did:plc:bulk1") 378 378 + updatedRec.Role = "admin" 379 379 + if err := op.BulkUpsert(ctx, uid, []BulkRecord{ 380 380 + {Collection: atproto.CrewCollection, Rkey: "bulk1", Data: updatedRec}, 381 381 + }); err != nil { 382 382 + t.Fatalf("BulkUpsert update: %v", err) 383 383 + } 384 384 + 385 385 + // Verify updated data 386 386 + _, val, err := op.GetRecord(ctx, uid, atproto.CrewCollection, "bulk1", cid.Undef) 387 387 + if err != nil { 388 388 + t.Fatalf("GetRecord after re-upsert: %v", err) 389 389 + } 390 390 + crew, ok := val.(*atproto.CrewRecord) 391 391 + if !ok { 392 392 + t.Fatalf("expected *CrewRecord, got %T", val) 393 393 + } 394 394 + if crew.Role != "admin" { 395 395 + t.Errorf("expected role=admin after re-upsert, got %q", crew.Role) 396 396 + } 397 397 + }) 398 398 + 399 399 + t.Run("GetRecord_CidMatch", func(t *testing.T) { 400 400 + op, uid := setup(t) 401 401 + ctx := context.Background() 402 402 + 403 403 + rec := newCrewRecord("did:plc:cidmatch") 404 404 + _, cc, err := op.PutRecord(ctx, uid, atproto.CrewCollection, "cidtest", rec) 405 405 + if err != nil { 406 406 + t.Fatalf("PutRecord: %v", err) 407 407 + } 408 408 + 409 409 + // Exact CID match succeeds 410 410 + _, _, err = op.GetRecord(ctx, uid, atproto.CrewCollection, "cidtest", cc) 411 411 + if err != nil { 412 412 + t.Errorf("GetRecord with matching CID should succeed: %v", err) 413 413 + } 414 414 + 415 415 + // Wrong CID fails 416 416 + wrongCid, _ := cid.Decode("bafyreigdvqptwntkto5jag4rr7oydencsj4m2t5pdgkhmdwyxlayuncm7e") 417 417 + _, _, err = op.GetRecord(ctx, uid, atproto.CrewCollection, "cidtest", wrongCid) 418 418 + if err == nil { 419 419 + t.Error("expected error with mismatched CID") 420 420 + } 421 421 + }) 422 422 + 423 423 + t.Run("GetRecord_NotFound", func(t *testing.T) { 424 424 + op, uid := setup(t) 425 425 + ctx := context.Background() 426 426 + 427 427 + _, _, err := op.GetRecord(ctx, uid, atproto.CrewCollection, "doesnotexist", cid.Undef) 428 428 + if err == nil { 429 429 + t.Error("expected error for non-existent record") 430 430 + } 431 431 + }) 432 432 + 433 433 + t.Run("GetRecordProof", func(t *testing.T) { 434 434 + op, uid := setup(t) 435 435 + ctx := context.Background() 436 436 + 437 437 + rec := newCrewRecord("did:plc:proof") 438 438 + _, _, err := op.PutRecord(ctx, uid, atproto.CrewCollection, "proofkey", rec) 439 439 + if err != nil { 440 440 + t.Fatalf("PutRecord: %v", err) 441 441 + } 442 442 + 443 443 + head, blocks, err := op.GetRecordProof(ctx, uid, atproto.CrewCollection, "proofkey") 444 444 + if err != nil { 445 445 + t.Fatalf("GetRecordProof: %v", err) 446 446 + } 447 447 + 448 448 + if !head.Defined() { 449 449 + t.Error("expected defined head CID") 450 450 + } 451 451 + if len(blocks) == 0 { 452 452 + t.Error("expected non-empty proof blocks") 453 453 + } 454 454 + }) 455 455 + 456 456 + t.Run("GetRepoRoot", func(t *testing.T) { 457 457 + op, uid := setup(t) 458 458 + ctx := context.Background() 459 459 + 460 460 + root1, err := op.GetRepoRoot(ctx, uid) 461 461 + if err != nil { 462 462 + t.Fatalf("GetRepoRoot: %v", err) 463 463 + } 464 464 + if !root1.Defined() { 465 465 + t.Error("expected defined root CID after InitNewActor") 466 466 + } 467 467 + 468 468 + // Write a record and verify root changes 469 469 + _, _, err = op.PutRecord(ctx, uid, atproto.CrewCollection, "roottest", newCrewRecord("did:plc:root")) 470 470 + if err != nil { 471 471 + t.Fatalf("PutRecord: %v", err) 472 472 + } 473 473 + 474 474 + root2, err := op.GetRepoRoot(ctx, uid) 475 475 + if err != nil { 476 476 + t.Fatalf("GetRepoRoot after write: %v", err) 477 477 + } 478 478 + if root1.Equals(root2) { 479 479 + t.Error("expected root to change after write") 480 480 + } 481 481 + }) 482 482 + 483 483 + t.Run("GetRepoRev", func(t *testing.T) { 484 484 + op, uid := setup(t) 485 485 + ctx := context.Background() 486 486 + 487 487 + rev1, err := op.GetRepoRev(ctx, uid) 488 488 + if err != nil { 489 489 + t.Fatalf("GetRepoRev: %v", err) 490 490 + } 491 491 + if rev1 == "" { 492 492 + t.Error("expected non-empty rev") 493 493 + } 494 494 + 495 495 + // Write a record and verify rev changes 496 496 + _, _, err = op.PutRecord(ctx, uid, atproto.CrewCollection, "revtest", newCrewRecord("did:plc:rev")) 497 497 + if err != nil { 498 498 + t.Fatalf("PutRecord: %v", err) 499 499 + } 500 500 + 501 501 + rev2, err := op.GetRepoRev(ctx, uid) 502 502 + if err != nil { 503 503 + t.Fatalf("GetRepoRev after write: %v", err) 504 504 + } 505 505 + if rev1 == rev2 { 506 506 + t.Error("expected rev to change after write") 507 507 + } 508 508 + }) 509 509 + 510 510 + t.Run("ReadRepo", func(t *testing.T) { 511 511 + op, uid := setup(t) 512 512 + ctx := context.Background() 513 513 + 514 514 + // Write something first 515 515 + _, _, err := op.PutRecord(ctx, uid, atproto.CrewCollection, "readrepo", newCrewRecord("did:plc:readrepo")) 516 516 + if err != nil { 517 517 + t.Fatalf("PutRecord: %v", err) 518 518 + } 519 519 + 520 520 + var buf bytes.Buffer 521 521 + if err := op.ReadRepo(ctx, uid, "", &buf); err != nil { 522 522 + t.Fatalf("ReadRepo: %v", err) 523 523 + } 524 524 + 525 525 + if buf.Len() == 0 { 526 526 + t.Error("expected non-empty CAR output") 527 527 + } 528 528 + }) 529 529 + 530 530 + t.Run("EventEmission_Create", func(t *testing.T) { 531 531 + op, uid := setup(t) 532 532 + ctx := context.Background() 533 533 + 534 534 + var events []*RepoEvent 535 535 + op.SetEventHandler(func(_ context.Context, evt *RepoEvent) { 536 536 + events = append(events, evt) 537 537 + }, false) 538 538 + 539 539 + rec := newCrewRecord("did:plc:evt-create") 540 540 + _, _, err := op.PutRecord(ctx, uid, atproto.CrewCollection, "evtcreate", rec) 541 541 + if err != nil { 542 542 + t.Fatalf("PutRecord: %v", err) 543 543 + } 544 544 + 545 545 + if len(events) != 1 { 546 546 + t.Fatalf("expected 1 event, got %d", len(events)) 547 547 + } 548 548 + 549 549 + evt := events[0] 550 550 + if evt.User != uid { 551 551 + t.Errorf("expected user=%d, got %d", uid, evt.User) 552 552 + } 553 553 + if !evt.NewRoot.Defined() { 554 554 + t.Error("expected defined NewRoot") 555 555 + } 556 556 + if evt.PrevData == nil { 557 557 + t.Error("expected non-nil PrevData") 558 558 + } 559 559 + if evt.Rev == "" { 560 560 + t.Error("expected non-empty Rev") 561 561 + } 562 562 + if len(evt.RepoSlice) == 0 { 563 563 + t.Error("expected non-empty RepoSlice") 564 564 + } 565 565 + if len(evt.Ops) != 1 { 566 566 + t.Fatalf("expected 1 op, got %d", len(evt.Ops)) 567 567 + } 568 568 + 569 569 + op0 := evt.Ops[0] 570 570 + if op0.Kind != EvtKindCreateRecord { 571 571 + t.Errorf("expected kind=create, got %q", op0.Kind) 572 572 + } 573 573 + if op0.Collection != atproto.CrewCollection { 574 574 + t.Errorf("expected collection=%s, got %q", atproto.CrewCollection, op0.Collection) 575 575 + } 576 576 + if op0.Rkey != "evtcreate" { 577 577 + t.Errorf("expected rkey=evtcreate, got %q", op0.Rkey) 578 578 + } 579 579 + if op0.RecCid == nil { 580 580 + t.Error("expected non-nil RecCid for create op") 581 581 + } 582 582 + }) 583 583 + 584 584 + t.Run("EventEmission_Update_PrevData", func(t *testing.T) { 585 585 + op, uid := setup(t) 586 586 + ctx := context.Background() 587 587 + 588 588 + // Create first (no event handler yet) 589 589 + rec := newCrewRecord("did:plc:evt-update") 590 590 + _, _, err := op.PutRecord(ctx, uid, atproto.CrewCollection, "evtupdate", rec) 591 591 + if err != nil { 592 592 + t.Fatalf("PutRecord: %v", err) 593 593 + } 594 594 + 595 595 + // Now set handler and update 596 596 + var events []*RepoEvent 597 597 + op.SetEventHandler(func(_ context.Context, evt *RepoEvent) { 598 598 + events = append(events, evt) 599 599 + }, false) 600 600 + 601 601 + updated := newCrewRecord("did:plc:evt-update") 602 602 + updated.Role = "admin" 603 603 + _, err = op.UpdateRecord(ctx, uid, atproto.CrewCollection, "evtupdate", updated) 604 604 + if err != nil { 605 605 + t.Fatalf("UpdateRecord: %v", err) 606 606 + } 607 607 + 608 608 + if len(events) != 1 { 609 609 + t.Fatalf("expected 1 event, got %d", len(events)) 610 610 + } 611 611 + 612 612 + evt := events[0] 613 613 + if evt.PrevData == nil { 614 614 + t.Error("expected non-nil PrevData on update event") 615 615 + } 616 616 + if evt.OldRoot == nil { 617 617 + t.Error("expected non-nil OldRoot on update event") 618 618 + } 619 619 + if evt.Since == nil { 620 620 + t.Error("expected non-nil Since on update event") 621 621 + } 622 622 + if len(evt.Ops) != 1 { 623 623 + t.Fatalf("expected 1 op, got %d", len(evt.Ops)) 624 624 + } 625 625 + if evt.Ops[0].Kind != EvtKindUpdateRecord { 626 626 + t.Errorf("expected kind=update, got %q", evt.Ops[0].Kind) 627 627 + } 628 628 + }) 629 629 + 630 630 + t.Run("EventEmission_Delete", func(t *testing.T) { 631 631 + op, uid := setup(t) 632 632 + ctx := context.Background() 633 633 + 634 634 + rec := newCrewRecord("did:plc:evt-delete") 635 635 + _, _, err := op.PutRecord(ctx, uid, atproto.CrewCollection, "evtdelete", rec) 636 636 + if err != nil { 637 637 + t.Fatalf("PutRecord: %v", err) 638 638 + } 639 639 + 640 640 + var events []*RepoEvent 641 641 + op.SetEventHandler(func(_ context.Context, evt *RepoEvent) { 642 642 + events = append(events, evt) 643 643 + }, false) 644 644 + 645 645 + if err := op.DeleteRecord(ctx, uid, atproto.CrewCollection, "evtdelete"); err != nil { 646 646 + t.Fatalf("DeleteRecord: %v", err) 647 647 + } 648 648 + 649 649 + if len(events) != 1 { 650 650 + t.Fatalf("expected 1 event, got %d", len(events)) 651 651 + } 652 652 + 653 653 + evt := events[0] 654 654 + if len(evt.Ops) != 1 { 655 655 + t.Fatalf("expected 1 op, got %d", len(evt.Ops)) 656 656 + } 657 657 + if evt.Ops[0].Kind != EvtKindDeleteRecord { 658 658 + t.Errorf("expected kind=delete, got %q", evt.Ops[0].Kind) 659 659 + } 660 660 + if evt.Ops[0].RecCid != nil { 661 661 + t.Error("expected nil RecCid for delete op") 662 662 + } 663 663 + }) 664 664 + 665 665 + t.Run("EventEmission_Hydrate", func(t *testing.T) { 666 666 + op, uid := setup(t) 667 667 + ctx := context.Background() 668 668 + 669 669 + var events []*RepoEvent 670 670 + op.SetEventHandler(func(_ context.Context, evt *RepoEvent) { 671 671 + events = append(events, evt) 672 672 + }, true) // hydrateRecords = true 673 673 + 674 674 + rec := newCrewRecord("did:plc:evt-hydrate") 675 675 + _, _, err := op.PutRecord(ctx, uid, atproto.CrewCollection, "evthydrate", rec) 676 676 + if err != nil { 677 677 + t.Fatalf("PutRecord: %v", err) 678 678 + } 679 679 + 680 680 + if len(events) != 1 { 681 681 + t.Fatalf("expected 1 event, got %d", len(events)) 682 682 + } 683 683 + if events[0].Ops[0].Record == nil { 684 684 + t.Error("expected non-nil Record when hydrateRecords=true") 685 685 + } 686 686 + }) 687 687 + 688 688 + // --- Error path and edge case tests --- 689 689 + 690 690 + t.Run("UpdateRecord_NotFound", func(t *testing.T) { 691 691 + op, uid := setup(t) 692 692 + ctx := context.Background() 693 693 + 694 694 + rec := newCrewRecord("did:plc:ghost") 695 695 + _, err := op.UpdateRecord(ctx, uid, atproto.CrewCollection, "nonexistent", rec) 696 696 + if err == nil { 697 697 + t.Error("expected error updating non-existent record") 698 698 + } 699 699 + }) 700 700 + 701 701 + t.Run("UpdateRecord_Hydrate", func(t *testing.T) { 702 702 + op, uid := setup(t) 703 703 + ctx := context.Background() 704 704 + 705 705 + rec := newCrewRecord("did:plc:hydrate-upd") 706 706 + _, _, err := op.PutRecord(ctx, uid, atproto.CrewCollection, "hydupd", rec) 707 707 + if err != nil { 708 708 + t.Fatalf("PutRecord: %v", err) 709 709 + } 710 710 + 711 711 + var events []*RepoEvent 712 712 + op.SetEventHandler(func(_ context.Context, evt *RepoEvent) { 713 713 + events = append(events, evt) 714 714 + }, true) 715 715 + 716 716 + updated := newCrewRecord("did:plc:hydrate-upd") 717 717 + updated.Role = "admin" 718 718 + _, err = op.UpdateRecord(ctx, uid, atproto.CrewCollection, "hydupd", updated) 719 719 + if err != nil { 720 720 + t.Fatalf("UpdateRecord: %v", err) 721 721 + } 722 722 + 723 723 + if len(events) != 1 { 724 724 + t.Fatalf("expected 1 event, got %d", len(events)) 725 725 + } 726 726 + if events[0].Ops[0].Record == nil { 727 727 + t.Error("expected non-nil Record on update with hydrateRecords=true") 728 728 + } 729 729 + }) 730 730 + 731 731 + t.Run("PutRecord_Hydrate", func(t *testing.T) { 732 732 + op, uid := setup(t) 733 733 + ctx := context.Background() 734 734 + 735 735 + var events []*RepoEvent 736 736 + op.SetEventHandler(func(_ context.Context, evt *RepoEvent) { 737 737 + events = append(events, evt) 738 738 + }, true) 739 739 + 740 740 + rec := newCrewRecord("did:plc:hydrate-put") 741 741 + _, _, err := op.PutRecord(ctx, uid, atproto.CrewCollection, "hydput", rec) 742 742 + if err != nil { 743 743 + t.Fatalf("PutRecord: %v", err) 744 744 + } 745 745 + 746 746 + if len(events) != 1 { 747 747 + t.Fatalf("expected 1 event, got %d", len(events)) 748 748 + } 749 749 + if events[0].Ops[0].Record == nil { 750 750 + t.Error("expected non-nil Record on put with hydrateRecords=true") 751 751 + } 752 752 + }) 753 753 + 754 754 + t.Run("InitNewActor_EmptyDID", func(t *testing.T) { 755 755 + op, uid := setup(t) 756 756 + ctx := context.Background() 757 757 + 758 758 + err := op.InitNewActor(ctx, uid, "", "", "", "", "") 759 759 + if err == nil { 760 760 + t.Error("expected error for empty DID") 761 761 + } 762 762 + }) 763 763 + 764 764 + t.Run("InitNewActor_ZeroUser", func(t *testing.T) { 765 765 + op, _ := setup(t) 766 766 + ctx := context.Background() 767 767 + 768 768 + err := op.InitNewActor(ctx, 0, "", "did:web:test", "", "", "") 769 769 + if err == nil { 770 770 + t.Error("expected error for zero user") 771 771 + } 772 772 + }) 773 773 + 774 774 + t.Run("InitNewActor_EventEmission", func(t *testing.T) { 775 775 + if len(freshSetup) == 0 || freshSetup[0] == nil { 776 776 + t.Skip("no freshSetup provided") 777 777 + } 778 778 + 779 779 + op, uid, did := freshSetup[0](t) 780 780 + ctx := context.Background() 781 781 + 782 782 + var events []*RepoEvent 783 783 + op.SetEventHandler(func(_ context.Context, evt *RepoEvent) { 784 784 + events = append(events, evt) 785 785 + }, true) 786 786 + 787 787 + err := op.InitNewActor(ctx, uid, "", did, "Test User", "", "") 788 788 + if err != nil { 789 789 + t.Fatalf("InitNewActor: %v", err) 790 790 + } 791 791 + 792 792 + if len(events) != 1 { 793 793 + t.Fatalf("expected 1 event from InitNewActor, got %d", len(events)) 794 794 + } 795 795 + evt := events[0] 796 796 + if len(evt.Ops) != 1 { 797 797 + t.Fatalf("expected 1 op, got %d", len(evt.Ops)) 798 798 + } 799 799 + if evt.Ops[0].Kind != EvtKindCreateRecord { 800 800 + t.Errorf("expected kind=create, got %q", evt.Ops[0].Kind) 801 801 + } 802 802 + if evt.Ops[0].Collection != "app.bsky.actor.profile" { 803 803 + t.Errorf("expected collection=app.bsky.actor.profile, got %q", evt.Ops[0].Collection) 804 804 + } 805 805 + if evt.Ops[0].Record == nil { 806 806 + t.Error("expected non-nil Record with hydrateRecords=true") 807 807 + } 808 808 + }) 809 809 + 810 810 + t.Run("GetRecordProof_NotFound", func(t *testing.T) { 811 811 + op, uid := setup(t) 812 812 + ctx := context.Background() 813 813 + 814 814 + _, _, err := op.GetRecordProof(ctx, uid, atproto.CrewCollection, "nonexistent") 815 815 + if err == nil { 816 816 + t.Error("expected error for proof of non-existent record") 817 817 + } 818 818 + }) 819 819 + 820 820 + t.Run("GetRecordProof_NoRepo", func(t *testing.T) { 821 821 + // Use a user ID that has no repo initialized — triggers OpenRepo error 822 822 + op, _ := setup(t) 823 823 + ctx := context.Background() 824 824 + 825 825 + _, _, err := op.GetRecordProof(ctx, models.Uid(9999), atproto.CrewCollection, "anything") 826 826 + if err == nil { 827 827 + t.Error("expected error for user with no repo") 828 828 + } 829 829 + }) 830 830 + 831 831 + t.Run("BatchWrite_Update", func(t *testing.T) { 832 832 + op, uid := setup(t) 833 833 + ctx := context.Background() 834 834 + 835 835 + // NOTE: BatchWrite internally uses r.PutRecord for updates (mst.Add), 836 836 + // which means the update write type is effectively a create-or-fail. 837 837 + // This test uses a fresh rkey to exercise the update code path. 838 838 + updated := newCrewRecord("did:plc:batchupd") 839 839 + updated.Role = "admin" 840 840 + writes := []*indigoatproto.RepoApplyWrites_Input_Writes_Elem{ 841 841 + { 842 842 + RepoApplyWrites_Update: &indigoatproto.RepoApplyWrites_Update{ 843 843 + Collection: atproto.CrewCollection, 844 844 + Rkey: "batchupd", 845 845 + Value: &lexutil.LexiconTypeDecoder{Val: updated}, 846 846 + }, 847 847 + }, 848 848 + } 849 849 + 850 850 + if err := op.BatchWrite(ctx, uid, writes); err != nil { 851 851 + t.Fatalf("BatchWrite update: %v", err) 852 852 + } 853 853 + 854 854 + // Verify data was written 855 855 + _, val, err := op.GetRecord(ctx, uid, atproto.CrewCollection, "batchupd", cid.Undef) 856 856 + if err != nil { 857 857 + t.Fatalf("GetRecord after batch update: %v", err) 858 858 + } 859 859 + crew, ok := val.(*atproto.CrewRecord) 860 860 + if !ok { 861 861 + t.Fatalf("expected *CrewRecord, got %T", val) 862 862 + } 863 863 + if crew.Role != "admin" { 864 864 + t.Errorf("expected role=admin, got %q", crew.Role) 865 865 + } 866 866 + }) 867 867 + 868 868 + t.Run("BatchWrite_AutoRkey", func(t *testing.T) { 869 869 + op, uid := setup(t) 870 870 + ctx := context.Background() 871 871 + 872 872 + var events []*RepoEvent 873 873 + op.SetEventHandler(func(_ context.Context, evt *RepoEvent) { 874 874 + events = append(events, evt) 875 875 + }, false) 876 876 + 877 877 + // Create with nil Rkey — should auto-generate TID 878 878 + writes := []*indigoatproto.RepoApplyWrites_Input_Writes_Elem{ 879 879 + { 880 880 + RepoApplyWrites_Create: &indigoatproto.RepoApplyWrites_Create{ 881 881 + Collection: atproto.CrewCollection, 882 882 + Rkey: nil, // auto-generate 883 883 + Value: &lexutil.LexiconTypeDecoder{Val: newCrewRecord("did:plc:autorkey")}, 884 884 + }, 885 885 + }, 886 886 + } 887 887 + 888 888 + if err := op.BatchWrite(ctx, uid, writes); err != nil { 889 889 + t.Fatalf("BatchWrite: %v", err) 890 890 + } 891 891 + 892 892 + if len(events) != 1 { 893 893 + t.Fatalf("expected 1 event, got %d", len(events)) 894 894 + } 895 895 + rkey := events[0].Ops[0].Rkey 896 896 + if len(rkey) != 13 { 897 897 + t.Errorf("expected 13-char auto-generated TID rkey, got %q (len=%d)", rkey, len(rkey)) 898 898 + } 899 899 + 900 900 + // Verify the record exists 901 901 + _, _, err := op.GetRecord(ctx, uid, atproto.CrewCollection, rkey, cid.Undef) 902 902 + if err != nil { 903 903 + t.Errorf("auto-rkey record not found: %v", err) 904 904 + } 905 905 + }) 906 906 + 907 907 + t.Run("BatchWrite_DeleteNotFound", func(t *testing.T) { 908 908 + op, uid := setup(t) 909 909 + ctx := context.Background() 910 910 + 911 911 + writes := []*indigoatproto.RepoApplyWrites_Input_Writes_Elem{ 912 912 + { 913 913 + RepoApplyWrites_Delete: &indigoatproto.RepoApplyWrites_Delete{ 914 914 + Collection: atproto.CrewCollection, 915 915 + Rkey: "nonexistent", 916 916 + }, 917 917 + }, 918 918 + } 919 919 + 920 920 + err := op.BatchWrite(ctx, uid, writes) 921 921 + if err == nil { 922 922 + t.Error("expected error deleting non-existent record in batch") 923 923 + } 924 924 + }) 925 925 + 926 926 + t.Run("BatchWrite_EmptyWriteElem", func(t *testing.T) { 927 927 + op, uid := setup(t) 928 928 + ctx := context.Background() 929 929 + 930 930 + // Write elem with no operation set 931 931 + writes := []*indigoatproto.RepoApplyWrites_Input_Writes_Elem{ 932 932 + {}, // all nil 933 933 + } 934 934 + 935 935 + err := op.BatchWrite(ctx, uid, writes) 936 936 + if err == nil { 937 937 + t.Error("expected error for empty write elem") 938 938 + } 939 939 + }) 940 940 + 941 941 + t.Run("BatchWrite_EventEmission", func(t *testing.T) { 942 942 + op, uid := setup(t) 943 943 + ctx := context.Background() 944 944 + 945 945 + // Create a record to delete 946 946 + _, _, err := op.PutRecord(ctx, uid, atproto.CrewCollection, "batchevtdel", newCrewRecord("did:plc:batchevtdel")) 947 947 + if err != nil { 948 948 + t.Fatalf("PutRecord: %v", err) 949 949 + } 950 950 + 951 951 + var events []*RepoEvent 952 952 + op.SetEventHandler(func(_ context.Context, evt *RepoEvent) { 953 953 + events = append(events, evt) 954 954 + }, true) 955 955 + 956 956 + rkey := "batchevt1" 957 957 + writes := []*indigoatproto.RepoApplyWrites_Input_Writes_Elem{ 958 958 + { 959 959 + RepoApplyWrites_Create: &indigoatproto.RepoApplyWrites_Create{ 960 960 + Collection: atproto.CrewCollection, 961 961 + Rkey: &rkey, 962 962 + Value: &lexutil.LexiconTypeDecoder{Val: newCrewRecord("did:plc:batchevt1")}, 963 963 + }, 964 964 + }, 965 965 + { 966 966 + RepoApplyWrites_Delete: &indigoatproto.RepoApplyWrites_Delete{ 967 967 + Collection: atproto.CrewCollection, 968 968 + Rkey: "batchevtdel", 969 969 + }, 970 970 + }, 971 971 + } 972 972 + 973 973 + if err := op.BatchWrite(ctx, uid, writes); err != nil { 974 974 + t.Fatalf("BatchWrite: %v", err) 975 975 + } 976 976 + 977 977 + if len(events) != 1 { 978 978 + t.Fatalf("expected 1 event, got %d", len(events)) 979 979 + } 980 980 + 981 981 + evt := events[0] 982 982 + if len(evt.Ops) != 2 { 983 983 + t.Fatalf("expected 2 ops in batch event, got %d", len(evt.Ops)) 984 984 + } 985 985 + if evt.PrevData == nil { 986 986 + t.Error("expected non-nil PrevData") 987 987 + } 988 988 + if evt.OldRoot == nil { 989 989 + t.Error("expected non-nil OldRoot") 990 990 + } 991 991 + 992 992 + // First op: create with hydrated record 993 993 + if evt.Ops[0].Kind != EvtKindCreateRecord { 994 994 + t.Errorf("op[0] expected kind=create, got %q", evt.Ops[0].Kind) 995 995 + } 996 996 + if evt.Ops[0].Record == nil { 997 997 + t.Error("op[0] expected non-nil Record with hydrateRecords=true") 998 998 + } 999 999 + 1000 1000 + // Second op: delete 1001 1001 + if evt.Ops[1].Kind != EvtKindDeleteRecord { 1002 1002 + t.Errorf("op[1] expected kind=delete, got %q", evt.Ops[1].Kind) 1003 1003 + } 1004 1004 + }) 1005 1005 + 1006 1006 + t.Run("BatchWrite_UpdateHydrate", func(t *testing.T) { 1007 1007 + op, uid := setup(t) 1008 1008 + ctx := context.Background() 1009 1009 + 1010 1010 + // NOTE: BatchWrite uses r.PutRecord for updates (mst.Add), so use a fresh rkey. 1011 1011 + var events []*RepoEvent 1012 1012 + op.SetEventHandler(func(_ context.Context, evt *RepoEvent) { 1013 1013 + events = append(events, evt) 1014 1014 + }, true) 1015 1015 + 1016 1016 + updated := newCrewRecord("did:plc:batchhydupd") 1017 1017 + updated.Role = "admin" 1018 1018 + writes := []*indigoatproto.RepoApplyWrites_Input_Writes_Elem{ 1019 1019 + { 1020 1020 + RepoApplyWrites_Update: &indigoatproto.RepoApplyWrites_Update{ 1021 1021 + Collection: atproto.CrewCollection, 1022 1022 + Rkey: "batchhydupd", 1023 1023 + Value: &lexutil.LexiconTypeDecoder{Val: updated}, 1024 1024 + }, 1025 1025 + }, 1026 1026 + } 1027 1027 + 1028 1028 + if err := op.BatchWrite(ctx, uid, writes); err != nil { 1029 1029 + t.Fatalf("BatchWrite: %v", err) 1030 1030 + } 1031 1031 + 1032 1032 + if len(events) != 1 { 1033 1033 + t.Fatalf("expected 1 event, got %d", len(events)) 1034 1034 + } 1035 1035 + if events[0].Ops[0].Kind != EvtKindUpdateRecord { 1036 1036 + t.Errorf("expected kind=update, got %q", events[0].Ops[0].Kind) 1037 1037 + } 1038 1038 + if events[0].Ops[0].Record == nil { 1039 1039 + t.Error("expected non-nil Record on batch update with hydrateRecords=true") 1040 1040 + } 1041 1041 + }) 1042 1042 + 1043 1043 + t.Run("BulkUpsert_EventEmission", func(t *testing.T) { 1044 1044 + op, uid := setup(t) 1045 1045 + ctx := context.Background() 1046 1046 + 1047 1047 + var events []*RepoEvent 1048 1048 + op.SetEventHandler(func(_ context.Context, evt *RepoEvent) { 1049 1049 + events = append(events, evt) 1050 1050 + }, true) 1051 1051 + 1052 1052 + records := []BulkRecord{ 1053 1053 + {Collection: atproto.CrewCollection, Rkey: "bulkevt1", Data: newCrewRecord("did:plc:bulkevt1")}, 1054 1054 + {Collection: atproto.CrewCollection, Rkey: "bulkevt2", Data: newCrewRecord("did:plc:bulkevt2")}, 1055 1055 + } 1056 1056 + 1057 1057 + if err := op.BulkUpsert(ctx, uid, records); err != nil { 1058 1058 + t.Fatalf("BulkUpsert: %v", err) 1059 1059 + } 1060 1060 + 1061 1061 + if len(events) != 1 { 1062 1062 + t.Fatalf("expected 1 event, got %d", len(events)) 1063 1063 + } 1064 1064 + if len(events[0].Ops) != 2 { 1065 1065 + t.Fatalf("expected 2 ops, got %d", len(events[0].Ops)) 1066 1066 + } 1067 1067 + // Both should be creates since records are new 1068 1068 + for i, eop := range events[0].Ops { 1069 1069 + if eop.Kind != EvtKindCreateRecord { 1070 1070 + t.Errorf("op[%d] expected kind=create, got %q", i, eop.Kind) 1071 1071 + } 1072 1072 + } 1073 1073 + }) 1074 1074 + 1075 1075 + t.Run("ReadRepo_WithSince", func(t *testing.T) { 1076 1076 + op, uid := setup(t) 1077 1077 + ctx := context.Background() 1078 1078 + 1079 1079 + // Get initial rev 1080 1080 + rev1, err := op.GetRepoRev(ctx, uid) 1081 1081 + if err != nil { 1082 1082 + t.Fatalf("GetRepoRev: %v", err) 1083 1083 + } 1084 1084 + 1085 1085 + // Write a record 1086 1086 + _, _, err = op.PutRecord(ctx, uid, atproto.CrewCollection, "since1", newCrewRecord("did:plc:since1")) 1087 1087 + if err != nil { 1088 1088 + t.Fatalf("PutRecord: %v", err) 1089 1089 + } 1090 1090 + 1091 1091 + // ReadRepo with since should produce smaller output than full export 1092 1092 + var fullBuf bytes.Buffer 1093 1093 + if err := op.ReadRepo(ctx, uid, "", &fullBuf); err != nil { 1094 1094 + t.Fatalf("ReadRepo full: %v", err) 1095 1095 + } 1096 1096 + 1097 1097 + var sinceBuf bytes.Buffer 1098 1098 + if err := op.ReadRepo(ctx, uid, rev1, &sinceBuf); err != nil { 1099 1099 + t.Fatalf("ReadRepo since: %v", err) 1100 1100 + } 1101 1101 + 1102 1102 + if sinceBuf.Len() == 0 { 1103 1103 + t.Error("expected non-empty incremental CAR") 1104 1104 + } 1105 1105 + if sinceBuf.Len() >= fullBuf.Len() { 1106 1106 + t.Errorf("expected incremental CAR (%d) < full CAR (%d)", sinceBuf.Len(), fullBuf.Len()) 1107 1107 + } 1108 1108 + }) 1109 1109 + 1110 1110 + t.Run("CreateRecord_NoEventWithoutHandler", func(t *testing.T) { 1111 1111 + op, uid := setup(t) 1112 1112 + ctx := context.Background() 1113 1113 + 1114 1114 + // Don't set event handler — should not panic 1115 1115 + rec := newCrewRecord("did:plc:nohandler") 1116 1116 + _, _, err := op.CreateRecord(ctx, uid, atproto.CrewCollection, rec) 1117 1117 + if err != nil { 1118 1118 + t.Fatalf("CreateRecord without handler: %v", err) 1119 1119 + } 1120 1120 + }) 1121 1121 + 1122 1122 + t.Run("CreateRecord_AlwaysHydrates", func(t *testing.T) { 1123 1123 + op, uid := setup(t) 1124 1124 + ctx := context.Background() 1125 1125 + 1126 1126 + var events []*RepoEvent 1127 1127 + op.SetEventHandler(func(_ context.Context, evt *RepoEvent) { 1128 1128 + events = append(events, evt) 1129 1129 + }, false) // hydration OFF 1130 1130 + 1131 1131 + rec := newCrewRecord("did:plc:always-hydrate") 1132 1132 + _, _, err := op.CreateRecord(ctx, uid, atproto.CrewCollection, rec) 1133 1133 + if err != nil { 1134 1134 + t.Fatalf("CreateRecord: %v", err) 1135 1135 + } 1136 1136 + 1137 1137 + if len(events) != 1 { 1138 1138 + t.Fatalf("expected 1 event, got %d", len(events)) 1139 1139 + } 1140 1140 + // CreateRecord always includes Record regardless of hydrateRecords 1141 1141 + if events[0].Ops[0].Record == nil { 1142 1142 + t.Error("expected non-nil Record on CreateRecord even with hydrateRecords=false") 1143 1143 + } 1144 1144 + }) 1145 1145 + 1146 1146 + t.Run("PutRecord_NoHydrate", func(t *testing.T) { 1147 1147 + op, uid := setup(t) 1148 1148 + ctx := context.Background() 1149 1149 + 1150 1150 + var events []*RepoEvent 1151 1151 + op.SetEventHandler(func(_ context.Context, evt *RepoEvent) { 1152 1152 + events = append(events, evt) 1153 1153 + }, false) // hydration OFF 1154 1154 + 1155 1155 + rec := newCrewRecord("did:plc:put-nohydrate") 1156 1156 + _, _, err := op.PutRecord(ctx, uid, atproto.CrewCollection, "putnohydrate", rec) 1157 1157 + if err != nil { 1158 1158 + t.Fatalf("PutRecord: %v", err) 1159 1159 + } 1160 1160 + 1161 1161 + if len(events) != 1 { 1162 1162 + t.Fatalf("expected 1 event, got %d", len(events)) 1163 1163 + } 1164 1164 + if events[0].Ops[0].Record != nil { 1165 1165 + t.Error("expected nil Record on PutRecord with hydrateRecords=false") 1166 1166 + } 1167 1167 + }) 1168 1168 + 1169 1169 + t.Run("UpdateRecord_NoHydrate", func(t *testing.T) { 1170 1170 + op, uid := setup(t) 1171 1171 + ctx := context.Background() 1172 1172 + 1173 1173 + // Create record first (no handler) 1174 1174 + rec := newCrewRecord("did:plc:upd-nohydrate") 1175 1175 + _, _, err := op.PutRecord(ctx, uid, atproto.CrewCollection, "updnohydrate", rec) 1176 1176 + if err != nil { 1177 1177 + t.Fatalf("PutRecord: %v", err) 1178 1178 + } 1179 1179 + 1180 1180 + var events []*RepoEvent 1181 1181 + op.SetEventHandler(func(_ context.Context, evt *RepoEvent) { 1182 1182 + events = append(events, evt) 1183 1183 + }, false) // hydration OFF 1184 1184 + 1185 1185 + updated := newCrewRecord("did:plc:upd-nohydrate") 1186 1186 + updated.Role = "admin" 1187 1187 + _, err = op.UpdateRecord(ctx, uid, atproto.CrewCollection, "updnohydrate", updated) 1188 1188 + if err != nil { 1189 1189 + t.Fatalf("UpdateRecord: %v", err) 1190 1190 + } 1191 1191 + 1192 1192 + if len(events) != 1 { 1193 1193 + t.Fatalf("expected 1 event, got %d", len(events)) 1194 1194 + } 1195 1195 + if events[0].Ops[0].Record != nil { 1196 1196 + t.Error("expected nil Record on UpdateRecord with hydrateRecords=false") 1197 1197 + } 1198 1198 + }) 1199 1199 + 1200 1200 + t.Run("BulkUpsert_NeverHydrates", func(t *testing.T) { 1201 1201 + op, uid := setup(t) 1202 1202 + ctx := context.Background() 1203 1203 + 1204 1204 + var events []*RepoEvent 1205 1205 + op.SetEventHandler(func(_ context.Context, evt *RepoEvent) { 1206 1206 + events = append(events, evt) 1207 1207 + }, true) // hydration ON — but BulkUpsert should still not hydrate 1208 1208 + 1209 1209 + records := []BulkRecord{ 1210 1210 + {Collection: atproto.CrewCollection, Rkey: "bulknohydrate", Data: newCrewRecord("did:plc:bulk-nohydrate")}, 1211 1211 + } 1212 1212 + if err := op.BulkUpsert(ctx, uid, records); err != nil { 1213 1213 + t.Fatalf("BulkUpsert: %v", err) 1214 1214 + } 1215 1215 + 1216 1216 + if len(events) != 1 { 1217 1217 + t.Fatalf("expected 1 event, got %d", len(events)) 1218 1218 + } 1219 1219 + if events[0].Ops[0].Record != nil { 1220 1220 + t.Error("expected nil Record on BulkUpsert even with hydrateRecords=true") 1221 1221 + } 1222 1222 + }) 1223 1223 + 1224 1224 + t.Run("RevChain_Sequential", func(t *testing.T) { 1225 1225 + op, uid := setup(t) 1226 1226 + ctx := context.Background() 1227 1227 + 1228 1228 + var events []*RepoEvent 1229 1229 + op.SetEventHandler(func(_ context.Context, evt *RepoEvent) { 1230 1230 + events = append(events, evt) 1231 1231 + }, false) 1232 1232 + 1233 1233 + // Get initial rev 1234 1234 + rev0, err := op.GetRepoRev(ctx, uid) 1235 1235 + if err != nil { 1236 1236 + t.Fatalf("GetRepoRev: %v", err) 1237 1237 + } 1238 1238 + 1239 1239 + // Write 3 records sequentially 1240 1240 + for i := 0; i < 3; i++ { 1241 1241 + _, _, err := op.PutRecord(ctx, uid, atproto.CrewCollection, fmt.Sprintf("chain%d", i), newCrewRecord(fmt.Sprintf("did:plc:chain%d", i))) 1242 1242 + if err != nil { 1243 1243 + t.Fatalf("PutRecord %d: %v", i, err) 1244 1244 + } 1245 1245 + } 1246 1246 + 1247 1247 + if len(events) != 3 { 1248 1248 + t.Fatalf("expected 3 events, got %d", len(events)) 1249 1249 + } 1250 1250 + 1251 1251 + // Revs increase monotonically (TIDs are lexicographically sortable) 1252 1252 + revs := []string{rev0, events[0].Rev, events[1].Rev, events[2].Rev} 1253 1253 + for i := 1; i < len(revs); i++ { 1254 1254 + if revs[i] <= revs[i-1] { 1255 1255 + t.Errorf("rev[%d]=%q should be > rev[%d]=%q", i, revs[i], i-1, revs[i-1]) 1256 1256 + } 1257 1257 + } 1258 1258 + 1259 1259 + // Each event's Since equals the previous rev 1260 1260 + for i, evt := range events { 1261 1261 + expectedSince := revs[i] // rev before this write 1262 1262 + if evt.Since == nil { 1263 1263 + t.Errorf("event[%d] Since is nil", i) 1264 1264 + } else if *evt.Since != expectedSince { 1265 1265 + t.Errorf("event[%d] Since=%q, expected %q", i, *evt.Since, expectedSince) 1266 1266 + } 1267 1267 + } 1268 1268 + 1269 1269 + // Each event's OldRoot equals the previous NewRoot 1270 1270 + // First event's OldRoot should be defined (repo initialized by setup) 1271 1271 + for i := 1; i < len(events); i++ { 1272 1272 + if events[i].OldRoot == nil { 1273 1273 + t.Errorf("event[%d] OldRoot is nil", i) 1274 1274 + } else if !events[i].OldRoot.Equals(events[i-1].NewRoot) { 1275 1275 + t.Errorf("event[%d] OldRoot=%s != event[%d] NewRoot=%s", i, *events[i].OldRoot, i-1, events[i-1].NewRoot) 1276 1276 + } 1277 1277 + } 1278 1278 + 1279 1279 + // Roots change each time 1280 1280 + roots := []cid.Cid{events[0].NewRoot, events[1].NewRoot, events[2].NewRoot} 1281 1281 + for i := 1; i < len(roots); i++ { 1282 1282 + if roots[i].Equals(roots[i-1]) { 1283 1283 + t.Errorf("root[%d] should differ from root[%d]", i, i-1) 1284 1284 + } 1285 1285 + } 1286 1286 + }) 1287 1287 + 1288 1288 + t.Run("ConcurrentWrites", func(t *testing.T) { 1289 1289 + op, uid := setup(t) 1290 1290 + ctx := context.Background() 1291 1291 + 1292 1292 + const n = 10 1293 1293 + var wg sync.WaitGroup 1294 1294 + errs := make([]error, n) 1295 1295 + 1296 1296 + for i := 0; i < n; i++ { 1297 1297 + wg.Add(1) 1298 1298 + go func(i int) { 1299 1299 + defer wg.Done() 1300 1300 + rkey := fmt.Sprintf("concurrent%d", i) 1301 1301 + rec := newCrewRecord(fmt.Sprintf("did:plc:concurrent%d", i)) 1302 1302 + _, _, errs[i] = op.PutRecord(ctx, uid, atproto.CrewCollection, rkey, rec) 1303 1303 + }(i) 1304 1304 + } 1305 1305 + wg.Wait() 1306 1306 + 1307 1307 + for i, err := range errs { 1308 1308 + if err != nil { 1309 1309 + t.Errorf("goroutine %d: PutRecord failed: %v", i, err) 1310 1310 + } 1311 1311 + } 1312 1312 + 1313 1313 + // Verify all records exist 1314 1314 + for i := 0; i < n; i++ { 1315 1315 + rkey := fmt.Sprintf("concurrent%d", i) 1316 1316 + _, _, err := op.GetRecord(ctx, uid, atproto.CrewCollection, rkey, cid.Undef) 1317 1317 + if err != nil { 1318 1318 + t.Errorf("record %q not found after concurrent writes: %v", rkey, err) 1319 1319 + } 1320 1320 + } 1321 1321 + }) 1322 1322 + 1323 1323 + t.Run("ReadRepo_ContainsRecords", func(t *testing.T) { 1324 1324 + op, uid := setup(t) 1325 1325 + ctx := context.Background() 1326 1326 + 1327 1327 + // Write 2 records with known rkeys 1328 1328 + _, _, err := op.PutRecord(ctx, uid, atproto.CrewCollection, "cartest1", newCrewRecord("did:plc:cartest1")) 1329 1329 + if err != nil { 1330 1330 + t.Fatalf("PutRecord 1: %v", err) 1331 1331 + } 1332 1332 + _, _, err = op.PutRecord(ctx, uid, atproto.CrewCollection, "cartest2", newCrewRecord("did:plc:cartest2")) 1333 1333 + if err != nil { 1334 1334 + t.Fatalf("PutRecord 2: %v", err) 1335 1335 + } 1336 1336 + 1337 1337 + // Export CAR 1338 1338 + var buf bytes.Buffer 1339 1339 + if err := op.ReadRepo(ctx, uid, "", &buf); err != nil { 1340 1340 + t.Fatalf("ReadRepo: %v", err) 1341 1341 + } 1342 1342 + 1343 1343 + // Parse it back 1344 1344 + r, err := repo.ReadRepoFromCar(ctx, &buf) 1345 1345 + if err != nil { 1346 1346 + t.Fatalf("ReadRepoFromCar: %v", err) 1347 1347 + } 1348 1348 + 1349 1349 + // Both records should be retrievable 1350 1350 + for _, rkey := range []string{"cartest1", "cartest2"} { 1351 1351 + rpath := atproto.CrewCollection + "/" + rkey 1352 1352 + _, _, err := r.GetRecord(ctx, rpath) 1353 1353 + if err != nil { 1354 1354 + t.Errorf("record %q not found in CAR: %v", rpath, err) 1355 1355 + } 1356 1356 + } 1357 1357 + 1358 1358 + // Unknown record should fail 1359 1359 + _, _, err = r.GetRecord(ctx, atproto.CrewCollection+"/doesnotexist") 1360 1360 + if err == nil { 1361 1361 + t.Error("expected error for non-existent record in CAR") 1362 1362 + } 1363 1363 + }) 1364 1364 + } 1365 1365 + 1366 1366 + // setupFreshRepoManager returns a RepoManager WITHOUT InitNewActor called. 1367 1367 + func setupFreshRepoManager(t *testing.T) (RepoOperator, models.Uid, string) { 1368 1368 + t.Helper() 1369 1369 + ctx := context.Background() 1370 1370 + keyPath := filepath.Join(t.TempDir(), "signing-key") 1371 1371 + if err := os.WriteFile(keyPath, sharedTestKey, 0600); err != nil { 1372 1372 + t.Fatalf("Failed to write signing key: %v", err) 1373 1373 + } 1374 1374 + pds, err := NewHoldPDS(ctx, "did:web:hold.init-test", "https://hold.init-test", "https://atcr.io", ":memory:", keyPath, false) 1375 1375 + if err != nil { 1376 1376 + t.Fatalf("NewHoldPDS: %v", err) 1377 1377 + } 1378 1378 + t.Cleanup(func() { pds.Close() }) 1379 1379 + return pds.repomgr, pds.uid, pds.did 1380 1380 + } 1381 1381 + 1382 1382 + // setupFreshDirectRepoOperator returns a DirectRepoOperator WITHOUT InitNewActor called. 1383 1383 + func setupFreshDirectRepoOperator(t *testing.T) (RepoOperator, models.Uid, string) { 1384 1384 + t.Helper() 1385 1385 + 1386 1386 + keyPath := filepath.Join(t.TempDir(), "signing-key") 1387 1387 + if err := os.WriteFile(keyPath, sharedTestKey, 0600); err != nil { 1388 1388 + t.Fatalf("Failed to write signing key: %v", err) 1389 1389 + } 1390 1390 + signingKey, err := oauth.GenerateOrLoadPDSKey(keyPath) 1391 1391 + if err != nil { 1392 1392 + t.Fatalf("GenerateOrLoadPDSKey: %v", err) 1393 1393 + } 1394 1394 + 1395 1395 + sqlStore := new(holddb.SQLiteStore) 1396 1396 + if err := sqlStore.Open(":memory:"); err != nil { 1397 1397 + t.Fatalf("SQLiteStore.Open: %v", err) 1398 1398 + } 1399 1399 + t.Cleanup(func() { sqlStore.Close() }) 1400 1400 + 1401 1401 + kmgr := NewHoldKeyManager(signingKey) 1402 1402 + op := NewDirectRepoOperator(sqlStore, kmgr) 1403 1403 + 1404 1404 + return op, models.Uid(1), "did:web:hold.test" 1405 1405 + } 1406 1406 + 1407 1407 + func TestRepoManager(t *testing.T) { 1408 1408 + runRepoOperatorTests(t, setupTestRepoOperator, setupFreshRepoManager) 1409 1409 + } 1410 1410 + 1411 1411 + func TestDirectRepoOperator(t *testing.T) { 1412 1412 + runRepoOperatorTests(t, setupTestDirectRepoOperator, setupFreshDirectRepoOperator) 1413 1413 + }

+62 -662

pkg/hold/pds/repomgr.go

··· 1 1 - // Package pds contains a vendored copy of RepoManager from github.com/bluesky-social/indigo 1 1 + package pds 2 2 + 3 3 + // repomgr.go — RepoManager manages ATProto repository operations for the hold PDS. 2 4 // 3 3 - // Source: github.com/bluesky-social/indigo/repomgr (v0.0.0-20251014222321) 4 4 - // Reference: github.com/streamplace/indigo (67ae5a5) for PutRecord implementation 5 5 - // Reason: The indigo library is unmaintained and contains a critical bug in UpdateRecord 5 5 + // Originally vendored from github.com/bluesky-social/indigo/repomgr with fixes: 6 6 + // - Fixed UpdateRecord bug (was calling PutRecord internally) 7 7 + // - Added PutRecord/UpsertRecord for explicit rkey operations 8 8 + // - Added prevData support for Sync 1.1 6 9 // 7 7 - // Modifications from original: 8 8 - // - Changed package from 'repomgr' to 'pds' for integration with hold service 9 9 - // - Fixed UpdateRecord bug (line 263): Changed r.PutRecord to r.UpdateRecord 10 10 - // (UpdateRecord was incorrectly calling PutRecord, causing incorrect MST operations) 11 11 - // - Removed 5 Prometheus metrics calls (openAndSigCheckDuration, calcDiffDuration, 12 12 - // writeCarSliceDuration, repoOpsImported) as metrics are not used in this project 13 13 - // - Added PutRecord method (lines 309-381) for creating records with explicit rkeys 14 14 - // (like CreateRecord but with specified rkey instead of auto-generated TID) 15 15 - // Based on streamplace/indigo implementation 16 16 - // - Added prevData to support Sync 1.1 17 17 - package pds 10 10 + // Implements the RepoOperator interface (see repo_operator.go). 11 11 + // See docs/REPOMGR_MIGRATION.md for planned migration to indigo/repo directly. 18 12 19 13 import ( 20 20 - "bytes" 21 14 "context" 22 22 - "errors" 23 15 "fmt" 24 16 "io" 25 17 "log/slog" 26 26 - "strings" 27 18 "sync" 28 19 29 20 holddb "atcr.io/pkg/hold/db" 30 21 atproto "github.com/bluesky-social/indigo/api/atproto" 31 22 bsky "github.com/bluesky-social/indigo/api/bsky" 32 23 "github.com/bluesky-social/indigo/atproto/syntax" 33 33 - lexutil "github.com/bluesky-social/indigo/lex/util" 34 24 "github.com/bluesky-social/indigo/models" 35 35 - "github.com/bluesky-social/indigo/mst" 36 25 "github.com/bluesky-social/indigo/repo" 37 26 "github.com/bluesky-social/indigo/util" 38 27 39 28 blocks "github.com/ipfs/go-block-format" 40 29 "github.com/ipfs/go-cid" 41 41 - "github.com/ipfs/go-datastore" 42 42 - blockstore "github.com/ipfs/go-ipfs-blockstore" 43 43 - ipld "github.com/ipfs/go-ipld-format" 44 44 - "github.com/ipld/go-car" 45 30 cbg "github.com/whyrusleeping/cbor-gen" 46 31 "go.opentelemetry.io/otel" 47 47 - "go.opentelemetry.io/otel/attribute" 48 48 - "gorm.io/gorm" 49 32 ) 50 33 51 34 func NewRepoManager(cs holddb.CarStore, kmgr KeyManager) *RepoManager { ··· 56 39 userLocks: make(map[models.Uid]*userLock), 57 40 kmgr: kmgr, 58 41 log: slog.Default().With("system", "repomgr"), 59 59 - noArchive: false, // NonArchivalCarstore not used in hold service 60 42 clk: clk, 61 43 } 62 44 } 63 45 64 64 - type KeyManager interface { 65 65 - VerifyUserSignature(context.Context, string, []byte, []byte) error 66 66 - SignForUser(context.Context, string, []byte) ([]byte, error) 67 67 - } 68 68 - 69 46 func (rm *RepoManager) SetEventHandler(cb func(context.Context, *RepoEvent), hydrateRecords bool) { 70 47 rm.events = cb 71 48 rm.hydrateRecords = hydrateRecords ··· 81 58 events func(context.Context, *RepoEvent) 82 59 hydrateRecords bool 83 60 84 84 - log *slog.Logger 85 85 - noArchive bool 86 86 - 61 61 + log *slog.Logger 87 62 clk *syntax.TIDClock 88 63 } 89 64 90 90 - // NextTID generates a new TID for use as a record key. 91 91 - func (rm *RepoManager) NextTID() string { 92 92 - return rm.clk.Next().String() 93 93 - } 94 94 - 95 95 - type ActorInfo struct { 96 96 - Did string 97 97 - Handle string 98 98 - DisplayName string 99 99 - Type string 100 100 - } 101 101 - 102 102 - type RepoEvent struct { 103 103 - User models.Uid 104 104 - OldRoot *cid.Cid 105 105 - NewRoot cid.Cid 106 106 - PrevData *cid.Cid // MST root CID of the previous commit (for firehose prevData field) 107 107 - Since *string 108 108 - Rev string 109 109 - RepoSlice []byte 110 110 - PDS uint 111 111 - Ops []RepoOp 112 112 - } 113 113 - 114 114 - type RepoOp struct { 115 115 - Kind EventKind 116 116 - Collection string 117 117 - Rkey string 118 118 - RecCid *cid.Cid 119 119 - Record any 120 120 - ActorInfo *ActorInfo 121 121 - } 122 122 - 123 123 - type EventKind string 124 124 - 125 125 - const ( 126 126 - EvtKindCreateRecord = EventKind("create") 127 127 - EvtKindUpdateRecord = EventKind("update") 128 128 - EvtKindDeleteRecord = EventKind("delete") 129 129 - ) 130 130 - 131 131 - type RepoHead struct { 132 132 - gorm.Model 133 133 - Usr models.Uid `gorm:"uniqueIndex"` 134 134 - Root string 135 135 - } 136 136 - 137 65 type userLock struct { 138 66 lk sync.Mutex 139 67 count int ··· 168 96 } 169 97 rm.lklk.Unlock() 170 98 } 171 171 - } 172 172 - 173 173 - func (rm *RepoManager) CarStore() holddb.CarStore { 174 174 - return rm.cs 175 99 } 176 100 177 101 func (rm *RepoManager) CreateRecord(ctx context.Context, user models.Uid, collection string, rec cbg.CBORMarshaler) (string, cid.Cid, error) { ··· 707 631 return head, bs.GetLoggedBlocks(), nil 708 632 } 709 633 710 710 - func (rm *RepoManager) GetProfile(ctx context.Context, uid models.Uid) (*bsky.ActorProfile, error) { 711 711 - bs, err := rm.cs.ReadOnlySession(uid) 712 712 - if err != nil { 713 713 - return nil, err 714 714 - } 715 715 - 716 716 - head, err := rm.cs.GetUserRepoHead(ctx, uid) 717 717 - if err != nil { 718 718 - return nil, err 719 719 - } 720 720 - 721 721 - r, err := repo.OpenRepo(ctx, bs, head) 722 722 - if err != nil { 723 723 - return nil, err 724 724 - } 725 725 - 726 726 - _, val, err := r.GetRecord(ctx, "app.bsky.actor.profile/self") 727 727 - if err != nil { 728 728 - return nil, err 729 729 - } 730 730 - 731 731 - ap, ok := val.(*bsky.ActorProfile) 732 732 - if !ok { 733 733 - return nil, fmt.Errorf("found wrong type in actor profile location in tree") 734 734 - } 735 735 - 736 736 - return ap, nil 737 737 - } 738 738 - 739 739 - func (rm *RepoManager) CheckRepoSig(ctx context.Context, r *repo.Repo, expdid string) error { 740 740 - ctx, span := otel.Tracer("repoman").Start(ctx, "CheckRepoSig") 741 741 - defer span.End() 742 742 - 743 743 - repoDid := r.RepoDid() 744 744 - if expdid != repoDid { 745 745 - return fmt.Errorf("DID in repo did not match (%q != %q)", expdid, repoDid) 746 746 - } 747 747 - 748 748 - scom := r.SignedCommit() 749 749 - 750 750 - usc := scom.Unsigned() 751 751 - sb, err := usc.BytesForSigning() 752 752 - if err != nil { 753 753 - return fmt.Errorf("commit serialization failed: %w", err) 754 754 - } 755 755 - if err := rm.kmgr.VerifyUserSignature(ctx, repoDid, scom.Sig, sb); err != nil { 756 756 - return fmt.Errorf("signature check failed (sig: %x) (sb: %x) : %w", scom.Sig, sb, err) 757 757 - } 758 758 - 759 759 - return nil 760 760 - } 761 761 - 762 762 - func (rm *RepoManager) HandleExternalUserEvent(ctx context.Context, pdsid uint, uid models.Uid, did string, since *string, nrev string, carslice []byte, ops []*atproto.SyncSubscribeRepos_RepoOp) error { 763 763 - if rm.noArchive { 764 764 - return rm.handleExternalUserEventNoArchive(ctx, pdsid, uid, did, since, nrev, carslice, ops) 765 765 - } else { 766 766 - return rm.handleExternalUserEventArchive(ctx, pdsid, uid, did, since, nrev, carslice, ops) 767 767 - } 768 768 - } 769 769 - 770 770 - func (rm *RepoManager) handleExternalUserEventNoArchive(ctx context.Context, pdsid uint, uid models.Uid, did string, since *string, nrev string, carslice []byte, ops []*atproto.SyncSubscribeRepos_RepoOp) error { 771 771 - ctx, span := otel.Tracer("repoman").Start(ctx, "HandleExternalUserEvent") 772 772 - defer span.End() 773 773 - 774 774 - span.SetAttributes(attribute.Int64("uid", int64(uid))) 775 775 - 776 776 - rm.log.Debug("HandleExternalUserEvent", "pds", pdsid, "uid", uid, "since", since, "nrev", nrev) 777 777 - 778 778 - unlock := rm.lockUser(ctx, uid) 779 779 - defer unlock() 780 780 - 781 781 - root, ds, err := rm.cs.ImportSlice(ctx, uid, since, carslice) 782 782 - if err != nil { 783 783 - return fmt.Errorf("importing external carslice: %w", err) 784 784 - } 785 785 - 786 786 - r, err := repo.OpenRepo(ctx, ds, root) 787 787 - if err != nil { 788 788 - return fmt.Errorf("opening external user repo (%d, root=%s): %w", uid, root, err) 789 789 - } 790 790 - 791 791 - if err := rm.CheckRepoSig(ctx, r, did); err != nil { 792 792 - return fmt.Errorf("check repo sig: %w", err) 793 793 - } 794 794 - 795 795 - // Capture previous MST root from old repo state if it exists 796 796 - var prevData *cid.Cid 797 797 - if ds.BaseCid().Defined() { 798 798 - oldrepo, err := repo.OpenRepo(ctx, ds, ds.BaseCid()) 799 799 - if err == nil { 800 800 - pd := oldrepo.DataCid() 801 801 - prevData = &pd 802 802 - } 803 803 - } 804 804 - 805 805 - evtops := make([]RepoOp, 0, len(ops)) 806 806 - for _, op := range ops { 807 807 - parts := strings.SplitN(op.Path, "/", 2) 808 808 - if len(parts) != 2 { 809 809 - return fmt.Errorf("invalid rpath in mst diff, must have collection and rkey") 810 810 - } 811 811 - 812 812 - switch EventKind(op.Action) { 813 813 - case EvtKindCreateRecord: 814 814 - rop := RepoOp{ 815 815 - Kind: EvtKindCreateRecord, 816 816 - Collection: parts[0], 817 817 - Rkey: parts[1], 818 818 - RecCid: (*cid.Cid)(op.Cid), 819 819 - } 820 820 - 821 821 - if rm.hydrateRecords { 822 822 - _, rec, err := r.GetRecord(ctx, op.Path) 823 823 - if err != nil { 824 824 - return fmt.Errorf("reading changed record from car slice: %w", err) 825 825 - } 826 826 - rop.Record = rec 827 827 - } 828 828 - 829 829 - evtops = append(evtops, rop) 830 830 - case EvtKindUpdateRecord: 831 831 - rop := RepoOp{ 832 832 - Kind: EvtKindUpdateRecord, 833 833 - Collection: parts[0], 834 834 - Rkey: parts[1], 835 835 - RecCid: (*cid.Cid)(op.Cid), 836 836 - } 837 837 - 838 838 - if rm.hydrateRecords { 839 839 - _, rec, err := r.GetRecord(ctx, op.Path) 840 840 - if err != nil { 841 841 - return fmt.Errorf("reading changed record from car slice: %w", err) 842 842 - } 843 843 - 844 844 - rop.Record = rec 845 845 - } 846 846 - 847 847 - evtops = append(evtops, rop) 848 848 - case EvtKindDeleteRecord: 849 849 - evtops = append(evtops, RepoOp{ 850 850 - Kind: EvtKindDeleteRecord, 851 851 - Collection: parts[0], 852 852 - Rkey: parts[1], 853 853 - }) 854 854 - default: 855 855 - return fmt.Errorf("unrecognized external user event kind: %q", op.Action) 856 856 - } 857 857 - } 858 858 - 859 859 - if rm.events != nil { 860 860 - rm.events(ctx, &RepoEvent{ 861 861 - User: uid, 862 862 - //OldRoot: prev, 863 863 - NewRoot: root, 864 864 - PrevData: prevData, 865 865 - Rev: nrev, 866 866 - Since: since, 867 867 - Ops: evtops, 868 868 - RepoSlice: carslice, 869 869 - PDS: pdsid, 870 870 - }) 871 871 - } 872 872 - 873 873 - return nil 874 874 - } 875 875 - 876 876 - func (rm *RepoManager) handleExternalUserEventArchive(ctx context.Context, pdsid uint, uid models.Uid, did string, since *string, nrev string, carslice []byte, ops []*atproto.SyncSubscribeRepos_RepoOp) error { 877 877 - ctx, span := otel.Tracer("repoman").Start(ctx, "HandleExternalUserEvent") 878 878 - defer span.End() 879 879 - 880 880 - span.SetAttributes(attribute.Int64("uid", int64(uid))) 881 881 - 882 882 - rm.log.Debug("HandleExternalUserEvent", "pds", pdsid, "uid", uid, "since", since, "nrev", nrev) 883 883 - 884 884 - unlock := rm.lockUser(ctx, uid) 885 885 - defer unlock() 886 886 - 887 887 - root, ds, err := rm.cs.ImportSlice(ctx, uid, since, carslice) 888 888 - if err != nil { 889 889 - return fmt.Errorf("importing external carslice: %w", err) 890 890 - } 891 891 - 892 892 - r, err := repo.OpenRepo(ctx, ds, root) 893 893 - if err != nil { 894 894 - return fmt.Errorf("opening external user repo (%d, root=%s): %w", uid, root, err) 895 895 - } 896 896 - 897 897 - if err := rm.CheckRepoSig(ctx, r, did); err != nil { 898 898 - return err 899 899 - } 900 900 - 901 901 - var skipcids map[cid.Cid]bool 902 902 - var prevData *cid.Cid 903 903 - if ds.BaseCid().Defined() { 904 904 - oldrepo, err := repo.OpenRepo(ctx, ds, ds.BaseCid()) 905 905 - if err != nil { 906 906 - return fmt.Errorf("failed to check data root in old repo: %w", err) 907 907 - } 908 908 - 909 909 - // Capture previous MST root for prevData 910 910 - pd := oldrepo.DataCid() 911 911 - prevData = &pd 912 912 - 913 913 - // if the old commit has a 'prev', CalcDiff will error out while trying 914 914 - // to walk it. This is an old repo thing that is being deprecated. 915 915 - // This check is a temporary workaround until all repos get migrated 916 916 - // and this becomes no longer an issue 917 917 - prev, _ := oldrepo.PrevCommit(ctx) 918 918 - if prev != nil { 919 919 - skipcids = map[cid.Cid]bool{ 920 920 - *prev: true, 921 921 - } 922 922 - } 923 923 - } 924 924 - 925 925 - if err := ds.CalcDiff(ctx, skipcids); err != nil { 926 926 - return fmt.Errorf("failed while calculating mst diff (since=%v): %w", since, err) 927 927 - } 928 928 - 929 929 - evtops := make([]RepoOp, 0, len(ops)) 930 930 - 931 931 - for _, op := range ops { 932 932 - parts := strings.SplitN(op.Path, "/", 2) 933 933 - if len(parts) != 2 { 934 934 - return fmt.Errorf("invalid rpath in mst diff, must have collection and rkey") 935 935 - } 936 936 - 937 937 - switch EventKind(op.Action) { 938 938 - case EvtKindCreateRecord: 939 939 - rop := RepoOp{ 940 940 - Kind: EvtKindCreateRecord, 941 941 - Collection: parts[0], 942 942 - Rkey: parts[1], 943 943 - RecCid: (*cid.Cid)(op.Cid), 944 944 - } 945 945 - 946 946 - if rm.hydrateRecords { 947 947 - _, rec, err := r.GetRecord(ctx, op.Path) 948 948 - if err != nil { 949 949 - return fmt.Errorf("reading changed record from car slice: %w", err) 950 950 - } 951 951 - rop.Record = rec 952 952 - } 953 953 - 954 954 - evtops = append(evtops, rop) 955 955 - case EvtKindUpdateRecord: 956 956 - rop := RepoOp{ 957 957 - Kind: EvtKindUpdateRecord, 958 958 - Collection: parts[0], 959 959 - Rkey: parts[1], 960 960 - RecCid: (*cid.Cid)(op.Cid), 961 961 - } 962 962 - 963 963 - if rm.hydrateRecords { 964 964 - _, rec, err := r.GetRecord(ctx, op.Path) 965 965 - if err != nil { 966 966 - return fmt.Errorf("reading changed record from car slice: %w", err) 967 967 - } 968 968 - 969 969 - rop.Record = rec 970 970 - } 971 971 - 972 972 - evtops = append(evtops, rop) 973 973 - case EvtKindDeleteRecord: 974 974 - evtops = append(evtops, RepoOp{ 975 975 - Kind: EvtKindDeleteRecord, 976 976 - Collection: parts[0], 977 977 - Rkey: parts[1], 978 978 - }) 979 979 - default: 980 980 - return fmt.Errorf("unrecognized external user event kind: %q", op.Action) 981 981 - } 982 982 - } 983 983 - 984 984 - rslice, err := ds.CloseWithRoot(ctx, root, nrev) 985 985 - if err != nil { 986 986 - return fmt.Errorf("close with root: %w", err) 987 987 - } 988 988 - 989 989 - if rm.events != nil { 990 990 - rm.events(ctx, &RepoEvent{ 991 991 - User: uid, 992 992 - //OldRoot: prev, 993 993 - NewRoot: root, 994 994 - PrevData: prevData, 995 995 - Rev: nrev, 996 996 - Since: since, 997 997 - Ops: evtops, 998 998 - RepoSlice: rslice, 999 999 - PDS: pdsid, 1000 1000 - }) 1001 1001 - } 1002 1002 - 1003 1003 - return nil 1004 1004 - } 1005 1005 - 1006 634 func (rm *RepoManager) BatchWrite(ctx context.Context, user models.Uid, writes []*atproto.RepoApplyWrites_Input_Writes_Elem) error { 1007 635 ctx, span := otel.Tracer("repoman").Start(ctx, "BatchWrite") 1008 636 defer span.End() ··· 1131 759 return nil 1132 760 } 1133 761 1134 1134 - func (rm *RepoManager) ImportNewRepo(ctx context.Context, user models.Uid, repoDid string, r io.Reader, rev *string) error { 1135 1135 - ctx, span := otel.Tracer("repoman").Start(ctx, "ImportNewRepo") 762 762 + // BulkUpsert writes multiple records in a single delta session and commit. 763 763 + // Each record is upserted: created if new, updated if it already exists. 764 764 + func (rm *RepoManager) BulkUpsert(ctx context.Context, user models.Uid, records []BulkRecord) error { 765 765 + ctx, span := otel.Tracer("repoman").Start(ctx, "BulkUpsert") 1136 766 defer span.End() 1137 767 1138 768 unlock := rm.lockUser(ctx, user) 1139 769 defer unlock() 1140 770 1141 1141 - currev, err := rm.cs.GetUserRepoRev(ctx, user) 771 771 + rev, err := rm.cs.GetUserRepoRev(ctx, user) 1142 772 if err != nil { 1143 773 return err 1144 774 } 1145 775 1146 1146 - curhead, err := rm.cs.GetUserRepoHead(ctx, user) 776 776 + ds, err := rm.cs.NewDeltaSession(ctx, user, &rev) 1147 777 if err != nil { 1148 778 return err 1149 779 } 1150 780 1151 1151 - if rev != nil && *rev == "" { 1152 1152 - rev = nil 1153 1153 - } 1154 1154 - if rev == nil { 1155 1155 - // if 'rev' is nil, this implies a fresh sync. 1156 1156 - // in this case, ignore any existing blocks we have and treat this like a clean import. 1157 1157 - curhead = cid.Undef 1158 1158 - } 1159 1159 - 1160 1160 - if rev != nil && *rev != currev { 1161 1161 - // TODO: we could probably just deal with this 1162 1162 - return fmt.Errorf("ImportNewRepo called with incorrect base") 1163 1163 - } 1164 1164 - 1165 1165 - // Capture previous MST root before import overwrites it 1166 1166 - var prevData *cid.Cid 1167 1167 - if curhead.Defined() { 1168 1168 - robs, err := rm.cs.ReadOnlySession(user) 1169 1169 - if err == nil { 1170 1170 - oldrepo, err := repo.OpenRepo(ctx, robs, curhead) 1171 1171 - if err == nil { 1172 1172 - pd := oldrepo.DataCid() 1173 1173 - prevData = &pd 1174 1174 - } 1175 1175 - } 1176 1176 - } 1177 1177 - 1178 1178 - err = rm.processNewRepo(ctx, user, r, rev, func(ctx context.Context, root cid.Cid, finish func(context.Context, string) ([]byte, error), bs blockstore.Blockstore) error { 1179 1179 - r, err := repo.OpenRepo(ctx, bs, root) 1180 1180 - if err != nil { 1181 1181 - return fmt.Errorf("opening new repo: %w", err) 1182 1182 - } 1183 1183 - 1184 1184 - scom := r.SignedCommit() 1185 1185 - 1186 1186 - usc := scom.Unsigned() 1187 1187 - sb, err := usc.BytesForSigning() 1188 1188 - if err != nil { 1189 1189 - return fmt.Errorf("commit serialization failed: %w", err) 1190 1190 - } 1191 1191 - if err := rm.kmgr.VerifyUserSignature(ctx, repoDid, scom.Sig, sb); err != nil { 1192 1192 - return fmt.Errorf("new user signature check failed: %w", err) 1193 1193 - } 1194 1194 - 1195 1195 - diffops, err := r.DiffSince(ctx, curhead) 1196 1196 - if err != nil { 1197 1197 - return fmt.Errorf("diff trees (curhead: %s): %w", curhead, err) 1198 1198 - } 1199 1199 - 1200 1200 - ops := make([]RepoOp, 0, len(diffops)) 1201 1201 - for _, op := range diffops { 1202 1202 - out, err := rm.processOp(ctx, bs, op, rm.hydrateRecords) 1203 1203 - if err != nil { 1204 1204 - rm.log.Error("failed to process repo op", "err", err, "path", op.Rpath, "repo", repoDid) 1205 1205 - } 1206 1206 - 1207 1207 - if out != nil { 1208 1208 - ops = append(ops, *out) 1209 1209 - } 1210 1210 - } 1211 1211 - 1212 1212 - slice, err := finish(ctx, scom.Rev) 1213 1213 - if err != nil { 1214 1214 - return err 1215 1215 - } 1216 1216 - 1217 1217 - if rm.events != nil { 1218 1218 - rm.events(ctx, &RepoEvent{ 1219 1219 - User: user, 1220 1220 - //OldRoot: oldroot, 1221 1221 - NewRoot: root, 1222 1222 - PrevData: prevData, 1223 1223 - Rev: scom.Rev, 1224 1224 - Since: &currev, 1225 1225 - RepoSlice: slice, 1226 1226 - Ops: ops, 1227 1227 - }) 1228 1228 - } 1229 1229 - 1230 1230 - return nil 1231 1231 - }) 1232 1232 - if err != nil { 1233 1233 - return fmt.Errorf("process new repo (current rev: %s): %w", currev, err) 1234 1234 - } 1235 1235 - 1236 1236 - return nil 1237 1237 - } 1238 1238 - 1239 1239 - func (rm *RepoManager) processOp(ctx context.Context, bs blockstore.Blockstore, op *mst.DiffOp, hydrateRecords bool) (*RepoOp, error) { 1240 1240 - parts := strings.SplitN(op.Rpath, "/", 2) 1241 1241 - if len(parts) != 2 { 1242 1242 - return nil, fmt.Errorf("repo mst had invalid rpath: %q", op.Rpath) 1243 1243 - } 1244 1244 - 1245 1245 - switch op.Op { 1246 1246 - case "add", "mut": 1247 1247 - 1248 1248 - kind := EvtKindCreateRecord 1249 1249 - if op.Op == "mut" { 1250 1250 - kind = EvtKindUpdateRecord 1251 1251 - } 1252 1252 - 1253 1253 - outop := &RepoOp{ 1254 1254 - Kind: kind, 1255 1255 - Collection: parts[0], 1256 1256 - Rkey: parts[1], 1257 1257 - RecCid: &op.NewCid, 1258 1258 - } 1259 1259 - 1260 1260 - if hydrateRecords { 1261 1261 - blk, err := bs.Get(ctx, op.NewCid) 1262 1262 - if err != nil { 1263 1263 - return nil, err 1264 1264 - } 1265 1265 - 1266 1266 - rec, err := lexutil.CborDecodeValue(blk.RawData()) 1267 1267 - if err != nil { 1268 1268 - if !errors.Is(err, lexutil.ErrUnrecognizedType) { 1269 1269 - return nil, err 1270 1270 - } 1271 1271 - 1272 1272 - rm.log.Warn("failed processing repo diff", "err", err) 1273 1273 - } else { 1274 1274 - outop.Record = rec 1275 1275 - } 1276 1276 - } 1277 1277 - 1278 1278 - return outop, nil 1279 1279 - case "del": 1280 1280 - return &RepoOp{ 1281 1281 - Kind: EvtKindDeleteRecord, 1282 1282 - Collection: parts[0], 1283 1283 - Rkey: parts[1], 1284 1284 - RecCid: nil, 1285 1285 - }, nil 1286 1286 - 1287 1287 - default: 1288 1288 - return nil, fmt.Errorf("diff returned invalid op type: %q", op.Op) 1289 1289 - } 1290 1290 - } 1291 1291 - 1292 1292 - func (rm *RepoManager) processNewRepo(ctx context.Context, user models.Uid, r io.Reader, rev *string, cb func(ctx context.Context, root cid.Cid, finish func(context.Context, string) ([]byte, error), bs blockstore.Blockstore) error) error { 1293 1293 - ctx, span := otel.Tracer("repoman").Start(ctx, "processNewRepo") 1294 1294 - defer span.End() 1295 1295 - 1296 1296 - carr, err := car.NewCarReader(r) 781 781 + head := ds.BaseCid() 782 782 + r, err := repo.OpenRepo(ctx, ds, head) 1297 783 if err != nil { 1298 784 return err 1299 785 } 1300 786 1301 1301 - if len(carr.Header.Roots) != 1 { 1302 1302 - return fmt.Errorf("invalid car file, header must have a single root (has %d)", len(carr.Header.Roots)) 787 787 + // Capture previous MST root before commit overwrites it 788 788 + var prevData *cid.Cid 789 789 + if head.Defined() { 790 790 + pd := r.DataCid() 791 791 + prevData = &pd 1303 792 } 1304 793 1305 1305 - membs := blockstore.NewBlockstore(datastore.NewMapDatastore()) 794 794 + ops := make([]RepoOp, 0, len(records)) 795 795 + for _, rec := range records { 796 796 + rpath := rec.Collection + "/" + rec.Rkey 1306 797 1307 1307 - for { 1308 1308 - blk, err := carr.Next() 1309 1309 - if err != nil { 1310 1310 - if err == io.EOF { 1311 1311 - break 1312 1312 - } 1313 1313 - return err 1314 1314 - } 798 798 + // Check if record exists to determine create vs update 799 799 + _, _, getErr := r.GetRecordBytes(ctx, rpath) 800 800 + recordExists := getErr == nil 1315 801 1316 1316 - if err := membs.Put(ctx, blk); err != nil { 1317 1317 - return err 802 802 + var cc cid.Cid 803 803 + var evtKind EventKind 804 804 + if recordExists { 805 805 + cc, err = r.UpdateRecord(ctx, rpath, rec.Data) 806 806 + evtKind = EvtKindUpdateRecord 807 807 + } else { 808 808 + cc, err = r.PutRecord(ctx, rpath, rec.Data) 809 809 + evtKind = EvtKindCreateRecord 1318 810 } 1319 1319 - } 1320 1320 - 1321 1321 - seen := make(map[cid.Cid]bool) 1322 1322 - 1323 1323 - root := carr.Header.Roots[0] 1324 1324 - // TODO: if there are blocks that get convergently recreated throughout 1325 1325 - // the repos lifecycle, this will end up erroneously not including 1326 1326 - // them. We should compute the set of blocks needed to read any repo 1327 1327 - // ops that happened in the commit and use that for our 'output' blocks 1328 1328 - cids, err := rm.walkTree(ctx, seen, root, membs, true) 1329 1329 - if err != nil { 1330 1330 - return fmt.Errorf("walkTree: %w", err) 1331 1331 - } 1332 1332 - 1333 1333 - ds, err := rm.cs.NewDeltaSession(ctx, user, rev) 1334 1334 - if err != nil { 1335 1335 - return fmt.Errorf("opening delta session: %w", err) 1336 1336 - } 1337 1337 - 1338 1338 - for _, c := range cids { 1339 1339 - blk, err := membs.Get(ctx, c) 1340 811 if err != nil { 1341 1341 - return fmt.Errorf("copying walked cids to carstore: %w", err) 812 812 + return fmt.Errorf("failed to write %s: %w", rpath, err) 1342 813 } 1343 814 1344 1344 - if err := ds.Put(ctx, blk); err != nil { 1345 1345 - return err 1346 1346 - } 815 815 + ops = append(ops, RepoOp{ 816 816 + Kind: evtKind, 817 817 + Collection: rec.Collection, 818 818 + Rkey: rec.Rkey, 819 819 + RecCid: &cc, 820 820 + }) 1347 821 } 1348 822 1349 1349 - finish := func(ctx context.Context, nrev string) ([]byte, error) { 1350 1350 - return ds.CloseWithRoot(ctx, root, nrev) 1351 1351 - } 1352 1352 - 1353 1353 - if err := cb(ctx, root, finish, ds); err != nil { 1354 1354 - return fmt.Errorf("cb errored root: %s, rev: %s: %w", root, stringOrNil(rev), err) 1355 1355 - } 1356 1356 - 1357 1357 - return nil 1358 1358 - } 1359 1359 - 1360 1360 - func stringOrNil(s *string) string { 1361 1361 - if s == nil { 1362 1362 - return "nil" 1363 1363 - } 1364 1364 - return *s 1365 1365 - } 1366 1366 - 1367 1367 - // walkTree returns all cids linked recursively by the root, skipping any cids 1368 1368 - // in the 'skip' map, and not erroring on 'not found' if prevMissing is set 1369 1369 - func (rm *RepoManager) walkTree(ctx context.Context, skip map[cid.Cid]bool, root cid.Cid, bs blockstore.Blockstore, prevMissing bool) ([]cid.Cid, error) { 1370 1370 - // TODO: what if someone puts non-cbor links in their repo? 1371 1371 - if root.Prefix().Codec != cid.DagCBOR { 1372 1372 - return nil, fmt.Errorf("can only handle dag-cbor objects in repos (%s is %d)", root, root.Prefix().Codec) 1373 1373 - } 1374 1374 - 1375 1375 - blk, err := bs.Get(ctx, root) 1376 1376 - if err != nil { 1377 1377 - return nil, err 1378 1378 - } 1379 1379 - 1380 1380 - var links []cid.Cid 1381 1381 - if err := cbg.ScanForLinks(bytes.NewReader(blk.RawData()), func(c cid.Cid) { 1382 1382 - if c.Prefix().Codec == cid.Raw { 1383 1383 - rm.log.Debug("skipping 'raw' CID in record", "recordCid", root, "rawCid", c) 1384 1384 - return 1385 1385 - } 1386 1386 - if skip[c] { 1387 1387 - return 1388 1388 - } 1389 1389 - 1390 1390 - links = append(links, c) 1391 1391 - skip[c] = true 1392 1392 - }); err != nil { 1393 1393 - return nil, err 1394 1394 - } 1395 1395 - 1396 1396 - out := []cid.Cid{root} 1397 1397 - skip[root] = true 1398 1398 - 1399 1399 - // TODO: should do this non-recursive since i expect these may get deep 1400 1400 - for _, c := range links { 1401 1401 - sub, err := rm.walkTree(ctx, skip, c, bs, prevMissing) 1402 1402 - if err != nil { 1403 1403 - if prevMissing && !ipld.IsNotFound(err) { 1404 1404 - return nil, err 1405 1405 - } 1406 1406 - } 1407 1407 - 1408 1408 - out = append(out, sub...) 1409 1409 - } 1410 1410 - 1411 1411 - return out, nil 1412 1412 - } 1413 1413 - 1414 1414 - func (rm *RepoManager) TakeDownRepo(ctx context.Context, uid models.Uid) error { 1415 1415 - unlock := rm.lockUser(ctx, uid) 1416 1416 - defer unlock() 1417 1417 - 1418 1418 - return rm.cs.WipeUserData(ctx, uid) 1419 1419 - } 1420 1420 - 1421 1421 - // ResetRepo is technically identical to TakeDownRepo, for now 1422 1422 - func (rm *RepoManager) ResetRepo(ctx context.Context, uid models.Uid) error { 1423 1423 - unlock := rm.lockUser(ctx, uid) 1424 1424 - defer unlock() 1425 1425 - 1426 1426 - return rm.cs.WipeUserData(ctx, uid) 1427 1427 - } 1428 1428 - 1429 1429 - func (rm *RepoManager) VerifyRepo(ctx context.Context, uid models.Uid) error { 1430 1430 - ses, err := rm.cs.ReadOnlySession(uid) 823 823 + nroot, nrev, err := r.Commit(ctx, rm.kmgr.SignForUser) 1431 824 if err != nil { 1432 825 return err 1433 826 } 1434 827 1435 1435 - r, err := repo.OpenRepo(ctx, ses, ses.BaseCid()) 828 828 + rslice, err := ds.CloseWithRoot(ctx, nroot, nrev) 1436 829 if err != nil { 1437 1437 - return err 830 830 + return fmt.Errorf("close with root: %w", err) 1438 831 } 1439 832 1440 1440 - if err := r.ForEach(ctx, "", func(k string, v cid.Cid) error { 1441 1441 - _, err := ses.Get(ctx, v) 1442 1442 - if err != nil { 1443 1443 - return fmt.Errorf("failed to get record %s (%s): %w", k, v, err) 1444 1444 - } 833 833 + var oldroot *cid.Cid 834 834 + if head.Defined() { 835 835 + oldroot = &head 836 836 + } 1445 837 1446 1446 - return nil 1447 1447 - }); err != nil { 1448 1448 - return err 838 838 + if rm.events != nil { 839 839 + rm.events(ctx, &RepoEvent{ 840 840 + User: user, 841 841 + OldRoot: oldroot, 842 842 + NewRoot: nroot, 843 843 + PrevData: prevData, 844 844 + Rev: nrev, 845 845 + Since: &rev, 846 846 + Ops: ops, 847 847 + RepoSlice: rslice, 848 848 + }) 1449 849 } 1450 850 1451 851 return nil

+6 -6

pkg/hold/pds/server.go

··· 41 41 appviewURL string 42 42 appviewMeta *atproto.AppviewMetadata 43 43 carstore holddb.CarStore 44 44 - repomgr *RepoManager 44 44 + repomgr RepoOperator 45 45 dbPath string 46 46 uid models.Uid 47 47 signingKey *atcrypto.PrivateKeyK256 ··· 106 106 // Create KeyManager wrapper for our signing key 107 107 kmgr := NewHoldKeyManager(signingKey) 108 108 109 109 - // Create RepoManager - it will handle all session/repo lifecycle 110 110 - rm := NewRepoManager(cs, kmgr) 109 109 + // Create repo operator - handles all session/repo lifecycle 110 110 + rm := NewDirectRepoOperator(cs, kmgr) 111 111 112 112 // Check if repo already exists, if not create initial commit 113 113 head, err := cs.GetUserRepoHead(ctx, uid) ··· 162 162 cs := sqlStore 163 163 uid := models.Uid(1) 164 164 kmgr := NewHoldKeyManager(signingKey) 165 165 - rm := NewRepoManager(cs, kmgr) 165 165 + rm := NewDirectRepoOperator(cs, kmgr) 166 166 167 167 head, err := cs.GetUserRepoHead(ctx, uid) 168 168 hasValidRepo := (err == nil && head.Defined()) ··· 200 200 return p.signingKey 201 201 } 202 202 203 203 - // RepomgrRef returns a reference to the RepoManager for event handler setup 204 204 - func (p *HoldPDS) RepomgrRef() *RepoManager { 203 203 + // RepomgrRef returns a reference to the RepoOperator for event handler setup 204 204 + func (p *HoldPDS) RepomgrRef() RepoOperator { 205 205 return p.repomgr 206 206 } 207 207