A container registry that uses the AT Protocol for manifest storage and S3 for blob storage. atcr.io
docker container atproto go

rebuild repomgr into a custom repo operator. up to 2x faster

evan.jarrett.net fcc5fa78 b235e4a7

verified
+2701 -777
+148
docs/REPOMGR_MIGRATION.md
··· 1 + # Incremental Migration: Vendored repomgr → Direct `indigo/repo` 2 + 3 + ## Context 4 + 5 + The hold PDS uses a vendored copy of indigo's `repomgr` (~1450 lines in `pkg/hold/pds/repomgr.go`). Upstream, repomgr is [soft-deprecated](https://github.com/bluesky-social/indigo/pull/1102#issuecomment-2985956040) — bnewbold recommends using `indigo/repo` directly (as [cocoon](https://github.com/haileyok/cocoon) does). The vendored copy already has custom patches (PutRecord, UpsertRecord, prevData) and will continue to drift. This migration defines a clean interface, then swaps the implementation behind it. 6 + 7 + ## Phase 0: Save plan to docs, remove dead code, define interface ✅ 8 + 9 + **Goal:** Persist this migration plan as a reference doc. Shrink repomgr.go from ~1450 lines to ~750 by removing dead code. Fix import.go encapsulation. Define `RepoOperator` interface so all code accesses repomgr through it. No behavior change. 10 + 11 + **Completed 2026-02-28.** repomgr.go: 1453 → 871 lines (-40%). import.go: 189 → 88 lines (-53%). New repo_operator.go: 82 lines. 12 + 13 + ### Step 1: Save plan to docs ✅ 14 + - Write this plan to `docs/REPOMGR_MIGRATION.md` 15 + 16 + ### Step 2: Dead code deleted from `repomgr.go` ✅ 17 + - `HandleExternalUserEvent()` + `handleExternalUserEventNoArchive()` + `handleExternalUserEventArchive()` 18 + - `ImportNewRepo()` + `processNewRepo()` + `walkTree()` + `processOp()` + `stringOrNil()` 19 + - `CheckRepoSig()` 20 + - `TakeDownRepo()`, `ResetRepo()`, `VerifyRepo()` 21 + - `GetProfile()` 22 + - `CarStore()` 23 + - `NextTID()` / `nextTID()` (removed entirely — `BatchWrite` uses `rm.clk.Next()` directly) 24 + - `noArchive` field removed from struct 25 + - 12 unused imports cleaned up 26 + 27 + ### Step 3: Fixed `import.go` encapsulation ✅ 28 + Added `BulkUpsert()` method to `RepoManager`. Rewrote `ImportFromCAR` to call `p.repomgr.BulkUpsert()` instead of reaching into `p.repomgr.lockUser`, `p.repomgr.cs`, `p.repomgr.kmgr`, `p.repomgr.events`. Removed the 88-line `bulkImportRecords()` private method. 29 + 30 + ### Step 4: Defined `RepoOperator` interface ✅ 31 + 32 + New file: `pkg/hold/pds/repo_operator.go` — interface with 16 methods, compile-time check, shared types (`RepoEvent`, `RepoOp`, `EventKind`, `BulkRecord`). 33 + 34 + ### Step 5: Updated callers to use interface ✅ 35 + - `pkg/hold/pds/server.go` — `repomgr *RepoManager` → `repomgr RepoOperator`, `RepomgrRef()` returns `RepoOperator` 36 + - Downstream callers (`hold/server.go`, `admin/handlers_relays.go`, tests) unchanged — they go through `RepomgrRef()` which returns the interface 37 + - `var _ RepoOperator = (*RepoManager)(nil)` compile-time check in `repo_operator.go` 38 + 39 + ### Verification: ✅ 40 + - `make lint` — 0 issues (also fixed pre-existing unchecked error in `events.go`) 41 + - `make test` — all tests pass 42 + 43 + --- 44 + 45 + ## Phase 1: Test hardening against the interface ✅ 46 + 47 + **Goal:** Write tests against `RepoOperator` that verify current behavior while `RepoManager` is the only implementation. These become the regression safety net when swapping to the new implementation in Phase 3. 48 + 49 + **Completed 2026-02-28.** New `repo_operator_test.go`: 38 subtests covering all CRUD, read, event emission, error paths, and edge cases. `runRepoOperatorTests(t, setup)` pattern ready for Phase 2's `DirectRepoOperator`. 50 + 51 + ### Gaps covered ✅ 52 + - `CreateRecord` — round-trip, TID 13-char rkey format, no-panic without event handler 53 + - `UpdateRecord` — CID changes, new data returned, non-existent record error, hydrated events 54 + - `PutRecord` — explicit rkey, duplicate rkey error, hydrated events 55 + - `UpsertRecord` — create path (created=true), update path (created=false, CID changes) 56 + - `DeleteRecord` — delete + verify gone, non-existent rkey error 57 + - `BatchWrite` — create+delete batch, update write type, auto-rkey (nil Rkey), delete-not-found error, empty write elem error, multi-op event emission with hydration, update hydration 58 + - `BulkUpsert` — create + re-upsert with changed data, multi-op event emission 59 + - `GetRecord` — CID match, CID mismatch error, not-found error 60 + - `GetRecordProof` — head CID + proof blocks, not-found error, no-repo error 61 + - `GetRepoRoot` — defined CID after init, changes after write 62 + - `GetRepoRev` — non-empty, changes after write 63 + - `ReadRepo` — non-empty CAR output, incremental export with `since` 64 + - `InitNewActor` — empty DID error, zero user error, event emission with hydration 65 + - Event emission — create/update/delete events verified: prevData, ops, oldRoot, newRoot, rev, since, repoSlice, hydration 66 + 67 + ### Coverage ✅ 68 + All RepoOperator methods 81–100%. Remaining uncovered lines are internal infrastructure error guards (`GetUserRepoRev`, `NewDeltaSession`, `OpenRepo`, `Commit`, `CloseWithRoot`) — not reachable without mocking the carstore. 69 + 70 + ### Files modified ✅ 71 + - `pkg/hold/pds/repo_operator_test.go` — new file, 38 subtests 72 + 73 + ### Verification ✅ 74 + - `make lint` — 0 issues 75 + - `make test` — all tests pass 76 + 77 + --- 78 + 79 + ## Phase 2: Build new implementation ✅ 80 + 81 + **Goal:** Create `DirectRepoOperator` using `indigo/repo` directly (cocoon pattern). 82 + 83 + **Completed 2026-02-28.** New `pkg/hold/pds/repo.go`: 548 lines (vs 852 in repomgr.go, ~36% reduction). All 37 subtests pass identically for both implementations. Race detector, shuffled order, and parallel execution all clean. 84 + 85 + ### New file: `pkg/hold/pds/repo.go` ✅ 86 + 87 + Key differences from vendored repomgr: 88 + - **Single `sync.Mutex`** instead of per-user lock map (`lklk` + `userLocks` + `userLock` struct + reference counting) 89 + - **No OpenTelemetry tracing** (`otel.Tracer` calls removed) 90 + - **No `gorm` dependency** (`RepoHead` struct removed, `gorm.io/gorm` dropped from go.mod) 91 + - **`openWriteSession` / `commitWrite` helpers** extract the repeated 6-step write pattern 92 + 93 + Core mutation pattern (same as current, just cleaner): 94 + 1. Lock → get rev → open delta session → open repo 95 + 2. Capture `r.DataCid()` for prevData 96 + 3. Perform operation(s) 97 + 4. `r.Commit()` → `ds.CloseWithRoot()` → emit event → unlock 98 + 99 + ### Shared types moved to `repo_operator.go` ✅ 100 + - `KeyManager` interface and `ActorInfo` struct moved from `repomgr.go` 101 + - Both implementations import from the same location 102 + 103 + ### Test wiring ✅ 104 + - `setupTestDirectRepoOperator` — creates carstore + key manager directly (no `NewHoldPDS`) 105 + - `runRepoOperatorTests` refactored to accept optional `freshSetup` for `InitNewActor_EventEmission` 106 + - `TestDirectRepoOperator` runs all 37 subtests identically 107 + 108 + ### Verification ✅ 109 + - `go build ./cmd/hold` — compiles 110 + - `TestRepoManager` — 37/37 subtests pass 111 + - `TestDirectRepoOperator` — 37/37 subtests pass 112 + - `-race -shuffle=on -count=5 -parallel=8` — all clean 113 + - `make lint` — 0 issues 114 + - `make test` — all tests pass 115 + 116 + --- 117 + 118 + ## Phase 3: Config flag + switchover 119 + 120 + **Goal:** Feature flag to select implementation, default old. 121 + 122 + ### Changes: 123 + - `pkg/hold/config.go` — add `UseDirectRepo bool` to DatabaseConfig 124 + - `pkg/hold/pds/server.go` — select implementation based on config in `NewHoldPDS`/`NewHoldPDSWithDB` 125 + - Regenerate example configs 126 + 127 + ### Verification: 128 + - Deploy with `use_direct_repo: false` (default) 129 + - Test with `use_direct_repo: true` in staging 130 + - `make lint && make test` 131 + 132 + --- 133 + 134 + ## Phase 4: Remove vendored repomgr 135 + 136 + **Goal:** After production validation, delete the old code. 137 + 138 + - Delete `repomgr.go` 139 + - Remove config flag, make `DirectRepoOperator` the only implementation 140 + - Rename `repo_direct.go` → `repo_operator_impl.go` 141 + - Regenerate example configs 142 + - `make lint && make test` 143 + 144 + --- 145 + 146 + ## Decision log 147 + 148 + - **`indigo/repo` over `atproto/repo`**: `atproto/repo` has the MST primitives (`Insert`, `Remove`, `ApplyOp`) but no high-level PDS API (`OpenRepo`, `CreateRecord`, `Commit(signFn)`). Its own doc.go says "does not yet work for implementing a repository host (PDS)." `indigo/repo` is what the reference PDS and cocoon use. The `RepoOperator` interface means we can swap later if `atproto/repo` adds PDS support.
+2 -2
go.mod
··· 25 25 github.com/gorilla/websocket v1.5.4-0.20250319132907-e064f32e3674 26 26 github.com/ipfs/go-block-format v0.2.3 27 27 github.com/ipfs/go-cid v0.6.0 28 - github.com/ipfs/go-datastore v0.9.1 29 28 github.com/ipfs/go-ipfs-blockstore v1.3.1 30 29 github.com/ipfs/go-ipld-cbor v0.2.1 31 30 github.com/ipfs/go-ipld-format v0.6.3 ··· 50 49 golang.org/x/image v0.36.0 51 50 golang.org/x/sys v0.41.0 52 51 golang.org/x/xerrors v0.0.0-20240903120638-7835f813f4da 53 - gorm.io/gorm v1.31.1 54 52 ) 55 53 56 54 require ( ··· 120 118 github.com/ipfs/bbloom v0.0.4 // indirect 121 119 github.com/ipfs/boxo v0.36.0 // indirect 122 120 github.com/ipfs/go-cidutil v0.1.1 // indirect 121 + github.com/ipfs/go-datastore v0.9.1 // indirect 123 122 github.com/ipfs/go-dsqueue v0.2.0 // indirect 124 123 github.com/ipfs/go-ipfs-ds-help v1.1.1 // indirect 125 124 github.com/ipfs/go-ipfs-util v0.0.3 // indirect ··· 213 212 google.golang.org/protobuf v1.36.11 // indirect 214 213 gopkg.in/yaml.v2 v2.4.0 // indirect 215 214 gopkg.in/yaml.v3 v3.0.1 // indirect 215 + gorm.io/gorm v1.31.1 // indirect 216 216 lukechampine.com/blake3 v1.4.1 // indirect 217 217 )
+4 -3
pkg/hold/pds/events.go
··· 56 56 Rev string `json:"rev" cborgen:"rev"` 57 57 Since *string `json:"since,omitempty" cborgen:"since,omitempty"` 58 58 PrevData string `json:"prevData,omitempty" cborgen:"prevData,omitempty"` // MST root CID string of previous commit 59 - Blocks []byte `json:"blocks" cborgen:"blocks"` // CAR slice bytes 59 + Blocks []byte `json:"blocks" cborgen:"blocks"` // CAR slice bytes 60 60 Ops []*atproto.SyncSubscribeRepos_RepoOp `json:"ops" cborgen:"ops"` 61 61 Time string `json:"time" cborgen:"time"` 62 62 Type string `json:"$type" cborgen:"$type"` // Always "#commit" ··· 170 170 } 171 171 } 172 172 173 - // Migration: add prev_data column if missing (existing databases) 174 - b.db.Exec("ALTER TABLE firehose_events ADD COLUMN prev_data TEXT") 173 + // Migration: add prev_data column if missing (existing databases). 174 + // Intentionally ignore error — fails with "duplicate column" if already present. 175 + _, _ = b.db.Exec("ALTER TABLE firehose_events ADD COLUMN prev_data TEXT") 175 176 176 177 // Load last sequence number from database 177 178 var lastSeq sql.NullInt64
+4 -104
pkg/hold/pds/import.go
··· 8 8 9 9 "github.com/bluesky-social/indigo/repo" 10 10 "github.com/ipfs/go-cid" 11 - "go.opentelemetry.io/otel" 12 11 ) 13 12 14 13 // rawCBOR wraps raw bytes to satisfy cbg.CBORMarshaler. ··· 20 19 return err 21 20 } 22 21 23 - // bulkRecord holds a single record to import. 24 - type bulkRecord struct { 25 - Collection string 26 - Rkey string 27 - Data rawCBOR 28 - } 29 - 30 22 // ImportResult summarizes a CAR import operation. 31 23 type ImportResult struct { 32 24 Total int ··· 52 44 } 53 45 54 46 // Collect all records 55 - var records []bulkRecord 47 + var records []BulkRecord 56 48 err = sourceRepo.ForEach(ctx, "", func(k string, v cid.Cid) error { 57 49 _, recBytes, err := sourceRepo.GetRecordBytes(ctx, k) 58 50 if err != nil { ··· 64 56 return fmt.Errorf("unexpected record path format: %s", k) 65 57 } 66 58 67 - records = append(records, bulkRecord{ 59 + records = append(records, BulkRecord{ 68 60 Collection: parts[0], 69 61 Rkey: parts[1], 70 62 Data: rawCBOR(*recBytes), ··· 79 71 return &ImportResult{PerCollection: map[string]int{}}, nil 80 72 } 81 73 82 - // Bulk upsert all records in a single commit 83 - if err := p.bulkImportRecords(ctx, records); err != nil { 74 + // Bulk upsert all records in a single commit via the RepoOperator interface 75 + if err := p.repomgr.BulkUpsert(ctx, p.uid, records); err != nil { 84 76 return nil, fmt.Errorf("failed to import records: %w", err) 85 77 } 86 78 ··· 94 86 } 95 87 return result, nil 96 88 } 97 - 98 - // bulkImportRecords writes all records in a single delta session + commit. 99 - // Each record is upserted: created if new, updated if exists. 100 - func (p *HoldPDS) bulkImportRecords(ctx context.Context, records []bulkRecord) error { 101 - ctx, span := otel.Tracer("repoman").Start(ctx, "BulkImportRecords") 102 - defer span.End() 103 - 104 - unlock := p.repomgr.lockUser(ctx, p.uid) 105 - defer unlock() 106 - 107 - rev, err := p.repomgr.cs.GetUserRepoRev(ctx, p.uid) 108 - if err != nil { 109 - return err 110 - } 111 - 112 - ds, err := p.repomgr.cs.NewDeltaSession(ctx, p.uid, &rev) 113 - if err != nil { 114 - return err 115 - } 116 - 117 - head := ds.BaseCid() 118 - r, err := repo.OpenRepo(ctx, ds, head) 119 - if err != nil { 120 - return err 121 - } 122 - 123 - // Capture previous MST root before commit overwrites it 124 - var prevData *cid.Cid 125 - if head.Defined() { 126 - pd := r.DataCid() 127 - prevData = &pd 128 - } 129 - 130 - ops := make([]RepoOp, 0, len(records)) 131 - for _, rec := range records { 132 - rpath := rec.Collection + "/" + rec.Rkey 133 - 134 - // Check if record exists to determine create vs update 135 - _, _, getErr := r.GetRecordBytes(ctx, rpath) 136 - recordExists := getErr == nil 137 - 138 - var cc cid.Cid 139 - var evtKind EventKind 140 - if recordExists { 141 - cc, err = r.UpdateRecord(ctx, rpath, rec.Data) 142 - evtKind = EvtKindUpdateRecord 143 - } else { 144 - cc, err = r.PutRecord(ctx, rpath, rec.Data) 145 - evtKind = EvtKindCreateRecord 146 - } 147 - if err != nil { 148 - return fmt.Errorf("failed to write %s: %w", rpath, err) 149 - } 150 - 151 - ops = append(ops, RepoOp{ 152 - Kind: evtKind, 153 - Collection: rec.Collection, 154 - Rkey: rec.Rkey, 155 - RecCid: &cc, 156 - }) 157 - } 158 - 159 - nroot, nrev, err := r.Commit(ctx, p.repomgr.kmgr.SignForUser) 160 - if err != nil { 161 - return err 162 - } 163 - 164 - rslice, err := ds.CloseWithRoot(ctx, nroot, nrev) 165 - if err != nil { 166 - return fmt.Errorf("close with root: %w", err) 167 - } 168 - 169 - var oldroot *cid.Cid 170 - if head.Defined() { 171 - oldroot = &head 172 - } 173 - 174 - if p.repomgr.events != nil { 175 - p.repomgr.events(ctx, &RepoEvent{ 176 - User: p.uid, 177 - OldRoot: oldroot, 178 - NewRoot: nroot, 179 - PrevData: prevData, 180 - Rev: nrev, 181 - Since: &rev, 182 - Ops: ops, 183 - RepoSlice: rslice, 184 - }) 185 - } 186 - 187 - return nil 188 - }
+548
pkg/hold/pds/repo.go
··· 1 + package pds 2 + 3 + // repo.go — DirectRepoOperator manages ATProto repository operations using 4 + // indigo/repo directly, replacing RepoManager with a simpler implementation. 5 + // 6 + // Key simplifications vs RepoManager: 7 + // - Single sync.Mutex instead of per-user lock map (hold is always uid=1) 8 + // - No OpenTelemetry tracing 9 + // - No gorm dependency 10 + // 11 + // Implements the RepoOperator interface (see repo_operator.go). 12 + // See docs/REPOMGR_MIGRATION.md for the migration plan. 13 + 14 + import ( 15 + "context" 16 + "fmt" 17 + "io" 18 + "log/slog" 19 + "sync" 20 + 21 + holddb "atcr.io/pkg/hold/db" 22 + atproto "github.com/bluesky-social/indigo/api/atproto" 23 + bsky "github.com/bluesky-social/indigo/api/bsky" 24 + "github.com/bluesky-social/indigo/atproto/syntax" 25 + "github.com/bluesky-social/indigo/models" 26 + "github.com/bluesky-social/indigo/repo" 27 + "github.com/bluesky-social/indigo/util" 28 + 29 + blocks "github.com/ipfs/go-block-format" 30 + "github.com/ipfs/go-cid" 31 + cbg "github.com/whyrusleeping/cbor-gen" 32 + ) 33 + 34 + // Compile-time check that DirectRepoOperator implements RepoOperator. 35 + var _ RepoOperator = (*DirectRepoOperator)(nil) 36 + 37 + // DirectRepoOperator implements RepoOperator using indigo/repo directly. 38 + type DirectRepoOperator struct { 39 + cs holddb.CarStore 40 + kmgr KeyManager 41 + mu sync.Mutex // single mutex (hold is always uid=1) 42 + events func(context.Context, *RepoEvent) 43 + hydrateRecords bool 44 + log *slog.Logger 45 + clk *syntax.TIDClock // for BatchWrite auto-rkey generation 46 + } 47 + 48 + // NewDirectRepoOperator creates a new DirectRepoOperator. 49 + func NewDirectRepoOperator(cs holddb.CarStore, kmgr KeyManager) *DirectRepoOperator { 50 + return &DirectRepoOperator{ 51 + cs: cs, 52 + kmgr: kmgr, 53 + log: slog.Default().With("system", "repo"), 54 + clk: syntax.NewTIDClock(0), 55 + } 56 + } 57 + 58 + // writeSession holds state for an in-progress write transaction. 59 + type writeSession struct { 60 + ds *holddb.DeltaSession 61 + r *repo.Repo 62 + head cid.Cid 63 + prevData *cid.Cid 64 + rev string 65 + } 66 + 67 + // openWriteSession opens a write session, acquiring the lock. 68 + // On success, the caller holds d.mu and must defer d.mu.Unlock(). 69 + // On error, the lock is released before returning. 70 + func (d *DirectRepoOperator) openWriteSession(ctx context.Context, user models.Uid) (*writeSession, error) { 71 + d.mu.Lock() 72 + 73 + rev, err := d.cs.GetUserRepoRev(ctx, user) 74 + if err != nil { 75 + d.mu.Unlock() 76 + return nil, err 77 + } 78 + 79 + ds, err := d.cs.NewDeltaSession(ctx, user, &rev) 80 + if err != nil { 81 + d.mu.Unlock() 82 + return nil, err 83 + } 84 + 85 + head := ds.BaseCid() 86 + 87 + r, err := repo.OpenRepo(ctx, ds, head) 88 + if err != nil { 89 + d.mu.Unlock() 90 + return nil, err 91 + } 92 + 93 + var prevData *cid.Cid 94 + if head.Defined() { 95 + pd := r.DataCid() 96 + prevData = &pd 97 + } 98 + 99 + return &writeSession{ 100 + ds: ds, 101 + r: r, 102 + head: head, 103 + prevData: prevData, 104 + rev: rev, 105 + }, nil 106 + } 107 + 108 + // commitWrite commits a write session and emits an event if configured. 109 + func (d *DirectRepoOperator) commitWrite(ctx context.Context, ws *writeSession, user models.Uid, ops []RepoOp) (cid.Cid, string, error) { 110 + nroot, nrev, err := ws.r.Commit(ctx, d.kmgr.SignForUser) 111 + if err != nil { 112 + return cid.Undef, "", err 113 + } 114 + 115 + rslice, err := ws.ds.CloseWithRoot(ctx, nroot, nrev) 116 + if err != nil { 117 + return cid.Undef, "", fmt.Errorf("close with root: %w", err) 118 + } 119 + 120 + if d.events != nil { 121 + var oldroot *cid.Cid 122 + if ws.head.Defined() { 123 + oldroot = &ws.head 124 + } 125 + 126 + d.events(ctx, &RepoEvent{ 127 + User: user, 128 + OldRoot: oldroot, 129 + NewRoot: nroot, 130 + PrevData: ws.prevData, 131 + Rev: nrev, 132 + Since: &ws.rev, 133 + Ops: ops, 134 + RepoSlice: rslice, 135 + }) 136 + } 137 + 138 + return nroot, nrev, nil 139 + } 140 + 141 + func (d *DirectRepoOperator) SetEventHandler(cb func(context.Context, *RepoEvent), hydrateRecords bool) { 142 + d.events = cb 143 + d.hydrateRecords = hydrateRecords 144 + } 145 + 146 + func (d *DirectRepoOperator) CreateRecord(ctx context.Context, user models.Uid, collection string, rec cbg.CBORMarshaler) (string, cid.Cid, error) { 147 + ws, err := d.openWriteSession(ctx, user) 148 + if err != nil { 149 + return "", cid.Undef, err 150 + } 151 + defer d.mu.Unlock() 152 + 153 + cc, tid, err := ws.r.CreateRecord(ctx, collection, rec) 154 + if err != nil { 155 + return "", cid.Undef, err 156 + } 157 + 158 + ops := []RepoOp{{ 159 + Kind: EvtKindCreateRecord, 160 + Collection: collection, 161 + Rkey: tid, 162 + Record: rec, // CreateRecord always includes Record (no hydration check) 163 + RecCid: &cc, 164 + }} 165 + 166 + if _, _, err := d.commitWrite(ctx, ws, user, ops); err != nil { 167 + return "", cid.Undef, err 168 + } 169 + 170 + return collection + "/" + tid, cc, nil 171 + } 172 + 173 + func (d *DirectRepoOperator) UpdateRecord(ctx context.Context, user models.Uid, collection, rkey string, rec cbg.CBORMarshaler) (cid.Cid, error) { 174 + ws, err := d.openWriteSession(ctx, user) 175 + if err != nil { 176 + return cid.Undef, err 177 + } 178 + defer d.mu.Unlock() 179 + 180 + rpath := collection + "/" + rkey 181 + cc, err := ws.r.UpdateRecord(ctx, rpath, rec) 182 + if err != nil { 183 + return cid.Undef, err 184 + } 185 + 186 + op := RepoOp{ 187 + Kind: EvtKindUpdateRecord, 188 + Collection: collection, 189 + Rkey: rkey, 190 + RecCid: &cc, 191 + } 192 + if d.hydrateRecords { 193 + op.Record = rec 194 + } 195 + 196 + if _, _, err := d.commitWrite(ctx, ws, user, []RepoOp{op}); err != nil { 197 + return cid.Undef, err 198 + } 199 + 200 + return cc, nil 201 + } 202 + 203 + func (d *DirectRepoOperator) PutRecord(ctx context.Context, user models.Uid, collection, rkey string, rec cbg.CBORMarshaler) (string, cid.Cid, error) { 204 + ws, err := d.openWriteSession(ctx, user) 205 + if err != nil { 206 + return "", cid.Undef, err 207 + } 208 + defer d.mu.Unlock() 209 + 210 + rpath := collection + "/" + rkey 211 + cc, err := ws.r.PutRecord(ctx, rpath, rec) 212 + if err != nil { 213 + return "", cid.Undef, err 214 + } 215 + 216 + op := RepoOp{ 217 + Kind: EvtKindCreateRecord, 218 + Collection: collection, 219 + Rkey: rkey, 220 + RecCid: &cc, 221 + } 222 + if d.hydrateRecords { 223 + op.Record = rec 224 + } 225 + 226 + if _, _, err := d.commitWrite(ctx, ws, user, []RepoOp{op}); err != nil { 227 + return "", cid.Undef, err 228 + } 229 + 230 + return rpath, cc, nil 231 + } 232 + 233 + func (d *DirectRepoOperator) UpsertRecord(ctx context.Context, user models.Uid, collection, rkey string, rec cbg.CBORMarshaler) (string, cid.Cid, bool, error) { 234 + ws, err := d.openWriteSession(ctx, user) 235 + if err != nil { 236 + return "", cid.Undef, false, err 237 + } 238 + defer d.mu.Unlock() 239 + 240 + rpath := collection + "/" + rkey 241 + 242 + // Check if record exists 243 + _, _, err = ws.r.GetRecordBytes(ctx, rpath) 244 + recordExists := err == nil 245 + 246 + var cc cid.Cid 247 + var evtKind EventKind 248 + if recordExists { 249 + cc, err = ws.r.UpdateRecord(ctx, rpath, rec) 250 + evtKind = EvtKindUpdateRecord 251 + } else { 252 + cc, err = ws.r.PutRecord(ctx, rpath, rec) 253 + evtKind = EvtKindCreateRecord 254 + } 255 + if err != nil { 256 + return "", cid.Undef, false, err 257 + } 258 + 259 + op := RepoOp{ 260 + Kind: evtKind, 261 + Collection: collection, 262 + Rkey: rkey, 263 + RecCid: &cc, 264 + } 265 + if d.hydrateRecords { 266 + op.Record = rec 267 + } 268 + 269 + if _, _, err := d.commitWrite(ctx, ws, user, []RepoOp{op}); err != nil { 270 + return "", cid.Undef, false, err 271 + } 272 + 273 + return rpath, cc, !recordExists, nil 274 + } 275 + 276 + func (d *DirectRepoOperator) DeleteRecord(ctx context.Context, user models.Uid, collection, rkey string) error { 277 + ws, err := d.openWriteSession(ctx, user) 278 + if err != nil { 279 + return err 280 + } 281 + defer d.mu.Unlock() 282 + 283 + rpath := collection + "/" + rkey 284 + if err := ws.r.DeleteRecord(ctx, rpath); err != nil { 285 + return err 286 + } 287 + 288 + ops := []RepoOp{{ 289 + Kind: EvtKindDeleteRecord, 290 + Collection: collection, 291 + Rkey: rkey, 292 + }} 293 + 294 + _, _, err = d.commitWrite(ctx, ws, user, ops) 295 + return err 296 + } 297 + 298 + func (d *DirectRepoOperator) BatchWrite(ctx context.Context, user models.Uid, writes []*atproto.RepoApplyWrites_Input_Writes_Elem) error { 299 + ws, err := d.openWriteSession(ctx, user) 300 + if err != nil { 301 + return err 302 + } 303 + defer d.mu.Unlock() 304 + 305 + ops := make([]RepoOp, 0, len(writes)) 306 + for _, w := range writes { 307 + switch { 308 + case w.RepoApplyWrites_Create != nil: 309 + c := w.RepoApplyWrites_Create 310 + var rkey string 311 + if c.Rkey != nil { 312 + rkey = *c.Rkey 313 + } else { 314 + rkey = d.clk.Next().String() 315 + } 316 + 317 + nsid := c.Collection + "/" + rkey 318 + cc, err := ws.r.PutRecord(ctx, nsid, c.Value.Val) 319 + if err != nil { 320 + return err 321 + } 322 + 323 + op := RepoOp{ 324 + Kind: EvtKindCreateRecord, 325 + Collection: c.Collection, 326 + Rkey: rkey, 327 + RecCid: &cc, 328 + } 329 + if d.hydrateRecords { 330 + op.Record = c.Value.Val 331 + } 332 + ops = append(ops, op) 333 + 334 + case w.RepoApplyWrites_Update != nil: 335 + u := w.RepoApplyWrites_Update 336 + 337 + // Known quirk: uses PutRecord (mst.Add) not UpdateRecord 338 + cc, err := ws.r.PutRecord(ctx, u.Collection+"/"+u.Rkey, u.Value.Val) 339 + if err != nil { 340 + return err 341 + } 342 + 343 + op := RepoOp{ 344 + Kind: EvtKindUpdateRecord, 345 + Collection: u.Collection, 346 + Rkey: u.Rkey, 347 + RecCid: &cc, 348 + } 349 + if d.hydrateRecords { 350 + op.Record = u.Value.Val 351 + } 352 + ops = append(ops, op) 353 + 354 + case w.RepoApplyWrites_Delete != nil: 355 + del := w.RepoApplyWrites_Delete 356 + 357 + if err := ws.r.DeleteRecord(ctx, del.Collection+"/"+del.Rkey); err != nil { 358 + return err 359 + } 360 + 361 + ops = append(ops, RepoOp{ 362 + Kind: EvtKindDeleteRecord, 363 + Collection: del.Collection, 364 + Rkey: del.Rkey, 365 + }) 366 + 367 + default: 368 + return fmt.Errorf("no operation set in write enum") 369 + } 370 + } 371 + 372 + _, _, err = d.commitWrite(ctx, ws, user, ops) 373 + return err 374 + } 375 + 376 + func (d *DirectRepoOperator) BulkUpsert(ctx context.Context, user models.Uid, records []BulkRecord) error { 377 + ws, err := d.openWriteSession(ctx, user) 378 + if err != nil { 379 + return err 380 + } 381 + defer d.mu.Unlock() 382 + 383 + ops := make([]RepoOp, 0, len(records)) 384 + for _, rec := range records { 385 + rpath := rec.Collection + "/" + rec.Rkey 386 + 387 + // Check if record exists to determine create vs update 388 + _, _, getErr := ws.r.GetRecordBytes(ctx, rpath) 389 + recordExists := getErr == nil 390 + 391 + var cc cid.Cid 392 + var evtKind EventKind 393 + if recordExists { 394 + cc, err = ws.r.UpdateRecord(ctx, rpath, rec.Data) 395 + evtKind = EvtKindUpdateRecord 396 + } else { 397 + cc, err = ws.r.PutRecord(ctx, rpath, rec.Data) 398 + evtKind = EvtKindCreateRecord 399 + } 400 + if err != nil { 401 + return fmt.Errorf("failed to write %s: %w", rpath, err) 402 + } 403 + 404 + // No hydration for BulkUpsert (records never included in events) 405 + ops = append(ops, RepoOp{ 406 + Kind: evtKind, 407 + Collection: rec.Collection, 408 + Rkey: rec.Rkey, 409 + RecCid: &cc, 410 + }) 411 + } 412 + 413 + _, _, err = d.commitWrite(ctx, ws, user, ops) 414 + return err 415 + } 416 + 417 + func (d *DirectRepoOperator) InitNewActor(ctx context.Context, user models.Uid, handle, did, displayname string, declcid, actortype string) error { 418 + d.mu.Lock() 419 + defer d.mu.Unlock() 420 + 421 + if did == "" { 422 + return fmt.Errorf("must specify DID for new actor") 423 + } 424 + 425 + if user == 0 { 426 + return fmt.Errorf("must specify user for new actor") 427 + } 428 + 429 + ds, err := d.cs.NewDeltaSession(ctx, user, nil) 430 + if err != nil { 431 + return fmt.Errorf("creating new delta session: %w", err) 432 + } 433 + 434 + r := repo.NewRepo(ctx, did, ds) 435 + 436 + profile := &bsky.ActorProfile{ 437 + DisplayName: &displayname, 438 + } 439 + 440 + _, err = r.PutRecord(ctx, "app.bsky.actor.profile/self", profile) 441 + if err != nil { 442 + return fmt.Errorf("setting initial actor profile: %w", err) 443 + } 444 + 445 + root, nrev, err := r.Commit(ctx, d.kmgr.SignForUser) 446 + if err != nil { 447 + return fmt.Errorf("committing repo for actor init: %w", err) 448 + } 449 + 450 + rslice, err := ds.CloseWithRoot(ctx, root, nrev) 451 + if err != nil { 452 + return fmt.Errorf("close with root: %w", err) 453 + } 454 + 455 + if d.events != nil { 456 + op := RepoOp{ 457 + Kind: EvtKindCreateRecord, 458 + Collection: "app.bsky.actor.profile", 459 + Rkey: "self", 460 + } 461 + 462 + if d.hydrateRecords { 463 + op.Record = profile 464 + } 465 + 466 + d.events(ctx, &RepoEvent{ 467 + User: user, 468 + NewRoot: root, 469 + Rev: nrev, 470 + Ops: []RepoOp{op}, 471 + RepoSlice: rslice, 472 + }) 473 + } 474 + 475 + return nil 476 + } 477 + 478 + func (d *DirectRepoOperator) GetRecord(ctx context.Context, user models.Uid, collection string, rkey string, maybeCid cid.Cid) (cid.Cid, cbg.CBORMarshaler, error) { 479 + bs, err := d.cs.ReadOnlySession(user) 480 + if err != nil { 481 + return cid.Undef, nil, err 482 + } 483 + 484 + head, err := d.cs.GetUserRepoHead(ctx, user) 485 + if err != nil { 486 + return cid.Undef, nil, err 487 + } 488 + 489 + r, err := repo.OpenRepo(ctx, bs, head) 490 + if err != nil { 491 + return cid.Undef, nil, err 492 + } 493 + 494 + ocid, val, err := r.GetRecord(ctx, collection+"/"+rkey) 495 + if err != nil { 496 + return cid.Undef, nil, err 497 + } 498 + 499 + if maybeCid.Defined() && ocid != maybeCid { 500 + return cid.Undef, nil, fmt.Errorf("record at specified key had different CID than expected") 501 + } 502 + 503 + return ocid, val, nil 504 + } 505 + 506 + func (d *DirectRepoOperator) GetRecordProof(ctx context.Context, user models.Uid, collection string, rkey string) (cid.Cid, []blocks.Block, error) { 507 + robs, err := d.cs.ReadOnlySession(user) 508 + if err != nil { 509 + return cid.Undef, nil, err 510 + } 511 + 512 + bs := util.NewLoggingBstore(robs) 513 + 514 + head, err := d.cs.GetUserRepoHead(ctx, user) 515 + if err != nil { 516 + return cid.Undef, nil, err 517 + } 518 + 519 + r, err := repo.OpenRepo(ctx, bs, head) 520 + if err != nil { 521 + return cid.Undef, nil, err 522 + } 523 + 524 + _, _, err = r.GetRecordBytes(ctx, collection+"/"+rkey) 525 + if err != nil { 526 + return cid.Undef, nil, err 527 + } 528 + 529 + return head, bs.GetLoggedBlocks(), nil 530 + } 531 + 532 + func (d *DirectRepoOperator) GetRepoRoot(ctx context.Context, user models.Uid) (cid.Cid, error) { 533 + d.mu.Lock() 534 + defer d.mu.Unlock() 535 + 536 + return d.cs.GetUserRepoHead(ctx, user) 537 + } 538 + 539 + func (d *DirectRepoOperator) GetRepoRev(ctx context.Context, user models.Uid) (string, error) { 540 + d.mu.Lock() 541 + defer d.mu.Unlock() 542 + 543 + return d.cs.GetUserRepoRev(ctx, user) 544 + } 545 + 546 + func (d *DirectRepoOperator) ReadRepo(ctx context.Context, user models.Uid, since string, w io.Writer) error { 547 + return d.cs.ReadUserCar(ctx, user, since, true, w) 548 + }
+96
pkg/hold/pds/repo_operator.go
··· 1 + // Package pds implements a minimal ATProto PDS for the hold service. 2 + package pds 3 + 4 + import ( 5 + "context" 6 + "io" 7 + 8 + atproto "github.com/bluesky-social/indigo/api/atproto" 9 + "github.com/bluesky-social/indigo/models" 10 + 11 + blocks "github.com/ipfs/go-block-format" 12 + "github.com/ipfs/go-cid" 13 + cbg "github.com/whyrusleeping/cbor-gen" 14 + ) 15 + 16 + // RepoOperator defines the interface for ATProto repository operations. 17 + // RepoManager implements this interface. Future implementations (e.g., using 18 + // indigo/repo directly) can be swapped in behind this interface. 19 + // See docs/REPOMGR_MIGRATION.md for the migration plan. 20 + type RepoOperator interface { 21 + // Record CRUD 22 + CreateRecord(ctx context.Context, user models.Uid, collection string, rec cbg.CBORMarshaler) (string, cid.Cid, error) 23 + UpdateRecord(ctx context.Context, user models.Uid, collection, rkey string, rec cbg.CBORMarshaler) (cid.Cid, error) 24 + PutRecord(ctx context.Context, user models.Uid, collection, rkey string, rec cbg.CBORMarshaler) (string, cid.Cid, error) 25 + UpsertRecord(ctx context.Context, user models.Uid, collection, rkey string, rec cbg.CBORMarshaler) (string, cid.Cid, bool, error) 26 + DeleteRecord(ctx context.Context, user models.Uid, collection, rkey string) error 27 + BatchWrite(ctx context.Context, user models.Uid, writes []*atproto.RepoApplyWrites_Input_Writes_Elem) error 28 + BulkUpsert(ctx context.Context, user models.Uid, records []BulkRecord) error 29 + 30 + // Read 31 + GetRecord(ctx context.Context, user models.Uid, collection string, rkey string, maybeCid cid.Cid) (cid.Cid, cbg.CBORMarshaler, error) 32 + GetRecordProof(ctx context.Context, user models.Uid, collection string, rkey string) (cid.Cid, []blocks.Block, error) 33 + GetRepoRoot(ctx context.Context, user models.Uid) (cid.Cid, error) 34 + GetRepoRev(ctx context.Context, user models.Uid) (string, error) 35 + ReadRepo(ctx context.Context, user models.Uid, since string, w io.Writer) error 36 + 37 + // Lifecycle 38 + InitNewActor(ctx context.Context, user models.Uid, handle, did, displayname string, declcid, actortype string) error 39 + SetEventHandler(cb func(context.Context, *RepoEvent), hydrateRecords bool) 40 + } 41 + 42 + // KeyManager handles cryptographic signing for repository commits. 43 + type KeyManager interface { 44 + VerifyUserSignature(context.Context, string, []byte, []byte) error 45 + SignForUser(context.Context, string, []byte) ([]byte, error) 46 + } 47 + 48 + // ActorInfo holds identity information for a repository actor. 49 + type ActorInfo struct { 50 + Did string 51 + Handle string 52 + DisplayName string 53 + Type string 54 + } 55 + 56 + // Compile-time check that RepoManager implements RepoOperator. 57 + var _ RepoOperator = (*RepoManager)(nil) 58 + 59 + // RepoEvent represents a mutation event emitted by a RepoOperator. 60 + type RepoEvent struct { 61 + User models.Uid 62 + OldRoot *cid.Cid 63 + NewRoot cid.Cid 64 + PrevData *cid.Cid // MST root CID of the previous commit (for firehose prevData field) 65 + Since *string 66 + Rev string 67 + RepoSlice []byte 68 + PDS uint 69 + Ops []RepoOp 70 + } 71 + 72 + // RepoOp represents a single operation within a RepoEvent. 73 + type RepoOp struct { 74 + Kind EventKind 75 + Collection string 76 + Rkey string 77 + RecCid *cid.Cid 78 + Record any 79 + ActorInfo *ActorInfo 80 + } 81 + 82 + // EventKind identifies the type of repository mutation. 83 + type EventKind string 84 + 85 + const ( 86 + EvtKindCreateRecord = EventKind("create") 87 + EvtKindUpdateRecord = EventKind("update") 88 + EvtKindDeleteRecord = EventKind("delete") 89 + ) 90 + 91 + // BulkRecord holds a single record for bulk import/upsert operations. 92 + type BulkRecord struct { 93 + Collection string 94 + Rkey string 95 + Data cbg.CBORMarshaler 96 + }
+418
pkg/hold/pds/repo_operator_benchmark_test.go
··· 1 + package pds 2 + 3 + import ( 4 + "bytes" 5 + "context" 6 + "fmt" 7 + "io" 8 + "log/slog" 9 + "os" 10 + "path/filepath" 11 + "testing" 12 + 13 + "atcr.io/pkg/atproto" 14 + "atcr.io/pkg/auth/oauth" 15 + holddb "atcr.io/pkg/hold/db" 16 + "github.com/bluesky-social/indigo/models" 17 + "github.com/ipfs/go-cid" 18 + 19 + indigoatproto "github.com/bluesky-social/indigo/api/atproto" 20 + lexutil "github.com/bluesky-social/indigo/lex/util" 21 + ) 22 + 23 + // benchSetup creates a RepoOperator and returns it with the user ID. 24 + type benchSetup func(b *testing.B) (RepoOperator, models.Uid) 25 + 26 + func suppressLogs(b *testing.B) { 27 + b.Helper() 28 + prev := slog.Default() 29 + slog.SetDefault(slog.New(slog.NewTextHandler(io.Discard, nil))) 30 + b.Cleanup(func() { slog.SetDefault(prev) }) 31 + } 32 + 33 + func setupBenchRepoManager(b *testing.B) (RepoOperator, models.Uid) { 34 + b.Helper() 35 + suppressLogs(b) 36 + ctx := context.Background() 37 + tmpDir := b.TempDir() 38 + keyPath := filepath.Join(tmpDir, "signing-key") 39 + 40 + if err := os.WriteFile(keyPath, sharedTestKey, 0600); err != nil { 41 + b.Fatalf("write signing key: %v", err) 42 + } 43 + 44 + pds, err := NewHoldPDS(ctx, "did:web:hold.bench", "https://hold.bench", "https://atcr.io", ":memory:", keyPath, false) 45 + if err != nil { 46 + b.Fatalf("NewHoldPDS: %v", err) 47 + } 48 + 49 + if err := pds.repomgr.InitNewActor(ctx, pds.uid, "", pds.did, "", "", ""); err != nil { 50 + b.Fatalf("InitNewActor: %v", err) 51 + } 52 + 53 + b.Cleanup(func() { pds.Close() }) 54 + return pds.repomgr, pds.uid 55 + } 56 + 57 + func setupBenchDirectRepoOperator(b *testing.B) (RepoOperator, models.Uid) { 58 + b.Helper() 59 + suppressLogs(b) 60 + ctx := context.Background() 61 + keyPath := filepath.Join(b.TempDir(), "signing-key") 62 + 63 + if err := os.WriteFile(keyPath, sharedTestKey, 0600); err != nil { 64 + b.Fatalf("write signing key: %v", err) 65 + } 66 + signingKey, err := oauth.GenerateOrLoadPDSKey(keyPath) 67 + if err != nil { 68 + b.Fatalf("GenerateOrLoadPDSKey: %v", err) 69 + } 70 + 71 + sqlStore := new(holddb.SQLiteStore) 72 + if err := sqlStore.Open(":memory:"); err != nil { 73 + b.Fatalf("SQLiteStore.Open: %v", err) 74 + } 75 + b.Cleanup(func() { sqlStore.Close() }) 76 + 77 + kmgr := NewHoldKeyManager(signingKey) 78 + op := NewDirectRepoOperator(sqlStore, kmgr) 79 + uid := models.Uid(1) 80 + 81 + if err := op.InitNewActor(ctx, uid, "", "did:web:hold.bench", "", "", ""); err != nil { 82 + b.Fatalf("InitNewActor: %v", err) 83 + } 84 + 85 + return op, uid 86 + } 87 + 88 + // seedRecords writes n records and returns their rkeys. 89 + func seedRecords(b *testing.B, op RepoOperator, uid models.Uid, n int) []string { 90 + b.Helper() 91 + ctx := context.Background() 92 + rkeys := make([]string, n) 93 + for i := 0; i < n; i++ { 94 + rkey := fmt.Sprintf("seed%d", i) 95 + _, _, err := op.PutRecord(ctx, uid, atproto.CrewCollection, rkey, newCrewRecord(fmt.Sprintf("did:plc:seed%d", i))) 96 + if err != nil { 97 + b.Fatalf("seed PutRecord %d: %v", i, err) 98 + } 99 + rkeys[i] = rkey 100 + } 101 + return rkeys 102 + } 103 + 104 + func runRepoOperatorBenchmarks(b *testing.B, name string, setup benchSetup) { 105 + b.Run(name, func(b *testing.B) { 106 + b.Run("CreateRecord", func(b *testing.B) { 107 + op, uid := setup(b) 108 + ctx := context.Background() 109 + 110 + b.ResetTimer() 111 + for i := 0; i < b.N; i++ { 112 + _, _, err := op.CreateRecord(ctx, uid, atproto.CrewCollection, newCrewRecord(fmt.Sprintf("did:plc:bench%d", i))) 113 + if err != nil { 114 + b.Fatalf("CreateRecord: %v", err) 115 + } 116 + } 117 + }) 118 + 119 + b.Run("PutRecord", func(b *testing.B) { 120 + op, uid := setup(b) 121 + ctx := context.Background() 122 + 123 + b.ResetTimer() 124 + for i := 0; i < b.N; i++ { 125 + rkey := fmt.Sprintf("put%d", i) 126 + _, _, err := op.PutRecord(ctx, uid, atproto.CrewCollection, rkey, newCrewRecord(fmt.Sprintf("did:plc:put%d", i))) 127 + if err != nil { 128 + b.Fatalf("PutRecord: %v", err) 129 + } 130 + } 131 + }) 132 + 133 + b.Run("UpdateRecord", func(b *testing.B) { 134 + op, uid := setup(b) 135 + ctx := context.Background() 136 + 137 + // Seed one record to update repeatedly 138 + _, _, err := op.PutRecord(ctx, uid, atproto.CrewCollection, "updbench", newCrewRecord("did:plc:updbench")) 139 + if err != nil { 140 + b.Fatalf("seed PutRecord: %v", err) 141 + } 142 + 143 + b.ResetTimer() 144 + for i := 0; i < b.N; i++ { 145 + rec := newCrewRecord("did:plc:updbench") 146 + rec.Role = fmt.Sprintf("role%d", i) 147 + _, err := op.UpdateRecord(ctx, uid, atproto.CrewCollection, "updbench", rec) 148 + if err != nil { 149 + b.Fatalf("UpdateRecord: %v", err) 150 + } 151 + } 152 + }) 153 + 154 + b.Run("DeleteRecord", func(b *testing.B) { 155 + op, uid := setup(b) 156 + ctx := context.Background() 157 + 158 + // Pre-create all records to delete 159 + rkeys := make([]string, b.N) 160 + for i := 0; i < b.N; i++ { 161 + rkey := fmt.Sprintf("del%d", i) 162 + _, _, err := op.PutRecord(ctx, uid, atproto.CrewCollection, rkey, newCrewRecord(fmt.Sprintf("did:plc:del%d", i))) 163 + if err != nil { 164 + b.Fatalf("seed PutRecord: %v", err) 165 + } 166 + rkeys[i] = rkey 167 + } 168 + 169 + b.ResetTimer() 170 + for i := 0; i < b.N; i++ { 171 + if err := op.DeleteRecord(ctx, uid, atproto.CrewCollection, rkeys[i]); err != nil { 172 + b.Fatalf("DeleteRecord: %v", err) 173 + } 174 + } 175 + }) 176 + 177 + b.Run("GetRecord", func(b *testing.B) { 178 + op, uid := setup(b) 179 + ctx := context.Background() 180 + 181 + _, _, err := op.PutRecord(ctx, uid, atproto.CrewCollection, "getbench", newCrewRecord("did:plc:getbench")) 182 + if err != nil { 183 + b.Fatalf("seed PutRecord: %v", err) 184 + } 185 + 186 + b.ResetTimer() 187 + for i := 0; i < b.N; i++ { 188 + _, _, err := op.GetRecord(ctx, uid, atproto.CrewCollection, "getbench", cid.Undef) 189 + if err != nil { 190 + b.Fatalf("GetRecord: %v", err) 191 + } 192 + } 193 + }) 194 + 195 + b.Run("GetRepoRev", func(b *testing.B) { 196 + op, uid := setup(b) 197 + ctx := context.Background() 198 + 199 + b.ResetTimer() 200 + for i := 0; i < b.N; i++ { 201 + _, err := op.GetRepoRev(ctx, uid) 202 + if err != nil { 203 + b.Fatalf("GetRepoRev: %v", err) 204 + } 205 + } 206 + }) 207 + 208 + b.Run("GetRepoRoot", func(b *testing.B) { 209 + op, uid := setup(b) 210 + ctx := context.Background() 211 + 212 + b.ResetTimer() 213 + for i := 0; i < b.N; i++ { 214 + _, err := op.GetRepoRoot(ctx, uid) 215 + if err != nil { 216 + b.Fatalf("GetRepoRoot: %v", err) 217 + } 218 + } 219 + }) 220 + 221 + // BatchWrite at different sizes — shows commit overhead vs per-record cost 222 + for _, size := range []int{1, 10, 100} { 223 + b.Run(fmt.Sprintf("BatchWrite_%d", size), func(b *testing.B) { 224 + op, uid := setup(b) 225 + ctx := context.Background() 226 + 227 + b.ResetTimer() 228 + for i := 0; i < b.N; i++ { 229 + writes := make([]*indigoatproto.RepoApplyWrites_Input_Writes_Elem, size) 230 + for j := 0; j < size; j++ { 231 + rkey := fmt.Sprintf("batch%d_%d", i, j) 232 + writes[j] = &indigoatproto.RepoApplyWrites_Input_Writes_Elem{ 233 + RepoApplyWrites_Create: &indigoatproto.RepoApplyWrites_Create{ 234 + Collection: atproto.CrewCollection, 235 + Rkey: &rkey, 236 + Value: &lexutil.LexiconTypeDecoder{Val: newCrewRecord(fmt.Sprintf("did:plc:batch%d_%d", i, j))}, 237 + }, 238 + } 239 + } 240 + if err := op.BatchWrite(ctx, uid, writes); err != nil { 241 + b.Fatalf("BatchWrite: %v", err) 242 + } 243 + } 244 + }) 245 + } 246 + 247 + // ReadRepo at different repo sizes 248 + for _, size := range []int{10, 100, 500} { 249 + b.Run(fmt.Sprintf("ReadRepo_%drecords", size), func(b *testing.B) { 250 + op, uid := setup(b) 251 + ctx := context.Background() 252 + seedRecords(b, op, uid, size) 253 + 254 + var buf bytes.Buffer 255 + b.ResetTimer() 256 + for i := 0; i < b.N; i++ { 257 + buf.Reset() 258 + if err := op.ReadRepo(ctx, uid, "", &buf); err != nil { 259 + b.Fatalf("ReadRepo: %v", err) 260 + } 261 + } 262 + b.SetBytes(int64(buf.Len())) 263 + }) 264 + } 265 + 266 + // Event overhead: with vs without handler 267 + b.Run("PutRecord_NoEvents", func(b *testing.B) { 268 + op, uid := setup(b) 269 + ctx := context.Background() 270 + // No event handler set 271 + 272 + b.ResetTimer() 273 + for i := 0; i < b.N; i++ { 274 + rkey := fmt.Sprintf("noevt%d", i) 275 + _, _, err := op.PutRecord(ctx, uid, atproto.CrewCollection, rkey, newCrewRecord(fmt.Sprintf("did:plc:noevt%d", i))) 276 + if err != nil { 277 + b.Fatalf("PutRecord: %v", err) 278 + } 279 + } 280 + }) 281 + 282 + b.Run("PutRecord_WithEvents", func(b *testing.B) { 283 + op, uid := setup(b) 284 + ctx := context.Background() 285 + op.SetEventHandler(func(_ context.Context, _ *RepoEvent) {}, false) 286 + 287 + b.ResetTimer() 288 + for i := 0; i < b.N; i++ { 289 + rkey := fmt.Sprintf("evt%d", i) 290 + _, _, err := op.PutRecord(ctx, uid, atproto.CrewCollection, rkey, newCrewRecord(fmt.Sprintf("did:plc:evt%d", i))) 291 + if err != nil { 292 + b.Fatalf("PutRecord: %v", err) 293 + } 294 + } 295 + }) 296 + 297 + b.Run("PutRecord_WithHydration", func(b *testing.B) { 298 + op, uid := setup(b) 299 + ctx := context.Background() 300 + op.SetEventHandler(func(_ context.Context, _ *RepoEvent) {}, true) 301 + 302 + b.ResetTimer() 303 + for i := 0; i < b.N; i++ { 304 + rkey := fmt.Sprintf("hyd%d", i) 305 + _, _, err := op.PutRecord(ctx, uid, atproto.CrewCollection, rkey, newCrewRecord(fmt.Sprintf("did:plc:hyd%d", i))) 306 + if err != nil { 307 + b.Fatalf("PutRecord: %v", err) 308 + } 309 + } 310 + }) 311 + 312 + // Read from a repo with many records (MST depth) 313 + b.Run("GetRecord_LargeRepo", func(b *testing.B) { 314 + op, uid := setup(b) 315 + ctx := context.Background() 316 + rkeys := seedRecords(b, op, uid, 500) 317 + target := rkeys[len(rkeys)/2] // pick a middle record 318 + 319 + b.ResetTimer() 320 + for i := 0; i < b.N; i++ { 321 + _, _, err := op.GetRecord(ctx, uid, atproto.CrewCollection, target, cid.Undef) 322 + if err != nil { 323 + b.Fatalf("GetRecord: %v", err) 324 + } 325 + } 326 + }) 327 + 328 + // ReadRepo incremental (since) vs full 329 + b.Run("ReadRepo_Incremental", func(b *testing.B) { 330 + op, uid := setup(b) 331 + ctx := context.Background() 332 + seedRecords(b, op, uid, 100) 333 + 334 + // Get rev before the last write 335 + rev, err := op.GetRepoRev(ctx, uid) 336 + if err != nil { 337 + b.Fatalf("GetRepoRev: %v", err) 338 + } 339 + 340 + // Write one more record 341 + _, _, err = op.PutRecord(ctx, uid, atproto.CrewCollection, "incremental", newCrewRecord("did:plc:incremental")) 342 + if err != nil { 343 + b.Fatalf("PutRecord: %v", err) 344 + } 345 + 346 + var buf bytes.Buffer 347 + b.ResetTimer() 348 + for i := 0; i < b.N; i++ { 349 + buf.Reset() 350 + if err := op.ReadRepo(ctx, uid, rev, &buf); err != nil { 351 + b.Fatalf("ReadRepo: %v", err) 352 + } 353 + } 354 + b.SetBytes(int64(buf.Len())) 355 + }) 356 + }) 357 + } 358 + 359 + func BenchmarkRepoManager(b *testing.B) { 360 + runRepoOperatorBenchmarks(b, "RepoManager", setupBenchRepoManager) 361 + } 362 + 363 + func BenchmarkDirectRepoOperator(b *testing.B) { 364 + runRepoOperatorBenchmarks(b, "DirectRepoOperator", setupBenchDirectRepoOperator) 365 + } 366 + 367 + // BenchmarkBatchVsSingle compares the cost of N individual PutRecord calls 368 + // vs a single BatchWrite with N records. 369 + func BenchmarkBatchVsSingle(b *testing.B) { 370 + for _, impl := range []struct { 371 + name string 372 + setup benchSetup 373 + }{ 374 + {"RepoManager", setupBenchRepoManager}, 375 + {"DirectRepoOperator", setupBenchDirectRepoOperator}, 376 + } { 377 + for _, size := range []int{1, 10, 50} { 378 + b.Run(fmt.Sprintf("%s/Single_%d", impl.name, size), func(b *testing.B) { 379 + op, uid := impl.setup(b) 380 + ctx := context.Background() 381 + 382 + b.ResetTimer() 383 + for i := 0; i < b.N; i++ { 384 + for j := 0; j < size; j++ { 385 + rkey := fmt.Sprintf("single%d_%d", i, j) 386 + _, _, err := op.PutRecord(ctx, uid, atproto.CrewCollection, rkey, newCrewRecord(fmt.Sprintf("did:plc:s%d_%d", i, j))) 387 + if err != nil { 388 + b.Fatalf("PutRecord: %v", err) 389 + } 390 + } 391 + } 392 + }) 393 + 394 + b.Run(fmt.Sprintf("%s/Batch_%d", impl.name, size), func(b *testing.B) { 395 + op, uid := impl.setup(b) 396 + ctx := context.Background() 397 + 398 + b.ResetTimer() 399 + for i := 0; i < b.N; i++ { 400 + writes := make([]*indigoatproto.RepoApplyWrites_Input_Writes_Elem, size) 401 + for j := 0; j < size; j++ { 402 + rkey := fmt.Sprintf("batch%d_%d", i, j) 403 + writes[j] = &indigoatproto.RepoApplyWrites_Input_Writes_Elem{ 404 + RepoApplyWrites_Create: &indigoatproto.RepoApplyWrites_Create{ 405 + Collection: atproto.CrewCollection, 406 + Rkey: &rkey, 407 + Value: &lexutil.LexiconTypeDecoder{Val: newCrewRecord(fmt.Sprintf("did:plc:b%d_%d", i, j))}, 408 + }, 409 + } 410 + } 411 + if err := op.BatchWrite(ctx, uid, writes); err != nil { 412 + b.Fatalf("BatchWrite: %v", err) 413 + } 414 + } 415 + }) 416 + } 417 + } 418 + }
+1413
pkg/hold/pds/repo_operator_test.go
··· 1 + package pds 2 + 3 + import ( 4 + "bytes" 5 + "context" 6 + "fmt" 7 + "os" 8 + "path/filepath" 9 + "strings" 10 + "sync" 11 + "testing" 12 + 13 + "atcr.io/pkg/atproto" 14 + "atcr.io/pkg/auth/oauth" 15 + holddb "atcr.io/pkg/hold/db" 16 + indigoatproto "github.com/bluesky-social/indigo/api/atproto" 17 + lexutil "github.com/bluesky-social/indigo/lex/util" 18 + "github.com/bluesky-social/indigo/models" 19 + "github.com/bluesky-social/indigo/repo" 20 + "github.com/ipfs/go-cid" 21 + ) 22 + 23 + // setupTestRepoOperator creates a fresh RepoOperator (backed by RepoManager) 24 + // and returns it along with the user ID. Each call gets an isolated instance. 25 + func setupTestRepoOperator(t *testing.T) (RepoOperator, models.Uid) { 26 + t.Helper() 27 + ctx := context.Background() 28 + tmpDir := t.TempDir() 29 + 30 + dbPath := ":memory:" 31 + keyPath := filepath.Join(tmpDir, "signing-key") 32 + 33 + if err := os.WriteFile(keyPath, sharedTestKey, 0600); err != nil { 34 + t.Fatalf("Failed to write signing key: %v", err) 35 + } 36 + 37 + pds, err := NewHoldPDS(ctx, "did:web:hold.test", "https://hold.test", "https://atcr.io", dbPath, keyPath, false) 38 + if err != nil { 39 + t.Fatalf("NewHoldPDS: %v", err) 40 + } 41 + 42 + if err := pds.repomgr.InitNewActor(ctx, pds.uid, "", pds.did, "", "", ""); err != nil { 43 + t.Fatalf("InitNewActor: %v", err) 44 + } 45 + 46 + t.Cleanup(func() { pds.Close() }) 47 + return pds.repomgr, pds.uid 48 + } 49 + 50 + // setupTestDirectRepoOperator creates a fresh DirectRepoOperator 51 + // and returns it along with the user ID. 52 + func setupTestDirectRepoOperator(t *testing.T) (RepoOperator, models.Uid) { 53 + t.Helper() 54 + ctx := context.Background() 55 + 56 + keyPath := filepath.Join(t.TempDir(), "signing-key") 57 + if err := os.WriteFile(keyPath, sharedTestKey, 0600); err != nil { 58 + t.Fatalf("Failed to write signing key: %v", err) 59 + } 60 + signingKey, err := oauth.GenerateOrLoadPDSKey(keyPath) 61 + if err != nil { 62 + t.Fatalf("GenerateOrLoadPDSKey: %v", err) 63 + } 64 + 65 + sqlStore := new(holddb.SQLiteStore) 66 + if err := sqlStore.Open(":memory:"); err != nil { 67 + t.Fatalf("SQLiteStore.Open: %v", err) 68 + } 69 + t.Cleanup(func() { sqlStore.Close() }) 70 + 71 + kmgr := NewHoldKeyManager(signingKey) 72 + op := NewDirectRepoOperator(sqlStore, kmgr) 73 + 74 + uid := models.Uid(1) 75 + if err := op.InitNewActor(ctx, uid, "", "did:web:hold.test", "", "", ""); err != nil { 76 + t.Fatalf("InitNewActor: %v", err) 77 + } 78 + 79 + return op, uid 80 + } 81 + 82 + // newCrewRecord creates a test crew record with the given member DID. 83 + func newCrewRecord(member string) *atproto.CrewRecord { 84 + return &atproto.CrewRecord{ 85 + Type: atproto.CrewCollection, 86 + Member: member, 87 + Role: "writer", 88 + Permissions: []string{"blob:read", "blob:write"}, 89 + AddedAt: "2026-01-01T00:00:00Z", 90 + } 91 + } 92 + 93 + // runRepoOperatorTests runs the full RepoOperator test suite against any implementation. 94 + // An optional freshSetup function returns an operator WITHOUT InitNewActor pre-called, 95 + // for testing InitNewActor event emission. 96 + func runRepoOperatorTests(t *testing.T, setup func(t *testing.T) (RepoOperator, models.Uid), freshSetup ...func(t *testing.T) (RepoOperator, models.Uid, string)) { 97 + t.Run("CreateRecord", func(t *testing.T) { 98 + op, uid := setup(t) 99 + ctx := context.Background() 100 + 101 + rec := newCrewRecord("did:plc:alice") 102 + path, cc, err := op.CreateRecord(ctx, uid, atproto.CrewCollection, rec) 103 + if err != nil { 104 + t.Fatalf("CreateRecord: %v", err) 105 + } 106 + 107 + if !strings.HasPrefix(path, atproto.CrewCollection+"/") { 108 + t.Errorf("path should start with collection, got %q", path) 109 + } 110 + if !cc.Defined() { 111 + t.Error("expected defined CID") 112 + } 113 + 114 + // Extract rkey and verify it looks like a TID (13 chars, base32-sortable) 115 + rkey := strings.TrimPrefix(path, atproto.CrewCollection+"/") 116 + if len(rkey) != 13 { 117 + t.Errorf("expected 13-char TID rkey, got %q (len=%d)", rkey, len(rkey)) 118 + } 119 + 120 + // Round-trip via GetRecord 121 + gotCid, _, err := op.GetRecord(ctx, uid, atproto.CrewCollection, rkey, cid.Undef) 122 + if err != nil { 123 + t.Fatalf("GetRecord: %v", err) 124 + } 125 + if !gotCid.Equals(cc) { 126 + t.Errorf("CID mismatch: create=%s get=%s", cc, gotCid) 127 + } 128 + }) 129 + 130 + t.Run("UpdateRecord", func(t *testing.T) { 131 + op, uid := setup(t) 132 + ctx := context.Background() 133 + 134 + rec := newCrewRecord("did:plc:bob") 135 + path, createCid, err := op.CreateRecord(ctx, uid, atproto.CrewCollection, rec) 136 + if err != nil { 137 + t.Fatalf("CreateRecord: %v", err) 138 + } 139 + rkey := strings.TrimPrefix(path, atproto.CrewCollection+"/") 140 + 141 + // Update with different data 142 + updated := newCrewRecord("did:plc:bob") 143 + updated.Role = "admin" 144 + updateCid, err := op.UpdateRecord(ctx, uid, atproto.CrewCollection, rkey, updated) 145 + if err != nil { 146 + t.Fatalf("UpdateRecord: %v", err) 147 + } 148 + 149 + if createCid.Equals(updateCid) { 150 + t.Error("expected CID to change after update") 151 + } 152 + 153 + // Verify new data via GetRecord 154 + _, val, err := op.GetRecord(ctx, uid, atproto.CrewCollection, rkey, cid.Undef) 155 + if err != nil { 156 + t.Fatalf("GetRecord: %v", err) 157 + } 158 + crew, ok := val.(*atproto.CrewRecord) 159 + if !ok { 160 + t.Fatalf("expected *CrewRecord, got %T", val) 161 + } 162 + if crew.Role != "admin" { 163 + t.Errorf("expected role=admin, got %q", crew.Role) 164 + } 165 + }) 166 + 167 + t.Run("PutRecord", func(t *testing.T) { 168 + op, uid := setup(t) 169 + ctx := context.Background() 170 + 171 + rec := newCrewRecord("did:plc:charlie") 172 + path, cc, err := op.PutRecord(ctx, uid, atproto.CrewCollection, "mykey", rec) 173 + if err != nil { 174 + t.Fatalf("PutRecord: %v", err) 175 + } 176 + 177 + if !strings.HasSuffix(path, "/mykey") { 178 + t.Errorf("expected path ending in /mykey, got %q", path) 179 + } 180 + if !cc.Defined() { 181 + t.Error("expected defined CID") 182 + } 183 + 184 + // Round-trip 185 + gotCid, _, err := op.GetRecord(ctx, uid, atproto.CrewCollection, "mykey", cid.Undef) 186 + if err != nil { 187 + t.Fatalf("GetRecord: %v", err) 188 + } 189 + if !gotCid.Equals(cc) { 190 + t.Errorf("CID mismatch") 191 + } 192 + }) 193 + 194 + t.Run("PutRecord_DuplicateRkey", func(t *testing.T) { 195 + op, uid := setup(t) 196 + ctx := context.Background() 197 + 198 + rec := newCrewRecord("did:plc:dave") 199 + _, _, err := op.PutRecord(ctx, uid, atproto.CrewCollection, "dupekey", rec) 200 + if err != nil { 201 + t.Fatalf("first PutRecord: %v", err) 202 + } 203 + 204 + rec2 := newCrewRecord("did:plc:dave2") 205 + _, _, err = op.PutRecord(ctx, uid, atproto.CrewCollection, "dupekey", rec2) 206 + if err == nil { 207 + t.Error("expected error on duplicate rkey PutRecord") 208 + } 209 + }) 210 + 211 + t.Run("UpsertRecord_Create", func(t *testing.T) { 212 + op, uid := setup(t) 213 + ctx := context.Background() 214 + 215 + rec := newCrewRecord("did:plc:eve") 216 + path, cc, created, err := op.UpsertRecord(ctx, uid, atproto.CrewCollection, "upsert1", rec) 217 + if err != nil { 218 + t.Fatalf("UpsertRecord: %v", err) 219 + } 220 + 221 + if !created { 222 + t.Error("expected created=true for new record") 223 + } 224 + if !strings.HasSuffix(path, "/upsert1") { 225 + t.Errorf("expected path ending in /upsert1, got %q", path) 226 + } 227 + if !cc.Defined() { 228 + t.Error("expected defined CID") 229 + } 230 + 231 + // Verify retrievable 232 + _, _, err = op.GetRecord(ctx, uid, atproto.CrewCollection, "upsert1", cid.Undef) 233 + if err != nil { 234 + t.Fatalf("GetRecord after upsert-create: %v", err) 235 + } 236 + }) 237 + 238 + t.Run("UpsertRecord_Update", func(t *testing.T) { 239 + op, uid := setup(t) 240 + ctx := context.Background() 241 + 242 + rec := newCrewRecord("did:plc:frank") 243 + _, cid1, _, err := op.UpsertRecord(ctx, uid, atproto.CrewCollection, "upsert2", rec) 244 + if err != nil { 245 + t.Fatalf("first UpsertRecord: %v", err) 246 + } 247 + 248 + updated := newCrewRecord("did:plc:frank") 249 + updated.Role = "admin" 250 + _, cid2, created, err := op.UpsertRecord(ctx, uid, atproto.CrewCollection, "upsert2", updated) 251 + if err != nil { 252 + t.Fatalf("second UpsertRecord: %v", err) 253 + } 254 + 255 + if created { 256 + t.Error("expected created=false for existing record") 257 + } 258 + if cid1.Equals(cid2) { 259 + t.Error("expected CID to change on upsert-update") 260 + } 261 + }) 262 + 263 + t.Run("DeleteRecord", func(t *testing.T) { 264 + op, uid := setup(t) 265 + ctx := context.Background() 266 + 267 + rec := newCrewRecord("did:plc:grace") 268 + path, _, err := op.CreateRecord(ctx, uid, atproto.CrewCollection, rec) 269 + if err != nil { 270 + t.Fatalf("CreateRecord: %v", err) 271 + } 272 + rkey := strings.TrimPrefix(path, atproto.CrewCollection+"/") 273 + 274 + if err := op.DeleteRecord(ctx, uid, atproto.CrewCollection, rkey); err != nil { 275 + t.Fatalf("DeleteRecord: %v", err) 276 + } 277 + 278 + _, _, err = op.GetRecord(ctx, uid, atproto.CrewCollection, rkey, cid.Undef) 279 + if err == nil { 280 + t.Error("expected error getting deleted record") 281 + } 282 + }) 283 + 284 + t.Run("DeleteRecord_NotFound", func(t *testing.T) { 285 + op, uid := setup(t) 286 + ctx := context.Background() 287 + 288 + err := op.DeleteRecord(ctx, uid, atproto.CrewCollection, "nonexistent") 289 + if err == nil { 290 + t.Error("expected error deleting non-existent record") 291 + } 292 + }) 293 + 294 + t.Run("BatchWrite_CreateAndDelete", func(t *testing.T) { 295 + op, uid := setup(t) 296 + ctx := context.Background() 297 + 298 + // First, create a record to delete in the batch 299 + rec := newCrewRecord("did:plc:todelete") 300 + path, _, err := op.CreateRecord(ctx, uid, atproto.CrewCollection, rec) 301 + if err != nil { 302 + t.Fatalf("CreateRecord: %v", err) 303 + } 304 + deleteRkey := strings.TrimPrefix(path, atproto.CrewCollection+"/") 305 + 306 + // Batch: create 2 + delete 1 307 + rkey1 := "batchkey1" 308 + rkey2 := "batchkey2" 309 + writes := []*indigoatproto.RepoApplyWrites_Input_Writes_Elem{ 310 + { 311 + RepoApplyWrites_Create: &indigoatproto.RepoApplyWrites_Create{ 312 + Collection: atproto.CrewCollection, 313 + Rkey: &rkey1, 314 + Value: &lexutil.LexiconTypeDecoder{Val: newCrewRecord("did:plc:batch1")}, 315 + }, 316 + }, 317 + { 318 + RepoApplyWrites_Create: &indigoatproto.RepoApplyWrites_Create{ 319 + Collection: atproto.CrewCollection, 320 + Rkey: &rkey2, 321 + Value: &lexutil.LexiconTypeDecoder{Val: newCrewRecord("did:plc:batch2")}, 322 + }, 323 + }, 324 + { 325 + RepoApplyWrites_Delete: &indigoatproto.RepoApplyWrites_Delete{ 326 + Collection: atproto.CrewCollection, 327 + Rkey: deleteRkey, 328 + }, 329 + }, 330 + } 331 + 332 + if err := op.BatchWrite(ctx, uid, writes); err != nil { 333 + t.Fatalf("BatchWrite: %v", err) 334 + } 335 + 336 + // Verify created records exist 337 + _, _, err = op.GetRecord(ctx, uid, atproto.CrewCollection, rkey1, cid.Undef) 338 + if err != nil { 339 + t.Errorf("batch-created record 1 not found: %v", err) 340 + } 341 + _, _, err = op.GetRecord(ctx, uid, atproto.CrewCollection, rkey2, cid.Undef) 342 + if err != nil { 343 + t.Errorf("batch-created record 2 not found: %v", err) 344 + } 345 + 346 + // Verify deleted record is gone 347 + _, _, err = op.GetRecord(ctx, uid, atproto.CrewCollection, deleteRkey, cid.Undef) 348 + if err == nil { 349 + t.Error("expected batch-deleted record to be gone") 350 + } 351 + }) 352 + 353 + t.Run("BulkUpsert", func(t *testing.T) { 354 + op, uid := setup(t) 355 + ctx := context.Background() 356 + 357 + records := []BulkRecord{ 358 + {Collection: atproto.CrewCollection, Rkey: "bulk1", Data: newCrewRecord("did:plc:bulk1")}, 359 + {Collection: atproto.CrewCollection, Rkey: "bulk2", Data: newCrewRecord("did:plc:bulk2")}, 360 + } 361 + 362 + if err := op.BulkUpsert(ctx, uid, records); err != nil { 363 + t.Fatalf("BulkUpsert: %v", err) 364 + } 365 + 366 + // Verify both exist 367 + _, _, err := op.GetRecord(ctx, uid, atproto.CrewCollection, "bulk1", cid.Undef) 368 + if err != nil { 369 + t.Errorf("bulk record 1 not found: %v", err) 370 + } 371 + _, _, err = op.GetRecord(ctx, uid, atproto.CrewCollection, "bulk2", cid.Undef) 372 + if err != nil { 373 + t.Errorf("bulk record 2 not found: %v", err) 374 + } 375 + 376 + // Re-upsert with changed data 377 + updatedRec := newCrewRecord("did:plc:bulk1") 378 + updatedRec.Role = "admin" 379 + if err := op.BulkUpsert(ctx, uid, []BulkRecord{ 380 + {Collection: atproto.CrewCollection, Rkey: "bulk1", Data: updatedRec}, 381 + }); err != nil { 382 + t.Fatalf("BulkUpsert update: %v", err) 383 + } 384 + 385 + // Verify updated data 386 + _, val, err := op.GetRecord(ctx, uid, atproto.CrewCollection, "bulk1", cid.Undef) 387 + if err != nil { 388 + t.Fatalf("GetRecord after re-upsert: %v", err) 389 + } 390 + crew, ok := val.(*atproto.CrewRecord) 391 + if !ok { 392 + t.Fatalf("expected *CrewRecord, got %T", val) 393 + } 394 + if crew.Role != "admin" { 395 + t.Errorf("expected role=admin after re-upsert, got %q", crew.Role) 396 + } 397 + }) 398 + 399 + t.Run("GetRecord_CidMatch", func(t *testing.T) { 400 + op, uid := setup(t) 401 + ctx := context.Background() 402 + 403 + rec := newCrewRecord("did:plc:cidmatch") 404 + _, cc, err := op.PutRecord(ctx, uid, atproto.CrewCollection, "cidtest", rec) 405 + if err != nil { 406 + t.Fatalf("PutRecord: %v", err) 407 + } 408 + 409 + // Exact CID match succeeds 410 + _, _, err = op.GetRecord(ctx, uid, atproto.CrewCollection, "cidtest", cc) 411 + if err != nil { 412 + t.Errorf("GetRecord with matching CID should succeed: %v", err) 413 + } 414 + 415 + // Wrong CID fails 416 + wrongCid, _ := cid.Decode("bafyreigdvqptwntkto5jag4rr7oydencsj4m2t5pdgkhmdwyxlayuncm7e") 417 + _, _, err = op.GetRecord(ctx, uid, atproto.CrewCollection, "cidtest", wrongCid) 418 + if err == nil { 419 + t.Error("expected error with mismatched CID") 420 + } 421 + }) 422 + 423 + t.Run("GetRecord_NotFound", func(t *testing.T) { 424 + op, uid := setup(t) 425 + ctx := context.Background() 426 + 427 + _, _, err := op.GetRecord(ctx, uid, atproto.CrewCollection, "doesnotexist", cid.Undef) 428 + if err == nil { 429 + t.Error("expected error for non-existent record") 430 + } 431 + }) 432 + 433 + t.Run("GetRecordProof", func(t *testing.T) { 434 + op, uid := setup(t) 435 + ctx := context.Background() 436 + 437 + rec := newCrewRecord("did:plc:proof") 438 + _, _, err := op.PutRecord(ctx, uid, atproto.CrewCollection, "proofkey", rec) 439 + if err != nil { 440 + t.Fatalf("PutRecord: %v", err) 441 + } 442 + 443 + head, blocks, err := op.GetRecordProof(ctx, uid, atproto.CrewCollection, "proofkey") 444 + if err != nil { 445 + t.Fatalf("GetRecordProof: %v", err) 446 + } 447 + 448 + if !head.Defined() { 449 + t.Error("expected defined head CID") 450 + } 451 + if len(blocks) == 0 { 452 + t.Error("expected non-empty proof blocks") 453 + } 454 + }) 455 + 456 + t.Run("GetRepoRoot", func(t *testing.T) { 457 + op, uid := setup(t) 458 + ctx := context.Background() 459 + 460 + root1, err := op.GetRepoRoot(ctx, uid) 461 + if err != nil { 462 + t.Fatalf("GetRepoRoot: %v", err) 463 + } 464 + if !root1.Defined() { 465 + t.Error("expected defined root CID after InitNewActor") 466 + } 467 + 468 + // Write a record and verify root changes 469 + _, _, err = op.PutRecord(ctx, uid, atproto.CrewCollection, "roottest", newCrewRecord("did:plc:root")) 470 + if err != nil { 471 + t.Fatalf("PutRecord: %v", err) 472 + } 473 + 474 + root2, err := op.GetRepoRoot(ctx, uid) 475 + if err != nil { 476 + t.Fatalf("GetRepoRoot after write: %v", err) 477 + } 478 + if root1.Equals(root2) { 479 + t.Error("expected root to change after write") 480 + } 481 + }) 482 + 483 + t.Run("GetRepoRev", func(t *testing.T) { 484 + op, uid := setup(t) 485 + ctx := context.Background() 486 + 487 + rev1, err := op.GetRepoRev(ctx, uid) 488 + if err != nil { 489 + t.Fatalf("GetRepoRev: %v", err) 490 + } 491 + if rev1 == "" { 492 + t.Error("expected non-empty rev") 493 + } 494 + 495 + // Write a record and verify rev changes 496 + _, _, err = op.PutRecord(ctx, uid, atproto.CrewCollection, "revtest", newCrewRecord("did:plc:rev")) 497 + if err != nil { 498 + t.Fatalf("PutRecord: %v", err) 499 + } 500 + 501 + rev2, err := op.GetRepoRev(ctx, uid) 502 + if err != nil { 503 + t.Fatalf("GetRepoRev after write: %v", err) 504 + } 505 + if rev1 == rev2 { 506 + t.Error("expected rev to change after write") 507 + } 508 + }) 509 + 510 + t.Run("ReadRepo", func(t *testing.T) { 511 + op, uid := setup(t) 512 + ctx := context.Background() 513 + 514 + // Write something first 515 + _, _, err := op.PutRecord(ctx, uid, atproto.CrewCollection, "readrepo", newCrewRecord("did:plc:readrepo")) 516 + if err != nil { 517 + t.Fatalf("PutRecord: %v", err) 518 + } 519 + 520 + var buf bytes.Buffer 521 + if err := op.ReadRepo(ctx, uid, "", &buf); err != nil { 522 + t.Fatalf("ReadRepo: %v", err) 523 + } 524 + 525 + if buf.Len() == 0 { 526 + t.Error("expected non-empty CAR output") 527 + } 528 + }) 529 + 530 + t.Run("EventEmission_Create", func(t *testing.T) { 531 + op, uid := setup(t) 532 + ctx := context.Background() 533 + 534 + var events []*RepoEvent 535 + op.SetEventHandler(func(_ context.Context, evt *RepoEvent) { 536 + events = append(events, evt) 537 + }, false) 538 + 539 + rec := newCrewRecord("did:plc:evt-create") 540 + _, _, err := op.PutRecord(ctx, uid, atproto.CrewCollection, "evtcreate", rec) 541 + if err != nil { 542 + t.Fatalf("PutRecord: %v", err) 543 + } 544 + 545 + if len(events) != 1 { 546 + t.Fatalf("expected 1 event, got %d", len(events)) 547 + } 548 + 549 + evt := events[0] 550 + if evt.User != uid { 551 + t.Errorf("expected user=%d, got %d", uid, evt.User) 552 + } 553 + if !evt.NewRoot.Defined() { 554 + t.Error("expected defined NewRoot") 555 + } 556 + if evt.PrevData == nil { 557 + t.Error("expected non-nil PrevData") 558 + } 559 + if evt.Rev == "" { 560 + t.Error("expected non-empty Rev") 561 + } 562 + if len(evt.RepoSlice) == 0 { 563 + t.Error("expected non-empty RepoSlice") 564 + } 565 + if len(evt.Ops) != 1 { 566 + t.Fatalf("expected 1 op, got %d", len(evt.Ops)) 567 + } 568 + 569 + op0 := evt.Ops[0] 570 + if op0.Kind != EvtKindCreateRecord { 571 + t.Errorf("expected kind=create, got %q", op0.Kind) 572 + } 573 + if op0.Collection != atproto.CrewCollection { 574 + t.Errorf("expected collection=%s, got %q", atproto.CrewCollection, op0.Collection) 575 + } 576 + if op0.Rkey != "evtcreate" { 577 + t.Errorf("expected rkey=evtcreate, got %q", op0.Rkey) 578 + } 579 + if op0.RecCid == nil { 580 + t.Error("expected non-nil RecCid for create op") 581 + } 582 + }) 583 + 584 + t.Run("EventEmission_Update_PrevData", func(t *testing.T) { 585 + op, uid := setup(t) 586 + ctx := context.Background() 587 + 588 + // Create first (no event handler yet) 589 + rec := newCrewRecord("did:plc:evt-update") 590 + _, _, err := op.PutRecord(ctx, uid, atproto.CrewCollection, "evtupdate", rec) 591 + if err != nil { 592 + t.Fatalf("PutRecord: %v", err) 593 + } 594 + 595 + // Now set handler and update 596 + var events []*RepoEvent 597 + op.SetEventHandler(func(_ context.Context, evt *RepoEvent) { 598 + events = append(events, evt) 599 + }, false) 600 + 601 + updated := newCrewRecord("did:plc:evt-update") 602 + updated.Role = "admin" 603 + _, err = op.UpdateRecord(ctx, uid, atproto.CrewCollection, "evtupdate", updated) 604 + if err != nil { 605 + t.Fatalf("UpdateRecord: %v", err) 606 + } 607 + 608 + if len(events) != 1 { 609 + t.Fatalf("expected 1 event, got %d", len(events)) 610 + } 611 + 612 + evt := events[0] 613 + if evt.PrevData == nil { 614 + t.Error("expected non-nil PrevData on update event") 615 + } 616 + if evt.OldRoot == nil { 617 + t.Error("expected non-nil OldRoot on update event") 618 + } 619 + if evt.Since == nil { 620 + t.Error("expected non-nil Since on update event") 621 + } 622 + if len(evt.Ops) != 1 { 623 + t.Fatalf("expected 1 op, got %d", len(evt.Ops)) 624 + } 625 + if evt.Ops[0].Kind != EvtKindUpdateRecord { 626 + t.Errorf("expected kind=update, got %q", evt.Ops[0].Kind) 627 + } 628 + }) 629 + 630 + t.Run("EventEmission_Delete", func(t *testing.T) { 631 + op, uid := setup(t) 632 + ctx := context.Background() 633 + 634 + rec := newCrewRecord("did:plc:evt-delete") 635 + _, _, err := op.PutRecord(ctx, uid, atproto.CrewCollection, "evtdelete", rec) 636 + if err != nil { 637 + t.Fatalf("PutRecord: %v", err) 638 + } 639 + 640 + var events []*RepoEvent 641 + op.SetEventHandler(func(_ context.Context, evt *RepoEvent) { 642 + events = append(events, evt) 643 + }, false) 644 + 645 + if err := op.DeleteRecord(ctx, uid, atproto.CrewCollection, "evtdelete"); err != nil { 646 + t.Fatalf("DeleteRecord: %v", err) 647 + } 648 + 649 + if len(events) != 1 { 650 + t.Fatalf("expected 1 event, got %d", len(events)) 651 + } 652 + 653 + evt := events[0] 654 + if len(evt.Ops) != 1 { 655 + t.Fatalf("expected 1 op, got %d", len(evt.Ops)) 656 + } 657 + if evt.Ops[0].Kind != EvtKindDeleteRecord { 658 + t.Errorf("expected kind=delete, got %q", evt.Ops[0].Kind) 659 + } 660 + if evt.Ops[0].RecCid != nil { 661 + t.Error("expected nil RecCid for delete op") 662 + } 663 + }) 664 + 665 + t.Run("EventEmission_Hydrate", func(t *testing.T) { 666 + op, uid := setup(t) 667 + ctx := context.Background() 668 + 669 + var events []*RepoEvent 670 + op.SetEventHandler(func(_ context.Context, evt *RepoEvent) { 671 + events = append(events, evt) 672 + }, true) // hydrateRecords = true 673 + 674 + rec := newCrewRecord("did:plc:evt-hydrate") 675 + _, _, err := op.PutRecord(ctx, uid, atproto.CrewCollection, "evthydrate", rec) 676 + if err != nil { 677 + t.Fatalf("PutRecord: %v", err) 678 + } 679 + 680 + if len(events) != 1 { 681 + t.Fatalf("expected 1 event, got %d", len(events)) 682 + } 683 + if events[0].Ops[0].Record == nil { 684 + t.Error("expected non-nil Record when hydrateRecords=true") 685 + } 686 + }) 687 + 688 + // --- Error path and edge case tests --- 689 + 690 + t.Run("UpdateRecord_NotFound", func(t *testing.T) { 691 + op, uid := setup(t) 692 + ctx := context.Background() 693 + 694 + rec := newCrewRecord("did:plc:ghost") 695 + _, err := op.UpdateRecord(ctx, uid, atproto.CrewCollection, "nonexistent", rec) 696 + if err == nil { 697 + t.Error("expected error updating non-existent record") 698 + } 699 + }) 700 + 701 + t.Run("UpdateRecord_Hydrate", func(t *testing.T) { 702 + op, uid := setup(t) 703 + ctx := context.Background() 704 + 705 + rec := newCrewRecord("did:plc:hydrate-upd") 706 + _, _, err := op.PutRecord(ctx, uid, atproto.CrewCollection, "hydupd", rec) 707 + if err != nil { 708 + t.Fatalf("PutRecord: %v", err) 709 + } 710 + 711 + var events []*RepoEvent 712 + op.SetEventHandler(func(_ context.Context, evt *RepoEvent) { 713 + events = append(events, evt) 714 + }, true) 715 + 716 + updated := newCrewRecord("did:plc:hydrate-upd") 717 + updated.Role = "admin" 718 + _, err = op.UpdateRecord(ctx, uid, atproto.CrewCollection, "hydupd", updated) 719 + if err != nil { 720 + t.Fatalf("UpdateRecord: %v", err) 721 + } 722 + 723 + if len(events) != 1 { 724 + t.Fatalf("expected 1 event, got %d", len(events)) 725 + } 726 + if events[0].Ops[0].Record == nil { 727 + t.Error("expected non-nil Record on update with hydrateRecords=true") 728 + } 729 + }) 730 + 731 + t.Run("PutRecord_Hydrate", func(t *testing.T) { 732 + op, uid := setup(t) 733 + ctx := context.Background() 734 + 735 + var events []*RepoEvent 736 + op.SetEventHandler(func(_ context.Context, evt *RepoEvent) { 737 + events = append(events, evt) 738 + }, true) 739 + 740 + rec := newCrewRecord("did:plc:hydrate-put") 741 + _, _, err := op.PutRecord(ctx, uid, atproto.CrewCollection, "hydput", rec) 742 + if err != nil { 743 + t.Fatalf("PutRecord: %v", err) 744 + } 745 + 746 + if len(events) != 1 { 747 + t.Fatalf("expected 1 event, got %d", len(events)) 748 + } 749 + if events[0].Ops[0].Record == nil { 750 + t.Error("expected non-nil Record on put with hydrateRecords=true") 751 + } 752 + }) 753 + 754 + t.Run("InitNewActor_EmptyDID", func(t *testing.T) { 755 + op, uid := setup(t) 756 + ctx := context.Background() 757 + 758 + err := op.InitNewActor(ctx, uid, "", "", "", "", "") 759 + if err == nil { 760 + t.Error("expected error for empty DID") 761 + } 762 + }) 763 + 764 + t.Run("InitNewActor_ZeroUser", func(t *testing.T) { 765 + op, _ := setup(t) 766 + ctx := context.Background() 767 + 768 + err := op.InitNewActor(ctx, 0, "", "did:web:test", "", "", "") 769 + if err == nil { 770 + t.Error("expected error for zero user") 771 + } 772 + }) 773 + 774 + t.Run("InitNewActor_EventEmission", func(t *testing.T) { 775 + if len(freshSetup) == 0 || freshSetup[0] == nil { 776 + t.Skip("no freshSetup provided") 777 + } 778 + 779 + op, uid, did := freshSetup[0](t) 780 + ctx := context.Background() 781 + 782 + var events []*RepoEvent 783 + op.SetEventHandler(func(_ context.Context, evt *RepoEvent) { 784 + events = append(events, evt) 785 + }, true) 786 + 787 + err := op.InitNewActor(ctx, uid, "", did, "Test User", "", "") 788 + if err != nil { 789 + t.Fatalf("InitNewActor: %v", err) 790 + } 791 + 792 + if len(events) != 1 { 793 + t.Fatalf("expected 1 event from InitNewActor, got %d", len(events)) 794 + } 795 + evt := events[0] 796 + if len(evt.Ops) != 1 { 797 + t.Fatalf("expected 1 op, got %d", len(evt.Ops)) 798 + } 799 + if evt.Ops[0].Kind != EvtKindCreateRecord { 800 + t.Errorf("expected kind=create, got %q", evt.Ops[0].Kind) 801 + } 802 + if evt.Ops[0].Collection != "app.bsky.actor.profile" { 803 + t.Errorf("expected collection=app.bsky.actor.profile, got %q", evt.Ops[0].Collection) 804 + } 805 + if evt.Ops[0].Record == nil { 806 + t.Error("expected non-nil Record with hydrateRecords=true") 807 + } 808 + }) 809 + 810 + t.Run("GetRecordProof_NotFound", func(t *testing.T) { 811 + op, uid := setup(t) 812 + ctx := context.Background() 813 + 814 + _, _, err := op.GetRecordProof(ctx, uid, atproto.CrewCollection, "nonexistent") 815 + if err == nil { 816 + t.Error("expected error for proof of non-existent record") 817 + } 818 + }) 819 + 820 + t.Run("GetRecordProof_NoRepo", func(t *testing.T) { 821 + // Use a user ID that has no repo initialized — triggers OpenRepo error 822 + op, _ := setup(t) 823 + ctx := context.Background() 824 + 825 + _, _, err := op.GetRecordProof(ctx, models.Uid(9999), atproto.CrewCollection, "anything") 826 + if err == nil { 827 + t.Error("expected error for user with no repo") 828 + } 829 + }) 830 + 831 + t.Run("BatchWrite_Update", func(t *testing.T) { 832 + op, uid := setup(t) 833 + ctx := context.Background() 834 + 835 + // NOTE: BatchWrite internally uses r.PutRecord for updates (mst.Add), 836 + // which means the update write type is effectively a create-or-fail. 837 + // This test uses a fresh rkey to exercise the update code path. 838 + updated := newCrewRecord("did:plc:batchupd") 839 + updated.Role = "admin" 840 + writes := []*indigoatproto.RepoApplyWrites_Input_Writes_Elem{ 841 + { 842 + RepoApplyWrites_Update: &indigoatproto.RepoApplyWrites_Update{ 843 + Collection: atproto.CrewCollection, 844 + Rkey: "batchupd", 845 + Value: &lexutil.LexiconTypeDecoder{Val: updated}, 846 + }, 847 + }, 848 + } 849 + 850 + if err := op.BatchWrite(ctx, uid, writes); err != nil { 851 + t.Fatalf("BatchWrite update: %v", err) 852 + } 853 + 854 + // Verify data was written 855 + _, val, err := op.GetRecord(ctx, uid, atproto.CrewCollection, "batchupd", cid.Undef) 856 + if err != nil { 857 + t.Fatalf("GetRecord after batch update: %v", err) 858 + } 859 + crew, ok := val.(*atproto.CrewRecord) 860 + if !ok { 861 + t.Fatalf("expected *CrewRecord, got %T", val) 862 + } 863 + if crew.Role != "admin" { 864 + t.Errorf("expected role=admin, got %q", crew.Role) 865 + } 866 + }) 867 + 868 + t.Run("BatchWrite_AutoRkey", func(t *testing.T) { 869 + op, uid := setup(t) 870 + ctx := context.Background() 871 + 872 + var events []*RepoEvent 873 + op.SetEventHandler(func(_ context.Context, evt *RepoEvent) { 874 + events = append(events, evt) 875 + }, false) 876 + 877 + // Create with nil Rkey — should auto-generate TID 878 + writes := []*indigoatproto.RepoApplyWrites_Input_Writes_Elem{ 879 + { 880 + RepoApplyWrites_Create: &indigoatproto.RepoApplyWrites_Create{ 881 + Collection: atproto.CrewCollection, 882 + Rkey: nil, // auto-generate 883 + Value: &lexutil.LexiconTypeDecoder{Val: newCrewRecord("did:plc:autorkey")}, 884 + }, 885 + }, 886 + } 887 + 888 + if err := op.BatchWrite(ctx, uid, writes); err != nil { 889 + t.Fatalf("BatchWrite: %v", err) 890 + } 891 + 892 + if len(events) != 1 { 893 + t.Fatalf("expected 1 event, got %d", len(events)) 894 + } 895 + rkey := events[0].Ops[0].Rkey 896 + if len(rkey) != 13 { 897 + t.Errorf("expected 13-char auto-generated TID rkey, got %q (len=%d)", rkey, len(rkey)) 898 + } 899 + 900 + // Verify the record exists 901 + _, _, err := op.GetRecord(ctx, uid, atproto.CrewCollection, rkey, cid.Undef) 902 + if err != nil { 903 + t.Errorf("auto-rkey record not found: %v", err) 904 + } 905 + }) 906 + 907 + t.Run("BatchWrite_DeleteNotFound", func(t *testing.T) { 908 + op, uid := setup(t) 909 + ctx := context.Background() 910 + 911 + writes := []*indigoatproto.RepoApplyWrites_Input_Writes_Elem{ 912 + { 913 + RepoApplyWrites_Delete: &indigoatproto.RepoApplyWrites_Delete{ 914 + Collection: atproto.CrewCollection, 915 + Rkey: "nonexistent", 916 + }, 917 + }, 918 + } 919 + 920 + err := op.BatchWrite(ctx, uid, writes) 921 + if err == nil { 922 + t.Error("expected error deleting non-existent record in batch") 923 + } 924 + }) 925 + 926 + t.Run("BatchWrite_EmptyWriteElem", func(t *testing.T) { 927 + op, uid := setup(t) 928 + ctx := context.Background() 929 + 930 + // Write elem with no operation set 931 + writes := []*indigoatproto.RepoApplyWrites_Input_Writes_Elem{ 932 + {}, // all nil 933 + } 934 + 935 + err := op.BatchWrite(ctx, uid, writes) 936 + if err == nil { 937 + t.Error("expected error for empty write elem") 938 + } 939 + }) 940 + 941 + t.Run("BatchWrite_EventEmission", func(t *testing.T) { 942 + op, uid := setup(t) 943 + ctx := context.Background() 944 + 945 + // Create a record to delete 946 + _, _, err := op.PutRecord(ctx, uid, atproto.CrewCollection, "batchevtdel", newCrewRecord("did:plc:batchevtdel")) 947 + if err != nil { 948 + t.Fatalf("PutRecord: %v", err) 949 + } 950 + 951 + var events []*RepoEvent 952 + op.SetEventHandler(func(_ context.Context, evt *RepoEvent) { 953 + events = append(events, evt) 954 + }, true) 955 + 956 + rkey := "batchevt1" 957 + writes := []*indigoatproto.RepoApplyWrites_Input_Writes_Elem{ 958 + { 959 + RepoApplyWrites_Create: &indigoatproto.RepoApplyWrites_Create{ 960 + Collection: atproto.CrewCollection, 961 + Rkey: &rkey, 962 + Value: &lexutil.LexiconTypeDecoder{Val: newCrewRecord("did:plc:batchevt1")}, 963 + }, 964 + }, 965 + { 966 + RepoApplyWrites_Delete: &indigoatproto.RepoApplyWrites_Delete{ 967 + Collection: atproto.CrewCollection, 968 + Rkey: "batchevtdel", 969 + }, 970 + }, 971 + } 972 + 973 + if err := op.BatchWrite(ctx, uid, writes); err != nil { 974 + t.Fatalf("BatchWrite: %v", err) 975 + } 976 + 977 + if len(events) != 1 { 978 + t.Fatalf("expected 1 event, got %d", len(events)) 979 + } 980 + 981 + evt := events[0] 982 + if len(evt.Ops) != 2 { 983 + t.Fatalf("expected 2 ops in batch event, got %d", len(evt.Ops)) 984 + } 985 + if evt.PrevData == nil { 986 + t.Error("expected non-nil PrevData") 987 + } 988 + if evt.OldRoot == nil { 989 + t.Error("expected non-nil OldRoot") 990 + } 991 + 992 + // First op: create with hydrated record 993 + if evt.Ops[0].Kind != EvtKindCreateRecord { 994 + t.Errorf("op[0] expected kind=create, got %q", evt.Ops[0].Kind) 995 + } 996 + if evt.Ops[0].Record == nil { 997 + t.Error("op[0] expected non-nil Record with hydrateRecords=true") 998 + } 999 + 1000 + // Second op: delete 1001 + if evt.Ops[1].Kind != EvtKindDeleteRecord { 1002 + t.Errorf("op[1] expected kind=delete, got %q", evt.Ops[1].Kind) 1003 + } 1004 + }) 1005 + 1006 + t.Run("BatchWrite_UpdateHydrate", func(t *testing.T) { 1007 + op, uid := setup(t) 1008 + ctx := context.Background() 1009 + 1010 + // NOTE: BatchWrite uses r.PutRecord for updates (mst.Add), so use a fresh rkey. 1011 + var events []*RepoEvent 1012 + op.SetEventHandler(func(_ context.Context, evt *RepoEvent) { 1013 + events = append(events, evt) 1014 + }, true) 1015 + 1016 + updated := newCrewRecord("did:plc:batchhydupd") 1017 + updated.Role = "admin" 1018 + writes := []*indigoatproto.RepoApplyWrites_Input_Writes_Elem{ 1019 + { 1020 + RepoApplyWrites_Update: &indigoatproto.RepoApplyWrites_Update{ 1021 + Collection: atproto.CrewCollection, 1022 + Rkey: "batchhydupd", 1023 + Value: &lexutil.LexiconTypeDecoder{Val: updated}, 1024 + }, 1025 + }, 1026 + } 1027 + 1028 + if err := op.BatchWrite(ctx, uid, writes); err != nil { 1029 + t.Fatalf("BatchWrite: %v", err) 1030 + } 1031 + 1032 + if len(events) != 1 { 1033 + t.Fatalf("expected 1 event, got %d", len(events)) 1034 + } 1035 + if events[0].Ops[0].Kind != EvtKindUpdateRecord { 1036 + t.Errorf("expected kind=update, got %q", events[0].Ops[0].Kind) 1037 + } 1038 + if events[0].Ops[0].Record == nil { 1039 + t.Error("expected non-nil Record on batch update with hydrateRecords=true") 1040 + } 1041 + }) 1042 + 1043 + t.Run("BulkUpsert_EventEmission", func(t *testing.T) { 1044 + op, uid := setup(t) 1045 + ctx := context.Background() 1046 + 1047 + var events []*RepoEvent 1048 + op.SetEventHandler(func(_ context.Context, evt *RepoEvent) { 1049 + events = append(events, evt) 1050 + }, true) 1051 + 1052 + records := []BulkRecord{ 1053 + {Collection: atproto.CrewCollection, Rkey: "bulkevt1", Data: newCrewRecord("did:plc:bulkevt1")}, 1054 + {Collection: atproto.CrewCollection, Rkey: "bulkevt2", Data: newCrewRecord("did:plc:bulkevt2")}, 1055 + } 1056 + 1057 + if err := op.BulkUpsert(ctx, uid, records); err != nil { 1058 + t.Fatalf("BulkUpsert: %v", err) 1059 + } 1060 + 1061 + if len(events) != 1 { 1062 + t.Fatalf("expected 1 event, got %d", len(events)) 1063 + } 1064 + if len(events[0].Ops) != 2 { 1065 + t.Fatalf("expected 2 ops, got %d", len(events[0].Ops)) 1066 + } 1067 + // Both should be creates since records are new 1068 + for i, eop := range events[0].Ops { 1069 + if eop.Kind != EvtKindCreateRecord { 1070 + t.Errorf("op[%d] expected kind=create, got %q", i, eop.Kind) 1071 + } 1072 + } 1073 + }) 1074 + 1075 + t.Run("ReadRepo_WithSince", func(t *testing.T) { 1076 + op, uid := setup(t) 1077 + ctx := context.Background() 1078 + 1079 + // Get initial rev 1080 + rev1, err := op.GetRepoRev(ctx, uid) 1081 + if err != nil { 1082 + t.Fatalf("GetRepoRev: %v", err) 1083 + } 1084 + 1085 + // Write a record 1086 + _, _, err = op.PutRecord(ctx, uid, atproto.CrewCollection, "since1", newCrewRecord("did:plc:since1")) 1087 + if err != nil { 1088 + t.Fatalf("PutRecord: %v", err) 1089 + } 1090 + 1091 + // ReadRepo with since should produce smaller output than full export 1092 + var fullBuf bytes.Buffer 1093 + if err := op.ReadRepo(ctx, uid, "", &fullBuf); err != nil { 1094 + t.Fatalf("ReadRepo full: %v", err) 1095 + } 1096 + 1097 + var sinceBuf bytes.Buffer 1098 + if err := op.ReadRepo(ctx, uid, rev1, &sinceBuf); err != nil { 1099 + t.Fatalf("ReadRepo since: %v", err) 1100 + } 1101 + 1102 + if sinceBuf.Len() == 0 { 1103 + t.Error("expected non-empty incremental CAR") 1104 + } 1105 + if sinceBuf.Len() >= fullBuf.Len() { 1106 + t.Errorf("expected incremental CAR (%d) < full CAR (%d)", sinceBuf.Len(), fullBuf.Len()) 1107 + } 1108 + }) 1109 + 1110 + t.Run("CreateRecord_NoEventWithoutHandler", func(t *testing.T) { 1111 + op, uid := setup(t) 1112 + ctx := context.Background() 1113 + 1114 + // Don't set event handler — should not panic 1115 + rec := newCrewRecord("did:plc:nohandler") 1116 + _, _, err := op.CreateRecord(ctx, uid, atproto.CrewCollection, rec) 1117 + if err != nil { 1118 + t.Fatalf("CreateRecord without handler: %v", err) 1119 + } 1120 + }) 1121 + 1122 + t.Run("CreateRecord_AlwaysHydrates", func(t *testing.T) { 1123 + op, uid := setup(t) 1124 + ctx := context.Background() 1125 + 1126 + var events []*RepoEvent 1127 + op.SetEventHandler(func(_ context.Context, evt *RepoEvent) { 1128 + events = append(events, evt) 1129 + }, false) // hydration OFF 1130 + 1131 + rec := newCrewRecord("did:plc:always-hydrate") 1132 + _, _, err := op.CreateRecord(ctx, uid, atproto.CrewCollection, rec) 1133 + if err != nil { 1134 + t.Fatalf("CreateRecord: %v", err) 1135 + } 1136 + 1137 + if len(events) != 1 { 1138 + t.Fatalf("expected 1 event, got %d", len(events)) 1139 + } 1140 + // CreateRecord always includes Record regardless of hydrateRecords 1141 + if events[0].Ops[0].Record == nil { 1142 + t.Error("expected non-nil Record on CreateRecord even with hydrateRecords=false") 1143 + } 1144 + }) 1145 + 1146 + t.Run("PutRecord_NoHydrate", func(t *testing.T) { 1147 + op, uid := setup(t) 1148 + ctx := context.Background() 1149 + 1150 + var events []*RepoEvent 1151 + op.SetEventHandler(func(_ context.Context, evt *RepoEvent) { 1152 + events = append(events, evt) 1153 + }, false) // hydration OFF 1154 + 1155 + rec := newCrewRecord("did:plc:put-nohydrate") 1156 + _, _, err := op.PutRecord(ctx, uid, atproto.CrewCollection, "putnohydrate", rec) 1157 + if err != nil { 1158 + t.Fatalf("PutRecord: %v", err) 1159 + } 1160 + 1161 + if len(events) != 1 { 1162 + t.Fatalf("expected 1 event, got %d", len(events)) 1163 + } 1164 + if events[0].Ops[0].Record != nil { 1165 + t.Error("expected nil Record on PutRecord with hydrateRecords=false") 1166 + } 1167 + }) 1168 + 1169 + t.Run("UpdateRecord_NoHydrate", func(t *testing.T) { 1170 + op, uid := setup(t) 1171 + ctx := context.Background() 1172 + 1173 + // Create record first (no handler) 1174 + rec := newCrewRecord("did:plc:upd-nohydrate") 1175 + _, _, err := op.PutRecord(ctx, uid, atproto.CrewCollection, "updnohydrate", rec) 1176 + if err != nil { 1177 + t.Fatalf("PutRecord: %v", err) 1178 + } 1179 + 1180 + var events []*RepoEvent 1181 + op.SetEventHandler(func(_ context.Context, evt *RepoEvent) { 1182 + events = append(events, evt) 1183 + }, false) // hydration OFF 1184 + 1185 + updated := newCrewRecord("did:plc:upd-nohydrate") 1186 + updated.Role = "admin" 1187 + _, err = op.UpdateRecord(ctx, uid, atproto.CrewCollection, "updnohydrate", updated) 1188 + if err != nil { 1189 + t.Fatalf("UpdateRecord: %v", err) 1190 + } 1191 + 1192 + if len(events) != 1 { 1193 + t.Fatalf("expected 1 event, got %d", len(events)) 1194 + } 1195 + if events[0].Ops[0].Record != nil { 1196 + t.Error("expected nil Record on UpdateRecord with hydrateRecords=false") 1197 + } 1198 + }) 1199 + 1200 + t.Run("BulkUpsert_NeverHydrates", func(t *testing.T) { 1201 + op, uid := setup(t) 1202 + ctx := context.Background() 1203 + 1204 + var events []*RepoEvent 1205 + op.SetEventHandler(func(_ context.Context, evt *RepoEvent) { 1206 + events = append(events, evt) 1207 + }, true) // hydration ON — but BulkUpsert should still not hydrate 1208 + 1209 + records := []BulkRecord{ 1210 + {Collection: atproto.CrewCollection, Rkey: "bulknohydrate", Data: newCrewRecord("did:plc:bulk-nohydrate")}, 1211 + } 1212 + if err := op.BulkUpsert(ctx, uid, records); err != nil { 1213 + t.Fatalf("BulkUpsert: %v", err) 1214 + } 1215 + 1216 + if len(events) != 1 { 1217 + t.Fatalf("expected 1 event, got %d", len(events)) 1218 + } 1219 + if events[0].Ops[0].Record != nil { 1220 + t.Error("expected nil Record on BulkUpsert even with hydrateRecords=true") 1221 + } 1222 + }) 1223 + 1224 + t.Run("RevChain_Sequential", func(t *testing.T) { 1225 + op, uid := setup(t) 1226 + ctx := context.Background() 1227 + 1228 + var events []*RepoEvent 1229 + op.SetEventHandler(func(_ context.Context, evt *RepoEvent) { 1230 + events = append(events, evt) 1231 + }, false) 1232 + 1233 + // Get initial rev 1234 + rev0, err := op.GetRepoRev(ctx, uid) 1235 + if err != nil { 1236 + t.Fatalf("GetRepoRev: %v", err) 1237 + } 1238 + 1239 + // Write 3 records sequentially 1240 + for i := 0; i < 3; i++ { 1241 + _, _, err := op.PutRecord(ctx, uid, atproto.CrewCollection, fmt.Sprintf("chain%d", i), newCrewRecord(fmt.Sprintf("did:plc:chain%d", i))) 1242 + if err != nil { 1243 + t.Fatalf("PutRecord %d: %v", i, err) 1244 + } 1245 + } 1246 + 1247 + if len(events) != 3 { 1248 + t.Fatalf("expected 3 events, got %d", len(events)) 1249 + } 1250 + 1251 + // Revs increase monotonically (TIDs are lexicographically sortable) 1252 + revs := []string{rev0, events[0].Rev, events[1].Rev, events[2].Rev} 1253 + for i := 1; i < len(revs); i++ { 1254 + if revs[i] <= revs[i-1] { 1255 + t.Errorf("rev[%d]=%q should be > rev[%d]=%q", i, revs[i], i-1, revs[i-1]) 1256 + } 1257 + } 1258 + 1259 + // Each event's Since equals the previous rev 1260 + for i, evt := range events { 1261 + expectedSince := revs[i] // rev before this write 1262 + if evt.Since == nil { 1263 + t.Errorf("event[%d] Since is nil", i) 1264 + } else if *evt.Since != expectedSince { 1265 + t.Errorf("event[%d] Since=%q, expected %q", i, *evt.Since, expectedSince) 1266 + } 1267 + } 1268 + 1269 + // Each event's OldRoot equals the previous NewRoot 1270 + // First event's OldRoot should be defined (repo initialized by setup) 1271 + for i := 1; i < len(events); i++ { 1272 + if events[i].OldRoot == nil { 1273 + t.Errorf("event[%d] OldRoot is nil", i) 1274 + } else if !events[i].OldRoot.Equals(events[i-1].NewRoot) { 1275 + t.Errorf("event[%d] OldRoot=%s != event[%d] NewRoot=%s", i, *events[i].OldRoot, i-1, events[i-1].NewRoot) 1276 + } 1277 + } 1278 + 1279 + // Roots change each time 1280 + roots := []cid.Cid{events[0].NewRoot, events[1].NewRoot, events[2].NewRoot} 1281 + for i := 1; i < len(roots); i++ { 1282 + if roots[i].Equals(roots[i-1]) { 1283 + t.Errorf("root[%d] should differ from root[%d]", i, i-1) 1284 + } 1285 + } 1286 + }) 1287 + 1288 + t.Run("ConcurrentWrites", func(t *testing.T) { 1289 + op, uid := setup(t) 1290 + ctx := context.Background() 1291 + 1292 + const n = 10 1293 + var wg sync.WaitGroup 1294 + errs := make([]error, n) 1295 + 1296 + for i := 0; i < n; i++ { 1297 + wg.Add(1) 1298 + go func(i int) { 1299 + defer wg.Done() 1300 + rkey := fmt.Sprintf("concurrent%d", i) 1301 + rec := newCrewRecord(fmt.Sprintf("did:plc:concurrent%d", i)) 1302 + _, _, errs[i] = op.PutRecord(ctx, uid, atproto.CrewCollection, rkey, rec) 1303 + }(i) 1304 + } 1305 + wg.Wait() 1306 + 1307 + for i, err := range errs { 1308 + if err != nil { 1309 + t.Errorf("goroutine %d: PutRecord failed: %v", i, err) 1310 + } 1311 + } 1312 + 1313 + // Verify all records exist 1314 + for i := 0; i < n; i++ { 1315 + rkey := fmt.Sprintf("concurrent%d", i) 1316 + _, _, err := op.GetRecord(ctx, uid, atproto.CrewCollection, rkey, cid.Undef) 1317 + if err != nil { 1318 + t.Errorf("record %q not found after concurrent writes: %v", rkey, err) 1319 + } 1320 + } 1321 + }) 1322 + 1323 + t.Run("ReadRepo_ContainsRecords", func(t *testing.T) { 1324 + op, uid := setup(t) 1325 + ctx := context.Background() 1326 + 1327 + // Write 2 records with known rkeys 1328 + _, _, err := op.PutRecord(ctx, uid, atproto.CrewCollection, "cartest1", newCrewRecord("did:plc:cartest1")) 1329 + if err != nil { 1330 + t.Fatalf("PutRecord 1: %v", err) 1331 + } 1332 + _, _, err = op.PutRecord(ctx, uid, atproto.CrewCollection, "cartest2", newCrewRecord("did:plc:cartest2")) 1333 + if err != nil { 1334 + t.Fatalf("PutRecord 2: %v", err) 1335 + } 1336 + 1337 + // Export CAR 1338 + var buf bytes.Buffer 1339 + if err := op.ReadRepo(ctx, uid, "", &buf); err != nil { 1340 + t.Fatalf("ReadRepo: %v", err) 1341 + } 1342 + 1343 + // Parse it back 1344 + r, err := repo.ReadRepoFromCar(ctx, &buf) 1345 + if err != nil { 1346 + t.Fatalf("ReadRepoFromCar: %v", err) 1347 + } 1348 + 1349 + // Both records should be retrievable 1350 + for _, rkey := range []string{"cartest1", "cartest2"} { 1351 + rpath := atproto.CrewCollection + "/" + rkey 1352 + _, _, err := r.GetRecord(ctx, rpath) 1353 + if err != nil { 1354 + t.Errorf("record %q not found in CAR: %v", rpath, err) 1355 + } 1356 + } 1357 + 1358 + // Unknown record should fail 1359 + _, _, err = r.GetRecord(ctx, atproto.CrewCollection+"/doesnotexist") 1360 + if err == nil { 1361 + t.Error("expected error for non-existent record in CAR") 1362 + } 1363 + }) 1364 + } 1365 + 1366 + // setupFreshRepoManager returns a RepoManager WITHOUT InitNewActor called. 1367 + func setupFreshRepoManager(t *testing.T) (RepoOperator, models.Uid, string) { 1368 + t.Helper() 1369 + ctx := context.Background() 1370 + keyPath := filepath.Join(t.TempDir(), "signing-key") 1371 + if err := os.WriteFile(keyPath, sharedTestKey, 0600); err != nil { 1372 + t.Fatalf("Failed to write signing key: %v", err) 1373 + } 1374 + pds, err := NewHoldPDS(ctx, "did:web:hold.init-test", "https://hold.init-test", "https://atcr.io", ":memory:", keyPath, false) 1375 + if err != nil { 1376 + t.Fatalf("NewHoldPDS: %v", err) 1377 + } 1378 + t.Cleanup(func() { pds.Close() }) 1379 + return pds.repomgr, pds.uid, pds.did 1380 + } 1381 + 1382 + // setupFreshDirectRepoOperator returns a DirectRepoOperator WITHOUT InitNewActor called. 1383 + func setupFreshDirectRepoOperator(t *testing.T) (RepoOperator, models.Uid, string) { 1384 + t.Helper() 1385 + 1386 + keyPath := filepath.Join(t.TempDir(), "signing-key") 1387 + if err := os.WriteFile(keyPath, sharedTestKey, 0600); err != nil { 1388 + t.Fatalf("Failed to write signing key: %v", err) 1389 + } 1390 + signingKey, err := oauth.GenerateOrLoadPDSKey(keyPath) 1391 + if err != nil { 1392 + t.Fatalf("GenerateOrLoadPDSKey: %v", err) 1393 + } 1394 + 1395 + sqlStore := new(holddb.SQLiteStore) 1396 + if err := sqlStore.Open(":memory:"); err != nil { 1397 + t.Fatalf("SQLiteStore.Open: %v", err) 1398 + } 1399 + t.Cleanup(func() { sqlStore.Close() }) 1400 + 1401 + kmgr := NewHoldKeyManager(signingKey) 1402 + op := NewDirectRepoOperator(sqlStore, kmgr) 1403 + 1404 + return op, models.Uid(1), "did:web:hold.test" 1405 + } 1406 + 1407 + func TestRepoManager(t *testing.T) { 1408 + runRepoOperatorTests(t, setupTestRepoOperator, setupFreshRepoManager) 1409 + } 1410 + 1411 + func TestDirectRepoOperator(t *testing.T) { 1412 + runRepoOperatorTests(t, setupTestDirectRepoOperator, setupFreshDirectRepoOperator) 1413 + }
+62 -662
pkg/hold/pds/repomgr.go
··· 1 - // Package pds contains a vendored copy of RepoManager from github.com/bluesky-social/indigo 1 + package pds 2 + 3 + // repomgr.go — RepoManager manages ATProto repository operations for the hold PDS. 2 4 // 3 - // Source: github.com/bluesky-social/indigo/repomgr (v0.0.0-20251014222321) 4 - // Reference: github.com/streamplace/indigo (67ae5a5) for PutRecord implementation 5 - // Reason: The indigo library is unmaintained and contains a critical bug in UpdateRecord 5 + // Originally vendored from github.com/bluesky-social/indigo/repomgr with fixes: 6 + // - Fixed UpdateRecord bug (was calling PutRecord internally) 7 + // - Added PutRecord/UpsertRecord for explicit rkey operations 8 + // - Added prevData support for Sync 1.1 6 9 // 7 - // Modifications from original: 8 - // - Changed package from 'repomgr' to 'pds' for integration with hold service 9 - // - Fixed UpdateRecord bug (line 263): Changed r.PutRecord to r.UpdateRecord 10 - // (UpdateRecord was incorrectly calling PutRecord, causing incorrect MST operations) 11 - // - Removed 5 Prometheus metrics calls (openAndSigCheckDuration, calcDiffDuration, 12 - // writeCarSliceDuration, repoOpsImported) as metrics are not used in this project 13 - // - Added PutRecord method (lines 309-381) for creating records with explicit rkeys 14 - // (like CreateRecord but with specified rkey instead of auto-generated TID) 15 - // Based on streamplace/indigo implementation 16 - // - Added prevData to support Sync 1.1 17 - package pds 10 + // Implements the RepoOperator interface (see repo_operator.go). 11 + // See docs/REPOMGR_MIGRATION.md for planned migration to indigo/repo directly. 18 12 19 13 import ( 20 - "bytes" 21 14 "context" 22 - "errors" 23 15 "fmt" 24 16 "io" 25 17 "log/slog" 26 - "strings" 27 18 "sync" 28 19 29 20 holddb "atcr.io/pkg/hold/db" 30 21 atproto "github.com/bluesky-social/indigo/api/atproto" 31 22 bsky "github.com/bluesky-social/indigo/api/bsky" 32 23 "github.com/bluesky-social/indigo/atproto/syntax" 33 - lexutil "github.com/bluesky-social/indigo/lex/util" 34 24 "github.com/bluesky-social/indigo/models" 35 - "github.com/bluesky-social/indigo/mst" 36 25 "github.com/bluesky-social/indigo/repo" 37 26 "github.com/bluesky-social/indigo/util" 38 27 39 28 blocks "github.com/ipfs/go-block-format" 40 29 "github.com/ipfs/go-cid" 41 - "github.com/ipfs/go-datastore" 42 - blockstore "github.com/ipfs/go-ipfs-blockstore" 43 - ipld "github.com/ipfs/go-ipld-format" 44 - "github.com/ipld/go-car" 45 30 cbg "github.com/whyrusleeping/cbor-gen" 46 31 "go.opentelemetry.io/otel" 47 - "go.opentelemetry.io/otel/attribute" 48 - "gorm.io/gorm" 49 32 ) 50 33 51 34 func NewRepoManager(cs holddb.CarStore, kmgr KeyManager) *RepoManager { ··· 56 39 userLocks: make(map[models.Uid]*userLock), 57 40 kmgr: kmgr, 58 41 log: slog.Default().With("system", "repomgr"), 59 - noArchive: false, // NonArchivalCarstore not used in hold service 60 42 clk: clk, 61 43 } 62 44 } 63 45 64 - type KeyManager interface { 65 - VerifyUserSignature(context.Context, string, []byte, []byte) error 66 - SignForUser(context.Context, string, []byte) ([]byte, error) 67 - } 68 - 69 46 func (rm *RepoManager) SetEventHandler(cb func(context.Context, *RepoEvent), hydrateRecords bool) { 70 47 rm.events = cb 71 48 rm.hydrateRecords = hydrateRecords ··· 81 58 events func(context.Context, *RepoEvent) 82 59 hydrateRecords bool 83 60 84 - log *slog.Logger 85 - noArchive bool 86 - 61 + log *slog.Logger 87 62 clk *syntax.TIDClock 88 63 } 89 64 90 - // NextTID generates a new TID for use as a record key. 91 - func (rm *RepoManager) NextTID() string { 92 - return rm.clk.Next().String() 93 - } 94 - 95 - type ActorInfo struct { 96 - Did string 97 - Handle string 98 - DisplayName string 99 - Type string 100 - } 101 - 102 - type RepoEvent struct { 103 - User models.Uid 104 - OldRoot *cid.Cid 105 - NewRoot cid.Cid 106 - PrevData *cid.Cid // MST root CID of the previous commit (for firehose prevData field) 107 - Since *string 108 - Rev string 109 - RepoSlice []byte 110 - PDS uint 111 - Ops []RepoOp 112 - } 113 - 114 - type RepoOp struct { 115 - Kind EventKind 116 - Collection string 117 - Rkey string 118 - RecCid *cid.Cid 119 - Record any 120 - ActorInfo *ActorInfo 121 - } 122 - 123 - type EventKind string 124 - 125 - const ( 126 - EvtKindCreateRecord = EventKind("create") 127 - EvtKindUpdateRecord = EventKind("update") 128 - EvtKindDeleteRecord = EventKind("delete") 129 - ) 130 - 131 - type RepoHead struct { 132 - gorm.Model 133 - Usr models.Uid `gorm:"uniqueIndex"` 134 - Root string 135 - } 136 - 137 65 type userLock struct { 138 66 lk sync.Mutex 139 67 count int ··· 168 96 } 169 97 rm.lklk.Unlock() 170 98 } 171 - } 172 - 173 - func (rm *RepoManager) CarStore() holddb.CarStore { 174 - return rm.cs 175 99 } 176 100 177 101 func (rm *RepoManager) CreateRecord(ctx context.Context, user models.Uid, collection string, rec cbg.CBORMarshaler) (string, cid.Cid, error) { ··· 707 631 return head, bs.GetLoggedBlocks(), nil 708 632 } 709 633 710 - func (rm *RepoManager) GetProfile(ctx context.Context, uid models.Uid) (*bsky.ActorProfile, error) { 711 - bs, err := rm.cs.ReadOnlySession(uid) 712 - if err != nil { 713 - return nil, err 714 - } 715 - 716 - head, err := rm.cs.GetUserRepoHead(ctx, uid) 717 - if err != nil { 718 - return nil, err 719 - } 720 - 721 - r, err := repo.OpenRepo(ctx, bs, head) 722 - if err != nil { 723 - return nil, err 724 - } 725 - 726 - _, val, err := r.GetRecord(ctx, "app.bsky.actor.profile/self") 727 - if err != nil { 728 - return nil, err 729 - } 730 - 731 - ap, ok := val.(*bsky.ActorProfile) 732 - if !ok { 733 - return nil, fmt.Errorf("found wrong type in actor profile location in tree") 734 - } 735 - 736 - return ap, nil 737 - } 738 - 739 - func (rm *RepoManager) CheckRepoSig(ctx context.Context, r *repo.Repo, expdid string) error { 740 - ctx, span := otel.Tracer("repoman").Start(ctx, "CheckRepoSig") 741 - defer span.End() 742 - 743 - repoDid := r.RepoDid() 744 - if expdid != repoDid { 745 - return fmt.Errorf("DID in repo did not match (%q != %q)", expdid, repoDid) 746 - } 747 - 748 - scom := r.SignedCommit() 749 - 750 - usc := scom.Unsigned() 751 - sb, err := usc.BytesForSigning() 752 - if err != nil { 753 - return fmt.Errorf("commit serialization failed: %w", err) 754 - } 755 - if err := rm.kmgr.VerifyUserSignature(ctx, repoDid, scom.Sig, sb); err != nil { 756 - return fmt.Errorf("signature check failed (sig: %x) (sb: %x) : %w", scom.Sig, sb, err) 757 - } 758 - 759 - return nil 760 - } 761 - 762 - func (rm *RepoManager) HandleExternalUserEvent(ctx context.Context, pdsid uint, uid models.Uid, did string, since *string, nrev string, carslice []byte, ops []*atproto.SyncSubscribeRepos_RepoOp) error { 763 - if rm.noArchive { 764 - return rm.handleExternalUserEventNoArchive(ctx, pdsid, uid, did, since, nrev, carslice, ops) 765 - } else { 766 - return rm.handleExternalUserEventArchive(ctx, pdsid, uid, did, since, nrev, carslice, ops) 767 - } 768 - } 769 - 770 - func (rm *RepoManager) handleExternalUserEventNoArchive(ctx context.Context, pdsid uint, uid models.Uid, did string, since *string, nrev string, carslice []byte, ops []*atproto.SyncSubscribeRepos_RepoOp) error { 771 - ctx, span := otel.Tracer("repoman").Start(ctx, "HandleExternalUserEvent") 772 - defer span.End() 773 - 774 - span.SetAttributes(attribute.Int64("uid", int64(uid))) 775 - 776 - rm.log.Debug("HandleExternalUserEvent", "pds", pdsid, "uid", uid, "since", since, "nrev", nrev) 777 - 778 - unlock := rm.lockUser(ctx, uid) 779 - defer unlock() 780 - 781 - root, ds, err := rm.cs.ImportSlice(ctx, uid, since, carslice) 782 - if err != nil { 783 - return fmt.Errorf("importing external carslice: %w", err) 784 - } 785 - 786 - r, err := repo.OpenRepo(ctx, ds, root) 787 - if err != nil { 788 - return fmt.Errorf("opening external user repo (%d, root=%s): %w", uid, root, err) 789 - } 790 - 791 - if err := rm.CheckRepoSig(ctx, r, did); err != nil { 792 - return fmt.Errorf("check repo sig: %w", err) 793 - } 794 - 795 - // Capture previous MST root from old repo state if it exists 796 - var prevData *cid.Cid 797 - if ds.BaseCid().Defined() { 798 - oldrepo, err := repo.OpenRepo(ctx, ds, ds.BaseCid()) 799 - if err == nil { 800 - pd := oldrepo.DataCid() 801 - prevData = &pd 802 - } 803 - } 804 - 805 - evtops := make([]RepoOp, 0, len(ops)) 806 - for _, op := range ops { 807 - parts := strings.SplitN(op.Path, "/", 2) 808 - if len(parts) != 2 { 809 - return fmt.Errorf("invalid rpath in mst diff, must have collection and rkey") 810 - } 811 - 812 - switch EventKind(op.Action) { 813 - case EvtKindCreateRecord: 814 - rop := RepoOp{ 815 - Kind: EvtKindCreateRecord, 816 - Collection: parts[0], 817 - Rkey: parts[1], 818 - RecCid: (*cid.Cid)(op.Cid), 819 - } 820 - 821 - if rm.hydrateRecords { 822 - _, rec, err := r.GetRecord(ctx, op.Path) 823 - if err != nil { 824 - return fmt.Errorf("reading changed record from car slice: %w", err) 825 - } 826 - rop.Record = rec 827 - } 828 - 829 - evtops = append(evtops, rop) 830 - case EvtKindUpdateRecord: 831 - rop := RepoOp{ 832 - Kind: EvtKindUpdateRecord, 833 - Collection: parts[0], 834 - Rkey: parts[1], 835 - RecCid: (*cid.Cid)(op.Cid), 836 - } 837 - 838 - if rm.hydrateRecords { 839 - _, rec, err := r.GetRecord(ctx, op.Path) 840 - if err != nil { 841 - return fmt.Errorf("reading changed record from car slice: %w", err) 842 - } 843 - 844 - rop.Record = rec 845 - } 846 - 847 - evtops = append(evtops, rop) 848 - case EvtKindDeleteRecord: 849 - evtops = append(evtops, RepoOp{ 850 - Kind: EvtKindDeleteRecord, 851 - Collection: parts[0], 852 - Rkey: parts[1], 853 - }) 854 - default: 855 - return fmt.Errorf("unrecognized external user event kind: %q", op.Action) 856 - } 857 - } 858 - 859 - if rm.events != nil { 860 - rm.events(ctx, &RepoEvent{ 861 - User: uid, 862 - //OldRoot: prev, 863 - NewRoot: root, 864 - PrevData: prevData, 865 - Rev: nrev, 866 - Since: since, 867 - Ops: evtops, 868 - RepoSlice: carslice, 869 - PDS: pdsid, 870 - }) 871 - } 872 - 873 - return nil 874 - } 875 - 876 - func (rm *RepoManager) handleExternalUserEventArchive(ctx context.Context, pdsid uint, uid models.Uid, did string, since *string, nrev string, carslice []byte, ops []*atproto.SyncSubscribeRepos_RepoOp) error { 877 - ctx, span := otel.Tracer("repoman").Start(ctx, "HandleExternalUserEvent") 878 - defer span.End() 879 - 880 - span.SetAttributes(attribute.Int64("uid", int64(uid))) 881 - 882 - rm.log.Debug("HandleExternalUserEvent", "pds", pdsid, "uid", uid, "since", since, "nrev", nrev) 883 - 884 - unlock := rm.lockUser(ctx, uid) 885 - defer unlock() 886 - 887 - root, ds, err := rm.cs.ImportSlice(ctx, uid, since, carslice) 888 - if err != nil { 889 - return fmt.Errorf("importing external carslice: %w", err) 890 - } 891 - 892 - r, err := repo.OpenRepo(ctx, ds, root) 893 - if err != nil { 894 - return fmt.Errorf("opening external user repo (%d, root=%s): %w", uid, root, err) 895 - } 896 - 897 - if err := rm.CheckRepoSig(ctx, r, did); err != nil { 898 - return err 899 - } 900 - 901 - var skipcids map[cid.Cid]bool 902 - var prevData *cid.Cid 903 - if ds.BaseCid().Defined() { 904 - oldrepo, err := repo.OpenRepo(ctx, ds, ds.BaseCid()) 905 - if err != nil { 906 - return fmt.Errorf("failed to check data root in old repo: %w", err) 907 - } 908 - 909 - // Capture previous MST root for prevData 910 - pd := oldrepo.DataCid() 911 - prevData = &pd 912 - 913 - // if the old commit has a 'prev', CalcDiff will error out while trying 914 - // to walk it. This is an old repo thing that is being deprecated. 915 - // This check is a temporary workaround until all repos get migrated 916 - // and this becomes no longer an issue 917 - prev, _ := oldrepo.PrevCommit(ctx) 918 - if prev != nil { 919 - skipcids = map[cid.Cid]bool{ 920 - *prev: true, 921 - } 922 - } 923 - } 924 - 925 - if err := ds.CalcDiff(ctx, skipcids); err != nil { 926 - return fmt.Errorf("failed while calculating mst diff (since=%v): %w", since, err) 927 - } 928 - 929 - evtops := make([]RepoOp, 0, len(ops)) 930 - 931 - for _, op := range ops { 932 - parts := strings.SplitN(op.Path, "/", 2) 933 - if len(parts) != 2 { 934 - return fmt.Errorf("invalid rpath in mst diff, must have collection and rkey") 935 - } 936 - 937 - switch EventKind(op.Action) { 938 - case EvtKindCreateRecord: 939 - rop := RepoOp{ 940 - Kind: EvtKindCreateRecord, 941 - Collection: parts[0], 942 - Rkey: parts[1], 943 - RecCid: (*cid.Cid)(op.Cid), 944 - } 945 - 946 - if rm.hydrateRecords { 947 - _, rec, err := r.GetRecord(ctx, op.Path) 948 - if err != nil { 949 - return fmt.Errorf("reading changed record from car slice: %w", err) 950 - } 951 - rop.Record = rec 952 - } 953 - 954 - evtops = append(evtops, rop) 955 - case EvtKindUpdateRecord: 956 - rop := RepoOp{ 957 - Kind: EvtKindUpdateRecord, 958 - Collection: parts[0], 959 - Rkey: parts[1], 960 - RecCid: (*cid.Cid)(op.Cid), 961 - } 962 - 963 - if rm.hydrateRecords { 964 - _, rec, err := r.GetRecord(ctx, op.Path) 965 - if err != nil { 966 - return fmt.Errorf("reading changed record from car slice: %w", err) 967 - } 968 - 969 - rop.Record = rec 970 - } 971 - 972 - evtops = append(evtops, rop) 973 - case EvtKindDeleteRecord: 974 - evtops = append(evtops, RepoOp{ 975 - Kind: EvtKindDeleteRecord, 976 - Collection: parts[0], 977 - Rkey: parts[1], 978 - }) 979 - default: 980 - return fmt.Errorf("unrecognized external user event kind: %q", op.Action) 981 - } 982 - } 983 - 984 - rslice, err := ds.CloseWithRoot(ctx, root, nrev) 985 - if err != nil { 986 - return fmt.Errorf("close with root: %w", err) 987 - } 988 - 989 - if rm.events != nil { 990 - rm.events(ctx, &RepoEvent{ 991 - User: uid, 992 - //OldRoot: prev, 993 - NewRoot: root, 994 - PrevData: prevData, 995 - Rev: nrev, 996 - Since: since, 997 - Ops: evtops, 998 - RepoSlice: rslice, 999 - PDS: pdsid, 1000 - }) 1001 - } 1002 - 1003 - return nil 1004 - } 1005 - 1006 634 func (rm *RepoManager) BatchWrite(ctx context.Context, user models.Uid, writes []*atproto.RepoApplyWrites_Input_Writes_Elem) error { 1007 635 ctx, span := otel.Tracer("repoman").Start(ctx, "BatchWrite") 1008 636 defer span.End() ··· 1131 759 return nil 1132 760 } 1133 761 1134 - func (rm *RepoManager) ImportNewRepo(ctx context.Context, user models.Uid, repoDid string, r io.Reader, rev *string) error { 1135 - ctx, span := otel.Tracer("repoman").Start(ctx, "ImportNewRepo") 762 + // BulkUpsert writes multiple records in a single delta session and commit. 763 + // Each record is upserted: created if new, updated if it already exists. 764 + func (rm *RepoManager) BulkUpsert(ctx context.Context, user models.Uid, records []BulkRecord) error { 765 + ctx, span := otel.Tracer("repoman").Start(ctx, "BulkUpsert") 1136 766 defer span.End() 1137 767 1138 768 unlock := rm.lockUser(ctx, user) 1139 769 defer unlock() 1140 770 1141 - currev, err := rm.cs.GetUserRepoRev(ctx, user) 771 + rev, err := rm.cs.GetUserRepoRev(ctx, user) 1142 772 if err != nil { 1143 773 return err 1144 774 } 1145 775 1146 - curhead, err := rm.cs.GetUserRepoHead(ctx, user) 776 + ds, err := rm.cs.NewDeltaSession(ctx, user, &rev) 1147 777 if err != nil { 1148 778 return err 1149 779 } 1150 780 1151 - if rev != nil && *rev == "" { 1152 - rev = nil 1153 - } 1154 - if rev == nil { 1155 - // if 'rev' is nil, this implies a fresh sync. 1156 - // in this case, ignore any existing blocks we have and treat this like a clean import. 1157 - curhead = cid.Undef 1158 - } 1159 - 1160 - if rev != nil && *rev != currev { 1161 - // TODO: we could probably just deal with this 1162 - return fmt.Errorf("ImportNewRepo called with incorrect base") 1163 - } 1164 - 1165 - // Capture previous MST root before import overwrites it 1166 - var prevData *cid.Cid 1167 - if curhead.Defined() { 1168 - robs, err := rm.cs.ReadOnlySession(user) 1169 - if err == nil { 1170 - oldrepo, err := repo.OpenRepo(ctx, robs, curhead) 1171 - if err == nil { 1172 - pd := oldrepo.DataCid() 1173 - prevData = &pd 1174 - } 1175 - } 1176 - } 1177 - 1178 - err = rm.processNewRepo(ctx, user, r, rev, func(ctx context.Context, root cid.Cid, finish func(context.Context, string) ([]byte, error), bs blockstore.Blockstore) error { 1179 - r, err := repo.OpenRepo(ctx, bs, root) 1180 - if err != nil { 1181 - return fmt.Errorf("opening new repo: %w", err) 1182 - } 1183 - 1184 - scom := r.SignedCommit() 1185 - 1186 - usc := scom.Unsigned() 1187 - sb, err := usc.BytesForSigning() 1188 - if err != nil { 1189 - return fmt.Errorf("commit serialization failed: %w", err) 1190 - } 1191 - if err := rm.kmgr.VerifyUserSignature(ctx, repoDid, scom.Sig, sb); err != nil { 1192 - return fmt.Errorf("new user signature check failed: %w", err) 1193 - } 1194 - 1195 - diffops, err := r.DiffSince(ctx, curhead) 1196 - if err != nil { 1197 - return fmt.Errorf("diff trees (curhead: %s): %w", curhead, err) 1198 - } 1199 - 1200 - ops := make([]RepoOp, 0, len(diffops)) 1201 - for _, op := range diffops { 1202 - out, err := rm.processOp(ctx, bs, op, rm.hydrateRecords) 1203 - if err != nil { 1204 - rm.log.Error("failed to process repo op", "err", err, "path", op.Rpath, "repo", repoDid) 1205 - } 1206 - 1207 - if out != nil { 1208 - ops = append(ops, *out) 1209 - } 1210 - } 1211 - 1212 - slice, err := finish(ctx, scom.Rev) 1213 - if err != nil { 1214 - return err 1215 - } 1216 - 1217 - if rm.events != nil { 1218 - rm.events(ctx, &RepoEvent{ 1219 - User: user, 1220 - //OldRoot: oldroot, 1221 - NewRoot: root, 1222 - PrevData: prevData, 1223 - Rev: scom.Rev, 1224 - Since: &currev, 1225 - RepoSlice: slice, 1226 - Ops: ops, 1227 - }) 1228 - } 1229 - 1230 - return nil 1231 - }) 1232 - if err != nil { 1233 - return fmt.Errorf("process new repo (current rev: %s): %w", currev, err) 1234 - } 1235 - 1236 - return nil 1237 - } 1238 - 1239 - func (rm *RepoManager) processOp(ctx context.Context, bs blockstore.Blockstore, op *mst.DiffOp, hydrateRecords bool) (*RepoOp, error) { 1240 - parts := strings.SplitN(op.Rpath, "/", 2) 1241 - if len(parts) != 2 { 1242 - return nil, fmt.Errorf("repo mst had invalid rpath: %q", op.Rpath) 1243 - } 1244 - 1245 - switch op.Op { 1246 - case "add", "mut": 1247 - 1248 - kind := EvtKindCreateRecord 1249 - if op.Op == "mut" { 1250 - kind = EvtKindUpdateRecord 1251 - } 1252 - 1253 - outop := &RepoOp{ 1254 - Kind: kind, 1255 - Collection: parts[0], 1256 - Rkey: parts[1], 1257 - RecCid: &op.NewCid, 1258 - } 1259 - 1260 - if hydrateRecords { 1261 - blk, err := bs.Get(ctx, op.NewCid) 1262 - if err != nil { 1263 - return nil, err 1264 - } 1265 - 1266 - rec, err := lexutil.CborDecodeValue(blk.RawData()) 1267 - if err != nil { 1268 - if !errors.Is(err, lexutil.ErrUnrecognizedType) { 1269 - return nil, err 1270 - } 1271 - 1272 - rm.log.Warn("failed processing repo diff", "err", err) 1273 - } else { 1274 - outop.Record = rec 1275 - } 1276 - } 1277 - 1278 - return outop, nil 1279 - case "del": 1280 - return &RepoOp{ 1281 - Kind: EvtKindDeleteRecord, 1282 - Collection: parts[0], 1283 - Rkey: parts[1], 1284 - RecCid: nil, 1285 - }, nil 1286 - 1287 - default: 1288 - return nil, fmt.Errorf("diff returned invalid op type: %q", op.Op) 1289 - } 1290 - } 1291 - 1292 - func (rm *RepoManager) processNewRepo(ctx context.Context, user models.Uid, r io.Reader, rev *string, cb func(ctx context.Context, root cid.Cid, finish func(context.Context, string) ([]byte, error), bs blockstore.Blockstore) error) error { 1293 - ctx, span := otel.Tracer("repoman").Start(ctx, "processNewRepo") 1294 - defer span.End() 1295 - 1296 - carr, err := car.NewCarReader(r) 781 + head := ds.BaseCid() 782 + r, err := repo.OpenRepo(ctx, ds, head) 1297 783 if err != nil { 1298 784 return err 1299 785 } 1300 786 1301 - if len(carr.Header.Roots) != 1 { 1302 - return fmt.Errorf("invalid car file, header must have a single root (has %d)", len(carr.Header.Roots)) 787 + // Capture previous MST root before commit overwrites it 788 + var prevData *cid.Cid 789 + if head.Defined() { 790 + pd := r.DataCid() 791 + prevData = &pd 1303 792 } 1304 793 1305 - membs := blockstore.NewBlockstore(datastore.NewMapDatastore()) 794 + ops := make([]RepoOp, 0, len(records)) 795 + for _, rec := range records { 796 + rpath := rec.Collection + "/" + rec.Rkey 1306 797 1307 - for { 1308 - blk, err := carr.Next() 1309 - if err != nil { 1310 - if err == io.EOF { 1311 - break 1312 - } 1313 - return err 1314 - } 798 + // Check if record exists to determine create vs update 799 + _, _, getErr := r.GetRecordBytes(ctx, rpath) 800 + recordExists := getErr == nil 1315 801 1316 - if err := membs.Put(ctx, blk); err != nil { 1317 - return err 802 + var cc cid.Cid 803 + var evtKind EventKind 804 + if recordExists { 805 + cc, err = r.UpdateRecord(ctx, rpath, rec.Data) 806 + evtKind = EvtKindUpdateRecord 807 + } else { 808 + cc, err = r.PutRecord(ctx, rpath, rec.Data) 809 + evtKind = EvtKindCreateRecord 1318 810 } 1319 - } 1320 - 1321 - seen := make(map[cid.Cid]bool) 1322 - 1323 - root := carr.Header.Roots[0] 1324 - // TODO: if there are blocks that get convergently recreated throughout 1325 - // the repos lifecycle, this will end up erroneously not including 1326 - // them. We should compute the set of blocks needed to read any repo 1327 - // ops that happened in the commit and use that for our 'output' blocks 1328 - cids, err := rm.walkTree(ctx, seen, root, membs, true) 1329 - if err != nil { 1330 - return fmt.Errorf("walkTree: %w", err) 1331 - } 1332 - 1333 - ds, err := rm.cs.NewDeltaSession(ctx, user, rev) 1334 - if err != nil { 1335 - return fmt.Errorf("opening delta session: %w", err) 1336 - } 1337 - 1338 - for _, c := range cids { 1339 - blk, err := membs.Get(ctx, c) 1340 811 if err != nil { 1341 - return fmt.Errorf("copying walked cids to carstore: %w", err) 812 + return fmt.Errorf("failed to write %s: %w", rpath, err) 1342 813 } 1343 814 1344 - if err := ds.Put(ctx, blk); err != nil { 1345 - return err 1346 - } 815 + ops = append(ops, RepoOp{ 816 + Kind: evtKind, 817 + Collection: rec.Collection, 818 + Rkey: rec.Rkey, 819 + RecCid: &cc, 820 + }) 1347 821 } 1348 822 1349 - finish := func(ctx context.Context, nrev string) ([]byte, error) { 1350 - return ds.CloseWithRoot(ctx, root, nrev) 1351 - } 1352 - 1353 - if err := cb(ctx, root, finish, ds); err != nil { 1354 - return fmt.Errorf("cb errored root: %s, rev: %s: %w", root, stringOrNil(rev), err) 1355 - } 1356 - 1357 - return nil 1358 - } 1359 - 1360 - func stringOrNil(s *string) string { 1361 - if s == nil { 1362 - return "nil" 1363 - } 1364 - return *s 1365 - } 1366 - 1367 - // walkTree returns all cids linked recursively by the root, skipping any cids 1368 - // in the 'skip' map, and not erroring on 'not found' if prevMissing is set 1369 - func (rm *RepoManager) walkTree(ctx context.Context, skip map[cid.Cid]bool, root cid.Cid, bs blockstore.Blockstore, prevMissing bool) ([]cid.Cid, error) { 1370 - // TODO: what if someone puts non-cbor links in their repo? 1371 - if root.Prefix().Codec != cid.DagCBOR { 1372 - return nil, fmt.Errorf("can only handle dag-cbor objects in repos (%s is %d)", root, root.Prefix().Codec) 1373 - } 1374 - 1375 - blk, err := bs.Get(ctx, root) 1376 - if err != nil { 1377 - return nil, err 1378 - } 1379 - 1380 - var links []cid.Cid 1381 - if err := cbg.ScanForLinks(bytes.NewReader(blk.RawData()), func(c cid.Cid) { 1382 - if c.Prefix().Codec == cid.Raw { 1383 - rm.log.Debug("skipping 'raw' CID in record", "recordCid", root, "rawCid", c) 1384 - return 1385 - } 1386 - if skip[c] { 1387 - return 1388 - } 1389 - 1390 - links = append(links, c) 1391 - skip[c] = true 1392 - }); err != nil { 1393 - return nil, err 1394 - } 1395 - 1396 - out := []cid.Cid{root} 1397 - skip[root] = true 1398 - 1399 - // TODO: should do this non-recursive since i expect these may get deep 1400 - for _, c := range links { 1401 - sub, err := rm.walkTree(ctx, skip, c, bs, prevMissing) 1402 - if err != nil { 1403 - if prevMissing && !ipld.IsNotFound(err) { 1404 - return nil, err 1405 - } 1406 - } 1407 - 1408 - out = append(out, sub...) 1409 - } 1410 - 1411 - return out, nil 1412 - } 1413 - 1414 - func (rm *RepoManager) TakeDownRepo(ctx context.Context, uid models.Uid) error { 1415 - unlock := rm.lockUser(ctx, uid) 1416 - defer unlock() 1417 - 1418 - return rm.cs.WipeUserData(ctx, uid) 1419 - } 1420 - 1421 - // ResetRepo is technically identical to TakeDownRepo, for now 1422 - func (rm *RepoManager) ResetRepo(ctx context.Context, uid models.Uid) error { 1423 - unlock := rm.lockUser(ctx, uid) 1424 - defer unlock() 1425 - 1426 - return rm.cs.WipeUserData(ctx, uid) 1427 - } 1428 - 1429 - func (rm *RepoManager) VerifyRepo(ctx context.Context, uid models.Uid) error { 1430 - ses, err := rm.cs.ReadOnlySession(uid) 823 + nroot, nrev, err := r.Commit(ctx, rm.kmgr.SignForUser) 1431 824 if err != nil { 1432 825 return err 1433 826 } 1434 827 1435 - r, err := repo.OpenRepo(ctx, ses, ses.BaseCid()) 828 + rslice, err := ds.CloseWithRoot(ctx, nroot, nrev) 1436 829 if err != nil { 1437 - return err 830 + return fmt.Errorf("close with root: %w", err) 1438 831 } 1439 832 1440 - if err := r.ForEach(ctx, "", func(k string, v cid.Cid) error { 1441 - _, err := ses.Get(ctx, v) 1442 - if err != nil { 1443 - return fmt.Errorf("failed to get record %s (%s): %w", k, v, err) 1444 - } 833 + var oldroot *cid.Cid 834 + if head.Defined() { 835 + oldroot = &head 836 + } 1445 837 1446 - return nil 1447 - }); err != nil { 1448 - return err 838 + if rm.events != nil { 839 + rm.events(ctx, &RepoEvent{ 840 + User: user, 841 + OldRoot: oldroot, 842 + NewRoot: nroot, 843 + PrevData: prevData, 844 + Rev: nrev, 845 + Since: &rev, 846 + Ops: ops, 847 + RepoSlice: rslice, 848 + }) 1449 849 } 1450 850 1451 851 return nil
+6 -6
pkg/hold/pds/server.go
··· 41 41 appviewURL string 42 42 appviewMeta *atproto.AppviewMetadata 43 43 carstore holddb.CarStore 44 - repomgr *RepoManager 44 + repomgr RepoOperator 45 45 dbPath string 46 46 uid models.Uid 47 47 signingKey *atcrypto.PrivateKeyK256 ··· 106 106 // Create KeyManager wrapper for our signing key 107 107 kmgr := NewHoldKeyManager(signingKey) 108 108 109 - // Create RepoManager - it will handle all session/repo lifecycle 110 - rm := NewRepoManager(cs, kmgr) 109 + // Create repo operator - handles all session/repo lifecycle 110 + rm := NewDirectRepoOperator(cs, kmgr) 111 111 112 112 // Check if repo already exists, if not create initial commit 113 113 head, err := cs.GetUserRepoHead(ctx, uid) ··· 162 162 cs := sqlStore 163 163 uid := models.Uid(1) 164 164 kmgr := NewHoldKeyManager(signingKey) 165 - rm := NewRepoManager(cs, kmgr) 165 + rm := NewDirectRepoOperator(cs, kmgr) 166 166 167 167 head, err := cs.GetUserRepoHead(ctx, uid) 168 168 hasValidRepo := (err == nil && head.Defined()) ··· 200 200 return p.signingKey 201 201 } 202 202 203 - // RepomgrRef returns a reference to the RepoManager for event handler setup 204 - func (p *HoldPDS) RepomgrRef() *RepoManager { 203 + // RepomgrRef returns a reference to the RepoOperator for event handler setup 204 + func (p *HoldPDS) RepomgrRef() RepoOperator { 205 205 return p.repomgr 206 206 } 207 207