Implement did:plc support for holds with the ability to import/export CARs. · evan.jarrett.net/at-container-registry@e3843db

+40 -3

CLAUDE.md

··· 36 36 go test -race ./... # race detector 37 37 38 38 # Docker 39 - docker build -t atcr.io/appview:latest . 39 + docker build -f Dockerfile.appview -t atcr.io/appview:latest . 40 40 docker build -f Dockerfile.hold -t atcr.io/hold:latest . 41 41 docker build -f Dockerfile.scanner -t atcr.io/scanner:latest . 42 42 docker-compose up -d ··· 53 53 # Usage report 54 54 go run ./cmd/usage-report --hold https://hold01.atcr.io 55 55 go run ./cmd/usage-report --hold https://hold01.atcr.io --from-manifests 56 + 57 + # Utilities 58 + go run ./cmd/db-migrate --help # SQLite → libsql migration 59 + go run ./cmd/record-query --help # Query ATProto relay by collection 60 + go run ./cmd/s3-test # S3 connectivity test 61 + go run ./cmd/healthcheck <url> # HTTP health check (for Docker) 56 62 ``` 57 63 58 64 ## Architecture Overview ··· 66 72 ### Four Components 67 73 68 74 1. **AppView** (`cmd/appview`) — OCI Distribution API server. Resolves identities, routes manifests to PDS, routes blobs to hold service, validates OAuth, issues registry JWTs. Includes web UI for browsing. 69 - 2. **Hold Service** (`cmd/hold`) — BYOS blob storage. Embedded PDS with captain/crew records, S3-compatible storage, presigned URLs. Optional subsystems: admin UI, quotas, billing (Stripe), GC, scan dispatch. 75 + 2. **Hold Service** (`cmd/hold`) — BYOS blob storage. Embedded PDS with captain/crew/stats/scan records (all ATProto records in CAR store), S3-compatible storage, presigned URLs. Supports did:web (default) or did:plc identity with auto-recovery. Optional subsystems: admin UI, quotas, billing (Stripe), GC, scan dispatch, Bluesky status posts. 70 76 3. **Scanner** (`scanner/cmd/scanner`) — Vulnerability scanning. Connects to hold via WebSocket, generates SBOMs (Syft), scans vulnerabilities (Grype). Priority queue with tier-based scheduling. 71 77 4. **Credential Helper** (`cmd/credential-helper`) — Docker credential helper implementing ATProto OAuth flow, exchanges OAuth token for registry JWT. 72 78 ··· 92 98 - **Sailors** = registry users, **Captains** = hold owners, **Crew** = hold members 93 99 - **Holds** = storage endpoints (BYOS), **Quartermaster/Bosun/Deckhand** = crew tiers 94 100 101 + ### Hold Embedded PDS Records 102 + 103 + The hold's embedded PDS stores all operational data as ATProto records in a CAR store (not SQLite). SQLite holds only the records index and events. 104 + 105 + | Collection | Cardinality | Description | 106 + |---|---|---| 107 + | `io.atcr.hold.captain` | Singleton | Hold identity, owner DID, settings | 108 + | `io.atcr.hold.crew` | Per-member | Crew membership + permissions | 109 + | `io.atcr.hold.layer` | Per-layer | Layer metadata (digest, size, media type) | 110 + | `io.atcr.hold.stats` | Per-repo | Push/pull counts per owner+repository | 111 + | `io.atcr.hold.scan` | Per-scan | Vulnerability scan results | 112 + | `app.bsky.feed.post` | Status posts | Online/offline status, push notifications | 113 + | `sh.tangled.actor.profile` | Singleton | Hold profile (name, description, avatar) | 114 + 95 115 ## Authentication 96 116 97 117 Three token types flow through the system: ··· 135 155 | OAuth client & session refresher | `pkg/auth/oauth/client.go` | 136 156 | OAuth P-256 key management | `pkg/auth/oauth/keys.go` | 137 157 | Hold PDS endpoints & auth | `pkg/hold/pds/xrpc.go`, `pkg/hold/pds/auth.go` | 158 + | Hold DID management (did:web, did:plc, PLC recovery) | `pkg/hold/pds/did.go` | 159 + | Hold captain records | `pkg/hold/pds/captain.go` | 160 + | Hold crew management | `pkg/hold/pds/crew.go` | 161 + | Hold push/pull stats (ATProto records in CAR store) | `pkg/hold/pds/stats.go` | 162 + | Hold layer records | `pkg/hold/pds/layer.go` | 163 + | Hold scan records & scanner integration | `pkg/hold/pds/scan.go`, `pkg/hold/pds/scan_broadcaster.go` | 164 + | Hold Bluesky status posts | `pkg/hold/pds/status.go` | 138 165 | Hold OCI upload endpoints | `pkg/hold/oci/xrpc.go` | 139 166 | Hold config | `pkg/hold/config.go` | 140 167 | AppView config | `pkg/appview/config.go` | ··· 163 190 - **Context keys** (`auth.method`, `puller.did`) exist because `Repository()` receives `context.Context` from the distribution library interface — context values are the only way to pass data from HTTP middleware into the distribution middleware layer. Both are copied into `RegistryContext` inside `Repository()`. 164 191 - **OAuth key types**: AppView uses P-256 (ES256) for OAuth, not K-256 like PDS keys 165 192 - **Confidential vs public clients**: Production uses P-256 key at `/var/lib/atcr/oauth/client.key` (auto-generated); localhost is always public client 193 + - **Hold stats are ATProto records in CAR store** — `io.atcr.hold.stats` records are stored via `repomgr.PutRecord()`, not in SQLite. Lost if CAR store is lost without backup. 194 + - **PLC auto-update on boot** — When using did:plc, `LoadOrCreateDID()` calls `EnsurePLCCurrent()` every startup. If local signing key or URL doesn't match plc.directory, it auto-updates (requires rotation key on disk). 195 + - **Hold CAR store is the source of truth** — Captain, crew, layer, stats, scan records, Bluesky posts, profiles are all ATProto records in the CAR store. SQLite holds only the records index and events. 166 196 167 197 ## Common Tasks 168 198 ··· 199 229 - **Adding new tables**: Add to `schema.sql` only (no migration needed) 200 230 - **Altering tables**: Create migration AND update `schema.sql` to keep them in sync 201 231 232 + **Hold DID recovery/migration (did:plc):** 233 + 1. Back up `rotation.key` and DID string (from `did.txt` or plc.directory) 234 + 2. Set `database.did_method: plc` and `database.did: "did:plc:..."` in config 235 + 3. Provide `rotation_key_path` — signing key auto-generates if missing 236 + 4. On boot: `LoadOrCreateDID()` adopts the DID, `EnsurePLCCurrent()` auto-updates PLC directory if keys/URL changed 237 + 5. Without rotation key: hold boots but logs warning about PLC mismatch 238 + 202 239 **Adding web UI features:** 203 240 - Add handler in `pkg/appview/handlers/` 204 - - Register route in `cmd/appview/serve.go` 241 + - Register route in `pkg/appview/routes/routes.go` 205 242 - Create template in `pkg/appview/templates/pages/` 206 243 207 244 ## Testing Strategy

+42 -19

README.md

··· 77 77 78 78 ### Running Your Own AppView 79 79 80 - **Using Docker Compose:** 81 - ```bash 82 - cp .env.appview.example .env.appview 83 - # Edit .env.appview with your configuration 84 - docker-compose up -d 85 - ``` 86 - 87 - **Local development:** 88 80 ```bash 89 81 # Build 90 82 go build -o bin/atcr-appview ./cmd/appview 91 - go build -o bin/atcr-hold ./cmd/hold 92 83 93 - # Configure 94 - cp .env.appview.example .env.appview 95 - # Edit .env.appview - set ATCR_DEFAULT_HOLD 96 - source .env.appview 84 + # Generate a config file with all defaults 85 + ./bin/atcr-appview config init config-appview.yaml 86 + # Edit config-appview.yaml — set server.default_hold_did at minimum 97 87 98 88 # Run 99 - ./bin/atcr-appview serve 89 + ./bin/atcr-appview serve --config config-appview.yaml 90 + ``` 91 + 92 + **Using Docker:** 93 + ```bash 94 + docker build -f Dockerfile.appview -t atcr-appview:latest . 95 + docker run -d -p 5000:5000 \ 96 + -v ./config-appview.yaml:/config.yaml:ro \ 97 + -v atcr-data:/var/lib/atcr \ 98 + atcr-appview:latest serve --config /config.yaml 100 99 ``` 101 100 102 101 See **[deploy/README.md](./deploy/README.md)** for production deployment. 102 + 103 + ### Running Your Own Hold (BYOS Storage) 104 + 105 + See **[docs/hold.md](./docs/hold.md)** for deploying your own storage backend. 103 106 104 107 ## Development 105 108 ··· 122 125 cmd/ 123 126 ├── appview/ # Registry server + web UI 124 127 ├── hold/ # Storage service (BYOS) 125 - └── credential-helper/ # Docker credential helper 128 + ├── credential-helper/ # Docker credential helper 129 + ├── oauth-helper/ # OAuth debug tool 130 + ├── healthcheck/ # HTTP health check (for Docker) 131 + ├── db-migrate/ # SQLite → libsql migration 132 + ├── usage-report/ # Hold storage usage report 133 + ├── record-query/ # Query ATProto relay by collection 134 + └── s3-test/ # S3 connectivity test 126 135 127 136 pkg/ 128 137 ├── appview/ 129 138 │ ├── db/ # SQLite database (migrations, queries, stores) 130 139 │ ├── handlers/ # HTTP handlers (home, repo, search, auth, settings) 140 + │ ├── holdhealth/ # Hold service health checker 131 141 │ ├── jetstream/ # ATProto Jetstream consumer 132 142 │ ├── middleware/ # Auth & registry middleware 133 - │ ├── storage/ # Storage routing (hold cache, blob proxy, repository) 143 + │ ├── ogcard/ # OpenGraph image generation 144 + │ ├── readme/ # Repository README fetcher 145 + │ ├── routes/ # HTTP route registration 146 + │ ├── storage/ # Storage routing (blob proxy, manifest store) 134 147 │ ├── public/ # Static assets (JS, CSS, install scripts) 135 148 │ └── templates/ # HTML templates 136 149 ├── atproto/ # ATProto client, records, manifest/tag stores 137 150 ├── auth/ 138 - │ ├── oauth/ # OAuth client, server, refresher, storage 151 + │ ├── oauth/ # OAuth client, refresher, storage 139 152 │ ├── token/ # JWT issuer, validator, claims 140 - │ └── atproto/ # Session validation 141 - └── hold/ # Hold service (authorization, storage, multipart, S3) 153 + │ └── holdlocal/ # Local hold authorization 154 + ├── config/ # Config marshaling (commented YAML) 155 + ├── hold/ 156 + │ ├── admin/ # Admin web UI 157 + │ ├── billing/ # Stripe billing integration 158 + │ ├── db/ # Vendored carstore (go-libsql) 159 + │ ├── gc/ # Garbage collection 160 + │ ├── oci/ # OCI upload endpoints 161 + │ ├── pds/ # Embedded PDS (DID, captain, crew, stats, scans) 162 + │ └── quota/ # Storage quotas 163 + ├── logging/ # Structured logging + remote shipping 164 + └── s3/ # S3 client utilities 142 165 ``` 143 166 144 167 ## License

+1

cmd/hold/main.go

··· 76 76 77 77 rootCmd.AddCommand(serveCmd) 78 78 rootCmd.AddCommand(configCmd) 79 + rootCmd.AddCommand(repoCmd) 79 80 } 80 81 81 82 func main() {

+146

cmd/hold/repo.go

··· 1 + package main 2 + 3 + import ( 4 + "context" 5 + "fmt" 6 + "log/slog" 7 + "os" 8 + 9 + "atcr.io/pkg/hold" 10 + holddb "atcr.io/pkg/hold/db" 11 + "atcr.io/pkg/hold/pds" 12 + 13 + "github.com/spf13/cobra" 14 + ) 15 + 16 + var repoCmd = &cobra.Command{ 17 + Use: "repo", 18 + Short: "Repository management commands", 19 + } 20 + 21 + var repoExportCmd = &cobra.Command{ 22 + Use: "export", 23 + Short: "Export the hold's repo as a CAR file to stdout", 24 + Long: `Export the hold's ATProto repository as a CAR (Content Addressable Archive) file. 25 + The CAR is written to stdout, so redirect to a file: 26 + 27 + atcr-hold repo export --config config.yaml > backup.car`, 28 + Args: cobra.NoArgs, 29 + RunE: func(cmd *cobra.Command, args []string) error { 30 + cfg, err := hold.LoadConfig(repoConfigFile) 31 + if err != nil { 32 + return fmt.Errorf("failed to load config: %w", err) 33 + } 34 + 35 + ctx := context.Background() 36 + holdPDS, cleanup, err := openHoldPDS(ctx, cfg) 37 + if err != nil { 38 + return err 39 + } 40 + defer cleanup() 41 + 42 + if err := holdPDS.ExportToCAR(ctx, os.Stdout); err != nil { 43 + return fmt.Errorf("failed to export: %w", err) 44 + } 45 + 46 + fmt.Fprintf(os.Stderr, "Export complete\n") 47 + return nil 48 + }, 49 + } 50 + 51 + var repoImportCmd = &cobra.Command{ 52 + Use: "import <file> [file...]", 53 + Short: "Import records from one or more CAR files", 54 + Long: `Import ATProto records from CAR files into the hold's repo. 55 + Records are upserted (existing records are overwritten). Multiple files can be 56 + imported additively. 57 + 58 + atcr-hold repo import --config config.yaml backup.car 59 + atcr-hold repo import --config config.yaml backup.car extra-records.car`, 60 + Args: cobra.MinimumNArgs(1), 61 + RunE: func(cmd *cobra.Command, args []string) error { 62 + cfg, err := hold.LoadConfig(repoConfigFile) 63 + if err != nil { 64 + return fmt.Errorf("failed to load config: %w", err) 65 + } 66 + 67 + ctx := context.Background() 68 + holdPDS, cleanup, err := openHoldPDS(ctx, cfg) 69 + if err != nil { 70 + return err 71 + } 72 + defer cleanup() 73 + 74 + for _, path := range args { 75 + f, err := os.Open(path) 76 + if err != nil { 77 + return fmt.Errorf("failed to open %s: %w", path, err) 78 + } 79 + 80 + result, err := holdPDS.ImportFromCAR(ctx, f) 81 + f.Close() 82 + if err != nil { 83 + return fmt.Errorf("failed to import %s: %w", path, err) 84 + } 85 + 86 + fmt.Fprintf(os.Stderr, "Imported %d records from %s\n", result.Total, path) 87 + for collection, count := range result.PerCollection { 88 + fmt.Fprintf(os.Stderr, " %s: %d\n", collection, count) 89 + } 90 + } 91 + 92 + return nil 93 + }, 94 + } 95 + 96 + var repoConfigFile string 97 + 98 + func init() { 99 + repoCmd.PersistentFlags().StringVarP(&repoConfigFile, "config", "c", "", "path to YAML configuration file") 100 + 101 + repoCmd.AddCommand(repoExportCmd) 102 + repoCmd.AddCommand(repoImportCmd) 103 + } 104 + 105 + // openHoldPDS creates a HoldPDS from config for offline CLI operations. 106 + // Returns the PDS and a cleanup function that must be deferred. 107 + func openHoldPDS(ctx context.Context, cfg *hold.Config) (*pds.HoldPDS, func(), error) { 108 + holdDID, err := pds.LoadOrCreateDID(ctx, pds.DIDConfig{ 109 + DID: cfg.Database.DID, 110 + DIDMethod: cfg.Database.DIDMethod, 111 + PublicURL: cfg.Server.PublicURL, 112 + DBPath: cfg.Database.Path, 113 + SigningKeyPath: cfg.Database.KeyPath, 114 + RotationKeyPath: cfg.Database.RotationKeyPath, 115 + PLCDirectoryURL: cfg.Database.PLCDirectoryURL, 116 + }) 117 + if err != nil { 118 + return nil, nil, fmt.Errorf("failed to resolve hold DID: %w", err) 119 + } 120 + slog.Info("Using hold DID", "did", holdDID) 121 + 122 + // Open shared database 123 + dbFilePath := cfg.Database.Path + "/db.sqlite3" 124 + libsqlCfg := holddb.LibsqlConfig{ 125 + SyncURL: cfg.Database.LibsqlSyncURL, 126 + AuthToken: cfg.Database.LibsqlAuthToken, 127 + SyncInterval: cfg.Database.LibsqlSyncInterval, 128 + } 129 + holdDB, err := holddb.OpenHoldDB(dbFilePath, libsqlCfg) 130 + if err != nil { 131 + return nil, nil, fmt.Errorf("failed to open hold database: %w", err) 132 + } 133 + 134 + holdPDS, err := pds.NewHoldPDSWithDB(ctx, holdDID, cfg.Server.PublicURL, cfg.Database.Path, cfg.Database.KeyPath, false, holdDB.DB) 135 + if err != nil { 136 + holdDB.Close() 137 + return nil, nil, fmt.Errorf("failed to initialize PDS: %w", err) 138 + } 139 + 140 + cleanup := func() { 141 + holdPDS.Close() 142 + holdDB.Close() 143 + } 144 + 145 + return holdPDS, cleanup, nil 146 + }

+8

config-hold.example.yaml

··· 69 69 path: /var/lib/atcr-hold 70 70 # PDS signing key path. Defaults to {database.path}/signing.key. 71 71 key_path: "" 72 + # DID method: 'web' (default, derived from public_url) or 'plc' (registered with PLC directory). 73 + did_method: web 74 + # Explicit DID for this hold. If set with did_method 'plc', adopts this identity instead of creating new. Use for recovery/migration. 75 + did: "" 76 + # PLC directory URL. Only used when did_method is 'plc'. Default: https://plc.directory 77 + plc_directory_url: https://plc.directory 78 + # Rotation key path for did:plc. Controls DID identity (separate from signing key). Defaults to {database.path}/rotation.key. 79 + rotation_key_path: "" 72 80 # libSQL sync URL (libsql://...). Works with Turso cloud, Bunny DB, or self-hosted libsql-server. Leave empty for local-only SQLite. 73 81 libsql_sync_url: "" 74 82 # Auth token for libSQL sync. Required if libsql_sync_url is set.

+4

deploy/upcloud/configs/hold.yaml.tmpl

··· 32 32 database: 33 33 path: "{{.BasePath}}" 34 34 key_path: "" 35 + did_method: web 36 + did: "" 37 + plc_directory_url: https://plc.directory 38 + rotation_key_path: "" 35 39 libsql_sync_url: "" 36 40 libsql_auth_token: "" 37 41 libsql_sync_interval: 1m0s

+161 -187

docs/appview.md

··· 6 6 7 7 **AppView** is the frontend server component of ATCR. It serves as the OCI-compliant registry API endpoint and web interface that Docker clients interact with when pushing and pulling container images. 8 8 9 - ### What AppView Does 10 - 11 9 AppView is the orchestration layer that: 12 10 13 11 - **Serves the OCI Distribution API V2** - Compatible with Docker, containerd, podman, and all OCI clients 14 12 - **Resolves ATProto identities** - Converts handles (`alice.bsky.social`) and DIDs (`did:plc:xyz123`) to PDS endpoints 15 13 - **Routes manifests** - Stores container image manifests as ATProto records in users' Personal Data Servers 16 14 - **Routes blobs** - Proxies blob (layer) operations to hold services for S3-compatible storage 17 - - **Provides web UI** - Browse repositories, search images, view tags, track pull counts, manage stars 18 - - **Manages authentication** - Validates OAuth tokens and issues registry JWTs to Docker clients 15 + - **Provides web UI** - Browse repositories, search images, view tags, track pull counts, manage stars, vulnerability scan results 16 + - **Manages authentication** - ATProto OAuth with device authorization flow, issues registry JWTs to Docker clients 19 17 20 18 ### The ATCR Ecosystem 21 19 22 20 AppView is the **frontend** of a multi-component architecture: 23 21 24 22 1. **AppView** (this component) - Registry API + web interface 25 - 2. **[Hold Service](https://atcr.io/r/evan.jarrett.net/atcr-hold)** - Storage backend with embedded PDS for blob storage 23 + 2. **[Hold Service](hold.md)** - Storage backend with embedded PDS for blob storage 26 24 3. **Credential Helper** - Client-side tool for ATProto OAuth authentication 27 25 28 26 **Data flow:** ··· 45 43 - Maintain full control over registry infrastructure 46 44 47 45 **Prerequisites:** 48 - - A running [Hold service](https://atcr.io/r/evan.jarrett.net/atcr-hold) (required for blob storage) 46 + - A running [Hold service](hold.md) (required for blob storage) 49 47 - (Optional) Domain name with SSL/TLS certificates for production 50 48 - (Optional) Access to ATProto Jetstream for real-time indexing 51 49 52 50 ## Quick Start 53 51 54 - ### Using Docker Compose 55 - 56 - The fastest way to run AppView alongside a Hold service: 52 + ### 1. Build the Docker image 57 53 58 54 ```bash 59 - # Clone repository 60 - git clone https://tangled.org/evan.jarrett.net/at-container-registry 61 - cd atcr 55 + docker build -t atcr-appview:latest -f Dockerfile.appview . 56 + ``` 62 57 63 - # Copy and configure environment 64 - cp .env.appview.example .env.appview 65 - # Edit .env.appview - set ATCR_DEFAULT_HOLD_DID (see Configuration below) 58 + This produces a ~30MB scratch image with a statically-linked binary. 66 59 67 - # Start services 68 - docker-compose up -d 60 + ### 2. Generate a config file 69 61 70 - # Verify 71 - curl http://localhost:5000/v2/ 62 + ```bash 63 + docker run --rm atcr-appview config init > config-appview.yaml 72 64 ``` 73 65 74 - ### Minimal Configuration 75 - 76 - At minimum, you must set: 66 + This creates a fully-commented YAML file with all available options and their defaults. You can also generate it from a local binary: 77 67 78 68 ```bash 79 - # Required: Default hold service for blob storage 80 - ATCR_DEFAULT_HOLD_DID=did:web:127.0.0.1:8080 81 - 82 - # Recommended for production 83 - ATCR_BASE_URL=https://registry.example.com 84 - ATCR_HTTP_ADDR=:5000 69 + ./bin/atcr-appview config init config-appview.yaml 85 70 ``` 86 71 87 - See **Configuration Reference** below for all options. 72 + ### 3. Set the required field 88 73 89 - ## Configuration Reference 74 + Edit `config-appview.yaml` and set `server.default_hold_did` to your hold service's DID: 90 75 91 - AppView is configured entirely via environment variables. Load them with: 92 - ```bash 93 - source .env.appview 94 - ./bin/atcr-appview serve 76 + ```yaml 77 + server: 78 + default_hold_did: "did:web:127.0.0.1:8080" # local dev 79 + # default_hold_did: "did:web:hold01.example.com" # production 95 80 ``` 96 81 97 - Or via Docker Compose (recommended). 82 + This is the **only required configuration field**. To find a hold's DID, visit its `/.well-known/did.json` endpoint. 98 83 99 - ### Server Configuration 84 + For production, also set your public URL: 100 85 101 - #### `ATCR_HTTP_ADDR` 102 - - **Default:** `:5000` 103 - - **Description:** HTTP listen address for the registry API and web UI 104 - - **Example:** `:5000`, `:8080`, `0.0.0.0:5000` 86 + ```yaml 87 + server: 88 + base_url: "https://registry.example.com" 89 + default_hold_did: "did:web:hold01.example.com" 90 + ``` 105 91 106 - #### `ATCR_BASE_URL` 107 - - **Default:** Auto-detected from `ATCR_HTTP_ADDR` (e.g., `http://127.0.0.1:5000`) 108 - - **Description:** Public URL for the AppView service. Used to generate OAuth redirect URIs and JWT realm claims. 109 - - **Development:** Auto-detection works fine (`http://127.0.0.1:5000`) 110 - - **Production:** Set to your public URL (e.g., `https://atcr.example.com`) 111 - - **Example:** `https://atcr.io`, `http://127.0.0.1:5000` 92 + ### 4. Run 112 93 113 - ### Storage Configuration 94 + ```bash 95 + docker run -d \ 96 + -v ./config-appview.yaml:/config.yaml:ro \ 97 + -v atcr-data:/var/lib/atcr \ 98 + -p 5000:5000 \ 99 + atcr-appview serve --config /config.yaml 100 + ``` 114 101 115 - #### `ATCR_DEFAULT_HOLD_DID` ⚠️ REQUIRED 116 - - **Default:** None (required) 117 - - **Description:** DID of the default hold service for blob storage. Used when users don't have their own hold configured in their sailor profile. AppView routes all blob operations to this hold. 118 - - **Format:** `did:web:hostname[:port]` 119 - - **Docker Compose:** `did:web:atcr-hold:8080` (internal Docker network) 120 - - **Local dev:** `did:web:127.0.0.1:8080` 121 - - **Production:** `did:web:hold01.atcr.io` 122 - - **Note:** This hold must be reachable from AppView. To find a hold's DID, visit `https://hold-url/.well-known/did.json` 102 + ### 5. Verify 123 103 124 - ### Authentication Configuration 104 + ```bash 105 + curl http://localhost:5000/v2/ 106 + # Should return: {} 125 107 126 - #### `ATCR_AUTH_KEY_PATH` 127 - - **Default:** `/var/lib/atcr/auth/private-key.pem` 128 - - **Description:** Path to JWT signing private key (RSA). Auto-generated if missing. 129 - - **Note:** Keep this secure - it signs all registry JWTs issued to Docker clients 108 + curl http://localhost:5000/health 109 + # Should return: {"status":"ok"} 110 + ``` 130 111 131 - #### `ATCR_AUTH_CERT_PATH` 132 - - **Default:** `/var/lib/atcr/auth/private-key.crt` 133 - - **Description:** Path to JWT signing certificate. Auto-generated if missing. 134 - - **Note:** Paired with `ATCR_AUTH_KEY_PATH` 112 + ## Configuration 135 113 136 - ### Web UI Configuration 114 + AppView uses YAML configuration with environment variable overrides. The generated `config-appview.yaml` is the canonical reference — every field is commented inline with its purpose and default value. 137 115 138 - #### `ATCR_UI_DATABASE_PATH` 139 - - **Default:** `/var/lib/atcr/ui.db` 140 - - **Description:** SQLite database path for UI data (OAuth sessions, stars, pull counts, repository metadata) 141 - - **Note:** For multi-instance deployments, use PostgreSQL (see production docs) 116 + ### Config loading priority (highest wins) 142 117 143 - ### Logging Configuration 118 + 1. Environment variables (`ATCR_` prefix) 119 + 2. YAML config file (`--config`) 120 + 3. Built-in defaults 144 121 145 - #### `ATCR_LOG_LEVEL` 146 - - **Default:** `info` 147 - - **Options:** `debug`, `info`, `warn`, `error` 148 - - **Description:** Log verbosity level 149 - - **Development:** Use `debug` for detailed troubleshooting 150 - - **Production:** Use `info` or `warn` 122 + ### Environment variable convention 151 123 152 - #### `ATCR_LOG_FORMATTER` 153 - - **Default:** `text` 154 - - **Options:** `text`, `json` 155 - - **Description:** Log output format 156 - - **Production:** Use `json` for structured logging (easier to parse with log aggregators) 124 + YAML paths map to env vars with `ATCR_` prefix and `_` separators: 157 125 158 - ### Hold Health Check Configuration 126 + ``` 127 + server.default_hold_did → ATCR_SERVER_DEFAULT_HOLD_DID 128 + server.base_url → ATCR_SERVER_BASE_URL 129 + ui.database_path → ATCR_UI_DATABASE_PATH 130 + jetstream.backfill_enabled → ATCR_JETSTREAM_BACKFILL_ENABLED 131 + ``` 159 132 160 - AppView periodically checks if hold services are reachable and caches results to display health indicators in the UI. 133 + ### Config sections overview 161 134 162 - #### `ATCR_HEALTH_CHECK_INTERVAL` 163 - - **Default:** `15m` 164 - - **Description:** How often to check health of hold endpoints in the background 165 - - **Format:** Duration string (e.g., `5m`, `15m`, `30m`, `1h`) 166 - - **Recommendation:** 15-30 minutes for production 135 + | Section | Purpose | Notes | 136 + |---------|---------|-------| 137 + | `server` | Listen address, public URL, hold DID, OAuth key, branding | Only `default_hold_did` is required | 138 + | `ui` | Database path, theme, libSQL sync | All have defaults; auto-creates DB on first run | 139 + | `auth` | JWT signing key/cert paths | Auto-generated on first run | 140 + | `jetstream` | Real-time ATProto event streaming, backfill sync | Runs automatically; backfill enabled by default | 141 + | `health` | Hold health check interval and cache TTL | Sensible defaults (15m) | 142 + | `log_shipper` | Remote log shipping (Victoria, OpenSearch, Loki) | Disabled by default | 143 + | `legal` | Terms/privacy page customization | Optional | 144 + | `credential_helper` | Credential helper download source | Optional | 167 145 168 - #### `ATCR_HEALTH_CACHE_TTL` 169 - - **Default:** `15m` 170 - - **Description:** How long to cache health check results before re-checking 171 - - **Format:** Duration string (e.g., `15m`, `30m`, `1h`) 172 - - **Note:** Should be >= `ATCR_HEALTH_CHECK_INTERVAL` for efficiency 146 + ### Auto-generated files 173 147 174 - ### Jetstream Configuration (ATProto Event Streaming) 148 + On first run, AppView auto-generates these under `/var/lib/atcr/`: 175 149 176 - Jetstream provides real-time indexing of ATProto records (manifests, tags) into the AppView database for the web UI. 150 + | File | Purpose | 151 + |------|---------| 152 + | `ui.db` | SQLite database (OAuth sessions, stars, pull counts, device approvals) | 153 + | `auth/private-key.pem` | RSA private key for signing registry JWTs | 154 + | `auth/private-key.crt` | X.509 certificate for JWT verification | 155 + | `oauth/client.key` | P-256 private key for OAuth client authentication | 177 156 178 - #### `JETSTREAM_URL` 179 - - **Default:** `wss://jetstream2.us-west.bsky.network/subscribe` 180 - - **Description:** Jetstream WebSocket URL for real-time ATProto events 181 - - **Note:** Connects to Bluesky's public Jetstream by default 157 + **Persist `/var/lib/atcr/` across restarts.** Losing the auth keys invalidates all active sessions; losing the database loses OAuth state and UI data. 182 158 183 - #### `ATCR_BACKFILL_ENABLED` 184 - - **Default:** `false` 185 - - **Description:** Enable periodic sync of historical ATProto records. Set to `true` for production to ensure database completeness. 186 - - **Recommendation:** Enable for production AppView instances 159 + ## Deployment 187 160 188 - #### `ATCR_RELAY_ENDPOINT` 189 - - **Default:** `https://relay1.us-east.bsky.network` 190 - - **Description:** ATProto relay endpoint for backfill sync API 191 - - **Note:** Used when `ATCR_BACKFILL_ENABLED=true` 161 + ### Docker (recommended) 192 162 193 - ### Legacy Configuration 163 + `Dockerfile.appview` builds a minimal scratch image (~30MB) containing: 164 + - Static `atcr-appview` binary (CGO-enabled with embedded SQLite) 165 + - `healthcheck` binary for container health checks 166 + - CA certificates and timezone data 194 167 195 - #### `TEST_MODE` 196 - - **Default:** `false` 197 - - **Description:** Enable test mode (skips some validations). Do not use in production. 168 + **Port:** `5000` (HTTP) 198 169 199 - ## Web Interface Features 170 + **Volume:** `/var/lib/atcr` (auth keys, database, OAuth keys) 200 171 201 - The AppView web UI provides: 172 + **Health check:** `GET /health` returns `{"status":"ok"}` 202 173 203 - - **Home page** - Featured repositories and recent pushes feed 204 - - **Repository pages** - View tags, manifests, pull instructions, health status 205 - - **Search** - Find repositories by owner handle or repository name 206 - - **User profiles** - View a user's repositories and activity 207 - - **Stars** - Favorite repositories (requires OAuth login) 208 - - **Pull counts** - Track image pull statistics 209 - - **Multi-arch support** - Display platform-specific manifests (linux/amd64, linux/arm64) 210 - - **Health indicators** - Real-time hold service reachability status 211 - - **Install scripts** - Host credential helper installation scripts at `/install.sh` 174 + ```bash 175 + docker run -d \ 176 + --name atcr-appview \ 177 + -v ./config-appview.yaml:/config.yaml:ro \ 178 + -v atcr-data:/var/lib/atcr \ 179 + -p 5000:5000 \ 180 + --health-cmd '/healthcheck http://localhost:5000/health' \ 181 + --health-interval 30s \ 182 + --restart unless-stopped \ 183 + atcr-appview serve --config /config.yaml 184 + ``` 212 185 213 - ## Deployment Scenarios 186 + ### Production with reverse proxy 214 187 215 - ### Public Registry (like atcr.io) 188 + AppView serves HTTP on port 5000. For production, put a reverse proxy in front for HTTPS termination. The repository includes a working Caddy + Docker Compose setup at [`deploy/docker-compose.prod.yml`](../deploy/docker-compose.prod.yml) that runs AppView, Hold, and Caddy together with automatic TLS. 216 189 217 - Open to all ATProto users: 190 + A minimal production compose override: 218 191 219 - ```bash 220 - # AppView config 221 - ATCR_BASE_URL=https://registry.example.com 222 - ATCR_DEFAULT_HOLD_DID=did:web:hold01.example.com 223 - ATCR_BACKFILL_ENABLED=true 192 + ```yaml 193 + services: 194 + atcr-appview: 195 + image: atcr-appview:latest 196 + command: ["serve", "--config", "/config.yaml"] 197 + environment: 198 + ATCR_SERVER_BASE_URL: https://registry.example.com 199 + ATCR_SERVER_DEFAULT_HOLD_DID: did:web:hold.example.com 200 + volumes: 201 + - ./config-appview.yaml:/config.yaml:ro 202 + - atcr-appview-data:/var/lib/atcr 203 + healthcheck: 204 + test: ["CMD", "/healthcheck", "http://localhost:5000/health"] 205 + interval: 30s 206 + timeout: 10s 207 + retries: 3 208 + start_period: 30s 224 209 225 - # Hold config (linked hold service) 226 - HOLD_PUBLIC=true # Allow public pulls 227 - HOLD_ALLOW_ALL_CREW=true # Allow all authenticated users to push 210 + volumes: 211 + atcr-appview-data: 228 212 ``` 229 213 230 - ### Private Organizational Registry 214 + ### Systemd (bare metal) 231 215 232 - Restricted to crew members only: 216 + For non-Docker deployments, see the systemd service templates in [`deploy/upcloud/`](../deploy/upcloud/) which include security hardening (dedicated user, filesystem protection, private tmp). 233 217 234 - ```bash 235 - # AppView config 236 - ATCR_BASE_URL=https://registry.internal.example.com 237 - ATCR_DEFAULT_HOLD_DID=did:web:hold.internal.example.com 218 + ## Deployment Scenarios 238 219 239 - # Hold config (linked hold service) 240 - HOLD_PUBLIC=false # Require auth for pulls 241 - HOLD_ALLOW_ALL_CREW=false # Only owner + explicit crew can push 242 - HOLD_OWNER=did:plc:your-org-did # Organization DID 243 - ``` 220 + ### Public Registry 244 221 245 - ### Development/Testing 222 + Open to all ATProto users: 246 223 247 - Local Docker Compose setup with Minio for S3-compatible storage: 224 + ```yaml 225 + # config-appview.yaml 226 + server: 227 + base_url: "https://registry.example.com" 228 + default_hold_did: "did:web:hold01.example.com" 229 + jetstream: 230 + backfill_enabled: true 231 + ``` 248 232 249 - ```bash 250 - # Start Minio (S3-compatible storage) 251 - docker run -p 9000:9000 -p 9001:9001 minio/minio server /data --console-address ":9001" 233 + The linked hold service should have `server.public: true` and `registration.allow_all_crew: true`. 252 234 253 - # AppView config 254 - ATCR_HTTP_ADDR=:5000 255 - ATCR_DEFAULT_HOLD_DID=did:web:atcr-hold:8080 256 - ATCR_LOG_LEVEL=debug 235 + ### Private Organizational Registry 257 236 258 - # Hold config (linked hold service) 259 - AWS_ACCESS_KEY_ID=minioadmin 260 - AWS_SECRET_ACCESS_KEY=minioadmin 261 - S3_BUCKET=test 262 - S3_ENDPOINT=http://minio:9000 263 - HOLD_PUBLIC=true 264 - HOLD_ALLOW_ALL_CREW=true 237 + Restricted to crew members only: 238 + 239 + ```yaml 240 + # config-appview.yaml 241 + server: 242 + base_url: "https://registry.internal.example.com" 243 + default_hold_did: "did:web:hold.internal.example.com" 265 244 ``` 266 245 267 - ## Production Deployment 246 + The linked hold service should have `server.public: false` and `registration.allow_all_crew: false`, with an explicit `registration.owner_did` set to the organization's DID. 268 247 269 - For production deployments with: 270 - - Multiple AppView instances (load balancing) 271 - - PostgreSQL database (instead of SQLite) 272 - - SSL/TLS certificates 273 - - Systemd service files 274 - - Log rotation 275 - - Monitoring 248 + ### Local Development 276 249 277 - See **[deploy/README.md](https://tangled.org/evan.jarrett.net/at-container-registry/blob/main/deploy/README.md)** for comprehensive production deployment guide. 250 + ```yaml 251 + # config-appview.yaml 252 + log_level: debug 253 + server: 254 + default_hold_did: "did:web:127.0.0.1:8080" 255 + test_mode: true # allows HTTP for DID resolution 256 + ``` 278 257 279 - ### Quick Production Checklist 258 + Run a hold service locally with Minio for S3-compatible storage. See [hold.md](hold.md) for hold setup. 280 259 281 - Before going to production: 260 + ## Web Interface 282 261 283 - - [ ] Set `ATCR_BASE_URL` to your public HTTPS URL 284 - - [ ] Set `ATCR_DEFAULT_HOLD_DID` to a production hold service 285 - - [ ] Enable Jetstream backfill (`ATCR_BACKFILL_ENABLED=true`) 286 - - [ ] Use `ATCR_LOG_FORMATTER=json` for structured logging 287 - - [ ] Secure JWT keys (`ATCR_AUTH_KEY_PATH`, `ATCR_AUTH_CERT_PATH`) 288 - - [ ] Configure SSL/TLS termination (nginx/Caddy/Cloudflare) 289 - - [ ] Set up database backups (if using SQLite, consider PostgreSQL) 290 - - [ ] Monitor hold health checks 291 - - [ ] Test OAuth flow end-to-end 292 - - [ ] Verify Docker push/pull works 262 + The AppView web UI provides: 293 263 294 - ## Configuration Files Reference 295 - 296 - - **[.env.appview.example](https://tangled.org/evan.jarrett.net/at-container-registry/blob/main/.env.appview.example)** - All available environment variables with documentation 297 - - **[deploy/.env.prod.template](https://tangled.org/evan.jarrett.net/at-container-registry/blob/main/deploy/.env.prod.template)** - Production configuration template 298 - - **[deploy/README.md](https://tangled.org/evan.jarrett.net/at-container-registry/blob/main/deploy/README.md)** - Production deployment guide 299 - - **[Hold Service Documentation](https://atcr.io/r/evan.jarrett.net/atcr-hold)** - Storage backend setup 264 + - **Home page** - Featured repositories and recent pushes 265 + - **Repository pages** - Tags, manifests, pull instructions, health status, vulnerability scan results 266 + - **Search** - Find repositories by owner handle or repository name 267 + - **User profiles** - View a user's repositories and starred images 268 + - **Stars** - Favorite repositories (requires login) 269 + - **Pull counts** - Image pull statistics 270 + - **Multi-arch support** - Platform-specific manifests (linux/amd64, linux/arm64, etc.) 271 + - **Health indicators** - Real-time hold service reachability 272 + - **Device management** - Approve and revoke Docker credential helper pairings 273 + - **Settings** - Choose default hold, view crew memberships, storage usage

+122 -314

docs/hold.md

··· 1 1 # ATCR Hold Service 2 2 3 - > The storage backend component of ATCR (ATProto Container Registry) 4 - 5 - ## Overview 6 - 7 - **Hold Service** is the storage backend component of ATCR. It enables BYOS (Bring Your Own Storage) - users can store their own container image layers in their own S3-compatible storage (AWS S3, Storj, Minio, UpCloud, etc.). Each hold runs as a full ATProto user with an embedded PDS, exposing both standard ATProto sync endpoints and custom XRPC endpoints for OCI multipart blob uploads. 8 - 9 - ### What Hold Service Does 10 - 11 - Hold Service is the storage layer that: 12 - 13 - - **Bring Your Own Storage (BYOS)** - Store your own container image layers in your own S3-compatible storage (AWS S3, Storj, Minio, UpCloud, etc.) 14 - - **Embedded ATProto PDS** - Each hold is a full ATProto user with its own DID, repository, and identity 15 - - **Custom XRPC Endpoints** - OCI-compatible multipart upload endpoints (`io.atcr.hold.*`) for blob operations 16 - - **Presigned URL Generation** - Creates time-limited S3 URLs for direct client-to-storage transfers (~99% bandwidth reduction) 17 - - **Crew Management** - Controls access via captain and crew records stored in the hold's embedded PDS 18 - - **Standard ATProto Sync** - Exposes com.atproto.sync.* endpoints for repository synchronization and firehose 19 - - **S3 Storage** - Works with any S3-compatible storage (AWS S3, Storj, Minio, UpCloud, Azure, GCS via S3 gateway) 20 - - **Bluesky Integration** - Optional: Posts container image push notifications from the hold's identity to Bluesky 3 + Hold Service is the BYOS (Bring Your Own Storage) blob storage backend for ATCR. It stores container image layers in your own S3-compatible storage (AWS S3, Storj, Minio, UpCloud, etc.) and generates presigned URLs so clients transfer data directly to/from S3. Each hold runs an embedded ATProto PDS with its own DID, repository, and crew-based access control. 21 4 22 - ### The ATCR Ecosystem 5 + Hold Service is one component of the ATCR ecosystem: 23 6 24 - Hold Service is the **storage backend** of a multi-component architecture: 7 + 1. **[AppView](https://atcr.io/r/evan.jarrett.net/atcr-appview)** — Registry API + web interface 8 + 2. **Hold Service** (this component) — Storage backend with embedded PDS 9 + 3. **Credential Helper** — Client-side tool for ATProto OAuth authentication 25 10 26 - 1. **[AppView](https://atcr.io/r/evan.jarrett.net/atcr-appview)** - Registry API + web interface 27 - 2. **Hold Service** (this component) - Storage backend with embedded PDS 28 - 3. **Credential Helper** - Client-side tool for ATProto OAuth authentication 29 - 30 - **Data flow:** 31 11 ``` 32 - Docker Client → AppView (resolves identity) → User's PDS (stores manifest) 33 - ↓ 34 - Hold Service (generates presigned URL) 35 - ↓ 36 - S3/Storj/etc. (client uploads/downloads blobs directly) 12 + Docker Client --> AppView (resolves identity) --> User's PDS (stores manifest) 13 + | 14 + Hold Service (generates presigned URL) 15 + | 16 + S3/Storj/etc. (client uploads/downloads directly) 37 17 ``` 38 18 39 - Manifests (small JSON metadata) live in users' ATProto PDS, while blobs (large binary layers) live in hold services. AppView orchestrates the routing, and hold services provide presigned URLs to eliminate bandwidth bottlenecks. 19 + Manifests (small JSON metadata) live in users' ATProto PDS. Blobs (large binary layers) live in hold services. AppView orchestrates the routing. 40 20 41 21 ## When to Run Your Own Hold 42 22 43 - Most users can push to the default hold at **https://hold01.atcr.io** - you don't need to run your own hold. 23 + Most users can push to the default hold at **https://hold01.atcr.io** — you don't need to run your own. 44 24 45 - **Run your own hold if you want to:** 46 - - Control where your container layer data is stored (own S3 bucket, Storj, etc.) 25 + Run your own hold if you want to: 26 + - Control where your container layer data is stored (own S3 bucket, geographic region) 47 27 - Manage access for a team or organization via crew membership 48 - - Reduce bandwidth costs by using presigned URLs for direct S3 transfers 49 28 - Run a shared hold for a community or project 50 - - Maintain data sovereignty (keep blobs in specific geographic regions) 29 + - Use a CDN pull zone for faster downloads 51 30 52 - **Prerequisites:** 53 - - S3-compatible storage (AWS S3, Storj, Minio, UpCloud, etc.) 54 - - (Optional) Domain name with SSL/TLS certificates for production 55 - - ATProto DID for hold owner (get from: `https://bsky.social/xrpc/com.atproto.identity.resolveHandle?handle=yourhandle.bsky.social`) 31 + **Prerequisites:** S3-compatible storage with a bucket already created, and a domain with TLS for production. 56 32 57 33 ## Quick Start 58 34 59 - ### Using Docker Compose 60 - 61 - The fastest way to run Hold service with S3 storage: 35 + ### 1. Generate Configuration 62 36 63 37 ```bash 64 - # Clone repository 65 - git clone https://tangled.org/evan.jarrett.net/at-container-registry 66 - cd atcr 67 - 68 - # Copy and configure environment 69 - cp .env.hold.example .env.hold 70 - # Edit .env.hold - set HOLD_PUBLIC_URL, HOLD_OWNER, S3 credentials (see Configuration below) 71 - 72 - # Start hold service 73 - docker-compose -f docker-compose.hold.yml up -d 74 - 75 - # Verify 76 - curl http://localhost:8080/.well-known/did.json 77 - ``` 78 - 79 - ### Minimal Configuration 80 - 81 - At minimum, you must set: 82 - 83 - ```bash 84 - # Required: Public URL (generates did:web identity) 85 - HOLD_PUBLIC_URL=https://hold.example.com 86 - 87 - # Required: Your ATProto DID (for captain record) 88 - HOLD_OWNER=did:plc:your-did-here 89 - 90 - # Required: S3 credentials and bucket 91 - AWS_ACCESS_KEY_ID=your-access-key 92 - AWS_SECRET_ACCESS_KEY=your-secret-key 93 - S3_BUCKET=your-bucket-name 38 + # Build the hold binary 39 + go build -o bin/atcr-hold ./cmd/hold 94 40 95 - # Recommended: Database directory for embedded PDS 96 - HOLD_DATABASE_DIR=/var/lib/atcr-hold 41 + # Generate a fully-commented config file with all defaults 42 + ./bin/atcr-hold config init config-hold.yaml 97 43 ``` 98 44 99 - See **Configuration Reference** below for all options. 100 - 101 - ## Configuration Reference 45 + Or generate config from Docker without building locally: 102 46 103 - Hold Service is configured entirely via environment variables. Load them with: 104 47 ```bash 105 - source .env.hold 106 - ./bin/atcr-hold 48 + docker run --rm -i $(docker build -q -f Dockerfile.hold .) config init > config-hold.yaml 107 49 ``` 108 50 109 - Or via Docker Compose (recommended). 110 - 111 - ### Server Configuration 112 - 113 - #### `HOLD_PUBLIC_URL` ⚠️ REQUIRED 114 - - **Default:** None (required) 115 - - **Description:** Public URL of this hold service. Used to generate the hold's did:web identity. The hostname becomes the hold's DID. 116 - - **Format:** `https://hold.example.com` or `http://127.0.0.1:8080` (development) 117 - - **Example:** `https://hold01.atcr.io` → DID is `did:web:hold01.atcr.io` 118 - - **Note:** This URL must be reachable by AppView and Docker clients 51 + The generated file documents every option with inline comments. Edit only what you need. 119 52 120 - #### `HOLD_SERVER_ADDR` 121 - - **Default:** `:8080` 122 - - **Description:** HTTP listen address for XRPC endpoints 123 - - **Example:** `:8080`, `:9000`, `0.0.0.0:8080` 53 + ### 2. Minimal Configuration 124 54 125 - #### `HOLD_PUBLIC` 126 - - **Default:** `false` 127 - - **Description:** Allow public blob reads (pulls) without authentication. Writes always require crew membership. 128 - - **Use cases:** 129 - - `true`: Public registry (anyone can pull, authenticated users can push if crew) 130 - - `false`: Private registry (authentication required for both push and pull) 55 + Only three things need to be set — everything else has sensible defaults: 131 56 132 - ### S3 Storage Configuration 57 + ```yaml 58 + storage: 59 + access_key: "YOUR_S3_ACCESS_KEY" 60 + secret_key: "YOUR_S3_SECRET_KEY" 61 + bucket: "your-bucket-name" 62 + endpoint: "https://gateway.storjshare.io" # omit for AWS S3 133 63 134 - S3 is the only supported storage backend. Presigned URLs enable direct client-to-storage transfers (~99% bandwidth reduction). 64 + server: 65 + public_url: "https://hold.example.com" 135 66 136 - ##### `AWS_ACCESS_KEY_ID` ⚠️ REQUIRED for S3 137 - - **Description:** S3 access key ID for authentication 138 - - **Example:** `AKIAIOSFODNN7EXAMPLE` 67 + registration: 68 + owner_did: "did:plc:your-did-here" 69 + ``` 139 70 140 - ##### `AWS_SECRET_ACCESS_KEY` ⚠️ REQUIRED for S3 141 - - **Description:** S3 secret access key for authentication 142 - - **Example:** `wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY` 71 + - **`server.public_url`** — Your hold's public HTTPS URL. This becomes the hold's `did:web` identity. 72 + - **`storage.bucket`** — S3 bucket name (must already exist). 73 + - **`registration.owner_did`** — Your ATProto DID. Creates you as captain (admin) on first boot. Get yours from: `https://bsky.social/xrpc/com.atproto.identity.resolveHandle?handle=yourhandle.bsky.social` 143 74 144 - ##### `AWS_REGION` 145 - - **Default:** `us-east-1` 146 - - **Description:** S3 region 147 - - **AWS regions:** `us-east-1`, `us-west-2`, `eu-west-1`, etc. 148 - - **UpCloud regions:** `us-chi1`, `us-nyc1`, `de-fra1`, `uk-lon1`, `sg-sin1` 149 - 150 - ##### `S3_BUCKET` ⚠️ REQUIRED for S3 151 - - **Description:** S3 bucket name where blobs will be stored 152 - - **Example:** `atcr-blobs`, `my-company-registry-blobs` 153 - - **Note:** Bucket must already exist 154 - 155 - ##### `S3_ENDPOINT` 156 - - **Default:** None (uses AWS S3) 157 - - **Description:** S3-compatible endpoint URL for non-AWS providers 158 - - **Storj:** `https://gateway.storjshare.io` 159 - - **UpCloud:** `https://[bucket-id].upcloudobjects.com` 160 - - **Minio:** `http://minio:9000` 161 - - **Note:** Leave empty for AWS S3 162 - 163 - ### Embedded PDS Configuration 164 - 165 - #### `HOLD_DATABASE_DIR` 166 - - **Default:** `/var/lib/atcr-hold` 167 - - **Description:** Directory path for embedded PDS carstore (SQLite database). Carstore creates `db.sqlite3` inside this directory. 168 - - **Note:** This must be a directory path, NOT a file path. If empty, embedded PDS is disabled (not recommended - hold authorization requires PDS). 169 - 170 - #### `HOLD_KEY_PATH` 171 - - **Default:** `{HOLD_DATABASE_DIR}/signing.key` 172 - - **Description:** Path to hold's signing key (secp256k1). Auto-generated on first run if missing. 173 - - **Note:** Keep this secure - it's used to sign ATProto commits in the hold's repository 174 - 175 - ### Access Control 176 - 177 - #### `HOLD_OWNER` 178 - - **Default:** None 179 - - **Description:** Your ATProto DID. Used to create the captain record and add you as the first crew member with admin role. 180 - - **Get your DID:** `https://bsky.social/xrpc/com.atproto.identity.resolveHandle?handle=yourhandle.bsky.social` 181 - - **Example:** `did:plc:abc123xyz789` 182 - - **Note:** If set, the hold will initialize with your DID as owner on first run 183 - 184 - #### `HOLD_ALLOW_ALL_CREW` 185 - - **Default:** `false` 186 - - **Description:** Allow any authenticated ATCR user to write to this hold (treat all as crew) 187 - - **Security model:** 188 - - `true`: Any authenticated user can push images (useful for shared/community holds) 189 - - `false`: Only hold owner and explicit crew members can push (verified via crew records in hold's PDS) 190 - - **Use cases:** 191 - - Public registry: `HOLD_PUBLIC=true, HOLD_ALLOW_ALL_CREW=true` 192 - - ATProto users only: `HOLD_PUBLIC=false, HOLD_ALLOW_ALL_CREW=true` 193 - - Private hold: `HOLD_PUBLIC=false, HOLD_ALLOW_ALL_CREW=false` (default) 194 - 195 - ### Bluesky Integration 196 - 197 - #### `HOLD_BLUESKY_POSTS_ENABLED` 198 - - **Default:** `false` 199 - - **Description:** Create Bluesky posts when users push container images. Posts include image name, tag, size, and layer count. 200 - - **Note:** Posts are created from the hold's embedded PDS identity (did:web). Requires hold to be crawled by Bluesky relay. 201 - - **Enable relay crawl:** `./deploy/request-crawl.sh hold.example.com` 202 - 203 - #### `HOLD_PROFILE_AVATAR` 204 - - **Default:** `https://imgs.blue/evan.jarrett.net/1TpTOdtS60GdJWBYEqtK22y688jajbQ9a5kbYRFtwuqrkBAE` 205 - - **Description:** URL to download avatar image for hold's Bluesky profile. Downloaded and uploaded as blob during bootstrap. 206 - - **Note:** Avatar is stored in hold's PDS and displayed on Bluesky profile 207 - 208 - ### Advanced Configuration 209 - 210 - #### `TEST_MODE` 211 - - **Default:** `false` 212 - - **Description:** Enable test mode (skips some validations). Do not use in production. 75 + ### 3. Build and Run with Docker 213 76 214 - ## XRPC Endpoints 77 + ```bash 78 + # Build the image 79 + docker build -f Dockerfile.hold -t atcr-hold:latest . 215 80 216 - Hold Service exposes two types of XRPC endpoints: 81 + # Run it 82 + docker run -d \ 83 + --name atcr-hold \ 84 + -p 8080:8080 \ 85 + -v $(pwd)/config-hold.yaml:/config.yaml:ro \ 86 + -v atcr-hold-data:/var/lib/atcr-hold \ 87 + atcr-hold:latest serve --config /config.yaml 88 + ``` 217 89 218 - ### ATProto Sync Endpoints (Standard) 219 - - `GET /.well-known/did.json` - DID document (did:web resolution) 220 - - `GET /xrpc/com.atproto.sync.getRepo` - Download full repository as CAR file 221 - - `GET /xrpc/com.atproto.sync.getBlob` - Get blob or presigned download URL 222 - - `GET /xrpc/com.atproto.sync.subscribeRepos` - WebSocket firehose for real-time events 223 - - `GET /xrpc/com.atproto.sync.listRepos` - List all repositories (single-user PDS) 224 - - `GET /xrpc/com.atproto.repo.describeRepo` - Repository metadata 225 - - `GET /xrpc/com.atproto.repo.getRecord` - Get record by collection and rkey 226 - - `GET /xrpc/com.atproto.repo.listRecords` - List records in collection 227 - - `POST /xrpc/com.atproto.repo.deleteRecord` - Delete record (owner/crew admin only) 90 + - **`/var/lib/atcr-hold`** — Persistent volume for the embedded PDS (carstore database + signing keys). Back this up. 91 + - **Port 8080** — Default listen address. Put a reverse proxy (Caddy, nginx) in front for TLS. 92 + - The image is built `FROM scratch` — the binary includes SQLite statically linked. 93 + - Optional: `docker build --build-arg BILLING_ENABLED=true` to include Stripe billing support. 228 94 229 - ### OCI Multipart Upload Endpoints (Custom) 230 - - `POST /xrpc/io.atcr.hold.initiateUpload` - Start multipart upload session 231 - - `POST /xrpc/io.atcr.hold.getPartUploadUrl` - Get presigned URL for uploading a part 232 - - `POST /xrpc/io.atcr.hold.completeUpload` - Finalize multipart upload 233 - - `POST /xrpc/io.atcr.hold.abortUpload` - Cancel multipart upload 234 - - `POST /xrpc/io.atcr.hold.notifyManifest` - Notify hold of manifest upload (creates layer records, Bluesky posts) 95 + ## Configuration 235 96 236 - ## Authorization Model 97 + Config loads in layers: **defaults → YAML file → environment variables**. Later layers override earlier ones. 237 98 238 - Hold Service uses crew membership records in its embedded PDS for access control: 99 + All YAML fields can be overridden with environment variables using the `HOLD_` prefix and `_` path separators. For example, `server.public_url` becomes `HOLD_SERVER_PUBLIC_URL`. 239 100 240 - ### Read Access (Blob Downloads) 101 + S3 credentials also accept standard AWS environment variable names: `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`, `AWS_REGION`, `S3_BUCKET`, `S3_ENDPOINT`. 241 102 242 - **Public Hold** (`HOLD_PUBLIC=true`): 243 - - Anonymous users: ✅ Allowed 244 - - Authenticated users: ✅ Allowed 103 + For the complete configuration reference with all options and defaults, see [`config-hold.example.yaml`](../config-hold.example.yaml) or run `atcr-hold config init`. 245 104 246 - **Private Hold** (`HOLD_PUBLIC=false`): 247 - - Anonymous users: ❌ Forbidden 248 - - Authenticated users with crew membership: ✅ Allowed 249 - - Crew must have `blob:read` permission 105 + ## Access Control 250 106 251 - ### Write Access (Blob Uploads) 107 + | Setting | Who can pull | Who can push | 108 + |---|---|---| 109 + | `server.public: true` | Anyone | Captain + crew with `blob:write` | 110 + | `server.public: false` (default) | Crew with `blob:read` | Captain + crew with `blob:write` | 111 + | + `registration.allow_all_crew: true` | (per above) | Any authenticated user | 252 112 253 - Regardless of `HOLD_PUBLIC` setting: 254 - - Hold owner (from captain record): ✅ Allowed 255 - - Crew members with `blob:write` permission: ✅ Allowed 256 - - Non-crew authenticated users: Depends on `HOLD_ALLOW_ALL_CREW` 257 - - `HOLD_ALLOW_ALL_CREW=true`: ✅ Allowed 258 - - `HOLD_ALLOW_ALL_CREW=false`: ❌ Forbidden 113 + The captain (set via `registration.owner_did`) has all permissions implicitly. `blob:write` implies `blob:read`. 259 114 260 - ### Authentication Method 115 + Authentication uses ATProto service tokens: AppView requests a token from the user's PDS scoped to the hold's DID, then includes it in XRPC requests. The hold validates the token and checks crew membership. 261 116 262 - AppView uses **service tokens** from user's PDS to authenticate with hold service: 263 - 1. AppView calls user's PDS: `com.atproto.server.getServiceAuth` with hold DID 264 - 2. User's PDS returns a service token scoped to the hold DID 265 - 3. AppView includes service token in XRPC requests to hold 266 - 4. Hold validates token and checks crew membership in its embedded PDS 117 + See [BYOS.md](BYOS.md) for the full authorization model. 267 118 268 - ## Deployment Scenarios 119 + ## Optional Subsystems 269 120 270 - ### Personal Hold (Single User) 121 + | Subsystem | Default | Config key | Notes | 122 + |---|---|---|---| 123 + | Admin panel | Enabled | `admin.enabled` | Web UI for crew, settings, and storage management | 124 + | Quotas | Disabled | `quota.tiers` | Tier-based storage limits (e.g., deckhand=5GB, bosun=50GB) | 125 + | Garbage collection | Disabled | `gc.enabled` | Nightly cleanup of orphaned blobs and records | 126 + | Vulnerability scanner | Disabled | `scanner.secret` | Requires separate scanner service; see [SBOM_SCANNING.md](SBOM_SCANNING.md) | 127 + | Billing (Stripe) | Disabled | Build flag + env | Build with `--build-arg BILLING_ENABLED=true`; see [BILLING.md](BILLING.md) | 128 + | Bluesky posts | Disabled | `registration.enable_bluesky_posts` | Posts push notifications from hold's identity | 271 129 272 - Your own storage for your images: 130 + ## Hold Identity 273 131 274 - ```bash 275 - # Hold config 276 - HOLD_PUBLIC_URL=https://hold.alice.com 277 - HOLD_OWNER=did:plc:alice-did 278 - HOLD_PUBLIC=false # Private (only you can pull) 279 - HOLD_ALLOW_ALL_CREW=false # Only you can push 280 - HOLD_DATABASE_DIR=/var/lib/atcr-hold 132 + **did:web (default)** — Derived from `server.public_url` with zero setup. `https://hold.example.com` becomes `did:web:hold.example.com`. The DID document is served at `/.well-known/did.json`. Tied to domain ownership — if you lose the domain, you lose the identity. 281 133 282 - # S3 storage (using Storj) 283 - AWS_ACCESS_KEY_ID=your-key 284 - AWS_SECRET_ACCESS_KEY=your-secret 285 - S3_BUCKET=alice-container-registry 286 - S3_ENDPOINT=https://gateway.storjshare.io 287 - ``` 134 + **did:plc (portable)** — Set `database.did_method: plc` in config. Registered with plc.directory. Survives domain changes. Requires a rotation key (auto-generated at `{database.path}/rotation.key`). Use `database.did` to adopt an existing DID for recovery or migration. 288 135 289 - ### Shared Hold (Team/Organization) 136 + ## Verification 290 137 291 - Shared storage for a team with crew members: 138 + After starting your hold, verify it's working: 292 139 293 140 ```bash 294 - # Hold config 295 - HOLD_PUBLIC_URL=https://hold.acme.corp 296 - HOLD_OWNER=did:plc:acme-org-did 297 - HOLD_PUBLIC=false # Private reads (crew only) 298 - HOLD_ALLOW_ALL_CREW=false # Explicit crew membership required 299 - HOLD_DATABASE_DIR=/var/lib/atcr-hold 300 - 301 - # S3 storage 302 - AWS_ACCESS_KEY_ID=your-key 303 - AWS_SECRET_ACCESS_KEY=your-secret 304 - S3_BUCKET=acme-registry-blobs 305 - ``` 306 - 307 - Then add crew members via XRPC or hold PDS records. 141 + # Health check — should return {"version":"..."} 142 + curl https://hold.example.com/xrpc/_health 308 143 309 - ### Public Hold (Community Registry) 310 - 311 - Open storage allowing anyone to push and pull: 144 + # DID document — should return valid JSON with service endpoints 145 + curl https://hold.example.com/.well-known/did.json 312 146 313 - ```bash 314 - # Hold config 315 - HOLD_PUBLIC_URL=https://hold.community.io 316 - HOLD_OWNER=did:plc:community-did 317 - HOLD_PUBLIC=true # Public reads (anyone can pull) 318 - HOLD_ALLOW_ALL_CREW=true # Any authenticated user can push 319 - HOLD_DATABASE_DIR=/var/lib/atcr-hold 147 + # Captain record — should show your owner DID 148 + curl "https://hold.example.com/xrpc/com.atproto.repo.listRecords?repo=HOLD_DID&collection=io.atcr.hold.captain" 320 149 321 - # S3 storage 322 - AWS_ACCESS_KEY_ID=your-key 323 - AWS_SECRET_ACCESS_KEY=your-secret 324 - S3_BUCKET=community-registry-blobs 150 + # Crew records 151 + curl "https://hold.example.com/xrpc/com.atproto.repo.listRecords?repo=HOLD_DID&collection=io.atcr.hold.crew" 325 152 ``` 326 153 327 - ### Development/Testing with Minio 154 + Replace `HOLD_DID` with your hold's DID (from the `/.well-known/did.json` response). 328 155 329 - For local development, use Minio as an S3-compatible storage: 156 + ## Docker Compose 330 157 331 - ```bash 332 - # Start Minio 333 - docker run -p 9000:9000 -p 9001:9001 minio/minio server /data --console-address ":9001" 158 + ```yaml 159 + services: 160 + atcr-hold: 161 + build: 162 + context: . 163 + dockerfile: Dockerfile.hold 164 + command: ["serve", "--config", "/config.yaml"] 165 + volumes: 166 + - ./config-hold.yaml:/config.yaml:ro 167 + - atcr-hold-data:/var/lib/atcr-hold 168 + ports: 169 + - "8080:8080" 170 + healthcheck: 171 + test: ["CMD", "/healthcheck", "http://localhost:8080/xrpc/_health"] 172 + interval: 30s 173 + timeout: 10s 174 + retries: 3 175 + start_period: 30s 334 176 335 - # Hold config 336 - HOLD_PUBLIC_URL=http://127.0.0.1:8080 337 - HOLD_OWNER=did:plc:your-test-did 338 - HOLD_PUBLIC=true 339 - HOLD_ALLOW_ALL_CREW=true 340 - HOLD_DATABASE_DIR=/tmp/atcr-hold 341 - 342 - # Minio S3 storage 343 - AWS_ACCESS_KEY_ID=minioadmin 344 - AWS_SECRET_ACCESS_KEY=minioadmin 345 - S3_BUCKET=test 346 - S3_ENDPOINT=http://localhost:9000 177 + volumes: 178 + atcr-hold-data: 347 179 ``` 348 180 349 - ## Production Deployment 350 - 351 - For production deployments with: 352 - - SSL/TLS certificates 353 - - S3 storage with presigned URLs 354 - - Proper access control 355 - - Systemd service files 356 - - Monitoring 357 - 358 - See **[deploy/README.md](https://tangled.org/evan.jarrett.net/at-container-registry/blob/main/deploy/README.md)** for comprehensive production deployment guide. 359 - 360 - ### Quick Production Checklist 361 - 362 - Before going to production: 181 + For production with TLS termination, see [`deploy/docker-compose.prod.yml`](../deploy/docker-compose.prod.yml) which includes a Caddy reverse proxy. 363 182 364 - - [ ] Set `HOLD_PUBLIC_URL` to your public HTTPS URL 365 - - [ ] Set `HOLD_OWNER` to your ATProto DID 366 - - [ ] Set `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`, `S3_BUCKET`, `S3_ENDPOINT` 367 - - [ ] Set `HOLD_DATABASE_DIR` to persistent directory 368 - - [ ] Configure `HOLD_PUBLIC` and `HOLD_ALLOW_ALL_CREW` for desired access model 369 - - [ ] Configure SSL/TLS termination (Caddy/nginx/Cloudflare) 370 - - [ ] Verify DID document: `curl https://hold.example.com/.well-known/did.json` 371 - - [ ] Test presigned URLs: Check logs for "presigned URL" messages during push 372 - - [ ] Monitor crew membership: `curl https://hold.example.com/xrpc/com.atproto.repo.listRecords?repo={holdDID}&collection=io.atcr.hold.crew` 373 - - [ ] (Optional) Enable Bluesky posts: `HOLD_BLUESKY_POSTS_ENABLED=true` 374 - - [ ] (Optional) Request relay crawl: `./deploy/request-crawl.sh hold.example.com` 183 + ## Further Reading 375 184 376 - ## Configuration Files Reference 377 - 378 - - **[.env.hold.example](https://tangled.org/evan.jarrett.net/at-container-registry/blob/main/.env.hold.example)** - All available environment variables with documentation 379 - - **[deploy/.env.prod.template](https://tangled.org/evan.jarrett.net/at-container-registry/blob/main/deploy/.env.prod.template)** - Production configuration template (includes both AppView and Hold) 380 - - **[deploy/README.md](https://tangled.org/evan.jarrett.net/at-container-registry/blob/main/deploy/README.md)** - Production deployment guide 381 - - **[AppView Documentation](https://atcr.io/r/evan.jarrett.net/atcr-appview)** - Registry API server setup 382 - - **[BYOS Architecture](https://tangled.org/evan.jarrett.net/at-container-registry/blob/main/docs/BYOS.md)** - Bring Your Own Storage technical design 185 + - [`config-hold.example.yaml`](../config-hold.example.yaml) — Complete configuration reference with inline comments 186 + - [BYOS.md](BYOS.md) — Bring Your Own Storage architecture and authorization model 187 + - [HOLD_XRPC_ENDPOINTS.md](HOLD_XRPC_ENDPOINTS.md) — XRPC endpoint reference 188 + - [BILLING.md](BILLING.md) — Stripe billing integration 189 + - [QUOTAS.md](QUOTAS.md) — Quota management 190 + - [SBOM_SCANNING.md](SBOM_SCANNING.md) — Vulnerability scanning

+1

go.mod

··· 9 9 github.com/aws/aws-sdk-go-v2/credentials v1.19.7 10 10 github.com/aws/aws-sdk-go-v2/service/s3 v1.96.0 11 11 github.com/bluesky-social/indigo v0.0.0-20260213003059-85cdd0d6871c 12 + github.com/did-method-plc/go-didplc v0.0.0-20251009212921-7b7a252b8019 12 13 github.com/distribution/distribution/v3 v3.0.0 13 14 github.com/distribution/reference v0.6.0 14 15 github.com/earthboundkid/versioninfo/v2 v2.24.1

+2

go.sum

··· 84 84 github.com/decred/dcrd/dcrec/secp256k1/v4 v4.4.0/go.mod h1:ZXNYxsqcloTdSy/rNShjYzMhyjf0LaoftYK0p+A3h40= 85 85 github.com/dgryski/go-rendezvous v0.0.0-20200823014737-9f7001d12a5f h1:lO4WD4F/rVNCu3HqELle0jiPLLBs70cWOduZpkS1E78= 86 86 github.com/dgryski/go-rendezvous v0.0.0-20200823014737-9f7001d12a5f/go.mod h1:cuUVRXasLTGF7a8hSLbxyZXjz+1KgoB3wDUb6vlszIc= 87 + github.com/did-method-plc/go-didplc v0.0.0-20251009212921-7b7a252b8019 h1:MhDee1P3Zar8u72U6RtOKvzSd7dBAU3l2hhrOLQsfB0= 88 + github.com/did-method-plc/go-didplc v0.0.0-20251009212921-7b7a252b8019/go.mod h1:dBm0+R8Diqo90As3Q6p2wXAdrGXJgPEWBKUnpV5SUzI= 87 89 github.com/distribution/distribution/v3 v3.0.0 h1:q4R8wemdRQDClzoNNStftB2ZAfqOiN6UX90KJc4HjyM= 88 90 github.com/distribution/distribution/v3 v3.0.0/go.mod h1:tRNuFoZsUdyRVegq8xGNeds4KLjwLCRin/tTo6i1DhU= 89 91 github.com/distribution/reference v0.6.0 h1:0IXCQ5g4/QMHHkarYzh5l+u8T3t73zM5QvfrDyIgxBk=

+1

pkg/appview/jetstream/processor_test.go

··· 75 75 size INTEGER NOT NULL, 76 76 media_type TEXT NOT NULL, 77 77 layer_index INTEGER NOT NULL, 78 + annotations TEXT, 78 79 PRIMARY KEY(manifest_id, layer_index) 79 80 ); 80 81

+1 -1

pkg/hold/admin/admin.go

··· 421 421 r.Get("/admin/api/relay/status", ui.handleRelayStatus) 422 422 423 423 // Logout 424 - r.Get("/admin/auth/logout", ui.handleLogout) 424 + r.Post("/admin/auth/logout", ui.handleLogout) 425 425 }) 426 426 } 427 427

+1 -1

pkg/hold/admin/auth.go

··· 13 13 // Get session cookie 14 14 token, ok := getSessionCookie(r) 15 15 if !ok { 16 - http.Redirect(w, r, "/admin/auth/login?return_to="+r.URL.Path, http.StatusFound) 16 + http.Redirect(w, r, "/admin/auth/login", http.StatusFound) 17 17 return 18 18 } 19 19

+2 -9

pkg/hold/admin/handlers_auth.go

··· 21 21 } 22 22 } 23 23 24 - returnTo := r.URL.Query().Get("return_to") 25 - if returnTo == "" { 26 - returnTo = "/admin" 27 - } 28 - 29 24 data := struct { 30 25 PageData 31 - ReturnTo string 32 - Error string 26 + Error string 33 27 }{ 34 28 PageData: PageData{ 35 29 Title: "Login", 36 30 ActivePage: "login", 37 31 HoldDID: ui.pds.DID(), 38 32 }, 39 - ReturnTo: returnTo, 40 - Error: r.URL.Query().Get("error"), 33 + Error: r.URL.Query().Get("error"), 41 34 } 42 35 43 36 ui.renderTemplate(w, "pages/login.html", data)

+3 -1

pkg/hold/admin/templates/components/nav.html

··· 8 8 <div class="flex items-center gap-3"> 9 9 {{template "admin-theme-toggle"}} 10 10 <span class="text-sm opacity-80">{{.User.Handle}}</span> 11 - <a href="/admin/auth/logout" class="btn btn-sm btn-ghost">Logout</a> 11 + <form method="POST" action="/admin/auth/logout" class="inline"> 12 + <button type="submit" class="btn btn-sm btn-ghost">Logout</button> 13 + </form> 12 14 </div> 13 15 {{end}} 14 16 </div>

-2

pkg/hold/admin/templates/pages/login.html

···

+25 -1

pkg/hold/config.go

··· 143 143 // PDS signing key path. 144 144 KeyPath string `yaml:"key_path" comment:"PDS signing key path. Defaults to {database.path}/signing.key."` 145 145 146 + // DID method for hold identity: "web" (default) or "plc". 147 + DIDMethod string `yaml:"did_method" comment:"DID method: 'web' (default, derived from public_url) or 'plc' (registered with PLC directory)."` 148 + 149 + // Explicit DID for this hold. Used for recovery/migration with did:plc. 150 + DID string `yaml:"did" comment:"Explicit DID for this hold. If set with did_method 'plc', adopts this identity instead of creating new. Use for recovery/migration."` 151 + 152 + // PLC directory URL. Only used when did_method is "plc". 153 + PLCDirectoryURL string `yaml:"plc_directory_url" comment:"PLC directory URL. Only used when did_method is 'plc'. Default: https://plc.directory"` 154 + 155 + // Rotation key path for did:plc. Separate from signing key for recovery. 156 + RotationKeyPath string `yaml:"rotation_key_path" comment:"Rotation key path for did:plc. Controls DID identity (separate from signing key). Defaults to {database.path}/rotation.key."` 157 + 146 158 // libSQL sync URL for embedded replica mode. 147 159 LibsqlSyncURL string `yaml:"libsql_sync_url" comment:"libSQL sync URL (libsql://...). Works with Turso cloud, Bunny DB, or self-hosted libsql-server. Leave empty for local-only SQLite."` 148 160 ··· 177 189 // Database defaults 178 190 v.SetDefault("database.path", "/var/lib/atcr-hold") 179 191 v.SetDefault("database.key_path", "") 192 + v.SetDefault("database.did_method", "web") 193 + v.SetDefault("database.did", "") 194 + v.SetDefault("database.plc_directory_url", "https://plc.directory") 195 + v.SetDefault("database.rotation_key_path", "") 180 196 v.SetDefault("database.libsql_sync_url", "") 181 197 v.SetDefault("database.libsql_auth_token", "") 182 198 v.SetDefault("database.libsql_sync_interval", "60s") ··· 267 283 return nil, fmt.Errorf("storage.bucket is required (env: S3_BUCKET) - S3 is the only supported storage backend") 268 284 } 269 285 270 - // Post-load: derive key path from database path if not set 286 + // Post-load: derive key paths from database path if not set 271 287 if cfg.Database.KeyPath == "" && cfg.Database.Path != "" { 272 288 cfg.Database.KeyPath = filepath.Join(cfg.Database.Path, "signing.key") 289 + } 290 + if cfg.Database.RotationKeyPath == "" && cfg.Database.Path != "" { 291 + cfg.Database.RotationKeyPath = filepath.Join(cfg.Database.Path, "rotation.key") 292 + } 293 + 294 + // Validate DID method 295 + if cfg.Database.DIDMethod != "" && cfg.Database.DIDMethod != "web" && cfg.Database.DIDMethod != "plc" { 296 + return nil, fmt.Errorf("database.did_method must be 'web' or 'plc', got %q", cfg.Database.DIDMethod) 273 297 } 274 298 275 299 // Store config path for subsystem config loading (e.g. billing)

+298 -12

pkg/hold/pds/did.go

··· 1 1 package pds 2 2 3 3 import ( 4 + "context" 4 5 "encoding/json" 5 6 "fmt" 7 + "log/slog" 6 8 "net/url" 9 + "os" 10 + "path/filepath" 11 + "strings" 12 + 13 + "atcr.io/pkg/auth/oauth" 14 + "github.com/bluesky-social/indigo/atproto/atcrypto" 15 + didplc "github.com/did-method-plc/go-didplc" 7 16 ) 8 17 9 18 // DIDDocument represents a did:web document ··· 32 41 ServiceEndpoint string `json:"serviceEndpoint"` 33 42 } 34 43 35 - // GenerateDIDDocument creates a DID document for a did:web identity 44 + // GenerateDIDDocument creates a DID document for the hold's identity. 45 + // It uses the hold's stored DID (which may be did:web or did:plc). 36 46 func (p *HoldPDS) GenerateDIDDocument(publicURL string) (*DIDDocument, error) { 37 - // Parse URL to extract host and port 47 + did := p.did 48 + 49 + // Parse URL for alsoKnownAs 38 50 u, err := url.Parse(publicURL) 39 51 if err != nil { 40 52 return nil, fmt.Errorf("failed to parse public URL: %w", err) 41 53 } 42 - 43 - hostname := u.Hostname() 44 - port := u.Port() 45 - 46 - // Build host string (include non-standard ports per did:web spec) 47 - host := hostname 48 - if port != "" && port != "80" && port != "443" { 49 - host = fmt.Sprintf("%s:%s", hostname, port) 54 + host := u.Hostname() 55 + if port := u.Port(); port != "" && port != "80" && port != "443" { 56 + host = fmt.Sprintf("%s:%s", host, port) 50 57 } 51 - 52 - did := fmt.Sprintf("did:web:%s", host) 53 58 54 59 // Get public key in multibase format using indigo's crypto 55 60 pubKey, err := p.signingKey.PublicKey() ··· 104 109 } 105 110 106 111 return json.MarshalIndent(doc, "", " ") 112 + } 113 + 114 + // DIDConfig holds parameters for DID creation/loading. 115 + type DIDConfig struct { 116 + DID string // Explicit DID for adoption/recovery (optional) 117 + DIDMethod string // "web" or "plc" 118 + PublicURL string 119 + DBPath string 120 + SigningKeyPath string 121 + RotationKeyPath string 122 + PLCDirectoryURL string 123 + } 124 + 125 + // LoadOrCreateDID returns the hold's DID, either by deriving it from the URL (did:web) 126 + // or by loading/creating a did:plc identity registered with the PLC directory. 127 + // 128 + // For did:plc, the priority is: config DID > did.txt > create new. 129 + // When an existing DID is found (config or did.txt), EnsurePLCCurrent is called 130 + // to auto-update the PLC directory if the signing key or URL has changed. 131 + func LoadOrCreateDID(ctx context.Context, cfg DIDConfig) (string, error) { 132 + if cfg.DIDMethod != "plc" { 133 + return GenerateDIDFromURL(cfg.PublicURL), nil 134 + } 135 + 136 + didPath := filepath.Join(cfg.DBPath, "did.txt") 137 + 138 + // Priority: config DID > did.txt > create new 139 + var did string 140 + if cfg.DID != "" { 141 + if !strings.HasPrefix(cfg.DID, "did:plc:") { 142 + return "", fmt.Errorf("database.did must be a did:plc identifier, got %q", cfg.DID) 143 + } 144 + did = cfg.DID 145 + slog.Info("Using DID from config (adoption/recovery)", "did", did) 146 + } else if data, err := os.ReadFile(didPath); err == nil { 147 + d := strings.TrimSpace(string(data)) 148 + if strings.HasPrefix(d, "did:plc:") { 149 + did = d 150 + slog.Info("Loaded existing did:plc identity", "did", did) 151 + } 152 + } 153 + 154 + if did != "" { 155 + // Persist to did.txt (may be from config on first adoption) 156 + if err := os.MkdirAll(filepath.Dir(didPath), 0755); err != nil { 157 + return "", fmt.Errorf("failed to create directory for did.txt: %w", err) 158 + } 159 + if err := os.WriteFile(didPath, []byte(did+"\n"), 0600); err != nil { 160 + return "", fmt.Errorf("failed to write did.txt: %w", err) 161 + } 162 + 163 + // Load signing key (generate if missing — recovery case) 164 + signingKey, err := oauth.GenerateOrLoadPDSKey(cfg.SigningKeyPath) 165 + if err != nil { 166 + return "", fmt.Errorf("failed to load signing key: %w", err) 167 + } 168 + 169 + // Try to load rotation key (optional — may be stored offline) 170 + rotationKey, _ := loadOptionalK256Key(cfg.RotationKeyPath) 171 + 172 + if err := EnsurePLCCurrent(ctx, did, rotationKey, signingKey, cfg.PublicURL, cfg.PLCDirectoryURL); err != nil { 173 + return "", fmt.Errorf("failed to ensure PLC identity is current: %w", err) 174 + } 175 + 176 + return did, nil 177 + } 178 + 179 + // No existing DID — create new genesis operation 180 + slog.Info("Creating new did:plc identity") 181 + 182 + // Load or generate signing key 183 + signingKey, err := oauth.GenerateOrLoadPDSKey(cfg.SigningKeyPath) 184 + if err != nil { 185 + return "", fmt.Errorf("failed to load signing key: %w", err) 186 + } 187 + 188 + // Load or generate rotation key 189 + rotationKey, err := oauth.GenerateOrLoadPDSKey(cfg.RotationKeyPath) 190 + if err != nil { 191 + return "", fmt.Errorf("failed to load rotation key: %w", err) 192 + } 193 + 194 + did, err = CreatePLCIdentity(ctx, rotationKey, signingKey, cfg.PublicURL, cfg.PLCDirectoryURL) 195 + if err != nil { 196 + return "", fmt.Errorf("failed to create PLC identity: %w", err) 197 + } 198 + 199 + // Persist DID 200 + if err := os.MkdirAll(filepath.Dir(didPath), 0755); err != nil { 201 + return "", fmt.Errorf("failed to create directory for did.txt: %w", err) 202 + } 203 + if err := os.WriteFile(didPath, []byte(did+"\n"), 0600); err != nil { 204 + return "", fmt.Errorf("failed to write did.txt: %w", err) 205 + } 206 + 207 + slog.Info("Created did:plc identity", 208 + "did", did, 209 + "plc_directory", cfg.PLCDirectoryURL, 210 + ) 211 + slog.Warn("Back up rotation.key and optionally remove it from the server. It is only needed for DID updates (URL changes, key rotation).", 212 + "rotation_key_path", cfg.RotationKeyPath, 213 + ) 214 + 215 + return did, nil 216 + } 217 + 218 + // loadOptionalK256Key attempts to load a K-256 private key from disk. 219 + // Returns nil if the file does not exist (key stored offline). 220 + func loadOptionalK256Key(path string) (*atcrypto.PrivateKeyK256, error) { 221 + data, err := os.ReadFile(path) 222 + if err != nil { 223 + return nil, err 224 + } 225 + key, err := atcrypto.ParsePrivateBytesK256(data) 226 + if err != nil { 227 + return nil, fmt.Errorf("failed to parse K-256 key from %s: %w", path, err) 228 + } 229 + return key, nil 230 + } 231 + 232 + // EnsurePLCCurrent checks the PLC directory for the given DID and updates it 233 + // if the local signing key or public URL doesn't match what's registered. 234 + // If rotationKey is nil, mismatches are logged as warnings but not fatal. 235 + func EnsurePLCCurrent(ctx context.Context, did string, rotationKey, signingKey *atcrypto.PrivateKeyK256, publicURL, plcDirectoryURL string) error { 236 + client := &didplc.Client{DirectoryURL: plcDirectoryURL} 237 + 238 + // Fetch current op log 239 + opLog, err := client.OpLog(ctx, did) 240 + if err != nil { 241 + return fmt.Errorf("failed to fetch PLC op log for %s: %w", did, err) 242 + } 243 + if len(opLog) == 0 { 244 + return fmt.Errorf("empty op log for %s", did) 245 + } 246 + 247 + lastEntry := opLog[len(opLog)-1] 248 + lastOp := lastEntry.Regular 249 + if lastOp == nil { 250 + // Last op is not a regular op (could be legacy or tombstone) — skip update 251 + slog.Warn("Last PLC operation is not a regular op, skipping auto-update", "did", did) 252 + return nil 253 + } 254 + 255 + // Compare local state vs PLC state 256 + sigPub, err := signingKey.PublicKey() 257 + if err != nil { 258 + return fmt.Errorf("failed to get signing public key: %w", err) 259 + } 260 + localVerificationKey := sigPub.DIDKey() 261 + plcVerificationKey := lastOp.VerificationMethods["atproto"] 262 + 263 + localEndpoint := publicURL 264 + var plcEndpoint string 265 + if svc, ok := lastOp.Services["atproto_pds"]; ok { 266 + plcEndpoint = svc.Endpoint 267 + } 268 + 269 + keyMatch := localVerificationKey == plcVerificationKey 270 + endpointMatch := localEndpoint == plcEndpoint 271 + 272 + if keyMatch && endpointMatch { 273 + slog.Info("PLC identity is current", "did", did) 274 + return nil 275 + } 276 + 277 + slog.Info("PLC identity needs update", 278 + "did", did, 279 + "signing_key_changed", !keyMatch, 280 + "endpoint_changed", !endpointMatch, 281 + ) 282 + 283 + if rotationKey == nil { 284 + slog.Warn("PLC document doesn't match local state but no rotation key available. Provide rotation key to auto-update PLC directory.", 285 + "did", did, 286 + "signing_key_changed", !keyMatch, 287 + "endpoint_changed", !endpointMatch, 288 + ) 289 + return nil 290 + } 291 + 292 + // Build update operation 293 + rotPub, err := rotationKey.PublicKey() 294 + if err != nil { 295 + return fmt.Errorf("failed to get rotation public key: %w", err) 296 + } 297 + 298 + // Extract hostname for alsoKnownAs 299 + u, err := url.Parse(publicURL) 300 + if err != nil { 301 + return fmt.Errorf("failed to parse public URL: %w", err) 302 + } 303 + host := u.Hostname() 304 + if port := u.Port(); port != "" && port != "80" && port != "443" { 305 + host = host + ":" + port 306 + } 307 + 308 + prevCID := lastEntry.AsOperation().CID().String() 309 + 310 + op := &didplc.RegularOp{ 311 + Type: "plc_operation", 312 + RotationKeys: []string{rotPub.DIDKey()}, 313 + VerificationMethods: map[string]string{ 314 + "atproto": localVerificationKey, 315 + }, 316 + AlsoKnownAs: []string{"at://" + host}, 317 + Services: map[string]didplc.OpService{ 318 + "atproto_pds": {Type: "AtprotoPersonalDataServer", Endpoint: publicURL}, 319 + "atcr_hold": {Type: "AtcrHoldService", Endpoint: publicURL}, 320 + }, 321 + Prev: &prevCID, 322 + } 323 + 324 + if err := op.Sign(rotationKey); err != nil { 325 + return fmt.Errorf("failed to sign PLC update operation: %w", err) 326 + } 327 + 328 + if err := client.Submit(ctx, did, op); err != nil { 329 + return fmt.Errorf("failed to submit PLC update: %w", err) 330 + } 331 + 332 + slog.Info("Updated PLC identity", 333 + "did", did, 334 + "signing_key_rotated", !keyMatch, 335 + "endpoint_changed", !endpointMatch, 336 + ) 337 + 338 + return nil 339 + } 340 + 341 + // CreatePLCIdentity creates a new did:plc identity by building a genesis operation, 342 + // signing it with the rotation key, and submitting it to the PLC directory. 343 + func CreatePLCIdentity(ctx context.Context, rotationKey, signingKey *atcrypto.PrivateKeyK256, publicURL, plcDirectoryURL string) (string, error) { 344 + rotPub, err := rotationKey.PublicKey() 345 + if err != nil { 346 + return "", fmt.Errorf("failed to get rotation public key: %w", err) 347 + } 348 + 349 + sigPub, err := signingKey.PublicKey() 350 + if err != nil { 351 + return "", fmt.Errorf("failed to get signing public key: %w", err) 352 + } 353 + 354 + // Extract hostname for alsoKnownAs 355 + u, err := url.Parse(publicURL) 356 + if err != nil { 357 + return "", fmt.Errorf("failed to parse public URL: %w", err) 358 + } 359 + host := u.Hostname() 360 + if port := u.Port(); port != "" && port != "80" && port != "443" { 361 + host = host + ":" + port 362 + } 363 + 364 + op := &didplc.RegularOp{ 365 + Type: "plc_operation", 366 + RotationKeys: []string{rotPub.DIDKey()}, 367 + VerificationMethods: map[string]string{ 368 + "atproto": sigPub.DIDKey(), 369 + }, 370 + AlsoKnownAs: []string{"at://" + host}, 371 + Services: map[string]didplc.OpService{ 372 + "atproto_pds": {Type: "AtprotoPersonalDataServer", Endpoint: publicURL}, 373 + "atcr_hold": {Type: "AtcrHoldService", Endpoint: publicURL}, 374 + }, 375 + Prev: nil, 376 + } 377 + 378 + if err := op.Sign(rotationKey); err != nil { 379 + return "", fmt.Errorf("failed to sign PLC genesis operation: %w", err) 380 + } 381 + 382 + did, err := op.DID() 383 + if err != nil { 384 + return "", fmt.Errorf("failed to compute DID from genesis operation: %w", err) 385 + } 386 + 387 + client := &didplc.Client{DirectoryURL: plcDirectoryURL} 388 + if err := client.Submit(ctx, did, op); err != nil { 389 + return "", fmt.Errorf("failed to submit genesis operation to PLC directory: %w", err) 390 + } 391 + 392 + return did, nil 107 393 } 108 394 109 395 // GenerateDIDFromURL creates a did:web identifier from a public URL

+11

pkg/hold/pds/export.go

··· 1 + package pds 2 + 3 + import ( 4 + "context" 5 + "io" 6 + ) 7 + 8 + // ExportToCAR streams the hold's repo as a CAR file to the writer. 9 + func (p *HoldPDS) ExportToCAR(ctx context.Context, w io.Writer) error { 10 + return p.repomgr.ReadRepo(ctx, p.uid, "", w) 11 + }

+180

pkg/hold/pds/import.go

··· 1 + package pds 2 + 3 + import ( 4 + "context" 5 + "fmt" 6 + "io" 7 + "strings" 8 + 9 + "github.com/bluesky-social/indigo/repo" 10 + "github.com/ipfs/go-cid" 11 + "go.opentelemetry.io/otel" 12 + ) 13 + 14 + // rawCBOR wraps raw bytes to satisfy cbg.CBORMarshaler. 15 + // Used to pass through record bytes from a CAR without decoding. 16 + type rawCBOR []byte 17 + 18 + func (r rawCBOR) MarshalCBOR(w io.Writer) error { 19 + _, err := w.Write(r) 20 + return err 21 + } 22 + 23 + // bulkRecord holds a single record to import. 24 + type bulkRecord struct { 25 + Collection string 26 + Rkey string 27 + Data rawCBOR 28 + } 29 + 30 + // ImportResult summarizes a CAR import operation. 31 + type ImportResult struct { 32 + Total int 33 + PerCollection map[string]int 34 + } 35 + 36 + // ImportFromCAR reads a CAR file and imports all records into the hold's repo. 37 + // Records are upserted (overwrite on conflict) in a single atomic commit. 38 + // The repo is initialized if it doesn't exist yet. 39 + func (p *HoldPDS) ImportFromCAR(ctx context.Context, r io.Reader) (*ImportResult, error) { 40 + // Ensure repo exists 41 + head, err := p.carstore.GetUserRepoHead(ctx, p.uid) 42 + if err != nil || !head.Defined() { 43 + if err := p.repomgr.InitNewActor(ctx, p.uid, "", p.did, "", "", ""); err != nil { 44 + return nil, fmt.Errorf("failed to initialize repo: %w", err) 45 + } 46 + } 47 + 48 + // Parse the CAR into an in-memory repo 49 + sourceRepo, err := repo.ReadRepoFromCar(ctx, r) 50 + if err != nil { 51 + return nil, fmt.Errorf("failed to read CAR: %w", err) 52 + } 53 + 54 + // Collect all records 55 + var records []bulkRecord 56 + err = sourceRepo.ForEach(ctx, "", func(k string, v cid.Cid) error { 57 + _, recBytes, err := sourceRepo.GetRecordBytes(ctx, k) 58 + if err != nil { 59 + return fmt.Errorf("failed to get record bytes for %s: %w", k, err) 60 + } 61 + 62 + parts := strings.SplitN(k, "/", 2) 63 + if len(parts) != 2 { 64 + return fmt.Errorf("unexpected record path format: %s", k) 65 + } 66 + 67 + records = append(records, bulkRecord{ 68 + Collection: parts[0], 69 + Rkey: parts[1], 70 + Data: rawCBOR(*recBytes), 71 + }) 72 + return nil 73 + }) 74 + if err != nil { 75 + return nil, fmt.Errorf("failed to iterate CAR records: %w", err) 76 + } 77 + 78 + if len(records) == 0 { 79 + return &ImportResult{PerCollection: map[string]int{}}, nil 80 + } 81 + 82 + // Bulk upsert all records in a single commit 83 + if err := p.bulkImportRecords(ctx, records); err != nil { 84 + return nil, fmt.Errorf("failed to import records: %w", err) 85 + } 86 + 87 + // Build result 88 + result := &ImportResult{ 89 + Total: len(records), 90 + PerCollection: make(map[string]int), 91 + } 92 + for _, rec := range records { 93 + result.PerCollection[rec.Collection]++ 94 + } 95 + return result, nil 96 + } 97 + 98 + // bulkImportRecords writes all records in a single delta session + commit. 99 + // Each record is upserted: created if new, updated if exists. 100 + func (p *HoldPDS) bulkImportRecords(ctx context.Context, records []bulkRecord) error { 101 + ctx, span := otel.Tracer("repoman").Start(ctx, "BulkImportRecords") 102 + defer span.End() 103 + 104 + unlock := p.repomgr.lockUser(ctx, p.uid) 105 + defer unlock() 106 + 107 + rev, err := p.repomgr.cs.GetUserRepoRev(ctx, p.uid) 108 + if err != nil { 109 + return err 110 + } 111 + 112 + ds, err := p.repomgr.cs.NewDeltaSession(ctx, p.uid, &rev) 113 + if err != nil { 114 + return err 115 + } 116 + 117 + head := ds.BaseCid() 118 + r, err := repo.OpenRepo(ctx, ds, head) 119 + if err != nil { 120 + return err 121 + } 122 + 123 + ops := make([]RepoOp, 0, len(records)) 124 + for _, rec := range records { 125 + rpath := rec.Collection + "/" + rec.Rkey 126 + 127 + // Check if record exists to determine create vs update 128 + _, _, getErr := r.GetRecordBytes(ctx, rpath) 129 + recordExists := getErr == nil 130 + 131 + var cc cid.Cid 132 + var evtKind EventKind 133 + if recordExists { 134 + cc, err = r.UpdateRecord(ctx, rpath, rec.Data) 135 + evtKind = EvtKindUpdateRecord 136 + } else { 137 + cc, err = r.PutRecord(ctx, rpath, rec.Data) 138 + evtKind = EvtKindCreateRecord 139 + } 140 + if err != nil { 141 + return fmt.Errorf("failed to write %s: %w", rpath, err) 142 + } 143 + 144 + ops = append(ops, RepoOp{ 145 + Kind: evtKind, 146 + Collection: rec.Collection, 147 + Rkey: rec.Rkey, 148 + RecCid: &cc, 149 + }) 150 + } 151 + 152 + nroot, nrev, err := r.Commit(ctx, p.repomgr.kmgr.SignForUser) 153 + if err != nil { 154 + return err 155 + } 156 + 157 + rslice, err := ds.CloseWithRoot(ctx, nroot, nrev) 158 + if err != nil { 159 + return fmt.Errorf("close with root: %w", err) 160 + } 161 + 162 + var oldroot *cid.Cid 163 + if head.Defined() { 164 + oldroot = &head 165 + } 166 + 167 + if p.repomgr.events != nil { 168 + p.repomgr.events(ctx, &RepoEvent{ 169 + User: p.uid, 170 + OldRoot: oldroot, 171 + NewRoot: nroot, 172 + Rev: nrev, 173 + Since: &rev, 174 + Ops: ops, 175 + RepoSlice: rslice, 176 + }) 177 + } 178 + 179 + return nil 180 + }

+14 -4

pkg/hold/server.go

··· 71 71 var xrpcHandler *pds.XRPCHandler 72 72 var s3Service *s3.S3Service 73 73 if cfg.Database.Path != "" { 74 - holdDID := pds.GenerateDIDFromURL(cfg.Server.PublicURL) 75 - slog.Info("Initializing embedded PDS", "did", holdDID) 74 + ctx := context.Background() 76 75 77 - ctx := context.Background() 78 - var err error 76 + holdDID, err := pds.LoadOrCreateDID(ctx, pds.DIDConfig{ 77 + DID: cfg.Database.DID, 78 + DIDMethod: cfg.Database.DIDMethod, 79 + PublicURL: cfg.Server.PublicURL, 80 + DBPath: cfg.Database.Path, 81 + SigningKeyPath: cfg.Database.KeyPath, 82 + RotationKeyPath: cfg.Database.RotationKeyPath, 83 + PLCDirectoryURL: cfg.Database.PLCDirectoryURL, 84 + }) 85 + if err != nil { 86 + return nil, fmt.Errorf("failed to resolve hold DID: %w", err) 87 + } 88 + slog.Info("Initializing embedded PDS", "did", holdDID) 79 89 80 90 if cfg.Database.Path != ":memory:" { 81 91 // File mode: open centralized shared DB (supports embedded replica sync)