A container registry that uses the AT Protocol for manifest storage and S3 for blob storage.
atcr.io
docker
container
atproto
go
evan.jarrett.net
ui fixes, add ability to warn/hide unreachable manifests from the ui. clean up docs
1# Bring Your Own Storage (BYOS)
2
3## Overview
4
5ATCR supports "Bring Your Own Storage" (BYOS) for blob storage. Users can:
6- Deploy their own hold service with embedded PDS
7- Control access via crew membership in the hold's PDS
8- Keep blob data in their own S3/Storj/Minio while manifests stay in their user PDS
9
10## Architecture
11
12```
13┌──────────────────────────────────────────┐
14│ ATCR AppView (API) │
15│ - Manifests → User's PDS │
16│ - Auth & service token management │
17│ - Blob routing via XRPC │
18│ - Profile management │
19└────────────┬─────────────────────────────┘
20 │
21 │ Hold discovery priority:
22 │ 1. io.atcr.sailor.profile.defaultHold (DID)
23 │ 2. io.atcr.hold records (legacy)
24 │ 3. AppView default_hold_did
25 ▼
26┌──────────────────────────────────────────┐
27│ User's PDS │
28│ - io.atcr.sailor.profile (hold DID) │
29│ - io.atcr.manifest (with holdDid) │
30└────────────┬─────────────────────────────┘
31 │
32 │ Service token from user's PDS
33 ▼
34┌──────────────────────────────────────────┐
35│ Hold Service (did:web:hold.example.com) │
36│ ├── Embedded PDS │
37│ │ ├── Captain record (ownership) │
38│ │ └── Crew records (access control) │
39│ ├── XRPC multipart upload endpoints │
40│ └── Storage driver (S3/Storj/etc.) │
41└──────────────────────────────────────────┘
42```
43
44## Hold Service Components
45
46Each hold is a full ATProto actor with:
47- **DID**: `did:web:hold.example.com` (hold's identity)
48- **Embedded PDS**: Stores captain + crew records (shared data)
49- **Storage backend**: S3, Storj, Minio, filesystem, etc.
50- **XRPC endpoints**: Standard ATProto + custom OCI multipart upload
51
52### Records in Hold's PDS
53
54**Captain record** (`io.atcr.hold.captain/self`):
55```json
56{
57 "$type": "io.atcr.hold.captain",
58 "owner": "did:plc:alice123",
59 "public": false,
60 "deployedAt": "2025-10-14T...",
61 "region": "iad",
62 "provider": "fly.io"
63}
64```
65
66**Crew records** (`io.atcr.hold.crew/{rkey}`):
67```json
68{
69 "$type": "io.atcr.hold.crew",
70 "member": "did:plc:bob456",
71 "role": "admin",
72 "permissions": ["blob:read", "blob:write"],
73 "addedAt": "2025-10-14T..."
74}
75```
76
77### Sailor Profile (User's PDS)
78
79Users set their preferred hold in their sailor profile:
80
81```json
82{
83 "$type": "io.atcr.sailor.profile",
84 "defaultHold": "did:web:hold.example.com",
85 "createdAt": "2025-10-02T...",
86 "updatedAt": "2025-10-02T..."
87}
88```
89
90## Deployment
91
92### Configuration
93
94Hold service is configured entirely via environment variables:
95
96```bash
97# Hold identity (REQUIRED)
98HOLD_PUBLIC_URL=https://hold.example.com
99HOLD_OWNER=did:plc:your-did-here
100
101# Storage backend
102STORAGE_DRIVER=s3
103AWS_ACCESS_KEY_ID=your_access_key
104AWS_SECRET_ACCESS_KEY=your_secret_key
105AWS_REGION=us-east-1
106S3_BUCKET=my-blobs
107
108# Access control
109HOLD_PUBLIC=false # Require authentication for reads
110HOLD_ALLOW_ALL_CREW=false # Only explicit crew members can write
111
112# Embedded PDS
113HOLD_DATABASE_PATH=/var/lib/atcr-hold/hold.db
114HOLD_DATABASE_KEY_PATH=/var/lib/atcr-hold/keys
115```
116
117### Running Locally
118
119```bash
120# Build
121go build -o bin/atcr-hold ./cmd/hold
122
123# Run (with env vars or .env file)
124export HOLD_PUBLIC_URL=http://localhost:8080
125export HOLD_OWNER=did:plc:your-did-here
126export STORAGE_DRIVER=filesystem
127export STORAGE_ROOT_DIR=/tmp/atcr-hold
128export HOLD_DATABASE_PATH=/tmp/atcr-hold/hold.db
129
130./bin/atcr-hold
131```
132
133On first run, the hold service creates:
134- Captain record in embedded PDS (making you the owner)
135- Crew record for owner with all permissions
136- DID document at `/.well-known/did.json`
137
138### Deploy to Fly.io
139
140```bash
141# Create fly.toml
142cat > fly.toml <<EOF
143app = "my-atcr-hold"
144primary_region = "ord"
145
146[env]
147 HOLD_PUBLIC_URL = "https://my-atcr-hold.fly.dev"
148 STORAGE_DRIVER = "s3"
149 AWS_REGION = "us-east-1"
150 S3_BUCKET = "my-blobs"
151 HOLD_PUBLIC = "false"
152 HOLD_ALLOW_ALL_CREW = "false"
153
154[http_service]
155 internal_port = 8080
156 force_https = true
157 auto_stop_machines = true
158 auto_start_machines = true
159 min_machines_running = 0
160
161[[vm]]
162 cpu_kind = "shared"
163 cpus = 1
164 memory_mb = 256
165EOF
166
167# Deploy
168fly launch
169fly deploy
170
171# Set secrets
172fly secrets set AWS_ACCESS_KEY_ID=...
173fly secrets set AWS_SECRET_ACCESS_KEY=...
174fly secrets set HOLD_OWNER=did:plc:your-did-here
175```
176
177## Request Flow
178
179### Push with BYOS
180
181```
1821. Client: docker push atcr.io/alice/myapp:latest
183
1842. AppView resolves alice → did:plc:alice123
185
1863. AppView discovers hold DID:
187 - Check alice's sailor profile for defaultHold
188 - Returns: "did:web:alice-storage.fly.dev"
189
1904. AppView gets service token from alice's PDS:
191 GET /xrpc/com.atproto.server.getServiceAuth?aud=did:web:alice-storage.fly.dev
192 Response: { "token": "eyJ..." }
193
1945. AppView initiates multipart upload to hold:
195 POST https://alice-storage.fly.dev/xrpc/io.atcr.hold.initiateUpload
196 Authorization: Bearer {serviceToken}
197 Body: { "digest": "sha256:abc..." }
198 Response: { "uploadId": "xyz" }
199
2006. For each part:
201 - AppView: POST /xrpc/io.atcr.hold.getPartUploadUrl
202 - Hold validates service token, checks crew membership
203 - Hold returns: { "url": "https://s3.../presigned" }
204 - Client uploads directly to S3 presigned URL
205
2067. AppView completes upload:
207 POST /xrpc/io.atcr.hold.completeUpload
208 Body: { "uploadId": "xyz", "digest": "sha256:abc...", "parts": [...] }
209
2108. Manifest stored in alice's PDS:
211 - holdDid: "did:web:alice-storage.fly.dev"
212 - holdEndpoint: "https://alice-storage.fly.dev" (backward compat)
213```
214
215### Pull with BYOS
216
217```
2181. Client: docker pull atcr.io/alice/myapp:latest
219
2202. AppView fetches manifest from alice's PDS
221
2223. Manifest contains:
223 - holdDid: "did:web:alice-storage.fly.dev"
224
2254. AppView caches hold DID for 10 minutes (covers pull operation)
226
2275. Client requests blob: GET /v2/alice/myapp/blobs/sha256:abc123
228
2296. AppView uses cached hold DID from manifest
230
2317. AppView gets service token from alice's PDS
232
2338. AppView calls hold XRPC:
234 GET /xrpc/com.atproto.sync.getBlob?did={userDID}&cid=sha256:abc123
235 Authorization: Bearer {serviceToken}
236 Response: { "url": "https://s3.../presigned-download" }
237
2389. AppView redirects client to presigned S3 URL
239
24010. Client downloads directly from S3
241```
242
243**Key insight:** Pull uses the `holdDid` stored in the manifest, ensuring blobs are fetched from where they were originally pushed.
244
245## Access Control
246
247### Read Access
248
249- **Public hold** (`HOLD_PUBLIC=true`): Anonymous + authenticated users
250- **Private hold** (`HOLD_PUBLIC=false`): Authenticated users with crew membership
251
252### Write Access
253
254- Hold owner (captain) OR crew members only
255- Verified via `io.atcr.hold.crew` records in hold's embedded PDS
256- Service token proves user identity (from user's PDS)
257
258### Authorization Flow
259
260```go
2611. AppView gets service token from user's PDS
2622. AppView sends request to hold with service token
2633. Hold validates service token (checks it's from user's PDS)
2644. Hold extracts user's DID from token
2655. Hold checks crew records in its embedded PDS
2666. If crew member found → allow, else → deny
267```
268
269## Managing Crew Members
270
271### Add Crew Member
272
273Use ATProto client to create crew record in hold's PDS:
274
275```bash
276# Via XRPC (if hold supports it)
277POST https://hold.example.com/xrpc/io.atcr.hold.requestCrew
278Authorization: Bearer {userOAuthToken}
279
280# Or manually via captain's OAuth to hold's PDS
281atproto put-record \
282 --pds https://hold.example.com \
283 --collection io.atcr.hold.crew \
284 --rkey "{memberDID}" \
285 --value '{
286 "$type": "io.atcr.hold.crew",
287 "member": "did:plc:bob456",
288 "role": "admin",
289 "permissions": ["blob:read", "blob:write"]
290 }'
291```
292
293### Remove Crew Member
294
295```bash
296atproto delete-record \
297 --pds https://hold.example.com \
298 --collection io.atcr.hold.crew \
299 --rkey "{memberDID}"
300```
301
302## Storage Drivers
303
304Hold service supports all distribution storage drivers:
305- **S3** - AWS S3, Minio, Storj (via S3 gateway)
306- **Filesystem** - Local disk (for testing)
307- **Azure** - Azure Blob Storage
308- **GCS** - Google Cloud Storage
309- **Swift** - OpenStack Swift
310
311## Example: Team Hold
312
313```bash
314# 1. Deploy hold service
315export HOLD_PUBLIC_URL=https://team-hold.fly.dev
316export HOLD_OWNER=did:plc:admin
317export HOLD_PUBLIC=false # Private
318export STORAGE_DRIVER=s3
319export AWS_ACCESS_KEY_ID=...
320export S3_BUCKET=team-blobs
321
322fly deploy
323
324# 2. Hold auto-creates captain + crew records on first run
325
326# 3. Admin adds team members via hold's PDS (requires OAuth)
327# (TODO: Implement crew management UI/CLI)
328
329# 4. Team members set their sailor profile:
330atproto put-record \
331 --collection io.atcr.sailor.profile \
332 --rkey "self" \
333 --value '{
334 "$type": "io.atcr.sailor.profile",
335 "defaultHold": "did:web:team-hold.fly.dev"
336 }'
337
338# 5. Team members can now push/pull using team hold
339```
340
341## Limitations
342
343### Current IAM Challenges
344
345See [EMBEDDED_PDS.md](./EMBEDDED_PDS.md#iam-challenges) for detailed discussion.
346
347**Known issues:**
3481. **RPC permission format**: Service tokens don't work with IP-based DIDs in local dev
3492. **Dynamic hold discovery**: AppView can't dynamically OAuth arbitrary holds from sailor profiles
3503. **Manual profile management**: No UI for updating sailor profile (must use ATProto client)
351
352**Workaround:** Use hostname-based DIDs (`did:web:hold.example.com`) and public holds for now.
353
354## Future Improvements
355
3561. **Crew management UI** - Web interface for adding/removing crew members
3572. **Dynamic OAuth** - Support for arbitrary BYOS holds without pre-configuration
3583. **Hold migration** - Tools for moving blobs between holds
3594. **Storage analytics** - Track usage per user/repository
3605. **Distributed cache** - Redis for hold DID cache in multi-instance deployments
361
362## References
363
364- [EMBEDDED_PDS.md](./EMBEDDED_PDS.md) - Embedded PDS architecture and IAM details
365- [ATProto Lexicon Spec](https://atproto.com/specs/lexicon)
366- [Distribution Storage Drivers](https://distribution.github.io/distribution/storage-drivers/)
367- [S3 Presigned URLs](https://docs.aws.amazon.com/AmazonS3/latest/userguide/PresignedUrlUploadObject.html)