evan.jarrett.net
A container registry that uses the AT Protocol for manifest storage and S3 for blob storage.
atcr.io
docker
container
atproto
go
1# ATCR UpCloud Deployment Guide
2
3This guide walks you through deploying ATCR on UpCloud with Rocky Linux.
4
5## Architecture
6
7- **AppView** (atcr.io) - OCI registry API + web UI
8- **Hold Service** (hold01.atcr.io) - Presigned URL generator for blob storage
9- **Caddy** - Reverse proxy with automatic HTTPS
10- **UpCloud Object Storage** (blobs.atcr.io) - S3-compatible blob storage
11
12## Prerequisites
13
14### 1. UpCloud Account
15- Active UpCloud account
16- Object Storage enabled
17- Billing configured
18
19### 2. Domain Names
20You need three DNS records:
21- `atcr.io` (or your domain) - AppView
22- `hold01.atcr.io` - Hold service
23- `blobs.atcr.io` - S3 storage (CNAME)
24
25### 3. ATProto Account
26- Bluesky/ATProto account
27- Your DID (get from: `https://bsky.social/xrpc/com.atproto.identity.resolveHandle?handle=yourhandle.bsky.social`)
28
29### 4. UpCloud Object Storage Bucket
30Create an S3 bucket in UpCloud Object Storage:
311. Go to UpCloud Console → Storage → Object Storage
322. Create new bucket (e.g., `atcr-blobs`)
333. Note the region (e.g., `us-chi1`)
344. Generate access credentials (Access Key ID + Secret)
355. Note the endpoint (e.g., `s3.us-chi1.upcloudobjects.com`)
36
37## Deployment Steps
38
39### Step 1: Configure DNS
40
41Set up DNS records (using Cloudflare or your DNS provider):
42
43```
44Type Name Value Proxy
45────────────────────────────────────────────────────────────────────────────
46A atcr.io [your-upcloud-ip] ☁️ DISABLED
47A hold01.atcr.io [your-upcloud-ip] ☁️ DISABLED
48CNAME blobs.atcr.io atcr-blobs.us-chi1.upcloudobjects.com ☁️ DISABLED
49```
50
51**IMPORTANT:**
52- **DISABLE Cloudflare proxy** (gray cloud, not orange) for all three domains
53- Proxied connections break Docker registry protocol and presigned URLs
54- You'll still get HTTPS via Caddy's Let's Encrypt integration
55
56Wait for DNS propagation (5-30 minutes). Verify with:
57```bash
58dig atcr.io
59dig hold01.atcr.io
60dig blobs.atcr.io
61```
62
63### Step 2: Create UpCloud Server
64
651. Go to UpCloud Console → Servers → Deploy a new server
662. Select location (match your S3 region if possible)
673. Select **Rocky Linux 9** operating system
684. Choose plan (minimum: 2 GB RAM, 1 CPU)
695. Configure hostname: `atcr`
706. Enable IPv4 public networking
717. **Optional:** Enable IPv6
728. **User data:** Paste contents of `deploy/init-upcloud.sh`
73 - Update `ATCR_REPO` variable with your git repository URL
74 - Or leave empty and manually copy files later
759. Create SSH key or use password authentication
7610. Click **Deploy**
77
78### Step 3: Wait for Initialization
79
80The init script will:
81- Update system packages (~2-5 minutes)
82- Install Docker and Docker Compose
83- Configure firewall
84- Clone repository (if ATCR_REPO configured)
85- Create systemd service
86- Create helper scripts
87
88Monitor progress:
89```bash
90# SSH into server
91ssh root@[your-upcloud-ip]
92
93# Check cloud-init logs
94tail -f /var/log/cloud-init-output.log
95```
96
97Wait for the completion message in the logs.
98
99### Step 4: Configure Environment
100
101Edit the environment configuration:
102
103```bash
104# SSH into server
105ssh root@[your-upcloud-ip]
106
107# Edit environment file
108cd /opt/atcr
109nano .env
110```
111
112**Required configuration:**
113
114```bash
115# Domains
116APPVIEW_DOMAIN=atcr.io
117HOLD_DOMAIN=hold01.atcr.io
118
119# Your ATProto DID
120HOLD_OWNER=did:plc:your-did-here
121
122# UpCloud S3 credentials
123AWS_ACCESS_KEY_ID=your-access-key-id
124AWS_SECRET_ACCESS_KEY=your-secret-access-key
125AWS_REGION=us-chi1
126S3_BUCKET=atcr-blobs
127
128# S3 endpoint (choose one):
129# Option 1: Custom domain (recommended)
130S3_ENDPOINT=https://blobs.atcr.io
131# Option 2: Direct UpCloud endpoint
132# S3_ENDPOINT=https://s3.us-chi1.upcloudobjects.com
133
134# Public access (optional)
135HOLD_PUBLIC=false # Set to true to allow anonymous pulls
136```
137
138Save and exit (Ctrl+X, Y, Enter).
139
140### Step 5: Start ATCR
141
142```bash
143# Start services
144systemctl start atcr
145
146# Check status
147systemctl status atcr
148
149# Verify containers are running
150docker ps
151```
152
153You should see three containers:
154- `atcr-caddy`
155- `atcr-appview`
156- `atcr-hold`
157
158### Step 6: Complete Hold OAuth Registration
159
160The hold service needs to register itself with your PDS:
161
162```bash
163# Get OAuth URL from logs
164/opt/atcr/get-hold-oauth.sh
165```
166
167Look for output like:
168```
169Visit this URL to authorize: https://bsky.social/oauth/authorize?...
170```
171
1721. Copy the URL and open in your browser
1732. Log in with your ATProto account
1743. Authorize the hold service
1754. Return to terminal
176
177The hold service will create records in your PDS:
178- `io.atcr.hold` - Hold definition
179- `io.atcr.hold.crew` - Your membership as captain
180
181Verify registration:
182```bash
183docker logs atcr-hold | grep -i "success\|registered\|created"
184```
185
186### Step 7: Test the Registry
187
188#### Test 1: Check endpoints
189
190```bash
191# AppView (should return {})
192curl https://atcr.io/v2/
193
194# Hold service (should return {"status":"ok"})
195curl https://hold01.atcr.io/health
196```
197
198#### Test 2: Configure Docker client
199
200On your local machine:
201
202```bash
203# Install credential helper
204# (Build from source or download release)
205go install atcr.io/cmd/docker-credential-atcr@latest
206
207# Configure Docker to use the credential helper
208# Add to ~/.docker/config.json:
209{
210 "credHelpers": {
211 "atcr.io": "atcr"
212 }
213}
214```
215
216#### Test 3: Push a test image
217
218```bash
219# Tag an image
220docker tag alpine:latest atcr.io/yourhandle/test:latest
221
222# Push to ATCR
223docker push atcr.io/yourhandle/test:latest
224
225# Pull from ATCR
226docker pull atcr.io/yourhandle/test:latest
227```
228
229### Step 8: Monitor and Maintain
230
231#### View logs
232
233```bash
234# All services
235/opt/atcr/logs.sh
236
237# Specific service
238/opt/atcr/logs.sh atcr-appview
239/opt/atcr/logs.sh atcr-hold
240/opt/atcr/logs.sh atcr-caddy
241
242# Or use docker directly
243docker logs -f atcr-appview
244```
245
246#### Enable debug logging
247
248Toggle debug logging at runtime without restarting the container:
249
250```bash
251# Enable debug logging (auto-reverts after 30 minutes)
252docker kill -s SIGUSR1 atcr-appview
253docker kill -s SIGUSR1 atcr-hold
254
255# Manually disable before timeout
256docker kill -s SIGUSR1 atcr-appview
257```
258
259When toggled, you'll see:
260```
261level=INFO msg="Log level changed" from=INFO to=DEBUG trigger=SIGUSR1 auto_revert_in=30m0s
262```
263
264**Note:** Despite the command name, `docker kill -s SIGUSR1` does NOT stop the container. It sends a user-defined signal that the application handles to toggle debug mode.
265
266#### Restart services
267
268```bash
269# Restart all
270systemctl restart atcr
271
272# Or use docker-compose
273cd /opt/atcr
274docker compose -f deploy/docker-compose.prod.yml restart
275```
276
277#### Rebuild after code changes
278
279```bash
280/opt/atcr/rebuild.sh
281```
282
283#### Update configuration
284
285```bash
286# Edit environment
287nano /opt/atcr/.env
288
289# Restart services
290systemctl restart atcr
291```
292
293## Architecture Details
294
295### Service Communication
296
297```
298Internet
299 ↓
300Caddy (443) ───────────┐
301 ├─→ atcr-appview:5000 (Registry API + Web UI)
302 └─→ atcr-hold:8080 (Presigned URL generator)
303 ↓
304 UpCloud S3 (blobs.atcr.io)
305```
306
307### Data Flow: Push
308
309```
3101. docker push atcr.io/user/image:tag
3112. AppView ← Docker client (manifest + blob metadata)
3123. AppView → ATProto PDS (store manifest record)
3134. Hold ← Docker client (request presigned URL)
3145. Hold → UpCloud S3 API (generate presigned URL)
3156. Hold → Docker client (return presigned URL)
3167. UpCloud S3 ← Docker client (upload blob directly)
317```
318
319### Data Flow: Pull
320
321```
3221. docker pull atcr.io/user/image:tag
3232. AppView ← Docker client (get manifest)
3243. AppView → ATProto PDS (fetch manifest record)
3254. AppView → Docker client (return manifest with holdEndpoint)
3265. Hold ← Docker client (request presigned URL)
3276. Hold → UpCloud S3 API (generate presigned URL)
3287. Hold → Docker client (return presigned URL)
3298. UpCloud S3 ← Docker client (download blob directly)
330```
331
332**Key insight:** The hold service only generates presigned URLs. Actual data transfer happens directly between Docker clients and S3, minimizing bandwidth costs.
333
334## Troubleshooting
335
336### Issue: "Cannot connect to registry"
337
338**Check DNS:**
339```bash
340dig atcr.io
341dig hold01.atcr.io
342```
343
344**Check Caddy logs:**
345```bash
346docker logs atcr-caddy
347```
348
349**Check firewall:**
350```bash
351firewall-cmd --list-all
352```
353
354### Issue: "Certificate errors"
355
356**Verify DNS is propagated:**
357```bash
358curl -I https://atcr.io
359```
360
361**Check Caddy is obtaining certificates:**
362```bash
363docker logs atcr-caddy | grep -i certificate
364```
365
366**Common causes:**
367- DNS not propagated (wait 30 minutes)
368- Cloudflare proxy enabled (must be disabled)
369- Port 80/443 blocked by firewall
370
371### Issue: "Presigned URLs fail"
372
373**Check S3 endpoint configuration:**
374```bash
375docker exec atcr-hold env | grep S3
376```
377
378**Verify custom domain CNAME:**
379```bash
380dig blobs.atcr.io CNAME
381```
382
383**Test S3 connectivity:**
384```bash
385docker exec atcr-hold wget -O- https://blobs.atcr.io/
386```
387
388**Common causes:**
389- Cloudflare proxy enabled on blobs.atcr.io
390- S3_ENDPOINT misconfigured
391- AWS credentials invalid
392
393### Issue: "Hold registration fails"
394
395**Check hold owner DID:**
396```bash
397docker exec atcr-hold env | grep HOLD_OWNER
398```
399
400**Verify OAuth flow:**
401```bash
402/opt/atcr/get-hold-oauth.sh
403```
404
405**Manual registration:**
406```bash
407# Get fresh OAuth URL
408docker restart atcr-hold
409docker logs -f atcr-hold
410```
411
412### Issue: "High bandwidth usage"
413
414Presigned URLs should eliminate hold bandwidth. If seeing high usage:
415
416**Verify presigned URLs are enabled:**
417```bash
418docker logs atcr-hold | grep -i presigned
419```
420
421**Check S3 configuration:**
422```bash
423docker exec atcr-hold env | grep S3_BUCKET
424# Should show your S3 bucket name
425```
426
427**Verify direct S3 access:**
428```bash
429# Push should show 307 redirects in logs
430docker logs -f atcr-hold
431# Then push an image
432```
433
434### Automatic Updates
435
436```bash
437# Install automatic updates
438dnf install -y dnf-automatic
439
440# Enable timer
441systemctl enable --now dnf-automatic.timer
442```
443
444### Monitoring
445
446```bash
447# Install monitoring tools
448dnf install -y htop iotop nethogs
449
450# Monitor resources
451htop
452
453# Monitor Docker
454docker stats
455```
456
457### Backups
458
459Critical data to backup:
460- `/opt/atcr/.env` - Configuration
461- Docker volumes:
462 - `atcr-appview-data` - Auth keys, UI database, OAuth tokens
463 - `caddy_data` - TLS certificates
464
465```bash
466# Backup volumes
467docker run --rm \
468 -v atcr-appview-data:/data \
469 -v /backup:/backup \
470 alpine tar czf /backup/atcr-appview-data.tar.gz /data
471```
472
473## Scaling Considerations
474
475### Single Server (Current Setup)
476- Suitable for: 100-1000 users
477- Bottleneck: AppView CPU (manifest queries)
478- Storage: Unlimited (S3)
479
480### Multi-Server (Future)
481- Multiple AppView instances behind load balancer
482- Shared Redis for hold cache (replace in-memory cache)
483- PostgreSQL for UI database (replace SQLite)
484- Multiple hold services (geo-distributed)
485
486## Support
487
488- Documentation: https://tangled.org/evan.jarrett.net/at-container-registry
489- Issues: https://tangled.org/evan.jarrett.net/at-container-registry/issues
490- Bluesky: @evan.jarrett.net