bretton.dev
A community based topic aggregation platform built on atproto
feat: Set up local development environment with Bluesky PDS
- Add docker-compose.dev.yml with Bluesky PDS (port 3001)
- Add .env.dev with development configuration
- Add Makefile with convenient dev commands (help, dev-up, dev-down, etc.)
- Add comprehensive docs/LOCAL_DEVELOPMENT.md guide
- Update CLAUDE.md and ATPROTO_GUIDE.md with correct architecture
- Remove custom carstore implementation (PDS handles this)
- Remove internal/atproto/repo wrapper (not needed)
- Add feed lexicon schemas (getAll, getCommunity, getTimeline)
- Update post lexicons to remove getFeed (replaced by feed queries)
- Update PROJECT_STRUCTURE.md to reflect new architecture
Architecture:
- PDS is self-contained with internal SQLite + CAR storage
- PostgreSQL database only used by Coves AppView for indexing
- AppView subscribes directly to PDS firehose (no relay needed for local dev)
- PDS runs on port 3001 to avoid conflicts with production PDS on 3000
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
+69
.env.dev
+69
.env.dev
···
1
+
# Coves Local Development Environment Configuration
2
+
# This file contains all environment variables for the local atProto development stack
3
+
# DO NOT commit secrets to version control in production!
4
+
5
+
# =============================================================================
6
+
# PostgreSQL Configuration (Shared Database)
7
+
# =============================================================================
8
+
# Uses existing database from internal/db/local_dev_db_compose/
9
+
POSTGRES_HOST=localhost
10
+
POSTGRES_PORT=5433
11
+
POSTGRES_DB=coves_dev
12
+
POSTGRES_USER=dev_user
13
+
POSTGRES_PASSWORD=dev_password
14
+
15
+
# =============================================================================
16
+
# PDS (Personal Data Server) Configuration
17
+
# =============================================================================
18
+
# PDS runs on port 3001 (to avoid conflict with production PDS on :3000)
19
+
PDS_HOSTNAME=localhost
20
+
PDS_PORT=3001
21
+
22
+
# DID PLC Directory (use Bluesky's for development)
23
+
PDS_DID_PLC_URL=https://plc.directory
24
+
25
+
# JWT Secret (for signing tokens - change in production!)
26
+
PDS_JWT_SECRET=local-dev-jwt-secret-change-in-production
27
+
28
+
# Admin password for PDS management
29
+
PDS_ADMIN_PASSWORD=admin
30
+
31
+
# Handle domains (users will get handles like alice.local.coves.dev)
32
+
PDS_SERVICE_HANDLE_DOMAINS=.local.coves.dev
33
+
34
+
# PLC Rotation Key (k256 private key in hex format - for local dev only)
35
+
# This is a randomly generated key for testing - DO NOT use in production
36
+
PDS_PLC_ROTATION_KEY=af514fb84c4356241deed29feb392d1ee359f99c05a7b8f7bff2e5f2614f64b2
37
+
38
+
# =============================================================================
39
+
# AppView Configuration (Your Go Application)
40
+
# =============================================================================
41
+
# AppView runs on port 8081 (to avoid conflicts)
42
+
APPVIEW_PORT=8081
43
+
44
+
# PDS Firehose URL (WebSocket connection - direct to PDS, no relay)
45
+
FIREHOSE_URL=ws://localhost:3001/xrpc/com.atproto.sync.subscribeRepos
46
+
47
+
# PDS URL (for XRPC calls)
48
+
PDS_URL=http://localhost:3001
49
+
50
+
# =============================================================================
51
+
# Development Settings
52
+
# =============================================================================
53
+
# Environment
54
+
ENV=development
55
+
NODE_ENV=development
56
+
57
+
# Logging
58
+
LOG_LEVEL=debug
59
+
LOG_ENABLED=true
60
+
61
+
# =============================================================================
62
+
# Notes
63
+
# =============================================================================
64
+
# - PDS port 3001 avoids conflict with your production PDS on :3000
65
+
# - AppView port 8081 avoids conflicts
66
+
# - PostgreSQL port 5433 matches your existing local dev database
67
+
# - All services connect to the shared PostgreSQL database
68
+
# - AppView subscribes directly to PDS firehose (no relay needed for local dev)
69
+
# - PDS firehose: ws://localhost:3001/xrpc/com.atproto.sync.subscribeRepos
+45
-83
ATPROTO_GUIDE.md
+45
-83
ATPROTO_GUIDE.md
···
24
24
### Key Components
25
25
1. **DIDs (Decentralized Identifiers)** - Persistent user identifiers (e.g., `did:plc:xyz123`)
26
26
2. **Handles** - Human-readable names that resolve to DIDs (e.g., `alice.bsky.social`)
27
-
3. **Repositories** - User data stored as signed Merkle trees in CAR files
27
+
3. **Repositories** - User data stored in PDS'
28
28
4. **Lexicons** - Schema definitions for data types and API methods
29
29
5. **XRPC** - The RPC protocol for client-server communication
30
30
6. **Firehose** - Real-time event stream of repository changes
31
31
32
32
## Architecture Overview
33
33
34
-
### Two-Database Pattern
35
-
AT Protocol requires two distinct data stores:
34
+
### Coves Architecture Pattern
35
+
Coves uses a simplified, single-database architecture that leverages existing atProto infrastructure:
36
36
37
-
#### 1. Repository Database (Source of Truth)
38
-
- **Purpose**: Stores user-generated content as immutable, signed records
39
-
- **Storage**: CAR files containing Merkle trees + PostgreSQL metadata
40
-
- **Access**: Through XRPC procedures that modify repositories
41
-
- **Properties**:
42
-
- Append-only (soft deletes via tombstones)
43
-
- Cryptographically verifiable
44
-
- User-controlled and portable
37
+
#### Components
45
38
46
-
#### 2. AppView Database (Query Layer)
47
-
- **Purpose**: Denormalized, indexed data optimized for queries
48
-
- **Storage**: PostgreSQL with application-specific schema
49
-
- **Access**: Through XRPC queries (read-only)
50
-
- **Properties**:
51
-
- Eventually consistent with repositories
52
-
- Can be rebuilt from repository data
53
-
- Application-specific aggregations
39
+
1. **PDS (Personal Data Server)**
40
+
- Managed by Bluesky's official PDS implementation
41
+
- Handles user repositories, DIDs, and CAR file storage
42
+
- Users can use our PDS or any external PDS (federated)
43
+
- Emits events to the Relay/firehose
44
+
45
+
2. **Relay (BigSky)**
46
+
- Aggregates firehose events from multiple PDSs
47
+
- For development: subscribes only to local dev PDS
48
+
- For production: can subscribe to multiple PDSs or public relay
49
+
50
+
3. **AppView Database (Single PostgreSQL)**
51
+
- **Purpose**: Denormalized, indexed data optimized for Coves queries
52
+
- **Storage**: PostgreSQL with Coves-specific schema
53
+
- **Contains**:
54
+
- Indexed posts, communities, feeds
55
+
- User read states and preferences
56
+
- PDS metadata and record references
57
+
- **Properties**:
58
+
- Eventually consistent with PDS repositories
59
+
- Can be rebuilt from firehose replay
60
+
- Application-specific aggregations
61
+
62
+
4. **Coves AppView (Go Application)**
63
+
- Subscribes to Relay firehose
64
+
- Indexes relevant records into PostgreSQL
65
+
- Serves XRPC queries for Coves features
66
+
- Implements custom feed algorithms
54
67
55
68
### Data Flow
56
69
57
70
```
58
71
Write Path:
59
-
Client → XRPC Procedure → Service → Write Repo → CAR Store
60
-
↓
61
-
Firehose Event
62
-
↓
63
-
AppView Indexer
64
-
↓
65
-
AppView Database
72
+
Client → PDS (via XRPC) → Repository Record Created
73
+
↓
74
+
Firehose Event
75
+
↓
76
+
Relay aggregates events
77
+
↓
78
+
Coves AppView subscribes
79
+
↓
80
+
Index in PostgreSQL
66
81
67
82
Read Path:
68
-
Client → XRPC Query → Service → Read Repo → AppView Database
83
+
Client → Coves AppView (via XRPC) → PostgreSQL Query → Response
69
84
```
85
+
86
+
**Key Point**: Coves AppView only reads from the firehose and indexes data. It does NOT write to CAR files or manage repositories directly - the PDS handles that.
70
87
71
88
## Lexicons
72
89
···
138
155
- Procedures often start with `create`, `update`, or `delete`
139
156
- Keep names descriptive but concise
140
157
141
-
## XRPC
142
-
143
-
### What is XRPC?
144
-
XRPC (Cross-Protocol RPC) is AT Protocol's HTTP-based RPC system:
145
-
- All methods live under `/xrpc/` path
146
-
- Method names map directly to Lexicon IDs
147
-
- Supports both JSON and binary data
148
-
149
-
### Request Format
150
-
```
151
-
# Query (GET)
152
-
GET /xrpc/social.coves.community.getCommunity?id=123
153
-
154
-
# Procedure (POST)
155
-
POST /xrpc/social.coves.community.createPost
156
-
Content-Type: application/json
157
-
Authorization: Bearer <token>
158
-
159
-
{"text": "Hello, Coves!"}
160
-
```
161
-
162
-
### Authentication
163
-
- Uses Bearer tokens in Authorization header
164
-
- Tokens are JWTs signed by the user's signing key
165
-
- Service auth for server-to-server calls
166
-
167
-
## Data Storage
168
-
169
-
### CAR Files
170
-
Content Addressable archive files store repository data:
171
-
- Contains IPLD blocks forming a Merkle tree
172
-
- Each block identified by CID (Content IDentifier)
173
-
- Enables cryptographic verification and efficient sync
174
-
175
-
### Record Keys (rkeys)
176
-
- Unique identifiers for records within a collection
177
-
- Can be TIDs (timestamp-based) or custom strings
178
-
- Must match pattern: `[a-zA-Z0-9._~-]{1,512}`
179
-
180
-
### Repository Structure
181
-
```
182
-
Repository (did:plc:user123)
183
-
├── social.coves.post
184
-
│ ├── 3kkreaz3amd27 (TID)
185
-
│ └── 3kkreaz3amd28 (TID)
186
-
├── social.coves.community.member
187
-
│ ├── community123
188
-
│ └── community456
189
-
└── app.bsky.actor.profile
190
-
└── self
191
-
```
192
158
193
159
## Identity & Authentication
194
160
···
204
170
2. HTTPS well-known: `https://alice.com/.well-known/atproto-did`
205
171
206
172
### Authentication Flow
207
-
1. Client creates session with identifier/password
208
-
2. Server returns access/refresh tokens
209
-
3. Client uses access token for API requests
210
-
4. Refresh when access token expires
173
+
1. Client creates session with OAuth
211
174
212
175
## Firehose & Sync
213
176
···
233
196
### Using Indigo Library
234
197
Bluesky's official Go implementation provides:
235
198
- Lexicon code generation
236
-
- CAR file handling
237
199
- XRPC client/server
238
200
- Firehose subscription
239
201
+14
-80
CLAUDE.md
+14
-80
CLAUDE.md
···
13
13
#### Human & LLM Readability Guidelines:
14
14
- Descriptive Naming: Use full words over abbreviations (e.g., CommunityGovernance not CommGov)
15
15
16
-
## Build Process
17
-
18
-
### Phase 1: Planning (Before Writing Code)
19
-
20
-
**ALWAYS START WITH:**
21
-
22
-
- [ ] Identify which atProto patterns apply (check ATPROTO_GUIDE.md or context7 https://context7.com/bluesky-social/atproto)
23
-
- [ ] Check if Indigo (also in context7) packages already solve this: https://context7.com/bluesky-social/indigo
24
-
- [ ] Define the XRPC interface first
25
-
- [ ] Write the Lexicon schema
26
-
- [ ] Plan the data flow: CAR store → AppView
27
-
- [ ] - Follow the two-database pattern: Repository (CAR files)(PostgreSQL for metadata) and AppView (PostgreSQL)
28
-
- [ ] **Identify auth requirements and data sensitivity**
29
-
30
-
### Phase 2: Test-First Implementation
31
-
32
-
**BUILD ORDER:**
33
-
34
-
1. **Domain Model** (`core/[domain]/[domain].go`)
35
-
36
-
- Start with the simplest struct
37
-
- Add validation methods
38
-
- Define error types
39
-
- **Add input validation from the start**
40
-
2. **Repository Interfaces** (`core/[domain]/repository.go`)
41
-
42
-
```go
43
-
type CommunityWriteRepository interface {
44
-
Create(ctx context.Context, community *Community) error
45
-
Update(ctx context.Context, community *Community) error
46
-
}
47
-
48
-
type CommunityReadRepository interface {
49
-
GetByID(ctx context.Context, id string) (*Community, error)
50
-
List(ctx context.Context, limit, offset int) ([]*Community, error)
51
-
}
52
-
```
53
-
54
-
3. **Service Tests** (`core/[domain]/service_test.go`)
55
-
56
-
- Write failing tests for happy path
57
-
- **Add tests for invalid inputs**
58
-
- **Add tests for unauthorized access**
59
-
- Mock repositories
60
-
4. **Service Implementation** (`core/[domain]/service.go`)
61
-
62
-
- Implement to pass tests
63
-
- **Validate all inputs before processing**
64
-
- **Check permissions before operations**
65
-
- Handle transactions
66
-
5. **Repository Implementations**
16
+
## atProto Essentials for Coves
67
17
68
-
- **Always use parameterized queries**
69
-
- **Never concatenate user input into queries**
70
-
- Write repo: `internal/atproto/carstore/[domain]_write_repo.go`
71
-
- Read repo: `db/appview/[domain]_read_repo.go`
72
-
6. **XRPC Handler** (`xrpc/handlers/[domain]_handler.go`)
18
+
### Architecture
19
+
- **PDS is Self-Contained**: Uses internal SQLite + CAR files (in Docker volume)
20
+
- **PostgreSQL for AppView Only**: One database for Coves AppView indexing
21
+
- **Don't Touch PDS Internals**: PDS manages its own storage, we just read from firehose
22
+
- **Data Flow**: Client → PDS → Firehose → AppView → PostgreSQL
73
23
74
-
- **Verify auth tokens/DIDs**
75
-
- Parse XRPC request
76
-
- Call service
77
-
- **Sanitize errors before responding**
78
-
79
-
### Phase 3: Integration
80
-
81
-
**WIRE IT UP:**
82
-
83
-
- [ ] Add to dependency injection in main.go
84
-
- [ ] Register XRPC routes with proper auth middleware
85
-
- [ ] Create migration if needed
86
-
- [ ] Write integration test including auth flows
24
+
### Always Consider:
25
+
- [ ] **Identity**: Every action needs DID verification
26
+
- [ ] **Record Types**: Define custom lexicons (e.g., `social.coves.post`, `social.coves.community`)
27
+
- [ ] **Is it federated-friendly?** (Can other PDSs interact with it?)
28
+
- [ ] **Does the Lexicon make sense?** (Would it work for other forums?)
29
+
- [ ] **AppView only indexes**: We don't write to CAR files, only read from firehose
87
30
88
31
## Security-First Building
89
32
···
104
47
- Error messages with internal details → Wrap errors properly
105
48
- Unbounded queries → Add limits/pagination
106
49
107
-
## Quick Decision Guide
108
-
109
-
### "Should I use X?"
110
-
111
-
1. Does Indigo have it? → Use it
112
-
2. Can PostgreSQL + Go do it securely? → Build it simple
113
-
3. Requires external dependency? → Check Context7 first
114
-
115
50
### "How should I structure this?"
116
51
117
52
1. One domain, one package
···
134
69
135
70
- [ ] Tests pass (including security tests)
136
71
- [ ] Follows atProto patterns
137
-
- [ ] No security checklist items missed
138
72
- [ ] Handles errors gracefully
139
73
- [ ] Works end-to-end with auth
140
74
141
75
## Quick Checks Before Committing
142
76
143
77
1. **Will it work?** (Integration test proves it)
144
-
2. 1. **Is it secure?** (Auth, validation, parameterized queries)
78
+
2. **Is it secure?** (Auth, validation, parameterized queries)
145
79
3. **Is it simple?** (Could you explain to a junior?)
146
80
4. **Is it complete?** (Test, implementation, documentation)
147
81
148
-
Remember: We're building a working product. Perfect is the enemy of shipped.
82
+
Remember: We're building a working product. Perfect is the enemy of shipped.
+161
Makefile
+161
Makefile
···
1
+
.PHONY: help dev-up dev-down dev-logs dev-status dev-reset dev-db-up dev-db-down dev-db-reset test clean
2
+
3
+
# Default target - show help
4
+
.DEFAULT_GOAL := help
5
+
6
+
# Colors for output
7
+
CYAN := \033[36m
8
+
RESET := \033[0m
9
+
GREEN := \033[32m
10
+
YELLOW := \033[33m
11
+
12
+
##@ General
13
+
14
+
help: ## Show this help message
15
+
@echo ""
16
+
@echo "$(CYAN)Coves Development Commands$(RESET)"
17
+
@echo ""
18
+
@awk 'BEGIN {FS = ":.*##"; printf "Usage: make $(CYAN)<target>$(RESET)\n"} \
19
+
/^[a-zA-Z_-]+:.*?##/ { printf " $(CYAN)%-15s$(RESET) %s\n", $$1, $$2 } \
20
+
/^##@/ { printf "\n$(YELLOW)%s$(RESET)\n", substr($$0, 5) }' $(MAKEFILE_LIST)
21
+
@echo ""
22
+
23
+
##@ Local Development (atProto Stack)
24
+
25
+
dev-up: ## Start PDS for local development
26
+
@echo "$(GREEN)Starting Coves development stack...$(RESET)"
27
+
@echo "$(YELLOW)Note: Make sure PostgreSQL is running on port 5433$(RESET)"
28
+
@echo "Run 'make dev-db-up' if database is not running"
29
+
@docker-compose -f docker-compose.dev.yml --env-file .env.dev up -d pds
30
+
@echo ""
31
+
@echo "$(GREEN)✓ Development stack started!$(RESET)"
32
+
@echo ""
33
+
@echo "Services available at:"
34
+
@echo " - PDS (XRPC): http://localhost:3001"
35
+
@echo " - PDS Firehose (WS): ws://localhost:3001/xrpc/com.atproto.sync.subscribeRepos"
36
+
@echo " - AppView (API): http://localhost:8081 (when uncommented)"
37
+
@echo ""
38
+
@echo "Run 'make dev-logs' to view logs"
39
+
40
+
dev-down: ## Stop the atProto development stack
41
+
@echo "$(YELLOW)Stopping Coves development stack...$(RESET)"
42
+
@docker-compose -f docker-compose.dev.yml down
43
+
@echo "$(GREEN)✓ Development stack stopped$(RESET)"
44
+
45
+
dev-logs: ## Tail logs from all development services
46
+
@docker-compose -f docker-compose.dev.yml logs -f
47
+
48
+
dev-status: ## Show status of all development containers
49
+
@echo "$(CYAN)Development Stack Status:$(RESET)"
50
+
@docker-compose -f docker-compose.dev.yml ps
51
+
@echo ""
52
+
@echo "$(CYAN)Database Status:$(RESET)"
53
+
@cd internal/db/local_dev_db_compose && docker-compose ps
54
+
55
+
dev-reset: ## Nuclear option - stop everything and remove all volumes
56
+
@echo "$(YELLOW)⚠️ WARNING: This will delete all PDS data and volumes!$(RESET)"
57
+
@read -p "Are you sure? (y/N): " confirm && [ "$$confirm" = "y" ] || exit 1
58
+
@echo "$(YELLOW)Stopping and removing containers and volumes...$(RESET)"
59
+
@docker-compose -f docker-compose.dev.yml down -v
60
+
@echo "$(GREEN)✓ Reset complete - all data removed$(RESET)"
61
+
@echo "Run 'make dev-up' to start fresh"
62
+
63
+
##@ Database Management
64
+
65
+
dev-db-up: ## Start local PostgreSQL database (port 5433)
66
+
@echo "$(GREEN)Starting local PostgreSQL database...$(RESET)"
67
+
@cd internal/db/local_dev_db_compose && docker-compose up -d
68
+
@echo "$(GREEN)✓ Database started on port 5433$(RESET)"
69
+
@echo "Connection: postgresql://dev_user:dev_password@localhost:5433/coves_dev"
70
+
71
+
dev-db-down: ## Stop local PostgreSQL database
72
+
@echo "$(YELLOW)Stopping local PostgreSQL database...$(RESET)"
73
+
@cd internal/db/local_dev_db_compose && docker-compose down
74
+
@echo "$(GREEN)✓ Database stopped$(RESET)"
75
+
76
+
dev-db-reset: ## Reset database (delete all data and restart)
77
+
@echo "$(YELLOW)⚠️ WARNING: This will delete all database data!$(RESET)"
78
+
@read -p "Are you sure? (y/N): " confirm && [ "$$confirm" = "y" ] || exit 1
79
+
@echo "$(YELLOW)Resetting database...$(RESET)"
80
+
@cd internal/db/local_dev_db_compose && docker-compose down -v
81
+
@cd internal/db/local_dev_db_compose && docker-compose up -d
82
+
@echo "$(GREEN)✓ Database reset complete$(RESET)"
83
+
84
+
##@ Testing
85
+
86
+
test: ## Run all tests with test database
87
+
@echo "$(GREEN)Starting test database...$(RESET)"
88
+
@cd internal/db/test_db_compose && ./start-test-db.sh
89
+
@echo "$(GREEN)Running tests...$(RESET)"
90
+
@./run-tests.sh
91
+
@echo "$(GREEN)✓ Tests complete$(RESET)"
92
+
93
+
test-db-reset: ## Reset test database
94
+
@echo "$(GREEN)Resetting test database...$(RESET)"
95
+
@cd internal/db/test_db_compose && ./reset-test-db.sh
96
+
@echo "$(GREEN)✓ Test database reset$(RESET)"
97
+
98
+
##@ Build & Run
99
+
100
+
build: ## Build the Coves server
101
+
@echo "$(GREEN)Building Coves server...$(RESET)"
102
+
@go build -o server ./cmd/server
103
+
@echo "$(GREEN)✓ Build complete: ./server$(RESET)"
104
+
105
+
run: ## Run the Coves server (requires database running)
106
+
@echo "$(GREEN)Starting Coves server...$(RESET)"
107
+
@go run ./cmd/server
108
+
109
+
##@ Cleanup
110
+
111
+
clean: ## Clean build artifacts and temporary files
112
+
@echo "$(YELLOW)Cleaning build artifacts...$(RESET)"
113
+
@rm -f server main validate-lexicon
114
+
@go clean
115
+
@echo "$(GREEN)✓ Clean complete$(RESET)"
116
+
117
+
clean-all: clean ## Clean everything including Docker volumes (DESTRUCTIVE)
118
+
@echo "$(YELLOW)⚠️ WARNING: This will remove ALL Docker volumes!$(RESET)"
119
+
@read -p "Are you sure? (y/N): " confirm && [ "$$confirm" = "y" ] || exit 1
120
+
@make dev-reset
121
+
@make dev-db-reset
122
+
@echo "$(GREEN)✓ All clean$(RESET)"
123
+
124
+
##@ Workflows (Common Tasks)
125
+
126
+
fresh-start: ## Complete fresh start (reset DB, reset stack, start everything)
127
+
@echo "$(CYAN)Starting fresh development environment...$(RESET)"
128
+
@make dev-db-reset
129
+
@make dev-reset || true
130
+
@sleep 2
131
+
@make dev-db-up
132
+
@sleep 2
133
+
@make dev-up
134
+
@echo ""
135
+
@echo "$(GREEN)✓ Fresh environment ready!$(RESET)"
136
+
@make dev-status
137
+
138
+
quick-restart: ## Quick restart of development stack (keeps data)
139
+
@make dev-down
140
+
@make dev-up
141
+
142
+
##@ Utilities
143
+
144
+
validate-lexicon: ## Validate all Lexicon schemas
145
+
@echo "$(GREEN)Validating Lexicon schemas...$(RESET)"
146
+
@./validate-lexicon
147
+
@echo "$(GREEN)✓ Lexicon validation complete$(RESET)"
148
+
149
+
db-shell: ## Open PostgreSQL shell for local database
150
+
@echo "$(CYAN)Connecting to local database...$(RESET)"
151
+
@PGPASSWORD=dev_password psql -h localhost -p 5433 -U dev_user -d coves_dev
152
+
153
+
##@ Documentation
154
+
155
+
docs: ## Open project documentation
156
+
@echo "$(CYAN)Project Documentation:$(RESET)"
157
+
@echo " - Setup Guide: docs/LOCAL_DEVELOPMENT.md"
158
+
@echo " - Project Structure: PROJECT_STRUCTURE.md"
159
+
@echo " - Build Guide: CLAUDE.md"
160
+
@echo " - atProto Guide: ATPROTO_GUIDE.md"
161
+
@echo " - PRD: PRD.md"
-23
PROJECT_STRUCTURE.md
-23
PROJECT_STRUCTURE.md
···
3
3
This document provides an overview of the Coves project directory structure, following atProto architecture patterns.
4
4
5
5
**Legend:**
6
-
- † = Planned but not yet implemented
7
6
- 🔒 = Security-sensitive files
8
7
9
8
```
···
43
42
└── build/ † # Build artifacts
44
43
```
45
44
46
-
## Implementation Status
47
-
48
-
### Completed ✓
49
-
- Basic repository structure
50
-
- User domain models
51
-
- CAR store foundation
52
-
- Lexicon schemas
53
-
- Database migrations
54
-
55
-
### In Progress 🚧
56
-
- Repository service implementation
57
-
- User service
58
-
- Basic authentication
59
-
60
-
### Planned 📋
61
-
- XRPC handlers
62
-
- AppView indexer
63
-
- Firehose implementation
64
-
- Community features
65
-
- Moderation system
66
-
- Feed algorithms
67
45
68
46
## Development Guidelines
69
47
···
72
50
1. **Start with Lexicons**: Define data schemas first
73
51
2. **Implement Core Domain**: Create models and interfaces
74
52
3. **Build Services**: Implement business logic
75
-
4. **Add Repositories**: Create data access layers
76
53
5. **Wire XRPC**: Connect handlers last
+144
docker-compose.dev.yml
+144
docker-compose.dev.yml
···
1
+
version: '3.8'
2
+
3
+
# Coves Local Development Stack
4
+
# Simple setup: PDS + AppView (no relay needed for local dev)
5
+
# AppView subscribes directly to PDS firehose at ws://localhost:3001/xrpc/com.atproto.sync.subscribeRepos
6
+
# Ports configured to avoid conflicts with production PDS on :3000
7
+
8
+
services:
9
+
# Bluesky Personal Data Server (PDS)
10
+
# Handles user repositories, DIDs, and CAR files
11
+
pds:
12
+
image: ghcr.io/bluesky-social/pds:latest
13
+
container_name: coves-dev-pds
14
+
ports:
15
+
- "3001:3000" # PDS XRPC API (avoiding production PDS on :3000)
16
+
environment:
17
+
# PDS Configuration
18
+
PDS_HOSTNAME: ${PDS_HOSTNAME:-localhost}
19
+
PDS_PORT: 3000
20
+
PDS_DATA_DIRECTORY: /pds
21
+
PDS_BLOBSTORE_DISK_LOCATION: /pds/blocks
22
+
PDS_DID_PLC_URL: ${PDS_DID_PLC_URL:-https://plc.directory}
23
+
# PDS_CRAWLERS not needed - we're not using a relay for local dev
24
+
25
+
# Note: PDS uses its own internal SQLite database and CAR file storage
26
+
# Our PostgreSQL database is only for the Coves AppView
27
+
28
+
# JWT secrets (for local dev only)
29
+
PDS_JWT_SECRET: ${PDS_JWT_SECRET:-local-dev-jwt-secret-change-in-production}
30
+
PDS_ADMIN_PASSWORD: ${PDS_ADMIN_PASSWORD:-admin}
31
+
PDS_PLC_ROTATION_KEY_K256_PRIVATE_KEY_HEX: ${PDS_PLC_ROTATION_KEY:-af514fb84c4356241deed29feb392d1ee359f99c05a7b8f7bff2e5f2614f64b2}
32
+
33
+
# Service endpoints
34
+
PDS_SERVICE_HANDLE_DOMAINS: ${PDS_SERVICE_HANDLE_DOMAINS:-.local.coves.dev}
35
+
36
+
# Dev mode settings (allows HTTP instead of HTTPS)
37
+
PDS_DEV_MODE: "true"
38
+
39
+
# Development settings
40
+
NODE_ENV: development
41
+
LOG_ENABLED: "true"
42
+
LOG_LEVEL: ${LOG_LEVEL:-debug}
43
+
volumes:
44
+
- pds-data:/pds
45
+
networks:
46
+
- coves-dev
47
+
healthcheck:
48
+
test: ["CMD", "curl", "-f", "http://localhost:3000/xrpc/_health"]
49
+
interval: 10s
50
+
timeout: 5s
51
+
retries: 5
52
+
53
+
# Indigo Relay (BigSky) - OPTIONAL for local dev
54
+
# WARNING: BigSky is designed to crawl the entire atProto network!
55
+
# For local dev, consider using direct PDS firehose instead (see AppView config below)
56
+
#
57
+
# To use relay: docker-compose -f docker-compose.dev.yml up pds relay
58
+
# To skip relay: docker-compose -f docker-compose.dev.yml up pds
59
+
#
60
+
# If using relay, you MUST manually configure it to only watch local PDS:
61
+
# 1. Start relay
62
+
# 2. Use admin API to block all domains except localhost
63
+
# curl -X POST http://localhost:2471/admin/pds/requestCrawl \
64
+
# -H "Authorization: Bearer dev-admin-key" \
65
+
# -d '{"hostname": "localhost:3001"}'
66
+
relay:
67
+
image: ghcr.io/bluesky-social/indigo:bigsky-0a2d4173e6e89e49b448f6bb0a6e1ab58d12b385
68
+
container_name: coves-dev-relay
69
+
ports:
70
+
- "2471:2470" # Relay firehose WebSocket (avoiding conflicts)
71
+
environment:
72
+
# Relay Configuration
73
+
BGS_ADMIN_KEY: ${BGS_ADMIN_KEY:-dev-admin-key}
74
+
BGS_PORT: 2470
75
+
76
+
# IMPORTANT: Allow insecure WebSocket for local PDS (ws:// instead of wss://)
77
+
BGS_CRAWL_INSECURE_WS: "true"
78
+
79
+
# Database connection (uses shared PostgreSQL for relay state)
80
+
DATABASE_URL: postgresql://${POSTGRES_USER}:${POSTGRES_PASSWORD}@host.docker.internal:${POSTGRES_PORT}/${POSTGRES_DB}?sslmode=disable
81
+
82
+
# Relay will discover PDSs automatically - use admin API to restrict!
83
+
# See comments above for how to configure allowlist
84
+
85
+
# Development settings
86
+
LOG_LEVEL: ${LOG_LEVEL:-debug}
87
+
networks:
88
+
- coves-dev
89
+
extra_hosts:
90
+
- "host.docker.internal:host-gateway"
91
+
depends_on:
92
+
pds:
93
+
condition: service_healthy
94
+
healthcheck:
95
+
test: ["CMD", "curl", "-f", "http://localhost:2470/xrpc/_health"]
96
+
interval: 10s
97
+
timeout: 5s
98
+
retries: 5
99
+
# Mark as optional - start with: docker-compose up pds relay
100
+
profiles:
101
+
- relay
102
+
103
+
# Coves AppView (Your Go Application)
104
+
# Subscribes to PDS firehose and indexes Coves-specific data
105
+
# Note: Uncomment when you have a Dockerfile for the AppView
106
+
# appview:
107
+
# build:
108
+
# context: .
109
+
# dockerfile: Dockerfile
110
+
# container_name: coves-dev-appview
111
+
# ports:
112
+
# - "8081:8080" # AppView API (avoiding conflicts)
113
+
# environment:
114
+
# # Database connection
115
+
# DATABASE_URL: postgresql://${POSTGRES_USER}:${POSTGRES_PASSWORD}@host.docker.internal:${POSTGRES_PORT}/${POSTGRES_DB}?sslmode=disable
116
+
#
117
+
# # PDS Firehose subscription (direct, no relay)
118
+
# FIREHOSE_URL: ws://pds:3000/xrpc/com.atproto.sync.subscribeRepos
119
+
#
120
+
# # PDS connection (for XRPC calls)
121
+
# PDS_URL: http://pds:3000
122
+
#
123
+
# # Application settings
124
+
# PORT: 8080
125
+
# ENV: development
126
+
# LOG_LEVEL: ${LOG_LEVEL:-debug}
127
+
# networks:
128
+
# - coves-dev
129
+
# extra_hosts:
130
+
# - "host.docker.internal:host-gateway"
131
+
# depends_on:
132
+
# - pds
133
+
134
+
# Note: PostgreSQL runs separately via internal/db/local_dev_db_compose/
135
+
# This stack connects to it via host.docker.internal:5433
136
+
137
+
networks:
138
+
coves-dev:
139
+
driver: bridge
140
+
name: coves-dev-network
141
+
142
+
volumes:
143
+
pds-data:
144
+
name: coves-dev-pds-data
+450
docs/LOCAL_DEVELOPMENT.md
+450
docs/LOCAL_DEVELOPMENT.md
···
1
+
# Coves Local Development Guide
2
+
3
+
Complete guide for setting up and running the Coves atProto development environment.
4
+
5
+
## Table of Contents
6
+
- [Quick Start](#quick-start)
7
+
- [Architecture Overview](#architecture-overview)
8
+
- [Prerequisites](#prerequisites)
9
+
- [Setup Instructions](#setup-instructions)
10
+
- [Using the Makefile](#using-the-makefile)
11
+
- [Development Workflow](#development-workflow)
12
+
- [Troubleshooting](#troubleshooting)
13
+
- [Environment Variables](#environment-variables)
14
+
15
+
## Quick Start
16
+
17
+
```bash
18
+
# 1. Start the PostgreSQL database
19
+
make dev-db-up
20
+
21
+
# 2. Start the PDS
22
+
make dev-up
23
+
24
+
# 3. View logs
25
+
make dev-logs
26
+
27
+
# 4. Check status
28
+
make dev-status
29
+
30
+
# 5. When done
31
+
make dev-down
32
+
```
33
+
34
+
## Architecture Overview
35
+
36
+
Coves uses a simplified single-database architecture with direct PDS firehose subscription:
37
+
38
+
```
39
+
┌─────────────────────────────────────────────┐
40
+
│ Coves Local Development Stack │
41
+
├─────────────────────────────────────────────┤
42
+
│ │
43
+
│ ┌──────────────┐ │
44
+
│ │ PDS │ │
45
+
│ │ :3001 │ │
46
+
│ │ │ │
47
+
│ │ Firehose───────────┐ │
48
+
│ └──────────────┘ │ │
49
+
│ │ │
50
+
│ ▼ │
51
+
│ ┌──────────────┐ │
52
+
│ │ Coves AppView│ │
53
+
│ │ (Go) │ │
54
+
│ │ :8081 │ │
55
+
│ └──────┬───────┘ │
56
+
│ │ │
57
+
│ ┌──────▼───────┐ │
58
+
│ │ PostgreSQL │ │
59
+
│ │ :5433 │ │
60
+
│ └──────────────┘ │
61
+
│ │
62
+
└─────────────────────────────────────────────┘
63
+
64
+
Your Production PDS (:3000) ← Runs independently
65
+
```
66
+
67
+
### Components
68
+
69
+
1. **PDS (Port 3001)** - Bluesky's Personal Data Server with:
70
+
- User repositories and CAR files (stored in Docker volume)
71
+
- Internal SQLite database for PDS metadata
72
+
- Firehose WebSocket: `ws://localhost:3001/xrpc/com.atproto.sync.subscribeRepos`
73
+
2. **PostgreSQL (Port 5433)** - Database for Coves AppView data only
74
+
3. **Coves AppView (Port 8081)** - Your Go application that:
75
+
- Subscribes directly to PDS firehose
76
+
- Indexes Coves-specific data to PostgreSQL
77
+
78
+
**Key Points:**
79
+
- ✅ Ports chosen to avoid conflicts with production PDS on :3000
80
+
- ✅ PDS is self-contained with its own SQLite database and CAR storage
81
+
- ✅ PostgreSQL is only used by the Coves AppView for indexing
82
+
- ✅ AppView subscribes directly to PDS firehose (no relay needed)
83
+
- ✅ Simple, clean architecture for local development
84
+
85
+
## Prerequisites
86
+
87
+
- **Docker & Docker Compose** - For running containerized services
88
+
- **Go 1.22+** - For building the Coves AppView
89
+
- **PostgreSQL client** (optional) - For database inspection
90
+
- **Make** (optional but recommended) - For convenient commands
91
+
92
+
## Setup Instructions
93
+
94
+
### Step 1: Start the Database
95
+
96
+
The PostgreSQL database must be running first:
97
+
98
+
```bash
99
+
# Start the database
100
+
make dev-db-up
101
+
102
+
# Verify it's running
103
+
make dev-status
104
+
```
105
+
106
+
**Connection Details:**
107
+
- Host: `localhost`
108
+
- Port: `5433`
109
+
- Database: `coves_dev`
110
+
- User: `dev_user`
111
+
- Password: `dev_password`
112
+
113
+
### Step 2: Start the PDS
114
+
115
+
Start the Personal Data Server:
116
+
117
+
```bash
118
+
# Start PDS
119
+
make dev-up
120
+
121
+
# View logs (follows in real-time)
122
+
make dev-logs
123
+
```
124
+
125
+
Wait for health checks to pass (~10-30 seconds).
126
+
127
+
### Step 3: Verify Services
128
+
129
+
```bash
130
+
# Check PDS is running
131
+
make dev-status
132
+
133
+
# Test PDS health endpoint
134
+
curl http://localhost:3001/xrpc/_health
135
+
136
+
# Test PDS firehose endpoint (should get WebSocket upgrade response)
137
+
curl -i -N -H "Connection: Upgrade" -H "Upgrade: websocket" \
138
+
http://localhost:3001/xrpc/com.atproto.sync.subscribeRepos
139
+
```
140
+
141
+
### Step 4: Run Coves AppView (When Ready)
142
+
143
+
When you have a Dockerfile for the AppView:
144
+
145
+
1. Uncomment the `appview` service in `docker-compose.dev.yml`
146
+
2. Restart the stack: `make dev-down && make dev-up`
147
+
148
+
Or run the AppView locally:
149
+
150
+
```bash
151
+
# Set environment variables
152
+
export DATABASE_URL="postgresql://dev_user:dev_password@localhost:5433/coves_dev?sslmode=disable"
153
+
export FIREHOSE_URL="ws://localhost:3001/xrpc/com.atproto.sync.subscribeRepos"
154
+
export PDS_URL="http://localhost:3001"
155
+
export PORT=8081
156
+
157
+
# Run the AppView
158
+
go run ./cmd/server
159
+
```
160
+
161
+
## Using the Makefile
162
+
163
+
The Makefile provides convenient commands for development. Run `make help` to see all available commands:
164
+
165
+
### General Commands
166
+
167
+
```bash
168
+
make help # Show all available commands with descriptions
169
+
```
170
+
171
+
### Development Stack Commands
172
+
173
+
```bash
174
+
make dev-up # Start PDS for local development
175
+
make dev-down # Stop the stack
176
+
make dev-logs # Tail logs from PDS
177
+
make dev-status # Show status of containers
178
+
make dev-reset # Nuclear option - remove all data and volumes
179
+
```
180
+
181
+
### Database Commands
182
+
183
+
```bash
184
+
make dev-db-up # Start PostgreSQL database
185
+
make dev-db-down # Stop PostgreSQL database
186
+
make dev-db-reset # Reset database (delete all data)
187
+
make db-shell # Open psql shell to the database
188
+
```
189
+
190
+
### Testing Commands
191
+
192
+
```bash
193
+
make test # Run all tests with test database
194
+
make test-db-reset # Reset test database
195
+
```
196
+
197
+
### Workflow Commands
198
+
199
+
```bash
200
+
make fresh-start # Complete fresh start (reset everything)
201
+
make quick-restart # Quick restart (keeps data)
202
+
```
203
+
204
+
### Build Commands
205
+
206
+
```bash
207
+
make build # Build the Coves server binary
208
+
make run # Run the Coves server
209
+
make clean # Clean build artifacts
210
+
```
211
+
212
+
### Utilities
213
+
214
+
```bash
215
+
make validate-lexicon # Validate all Lexicon schemas
216
+
make docs # Show documentation file locations
217
+
```
218
+
219
+
## Development Workflow
220
+
221
+
### Typical Development Session
222
+
223
+
```bash
224
+
# 1. Start fresh environment
225
+
make fresh-start
226
+
227
+
# 2. Work on code...
228
+
229
+
# 3. Restart services as needed
230
+
make quick-restart
231
+
232
+
# 4. View logs
233
+
make dev-logs
234
+
235
+
# 5. Run tests
236
+
make test
237
+
238
+
# 6. Clean up when done
239
+
make dev-down
240
+
```
241
+
242
+
### Testing Lexicon Changes
243
+
244
+
```bash
245
+
# 1. Edit Lexicon files in internal/atproto/lexicon/
246
+
247
+
# 2. Validate schemas
248
+
make validate-lexicon
249
+
250
+
# 3. Restart services to pick up changes
251
+
make quick-restart
252
+
```
253
+
254
+
### Database Inspection
255
+
256
+
```bash
257
+
# Open PostgreSQL shell
258
+
make db-shell
259
+
260
+
# Or use psql directly
261
+
PGPASSWORD=dev_password psql -h localhost -p 5433 -U dev_user -d coves_dev
262
+
```
263
+
264
+
### Viewing Logs
265
+
266
+
```bash
267
+
# Follow all logs
268
+
make dev-logs
269
+
270
+
# Or use docker-compose directly
271
+
docker-compose -f docker-compose.dev.yml logs -f pds
272
+
docker-compose -f docker-compose.dev.yml logs -f relay
273
+
```
274
+
275
+
## Troubleshooting
276
+
277
+
### Port Already in Use
278
+
279
+
**Problem:** Error binding to port 3000, 5433, etc.
280
+
281
+
**Solution:**
282
+
- The dev environment uses non-standard ports to avoid conflicts
283
+
- PDS: 3001 (not 3000)
284
+
- PostgreSQL: 5433 (not 5432)
285
+
- Relay: 2471 (not 2470)
286
+
- AppView: 8081 (not 8080)
287
+
288
+
If you still have conflicts, check what's using the port:
289
+
290
+
```bash
291
+
# Check what's using a port
292
+
lsof -i :3001
293
+
lsof -i :5433
294
+
295
+
# Kill the process
296
+
kill -9 <PID>
297
+
```
298
+
299
+
### Database Connection Failed
300
+
301
+
**Problem:** Services can't connect to PostgreSQL
302
+
303
+
**Solution:**
304
+
305
+
```bash
306
+
# Ensure database is running
307
+
make dev-db-up
308
+
309
+
# Check database logs
310
+
cd internal/db/local_dev_db_compose && docker-compose logs
311
+
312
+
# Verify connection manually
313
+
PGPASSWORD=dev_password psql -h localhost -p 5433 -U dev_user -d coves_dev
314
+
```
315
+
316
+
### PDS Health Check Failing
317
+
318
+
**Problem:** PDS container keeps restarting
319
+
320
+
**Solution:**
321
+
322
+
```bash
323
+
# Check PDS logs
324
+
docker-compose -f docker-compose.dev.yml logs pds
325
+
326
+
# Common issues:
327
+
# 1. Database not accessible - ensure DB is running
328
+
# 2. Invalid environment variables - check .env.dev
329
+
# 3. Port conflict - ensure port 3001 is free
330
+
```
331
+
332
+
### AppView Not Receiving Firehose Events
333
+
334
+
**Problem:** AppView isn't receiving events from PDS firehose
335
+
336
+
**Solution:**
337
+
338
+
```bash
339
+
# Check PDS logs for firehose activity
340
+
docker-compose -f docker-compose.dev.yml logs pds
341
+
342
+
# Verify firehose endpoint is accessible
343
+
curl -i -N -H "Connection: Upgrade" -H "Upgrade: websocket" \
344
+
http://localhost:3001/xrpc/com.atproto.sync.subscribeRepos
345
+
346
+
# Check AppView is connecting to correct URL:
347
+
# FIREHOSE_URL=ws://localhost:3001/xrpc/com.atproto.sync.subscribeRepos
348
+
```
349
+
350
+
### Fresh Start Not Working
351
+
352
+
**Problem:** `make fresh-start` fails
353
+
354
+
**Solution:**
355
+
356
+
```bash
357
+
# Manually clean everything
358
+
docker-compose -f docker-compose.dev.yml down -v
359
+
cd internal/db/local_dev_db_compose && docker-compose down -v
360
+
docker volume prune -f
361
+
docker network prune -f
362
+
363
+
# Then start fresh
364
+
make dev-db-up
365
+
sleep 2
366
+
make dev-up
367
+
```
368
+
369
+
### Production PDS Interference
370
+
371
+
**Problem:** Dev environment conflicts with your production PDS
372
+
373
+
**Solution:**
374
+
- Dev PDS runs on port 3001 (production is 3000)
375
+
- Dev services use different handle domain (`.local.coves.dev`)
376
+
- They should not interfere unless you have custom networking
377
+
378
+
```bash
379
+
# Verify production PDS is still accessible
380
+
curl http://localhost:3000/xrpc/_health
381
+
382
+
# Verify dev PDS is separate
383
+
curl http://localhost:3001/xrpc/_health
384
+
```
385
+
386
+
## Environment Variables
387
+
388
+
All configuration is in `.env.dev`:
389
+
390
+
### Database Configuration
391
+
```bash
392
+
POSTGRES_HOST=localhost
393
+
POSTGRES_PORT=5433
394
+
POSTGRES_DB=coves_dev
395
+
POSTGRES_USER=dev_user
396
+
POSTGRES_PASSWORD=dev_password
397
+
```
398
+
399
+
### PDS Configuration
400
+
```bash
401
+
PDS_HOSTNAME=localhost
402
+
PDS_PORT=3001
403
+
PDS_JWT_SECRET=local-dev-jwt-secret-change-in-production
404
+
PDS_ADMIN_PASSWORD=admin
405
+
PDS_SERVICE_HANDLE_DOMAINS=.local.coves.dev
406
+
```
407
+
408
+
### Relay Configuration
409
+
```bash
410
+
BGS_PORT=2471
411
+
BGS_ADMIN_KEY=dev-admin-key
412
+
```
413
+
414
+
### AppView Configuration
415
+
```bash
416
+
APPVIEW_PORT=8081
417
+
FIREHOSE_URL=ws://localhost:3001/xrpc/com.atproto.sync.subscribeRepos
418
+
PDS_URL=http://localhost:3001
419
+
```
420
+
421
+
### Development Settings
422
+
```bash
423
+
ENV=development
424
+
LOG_LEVEL=debug
425
+
```
426
+
427
+
## Next Steps
428
+
429
+
1. **Build the Firehose Subscriber** - Create the AppView component that subscribes to the relay
430
+
2. **Define Custom Lexicons** - Create Coves-specific schemas in `internal/atproto/lexicon/social/coves/`
431
+
3. **Implement XRPC Handlers** - Build the API endpoints for Coves features
432
+
4. **Create Integration Tests** - Use Testcontainers to test the full stack
433
+
434
+
## Additional Resources
435
+
436
+
- [CLAUDE.md](../CLAUDE.md) - Build guidelines and security practices
437
+
- [ATPROTO_GUIDE.md](../ATPROTO_GUIDE.md) - Comprehensive atProto implementation guide
438
+
- [PROJECT_STRUCTURE.md](../PROJECT_STRUCTURE.md) - Project organization
439
+
- [PRD.md](../PRD.md) - Product requirements and roadmap
440
+
441
+
## Getting Help
442
+
443
+
- Check logs: `make dev-logs`
444
+
- View status: `make dev-status`
445
+
- Reset everything: `make fresh-start`
446
+
- Inspect database: `make db-shell`
447
+
448
+
For issues with atProto concepts, see [ATPROTO_GUIDE.md](../ATPROTO_GUIDE.md).
449
+
450
+
For build process questions, see [CLAUDE.md](../CLAUDE.md).
-104
internal/atproto/carstore/README.md
-104
internal/atproto/carstore/README.md
···
1
-
# CarStore Package
2
-
3
-
This package provides integration with Indigo's carstore for managing ATProto repository CAR files in the Coves platform.
4
-
5
-
## Overview
6
-
7
-
The carstore package wraps Indigo's carstore implementation to provide:
8
-
- Filesystem-based storage of CAR (Content Addressable aRchive) files
9
-
- PostgreSQL metadata tracking via GORM
10
-
- DID to UID mapping for user repositories
11
-
- Automatic garbage collection and compaction
12
-
13
-
## Architecture
14
-
15
-
```
16
-
[Repository Service]
17
-
↓
18
-
[RepoStore] ← Provides DID-based interface
19
-
↓
20
-
[CarStore] ← Wraps Indigo's carstore
21
-
↓
22
-
[Indigo CarStore] ← Actual implementation
23
-
↓
24
-
[PostgreSQL + Filesystem]
25
-
```
26
-
27
-
## Components
28
-
29
-
### CarStore (`carstore.go`)
30
-
Wraps Indigo's carstore implementation, providing methods for:
31
-
- `ImportSlice`: Import CAR data for a user
32
-
- `ReadUserCar`: Export user's repository as CAR
33
-
- `GetUserRepoHead`: Get latest repository state
34
-
- `CompactUserShards`: Run garbage collection
35
-
- `WipeUserData`: Delete all user data
36
-
37
-
### UserMapping (`user_mapping.go`)
38
-
Maps DIDs (Decentralized Identifiers) to numeric UIDs required by Indigo's carstore:
39
-
- DIDs are strings like `did:plc:abc123xyz`
40
-
- UIDs are numeric identifiers (models.Uid)
41
-
- Maintains bidirectional mapping in PostgreSQL
42
-
43
-
### RepoStore (`repo_store.go`)
44
-
Combines CarStore with UserMapping to provide DID-based operations:
45
-
- `ImportRepo`: Import repository for a DID
46
-
- `ReadRepo`: Export repository for a DID
47
-
- `GetRepoHead`: Get latest state for a DID
48
-
- `CompactRepo`: Run garbage collection for a DID
49
-
- `DeleteRepo`: Remove all data for a DID
50
-
51
-
## Data Flow
52
-
53
-
### Creating a New Repository
54
-
1. Service calls `RepoStore.ImportRepo(did, carData)`
55
-
2. RepoStore maps DID to UID via UserMapping
56
-
3. CarStore imports the CAR slice
57
-
4. Indigo's carstore:
58
-
- Stores CAR data as file on disk
59
-
- Records metadata in PostgreSQL
60
-
61
-
### Reading a Repository
62
-
1. Service calls `RepoStore.ReadRepo(did)`
63
-
2. RepoStore maps DID to UID
64
-
3. CarStore reads user's CAR data
65
-
4. Returns complete CAR file
66
-
67
-
## Database Schema
68
-
69
-
### user_maps table
70
-
```sql
71
-
CREATE TABLE user_maps (
72
-
uid SERIAL PRIMARY KEY,
73
-
did VARCHAR UNIQUE NOT NULL,
74
-
created_at BIGINT,
75
-
updated_at BIGINT
76
-
);
77
-
```
78
-
79
-
### Indigo's tables (auto-created)
80
-
- `car_shards`: Metadata about CAR file shards
81
-
- `block_refs`: Block reference tracking
82
-
83
-
## Storage
84
-
85
-
CAR files are stored on the filesystem at the path specified during initialization (e.g., `./data/carstore/`). The storage is organized by Indigo's carstore implementation, typically with sharding for performance.
86
-
87
-
## Configuration
88
-
89
-
Initialize the carstore with:
90
-
```go
91
-
carDirs := []string{"./data/carstore"}
92
-
repoStore, err := carstore.NewRepoStore(gormDB, carDirs)
93
-
```
94
-
95
-
## Future Enhancements
96
-
97
-
Current implementation supports repository-level operations. Record-level CRUD operations would require:
98
-
1. Reading the CAR file
99
-
2. Parsing into a repository structure
100
-
3. Modifying records
101
-
4. Re-serializing as CAR
102
-
5. Writing back to carstore
103
-
104
-
This is planned for future XRPC implementation.
-100
internal/atproto/carstore/carstore.go
-100
internal/atproto/carstore/carstore.go
···
1
-
package carstore
2
-
3
-
import (
4
-
"context"
5
-
"fmt"
6
-
"io"
7
-
8
-
"github.com/bluesky-social/indigo/carstore"
9
-
"github.com/bluesky-social/indigo/models"
10
-
"github.com/ipfs/go-cid"
11
-
"gorm.io/gorm"
12
-
)
13
-
14
-
// CarStore wraps Indigo's carstore for managing ATProto repository CAR files
15
-
type CarStore struct {
16
-
cs carstore.CarStore
17
-
}
18
-
19
-
// NewCarStore creates a new CarStore instance using Indigo's implementation
20
-
func NewCarStore(db *gorm.DB, carDirs []string) (*CarStore, error) {
21
-
// Initialize Indigo's carstore
22
-
cs, err := carstore.NewCarStore(db, carDirs)
23
-
if err != nil {
24
-
return nil, fmt.Errorf("initializing carstore: %w", err)
25
-
}
26
-
27
-
return &CarStore{
28
-
cs: cs,
29
-
}, nil
30
-
}
31
-
32
-
// ImportSlice imports a CAR file slice for a user
33
-
func (c *CarStore) ImportSlice(ctx context.Context, uid models.Uid, since *string, carData []byte) (cid.Cid, error) {
34
-
rootCid, _, err := c.cs.ImportSlice(ctx, uid, since, carData)
35
-
if err != nil {
36
-
return cid.Undef, fmt.Errorf("importing CAR slice for UID %d: %w", uid, err)
37
-
}
38
-
return rootCid, nil
39
-
}
40
-
41
-
// ReadUserCar reads a user's repository CAR file
42
-
func (c *CarStore) ReadUserCar(ctx context.Context, uid models.Uid, sinceRev string, incremental bool, w io.Writer) error {
43
-
if err := c.cs.ReadUserCar(ctx, uid, sinceRev, incremental, w); err != nil {
44
-
return fmt.Errorf("reading user CAR for UID %d: %w", uid, err)
45
-
}
46
-
return nil
47
-
}
48
-
49
-
// GetUserRepoHead gets the latest repository head CID for a user
50
-
func (c *CarStore) GetUserRepoHead(ctx context.Context, uid models.Uid) (cid.Cid, error) {
51
-
head, err := c.cs.GetUserRepoHead(ctx, uid)
52
-
if err != nil {
53
-
return cid.Undef, fmt.Errorf("getting repo head for UID %d: %w", uid, err)
54
-
}
55
-
return head, nil
56
-
}
57
-
58
-
// CompactUserShards performs garbage collection and compaction for a user's data
59
-
func (c *CarStore) CompactUserShards(ctx context.Context, uid models.Uid, aggressive bool) error {
60
-
_, err := c.cs.CompactUserShards(ctx, uid, aggressive)
61
-
if err != nil {
62
-
return fmt.Errorf("compacting shards for UID %d: %w", uid, err)
63
-
}
64
-
return nil
65
-
}
66
-
67
-
// WipeUserData removes all data for a user
68
-
func (c *CarStore) WipeUserData(ctx context.Context, uid models.Uid) error {
69
-
if err := c.cs.WipeUserData(ctx, uid); err != nil {
70
-
return fmt.Errorf("wiping data for UID %d: %w", uid, err)
71
-
}
72
-
return nil
73
-
}
74
-
75
-
// NewDeltaSession creates a new session for writing deltas
76
-
func (c *CarStore) NewDeltaSession(ctx context.Context, uid models.Uid, since *string) (*carstore.DeltaSession, error) {
77
-
session, err := c.cs.NewDeltaSession(ctx, uid, since)
78
-
if err != nil {
79
-
return nil, fmt.Errorf("creating delta session for UID %d: %w", uid, err)
80
-
}
81
-
return session, nil
82
-
}
83
-
84
-
// ReadOnlySession creates a read-only session for reading user data
85
-
func (c *CarStore) ReadOnlySession(uid models.Uid) (*carstore.DeltaSession, error) {
86
-
session, err := c.cs.ReadOnlySession(uid)
87
-
if err != nil {
88
-
return nil, fmt.Errorf("creating read-only session for UID %d: %w", uid, err)
89
-
}
90
-
return session, nil
91
-
}
92
-
93
-
// Stat returns statistics about the carstore
94
-
func (c *CarStore) Stat(ctx context.Context, uid models.Uid) ([]carstore.UserStat, error) {
95
-
stats, err := c.cs.Stat(ctx, uid)
96
-
if err != nil {
97
-
return nil, fmt.Errorf("getting stats for UID %d: %w", uid, err)
98
-
}
99
-
return stats, nil
100
-
}
-122
internal/atproto/carstore/repo_store.go
-122
internal/atproto/carstore/repo_store.go
···
1
-
package carstore
2
-
3
-
import (
4
-
"bytes"
5
-
"context"
6
-
"fmt"
7
-
"io"
8
-
9
-
"github.com/bluesky-social/indigo/models"
10
-
"github.com/ipfs/go-cid"
11
-
"gorm.io/gorm"
12
-
)
13
-
14
-
// RepoStore combines CarStore with UserMapping to provide DID-based repository storage
15
-
type RepoStore struct {
16
-
cs *CarStore
17
-
mapping *UserMapping
18
-
}
19
-
20
-
// NewRepoStore creates a new RepoStore instance
21
-
func NewRepoStore(db *gorm.DB, carDirs []string) (*RepoStore, error) {
22
-
// Create carstore
23
-
cs, err := NewCarStore(db, carDirs)
24
-
if err != nil {
25
-
return nil, fmt.Errorf("creating carstore: %w", err)
26
-
}
27
-
28
-
// Create user mapping
29
-
mapping, err := NewUserMapping(db)
30
-
if err != nil {
31
-
return nil, fmt.Errorf("creating user mapping: %w", err)
32
-
}
33
-
34
-
return &RepoStore{
35
-
cs: cs,
36
-
mapping: mapping,
37
-
}, nil
38
-
}
39
-
40
-
// ImportRepo imports a repository CAR file for a DID
41
-
func (rs *RepoStore) ImportRepo(ctx context.Context, did string, carData io.Reader) (cid.Cid, error) {
42
-
uid, err := rs.mapping.GetOrCreateUID(ctx, did)
43
-
if err != nil {
44
-
return cid.Undef, fmt.Errorf("getting UID for DID %s: %w", did, err)
45
-
}
46
-
47
-
// Read all data from the reader
48
-
data, err := io.ReadAll(carData)
49
-
if err != nil {
50
-
return cid.Undef, fmt.Errorf("reading CAR data: %w", err)
51
-
}
52
-
53
-
return rs.cs.ImportSlice(ctx, uid, nil, data)
54
-
}
55
-
56
-
// ReadRepo reads a repository CAR file for a DID
57
-
func (rs *RepoStore) ReadRepo(ctx context.Context, did string, sinceRev string) ([]byte, error) {
58
-
uid, err := rs.mapping.GetUID(did)
59
-
if err != nil {
60
-
return nil, fmt.Errorf("getting UID for DID %s: %w", did, err)
61
-
}
62
-
63
-
var buf bytes.Buffer
64
-
err = rs.cs.ReadUserCar(ctx, uid, sinceRev, false, &buf)
65
-
if err != nil {
66
-
return nil, fmt.Errorf("reading repo for DID %s: %w", did, err)
67
-
}
68
-
69
-
return buf.Bytes(), nil
70
-
}
71
-
72
-
// GetRepoHead gets the latest repository head CID for a DID
73
-
func (rs *RepoStore) GetRepoHead(ctx context.Context, did string) (cid.Cid, error) {
74
-
uid, err := rs.mapping.GetUID(did)
75
-
if err != nil {
76
-
return cid.Undef, fmt.Errorf("getting UID for DID %s: %w", did, err)
77
-
}
78
-
79
-
return rs.cs.GetUserRepoHead(ctx, uid)
80
-
}
81
-
82
-
// CompactRepo performs garbage collection for a DID's repository
83
-
func (rs *RepoStore) CompactRepo(ctx context.Context, did string) error {
84
-
uid, err := rs.mapping.GetUID(did)
85
-
if err != nil {
86
-
return fmt.Errorf("getting UID for DID %s: %w", did, err)
87
-
}
88
-
89
-
return rs.cs.CompactUserShards(ctx, uid, false)
90
-
}
91
-
92
-
// DeleteRepo removes all data for a DID's repository
93
-
func (rs *RepoStore) DeleteRepo(ctx context.Context, did string) error {
94
-
uid, err := rs.mapping.GetUID(did)
95
-
if err != nil {
96
-
return fmt.Errorf("getting UID for DID %s: %w", did, err)
97
-
}
98
-
99
-
return rs.cs.WipeUserData(ctx, uid)
100
-
}
101
-
102
-
// HasRepo checks if a repository exists for a DID
103
-
func (rs *RepoStore) HasRepo(ctx context.Context, did string) (bool, error) {
104
-
uid, err := rs.mapping.GetUID(did)
105
-
if err != nil {
106
-
// If no UID mapping exists, repo doesn't exist
107
-
return false, nil
108
-
}
109
-
110
-
// Try to get the repo head
111
-
head, err := rs.cs.GetUserRepoHead(ctx, uid)
112
-
if err != nil {
113
-
return false, nil
114
-
}
115
-
116
-
return head.Defined(), nil
117
-
}
118
-
119
-
// GetOrCreateUID gets or creates a UID for a DID
120
-
func (rs *RepoStore) GetOrCreateUID(ctx context.Context, did string) (models.Uid, error) {
121
-
return rs.mapping.GetOrCreateUID(ctx, did)
122
-
}
-127
internal/atproto/carstore/user_mapping.go
-127
internal/atproto/carstore/user_mapping.go
···
1
-
package carstore
2
-
3
-
import (
4
-
"context"
5
-
"fmt"
6
-
"sync"
7
-
8
-
"github.com/bluesky-social/indigo/models"
9
-
"gorm.io/gorm"
10
-
)
11
-
12
-
// UserMapping manages the mapping between DIDs and numeric UIDs required by Indigo's carstore
13
-
type UserMapping struct {
14
-
db *gorm.DB
15
-
mu sync.RWMutex
16
-
didToUID map[string]models.Uid
17
-
uidToDID map[models.Uid]string
18
-
nextUID models.Uid
19
-
}
20
-
21
-
// UserMap represents the database model for DID to UID mapping
22
-
type UserMap struct {
23
-
UID models.Uid `gorm:"primaryKey;autoIncrement"`
24
-
DID string `gorm:"column:did;uniqueIndex;not null"`
25
-
CreatedAt int64
26
-
UpdatedAt int64
27
-
}
28
-
29
-
// NewUserMapping creates a new UserMapping instance
30
-
func NewUserMapping(db *gorm.DB) (*UserMapping, error) {
31
-
// Auto-migrate the user mapping table
32
-
if err := db.AutoMigrate(&UserMap{}); err != nil {
33
-
return nil, fmt.Errorf("migrating user mapping table: %w", err)
34
-
}
35
-
36
-
um := &UserMapping{
37
-
db: db,
38
-
didToUID: make(map[string]models.Uid),
39
-
uidToDID: make(map[models.Uid]string),
40
-
nextUID: 1,
41
-
}
42
-
43
-
// Load existing mappings
44
-
if err := um.loadMappings(); err != nil {
45
-
return nil, fmt.Errorf("loading user mappings: %w", err)
46
-
}
47
-
48
-
return um, nil
49
-
}
50
-
51
-
// loadMappings loads all existing DID to UID mappings from the database
52
-
func (um *UserMapping) loadMappings() error {
53
-
var mappings []UserMap
54
-
if err := um.db.Find(&mappings).Error; err != nil {
55
-
return fmt.Errorf("querying user mappings: %w", err)
56
-
}
57
-
58
-
um.mu.Lock()
59
-
defer um.mu.Unlock()
60
-
61
-
for _, m := range mappings {
62
-
um.didToUID[m.DID] = m.UID
63
-
um.uidToDID[m.UID] = m.DID
64
-
if m.UID >= um.nextUID {
65
-
um.nextUID = m.UID + 1
66
-
}
67
-
}
68
-
69
-
return nil
70
-
}
71
-
72
-
// GetOrCreateUID gets or creates a UID for a given DID
73
-
func (um *UserMapping) GetOrCreateUID(ctx context.Context, did string) (models.Uid, error) {
74
-
um.mu.RLock()
75
-
if uid, exists := um.didToUID[did]; exists {
76
-
um.mu.RUnlock()
77
-
return uid, nil
78
-
}
79
-
um.mu.RUnlock()
80
-
81
-
// Need to create a new mapping
82
-
um.mu.Lock()
83
-
defer um.mu.Unlock()
84
-
85
-
// Double-check in case another goroutine created it
86
-
if uid, exists := um.didToUID[did]; exists {
87
-
return uid, nil
88
-
}
89
-
90
-
// Create new mapping
91
-
userMap := &UserMap{
92
-
DID: did,
93
-
}
94
-
95
-
if err := um.db.Create(userMap).Error; err != nil {
96
-
return 0, fmt.Errorf("creating user mapping for DID %s: %w", did, err)
97
-
}
98
-
99
-
um.didToUID[did] = userMap.UID
100
-
um.uidToDID[userMap.UID] = did
101
-
102
-
return userMap.UID, nil
103
-
}
104
-
105
-
// GetUID returns the UID for a DID, or an error if not found
106
-
func (um *UserMapping) GetUID(did string) (models.Uid, error) {
107
-
um.mu.RLock()
108
-
defer um.mu.RUnlock()
109
-
110
-
uid, exists := um.didToUID[did]
111
-
if !exists {
112
-
return 0, fmt.Errorf("UID not found for DID: %s", did)
113
-
}
114
-
return uid, nil
115
-
}
116
-
117
-
// GetDID returns the DID for a UID, or an error if not found
118
-
func (um *UserMapping) GetDID(uid models.Uid) (string, error) {
119
-
um.mu.RLock()
120
-
defer um.mu.RUnlock()
121
-
122
-
did, exists := um.uidToDID[uid]
123
-
if !exists {
124
-
return "", fmt.Errorf("DID not found for UID: %d", uid)
125
-
}
126
-
return did, nil
127
-
}
-201
internal/atproto/repo/wrapper.go
-201
internal/atproto/repo/wrapper.go
···
1
-
package repo
2
-
3
-
import (
4
-
"bytes"
5
-
"context"
6
-
"fmt"
7
-
8
-
"github.com/bluesky-social/indigo/mst"
9
-
"github.com/bluesky-social/indigo/repo"
10
-
"github.com/ipfs/go-cid"
11
-
blockstore "github.com/ipfs/go-ipfs-blockstore"
12
-
cbornode "github.com/ipfs/go-ipld-cbor"
13
-
cbg "github.com/whyrusleeping/cbor-gen"
14
-
)
15
-
16
-
// Wrapper provides a thin wrapper around Indigo's repo package
17
-
type Wrapper struct {
18
-
repo *repo.Repo
19
-
blockstore blockstore.Blockstore
20
-
}
21
-
22
-
// NewWrapper creates a new wrapper for a repository with the provided blockstore
23
-
func NewWrapper(did string, signingKey interface{}, bs blockstore.Blockstore) (*Wrapper, error) {
24
-
// Create new repository with the provided blockstore
25
-
r := repo.NewRepo(context.Background(), did, bs)
26
-
27
-
return &Wrapper{
28
-
repo: r,
29
-
blockstore: bs,
30
-
}, nil
31
-
}
32
-
33
-
// OpenWrapper opens an existing repository from CAR data with the provided blockstore
34
-
func OpenWrapper(carData []byte, signingKey interface{}, bs blockstore.Blockstore) (*Wrapper, error) {
35
-
r, err := repo.ReadRepoFromCar(context.Background(), bytes.NewReader(carData))
36
-
if err != nil {
37
-
return nil, fmt.Errorf("failed to read repo from CAR: %w", err)
38
-
}
39
-
40
-
return &Wrapper{
41
-
repo: r,
42
-
blockstore: bs,
43
-
}, nil
44
-
}
45
-
46
-
// CreateRecord adds a new record to the repository
47
-
func (w *Wrapper) CreateRecord(collection string, recordKey string, record cbg.CBORMarshaler) (cid.Cid, string, error) {
48
-
// The repo.CreateRecord generates its own key, so we'll use that
49
-
recordCID, rkey, err := w.repo.CreateRecord(context.Background(), collection, record)
50
-
if err != nil {
51
-
return cid.Undef, "", fmt.Errorf("failed to create record: %w", err)
52
-
}
53
-
54
-
// If a specific key was requested, we'd need to use PutRecord instead
55
-
if recordKey != "" {
56
-
// Use PutRecord for specific keys
57
-
path := fmt.Sprintf("%s/%s", collection, recordKey)
58
-
recordCID, err = w.repo.PutRecord(context.Background(), path, record)
59
-
if err != nil {
60
-
return cid.Undef, "", fmt.Errorf("failed to put record with key: %w", err)
61
-
}
62
-
return recordCID, recordKey, nil
63
-
}
64
-
65
-
return recordCID, rkey, nil
66
-
}
67
-
68
-
// GetRecord retrieves a record from the repository
69
-
func (w *Wrapper) GetRecord(collection string, recordKey string) (cid.Cid, []byte, error) {
70
-
path := fmt.Sprintf("%s/%s", collection, recordKey)
71
-
72
-
recordCID, rec, err := w.repo.GetRecord(context.Background(), path)
73
-
if err != nil {
74
-
return cid.Undef, nil, fmt.Errorf("failed to get record: %w", err)
75
-
}
76
-
77
-
// Encode record to CBOR
78
-
buf := new(bytes.Buffer)
79
-
if err := rec.(cbg.CBORMarshaler).MarshalCBOR(buf); err != nil {
80
-
return cid.Undef, nil, fmt.Errorf("failed to encode record: %w", err)
81
-
}
82
-
83
-
return recordCID, buf.Bytes(), nil
84
-
}
85
-
86
-
// UpdateRecord updates an existing record in the repository
87
-
func (w *Wrapper) UpdateRecord(collection string, recordKey string, record cbg.CBORMarshaler) (cid.Cid, error) {
88
-
path := fmt.Sprintf("%s/%s", collection, recordKey)
89
-
90
-
// Check if record exists
91
-
_, _, err := w.repo.GetRecord(context.Background(), path)
92
-
if err != nil {
93
-
return cid.Undef, fmt.Errorf("record not found: %w", err)
94
-
}
95
-
96
-
// Update the record
97
-
recordCID, err := w.repo.UpdateRecord(context.Background(), path, record)
98
-
if err != nil {
99
-
return cid.Undef, fmt.Errorf("failed to update record: %w", err)
100
-
}
101
-
102
-
return recordCID, nil
103
-
}
104
-
105
-
// DeleteRecord removes a record from the repository
106
-
func (w *Wrapper) DeleteRecord(collection string, recordKey string) error {
107
-
path := fmt.Sprintf("%s/%s", collection, recordKey)
108
-
109
-
if err := w.repo.DeleteRecord(context.Background(), path); err != nil {
110
-
return fmt.Errorf("failed to delete record: %w", err)
111
-
}
112
-
113
-
return nil
114
-
}
115
-
116
-
// ListRecords returns all records in a collection
117
-
func (w *Wrapper) ListRecords(collection string) ([]RecordInfo, error) {
118
-
var records []RecordInfo
119
-
120
-
err := w.repo.ForEach(context.Background(), collection, func(k string, v cid.Cid) error {
121
-
// Skip if not in the requested collection
122
-
if len(k) <= len(collection)+1 || k[:len(collection)] != collection || k[len(collection)] != '/' {
123
-
return nil
124
-
}
125
-
126
-
recordKey := k[len(collection)+1:]
127
-
records = append(records, RecordInfo{
128
-
Collection: collection,
129
-
RecordKey: recordKey,
130
-
CID: v,
131
-
})
132
-
133
-
return nil
134
-
})
135
-
136
-
if err != nil {
137
-
return nil, fmt.Errorf("failed to list records: %w", err)
138
-
}
139
-
140
-
return records, nil
141
-
}
142
-
143
-
// Commit creates a new signed commit
144
-
func (w *Wrapper) Commit(did string, signingKey interface{}) (*repo.SignedCommit, error) {
145
-
// The commit function expects a signing function with context
146
-
signingFunc := func(ctx context.Context, did string, data []byte) ([]byte, error) {
147
-
// TODO: Implement proper signing based on signingKey type
148
-
return []byte("mock-signature"), nil
149
-
}
150
-
151
-
_, _, err := w.repo.Commit(context.Background(), signingFunc)
152
-
if err != nil {
153
-
return nil, fmt.Errorf("failed to commit: %w", err)
154
-
}
155
-
156
-
// Return the signed commit from the repo
157
-
sc := w.repo.SignedCommit()
158
-
159
-
return &sc, nil
160
-
}
161
-
162
-
// GetHeadCID returns the CID of the current repository head
163
-
func (w *Wrapper) GetHeadCID() (cid.Cid, error) {
164
-
// TODO: Implement this properly
165
-
// The repo package doesn't expose a direct way to get the head CID
166
-
return cid.Undef, fmt.Errorf("not implemented")
167
-
}
168
-
169
-
// Export exports the repository as a CAR file
170
-
func (w *Wrapper) Export() ([]byte, error) {
171
-
// TODO: Implement proper CAR export using Indigo's carstore functionality
172
-
// For now, return a placeholder
173
-
return nil, fmt.Errorf("CAR export not yet implemented")
174
-
}
175
-
176
-
// GetMST returns the underlying Merkle Search Tree
177
-
func (w *Wrapper) GetMST() (*mst.MerkleSearchTree, error) {
178
-
// TODO: Implement MST access
179
-
return nil, fmt.Errorf("not implemented")
180
-
}
181
-
182
-
// RecordInfo contains information about a record
183
-
type RecordInfo struct {
184
-
Collection string
185
-
RecordKey string
186
-
CID cid.Cid
187
-
}
188
-
189
-
// DecodeRecord decodes CBOR data into a record structure
190
-
func DecodeRecord(data []byte, v interface{}) error {
191
-
return cbornode.DecodeInto(data, v)
192
-
}
193
-
194
-
// EncodeRecord encodes a record structure into CBOR data
195
-
func EncodeRecord(v cbg.CBORMarshaler) ([]byte, error) {
196
-
buf := new(bytes.Buffer)
197
-
if err := v.MarshalCBOR(buf); err != nil {
198
-
return nil, err
199
-
}
200
-
return buf.Bytes(), nil
201
-
}
server
server
This is a binary file and will not be displayed.
+28
-34
tests/lexicon_validation_test.go
+28
-34
tests/lexicon_validation_test.go
···
1
1
package tests
2
2
3
3
import (
4
+
"os"
5
+
"path/filepath"
4
6
"strings"
5
7
"testing"
6
8
···
17
19
t.Fatalf("Failed to load lexicon schemas: %v", err)
18
20
}
19
21
20
-
// Test that we can resolve our key schemas
21
-
expectedSchemas := []string{
22
-
"social.coves.actor.profile",
23
-
"social.coves.actor.subscription",
24
-
"social.coves.actor.membership",
25
-
"social.coves.community.profile",
26
-
"social.coves.community.rules",
27
-
"social.coves.community.wiki",
28
-
"social.coves.post.text",
29
-
"social.coves.post.image",
30
-
"social.coves.post.video",
31
-
"social.coves.post.article",
32
-
"social.coves.richtext.facet",
33
-
"social.coves.embed.image",
34
-
"social.coves.embed.video",
35
-
"social.coves.embed.external",
36
-
"social.coves.embed.post",
37
-
"social.coves.interaction.vote",
38
-
"social.coves.interaction.tag",
39
-
"social.coves.interaction.comment",
40
-
"social.coves.interaction.share",
41
-
"social.coves.moderation.vote",
42
-
"social.coves.moderation.tribunalVote",
43
-
"social.coves.moderation.ruleProposal",
22
+
// Walk through the directory and find all lexicon files
23
+
var lexiconFiles []string
24
+
err := filepath.Walk(schemaPath, func(path string, info os.FileInfo, err error) error {
25
+
if err != nil {
26
+
return err
27
+
}
28
+
if strings.HasSuffix(path, ".json") && !info.IsDir() {
29
+
lexiconFiles = append(lexiconFiles, path)
30
+
}
31
+
return nil
32
+
})
33
+
if err != nil {
34
+
t.Fatalf("Failed to walk directory: %v", err)
44
35
}
45
36
46
-
for _, schemaID := range expectedSchemas {
37
+
t.Logf("Found %d lexicon files to validate", len(lexiconFiles))
38
+
39
+
// Extract schema IDs from file paths and test resolution
40
+
for _, filePath := range lexiconFiles {
41
+
// Convert file path to schema ID
42
+
// e.g., ../internal/atproto/lexicon/social/coves/actor/profile.json -> social.coves.actor.profile
43
+
relPath, _ := filepath.Rel(schemaPath, filePath)
44
+
relPath = strings.TrimSuffix(relPath, ".json")
45
+
schemaID := strings.ReplaceAll(relPath, string(filepath.Separator), ".")
46
+
47
47
t.Run(schemaID, func(t *testing.T) {
48
48
if _, err := catalog.Resolve(schemaID); err != nil {
49
49
t.Errorf("Failed to resolve schema %s: %v", schemaID, err)
···
137
137
"community": "did:plc:programming123",
138
138
"postType": "text",
139
139
"title": "Test Post",
140
-
"text": "This is a test post",
141
-
"tags": []string{"test", "golang"},
142
-
"language": "en",
143
-
"contentWarnings": []string{},
140
+
"content": "This is a test post",
144
141
"createdAt": "2025-01-09T14:30:00Z",
145
142
},
146
143
shouldFail: false,
···
153
150
"community": "did:plc:programming123",
154
151
"postType": "invalid-type",
155
152
"title": "Test Post",
156
-
"text": "This is a test post",
157
-
"tags": []string{"test"},
158
-
"language": "en",
159
-
"contentWarnings": []string{},
153
+
"content": "This is a test post",
160
154
"createdAt": "2025-01-09T14:30:00Z",
161
155
},
162
156
shouldFail: true,
···
215
209
if err != nil {
216
210
t.Errorf("Expected lenient validation to pass, got error: %v", err)
217
211
}
218
-
}
212
+
}