feat(aggregator): add setup scripts for aggregator registration

+591

docs/aggregators/SETUP_GUIDE.md

··· 1 + # Aggregator Setup Guide 2 + 3 + This guide explains how to set up and register an aggregator with Coves instances. 4 + 5 + ## Table of Contents 6 + 7 + - [Overview](#overview) 8 + - [Architecture](#architecture) 9 + - [Prerequisites](#prerequisites) 10 + - [Quick Start](#quick-start) 11 + - [Detailed Setup Steps](#detailed-setup-steps) 12 + - [Authorization Process](#authorization-process) 13 + - [Posting to Communities](#posting-to-communities) 14 + - [Rate Limits](#rate-limits) 15 + - [Security Best Practices](#security-best-practices) 16 + - [Troubleshooting](#troubleshooting) 17 + - [API Reference](#api-reference) 18 + 19 + ## Overview 20 + 21 + **Aggregators** are automated services that post content to Coves communities. They are similar to Bluesky's feed generators and labelers - self-managed external services that integrate with the platform. 22 + 23 + **Key characteristics**: 24 + - Self-owned: You create and manage your own PDS account 25 + - Domain-verified: Prove ownership via `.well-known/atproto-did` 26 + - Community-authorized: Moderators grant posting permission per-community 27 + - Rate-limited: 10 posts per hour per community 28 + 29 + **Example use cases**: 30 + - RSS feed aggregators (tech news, blog posts) 31 + - Social media cross-posters (Twitter → Coves) 32 + - Event notifications (GitHub releases, weather alerts) 33 + - Content curation bots (daily links, summaries) 34 + 35 + ## Architecture 36 + 37 + ### Data Flow 38 + 39 + ``` 40 + ┌──────────────────────────────────────────────────────────┐ 41 + │ 1. One-Time Setup │ 42 + ├──────────────────────────────────────────────────────────┤ 43 + │ Aggregator creates PDS account │ 44 + │ ↓ │ 45 + │ Proves domain ownership (.well-known) │ 46 + │ ↓ │ 47 + │ Registers with Coves (enters users table) │ 48 + │ ↓ │ 49 + │ Writes service declaration │ 50 + │ ↓ │ 51 + │ Jetstream indexes into aggregators table │ 52 + └──────────────────────────────────────────────────────────┘ 53 + 54 + ┌──────────────────────────────────────────────────────────┐ 55 + │ 2. Per-Community Authorization │ 56 + ├──────────────────────────────────────────────────────────┤ 57 + │ Moderator writes authorization record │ 58 + │ ↓ │ 59 + │ Jetstream indexes into aggregator_authorizations │ 60 + └──────────────────────────────────────────────────────────┘ 61 + 62 + ┌──────────────────────────────────────────────────────────┐ 63 + │ 3. Posting (Ongoing) │ 64 + ├──────────────────────────────────────────────────────────┤ 65 + │ Aggregator calls post creation endpoint │ 66 + │ ↓ │ 67 + │ Handler validates: │ 68 + │ - Author in users table ✓ │ 69 + │ - Author in aggregators table ✓ │ 70 + │ - Authorization exists ✓ │ 71 + │ - Rate limit not exceeded ✓ │ 72 + │ ↓ │ 73 + │ Post written to community's PDS │ 74 + │ ↓ │ 75 + │ Jetstream indexes post │ 76 + └──────────────────────────────────────────────────────────┘ 77 + ``` 78 + 79 + ### Database Tables 80 + 81 + **users** - All actors (users, communities, aggregators) 82 + ```sql 83 + CREATE TABLE users ( 84 + did TEXT PRIMARY KEY, 85 + handle TEXT NOT NULL, 86 + pds_url TEXT, 87 + indexed_at TIMESTAMPTZ 88 + ); 89 + ``` 90 + 91 + **aggregators** - Aggregator-specific metadata 92 + ```sql 93 + CREATE TABLE aggregators ( 94 + did TEXT PRIMARY KEY, 95 + display_name TEXT NOT NULL, 96 + description TEXT, 97 + avatar_url TEXT, 98 + config_schema JSONB, 99 + source_url TEXT, 100 + maintainer_did TEXT, 101 + record_uri TEXT NOT NULL UNIQUE, 102 + record_cid TEXT NOT NULL, 103 + created_at TIMESTAMPTZ, 104 + indexed_at TIMESTAMPTZ 105 + ); 106 + ``` 107 + 108 + **aggregator_authorizations** - Community authorizations 109 + ```sql 110 + CREATE TABLE aggregator_authorizations ( 111 + id BIGSERIAL PRIMARY KEY, 112 + aggregator_did TEXT NOT NULL, 113 + community_did TEXT NOT NULL, 114 + enabled BOOLEAN NOT NULL DEFAULT true, 115 + config JSONB, 116 + created_by TEXT, 117 + record_uri TEXT NOT NULL UNIQUE, 118 + record_cid TEXT NOT NULL, 119 + UNIQUE(aggregator_did, community_did) 120 + ); 121 + ``` 122 + 123 + ## Prerequisites 124 + 125 + 1. **Domain ownership**: You must own a domain where you can host static files over HTTPS 126 + 2. **Web server**: Ability to serve the `.well-known/atproto-did` file 127 + 3. **Development tools**: `curl`, `jq`, basic shell scripting knowledge 128 + 4. **Email address**: For creating the PDS account 129 + 130 + **Optional**: 131 + - Custom avatar image (PNG/JPEG/WebP, max 1MB) 132 + - GitHub repository for source code transparency 133 + 134 + ## Quick Start 135 + 136 + We provide automated setup scripts: 137 + 138 + ```bash 139 + cd scripts/aggregator-setup 140 + 141 + # Make scripts executable 142 + chmod +x *.sh 143 + 144 + # Run setup scripts in order 145 + ./1-create-pds-account.sh 146 + ./2-setup-wellknown.sh 147 + # (Upload .well-known to your web server) 148 + ./3-register-with-coves.sh 149 + ./4-create-service-declaration.sh 150 + ``` 151 + 152 + See [scripts/aggregator-setup/README.md](../../scripts/aggregator-setup/README.md) for detailed script documentation. 153 + 154 + ## Detailed Setup Steps 155 + 156 + ### Step 1: Create PDS Account 157 + 158 + Your aggregator needs its own atProto identity (DID). The easiest way is to create an account on an existing PDS. 159 + 160 + **Using an existing PDS (recommended)**: 161 + 162 + ```bash 163 + curl -X POST https://bsky.social/xrpc/com.atproto.server.createAccount \ 164 + -H "Content-Type: application/json" \ 165 + -d '{ 166 + "handle": "mynewsbot.bsky.social", 167 + "email": "bot@example.com", 168 + "password": "secure-password-here" 169 + }' 170 + ``` 171 + 172 + **Response**: 173 + ```json 174 + { 175 + "accessJwt": "eyJ...", 176 + "refreshJwt": "eyJ...", 177 + "handle": "mynewsbot.bsky.social", 178 + "did": "did:plc:abc123...", 179 + "didDoc": {...} 180 + } 181 + ``` 182 + 183 + **Save these credentials securely!** You'll need the DID and access token for all subsequent operations. 184 + 185 + **Alternative**: Run your own PDS or use `did:web` (advanced). 186 + 187 + ### Step 2: Prove Domain Ownership 188 + 189 + To register with Coves, you must prove you own a domain by serving your DID at `https://yourdomain.com/.well-known/atproto-did`. 190 + 191 + **Create the file**: 192 + 193 + ```bash 194 + mkdir -p .well-known 195 + echo "did:plc:abc123..." > .well-known/atproto-did 196 + ``` 197 + 198 + **Upload to your web server** so it's accessible at: 199 + ``` 200 + https://rss-bot.example.com/.well-known/atproto-did 201 + ``` 202 + 203 + **Verify it works**: 204 + ```bash 205 + curl https://rss-bot.example.com/.well-known/atproto-did 206 + # Should return: did:plc:abc123... 207 + ``` 208 + 209 + **Nginx configuration example**: 210 + ```nginx 211 + location /.well-known/atproto-did { 212 + alias /var/www/.well-known/atproto-did; 213 + default_type text/plain; 214 + add_header Access-Control-Allow-Origin *; 215 + } 216 + ``` 217 + 218 + ### Step 3: Register with Coves 219 + 220 + Call the registration endpoint to register your aggregator DID with the Coves instance. 221 + 222 + **Endpoint**: `POST /xrpc/social.coves.aggregator.register` 223 + 224 + **Request**: 225 + ```bash 226 + curl -X POST https://api.coves.social/xrpc/social.coves.aggregator.register \ 227 + -H "Content-Type: application/json" \ 228 + -d '{ 229 + "did": "did:plc:abc123...", 230 + "domain": "rss-bot.example.com" 231 + }' 232 + ``` 233 + 234 + **Response** (Success): 235 + ```json 236 + { 237 + "did": "did:plc:abc123...", 238 + "handle": "mynewsbot.bsky.social", 239 + "message": "Aggregator registered successfully. Next step: create a service declaration record at at://did:plc:abc123.../social.coves.aggregator.service/self" 240 + } 241 + ``` 242 + 243 + **What happens**: 244 + 1. Coves fetches `https://rss-bot.example.com/.well-known/atproto-did` 245 + 2. Verifies it contains your DID 246 + 3. Resolves your DID to get handle and PDS URL 247 + 4. Inserts you into the `users` table 248 + 249 + **You're now registered!** But you need to create a service declaration next. 250 + 251 + ### Step 4: Create Service Declaration 252 + 253 + Write a `social.coves.aggregator.service` record to your repository. This contains metadata about your aggregator and gets indexed by Coves' Jetstream consumer. 254 + 255 + **Endpoint**: `POST https://your-pds.com/xrpc/com.atproto.repo.createRecord` 256 + 257 + **Request**: 258 + ```bash 259 + curl -X POST https://bsky.social/xrpc/com.atproto.repo.createRecord \ 260 + -H "Authorization: Bearer YOUR_ACCESS_TOKEN" \ 261 + -H "Content-Type: application/json" \ 262 + -d '{ 263 + "repo": "did:plc:abc123...", 264 + "collection": "social.coves.aggregator.service", 265 + "rkey": "self", 266 + "record": { 267 + "$type": "social.coves.aggregator.service", 268 + "did": "did:plc:abc123...", 269 + "displayName": "RSS News Aggregator", 270 + "description": "Aggregates tech news from various RSS feeds", 271 + "sourceUrl": "https://github.com/yourname/rss-aggregator", 272 + "maintainer": "did:plc:your-personal-did", 273 + "createdAt": "2024-01-15T12:00:00Z" 274 + } 275 + }' 276 + ``` 277 + 278 + **Response**: 279 + ```json 280 + { 281 + "uri": "at://did:plc:abc123.../social.coves.aggregator.service/self", 282 + "cid": "bafyrei..." 283 + } 284 + ``` 285 + 286 + **Optional fields**: 287 + - `avatar`: Blob reference to avatar image 288 + - `configSchema`: JSON Schema for community-specific configuration 289 + 290 + **Wait 5-10 seconds** for Jetstream to index your service declaration into the `aggregators` table. 291 + 292 + ## Authorization Process 293 + 294 + Before you can post to a community, a moderator must authorize your aggregator. 295 + 296 + ### How Authorization Works 297 + 298 + 1. **Moderator decision**: Community moderator evaluates your aggregator 299 + 2. **Authorization record**: Moderator writes `social.coves.aggregator.authorization` to community's repo 300 + 3. **Jetstream indexing**: Record gets indexed into `aggregator_authorizations` table 301 + 4. **Posting enabled**: You can now post to that community 302 + 303 + ### Authorization Record Structure 304 + 305 + **Location**: `at://{community_did}/social.coves.aggregator.authorization/{rkey}` 306 + 307 + **Example**: 308 + ```json 309 + { 310 + "$type": "social.coves.aggregator.authorization", 311 + "aggregatorDid": "did:plc:abc123...", 312 + "communityDid": "did:plc:community123...", 313 + "enabled": true, 314 + "createdBy": "did:plc:moderator...", 315 + "createdAt": "2024-01-15T12:00:00Z", 316 + "config": { 317 + "maxPostsPerHour": 5, 318 + "allowedCategories": ["tech", "news"] 319 + } 320 + } 321 + ``` 322 + 323 + ### Checking Your Authorizations 324 + 325 + **Endpoint**: `GET /xrpc/social.coves.aggregator.getAuthorizations` 326 + 327 + ```bash 328 + curl "https://api.coves.social/xrpc/social.coves.aggregator.getAuthorizations?aggregatorDid=did:plc:abc123...&enabledOnly=true" 329 + ``` 330 + 331 + **Response**: 332 + ```json 333 + { 334 + "authorizations": [ 335 + { 336 + "aggregatorDid": "did:plc:abc123...", 337 + "communityDid": "did:plc:community123...", 338 + "communityHandle": "~tech@coves.social", 339 + "enabled": true, 340 + "createdAt": "2024-01-15T12:00:00Z", 341 + "config": {...} 342 + } 343 + ] 344 + } 345 + ``` 346 + 347 + ## Posting to Communities 348 + 349 + Once authorized, you can post to communities using the standard post creation endpoint. 350 + 351 + ### Create Post 352 + 353 + **Endpoint**: `POST /xrpc/social.coves.community.post.create` 354 + 355 + **Request**: 356 + ```bash 357 + curl -X POST https://api.coves.social/xrpc/social.coves.community.post.create \ 358 + -H "Authorization: Bearer YOUR_ACCESS_TOKEN" \ 359 + -H "Content-Type: application/json" \ 360 + -d '{ 361 + "communityDid": "did:plc:community123...", 362 + "post": { 363 + "text": "New blog post: Understanding atProto Identity\nhttps://example.com/post", 364 + "createdAt": "2024-01-15T12:00:00Z", 365 + "facets": [ 366 + { 367 + "index": { "byteStart": 50, "byteEnd": 75 }, 368 + "features": [ 369 + { 370 + "$type": "social.coves.richtext.facet#link", 371 + "uri": "https://example.com/post" 372 + } 373 + ] 374 + } 375 + ] 376 + } 377 + }' 378 + ``` 379 + 380 + **Response**: 381 + ```json 382 + { 383 + "uri": "at://did:plc:abc123.../social.coves.community.post/3k...", 384 + "cid": "bafyrei..." 385 + } 386 + ``` 387 + 388 + ### Post Validation 389 + 390 + The handler validates: 391 + 1. **Authentication**: Valid JWT token 392 + 2. **Author exists**: DID in `users` table 393 + 3. **Is aggregator**: DID in `aggregators` table 394 + 4. **Authorization**: Active authorization for (aggregator, community) 395 + 5. **Rate limit**: Less than 10 posts/hour to this community 396 + 6. **Content**: Valid post structure per lexicon 397 + 398 + ### Rate Limits 399 + 400 + **Per-community rate limit**: 10 posts per hour 401 + 402 + This is tracked in the `aggregator_posts` table and enforced at the handler level. 403 + 404 + **Why?**: Prevents spam while allowing useful bot activity. 405 + 406 + **Best practices**: 407 + - Batch similar content 408 + - Post only high-quality content 409 + - Respect community guidelines 410 + - Monitor your posting rate 411 + 412 + ## Security Best Practices 413 + 414 + ### Credential Management 415 + 416 + ✅ **DO**: 417 + - Store credentials in environment variables or secret management 418 + - Use HTTPS for all API calls 419 + - Rotate access tokens regularly (use refresh tokens) 420 + - Keep `aggregator-config.env` out of version control 421 + 422 + ❌ **DON'T**: 423 + - Hardcode credentials in source code 424 + - Commit credentials to Git 425 + - Share access tokens publicly 426 + - Reuse personal credentials for bots 427 + 428 + ### Domain Security 429 + 430 + ✅ **DO**: 431 + - Use HTTPS for `.well-known` endpoint 432 + - Keep domain under your control 433 + - Monitor for unauthorized changes 434 + - Use DNSSEC if possible 435 + 436 + ❌ **DON'T**: 437 + - Use HTTP (will fail verification) 438 + - Use shared/untrusted hosting 439 + - Allow others to modify `.well-known` files 440 + - Use expired SSL certificates 441 + 442 + ### Content Security 443 + 444 + ✅ **DO**: 445 + - Validate all external content before posting 446 + - Sanitize URLs and text 447 + - Rate-limit your own posting 448 + - Implement circuit breakers for failures 449 + 450 + ❌ **DON'T**: 451 + - Post unvalidated user input 452 + - Include malicious links 453 + - Spam communities 454 + - Bypass rate limits 455 + 456 + ## Troubleshooting 457 + 458 + ### Registration Errors 459 + 460 + #### Error: "DomainVerificationFailed" 461 + 462 + **Cause**: `.well-known/atproto-did` not accessible or contains wrong DID 463 + 464 + **Solutions**: 465 + 1. Verify file is accessible: `curl https://yourdomain.com/.well-known/atproto-did` 466 + 2. Check content matches your DID exactly (no extra whitespace) 467 + 3. Ensure HTTPS is working (not HTTP) 468 + 4. Check web server logs for access errors 469 + 5. Verify firewall rules allow HTTPS traffic 470 + 471 + #### Error: "AlreadyRegistered" 472 + 473 + **Cause**: This DID is already registered with this Coves instance 474 + 475 + **Solutions**: 476 + - This is safe to ignore if you're re-running setup 477 + - If you need to update info, just create a new service declaration 478 + - Contact instance admin if you need to remove registration 479 + 480 + #### Error: "DIDResolutionFailed" 481 + 482 + **Cause**: Could not resolve DID document from PLC directory 483 + 484 + **Solutions**: 485 + 1. Verify DID exists: `curl https://plc.directory/{your-did}` 486 + 2. Wait 30 seconds and retry (PLC propagation delay) 487 + 3. Check PDS is accessible 488 + 4. Verify DID format is correct (must start with `did:plc:` or `did:web:`) 489 + 490 + ### Posting Errors 491 + 492 + #### Error: "NotAuthorized" 493 + 494 + **Cause**: No active authorization for this (aggregator, community) pair 495 + 496 + **Solutions**: 497 + 1. Check authorizations: `GET /xrpc/social.coves.aggregator.getAuthorizations` 498 + 2. Contact community moderator to request authorization 499 + 3. Verify authorization wasn't disabled 500 + 4. Wait for Jetstream to index authorization (5-10 seconds) 501 + 502 + #### Error: "RateLimitExceeded" 503 + 504 + **Cause**: Exceeded 10 posts/hour to this community 505 + 506 + **Solutions**: 507 + 1. Wait for the rate limit window to reset 508 + 2. Batch posts to stay under limit 509 + 3. Distribute posts across multiple communities 510 + 4. Implement posting queue in your aggregator 511 + 512 + ### Service Declaration Not Appearing 513 + 514 + **Symptoms**: Service declaration created but not in `aggregators` table 515 + 516 + **Solutions**: 517 + 1. Wait 5-10 seconds for Jetstream to index 518 + 2. Check Jetstream consumer logs for errors 519 + 3. Verify record was created: Check PDS at `at://your-did/social.coves.aggregator.service/self` 520 + 4. Verify `$type` field is exactly `"social.coves.aggregator.service"` 521 + 5. Check `displayName` is not empty (required field) 522 + 523 + ## API Reference 524 + 525 + ### Registration Endpoint 526 + 527 + **`POST /xrpc/social.coves.aggregator.register`** 528 + 529 + **Input**: 530 + ```typescript 531 + { 532 + did: string // DID of aggregator (did:plc or did:web) 533 + domain: string // Domain serving .well-known/atproto-did 534 + } 535 + ``` 536 + 537 + **Output**: 538 + ```typescript 539 + { 540 + did: string // Registered DID 541 + handle: string // Handle from DID document 542 + message: string // Next steps message 543 + } 544 + ``` 545 + 546 + **Errors**: 547 + - `InvalidDID`: DID format invalid 548 + - `DomainVerificationFailed`: .well-known verification failed 549 + - `AlreadyRegistered`: DID already registered 550 + - `DIDResolutionFailed`: Could not resolve DID 551 + 552 + ### Query Endpoints 553 + 554 + **`GET /xrpc/social.coves.aggregator.getServices`** 555 + 556 + Get aggregator service details. 557 + 558 + **Parameters**: 559 + - `dids`: Array of DIDs (comma-separated) 560 + 561 + **`GET /xrpc/social.coves.aggregator.getAuthorizations`** 562 + 563 + List communities that authorized an aggregator. 564 + 565 + **Parameters**: 566 + - `aggregatorDid`: Aggregator DID 567 + - `enabledOnly`: Filter to enabled only (default: false) 568 + 569 + **`GET /xrpc/social.coves.aggregator.listForCommunity`** 570 + 571 + List aggregators authorized by a community. 572 + 573 + **Parameters**: 574 + - `communityDid`: Community DID 575 + - `enabledOnly`: Filter to enabled only (default: false) 576 + 577 + ## Further Reading 578 + 579 + - [Aggregator PRD](PRD_AGGREGATORS.md) - Architecture and design decisions 580 + - [atProto Guide](../../ATPROTO_GUIDE.md) - atProto fundamentals 581 + - [Communities PRD](../PRD_COMMUNITIES.md) - Community system overview 582 + - [Setup Scripts README](../../scripts/aggregator-setup/README.md) - Script documentation 583 + 584 + ## Support 585 + 586 + For issues or questions: 587 + 588 + 1. Check this guide's troubleshooting section 589 + 2. Review the PRD and architecture docs 590 + 3. Check Coves GitHub issues 591 + 4. Ask in Coves developer community

+95

scripts/aggregator-setup/1-create-pds-account.sh

··· 1 + #!/bin/bash 2 + 3 + # Script: 1-create-pds-account.sh 4 + # Purpose: Create a PDS account for your aggregator 5 + # 6 + # This script helps you create an account on a PDS (Personal Data Server). 7 + # The PDS will automatically create a DID:PLC for you. 8 + 9 + set -e 10 + 11 + echo "================================================" 12 + echo "Step 1: Create PDS Account for Your Aggregator" 13 + echo "================================================" 14 + echo "" 15 + 16 + # Get PDS URL 17 + read -p "Enter PDS URL (default: https://bsky.social): " PDS_URL 18 + PDS_URL=${PDS_URL:-https://bsky.social} 19 + 20 + # Get credentials 21 + read -p "Enter desired handle (e.g., mynewsbot.bsky.social): " HANDLE 22 + read -p "Enter email: " EMAIL 23 + read -sp "Enter password: " PASSWORD 24 + echo "" 25 + 26 + # Validate inputs 27 + if [ -z "$HANDLE" ] || [ -z "$EMAIL" ] || [ -z "$PASSWORD" ]; then 28 + echo "Error: All fields are required" 29 + exit 1 30 + fi 31 + 32 + echo "" 33 + echo "Creating account on $PDS_URL..." 34 + 35 + # Create account via com.atproto.server.createAccount 36 + RESPONSE=$(curl -s -X POST "$PDS_URL/xrpc/com.atproto.server.createAccount" \ 37 + -H "Content-Type: application/json" \ 38 + -d "{ 39 + \"handle\": \"$HANDLE\", 40 + \"email\": \"$EMAIL\", 41 + \"password\": \"$PASSWORD\" 42 + }") 43 + 44 + # Check if successful 45 + if echo "$RESPONSE" | jq -e '.error' > /dev/null 2>&1; then 46 + echo "Error creating account:" 47 + echo "$RESPONSE" | jq '.' 48 + exit 1 49 + fi 50 + 51 + # Extract DID and access token 52 + DID=$(echo "$RESPONSE" | jq -r '.did') 53 + ACCESS_JWT=$(echo "$RESPONSE" | jq -r '.accessJwt') 54 + REFRESH_JWT=$(echo "$RESPONSE" | jq -r '.refreshJwt') 55 + 56 + if [ -z "$DID" ] || [ "$DID" = "null" ]; then 57 + echo "Error: Failed to extract DID from response" 58 + echo "$RESPONSE" | jq '.' 59 + exit 1 60 + fi 61 + 62 + echo "" 63 + echo "✓ Account created successfully!" 64 + echo "" 65 + echo "=== Save these credentials ====" 66 + echo "DID: $DID" 67 + echo "Handle: $HANDLE" 68 + echo "PDS URL: $PDS_URL" 69 + echo "Email: $EMAIL" 70 + echo "Password: [hidden]" 71 + echo "Access JWT: $ACCESS_JWT" 72 + echo "Refresh JWT: $REFRESH_JWT" 73 + echo "===============================" 74 + echo "" 75 + 76 + # Save to config file 77 + CONFIG_FILE="aggregator-config.env" 78 + cat > "$CONFIG_FILE" <<EOF 79 + # Aggregator Account Configuration 80 + # Generated: $(date) 81 + 82 + AGGREGATOR_DID="$DID" 83 + AGGREGATOR_HANDLE="$HANDLE" 84 + AGGREGATOR_PDS_URL="$PDS_URL" 85 + AGGREGATOR_EMAIL="$EMAIL" 86 + AGGREGATOR_PASSWORD="$PASSWORD" 87 + AGGREGATOR_ACCESS_JWT="$ACCESS_JWT" 88 + AGGREGATOR_REFRESH_JWT="$REFRESH_JWT" 89 + EOF 90 + 91 + echo "✓ Configuration saved to $CONFIG_FILE" 92 + echo "" 93 + echo "IMPORTANT: Keep this file secure! It contains your credentials." 94 + echo "" 95 + echo "Next step: Run ./2-setup-wellknown.sh"

+93

scripts/aggregator-setup/2-setup-wellknown.sh

··· 1 + #!/bin/bash 2 + 3 + # Script: 2-setup-wellknown.sh 4 + # Purpose: Generate .well-known/atproto-did file for domain verification 5 + # 6 + # This script creates the .well-known/atproto-did file that proves you own your domain. 7 + # You'll need to host this file at https://yourdomain.com/.well-known/atproto-did 8 + 9 + set -e 10 + 11 + echo "================================================" 12 + echo "Step 2: Setup .well-known/atproto-did" 13 + echo "================================================" 14 + echo "" 15 + 16 + # Load config if available 17 + if [ -f "aggregator-config.env" ]; then 18 + source aggregator-config.env 19 + echo "✓ Loaded configuration from aggregator-config.env" 20 + echo " DID: $AGGREGATOR_DID" 21 + echo "" 22 + else 23 + echo "Configuration file not found. Please run 1-create-pds-account.sh first." 24 + exit 1 25 + fi 26 + 27 + # Get domain 28 + read -p "Enter your aggregator's domain (e.g., rss-bot.example.com): " DOMAIN 29 + 30 + if [ -z "$DOMAIN" ]; then 31 + echo "Error: Domain is required" 32 + exit 1 33 + fi 34 + 35 + # Save domain to config 36 + echo "" >> aggregator-config.env 37 + echo "AGGREGATOR_DOMAIN=\"$DOMAIN\"" >> aggregator-config.env 38 + 39 + echo "" 40 + echo "Creating .well-known directory..." 41 + mkdir -p .well-known 42 + 43 + # Create the atproto-did file 44 + echo "$AGGREGATOR_DID" > .well-known/atproto-did 45 + 46 + echo "✓ Created .well-known/atproto-did with content: $AGGREGATOR_DID" 47 + echo "" 48 + 49 + echo "================================================" 50 + echo "Next Steps:" 51 + echo "================================================" 52 + echo "" 53 + echo "1. Upload the .well-known directory to your web server" 54 + echo " The file must be accessible at:" 55 + echo " https://$DOMAIN/.well-known/atproto-did" 56 + echo "" 57 + echo "2. Verify it's working by running:" 58 + echo " curl https://$DOMAIN/.well-known/atproto-did" 59 + echo " (Should return: $AGGREGATOR_DID)" 60 + echo "" 61 + echo "3. Once verified, run: ./3-register-with-coves.sh" 62 + echo "" 63 + 64 + # Create nginx example 65 + cat > nginx-example.conf <<EOF 66 + # Example nginx configuration for serving .well-known 67 + # Add this to your nginx server block: 68 + 69 + location /.well-known/atproto-did { 70 + alias /path/to/your/.well-known/atproto-did; 71 + default_type text/plain; 72 + add_header Access-Control-Allow-Origin *; 73 + } 74 + EOF 75 + 76 + echo "✓ Created nginx-example.conf for reference" 77 + echo "" 78 + 79 + # Create Apache example 80 + cat > apache-example.conf <<EOF 81 + # Example Apache configuration for serving .well-known 82 + # Add this to your Apache virtual host: 83 + 84 + Alias /.well-known /path/to/your/.well-known 85 + <Directory /path/to/your/.well-known> 86 + Options None 87 + AllowOverride None 88 + Require all granted 89 + Header set Access-Control-Allow-Origin "*" 90 + </Directory> 91 + EOF 92 + 93 + echo "✓ Created apache-example.conf for reference"

+103

scripts/aggregator-setup/3-register-with-coves.sh

··· 1 + #!/bin/bash 2 + 3 + # Script: 3-register-with-coves.sh 4 + # Purpose: Register your aggregator with a Coves instance 5 + # 6 + # This script calls the social.coves.aggregator.register XRPC endpoint 7 + # to register your aggregator DID with the Coves instance. 8 + 9 + set -e 10 + 11 + echo "================================================" 12 + echo "Step 3: Register with Coves Instance" 13 + echo "================================================" 14 + echo "" 15 + 16 + # Load config if available 17 + if [ -f "aggregator-config.env" ]; then 18 + source aggregator-config.env 19 + echo "✓ Loaded configuration from aggregator-config.env" 20 + echo " DID: $AGGREGATOR_DID" 21 + echo " Domain: $AGGREGATOR_DOMAIN" 22 + echo "" 23 + else 24 + echo "Configuration file not found. Please run previous scripts first." 25 + exit 1 26 + fi 27 + 28 + # Validate domain is set 29 + if [ -z "$AGGREGATOR_DOMAIN" ]; then 30 + echo "Error: AGGREGATOR_DOMAIN not set. Please run 2-setup-wellknown.sh first." 31 + exit 1 32 + fi 33 + 34 + # Get Coves instance URL 35 + read -p "Enter Coves instance URL (default: https://api.coves.social): " COVES_URL 36 + COVES_URL=${COVES_URL:-https://api.coves.social} 37 + 38 + echo "" 39 + echo "Verifying .well-known/atproto-did is accessible..." 40 + 41 + # Verify .well-known is accessible 42 + WELLKNOWN_URL="https://$AGGREGATOR_DOMAIN/.well-known/atproto-did" 43 + WELLKNOWN_CONTENT=$(curl -s "$WELLKNOWN_URL" || echo "ERROR") 44 + 45 + if [ "$WELLKNOWN_CONTENT" = "ERROR" ]; then 46 + echo "✗ Error: Could not access $WELLKNOWN_URL" 47 + echo " Please ensure the file is uploaded and accessible." 48 + exit 1 49 + elif [ "$WELLKNOWN_CONTENT" != "$AGGREGATOR_DID" ]; then 50 + echo "✗ Error: .well-known/atproto-did contains wrong DID" 51 + echo " Expected: $AGGREGATOR_DID" 52 + echo " Got: $WELLKNOWN_CONTENT" 53 + exit 1 54 + fi 55 + 56 + echo "✓ .well-known/atproto-did is correctly configured" 57 + echo "" 58 + 59 + echo "Registering with $COVES_URL..." 60 + 61 + # Call registration endpoint 62 + RESPONSE=$(curl -s -X POST "$COVES_URL/xrpc/social.coves.aggregator.register" \ 63 + -H "Content-Type: application/json" \ 64 + -d "{ 65 + \"did\": \"$AGGREGATOR_DID\", 66 + \"domain\": \"$AGGREGATOR_DOMAIN\" 67 + }") 68 + 69 + # Check if successful 70 + if echo "$RESPONSE" | jq -e '.error' > /dev/null 2>&1; then 71 + echo "✗ Registration failed:" 72 + echo "$RESPONSE" | jq '.' 73 + exit 1 74 + fi 75 + 76 + # Extract response 77 + REGISTERED_DID=$(echo "$RESPONSE" | jq -r '.did') 78 + REGISTERED_HANDLE=$(echo "$RESPONSE" | jq -r '.handle') 79 + MESSAGE=$(echo "$RESPONSE" | jq -r '.message') 80 + 81 + if [ -z "$REGISTERED_DID" ] || [ "$REGISTERED_DID" = "null" ]; then 82 + echo "✗ Error: Unexpected response format" 83 + echo "$RESPONSE" | jq '.' 84 + exit 1 85 + fi 86 + 87 + echo "" 88 + echo "✓ Registration successful!" 89 + echo "" 90 + echo "=== Registration Details ====" 91 + echo "DID: $REGISTERED_DID" 92 + echo "Handle: $REGISTERED_HANDLE" 93 + echo "Message: $MESSAGE" 94 + echo "=============================" 95 + echo "" 96 + 97 + # Save Coves URL to config 98 + echo "" >> aggregator-config.env 99 + echo "COVES_INSTANCE_URL=\"$COVES_URL\"" >> aggregator-config.env 100 + 101 + echo "✓ Updated aggregator-config.env with Coves instance URL" 102 + echo "" 103 + echo "Next step: Run ./4-create-service-declaration.sh"

+125

scripts/aggregator-setup/4-create-service-declaration.sh

··· 1 + #!/bin/bash 2 + 3 + # Script: 4-create-service-declaration.sh 4 + # Purpose: Create aggregator service declaration record 5 + # 6 + # This script writes a social.coves.aggregator.service record to your aggregator's repository. 7 + # This record contains metadata about your aggregator (name, description, etc.) and will be 8 + # indexed by Coves' Jetstream consumer into the aggregators table. 9 + 10 + set -e 11 + 12 + echo "================================================" 13 + echo "Step 4: Create Service Declaration" 14 + echo "================================================" 15 + echo "" 16 + 17 + # Load config if available 18 + if [ -f "aggregator-config.env" ]; then 19 + source aggregator-config.env 20 + echo "✓ Loaded configuration from aggregator-config.env" 21 + echo " DID: $AGGREGATOR_DID" 22 + echo " PDS URL: $AGGREGATOR_PDS_URL" 23 + echo "" 24 + else 25 + echo "Configuration file not found. Please run previous scripts first." 26 + exit 1 27 + fi 28 + 29 + # Validate required fields 30 + if [ -z "$AGGREGATOR_ACCESS_JWT" ]; then 31 + echo "Error: AGGREGATOR_ACCESS_JWT not set. Please run 1-create-pds-account.sh first." 32 + exit 1 33 + fi 34 + 35 + echo "Enter aggregator metadata:" 36 + echo "" 37 + 38 + # Get metadata from user 39 + read -p "Display Name (e.g., 'RSS News Aggregator'): " DISPLAY_NAME 40 + read -p "Description: " DESCRIPTION 41 + read -p "Source URL (e.g., 'https://github.com/yourname/aggregator'): " SOURCE_URL 42 + read -p "Maintainer DID (your personal DID, optional): " MAINTAINER_DID 43 + 44 + if [ -z "$DISPLAY_NAME" ]; then 45 + echo "Error: Display name is required" 46 + exit 1 47 + fi 48 + 49 + echo "" 50 + echo "Creating service declaration record..." 51 + 52 + # Build the service record 53 + SERVICE_RECORD=$(cat <<EOF 54 + { 55 + "\$type": "social.coves.aggregator.service", 56 + "did": "$AGGREGATOR_DID", 57 + "displayName": "$DISPLAY_NAME", 58 + "description": "$DESCRIPTION", 59 + "sourceUrl": "$SOURCE_URL", 60 + "maintainer": "$MAINTAINER_DID", 61 + "createdAt": "$(date -u +"%Y-%m-%dT%H:%M:%SZ")" 62 + } 63 + EOF 64 + ) 65 + 66 + # Call com.atproto.repo.createRecord 67 + RESPONSE=$(curl -s -X POST "$AGGREGATOR_PDS_URL/xrpc/com.atproto.repo.createRecord" \ 68 + -H "Authorization: Bearer $AGGREGATOR_ACCESS_JWT" \ 69 + -H "Content-Type: application/json" \ 70 + -d "{ 71 + \"repo\": \"$AGGREGATOR_DID\", 72 + \"collection\": \"social.coves.aggregator.service\", 73 + \"rkey\": \"self\", 74 + \"record\": $SERVICE_RECORD 75 + }") 76 + 77 + # Check if successful 78 + if echo "$RESPONSE" | jq -e '.error' > /dev/null 2>&1; then 79 + echo "✗ Failed to create service declaration:" 80 + echo "$RESPONSE" | jq '.' 81 + exit 1 82 + fi 83 + 84 + # Extract response 85 + RECORD_URI=$(echo "$RESPONSE" | jq -r '.uri') 86 + RECORD_CID=$(echo "$RESPONSE" | jq -r '.cid') 87 + 88 + if [ -z "$RECORD_URI" ] || [ "$RECORD_URI" = "null" ]; then 89 + echo "✗ Error: Unexpected response format" 90 + echo "$RESPONSE" | jq '.' 91 + exit 1 92 + fi 93 + 94 + echo "" 95 + echo "✓ Service declaration created successfully!" 96 + echo "" 97 + echo "=== Record Details ====" 98 + echo "URI: $RECORD_URI" 99 + echo "CID: $RECORD_CID" 100 + echo "=======================" 101 + echo "" 102 + 103 + # Save to config 104 + echo "" >> aggregator-config.env 105 + echo "SERVICE_DECLARATION_URI=\"$RECORD_URI\"" >> aggregator-config.env 106 + echo "SERVICE_DECLARATION_CID=\"$RECORD_CID\"" >> aggregator-config.env 107 + 108 + echo "✓ Updated aggregator-config.env" 109 + echo "" 110 + echo "================================================" 111 + echo "Setup Complete!" 112 + echo "================================================" 113 + echo "" 114 + echo "Your aggregator is now registered with Coves!" 115 + echo "" 116 + echo "Next steps:" 117 + echo "1. Wait a few seconds for Jetstream to index your service declaration" 118 + echo "2. Verify your aggregator appears in the aggregators list" 119 + echo "3. Community moderators can now authorize your aggregator" 120 + echo "4. Once authorized, you can start posting to communities" 121 + echo "" 122 + echo "To test posting, use the Coves XRPC endpoint:" 123 + echo " POST $COVES_INSTANCE_URL/xrpc/social.coves.community.post.create" 124 + echo "" 125 + echo "See docs/aggregators/SETUP_GUIDE.md for more information"

+252

scripts/aggregator-setup/README.md

··· 1 + # Aggregator Setup Scripts 2 + 3 + This directory contains scripts to help you set up and register your aggregator with Coves instances. 4 + 5 + ## Overview 6 + 7 + Aggregators are automated services that post content to Coves communities. They are similar to Bluesky's feed generators and labelers. To use aggregators with Coves, you need to: 8 + 9 + 1. Create a PDS account for your aggregator (gets you a DID) 10 + 2. Prove you own a domain via `.well-known/atproto-did` 11 + 3. Register with a Coves instance 12 + 4. Create a service declaration record 13 + 14 + These scripts automate this process for you. 15 + 16 + ## Prerequisites 17 + 18 + - **Domain ownership**: You must own a domain where you can host the `.well-known/atproto-did` file 19 + - **Web server**: Ability to serve static files over HTTPS 20 + - **Tools**: `curl`, `jq` (for JSON processing) 21 + - **Account**: Email address for creating the PDS account 22 + 23 + ## Quick Start 24 + 25 + ### Interactive Setup (Recommended) 26 + 27 + Run the scripts in order: 28 + 29 + ```bash 30 + # Make scripts executable 31 + chmod +x *.sh 32 + 33 + # Step 1: Create PDS account 34 + ./1-create-pds-account.sh 35 + 36 + # Step 2: Generate .well-known file 37 + ./2-setup-wellknown.sh 38 + 39 + # Step 3: Register with Coves (after uploading .well-known) 40 + ./3-register-with-coves.sh 41 + 42 + # Step 4: Create service declaration 43 + ./4-create-service-declaration.sh 44 + ``` 45 + 46 + ### Automated Setup Example 47 + 48 + For a reference implementation of automated setup, see the Kagi News aggregator at [aggregators/kagi-news/scripts/setup.sh](../../aggregators/kagi-news/scripts/setup.sh). 49 + 50 + The Kagi script shows how to automate all 4 steps (with the manual .well-known upload step in between). 51 + 52 + ## Script Reference 53 + 54 + ### 1-create-pds-account.sh 55 + 56 + **Purpose**: Creates a PDS account for your aggregator 57 + 58 + **Prompts for**: 59 + - PDS URL (default: https://bsky.social) 60 + - Handle (e.g., mynewsbot.bsky.social) 61 + - Email 62 + - Password 63 + 64 + **Outputs**: 65 + - `aggregator-config.env` - Configuration file with DID and credentials 66 + - Prints your DID and access tokens 67 + 68 + **Notes**: 69 + - Keep the config file secure! It contains your credentials 70 + - The PDS automatically generates a DID:PLC for you 71 + - You can use any PDS service, not just bsky.social 72 + 73 + ### 2-setup-wellknown.sh 74 + 75 + **Purpose**: Generates the `.well-known/atproto-did` file for domain verification 76 + 77 + **Prompts for**: 78 + - Your domain (e.g., rss-bot.example.com) 79 + 80 + **Outputs**: 81 + - `.well-known/atproto-did` - File containing your DID 82 + - `nginx-example.conf` - Example nginx configuration 83 + - `apache-example.conf` - Example Apache configuration 84 + 85 + **Manual step required**: 86 + Upload the `.well-known` directory to your web server. The file must be accessible at: 87 + ``` 88 + https://yourdomain.com/.well-known/atproto-did 89 + ``` 90 + 91 + **Verify it works**: 92 + ```bash 93 + curl https://yourdomain.com/.well-known/atproto-did 94 + # Should return your DID (e.g., did:plc:abc123...) 95 + ``` 96 + 97 + ### 3-register-with-coves.sh 98 + 99 + **Purpose**: Registers your aggregator with a Coves instance 100 + 101 + **Prompts for**: 102 + - Coves instance URL (default: https://api.coves.social) 103 + 104 + **Prerequisites**: 105 + - `.well-known/atproto-did` must be accessible from your domain 106 + - Scripts 1 and 2 must be completed 107 + 108 + **What it does**: 109 + 1. Verifies your `.well-known/atproto-did` is accessible 110 + 2. Calls `social.coves.aggregator.register` XRPC endpoint 111 + 3. Coves verifies domain ownership 112 + 4. Inserts your aggregator into the `users` table 113 + 114 + **Outputs**: 115 + - Updates `aggregator-config.env` with Coves instance URL 116 + - Prints registration confirmation 117 + 118 + ### 4-create-service-declaration.sh 119 + 120 + **Purpose**: Creates the service declaration record in your repository 121 + 122 + **Prompts for**: 123 + - Display name (e.g., "RSS News Aggregator") 124 + - Description 125 + - Source URL (GitHub repo, etc.) 126 + - Maintainer DID (optional) 127 + 128 + **What it does**: 129 + 1. Creates a `social.coves.aggregator.service` record at `at://your-did/social.coves.aggregator.service/self` 130 + 2. Jetstream consumer will index this into the `aggregators` table 131 + 3. Communities can now discover and authorize your aggregator 132 + 133 + **Outputs**: 134 + - Updates `aggregator-config.env` with record URI and CID 135 + - Prints record details 136 + 137 + ## Configuration File 138 + 139 + After running the scripts, you'll have an `aggregator-config.env` file with: 140 + 141 + ```bash 142 + AGGREGATOR_DID="did:plc:..." 143 + AGGREGATOR_HANDLE="mynewsbot.bsky.social" 144 + AGGREGATOR_PDS_URL="https://bsky.social" 145 + AGGREGATOR_EMAIL="bot@example.com" 146 + AGGREGATOR_PASSWORD="..." 147 + AGGREGATOR_ACCESS_JWT="..." 148 + AGGREGATOR_REFRESH_JWT="..." 149 + AGGREGATOR_DOMAIN="rss-bot.example.com" 150 + COVES_INSTANCE_URL="https://api.coves.social" 151 + SERVICE_DECLARATION_URI="at://did:plc:.../social.coves.aggregator.service/self" 152 + SERVICE_DECLARATION_CID="..." 153 + ``` 154 + 155 + **Use this in your aggregator code** to authenticate and post. 156 + 157 + ## What Happens Next? 158 + 159 + After completing all 4 steps: 160 + 161 + 1. **Your aggregator is registered** in the Coves instance's `users` table 162 + 2. **Your service declaration is indexed** in the `aggregators` table (takes a few seconds) 163 + 3. **Community moderators can now authorize** your aggregator for their communities 164 + 4. **Once authorized**, your aggregator can post to those communities 165 + 166 + ## Creating an Authorization 167 + 168 + Authorizations are created by community moderators, not by aggregators. The moderator writes a `social.coves.aggregator.authorization` record to their community's repository. 169 + 170 + See `docs/aggregators/SETUP_GUIDE.md` for more information on the authorization process. 171 + 172 + ## Posting to Communities 173 + 174 + Once authorized, your aggregator can post using: 175 + 176 + ```bash 177 + curl -X POST https://api.coves.social/xrpc/social.coves.community.post.create \ 178 + -H "Authorization: Bearer $AGGREGATOR_ACCESS_JWT" \ 179 + -H "Content-Type: application/json" \ 180 + -d '{ 181 + "communityDid": "did:plc:...", 182 + "post": { 183 + "text": "Your post content", 184 + "createdAt": "2024-01-15T12:00:00Z" 185 + } 186 + }' 187 + ``` 188 + 189 + ## Troubleshooting 190 + 191 + ### Error: "DomainVerificationFailed" 192 + 193 + - Verify `.well-known/atproto-did` is accessible: `curl https://yourdomain.com/.well-known/atproto-did` 194 + - Check the content matches your DID exactly (no extra whitespace) 195 + - Ensure HTTPS is working (not HTTP) 196 + - Check CORS headers if accessing from browser 197 + 198 + ### Error: "AlreadyRegistered" 199 + 200 + - You've already registered this DID with this Coves instance 201 + - This is safe to ignore if you're re-running the setup 202 + 203 + ### Error: "DIDResolutionFailed" 204 + 205 + - Your DID might be invalid or not found in the PLC directory 206 + - Verify your DID exists: `curl https://plc.directory/<your-did>` 207 + - Wait a few seconds and try again (PLC directory might be propagating) 208 + 209 + ### Service declaration not appearing 210 + 211 + - Wait 5-10 seconds for Jetstream consumer to index it 212 + - Check the Jetstream logs for errors 213 + - Verify the record was created: Check your PDS at `at://your-did/social.coves.aggregator.service/self` 214 + 215 + ## Example: Kagi News Aggregator 216 + 217 + For a complete reference implementation, see the Kagi News aggregator at `aggregators/kagi-news/`. 218 + 219 + The Kagi aggregator includes an automated setup script at [aggregators/kagi-news/scripts/setup.sh](../../aggregators/kagi-news/scripts/setup.sh) that demonstrates how to: 220 + 221 + - Automate the entire registration process 222 + - Use environment variables for configuration 223 + - Handle errors gracefully 224 + - Integrate the setup into your aggregator project 225 + 226 + This shows how you can package scripts 1-4 into a single automated flow for your specific aggregator. 227 + 228 + ## Security Notes 229 + 230 + - **Never commit `aggregator-config.env`** to version control 231 + - Store credentials securely (use environment variables or secret management) 232 + - Rotate access tokens regularly 233 + - Use HTTPS for all API calls 234 + - Validate community authorization before posting 235 + 236 + ## More Information 237 + 238 + - [Aggregator Setup Guide](../../docs/aggregators/SETUP_GUIDE.md) 239 + - [Aggregator PRD](../../docs/aggregators/PRD_AGGREGATORS.md) 240 + - [atProto Identity Guide](../../ATPROTO_GUIDE.md) 241 + - [Coves Communities PRD](../../docs/PRD_COMMUNITIES.md) 242 + 243 + ## Support 244 + 245 + If you encounter issues: 246 + 247 + 1. Check the troubleshooting section above 248 + 2. Review the full documentation in `docs/aggregators/` 249 + 3. Open an issue on GitHub with: 250 + - Which script failed 251 + - Error message 252 + - Your domain (without credentials)