bretton.dev
A community based topic aggregation platform built on atproto
feat(lexicons): add aggregator lexicon schemas
Add complete lexicon definitions for the aggregator system following
atProto patterns (Feed Generator + Labeler model).
Records (2):
- service.json: Aggregator service declaration with config schema
- authorization.json: Community authorization with enabled status
Procedures (3):
- enable.json: Enable aggregator for community (moderator only)
- disable.json: Disable aggregator
- updateConfig.json: Update aggregator configuration
Queries (3):
- getServices.json: Fetch aggregator details by DIDs
- getAuthorizations.json: List communities using aggregator
- listForCommunity.json: List aggregators for community
Definitions:
- aggregatorView: Basic aggregator metadata
- aggregatorViewDetailed: Aggregator with stats
- authorizationView: Authorization from community perspective
- communityAuthView: Authorization from aggregator perspective
Design decisions:
- Removed aggregatorType enum (too rigid for alpha)
- Used JSON Schema for configSchema (validation + UI generation)
- Followed Bluesky feed generator patterns
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
+936
-1278
+224
-1278
docs/aggregators/PRD_AGGREGATORS.md
+224
-1278
docs/aggregators/PRD_AGGREGATORS.md
···
1
1
# Aggregators PRD: Automated Content Posting System
2
2
3
-
**Status:** Planning / Design Phase
3
+
**Status:** In Development - Phase 1 (Core Infrastructure)
4
4
**Owner:** Platform Team
5
-
**Last Updated:** 2025-10-19
5
+
**Last Updated:** 2025-10-20
6
+
7
+
---
6
8
7
9
## Overview
8
10
9
-
Coves Aggregators are autonomous services that automatically post content to communities. Each aggregator is identified by its own DID and operates as a specialized actor within the atProto ecosystem. This system enables communities to have automated content feeds (RSS, sports results, TV/movie discussion threads, Bluesky mirrors, etc.) while maintaining full community control over which aggregators can post and what content they can create.
11
+
Coves Aggregators are autonomous services that automatically post content to communities. Each aggregator is identified by its own DID and operates as a specialized actor within the atProto ecosystem. This enables communities to have automated content feeds (RSS, sports results, TV/movie discussion threads, Bluesky mirrors, etc.) while maintaining full community control.
10
12
11
13
**Key Differentiator:** Unlike other platforms where users manually aggregate content, Coves communities can enable automated aggregators to handle routine posting tasks, creating a more dynamic and up-to-date community experience.
12
14
15
+
---
16
+
13
17
## Architecture Principles
14
18
15
19
### ✅ atProto-Compliant Design
16
20
17
-
Aggregators follow established atProto patterns for autonomous services:
18
-
19
-
**Pattern:** Feed Generators + Labelers Model
20
-
- Each aggregator has its own DID (like feed generators)
21
-
- Declaration record published in aggregator's repo (like `app.bsky.feed.generator`)
22
-
- DID document advertises service endpoint
23
-
- Service makes authenticated XRPC calls
24
-
- Communities explicitly authorize aggregators (like subscribing to labelers)
25
-
26
-
**Key Design Decisions:**
21
+
Aggregators follow established atProto patterns for autonomous services (Feed Generators + Labelers model):
27
22
28
23
1. **Aggregators are Actors, Not a Separate System**
29
-
- Aggregators authenticate as themselves (their DID)
24
+
- Each aggregator has its own DID
25
+
- Authenticate as themselves via JWT
30
26
- Use existing `social.coves.post.create` endpoint
31
27
- Post record's `author` field = aggregator DID (server-populated)
32
28
- No separate posting API needed
33
29
34
30
2. **Community Authorization Model**
35
-
- Communities create `social.coves.aggregator.authorization` records
36
-
- These records grant specific aggregators permission to post
37
-
- Authorizations include configuration (which RSS feeds, which users to mirror, etc.)
38
-
- Can be enabled/disabled at any time
31
+
- Communities create `social.coves.aggregator.authorization` records in their repo
32
+
- Records grant specific aggregators permission to post
33
+
- Include aggregator-specific configuration
34
+
- Can be enabled/disabled without deletion
39
35
40
36
3. **Hybrid Hosting**
41
-
- Coves can host official aggregators (RSS, sports, media)
42
-
- Third parties can build and host their own aggregators
43
-
- SDK provided for easy aggregator development
44
-
- All aggregators use same authorization system
45
-
46
-
---
47
-
48
-
## Architecture Overview
49
-
50
-
```
51
-
┌────────────────────────────────────────────────────────────┐
52
-
│ Aggregator Service (External) │
53
-
│ DID: did:web:rss-bot.coves.social │
54
-
│ │
55
-
│ - Watches external data sources (RSS, APIs, etc.) │
56
-
│ - Processes content (LLM deduplication, formatting) │
57
-
│ - Queries which communities have authorized it │
58
-
│ - Creates posts via social.coves.post.create │
59
-
│ - Responds to config queries via XRPC │
60
-
└────────────────────────────────────────────────────────────┘
61
-
│
62
-
│ 1. Authenticate as aggregator DID (JWT)
63
-
│ 2. Call social.coves.post.create
64
-
│ {
65
-
│ community: "did:plc:gaming123",
66
-
│ title: "...",
67
-
│ content: "...",
68
-
│ federatedFrom: { platform: "rss", ... }
69
-
│ }
70
-
▼
71
-
┌────────────────────────────────────────────────────────────┐
72
-
│ Coves AppView (social.coves.post.create Handler) │
73
-
│ │
74
-
│ 1. Extract DID from JWT (aggregator's DID) │
75
-
│ 2. Check if DID is registered aggregator │
76
-
│ 3. Validate authorization record exists & enabled │
77
-
│ 4. Apply aggregator-specific rate limits │
78
-
│ 5. Validate content against community rules │
79
-
│ 6. Create post with author = aggregator DID │
80
-
└────────────────────────────────────────────────────────────┘
81
-
│
82
-
│ Post record created:
83
-
│ {
84
-
│ $type: "social.coves.post.record",
85
-
│ author: "did:web:rss-bot.coves.social",
86
-
│ community: "did:plc:gaming123",
87
-
│ title: "Tech News Roundup",
88
-
│ content: "...",
89
-
│ federatedFrom: {
90
-
│ platform: "rss",
91
-
│ uri: "https://techcrunch.com/..."
92
-
│ }
93
-
│ }
94
-
▼
95
-
┌────────────────────────────────────────────────────────────┐
96
-
│ Jetstream → AppView Indexing │
97
-
│ - Post indexed with aggregator attribution │
98
-
│ - UI shows: "🤖 Posted by RSS Aggregator" │
99
-
│ - Community feed includes automated posts │
100
-
└────────────────────────────────────────────────────────────┘
101
-
```
37
+
- Coves can host official aggregators
38
+
- Third parties can build and host their own
39
+
- All use same authorization system
102
40
103
41
---
104
42
105
-
## Use Cases
106
-
107
-
### 1. RSS News Aggregator
108
-
**Problem:** Multiple users posting the same breaking news from different sources
109
-
**Solution:** RSS aggregator with LLM deduplication
110
-
- Watches configured RSS feeds
111
-
- Uses LLM to identify duplicate stories from different outlets
112
-
- Creates single "megathread" with all sources linked
113
-
- Posts unbiased summary of event
114
-
- Automatically tags with relevant topics
115
-
116
-
**Community Config:**
117
-
```json
118
-
{
119
-
"aggregatorDid": "did:web:rss-bot.coves.social",
120
-
"enabled": true,
121
-
"config": {
122
-
"feeds": [
123
-
"https://techcrunch.com/feed",
124
-
"https://arstechnica.com/feed"
125
-
],
126
-
"topics": ["technology", "ai"],
127
-
"dedupeWindow": "6h",
128
-
"minSources": 2
129
-
}
130
-
}
131
-
```
132
-
133
-
### 2. Bluesky Post Mirror
134
-
**Problem:** Want to surface specific Bluesky discussions in community
135
-
**Solution:** Bluesky mirror aggregator
136
-
- Monitors specific users or hashtags on Bluesky
137
-
- Creates posts in community when criteria met
138
-
- Preserves `originalAuthor` metadata
139
-
- Links back to original Bluesky thread
140
-
141
-
**Community Config:**
142
-
```json
143
-
{
144
-
"aggregatorDid": "did:web:bsky-mirror.coves.social",
145
-
"enabled": true,
146
-
"config": {
147
-
"mirrorUsers": [
148
-
"alice.bsky.social",
149
-
"bob.bsky.social"
150
-
],
151
-
"hashtags": ["covesalpha"],
152
-
"minLikes": 10
153
-
}
154
-
}
155
-
```
43
+
## Core Components
156
44
157
-
### 3. Sports Results Aggregator
158
-
**Problem:** Need post-game threads created immediately after games end
159
-
**Solution:** Sports aggregator watching game APIs
160
-
- Monitors sports APIs for game completions
161
-
- Creates post-game thread with final score, stats
162
-
- Tags with team names and league
163
-
- Posts within minutes of game ending
45
+
### 1. Service Declaration Record
46
+
**Lexicon:** `social.coves.aggregator.service`
47
+
**Location:** Aggregator's repository
48
+
**Key:** `literal:self`
164
49
165
-
**Community Config:**
166
-
```json
167
-
{
168
-
"aggregatorDid": "did:web:sports-bot.coves.social",
169
-
"enabled": true,
170
-
"config": {
171
-
"league": "NBA",
172
-
"teams": ["Lakers", "Warriors"],
173
-
"includeStats": true,
174
-
"autoPin": true
175
-
}
176
-
}
177
-
```
50
+
Declares aggregator existence and provides metadata for discovery.
178
51
179
-
### 4. TV/Movie Discussion Aggregator
180
-
**Problem:** Want episode discussion threads created when shows air
181
-
**Solution:** Media aggregator tracking release schedules
182
-
- Uses TMDB/IMDB APIs for release dates
183
-
- Creates discussion threads when episodes/movies release
184
-
- Includes metadata (cast, synopsis, ratings)
185
-
- Automatically pins for premiere episodes
52
+
**Required Fields:**
53
+
- `did` - Aggregator's DID (must match repo)
54
+
- `displayName` - Human-readable name
55
+
- `createdAt` - Creation timestamp
186
56
187
-
**Community Config:**
188
-
```json
189
-
{
190
-
"aggregatorDid": "did:web:media-bot.coves.social",
191
-
"enabled": true,
192
-
"config": {
193
-
"shows": [
194
-
{"tmdbId": "1234", "name": "Breaking Bad"}
195
-
],
196
-
"createOn": "airDate",
197
-
"timezone": "America/New_York",
198
-
"spoilerProtection": true
199
-
}
200
-
}
201
-
```
57
+
**Optional Fields:**
58
+
- `description` - What this aggregator does
59
+
- `avatar` - Avatar image blob
60
+
- `configSchema` - JSON Schema for community config validation
61
+
- `sourceUrl` - Link to source code (transparency)
62
+
- `maintainer` - DID of maintainer
202
63
203
64
---
204
65
205
-
## Lexicon Schemas
66
+
### 2. Authorization Record
67
+
**Lexicon:** `social.coves.aggregator.authorization`
68
+
**Location:** Community's repository
69
+
**Key:** `any`
206
70
207
-
### 1. Aggregator Service Declaration
71
+
Grants an aggregator permission to post with specific configuration.
208
72
209
-
**Collection:** `social.coves.aggregator.service`
210
-
**Key:** `literal:self` (one per aggregator account)
211
-
**Location:** Aggregator's own repository
212
-
213
-
This record declares the existence of an aggregator service and provides metadata for discovery.
214
-
215
-
```json
216
-
{
217
-
"lexicon": 1,
218
-
"id": "social.coves.aggregator.service",
219
-
"defs": {
220
-
"main": {
221
-
"type": "record",
222
-
"description": "Declaration of an aggregator service that can post to communities",
223
-
"key": "literal:self",
224
-
"record": {
225
-
"type": "object",
226
-
"required": ["did", "displayName", "createdAt", "aggregatorType"],
227
-
"properties": {
228
-
"did": {
229
-
"type": "string",
230
-
"format": "did",
231
-
"description": "DID of the aggregator service (must match repo DID)"
232
-
},
233
-
"displayName": {
234
-
"type": "string",
235
-
"maxGraphemes": 64,
236
-
"maxLength": 640,
237
-
"description": "Human-readable name (e.g., 'RSS News Aggregator')"
238
-
},
239
-
"description": {
240
-
"type": "string",
241
-
"maxGraphemes": 300,
242
-
"maxLength": 3000,
243
-
"description": "Description of what this aggregator does"
244
-
},
245
-
"avatar": {
246
-
"type": "blob",
247
-
"accept": ["image/png", "image/jpeg"],
248
-
"maxSize": 1000000,
249
-
"description": "Avatar image for bot identity"
250
-
},
251
-
"aggregatorType": {
252
-
"type": "string",
253
-
"knownValues": [
254
-
"social.coves.aggregator.types#rss",
255
-
"social.coves.aggregator.types#blueskyMirror",
256
-
"social.coves.aggregator.types#sports",
257
-
"social.coves.aggregator.types#media",
258
-
"social.coves.aggregator.types#custom"
259
-
],
260
-
"description": "Type of aggregator for categorization"
261
-
},
262
-
"configSchema": {
263
-
"type": "unknown",
264
-
"description": "JSON Schema describing config options for this aggregator. Communities use this to know what configuration fields are available."
265
-
},
266
-
"sourceUrl": {
267
-
"type": "string",
268
-
"format": "uri",
269
-
"description": "URL to aggregator's source code (for transparency)"
270
-
},
271
-
"maintainer": {
272
-
"type": "string",
273
-
"format": "did",
274
-
"description": "DID of person/organization maintaining this aggregator"
275
-
},
276
-
"createdAt": {
277
-
"type": "string",
278
-
"format": "datetime"
279
-
}
280
-
}
281
-
}
282
-
}
283
-
}
284
-
}
285
-
```
73
+
**Required Fields:**
74
+
- `aggregatorDid` - DID of authorized aggregator
75
+
- `communityDid` - DID of community (must match repo)
76
+
- `enabled` - Active status (toggleable)
77
+
- `createdAt` - When authorized
286
78
287
-
**Example Record:**
288
-
```json
289
-
{
290
-
"$type": "social.coves.aggregator.service",
291
-
"did": "did:web:rss-bot.coves.social",
292
-
"displayName": "RSS News Aggregator",
293
-
"description": "Automatically posts breaking news from configured RSS feeds with LLM-powered deduplication",
294
-
"aggregatorType": "social.coves.aggregator.types#rss",
295
-
"configSchema": {
296
-
"type": "object",
297
-
"properties": {
298
-
"feeds": {
299
-
"type": "array",
300
-
"items": { "type": "string", "format": "uri" }
301
-
},
302
-
"topics": {
303
-
"type": "array",
304
-
"items": { "type": "string" }
305
-
},
306
-
"dedupeWindow": { "type": "string" },
307
-
"minSources": { "type": "integer", "minimum": 1 }
308
-
}
309
-
},
310
-
"sourceUrl": "https://github.com/coves-social/rss-aggregator",
311
-
"maintainer": "did:plc:coves-platform",
312
-
"createdAt": "2025-10-19T12:00:00Z"
313
-
}
314
-
```
79
+
**Optional Fields:**
80
+
- `config` - Aggregator-specific config (validated against schema)
81
+
- `createdBy` - Moderator who authorized
82
+
- `disabledAt` / `disabledBy` - Audit trail
315
83
316
84
---
317
85
318
-
### 2. Community Authorization Record
319
-
320
-
**Collection:** `social.coves.aggregator.authorization`
321
-
**Key:** `any` (one per aggregator per community)
322
-
**Location:** Community's repository
323
-
324
-
This record grants an aggregator permission to post to a community and contains aggregator-specific configuration.
86
+
## Data Flow
325
87
326
-
```json
327
-
{
328
-
"lexicon": 1,
329
-
"id": "social.coves.aggregator.authorization",
330
-
"defs": {
331
-
"main": {
332
-
"type": "record",
333
-
"description": "Authorization for an aggregator to post to a community with specific configuration",
334
-
"key": "any",
335
-
"record": {
336
-
"type": "object",
337
-
"required": ["aggregatorDid", "communityDid", "createdAt", "enabled"],
338
-
"properties": {
339
-
"aggregatorDid": {
340
-
"type": "string",
341
-
"format": "did",
342
-
"description": "DID of the authorized aggregator"
343
-
},
344
-
"communityDid": {
345
-
"type": "string",
346
-
"format": "did",
347
-
"description": "DID of the community granting access (must match repo DID)"
348
-
},
349
-
"enabled": {
350
-
"type": "boolean",
351
-
"description": "Whether this aggregator is currently active. Can be toggled without deleting the record."
352
-
},
353
-
"config": {
354
-
"type": "unknown",
355
-
"description": "Aggregator-specific configuration. Must conform to the aggregator's configSchema."
356
-
},
357
-
"createdAt": {
358
-
"type": "string",
359
-
"format": "datetime"
360
-
},
361
-
"createdBy": {
362
-
"type": "string",
363
-
"format": "did",
364
-
"description": "DID of moderator who authorized this aggregator"
365
-
},
366
-
"disabledAt": {
367
-
"type": "string",
368
-
"format": "datetime",
369
-
"description": "When this authorization was disabled (if enabled=false)"
370
-
},
371
-
"disabledBy": {
372
-
"type": "string",
373
-
"format": "did",
374
-
"description": "DID of moderator who disabled this aggregator"
375
-
}
376
-
}
377
-
}
378
-
}
379
-
}
380
-
}
381
88
```
382
-
383
-
**Example Record:**
384
-
```json
385
-
{
386
-
"$type": "social.coves.aggregator.authorization",
387
-
"aggregatorDid": "did:web:rss-bot.coves.social",
388
-
"communityDid": "did:plc:gaming123",
389
-
"enabled": true,
390
-
"config": {
391
-
"feeds": [
392
-
"https://techcrunch.com/feed",
393
-
"https://arstechnica.com/feed"
394
-
],
395
-
"topics": ["technology", "ai", "gaming"],
396
-
"dedupeWindow": "6h",
397
-
"minSources": 2
398
-
},
399
-
"createdAt": "2025-10-19T14:00:00Z",
400
-
"createdBy": "did:plc:alice123"
401
-
}
402
-
```
403
-
404
-
---
405
-
406
-
### 3. Aggregator Type Definitions
407
-
408
-
**Collection:** `social.coves.aggregator.types`
409
-
**Purpose:** Define known aggregator types for categorization
410
-
411
-
```json
412
-
{
413
-
"lexicon": 1,
414
-
"id": "social.coves.aggregator.types",
415
-
"defs": {
416
-
"rss": {
417
-
"type": "string",
418
-
"description": "Aggregator that monitors RSS/Atom feeds"
419
-
},
420
-
"blueskyMirror": {
421
-
"type": "string",
422
-
"description": "Aggregator that mirrors Bluesky posts"
423
-
},
424
-
"sports": {
425
-
"type": "string",
426
-
"description": "Aggregator for sports scores and game threads"
427
-
},
428
-
"media": {
429
-
"type": "string",
430
-
"description": "Aggregator for TV/movie discussion threads"
431
-
},
432
-
"custom": {
433
-
"type": "string",
434
-
"description": "Custom third-party aggregator"
435
-
}
436
-
}
437
-
}
89
+
Aggregator Service (External)
90
+
│
91
+
│ 1. Authenticates as aggregator DID (JWT)
92
+
│ 2. Calls social.coves.post.create
93
+
▼
94
+
Coves AppView Handler
95
+
│
96
+
│ 1. Extract DID from JWT
97
+
│ 2. Check if DID is registered aggregator
98
+
│ 3. Validate authorization exists & enabled
99
+
│ 4. Apply aggregator rate limits
100
+
│ 5. Create post with author = aggregator DID
101
+
▼
102
+
Jetstream → AppView Indexing
103
+
│
104
+
│ Post indexed with aggregator attribution
105
+
│ UI shows: "🤖 Posted by [Aggregator Name]"
106
+
▼
107
+
Community Feed
438
108
```
439
109
440
110
---
···
443
113
444
114
### For Communities (Moderators)
445
115
446
-
#### `social.coves.aggregator.enable`
447
-
Enable an aggregator for a community
448
-
449
-
**Input:**
450
-
```json
451
-
{
452
-
"aggregatorDid": "did:web:rss-bot.coves.social",
453
-
"config": {
454
-
"feeds": ["https://techcrunch.com/feed"],
455
-
"topics": ["technology"]
456
-
}
457
-
}
458
-
```
459
-
460
-
**Output:**
461
-
```json
462
-
{
463
-
"uri": "at://did:plc:gaming123/social.coves.aggregator.authorization/3jui7kd58dt2g",
464
-
"cid": "bafyreif5...",
465
-
"authorization": {
466
-
"aggregatorDid": "did:web:rss-bot.coves.social",
467
-
"communityDid": "did:plc:gaming123",
468
-
"enabled": true,
469
-
"config": {...},
470
-
"createdAt": "2025-10-19T14:00:00Z"
471
-
}
472
-
}
473
-
```
474
-
475
-
**Behavior:**
476
-
- Validates caller is community moderator
477
-
- Validates aggregator exists and has service declaration
478
-
- Validates config against aggregator's configSchema
479
-
- Creates authorization record in community's repo
480
-
- Indexes to AppView for authorization checks
481
-
482
-
**Errors:**
483
-
- `NotAuthorized` - Caller is not a moderator
484
-
- `AggregatorNotFound` - Aggregator DID doesn't exist
485
-
- `InvalidConfig` - Config doesn't match configSchema
486
-
487
-
---
488
-
489
-
#### `social.coves.aggregator.disable`
490
-
Disable an aggregator for a community
491
-
492
-
**Input:**
493
-
```json
494
-
{
495
-
"aggregatorDid": "did:web:rss-bot.coves.social"
496
-
}
497
-
```
498
-
499
-
**Output:**
500
-
```json
501
-
{
502
-
"uri": "at://did:plc:gaming123/social.coves.aggregator.authorization/3jui7kd58dt2g",
503
-
"disabled": true,
504
-
"disabledAt": "2025-10-19T15:00:00Z"
505
-
}
506
-
```
507
-
508
-
**Behavior:**
509
-
- Validates caller is community moderator
510
-
- Updates authorization record (sets `enabled=false`, `disabledAt`, `disabledBy`)
511
-
- Aggregator can no longer post until re-enabled
512
-
513
-
---
514
-
515
-
#### `social.coves.aggregator.updateConfig`
516
-
Update configuration for an enabled aggregator
517
-
518
-
**Input:**
519
-
```json
520
-
{
521
-
"aggregatorDid": "did:web:rss-bot.coves.social",
522
-
"config": {
523
-
"feeds": ["https://techcrunch.com/feed", "https://arstechnica.com/feed"],
524
-
"topics": ["technology", "gaming"]
525
-
}
526
-
}
527
-
```
528
-
529
-
**Output:**
530
-
```json
531
-
{
532
-
"uri": "at://did:plc:gaming123/social.coves.aggregator.authorization/3jui7kd58dt2g",
533
-
"cid": "bafyreif6...",
534
-
"config": {...}
535
-
}
536
-
```
537
-
538
-
---
539
-
540
-
#### `social.coves.aggregator.listForCommunity`
541
-
List all aggregators (enabled and disabled) for a community
542
-
543
-
**Input:**
544
-
```json
545
-
{
546
-
"community": "did:plc:gaming123",
547
-
"enabledOnly": false,
548
-
"limit": 50,
549
-
"cursor": "..."
550
-
}
551
-
```
552
-
553
-
**Output:**
554
-
```json
555
-
{
556
-
"aggregators": [
557
-
{
558
-
"aggregatorDid": "did:web:rss-bot.coves.social",
559
-
"displayName": "RSS News Aggregator",
560
-
"description": "...",
561
-
"aggregatorType": "social.coves.aggregator.types#rss",
562
-
"enabled": true,
563
-
"config": {...},
564
-
"createdAt": "2025-10-19T14:00:00Z"
565
-
}
566
-
],
567
-
"cursor": "..."
568
-
}
569
-
```
570
-
571
-
---
116
+
- **`social.coves.aggregator.enable`** - Create authorization record
117
+
- **`social.coves.aggregator.disable`** - Set enabled=false
118
+
- **`social.coves.aggregator.updateConfig`** - Update config
119
+
- **`social.coves.aggregator.listForCommunity`** - List aggregators for community
572
120
573
121
### For Aggregators
574
122
575
-
#### Existing: `social.coves.post.create`
576
-
**Modified Behavior:** Now handles aggregator authentication
577
-
578
-
**Authorization Flow:**
579
-
1. Extract DID from JWT
580
-
2. Check if DID is registered aggregator (query `aggregators` table)
581
-
3. If aggregator:
582
-
- Validate authorization record exists for this community
583
-
- Check `enabled=true`
584
-
- Apply aggregator rate limits (e.g., 10 posts/hour)
585
-
4. If regular user:
586
-
- Validate membership, bans, etc. (existing logic)
587
-
5. Create post with `author = actorDID`
588
-
589
-
**Rate Limits:**
590
-
- Regular users: 20 posts/hour per community
591
-
- Aggregators: 10 posts/hour per community (to prevent spam)
592
-
593
-
---
594
-
595
-
#### `social.coves.aggregator.getAuthorizations`
596
-
Get list of communities that have authorized this aggregator
597
-
598
-
**Input:**
599
-
```json
600
-
{
601
-
"aggregatorDid": "did:web:rss-bot.coves.social",
602
-
"enabledOnly": true,
603
-
"limit": 100,
604
-
"cursor": "..."
605
-
}
606
-
```
607
-
608
-
**Output:**
609
-
```json
610
-
{
611
-
"authorizations": [
612
-
{
613
-
"communityDid": "did:plc:gaming123",
614
-
"communityName": "Gaming News",
615
-
"enabled": true,
616
-
"config": {...},
617
-
"createdAt": "2025-10-19T14:00:00Z"
618
-
}
619
-
],
620
-
"cursor": "..."
621
-
}
622
-
```
623
-
624
-
**Use Case:** Aggregator queries this to know which communities to post to
625
-
626
-
---
123
+
- **`social.coves.post.create`** - Modified to handle aggregator auth
124
+
- **`social.coves.aggregator.getAuthorizations`** - Query authorized communities
627
125
628
126
### For Discovery
629
127
630
-
#### `social.coves.aggregator.list`
631
-
List all available aggregators
632
-
633
-
**Input:**
634
-
```json
635
-
{
636
-
"type": "social.coves.aggregator.types#rss",
637
-
"limit": 50,
638
-
"cursor": "..."
639
-
}
640
-
```
641
-
642
-
**Output:**
643
-
```json
644
-
{
645
-
"aggregators": [
646
-
{
647
-
"did": "did:web:rss-bot.coves.social",
648
-
"displayName": "RSS News Aggregator",
649
-
"description": "...",
650
-
"aggregatorType": "social.coves.aggregator.types#rss",
651
-
"avatar": "...",
652
-
"maintainer": "did:plc:coves-platform",
653
-
"sourceUrl": "https://github.com/coves-social/rss-aggregator"
654
-
}
655
-
],
656
-
"cursor": "..."
657
-
}
658
-
```
659
-
660
-
---
661
-
662
-
#### `social.coves.aggregator.get`
663
-
Get detailed information about a specific aggregator
664
-
665
-
**Input:**
666
-
```json
667
-
{
668
-
"aggregatorDid": "did:web:rss-bot.coves.social"
669
-
}
670
-
```
671
-
672
-
**Output:**
673
-
```json
674
-
{
675
-
"did": "did:web:rss-bot.coves.social",
676
-
"displayName": "RSS News Aggregator",
677
-
"description": "...",
678
-
"aggregatorType": "social.coves.aggregator.types#rss",
679
-
"configSchema": {...},
680
-
"sourceUrl": "...",
681
-
"maintainer": "...",
682
-
"stats": {
683
-
"communitiesUsing": 42,
684
-
"postsCreated": 1337,
685
-
"createdAt": "2025-10-19T12:00:00Z"
686
-
}
687
-
}
688
-
```
128
+
- **`social.coves.aggregator.getServices`** - Fetch aggregator details by DID(s)
689
129
690
130
---
691
131
692
132
## Database Schema
693
133
694
134
### `aggregators` Table
695
-
Indexed aggregator service declarations from Jetstream
696
-
697
-
```sql
698
-
CREATE TABLE aggregators (
699
-
did TEXT PRIMARY KEY,
700
-
display_name TEXT NOT NULL,
701
-
description TEXT,
702
-
aggregator_type TEXT NOT NULL,
703
-
config_schema JSONB,
704
-
avatar_url TEXT,
705
-
source_url TEXT,
706
-
maintainer_did TEXT,
707
-
708
-
-- Indexing metadata
709
-
record_uri TEXT NOT NULL,
710
-
record_cid TEXT NOT NULL,
711
-
indexed_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
712
-
713
-
-- Stats (cached)
714
-
communities_using INTEGER NOT NULL DEFAULT 0,
715
-
posts_created BIGINT NOT NULL DEFAULT 0,
135
+
Indexes aggregator service declarations from Jetstream.
716
136
717
-
CONSTRAINT aggregators_type_check CHECK (
718
-
aggregator_type IN (
719
-
'social.coves.aggregator.types#rss',
720
-
'social.coves.aggregator.types#blueskyMirror',
721
-
'social.coves.aggregator.types#sports',
722
-
'social.coves.aggregator.types#media',
723
-
'social.coves.aggregator.types#custom'
724
-
)
725
-
)
726
-
);
727
-
728
-
CREATE INDEX idx_aggregators_type ON aggregators(aggregator_type);
729
-
CREATE INDEX idx_aggregators_indexed_at ON aggregators(indexed_at DESC);
730
-
```
731
-
732
-
---
137
+
**Key Columns:**
138
+
- `did` (PK) - Aggregator DID
139
+
- `display_name`, `description` - Service metadata
140
+
- `config_schema` - JSON Schema for config validation
141
+
- `avatar_url`, `source_url`, `maintainer_did` - Metadata
142
+
- `record_uri`, `record_cid` - atProto record metadata
143
+
- `communities_using`, `posts_created` - Cached stats (updated by triggers)
733
144
734
145
### `aggregator_authorizations` Table
735
-
Indexed authorization records from communities
146
+
Indexes community authorization records from Jetstream.
736
147
737
-
```sql
738
-
CREATE TABLE aggregator_authorizations (
739
-
id BIGSERIAL PRIMARY KEY,
148
+
**Key Columns:**
149
+
- `aggregator_did`, `community_did` - Authorization pair (unique together)
150
+
- `enabled` - Active status
151
+
- `config` - Community-specific JSON config
152
+
- `created_by`, `disabled_by` - Audit trail
153
+
- `record_uri`, `record_cid` - atProto record metadata
740
154
741
-
-- Authorization identity
742
-
aggregator_did TEXT NOT NULL REFERENCES aggregators(did) ON DELETE CASCADE,
743
-
community_did TEXT NOT NULL,
155
+
**Critical Indexes:**
156
+
- `idx_aggregator_auth_lookup` - Fast (aggregator_did, community_did, enabled) lookups for post creation
744
157
745
-
-- Authorization state
746
-
enabled BOOLEAN NOT NULL DEFAULT true,
747
-
config JSONB,
158
+
### `aggregator_posts` Table
159
+
AppView-only tracking for rate limiting and stats (not from lexicon).
748
160
749
-
-- Audit trail
750
-
created_at TIMESTAMPTZ NOT NULL,
751
-
created_by TEXT NOT NULL, -- DID of moderator
752
-
disabled_at TIMESTAMPTZ,
753
-
disabled_by TEXT, -- DID of moderator
754
-
755
-
-- atProto record metadata
756
-
record_uri TEXT NOT NULL UNIQUE,
757
-
record_cid TEXT NOT NULL,
758
-
indexed_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
759
-
760
-
UNIQUE(aggregator_did, community_did)
761
-
);
762
-
763
-
CREATE INDEX idx_aggregator_auth_agg_did ON aggregator_authorizations(aggregator_did) WHERE enabled = true;
764
-
CREATE INDEX idx_aggregator_auth_comm_did ON aggregator_authorizations(community_did) WHERE enabled = true;
765
-
CREATE INDEX idx_aggregator_auth_enabled ON aggregator_authorizations(enabled);
766
-
```
161
+
**Key Columns:**
162
+
- `aggregator_did`, `community_did`, `post_uri`
163
+
- `created_at` - For rate limit calculations
767
164
768
165
---
769
166
770
-
### `aggregator_posts` Table
771
-
Track posts created by aggregators (for rate limiting and stats)
772
-
773
-
```sql
774
-
CREATE TABLE aggregator_posts (
775
-
id BIGSERIAL PRIMARY KEY,
167
+
## Security
776
168
777
-
aggregator_did TEXT NOT NULL REFERENCES aggregators(did) ON DELETE CASCADE,
778
-
community_did TEXT NOT NULL,
779
-
post_uri TEXT NOT NULL,
780
-
post_cid TEXT NOT NULL,
169
+
### Authentication
170
+
- DID-based authentication via JWT signatures
171
+
- No shared secrets or API keys
172
+
- Aggregators can only post to authorized communities
781
173
782
-
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
174
+
### Authorization Checks
175
+
- Server validates aggregator status (not client-provided)
176
+
- Checks `aggregator_authorizations` table on every post
177
+
- Config validated against aggregator's JSON schema
783
178
784
-
UNIQUE(post_uri)
785
-
);
179
+
### Rate Limiting
180
+
- Aggregators: 10 posts/hour per community
181
+
- Tracked via `aggregator_posts` table
182
+
- Prevents spam
786
183
787
-
CREATE INDEX idx_aggregator_posts_agg_did_created ON aggregator_posts(aggregator_did, created_at DESC);
788
-
CREATE INDEX idx_aggregator_posts_comm_did_created ON aggregator_posts(community_did, created_at DESC);
789
-
790
-
-- For rate limiting: count posts in last hour
791
-
CREATE INDEX idx_aggregator_posts_rate_limit ON aggregator_posts(aggregator_did, community_did, created_at DESC);
792
-
```
184
+
### Audit Trail
185
+
- `created_by` / `disabled_by` track moderator actions
186
+
- Full history preserved in authorization records
793
187
794
188
---
795
189
796
-
## Implementation Plan
190
+
## Implementation Phases
797
191
798
-
### Phase 1: Core Infrastructure (Coves AppView)
799
-
192
+
### ✅ Phase 1: Core Infrastructure (COMPLETE)
193
+
**Status:** ✅ COMPLETE - All components implemented and tested
800
194
**Goal:** Enable aggregator authentication and authorization
801
195
802
-
#### 1.1 Database Setup
803
-
- [ ] Create migration for `aggregators` table
804
-
- [ ] Create migration for `aggregator_authorizations` table
805
-
- [ ] Create migration for `aggregator_posts` table
806
-
807
-
#### 1.2 Lexicon Definitions
808
-
- [ ] Create `social.coves.aggregator.service.json`
809
-
- [ ] Create `social.coves.aggregator.authorization.json`
810
-
- [ ] Create `social.coves.aggregator.types.json`
811
-
- [ ] Generate Go types from lexicons
812
-
813
-
#### 1.3 Repository Layer
814
-
```go
815
-
// internal/core/aggregators/repository.go
816
-
817
-
type Repository interface {
818
-
// Aggregator management
819
-
CreateAggregator(ctx context.Context, agg *Aggregator) error
820
-
GetAggregator(ctx context.Context, did string) (*Aggregator, error)
821
-
ListAggregators(ctx context.Context, filter AggregatorFilter) ([]*Aggregator, error)
822
-
UpdateAggregatorStats(ctx context.Context, did string, stats Stats) error
823
-
824
-
// Authorization management
825
-
CreateAuthorization(ctx context.Context, auth *Authorization) error
826
-
GetAuthorization(ctx context.Context, aggDID, commDID string) (*Authorization, error)
827
-
ListAuthorizationsForAggregator(ctx context.Context, aggDID string, enabledOnly bool) ([]*Authorization, error)
828
-
ListAuthorizationsForCommunity(ctx context.Context, commDID string) ([]*Authorization, error)
829
-
UpdateAuthorization(ctx context.Context, auth *Authorization) error
830
-
IsAuthorized(ctx context.Context, aggDID, commDID string) (bool, error)
831
-
832
-
// Post tracking (for rate limiting)
833
-
RecordAggregatorPost(ctx context.Context, aggDID, commDID, postURI string) error
834
-
CountRecentPosts(ctx context.Context, aggDID, commDID string, since time.Time) (int, error)
835
-
}
836
-
```
837
-
838
-
#### 1.4 Service Layer
839
-
```go
840
-
// internal/core/aggregators/service.go
841
-
842
-
type Service interface {
843
-
// For communities (moderators)
844
-
EnableAggregator(ctx context.Context, commDID, aggDID string, config map[string]interface{}) (*Authorization, error)
845
-
DisableAggregator(ctx context.Context, commDID, aggDID string) error
846
-
UpdateAggregatorConfig(ctx context.Context, commDID, aggDID string, config map[string]interface{}) error
847
-
ListCommunityAggregators(ctx context.Context, commDID string, enabledOnly bool) ([]*AggregatorInfo, error)
848
-
849
-
// For aggregators
850
-
GetAuthorizedCommunities(ctx context.Context, aggDID string) ([]*CommunityAuth, error)
851
-
852
-
// For discovery
853
-
ListAggregators(ctx context.Context, filter AggregatorFilter) ([]*Aggregator, error)
854
-
GetAggregator(ctx context.Context, did string) (*AggregatorDetail, error)
855
-
856
-
// Internal: called by post creation handler
857
-
ValidateAggregatorPost(ctx context.Context, aggDID, commDID string) error
858
-
}
859
-
```
196
+
**Components:**
197
+
- ✅ Lexicon schemas (9 files)
198
+
- ✅ Database migrations (2 migrations: 3 tables, 2 triggers, indexes)
199
+
- ✅ Repository layer (CRUD operations, bulk queries, optimized indexes)
200
+
- ✅ Service layer (business logic, validation, rate limiting)
201
+
- ✅ Modified post creation handler (aggregator authentication & authorization)
202
+
- ✅ XRPC query handlers (getServices, getAuthorizations, listForCommunity)
203
+
- ✅ Jetstream consumer (indexes service & authorization records from firehose)
204
+
- ✅ Integration tests (10+ test suites, E2E validation)
205
+
- ✅ E2E test validation (verified records exist in both PDS and AppView)
860
206
861
-
#### 1.5 Modify Post Creation Handler
862
-
```go
863
-
// internal/api/handlers/post/create.go
207
+
**Milestone:** ✅ ACHIEVED - Aggregators can authenticate and post to authorized communities
864
208
865
-
func CreatePost(ctx context.Context, input *CreatePostInput) (*CreatePostOutput, error) {
866
-
actorDID := GetDIDFromAuth(ctx)
209
+
**Deferred to Phase 2:**
210
+
- Write-forward operations (enable, disable, updateConfig) - require PDS integration
211
+
- Moderator permission checks - require communities ownership validation
867
212
868
-
// Check if actor is an aggregator
869
-
if isAggregator, _ := aggregatorService.IsAggregator(ctx, actorDID); isAggregator {
870
-
// Validate aggregator authorization
871
-
if err := aggregatorService.ValidateAggregatorPost(ctx, actorDID, input.Community); err != nil {
872
-
return nil, err
873
-
}
213
+
---
874
214
875
-
// Apply aggregator rate limits
876
-
if err := rateLimitAggregator(ctx, actorDID, input.Community); err != nil {
877
-
return nil, ErrRateLimitExceeded
878
-
}
879
-
} else {
880
-
// Regular user validation (existing logic)
881
-
// ... membership checks, ban checks, etc.
882
-
}
215
+
### Phase 2: Aggregator SDK (Post-Alpha)
216
+
**Deferred** - Will build SDK after Phase 1 is validated in production.
883
217
884
-
// Create post (author will be actorDID - either user or aggregator)
885
-
post, err := postService.CreatePost(ctx, actorDID, input)
886
-
if err != nil {
887
-
return nil, err
888
-
}
218
+
Core functionality works without SDK - aggregators just need to:
219
+
1. Create atProto account (get DID)
220
+
2. Publish service declaration record
221
+
3. Sign JWTs with their DID keys
222
+
4. Call existing XRPC endpoints
889
223
890
-
// If aggregator, track the post
891
-
if isAggregator {
892
-
_ = aggregatorService.RecordPost(ctx, actorDID, input.Community, post.URI)
893
-
}
224
+
---
894
225
895
-
return post, nil
896
-
}
897
-
```
226
+
### Phase 3: Reference Implementation (Future)
227
+
**Deferred** - First aggregator will likely be built inline to validate the system.
898
228
899
-
#### 1.6 XRPC Handlers
900
-
- [ ] `social.coves.aggregator.enable` handler
901
-
- [ ] `social.coves.aggregator.disable` handler
902
-
- [ ] `social.coves.aggregator.updateConfig` handler
903
-
- [ ] `social.coves.aggregator.listForCommunity` handler
904
-
- [ ] `social.coves.aggregator.getAuthorizations` handler
905
-
- [ ] `social.coves.aggregator.list` handler
906
-
- [ ] `social.coves.aggregator.get` handler
229
+
Potential first aggregator: RSS news bot for select communities.
907
230
908
-
#### 1.7 Jetstream Consumer
909
-
```go
910
-
// internal/atproto/jetstream/aggregator_consumer.go
231
+
---
911
232
912
-
func (c *AggregatorConsumer) HandleEvent(ctx context.Context, evt *jetstream.Event) error {
913
-
switch evt.Collection {
914
-
case "social.coves.aggregator.service":
915
-
switch evt.Operation {
916
-
case "create", "update":
917
-
return c.indexAggregatorService(ctx, evt)
918
-
case "delete":
919
-
return c.deleteAggregator(ctx, evt.DID)
920
-
}
233
+
## Key Design Decisions
921
234
922
-
case "social.coves.aggregator.authorization":
923
-
switch evt.Operation {
924
-
case "create", "update":
925
-
return c.indexAuthorization(ctx, evt)
926
-
case "delete":
927
-
return c.deleteAuthorization(ctx, evt.URI)
928
-
}
929
-
}
930
-
return nil
931
-
}
932
-
```
235
+
### 2025-10-20: Remove `aggregatorType` Field
236
+
**Decision:** Removed `aggregatorType` enum from service declaration and database.
933
237
934
-
#### 1.8 Integration Tests
935
-
- [ ] Test aggregator service indexing from Jetstream
936
-
- [ ] Test authorization record indexing
937
-
- [ ] Test `social.coves.post.create` with aggregator auth
938
-
- [ ] Test authorization validation (enabled/disabled)
939
-
- [ ] Test rate limiting for aggregators
940
-
- [ ] Test config validation against schema
238
+
**Rationale:**
239
+
- Pre-production - can break things
240
+
- Over-engineering for alpha
241
+
- Description field is sufficient for discovery
242
+
- Avoids rigid categorization
243
+
- Can add tags later if needed
941
244
942
-
**Milestone:** Aggregators can authenticate and post to communities with authorization
245
+
**Impact:**
246
+
- Simplified lexicons
247
+
- Removed database constraint
248
+
- More flexible for third-party developers
943
249
944
250
---
945
251
946
-
### Phase 2: Aggregator SDK (Go)
252
+
### 2025-10-19: Reuse `social.coves.post.create` Endpoint
253
+
**Decision:** Aggregators use existing post creation endpoint.
947
254
948
-
**Goal:** Provide SDK for building aggregators easily
949
-
950
-
#### 2.1 SDK Core
951
-
```go
952
-
// github.com/coves-social/aggregator-sdk-go
953
-
954
-
package aggregator
255
+
**Rationale:**
256
+
- Post record already server-populates `author` from JWT
257
+
- Simpler: one code path for all post creation
258
+
- Follows atProto principle: actors are actors
259
+
- `federatedFrom` field handles external content attribution
955
260
956
-
type Aggregator interface {
957
-
// Identity
958
-
GetDID() string
959
-
GetDisplayName() string
960
-
GetDescription() string
961
-
GetType() string
962
-
GetConfigSchema() map[string]interface{}
963
-
964
-
// Lifecycle
965
-
Start(ctx context.Context) error
966
-
Stop() error
967
-
968
-
// Posting (provided by SDK)
969
-
CreatePost(ctx context.Context, communityDID string, post Post) error
970
-
GetAuthorizedCommunities(ctx context.Context) ([]*CommunityAuth, error)
971
-
}
261
+
**Implementation:**
262
+
- Add branching logic in post handler: if aggregator, check authorization; else check membership
263
+
- Apply different rate limits based on actor type
972
264
973
-
type BaseAggregator struct {
974
-
DID string
975
-
DisplayName string
976
-
Description string
977
-
Type string
978
-
PrivateKey crypto.PrivateKey
979
-
CovesAPIURL string
265
+
---
980
266
981
-
client *http.Client
982
-
}
267
+
### 2025-10-19: Config as JSON Schema
268
+
**Decision:** Aggregators declare `configSchema` in service record.
983
269
984
-
type Post struct {
985
-
Title string
986
-
Content string
987
-
Embed interface{}
988
-
FederatedFrom *FederatedSource
989
-
ContentLabels []string
990
-
}
270
+
**Rationale:**
271
+
- Communities need to know what config options are available
272
+
- JSON Schema is standard and well-supported
273
+
- Enables UI auto-generation (forms from schema)
274
+
- Validation at authorization creation time
275
+
- Flexible: each aggregator has different config needs
991
276
992
-
type FederatedSource struct {
993
-
Platform string // "rss", "bluesky", etc.
994
-
URI string
995
-
ID string
996
-
OriginalCreatedAt time.Time
997
-
}
277
+
---
998
278
999
-
// Helper methods provided by SDK
1000
-
func (a *BaseAggregator) CreatePost(ctx context.Context, communityDID string, post Post) error {
1001
-
// 1. Sign JWT with aggregator's private key
1002
-
token := a.signJWT()
279
+
## Use Cases
1003
280
1004
-
// 2. Call social.coves.post.create via XRPC
1005
-
resp, err := a.client.Post(
1006
-
a.CovesAPIURL + "/xrpc/social.coves.post.create",
1007
-
&CreatePostInput{
1008
-
Community: communityDID,
1009
-
Title: post.Title,
1010
-
Content: post.Content,
1011
-
Embed: post.Embed,
1012
-
FederatedFrom: post.FederatedFrom,
1013
-
ContentLabels: post.ContentLabels,
1014
-
},
1015
-
&CreatePostOutput{},
1016
-
WithAuth(token),
1017
-
)
281
+
### RSS News Aggregator
282
+
Watches configured RSS feeds, uses LLM for deduplication, posts news articles to community.
1018
283
1019
-
return err
284
+
**Community Config Example:**
285
+
```json
286
+
{
287
+
"feeds": ["https://techcrunch.com/feed"],
288
+
"topics": ["technology"],
289
+
"dedupeWindow": "6h"
1020
290
}
291
+
```
1021
292
1022
-
func (a *BaseAggregator) GetAuthorizedCommunities(ctx context.Context) ([]*CommunityAuth, error) {
1023
-
// Call social.coves.aggregator.getAuthorizations
1024
-
token := a.signJWT()
293
+
---
1025
294
1026
-
resp, err := a.client.Get(
1027
-
a.CovesAPIURL + "/xrpc/social.coves.aggregator.getAuthorizations",
1028
-
map[string]string{"aggregatorDid": a.DID, "enabledOnly": "true"},
1029
-
&GetAuthorizationsOutput{},
1030
-
WithAuth(token),
1031
-
)
295
+
### Bluesky Post Mirror
296
+
Monitors specific users/hashtags on Bluesky, creates posts in community with original author metadata.
1032
297
1033
-
return resp.Authorizations, err
298
+
**Community Config Example:**
299
+
```json
300
+
{
301
+
"mirrorUsers": ["alice.bsky.social"],
302
+
"hashtags": ["covesalpha"],
303
+
"minLikes": 10
1034
304
}
1035
305
```
1036
306
1037
-
#### 2.2 SDK Documentation
1038
-
- [ ] README with quickstart guide
1039
-
- [ ] Example aggregators (RSS, Bluesky mirror)
1040
-
- [ ] API reference documentation
1041
-
- [ ] Configuration schema guide
1042
-
1043
-
**Milestone:** Third parties can build aggregators using SDK
1044
-
1045
307
---
1046
308
1047
-
### Phase 3: Reference Aggregator (RSS)
1048
-
1049
-
**Goal:** Build working RSS aggregator as reference implementation
1050
-
1051
-
#### 3.1 RSS Aggregator Implementation
1052
-
```go
1053
-
// github.com/coves-social/rss-aggregator
1054
-
1055
-
package main
1056
-
1057
-
import "github.com/coves-social/aggregator-sdk-go"
1058
-
1059
-
type RSSAggregator struct {
1060
-
*aggregator.BaseAggregator
1061
-
1062
-
// RSS-specific config
1063
-
pollInterval time.Duration
1064
-
llmClient *openai.Client
1065
-
}
1066
-
1067
-
func (r *RSSAggregator) Start(ctx context.Context) error {
1068
-
// 1. Get authorized communities
1069
-
communities, err := r.GetAuthorizedCommunities(ctx)
1070
-
if err != nil {
1071
-
return err
1072
-
}
1073
-
1074
-
// 2. Start polling loop
1075
-
ticker := time.NewTicker(r.pollInterval)
1076
-
defer ticker.Stop()
1077
-
1078
-
for {
1079
-
select {
1080
-
case <-ticker.C:
1081
-
r.pollFeeds(ctx, communities)
1082
-
case <-ctx.Done():
1083
-
return nil
1084
-
}
1085
-
}
1086
-
}
1087
-
1088
-
func (r *RSSAggregator) pollFeeds(ctx context.Context, communities []*CommunityAuth) {
1089
-
for _, comm := range communities {
1090
-
// Get RSS feeds from community config
1091
-
feeds := comm.Config["feeds"].([]string)
1092
-
1093
-
for _, feedURL := range feeds {
1094
-
items, err := r.fetchFeed(feedURL)
1095
-
if err != nil {
1096
-
continue
1097
-
}
1098
-
1099
-
// Process new items
1100
-
for _, item := range items {
1101
-
// Check if already posted
1102
-
if r.alreadyPosted(item.GUID) {
1103
-
continue
1104
-
}
1105
-
1106
-
// LLM deduplication logic
1107
-
duplicate := r.findDuplicate(item, comm.CommunityDID)
1108
-
if duplicate != nil {
1109
-
r.addToMegathread(duplicate, item)
1110
-
continue
1111
-
}
1112
-
1113
-
// Create new post
1114
-
post := aggregator.Post{
1115
-
Title: item.Title,
1116
-
Content: r.summarize(item),
1117
-
FederatedFrom: &aggregator.FederatedSource{
1118
-
Platform: "rss",
1119
-
URI: item.Link,
1120
-
OriginalCreatedAt: item.PublishedAt,
1121
-
},
1122
-
}
1123
-
1124
-
err = r.CreatePost(ctx, comm.CommunityDID, post)
1125
-
if err != nil {
1126
-
log.Printf("Failed to create post: %v", err)
1127
-
continue
1128
-
}
1129
-
1130
-
r.markPosted(item.GUID)
1131
-
}
1132
-
}
1133
-
}
1134
-
}
1135
-
1136
-
func (r *RSSAggregator) summarize(item *RSSItem) string {
1137
-
// Use LLM to create unbiased summary
1138
-
prompt := fmt.Sprintf("Summarize this news article in 2-3 sentences: %s", item.Description)
1139
-
summary, _ := r.llmClient.Complete(prompt)
1140
-
return summary
1141
-
}
309
+
### Sports Results
310
+
Monitors sports APIs, creates post-game threads with scores and stats.
1142
311
1143
-
func (r *RSSAggregator) findDuplicate(item *RSSItem, communityDID string) *Post {
1144
-
// Use LLM to detect semantic duplicates
1145
-
// Query recent posts in community
1146
-
// Compare with embeddings/similarity
1147
-
return nil // or duplicate post
312
+
**Community Config Example:**
313
+
```json
314
+
{
315
+
"league": "NBA",
316
+
"teams": ["Lakers", "Warriors"],
317
+
"includeStats": true
1148
318
}
1149
319
```
1150
320
1151
-
#### 3.2 Deployment
1152
-
- [ ] Dockerfile for RSS aggregator
1153
-
- [ ] Kubernetes manifests (for Coves-hosted instance)
1154
-
- [ ] Environment configuration guide
1155
-
- [ ] Monitoring and logging setup
1156
-
1157
-
#### 3.3 Testing
1158
-
- [ ] Unit tests for feed parsing
1159
-
- [ ] Integration tests with mock Coves API
1160
-
- [ ] E2E test with real Coves instance
1161
-
- [ ] LLM deduplication accuracy tests
1162
-
1163
-
**Milestone:** RSS aggregator running in production for select communities
1164
-
1165
-
---
1166
-
1167
-
### Phase 4: Additional Aggregators
1168
-
1169
-
#### 4.1 Bluesky Mirror Aggregator
1170
-
- [ ] Monitor Jetstream for specific users/hashtags
1171
-
- [ ] Preserve `originalAuthor` metadata
1172
-
- [ ] Link back to original Bluesky post
1173
-
- [ ] Rate limiting (don't flood community)
1174
-
1175
-
#### 4.2 Sports Aggregator
1176
-
- [ ] Integrate with ESPN/TheSportsDB APIs
1177
-
- [ ] Monitor game completions
1178
-
- [ ] Create post-game threads with stats
1179
-
- [ ] Auto-pin major games
1180
-
1181
-
#### 4.3 Media (TV/Movie) Aggregator
1182
-
- [ ] Integrate with TMDB API
1183
-
- [ ] Track show release schedules
1184
-
- [ ] Create episode discussion threads
1185
-
- [ ] Spoiler protection tags
1186
-
1187
-
**Milestone:** Multiple official aggregators available for communities
1188
-
1189
-
---
1190
-
1191
-
## Security Considerations
1192
-
1193
-
### Authentication
1194
-
✅ **DID-based Authentication**
1195
-
- Aggregators sign JWTs with their private keys
1196
-
- Server validates JWT signature against DID document
1197
-
- No shared secrets or API keys
1198
-
1199
-
✅ **Scoped Authorization**
1200
-
- Authorization records are per-community
1201
-
- Aggregator can only post to authorized communities
1202
-
- Communities can revoke at any time
1203
-
1204
-
### Rate Limiting
1205
-
✅ **Per-Aggregator Limits**
1206
-
- 10 posts/hour per community (configurable)
1207
-
- Prevents aggregator spam
1208
-
- Separate from user rate limits
1209
-
1210
-
✅ **Global Limits**
1211
-
- Total posts across all communities: 100/hour
1212
-
- Prevents runaway aggregators
1213
-
1214
-
### Content Validation
1215
-
✅ **Community Rules**
1216
-
- Aggregator posts validated against community content rules
1217
-
- No special exemptions (same rules as users)
1218
-
- Community can ban specific content patterns
1219
-
1220
-
✅ **Config Validation**
1221
-
- Authorization config validated against aggregator's configSchema
1222
-
- Prevents injection attacks via config
1223
-
- JSON schema validation
1224
-
1225
-
### Monitoring & Auditing
1226
-
✅ **Audit Trail**
1227
-
- All aggregator posts logged
1228
-
- `created_by` tracks which moderator authorized
1229
-
- `disabled_by` tracks who revoked access
1230
-
- Full history preserved
1231
-
1232
-
✅ **Abuse Detection**
1233
-
- Monitor for spam patterns
1234
-
- Alert if aggregator posts rejected repeatedly
1235
-
- Auto-disable after threshold violations
1236
-
1237
-
### Transparency
1238
-
✅ **Open Source**
1239
-
- Official aggregators open source
1240
-
- Source URL in service declaration
1241
-
- Community can audit behavior
1242
-
1243
-
✅ **Attribution**
1244
-
- Posts clearly show aggregator authorship
1245
-
- UI shows "🤖 Posted by [Aggregator Name]"
1246
-
- No attempt to impersonate users
1247
-
1248
-
---
1249
-
1250
-
## UI/UX Considerations
1251
-
1252
-
### Community Settings
1253
-
**Aggregator Management Page:**
1254
-
- List of available aggregators (with descriptions, types)
1255
-
- "Enable" button opens config modal
1256
-
- Config form generated from aggregator's configSchema
1257
-
- Toggle to enable/disable without deleting config
1258
-
- Stats: posts created, last active
1259
-
1260
-
**Post Display:**
1261
-
- Posts from aggregators have bot badge: "🤖"
1262
-
- Shows aggregator name (e.g., "Posted by RSS News Bot")
1263
-
- `federatedFrom` shows original source
1264
-
- Link to original content (RSS article, Bluesky post, etc.)
1265
-
1266
-
### User Preferences
1267
-
- Option to hide all aggregator posts
1268
-
- Option to hide specific aggregators
1269
-
- Filter posts by "user-created only" or "include bots"
1270
-
1271
321
---
1272
322
1273
323
## Success Metrics
1274
324
1275
-
### Pre-Launch Checklist
1276
-
- [ ] Lexicons defined and validated
1277
-
- [ ] Database migrations tested
1278
-
- [ ] Jetstream consumer indexes aggregator records
1279
-
- [ ] Post creation handler validates aggregator auth
1280
-
- [ ] Rate limiting prevents spam
1281
-
- [ ] SDK published and documented
1282
-
- [ ] Reference RSS aggregator working
1283
-
- [ ] E2E tests passing
1284
-
- [ ] Security audit completed
1285
-
1286
325
### Alpha Goals
1287
-
- 3+ official aggregators (RSS, Bluesky mirror, sports)
1288
-
- 10+ communities using aggregators
1289
-
- < 0.1% spam posts (false positives)
1290
-
- Aggregator posts appear in feed within 1 minute
326
+
- ✅ Lexicons validated
327
+
- ✅ Database migrations tested
328
+
- ⏳ Jetstream consumer indexes records
329
+
- ⏳ Post creation validates aggregator auth
330
+
- ⏳ Rate limiting prevents spam
331
+
- ⏳ Integration tests passing
1291
332
1292
-
### Beta Goals
1293
-
- Third-party aggregators launched
1294
-
- 50+ communities using aggregators
1295
-
- Developer documentation complete
1296
-
- Marketplace/directory for discovery
333
+
### Beta Goals (Future)
334
+
- First aggregator deployed in production
335
+
- 3+ communities using aggregators
336
+
- < 0.1% spam posts
337
+
- Third-party developer documentation
1297
338
1298
339
---
1299
340
1300
-
## Out of Scope (Future Versions)
341
+
## Out of Scope (Future)
1301
342
1302
-
### Aggregator Marketplace
1303
-
- [ ] Community ratings/reviews for aggregators
1304
-
- [ ] Featured aggregators
1305
-
- [ ] Paid aggregators (premium features)
1306
-
- [ ] Aggregator analytics dashboard
1307
-
1308
-
### Advanced Features
1309
-
- [ ] Scheduled posts (post at specific time)
1310
-
- [ ] Content moderation integration (auto-label NSFW)
1311
-
- [ ] Multi-community posting (single post to multiple communities)
1312
-
- [ ] Interactive aggregators (respond to comments)
1313
-
- [ ] Aggregator-to-aggregator communication (chains)
1314
-
1315
-
### Federation
1316
-
- [ ] Cross-instance aggregator discovery
1317
-
- [ ] Aggregator migration (change hosting provider)
1318
-
- [ ] Federated aggregator authorization (trust other instances' aggregators)
1319
-
1320
-
---
1321
-
1322
-
## Technical Decisions Log
1323
-
1324
-
### 2025-10-19: Reuse `social.coves.post.create` Endpoint
1325
-
1326
-
**Decision:** Aggregators use existing post creation endpoint, not a separate `social.coves.aggregator.post.create`
1327
-
1328
-
**Rationale:**
1329
-
- Post record already server-populates `author` field from JWT
1330
-
- Aggregators authenticate as themselves → `author = aggregator DID`
1331
-
- Simpler: one code path for all post creation
1332
-
- Follows atProto principle: actors are actors (users, bots, aggregators)
1333
-
- `federatedFrom` field already handles external content attribution
1334
-
1335
-
**Implementation:**
1336
-
- Add authorization check to `social.coves.post.create` handler
1337
-
- Check if authenticated DID is aggregator
1338
-
- Validate authorization record exists and enabled
1339
-
- Apply aggregator-specific rate limits
1340
-
- Otherwise same logic as user posts
1341
-
1342
-
**Trade-offs Accepted:**
1343
-
- Post creation handler has branching logic (user vs aggregator)
1344
-
- But: keeps lexicon simple, reuses existing validation
1345
-
1346
-
---
1347
-
1348
-
### 2025-10-19: Hybrid Hosting Model
1349
-
1350
-
**Decision:** Support both Coves-hosted and third-party aggregators
1351
-
1352
-
**Rationale:**
1353
-
- Coves can provide high-quality official aggregators (RSS, sports, media)
1354
-
- Third parties can build specialized aggregators (niche communities)
1355
-
- SDK makes it easy to build custom aggregators
1356
-
- Follows feed generator model (anyone can run one)
1357
-
- Decentralization-friendly
1358
-
1359
-
**Requirements:**
1360
-
- SDK must be well-documented and maintained
1361
-
- Authorization system must be DID-agnostic (works for any DID)
1362
-
- Discovery system shows all aggregators (official + third-party)
1363
-
1364
-
---
1365
-
1366
-
### 2025-10-19: Config as JSON Schema
1367
-
1368
-
**Decision:** Aggregators declare configSchema in their service record
1369
-
1370
-
**Rationale:**
1371
-
- Communities need to know what config options are available
1372
-
- JSON Schema is standard, well-supported
1373
-
- Enables UI auto-generation (forms from schema)
1374
-
- Validation at authorization creation time
1375
-
- Flexible: each aggregator can have different config structure
1376
-
1377
-
**Example:**
1378
-
```json
1379
-
{
1380
-
"configSchema": {
1381
-
"type": "object",
1382
-
"properties": {
1383
-
"feeds": {
1384
-
"type": "array",
1385
-
"items": { "type": "string", "format": "uri" },
1386
-
"description": "RSS feed URLs to monitor"
1387
-
},
1388
-
"topics": {
1389
-
"type": "array",
1390
-
"items": { "type": "string" },
1391
-
"description": "Topics to filter posts by"
1392
-
}
1393
-
},
1394
-
"required": ["feeds"]
1395
-
}
1396
-
}
1397
-
```
1398
-
1399
-
**Trade-offs Accepted:**
1400
-
- More complex than simple key-value config
1401
-
- But: better UX (self-documenting), prevents errors
343
+
- Aggregator marketplace with ratings/reviews
344
+
- UI for aggregator management (alpha uses XRPC only)
345
+
- Scheduled posts
346
+
- Interactive aggregators (respond to comments)
347
+
- Cross-instance aggregator discovery
348
+
- SDK (deferred until post-alpha)
349
+
- LLM features (deferred)
1402
350
1403
351
---
1404
352
1405
353
## References
1406
354
1407
355
- atProto Lexicon Spec: https://atproto.com/specs/lexicon
1408
-
- Feed Generator Starter Kit: https://github.com/bluesky-social/feed-generator
1409
-
- Labeler Implementation: https://github.com/bluesky-social/atproto/tree/main/packages/ozone
1410
-
- JSON Schema Spec: https://json-schema.org/
1411
-
- Coves Communities PRD: [PRD_COMMUNITIES.md](PRD_COMMUNITIES.md)
1412
-
- Coves Posts Implementation: [IMPLEMENTATION_POST_CREATION.md](IMPLEMENTATION_POST_CREATION.md)
356
+
- Feed Generator Pattern: https://github.com/bluesky-social/feed-generator
357
+
- Labeler Pattern: https://github.com/bluesky-social/atproto/tree/main/packages/ozone
358
+
- JSON Schema: https://json-schema.org/