Spark Feed Generator#

A template for creating custom feed generators for sprk.so.

Architecture#

The feed generator has three main components:

1. Ingester (ingester/) - Consumes the Spark firehose in real-time and indexes records to MongoDB. By default, it indexes so.sprk.feed.post records. Handlers for follow, like, and repost are included but disabled - enable them by uncommenting in ingester/index.ts.

2. Algorithms (algos/) - Define how posts are selected and sorted for your feed. Each algorithm exports:

   • handler - Query function that returns posts from the database
   • needsAuth - Whether the feed requires user authentication
   • publisherDid - The DID of the feed publisher
   • rkey - Unique identifier for this algorithm

3. API Server - Exposes XRPC endpoints that Spark clients call to fetch feed content (so.sprk.feed.getFeedSkeleton, so.sprk.feed.describeFeedGenerator). You won't need to modify these when creating a feed.

Creating Custom Feeds#

For topic/community feeds: Filter posts at the ingester level. Modify the handler in ingester/handlers/post.ts to only index posts matching your criteria (hashtags, keywords, specific authors, etc.).

For personalized/sorted feeds: Create a new algorithm in algos/. Copy simple-desc.ts as a starting point, then modify the query logic. Register your algorithm in algos/index.ts.

Example algorithm structure:

// algos/my-feed.ts
export const info = {
  handler: async (ctx, params, did) => {
    // Query posts from ctx.db.models.Post
    // Return { cursor, feed: [{ post: "at://..." }] }
  },
  needsAuth: false, // Set true if feed needs user's DID
  publisherDid: "did:plc:your-did",
  rkey: "my-feed", // at://your-did/so.sprk.feed.generator/my-feed
} as Algorithm;

Running#

Prerequisites#

Create a .env file in the project root:

# Required
FEEDGEN_DOMAIN=feeds.example.com    # Your feed generator's domain
SPRK_DB_NAME=feed-gen               # MongoDB database name
SPRK_DB_URI=mongodb://localhost:27017
SPRK_DB_USER=username
SPRK_DB_PASS=password

# Optional
SPRK_HOST=0.0.0.0                   # Server bind address
SPRK_PORT=3000                      # Server port
NODE_ENV=production

Docker Compose (Recommended)#

Starts both the feed generator and a MongoDB instance:

# Start services
docker-compose up -d

# View logs
docker-compose logs -f feed-gen

# Stop services
docker-compose down

Manual Setup#

Requires Deno and a MongoDB instance.

# Install dependencies
deno install

# Start server
deno task start

The server will be available at http://localhost:3000.

Endpoints#