Python Firehose Consumer - Quick Start#

Get up and running in 5 minutes!

TL;DR#

# 1. Build and start
docker-compose up -d python-firehose

# 2. Watch it work
docker-compose logs -f python-firehose

# 3. Verify events flowing
docker-compose exec redis redis-cli XLEN firehose:events

# 4. That's it! Your TypeScript workers consume from Redis (no changes needed)

What This Does#

✅ Connects to AT Protocol firehose (Bluesky network)
✅ Receives events (posts, likes, follows, etc.)
✅ Pushes to Redis streams (firehose:events)
✅ Your TypeScript workers consume from Redis (unchanged)
✅ Saves cursor for restart recovery

Why Use This?#

Problem: TypeScript firehose needs 32 workers using ~64GB RAM due to Node.js memory limits.

Solution: Python handles firehose ingestion in 1 process using ~2GB RAM. TypeScript workers stay the same.

Result: 85% memory reduction, simpler deployment, same functionality.

Configuration#

Works out-of-the-box with defaults. To customize:

# docker-compose.yml
python-firehose:
  environment:
    - RELAY_URL=wss://bsky.network              # Firehose URL
    - REDIS_URL=redis://redis:6379              # Redis connection
    - REDIS_STREAM_KEY=firehose:events          # Stream name (match TypeScript)
    - REDIS_CURSOR_KEY=firehose:python_cursor   # Cursor storage key
    - REDIS_MAX_STREAM_LEN=500000               # Max events in stream

Monitoring#

Check if it's running#

docker-compose ps python-firehose
# Should show: Up (healthy)

View logs#

docker-compose logs -f python-firehose

# You should see:
# [INFO] Connected to firehose successfully
# [INFO] Processed 1,000 events (~2,500 events/sec, cursor: 123456789)

Check Redis stream#

# Number of events in stream
docker-compose exec redis redis-cli XLEN firehose:events

# Current cursor position
docker-compose exec redis redis-cli GET firehose:python_cursor

# Watch events flowing (real-time)
docker-compose exec redis redis-cli MONITOR | grep firehose:events

Performance metrics#

# Memory usage
docker stats python-firehose

# Should show: ~1-2GB RAM (vs 64GB for 32 TypeScript workers)

Troubleshooting#

"Connection refused" error#

Redis isn't ready yet. Wait for Redis to be healthy:

docker-compose up -d redis
docker-compose logs -f redis
# Wait for: "Ready to accept connections"

"READONLY" error#

Connected to Redis replica instead of master. Check REDIS_URL:

# Should point to master, not replica
REDIS_URL=redis://redis:6379  # ✅ Correct
REDIS_URL=redis://replica:6379  # ❌ Wrong

No events showing up#

Check firehose connection:

# Look for connection success
docker-compose logs python-firehose | grep "Connected to firehose"

# Check for errors
docker-compose logs python-firehose | grep ERROR

High memory usage#

Python should use much less memory than TypeScript. If not:

# Check for memory leaks
docker stats python-firehose

# Restart if needed
docker-compose restart python-firehose

Migrating from TypeScript Firehose#

Step 1: Start Python consumer#

docker-compose up -d python-firehose

Step 2: Verify events flowing#

# Should see increasing numbers
docker-compose exec redis redis-cli XLEN firehose:events
# Wait 10 seconds
docker-compose exec redis redis-cli XLEN firehose:events

Step 3: Update TypeScript (optional)#

Your TypeScript workers already consume from Redis, so they'll automatically pick up events from Python.

If you want to disable TypeScript's firehose connection:

// server/index.ts
// await firehoseClient.connect();  // Disable this line

Step 4: Reduce workers (optional)#

Since Python handles ingestion efficiently, reduce TypeScript workers:

# docker-compose.yml
app:
  deploy:
    replicas: 4  # Down from 32

Common Commands#

# Start
docker-compose up -d python-firehose

# Stop
docker-compose stop python-firehose

# Restart
docker-compose restart python-firehose

# View logs
docker-compose logs -f python-firehose

# Check health
docker-compose ps python-firehose

# Remove (keeps Redis data)
docker-compose down python-firehose

What's Next?#

Read README.md for detailed documentation
See PYTHON_FIREHOSE_MIGRATION.md for migration guide
Check firehose_consumer.py to understand the code

Need Help?#

Check logs: docker-compose logs -f python-firehose
Verify Redis: docker-compose exec redis redis-cli ping
Test connection: curl -v https://bsky.network (should connect)

Remember: This is just the firehose ingestion layer. All your TypeScript business logic stays the same!