dollspace.gay
A third party ATProto appview
1# Python Firehose Consumer - Documentation Index
2
3**Quick Navigation** - Choose your path:
4
5## 🚀 I Want to Get Started Now
6→ **[QUICKSTART.md](./QUICKSTART.md)** - 5-minute setup guide
7
8## 📖 I Want to Understand This
9→ **[SUMMARY.md](./SUMMARY.md)** - High-level overview and rationale
10→ **[README.md](./README.md)** - Detailed documentation
11
12## 🔄 I Want to Migrate from TypeScript
13→ **[../PYTHON_FIREHOSE_MIGRATION.md](../PYTHON_FIREHOSE_MIGRATION.md)** - Step-by-step migration guide
14→ **[../ARCHITECTURE_COMPARISON.md](../ARCHITECTURE_COMPARISON.md)** - Before/after comparison
15
16## 💻 I Want to Read the Code
17→ **[firehose_consumer.py](./firehose_consumer.py)** - Main consumer (well-documented, ~400 lines)
18→ **[requirements.txt](./requirements.txt)** - Python dependencies
19→ **[Dockerfile](./Dockerfile)** - Container image
20
21---
22
23## What Is This?
24
25A **high-performance Python service** that replaces TypeScript firehose connections to eliminate worker overhead and memory limitations.
26
27### The Problem
28- TypeScript needs 32 workers = 64GB RAM
29- V8 heap limits, complex coordination
30- Database connection pool exhaustion
31
32### The Solution
33- Python handles firehose → Redis (1 process, 2GB RAM)
34- TypeScript handles processing (4 workers, 8GB RAM)
35- **Total: 85% memory reduction, same functionality**
36
37---
38
39## File Guide
40
41| File | Purpose | Read This If... |
42|------|---------|-----------------|
43| **QUICKSTART.md** | 5-minute getting started | You want to deploy now |
44| **SUMMARY.md** | Executive summary | You want the big picture |
45| **README.md** | Complete documentation | You want all the details |
46| **firehose_consumer.py** | Main Python script | You want to understand the code |
47| **Dockerfile** | Container image | You want to customize deployment |
48| **requirements.txt** | Python dependencies | You want to know what's installed |
49| **INDEX.md** | This file | You want to navigate the docs |
50
51---
52
53## Quick Commands
54
55```bash
56# Deploy
57docker-compose up -d python-firehose
58
59# Logs
60docker-compose logs -f python-firehose
61
62# Status
63docker-compose ps python-firehose
64
65# Verify events
66docker-compose exec redis redis-cli XLEN firehose:events
67
68# Memory usage
69docker stats python-firehose
70```
71
72---
73
74## Key Concepts
75
76### 1. Hybrid Architecture
77- **Python**: Firehose ingestion only (500 lines)
78- **TypeScript**: All business logic (10,000+ lines, unchanged)
79
80### 2. Redis as Bridge
81- Python pushes to `firehose:events` stream
82- TypeScript workers consume from same stream
83- Same format, no changes needed
84
85### 3. Drop-in Replacement
86- TypeScript workers don't know Python exists
87- Events arrive in Redis same as before
88- Zero business logic changes required
89
90---
91
92## Documentation Tree
93
94```
95python-firehose/
96├── INDEX.md ← You are here
97├── QUICKSTART.md ← Start here for quick deploy
98├── SUMMARY.md ← Overview and rationale
99├── README.md ← Complete documentation
100├── firehose_consumer.py ← Main code
101├── Dockerfile ← Container config
102├── requirements.txt ← Dependencies
103└── .gitignore ← Git ignore rules
104
105../
106├── PYTHON_FIREHOSE_MIGRATION.md ← Migration guide
107└── ARCHITECTURE_COMPARISON.md ← Before/after comparison
108```
109
110---
111
112## Next Steps
113
1141. **New user?** → Read [QUICKSTART.md](./QUICKSTART.md)
1152. **Want context?** → Read [SUMMARY.md](./SUMMARY.md)
1163. **Migrating?** → Read [../PYTHON_FIREHOSE_MIGRATION.md](../PYTHON_FIREHOSE_MIGRATION.md)
1174. **Need details?** → Read [README.md](./README.md)
118
119---
120
121**Remember**: This is just the ingestion layer. Your TypeScript business logic stays unchanged!