+17 -2

.env.example

··· 17 17 # Osprey Integration (federated labeling) 18 18 OSPREY_ENABLED=false 19 19 20 20 - # Other settings 21 21 - BACKFILL_DAYS=2 20 20 + # Backfill Configuration 21 21 + # Set BACKFILL_DAYS to automatically run historical backfill when the system starts 22 22 + # 0 = disabled (no backfill) 23 23 + # -1 = total backfill (entire available history) 24 24 + # >0 = backfill X days of history (e.g., 7 for last 7 days) 25 25 + BACKFILL_DAYS=0 26 26 + 27 27 + # Backfill Resource Throttling (optional - defaults shown below) 28 28 + # Uncomment and adjust based on your server's capacity 29 29 + # BACKFILL_BATCH_SIZE=5 # Events to process before delaying 30 30 + # BACKFILL_BATCH_DELAY_MS=2000 # Milliseconds to wait between batches 31 31 + # BACKFILL_MAX_CONCURRENT=2 # Maximum concurrent processing operations 32 32 + # BACKFILL_MAX_MEMORY_MB=512 # Pause if memory exceeds this limit 33 33 + # BACKFILL_USE_IDLE=true # Use idle processing time 34 34 + # BACKFILL_DB_POOL_SIZE=2 # Database connection pool size for backfill 35 35 + 36 36 + # Data Retention 22 37 DATA_RETENTION_DAYS=0

+208

QUICKSTART-BACKFILL.md

··· 1 1 + # Quick Start: Automatic Python Backfill 2 2 + 3 3 + This guide shows you how to enable automatic historical data backfill for your AT Protocol AppView. 4 4 + 5 5 + ## What is Backfill? 6 6 + 7 7 + Backfill retrieves historical posts, likes, follows, and other events from the AT Protocol network and stores them in your database. This is useful when: 8 8 + - Setting up a new AppView instance 9 9 + - You want to populate your database with historical data 10 10 + - Users expect to see past posts in their feeds 11 11 + 12 12 + ## How to Enable Backfill 13 13 + 14 14 + The Python backfill service runs automatically when you set the `BACKFILL_DAYS` environment variable. 15 15 + 16 16 + ### Option 1: Using Environment Variables (Recommended) 17 17 + 18 18 + ```bash 19 19 + # Set the backfill duration (pick one): 20 20 + export BACKFILL_DAYS=7 # Backfill last 7 days 21 21 + export BACKFILL_DAYS=30 # Backfill last 30 days 22 22 + export BACKFILL_DAYS=-1 # Backfill ALL available history 23 23 + 24 24 + # Optional: Configure backfill performance (defaults are conservative) 25 25 + export BACKFILL_BATCH_SIZE=5 # Events per batch 26 26 + export BACKFILL_BATCH_DELAY_MS=2000 # Delay between batches (ms) 27 27 + export BACKFILL_MAX_MEMORY_MB=512 # Memory limit 28 28 + 29 29 + # Start your services 30 30 + docker-compose up -d 31 31 + ``` 32 32 + 33 33 + ### Option 2: Using .env File 34 34 + 35 35 + 1. Copy `.env.example` to `.env` if you haven't already: 36 36 + ```bash 37 37 + cp .env.example .env 38 38 + ``` 39 39 + 40 40 + 2. Edit `.env` and set `BACKFILL_DAYS`: 41 41 + ```bash 42 42 + # In .env file: 43 43 + BACKFILL_DAYS=7 44 44 + ``` 45 45 + 46 46 + 3. Start your services: 47 47 + ```bash 48 48 + docker-compose up -d 49 49 + ``` 50 50 + 51 51 + ## Backfill Configuration Options 52 52 + 53 53 + | Variable | Default | Description | 54 54 + |----------|---------|-------------| 55 55 + | `BACKFILL_DAYS` | `0` | `0`=disabled, `-1`=all history, `>0`=specific days | 56 56 + | `BACKFILL_BATCH_SIZE` | `5` | Events to process before pausing | 57 57 + | `BACKFILL_BATCH_DELAY_MS` | `2000` | Milliseconds to wait between batches | 58 58 + | `BACKFILL_MAX_CONCURRENT` | `2` | Max concurrent processing operations | 59 59 + | `BACKFILL_MAX_MEMORY_MB` | `512` | Pause if memory exceeds this limit | 60 60 + | `BACKFILL_USE_IDLE` | `true` | Use idle CPU time for processing | 61 61 + | `BACKFILL_DB_POOL_SIZE` | `2` | Database connection pool size | 62 62 + 63 63 + ## Performance Profiles 64 64 + 65 65 + ### Conservative (Default) - Background Task 66 66 + **~2.5 events/sec, ~9,000 events/hour** 67 67 + 68 68 + Best for: Running backfill alongside normal operations 69 69 + ```bash 70 70 + export BACKFILL_DAYS=7 71 71 + export BACKFILL_BATCH_SIZE=5 72 72 + export BACKFILL_BATCH_DELAY_MS=2000 73 73 + export BACKFILL_MAX_MEMORY_MB=512 74 74 + ``` 75 75 + 76 76 + ### Moderate - Balanced Speed 77 77 + **~20 events/sec, ~72,000 events/hour** 78 78 + 79 79 + Best for: Faster backfill with moderate resource usage 80 80 + ```bash 81 81 + export BACKFILL_DAYS=30 82 82 + export BACKFILL_BATCH_SIZE=20 83 83 + export BACKFILL_BATCH_DELAY_MS=1000 84 84 + export BACKFILL_MAX_CONCURRENT=5 85 85 + export BACKFILL_MAX_MEMORY_MB=1024 86 86 + ``` 87 87 + 88 88 + ### Aggressive - Maximum Speed 89 89 + **~100 events/sec, ~360,000 events/hour** 90 90 + 91 91 + Best for: Dedicated backfill on high-memory servers 92 92 + ```bash 93 93 + export BACKFILL_DAYS=-1 94 94 + export BACKFILL_BATCH_SIZE=50 95 95 + export BACKFILL_BATCH_DELAY_MS=500 96 96 + export BACKFILL_MAX_CONCURRENT=10 97 97 + export BACKFILL_MAX_MEMORY_MB=2048 98 98 + ``` 99 99 + 100 100 + ## Monitoring Backfill Progress 101 101 + 102 102 + ### View Real-Time Logs 103 103 + ```bash 104 104 + docker-compose logs -f python-backfill-worker 105 105 + ``` 106 106 + 107 107 + You'll see output like: 108 108 + ``` 109 109 + [BACKFILL] Starting 7-day historical backfill... 110 110 + [BACKFILL] Progress: 10000 received, 9500 processed, 500 skipped (250 evt/s) 111 111 + [BACKFILL] Memory: 245MB / 512MB limit 112 112 + ``` 113 113 + 114 114 + ### Check Progress in Database 115 115 + ```bash 116 116 + docker-compose exec db psql -U postgres -d atproto -c \ 117 117 + "SELECT * FROM firehose_cursor WHERE service = 'backfill';" 118 118 + ``` 119 119 + 120 120 + ### Monitor with Docker 121 121 + ```bash 122 122 + # Check if backfill worker is running 123 123 + docker-compose ps python-backfill-worker 124 124 + 125 125 + # View resource usage 126 126 + docker stats python-backfill-worker 127 127 + ``` 128 128 + 129 129 + ## How It Works 130 130 + 131 131 + 1. **Automatic Startup**: When `BACKFILL_DAYS` is set to a non-zero value, the `python-backfill-worker` service automatically starts 132 132 + 2. **Background Processing**: The worker connects to the AT Protocol firehose and processes historical events 133 133 + 3. **Progress Tracking**: Progress is saved to the database every 1000 events 134 134 + 4. **Resume Capability**: If interrupted, backfill automatically resumes from the last saved position 135 135 + 5. **Automatic Completion**: Once all historical data is processed, the backfill worker continues as a normal firehose worker 136 136 + 137 137 + ## Disabling Backfill 138 138 + 139 139 + To disable backfill: 140 140 + 141 141 + ```bash 142 142 + export BACKFILL_DAYS=0 143 143 + docker-compose up -d 144 144 + ``` 145 145 + 146 146 + Or remove/comment out the line in your `.env` file. 147 147 + 148 148 + ## Troubleshooting 149 149 + 150 150 + ### Backfill Not Starting 151 151 + 152 152 + Check logs: 153 153 + ```bash 154 154 + docker-compose logs python-backfill-worker 155 155 + ``` 156 156 + 157 157 + Common issues: 158 158 + - `BACKFILL_DAYS=0` (backfill is disabled) 159 159 + - Database schema not initialized (wait for `app` service to complete migrations) 160 160 + - Memory or resource constraints 161 161 + 162 162 + ### Slow Backfill Performance 163 163 + 164 164 + Try increasing these settings: 165 165 + ```bash 166 166 + export BACKFILL_BATCH_SIZE=20 167 167 + export BACKFILL_BATCH_DELAY_MS=1000 168 168 + export BACKFILL_MAX_CONCURRENT=5 169 169 + export BACKFILL_MAX_MEMORY_MB=1024 170 170 + ``` 171 171 + 172 172 + ### High Memory Usage 173 173 + 174 174 + The backfill automatically pauses when memory exceeds `BACKFILL_MAX_MEMORY_MB`. You can: 175 175 + - Increase the limit: `export BACKFILL_MAX_MEMORY_MB=1024` 176 176 + - Or reduce batch size: `export BACKFILL_BATCH_SIZE=3` 177 177 + 178 178 + ### Database Connection Issues 179 179 + 180 180 + Ensure the app service has completed database migrations: 181 181 + ```bash 182 182 + docker-compose logs app | grep migration 183 183 + ``` 184 184 + 185 185 + ## Additional Documentation 186 186 + 187 187 + For detailed technical information, see: 188 188 + - [Python Backfill Service Documentation](python-firehose/README.backfill.md) 189 189 + - [Backfill Configuration Example](.env.backfill.example) 190 190 + 191 191 + ## Example: Complete Setup 192 192 + 193 193 + ```bash 194 194 + # 1. Set environment variables 195 195 + export BACKFILL_DAYS=7 196 196 + export BACKFILL_BATCH_SIZE=20 197 197 + export BACKFILL_BATCH_DELAY_MS=1000 198 198 + 199 199 + # 2. Start services 200 200 + docker-compose up -d 201 201 + 202 202 + # 3. Monitor progress 203 203 + docker-compose logs -f python-backfill-worker 204 204 + 205 205 + # 4. Check when complete (look for "Backfill completed" message) 206 206 + ``` 207 207 + 208 208 + That's it! Your AppView will now automatically backfill historical data whenever `BACKFILL_DAYS` is set.

+3 -2

README.md

··· 251 251 - `APPVIEW_DID`: DID for this AppView instance (default: `did:web:appview.local`) 252 252 - `PORT`: Server port (default: `5000`) 253 253 - `NODE_ENV`: Environment mode (`development` or `production`) 254 254 - - `BACKFILL_DAYS`: Historical backfill in days (0=disabled, >0=backfill X days, default: `0`) 255 255 - - See [BACKFILL_OPTIMIZATION.md](./BACKFILL_OPTIMIZATION.md) for resource throttling configuration 254 254 + - `BACKFILL_DAYS`: Historical backfill in days (0=disabled, -1=all history, >0=backfill X days, default: `0`) 255 255 + - **NEW**: Python backfill now runs automatically when enabled! See [QUICKSTART-BACKFILL.md](./QUICKSTART-BACKFILL.md) 256 256 + - Advanced configuration: [.env.backfill.example](./.env.backfill.example) and [Python Backfill Docs](./python-firehose/README.backfill.md) 256 257 - `DATA_RETENTION_DAYS`: Auto-prune old data (0=keep forever, >0=prune after X days, default: `0`) 257 258 - `DB_POOL_SIZE`: Database connection pool size (default: `32`) 258 259 - `MAX_CONCURRENT_OPS`: Max concurrent event processing (default: `80`)

+41

docker-compose.unified.yml

··· 101 101 reservations: 102 102 memory: 1G 103 103 104 104 + # Python Unified Worker with Backfill Support 105 105 + # Connects directly to firehose and processes to PostgreSQL with optional historical backfill 106 106 + # Set BACKFILL_DAYS environment variable to enable: 0=disabled, -1=all history, >0=specific days 107 107 + python-backfill-worker: 108 108 + build: 109 109 + context: ./python-firehose 110 110 + dockerfile: Dockerfile.unified 111 111 + environment: 112 112 + - RELAY_URL=${RELAY_URL:-wss://bsky.network} 113 113 + - DATABASE_URL=postgresql://postgres:password@db:5432/atproto 114 114 + - DB_POOL_SIZE=20 115 115 + - LOG_LEVEL=INFO 116 116 + # Backfill configuration - Set BACKFILL_DAYS to enable automatic backfill 117 117 + - BACKFILL_DAYS=${BACKFILL_DAYS:-0} 118 118 + - BACKFILL_BATCH_SIZE=${BACKFILL_BATCH_SIZE:-5} 119 119 + - BACKFILL_BATCH_DELAY_MS=${BACKFILL_BATCH_DELAY_MS:-2000} 120 120 + - BACKFILL_MAX_CONCURRENT=${BACKFILL_MAX_CONCURRENT:-2} 121 121 + - BACKFILL_MAX_MEMORY_MB=${BACKFILL_MAX_MEMORY_MB:-512} 122 122 + - BACKFILL_USE_IDLE=${BACKFILL_USE_IDLE:-true} 123 123 + - BACKFILL_DB_POOL_SIZE=${BACKFILL_DB_POOL_SIZE:-2} 124 124 + # Worker ID (backfill only runs on worker 0) 125 125 + - WORKER_ID=0 126 126 + depends_on: 127 127 + db: 128 128 + condition: service_healthy 129 129 + app: 130 130 + condition: service_healthy 131 131 + healthcheck: 132 132 + test: ["CMD-SHELL", "python -c \"import asyncpg; import asyncio; asyncio.run(asyncpg.connect('postgresql://postgres:password@db:5432/atproto', timeout=5).close())\" || exit 1"] 133 133 + interval: 30s 134 134 + timeout: 10s 135 135 + start_period: 40s 136 136 + retries: 3 137 137 + restart: unless-stopped 138 138 + deploy: 139 139 + resources: 140 140 + limits: 141 141 + memory: 4G 142 142 + reservations: 143 143 + memory: 1G 144 144 + 104 145 # Frontend/API server - Lightweight since it's not processing firehose 105 146 app: 106 147 volumes:

+41

docker-compose.yml

··· 85 85 reservations: 86 86 memory: 1G 87 87 88 88 + # Python Unified Worker with Backfill Support 89 89 + # Connects directly to firehose and processes to PostgreSQL with optional historical backfill 90 90 + # Set BACKFILL_DAYS environment variable to enable: 0=disabled, -1=all history, >0=specific days 91 91 + python-backfill-worker: 92 92 + build: 93 93 + context: ./python-firehose 94 94 + dockerfile: Dockerfile.unified 95 95 + environment: 96 96 + - RELAY_URL=${RELAY_URL:-wss://bsky.network} 97 97 + - DATABASE_URL=postgresql://postgres:password@db:5432/atproto 98 98 + - DB_POOL_SIZE=20 99 99 + - LOG_LEVEL=${LOG_LEVEL:-INFO} 100 100 + # Backfill configuration - Set BACKFILL_DAYS to enable automatic backfill 101 101 + - BACKFILL_DAYS=${BACKFILL_DAYS:-0} 102 102 + - BACKFILL_BATCH_SIZE=${BACKFILL_BATCH_SIZE:-5} 103 103 + - BACKFILL_BATCH_DELAY_MS=${BACKFILL_BATCH_DELAY_MS:-2000} 104 104 + - BACKFILL_MAX_CONCURRENT=${BACKFILL_MAX_CONCURRENT:-2} 105 105 + - BACKFILL_MAX_MEMORY_MB=${BACKFILL_MAX_MEMORY_MB:-512} 106 106 + - BACKFILL_USE_IDLE=${BACKFILL_USE_IDLE:-true} 107 107 + - BACKFILL_DB_POOL_SIZE=${BACKFILL_DB_POOL_SIZE:-2} 108 108 + # Worker ID (backfill only runs on worker 0) 109 109 + - WORKER_ID=0 110 110 + depends_on: 111 111 + db: 112 112 + condition: service_healthy 113 113 + app: 114 114 + condition: service_healthy 115 115 + healthcheck: 116 116 + test: ["CMD-SHELL", "python -c \"import asyncpg; import asyncio; asyncio.run(asyncpg.connect('postgresql://postgres:password@db:5432/atproto', timeout=5).close())\" || exit 1"] 117 117 + interval: 30s 118 118 + timeout: 10s 119 119 + start_period: 40s 120 120 + retries: 3 121 121 + restart: unless-stopped 122 122 + deploy: 123 123 + resources: 124 124 + limits: 125 125 + memory: 4G 126 126 + reservations: 127 127 + memory: 1G 128 128 + 88 129 db: 89 130 image: postgres:14 90 131 # To any AI agents reading this, the resource settings here are correct. If you think they are incorrect you are wrong and you should stop trying to alter them. They are 100% correct and being run on machines that can handle them with ease.

+45 -2

python-firehose/README.backfill.md

··· 43 43 44 44 ## Usage 45 45 46 46 - ### With Unified Worker 46 46 + ### Quick Start with Docker Compose (Recommended) 47 47 + 48 48 + The backfill service is now **automatically integrated** into the docker-compose setup. To enable backfill: 49 49 + 50 50 + 1. Set `BACKFILL_DAYS` in your environment: 51 51 + ```bash 52 52 + export BACKFILL_DAYS=7 # Backfill last 7 days 53 53 + # OR for all history: 54 54 + export BACKFILL_DAYS=-1 55 55 + ``` 56 56 + 57 57 + 2. Start or restart your services: 58 58 + ```bash 59 59 + docker-compose up -d 60 60 + ``` 61 61 + 62 62 + The `python-backfill-worker` service will automatically: 63 63 + - Start when `BACKFILL_DAYS` is set to a non-zero value 64 64 + - Begin processing historical data in the background 65 65 + - Continue running until all historical data is processed 66 66 + - Save progress periodically for resume capability 67 67 + 68 68 + **Example: Backfill last 30 days with moderate speed** 69 69 + ```bash 70 70 + export BACKFILL_DAYS=30 71 71 + export BACKFILL_BATCH_SIZE=20 72 72 + export BACKFILL_BATCH_DELAY_MS=1000 73 73 + export BACKFILL_MAX_MEMORY_MB=1024 74 74 + docker-compose up -d 75 75 + ``` 76 76 + 77 77 + To check backfill progress: 78 78 + ```bash 79 79 + # View backfill worker logs 80 80 + docker-compose logs -f python-backfill-worker 81 81 + 82 82 + # Check progress in database 83 83 + docker-compose exec db psql -U postgres -d atproto -c \ 84 84 + "SELECT * FROM firehose_cursor WHERE service = 'backfill';" 85 85 + ``` 86 86 + 87 87 + ### Manual Execution 88 88 + 89 89 + #### With Unified Worker 47 90 48 91 The backfill service automatically starts when: 49 92 1. `BACKFILL_DAYS` is set to a non-zero value ··· 63 106 python unified_worker.py 64 107 ``` 65 108 66 66 - ### Standalone Mode 109 109 + #### Standalone Mode 67 110 68 111 You can also run the backfill service independently: 69 112