QuickDID is a high-performance AT Protocol identity resolution service written in Rust. It provides handle-to-DID resolution with Redis-backed caching and queue processing.
···11# QuickDID - Development Guide for Claude
2233## Overview
44-QuickDID is a high-performance AT Protocol identity resolution service written in Rust. It provides handle-to-DID resolution with multi-layer caching (Redis, SQLite, in-memory), queue processing, metrics support, and proactive cache refreshing.
44+QuickDID is a high-performance AT Protocol identity resolution service written in Rust. It provides bidirectional handle-to-DID and DID-to-handle resolution with multi-layer caching (Redis, SQLite, in-memory), queue processing, metrics support, proactive cache refreshing, and real-time cache updates via Jetstream consumer.
5566## Configuration
77···49491. **Handle Resolution** (`src/handle_resolver/`)
5050 - `BaseHandleResolver`: Core resolution using DNS and HTTP
5151 - `RateLimitedHandleResolver`: Semaphore-based rate limiting with optional timeout
5252- - `CachingHandleResolver`: In-memory caching layer
5353- - `RedisHandleResolver`: Redis-backed persistent caching
5454- - `SqliteHandleResolver`: SQLite-backed persistent caching
5252+ - `CachingHandleResolver`: In-memory caching layer with bidirectional support
5353+ - `RedisHandleResolver`: Redis-backed persistent caching with bidirectional lookups
5454+ - `SqliteHandleResolver`: SQLite-backed persistent caching with bidirectional support
5555 - `ProactiveRefreshResolver`: Automatically refreshes cache entries before expiration
5656+ - All resolvers implement `HandleResolver` trait with:
5757+ - `resolve`: Handle-to-DID resolution
5858+ - `purge`: Remove entries by handle or DID
5959+ - `set`: Manually update handle-to-DID mappings
5660 - Uses binary serialization via `HandleResolutionResult` for space efficiency
5761 - Resolution stack: Cache → ProactiveRefresh (optional) → RateLimited (optional) → Base → DNS/HTTP
5862 - Includes resolution timing measurements for metrics
···8286 - Tracks counters, gauges, and timings
8387 - Configurable tags for environment/service identification
8488 - No-op adapter for development environments
8989+ - Metrics for Jetstream event processing
9090+9191+6. **Jetstream Consumer** (`src/jetstream_handler.rs`)
9292+ - Consumes AT Protocol firehose events via WebSocket
9393+ - Processes Account events (purges deleted/deactivated accounts)
9494+ - Processes Identity events (updates handle-to-DID mappings)
9595+ - Automatic reconnection with exponential backoff
9696+ - Comprehensive metrics for event processing
9797+ - Spawned as cancellable task using task manager
85988699## Key Technical Details
87100···91104- Other DID methods stored with full identifier
9210593106### Redis Integration
9494-- **Caching**: Uses MetroHash64 for key generation, stores binary data
107107+- **Bidirectional Caching**:
108108+ - Stores both handle→DID and DID→handle mappings
109109+ - Uses MetroHash64 for key generation
110110+ - Binary data storage for efficiency
111111+ - Automatic synchronization of both directions
95112- **Queuing**: Reliable queue with processing/dead letter queues
96113- **Key Prefixes**: Configurable via `QUEUE_REDIS_PREFIX` environment variable
97114···101118 - Acquire semaphore permit (with optional timeout)
102119 - If timeout configured and exceeded, return error
1031203. Perform DNS TXT lookup or HTTP well-known query
104104-4. Cache result with appropriate TTL
121121+4. Cache result with appropriate TTL in both directions (handle→DID and DID→handle)
1051225. Return DID or error
123123+124124+### Cache Management Operations
125125+- **Purge**: Removes entries by either handle or DID
126126+ - Uses `atproto_identity::resolve::parse_input` for identifier detection
127127+ - Removes both handle→DID and DID→handle mappings
128128+ - Chains through all resolver layers
129129+- **Set**: Manually updates handle-to-DID mappings
130130+ - Updates both directions in cache
131131+ - Normalizes handles to lowercase
132132+ - Chains through all resolver layers
106133107134## Environment Variables
108135···154181- `PROACTIVE_REFRESH_ENABLED`: Enable proactive cache refreshing (default: false)
155182- `PROACTIVE_REFRESH_THRESHOLD`: Refresh when TTL remaining is below this threshold (0.0-1.0, default: 0.8)
156183184184+### Optional - Jetstream Consumer
185185+- `JETSTREAM_ENABLED`: Enable Jetstream consumer for real-time cache updates (default: false)
186186+- `JETSTREAM_HOSTNAME`: Jetstream WebSocket hostname (default: jetstream.atproto.tools)
187187+157188## Error Handling
158189159190All error strings must use this format:
···190221### Test Coverage Areas
191222- Handle resolution with various DID methods
192223- Binary serialization/deserialization
193193-- Redis caching and expiration
224224+- Redis caching and expiration with bidirectional lookups
194225- Queue processing logic
195226- HTTP endpoint responses
227227+- Jetstream event handler processing
228228+- Purge and set operations across resolver layers
196229197230## Development Patterns
198231···239272240273### Debugging Resolution Issues
2412741. Enable debug logging: `RUST_LOG=debug`
242242-2. Check Redis cache: `redis-cli GET "handle:<hash>"`
275275+2. Check Redis cache:
276276+ - Handle lookup: `redis-cli GET "handle:<hash>"`
277277+ - DID lookup: `redis-cli GET "handle:<hash>"` (same key format)
2432783. Check SQLite cache: `sqlite3 quickdid.db "SELECT * FROM handle_resolution_cache;"`
2442794. Monitor queue processing in logs
2452805. Check rate limiting: Look for "Rate limit permit acquisition timed out" errors
2462816. Verify DNS/HTTP connectivity to AT Protocol infrastructure
2472827. Monitor metrics for resolution timing and cache hit rates
283283+8. Check Jetstream consumer status:
284284+ - Look for "Jetstream consumer" log entries
285285+ - Monitor `jetstream.*` metrics
286286+ - Check reconnection attempts in logs
248287249288## Dependencies
250289- `atproto-identity`: Core AT Protocol identity resolution
290290+- `atproto-jetstream`: AT Protocol Jetstream event consumer
251291- `bincode`: Binary serialization
252292- `deadpool-redis`: Redis connection pooling
253293- `metrohash`: Fast non-cryptographic hashing
+29-1
README.md
···2121## Features
22222323- **Fast Handle Resolution**: Resolves AT Protocol handles to DIDs using DNS TXT records and HTTP well-known endpoints
2424+- **Bidirectional Caching**: Supports both handle-to-DID and DID-to-handle lookups with automatic cache synchronization
2425- **Multi-Layer Caching**: Flexible caching with three tiers:
2526 - In-memory caching with configurable TTL (default: 600 seconds)
2627 - Redis-backed persistent caching (default: 90-day TTL)
2728 - SQLite-backed persistent caching (default: 90-day TTL)
2929+- **Jetstream Consumer**: Real-time cache updates from AT Protocol firehose:
3030+ - Processes Account and Identity events
3131+ - Automatically purges deleted/deactivated accounts
3232+ - Updates handle-to-DID mappings in real-time
3333+ - Comprehensive metrics for event processing
3434+ - Automatic reconnection with backoff
2835- **HTTP Caching**: Client-side caching support with:
2936 - ETag generation with configurable seed for cache invalidation
3037 - Cache-Control headers with max-age, stale-while-revalidate, and stale-if-error directives
···3946- **Metrics & Monitoring**:
4047 - StatsD metrics support for counters, gauges, and timings
4148 - Resolution timing measurements
4949+ - Jetstream event processing metrics
4250 - Configurable tags for environment/service identification
4351 - Integration guides for Telegraf and TimescaleDB
4452 - Configurable bind address for StatsD UDP socket (IPv4/IPv6)
···5159 - Redis-based deduplication for queue items
5260 - Prevents duplicate handle resolution work
5361 - Configurable TTL for deduplication keys
6262+- **Cache Management APIs**:
6363+ - `purge` method for removing entries by handle or DID
6464+ - `set` method for manually updating handle-to-DID mappings
6565+ - Chainable operations across resolver layers
5466- **AT Protocol Compatible**: Implements XRPC endpoints for seamless integration with AT Protocol infrastructure
5567- **Comprehensive Error Handling**: Structured errors with unique identifiers (e.g., `error-quickdid-config-1`), health checks, and graceful shutdown
5668- **12-Factor App**: Environment-based configuration following cloud-native best practices
···164176- `PROACTIVE_REFRESH_ENABLED`: Enable proactive cache refreshing (default: false)
165177- `PROACTIVE_REFRESH_THRESHOLD`: Refresh when TTL remaining is below this threshold (0.0-1.0, default: 0.8)
166178179179+#### Jetstream Consumer
180180+- `JETSTREAM_ENABLED`: Enable Jetstream consumer for real-time cache updates (default: false)
181181+- `JETSTREAM_HOSTNAME`: Jetstream WebSocket hostname (default: jetstream.atproto.tools)
182182+167183#### Static Files
168184- `STATIC_FILES_DIR`: Directory for serving static files (default: www)
169185···172188173189### Production Examples
174190175175-#### Redis-based with Metrics (Multi-instance/HA)
191191+#### Redis-based with Metrics and Jetstream (Multi-instance/HA)
176192```bash
177193HTTP_EXTERNAL=quickdid.example.com \
178194HTTP_PORT=3000 \
···187203METRICS_PREFIX=quickdid \
188204METRICS_TAGS=env:prod,service:quickdid \
189205CACHE_MAX_AGE=86400 \
206206+JETSTREAM_ENABLED=true \
207207+JETSTREAM_HOSTNAME=jetstream.atproto.tools \
190208RUST_LOG=info \
191209./target/release/quickdid
192210```
···213231 ↓ ↓ ↓ ↓
214232 Memory/Redis/ Background Semaphore AT Protocol
215233 SQLite Refresher (optional) Infrastructure
234234+ ↑
235235+ Jetstream Consumer ← Real-time Updates from AT Protocol Firehose
216236```
217237218238### Cache Priority
···2212412. SQLite (if configured) - Best for single-instance with persistence
2222423. In-memory (fallback) - Always available
223243244244+### Real-time Cache Updates
245245+When Jetstream is enabled, QuickDID maintains cache consistency by:
246246+- Processing Account events to purge deleted/deactivated accounts
247247+- Processing Identity events to update handle-to-DID mappings
248248+- Automatically reconnecting with exponential backoff on connection failures
249249+- Tracking metrics for successful and failed event processing
250250+224251### Deployment Strategies
225252226253- **Single-instance**: Use SQLite for both caching and queuing
227254- **Multi-instance/HA**: Use Redis for distributed caching and queuing
228255- **Development**: Use in-memory caching with MPSC queuing
256256+- **Real-time sync**: Enable Jetstream consumer for live cache updates
229257230258## API Endpoints
231259
+119-2
docs/configuration-reference.md
···1010- [Queue Configuration](#queue-configuration)
1111- [Rate Limiting Configuration](#rate-limiting-configuration)
1212- [HTTP Caching Configuration](#http-caching-configuration)
1313+- [Metrics Configuration](#metrics-configuration)
1414+- [Proactive Refresh Configuration](#proactive-refresh-configuration)
1515+- [Jetstream Consumer Configuration](#jetstream-consumer-configuration)
1616+- [Static Files Configuration](#static-files-configuration)
1317- [Configuration Examples](#configuration-examples)
1418- [Validation Rules](#validation-rules)
1519···761765- TTL=3600s (1 hour), threshold=0.8: Refresh after 48 minutes
762766- TTL=86400s (1 day), threshold=0.8: Refresh after 19.2 hours
763767768768+## Jetstream Consumer Configuration
769769+770770+### `JETSTREAM_ENABLED`
771771+772772+**Required**: No
773773+**Type**: Boolean
774774+**Default**: `false`
775775+776776+Enable Jetstream consumer for real-time cache updates from the AT Protocol firehose. When enabled, QuickDID connects to the Jetstream WebSocket service to receive live updates about account and identity changes.
777777+778778+**How it works**:
779779+- Subscribes to Account and Identity events from the firehose
780780+- Processes Account events to purge deleted/deactivated accounts
781781+- Processes Identity events to update handle-to-DID mappings
782782+- Automatically reconnects with exponential backoff on connection failures
783783+- Tracks metrics for successful and failed event processing
784784+785785+**Examples**:
786786+```bash
787787+# Enable Jetstream consumer (recommended for production)
788788+JETSTREAM_ENABLED=true
789789+790790+# Disable Jetstream consumer (default)
791791+JETSTREAM_ENABLED=false
792792+```
793793+794794+**Benefits**:
795795+- Real-time cache synchronization with AT Protocol network
796796+- Automatic removal of deleted/deactivated accounts
797797+- Immediate handle change updates
798798+- Reduces stale data in cache
799799+800800+**Considerations**:
801801+- Requires stable WebSocket connection
802802+- Increases network traffic (incoming events)
803803+- Best for services requiring up-to-date handle mappings
804804+- Automatically handles reconnection on failures
805805+806806+### `JETSTREAM_HOSTNAME`
807807+808808+**Required**: No
809809+**Type**: String
810810+**Default**: `jetstream.atproto.tools`
811811+812812+The hostname of the Jetstream WebSocket service to connect to for real-time AT Protocol events. Only used when `JETSTREAM_ENABLED=true`.
813813+814814+**Examples**:
815815+```bash
816816+# Production firehose (default)
817817+JETSTREAM_HOSTNAME=jetstream.atproto.tools
818818+819819+# Staging environment
820820+JETSTREAM_HOSTNAME=jetstream-staging.atproto.tools
821821+822822+# Local development firehose
823823+JETSTREAM_HOSTNAME=localhost:6008
824824+825825+# Custom deployment
826826+JETSTREAM_HOSTNAME=jetstream.example.com
827827+```
828828+829829+**Event Processing**:
830830+- **Account events**:
831831+ - `status: deleted` → Purges handle and DID from all caches
832832+ - `status: deactivated` → Purges handle and DID from all caches
833833+ - Other statuses → Ignored
834834+835835+- **Identity events**:
836836+ - Updates handle-to-DID mapping in cache
837837+ - Removes old handle mapping if changed
838838+ - Maintains bidirectional cache consistency
839839+840840+**Metrics Tracked** (when metrics are enabled):
841841+- `jetstream.events.received`: Total events received
842842+- `jetstream.events.processed`: Successfully processed events
843843+- `jetstream.events.failed`: Failed event processing
844844+- `jetstream.connections.established`: Successful connections
845845+- `jetstream.connections.failed`: Failed connection attempts
846846+847847+**Reconnection Behavior**:
848848+- Initial retry delay: 1 second
849849+- Maximum retry delay: 60 seconds
850850+- Exponential backoff with jitter
851851+- Automatic recovery on transient failures
852852+853853+**Recommendations**:
854854+- **Production**: Use default `jetstream.atproto.tools`
855855+- **Development**: Consider local firehose for testing
856856+- **High availability**: Monitor connection metrics
857857+- **Network issues**: Check WebSocket connectivity
858858+764859## Static Files Configuration
765860766861### `STATIC_FILES_DIR`
···10261121PROACTIVE_REFRESH_ENABLED=true
10271122PROACTIVE_REFRESH_THRESHOLD=0.8
1028112311241124+# Jetstream Consumer (optional, recommended for real-time sync)
11251125+JETSTREAM_ENABLED=true
11261126+JETSTREAM_HOSTNAME=jetstream.atproto.tools
11271127+10291128# HTTP Caching (Cache-Control headers)
10301129CACHE_MAX_AGE=86400 # 24 hours
10311130CACHE_STALE_IF_ERROR=172800 # 48 hours
···10591158# Rate Limiting (optional, recommended for production)
10601159RESOLVER_MAX_CONCURRENT=100
10611160RESOLVER_MAX_CONCURRENT_TIMEOUT_MS=5000 # 5 second timeout
11611161+11621162+# Jetstream Consumer (optional, recommended for real-time sync)
11631163+JETSTREAM_ENABLED=true
11641164+JETSTREAM_HOSTNAME=jetstream.atproto.tools
1062116510631166# HTTP Caching (Cache-Control headers)
10641167CACHE_MAX_AGE=86400 # 24 hours
···11101213# Proactive Refresh (recommended for HA)
11111214PROACTIVE_REFRESH_ENABLED=true
11121215PROACTIVE_REFRESH_THRESHOLD=0.7 # More aggressive for HA
12161216+12171217+# Jetstream Consumer (recommended for real-time sync in HA)
12181218+JETSTREAM_ENABLED=true
12191219+JETSTREAM_HOSTNAME=jetstream.atproto.tools
1113122011141221# Logging
11151222RUST_LOG=warn
···11571264 CACHE_TTL_REDIS: 86400
11581265 QUEUE_ADAPTER: redis
11591266 QUEUE_REDIS_TIMEOUT: 5
12671267+ JETSTREAM_ENABLED: true
12681268+ JETSTREAM_HOSTNAME: jetstream.atproto.tools
11601269 RUST_LOG: info
11611270 ports:
11621271 - "8080:8080"
···11861295 QUEUE_ADAPTER: sqlite
11871296 QUEUE_BUFFER_SIZE: 5000
11881297 QUEUE_SQLITE_MAX_SIZE: 10000
12981298+ JETSTREAM_ENABLED: true
12991299+ JETSTREAM_HOSTNAME: jetstream.atproto.tools
11891300 RUST_LOG: info
11901301 ports:
11911302 - "8080:8080"
···128113922. **Single-instance deployments**: Use SQLite for persistent caching and queuing
128213933. **Development/testing**: Use memory-only caching with MPSC queuing
128313944. **Hybrid setups**: Configure both Redis and SQLite for redundancy
12841284-5. **Queue adapter guidelines**:
13951395+5. **Real-time sync**: Enable Jetstream consumer for live cache updates
13961396+6. **Queue adapter guidelines**:
12851397 - Redis: Best for multi-instance deployments with distributed processing
12861398 - SQLite: Best for single-instance deployments needing persistence
12871399 - MPSC: Best for single-instance deployments without persistence needs
12881288-6. **Cache TTL guidelines**:
14001400+7. **Cache TTL guidelines**:
12891401 - Redis: Shorter TTLs (1-7 days) for frequently updated handles
12901402 - SQLite: Longer TTLs (7-90 days) for stable single-instance caching
12911403 - Memory: Short TTLs (5-30 minutes) as fallback
14041404+8. **Jetstream guidelines**:
14051405+ - Production: Enable for real-time cache synchronization
14061406+ - High-traffic: Essential for reducing stale data
14071407+ - Development: Can be disabled for simpler testing
14081408+ - Monitor WebSocket connection health in production
1292140912931410### Monitoring
12941411
+33-4
docs/production-deployment.md
···296296PROACTIVE_REFRESH_THRESHOLD=0.8
297297298298# ----------------------------------------------------------------------------
299299+# JETSTREAM CONSUMER CONFIGURATION
300300+# ----------------------------------------------------------------------------
301301+302302+# Enable Jetstream consumer for real-time cache updates (default: false)
303303+# When enabled, connects to AT Protocol firehose for live updates
304304+# Processes Account events (deleted/deactivated) and Identity events (handle changes)
305305+# Automatically reconnects with exponential backoff on connection failures
306306+JETSTREAM_ENABLED=false
307307+308308+# Jetstream WebSocket hostname (default: jetstream.atproto.tools)
309309+# The firehose service to connect to for real-time AT Protocol events
310310+# Examples:
311311+# - jetstream.atproto.tools (production firehose)
312312+# - jetstream-staging.atproto.tools (staging environment)
313313+# - localhost:6008 (local development)
314314+JETSTREAM_HOSTNAME=jetstream.atproto.tools
315315+316316+# ----------------------------------------------------------------------------
299317# STATIC FILES CONFIGURATION
300318# ----------------------------------------------------------------------------
301319···413431414432## Docker Compose Setup
415433416416-### Redis-based Production Setup
434434+### Redis-based Production Setup with Jetstream
417435418418-Create a `docker-compose.yml` file for a complete production setup with Redis:
436436+Create a `docker-compose.yml` file for a complete production setup with Redis and optional Jetstream consumer:
419437420438```yaml
421439version: '3.8'
···504522 driver: local
505523```
506524507507-### SQLite-based Single-Instance Setup
525525+### SQLite-based Single-Instance Setup with Jetstream
508526509509-For single-instance deployments without Redis, create a simpler `docker-compose.sqlite.yml`:
527527+For single-instance deployments without Redis, create a simpler `docker-compose.sqlite.yml` with optional Jetstream consumer:
510528511529```yaml
512530version: '3.8'
···524542 QUEUE_ADAPTER: sqlite
525543 QUEUE_BUFFER_SIZE: 5000
526544 QUEUE_SQLITE_MAX_SIZE: 10000
545545+ # Optional: Enable Jetstream for real-time cache updates
546546+ # JETSTREAM_ENABLED: true
547547+ # JETSTREAM_HOSTNAME: jetstream.atproto.tools
527548 RUST_LOG: info
528549 ports:
529550 - "8080:8080"
···9059262. **SQLite** (persistent, best for single-instance)
9069273. **Memory** (fast, but lost on restart)
907928929929+**Real-time Updates with Jetstream**: When `JETSTREAM_ENABLED=true`, QuickDID:
930930+- Connects to AT Protocol firehose for live cache updates
931931+- Processes Account events to purge deleted/deactivated accounts
932932+- Processes Identity events to update handle-to-DID mappings
933933+- Automatically reconnects with exponential backoff on failures
934934+- Tracks metrics for successful and failed event processing
935935+908936**Recommendations by Deployment Type**:
909937- **Single instance, persistent**: Use SQLite for both caching and queuing (`SQLITE_URL=sqlite:./quickdid.db`, `QUEUE_ADAPTER=sqlite`)
910938- **Multi-instance, HA**: Use Redis for both caching and queuing (`REDIS_URL=redis://redis:6379/0`, `QUEUE_ADAPTER=redis`)
939939+- **Real-time sync**: Enable Jetstream consumer (`JETSTREAM_ENABLED=true`) for live cache updates
911940- **Testing/development**: Use memory-only caching with MPSC queuing (`QUEUE_ADAPTER=mpsc`)
912941- **Hybrid**: Configure both Redis and SQLite for redundancy
913942
+4-1
src/bin/quickdid.rs
···633633 compression: false,
634634 zstd_dictionary_location: String::new(),
635635 jetstream_hostname: jetstream_hostname.clone(),
636636- collections: vec![], // Listen to all collections
636636+ // Listen to the "community.lexicon.collection.fake" collection
637637+ // so that we keep an active connection open but only for
638638+ // account and identity events.
639639+ collections: vec!["community.lexicon.collection.fake".to_string()], // Listen to all collections
637640 dids: vec![],
638641 max_message_size_bytes: None,
639642 cursor: None,