turso bad_gateway and http errors were only logged to stderr via
logfire.err(), not recorded on the otel spans. this made debugging
via logfire difficult since errors didn't show up as exceptions.

now calls span.recordError() + adds turso.status and turso.response
attributes so these errors are visible in logfire dashboard.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

get_document was only looking for pages[] at the top level, which works
for pub.leaflet.document records. site.standard.document records nest
pages under content.pages[] instead.

now checks both locations to extract plaintext content.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

Custom domain Leaflet sites (e.g., cailean.journal.ewancroft.uk) use
site.standard.document but have content.$type = "pub.leaflet.content".

- Add content_type field to ExtractedDocument
- Extract content.$type during document extraction
- Use as fallback in platform detection when basePath doesn't match

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

- add stats_buffer.zig: atomic counters + background sync to Turso (5s interval)
- stats.zig: recordSearch/recordCacheHit/etc now instant (~1us) via buffer
- LocalDb.zig: add similarity_cache table for local caching
- search.zig: local-first cache lookups + cached doc count (5min refresh)
- sync.zig: sync similarity_cache from Turso on startup

before: search latency 100-7600ms (blocked on Turso stats writes)
after: search latency <10ms (stats buffered, synced in background)

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

updates to version with proper parent-child span relationships

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

uses otel-backed logfire-zig with proper Linux resource detection
(bypasses otel-zig's macOS-only ProcessDetector).

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

This reverts commit 0f8728781f380cbe2dc9ce61ae635e9ba50a11af.

- uses otel-zig for proper OpenTelemetry protocol compliance
- same API, no code changes required
- OTLP protobuf export instead of JSON

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

- index stats, usage metrics, latency percentiles
- timeline chart for docs indexed per day
- p50 latency comparison chart
- clear labeling (e.g., "similar cache hit" not just "cache")

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

- tap now signals on site.standard.document (not pub.leaflet.document)
- embeddings are auto-generated by backend worker since a4509a4
- consolidate stack section: Fly hosts Zig backend, Turso with Voyage vectors
- link tap from first mention in "how it works"
- add greengale to known platforms in CLAUDE.md

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

- remove diagnose-latency.py (logfire shows latency distributions)
- simplify exercise-api to just hit endpoints, point to logfire for results

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

upstream fix addresses dangling pointer issue when Span.Data is copied.
string attributes now correctly survive struct copies.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

escapes non-ASCII bytes to fix unicode errors

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

fixes string attribute serialization using raw mode

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

fixes string attribute serialization bug

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

logfire-zig now copies string attributes internally, so arena-allocated
query params can safely be included in span attributes. this enables
seeing the actual search query in logfire traces.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

- http.search now shows: query, tag, platform filters
- http.similar now shows: uri

makes it easy to see what was searched at a glance

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

- include error name, SQL, and response preview in error logs
- local db errors now show: error name + sql
- turso errors now show: status + sql + response body preview

makes debugging db errors much easier

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

- db.query spans for Turso HTTP queries (with sql, args_count)
- db.batch spans for Turso batch queries
- db.local.query spans for local SQLite queries

queries now appear as children of HTTP handler spans

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

child spans now correctly set parent_span_id

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

fixes concurrent requests sharing trace IDs

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

numeric values (asInt, count, bucketCounts) now serialize as numbers
instead of strings per OTLP JSON spec

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

- update to logfire-zig with auto-flush, proper trace_id per root span,
and delta temporality for metrics
- add UTF-8 sanitization in embedder to fix Voyage API JSON serialization
(invalid UTF-8 caused json.Stringify to output byte arrays)
- tone down debug logging (remove per-document logs, reduce tap message
count logging frequency)

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

- add logfire-zig dependency from tangled.sh
- configure logfire in main.zig (reads LOGFIRE_WRITE_TOKEN from env)
- add spans to HTTP handlers (search, tags, popular, similar)
- add spans to embedder batch processing and voyage API calls
- add span to TAP record indexing
- replace std.debug.print with logfire.info/warn/err throughout
- add counters for search requests, documents indexed, publications indexed

when LOGFIRE_WRITE_TOKEN is not set, falls back to console output

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

- remove zig-logfire-client.md (pre-implementation design notes)
- add logfire-zig-adoption.md (concrete integration guide)

the client exists now, design exploration is obsolete.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

the fetchLeafletContent function was a defensive fallback for when
content.pages extraction failed. now that extraction handles the
nested path correctly, it's never triggered. removing ~40 lines.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

- clarify textContent is intentionally null for leaflet (record size)
- remove PDS fetch section (unnecessary with correct extraction)
- simplify to key insight: extract from content.pages correctly
- add note about leaflet phasing out pub.leaflet.document records

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

- add embedder.zig: background worker that generates embeddings via Voyage API
- polls for documents missing embeddings, batches them, updates Turso
- content truncation to stay under Voyage token limits
- fix indexer to use ON CONFLICT instead of INSERT OR REPLACE
(INSERT OR REPLACE was nuking embeddings on document updates)

no more dependency on manual backfill scripts for new documents.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

- add content-extraction.md documenting site.standard.document gotchas
- links to eli mallon's question about the content field wrapper pattern
- move old planning docs to docs/scratch/

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

- fix ExtractedDocument docstring: only content/tags are allocated,
other fields borrow from parsed JSON
- add takeContent() method for clean ownership transfer pattern
(per arraylist.md: toOwnedSlice vs deinit patterns)
- use takeContent() in fetchLeafletContent instead of fragile
partial deinit that manually freed only tags
- remove verbose ACK logging that printed for every message

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

This commit fixes indexing of documents published via site.standard.document
that contain embedded pub.leaflet.content. This was a multi-layered debugging
saga worth documenting.

## The Problem

Search for "set of shared standards" returned nothing, despite the document
existing at lab.leaflet.pub/3md4qsktbms24.

## Root Causes (in order of discovery)

1. **Fly secret overriding fly.toml** - TAP_SIGNAL_COLLECTION was set as a
Fly secret with wrong value ("pub.leaflet.document"), which overrode the
correct fly.toml value. Secrets take precedence over env vars in fly.toml.
Fix: `fly secrets unset TAP_SIGNAL_COLLECTION -a leaflet-search-tap`

2. **Extractor missing content.pages path** - The extractor only checked
`record.pages` (pub.leaflet.document) but not `record.content.pages`
(site.standard.document with embedded pub.leaflet.content). The content
structure differs between lexicons.

3. **Platform detection from collection only** - site.standard.document
gets platform="other" from collection detection, but the actual platform
(leaflet/pckt/offprint) should be inferred from the publication's basePath.

4. **Use-after-free in indexer** - base_path was read from query result,
but row.text() returns memory that's freed by result.deinit(). Fixed by
copying to stack buffer.

5. **Startup timeout** - HTTP server started after slow initialization,
causing Fly proxy timeouts. Reordered to start HTTP first, then init
services in background.

## The Saga of Failure Modes

- Repeatedly trying to curl from TAP container (which has no curl)
- Hallucinating TAP API endpoints that don't exist
- Falling back to backfill scripts instead of fixing root cause
- Not verifying Fly secrets vs fly.toml precedence
- Writing custom PDS client when zat already has everything needed

## How Content Extraction Now Works

1. TAP receives site.standard.document (signal collection)
2. Extractor tries: textContent → pages → content.pages
3. If content still empty AND content.$type == "pub.leaflet.content",
fetch pub.leaflet.document from PDS as fallback
4. Indexer infers platform from basePath (leaflet.pub → leaflet, etc.)

## Files Changed

- tap/fly.toml: signal collection → site.standard.document
- backend/src/extractor.zig: check content.pages, add test
- backend/src/tap.zig: PDS fetch fallback for empty content
- backend/src/indexer.zig: platform detection from basePath, fix UAF
- backend/src/main.zig: start HTTP first, init services in background
- backend/src/db/mod.zig: split init for staged startup
- scripts/backfill-pds: same content.pages fix
- docs/tap.md: updated documentation

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

practitioner notes for a potential official logfire client for zig,
referencing find-bufo rust implementation and OTLP protocol details.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

- bump RAM from 256mb to 512mb (fixes SIGABRT crashes under load)
- disable auto_stop to eliminate cold start latency
- add bench-search script for performance testing
- gitignore .loq_cache

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

Root cause: WAL file grew to 498MB (main db was 113MB) without ever
being checkpointed back to main database. Large WAL files cause FTS5
queries to fail with generic SQLite errors.

Changes:
- Add PRAGMA wal_checkpoint(TRUNCATE) after full sync
- Add PRAGMA wal_checkpoint(PASSIVE) after incremental sync
- Add integrity check on startup that auto-deletes corrupt db
- Store db path for potential auto-recovery

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

previously timing data was only loaded from disk when record() was
called. if the first request after restart was to /api/dashboard,
the timing data would appear empty.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

removed interval batching - just persist immediately.
simpler code, no data loss between deploys.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

- restored original timing table (no inline charts)
- added new "latency history (24h)" section with area chart
- shows search and similar endpoints as colored lines
- grafana-style visualization

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

- timing now stores hourly buckets for last 24 hours
- dashboard API returns history array per endpoint
- frontend renders mini bar charts under each endpoint
- renamed "activity" to "documents indexed" for clarity

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

saves timing stats to /data/timing.bin every 100 requests.
loads on first request after restart. survives deploys.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

the stats table (searches, started_at, errors, cache hits) doesn't
sync to the local replica, so we must query Turso for it.
document/publication counts can still use fast local queries.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

routes dashboard and stats queries to local SQLite first,
falling back to Turso only when local db unavailable.
reduces dashboard latency from ~6s to <100ms.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

- highlightTerms() now extracts quoted phrases as single terms
- add daily scheduled workflow for embedding backfill (6am UTC)
- can also trigger backfill manually via workflow_dispatch

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

- search() now accepts platform (leaflet|pckt|offprint|greengale|other)
- search() now accepts since (ISO date) for filtering recent content
- SearchResult includes platform field from API
- trimmed prompts for fewer tokens
- added docs/api.md with full API reference
- added scripts/test_live.py for testing with FastMCP client

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

- use same background/border/colors as .tag elements
- add flex: none to prevent expansion from global input[type="text"]
- add 44px touch targets on mobile and touch devices
- add hover state matching .tag:hover

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

- delete stale publication/self records that cause wrong basePath lookups
- add platform-aware fallback: match greengale docs to greengale pubs
- re-derive basePath for affected documents
- fix tag input CSS specificity (input.tag-input beats input[type="text"])

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

users can now type any tag to filter by, not just popular tags
- small input field after the popular tag chips
- enter key applies the filter
- consistent styling with existing UI

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

- add basePath detection migration for greengale.app/*
- add greengale to frontend platform filter and config
- update docs and README
- ~29 existing documents will be backfilled on deploy

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

0.27ms → 270µs

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

- >=10ms: whole ms
- 1-10ms: 1 decimal
- <1ms: 2 decimals

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>