prefect server in zig

add database migrations planning document

- docs/migrations.md: comprehensive analysis of migration options
- zmig (sqlite-only zig tool)
- atlas (external go-based tool, multi-dialect)
- minimal DIY approach
- hybrid recommendation (atlas for dev, embedded SQL for runtime)
- phased implementation plan
- references to python prefect alembic patterns

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

+174
+1
docs/README.md
··· 6 6 7 7 - [web server](./web-server.md) - http layer, route dispatch 8 8 - [database](./database.md) - sqlite + postgres persistence 9 + - [migrations](./migrations.md) - schema versioning strategy (planning) 9 10 - [services](./services.md) - background workers 10 11 11 12 broker and messaging details in `src/broker/CLAUDE.md` and `src/services/CLAUDE.md`.
+8
docs/database.md
··· 122 122 123 123 the `Transaction` type ensures all statements execute on the same connection, which is critical for postgres where the pool might otherwise give different connections for each query. 124 124 125 + ## migrations 126 + 127 + **current state**: no migration system. schema is applied via `CREATE TABLE IF NOT EXISTS` on startup. 128 + 129 + see [migrations.md](./migrations.md) for planning on proper migration support. 130 + 131 + see [python-reference/](./python-reference/) for how prefect python handles migrations with alembic. 132 + 125 133 ## testing 126 134 127 135 ```bash
+165
docs/migrations.md
··· 1 + # database migrations 2 + 3 + ## current state: no migration system 4 + 5 + we currently have no migration system. schema changes are: 6 + - embedded in `src/db/schema/sqlite.zig` and `src/db/schema/postgres.zig` 7 + - applied via `CREATE TABLE IF NOT EXISTS` on startup 8 + - manually applied to existing databases (`ALTER TABLE ...`) 9 + 10 + this works for development but **will not work for production**. 11 + 12 + ## requirements 13 + 14 + 1. **version tracking** - know which migrations have been applied 15 + 2. **dual dialect support** - sqlite and postgres, possibly with different SQL 16 + 3. **forward migrations** - apply schema changes 17 + 4. **rollback capability** - undo changes (nice to have) 18 + 5. **embedded runtime** - apply migrations on server startup 19 + 6. **CLI tooling** - create/manage migrations during development 20 + 21 + ## options evaluated 22 + 23 + ### option 1: zmig (sqlite-only) 24 + 25 + [zmig](https://github.com/Jeansidharta/zmig) - native zig migration tool 26 + 27 + **pros:** 28 + - pure zig, embeds migrations in binary 29 + - timestamp-based ordering 30 + - up/down migrations 31 + - runtime API: `applyMigrations(db, alloc, options)` 32 + - requires zig 0.15.2 (we use this) 33 + 34 + **cons:** 35 + - sqlite only - no postgres support 36 + - would need to fork/extend for postgres 37 + 38 + **migration format:** 39 + ``` 40 + migrations/ 41 + ├── 1758503588032-add_empirical_policy.up.sql 42 + └── 1758503588032-add_empirical_policy.down.sql 43 + ``` 44 + 45 + ### option 2: atlas (external tool) 46 + 47 + [atlas](https://atlasgo.io/) - schema-as-code migration tool (go-based) 48 + 49 + **pros:** 50 + - supports sqlite AND postgres 51 + - declarative schema definitions 52 + - automatic migration generation 53 + - k8s operator available 54 + - language-agnostic CLI 55 + 56 + **cons:** 57 + - external dependency (go binary) 58 + - not embedded in our binary 59 + - adds operational complexity 60 + - schema defined separately from zig code 61 + 62 + ### option 3: minimal DIY 63 + 64 + build minimal migration system ourselves: 65 + 66 + **pros:** 67 + - tailored to our exact needs 68 + - no external dependencies 69 + - full control over dialect differences 70 + - embeds in binary 71 + 72 + **cons:** 73 + - development effort 74 + - must handle edge cases ourselves 75 + 76 + ### option 4: hybrid approach (recommended) 77 + 78 + **development**: use atlas CLI for migration creation/testing 79 + **runtime**: embed migration SQL files and apply with simple zig code 80 + 81 + ``` 82 + migrations/ 83 + ├── versions.txt # ordered list of migration IDs 84 + ├── 001_initial/ 85 + │ ├── sqlite.up.sql 86 + │ ├── sqlite.down.sql 87 + │ ├── postgres.up.sql 88 + │ └── postgres.down.sql 89 + └── 002_add_empirical_policy/ 90 + ├── sqlite.up.sql 91 + └── postgres.up.sql 92 + ``` 93 + 94 + runtime logic (minimal zig code): 95 + ```zig 96 + pub fn applyMigrations(db: *Backend) !void { 97 + // 1. create migrations table if not exists 98 + // 2. read applied migrations from table 99 + // 3. read embedded migrations from @embedFile 100 + // 4. apply any pending migrations in order 101 + // 5. record applied migrations 102 + } 103 + ``` 104 + 105 + ## migration table schema 106 + 107 + ```sql 108 + CREATE TABLE IF NOT EXISTS _migrations ( 109 + id TEXT PRIMARY KEY, -- "001_initial" 110 + applied_at TEXT NOT NULL, -- ISO timestamp 111 + checksum TEXT -- SHA256 of migration SQL (optional) 112 + ); 113 + ``` 114 + 115 + ## implementation phases 116 + 117 + ### phase 1: tracking (minimal) 118 + 119 + 1. add `_migrations` table to schema 120 + 2. on startup, record current "version" (e.g., "001_initial") 121 + 3. log warning if database version != code version 122 + 123 + ### phase 2: embedded migrations 124 + 125 + 1. create `migrations/` directory with SQL files 126 + 2. embed migrations in binary via `@embedFile` 127 + 3. apply pending migrations on startup 128 + 4. support `--migrate-only` CLI flag 129 + 130 + ### phase 3: development tooling 131 + 132 + 1. add `zig build migrate:new` or similar 133 + 2. generate timestamped migration directories 134 + 3. optionally integrate atlas for schema diffing 135 + 136 + ### phase 4: rollbacks (if needed) 137 + 138 + 1. add down.sql files 139 + 2. add `--rollback` CLI flag 140 + 3. track rollback state 141 + 142 + ## alembic patterns to preserve 143 + 144 + from python prefect (see `docs/python-reference/migrations.md`): 145 + 146 + 1. **separate dialect chains** - different SQL per database 147 + 2. **migration notes** - document changes with both dialect IDs 148 + 3. **batch mode for sqlite** - table recreation for ALTER 149 + 4. **transaction per migration** - isolation 150 + 5. **migration lock** - prevent concurrent execution 151 + 152 + ## questions to resolve 153 + 154 + 1. **when to migrate?** startup vs explicit command vs both? 155 + 2. **failure handling?** abort startup? partial state? 156 + 3. **backwards compatibility?** can old code read new schema? 157 + 4. **testing?** how to test migrations in CI? 158 + 159 + ## references 160 + 161 + - [zmig](https://github.com/Jeansidharta/zmig) - zig sqlite migrations 162 + - [zigmigrate](https://github.com/eugenepentland/zigmigrate) - zig sqlite migrations (goose-inspired) 163 + - [atlas](https://atlasgo.io/) - schema-as-code tool 164 + - [flyway](https://www.bytebase.com/blog/flyway-vs-liquibase/) - java migration tool 165 + - python prefect: `~/github.com/prefecthq/prefect/src/prefect/server/database/_migrations/`