teal.fm / teal
Your music, beautifully tracked. All yours. (coming soon) teal.fm
teal-fm atproto
fork atom

clean up

Natalie B. 21121e7d 363fa5b9

-564
unified split
-209
.github/SQLX_AND_CROSS_COMPILATION.md
··· 1 - # SQLx Offline and Cross-Compilation Setup 2 - 3 - This document explains the configuration changes made to support SQLx offline builds and fix cross-compilation issues. 4 - 5 - ## Problems Solved 6 - 7 - ### 1. SQLx Offline Compilation 8 - When `SQLX_OFFLINE=true`, each Rust project needs access to `.sqlx` query metadata files in their local directory. Previously, these files only existed at the workspace root, causing builds to fail outside Docker. 9 - 10 - ### 2. Cross-Compilation OpenSSL Issues 11 - Cross-compilation was failing due to OpenSSL dependencies being pulled in by various crates, which is notoriously difficult to cross-compile. 12 - 13 - ## Solutions Implemented 14 - 15 - ### SQLx Offline Setup 16 - 17 - #### Script: `scripts/setup-sqlx-offline.sh` 18 - - Automatically copies `.sqlx` files from workspace root to each SQLx-dependent project 19 - - Identifies projects that use SQLx by checking `Cargo.toml` files 20 - - Projects that receive `.sqlx` files: 21 - - `apps/aqua` 22 - - `services/cadet` 23 - - `services/satellite` 24 - 25 - #### CI Integration 26 - The script is now called in all CI jobs that build Rust code: 27 - - `setup-and-build` job 28 - - `rust-cross-compile` job 29 - - `rust-quality` job 30 - - `security-audit` job 31 - 32 - ### Cross-Compilation Fixes 33 - 34 - #### 1. Replaced OpenSSL with rustls 35 - Updated workspace dependencies to use rustls instead of OpenSSL: 36 - 37 - ```toml 38 - # Root Cargo.toml 39 - tokio = { version = "1.0", features = [ 40 - "rt-multi-thread", 41 - "macros", 42 - "time", # Required for tokio::time module 43 - "net", # Required for networking 44 - "sync", # Required for synchronization primitives 45 - ] } 46 - 47 - sqlx = { version = "0.8", features = [ 48 - "runtime-tokio", 49 - "postgres", 50 - "uuid", 51 - "tls-rustls", # Instead of default OpenSSL 52 - ] } 53 - 54 - reqwest = { version = "0.12", default-features = false, features = [ 55 - "json", 56 - "rustls-tls", # Instead of default native-tls 57 - "stream", 58 - "gzip", 59 - ] } 60 - 61 - tokio-tungstenite = { version = "*", default-features = false, features = [ 62 - "rustls-tls-webpki-roots", # Instead of default native-tls 63 - "connect", # For connect_async function 64 - "handshake", # For accept_async function (tests) 65 - ] } 66 - ``` 67 - 68 - #### 2. Fixed Workspace Dependency Conflicts 69 - The `services/Cargo.toml` was overriding workspace dependencies with different configurations. Fixed by: 70 - - Changing `reqwest = { version = "0.12", features = ["json"] }` to `reqwest.workspace = true` 71 - - Changing `tokio-tungstenite = "0.24"` to `tokio-tungstenite.workspace = true` 72 - 73 - #### 3. Enhanced Cross.toml Configuration 74 - Created comprehensive `Cross.toml` files for cross-compilation: 75 - 76 - ```toml 77 - [build.env] 78 - passthrough = [ 79 - "CARGO_HOME", 80 - "CARGO_TARGET_DIR", 81 - "SQLX_OFFLINE", 82 - "PKG_CONFIG_ALLOW_CROSS", 83 - ] 84 - 85 - [target.aarch64-unknown-linux-gnu] 86 - image = "ghcr.io/cross-rs/aarch64-unknown-linux-gnu:main" 87 - 88 - [target.aarch64-unknown-linux-gnu.env] 89 - passthrough = ["CARGO_HOME", "CARGO_TARGET_DIR", "SQLX_OFFLINE"] 90 - PKG_CONFIG_ALLOW_CROSS = "1" 91 - RUSTFLAGS = "-C target-feature=+crt-static -C link-arg=-s" 92 - CC_aarch64_unknown_linux_gnu = "aarch64-linux-gnu-gcc" 93 - CXX_aarch64_unknown_linux_gnu = "aarch64-linux-gnu-g++" 94 - ``` 95 - 96 - #### 4. Improved CI Cross-Compilation 97 - - Uses latest `cross` from Git: `cargo install cross --git https://github.com/cross-rs/cross` 98 - - Sets `PKG_CONFIG_ALLOW_CROSS=1` environment variable 99 - - Copies `.sqlx` files to individual service directories during cross-compilation 100 - - Improved executable collection with better filtering 101 - 102 - ### Executable Collection 103 - 104 - Enhanced executable collection in CI: 105 - - Filters out build artifacts (`.d` files and temporary files with `-` in names) 106 - - Handles missing target directories gracefully 107 - - Collects executables for both x86_64 and aarch64 targets 108 - - Provides clear logging of collected executables 109 - 110 - ## File Changes 111 - 112 - ### Created 113 - - `scripts/setup-sqlx-offline.sh` - SQLx offline setup script 114 - - `Cross.toml` - Root cross-compilation config 115 - - `services/Cross.toml` - Services cross-compilation config 116 - - `apps/aqua/Cross.toml` - Aqua cross-compilation config 117 - 118 - ### Modified 119 - - `Cargo.toml` - Updated workspace dependencies to use rustls 120 - - `services/Cargo.toml` - Fixed workspace dependency usage 121 - - `.github/workflows/ci.yml` - Added SQLx setup and improved cross-compilation 122 - 123 - ## Usage 124 - 125 - ### Running SQLx Setup Locally 126 - ```bash 127 - ./scripts/setup-sqlx-offline.sh 128 - ``` 129 - 130 - ### Cross-Compilation Locally 131 - ```bash 132 - # Install cross if not already installed 133 - cargo install cross --git https://github.com/cross-rs/cross 134 - 135 - # Add target 136 - rustup target add aarch64-unknown-linux-gnu 137 - 138 - # Set environment 139 - export PKG_CONFIG_ALLOW_CROSS=1 140 - export SQLX_OFFLINE=true 141 - 142 - # Run SQLx setup 143 - ./scripts/setup-sqlx-offline.sh 144 - 145 - # Cross-compile services 146 - cd services 147 - cross build --release --target aarch64-unknown-linux-gnu 148 - 149 - # Cross-compile apps 150 - cd ../apps/aqua 151 - cross build --release --target aarch64-unknown-linux-gnu 152 - ``` 153 - 154 - ### Updating SQLx Queries 155 - When you add or modify SQL queries: 156 - 157 - 1. Generate new query metadata from the services directory: 158 - ```bash 159 - cd services 160 - cargo sqlx prepare 161 - ``` 162 - 163 - 2. Update all project copies: 164 - ```bash 165 - ./scripts/setup-sqlx-offline.sh 166 - ``` 167 - 168 - ## Troubleshooting 169 - 170 - ### Cross-Compilation Still Fails 171 - - Ensure you're using the latest `cross` from Git 172 - - Check that `PKG_CONFIG_ALLOW_CROSS=1` is set 173 - - Verify no dependencies are pulling in OpenSSL (use `cargo tree | grep openssl`) 174 - 175 - ### tokio-tungstenite Import Errors 176 - If you see "unresolved import `tokio_tungstenite::connect_async`" errors: 177 - - Ensure the `connect` feature is enabled for client functionality 178 - - Add the `handshake` feature if using `accept_async` for server functionality 179 - - Check that `default-features = false` is set to avoid OpenSSL dependencies 180 - 181 - ### tokio Module Errors 182 - If you see "could not find `time` in `tokio`" or similar errors: 183 - - Ensure tokio workspace dependency includes required features: `time`, `net`, `sync` 184 - - These features are often included in default features but must be explicit when using `default-features = false` 185 - 186 - ### SQLx Offline Errors 187 - - Run `./scripts/setup-sqlx-offline.sh` after any SQL query changes 188 - - Ensure `.sqlx` directory exists in workspace root 189 - - Check that `SQLX_OFFLINE=true` is set in CI environment 190 - 191 - ### Missing Executables 192 - - Check that build succeeded without errors 193 - - Verify executable names match expected service/app names 194 - - Look for executables in `artifacts/` directory structure 195 - 196 - ## Dependencies Avoided 197 - To maintain cross-compilation compatibility, avoid dependencies that: 198 - - Default to OpenSSL (use rustls variants) 199 - - Require system libraries not available in cross containers 200 - - Have complex native build requirements 201 - 202 - ### Common Feature Configuration Issues 203 - When disabling default features to avoid OpenSSL: 204 - - `tokio`: Must explicitly enable `time`, `net`, `sync` features for common functionality 205 - - `tokio-tungstenite`: Must explicitly enable `connect` and `handshake` features 206 - - `reqwest`: Must explicitly enable required features like `json`, `stream`, `gzip` 207 - - `sqlx`: Use `tls-rustls` instead of default OpenSSL TLS 208 - 209 - Always test cross-compilation locally before pushing changes.
-355
docs/migration-troubleshooting.md
··· 1 - # Migration Troubleshooting Guide 2 - 3 - ## Common Migration Issues and Solutions 4 - 5 - ### Issue: "cannot drop function because other objects depend on it" 6 - 7 - **Error Message:** 8 - ``` 9 - error: while executing migration 20241220000008: error returned from database: cannot drop function extract_discriminant(text) because other objects depend on it 10 - ``` 11 - 12 - **Cause:** 13 - This error occurs when trying to drop database functions that have dependent objects (views, other functions, triggers, etc.) without properly handling the dependencies. 14 - 15 - **Solution:** 16 - 17 - #### Option 1: Fix the Migration (Recommended) 18 - Update the problematic migration to handle dependencies properly: 19 - 20 - 1. **Edit the migration file** (e.g., `20241220000008_fix_discriminant_case_sensitivity.sql`): 21 - 22 - ```sql 23 - -- Drop dependent views first, then functions, then recreate everything 24 - DROP VIEW IF EXISTS discriminant_analysis CASCADE; 25 - DROP VIEW IF EXISTS discriminant_stats CASCADE; 26 - 27 - -- Drop existing functions with CASCADE to handle dependencies 28 - DROP FUNCTION IF EXISTS extract_discriminant(TEXT) CASCADE; 29 - DROP FUNCTION IF EXISTS get_base_name(TEXT) CASCADE; 30 - DROP FUNCTION IF EXISTS extract_edition_discriminant(TEXT) CASCADE; 31 - 32 - -- Then recreate functions and views... 33 - ``` 34 - 35 - 2. **Reset the migration state** if the migration was partially applied: 36 - 37 - ```bash 38 - # Connect to your database and reset the specific migration 39 - psql $DATABASE_URL -c "DELETE FROM _sqlx_migrations WHERE version = '20241220000008';" 40 - 41 - # Or reset all migrations and start fresh (WARNING: This drops all data) 42 - psql $DATABASE_URL -c "DROP SCHEMA public CASCADE; CREATE SCHEMA public;" 43 - ``` 44 - 45 - 3. **Run migrations again**: 46 - ```bash 47 - cd services 48 - DATABASE_URL="your_database_url" sqlx migrate run 49 - ``` 50 - 51 - #### Option 2: Manual Dependency Cleanup 52 - If you can't modify the migration file: 53 - 54 - 1. **Identify dependencies**: 55 - ```sql 56 - -- Find objects that depend on the function 57 - SELECT 58 - p.proname as function_name, 59 - d.objid, 60 - d.classid::regclass as object_type, 61 - d.refobjid 62 - FROM pg_depend d 63 - JOIN pg_proc p ON d.refobjid = p.oid 64 - WHERE p.proname = 'extract_discriminant'; 65 - ``` 66 - 67 - 2. **Drop dependencies manually**: 68 - ```sql 69 - -- Drop dependent views 70 - DROP VIEW IF EXISTS discriminant_analysis CASCADE; 71 - DROP VIEW IF EXISTS discriminant_stats CASCADE; 72 - DROP VIEW IF EXISTS track_variants CASCADE; 73 - DROP VIEW IF EXISTS release_variants CASCADE; 74 - 75 - -- Drop the functions 76 - DROP FUNCTION IF EXISTS extract_discriminant(TEXT) CASCADE; 77 - DROP FUNCTION IF EXISTS get_base_name(TEXT) CASCADE; 78 - DROP FUNCTION IF EXISTS extract_edition_discriminant(TEXT) CASCADE; 79 - ``` 80 - 81 - 3. **Continue with migration**: 82 - ```bash 83 - DATABASE_URL="your_database_url" sqlx migrate run 84 - ``` 85 - 86 - ### Issue: "migration was previously applied but has been modified" 87 - 88 - **Error Message:** 89 - ``` 90 - error: migration 20241220000008 was previously applied but has been modified 91 - ``` 92 - 93 - **Cause:** 94 - The migration file has been changed after it was already applied to the database. 95 - 96 - **Solutions:** 97 - 98 - #### Option 1: Reset Migration State 99 - ```bash 100 - # Remove the specific migration from tracking 101 - psql $DATABASE_URL -c "DELETE FROM _sqlx_migrations WHERE version = '20241220000008';" 102 - 103 - # Run migrations again 104 - DATABASE_URL="your_database_url" sqlx migrate run 105 - ``` 106 - 107 - #### Option 2: Create a New Migration 108 - ```bash 109 - # Create a new migration with your changes 110 - sqlx migrate add fix_discriminant_case_sensitivity_v2 111 - 112 - # Copy your changes to the new migration file 113 - # Run the new migration 114 - DATABASE_URL="your_database_url" sqlx migrate run 115 - ``` 116 - 117 - #### Option 3: Full Reset (WARNING: Destroys all data) 118 - ```bash 119 - # Connect to database and reset everything 120 - psql $DATABASE_URL -c "DROP SCHEMA public CASCADE; CREATE SCHEMA public;" 121 - 122 - # Run all migrations from scratch 123 - DATABASE_URL="your_database_url" sqlx migrate run 124 - ``` 125 - 126 - ### Issue: "No such file or directory" when running migrations 127 - 128 - **Error Message:** 129 - ``` 130 - error: while resolving migrations: No such file or directory (os error 2) 131 - ``` 132 - 133 - **Cause:** 134 - The migration directory is not found in the expected location. 135 - 136 - **Solutions:** 137 - 138 - #### Option 1: Check Migration Directory Location 139 - ```bash 140 - # Check where sqlx expects migrations 141 - cat services/.sqlx/.sqlxrc 142 - 143 - # Ensure migrations exist in the correct location 144 - ls -la services/migrations/ 145 - ``` 146 - 147 - #### Option 2: Copy Migrations to Correct Location 148 - ```bash 149 - # If migrations are in wrong location, copy them 150 - cp migrations/*.sql services/migrations/ 151 - 152 - # Or create symlink 153 - ln -s ../migrations services/migrations 154 - ``` 155 - 156 - #### Option 3: Update sqlx Configuration 157 - Edit `services/.sqlx/.sqlxrc`: 158 - ```toml 159 - [database] 160 - url = "postgres://localhost/teal" 161 - migrations = "../migrations" # Update path as needed 162 - ``` 163 - 164 - ### Issue: Database Connection Problems 165 - 166 - **Error Messages:** 167 - - `Connection refused (os error 61)` 168 - - `password authentication failed` 169 - - `database "teal_test" does not exist` 170 - 171 - **Solutions:** 172 - 173 - #### Connection Refused 174 - ```bash 175 - # Check if database is running 176 - docker ps | grep postgres 177 - 178 - # Start database if needed 179 - docker-compose -f compose.db-test.yml up -d 180 - 181 - # Wait for database to start 182 - sleep 5 183 - ``` 184 - 185 - #### Authentication Issues 186 - ```bash 187 - # Check connection string format 188 - DATABASE_URL="postgres://username:password@host:port/database" 189 - 190 - # Example for test database 191 - DATABASE_URL="postgres://postgres:testpass123@localhost:5433/teal_test" 192 - ``` 193 - 194 - #### Database Doesn't Exist 195 - ```bash 196 - # Create database 197 - docker exec postgres_container psql -U postgres -c "CREATE DATABASE teal_test;" 198 - 199 - # Or recreate test environment 200 - docker-compose -f compose.db-test.yml down 201 - docker-compose -f compose.db-test.yml up -d 202 - ``` 203 - 204 - ## Migration Best Practices 205 - 206 - ### 1. Handle Dependencies Properly 207 - Always use `CASCADE` when dropping objects with dependencies: 208 - ```sql 209 - DROP FUNCTION function_name(args) CASCADE; 210 - DROP VIEW view_name CASCADE; 211 - ``` 212 - 213 - ### 2. Test Migrations Locally 214 - ```bash 215 - # Use test database for migration testing 216 - DATABASE_URL="postgres://localhost:5433/teal_test" sqlx migrate run 217 - 218 - # Verify results 219 - psql "postgres://localhost:5433/teal_test" -c "SELECT extract_discriminant('Test (Example)');" 220 - ``` 221 - 222 - ### 3. Backup Before Major Migrations 223 - ```bash 224 - # Create backup 225 - pg_dump $DATABASE_URL > backup_before_migration.sql 226 - 227 - # Apply migrations 228 - sqlx migrate run 229 - 230 - # Restore if needed 231 - psql $DATABASE_URL < backup_before_migration.sql 232 - ``` 233 - 234 - ### 4. Version Control Migration Files 235 - - Never modify applied migrations 236 - - Create new migrations for changes 237 - - Use descriptive migration names 238 - - Include rollback instructions in comments 239 - 240 - ### 5. Migration File Structure 241 - ```sql 242 - -- Migration: descriptive_name 243 - -- Purpose: Brief description of what this migration does 244 - -- Dependencies: List any required prior migrations 245 - -- Rollback: Instructions for manual rollback if needed 246 - 247 - -- Drop dependencies first 248 - DROP VIEW IF EXISTS dependent_view CASCADE; 249 - 250 - -- Make changes 251 - CREATE OR REPLACE FUNCTION new_function() ...; 252 - 253 - -- Recreate dependencies 254 - CREATE VIEW dependent_view AS ...; 255 - 256 - -- Update existing data if needed 257 - UPDATE table_name SET column = new_value WHERE condition; 258 - 259 - -- Add comments 260 - COMMENT ON FUNCTION new_function IS 'Description of function purpose'; 261 - ``` 262 - 263 - ## Emergency Recovery 264 - 265 - ### Complete Database Reset 266 - If migrations are completely broken: 267 - 268 - ```bash 269 - # 1. Stop all services 270 - docker-compose down 271 - 272 - # 2. Remove database volume (WARNING: Destroys all data) 273 - docker volume rm teal_postgres_data 274 - 275 - # 3. Start fresh 276 - docker-compose up -d postgres 277 - 278 - # 4. Wait for database to initialize 279 - sleep 10 280 - 281 - # 5. Run all migrations from scratch 282 - DATABASE_URL="your_database_url" sqlx migrate run 283 - ``` 284 - 285 - ### Partial Recovery 286 - If only discriminant system is broken: 287 - 288 - ```sql 289 - -- Remove discriminant-related objects 290 - DROP VIEW IF EXISTS discriminant_analysis CASCADE; 291 - DROP VIEW IF EXISTS discriminant_stats CASCADE; 292 - DROP VIEW IF EXISTS track_variants CASCADE; 293 - DROP VIEW IF EXISTS release_variants CASCADE; 294 - DROP FUNCTION IF EXISTS extract_discriminant(TEXT) CASCADE; 295 - DROP FUNCTION IF EXISTS get_base_name(TEXT) CASCADE; 296 - DROP FUNCTION IF EXISTS extract_edition_discriminant(TEXT) CASCADE; 297 - 298 - -- Remove discriminant columns 299 - ALTER TABLE plays DROP COLUMN IF EXISTS track_discriminant; 300 - ALTER TABLE plays DROP COLUMN IF EXISTS release_discriminant; 301 - ALTER TABLE recordings DROP COLUMN IF EXISTS discriminant; 302 - ALTER TABLE releases DROP COLUMN IF EXISTS discriminant; 303 - 304 - -- Mark discriminant migrations as not applied 305 - DELETE FROM _sqlx_migrations WHERE version >= '20241220000006'; 306 - 307 - -- Re-run discriminant migrations 308 - ``` 309 - 310 - ## Getting Help 311 - 312 - ### Debug Information to Collect 313 - When reporting migration issues, include: 314 - 315 - 1. **Error message** (full stack trace) 316 - 2. **Migration file content** that's causing issues 317 - 3. **Database state**: 318 - ```sql 319 - SELECT version FROM _sqlx_migrations ORDER BY version; 320 - \df extract_discriminant 321 - \dv discriminant_* 322 - ``` 323 - 4. **Environment details**: 324 - - Database version: `SELECT version();` 325 - - Operating system 326 - - sqlx version: `cargo sqlx --version` 327 - 328 - ### Useful Debugging Commands 329 - ```sql 330 - -- Check applied migrations 331 - SELECT * FROM _sqlx_migrations ORDER BY version; 332 - 333 - -- Check function definitions 334 - \df+ extract_discriminant 335 - 336 - -- Check view definitions 337 - \d+ discriminant_analysis 338 - 339 - -- Check table schemas 340 - \d+ plays 341 - \d+ recordings 342 - \d+ releases 343 - 344 - -- Test function directly 345 - SELECT extract_discriminant('Test (Example)'); 346 - ``` 347 - 348 - ## Contact and Support 349 - 350 - For persistent migration issues: 351 - 1. Check this troubleshooting guide first 352 - 2. Review the specific migration file causing issues 353 - 3. Try solutions in order of preference (fix migration → manual cleanup → reset) 354 - 4. Create minimal reproduction case for complex issues 355 - 5. Document exact steps that led to the error for support requests