slices.network / quickslice
Auto-indexing service and GraphQL API for AT Protocol Records quickslice.slices.network/
atproto gleam graphql
fork atom

docs: add rkey notification sorting implementation plan

chadtmiller.com bdb0de61 62aeb7c4

+396
unified split
+396
dev-docs/plans/2025-12-26-rkey-notification-sorting.md
··· 1 + # Rkey-Based Notification Sorting Implementation Plan 2 + 3 + > **For Claude:** REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task. 4 + 5 + **Goal:** Sort notifications by TID (timestamp identifier) extracted from record rkey for chronological ordering across collections. 6 + 7 + **Architecture:** Add a generated `rkey` column to the record table that extracts the last path segment from the URI. Use this for sorting notifications instead of `indexed_at`, since TIDs are lexicographically sortable and encode creation timestamps. 8 + 9 + **Tech Stack:** Gleam, SQLite, GraphQL (swell) 10 + 11 + --- 12 + 13 + ### Task 1: Add Migration for rkey Generated Column 14 + 15 + **Files:** 16 + - Create: `server/db/migrations/20241227000001_add_rkey_column.sql` 17 + 18 + **Step 1: Create the migration file** 19 + 20 + ```sql 21 + -- migrate:up 22 + 23 + -- Add rkey as a generated column extracted from uri 24 + -- URI format: at://did/collection/rkey 25 + -- We extract everything after the last '/' 26 + ALTER TABLE record ADD COLUMN rkey TEXT 27 + GENERATED ALWAYS AS ( 28 + substr(uri, instr(substr(uri, instr(substr(uri, 6), '/') + 6), '/') + instr(substr(uri, 6), '/') + 6) 29 + ) STORED; 30 + 31 + -- Index for efficient sorting by rkey (TID-based chronological order) 32 + CREATE INDEX IF NOT EXISTS idx_record_rkey ON record(rkey DESC); 33 + 34 + -- migrate:down 35 + 36 + DROP INDEX IF EXISTS idx_record_rkey; 37 + -- Note: SQLite doesn't support DROP COLUMN, would need table rebuild 38 + ``` 39 + 40 + **Step 2: Test the SQL extraction logic manually** 41 + 42 + Run in SQLite to verify the extraction works: 43 + ```sql 44 + SELECT 45 + uri, 46 + substr(uri, instr(substr(uri, instr(substr(uri, 6), '/') + 6), '/') + instr(substr(uri, 6), '/') + 6) as rkey 47 + FROM ( 48 + SELECT 'at://did:plc:abc123/social.grain.photo/3maungnepas2t' as uri 49 + ); 50 + ``` 51 + 52 + Expected: `3maungnepas2t` 53 + 54 + **Step 3: Commit** 55 + 56 + ```bash 57 + git add server/db/migrations/20241227000001_add_rkey_column.sql 58 + git commit -m "feat(db): add rkey generated column for TID-based sorting" 59 + ``` 60 + 61 + --- 62 + 63 + ### Task 2: Update Record Type to Include rkey 64 + 65 + **Files:** 66 + - Modify: `server/src/database/types.gleam` 67 + 68 + **Step 1: Add rkey field to Record type** 69 + 70 + In `server/src/database/types.gleam`, update the Record type: 71 + 72 + ```gleam 73 + pub type Record { 74 + Record( 75 + uri: String, 76 + cid: String, 77 + did: String, 78 + collection: String, 79 + json: String, 80 + indexed_at: String, 81 + rkey: String, 82 + ) 83 + } 84 + ``` 85 + 86 + **Step 2: Run build to find all places that need updating** 87 + 88 + ```bash 89 + cd /Users/chadmiller/code/quickslice/server && gleam build 2>&1 90 + ``` 91 + 92 + Expected: Compile errors showing all places constructing Record 93 + 94 + **Step 3: Commit** 95 + 96 + ```bash 97 + git add server/src/database/types.gleam 98 + git commit -m "feat(types): add rkey field to Record type" 99 + ``` 100 + 101 + --- 102 + 103 + ### Task 3: Update Record Decoder and Columns 104 + 105 + **Files:** 106 + - Modify: `server/src/database/repositories/records.gleam` 107 + 108 + **Step 1: Update record_columns function** 109 + 110 + Find `record_columns` and add `rkey`: 111 + 112 + ```gleam 113 + fn record_columns(exec: Executor) -> String { 114 + "uri, cid, did, collection, json, indexed_at, rkey" 115 + } 116 + ``` 117 + 118 + **Step 2: Update record_decoder function** 119 + 120 + Find `record_decoder` and add rkey decoding: 121 + 122 + ```gleam 123 + fn record_decoder() -> decode.Decoder(Record) { 124 + use uri <- decode.field(0, decode.string) 125 + use cid <- decode.field(1, decode.string) 126 + use did <- decode.field(2, decode.string) 127 + use collection <- decode.field(3, decode.string) 128 + use json <- decode.field(4, decode.string) 129 + use indexed_at <- decode.field(5, decode.string) 130 + use rkey <- decode.field(6, decode.string) 131 + decode.success(Record(uri:, cid:, did:, collection:, json:, indexed_at:, rkey:)) 132 + } 133 + ``` 134 + 135 + **Step 3: Fix any remaining Record constructor calls** 136 + 137 + Search for `Record(` and update any direct constructions to include rkey. For test fixtures, extract rkey from uri: 138 + 139 + ```gleam 140 + // Helper to extract rkey from uri for tests 141 + fn extract_rkey(uri: String) -> String { 142 + uri 143 + |> string.split("/") 144 + |> list.last 145 + |> result.unwrap("") 146 + } 147 + ``` 148 + 149 + **Step 4: Run tests to verify** 150 + 151 + ```bash 152 + cd /Users/chadmiller/code/quickslice/server && gleam test 2>&1 | tail -20 153 + ``` 154 + 155 + **Step 5: Commit** 156 + 157 + ```bash 158 + git add server/src/database/repositories/records.gleam 159 + git commit -m "feat(records): update decoder to include rkey column" 160 + ``` 161 + 162 + --- 163 + 164 + ### Task 4: Update get_notifications to Sort by rkey 165 + 166 + **Files:** 167 + - Modify: `server/src/database/repositories/records.gleam` 168 + 169 + **Step 1: Update ORDER BY clause** 170 + 171 + In `get_notifications` function (around line 1143), change: 172 + 173 + ```gleam 174 + // Before: 175 + <> " ORDER BY indexed_at DESC, uri DESC LIMIT " 176 + 177 + // After: 178 + <> " ORDER BY rkey DESC, uri DESC LIMIT " 179 + ``` 180 + 181 + **Step 2: Update cursor WHERE clause** 182 + 183 + In the cursor clause building (around line 1118), change: 184 + 185 + ```gleam 186 + // Before: 187 + #(" AND (indexed_at, uri) < (" <> p1 <> ", " <> p2 <> ")", [ 188 + Text(indexed_at_value), 189 + Text(cid_value), 190 + ]) 191 + 192 + // After: 193 + #(" AND (rkey, uri) < (" <> p1 <> ", " <> p2 <> ")", [ 194 + Text(rkey_value), 195 + Text(uri_value), 196 + ]) 197 + ``` 198 + 199 + Note: The cursor now uses `(rkey, uri)` tuple instead of `(indexed_at, cid)`. 200 + 201 + **Step 3: Update cursor decoding** 202 + 203 + Update the cursor value extraction: 204 + 205 + ```gleam 206 + // The cursor format is now: rkey|uri (encoded as base64) 207 + let rkey_value = 208 + decoded.field_values |> list.first |> result.unwrap("") 209 + let uri_value = decoded.cid // cid field now holds uri for this cursor 210 + ``` 211 + 212 + **Step 4: Run tests** 213 + 214 + ```bash 215 + cd /Users/chadmiller/code/quickslice/server && gleam test 2>&1 | tail -20 216 + ``` 217 + 218 + **Step 5: Commit** 219 + 220 + ```bash 221 + git add server/src/database/repositories/records.gleam 222 + git commit -m "feat(notifications): sort by rkey for TID-based chronological order" 223 + ``` 224 + 225 + --- 226 + 227 + ### Task 5: Update Notification Cursor Generation 228 + 229 + **Files:** 230 + - Modify: `server/src/database/repositories/records.gleam` 231 + 232 + **Step 1: Update end_cursor generation** 233 + 234 + In `get_notifications` (around line 1156-1162), update cursor generation: 235 + 236 + ```gleam 237 + let end_cursor = case list.last(trimmed) { 238 + Ok(record) -> { 239 + // Encode cursor as rkey|uri for notification pagination 240 + let cursor_content = record.rkey <> "|" <> record.uri 241 + Some(pagination.encode_base64(cursor_content)) 242 + } 243 + Error(_) -> None 244 + } 245 + ``` 246 + 247 + **Step 2: Run tests** 248 + 249 + ```bash 250 + cd /Users/chadmiller/code/quickslice/server && gleam test 2>&1 | tail -20 251 + ``` 252 + 253 + **Step 3: Commit** 254 + 255 + ```bash 256 + git add server/src/database/repositories/records.gleam 257 + git commit -m "feat(notifications): update cursor to use rkey|uri format" 258 + ``` 259 + 260 + --- 261 + 262 + ### Task 6: Add rkey to Pagination Field Extraction 263 + 264 + **Files:** 265 + - Modify: `server/src/database/queries/pagination.gleam` 266 + 267 + **Step 1: Add rkey to extract_field_value** 268 + 269 + Update the `extract_field_value` function: 270 + 271 + ```gleam 272 + pub fn extract_field_value(record: Record, field: String) -> String { 273 + case field { 274 + "uri" -> record.uri 275 + "cid" -> record.cid 276 + "did" -> record.did 277 + "collection" -> record.collection 278 + "indexed_at" -> record.indexed_at 279 + "rkey" -> record.rkey 280 + _ -> extract_json_field(record.json, field) 281 + } 282 + } 283 + ``` 284 + 285 + **Step 2: Add rkey to build_cursor_field_reference** 286 + 287 + ```gleam 288 + fn build_cursor_field_reference(exec: Executor, field: String) -> String { 289 + case field { 290 + "uri" | "cid" | "did" | "collection" | "indexed_at" | "rkey" -> field 291 + _ -> executor.json_extract(exec, "json", field) 292 + } 293 + } 294 + ``` 295 + 296 + **Step 3: Add rkey to build_order_by** 297 + 298 + ```gleam 299 + // In build_order_by, update the field_ref case: 300 + let field_ref = case field_name { 301 + "uri" | "cid" | "did" | "collection" | "indexed_at" | "rkey" -> 302 + table_prefix <> field_name 303 + // ... rest unchanged 304 + } 305 + ``` 306 + 307 + **Step 4: Run tests** 308 + 309 + ```bash 310 + cd /Users/chadmiller/code/quickslice/server && gleam test 2>&1 | tail -20 311 + ``` 312 + 313 + **Step 5: Commit** 314 + 315 + ```bash 316 + git add server/src/database/queries/pagination.gleam 317 + git commit -m "feat(pagination): add rkey field support" 318 + ``` 319 + 320 + --- 321 + 322 + ### Task 7: Update Notification E2E Tests 323 + 324 + **Files:** 325 + - Modify: `server/test/graphql/notifications_e2e_test.gleam` 326 + 327 + **Step 1: Update test fixtures to use TID-like rkeys** 328 + 329 + Ensure test records use TID-format rkeys so sorting is predictable: 330 + 331 + ```gleam 332 + // Use TIDs that sort in a known order 333 + // Earlier TID (should appear second when sorted DESC) 334 + let earlier_rkey = "3kaungnepas2t" 335 + // Later TID (should appear first when sorted DESC) 336 + let later_rkey = "3maungnepas2t" 337 + ``` 338 + 339 + **Step 2: Update assertions to verify rkey-based ordering** 340 + 341 + ```gleam 342 + // Records should be ordered by rkey DESC 343 + // So later_rkey record should come before earlier_rkey record 344 + ``` 345 + 346 + **Step 3: Run notification tests** 347 + 348 + ```bash 349 + cd /Users/chadmiller/code/quickslice/server && gleam test 2>&1 | grep -A5 "notification" 350 + ``` 351 + 352 + **Step 4: Commit** 353 + 354 + ```bash 355 + git add server/test/graphql/notifications_e2e_test.gleam 356 + git commit -m "test(notifications): update e2e tests for rkey-based sorting" 357 + ``` 358 + 359 + --- 360 + 361 + ### Task 8: Run Full Test Suite and Verify 362 + 363 + **Step 1: Run all tests** 364 + 365 + ```bash 366 + cd /Users/chadmiller/code/quickslice/server && gleam test 2>&1 367 + ``` 368 + 369 + Expected: All tests pass 370 + 371 + **Step 2: Verify migration works on fresh database** 372 + 373 + ```bash 374 + cd /Users/chadmiller/code/quickslice/server && rm -f quickslice.db && gleam run -- migrate 375 + ``` 376 + 377 + **Step 3: Final commit if any fixes needed** 378 + 379 + ```bash 380 + git add -A 381 + git commit -m "fix: address test failures from rkey sorting migration" 382 + ``` 383 + 384 + --- 385 + 386 + ## Summary 387 + 388 + After completing these tasks: 389 + 390 + 1. **Records table** has a generated `rkey` column that extracts the last URI segment 391 + 2. **Notifications** are sorted by `rkey DESC` instead of `indexed_at DESC` 392 + 3. **Cursor pagination** uses `(rkey, uri)` tuple for stable pagination 393 + 4. **TID-based records** (majority) sort chronologically by creation time 394 + 5. **Non-TID rkeys** sort lexicographically (deterministic, just not chronological) 395 + 396 + This provides chronological notification ordering across collections without relying on arbitrary `indexed_at` timestamps.