microcosm.blue / microcosm-rs
Constellation, Spacedust, Slingshot, UFOs: atproto crates and services for microcosm
fork atom

Convert existing REST /links/count endpoint to XRPC equivalent #8

closed opened by seoul.systems targeting main from seoul.systems/microcosm-rs: xrpc_backlinks_count

Add getCounts XRPC equivalent to REST /links/count

Simple conversion of the existing endpoint from REST to XRPC. Consequtively marked the pre-existing REST endpoint as deprecated.

In addition we now ignore rocks.test

Labels

None yet.

Participants 2
AT URI
at://did:plc:53wellrw53o7sw4zlpfenvuh/sh.tangled.repo.pull/3mcybnzocig22
+317 -161
Interdiff #3 → #4
.gitignore

This file has not been changed.

.prettierrc

This file has not been changed.

+12 -3
constellation/src/server/mod.rs
··· 17 17 use tokio::task::spawn_blocking; 18 18 use tokio_util::sync::CancellationToken; 19 19 20 - use crate::storage::{LinkReader, StorageStats}; 20 + use crate::storage::{LinkReader, Order, StorageStats}; 21 21 use crate::{CountsByCount, Did, RecordId}; 22 22 23 23 mod acceptable; ··· 456 456 /// Set the max number of links to return per page of results 457 457 #[serde(default = "get_default_cursor_limit")] 458 458 limit: u64, 459 - // TODO: allow reverse (er, forward) order as well 459 + /// Allow returning links in reverse order (default: false) 460 + #[serde(default)] 461 + reverse: bool, 460 462 } 461 463 #[derive(Template, Serialize)] 462 464 #[template(path = "get-backlinks.html.j2")] ··· 502 504 }; 503 505 let path = format!(".{path}"); 504 506 507 + let order = if query.reverse { 508 + Order::OldestToNewest 509 + } else { 510 + Order::NewestToOldest 511 + }; 512 + 505 513 let paged = store 506 514 .get_links( 507 515 &query.subject, 508 516 collection, 509 517 &path, 518 + order, 510 519 limit, 511 520 until, 512 521 &filter_dids, ··· 555 564 from_dids: Option<String>, // comma separated: gross 556 565 #[serde(default = "get_default_cursor_limit")] 557 566 limit: u64, 558 - // TODO: allow reverse (er, forward) order as well 559 567 } 560 568 #[derive(Template, Serialize)] 561 569 #[template(path = "links.html.j2")] ··· 609 617 &query.target, 610 618 &query.collection, 611 619 &query.path, 620 + Order::NewestToOldest, 612 621 limit, 613 622 until, 614 623 &filter_dids,
+55 -53
constellation/src/storage/mem_store.rs
··· 1 1 use super::{ 2 + LinkReader, LinkStorage, Order, PagedAppendingCollection, PagedOrderedCollection, StorageStats, 2 - LinkReader, LinkStorage, PagedAppendingCollection, PagedOrderedCollection, StorageStats, 3 3 }; 4 4 use crate::{ActionableEvent, CountsByCount, Did, RecordId}; 5 5 use anyhow::Result; ··· 147 147 ) -> Result<PagedOrderedCollection<(String, u64, u64), String>> { 148 148 let data = self.0.lock().unwrap(); 149 149 let Some(paths) = data.targets.get(&Target::new(target)) else { 150 + return Ok(PagedOrderedCollection::empty()); 150 - return Ok(PagedOrderedCollection::default()); 151 151 }; 152 152 let Some(linkers) = paths.get(&Source::new(collection, path)) else { 153 + return Ok(PagedOrderedCollection::empty()); 153 - return Ok(PagedOrderedCollection::default()); 154 154 }; 155 155 156 156 let path_to_other = RecordPath::new(path_to_other); ··· 239 239 target: &str, 240 240 collection: &str, 241 241 path: &str, 242 + order: Order, 242 243 limit: u64, 244 + until: Option<u64>, // paged iteration endpoint 243 - until: Option<u64>, 244 245 filter_dids: &HashSet<Did>, 245 246 ) -> Result<PagedAppendingCollection<RecordId>> { 246 247 let data = self.0.lock().unwrap(); 247 248 let Some(paths) = data.targets.get(&Target::new(target)) else { 249 + return Ok(PagedAppendingCollection::empty()); 248 - return Ok(PagedAppendingCollection { 249 - version: (0, 0), 250 - items: Vec::new(), 251 - next: None, 252 - total: 0, 253 - }); 254 250 }; 255 251 let Some(did_rkeys) = paths.get(&Source::new(collection, path)) else { 252 + return Ok(PagedAppendingCollection::empty()); 256 - return Ok(PagedAppendingCollection { 257 - version: (0, 0), 258 - items: Vec::new(), 259 - next: None, 260 - total: 0, 261 - }); 262 253 }; 263 254 264 255 let did_rkeys: Vec<_> = if !filter_dids.is_empty() { ··· 275 266 did_rkeys.to_vec() 276 267 }; 277 268 269 + let total = did_rkeys.len() as u64; 270 + 271 + // backlinks are stored oldest-to-newest (ascending index with increasing age) 272 + let (start, take, next_until) = match order { 273 + Order::OldestToNewest => { 274 + let start = until.unwrap_or(0); 275 + let next = start + limit + 1; 276 + let next_until = if next < total { Some(next) } else { None }; 277 + (start, limit, next_until) 278 + } 279 + Order::NewestToOldest => { 280 + let until = until.unwrap_or(total); 281 + match until.checked_sub(limit) { 282 + Some(s) if s > 0 => (s, limit, Some(s)), 283 + Some(s) => (s, limit, None), 284 + None => (0, until, None), 285 + } 286 + } 287 + }; 278 - let total = did_rkeys.len(); 279 - let end = until 280 - .map(|u| std::cmp::min(u as usize, total)) 281 - .unwrap_or(total); 282 - let begin = end.saturating_sub(limit as usize); 283 - let next = if begin == 0 { None } else { Some(begin as u64) }; 284 288 289 + let alive = did_rkeys.iter().flatten().count() as u64; 285 - let alive = did_rkeys.iter().flatten().count(); 286 290 let gone = total - alive; 287 291 292 + let items = did_rkeys 288 - let items: Vec<_> = did_rkeys[begin..end] 289 293 .iter() 294 + .skip(start as usize) 295 + .take(take as usize) 290 - .rev() 291 296 .flatten() 292 297 .filter(|(did, _)| *data.dids.get(did).expect("did must be in dids")) 293 298 .map(|(did, rkey)| RecordId { 294 299 did: did.clone(), 295 300 rkey: rkey.0.clone(), 296 301 collection: collection.to_string(), 302 + }); 303 + 304 + let items: Vec<_> = match order { 305 + Order::OldestToNewest => items.collect(), // links are stored oldest first 306 + Order::NewestToOldest => items.rev().collect(), 307 + }; 297 - }) 298 - .collect(); 299 308 300 309 Ok(PagedAppendingCollection { 310 + version: (total, gone), 301 - version: (total as u64, gone as u64), 302 311 items, 312 + next: next_until, 313 + total: alive, 303 - next, 304 - total: alive as u64, 305 314 }) 306 315 } 307 316 ··· 315 324 ) -> Result<PagedAppendingCollection<Did>> { 316 325 let data = self.0.lock().unwrap(); 317 326 let Some(paths) = data.targets.get(&Target::new(target)) else { 327 + return Ok(PagedAppendingCollection::empty()); 318 - return Ok(PagedAppendingCollection { 319 - version: (0, 0), 320 - items: Vec::new(), 321 - next: None, 322 - total: 0, 323 - }); 324 328 }; 325 329 let Some(did_rkeys) = paths.get(&Source::new(collection, path)) else { 330 + return Ok(PagedAppendingCollection::empty()); 326 - return Ok(PagedAppendingCollection { 327 - version: (0, 0), 328 - items: Vec::new(), 329 - next: None, 330 - total: 0, 331 - }); 332 331 }; 333 332 334 333 let dids: Vec<Option<Did>> = { ··· 348 347 .collect() 349 348 }; 350 349 350 + let total = dids.len() as u64; 351 + let until = until.unwrap_or(total); 352 + let (start, take, next_until) = match until.checked_sub(limit) { 353 + Some(s) if s > 0 => (s, limit, Some(s)), 354 + Some(s) => (s, limit, None), 355 + None => (0, until, None), 356 + }; 351 - let total = dids.len(); 352 - let end = until 353 - .map(|u| std::cmp::min(u as usize, total)) 354 - .unwrap_or(total); 355 - let begin = end.saturating_sub(limit as usize); 356 - let next = if begin == 0 { None } else { Some(begin as u64) }; 357 357 358 + let alive = dids.iter().flatten().count() as u64; 358 - let alive = dids.iter().flatten().count(); 359 359 let gone = total - alive; 360 360 361 + let items: Vec<Did> = dids 361 - let items: Vec<Did> = dids[begin..end] 362 362 .iter() 363 + .skip(start as usize) 364 + .take(take as usize) 363 365 .rev() 364 366 .flatten() 365 367 .filter(|did| *data.dids.get(did).expect("did must be in dids")) ··· 367 369 .collect(); 368 370 369 371 Ok(PagedAppendingCollection { 372 + version: (total, gone), 370 - version: (total as u64, gone as u64), 371 373 items, 374 + next: next_until, 375 + total: alive, 372 - next, 373 - total: alive as u64, 374 376 }) 375 377 } 376 378
+195 -76
constellation/src/storage/mod.rs
··· 11 11 #[cfg(feature = "rocks")] 12 12 pub use rocks_store::RocksStorage; 13 13 14 + /// Ordering for paginated link queries 15 + #[derive(Debug, Clone, Copy, PartialEq, Eq)] 16 + pub enum Order { 17 + /// Newest links first (default) 18 + NewestToOldest, 19 + /// Oldest links first 20 + OldestToNewest, 21 + } 22 + 23 + #[derive(Debug, Default, PartialEq)] 14 - #[derive(Debug, PartialEq)] 15 24 pub struct PagedAppendingCollection<T> { 16 25 pub version: (u64, u64), // (collection length, deleted item count) // TODO: change to (total, active)? since dedups isn't "deleted" 17 26 pub items: Vec<T>, ··· 19 28 pub total: u64, 20 29 } 21 30 31 + impl<T> PagedAppendingCollection<T> { 32 + pub(crate) fn empty() -> Self { 33 + Self { 34 + version: (0, 0), 35 + items: Vec::new(), 36 + next: None, 37 + total: 0, 38 + } 39 + } 40 + } 41 + 22 42 /// A paged collection whose keys are sorted instead of indexed 23 43 /// 24 44 /// this has weaker guarantees than PagedAppendingCollection: it might 25 45 /// return a totally consistent snapshot. but it should avoid duplicates 26 46 /// and each page should at least be internally consistent. 47 + #[derive(Debug, PartialEq)] 27 - #[derive(Debug, PartialEq, Default)] 28 48 pub struct PagedOrderedCollection<T, K: Ord> { 29 49 pub items: Vec<T>, 30 50 pub next: Option<K>, 31 51 } 32 52 53 + impl<T, K: Ord> PagedOrderedCollection<T, K> { 54 + pub(crate) fn empty() -> Self { 55 + Self { 56 + items: Vec::new(), 57 + next: None, 58 + } 59 + } 60 + } 61 + 33 62 #[derive(Debug, Deserialize, Serialize, PartialEq)] 34 63 pub struct StorageStats { 35 64 /// estimate of how many accounts we've seen create links. the _subjects_ of any links are not represented here. ··· 82 111 83 112 fn get_distinct_did_count(&self, target: &str, collection: &str, path: &str) -> Result<u64>; 84 113 114 + #[allow(clippy::too_many_arguments)] 85 115 fn get_links( 86 116 &self, 87 117 target: &str, 88 118 collection: &str, 89 119 path: &str, 120 + order: Order, 90 121 limit: u64, 91 122 until: Option<u64>, 92 123 filter_dids: &HashSet<Did>, ··· 180 211 "a.com", 181 212 "app.t.c", 182 213 ".abc.uri", 214 + Order::NewestToOldest, 183 215 100, 184 216 None, 185 217 &HashSet::default() 186 218 )?, 219 + PagedAppendingCollection::empty() 187 - PagedAppendingCollection { 188 - version: (0, 0), 189 - items: vec![], 190 - next: None, 191 - total: 0, 192 - } 193 220 ); 194 221 assert_eq!( 195 222 storage.get_distinct_dids("a.com", "app.t.c", ".abc.uri", 100, None)?, 223 + PagedAppendingCollection::empty() 196 - PagedAppendingCollection { 197 - version: (0, 0), 198 - items: vec![], 199 - next: None, 200 - total: 0, 201 - } 202 224 ); 203 225 assert_eq!(storage.get_all_counts("bad-example.com")?, HashMap::new()); 204 226 assert_eq!( ··· 683 705 "a.com", 684 706 "app.t.c", 685 707 ".abc.uri", 708 + Order::NewestToOldest, 686 709 100, 687 710 None, 688 711 &HashSet::default() ··· 727 750 0, 728 751 )?; 729 752 } 753 + 754 + let sub = "a.com"; 755 + let col = "app.t.c"; 756 + let path = ".abc.uri"; 757 + let order = Order::NewestToOldest; 758 + let dids_filter = HashSet::new(); 759 + 760 + // --- --- round one! --- --- // 761 + // all backlinks 762 + let links = storage.get_links(sub, col, path, order, 2, None, &dids_filter)?; 730 - let links = 731 - storage.get_links("a.com", "app.t.c", ".abc.uri", 2, None, &HashSet::default())?; 732 - let dids = storage.get_distinct_dids("a.com", "app.t.c", ".abc.uri", 2, None)?; 733 763 assert_eq!( 734 764 links, 735 765 PagedAppendingCollection { ··· 737 767 items: vec![ 738 768 RecordId { 739 769 did: "did:plc:asdf-5".into(), 770 + collection: col.into(), 740 - collection: "app.t.c".into(), 741 771 rkey: "asdf".into(), 742 772 }, 743 773 RecordId { 744 774 did: "did:plc:asdf-4".into(), 775 + collection: col.into(), 745 - collection: "app.t.c".into(), 746 776 rkey: "asdf".into(), 747 777 }, 748 778 ], ··· 750 780 total: 5, 751 781 } 752 782 ); 783 + // distinct dids 784 + let dids = storage.get_distinct_dids(sub, col, path, 2, None)?; 753 785 assert_eq!( 754 786 dids, 755 787 PagedAppendingCollection { ··· 759 791 total: 5, 760 792 } 761 793 ); 794 + 795 + // --- --- round two! --- --- // 796 + // all backlinks 797 + let links = storage.get_links(sub, col, path, order, 2, links.next, &dids_filter)?; 762 - let links = storage.get_links( 763 - "a.com", 764 - "app.t.c", 765 - ".abc.uri", 766 - 2, 767 - links.next, 768 - &HashSet::default(), 769 - )?; 770 - let dids = storage.get_distinct_dids("a.com", "app.t.c", ".abc.uri", 2, dids.next)?; 771 798 assert_eq!( 772 799 links, 773 800 PagedAppendingCollection { ··· 775 802 items: vec![ 776 803 RecordId { 777 804 did: "did:plc:asdf-3".into(), 805 + collection: col.into(), 778 - collection: "app.t.c".into(), 779 806 rkey: "asdf".into(), 780 807 }, 781 808 RecordId { 782 809 did: "did:plc:asdf-2".into(), 810 + collection: col.into(), 783 - collection: "app.t.c".into(), 784 811 rkey: "asdf".into(), 785 812 }, 786 813 ], ··· 788 815 total: 5, 789 816 } 790 817 ); 818 + // distinct dids 819 + let dids = storage.get_distinct_dids(sub, col, path, 2, dids.next)?; 791 820 assert_eq!( 792 821 dids, 793 822 PagedAppendingCollection { ··· 797 826 total: 5, 798 827 } 799 828 ); 829 + 830 + // --- --- round three! --- --- // 831 + // all backlinks 832 + let links = storage.get_links(sub, col, path, order, 2, links.next, &dids_filter)?; 800 - let links = storage.get_links( 801 - "a.com", 802 - "app.t.c", 803 - ".abc.uri", 804 - 2, 805 - links.next, 806 - &HashSet::default(), 807 - )?; 808 - let dids = storage.get_distinct_dids("a.com", "app.t.c", ".abc.uri", 2, dids.next)?; 809 833 assert_eq!( 810 834 links, 811 835 PagedAppendingCollection { 812 836 version: (5, 0), 813 837 items: vec![RecordId { 814 838 did: "did:plc:asdf-1".into(), 839 + collection: col.into(), 815 - collection: "app.t.c".into(), 816 840 rkey: "asdf".into(), 817 841 },], 818 842 next: None, 819 843 total: 5, 820 844 } 821 845 ); 846 + // distinct dids 847 + let dids = storage.get_distinct_dids(sub, col, path, 2, dids.next)?; 822 848 assert_eq!( 823 849 dids, 824 850 PagedAppendingCollection { ··· 828 854 total: 5, 829 855 } 830 856 ); 857 + 831 858 assert_stats(storage.get_stats()?, 5..=5, 1..=1, 5..=5); 832 859 }); 833 860 861 + test_each_storage!(get_links_reverse_order, |storage| { 862 + for i in 1..=5 { 863 + storage.push( 864 + &ActionableEvent::CreateLinks { 865 + record_id: RecordId { 866 + did: format!("did:plc:asdf-{i}").into(), 867 + collection: "app.t.c".into(), 868 + rkey: "asdf".into(), 869 + }, 870 + links: vec![CollectedLink { 871 + target: Link::Uri("a.com".into()), 872 + path: ".abc.uri".into(), 873 + }], 874 + }, 875 + 0, 876 + )?; 877 + } 878 + 879 + // Test OldestToNewest order (oldest first) 834 - test_each_storage!(get_filtered_links, |storage| { 835 880 let links = storage.get_links( 836 881 "a.com", 837 882 "app.t.c", 838 883 ".abc.uri", 884 + Order::OldestToNewest, 839 885 2, 840 886 None, 887 + &HashSet::default(), 841 - &HashSet::from([Did("did:plc:linker".to_string())]), 842 888 )?; 843 889 assert_eq!( 844 890 links, 845 891 PagedAppendingCollection { 892 + version: (5, 0), 893 + items: vec![ 894 + RecordId { 895 + did: "did:plc:asdf-1".into(), 896 + collection: "app.t.c".into(), 897 + rkey: "asdf".into(), 898 + }, 899 + RecordId { 900 + did: "did:plc:asdf-2".into(), 901 + collection: "app.t.c".into(), 902 + rkey: "asdf".into(), 903 + }, 904 + ], 905 + next: Some(3), 906 + total: 5, 846 - version: (0, 0), 847 - items: vec![], 848 - next: None, 849 - total: 0, 850 907 } 851 908 ); 909 + // Test NewestToOldest order (newest first) 910 + let links = storage.get_links( 911 + "a.com", 912 + "app.t.c", 913 + ".abc.uri", 914 + Order::NewestToOldest, 915 + 2, 916 + None, 917 + &HashSet::default(), 918 + )?; 919 + assert_eq!( 920 + links, 921 + PagedAppendingCollection { 922 + version: (5, 0), 923 + items: vec![ 924 + RecordId { 925 + did: "did:plc:asdf-5".into(), 926 + collection: "app.t.c".into(), 927 + rkey: "asdf".into(), 928 + }, 929 + RecordId { 930 + did: "did:plc:asdf-4".into(), 931 + collection: "app.t.c".into(), 932 + rkey: "asdf".into(), 933 + }, 934 + ], 935 + next: Some(3), 936 + total: 5, 937 + } 938 + ); 939 + assert_stats(storage.get_stats()?, 5..=5, 1..=1, 5..=5); 940 + }); 941 + 942 + test_each_storage!(get_filtered_links, |storage| { 943 + let links = storage.get_links( 944 + "a.com", 945 + "app.t.c", 946 + ".abc.uri", 947 + Order::NewestToOldest, 948 + 2, 949 + None, 950 + &HashSet::from([Did("did:plc:linker".to_string())]), 951 + )?; 952 + assert_eq!(links, PagedAppendingCollection::empty()); 852 953 853 954 storage.push( 854 955 &ActionableEvent::CreateLinks { ··· 869 970 "a.com", 870 971 "app.t.c", 871 972 ".abc.uri", 973 + Order::NewestToOldest, 872 974 2, 873 975 None, 874 976 &HashSet::from([Did("did:plc:linker".to_string())]), ··· 891 993 "a.com", 892 994 "app.t.c", 893 995 ".abc.uri", 996 + Order::NewestToOldest, 894 997 2, 895 998 None, 896 999 &HashSet::from([Did("did:plc:someone-else".to_string())]), 897 1000 )?; 1001 + assert_eq!(links, PagedAppendingCollection::empty()); 898 - assert_eq!( 899 - links, 900 - PagedAppendingCollection { 901 - version: (0, 0), 902 - items: vec![], 903 - next: None, 904 - total: 0, 905 - } 906 - ); 907 1002 908 1003 storage.push( 909 1004 &ActionableEvent::CreateLinks { ··· 938 1033 "a.com", 939 1034 "app.t.c", 940 1035 ".abc.uri", 1036 + Order::NewestToOldest, 941 1037 2, 942 1038 None, 943 1039 &HashSet::from([Did("did:plc:linker".to_string())]), ··· 967 1063 "a.com", 968 1064 "app.t.c", 969 1065 ".abc.uri", 1066 + Order::NewestToOldest, 970 1067 2, 971 1068 None, 972 1069 &HashSet::from([ ··· 999 1096 "a.com", 1000 1097 "app.t.c", 1001 1098 ".abc.uri", 1099 + Order::NewestToOldest, 1002 1100 2, 1003 1101 None, 1004 1102 &HashSet::from([Did("did:plc:someone-unknown".to_string())]), 1005 1103 )?; 1104 + assert_eq!(links, PagedAppendingCollection::empty()); 1006 - assert_eq!( 1007 - links, 1008 - PagedAppendingCollection { 1009 - version: (0, 0), 1010 - items: vec![], 1011 - next: None, 1012 - total: 0, 1013 - } 1014 - ); 1015 1105 }); 1016 1106 1017 1107 test_each_storage!(get_links_exact_multiple, |storage| { ··· 1031 1121 0, 1032 1122 )?; 1033 1123 } 1124 + let links = storage.get_links( 1125 + "a.com", 1126 + "app.t.c", 1127 + ".abc.uri", 1128 + Order::NewestToOldest, 1129 + 2, 1130 + None, 1131 + &HashSet::default(), 1132 + )?; 1034 - let links = 1035 - storage.get_links("a.com", "app.t.c", ".abc.uri", 2, None, &HashSet::default())?; 1036 1133 assert_eq!( 1037 1134 links, 1038 1135 PagedAppendingCollection { ··· 1057 1154 "a.com", 1058 1155 "app.t.c", 1059 1156 ".abc.uri", 1157 + Order::NewestToOldest, 1060 1158 2, 1061 1159 links.next, 1062 1160 &HashSet::default(), ··· 1101 1199 0, 1102 1200 )?; 1103 1201 } 1202 + let links = storage.get_links( 1203 + "a.com", 1204 + "app.t.c", 1205 + ".abc.uri", 1206 + Order::NewestToOldest, 1207 + 2, 1208 + None, 1209 + &HashSet::default(), 1210 + )?; 1104 - let links = 1105 - storage.get_links("a.com", "app.t.c", ".abc.uri", 2, None, &HashSet::default())?; 1106 1211 assert_eq!( 1107 1212 links, 1108 1213 PagedAppendingCollection { ··· 1141 1246 "a.com", 1142 1247 "app.t.c", 1143 1248 ".abc.uri", 1249 + Order::NewestToOldest, 1144 1250 2, 1145 1251 links.next, 1146 1252 &HashSet::default(), ··· 1185 1291 0, 1186 1292 )?; 1187 1293 } 1294 + let links = storage.get_links( 1295 + "a.com", 1296 + "app.t.c", 1297 + ".abc.uri", 1298 + Order::NewestToOldest, 1299 + 2, 1300 + None, 1301 + &HashSet::default(), 1302 + )?; 1188 - let links = 1189 - storage.get_links("a.com", "app.t.c", ".abc.uri", 2, None, &HashSet::default())?; 1190 1303 assert_eq!( 1191 1304 links, 1192 1305 PagedAppendingCollection { ··· 1219 1332 "a.com", 1220 1333 "app.t.c", 1221 1334 ".abc.uri", 1335 + Order::NewestToOldest, 1222 1336 2, 1223 1337 links.next, 1224 1338 &HashSet::default(), ··· 1256 1370 0, 1257 1371 )?; 1258 1372 } 1373 + let links = storage.get_links( 1374 + "a.com", 1375 + "app.t.c", 1376 + ".abc.uri", 1377 + Order::NewestToOldest, 1378 + 2, 1379 + None, 1380 + &HashSet::default(), 1381 + )?; 1259 - let links = 1260 - storage.get_links("a.com", "app.t.c", ".abc.uri", 2, None, &HashSet::default())?; 1261 1382 assert_eq!( 1262 1383 links, 1263 1384 PagedAppendingCollection { ··· 1286 1407 "a.com", 1287 1408 "app.t.c", 1288 1409 ".abc.uri", 1410 + Order::NewestToOldest, 1289 1411 2, 1290 1412 links.next, 1291 1413 &HashSet::default(), ··· 1372 1494 &HashSet::new(), 1373 1495 &HashSet::new(), 1374 1496 )?, 1497 + PagedOrderedCollection::empty() 1375 - PagedOrderedCollection { 1376 - items: vec![], 1377 - next: None, 1378 - } 1379 1498 ); 1380 1499 }); 1381 1500
+41 -25
constellation/src/storage/rocks_store.rs
··· 1 1 use super::{ 2 + ActionableEvent, LinkReader, LinkStorage, Order, PagedAppendingCollection, 3 + PagedOrderedCollection, StorageStats, 2 - ActionableEvent, LinkReader, LinkStorage, PagedAppendingCollection, PagedOrderedCollection, 3 - StorageStats, 4 4 }; 5 5 use crate::{CountsByCount, Did, RecordId}; 6 6 use anyhow::{bail, Result}; ··· 960 960 961 961 let Some(target_id) = self.target_id_table.get_id_val(&self.db, &target_key)? else { 962 962 eprintln!("nothin doin for this target, {target_key:?}"); 963 + return Ok(PagedOrderedCollection::empty()); 963 - return Ok(Default::default()); 964 964 }; 965 965 966 966 let filter_did_ids: HashMap<DidId, bool> = filter_dids ··· 1127 1127 target: &str, 1128 1128 collection: &str, 1129 1129 path: &str, 1130 + order: Order, 1130 1131 limit: u64, 1131 1132 until: Option<u64>, 1132 1133 filter_dids: &HashSet<Did>, ··· 1138 1139 ); 1139 1140 1140 1141 let Some(target_id) = self.target_id_table.get_id_val(&self.db, &target_key)? else { 1142 + return Ok(PagedAppendingCollection::empty()); 1141 - return Ok(PagedAppendingCollection { 1142 - version: (0, 0), 1143 - items: Vec::new(), 1144 - next: None, 1145 - total: 0, 1146 - }); 1147 1143 }; 1148 1144 1149 1145 let mut linkers = self.get_target_linkers(&target_id)?; ··· 1167 1163 1168 1164 let (alive, gone) = linkers.count(); 1169 1165 let total = alive + gone; 1166 + 1167 + let (start, take, next_until) = match order { 1168 + // OldestToNewest: start from the beginning, paginate forward 1169 + Order::OldestToNewest => { 1170 + let start = until.unwrap_or(0); 1171 + let next = start + limit + 1; 1172 + let next_until = if next < total { Some(next) } else { None }; 1173 + (start, limit, next_until) 1174 + } 1175 + // NewestToOldest: start from the end, paginate backward 1176 + Order::NewestToOldest => { 1177 + let until = until.unwrap_or(total); 1178 + match until.checked_sub(limit) { 1179 + Some(s) if s > 0 => (s, limit, Some(s)), 1180 + Some(s) => (s, limit, None), 1181 + None => (0, until, None), 1182 + } 1183 + } 1184 + }; 1170 - let end = until.map(|u| std::cmp::min(u, total)).unwrap_or(total) as usize; 1171 - let begin = end.saturating_sub(limit as usize); 1172 - let next = if begin == 0 { None } else { Some(begin as u64) }; 1173 1185 1186 + let did_id_rkeys = linkers.0.iter().skip(start as usize).take(take as usize); 1187 + let did_id_rkeys: Vec<_> = match order { 1188 + Order::OldestToNewest => did_id_rkeys.collect(), 1189 + Order::NewestToOldest => did_id_rkeys.rev().collect(), 1190 + }; 1174 - let did_id_rkeys = linkers.0[begin..end].iter().rev().collect::<Vec<_>>(); 1175 1191 1176 1192 let mut items = Vec::with_capacity(did_id_rkeys.len()); 1177 1193 // TODO: use get-many (or multi-get or whatever it's called) ··· 1201 1217 Ok(PagedAppendingCollection { 1202 1218 version: (total, gone), 1203 1219 items, 1220 + next: next_until, 1204 - next, 1205 1221 total: alive, 1206 1222 }) 1207 1223 } ··· 1221 1237 ); 1222 1238 1223 1239 let Some(target_id) = self.target_id_table.get_id_val(&self.db, &target_key)? else { 1240 + return Ok(PagedAppendingCollection::empty()); 1224 - return Ok(PagedAppendingCollection { 1225 - version: (0, 0), 1226 - items: Vec::new(), 1227 - next: None, 1228 - total: 0, 1229 - }); 1230 1241 }; 1231 1242 1232 1243 let linkers = self.get_distinct_target_linkers(&target_id)?; 1233 1244 1234 1245 let (alive, gone) = linkers.count(); 1235 1246 let total = alive + gone; 1236 - let end = until.map(|u| std::cmp::min(u, total)).unwrap_or(total) as usize; 1237 - let begin = end.saturating_sub(limit as usize); 1238 - let next = if begin == 0 { None } else { Some(begin as u64) }; 1239 1247 1248 + let until = until.unwrap_or(total); 1249 + let (start, take, next_until) = match until.checked_sub(limit) { 1250 + Some(s) if s > 0 => (s, limit, Some(s)), 1251 + Some(s) => (s, limit, None), 1252 + None => (0, until, None), 1253 + }; 1254 + 1255 + let did_id_rkeys = linkers.0.iter().skip(start as usize).take(take as usize); 1256 + let did_id_rkeys: Vec<_> = did_id_rkeys.rev().collect(); 1240 - let did_id_rkeys = linkers.0[begin..end].iter().rev().collect::<Vec<_>>(); 1241 1257 1242 1258 let mut items = Vec::with_capacity(did_id_rkeys.len()); 1243 1259 // TODO: use get-many (or multi-get or whatever it's called) ··· 1263 1279 Ok(PagedAppendingCollection { 1264 1280 version: (total, gone), 1265 1281 items, 1282 + next: next_until, 1266 - next, 1267 1283 total: alive, 1268 1284 }) 1269 1285 }
+4
constellation/templates/base.html.j2
··· 40 40 padding: 0.5em 0.3em; 41 41 max-width: 100%; 42 42 } 43 + pre.code input { 44 + margin: 0; 45 + padding: 0; 46 + } 43 47 .stat { 44 48 color: #f90; 45 49 font-size: 1.618rem;
+2 -1
constellation/templates/get-backlinks.html.j2
··· 6 6 7 7 {% block content %} 8 8 9 + {% call try_it::get_backlinks(query.subject, query.source, query.did, query.limit, query.reverse) %} 9 - {% call try_it::get_backlinks(query.subject, query.source, query.did, query.limit) %} 10 10 11 11 <h2> 12 12 Links to <code>{{ query.subject }}</code> ··· 40 40 <input type="hidden" name="did" value="{{ did }}" /> 41 41 {% endfor %} 42 42 <input type="hidden" name="cursor" value={{ c|json|safe }} /> 43 + <input type="hidden" name="reverse" value="{{ query.reverse }}"> 43 44 <button type="submit">next page&hellip;</button> 44 45 </form> 45 46 {% else %}
+4 -1
constellation/templates/hello.html.j2
··· 49 49 <li><p><code>source</code>: required. Example: <code>app.bsky.feed.like:subject.uri</code></p></li> 50 50 <li><p><code>did</code>: optional, filter links to those from specific users. Include multiple times to filter by multiple users. Example: <code>did=did:plc:vc7f4oafdgxsihk4cry2xpze&did=did:plc:vc7f4oafdgxsihk4cry2xpze</code></p></li> 51 51 <li><p><code>limit</code>: optional. Default: <code>16</code>. Maximum: <code>100</code></p></li> 52 + <li><p><code>reverse</code>: optional, return links in reverse order. Default: <code>false</code></p></li> 52 53 </ul> 53 54 54 55 <p style="margin-bottom: 0"><strong>Try it:</strong></p> 55 - {% call try_it::get_backlinks("at://did:plc:a4pqq234yw7fqbddawjo7y35/app.bsky.feed.post/3m237ilwc372e", "app.bsky.feed.like:subject.uri", [""], 16) %} 56 + {% call 57 + try_it::get_backlinks("at://did:plc:a4pqq234yw7fqbddawjo7y35/app.bsky.feed.post/3m237ilwc372e", "app.bsky.feed.like:subject.uri", [""], 16, false) %} 56 58 57 59 58 60 <h3 class="route"><code>GET /xrpc/blue.microcosm.links.getManyToManyCounts</code></h3> ··· 96 98 <li><p><code>did</code>: optional, filter links to those from specific users. Include multiple times to filter by multiple users. Example: <code>did=did:plc:vc7f4oafdgxsihk4cry2xpze&did=did:plc:vc7f4oafdgxsihk4cry2xpze</code></p></li> 97 99 <li><p><code>from_dids</code> [deprecated]: optional. Use <code>did</code> instead. Example: <code>from_dids=did:plc:vc7f4oafdgxsihk4cry2xpze,did:plc:vc7f4oafdgxsihk4cry2xpze</code></p></li> 98 100 <li><p><code>limit</code>: optional. Default: <code>16</code>. Maximum: <code>100</code></p></li> 101 + <li><p><code>reverse</code>: optional, return links in reverse order. Default: <code>false</code></p></li> 99 102 </ul> 100 103 101 104 <p style="margin-bottom: 0"><strong>Try it:</strong></p>
+4 -2
constellation/templates/try-it-macros.html.j2
··· 1 - {% macro get_backlinks(subject, source, dids, limit) %} 1 + {% macro get_backlinks(subject, source, dids, limit, reverse) %} 2 2 <form method="get" action="/xrpc/blue.microcosm.links.getBacklinks"> 3 3 <pre class="code"><strong>GET</strong> /xrpc/blue.microcosm.links.getBacklinks 4 4 ?subject= <input type="text" name="subject" value="{{ subject }}" placeholder="at-uri, did, uri..." /> ··· 6 6 {%- for did in dids %}{% if !did.is_empty() %} 7 7 &did= <input type="text" name="did" value="{{ did }}" placeholder="did:plc:..." />{% endif %}{% endfor %} 8 8 <span id="did-placeholder"></span> <button id="add-did">+ did filter</button> 9 - &limit= <input type="number" name="limit" value="{{ limit }}" max="100" placeholder="100" /> <button type="submit">get links</button></pre> 9 + &limit= <input type="number" name="limit" value="{{ limit }}" max="100" placeholder="100" /> 10 + &reverse= <input type="checkbox" name="reverse" value="true" {% if reverse %}checked{% endif %}> 11 + <button type="submit">get links</button></pre> 10 12 </form> 11 13 <script> 12 14 const addDidButton = document.getElementById('add-did');
lexicons/blue.microcosm/links/getBacklinks.json

This file has not been changed.

lexicons/blue.microcosm/links/getBacklinksCount.json

This file has not been changed.

lexicons/blue.microcosm/links/getManyToMany.json

This file has not been changed.

lexicons/blue.microcosm/links/getManyToManyCounts.json

This file has not been changed.

History

8 rounds 13 comments
sign up or login to add to the discussion
seoul.systems submitted #7
diff interdiff
8 commits
expand
c9e714e7
Add getCounts XRPC equivalent to REST /links/count
09f58297
Modify backlinks counting XRPC endpoint name
f5c3fe45
Mark /links/count REST endpoint as deprecated
92fc8620
Remove .uri suffix
ba065299
Reformat existing lexicons
410a3830
Remove wrongly commited getManyToMany lexicon
7faaa631
Fix Git whitespace error in "hello" template
d16917f1
Format .prettierrc and fix Git whitespace error
expand 3 comments
seoul.systems

I'm giving up on the whitespace issue...

max@max-mbpro ~/dev/microcosm-rs (xrpc_backlinks_count) $ git diff --check upstream/main
max@max-mbpro ~/dev/microcosm-rs (xrpc_backlinks_count) $

git-diff --checked doesn't indicate any errors either.

As far as I can tell this seems to be a confirmed Tangled issue

bad-example.com

happy to do merging locally as needed!

bad-example.com

merged. thanks!

closed without merging
seoul.systems submitted #6
diff interdiff
7 commits
expand
c9e714e7
Add getCounts XRPC equivalent to REST /links/count
09f58297
Modify backlinks counting XRPC endpoint name
f5c3fe45
Mark /links/count REST endpoint as deprecated
92fc8620
Remove .uri suffix
ba065299
Reformat existing lexicons
410a3830
Remove wrongly commited getManyToMany lexicon
7faaa631
Fix Git whitespace error in "hello" template
expand 0 comments
seoul.systems submitted #5
diff interdiff
6 commits
expand
c9e714e7
Add getCounts XRPC equivalent to REST /links/count
09f58297
Modify backlinks counting XRPC endpoint name
f5c3fe45
Mark /links/count REST endpoint as deprecated
92fc8620
Remove .uri suffix
ba065299
Reformat existing lexicons
410a3830
Remove wrongly commited getManyToMany lexicon
expand 0 comments
seoul.systems submitted #4
diff
3 commits
expand
c9e714e7
Add getCounts XRPC equivalent to REST /links/count
09f58297
Modify backlinks counting XRPC endpoint name
f5c3fe45
Mark /links/count REST endpoint as deprecated
expand 4 comments
seoul.systems

Tangled somehow complains about merge conflicts here, but I couldn't find any after rebasing on upstream/main again?!

bad-example.com

very weird!

one tiny thing left: the source for blocks is app.bsky.graph.block:subject (no .uri suffix on the path) -- it's in the hello.html template.

bad-example.com

i think some many-to-many order stuff ended up on this branch but i'm too tired for git rn.

if you don't get to it first i'm happy to fix the source and git stuff and merge when i can get to it :)

seoul.systems

Sorry about the the m2m stuff that I didn't catch before opening the PR. I did clean this up again, and nothing belonging there should be remaining here; Addresses your above comment regarding the .uri suffix as well.

seoul.systems submitted #3
diff interdiff
3 commits
expand
ce009bd4
Add getCounts XRPC equivalent to REST /links/count
c39d6a0a
Modify backlinks counting XRPC endpoint name
9b992af4
Mark /links/count REST endpoint as deprecated
expand 0 comments
seoul.systems submitted #2
diff interdiff
1 commit
expand
ce009bd4
Add getCounts XRPC equivalent to REST /links/count
expand 6 comments
seoul.systems

Had to do some rebasing and cleanup, but good to go now I think.

bad-example.com

sweet!

(btw the local/ gitignore is there for doing local/rocks.test, but i'm fine adding it to the top-level gitignore too!)

bad-example.com

i think links.getCounts is a little too broad -- what do you think of links.getBacklinksCount?

bad-example.com

we also throw a deprecated warning under /links/count on the main page template, matching the one for /links.

bad-example.com

it's weird but the whitespace in the forms in try-it-macros is significant (one of my moments of knowingly doing things the wrong way because i was bored, sorry!)

the new form for this endpoint needs little tweaking to match the others.

seoul.systems

Ah got it. The rocks tests ended up cluttering the top-level directory so I just added them to .gitignore. I often keep a .local file in .gitignores to ignore whatever's supposed to stay in one's local copy and somehow assumed this had the same intent.

Changed the function and endpoint/method name as requested to get_backlinks_count/getBacklinksCount

The endpoint in the try-it-macro should now match the other existing ones down to how they're formatted with whitespace

seoul.systems submitted #1
diff interdiff
3 commits
expand
6221bbb7
Make metrics collection opt-in
f0ef2095
Increase constellation response limits
2a98c132
Add getCounts XRPC equivalent to REST /links/count
expand 0 comments
seoul.systems submitted #0
diff
7 commits
expand
6221bbb7
Make metrics collection opt-in
f0ef2095
Increase constellation response limits
a4177589
wip: m2m
81b7df55
Add tests for new get_many_to_many query handler
c6742917
Fix get_m2m_empty test
f0b16b50
Replace tuple with RecordsBySubject struct
740509e5
Add getCounts XRPC equivalent to REST /links/count
expand 0 comments