Constellation, Spacedust, Slingshot, UFOs: atproto crates and services for microcosm

Convert existing REST /links/count endpoint to XRPC equivalent #8

closed opened by seoul.systems targeting main from seoul.systems/microcosm-rs: xrpc_backlinks_count

Add getCounts XRPC equivalent to REST /links/count

Simple conversion of the existing endpoint from REST to XRPC. Consequtively marked the pre-existing REST endpoint as deprecated.

In addition we now ignore rocks.test

Labels

None yet.

Participants 2
AT URI
at://did:plc:53wellrw53o7sw4zlpfenvuh/sh.tangled.repo.pull/3mcybnzocig22
+270 -50
Diff #1
+1
.gitignore
··· 1 1 /target 2 2 local/ 3 + rocks.test
+4
.prettierrc
··· 1 + { 2 + "tabWidth": 2, 3 + "useTabs": false 4 + }
+47
constellation/src/server/mod.rs
··· 78 78 }), 79 79 ) 80 80 .route( 81 + "/xrpc/blue.microcosm.links.getCounts", 82 + get({ 83 + let store = store.clone(); 84 + move |accept, query| async { 85 + spawn_blocking(|| get_counts(accept, query, store)) 86 + .await 87 + .map_err(to500)? 88 + } 89 + }), 90 + ) 91 + .route( 81 92 "/links/count/distinct-dids", 82 93 get({ 83 94 let store = store.clone(); ··· 343 354 #[serde(skip_serializing)] 344 355 query: GetLinksCountQuery, 345 356 } 357 + #[deprecated] 346 358 fn count_links( 347 359 accept: ExtractAccept, 348 360 query: Query<GetLinksCountQuery>, ··· 361 373 } 362 374 363 375 #[derive(Clone, Deserialize)] 376 + struct GetItemsCountQuery { 377 + subject: String, 378 + source: String, 379 + } 380 + #[derive(Template, Serialize)] 381 + #[template(path = "get-counts.html.j2")] 382 + struct GetItemsCountResponse { 383 + total: u64, 384 + #[serde(skip_serializing)] 385 + query: GetItemsCountQuery, 386 + } 387 + fn get_counts( 388 + accept: ExtractAccept, 389 + query: axum_extra::extract::Query<GetItemsCountQuery>, 390 + store: impl LinkReader, 391 + ) -> Result<impl IntoResponse, http::StatusCode> { 392 + let Some((collection, path)) = query.source.split_once(':') else { 393 + return Err(http::StatusCode::BAD_REQUEST); 394 + }; 395 + let path = format!(".{path}"); 396 + let total = store 397 + .get_count(&query.subject, collection, &path) 398 + .map_err(|_| http::StatusCode::INTERNAL_SERVER_ERROR)?; 399 + 400 + Ok(acceptable( 401 + accept, 402 + GetItemsCountResponse { 403 + total, 404 + query: (*query).clone(), 405 + }, 406 + )) 407 + } 408 + 409 + #[derive(Clone, Deserialize)] 364 410 struct GetDidsCountQuery { 365 411 target: String, 366 412 collection: String, ··· 659 705 #[serde(skip_serializing)] 660 706 query: GetAllLinksQuery, 661 707 } 708 + #[deprecated] 662 709 fn count_all_links( 663 710 accept: ExtractAccept, 664 711 query: Query<GetAllLinksQuery>,
+1 -1
constellation/templates/dids.html.j2
··· 27 27 {% for did in linking_dids %} 28 28 <pre style="display: block; margin: 1em 2em" class="code"><strong>DID</strong>: {{ did.0 }} 29 29 -> see <a href="/links/all?target={{ did.0|urlencode }}">links to this DID</a> 30 - -> browse <a href="https://pdsls.dev/at://{{ did.0 }}">this DID record</a></pre> 30 + -> browse <a href="https://pdsls.dev/at://{{ did.0|urlencode }}">this DID record</a></pre> 31 31 {% endfor %} 32 32 33 33 {% if let Some(c) = cursor %}
+38
constellation/templates/get-counts.html.j2
··· 1 + {% extends "base.html.j2" %} 2 + {% import "try-it-macros.html.j2" as try_it %} 3 + 4 + {% block title %}Link Count{% endblock %} 5 + {% block description %}Count of {{ query.source }} records linking to {{ query.subject }}{% endblock %} 6 + 7 + {% block content %} 8 + 9 + {% call try_it::get_counts( 10 + query.subject, 11 + query.source, 12 + ) %} 13 + 14 + <h2> 15 + Total links to <code>{{ query.subject }}</code> 16 + {% if let Some(browseable_uri) = query.subject|to_browseable %} 17 + <small style="font-weight: normal; font-size: 1rem"><a href="{{ browseable_uri }}">browse record</a></small> 18 + {% endif %} 19 + </h2> 20 + 21 + <p><strong><code>{{ total|human_number }}</code></strong> total links from <code>{{ query.source }}</code> to <code>{{ query.subject }}</code></p> 22 + 23 + <ul> 24 + <li> 25 + See direct backlinks at <code>/xrpc/blue.microcosm.links.getBacklinks</code>: 26 + <a href="/xrpc/blue.microcosm.links.getBacklinks?subject={{ query.subject|urlencode }}&source={{ query.source|urlencode }}"> 27 + /xrpc/blue.microcosm.links.getBacklinks?subject={{ query.subject }}&source={{ query.source }} 28 + </a> 29 + </li> 30 + <li>See all links to this target at <code>/links/all</code>: <a href="/links/all?target={{ query.subject|urlencode }}">/links/all?target={{ query.subject }}</a></li> 31 + </ul> 32 + 33 + <details> 34 + <summary>Raw JSON response</summary> 35 + <pre class="code">{{ self|tojson }}</pre> 36 + </details> 37 + 38 + {% endblock %}
+13
constellation/templates/hello.html.j2
··· 134 134 <p style="margin-bottom: 0"><strong>Try it:</strong></p> 135 135 {% call try_it::links_count("did:plc:vc7f4oafdgxsihk4cry2xpze", "app.bsky.graph.block", ".subject") %} 136 136 137 + <h3 class="route"><code>GET /xrpc/blue.microcosm.links.getCounts</code></h3> 138 + 139 + <p>The total number of links pointing at a given target.</p> 140 + 141 + <h4>Query parameters:</h4> 142 + 143 + <ul> 144 + <li><code>subject</code>: required, must url-encode. The target being linked to. Example: <code>did:plc:vc7f4oafdgxsihk4cry2xpze</code> or <code>at://did:plc:vc7f4oafdgxsihk4cry2xpze/app.bsky.feed.post/3lgwdn7vd722r</code></li> 145 + <li><code>source</code>: required. Collection and path specification for the primary link. Example: <code>app.bsky.feed.like:subject.uri</code></li> 146 + </ul> 147 + 148 + <p style="margin-bottom: 0"><strong>Try it:</strong></p> 149 + {% call try_it::get_counts("did:plc:vc7f4oafdgxsihk4cry2xpze", "app.bsky.graph.block:subject.uri") %} 137 150 138 151 <h3 class="route"><code>GET /links/count/distinct-dids</code></h3> 139 152
+10
constellation/templates/try-it-macros.html.j2
··· 102 102 </form> 103 103 {% endmacro %} 104 104 105 + {% macro get_counts(subject, source) %} 106 + <form method="get" action="/xrpc/blue.microcosm.links.getCounts"> 107 + <pre class="code"> 108 + <strong>GET</strong> /xrpc/blue.microcosm.links.getCounts 109 + ?subject= <input type="text" name="subject" value="{{ subject }}" placeholder="subject" /> 110 + &source= <input type="text" name="source" value="{{ source }}" placeholder="source" /> 111 + <button type="submit">get links count</button> 112 + </pre> 113 + </form> 114 + {% endmacro %} 105 115 106 116 {% macro links_count(target, collection, path) %} 107 117 <form method="get" action="/links/count">
+3 -13
lexicons/blue.microcosm/links/getBacklinks.json
··· 7 7 "description": "a list of records linking to any record, identity, or uri", 8 8 "parameters": { 9 9 "type": "params", 10 - "required": [ 11 - "subject", 12 - "source" 13 - ], 10 + "required": ["subject", "source"], 14 11 "properties": { 15 12 "subject": { 16 13 "type": "string", ··· 42 39 "encoding": "application/json", 43 40 "schema": { 44 41 "type": "object", 45 - "required": [ 46 - "total", 47 - "records" 48 - ], 42 + "required": ["total", "records"], 49 43 "properties": { 50 44 "total": { 51 45 "type": "integer", ··· 68 62 }, 69 63 "linkRecord": { 70 64 "type": "object", 71 - "required": [ 72 - "did", 73 - "collection", 74 - "rkey" 75 - ], 65 + "required": ["did", "collection", "rkey"], 76 66 "properties": { 77 67 "did": { 78 68 "type": "string",
+38
lexicons/blue.microcosm/links/getCounts.json
··· 1 + { 2 + "lexicon": 1, 3 + "id": "blue.microcosm.links.getCounts", 4 + "defs": { 5 + "main": { 6 + "type": "query", 7 + "description": "count records that link to another record", 8 + "parameters": { 9 + "type": "params", 10 + "required": ["subject", "source"], 11 + "properties": { 12 + "subject": { 13 + "type": "string", 14 + "format": "uri", 15 + "description": "the primary target being linked to (at-uri, did, or uri)" 16 + }, 17 + "source": { 18 + "type": "string", 19 + "description": "collection and path specification for the primary link" 20 + } 21 + } 22 + }, 23 + "output": { 24 + "encoding": "application/json", 25 + "schema": { 26 + "type": "object", 27 + "required": ["total"], 28 + "properties": { 29 + "total": { 30 + "type": "integer", 31 + "description": "total number of matching links" 32 + } 33 + } 34 + } 35 + } 36 + } 37 + } 38 + }
+109
lexicons/blue.microcosm/links/getManyToMany.json
··· 1 + { 2 + "lexicon": 1, 3 + "id": "blue.microcosm.links.getManyToMany", 4 + "defs": { 5 + "main": { 6 + "type": "query", 7 + "description": "Get records that link to a primary subject, grouped by the secondary subjects they also reference", 8 + "parameters": { 9 + "type": "params", 10 + "required": ["subject", "source", "pathToOther"], 11 + "properties": { 12 + "subject": { 13 + "type": "string", 14 + "format": "uri", 15 + "description": "the primary target being linked to (at-uri, did, or uri)" 16 + }, 17 + "source": { 18 + "type": "string", 19 + "description": "collection and path specification for the primary link (e.g., 'app.bsky.feed.like:subject.uri')" 20 + }, 21 + "pathToOther": { 22 + "type": "string", 23 + "description": "path to the secondary link in the many-to-many record (e.g., 'otherThing.uri')" 24 + }, 25 + "did": { 26 + "type": "array", 27 + "description": "filter links to those from specific users", 28 + "items": { 29 + "type": "string", 30 + "format": "did" 31 + } 32 + }, 33 + "otherSubject": { 34 + "type": "array", 35 + "description": "filter secondary links to specific subjects", 36 + "items": { 37 + "type": "string" 38 + } 39 + }, 40 + "limit": { 41 + "type": "integer", 42 + "minimum": 1, 43 + "maximum": 100, 44 + "default": 16, 45 + "description": "number of results to return" 46 + } 47 + } 48 + }, 49 + "output": { 50 + "encoding": "application/json", 51 + "schema": { 52 + "type": "object", 53 + "required": ["linking_records"], 54 + "properties": { 55 + "linking_records": { 56 + "type": "array", 57 + "items": { 58 + "type": "ref", 59 + "ref": "#recordsBySubject" 60 + } 61 + }, 62 + "cursor": { 63 + "type": "string", 64 + "description": "pagination cursor" 65 + } 66 + } 67 + } 68 + } 69 + }, 70 + "recordsBySubject": { 71 + "type": "object", 72 + "required": ["subject", "records"], 73 + "properties": { 74 + "subject": { 75 + "type": "string", 76 + "description": "the secondary subject that these records link to" 77 + }, 78 + "records": { 79 + "type": "array", 80 + "items": { 81 + "type": "ref", 82 + "ref": "#linkRecord" 83 + } 84 + } 85 + } 86 + }, 87 + "linkRecord": { 88 + "type": "object", 89 + "required": ["did", "collection", "rkey"], 90 + "description": "A record identifier consisting of a DID, collection, and record key", 91 + "properties": { 92 + "did": { 93 + "type": "string", 94 + "format": "did", 95 + "description": "the DID of the linking record's repository" 96 + }, 97 + "collection": { 98 + "type": "string", 99 + "format": "nsid", 100 + "description": "the collection of the linking record" 101 + }, 102 + "rkey": { 103 + "type": "string", 104 + "format": "record-key" 105 + } 106 + } 107 + } 108 + } 109 + }
+3 -13
lexicons/blue.microcosm/links/getManyToManyCounts.json
··· 7 7 "description": "count many-to-many relationships with secondary link paths", 8 8 "parameters": { 9 9 "type": "params", 10 - "required": [ 11 - "subject", 12 - "source", 13 - "pathToOther" 14 - ], 10 + "required": ["subject", "source", "pathToOther"], 15 11 "properties": { 16 12 "subject": { 17 13 "type": "string", ··· 54 50 "encoding": "application/json", 55 51 "schema": { 56 52 "type": "object", 57 - "required": [ 58 - "counts_by_other_subject" 59 - ], 53 + "required": ["counts_by_other_subject"], 60 54 "properties": { 61 55 "counts_by_other_subject": { 62 56 "type": "array", ··· 75 69 }, 76 70 "countBySubject": { 77 71 "type": "object", 78 - "required": [ 79 - "subject", 80 - "total", 81 - "distinct" 82 - ], 72 + "required": ["subject", "total", "distinct"], 83 73 "properties": { 84 74 "subject": { 85 75 "type": "string",
-2
spacedust/src/error.rs
··· 30 30 TooManySourcesWanted, 31 31 #[error("more wantedSubjectDids were requested than allowed (max 10,000)")] 32 32 TooManyDidsWanted, 33 - #[error("more wantedSubjectPrefixes were requested than allowed (max 100)")] 34 - TooManySubjectPrefixesWanted, 35 33 #[error("more wantedSubjects were requested than allowed (max 50,000)")] 36 34 TooManySubjectsWanted, 37 35 }
+2 -11
spacedust/src/server.rs
··· 227 227 #[serde(default)] 228 228 pub wanted_subjects: HashSet<String>, 229 229 #[serde(default)] 230 - pub wanted_subject_prefixes: HashSet<String>, 231 - #[serde(default)] 232 230 pub wanted_subject_dids: HashSet<String>, 233 231 #[serde(default)] 234 232 pub wanted_sources: HashSet<String>, ··· 243 241 /// 244 242 /// The at-uri must be url-encoded 245 243 /// 246 - /// Pass this parameter multiple times to specify multiple subjects, like 244 + /// Pass this parameter multiple times to specify multiple collections, like 247 245 /// `wantedSubjects=[...]&wantedSubjects=[...]` 248 246 pub wanted_subjects: String, 249 - /// One or more at-uri, URI, or DID prefixes to receive links about 250 - /// 251 - /// The uri must be url-encoded 252 - /// 253 - /// Pass this parameter multiple times to specify multiple prefixes, like 254 - /// `wantedSubjectPrefixes=[...]&wantedSubjectPrefixes=[...]` 255 - pub wanted_subject_prefixes: String, 256 247 /// One or more DIDs to receive links about 257 248 /// 258 - /// Pass this parameter multiple times to specify multiple subjects 249 + /// Pass this parameter multiple times to specify multiple collections 259 250 pub wanted_subject_dids: String, 260 251 /// One or more link sources to receive links about 261 252 ///
+1 -10
spacedust/src/subscriber.rs
··· 124 124 let query = &self.query; 125 125 126 126 // subject + subject DIDs are logical OR 127 - if !(query.wanted_subjects.is_empty() 128 - && query.wanted_subject_prefixes.is_empty() 129 - && query.wanted_subject_dids.is_empty() 127 + if !(query.wanted_subjects.is_empty() && query.wanted_subject_dids.is_empty() 130 128 || query.wanted_subjects.contains(&properties.subject) 131 - || query 132 - .wanted_subject_prefixes 133 - .iter() 134 - .any(|p| properties.subject.starts_with(p)) 135 129 || properties 136 130 .subject_did 137 131 .as_ref() ··· 160 154 } 161 155 if opts.wanted_subject_dids.len() > 10_000 { 162 156 return Err(SubscriberUpdateError::TooManyDidsWanted); 163 - } 164 - if opts.wanted_subject_prefixes.len() > 100 { 165 - return Err(SubscriberUpdateError::TooManySubjectPrefixesWanted); 166 157 } 167 158 if opts.wanted_subjects.len() > 50_000 { 168 159 return Err(SubscriberUpdateError::TooManySubjectsWanted);

History

8 rounds 13 comments
sign up or login to add to the discussion
8 commits
expand
Add getCounts XRPC equivalent to REST /links/count
Modify backlinks counting XRPC endpoint name
Mark /links/count REST endpoint as deprecated
Remove .uri suffix
Reformat existing lexicons
Remove wrongly commited getManyToMany lexicon
Fix Git whitespace error in "hello" template
Format .prettierrc and fix Git whitespace error
expand 3 comments

I'm giving up on the whitespace issue...

max@max-mbpro ~/dev/microcosm-rs (xrpc_backlinks_count) $ git diff --check upstream/main
max@max-mbpro ~/dev/microcosm-rs (xrpc_backlinks_count) $

git-diff --checked doesn't indicate any errors either.

As far as I can tell this seems to be a confirmed Tangled issue

happy to do merging locally as needed!

merged. thanks!

closed without merging
7 commits
expand
Add getCounts XRPC equivalent to REST /links/count
Modify backlinks counting XRPC endpoint name
Mark /links/count REST endpoint as deprecated
Remove .uri suffix
Reformat existing lexicons
Remove wrongly commited getManyToMany lexicon
Fix Git whitespace error in "hello" template
expand 0 comments
6 commits
expand
Add getCounts XRPC equivalent to REST /links/count
Modify backlinks counting XRPC endpoint name
Mark /links/count REST endpoint as deprecated
Remove .uri suffix
Reformat existing lexicons
Remove wrongly commited getManyToMany lexicon
expand 0 comments
3 commits
expand
Add getCounts XRPC equivalent to REST /links/count
Modify backlinks counting XRPC endpoint name
Mark /links/count REST endpoint as deprecated
expand 4 comments

Tangled somehow complains about merge conflicts here, but I couldn't find any after rebasing on upstream/main again?!

very weird!

one tiny thing left: the source for blocks is app.bsky.graph.block:subject (no .uri suffix on the path) -- it's in the hello.html template.

i think some many-to-many order stuff ended up on this branch but i'm too tired for git rn.

if you don't get to it first i'm happy to fix the source and git stuff and merge when i can get to it :)

Sorry about the the m2m stuff that I didn't catch before opening the PR. I did clean this up again, and nothing belonging there should be remaining here; Addresses your above comment regarding the .uri suffix as well.

3 commits
expand
Add getCounts XRPC equivalent to REST /links/count
Modify backlinks counting XRPC endpoint name
Mark /links/count REST endpoint as deprecated
expand 0 comments
1 commit
expand
Add getCounts XRPC equivalent to REST /links/count
expand 6 comments

Had to do some rebasing and cleanup, but good to go now I think.

sweet!

(btw the local/ gitignore is there for doing local/rocks.test, but i'm fine adding it to the top-level gitignore too!)

i think links.getCounts is a little too broad -- what do you think of links.getBacklinksCount?

we also throw a deprecated warning under /links/count on the main page template, matching the one for /links.

it's weird but the whitespace in the forms in try-it-macros is significant (one of my moments of knowingly doing things the wrong way because i was bored, sorry!)

the new form for this endpoint needs little tweaking to match the others.

Ah got it. The rocks tests ended up cluttering the top-level directory so I just added them to .gitignore. I often keep a .local file in .gitignores to ignore whatever's supposed to stay in one's local copy and somehow assumed this had the same intent.

Changed the function and endpoint/method name as requested to get_backlinks_count/getBacklinksCount

The endpoint in the try-it-macro should now match the other existing ones down to how they're formatted with whitespace

3 commits
expand
Make metrics collection opt-in
Increase constellation response limits
Add getCounts XRPC equivalent to REST /links/count
expand 0 comments
7 commits
expand
Make metrics collection opt-in
Increase constellation response limits
wip: m2m
Add tests for new get_many_to_many query handler
Fix get_m2m_empty test
Replace tuple with RecordsBySubject struct
Add getCounts XRPC equivalent to REST /links/count
expand 0 comments