Constellation, Spacedust, Slingshot, UFOs: atproto crates and services for microcosm

Convert existing REST /links/count endpoint to XRPC equivalent #8

closed opened by seoul.systems targeting main from seoul.systems/microcosm-rs: xrpc_backlinks_count

Add getCounts XRPC equivalent to REST /links/count

Simple conversion of the existing endpoint from REST to XRPC. Consequtively marked the pre-existing REST endpoint as deprecated.

In addition we now ignore rocks.test

Labels

None yet.

Participants 2
AT URI
at://did:plc:53wellrw53o7sw4zlpfenvuh/sh.tangled.repo.pull/3mcybnzocig22
+24 -4
Interdiff #1 #2
.gitignore

This file has not been changed.

.prettierrc

This file has not been changed.

constellation/src/server/mod.rs

This file has not been changed.

+1 -1
constellation/templates/dids.html.j2
··· 27 {% for did in linking_dids %} 28 <pre style="display: block; margin: 1em 2em" class="code"><strong>DID</strong>: {{ did.0 }} 29 -> see <a href="/links/all?target={{ did.0|urlencode }}">links to this DID</a> 30 - -> browse <a href="https://pdsls.dev/at://{{ did.0|urlencode }}">this DID record</a></pre> 31 {% endfor %} 32 33 {% if let Some(c) = cursor %}
··· 27 {% for did in linking_dids %} 28 <pre style="display: block; margin: 1em 2em" class="code"><strong>DID</strong>: {{ did.0 }} 29 -> see <a href="/links/all?target={{ did.0|urlencode }}">links to this DID</a> 30 + -> browse <a href="https://pdsls.dev/at://{{ did.0 }}">this DID record</a></pre> 31 {% endfor %} 32 33 {% if let Some(c) = cursor %}
constellation/templates/get-counts.html.j2

This file has not been changed.

constellation/templates/hello.html.j2

This file has not been changed.

constellation/templates/try-it-macros.html.j2

This file has not been changed.

lexicons/blue.microcosm/links/getBacklinks.json

This file has not been changed.

lexicons/blue.microcosm/links/getCounts.json

This file has not been changed.

lexicons/blue.microcosm/links/getManyToMany.json

This file has not been changed.

lexicons/blue.microcosm/links/getManyToManyCounts.json

This file has not been changed.

+2
spacedust/src/error.rs
··· 30 TooManySourcesWanted, 31 #[error("more wantedSubjectDids were requested than allowed (max 10,000)")] 32 TooManyDidsWanted, 33 #[error("more wantedSubjects were requested than allowed (max 50,000)")] 34 TooManySubjectsWanted, 35 }
··· 30 TooManySourcesWanted, 31 #[error("more wantedSubjectDids were requested than allowed (max 10,000)")] 32 TooManyDidsWanted, 33 + #[error("more wantedSubjectPrefixes were requested than allowed (max 100)")] 34 + TooManySubjectPrefixesWanted, 35 #[error("more wantedSubjects were requested than allowed (max 50,000)")] 36 TooManySubjectsWanted, 37 }
+11 -2
spacedust/src/server.rs
··· 227 #[serde(default)] 228 pub wanted_subjects: HashSet<String>, 229 #[serde(default)] 230 pub wanted_subject_dids: HashSet<String>, 231 #[serde(default)] 232 pub wanted_sources: HashSet<String>, ··· 241 /// 242 /// The at-uri must be url-encoded 243 /// 244 - /// Pass this parameter multiple times to specify multiple collections, like 245 /// `wantedSubjects=[...]&wantedSubjects=[...]` 246 pub wanted_subjects: String, 247 /// One or more DIDs to receive links about 248 /// 249 - /// Pass this parameter multiple times to specify multiple collections 250 pub wanted_subject_dids: String, 251 /// One or more link sources to receive links about 252 ///
··· 227 #[serde(default)] 228 pub wanted_subjects: HashSet<String>, 229 #[serde(default)] 230 + pub wanted_subject_prefixes: HashSet<String>, 231 + #[serde(default)] 232 pub wanted_subject_dids: HashSet<String>, 233 #[serde(default)] 234 pub wanted_sources: HashSet<String>, ··· 243 /// 244 /// The at-uri must be url-encoded 245 /// 246 + /// Pass this parameter multiple times to specify multiple subjects, like 247 /// `wantedSubjects=[...]&wantedSubjects=[...]` 248 pub wanted_subjects: String, 249 + /// One or more at-uri, URI, or DID prefixes to receive links about 250 + /// 251 + /// The uri must be url-encoded 252 + /// 253 + /// Pass this parameter multiple times to specify multiple prefixes, like 254 + /// `wantedSubjectPrefixes=[...]&wantedSubjectPrefixes=[...]` 255 + pub wanted_subject_prefixes: String, 256 /// One or more DIDs to receive links about 257 /// 258 + /// Pass this parameter multiple times to specify multiple subjects 259 pub wanted_subject_dids: String, 260 /// One or more link sources to receive links about 261 ///
+10 -1
spacedust/src/subscriber.rs
··· 124 let query = &self.query; 125 126 // subject + subject DIDs are logical OR 127 - if !(query.wanted_subjects.is_empty() && query.wanted_subject_dids.is_empty() 128 || query.wanted_subjects.contains(&properties.subject) 129 || properties 130 .subject_did 131 .as_ref() ··· 154 } 155 if opts.wanted_subject_dids.len() > 10_000 { 156 return Err(SubscriberUpdateError::TooManyDidsWanted); 157 } 158 if opts.wanted_subjects.len() > 50_000 { 159 return Err(SubscriberUpdateError::TooManySubjectsWanted);
··· 124 let query = &self.query; 125 126 // subject + subject DIDs are logical OR 127 + if !(query.wanted_subjects.is_empty() 128 + && query.wanted_subject_prefixes.is_empty() 129 + && query.wanted_subject_dids.is_empty() 130 || query.wanted_subjects.contains(&properties.subject) 131 + || query 132 + .wanted_subject_prefixes 133 + .iter() 134 + .any(|p| properties.subject.starts_with(p)) 135 || properties 136 .subject_did 137 .as_ref() ··· 160 } 161 if opts.wanted_subject_dids.len() > 10_000 { 162 return Err(SubscriberUpdateError::TooManyDidsWanted); 163 + } 164 + if opts.wanted_subject_prefixes.len() > 100 { 165 + return Err(SubscriberUpdateError::TooManySubjectPrefixesWanted); 166 } 167 if opts.wanted_subjects.len() > 50_000 { 168 return Err(SubscriberUpdateError::TooManySubjectsWanted);

History

8 rounds 13 comments
sign up or login to add to the discussion
8 commits
expand
Add getCounts XRPC equivalent to REST /links/count
Modify backlinks counting XRPC endpoint name
Mark /links/count REST endpoint as deprecated
Remove .uri suffix
Reformat existing lexicons
Remove wrongly commited getManyToMany lexicon
Fix Git whitespace error in "hello" template
Format .prettierrc and fix Git whitespace error
expand 3 comments

I'm giving up on the whitespace issue...

max@max-mbpro ~/dev/microcosm-rs (xrpc_backlinks_count) $ git diff --check upstream/main
max@max-mbpro ~/dev/microcosm-rs (xrpc_backlinks_count) $

git-diff --checked doesn't indicate any errors either.

As far as I can tell this seems to be a confirmed Tangled issue

happy to do merging locally as needed!

merged. thanks!

closed without merging
7 commits
expand
Add getCounts XRPC equivalent to REST /links/count
Modify backlinks counting XRPC endpoint name
Mark /links/count REST endpoint as deprecated
Remove .uri suffix
Reformat existing lexicons
Remove wrongly commited getManyToMany lexicon
Fix Git whitespace error in "hello" template
expand 0 comments
6 commits
expand
Add getCounts XRPC equivalent to REST /links/count
Modify backlinks counting XRPC endpoint name
Mark /links/count REST endpoint as deprecated
Remove .uri suffix
Reformat existing lexicons
Remove wrongly commited getManyToMany lexicon
expand 0 comments
3 commits
expand
Add getCounts XRPC equivalent to REST /links/count
Modify backlinks counting XRPC endpoint name
Mark /links/count REST endpoint as deprecated
expand 4 comments

Tangled somehow complains about merge conflicts here, but I couldn't find any after rebasing on upstream/main again?!

very weird!

one tiny thing left: the source for blocks is app.bsky.graph.block:subject (no .uri suffix on the path) -- it's in the hello.html template.

i think some many-to-many order stuff ended up on this branch but i'm too tired for git rn.

if you don't get to it first i'm happy to fix the source and git stuff and merge when i can get to it :)

Sorry about the the m2m stuff that I didn't catch before opening the PR. I did clean this up again, and nothing belonging there should be remaining here; Addresses your above comment regarding the .uri suffix as well.

3 commits
expand
Add getCounts XRPC equivalent to REST /links/count
Modify backlinks counting XRPC endpoint name
Mark /links/count REST endpoint as deprecated
expand 0 comments
1 commit
expand
Add getCounts XRPC equivalent to REST /links/count
expand 6 comments

Had to do some rebasing and cleanup, but good to go now I think.

sweet!

(btw the local/ gitignore is there for doing local/rocks.test, but i'm fine adding it to the top-level gitignore too!)

i think links.getCounts is a little too broad -- what do you think of links.getBacklinksCount?

we also throw a deprecated warning under /links/count on the main page template, matching the one for /links.

it's weird but the whitespace in the forms in try-it-macros is significant (one of my moments of knowingly doing things the wrong way because i was bored, sorry!)

the new form for this endpoint needs little tweaking to match the others.

Ah got it. The rocks tests ended up cluttering the top-level directory so I just added them to .gitignore. I often keep a .local file in .gitignores to ignore whatever's supposed to stay in one's local copy and somehow assumed this had the same intent.

Changed the function and endpoint/method name as requested to get_backlinks_count/getBacklinksCount

The endpoint in the try-it-macro should now match the other existing ones down to how they're formatted with whitespace

3 commits
expand
Make metrics collection opt-in
Increase constellation response limits
Add getCounts XRPC equivalent to REST /links/count
expand 0 comments
7 commits
expand
Make metrics collection opt-in
Increase constellation response limits
wip: m2m
Add tests for new get_many_to_many query handler
Fix get_m2m_empty test
Replace tuple with RecordsBySubject struct
Add getCounts XRPC equivalent to REST /links/count
expand 0 comments