microcosm.blue
Constellation, Spacedust, Slingshot, UFOs: atproto crates and services for microcosm
actually update the main readme a bit
bad-example.com
cozy-setup (move to another repo).md
legacy/cozy-setup (move to another repo).md
cozy-setup (move to another repo).md
legacy/cozy-setup (move to another repo).md
+35
legacy/old-readme-details.md
+35
legacy/old-readme-details.md
···
1
+
[Constellation](./constellation/)
2
+
--------------------------------------------
3
+
4
+
A global atproto backlink index ✨
5
+
6
+
- Self hostable: handles the full write throughput of the global atproto firehose on a raspberry pi 4b + single SSD
7
+
- Storage efficient: less than 2GB/day disk consumption indexing all references in all lexicons and all non-atproto URLs
8
+
- Handles record deletion, account de/re-activation, and account deletion, ensuring accurate link counts and respecting users data choices
9
+
- Simple JSON API
10
+
11
+
All social interactions in atproto tend to be represented by links (or references) between PDS records. This index can answer questions like "how many likes does a bsky post have", "who follows an account", "what are all the comments on a [frontpage](https://frontpage.fyi/) post", and more.
12
+
13
+
- **status**: works! api is unstable and likely to change, and no known instances have a full network backfill yet.
14
+
- source: [./constellation/](./constellation/)
15
+
- public instance: [constellation.microcosm.blue](https://constellation.microcosm.blue/)
16
+
17
+
_note: the public instance currently runs on a little raspberry pi in my house, feel free to use it! it comes with only with best-effort uptime, no commitment to not breaking the api for now, and possible rate-limiting. if you want to be nice you can put your project name and bsky username (or email) in your user-agent header for api requests._
18
+
19
+
20
+
App: Spacedust
21
+
--------------
22
+
23
+
A notification subscription service 💫
24
+
25
+
using the same "link source" concept as [constellation](./constellation/), offer webhook notifications for new references created to records
26
+
27
+
- **status**: in design
28
+
29
+
30
+
Library: [links](./links/)
31
+
------------------------------------
32
+
33
+
A rust crate (not published on crates.io yet) for optimistically parsing links out of arbitrary atproto PDS records, and potentially canonicalizing them
34
+
35
+
- **status**: unstable, might remain an internal lib for constellation (and spacedust, soon)
+123
legacy/original-notes.md
+123
legacy/original-notes.md
···
1
+
---
2
+
3
+
4
+
old notes follow, ignore
5
+
------------------------
6
+
7
+
8
+
as far as i can tell, atproto lexicons today don't follow much of a convention for referencing across documents: sometimes it's a StrongRef, sometimes it's a DID, sometimes it's a bare at-uri. lexicon authors choose any old link-sounding key name for the key in their document.
9
+
10
+
it's pretty messy so embrace the mess: atproto wants to be part of the web, so this library will also extract URLs and other URIs if you want it to. all the links.
11
+
12
+
13
+
why
14
+
---
15
+
16
+
the atproto firehose that bluesky sprays at you will contain raw _contents_ from peoples' pdses. these are isolated, decontextualized updates. it's very easy to build some kinds of interesting downstream apps off of this feed.
17
+
18
+
- bluesky posts (firesky, deletions, )
19
+
- blueksy post stats (emojis, )
20
+
- trending keywords ()
21
+
22
+
but bringing almost kind of _context_ into your project requires a big step up in complexity and potentially cost: you're entering "appview" territory. _how many likes does a post have? who follows this account?_
23
+
24
+
you own your atproto data: it's kept in your personal data repository (PDS) and noone else can write to it. when someone likes your post, they create a "like" record in their _own_ pds, and that like belongs to _them_, not to you/your post.
25
+
26
+
in the firehose you'll see a `app.bsky.feed.post` record created, with no details about who has liked it. then you'll see separate `app.bsky.feed.like` records show up for each like that comes in on that post, with no context about the post except a random-looking reference to it. storing these in order to do so is up to you!
27
+
28
+
**so, why**
29
+
30
+
everything is links, and they're a mess, but they all kinda work the same, so maybe some tooling can bring down that big step in complexity from firehose raw-content apps -> apps requiring any social context.
31
+
32
+
everything is links:
33
+
34
+
- likes
35
+
- follows
36
+
- blocks
37
+
- reposts
38
+
- quotes
39
+
40
+
some low-level things you could make from links:
41
+
42
+
- notification streams (part of ucosm)
43
+
- a global reverse index (part of ucosm)
44
+
45
+
i think that making these low-level services as easy to use as jetstream could open up pathways for building more atproto apps that operate at full scale with interesting features for reasonable effort at low cost to operate.
46
+
47
+
48
+
extracting links
49
+
---------------
50
+
51
+
52
+
- low-level: pass a &str of a field value and get a parsed link back
53
+
54
+
- med-level: pass a &str of record in json form and get a list of parsed links + json paths back. (todo: should also handle dag-cbor prob?)
55
+
56
+
- high-ish level: pass the json record and maybe apply some pre-loaded rules based on known lexicons to get the best result.
57
+
58
+
for now, a link is only considered if it matches for the entire value of the record's field -- links embedded in text content are not included. note that urls in bluesky posts _will_ still be extracted, since they are broken out into facets.
59
+
60
+
61
+
resolving / canonicalizing links
62
+
--------------------------------
63
+
64
+
65
+
### at-uris
66
+
67
+
every at-uri has at least two equivalent forms, one with a `DID`, and one with an account handle. the at-uri spec [illustrates this by example](https://atproto.com/specs/at-uri-scheme):
68
+
69
+
- `at://did:plc:44ybard66vv44zksje25o7dz/app.bsky.feed.post/3jwdwj2ctlk26`
70
+
- `at://bnewbold.bsky.team/app.bsky.feed.post/3jwdwj2ctlk26`
71
+
72
+
some applications, like a reverse link index, may wish to canonicalize at-uris to a single form. the `DID`-form is stable as an account changes its handle and probably the right choice to canonicalize to, but maybe some apps would actually perfer to canonicalise to handles?
73
+
74
+
hopefully atrium will make it easy to resolve at-uris.
75
+
76
+
77
+
### urls
78
+
79
+
canonicalizing URLs is more annoying but also a bit more established. lots of details.
80
+
81
+
- do we have to deal with punycode?
82
+
- follow redirects (todo: only permanent ones, or all?)
83
+
- check for rel=canonical http header and possibly follow it
84
+
- check link rel=canonical meta tag and possibly follow it
85
+
- do we need to check site maps??
86
+
- do we have to care at all about AMP?
87
+
- do we want anything to do with url shorteners??
88
+
- how do multilingual sites affect this?
89
+
- do we have to care about `script type="application/ld+json"` ???
90
+
91
+
ugh. is there a crate for this.
92
+
93
+
94
+
### relative uris?
95
+
96
+
links might be relative, in which case they might need to be made absolute before being useful. is that a concern for this library, or up to the user? (seems like we might not have context here to determine its absolute)
97
+
98
+
99
+
### canonicalizing
100
+
101
+
there should be a few async functions available to canonicalize already-parsed links.
102
+
103
+
- what happens if a link can't be resolved?
104
+
105
+
106
+
---
107
+
108
+
- using `tinyjson` because it's nice -- maybe should switch to serde_json to share deps with atrium?
109
+
110
+
- would use atrium for parsing at-uris, but it's not in there. there's a did-only version in the non-lib commands.rs. its identifier parser is strict to did + handle, which makes sense, but for our purposes we might want to allow unknown methods too?
111
+
112
+
- rsky-syntax has an aturi
113
+
- adenosyne also
114
+
- might come back to these
115
+
116
+
117
+
-------
118
+
119
+
rocks
120
+
121
+
```bash
122
+
ROCKSDB_LIB_DIR=/nix/store/z2chn0hsik0clridr8mlprx1cngh1g3c-rocksdb-9.7.3/lib/ cargo build
123
+
```
+41
-137
readme.md
+41
-137
readme.md
···
1
-
microcosm: links
2
-
================
3
-
4
-
this repo contains libraries and apps for working with cross-record references in at-protocol.
5
-
1
+
microcosm
2
+
=========
6
3
7
-
App: [Constellation](./constellation/)
8
-
--------------------------------------------
4
+
This repo contains APIs and libraries for [atproto](https://atproto.com/) services from [microcosm](https://microcosm.blue):
9
5
10
-
A global atproto backlink index ✨
11
6
12
-
- Self hostable: handles the full write throughput of the global atproto firehose on a raspberry pi 4b + single SSD
13
-
- Storage efficient: less than 2GB/day disk consumption indexing all references in all lexicons and all non-atproto URLs
14
-
- Handles record deletion, account de/re-activation, and account deletion, ensuring accurate link counts and respecting users data choices
15
-
- Simple JSON API
16
-
17
-
All social interactions in atproto tend to be represented by links (or references) between PDS records. This index can answer questions like "how many likes does a bsky post have", "who follows an account", "what are all the comments on a [frontpage](https://frontpage.fyi/) post", and more.
18
-
19
-
- **status**: works! api is unstable and likely to change, and no known instances have a full network backfill yet.
20
-
- source: [./constellation/](./constellation/)
21
-
- public instance: [constellation.microcosm.blue](https://constellation.microcosm.blue/)
22
-
23
-
_note: the public instance currently runs on a little raspberry pi in my house, feel free to use it! it comes with only with best-effort uptime, no commitment to not breaking the api for now, and possible rate-limiting. if you want to be nice you can put your project name and bsky username (or email) in your user-agent header for api requests._
24
-
25
-
26
-
App: Spacedust
27
-
--------------
28
-
29
-
A notification subscription service 💫
30
-
31
-
using the same "link source" concept as [constellation](./constellation/), offer webhook notifications for new references created to records
32
-
33
-
- **status**: in design
34
-
35
-
36
-
Library: [links](./links/)
7
+
🌌 [Constellation](./constellation/)
37
8
------------------------------------
38
9
39
-
A rust crate (not published on crates.io yet) for optimistically parsing links out of arbitrary atproto PDS records, and potentially canonicalizing them
40
-
41
-
- **status**: unstable, might remain an internal lib for constellation (and spacedust, soon)
42
-
43
-
44
-
45
-
---
46
-
47
-
48
-
old notes follow, ignore
49
-
------------------------
50
-
51
-
52
-
as far as i can tell, atproto lexicons today don't follow much of a convention for referencing across documents: sometimes it's a StrongRef, sometimes it's a DID, sometimes it's a bare at-uri. lexicon authors choose any old link-sounding key name for the key in their document.
53
-
54
-
it's pretty messy so embrace the mess: atproto wants to be part of the web, so this library will also extract URLs and other URIs if you want it to. all the links.
55
-
56
-
57
-
why
58
-
---
59
-
60
-
the atproto firehose that bluesky sprays at you will contain raw _contents_ from peoples' pdses. these are isolated, decontextualized updates. it's very easy to build some kinds of interesting downstream apps off of this feed.
61
-
62
-
- bluesky posts (firesky, deletions, )
63
-
- blueksy post stats (emojis, )
64
-
- trending keywords ()
65
-
66
-
but bringing almost kind of _context_ into your project requires a big step up in complexity and potentially cost: you're entering "appview" territory. _how many likes does a post have? who follows this account?_
67
-
68
-
you own your atproto data: it's kept in your personal data repository (PDS) and noone else can write to it. when someone likes your post, they create a "like" record in their _own_ pds, and that like belongs to _them_, not to you/your post.
69
-
70
-
in the firehose you'll see a `app.bsky.feed.post` record created, with no details about who has liked it. then you'll see separate `app.bsky.feed.like` records show up for each like that comes in on that post, with no context about the post except a random-looking reference to it. storing these in order to do so is up to you!
71
-
72
-
**so, why**
73
-
74
-
everything is links, and they're a mess, but they all kinda work the same, so maybe some tooling can bring down that big step in complexity from firehose raw-content apps -> apps requiring any social context.
75
-
76
-
everything is links:
77
-
78
-
- likes
79
-
- follows
80
-
- blocks
81
-
- reposts
82
-
- quotes
83
-
84
-
some low-level things you could make from links:
85
-
86
-
- notification streams (part of ucosm)
87
-
- a global reverse index (part of ucosm)
88
-
89
-
i think that making these low-level services as easy to use as jetstream could open up pathways for building more atproto apps that operate at full scale with interesting features for reasonable effort at low cost to operate.
90
-
10
+
A global atproto interactions backlink index as a simple JSON API. Works with every lexicon, runs on a raspberry pi, consumes less than 2GiB of disk per day. Handles record deletion, account de/re-activation, and account deletion, ensuring accurate link counts while respecting users' data choices.
91
11
92
-
extracting links
93
-
---------------
12
+
- source: [./constellation/](./constellation/)
13
+
- [public instance + API docs](https://constellation.microcosm.blue/)
14
+
- status: used in production. APIs will change but backwards compatibility will be maintained as long as needed.
94
15
95
16
96
-
- low-level: pass a &str of a field value and get a parsed link back
17
+
🎇 [Spacedust](./spacedust/)
18
+
----------------------------
97
19
98
-
- med-level: pass a &str of record in json form and get a list of parsed links + json paths back. (todo: should also handle dag-cbor prob?)
20
+
A global atproto interactions firehose. Extracts all at-uris, DIDs, and URLs from every lexicon in the firehose, and exposes them over a websocket modelled after [jetstream](github.com/bluesky-social/jetstream).
99
21
100
-
- high-ish level: pass the json record and maybe apply some pre-loaded rules based on known lexicons to get the best result.
22
+
- source: [./spacedust/](./spacedust/)
23
+
- [public instance + API docs](https://spacedust.microcosm.blue/)
24
+
- status: v0: the basics work and the APIs are in place! missing cursor replay, forward link storage, and delete event link hydration.
101
25
102
-
for now, a link is only considered if it matches for the entire value of the record's field -- links embedded in text content are not included. note that urls in bluesky posts _will_ still be extracted, since they are broken out into facets.
26
+
Demos:
103
27
28
+
- [Spacedust notifications](https://notifications.microcosm.blue/): web push notifications for _every_ atproto app
29
+
- [Zero-Bluesky real-time interaction-updating post embed](https://bsky.bad-example.com/zero-bluesky-realtime-embed/)
104
30
105
-
resolving / canonicalizing links
106
-
--------------------------------
107
31
32
+
🛰️ [Slingshot](./slingshot)
33
+
---------------------------
108
34
109
-
### at-uris
35
+
A fast, eager, production-grade edge cache for atproto records and identities. Pre-caches all records from the firehose and maintains a longer-term cache of requested records on disk.
110
36
111
-
every at-uri has at least two equivalent forms, one with a `DID`, and one with an account handle. the at-uri spec [illustrates this by example](https://atproto.com/specs/at-uri-scheme):
37
+
- source: [./slingshot/](./slingshot/)
38
+
- [public instance + API docs](https://slingshot.microcosm.blue/)
39
+
- status: v0: most XRPC APIs are working. cache storage is being reworked.
112
40
113
-
- `at://did:plc:44ybard66vv44zksje25o7dz/app.bsky.feed.post/3jwdwj2ctlk26`
114
-
- `at://bnewbold.bsky.team/app.bsky.feed.post/3jwdwj2ctlk26`
115
41
116
-
some applications, like a reverse link index, may wish to canonicalize at-uris to a single form. the `DID`-form is stable as an account changes its handle and probably the right choice to canonicalize to, but maybe some apps would actually perfer to canonicalise to handles?
42
+
🛸 [UFOs API](./ufos)
43
+
---------------------
117
44
118
-
hopefully atrium will make it easy to resolve at-uris.
45
+
Timeseries stats and sample records for every [collection](https://atproto.com/guides/glossary#collection) ever seen in the atproto firehose. Unique users are counted in hyperloglog sketches enabling arbitrary cardinality aggregation across time buckets and/or NSIDs.
119
46
47
+
- source: [./ufos/](./ufos/)
48
+
- [public instance + API docs](https://ufos-api.microcosm.blue/)
49
+
- status: Used in production. It has APIs and they work! Needs improvement on indexing; needs more indexes and some more APIs to the data exposed.
120
50
121
-
### urls
51
+
See also: [UFOs atproto explorer](https://ufos.microcosm.blue/) built on UFOs API. ([source](github.com/at-microcosm/spacedust-utils))
122
52
123
-
canonicalizing URLs is more annoying but also a bit more established. lots of details.
124
53
125
-
- do we have to deal with punycode?
126
-
- follow redirects (todo: only permanent ones, or all?)
127
-
- check for rel=canonical http header and possibly follow it
128
-
- check link rel=canonical meta tag and possibly follow it
129
-
- do we need to check site maps??
130
-
- do we have to care at all about AMP?
131
-
- do we want anything to do with url shorteners??
132
-
- how do multilingual sites affect this?
133
-
- do we have to care about `script type="application/ld+json"` ???
134
-
135
-
ugh. is there a crate for this.
136
-
137
-
138
-
### relative uris?
139
-
140
-
links might be relative, in which case they might need to be made absolute before being useful. is that a concern for this library, or up to the user? (seems like we might not have context here to determine its absolute)
141
-
142
-
143
-
### canonicalizing
144
-
145
-
there should be a few async functions available to canonicalize already-parsed links.
146
-
147
-
- what happens if a link can't be resolved?
54
+
💫 [Links](./links)
55
+
-------------------
148
56
149
-
150
-
---
57
+
Rust library for parsing and extracting links (at-uris, DIDs, and URLs) from atproto records.
151
58
152
-
- using `tinyjson` because it's nice -- maybe should switch to serde_json to share deps with atrium?
59
+
- source: [./links/](./links/)
60
+
- status: not yet published to crates.io; needs some rework
153
61
154
-
- would use atrium for parsing at-uris, but it's not in there. there's a did-only version in the non-lib commands.rs. its identifier parser is strict to did + handle, which makes sense, but for our purposes we might want to allow unknown methods too?
155
-
156
-
- rsky-syntax has an aturi
157
-
- adenosyne also
158
-
- might come back to these
159
62
63
+
🔭 Deprecated: [Who am I](./who-am-i)
64
+
-------------------------------------
160
65
161
-
-------
66
+
An identity bridge for microcosm demos, that kinda worked. Fixing its problems is about equivalent to reinventing a lot of OIDC, so it's being retired.
162
67
163
-
rocks
68
+
- source: [./who-am-i/](./who-am-i/)
69
+
- status: ready for retirement.
164
70
165
-
```bash
166
-
ROCKSDB_LIB_DIR=/nix/store/z2chn0hsik0clridr8mlprx1cngh1g3c-rocksdb-9.7.3/lib/ cargo build
167
-
```
71
+
Still in use for the Spacedust Notifications demo, but that will hopefully be migrated to use atproto oauth directly instead.
ufos ops (move to micro-ops).md
legacy/ufos ops (move to micro-ops).md
ufos ops (move to micro-ops).md
legacy/ufos ops (move to micro-ops).md