bad-example.com
APIs for links and references in the ATmosphere
meta descriptions etc
+1
constellation/readme.md
+1
constellation/readme.md
···
4
5
- Self hostable: handles the full write throughput of the global atproto firehose on a raspberry pi 4b + single SSD
6
- Storage efficient: less than 2GB/day disk consumption indexing all references in all lexicons and all non-atproto URLs
7
- Simple JSON API
8
9
All social interactions in atproto tend to be represented by links (or references) between PDS records. This index can answer questions like "how many likes does a bsky post have", "who follows an account", "what are all the comments on a [frontpage](https://frontpage.fyi/) post", and more.
···
4
5
- Self hostable: handles the full write throughput of the global atproto firehose on a raspberry pi 4b + single SSD
6
- Storage efficient: less than 2GB/day disk consumption indexing all references in all lexicons and all non-atproto URLs
7
+
- Handles record deletion, account de/re-activation, and account deletion, ensuring accurate link counts and respecting users data choices
8
- Simple JSON API
9
10
All social interactions in atproto tend to be represented by links (or references) between PDS records. This index can answer questions like "how many likes does a bsky post have", "who follows an account", "what are all the comments on a [frontpage](https://frontpage.fyi/) post", and more.
+7
-4
constellation/src/server/mod.rs
+7
-4
constellation/src/server/mod.rs
···
4
use serde::{Deserialize, Serialize};
5
use serde_with::serde_as;
6
use std::collections::HashMap;
7
-
use std::time::{UNIX_EPOCH, Duration};
8
use tokio::net::{TcpListener, ToSocketAddrs};
9
use tokio::task::block_in_place;
10
use tokio_util::sync::CancellationToken;
···
21
const DEFAULT_CURSOR_LIMIT_MAX: u64 = 100;
22
23
const INDEX_BEGAN_AT_TS: u64 = 1738083600; // TODO: not this
24
-
25
26
pub async fn serve<S, A>(store: S, addr: A, stay_alive: CancellationToken) -> anyhow::Result<()>
27
where
···
106
days_indexed: u64,
107
stats: StorageStats,
108
}
109
-
fn hello(accept: ExtractAccept, store: impl LinkReader) -> Result<impl IntoResponse, http::StatusCode> {
110
let stats = store
111
.get_stats()
112
.map_err(|_| http::StatusCode::INTERNAL_SERVER_ERROR)?;
113
let days_indexed = (UNIX_EPOCH + Duration::from_secs(INDEX_BEGAN_AT_TS))
114
.elapsed()
115
.map_err(|_| http::StatusCode::INTERNAL_SERVER_ERROR)?
116
-
.as_secs() / 86400;
117
Ok(acceptable(accept, HelloReponse {
118
help: "open this URL in a web browser (or request with Accept: text/html) for information about this API.",
119
days_indexed,
···
4
use serde::{Deserialize, Serialize};
5
use serde_with::serde_as;
6
use std::collections::HashMap;
7
+
use std::time::{Duration, UNIX_EPOCH};
8
use tokio::net::{TcpListener, ToSocketAddrs};
9
use tokio::task::block_in_place;
10
use tokio_util::sync::CancellationToken;
···
21
const DEFAULT_CURSOR_LIMIT_MAX: u64 = 100;
22
23
const INDEX_BEGAN_AT_TS: u64 = 1738083600; // TODO: not this
24
25
pub async fn serve<S, A>(store: S, addr: A, stay_alive: CancellationToken) -> anyhow::Result<()>
26
where
···
105
days_indexed: u64,
106
stats: StorageStats,
107
}
108
+
fn hello(
109
+
accept: ExtractAccept,
110
+
store: impl LinkReader,
111
+
) -> Result<impl IntoResponse, http::StatusCode> {
112
let stats = store
113
.get_stats()
114
.map_err(|_| http::StatusCode::INTERNAL_SERVER_ERROR)?;
115
let days_indexed = (UNIX_EPOCH + Duration::from_secs(INDEX_BEGAN_AT_TS))
116
.elapsed()
117
.map_err(|_| http::StatusCode::INTERNAL_SERVER_ERROR)?
118
+
.as_secs()
119
+
/ 86400;
120
Ok(acceptable(accept, HelloReponse {
121
help: "open this URL in a web browser (or request with Accept: text/html) for information about this API.",
122
days_indexed,
+1
-1
constellation/src/storage/mod.rs
+1
-1
constellation/src/storage/mod.rs
+2
-1
constellation/templates/base.html.j2
+2
-1
constellation/templates/base.html.j2
···
2
<html lang="en">
3
<head>
4
<meta charset="utf8">
5
+
<title>{% block title %}{% endblock %} — Constellation</title>
6
<meta name="viewport" content="width=device-width, initial-scale=1" />
7
+
<meta name="description" content="{% block description %}Constellation is a self-hosted JSON API to an atproto-wide index of PDS record back-links. You can use it to query social interactions in real time.{% endblock %}" />
8
<style>
9
body {
10
font-family: sans-serif;
+2
-1
constellation/templates/dids-count.html.j2
+2
-1
constellation/templates/dids-count.html.j2
···
1
{% extends "base.html.j2" %}
2
{% import "try-it-macros.html.j2" as try_it %}
3
4
+
{% block title %}Distinct DIDs{% endblock %}
5
+
{% block description %}Count of distinct DIDs with {{ query.collection }} records linking to {{ query.target }} at JSON path {{ query.path }}{% endblock %}
6
7
{% block content %}
8
+1
constellation/templates/dids.html.j2
+1
constellation/templates/dids.html.j2
+1
constellation/templates/explore-links.html.j2
+1
constellation/templates/explore-links.html.j2
+2
constellation/templates/hello.html.j2
+2
constellation/templates/hello.html.j2
···
16
<li>and more</li>
17
</ul>
18
19
<p>
20
This server has indexed <span class="stat">{{ stats.linking_records|human_number }}</span> links between <span class="stat">{{ stats.targetables|human_number }}</span> targets and sources from <span class="stat">{{ stats.dids|human_number }}</span> identities over <span class="stat">{{ days_indexed|human_number }}</span> days.<br/>
21
<small>(indexing new records in real time, backfill still TODO)</small>
···
16
<li>and more</li>
17
</ul>
18
19
+
<p>It works by recursively walking <em>all</em> records coming through the firehose, searching for anything that looks like a link. Links are indexed by the target they point at, the collection the record came from, and the JSON path to the link in that record.</p>
20
+
21
<p>
22
This server has indexed <span class="stat">{{ stats.linking_records|human_number }}</span> links between <span class="stat">{{ stats.targetables|human_number }}</span> targets and sources from <span class="stat">{{ stats.dids|human_number }}</span> identities over <span class="stat">{{ days_indexed|human_number }}</span> days.<br/>
23
<small>(indexing new records in real time, backfill still TODO)</small>
+1
constellation/templates/links-count.html.j2
+1
constellation/templates/links-count.html.j2
+1
constellation/templates/links.html.j2
+1
constellation/templates/links.html.j2
+1
readme.md
+1
readme.md
···
11
12
- Self hostable: handles the full write throughput of the global atproto firehose on a raspberry pi 4b + single SSD
13
- Storage efficient: less than 2GB/day disk consumption indexing all references in all lexicons and all non-atproto URLs
14
- Simple JSON API
15
16
All social interactions in atproto tend to be represented by links (or references) between PDS records. This index can answer questions like "how many likes does a bsky post have", "who follows an account", "what are all the comments on a [frontpage](https://frontpage.fyi/) post", and more.
···
11
12
- Self hostable: handles the full write throughput of the global atproto firehose on a raspberry pi 4b + single SSD
13
- Storage efficient: less than 2GB/day disk consumption indexing all references in all lexicons and all non-atproto URLs
14
+
- Handles record deletion, account de/re-activation, and account deletion, ensuring accurate link counts and respecting users data choices
15
- Simple JSON API
16
17
All social interactions in atproto tend to be represented by links (or references) between PDS records. This index can answer questions like "how many likes does a bsky post have", "who follows an account", "what are all the comments on a [frontpage](https://frontpage.fyi/) post", and more.