vicwalker.dev.br / pds-mirror
Mirror from bluesky-social/pds
fork atom

README updates (#34)

* update env

* updated to remove manual instructions and explain current state, etc

* formatting

* tweaks

---------

Co-authored-by: Jake Gold <jake@blueskyweb.xyz>

authored by

Daniel Holmgren
Jake Gold and committed by
GitHub 26c06028 643a9ffa

+86 -282
+60 -282
README.md
··· 4 4 5 5 ## Table of Contents 6 6 7 - * [FAQ](#faq) 7 + <!-- markdown-toc -i README.md --> 8 + 9 + <!-- toc --> 10 + 11 + - [FAQ](#faq) 8 12 * [What is Bluesky?](#what-is-bluesky) 9 13 * [What is AT Protocol?](#what-is-at-protocol) 10 - * [How can developers get invite codes?](#how-can-developers-get-invite-codes) 11 14 * [Where is the code?](#where-is-the-code) 12 15 * [What is the current status of federation?](#what-is-the-current-status-of-federation) 13 16 * [What should I know about running a PDS in the developer sandbox?](#what-should-i-know-about-running-a-pds-in-the-developer-sandbox) 14 - * [Self\-hosting PDS](#self-hosting-pds) 15 - * [Preparation for self\-hosting PDS](#preparation-for-self-hosting-pds) 17 + - [Self-hosting PDS](#self-hosting-pds) 18 + * [Preparation for self-hosting PDS](#preparation-for-self-hosting-pds) 16 19 * [Open your cloud firewall for HTTP and HTTPS](#open-your-cloud-firewall-for-http-and-https) 17 20 * [Configure DNS for your domain](#configure-dns-for-your-domain) 18 21 * [Check that DNS is working as expected](#check-that-dns-is-working-as-expected) 19 - * [Automatic install on Ubuntu 20\.04/22\.04 or Debian 11/12](#automatic-install-on-ubuntu-20042204-or-debian-1112) 20 - * [Installing manually on Ubuntu 22\.04](#installing-manually-on-ubuntu-2204) 21 - * [Open ports on your Linux firewall](#open-ports-on-your-linux-firewall) 22 - * [Install Docker](#install-docker) 23 - * [Uninstall old versions](#uninstall-old-versions) 24 - * [Set up the repository](#set-up-the-repository) 25 - * [Install Docker Engine](#install-docker-engine) 26 - * [Verify Docker Engine installation](#verify-docker-engine-installation) 27 - * [Set up the PDS directory](#set-up-the-pds-directory) 28 - * [Create the Caddyfile](#create-the-caddyfile) 29 - * [Create the PDS env configuration file](#create-the-pds-env-configuration-file) 30 - * [Start the PDS containers](#start-the-pds-containers) 31 - * [Download the Docker compose file](#download-the-docker-compose-file) 32 - * [Create the systemd service](#create-the-systemd-service) 33 - * [Start the service](#start-the-service) 34 - * [Verify your PDS is online](#verify-your-pds-is-online) 35 - * [Obtain your PDS admin password](#obtain-your-pds-admin-password) 36 - * [Generate an invite code for your PDS](#generate-an-invite-code-for-your-pds) 37 - * [Connecting to your server](#connecting-to-your-server) 38 - * [Manually updating your PDS](#manually-updating-your-pds) 39 - * [PDS environment variables](#pds-environment-variables) 22 + * [Installer on Ubuntu 20.04/22.04 and Debian 11/12](#installer-on-ubuntu-20042204-and-debian-1112) 23 + * [Verifying that your PDS is online and accessible](#verifying-that-your-pds-is-online-and-accessible) 24 + * [Creating an account using pdsadmin](#creating-an-account-using-pdsadmin) 25 + * [Creating an account using an invite code](#creating-an-account-using-an-invite-code) 26 + * [Using the Bluesky app with your PDS](#using-the-bluesky-app-with-your-pds) 27 + * [Updating your PDS](#updating-your-pds) 40 28 29 + <!-- tocstop --> 41 30 42 31 ## FAQ 43 32 ··· 45 34 46 35 Bluesky is a social media application built on AT Protocol. 47 36 48 - Please visit the [Bluesky website](https://bsky.app/) for more information. 37 + Please visit the [Bluesky website](https://bsky.social/) for more information. 49 38 50 39 ### What is AT Protocol? 51 40 ··· 53 42 54 43 Please visit the [AT Protocol docs](https://atproto.com/guides/overview) for additional information. 55 44 56 - ### How can developers get invite codes? 57 - 58 - There is no invite required to join the sandbox network. Simply set up your own PDS and generate your own invite codes to create accounts. If you desire an account on the production network (on the official Bluesky PDS) please check out the [Bluesky Developer Waitlist](https://docs.google.com/forms/d/e/1FAIpQLSfCuguykw3HaPxIZMJQKRu8_-vsRew90NALVTDOjCSPDmsGNg/viewform) which prioritizes access for developers wanting to build software on atproto. 59 - 60 45 ### Where is the code? 61 46 62 - * [Canonical TypeScript code](https://github.com/bluesky-social/atproto) 63 - * [Experimental Go code](https://github.com/bluesky-social/indigo) 47 + * [TypeScript code](https://github.com/bluesky-social/atproto) 48 + * [Go code](https://github.com/bluesky-social/indigo) 64 49 65 50 ### What is the current status of federation? 66 51 67 - We do not currently support PDS federation on the production network but it is now possible to federate in the developer sandbox. 52 + As of Feb, 2024, the AT Protocol data service (PDS) is now open to federation for self-hosters! 53 + 54 + ✅ Federated domain handles (e.g. `@nytimes.com`) 55 + 56 + ✅ Federated feed generators (custom algorithms) 57 + 58 + ✅ Federated relays (event firehose) 59 + 60 + ✅ Federated app views (API service) 61 + 62 + ✅ Federated data for self-hosters (PDS hosting) 63 + 64 + 🟩 Federated moderation (labeling) (coming soon) 65 + 66 + 🟩 Federated data for large service providers (coming soon) 68 67 69 68 ### What should I know about running a PDS in the developer sandbox? 69 + 70 + Developers may now run self-hosted PDS hosts on the production network! 71 + 72 + Though it is still recommended to run experiments in the developer sandbox network. 70 73 71 74 Read the [SANDBOX.md](https://github.com/bluesky-social/pds/blob/main/SANDBOX.md) for an overview of the sandbox network. 72 75 ··· 134 137 135 138 These should all return your server's public IP. 136 139 137 - ### Automatic install on Ubuntu 20.04/22.04 or Debian 11/12 140 + ### Installer on Ubuntu 20.04/22.04 and Debian 11/12 138 141 139 - On your server via ssh, run the installer script: 142 + On your server via ssh, download the installer script using wget: 140 143 141 144 ```bash 142 145 wget https://raw.githubusercontent.com/bluesky-social/pds/main/installer.sh 143 146 ``` 144 147 145 - ```bash 146 - sudo bash installer.sh 147 - ``` 148 - 149 - ### Installing manually on Ubuntu 22.04 150 - 151 - #### Open ports on your Linux firewall 152 - 153 - If your server is running a Linux firewall managed with `ufw`, you will need to open these ports: 154 - 155 - ```bash 156 - $ sudo ufw allow 80/tcp 157 - $ sudo ufw allow 443/tcp 158 - ``` 159 - 160 - #### Install Docker 161 - 162 - On your server, install Docker CE (Community Edition), using the the following instructions. For other operating systems you may reference the [official Docker install guides](https://docs.docker.com/engine/install/). 163 - 164 - **Note:** All of the following commands should be run on your server via ssh. 165 - 166 - ##### Uninstall old versions 167 - 168 - ```bash 169 - sudo apt-get remove docker docker-engine docker.io containerd runc 170 - ``` 171 - 172 - ##### Set up the repository 173 - 174 - ```bash 175 - sudo apt-get update 176 - sudo apt-get install \ 177 - ca-certificates \ 178 - curl \ 179 - gnupg 180 - ``` 181 - 182 - ```bash 183 - sudo install -m 0755 -d /etc/apt/keyrings 184 - curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg 185 - sudo chmod a+r /etc/apt/keyrings/docker.gpg 186 - ``` 187 - 188 - ```bash 189 - echo \ 190 - "deb [arch="$(dpkg --print-architecture)" signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu \ 191 - "$(. /etc/os-release && echo "$VERSION_CODENAME")" stable" | \ 192 - sudo tee /etc/apt/sources.list.d/docker.list > /dev/null 193 - ``` 194 - 195 - ##### Install Docker Engine 196 - 197 - ```bash 198 - sudo apt-get update 199 - ``` 200 - 201 - ```bash 202 - sudo apt-get install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin 203 - ``` 204 - 205 - ##### Verify Docker Engine installation 206 - 207 - ```bash 208 - sudo docker run hello-world 209 - ``` 210 - 211 - #### Set up the PDS directory 212 - 213 - ```bash 214 - sudo mkdir /pds 215 - sudo mkdir --parents /pds/caddy/data 216 - sudo mkdir --parents /pds/caddy/etc/caddy 217 - ``` 218 - 219 - #### Create the Caddyfile 220 - 221 - Be sure to replace `example.com` with your own domain. 222 - 223 - ```bash 224 - cat <<CADDYFILE | sudo tee /pds/caddy/etc/caddy/Caddyfile 225 - { 226 - email you@example.com 227 - on_demand_tls { 228 - ask http://localhost:3000 229 - } 230 - } 231 - 232 - *.example.com, example.com { 233 - tls { 234 - on_demand 235 - } 236 - reverse_proxy http://localhost:3000 237 - } 238 - CADDYFILE 239 - ``` 240 - 241 - #### Create the PDS env configuration file 242 - 243 - You should fill in the first 5 values, but leave the rest untouched unless you have good reason to change it. 244 - 245 - See the PDS environment variables section at the end of this README for explanations of each value 246 - 247 - Your PDS will need two secp256k1 private keys provided as hex strings. You can securely generate these keys using `openssl` with the following command: 248 - 249 - **Note:** 250 - * Replace `example.com` with your domain name. 251 - 252 - ```bash 253 - PDS_HOSTNAME="example.com" 254 - PDS_JWT_SECRET="$(openssl rand --hex 16)" 255 - PDS_ADMIN_PASSWORD="$(openssl rand --hex 16)" 256 - PDS_REPO_SIGNING_KEY_K256_PRIVATE_KEY_HEX="$(openssl ecparam --name secp256k1 --genkey --noout --outform DER | tail --bytes=+8 | head --bytes=32 | xxd --plain --cols 32)" 257 - PDS_PLC_ROTATION_KEY_K256_PRIVATE_KEY_HEX="$(openssl ecparam --name secp256k1 --genkey --noout --outform DER | tail --bytes=+8 | head --bytes=32 | xxd --plain --cols 32)" 258 - 259 - cat <<PDS_CONFIG | sudo tee /pds/pds.env 260 - PDS_HOSTNAME=${PDS_HOSTNAME} 261 - PDS_JWT_SECRET=${PDS_JWT_SECRET} 262 - PDS_ADMIN_PASSWORD=${PDS_ADMIN_PASSWORD} 263 - PDS_REPO_SIGNING_KEY_K256_PRIVATE_KEY_HEX=${PDS_REPO_SIGNING_KEY_K256_PRIVATE_KEY_HEX} 264 - PDS_PLC_ROTATION_KEY_K256_PRIVATE_KEY_HEX=${PDS_PLC_ROTATION_KEY_K256_PRIVATE_KEY_HEX} 265 - PDS_DB_SQLITE_LOCATION=/pds/pds.sqlite 266 - PDS_BLOBSTORE_DISK_LOCATION=/pds/blocks 267 - PDS_DID_PLC_URL=https://plc.bsky-sandbox.dev 268 - PDS_BSKY_APP_VIEW_URL=https://api.bsky-sandbox.dev 269 - PDS_BSKY_APP_VIEW_DID=did:web:api.bsky-sandbox.dev 270 - PDS_CRAWLERS=https://bgs.bsky-sandbox.dev 271 - PDS_CONFIG 272 - ``` 273 - 274 - #### Start the PDS containers 275 - 276 - ##### Download the Docker compose file 277 - 278 - Download the `compose.yaml` to run your PDS, which includes the following containers: 279 - 280 - * `pds` Node PDS server running on http://localhost:3000 281 - * `caddy` HTTP reverse proxy handling TLS and proxying requests to the PDS server 282 - * `watchtower` Daemon responsible for auto-updating containers to keep the server secure and federating 283 - 284 - ```bash 285 - curl https://raw.githubusercontent.com/bluesky-social/pds/main/compose.yaml | sudo tee /pds/compose.yaml 286 - ``` 287 - 288 - ##### Create the systemd service 148 + or download it using curl: 289 149 290 150 ```bash 291 - cat <<SYSTEMD_UNIT_FILE | sudo tee /etc/systemd/system/pds.service 292 - [Unit] 293 - Description=Bluesky PDS Service 294 - Documentation=https://github.com/bluesky-social/pds 295 - Requires=docker.service 296 - After=docker.service 297 - 298 - [Service] 299 - Type=oneshot 300 - RemainAfterExit=yes 301 - WorkingDirectory=/pds 302 - ExecStart=/usr/bin/docker compose --file /pds/compose.yaml up --detach 303 - ExecStop=/usr/bin/docker compose --file /pds/compose.yaml down 304 - 305 - [Install] 306 - WantedBy=default.target 307 - SYSTEMD_UNIT_FILE 151 + curl https://raw.githubusercontent.com/bluesky-social/pds/main/installer.sh >installer.sh 308 152 ``` 309 153 310 - ##### Start the service 154 + And then run the installer using bash: 311 155 312 - **Reload the systemd daemon to create the new service:** 313 156 ```bash 314 - sudo systemctl daemon-reload 157 + sudo bash installer.sh 315 158 ``` 316 159 317 - **Enable the systemd service:** 318 - ```bash 319 - sudo systemctl enable pds 320 - ``` 160 + ### Verifying that your PDS is online and accessible 321 161 322 - **Start the pds systemd service:** 323 - ```bash 324 - sudo systemctl start pds 325 - ``` 162 + You can check if your server is online and healthy by requesting the healthcheck endpoint. 326 163 327 - **Ensure that containers are running** 164 + You can visit `https://example.com/xrpc/_health` in your browser. You should see a JSON response with a version. 328 165 329 - There should be a caddy, pds, and watchtower container running. 166 + For example: 330 167 331 - ```bash 332 - sudo systemctl status pds 333 168 ``` 334 - 335 - ```bash 336 - sudo docker ps 337 - ``` 338 - 339 - ### Verify your PDS is online 340 - 341 - You can check if your server is online and healthy by requesting the healthcheck endpoint. 342 - 343 - ```bash 344 - curl https://example.com/xrpc/_health 345 169 {"version":"0.2.2-beta.2"} 346 170 ``` 347 171 348 - ### Obtain your PDS admin password 349 - 350 - Your PDS admin password should be in your `pds.env` file if you used the installer script. 172 + ### Creating an account using pdsadmin 351 173 352 - **For example:** 174 + Using ssh on your server, use `pdsadmin` to create an account if you haven't already. 353 175 354 176 ```bash 355 - $ source /pds/pds.env 356 - $ echo $PDS_ADMIN_PASSWORD 357 - a7b5970b6a5077bb41fc68a26d30adda 177 + sudo pdsadmin account create 358 178 ``` 359 - ### Generate an invite code for your PDS 360 179 361 - By default, your PDS will require an invite code to create an account. 180 + ### Creating an account using an invite code 362 181 363 - You can generate a new invite code with the following command: 182 + Using ssh on your server, use `pdsadmin` to create an invite code. 364 183 365 184 ```bash 366 - PDS_HOSTNAME="example.com" 367 - PDS_ADMIN_PASSWORD="<YOUR PDS ADMIN PASSWORD>" 368 - 369 - curl --silent \ 370 - --show-error \ 371 - --request POST \ 372 - --user "admin:${PDS_ADMIN_PASSWORD}" \ 373 - --header "Content-Type: application/json" \ 374 - --data '{"useCount": 1}' \ 375 - https://${PDS_HOSTNAME}/xrpc/com.atproto.server.createInviteCode 185 + sudo pdsadmin create-invite-code 376 186 ``` 377 187 378 - **Note:** the `useCount` field specifies how many times an invite code can be used 188 + When creating an account using the app, enter this invite code. 379 189 380 - ### Connecting to your server 190 + ### Using the Bluesky app with your PDS 381 191 382 - You can use the Bluesky app to connect to your server to create an account. 192 + You can use the Bluesky app to connect to your PDS. 383 193 384 194 1. Get the Bluesky app 385 - * [Bluesky for Web (sandbox)](https://app.bsky-sandbox.dev/) 195 + * [Bluesky for Web](https://bsky.app/) 386 196 * [Bluesky for iPhone](https://apps.apple.com/us/app/bluesky-social/id6444370199) 387 197 * [Bluesky for Android](https://play.google.com/store/apps/details?id=xyz.blueskyweb.app) 388 198 1. Enter the URL of your PDS (e.g. `https://example.com/`) 389 - 1. Create an account using the generated invite code 390 - 1. Create a post 391 199 392 - _Note: because we use on-the-fly TLS certs, it may take 10-30s for your handle to be accessible. If you aren't seeing your first post/profile, wait 30s and try to make another post._ 200 + _Note: because the subdomain TLS certificate is created on-demand, it may take 10-30s for your handle to be accessible. If you aren't seeing your first post/profile, wait 30s and try to make another post._ 393 201 394 - Checkout [SANDBOX.md](./SANDBOX.md) for an overview of participating in the sandbox network. 202 + ### Updating your PDS 395 203 396 - ### Manually updating your PDS 204 + It is recommended that you keep your PDS up to date with new versions, otherwise things may break. You can use the `pdsadmin` tool to update your PDS. 397 205 398 - If you use use Docker `compose.yaml` file in this repo, your PDS will automatically update nightly. To manually update to the latest version use the following commands. 399 - 400 - **Pull the latest PDS container image:** 401 206 ```bash 402 - sudo docker pull ghcr.io/bluesky-social/pds:latest 403 - ``` 404 - 405 - **Restart PDS with the new container image:** 406 - ```bash 407 - sudo systemctl restart pds 207 + sudo pdsadmin update 408 208 ``` 409 - 410 - ## PDS environment variables 411 - 412 - You will need to customize various settings configured through the PDS environment variables. See the below table to find the variables you'll need to set. 413 - 414 - | Environment Variable | Value | Should update? | Notes | 415 - | ----------------------------------------- | ---------------------------- | -------------- | --------------------------------------------------------------------------- | 416 - | PDS_HOSTNAME | example.com | ✅ | Public domain you intend to deploy your service at | 417 - | PDS_JWT_SECRET | jwt-secret | ✅ | Use a secure high-entropy string that is 32 characters in length | 418 - | PDS_ADMIN_PASSWORD | admin-pass | ✅ | Use a secure high-entropy string that is 32 characters in length | 419 - | PDS_REPO_SIGNING_KEY_K256_PRIVATE_KEY_HEX | 3ee68... | ✅ | See above Generate Keys section - once set, do not change | 420 - | PDS_PLC_ROTATION_KEY_K256_PRIVATE_KEY_HEX | e049f... | ✅ | See above Generate Keys section - once set, do not change | 421 - | PDS_DB_SQLITE_LOCATION | /pds/pds.sqlite | ❌ | Or use `PDS_DB_POSTGRES_URL` depending on which database you intend to use | 422 - | PDS_BLOBSTORE_DISK_LOCATION | /pds/blocks | ❌ | Only update if you update the mounted volume for your docker image as well | 423 - | PDS_DID_PLC_URL | https://plc.bsky-sandbox.dev | ❌ | Do not adjust if you intend to federate with the Bluesky federation sandbox | 424 - | PDS_BSKY_APP_VIEW_URL | https://api.bsky-sandbox.dev | ❌ | Do not adjust if you intend to federate with the Bluesky federation sandbox | 425 - | PDS_BSKY_APP_VIEW_DID | did:web:api.bsky-sandbox.dev | ❌ | Do not adjust if you intend to federate with the Bluesky federation sandbox | 426 - | PDS_CRAWLERS | https://bgs.bsky-sandbox.dev | ❌ | Do not adjust if you intend to federate with the Bluesky federation sandbox | 427 - 428 - There are additional environment variables that can be tweaked depending on how you're running your service. For instance, storing blobs in AWS S3, keys in AWS KMS, or setting up an email service. 429 - 430 - Feel free to explore those [Here](https://github.com/bluesky-social/atproto/blob/main/packages/pds/src/config/env.ts). However, we will not be providing support for more advanced configurations.
+26
SANDBOX.md
··· 10 10 11 11 Given that this is a testing environment, we will be defederating from any instances that do not abide by these guidelines, or that cause unnecessary trouble, and will not be providing specific justifications for these decisions. 12 12 13 + <!-- markdown-toc -i SANDBOX.md --> 14 + 15 + <!-- toc --> 16 + 17 + - [Guidelines that must be followed](#guidelines-that-must-be-followed) 18 + * [Post responsibly](#post-responsibly) 19 + * [Keep the emphasis on testing](#keep-the-emphasis-on-testing) 20 + * [Do limit account creation](#do-limit-account-creation) 21 + * [Don’t expect persistence or uptime](#dont-expect-persistence-or-uptime) 22 + * [Don't advertise your service as being "Bluesky"](#dont-advertise-your-service-as-being-bluesky) 23 + * [Do not mirror sandbox did:plcs to production](#do-not-mirror-sandbox-didplcs-to-production) 24 + * [Status and Wipes](#status-and-wipes) 25 + + [🐉 Beware of dragons!](#%F0%9F%90%89-beware-of-dragons) 26 + + [Routine wipes](#routine-wipes) 27 + - [Getting started](#getting-started) 28 + * [Auto-updates](#auto-updates) 29 + * [Odds & Ends & Warnings & Reminders](#odds--ends--warnings--reminders) 30 + - [Learn more about atproto federation](#learn-more-about-atproto-federation) 31 + * [Network Services](#network-services) 32 + + [PLC](#plc) 33 + + [BGS](#bgs) 34 + + [Bluesky App View](#bluesky-app-view) 35 + - [The PDS](#the-pds) 36 + 37 + <!-- tocstop --> 38 + 13 39 # Guidelines that must be followed 14 40 15 41 Using the sandbox environment means you agree to adhere to our Guidelines. Please read the following carefully: