tranquil.farm / tranquil-pds
Our Personal Data Server from scratch! tranquil.farm
oauth atproto pds rust postgresql objectstorage fun
fork atom

refactor: toml config #24

merged opened by isabelroses.com targeting main

most certainly a improvement but almost certainly going to have some errors

Labels

None yet.

assignee

None yet.

Participants 2
AT URI
at://did:plc:qxichs7jsycphrsmbujwqbfb/sh.tangled.repo.pull/3meorfx6aki22
+38 -53
Interdiff #1 → #2
unified split
.env.example

This file has not been changed.

Cargo.lock

This file has not been changed.

Cargo.toml

This file has not been changed.

README.md

This file has not been changed.

crates/tranquil-auth/Cargo.toml

This file has not been changed.

crates/tranquil-auth/src/token.rs

This file has not been changed.

crates/tranquil-cache/Cargo.toml

This file has not been changed.

+2 -2
crates/tranquil-cache/src/lib.rs
··· 213 213 (cache, rate_limiter) 214 214 } 215 215 Err(e) => { 216 - tracing::error!("ripple engine failed to start: {e}. running without cache."); 216 + tracing::error!("ripple engine failed to start: {e:#}. running without cache."); 217 217 (Arc::new(NoOpCache), Arc::new(NoOpRateLimiter)) 218 218 } 219 219 } 220 220 } 221 221 Err(e) => { 222 - tracing::error!("ripple config error: {e}. running without cache."); 222 + tracing::error!("ripple config error: {e:#}. running without cache."); 223 223 (Arc::new(NoOpCache), Arc::new(NoOpRateLimiter)) 224 224 } 225 225 }
crates/tranquil-comms/Cargo.toml

This file has not been changed.

crates/tranquil-comms/src/sender.rs

This file has not been changed.

crates/tranquil-config/Cargo.toml

This file has not been changed.

+15 -24
crates/tranquil-config/src/lib.rs
··· 123 123 /// 124 124 /// Returns `Ok(())` when the configuration is consistent, or a 125 125 /// [`ConfigError`] listing every problem found. 126 - pub fn validate(&self) -> Result<(), ConfigError> { 126 + pub fn validate(&self, ignore_secrets: bool) -> Result<(), ConfigError> { 127 127 let mut errors = Vec::new(); 128 128 129 + // -- database --------------------------------------------------------- 130 + if self.database.url.is_none() { 131 + errors.push( 132 + "database.url (DATABASE_URL) is required".to_string(), 133 + ); 134 + } 135 + 129 136 // -- secrets ---------------------------------------------------------- 130 - if !self.secrets.allow_insecure && !cfg!(test) { 137 + if !ignore_secrets && !self.secrets.allow_insecure && !cfg!(test) { 131 138 if let Some(ref s) = self.secrets.jwt_secret { 132 139 if s.len() < 32 { 133 140 errors.push( ··· 194 201 ); 195 202 } 196 203 } 197 - "filesystem" => { 198 - if self.storage.path.is_none() { 199 - errors.push( 200 - "storage.backend is \"filesystem\" but storage.path \ 201 - (BLOB_STORAGE_PATH) is not set" 202 - .to_string(), 203 - ); 204 - } 205 - } 204 + "filesystem" => {} 206 205 other => { 207 206 errors.push(format!( 208 207 "storage.backend must be \"filesystem\" or \"s3\", got \"{other}\"" ··· 222 221 ); 223 222 } 224 223 } 225 - "filesystem" => { 226 - if self.backup.path.is_none() { 227 - errors.push( 228 - "backup.backend is \"filesystem\" but backup.path \ 229 - (BACKUP_STORAGE_PATH) is not set" 230 - .to_string(), 231 - ); 232 - } 233 - } 224 + "filesystem" => {} 234 225 other => { 235 226 errors.push(format!( 236 227 "backup.backend must be \"filesystem\" or \"s3\", got \"{other}\"" ··· 592 583 pub backend: String, 593 584 594 585 /// Path on disk for the filesystem blob backend. 595 - #[config(env = "BLOB_STORAGE_PATH")] 596 - pub path: Option<String>, 586 + #[config(env = "BLOB_STORAGE_PATH", default = "/var/lib/tranquil-pds/blobs")] 587 + pub path: String, 597 588 598 589 /// S3 bucket name for blob storage. 599 590 #[config(env = "S3_BUCKET")] ··· 615 606 pub backend: String, 616 607 617 608 /// Path on disk for the filesystem backup backend. 618 - #[config(env = "BACKUP_STORAGE_PATH")] 619 - pub path: Option<String>, 609 + #[config(env = "BACKUP_STORAGE_PATH", default = "/var/lib/tranquil-pds/backups")] 610 + pub path: String, 620 611 621 612 /// S3 bucket name for backups. 622 613 #[config(env = "BACKUP_S3_BUCKET")]
crates/tranquil-infra/Cargo.toml

This file has not been changed.

crates/tranquil-infra/src/lib.rs

This file has not been changed.

crates/tranquil-pds/Cargo.toml

This file has not been changed.

crates/tranquil-pds/src/api/admin/account/email.rs

This file has not been changed.

crates/tranquil-pds/src/api/admin/account/update.rs

This file has not been changed.

crates/tranquil-pds/src/api/delegation.rs

This file has not been changed.

crates/tranquil-pds/src/api/discord_webhook.rs

This file has not been changed.

crates/tranquil-pds/src/api/identity/account.rs

This file has not been changed.

crates/tranquil-pds/src/api/identity/did.rs

This file has not been changed.

crates/tranquil-pds/src/api/identity/plc/request.rs

This file has not been changed.

crates/tranquil-pds/src/api/identity/plc/submit.rs

This file has not been changed.

crates/tranquil-pds/src/api/moderation/mod.rs

This file has not been changed.

crates/tranquil-pds/src/api/notification_prefs.rs

This file has not been changed.

crates/tranquil-pds/src/api/proxy.rs

This file has not been changed.

crates/tranquil-pds/src/api/proxy_client.rs

This file has not been changed.

crates/tranquil-pds/src/api/repo/blob.rs

This file has not been changed.

crates/tranquil-pds/src/api/repo/import.rs

This file has not been changed.

crates/tranquil-pds/src/api/repo/meta.rs

This file has not been changed.

crates/tranquil-pds/src/api/repo/record/read.rs

This file has not been changed.

crates/tranquil-pds/src/api/server/account_status.rs

This file has not been changed.

crates/tranquil-pds/src/api/server/email.rs

This file has not been changed.

crates/tranquil-pds/src/api/server/invite.rs

This file has not been changed.

crates/tranquil-pds/src/api/server/meta.rs

This file has not been changed.

crates/tranquil-pds/src/api/server/migration.rs

This file has not been changed.

crates/tranquil-pds/src/api/server/passkey_account.rs

This file has not been changed.

crates/tranquil-pds/src/api/server/password.rs

This file has not been changed.

crates/tranquil-pds/src/api/server/session.rs

This file has not been changed.

crates/tranquil-pds/src/api/server/totp.rs

This file has not been changed.

crates/tranquil-pds/src/api/server/verify_email.rs

This file has not been changed.

crates/tranquil-pds/src/api/server/verify_token.rs

This file has not been changed.

crates/tranquil-pds/src/api/telegram_webhook.rs

This file has not been changed.

crates/tranquil-pds/src/appview/mod.rs

This file has not been changed.

crates/tranquil-pds/src/auth/service.rs

This file has not been changed.

crates/tranquil-pds/src/auth/verification_token.rs

This file has not been changed.

crates/tranquil-pds/src/comms/service.rs

This file has not been changed.

crates/tranquil-pds/src/config.rs

This file has not been changed.

crates/tranquil-pds/src/crawlers.rs

This file has not been changed.

crates/tranquil-pds/src/handle/mod.rs

This file has not been changed.

crates/tranquil-pds/src/lib.rs

This file has not been changed.

+11 -6
crates/tranquil-pds/src/main.rs
··· 34 34 #[derive(Subcommand)] 35 35 enum Command { 36 36 /// Validate the configuration and exit 37 - Validate, 37 + Validate { 38 + /// Skip validation of secrets and database URL (useful when secrets 39 + /// are provided at runtime via environment variables / secret files) 40 + #[arg(long)] 41 + ignore_secrets: bool, 42 + }, 38 43 /// Print a TOML configuration template to stdout 39 44 ConfigTemplate, 40 45 } ··· 52 57 print!("{}", tranquil_config::template()); 53 58 ExitCode::SUCCESS 54 59 } 55 - Command::Validate => { 60 + Command::Validate { ignore_secrets } => { 56 61 let config = match tranquil_config::load(cli.config.as_ref()) { 57 62 Ok(c) => c, 58 63 Err(e) => { 59 - eprintln!("Failed to load configuration: {e}"); 64 + eprintln!("Failed to load configuration: {e:#}"); 60 65 return ExitCode::FAILURE; 61 66 } 62 67 }; 63 - match config.validate() { 68 + match config.validate(*ignore_secrets) { 64 69 Ok(()) => { 65 70 println!("Configuration is valid."); 66 71 ExitCode::SUCCESS ··· 79 84 let config = match tranquil_config::load(cli.config.as_ref()) { 80 85 Ok(c) => c, 81 86 Err(e) => { 82 - error!("Failed to load configuration: {e}"); 87 + error!("Failed to load configuration: {e:#}"); 83 88 return ExitCode::FAILURE; 84 89 } 85 90 }; 86 91 87 - if let Err(e) = config.validate() { 92 + if let Err(e) = config.validate(false) { 88 93 error!("{e}"); 89 94 return ExitCode::FAILURE; 90 95 }
crates/tranquil-pds/src/moderation/mod.rs

This file has not been changed.

crates/tranquil-pds/src/oauth/endpoints/authorize.rs

This file has not been changed.

crates/tranquil-pds/src/oauth/endpoints/metadata.rs

This file has not been changed.

crates/tranquil-pds/src/oauth/endpoints/token/grants.rs

This file has not been changed.

crates/tranquil-pds/src/oauth/endpoints/token/helpers.rs

This file has not been changed.

crates/tranquil-pds/src/oauth/endpoints/token/introspect.rs

This file has not been changed.

crates/tranquil-pds/src/plc/mod.rs

This file has not been changed.

crates/tranquil-pds/src/scheduled.rs

This file has not been changed.

crates/tranquil-pds/src/sso/config.rs

This file has not been changed.

crates/tranquil-pds/src/sso/endpoints.rs

This file has not been changed.

+2 -1
crates/tranquil-pds/src/state.rs
··· 205 205 impl AppState { 206 206 pub async fn new(shutdown: CancellationToken) -> Result<Self, Box<dyn Error>> { 207 207 let cfg = tranquil_config::get(); 208 - let database_url = &cfg.database.url; 208 + let database_url = cfg.database.url.as_deref() 209 + .expect("database.url (DATABASE_URL) must be set"); 209 210 210 211 let max_connections = cfg.database.max_connections; 211 212
crates/tranquil-pds/src/sync/subscribe_repos.rs

This file has not been changed.

crates/tranquil-pds/src/sync/verify.rs

This file has not been changed.

crates/tranquil-pds/src/util.rs

This file has not been changed.

crates/tranquil-ripple/Cargo.toml

This file has not been changed.

crates/tranquil-ripple/src/config.rs

This file has not been changed.

crates/tranquil-storage/Cargo.toml

This file has not been changed.

+2 -16
crates/tranquil-storage/src/lib.rs
··· 716 716 } 717 717 _ => { 718 718 tracing::info!("Initializing filesystem blob storage"); 719 - let path = cfg.storage.path.clone().unwrap_or_else(|| { 720 - panic!( 721 - "storage.path (BLOB_STORAGE_PATH) not set. \ 722 - Set BLOB_STORAGE_PATH to a valid directory path." 723 - ); 724 - }); 719 + let path = cfg.storage.path.clone(); 725 720 FilesystemBlobStorage::new(path) 726 721 .await 727 722 .unwrap_or_else(|e| { ··· 770 765 None 771 766 } 772 767 _ => { 773 - let path = match cfg.backup.path.clone() { 774 - Some(p) => p, 775 - None => { 776 - tracing::error!( 777 - "backup.path (BACKUP_STORAGE_PATH) not set. \ 778 - Backups will be disabled." 779 - ); 780 - return None; 781 - } 782 - }; 768 + let path = cfg.backup.path.clone(); 783 769 FilesystemBackupStorage::new(path).await.map_or_else( 784 770 |e| { 785 771 tracing::error!(
docker-compose.prod.yaml

This file has not been changed.

docker-compose.yaml

This file has not been changed.

docs/install-containers.md

This file has not been changed.

docs/install-debian.md

This file has not been changed.

docs/install-kubernetes.md

This file has not been changed.

+6 -4
example.toml
··· 99 99 # PostgreSQL connection URL. 100 100 # 101 101 # Can also be specified via environment variable `DATABASE_URL`. 102 - # 103 - # Required! This value must be specified. 104 102 #url = 105 103 106 104 # Maximum number of connections in the pool. ··· 166 164 # Path on disk for the filesystem blob backend. 167 165 # 168 166 # Can also be specified via environment variable `BLOB_STORAGE_PATH`. 169 - #path = 167 + # 168 + # Default value: "/var/lib/tranquil-pds/blobs" 169 + #path = "/var/lib/tranquil-pds/blobs" 170 170 171 171 # S3 bucket name for blob storage. 172 172 # ··· 196 196 # Path on disk for the filesystem backup backend. 197 197 # 198 198 # Can also be specified via environment variable `BACKUP_STORAGE_PATH`. 199 - #path = 199 + # 200 + # Default value: "/var/lib/tranquil-pds/backups" 201 + #path = "/var/lib/tranquil-pds/backups" 200 202 201 203 # S3 bucket name for backups. 202 204 #

History

4 rounds 2 comments
sign up or login to add to the discussion
isabelroses.com submitted #3
diff interdiff
1 commit
expand
c66fe45e
refactor: toml config
expand 0 comments
pull request successfully merged
isabelroses.com submitted #2
diff
1 commit
expand
1b32bf4b
refactor: toml config
expand 0 comments
isabelroses.com submitted #1
diff interdiff
1 commit
expand
21b7ee92
refactor: toml config
expand 1 comment
nel.pet

oookay finally got a chance to try this out and noticed some usability mehs

the blob storage paths and the backup storage paths should have default values. doesnt make much sense to me to not have that.

validate should probably have a --ignore-secrets flag or similar? since most people will probably want to set the secrets in an env file and not in the config file. necessary for the nix module too if we want that to validate the config during build

anywhere possible errors from trying to load the config are printed to the user should also have some more handling to actually print the error properly. as https://github.com/LukasKalbertodt/confique/blob/main/src/error.rs#L9 mentions just printing it doesnt actually do that which ends up giving very cryptic and non-specific messages to the user just saying loading the config failed without giving proper reasons why. just doing e:# seemed fine for now for me locally. in the future we'll probably want to walk the sources properly but it can wait

isabelroses.com submitted #0
diff
1 commit
expand
13867bba
refactor: toml config
expand 1 comment
nel.pet

overall looks really good! i only really have some nitpicks about naming and default values.

server.service_handle_domains should be server.user_handle_domains imo. i never quite liked the "service handle domain" name. its confusing imo. and describeServer calls them user domains so i think we should align with that

server.enable_self_hosted_did_web makes it sound like its the opposite of what it is. should be server.enable_pds_hosted_did_web instead. also imo we should default this to false? given the consequences of having this enabled it should be opt-in imo

server.invite_code_required should probably default to true? thats what ref impl does and i think thats sensible

crawlers.urls should be firehose.crawlers imo. i think it makes sense to keep it with the rest of the sync related config (or have a layer of nesting more and have sync.crawlers and sync.firehose. sounds messy tho)

all of ripple.* should probably go under cache.ripple.* (as well as add a cache.backend option) since ripple is an in-house in-process replacement for valkey

also perhaps make it clear in the docs that you can set config options with env vars too? + the config cli flag and env var

idk how i feel about the config static being a OnceLock and not a LazyLock and all the panicing with init() and get(). but i understand getting it to work with a LazyLock is annoying due to the fallibility of config loading. probably going to explore how to handle that in the future. not going to block this PR on that