this repo has no description
lewis.moe
1# BSPDS Production Kubernetes Deployment
2
3> **Warning**: These instructions are untested and theoretical, written from the top of Lewis' head. They may contain errors or omissions. This warning will be removed once the guide has been verified.
4
5This guide covers deploying BSPDS on a production multi-node Kubernetes cluster with high availability, auto-scaling, and proper secrets management.
6
7## Architecture Overview
8
9```
10 ┌─────────────────────────────────────────────────┐
11 │ Kubernetes Cluster │
12 │ │
13 Internet ──────►│ Ingress Controller (nginx/traefik) │
14 │ │ │
15 │ ▼ │
16 │ ┌─────────────┐ │
17 │ │ Service │◄── HPA (2-10 replicas) │
18 │ └──────┬──────┘ │
19 │ │ │
20 │ ┌────┴────┐ │
21 │ ▼ ▼ │
22 │ ┌─────┐ ┌─────┐ │
23 │ │BSPDS│ │BSPDS│ ... (pods) │
24 │ └──┬──┘ └──┬──┘ │
25 │ │ │ │
26 │ ▼ ▼ │
27 │ ┌──────────────────────────────────────┐ │
28 │ │ PostgreSQL │ MinIO │ Valkey │ │
29 │ │ (HA/Operator)│ (StatefulSet) │ (Sentinel) │
30 │ └──────────────────────────────────────┘ │
31 └─────────────────────────────────────────────────┘
32```
33
34## Prerequisites
35
36- Kubernetes cluster (1.30+) with at least 3 nodes (1.34 is current stable)
37- `kubectl` configured to access your cluster
38- `helm` 3.x installed
39- Storage class that supports `ReadWriteOnce` (for databases)
40- Ingress controller installed (nginx-ingress or traefik)
41- cert-manager installed for TLS certificates
42
43### Quick Prerequisites Setup
44
45If you need to install prerequisites:
46
47```bash
48# Install nginx-ingress (chart v4.14.1 - December 2025)
49helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx
50helm repo update
51helm install ingress-nginx ingress-nginx/ingress-nginx \
52 --namespace ingress-nginx --create-namespace \
53 --version 4.14.1
54
55# Install cert-manager (v1.19.2 - December 2025)
56helm repo add jetstack https://charts.jetstack.io
57helm repo update
58helm install cert-manager jetstack/cert-manager \
59 --namespace cert-manager --create-namespace \
60 --version v1.19.2 \
61 --set installCRDs=true
62```
63
64---
65
66## 1. Create Namespace
67
68```bash
69kubectl create namespace bspds
70kubectl config set-context --current --namespace=bspds
71```
72
73## 2. Create Secrets
74
75Generate secure passwords and secrets:
76
77```bash
78# Generate secrets
79DB_PASSWORD=$(openssl rand -base64 32)
80MINIO_PASSWORD=$(openssl rand -base64 32)
81JWT_SECRET=$(openssl rand -base64 48)
82DPOP_SECRET=$(openssl rand -base64 48)
83MASTER_KEY=$(openssl rand -base64 48)
84
85# Create Kubernetes secrets
86kubectl create secret generic bspds-db-credentials \
87 --from-literal=username=bspds \
88 --from-literal=password="$DB_PASSWORD"
89
90kubectl create secret generic bspds-minio-credentials \
91 --from-literal=root-user=minioadmin \
92 --from-literal=root-password="$MINIO_PASSWORD"
93
94kubectl create secret generic bspds-secrets \
95 --from-literal=jwt-secret="$JWT_SECRET" \
96 --from-literal=dpop-secret="$DPOP_SECRET" \
97 --from-literal=master-key="$MASTER_KEY"
98
99# Save secrets locally (KEEP SECURE!)
100echo "DB_PASSWORD=$DB_PASSWORD" > secrets.txt
101echo "MINIO_PASSWORD=$MINIO_PASSWORD" >> secrets.txt
102echo "JWT_SECRET=$JWT_SECRET" >> secrets.txt
103echo "DPOP_SECRET=$DPOP_SECRET" >> secrets.txt
104echo "MASTER_KEY=$MASTER_KEY" >> secrets.txt
105chmod 600 secrets.txt
106```
107
108## 3. Deploy PostgreSQL
109
110### Option A: CloudNativePG Operator (Recommended for HA)
111
112```bash
113# Install CloudNativePG operator (v1.28.0 - December 2025)
114kubectl apply --server-side -f \
115 https://raw.githubusercontent.com/cloudnative-pg/cloudnative-pg/release-1.28/releases/cnpg-1.28.0.yaml
116
117# Wait for operator
118kubectl wait --for=condition=available --timeout=120s \
119 deployment/cnpg-controller-manager -n cnpg-system
120```
121
122```bash
123cat <<EOF | kubectl apply -f -
124apiVersion: postgresql.cnpg.io/v1
125kind: Cluster
126metadata:
127 name: bspds-db
128 namespace: bspds
129spec:
130 instances: 3
131
132 postgresql:
133 parameters:
134 max_connections: "200"
135 shared_buffers: "256MB"
136
137 bootstrap:
138 initdb:
139 database: pds
140 owner: bspds
141 secret:
142 name: bspds-db-credentials
143
144 storage:
145 size: 20Gi
146 storageClass: standard # adjust for your cluster
147
148 resources:
149 requests:
150 memory: "512Mi"
151 cpu: "250m"
152 limits:
153 memory: "1Gi"
154 cpu: "1000m"
155
156 affinity:
157 podAntiAffinityType: required
158EOF
159```
160
161### Option B: Simple StatefulSet (Single Instance)
162
163```bash
164cat <<EOF | kubectl apply -f -
165apiVersion: v1
166kind: PersistentVolumeClaim
167metadata:
168 name: bspds-db-pvc
169 namespace: bspds
170spec:
171 accessModes:
172 - ReadWriteOnce
173 resources:
174 requests:
175 storage: 20Gi
176---
177apiVersion: apps/v1
178kind: StatefulSet
179metadata:
180 name: bspds-db
181 namespace: bspds
182spec:
183 serviceName: bspds-db
184 replicas: 1
185 selector:
186 matchLabels:
187 app: bspds-db
188 template:
189 metadata:
190 labels:
191 app: bspds-db
192 spec:
193 containers:
194 - name: postgres
195 image: postgres:18-alpine
196 ports:
197 - containerPort: 5432
198 env:
199 - name: POSTGRES_DB
200 value: pds
201 - name: POSTGRES_USER
202 valueFrom:
203 secretKeyRef:
204 name: bspds-db-credentials
205 key: username
206 - name: POSTGRES_PASSWORD
207 valueFrom:
208 secretKeyRef:
209 name: bspds-db-credentials
210 key: password
211 - name: PGDATA
212 value: /var/lib/postgresql/data/pgdata
213 volumeMounts:
214 - name: data
215 mountPath: /var/lib/postgresql/data
216 resources:
217 requests:
218 memory: "256Mi"
219 cpu: "100m"
220 limits:
221 memory: "1Gi"
222 cpu: "500m"
223 livenessProbe:
224 exec:
225 command: ["pg_isready", "-U", "bspds", "-d", "pds"]
226 initialDelaySeconds: 30
227 periodSeconds: 10
228 readinessProbe:
229 exec:
230 command: ["pg_isready", "-U", "bspds", "-d", "pds"]
231 initialDelaySeconds: 5
232 periodSeconds: 5
233 volumes:
234 - name: data
235 persistentVolumeClaim:
236 claimName: bspds-db-pvc
237---
238apiVersion: v1
239kind: Service
240metadata:
241 name: bspds-db-rw
242 namespace: bspds
243spec:
244 selector:
245 app: bspds-db
246 ports:
247 - port: 5432
248 targetPort: 5432
249EOF
250```
251
252## 4. Deploy MinIO
253
254```bash
255cat <<EOF | kubectl apply -f -
256apiVersion: v1
257kind: PersistentVolumeClaim
258metadata:
259 name: bspds-minio-pvc
260 namespace: bspds
261spec:
262 accessModes:
263 - ReadWriteOnce
264 resources:
265 requests:
266 storage: 50Gi
267---
268apiVersion: apps/v1
269kind: StatefulSet
270metadata:
271 name: bspds-minio
272 namespace: bspds
273spec:
274 serviceName: bspds-minio
275 replicas: 1
276 selector:
277 matchLabels:
278 app: bspds-minio
279 template:
280 metadata:
281 labels:
282 app: bspds-minio
283 spec:
284 containers:
285 - name: minio
286 image: minio/minio:RELEASE.2025-10-15T17-29-55Z
287 args:
288 - server
289 - /data
290 - --console-address
291 - ":9001"
292 ports:
293 - containerPort: 9000
294 name: api
295 - containerPort: 9001
296 name: console
297 env:
298 - name: MINIO_ROOT_USER
299 valueFrom:
300 secretKeyRef:
301 name: bspds-minio-credentials
302 key: root-user
303 - name: MINIO_ROOT_PASSWORD
304 valueFrom:
305 secretKeyRef:
306 name: bspds-minio-credentials
307 key: root-password
308 volumeMounts:
309 - name: data
310 mountPath: /data
311 resources:
312 requests:
313 memory: "256Mi"
314 cpu: "100m"
315 limits:
316 memory: "512Mi"
317 cpu: "500m"
318 livenessProbe:
319 httpGet:
320 path: /minio/health/live
321 port: 9000
322 initialDelaySeconds: 30
323 periodSeconds: 10
324 readinessProbe:
325 httpGet:
326 path: /minio/health/ready
327 port: 9000
328 initialDelaySeconds: 10
329 periodSeconds: 5
330 volumes:
331 - name: data
332 persistentVolumeClaim:
333 claimName: bspds-minio-pvc
334---
335apiVersion: v1
336kind: Service
337metadata:
338 name: bspds-minio
339 namespace: bspds
340spec:
341 selector:
342 app: bspds-minio
343 ports:
344 - port: 9000
345 targetPort: 9000
346 name: api
347 - port: 9001
348 targetPort: 9001
349 name: console
350EOF
351```
352
353### Initialize MinIO Bucket
354
355```bash
356kubectl run minio-init --rm -it --restart=Never \
357 --image=minio/mc:RELEASE.2025-07-16T15-35-03Z \
358 --env="MINIO_ROOT_USER=minioadmin" \
359 --env="MINIO_ROOT_PASSWORD=$(kubectl get secret bspds-minio-credentials -o jsonpath='{.data.root-password}' | base64 -d)" \
360 --command -- sh -c "
361 mc alias set local http://bspds-minio:9000 \$MINIO_ROOT_USER \$MINIO_ROOT_PASSWORD &&
362 mc mb --ignore-existing local/pds-blobs
363 "
364```
365
366## 5. Deploy Valkey
367
368```bash
369cat <<EOF | kubectl apply -f -
370apiVersion: v1
371kind: PersistentVolumeClaim
372metadata:
373 name: bspds-valkey-pvc
374 namespace: bspds
375spec:
376 accessModes:
377 - ReadWriteOnce
378 resources:
379 requests:
380 storage: 5Gi
381---
382apiVersion: apps/v1
383kind: StatefulSet
384metadata:
385 name: bspds-valkey
386 namespace: bspds
387spec:
388 serviceName: bspds-valkey
389 replicas: 1
390 selector:
391 matchLabels:
392 app: bspds-valkey
393 template:
394 metadata:
395 labels:
396 app: bspds-valkey
397 spec:
398 containers:
399 - name: valkey
400 image: valkey/valkey:9-alpine
401 args:
402 - valkey-server
403 - --appendonly
404 - "yes"
405 - --maxmemory
406 - "256mb"
407 - --maxmemory-policy
408 - allkeys-lru
409 ports:
410 - containerPort: 6379
411 volumeMounts:
412 - name: data
413 mountPath: /data
414 resources:
415 requests:
416 memory: "128Mi"
417 cpu: "50m"
418 limits:
419 memory: "300Mi"
420 cpu: "200m"
421 livenessProbe:
422 exec:
423 command: ["valkey-cli", "ping"]
424 initialDelaySeconds: 10
425 periodSeconds: 5
426 readinessProbe:
427 exec:
428 command: ["valkey-cli", "ping"]
429 initialDelaySeconds: 5
430 periodSeconds: 3
431 volumes:
432 - name: data
433 persistentVolumeClaim:
434 claimName: bspds-valkey-pvc
435---
436apiVersion: v1
437kind: Service
438metadata:
439 name: bspds-valkey
440 namespace: bspds
441spec:
442 selector:
443 app: bspds-valkey
444 ports:
445 - port: 6379
446 targetPort: 6379
447EOF
448```
449
450## 6. Build and Push BSPDS Image
451
452```bash
453# Build image
454cd /path/to/bspds
455docker build -t your-registry.com/bspds:latest .
456docker push your-registry.com/bspds:latest
457```
458
459If using a private registry, create an image pull secret:
460
461```bash
462kubectl create secret docker-registry regcred \
463 --docker-server=your-registry.com \
464 --docker-username=your-username \
465 --docker-password=your-password \
466 --docker-email=your-email
467```
468
469## 7. Run Database Migrations
470
471BSPDS runs migrations automatically on startup. However, if you want to run migrations separately (recommended for zero-downtime deployments), you can use a Job:
472
473```bash
474cat <<'EOF' | kubectl apply -f -
475apiVersion: batch/v1
476kind: Job
477metadata:
478 name: bspds-migrate
479 namespace: bspds
480spec:
481 ttlSecondsAfterFinished: 300
482 template:
483 spec:
484 restartPolicy: Never
485 containers:
486 - name: migrate
487 image: your-registry.com/bspds:latest
488 command: ["/usr/local/bin/bspds"]
489 args: ["--migrate-only"] # Add this flag to your app, or remove this Job
490 env:
491 - name: DB_PASSWORD
492 valueFrom:
493 secretKeyRef:
494 name: bspds-db-credentials
495 key: password
496 - name: DATABASE_URL
497 value: "postgres://bspds:$(DB_PASSWORD)@bspds-db-rw:5432/pds"
498EOF
499
500kubectl wait --for=condition=complete --timeout=120s job/bspds-migrate
501```
502
503> **Note**: If your BSPDS image doesn't have a `--migrate-only` flag, you can skip this step. The app will run migrations on first startup. Alternatively, build a separate migration image with `sqlx-cli` installed.
504
505## 8. Deploy BSPDS Application
506
507```bash
508cat <<EOF | kubectl apply -f -
509apiVersion: v1
510kind: ConfigMap
511metadata:
512 name: bspds-config
513 namespace: bspds
514data:
515 PDS_HOSTNAME: "pds.example.com"
516 SERVER_HOST: "0.0.0.0"
517 SERVER_PORT: "3000"
518 S3_ENDPOINT: "http://bspds-minio:9000"
519 AWS_REGION: "us-east-1"
520 S3_BUCKET: "pds-blobs"
521 VALKEY_URL: "redis://bspds-valkey:6379"
522 APPVIEW_URL: "https://api.bsky.app"
523 CRAWLERS: "https://bsky.network"
524 FRONTEND_DIR: "/app/frontend/dist"
525---
526apiVersion: apps/v1
527kind: Deployment
528metadata:
529 name: bspds
530 namespace: bspds
531spec:
532 replicas: 2
533 selector:
534 matchLabels:
535 app: bspds
536 template:
537 metadata:
538 labels:
539 app: bspds
540 spec:
541 imagePullSecrets:
542 - name: regcred # Remove if using public registry
543 affinity:
544 podAntiAffinity:
545 preferredDuringSchedulingIgnoredDuringExecution:
546 - weight: 100
547 podAffinityTerm:
548 labelSelector:
549 matchLabels:
550 app: bspds
551 topologyKey: kubernetes.io/hostname
552 containers:
553 - name: bspds
554 image: your-registry.com/bspds:latest
555 ports:
556 - containerPort: 3000
557 name: http
558 envFrom:
559 - configMapRef:
560 name: bspds-config
561 env:
562 - name: DB_PASSWORD
563 valueFrom:
564 secretKeyRef:
565 name: bspds-db-credentials
566 key: password
567 - name: DATABASE_URL
568 value: "postgres://bspds:$(DB_PASSWORD)@bspds-db-rw:5432/pds"
569 - name: AWS_ACCESS_KEY_ID
570 valueFrom:
571 secretKeyRef:
572 name: bspds-minio-credentials
573 key: root-user
574 - name: AWS_SECRET_ACCESS_KEY
575 valueFrom:
576 secretKeyRef:
577 name: bspds-minio-credentials
578 key: root-password
579 - name: JWT_SECRET
580 valueFrom:
581 secretKeyRef:
582 name: bspds-secrets
583 key: jwt-secret
584 - name: DPOP_SECRET
585 valueFrom:
586 secretKeyRef:
587 name: bspds-secrets
588 key: dpop-secret
589 - name: MASTER_KEY
590 valueFrom:
591 secretKeyRef:
592 name: bspds-secrets
593 key: master-key
594 resources:
595 requests:
596 memory: "256Mi"
597 cpu: "100m"
598 limits:
599 memory: "1Gi"
600 cpu: "1000m"
601 livenessProbe:
602 httpGet:
603 path: /xrpc/_health
604 port: 3000
605 initialDelaySeconds: 30
606 periodSeconds: 10
607 failureThreshold: 3
608 readinessProbe:
609 httpGet:
610 path: /xrpc/_health
611 port: 3000
612 initialDelaySeconds: 5
613 periodSeconds: 5
614 failureThreshold: 3
615 securityContext:
616 runAsNonRoot: true
617 runAsUser: 1000
618 allowPrivilegeEscalation: false
619---
620apiVersion: v1
621kind: Service
622metadata:
623 name: bspds
624 namespace: bspds
625spec:
626 selector:
627 app: bspds
628 ports:
629 - port: 80
630 targetPort: 3000
631 name: http
632EOF
633```
634
635## 9. Configure Horizontal Pod Autoscaler
636
637```bash
638cat <<EOF | kubectl apply -f -
639apiVersion: autoscaling/v2
640kind: HorizontalPodAutoscaler
641metadata:
642 name: bspds
643 namespace: bspds
644spec:
645 scaleTargetRef:
646 apiVersion: apps/v1
647 kind: Deployment
648 name: bspds
649 minReplicas: 2
650 maxReplicas: 10
651 metrics:
652 - type: Resource
653 resource:
654 name: cpu
655 target:
656 type: Utilization
657 averageUtilization: 70
658 - type: Resource
659 resource:
660 name: memory
661 target:
662 type: Utilization
663 averageUtilization: 80
664 behavior:
665 scaleDown:
666 stabilizationWindowSeconds: 300
667 policies:
668 - type: Pods
669 value: 1
670 periodSeconds: 60
671 scaleUp:
672 stabilizationWindowSeconds: 0
673 policies:
674 - type: Percent
675 value: 100
676 periodSeconds: 15
677 - type: Pods
678 value: 4
679 periodSeconds: 15
680 selectPolicy: Max
681EOF
682```
683
684## 10. Configure Pod Disruption Budget
685
686```bash
687cat <<EOF | kubectl apply -f -
688apiVersion: policy/v1
689kind: PodDisruptionBudget
690metadata:
691 name: bspds
692 namespace: bspds
693spec:
694 minAvailable: 1
695 selector:
696 matchLabels:
697 app: bspds
698EOF
699```
700
701## 11. Configure TLS with cert-manager
702
703```bash
704cat <<EOF | kubectl apply -f -
705apiVersion: cert-manager.io/v1
706kind: ClusterIssuer
707metadata:
708 name: letsencrypt-prod
709spec:
710 acme:
711 server: https://acme-v02.api.letsencrypt.org/directory
712 email: your-email@example.com
713 privateKeySecretRef:
714 name: letsencrypt-prod
715 solvers:
716 - http01:
717 ingress:
718 class: nginx
719EOF
720```
721
722## 12. Configure Ingress
723
724```bash
725cat <<EOF | kubectl apply -f -
726apiVersion: networking.k8s.io/v1
727kind: Ingress
728metadata:
729 name: bspds
730 namespace: bspds
731 annotations:
732 cert-manager.io/cluster-issuer: letsencrypt-prod
733 nginx.ingress.kubernetes.io/proxy-read-timeout: "86400"
734 nginx.ingress.kubernetes.io/proxy-send-timeout: "86400"
735 nginx.ingress.kubernetes.io/proxy-body-size: "100m"
736 nginx.ingress.kubernetes.io/proxy-buffering: "off"
737 nginx.ingress.kubernetes.io/websocket-services: "bspds"
738spec:
739 ingressClassName: nginx
740 tls:
741 - hosts:
742 - pds.example.com
743 secretName: bspds-tls
744 rules:
745 - host: pds.example.com
746 http:
747 paths:
748 - path: /
749 pathType: Prefix
750 backend:
751 service:
752 name: bspds
753 port:
754 number: 80
755EOF
756```
757
758## 13. Configure Network Policies (Optional but Recommended)
759
760```bash
761cat <<EOF | kubectl apply -f -
762apiVersion: networking.k8s.io/v1
763kind: NetworkPolicy
764metadata:
765 name: bspds-network-policy
766 namespace: bspds
767spec:
768 podSelector:
769 matchLabels:
770 app: bspds
771 policyTypes:
772 - Ingress
773 - Egress
774 ingress:
775 - from:
776 - namespaceSelector:
777 matchLabels:
778 kubernetes.io/metadata.name: ingress-nginx
779 ports:
780 - protocol: TCP
781 port: 3000
782 egress:
783 - to:
784 - podSelector:
785 matchLabels:
786 app: bspds-db
787 ports:
788 - protocol: TCP
789 port: 5432
790 - to:
791 - podSelector:
792 matchLabels:
793 app: bspds-minio
794 ports:
795 - protocol: TCP
796 port: 9000
797 - to:
798 - podSelector:
799 matchLabels:
800 app: bspds-valkey
801 ports:
802 - protocol: TCP
803 port: 6379
804 - to: # Allow DNS
805 - namespaceSelector: {}
806 podSelector:
807 matchLabels:
808 k8s-app: kube-dns
809 ports:
810 - protocol: UDP
811 port: 53
812 - to: # Allow external HTTPS (for federation)
813 - ipBlock:
814 cidr: 0.0.0.0/0
815 ports:
816 - protocol: TCP
817 port: 443
818EOF
819```
820
821## 14. Deploy Prometheus Monitoring (Optional)
822
823```bash
824cat <<EOF | kubectl apply -f -
825apiVersion: monitoring.coreos.com/v1
826kind: ServiceMonitor
827metadata:
828 name: bspds
829 namespace: bspds
830 labels:
831 release: prometheus
832spec:
833 selector:
834 matchLabels:
835 app: bspds
836 endpoints:
837 - port: http
838 path: /metrics
839 interval: 30s
840EOF
841```
842
843---
844
845## Verification
846
847```bash
848# Check all pods are running
849kubectl get pods -n bspds
850
851# Check services
852kubectl get svc -n bspds
853
854# Check ingress
855kubectl get ingress -n bspds
856
857# Check certificate
858kubectl get certificate -n bspds
859
860# Test health endpoint
861curl -s https://pds.example.com/xrpc/_health | jq
862
863# Test DID endpoint
864curl -s https://pds.example.com/.well-known/atproto-did
865```
866
867---
868
869## Maintenance
870
871### View Logs
872
873```bash
874# All BSPDS pods
875kubectl logs -l app=bspds -n bspds -f
876
877# Specific pod
878kubectl logs -f deployment/bspds -n bspds
879```
880
881### Scale Manually
882
883```bash
884kubectl scale deployment bspds --replicas=5 -n bspds
885```
886
887### Update BSPDS
888
889```bash
890# Build and push new image
891docker build -t your-registry.com/bspds:v1.2.3 .
892docker push your-registry.com/bspds:v1.2.3
893
894# Update deployment
895kubectl set image deployment/bspds bspds=your-registry.com/bspds:v1.2.3 -n bspds
896
897# Watch rollout
898kubectl rollout status deployment/bspds -n bspds
899```
900
901### Backup Database
902
903```bash
904# For CloudNativePG
905kubectl cnpg backup bspds-db -n bspds
906
907# For StatefulSet
908kubectl exec -it bspds-db-0 -n bspds -- pg_dump -U bspds pds > backup-$(date +%Y%m%d).sql
909```
910
911### Run Migrations
912
913If you have a migration Job defined, you can re-run it:
914
915```bash
916# Delete old job first (if exists)
917kubectl delete job bspds-migrate -n bspds --ignore-not-found
918
919# Re-apply the migration job from step 7
920# Or simply restart the deployment - BSPDS runs migrations on startup
921kubectl rollout restart deployment/bspds -n bspds
922```
923
924---
925
926## Troubleshooting
927
928### Pod Won't Start
929
930```bash
931kubectl describe pod -l app=bspds -n bspds
932kubectl logs -l app=bspds -n bspds --previous
933```
934
935### Database Connection Issues
936
937```bash
938# Test connectivity from a debug pod
939kubectl run debug --rm -it --restart=Never --image=postgres:18-alpine -- \
940 psql "postgres://bspds:PASSWORD@bspds-db-rw:5432/pds" -c "SELECT 1"
941```
942
943### Certificate Issues
944
945```bash
946kubectl describe certificate bspds-tls -n bspds
947kubectl describe certificaterequest -n bspds
948kubectl logs -l app.kubernetes.io/name=cert-manager -n cert-manager
949```
950
951### View Resource Usage
952
953```bash
954kubectl top pods -n bspds
955kubectl top nodes
956```