this repo has no description
lewis.moe
1# BSPDS Production Kubernetes Deployment
2> **Warning**: These instructions are untested and theoretical, written from the top of Lewis' head. They may contain errors or omissions. This warning will be removed once the guide has been verified.
3This guide covers deploying BSPDS on a production multi-node Kubernetes cluster with high availability, auto-scaling, and proper secrets management.
4## Architecture Overview
5```
6 ┌─────────────────────────────────────────────────┐
7 │ Kubernetes Cluster │
8 │ │
9 Internet ──────►│ Ingress Controller (nginx/traefik) │
10 │ │ │
11 │ ▼ │
12 │ ┌─────────────┐ │
13 │ │ Service │◄── HPA (2-10 replicas) │
14 │ └──────┬──────┘ │
15 │ │ │
16 │ ┌────┴────┐ │
17 │ ▼ ▼ │
18 │ ┌─────┐ ┌─────┐ │
19 │ │BSPDS│ │BSPDS│ ... (pods) │
20 │ └──┬──┘ └──┬──┘ │
21 │ │ │ │
22 │ ▼ ▼ │
23 │ ┌──────────────────────────────────────┐ │
24 │ │ PostgreSQL │ MinIO │ Valkey │ │
25 │ │ (HA/Operator)│ (StatefulSet) │ (Sentinel) │
26 │ └──────────────────────────────────────┘ │
27 └─────────────────────────────────────────────────┘
28```
29## Prerequisites
30- Kubernetes cluster (1.30+) with at least 3 nodes (1.34 is current stable)
31- `kubectl` configured to access your cluster
32- `helm` 3.x installed
33- Storage class that supports `ReadWriteOnce` (for databases)
34- Ingress controller installed (nginx-ingress or traefik)
35- cert-manager installed for TLS certificates
36### Quick Prerequisites Setup
37If you need to install prerequisites:
38```bash
39# Install nginx-ingress (chart v4.14.1 - December 2025)
40helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx
41helm repo update
42helm install ingress-nginx ingress-nginx/ingress-nginx \
43 --namespace ingress-nginx --create-namespace \
44 --version 4.14.1
45# Install cert-manager (v1.19.2 - December 2025)
46helm repo add jetstack https://charts.jetstack.io
47helm repo update
48helm install cert-manager jetstack/cert-manager \
49 --namespace cert-manager --create-namespace \
50 --version v1.19.2 \
51 --set installCRDs=true
52```
53---
54## 1. Create Namespace
55```bash
56kubectl create namespace bspds
57kubectl config set-context --current --namespace=bspds
58```
59## 2. Create Secrets
60Generate secure passwords and secrets:
61```bash
62# Generate secrets
63DB_PASSWORD=$(openssl rand -base64 32)
64MINIO_PASSWORD=$(openssl rand -base64 32)
65JWT_SECRET=$(openssl rand -base64 48)
66DPOP_SECRET=$(openssl rand -base64 48)
67MASTER_KEY=$(openssl rand -base64 48)
68# Create Kubernetes secrets
69kubectl create secret generic bspds-db-credentials \
70 --from-literal=username=bspds \
71 --from-literal=password="$DB_PASSWORD"
72kubectl create secret generic bspds-minio-credentials \
73 --from-literal=root-user=minioadmin \
74 --from-literal=root-password="$MINIO_PASSWORD"
75kubectl create secret generic bspds-secrets \
76 --from-literal=jwt-secret="$JWT_SECRET" \
77 --from-literal=dpop-secret="$DPOP_SECRET" \
78 --from-literal=master-key="$MASTER_KEY"
79# Save secrets locally (KEEP SECURE!)
80echo "DB_PASSWORD=$DB_PASSWORD" > secrets.txt
81echo "MINIO_PASSWORD=$MINIO_PASSWORD" >> secrets.txt
82echo "JWT_SECRET=$JWT_SECRET" >> secrets.txt
83echo "DPOP_SECRET=$DPOP_SECRET" >> secrets.txt
84echo "MASTER_KEY=$MASTER_KEY" >> secrets.txt
85chmod 600 secrets.txt
86```
87## 3. Deploy PostgreSQL
88### Option A: CloudNativePG Operator (Recommended for HA)
89```bash
90# Install CloudNativePG operator (v1.28.0 - December 2025)
91kubectl apply --server-side -f \
92 https://raw.githubusercontent.com/cloudnative-pg/cloudnative-pg/release-1.28/releases/cnpg-1.28.0.yaml
93# Wait for operator
94kubectl wait --for=condition=available --timeout=120s \
95 deployment/cnpg-controller-manager -n cnpg-system
96```
97```bash
98cat <<EOF | kubectl apply -f -
99apiVersion: postgresql.cnpg.io/v1
100kind: Cluster
101metadata:
102 name: bspds-db
103 namespace: bspds
104spec:
105 instances: 3
106 postgresql:
107 parameters:
108 max_connections: "200"
109 shared_buffers: "256MB"
110 bootstrap:
111 initdb:
112 database: pds
113 owner: bspds
114 secret:
115 name: bspds-db-credentials
116 storage:
117 size: 20Gi
118 storageClass: standard # adjust for your cluster
119 resources:
120 requests:
121 memory: "512Mi"
122 cpu: "250m"
123 limits:
124 memory: "1Gi"
125 cpu: "1000m"
126 affinity:
127 podAntiAffinityType: required
128EOF
129```
130### Option B: Simple StatefulSet (Single Instance)
131```bash
132cat <<EOF | kubectl apply -f -
133apiVersion: v1
134kind: PersistentVolumeClaim
135metadata:
136 name: bspds-db-pvc
137 namespace: bspds
138spec:
139 accessModes:
140 - ReadWriteOnce
141 resources:
142 requests:
143 storage: 20Gi
144---
145apiVersion: apps/v1
146kind: StatefulSet
147metadata:
148 name: bspds-db
149 namespace: bspds
150spec:
151 serviceName: bspds-db
152 replicas: 1
153 selector:
154 matchLabels:
155 app: bspds-db
156 template:
157 metadata:
158 labels:
159 app: bspds-db
160 spec:
161 containers:
162 - name: postgres
163 image: postgres:18-alpine
164 ports:
165 - containerPort: 5432
166 env:
167 - name: POSTGRES_DB
168 value: pds
169 - name: POSTGRES_USER
170 valueFrom:
171 secretKeyRef:
172 name: bspds-db-credentials
173 key: username
174 - name: POSTGRES_PASSWORD
175 valueFrom:
176 secretKeyRef:
177 name: bspds-db-credentials
178 key: password
179 - name: PGDATA
180 value: /var/lib/postgresql/data/pgdata
181 volumeMounts:
182 - name: data
183 mountPath: /var/lib/postgresql/data
184 resources:
185 requests:
186 memory: "256Mi"
187 cpu: "100m"
188 limits:
189 memory: "1Gi"
190 cpu: "500m"
191 livenessProbe:
192 exec:
193 command: ["pg_isready", "-U", "bspds", "-d", "pds"]
194 initialDelaySeconds: 30
195 periodSeconds: 10
196 readinessProbe:
197 exec:
198 command: ["pg_isready", "-U", "bspds", "-d", "pds"]
199 initialDelaySeconds: 5
200 periodSeconds: 5
201 volumes:
202 - name: data
203 persistentVolumeClaim:
204 claimName: bspds-db-pvc
205---
206apiVersion: v1
207kind: Service
208metadata:
209 name: bspds-db-rw
210 namespace: bspds
211spec:
212 selector:
213 app: bspds-db
214 ports:
215 - port: 5432
216 targetPort: 5432
217EOF
218```
219## 4. Deploy MinIO
220```bash
221cat <<EOF | kubectl apply -f -
222apiVersion: v1
223kind: PersistentVolumeClaim
224metadata:
225 name: bspds-minio-pvc
226 namespace: bspds
227spec:
228 accessModes:
229 - ReadWriteOnce
230 resources:
231 requests:
232 storage: 50Gi
233---
234apiVersion: apps/v1
235kind: StatefulSet
236metadata:
237 name: bspds-minio
238 namespace: bspds
239spec:
240 serviceName: bspds-minio
241 replicas: 1
242 selector:
243 matchLabels:
244 app: bspds-minio
245 template:
246 metadata:
247 labels:
248 app: bspds-minio
249 spec:
250 containers:
251 - name: minio
252 image: minio/minio:RELEASE.2025-10-15T17-29-55Z
253 args:
254 - server
255 - /data
256 - --console-address
257 - ":9001"
258 ports:
259 - containerPort: 9000
260 name: api
261 - containerPort: 9001
262 name: console
263 env:
264 - name: MINIO_ROOT_USER
265 valueFrom:
266 secretKeyRef:
267 name: bspds-minio-credentials
268 key: root-user
269 - name: MINIO_ROOT_PASSWORD
270 valueFrom:
271 secretKeyRef:
272 name: bspds-minio-credentials
273 key: root-password
274 volumeMounts:
275 - name: data
276 mountPath: /data
277 resources:
278 requests:
279 memory: "256Mi"
280 cpu: "100m"
281 limits:
282 memory: "512Mi"
283 cpu: "500m"
284 livenessProbe:
285 httpGet:
286 path: /minio/health/live
287 port: 9000
288 initialDelaySeconds: 30
289 periodSeconds: 10
290 readinessProbe:
291 httpGet:
292 path: /minio/health/ready
293 port: 9000
294 initialDelaySeconds: 10
295 periodSeconds: 5
296 volumes:
297 - name: data
298 persistentVolumeClaim:
299 claimName: bspds-minio-pvc
300---
301apiVersion: v1
302kind: Service
303metadata:
304 name: bspds-minio
305 namespace: bspds
306spec:
307 selector:
308 app: bspds-minio
309 ports:
310 - port: 9000
311 targetPort: 9000
312 name: api
313 - port: 9001
314 targetPort: 9001
315 name: console
316EOF
317```
318### Initialize MinIO Bucket
319```bash
320kubectl run minio-init --rm -it --restart=Never \
321 --image=minio/mc:RELEASE.2025-07-16T15-35-03Z \
322 --env="MINIO_ROOT_USER=minioadmin" \
323 --env="MINIO_ROOT_PASSWORD=$(kubectl get secret bspds-minio-credentials -o jsonpath='{.data.root-password}' | base64 -d)" \
324 --command -- sh -c "
325 mc alias set local http://bspds-minio:9000 \$MINIO_ROOT_USER \$MINIO_ROOT_PASSWORD &&
326 mc mb --ignore-existing local/pds-blobs
327 "
328```
329## 5. Deploy Valkey
330```bash
331cat <<EOF | kubectl apply -f -
332apiVersion: v1
333kind: PersistentVolumeClaim
334metadata:
335 name: bspds-valkey-pvc
336 namespace: bspds
337spec:
338 accessModes:
339 - ReadWriteOnce
340 resources:
341 requests:
342 storage: 5Gi
343---
344apiVersion: apps/v1
345kind: StatefulSet
346metadata:
347 name: bspds-valkey
348 namespace: bspds
349spec:
350 serviceName: bspds-valkey
351 replicas: 1
352 selector:
353 matchLabels:
354 app: bspds-valkey
355 template:
356 metadata:
357 labels:
358 app: bspds-valkey
359 spec:
360 containers:
361 - name: valkey
362 image: valkey/valkey:9-alpine
363 args:
364 - valkey-server
365 - --appendonly
366 - "yes"
367 - --maxmemory
368 - "256mb"
369 - --maxmemory-policy
370 - allkeys-lru
371 ports:
372 - containerPort: 6379
373 volumeMounts:
374 - name: data
375 mountPath: /data
376 resources:
377 requests:
378 memory: "128Mi"
379 cpu: "50m"
380 limits:
381 memory: "300Mi"
382 cpu: "200m"
383 livenessProbe:
384 exec:
385 command: ["valkey-cli", "ping"]
386 initialDelaySeconds: 10
387 periodSeconds: 5
388 readinessProbe:
389 exec:
390 command: ["valkey-cli", "ping"]
391 initialDelaySeconds: 5
392 periodSeconds: 3
393 volumes:
394 - name: data
395 persistentVolumeClaim:
396 claimName: bspds-valkey-pvc
397---
398apiVersion: v1
399kind: Service
400metadata:
401 name: bspds-valkey
402 namespace: bspds
403spec:
404 selector:
405 app: bspds-valkey
406 ports:
407 - port: 6379
408 targetPort: 6379
409EOF
410```
411## 6. Build and Push BSPDS Image
412```bash
413# Build image
414cd /path/to/bspds
415docker build -t your-registry.com/bspds:latest .
416docker push your-registry.com/bspds:latest
417```
418If using a private registry, create an image pull secret:
419```bash
420kubectl create secret docker-registry regcred \
421 --docker-server=your-registry.com \
422 --docker-username=your-username \
423 --docker-password=your-password \
424 --docker-email=your-email
425```
426## 7. Run Database Migrations
427BSPDS runs migrations automatically on startup. However, if you want to run migrations separately (recommended for zero-downtime deployments), you can use a Job:
428```bash
429cat <<'EOF' | kubectl apply -f -
430apiVersion: batch/v1
431kind: Job
432metadata:
433 name: bspds-migrate
434 namespace: bspds
435spec:
436 ttlSecondsAfterFinished: 300
437 template:
438 spec:
439 restartPolicy: Never
440 containers:
441 - name: migrate
442 image: your-registry.com/bspds:latest
443 command: ["/usr/local/bin/bspds"]
444 args: ["--migrate-only"] # Add this flag to your app, or remove this Job
445 env:
446 - name: DB_PASSWORD
447 valueFrom:
448 secretKeyRef:
449 name: bspds-db-credentials
450 key: password
451 - name: DATABASE_URL
452 value: "postgres://bspds:$(DB_PASSWORD)@bspds-db-rw:5432/pds"
453EOF
454kubectl wait --for=condition=complete --timeout=120s job/bspds-migrate
455```
456> **Note**: If your BSPDS image doesn't have a `--migrate-only` flag, you can skip this step. The app will run migrations on first startup. Alternatively, build a separate migration image with `sqlx-cli` installed.
457## 8. Deploy BSPDS Application
458```bash
459cat <<EOF | kubectl apply -f -
460apiVersion: v1
461kind: ConfigMap
462metadata:
463 name: bspds-config
464 namespace: bspds
465data:
466 PDS_HOSTNAME: "pds.example.com"
467 SERVER_HOST: "0.0.0.0"
468 SERVER_PORT: "3000"
469 S3_ENDPOINT: "http://bspds-minio:9000"
470 AWS_REGION: "us-east-1"
471 S3_BUCKET: "pds-blobs"
472 VALKEY_URL: "redis://bspds-valkey:6379"
473 APPVIEW_URL: "https://api.bsky.app"
474 CRAWLERS: "https://bsky.network"
475 FRONTEND_DIR: "/app/frontend/dist"
476---
477apiVersion: apps/v1
478kind: Deployment
479metadata:
480 name: bspds
481 namespace: bspds
482spec:
483 replicas: 2
484 selector:
485 matchLabels:
486 app: bspds
487 template:
488 metadata:
489 labels:
490 app: bspds
491 spec:
492 imagePullSecrets:
493 - name: regcred # Remove if using public registry
494 affinity:
495 podAntiAffinity:
496 preferredDuringSchedulingIgnoredDuringExecution:
497 - weight: 100
498 podAffinityTerm:
499 labelSelector:
500 matchLabels:
501 app: bspds
502 topologyKey: kubernetes.io/hostname
503 containers:
504 - name: bspds
505 image: your-registry.com/bspds:latest
506 ports:
507 - containerPort: 3000
508 name: http
509 envFrom:
510 - configMapRef:
511 name: bspds-config
512 env:
513 - name: DB_PASSWORD
514 valueFrom:
515 secretKeyRef:
516 name: bspds-db-credentials
517 key: password
518 - name: DATABASE_URL
519 value: "postgres://bspds:$(DB_PASSWORD)@bspds-db-rw:5432/pds"
520 - name: AWS_ACCESS_KEY_ID
521 valueFrom:
522 secretKeyRef:
523 name: bspds-minio-credentials
524 key: root-user
525 - name: AWS_SECRET_ACCESS_KEY
526 valueFrom:
527 secretKeyRef:
528 name: bspds-minio-credentials
529 key: root-password
530 - name: JWT_SECRET
531 valueFrom:
532 secretKeyRef:
533 name: bspds-secrets
534 key: jwt-secret
535 - name: DPOP_SECRET
536 valueFrom:
537 secretKeyRef:
538 name: bspds-secrets
539 key: dpop-secret
540 - name: MASTER_KEY
541 valueFrom:
542 secretKeyRef:
543 name: bspds-secrets
544 key: master-key
545 resources:
546 requests:
547 memory: "256Mi"
548 cpu: "100m"
549 limits:
550 memory: "1Gi"
551 cpu: "1000m"
552 livenessProbe:
553 httpGet:
554 path: /xrpc/_health
555 port: 3000
556 initialDelaySeconds: 30
557 periodSeconds: 10
558 failureThreshold: 3
559 readinessProbe:
560 httpGet:
561 path: /xrpc/_health
562 port: 3000
563 initialDelaySeconds: 5
564 periodSeconds: 5
565 failureThreshold: 3
566 securityContext:
567 runAsNonRoot: true
568 runAsUser: 1000
569 allowPrivilegeEscalation: false
570---
571apiVersion: v1
572kind: Service
573metadata:
574 name: bspds
575 namespace: bspds
576spec:
577 selector:
578 app: bspds
579 ports:
580 - port: 80
581 targetPort: 3000
582 name: http
583EOF
584```
585## 9. Configure Horizontal Pod Autoscaler
586```bash
587cat <<EOF | kubectl apply -f -
588apiVersion: autoscaling/v2
589kind: HorizontalPodAutoscaler
590metadata:
591 name: bspds
592 namespace: bspds
593spec:
594 scaleTargetRef:
595 apiVersion: apps/v1
596 kind: Deployment
597 name: bspds
598 minReplicas: 2
599 maxReplicas: 10
600 metrics:
601 - type: Resource
602 resource:
603 name: cpu
604 target:
605 type: Utilization
606 averageUtilization: 70
607 - type: Resource
608 resource:
609 name: memory
610 target:
611 type: Utilization
612 averageUtilization: 80
613 behavior:
614 scaleDown:
615 stabilizationWindowSeconds: 300
616 policies:
617 - type: Pods
618 value: 1
619 periodSeconds: 60
620 scaleUp:
621 stabilizationWindowSeconds: 0
622 policies:
623 - type: Percent
624 value: 100
625 periodSeconds: 15
626 - type: Pods
627 value: 4
628 periodSeconds: 15
629 selectPolicy: Max
630EOF
631```
632## 10. Configure Pod Disruption Budget
633```bash
634cat <<EOF | kubectl apply -f -
635apiVersion: policy/v1
636kind: PodDisruptionBudget
637metadata:
638 name: bspds
639 namespace: bspds
640spec:
641 minAvailable: 1
642 selector:
643 matchLabels:
644 app: bspds
645EOF
646```
647## 11. Configure TLS with cert-manager
648```bash
649cat <<EOF | kubectl apply -f -
650apiVersion: cert-manager.io/v1
651kind: ClusterIssuer
652metadata:
653 name: letsencrypt-prod
654spec:
655 acme:
656 server: https://acme-v02.api.letsencrypt.org/directory
657 email: your-email@example.com
658 privateKeySecretRef:
659 name: letsencrypt-prod
660 solvers:
661 - http01:
662 ingress:
663 class: nginx
664EOF
665```
666## 12. Configure Ingress
667```bash
668cat <<EOF | kubectl apply -f -
669apiVersion: networking.k8s.io/v1
670kind: Ingress
671metadata:
672 name: bspds
673 namespace: bspds
674 annotations:
675 cert-manager.io/cluster-issuer: letsencrypt-prod
676 nginx.ingress.kubernetes.io/proxy-read-timeout: "86400"
677 nginx.ingress.kubernetes.io/proxy-send-timeout: "86400"
678 nginx.ingress.kubernetes.io/proxy-body-size: "100m"
679 nginx.ingress.kubernetes.io/proxy-buffering: "off"
680 nginx.ingress.kubernetes.io/websocket-services: "bspds"
681spec:
682 ingressClassName: nginx
683 tls:
684 - hosts:
685 - pds.example.com
686 secretName: bspds-tls
687 rules:
688 - host: pds.example.com
689 http:
690 paths:
691 - path: /
692 pathType: Prefix
693 backend:
694 service:
695 name: bspds
696 port:
697 number: 80
698EOF
699```
700## 13. Configure Network Policies (Optional but Recommended)
701```bash
702cat <<EOF | kubectl apply -f -
703apiVersion: networking.k8s.io/v1
704kind: NetworkPolicy
705metadata:
706 name: bspds-network-policy
707 namespace: bspds
708spec:
709 podSelector:
710 matchLabels:
711 app: bspds
712 policyTypes:
713 - Ingress
714 - Egress
715 ingress:
716 - from:
717 - namespaceSelector:
718 matchLabels:
719 kubernetes.io/metadata.name: ingress-nginx
720 ports:
721 - protocol: TCP
722 port: 3000
723 egress:
724 - to:
725 - podSelector:
726 matchLabels:
727 app: bspds-db
728 ports:
729 - protocol: TCP
730 port: 5432
731 - to:
732 - podSelector:
733 matchLabels:
734 app: bspds-minio
735 ports:
736 - protocol: TCP
737 port: 9000
738 - to:
739 - podSelector:
740 matchLabels:
741 app: bspds-valkey
742 ports:
743 - protocol: TCP
744 port: 6379
745 - to: # Allow DNS
746 - namespaceSelector: {}
747 podSelector:
748 matchLabels:
749 k8s-app: kube-dns
750 ports:
751 - protocol: UDP
752 port: 53
753 - to: # Allow external HTTPS (for federation)
754 - ipBlock:
755 cidr: 0.0.0.0/0
756 ports:
757 - protocol: TCP
758 port: 443
759EOF
760```
761## 14. Deploy Prometheus Monitoring (Optional)
762```bash
763cat <<EOF | kubectl apply -f -
764apiVersion: monitoring.coreos.com/v1
765kind: ServiceMonitor
766metadata:
767 name: bspds
768 namespace: bspds
769 labels:
770 release: prometheus
771spec:
772 selector:
773 matchLabels:
774 app: bspds
775 endpoints:
776 - port: http
777 path: /metrics
778 interval: 30s
779EOF
780```
781---
782## Verification
783```bash
784# Check all pods are running
785kubectl get pods -n bspds
786# Check services
787kubectl get svc -n bspds
788# Check ingress
789kubectl get ingress -n bspds
790# Check certificate
791kubectl get certificate -n bspds
792# Test health endpoint
793curl -s https://pds.example.com/xrpc/_health | jq
794# Test DID endpoint
795curl -s https://pds.example.com/.well-known/atproto-did
796```
797---
798## Maintenance
799### View Logs
800```bash
801# All BSPDS pods
802kubectl logs -l app=bspds -n bspds -f
803# Specific pod
804kubectl logs -f deployment/bspds -n bspds
805```
806### Scale Manually
807```bash
808kubectl scale deployment bspds --replicas=5 -n bspds
809```
810### Update BSPDS
811```bash
812# Build and push new image
813docker build -t your-registry.com/bspds:v1.2.3 .
814docker push your-registry.com/bspds:v1.2.3
815# Update deployment
816kubectl set image deployment/bspds bspds=your-registry.com/bspds:v1.2.3 -n bspds
817# Watch rollout
818kubectl rollout status deployment/bspds -n bspds
819```
820### Backup Database
821```bash
822# For CloudNativePG
823kubectl cnpg backup bspds-db -n bspds
824# For StatefulSet
825kubectl exec -it bspds-db-0 -n bspds -- pg_dump -U bspds pds > backup-$(date +%Y%m%d).sql
826```
827### Run Migrations
828If you have a migration Job defined, you can re-run it:
829```bash
830# Delete old job first (if exists)
831kubectl delete job bspds-migrate -n bspds --ignore-not-found
832# Re-apply the migration job from step 7
833# Or simply restart the deployment - BSPDS runs migrations on startup
834kubectl rollout restart deployment/bspds -n bspds
835```
836---
837## Troubleshooting
838### Pod Won't Start
839```bash
840kubectl describe pod -l app=bspds -n bspds
841kubectl logs -l app=bspds -n bspds --previous
842```
843### Database Connection Issues
844```bash
845# Test connectivity from a debug pod
846kubectl run debug --rm -it --restart=Never --image=postgres:18-alpine -- \
847 psql "postgres://bspds:PASSWORD@bspds-db-rw:5432/pds" -c "SELECT 1"
848```
849### Certificate Issues
850```bash
851kubectl describe certificate bspds-tls -n bspds
852kubectl describe certificaterequest -n bspds
853kubectl logs -l app.kubernetes.io/name=cert-manager -n cert-manager
854```
855### View Resource Usage
856```bash
857kubectl top pods -n bspds
858kubectl top nodes
859```