kubernetes deployment#

deploy prefect-server to kubernetes with horizontal scaling support.

architecture#

                    ┌─────────────────┐
                    │  load balancer  │
                    │    (service)    │
                    └────────┬────────┘
                             │
         ┌───────────────────┼───────────────────┐
         │                   │                   │
    ┌────▼────┐         ┌────▼────┐         ┌────▼────┐
    │ api-1   │         │ api-2   │         │ api-N   │
    │ --no-   │         │ --no-   │         │ --no-   │
    │services │         │services │         │services │
    └────┬────┘         └────┬────┘         └────┬────┘
         │                   │                   │
         └───────────────────┼───────────────────┘
                             │
    ┌────────────────────────┼────────────────────────┐
    │                        │                        │
┌───▼───┐            ┌───────▼───────┐          ┌────▼────┐
│ redis │◄───────────│   services    │──────────►│postgres │
│(broker)│           │ (single inst) │          │  (db)   │
└───────┘            └───────────────┘          └─────────┘

quick start#

# deploy everything
kubectl apply -k k8s/

# check status
kubectl -n prefect get pods

# port-forward to access locally
kubectl -n prefect port-forward svc/prefect-server 4200:4200

# open http://localhost:4200/api/health

components#

scaling#

# scale API servers
kubectl -n prefect scale deployment/prefect-api --replicas=5

# background services must stay at 1 replica

configuration#

edit configmap.yaml for environment variables:

• PREFECT_DATABASE_BACKEND: sqlite or postgres
• PREFECT_BROKER_BACKEND: memory or redis
• PREFECT_SERVER_LOGGING_LEVEL: DEBUG, INFO, WARNING, ERROR

edit configmap.yaml secrets section for:

• PREFECT_DATABASE_URL: postgres connection string

commands#

single binary with subcommands:

prefect-server              # server + services (default)
prefect-server --no-services # API only (horizontal scaling)
prefect-server services      # background services only

image registry auth (atcr.io)#

atcr.io uses a credential helper for docker CLI auth. k8s needs explicit credentials:

# extract creds from docker credential helper
echo "atcr.io" | docker-credential-atcr get
# returns: {"ServerURL":"atcr.io","Username":"<handle>","Secret":"<token>"}

# create k8s secret with those values
kubectl -n prefect create secret docker-registry atcr-creds \
  --docker-server=atcr.io \
  --docker-username=<handle> \
  --docker-password=<token>

# patch deployments to use the secret
kubectl -n prefect patch deployment prefect-api \
  -p '{"spec":{"template":{"spec":{"imagePullSecrets":[{"name":"atcr-creds"}]}}}}'
kubectl -n prefect patch deployment prefect-services \
  -p '{"spec":{"template":{"spec":{"imagePullSecrets":[{"name":"atcr-creds"}]}}}}'

alternatively, add imagePullSecrets directly to the deployment manifests.

production considerations#

1. postgres: use a managed database (RDS, Cloud SQL, etc.)
2. redis: use a managed redis (ElastiCache, Memorystore, etc.)
3. ingress: add an ingress controller for external access
4. TLS: configure TLS termination at ingress
5. persistence: add PVC for postgres if using in-cluster
6. HPA: add horizontal pod autoscaler for API servers