Introduction

Docker packages a Machine Learning API into a portable container image. Kubernetes operates that image across one or more machines. It schedules containers, maintains the desired number of replicas, replaces failed instances, provides stable service discovery, controls configuration, and coordinates application updates.

In this lesson, the FastAPI customer-churn service from the previous lessons is deployed to Kubernetes. The workload uses an immutable container image containing the API and approved model pipeline. Kubernetes manages multiple application Pods behind a stable Service.

Kubernetes can improve availability and operational consistency, but it does not validate the Machine Learning model. Model quality, input correctness, drift, responsible use, security, and business outcomes still require separate controls.

Learning Objectives

Prerequisites

docker version
kubectl version --client
kubectl config current-context

What Is Kubernetes?

Kubernetes is a container-orchestration platform. Users declare the desired state of workloads through Kubernetes resources, and controllers continually work to move the actual cluster state toward that desired state.

Kubernetes Architecture

Core Kubernetes Resources

Pod, ReplicaSet, and Deployment

A Pod is an execution unit and may be replaced at any time. A Deployment manages ReplicaSets, while a ReplicaSet maintains the desired number of matching Pods.

Create a Namespace

apiVersion: v1
kind: Namespace
metadata:
  name: ml-inference
  labels:
    app.kubernetes.io/part-of: machine-learning
    environment: development

kubectl apply \
  --filename namespace.yaml

kubectl get namespaces

Use Consistent Labels

labels:
  app.kubernetes.io/name: customer-churn-api
  app.kubernetes.io/instance: customer-churn-api-production
  app.kubernetes.io/version: "1.0.0"
  app.kubernetes.io/component: inference-api
  app.kubernetes.io/part-of: machine-learning
  app.kubernetes.io/managed-by: kubectl

Labels support selection, grouping, policy, monitoring, and operational reporting. Service and Deployment selectors must match the intended Pods exactly.

Create a ConfigMap

apiVersion: v1
kind: ConfigMap
metadata:
  name: customer-churn-api-config
  namespace: ml-inference
data:
  MODEL_PATH: /app/artifacts/customer_churn_pipeline.joblib
  METADATA_PATH: /app/artifacts/model_metadata.json
  MAXIMUM_BATCH_SIZE: "100"
  LOG_LEVEL: INFO

kubectl apply \
  --filename configmap.yaml

Handle Sensitive Configuration

apiVersion: v1
kind: Secret
metadata:
  name: customer-churn-api-secret
  namespace: ml-inference
type: Opaque
stringData:
  API_KEY: replace-through-approved-secret-process

Create the ML API Deployment

apiVersion: apps/v1
kind: Deployment
metadata:
  name: customer-churn-api
  namespace: ml-inference
  labels:
    app.kubernetes.io/name: customer-churn-api
    app.kubernetes.io/component: inference-api
spec:
  replicas: 2

  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 0
      maxSurge: 1

  selector:
    matchLabels:
      app.kubernetes.io/name: customer-churn-api

  template:
    metadata:
      labels:
        app.kubernetes.io/name: customer-churn-api
        app.kubernetes.io/component: inference-api
        app.kubernetes.io/version: "1.0.0"

    spec:
      serviceAccountName: customer-churn-api

      securityContext:
        runAsNonRoot: true
        seccompProfile:
          type: RuntimeDefault

      containers:
        - name: ml-api
          image: registry.example.com/ml/customer-churn-api:1.0.0
          imagePullPolicy: IfNotPresent

          ports:
            - name: http
              containerPort: 8000
              protocol: TCP

          envFrom:
            - configMapRef:
                name: customer-churn-api-config

          env:
            - name: API_KEY
              valueFrom:
                secretKeyRef:
                  name: customer-churn-api-secret
                  key: API_KEY

          resources:
            requests:
              cpu: "250m"
              memory: "512Mi"
            limits:
              cpu: "1"
              memory: "1Gi"

          startupProbe:
            httpGet:
              path: /health
              port: http
            periodSeconds: 5
            failureThreshold: 30

          readinessProbe:
            httpGet:
              path: /ready
              port: http
            periodSeconds: 10
            timeoutSeconds: 3
            failureThreshold: 3

          livenessProbe:
            httpGet:
              path: /health
              port: http
            periodSeconds: 20
            timeoutSeconds: 3
            failureThreshold: 3

          securityContext:
            allowPrivilegeEscalation: false
            readOnlyRootFilesystem: true
            capabilities:
              drop:
                - ALL

          volumeMounts:
            - name: temporary-files
              mountPath: /tmp

      volumes:
        - name: temporary-files
          emptyDir: {}

Resource Requests and Limits

Startup, Readiness, and Liveness Probes

Create a Service

apiVersion: v1
kind: Service
metadata:
  name: customer-churn-api
  namespace: ml-inference
  labels:
    app.kubernetes.io/name: customer-churn-api
spec:
  type: ClusterIP

  selector:
    app.kubernetes.io/name: customer-churn-api

  ports:
    - name: http
      protocol: TCP
      port: 80
      targetPort: http

kubectl apply \
  --filename deployment.yaml \
  --filename service.yaml

Kubernetes Service Types

Local Port Forwarding

kubectl port-forward \
  --namespace ml-inference \
  service/customer-churn-api \
  8000:80

curl http://127.0.0.1:8000/health

curl http://127.0.0.1:8000/ready

Test the Kubernetes Prediction API

curl \
  --request POST \
  --header "Content-Type: application/json" \
  --data '{
    "tenure_months": 18,
    "monthly_charges": 64.5,
    "total_charges": 1161.0,
    "contract_type": "Monthly",
    "payment_method": "Electronic",
    "tech_support": "No"
  }' \
  http://127.0.0.1:8000/predictions

Inspect Kubernetes Resources

kubectl get all \
  --namespace ml-inference

kubectl get pods \
  --namespace ml-inference \
  --output wide

kubectl describe deployment \
  customer-churn-api \
  --namespace ml-inference

kubectl get endpoints \
  customer-churn-api \
  --namespace ml-inference

View Application Logs

kubectl logs \
  --namespace ml-inference \
  deployment/customer-churn-api

kubectl logs \
  --namespace ml-inference \
  deployment/customer-churn-api \
  --follow \
  --tail=100

kubectl logs \
  --namespace ml-inference \
  pod/replace-with-pod-name \
  --previous

Scale the Deployment Manually

kubectl scale deployment \
  customer-churn-api \
  --namespace ml-inference \
  --replicas=3

kubectl get pods \
  --namespace ml-inference \
  --watch

Configure Horizontal Pod Autoscaling

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: customer-churn-api
  namespace: ml-inference
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: customer-churn-api

  minReplicas: 2
  maxReplicas: 10

  behavior:
    scaleUp:
      stabilizationWindowSeconds: 0
      policies:
        - type: Percent
          value: 100
          periodSeconds: 60

    scaleDown:
      stabilizationWindowSeconds: 300
      policies:
        - type: Percent
          value: 25
          periodSeconds: 60

  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 60

kubectl apply \
  --filename hpa.yaml

kubectl get hpa \
  --namespace ml-inference

Choose Useful Scaling Signals

Perform a Rolling Update

kubectl set image deployment/customer-churn-api \
  ml-api=registry.example.com/ml/customer-churn-api:1.1.0 \
  --namespace ml-inference

kubectl rollout status \
  deployment/customer-churn-api \
  --namespace ml-inference

kubectl rollout history \
  deployment/customer-churn-api \
  --namespace ml-inference

Roll Back a Deployment

kubectl rollout undo \
  deployment/customer-churn-api \
  --namespace ml-inference

kubectl rollout status \
  deployment/customer-churn-api \
  --namespace ml-inference

Add a Pod Disruption Budget

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: customer-churn-api
  namespace: ml-inference
spec:
  minAvailable: 1

  selector:
    matchLabels:
      app.kubernetes.io/name: customer-churn-api

Spread Replicas Across Nodes

spec:
  template:
    spec:
      topologySpreadConstraints:
        - maxSkew: 1
          topologyKey: kubernetes.io/hostname
          whenUnsatisfiable: ScheduleAnyway
          labelSelector:
            matchLabels:
              app.kubernetes.io/name: customer-churn-api

Topology-spread constraints can reduce concentration of replicas on one node. Scheduling behavior must be evaluated against available nodes, zones, labels, capacity, and availability requirements.

Schedule Accelerator Workloads

resources:
  requests:
    cpu: "1"
    memory: "4Gi"
    nvidia.com/gpu: "1"
  limits:
    cpu: "2"
    memory: "8Gi"
    nvidia.com/gpu: "1"

Model Artifact Distribution

Example Init Container Pattern

spec:
  template:
    spec:
      initContainers:
        - name: model-downloader
          image: registry.example.com/ml/model-downloader:1.0.0
          args:
            - download
            - --model-version=1.0.0
            - --destination=/models/model.joblib
          volumeMounts:
            - name: model-files
              mountPath: /models

      containers:
        - name: ml-api
          image: registry.example.com/ml/customer-churn-api:1.0.0
          env:
            - name: MODEL_PATH
              value: /models/model.joblib
          volumeMounts:
            - name: model-files
              mountPath: /models
              readOnly: true

      volumes:
        - name: model-files
          emptyDir: {}

Service Accounts and Access

apiVersion: v1
kind: ServiceAccount
metadata:
  name: customer-churn-api
  namespace: ml-inference
automountServiceAccountToken: false

Network Policies

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: customer-churn-api-ingress
  namespace: ml-inference
spec:
  podSelector:
    matchLabels:
      app.kubernetes.io/name: customer-churn-api

  policyTypes:
    - Ingress

  ingress:
    - from:
        - namespaceSelector:
            matchLabels:
              access-to-ml-api: "true"
      ports:
        - protocol: TCP
          port: 8000

Pod Security Practices

Kubernetes Observability

kubectl get events \
  --namespace ml-inference \
  --sort-by=.metadata.creationTimestamp

ML Observability Beyond Kubernetes

Kubernetes Cost Considerations

Troubleshooting Kubernetes ML Workloads

kubectl describe pod \
  replace-with-pod-name \
  --namespace ml-inference

kubectl logs \
  replace-with-pod-name \
  --namespace ml-inference

kubectl get events \
  --namespace ml-inference \
  --sort-by=.metadata.creationTimestamp

Kubernetes in CI/CD

GitOps for Kubernetes

GitOps uses version-controlled declarative configuration as the desired deployment state. An authorized controller can reconcile the cluster with approved repository changes.

Responsible Kubernetes ML

Common Kubernetes Mistakes

Clean Up Learning Resources

kubectl delete \
  --filename hpa.yaml \
  --filename pod-disruption-budget.yaml \
  --filename service.yaml \
  --filename deployment.yaml \
  --filename configmap.yaml \
  --namespace ml-inference

kubectl delete namespace \
  ml-inference

Recommended Kubernetes Workflow

Hands-On Practice

Knowledge Check

Kubernetes Readiness Checklist

Lesson Summary