Running Kubernetes on Spot: Interruptions, PDBs, and Node Pools That Survive

Spot (AWS), Spot VMs (Azure), and preemptible VMs (GCP) are the highest-leverage cost cut most platform teams have. They are also the fastest way to create a self-inflicted outage if you treat them like cheaper on-demand.

The rule: Spot is for capacity that can die in two minutes. Everything else stays on-demand.

What an interruption actually looks like

On AWS, the node gets a rebalance recommendation and then a two-minute instance interruption notice. kubelet should cordon and drain. Your pods get SIGTERM, then terminationGracePeriodSeconds, then SIGKILL.

If your app ignores SIGTERM, the grace period is fiction. If your PDB forbids eviction, the drain stalls and the instance disappears anyway.

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: api
spec:
  minAvailable: 2
  selector:
    matchLabels:
      app: api

minAvailable: 2 with two replicas means zero voluntary evictions. Spot drain is voluntary. The node dies anyway. You now have a hard kill, not a drain.

For Spot-tolerant services, use:

spec:
  maxUnavailable: 1

and run at least three replicas across zones so one interruption is not a brownout.

Split node pools on purpose

Do not run a single mixed pool and hope the scheduler does the right thing.

On-demand pool — system-critical and stateful:

• kube-system (CoreDNS, metrics-server, CNI)
• ingress controllers
• cluster autoscaler / Karpenter controller
• Prometheus / Loki (or run them elsewhere)
• StatefulSets with local disks you cannot lose

Spot pool — stateless and restartable:

• stateless APIs
• workers, consumers, batch
• CI runners
• preview environments

Taint the Spot pool and tolerate it only where you mean it:

# node
spec:
  taints:
    - key: spot
      value: "true"
      effect: NoSchedule
---
# workload that may run on Spot
spec:
  tolerations:
    - key: spot
      operator: Equal
      value: "true"
      effect: NoSchedule
  topologySpreadConstraints:
    - maxSkew: 1
      topologyKey: topology.kubernetes.io/zone
      whenUnsatisfiable: DoNotSchedule
      labelSelector:
        matchLabels:
          app: api
  affinity:
    nodeAffinity:
      preferredDuringSchedulingIgnoredDuringExecution:
        - weight: 80
          preference:
            matchExpressions:
              - key: karpenter.sh/capacity-type
                operator: In
                values: ["spot"]
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
          - matchExpressions:
              - key: kubernetes.io/os
                operator: In
                values: ["linux"]

Prefer Spot, require zone spread, and keep a fallback of on-demand when Spot is gone.

Karpenter / Cluster Autoscaler notes

With Karpenter, use a NodePool that allows Spot and a separate NodePool that only allows on-demand for tainted system workloads.

apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: spot-stateless
spec:
  template:
    spec:
      taints:
        - key: spot
          value: "true"
          effect: NoSchedule
      requirements:
        - key: karpenter.sh/capacity-type
          operator: In
          values: ["spot"]
        - key: kubernetes.io/arch
          operator: In
          values: ["amd64"]
  disruption:
    consolidationPolicy: WhenEmptyOrUnderutilized
    consolidateAfter: 5m

Consolidation on Spot is aggressive by design. If you consolidate under a PDB that cannot be satisfied, pods go Pending. Watch karpenter_disruption_* metrics before you tighten consolidateAfter.

Cluster Autoscaler users: enable --expander=least-waste or priority expanders so on-demand pools are not scaled first for Spot-tolerant pods. Annotate pods that must not land on Spot (cluster-autoscaler.kubernetes.io/safe-to-evict: "false" is not a Spot shield; it only delays CA eviction).

Application contract

A service is Spot-safe only if all of these are true:

1. It is replicated (replicas >= 3 for anything user-facing).
2. PDB allows at least one eviction.
3. terminationGracePeriodSeconds matches real shutdown time (drain DB connections, finish in-flight HTTP).
4. Readiness fails fast when the process is shutting down (stop taking new work on SIGTERM).
5. No local disk that is the source of truth.
6. Clients retry (idempotent writes, backoff).

spec:
  terminationGracePeriodSeconds: 45
  containers:
    - name: api
      lifecycle:
        preStop:
          exec:
            command: ["/bin/sh", "-c", "sleep 5"]
      readinessProbe:
        httpGet:
          path: /ready
          port: 8080
        periodSeconds: 5

The preStop sleep is a cheap way to let the Service endpoint controller drop the pod from endpoints before SIGTERM hits the process. Pair it with a readiness gate if you are on a recent Kubernetes.

Mixed capacity for the same Deployment

Two common patterns:

Preferred Spot, required capacity. One Deployment, prefer Spot, allow on-demand. Simple. During a Spot drought, you pay on-demand automatically.

Hard split. Two Deployments or a topology that pins replicas/2 to on-demand. More control, more YAML. Use this for SLOs that cannot tolerate a full Spot reclaim in one AZ.

Do not put a single-replica Redis on Spot and call it “HA because Cluster Autoscaler will replace it.” Replacement is not zero-downtime.

Common pitfalls

1. PDB minAvailable equal to replica count — drain cannot proceed; interruption is a hard kill.
2. System pods on Spot — CoreDNS blip looks like a cluster outage.
3. Grace period of 30s on a worker that needs 2 minutes to finish a message — you will double-publish or lose work unless the consumer is idempotent.
4. One AZ Spot pool — reclaim events are correlated inside a capacity pool.
5. Ignoring rebalance recommendations — by the time the interruption notice arrives you have 120 seconds. Drain earlier when the rebalance event fires (AWS Node Termination Handler or Karpenter interruption queue).

Checklist before you flip a pool to Spot

• System and stateful workloads tainted off Spot
• User-facing services: 3+ replicas, maxUnavailable: 1, multi-AZ
• NTH / Karpenter interruption handling installed and paging
• Load test a drain (kubectl drain) in staging
• Cost dashboard shows Spot vs on-demand hours, not just “nodes”

See also Kubernetes cost and security hygiene.