Karpenter Disruption: Consolidation, Budgets, and Why Pods Go Pending at 3am

Karpenter does two jobs: it launches the right node for pending pods, and it throws nodes away when they are empty, underutilized, drifted, or interrupted. The second job is where most incidents come from.

If you only tuned requirements and never read disruption, you have an autoscaler that is also a chaos monkey.

What “disruption” means

A NodeClaim can be disrupted for:

Consolidation is optional. Drift is not — if you change an AMI alias, Karpenter will roll the fleet.

apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: stateless
spec:
  disruption:
    consolidationPolicy: WhenEmptyOrUnderutilized
    consolidateAfter: 1m
    budgets:
      - nodes: "10%"
      - nodes: "0"
        schedule: "0 8 * * 1-5"
        duration: 10h
        reasons:
          - Underutilized
          - Drifted

Read that budget as: during weekday business hours, do not consolidate or roll for drift. Empty-node cleanup can still happen unless you add a second budget that covers Empty.

Budgets are not PDBs

• PDB — Kubernetes will not voluntarily evict a pod if the budget would be violated.
• Karpenter budget — Karpenter will not start more node disruptions than you allow.

They stack. A PDB that cannot be satisfied (minAvailable: 100% on two replicas) means Karpenter waits, then either gives up or, for interruption, the instance dies anyway.

For Spot-tolerant apps use maxUnavailable: 1 and three replicas. See running Kubernetes on Spot.

consolidateAfter is a loaded gun

WhenEmptyOrUnderutilized + consolidateAfter: 0s (or a few seconds) means Karpenter will reshuffle as soon as utilization dips. Batch jobs finishing at 03:00 will trigger a pack, which evicts a remaining API replica, which pages.

Start conservative:

disruption:
  consolidationPolicy: WhenEmpty
  consolidateAfter: 5m

WhenEmpty only deletes nodes that have no real work left. Turn on underutilized consolidation after you have:

• PDBs on every user-facing Deployment
• zone spread
• a weekday nodes: "0" budget for Underutilized

Then raise to WhenEmptyOrUnderutilized with consolidateAfter: 10m and watch karpenter_voluntary_disruption_queue_failures_total.

Drift will roll your cluster

Pin AMIs if you do not want a surprise rollout:

apiVersion: karpenter.k8s.aws/v1
kind: EC2NodeClass
metadata:
  name: default
spec:
  amiSelectorTerms:
    - alias: al2023@v20240807
  amiFamily: AL2

Using alias: al2023@latest means every new Karpenter-discovered AMI is drift. That is fine if you budget it (nodes: 1 and a weekday freeze). It is not fine on Friday afternoon with no budget.

When you do want a rollout, change the alias in a PR and let budgets drip nodes: nodes: "5%" overnight.

do-not-disrupt is an escape hatch, not a strategy

metadata:
  annotations:
    karpenter.sh/do-not-disrupt: "true"

Put this on:

• the Karpenter controller Deployment itself
• ingress, CoreDNS, and other cluster-critical singletons if they ever land on Karpenter nodes (they should not)
• a one-off debug pod

If half your Deployments have it, consolidation never saves money and you will wonder why NodePools grow forever.

Interruption handling

Wire the interruption queue (SQS) so Karpenter gets rebalance and spot notices before the two-minute kill. Without it, drain starts when kubelet is already dying.

Taint / isolate controllers on on-demand. A Spot interruption of the Karpenter replica that is supposed to replace Spot nodes is a sad loop.

Common pitfalls

1. No budgets. Consolidation and drift can hit 30% of nodes at once.
2. PDB minAvailable = replica count. Voluntary drain never completes; interruption is a hard kill.
3. consolidateAfter: 0s in prod. Cute in a demo cluster.
4. @latest AMI with no freeze window.
5. System pods scheduled onto the consolidating NodePool.

A sane production starting point

disruption:
  consolidationPolicy: WhenEmptyOrUnderutilized
  consolidateAfter: 15m
  budgets:
    - nodes: "10%"
    - nodes: "0"
      schedule: "0 7 * * 1-5"
      duration: 12h
      reasons: ["Underutilized", "Drifted"]
    - nodes: "1"
      reasons: ["Drifted"]

Plus: critical workloads off Spot, PDBs that allow one eviction, interruption queue configured.

Measure karpenter_nodes_terminated by reason for a week before you tighten consolidateAfter.