Post

(Pt. 1) karpenter: how a pending pod becomes a node

(Pt. 1) karpenter: how a pending pod becomes a node

A four-part series on how Karpenter decides what hardware a Kubernetes cluster should run, and what Exa’s fork changes for fleets where the answer is measured in GPUs:

  1. How a pending pod becomes a node (this post)
  2. How a node earns its deletion
  3. kraftsman: when the control loop is the bottleneck
  4. kraftsman: pricing the fleet

0/8 nodes are available: 4 Insufficient nvidia.com/gpu, 8 Insufficient cpu. The pod sits Pending and kube-scheduler will sit with it forever, because its whole job is picking among nodes that already exist and none of them fit.

Two minutes later an instance is booting, a Node object appears, and the pod binds. Something in between did the scheduling work, and it wasn’t kube-scheduler.

That something is Karpenter. People call it an autoscaler, which is close but not quite the right shape. It’s better held as one scheduling simulator driven by two control loops.

A provisioning loop watches pending pods and asks what hardware would make them schedulable, then buys it. A disruption loop watches existing nodes and asks whether their pods could run elsewhere for less, then kills or replaces them. The first adds capacity, the second removes it, and pod binding stays with kube-scheduler. Part 2 is all about that second loop.

So let’s follow one pod from Pending to a machine, and meet the objects it passes through on the way.

Overview

Everything below happens inside one controller process. The double lines mark the two boundaries that matter: where Karpenter’s internal simulation ends and real API objects begin, and where the API objects end and a cloud instance begins.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
ONE PENDING POD BECOMES ONE NODE

  ┌──────────┐
  │   pod    │  Pending, unschedulable on every existing node
  └────┬─────┘
       │ pod watch event
═══════╪══════════════ karpenter controller ══════════════════════════
       ▼
  ┌───────────┐   ┌──────────────────────────────────────────────┐
  │  batcher  │──▶│ scheduler simulation                          │
  │ idle 1s,  │   │  1. ride existing nodes if they fit           │
  │ max 10s   │   │  2. otherwise open NodeClaims and pack pods   │
  └───────────┘   │     into them, narrowing instance types       │
                  └──────────────────────┬───────────────────────┘
                                         │
                                         ▼
                                NodeClaim object created
═══════════════════════════════╪═════════ apiserver ═══════════════════
                               ▼
                    cloudprovider.Create()
                               │
                               ▼
                    instance boots, kubelet starts
                               │
                               ▼
                    Node registers ──▶ labels synced, taints cleared
                               │
                               ▼
                    kube-scheduler finally binds the pod

The actual pod binding happens at the bottom of that diagram, done by kube-scheduler on a node Karpenter bought. Karpenter never binds a pod in its life. Its scheduler is a simulator whose only output is a decision about hardware.

The vocabulary

The pod’s trip passes through four object kinds, and the series leans on all of them.

1
2
3
  NodePool ──▶ NodeClass ──▶ NodeClaim ──▶ Node
  policy        provider      one capacity  registered
  you write     config        request       machine

A NodePool is the policy surface. It lists the acceptable instance types and zones, the taints each node carries, how much of the pool may be disrupted at once, and the nodeClassRef the claims point at. The NodeClass holds everything needed to actually boot a machine in the pool’s shape, like an EC2NodeClass carrying the AMI family, subnet and security group selectors, and capacity reservation selectors on AWS.

A NodeClaim is where the two meet. Karpenter stamps one per piece of capacity it wants, with the pool’s requirements resolved against real pending pods. When the kubelet on the resulting instance registers with the apiserver, the Node object is what shows up, linked back to its claim by the provider ID.

The pair people conflate is NodeClaim and Node. The claim is Karpenter’s intent before hardware exists while the Node is the kubelet’s presence after it. Everything interesting about lifecycle lives in the gap between them, bridged by matching the claim’s status.providerID to node.spec.providerID.

One simulator, two loops

1
2
3
4
5
6
7
8
9
10
11
12
13
  KUBE-SCHEDULER                   KARPENTER'S SCHEDULER

  pod pending?                     provisioner           disruption
       │                           pod pending,          pretend a node
  nodes that exist                 what node fits it?    is gone, where do
       │                                  │              its pods land?
       ▼                                  ▼                     │
  pick one, bind the pod           create claims                ▼
                                   (adds capacity)       delete or replace
                                                         (removes capacity)

                                   both run the same fit simulation
                                   over nodes that exist + could exist

Two consequences follow from putting the intelligence in a shared simulator rather than in either loop.

First, the scheduler never places anything. kube-scheduler still binds every pod to a node. Karpenter’s scheduling package is only a simulator: if this pod needed a node, what would that node look like? The answer comes back as NodeClaims, not bindings.

Second, each loop only moves in one direction. The provisioner can add a node but never remove one, because a pod that can’t schedule is never an argument for deleting capacity. The disruption loop owns every removal.

What people picture as a separate repacker is the same fit simulation pointed at existing nodes instead of pending ones. That’s also what lets a node’s pods replay through the logic that placed them.

That asymmetry is the whole reason Part 2 exists.

The batch window

The provisioner watches pod events and holds them in a batcher. One second of quiet ends the window, and ten seconds caps it, so a burst of pods gets solved in one pass instead of fifty.

1
2
3
4
pods pending    │─pod1─pod2─pod3────────────pod4──pod5─│
                │        └─<1s quiet→┘        └─<1s quiet→┘
batches         └── batch A: {1,2,3} ──┘      └─ batch B: {4,5} ─┘
                (max 10s caps a sustained burst)

Batching is the difference between autoscaling and just relaunching pods on nodes. Fifty pods arriving together can share one simulation and land on one right-sized node instead of racing through fifty separate launches. The knobs are BATCH_IDLE_DURATION (1s) and BATCH_MAX_DURATION (10s) in options.go.

Not every Pending pod joins the batch. The entry ticket is a pod kube-scheduler has already tried and marked Unschedulable. GetProvisionablePods then drops the ones Karpenter has no business provisioning for:

  • DaemonSet pods and static pods, which manage their own placement
  • pods kube-scheduler already won a preemption for (status.nominatedNodeName is set), because they’re about to land on capacity that eviction frees
  • pods that already have a nodeName

The provisioner also validates what’s left, rejecting a pod whose PVC topology can’t be satisfied or whose selector names a NodePool that doesn’t exist. The rest get fed to the simulator.

The simulation, walked through

The scheduler’s job per batch is a greedy bin-pack with a twist: it can fabricate bins.

Per NodePool it builds a NodeClaimTemplate from the pool’s requirements, labels, and taints plus the provider’s instance type catalog. Every existing node gets wrapped as an ExistingNode. Then it walks the pods largest first and gives each one three chances in order: an existing node, a claim opened earlier in the pass, or a fresh claim.

“Largest” means cpu, with memory as the tiebreak. GPU isn’t part of the sort order, but it is very much part of the fit check. Whether a pod fits somewhere is judged on every resource it requests, gpu included, plus its taints, selectors, and topology rules.

So let’s trace three pending pods through one pass, with nothing else in the cluster and a pool that allows two instance types.

1
2
3
4
5
6
7
8
instance types          cpu   gpu   price (on-demand)
  m5.2xlarge             8    -      $0.38
  g5.2xlarge             8    1      $1.21

pods                    requests
  pod-a                 6 cpu
  pod-b                 4 cpu, 1 nvidia.com/gpu
  pod-c                 3 cpu
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
pod, largest first     claim-1                     claim-2
                       types that still fit        types that still fit

1. pod-a  6cpu         opens claim-1
                       m5.2x  g5.2x

2. pod-b  4cpu+1gpu    6+4 = 10cpu, too big        opens claim-2
                       for either type             g5.2x only, since
                                                   m5.2x has no gpu

3. pod-c  3cpu         6+3 = 9cpu, too big         4+3 = 7cpu, fits
                                                   g5.2x

result                 pod-a, on the cheaper       pod-b + pod-c, on
                       of the two at launch        a g5.2xlarge

Two things in that table carry the whole mechanism.

The first is narrowing. A claim starts out able to become any type the pool allows, and each pod that joins intersects its needs into it. The type list then shrinks to whatever can still host everything on it, which is why claim-2 can only ever be a GPU type after pod-b arrives.

The second is that packing goes by fit rather than need. pod-c doesn’t want a GPU but claim-2 is where it fits, so it rides along. Among the claims already open the scheduler tries the one with the fewest pods first and takes the first that works.

The example skips DaemonSet overhead, but the real scheduler subtracts it before placing anything. An 8-cpu type offers a bit less than 8, and the GPU device plugin’s own request is already counted before pod-b lands. Topology spread is tracked as per-domain pod counts, so a DoNotSchedule spread across zones forces pods onto claims in different zones.

The output is a set of simulated claims, each holding its pods and the types it could still become.

From claim to machine

A simulated claim isn’t a specific machine so much as a shopping list. claim-1 above says “m5.2xlarge or g5.2xlarge, either zone, spot or on-demand” and leaves the final pick to the cloud provider. The provider takes the cheapest option it can actually get. That flexibility is deliberate, because a claim with many ways to be satisfied rarely fails on a sold-out type.

ToNodeClaim writes that list into a real NodeClaim object, tidying a few things on the way:

  • the instance types are sorted cheapest-first and cut off at 600 (MaxInstanceTypes), since the provider will only weigh so many options in one launch call
  • the claim records how the surviving types are actually sold, whether spot, on-demand, or reserved, so it never asks for spot on a type that only comes on-demand
  • any label the pool pins to a single value gets stamped on the claim, and the owner reference points at the NodePool so deleting the pool cleans up its claims

Then provisioner.CreateNodeClaims writes the objects, and the launch path hands them to the cloud provider one at a time.

What the cloud provider sells

The hardware catalog arrives as data through a small CloudProvider interface: GetInstanceTypes for the catalog, Create/Delete/Get/List for NodeClaims, and IsDrifted.

An InstanceType is a name, a capacity map, and a list of Offerings. Each offering is one (instance type, zone, capacity type) combination with a price and an availability bit, and it’s the unit everything gets priced in.

1
2
3
4
5
6
7
8
instance type g5.2xlarge, its offerings:

  zone    capacity-type   price     available
  a       spot            $0.51     yes
  b       spot            $0.47     no   ← ICE'd: marked unavailable after
  a       on-demand       $1.21     yes     InsufficientCapacity
  b       on-demand       $1.21     yes
  b       reserved        ~$0       yes  ← you're already paying for it

Capacity type is where the price differences live. On-demand is the list price. Spot is the same hardware at a market price, cheaper and revocable on two minutes’ notice, so cheap and interruptible are one property. Reserved is capacity you already paid for, like AWS On-Demand Capacity Reservations, and the NodeClass’s capacityReservationSelectorTerms pick which ones a pool may use.

The AWS provider prices reserved offerings at on-demand divided by ten million (see reserved_capacity_resolver.go). At near zero the scheduler grabs reserved capacity wherever a pod is eligible, and each offering tracks its remaining ReservationCapacity so a reservation can’t be oversubscribed. GPU capacity blocks ride the same path.

The price list isn’t decorative. Instance types get sorted cheapest-first at materialization, offerings are priced per zone because spot markets are zonal, and Part 2’s entire consolidation engine is “is there a cheaper offering for these pods.”

The gap between claim and node

A created NodeClaim is a promise. The lifecycle controller walks it through three conditions.

1
2
3
4
5
6
7
8
9
10
11
12
                         claim           instance         kubelet         node
                         created         boots            registers       Ready

  Launched                                   ✓
  Registered                                                  ✓
  Initialized                                                               ✓

  Launched    = instance exists, providerID stamped on the claim
  Registered  = Node appeared and matched providerID, labels
                synced, unregistered taint removed
  Initialized = Node Ready, startup taints gone, extended
                resources (nvidia.com/gpu) registered

Each condition is a distinct way to be stuck, and the liveness controller times them out separately. No instance after five minutes means the provider never delivered, so the claim is deleted and re-provisioned. An instance that hasn’t registered after fifteen minutes gets the same treatment. The machine exists but the kubelet never showed up, usually a bad AMI or bootstrap, and the fleet is better off trying again elsewhere.

Initialized is the slow one, because it waits on image pulls, CNI setup, and device plugins. A GPU node isn’t useful until nvidia.com/gpu appears in allocatable, and that is exactly what this condition waits for. Until then taints keep stray pods off the node, karpenter.sh/unregistered:NoExecute before registration and the pool’s startup taints after it.

The shadow cluster

Every simulation above reads from an in-memory mirror called state.Cluster rather than from the apiserver. Watches keep it current, and it holds one StateNode per real node and per in-flight claim.

1
2
3
4
5
6
7
8
  apiserver                              state.Cluster (in memory)

  Node foo               ──watch──▶      StateNode foo
  pods bound to foo      ──watch──▶        labels, taints
  NodeClaim for foo      ──watch──▶        pods' requests, summed
                                           daemonset requests
                                           host ports, volumes
                                           nominated?  marked for deletion?

Summing requests once per node instead of once per pod per pass is what makes a full scheduling simulation cheap enough to run on every pod batch and every ten-second disruption poll.

The last two fields on that card are how the two loops stay out of each other’s way, which they need because both act on the same pods. Here’s the race they would have without them.

1
2
3
4
5
6
7
8
9
10
WITHOUT THE NOTE                        WITH THE NOTE

t=0   provisioner: pod-x fits on        t=0   provisioner: pod-x fits on
      node foo, buy nothing                   foo, foo nominated for 20s
t=2   disruption: foo is nearly         t=2   disruption: foo is nominated,
      empty, delete it                        not a candidate
t=5   kube-scheduler: foo is            t=5   kube-scheduler binds pod-x
      tainted, nothing fits pod-x             to foo
t=6   pod-x pending again, next         t=20  note expires, foo is a
      pass buys a node it didn't need         normal candidate again

The note is NominateNodeForPod in cluster.go. After each batch the provisioner marks every existing node that received a pending pod as nominated for about twenty seconds, which covers the gap between simulation and bind. Disruption checks the mark twice, once when picking candidates and again when validating a command. It’s also what emits the Nominated event on a pod, so Pod should schedule on: node/X is the provisioner calling its shot.

The other field runs the other way.

1
2
3
4
5
6
7
  provisioner writes:               disruption writes:
  nominated (20s)                   marked for deletion
        │                                  │
        ▼                                  ▼
  disruption reads:                 provisioner reads:
  hands off this node               this node isn't capacity,
                                    and its pods need homes too

When a disruption command starts executing it marks its candidates for deletion. The provisioner stops counting them as room and schedules for their evacuating pods as if they were already pending. The ordering matters, and it’s why Part 2’s queue marks candidates after launching replacements. Mark first, and the provisioner races the queue to buy the same capacity.

Everything so far only grows the cluster. The provisioner has no path that ends with fewer nodes and no reason to have one. Making the fleet smaller and cheaper is a different loop’s job, running on a ten-second poll, and it’s what Part 2 walks through.

This post is licensed under CC BY 4.0 by the author.