(Pt. 4) kraftsman: pricing the fleet

A four-part series on how Karpenter decides what hardware a Kubernetes cluster should run, and what Exa’s fork changes for fleets where the answer is measured in GPUs:

1. How a pending pod becomes a node
2. How a node earns its deletion
3. kraftsman: when the control loop is the bottleneck
4. kraftsman: pricing the fleet (this post)

Part 3 was about the control loop keeping up. The other thing a GPU fleet does is spend money, and this is where upstream’s model shows its assumptions.

Karpenter was built to be safe first. A replacement gets priced at its worst-case compatible offering, one node replaces into at most one node, and packing only asks whether a pod fits. Those are reasonable defaults when instances are cheap and workloads are latency-sensitive.

But they leave money on the table in three specific ways on a fleet where a node is eight accelerators and the workloads are durable batch jobs. A claim priced at its worst case gets vetoed by the single most expensive zone it might land in. A spot pool with one spiked zone loses the savings available in five others.

Fit-only packing ignores that growing a claim can cost more than opening a cheaper one. And the 1:1 replacement rule hits the node with the most to save hardest: a fat node running half-empty can never consolidate, because its pods won’t fit a single cheaper node.

The kraftsman changes follow a request through its life: what a claim reserves before it exists, what a pod costs to place, and what a node takes to die.

  a request's life, and where the fork touches it:

    claim reserves   DaemonSet overhead, charged per realization
    pod places       fit-only or marginal-cost
    claim launches   spot-first, deferrable, lease-backed
    node dies        splits, OD→spot retry, lease reclaim

The DaemonSet tax, charged fairly

Part 1 mentioned that every claim reserves the sum of its compatible DaemonSets before a workload pod is even considered. What “compatible” means is where the subtlety lives.

A NodePool template’s requirements are broad: several zones, several label realizations, several instance shapes. Different DaemonSets select different realizations. A GPU device plugin runs on GPU nodes, a TPU plugin on TPU nodes, the CNI agent on everything. No concrete node ever carries both plugins, because no node is both.

Upstream checks a DaemonSet’s compatibility against the instance type’s labels only. A selector on a label the instance type doesn’t carry (a zone, a custom accelerator label) counts as compatible, so every such DaemonSet gets summed into the claim, including ones that can’t co-reside. On a pool spanning accelerator classes that’s a double tax on every node. The visible symptom is oversized claims: nodes bought one size up to hold capacity for daemons that were never coming.

Kraftsman’s daemonoverhead.go instead enumerates the label realizations where DaemonSets disagree. It sums the DaemonSet requests within each realization and charges the element-wise maximum across them.

pool template can realize: {gpu node} or {tpu node}

DaemonSets:  cilium (all)         200m cpu
             gpu-plugin (gpu)     500m cpu
             tpu-plugin (tpu)     300m cpu

upstream charges every claim:     200+500+300 = 1000m   ← a node that can't exist
realization-aware charges:        max(200+500, 200+300) = 700m per resource

OR semantics in nodeSelectorTerms and absent labels are handled conservatively, so a DaemonSet that might match a realization is charged to it. Past 4,096 realizations it falls back to the old over-reservation rather than blow up.

The result is claims sized for daemons that could actually coexist. On a mixed-accelerator pool that’s the difference between a node that fits the workload and a node that’s one size too big.

What a pod costs to place

Provisioning packing is fit-only: a pending pod joins the least-loaded in-flight claim that can hold it, and a new claim opens only when nothing fits. That’s the right default for bin-packing since it fills claims, but “fits” is the only criterion it checks and fitting isn’t free.

Growing a claim can force it onto a bigger instance type. The difference between “fits” and “cheap” is a price nobody computed.

The marginal-cost packing policy, selected per pool via the karpenter.sh/nodeclaim-packing-policy annotation in packing.go, prices every candidate placement.

Growing an in-flight claim costs the increase in its cheapest launch price. Opening a new claim costs that claim’s cheapest price. The pod goes wherever the delta is smaller, with ties preferring the in-flight claim.

pod: 8 cpu.  claim-1 is a g5.2xlarge-shaped claim with room.

  grow claim-1:   its instance options narrow to the next size up
                  → cheapest launch price rises $1.21 → $2.42   delta +$1.21
  open claim-2:   an m5.2xlarge at spot                          delta +$0.17

  binpack:        pod joins claim-1 (first fit)
  marginal-cost:  pod opens claim-2, $0.17 beats $1.21

Two degradations keep it predictable. A placement with no priceable options falls back to binpacking rather than splitting arbitrarily. Binpack pools price every move at zero, so behavior is untouched unless you opt in.

On pools where instance types have big price steps (anything with accelerators), “does the pod fit” stops being the only question worth asking.

The node that couldn’t die

Back to Part 2’s third gap. Single-node consolidation upstream either deletes a node or replaces it 1:1 with something cheaper.

The fat half-empty node, the one holding most of the pool’s idle spend, fails both checks: it’s not empty, and its pods won’t fit one cheaper node. So it stays, pass after pass, the best consolidation target in the fleet and an impossible one.

Kraftsman’s split fallback re-runs the same candidate under a price ceiling. The re-simulation forbids replacement capacity priced at or above the candidate’s own price. The scheduler then does what it naturally does: split the pods across several claims that each come in under the ceiling.

candidate: 8-GPU on-demand node, $30/hr, running 5 GPUs of pods

upstream:   5 GPUs won't fit any single node under $30  → no-op, forever
split:      ceiling $30 on every new claim
            → claim-2: 4-GPU spot node  $11
            → claim-3: 1-GPU spot node   $3
            total $14 < $30, margin ≥ 5%   → REPLACE 1 → 2

The guardrails are what make it a fallback and not a landmine: it only applies to single-candidate consolidations with at least two reschedulable pods.

The replacement count is bounded by max-consolidation-replacements, which defaults to one. That makes the split doubly opt-in: set CONSOLIDATION_SPLIT_FALLBACK=true and raise the bound. Attempts are also capped per pass by consolidation-split-max-attempts, because each one costs a full simulation. The split also has to beat a minimum savings margin (consolidation-split-min-savings, default 5%), so a node doesn’t churn into three nodes for pocket change.

Budget-exhausted attempts record themselves as inconclusive rather than no-op, so a later pass can try again instead of caching the wrong verdict.

When spot is the answer but only in some zones

Part 2’s worst-case pricing has a sharper version of the same problem. An on-demand candidate’s replacement claim gets priced at its worst compatible offering across every zone the claim allows. One zone where spot spiked past the budget vetoes the whole replacement even when the other five offer real savings.

claim allows zones a..f; priced at the worst compatible offering:

  a  spot $0.51   b  spot $0.47   c  spot $0.62
  d  spot $0.58   e  spot $0.55   f  spot $1.40  ← one spiked zone

  worst case $1.40 > candidate $1.21     → vetoed, nothing happens

  retry, spot only, zones that beat the budget:
  a..e  worst case $0.62 < $1.21         → pinned to spot in a..e

The OD-to-spot retry in consolidation.go re-prices the empty-handed claims. Each gets narrowed to spot only and to the zones whose spot offerings beat the budget, then checked again.

The narrowing is also the guarantee. The launched claim is pinned to those zones and to spot, so the worst case the retry priced is the worst the launch can do. Insufficient spot capacity fails the launch instead of silently falling back to another on-demand node at the price you were trying to leave.

It’s enabled by default (OD_TO_SPOT_CONSOLIDATION) because “the on-demand node stays” is a safe failure and “a spot node appears in a cheap zone” is a common success on a multi-zone pool.

Two related knobs round out the price checks:

1. spot-to-spot-min-instance-types makes upstream’s 15-option floor configurable. Fifteen assumes instance-type-diverse pools. A GPU pool pinned to one family can never present fifteen cheaper types, so it could never consolidate spot-to-spot at all. The launch is also capped to that many cheapest options. The launched type then sits inside the priced set and can’t be immediately re-consolidated, which is the churn loop the floor exists to prevent.
2. consolidation-replace-min-savings sets a fleet-wide savings floor on every replace decision, same idea as the split margin: moving a node has a real cost in disruption, and saving two cents doesn’t cover it.

When the cloud can’t sell yet

All of the above treats capacity markets as something you query. There’s a second mode a spot-heavy provider wants: treat them as something you wait on and probe. Kraftsman adds three hooks in spotfirst.go for providers that prefer spot but may temporarily fall back to on-demand:

1. LaunchDeferredError lets the provider say “not yet” instead of “failed.” A deferred launch requeues on the provider’s RetryAfter cadence rather than controller backoff. The deferral isn’t counted as a failure, though the five-minute launch clock keeps running from claim creation.
2. WithUnavailableOfferingsIgnored bypasses the insufficient-capacity cache for chosen capacity types. The ICE cache exists to stop hammering a sold-out market, but a market you stopped probing is a market you can’t notice refilling. Structural incompatibilities stay unavailable regardless.
3. DriftReasonOnDemandLeaseExpired is the reclaim path. When the provider falls back to on-demand under a lease, it marks the NodeClaim drifted with this reason once the lease expires. The drift controller sorts those candidates first and pins their replacements to spot. An insufficient-capacity launch then fails the command (and keeps the on-demand node) instead of falling back to another on-demand node.

The lease reclaim path, laid out:

  spot unavailable at launch
    │
    ▼
  provider falls back to on-demand under a lease
    │
    ▼
  lease expires ──▶ claim marked drifted (OnDemandLeaseExpired)
    │
    ▼
  drift sorts it first, replacement pinned to spot
    │
    ├─ spot capacity exists ────▶ swap completes
    └─ insufficient capacity ───▶ launch fails, on-demand stays

The lease swap is price correction rather than template drift, and it gets scheduled like one. The provider that uses these hooks is Exa-internal. On a spot-first fleet, though, “temporarily on-demand” is a state the autoscaler should model rather than a fact it should accept.

The lifecycle edges a self-managed fleet hits

Upstream’s two timeouts (five minutes to launch, fifteen to register) assume launch providers and boot times from a managed-node world. The fork’s fleet boots things that violate both. The registration timeout became configurable per pool and a third, opt-in timeout got added.

• karpenter.sh/nodeclaim-registration-timeout on a NodePool overrides the fifteen-minute registration budget per pool. Some accelerator capacity legitimately takes fifteen to twenty-five minutes from insert to kubelet. The upstream default deletes those claims mid-boot, replacing a slow node with a fresh timeout forever.
• NODECLAIM_INITIALIZATION_TIMEOUT covers the gap after registration. A node can register and then never initialize: startup taints never clear, extended resources never appear. Upstream keeps it forever: an instance billing full price, running no workload, and distorting every simulation that models its phantom capacity. The timeout deletes it like the others. It’s a fleet-wide setting, off unless you set it, and measured from registration rather than creation so the clock only starts once boot could plausibly have finished.
• The garbage collector reclaims claims that launched but never registered, once the instance has verifiably vanished. That covers a spot preemption during boot or an insert that failed after being accepted. A five-minute grace period covers the provider’s list-after-create consistency window first. Upstream holds those claims until the registration timeout, holding their pending pods hostage the whole time.
• And the inverse leak: a kubelet that registers after its claim was already being deleted creates a Node with no owner reference and no termination finalizer. On a managed cloud the cloud-controller-manager reaps it when the instance disappears. On a self-managed one nothing does, so it sits NotReady forever. The lifecycle controller deletes those nodes once the provider confirms the instance is gone.

None of these is a big change. They’re the seams you only find once your nodes aren’t all EKS-shaped.

What’s left

Put next to upstream, the delta reads as one theme: every place upstream chose the safe interpretation of cost, the fork makes the cost explicit and lets you set the floor.

upstream	kraftsman
daemonset tax = sum of everything compatible	element-wise max across realizations
fit-only packing	per-pool marginal-cost pricing
replace 1:1 or delete	bounded 1→N splits with a savings floor
worst-case price vetoes the claim	re-price spot-only in the cheap zones
spot-to-spot needs 15 cheaper types	configurable floor, launch capped to priced set
any cheaper replacement wins	minimum savings margins
launch fails or succeeds	launch can defer, leases get reclaimed to spot
5m/15m global timeouts	per-pool registration timeout, opt-in initialization timeout

The architecture is still Part 1’s. Pending pods become claims, claims become instances, a second loop asks whether the fleet could be cheaper, and a queue makes it so.

The assumptions inside that loop were all calibrated for a fleet where compute is cheap: what a node reserves, what a pod costs to place, what a replacement may become, how long boot can take. Point the same loop at hardware that isn’t. The scheduler already knows how to solve the problem, and you mostly just have to let it see the prices.