Kubernetes Ecosystem: From User to Contributor, Episode 9
← EP08: Karpenter · EP09: Karpenter vs Cluster Autoscaler · EP10: KubeVirt →

10 min read

TL;DR

• Karpenter vs Cluster Autoscaler comes down to an architectural ceiling: Cluster Autoscaler has to work generically across every cloud’s own autoscaling-group abstraction, which caps how smart its instance-selection can ever be
• Karpenter throws away the node-group abstraction and talks to the cloud’s instance-provisioning API directly — that’s the actual reason AWS built a new tool instead of extending Cluster Autoscaler
• Karpenter provisions faster and consolidates more aggressively for cost savings; Cluster Autoscaler’s scale-down behavior is deliberately more conservative
• Cluster Autoscaler remains the only mature option for several smaller cloud providers that don’t have a Karpenter provider implementation yet
• Recommendation: use Karpenter on AWS (and increasingly GKE) if you want its cost and speed advantages; stay on Cluster Autoscaler if you need one consistent tool across multiple clouds or you’re on a cloud Karpenter doesn’t support yet
• Contribution opportunity: building a Karpenter provider for a smaller, currently-unsupported cloud is real, meaningful, and directly helps teams stuck on Cluster Autoscaler’s more limited model purely for lack of an alternative

The Big Picture

CLUSTER AUTOSCALER                          KARPENTER
───────────────────                          ─────────
Generic abstraction: "a node group           Direct: "the cloud's actual
that can scale from N to M"                  instance-provisioning API"
        │                                            │
Has to work the same way whether             Talks to EC2's RunInstances API
it's an AWS ASG, a GCP MIG, or an            (or the equivalent) directly —
Azure VMSS — lowest common                   no generic abstraction ceiling
denominator by necessity                     to work around
        │                                            │
Instance type is WHATEVER the                Instance type is COMPUTED per
node group was pre-configured with           pending pod, from a flexible
                                              allowed range

Karpenter vs Cluster Autoscaler isn’t “new tool, old tool” — it’s a direct consequence of Cluster Autoscaler’s cross-cloud genericness being both its strength (works everywhere) and its ceiling (can never be smarter than the lowest common denominator of every cloud’s node-group abstraction).

Why AWS Built Karpenter Instead of Improving Cluster Autoscaler

Cluster Autoscaler was designed to work identically across clouds by scaling pre-existing node groups — ASGs on AWS, Managed Instance Groups on GCP, VM Scale Sets on Azure. That design constraint is exactly what limits it: it can only ever choose among the instance types and sizes someone already configured into a node group ahead of time, and it can only scale that group up or down as a unit.

AWS’s actual motivation for building Karpenter was to remove that ceiling entirely for their own cloud — by talking to EC2’s provisioning APIs directly, Karpenter can select from the full range of instance types AWS offers for every single provisioning decision, not just whatever a handful of pre-configured node groups happen to offer.

Provisioning Speed Compared

# Cluster Autoscaler: must first identify which existing node group to
# scale, then wait for that group's own scaling mechanism (an ASG launch,
# for instance) to complete
$ kubectl get events --field-selector reason=TriggeredScaleUp
# typically 1-3 minutes to a new node being schedulable

# Karpenter: computes the instance directly and calls the provisioning
# API without an intermediate node-group scaling step
$ kubectl get nodeclaims
# typically under a minute from pending pod to a schedulable node

The speed difference isn’t marginal at scale — for workloads with bursty, latency-sensitive scaling needs (batch job spikes, CI runner fleets), the extra minute or two Cluster Autoscaler’s node-group indirection adds is a real, felt difference, not a rounding error.

Cost Efficiency: Consolidation vs CA’s More Conservative Scale-Down

Karpenter’s consolidation behavior (EP08) actively looks for cheaper node configurations continuously, not just when pods are pending. Cluster Autoscaler’s scale-down logic is deliberately more conservative — it removes clearly-empty or clearly-underutilized nodes, but doesn’t proactively repack workloads onto fewer, better-fitting nodes the way Karpenter does by design. Teams migrating from Cluster Autoscaler to Karpenter commonly report meaningful compute cost reductions purely from this behavioral difference, independent of any instance-selection improvement.

Where Cluster Autoscaler Is Still the Right Choice

Multi-cloud consistency needs: if your platform runs on AWS, GCP, and Azure and you want one autoscaling tool behaving identically everywhere, Cluster Autoscaler’s cross-cloud design is a genuine advantage — Karpenter’s provider maturity still varies significantly by cloud.

Clouds without a mature Karpenter provider: several smaller cloud providers have no Karpenter implementation at all — Cluster Autoscaler, or that cloud’s own native autoscaler, remains the only real option.

Teams not hitting Cluster Autoscaler’s actual limits: if your workloads are stable, predictable, and your existing node groups already fit them well, Karpenter’s advantages may not be worth a migration — Cluster Autoscaler is mature, stable, and well-understood.

The Recommendation

On AWS specifically, and increasingly on GKE: default to Karpenter. The provisioning speed and consolidation cost savings are real and well-documented at this point, and this is where Karpenter’s provider maturity is strongest.

On any cloud without a mature Karpenter provider, or in a genuinely multi-cloud platform wanting one consistent tool: stay on Cluster Autoscaler. Don’t migrate for the sake of using the newer tool if your actual cloud or requirements don’t play to Karpenter’s strengths yet.

Don’t run both against the same node pool. Pick one scaler per cluster (or per clearly-separated node pool if you’re genuinely transitioning) — having both react to the same pending pods produces exactly the kind of conflicting-controller behavior you’d expect.

Production Gotchas

Migrating from Cluster Autoscaler to Karpenter mid-cluster requires careful sequencing, not a simultaneous cutover. Run them against separate, clearly labeled node pools during migration, and fully decommission Cluster Autoscaler’s management of a pool before letting Karpenter manage the same workloads.

Cluster Autoscaler’s node-group-based cost estimates and Karpenter’s per-instance cost awareness aren’t directly comparable without normalizing for what each is actually measuring. Don’t assume a raw percentage cost-savings figure from a vendor blog post transfers directly to your own workload mix.

Karpenter provider maturity genuinely varies by cloud — check the specific provider’s current feature list, not just “does Karpenter support my cloud” as a yes/no question.

Quick Reference

Contribution Opportunity: A Karpenter Provider for an Unsupported Cloud

The limitation: Teams running on smaller cloud providers — several exist with real production Kubernetes usage but no Karpenter implementation — are stuck with Cluster Autoscaler’s node-group model purely because nobody has built the equivalent Karpenter provider for their cloud, not because Cluster Autoscaler is actually the better fit for their workload.

Why it’s hard to fix: Building a new cloud provider for Karpenter means implementing that cloud’s instance-provisioning API integration, its spot/preemptible-equivalent interruption handling, and its networking/subnet discovery model from scratch — a genuine, multi-week engineering effort with no existing template for that specific cloud, even though the AWS and GCP providers exist as architectural references.

What a contribution-shaped fix looks like: Karpenter’s core (kubernetes-sigs/karpenter) is explicitly designed to support multiple cloud providers as separate implementations of a defined interface — the AWS and GCP provider source code is the reference for what a new provider needs to implement. For an engineer who already runs production Kubernetes on an unsupported cloud, building a minimal provider — even one covering just basic on-demand instance provisioning, without full spot/consolidation parity at first — is a real, high-value, currently-missing contribution that directly serves other teams on that same cloud stuck with no alternative to Cluster Autoscaler.

Key Takeaways

• Karpenter exists because Cluster Autoscaler’s cross-cloud node-group abstraction is a genuine architectural ceiling, not because Cluster Autoscaler was poorly built
• Karpenter provisions faster and consolidates more aggressively for cost savings — real, measurable advantages on the clouds it supports well
• Cluster Autoscaler remains the right choice for genuine multi-cloud consistency needs and for clouds without a mature Karpenter provider
• Never run both scalers against the same node pool simultaneously
• Building a Karpenter provider for a currently-unsupported cloud is a real, high-value contribution with existing architectural references (AWS, GCP) to learn from

What’s Next

Every tool so far in this series has assumed container workloads. EP10 closes the series with KubeVirt — running actual virtual machines as first-class citizens alongside pods on the same cluster, and why that migration path still matters in a container-first world.

Next: EP10 — KubeVirt: Running VMs on Kubernetes — and Why That Still Matters

Get EP10 in your inbox when it publishes → linuxcent.com/subscribe

The post Karpenter vs Cluster Autoscaler: Why AWS Built Its Own Scaler appeared first on Linuxcent.

Kubernetes Ecosystem: From User to Contributor, Episode 8
← EP07: Crossplane vs Terraform · EP08: Karpenter · EP09: Karpenter vs Cluster Autoscaler →

11 min read

TL;DR

• Karpenter node provisioning means no pre-defined node groups at all — it looks at pending pods’ actual resource requests and provisions the specific instance type and size that fits, directly
• NodePool and NodeClass are Karpenter’s two core CRDs: NodePool declares provisioning constraints and instance-type flexibility, NodeClass declares the cloud-specific details (AMI, subnets, security groups)
• Consolidation is Karpenter’s continuous bin-packing behavior — it doesn’t just scale up when pods are pending, it actively replaces underutilized nodes with better-fitting ones to reduce cost
• Karpenter handles spot interruption notices natively, draining gracefully before the two-minute warning expires, rather than relying on a separate spot-handling daemon
• Originally AWS-only, Karpenter has been donated to Kubernetes SIGs specifically to become a cross-cloud project — provider parity for GKE, AKS, and others is real, current, in-progress work
• Contribution opportunity: non-AWS provider feature parity is an explicitly open area with active upstream tracking — a genuinely current place to contribute

The Big Picture

CLUSTER AUTOSCALER MODEL                    KARPENTER MODEL
─────────────────────────                    ────────────────
Pre-defined node groups                      No node groups
(ASG A: m5.large, ASG B: m5.xlarge, ...)     Pending pod: needs 2 vCPU, 4Gi
        │                                            │
Pod pending, no capacity                     Karpenter evaluates: cheapest
        │                                    instance type that actually
Scale UP the node group                      fits, from a flexible list —
that (roughly) fits                          could be any instance family
        │                                    allowed by the NodePool
New node joins — may be                              │
oversized or undersized                      Provisions exactly that instance
for the actual pod                           — right-sized to the real
                                              pending workload

Karpenter node provisioning removes the middle abstraction layer entirely — instead of scaling a pre-sized group and hoping the group’s instance type roughly matches what’s pending, it computes the actual best-fit instance for the actual pending pods, every time.

NodePool and NodeClass: Karpenter’s Core CRDs

apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: general-purpose
spec:
  template:
    spec:
      requirements:
      - key: karpenter.k8s.aws/instance-category
        operator: In
        values: ["c", "m", "r"]      # flexible across instance families
      - key: karpenter.k8s.aws/instance-generation
        operator: Gt
        values: ["4"]
      nodeClassRef:
        name: default
  disruption:
    consolidationPolicy: WhenUnderutilized
    expireAfter: 720h
---
apiVersion: karpenter.k8s.aws/v1
kind: EC2NodeClass
metadata:
  name: default
spec:
  amiFamily: AL2023
  subnetSelectorTerms:
  - tags: {karpenter.sh/discovery: my-cluster}
  securityGroupSelectorTerms:
  - tags: {karpenter.sh/discovery: my-cluster}

NodePool says “here’s the range of instance types you’re allowed to choose from, and here’s the disruption policy” — it’s about scheduling flexibility. EC2NodeClass (or the equivalent for other providers) says “here’s the actual cloud-specific detail” — AMI, subnets, security groups. Splitting these two concerns is deliberate: a platform team can offer multiple NodePools with different cost/performance trade-offs, all referencing the same underlying NodeClass.

How Karpenter Actually Picks an Instance Type

$ kubectl get nodeclaims
NAME            TYPE          ZONE         NODE               READY   AGE
general-x7k2l   c6a.xlarge    us-east-1a   ip-10-0-1-42...    True    45s

$ kubectl describe nodeclaim general-x7k2l
...
Events:
  Reason              Message
  ------              -------
  Launched            Launched instance: i-0abc123... c6a.xlarge
  #                    ^^^^^^^^^^ — chosen because it was the cheapest
  #                    instance type in the allowed range that fit
  #                    the pending pods' actual CPU/memory requests

A NodeClaim is the record of one provisioning decision — it shows exactly which instance type Karpenter chose and why, unlike a node-group scale-up event, which just tells you the group’s already-fixed instance type was used again regardless of fit.

Consolidation: Karpenter’s Continuous Bin-Packing

# Karpenter continuously evaluates whether existing nodes could be
# consolidated into fewer, better-utilized nodes
$ kubectl get nodeclaims -o wide
NAME            TYPE         CPU-UTIL   MEM-UTIL
node-a          m5.2xlarge   15%        20%
node-b          m5.2xlarge   18%        22%
#                                             both underutilized — Karpenter
#                                             may consolidate these two onto
#                                             a single, smaller instance

This is the behavior that most differentiates Karpenter from a traditional autoscaler: it doesn’t just react to pending pods by scaling up. It continuously looks for opportunities to replace a set of underutilized nodes with fewer, better-fitting ones — actively working to reduce cost, not just meet demand.

Interruption Handling: Spot Instances Done Right

# Karpenter watches for AWS's spot interruption notice natively
$ kubectl get events --field-selector reason=DisruptionTerminating
LAST SEEN   REASON                  MESSAGE
5s          DisruptionTerminating   Node terminating due to spot interruption,
                                     draining pods gracefully before 2-minute deadline

Before Karpenter, handling spot interruptions gracefully typically meant running a separate tool (like AWS Node Termination Handler) alongside your autoscaler. Karpenter builds this in directly — it’s part of the same controller making the original provisioning decision, not a bolted-on separate system watching for the same signal independently.

Production Gotchas

Aggressive consolidation without a properly configured PodDisruptionBudget can cause more pod churn than teams expect. Karpenter respects PDBs, but if you haven’t set them, consolidation can evict pods more freely than a team used to Cluster Autoscaler’s more conservative default behavior anticipated.

A misconfigured NodeClass (wrong subnet tags, wrong security group selector) fails silently from the scheduler’s point of view — pods just stay pending, and the actual error is in Karpenter’s controller logs or NodeClaim events, not anywhere the standard kubectl get pods workflow surfaces by default.

Karpenter’s own controller needs real resource requests and, ideally, its own dedicated nodes or a stable node pool — running the thing that provisions your nodes on a node that might itself get consolidated away is a bootstrapping problem worth designing around explicitly.

Quick Reference

kubectl get nodepools                       # provisioning policies defined
kubectl get nodeclasses                     # cloud-specific node configuration
kubectl get nodeclaims                      # individual provisioning decisions
kubectl describe nodeclaim <name>            # why this specific instance was chosen
kubectl get events --field-selector reason=DisruptionTerminating   # interruption/consolidation activity

Contribution Opportunity: Closing Non-AWS Provider Feature Parity

The limitation: Karpenter started as an AWS-specific project and has since been donated to Kubernetes SIGs specifically to become a genuinely cross-cloud tool. The GKE provider and others are real and actively developed, but feature parity with the mature AWS provider — specific instance-selection heuristics, certain disruption/consolidation behaviors, provider-specific NodeClass capabilities — isn’t complete yet, and this is openly tracked, not hidden.

Why it’s hard to fix: Each cloud’s instance-provisioning API, spot-interruption signaling mechanism, and networking model differs meaningfully — replicating AWS provider behavior on GCP or Azure isn’t a port, it’s a re-implementation against a different API with different constraints and different edge cases, done by a provider team with less historical runtime than the original AWS implementation had.

What a contribution-shaped fix looks like: The kubernetes-sigs/karpenter-provider-gcp (and other provider) repositories maintain their own issue trackers with specific, scoped feature-parity gaps against the AWS implementation — this isn’t a vague “make it better,” it’s a list of concrete, individually-tractable items. Picking one specific parity gap, understanding how the AWS provider solved the equivalent problem, and implementing the analogous behavior for the target cloud is real, wanted, trackable upstream work — precisely the shape of contribution this series has been pointing at throughout.

Key Takeaways

• Karpenter provisions the actual best-fit instance for pending pods directly, with no pre-defined node-group middle layer
• NodePool (scheduling flexibility) and NodeClass (cloud-specific detail) are deliberately separated concerns
• Consolidation is active, continuous bin-packing — Karpenter looks for cost savings, not just capacity needs
• Native spot interruption handling removes the need for a separate termination-handling tool
• Non-AWS provider feature parity is explicitly open, tracked work — a real, current, well-scoped contribution opportunity in a project under active cross-cloud expansion

What’s Next

EP09 puts Karpenter head-to-head against the tool it’s increasingly replacing — Cluster Autoscaler — and gives a clear recommendation for when the older, node-group model is still the right choice.

Next: EP09 — Karpenter vs Cluster Autoscaler: Why AWS Built Its Own Scaler

Get EP09 in your inbox when it publishes → linuxcent.com/subscribe

The post Karpenter: Just-in-Time Node Provisioning for Kubernetes appeared first on Linuxcent.