Kubernetes Ecosystem: From User to Contributor, Episode 5
← EP04: Rancher · EP05: Cluster API · EP06: Crossplane →
11 min read
TL;DR
- Cluster API (CAPI) declares Kubernetes clusters themselves — not just workloads running inside them — as Kubernetes objects:
Cluster,Machine,MachineDeployment, reconciled by controllers the same way aDeploymentreconciles pods - CAPI itself is infrastructure-agnostic — the actual provisioning logic lives in separate infrastructure providers (AWS, Azure, GCP, vSphere, and dozens more), each implementing the same core contract
- Bootstrapping is genuinely awkward by necessity: you need a Kubernetes cluster to run CAPI’s controllers before CAPI can create your real cluster — solved by a temporary “kind” cluster and a pivot step that moves CAPI’s own resources into the cluster it just created
- Rancher’s own newer provisioning (EP04) increasingly builds on CAPI patterns rather than reinventing cluster lifecycle management from scratch
- Provider version compatibility is a real, ongoing constraint — CAPI core and each infrastructure provider version independently, and not every combination is supported
- Contribution opportunity:
clusterctl move, the pivot operation, has well-documented fragility with resources it doesn’t natively understand — a concrete, scoped gap
The Big Picture
Cluster (the K8s object, not the K8s cluster itself)
│
├── Represents: this Cluster SHOULD exist
│
▼
MachineDeployment ──── mirrors Deployment/ReplicaSet/Pod exactly ────┐
│ │
▼ │
MachineSet │
│ │
▼ │
Machine ────────► Infrastructure Provider (AWS/Azure/GCP/vSphere/...) │
│ actually creates the VM/instance │
▼ │
Bootstrap Provider (kubeadm, typically) │
actually turns that VM into a working Kubernetes node ────────────────┘
Cluster API’s declarative cluster lifecycle model is the same reconciliation pattern Kubernetes already uses for workloads, applied one layer up: instead of a Deployment controller reconciling Pod objects into running containers, CAPI’s controllers reconcile Machine objects into running cloud instances that then join a cluster as nodes.
The Core Abstraction: Clusters and Machines as Kubernetes Objects
$ kubectl apply -f - <<EOF
apiVersion: cluster.x-k8s.io/v1beta1
kind: Cluster
metadata:
name: prod-us-east
spec:
clusterNetwork:
pods:
cidrBlocks: ["192.168.0.0/16"]
infrastructureRef:
apiVersion: infrastructure.cluster.x-k8s.io/v1beta2
kind: AWSCluster
name: prod-us-east
EOF
$ kubectl get clusters
NAME PHASE AGE
prod-us-east Provisioning 45s
$ kubectl get machines
NAME CLUSTER PHASE VERSION
prod-us-east-cp-x7k2l prod-us-east Provisioning v1.28.5
The Cluster object is a declaration of intent, not the cluster itself — a management cluster (a separate, already-running Kubernetes cluster whose only job is to run CAPI’s controllers) watches these objects and does the actual work of calling out to AWS, Azure, or whatever provider is referenced, creating instances, and bootstrapping Kubernetes on them.
The Provider Model: How CAPI Stays Infrastructure-Agnostic
CAPI’s core (cluster-api) knows nothing about AWS, Azure, or any specific cloud. That knowledge lives in separate, independently-versioned infrastructure providers:
$ clusterctl init --infrastructure aws
Fetching providers
Installing cert-manager
Installing Provider="cluster-api" Version="v1.6.2"
Installing Provider="bootstrap-kubeadm" Version="v1.6.2"
Installing Provider="control-plane-kubeadm" Version="v1.6.2"
Installing Provider="infrastructure-aws" Version="v2.4.0"
# ^^^^^^ — versioned independently
# from core CAPI above
This split — core lifecycle logic separate from provider-specific implementation — is the same architectural pattern CNI and CSI use elsewhere in Kubernetes: a stable core contract, swappable implementations. It’s also exactly why CAPI’s ecosystem includes dozens of infrastructure providers (AWS, Azure, GCP, vSphere, OpenStack, Docker, bare metal, and many more) maintained by different teams at different paces.
A Management Cluster Managing Itself and Others: The Pivot
The genuinely awkward part of CAPI’s bootstrap story: you need a running Kubernetes cluster to host CAPI’s controllers before CAPI can create your first real cluster. The standard pattern:
# Step 1: spin up a throwaway local cluster just to run CAPI controllers
$ kind create cluster --name capi-bootstrap
$ clusterctl init --infrastructure aws
# Step 2: use that temporary management cluster to provision the REAL cluster
$ clusterctl generate cluster prod-us-east --infrastructure aws | kubectl apply -f -
# Step 3: move CAPI's own resources OFF the throwaway cluster and ONTO
# the cluster that was just created — "pivoting" management to itself
$ clusterctl move --to-kubeconfig=./prod-us-east.kubeconfig
Performing move...
Discovering Cluster API objects
Moving Cluster API objects: Clusters=1, Machines=3, ...
After the pivot, the cluster CAPI created is now managing its own lifecycle (and can go on to manage other clusters too) — the temporary kind cluster can be torn down. This bootstrap-then-pivot dance is elegant in theory and one of the more fragile operational moments in CAPI’s lifecycle in practice.
How Rancher and Others Build On CAPI
Rancher’s newer cluster provisioning (EP04) increasingly leans on CAPI patterns rather than maintaining entirely separate provisioning logic — the industry direction across the Kubernetes ecosystem has been toward CAPI as the shared substrate for “declare a cluster, get a cluster,” with vendors building their own UX and opinionated defaults on top rather than reinventing the reconciliation model itself.
⚠ Production Gotchas
Provider version compatibility is a real support matrix, not a “probably fine” assumption. Core CAPI and each infrastructure provider version independently — upgrading one without checking the compatibility matrix for the other is a common source of cryptic reconciliation failures.
clusterctl move is a rare, high-stakes operation — most teams run it once per cluster’s lifetime, if ever, which means nobody on the team has recent hands-on experience when something goes wrong. Test the pivot in a non-production scenario before relying on it for anything real.
A Machine stuck in Provisioning can mean the infrastructure provider, the bootstrap provider, or the actual cloud API — three different places to look, and the Machine object’s own status doesn’t always make it obvious which. Check the infrastructure-specific object (AWSMachine, AzureMachine, etc.) directly, not just the generic Machine.
Quick Reference
clusterctl init --infrastructure <provider> # install CAPI + a provider on the management cluster
clusterctl generate cluster <name> --infrastructure <provider> # generate cluster manifests
kubectl get clusters # cluster lifecycle phase
kubectl get machines # per-node provisioning phase
kubectl get awsmachines / azuremachines / ... # provider-specific detail
clusterctl move --to-kubeconfig=<path> # pivot management to another cluster
clusterctl describe cluster <name> # human-readable status tree
Contribution Opportunity: clusterctl move‘s Fragility With Non-Native Resources
The limitation: clusterctl move knows how to move CAPI’s own well-known resource types between management clusters cleanly. When a provider or an operator has added custom resources that reference or extend CAPI objects — a common real-world pattern — move doesn’t always understand the relationship, and teams have reported needing manual intervention (patching, reapplying, or reordering) to get a full pivot to succeed cleanly. This is documented in multiple open issues against the project, not a rare edge case.
Why it’s hard to fix: move‘s core logic has to correctly identify and preserve object references and ownership across an arbitrary graph of custom resources it wasn’t necessarily designed to know about — building a fully general solution risks either false confidence (silently missing a reference) or false failure (over-cautiously blocking a move that would have been fine). The CAPI maintainers have to weigh correctness against usability here, and it’s a genuinely hard design problem, not a simple bug.
What a contribution-shaped fix looks like: Two realistic, scoped starting points: (1) a --dry-run-style pre-flight checker for clusterctl move that specifically scans for custom resources referencing CAPI objects and flags them before the move attempt, rather than discovering the gap mid-operation; or (2) contributing a documented, tested procedure (and ideally a small helper tool) for the specific pattern of “extra resources referencing Machine/Cluster objects” that’s already been reported in the project’s issue tracker — turning a known, recurring support question into a documented, repeatable procedure.
Key Takeaways
- CAPI applies Kubernetes’ own reconciliation pattern one layer up —
ClusterandMachineobjects are declarations, reconciled into real infrastructure by provider-specific controllers - The core/provider split keeps CAPI infrastructure-agnostic, at the cost of independent versioning you have to track across a real compatibility matrix
- The bootstrap-then-pivot pattern is CAPI’s most elegant and most operationally fragile moment — rehearse it before you need it for real
- Rancher and other platform tools increasingly build their own provisioning UX on top of CAPI’s reconciliation model rather than replacing it
- The clearest contribution opportunity is
clusterctl move‘s handling of non-native custom resources — a documented, scoped gap with real prior art in the issue tracker
What’s Next
CAPI treats infrastructure — VMs, networks, load balancers — as the thing being reconciled into existence from Kubernetes objects. EP06 takes that same idea and generalizes it as far as it can go: Crossplane turns Kubernetes into a control plane for effectively any cloud resource, not just the ones needed to run Kubernetes itself.
Next: EP06 — Crossplane: Kubernetes as the Universal Control Plane
Get EP06 in your inbox when it publishes → linuxcent.com/subscribe