Rancher

Cluster API: Declarative Cluster Lifecycle — Rancher’s Foundation Layer

by
Reading Time: 5 minutes

Kubernetes Ecosystem: From User to Contributor, Episode 5
← EP04: Rancher · EP05: Cluster API · EP06: Crossplane →

11 min read

TL;DR

  • Cluster API (CAPI) declares Kubernetes clusters themselves — not just workloads running inside them — as Kubernetes objects: Cluster, Machine, MachineDeployment, reconciled by controllers the same way a Deployment reconciles pods
  • CAPI itself is infrastructure-agnostic — the actual provisioning logic lives in separate infrastructure providers (AWS, Azure, GCP, vSphere, and dozens more), each implementing the same core contract
  • Bootstrapping is genuinely awkward by necessity: you need a Kubernetes cluster to run CAPI’s controllers before CAPI can create your real cluster — solved by a temporary “kind” cluster and a pivot step that moves CAPI’s own resources into the cluster it just created
  • Rancher’s own newer provisioning (EP04) increasingly builds on CAPI patterns rather than reinventing cluster lifecycle management from scratch
  • Provider version compatibility is a real, ongoing constraint — CAPI core and each infrastructure provider version independently, and not every combination is supported
  • Contribution opportunity: clusterctl move, the pivot operation, has well-documented fragility with resources it doesn’t natively understand — a concrete, scoped gap

The Big Picture

Cluster (the K8s object, not the K8s cluster itself)
  │
  ├── Represents: this Cluster SHOULD exist
  │
  ▼
MachineDeployment  ──── mirrors Deployment/ReplicaSet/Pod exactly ────┐
  │                                                                     │
  ▼                                                                     │
MachineSet                                                              │
  │                                                                     │
  ▼                                                                     │
Machine  ────────► Infrastructure Provider (AWS/Azure/GCP/vSphere/...)  │
  │                  actually creates the VM/instance                  │
  ▼                                                                     │
Bootstrap Provider (kubeadm, typically)                                │
  actually turns that VM into a working Kubernetes node ────────────────┘

Cluster API’s declarative cluster lifecycle model is the same reconciliation pattern Kubernetes already uses for workloads, applied one layer up: instead of a Deployment controller reconciling Pod objects into running containers, CAPI’s controllers reconcile Machine objects into running cloud instances that then join a cluster as nodes.

The Core Abstraction: Clusters and Machines as Kubernetes Objects

$ kubectl apply -f - <<EOF
apiVersion: cluster.x-k8s.io/v1beta1
kind: Cluster
metadata:
  name: prod-us-east
spec:
  clusterNetwork:
    pods:
      cidrBlocks: ["192.168.0.0/16"]
  infrastructureRef:
    apiVersion: infrastructure.cluster.x-k8s.io/v1beta2
    kind: AWSCluster
    name: prod-us-east
EOF

$ kubectl get clusters
NAME           PHASE          AGE
prod-us-east   Provisioning   45s

$ kubectl get machines
NAME                     CLUSTER        PHASE         VERSION
prod-us-east-cp-x7k2l    prod-us-east   Provisioning  v1.28.5

The Cluster object is a declaration of intent, not the cluster itself — a management cluster (a separate, already-running Kubernetes cluster whose only job is to run CAPI’s controllers) watches these objects and does the actual work of calling out to AWS, Azure, or whatever provider is referenced, creating instances, and bootstrapping Kubernetes on them.

The Provider Model: How CAPI Stays Infrastructure-Agnostic

CAPI’s core (cluster-api) knows nothing about AWS, Azure, or any specific cloud. That knowledge lives in separate, independently-versioned infrastructure providers:

$ clusterctl init --infrastructure aws
Fetching providers
Installing cert-manager
Installing Provider="cluster-api" Version="v1.6.2"
Installing Provider="bootstrap-kubeadm" Version="v1.6.2"
Installing Provider="control-plane-kubeadm" Version="v1.6.2"
Installing Provider="infrastructure-aws" Version="v2.4.0"
#                                          ^^^^^^ — versioned independently
#                                          from core CAPI above

This split — core lifecycle logic separate from provider-specific implementation — is the same architectural pattern CNI and CSI use elsewhere in Kubernetes: a stable core contract, swappable implementations. It’s also exactly why CAPI’s ecosystem includes dozens of infrastructure providers (AWS, Azure, GCP, vSphere, OpenStack, Docker, bare metal, and many more) maintained by different teams at different paces.

A Management Cluster Managing Itself and Others: The Pivot

The genuinely awkward part of CAPI’s bootstrap story: you need a running Kubernetes cluster to host CAPI’s controllers before CAPI can create your first real cluster. The standard pattern:

# Step 1: spin up a throwaway local cluster just to run CAPI controllers
$ kind create cluster --name capi-bootstrap
$ clusterctl init --infrastructure aws

# Step 2: use that temporary management cluster to provision the REAL cluster
$ clusterctl generate cluster prod-us-east --infrastructure aws | kubectl apply -f -

# Step 3: move CAPI's own resources OFF the throwaway cluster and ONTO
# the cluster that was just created — "pivoting" management to itself
$ clusterctl move --to-kubeconfig=./prod-us-east.kubeconfig
Performing move...
Discovering Cluster API objects
Moving Cluster API objects: Clusters=1, Machines=3, ...

After the pivot, the cluster CAPI created is now managing its own lifecycle (and can go on to manage other clusters too) — the temporary kind cluster can be torn down. This bootstrap-then-pivot dance is elegant in theory and one of the more fragile operational moments in CAPI’s lifecycle in practice.

How Rancher and Others Build On CAPI

Rancher’s newer cluster provisioning (EP04) increasingly leans on CAPI patterns rather than maintaining entirely separate provisioning logic — the industry direction across the Kubernetes ecosystem has been toward CAPI as the shared substrate for “declare a cluster, get a cluster,” with vendors building their own UX and opinionated defaults on top rather than reinventing the reconciliation model itself.

⚠ Production Gotchas

Provider version compatibility is a real support matrix, not a “probably fine” assumption. Core CAPI and each infrastructure provider version independently — upgrading one without checking the compatibility matrix for the other is a common source of cryptic reconciliation failures.

clusterctl move is a rare, high-stakes operation — most teams run it once per cluster’s lifetime, if ever, which means nobody on the team has recent hands-on experience when something goes wrong. Test the pivot in a non-production scenario before relying on it for anything real.

A Machine stuck in Provisioning can mean the infrastructure provider, the bootstrap provider, or the actual cloud API — three different places to look, and the Machine object’s own status doesn’t always make it obvious which. Check the infrastructure-specific object (AWSMachine, AzureMachine, etc.) directly, not just the generic Machine.

Quick Reference

clusterctl init --infrastructure <provider>   # install CAPI + a provider on the management cluster
clusterctl generate cluster <name> --infrastructure <provider>   # generate cluster manifests
kubectl get clusters                           # cluster lifecycle phase
kubectl get machines                           # per-node provisioning phase
kubectl get awsmachines / azuremachines / ...   # provider-specific detail
clusterctl move --to-kubeconfig=<path>          # pivot management to another cluster
clusterctl describe cluster <name>              # human-readable status tree

Contribution Opportunity: clusterctl move‘s Fragility With Non-Native Resources

The limitation: clusterctl move knows how to move CAPI’s own well-known resource types between management clusters cleanly. When a provider or an operator has added custom resources that reference or extend CAPI objects — a common real-world pattern — move doesn’t always understand the relationship, and teams have reported needing manual intervention (patching, reapplying, or reordering) to get a full pivot to succeed cleanly. This is documented in multiple open issues against the project, not a rare edge case.

Why it’s hard to fix: move‘s core logic has to correctly identify and preserve object references and ownership across an arbitrary graph of custom resources it wasn’t necessarily designed to know about — building a fully general solution risks either false confidence (silently missing a reference) or false failure (over-cautiously blocking a move that would have been fine). The CAPI maintainers have to weigh correctness against usability here, and it’s a genuinely hard design problem, not a simple bug.

What a contribution-shaped fix looks like: Two realistic, scoped starting points: (1) a --dry-run-style pre-flight checker for clusterctl move that specifically scans for custom resources referencing CAPI objects and flags them before the move attempt, rather than discovering the gap mid-operation; or (2) contributing a documented, tested procedure (and ideally a small helper tool) for the specific pattern of “extra resources referencing Machine/Cluster objects” that’s already been reported in the project’s issue tracker — turning a known, recurring support question into a documented, repeatable procedure.

Key Takeaways

  • CAPI applies Kubernetes’ own reconciliation pattern one layer up — Cluster and Machine objects are declarations, reconciled into real infrastructure by provider-specific controllers
  • The core/provider split keeps CAPI infrastructure-agnostic, at the cost of independent versioning you have to track across a real compatibility matrix
  • The bootstrap-then-pivot pattern is CAPI’s most elegant and most operationally fragile moment — rehearse it before you need it for real
  • Rancher and other platform tools increasingly build their own provisioning UX on top of CAPI’s reconciliation model rather than replacing it
  • The clearest contribution opportunity is clusterctl move‘s handling of non-native custom resources — a documented, scoped gap with real prior art in the issue tracker

What’s Next

CAPI treats infrastructure — VMs, networks, load balancers — as the thing being reconciled into existence from Kubernetes objects. EP06 takes that same idea and generalizes it as far as it can go: Crossplane turns Kubernetes into a control plane for effectively any cloud resource, not just the ones needed to run Kubernetes itself.

Next: EP06 — Crossplane: Kubernetes as the Universal Control Plane

Get EP06 in your inbox when it publishes → linuxcent.com/subscribe

Rancher: Multi-Cluster Kubernetes Management at Scale

by
Reading Time: 5 minutes

Kubernetes Ecosystem: From User to Contributor, Episode 4
← EP03: k3s vs MicroK8s vs Minikube · EP04: Rancher · EP05: Cluster API →

11 min read

TL;DR

  • Rancher multi-cluster management means one Rancher server managing many downstream Kubernetes clusters — its own RKE2/k3s clusters, or imported EKS/GKE/AKS clusters — from a single pane of glass
  • Rancher doesn’t proxy every API call through itself; it deploys a lightweight agent into each downstream cluster that phones home, then aggregates each cluster’s API through that agent
  • Fleet, Rancher’s built-in GitOps engine, is what actually pushes manifests to potentially hundreds of clusters from a single git repository — this is the feature that makes “fleet” in the product’s marketing literal, not aspirational
  • Rancher’s Projects group namespaces within one cluster for permission management — they are not a cross-cluster grouping, a common misunderstanding
  • The Rancher server itself becomes something you now have to operate: HA, upgrades, and version compatibility with every downstream cluster’s Kubernetes version are real, ongoing operational work
  • Contribution opportunity: Fleet’s multi-cluster drift visibility has real, specific gaps — covered below

The Big Picture

                    ┌─────────────────────────┐
                    │      RANCHER SERVER       │
                    │  (itself a K8s cluster,   │
                    │   ideally HA, 3+ nodes)   │
                    └────────────┬────────────┘
                                 │ agents phone home,
                                 │ API aggregated back
              ┌──────────────────┼──────────────────┐
              │                  │                  │
        ┌─────▼─────┐     ┌──────▼──────┐    ┌──────▼──────┐
        │  RKE2      │     │  Imported    │    │  Imported    │
        │  cluster   │     │  EKS cluster │    │  GKE cluster │
        │ (Rancher-  │     │ (Rancher     │    │ (Rancher     │
        │  provisioned)│    │  didn't      │    │  didn't      │
        │            │     │  create it)  │    │  create it)  │
        └────────────┘     └─────────────┘    └─────────────┘

Rancher multi-cluster management works by inverting the connection direction most people assume: Rancher doesn’t reach out and control downstream clusters directly. Each downstream cluster runs a small agent that establishes an outbound connection back to the Rancher server — which is why Rancher can manage a cluster sitting behind NAT or a restrictive firewall, as long as that cluster can reach out.

How Rancher Actually Manages Clusters It Didn’t Create

# Import an existing cluster Rancher never touched at creation time
$ kubectl apply -f https://rancher.example.com/v3/import/<token>.yaml
# This installs the cattle-cluster-agent into the target cluster —
# that agent is the only thing Rancher needs to start managing it

$ kubectl get pods -n cattle-system
NAME                                    READY   STATUS    RESTARTS
cattle-cluster-agent-7d8f9c-x2k9l       1/1     Running   0

Once the agent is running, Rancher’s UI and API present that cluster’s resources as if you were talking to it directly — the agent maintains the tunnel and relays API calls both ways. This is the architectural reason Rancher can manage a genuinely heterogeneous fleet: RKE2, k3s, EKS, GKE, AKS, and on-prem clusters all look identical to Rancher once the same agent is running in each.

RKE2 and k3s: Rancher’s Own Cluster Distributions

Rancher can also provision brand-new clusters directly, using its own distributions:

# Provisioning a new downstream cluster via Rancher's cluster API
# (typically done through the UI, but expressible as a CR)
$ kubectl apply -f - <<EOF
apiVersion: provisioning.cattle.io/v1
kind: Cluster
metadata:
  name: edge-fleet-01
  namespace: fleet-default
spec:
  kubernetesVersion: v1.28.9+rke2r1
  rkeConfig:
    machinePools:
    - name: pool-01
      quantity: 3
EOF

RKE2 (“RKE Government,” a CIS-hardened, more security-focused distribution) and k3s (the lightweight distribution covered in EP03) are both Rancher/SUSE projects, and Rancher treats them as first-class provisioning targets — this is the direct product connection between “the lightweight Kubernetes distro you picked in EP03” and “the fleet manager covered in this episode.”

Fleet: GitOps at Fleet Scale

# Fleet watches a git repo and deploys its manifests to a TARGETED
# set of clusters based on label selectors — not necessarily all of them
$ kubectl apply -f - <<EOF
apiVersion: fleet.cattle.io/v1alpha1
kind: GitRepo
metadata:
  name: platform-baseline
  namespace: fleet-default
spec:
  repo: https://github.com/example-org/platform-manifests
  branch: main
  targets:
  - clusterSelector:
      matchLabels:
        env: production
EOF

$ kubectl get gitrepo -n fleet-default
NAME                REPO                                          COMMIT     BUNDLESREADY
platform-baseline   https://github.com/example-org/platform-...   a1b2c3d    12/14
#                                                                              ^^^^^ — 2 clusters
#                                                                              haven't converged yet

BUNDLESREADY 12/14 is the number that matters at fleet scale — it tells you how many of the targeted clusters have actually converged to the git state, but notice it doesn’t tell you why the other 2 haven’t, or which 2 they are, without drilling into each bundle individually. That’s the exact gap covered in this episode’s contribution section.

Projects and RBAC: Rancher’s Multi-Tenancy Layer

A common misconception worth correcting directly: Rancher’s Projects group namespaces within a single cluster for permission and resource-quota management — they are not a mechanism for grouping resources across clusters. Cross-cluster access control is handled separately, through Cluster-level and Global roles assigned per user or group.

Global scope        → applies across every cluster Rancher manages
  └── Cluster scope  → applies to all namespaces in one specific cluster
        └── Project scope → applies to a defined subset of namespaces
              within that one cluster (Rancher's own grouping construct)

Getting this hierarchy backwards — assuming a Project spans clusters — is one of the most common Rancher RBAC mistakes teams make when first designing their permission model.

⚠ Production Gotchas

Rancher server itself needs HA, and losing it doesn’t take down downstream clusters — but it does take down your ability to manage them centrally. Downstream clusters keep running their workloads fine if Rancher server is unreachable; you just lose the single-pane-of-glass view and Fleet’s GitOps reconciliation until it’s back.

Version skew between Rancher server and downstream Kubernetes versions is a real, documented compatibility matrix — not a “should mostly work” situation. Upgrading Rancher server ahead of your downstream clusters’ Kubernetes versions (or vice versa, letting downstream clusters drift too far ahead) can break agent compatibility. Check Rancher’s official support matrix before any upgrade, not after something breaks.

Agent reconnection storms after a Rancher server upgrade or restart are a known operational event, not a bug report. If you manage dozens of downstream clusters, expect a burst of reconnection activity immediately after any Rancher server maintenance — plan maintenance windows with that in mind.

Quick Reference

kubectl apply -f import.yaml              # import an existing cluster
kubectl get clusters.provisioning.cattle.io -A   # all clusters Rancher manages
kubectl get gitrepo -n fleet-default       # Fleet GitOps repo status
kubectl get bundles -n fleet-default       # per-cluster deployment bundle status
kubectl get pods -n cattle-system          # agent health, on a downstream cluster

Contribution Opportunity: Fleet’s Multi-Cluster Drift Visibility

The limitation: Fleet’s BUNDLESREADY count tells you how many targeted clusters have converged, but drilling into why a specific cluster hasn’t — a stuck rollout, a resource conflict, a cluster that’s unreachable — still requires checking that cluster’s bundle status individually. At a fleet of dozens or hundreds of clusters, there’s no aggregated view that surfaces “these 3 clusters are all failing for the same underlying reason” without manual cross-referencing.

Why it’s hard to fix: Aggregating meaningful failure reasons across a heterogeneous fleet is genuinely harder than it sounds — a “failed” bundle on one cluster might be a transient network blip, on another a real manifest conflict, and on a third a resource quota limit. Building a dashboard that correctly buckets and summarizes those different failure classes without producing a wall of noise is a real UX and data-modeling problem, and it’s not the kind of thing that gets prioritized ahead of core provisioning reliability work.

What a contribution-shaped fix looks like: A scoped, achievable starting point: a fleet CLI plugin or a Rancher UI extension that queries all Bundle resources across the fleet’s clusters, groups them by failure-reason similarity (using the existing status conditions Fleet already populates — this is a client-side aggregation problem, not a new backend feature), and surfaces a ranked summary. This is buildable against Fleet’s existing CRDs and status fields without needing to modify Fleet’s core reconciliation logic — exactly the kind of contribution an operator who’s felt this specific pain at scale is positioned to build and upstream.

Key Takeaways

  • Rancher manages downstream clusters through an outbound-connecting agent, not by reaching in — this is why it can manage clusters behind NAT or restrictive firewalls
  • Fleet is the actual mechanism for GitOps at fleet scale, targeting clusters by label selector and reporting convergence via BUNDLESREADY counts
  • Projects group namespaces within one cluster, not across clusters — a frequent RBAC design mistake starts from getting this backwards
  • The Rancher server becomes real infrastructure you operate: HA, version-compatibility matrices, and post-upgrade agent reconnection are ongoing operational realities
  • The clearest contribution opportunity is Fleet’s drift-visibility gap at scale — a client-side aggregation problem buildable against existing CRDs, not a core-logic change

What’s Next

Rancher’s own cluster provisioning sits on top of a more general pattern: declaring cluster lifecycle as Kubernetes resources. EP05 covers Cluster API directly — the CNCF project Rancher’s own provisioning increasingly builds on, and the pattern several other tools in this series also depend on.

Next: EP05 — Cluster API: Declarative Cluster Lifecycle — Rancher’s Foundation Layer

Get EP05 in your inbox when it publishes → linuxcent.com/subscribe