Kubernetes Ecosystem: From User to Contributor, Episode 4
← EP03: k3s vs MicroK8s vs Minikube · EP04: Rancher · EP05: Cluster API →
11 min read
TL;DR
- Rancher multi-cluster management means one Rancher server managing many downstream Kubernetes clusters — its own RKE2/k3s clusters, or imported EKS/GKE/AKS clusters — from a single pane of glass
- Rancher doesn’t proxy every API call through itself; it deploys a lightweight agent into each downstream cluster that phones home, then aggregates each cluster’s API through that agent
- Fleet, Rancher’s built-in GitOps engine, is what actually pushes manifests to potentially hundreds of clusters from a single git repository — this is the feature that makes “fleet” in the product’s marketing literal, not aspirational
- Rancher’s Projects group namespaces within one cluster for permission management — they are not a cross-cluster grouping, a common misunderstanding
- The Rancher server itself becomes something you now have to operate: HA, upgrades, and version compatibility with every downstream cluster’s Kubernetes version are real, ongoing operational work
- Contribution opportunity: Fleet’s multi-cluster drift visibility has real, specific gaps — covered below
The Big Picture
┌─────────────────────────┐
│ RANCHER SERVER │
│ (itself a K8s cluster, │
│ ideally HA, 3+ nodes) │
└────────────┬────────────┘
│ agents phone home,
│ API aggregated back
┌──────────────────┼──────────────────┐
│ │ │
┌─────▼─────┐ ┌──────▼──────┐ ┌──────▼──────┐
│ RKE2 │ │ Imported │ │ Imported │
│ cluster │ │ EKS cluster │ │ GKE cluster │
│ (Rancher- │ │ (Rancher │ │ (Rancher │
│ provisioned)│ │ didn't │ │ didn't │
│ │ │ create it) │ │ create it) │
└────────────┘ └─────────────┘ └─────────────┘
Rancher multi-cluster management works by inverting the connection direction most people assume: Rancher doesn’t reach out and control downstream clusters directly. Each downstream cluster runs a small agent that establishes an outbound connection back to the Rancher server — which is why Rancher can manage a cluster sitting behind NAT or a restrictive firewall, as long as that cluster can reach out.
How Rancher Actually Manages Clusters It Didn’t Create
# Import an existing cluster Rancher never touched at creation time
$ kubectl apply -f https://rancher.example.com/v3/import/<token>.yaml
# This installs the cattle-cluster-agent into the target cluster —
# that agent is the only thing Rancher needs to start managing it
$ kubectl get pods -n cattle-system
NAME READY STATUS RESTARTS
cattle-cluster-agent-7d8f9c-x2k9l 1/1 Running 0
Once the agent is running, Rancher’s UI and API present that cluster’s resources as if you were talking to it directly — the agent maintains the tunnel and relays API calls both ways. This is the architectural reason Rancher can manage a genuinely heterogeneous fleet: RKE2, k3s, EKS, GKE, AKS, and on-prem clusters all look identical to Rancher once the same agent is running in each.
RKE2 and k3s: Rancher’s Own Cluster Distributions
Rancher can also provision brand-new clusters directly, using its own distributions:
# Provisioning a new downstream cluster via Rancher's cluster API
# (typically done through the UI, but expressible as a CR)
$ kubectl apply -f - <<EOF
apiVersion: provisioning.cattle.io/v1
kind: Cluster
metadata:
name: edge-fleet-01
namespace: fleet-default
spec:
kubernetesVersion: v1.28.9+rke2r1
rkeConfig:
machinePools:
- name: pool-01
quantity: 3
EOF
RKE2 (“RKE Government,” a CIS-hardened, more security-focused distribution) and k3s (the lightweight distribution covered in EP03) are both Rancher/SUSE projects, and Rancher treats them as first-class provisioning targets — this is the direct product connection between “the lightweight Kubernetes distro you picked in EP03” and “the fleet manager covered in this episode.”
Fleet: GitOps at Fleet Scale
# Fleet watches a git repo and deploys its manifests to a TARGETED
# set of clusters based on label selectors — not necessarily all of them
$ kubectl apply -f - <<EOF
apiVersion: fleet.cattle.io/v1alpha1
kind: GitRepo
metadata:
name: platform-baseline
namespace: fleet-default
spec:
repo: https://github.com/example-org/platform-manifests
branch: main
targets:
- clusterSelector:
matchLabels:
env: production
EOF
$ kubectl get gitrepo -n fleet-default
NAME REPO COMMIT BUNDLESREADY
platform-baseline https://github.com/example-org/platform-... a1b2c3d 12/14
# ^^^^^ — 2 clusters
# haven't converged yet
BUNDLESREADY 12/14 is the number that matters at fleet scale — it tells you how many of the targeted clusters have actually converged to the git state, but notice it doesn’t tell you why the other 2 haven’t, or which 2 they are, without drilling into each bundle individually. That’s the exact gap covered in this episode’s contribution section.
Projects and RBAC: Rancher’s Multi-Tenancy Layer
A common misconception worth correcting directly: Rancher’s Projects group namespaces within a single cluster for permission and resource-quota management — they are not a mechanism for grouping resources across clusters. Cross-cluster access control is handled separately, through Cluster-level and Global roles assigned per user or group.
Global scope → applies across every cluster Rancher manages
└── Cluster scope → applies to all namespaces in one specific cluster
└── Project scope → applies to a defined subset of namespaces
within that one cluster (Rancher's own grouping construct)
Getting this hierarchy backwards — assuming a Project spans clusters — is one of the most common Rancher RBAC mistakes teams make when first designing their permission model.
⚠ Production Gotchas
Rancher server itself needs HA, and losing it doesn’t take down downstream clusters — but it does take down your ability to manage them centrally. Downstream clusters keep running their workloads fine if Rancher server is unreachable; you just lose the single-pane-of-glass view and Fleet’s GitOps reconciliation until it’s back.
Version skew between Rancher server and downstream Kubernetes versions is a real, documented compatibility matrix — not a “should mostly work” situation. Upgrading Rancher server ahead of your downstream clusters’ Kubernetes versions (or vice versa, letting downstream clusters drift too far ahead) can break agent compatibility. Check Rancher’s official support matrix before any upgrade, not after something breaks.
Agent reconnection storms after a Rancher server upgrade or restart are a known operational event, not a bug report. If you manage dozens of downstream clusters, expect a burst of reconnection activity immediately after any Rancher server maintenance — plan maintenance windows with that in mind.
Quick Reference
kubectl apply -f import.yaml # import an existing cluster
kubectl get clusters.provisioning.cattle.io -A # all clusters Rancher manages
kubectl get gitrepo -n fleet-default # Fleet GitOps repo status
kubectl get bundles -n fleet-default # per-cluster deployment bundle status
kubectl get pods -n cattle-system # agent health, on a downstream cluster
Contribution Opportunity: Fleet’s Multi-Cluster Drift Visibility
The limitation: Fleet’s BUNDLESREADY count tells you how many targeted clusters have converged, but drilling into why a specific cluster hasn’t — a stuck rollout, a resource conflict, a cluster that’s unreachable — still requires checking that cluster’s bundle status individually. At a fleet of dozens or hundreds of clusters, there’s no aggregated view that surfaces “these 3 clusters are all failing for the same underlying reason” without manual cross-referencing.
Why it’s hard to fix: Aggregating meaningful failure reasons across a heterogeneous fleet is genuinely harder than it sounds — a “failed” bundle on one cluster might be a transient network blip, on another a real manifest conflict, and on a third a resource quota limit. Building a dashboard that correctly buckets and summarizes those different failure classes without producing a wall of noise is a real UX and data-modeling problem, and it’s not the kind of thing that gets prioritized ahead of core provisioning reliability work.
What a contribution-shaped fix looks like: A scoped, achievable starting point: a fleet CLI plugin or a Rancher UI extension that queries all Bundle resources across the fleet’s clusters, groups them by failure-reason similarity (using the existing status conditions Fleet already populates — this is a client-side aggregation problem, not a new backend feature), and surfaces a ranked summary. This is buildable against Fleet’s existing CRDs and status fields without needing to modify Fleet’s core reconciliation logic — exactly the kind of contribution an operator who’s felt this specific pain at scale is positioned to build and upstream.
Key Takeaways
- Rancher manages downstream clusters through an outbound-connecting agent, not by reaching in — this is why it can manage clusters behind NAT or restrictive firewalls
- Fleet is the actual mechanism for GitOps at fleet scale, targeting clusters by label selector and reporting convergence via
BUNDLESREADYcounts - Projects group namespaces within one cluster, not across clusters — a frequent RBAC design mistake starts from getting this backwards
- The Rancher server becomes real infrastructure you operate: HA, version-compatibility matrices, and post-upgrade agent reconnection are ongoing operational realities
- The clearest contribution opportunity is Fleet’s drift-visibility gap at scale — a client-side aggregation problem buildable against existing CRDs, not a core-logic change
What’s Next
Rancher’s own cluster provisioning sits on top of a more general pattern: declaring cluster lifecycle as Kubernetes resources. EP05 covers Cluster API directly — the CNCF project Rancher’s own provisioning increasingly builds on, and the pattern several other tools in this series also depend on.
Next: EP05 — Cluster API: Declarative Cluster Lifecycle — Rancher’s Foundation Layer
Get EP05 in your inbox when it publishes → linuxcent.com/subscribe