Kubernetes Ecosystem: From User to Contributor, Episode 10 (Series Finale)
← EP09: Karpenter vs Cluster Autoscaler · EP10: KubeVirt · All Kubernetes Ecosystem Episodes →

11 min read

TL;DR

• KubeVirt runs full virtual machines as first-class Kubernetes objects — a VirtualMachineInstance wraps a real QEMU/KVM VM inside a special pod, scheduled and managed the same way any other workload is
• The real reason to run KubeVirt isn’t nostalgia for VMs — it’s giving legacy, non-containerizable, or Windows workloads the same control plane, networking, and operational tooling as everything else, instead of maintaining a separate VM infrastructure stack alongside Kubernetes
• Multus (multiple network interfaces per pod) and CDI (Containerized Data Importer, for getting disk images into the cluster) are the supporting pieces that make VM networking and storage work inside a Kubernetes-native model
• Live migration — moving a running VM to another node with no downtime — is KubeVirt’s answer to node maintenance, and it’s meaningfully different from how Kubernetes handles pod disruption
• Nodes running VM workloads need actual hardware virtualization support (KVM) — the same nested-virtualization requirement that constrained Minikube’s VM drivers back in EP02, now showing up at the production node level
• Contribution opportunity: GPU and device-plugin support for VM workloads specifically lags behind the maturity of the container device-plugin ecosystem — a real, scoped gap

The Big Picture

CONTAINER POD                                VM VIA KUBEVIRT
──────────────                                ───────────────
Pod                                          VirtualMachineInstance
  └── container                                └── virt-launcher Pod
        runs directly on the                        └── QEMU/KVM process
        node's kernel via                                running a REAL VM,
        containerd/runc                                    with its own kernel
                                                             (Linux, Windows,
Scheduling, networking,                                     anything QEMU
resource limits — standard                                  supports)
Kubernetes primitives
                                              Scheduling, networking, resource
                                              limits — SAME standard Kubernetes
                                              primitives, wrapping a VM instead

What is KubeVirt? It’s the project that makes a virtual machine look like just another workload to Kubernetes — the VM runs inside a virt-launcher pod, so everything Kubernetes already does for scheduling, networking policy, and resource accounting applies to the VM too, without Kubernetes itself needing to understand virtualization at all.

The Core Abstraction: VirtualMachine, VirtualMachineInstance, and virt-launcher

apiVersion: kubevirt.io/v1
kind: VirtualMachine
metadata:
  name: legacy-app-vm
spec:
  running: true
  template:
    spec:
      domain:
        cpu: {cores: 2}
        memory: {guest: 4Gi}
        devices:
          disks:
          - name: rootdisk
            disk: {bus: virtio}
      volumes:
      - name: rootdisk
        dataVolume: {name: legacy-app-disk}

$ kubectl get vmi
NAME            AGE   PHASE     IP            NODENAME
legacy-app-vm   2m    Running   10.244.1.15   worker-03

$ kubectl get pods
NAME                                READY   STATUS
virt-launcher-legacy-app-vm-x7k2l   2/2     Running
#     ^^^^^^^^^^^^ — this IS the pod hosting the real QEMU process

VirtualMachine is the persistent declaration (should this VM exist, running or stopped); VirtualMachineInstance is the actual running instance; virt-launcher is the pod — an ordinary Kubernetes pod from the scheduler’s point of view — that contains the QEMU process actually executing the VM. This layering mirrors Deployment/ReplicaSet/Pod deliberately, the same pattern CAPI (EP05) uses for Cluster/Machine.

Why Run VMs on Kubernetes At All

The honest case for KubeVirt isn’t “VMs are better than containers” — it’s consolidation. Most real infrastructure estates have some workloads that genuinely can’t be containerized cleanly: legacy applications tightly coupled to a specific OS version, Windows-only software, or workloads with licensing/compliance requirements demanding a VM-level boundary. Without KubeVirt, those workloads need an entirely separate VM infrastructure stack — its own hypervisor management, its own networking, its own monitoring — running alongside Kubernetes rather than on it. KubeVirt collapses that into one control plane, one set of operational tooling, one team’s expertise, for both categories of workload.

Networking and Storage for VMs: Multus, DataVolumes, CDI

# Multus lets a VM's pod attach to more than one network —
# often needed for VMs expecting a dedicated network interface,
# unlike a typical single-network container workload
$ kubectl get network-attachment-definitions

# CDI (Containerized Data Importer) gets an existing VM disk image
# into the cluster as a DataVolume
$ kubectl apply -f - <<EOF
apiVersion: cdi.kubevirt.io/v1beta1
kind: DataVolume
metadata:
  name: legacy-app-disk
spec:
  source:
    http:
      url: "https://images.example.com/legacy-app.qcow2"
  pvc:
    accessModes: ["ReadWriteOnce"]
    resources:
      requests: {storage: 20Gi}
EOF

Regular container images assume a filesystem layer model that doesn’t map cleanly onto “import this existing 20GB qcow2 disk image a legacy app already runs from.” CDI exists specifically to bridge that gap — pulling real VM disk images from HTTP, S3, or a registry into a DataVolume a VirtualMachine can boot from.

Live Migration: KubeVirt’s Answer to Node Maintenance

$ virtctl migrate legacy-app-vm
VM legacy-app-vm was scheduled to migrate

$ kubectl get vmim
NAME                    PHASE       VMI
legacy-app-vm-migrate   Running     legacy-app-vm

# The VM keeps running, on a different node, with no restart —
# genuinely different from how Kubernetes handles pod disruption

A pod being evicted for node maintenance simply gets rescheduled — it restarts fresh somewhere else. A live-migrated VM keeps running throughout, its in-memory state transferred to the new node without a restart, the same live-migration capability traditional hypervisor platforms have offered for years. This is one behavior category where “VM on Kubernetes” is genuinely doing something a plain pod cannot.

Production Gotchas

Nodes need real KVM support — nested virtualization if the node itself is already a VM (common on cloud instances). This is the same underlying requirement that constrained Minikube’s VM drivers back in EP02, now at the production node level: if your cloud instance type doesn’t expose nested virtualization, KubeVirt workloads simply won’t schedule there.

VM disk I/O performance depends heavily on the underlying storage class and driver (virtio vs alternatives) — don’t assume container-workload storage benchmarks transfer to VM workloads on the same cluster.

Live migration isn’t guaranteed to succeed for every VM under every condition — a VM under heavy memory-write load, or a migration across a congested network path, can fail or take far longer than expected. Test migration behavior under realistic load before depending on it for a real maintenance window.

Windows VMs specifically need the right virtio drivers pre-installed in the image, or networking/storage won’t work post-boot — this is a common first-VM stumbling block distinct from anything Linux-VM users hit.

Quick Reference

kubectl get vm                       # VirtualMachine declarations
kubectl get vmi                      # running VirtualMachineInstances
kubectl get vmim                     # migration status
virtctl start / stop / restart <vm>  # lifecycle control
virtctl console <vm>                 # serial console access
virtctl migrate <vm>                 # trigger a live migration
kubectl get datavolumes              # CDI-imported disk images

Contribution Opportunity: Device-Plugin Parity for VM Workloads

The limitation: Kubernetes’ device-plugin ecosystem for GPUs and other specialized hardware is mature and well-supported for container workloads. Extending that same hardware access cleanly into KubeVirt-managed VMs — GPU passthrough with the same operational ergonomics containers already have — is real, current work with meaningfully more rough edges than the container-side equivalent.

Why it’s hard to fix: Passing a physical device through to a VM (as opposed to a container, which can often share the host’s device more directly) involves IOMMU configuration, PCI passthrough mechanics, and driver considerations inside the guest OS that simply don’t exist for container device access — it’s a genuinely harder problem, not a lagging implementation of an equally-easy one.

What a contribution-shaped fix looks like: KubeVirt’s own documentation and issue tracker identify specific, scoped GPU/device-passthrough gaps — particular hardware combinations without a documented working configuration, or specific device-plugin integrations that work for containers but haven’t been extended to the VM path. Reproducing one of these documented gaps on real hardware you have access to, and contributing either the fix or a verified, tested configuration guide back to the project, is exactly the kind of practitioner-shaped contribution this series has pointed toward in every episode — you need access to the hardware and patience to reproduce, not novel research.

Key Takeaways

• KubeVirt makes a VM look like an ordinary Kubernetes workload by wrapping it in a virt-launcher pod — the same scheduling, networking, and resource primitives apply to both
• The real case for running VMs on Kubernetes is infrastructure consolidation — one control plane for both container and VM workloads, not VM enthusiasm for its own sake
• Multus and CDI solve the networking and storage problems that don’t map cleanly from the container model onto VM disk images and multi-interface networking
• Live migration is a genuine capability gap between VMs and plain pods — Kubernetes’ own disruption model for pods has no equivalent
• The clearest current contribution opportunity is closing specific, documented device-passthrough gaps for VM workloads — hardware-access work, not abstract feature design

Series Wrap: From User to Contributor

Ten episodes, ten tools, and one repeated pattern: every project in this series exists because something else fell short — k3s and MicroK8s because upstream Kubernetes was too heavy for the edge, Karpenter because Cluster Autoscaler’s node-group abstraction had a ceiling, Crossplane because Terraform’s model didn’t fit a fully Kubernetes-native platform team. Each one of those gaps was found by someone who was using the previous tool closely enough to feel exactly where it stopped working.

That’s the actual path from user to contributor this series has been pointing at in every Contribution Opportunity section: not a certification, not a novel research breakthrough — close, careful attention to a tool you already run, precise enough to name what’s actually missing and specific enough to describe what fixing it would look like. Every gap named across these ten episodes is real, documented, and waiting for exactly that kind of attention.

Get future series and episodes in your inbox → linuxcent.com/subscribe

The post KubeVirt: Running VMs on Kubernetes — and Why That Still Matters appeared first on Linuxcent.

Kubernetes Ecosystem: From User to Contributor, Episode 6
← EP05: Cluster API · EP06: Crossplane · EP07: Crossplane vs Terraform →

12 min read

TL;DR

• Crossplane extends the exact reconciliation pattern EP05 covered for cluster infrastructure to any cloud resource — an S3 bucket, an RDS instance, a DNS record all become Kubernetes CRDs, continuously reconciled
• Managed Resources represent one real cloud resource each; Compositions bundle several Managed Resources behind a single, simpler custom API a platform team defines and app teams consume
• Composition Functions are Crossplane’s newer, more flexible replacement for its older YAML-based patch-and-transform templating — real code (Go, Python, or others) instead of declarative patches
• Crossplane continuously reconciles like any Kubernetes controller — a manual change to a cloud resource outside Crossplane gets reverted on the next reconcile loop, which is a real surprise for teams used to Terraform’s plan/apply model
• Provider CRD counts can bloat a cluster’s etcd significantly — this drove the ecosystem’s move toward smaller, split “provider families” instead of one monolithic provider per cloud
• Contribution opportunity: several providers still haven’t migrated to the family-split pattern — a real, currently-tracked, achievable upstream contribution

The Big Picture

App team writes:                    Platform team defined this Composition
                                     once, behind the scenes:
apiVersion: platform.example.com/v1
kind: Database                       XRD "Database" ─── composes ───┐
metadata:                                                             │
  name: my-app-db                                                     ▼
spec:                                                        ┌────────────────┐
  size: small                                                │ RDSInstance    │
                                                                │ SecurityGroup  │
   │                                                            │ ParameterGroup │
   │ app team never sees                                        └────────────────┘
   │ or touches these three                                     each a real Managed
   ▼                                                            Resource, a real
Crossplane reconciles all three,                                cloud API call
continuously, forever

Crossplane’s pitch as a universal control plane is literal: instead of app teams filing tickets or writing their own Terraform for a database, they request a Database — a custom API the platform team designed — and Crossplane’s controllers translate that into the actual RDS instance, security group, and parameter group underneath, then keep reconciling all three toward the declared state indefinitely.

Managed Resources: Cloud Infrastructure as Kubernetes CRDs

$ kubectl apply -f - <<EOF
apiVersion: s3.aws.upbound.io/v1beta1
kind: Bucket
metadata:
  name: app-uploads-prod
spec:
  forProvider:
    region: us-east-1
  providerConfigRef:
    name: aws-prod
EOF

$ kubectl get bucket app-uploads-prod
NAME               READY   SYNCED   AGE
app-uploads-prod   True    True     30s
#                  ^^^^    ^^^^^^ — READY: resource exists and is healthy
#                          SYNCED: Crossplane's last reconcile succeeded

Every field under forProvider maps directly to that cloud API’s actual parameters — this is a thin, honest translation layer, not an abstraction hiding what’s actually being created. READY/SYNCED becoming True means an actual S3 bucket now exists in that AWS account, exactly as declared.

Compositions and XRDs: Building Your Own Abstract Platform API

This is Crossplane’s real differentiator over just using individual Managed Resources directly:

# The platform team defines the abstract API app teams will see
apiVersion: apiextensions.crossplane.io/v1
kind: CompositeResourceDefinition
metadata:
  name: xdatabases.platform.example.com
spec:
  group: platform.example.com
  names:
    kind: XDatabase
    plural: xdatabases
  claimNames:
    kind: Database        # ← this is what app teams actually create
    plural: databases
  versions:
  - name: v1
    schema:
      openAPIV3Schema:
        properties:
          spec:
            properties:
              size: {type: string, enum: ["small", "medium", "large"]}

App teams interact only with the simple Database claim shown in the Big Picture diagram above. The Composition resource (not shown here for brevity) is what actually maps size: small to a specific RDS instance class, storage size, and backup configuration — the platform team’s opinions, encoded once, consumed self-service by every app team afterward.

Composition Functions: Crossplane’s Newer, More Flexible Approach

Older Crossplane Compositions used a YAML-based “patch and transform” templating language to map the abstract API’s fields onto Managed Resource fields — functional, but limited for anything beyond straightforward field mapping. Composition Functions replace that with actual executable code:

$ crossplane beta render xr.yaml composition.yaml functions.yaml
---
apiVersion: rds.aws.upbound.io/v1alpha1
kind: Instance
metadata:
  name: my-app-db-instance
spec:
  forProvider:
    instanceClass: db.t3.micro   # ← computed by real Go logic based on
                                  #   spec.size, not a static YAML patch
    engine: postgres

Composition Functions run as small, packaged pieces of logic (often distributed as OCI images) that Crossplane’s engine invokes during reconciliation — giving platform teams real conditionals, loops, and validation instead of the older templating language’s more limited patch syntax.

Providers and the Provider Ecosystem

Each cloud’s resources are supplied by a separate provider — provider-aws, provider-gcp, provider-azure, and increasingly split into smaller provider families (provider-aws-s3, provider-aws-rds, etc.) rather than one enormous provider per cloud:

$ kubectl get providers
NAME                   INSTALLED   HEALTHY   AGE
provider-aws-s3        True        True      10d
provider-aws-rds       True        True      10d
#         ^^^^^^ — installing only the families you actually use, instead
#                  of one monolithic provider-aws with every AWS service's
#                  CRDs installed regardless of whether you use them

The family split exists specifically because a single monolithic cloud provider can register thousands of CRDs — a real, measurable strain on a cluster’s etcd and API server that the ecosystem is still in the process of migrating away from.

Production Gotchas

Crossplane reconciles continuously — a manual change to a cloud resource outside Crossplane gets reverted on the next loop. Teams coming from Terraform’s plan/apply model, where nothing changes until you explicitly run apply again, are frequently surprised the first time a manual “quick fix” in the AWS console gets silently undone minutes later.

Monolithic providers can register thousands of CRDs, and that has a real, measurable etcd and API-server cost. If you’re on an older, non-family provider version and seeing API server memory pressure, check CRD count before assuming it’s an unrelated capacity issue.

Deleting a Composition’s underlying claim doesn’t always tear down cleanly if finalizers on the Managed Resources are stuck — a Managed Resource that failed to delete cleanly from the cloud side (a non-empty S3 bucket, for instance) will block the whole claim’s deletion until that’s resolved manually.

Quick Reference

kubectl get managed                        # every Managed Resource, all providers
kubectl get compositeresourcedefinitions   # XRDs — the abstract APIs defined
kubectl get compositions                   # the mapping logic behind each XRD
kubectl get providers                       # installed providers + health
crossplane beta render <xr> <comp> <fns>    # render a Composition locally, no cluster needed
kubectl describe <managed-resource-kind> <name>   # sync status + underlying cloud errors

Contribution Opportunity: Migrating Providers to the Family Pattern

The limitation: Not every Crossplane provider has migrated from the older, monolithic-per-cloud model to the smaller “provider family” pattern that registers only the CRDs for services actually in use. Clusters running an un-migrated provider carry the etcd and API-server overhead of thousands of unused CRDs, and this is a known, actively-discussed problem in the Crossplane community — not a hypothetical one.

Why it’s hard to fix: Splitting a monolithic provider into families isn’t a mechanical find-and-replace — it means restructuring code generation, versioning, and release processes for every resource type the provider covers, while keeping a migration path that doesn’t break existing users who depend on the old provider’s CRDs. It’s real, unglamorous engineering work that has to happen provider-by-provider, cloud-by-cloud, and each provider’s maintainer bandwidth varies.

What a contribution-shaped fix looks like: The Crossplane and Upbound-maintained provider repositories publicly track which providers still need family-splitting — this is documented, wanted work, not a gap you’d have to go discover yourself. A concrete starting contribution: pick one still-monolithic provider (checking the project’s own tracking issues for an unclaimed one), and work through the documented family-split process the already-migrated providers (like provider-aws) used as a reference implementation. This is real upstream OSS work with an existing template to follow, not a design problem you have to solve from scratch.

Key Takeaways

• Crossplane’s Managed Resources make individual cloud resources real Kubernetes CRDs, continuously reconciled rather than applied once
• Compositions and XRDs are the actual value proposition: platform teams define a simple, opinionated API once; app teams self-serve against it without needing to know what’s underneath
• Composition Functions replace older YAML patch-and-transform templating with real executable logic — a genuinely evolving, more flexible part of the project
• Continuous reconciliation means manual out-of-band changes get reverted — a real behavioral difference from Terraform’s plan/apply model, not just a implementation detail
• The provider family migration is documented, wanted, achievable contribution work — not a gap you’d need to discover on your own

What’s Next

Crossplane’s composition model and Terraform’s HCL module model solve the same underlying problem — reusable, parameterized infrastructure definitions — from genuinely different architectural starting points. EP07 puts them side by side and gives a clear recommendation for which fits which team.

Next: EP07 — Crossplane vs Terraform: Composition vs HCL for Infrastructure as Code

Get EP07 in your inbox when it publishes → linuxcent.com/subscribe

The post Crossplane: Kubernetes as the Universal Control Plane appeared first on Linuxcent.

Kubernetes Ecosystem: From User to Contributor, Episode 5
← EP04: Rancher · EP05: Cluster API · EP06: Crossplane →

11 min read

TL;DR

• Cluster API (CAPI) declares Kubernetes clusters themselves — not just workloads running inside them — as Kubernetes objects: Cluster, Machine, MachineDeployment, reconciled by controllers the same way a Deployment reconciles pods
• CAPI itself is infrastructure-agnostic — the actual provisioning logic lives in separate infrastructure providers (AWS, Azure, GCP, vSphere, and dozens more), each implementing the same core contract
• Bootstrapping is genuinely awkward by necessity: you need a Kubernetes cluster to run CAPI’s controllers before CAPI can create your real cluster — solved by a temporary “kind” cluster and a pivot step that moves CAPI’s own resources into the cluster it just created
• Rancher’s own newer provisioning (EP04) increasingly builds on CAPI patterns rather than reinventing cluster lifecycle management from scratch
• Provider version compatibility is a real, ongoing constraint — CAPI core and each infrastructure provider version independently, and not every combination is supported
• Contribution opportunity: clusterctl move, the pivot operation, has well-documented fragility with resources it doesn’t natively understand — a concrete, scoped gap

The Big Picture

Cluster (the K8s object, not the K8s cluster itself)
  │
  ├── Represents: this Cluster SHOULD exist
  │
  ▼
MachineDeployment  ──── mirrors Deployment/ReplicaSet/Pod exactly ────┐
  │                                                                     │
  ▼                                                                     │
MachineSet                                                              │
  │                                                                     │
  ▼                                                                     │
Machine  ────────► Infrastructure Provider (AWS/Azure/GCP/vSphere/...)  │
  │                  actually creates the VM/instance                  │
  ▼                                                                     │
Bootstrap Provider (kubeadm, typically)                                │
  actually turns that VM into a working Kubernetes node ────────────────┘

Cluster API’s declarative cluster lifecycle model is the same reconciliation pattern Kubernetes already uses for workloads, applied one layer up: instead of a Deployment controller reconciling Pod objects into running containers, CAPI’s controllers reconcile Machine objects into running cloud instances that then join a cluster as nodes.

The Core Abstraction: Clusters and Machines as Kubernetes Objects

$ kubectl apply -f - <<EOF
apiVersion: cluster.x-k8s.io/v1beta1
kind: Cluster
metadata:
  name: prod-us-east
spec:
  clusterNetwork:
    pods:
      cidrBlocks: ["192.168.0.0/16"]
  infrastructureRef:
    apiVersion: infrastructure.cluster.x-k8s.io/v1beta2
    kind: AWSCluster
    name: prod-us-east
EOF

$ kubectl get clusters
NAME           PHASE          AGE
prod-us-east   Provisioning   45s

$ kubectl get machines
NAME                     CLUSTER        PHASE         VERSION
prod-us-east-cp-x7k2l    prod-us-east   Provisioning  v1.28.5

The Cluster object is a declaration of intent, not the cluster itself — a management cluster (a separate, already-running Kubernetes cluster whose only job is to run CAPI’s controllers) watches these objects and does the actual work of calling out to AWS, Azure, or whatever provider is referenced, creating instances, and bootstrapping Kubernetes on them.

The Provider Model: How CAPI Stays Infrastructure-Agnostic

CAPI’s core (cluster-api) knows nothing about AWS, Azure, or any specific cloud. That knowledge lives in separate, independently-versioned infrastructure providers:

$ clusterctl init --infrastructure aws
Fetching providers
Installing cert-manager
Installing Provider="cluster-api" Version="v1.6.2"
Installing Provider="bootstrap-kubeadm" Version="v1.6.2"
Installing Provider="control-plane-kubeadm" Version="v1.6.2"
Installing Provider="infrastructure-aws" Version="v2.4.0"
#                                          ^^^^^^ — versioned independently
#                                          from core CAPI above

This split — core lifecycle logic separate from provider-specific implementation — is the same architectural pattern CNI and CSI use elsewhere in Kubernetes: a stable core contract, swappable implementations. It’s also exactly why CAPI’s ecosystem includes dozens of infrastructure providers (AWS, Azure, GCP, vSphere, OpenStack, Docker, bare metal, and many more) maintained by different teams at different paces.

A Management Cluster Managing Itself and Others: The Pivot

The genuinely awkward part of CAPI’s bootstrap story: you need a running Kubernetes cluster to host CAPI’s controllers before CAPI can create your first real cluster. The standard pattern:

# Step 1: spin up a throwaway local cluster just to run CAPI controllers
$ kind create cluster --name capi-bootstrap
$ clusterctl init --infrastructure aws

# Step 2: use that temporary management cluster to provision the REAL cluster
$ clusterctl generate cluster prod-us-east --infrastructure aws | kubectl apply -f -

# Step 3: move CAPI's own resources OFF the throwaway cluster and ONTO
# the cluster that was just created — "pivoting" management to itself
$ clusterctl move --to-kubeconfig=./prod-us-east.kubeconfig
Performing move...
Discovering Cluster API objects
Moving Cluster API objects: Clusters=1, Machines=3, ...

After the pivot, the cluster CAPI created is now managing its own lifecycle (and can go on to manage other clusters too) — the temporary kind cluster can be torn down. This bootstrap-then-pivot dance is elegant in theory and one of the more fragile operational moments in CAPI’s lifecycle in practice.

How Rancher and Others Build On CAPI

Rancher’s newer cluster provisioning (EP04) increasingly leans on CAPI patterns rather than maintaining entirely separate provisioning logic — the industry direction across the Kubernetes ecosystem has been toward CAPI as the shared substrate for “declare a cluster, get a cluster,” with vendors building their own UX and opinionated defaults on top rather than reinventing the reconciliation model itself.

Production Gotchas

Provider version compatibility is a real support matrix, not a “probably fine” assumption. Core CAPI and each infrastructure provider version independently — upgrading one without checking the compatibility matrix for the other is a common source of cryptic reconciliation failures.

clusterctl move is a rare, high-stakes operation — most teams run it once per cluster’s lifetime, if ever, which means nobody on the team has recent hands-on experience when something goes wrong. Test the pivot in a non-production scenario before relying on it for anything real.

A Machine stuck in Provisioning can mean the infrastructure provider, the bootstrap provider, or the actual cloud API — three different places to look, and the Machine object’s own status doesn’t always make it obvious which. Check the infrastructure-specific object (AWSMachine, AzureMachine, etc.) directly, not just the generic Machine.

Quick Reference

clusterctl init --infrastructure <provider>   # install CAPI + a provider on the management cluster
clusterctl generate cluster <name> --infrastructure <provider>   # generate cluster manifests
kubectl get clusters                           # cluster lifecycle phase
kubectl get machines                           # per-node provisioning phase
kubectl get awsmachines / azuremachines / ...   # provider-specific detail
clusterctl move --to-kubeconfig=<path>          # pivot management to another cluster
clusterctl describe cluster <name>              # human-readable status tree

Contribution Opportunity: clusterctl move‘s Fragility With Non-Native Resources

The limitation: clusterctl move knows how to move CAPI’s own well-known resource types between management clusters cleanly. When a provider or an operator has added custom resources that reference or extend CAPI objects — a common real-world pattern — move doesn’t always understand the relationship, and teams have reported needing manual intervention (patching, reapplying, or reordering) to get a full pivot to succeed cleanly. This is documented in multiple open issues against the project, not a rare edge case.

Why it’s hard to fix: move‘s core logic has to correctly identify and preserve object references and ownership across an arbitrary graph of custom resources it wasn’t necessarily designed to know about — building a fully general solution risks either false confidence (silently missing a reference) or false failure (over-cautiously blocking a move that would have been fine). The CAPI maintainers have to weigh correctness against usability here, and it’s a genuinely hard design problem, not a simple bug.

What a contribution-shaped fix looks like: Two realistic, scoped starting points: (1) a --dry-run-style pre-flight checker for clusterctl move that specifically scans for custom resources referencing CAPI objects and flags them before the move attempt, rather than discovering the gap mid-operation; or (2) contributing a documented, tested procedure (and ideally a small helper tool) for the specific pattern of “extra resources referencing Machine/Cluster objects” that’s already been reported in the project’s issue tracker — turning a known, recurring support question into a documented, repeatable procedure.

Key Takeaways

• CAPI applies Kubernetes’ own reconciliation pattern one layer up — Cluster and Machine objects are declarations, reconciled into real infrastructure by provider-specific controllers
• The core/provider split keeps CAPI infrastructure-agnostic, at the cost of independent versioning you have to track across a real compatibility matrix
• The bootstrap-then-pivot pattern is CAPI’s most elegant and most operationally fragile moment — rehearse it before you need it for real
• Rancher and other platform tools increasingly build their own provisioning UX on top of CAPI’s reconciliation model rather than replacing it
• The clearest contribution opportunity is clusterctl move‘s handling of non-native custom resources — a documented, scoped gap with real prior art in the issue tracker

What’s Next

CAPI treats infrastructure — VMs, networks, load balancers — as the thing being reconciled into existence from Kubernetes objects. EP06 takes that same idea and generalizes it as far as it can go: Crossplane turns Kubernetes into a control plane for effectively any cloud resource, not just the ones needed to run Kubernetes itself.

Next: EP06 — Crossplane: Kubernetes as the Universal Control Plane

Get EP06 in your inbox when it publishes → linuxcent.com/subscribe

The post Cluster API: Declarative Cluster Lifecycle — Rancher’s Foundation Layer appeared first on Linuxcent.

Kubernetes Ecosystem: From User to Contributor, Episode 4
← EP03: k3s vs MicroK8s vs Minikube · EP04: Rancher · EP05: Cluster API →

11 min read

TL;DR

• Rancher multi-cluster management means one Rancher server managing many downstream Kubernetes clusters — its own RKE2/k3s clusters, or imported EKS/GKE/AKS clusters — from a single pane of glass
• Rancher doesn’t proxy every API call through itself; it deploys a lightweight agent into each downstream cluster that phones home, then aggregates each cluster’s API through that agent
• Fleet, Rancher’s built-in GitOps engine, is what actually pushes manifests to potentially hundreds of clusters from a single git repository — this is the feature that makes “fleet” in the product’s marketing literal, not aspirational
• Rancher’s Projects group namespaces within one cluster for permission management — they are not a cross-cluster grouping, a common misunderstanding
• The Rancher server itself becomes something you now have to operate: HA, upgrades, and version compatibility with every downstream cluster’s Kubernetes version are real, ongoing operational work
• Contribution opportunity: Fleet’s multi-cluster drift visibility has real, specific gaps — covered below

The Big Picture

                    ┌─────────────────────────┐
                    │      RANCHER SERVER       │
                    │  (itself a K8s cluster,   │
                    │   ideally HA, 3+ nodes)   │
                    └────────────┬────────────┘
                                 │ agents phone home,
                                 │ API aggregated back
              ┌──────────────────┼──────────────────┐
              │                  │                  │
        ┌─────▼─────┐     ┌──────▼──────┐    ┌──────▼──────┐
        │  RKE2      │     │  Imported    │    │  Imported    │
        │  cluster   │     │  EKS cluster │    │  GKE cluster │
        │ (Rancher-  │     │ (Rancher     │    │ (Rancher     │
        │  provisioned)│    │  didn't      │    │  didn't      │
        │            │     │  create it)  │    │  create it)  │
        └────────────┘     └─────────────┘    └─────────────┘

Rancher multi-cluster management works by inverting the connection direction most people assume: Rancher doesn’t reach out and control downstream clusters directly. Each downstream cluster runs a small agent that establishes an outbound connection back to the Rancher server — which is why Rancher can manage a cluster sitting behind NAT or a restrictive firewall, as long as that cluster can reach out.

How Rancher Actually Manages Clusters It Didn’t Create

# Import an existing cluster Rancher never touched at creation time
$ kubectl apply -f https://rancher.example.com/v3/import/<token>.yaml
# This installs the cattle-cluster-agent into the target cluster —
# that agent is the only thing Rancher needs to start managing it

$ kubectl get pods -n cattle-system
NAME                                    READY   STATUS    RESTARTS
cattle-cluster-agent-7d8f9c-x2k9l       1/1     Running   0

Once the agent is running, Rancher’s UI and API present that cluster’s resources as if you were talking to it directly — the agent maintains the tunnel and relays API calls both ways. This is the architectural reason Rancher can manage a genuinely heterogeneous fleet: RKE2, k3s, EKS, GKE, AKS, and on-prem clusters all look identical to Rancher once the same agent is running in each.

RKE2 and k3s: Rancher’s Own Cluster Distributions

Rancher can also provision brand-new clusters directly, using its own distributions:

# Provisioning a new downstream cluster via Rancher's cluster API
# (typically done through the UI, but expressible as a CR)
$ kubectl apply -f - <<EOF
apiVersion: provisioning.cattle.io/v1
kind: Cluster
metadata:
  name: edge-fleet-01
  namespace: fleet-default
spec:
  kubernetesVersion: v1.28.9+rke2r1
  rkeConfig:
    machinePools:
    - name: pool-01
      quantity: 3
EOF

RKE2 (“RKE Government,” a CIS-hardened, more security-focused distribution) and k3s (the lightweight distribution covered in EP03) are both Rancher/SUSE projects, and Rancher treats them as first-class provisioning targets — this is the direct product connection between “the lightweight Kubernetes distro you picked in EP03” and “the fleet manager covered in this episode.”

Fleet: GitOps at Fleet Scale

# Fleet watches a git repo and deploys its manifests to a TARGETED
# set of clusters based on label selectors — not necessarily all of them
$ kubectl apply -f - <<EOF
apiVersion: fleet.cattle.io/v1alpha1
kind: GitRepo
metadata:
  name: platform-baseline
  namespace: fleet-default
spec:
  repo: https://github.com/example-org/platform-manifests
  branch: main
  targets:
  - clusterSelector:
      matchLabels:
        env: production
EOF

$ kubectl get gitrepo -n fleet-default
NAME                REPO                                          COMMIT     BUNDLESREADY
platform-baseline   https://github.com/example-org/platform-...   a1b2c3d    12/14
#                                                                              ^^^^^ — 2 clusters
#                                                                              haven't converged yet

BUNDLESREADY 12/14 is the number that matters at fleet scale — it tells you how many of the targeted clusters have actually converged to the git state, but notice it doesn’t tell you why the other 2 haven’t, or which 2 they are, without drilling into each bundle individually. That’s the exact gap covered in this episode’s contribution section.

Projects and RBAC: Rancher’s Multi-Tenancy Layer

A common misconception worth correcting directly: Rancher’s Projects group namespaces within a single cluster for permission and resource-quota management — they are not a mechanism for grouping resources across clusters. Cross-cluster access control is handled separately, through Cluster-level and Global roles assigned per user or group.

Global scope        → applies across every cluster Rancher manages
  └── Cluster scope  → applies to all namespaces in one specific cluster
        └── Project scope → applies to a defined subset of namespaces
              within that one cluster (Rancher's own grouping construct)

Getting this hierarchy backwards — assuming a Project spans clusters — is one of the most common Rancher RBAC mistakes teams make when first designing their permission model.

Production Gotchas

Rancher server itself needs HA, and losing it doesn’t take down downstream clusters — but it does take down your ability to manage them centrally. Downstream clusters keep running their workloads fine if Rancher server is unreachable; you just lose the single-pane-of-glass view and Fleet’s GitOps reconciliation until it’s back.

Version skew between Rancher server and downstream Kubernetes versions is a real, documented compatibility matrix — not a “should mostly work” situation. Upgrading Rancher server ahead of your downstream clusters’ Kubernetes versions (or vice versa, letting downstream clusters drift too far ahead) can break agent compatibility. Check Rancher’s official support matrix before any upgrade, not after something breaks.

Agent reconnection storms after a Rancher server upgrade or restart are a known operational event, not a bug report. If you manage dozens of downstream clusters, expect a burst of reconnection activity immediately after any Rancher server maintenance — plan maintenance windows with that in mind.

Quick Reference

kubectl apply -f import.yaml              # import an existing cluster
kubectl get clusters.provisioning.cattle.io -A   # all clusters Rancher manages
kubectl get gitrepo -n fleet-default       # Fleet GitOps repo status
kubectl get bundles -n fleet-default       # per-cluster deployment bundle status
kubectl get pods -n cattle-system          # agent health, on a downstream cluster

Contribution Opportunity: Fleet’s Multi-Cluster Drift Visibility

The limitation: Fleet’s BUNDLESREADY count tells you how many targeted clusters have converged, but drilling into why a specific cluster hasn’t — a stuck rollout, a resource conflict, a cluster that’s unreachable — still requires checking that cluster’s bundle status individually. At a fleet of dozens or hundreds of clusters, there’s no aggregated view that surfaces “these 3 clusters are all failing for the same underlying reason” without manual cross-referencing.

Why it’s hard to fix: Aggregating meaningful failure reasons across a heterogeneous fleet is genuinely harder than it sounds — a “failed” bundle on one cluster might be a transient network blip, on another a real manifest conflict, and on a third a resource quota limit. Building a dashboard that correctly buckets and summarizes those different failure classes without producing a wall of noise is a real UX and data-modeling problem, and it’s not the kind of thing that gets prioritized ahead of core provisioning reliability work.

What a contribution-shaped fix looks like: A scoped, achievable starting point: a fleet CLI plugin or a Rancher UI extension that queries all Bundle resources across the fleet’s clusters, groups them by failure-reason similarity (using the existing status conditions Fleet already populates — this is a client-side aggregation problem, not a new backend feature), and surfaces a ranked summary. This is buildable against Fleet’s existing CRDs and status fields without needing to modify Fleet’s core reconciliation logic — exactly the kind of contribution an operator who’s felt this specific pain at scale is positioned to build and upstream.

Key Takeaways

• Rancher manages downstream clusters through an outbound-connecting agent, not by reaching in — this is why it can manage clusters behind NAT or restrictive firewalls
• Fleet is the actual mechanism for GitOps at fleet scale, targeting clusters by label selector and reporting convergence via BUNDLESREADY counts
• Projects group namespaces within one cluster, not across clusters — a frequent RBAC design mistake starts from getting this backwards
• The Rancher server becomes real infrastructure you operate: HA, version-compatibility matrices, and post-upgrade agent reconnection are ongoing operational realities
• The clearest contribution opportunity is Fleet’s drift-visibility gap at scale — a client-side aggregation problem buildable against existing CRDs, not a core-logic change

What’s Next

Rancher’s own cluster provisioning sits on top of a more general pattern: declaring cluster lifecycle as Kubernetes resources. EP05 covers Cluster API directly — the CNCF project Rancher’s own provisioning increasingly builds on, and the pattern several other tools in this series also depend on.

Next: EP05 — Cluster API: Declarative Cluster Lifecycle — Rancher’s Foundation Layer

Get EP05 in your inbox when it publishes → linuxcent.com/subscribe

The post Rancher: Multi-Cluster Kubernetes Management at Scale appeared first on Linuxcent.

Kubernetes CRDs & Operators: Extending the API, Episode 7
What Is a CRD? · CRDs You Already Use · CRD Anatomy · Write Your First CRD · CEL Validation · Controller Loop · Build an Operator · CRD Versioning · Admission Webhooks · CRDs in Production

TL;DR

• Building a Kubernetes operator means writing a Go reconciler with controller-runtime — kubebuilder scaffolds the project structure, RBAC markers, and Makefile targets so you focus on the reconcile logic
  (kubebuilder = a CLI and framework that generates the operator project scaffold; controller-runtime = the Go library that provides the informer cache, work queue, and reconciler interface)
• The reconciler for BackupPolicy in this episode creates and manages a CronJob — it is the behavior layer for the CRD built in EP03–EP05
• RBAC is expressed as Go code comments (//+kubebuilder:rbac:...) — kubebuilder generates the ClusterRole YAML from them
• Run the operator locally with make run during development; no cluster deployment needed until ready
• The same project that builds the operator also builds and installs the CRD — make install applies the CRD YAML generated from your Go types
• Testing: the operator ships with envtest — a local API server + etcd for controller testing without a real cluster

The Big Picture

  OPERATOR PROJECT STRUCTURE (kubebuilder scaffold)

  backup-operator/
  ├── api/v1alpha1/
  │   ├── backuppolicy_types.go     ← Go types that define CRD schema
  │   └── groupversion_info.go
  ├── internal/controller/
  │   └── backuppolicy_controller.go ← reconcile logic (our main focus)
  ├── config/
  │   ├── crd/                       ← generated CRD YAML
  │   ├── rbac/                      ← generated RBAC YAML
  │   └── manager/                   ← controller Deployment YAML
  ├── cmd/main.go                    ← entrypoint, sets up the manager
  └── Makefile                       ← build, test, install, deploy targets

  FLOW:
  Go types → kubebuilder generate → CRD YAML + RBAC YAML
  Reconcile function → runs in cluster → watches BackupPolicy → manages CronJobs

Building a Kubernetes operator with controller-runtime is where CRDs become living infrastructure — the BackupPolicy objects created in EP04 now get actual behavior attached to them.

Prerequisites

# Go 1.22+
go version

# kubebuilder CLI
curl -L -o kubebuilder \
  https://github.com/kubernetes-sigs/kubebuilder/releases/latest/download/kubebuilder_linux_amd64
chmod +x kubebuilder
sudo mv kubebuilder /usr/local/bin/

# A running cluster (kind works well for development)
kind create cluster --name operator-dev

# Verify kubectl works
kubectl cluster-info --context kind-operator-dev

Step 1: Scaffold the Project

mkdir backup-operator && cd backup-operator

# Initialize the Go module and project structure
kubebuilder init \
  --domain storage.example.com \
  --repo github.com/example/backup-operator

# Create the API (Go types + controller scaffold)
kubebuilder create api \
  --group storage \
  --version v1alpha1 \
  --kind BackupPolicy \
  --resource \
  --controller

When prompted:

Create Resource [y/n]: y
Create Controller [y/n]: y

The generated directory tree:

backup-operator/
├── api/
│   └── v1alpha1/
│       ├── backuppolicy_types.go
│       └── groupversion_info.go
├── internal/
│   └── controller/
│       └── backuppolicy_controller.go
├── cmd/
│   └── main.go
├── config/
│   ├── crd/bases/
│   ├── rbac/
│   └── manager/
├── go.mod
├── go.sum
└── Makefile

Step 2: Define the Go Types

Edit api/v1alpha1/backuppolicy_types.go to match the schema from EP03:

package v1alpha1

import (
    metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
)

// BackupTarget specifies a namespace to include in the backup.
type BackupTarget struct {
    Namespace      string `json:"namespace"`
    IncludeSecrets bool   `json:"includeSecrets,omitempty"`
}

// BackupPolicySpec defines the desired state of BackupPolicy.
type BackupPolicySpec struct {
    // Schedule is a cron expression for when to run backups.
    // +kubebuilder:validation:Pattern=`^(\*|[0-9,\-\/]+) (\*|[0-9,\-\/]+) (\*|[0-9,\-\/]+) (\*|[0-9,\-\/]+) (\*|[0-9,\-\/]+)$`
    Schedule string `json:"schedule"`

    // RetentionDays is how long to keep backup snapshots.
    // +kubebuilder:validation:Minimum=1
    // +kubebuilder:validation:Maximum=365
    RetentionDays int32 `json:"retentionDays"`

    // StorageClass is the storage class to use for backup volumes.
    // +kubebuilder:default=standard
    // +kubebuilder:validation:Enum=standard;premium;encrypted;archive
    StorageClass string `json:"storageClass,omitempty"`

    // Targets lists the namespaces and resources to include.
    // +kubebuilder:validation:MaxItems=20
    Targets []BackupTarget `json:"targets,omitempty"`

    // Suspended pauses backup execution when true.
    // +kubebuilder:default=false
    Suspended bool `json:"suspended,omitempty"`
}

// BackupPolicyStatus defines the observed state of BackupPolicy.
type BackupPolicyStatus struct {
    // Conditions reflect the current state of the BackupPolicy.
    Conditions []metav1.Condition `json:"conditions,omitempty"`

    // LastBackupTime is when the most recent backup completed.
    LastBackupTime *metav1.Time `json:"lastBackupTime,omitempty"`

    // CronJobName is the name of the managed CronJob.
    CronJobName string `json:"cronJobName,omitempty"`
}

// +kubebuilder:object:root=true
// +kubebuilder:subresource:status
// +kubebuilder:printcolumn:name="Schedule",type=string,JSONPath=`.spec.schedule`
// +kubebuilder:printcolumn:name="Retention",type=integer,JSONPath=`.spec.retentionDays`
// +kubebuilder:printcolumn:name="Suspended",type=boolean,JSONPath=`.spec.suspended`
// +kubebuilder:printcolumn:name="Ready",type=string,JSONPath=`.status.conditions[?(@.type=='Ready')].status`
// +kubebuilder:printcolumn:name="Age",type=date,JSONPath=`.metadata.creationTimestamp`

// BackupPolicy is the Schema for the backuppolicies API.
type BackupPolicy struct {
    metav1.TypeMeta   `json:",inline"`
    metav1.ObjectMeta `json:"metadata,omitempty"`

    Spec   BackupPolicySpec   `json:"spec,omitempty"`
    Status BackupPolicyStatus `json:"status,omitempty"`
}

// +kubebuilder:object:root=true

// BackupPolicyList contains a list of BackupPolicy.
type BackupPolicyList struct {
    metav1.TypeMeta `json:",inline"`
    metav1.ListMeta `json:"metadata,omitempty"`
    Items           []BackupPolicy `json:"items"`
}

func init() {
    SchemeBuilder.Register(&BackupPolicy{}, &BackupPolicyList{})
}

Regenerate the CRD YAML and DeepCopy methods:

make generate   # regenerates zz_generated.deepcopy.go
make manifests  # regenerates CRD YAML under config/crd/bases/

Step 3: Write the Reconciler

Edit internal/controller/backuppolicy_controller.go:

package controller

import (
    "context"
    "fmt"

    batchv1 "k8s.io/api/batch/v1"
    corev1 "k8s.io/api/core/v1"
    apierrors "k8s.io/apimachinery/pkg/api/errors"
    "k8s.io/apimachinery/pkg/api/meta"
    metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
    "k8s.io/apimachinery/pkg/runtime"
    "k8s.io/apimachinery/pkg/types"
    ctrl "sigs.k8s.io/controller-runtime"
    "sigs.k8s.io/controller-runtime/pkg/client"
    "sigs.k8s.io/controller-runtime/pkg/log"

    storagev1alpha1 "github.com/example/backup-operator/api/v1alpha1"
)

// BackupPolicyReconciler reconciles BackupPolicy objects.
type BackupPolicyReconciler struct {
    client.Client
    Scheme *runtime.Scheme
}

// RBAC markers — kubebuilder generates ClusterRole YAML from these comments.
//+kubebuilder:rbac:groups=storage.example.com,resources=backuppolicies,verbs=get;list;watch;create;update;patch;delete
//+kubebuilder:rbac:groups=storage.example.com,resources=backuppolicies/status,verbs=get;update;patch
//+kubebuilder:rbac:groups=storage.example.com,resources=backuppolicies/finalizers,verbs=update
//+kubebuilder:rbac:groups=batch,resources=cronjobs,verbs=get;list;watch;create;update;patch;delete

func (r *BackupPolicyReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
    logger := log.FromContext(ctx)

    // Step 1: Fetch the BackupPolicy
    bp := &storagev1alpha1.BackupPolicy{}
    if err := r.Get(ctx, req.NamespacedName, bp); err != nil {
        if apierrors.IsNotFound(err) {
            // Object deleted before we could reconcile — nothing to do.
            return ctrl.Result{}, nil
        }
        return ctrl.Result{}, fmt.Errorf("fetching BackupPolicy: %w", err)
    }

    // Step 2: Define the desired CronJob name
    cronJobName := fmt.Sprintf("%s-backup", bp.Name)

    // Step 3: Fetch the existing CronJob (if any)
    existing := &batchv1.CronJob{}
    err := r.Get(ctx, types.NamespacedName{Name: cronJobName, Namespace: bp.Namespace}, existing)
    notFound := apierrors.IsNotFound(err)
    if err != nil && !notFound {
        return ctrl.Result{}, fmt.Errorf("fetching CronJob: %w", err)
    }

    // Step 4: Build the desired CronJob
    desired := r.buildCronJob(bp, cronJobName)

    // Step 5: Create or update
    if notFound {
        logger.Info("Creating CronJob", "name", cronJobName)
        if err := r.Create(ctx, desired); err != nil {
            return ctrl.Result{}, fmt.Errorf("creating CronJob: %w", err)
        }
    } else {
        // Update schedule and suspend state if they differ
        if existing.Spec.Schedule != desired.Spec.Schedule ||
            existing.Spec.Suspend != desired.Spec.Suspend {
            existing.Spec.Schedule = desired.Spec.Schedule
            existing.Spec.Suspend = desired.Spec.Suspend
            logger.Info("Updating CronJob", "name", cronJobName)
            if err := r.Update(ctx, existing); err != nil {
                return ctrl.Result{}, fmt.Errorf("updating CronJob: %w", err)
            }
        }
    }

    // Step 6: Update status
    bpCopy := bp.DeepCopy()
    meta.SetStatusCondition(&bpCopy.Status.Conditions, metav1.Condition{
        Type:               "Ready",
        Status:             metav1.ConditionTrue,
        Reason:             "CronJobReady",
        Message:            fmt.Sprintf("CronJob %s is configured", cronJobName),
        ObservedGeneration: bp.Generation,
    })
    bpCopy.Status.CronJobName = cronJobName

    if err := r.Status().Update(ctx, bpCopy); err != nil {
        return ctrl.Result{}, fmt.Errorf("updating status: %w", err)
    }

    return ctrl.Result{}, nil
}

func (r *BackupPolicyReconciler) buildCronJob(bp *storagev1alpha1.BackupPolicy, name string) *batchv1.CronJob {
    suspend := bp.Spec.Suspended
    retentionArg := fmt.Sprintf("--retention-days=%d", bp.Spec.RetentionDays)

    cj := &batchv1.CronJob{
        ObjectMeta: metav1.ObjectMeta{
            Name:      name,
            Namespace: bp.Namespace,
            Labels: map[string]string{
                "app.kubernetes.io/managed-by": "backup-operator",
                "backuppolicy":                 bp.Name,
            },
        },
        Spec: batchv1.CronJobSpec{
            Schedule: bp.Spec.Schedule,
            Suspend:  &suspend,
            JobTemplate: batchv1.JobTemplateSpec{
                Spec: batchv1.JobSpec{
                    Template: corev1.PodTemplateSpec{
                        Spec: corev1.PodSpec{
                            RestartPolicy: corev1.RestartPolicyOnFailure,
                            Containers: []corev1.Container{
                                {
                                    Name:    "backup",
                                    Image:   "backup-tool:latest",
                                    Args:    []string{retentionArg},
                                },
                            },
                        },
                    },
                },
            },
        },
    }

    // Set owner reference — CronJob is garbage-collected when BackupPolicy is deleted
    _ = ctrl.SetControllerReference(bp, cj, r.Scheme)
    return cj
}

// SetupWithManager registers the controller with the manager and declares what to watch.
func (r *BackupPolicyReconciler) SetupWithManager(mgr ctrl.Manager) error {
    return ctrl.NewControllerManagedBy(mgr).
        For(&storagev1alpha1.BackupPolicy{}).
        Owns(&batchv1.CronJob{}).    // reconcile BackupPolicy when owned CronJob changes
        Complete(r)
}

Step 4: Install the CRD and Run Locally

# Install the CRD into the cluster
make install

customresourcedefinition.apiextensions.k8s.io/backuppolicies.storage.example.com created

# Run the controller locally (outside the cluster)
make run

2026-04-25T08:00:00Z  INFO  Starting manager
2026-04-25T08:00:00Z  INFO  Starting workers  {"controller": "backuppolicy", "worker count": 1}

In a separate terminal:

kubectl apply -f - <<'EOF'
apiVersion: storage.example.com/v1alpha1
kind: BackupPolicy
metadata:
  name: nightly
  namespace: default
spec:
  schedule: "0 2 * * *"
  retentionDays: 30
EOF

Watch the controller output:

2026-04-25T08:01:00Z  INFO  Creating CronJob  {"name": "nightly-backup"}

Check the result:

kubectl get bp nightly

NAME      SCHEDULE    RETENTION   SUSPENDED   READY   AGE
nightly   0 2 * * *   30          false       True    10s

kubectl get cronjob nightly-backup

NAME             SCHEDULE    SUSPEND   ACTIVE   LAST SCHEDULE   AGE
nightly-backup   0 2 * * *   False     0        <none>          10s

Test self-healing — delete the CronJob and watch the controller recreate it:

kubectl delete cronjob nightly-backup
# Controller output:
# 2026-04-25T08:02:00Z  INFO  Creating CronJob  {"name": "nightly-backup"}

kubectl get cronjob nightly-backup
# Back within seconds

Test suspend:

kubectl patch bp nightly --type=merge -p '{"spec":{"suspended":true}}'
kubectl get cronjob nightly-backup -o jsonpath='{.spec.suspend}'
# true

Step 5: Deploy to Cluster

When ready for in-cluster deployment:

# Build and push the controller image
make docker-build docker-push IMG=your-registry/backup-operator:v0.1.0

# Deploy to cluster (creates Deployment, RBAC, CRD)
make deploy IMG=your-registry/backup-operator:v0.1.0

kubectl get pods -n backup-operator-system

NAME                                          READY   STATUS    RESTARTS   AGE
backup-operator-controller-manager-abc123     2/2     Running   0          30s

Understanding the RBAC Markers

The //+kubebuilder:rbac:... comments in the controller generate the ClusterRole YAML when you run make manifests:

//+kubebuilder:rbac:groups=storage.example.com,resources=backuppolicies,verbs=get;list;watch;create;update;patch;delete
//+kubebuilder:rbac:groups=storage.example.com,resources=backuppolicies/status,verbs=get;update;patch
//+kubebuilder:rbac:groups=batch,resources=cronjobs,verbs=get;list;watch;create;update;patch;delete

Generated YAML under config/rbac/role.yaml:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: manager-role
rules:
  - apiGroups: ["storage.example.com"]
    resources: ["backuppolicies"]
    verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
  - apiGroups: ["storage.example.com"]
    resources: ["backuppolicies/status"]
    verbs: ["get", "update", "patch"]
  - apiGroups: ["batch"]
    resources: ["cronjobs"]
    verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]

This approach keeps RBAC co-located with the code that needs it — if you add a new resource access in the controller, you add the marker next to it.

Common Mistakes

Not setting an owner reference on child resources. Without ctrl.SetControllerReference(parent, child, scheme), deleting the BackupPolicy leaves orphaned CronJobs. Owner references enable automatic garbage collection of child resources.

Updating the object after r.Get() without handling conflicts. If two reconciles run concurrently (possible after a controller restart), both may try to update the same resource. The API server uses resource version for optimistic concurrency — you will get a conflict error. Retry the reconcile on conflict errors rather than failing.

Writing to bp directly instead of bp.DeepCopy() for status updates. If the status update fails and you retry, the original bp object now has the modified status in memory. Always update a deep copy when writing status so the in-memory state stays consistent with what was actually persisted.

Not watching owned resources. If you forget .Owns(&batchv1.CronJob{}) in SetupWithManager, the controller will not reconcile when a CronJob is deleted. Self-healing requires watching the resources you manage.

Quick Reference

# Scaffold a new API + controller
kubebuilder create api --group mygroup --version v1alpha1 --kind MyKind

# Regenerate deep copy methods after changing types
make generate

# Regenerate CRD YAML + RBAC from markers
make manifests

# Install CRD into current cluster
make install

# Run controller locally (outside cluster)
make run

# Build + push image, then deploy to cluster
make docker-build docker-push IMG=registry/operator:tag
make deploy IMG=registry/operator:tag

# Uninstall CRD (WARNING: deletes all instances)
make uninstall

Key Takeaways

• kubebuilder scaffolds the project; you write the types and the reconcile function
• Go struct markers (//+kubebuilder:...) generate the CRD YAML and RBAC — keep them close to the code they describe
• ctrl.SetControllerReference enables automatic garbage collection of child resources
• Always deep-copy the object before writing status; retry on conflict errors
• make run runs the controller locally — no Docker build needed during development

What’s Next

EP08: Kubernetes CRD Versioning covers how to evolve the BackupPolicy schema from v1alpha1 to v1 without breaking existing clients — storage versions, conversion webhooks, and the hub-and-spoke model for safe API evolution in production clusters.

Get EP08 in your inbox when it publishes → subscribe at linuxcent.com

The post Build a Simple Kubernetes Operator with controller-runtime and kubebuilder appeared first on Linuxcent.

Kubernetes CRDs & Operators: Extending the API, Episode 4
What Is a CRD? · CRDs You Already Use · CRD Anatomy · Write Your First CRD · CEL Validation · Controller Loop · Build an Operator · CRD Versioning · Admission Webhooks · CRDs in Production

TL;DR

• Writing a Kubernetes CRD requires five YAML files: the CRD itself, a ClusterRole/ClusterRoleBinding, a namespaced Role/RoleBinding for consumers, and a sample custom resource
• The BackupPolicy CRD built in this episode is the running example throughout the rest of the series — operators, versioning, and production patterns all use it
• Apply the CRD, verify it with kubectl get crds, create a custom resource, and watch the API server validate your spec
• RBAC for CRDs follows the same Role/ClusterRole model as built-in resources — the generated resource name is {plural}.{group}
• Schema validation fires at apply time: bad field types, missing required fields, and out-of-range values all return clear errors before anything reaches etcd
• Without a controller, a BackupPolicy is stored in etcd but nothing acts on it — that is the topic of EP05 and EP07

The Big Picture

  WHAT WE'RE BUILDING IN THIS EPISODE

  1. backuppolicies-crd.yaml        ← registers the BackupPolicy type
  2. backuppolicies-rbac.yaml       ← controls who can create/view/delete
  3. nightly-backup.yaml            ← our first custom resource instance

  After applying:

  kubectl get crds | grep backup      ← BackupPolicy type exists
  kubectl get backuppolicies -n demo  ← nightly instance exists
  kubectl describe bp nightly -n demo ← spec visible, status empty
  kubectl apply -f bad-backup.yaml    ← schema validation rejects bad data

Writing your first Kubernetes CRD is the step that bridges understanding CRDs conceptually to operating them in a real cluster. This episode is hands-on — every block of YAML is something you apply and verify.

Prerequisites

You need a running Kubernetes cluster and kubectl configured. Any of these work:

# Local options
kind create cluster --name crd-demo
# or
minikube start

# Verify cluster access
kubectl cluster-info
kubectl get nodes

Step 1: Write the CRD

Save this as backuppolicies-crd.yaml:

apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: backuppolicies.storage.example.com
spec:
  group: storage.example.com
  scope: Namespaced
  names:
    plural:     backuppolicies
    singular:   backuppolicy
    kind:       BackupPolicy
    shortNames:
      - bp
    categories:
      - storage
  versions:
    - name: v1alpha1
      served: true
      storage: true
      schema:
        openAPIV3Schema:
          type: object
          required: ["spec"]
          properties:
            spec:
              type: object
              required: ["schedule", "retentionDays"]
              properties:
                schedule:
                  type: string
                  description: "Cron expression (e.g. '0 2 * * *' for 02:00 daily)"
                retentionDays:
                  type: integer
                  minimum: 1
                  maximum: 365
                  description: "How many days to retain backup snapshots"
                storageClass:
                  type: string
                  default: "standard"
                  description: "StorageClass to use for backup volumes"
                targets:
                  type: array
                  description: "Namespaces and resources to include in the backup"
                  maxItems: 20
                  items:
                    type: object
                    required: ["namespace"]
                    properties:
                      namespace:
                        type: string
                      includeSecrets:
                        type: boolean
                        default: false
                suspended:
                  type: boolean
                  default: false
                  description: "Set to true to pause backup execution"
            status:
              type: object
              x-kubernetes-preserve-unknown-fields: true
      subresources:
        status: {}
      additionalPrinterColumns:
        - name: Schedule
          type: string
          jsonPath: .spec.schedule
        - name: Retention
          type: integer
          jsonPath: .spec.retentionDays
        - name: Suspended
          type: boolean
          jsonPath: .spec.suspended
        - name: Ready
          type: string
          jsonPath: .status.conditions[?(@.type=='Ready')].status
        - name: Age
          type: date
          jsonPath: .metadata.creationTimestamp

Apply it:

kubectl apply -f backuppolicies-crd.yaml

Verify it registered correctly:

kubectl get crds backuppolicies.storage.example.com

NAME                                    CREATED AT
backuppolicies.storage.example.com      2026-04-25T08:00:00Z

Check the API server now knows about it:

kubectl api-resources | grep backuppolic

backuppolicies    bp    storage.example.com/v1alpha1    true    BackupPolicy

Check it is Established:

kubectl get crd backuppolicies.storage.example.com \
  -o jsonpath='{.status.conditions[?(@.type=="Established")].status}'

True

If you see False or empty output, wait a few seconds and retry — the API server takes a moment to register new CRDs.

Step 2: Write RBAC

CRDs follow the same RBAC model as built-in resources. The resource name is {plural}.{group}.

Save this as backuppolicies-rbac.yaml:

# ClusterRole for operators/controllers that manage BackupPolicy objects
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: backuppolicy-controller
rules:
  - apiGroups: ["storage.example.com"]
    resources: ["backuppolicies"]
    verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
  - apiGroups: ["storage.example.com"]
    resources: ["backuppolicies/status"]
    verbs: ["get", "update", "patch"]
  - apiGroups: ["storage.example.com"]
    resources: ["backuppolicies/finalizers"]
    verbs: ["update"]
---
# Role for application teams to manage BackupPolicies in their namespace
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: backuppolicy-editor
rules:
  - apiGroups: ["storage.example.com"]
    resources: ["backuppolicies"]
    verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
---
# Read-only role for auditors
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: backuppolicy-viewer
rules:
  - apiGroups: ["storage.example.com"]
    resources: ["backuppolicies"]
    verbs: ["get", "list", "watch"]

kubectl apply -f backuppolicies-rbac.yaml

Verify the roles exist:

kubectl get clusterrole | grep backuppolicy

backuppolicy-controller   2026-04-25T08:01:00Z
backuppolicy-editor       2026-04-25T08:01:00Z
backuppolicy-viewer       2026-04-25T08:01:00Z

Note on backuppolicies/status: The separate status RBAC rule is only meaningful if you enabled the status subresource (we did). Without it, status and spec share the same update path.

Step 3: Create a Namespace and Your First Custom Resource

kubectl create namespace demo

Save this as nightly-backup.yaml:

apiVersion: storage.example.com/v1alpha1
kind: BackupPolicy
metadata:
  name: nightly
  namespace: demo
  labels:
    app.kubernetes.io/managed-by: manual
spec:
  schedule: "0 2 * * *"
  retentionDays: 30
  storageClass: standard
  targets:
    - namespace: production
      includeSecrets: false
    - namespace: staging
      includeSecrets: false
  suspended: false

Apply it:

kubectl apply -f nightly-backup.yaml

Get it back:

kubectl get backuppolicies -n demo

NAME      SCHEDULE    RETENTION   SUSPENDED   READY   AGE
nightly   0 2 * * *   30          false       <none>  5s

The Ready column is <none> because there is no controller writing status yet. The custom resource exists and is stored in etcd, but nothing is acting on it.

Describe it:

kubectl describe bp nightly -n demo

Name:         nightly
Namespace:    demo
Labels:       app.kubernetes.io/managed-by=manual
Annotations:  <none>
API Version:  storage.example.com/v1alpha1
Kind:         BackupPolicy
Metadata:
  Creation Timestamp:  2026-04-25T08:05:00Z
  ...
Spec:
  Retention Days:  30
  Schedule:        0 2 * * *
  Storage Class:   standard
  Suspended:       false
  Targets:
    Include Secrets:  false
    Namespace:        production
    Include Secrets:  false
    Namespace:        staging
Status:
Events:  <none>

Step 4: Test Schema Validation

The API server now validates every BackupPolicy against the schema. Try creating an invalid one:

kubectl apply -f - <<'EOF'
apiVersion: storage.example.com/v1alpha1
kind: BackupPolicy
metadata:
  name: bad-policy
  namespace: demo
spec:
  schedule: "not-a-cron"
  retentionDays: 500
EOF

The BackupPolicy "bad-policy" is invalid:
  spec.retentionDays: Invalid value: 500:
    spec.retentionDays in body should be less than or equal to 365

Missing required field:

kubectl apply -f - <<'EOF'
apiVersion: storage.example.com/v1alpha1
kind: BackupPolicy
metadata:
  name: missing-schedule
  namespace: demo
spec:
  retentionDays: 7
EOF

The BackupPolicy "missing-schedule" is invalid:
  spec.schedule: Required value

Wrong type:

kubectl apply -f - <<'EOF'
apiVersion: storage.example.com/v1alpha1
kind: BackupPolicy
metadata:
  name: wrong-type
  namespace: demo
spec:
  schedule: "0 2 * * *"
  retentionDays: "thirty"
EOF

The BackupPolicy "wrong-type" is invalid:
  spec.retentionDays: Invalid value: "string":
    spec.retentionDays in body must be of type integer: "string"

All validation fires at the API boundary — before etcd, before any controller sees the object.

Step 5: Verify Default Values Apply

The schema defines storageClass: default: "standard" and suspended: default: false. Verify they are applied even when not specified:

kubectl apply -f - <<'EOF'
apiVersion: storage.example.com/v1alpha1
kind: BackupPolicy
metadata:
  name: minimal
  namespace: demo
spec:
  schedule: "0 0 * * 0"
  retentionDays: 7
EOF

kubectl get bp minimal -n demo -o jsonpath='{.spec.storageClass}'

standard

kubectl get bp minimal -n demo -o jsonpath='{.spec.suspended}'

false

Defaults are injected by the API server at admission time. They appear in etcd and in every kubectl get -o yaml output — the stored object includes the defaults even if the user did not specify them.

Step 6: Explore the API Endpoints

Your custom resource is now available at standard REST endpoints:

kubectl proxy --port=8001 &

# List all BackupPolicies in the demo namespace
curl -s http://localhost:8001/apis/storage.example.com/v1alpha1/namespaces/demo/backuppolicies \
  | jq '.items[].metadata.name'

"nightly"
"minimal"

# Get a specific BackupPolicy
curl -s http://localhost:8001/apis/storage.example.com/v1alpha1/namespaces/demo/backuppolicies/nightly \
  | jq '.spec'

This is how controllers discover and watch custom resources — via the same API server endpoints, using informers that wrap these REST calls with efficient list-and-watch semantics.

Step 7: Clean Up

kubectl delete namespace demo
kubectl delete -f backuppolicies-rbac.yaml
kubectl delete -f backuppolicies-crd.yaml   # WARNING: deletes all BackupPolicy instances first

Common Mistakes

metadata.name does not match {plural}.{group}. The most common error. If you name the CRD backuppolicy.storage.example.com (singular) but the spec says plural: backuppolicies, the API server rejects it. The name must always be {plural}.{group}.

No required fields on spec. Without required constraints, kubectl apply accepts an empty spec: {}. The controller then receives objects with no configuration and has to handle the nil case. Define required fields in the schema.

Forgetting subresources: status: {}. Without this, controllers writing .status also overwrite .spec on full PUT updates. This causes status updates to reset user edits. Enable the status subresource from day one.

Not testing validation errors. Schema validation is the first line of defense. Always explicitly test that your required fields are required, types are enforced, and range constraints work — before deploying the controller.

Quick Reference

# All kubectl operations work on custom resources
kubectl get      backuppolicies -n demo
kubectl get      bp -n demo                  # shortName
kubectl describe bp nightly -n demo
kubectl edit     bp nightly -n demo
kubectl delete   bp nightly -n demo

# Output formats
kubectl get bp -n demo -o yaml
kubectl get bp -n demo -o json
kubectl get bp -n demo -o jsonpath='{.items[*].metadata.name}'

# Watch for changes
kubectl get bp -n demo -w

# List across all namespaces
kubectl get bp -A

# Patch spec
kubectl patch bp nightly -n demo \
  --type=merge -p '{"spec":{"suspended":true}}'

Key Takeaways

• A working CRD deployment needs: the CRD YAML, RBAC ClusterRoles, and at least one sample custom resource
• The API server validates all custom resources against the schema at apply time — errors are surfaced immediately, not inside the controller
• Default values in the schema are injected at admission time and appear in every stored object
• RBAC for custom resources uses {plural}.{group} as the resource name — status and finalizers are separate sub-resources
• Without a controller, custom resources are stored in etcd and serve as validated configuration — nothing acts on them until a controller is deployed

What’s Next

EP05: Kubernetes CRD CEL Validation extends schema validation beyond simple type and range checks — cross-field rules (“if storageClass is premium, retentionDays must be at most 90″), regex validation beyond pattern, and immutable field enforcement. All without an admission webhook.

Get EP05 in your inbox when it publishes → subscribe at linuxcent.com

The post Write Your First Kubernetes CRD: A Hands-On YAML Walkthrough appeared first on Linuxcent.

Kubernetes CRDs & Operators: Extending the API, Episode 3
What Is a CRD? · CRDs You Already Use · CRD Anatomy · Write Your First CRD · CEL Validation · Controller Loop · Build an Operator · CRD Versioning · Admission Webhooks · CRDs in Production

TL;DR

• The Kubernetes CRD schema is defined in spec.versions[].schema.openAPIV3Schema — the API server uses it to validate every custom resource create and update before storing in etcd
  (OpenAPI v3 schema = a JSON Schema dialect that describes the structure, types, and constraints of your resource’s fields)
• spec.versions is a list — CRDs can serve multiple API versions simultaneously; exactly one version must have storage: true
• scope: Namespaced vs scope: Cluster controls whether custom resources live inside a namespace or at cluster level (like PersistentVolume vs PersistentVolumeClaim)
• spec.names defines the plural, singular, kind, and optional shortNames used in kubectl and RBAC
• The status subresource (subresources.status: {}) separates user writes (spec) from controller writes (status) — enabling optimistic concurrency and kubectl status support
• The scale subresource (subresources.scale) makes your custom resource compatible with kubectl scale and the HorizontalPodAutoscaler

The Big Picture

  ANATOMY OF A CUSTOMRESOURCEDEFINITION

  apiVersion: apiextensions.k8s.io/v1
  kind: CustomResourceDefinition
  metadata:
    name: {plural}.{group}        ← MUST be exactly this format
  spec:
    group: {group}                ← API group (e.g. storage.example.com)
    scope: Namespaced | Cluster   ← where instances live
    names:                        ← how kubectl refers to this resource
      plural: backuppolicies
      singular: backuppolicy
      kind: BackupPolicy
      shortNames: [bp]
    versions:                     ← can be a list; one must have storage: true
      - name: v1alpha1
        served: true              ← API server responds to this version
        storage: true             ← etcd stores objects in this version
        schema:
          openAPIV3Schema:        ← validation schema for ALL objects of this type
            type: object
            properties:
              spec: {...}
              status: {...}
        subresources:
          status: {}              ← enables separate status write path
          scale:                  ← enables kubectl scale + HPA
            specReplicasPath: .spec.replicas
            statusReplicasPath: .status.replicas
        additionalPrinterColumns: ← extra columns in kubectl get output
          - name: Schedule
            type: string
            jsonPath: .spec.schedule

Understanding the Kubernetes CRD schema is the prerequisite for writing a CRD that behaves correctly in production — validation catches bad data at the API boundary, the status subresource prevents controller race conditions, and scope determines your entire RBAC and multi-tenancy model.

spec.group and metadata.name

The group is a reverse-DNS identifier for your API. Convention:

storage.example.com     ← domain you control + functional area
monitoring.myteam.io
databases.platform.company.com

The CRD’s metadata.name must be exactly {plural}.{group}:

metadata:
  name: backuppolicies.storage.example.com
spec:
  group: storage.example.com
  names:
    plural: backuppolicies

If these do not match, the API server rejects the CRD with a validation error. This is the most common first-timer mistake.

spec.scope: Namespaced vs Cluster

  SCOPE DETERMINES WHERE INSTANCES LIVE

  Namespaced (scope: Namespaced)       Cluster (scope: Cluster)
  ─────────────────────────────         ──────────────────────────
  kubectl get backuppolicies -n prod    kubectl get clusterbackuppolicies
  kubectl get backuppolicies -A         (no -n flag, no namespace)

  Analogous to: Pod, Deployment,        Analogous to: PersistentVolume,
                ConfigMap                             ClusterRole, Node

Namespaced: Use when instances are per-tenant or per-application. Users with namespace-scoped RBAC can manage their own instances without cluster-admin. Most CRDs should be namespaced.

Cluster-scoped: Use when instances represent cluster-wide configuration — a ClusterIssuer (cert-manager), ClusterSecretStore (ESO), a StorageClass-like concept. Requires cluster-level RBAC to create/modify.

You cannot change scope after a CRD is created without deleting and recreating it (which deletes all instances). Choose carefully.

spec.versions: Serving Multiple API Versions

spec:
  versions:
    - name: v1alpha1
      served: true
      storage: false       # not stored; converted on read
      schema:
        openAPIV3Schema: {...}
    - name: v1beta1
      served: true
      storage: false
      schema:
        openAPIV3Schema: {...}
    - name: v1
      served: true
      storage: true        # etcd stores in this version
      schema:
        openAPIV3Schema: {...}

Rules:
– served: true means the API server accepts requests at this version
– served: false means the API server returns 404 for that version — use to deprecate
– Exactly one version must have storage: true — this is what gets written to etcd
– When a client requests a non-storage version, the API server converts on the fly (or calls your conversion webhook — see EP08)

Early in development, start with v1alpha1 storage: true. Promote to v1 when the schema is stable. EP08 covers how to do this without losing data.

spec.names: What kubectl Sees

spec:
  names:
    plural:     backuppolicies     # kubectl get backuppolicies
    singular:   backuppolicy       # kubectl get backuppolicy (also works)
    kind:       BackupPolicy       # used in YAML apiVersion/kind
    listKind:   BackupPolicyList   # optional; auto-derived if omitted
    shortNames:                    # kubectl get bp
      - bp
    categories:                    # kubectl get all includes this type
      - all

categories is worth noting: if you add all to categories, your custom resources appear when someone runs kubectl get all -n mynamespace. Most CRDs deliberately do not add this — it clutters get all output. Only add it if your resource is a primary operational concern.

schema.openAPIV3Schema: Validation

The schema is where you define field types, required fields, constraints, and descriptions. The API server validates every create and update against this schema before writing to etcd.

schema:
  openAPIV3Schema:
    type: object
    required: ["spec"]
    properties:
      spec:
        type: object
        required: ["schedule", "retentionDays"]
        properties:
          schedule:
            type: string
            description: "Cron expression for backup schedule"
            pattern: '^(\*|[0-9,\-\/]+)\s+(\*|[0-9,\-\/]+)\s+(\*|[0-9,\-\/]+)\s+(\*|[0-9,\-\/]+)\s+(\*|[0-9,\-\/]+)$'
          retentionDays:
            type: integer
            minimum: 1
            maximum: 365
          storageClass:
            type: string
            default: "standard"        # default value (Kubernetes 1.17+)
          targets:
            type: array
            maxItems: 10
            items:
              type: object
              required: ["name"]
              properties:
                name:
                  type: string
                namespace:
                  type: string
                  default: "default"
      status:
        type: object
        x-kubernetes-preserve-unknown-fields: true   # controllers write arbitrary status

Field types available

x-kubernetes-preserve-unknown-fields: true

This tells the API server not to prune fields it does not know about. Use it on status (controllers write whatever they need) and on fields that are intentionally free-form (like a config field that accepts arbitrary YAML). Avoid it on spec — it bypasses validation.

Validation behavior in practice

# This will fail with a clear error:
kubectl apply -f - <<EOF
apiVersion: storage.example.com/v1alpha1
kind: BackupPolicy
metadata:
  name: bad
  namespace: default
spec:
  schedule: "not-a-cron"    # fails pattern validation
  retentionDays: 500         # fails maximum: 365
EOF

The BackupPolicy "bad" is invalid:
  spec.schedule: Invalid value: "not-a-cron": spec.schedule in body should match
    '^(\*|[0-9,\-\/]+)\s+...'
  spec.retentionDays: Invalid value: 500: spec.retentionDays in body should be
    less than or equal to 365

Schema validation catches configuration mistakes at apply time, not at runtime inside a pod. This is one of the core advantages of expressing domain configuration as CRDs rather than ConfigMaps.

additionalPrinterColumns: What kubectl get Shows

By default, kubectl get backuppolicies shows only NAME and AGE. You can add columns:

additionalPrinterColumns:
  - name: Schedule
    type: string
    jsonPath: .spec.schedule
    description: Cron schedule for backups
  - name: Retention
    type: integer
    jsonPath: .spec.retentionDays
    priority: 1          # 0 = always shown; 1 = only with -o wide
  - name: Ready
    type: string
    jsonPath: .status.conditions[?(@.type=='Ready')].status
  - name: Age
    type: date
    jsonPath: .metadata.creationTimestamp

Result:

NAME        SCHEDULE      READY   AGE
nightly     0 2 * * *     True    3d
weekly      0 0 * * 0     False   7d

Good printer columns turn kubectl get into a useful operational dashboard. Include Ready (from status conditions) so operators can immediately see which custom resources are healthy without running kubectl describe.

The Status Subresource

subresources:
  status: {}

Without the status subresource, spec and status are part of the same object. Any user with update permission on the CRD can modify both. Controllers write status through the same path as users write spec.

With the status subresource enabled:
– kubectl apply / kubectl patch only update spec — the status block is stripped
– Controllers use the /status subresource endpoint to write status
– RBAC can grant update on backuppolicies (spec) independently from update on backuppolicies/status

  WITHOUT status subresource:         WITH status subresource:
  ─────────────────────────            ──────────────────────────
  PUT /backuppolicies/nightly          PUT /backuppolicies/nightly
  → updates spec AND status            → updates spec only

                                       PUT /backuppolicies/nightly/status
                                       → updates status only (controller path)

Always enable the status subresource on production CRDs. The split between spec and status is fundamental to the Kubernetes API contract. Without it, a controller updating status can accidentally overwrite spec changes made by a user at the same time.

The Scale Subresource

subresources:
  scale:
    specReplicasPath: .spec.replicas
    statusReplicasPath: .status.replicas
    labelSelectorPath: .status.labelSelector

This makes your custom resource compatible with:

kubectl scale backuppolicy nightly --replicas=3

And with HorizontalPodAutoscaler targeting your custom resource. If your CRD manages something replica-based (workers, shards, connections), enabling the scale subresource lets it plug into the standard Kubernetes autoscaling ecosystem without extra plumbing.

Common Mistakes

Forgetting x-kubernetes-preserve-unknown-fields: true on status. If you validate the status field with a strict schema but do not add this, the API server will prune any status fields the controller writes that are not in the schema. The controller’s status updates will silently lose fields. Either define the full status schema or use x-kubernetes-preserve-unknown-fields: true.

Using scope: Cluster for resources that should be namespaced. Once a CRD is created as cluster-scoped, you cannot make it namespaced without deleting and recreating it. Plan scope before deploying to production.

Not enabling the status subresource. Without it, controllers writing status can race with users updating spec. It also means kubectl patch --subresource=status does not work and some tooling behaves unexpectedly. Enable it from the start.

Loose schema with no required fields. An openAPIV3Schema with no required constraint accepts objects with empty spec. This usually means your controller gets called with a resource that is missing mandatory configuration. Define required fields and validate them at the API boundary, not inside the controller.

Quick Reference

# Inspect the full schema of a CRD
kubectl get crd backuppolicies.storage.example.com -o yaml | \
  yq '.spec.versions[0].schema'

# Check what subresources are enabled
kubectl get crd certificates.cert-manager.io -o jsonpath=\
  '{.spec.versions[0].subresources}'

# See all served versions for a CRD
kubectl get crd prometheuses.monitoring.coreos.com \
  -o jsonpath='{.spec.versions[*].name}'

# Check which version is the storage version
kubectl get crd certificates.cert-manager.io \
  -o jsonpath='{.spec.versions[?(@.storage==true)].name}'

# Describe the printer columns for a CRD
kubectl get crd scaledobjects.keda.sh \
  -o jsonpath='{.spec.versions[0].additionalPrinterColumns}'

Key Takeaways

• spec.versions allows serving and storing multiple API versions; only one version has storage: true
• scope (Namespaced vs Cluster) cannot be changed after creation — choose deliberately
• openAPIV3Schema validates every CR at the API boundary, before etcd storage
• The status subresource separates the user write path (spec) from the controller write path (status) — always enable it
• additionalPrinterColumns makes kubectl get operationally useful; include a Ready column from status conditions

What’s Next

EP04: Write Your First Kubernetes CRD puts the anatomy into practice — a complete hands-on walkthrough building a BackupPolicy CRD from scratch, applying it to a cluster, creating instances, and verifying validation, RBAC, and status behavior.

Get EP04 in your inbox when it publishes → subscribe at linuxcent.com

The post Kubernetes CRD Schema Explained: Versions, Validation, and Status Subresource appeared first on Linuxcent.

Kubernetes CRDs & Operators: Extending the API, Episode 1
What Is a CRD? · CRDs You Already Use · CRD Anatomy · Write Your First CRD · CEL Validation · Controller Loop · Build an Operator · CRD Versioning · Admission Webhooks · CRDs in Production

TL;DR

• A Kubernetes CRD (Custom Resource Definition) is how you add new resource types to the Kubernetes API — the same way Deployment and Service exist natively, you can make BackupPolicy or Certificate exist too
  (CRD = the schema/blueprint; Custom Resource = an instance of that schema, just like a Pod is an instance of the Pod schema)
• Every kubectl get crds on a real cluster shows dozens of them — cert-manager, KEDA, Prometheus Operator, Crossplane all ship their own CRDs
• CRDs are served by the same API server as built-in resources — kubectl, RBAC, watches, and events all work identically
• A CRD alone does nothing — a controller watches the custom resources and acts on them; together they form an Operator
• CRDs live in etcd just like Pods and Deployments — they survive API server restarts and cluster upgrades
• You do not need to modify Kubernetes source code or restart the API server to add a CRD

The Big Picture

  HOW KUBERNETES CRDs EXTEND THE API

  ┌──────────────────────────────────────────────────────────────┐
  │  Kubernetes API Server                                       │
  │                                                              │
  │  Built-in resources          Custom resources (via CRD)      │
  │  ─────────────────           ──────────────────────────      │
  │  Pod                         Certificate     (cert-manager)  │
  │  Deployment                  ScaledObject    (KEDA)          │
  │  Service                     ExternalSecret  (ESO)           │
  │  ConfigMap                   BackupPolicy    (your team)     │
  │  ...                         ...                             │
  │                                                              │
  │  All resources: same API, same kubectl, same RBAC, same etcd │
  └──────────────────────────────────────────────────────────────┘
            ▲                          ▲
            │ built in                 │ registered at runtime
            │                         │
         Kubernetes              CustomResourceDefinition
          binary                    (a YAML you apply)

What is a Kubernetes CRD? It is a resource that defines resources — a schema registration that teaches the API server about a new object type you want to use in your cluster.

What Problem CRDs Solve

Kubernetes ships with roughly 50 resource types: Pods, Deployments, Services, ConfigMaps, Secrets, PersistentVolumes, and so on. These cover the general-purpose building blocks for running containerized workloads.

But the moment you operate real infrastructure, you hit the edges. You want to express:

• “This database should have three replicas with point-in-time recovery enabled” — not a Deployment
• “This TLS certificate for api.example.com should renew 30 days before expiry” — not a Secret
• “This queue consumer should scale to zero when the queue is empty” — not a HorizontalPodAutoscaler

Before CRDs (pre-2017), the only options were: use ConfigMaps as a poor substitute (no schema, no validation, no dedicated RBAC), or fork Kubernetes and add the resource natively (impractical for everyone outside the core team).

CRDs, introduced as stable in Kubernetes 1.16, solved this by letting you register a new resource type with the API server at runtime — without touching Kubernetes source code, without restarting the API server, without any special access beyond being able to create cluster-scoped resources.

The Kubernetes API: A Brief Mental Model

Before CRDs make sense, the API model needs to be clear.

  KUBERNETES API STRUCTURE

  apiVersion: apps/v1       ← API group (apps) + version (v1)
  kind: Deployment          ← resource type
  metadata:
    name: web               ← instance name
    namespace: default      ← namespace scope
  spec:
    replicas: 3             ← desired state

Every Kubernetes resource has:
– A group (e.g., apps, batch, networking.k8s.io) — or no group for core resources
– A version (e.g., v1, v1beta1)
– A kind (e.g., Deployment, Pod)
– A scope: namespaced or cluster-wide

The API server is a registry. Each group/version/kind combination maps to a Go struct that knows how to validate, store, and serve that resource type.

A CRD registers a new entry in that registry. You supply the group, version, kind, and schema. The API server handles everything else — serving it via REST, storing it in etcd, exposing it to kubectl.

What a CRD Looks Like

Here is the smallest possible CRD — it creates a new BackupPolicy resource type in the storage.example.com API group:

apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: backuppolicies.storage.example.com
spec:
  group: storage.example.com
  versions:
    - name: v1alpha1
      served: true
      storage: true
      schema:
        openAPIV3Schema:
          type: object
          properties:
            spec:
              type: object
              properties:
                schedule:
                  type: string
                retentionDays:
                  type: integer
  scope: Namespaced
  names:
    plural: backuppolicies
    singular: backuppolicy
    kind: BackupPolicy
    shortNames:
      - bp

Apply it:

kubectl apply -f backuppolicy-crd.yaml

Now create an instance:

apiVersion: storage.example.com/v1alpha1
kind: BackupPolicy
metadata:
  name: nightly
  namespace: default
spec:
  schedule: "0 2 * * *"
  retentionDays: 30

kubectl apply -f nightly-backup.yaml
kubectl get backuppolicies
kubectl get bp            # shortName works
kubectl describe bp nightly

The API server validates the spec against the schema, stores it in etcd, and returns it via all the standard API endpoints — all without a single line of custom code.

CRD vs Built-In Resource: What Is Different?

Not much, deliberately.

The last row is the critical one. When you create a Deployment, the deployment controller immediately starts managing ReplicaSets. When you create a BackupPolicy, nothing happens — until you write and deploy a controller that watches BackupPolicy objects and acts on them.

That controller + the CRD is what people call an Operator.

A Real Cluster: What You Actually See

Run this on any cluster running cert-manager, Prometheus Operator, or any other tooling:

kubectl get crds

Sample output (abbreviated):

NAME                                                  CREATED AT
certificates.cert-manager.io                          2024-11-01T08:12:00Z
certificaterequests.cert-manager.io                   2024-11-01T08:12:00Z
issuers.cert-manager.io                               2024-11-01T08:12:00Z
clusterissuers.cert-manager.io                        2024-11-01T08:12:00Z
scaledobjects.keda.sh                                 2024-11-01T08:13:00Z
scaledjobs.keda.sh                                    2024-11-01T08:13:00Z
externalsecrets.external-secrets.io                   2024-11-01T08:14:00Z
prometheuses.monitoring.coreos.com                    2024-11-01T08:15:00Z
servicemonitors.monitoring.coreos.com                 2024-11-01T08:15:00Z

Every tool that ships as a CRD-based system registers its resource types here first. The count often surprises engineers: a production cluster with a typical toolchain easily has 40–80 CRDs.

Check how many are on your cluster:

kubectl get crds --no-headers | wc -l

How the API Server Handles a CRD

When you apply a CRD, the API server does three things:

  CRD REGISTRATION FLOW

  kubectl apply -f my-crd.yaml
          │
          ▼
  1. API server validates the CRD manifest
     (is the schema valid OpenAPI v3? are names correct?)
          │
          ▼
  2. CRD stored in etcd
     (under /registry/apiextensions.k8s.io/customresourcedefinitions/)
          │
          ▼
  3. New REST endpoints activated immediately:
     GET  /apis/storage.example.com/v1alpha1/namespaces/{ns}/backuppolicies
     POST /apis/storage.example.com/v1alpha1/namespaces/{ns}/backuppolicies
     ...

From this point, any kubectl get backuppolicies or API call to those endpoints is handled exactly like a built-in resource call — the API server serves it from etcd, applies RBAC, runs admission webhooks, and returns standard JSON.

No restart required. The new endpoints appear within seconds.

The Difference Between CRD and CR

Two terms that are easily confused:

• CRD (CustomResourceDefinition) — the schema/blueprint. There is one CRD per resource type. certificates.cert-manager.io is a CRD.
• CR (Custom Resource) — an instance of a CRD. Every Certificate object you create is a custom resource. You can have thousands of CRs per CRD.

  CRD (one)          →  Custom Resource (many)
  ─────────             ─────────────────────
  certificates          web-tls           (namespace: production)
  .cert-manager.io      api-tls           (namespace: production)
                        admin-tls         (namespace: staging)
                        ...

The CRD is applied once (usually by the tool’s Helm chart). Custom resources are created by your users, your CI pipeline, or your GitOps system throughout the life of the cluster.

Where CRDs Fit in the Kubernetes Extension Model

CRDs are one of three ways to extend Kubernetes:

  KUBERNETES EXTENSION MECHANISMS

  1. CRDs + Controllers (Operators)
     Add new resource types + behavior
     → cert-manager, KEDA, Argo CD, Crossplane
     Used for: domain-specific abstractions, infrastructure management

  2. Admission Webhooks
     Intercept API requests to validate or mutate objects
     → OPA/Gatekeeper, Kyverno, Istio injection
     Used for: policy enforcement, sidecar injection, defaulting

  3. API Aggregation (AA)
     Register a fully separate API server behind the main API server
     → metrics-server, custom autoscalers
     Used for: when you need non-CRUD semantics (e.g. exec, attach, streaming)

For 95% of use cases, CRDs + controllers are the right mechanism. API aggregation is complex and only warranted for non-standard API semantics. Admission webhooks are complementary to CRDs, not an alternative.

Common Mistakes

Confusing the CRD with the controller. The CRD is just a schema registration — it does not execute code. If you apply a CRD but do not deploy its controller, creating custom resources will succeed (the API server accepts them) but nothing will happen. This catches many people the first time they try to use cert-manager by only applying the CRDs without installing the cert-manager controller.

Assuming CRD deletion is safe. Deleting a CRD deletes all custom resources of that type from etcd. There is no “are you sure?” prompt. If you delete the certificates.cert-manager.io CRD, every Certificate object in every namespace is gone.

Treating CRDs as ConfigMap replacements. Some teams store configuration in CRDs purely to get schema validation. This works, but without a controller, the custom resources are inert data. If you only need configuration storage with validation, a CRD is viable — just be explicit that there is no reconciliation loop.

Quick Reference

# List all CRDs in the cluster
kubectl get crds

# Inspect a specific CRD's schema
kubectl get crd certificates.cert-manager.io -o yaml

# List all custom resources of a type
kubectl get certificates -A

# Get details on a specific custom resource
kubectl describe certificate web-tls -n production

# Delete a CRD (WARNING: deletes all instances)
kubectl delete crd backuppolicies.storage.example.com

# Check if a CRD is established (ready to use)
kubectl get crd backuppolicies.storage.example.com \
  -o jsonpath='{.status.conditions[?(@.type=="Established")].status}'
# Returns: True

Key Takeaways

• A Kubernetes CRD registers a new resource type with the API server — no source code changes, no restart required
• Custom resources behave identically to built-in resources: kubectl, RBAC, watches, etcd, admission webhooks all work the same way
• The CRD is just the schema; a controller gives custom resources behavior — together they form an Operator
• Every production cluster running modern tooling already uses dozens of CRDs
• Deleting a CRD deletes all its instances — treat CRDs as production-critical objects

What’s Next

EP02: CRDs You Already Use makes this concrete before we go deeper — we walk through cert-manager’s Certificate, KEDA’s ScaledObject, and External Secrets’ ExternalSecret as working examples, so you understand what a well-designed CRD looks like from a user’s perspective before you design your own.

Get EP02 in your inbox when it publishes → subscribe at linuxcent.com

The post What Is a Kubernetes CRD? How Custom Resources Extend the API appeared first on Linuxcent.

Introduction

Ten years after the first commit, Kubernetes is not exciting in the way it was in 2015. That’s a compliment. The system is stable. The APIs are mature. The migrations — dockershim, PSP, cloud provider code — are behind us.

What the 1.33–1.35 cycle shows is a project focused on precision: removing edge cases, promoting long-running alpha features to stable, and making the scheduler, storage, and security model more correct rather than more powerful. That’s what a mature infrastructure platform looks like.

Here’s what happened and where the project is headed.

Kubernetes 1.33 — Sidecar Resize, In-Place Resize Beta (April 2025)

Code name: Octarine

In-Place Pod Vertical Scaling reaches Beta

After landing as alpha in 1.27, in-place pod resource resizing became beta in 1.33 — enabled by default via the InPlacePodVerticalScaling feature gate.

The capability: change CPU and memory requests/limits on a running container without terminating and restarting the pod.

# Resize a running container's CPU limit without restart
kubectl patch pod api-pod-xyz --type='json' -p='[
  {
    "op": "replace",
    "path": "/spec/containers/0/resources/requests/cpu",
    "value": "2"
  },
  {
    "op": "replace",
    "path": "/spec/containers/0/resources/limits/cpu",
    "value": "4"
  }
]'

# Verify the resize was applied
kubectl get pod api-pod-xyz -o jsonpath='{.status.containerStatuses[0].resources}'

Why this matters operationally: Before in-place resize, vertical scaling meant terminating the pod, losing in-memory state, waiting for a new pod to become ready. For databases with warm buffer pools, JVM applications with loaded heap caches, or any workload where startup cost is significant, this was a serious limitation. Vertical Pod Autoscaler (VPA) worked around it by restarting pods — acceptable for stateless workloads, problematic for stateful ones.

In 1.33, resizing also works for sidecar containers, combining two 1.32-stable features.

Sidecar Containers — Full Maturity

The first feature to formally combine sidecar and in-place resize: you can now vertically scale a service mesh proxy (Envoy sidecar) without restarting the application pod. For high-traffic services where the proxy itself becomes the CPU bottleneck, this is directly actionable.

Gateway API v1.4 (October 2025)

Gateway API continued its rapid iteration with v1.4:

BackendTLSPolicy (Standard channel): Configure TLS between the gateway and the backend service — not just TLS termination at the gateway, but end-to-end encryption:

apiVersion: gateway.networking.k8s.io/v1alpha3
kind: BackendTLSPolicy
metadata:
  name: api-backend-tls
spec:
  targetRefs:
  - group: ""
    kind: Service
    name: api-service
  validation:
    caCertificateRefs:
    - name: internal-ca
      group: ""
      kind: ConfigMap
    hostname: api.internal.corp

Gateway Client Certificate Validation: The gateway can now validate client certificates — mutual TLS for ingress traffic, not just between services.

TLSRoute to Standard: TLS routing (based on SNI, not HTTP host headers) graduated to the standard channel — enabling TCP workloads with TLS passthrough through the Gateway API model.

ListenerSet: Group multiple Gateway listeners — useful for shared infrastructure where multiple teams need to attach routes to the same gateway without managing separate Gateway resources.

Kubernetes 1.34 — Scheduler Improvements, DRA Continues (August 2025)

The 1.34 release focused on the scheduler and Dynamic Resource Allocation:

DRA structured parameters stabilization: The Dynamic Resource Allocation API matured its parameter model — resource drivers can expose structured claims that the scheduler understands, enabling topology-aware placement of GPU workloads:

apiVersion: resource.k8s.io/v1alpha3
kind: ResourceClaim
metadata:
  name: gpu-claim
spec:
  devices:
    requests:
    - name: gpu
      deviceClassName: gpu.nvidia.com
      selectors:
      - cel:
          expression: device.attributes["nvidia.com/gpu-product"].string() == "A100-SXM4-80GB"
      count: 2

Scheduler QueueingHint stable: Plugins can now tell the scheduler when to re-queue a pod for scheduling — instead of the scheduler periodically retrying all unschedulable pods, plugins signal when relevant cluster state has changed. This significantly reduces scheduler CPU consumption in large clusters with many unschedulable pods.

Fine-grained node authorization improvements: Kubelets can now be restricted from accessing Service resources they don’t need — further reducing the blast radius of a compromised kubelet.

Kubernetes 1.35 — In-Place Resize GA, Memory Limits Unlocked (December 2025)

In-Place Pod Vertical Scaling Graduates to Stable

After landing in alpha (1.27), beta (1.33), in-place resize graduated to GA in 1.35. Two significant improvements accompanied GA:

Memory limit decreases now permitted: Previously, you could increase memory limits in-place but not decrease them. The restriction existed because the kernel doesn’t immediately reclaim memory when the limit is lowered — the OOM killer would need to run. 1.35 lifts this restriction with proper handling: the kernel is instructed to reclaim, and the pod status reflects the resize progress.

Pod-Level Resources (alpha in 1.35): Specify resource requests and limits at the pod level rather than per-container — with in-place resize support. Useful for init containers and sidecar patterns where total pod resources matter more than per-container allocation.

spec:
  # Pod-level resources (alpha) — total budget for all containers
  resources:
    requests:
      cpu: "4"
      memory: "8Gi"
  containers:
  - name: application
    image: myapp:latest
    # No per-container resources; pod-level applies
  - name: log-collector
    image: fluentbit:latest
    restartPolicy: Always  # sidecar

Other 1.35 Highlights

Topology Spread Constraints improvements: Better handling of unschedulable scenarios — whenUnsatisfiable: ScheduleAnyway now has smarter fallback behavior.

VolumeAttributesClass stable: Change storage performance characteristics (IOPS, throughput) of a PersistentVolume without re-provisioning — the storage equivalent of in-place pod resize.

# Change volume IOPS without re-provisioning
kubectl patch pvc database-pvc --type='merge' -p='
  {"spec": {"volumeAttributesClassName": "high-performance"}}'

Job success policy improvements: Declare a Job successful when a subset of pods complete successfully — for distributed training jobs where not all workers need to finish.

What’s in Kubernetes 1.36 (April 22, 2026)

Kubernetes 1.36 is on track for April 22, 2026 release. Based on the enhancement tracking and KEP (Kubernetes Enhancement Proposal) pipeline, expected highlights include:

• DRA continuing toward stable
• Pod-level resources moving to beta
• Scheduler improvements for AI/ML workload placement
• Further Gateway API integration as core networking model

The project has reached a rhythm: four releases per year, each focused on advancing a predictable set of features through alpha → beta → stable. The drama of the 2019–2022 period (PSP, dockershim, API removals) is behind it.

The State of the Ecosystem in 2026

Control Plane Deployment Models

CNI Landscape

eBPF-based CNIs (Cilium, Calico in eBPF mode) are now the default recommendation for production clusters. The iptables era of Kubernetes networking is ending.

Security Stack in 2026

A hardened Kubernetes cluster in 2026 runs:

Cluster provisioning:    Cluster API + GitOps (Flux/ArgoCD)
Admission control:       Pod Security Admission (restricted) + Kyverno or OPA/Gatekeeper
Runtime security:        Falco (eBPF-based syscall monitoring)
Network security:        Cilium NetworkPolicy + Cilium Cluster Mesh for multi-cluster
Image security:          Cosign signing in CI + admission webhook for signature verification
Secret management:       External Secrets Operator → HashiCorp Vault or cloud KMS
Observability:           Prometheus + Grafana + Hubble (network flows) + OpenTelemetry

The Permanent Principles That Haven’t Changed

Looking across twelve years and 35 minor versions, some things have not changed:

The API as the universal interface: Everything in Kubernetes is a resource. This remains the most important architectural decision — it makes every tool, every controller, every GitOps system work with the same model.

Reconciliation loops: Every Kubernetes controller watches actual state and drives it toward desired state. The controller pattern from 2014 is unchanged. CRDs and Operators are just more instances of it.

Labels and selectors: The flexible grouping mechanism from 1.0 is still the primary way Kubernetes components find each other. Services find pods. HPA finds Deployments. Operators find their managed resources.

Declarative, not imperative: You describe what you want. Kubernetes figures out how to achieve and maintain it. This principle, inherited from Borg’s BCL configuration, underlies everything from Deployments to Crossplane’s cloud resource management.

What’s Coming: The Next Five Years

WebAssembly on Kubernetes: The Wasm ecosystem (wasmCloud, SpinKube) is building toward running WebAssembly workloads as first-class Kubernetes pods — near-native performance, smaller images, stronger isolation than containers. Still early, but gaining real adoption.

AI inference as infrastructure: LLM serving is becoming a cluster primitive. Tools like KServe and vLLM on Kubernetes are moving from research to production. The scheduler, resource model, and networking will continue adapting to inference workload patterns.

Confidential computing: AMD SEV, Intel TDX, and ARM CCA provide hardware-level memory encryption for pods. The RuntimeClass mechanism and ongoing kernel work are making confidential Kubernetes workloads operational rather than experimental.

Leaner distributions: k3s, k0s, Talos, and Flatcar-based minimal Kubernetes distributions are growing in adoption for edge, IoT, and resource-constrained environments. The pressure is toward smaller, more auditable control planes.

Key Takeaways

• In-place pod vertical scaling went from alpha (1.27) to stable (1.35) — live CPU and memory resize without pod restart changes the economics of stateful workload management
• Gateway API v1.4 completes the ingress replacement story: BackendTLSPolicy, client certificate validation, and TLSRoute in standard channel
• VolumeAttributesClass stable (1.35): Change storage performance in-place — the storage parallel to pod resource resize
• The eBPF era of Kubernetes networking is established: Cilium as default CNI in GKE, growing in EKS/AKS, replacing iptables-based kube-proxy
• The Kubernetes project in 2026 is focused on precision — promoting mature features to stable, reducing edge cases, improving scheduler efficiency — not adding new abstractions
• WebAssembly, confidential computing, and AI inference scheduling are the frontiers to watch

Series Wrap-Up

From 47,501 lines of Go in a 250-file GitHub commit to the operating system of the cloud — and still reconciling.

← EP07: Platform Engineering Era

Series: Kubernetes: From Borg to Platform Engineering | linuxcent.com

The post Kubernetes Today: v1.33 to v1.35, In-Place Resize GA, and What Comes Next appeared first on Linuxcent.

Introduction

By 2023, the question had shifted from “how do we run Kubernetes?” to “how do we let other engineers run their workloads on Kubernetes without becoming a bottleneck?”

This is the platform engineering problem. And it drove the tooling that defined 2023–2025: GitOps as the deployment standard, Cluster API for Kubernetes-on-Kubernetes provisioning, AI/ML workloads forcing new scheduling capabilities, and the Kubernetes project itself shedding more weight to become faster to release and operate.

GitOps: Principle Becomes Practice

GitOps as a term was coined by Weaveworks in 2017. By 2023, it was no longer a debate — it was the default deployment model for organizations running Kubernetes at scale.

The principle: the desired state of your cluster lives in Git. A controller watches the repository and reconciles the cluster state to match. Every deployment is a PR merge. The audit trail is the Git history.

Flux v2 (CNCF graduated) and ArgoCD (CNCF incubating) became the two dominant implementations:

# Flux: GitRepository + Kustomization
apiVersion: source.toolkit.fluxcd.io/v1
kind: GitRepository
metadata:
  name: production-config
  namespace: flux-system
spec:
  interval: 1m
  url: https://github.com/org/k8s-config
  ref:
    branch: main
---
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: production-apps
  namespace: flux-system
spec:
  interval: 10m
  path: ./clusters/production
  prune: true          # Remove resources deleted from Git
  sourceRef:
    kind: GitRepository
    name: production-config
  healthChecks:
  - apiVersion: apps/v1
    kind: Deployment
    name: api
    namespace: production

The prune: true behavior is critical: resources deleted from Git are deleted from the cluster. This is what makes GitOps a security control — unknown resources that aren’t in Git get removed. No more accumulation of forgotten test deployments, rogue debug pods, or unauthorized configuration changes that outlive the engineer who made them.

ArgoCD’s Application model added a UI, synchronization policies, and multi-cluster management:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: production-api
  namespace: argocd
spec:
  project: production
  source:
    repoURL: https://github.com/org/apps
    targetRevision: HEAD
    path: api/production
  destination:
    server: https://kubernetes.default.svc
    namespace: api
  syncPolicy:
    automated:
      prune: true
      selfHeal: true    # Revert manual kubectl changes
    syncOptions:
    - CreateNamespace=true

The selfHeal: true option is where GitOps becomes enforceable: any manual change made with kubectl is automatically reverted within the sync interval. For compliance-sensitive environments, this is a configuration drift prevention control.

Cluster API: Kubernetes Managing Kubernetes

Cluster API (cluster-sigs/cluster-api) flipped the usual model: instead of using tools like Terraform or Ansible to provision Kubernetes clusters, Cluster API lets you manage Kubernetes clusters as Kubernetes resources — using a management cluster to provision and manage workload clusters.

# Create a new Kubernetes cluster as a Kubernetes resource
apiVersion: cluster.x-k8s.io/v1beta1
kind: Cluster
metadata:
  name: workload-cluster-prod
spec:
  clusterNetwork:
    pods:
      cidrBlocks: ["192.168.0.0/16"]
  infrastructureRef:
    apiVersion: infrastructure.cluster.x-k8s.io/v1beta2
    kind: AWSCluster
    name: workload-cluster-prod
  controlPlaneRef:
    apiVersion: controlplane.cluster.x-k8s.io/v1beta1
    kind: KubeadmControlPlane
    name: workload-cluster-prod-control-plane

Cluster API reconciliation handles cluster provisioning, scaling, upgrades, and deletion — all through the Kubernetes API, with all the tooling (RBAC, audit logging, GitOps integration) that entails. Multi-cluster platform teams could now manage hundreds of workload clusters from a single management cluster.

Kubernetes 1.28 — Sidecar Containers Alpha (August 2023)

Sidecar containers had been a Kubernetes pattern since 2015 — a helper container in the same pod as the main application. But there was no native sidecar lifecycle management. Sidecars were just regular init containers or additional containers, which meant:
– Init container sidecars ran before the application and had to block until they succeeded
– Regular container sidecars had no ordering guarantees at startup
– At pod termination, sidecars could die before the application finished draining

1.28 introduced native sidecar support: a new restartPolicy field for init containers:

spec:
  initContainers:
  - name: log-collector
    image: fluentbit:latest
    restartPolicy: Always    # This makes it a sidecar
    # Starts before main containers, stays running, stops after main containers exit
  containers:
  - name: application
    image: myapp:latest

A sidecar container (init container with restartPolicy: Always):
– Starts before application containers
– Stays running throughout the pod lifecycle
– Terminates automatically after all main containers exit
– Restarts if it crashes (unlike regular init containers)

This solved the service mesh sidecar problem: Istio and Linkerd injected Envoy proxies as regular containers, leading to race conditions where the proxy hadn’t started when the application tried to make outbound connections. Native sidecar lifecycle guarantees the proxy is ready before the application starts.

Also in 1.28:
– Retroactive default StorageClass assignment: Existing PVCs without a StorageClass assignment get the default applied retroactively — useful for migrations
– Non-graceful node shutdown stable: Handle node power failures without manual pod cleanup
– Recovery from volume expansion failure: Previously, a failed volume expansion left the PVC in a broken state; 1.28 introduced a mechanism to recover

AI/ML Workloads Force New Kubernetes Capabilities

The LLM wave of 2023 drove GPU workloads onto Kubernetes at a scale and urgency the project hadn’t anticipated. Running LLM inference on Kubernetes required solving problems that CPU-centric cluster scheduling hadn’t encountered:

GPU topology awareness: Inference across multiple GPUs requires GPUs connected by NVLink or on the same PCIe switch, not arbitrary GPUs from different nodes or different PCIe buses. The Dynamic Resource Allocation API (1.26 alpha) was designed exactly for this.

Fractional GPU allocation: NVIDIA’s time-slicing and MIG (Multi-Instance GPU) allow multiple pods to share a single GPU. The GPU operator (NVIDIA) manages this at the node level:

# Check GPU resources visible to Kubernetes
kubectl get nodes -o custom-columns=\
  "NODE:.metadata.name,GPU:.status.allocatable.nvidia\.com/gpu"
# NODE       GPU
# gpu-node-1   8
# gpu-node-2   8

Batch scheduling for training jobs: Training runs require all workers to start simultaneously — a single missing GPU makes the entire job stall. The Kubernetes Job API doesn’t guarantee this. Projects like Volcano (CNCF incubating) and Kueue (Kubernetes SIG Scheduling) added gang scheduling: a job only starts when all requested resources are available.

# Kueue: queue AI training jobs with resource quotas
apiVersion: kueue.x-k8s.io/v1beta1
kind: ClusterQueue
metadata:
  name: gpu-queue
spec:
  namespaceSelector: {}
  resourceGroups:
  - coveredResources: ["nvidia.com/gpu", "cpu", "memory"]
    flavors:
    - name: a100-80gb
      resources:
      - name: nvidia.com/gpu
        nominalQuota: 16

Kubernetes 1.29 — Sidecar to Beta, Load Balancer IP Mode (December 2023)

• Sidecar containers beta: The lifecycle semantics were refined based on 1.28 alpha feedback
• Load balancer IP mode alpha: Distinguish between load balancers that use virtual IPs (kube-proxy handles the traffic) vs. those that handle traffic directly (no need for kube-proxy rules) — important for eBPF-based load balancers
• ReadWriteOncePod volume access stable

Kubernetes 1.30 — Structured Authorization Config (April 2024)

• Structured authorization configuration beta: Define multiple authorization webhooks with explicit ordering, failure modes, and connection settings — replacing the flat --authorization-mode flag
• Sidecar containers beta continues
• Node memory swap support beta: Allow pods to use swap memory — controversial but necessary for workloads with bursty memory patterns that prefer using swap over OOM kill

# Node with swap enabled — kubelet config
kind: KubeletConfiguration
memorySwap:
  swapBehavior: LimitedSwap

The swap support feature reversed a long-standing Kubernetes hard stance: swap was disabled since 1.0 because its interaction with Kubernetes memory accounting was unpredictable. The 1.30 approach adds proper accounting and policies.

Kubernetes 1.31 — Cloud Provider Code Removal Complete (August 2024)

1.31 marked the completion of the cloud provider code removal — the 1.5 million line migration that had been running since 1.26. Core binaries are 40% smaller. The API server, controller manager, and scheduler no longer contain vendor-specific code.

Also in 1.31:
– Persistent Volume health monitor stable
– AppArmor support stable: AppArmor profiles for pods using the native Kubernetes field (not annotations)
– Traffic distribution for Services beta: Express topology preferences for Service routing (prefer local node, prefer same zone)

# Traffic distribution: prefer endpoints in the same zone
apiVersion: v1
kind: Service
metadata:
  name: api
spec:
  trafficDistribution: PreferClose
  selector:
    app: api
  ports:
  - port: 80
    targetPort: 8080

Kubernetes 1.32 — Sidecar Stable, DRA Beta (December 2024)

• Sidecar containers stable: After nearly a decade of workarounds, the sidecar pattern is a first-class Kubernetes primitive
• Dynamic Resource Allocation beta: GPU and specialized hardware scheduling ready for production evaluation
• Job API improvements: Success and failure policies for indexed jobs — granular control over batch workload behavior
• Custom Resource field selectors: Filter CRDs on arbitrary fields — making large CRD-based systems more efficient to query

Crossplane: Kubernetes as the Control Plane for Everything

Crossplane (CNCF graduated) extended the Kubernetes API model beyond the cluster itself. Using CRDs and controllers, Crossplane lets you manage cloud resources (RDS databases, S3 buckets, VPCs, IAM roles) as Kubernetes resources — provisioned, updated, and deleted through the Kubernetes API.

# Crossplane: provision an RDS PostgreSQL instance as a Kubernetes resource
apiVersion: database.aws.crossplane.io/v1beta1
kind: RDSInstance
metadata:
  name: production-db
spec:
  forProvider:
    region: us-east-1
    dbInstanceClass: db.r6g.xlarge
    masterUsername: admin
    engine: postgres
    engineVersion: "15"
    allocatedStorage: 100
    multiAZ: true
  writeConnectionSecretsToRef:
    name: production-db-credentials
    namespace: production

For platform teams, Crossplane means a single control plane — the Kubernetes API — for both compute workloads and cloud infrastructure. GitOps tools (Flux, ArgoCD) manage both.

Key Takeaways

• GitOps (Flux, ArgoCD) became the production deployment standard — not for ideological reasons, but because the audit trail, drift detection, and self-healing properties solve real operational and compliance problems
• Cluster API made Kubernetes cluster lifecycle (provisioning, upgrades, deletion) a Kubernetes-native operation — the same API, tooling, and audit trail
• Native sidecar containers (1.28 alpha → 1.32 stable) finally resolved the lifecycle ordering problem that service meshes and log collectors had worked around for years
• AI/ML workloads drove new scheduling capabilities (DRA, gang scheduling via Kueue/Volcano) and made GPU topology awareness a first-class concern
• Crossplane generalized the Kubernetes API model to cloud infrastructure — the cluster is now a control plane for everything, not just containers

What’s Next

← EP06: The Runtime Reckoning | EP08: Kubernetes Today →

Series: Kubernetes: From Borg to Platform Engineering | linuxcent.com

The post The Platform Engineering Era: GitOps, AI Workloads, and Leaner Kubernetes (2023–2025) appeared first on Linuxcent.