Kubernetes Ecosystem: From User to Contributor, Episode 2
← EP01: MicroK8s Explained · EP02: Minikube · EP03: k3s vs MicroK8s vs Minikube →

11 min read

TL;DR

• What is Minikube? A tool that runs a single-node Kubernetes cluster inside a VM or a container on your local machine — the oldest and most widely adopted “local Kubernetes” tool in the ecosystem
• Unlike MicroK8s’s bare-metal snap install, Minikube’s default drivers isolate the cluster inside a VM (VirtualBox, HyperKit, Hyper-V, KVM2) or a Docker container — a deliberate isolation trade-off, not an accident
• minikube addons, minikube profile, and multi-node support let you run several named clusters side by side, each with its own driver and Kubernetes version
• LoadBalancer services don’t resolve to anything real on their own — minikube tunnel or minikube service are required, and this trips up almost everyone the first time
• The VM overhead that makes Minikube heavier than MicroK8s is also what makes it a more faithful stand-in for a real cloud node, particularly for testing kernel-adjacent behavior
• Contribution opportunity: feature parity across Minikube’s own driver list is uneven, and closing specific gaps there is a well-scoped, achievable contribution

The Big Picture

MICROK8S: BARE-METAL SNAP                   MINIKUBE: ISOLATED VM/CONTAINER
──────────────────────────                   ──────────────────────────────
Host OS
  └── microk8s snap                         Host OS
        ├── kubelet                           └── Driver (VirtualBox / KVM2 /
        ├── kube-apiserver                        HyperKit / Docker / Podman)
        ├── containerd                              └── VM or container
        └── Dqlite                                        ├── kubelet
                                                            ├── kube-apiserver
No VM boundary — cluster                                   ├── etcd
runs directly on the host                                  └── containerd
kernel and network stack
                                              Full isolation boundary between
                                              cluster and host — closer to how
                                              a real cloud node actually looks

What is Minikube? It’s the tool that popularized “just run a Kubernetes cluster on your laptop” — a single command that provisions a VM or container, installs a full Kubernetes control plane and node inside it, and hands you a working kubectl context. The isolation boundary that VM makes MicroK8s’s bare-metal install avoid is the entire point: Minikube trades startup speed and resource overhead for a cluster that behaves more like a real, separate node — the same control-plane/node split covered in detail in this site’s Kubernetes history series, just shrunk down to fit on a laptop.

The Driver Model: How Minikube Actually Runs Your Cluster

Minikube doesn’t run Kubernetes directly on your host. It provisions a driver-specific environment first, then runs Kubernetes inside that:

$ minikube start --driver=docker
😄  minikube v1.32.0 on Darwin 14.2
✨  Using the docker driver based on user configuration
👍  Starting control plane node minikube in cluster minikube
🚜  Pulling base image ...
🔥  Creating docker container (CPUs=2, Memory=4000MB) ...
🐳  Preparing Kubernetes v1.28.3 on Docker 24.0.7 ...
🔎  Verifying Kubernetes components...
🌟  Enabled addons: default-storageclass, storage-provisioner
🏄  Done! kubectl is now configured to use "minikube" cluster

$ minikube status
minikube
type: Control Plane
host: Running
kubelet: Running
apiserver: Running
kubeconfig: Configured

The --driver flag is the real decision point. docker/podman drivers run the cluster as a container, which is fast to start but shares the host kernel — you don’t get true kernel-level isolation. virtualbox/hyperkit/kvm2/hyperv drivers run a full VM, which is slower to start (30–90 seconds, versus 10–20 for the container driver) but gives the cluster its own kernel, its own network namespace, and behavior much closer to an actual cloud instance.

Addons and Profiles: Minikube’s Answer to Multi-Environment Testing

$ minikube addons list
|-----------------------------|----------|--------------|
| ADDON NAME                  | PROFILE  | STATUS       |
|-----------------------------|----------|--------------|
| ingress                     | minikube | disabled     |
| metrics-server              | minikube | disabled     |
| dashboard                   | minikube | disabled     |
| registry                    | minikube | disabled     |

$ minikube addons enable ingress
🔎  Verifying ingress addon...
🌟  The 'ingress' addon is enabled

# Run a second, independent cluster on a different Kubernetes version
$ minikube start -p old-version --kubernetes-version=v1.26.0
$ minikube profile list
|----------|-----------|---------|--------------|------|
| Profile  | VM Driver | Runtime | IP           | Ver  |
|----------|-----------|---------|--------------|------|
| minikube | docker    | docker  | 192.168.49.2 | v1.28.3 |
| old-version | docker | docker  | 192.168.58.2 | v1.26.0 |

Profiles are Minikube’s way of running multiple, fully independent clusters side by side — useful for testing an upgrade path or comparing behavior across Kubernetes versions without tearing anything down. MicroK8s has no equivalent to this; it’s a genuine Minikube differentiator, not just a different flavor of the same feature.

Where the VM Overhead Actually Shows Up

The isolation Minikube provides isn’t free, and it shows up in three concrete places: startup time (a VM driver cold-start is measured in tens of seconds, not the few seconds a bare-metal snap install takes), memory floor (a VM needs to reserve memory for its own kernel and init system before Kubernetes gets any of it), and CI runners specifically — many hosted CI environments (GitHub Actions’ standard runners, for example) don’t support nested virtualization, which rules out VM drivers entirely and forces the docker driver, quietly giving up the isolation benefit that was the reason to pick Minikube over MicroK8s in the first place.

Networking Quirks: LoadBalancer Services and minikube tunnel

This is the single most common point of confusion for anyone coming from a real cloud cluster:

$ kubectl expose deployment web --type=LoadBalancer --port=80
service/web exposed

$ kubectl get svc web
NAME   TYPE           CLUSTER-IP     EXTERNAL-IP   PORT(S)
web    LoadBalancer   10.96.34.201   <pending>     80:31234/TCP
#                                    ^^^^^^^^^ stays pending forever —
#                                    there's no cloud load balancer to provision one

Minikube has no cloud provider to actually satisfy a LoadBalancer request. Two ways to actually reach the service:

# Option 1: minikube tunnel — creates a real route to LoadBalancer services,
# must stay running in a foreground terminal the whole time
$ minikube tunnel
✅  Tunnel successfully started

# Option 2: minikube service — opens the service in a browser via NodePort,
# no LoadBalancer semantics, but doesn't require a background process
$ minikube service web --url
http://192.168.49.2:31234

minikube tunnel is the closer match to real LoadBalancer behavior, but it’s a foreground process that silently stops working if the terminal closes or the machine sleeps — a frequent source of “it worked five minutes ago” confusion.

Production Gotchas

Nested virtualization isn’t available everywhere. Many hosted CI runners and some cloud dev environments don’t expose the CPU virtualization extensions Minikube’s VM drivers need — you’ll get a driver failure that looks like a Minikube bug but is actually a host capability gap. Falling back to --driver=docker works, but changes the isolation guarantees you were relying on.

The docker driver shares your host’s Docker daemon resource limits. If your host Docker Desktop is capped at 4GB, that’s a hard ceiling for everything running inside the Minikube container too — VM drivers get their own explicit memory allocation instead.

minikube tunnel dying silently is the most common “why can’t I reach my LoadBalancer” support question. It doesn’t reliably surface a clear error when it stops — check minikube tunnel‘s own terminal output before assuming the Kubernetes side is broken.

Addon behavior differs meaningfully by driver. The ingress addon’s interaction with host networking is different between a VM driver (which gets its own IP on a virtual network) and the docker driver (which shares the host’s Docker network) — a setup that works on one driver doesn’t automatically work identically on another.

Quick Reference

minikube start --driver=<docker|virtualbox|hyperkit|kvm2|hyperv>
minikube status                    # cluster health
minikube addons list                # available and enabled add-ons
minikube addons enable <name>       # enable one
minikube profile list               # all named clusters
minikube start -p <name>            # start/create a named profile
minikube tunnel                     # real LoadBalancer routing (foreground)
minikube service <name> --url       # NodePort-based access, no LB semantics
minikube delete -p <name>           # tear down a specific profile
minikube ssh                        # shell into the cluster's VM/container

Contribution Opportunity: Closing Minikube’s Driver Feature-Parity Gaps

The limitation: Minikube supports over a dozen drivers (docker, podman, virtualbox, hyperkit, kvm2, hyperv, vfkit, qemu, and more), and features don’t land on all of them at the same time or with the same fidelity. GPU passthrough, specific CNI plugin support, and certain addon behaviors work reliably on some drivers and only partially — or not at all — on others. A user picking a driver based on their OS often has no easy way to know upfront which features they’re implicitly giving up.

Why it’s hard to fix: Each driver wraps a fundamentally different underlying technology (a type-2 hypervisor, a container runtime, a different hypervisor API per OS), so a feature that’s straightforward on one driver can require an entirely separate implementation path on another — this isn’t a matter of one team finishing a checklist, it’s N different integration surfaces that each need their own maintainer attention, and Minikube’s driver maintainers are a much smaller, more fragmented group than the core Kubernetes maintainers.

What a contribution-shaped fix looks like: The achievable starting point isn’t “add GPU support to every driver” — it’s picking one specific, well-documented gap (say, a particular addon’s known behavior difference on hyperv versus kvm2), reproducing it precisely, and either fixing the driver-specific code path in kubernetes/minikube or, just as valuably, contributing a clear compatibility matrix to the project’s docs so the next person doesn’t discover the gap by trial and error. Minikube’s own GitHub issues are full of exactly these driver-specific reports sitting unresolved for lack of someone who reproduces and narrows them down.

Key Takeaways

• Minikube isolates the cluster inside a VM or container, trading startup speed and resource overhead for isolation closer to a real cloud node
• Profiles let you run multiple independent, differently-versioned clusters side by side — a genuine capability MicroK8s doesn’t have
• LoadBalancer services need minikube tunnel or minikube service — there’s no cloud provider underneath to satisfy the request automatically
• Driver choice has real consequences: VM drivers need nested virtualization support that not every host or CI runner provides, and feature parity across drivers is uneven
• The clearest contribution opportunity is narrowing and documenting (or fixing) specific driver feature-parity gaps — achievable without deep hypervisor expertise

What’s Next

EP01 and EP02 covered MicroK8s and Minikube individually. EP03 puts them head-to-head against k3s — the third major lightweight Kubernetes option — on the criteria that actually matter when picking one: resource footprint, HA story, and how much you’re willing to trade control for convenience.

Next: EP03 — k3s vs MicroK8s vs Minikube: Which Lightweight Kubernetes Fits Your Use Case

Get EP03 in your inbox when it publishes → linuxcent.com/subscribe

The post Minikube: Local Kubernetes Done Right — and Where It Breaks Down appeared first on Linuxcent.

Kubernetes Ecosystem: From User to Contributor, Episode 1
← Kubernetes: From Borg to Platform Engineering · EP01: MicroK8s Explained · All Kubernetes Ecosystem Episodes →

11 min read

TL;DR

• What is MicroK8s? A full, CNCF-conformant Kubernetes cluster packaged into a single snap package — install and have a running cluster in under 60 seconds, no separate container runtime or CNI install required
• Built by Canonical for edge, IoT, and ephemeral CI clusters, where minimal footprint and single-command lifecycle matter more than fine-grained tuning
• Add-ons (dns, storage, ingress, metallb, cilium, gpu) replace what you’d otherwise hand-install and wire together yourself
• High-availability mode uses Dqlite (a distributed, Raft-backed SQLite) instead of etcd — a deliberate design choice with a real operational trade-off
• The same “just works” install model that makes MicroK8s fast to stand up also hides some of the control-plane tuning knobs production operators expect from etcd
• Contribution opportunity: Dqlite’s operational tooling is years behind etcd’s — a real, specific, achievable gap, covered in depth below

The Big Picture

TRADITIONAL KUBERNETES INSTALL              MICROK8S INSTALL
─────────────────────────────               ──────────────────
Install a container runtime                 $ sudo snap install microk8s --classic
Install kubelet, kubeadm, kubectl            $ microk8s status --wait-ready
Stand up and configure etcd
kubeadm init, join workers                          │
Install a CNI plugin                                ▼
Install an ingress controller                One snap install =
Install a storage provisioner                kubelet + kube-apiserver + kube-scheduler
Wire it all together, hope                   + kube-controller-manager + containerd
the versions are compatible                  + Dqlite + CNI (via add-on)
                                              — one running cluster, one command

What is MicroK8s? It’s Canonical’s single-binary distribution that packages an entire Kubernetes control plane and node into one snap package, so snap install microk8s produces a running, CNCF-conformant cluster without a separate container runtime, etcd install, or CNI setup step. The trade for that speed is control: the same bundling that removes a dozen manual steps also removes a dozen places you’d normally tune something.

How MicroK8s Fits an Entire Cluster Into One Snap

A snap package is a self-contained, confined Linux application bundle — MicroK8s ships its own containerd, its own CNI defaults, and its own datastore inside that bundle, rather than expecting the host to provide them.

$ sudo snap install microk8s --classic
microk8s (1.29/stable) v1.29.1 from Canonical✓ installed

$ microk8s status --wait-ready
microk8s is running
high-availability: no
  datastore master nodes: 127.0.0.1:19001
  datastore standby nodes: none
addons:
  enabled:
    ha-cluster            # always on — this is the Dqlite HA layer, even for a single node
  disabled:
    dns                   # ← you enable what you need, nothing runs by default beyond core
    ingress
    storage
    ...

--classic confinement is required because MicroK8s needs broader host access than a strictly-confined snap allows — it manages network interfaces, iptables/nftables rules, and mounts. This is the first place production operators coming from a bare-metal kubeadm install get surprised: MicroK8s’s snap confinement model means some host interactions happen through paths a traditional install never touches, and debugging network issues sometimes means understanding snap’s confinement boundaries, not just Kubernetes networking.

The Add-on Model: How MicroK8s Replaces a Day of Cluster Bootstrapping

Everything beyond the bare control plane is an add-on, enabled with one command:

$ microk8s enable dns storage ingress
Infer repository core for addon dns
Enabling DNS
...
DNS is enabled
Infer repository core for addon storage
Enabling default storage class
...
Storage is enabled
Infer repository core for addon ingress
Ingress controller is enabled

$ microk8s kubectl get pods -A
NAMESPACE     NAME                                      READY   STATUS    RESTARTS
kube-system   coredns-864597b5fd-x7k2p                  1/1     Running   0
kube-system   hostpath-provisioner-5c65c9c74f-j9qmz      1/1     Running   0
ingress       nginx-ingress-microk8s-controller-abcde    1/1     Running   0

Each add-on is a maintained, version-pinned bundle — enabling ingress doesn’t pull the latest ingress-nginx release, it pulls whatever version that MicroK8s release channel has validated. That’s the same trade seen everywhere else in this tool: predictability and speed, at the cost of being slightly behind upstream and unable to mix-and-match component versions the way a hand-built cluster can.

The cilium add-on is worth calling out specifically for platform teams already standardizing on eBPF-based networking elsewhere — it replaces MicroK8s’s default CNI with Cilium, giving you the same eBPF-enforced network policy model covered in the TC eBPF episode of the eBPF series, without a separate Cilium install process.

Dqlite Instead of etcd: MicroK8s’s Most Debated Design Choice

For high availability, upstream Kubernetes distributions almost universally reach for etcd — a mature, Raft-based key-value store with over a decade of production hardening. MicroK8s uses Dqlite instead: a distributed SQLite built by Canonical, also Raft-based, but SQL-native rather than a plain key-value store.

# Convert a single node into a 3-node HA cluster
$ microk8s add-node
From the target node, run:
microk8s join 10.0.1.15:25000/abc123... --worker

$ microk8s status
microk8s is running
high-availability: yes
  datastore master nodes: 10.0.1.10:19001 10.0.1.12:19001 10.0.1.15:19001
  datastore standby nodes: none

The reasoning behind Dqlite is architectural: Canonical wanted a datastore that could also serve their other projects (LXD, for instance) with a SQL interface, not just Kubernetes’ key-value needs, and one they could tightly control the release cadence of rather than depending on the etcd project’s own timeline. That’s a legitimate engineering decision — but it means MicroK8s’s HA story runs on a datastore with a small fraction of etcd’s operational track record.

Where MicroK8s Actually Runs in Production

MicroK8s’s real fit is narrower than “any Kubernetes workload”: edge and IoT deployments where a device needs a full, conformant cluster with no external dependencies (a factory sensor gateway, a retail point-of-sale cluster); CI/CD pipelines that need a disposable, fast-booting cluster per test run; and single-node developer or demo environments where Minikube’s VM overhead isn’t wanted.

It’s a weaker fit for large multi-tenant production clusters where teams already have deep etcd operational expertise, need fine-grained control-plane component versioning, or run at a scale where Dqlite’s newer, less-battle-tested Raft implementation is a harder sell to a risk-averse platform team.

Production Gotchas

Dqlite HA needs an odd number of nodes, same as etcd — but the community knowledge base is much thinner. A 2-node or 4-node Dqlite cluster has the identical split-brain risk etcd has at even node counts. The difference is that when something goes wrong, etcd has ten years of Stack Overflow answers and postmortems; Dqlite has a fraction of that.

microk8s kubectl and a separately-installed kubectl are not automatically the same context. Running both on one host is a common source of “why isn’t my change showing up” confusion — always check which kubeconfig each one is actually pointed at.

Add-on versions lag upstream by design. If a CVE fix or a new feature lands in upstream ingress-nginx or Cilium, MicroK8s’s bundled add-on version won’t have it until the next MicroK8s release validates it. Don’t assume microk8s enable X gets you the latest X.

Classic confinement means MicroK8s can conflict with other host-level network tooling. Firewalls, VPN clients, or other snap-confined networking tools on the same host can produce iptables/nftables rule conflicts that look like a Kubernetes networking bug but are actually a confinement-boundary interaction.

Quick Reference

microk8s status --wait-ready       # cluster health, HA state, enabled add-ons
microk8s enable <addon>            # dns, storage, ingress, metallb, cilium, gpu, ...
microk8s disable <addon>           # remove an add-on
microk8s kubectl <args>            # bundled kubectl, uses MicroK8s's own kubeconfig
microk8s add-node                  # generate a join token for HA/worker expansion
microk8s join <token>               # join a node using that token
microk8s inspect                   # generate a full diagnostic tarball for support/debugging
microk8s remove-node <node>         # remove a node from an HA cluster

Contribution Opportunity: Dqlite’s Missing Decade of Tooling

The limitation: etcd operators have etcdctl endpoint status, etcdctl endpoint health, mature snapshot/restore tooling, and a decade of documented failure-mode runbooks. Dqlite’s CLI surface for diagnosing a struggling Raft cluster — checking leader state, log index lag between nodes, or safely restoring from a snapshot after a prolonged partition — is meaningfully thinner. When a MicroK8s HA node fails to rejoin cleanly, there’s far less prior art to lean on than for the etcd equivalent.

Why it’s hard to fix: This isn’t a bug sitting in an issue tracker waiting for a quick patch. It’s a maturity gap that comes from Dqlite being a newer, narrower-scope project (built primarily to serve Canonical’s own products) maintained by a much smaller team than the decade of enterprise-scale usage that produced etcd’s tooling. Closing that gap is a sustained, multi-quarter documentation-and-tooling effort, not a single PR — and it competes for the Dqlite maintainers’ time against Canonical’s own product roadmap, which doesn’t automatically prioritize the broader Kubernetes community’s operational wishlist.

What a contribution-shaped fix looks like: Two concrete, achievable starting points that don’t require deep Raft-internals expertise: (1) a dqlite-side diagnostic command mirroring etcdctl endpoint status — human-readable leader/term/log-index output — contributed to canonical/go-dqlite; or (2) reproducing and documenting specific HA failure scenarios (node rejoin after a prolonged network partition, recovery from a minority-node failure) as runbooks in canonical/microk8s‘s own documentation repo, with the exact recovery commands verified against a real reproduction. The second option in particular is the kind of contribution an engineer who’s actually operated MicroK8s in production is uniquely positioned to make — it needs careful reproduction and clear writing, not kernel-level systems expertise.

Key Takeaways

• MicroK8s packages a full, CNCF-conformant Kubernetes cluster into a single snap install — no separate runtime, etcd, or CNI setup required
• The add-on model trades version flexibility for predictability: what you enable is validated and bundled, not necessarily the latest upstream release
• Dqlite replaces etcd for HA — a legitimate architectural choice, but one with far less operational tooling and community track record behind it
• MicroK8s’s real fit is edge/IoT, ephemeral CI clusters, and single-node dev environments — not large multi-tenant production clusters with deep etcd expertise already in place
• The clearest contribution opportunity here doesn’t require Raft internals — it requires operating MicroK8s in production long enough to hit a real failure mode and documenting it precisely

What’s Next

MicroK8s trades control for a near-instant single-command cluster. EP02 looks at Minikube — the other dominant “local Kubernetes” tool, built on a different trade-off entirely: a full VM (or container-based driver) per cluster instead of a bare-metal snap install, and where that heavier model actually earns its overhead.

Next: EP02 — Minikube: Local Kubernetes Done Right — and Where It Breaks Down

Get EP02 in your inbox when it publishes → linuxcent.com/subscribe

The post MicroK8s Explained: The Single-Binary Kubernetes for Edge and IoT appeared first on Linuxcent.

Introduction

Ten years after the first commit, Kubernetes is not exciting in the way it was in 2015. That’s a compliment. The system is stable. The APIs are mature. The migrations — dockershim, PSP, cloud provider code — are behind us.

What the 1.33–1.35 cycle shows is a project focused on precision: removing edge cases, promoting long-running alpha features to stable, and making the scheduler, storage, and security model more correct rather than more powerful. That’s what a mature infrastructure platform looks like.

Here’s what happened and where the project is headed.

Kubernetes 1.33 — Sidecar Resize, In-Place Resize Beta (April 2025)

Code name: Octarine

In-Place Pod Vertical Scaling reaches Beta

After landing as alpha in 1.27, in-place pod resource resizing became beta in 1.33 — enabled by default via the InPlacePodVerticalScaling feature gate.

The capability: change CPU and memory requests/limits on a running container without terminating and restarting the pod.

# Resize a running container's CPU limit without restart
kubectl patch pod api-pod-xyz --type='json' -p='[
  {
    "op": "replace",
    "path": "/spec/containers/0/resources/requests/cpu",
    "value": "2"
  },
  {
    "op": "replace",
    "path": "/spec/containers/0/resources/limits/cpu",
    "value": "4"
  }
]'

# Verify the resize was applied
kubectl get pod api-pod-xyz -o jsonpath='{.status.containerStatuses[0].resources}'

Why this matters operationally: Before in-place resize, vertical scaling meant terminating the pod, losing in-memory state, waiting for a new pod to become ready. For databases with warm buffer pools, JVM applications with loaded heap caches, or any workload where startup cost is significant, this was a serious limitation. Vertical Pod Autoscaler (VPA) worked around it by restarting pods — acceptable for stateless workloads, problematic for stateful ones.

In 1.33, resizing also works for sidecar containers, combining two 1.32-stable features.

Sidecar Containers — Full Maturity

The first feature to formally combine sidecar and in-place resize: you can now vertically scale a service mesh proxy (Envoy sidecar) without restarting the application pod. For high-traffic services where the proxy itself becomes the CPU bottleneck, this is directly actionable.

Gateway API v1.4 (October 2025)

Gateway API continued its rapid iteration with v1.4:

BackendTLSPolicy (Standard channel): Configure TLS between the gateway and the backend service — not just TLS termination at the gateway, but end-to-end encryption:

apiVersion: gateway.networking.k8s.io/v1alpha3
kind: BackendTLSPolicy
metadata:
  name: api-backend-tls
spec:
  targetRefs:
  - group: ""
    kind: Service
    name: api-service
  validation:
    caCertificateRefs:
    - name: internal-ca
      group: ""
      kind: ConfigMap
    hostname: api.internal.corp

Gateway Client Certificate Validation: The gateway can now validate client certificates — mutual TLS for ingress traffic, not just between services.

TLSRoute to Standard: TLS routing (based on SNI, not HTTP host headers) graduated to the standard channel — enabling TCP workloads with TLS passthrough through the Gateway API model.

ListenerSet: Group multiple Gateway listeners — useful for shared infrastructure where multiple teams need to attach routes to the same gateway without managing separate Gateway resources.

Kubernetes 1.34 — Scheduler Improvements, DRA Continues (August 2025)

The 1.34 release focused on the scheduler and Dynamic Resource Allocation:

DRA structured parameters stabilization: The Dynamic Resource Allocation API matured its parameter model — resource drivers can expose structured claims that the scheduler understands, enabling topology-aware placement of GPU workloads:

apiVersion: resource.k8s.io/v1alpha3
kind: ResourceClaim
metadata:
  name: gpu-claim
spec:
  devices:
    requests:
    - name: gpu
      deviceClassName: gpu.nvidia.com
      selectors:
      - cel:
          expression: device.attributes["nvidia.com/gpu-product"].string() == "A100-SXM4-80GB"
      count: 2

Scheduler QueueingHint stable: Plugins can now tell the scheduler when to re-queue a pod for scheduling — instead of the scheduler periodically retrying all unschedulable pods, plugins signal when relevant cluster state has changed. This significantly reduces scheduler CPU consumption in large clusters with many unschedulable pods.

Fine-grained node authorization improvements: Kubelets can now be restricted from accessing Service resources they don’t need — further reducing the blast radius of a compromised kubelet.

Kubernetes 1.35 — In-Place Resize GA, Memory Limits Unlocked (December 2025)

In-Place Pod Vertical Scaling Graduates to Stable

After landing in alpha (1.27), beta (1.33), in-place resize graduated to GA in 1.35. Two significant improvements accompanied GA:

Memory limit decreases now permitted: Previously, you could increase memory limits in-place but not decrease them. The restriction existed because the kernel doesn’t immediately reclaim memory when the limit is lowered — the OOM killer would need to run. 1.35 lifts this restriction with proper handling: the kernel is instructed to reclaim, and the pod status reflects the resize progress.

Pod-Level Resources (alpha in 1.35): Specify resource requests and limits at the pod level rather than per-container — with in-place resize support. Useful for init containers and sidecar patterns where total pod resources matter more than per-container allocation.

spec:
  # Pod-level resources (alpha) — total budget for all containers
  resources:
    requests:
      cpu: "4"
      memory: "8Gi"
  containers:
  - name: application
    image: myapp:latest
    # No per-container resources; pod-level applies
  - name: log-collector
    image: fluentbit:latest
    restartPolicy: Always  # sidecar

Other 1.35 Highlights

Topology Spread Constraints improvements: Better handling of unschedulable scenarios — whenUnsatisfiable: ScheduleAnyway now has smarter fallback behavior.

VolumeAttributesClass stable: Change storage performance characteristics (IOPS, throughput) of a PersistentVolume without re-provisioning — the storage equivalent of in-place pod resize.

# Change volume IOPS without re-provisioning
kubectl patch pvc database-pvc --type='merge' -p='
  {"spec": {"volumeAttributesClassName": "high-performance"}}'

Job success policy improvements: Declare a Job successful when a subset of pods complete successfully — for distributed training jobs where not all workers need to finish.

What’s in Kubernetes 1.36 (April 22, 2026)

Kubernetes 1.36 is on track for April 22, 2026 release. Based on the enhancement tracking and KEP (Kubernetes Enhancement Proposal) pipeline, expected highlights include:

• DRA continuing toward stable
• Pod-level resources moving to beta
• Scheduler improvements for AI/ML workload placement
• Further Gateway API integration as core networking model

The project has reached a rhythm: four releases per year, each focused on advancing a predictable set of features through alpha → beta → stable. The drama of the 2019–2022 period (PSP, dockershim, API removals) is behind it.

The State of the Ecosystem in 2026

Control Plane Deployment Models

CNI Landscape

eBPF-based CNIs (Cilium, Calico in eBPF mode) are now the default recommendation for production clusters. The iptables era of Kubernetes networking is ending.

Security Stack in 2026

A hardened Kubernetes cluster in 2026 runs:

Cluster provisioning:    Cluster API + GitOps (Flux/ArgoCD)
Admission control:       Pod Security Admission (restricted) + Kyverno or OPA/Gatekeeper
Runtime security:        Falco (eBPF-based syscall monitoring)
Network security:        Cilium NetworkPolicy + Cilium Cluster Mesh for multi-cluster
Image security:          Cosign signing in CI + admission webhook for signature verification
Secret management:       External Secrets Operator → HashiCorp Vault or cloud KMS
Observability:           Prometheus + Grafana + Hubble (network flows) + OpenTelemetry

The Permanent Principles That Haven’t Changed

Looking across twelve years and 35 minor versions, some things have not changed:

The API as the universal interface: Everything in Kubernetes is a resource. This remains the most important architectural decision — it makes every tool, every controller, every GitOps system work with the same model.

Reconciliation loops: Every Kubernetes controller watches actual state and drives it toward desired state. The controller pattern from 2014 is unchanged. CRDs and Operators are just more instances of it.

Labels and selectors: The flexible grouping mechanism from 1.0 is still the primary way Kubernetes components find each other. Services find pods. HPA finds Deployments. Operators find their managed resources.

Declarative, not imperative: You describe what you want. Kubernetes figures out how to achieve and maintain it. This principle, inherited from Borg’s BCL configuration, underlies everything from Deployments to Crossplane’s cloud resource management.

What’s Coming: The Next Five Years

WebAssembly on Kubernetes: The Wasm ecosystem (wasmCloud, SpinKube) is building toward running WebAssembly workloads as first-class Kubernetes pods — near-native performance, smaller images, stronger isolation than containers. Still early, but gaining real adoption.

AI inference as infrastructure: LLM serving is becoming a cluster primitive. Tools like KServe and vLLM on Kubernetes are moving from research to production. The scheduler, resource model, and networking will continue adapting to inference workload patterns.

Confidential computing: AMD SEV, Intel TDX, and ARM CCA provide hardware-level memory encryption for pods. The RuntimeClass mechanism and ongoing kernel work are making confidential Kubernetes workloads operational rather than experimental.

Leaner distributions: k3s, k0s, Talos, and Flatcar-based minimal Kubernetes distributions are growing in adoption for edge, IoT, and resource-constrained environments. The pressure is toward smaller, more auditable control planes.

Key Takeaways

• In-place pod vertical scaling went from alpha (1.27) to stable (1.35) — live CPU and memory resize without pod restart changes the economics of stateful workload management
• Gateway API v1.4 completes the ingress replacement story: BackendTLSPolicy, client certificate validation, and TLSRoute in standard channel
• VolumeAttributesClass stable (1.35): Change storage performance in-place — the storage parallel to pod resource resize
• The eBPF era of Kubernetes networking is established: Cilium as default CNI in GKE, growing in EKS/AKS, replacing iptables-based kube-proxy
• The Kubernetes project in 2026 is focused on precision — promoting mature features to stable, reducing edge cases, improving scheduler efficiency — not adding new abstractions
• WebAssembly, confidential computing, and AI inference scheduling are the frontiers to watch

Series Wrap-Up

From 47,501 lines of Go in a 250-file GitHub commit to the operating system of the cloud — and still reconciling.

← EP07: Platform Engineering Era

Series: Kubernetes: From Borg to Platform Engineering | linuxcent.com

The post Kubernetes Today: v1.33 to v1.35, In-Place Resize GA, and What Comes Next appeared first on Linuxcent.

Introduction

By 2023, the question had shifted from “how do we run Kubernetes?” to “how do we let other engineers run their workloads on Kubernetes without becoming a bottleneck?”

This is the platform engineering problem. And it drove the tooling that defined 2023–2025: GitOps as the deployment standard, Cluster API for Kubernetes-on-Kubernetes provisioning, AI/ML workloads forcing new scheduling capabilities, and the Kubernetes project itself shedding more weight to become faster to release and operate.

GitOps: Principle Becomes Practice

GitOps as a term was coined by Weaveworks in 2017. By 2023, it was no longer a debate — it was the default deployment model for organizations running Kubernetes at scale.

The principle: the desired state of your cluster lives in Git. A controller watches the repository and reconciles the cluster state to match. Every deployment is a PR merge. The audit trail is the Git history.

Flux v2 (CNCF graduated) and ArgoCD (CNCF incubating) became the two dominant implementations:

# Flux: GitRepository + Kustomization
apiVersion: source.toolkit.fluxcd.io/v1
kind: GitRepository
metadata:
  name: production-config
  namespace: flux-system
spec:
  interval: 1m
  url: https://github.com/org/k8s-config
  ref:
    branch: main
---
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: production-apps
  namespace: flux-system
spec:
  interval: 10m
  path: ./clusters/production
  prune: true          # Remove resources deleted from Git
  sourceRef:
    kind: GitRepository
    name: production-config
  healthChecks:
  - apiVersion: apps/v1
    kind: Deployment
    name: api
    namespace: production

The prune: true behavior is critical: resources deleted from Git are deleted from the cluster. This is what makes GitOps a security control — unknown resources that aren’t in Git get removed. No more accumulation of forgotten test deployments, rogue debug pods, or unauthorized configuration changes that outlive the engineer who made them.

ArgoCD’s Application model added a UI, synchronization policies, and multi-cluster management:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: production-api
  namespace: argocd
spec:
  project: production
  source:
    repoURL: https://github.com/org/apps
    targetRevision: HEAD
    path: api/production
  destination:
    server: https://kubernetes.default.svc
    namespace: api
  syncPolicy:
    automated:
      prune: true
      selfHeal: true    # Revert manual kubectl changes
    syncOptions:
    - CreateNamespace=true

The selfHeal: true option is where GitOps becomes enforceable: any manual change made with kubectl is automatically reverted within the sync interval. For compliance-sensitive environments, this is a configuration drift prevention control.

Cluster API: Kubernetes Managing Kubernetes

Cluster API (cluster-sigs/cluster-api) flipped the usual model: instead of using tools like Terraform or Ansible to provision Kubernetes clusters, Cluster API lets you manage Kubernetes clusters as Kubernetes resources — using a management cluster to provision and manage workload clusters.

# Create a new Kubernetes cluster as a Kubernetes resource
apiVersion: cluster.x-k8s.io/v1beta1
kind: Cluster
metadata:
  name: workload-cluster-prod
spec:
  clusterNetwork:
    pods:
      cidrBlocks: ["192.168.0.0/16"]
  infrastructureRef:
    apiVersion: infrastructure.cluster.x-k8s.io/v1beta2
    kind: AWSCluster
    name: workload-cluster-prod
  controlPlaneRef:
    apiVersion: controlplane.cluster.x-k8s.io/v1beta1
    kind: KubeadmControlPlane
    name: workload-cluster-prod-control-plane

Cluster API reconciliation handles cluster provisioning, scaling, upgrades, and deletion — all through the Kubernetes API, with all the tooling (RBAC, audit logging, GitOps integration) that entails. Multi-cluster platform teams could now manage hundreds of workload clusters from a single management cluster.

Kubernetes 1.28 — Sidecar Containers Alpha (August 2023)

Sidecar containers had been a Kubernetes pattern since 2015 — a helper container in the same pod as the main application. But there was no native sidecar lifecycle management. Sidecars were just regular init containers or additional containers, which meant:
– Init container sidecars ran before the application and had to block until they succeeded
– Regular container sidecars had no ordering guarantees at startup
– At pod termination, sidecars could die before the application finished draining

1.28 introduced native sidecar support: a new restartPolicy field for init containers:

spec:
  initContainers:
  - name: log-collector
    image: fluentbit:latest
    restartPolicy: Always    # This makes it a sidecar
    # Starts before main containers, stays running, stops after main containers exit
  containers:
  - name: application
    image: myapp:latest

A sidecar container (init container with restartPolicy: Always):
– Starts before application containers
– Stays running throughout the pod lifecycle
– Terminates automatically after all main containers exit
– Restarts if it crashes (unlike regular init containers)

This solved the service mesh sidecar problem: Istio and Linkerd injected Envoy proxies as regular containers, leading to race conditions where the proxy hadn’t started when the application tried to make outbound connections. Native sidecar lifecycle guarantees the proxy is ready before the application starts.

Also in 1.28:
– Retroactive default StorageClass assignment: Existing PVCs without a StorageClass assignment get the default applied retroactively — useful for migrations
– Non-graceful node shutdown stable: Handle node power failures without manual pod cleanup
– Recovery from volume expansion failure: Previously, a failed volume expansion left the PVC in a broken state; 1.28 introduced a mechanism to recover

AI/ML Workloads Force New Kubernetes Capabilities

The LLM wave of 2023 drove GPU workloads onto Kubernetes at a scale and urgency the project hadn’t anticipated. Running LLM inference on Kubernetes required solving problems that CPU-centric cluster scheduling hadn’t encountered:

GPU topology awareness: Inference across multiple GPUs requires GPUs connected by NVLink or on the same PCIe switch, not arbitrary GPUs from different nodes or different PCIe buses. The Dynamic Resource Allocation API (1.26 alpha) was designed exactly for this.

Fractional GPU allocation: NVIDIA’s time-slicing and MIG (Multi-Instance GPU) allow multiple pods to share a single GPU. The GPU operator (NVIDIA) manages this at the node level:

# Check GPU resources visible to Kubernetes
kubectl get nodes -o custom-columns=\
  "NODE:.metadata.name,GPU:.status.allocatable.nvidia\.com/gpu"
# NODE       GPU
# gpu-node-1   8
# gpu-node-2   8

Batch scheduling for training jobs: Training runs require all workers to start simultaneously — a single missing GPU makes the entire job stall. The Kubernetes Job API doesn’t guarantee this. Projects like Volcano (CNCF incubating) and Kueue (Kubernetes SIG Scheduling) added gang scheduling: a job only starts when all requested resources are available.

# Kueue: queue AI training jobs with resource quotas
apiVersion: kueue.x-k8s.io/v1beta1
kind: ClusterQueue
metadata:
  name: gpu-queue
spec:
  namespaceSelector: {}
  resourceGroups:
  - coveredResources: ["nvidia.com/gpu", "cpu", "memory"]
    flavors:
    - name: a100-80gb
      resources:
      - name: nvidia.com/gpu
        nominalQuota: 16

Kubernetes 1.29 — Sidecar to Beta, Load Balancer IP Mode (December 2023)

• Sidecar containers beta: The lifecycle semantics were refined based on 1.28 alpha feedback
• Load balancer IP mode alpha: Distinguish between load balancers that use virtual IPs (kube-proxy handles the traffic) vs. those that handle traffic directly (no need for kube-proxy rules) — important for eBPF-based load balancers
• ReadWriteOncePod volume access stable

Kubernetes 1.30 — Structured Authorization Config (April 2024)

• Structured authorization configuration beta: Define multiple authorization webhooks with explicit ordering, failure modes, and connection settings — replacing the flat --authorization-mode flag
• Sidecar containers beta continues
• Node memory swap support beta: Allow pods to use swap memory — controversial but necessary for workloads with bursty memory patterns that prefer using swap over OOM kill

# Node with swap enabled — kubelet config
kind: KubeletConfiguration
memorySwap:
  swapBehavior: LimitedSwap

The swap support feature reversed a long-standing Kubernetes hard stance: swap was disabled since 1.0 because its interaction with Kubernetes memory accounting was unpredictable. The 1.30 approach adds proper accounting and policies.

Kubernetes 1.31 — Cloud Provider Code Removal Complete (August 2024)

1.31 marked the completion of the cloud provider code removal — the 1.5 million line migration that had been running since 1.26. Core binaries are 40% smaller. The API server, controller manager, and scheduler no longer contain vendor-specific code.

Also in 1.31:
– Persistent Volume health monitor stable
– AppArmor support stable: AppArmor profiles for pods using the native Kubernetes field (not annotations)
– Traffic distribution for Services beta: Express topology preferences for Service routing (prefer local node, prefer same zone)

# Traffic distribution: prefer endpoints in the same zone
apiVersion: v1
kind: Service
metadata:
  name: api
spec:
  trafficDistribution: PreferClose
  selector:
    app: api
  ports:
  - port: 80
    targetPort: 8080

Kubernetes 1.32 — Sidecar Stable, DRA Beta (December 2024)

• Sidecar containers stable: After nearly a decade of workarounds, the sidecar pattern is a first-class Kubernetes primitive
• Dynamic Resource Allocation beta: GPU and specialized hardware scheduling ready for production evaluation
• Job API improvements: Success and failure policies for indexed jobs — granular control over batch workload behavior
• Custom Resource field selectors: Filter CRDs on arbitrary fields — making large CRD-based systems more efficient to query

Crossplane: Kubernetes as the Control Plane for Everything

Crossplane (CNCF graduated) extended the Kubernetes API model beyond the cluster itself. Using CRDs and controllers, Crossplane lets you manage cloud resources (RDS databases, S3 buckets, VPCs, IAM roles) as Kubernetes resources — provisioned, updated, and deleted through the Kubernetes API.

# Crossplane: provision an RDS PostgreSQL instance as a Kubernetes resource
apiVersion: database.aws.crossplane.io/v1beta1
kind: RDSInstance
metadata:
  name: production-db
spec:
  forProvider:
    region: us-east-1
    dbInstanceClass: db.r6g.xlarge
    masterUsername: admin
    engine: postgres
    engineVersion: "15"
    allocatedStorage: 100
    multiAZ: true
  writeConnectionSecretsToRef:
    name: production-db-credentials
    namespace: production

For platform teams, Crossplane means a single control plane — the Kubernetes API — for both compute workloads and cloud infrastructure. GitOps tools (Flux, ArgoCD) manage both.

Key Takeaways

• GitOps (Flux, ArgoCD) became the production deployment standard — not for ideological reasons, but because the audit trail, drift detection, and self-healing properties solve real operational and compliance problems
• Cluster API made Kubernetes cluster lifecycle (provisioning, upgrades, deletion) a Kubernetes-native operation — the same API, tooling, and audit trail
• Native sidecar containers (1.28 alpha → 1.32 stable) finally resolved the lifecycle ordering problem that service meshes and log collectors had worked around for years
• AI/ML workloads drove new scheduling capabilities (DRA, gang scheduling via Kueue/Volcano) and made GPU topology awareness a first-class concern
• Crossplane generalized the Kubernetes API model to cloud infrastructure — the cluster is now a control plane for everything, not just containers

What’s Next

← EP06: The Runtime Reckoning | EP08: Kubernetes Today →

Series: Kubernetes: From Borg to Platform Engineering | linuxcent.com

The post The Platform Engineering Era: GitOps, AI Workloads, and Leaner Kubernetes (2023–2025) appeared first on Linuxcent.

Introduction

By 2018, Kubernetes had won the orchestration market. The question was no longer “which orchestrator?” — it was “how do we run complex workloads on it, and how do we do it safely?”

The 2018–2020 period is defined by three parallel tracks: the Operator pattern maturing into a serious engineering discipline, the service mesh debate consuming enormous community energy, and the security model evolving from “trust everything in the cluster” toward something resembling defense-in-depth.

The OperatorHub Era

The Operator pattern, introduced by CoreOS engineers in 2016, reached critical mass in 2018–2019. In November 2018, Red Hat launched OperatorHub.io — a registry for Kubernetes Operators covering databases (PostgreSQL, MongoDB, CockroachDB), messaging (Kafka, RabbitMQ), monitoring (Prometheus), and more.

The Operator SDK (Red Hat, 2018) gave teams a framework for building Operators in Go, Ansible, or Helm — lowering the barrier from “you need to write a Kubernetes controller from scratch” to “fill in the reconciliation logic.”

The maturity model for Operators was codified into five levels:

Most production Operators in 2019 were at Level 1–2. Getting to Level 3+ required encoding significant domain knowledge — the kind that previously lived in a senior database administrator’s head.

Kubernetes 1.11 — CoreDNS Default, Load Balancing Stable (June 2018)

• CoreDNS replaced kube-dns as the default DNS provider. CoreDNS is plugin-based — you can extend it for custom DNS resolution logic (split DNS, external name resolution, DNS-based service discovery for non-Kubernetes services)
• IPVS-based kube-proxy stable: The load balancing mode for Services switched from iptables to IPVS (IP Virtual Server), enabling O(1) service routing instead of O(n) iptables rule traversal — critical at scale
• TLS bootstrapping stable: Kubelet automatic certificate rotation — kubelets no longer needed manual certificate management

The IPVS kube-proxy mode is a good example of a performance improvement that also has security implications. iptables rules degrade linearly with rule count; at 10,000+ services, iptables becomes a performance and debuggability problem. IPVS uses a hash table — O(1) lookups regardless of service count.

Kubernetes 1.12 — 1.13: Amazon EKS, Runtime Security (September–December 2018)

Amazon EKS Goes GA (June 2018)

Amazon EKS became generally available in June 2018. This was significant not just for AWS customers but for the entire ecosystem: EKS’s launch meant every major cloud provider now had a production-grade managed Kubernetes offering.

EKS’s initial release was deliberately limited — managed control plane, self-managed worker nodes. This contrasted with GKE’s more automated approach, and the community noticed. GKE had been running managed Kubernetes longer, and it showed in feature completeness.

1.12 (September 2018)

• RuntimeClass alpha: A mechanism to specify which container runtime to use for a pod — containerd, gVisor, Kata Containers. The foundation for confidential computing workloads where you want hardware-isolated containers
• RBAC delegation: Service accounts could now grant RBAC permissions they themselves held — enabling Operators to manage RBAC for the applications they deploy
• Volume snapshot alpha: Create point-in-time snapshots of PersistentVolumes — the beginning of Kubernetes-native backup primitives

1.13 (December 2018)

• kubeadm graduates to GA: The cluster bootstrapping tool was now stable and recommended for production
• CoreDNS stable
• CSI stable: Storage drivers could be shipped entirely out of tree

Kubernetes 1.14 — Windows Containers Go Stable (March 2019)

Windows Server container support graduated to stable in 1.14. For the first time, Kubernetes clusters could run Windows workloads as first-class citizens — .NET Framework applications, IIS, SQL Server containers alongside Linux-based microservices.

The implementation required significant work: Windows containers have different networking models, different filesystem semantics, and different process models than Linux containers. Making them a first-class Kubernetes citizen meant handling all of those differences in the node components.

Also in 1.14:
– PersistentVolume and StorageClass improvements
– kubectl improvements: kubectl diff — show what would change before applying a manifest

The PodSecurityPolicy Problem

PodSecurityPolicy (PSP) was alpha in Kubernetes 1.3, beta in 1.8, and would remain in beta until it was deprecated in 1.21. It was simultaneously the most important security primitive in Kubernetes and the most broken.

PSP let administrators define what a pod was allowed to do:

apiVersion: policy/v1beta1
kind: PodSecurityPolicy
metadata:
  name: restricted
spec:
  privileged: false
  allowPrivilegeEscalation: false
  requiredDropCapabilities:
    - ALL
  volumes:
    - 'configMap'
    - 'emptyDir'
    - 'projected'
    - 'secret'
    - 'downwardAPI'
    - 'persistentVolumeClaim'
  hostNetwork: false
  hostIPC: false
  hostPID: false
  runAsUser:
    rule: MustRunAsNonRoot
  seLinux:
    rule: RunAsAny
  supplementalGroups:
    rule: MustRunAs
    ranges:
      - min: 1
        max: 65535
  fsGroup:
    rule: MustRunAs
    ranges:
      - min: 1
        max: 65535
  readOnlyRootFilesystem: false

The problem: the admission mechanism was confusing, the UX was hostile, and the authorization model (who could use which PSP) led to privilege escalation paths that were non-obvious. Many teams either disabled PSP entirely or created a permissive policy that made it functionally useless.

The community would spend years working toward a replacement. In 2021 it was deprecated; in 1.25 (2022) it was removed. The replacement — Pod Security Admission — is discussed in EP05.

Kubernetes 1.15 — 1.17: Custom Resource Maturity (2019)

1.15 (June 2019)

• CRDs continue maturing: Structural schemas, pruning of unknown fields — making CRDs behave more like first-class API types
• Kustomize integrated into kubectl: Template-free Kubernetes configuration customization. Where Helm uses Go templates, Kustomize uses overlays — a base configuration plus environment-specific patches

# kustomization.yaml — base + production overlay
bases:
  - ../../base
patches:
  - deployment-replicas.yaml
  - resource-limits.yaml
configMapGenerator:
  - name: app-config
    literals:
      - ENV=production

1.16 (September 2019)

• CRDs graduate to GA (apps/v1, not extensions/v1beta1)
• Admission webhooks stable: Validating and mutating webhooks that intercept every API request. This is the foundation for OPA/Gatekeeper, Kyverno, and all policy-as-code enforcement in Kubernetes

The admission webhook framework’s graduation to stable in 1.16 was more significant than it appeared. It meant that any security policy engine — OPA/Gatekeeper, Kyverno, Styra, etc. — could now enforce policies on any Kubernetes resource creation or modification, using a stable, documented API.

• Removal of several deprecated beta APIs: extensions/v1beta1 Deployments, DaemonSets, ReplicaSets — a preview of the more aggressive API cleanup that would come in 1.22

1.17 (December 2019)

• Volume snapshots beta
• Cloud Provider labels stable

OPA/Gatekeeper: Policy as Code Enters the Mainstream

Open Policy Agent (OPA) + Gatekeeper emerged as the policy engine of choice for Kubernetes in 2019. Gatekeeper uses the admission webhook framework to intercept API requests and evaluate them against Rego policies:

# Deny containers running as root
package kubernetes.admission

deny[msg] {
  input.request.kind.kind == "Pod"
  container := input.request.object.spec.containers[_]
  container.securityContext.runAsUser == 0
  msg := sprintf("Container %v must not run as root", [container.name])
}

The OPA/Gatekeeper model represented a shift in security thinking: instead of configuring security at the cluster level, you codify security policy in a language (Rego) and enforce it uniformly across all admission requests. Policies can be tested, versioned, and reviewed like code.

Kubernetes 1.18 — Topology-Aware Routing, Immutability (March 2020)

• Topology-aware service routing alpha: Route service traffic to endpoints in the same zone/node as the caller — reducing cross-zone data transfer costs and latency
• Immutable ConfigMaps and Secrets alpha: Mark a ConfigMap or Secret as immutable — the API server rejects updates, preventing accidental mutation of configuration that applications have already loaded
• IngressClass: A mechanism to specify which Ingress controller should handle an Ingress resource — enabling multiple ingress controllers in the same cluster

# Immutable secret — once set, cannot be changed
apiVersion: v1
kind: Secret
metadata:
  name: db-credentials
immutable: true
data:
  password: dGhpcyBpcyBhIHRlc3Q=

The Falco Adoption Wave

CNCF-donated Falco (originated by Sysdig) became the standard tool for Kubernetes runtime security in this period. Falco uses eBPF probes or kernel modules to monitor syscalls and generate alerts based on rules:

# Falco rule: detect shell spawned in a container
- rule: Terminal shell in container
  desc: A shell was spawned in a container
  condition: >
    spawned_process and container and
    shell_procs and proc.tty != 0
  output: >
    A shell was spawned in a container
    (user=%user.name container=%container.name
     shell=%proc.name parent=%proc.pname)
  priority: WARNING

Falco addressed the gap that PodSecurityPolicy couldn’t: admission-time policy prevents known-bad configurations from running, but it can’t detect a compromise that happens at runtime — a shell spawned by an exploited web application, for example.

The Service Mesh Exhaustion

By 2019, the service mesh landscape was producing more overhead than value for many teams. Istio’s operational complexity — its control plane components, its sidecar injection model, its frequent breaking changes between versions — burned teams that adopted it early.

The community questions were real: do you actually need mTLS between every service in your cluster? Is the operational cost of a service mesh worth the security benefit for every organization?

Linkerd 2.x (Buoyant) positioned itself as the lightweight alternative — simpler to operate, less configuration surface, Rust-based proxy instead of Envoy. For teams that wanted the security benefit (mTLS) without the complexity cost, Linkerd 2.x was often the better choice.

The honest answer in 2019-2020: service meshes were the right architecture for organizations with hundreds of services and dedicated platform teams. For most organizations, they were complexity that outpaced the threat model.

Key Takeaways

• The Operator pattern matured from a pattern into an engineering discipline with tooling (Operator SDK), a registry (OperatorHub), and a capability maturity model
• EKS going GA completed the managed Kubernetes trifecta — every major cloud provider was now committed
• CRDs graduating to stable in 1.16 was the foundation for everything built on Kubernetes extensibility — Operators, policy engines, GitOps tools
• Admission webhooks graduating to stable enabled the policy-as-code ecosystem (OPA/Gatekeeper, Kyverno) — the only viable alternative to PSP’s broken model
• Falco established runtime security as a distinct discipline from admission-time policy enforcement
• Service mesh adoption was real but the complexity cost was frequently underestimated; many teams that adopted Istio in 2018-2019 spent 2019-2020 managing it

What’s Next

← EP03: Enterprise Awakening | EP05: Security Hardens →

Series: Kubernetes: From Borg to Platform Engineering | linuxcent.com

The post The Operator Era: Stateful Workloads, Service Mesh, and the Cloud-Native Stack (2018–2020) appeared first on Linuxcent.

Introduction

By the end of 2016, engineers were running Kubernetes in production. Not as an experiment — in production, handling real traffic. And that’s where the real gaps became visible.

The 2016–2018 period is the era when Kubernetes grew up. RBAC went stable. CRDs replaced the fragile ThirdPartyResource hack. The major cloud providers launched managed services. Helm became the standard for packaging. And the security posture, which had been an afterthought in the Borg-derived model, started getting serious attention.

Kubernetes 1.6 — The RBAC Milestone (March 2017)

Kubernetes 1.6 is the release that made enterprise Kubernetes possible. The headline feature: RBAC (Role-Based Access Control) promoted to beta, enabled by default.

Before RBAC, Kubernetes had attribute-based access control (ABAC) — a flat policy file on the API server that required a restart to change. It worked, but it was operationally painful and offered no granularity at the namespace level.

RBAC introduced four objects:
– Role: A set of permissions scoped to a namespace
– ClusterRole: A set of permissions cluster-wide or reusable across namespaces
– RoleBinding: Assigns a Role to a user/group/service account in a namespace
– ClusterRoleBinding: Assigns a ClusterRole cluster-wide

# Example: read-only access to pods in the dev namespace
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  namespace: dev
  name: pod-reader
rules:
- apiGroups: [""]
  resources: ["pods"]
  verbs: ["get", "watch", "list"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: read-pods
  namespace: dev
subjects:
- kind: User
  name: alice
  apiGroup: rbac.authorization.k8s.io
roleRef:
  kind: Role
  name: pod-reader
  apiGroup: rbac.authorization.k8s.io

Also in 1.6:
– etcd v3 as default: Better performance, watch semantics, and transaction support
– Node Authorization mode: Kubelets can now only access secrets and pods bound to their own node — a critical lateral movement restriction
– Audit logging (alpha): API server logs every request — who did what, to which resource, at what time
– Scale: Tested to 5,000 nodes per cluster

The node authorization mode deserves more attention than it typically gets. Before 1.6, a compromised kubelet could read all secrets in the cluster. Node authorization restricted the kubelet to only the secrets it needed for pods scheduled on that node. This single change dramatically reduced the blast radius of a node compromise.

Kubernetes 1.7 — Custom Resource Definitions (June 2017)

The most significant architectural decision in Kubernetes history after the initial design: ThirdPartyResources (TPRs) were replaced with CustomResourceDefinitions (CRDs).

TPRs were a fragile mechanism introduced in 1.2 that let users define custom API types. They had serious limitations: no schema validation, no versioning, data loss bugs, and poor upgrade behavior. In 1.7, they were replaced with CRDs.

CRDs are what make the Kubernetes API extension model work. They let you define new resource types that the API server stores and serves, with optional schema validation via OpenAPI v3 schemas, version conversion, and admission webhook integration.

apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: databases.stable.example.com
spec:
  group: stable.example.com
  versions:
  - name: v1
    served: true
    storage: true
    schema:
      openAPIV3Schema:
        type: object
        properties:
          spec:
            type: object
            properties:
              size:
                type: string
              version:
                type: string
  scope: Namespaced
  names:
    plural: databases
    singular: database
    kind: Database

CRDs enabled the entire Operator ecosystem that would define the next phase of Kubernetes. Without stable, schema-validated custom resources, you can’t build reliable controllers on top of them.

Also in 1.7:
– Secrets encryption at rest (alpha): Finally, secrets stored in etcd could be encrypted with AES-CBC or AES-GCM
– Network Policy promoted to stable: CNI plugins implementing NetworkPolicy could now enforce pod-level ingress/egress rules
– API aggregation layer: Extend the Kubernetes API with custom API servers — the foundation for metrics-server and other API extensions

Kubernetes 1.8 — RBAC Goes Stable (September 2017)

RBAC graduated to stable in 1.8. This was the point of no return for enterprise adoption. Security teams could now enforce least-privilege on Kubernetes API access with a documented, stable API.

Key additions:
– Storage Classes stable: Dynamic volume provisioning — request a PersistentVolume and have the underlying storage (EBS, GCE PD, NFS) automatically provisioned
– Workloads API (apps/v1beta2): Deployments, ReplicaSets, DaemonSets, and StatefulSets all moved under a unified API group, signaling they were heading toward stable

The admission webhook framework — which would become the foundation for policy enforcement tools like OPA/Gatekeeper — was also being refined in this period.

The Cloud Provider Moment (2017–2018)

October 2017: Docker Surrenders

At DockerCon Europe in October 2017, Docker Inc. announced that Docker Enterprise Edition would ship with Kubernetes support alongside Docker Swarm. This was, effectively, Docker Inc. conceding the orchestration market to Kubernetes. Swarm remained available, but the message was clear: Kubernetes was the production standard.

October 2017: Microsoft Previews AKS

Microsoft previewed Azure Kubernetes Service at DockerCon Europe. The managed Kubernetes race was on.

November 2017: Amazon Announces EKS

At AWS re:Invent 2017, Amazon announced Elastic Kubernetes Service. The three major cloud providers — Google (GKE, running since 2014), Microsoft (AKS), and Amazon (EKS) — were all committed to managed Kubernetes.

For enterprise buyers, this was the signal they needed. Kubernetes was no longer a bet on an experimental technology — it was the supported, managed offering from every major cloud provider.

Kubernetes 1.9 — Workloads API Stable (December 2017)

The Workloads API (apps/v1) went stable in 1.9. This matters because it locked in the API contract for Deployments, ReplicaSets, DaemonSets, and StatefulSets. Infrastructure built on these APIs would not break on upgrades.

# apps/v1 Deployment — the stable form that operators rely on
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
spec:
  replicas: 3
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:1.14.2
        ports:
        - containerPort: 80
        resources:
          requests:
            memory: "64Mi"
            cpu: "250m"
          limits:
            memory: "128Mi"
            cpu: "500m"

Also in 1.9:
– Windows container support moved to beta — actual Windows Server 2016 nodes in a cluster
– CoreDNS available as an alternative to kube-dns: A more extensible, plugin-based DNS server that would replace kube-dns as the default in 1.11

Kubernetes 1.10 — Storage, Auth, and Scale (March 2018)

1.10 continued the enterprise hardening:
– CSI (Container Storage Interface) beta: A standardized interface between Kubernetes and storage providers. Before CSI, storage drivers were compiled into the kubelet binary. CSI moved them out-of-tree, allowing storage vendors to ship their own drivers without waiting for a Kubernetes release
– External credential providers (alpha): Authenticate against external systems (cloud IAM, HashiCorp Vault) for kubeconfig credentials
– Node problem detector stable: Detect and report node-level problems (kernel deadlocks, corrupted file systems) as Kubernetes events and node conditions

The CSI transition was one of the most important infrastructure decisions of this period. It decoupled storage driver development from the Kubernetes release cycle — a necessary step for cloud providers to ship storage integrations rapidly and independently.

The Istio Announcement and Service Mesh Wars (May 2017)

Google and IBM announced Istio in May 2017 — a service mesh that layered mTLS, traffic management, and observability on top of existing Kubernetes deployments without changing application code.

Istio’s architecture: sidecar proxies (Envoy) injected into every pod, managed by a control plane. Every service-to-service call passes through the sidecar, enabling:
– Mutual TLS between services (zero-trust networking at the service layer)
– Fine-grained traffic control (canary releases, circuit breaking, retries)
– Distributed tracing and metrics

Linkerd (from Buoyant) had been working on the same problem since 2016. The two projects would compete for the “service mesh standard” throughout 2017–2019.

The service mesh conversation was fundamentally a security architecture conversation: how do you enforce mutual authentication and encryption between services in a Kubernetes cluster without requiring application developers to implement it?

CoreOS Acquisition and the Operator Pattern (2018)

In January 2018, Red Hat acquired CoreOS for $250 million. CoreOS had contributed two things that would permanently shape Kubernetes:

1. The Operator Pattern (introduced by CoreOS engineers Brandon Philips and Josh Wood in 2016): An Operator is a custom controller that uses CRDs to manage the lifecycle of complex, stateful applications. The etcd Operator (CoreOS’s own) was the first — it automated etcd cluster creation, scaling, backup, and failure recovery. The pattern generalized: a Prometheus Operator, a PostgreSQL Operator, a Kafka Operator.

The Operator pattern is the answer to the question “how do you encode operational knowledge into software?” A human operator knows how to deploy, scale, backup, and recover a database. An Operator codifies that knowledge into a controller loop.

# Operator pattern: watch CRD → reconcile → manage application
CRD (EtcdCluster) → Operator Controller watches → creates/updates Pods, Services, Snapshots

2. etcd: The distributed key-value store that backs the Kubernetes control plane. CoreOS built and maintained etcd. Red Hat acquiring CoreOS meant that the company maintaining Kubernetes’s most critical dependency (after the kernel) was now inside the Red Hat/IBM orbit.

Helm 2 and the Charts Ecosystem

By 2017–2018, Helm had become the de facto package manager for Kubernetes. The public Helm chart repository hosted hundreds of charts — databases (PostgreSQL, MySQL, Redis), monitoring (Prometheus, Grafana), ingress controllers (nginx), CI/CD tools (Jenkins, GitLab Runner).

Helm 2 introduced Tiller — a server-side component that managed release state in the cluster. Tiller became the most criticized security decision in the Kubernetes ecosystem: Tiller ran with cluster-admin privileges by default, meaning any user who could reach Tiller’s gRPC endpoint could do anything in the cluster.

Security teams hated Tiller. The Helm team addressed it in Helm 3 (2019) by removing Tiller entirely and storing release state as Kubernetes Secrets instead.

Key Takeaways

• RBAC going stable in 1.8 was the single most important security event in early Kubernetes history — it gave enterprises the access control model they needed for production
• CRDs replacing TPRs in 1.7 enabled the entire Operator ecosystem that would define the next phase of Kubernetes
• Docker Inc.’s October 2017 announcement that it would support Kubernetes in Docker EE effectively ended the container orchestration wars
• The three major cloud providers (GKE, AKS, EKS) all standardizing on managed Kubernetes drove enterprise adoption faster than any feature announcement could
• The Operator pattern — Kubernetes controllers that encode operational knowledge — emerged from CoreOS and became the standard model for managing complex stateful applications
• Helm filled a real gap but Tiller’s cluster-admin model was a security debt the community had to repay in Helm 3

What’s Next

← EP02: The Container Wars | EP04: The Operator Era →

Series: Kubernetes: From Borg to Platform Engineering | linuxcent.com

The post Enterprise Awakening: RBAC, CRDs, Cloud Providers, and Helm Goes Mainstream (2016–2018) appeared first on Linuxcent.

Introduction

Three orchestration systems entered the arena in 2015. Only one would still matter three years later.

Docker had created the container revolution. Now everyone needed to run containers at scale, and three camps formed around three very different philosophies. Understanding why Kubernetes won — and how close it came to not winning — explains most of the design choices that still shape Kubernetes today.

The State of Container Orchestration in 2014

When Kubernetes made its public debut at DockerCon 2014, it entered a space that didn’t yet have a name. “Container orchestration” wasn’t a category. It was a problem people had started to feel but not yet articulate.

Three approaches emerged nearly simultaneously:

Docker Swarm (announced December 2014): Docker’s answer to orchestration, built on the premise that the tool you use to run containers should also be the tool you use to cluster them. Swarm used the same Docker CLI and Docker API — zero new concepts for developers already using Docker.

Apache Mesos (Mesosphere Marathon): Mesos predated Docker. It was a distributed systems kernel originally developed at Berkeley, used in production at Twitter, Airbnb, and Apple. Marathon was the framework for running long-running services on top of Mesos. Mesos could run Docker containers, Hadoop jobs, and Spark workloads on the same cluster. Serious infrastructure engineers took it seriously.

Kubernetes: The newcomer with Google’s name behind it, but no track record outside Google, and early versions that required significant operational expertise to run.

Kubernetes v1.0: July 21, 2015

The 1.0 release landed at the first CloudNativeCon/KubeCon in San Francisco on July 21, 2015. The timing was deliberate — it coincided with the announcement of the Cloud Native Computing Foundation.

What shipped in 1.0:

• Pods: The core scheduling unit — one or more containers sharing a network namespace and storage
• Replication Controllers: Keep N copies of a pod running (later replaced by ReplicaSets and Deployments)
• Services: A stable virtual IP and DNS name in front of a set of pods
• Namespaces: Soft multi-tenancy boundaries within a cluster
• Labels and Selectors: The flexible grouping mechanism that makes everything composable
• Persistent Volumes (basic): Pods could mount persistent storage
• kubectl: The command-line interface

What was not in 1.0:
– No RBAC (Role-Based Access Control)
– No network policy
– No autoscaling
– No Ingress resources
– No StatefulSets
– No DaemonSets (added in 1.1)
– Secrets were stored in plaintext in etcd

The security posture of a fresh Kubernetes 1.0 cluster was essentially: “trust everything inside the cluster.” That was the inherited assumption from Borg.

The CNCF Formation

Alongside the 1.0 release, Google donated Kubernetes to the newly formed Cloud Native Computing Foundation — a Linux Foundation project. This was a critical strategic move.

By donating Kubernetes to a neutral foundation, Google:
1. Removed the perception of a single vendor controlling the project
2. Created a governance model that made enterprise adoption politically safe
3. Invited competitors (Red Hat, CoreOS, Docker, Microsoft) to contribute without ceding control to them

The CNCF’s initial Technical Oversight Committee included engineers from Google, Red Hat, Twitter, Cisco, and others. This governance model would later become the template for every CNCF project that followed.

v1.1 — v1.5: Building the Foundation (Late 2015–2016)

Kubernetes 1.1 (November 2015)

• Horizontal Pod Autoscaler (HPA): Automatically scale pod count based on CPU utilization
• HTTP load balancing: Ingress API added as alpha — pods could now be exposed via HTTP routing rules
• Job objects: Run a task to completion, not just keep it running
• Performance: 30% throughput improvement, pods per minute scheduling rate improved significantly

Kubernetes 1.2 (March 2016)

• Deployments promoted to beta: Rolling updates, rollback, pause/resume — the deployment primitive that engineers actually use for application deployments
• ConfigMaps: Decouple configuration from container images (no more baking config into images)
• Daemon Sets stable: Run exactly one pod per node — the pattern for node agents (log shippers, monitoring agents, network plugins)
• Scale: Tested to 1,000 nodes and 30,000 pods per cluster

Kubernetes 1.3 (July 2016)

• StatefulSets (then called PetSets, alpha): Ordered, persistent-identity pods — the first serious attempt to run databases and stateful applications
• Cross-cluster federation (alpha): Run workloads across multiple clusters
• PodDisruptionBudgets (alpha): Control how many pods can be unavailable during voluntary disruptions — critical for safe rolling updates
• rkt integration (Rktnetes): First Container Runtime Interface experiment — the kubelet talking to something other than Docker

Kubernetes 1.4 (September 2016)

• kubeadm: A tool to bootstrap a Kubernetes cluster in two commands. Before kubeadm, setting up a cluster required following Kelsey Hightower’s “Kubernetes the Hard Way” — valuable for learning, painful for production
• ScheduledJobs (CronJobs): Run a job on a schedule
• PodPresets: Inject common configuration into pods at admission time
• Init Containers beta: Containers that run to completion before the main application containers start — the clean solution for initialization sequencing

Kubernetes 1.5 (December 2016)

• StatefulSets promoted to beta
• PodDisruptionBudgets to beta
• Windows Server container support (alpha): First step toward a non-Linux node
• CRI (Container Runtime Interface) alpha: The abstraction layer that would eventually allow Kubernetes to run containerd, CRI-O, and others instead of depending on Docker
• OpenAPI spec: Machine-readable API documentation, enabling client code generation

Helm: The Missing Package Manager (February 2016)

Kubernetes gave you primitives. It did not give you a way to install applications composed of those primitives. In February 2016, Deis (later acquired by Microsoft) released Helm — a package manager for Kubernetes.

Helm introduced two concepts that stuck:
– Charts: A collection of Kubernetes manifests bundled with templating and default values
– Releases: An installed instance of a chart, with its own lifecycle (install, upgrade, rollback, delete)

Helm’s immediate adoption signaled something important: the community was already thinking in terms of applications, not just raw primitives. Infrastructure engineers needed a layer of abstraction above YAML.

The Battle Lines Harden

By mid-2016, the three-way contest was becoming clearer:

Docker Swarm’s advantage: Zero friction for existing Docker users. docker swarm init + docker stack deploy. No new CLI, no new API, no new mental model. For small teams running straightforward applications, it was compelling.

Mesos’s advantage: Proven at Google-scale before Kubernetes existed. Twitter ran Mesos in production. It could run heterogeneous workloads (Docker containers, Hadoop, Spark) on the same cluster. Enterprise data teams already had Mesos expertise.

Kubernetes’s advantage: The Google name, rapidly growing community, and a design that was clearly winning the feature race. But operational complexity was real — running Kubernetes well in 2016 required significant investment.

The Turning Point Nobody Talks About

The real moment that decided the container wars wasn’t a feature announcement. It was cloud provider behavior.

Google Kubernetes Engine (GKE) — then called Google Container Engine — had been running since 2014. It was the first managed Kubernetes service, and it worked. In 2016, both Microsoft and Amazon were working on managed Kubernetes offerings. Neither chose Docker Swarm. Neither chose Mesos.

When cloud providers converge on a technology, the market follows. By the time Amazon announced EKS and Microsoft announced AKS in late 2017, the decision was already made.

The Security Debt Accumulates

Running through the 1.0–1.5 feature list reveals a security architecture that was being designed in flight:

• etcd stored secrets as base64-encoded strings — not encrypted. Kubernetes 1.7 (2017) would add encryption at rest, but it required explicit configuration
• The API server was unauthenticated by default in early versions — you needed to configure authentication
• Network traffic between pods was unrestricted — all pods could reach all other pods on all ports, across all namespaces. NetworkPolicy existed as alpha in 1.3 but required a CNI plugin that supported it
• The kubelet’s API was open — in early Kubernetes, the kubelet’s HTTP API was accessible without authentication from within the cluster

These weren’t oversights — they were reasonable defaults for an internal cluster managed by a single team. They became liabilities as Kubernetes moved into multi-tenant enterprise environments.

KubeCon: A Community Forms

The first KubeCon conference ran November 9-11, 2015, in San Francisco — a small gathering of a few hundred engineers. By November 2016, KubeCon North America in Seattle drew thousands. The growth was not marketing-driven; it was practitioners solving real problems and sharing what they learned.

This community dynamic was qualitatively different from the Docker Swarm and Mesos ecosystems. Kubernetes had a contributor culture — pull requests, SIG (Special Interest Group) meetings, public design docs. The project was being built in the open, and engineers could see it happening.

Key Takeaways

• Kubernetes 1.0 shipped in July 2015 with the basics functional but security model immature — no RBAC, no network policy, secrets stored in plaintext
• The CNCF governance model was the strategic move that made enterprise adoption politically safe — no single vendor controls the project
• Helm filled the missing application packaging layer that raw Kubernetes couldn’t provide
• The container wars were decided not by technical superiority alone, but by cloud provider alignment — when Google, Microsoft, and Amazon all built managed Kubernetes, the market followed
• v1.1–v1.5 established the core workload primitives: Deployments, StatefulSets, DaemonSets, Jobs, ConfigMaps, HPA — most of these remain the daily vocabulary of Kubernetes operations

What’s Next

← EP01: The Borg Legacy | EP03: Enterprise Awakening →

Series: Kubernetes: From Borg to Platform Engineering | linuxcent.com

The post The Container Wars: Kubernetes 1.0, CNCF, and the Fight for Orchestration (2014–2016) appeared first on Linuxcent.

Introduction

Every piece of infrastructure has a lineage. Kubernetes didn’t appear from nowhere in 2014. It is, in almost every meaningful sense, Google’s Borg system rebuilt for the world — with a decade of hard lessons baked in.

To understand Kubernetes, you have to understand what came before it. And what came before it ran (and still runs) more compute than most organizations will ever touch.

Google’s Scale Problem (2003)

By the early 2000s, Google was running hundreds of thousands of jobs across tens of thousands of machines. Web indexing, ads, Gmail, Maps — all of these needed compute, and none of them could afford to waste it.

In 2003-2004, Google engineer Rohit Seth proposed a kernel feature called cgroups (control groups) — a mechanism to limit, prioritize, account, and isolate resource usage of process groups. The Linux kernel merged cgroups in 2.6.24 (2008). This was the primitive that would later make containers possible.

Simultaneously, Google built Borg — an internal cluster management system that could run hundreds of thousands of jobs, from many thousands of different applications, across many clusters, with each cluster having up to tens of thousands of machines. Borg was never open-sourced. It ran (and still runs) Google’s entire production workload.

What Borg Got Right

Borg introduced concepts that engineers didn’t yet have names for. They became the vocabulary of modern infrastructure:

Workload types:
Borg separated workloads into two classes: long-running services (high-priority, latency-sensitive) and batch jobs (best-effort, preemptible). Kubernetes would later call these Deployments and Jobs.

Declarative specification:
Borg jobs were described in a configuration language (BCL, a dialect of GCL). You declared what you wanted; Borg figured out how to achieve it. Sound familiar?

Resource limits and requests:
Borg tasks had both a request (what you need) and a limit (what you can use). Kubernetes adopted this model directly — resources.requests and resources.limits in pod specs trace directly back to Borg.

Health checking and rescheduling:
Borg monitored task health and automatically rescheduled failed tasks. The kubelet’s liveness and readiness probes are descendants of this.

Cell (cluster) topology:
Borg organized machines into “cells” — what Kubernetes calls clusters. The Borgmaster (control plane) managed the cell.

Omega: The Sequel That Didn’t Ship

Around 2011, Google started building Omega — a more flexible scheduler designed to address Borg’s limitations. Borg had a monolithic scheduler; Omega introduced a shared-state, optimistic-concurrency model where multiple schedulers could operate concurrently without stepping on each other.

A 2013 paper from Google (“Omega: flexible, scalable schedulers for large compute clusters”) made these ideas public. Omega itself stayed internal, but many of its scheduling concepts influenced Kubernetes’ extensible scheduler design.

The Docker Moment (March 2013)

On March 15, 2013, Solomon Hykes stood at PyCon and demonstrated Docker with a five-minute talk titled “The future of Linux Containers.” The demo ran a container. That was it. The room understood immediately.

Docker solved the packaging and distribution problem. Linux had had containers (via LXC and cgroups/namespaces) for years, but running one required deep kernel knowledge. Docker wrapped all of that in a UX that a developer could actually use.

Google’s engineers watched. They recognized the pattern: Docker was doing for containers what the smartphone did for mobile computing — making an existing capability accessible to everyone.

The Google engineers building the next generation of infrastructure realized: once containers become ubiquitous, someone will need to orchestrate them at scale. And they had already built that system internally, twice.

The Decision to Open-Source (Fall 2013)

In late 2013, a small group of Google engineers — Brendan Burns, Joe Beda, Craig McLuckie, Ville Aikas, Tim Hockin, Dawn Chen, Brian Grant, and Daniel Smith — began a new project internally codenamed “Project Seven” (a reference to the Borg drone Seven of Nine).

The core insight: Google’s competitive advantage in infrastructure came from what ran on the cluster management system, not the system itself. Open-sourcing a Kubernetes-like system would benefit Google by standardizing the ecosystem around patterns Google already understood better than anyone.

The initial design decisions were deliberate:

• Go as the implementation language: Fast compilation, good concurrency primitives, easy deployment as static binaries
• REST API as the primary interface: Everything in Kubernetes is an API resource. This is not accidental — it makes the system composable and automatable from day one
• Labels and selectors over hierarchical naming: Borg used a hierarchical job/task naming scheme; Kubernetes chose a flat namespace with label-based grouping, which proved far more flexible
• Reconciliation loops everywhere: Every Kubernetes controller is a loop that watches actual state and drives it toward desired state. This is the controller pattern, and it is the heart of Kubernetes extensibility

First Commit: June 6, 2014

The first public commit landed on GitHub on June 6, 2014: 250 files, 47,501 lines of Go, Bash, and Markdown.

Three days later, on June 10, 2014, Eric Brewer (VP of Infrastructure at Google) announced Kubernetes publicly at DockerCon 2014. The announcement framed it explicitly as bringing Google’s infrastructure learnings to the community.

By July 10, 2014, Microsoft, Red Hat, IBM, and Docker had joined the contributor community.

What Kubernetes Deliberately Left Out of Borg

The designers made intentional decisions about what not to carry forward:

No proprietary language: Borg’s BCL/GCL was Google-internal. Kubernetes used plain JSON (later YAML) manifests — standard formats any tool could read and write.

No magic autoscaling by default: Borg aggressively reclaimed resources. Kubernetes launched without this, adding HPA (Horizontal Pod Autoscaler) later, allowing operators to control the behavior.

No built-in service discovery tied to the scheduler: Borg had tight coupling between scheduling and name resolution. Kubernetes separated these: Services (kube-proxy, DNS) are distinct from the scheduler, allowing them to evolve independently.

The Borg Paper (2015)

In April 2015, Google published “Large-scale cluster management at Google with Borg” — the first public detailed description of the system. Reading it alongside the Kubernetes documentation reveals how directly the design decisions transferred.

Key numbers from the paper:
– Borg ran hundreds of thousands of jobs from thousands of applications
– Typical cell: 10,000 machines
– Utilization improvements from bin-packing: significant enough to justify the entire engineering investment

The paper is required reading for anyone who wants to understand why Kubernetes is designed the way it is — not as a series of arbitrary choices but as a deliberately evolved system.

The Lineage That Matters for Security

From a security architecture perspective, the Borg lineage matters because the isolation model was designed for a trusted-internal environment, not a multi-tenant hostile-external one. This created a debt that Kubernetes has spent years paying down:

• Namespaces are a soft boundary, not a hard isolation primitive — just as Borg’s cells were
• The default-allow network model reflects Borg’s assumption of a trusted internal network
• No built-in admission control at launch — Borg trusted its job submitters

Understanding this history explains why features like NetworkPolicy, PodSecurity, RBAC, and OPA/Gatekeeper were retrofitted over years rather than built-in from day one. The system was designed by and for Google’s internal trust model. The security hardening came as it entered the wild.

Key Takeaways

• Kubernetes is Google’s Borg system rebuilt for the world, carrying 10+ years of cluster management experience
• Core Kubernetes primitives — resource requests/limits, declarative specs, health-based rescheduling, label-based grouping — map directly to Borg concepts
• The decision to open-source was strategic, not altruistic: Google wanted to standardize the ecosystem on patterns it already mastered
• The security gaps in early Kubernetes (no default network isolation, permissive RBAC, no pod-level security controls) trace directly to Borg’s trusted-internal-network assumptions
• Docker’s accessibility breakthrough created the demand; Google’s Borg experience supplied the architecture

What’s Next

EP02: The Container Wars → — Kubernetes 1.0, the CNCF formation, and the three-way fight between Docker Swarm, Apache Mesos, and Kubernetes for control of the container orchestration market.

Series: Kubernetes: From Borg to Platform Engineering | linuxcent.com

The post The Borg Legacy: How Google Built the Blueprint for Kubernetes (2003–2014) appeared first on Linuxcent.