Container Orchestration

The Container Wars: Kubernetes 1.0, CNCF, and the Fight for Orchestration (2014–2016)

by
Reading Time: 6 minutes

CISSP Domain Mapping

Domain Relevance
Domain 3 — Security Architecture Early Kubernetes security model: flat networking, no RBAC, container isolation gaps
Domain 8 — Software Security Container image trust model begins taking shape; Docker Hub as central registry

Introduction

Three orchestration systems entered the arena in 2015. Only one would still matter three years later.

Docker had created the container revolution. Now everyone needed to run containers at scale, and three camps formed around three very different philosophies. Understanding why Kubernetes won — and how close it came to not winning — explains most of the design choices that still shape Kubernetes today.

The State of Container Orchestration in 2014

When Kubernetes made its public debut at DockerCon 2014, it entered a space that didn’t yet have a name. “Container orchestration” wasn’t a category. It was a problem people had started to feel but not yet articulate.

Three approaches emerged nearly simultaneously:

Docker Swarm (announced December 2014): Docker’s answer to orchestration, built on the premise that the tool you use to run containers should also be the tool you use to cluster them. Swarm used the same Docker CLI and Docker API — zero new concepts for developers already using Docker.

Apache Mesos (Mesosphere Marathon): Mesos predated Docker. It was a distributed systems kernel originally developed at Berkeley, used in production at Twitter, Airbnb, and Apple. Marathon was the framework for running long-running services on top of Mesos. Mesos could run Docker containers, Hadoop jobs, and Spark workloads on the same cluster. Serious infrastructure engineers took it seriously.

Kubernetes: The newcomer with Google’s name behind it, but no track record outside Google, and early versions that required significant operational expertise to run.

Kubernetes v1.0: July 21, 2015

The 1.0 release landed at the first CloudNativeCon/KubeCon in San Francisco on July 21, 2015. The timing was deliberate — it coincided with the announcement of the Cloud Native Computing Foundation.

What shipped in 1.0:

  • Pods: The core scheduling unit — one or more containers sharing a network namespace and storage
  • Replication Controllers: Keep N copies of a pod running (later replaced by ReplicaSets and Deployments)
  • Services: A stable virtual IP and DNS name in front of a set of pods
  • Namespaces: Soft multi-tenancy boundaries within a cluster
  • Labels and Selectors: The flexible grouping mechanism that makes everything composable
  • Persistent Volumes (basic): Pods could mount persistent storage
  • kubectl: The command-line interface

What was not in 1.0:
– No RBAC (Role-Based Access Control)
– No network policy
– No autoscaling
– No Ingress resources
– No StatefulSets
– No DaemonSets (added in 1.1)
– Secrets were stored in plaintext in etcd

The security posture of a fresh Kubernetes 1.0 cluster was essentially: “trust everything inside the cluster.” That was the inherited assumption from Borg.

The CNCF Formation

Alongside the 1.0 release, Google donated Kubernetes to the newly formed Cloud Native Computing Foundation — a Linux Foundation project. This was a critical strategic move.

By donating Kubernetes to a neutral foundation, Google:
1. Removed the perception of a single vendor controlling the project
2. Created a governance model that made enterprise adoption politically safe
3. Invited competitors (Red Hat, CoreOS, Docker, Microsoft) to contribute without ceding control to them

The CNCF’s initial Technical Oversight Committee included engineers from Google, Red Hat, Twitter, Cisco, and others. This governance model would later become the template for every CNCF project that followed.

v1.1 — v1.5: Building the Foundation (Late 2015–2016)

Kubernetes 1.1 (November 2015)

  • Horizontal Pod Autoscaler (HPA): Automatically scale pod count based on CPU utilization
  • HTTP load balancing: Ingress API added as alpha — pods could now be exposed via HTTP routing rules
  • Job objects: Run a task to completion, not just keep it running
  • Performance: 30% throughput improvement, pods per minute scheduling rate improved significantly

Kubernetes 1.2 (March 2016)

  • Deployments promoted to beta: Rolling updates, rollback, pause/resume — the deployment primitive that engineers actually use for application deployments
  • ConfigMaps: Decouple configuration from container images (no more baking config into images)
  • Daemon Sets stable: Run exactly one pod per node — the pattern for node agents (log shippers, monitoring agents, network plugins)
  • Scale: Tested to 1,000 nodes and 30,000 pods per cluster

Kubernetes 1.3 (July 2016)

  • StatefulSets (then called PetSets, alpha): Ordered, persistent-identity pods — the first serious attempt to run databases and stateful applications
  • Cross-cluster federation (alpha): Run workloads across multiple clusters
  • PodDisruptionBudgets (alpha): Control how many pods can be unavailable during voluntary disruptions — critical for safe rolling updates
  • rkt integration (Rktnetes): First Container Runtime Interface experiment — the kubelet talking to something other than Docker

Kubernetes 1.4 (September 2016)

  • kubeadm: A tool to bootstrap a Kubernetes cluster in two commands. Before kubeadm, setting up a cluster required following Kelsey Hightower’s “Kubernetes the Hard Way” — valuable for learning, painful for production
  • ScheduledJobs (CronJobs): Run a job on a schedule
  • PodPresets: Inject common configuration into pods at admission time
  • Init Containers beta: Containers that run to completion before the main application containers start — the clean solution for initialization sequencing

Kubernetes 1.5 (December 2016)

  • StatefulSets promoted to beta
  • PodDisruptionBudgets to beta
  • Windows Server container support (alpha): First step toward a non-Linux node
  • CRI (Container Runtime Interface) alpha: The abstraction layer that would eventually allow Kubernetes to run containerd, CRI-O, and others instead of depending on Docker
  • OpenAPI spec: Machine-readable API documentation, enabling client code generation

Helm: The Missing Package Manager (February 2016)

Kubernetes gave you primitives. It did not give you a way to install applications composed of those primitives. In February 2016, Deis (later acquired by Microsoft) released Helm — a package manager for Kubernetes.

Helm introduced two concepts that stuck:
– Charts: A collection of Kubernetes manifests bundled with templating and default values
– Releases: An installed instance of a chart, with its own lifecycle (install, upgrade, rollback, delete)

Helm’s immediate adoption signaled something important: the community was already thinking in terms of applications, not just raw primitives. Infrastructure engineers needed a layer of abstraction above YAML.

The Battle Lines Harden

By mid-2016, the three-way contest was becoming clearer:

Docker Swarm’s advantage: Zero friction for existing Docker users. docker swarm init + docker stack deploy. No new CLI, no new API, no new mental model. For small teams running straightforward applications, it was compelling.

Mesos’s advantage: Proven at Google-scale before Kubernetes existed. Twitter ran Mesos in production. It could run heterogeneous workloads (Docker containers, Hadoop, Spark) on the same cluster. Enterprise data teams already had Mesos expertise.

Kubernetes’s advantage: The Google name, rapidly growing community, and a design that was clearly winning the feature race. But operational complexity was real — running Kubernetes well in 2016 required significant investment.

The Turning Point Nobody Talks About

The real moment that decided the container wars wasn’t a feature announcement. It was cloud provider behavior.

Google Kubernetes Engine (GKE) — then called Google Container Engine — had been running since 2014. It was the first managed Kubernetes service, and it worked. In 2016, both Microsoft and Amazon were working on managed Kubernetes offerings. Neither chose Docker Swarm. Neither chose Mesos.

When cloud providers converge on a technology, the market follows. By the time Amazon announced EKS and Microsoft announced AKS in late 2017, the decision was already made.

The Security Debt Accumulates

Running through the 1.0–1.5 feature list reveals a security architecture that was being designed in flight:

  • etcd stored secrets as base64-encoded strings — not encrypted. Kubernetes 1.7 (2017) would add encryption at rest, but it required explicit configuration
  • The API server was unauthenticated by default in early versions — you needed to configure authentication
  • Network traffic between pods was unrestricted — all pods could reach all other pods on all ports, across all namespaces. NetworkPolicy existed as alpha in 1.3 but required a CNI plugin that supported it
  • The kubelet’s API was open — in early Kubernetes, the kubelet’s HTTP API was accessible without authentication from within the cluster

These weren’t oversights — they were reasonable defaults for an internal cluster managed by a single team. They became liabilities as Kubernetes moved into multi-tenant enterprise environments.

KubeCon: A Community Forms

The first KubeCon conference ran November 9-11, 2015, in San Francisco — a small gathering of a few hundred engineers. By November 2016, KubeCon North America in Seattle drew thousands. The growth was not marketing-driven; it was practitioners solving real problems and sharing what they learned.

This community dynamic was qualitatively different from the Docker Swarm and Mesos ecosystems. Kubernetes had a contributor culture — pull requests, SIG (Special Interest Group) meetings, public design docs. The project was being built in the open, and engineers could see it happening.

Key Takeaways

  • Kubernetes 1.0 shipped in July 2015 with the basics functional but security model immature — no RBAC, no network policy, secrets stored in plaintext
  • The CNCF governance model was the strategic move that made enterprise adoption politically safe — no single vendor controls the project
  • Helm filled the missing application packaging layer that raw Kubernetes couldn’t provide
  • The container wars were decided not by technical superiority alone, but by cloud provider alignment — when Google, Microsoft, and Amazon all built managed Kubernetes, the market followed
  • v1.1–v1.5 established the core workload primitives: Deployments, StatefulSets, DaemonSets, Jobs, ConfigMaps, HPA — most of these remain the daily vocabulary of Kubernetes operations

What’s Next

← EP01: The Borg Legacy | EP03: Enterprise Awakening →

Series: Kubernetes: From Borg to Platform Engineering | linuxcent.com

The Borg Legacy: How Google Built the Blueprint for Kubernetes (2003–2014)

by
Reading Time: 5 minutes

CISSP Domain Mapping

Domain Relevance
Domain 3 — Security Architecture Workload isolation principles from Borg carried directly into Kubernetes namespace and cgroup model
Domain 7 — Security Operations Cluster-wide resource governance and auditing concepts established by Borg

Introduction

Every piece of infrastructure has a lineage. Kubernetes didn’t appear from nowhere in 2014. It is, in almost every meaningful sense, Google’s Borg system rebuilt for the world — with a decade of hard lessons baked in.

To understand Kubernetes, you have to understand what came before it. And what came before it ran (and still runs) more compute than most organizations will ever touch.

Google’s Scale Problem (2003)

By the early 2000s, Google was running hundreds of thousands of jobs across tens of thousands of machines. Web indexing, ads, Gmail, Maps — all of these needed compute, and none of them could afford to waste it.

In 2003-2004, Google engineer Rohit Seth proposed a kernel feature called cgroups (control groups) — a mechanism to limit, prioritize, account, and isolate resource usage of process groups. The Linux kernel merged cgroups in 2.6.24 (2008). This was the primitive that would later make containers possible.

Simultaneously, Google built Borg — an internal cluster management system that could run hundreds of thousands of jobs, from many thousands of different applications, across many clusters, with each cluster having up to tens of thousands of machines. Borg was never open-sourced. It ran (and still runs) Google’s entire production workload.

What Borg Got Right

Borg introduced concepts that engineers didn’t yet have names for. They became the vocabulary of modern infrastructure:

Workload types:
Borg separated workloads into two classes: long-running services (high-priority, latency-sensitive) and batch jobs (best-effort, preemptible). Kubernetes would later call these Deployments and Jobs.

Declarative specification:
Borg jobs were described in a configuration language (BCL, a dialect of GCL). You declared what you wanted; Borg figured out how to achieve it. Sound familiar?

Resource limits and requests:
Borg tasks had both a request (what you need) and a limit (what you can use). Kubernetes adopted this model directly — resources.requests and resources.limits in pod specs trace directly back to Borg.

Health checking and rescheduling:
Borg monitored task health and automatically rescheduled failed tasks. The kubelet’s liveness and readiness probes are descendants of this.

Cell (cluster) topology:
Borg organized machines into “cells” — what Kubernetes calls clusters. The Borgmaster (control plane) managed the cell.

Omega: The Sequel That Didn’t Ship

Around 2011, Google started building Omega — a more flexible scheduler designed to address Borg’s limitations. Borg had a monolithic scheduler; Omega introduced a shared-state, optimistic-concurrency model where multiple schedulers could operate concurrently without stepping on each other.

A 2013 paper from Google (“Omega: flexible, scalable schedulers for large compute clusters”) made these ideas public. Omega itself stayed internal, but many of its scheduling concepts influenced Kubernetes’ extensible scheduler design.

The Docker Moment (March 2013)

On March 15, 2013, Solomon Hykes stood at PyCon and demonstrated Docker with a five-minute talk titled “The future of Linux Containers.” The demo ran a container. That was it. The room understood immediately.

Docker solved the packaging and distribution problem. Linux had had containers (via LXC and cgroups/namespaces) for years, but running one required deep kernel knowledge. Docker wrapped all of that in a UX that a developer could actually use.

Google’s engineers watched. They recognized the pattern: Docker was doing for containers what the smartphone did for mobile computing — making an existing capability accessible to everyone.

The Google engineers building the next generation of infrastructure realized: once containers become ubiquitous, someone will need to orchestrate them at scale. And they had already built that system internally, twice.

The Decision to Open-Source (Fall 2013)

In late 2013, a small group of Google engineers — Brendan Burns, Joe Beda, Craig McLuckie, Ville Aikas, Tim Hockin, Dawn Chen, Brian Grant, and Daniel Smith — began a new project internally codenamed “Project Seven” (a reference to the Borg drone Seven of Nine).

The core insight: Google’s competitive advantage in infrastructure came from what ran on the cluster management system, not the system itself. Open-sourcing a Kubernetes-like system would benefit Google by standardizing the ecosystem around patterns Google already understood better than anyone.

The initial design decisions were deliberate:

  • Go as the implementation language: Fast compilation, good concurrency primitives, easy deployment as static binaries
  • REST API as the primary interface: Everything in Kubernetes is an API resource. This is not accidental — it makes the system composable and automatable from day one
  • Labels and selectors over hierarchical naming: Borg used a hierarchical job/task naming scheme; Kubernetes chose a flat namespace with label-based grouping, which proved far more flexible
  • Reconciliation loops everywhere: Every Kubernetes controller is a loop that watches actual state and drives it toward desired state. This is the controller pattern, and it is the heart of Kubernetes extensibility

First Commit: June 6, 2014

The first public commit landed on GitHub on June 6, 2014: 250 files, 47,501 lines of Go, Bash, and Markdown.

Three days later, on June 10, 2014, Eric Brewer (VP of Infrastructure at Google) announced Kubernetes publicly at DockerCon 2014. The announcement framed it explicitly as bringing Google’s infrastructure learnings to the community.

By July 10, 2014, Microsoft, Red Hat, IBM, and Docker had joined the contributor community.

What Kubernetes Deliberately Left Out of Borg

The designers made intentional decisions about what not to carry forward:

No proprietary language: Borg’s BCL/GCL was Google-internal. Kubernetes used plain JSON (later YAML) manifests — standard formats any tool could read and write.

No magic autoscaling by default: Borg aggressively reclaimed resources. Kubernetes launched without this, adding HPA (Horizontal Pod Autoscaler) later, allowing operators to control the behavior.

No built-in service discovery tied to the scheduler: Borg had tight coupling between scheduling and name resolution. Kubernetes separated these: Services (kube-proxy, DNS) are distinct from the scheduler, allowing them to evolve independently.

The Borg Paper (2015)

In April 2015, Google published “Large-scale cluster management at Google with Borg” — the first public detailed description of the system. Reading it alongside the Kubernetes documentation reveals how directly the design decisions transferred.

Key numbers from the paper:
– Borg ran hundreds of thousands of jobs from thousands of applications
– Typical cell: 10,000 machines
– Utilization improvements from bin-packing: significant enough to justify the entire engineering investment

The paper is required reading for anyone who wants to understand why Kubernetes is designed the way it is — not as a series of arbitrary choices but as a deliberately evolved system.

The Lineage That Matters for Security

From a security architecture perspective, the Borg lineage matters because the isolation model was designed for a trusted-internal environment, not a multi-tenant hostile-external one. This created a debt that Kubernetes has spent years paying down:

  • Namespaces are a soft boundary, not a hard isolation primitive — just as Borg’s cells were
  • The default-allow network model reflects Borg’s assumption of a trusted internal network
  • No built-in admission control at launch — Borg trusted its job submitters

Understanding this history explains why features like NetworkPolicy, PodSecurity, RBAC, and OPA/Gatekeeper were retrofitted over years rather than built-in from day one. The system was designed by and for Google’s internal trust model. The security hardening came as it entered the wild.

Key Takeaways

  • Kubernetes is Google’s Borg system rebuilt for the world, carrying 10+ years of cluster management experience
  • Core Kubernetes primitives — resource requests/limits, declarative specs, health-based rescheduling, label-based grouping — map directly to Borg concepts
  • The decision to open-source was strategic, not altruistic: Google wanted to standardize the ecosystem on patterns it already mastered
  • The security gaps in early Kubernetes (no default network isolation, permissive RBAC, no pod-level security controls) trace directly to Borg’s trusted-internal-network assumptions
  • Docker’s accessibility breakthrough created the demand; Google’s Borg experience supplied the architecture

What’s Next

EP02: The Container Wars → — Kubernetes 1.0, the CNCF formation, and the three-way fight between Docker Swarm, Apache Mesos, and Kubernetes for control of the container orchestration market.

Series: Kubernetes: From Borg to Platform Engineering | linuxcent.com