Forensics

Reading Time: 15 minutes

What is purple team security → OWASP Top 10 mapped to cloud infrastructure → Cloud security breaches 2020–2025 → Broken access control in AWS → MFA fatigue attacks → CI/CD secrets exposure → SSRF to cloud metadata → Kubernetes container escape → Supply chain attack detection → Cloud lateral movement IAM → Detection engineering with eBPF → Cloud Incident Response Playbook

---

TL;DR

• A cloud incident response playbook is not documentation you write after a breach — it is the executable sequence your team runs in the first 24 hours, rehearsed before the breach happens
• The ChangeHealthcare attack (February 2024) disrupted $22 billion in medical claims processing and exposed 190 million Americans’ health data; the initial vector was a single set of stolen credentials and a Citrix portal with no MFA
• Hours 0–1: declare the incident immediately, scope the blast radius, and start querying CloudTrail — do not investigate quietly
• Hours 1–4: contain by revoking credentials and isolating infrastructure, but preserve evidence before any remediation — forensic snapshots and log exports before terminating anything
• Hours 4–12: trace lateral movement via AssumeRole chains, identify persistence mechanisms (new IAM users/roles, Lambda backdoors, modified images), and confirm the full data access scope
• Hours 12–24: eradicate from known-good baselines, not by patching compromised instances; recover dev → staging → prod; trigger regulatory notification timers

---

OWASP Mapping: Cross-cutting — incident response is not mapped to a single OWASP category because a breach can enter through any of them. IR quality is the backstop when prevention fails across A01 (broken access control), A07 (authentication failures), A08 (supply chain), and every other vector. The 24-hour window covered here applies regardless of initial entry point.

---

The Big Picture

┌─────────────────────────────────────────────────────────────────────────┐
│            CLOUD INCIDENT RESPONSE: THE 24-HOUR SEQUENCE                │
│                                                                         │
│  ALERT                                                                  │
│    GuardDuty / Falco / anomaly detection fires                          │
│    ↓                                                                    │
│  TRIAGE  [0–1h]                                                         │
│    Declare incident → scope blast radius → open incident channel        │
│    Is the attacker still active? What data is at risk?                  │
│    ↓                                                                    │
│  CONTAIN  [1–4h]                                                        │
│    Revoke credentials → isolate compute → cordon K8s nodes             │
│    !! Do NOT terminate instances before snapshot !!                     │
│    ↓                                                                    │
│  PRESERVE  [1–4h, parallel with contain]                                │
│    EBS snapshots → CloudTrail log export → VPC Flow export              │
│    Forensic copy before any remediation changes the system state        │
│    ↓                                                                    │
│  INVESTIGATE  [4–12h]                                                   │
│    AssumeRole chain analysis → data access scope → persistence hunt     │
│    eBPF/Falco/Tetragon evidence if available (see EP11)                 │
│    ↓                                                                    │
│  ERADICATE  [12–24h]                                                    │
│    Remove persistence → rotate ALL credentials in blast radius          │
│    Replace compromised instances from known-good hardened AMI           │
│    ↓                                                                    │
│  RECOVER  [12–24h]                                                      │
│    dev → staging → prod sequence. Never prod-first.                     │
│    Verify monitoring before declaring all-clear                         │
│    ↓                                                                    │
│  LEARN                                                                  │
│    Post-incident review → timeline → regulatory notifications           │
│    Update playbook before the next incident                             │
└─────────────────────────────────────────────────────────────────────────┘

A cloud incident response playbook that exists only as a document is not an incident response capability. The sequence above is only useful if your team has rehearsed it — run it as a tabletop, run it in a chaos exercise, run it on a simulated breach in a non-prod account. The first time through this sequence should not be during an actual breach.

---

The Incident: ChangeHealthcare (February 2024)

On February 21, 2024, ransomware attacked Change Healthcare, a UnitedHealth Group subsidiary that processes roughly 50% of US medical claims. By the time containment completed, the damage was:

• $22 billion in medical claims processing disrupted
• 190 million Americans’ health data potentially exposed
• Hospitals unable to process insurance claims for weeks — some faced payroll crises because they couldn’t get reimbursed for care already delivered
• A $22 million ransom paid to ALPHV/BlackCat, followed by ALPHV exit-scamming the affiliate (keeping the ransom), followed by RansomHub re-extorting with the same data

The initial vector: a Citrix remote access portal with no MFA enforced. A single set of stolen credentials. That’s it.

What made the outcome as severe as it was: the attackers had nine days of dwell time before the ransomware detonated. Nine days of lateral movement, data staging, and backup discovery before the explosion. The first 24 hours after detection determine whether you contain an intrusion or respond to a full-scale breach. The ChangeHealthcare team was responding to a full-scale breach because the first 24 hours happened nine days before anyone knew there was an incident.

There is an inverse relationship between incident response quality and preparation investment. Teams that contain in four hours practiced containing in four hours. Teams that discover they have no forensic evidence discover that during the investigation, not before it.

---

Hour 0–1: Detect and Declare

Step 1: Declare — Do Not Investigate Quietly

The instinct when something looks suspicious is to investigate before escalating. That instinct is wrong in cloud incidents. Every minute of quiet investigation is a minute the attacker may be escalating privileges, staging data, or discovering your backups.

Declare the incident immediately. The threshold for declaration is suspicion, not confirmation.

Who to notify in the first 15 minutes:
– CISO (or on-call security lead)
– Legal counsel (regulatory clock starts now; you need legal involved from minute one)
– On-call SRE lead (you will need infrastructure access)
– Communications lead (if external-facing systems are involved)

Operational setup:
1. Create a dedicated incident Slack channel: #incident-YYYY-MM-DD-brief-descriptor
2. Start an incident log — a shared doc, timestamped, with every action taken and by whom. This becomes your evidence log and your regulatory submission document.
3. Assign a scribe. The incident commander should not also be taking notes.

Step 2: Scope the Blast Radius

Before touching anything, answer three questions:

1. Is the attacker still active? (Is this ongoing or historical?)
2. What is the potential blast radius? (Which accounts, regions, services, principals are in scope?)
3. What data is at risk? (PII, credentials, intellectual property, PHI/PII with regulatory implications?)

Step 3: Initial CloudTrail Query

# Run this before touching anything — you want a clean baseline
aws cloudtrail lookup-events \
  --lookup-attributes AttributeKey=Username,AttributeValue=suspected-role \
  --start-time $(date -d '1 hour ago' --iso-8601=seconds) \
  --query 'Events[*].[EventTime,EventName,Resources[0].ResourceName]' \
  --output table

# If you don't know the principal yet — look for unusual API activity
# across all principals in the last hour
aws cloudtrail lookup-events \
  --start-time $(date -d '1 hour ago' --iso-8601=seconds) \
  --query 'Events[*].{Time:EventTime,User:Username,Event:EventName,Source:EventSource}' \
  --output json | \
  jq 'sort_by(.Time) | reverse | .[:50]'
# Look for: CreateUser, AttachRolePolicy, PutRolePolicy, CreateAccessKey,
#           GetSecretValue, ListBuckets, DescribeInstances in rapid succession

# Check GuardDuty for the triggering finding
DETECTOR_ID=$(aws guardduty list-detectors --query 'DetectorIds[0]' --output text)

aws guardduty get-findings \
  --detector-id "${DETECTOR_ID}" \
  --finding-ids $(aws guardduty list-findings \
    --detector-id "${DETECTOR_ID}" \
    --finding-criteria '{
      "Criterion": {
        "updatedAt": {"Gte": '$(date -d '24 hours ago' +%s000)'}
      }
    }' \
    --sort-criteria '{"AttributeName":"updatedAt","OrderBy":"DESC"}' \
    --max-results 10 \
    --query 'FindingIds' --output text) | \
  jq '.Findings[] | {type: .Type, severity: .Severity, time: .UpdatedAt, detail: .Description}'

---

Hour 1–4: Contain Without Destroying Evidence

The central tension in early containment: you need to stop the bleeding, but you also need the evidence. Terminating a compromised EC2 instance stops the threat on that instance — it also destroys the process table, network connections, in-memory artifacts, and filesystem state that the investigation needs.

The order of operations:
1. Preserve (snapshot, export logs)
2. Contain (revoke credentials, isolate network)
3. Never terminate before step 1

Evidence Preservation (Before Any Containment Action)

# Create EBS snapshots of ALL volumes on compromised instances
# Do this FIRST — before network isolation, before anything
aws ec2 describe-instances \
  --instance-ids i-compromised-instance-id \
  --query 'Reservations[].Instances[].BlockDeviceMappings[].Ebs.VolumeId' \
  --output text | tr '\t' '\n' | \
  while read vol_id; do
    echo "Snapshotting volume: ${vol_id}"
    aws ec2 create-snapshot \
      --volume-id "${vol_id}" \
      --description "IR evidence - $(date --iso-8601) - ${vol_id}" \
      --tag-specifications "ResourceType=snapshot,Tags=[{Key=incident,Value=active},{Key=preserve,Value=legal-hold}]"
  done

# Export CloudTrail logs for the incident window to a local IR evidence directory
# Use a time window that starts 24 hours before the suspected compromise
aws s3 sync \
  s3://your-cloudtrail-bucket/AWSLogs/123456789012/CloudTrail/ \
  ./ir-evidence/cloudtrail/ \
  --exclude "*" \
  --include "*/2024/02/21/*" \
  --include "*/2024/02/22/*"

# Export VPC Flow Logs for the incident window
# These show network connections that CloudTrail doesn't capture
aws logs filter-log-events \
  --log-group-name /aws/vpc/flowlogs \
  --start-time $(date -d '24 hours ago' +%s000) \
  --end-time $(date +%s000) \
  --query 'events[*].message' \
  --output text > ./ir-evidence/vpc-flow-logs.txt

Containment Action 1: Revoke the Compromised Credential

# Option A: Disable an IAM user's access key (reversible — preserves key for forensics)
aws iam update-access-key \
  --user-name compromised-user \
  --access-key-id AKIAIOSFODNN7EXAMPLE \
  --status Inactive

# Option B: If the compromised principal is an IAM role —
# attach a deny-all inline policy (fastest, takes effect immediately)
aws iam put-role-policy \
  --role-name compromised-role \
  --policy-name incident-deny-all \
  --policy-document '{
    "Version": "2012-10-17",
    "Statement": [
      {
        "Sid": "IncidentDenyAll",
        "Effect": "Deny",
        "Action": "*",
        "Resource": "*"
      }
    ]
  }'

# Option C: If you need to revoke ALL active sessions for a role immediately
# (active STS sessions are not invalidated by the deny policy alone
#  until the session token expires — use this to force immediate revocation)
aws iam put-role-policy \
  --role-name compromised-role \
  --policy-name incident-deny-all \
  --policy-document '{
    "Version": "2012-10-17",
    "Statement": [
      {
        "Effect": "Deny",
        "Action": "*",
        "Resource": "*",
        "Condition": {
          "DateLessThan": {
            "aws:TokenIssueTime": "'$(date --iso-8601=seconds)'"
          }
        }
      }
    ]
  }'
# This denies all requests where the token was issued before right now
# — effectively invalidating all existing sessions for this role

Containment Action 2: Isolate Affected EC2 Instances

# Create an isolation security group — no ingress, no egress
# except SSH from your IR bastion (for forensic access if needed)
ISOLATION_SG=$(aws ec2 create-security-group \
  --group-name "incident-isolation-$(date +%Y%m%d)" \
  --description "Incident isolation - no network access except IR bastion" \
  --vpc-id vpc-your-vpc-id \
  --query 'GroupId' \
  --output text)

echo "Isolation SG created: ${ISOLATION_SG}"

# Add ingress rule: only from IR bastion (for forensic access)
# Remove this rule entirely if you don't need it
aws ec2 authorize-security-group-ingress \
  --group-id "${ISOLATION_SG}" \
  --protocol tcp \
  --port 22 \
  --cidr YOUR-IR-BASTION-IP/32

# Apply the isolation SG to the compromised instance
# This replaces all existing security groups — the instance is now isolated
aws ec2 modify-instance-attribute \
  --instance-id i-compromised-instance-id \
  --groups "${ISOLATION_SG}"

Important: Do not terminate the instance. The isolated instance remains available for forensic analysis via the IR bastion. Termination destroys volatile evidence. You terminate after the investigation is complete and legal has cleared the evidence for destruction.

Containment Action 3: Kubernetes — Cordon, Don’t Delete

# Cordon the compromised node — prevents new pod scheduling
kubectl cordon node/compromised-node-name

# Label the node for IR tracking
kubectl label node/compromised-node-name incident=active preserve=legal-hold

# If a specific pod is the concern — do NOT kubectl delete pod
# Instead, collect forensic information first
POD_NAME="compromised-pod"
NAMESPACE="production"

# Capture the full pod spec and status
kubectl get pod "${POD_NAME}" -n "${NAMESPACE}" -o json > \
  ./ir-evidence/pod-spec-${POD_NAME}.json

# Capture environment variables (may contain credential evidence)
kubectl exec "${POD_NAME}" -n "${NAMESPACE}" -- env > \
  ./ir-evidence/pod-env-${POD_NAME}.txt 2>/dev/null

# Capture running processes
kubectl exec "${POD_NAME}" -n "${NAMESPACE}" -- ps auxf > \
  ./ir-evidence/pod-processes-${POD_NAME}.txt 2>/dev/null

# Capture network connections
kubectl exec "${POD_NAME}" -n "${NAMESPACE}" -- ss -tunapw > \
  ./ir-evidence/pod-netstat-${POD_NAME}.txt 2>/dev/null

# Now you can delete the pod if needed — you have the evidence

---

Hour 4–12: Investigate the Blast Radius

Containment stops the active threat. Investigation answers: what did they do, where did they go, and what did they touch?

Trace the Lateral Movement

The most important lateral movement mechanism in AWS is AssumeRole chaining — a compromised principal assumes a role, which has permissions to assume another role, building a privilege escalation path. IAM attack path reconstruction requires following this chain through CloudTrail.

# Find all AssumeRole events from the compromised principal
# This shows every role the attacker assumed after initial compromise
aws cloudtrail lookup-events \
  --lookup-attributes AttributeKey=EventName,AttributeValue=AssumeRole \
  --start-time "2024-02-21T00:00:00Z" \
  --end-time "2024-02-22T23:59:59Z" \
  --output json | \
  jq '.Events[] | 
    (.CloudTrailEvent | fromjson) | 
    select(.userIdentity.arn | contains("compromised-role")) | 
    {
      time: .eventTime,
      caller: .userIdentity.arn,
      assumed_role: .requestParameters.roleArn,
      session_name: .requestParameters.roleSessionName,
      source_ip: .sourceIPAddress
    }'

# Follow the chain — get ALL roles assumed during the incident window
# regardless of source, then trace connections manually
aws cloudtrail lookup-events \
  --lookup-attributes AttributeKey=EventName,AttributeValue=AssumeRole \
  --start-time "2024-02-21T00:00:00Z" \
  --end-time "2024-02-22T23:59:59Z" \
  --output json | \
  jq -r '.Events[] | 
    (.CloudTrailEvent | fromjson) | 
    [.eventTime, .userIdentity.arn, .requestParameters.roleArn, .sourceIPAddress] | 
    @tsv' | \
  sort -k1
# Build the graph manually: which ARN called AssumeRole for which target role
# Any role not in your expected deployment automation is suspicious

Find What Data Was Accessed

# S3 GetObject events — shows every object the attacker read
# NOTE: S3 data events are NOT enabled by default in CloudTrail
# If you haven't pre-enabled them, this query returns nothing useful
aws cloudtrail lookup-events \
  --lookup-attributes AttributeKey=EventName,AttributeValue=GetObject \
  --start-time "2024-02-21T00:00:00Z" \
  --end-time "2024-02-22T23:59:59Z" \
  --output json | \
  jq '.Events[] | 
    (.CloudTrailEvent | fromjson) | 
    {
      time: .eventTime,
      user: .userIdentity.arn,
      bucket: .requestParameters.bucketName,
      key: .requestParameters.key,
      source_ip: .sourceIPAddress
    }'

# Secrets Manager — what secrets were accessed?
aws cloudtrail lookup-events \
  --lookup-attributes AttributeKey=EventName,AttributeValue=GetSecretValue \
  --start-time "2024-02-21T00:00:00Z" \
  --output json | \
  jq '.Events[] | 
    (.CloudTrailEvent | fromjson) | 
    {
      time: .eventTime,
      user: .userIdentity.arn,
      secret: .requestParameters.secretId,
      source_ip: .sourceIPAddress
    }'

# KMS — what was decrypted?
aws cloudtrail lookup-events \
  --lookup-attributes AttributeKey=EventName,AttributeValue=Decrypt \
  --start-time "2024-02-21T00:00:00Z" \
  --output json | \
  jq '.Events[] | 
    (.CloudTrailEvent | fromjson) | 
    {
      time: .eventTime,
      user: .userIdentity.arn,
      key_id: .requestParameters.keyId,
      source_ip: .sourceIPAddress
    }'

Hunt for Persistence Mechanisms

Attackers establish persistence before detonating ransomware or before exfiltrating at scale. The most common persistence mechanisms in AWS:

# New IAM users created during the incident window
aws iam list-users \
  --query 'Users[?CreateDate>=`2024-02-21T00:00:00Z`].[UserName,CreateDate,UserId]' \
  --output table

# New IAM roles created during the incident window
aws iam list-roles \
  --query 'Roles[?CreateDate>=`2024-02-21T00:00:00Z`].[RoleName,CreateDate,RoleId]' \
  --output table

# New IAM access keys created for existing users
aws cloudtrail lookup-events \
  --lookup-attributes AttributeKey=EventName,AttributeValue=CreateAccessKey \
  --start-time "2024-02-21T00:00:00Z" \
  --output json | \
  jq '.Events[] | (.CloudTrailEvent | fromjson) | {time: .eventTime, user: .requestParameters.userName, by: .userIdentity.arn}'

# Lambda functions with recent code modifications
# (Lambda is a common backdoor target — function code is easy to modify)
aws lambda list-functions \
  --query 'Functions[?LastModified>=`2024-02-21`].[FunctionName,LastModified,Runtime]' \
  --output table

# For any recently modified function — check for unexpected environment variables
aws lambda get-function-configuration \
  --function-name suspicious-function-name \
  --query '{env: Environment.Variables, role: Role, handler: Handler}'

# CloudFormation stacks created or modified during incident window
aws cloudtrail lookup-events \
  --lookup-attributes AttributeKey=EventName,AttributeValue=CreateStack \
  --start-time "2024-02-21T00:00:00Z" \
  --output json | \
  jq '.Events[] | (.CloudTrailEvent | fromjson) | {time: .eventTime, stack: .requestParameters.stackName, by: .userIdentity.arn}'

# EC2 user-data modifications (backdoor via user data on restart)
aws cloudtrail lookup-events \
  --lookup-attributes AttributeKey=EventName,AttributeValue=ModifyInstanceAttribute \
  --start-time "2024-02-21T00:00:00Z" \
  --output json | \
  jq '.Events[] | (.CloudTrailEvent | fromjson) | select(.requestParameters | has("userData")) | {time: .eventTime, instance: .requestParameters.instanceId, by: .userIdentity.arn}'

eBPF and Falco Evidence (If Available)

If your environment runs Falco or Cilium Tetragon (see detection engineering with eBPF), the kernel-level telemetry from EP11 is now forensic evidence:

# Tetragon: export process execution events for the incident window
# Tetragon writes to /var/log/tetragon/tetragon.log by default
# Filter by the time window and affected pod/node

# On the affected node (or via log aggregation if you ship to a SIEM):
cat /var/log/tetragon/tetragon.log | \
  jq 'select(.time >= "2024-02-21T00:00:00Z" and .time <= "2024-02-22T23:59:59Z") |
    select(.process_exec != null) |
    {
      time: .time,
      pod: .process_exec.process.pod.name,
      ns: .process_exec.process.pod.namespace,
      binary: .process_exec.process.binary,
      args: .process_exec.process.arguments,
      parent: .process_exec.parent.binary
    }' | head -100

# Falco: pull alerts from the incident window out of your SIEM/log store
# If you're running Falco with file output:
grep "2024-02-21\|2024-02-22" /var/log/falco/events.json | \
  jq 'select(.priority == "Critical" or .priority == "Error") |
    {time: .time, rule: .rule, output: .output, pod: .output_fields."k8s.pod.name"}' | \
  head -50

Process lineage from Tetragon (which parent process spawned which child) is often the clearest signal of container escape or lateral movement within a cluster. It shows attack paths that API-layer logging cannot reconstruct.

---

Hour 12–24: Eradicate and Recover

Remove Persistence

Work through the persistence findings from the investigation phase in order:

# Delete unauthorized IAM users created during the incident
# First: disable their access keys
aws iam list-access-keys --user-name attacker-created-user \
  --query 'AccessKeyMetadata[].AccessKeyId' --output text | \
  tr '\t' '\n' | \
  while read key_id; do
    aws iam update-access-key --user-name attacker-created-user \
      --access-key-id "${key_id}" --status Inactive
  done

# Then: detach all policies, remove from groups, delete login profile, delete user
aws iam detach-user-policy --user-name attacker-created-user \
  --policy-arn arn:aws:iam::123456789012:policy/attached-policy
aws iam delete-user --user-name attacker-created-user

# Rotate ALL credentials that could have been accessed during the incident window
# Not just the initial compromise — every secret in the blast radius

# List all IAM user access keys in the affected account
aws iam list-users --query 'Users[].UserName' --output text | tr '\t' '\n' | \
  while read user; do
    aws iam list-access-keys --user-name "${user}" \
      --query 'AccessKeyMetadata[?Status==`Active`].{User:UserName,Key:AccessKeyId}' \
      --output json
  done | jq -s 'flatten'
# For each key: create new key → update application config → delete old key

# Remove Lambda backdoors — restore from last known-good deployment
# Do NOT patch the modified function — replace the entire deployment package
aws lambda update-function-code \
  --function-name backdoored-function \
  --s3-bucket your-code-bucket \
  --s3-key known-good/function-v1.2.3.zip

# Reset environment variables (remove anything added during incident)
aws lambda update-function-configuration \
  --function-name backdoored-function \
  --environment 'Variables={EXPECTED_VAR=expected_value}'

Replace Compromised Instances From Known-Good Baselines

Do not patch a compromised instance and return it to production. The instance’s integrity is unknown — the attacker may have modified binaries, installed kernel modules, or altered the init system in ways that a filesystem scan won’t catch.

Replace from a known-good hardened image:

# Launch a replacement from a hardened baseline AMI
# If you're running a Stratum-built image pipeline, this is where it pays off:
# you have a signed, hardened, versioned AMI to replace from

aws ec2 run-instances \
  --image-id ami-known-good-hardened-baseline \
  --instance-type t3.medium \
  --subnet-id subnet-your-private-subnet \
  --security-groups sg-your-normal-sg \
  --iam-instance-profile Name=your-instance-profile \
  --tag-specifications \
    'ResourceType=instance,Tags=[{Key=Name,Value=replacement-post-incident},{Key=incident-id,Value=2024-02-21}]' \
  --user-data file://init-script.sh

If you don’t have a hardened AMI pipeline, this incident is the forcing function to build one. Rebuilding from a generic AMI means re-running your full configuration management stack and hoping nothing drifts. Rebuilding from a known-good hardened baseline means launching and verifying.

Recovery Sequence

dev → staging → prod

Not prod first. Not all at once.

Bring dev back up. Verify monitoring and alerting are functional — specifically, verify that the detection that fired during this incident still fires in dev. If you can’t reproduce the detection in dev, you don’t know if it’s working.

Promote to staging. Run your standard smoke tests plus whatever you added to your detection suite based on this incident.

Promote to prod only after staging has been clean for at least four hours.

---

The Post-Incident Review

Schedule it within 72 hours of resolution. Not a blame session — a timeline reconstruction and process improvement meeting. What to document:

Timeline reconstruction (to the minute):

Time	Event	Who	Evidence Source
Feb 21 12:47	Initial compromise — credential used from unexpected IP	Attacker	CloudTrail
Feb 21 12:51	First AssumeRole to production role	Attacker	CloudTrail
Feb 21 13:15	S3 ListBuckets on customer-data bucket	Attacker	CloudTrail data events
Feb 21 21:30	GuardDuty fires: UnauthorizedAccess:IAMUser/AnomalousBehavior	GuardDuty	GuardDuty finding
Feb 21 21:35	On-call engineer acknowledges alert	SRE	PagerDuty
Feb 21 21:50	Incident declared, channel created	IR lead	Slack

Key metrics to measure and improve:

• Mean Time to Detect (MTTD): Time between initial compromise and first alert
• Mean Time to Declare (MTTDeclare): Time between first alert and formal incident declaration
• Mean Time to Contain (MTTC): Time between declaration and credential revocation + network isolation
• Blast radius: Accounts, services, data classifications confirmed in scope

Regulatory notification requirements (know these before the incident):

• GDPR: 72 hours from discovery to supervisory authority notification
• HIPAA: 60 days from discovery to individual notification; 60 days to HHS for breaches affecting 500+ individuals
• CCPA: “expedient” notification to individuals; no fixed statutory window for regulator notification but AG guidance suggests 72 hours
• SEC (public companies): 4 business days from determining the incident is “material”
• Check your state breach notification laws — 50 states, 50 different windows

---

⚠ Production Gotchas

Revoking a credential mid-operation breaks running jobs. If the compromised IAM role is used by production services, the deny-all policy will immediately break those services. Have a plan for emergency credential rotation before you act — either a separate role for legitimate services or a maintenance window. The contain-vs-service-availability tradeoff is a real one; make it deliberately, document it in the incident log.

CloudTrail data events are not enabled by default. Management events (API calls like CreateUser, RunInstances, AssumeRole) are enabled. Data events (S3 GetObject, Lambda function invocations, DynamoDB item-level activity) must be explicitly enabled and cost extra. If you discover during an incident that you needed S3 data events and didn’t have them, you cannot reconstruct what data the attacker accessed. Enable them before the incident.

Forensic snapshots cost money. EBS snapshot storage is not free, and snapshotting every volume on every compromised instance adds up. Have a pre-approved IR budget that includes forensic snapshot costs — getting financial approval in the middle of an active incident is a delay you don’t want.

Legal hold means don’t delete anything. Once legal is involved, no evidence can be destroyed without legal clearance. That includes the compromised EC2 instances, the forensic snapshots, the log exports, and the incident Slack channel. Set legal-hold tags on all IR artifacts immediately and don’t clean up until legal explicitly says to.

The attacker may still be in. Containment removes one credential and one network path. If the attacker established multiple persistence mechanisms before you detected them, containment is the beginning of the eradication phase, not the end. Assume they’re still in until the persistence hunt is complete.

Multi-account blast radius compounds quickly. AssumeRole chains can cross account boundaries. A compromised role in account A that can assume a role in account B means the blast radius spans both accounts, and CloudTrail logging in account A does not show what the attacker did after assuming the role in account B. Pull CloudTrail from every account in the blast radius.

---

Quick Reference: IR Checklist — First 24 Hours

Hour 0–1: Declare and Scope

• [ ] Declare incident — do not investigate quietly
• [ ] Notify: CISO, Legal, on-call SRE lead
• [ ] Create incident Slack channel: #incident-YYYY-MM-DD-descriptor
• [ ] Start timestamped incident log (shared doc, assign scribe)
• [ ] Query CloudTrail: last 1–2 hours of suspected principal activity
• [ ] Check GuardDuty for active findings
• [ ] Answer: active or historical? blast radius? data at risk?

Hour 1–4: Preserve, Then Contain

• [ ] FIRST: Snapshot all volumes on compromised EC2 instances
• [ ] FIRST: Export CloudTrail logs for incident window to IR evidence directory
• [ ] FIRST: Export VPC Flow Logs for incident window
• [ ] Revoke compromised IAM credential (disable key or attach deny-all policy)
• [ ] For role sessions: use DateLessThan condition to invalidate active sessions
• [ ] Apply isolation security group to compromised EC2 instances (do NOT terminate)
• [ ] Cordon compromised Kubernetes nodes (do NOT delete pods before forensic capture)
• [ ] Collect pod forensics: spec, env vars, process list, network connections

Hour 4–12: Investigate

• [ ] Trace AssumeRole chain from compromised principal — build the lateral movement graph
• [ ] Query S3 GetObject, GetSecretValue, Decrypt events for data access scope
• [ ] Hunt persistence: new IAM users/roles, new access keys, Lambda modifications
• [ ] Check EC2 user-data modifications, new CloudFormation stacks
• [ ] Pull Tetragon/Falco evidence if available — process lineage and connection logs
• [ ] Cross-account check: pull CloudTrail from every account reached via AssumeRole

Hour 12–24: Eradicate and Recover

• [ ] Delete all unauthorized IAM users/roles/access keys created during incident
• [ ] Rotate ALL credentials in the blast radius (not just the initial compromise)
• [ ] Remove Lambda backdoors — replace entire deployment package, reset environment
• [ ] Replace compromised instances from known-good hardened AMI (do not patch-in-place)
• [ ] Recover: dev → staging → prod. Verify detection fires in dev before promoting.
• [ ] Declare all-clear only after monitoring shows clean in prod for 4+ hours

Ongoing: Regulatory and Communication

• [ ] Log discovery time — regulatory clocks (GDPR 72h, HIPAA 60d) start at discovery
• [ ] Legal hold on all IR artifacts — do not delete without legal clearance
• [ ] Schedule post-incident review within 72 hours of resolution
• [ ] Update this playbook before the next incident

---

Key Takeaways

• A cloud incident response playbook only works if it has been rehearsed before the incident — the ChangeHealthcare attack showed that nine days of undetected dwell time transforms a credential theft into a national healthcare disruption
• Preserve before you contain: snapshot volumes and export logs before revoking credentials or isolating instances — forensic evidence destroyed during hasty containment cannot be reconstructed
• The contain-vs-evidence tension is real and deliberate: isolated EC2 instances remain available for forensic access via IR bastion; terminated instances do not
• CloudTrail data events (S3 GetObject, Lambda invocations) are not enabled by default — if you need them during an incident and haven’t pre-enabled them, your data access scope is unknown
• Recovery sequence is dev → staging → prod, and you verify detection fires in dev before promoting — if you can’t reproduce the detection that caught the original incident, you don’t know if it still works

---

What’s Next

This playbook is reactive. You run it after something goes wrong. EP13 is about making it proactive — running structured attack simulations against your own infrastructure on a regular cadence so the first time your team works through this sequence is not during an actual breach. Continuous purple team testing means your IR team has muscle memory for the playbook, your detection tooling is validated against real attack patterns, and your blast radius assumptions are tested before an attacker tests them for you.

Get EP13 in your inbox when it publishes → subscribe at linuxcent.com