What is purple team security → OWASP Top 10 mapped to cloud infrastructure → Cloud security breaches 2020–2025 → Broken access control in AWS → MFA fatigue attacks → CI/CD secrets exposure

TL;DR

• CI/CD secrets exposure is OWASP A08 + A02: credentials committed to repositories or stored in pipeline environment variables can be exfiltrated when the platform is compromised, and automated scanners find them within seconds of a public commit
• The CircleCI breach (January 2023): an engineer’s laptop was compromised via malware → session token stolen → attacker accessed CircleCI production systems → all customer environment variables (AWS keys, GitHub tokens, SSH keys) exfiltrated
• The structural problem: long-lived credentials stored in a CI/CD platform are only as secure as the platform itself — if the platform is compromised, all stored secrets are compromised
• The structural fix: OIDC workload identity replaces stored credentials with short-lived tokens issued at job runtime — there is nothing to exfiltrate
• Pre-commit hooks and CI-layer secret scanning are detection layers, not structural fixes — they catch accidents, not determined attackers
• Automated secret scanners (TruffleHog, Gitleaks) find credentials in public repos within 60–90 seconds of commit

OWASP Mapping: A08 Software and Data Integrity Failures — build pipeline integrity. A02 Cryptographic Failures — secrets stored in ways that allow exfiltration.

The Big Picture

┌─────────────────────────────────────────────────────────────────────┐
│                  CI/CD SECRETS ATTACK SURFACE                       │
│                                                                     │
│   VECTOR 1: COMMITTED TO VCS                                        │
│   Developer ── git commit ──▶ .env with AWS_SECRET_KEY              │
│   Automated scanner ──────▶  clones within 60 seconds              │
│   Attacker ───────────────▶  accesses AWS before dev notices        │
│                                                                     │
│   VECTOR 2: STORED IN CI/CD PLATFORM                                │
│   DevOps ─── configures ──▶  AWS_ACCESS_KEY_ID in CircleCI         │
│   Attacker compromises CircleCI → exfiltrates all org env vars      │
│                                                                     │
│   VECTOR 3: IN CONTAINER/PROCESS ENV                                │
│   kubectl exec / docker inspect ──▶  printenv shows credentials     │
│   Anyone with container exec access = credential access             │
│                                                                     │
│   VECTOR 4: IN BUILD ARTIFACTS / LOGS                               │
│   Build log: "Using token: ghp_xxxxxxxxxxxx..." → exposed in log   │
│                                                                     │
│   ═══════════════════════════════════════════════════════           │
│   STRUCTURAL FIX: OIDC WORKLOAD IDENTITY                            │
│   No stored credential → nothing to commit, nothing to exfiltrate  │
│   CI job requests token at runtime → 1-hour TTL → expired          │
│                                                                     │
└─────────────────────────────────────────────────────────────────────┘

CI/CD secrets exposure is not primarily a developer discipline problem — it is a structural problem. When credentials are stored in a CI/CD platform, in environment variables, or in version control, the only question is when they will be exposed, not whether. The structural answer replaces stored credentials with dynamically issued, short-lived tokens that cannot be exfiltrated because they don’t persist.

The 25-Minute Compromise: How Automated Scanning Works Against You

At 2:47 AM, a developer committed a .env file to a public GitHub repository. It contained:

DATABASE_URL=postgres://admin:prod_p@ssw0rd@db.internal.company.com:5432/customers
AWS_ACCESS_KEY_ID=AKIAIOSFODNN7EXAMPLE
AWS_SECRET_ACCESS_KEY=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
STRIPE_SECRET_KEY=sk_live_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
GITHUB_TOKEN=ghp_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

At 2:48 AM — 60 seconds later — an automated scanner had cloned the repository. These scanners run continuously against GitHub’s public event stream, looking for credential patterns in new commits, new files, and new repository forks.

At 3:12 AM — 25 minutes after the commit — the database started receiving unusual queries. The automated scanning infrastructure is not operated by individuals manually watching for leaks. It is fully automated: pattern match → clone → test credential validity → if valid, begin exploitation or sell.

GitHub now runs its own secret scanning and immediately invalidates some credential types (GitHub tokens, AWS IAM keys partnered with AWS) when detected in public repositories. This covers a subset of credential types. It does not cover database passwords, service-specific tokens for non-partnered services, or private repository commits that become public via fork.

The CircleCI Breach: Platform-Level Credential Exfiltration

The CircleCI breach (January 2023) is the definitive example of CI/CD platform-level secrets exposure. The attack chain:

1. CircleCI engineer's laptop compromised via malware (initial vector not fully disclosed)
2. Malware steals a 2FA-authenticated SSO session token
3. Session token valid, not expired
4. Attacker uses session token to authenticate to CircleCI internal systems
5. From internal access, attacker reaches production database
6. Production database contains encrypted customer secrets (environment variables)
7. Database also contains the encryption keys (in accessible internal system)
8. Attacker exfiltrates: encrypted secrets + encryption keys = plaintext secrets

What was stored in CircleCI environment variables by customers:
– AWS IAM access key ID and secret access key pairs
– GitHub personal access tokens and OAuth tokens
– DockerHub credentials
– SSH private keys (for deployment access)
– Heroku API keys
– Stripe, Twilio, SendGrid API keys
– Internal service account credentials

CircleCI could not determine which customer secrets were accessed and which were not — they notified all customers to rotate all credentials stored in their system.

The scale of the blast radius: Any customer who had stored long-lived credentials in CircleCI environment variables was potentially compromised. The credential was valid. The CircleCI platform’s encryption only protected against offline attacks — an attacker with internal database access and access to the key management system had everything needed to decrypt.

Red Phase: Enumerating Secrets Exposure in Your Pipeline

Scanning Repositories for Committed Secrets

# Install: pip install trufflehog3 or use the Docker image
docker run --rm \
  -v "$(pwd):/repo" \
  trufflesecurity/trufflehog:latest \
  git file:///repo \
  --json \
  --only-verified \
  2>/dev/null | \
  jq '{
    file: .SourceMetadata.Data.Git.file,
    commit: .SourceMetadata.Data.Git.commit,
    detector: .DetectorName,
    verified: .Verified,
    line: .SourceMetadata.Data.Git.line
  }'

# Gitleaks: alternative scanner with SARIF output for CI integration
gitleaks detect \
  --source . \
  --report-format sarif \
  --report-path gitleaks-report.sarif \
  --verbose

# Or: scan entire git history (catches secrets that were committed then deleted)
gitleaks detect \
  --source . \
  --log-opts="--all" \
  --report-format json \
  --report-path gitleaks-history.json

# Scan a specific GitHub organization's public repositories
# (test your own org before red team exercises)
trufflehog github \
  --org your-github-org \
  --token "${GITHUB_TOKEN}" \
  --json \
  --only-verified \
  2>/dev/null | \
  jq '{
    repo: .SourceMetadata.Data.Github.repository,
    file: .SourceMetadata.Data.Github.file,
    detector: .DetectorName,
    verified: .Verified
  }'

Enumerating Secrets in CI/CD Platform Environment Variables

# GitHub Actions: list secrets defined in a repository
# (shows names only — values are not returned by API, but names reveal what's stored)
curl -H "Authorization: Bearer ${GITHUB_TOKEN}" \
  -H "Accept: application/vnd.github+json" \
  "https://api.github.com/repos/your-org/your-repo/actions/secrets" | \
  jq '.secrets[] | {name: .name, updated: .updated_at}'

# GitHub Actions: list organization-level secrets
curl -H "Authorization: Bearer ${GITHUB_TOKEN}" \
  -H "Accept: application/vnd.github+json" \
  "https://api.github.com/orgs/your-org/actions/secrets" | \
  jq '.secrets[] | {name: .name, visibility: .visibility, updated: .updated_at}'

# Check for credentials in running pod environment variables (Kubernetes)
# This is what an attacker with kubectl exec access would do
kubectl get pods -A -o json | \
  jq -r '.items[] | 
    .metadata.namespace + "/" + .metadata.name + ": " + 
    ([.spec.containers[].env[]? | 
      select(.name | test("KEY|SECRET|TOKEN|PASSWORD|CREDENTIAL|API"; "i")) |
      .name
    ] | join(", "))' | \
  grep -v ": $"  # Only show pods with matching env var names

Testing Whether AWS Keys in CI/CD Are Over-Permissioned

# If you find an AWS access key in a scan — test its permissions
# (on your own test account's keys only)
aws sts get-caller-identity
# Returns: account, user/role ARN, caller ID

# What can this key do?
aws iam simulate-principal-policy \
  --policy-source-arn $(aws sts get-caller-identity --query Arn --output text) \
  --action-names "s3:*" "ec2:*" "iam:*" "sts:AssumeRole" \
  --query 'EvaluationResults[?EvalDecision==`allowed`].EvalActionName' \
  --output text

Blue Phase: Detection Across the Secret Lifecycle

GitHub Secret Scanning Alerts

# List secret scanning alerts in a repository via GitHub API
curl -H "Authorization: Bearer ${GITHUB_TOKEN}" \
  -H "Accept: application/vnd.github+json" \
  "https://api.github.com/repos/your-org/your-repo/secret-scanning/alerts?state=open" | \
  jq '.[] | {
    type: .secret_type,
    state: .state,
    created: .created_at,
    url: .html_url
  }'

CloudTrail: Detecting API Activity from CI/CD Credentials

When a CI/CD credential is used by an attacker, the CloudTrail events show unusual patterns:

# Find API calls from CI/CD credentials outside normal working hours
# or from unexpected IPs (attacker using the stolen key)
aws cloudtrail lookup-events \
  --lookup-attributes AttributeKey=Username,AttributeValue=ci-deploy-user \
  --start-time "$(date -d '7 days ago' --iso-8601=seconds)" \
  --query 'Events[].{Time:EventTime,Name:EventName,IP:CloudTrailEvent}' \
  --output json | \
  jq '.[] | {
    time: .Time,
    event: .Name,
    ip: (.IP | fromjson | .sourceIPAddress),
    user_agent: (.IP | fromjson | .userAgent)
  }' | \
  jq 'select(.ip | test("^(10\\.|172\\.(1[6-9]|2[0-9]|3[01])\\.|192\\.168\\.)") | not)'
  # Filter: events from non-RFC1918 IPs (outside your known CI/CD IP ranges)

SIEM Query: Credential Used in Multiple Regions Simultaneously

A credential being used from multiple regions simultaneously is a strong indicator of compromise:

-- Athena query against CloudTrail logs
-- Detect: same access key used from multiple regions in same hour
SELECT
  userIdentity.accessKeyId,
  userIdentity.userName,
  COUNT(DISTINCT awsRegion) as region_count,
  ARRAY_AGG(DISTINCT awsRegion) as regions,
  COUNT(DISTINCT sourceIPAddress) as ip_count,
  ARRAY_AGG(DISTINCT sourceIPAddress) as source_ips,
  DATE_TRUNC('hour', from_iso8601_timestamp(eventTime)) as hour
FROM cloudtrail_logs
WHERE
  userIdentity.type = 'IAMUser'
  AND from_iso8601_timestamp(eventTime) > current_timestamp - interval '7' day
GROUP BY
  userIdentity.accessKeyId,
  userIdentity.userName,
  DATE_TRUNC('hour', from_iso8601_timestamp(eventTime))
HAVING COUNT(DISTINCT awsRegion) > 2
ORDER BY region_count DESC;

GuardDuty: Credential Exfiltration Indicators

# GuardDuty findings relevant to CI/CD credential compromise
DETECTOR_ID=$(aws guardduty list-detectors --query 'DetectorIds[0]' --output text)

aws guardduty list-findings \
  --detector-id "${DETECTOR_ID}" \
  --finding-criteria '{
    "Criterion": {
      "type": {
        "Equals": [
          "UnauthorizedAccess:IAMUser/TorIPCaller",
          "UnauthorizedAccess:IAMUser/MaliciousIPCaller",
          "Discovery:IAMUser/AnomalousBehavior",
          "Exfiltration:IAMUser/AnomalousBehavior",
          "CredentialAccess:IAMUser/AnomalousBehavior"
        ]
      }
    }
  }' \
  --query 'FindingIds' --output text | \
  xargs -n 10 aws guardduty get-findings \
    --detector-id "${DETECTOR_ID}" \
    --finding-ids | \
  jq '.Findings[] | {type: .Type, user: .Resource.AccessKeyDetails.UserName, severity: .Severity}'

Purple Phase: The Structural Fix

Fix 1: OIDC Workload Identity — Eliminate Stored Credentials

This is the structural solution. Instead of storing an AWS IAM access key in your CI/CD platform, the CI/CD job authenticates to AWS using an OIDC token issued by the CI/CD provider. AWS validates the token against a pre-configured trust policy and issues temporary credentials valid for the duration of the job.

The OIDC workload identity approach eliminates static cloud access keys entirely — there is no secret to commit, no secret to exfiltrate from the CI/CD platform, and no long-lived credential to rotate on breach.

GitHub Actions with AWS OIDC — complete setup:

# .github/workflows/deploy.yml
name: Deploy to AWS

on:
  push:
    branches: [main]

permissions:
  id-token: write   # Required for OIDC token request
  contents: read

jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Configure AWS credentials via OIDC
        uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: arn:aws:iam::123456789012:role/github-actions-deploy-role
          role-session-name: github-actions-${{ github.run_id }}
          aws-region: us-east-1
          # No AWS_ACCESS_KEY_ID or AWS_SECRET_ACCESS_KEY needed

      - name: Deploy
        run: aws s3 sync ./dist s3://your-bucket/

AWS IAM trust policy for GitHub Actions OIDC:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Federated": "arn:aws:iam::123456789012:oidc-provider/token.actions.githubusercontent.com"
      },
      "Action": "sts:AssumeRoleWithWebIdentity",
      "Condition": {
        "StringEquals": {
          "token.actions.githubusercontent.com:aud": "sts.amazonaws.com"
        },
        "StringLike": {
          "token.actions.githubusercontent.com:sub": "repo:your-org/your-repo:ref:refs/heads/main"
        }
      }
    }
  ]
}

# Create the OIDC provider in AWS (one-time setup)
aws iam create-open-id-connect-provider \
  --url https://token.actions.githubusercontent.com \
  --client-id-list sts.amazonaws.com \
  --thumbprint-list "6938fd4d98bab03faadb97b34396831e3780aea1"

# Create the IAM role with the trust policy above
aws iam create-role \
  --role-name github-actions-deploy-role \
  --assume-role-policy-document file://github-actions-trust-policy.json

# Attach a least-privilege policy to the role
aws iam attach-role-policy \
  --role-name github-actions-deploy-role \
  --policy-arn arn:aws:iam::123456789012:policy/deploy-policy

Fix 2: Pre-Commit Hooks — Catch Accidents Before They Reach VCS

Pre-commit hooks don’t stop a determined attacker. They catch accidents — the developer who forgets to move a .env file to .gitignore before staging all files.

# Install pre-commit framework
pip install pre-commit

# .pre-commit-config.yaml in your repository root
cat > .pre-commit-config.yaml << 'EOF'
repos:
  - repo: https://github.com/gitleaks/gitleaks
    rev: v8.18.4
    hooks:
      - id: gitleaks
        name: Detect hardcoded secrets
        entry: gitleaks protect --staged --redact --verbose
        language: golang
        pass_filenames: false

  - repo: https://github.com/pre-commit/pre-commit-hooks
    rev: v4.5.0
    hooks:
      - id: detect-private-key
      - id: check-added-large-files
        args: ['--maxkb=1000']
EOF

# Install the hooks in the local repository
pre-commit install

# Test against staged files
pre-commit run --all-files

Fix 3: CI-Layer Secret Scanning — Block Before Merge

# GitHub Actions: secret scanning as a required status check
# .github/workflows/secret-scan.yml
name: Secret Scan

on:
  pull_request:
    types: [opened, synchronize]

jobs:
  secret-scan:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0  # Full history for git log scanning

      - name: Run TruffleHog
        uses: trufflesecurity/trufflehog@main
        with:
          path: ./
          base: ${{ github.event.repository.default_branch }}
          head: HEAD
          extra_args: --only-verified --json

# GitLab CI: secret detection built-in template
include:
  - template: Security/Secret-Detection.gitlab-ci.yml

secret_detection:
  stage: test
  variables:
    SECRET_DETECTION_HISTORIC_SCAN: "true"  # Scan full history

Fix 4: Audit and Rotate Existing CI/CD Platform Secrets

After implementing OIDC, the migration path for existing stored credentials:

#!/bin/bash
# Purple Team EP06 — CI/CD Secrets Migration Audit
# Identifies AWS IAM keys stored in CI/CD that should be replaced with OIDC

echo "=== AWS IAM Keys Potentially Stored in CI/CD ==="
echo "--- Keys not used from expected CI/CD IPs in last 30 days ---"

# Get all IAM access keys
aws iam list-users --query 'Users[].UserName' --output text | tr '\t' '\n' | \
  while read user; do
    keys=$(aws iam list-access-keys --user-name "$user" \
      --query 'AccessKeyMetadata[?Status==`Active`].{Key:AccessKeyId,Created:CreateDate}' \
      --output json)

    if [ "$(echo "$keys" | jq length)" -gt 0 ]; then
      echo ""
      echo "User: $user"
      echo "$keys" | jq -r '.[] | "  Key: " + .Key + " | Created: " + .Created'

      # Check last used
      echo "$keys" | jq -r '.[].Key' | while read key_id; do
        last_used=$(aws iam get-access-key-last-used --access-key-id "$key_id" \
          --query 'AccessKeyLastUsed.{Date:LastUsedDate,Service:ServiceName,Region:Region}' \
          --output json)
        echo "  Last used: $(echo "$last_used" | jq -r '.Date // "Never"') | Service: $(echo "$last_used" | jq -r '.Service // "N/A"')"
      done
    fi
  done

echo ""
echo "=== MIGRATION CHECKLIST ==="
echo "  1. For each CI/CD IAM key above:"
echo "     a. Identify which CI/CD platform uses it"
echo "     b. Set up OIDC trust policy for that platform"
echo "     c. Update pipeline to use OIDC (no stored key)"
echo "     d. Disable and then delete the IAM key"
echo "     e. Verify pipelines still work"

Run This in Your Own Environment: Secrets Exposure Audit

#!/bin/bash
# Purple Team EP06 — CI/CD Secrets Exposure Audit
# Run from your workstation with git and trufflehog installed

echo "=== 1. Scan Local Repository for Committed Secrets ==="
if command -v trufflehog > /dev/null 2>&1; then
  trufflehog git file://$(pwd) --only-verified --json 2>/dev/null | \
    jq '{file: .SourceMetadata.Data.Git.file, detector: .DetectorName}' || \
    echo "  No verified secrets found in git history"
else
  echo "  Install trufflehog: pip install trufflehog3"
fi

echo ""
echo "=== 2. Check for .env Files in Git History ==="
git log --all --full-history -- "*.env" "**/.env" ".env.*" 2>/dev/null | \
  grep "^commit" | head -5 | \
  while read _ commit; do
    echo "  .env file committed: $commit"
    git show "$commit" --stat | head -3
  done

echo ""
echo "=== 3. Check Running Pods for Credential Env Vars (Kubernetes) ==="
if command -v kubectl > /dev/null 2>&1; then
  kubectl get pods -A -o json 2>/dev/null | \
    jq -r '.items[] | 
      .metadata.namespace + "/" + .metadata.name + ": " + 
      ([.spec.containers[].env[]? | 
        select(.name | test("KEY|SECRET|TOKEN|PASSWORD|CREDENTIAL"; "i")) |
        .name
      ] | join(", "))' | \
    grep -v ": $" | head -20
else
  echo "  kubectl not found"
fi

echo ""
echo "=== 4. GitHub Actions Secrets Inventory ==="
if [ -n "${GITHUB_TOKEN}" ]; then
  REPO="your-org/your-repo"  # Update this
  curl -s -H "Authorization: Bearer ${GITHUB_TOKEN}" \
    -H "Accept: application/vnd.github+json" \
    "https://api.github.com/repos/${REPO}/actions/secrets" | \
    jq '.secrets[] | {name: .name, updated: .updated_at}'
else
  echo "  Set GITHUB_TOKEN to enumerate repository secrets"
fi

Common Mistakes When Addressing CI/CD Secrets Exposure

Treating secret scanning as the primary control. TruffleHog and Gitleaks catch what gets committed. They do not prevent the CircleCI attack class — an attacker who compromises the CI/CD platform itself bypasses all scanning controls. Scanning is detection; OIDC workload identity is prevention.

Rotating compromised keys without checking CloudTrail for use. When a secret is exposed, the first question is not “rotate it” — it is “was it used?” Check CloudTrail for any API activity from the key between the suspected exposure time and the rotation. If the key was used, you have an active incident, not just a credential rotation task.

Using OIDC trust policies that are too broad. The GitHub Actions OIDC trust policy in the fix section uses a StringLike condition on the sub claim to scope to a specific repository and branch. If you use StringLike: "*" instead, any GitHub Actions job in any repository can assume your role. Always scope OIDC trust policies to the specific repository, branch, and environment that needs the access.

Not scanning git history — only the working tree. Secrets that were committed and then deleted are still in git history. git rm removes the file from the working tree but not from the object store. TruffleHog and Gitleaks scan history by default when given the --all flag. Scanning only the current working tree misses all historical exposures.

Forgetting third-party GitHub Actions. The supply chain attack surface includes the Actions you reference in your workflows. An Action pinned to a mutable tag (@main, @v1) can be changed by the maintainer. Pin to a specific commit SHA and verify the Action’s provenance.

# Vulnerable: mutable tag
- uses: aws-actions/configure-aws-credentials@v4

# Secure: pinned SHA
- uses: aws-actions/configure-aws-credentials@e3dd6a429d7300a6a4c196c26e831c1e4c763fe4

Quick Reference

Key Takeaways

• CI/CD secrets exposure is structural: long-lived credentials in a CI/CD platform are only as secure as that platform — the CircleCI breach proved that encryption alone is insufficient if the attacker can access the keys
• Automated secret scanners find publicly committed credentials within 60–90 seconds — rotation must happen faster than that or assume compromise
• Pre-commit hooks and CI secret scanning catch accidents; they do not prevent determined attackers who compromise the platform itself
• OIDC workload identity is the structural fix: no stored credential means no credential to exfiltrate
• When rotating a compromised key, check CloudTrail for usage between exposure and rotation before closing the incident
• OIDC trust policies must be scoped to specific repositories and branches — a wildcard trust policy recreates the exposure in a different form
• Pin third-party GitHub Actions to commit SHAs, not mutable tags — mutable tags are a supply chain attack surface

What’s Next

EP07 covers SSRF to cloud metadata: how an SSRF vulnerability in any application layer becomes a straight line to IAM credentials when IMDSv2 is not enforced. The Capital One breach anatomy — WAF SSRF → EC2 metadata → IAM role credentials → 100 million S3 records — in full technical detail, with the simulation commands and the one-line enforcement fix. If you’ve addressed identity and secrets, the network attack paths are where EP07 through EP10 focus.

Get EP07 in your inbox when it publishes → subscribe at linuxcent.com

The post CI/CD Secrets Exposure: How Supply Chain Attacks Target Your Pipeline appeared first on Linuxcent.

Introduction

The 2020–2022 period redefined what “secure Kubernetes” meant. A global pandemic moved workloads to cloud-native infrastructure faster than security practices could follow. SolarWinds happened. Log4Shell happened. The software supply chain became a crisis.

At the same time, the Kubernetes project was doing something it had been reluctant to do: removing APIs and features, including PodSecurityPolicy — the primary security primitive that most enterprise clusters depended on. The replacement was simpler, but the migration was not.

Kubernetes 1.19 — LTS Behavior, Ingress Stable (August 2020)

1.19 extended the support window to one year (from nine months). This was an acknowledgment that enterprise organizations couldn’t upgrade four times per year — a common complaint from operations teams.

• Ingress graduated to stable: networking.k8s.io/v1 — after years as a beta resource, Ingress finally had a stable API
• Immutable ConfigMaps and Secrets to beta: Configuration protection becomes broadly available
• EndpointSlices to GA: The replacement for Endpoints — shards pod-to-service mappings to avoid the single large Endpoints object that caused control plane stress at scale (10,000+ endpoints for a single service)
• Structured logging (alpha): Machine-parseable log output from Kubernetes control plane components — a prerequisite for reliable SIEM integration

# EndpointSlice: distributed representation of service endpoints
kubectl get endpointslices -n production -l kubernetes.io/service-name=api-service
NAME                  ADDRESSTYPE   PORTS   ENDPOINTS                                   AGE
api-service-abc12     IPv4          8080    10.0.1.5,10.0.1.6,10.0.1.7 + 47 more...   2d
api-service-def34     IPv4          8080    10.0.2.1,10.0.2.2,10.0.2.3 + 47 more...   2d

Kubernetes 1.20 — Dockershim Deprecated (December 2020)

The announcement in 1.20 that the Docker shim was deprecated caused more panic than any previous Kubernetes deprecation. The message was misread by many as “Kubernetes is dropping Docker support” — the PR catastrophe that followed required the Kubernetes blog to publish a dedicated clarification post.

The reality: Docker-built images continued to work on Kubernetes. What was being removed was the code in the kubelet that talked directly to Docker’s daemon using a non-standard interface, rather than through the Container Runtime Interface (CRI). Docker images conform to the OCI (Open Container Initiative) image specification — they run on any CRI-compliant runtime.

The migration path:
– containerd: The runtime that Docker itself used internally. Moving to containerd meant removing the Docker layer entirely — the kubelet talks directly to containerd via CRI
– CRI-O: An OCI-focused runtime designed specifically for Kubernetes, minimal and purpose-built

# Before (Docker socket): kubelet → dockershim → Docker daemon → containerd → runc
# After (direct CRI):     kubelet → containerd → runc
#                    or:  kubelet → CRI-O → runc

# Check runtime in use on a node
kubectl get node worker-1 -o jsonpath='{.status.nodeInfo.containerRuntimeVersion}'
# containerd://1.6.4

Also in 1.20:
– API Priority and Fairness beta: Rate-limit API server requests by priority — prevents a runaway controller from starving other API clients
– CronJobs stable: Scheduled jobs graduate after years in beta
– Volume snapshot stable

The SolarWinds Context (December 2020)

The SolarWinds supply chain attack, disclosed in December 2020, didn’t directly target Kubernetes. But it accelerated an existing conversation in the cloud-native community: if the build pipeline is compromised, signed binaries mean nothing. If the image registry is compromised, admission control on image names means nothing.

The attack catalyzed work on several fronts:
– Sigstore: An open-source project (Google, Red Hat, Purdue University) for signing and verifying software artifacts including container images
– SLSA (Supply chain Levels for Software Artifacts): A framework for incrementally improving supply chain security, from basic build provenance to hermetic builds with verified dependencies
– SBOM (Software Bill of Materials): A machine-readable inventory of software components in an image — required by US Executive Order 14028 (May 2021) for software sold to the federal government

Kubernetes 1.21 — PodSecurityPolicy Deprecation (April 2021)

PodSecurityPolicy was deprecated in 1.21, announcing its removal in 1.25. The deprecation was contentious — PSP was the only built-in mechanism for enforcing pod security constraints, and every security-conscious cluster depended on it, despite its many flaws.

The replacement approach: Pod Security Standards — three predefined security profiles:

Other 1.21 highlights:
– CronJobs stable
– Immutable ConfigMaps and Secrets stable
– Graceful node shutdown beta: The kubelet gracefully terminates pods when a node shuts down (not just when the kubelet stops)
– PodDisruptionBudget stable

Kubernetes 1.22 — The Great API Removal (August 2021)

1.22 was the most disruptive Kubernetes release for operations teams since 1.0. Several long-lived beta APIs were removed:

Teams with Helm charts, Terraform modules, and CI/CD pipelines built against beta API versions had to update their manifests. This was the moment that finally drove home the message: beta APIs in Kubernetes are not stable — they will be removed.

Also in 1.22:
– Server-Side Apply stable: Apply semantics moved server-side — field ownership tracking, conflict detection, and merge strategies are handled by the API server rather than client-side kubectl
– Memory manager stable: Better NUMA-aware memory allocation for latency-sensitive workloads
– Bound Service Account Token Volumes stable: Time-limited, audience-bound tokens for pods — replacing the long-lived, cluster-wide service account tokens that were a persistent security concern

# Bound service account token — expires, audience-restricted
# Projected volume mounts a time-limited token (default 1h expiry)
volumes:
- name: token
  projected:
    sources:
    - serviceAccountToken:
        audience: api
        expirationSeconds: 3600
        path: token

The bound token change was significant from a security perspective: previously, a service account token extracted from a pod would be valid indefinitely, for any audience. Projected tokens expire and are tied to a specific audience.

Pod Security Admission (Kubernetes 1.22, GA in 1.25)

The replacement for PodSecurityPolicy was Pod Security Admission — an admission controller built into the API server (no webhook required) that enforces the three Pod Security Standards at the namespace level:

# Namespace-level security enforcement
apiVersion: v1
kind: Namespace
metadata:
  name: production
  labels:
    pod-security.kubernetes.io/enforce: restricted
    pod-security.kubernetes.io/enforce-version: v1.25
    pod-security.kubernetes.io/warn: restricted
    pod-security.kubernetes.io/warn-version: v1.25

The three modes:
– enforce: Reject pods that violate the policy
– audit: Allow the pod but add an audit annotation
– warn: Allow the pod and send a warning to the client

Pod Security Admission is deliberately simpler than PSP. It does less — it enforces three fixed profiles, not arbitrary rules. For arbitrary policy, you still need OPA/Gatekeeper or Kyverno. But the simplicity means it works reliably, with no authorization edge cases.

Kubernetes 1.23 — Dual-Stack Stable, HPA v2 Stable (December 2021)

• IPv4/IPv6 dual-stack stable: Pods and Services can have both IPv4 and IPv6 addresses — critical for organizations running mixed-stack networks or migrating from IPv4 to IPv6
• HPA v2 stable: Horizontal Pod Autoscaler with support for multiple metrics (CPU, memory, custom metrics from Prometheus, external metrics). Scale on Prometheus metrics, not just CPU:

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: api-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: api
  minReplicas: 2
  maxReplicas: 20
  metrics:
  - type: Pods
    pods:
      metric:
        name: http_requests_per_second
      target:
        type: AverageValue
        averageValue: 1000m

• FlexVolume deprecated (in favor of CSI): Another step in the driver out-of-tree migration

The Log4Shell Moment (December 2021)

Log4Shell (CVE-2021-44228) hit on December 9, 2021. The vulnerability allowed unauthenticated remote code execution in any Java application using Log4j 2.x. The blast radius was enormous — Log4j was in everything.

For Kubernetes operators, Log4Shell crystallized several operational realities:

Inventory problem: Do you know which of your pods is running a Java application? Do you know which version of Log4j it includes? Without an SBOM pipeline and admission-time image scanning, you probably don’t have a reliable answer.

Patch velocity problem: Once you know which images are vulnerable, how quickly can you rebuild and redeploy? Organizations with GitOps pipelines and image update automation (Flux’s image reflector, ArgoCD Image Updater) could respond in hours. Organizations without this infrastructure measured response time in days.

Runtime detection problem: Can you detect exploitation attempts in real time? Falco rules for Log4Shell JNDI lookup patterns were available within hours of disclosure — but only organizations already running Falco could use them.

Log4Shell made the case for supply chain security, image scanning, SBOM generation, and runtime detection tooling more effectively than any conference talk.

Sigstore and the Supply Chain Response

In 2021, Sigstore reached a point where its tooling — cosign (image signing), rekor (transparency log), fulcio (keyless signing via OIDC) — was production-ready.

The keyless signing model was significant: instead of managing long-lived signing keys (which themselves become a supply chain risk), fulcio issues short-lived certificates tied to an OIDC identity (a GitHub Actions workflow, a GitLab CI job). The signature proves that a specific workflow built the image.

# Sign an image as part of CI (keyless, OIDC-based)
cosign sign --yes ghcr.io/org/app:v1.0.0

# Verify before deploying
cosign verify \
  --certificate-identity-regexp "https://github.com/org/app/.github/workflows/build.yml" \
  --certificate-oidc-issuer https://token.actions.githubusercontent.com \
  ghcr.io/org/app:v1.0.0

Policy engines (OPA/Gatekeeper, Kyverno) could be configured to reject pods using unsigned or unverified images at admission time — closing the loop from build provenance to runtime enforcement.

Key Takeaways

• Dockershim deprecation in 1.20 was about removing the non-standard interface, not about dropping Docker image compatibility — containers built with Docker run on containerd or CRI-O without changes
• The API removals in 1.22 were operationally painful but necessary — beta APIs in Kubernetes are not production-stable commitments
• Pod Security Admission (PSP’s replacement) trades power for reliability — three fixed profiles enforced at the namespace level, built into the API server, no authorization edge cases
• SolarWinds and Log4Shell made supply chain security a board-level concern; Sigstore, SBOM, and admission-time image verification moved from “nice to have” to operational requirements
• Bound service account tokens (1.22 stable) addressed a persistent security gap: pod tokens that expire and are audience-restricted rather than long-lived cluster-wide credentials

What’s Next

← EP04: The Operator Era | EP06: The Runtime Reckoning →

Series: Kubernetes: From Borg to Platform Engineering | linuxcent.com

The post Security Hardens: Supply Chain, Pod Security, and the API Cleanup (2020–2022) appeared first on Linuxcent.