9 min read

EP01: Product Quality vs Code Quality · All The Legible Repo Episodes →

This series is about the quality layer your CI can’t see. Each episode takes one failure that linters, scanners, and test suites never catch, shows the incident that proves it, and ends with one command you can run today. This opener names the problem — and introduces the gate that measures it.

Table of Contents

• TL;DR
• Quick Check: What Grade Is Your Repo Right Now
• The 500 Nobody Reported
• Where Product Quality Sits (and Why Linters Can’t See It)
• The Questions That Decide Whether a Stranger Stays
• How the Gate Works: Scorecard Plus Cold-Start
• What This Means for Your Repos Right Now
• Production Gotchas
• Quick Reference
• Framework Alignment
• Key Takeaways
• What’s Next

TL;DR

• Product quality vs code quality is the gap between “the tests pass” and “a stranger can actually use this” — and no linter measures it.
• The costliest failures are silent: the person who hits friction in the first ten minutes never files an issue. They close the tab.
• Legibility is checkable: README length, copy-paste quickstart, .env.example, actionable errors, a published artifact that still installs today.
• Invigil (Apache-2.0) mechanizes ~35 of these checks into gate levels G1–G7 with a letter grade — and prints the exact fix for every failure.
• It grades itself in CI: a pull request that lowers Invigil’s own score doesn’t merge.

Quick Check: What Grade Is Your Repo Right Now

Before the story, the evidence. Two commands, two minutes, on any repo you maintain:

pip install invigil
invigil score . --offline

Sample output, annotated:

Invigil — myproject
Gate G2 · Grade C+ · 19/27 (70%)          ← gate = maturity rung, grade = weighted score

FAIL [G1] README is a landing page (≤300 lines)     (effort: minutes)
      fix: move deep-dive sections to docs/; keep quickstart + pitch
FAIL [G1] .env.example documents every config var   (effort: minutes)
      fix: create .env.example listing each var with purpose + default
FAIL [G2] Errors carry a correlation ID             (effort: hours)
      fix: add a global exception handler returning {"error_id": ...}

Every failing line names the check, the effort class, and the exact fix. However you feel about the individual opinions, notice what just happened: nothing in your existing CI produces this view.

The 500 Nobody Reported

The day I renamed a package and pushed the new wheel, every UI page it served returned a 500. The commit message said “verified.” I had verified the import — not the experience. No test caught it, because the tests ran against my source tree, not against the artifact a stranger downloads. And no user caught it for me. The first stranger who hit that 500 did what strangers do: closed the tab and never came back.

That is the failure mode that should keep maintainers up at night. Absence of complaints is not absence of problems. Silence is the loudest negative signal a project gets.

A clean-virtualenv install from an empty directory found the bug in minutes. That habit — being your own first angry user — became a doctrine. Later, the doctrine became a CI gate called Invigil, because habits don’t run nightly and machines do.

Where Product Quality Sits (and Why Linters Can’t See It)

                    ┌─────────────────────────────────────────┐
                    │        WHAT YOUR CI CHECKS TODAY        │
                    │  ruff / eslint      → code style        │
                    │  pytest / jest      → source behavior   │
                    │  Trivy / Dependabot → CVEs, deps        │
                    │  Scorecard          → supply chain      │
                    └────────────────┬────────────────────────┘
                                     │  all green ✅
                                     ▼
                    ┌─────────────────────────────────────────┐
                    │        WHAT THE STRANGER MEETS          │
                    │  README (landing page or wall of text?) │
                    │  Quickstart (works from empty dir?)     │
                    │  Published artifact (installs TODAY?)   │
                    │  First error (fix included or trace?)   │
                    │  llms.txt / AGENTS.md (agent-readable?) │
                    └─────────────────────────────────────────┘
                          nothing above checks this layer

The product quality vs code quality distinction is exactly this diagram. As a result, a repo can be immaculate in the top box and unusable in the bottom one — green CI, linted code, zero CVEs, and a quickstart that fails on the first copy-paste. In contrast to code quality, product quality has no reflexive tooling. Every good maintainer checks these things by hand, occasionally, when they remember. Nobody’s CI does it on every pull request.

I build hardened infrastructure for a living, and the same lesson repeats there: a standard that isn’t enforced mechanically is a wish. That’s why Linux hardening as code beats hardening runbooks — and it’s why legibility needs a gate, not a checklist.

The Questions That Decide Whether a Stranger Stays

Specifically, the gate asks the questions your CI never asks:

• Can someone get from “found the repo” to “it worked on my machine” in ten minutes?
• When something fails, does the error include the fix — or a traceback?
• Is the README a landing page, or 600 lines of accumulated documentation?
• Does the artifact you published still install today, after your dependencies drifted?
• Is there an .env.example, or do users reverse-engineer your config from source?
• Can an AI agent — now often the first reader — parse your llms.txt and AGENTS.md without hitting stale paths or a leaked key?

Each question maps to a mechanical check. Together, ~35 checks roll up into gate levels G1–G7 — a maturity ladder, not a binary pass/fail — plus a weighted letter grade. A repo reaches gate Gn only when every mandatory check at or below n passes.

How the Gate Works: Scorecard Plus Cold-Start

Layer 1 — the scorecard (every PR, seconds)

The static layer inspects the repo and its metadata: LICENSE, README length, quickstart shape, tracked secrets, SHA-pinned actions, enforced lockfile, coverage floor, docs index, llms.txt/AGENTS.md hygiene, and more. It runs offline in a pre-commit hook in roughly 120 ms, because a gate that adds friction is a gate that gets uninstalled.

invigil score . --format markdown   # PR-comment-ready table

Layer 2 — the cold-start gate (nightly)

This is the layer that would have caught my 500. Instead of testing the source tree, it boots the published artifact — the wheel on PyPI, the image on GHCR — on a clean runner and probes its surface within a ten-minute budget:

# .invigil.yml
artifacts:
  - { type: pypi, name: "myapp[all]" }
  - { type: ghcr, image: ghcr.io/me/myapp:latest, port: 8000 }
probes:
  - { url: "/", expect_status: 200 }

Because it installs from the real registry into a real empty environment, it catches the class of bug where CI passes but the shipped thing is broken: the missing template directory, the config default pointing at localhost, the dependency that resolved differently after an upstream release.

What This Means for Your Repos Right Now

Start in report-only mode. The progressive profile scores everything and gates nothing — you get the visibility without a wall of red blocking your next merge. Flip to enforce once the grade stabilizes, the same way you’d introduce any merge check.

The doctrine is opinionated, and that’s deliberate — but the gate bends instead of breaking. Profiles (strict | progressive | light), per-check weights, and optional flags let a team disagree with a specific opinion without forking the tool. Additionally, network-dependent checks that time out become SKIPs excluded from the grade — never a false downgrade that erodes trust in the number.

One more thing, because trust matters for a tool that grades others: Invigil grades itself in CI. A pull request that lowers its own score won’t merge. The gate passes its own gate — currently G5, grade A+.

Production Gotchas

Enforcing on day one. Turning on enforce: true before the team has seen the report produces a wall of failures and an uninstall. What breaks: adoption. How to detect it: grumbling in your PR comments. The fix: progressive first, enforce after two weeks of stable grades.

Treating the grade as the goal. The grade is a proxy for a stranger’s first ten minutes. Gaming it (a hollow .env.example, a README split that hides the quickstart) passes the check and still loses the user. The fix is cultural, not mechanical — review the fix, not just the score delta.

Skipping the cold-start layer because “CI already tests installs.” CI installs from the source tree with your lockfile present. The stranger installs from the registry into nothing. These diverge silently after any packaging change — that divergence is invisible until you test the published artifact itself.

Quick Reference

Framework Alignment

Key Takeaways

• Product quality and code quality are different layers; your CI only watches one of them.
• Test the experience, not the import — the developer’s machine is a lie, and so is the source tree.
• Silence is data: the users you never hear from are the ones who hit the friction.
• A quality bar you can’t measure is an opinion; a gate you bypass is dead weight — make it fast, bendable, and report-only by default.
• Trust tools that hold themselves to their own standard: Invigil’s own PRs merge only if its self-grade holds.
• Defaults are never neutral — the same reason cloud AMI security risks demand custom images applies to your repo’s out-of-the-box experience.

What’s Next

EP02 goes deep on the layer that caught my 500: testing the published artifact, not the source tree. Clean-runner boots, real-registry installs, probe budgets — and why “it works in CI” is a statement about your lockfile, not your users. EP02: How to Test Your Published PyPI Package — Before a Stranger Does.

Invigil is Apache-2.0 and built in the open. If this episode named a failure you’ve shipped (we all have), there are more checks waiting to be written — good-first-issues with acceptance criteria at github.com/invigil/invigil. Pick one, or open a Discussion and say hello. First-time contributors get fast reviews and release-notes credit.

Get EP02 in your inbox when it publishes → subscribe

The post Product Quality vs Code Quality: Why Your Green CI Still Loses Users appeared first on Linuxcent.

What is purple team security → OWASP Top 10 mapped to cloud infrastructure → Cloud security breaches 2020–2025 → Broken access control in AWS → MFA fatigue attacks → CI/CD secrets exposure

TL;DR

• CI/CD secrets exposure is OWASP A08 + A02: credentials committed to repositories or stored in pipeline environment variables can be exfiltrated when the platform is compromised, and automated scanners find them within seconds of a public commit
• The CircleCI breach (January 2023): an engineer’s laptop was compromised via malware → session token stolen → attacker accessed CircleCI production systems → all customer environment variables (AWS keys, GitHub tokens, SSH keys) exfiltrated
• The structural problem: long-lived credentials stored in a CI/CD platform are only as secure as the platform itself — if the platform is compromised, all stored secrets are compromised
• The structural fix: OIDC workload identity replaces stored credentials with short-lived tokens issued at job runtime — there is nothing to exfiltrate
• Pre-commit hooks and CI-layer secret scanning are detection layers, not structural fixes — they catch accidents, not determined attackers
• Automated secret scanners (TruffleHog, Gitleaks) find credentials in public repos within 60–90 seconds of commit

OWASP Mapping: A08 Software and Data Integrity Failures — build pipeline integrity. A02 Cryptographic Failures — secrets stored in ways that allow exfiltration.

The Big Picture

┌─────────────────────────────────────────────────────────────────────┐
│                  CI/CD SECRETS ATTACK SURFACE                       │
│                                                                     │
│   VECTOR 1: COMMITTED TO VCS                                        │
│   Developer ── git commit ──▶ .env with AWS_SECRET_KEY              │
│   Automated scanner ──────▶  clones within 60 seconds              │
│   Attacker ───────────────▶  accesses AWS before dev notices        │
│                                                                     │
│   VECTOR 2: STORED IN CI/CD PLATFORM                                │
│   DevOps ─── configures ──▶  AWS_ACCESS_KEY_ID in CircleCI         │
│   Attacker compromises CircleCI → exfiltrates all org env vars      │
│                                                                     │
│   VECTOR 3: IN CONTAINER/PROCESS ENV                                │
│   kubectl exec / docker inspect ──▶  printenv shows credentials     │
│   Anyone with container exec access = credential access             │
│                                                                     │
│   VECTOR 4: IN BUILD ARTIFACTS / LOGS                               │
│   Build log: "Using token: ghp_xxxxxxxxxxxx..." → exposed in log   │
│                                                                     │
│   ═══════════════════════════════════════════════════════           │
│   STRUCTURAL FIX: OIDC WORKLOAD IDENTITY                            │
│   No stored credential → nothing to commit, nothing to exfiltrate  │
│   CI job requests token at runtime → 1-hour TTL → expired          │
│                                                                     │
└─────────────────────────────────────────────────────────────────────┘

CI/CD secrets exposure is not primarily a developer discipline problem — it is a structural problem. When credentials are stored in a CI/CD platform, in environment variables, or in version control, the only question is when they will be exposed, not whether. The structural answer replaces stored credentials with dynamically issued, short-lived tokens that cannot be exfiltrated because they don’t persist.

The 25-Minute Compromise: How Automated Scanning Works Against You

At 2:47 AM, a developer committed a .env file to a public GitHub repository. It contained:

DATABASE_URL=postgres://admin:prod_p@ssw0rd@db.internal.company.com:5432/customers
AWS_ACCESS_KEY_ID=AKIAIOSFODNN7EXAMPLE
AWS_SECRET_ACCESS_KEY=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
STRIPE_SECRET_KEY=sk_live_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
GITHUB_TOKEN=ghp_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

At 2:48 AM — 60 seconds later — an automated scanner had cloned the repository. These scanners run continuously against GitHub’s public event stream, looking for credential patterns in new commits, new files, and new repository forks.

At 3:12 AM — 25 minutes after the commit — the database started receiving unusual queries. The automated scanning infrastructure is not operated by individuals manually watching for leaks. It is fully automated: pattern match → clone → test credential validity → if valid, begin exploitation or sell.

GitHub now runs its own secret scanning and immediately invalidates some credential types (GitHub tokens, AWS IAM keys partnered with AWS) when detected in public repositories. This covers a subset of credential types. It does not cover database passwords, service-specific tokens for non-partnered services, or private repository commits that become public via fork.

The CircleCI Breach: Platform-Level Credential Exfiltration

The CircleCI breach (January 2023) is the definitive example of CI/CD platform-level secrets exposure. The attack chain:

1. CircleCI engineer's laptop compromised via malware (initial vector not fully disclosed)
2. Malware steals a 2FA-authenticated SSO session token
3. Session token valid, not expired
4. Attacker uses session token to authenticate to CircleCI internal systems
5. From internal access, attacker reaches production database
6. Production database contains encrypted customer secrets (environment variables)
7. Database also contains the encryption keys (in accessible internal system)
8. Attacker exfiltrates: encrypted secrets + encryption keys = plaintext secrets

What was stored in CircleCI environment variables by customers:
– AWS IAM access key ID and secret access key pairs
– GitHub personal access tokens and OAuth tokens
– DockerHub credentials
– SSH private keys (for deployment access)
– Heroku API keys
– Stripe, Twilio, SendGrid API keys
– Internal service account credentials

CircleCI could not determine which customer secrets were accessed and which were not — they notified all customers to rotate all credentials stored in their system.

The scale of the blast radius: Any customer who had stored long-lived credentials in CircleCI environment variables was potentially compromised. The credential was valid. The CircleCI platform’s encryption only protected against offline attacks — an attacker with internal database access and access to the key management system had everything needed to decrypt.

Red Phase: Enumerating Secrets Exposure in Your Pipeline

Scanning Repositories for Committed Secrets

# Install: pip install trufflehog3 or use the Docker image
docker run --rm \
  -v "$(pwd):/repo" \
  trufflesecurity/trufflehog:latest \
  git file:///repo \
  --json \
  --only-verified \
  2>/dev/null | \
  jq '{
    file: .SourceMetadata.Data.Git.file,
    commit: .SourceMetadata.Data.Git.commit,
    detector: .DetectorName,
    verified: .Verified,
    line: .SourceMetadata.Data.Git.line
  }'

# Gitleaks: alternative scanner with SARIF output for CI integration
gitleaks detect \
  --source . \
  --report-format sarif \
  --report-path gitleaks-report.sarif \
  --verbose

# Or: scan entire git history (catches secrets that were committed then deleted)
gitleaks detect \
  --source . \
  --log-opts="--all" \
  --report-format json \
  --report-path gitleaks-history.json

# Scan a specific GitHub organization's public repositories
# (test your own org before red team exercises)
trufflehog github \
  --org your-github-org \
  --token "${GITHUB_TOKEN}" \
  --json \
  --only-verified \
  2>/dev/null | \
  jq '{
    repo: .SourceMetadata.Data.Github.repository,
    file: .SourceMetadata.Data.Github.file,
    detector: .DetectorName,
    verified: .Verified
  }'

Enumerating Secrets in CI/CD Platform Environment Variables

# GitHub Actions: list secrets defined in a repository
# (shows names only — values are not returned by API, but names reveal what's stored)
curl -H "Authorization: Bearer ${GITHUB_TOKEN}" \
  -H "Accept: application/vnd.github+json" \
  "https://api.github.com/repos/your-org/your-repo/actions/secrets" | \
  jq '.secrets[] | {name: .name, updated: .updated_at}'

# GitHub Actions: list organization-level secrets
curl -H "Authorization: Bearer ${GITHUB_TOKEN}" \
  -H "Accept: application/vnd.github+json" \
  "https://api.github.com/orgs/your-org/actions/secrets" | \
  jq '.secrets[] | {name: .name, visibility: .visibility, updated: .updated_at}'

# Check for credentials in running pod environment variables (Kubernetes)
# This is what an attacker with kubectl exec access would do
kubectl get pods -A -o json | \
  jq -r '.items[] | 
    .metadata.namespace + "/" + .metadata.name + ": " + 
    ([.spec.containers[].env[]? | 
      select(.name | test("KEY|SECRET|TOKEN|PASSWORD|CREDENTIAL|API"; "i")) |
      .name
    ] | join(", "))' | \
  grep -v ": $"  # Only show pods with matching env var names

Testing Whether AWS Keys in CI/CD Are Over-Permissioned

# If you find an AWS access key in a scan — test its permissions
# (on your own test account's keys only)
aws sts get-caller-identity
# Returns: account, user/role ARN, caller ID

# What can this key do?
aws iam simulate-principal-policy \
  --policy-source-arn $(aws sts get-caller-identity --query Arn --output text) \
  --action-names "s3:*" "ec2:*" "iam:*" "sts:AssumeRole" \
  --query 'EvaluationResults[?EvalDecision==`allowed`].EvalActionName' \
  --output text

Blue Phase: Detection Across the Secret Lifecycle

GitHub Secret Scanning Alerts

# List secret scanning alerts in a repository via GitHub API
curl -H "Authorization: Bearer ${GITHUB_TOKEN}" \
  -H "Accept: application/vnd.github+json" \
  "https://api.github.com/repos/your-org/your-repo/secret-scanning/alerts?state=open" | \
  jq '.[] | {
    type: .secret_type,
    state: .state,
    created: .created_at,
    url: .html_url
  }'

CloudTrail: Detecting API Activity from CI/CD Credentials

When a CI/CD credential is used by an attacker, the CloudTrail events show unusual patterns:

# Find API calls from CI/CD credentials outside normal working hours
# or from unexpected IPs (attacker using the stolen key)
aws cloudtrail lookup-events \
  --lookup-attributes AttributeKey=Username,AttributeValue=ci-deploy-user \
  --start-time "$(date -d '7 days ago' --iso-8601=seconds)" \
  --query 'Events[].{Time:EventTime,Name:EventName,IP:CloudTrailEvent}' \
  --output json | \
  jq '.[] | {
    time: .Time,
    event: .Name,
    ip: (.IP | fromjson | .sourceIPAddress),
    user_agent: (.IP | fromjson | .userAgent)
  }' | \
  jq 'select(.ip | test("^(10\\.|172\\.(1[6-9]|2[0-9]|3[01])\\.|192\\.168\\.)") | not)'
  # Filter: events from non-RFC1918 IPs (outside your known CI/CD IP ranges)

SIEM Query: Credential Used in Multiple Regions Simultaneously

A credential being used from multiple regions simultaneously is a strong indicator of compromise:

-- Athena query against CloudTrail logs
-- Detect: same access key used from multiple regions in same hour
SELECT
  userIdentity.accessKeyId,
  userIdentity.userName,
  COUNT(DISTINCT awsRegion) as region_count,
  ARRAY_AGG(DISTINCT awsRegion) as regions,
  COUNT(DISTINCT sourceIPAddress) as ip_count,
  ARRAY_AGG(DISTINCT sourceIPAddress) as source_ips,
  DATE_TRUNC('hour', from_iso8601_timestamp(eventTime)) as hour
FROM cloudtrail_logs
WHERE
  userIdentity.type = 'IAMUser'
  AND from_iso8601_timestamp(eventTime) > current_timestamp - interval '7' day
GROUP BY
  userIdentity.accessKeyId,
  userIdentity.userName,
  DATE_TRUNC('hour', from_iso8601_timestamp(eventTime))
HAVING COUNT(DISTINCT awsRegion) > 2
ORDER BY region_count DESC;

GuardDuty: Credential Exfiltration Indicators

# GuardDuty findings relevant to CI/CD credential compromise
DETECTOR_ID=$(aws guardduty list-detectors --query 'DetectorIds[0]' --output text)

aws guardduty list-findings \
  --detector-id "${DETECTOR_ID}" \
  --finding-criteria '{
    "Criterion": {
      "type": {
        "Equals": [
          "UnauthorizedAccess:IAMUser/TorIPCaller",
          "UnauthorizedAccess:IAMUser/MaliciousIPCaller",
          "Discovery:IAMUser/AnomalousBehavior",
          "Exfiltration:IAMUser/AnomalousBehavior",
          "CredentialAccess:IAMUser/AnomalousBehavior"
        ]
      }
    }
  }' \
  --query 'FindingIds' --output text | \
  xargs -n 10 aws guardduty get-findings \
    --detector-id "${DETECTOR_ID}" \
    --finding-ids | \
  jq '.Findings[] | {type: .Type, user: .Resource.AccessKeyDetails.UserName, severity: .Severity}'

Purple Phase: The Structural Fix

Fix 1: OIDC Workload Identity — Eliminate Stored Credentials

This is the structural solution. Instead of storing an AWS IAM access key in your CI/CD platform, the CI/CD job authenticates to AWS using an OIDC token issued by the CI/CD provider. AWS validates the token against a pre-configured trust policy and issues temporary credentials valid for the duration of the job.

The OIDC workload identity approach eliminates static cloud access keys entirely — there is no secret to commit, no secret to exfiltrate from the CI/CD platform, and no long-lived credential to rotate on breach.

GitHub Actions with AWS OIDC — complete setup:

# .github/workflows/deploy.yml
name: Deploy to AWS

on:
  push:
    branches: [main]

permissions:
  id-token: write   # Required for OIDC token request
  contents: read

jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Configure AWS credentials via OIDC
        uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: arn:aws:iam::123456789012:role/github-actions-deploy-role
          role-session-name: github-actions-${{ github.run_id }}
          aws-region: us-east-1
          # No AWS_ACCESS_KEY_ID or AWS_SECRET_ACCESS_KEY needed

      - name: Deploy
        run: aws s3 sync ./dist s3://your-bucket/

AWS IAM trust policy for GitHub Actions OIDC:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Federated": "arn:aws:iam::123456789012:oidc-provider/token.actions.githubusercontent.com"
      },
      "Action": "sts:AssumeRoleWithWebIdentity",
      "Condition": {
        "StringEquals": {
          "token.actions.githubusercontent.com:aud": "sts.amazonaws.com"
        },
        "StringLike": {
          "token.actions.githubusercontent.com:sub": "repo:your-org/your-repo:ref:refs/heads/main"
        }
      }
    }
  ]
}

# Create the OIDC provider in AWS (one-time setup)
aws iam create-open-id-connect-provider \
  --url https://token.actions.githubusercontent.com \
  --client-id-list sts.amazonaws.com \
  --thumbprint-list "6938fd4d98bab03faadb97b34396831e3780aea1"

# Create the IAM role with the trust policy above
aws iam create-role \
  --role-name github-actions-deploy-role \
  --assume-role-policy-document file://github-actions-trust-policy.json

# Attach a least-privilege policy to the role
aws iam attach-role-policy \
  --role-name github-actions-deploy-role \
  --policy-arn arn:aws:iam::123456789012:policy/deploy-policy

Fix 2: Pre-Commit Hooks — Catch Accidents Before They Reach VCS

Pre-commit hooks don’t stop a determined attacker. They catch accidents — the developer who forgets to move a .env file to .gitignore before staging all files.

# Install pre-commit framework
pip install pre-commit

# .pre-commit-config.yaml in your repository root
cat > .pre-commit-config.yaml << 'EOF'
repos:
  - repo: https://github.com/gitleaks/gitleaks
    rev: v8.18.4
    hooks:
      - id: gitleaks
        name: Detect hardcoded secrets
        entry: gitleaks protect --staged --redact --verbose
        language: golang
        pass_filenames: false

  - repo: https://github.com/pre-commit/pre-commit-hooks
    rev: v4.5.0
    hooks:
      - id: detect-private-key
      - id: check-added-large-files
        args: ['--maxkb=1000']
EOF

# Install the hooks in the local repository
pre-commit install

# Test against staged files
pre-commit run --all-files

Fix 3: CI-Layer Secret Scanning — Block Before Merge

# GitHub Actions: secret scanning as a required status check
# .github/workflows/secret-scan.yml
name: Secret Scan

on:
  pull_request:
    types: [opened, synchronize]

jobs:
  secret-scan:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0  # Full history for git log scanning

      - name: Run TruffleHog
        uses: trufflesecurity/trufflehog@main
        with:
          path: ./
          base: ${{ github.event.repository.default_branch }}
          head: HEAD
          extra_args: --only-verified --json

# GitLab CI: secret detection built-in template
include:
  - template: Security/Secret-Detection.gitlab-ci.yml

secret_detection:
  stage: test
  variables:
    SECRET_DETECTION_HISTORIC_SCAN: "true"  # Scan full history

Fix 4: Audit and Rotate Existing CI/CD Platform Secrets

After implementing OIDC, the migration path for existing stored credentials:

#!/bin/bash
# Purple Team EP06 — CI/CD Secrets Migration Audit
# Identifies AWS IAM keys stored in CI/CD that should be replaced with OIDC

echo "=== AWS IAM Keys Potentially Stored in CI/CD ==="
echo "--- Keys not used from expected CI/CD IPs in last 30 days ---"

# Get all IAM access keys
aws iam list-users --query 'Users[].UserName' --output text | tr '\t' '\n' | \
  while read user; do
    keys=$(aws iam list-access-keys --user-name "$user" \
      --query 'AccessKeyMetadata[?Status==`Active`].{Key:AccessKeyId,Created:CreateDate}' \
      --output json)

    if [ "$(echo "$keys" | jq length)" -gt 0 ]; then
      echo ""
      echo "User: $user"
      echo "$keys" | jq -r '.[] | "  Key: " + .Key + " | Created: " + .Created'

      # Check last used
      echo "$keys" | jq -r '.[].Key' | while read key_id; do
        last_used=$(aws iam get-access-key-last-used --access-key-id "$key_id" \
          --query 'AccessKeyLastUsed.{Date:LastUsedDate,Service:ServiceName,Region:Region}' \
          --output json)
        echo "  Last used: $(echo "$last_used" | jq -r '.Date // "Never"') | Service: $(echo "$last_used" | jq -r '.Service // "N/A"')"
      done
    fi
  done

echo ""
echo "=== MIGRATION CHECKLIST ==="
echo "  1. For each CI/CD IAM key above:"
echo "     a. Identify which CI/CD platform uses it"
echo "     b. Set up OIDC trust policy for that platform"
echo "     c. Update pipeline to use OIDC (no stored key)"
echo "     d. Disable and then delete the IAM key"
echo "     e. Verify pipelines still work"

Run This in Your Own Environment: Secrets Exposure Audit

#!/bin/bash
# Purple Team EP06 — CI/CD Secrets Exposure Audit
# Run from your workstation with git and trufflehog installed

echo "=== 1. Scan Local Repository for Committed Secrets ==="
if command -v trufflehog > /dev/null 2>&1; then
  trufflehog git file://$(pwd) --only-verified --json 2>/dev/null | \
    jq '{file: .SourceMetadata.Data.Git.file, detector: .DetectorName}' || \
    echo "  No verified secrets found in git history"
else
  echo "  Install trufflehog: pip install trufflehog3"
fi

echo ""
echo "=== 2. Check for .env Files in Git History ==="
git log --all --full-history -- "*.env" "**/.env" ".env.*" 2>/dev/null | \
  grep "^commit" | head -5 | \
  while read _ commit; do
    echo "  .env file committed: $commit"
    git show "$commit" --stat | head -3
  done

echo ""
echo "=== 3. Check Running Pods for Credential Env Vars (Kubernetes) ==="
if command -v kubectl > /dev/null 2>&1; then
  kubectl get pods -A -o json 2>/dev/null | \
    jq -r '.items[] | 
      .metadata.namespace + "/" + .metadata.name + ": " + 
      ([.spec.containers[].env[]? | 
        select(.name | test("KEY|SECRET|TOKEN|PASSWORD|CREDENTIAL"; "i")) |
        .name
      ] | join(", "))' | \
    grep -v ": $" | head -20
else
  echo "  kubectl not found"
fi

echo ""
echo "=== 4. GitHub Actions Secrets Inventory ==="
if [ -n "${GITHUB_TOKEN}" ]; then
  REPO="your-org/your-repo"  # Update this
  curl -s -H "Authorization: Bearer ${GITHUB_TOKEN}" \
    -H "Accept: application/vnd.github+json" \
    "https://api.github.com/repos/${REPO}/actions/secrets" | \
    jq '.secrets[] | {name: .name, updated: .updated_at}'
else
  echo "  Set GITHUB_TOKEN to enumerate repository secrets"
fi

Common Mistakes When Addressing CI/CD Secrets Exposure

Treating secret scanning as the primary control. TruffleHog and Gitleaks catch what gets committed. They do not prevent the CircleCI attack class — an attacker who compromises the CI/CD platform itself bypasses all scanning controls. Scanning is detection; OIDC workload identity is prevention.

Rotating compromised keys without checking CloudTrail for use. When a secret is exposed, the first question is not “rotate it” — it is “was it used?” Check CloudTrail for any API activity from the key between the suspected exposure time and the rotation. If the key was used, you have an active incident, not just a credential rotation task.

Using OIDC trust policies that are too broad. The GitHub Actions OIDC trust policy in the fix section uses a StringLike condition on the sub claim to scope to a specific repository and branch. If you use StringLike: "*" instead, any GitHub Actions job in any repository can assume your role. Always scope OIDC trust policies to the specific repository, branch, and environment that needs the access.

Not scanning git history — only the working tree. Secrets that were committed and then deleted are still in git history. git rm removes the file from the working tree but not from the object store. TruffleHog and Gitleaks scan history by default when given the --all flag. Scanning only the current working tree misses all historical exposures.

Forgetting third-party GitHub Actions. The supply chain attack surface includes the Actions you reference in your workflows. An Action pinned to a mutable tag (@main, @v1) can be changed by the maintainer. Pin to a specific commit SHA and verify the Action’s provenance.

# Vulnerable: mutable tag
- uses: aws-actions/configure-aws-credentials@v4

# Secure: pinned SHA
- uses: aws-actions/configure-aws-credentials@e3dd6a429d7300a6a4c196c26e831c1e4c763fe4

Quick Reference

Key Takeaways

• CI/CD secrets exposure is structural: long-lived credentials in a CI/CD platform are only as secure as that platform — the CircleCI breach proved that encryption alone is insufficient if the attacker can access the keys
• Automated secret scanners find publicly committed credentials within 60–90 seconds — rotation must happen faster than that or assume compromise
• Pre-commit hooks and CI secret scanning catch accidents; they do not prevent determined attackers who compromise the platform itself
• OIDC workload identity is the structural fix: no stored credential means no credential to exfiltrate
• When rotating a compromised key, check CloudTrail for usage between exposure and rotation before closing the incident
• OIDC trust policies must be scoped to specific repositories and branches — a wildcard trust policy recreates the exposure in a different form
• Pin third-party GitHub Actions to commit SHAs, not mutable tags — mutable tags are a supply chain attack surface

What’s Next

EP07 covers SSRF to cloud metadata: how an SSRF vulnerability in any application layer becomes a straight line to IAM credentials when IMDSv2 is not enforced. The Capital One breach anatomy — WAF SSRF → EC2 metadata → IAM role credentials → 100 million S3 records — in full technical detail, with the simulation commands and the one-line enforcement fix. If you’ve addressed identity and secrets, the network attack paths are where EP07 through EP10 focus.

Get EP07 in your inbox when it publishes → subscribe at linuxcent.com

The post CI/CD Secrets Exposure: How Supply Chain Attacks Target Your Pipeline appeared first on Linuxcent.

OS Hardening as Code, Episode 5
Cloud AMI Security Risks · Linux Hardening as Code · Multi-Cloud OS Hardening · Automated OpenSCAP Compliance · CI/CD Compliance Gate**

Note: the tool in this series was released as Stratum and renamed to BakeX at
v0.6.0 — same project, same license, same team. Commands below use the current bakex
CLI. If you arrived here looking for stratum or pip install stratumoss, you’re in the
right place: github.com/invicton/bakex.

TL;DR

• A CI/CD compliance gate turns an OS hardening grade from a report into a build constraint — unhardened images fail the pipeline before they can be deployed
• POST /api/pipeline/scan scores an image against a pass_threshold and a severity_threshold, and returns a passed boolean
• The endpoint returns HTTP 200 even when the gate fails. curl -sf will not catch it — you must parse .passed. This is the single most important detail on this page
• The gate is two-dimensional: a score floor and a severity ceiling, so one critical finding blocks a release that scores 94
• GitHub Actions, GitLab CI, Jenkins, and Tekton integrations are one curl plus one jq
• The structural guarantee: an image that doesn’t pass the gate doesn’t reach the deploy job

The Problem: A Grade No One Checks Is Decoration

Pipeline without compliance gate:
  Build → Test → Security scan (results to dashboard) → Deploy

What actually happens:
  Build → Test → Security scan → "C grade, but we need to ship" → Deploy anyway
                                           │
                                           └─ Dashboard shows C grade
                                              Nobody is paged
                                              Deployment succeeds

A CI/CD compliance gate means the pipeline can’t continue if the grade is below threshold.

EP04 showed that automated OpenSCAP compliance gives every image a verified, reproducible grade before deployment. What it assumed is that someone checks the grade before deploying. They don’t — not under deadline pressure, not when the image has been “working fine for months,” not at 2am.

The same problem that made hardening runbooks skippable applies to compliance grades: if checking the grade is a discretionary step, it will be skipped.

A new microservice was deployed from an unhardened base image. The team had built it quickly during a sprint, used a community AMI as the base, and planned to harden it “in the next sprint.”

Three weeks later, a penetration test found it. SSH password authentication enabled. Three unnecessary services running — one of them with a known CVE. The finding: the instance had full inbound access from the VPC and was reachable from a compromised adjacent instance.

The deployment had gone through the normal CI/CD pipeline. Unit tests passed. Integration tests passed. A vulnerability scan ran. The scan produced a report that went to a dashboard. Nobody had a gate set up to fail the build if the image was unhardened.

The hardening work from the “next sprint” plan would have taken four hours. The pentest remediation took a week, plus the time to investigate what had been exposed during the three weeks the instance was running.

The CI/CD pipeline had every check except the one that would have caught the base image problem before the first deployment.

The Pipeline API

The Pipeline API is a single HTTP endpoint that takes an image ID, scans it, and returns a verdict:

curl -s -X POST https://bakex.yourdomain.com/api/pipeline/scan \
  -H "X-API-Key: ${BAKEX_TOKEN}" \
  -H "Content-Type: application/json" \
  -d '{
    "image_id": "ami-0a7f3c9e82d1b4c05",
    "provider": "aws",
    "region": "us-east-1",
    "pass_threshold": 75.0,
    "severity_threshold": "high",
    "wait": true
  }'

Authentication takes either X-API-Key or Authorization: Bearer; keys are created at
/settings/api-keys. With wait: true the request blocks until the scan completes — which is what
you want in CI, where a job that returns before the answer exists is worse than a slow one. There’s
a timeout_seconds (default 900) for when it doesn’t.

The response is the same shape whether you passed or failed:

{
  "job_id": "7f3c9e82-4d1b-4c05-a7f3-c9e82d1b4c05",
  "status": "complete",
  "passed": false,
  "grade": "C",
  "score_pct": 72.0,
  "severity_counts": { "critical": 0, "high": 2, "medium": 5, "low": 11 },
  "threshold_violations": ["high"],
  "pass_threshold": 75.0,
  "severity_threshold": "high",
  "image_id": "ami-0c9d5e3f81a2b6e07",
  "sarif_url": ".../api/auditor/scan-image/7f3c9e82.../report?fmt=sarif",
  "html_report_url": ".../api/auditor/scan-image/7f3c9e82.../report"
}

The detail that will silently break your gate

A failed gate still returns HTTP 200. There is no 4xx on failure — the verdict is in the
passed field, not the status code.

That means the pattern everyone reaches for first is wrong:

# WRONG — this never fails. -f only reacts to HTTP >= 400,
# and a failed gate returns 200.
curl -sf -X POST .../api/pipeline/scan -d '...' || exit 1

You have to read the body:

# RIGHT
RESULT=$(curl -s -X POST "${BAKEX_URL}/api/pipeline/scan" \
  -H "X-API-Key: ${BAKEX_TOKEN}" \
  -H "Content-Type: application/json" \
  -d "{\"image_id\": \"${AMI_ID}\", \"pass_threshold\": 75.0, \"severity_threshold\": \"high\"}")

echo "$RESULT" | jq -r '"grade=\(.grade) score=\(.score_pct) passed=\(.passed)"'

if [ "$(echo "$RESULT" | jq -r '.passed')" != "true" ]; then
  echo "Compliance gate failed — violations: $(echo "$RESULT" | jq -c '.threshold_violations')"
  echo "Report: $(echo "$RESULT" | jq -r '.html_report_url')"
  exit 1
fi

A gate that reports failure and exits 0 is worse than no gate, because it produces a green
pipeline and the belief that something was checked.

Two thresholds, not one

passed is the AND of two independent conditions:

passed = (score_pct >= pass_threshold) AND (no findings at or above severity_threshold)

severity_threshold: "high" means any critical or high finding fails the build regardless of
score. An image can score 94 — a comfortable A — and still fail on a single critical finding. That
is the right default: scores average away the thing that gets you breached.

GitHub Actions Integration

# .github/workflows/deploy.yml

jobs:
  build-image:
    runs-on: ubuntu-latest
    outputs:
      ami_id: ${{ steps.build.outputs.ami_id }}
    steps:
      - name: Build hardened AMI
        id: build
        run: |
          AMI_ID=$(bakex build blueprints/ubuntu/22.04/cis-l1-aws.yaml --json \
            | jq -r '.artifact_id')
          echo "ami_id=${AMI_ID}" >> $GITHUB_OUTPUT

  compliance-gate:
    runs-on: ubuntu-latest
    needs: build-image
    steps:
      - name: BakeX compliance gate
        run: |
          RESULT=$(curl -s -X POST ${{ vars.BAKEX_URL }}/api/pipeline/scan \
            -H "X-API-Key: ${{ secrets.BAKEX_TOKEN }}" \
            -H "Content-Type: application/json" \
            -d "{\"image_id\": \"${{ needs.build-image.outputs.ami_id }}\",
                 \"pass_threshold\": 75.0, \"severity_threshold\": \"high\"}")

          echo "$RESULT" | jq -r '"grade=\(.grade) score=\(.score_pct)"'

          # Must check .passed — the endpoint returns 200 on failure
          if [ "$(echo "$RESULT" | jq -r '.passed')" != "true" ]; then
            echo "::error::Compliance gate failed: $(echo "$RESULT" | jq -c '.threshold_violations')"
            exit 1
          fi

      - name: Upload SARIF to code scanning
        if: always()
        run: |
          curl -s -o bakex.sarif "$(echo "$RESULT" | jq -r '.sarif_url')"
      - uses: github/codeql-action/upload-sarif@v3
        if: always()
        with:
          sarif_file: bakex.sarif

  deploy:
    runs-on: ubuntu-latest
    needs: [build-image, compliance-gate]
    steps:
      - name: Deploy to staging
        run: |
          aws autoscaling update-auto-scaling-group \
            --auto-scaling-group-name my-asg \
            --launch-template "ImageId=${{ needs.build-image.outputs.ami_id }}"

The deploy job only runs if compliance-gate passes. The AMI doesn’t reach the autoscaling group if it doesn’t meet the grade threshold.

GitLab CI Integration

# .gitlab-ci.yml

stages:
  - build
  - compliance
  - deploy

build-image:
  stage: build
  script:
    - |
      AMI_ID=$(bakex build blueprints/ubuntu/22.04/cis-l1-aws.yaml --json \
        | jq -r '.artifact_id')
      echo "AMI_ID=${AMI_ID}" >> build.env
  artifacts:
    reports:
      dotenv: build.env

compliance-gate:
  stage: compliance
  needs: [build-image]
  script:
    - |
      RESULT=$(curl -s -X POST ${BAKEX_URL}/api/pipeline/scan \
        -H "X-API-Key: ${BAKEX_TOKEN}" \
        -H "Content-Type: application/json" \
        -d "{\"image_id\": \"${AMI_ID}\", \"pass_threshold\": 75.0,
             \"severity_threshold\": \"high\"}")
      echo "$RESULT" | jq -r '"grade=\(.grade) score=\(.score_pct) passed=\(.passed)"'
      test "$(echo "$RESULT" | jq -r '.passed')" = "true"

deploy:
  stage: deploy
  needs: [build-image, compliance-gate]
  script:
    - ./deploy.sh ${AMI_ID}

What the Failed Gate Tells You

The value of the CI/CD compliance gate is not just that it blocks bad images — it’s that the failure output tells engineers what to fix.

The response carries three things an engineer can act on immediately:

$ echo "$RESULT" | jq '{grade, score_pct, threshold_violations, severity_counts}'
{
  "grade": "C",
  "score_pct": 72.0,
  "threshold_violations": ["high"],
  "severity_counts": { "critical": 0, "high": 2, "medium": 5, "low": 11 }
}

threshold_violations names the severities that broke the gate — here, two high findings, not the
score. That distinction matters: an engineer who reads “grade C” starts a broad hardening project,
while one who reads “two high findings” goes and fixes two things.

For the rule-level detail, follow sarif_url. Pushing that SARIF into GitHub code scanning (as in
the workflow above) puts each finding on the pull request diff, which is where someone will actually
read it — a link to a dashboard in a CI log is a link nobody clicks.

Thresholds by Environment

Not all environments need the same bar, and both dimensions are per-request — so the environment
distinction lives in your pipeline, not in BakeX config:

# Production — high score floor, nothing high or above
PASS=90.0 ; SEV=high

# Staging — lower floor, still no criticals
PASS=75.0 ; SEV=critical

# Development — score only, severity effectively off
PASS=60.0 ; SEV=low

curl -s -X POST "${BAKEX_URL}/api/pipeline/scan" \
  -H "X-API-Key: ${BAKEX_TOKEN}" -H "Content-Type: application/json" \
  -d "{\"image_id\": \"${AMI_ID}\", \"pass_threshold\": ${PASS}, \"severity_threshold\": \"${SEV}\"}"

Note that severity_threshold gets stricter as it goes down the list: low fails on any finding
at all, critical fails only on criticals. It reads backwards the first time. Development wanting a
permissive gate wants critical, not low.

Production Gotchas

The 200-on-failure behaviour is the whole ballgame. Repeating it because it is the one thing that
turns this page from useful to harmful if missed: check .passed. Never rely on curl -f, and never
rely on the HTTP status.

Scans take minutes, and wait: true blocks. The endpoint provisions an instance from the image
and scans it. With wait: true your CI job blocks for the duration; timeout_seconds defaults to
900. Set your CI step timeout above that, or use wait: false and poll GET /api/pipeline/scan/{job_id}.

Token rotation. The API key should rotate on the same schedule as other service credentials, and
environments should use different keys — a leaked staging key must not be able to satisfy a
production gate.

The gate needs a reachable BakeX server. This is an HTTP API, not a self-contained action: the
runner must reach the BakeX instance, and that instance needs cloud credentials for the provider
whose image it is scanning.

Key Takeaways

• A CI/CD compliance gate turns a compliance grade from a dashboard metric into a pipeline constraint — the image doesn’t deploy if it doesn’t pass
• POST /api/pipeline/scan is a single HTTP call that any CI/CD system can make — no agent, no plugin, no SDK required
• The endpoint returns 200 even when the gate fails. Parse .passed; curl -sf || exit 1 produces a green pipeline and a false sense of security
• The verdict is two-dimensional — a score floor AND a severity ceiling — so a single critical finding blocks an image that scores 94
• threshold_violations tells an engineer why it failed, which is the difference between “fix two high findings” and “start a hardening project”
• Push the sarif_url into GitHub code scanning so findings land on the pull request, not in a CI log

What’s Next

The CI/CD compliance gate closes the final gap: even if an unhardened image gets built, it can’t deploy. EP05 is the bookmark episode — this is the point where OS hardening becomes structurally enforced rather than procedurally expected.

EP06 is the series closer. For five episodes, you’ve been using BakeX as a user. What does it look like to run it yourself — extend it with a custom provider, deploy it in your own infrastructure, or contribute a blueprint back?

BakeX is Apache 2.0. EP06 is the architecture reveal, the deployment guide, and the extension points for everything the series taught.

Next: BakeX — open-source OS hardening platform for multi-cloud infrastructure

Get EP06 in your inbox when it publishes → linuxcent.com/subscribe

The post The Pipeline Gate — Hardened Images as a CI/CD Build Constraint appeared first on Linuxcent.