OS Hardening as Code, Episode 5
Cloud AMI Security Risks · Linux Hardening as Code · Multi-Cloud OS Hardening · Automated OpenSCAP Compliance · CI/CD Compliance Gate**

Note: the tool in this series was released as Stratum and renamed to BakeX at
v0.6.0 — same project, same license, same team. Commands below use the current bakex
CLI. If you arrived here looking for stratum or pip install stratumoss, you’re in the
right place: github.com/invicton/bakex.

TL;DR

• A CI/CD compliance gate turns an OS hardening grade from a report into a build constraint — unhardened images fail the pipeline before they can be deployed
• POST /api/pipeline/scan scores an image against a pass_threshold and a severity_threshold, and returns a passed boolean
• The endpoint returns HTTP 200 even when the gate fails. curl -sf will not catch it — you must parse .passed. This is the single most important detail on this page
• The gate is two-dimensional: a score floor and a severity ceiling, so one critical finding blocks a release that scores 94
• GitHub Actions, GitLab CI, Jenkins, and Tekton integrations are one curl plus one jq
• The structural guarantee: an image that doesn’t pass the gate doesn’t reach the deploy job

The Problem: A Grade No One Checks Is Decoration

Pipeline without compliance gate:
  Build → Test → Security scan (results to dashboard) → Deploy

What actually happens:
  Build → Test → Security scan → "C grade, but we need to ship" → Deploy anyway
                                           │
                                           └─ Dashboard shows C grade
                                              Nobody is paged
                                              Deployment succeeds

A CI/CD compliance gate means the pipeline can’t continue if the grade is below threshold.

EP04 showed that automated OpenSCAP compliance gives every image a verified, reproducible grade before deployment. What it assumed is that someone checks the grade before deploying. They don’t — not under deadline pressure, not when the image has been “working fine for months,” not at 2am.

The same problem that made hardening runbooks skippable applies to compliance grades: if checking the grade is a discretionary step, it will be skipped.

A new microservice was deployed from an unhardened base image. The team had built it quickly during a sprint, used a community AMI as the base, and planned to harden it “in the next sprint.”

Three weeks later, a penetration test found it. SSH password authentication enabled. Three unnecessary services running — one of them with a known CVE. The finding: the instance had full inbound access from the VPC and was reachable from a compromised adjacent instance.

The deployment had gone through the normal CI/CD pipeline. Unit tests passed. Integration tests passed. A vulnerability scan ran. The scan produced a report that went to a dashboard. Nobody had a gate set up to fail the build if the image was unhardened.

The hardening work from the “next sprint” plan would have taken four hours. The pentest remediation took a week, plus the time to investigate what had been exposed during the three weeks the instance was running.

The CI/CD pipeline had every check except the one that would have caught the base image problem before the first deployment.

The Pipeline API

The Pipeline API is a single HTTP endpoint that takes an image ID, scans it, and returns a verdict:

curl -s -X POST https://bakex.yourdomain.com/api/pipeline/scan \
  -H "X-API-Key: ${BAKEX_TOKEN}" \
  -H "Content-Type: application/json" \
  -d '{
    "image_id": "ami-0a7f3c9e82d1b4c05",
    "provider": "aws",
    "region": "us-east-1",
    "pass_threshold": 75.0,
    "severity_threshold": "high",
    "wait": true
  }'

Authentication takes either X-API-Key or Authorization: Bearer; keys are created at
/settings/api-keys. With wait: true the request blocks until the scan completes — which is what
you want in CI, where a job that returns before the answer exists is worse than a slow one. There’s
a timeout_seconds (default 900) for when it doesn’t.

The response is the same shape whether you passed or failed:

{
  "job_id": "7f3c9e82-4d1b-4c05-a7f3-c9e82d1b4c05",
  "status": "complete",
  "passed": false,
  "grade": "C",
  "score_pct": 72.0,
  "severity_counts": { "critical": 0, "high": 2, "medium": 5, "low": 11 },
  "threshold_violations": ["high"],
  "pass_threshold": 75.0,
  "severity_threshold": "high",
  "image_id": "ami-0c9d5e3f81a2b6e07",
  "sarif_url": ".../api/auditor/scan-image/7f3c9e82.../report?fmt=sarif",
  "html_report_url": ".../api/auditor/scan-image/7f3c9e82.../report"
}

The detail that will silently break your gate

A failed gate still returns HTTP 200. There is no 4xx on failure — the verdict is in the
passed field, not the status code.

That means the pattern everyone reaches for first is wrong:

# WRONG — this never fails. -f only reacts to HTTP >= 400,
# and a failed gate returns 200.
curl -sf -X POST .../api/pipeline/scan -d '...' || exit 1

You have to read the body:

# RIGHT
RESULT=$(curl -s -X POST "${BAKEX_URL}/api/pipeline/scan" \
  -H "X-API-Key: ${BAKEX_TOKEN}" \
  -H "Content-Type: application/json" \
  -d "{\"image_id\": \"${AMI_ID}\", \"pass_threshold\": 75.0, \"severity_threshold\": \"high\"}")

echo "$RESULT" | jq -r '"grade=\(.grade) score=\(.score_pct) passed=\(.passed)"'

if [ "$(echo "$RESULT" | jq -r '.passed')" != "true" ]; then
  echo "Compliance gate failed — violations: $(echo "$RESULT" | jq -c '.threshold_violations')"
  echo "Report: $(echo "$RESULT" | jq -r '.html_report_url')"
  exit 1
fi

A gate that reports failure and exits 0 is worse than no gate, because it produces a green
pipeline and the belief that something was checked.

Two thresholds, not one

passed is the AND of two independent conditions:

passed = (score_pct >= pass_threshold) AND (no findings at or above severity_threshold)

severity_threshold: "high" means any critical or high finding fails the build regardless of
score. An image can score 94 — a comfortable A — and still fail on a single critical finding. That
is the right default: scores average away the thing that gets you breached.

GitHub Actions Integration

# .github/workflows/deploy.yml

jobs:
  build-image:
    runs-on: ubuntu-latest
    outputs:
      ami_id: ${{ steps.build.outputs.ami_id }}
    steps:
      - name: Build hardened AMI
        id: build
        run: |
          AMI_ID=$(bakex build blueprints/ubuntu/22.04/cis-l1-aws.yaml --json \
            | jq -r '.artifact_id')
          echo "ami_id=${AMI_ID}" >> $GITHUB_OUTPUT

  compliance-gate:
    runs-on: ubuntu-latest
    needs: build-image
    steps:
      - name: BakeX compliance gate
        run: |
          RESULT=$(curl -s -X POST ${{ vars.BAKEX_URL }}/api/pipeline/scan \
            -H "X-API-Key: ${{ secrets.BAKEX_TOKEN }}" \
            -H "Content-Type: application/json" \
            -d "{\"image_id\": \"${{ needs.build-image.outputs.ami_id }}\",
                 \"pass_threshold\": 75.0, \"severity_threshold\": \"high\"}")

          echo "$RESULT" | jq -r '"grade=\(.grade) score=\(.score_pct)"'

          # Must check .passed — the endpoint returns 200 on failure
          if [ "$(echo "$RESULT" | jq -r '.passed')" != "true" ]; then
            echo "::error::Compliance gate failed: $(echo "$RESULT" | jq -c '.threshold_violations')"
            exit 1
          fi

      - name: Upload SARIF to code scanning
        if: always()
        run: |
          curl -s -o bakex.sarif "$(echo "$RESULT" | jq -r '.sarif_url')"
      - uses: github/codeql-action/upload-sarif@v3
        if: always()
        with:
          sarif_file: bakex.sarif

  deploy:
    runs-on: ubuntu-latest
    needs: [build-image, compliance-gate]
    steps:
      - name: Deploy to staging
        run: |
          aws autoscaling update-auto-scaling-group \
            --auto-scaling-group-name my-asg \
            --launch-template "ImageId=${{ needs.build-image.outputs.ami_id }}"

The deploy job only runs if compliance-gate passes. The AMI doesn’t reach the autoscaling group if it doesn’t meet the grade threshold.

GitLab CI Integration

# .gitlab-ci.yml

stages:
  - build
  - compliance
  - deploy

build-image:
  stage: build
  script:
    - |
      AMI_ID=$(bakex build blueprints/ubuntu/22.04/cis-l1-aws.yaml --json \
        | jq -r '.artifact_id')
      echo "AMI_ID=${AMI_ID}" >> build.env
  artifacts:
    reports:
      dotenv: build.env

compliance-gate:
  stage: compliance
  needs: [build-image]
  script:
    - |
      RESULT=$(curl -s -X POST ${BAKEX_URL}/api/pipeline/scan \
        -H "X-API-Key: ${BAKEX_TOKEN}" \
        -H "Content-Type: application/json" \
        -d "{\"image_id\": \"${AMI_ID}\", \"pass_threshold\": 75.0,
             \"severity_threshold\": \"high\"}")
      echo "$RESULT" | jq -r '"grade=\(.grade) score=\(.score_pct) passed=\(.passed)"'
      test "$(echo "$RESULT" | jq -r '.passed')" = "true"

deploy:
  stage: deploy
  needs: [build-image, compliance-gate]
  script:
    - ./deploy.sh ${AMI_ID}

What the Failed Gate Tells You

The value of the CI/CD compliance gate is not just that it blocks bad images — it’s that the failure output tells engineers what to fix.

The response carries three things an engineer can act on immediately:

$ echo "$RESULT" | jq '{grade, score_pct, threshold_violations, severity_counts}'
{
  "grade": "C",
  "score_pct": 72.0,
  "threshold_violations": ["high"],
  "severity_counts": { "critical": 0, "high": 2, "medium": 5, "low": 11 }
}

threshold_violations names the severities that broke the gate — here, two high findings, not the
score. That distinction matters: an engineer who reads “grade C” starts a broad hardening project,
while one who reads “two high findings” goes and fixes two things.

For the rule-level detail, follow sarif_url. Pushing that SARIF into GitHub code scanning (as in
the workflow above) puts each finding on the pull request diff, which is where someone will actually
read it — a link to a dashboard in a CI log is a link nobody clicks.

Thresholds by Environment

Not all environments need the same bar, and both dimensions are per-request — so the environment
distinction lives in your pipeline, not in BakeX config:

# Production — high score floor, nothing high or above
PASS=90.0 ; SEV=high

# Staging — lower floor, still no criticals
PASS=75.0 ; SEV=critical

# Development — score only, severity effectively off
PASS=60.0 ; SEV=low

curl -s -X POST "${BAKEX_URL}/api/pipeline/scan" \
  -H "X-API-Key: ${BAKEX_TOKEN}" -H "Content-Type: application/json" \
  -d "{\"image_id\": \"${AMI_ID}\", \"pass_threshold\": ${PASS}, \"severity_threshold\": \"${SEV}\"}"

Note that severity_threshold gets stricter as it goes down the list: low fails on any finding
at all, critical fails only on criticals. It reads backwards the first time. Development wanting a
permissive gate wants critical, not low.

Production Gotchas

The 200-on-failure behaviour is the whole ballgame. Repeating it because it is the one thing that
turns this page from useful to harmful if missed: check .passed. Never rely on curl -f, and never
rely on the HTTP status.

Scans take minutes, and wait: true blocks. The endpoint provisions an instance from the image
and scans it. With wait: true your CI job blocks for the duration; timeout_seconds defaults to
900. Set your CI step timeout above that, or use wait: false and poll GET /api/pipeline/scan/{job_id}.

Token rotation. The API key should rotate on the same schedule as other service credentials, and
environments should use different keys — a leaked staging key must not be able to satisfy a
production gate.

The gate needs a reachable BakeX server. This is an HTTP API, not a self-contained action: the
runner must reach the BakeX instance, and that instance needs cloud credentials for the provider
whose image it is scanning.

Key Takeaways

• A CI/CD compliance gate turns a compliance grade from a dashboard metric into a pipeline constraint — the image doesn’t deploy if it doesn’t pass
• POST /api/pipeline/scan is a single HTTP call that any CI/CD system can make — no agent, no plugin, no SDK required
• The endpoint returns 200 even when the gate fails. Parse .passed; curl -sf || exit 1 produces a green pipeline and a false sense of security
• The verdict is two-dimensional — a score floor AND a severity ceiling — so a single critical finding blocks an image that scores 94
• threshold_violations tells an engineer why it failed, which is the difference between “fix two high findings” and “start a hardening project”
• Push the sarif_url into GitHub code scanning so findings land on the pull request, not in a CI log

What’s Next

The CI/CD compliance gate closes the final gap: even if an unhardened image gets built, it can’t deploy. EP05 is the bookmark episode — this is the point where OS hardening becomes structurally enforced rather than procedurally expected.

EP06 is the series closer. For five episodes, you’ve been using BakeX as a user. What does it look like to run it yourself — extend it with a custom provider, deploy it in your own infrastructure, or contribute a blueprint back?

BakeX is Apache 2.0. EP06 is the architecture reveal, the deployment guide, and the extension points for everything the series taught.

Next: BakeX — open-source OS hardening platform for multi-cloud infrastructure

Get EP06 in your inbox when it publishes → linuxcent.com/subscribe

The post The Pipeline Gate — Hardened Images as a CI/CD Build Constraint appeared first on Linuxcent.

Introduction

By 2023, the question had shifted from “how do we run Kubernetes?” to “how do we let other engineers run their workloads on Kubernetes without becoming a bottleneck?”

This is the platform engineering problem. And it drove the tooling that defined 2023–2025: GitOps as the deployment standard, Cluster API for Kubernetes-on-Kubernetes provisioning, AI/ML workloads forcing new scheduling capabilities, and the Kubernetes project itself shedding more weight to become faster to release and operate.

GitOps: Principle Becomes Practice

GitOps as a term was coined by Weaveworks in 2017. By 2023, it was no longer a debate — it was the default deployment model for organizations running Kubernetes at scale.

The principle: the desired state of your cluster lives in Git. A controller watches the repository and reconciles the cluster state to match. Every deployment is a PR merge. The audit trail is the Git history.

Flux v2 (CNCF graduated) and ArgoCD (CNCF incubating) became the two dominant implementations:

# Flux: GitRepository + Kustomization
apiVersion: source.toolkit.fluxcd.io/v1
kind: GitRepository
metadata:
  name: production-config
  namespace: flux-system
spec:
  interval: 1m
  url: https://github.com/org/k8s-config
  ref:
    branch: main
---
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: production-apps
  namespace: flux-system
spec:
  interval: 10m
  path: ./clusters/production
  prune: true          # Remove resources deleted from Git
  sourceRef:
    kind: GitRepository
    name: production-config
  healthChecks:
  - apiVersion: apps/v1
    kind: Deployment
    name: api
    namespace: production

The prune: true behavior is critical: resources deleted from Git are deleted from the cluster. This is what makes GitOps a security control — unknown resources that aren’t in Git get removed. No more accumulation of forgotten test deployments, rogue debug pods, or unauthorized configuration changes that outlive the engineer who made them.

ArgoCD’s Application model added a UI, synchronization policies, and multi-cluster management:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: production-api
  namespace: argocd
spec:
  project: production
  source:
    repoURL: https://github.com/org/apps
    targetRevision: HEAD
    path: api/production
  destination:
    server: https://kubernetes.default.svc
    namespace: api
  syncPolicy:
    automated:
      prune: true
      selfHeal: true    # Revert manual kubectl changes
    syncOptions:
    - CreateNamespace=true

The selfHeal: true option is where GitOps becomes enforceable: any manual change made with kubectl is automatically reverted within the sync interval. For compliance-sensitive environments, this is a configuration drift prevention control.

Cluster API: Kubernetes Managing Kubernetes

Cluster API (cluster-sigs/cluster-api) flipped the usual model: instead of using tools like Terraform or Ansible to provision Kubernetes clusters, Cluster API lets you manage Kubernetes clusters as Kubernetes resources — using a management cluster to provision and manage workload clusters.

# Create a new Kubernetes cluster as a Kubernetes resource
apiVersion: cluster.x-k8s.io/v1beta1
kind: Cluster
metadata:
  name: workload-cluster-prod
spec:
  clusterNetwork:
    pods:
      cidrBlocks: ["192.168.0.0/16"]
  infrastructureRef:
    apiVersion: infrastructure.cluster.x-k8s.io/v1beta2
    kind: AWSCluster
    name: workload-cluster-prod
  controlPlaneRef:
    apiVersion: controlplane.cluster.x-k8s.io/v1beta1
    kind: KubeadmControlPlane
    name: workload-cluster-prod-control-plane

Cluster API reconciliation handles cluster provisioning, scaling, upgrades, and deletion — all through the Kubernetes API, with all the tooling (RBAC, audit logging, GitOps integration) that entails. Multi-cluster platform teams could now manage hundreds of workload clusters from a single management cluster.

Kubernetes 1.28 — Sidecar Containers Alpha (August 2023)

Sidecar containers had been a Kubernetes pattern since 2015 — a helper container in the same pod as the main application. But there was no native sidecar lifecycle management. Sidecars were just regular init containers or additional containers, which meant:
– Init container sidecars ran before the application and had to block until they succeeded
– Regular container sidecars had no ordering guarantees at startup
– At pod termination, sidecars could die before the application finished draining

1.28 introduced native sidecar support: a new restartPolicy field for init containers:

spec:
  initContainers:
  - name: log-collector
    image: fluentbit:latest
    restartPolicy: Always    # This makes it a sidecar
    # Starts before main containers, stays running, stops after main containers exit
  containers:
  - name: application
    image: myapp:latest

A sidecar container (init container with restartPolicy: Always):
– Starts before application containers
– Stays running throughout the pod lifecycle
– Terminates automatically after all main containers exit
– Restarts if it crashes (unlike regular init containers)

This solved the service mesh sidecar problem: Istio and Linkerd injected Envoy proxies as regular containers, leading to race conditions where the proxy hadn’t started when the application tried to make outbound connections. Native sidecar lifecycle guarantees the proxy is ready before the application starts.

Also in 1.28:
– Retroactive default StorageClass assignment: Existing PVCs without a StorageClass assignment get the default applied retroactively — useful for migrations
– Non-graceful node shutdown stable: Handle node power failures without manual pod cleanup
– Recovery from volume expansion failure: Previously, a failed volume expansion left the PVC in a broken state; 1.28 introduced a mechanism to recover

AI/ML Workloads Force New Kubernetes Capabilities

The LLM wave of 2023 drove GPU workloads onto Kubernetes at a scale and urgency the project hadn’t anticipated. Running LLM inference on Kubernetes required solving problems that CPU-centric cluster scheduling hadn’t encountered:

GPU topology awareness: Inference across multiple GPUs requires GPUs connected by NVLink or on the same PCIe switch, not arbitrary GPUs from different nodes or different PCIe buses. The Dynamic Resource Allocation API (1.26 alpha) was designed exactly for this.

Fractional GPU allocation: NVIDIA’s time-slicing and MIG (Multi-Instance GPU) allow multiple pods to share a single GPU. The GPU operator (NVIDIA) manages this at the node level:

# Check GPU resources visible to Kubernetes
kubectl get nodes -o custom-columns=\
  "NODE:.metadata.name,GPU:.status.allocatable.nvidia\.com/gpu"
# NODE       GPU
# gpu-node-1   8
# gpu-node-2   8

Batch scheduling for training jobs: Training runs require all workers to start simultaneously — a single missing GPU makes the entire job stall. The Kubernetes Job API doesn’t guarantee this. Projects like Volcano (CNCF incubating) and Kueue (Kubernetes SIG Scheduling) added gang scheduling: a job only starts when all requested resources are available.

# Kueue: queue AI training jobs with resource quotas
apiVersion: kueue.x-k8s.io/v1beta1
kind: ClusterQueue
metadata:
  name: gpu-queue
spec:
  namespaceSelector: {}
  resourceGroups:
  - coveredResources: ["nvidia.com/gpu", "cpu", "memory"]
    flavors:
    - name: a100-80gb
      resources:
      - name: nvidia.com/gpu
        nominalQuota: 16

Kubernetes 1.29 — Sidecar to Beta, Load Balancer IP Mode (December 2023)

• Sidecar containers beta: The lifecycle semantics were refined based on 1.28 alpha feedback
• Load balancer IP mode alpha: Distinguish between load balancers that use virtual IPs (kube-proxy handles the traffic) vs. those that handle traffic directly (no need for kube-proxy rules) — important for eBPF-based load balancers
• ReadWriteOncePod volume access stable

Kubernetes 1.30 — Structured Authorization Config (April 2024)

• Structured authorization configuration beta: Define multiple authorization webhooks with explicit ordering, failure modes, and connection settings — replacing the flat --authorization-mode flag
• Sidecar containers beta continues
• Node memory swap support beta: Allow pods to use swap memory — controversial but necessary for workloads with bursty memory patterns that prefer using swap over OOM kill

# Node with swap enabled — kubelet config
kind: KubeletConfiguration
memorySwap:
  swapBehavior: LimitedSwap

The swap support feature reversed a long-standing Kubernetes hard stance: swap was disabled since 1.0 because its interaction with Kubernetes memory accounting was unpredictable. The 1.30 approach adds proper accounting and policies.

Kubernetes 1.31 — Cloud Provider Code Removal Complete (August 2024)

1.31 marked the completion of the cloud provider code removal — the 1.5 million line migration that had been running since 1.26. Core binaries are 40% smaller. The API server, controller manager, and scheduler no longer contain vendor-specific code.

Also in 1.31:
– Persistent Volume health monitor stable
– AppArmor support stable: AppArmor profiles for pods using the native Kubernetes field (not annotations)
– Traffic distribution for Services beta: Express topology preferences for Service routing (prefer local node, prefer same zone)

# Traffic distribution: prefer endpoints in the same zone
apiVersion: v1
kind: Service
metadata:
  name: api
spec:
  trafficDistribution: PreferClose
  selector:
    app: api
  ports:
  - port: 80
    targetPort: 8080

Kubernetes 1.32 — Sidecar Stable, DRA Beta (December 2024)

• Sidecar containers stable: After nearly a decade of workarounds, the sidecar pattern is a first-class Kubernetes primitive
• Dynamic Resource Allocation beta: GPU and specialized hardware scheduling ready for production evaluation
• Job API improvements: Success and failure policies for indexed jobs — granular control over batch workload behavior
• Custom Resource field selectors: Filter CRDs on arbitrary fields — making large CRD-based systems more efficient to query

Crossplane: Kubernetes as the Control Plane for Everything

Crossplane (CNCF graduated) extended the Kubernetes API model beyond the cluster itself. Using CRDs and controllers, Crossplane lets you manage cloud resources (RDS databases, S3 buckets, VPCs, IAM roles) as Kubernetes resources — provisioned, updated, and deleted through the Kubernetes API.

# Crossplane: provision an RDS PostgreSQL instance as a Kubernetes resource
apiVersion: database.aws.crossplane.io/v1beta1
kind: RDSInstance
metadata:
  name: production-db
spec:
  forProvider:
    region: us-east-1
    dbInstanceClass: db.r6g.xlarge
    masterUsername: admin
    engine: postgres
    engineVersion: "15"
    allocatedStorage: 100
    multiAZ: true
  writeConnectionSecretsToRef:
    name: production-db-credentials
    namespace: production

For platform teams, Crossplane means a single control plane — the Kubernetes API — for both compute workloads and cloud infrastructure. GitOps tools (Flux, ArgoCD) manage both.

Key Takeaways

• GitOps (Flux, ArgoCD) became the production deployment standard — not for ideological reasons, but because the audit trail, drift detection, and self-healing properties solve real operational and compliance problems
• Cluster API made Kubernetes cluster lifecycle (provisioning, upgrades, deletion) a Kubernetes-native operation — the same API, tooling, and audit trail
• Native sidecar containers (1.28 alpha → 1.32 stable) finally resolved the lifecycle ordering problem that service meshes and log collectors had worked around for years
• AI/ML workloads drove new scheduling capabilities (DRA, gang scheduling via Kueue/Volcano) and made GPU topology awareness a first-class concern
• Crossplane generalized the Kubernetes API model to cloud infrastructure — the cluster is now a control plane for everything, not just containers

What’s Next

← EP06: The Runtime Reckoning | EP08: Kubernetes Today →

Series: Kubernetes: From Borg to Platform Engineering | linuxcent.com

The post The Platform Engineering Era: GitOps, AI Workloads, and Leaner Kubernetes (2023–2025) appeared first on Linuxcent.