Compliance Grading — Automated OpenSCAP with A-F Scores Before Deployment

Reading Time: 6 minutes

OS Hardening as Code, Episode 4
Cloud AMI Security Risks · Linux Hardening as Code · Multi-Cloud OS Hardening · Automated OpenSCAP Compliance**

Note: the tool in this series was released as Stratum and renamed to BakeX at
v0.6.0 — same project, same license, same team. Commands below use the current bakex
CLI. If you arrived here looking for stratum or pip install stratumoss, you’re in the
right place: github.com/invicton/bakex.


TL;DR

  • “We use CIS L1” means nothing without a verified grade — automated OpenSCAP compliance provides one before any instance is deployed
  • BakeX runs OpenSCAP as a stage of every build, and the scan result carries a letter grade A–F
  • The grade is OpenSCAP’s own XCCDF score mapped to a letter: A ≥ 90, B ≥ 75, C ≥ 60, D ≥ 40, F below that
  • SARIF output is machine-readable — importable directly into GitHub Advanced Security, Jira, or any SIEM
  • Scanning and baseline comparison live in the web UI and HTTP API, not the CLI — the CLI is validate and build
  • A build whose scan fails the blueprint’s threshold ends in Status: failed with exit code 1, and no image is snapshotted

The Problem: A Grade That’s Never Been Verified Is Not a Grade

Security audit request:
"Provide CIS L1 compliance evidence for all production instances"

Team response:
  Instance A: "CIS L1 hardened" — OpenSCAP last run: 4 months ago
  Instance B: "CIS L1 hardened" — OpenSCAP last run: never
  Instance C: "CIS L1 hardened" — OpenSCAP version: 1.2 (current: 1.3.8)
  Instance D: "CIS L1 hardened" — manual scan output: "87% passing"
  Instance E: "CIS L1 hardened" — manual scan output: "91% passing"

"Which profile was used for D and E? Are they comparable?"
"Were they scanned before or after a recent kernel update?"
"Why is C running an old OpenSCAP version?"

Automated OpenSCAP compliance means the grade is generated the same way, on every image, every time, before the image is ever deployed.

EP03 showed that the same HardeningBlueprint YAML builds consistent OS images across six cloud providers. What it left open is the question every auditor eventually asks: how do you know the Ansible hardening actually did what you think it did? Running Ansible-Lockdown successfully means the tasks ran. It does not mean every CIS control is satisfied — some controls can’t be applied by Ansible alone, some require manual verification, and some interact with the environment in unexpected ways.


A compliance team requested CIS L2 evidence for a SOC 2 Type II audit. The security team had been running OpenSCAP scans — but manually, on-demand, using slightly different profiles across teams, with no standard for how to store or compare results.

The audit found four problems:
1. Two instances had been scanned with CIS L1, not L2, despite being labeled “CIS L2”
2. Three instances hadn’t been scanned in over six months
3. The scan outputs from different teams were in different formats (HTML vs XML vs text)
4. Two instances showed “91% passing” and “89% passing” — with no documentation of whether those were acceptable thresholds or what the failing controls were

The audit took two weeks to resolve. The finding wasn’t a security failure — it was a documentation and process failure. But it consumed two weeks of engineering time and appeared in the audit report as a gap.

The root cause: compliance scanning was a manual step that produced inconsistent output in an inconsistent format.


How Automated OpenSCAP Compliance Works

Scanning is a stage of the build, not an afterthought you remember to run:

bakex build blueprints/ubuntu/22.04/cis-l1-aws.yaml
      │
      ├─ Provisioning via aws
      │
      ├─ Applying pre-hardening system configuration
      │    (hostname, filesystem, users)
      │
      ├─ Applying Ansible-Lockdown hardening roles
      │
      ├─ Running OpenSCAP compliance scan
      │    ├── benchmark:  xccdf_org.ssgproject.content_benchmark_UBUNTU2204
      │    ├── profile:    ...content_profile_cis_level1_server
      │    └── datastream: ssg-ubuntu2204-ds.xml
      │
      ├─ Snapshotting golden image
      │
      └─ Image ready: ami-0a7f3c9e82d1b4c05

All three compliance identifiers come from the blueprint’s compliance block, and they are full
XCCDF strings rather than friendly names like cis-l1 — they’re handed to oscap unmodified, so
there is no name-mapping layer that can silently pick the wrong profile. That single detail
answers the audit question “which profile was actually used?” without anyone having to remember.

Ubuntu is a special case worth knowing: it ships no SCAP content package in the archive, so BakeX
downloads the matching datastream from a ComplianceAsCode release and checksum-verifies it rather
than failing or silently scanning nothing.


The A-F Grade Calculation

The grade is deliberately boring, and that is the point. BakeX does not invent a scoring model —
it takes OpenSCAP’s own XCCDF score and maps it to a letter:

def score_to_grade(score: float) -> str:
    if score >= 90: return "A"
    if score >= 75: return "B"
    if score >= 60: return "C"
    if score >= 40: return "D"
    return "F"
Grade Score Meaning
A ≥ 90 Production-ready, minimal exceptions
B ≥ 75 Acceptable with documented exceptions
C ≥ 60 Below standard — deploy with caution
D ≥ 40 Significant gaps — do not deploy to production
F < 40 Hardening failed

The thresholds are fixed, not per-blueprint tunables. That is a defensible choice: a grade you can
adjust in the file being graded is not evidence, it’s decoration. If an A means ≥ 90 everywhere,
two teams’ grades are comparable without reading their blueprints — which was exactly the failure
in the audit story above.

What is configurable is when the build refuses to continue:

compliance:
  benchmark: xccdf_org.ssgproject.content_benchmark_UBUNTU2204
  profile: xccdf_org.ssgproject.content_profile_cis_level1_server
  datastream: /usr/share/xml/scap/ssg/content/ssg-ubuntu2204-ds.xml
  fail_on_findings: true      # findings at/above the threshold fail the build
  severity_threshold: medium  # critical | high | medium | low

fail_on_findings with a severity_threshold is severity-based rather than score-based, which
tends to match how people actually reason about risk: one critical finding should block a release
even when 94% of rules pass. When it trips, the build ends in Status: failed, exit code 1, and
no image is snapshotted.


Where the Scan Surface Actually Lives

Worth being blunt about this, because it is the most common wrong assumption: there is no
bakex scan command.
The CLI is two verbs — validate and build. Scanning, history, and
baseline comparison live in the web app and its HTTP API, because scan results need somewhere to
persist and something to render them.

Start the server and the whole surface is there:

bakex serve --port 8000

The auditor API is mounted at /api/auditor:

Endpoint What it does
POST /api/auditor/scan-image Scan an image and return a job
POST /api/auditor/scan-container Same, for a container image
GET /api/auditor/jobs List scan jobs
GET /api/auditor/jobs/{job_id} One job, with grade and severity counts
GET /api/auditor/jobs/{job_id}/compare/{baseline_id} Diff a scan against a baseline
GET /api/auditor/scan-image/{job_id}/report?fmt=… Export the report
GET /api/auditor/scan-image/{job_id}/badge.svg Grade badge for a README

SARIF Export

The report endpoint speaks three formats, selected by query parameter:

# Human-readable — printable HTML, print-to-PDF from the browser
curl "http://localhost:8000/api/auditor/scan-image/$JOB/report?fmt=html"

# Machine-readable job dict
curl "http://localhost:8000/api/auditor/scan-image/$JOB/report?fmt=json"

# SARIF 2.1.0 — the one that matters for CI
curl -o scan.sarif.json \
  "http://localhost:8000/api/auditor/scan-image/$JOB/report?fmt=sarif"

SARIF 2.1.0 is the standard interchange format for security scan results, which means the OpenSCAP
findings land wherever your other scanners’ findings already land:

  • GitHub Advanced Security — upload with github/codeql-action/upload-sarif; findings appear in the Security tab, annotated on the PR
  • Azure DevOps — native SARIF viewer
  • Splunk / SIEM — structured JSON, parseable as events
  • AWS Security Hub — importable as findings via the Security Hub API

For audit purposes the SARIF file is the evidence artifact: it carries every rule result, the
profile that was used, and the timestamp. “91% passing” in a spreadsheet is a claim. A SARIF file
in the Security tab is a record.

The badge endpoint is the small touch that gets used most — badge.svg renders the letter grade,
so a repo’s README can show the compliance grade of the image it builds, next to the CI badge.


Drift: Comparing Against a Baseline

The comparison endpoint takes two job IDs — a current scan and a stored baseline — and reports the
delta, including the change in score:

curl "http://localhost:8000/api/auditor/jobs/$CURRENT/compare/$BASELINE"

That is the mechanism behind “what changed since we built this.” You scan the image at build time,
keep that job as the baseline, and re-scan later; the comparison tells you which rules moved and
which direction the score went. It is how you find the instance somebody modified “temporarily”
and never reverted.

The honest limitation: this compares scan jobs, so drift detection is as good as your discipline
about scanning on a schedule. Nothing re-scans your fleet for you.


What Controls Typically Block an A Grade

For Ubuntu 22.04 CIS L1 builds in most cloud environments, these are the controls that most commonly prevent an A grade:

Control Why it often fails Fix
1.1.7 /var/log/audit separate partition Cloud images don’t have separate volumes at build time Add EBS volume, configure at launch
1.6.1 AppArmor bootloader config GRUB parameters not set correctly Update /etc/default/grub, run update-grub
3.1.1 Disable IPv6 Cloud networking sometimes requires IPv6 Override with documented reason if intentional
5.2.21 SSH MaxStartups Default sshd_config not updated Add MaxStartups 10:30:60 to sshd_config
6.1.10 World-writable files Some package installations leave world-writable files Post-install cleanup in Ansible role

The first two (separate audit partition, AppArmor bootloader) are the most common A→B blockers and often require architecture decisions about how volumes are provisioned at launch versus build time.


Key Takeaways

  • Automated OpenSCAP compliance means every image has a verified, reproducible grade generated by the same scanner with the same profile, before it’s ever deployed
  • The grade is OpenSCAP’s own XCCDF score mapped to a fixed scale (A ≥ 90, B ≥ 75, C ≥ 60, D ≥ 40) — fixed on purpose, so grades from two teams are comparable without reading their blueprints
  • The build gate is severity-based, not score-based: fail_on_findings plus severity_threshold blocks a release on one critical finding even when most rules pass
  • SARIF 2.1.0 export makes scan results importable into GitHub Advanced Security, Azure DevOps, SIEM, and audit tooling — the SARIF file is the evidence artifact
  • Scanning and baseline comparison are HTTP API surfaces, not CLI commands; the CLI is validate and build

What’s Next

Automated OpenSCAP compliance gives every image a verified grade before deployment. What EP04 left open is what happens after the grade is known — specifically, what prevents an engineer from deploying a C-grade image to production “just this once.”

The Pipeline API is the answer. EP05 covers the CI/CD compliance gate: POST /api/pipeline/scan fails the build if the image grade is below threshold. The unhardened image never reaches production — not because engineers are disciplined, but because the pipeline won’t let it through.

Next: CI/CD compliance gate — block unhardened images before they reach production

Get EP05 in your inbox when it publishes → linuxcent.com/subscribe

Hardening Blueprint as Code — Declare Your OS Baseline in YAML

Reading Time: 8 minutes

OS Hardening as Code, Episode 2
Cloud AMI Security Risks · Linux Hardening as Code**

Note: the tool in this series was released as Stratum and renamed to BakeX at
v0.6.0 — same project, same license, same team. Commands below use the current bakex
CLI. If you arrived here looking for stratum or pip install stratumoss, you’re in the
right place: github.com/invicton/bakex.


TL;DR

  • A hardening runbook is a list of steps someone runs. A HardeningBlueprint YAML is a build artifact — if it wasn’t applied, the image doesn’t exist
  • Linux hardening as code means declaring your entire OS security baseline in a single YAML file and building it reproducibly across any provider
  • bakex build blueprints/ubuntu/22.04/cis-l1-aws.yaml either produces a hardened image or fails — there is no partial state
  • The blueprint includes: target OS/provider/base image, compliance benchmark, filesystem layout, users, and per-rule overrides with documented justifications
  • One blueprint file = one source of truth for your hardening posture, version-controlled and reviewable like any other infrastructure code
  • Post-build OpenSCAP scan runs automatically — the image only snapshots if it passes

The Problem: A Runbook That Gets Skipped Once Is a Runbook That Gets Skipped

Hardening runbook
       │
       ▼
  Human executes
  steps manually
       │
       ├─── 47 deployments: followed correctly
       │
       └─── 1 deployment at 2am: step 12 skipped
                    │
                    ▼
           Instance in production
           without audit logging,
           SSH password auth enabled,
           unnecessary services running

Linux hardening as code eliminates the human decision point. If the blueprint wasn’t applied, the image doesn’t exist.

EP01 showed that default cloud AMIs arrive pre-broken — unnecessary services, no audit logging, weak kernel parameters, SSH configured for convenience not security. The obvious response is a hardening script. But a script run by a human is still a process step. It can be skipped. It can be done halfway. It can drift across different engineers who each interpret “run the hardening script” slightly differently.


A production deployment last year. The platform team had a solid CIS L1 hardening runbook — 68 steps, well-documented, followed consistently. Then a critical incident at 2am required three new instances to be deployed on short notice. The engineer on call ran the provisioning script and, under pressure, skipped the hardening step with the intention of running it the next morning.

They didn’t. The three instances stayed in production unhardened for six weeks before an automated scan caught them. Audit logging wasn’t configured. SSH was accepting password authentication. Two unnecessary services were running that weren’t in the approved software list.

Nothing was breached. But the finding went into the next compliance report as a gap, the team spent a week remediating, and the post-mortem conclusion was “we need better runbook discipline.”

That’s the wrong conclusion. The runbook isn’t the problem. The problem is that hardening was a process step instead of a build constraint.


What Linux Hardening as Code Actually Means

Linux hardening as code is the same principle as infrastructure as code applied to OS security posture: the desired state is declared in a file, the file is the source of truth, and the execution is deterministic and repeatable.

HardeningBlueprint YAML
         │
         ▼
  bakex build
         │
  ┌──────┴──────────────────┐
  │  Provider Layer          │
  │  (cloud-init, disk       │
  │   names, metadata        │
  │   endpoint per provider) │
  └──────┬──────────────────┘
         │
  ┌──────┴──────────────────┐
  │  Ansible-Lockdown        │
  │  (CIS L1/L2, STIG —      │
  │   the hardening steps)   │
  └──────┬──────────────────┘
         │
  ┌──────┴──────────────────┐
  │  OpenSCAP Scanner        │
  │  (post-build verify)     │
  └──────┬──────────────────┘
         │
         ▼
  Golden Image (AMI/GCP image/Azure image)
  + Compliance grade in image metadata

The YAML file is what you write. BakeX handles the rest.


The HardeningBlueprint YAML

The blueprint is the complete, auditable declaration of your OS security posture:

# blueprints/ubuntu/22.04/cis-l1-aws.yaml
bakex_version: "0.6.0"
kind: HardeningBlueprint

metadata:
  name: ubuntu22-cis-l1-aws
  version: "1.0.0"
  description: >
    CIS Ubuntu Linux 22.04 LTS Benchmark — Level 1 Server profile for AWS.
  tags: [ubuntu, ubuntu22.04, cis, level1, server, aws]

target:
  os: ubuntu22.04
  arch: x86_64
  provider: aws
  base_image: ami-0c7217cdde317cfec   # Ubuntu 22.04 LTS, us-east-1 (x86_64)
  instance_type: t3.medium
  root_volume_size_gb: 20

system:
  hostname: hardened-node
  timezone: UTC
  locale: en_US.UTF-8
  selinux_mode: null              # Ubuntu uses AppArmor; null skips SELinux

filesystem:                       # CIS 1.1.x: separate mounts with noexec/nosuid/nodev
  - device: tmpfs
    mountpoint: /tmp
    fstype: tmpfs
    options: [rw, nosuid, nodev, noexec, relatime]
    size: 2G

  - device: tmpfs
    mountpoint: /dev/shm
    fstype: tmpfs
    options: [rw, nosuid, nodev, noexec, relatime]

users:
  root:
    lock: true                    # CIS 5.4.2: lock the root account
  accounts:
    - name: bakex-admin
      groups: [sudo]
      shell: /bin/bash
      ssh_authorized_keys: []     # Add: - "ssh-ed25519 AAAA..."

compliance:
  benchmark: xccdf_org.ssgproject.content_benchmark_UBUNTU2204
  profile: xccdf_org.ssgproject.content_profile_cis_level1_server
  datastream: /usr/share/xml/scap/ssg/content/ssg-ubuntu2204-ds.xml
  fail_on_findings: true
  severity_threshold: medium

controls:                         # per-rule overrides, keyed by XCCDF rule ID
  xccdf_org.ssgproject.content_rule_grub2_enable_selinux:
    enabled: false
    justification: >
      Ubuntu Linux uses AppArmor as the mandatory access control framework.
      SELinux is not installed or applicable on this distribution.

  xccdf_org.ssgproject.content_rule_package_telnet_removed: true

That is not a simplified illustration — it is the shipped cis-l1-aws.yaml, trimmed of comments. Each section is explicit:

target — which OS, which provider, and which base image to harden. This is the only provider-specific section. Everything below it is portable.

compliance — the OpenSCAP benchmark, profile, and datastream. These are full XCCDF identifiers, not friendly names like cis-l1, because they’re passed straight to oscap. Ubuntu ships no SCAP content package, so BakeX downloads the matching datastream from a ComplianceAsCode release and checksum-verifies it.

filesystem — a list of mounts, each with its CIS-required options.

users — root lock state and the service accounts baked into the image.

controls — documented exceptions, keyed by XCCDF rule ID. Not suppressions — overrides with a recorded justification. A bare true enforces a rule; a map with enabled: false disables it and demands a reason. This is the difference between “we turned off this control” and “this control is not applicable, documented here.”


Building the Image

# Validate the blueprint before building — exit 0 if valid, 1 if not
bakex validate blueprints/ubuntu/22.04/cis-l1-aws.yaml

# Build — this will take 15-25 minutes
bakex build blueprints/ubuntu/22.04/cis-l1-aws.yaml

Validation is fast and offline:

OK    blueprints/ubuntu/22.04/cis-l1-aws.yaml  (HardeningBlueprint 'ubuntu22-cis-l1-aws')

1/1 blueprint(s) valid.

The build walks five stages, and the provider is read from the blueprint — there is no --provider flag to get wrong:

Building 'ubuntu22-cis-l1-aws' (aws) → job 7f3c9e82-4d1b-4c05-a7f3-c9e82d1b4c05
[2026-07-27T15:42:01] Provisioning via aws
[2026-07-27T15:42:45] Applying pre-hardening system configuration (hostname, filesystem, users)
[2026-07-27T15:43:12] Applying Ansible-Lockdown hardening roles
[2026-07-27T15:52:11] Running OpenSCAP compliance scan
[2026-07-27T15:54:10] Snapshotting golden image
[2026-07-27T15:54:47] Image ready: ami-0a7f3c9e82d1b4c05

Status: complete

Two things worth noticing. bakex build also accepts a bundled profile name, so bakex build ubuntu22-cis-l1-aws does the same thing without a path. And --json emits the job record instead of the log, which is what you want in CI.

If the post-build scan comes back below the configured threshold, the build fails — no AMI is created. The instance is terminated. The image does not exist. Exit code is 1, and Status: failed carries the reason.

That is the structural guarantee. You cannot skip a build step at 2am because at 2am you’re calling bakex build, not running steps manually.


The Control Override Mechanism

The override mechanism is what separates this from checkbox compliance.

Every security benchmark has controls that conflict with how production environments actually work. CIS L1 recommends /tmp on a separate partition. Many cloud instances use tmpfs with equivalent nodev, nosuid, noexec mount options. The intent of the control is satisfied. The literal implementation differs.

Without an override mechanism, you have two bad options: fail the scan (noisy, meaningless), or configure the scanner to ignore the control (undocumented, invisible to auditors).

The blueprint’s controls section gives you a third option: record the override and its justification in the same version-controlled artifact that produced the image.

controls:
  xccdf_org.ssgproject.content_rule_grub2_enable_selinux:
    enabled: false
    justification: >
      Ubuntu Linux uses AppArmor as the mandatory access control framework.
      SELinux is not installed or applicable on this distribution.

Note that the key is the full XCCDF rule ID, not a CIS section number. That is deliberate — the rule ID is what OpenSCAP reports, so the override and the scanner finding line up exactly with no translation table in between.

Be precise about what this does today. As of v0.6.0 the controls block is declarative:
it records intent alongside the blueprint, and the UI reads it back, but it is not yet compiled
into an OpenSCAP tailoring file. The scan still evaluates the full profile, so an overridden
rule still shows up as a finding and still counts against the score. What you get right now is
provenance — the justification lives in git, next to the thing that built the image, reviewable
in a pull request — not automatic score adjustment.

That is a smaller claim than “documented exceptions are counted as compliant,” and it’s the
true one. Wiring overrides through to scan tailoring is the obvious next step, and it’s the
kind of well-scoped gap that makes a good first contribution.


What the Blueprint Gives You That a Script Doesn’t

Hardening script HardeningBlueprint YAML
Version-controlled Possible but not enforced Always — it’s a file
Auditable exceptions Typically not Built-in override mechanism
Post-build verification Manual or none Automatic OpenSCAP scan
Image exists only if hardened No Yes — build fails if scan fails
Multi-cloud portability Requires separate scripts Swap the target block; compliance sections stay identical
Drift detection Not possible Rescan instance against original grade
Skippable at 2am Yes No — you’d have to change the build process

The last row is the one that matters. A script is skippable because there’s a human in the loop. A blueprint is a build artifact — you can’t deploy the image without the blueprint having been applied, because the image is what the blueprint produces.


Validating a Blueprint Before Building

# Schema validation — one file
bakex validate blueprints/ubuntu/22.04/cis-l1-aws.yaml

# Or the whole library at once
bakex validate blueprints/**/*.yaml

# Machine-readable, for CI and agents
bakex validate blueprints/ubuntu/22.04/cis-l1-aws.yaml --json

bakex validate exits 0 when every file is valid and 1 when any file fails, which makes it a one-line CI gate. It never touches a cloud API — it is pure schema and cross-field checking, so it runs in your pipeline before you’ve paid for a build instance.

That cross-field part matters more than it sounds. Validation rejects OS/provider combinations the catalog doesn’t support, so a blueprint asking for an OS that provider can’t supply fails at validation time rather than fifteen minutes into a build.

If you want to generate or check blueprints from something other than the CLI, the format is published as a JSON Schema (Draft 2020-12) at docs/schema/hardening-blueprint.schema.json. Point your editor at it for autocomplete, or hand it to an LLM and let it write the blueprint.


Production Gotchas

Build time is 15–25 minutes. Ansible-Lockdown applies 144+ tasks for CIS L1. Build this into your pipeline timing — don’t expect golden images in 3 minutes.

Cloud-init ordering matters. On AWS, certain hardening steps (sysctl tuning, PAM configuration) interact with cloud-init. The BakeX provider layer handles sequencing — but if you add custom hardening roles, test the cloud-init interaction explicitly.

Some CIS controls conflict with managed service requirements. AWS Systems Manager Session Manager requires specific SSH configuration. RDS requires specific networking settings. Use the controls override section to document these — don’t suppress them silently.

Kernel parameter hardening requires a reboot. Controls in the 3.x (network parameters) and 1.5.x (kernel modules) sections apply sysctl changes that take effect on reboot. The BakeX build process reboots the instance before the OpenSCAP scan — don’t skip the reboot if you’re building manually.


Key Takeaways

  • Linux hardening as code means the blueprint YAML is the build artifact — the image either exists and is hardened, or it doesn’t exist
  • The controls override mechanism is the difference between undocumented suppressions and auditable, reasoned exceptions
  • Post-build OpenSCAP scan runs automatically — a failing grade blocks image creation
  • bakex validate is an offline, exit-code-shaped CI gate — it catches unsupported OS/provider pairs before a build instance is ever launched
  • The compliance sections are portable across providers (EP03 covers this): swap the target block, and the benchmark, filesystem, users, and control overrides stay byte-identical
  • Version-controlling the blueprint gives you a complete history of what your OS security posture was at any point in time — the same way Terraform state tracks infrastructure

What’s Next

One blueprint, one provider. EP02 showed that the skip-at-2am problem is solved when hardening is a build artifact rather than a process step.

What it didn’t address: what happens when you expand to a second cloud. GCP uses different disk names. Azure cloud-init fires in a different order. The AWS metadata endpoint IP is different from every other provider. If you maintain separate hardening scripts per cloud, they drift within a month.

EP03 covers multi-cloud OS hardening: one compliance posture, six providers, no drift — and it shows the diff that proves it.

Next: multi-cloud OS hardening — one blueprint for AWS, GCP, and Azure

Get EP03 in your inbox when it publishes → linuxcent.com/subscribe