OS Hardening as Code, Episode 3
Cloud AMI Security Risks · Linux Hardening as Code · Multi-Cloud OS Hardening**
Note: the tool in this series was released as Stratum and renamed to BakeX at
v0.6.0 — same project, same license, same team. Commands below use the currentbakex
CLI. If you arrived here looking forstratumorpip install stratumoss, you’re in the
right place: github.com/invicton/bakex.
TL;DR
- Multi-cloud OS hardening with separate scripts per provider means three scripts that drift within weeks
- A HardeningBlueprint YAML separates compliance intent (portable) from provider details (handled by BakeX’s provider layer)
- You keep one blueprint file per provider — and every section except
targetstays byte-identical across all six. The diff below proves it - Provider-specific differences — disk names, cloud-init ordering, base image identifiers — are abstracted away from the blueprint author
- The compliance posture becomes reviewable in a pull request: a control change touches six files identically, and a reviewer can see that at a glance
- Six providers ship as working blueprints today: AWS, GCP, Azure, DigitalOcean, Linode, Proxmox
The Problem: Three Clouds, Three Scripts, Three Ways to Drift
AWS hardening script GCP hardening script Azure hardening script
├── /dev/xvd* disk refs ├── /dev/sda* disk refs ├── /dev/sda* disk refs
├── 169.254.169.254 IMDS ├── 169.254.169.254 IMDS ├── 169.254.169.254 IMDS
├── cloud-init order A ├── cloud-init order B ├── cloud-init order C
└── Updated: Jan 2025 └── Updated: Aug 2024 └── Updated: Mar 2024
│
└─ 5 months behind
on CIS updates
Multi-cloud OS hardening starts as a copy-paste of the AWS script. Within a month, the clouds diverge.
EP02 showed that a HardeningBlueprint YAML eliminates the skip-at-2am problem by making hardening a build artifact. What it assumed — quietly — is that you’re building for one provider. The moment you expand to a second cloud, the provider-specific details in the blueprint become a problem: disk names differ, cloud-init fires in a different order, and AWS-specific assumptions break silently on GCP.
We expanded from AWS to GCP six months ago. The EC2 hardening script had been working reliably for over a year. The GCP engineer took the AWS script, made some quick changes, and started building images.
The first GCP images had a subtle problem: the /tmp and /home separate partition entries in /etc/fstab referenced /dev/xvdb — an AWS disk naming convention. GCP uses /dev/sdb. The fstab entries were silently ignored. The mounts existed but weren’t restricted. The CIS controls for separate filesystem partitions were listed as passing in the scan output because the Ansible task had “run successfully” — it just hadn’t done what we thought.
It took a pentest three months later to catch it. The finding: six production GCP instances with /tmp not mounted with noexec, nosuid, nodev — despite our “CIS L1 hardened” label.
The root cause wasn’t the engineer. It was a hardening approach that required cloud-specific knowledge embedded in the script rather than in a provider abstraction layer.
How BakeX Separates Compliance Intent from Provider Details
Multi-cloud OS hardening works when the compliance intent and the provider details are kept strictly separate.
HardeningBlueprint YAML
(compliance intent — portable)
│
▼
BakeX Provider Layer
┌─────────────────────────────────────────────┐
│ AWS │ GCP │ Azure │
│ /dev/xvd* │ /dev/sda* │ /dev/sda* │
│ IMDS v2 │ GCP IMDS │ Azure IMDS │
│ cloud-init │ cloud-init │ waagent │
│ order A │ order B │ order C │
└─────────────────────────────────────────────┘
│
▼
Ansible-Lockdown + Provider-Aware Configuration
│
▼
OpenSCAP Scan
│
▼
Golden Image (AMI / GCP Image / Azure Image)
The blueprint author declares what should be true about the OS. BakeX’s provider layer handles how that’s achieved on each cloud.
The disk naming, cloud-init sequencing, metadata endpoint configuration, and provider-specific package repositories are all abstracted into the provider layer. They never appear in the blueprint file.
The Same Blueprint Across Six Providers
Here is the part people expect to be a flag, and isn’t. There is no --provider switch. The
provider is a field inside the blueprint, so you keep one file per target:
$ ls blueprints/ubuntu/22.04/
cis-l1-aws.yaml cis-l1-digitalocean.yaml cis-l1-linode.yaml
cis-l1-azure.yaml cis-l1-gcp.yaml cis-l1-proxmox.yaml
# Validate all six at once — offline, no cloud API calls
$ bakex validate blueprints/ubuntu/22.04/*.yaml
OK blueprints/ubuntu/22.04/cis-l1-aws.yaml (HardeningBlueprint 'ubuntu22-cis-l1-aws')
...
6/6 blueprint(s) valid.
# Build one
$ bakex build blueprints/ubuntu/22.04/cis-l1-gcp.yaml
Building 'ubuntu22-cis-l1-gcp' (gcp) → job 7f3c9e82-…
That design choice looks like more files, and it is. What you get for it is that a blueprint
is completely self-describing: the file names its own cloud and its own base image, so it
builds the same way on your laptop, in CI, and on a colleague’s machine with no flags to
forget and no environment to match.
The claim worth testing: how much actually differs between those six files?
I parsed all six and compared every section except metadata and target:
compliance identical across all 6
controls identical across all 6 (same rules, same enable state)
filesystem identical across all 6
users identical across all 6
system identical across all 6
Only the target block changes, and it changes in exactly the way you’d expect:
| Provider | instance_type |
base_image |
|---|---|---|
| aws | t3.medium |
ami-0c7217cdde317cfec |
| gcp | e2-medium |
projects/ubuntu-os-cloud/global/images/family/ubuntu-2204-lts |
| azure | Standard_B2s |
Canonical:0001-com-ubuntu-server-jammy:22_04-lts-gen2:latest |
| digitalocean | s-2vcpu-4gb |
ubuntu-22-04-x64 |
| linode | g6-standard-2 |
linode/ubuntu22.04 |
| proxmox | 2c-4g |
9000 (VE template VMID) |
Six wildly different ways of naming “Ubuntu 22.04 LTS” and sizing a 2 vCPU / 4 GB box. That
is the entire provider-specific surface. The CIS benchmark, the profile, the datastream, the
mount options, the locked root account, the control overrides and their justifications — all
byte-identical.
One honest caveat, because I checked rather than assumed: the six files are semantically
identical but not textually so. The justification string on one disabled SELinux rule is
worded slightly differently between files — same rule, same enabled: false, same meaning,
different prose. It’s a cosmetic inconsistency in the shipped library, not a behavioural one,
and it’s the kind of thing you only find by diffing rather than trusting the header comment.
If you change the compliance posture — add a control override, tighten a mount option — you
change it identically in six files and rebuild. A reviewer sees six identical hunks in the
diff. A sixth hunk that looks different is a bug, and it’s visible in code review rather than
three months later in a pentest.
What the Provider Layer Handles
The provider layer is where the cloud-specific knowledge lives, so the blueprint author doesn’t have to carry it:
Disk naming:
| Provider | OS disk | Ephemeral | Data |
|---|---|---|---|
| AWS | /dev/xvda |
/dev/xvdb |
/dev/xvdc+ |
| GCP | /dev/sda |
— | /dev/sdb+ |
| Azure | /dev/sda |
/dev/sdb (temp disk) |
/dev/sdc+ |
| DigitalOcean | /dev/vda |
— | /dev/vdb+ |
The CIS controls for separate /tmp and /home partitions reference disk paths that differ across these providers. The provider layer translates the blueprint’s filesystem.tmp declaration into the correct fstab entries for the target cloud.
Cloud-init ordering:
Different providers initialize services in different orders. On AWS, the network is available before cloud-init runs most tasks. On GCP, some network configuration happens after cloud-init starts. On Azure, the waagent handles some configuration that cloud-init handles elsewhere.
The provider layer sequences the hardening steps to run in the correct order for each provider — specifically, it waits for network availability before applying network-level hardening, and ensures the package manager is configured before running Ansible roles that require package installation.
Metadata endpoint configuration:
CIS controls include restrictions on access to the instance metadata service (IMDSv2 enforcement on AWS, equivalent controls on GCP/Azure). The provider layer applies the correct restriction for each cloud — the blueprint just declares compliance: benchmark: cis-l1.
Building Every Provider
There is no built-in fan-out flag, and honestly none is needed — the CLI is exit-code shaped,
so the shell already does this well:
# Validate everything first; stop before spending money if anything is wrong
bakex validate blueprints/ubuntu/22.04/*.yaml || exit 1
# Then build each target
for bp in blueprints/ubuntu/22.04/cis-l1-*.yaml; do
bakex build "$bp" --json > "builds/$(basename "$bp" .yaml).json" &
done
wait
--json emits the job record — id, profile name, provider, status, artifact ID, error — which
is what you want when six builds are writing to six files at once. Every build either lands a
complete status with an artifact ID, or a failed status with the reason. Nothing produces
a half-hardened image.
The validate-then-build ordering matters more than it looks. Validation is offline and takes
milliseconds; a build takes 15–25 minutes and costs money. Catching a malformed blueprint or an
unsupported OS/provider pair in the first step means you never launch the instance.
Blueprint Versioning and Drift
Version-controlling the blueprint file solves a problem multi-cloud environments hit
consistently: knowing what your OS security posture was six months ago. The blueprint is the
answer — it’s a file, in git, with a commit history and a reviewer’s name on every change.
Re-scanning a running instance against the posture that built it is a separate job, and it
does not live in the CLI. bakex validate and bakex build are the two CLI verbs; scanning
and drift comparison are in the web UI and the HTTP API, where the scan results have somewhere
to live. EP04 covers that surface in detail — the A–F grade, the SARIF export, and comparing a
current scan against a stored baseline.
The useful discipline in the meantime is unglamorous: rebuild from the blueprint rather than
patching running instances. An instance that drifted is a symptom; the blueprint is the cure,
and re-baking is cheaper than reconciling.
Production Gotchas
Provider-specific CIS controls exist. CIS AWS Foundations Benchmark and CIS GCP Benchmark include cloud-specific controls (VPC flow logs, CloudTrail, etc.) that are separate from the OS-level CIS controls. The blueprint handles OS-level controls. Cloud-level controls (IAM, logging, network configuration) belong in your cloud security posture management tooling.
Build costs vary by provider. On AWS, the build instance is a t3.medium for 15–20 minutes (~$0.02). On GCP and Azure, equivalent pricing applies. For multi-provider builds, run them in regions close to your primary workloads to minimize image transfer time.
Proxmox is a template VMID, not an image name. Every cloud provider names its base image with a string; Proxmox names it with a number — the VE template’s VMID (9000 in the shipped blueprint). The provider talks to the Proxmox API remotely via proxmoxer, so no agent on the host is required, but it does need host credentials and it discovers the built VM’s IP through the QEMU guest agent inside the VM. If the guest agent isn’t installed in your template, the build will provision and then hang waiting for an address.
KVM and Proxmox are deliberately different providers. They look interchangeable and aren’t: a KVM target names a downloadable cloud image, a Proxmox target names a VE template that already exists on your host. Don’t assume a blueprint written for one works on the other.
GCP image sharing across projects requires explicit IAM. GCP machine images aren’t automatically available to other projects in the organization. BakeX builds the image; sharing it is a GCP IAM operation you configure at the project or organization level — there’s no BakeX command that grants cross-project access for you.
Key Takeaways
- Multi-cloud OS hardening with separate scripts per provider creates inevitable drift; a provider-abstracted blueprint eliminates it
- BakeX ships working blueprints for AWS, GCP, Azure, DigitalOcean, Linode, and Proxmox — one file per provider, with
targetas the only section that differs - The provider is a field in the blueprint, not a CLI flag: every file is self-describing and builds identically in CI, locally, or on a teammate’s machine
- Fan-out is a shell loop over exit codes, not a framework feature — validate all six offline first, then build in parallel with
--json - Blueprint version control is the single source of truth for OS security posture history — and a compliance change that isn’t identical across all six providers shows up as an odd hunk in code review
What’s Next
Six providers, one compliance posture, and a diff that proves it. EP03 showed that the multi-cloud drift problem disappears when provider details are confined to a single block of the blueprint.
What neither EP02 nor EP03 answered is the auditor’s question: how do you know the image is actually compliant? “We ran CIS L1” is not an answer. “Grade A, 98/100 controls, SARIF export attached” is.
EP04 covers automated OpenSCAP compliance: the post-build scan in detail — how the A-F grade is calculated, what controls block an A grade, how SARIF exports work, and how drift detection catches what changed after deployment.
Next: automated OpenSCAP compliance — CIS benchmark grading before deployment
Get EP04 in your inbox when it publishes → linuxcent.com/subscribe
