Infrastructure as Code

Crossplane vs Terraform: Composition vs HCL for Infrastructure as Code

by
Reading Time: 5 minutes

Kubernetes Ecosystem: From User to Contributor, Episode 7
← EP06: Crossplane · EP07: Crossplane vs Terraform · EP08: Karpenter →

11 min read

TL;DR

  • Crossplane vs Terraform is fundamentally a continuous-reconciliation model against a plan/apply model — not just two different syntaxes for the same idea
  • Crossplane needs a live Kubernetes cluster to run at all; Terraform needs nothing but a state file and network access to the providers it’s calling
  • Terraform’s provider registry is a decade deep and covers services Crossplane’s younger ecosystem hasn’t reached yet — SaaS tools, monitoring platforms, and services with no cloud-infrastructure angle at all
  • Crossplane’s Compositions give app teams a genuinely self-service, in-cluster API; Terraform’s modules give infra teams reusable code, but consuming a module still means running Terraform yourself
  • Recommendation: many real platform teams use both — Terraform (or CAPI, EP05) to bootstrap the cluster and its surrounding VPC/networking, then Crossplane running inside that cluster for the self-service, app-team-facing layer
  • Contribution opportunity: Crossplane’s provider coverage gap against Terraform’s registry is real, specific, and a legitimate place to build a brand-new provider

The Big Picture

TERRAFORM                                   CROSSPLANE
──────────                                   ──────────
terraform plan                               kubectl apply -f resource.yaml
  │  (shows what WOULD change)                    │
  ▼                                                ▼
terraform apply                              Crossplane controller reconciles
  │  (changes happen once, here)                   │  (continuously, forever,
  ▼                                                 │   not just at apply time)
State file (local or remote backend)               ▼
tracks what Terraform created                 Kubernetes etcd IS the state —
                                                the CR's status field tracks
No live cluster or control                     sync state
plane required to run this
                                               Requires a running Kubernetes
                                               cluster as the control plane

Crossplane vs Terraform is best understood through that control-flow difference first, before comparing any specific feature: Terraform changes things at discrete moments you trigger; Crossplane’s controllers are always watching, always correcting drift, for as long as the cluster runs.

The Fundamental Model Difference: Continuous Reconciliation vs Plan/Apply

Terraform’s model gives you an explicit review step — terraform plan shows exactly what will change before anything does, and nothing changes again until you run apply a second time. Crossplane’s model (covered in EP06) has no equivalent pause: once a Managed Resource or Composition claim exists, Crossplane’s controllers reconcile it toward the desired state continuously, including reverting manual out-of-band changes automatically.

Neither is objectively better — they’re suited to different operating assumptions. Terraform’s model fits teams who want a deliberate, reviewed change process. Crossplane’s fits teams who want infrastructure to behave like every other Kubernetes-native resource: self-healing, always converging, no separate “did anyone remember to re-apply” step.

State Management: etcd + CRDs vs Terraform State Files

# Terraform: state lives in a file (local or remote — S3, Terraform Cloud, etc.)
$ terraform state list
aws_s3_bucket.uploads
aws_db_instance.main

# Crossplane: "state" is just the live cluster's etcd — the CR's own status
$ kubectl get bucket uploads -o jsonpath='{.status.conditions}'
[{"type":"Ready","status":"True"},{"type":"Synced","status":"True"}]

Terraform’s state file is a single point of coordination that has to be locked correctly for concurrent runs to be safe — a well-understood but real operational concern (remote state backends, state locking, occasional manual state surgery after a botched apply). Crossplane sidesteps a separate state file entirely, but that means the health of your Kubernetes cluster’s etcd is the health of your infrastructure’s state — a different, not smaller, operational responsibility.

Composition vs Modules: Reusable Infrastructure Patterns Compared

Terraform modules are reusable code that whoever runs Terraform includes in their own configuration — genuinely reusable, but still something each consumer runs themselves. Crossplane Compositions (EP06) are reusable inside the cluster — an app team doesn’t run anything, they just create a custom resource the platform team already defined, and Crossplane’s controllers do the rest without the app team ever touching Terraform or Crossplane’s own tooling directly.

That’s the real practical difference for organizational self-service: Compositions remove the “app team needs to know how to run our IaC tool” step entirely. Modules still require the consumer to run Terraform, even if they didn’t write the module.

Ecosystem Maturity: Terraform’s Decade-Deep Provider Registry vs Crossplane’s Younger One

Terraform’s provider registry covers not just the major clouds but a long tail of SaaS platforms, monitoring tools, DNS providers, and internal enterprise systems that have no “cloud infrastructure” angle at all — a decade of community and vendor-contributed providers. Crossplane’s provider ecosystem, while actively growing and covering the major clouds thoroughly, has real, documented gaps once you look past core compute/storage/networking/database resources into more specialized or less common services.

The Recommendation: Which One, and When to Use Both

If your platform team is Kubernetes-native and wants to offer app teams a true self-service infrastructure API without teaching them a separate IaC tool: Crossplane. That’s the specific problem its Composition model solves better than anything Terraform offers.

If you need broad provider coverage beyond core cloud infrastructure, or you don’t want infrastructure lifecycle tied to a live Kubernetes control plane’s uptime: Terraform. Its registry depth and its independence from any running cluster are real advantages Crossplane doesn’t currently match.

The honest answer for a lot of real platform teams is both, at different layers. Use Terraform (or Cluster API, EP05) to bootstrap the Kubernetes cluster itself and its surrounding cloud networking — the layer that has to exist before Crossplane can run at all — then run Crossplane inside that cluster for the ongoing, self-service, app-team-facing infrastructure requests. This isn’t a compromise; it’s matching each tool to the layer it’s actually better suited for.

⚠ Production Gotchas

Don’t manage the same cloud resource with both Terraform and Crossplane simultaneously. Both tools will detect the other’s changes as drift and fight to revert them — pick one owner per resource, even when both tools are in use across your stack at different layers.

Terraform’s plan/apply gives you a review window Crossplane doesn’t — build your own review gate if you need one with Crossplane (a PR-based GitOps flow with required approval before a claim manifest merges is the common substitute).

Crossplane’s continuous reconciliation means a broken provider or a cloud API outage shows up as a stuck Synced: False condition, not a failed one-time command — monitoring needs to watch for stuck conditions over time, not just command exit codes the way Terraform CI pipelines typically do.

Quick Reference

Terraform Crossplane
Change model Plan → Apply (explicit) Continuous reconciliation
Requires a live cluster No Yes
State State file (local/remote) Kubernetes etcd + CR status
Reusable patterns Modules (you still run them) Compositions (app team just creates a claim)
Provider breadth Very broad, decade-deep Growing, strong on core cloud, gaps elsewhere
Manual drift Detected at next plan, not auto-reverted Auto-reverted on next reconcile

Contribution Opportunity: Building a Missing Crossplane Provider

The limitation: For a meaningful number of services Terraform has supported for years — smaller SaaS platforms, specialized monitoring tools, niche infrastructure services — there’s no Crossplane provider equivalent yet. Anyone wanting to manage that service the Crossplane way currently can’t, full stop.

Why it’s hard to fix: Building a new provider means implementing a real API client, defining CRD schemas that faithfully map the service’s actual parameters, and maintaining it as that service’s API evolves — real, ongoing engineering commitment, not a one-time script. That’s exactly why the ecosystem’s provider list still trails Terraform’s, despite Crossplane’s core reconciliation engine being mature: the core is one thing to maintain, but each provider is its own ongoing surface area.

What a contribution-shaped fix looks like: Crossplane’s provider-template repository exists specifically to make starting a new provider tractable — it scaffolds the boilerplate (code generation, CRD structure, controller wiring) so a new provider author focuses on the actual API mapping, not plumbing. Picking one service you already use via Terraform that has no Crossplane equivalent, and building a minimal provider covering just the 2-3 resource types you actually need, is a real, bounded, achievable contribution — and one the Crossplane community actively wants, given how directly it grows the ecosystem.

Key Takeaways

  • Crossplane’s continuous reconciliation and Terraform’s plan/apply are different operating models, not different syntaxes for the same thing — pick based on which review/change process fits your team
  • Crossplane requires a live cluster to function at all; Terraform doesn’t, which matters for bootstrapping order
  • Compositions remove the “app team has to run our IaC tool” step that Terraform modules still require
  • Terraform’s provider registry breadth remains a real advantage for anything beyond core cloud infrastructure
  • Many real platform teams run both at different layers — Terraform/CAPI to bootstrap the cluster, Crossplane inside it for self-service — and that’s a legitimate architecture, not indecision

What’s Next

Everything so far in this series has been about provisioning clusters and the infrastructure around them. EP08 shifts to what happens inside an already-running cluster when pods can’t be scheduled: Karpenter’s just-in-time node provisioning, and why it replaced the node-group model most teams started with.

Next: EP08 — Karpenter: Just-in-Time Node Provisioning for Kubernetes

Get EP08 in your inbox when it publishes → linuxcent.com/subscribe

Crossplane: Kubernetes as the Universal Control Plane

by
Reading Time: 5 minutes

Kubernetes Ecosystem: From User to Contributor, Episode 6
← EP05: Cluster API · EP06: Crossplane · EP07: Crossplane vs Terraform →

12 min read

TL;DR

  • Crossplane extends the exact reconciliation pattern EP05 covered for cluster infrastructure to any cloud resource — an S3 bucket, an RDS instance, a DNS record all become Kubernetes CRDs, continuously reconciled
  • Managed Resources represent one real cloud resource each; Compositions bundle several Managed Resources behind a single, simpler custom API a platform team defines and app teams consume
  • Composition Functions are Crossplane’s newer, more flexible replacement for its older YAML-based patch-and-transform templating — real code (Go, Python, or others) instead of declarative patches
  • Crossplane continuously reconciles like any Kubernetes controller — a manual change to a cloud resource outside Crossplane gets reverted on the next reconcile loop, which is a real surprise for teams used to Terraform’s plan/apply model
  • Provider CRD counts can bloat a cluster’s etcd significantly — this drove the ecosystem’s move toward smaller, split “provider families” instead of one monolithic provider per cloud
  • Contribution opportunity: several providers still haven’t migrated to the family-split pattern — a real, currently-tracked, achievable upstream contribution

The Big Picture

App team writes:                    Platform team defined this Composition
                                     once, behind the scenes:
apiVersion: platform.example.com/v1
kind: Database                       XRD "Database" ─── composes ───┐
metadata:                                                             │
  name: my-app-db                                                     ▼
spec:                                                        ┌────────────────┐
  size: small                                                │ RDSInstance    │
                                                                │ SecurityGroup  │
   │                                                            │ ParameterGroup │
   │ app team never sees                                        └────────────────┘
   │ or touches these three                                     each a real Managed
   ▼                                                            Resource, a real
Crossplane reconciles all three,                                cloud API call
continuously, forever

Crossplane’s pitch as a universal control plane is literal: instead of app teams filing tickets or writing their own Terraform for a database, they request a Database — a custom API the platform team designed — and Crossplane’s controllers translate that into the actual RDS instance, security group, and parameter group underneath, then keep reconciling all three toward the declared state indefinitely.

Managed Resources: Cloud Infrastructure as Kubernetes CRDs

$ kubectl apply -f - <<EOF
apiVersion: s3.aws.upbound.io/v1beta1
kind: Bucket
metadata:
  name: app-uploads-prod
spec:
  forProvider:
    region: us-east-1
  providerConfigRef:
    name: aws-prod
EOF

$ kubectl get bucket app-uploads-prod
NAME               READY   SYNCED   AGE
app-uploads-prod   True    True     30s
#                  ^^^^    ^^^^^^ — READY: resource exists and is healthy
#                          SYNCED: Crossplane's last reconcile succeeded

Every field under forProvider maps directly to that cloud API’s actual parameters — this is a thin, honest translation layer, not an abstraction hiding what’s actually being created. READY/SYNCED becoming True means an actual S3 bucket now exists in that AWS account, exactly as declared.

Compositions and XRDs: Building Your Own Abstract Platform API

This is Crossplane’s real differentiator over just using individual Managed Resources directly:

# The platform team defines the abstract API app teams will see
apiVersion: apiextensions.crossplane.io/v1
kind: CompositeResourceDefinition
metadata:
  name: xdatabases.platform.example.com
spec:
  group: platform.example.com
  names:
    kind: XDatabase
    plural: xdatabases
  claimNames:
    kind: Database        # ← this is what app teams actually create
    plural: databases
  versions:
  - name: v1
    schema:
      openAPIV3Schema:
        properties:
          spec:
            properties:
              size: {type: string, enum: ["small", "medium", "large"]}

App teams interact only with the simple Database claim shown in the Big Picture diagram above. The Composition resource (not shown here for brevity) is what actually maps size: small to a specific RDS instance class, storage size, and backup configuration — the platform team’s opinions, encoded once, consumed self-service by every app team afterward.

Composition Functions: Crossplane’s Newer, More Flexible Approach

Older Crossplane Compositions used a YAML-based “patch and transform” templating language to map the abstract API’s fields onto Managed Resource fields — functional, but limited for anything beyond straightforward field mapping. Composition Functions replace that with actual executable code:

$ crossplane beta render xr.yaml composition.yaml functions.yaml
---
apiVersion: rds.aws.upbound.io/v1alpha1
kind: Instance
metadata:
  name: my-app-db-instance
spec:
  forProvider:
    instanceClass: db.t3.micro   # ← computed by real Go logic based on
                                  #   spec.size, not a static YAML patch
    engine: postgres

Composition Functions run as small, packaged pieces of logic (often distributed as OCI images) that Crossplane’s engine invokes during reconciliation — giving platform teams real conditionals, loops, and validation instead of the older templating language’s more limited patch syntax.

Providers and the Provider Ecosystem

Each cloud’s resources are supplied by a separate provider — provider-aws, provider-gcp, provider-azure, and increasingly split into smaller provider families (provider-aws-s3, provider-aws-rds, etc.) rather than one enormous provider per cloud:

$ kubectl get providers
NAME                   INSTALLED   HEALTHY   AGE
provider-aws-s3        True        True      10d
provider-aws-rds       True        True      10d
#         ^^^^^^ — installing only the families you actually use, instead
#                  of one monolithic provider-aws with every AWS service's
#                  CRDs installed regardless of whether you use them

The family split exists specifically because a single monolithic cloud provider can register thousands of CRDs — a real, measurable strain on a cluster’s etcd and API server that the ecosystem is still in the process of migrating away from.

⚠ Production Gotchas

Crossplane reconciles continuously — a manual change to a cloud resource outside Crossplane gets reverted on the next loop. Teams coming from Terraform’s plan/apply model, where nothing changes until you explicitly run apply again, are frequently surprised the first time a manual “quick fix” in the AWS console gets silently undone minutes later.

Monolithic providers can register thousands of CRDs, and that has a real, measurable etcd and API-server cost. If you’re on an older, non-family provider version and seeing API server memory pressure, check CRD count before assuming it’s an unrelated capacity issue.

Deleting a Composition’s underlying claim doesn’t always tear down cleanly if finalizers on the Managed Resources are stuck — a Managed Resource that failed to delete cleanly from the cloud side (a non-empty S3 bucket, for instance) will block the whole claim’s deletion until that’s resolved manually.

Quick Reference

kubectl get managed                        # every Managed Resource, all providers
kubectl get compositeresourcedefinitions   # XRDs — the abstract APIs defined
kubectl get compositions                   # the mapping logic behind each XRD
kubectl get providers                       # installed providers + health
crossplane beta render <xr> <comp> <fns>    # render a Composition locally, no cluster needed
kubectl describe <managed-resource-kind> <name>   # sync status + underlying cloud errors

Contribution Opportunity: Migrating Providers to the Family Pattern

The limitation: Not every Crossplane provider has migrated from the older, monolithic-per-cloud model to the smaller “provider family” pattern that registers only the CRDs for services actually in use. Clusters running an un-migrated provider carry the etcd and API-server overhead of thousands of unused CRDs, and this is a known, actively-discussed problem in the Crossplane community — not a hypothetical one.

Why it’s hard to fix: Splitting a monolithic provider into families isn’t a mechanical find-and-replace — it means restructuring code generation, versioning, and release processes for every resource type the provider covers, while keeping a migration path that doesn’t break existing users who depend on the old provider’s CRDs. It’s real, unglamorous engineering work that has to happen provider-by-provider, cloud-by-cloud, and each provider’s maintainer bandwidth varies.

What a contribution-shaped fix looks like: The Crossplane and Upbound-maintained provider repositories publicly track which providers still need family-splitting — this is documented, wanted work, not a gap you’d have to go discover yourself. A concrete starting contribution: pick one still-monolithic provider (checking the project’s own tracking issues for an unclaimed one), and work through the documented family-split process the already-migrated providers (like provider-aws) used as a reference implementation. This is real upstream OSS work with an existing template to follow, not a design problem you have to solve from scratch.

Key Takeaways

  • Crossplane’s Managed Resources make individual cloud resources real Kubernetes CRDs, continuously reconciled rather than applied once
  • Compositions and XRDs are the actual value proposition: platform teams define a simple, opinionated API once; app teams self-serve against it without needing to know what’s underneath
  • Composition Functions replace older YAML patch-and-transform templating with real executable logic — a genuinely evolving, more flexible part of the project
  • Continuous reconciliation means manual out-of-band changes get reverted — a real behavioral difference from Terraform’s plan/apply model, not just a implementation detail
  • The provider family migration is documented, wanted, achievable contribution work — not a gap you’d need to discover on your own

What’s Next

Crossplane’s composition model and Terraform’s HCL module model solve the same underlying problem — reusable, parameterized infrastructure definitions — from genuinely different architectural starting points. EP07 puts them side by side and gives a clear recommendation for which fits which team.

Next: EP07 — Crossplane vs Terraform: Composition vs HCL for Infrastructure as Code

Get EP07 in your inbox when it publishes → linuxcent.com/subscribe

Cloud-Native Hardening: Securing the AWS Identity Perimeter

by
Reading Time: 6 minutes

Zero to Hero: Cybersecurity Architecture Masterclass, Module 3
← Module 2: Proactive Design · Module 3: Cloud-Native Hardening · Module 4: Resilience & Survival →

12 min read

TL;DR

  • Cloud native infrastructure hardening starts from a different assumption than on-prem hardening: there is no network perimeter, only an identity perimeter — every AWS API call is the boundary
  • IMDSv1 (the EC2 metadata service without a token) is the single highest-leverage cloud-native hardening fix available — it turned an SSRF bug into the Capital One breach
  • IAM policy design is architecture, not IT administration: least privilege, permission boundaries, and SCPs compose into the actual perimeter
  • Infrastructure-as-code scanning (checkov, tfsec) catches identity-perimeter mistakes in a pull request instead of in an incident
  • aws iam simulate-principal-policy answers “can this role actually do that?” definitively, without waiting to find out in production
  • Recommendation: treat IMDSv2 enforcement and IAM least-privilege review as pipeline gates, not periodic audits — the same “build constraint, not process step” principle from the OS Hardening series

The Big Picture: The Perimeter Moved to the API Call

ON-PREM MODEL                          CLOUD-NATIVE MODEL
──────────────                          ──────────────────
Firewall at network edge                No fixed network edge
        │                                        │
Trusted internal subnet                 Every API call carries its
        │                                 own identity + policy
Server assumed safe if                          │
inside the firewall                     IAM evaluates: who is this,
                                          what can they do, right now
                                                 │
                                          Perimeter = the IAM policy
                                          attached to the caller

Cloud-native infrastructure hardening means accepting that the network no longer defines what’s trusted — the AWS identity perimeter, enforced entirely through IAM policy evaluation on every single API call, is the only perimeter that actually exists. Module 1 called this the shift from network-centric to identity-centric trust; this module makes it concrete with the two failures that actually break it in production: a leaky metadata service and an over-permissioned role.

The Breach That Made IMDSv2 Mandatory

In 2019, a misconfigured WAF in front of a bank’s application allowed a Server-Side Request Forgery (SSRF) — an attacker convinced the application server to make an HTTP request to http://169.254.169.254, the EC2 instance metadata endpoint. IMDSv1 answered with no authentication required at all: temporary IAM credentials for the role attached to that instance, handed to anyone who could make the server issue that one request.

Those credentials had read access to S3. The attacker used them to exfiltrate over 100 million customer records. This is the Capital One breach — covered in full in the Purple Team series — and it is the single clearest illustration in cloud history of why “the perimeter is the identity, not the network” isn’t a slogan — it’s a description of exactly where that breach actually happened. The WAF misconfiguration was the entry point. The metadata service handing out credentials with zero verification was the architectural failure that turned an SSRF bug into a 100-million-record breach.

IMDSv2 closes this specific gap by requiring a session token, fetched via a PUT request, before any metadata GET request is honored — and that PUT request cannot be replayed through a typical SSRF, because SSRF vulnerabilities almost always only allow GET-style requests to be forged. This single setting is the highest-leverage cloud-native hardening control available, and it should be enforced at the account level, not left as an opt-in per instance:

# Check whether IMDSv2 is enforced (HttpTokens: required) on an instance
$ aws ec2 describe-instances --instance-ids i-0abc123 \
    --query 'Reservations[].Instances[].MetadataOptions'
{
    "HttpTokens": "required",
    "HttpPutResponseHopLimit": 1,
    "HttpEndpoint": "enabled"
}
# "required" = IMDSv2 only. "optional" = IMDSv1 still works — the gap.

# Enforce it account-wide for all new instances
$ aws ec2 modify-instance-metadata-defaults \
    --http-tokens required --http-put-response-hop-limit 1

IAM Policy Design Is Architecture

If the metadata service is one way the identity perimeter leaks, an over-permissioned IAM policy is the other — and it’s far more common, because it doesn’t require a bug at all. It only requires a policy written with "Resource": "*" because scoping it felt like it would slow down a deploy.

Least privilege means a role can do exactly what its function requires and nothing else — not “read-only across the account,” but “read this specific S3 prefix, write to this specific queue.”

Permission boundaries cap what a role can ever be granted, even by someone with iam:CreatePolicy access — a safety rail against exactly the kind of iam:PassRole privilege escalation covered in the Cloud IAM series, not just against the policy as originally written.

Service Control Policies (SCPs) apply at the AWS Organization level, capping what any role in an account can do regardless of how permissive that account’s own IAM policies are — the outermost layer of the identity perimeter, and the one that survives a single account being compromised.

{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Action": ["s3:GetObject"],
    "Resource": "arn:aws:s3:::billing-invoices/tenant-4471/*"
  }]
}

That policy can only ever read one tenant’s invoice prefix. Compare it to "Resource": "arn:aws:s3:::billing-invoices/*" — functionally identical for the one use case the developer was testing, and catastrophically different the day this role’s credentials leak.

Quick Check: Can This Role Actually Do That?

Don’t wait to find out in production. aws iam simulate-principal-policy evaluates a specific action against a role’s actual attached and inline policies — including SCPs and permission boundaries — and gives you a definitive allow/deny before anything runs:

$ aws iam simulate-principal-policy \
    --policy-source-arn arn:aws:iam::123456789012:role/billing-api-role \
    --action-names s3:GetObject \
    --resource-arns arn:aws:s3:::billing-invoices/tenant-9982/*

{
  "EvaluationResults": [{
    "EvalActionName": "s3:GetObject",
    "EvalResourceName": "arn:aws:s3:::billing-invoices/tenant-9982/*",
    "EvalDecision": "explicitDeny",     # ← the answer you needed before deploying
    "MatchedStatements": [...]
  }]
}

explicitDeny here means some policy statement — the role’s own policy, a permission boundary, or an SCP — explicitly blocks the action, and that takes precedence over any Allow anywhere else in the policy chain (Module 1’s deny-by-default evaluation model, in practice). Run this simulation as part of code review for any new IAM policy, not after the role is already attached to a running service.

Catching This Before It Ships: Cloud-Native Hardening via IaC Scanning

Manually reviewing every Terraform IAM policy in every pull request doesn’t scale past a handful of engineers. checkov and tfsec scan infrastructure-as-code for exactly the patterns above — wildcard resources, IMDSv1 left enabled, public S3 buckets — as a CI step, before terraform apply ever runs:

$ checkov -d ./terraform --check CKV_AWS_79,CKV_AWS_8

Check: CKV_AWS_79: "Ensure Instance Metadata Service Version 1 is not enabled"
    FAILED for resource: aws_instance.billing_api
    File: main.tf:14-22

Check: CKV_AWS_8: "Ensure IAM policies do not allow full administrative privileges"
    FAILED for resource: aws_iam_role_policy.billing_api_policy
    File: iam.tf:8-15
        Resource: "*"

A failed checkov check blocking a pull request is the identity-perimeter equivalent of Stratum’s pipeline gate refusing to snapshot an unhardened image — the unsafe configuration never reaches an account where it can be exploited, because the check runs before merge, not after an audit finds it months later.

Production Gotchas

IMDSv2 enforcement can break old SDKs and tools silently. Some older AWS SDK versions and third-party agents assume IMDSv1 and simply fail to fetch credentials once HttpTokens: required is set — test in staging before enforcing account-wide.

iam simulate-principal-policy doesn’t account for resource-based policies on the target. It evaluates the principal’s policies correctly, but if the target (an S3 bucket, a KMS key) has its own resource policy denying access, you need simulate-custom-policy with both policies supplied to get the full picture.

SCPs fail closed in a way that’s easy to misdiagnose. An SCP deny produces the same AccessDenied error as a missing IAM permission — check the SCP layer explicitly before assuming the role’s own policy is the problem, or you’ll spend an hour widening a policy that was never the actual blocker.

checkov/tfsec false positives erode trust in the gate fast. Suppress specific, documented exceptions inline (#checkov:skip=CKV_AWS_79:reason) rather than disabling the check account-wide the first time it blocks something legitimate.

Framework Alignment

Framework Control / ID Architectural Mapping
NIST CSF 2.0 PR.AA-05 Access permissions are managed, incorporating least privilege and separation of duties.
NIST SP 800-207 Zero Trust The identity perimeter, enforced per-API-call, is the direct implementation of continuous verification.
ISO 27001:2022 8.2 Privileged access rights are restricted and managed.
SOC 2 CC6.3 The entity authorizes, modifies, or removes access based on roles and responsibilities.

Key Takeaways

  • The identity perimeter, not the network, is what cloud-native hardening actually secures — every IAM policy evaluation is a perimeter check
  • IMDSv2 enforcement is the single highest-leverage fix available and should be an account-wide default, not an opt-in
  • Least privilege, permission boundaries, and SCPs are three layers of the same perimeter — design all three deliberately, don’t rely on one
  • aws iam simulate-principal-policy gives a definitive answer before deployment instead of an incident after
  • IaC scanning turns identity-perimeter mistakes into blocked pull requests instead of production findings

What’s Next

Module 3 hardened the identity perimeter against external and lateral threats. Module 4 asks what happens after a perimeter fails anyway — specifically, how immutable, WORM-locked data architecture makes ransomware and mass-deletion attacks survivable even when an attacker has already gotten past every control this module covers.

Next: Module 4: Resilience & Survival — Immutable Data Architecture and Surviving Ransomware via WORM

Get the full masterclass in your inbox → linuxcent.com/subscribe

BakeX — OS Hardening as a Platform

by
Reading Time: 8 minutes

OS Hardening as Code, Episode 6
Cloud AMI Security Risks · Linux Hardening as Code · Multi-Cloud OS Hardening · Automated OpenSCAP Compliance · CI/CD Compliance Gate · BakeX Platform**

Note: this series was written when the project was called Stratum. It was renamed to
BakeX at v0.6.0 — same project, same Apache 2.0 license, same team. The old
github.com/rrskris/Stratum URL redirects here, and pip install stratumoss is retired in
favour of pip install bakex. Current home:
github.com/invicton/bakex.

TL;DR

  • BakeX is open-source under Apache 2.0 — the engine, blueprint format, scanner, and Pipeline API are all in the repository
  • Self-hostable end to end: nothing is locked to a hosted service, and there is no paid tier gating the pipeline
  • Two real extension points: provider plugins (drop-in .py or a bakex.providers entry point) and blueprints (pure YAML, no code)
  • Architecture: Blueprint YAML → Engine → Provider Layer → Ansible-Lockdown → OpenSCAP → Golden Image → Pipeline API
  • The series taught the user-facing interface for five episodes; EP06 covers what’s underneath and how to build on it
  • Installation is git clone + docker compose up, or pip install bakex for the CLI and web app

The Series Arc, Inverted

EP01 showed that default cloud AMIs arrive pre-broken. By the time you reach EP06, that problem has a complete solution:

EP01 — The problem:
  Default AMI → Production → Security audit finds gaps
  (unknown OS baseline, unverified hardening, no evidence)

EP06 — The solution:
  HardeningBlueprint YAML
           ↓
    bakex validate          ← EP02 (blueprint as code)
    bakex build             ← EP02
      one file per provider ← EP03 (multi-cloud)
           ↓
    OpenSCAP scan           ← EP04 (compliance grading)
    Grade: A (score 94)
           ↓
    POST /api/pipeline/scan ← EP05 (CI/CD gate)
    passed: true
           ↓
    Production deployment
    (Grade A, SARIF attached, blueprint version-controlled)

For five episodes, you’ve used BakeX as a user. This episode covers what it looks like to run it yourself, extend it, and build on it.

I’ve spent years watching infrastructure teams solve the same OS hardening problem in slightly different ways. Custom scripts that drift. OpenSCAP runs that produce evidence no one reads. Compliance checklists completed by humans who have competing priorities.

The tools exist. ansible-lockdown applies CIS controls reliably. OpenSCAP verifies them accurately. The CI/CD systems can enforce anything you can express as a pass/fail. The gap isn’t the tooling — it’s the integration layer that ties them together into a reproducible, auditable pipeline.

BakeX is that integration layer, open-sourced.

The philosophy is the same as Terraform applied to OS security posture: declare the desired state in a version-controlled file, apply it reproducibly, and verify it automatically. The skip-at-2am problem disappears not because engineers are more careful, but because there’s no step to skip.

The Architecture

┌─────────────────────────────────────────────────────────┐
│                 HardeningBlueprint YAML                  │
│         (version-controlled, provider-agnostic)          │
└─────────────────────┬───────────────────────────────────┘
                      │
                      ▼
┌─────────────────────────────────────────────────────────┐
│                    BakeX Engine                          │
│                  (Apache 2.0, OSS)                       │
│  ┌─────────────┐  ┌──────────────┐  ┌────────────────┐  │
│  │  Blueprint  │  │   Provider   │  │    Scheduler   │  │
│  │   Parser    │  │    Layer     │  │  (parallel     │  │
│  │             │  │  AWS  GCP    │  │   multi-cloud  │  │
│  │  Validates  │  │  Azure DO    │  │   builds)      │  │
│  │  schema +   │  │  Linode      │  │                │  │
│  │  overrides  │  │  Proxmox     │  │                │  │
│  └─────────────┘  └──────────────┘  └────────────────┘  │
└─────────────────────┬───────────────────────────────────┘
                      │
           ┌──────────┴──────────┐
           ▼                     ▼
  ┌─────────────────┐   ┌─────────────────┐
  │ Ansible-Lockdown │   │  OpenSCAP       │
  │  Runner          │   │  Scanner        │
  │                  │   │                 │
  │  UBUNTU22-CIS    │   │  A-F grade      │
  │  RHEL8-STIG      │   │  SARIF export   │
  │  Custom roles    │   │  Drift detect   │
  └────────┬─────────┘   └────────┬────────┘
           │                      │
           └──────────┬───────────┘
                      │
                      ▼
         ┌─────────────────────────┐
         │   Golden Image          │
         │   (AMI / GCP / Azure)   │
         │   + compliance metadata │
         └────────────┬────────────┘
                      │
                      ▼
         ┌─────────────────────────┐
         │   Pipeline API          │
         │   (Apache 2.0, OSS)     │
         │                         │
         │  POST /api/pipeline/scan │
         │  ← CI/CD gate           │
         └─────────────────────────┘

Every component is open-source under Apache 2.0. The engine, provider layer, Ansible runner, OpenSCAP scanner, and Pipeline API are all in the repository. Nothing is locked to a hosted service.

Installation

Three ways in, depending on how much you want installed on the host.

Docker Compose — recommended, everything preinstalled:

git clone https://github.com/invicton/bakex.git
cd bakex
docker compose up

Open http://localhost:8001. Log in with any username and the admin token as the password —
it’s generated on first start and written to data/.admin_token. Set BAKEX_ADMIN_TOKEN and
BAKEX_SECRET_KEY in docker-compose.yml if you want logins that survive a rebuild.

Compose mounts ~/.aws, ~/.config/gcloud, and ~/.ssh read-only, plus persistent ./data,
./profiles, and ./plugins/providers. That last mount is the one to notice — it’s the
drop-in directory for provider plugins, which matters in the next section.

Published image:

docker run -p 8000:8000 rrskris/bakex:latest

PyPI — CLI and web app:

pip install "bakex[all-providers]"   # or pick extras: aws, gcp, azure, linode, digitalocean, proxmox
bakex serve --port 8000

One caveat worth stating plainly rather than letting you discover it: the extras install each
provider’s cloud SDK, and Ansible and OpenSCAP must be present on the host for real builds. If you
want the batteries-included path, use Compose. bakex validate works anywhere with no host
dependencies at all.

There is no Helm chart. BakeX is a build tool that talks to cloud APIs, not a cluster workload —
it does not need to live in Kubernetes to harden images for it.

The Three Extension Points

1. Blueprints — the extension point with no code in it

The highest-leverage way to extend BakeX isn’t Python. It’s a YAML file.

A blueprint is a complete, self-contained description of a hardened OS on a specific provider,
and the library ships 18 of them. Adding the nineteenth — say Ubuntu 24.04, or CIS Level 2 for a
distro that only has Level 1 today — requires no engine changes, because the benchmark, profile,
and datastream are just strings handed to oscap.

The full format is published as a JSON Schema (Draft 2020-12) at
docs/schema/hardening-blueprint.schema.json. Point your editor at it for autocomplete and
inline validation, or hand it to an LLM and let it draft the blueprint — the schema was published
partly so that agents could write these correctly without reading the source.

The loop is short enough to run in a coffee break:

$EDITOR blueprints/ubuntu/24.04/cis-l1-aws.yaml
bakex validate blueprints/ubuntu/24.04/cis-l1-aws.yaml

Validation is offline and checks more than syntax — it rejects OS/provider combinations the
catalog doesn’t support, so you find out that a distro isn’t available on your target cloud in
milliseconds rather than fifteen minutes into a paid build.

2. Provider Plugins

Adding a cloud means implementing four methods. That’s the whole interface
(bakex/plugins/base_provider.py):

# plugins/providers/my_cloud.py
from bakex.plugins.base_provider import BaseProvider, ProviderResult
from bakex.core.models import ComplianceProfile

class MyCloudProvider(BaseProvider):
    name = "my-cloud"          # matches target.provider in a blueprint

    def provision(self, profile: ComplianceProfile, **kwargs) -> str:
        """Launch a build instance; return its instance ID."""
        ...

    def run_ansible(self, instance_id: str, profile: ComplianceProfile) -> None:
        """Apply the Ansible-Lockdown hardening roles."""
        ...

    def snapshot(self, instance_id: str, profile: ComplianceProfile) -> ProviderResult:
        """Capture the golden image; return the artifact ID."""
        ...

    def teardown(self, instance_id: str) -> None:
        """Destroy the ephemeral build instance."""
        ...

There is no registration command. The loader (bakex/plugins/loader.py) is hybrid and finds
plugins two ways:

  1. Drop-in — put the .py file in plugins/providers/. That directory is a Compose volume
    mount, so a plugin dropped there is live in the container without rebuilding an image.
  2. Entry point — ship a pip-installable package declaring a bakex.providers entry point.
    This is how a third party distributes a provider without touching the BakeX repo.

Entry points load first and drop-ins load second, so a local file deliberately shadows an
installed package of the same name — which is exactly what you want when debugging someone
else’s provider.

The plugin becomes usable by writing provider: my-cloud in a blueprint’s target block. There
is no --provider flag to pass, because there is no --provider flag anywhere.

One honest note on the validation interaction from EP02: the compatibility check only objects
when both the OS and the provider are in the catalog. An unknown provider is assumed to be a
valid third-party plugin rather than an error — existence is the plugin registry’s call at build
time, compatibility is validation’s. That’s what makes shipping a provider out-of-tree possible
at all.

3. Pipeline Integrations

Beyond the curl-based gate from EP05, BakeX has a webhook system. Webhooks are registered through
the API rather than a config file, so they can be managed by the same automation that manages
everything else:

curl -X POST http://localhost:8001/api/webhooks \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://hooks.slack.com/services/…",
    "events": ["build.failed", "scan.grade_change"],
    "label": "platform-security alerts"
  }'

Five events fire: build.complete, build.failed, scan.complete, scan.failed, and
scan.grade_change. Registering an unknown event name is a 422 rather than a silent no-op —
a small thing that saves an afternoon.

scan.grade_change is the one to wire up first. A build that fails is loud on its own; a base
image that quietly slid from A to B between two scans is the signal nobody notices.

Deliveries are signed. The registration response returns a secret once, and every request
carries an X-BakeX-Signature: sha256=… HMAC so the receiver can verify the payload came from
your BakeX instance and not from anyone who guessed the endpoint URL.

There’s a defensive detail here that’s worth calling out, because it’s the kind of thing that
usually ships broken: webhook target URLs are checked against loopback, private, link-local, and
reserved ranges — including 169.254.169.254 — and they’re re-resolved at send time, not just
at registration. A user-configurable URL that the server will fetch is a textbook SSRF into the
cloud metadata endpoint, and on a tool that holds cloud credentials that would be a very bad day.

The Open-Core Model

BakeX sits alongside the tools that became infrastructure standards by being genuinely usable
before they were commercial:

Tool Model
Terraform / OpenTofu Core OSS, enterprise features in paid tier
Cilium / Isovalent Core OSS, enterprise support/features in paid tier
Vault / HCP Vault Core OSS, hosted/enterprise in paid tier
BakeX Engine + blueprint + scanner + Pipeline API: Apache 2.0, no paid tier today

Everything taught in this series — the blueprint format, the build pipeline, the compliance
grading, the CI/CD gate — is in the repository. There is no feature held back, because there is
currently nothing to hold it back for. Self-host it, extend it, fork it.

Worth being straight about where the project actually is: BakeX is young. It has signed releases,
SBOMs and provenance attestations, an OpenSSF Scorecard, a published JSON Schema, and over a
thousand tests — the engineering is in good order. What it does not yet have is users. If you’re
reading this and the shape of the tool fits your problem, you would be early, and early is when
your opinion changes the design.

The repository is at: github.com/invicton/bakex

What This Series Taught

EP01 — EP06 in one view:

Episode What you learned What BakeX does
EP01 Default AMIs are insecure by design Replaces the default AMI with a hardened golden image
EP02 Blueprint as code — the 2am skip disappears HardeningBlueprint YAML, bakex validate / bakex build
EP03 One posture, six providers, no drift 18 shipped blueprints; only target differs across providers
EP04 Automated OpenSCAP — grade at build time A–F from the XCCDF score, SARIF 2.1.0 export, baseline compare
EP05 CI/CD gate — the unhardened image never deploys Pipeline API: POST /api/pipeline/scan, parse .passed
EP06 The platform — OSS, self-hostable, extendable Apache 2.0, Compose install, blueprints + provider plugins

What’s Next

This series closes the OS hardening gap. The same principle — declare desired state, build
reproducibly, verify automatically — applies to every layer of your infrastructure.

Write the next blueprint

The most useful thing you can do with what this series taught is add a blueprint, and it is
genuinely pure YAML — no Python, no engine changes, no build system to learn.

You’ve spent five episodes on Ubuntu 22.04 CIS Level 1. The natural next one is Level 2 for the
same OS: #1 — Ubuntu 22.04 CIS Level 2. The
issue carries the acceptance criteria and the exact verify command, and the review loop is
bakex validate returning 0.

If a different distro is closer to what you actually run, the whole set is filed and labelled:
good first issues, blueprint label.
RHEL 9, AlmaLinux 9, Rocky 9, Debian 12, and Amazon Linux 2023 all have gaps. Each one is one
file, and each is the sort of contribution that takes an evening.

GitHub: github.com/invicton/bakex

Elsewhere on the blog

If you’ve been following the eBPF: From Kernel to Cloud series,
EP10 covers what happens when you combine kernel-level observability with the hardened base BakeX
produces: every connection, every process spawn, every file access — visible from the host kernel,
on an OS baseline you can verify.

The next series is the Purple Team Playbook — real attack paths against cloud and Kubernetes
infrastructure, how they’re detected, and how they’re closed.

Get new episodes in your inbox → linuxcent.com/subscribe

Hardening Blueprint as Code — Declare Your OS Baseline in YAML

by
Reading Time: 8 minutes

OS Hardening as Code, Episode 2
Cloud AMI Security Risks · Linux Hardening as Code**

Note: the tool in this series was released as Stratum and renamed to BakeX at
v0.6.0 — same project, same license, same team. Commands below use the current bakex
CLI. If you arrived here looking for stratum or pip install stratumoss, you’re in the
right place: github.com/invicton/bakex.

TL;DR

  • A hardening runbook is a list of steps someone runs. A HardeningBlueprint YAML is a build artifact — if it wasn’t applied, the image doesn’t exist
  • Linux hardening as code means declaring your entire OS security baseline in a single YAML file and building it reproducibly across any provider
  • bakex build blueprints/ubuntu/22.04/cis-l1-aws.yaml either produces a hardened image or fails — there is no partial state
  • The blueprint includes: target OS/provider/base image, compliance benchmark, filesystem layout, users, and per-rule overrides with documented justifications
  • One blueprint file = one source of truth for your hardening posture, version-controlled and reviewable like any other infrastructure code
  • Post-build OpenSCAP scan runs automatically — the image only snapshots if it passes

The Problem: A Runbook That Gets Skipped Once Is a Runbook That Gets Skipped

Hardening runbook
       │
       ▼
  Human executes
  steps manually
       │
       ├─── 47 deployments: followed correctly
       │
       └─── 1 deployment at 2am: step 12 skipped
                    │
                    ▼
           Instance in production
           without audit logging,
           SSH password auth enabled,
           unnecessary services running

Linux hardening as code eliminates the human decision point. If the blueprint wasn’t applied, the image doesn’t exist.

EP01 showed that default cloud AMIs arrive pre-broken — unnecessary services, no audit logging, weak kernel parameters, SSH configured for convenience not security. The obvious response is a hardening script. But a script run by a human is still a process step. It can be skipped. It can be done halfway. It can drift across different engineers who each interpret “run the hardening script” slightly differently.

A production deployment last year. The platform team had a solid CIS L1 hardening runbook — 68 steps, well-documented, followed consistently. Then a critical incident at 2am required three new instances to be deployed on short notice. The engineer on call ran the provisioning script and, under pressure, skipped the hardening step with the intention of running it the next morning.

They didn’t. The three instances stayed in production unhardened for six weeks before an automated scan caught them. Audit logging wasn’t configured. SSH was accepting password authentication. Two unnecessary services were running that weren’t in the approved software list.

Nothing was breached. But the finding went into the next compliance report as a gap, the team spent a week remediating, and the post-mortem conclusion was “we need better runbook discipline.”

That’s the wrong conclusion. The runbook isn’t the problem. The problem is that hardening was a process step instead of a build constraint.

What Linux Hardening as Code Actually Means

Linux hardening as code is the same principle as infrastructure as code applied to OS security posture: the desired state is declared in a file, the file is the source of truth, and the execution is deterministic and repeatable.

HardeningBlueprint YAML
         │
         ▼
  bakex build
         │
  ┌──────┴──────────────────┐
  │  Provider Layer          │
  │  (cloud-init, disk       │
  │   names, metadata        │
  │   endpoint per provider) │
  └──────┬──────────────────┘
         │
  ┌──────┴──────────────────┐
  │  Ansible-Lockdown        │
  │  (CIS L1/L2, STIG —      │
  │   the hardening steps)   │
  └──────┬──────────────────┘
         │
  ┌──────┴──────────────────┐
  │  OpenSCAP Scanner        │
  │  (post-build verify)     │
  └──────┬──────────────────┘
         │
         ▼
  Golden Image (AMI/GCP image/Azure image)
  + Compliance grade in image metadata

The YAML file is what you write. BakeX handles the rest.

The HardeningBlueprint YAML

The blueprint is the complete, auditable declaration of your OS security posture:

# blueprints/ubuntu/22.04/cis-l1-aws.yaml
bakex_version: "0.6.0"
kind: HardeningBlueprint

metadata:
  name: ubuntu22-cis-l1-aws
  version: "1.0.0"
  description: >
    CIS Ubuntu Linux 22.04 LTS Benchmark — Level 1 Server profile for AWS.
  tags: [ubuntu, ubuntu22.04, cis, level1, server, aws]

target:
  os: ubuntu22.04
  arch: x86_64
  provider: aws
  base_image: ami-0c7217cdde317cfec   # Ubuntu 22.04 LTS, us-east-1 (x86_64)
  instance_type: t3.medium
  root_volume_size_gb: 20

system:
  hostname: hardened-node
  timezone: UTC
  locale: en_US.UTF-8
  selinux_mode: null              # Ubuntu uses AppArmor; null skips SELinux

filesystem:                       # CIS 1.1.x: separate mounts with noexec/nosuid/nodev
  - device: tmpfs
    mountpoint: /tmp
    fstype: tmpfs
    options: [rw, nosuid, nodev, noexec, relatime]
    size: 2G

  - device: tmpfs
    mountpoint: /dev/shm
    fstype: tmpfs
    options: [rw, nosuid, nodev, noexec, relatime]

users:
  root:
    lock: true                    # CIS 5.4.2: lock the root account
  accounts:
    - name: bakex-admin
      groups: [sudo]
      shell: /bin/bash
      ssh_authorized_keys: []     # Add: - "ssh-ed25519 AAAA..."

compliance:
  benchmark: xccdf_org.ssgproject.content_benchmark_UBUNTU2204
  profile: xccdf_org.ssgproject.content_profile_cis_level1_server
  datastream: /usr/share/xml/scap/ssg/content/ssg-ubuntu2204-ds.xml
  fail_on_findings: true
  severity_threshold: medium

controls:                         # per-rule overrides, keyed by XCCDF rule ID
  xccdf_org.ssgproject.content_rule_grub2_enable_selinux:
    enabled: false
    justification: >
      Ubuntu Linux uses AppArmor as the mandatory access control framework.
      SELinux is not installed or applicable on this distribution.

  xccdf_org.ssgproject.content_rule_package_telnet_removed: true

That is not a simplified illustration — it is the shipped cis-l1-aws.yaml, trimmed of comments. Each section is explicit:

target — which OS, which provider, and which base image to harden. This is the only provider-specific section. Everything below it is portable.

compliance — the OpenSCAP benchmark, profile, and datastream. These are full XCCDF identifiers, not friendly names like cis-l1, because they’re passed straight to oscap. Ubuntu ships no SCAP content package, so BakeX downloads the matching datastream from a ComplianceAsCode release and checksum-verifies it.

filesystem — a list of mounts, each with its CIS-required options.

users — root lock state and the service accounts baked into the image.

controls — documented exceptions, keyed by XCCDF rule ID. Not suppressions — overrides with a recorded justification. A bare true enforces a rule; a map with enabled: false disables it and demands a reason. This is the difference between “we turned off this control” and “this control is not applicable, documented here.”

Building the Image

# Validate the blueprint before building — exit 0 if valid, 1 if not
bakex validate blueprints/ubuntu/22.04/cis-l1-aws.yaml

# Build — this will take 15-25 minutes
bakex build blueprints/ubuntu/22.04/cis-l1-aws.yaml

Validation is fast and offline:

OK    blueprints/ubuntu/22.04/cis-l1-aws.yaml  (HardeningBlueprint 'ubuntu22-cis-l1-aws')

1/1 blueprint(s) valid.

The build walks five stages, and the provider is read from the blueprint — there is no --provider flag to get wrong:

Building 'ubuntu22-cis-l1-aws' (aws) → job 7f3c9e82-4d1b-4c05-a7f3-c9e82d1b4c05
[2026-07-27T15:42:01] Provisioning via aws
[2026-07-27T15:42:45] Applying pre-hardening system configuration (hostname, filesystem, users)
[2026-07-27T15:43:12] Applying Ansible-Lockdown hardening roles
[2026-07-27T15:52:11] Running OpenSCAP compliance scan
[2026-07-27T15:54:10] Snapshotting golden image
[2026-07-27T15:54:47] Image ready: ami-0a7f3c9e82d1b4c05

Status: complete

Two things worth noticing. bakex build also accepts a bundled profile name, so bakex build ubuntu22-cis-l1-aws does the same thing without a path. And --json emits the job record instead of the log, which is what you want in CI.

If the post-build scan comes back below the configured threshold, the build fails — no AMI is created. The instance is terminated. The image does not exist. Exit code is 1, and Status: failed carries the reason.

That is the structural guarantee. You cannot skip a build step at 2am because at 2am you’re calling bakex build, not running steps manually.

The Control Override Mechanism

The override mechanism is what separates this from checkbox compliance.

Every security benchmark has controls that conflict with how production environments actually work. CIS L1 recommends /tmp on a separate partition. Many cloud instances use tmpfs with equivalent nodev, nosuid, noexec mount options. The intent of the control is satisfied. The literal implementation differs.

Without an override mechanism, you have two bad options: fail the scan (noisy, meaningless), or configure the scanner to ignore the control (undocumented, invisible to auditors).

The blueprint’s controls section gives you a third option: record the override and its justification in the same version-controlled artifact that produced the image.

controls:
  xccdf_org.ssgproject.content_rule_grub2_enable_selinux:
    enabled: false
    justification: >
      Ubuntu Linux uses AppArmor as the mandatory access control framework.
      SELinux is not installed or applicable on this distribution.

Note that the key is the full XCCDF rule ID, not a CIS section number. That is deliberate — the rule ID is what OpenSCAP reports, so the override and the scanner finding line up exactly with no translation table in between.

Be precise about what this does today. As of v0.6.0 the controls block is declarative:
it records intent alongside the blueprint, and the UI reads it back, but it is not yet compiled
into an OpenSCAP tailoring file. The scan still evaluates the full profile, so an overridden
rule still shows up as a finding and still counts against the score. What you get right now is
provenance — the justification lives in git, next to the thing that built the image, reviewable
in a pull request — not automatic score adjustment.

That is a smaller claim than “documented exceptions are counted as compliant,” and it’s the
true one. Wiring overrides through to scan tailoring is the obvious next step, and it’s the
kind of well-scoped gap that makes a good first contribution.

What the Blueprint Gives You That a Script Doesn’t

Hardening script HardeningBlueprint YAML
Version-controlled Possible but not enforced Always — it’s a file
Auditable exceptions Typically not Built-in override mechanism
Post-build verification Manual or none Automatic OpenSCAP scan
Image exists only if hardened No Yes — build fails if scan fails
Multi-cloud portability Requires separate scripts Swap the target block; compliance sections stay identical
Drift detection Not possible Rescan instance against original grade
Skippable at 2am Yes No — you’d have to change the build process

The last row is the one that matters. A script is skippable because there’s a human in the loop. A blueprint is a build artifact — you can’t deploy the image without the blueprint having been applied, because the image is what the blueprint produces.

Validating a Blueprint Before Building

# Schema validation — one file
bakex validate blueprints/ubuntu/22.04/cis-l1-aws.yaml

# Or the whole library at once
bakex validate blueprints/**/*.yaml

# Machine-readable, for CI and agents
bakex validate blueprints/ubuntu/22.04/cis-l1-aws.yaml --json

bakex validate exits 0 when every file is valid and 1 when any file fails, which makes it a one-line CI gate. It never touches a cloud API — it is pure schema and cross-field checking, so it runs in your pipeline before you’ve paid for a build instance.

That cross-field part matters more than it sounds. Validation rejects OS/provider combinations the catalog doesn’t support, so a blueprint asking for an OS that provider can’t supply fails at validation time rather than fifteen minutes into a build.

If you want to generate or check blueprints from something other than the CLI, the format is published as a JSON Schema (Draft 2020-12) at docs/schema/hardening-blueprint.schema.json. Point your editor at it for autocomplete, or hand it to an LLM and let it write the blueprint.

Production Gotchas

Build time is 15–25 minutes. Ansible-Lockdown applies 144+ tasks for CIS L1. Build this into your pipeline timing — don’t expect golden images in 3 minutes.

Cloud-init ordering matters. On AWS, certain hardening steps (sysctl tuning, PAM configuration) interact with cloud-init. The BakeX provider layer handles sequencing — but if you add custom hardening roles, test the cloud-init interaction explicitly.

Some CIS controls conflict with managed service requirements. AWS Systems Manager Session Manager requires specific SSH configuration. RDS requires specific networking settings. Use the controls override section to document these — don’t suppress them silently.

Kernel parameter hardening requires a reboot. Controls in the 3.x (network parameters) and 1.5.x (kernel modules) sections apply sysctl changes that take effect on reboot. The BakeX build process reboots the instance before the OpenSCAP scan — don’t skip the reboot if you’re building manually.

Key Takeaways

  • Linux hardening as code means the blueprint YAML is the build artifact — the image either exists and is hardened, or it doesn’t exist
  • The controls override mechanism is the difference between undocumented suppressions and auditable, reasoned exceptions
  • Post-build OpenSCAP scan runs automatically — a failing grade blocks image creation
  • bakex validate is an offline, exit-code-shaped CI gate — it catches unsupported OS/provider pairs before a build instance is ever launched
  • The compliance sections are portable across providers (EP03 covers this): swap the target block, and the benchmark, filesystem, users, and control overrides stay byte-identical
  • Version-controlling the blueprint gives you a complete history of what your OS security posture was at any point in time — the same way Terraform state tracks infrastructure

What’s Next

One blueprint, one provider. EP02 showed that the skip-at-2am problem is solved when hardening is a build artifact rather than a process step.

What it didn’t address: what happens when you expand to a second cloud. GCP uses different disk names. Azure cloud-init fires in a different order. The AWS metadata endpoint IP is different from every other provider. If you maintain separate hardening scripts per cloud, they drift within a month.

EP03 covers multi-cloud OS hardening: one compliance posture, six providers, no drift — and it shows the diff that proves it.

Next: multi-cloud OS hardening — one blueprint for AWS, GCP, and Azure

Get EP03 in your inbox when it publishes → linuxcent.com/subscribe