Immutable OS Series, Episode 1
← Stratum EP06: Stratum — OS Hardening as a Platform · EP01: What Is an Immutable OS? · EP02: Atomic OS Updates Explained →

TL;DR

• An immutable OS is one where the running root filesystem is read-only — the only way to change it is to boot a new, versioned image, never to mutate the one that’s live
• Hardening an image proves it’s correct at build time. Immutability is what keeps that proof true after the image boots into production
• The mechanism is atomic A/B updates: a new OS image is staged fully, then swapped in as one operation — the system is never caught half-updated
• A bad update is one command away from undone: rpm-ostree rollback && systemctl reboot — no reinstall, no image rebuild
• bootc, Fedora CoreOS/Silverblue, and Talos Linux are three real implementations of this model, each targeting a different deployment shape
• This is not a replacement for Stratum’s hardening pipeline — it’s what keeps a hardened image hardened after it ships

The Big Picture: A Snapshot vs. a Guarantee

TRADITIONAL MUTABLE OS                    IMMUTABLE OS
────────────────────────                  ────────────

Golden image (grade: A)                   Deployment A (active, read-only)
        │ boots into prod                          │
        ▼                                           │  atomic swap
Running root filesystem (read-write)                ▼
        │                                  Deployment B (staged)
        │  SSH fix, config-mgmt run,               │
        │  ad-hoc package install                   │  if boot fails
        ▼                                           ▼
Drifted state — no build artifact         Rollback (one command,
matches what's actually running            no reinstall)

An immutable OS is a system whose root filesystem cannot be changed in place — every change ships as a new, complete, versioned image, and the system swaps to it atomically or not at all. That’s the one-sentence answer, and it’s the reason this series exists: a hardening pipeline can prove an image is correct on the day it’s built, but on a traditional mutable root filesystem, nothing stops that proof from becoming false the day after.

The Gap Stratum’s Grade Doesn’t Cover

Stratum’s series ended with a hardened, graded, pipeline-gated image — POST /api/pipeline/scan fails the build if the grade drops below B, so an unhardened image never reaches production. That solved a real problem: images used to ship broken by default, and now they don’t.

But watch what happens six weeks later. An on-call engineer SSHes into a production node at 2 a.m. to unblock an incident and leaves behind a one-line iptables rule that was never reviewed. A config-management run pushes an unrelated package upgrade because someone’s playbook target list was too broad. A well-meaning teammate installs a debugging tool “just for now” and forgets to remove it. None of this touches the build pipeline. None of it fails a scan, because no scan runs again after the image ships.

Six months later, an auditor asks for evidence that the instance matches its compliance grade. The honest answer is: it did, once, the day it was built. Nobody can say what’s true about it now — the golden image and the running system are two different, unreconciled things.

That’s the gap. Hardening is a build-time guarantee. Immutability is what makes it a runtime guarantee too, because there’s no path left for a change to happen except through the build pipeline that produced the image in the first place.

From Golden Images to Immutable OS: A Short History

Golden images (Stratum’s territory) solved the “every instance starts insecure” problem by baking the correct configuration in at build time — the same idea as infrastructure-as-code applied to an OS baseline. Configuration management tools (Ansible, Chef, Puppet) then tried to solve drift by re-applying the desired state on a schedule, converging the system back toward correctness every run.

Convergence is not the same as prevention. A config-management run that fires every 30 minutes still leaves a 29-minute window where the system can be anything. And convergence tools can only fix drift they know to look for — an ad-hoc apt install that isn’t in anyone’s playbook just sits there, invisible, until someone happens to notice.

Immutable OS designs remove the window entirely. If the root filesystem is mounted read-only, apt install on a running node doesn’t drift the system — it fails, because there’s nowhere to write the new package. The only way to add that package is to build a new image and boot into it. Prevention replaces convergence.

How Atomic Updates Actually Work

The core mechanism, used by ostree-based systems (Fedora CoreOS, Silverblue) and bootc alike, is A/B deployment:

1. Two deployment slots exist on disk at all times — call them A (active) and B (staged). Only one is booted at a time.
2. An update downloads and assembles the entire new OS image into the inactive slot. This can take minutes. The running system is completely unaffected while it happens — there is no partial state visible to production traffic.
3. The bootloader entry swaps atomically. This is a single operation, not a sequence of file writes — the system either boots the new deployment on next reboot, or it doesn’t. There’s no window where half the files are new and half are old.
4. If the new deployment fails to boot or fails a health check, rolling back means booting the previous slot — the old deployment was never deleted, never modified. It’s still exactly what it was before the update.

# Check current and staged deployments
$ rpm-ostree status
State: idle
Deployments:
● ostree://fedora:fedora/38/x86_64/coreos
                   Version: 38.20240210.3.0 (2024-02-10T09:14:22Z)
                   Commit: 8f2a1c...

  ostree://fedora:fedora/38/x86_64/coreos
                   Version: 38.20240115.2.0 (2024-01-15T11:02:03Z)
                   Commit: 3b7e9d...

# Roll back to the previous deployment — no rebuild, no reinstall
$ rpm-ostree rollback
Moving 'ostree://fedora:fedora/38/x86_64/coreos' (38.20240115.2.0) to be first deployment
Run "systemctl reboot" to start a rollback

$ systemctl reboot

The ● marks the currently booted deployment. The second entry never disappeared when the update landed — it’s exactly the filesystem that was running two weeks ago, byte for byte, ready to boot again.

bootc — covered in depth in EP04 — applies the same A/B model but defines the OS image as an OCI container image, built with a standard Containerfile and pushed to a normal container registry. The deployment mechanism is the same; the packaging format is the one most infrastructure teams already have tooling for.

What You Give Up, and What You Get Back

The trade-off is real: an immutable OS removes a workflow a lot of engineers rely on — the quick SSH fix. That’s not a bug in the design. It’s the entire point. If the quick fix is impossible, it can’t happen accidentally, and it can’t happen without going through review.

Three Ways This Actually Ships Today

This series covers each of these in depth over the coming episodes — for now, know they exist and roughly where each one fits:

• Fedora CoreOS / Silverblue (EP03) — ostree-based, general-purpose immutable Linux. CoreOS targets servers and container hosts; Silverblue targets immutable desktops. Both use rpm-ostree for the deployment model shown above.
• bootc (EP04) — an immutable OS image defined as a container image and booted directly, no separate “OS build” toolchain from your application build toolchain. Newer, and increasingly the direction RHEL-family distros are heading.
• Talos Linux (EP05) — purpose-built for Kubernetes nodes. No SSH, no shell, no package manager at all — the only interface is an API (talosctl). The most aggressive point on this spectrum: not just read-only, but no interactive access whatsoever.

None of these require you to abandon Stratum. A bootc image or a Fedora CoreOS image can still be built from a hardened, CIS-benchmarked base — the hardening pipeline and the immutability model solve different problems and compose cleanly.

Production Gotchas

Immutability doesn’t mean “no state.” /etc and /var are typically still writable on ostree-based systems (application data, logs, local config overrides have to live somewhere). “Immutable” means the OS binaries and base configuration can’t be mutated in place — read the docs for your specific distro to know exactly what’s writable.

Rollback isn’t instant if you don’t test it first. rpm-ostree rollback works, but if you’ve never practiced it, the first time you run it under incident pressure is the wrong time to discover a health check you forgot to configure. Rehearse rollback the same way you’d rehearse a database failover.

Container image tooling doesn’t automatically make an OS image safe. bootc images are built like container images, which means it’s easy to accidentally treat them like disposable containers instead of long-lived OS deployments — with all the patching and lifecycle discipline that implies.

Not everything you run today has an immutable-OS story yet. Legacy configuration management (Puppet/Chef agents that expect to write to /etc continuously) and some monitoring agents assume a mutable filesystem. Check compatibility before you migrate a fleet.

Quick Reference

# ostree/rpm-ostree (Fedora CoreOS, Silverblue)
rpm-ostree status                  # current + staged deployments
rpm-ostree upgrade                 # stage the next image
rpm-ostree rollback                # revert to the previous deployment
ostree admin status                # lower-level deployment inspection

# bootc
bootc status                       # current + staged image, digest-pinned
bootc upgrade                      # pull and stage the next image
bootc rollback                     # revert to the previous deployment

# Talos Linux (API-only, no shell)
talosctl version                   # node + API version
talosctl get machineconfig         # current applied config
talosctl upgrade --image <ref>     # stage a new node image

Key Takeaways

• A hardened image is a build-time guarantee; an immutable OS is what makes that guarantee hold at runtime too
• Atomic A/B deployment means the system is never caught half-updated, and the previous deployment is always intact for rollback
• Config-management convergence fights drift on a schedule; immutability removes the writable path drift needs to happen at all
• rpm-ostree/bootc give you an exact, versioned answer to “what changed” instead of shell history and guesswork
• This composes with Stratum’s hardening pipeline — it doesn’t replace it

What’s Next

EP01 established the gap: hardening proves an image correct once, at build time, and a mutable root filesystem gives that proof an expiration date nobody tracks. EP02 goes one level deeper into the mechanism that closes it — exactly how ostree and bootc implement atomic A/B updates under the hood, including how the bootloader is involved and what “atomic” actually guarantees.

Next: EP02 — Atomic OS Updates Explained: How ostree and bootc Actually Work

Get EP02 in your inbox when it publishes → linuxcent.com/subscribe

The post What Is an Immutable OS — and Why Hardening Isn’t Enough appeared first on Linuxcent.