Immutable OS Series, Episode 2
← EP01: What Is an Immutable OS? · EP02: Atomic OS Updates Explained · All Immutable OS Episodes →

TL;DR

• Atomic OS updates explained at the mechanism level: ostree stores every deployment as a content-addressed commit, not a set of files you overwrite — “atomic” is a property of the filesystem layout, not a promise a script makes
• The actual atomicity boundary is a single bootloader configuration write — everything before that point is fully reversible, and everything after it is a clean boot into a complete, self-contained deployment
• bootc builds on the same ostree deployment model but starts from a Containerfile, so building a bootable OS image uses the same toolchain as building an application container
• Power loss mid-update is a non-event: the system reboots into whatever the bootloader pointed at before the write, because the new deployment was never referenced until that one atomic write succeeded
• Rollback targets aren’t kept forever — garbage collection and configurable deployment limits mean “you can always roll back” has a real, finite window
• This is the mechanism EP01 described in outline; this episode is what actually happens on disk

The Big Picture: A Commit Graph, Not a File Tree

ostree REPOSITORY (content-addressed objects)
─────────────────────────────────────────────
  commit A (hash 8f2a1c...)  ──parent──▶  commit B (hash 3b7e9d...)
       │                                        │
       │ checked out as                         │ checked out as
       ▼                                        ▼
  /ostree/deploy/os/deploy/8f2a1c...    /ostree/deploy/os/deploy/3b7e9d...
  (READ-ONLY bind mount → /)            (READ-ONLY bind mount → /, once active)

BOOTLOADER CONFIG (the atomicity boundary)
─────────────────────────────────────────────
  grub.cfg / loader entries
       │
       └── points to exactly ONE deployment directory at a time
           Changing this pointer IS the update. Nothing else has
           to happen for the new deployment to become "the OS."

Atomic OS updates explained simply: ostree never edits a running deployment’s files. It writes an entirely new, complete deployment as a set of immutable, content-addressed objects somewhere else on disk, and the update becomes real the instant a single bootloader entry is rewritten to point at it. EP01 showed this from the outside — rpm-ostree status, rollback, a clean before/after. This episode is what’s actually happening underneath those commands.

Every Deployment Is a Commit, Not a Directory You Edited

A traditional package manager mutates files in place: apt upgrade overwrites /usr/bin/curl with a new binary, in the same inode, on the same live filesystem the kernel and every running process are using. If that write is interrupted, or if two updates race, the result is whatever state the filesystem happened to be in when things stopped — there’s no defined “before” state to return to, because the before state was destroyed in place.

This is the same declarative-artifact idea Stratum’s HardeningBlueprint YAML applies to OS hardening — the artifact either fully exists or the build failed, with nothing skippable in between — extended down to the filesystem itself.

ostree does something structurally different: every file in a deployment is stored as an object named by the SHA-256 hash of its content, inside a repository (/ostree/repo). A deployment is a commit — a tree of these hashed objects, checksummed all the way up, the same content-addressing model Git uses for a repository’s history. Deploying an update means:

1. Pull or build the new commit into the local ostree repository (pure object storage — this doesn’t touch the running system at all)
2. Check out that commit into a new deployment directory (/ostree/deploy/<os>/deploy/<checksum>) — still doesn’t touch the running system
3. Write a new bootloader entry pointing at that new deployment directory
4. Reboot

Steps 1 and 2 can take minutes, involve gigabytes of I/O, and fail halfway through with zero consequence — the running system’s deployment directory was never opened for writing. There is no partial-update state visible to anything, because nothing that’s currently running was ever touched.

The Atomicity Boundary: One Bootloader Write

“Atomic” specifically refers to step 3. Rewriting a bootloader entry (a GRUB grub.cfg regeneration, or a systemd-boot loader entry file) is small enough to be a single filesystem operation — either the new entry exists on disk, or it doesn’t. There’s no meaningful “half-written bootloader entry” state that a power failure can leave you in: at boot, the firmware reads whatever bootloader configuration fully exists, and that configuration names exactly one deployment.

POWER LOSS DURING STEP 1 or 2 (pulling/staging the new commit)
────────────────────────────────────────────────────────────
Next boot: bootloader entry still points at the OLD deployment.
The new commit's partial objects sit in the repo, orphaned,
inert. System boots exactly as if the update never started.

POWER LOSS DURING STEP 3 (bootloader entry write)
────────────────────────────────────────────────────────────
Filesystem-level atomic rename guarantees the entry write itself
either completes or doesn't. Next boot: either the old deployment
(write didn't land) or the new one (write landed) — never a
corrupted bootloader config caught in between.

POWER LOSS AFTER STEP 3, BEFORE REBOOT
────────────────────────────────────────────────────────────
Doesn't matter — the running system hasn't changed. The new
deployment activates on the NEXT boot, whenever that happens.

This is the property EP01 called “the system is never caught half-updated” — and now you can see exactly why: every step before the bootloader write is invisible to the running system, and the bootloader write itself is small enough that the filesystem’s own atomic-rename guarantee covers it. There’s no custom transaction logic to trust. It’s a property of doing the update in the right order, using a write that was already atomic.

bootc: The Same Model, a Container Build Toolchain

bootc uses this identical deployment mechanism — the on-disk layout, the bootloader swap, the rollback behavior are all the same ostree machinery. What bootc changes is how the commit gets built in the first place.

# Containerfile — this IS the OS image definition
FROM quay.io/fedora/fedora-bootc:41

RUN dnf install -y nginx && \
    systemctl enable nginx && \
    dnf clean all

# Standard container build — no special OS-image tooling required

# Build it exactly like an application container
$ podman build -t myregistry.example.com/os/web-node:v12 .
$ podman push myregistry.example.com/os/web-node:v12

# On the target machine — pulls the image, converts it to an
# ostree commit, stages it as the next deployment
$ bootc switch myregistry.example.com/os/web-node:v12
Queued for next boot: myregistry.example.com/os/web-node:v12
Please reboot to complete the update.

$ systemctl reboot

bootc switch and bootc upgrade do the same three-step dance as raw ostree — pull the new commit (here, derived from a container image’s layers instead of an RPM-based tree), stage a deployment directory, write the bootloader entry — the difference is entirely in step 1: bootc converts OCI container image layers into an ostree commit instead of building one from package installation directly. Your existing container registry, existing Containerfile conventions, and existing image-signing pipeline all apply unchanged to what is, underneath, a bootable operating system.

Where ostree and bootc Actually Diverge

Nothing about atomicity, rollback safety, or the deployment model changes between the two — bootc’s entire value proposition is packaging the same guarantee behind tooling most infrastructure teams already have muscle memory for.

The Part EP01 Didn’t Mention: Rollback Has a Shelf Life

“The previous deployment is always intact for rollback” (EP01’s phrasing) is true, but not indefinitely. Each deployment consumes real disk space — a full OS tree’s worth of objects, though ostree deduplicates identical objects across commits so an incremental update doesn’t cost a second full copy. Two mechanisms limit how far back you can actually roll:

Deployment count limits. Most configurations keep a bounded number of deployments (commonly 2–3). Once you’ve upgraded past that limit, the oldest deployment is pruned — rpm-ostree cleanup or an automatic policy removes it, and its objects become eligible for garbage collection if nothing else references them.

Garbage collection reclaims orphaned objects. ostree prune (or rpm-ostree cleanup -p) removes any object in the repository not reachable from a currently-kept deployment or a pinned ref. If you pruned a deployment last week and you need to roll back to it today, that commit is gone — not degraded, not slow to restore, simply no longer present.

# See exactly what's kept and what's eligible for cleanup
$ ostree admin status
  fedora-coreos 38.20240210.3.0 (booted)   # current
  fedora-coreos 38.20240115.2.0            # one rollback available

# Pin a deployment explicitly if you need a longer-lived rollback
# target than the default retention policy provides
$ ostree admin pin 1

If your incident-response plan assumes “we can always roll back to last month’s known-good state,” verify that against your actual retention policy — the default is usually one previous deployment, not an archive.

Quick Reference

# Inspect the commit graph and current deployments
ostree admin status                      # deployments + which is booted
ostree log <ref>                         # commit history for a branch
ostree show <checksum>                   # inspect a specific commit

# rpm-ostree (Fedora CoreOS / Silverblue)
rpm-ostree status                        # current + staged, same as EP01
rpm-ostree cleanup -p                    # prune old deployments + GC

# bootc
bootc status                             # current + staged image
bootc switch <image-ref>                 # move to a different image
bootc upgrade                            # pull latest tag, stage it
bootc rollback                           # revert to previous deployment

Production Gotchas

“Atomic” doesn’t mean “instant.” Staging a new deployment can take as long as a full OS install — the atomicity guarantee is about the swap being indivisible, not about the whole process being fast. Budget real time for the pull-and-stage phase in maintenance windows.

Deduplication means disk usage doesn’t scale linearly with deployment count, but it isn’t free either. A kernel or major package version bump touches enough objects that “just keep 5 deployments for safety” can use more disk than teams expect. Monitor /ostree/repo size, don’t assume it’s negligible.

Pinning a deployment and forgetting about it silently defeats garbage collection. ostree admin pin is the right tool for “I need to guarantee this stays available,” but a pinned deployment never gets reclaimed automatically — audit pins periodically or disk usage grows unbounded.

bootc’s registry dependency is a new failure mode ostree-native updates didn’t have. If your container registry is unreachable, bootc upgrade fails the same way a registry-down event fails an application deployment — factor registry availability into your OS update SLA the same way you already do for app deployments.

Key Takeaways

• Every ostree deployment is a content-addressed commit, not a set of files mutated in place — that’s what makes “atomic” a filesystem property instead of a script’s promise
• The actual atomicity boundary is a single bootloader entry write; everything before it is invisible to the running system, everything after it takes effect on next boot
• bootc uses the identical deployment mechanism, but builds commits from standard Containerfiles and distributes them through standard container registries
• Rollback is real but bounded — deployment limits and garbage collection mean “always roll back” has a specific, checkable retention window, not an unlimited one
• ostree and bootc differ in build/distribution tooling, not in the safety guarantees the deployment model provides

What’s Next

EP02 covered the mechanism in the abstract. EP03 runs it day-to-day — Fedora CoreOS and Silverblue in practice: what changes about dnf install, package layering, troubleshooting, and rollback when you’re actually living on top of this model instead of reading about it.

Next: EP03 — Fedora CoreOS / Silverblue in Practice

Get EP03 in your inbox when it publishes → linuxcent.com/subscribe

The post Atomic OS Updates Explained: How ostree and bootc Actually Work appeared first on Linuxcent.

Immutable OS Series, Episode 1
← Stratum EP06: Stratum — OS Hardening as a Platform · EP01: What Is an Immutable OS? · EP02: Atomic OS Updates Explained →

TL;DR

• An immutable OS is one where the running root filesystem is read-only — the only way to change it is to boot a new, versioned image, never to mutate the one that’s live
• Hardening an image proves it’s correct at build time. Immutability is what keeps that proof true after the image boots into production
• The mechanism is atomic A/B updates: a new OS image is staged fully, then swapped in as one operation — the system is never caught half-updated
• A bad update is one command away from undone: rpm-ostree rollback && systemctl reboot — no reinstall, no image rebuild
• bootc, Fedora CoreOS/Silverblue, and Talos Linux are three real implementations of this model, each targeting a different deployment shape
• This is not a replacement for Stratum’s hardening pipeline — it’s what keeps a hardened image hardened after it ships

The Big Picture: A Snapshot vs. a Guarantee

TRADITIONAL MUTABLE OS                    IMMUTABLE OS
────────────────────────                  ────────────

Golden image (grade: A)                   Deployment A (active, read-only)
        │ boots into prod                          │
        ▼                                           │  atomic swap
Running root filesystem (read-write)                ▼
        │                                  Deployment B (staged)
        │  SSH fix, config-mgmt run,               │
        │  ad-hoc package install                   │  if boot fails
        ▼                                           ▼
Drifted state — no build artifact         Rollback (one command,
matches what's actually running            no reinstall)

An immutable OS is a system whose root filesystem cannot be changed in place — every change ships as a new, complete, versioned image, and the system swaps to it atomically or not at all. That’s the one-sentence answer, and it’s the reason this series exists: a hardening pipeline can prove an image is correct on the day it’s built, but on a traditional mutable root filesystem, nothing stops that proof from becoming false the day after.

The Gap Stratum’s Grade Doesn’t Cover

Stratum’s series ended with a hardened, graded, pipeline-gated image — POST /api/pipeline/scan fails the build if the grade drops below B, so an unhardened image never reaches production. That solved a real problem: images used to ship broken by default, and now they don’t.

But watch what happens six weeks later. An on-call engineer SSHes into a production node at 2 a.m. to unblock an incident and leaves behind a one-line iptables rule that was never reviewed. A config-management run pushes an unrelated package upgrade because someone’s playbook target list was too broad. A well-meaning teammate installs a debugging tool “just for now” and forgets to remove it. None of this touches the build pipeline. None of it fails a scan, because no scan runs again after the image ships.

Six months later, an auditor asks for evidence that the instance matches its compliance grade. The honest answer is: it did, once, the day it was built. Nobody can say what’s true about it now — the golden image and the running system are two different, unreconciled things.

That’s the gap. Hardening is a build-time guarantee. Immutability is what makes it a runtime guarantee too, because there’s no path left for a change to happen except through the build pipeline that produced the image in the first place.

From Golden Images to Immutable OS: A Short History

Golden images (Stratum’s territory) solved the “every instance starts insecure” problem by baking the correct configuration in at build time — the same idea as infrastructure-as-code applied to an OS baseline. Configuration management tools (Ansible, Chef, Puppet) then tried to solve drift by re-applying the desired state on a schedule, converging the system back toward correctness every run.

Convergence is not the same as prevention. A config-management run that fires every 30 minutes still leaves a 29-minute window where the system can be anything. And convergence tools can only fix drift they know to look for — an ad-hoc apt install that isn’t in anyone’s playbook just sits there, invisible, until someone happens to notice.

Immutable OS designs remove the window entirely. If the root filesystem is mounted read-only, apt install on a running node doesn’t drift the system — it fails, because there’s nowhere to write the new package. The only way to add that package is to build a new image and boot into it. Prevention replaces convergence.

How Atomic Updates Actually Work

The core mechanism, used by ostree-based systems (Fedora CoreOS, Silverblue) and bootc alike, is A/B deployment:

1. Two deployment slots exist on disk at all times — call them A (active) and B (staged). Only one is booted at a time.
2. An update downloads and assembles the entire new OS image into the inactive slot. This can take minutes. The running system is completely unaffected while it happens — there is no partial state visible to production traffic.
3. The bootloader entry swaps atomically. This is a single operation, not a sequence of file writes — the system either boots the new deployment on next reboot, or it doesn’t. There’s no window where half the files are new and half are old.
4. If the new deployment fails to boot or fails a health check, rolling back means booting the previous slot — the old deployment was never deleted, never modified. It’s still exactly what it was before the update.

# Check current and staged deployments
$ rpm-ostree status
State: idle
Deployments:
● ostree://fedora:fedora/38/x86_64/coreos
                   Version: 38.20240210.3.0 (2024-02-10T09:14:22Z)
                   Commit: 8f2a1c...

  ostree://fedora:fedora/38/x86_64/coreos
                   Version: 38.20240115.2.0 (2024-01-15T11:02:03Z)
                   Commit: 3b7e9d...

# Roll back to the previous deployment — no rebuild, no reinstall
$ rpm-ostree rollback
Moving 'ostree://fedora:fedora/38/x86_64/coreos' (38.20240115.2.0) to be first deployment
Run "systemctl reboot" to start a rollback

$ systemctl reboot

The ● marks the currently booted deployment. The second entry never disappeared when the update landed — it’s exactly the filesystem that was running two weeks ago, byte for byte, ready to boot again.

bootc — covered in depth in EP04 — applies the same A/B model but defines the OS image as an OCI container image, built with a standard Containerfile and pushed to a normal container registry. The deployment mechanism is the same; the packaging format is the one most infrastructure teams already have tooling for.

What You Give Up, and What You Get Back

The trade-off is real: an immutable OS removes a workflow a lot of engineers rely on — the quick SSH fix. That’s not a bug in the design. It’s the entire point. If the quick fix is impossible, it can’t happen accidentally, and it can’t happen without going through review.

Three Ways This Actually Ships Today

This series covers each of these in depth over the coming episodes — for now, know they exist and roughly where each one fits:

• Fedora CoreOS / Silverblue (EP03) — ostree-based, general-purpose immutable Linux. CoreOS targets servers and container hosts; Silverblue targets immutable desktops. Both use rpm-ostree for the deployment model shown above.
• bootc (EP04) — an immutable OS image defined as a container image and booted directly, no separate “OS build” toolchain from your application build toolchain. Newer, and increasingly the direction RHEL-family distros are heading.
• Talos Linux (EP05) — purpose-built for Kubernetes nodes. No SSH, no shell, no package manager at all — the only interface is an API (talosctl). The most aggressive point on this spectrum: not just read-only, but no interactive access whatsoever.

None of these require you to abandon Stratum. A bootc image or a Fedora CoreOS image can still be built from a hardened, CIS-benchmarked base — the hardening pipeline and the immutability model solve different problems and compose cleanly.

Production Gotchas

Immutability doesn’t mean “no state.” /etc and /var are typically still writable on ostree-based systems (application data, logs, local config overrides have to live somewhere). “Immutable” means the OS binaries and base configuration can’t be mutated in place — read the docs for your specific distro to know exactly what’s writable.

Rollback isn’t instant if you don’t test it first. rpm-ostree rollback works, but if you’ve never practiced it, the first time you run it under incident pressure is the wrong time to discover a health check you forgot to configure. Rehearse rollback the same way you’d rehearse a database failover.

Container image tooling doesn’t automatically make an OS image safe. bootc images are built like container images, which means it’s easy to accidentally treat them like disposable containers instead of long-lived OS deployments — with all the patching and lifecycle discipline that implies.

Not everything you run today has an immutable-OS story yet. Legacy configuration management (Puppet/Chef agents that expect to write to /etc continuously) and some monitoring agents assume a mutable filesystem. Check compatibility before you migrate a fleet.

Quick Reference

# ostree/rpm-ostree (Fedora CoreOS, Silverblue)
rpm-ostree status                  # current + staged deployments
rpm-ostree upgrade                 # stage the next image
rpm-ostree rollback                # revert to the previous deployment
ostree admin status                # lower-level deployment inspection

# bootc
bootc status                       # current + staged image, digest-pinned
bootc upgrade                      # pull and stage the next image
bootc rollback                     # revert to the previous deployment

# Talos Linux (API-only, no shell)
talosctl version                   # node + API version
talosctl get machineconfig         # current applied config
talosctl upgrade --image <ref>     # stage a new node image

Key Takeaways

• A hardened image is a build-time guarantee; an immutable OS is what makes that guarantee hold at runtime too
• Atomic A/B deployment means the system is never caught half-updated, and the previous deployment is always intact for rollback
• Config-management convergence fights drift on a schedule; immutability removes the writable path drift needs to happen at all
• rpm-ostree/bootc give you an exact, versioned answer to “what changed” instead of shell history and guesswork
• This composes with Stratum’s hardening pipeline — it doesn’t replace it

What’s Next

EP01 established the gap: hardening proves an image correct once, at build time, and a mutable root filesystem gives that proof an expiration date nobody tracks. EP02 goes one level deeper into the mechanism that closes it — exactly how ostree and bootc implement atomic A/B updates under the hood, including how the bootloader is involved and what “atomic” actually guarantees.

Next: EP02 — Atomic OS Updates Explained: How ostree and bootc Actually Work

Get EP02 in your inbox when it publishes → linuxcent.com/subscribe

The post What Is an Immutable OS — and Why Hardening Isn’t Enough appeared first on Linuxcent.