Immutable OS Series, Episode 2
← EP01: What Is an Immutable OS? · EP02: Atomic OS Updates Explained · All Immutable OS Episodes →
TL;DR
- Atomic OS updates explained at the mechanism level: ostree stores every deployment as a content-addressed commit, not a set of files you overwrite — “atomic” is a property of the filesystem layout, not a promise a script makes
- The actual atomicity boundary is a single bootloader configuration write — everything before that point is fully reversible, and everything after it is a clean boot into a complete, self-contained deployment
- bootc builds on the same ostree deployment model but starts from a
Containerfile, so building a bootable OS image uses the same toolchain as building an application container - Power loss mid-update is a non-event: the system reboots into whatever the bootloader pointed at before the write, because the new deployment was never referenced until that one atomic write succeeded
- Rollback targets aren’t kept forever — garbage collection and configurable deployment limits mean “you can always roll back” has a real, finite window
- This is the mechanism EP01 described in outline; this episode is what actually happens on disk
The Big Picture: A Commit Graph, Not a File Tree
ostree REPOSITORY (content-addressed objects)
─────────────────────────────────────────────
commit A (hash 8f2a1c...) ──parent──▶ commit B (hash 3b7e9d...)
│ │
│ checked out as │ checked out as
▼ ▼
/ostree/deploy/os/deploy/8f2a1c... /ostree/deploy/os/deploy/3b7e9d...
(READ-ONLY bind mount → /) (READ-ONLY bind mount → /, once active)
BOOTLOADER CONFIG (the atomicity boundary)
─────────────────────────────────────────────
grub.cfg / loader entries
│
└── points to exactly ONE deployment directory at a time
Changing this pointer IS the update. Nothing else has
to happen for the new deployment to become "the OS."
Atomic OS updates explained simply: ostree never edits a running deployment’s files. It writes an entirely new, complete deployment as a set of immutable, content-addressed objects somewhere else on disk, and the update becomes real the instant a single bootloader entry is rewritten to point at it. EP01 showed this from the outside — rpm-ostree status, rollback, a clean before/after. This episode is what’s actually happening underneath those commands.
Every Deployment Is a Commit, Not a Directory You Edited
A traditional package manager mutates files in place: apt upgrade overwrites /usr/bin/curl with a new binary, in the same inode, on the same live filesystem the kernel and every running process are using. If that write is interrupted, or if two updates race, the result is whatever state the filesystem happened to be in when things stopped — there’s no defined “before” state to return to, because the before state was destroyed in place.
This is the same declarative-artifact idea Stratum’s HardeningBlueprint YAML applies to OS hardening — the artifact either fully exists or the build failed, with nothing skippable in between — extended down to the filesystem itself.
ostree does something structurally different: every file in a deployment is stored as an object named by the SHA-256 hash of its content, inside a repository (/ostree/repo). A deployment is a commit — a tree of these hashed objects, checksummed all the way up, the same content-addressing model Git uses for a repository’s history. Deploying an update means:
- Pull or build the new commit into the local ostree repository (pure object storage — this doesn’t touch the running system at all)
- Check out that commit into a new deployment directory (
/ostree/deploy/<os>/deploy/<checksum>) — still doesn’t touch the running system - Write a new bootloader entry pointing at that new deployment directory
- Reboot
Steps 1 and 2 can take minutes, involve gigabytes of I/O, and fail halfway through with zero consequence — the running system’s deployment directory was never opened for writing. There is no partial-update state visible to anything, because nothing that’s currently running was ever touched.
The Atomicity Boundary: One Bootloader Write
“Atomic” specifically refers to step 3. Rewriting a bootloader entry (a GRUB grub.cfg regeneration, or a systemd-boot loader entry file) is small enough to be a single filesystem operation — either the new entry exists on disk, or it doesn’t. There’s no meaningful “half-written bootloader entry” state that a power failure can leave you in: at boot, the firmware reads whatever bootloader configuration fully exists, and that configuration names exactly one deployment.
POWER LOSS DURING STEP 1 or 2 (pulling/staging the new commit)
────────────────────────────────────────────────────────────
Next boot: bootloader entry still points at the OLD deployment.
The new commit's partial objects sit in the repo, orphaned,
inert. System boots exactly as if the update never started.
POWER LOSS DURING STEP 3 (bootloader entry write)
────────────────────────────────────────────────────────────
Filesystem-level atomic rename guarantees the entry write itself
either completes or doesn't. Next boot: either the old deployment
(write didn't land) or the new one (write landed) — never a
corrupted bootloader config caught in between.
POWER LOSS AFTER STEP 3, BEFORE REBOOT
────────────────────────────────────────────────────────────
Doesn't matter — the running system hasn't changed. The new
deployment activates on the NEXT boot, whenever that happens.
This is the property EP01 called “the system is never caught half-updated” — and now you can see exactly why: every step before the bootloader write is invisible to the running system, and the bootloader write itself is small enough that the filesystem’s own atomic-rename guarantee covers it. There’s no custom transaction logic to trust. It’s a property of doing the update in the right order, using a write that was already atomic.
bootc: The Same Model, a Container Build Toolchain
bootc uses this identical deployment mechanism — the on-disk layout, the bootloader swap, the rollback behavior are all the same ostree machinery. What bootc changes is how the commit gets built in the first place.
# Containerfile — this IS the OS image definition
FROM quay.io/fedora/fedora-bootc:41
RUN dnf install -y nginx && \
systemctl enable nginx && \
dnf clean all
# Standard container build — no special OS-image tooling required
# Build it exactly like an application container
$ podman build -t myregistry.example.com/os/web-node:v12 .
$ podman push myregistry.example.com/os/web-node:v12
# On the target machine — pulls the image, converts it to an
# ostree commit, stages it as the next deployment
$ bootc switch myregistry.example.com/os/web-node:v12
Queued for next boot: myregistry.example.com/os/web-node:v12
Please reboot to complete the update.
$ systemctl reboot
bootc switch and bootc upgrade do the same three-step dance as raw ostree — pull the new commit (here, derived from a container image’s layers instead of an RPM-based tree), stage a deployment directory, write the bootloader entry — the difference is entirely in step 1: bootc converts OCI container image layers into an ostree commit instead of building one from package installation directly. Your existing container registry, existing Containerfile conventions, and existing image-signing pipeline all apply unchanged to what is, underneath, a bootable operating system.
Where ostree and bootc Actually Diverge
| Raw ostree (Fedora CoreOS style) | bootc | |
|---|---|---|
| Image defined as | rpm-ostree compose treefile (custom format) |
Standard Containerfile |
| Build tooling | ostree/rpm-ostree-specific | Any OCI-compatible builder (podman, buildah, docker) |
| Registry/distribution | ostree’s own HTTP-based repo protocol, or OSTree-in-OCI | Standard container registry (Quay, Docker Hub, ECR, GHCR) |
| Deployment mechanism on disk | ostree commits, A/B deployments | Identical — ostree commits, A/B deployments |
| Rollback command | rpm-ostree rollback |
bootc rollback |
| Best fit | Teams already fluent in ostree/ Fedora tooling | Teams that want OS images to fit their existing container CI/CD |
Nothing about atomicity, rollback safety, or the deployment model changes between the two — bootc’s entire value proposition is packaging the same guarantee behind tooling most infrastructure teams already have muscle memory for.
The Part EP01 Didn’t Mention: Rollback Has a Shelf Life
“The previous deployment is always intact for rollback” (EP01’s phrasing) is true, but not indefinitely. Each deployment consumes real disk space — a full OS tree’s worth of objects, though ostree deduplicates identical objects across commits so an incremental update doesn’t cost a second full copy. Two mechanisms limit how far back you can actually roll:
Deployment count limits. Most configurations keep a bounded number of deployments (commonly 2–3). Once you’ve upgraded past that limit, the oldest deployment is pruned — rpm-ostree cleanup or an automatic policy removes it, and its objects become eligible for garbage collection if nothing else references them.
Garbage collection reclaims orphaned objects. ostree prune (or rpm-ostree cleanup -p) removes any object in the repository not reachable from a currently-kept deployment or a pinned ref. If you pruned a deployment last week and you need to roll back to it today, that commit is gone — not degraded, not slow to restore, simply no longer present.
# See exactly what's kept and what's eligible for cleanup
$ ostree admin status
fedora-coreos 38.20240210.3.0 (booted) # current
fedora-coreos 38.20240115.2.0 # one rollback available
# Pin a deployment explicitly if you need a longer-lived rollback
# target than the default retention policy provides
$ ostree admin pin 1
If your incident-response plan assumes “we can always roll back to last month’s known-good state,” verify that against your actual retention policy — the default is usually one previous deployment, not an archive.
Quick Reference
# Inspect the commit graph and current deployments
ostree admin status # deployments + which is booted
ostree log <ref> # commit history for a branch
ostree show <checksum> # inspect a specific commit
# rpm-ostree (Fedora CoreOS / Silverblue)
rpm-ostree status # current + staged, same as EP01
rpm-ostree cleanup -p # prune old deployments + GC
# bootc
bootc status # current + staged image
bootc switch <image-ref> # move to a different image
bootc upgrade # pull latest tag, stage it
bootc rollback # revert to previous deployment
Production Gotchas
“Atomic” doesn’t mean “instant.” Staging a new deployment can take as long as a full OS install — the atomicity guarantee is about the swap being indivisible, not about the whole process being fast. Budget real time for the pull-and-stage phase in maintenance windows.
Deduplication means disk usage doesn’t scale linearly with deployment count, but it isn’t free either. A kernel or major package version bump touches enough objects that “just keep 5 deployments for safety” can use more disk than teams expect. Monitor /ostree/repo size, don’t assume it’s negligible.
Pinning a deployment and forgetting about it silently defeats garbage collection. ostree admin pin is the right tool for “I need to guarantee this stays available,” but a pinned deployment never gets reclaimed automatically — audit pins periodically or disk usage grows unbounded.
bootc’s registry dependency is a new failure mode ostree-native updates didn’t have. If your container registry is unreachable, bootc upgrade fails the same way a registry-down event fails an application deployment — factor registry availability into your OS update SLA the same way you already do for app deployments.
Key Takeaways
- Every ostree deployment is a content-addressed commit, not a set of files mutated in place — that’s what makes “atomic” a filesystem property instead of a script’s promise
- The actual atomicity boundary is a single bootloader entry write; everything before it is invisible to the running system, everything after it takes effect on next boot
- bootc uses the identical deployment mechanism, but builds commits from standard
Containerfiles and distributes them through standard container registries - Rollback is real but bounded — deployment limits and garbage collection mean “always roll back” has a specific, checkable retention window, not an unlimited one
- ostree and bootc differ in build/distribution tooling, not in the safety guarantees the deployment model provides
What’s Next
EP02 covered the mechanism in the abstract. EP03 runs it day-to-day — Fedora CoreOS and Silverblue in practice: what changes about dnf install, package layering, troubleshooting, and rollback when you’re actually living on top of this model instead of reading about it.
Next: EP03 — Fedora CoreOS / Silverblue in Practice
Get EP03 in your inbox when it publishes → linuxcent.com/subscribe