11 min read

A Node.js readline silent exit happens when rl.question() never fires its callback. The interface closes first, the awaited promise stays pending forever, and the event loop drains. As a result, the process exits cleanly with status 0 — mid-prompt, no error, no stack trace. I hit this in three separate places in sarvam-cli, an MIT-licensed agentic coding CLI. The fix is eight lines. The interesting part is what each call site should do when input ends, and why “decline” is the only safe answer at an approval prompt.

Table of Contents

• TL;DR
• The Symptom: a % at the End of a Terminal Paste
• Why My First Diagnosis Was Wrong
• What Actually Causes the Node.js readline Silent Exit
• Why Exit Code 0 Is the Real Damage
• Three Call Sites, One Copy-Pasted Habit
• The Fix: Resolve to a Sentinel When the Interface Closes
• What null Should Mean at Each Call Site
• Why a Partial Write Is Worse Than No Write
• Proving It: Testing the Interactive Path in a pty
• Quick Reference
• Beyond Node: Any Callback With a Path That Never Runs
• CISSP Domain Mapping
• Key Takeaways
• Try sarvam-cli

TL;DR

• rl.question() never fires its callback if the readline interface closes first. Ctrl+D, Ctrl+C, and exhausted piped stdin all close it.
• A pending promise is not a running task. Node finds nothing scheduled, drains the loop, and exits with status 0 — so the failure presents as success.
• The fix is to resolve to null on close, not to reject. End of input is normal, not exceptional.
• What null means differs per call site: exit at a REPL prompt, decline at a consent gate, abort without writing in a config wizard.
• At an approval prompt, === "y" fails closed and !== "n" fails open. Same line count, opposite blast radius.
• Test the interactive path in a pseudo-terminal. Piping only exercises the non-TTY code path, and your users are on the other one.

The Symptom: a % at the End of a Terminal Paste

Someone sent me a session transcript from sarvam-cli. The last four lines:

❯ /model
Current model: sarvam-105b
Available: sarvam-105b
model> %

That trailing % is zsh telling you the previous command produced output with no final newline. Specifically, it only appears when zsh has regained control — which means the process exited. While sitting at a prompt. Having printed model> and then simply stopped existing.

No error. No traceback. Nothing in the logs.

Why My First Diagnosis Was Wrong

My first theory was wrong, and it is worth saying so. I assumed stdin contention. The code attached a raw process.stdin.on("data") listener for a Ctrl+O keybinding while a readline interface was consuming the same stream. Two readers, one pipe — a classic. I wrote it up confidently.

Then I reproduced it before fixing it, and the theory collapsed. The /model flow completed perfectly. Additionally, the line buffer survived a mid-line keypress intact. Whatever killed the process, it was not stdin contention.

What Actually Causes the Node.js readline Silent Exit

Here is the code every Node CLI writes to get an async prompt:

const rl = readline.createInterface({ input: process.stdin, output: process.stdout });
const ask = (q) => new Promise((resolve) => rl.question(q, (a) => resolve(a)));

const name = await ask("Your name: ");

rl.question() registers a one-shot callback for the next line of input. That is the whole contract. Consequently, there is exactly one way for it to never be called: the interface closes before a line arrives.

Three ordinary things close it:

• Ctrl+D — EOF at the terminal
• Ctrl+C — readline’s default SIGINT behaviour closes the interface
• Piped stdin running out of lines — printf 'a\nb\n' | sarvam when the CLI asks five questions

When any of those happen, the callback is dropped. The promise attached to it stays pending. await never returns.

Now the part that turns a bug into a silent bug. A pending promise is not a running task. Node does not wait on it, because nothing is scheduled — no timer, no socket, no handle. The event loop finds itself with nothing to do, and does the correct thing:

It exits. Status 0.

From the outside, your program reported success. From the inside, it never finished the line it was on.

$ printf '\n\n\n' | sarvam --init
  sarvam init
  ----------------

Provider [a/b] (default: a): API key: $ echo $?
0

Success. Nothing written.

Why Exit Code 0 Is the Real Damage

A crash is loud. You get a stack trace, a non-zero status, and CI goes red. This is the opposite. It is a false success, and false success is the failure mode that survives longest in production, because nothing is watching for it.

That gap between “the process succeeded” and “the product worked” is exactly the product quality vs code quality split — no test suite in this repo was wrong, and every one of them passed.

Concretely, in my case:

sarvam --init && echo "configured" && deploy.sh

--init exits 0 having written no config file. However, the && chain proceeds anyway. Then deploy.sh runs against a machine that was never configured. The one signal a shell script has for “did this work” was lying.

Three Call Sites, One Copy-Pasted Habit

Once I understood the mechanism, I grepped for the shape rather than the symptom — rl.question wrapped in a new Promise. It appeared three times.

1. The main REPL prompt. Ctrl+D exited silently. In practice, this is the one everyone would eventually notice and shrug at, because “Ctrl+D quits” looks like intended behaviour when the process disappears.

2. A nested sub-prompt. The one in the transcript. Identical cause, more visible, because it left a half-drawn prompt on screen.

3. The --init configuration wizard. The worst of the three, and the one nobody would have found by hand. Specifically, it only misbehaves under piped stdin — which is exactly how CI and setup scripts drive it.

Same eight lines, copy-pasted, three times. That is the honest reason it appeared three times: not three bugs, one habit.

The Fix: Resolve to a Sentinel When the Interface Closes

Resolve to null when the interface closes, so every caller gets a value instead of hanging:

let closed = false;
rl.on("close", () => {
  closed = true;
});

const ask = (q: string): Promise<string | null> =>
  new Promise((resolve) => {
    if (closed) return resolve(null);

    let answered = false;
    const onClose = () => {
      if (!answered) {
        answered = true;
        resolve(null);
      }
    };

    rl.once("close", onClose);
    rl.question(q, (a) => {
      answered = true;
      rl.removeListener("close", onClose);
      resolve(a);
    });
  });

Three details matter more than they look:

• resolve(null), not reject(). End of input is not exceptional. It is the normal way a pipe finishes and the normal way a user quits. Rejecting forces every call site into a try/catch and tempts people to swallow it.
• The answered guard. Without it, a close event firing after a legitimate answer double-resolves. That is harmless with promises, but it hides ordering bugs.
• removeListener on the success path. A long REPL session asks hundreds of questions. Without this you accumulate a close listener per prompt, and Node starts warning you about a leak around 11.

What null Should Mean at Each Call Site

This is where a mechanical fix becomes a design decision. null means “input ended.” What you should do about that differs at every call site. Furthermore, getting it wrong at one of them is a security bug.

At the main prompt — exit cleanly.

const input = await ask("❯ ");
if (input === null) break;   // fall through to the normal shutdown path

At an approval prompt — decline. Always.

const ans = await ask(`▸ ${tool}: ${summary} [y/N] `);
if (ans === null) return false;  // stdin closed — never assume consent
return ans.toLowerCase().trim() === "y";

This is the one that matters. sarvam-cli is an agentic assistant that can run shell commands and write files, gated behind a [y/N] prompt. Therefore, “input ended” must never be read as approval.

Note how easily it goes the other way. Had the original been written as a denial check:

return ans.toLowerCase().trim() !== "n";   // looks equivalent. isn't.

…then an empty or absent answer becomes approval, and a closed stdin auto-approves every pending action. Same number of lines, opposite blast radius. Fail-closed is not a philosophy here. It is a comparison operator — and it is the most concrete example of cybersecurity architecture principles I have shipped in one line of code.

Why a Partial Write Is Worse Than No Write

In the config wizard — abort without writing.

const answers: string[] = [];
for (const q of questions) {
  const a = await ask(q);
  if (a === null) {
    rl.close();
    console.error("\n  init aborted — input ended before every question was answered.");
    console.error(`  Nothing written to ${CONFIG_PATH}.\n`);
    return null;   // caller exits non-zero
  }
  answers.push(a);
}

The tempting alternative is to save whatever you collected. Don’t. In this codebase a partial config with an empty apiKey field is worse than no file at all, because of a second bug it interacts with:

// The config file wins over the environment...
const apiKey = file.apiKey ?? process.env.API_KEY ?? "";

?? only falls through on null/undefined. An empty string is a defined value. As a result, a config file containing "apiKey": "" permanently shadows the environment variable. Export API_KEY all you like — the CLI reports “no API key found” forever, and the file causing it looks empty and harmless.

A partial write turns a clean failure into a persistent one. No write beats a bad write.

On the caller side, actually surface it:

if (args.init) {
  const created = await initConfigInteractive();
  process.exit(created ? 0 : 1);
}

Proving It: Testing the Interactive Path in a pty

You cannot test this properly with a pipe. readline behaves differently when stdin is not a TTY, so piping only exercises one of the two paths — and the interactive path is the one your users are on.

The one-liner smoke test is what I would add to CI first:

$ printf '\n\n\n' | sarvam --init; echo "exit=$?"
  init aborted — input ended before every question was answered.
  Nothing written to /home/vamshi/.sarvam/config.json.
exit=1          # ← was 0 before the fix. Non-zero is the whole point.

For the interactive path, drive a pseudo-terminal. Additionally, this needs no new dependencies — it is Python’s stdlib:

#!/usr/bin/env python3
"""Send Ctrl+D at a prompt and assert the CLI exits like it means it."""
import os, pty, subprocess, time

master, slave = pty.openpty()
p = subprocess.Popen(["sarvam"], stdin=slave, stdout=slave, stderr=slave, close_fds=True)
os.close(slave)

time.sleep(1.0)            # let it draw its prompt
os.write(master, b"\x04")  # Ctrl+D  (use b"\x03" for Ctrl+C)
time.sleep(1.0)

print("exit:", p.wait(timeout=5))

Running it against the fixed build:

$ python3 test_eof.py
exit: 0         # ← clean shutdown, farewell line printed
                #   before the fix this also printed 0 — but with no farewell

That distinction is the whole test. Together, the exit code and the farewell line tell you which of the two happened. This is what continuous security validation looks like at CLI scale: assert the observable behaviour, not just the status.

Quick Reference

Expected behaviour after the fix:

Use 130 for SIGINT. That is the 128 + signal convention, and the only way a wrapping script can tell “the user interrupted this” from “this finished.” However, it needs an explicit handler, because readline’s default is a silent close:

let interrupted = false;
rl.on("SIGINT", () => {
  interrupted = true;
  rl.close();
});
// …after the loop:
if (interrupted) process.exitCode = 130;

Beyond Node: Any Callback With a Path That Never Runs

The specific API is Node’s. However, the pattern is everywhere: an async primitive whose completion callback has a path that never runs.

Any time you wrap a callback API in a promise, ask the same question — what are all the ways this callback might not be called? Closed streams, cancelled requests, timed-out sockets, aborted signals. In every one of those cases, a bare new Promise(resolve => api(cb)) becomes a permanent hang. Moreover, in an event-loop runtime, a permanent hang looks exactly like a clean exit.

The tell is a process that ends without printing whatever it normally prints on the way out. If your CLI has a farewell line, a summary, or a flush, its absence is your signal — not the exit code, which is lying.

CISSP Domain Mapping

Key Takeaways

1. rl.question() never fires if the interface closes first. Ctrl+D, Ctrl+C, and exhausted piped stdin all close it.
2. A pending promise is not a running task. Node exits cleanly when the loop empties, so the failure presents as success.
3. Resolve to a sentinel, don’t reject. End of input is normal, not exceptional.
4. Decide what “input ended” means per call site. Exit at a prompt, decline at a consent gate, abort at a wizard.
5. Never assume consent from absent input. Write === "y", never !== "n".
6. A partial write can be worse than no write — especially where an empty string is a meaningful, shadowing value.
7. Test the interactive path in a pty. A pipe tests the other code path entirely.
8. Reproduce before you fix. My confident first diagnosis was wrong, and only a reproduction attempt caught it before it became a wasted refactor.

Try sarvam-cli

The CLI in this post is sarvam-cli — an MIT-licensed, open-source agentic coding assistant powered by Sarvam AI. It reads, writes, and edits files and runs shell commands in your project, with your approval before any side effect. That approval gate is exactly the one discussed above, which is why the fail-closed behaviour mattered enough to write up.

git clone https://github.com/indic-ai-contribs/sarvam-cli.git
cd sarvam-cli
npm install
npm run build
npm link

sarvam --init     # exits non-zero now if you don't finish the wizard

The fixes described here shipped in v0.2.9 and v0.2.10. Issues and pull requests are welcome — particularly from anyone who has fought the same class of bug in their own CLI. If the project is useful to you, a star on the sarvam-cli GitHub repo genuinely helps it reach more Indian-language AI developers.

Get the next deep-dive in your inbox when it publishes → subscribe to linuxcent.com

The post Exit Code 0 Lied: The Silent Node.js Bug That Hit sarvam-cli Three Times appeared first on Linuxcent.

9 min read

EP01: Product Quality vs Code Quality · All The Legible Repo Episodes →

This series is about the quality layer your CI can’t see. Each episode takes one failure that linters, scanners, and test suites never catch, shows the incident that proves it, and ends with one command you can run today. This opener names the problem — and introduces the gate that measures it.

Table of Contents

• TL;DR
• Quick Check: What Grade Is Your Repo Right Now
• The 500 Nobody Reported
• Where Product Quality Sits (and Why Linters Can’t See It)
• The Questions That Decide Whether a Stranger Stays
• How the Gate Works: Scorecard Plus Cold-Start
• What This Means for Your Repos Right Now
• Production Gotchas
• Quick Reference
• Framework Alignment
• Key Takeaways
• What’s Next

TL;DR

• Product quality vs code quality is the gap between “the tests pass” and “a stranger can actually use this” — and no linter measures it.
• The costliest failures are silent: the person who hits friction in the first ten minutes never files an issue. They close the tab.
• Legibility is checkable: README length, copy-paste quickstart, .env.example, actionable errors, a published artifact that still installs today.
• Invigil (Apache-2.0) mechanizes ~35 of these checks into gate levels G1–G7 with a letter grade — and prints the exact fix for every failure.
• It grades itself in CI: a pull request that lowers Invigil’s own score doesn’t merge.

Quick Check: What Grade Is Your Repo Right Now

Before the story, the evidence. Two commands, two minutes, on any repo you maintain:

pip install invigil
invigil score . --offline

Sample output, annotated:

Invigil — myproject
Gate G2 · Grade C+ · 19/27 (70%)          ← gate = maturity rung, grade = weighted score

FAIL [G1] README is a landing page (≤300 lines)     (effort: minutes)
      fix: move deep-dive sections to docs/; keep quickstart + pitch
FAIL [G1] .env.example documents every config var   (effort: minutes)
      fix: create .env.example listing each var with purpose + default
FAIL [G2] Errors carry a correlation ID             (effort: hours)
      fix: add a global exception handler returning {"error_id": ...}

Every failing line names the check, the effort class, and the exact fix. However you feel about the individual opinions, notice what just happened: nothing in your existing CI produces this view.

The 500 Nobody Reported

The day I renamed a package and pushed the new wheel, every UI page it served returned a 500. The commit message said “verified.” I had verified the import — not the experience. No test caught it, because the tests ran against my source tree, not against the artifact a stranger downloads. And no user caught it for me. The first stranger who hit that 500 did what strangers do: closed the tab and never came back.

That is the failure mode that should keep maintainers up at night. Absence of complaints is not absence of problems. Silence is the loudest negative signal a project gets.

A clean-virtualenv install from an empty directory found the bug in minutes. That habit — being your own first angry user — became a doctrine. Later, the doctrine became a CI gate called Invigil, because habits don’t run nightly and machines do.

Where Product Quality Sits (and Why Linters Can’t See It)

                    ┌─────────────────────────────────────────┐
                    │        WHAT YOUR CI CHECKS TODAY        │
                    │  ruff / eslint      → code style        │
                    │  pytest / jest      → source behavior   │
                    │  Trivy / Dependabot → CVEs, deps        │
                    │  Scorecard          → supply chain      │
                    └────────────────┬────────────────────────┘
                                     │  all green ✅
                                     ▼
                    ┌─────────────────────────────────────────┐
                    │        WHAT THE STRANGER MEETS          │
                    │  README (landing page or wall of text?) │
                    │  Quickstart (works from empty dir?)     │
                    │  Published artifact (installs TODAY?)   │
                    │  First error (fix included or trace?)   │
                    │  llms.txt / AGENTS.md (agent-readable?) │
                    └─────────────────────────────────────────┘
                          nothing above checks this layer

The product quality vs code quality distinction is exactly this diagram. As a result, a repo can be immaculate in the top box and unusable in the bottom one — green CI, linted code, zero CVEs, and a quickstart that fails on the first copy-paste. In contrast to code quality, product quality has no reflexive tooling. Every good maintainer checks these things by hand, occasionally, when they remember. Nobody’s CI does it on every pull request.

I build hardened infrastructure for a living, and the same lesson repeats there: a standard that isn’t enforced mechanically is a wish. That’s why Linux hardening as code beats hardening runbooks — and it’s why legibility needs a gate, not a checklist.

The Questions That Decide Whether a Stranger Stays

Specifically, the gate asks the questions your CI never asks:

• Can someone get from “found the repo” to “it worked on my machine” in ten minutes?
• When something fails, does the error include the fix — or a traceback?
• Is the README a landing page, or 600 lines of accumulated documentation?
• Does the artifact you published still install today, after your dependencies drifted?
• Is there an .env.example, or do users reverse-engineer your config from source?
• Can an AI agent — now often the first reader — parse your llms.txt and AGENTS.md without hitting stale paths or a leaked key?

Each question maps to a mechanical check. Together, ~35 checks roll up into gate levels G1–G7 — a maturity ladder, not a binary pass/fail — plus a weighted letter grade. A repo reaches gate Gn only when every mandatory check at or below n passes.

How the Gate Works: Scorecard Plus Cold-Start

Layer 1 — the scorecard (every PR, seconds)

The static layer inspects the repo and its metadata: LICENSE, README length, quickstart shape, tracked secrets, SHA-pinned actions, enforced lockfile, coverage floor, docs index, llms.txt/AGENTS.md hygiene, and more. It runs offline in a pre-commit hook in roughly 120 ms, because a gate that adds friction is a gate that gets uninstalled.

invigil score . --format markdown   # PR-comment-ready table

Layer 2 — the cold-start gate (nightly)

This is the layer that would have caught my 500. Instead of testing the source tree, it boots the published artifact — the wheel on PyPI, the image on GHCR — on a clean runner and probes its surface within a ten-minute budget:

# .invigil.yml
artifacts:
  - { type: pypi, name: "myapp[all]" }
  - { type: ghcr, image: ghcr.io/me/myapp:latest, port: 8000 }
probes:
  - { url: "/", expect_status: 200 }

Because it installs from the real registry into a real empty environment, it catches the class of bug where CI passes but the shipped thing is broken: the missing template directory, the config default pointing at localhost, the dependency that resolved differently after an upstream release.

What This Means for Your Repos Right Now

Start in report-only mode. The progressive profile scores everything and gates nothing — you get the visibility without a wall of red blocking your next merge. Flip to enforce once the grade stabilizes, the same way you’d introduce any merge check.

The doctrine is opinionated, and that’s deliberate — but the gate bends instead of breaking. Profiles (strict | progressive | light), per-check weights, and optional flags let a team disagree with a specific opinion without forking the tool. Additionally, network-dependent checks that time out become SKIPs excluded from the grade — never a false downgrade that erodes trust in the number.

One more thing, because trust matters for a tool that grades others: Invigil grades itself in CI. A pull request that lowers its own score won’t merge. The gate passes its own gate — currently G5, grade A+.

Production Gotchas

Enforcing on day one. Turning on enforce: true before the team has seen the report produces a wall of failures and an uninstall. What breaks: adoption. How to detect it: grumbling in your PR comments. The fix: progressive first, enforce after two weeks of stable grades.

Treating the grade as the goal. The grade is a proxy for a stranger’s first ten minutes. Gaming it (a hollow .env.example, a README split that hides the quickstart) passes the check and still loses the user. The fix is cultural, not mechanical — review the fix, not just the score delta.

Skipping the cold-start layer because “CI already tests installs.” CI installs from the source tree with your lockfile present. The stranger installs from the registry into nothing. These diverge silently after any packaging change — that divergence is invisible until you test the published artifact itself.

Quick Reference

Framework Alignment

Key Takeaways

• Product quality and code quality are different layers; your CI only watches one of them.
• Test the experience, not the import — the developer’s machine is a lie, and so is the source tree.
• Silence is data: the users you never hear from are the ones who hit the friction.
• A quality bar you can’t measure is an opinion; a gate you bypass is dead weight — make it fast, bendable, and report-only by default.
• Trust tools that hold themselves to their own standard: Invigil’s own PRs merge only if its self-grade holds.
• Defaults are never neutral — the same reason cloud AMI security risks demand custom images applies to your repo’s out-of-the-box experience.

What’s Next

EP02 goes deep on the layer that caught my 500: testing the published artifact, not the source tree. Clean-runner boots, real-registry installs, probe budgets — and why “it works in CI” is a statement about your lockfile, not your users. EP02: How to Test Your Published PyPI Package — Before a Stranger Does.

Invigil is Apache-2.0 and built in the open. If this episode named a failure you’ve shipped (we all have), there are more checks waiting to be written — good-first-issues with acceptance criteria at github.com/invigil/invigil. Pick one, or open a Discussion and say hello. First-time contributors get fast reviews and release-notes credit.

Get EP02 in your inbox when it publishes → subscribe

The post Product Quality vs Code Quality: Why Your Green CI Still Loses Users appeared first on Linuxcent.

OS Hardening as Code, Episode 6
Cloud AMI Security Risks · Linux Hardening as Code · Multi-Cloud OS Hardening · Automated OpenSCAP Compliance · CI/CD Compliance Gate · BakeX Platform**

Note: this series was written when the project was called Stratum. It was renamed to
BakeX at v0.6.0 — same project, same Apache 2.0 license, same team. The old
github.com/rrskris/Stratum URL redirects here, and pip install stratumoss is retired in
favour of pip install bakex. Current home:
github.com/invicton/bakex.

TL;DR

• BakeX is open-source under Apache 2.0 — the engine, blueprint format, scanner, and Pipeline API are all in the repository
• Self-hostable end to end: nothing is locked to a hosted service, and there is no paid tier gating the pipeline
• Two real extension points: provider plugins (drop-in .py or a bakex.providers entry point) and blueprints (pure YAML, no code)
• Architecture: Blueprint YAML → Engine → Provider Layer → Ansible-Lockdown → OpenSCAP → Golden Image → Pipeline API
• The series taught the user-facing interface for five episodes; EP06 covers what’s underneath and how to build on it
• Installation is git clone + docker compose up, or pip install bakex for the CLI and web app

The Series Arc, Inverted

EP01 showed that default cloud AMIs arrive pre-broken. By the time you reach EP06, that problem has a complete solution:

EP01 — The problem:
  Default AMI → Production → Security audit finds gaps
  (unknown OS baseline, unverified hardening, no evidence)

EP06 — The solution:
  HardeningBlueprint YAML
           ↓
    bakex validate          ← EP02 (blueprint as code)
    bakex build             ← EP02
      one file per provider ← EP03 (multi-cloud)
           ↓
    OpenSCAP scan           ← EP04 (compliance grading)
    Grade: A (score 94)
           ↓
    POST /api/pipeline/scan ← EP05 (CI/CD gate)
    passed: true
           ↓
    Production deployment
    (Grade A, SARIF attached, blueprint version-controlled)

For five episodes, you’ve used BakeX as a user. This episode covers what it looks like to run it yourself, extend it, and build on it.

I’ve spent years watching infrastructure teams solve the same OS hardening problem in slightly different ways. Custom scripts that drift. OpenSCAP runs that produce evidence no one reads. Compliance checklists completed by humans who have competing priorities.

The tools exist. ansible-lockdown applies CIS controls reliably. OpenSCAP verifies them accurately. The CI/CD systems can enforce anything you can express as a pass/fail. The gap isn’t the tooling — it’s the integration layer that ties them together into a reproducible, auditable pipeline.

BakeX is that integration layer, open-sourced.

The philosophy is the same as Terraform applied to OS security posture: declare the desired state in a version-controlled file, apply it reproducibly, and verify it automatically. The skip-at-2am problem disappears not because engineers are more careful, but because there’s no step to skip.

The Architecture

┌─────────────────────────────────────────────────────────┐
│                 HardeningBlueprint YAML                  │
│         (version-controlled, provider-agnostic)          │
└─────────────────────┬───────────────────────────────────┘
                      │
                      ▼
┌─────────────────────────────────────────────────────────┐
│                    BakeX Engine                          │
│                  (Apache 2.0, OSS)                       │
│  ┌─────────────┐  ┌──────────────┐  ┌────────────────┐  │
│  │  Blueprint  │  │   Provider   │  │    Scheduler   │  │
│  │   Parser    │  │    Layer     │  │  (parallel     │  │
│  │             │  │  AWS  GCP    │  │   multi-cloud  │  │
│  │  Validates  │  │  Azure DO    │  │   builds)      │  │
│  │  schema +   │  │  Linode      │  │                │  │
│  │  overrides  │  │  Proxmox     │  │                │  │
│  └─────────────┘  └──────────────┘  └────────────────┘  │
└─────────────────────┬───────────────────────────────────┘
                      │
           ┌──────────┴──────────┐
           ▼                     ▼
  ┌─────────────────┐   ┌─────────────────┐
  │ Ansible-Lockdown │   │  OpenSCAP       │
  │  Runner          │   │  Scanner        │
  │                  │   │                 │
  │  UBUNTU22-CIS    │   │  A-F grade      │
  │  RHEL8-STIG      │   │  SARIF export   │
  │  Custom roles    │   │  Drift detect   │
  └────────┬─────────┘   └────────┬────────┘
           │                      │
           └──────────┬───────────┘
                      │
                      ▼
         ┌─────────────────────────┐
         │   Golden Image          │
         │   (AMI / GCP / Azure)   │
         │   + compliance metadata │
         └────────────┬────────────┘
                      │
                      ▼
         ┌─────────────────────────┐
         │   Pipeline API          │
         │   (Apache 2.0, OSS)     │
         │                         │
         │  POST /api/pipeline/scan │
         │  ← CI/CD gate           │
         └─────────────────────────┘

Every component is open-source under Apache 2.0. The engine, provider layer, Ansible runner, OpenSCAP scanner, and Pipeline API are all in the repository. Nothing is locked to a hosted service.

Installation

Three ways in, depending on how much you want installed on the host.

Docker Compose — recommended, everything preinstalled:

git clone https://github.com/invicton/bakex.git
cd bakex
docker compose up

Open http://localhost:8001. Log in with any username and the admin token as the password —
it’s generated on first start and written to data/.admin_token. Set BAKEX_ADMIN_TOKEN and
BAKEX_SECRET_KEY in docker-compose.yml if you want logins that survive a rebuild.

Compose mounts ~/.aws, ~/.config/gcloud, and ~/.ssh read-only, plus persistent ./data,
./profiles, and ./plugins/providers. That last mount is the one to notice — it’s the
drop-in directory for provider plugins, which matters in the next section.

Published image:

docker run -p 8000:8000 rrskris/bakex:latest

PyPI — CLI and web app:

pip install "bakex[all-providers]"   # or pick extras: aws, gcp, azure, linode, digitalocean, proxmox
bakex serve --port 8000

One caveat worth stating plainly rather than letting you discover it: the extras install each
provider’s cloud SDK, and Ansible and OpenSCAP must be present on the host for real builds. If you
want the batteries-included path, use Compose. bakex validate works anywhere with no host
dependencies at all.

There is no Helm chart. BakeX is a build tool that talks to cloud APIs, not a cluster workload —
it does not need to live in Kubernetes to harden images for it.

The Three Extension Points

1. Blueprints — the extension point with no code in it

The highest-leverage way to extend BakeX isn’t Python. It’s a YAML file.

A blueprint is a complete, self-contained description of a hardened OS on a specific provider,
and the library ships 18 of them. Adding the nineteenth — say Ubuntu 24.04, or CIS Level 2 for a
distro that only has Level 1 today — requires no engine changes, because the benchmark, profile,
and datastream are just strings handed to oscap.

The full format is published as a JSON Schema (Draft 2020-12) at
docs/schema/hardening-blueprint.schema.json. Point your editor at it for autocomplete and
inline validation, or hand it to an LLM and let it draft the blueprint — the schema was published
partly so that agents could write these correctly without reading the source.

The loop is short enough to run in a coffee break:

$EDITOR blueprints/ubuntu/24.04/cis-l1-aws.yaml
bakex validate blueprints/ubuntu/24.04/cis-l1-aws.yaml

Validation is offline and checks more than syntax — it rejects OS/provider combinations the
catalog doesn’t support, so you find out that a distro isn’t available on your target cloud in
milliseconds rather than fifteen minutes into a paid build.

2. Provider Plugins

Adding a cloud means implementing four methods. That’s the whole interface
(bakex/plugins/base_provider.py):

# plugins/providers/my_cloud.py
from bakex.plugins.base_provider import BaseProvider, ProviderResult
from bakex.core.models import ComplianceProfile

class MyCloudProvider(BaseProvider):
    name = "my-cloud"          # matches target.provider in a blueprint

    def provision(self, profile: ComplianceProfile, **kwargs) -> str:
        """Launch a build instance; return its instance ID."""
        ...

    def run_ansible(self, instance_id: str, profile: ComplianceProfile) -> None:
        """Apply the Ansible-Lockdown hardening roles."""
        ...

    def snapshot(self, instance_id: str, profile: ComplianceProfile) -> ProviderResult:
        """Capture the golden image; return the artifact ID."""
        ...

    def teardown(self, instance_id: str) -> None:
        """Destroy the ephemeral build instance."""
        ...

There is no registration command. The loader (bakex/plugins/loader.py) is hybrid and finds
plugins two ways:

1. Drop-in — put the .py file in plugins/providers/. That directory is a Compose volume
   mount, so a plugin dropped there is live in the container without rebuilding an image.
2. Entry point — ship a pip-installable package declaring a bakex.providers entry point.
   This is how a third party distributes a provider without touching the BakeX repo.

Entry points load first and drop-ins load second, so a local file deliberately shadows an
installed package of the same name — which is exactly what you want when debugging someone
else’s provider.

The plugin becomes usable by writing provider: my-cloud in a blueprint’s target block. There
is no --provider flag to pass, because there is no --provider flag anywhere.

One honest note on the validation interaction from EP02: the compatibility check only objects
when both the OS and the provider are in the catalog. An unknown provider is assumed to be a
valid third-party plugin rather than an error — existence is the plugin registry’s call at build
time, compatibility is validation’s. That’s what makes shipping a provider out-of-tree possible
at all.

3. Pipeline Integrations

Beyond the curl-based gate from EP05, BakeX has a webhook system. Webhooks are registered through
the API rather than a config file, so they can be managed by the same automation that manages
everything else:

curl -X POST http://localhost:8001/api/webhooks \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://hooks.slack.com/services/…",
    "events": ["build.failed", "scan.grade_change"],
    "label": "platform-security alerts"
  }'

Five events fire: build.complete, build.failed, scan.complete, scan.failed, and
scan.grade_change. Registering an unknown event name is a 422 rather than a silent no-op —
a small thing that saves an afternoon.

scan.grade_change is the one to wire up first. A build that fails is loud on its own; a base
image that quietly slid from A to B between two scans is the signal nobody notices.

Deliveries are signed. The registration response returns a secret once, and every request
carries an X-BakeX-Signature: sha256=… HMAC so the receiver can verify the payload came from
your BakeX instance and not from anyone who guessed the endpoint URL.

There’s a defensive detail here that’s worth calling out, because it’s the kind of thing that
usually ships broken: webhook target URLs are checked against loopback, private, link-local, and
reserved ranges — including 169.254.169.254 — and they’re re-resolved at send time, not just
at registration. A user-configurable URL that the server will fetch is a textbook SSRF into the
cloud metadata endpoint, and on a tool that holds cloud credentials that would be a very bad day.

The Open-Core Model

BakeX sits alongside the tools that became infrastructure standards by being genuinely usable
before they were commercial:

Everything taught in this series — the blueprint format, the build pipeline, the compliance
grading, the CI/CD gate — is in the repository. There is no feature held back, because there is
currently nothing to hold it back for. Self-host it, extend it, fork it.

Worth being straight about where the project actually is: BakeX is young. It has signed releases,
SBOMs and provenance attestations, an OpenSSF Scorecard, a published JSON Schema, and over a
thousand tests — the engineering is in good order. What it does not yet have is users. If you’re
reading this and the shape of the tool fits your problem, you would be early, and early is when
your opinion changes the design.

The repository is at: github.com/invicton/bakex

What This Series Taught

EP01 — EP06 in one view:

What’s Next

This series closes the OS hardening gap. The same principle — declare desired state, build
reproducibly, verify automatically — applies to every layer of your infrastructure.

Write the next blueprint

The most useful thing you can do with what this series taught is add a blueprint, and it is
genuinely pure YAML — no Python, no engine changes, no build system to learn.

You’ve spent five episodes on Ubuntu 22.04 CIS Level 1. The natural next one is Level 2 for the
same OS: #1 — Ubuntu 22.04 CIS Level 2. The
issue carries the acceptance criteria and the exact verify command, and the review loop is
bakex validate returning 0.

If a different distro is closer to what you actually run, the whole set is filed and labelled:
good first issues, blueprint label.
RHEL 9, AlmaLinux 9, Rocky 9, Debian 12, and Amazon Linux 2023 all have gaps. Each one is one
file, and each is the sort of contribution that takes an evening.

GitHub: github.com/invicton/bakex

Elsewhere on the blog

If you’ve been following the eBPF: From Kernel to Cloud series,
EP10 covers what happens when you combine kernel-level observability with the hardened base BakeX
produces: every connection, every process spawn, every file access — visible from the host kernel,
on an OS baseline you can verify.

The next series is the Purple Team Playbook — real attack paths against cloud and Kubernetes
infrastructure, how they’re detected, and how they’re closed.

Get new episodes in your inbox → linuxcent.com/subscribe

The post BakeX — OS Hardening as a Platform appeared first on Linuxcent.