kibble

Dogfood your docs.

Verify your documentation in CI. Runs install instructions and examples in a clean container, then checks documented commands and flags against the installed tool.

_{VERIFIED · FAIL · GAP · DRIFT · BLOCKED · SKIP}

myrepo
  ! brew             0s  GAP  no formula, cask, or alias named "acme/tap/nope" in the c…
  – doc-coverage     0s  SKIP  no command list in the help output to check against
  ✗ example         22s  FAIL  b1 "mytool --nosuchflag" exited 2: flag provided but not d…
      myrepo/README.md:17
      $ mytool --nosuchflag
        exited 2: flag provided but not defined: -nosuchflag
  ✓ flag-check       0s  2 cited flags ok, 0 subcommands cited
  ✓ go-install      22s  v1.4.0

FAILED  a documented line ran and did not work
2 passed  1 failed  1 gap  1 other  5 checks in 45s

kibble does not check whether your documentation exists. It checks whether the interface your documentation promises still exists. It reads your README, everything in docs/, and the instructional documents beside them, runs what they tell a reader to run in a clean Linux container from zero, and puts the flags and subcommands they cite to the binary it just installed. A page still teaching a flag the tool dropped is caught as drift, even though nothing failed to run.

What comes back is a verdict, not a pass. kibble separates documentation that is wrong from documentation it could not check, so a green run means something specific rather than nothing went visibly bang. That distinction is pinned in both directions: a corpus of real repositories proves correct documentation passes, and deliberate one-line corruptions of those same documents prove rot gets caught. A verifier that only ever agrees with you is worse than no verifier, so kibble is tested against its own willingness to say yes.

What rots, and who reads it

The stranger is not always a person now. Coding agents install tools by doing what the README says, and they fail differently than people do. Someone who follows a broken instruction knows they followed it correctly, reads the error, and works around the document. An agent cannot tell a stale command from its own mistake, so it retries, invents variants, and reports success it did not have. A line that has been wrong for six months gets run all day by something that will never complain about it. Stale documentation used to cost a reader ten minutes. Now it is instructions a machine executes without ever telling you they were wrong.

Your README tells people to run go install ..., then some setup, then a quickstart. Every one of those rots the moment the code moves, and you are the last to know. kibble reads those steps from the README and from everything instructional beside it: a docs tree, a getting-started, an install guide, a tutorial, a cookbook, an FAQ. Those rot faster than the front page, because they are where instructions go once it fills up and nobody reads them on the way past. Documents that describe the project rather than instruct a reader are left alone, since running a contributing guide's test commands proves nothing about the docs a user follows. A piped shell installer and a system package install are recorded but not run, and reported as skips with a reason, so an install kibble declines to execute is still visible rather than silently missed.

Commands are not the only thing that rots. kibble takes the flags and subcommands your documents cite and puts them to the binary it just installed, so a page still teaching a flag the tool dropped is caught as drift rather than passing because nobody happened to run that line. It also reads the binary's own help and reports the documented surface against the real one.

That container is one environment, not every environment. kibble answers whether your documentation works from zero in a reproducible Linux container, which is a narrower claim than whether it works for every reader: a macOS path, an ARM host, musl, a corporate proxy, and a private registry are all outside what it sees. The narrow claim is the one worth making, because it is the one kibble can actually settle.

Install

brew install dcadolph/tap/kibble

Or, with a Go toolchain:

go install github.com/dcadolph/kibble@latest

Requires Docker on the host. A drop-in replacement that speaks docker's command line, such as Podman, works through KIBBLE_DOCKER=podman.

Usage

Point it at one or more repository directories, or none at all:

kibble
kibble ./myrepo
kibble ./repo-a ./repo-b

With no path it checks the directory you are standing in, so cd into a project and run it. There are no prompts. kibble's home is a CI job, and a tool that stops to ask a question there either hangs on a closed pipe or needs a flag to defeat it.

A run that finds something says where it is:

myrepo
  ! brew             0s  GAP  no formula, cask, or alias named "acme/tap/nope" in the c…
  – doc-coverage     0s  SKIP  no command list in the help output to check against
  ✗ example         22s  FAIL  b1 "mytool --nosuchflag" exited 2: flag provided but not d…
      myrepo/README.md:17
      $ mytool --nosuchflag
        exited 2: flag provided but not defined: -nosuchflag
  ✓ flag-check       0s  2 cited flags ok, 0 subcommands cited
  ✓ go-install      22s  v1.4.0

FAILED  a documented line ran and did not work
2 passed  1 failed  1 gap  1 other  5 checks in 45s

The file and line under a failure are the point. A verdict a reader has to go hunting for in a log costs them the search, so the broken line, where it is written, and what the tool said are printed together. In CI the same three become an annotation on that README line.

A session the container ends early is an ERROR with no verdict on its examples, and the lines it never reached are BLOCKED. A file a document reads that no earlier line is seen to create is checked at run time, just before the line that reads it.

Use it in CI

Add a workflow that fails a pull request when a documented install breaks:

name: docs
on: pull_request
jobs:
  kibble:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: dcadolph/kibble@v0.27.2
        with:
          repo: .
          # args: -strict   # fail on timeouts, smoke failures, drift, and gaps too

The runner already has Docker. A failed install or example is annotated on the exact README line that broke, so it shows up inline in the pull request the way a failing test does; doc drift becomes a warning annotation, and the job summary gets the full results table. kibble runs on its own README this way on every commit.

Pin an exact release tag so a run stays reproducible a year from now: the action downloads the released binary for that version and verifies its checksum before running it, so a CI log can always say which kibble produced its verdict.

The exit code tells a script what happened:

• 0 means at least one check ran and none failed.
• 1 means the run failed. -strict widens what counts as a failure.
• 2 means a usage error, such as no path and no README in the current directory, Docker unavailable, or less than 2 GiB free in Docker.
• 3 means nothing was checked: every row is a skip, so no install ran and no example could run either. The action fails on purpose, since a check that ran nothing must not pass.
• 130 means the run was interrupted.

Flags

Flag	Default	What
-image	golang:1.26	Fallback image when no toolchain is detected.
-timeout	240s	Per-step build timeout.
-workers	3	Max concurrent installs.
-json	false	Emit results as JSON to stdout.
-triage	false	After the table, list every non-working result in full.
-version	false	Print the version and exit.
-strict	false	Also fail on timeouts, smoke failures, drift, and gaps.
-examples	true	Replay each document's example blocks in the container.
-plan	false	Print the example plans as JSON and exit.
-suggest	false	Propose a .kibble.yml using a model and exit.
-mcp	false	Serve the Model Context Protocol over stdio.
-brew-install	false	Run documented brew installs for real instead of checking the formula exists.

One dash or two, either works: -strict and --strict name the same flag.

What the verdicts mean

kibble runs installs from zero, smoke-tests what lands, replays quickstarts in one session, and checks cited flags and subcommands against the binary's own help. Every result is one verdict, and the boundaries between them are the product. Only one verdict claims the documented step works:

• VERIFIED built or installed and the binary answered a smoke test. It is the only green verdict.
• BUILT, RAN, EXISTS, and CROSS-ARCH all mean a step ran or a package exists but nothing proved it works: BUILT compiled but the smoke test failed, RAN finished but produced no binary to test, EXISTS confirms a package is in its index without installing it, and CROSS-ARCH installed a binary for another architecture this host cannot run. None of them is a pass, and none is an accusation.
• FAIL ran and did not work, or named a package that does not exist.
• SKIP means kibble chose not to run the line, before running it, on evidence it holds: a placeholder to fill in, a block scoped to another platform, a command the shell reported missing. A machine-readable reason says which.
• BLOCKED means kibble ran the line and could not establish whether it works. The output resembled a condition kibble excuses, and resembling is all it did: a 403 is a missing account or a wrong argument, a refused connection is an absent service or a misnamed port, an exit 1 with no output is a search finding nothing or a command dying quietly. A skip would claim the document is fine. BLOCKED claims nothing.
• SKIP with the reason no-install-route means the documents hold no install of the tool itself that kibble extracts. The detail names the README, how many install documents were read, and the routes seen. It is not a GAP, since the documents may describe routes kibble does not run, such as apt or brew.
• BLOCKED with the reason inherited-state means a line that would have been a plain FAIL ran in a directory an earlier section entered with cd, pushd, or popd. Documents cd once in setup and stay there, so the failure may come from what that directory holds rather than from what the line says.
• BLOCKED with the reason checkout-state means a line that would have been a plain FAIL ran in the repository's own checkout and its output says the target already exists or is already initialized. The collision is with the checkout kibble chose, so the document may be fine.
• A passing line that writes a titled fence to its file reads VERIFIED "wrote X as the document shows it". The document supplied the content, so only the write is claimed.
• An install whose product is a shell function the documented sourcing defines is VERIFIED with the version it reports, since a function leaves no binary to find.
• GAP means the document is incomplete: a file, directory, or setting nothing creates. A skipped block of package-manager installs alone cannot have made the missing file.
• DRIFT means the docs cite a flag or subcommand the binary no longer has.
• TIMEOUT and ERROR keep slow networks and kibble's own trouble out of your verdict.

When a tool reports a missing file, program, git history, preset, or configuration, kibble reads that as evidence. If the document named the file, the verdict is GAP. If the tool reached for something the document never mentions, the verdict is BLOCKED, since the document may be incomplete or the tool may have a bug and kibble cannot tell which. If the container lacks a program, the verdict is SKIP. An operand written as a stand-in name, such as my-project or repo1, is a placeholder the reader fills in, so that line is a SKIP. Documents under directories named for an older version are not replayed, so a project's current docs are tested and its archived ones are not. Each document after the first starts from the tree and the tool the install left. What a document writes in the home directory still carries over to the next.

Every verdict also carries a coarse bucket (works, unverified, broken, doc-drift, not-attempted, inconclusive) so a consumer can condense the fine verdicts without hard-coding them. By default only FAIL and ERROR fail a run; -strict also fails everything outside the works and not-attempted buckets, so an unverified step, a drift, or a gap all count when you demand proof. The full reasoning, including why a check that cries wolf is worse than no check, is in docs/DESIGN.md.

A VERIFIED install row also carries version and latest in the JSON when the version the binary reported is one of the repository's release tags. The table marks the row behind when that version is older than the newest tag. This changes no verdict, since registries lag tags by design.

Why believe any of it

A verifier is worth exactly what its false positives and false negatives are worth, so both are pinned and enforced weekly against real repositories.

corpus/repos.tsv holds ripgrep, fd, glow, nodemon, httpie, and rich-cli, each at a fixed commit with the verdict counts a person read the documentation and verified by hand. A scheduled run fails when any count moves, because a moved count means kibble's judgment changed and the repository did not.

corpus/mutations.tsv runs the other direction. Each row corrupts exactly one documented line of a pinned repository, a crate name typo, a brew formula that no longer resolves, a file name no step creates, and requires kibble to reach a named verdict on the damage in a finding that quotes the mutated token. A count alone would not do, since a coincidence can produce a count.

What the weekly job prints, with each repository's counts in VERIFIED, FAIL, GAP order:

ripgrep: 3,0,0 as expected
nodemon: 2,0,0 as expected
fd: 6,0,0 as expected
glow: 5,0,0 as expected
httpie: 0,0,0 as expected
rich-cli: 2,0,1 as expected
ripgrep: FAIL=1 naming ripgreppp as expected
rich-cli: FAIL=1 naming richhh as expected
ripgrep: GAP=1 naming READMEE.md as expected

Correct documentation passes and one edit of rot flips it. Both halves are the claim; a tool that only holds one of them is either a rubber stamp or a nuisance. The same property is pinned below the integration layer by a unit suite that mutates a healthy document one edit at a time and requires the untouched copy to come back clean first. Reasoning in docs/DESIGN.md.

What kibble does not check

The container is Linux, x86 or ARM depending on the runner, with a network and no credentials. Documentation that only breaks on macOS or Windows, only under musl, only behind a proxy or a corporate CA, or only with a private registry configured, is documentation kibble cannot speak to. A block whose own prose scopes it to another operating system is skipped and said so, rather than run and blamed. Nothing here is a verdict about your reader's laptop; it is a verdict about a clean machine.

Security

kibble executes commands it read out of documentation, which means a README is untrusted input: anyone who can change the docs can change what runs. Every command runs in a fresh container with nothing mounted from the host, most Linux capabilities dropped, and no-new-privileges set. It is not an unprivileged container: the session is root inside it, because installing the packages a document depends on needs to be. The network stays open too, because verifying an install is fetching it. Treat a kibble run the way you treat a build script, and read docs/SECURITY.md before pointing it at a repository you do not trust.

Configuration

Most repositories need none. When a heuristic cannot settle a call, a .kibble.yml at the repository root does: fixtures, environment, substitutions, background services with readiness probes, and per-line run or skip rules, so the run stays reproducible and the engine stays the thing that decides pass or fail. -suggest has a model draft the file for you to review; -mcp serves the same engine to an agent. All of it is in docs/CONFIG.md.

Roadmap

• JUnit XML output for CI systems that are not GitHub.

Why "kibble"

Eating your own dog food means using what you ship the way a stranger would. Nobody does it for documentation. Your machine already has everything installed, and you stopped reading your own docs the day the workflow became muscle memory. kibble is the bowl: it feeds your docs back to a fresh machine and tells you whether they still go down.

License

MIT. See LICENSE.