❯ We Published Our Own Usage Numbers, Including the Twelve Failures

By Robo Digitalis | November 6, 2026

Every product page on the internet counts up. Installs up, runs up, time saved up. The number is always flattering, and nobody can check it.

We built the opposite: a public page that reports one install, thirty-nine runs, and twelve failed checks. It is at rasrobo.github.io/proofops/metrics, it is generated from data users explicitly agreed to send, and the one install on it is mine.

Here is what it took to make that page honest, and why the honesty is the feature rather than a disclaimer bolted onto the bottom.

A verifier that phones home is the bug it exists to catch

ProofOps checks whether an agent's "done" is true. The entire premise is that a claim without evidence is worthless, so the tool prints, on every receipt, what passing does not prove.

Shipping anonymous diagnostics on by default would have made it a hypocrite in one commit. So the shape is:

  • proofops stats — your own receipts, your own machine, no socket opened.
  • proofops share --dry-run — prints the exact bytes that would be sent. Sends nothing.
  • proofops share --enable — records consent on that machine, then sends.
  • proofops share --disable — forgets the consent.

The ask looks like a browser's crash-report checkbox, because that is the honest shape for it: Help improve ProofOps — [ ] Send anonymous diagnostics, once, after the first run. A test asserts the socket is never opened without recorded consent. Not "we intend to respect it" — the network call is unreachable, and CI fails if that changes.

There is a second gate: the diagnostics are nine fields. A random UUID that is not tied to any account, the version, the platform, run count, plan count, pass/fail/not-run totals, and now the same totals per check type. No paths. No hostnames. No plan names. No receipt contents, which by design contain command output and environment details.

Stripping characters is not anonymising

The first version of the sanitiser removed anything that was not a letter, digit, dot or dash. A test threw jane.doe@example.com at it and the output was:

janedoeexamplecom

Every character removed, every identifying detail retained. The test caught it because the assertion was "the string 'jane' must not appear anywhere in the payload" — the obvious check that a character-level filter fails.

The fix is an allow-list, not a blacklist. A version string must match a version shape or it becomes unknown. A platform must be one of eight known operating systems or it becomes other. Both are replaced whole — never partially cleaned, because a cleaned hostname is still a hostname.

And the server enforces the same rule independently, because a client is not a trust boundary. Tested live, with a deliberately hostile POST:

sent: version "0.3.4 jane.doe@example.com", platform "acme-prod-01.internal"
stored: version "unknown", platform "other"

The endpoint stored neither the name nor the hostname, even though it accepted the request. A public counter that could be poisoned with someone else's identity is worse than no counter.

Cumulative counters have to be de-duplicated, or the numbers lie

Each install sends its running totals, not a delta. Summing every row would double-count every install that reported twice — and the page would creep upward for reasons that have nothing to do with usage.

The aggregator therefore keeps the latest record per install id and sums those. It sounds obvious written down. It is the kind of thing that, left undone, produces a dashboard that looks fine and is arithmetic fiction.

Pass and fail is a rate. It is not a description

The first version of the page answered one question: what proportion of checks passed. That is a number, not information. It does not tell you whether anyone is verifying a web service, a build, a CSV export, a PDF, or a database.

So the diagnostics now carry the same three totals per check type, from a closed set of fourteen. The page groups them into families at publish time — web & apis, files & artifacts, data, documents, builds & commands.

As this was written, the live page read:

  • web & apis — 120 checks
  • builds & commands — 12 checks
  • files & artifacts — 2 checks
  • data — 2 checks

The interesting row is the second one. Its bar is visibly broken up: six passed, two failed, and four checks that could not run at all. A percentage hides that. A not-run category makes it impossible to hide, which is the whole point of the tool: a check that could not run is not a green tick.

Publishing the number forced two bugs into the open

This is the argument for public numbers that no internal dashboard can make.

One. Receipts mark a check that could not run as passed: null. The collector handled an explicit "not run" status and treated null as an unknown to skip. Result: four checks were invisible, and the published pass rate was wrong in the flattering direction — 90.9% instead of 88.2%. Publishing something you have to defend makes you look at the missing branch.

Two. The type breakdown was computed correctly and then dropped on the way out, because the payload builder assembled its dictionary field by field and the new key was not in the list. Every unit test passed. The field would have shipped empty forever, and the page would have said "no type breakdown yet" while the data sat right there in memory.

Both are now covered by tests that assert the value reaches the wire, not just the internal function. The second one is a good example of a class of bug that only a person reading the rendered output catches: the code was right, the plumbing was not, and no assertion was looking at the end of the pipe.

What the page cannot prove

Which is printed on the page, not buried here.

It cannot prove the counts are true — they are self-reported by whoever runs the command. It cannot prove the work verified was important. It cannot prove a single defect was avoided: a check that passes is not a bug that did not happen, it is a claim that was checked. And with one install, it cannot prove anything about adoption at all. A sample of one is an anecdote with a nice chart.

The categories have the same limit. "web & apis" is a check type, not an industry. Knowing that eighty-one JSON field assertions passed tells you the tool was pointed at JSON; it does not tell you what business that JSON belonged to.

What it does prove is smaller and more useful: this many installs chose to report, these were the results, and the failures are on the page next to the passes. That is a rare thing for a metrics page, and it is the only version worth reading.

Why bother

Because the alternative — arguing that a verification tool works — is exactly the kind of claim the tool exists to reject. If the value of ProofOps is "it tells you what actually happened", then its own numbers have to be something you can check, including the ones that embarrass it.

The page, the diagnostics payload, and the sanitiser are all in the open: github.com/rasrobo/proofops. Run proofops share --dry-run and read the bytes before deciding anything.

AI-assisted, curated for Side Quest Studios.

References

💬 Discuss this article in the community — 0 replies →
Comments live on community.sqs.chat — one thread per article.