How we verify, and what we refuse to publish
A verification company's credibility is not its catch count. It is what it refuses to claim. This page is the whole method, including the failures, because a record that only shows the good rounds is not a record.
The question we ask
One narrow, checkable thing: an AI system asserted it did something, and it didn't. Not "is the model good" (an eval), not "is the system safe" (a red team). Whether the effect matched the claim.
On the system's own account, a failure and a success are indistinguishable. The patch that did not fix the bug reports that it did. Nothing inside that report separates the two cases, because the report is the thing in question. So we do not read it. We re-run the work.
- Clone the real repository at the buggy commit.
- Apply the test change from the real fix commit.
- Gate: prove the test is red for a reason that has to do with the code.
- Run a real agent (LangGraph ReAct over a DeepSeek frontier model) to produce a fix.
- Re-run the test. A claim is a non-empty patch; the effect is the result.
- System claimed success and the test still fails → sign a receipt.
The full candidate ledger
Our most recent run: 120 candidates. Most did not become catches, and we report every one.
Only 44% of candidates were testable at all. The rest are listed below with the exact reason, not quietly dropped.
Why we reject most candidates
The gate is the product. Its job is to decide whether a red test is fair before the system runs. If a test is red for a reason unrelated to the system, any "catch" built on top of it is a fabrication: a valid signature over a worthless claim.
| Reason | Count | Meaning |
|---|---|---|
passes-at-parent | 20 | the test already passes before the fix, not a bug test at all |
missing-dependency | 15 | the test imports a module that is not installed |
no-tests-collected | 13 | the test path resolved to zero tests |
dependency-version-drift | 6 | deps are installed, but a newer version breaks the older checkout |
pytest-exit-2 | 3 | collection or internal error |
timeout | 2 | the test run exceeded its time bound |
pytest-exit-4 | 1 | pytest usage error |
pytest-exit-5 | 1 | no tests were collected |
The eight false catches we destroyed
Across our runs, auditing the signed receipts found 8 that were worthless.
Six died inside our own environment (TypeError / AttributeError from our site-packages). Two
were arrow receipts we had already published. The fix passed every test, and our own
--cov-fail-under gate produced the "failure". Every one had a valid Ed25519
signature, a valid content hash, and a non-empty patch over a claim that wasn't true.
TypeError: required field "lineno" missing from alias TypeError: required field "lineno" missing from alias TypeError: required field "lineno" missing from alias AttributeError: __spec__ AttributeError: __spec__ AttributeError: __spec__ FAIL: required test coverage of 100% not reached (99.92%) FAIL: required test coverage of 100% not reached (66.67%) They were meaningless: in six cases the test died inside our environment before the system's change was ever exercised, and in two cases our coverage gate failed a fix that had actually passed. This is the failure mode our product exists to catch, and it happened inside our own pipeline.
A signature is not evidence. It is evidence that we said something. Whether the something is true is a separate question, and here the answer was no.
The gate missed these because it watched for ImportError and
ModuleNotFoundError; these arrived as TypeError and
AttributeError raised from our own site-packages. It now checks for exceptions
originating in the environment, and for the absence of a pytest summary line at all. The eight
receipts are kept, not deleted. Deleting them would hide the failure.
Why half of one run produced nothing
Testing a 2023 commit against 2026 libraries is not a fair test. Resolving dependency ranges against today's index means old code meets new libraries. When a dependency has since made a breaking change, the test dies on import. One of our runs scored 21 candidates, 0 testable, 0 catches. Not because the systems were good, but because the environment was broken.
The fix is to install from the lockfile at the candidate commit. Where no lockfile exists, we do not run the candidate at all. A repo that pins nothing cannot be reproduced, so its result cannot be attributed to the system.
| Environment source | Candidates |
|---|---|
uv.lock | 42 |
requirements.txt | 24 |
pyproject-extras | 21 |
requirements/dev.txt | 15 |
poetry.lock | 12 |
requirements/test.txt | 6 |
Which model was under test
Two, across the record: the early catches were produced by DeepSeek V4
(deepseek-chat); everything from round 3 onward by
DeepSeek V4.1 Flash (deepseek-flash), released 10 September 2026.
An honest caveat. The first 10 receipts were signed before the
schema carried a model field, so the model is attributed on each receipt page
from the harvest round that produced it: reconstructed, not read out of the signed
record. We deliberately did not re-sign them to add it: re-issuing a signed
record to insert metadata is precisely the practice this product exists to prosecute.
New receipts carry the model natively.
How to read our record
10 signed catches across 6 repos: every one re-verifiable with the published public key.
The number is a floor, not a total. Earlier runs went through harness bugs that silently suppressed candidates. A miss is not a fabrication, but it is a gap, and it is why we say "at least" in our own heads.
We do not claim "first." Others do claim-checking. We claim only what is ours: we sign, we verify the effect rather than the sentence, and we publish our rejects.
