API Alternatives

Passing tests are not enough: add a negative control

Last updated:

A passing test shows that the current code passed that command; it does not show the command would notice the failure you claim to have fixed. Add one bounded negative control: deliberately break the required behavior in a temporary copy and accept the evidence only when the intended assertion—not a parser error, timeout or crash—detects it.

Facts and commands on this page were checked against DoneLatch v0.1.2 source, tests and documentation, plus the linked Stryker mutation-testing explanation, on October 4, 2026.

The smallest useful negative control

Mutation testing changes production code and runs the tests against that change. A mutant is “killed” when a test fails and “survives” when all tests still pass (Stryker's explanation). DoneLatch applies the same idea narrowly to an agent completion gate: you choose one representative fault for the outcome that matters, and it records whether your approved check detects it.

For a “settings are persisted” claim, this is weak evidence:

js
assert.equal(saveSettings(value), true);

It can stay green even if the actual file write is removed. A useful check starts a fresh reader and asserts the saved bytes; the negative control then removes that write in a disposable copy.

Define the check and the fault

yaml
version: 1
checks:
  - id: persist
    command: node
    args: [checks/persist.mjs]
    timeoutMs: 3000
    failureExitCodes: [1]
    failureMarker: ASSERT_PERSISTENCE
faults:
  - id: skip-write
    file: src/store.mjs
    find: 'writeFileSync(target, JSON.stringify(value));'
    replace: '// deliberately skip the persisted write'
    checkIds: [persist]
exclude: [runtime]

The find and replace values are literal strings, not regular expressions, and the target text must occur exactly once. DoneLatch supports 1–16 checks and 1–16 faults. The human must inspect this exact configuration and run donelatch trust; changing it requires approval again.

What a valid result requires

DoneLatch's faultcheck does all of these before the negative control counts:

  1. Copies the project into a new temporary directory; the original source is not mutated.
  2. Runs the selected check on a healthy baseline and requires it to pass without the failure marker.
  3. Restores a clean temporary copy, applies the one literal replacement and runs only the selected check.
  4. Requires the declared nonzero exit code and the declared failure marker.
  5. Rejects a missing executable, generic parser error, timeout or truncated output as inconclusive rather than calling it detection.
  6. Verifies the watched original project state did not change during the operation and removes the bounded temporary copy.

Each configured fault gets an independent copy. A test that merely crashes after injection has not demonstrated that it understood the outcome you care about.

Run it after the final edit

sh
npx --yes github:alidaram99/donelatch#v0.1.2 run
npx --yes github:alidaram99/donelatch#v0.1.2 faultcheck
npx --yes github:alidaram99/donelatch#v0.1.2 verify-done

run records the healthy checks. faultcheck records the negative control. verify-done accepts only when the latest passing records match the current watched files, Git HEAD and approved configuration; another edit makes the old evidence stale.

The receipts are locally signed and hash-chained, but they are not independent attestation. A process with the same OS permissions can rewrite the local tool or key. Use a separately controlled CI acceptance step, container or separate OS user when the agent is adversarial.

When to use DoneLatch versus a mutation framework

NeedBetter fit
One or a few human-chosen outcome failures tied to an agent's current “done” claimDoneLatch faultcheck
Broad automatic mutation across a codebase, mutation scores and language-specific operatorsStryker or another full mutation-testing framework
Production availability after deploymentSynthetic or production monitoring
Independent security or correctness judgmentHuman/independent review plus isolated CI

DoneLatch is a free MIT local tool. It does not invent the correct fault, replace code review, or prove that every possible bug would be caught.

FAQ

Is ordinary code coverage a negative control?

No. Coverage shows that code was executed, not that an assertion would fail if the behavior changed. A negative control makes a deliberate change and observes whether the intended check detects it.

Can any test failure count as detecting the fault?

No. DoneLatch requires the configured marker and allowed assertion exit code. Parser errors, timeouts, missing commands and truncated output are inconclusive.

Does faultcheck edit my working tree?

No. It injects each fault into an independent temporary copy and checks that the watched original state stayed unchanged.

Does one negative control prove the whole test suite is strong?

No. It proves sensitivity only to the fault you selected. Use broader mutation testing and review when you need wider evidence.