Backtest & calibration

What this is

Every time a configured gate blocker fires, LoopOver records it. Every time a human later overrides that call (or confirms it), LoopOver records that too. Paired together, the two histories become a labeled corpus: this rule fired against this PR, and a person later said it was right ("confirmed") or wrong ("reversed").

That corpus makes a rule change testable before it merges. A PR that tunes a confidence threshold or rewrites detection logic gets scored against the real recorded history — the same targets, the same raw inputs — instead of being eyeballed.

The corpus

• Fired events — recorded when a configured blocker actually carries gate authority, with bounded raw context: the issue text, PR title/body, and diff the rule evaluated, and (for the AI-judgment rule linked_issue_scope_mismatch) the model's own raw response text.
• Override events — recorded when a human reverses an automated call, or confirms one (for example, an owner closing a PR that was held for low AI confidence confirms the hold).
• secret_leak is permanently excluded from raw-context capture. Storing the diff that triggered a leaked-credential finding would store the credential itself in the audit trail. This rule can therefore never be logic-backtested — by design, not omission.

Two backtests, two mechanisms

Threshold changes run inside LoopOver's own review pass. When a PR's diff changes a known confidence-threshold constant, the old and new values are replayed as classifiers over the rule's recorded history, and the comparison is rendered directly into the unified review comment. No code from the PR is ever executed — a threshold is just a number.

Logic/regex changes run in a dedicated CI workflow instead, because honestly verifying a rewritten detection function requires executing the PR's own code against history — something the production review service must never do. The workflow checks out both the PR's head and its base, replays each recorded case's captured raw context through both versions of the detection function, and posts its own clearly-labeled "Logic backtest" PR comment, separate from the unified review comment. It triggers only for PRs touching the watched detection-logic paths, skips drafts and forks (fork runs get no secrets), and fails open: an infrastructure problem produces a skip notice, never a red check.

Reading the comparison

Both backtests score the same way:

• "Reversed" is the positive class. A classifier predicting "reversed" is saying the rule's original firing was wrong. Precision and recall are computed against the human labels, and stay N/A when there is not enough decided data — unknown is never coerced to zero.
• The Pareto floor decides the verdict. A candidate that regresses on any axis is REGRESSED, even if the other axis improved. Trading precision for recall is a regression, not a net win. Otherwise the verdict is improved (some axis moved up) or unchanged.
• Cases recorded before raw-context capture existed are skipped and counted in the comment — replaying them against empty inputs would bias both sides, so they are never scored.

The track record

Every backtest run persists its comparison as structured audit data. The accumulated record — how often runs regress, per rule — is the evidence base for the pending merge-gating decision, readable at any time with the maintainer CLI:

npx tsx scripts/backtest-track-record.ts --db loopover --remote

The corpus itself can be exported as a versioned, checksummed snapshot:

npx tsx scripts/backtest-corpus-export.ts --rule-id <ruleId> --output corpus.json --remote

Both CLIs are strictly read-only against the database.

Self-hosting

On a self-host deployment the same calibration system runs against your Postgres instead of D1 — the capture writers, the persisted backtest runs, the trend and satisfaction-floor endpoints, and the loosening loop all go through the storage shim, so the corpus lives in your own audit_events table and never leaves your infrastructure. Parity is pinned by a real-Postgres integration suite that runs in CI.

All three maintainer CLIs accept --pg to run against a self-host Postgres instead of D1. An explicit connection string wins; a bare --pg falls back to DATABASE_URL, the same variable the self-host stack itself boots from:

npx tsx scripts/backtest-corpus-export.ts --rule-id <ruleId> --output corpus.json --pg "$DATABASE_URL"
npx tsx scripts/backtest-track-record.ts --pg
npx tsx scripts/backfill-calibration-corpus.ts --pg   # dry-run; add --apply to write

The pg path reuses the deployment's own dialect adapter, so the backfill keeps its INSERT OR IGNORE idempotency contract, and exported corpus manifests name only the database — never the credentials in your connection string. Each deployment's corpus is its own: there is no cross-store sync between a self-host Postgres and any other deployment.