Op2ra Information

Op2ra audits the instruments your numbers come from. Operations produce evidence. Evidence produces numbers. Numbers become assertions. We determine whether those assertions deserve to be believed and we publish what we could not check. The failure we catch is not a wrong number. It is an unfalsifiable one: a retrieval score with no audited test set, a campaign lift below its own detection floor, a saving with no written counterfactual. Each is defensible. None is evidence. The method is the same on every surface. Hold something out. Count the search — every variant tried, including the deleted ones. Correct for the count. Then declare what you could not see. The fourth step is enforced in code, not encouraged. Our report object refuses to construct if the limits list is empty, the resolution floor unstated, calibration claimed without error rates, or the findings list empty. That last rule exists because our own code audit once shipped a report that printed "no findings" over five defects it had already detected. The rule is in the codebase because the failure happened to us. FIVE SURFACES Documents — retrieval and search: is your evaluation set measuring your retriever, or your wording? Growth — is the win larger than the experiment could detect? Money — savings claims and track records: how many things were tried before this one worked? Code — where a system reports success while doing less than it claims. Assistant — grounded answers, and the instrument that checks them. WHAT IS ACTUALLY CALIBRATED One surface is calibrated against labelled data: 59 test suites over seven real corpora with human-authored queries. Three are calibrated against generators — simulated experiments, synthetic statements, constructed fixtures — a weaker kind of evidence, and every one of those reports says so. The fifth has no reference standard at all. Send one artifact. The first look costs nothing, and if we find nothing we will say that too. rishabh@op2ra.com