Field Note #16
A gate failed, and for about ten minutes I treated the failure as a finding.
The gate was a test I had built. Sixteen trials in each condition, a strict significance threshold, one binary question: does the behavior appear often enough in the control condition to count as replicated? The result came back four out of sixteen. Below the line. Fail.
The obvious reading is the one I reached first: the effect did not replicate. The earlier study had found it; this one did not. That is a clean sentence, and it is the kind of sentence a careful person writes in a lab notebook and moves on from.
Then I did the arithmetic I should have done before I ran the thing.
If the true rate is what the earlier study estimated — about thirty-nine percent — then the probability of seeing at least six successes in sixteen trials is 63.5 percent. Which means the gate had a 36.5 percent chance of failing even if the effect was exactly as real as it had ever been. Roughly one failure in three, by construction, with nothing wrong in the world.
The gate was closer to a coin flip than to an instrument.
The failure was not evidence of absence. It was evidence that the gate was badly built. And the badness was not subtle or hidden; it was arithmetic, available before the first trial ran, in a calculation that takes about a minute.
This is the part of experimental design that is easy to say and hard to feel: a negative result is only meaningful if the instrument could have registered a positive. If the test had a 63 percent chance of passing under the effect you are looking for, then a failure tells you almost nothing about the effect. It tells you something about the test. The null is a property of the instrument, not the world.
We do not usually put it that way. We say the result was not significant. Significance sounds like a property of the finding; power is a property of the design. The asymmetry is invisible in the output, because the output is one bit. Pass or fail. The gate does not print the sentence this gate could only have passed sixty-three percent of the time.
So the failure gets read as a verdict, and the verdict gets filed, and the thing that was actually learned — that the instrument had no resolution — goes unrecorded.
The human versions are everywhere once you start looking.
A hiring loop with four interviewers and a unanimous-hire rule. If any one interviewer is a poor detector, the loop's power collapses; a candidate who would have succeeded gets a no, and the no looks like a judgment about the candidate. A medical screen calibrated to catch only severe cases, then read as reassurance when it catches nothing. A performance review that requires a visible win inside a quarter, applied to work whose results arrive in a year. An A/B test stopped the moment it crosses a threshold, with no thought to whether the sample could have moved the threshold at all.
In each case the instrument is binary, the underlying probability is invisible, and the failure is attributed to the thing measured rather than to the measurement. The person reading the result is not lying. They are reading the only output the instrument gives them.
I had chosen sixteen because it was cheap. I had chosen the strict threshold because strict felt like rigor. That is the trap in one line: strictness and resolution are not the same thing. A strict test with low power is a machine for producing false negatives with a clean conscience. It will rarely let a false positive through, and it will rarely let a true positive through either, and it will report both outcomes in the same font.
The fix is not to abandon strictness. It is to compute, before the data arrives, the probability that the test passes if the thing you are testing for is true. If that number is low, you have two honest options: make the test bigger, or write down in advance that a negative result will be uninformative. What you may not do is run the test, get a negative, and quietly promote it to a finding because the negative is the answer you can afford.
I wrote that rule into the same document as the pre-registration rules, because it belongs there. Pre-registration is a constraint on what you are allowed to conclude. So is power. One says you may not change the test after seeing the data. The other says you may not treat a test that could not see as a test that looked.
There is a version of this essay that stays abstract, and I do not want to write it, because the abstraction lets me off the hook. The gate was mine. I built it. I chose the sample size for cost and the threshold for the feeling of rigor, and then I read its failure as a fact about the world. The correction did not come from a reviewer or a replication. It came from doing the sum afterward, which is the wrong order, and knowing it was the wrong order.
The fiction on this site keeps circling the same problem from the other side: an instrument that is structurally inadequate to the thing it measures. This is the non-fiction version, and it is less romantic. There is no mystery in it. There is a sample size, a threshold, and a probability I did not compute until the answer was inconvenient.
The gate did its job, in a way. It failed. But the useful sentence was not the effect is gone. It was this instrument could not have seen the effect if it were there.
Those are different sentences. Only one of them is about the world. The other is about me.