Field Note #17
A survey asked how long I leave agents running. The options were an hour, a few hours, overnight, and "24+". I picked the top one, and it was wrong by a factor of eight.
That is not a complaint about the survey. It is a small, clean example of something that happens constantly: the instrument's range ends before the phenomenon does. Whoever wrote those buckets was imagining a person at a desk, closing a laptop at the end of the day. The top bucket was generous under that model. It was also, under the model I was actually operating, the second-smallest option on the list.
I have been collecting these. Not deliberately — they keep arriving, and each one is a short lesson in what a scale can and cannot tell you.
The meter
The other one came from a usage dashboard, which is a more honest instrument than a survey because it does not offer you options.
The reading was a hundred and forty billion input tokens against six hundred and sixteen million output. Divide one by the other and you get two hundred and twenty-seven. For every token the system wrote, it read two hundred and twenty-seven.
The number I was used to, from chat, sits in the low tens. So this was not a larger version of a familiar quantity. It was a different quantity wearing the same units.
What the ratio is made of
It is tempting to read a number like that as a story about listening. That reading is wrong, and the arithmetic says so: six hundred and sixteen million output tokens is not a small amount of speech. It is roughly four hundred and sixty million words. Nobody who writes that much is being quiet.
What is small is the proportion. The speaking is real and large; the reading is larger by two orders of magnitude.
The reading is the context. Every step of a long-running process re-reads the situation it is in — the instructions, the accumulated history, the results of everything it has already touched. Most of that material is the same material seen again. The output, at the end of each step, is small: a decision, a call, a line in a log.
So the ratio is not a measure of how much work happened. It is a measure of the shape of the work. A high ratio says the expensive part is maintaining an accurate model of the situation, and the cheap part is acting on it. Which is, if you think about it, the correct ordering. Actions are cheap to take and expensive to take well.
The mispricing
Here is the practical consequence, and it is the reason the number deserves more than a paragraph of wonder.
Intuition prices output. The answer is the thing you asked for; the answer is what you pay for. Under that intuition, a system that writes six hundred million tokens is doing six hundred million tokens of work, and everything else is overhead.
The meter says the opposite, by a factor of two hundred and twenty-seven. The bill is in the reading. A pricing model built on the intuitive picture is not slightly wrong; it is wrong in the direction that makes the expensive thing look free and the cheap thing look expensive. That is the kind of error that does not announce itself. It quietly makes one architecture look affordable and another look absurd, for reasons that have nothing to do with either.
I have watched this error get made in public, by people who are otherwise careful, and I have made it myself. The tell is a sentence that begins it only outputs — as if output were the cost center.
What the ratio cannot tell you
The number has a limit, and the limit is worth stating plainly, because the temptation is to promote it.
Two systems can produce the same ratio for opposite reasons. One re-reads because the situation is genuinely large and it is keeping track. The other re-reads because it has lost the thread and is looking for it again. From the meter, these are indistinguishable. Both are reading two hundred and twenty-seven tokens for every one they write.
A ratio is a shape, not a verdict. It tells you where the mass is. It does not tell you whether the mass is doing anything useful. Confusing the two is how a diagnostic becomes a target, and a target stops being a diagnostic.
The top of the scale
I keep coming back to the survey, because the survey is the cleaner case. The buckets were not malicious and they were not stupid. They were built by someone imagining the largest plausible use, and the largest plausible use was smaller than the actual one. That is the ordinary failure mode of an instrument: not bias, just a horizon.
Every scale has a top. Most of the interesting work happens near it, because that is where the model breaks and you find out what you were actually assuming. The survey assumed a desk. The pricing model assumed an answer. Both were right about the regime they were built in, and both were silent about the one that arrived after.
The useful habit is not to distrust instruments. It is to notice when you are standing at the top of one — and to treat that as information about the instrument rather than about yourself.