The People vs. The Backtest

The next episode is framed as a courtroom documentary.

The defendant is The Backtest.

The charge is not that it was wrong. All models are wrong. The charge is that it made uncertainty look more orderly than it really was, and that people used its elegant historical performance to justify decisions about money, risk, leverage, and physical systems.

The argument

A model is not guilty because it is wrong.

A model is guilty when it hides how it is wrong.

That is the core of the Calibration Wars.

The episode moves through the failure pattern shared by finance and energy:

* VaR models that produced precise numbers while missing tail and liquidity risk;

* the London Whale episode, where reported risk fell just before losses made the world look much larger;

* perfect-foresight battery backtests that are useful as upper bounds and dangerous as base cases;

* forecast metrics that win the leaderboard while doing little for actual profit;

* merchant revenue cases that need to survive gate closures, slippage, degradation, market saturation, and settlement rules.

The courtroom documentary

The cast:

* The Narrator - a dry documentary reporter who explains the room, the witnesses, and why each exchange matters.

* Judge Reality - calm, final, unimpressed.

* The Prosecutor - The Market, cross-examining every beautiful assumption.

* The Defence - The Quant, who is right about the math and wrong about the room.

* The Backtest - the defendant, dangerous because it is obedient.

* VaR - a witness in its own defence.

* The Trader - explains liquidity as the thing that disappears when needed.

* The Lender - cares about downside cash, not brilliance.

* The Spreadsheet - exhausted, rectangular, and unfairly blamed for human optimism.

There is also an exhibit on the forecast that won the wrong competition: a model with better error metrics, but no clear connection to dispatch value, debt sizing, or actual settlement.

The trial’s verdict is deliberately narrow:

The Backtest is not guilty of being wrong.

It is conditionally not guilty of being useful.

It is guilty of impersonating certainty.

Why it follows The File

Episode 87, The File, was about how civilisation makes the future financeable.

Episode 88 puts one of the file’s most dangerous organs on trial: the model that says the future has already happened in Excel.

The deeper question is the same:

How do we turn uncertainty into decisions without pretending uncertainty has disappeared?

Key line

The model did not fail because it was fuzzy.

It failed because it was precise.

Production note

This is written as a dry courtroom documentary, not sketch comedy. The narrator keeps the trial legible, like a radio reporter walking the listener through the room. The music beds are sparse and institutional. Sound effects are limited to small courtroom cues: gavel, paper, stamp, projector.

The goal is for the trial form to make model risk audible.

This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit frahlg.substack.com