Backtesting trading robots — and the lie of the perfect curve

I've built hundreds of trading robots. The hardest part was never getting a backtest to look good — that's trivial. The hard part is getting one to keep working after you switch it on with real money. Those are two completely different problems, and the gap between them has a name: curve fitting.

Here's the picture every backtest is secretly trying to sell you.

The dream backtest: every arrow sells the exact top and buys the exact bottom. On past data it literally cannot lose. That perfection is the warning sign — not the green light.

Look at that chart. Every swing high is sold, every swing low is bought. The equity curve only goes up. If a robot showed you this, you'd fund it tomorrow. And that's exactly the trap, because hitting every top and every bottom isn't skill — it's hindsight. The price already happened. Fitting a strategy to it is like betting on a horse race after you've watched the replay.

What curve fitting actually is

Most strategies have a handful of knobs — call it x parameters. A moving-average length here, a stop-loss distance there, an RSI threshold, a time-of-day filter, a position-sizing rule. Maybe five of them. Maybe twenty.

Now you point that strategy at historical price data and start turning the knobs. Bump the MA from 20 to 23 — backtest improves. Tighten the stop by 4 ticks — improves again. Add a filter that skips Mondays because Mondays happened to be bad in your sample — even better. You repeat this, consciously or with an optimizer running thousands of combinations, until the curve is perfect for that time period.

And it works. That's the seductive part. A curve-fitted robot can run beautifully for weeks, sometimes months. I've had a few last over a year. But what you've actually built isn't a model of how the market behaves — it's a model of what the market did, noise and all.

Market data is not the same as market behavior. Data is behavior plus random noise. When you optimize hard enough, you stop fitting the behavior and start fitting the noise — and noise, by definition, doesn't repeat.

That distinction is the whole game. Real edges are rare and broad: they show up across many parameter values, many instruments, many years. A curve-fitted edge is narrow and brittle: it lives at one specific setting, on one specific stretch of history, and it dies the moment the world stops repeating itself.

Why it always breaks eventually

Markets are not a fixed system. They drift constantly — liquidity moves, volatility regimes shift, a competitor's algo changes how a level reacts, central-bank policy turns, the dominant participants rotate. None of these are dramatic on any given day. They're small. But your finely-tuned parameters were balanced on the exact shape of the old regime, and small changes are enough to knock them off.

So the robot that bought every dip starts buying dips that keep falling. The filter that skipped the bad days now skips the good ones. The edge degrades — sometimes slowly, sometimes off a cliff. This is the honest pattern almost nobody shares: the skill isn't only finding an edge, it's recognizing when the edge has stopped being one and turning the thing off. That's the call I've made on every robot I've run, including the ones that made good money first.

How I try not to fool myself

You can't eliminate curve fitting, but you can fight it. The principles below are the consensus across the practitioner and academic work linked at the bottom:

Keep it simple. Fewer parameters means fewer degrees of freedom to overfit. Every knob you add is another way to memorize the past.
Out-of-sample, always. Tune on one slice of history, then test on a slice the strategy has never seen. If it falls apart there, you fit noise.
Prefer flat parameter plateaus, not sharp peaks. If the strategy only works at length = 23 and collapses at 21 or 25, that's a spike on a noise surface, not an edge. You want a wide region where nearby settings all work.
Walk forward. Re-fit on a rolling window and trade the next window, repeatedly, the way you'd actually have to live it.
Incubate it. Run it forward on fresh data (paper or tiny size) before trusting it. The market doesn't care about your backtest; it only votes in real time.
Count your tries. If you tested 5,000 configurations and kept the best one, the "best" is probably luck. This is the multiple-testing problem, and there's now real math — the Probability of Backtest Overfitting — for putting a number on it.

None of this gives you the chart at the top of this article. That chart is a fantasy. What good process gives you is a strategy with more losses in the backtest but a real chance of surviving contact with a market that never stops changing.

A beautiful backtest costs nothing. A strategy that still works next quarter is the only thing worth having.

What curve fitting actually is

Why it always breaks eventually

How I try not to fool myself

Further reading