Technical Analysis · Chapter 11 of 12
Testing a technical strategy without fooling yourself
The methodology chapter. If you internalise only this one, you will be ahead of almost everyone who trades on charts.
Everything in this book has pointed here. A technical rule is a hypothesis, and a hypothesis you have not tested against a proper control is a belief. The distance between the two is where most trading money is lost, and closing it is entirely a matter of method rather than of intelligence.
The five steps, in order
1. State the rule mechanically. Every term must be codeable. “A strong bullish candle near support in an uptrend” contains four undefined terms. If you cannot write it in code, you are not applying it consistently either — which means your live results and your recollection are both unreliable.
2. Build a matched control. This is the step that decides whether anything you conclude means anything, and it is skipped by almost everyone. The control must match your strategy on everything except the signal: same instrument, same period, same number of trades, same holding period, same position sizing. The only difference is that entries are random.
Matching the control to the entry is the hard part and the important part.
If your entry happens after a pullback, a control that enters at random closes is not a control — it measures short-horizon mean reversion and credits it to your signal. The control must also enter after a pullback.
If your entry only fires in high-volatility regimes, the control must be restricted to high-volatility regimes.
If your strategy holds for 10 days, the control holds for 10 days.
And average at least five random draws. A single random comparison is under-powered and will land high or low by chance, which will then mislead you in whichever direction it happened to fall.
3. Compute expectancy, never a hit rate. Expectancy per trade is (win% × average win) − (loss% × average loss), net of all costs. Express it in R — multiples of the amount risked — so results are comparable across instruments and position sizes.
A 70% win rate sounds excellent and loses money if the average win is 0.3R and the average loss is 1R: (0.7 × 0.3) − (0.3 × 1.0) = −0.09R per trade. A 35% win rate with a 3R average win and 1R average loss returns +0.4R per trade. Hit rate on its own carries no information about whether a strategy makes money, and it is the number almost every published strategy leads with.
4. Split by regime and by year. Report every single year separately, plus a split by volatility regime. An aggregate number hides the strategy that made everything in one eighteen-month window and nothing since — which is by far the most common shape of a backtest that looks good and fails live.
5. Vary the size. Re-run at 2× and 5× the assumed position size with an impact model. Where does the edge disappear? That number is the strategy's capacity, and it is a real answer to a question every serious person will ask.
The seven ways a backtest lies
| Failure | What it looks like | The fix |
|---|---|---|
| Lookahead | Using information not available at decision time | Point-in-time data. Never enter at an adjusted price. Check that every input's timestamp precedes the decision |
| Survivorship | Universe built from today's members | Point-in-time membership, including delisted names |
| Multiple testing | 200 parameter sets, the best one looks great | Count your hypotheses. Hold out data you have literally never looked at |
| Cost naivety | A flat basis-point assumption | Real fee schedule, half-spread each way, and a size-dependent impact term |
| Fill fantasy | Assuming you got the touch price, or the open | Fill only against traded volume; cap participation at a realistic fraction |
| Regime luck | One great window, flat elsewhere | Report every year separately, and say so plainly |
| The missing control | Comparing to nothing at all | A matched random control, averaged over at least five draws |
Walk-forward, which is how professionals do it
In-sample and out-of-sample is the minimum. Walk-forward is the honest version: fit or choose parameters on a window, evaluate on the next window, roll forward, and never look at an evaluation window while making decisions about it. The result is a concatenated out-of-sample record that reflects how the strategy would actually have been operated — including the periodic re-fitting, which is itself a source of degradation people forget to model.
The discipline it enforces is psychological as much as statistical. Once you have seen the out-of-sample result, you cannot un-see it, and every subsequent adjustment is contaminated. Walk-forward makes that contamination visible because a re-run after tweaking is obviously a second look.
A checklist to run before you believe your own result
- Have I stated the rule in code, with every parameter named?
- Do I have a control that matches my entry's character, averaged over five or more draws?
- Is my expectancy in R and net of realistic costs including impact?
- Have I reported every year separately?
- How many variants did I try before this one, and have I written that number down?
- Does the result survive small changes to each parameter, or does it live at one setting?
- At what size does the edge disappear?
- If I flip the signal's sign, do I get a symmetrically bad result? If a reversed signal is also profitable, I am measuring exposure rather than signal.
- Have I checked for a data error at the dates of my largest gains? The single biggest winner in a backtest is a bad print more often than anyone admits.
That last two are worth dwelling on, because they catch different things than the rest. Reversing the signal is close to a free control. And auditing your best trades — rather than your worst — is where data errors hide, because a bad print produces an enormous fictional gain and nobody ever investigates a win.
Check yourself