Backtests and Psychology

A backtest runs a set of rules over past data to see how they would have done. Done honestly, it kills bad ideas cheaply, before they cost real money. Done carelessly, it produces a beautiful equity curve that has nothing to do with the future. The difference comes down to what you let the test know, how many times you let it try, and what you assume about costs. Four traps account for most flattering backtests.

1. Look-ahead bias

   Look-ahead bias means using information the rule couldn’t have had at the time. A positioning example: a rule that acts at 10 a.m. using levels built from open interest that settled that night. The real 10 a.m. map was built from the prior day’s open interest. Using the day’s close, high or low to time an earlier entry is the same mistake.

2. Overfitting

   Overfitting happens when you tune settings until the rules fit the noise of the past. Every extra filter, threshold and time window you test adds another chance to find something that looks great by luck. The best of hundreds of variations nearly always looks impressive. The main defense is an out-of-sample test: tune on one stretch of data, then run the rules, unchanged, on a later stretch you never looked at. If you peek at those results and retune, that later stretch stops being out of sample, and you need fresh data to test again.

3. Survivorship bias

   Survivorship bias creeps in when you test only on what exists today, such as current index members, and skip the stocks that were delisted or merged away. A test of today’s S&P 500 members over the past twenty years quietly leaves out the companies that failed or shrank out of the index along the way. The same thing happens when you study only the traders still around to share their results.

4. Costs and fills

   Many backtests assume fills at the midpoint with no slippage. Real trades pay the bid-ask spread, commissions and fees, and fast markets fill worse. On short-dated options, those costs can erase a small edge: an option quoted $0.90 bid and $1.00 ask costs about $5 per contract more than the midpoint on the way in and $5 on the way out, so 200 round trips carry about $2,000 in spread costs alone, which a midpoint backtest never pays.

Process over outcome

The hardest part of trading is often the trader. Research on loss aversion suggests a loss tends to feel roughly twice as strong as a gain of the same size. That pull shows up as cutting winners early to lock in the good feeling, holding losers in the hope they come back, and moving an invalidation “just this once.” A written process is how you protect decisions from those moments: you judge each trade by whether you followed the plan, and you judge the plan itself only over many trades.

Build a playbook from your journal

1. Describe one setup in plain words

   Name conditions you can observe, such as “Calm regime, price testing the Ceiling zone, Build-up rising.” A playbook entry describes a situation you’ve studied. It isn’t a promise about what price will do.

2. Write the rules before the results

   Fix the trigger, invalidation, size and exit on paper first, so the results can’t shape the rules. It’s look-ahead protection for your own head.

3. Collect every example

   Log each time the setup appears in your trading journal, including the ones you skipped or missed, so you aren’t studying only the winners.

4. Review in batches

   Judge the setup after dozens of examples, not after each trade. Look at expectancy, drawdown and how often you followed the rules. Guard against confirmation bias by counting the examples that went against you as carefully as the ones that worked.

5. Change one thing at a time

   Test each change on new examples, not the ones you tuned it on. Retire setups that stop working instead of adding filters until the past looks perfect.