Backtesting trading signals
Backtesting applies a fully specified trading rule to historical data to estimate how it would have performed. Its results are only as credible as its data and design: look-ahead bias, survivorship bias, unrealistic costs and testing many variants can all make a rule look better than it is. Past results do not guarantee future results.
Loading the interactive OptionMind workspace…
Common sources of bias
Look-ahead bias uses information that was not available when the decision was made, such as trading at the open on that day’s close, using later-revised data or picking stocks by later index membership. Point-in-time data with a timestamp for every input helps prevent it.
Survivorship bias tests only what survived, such as today’s index members without the stocks that were delisted; quietly dropping trades whose outcome could not be measured has a similar effect. Both flatter results. Cost assumptions matter as much: filling options at the mid-price ignores the bid-ask spread, which can erase a small edge.
Out-of-sample and multiple testing
Design and tune a rule on one period, then test it once on later data it has never seen (out-of-sample, or a holdout). Walk-forward testing repeats this on rolling windows. Re-tuning after looking at the holdout turns it into in-sample data.
Testing many variants inflates false discoveries: with 20 independent variants each tested at a 5% significance level, the chance that at least one looks significant by luck alone is 1 − 0.95²⁰ ≈ 64%. Pre-specifying the main test, recording every variant tried and tightening the threshold (Bonferroni: 0.05 ÷ 20 = 0.0025) reduce that risk.
Worked example
A rule produces 120 trades: 66 winners averaging $150 and 54 losers averaging −$110. Win rate = 66 ÷ 120 = 55%. Expectancy = (66 × 150 − 54 × 110) ÷ 120 = (9,900 − 5,940) ÷ 120 = $33 per trade, and profit factor = 9,900 ÷ 5,940 ≈ 1.67.
The 95% Wilson interval for a 55% win rate on 120 trades runs from about 46% to 64%, so the true rate could be well below or above 55%. The result needs more trades and an out-of-sample test before it can be called an edge.
How Edge Validation measures it
OptionMind’s Edge Validation page measures the Opportunity Scanner’s own episodes forward in time instead of replaying old data. Episodes are recorded when detected, results are isolated by score version, and every triggered episode stays visible, including pending and unavailable outcomes. P&L assumes one contract entered at the ask and exited at the bid after estimated round-trip fees.
It reports expectancy, win rate with a Wilson 95% interval, profit factor and maximum drawdown by ticker, DTE, strategy and volatility regime. Results are descriptive from 30 measured episodes and reliable only with at least 100, 80% measurement coverage and enough distinct sessions and tickers (10 each by default); no score change is eligible until an untouched chronological holdout passes. A separate shadow-only experiment compares candidates with near-miss and control episodes at 15- and 60-minute horizons.
Limitations
Any sample describes only its own period. Overlapping signals are correlated, so the effective sample is smaller than the count; Edge Validation notes that cross-ticker, time and regime correlation has not yet been estimated and size-aware slippage has not yet been measured. Markets change, and past results do not guarantee future results.
Frequently asked questions
What is the difference between backtesting and forward testing?
A backtest replays a rule on historical data. A forward test records signals as they occur and measures outcomes afterwards, which rules out look-ahead bias by design but takes longer to build a useful sample.
How many trades does a backtest need?
There is no universal number; it depends on how variable the results are, and confidence intervals show how precise an estimate is. Edge Validation’s own policy treats fewer than 30 measured episodes as insufficient and needs at least 100 for a reliable baseline.
Does a profitable backtest mean a strategy will work?
No. Overfitting, costs, changing market regimes and luck can all make past results unrepresentative. Past results do not guarantee future results.