A strong in-sample Sharpe often reflects search bias, not alpha. The papers below cover backtest overfitting, adjusted significance tests, and open replication — the minimum reading before you trust a new strategy pipeline.
Overfitting and significance
When you test many variants, the best backtest is almost always lucky. Bailey and Lopez de Prado quantify this; Harvey and Liu extend the logic to published factor discoveries.
The Probability of Backtest Overfitting
CSCV and PBO metrics for estimating how much of a backtest edge comes from selection bias rather than signal.
The Deflated Sharpe Ratio
Adjusts Sharpe for non-normality, sample length, and the number of trials run before picking a strategy.
False (and Missed) Discoveries in Financial Economics
Updated framework for multiple testing in published finance research. Directly relevant when mining factors or ML features.
In-Sample and Out-of-Sample Sharpe Ratios for Linear Predictive Models
Closed-form link between in-sample and out-of-sample Sharpe ratios for linear strategies. Shows replication ratios collapse for many-asset, weak-signal portfolios and improve with longer training windows.
A Reality Check for Data Snooping
Bootstrap reality check for the best of many backtested models. Foundational when you run parameter or rule search before picking a strategy.
The Minimum Backtest Length
Minimum sample length needed for a Sharpe to be statistically distinguishable from noise given trial count and skew. Pairs with the deflated Sharpe entry above.
Data-Snooping, Technical Trading Rule Performance, and the Bootstrap
Early large-scale demonstration that the best technical rule from a huge search can look profitable in sample while failing out of sample without a snooping adjustment.
Replication and open data
Modern factor and signal research should be reproducible. Chen and Zimmermann's open-source asset pricing project is the benchmark for checking whether a backtest survives clean data and code.
Open Source Cross-Sectional Asset Pricing
Replicates hundreds of cross-sectional signals with shared code and data. Essential before claiming a new anomaly is real. Open data portal
Recent applied work
Deep Reinforcement Learning for Cryptocurrency Trading: Practical Approach to Address Backtest Overfitting
Walk-forward and hold-out design for RL trading systems. Useful template for any strategy with heavy hyperparameter search.
Before you ship
- Report how many variants you tested before picking the winner
- Use purged or walk-forward validation, not a single random split
- Replicate on an independent data vendor when possible
- Apply transaction costs and capacity constraints early
Pair this list with our ML signals reading list for validation methodology in production pipelines.