Financial Machine Learning: Methods for Return Prediction, Factor Models, and Portfolio Choice

Machine learning has become an indispensable tool for understanding financial markets. Two conditions make finance particularly fertile ground for ML methods: information sets are large, and functional forms relating information to prices are unknown. This survey covers the key methods and findings across return prediction, factor models, and portfolio choice.

Why Finance Needs Machine Learning

Asset prices are predictions. A price reflects investors’ discounted expectations of future payoffs given their information set. Researchers face two fundamental challenges in studying this relationship: they cannot observe all relevant conditioning information, and they cannot specify the correct functional form linking information to prices.

Traditional econometric approaches impose specific functional forms motivated by theory, then estimate parameters within those constraints. Machine learning takes a different approach—it uses high-dimensional models, regularization to prevent overfitting, and efficient algorithms to search among many specifications. As Breiman (2001) argued, focusing on prediction accuracy when the true data model is unknown distinguishes ML from classical statistics.

Finance presents three specific challenges for ML:

• Small data: Macro-finance time series contain only hundreds of observations, unlike other ML domains with effectively unlimited data
• Low signal-to-noise ratios: Market efficiency competes away return predictability, so signals are weak
• Structural instability: Markets evolve, making previously reliable patterns obsolete

Economic theory helps address these challenges. Theory and model parameters substitute for each other—more structure means fewer parameters to estimate.

The Virtue of Model Complexity

Traditional econometrics favors parsimony. Machine learning challenges this directly.

Kelly et al. (2022) show theoretically that, in the realistic case of misspecified models, expected out-of-sample forecasting and portfolio performance are increasing in model complexity. The intuition: misspecification bias from using too simple a model is more costly than the shrinkage bias introduced by overparameterization.

This “virtue of complexity” emerges from two effects:

1. Larger models better approximate the unknown true data-generating process
2. In overparameterized models (parameters > observations), the minimum-norm solution acts as implicit regularization—the larger the model, the stronger the implicit shrinkage

The result challenges the parsimony dogma. With appropriate explicit shrinkage (ridge regularization), Sharpe ratios of machine learning portfolios increase monotonically with model complexity.

A related concept is the complexity wedge—the gap between in-sample and out-of-sample performance. This wedge has two components: traditional overfitting (in-sample performance inflated relative to truth) and limits to learning (out-of-sample performance below the true model’s). Even with the best ML methods, attainable Sharpe ratios are attenuated by roughly an order of magnitude relative to the true data-generating process.

Return Prediction

Linear Benchmarks

The foundational panel model regresses next-period returns on stock characteristics:

Ri,t+1 = β’zi,t + εi,t+1

Haugen and Baker (1996) and Lewellen (2015) established that linear models using dozens of predictors produce significant out-of-sample R² (~1% monthly) and strong trading strategy performance (Sharpe ratios above 1.7 equal-weighted). These results demonstrate that combining many signals adds value beyond single-characteristic sorts.

Penalized Regression

Unregularized OLS with many predictors fails dramatically—Gu et al. (2020) show an R² of −35% monthly when using ~1,000 predictors without regularization. Elastic net penalization (combining L1 and L2 penalties) reverses this, producing positive R² and Sharpe ratios near 1.3.

Lasso’s sparsity-inducing geometry also provides variable selection. Freyberger et al. (2020) find that fewer than half of commonly studied stock signals have independent predictive power once nonlinear effects are accommodated through a penalized generalized additive model.

Dimension Reduction

Principal components regression (PCR) and partial least squares (PLS) reduce high-dimensional predictors to low-dimensional factors before forecasting. PLS improves on PCR by selecting components that maximize predictive correlation with the target, not just variance among predictors. Kelly and Pruitt (2013) achieve out-of-sample R² of 13% for annual market returns using PLS applied to the cross section of valuation ratios.

Neural Networks

Neural networks are universal approximators that can capture nonlinear interactions among predictors. Gu et al. (2020) compare models on a panel of US stocks from 1957–2016:

Three findings stand out. First, regularization is essential—unregularized linear models fail. Second, nonlinear models, especially neural networks, outperform linear alternatives. Third, the improvement is especially large for large-cap stocks (R² of 0.70%), ruling out the explanation that ML exploits small-stock illiquidity.

Neural network predictions also reveal important interaction effects. The reversal effect is strongest and linear for small stocks but concave for mega-caps. These nonlinearities explain why neural networks outperform even well-regularized linear models.

Alternative Data

Text data: Supervised sentiment extraction methods (Ke et al., 2019) outperform commercial sentiment vendors and dictionary-based approaches. Large language models (BERT, GPT) provide richer text representations that capture contextual meaning, further improving return predictions beyond bag-of-words methods.

Image data: Jiang et al. (2022) apply convolutional neural networks to OHLC price charts, achieving equal-weighted Sharpe ratios of 7.2 on weekly rebalancing—far exceeding standard momentum and reversal strategies. The CNN detects a previously undocumented pattern: stocks whose latest close price sits near the bottom of their recent high-low range appreciate in the following week.

Factor Models and Risk-Return Tradeoffs

Unconditional Models

The Arbitrage Pricing Theory provides a blueprint for data-driven factor analysis. PCA on the return matrix estimates latent factors and loadings without imposing specific factor identities. While PCA struggles on individual stock panels, it succeeds on panels of anomaly portfolios, pricing them with small errors (Kelly et al., 2020; Kozak et al., 2018).

For non-tradable factors (consumption growth, liquidity, intermediary capital), the three-pass estimator (Giglio and Xiu, 2021) marries PCA with Fama-MacBeth regression. It estimates latent factors in the first pass, uses them as controls in cross-sectional regression in the second pass, and projects the non-tradable factor onto estimated factors in the third pass. This approach eliminates omitted variable bias and measurement error that plague conventional two-pass regressions.

Conditional Models: IPCA

Instrumented PCA (Kelly et al., 2020) links factor loadings directly to observable characteristics:

βi,t = Zt Γβ

This parameterization achieves conditional factor models without separately estimating betas for each asset-period. IPCA delivers similar total R² to observable factor models (CAPM, Fama-French) while using 95% fewer parameters, and substantially higher predictive R². Most importantly, IPCA reduces the number of characteristic-sorted portfolios with significant alpha from 29 (under the six-factor Fama-French-Carhart model) to just 4.

Generalizing to nonlinear beta functions via neural networks (the conditional autoencoder of Gu et al., 2020), predictive R² improves further while total R² stays similar—the nonlinear model provides a more accurate description of conditional risk compensation.

Alphas and Multiple Testing

Da et al. (2022) show that any feasible trading strategy’s Sharpe ratio is bounded by the posterior estimate of alpha given historical data. With a 27-factor model, the estimated infeasible Sharpe ratio exceeds 2.5, roughly four times the feasible Sharpe ratio (~0.5) achievable by machine learning strategies. This gap suggests the APT works reasonably well—high theoretical Sharpe ratios are inaccessible due to statistical limits to learning.

For multiple hypothesis testing across hundreds of anomalies, controlling false discovery rates is essential. Jensen et al. (2021) demonstrate that a Bayesian hierarchical approach to multiple testing—which shrinks alpha estimates toward zero and borrows strength across factors—delivers better economic performance than conservative FDR control methods.

Optimal Portfolio Construction

The Plug-In Problem

The classical Markowitz solution w = (1/γ)Σ⁻¹μ* requires knowing the return distribution. Kan and Zhou (2007) show the expected utility loss from using sample estimates grows with N (number of assets) and is severe even at moderate dimensions. Crucially, the plug-in estimator is inadmissible—simple tweaks like tilting toward the risk-free asset uniformly improve expected utility.

This inadmissibility reveals a fundamental insight: once estimation uncertainty enters the problem, the portfolio rule that maximizes out-of-sample utility cannot be pinned down analytically. This ambiguity is precisely what machine learning addresses.

Maximum Sharpe Ratio Regression

Britten-Jones (1999) established that the OLS coefficient in the regression 1 = w’Rt + ut is proportional to the tangency portfolio. This maximum Sharpe ratio regression (MSRR) formulation integrates estimation and utility maximization in one step, and immediately admits machine learning extensions:

• Ridge regularization shrinks factor covariances toward identity, connecting to Kozak et al. (2020)’s shrunk SDF
• Lasso regularization enforces leverage constraints and performs variable selection
• Neural network weight functions parameterize nonlinear mappings from characteristics to portfolio weights

Simon et al. (2022) demonstrate that a neural network MSRR achieves Sharpe ratios 40% above the linear specification using the same predictors (2.5 vs. 1.8). Didisheim et al. (2023) prove theoretically that portfolio Sharpe ratios increase with model complexity under appropriate shrinkage.

SDF Estimation

The equivalence between mean-variance efficiency and SDF pricing connects portfolio construction to asset pricing theory. Kozak et al. (2020) regularize SDF estimation via ridge, showing that ridge shrinkage induces heavier penalization for lower-ranked principal components—economically, this reflects skepticism that low-variance combinations of factors drive large SDF volatility.

Chen et al. (2021) extend this with a GAN-based adversarial framework: one network estimates SDF weights that minimize pricing errors, while a competing network finds the instruments (test assets) that expose the SDF’s worst failures. This adversarial structure provides efficient GMM estimation without specifying instruments in advance.

Trading Costs and Multi-Period Problems

In the presence of trading costs, optimal portfolios depend on expected returns at all future horizons, not just one period ahead. Jensen et al. (2022) resolve the resulting complexity by using economic structure to derive a portfolio rule that depends on an “aim” portfolio—an exponentially weighted average of Markowitz portfolios across horizons—then parameterize this aim portfolio with a random feature neural network. Their one-step, cost-aware algorithm delivers superior net-of-cost performance versus standard two-step approaches that learn portfolios ignoring costs then smooth afterward.

Practical Guidance

Several principles emerge from this literature:

On model selection: Use cross-validation with temporal ordering to avoid information leakage. Embed an embargo period between training and validation samples proportional to serial correlation strength.

On complexity: Include all plausibly relevant predictors. Use rich nonlinear models rather than simple linear specifications. Employ prudent shrinkage, particularly ridge regularization. Evaluate economic performance (Sharpe ratio, utility) rather than statistical fit (R²), since the latter is an unreliable proxy for the former.

On combining ML with economics: Economic structure substitutes for parameters. Use theory to constrain functional forms where available (e.g., IPCA links betas to characteristics; Jensen et al. use the Garleanu-Pedersen portfolio structure) and ML to handle functional form ambiguity where theory is silent.

On evaluation: Out-of-sample performance is the ultimate criterion. Report value-weighted as well as equal-weighted results to confirm that ML captures economically meaningful predictability rather than small-stock microstructure effects.

Conclusion

Machine learning has transformed empirical finance. Neural networks, penalized regression, and dimension reduction methods all outperform unregularized linear models, with nonlinear models providing the largest gains. The virtue of complexity—that more parameters improve out-of-sample performance when accompanied by appropriate regularization—overturns the parsimony dogma that governed econometrics for decades.

The most productive research combines economic structure with machine learning flexibility. Economic theory constrains what relationships to look for; ML determines their precise functional form from data. Neither pure structural modeling nor atheoretical prediction alone captures the full richness of financial markets.

The frontier lies in applying these methods beyond return prediction: to structural economic models, equilibrium analysis, and domains including corporate finance, household finance, and real estate. Start by applying regularized nonlinear models to your prediction problem, evaluate performance out-of-sample using economic criteria, and use economic restrictions to guide where theory can substitute for parameters.