Forecasting S&P 500 volatility with deep learning
MSc dissertation, University of Liverpool. A head-to-head between econometric volatility models, an LSTM, and a hybrid of the two.
The question
Volatility forecasting has two traditions that mostly ignore each other. Econometrics has spent forty years on the GARCH family — models with strong structural assumptions about how volatility clusters and decays. Machine learning brings sequence models that assume almost nothing and learn the pattern from data.
The obvious move is to combine them: let the econometric model capture the structure it is good at, and hand the leftovers to a neural network. That was my hypothesis. I wanted to know whether a hybrid genuinely beats either component on its own, on daily S&P 500 returns.
Approach
I ran it as a bake-off rather than trying to prove one model right.
- Econometric baselines. ARCH, GARCH, EGARCH and GJR-GARCH, fitted and compared on AIC and RMSE. The asymmetric models matter here because equity volatility responds more sharply to falls than to rises.
- Deep learning benchmark. A standalone LSTM on the historical return series, with VIX added as a market-sentiment feature — forward-looking implied volatility rather than purely backward-looking realised volatility.
- The hybrid. Take the residuals from the winning GARCH model — the part the econometric model could not explain — and train the LSTM on those.
What I found
The hybrid lost. The standalone LSTM, fed the raw series plus VIX, outperformed both the econometric baselines and the EGARCH–LSTM hybrid I had built the project around.
Residuals from a misspecified model are not clean signal. They are signal plus whatever the econometric model got wrong. By training the LSTM on GARCH residuals I was handing it a target contaminated by the baseline's own errors, and the network spent its capacity learning those errors rather than learning the market.
The standalone LSTM never had that handicap. Given the raw series and a genuinely informative extra feature in VIX, it found the structure directly.
What I'd do differently
Two things, and I'd rather say them than have someone find them.
- I let the scope run. I kept adding model variants until my supervisor told me, in as many words, to stop building and start writing. The project was better for the constraint. I now treat "what is the smallest version that answers the question" as the first step rather than the last.
- I picked the architecture before testing the premise. The hybrid was the plan from day one, so the first real test of whether hybridising was a good idea happened far too late. Cheap falsification first, expensive build second.
Why this one matters to me
It is the last substantial piece of work where I understood every line I wrote. Not "it ran" — understood. That is the bar I hold myself to, and it is why a negative result is on this page instead of quietly missing from it.