Hidden Markov Model Regime: Inferring Market States

Key Takeaways

• An HMM assigns a posterior probability to each hidden regime at every bar by running the forward algorithm, which updates state probabilities without ever using future data.
• A two-state HMM on SPY (2010–2019) found a calm state with 0.7 percent daily mean and 0.7 percent volatility vs a stressed state with -0.12 percent mean and 2.1 percent volatility; the February 2020 stressed probability exceeded 0.90 within days of the COVID shock.
• The most common error is using the Viterbi or smoothed state sequence in backtests, both use future data; only the real-time forward probability is valid for generating live signals.
• Strategies that trade directly on single-bar state flips generate excessive whipsaw; requiring the probability to hold above a threshold for several bars substantially reduces transaction costs.

What It Is

A hidden Markov model regime framework has three ingredients: a finite set of states S = {1, 2, ..., N}, a state transition matrix A where A_{ij} is the probability of moving from state i to state j, and an emission model that gives the distribution of observed returns given the current state. The state sequence is Markov and the emissions are conditionally independent given the states.

For market regime detection, the observed series is usually daily returns, realized volatility, or a vector of both. Two-state HMMs capture a bull/bear or calm/stressed dichotomy. Three-state versions add a transitional or crisis regime.

The Intuition

A pure regression on market returns treats every day as drawn from the same distribution. That assumption fails whenever the world changes, which in finance is often. An HMM keeps a probability distribution over "which regime is active today" and updates it each time a new return arrives.

The payoff for trading is a soft regime label you can trust. Instead of a rule like "VIX above 25 means risk-off," the HMM reads the joint behavior of several observables and returns a posterior probability. That posterior naturally smooths over single-day noise and responds faster to clusters of evidence.

How It Works

Let y_1, ..., y_T be the observed return series and s_1, ..., s_T the hidden state sequence. The model parameters are lambda = (A, B, pi), where A is the N x N transition matrix, B is the emission distribution (e.g. Gaussian per state), and pi is the initial state distribution. Three classic problems are solved:

1. Likelihood P(y | lambda) via the forward algorithm.
2. Most likely state sequence via the Viterbi algorithm.
3. Parameter estimation via the Baum-Welch (expectation-maximization) algorithm.

The forward recursion computes alpha_t(i) = P(y_1, ..., y_t, s_t = i | lambda):

alpha_1(i)   = pi_i * b_i(y_1)
alpha_{t+1}(j) = [ sum_i alpha_t(i) * A_{ij} ] * b_j(y_{t+1})

The filtered regime probability at time t is:

P(s_t = i | y_1..t) = alpha_t(i) / sum_j alpha_t(j)

For Gaussian emissions, b_i(y) = N(y | mu_i, sigma_i^2). Baum-Welch iterates expectation and maximization steps until the log-likelihood stops improving.

Worked Example

Fit a two-state Gaussian HMM on daily SPY returns from 2010 to 2019. Suppose Baum-Welch converges to:

• State 1 (calm): mu = +0.06 percent per day, sigma = 0.7 percent, self-transition 0.98
• State 2 (stressed): mu = -0.12 percent per day, sigma = 2.1 percent, self-transition 0.90

Expected duration in state 1 is 1 / (1 - 0.98) = 50 days. In state 2 it is 10 days. On February 27, 2020, the forward pass returns P(state 2) = 0.15. After three more large down days, that probability exceeds 0.90 and stays there through April. A strategy that cut beta to 0.3 when P(state 2) > 0.80 and restored full exposure when it fell below 0.30 would have bypassed the worst drawdown of the year.

Common Mistakes

1. Confusing HMM with Markov switching regression. Hamilton's model embeds a regression inside the regime framework. A plain HMM on returns does not. Do not claim one when you fit the other.

2. Starting Baum-Welch from bad initial values. The EM algorithm climbs to a local maximum of the likelihood, not the global one. Random restarts or k-means initialization on observed features reduce the chance of a bad fit.

3. Over-interpreting regime labels. The model does not know what "bull" or "bear" means. After fitting, you must inspect the regime means and volatilities to attach a label. If you re-fit on a different window, the indices may swap.

4. Ignoring regime persistence in the horizon. A regime with self-transition 0.70 has an expected duration of only 3.3 bars. Trading it directly generates whipsaw and costs. Either enforce persistence via a prior in Bayesian HMMs or require the posterior to exceed threshold for several bars.

5. Assuming Gaussian emissions without checking residuals. Within-regime returns still have fat tails. Consider Student-t emissions or GARCH-within-regime extensions if standardized residuals show excess kurtosis.

Frequently Asked Questions

Q: What is a hidden Markov model regime in simple terms? It is a statistical model that assumes the market operates in a small number of unobservable states, such as calm and stressed, and that each state generates returns with its own mean and volatility. The model infers the probability of each state at every bar from the observed return sequence.

Q: How does a hidden Markov model affect investment decisions? It provides a probability-weighted view of the current market state rather than a binary rule like "VIX above 25." That posterior can be used to scale beta, rotate factor exposures, or switch between a risk-on and risk-off portfolio configuration, with transitions that respond to clusters of evidence rather than single-day spikes.

Q: What is a real-world example of HMM regime detection? A two-state HMM trained on SPY daily returns assigned the stressed state a probability above 0.90 by March 2020, after only a few days of large negative returns. A strategy cutting beta to 0.3 when stressed probability exceeded 0.80 would have bypassed most of the COVID drawdown and restored full exposure by April when the probability fell below 0.30.

Q: How can investors implement an HMM without look-ahead bias? Use only the filtered probability computed by the forward algorithm, the real-time estimate that conditions solely on returns up to the current bar. Do not use the Viterbi sequence (which finds the globally optimal path using all data) or the backward smoother outputs in any live signal; both incorporate future information.

Q: How is a hidden Markov model different from a regime-switching regression? Hamilton's regime-switching model embeds a regression structure in each state, which makes the state dynamics depend on specific economic covariates. A plain HMM simply specifies an emission distribution, typically Gaussian, per state, without covariates. For pure market-state detection from return series, the two approaches produce similar results; the regime-switching regression is more appropriate when macro variables are used as inputs.

Sources

1. Rabiner, L.R. (1989). "A Tutorial on Hidden Markov Models and Selected Applications in Speech Recognition." Proceedings of the IEEE 77(2), 257-286. https://www.ece.ucsb.edu/Faculty/Rabiner/ece259/Reprints/tutorial%20on%20hmm%20and%20applications.pdf
2. Hamilton, J.D. (1989). "A New Approach to the Economic Analysis of Nonstationary Time Series and the Business Cycle." Econometrica 57(2), 357-384. https://users.ssc.wisc.edu/~behansen/718/Hamilton1989.pdf
3. Bhar, R. and Hamori, S. (2004). Hidden Markov Models: Applications to Financial Economics. Springer. https://link.springer.com/book/10.1007/978-1-4020-7940-1
4. Nystrup, P., Madsen, H., Lindstrom, E. (2017). "Long Memory of Financial Time Series and Hidden Markov Models with Time-Varying Parameters." Journal of Forecasting. https://onlinelibrary.wiley.com/doi/10.1002/for.2447

Disclaimer

This article is educational content only and is not financial advice. Nothing here is a recommendation to buy, sell, or hold any security. Consult a licensed advisor before making investment decisions.