Forecasting Non-Stationary Processes

18 Apr 2012 13:22

---

Some non-stationary processes are in fact easy to forecast: periodic ones, for example, are strictly speaking not stationary. An ergodic Markov chain started far from its invariant distribution is also non-stationary, but easy to predict (it will approach the stationary distribution). Both of these cases are conditionally stationary, which I think is all that's really needed.

What's more interesting is the problem of so to speak really non-stationary processes. It's hard to imagine that there is any way to truly predict an arbitrary non-stationary process. (Basically: as soon as you think you have established a trend-line, the Adversary can always reverse the trend, without creating any problems of consistency with earlier data.) If you can constrain the class of allowable non-stationary processes, however, then something might be possible. Alternately, one might lower expectations, not to actually predicting well, but to predicting with low regret.

I actually have an Idea about using model averaging here, but need to find the time to work on it.

See also: Ensemble Methods in Machine Learning; Low-Regret Learning; Time Series; Universal Prediction

Recommended (very misc):
• S. Caires and J. A. Ferreira, "On the Non-parametric Prediction of Conditionally Stationary Sequences", Statistical Inference for Stochastic Processes 8 (2005): 151--184
• R. Dahlhaus, "Fitting Time Series Models to Nonstationary Processes", Annals of Statistics 25 (1997): 1--37
• Mark Herbster and Manfred K. Warmuth, "Tracking the Best Expert", Machine Learning 32 (1998): 151--178 [PS version via Dr. Herbster]
• Elad Hazan and Satyen Kale, "Extracting certainty from uncertainty: regret bounded by variation in costs", Machine Learning 80 (2010): 165--188
• Jeremy Zico Kolter and Marcus A. Maloof
  • "Dynamic Weighted Majority: An Ensemble Method for Drifting Concepts", Journal of Machine Learning Research 8 (2007): 2755--2790
  • "Using Additive Expert Ensembles to Cope with Concept Drift", ICML 2005 [PDF reprint via Kolter]
• Claire Monteleoni and Tommi S. Jaakkola, "Online Learning of Non-stationary Sequences", pp. 1093--1100 in NIPS 2003 (vol. 16) [Figuring out at what rate to switch between experts]
• Maxim Raginsky, Roummel F. Marcia, Jorge Silva and Rebecca M. Willett
  • "Sequential Probability Assignment via Online Convex Programming Using Exponential Families" [ISIT 2009; PDF]
  • "Sequential anomaly detection in the presence of noise and limited feedback", arxiv:0911.2904
• Kyupil Yeon, Moon Sup Song, Yongdai Kim, Hosik Choi, Cheolwoo Park, "Model averaging via penalized regression for tracking concept drift", Journal of Computational and Graphical Statistics online before print (2010)

Modesty forbids me to recommend:
• CRS, Abigail Z. Jacobs, Kristina Lisa Klinkner and Aaron Clauset, "Adapting to Non-stationarity with Growing Expert Ensembles", arxiv:1103.0949

To read:
• István Berkes, Lajos Horváth and Shiqing Ling, "Estimation in nonstationary random coefficient autoregressive models", Journal of Time Series Analysis 30 (2009): 395--416 ["the unit root problem does not exist in the RCA model"!]
• Satish T. S. Bukkapatnam and Changqing Cheng, "Forecasting the evolution of nonlinear and nonstationary systems using recurrence-based local Gaussian process models", Physical Review E 82 (2010): 056206
• Alexey Chernov, Vladimir Vovk, "Prediction with Advice of Unknown Number of Experts", arxiv:1006.0475
• Michael P. Clements and David F. Hendry, Forecasting Non-Stationary Economic Time Series
• Rainer Dahlhaus and Wolfgang Polonik, "Empirical spectral processes for locally stationary time series", Bernoulli 15 (2009): 1--39, arxiv:902.1448
• Ching-Kang Ing, Jin-Lung Lin, Shu-Hui Yu, "Toward optimal multistep forecasts in non-stationary autoregressions", Bernoulli 15 (2009): 402--437 = arxiv:0906.2266 ["Optimal" assuming that you know you are facing a linear AR model.]
• Yan Karklin and Michael S. Lewicki, "A Hierarchical Bayesian Model for Learning Nonlinear Statistical Regularities in Nonstationary Natural Signals", Neural Computation 17 (2005): 397--423
• Zudi Lu, Dag Johan Steinskog, Dag Tjostheim and Qiwei Yao, "Adaptively Varying-Coefficient Spatiotemporal Models", Journal of the Royal Statistical Society B 71 (2009): 859--880 [PDF preprint]
• Joaquin Quinonero-Candela, Masashi Sugiyama, Anton Schwaighofer and Neil D. Lawrence (eds.), Dataset Shift in Machine Learning
• Joshua W. Robinson, Alexander J. Hartemink, "Learning Non-Stationary Dynamic Bayesian Networks", Journal of Machine Learning Research 11 (2010): 3647--3680
• Masashi Sugiyama and Motoaki Kawanabe, Machine Learning in Non-Stationary Environments: Introduction to Covariate Shift Adaptation [blurb]
• P. F. Verdes, P. M. Granitto and H. A. Ceccatto, "Overembedding Method for Modeling Nonstationary Systems", Physical Review Letters 96 (2006): 118701
• Ou Zhao, Michael Woodroofe, "Estimating a monotone trend", arxiv:0812.3188
• Shuheng Zhou, John Lafferty, Larry Wasserman, "Time Varying Undirected Graphs", arxiv:0802.2758

To write:
• CRS + co-conspirators to be named later, "This Time Is Different"