Time-Series Model Selection and Hyperparameter Tuning
Chronological resampling, nested evaluation, gap design, and leakage control
QM020 · Machine Learning · Advanced
Core idea. Hyperparameter tuning is part of the forecasting algorithm. Its folds, preprocessing, target maturity, and selection rule must use only information admissible at each simulated forecast origin.
Use it for. Choosing penalty strengths, tree depth, learning rates, feature-selection thresholds, model families, or ensemble settings in time-ordered prediction problems.
It does not establish. That one resampling scheme is universally correct, that ordinary K-fold cross-validation is mathematically invalid in every time-series problem, or that tuning protects against a poorly chosen final evaluation design.
The Question
A machine-learning model is rarely just an estimator with fixed settings. Researchers also choose hyperparameters, preprocessing rules, features, and sometimes the model family itself.
If those choices are made using future observations, the final forecast evaluation can be biased even when the final coefficient fit uses only past data.
The real question is therefore:
How should model selection be embedded inside a time-series forecasting experiment so that both fitting and tuning obey the information available at each forecast origin?
Why It Matters
Model selection can overfit a finite validation criterion just as coefficient estimation can overfit a training sample. Cawley and Talbot emphasize that optimizing a noisy model-selection criterion can itself create selection bias (Cawley and Talbot 2010). Varma and Simon show why using the same cross-validation process both to select a model and to report its performance can produce optimistically biased error estimates (Varma and Simon 2006).
In time series there is an additional issue: chronology. Future observations can contaminate training through fold construction, preprocessing, feature engineering, or labels whose outcomes have not matured by the forecast origin.
Intuition
Think of a historical backtest that issues a forecast at the end of each month.
At a particular origin \(t\):
- all raw inputs must have been available by \(t\);
- any target used for training must already be fully realized by \(t\);
- scalers, imputers, feature selectors, and dimensionality reductions must be fitted only on the admissible training data;
- hyperparameter selection must use only folds contained within that admissible history; and
- the forecast for \(t+1\) or a longer horizon must not influence any of those choices.
A model that obeys these rules can still overfit. But it avoids the more basic mistake of training or tuning on unavailable information.
The Method
Separate tuning from evaluation
Let \(\theta\) denote model parameters fitted from data and \(\lambda\) denote hyperparameters selected by a validation rule.
A clean design distinguishes:
- inner estimation/tuning: choose \(\lambda\) and fit \(\theta\) using only admissible historical data;
- outer evaluation: score the resulting procedure on observations that did not participate in selecting \(\lambda\).
If the same observations determine \(\lambda\) and are then used to claim out-of-sample performance, the performance estimate is conditional on selection using those observations.
Expanding and rolling validation
For ordered observations \(1,\ldots,T\), an expanding validation design might use:
\[ \mathcal T_k=\{1,\ldots,t_k\}, \qquad \mathcal V_k=\{t_k+g+1,\ldots,t_k+g+h\}, \]
where \(g\ge 0\) is a gap and \(h\) is validation-block length.
A rolling design replaces the expanding training set with a fixed-length window:
\[ \mathcal T_k=\{t_k-w+1,\ldots,t_k\}. \]
The choice between rolling and expanding windows reflects a modeling view about stability versus adaptation; it is not a purely mechanical tuning detail. QM002 — Rolling vs Expanding Windows develops that distinction.
The role of a gap
A gap excludes observations immediately before the validation block from model fitting. In software such as scikit-learn, TimeSeriesSplit(gap=...) implements this index separation (scikit-learn developers, n.d.-b).
But a gap should not be chosen by habit. Its purpose depends on the data-generating and labeling process. Examples include:
- an \(h\)-period forward-return label that is not fully realized at the forecast cutoff;
- overlapping target windows;
- features whose construction uses a trailing window extending beyond an apparent row timestamp; or
- operational delays between observation and actual availability.
The relevant exclusion can therefore be driven by label maturity and information availability, not just by serial correlation.
Is Ordinary K-Fold Always Wrong for Time Series?
No. That blanket statement is too strong.
Bergmeir, Hyndman, and Koo show that standard K-fold cross-validation can be valid for evaluating purely autoregressive prediction under conditions that yield uncorrelated errors (Bergmeir et al. 2018). Their result is an important warning against turning a practical default into a universal theorem.
For many financial/economic workflows, however, random K-fold remains a poor default because:
- the process may be nonstationary;
- features may include revised or delayed data;
- target windows may overlap;
- the intended deployment is explicitly forward through time; and
- the researcher wants to mimic the information set available at historical forecast origins.
Thus the recommended default is chronological resampling because it matches the operational forecasting problem, not because every non-chronological split is mathematically invalid in every autoregressive setting.
Preprocessing Belongs Inside the Fold
Suppose a StandardScaler is fitted using all observations before cross-validation. The validation fold then influences the training-feature means and standard deviations. That is leakage.
The correct pattern is:
- split the fold;
- fit preprocessing on the fold’s training observations;
- transform training and validation data with that fitted object;
- fit the predictive model; and
- evaluate on the validation fold.
Pipelines make this discipline easier to enforce (scikit-learn developers, n.d.-a). The same logic applies to imputation, PCA, feature selection, winsorization thresholds learned from data, and target transformations whose parameters are estimated.
Nested Historical Evaluation
For a final historical performance estimate, the entire tuning procedure should be rerun inside each outer forecast origin.
At outer origin \(t\):
- construct the admissible history \(\mathcal H_t\);
- build inner chronological folds within \(\mathcal H_t\);
- evaluate candidate hyperparameters only on those inner folds;
- choose the hyperparameter using a predeclared rule;
- refit on the admissible history using that choice; and
- forecast the outer evaluation target.
This is computationally heavier than tuning once on the full sample, but it answers a different and more defensible question: what would this model-selection procedure have produced using only historical information?
How to Choose the Objective
The tuning loss should match the research objective closely enough to be meaningful.
For point forecasts, common choices include MSE and MAE. For probabilistic forecasts, proper scoring rules are more appropriate. For portfolio systems, one may be tempted to tune directly on downstream Sharpe or CAGR. That can be legitimate as a decision-focused objective, but it creates additional sampling noise, path dependence, and multiple-testing risk.
If a model is selected on portfolio performance, the research should state that explicitly rather than describing the exercise as pure forecast-error tuning.
Financial / Economic Example
Suppose a monthly cross-asset model predicts next-month returns from lagged price features. Candidate Ridge penalties are \(\{0.1,1,10,100\}\).
At each outer month-end:
- only completed historical targets are admitted;
- inner folds move forward through the available history;
- the scaler and Ridge model are refitted inside each fold;
- the penalty with the lowest average inner validation loss is selected; and
- the selected model is refitted on the current history and used to produce the next forecast.
The resulting outer forecasts can then be mapped to ranks and portfolio decisions without allowing later months to influence the penalty chosen at earlier origins.
Implementation
The accompanying example implements a small nested expanding-window tuner. Its splitter is deliberately explicit rather than presented as a universal library recipe.
A simplified pattern is:
for outer_origin in outer_origins:
X_hist, y_hist = admissible_history(outer_origin)
best = None
for candidate in grid:
fold_losses = []
for train_idx, val_idx in inner_splits(X_hist):
model = make_pipeline(StandardScaler(), Ridge(alpha=candidate))
model.fit(X_hist[train_idx], y_hist[train_idx])
fold_losses.append(loss(y_hist[val_idx], model.predict(X_hist[val_idx])))
# choose candidate from inner-fold losses onlyHands-on Lab
From the QM020 folder:
python labs/python/qm020_time_series_tuning.pyThe synthetic series contains a mild regime change. Compare a single full-history tuning decision with repeated nested historical tuning. The exercise is illustrative; neither procedure is guaranteed to dominate in all samples.
Common Mistakes
- Randomly shuffling a deployment-mimicking forecasting problem. The folds no longer reproduce the historical information flow.
- Using the final test period to pick hyperparameters. The test is no longer untouched.
- Fitting preprocessing before splitting. Validation information leaks into training transformations.
- Ignoring target maturity. A row dated \(t\) can still contain a target whose realization occurs after \(t\).
- Using
gap=0automatically. The appropriate gap depends on the label and information clock. - Using a large gap automatically. Throwing away data without a design reason can reduce efficiency without fixing the real timing issue.
- Reporting the best of many tried designs without accounting for the search. Model-selection risk includes researcher degrees of freedom, not only software hyperparameters.
- Claiming standard K-fold is universally invalid for time series. The literature contains important conditional validity results (Bergmeir et al. 2018).
When Not to Use It
Nested historical tuning can be unnecessary when a model has no meaningful hyperparameters, when a setting is fixed ex ante by design, or when the sample is too small for a stable inner/outer separation. In those cases, a simpler predeclared model may be more credible than an elaborate but weakly identified tuning process.
Used in SlackQuant Research
- Equity Alpha — benchmark-aware global equity ETF alpha system using point-in-time selection, rank-based decisions, and active-risk evaluation.
Forecast-to-Rank Allocation (F2R) applies chronological model-selection discipline when conventional supervised forecasting candidates are compared over historical decision dates.
- The Decision Value of Price and Macro Information uses time-ordered forecasting evaluation and model-selection logic.
- F2R Model-Adoption Research compares forecasting architectures under a common historical information and evaluation clock.
The public Method documents the reusable information-set and tuning discipline without disclosing protected live model settings.