A time-series foundation model, from scratch
How far does a small transformer get on a public zero-shot forecasting benchmark when it is pretrained from scratch, on data checked for overlap with the benchmark, and judged by rules fixed in advance?
A decoder-only transformer, its tokenizers, training loop and sampling written in plain PyTorch; pretrained on a pinned, leakage-audited corpus plus synthetic Gaussian-process series; scored zero-shot on held-out datasets with the official GIFT-Eval scoring code; with a pre-registered study of daily crypto-market volatility across the 2020 and 2022 market breaks.
Problem
Foundation models for time series are pretrained on large collections of series and asked to forecast new ones with no further training. Benchmarks such as GIFT-Eval make them comparable, but results are only meaningful if no benchmark data leaks into pretraining and if the evaluation is fixed before the scores are seen. This project builds the whole pipeline itself and holds it to both standards.
Data, and a leakage finding
Pretraining uses 77 datasets (1,087,018,264 points) from GiftEvalPretrain, the corpus its authors describe as non-leaking, plus synthetic Gaussian-process series. Every raw file is pinned by its SHA-256 fingerprint. A content-based check compared every pretraining series with every GIFT-Eval series, at native frequency and after daily aggregation, and found genuine copies:
beijing_air_quality↔kdd_cup_2018_with_missing/H: 59 seriesfred_md↔m4_monthly: 8 seriesm1_monthly↔m4_monthly: 2 seriespedestrian_counts↔m4_hourly: 46 seriestourism_monthly↔m4_monthly: 28 seriesweather↔temperature_rain_with_missing: 499 series
All 801 flagged series were removed, and the re-run on the final corpus finds 0 matches. A first version of the check also produced false matches from linearly interpolated gaps, which look identical in every series once normalised; inspecting each match separated those from real copies.
Method
- Model: decoder-only transformer with pre-norm RMSNorm, rotary position embeddings, SwiGLU and a key/value cache, written without model libraries. Two tokenizers: mean-scaled 4,096-bin quantisation with cross-entropy, and patches with a non-crossing quantile head.
- Training: mixed precision, gradient accumulation, cosine schedule, checkpoints to Google Drive every 10 minutes; a resumed run is bit-identical to an uninterrupted one (tested). Trained on a free Colab T4; profiling showed half of each step went to cumulative-sum kernels, which were replaced by equivalent cheaper operations.
- Evaluation: GluonTS exactly as GIFT-Eval scores the leaderboard, verified by reproducing upstream's Seasonal Naive on every metric. The split, metrics, baselines, primary comparison and decision rule were committed and tagged in git before any test-set scoring; model selection used only held-out validation datasets.
Results
Every model was scored zero-shot on 16 configurations built from datasets held out of pretraining entirely, with the official GIFT-Eval scoring code. Error falls steadily with model size: the largest model, patch_s2, has CRPS 0.605 and MASE 0.840 relative to Seasonal Naive, so it beats the naive baseline on both, and the pre-registered rule selects it.
| Run | Parameters | CRPS | MASE |
|---|---|---|---|
| patch_s0 | 0.9M | 0.740 | 1.139 |
| patch_s1 | 2.9M | 0.636 | 0.910 |
| patch_s2 | 10.1M | 0.605 | 0.840 |
Validation loss falls with training compute as a power law with exponent 0.058 (95% bootstrap interval 0.051 to 0.064) across 3 patch-model sizes.
For context, the published GIFT-Eval leaderboard results of other models are below, recomputed from upstream's result files. They are other models' scores on the benchmark's test split, not ours; scoring our model there is pre-registered future work.
| Model | CRPS | MASE | 80% coverage |
|---|---|---|---|
| TimesFM-2.5, 231M (reported) | 0.490 | 0.705 | n/a |
| FlowState, 9.1M (reported) | 0.502 | 0.726 | n/a |
| AutoARIMA (reported) | 0.912 | 1.074 | n/a |
| Seasonal Naive (run by us) | 1.000 | 1.000 | 0.90 |
| AutoETS (reported) | 7.489 | 1.212 | n/a |
Market volatility study
Daily realized volatility of BTCUSDT and ETHUSDT, computed from Binance Vision 5-minute data, forecast 22 days ahead from every day of 2020 to 2026, with the 2020 crash and the 2022 collapses as stress regimes. The design was pre-registered separately.
| Model | Loss ratio | 80% coverage |
|---|---|---|
| Random walk | 1 (reference) | 0.80 |
| HAR-RV | 0.755 [0.731, 0.783] | 0.81 |
The baselines were run as designed; HAR-RV, the standard volatility model, is the bar for the transformer in this study. These are statistical forecast scores only; no trading performance is claimed.
What didn't work, and limits
- The first GPU run diverged: future patches reach about a million in scaled units when the context is near zero, overflowing half precision. Clipping fixed it; the broken runs are discarded and recorded in the repository.
- Prediction intervals are too narrow, worst at long horizons, because longer forecasts feed back medians.
- Scope was cut before test scoring to fit a free GPU and a deadline, and the cuts are recorded in the repository: three model sizes instead of five, the second tokenizer trained but not scored, and one seed, so seed-to-seed variation is not measured. The pre-registered test-split scoring of our model is future work, so no benchmark ranking is claimed.
- The leakage check finds exact and rescaled copies only, not noisy or re-sampled ones.
Engineering
- Every document in the repository is generated from committed result files, and a test fails if one contains a number that is not in them. This page is generated the same way.
- A tiny configuration runs the whole pipeline, from verified download to evaluation, in continuous integration on every push.
Python 3.12 · PyTorch · NumPy · pandas · Parquet · GluonTS · statsforecast · pytest · Hypothesis · ruff · mypy · uv · GitHub Actions · Google Colab