Forecast Interval Lab

How often does a “90% interval” miss?

A forecast of hourly rentals gives one number. If I use that number to think about a busy hour, I also need to see how far it can miss. The data here count rentals across the bike-sharing system; they do not say how many bikes will be available at a particular station.

I used the earlier Bike Hour Forecast study as a starting point for a separate experiment. I trained one new point predictor, chose its checkpoint on the first quarter of 2012, and used the next quarter's errors to set interval widths. For a reader new to prediction intervals, this means drawing a band around a forecast using errors on a separate period, then checking how often later observations land inside it.

The headline looked reassuring: the fixed 90% band covered 3,894 of 4,258 eligible test hours, or 91.45%. Looking month by month changed the story. September covered 87.64%; October covered 86.11%. The 364 missed hours remain in the package, including 288 observations above the band. A system-wide average hid periods with more misses.

I kept those thresholds fixed. Changing them after seeing these months would answer a different question. The hourly windows overlap and seasons change, so this time series does not establish the exchangeability assumption behind the usual marginal coverage statement. The app lets a reader inspect the fixed model and band, or try the rank arithmetic separately. It cannot promise that a particular future hour has a 90% chance of falling inside its interval.

The training and independent rerun were completed for this frozen experiment on November 4, 2026. This publication reuses that model, calibration and complete results; it is not another training run. The dataset and earlier benchmark results were already known.

为什么“90%区间”在某个月仍会频繁漏掉?

预测某个小时会有多少次租车,只给一个数字还不够。我还想知道,这个数字可能偏离多少。这里的数据是共享单车系统的租用总次数,不能据此判断某个站点还剩几辆车。

我沿着前面的 Bike Hour Forecast 做了一次单独实验:重新训练一个点预测模型,用2012年第一季度选轮次,再用第二季度的误差确定区间宽度。可以把它理解为:先根据另一段时间里犯过的错,在预测数字周围画一条带子,再去检查后来的真实记录是否落在里面。

整体数字看起来不错:固定的90%名义区间,在4,258个符合条件的测试小时里覆盖了3,894个,实际是91.45%。但拆到月份,9月只有87.64%,10月只有86.11%。全部364个漏掉的小时都保留了,其中288个真实值高于区间上界。整体平均掩盖了失准更明显的时段。

我没有看完这些月份再改阈值。相邻小时的历史窗口互相重叠,季节也在变化,不能据此认定满足常见覆盖结论所需的可交换性。页面可以运行固定模型和区间,也可以单独试算排名规则;它不能保证未来某一次预测有90%的概率落在区间里。

这次冻结实验及独立重跑在2026年11月4日完成;本次发布沿用原模型、校准与完整结果,不是又训练了一遍。原数据和前一次实验的结果也已经看过,不冒充盲测。下面保留完整方法、所有分母和失败边界。

I trained a small hourly rental predictor with separate model-selection and residual-calibration periods, then measured interval coverage in later months. The browser app runs the selected point model, checks the full stored calibration-score list, constructs three fixed rank intervals and exports complete reports. A separate calculator exposes small-sample, tie and exact-rank behavior on user-supplied scores.

The historical time series has overlapping windows and changing seasons. It is not established exchangeable. Nominal interval levels are targets for the construction, not per-input probabilities or future-month coverage guarantees.

Full eligible test results

Nominal level Error fraction Rank / 2,134 scores Error threshold Covered / 4,258 Observed coverage Mean width (rentals)
95% 1/20 2,029 127.099692 4,037 94.8098% 223.8812
90% (primary) 1/10 1,922 96.621898 3,894 91.4514% 172.9837
80% 1/5 1,708 62.429339 3,511 82.4566% 114.2475

The primary 90% interval misses 364 targets: 76 below and 288 above. Every miss is retained in evaluation-001/all-primary-misses.json. All 12,774 interval rows, all 4,258 point predictions and every original record's eligibility/reason remain available. The point model has MAE 36.2158, RMSE 59.3038 and RMSLE 0.272042 on these same test hours. It was not retuned to improve on the earlier point-only experiment.

Test month Eligible hours Primary observed coverage
2012-07 744 91.5323%
2012-08 744 92.0699%
2012-09 720 87.6389%
2012-10 684 86.1111%
2012-11 670 94.6269%
2012-12 696 96.8391%

Overall coverage near a nominal target does not establish coverage within each month. These finite observations also do not prove or disprove the marginal exchangeable-score theorem. All three levels and every month are reported without a test-based threshold update.

All 40 epochs, complete monthly coverage and width, and four-role inventory

Four separate data roles

The original UCI Bike Sharing source contains 17,379 hourly records from Washington D.C.'s Capital Bikeshare in 2011–2012. Its first-to-last naive calendar span has 165 absent hours. The source files and previous benchmark results were already inspected in the earlier local Bike Hour Forecast work. This is a new four-role interval experiment on a known dataset, not a new or historically blind benchmark.

Role by target timestamp Original targets Eligible 24-hour contexts Omitted
Train: before 2012-01-01 8,645 7,279 1,366
Select: 2012 Q1 2,176 1,986 190
Calibrate: 2012 Q2 2,182 2,134 48
Test: 2012 Q3–Q4 4,376 4,258 118

Only targets with all 24 preceding nominal hours are eligible. No zero filling, interpolation or compression to the previous observed row occurs. All 1,722 omitted targets stay in the 17,379-row inventory, including 24 initial targets. Excluding incomplete contexts can select easier periods. Results cover eligible targets only.

Selection labels choose the epoch; calibration labels are used afterward for absolute residuals and rank thresholds. This separation keeps calibration labels out of checkpoint selection. The observations still overlap in time and shift with season, so this does not establish exchangeability or independence.

Evaluation is rolling one hour ahead: actual earlier calibration/test counts may enter later contexts once they have occurred. It is not a recursive six-month forecast from one fixed origin. Source timestamps have no offset; nominal calendar arithmetic does not reconstruct UTC or daylight saving time. Recorded rentals are not available bikes, unmet demand or validated current-city requirements.

One actual point fit

The fresh seed-321 model is a float64 NumPy network 35 → 32 tanh → 1, with 1,185 parameters. Features are 24 log1p earlier counts, cyclic target hour, Monday-first weekday indicators and cyclic target month. Feature and log-residual population means/standard deviations use eligible training data only. No target-time weather, casual/registered components or future counts enter features.

Training runs all 40 declared epochs, batch 128, Adam 0.002, coupled weight regularization 0.0001, fixed initialization/shuffle seeds. There are 2,280 updates and 291,160 training-row visits. The earliest minimum Q1 selection MAE chooses epoch 39. All 40 trace entries and the final epoch remain available. This is a fresh gradient fit, not a rescore of old saved weights. Three interval levels reuse the same selected model and calibration scores; they are not three models.

The predicted log count is clipped to [0, log1p(100000)] as a numerical guard. This is not a confidence bound or an outcome-support maximum. Interval upper endpoints are not clipped to 100,000.

Exact residual-rank rule

With the selected point predictor fixed, compute s_i = abs(y_i - point_i) on every eligible Q2 calibration target. For integer error fraction a/d, calculate the one-based order-statistic rank:

k = ceil((n + 1) × (d - a) / d)
q = the kth smallest calibration score, when k <= n
q = unbounded, when k > n
interval = [max(0, point - q), point + q]

Integer arithmetic computes the rank without a binary-float ceiling or an interpolated percentile. The lower bound is clipped to zero because the outcome domain is nonnegative. Coverage uses the inclusive score rule abs(truth - point) <= q; endpoint numbers are floating-point representations. Ties are retained and can give conservative coverage.

An unbounded rule serializes threshold: null, unbounded: true, and interval lower: 0, upper: null, width: null. It never substitutes a finite largest score. For example, four scores with a/d = 1/10 require rank five, hence an unbounded interval. With n = 49 and a/d = 29/50, the exact rank is 21; a direct binary-float expression can incorrectly ceil to 22.

This is the absolute-residual instance of the score-rank construction discussed by Angelopoulos and Bates, §2.3 and Appendix D of arXiv:2107.07511v6. Its marginal lower-coverage statement requires exchangeable calibration/new scores with the fitted predictor/scoring rule independent of calibration labels. It is not a promise for each fixed input or every future month. §4.6 discusses distribution drift; this experiment does not implement those weighted/adaptive methods or establish their assumptions. No paper code, prose or figures are copied into the release.

Use the local app

python3 -m http.server 8000 --bind 127.0.0.1
# Open http://127.0.0.1:8000.

The main panel accepts exactly 24 consecutive nominal-hour records, oldest first, each with exactly timestamp and count. Use YYYY-MM-DDTHH:00:00, years 1900–2100 and integer-valued counts 0–100,000; the derived target hour must stay in range. JSON 1 and 1.0 have the same value. Booleans, fractions, missing hours, duplicates, offsets, impossible dates, extra fields and inputs above 64 KiB reject. Editing the context or nominal level clears the previous result/download link.

The point-model byte digest binds the fixed calibration artifact. Its reader recomputes all three ranks from the complete stored 2,134 scores and verifies the saved thresholds. The explicit download retains the full input, point/raw output, selected epoch, rank, score count, threshold, endpoints, model/calibration byte digests and data/code/method attribution. Digests check consistency, not authenticity, exchangeability or model quality.

The separate rank calculator accepts 1–10,000 nonnegative finite scores, a point estimate from 0 to 1e9, and integer 0 < a < d <= 10000. Score values also stay within 0–1e9; score JSON is limited to 1 MiB. It computes arithmetic for a nonnegative outcome domain without changing the fixed study's scores or fitting a model. Inputs are unauthenticated, and the download includes the complete score list. A calculator result does not establish valid calibration data or future coverage.

Both workflows run in browser memory. Static code/model/scores/figure and a requested fixed example are fetched. There is no inference API, storage or autosave. Explicit downloads contain inputs; inspect them before sharing. Arbitrary user histories and scores retain their own terms. No current deployment, new-city validity or adoption is established by local checks.

Reproduce and inspect

Development used Python 3.14.7, NumPy 2.5.4 and Node 24.19.0 on CPU. No pretrained weights, GPU, model API, private/employer/competition material or paid compute was used. Timing is a descriptive host observation, not a performance benchmark.

python3 -m venv .venv
.venv/bin/python -m pip install -r requirements.txt
PYTHONDONTWRITEBYTECODE=1 .venv/bin/python test_core.py
PYTHONDONTWRITEBYTECODE=1 python3 test_quantiles.py
PYTHONDONTWRITEBYTECODE=1 python3 verify_results.py
PYTHONDONTWRITEBYTECODE=1 .venv/bin/python verify_portable.py
node test_intervals.mjs
PYTHONDONTWRITEBYTECODE=1 .venv/bin/python infer.py evaluation-001/example-context.json

The prefit checks include an independently written tiny-network MSE objective/finite differences, explicit Adam history, calendar/window/role boundaries and train-only normalizers. Rank tests exhaust all 120 permutations of five distinct scores: with four calibration values, α = 1/5 covers 96/120 held-out positions, while α = 1/10 is unbounded and covers all 120. Tie and exact-rational boundary cases are separate unit tests; these counts are not bike performance or another fit.

verify_results.py separately reconstructs the entire source/window inventory and training statistics, all 8,378 selection/calibration/test point forecasts, all 2,134 calibration scores, Fraction-based ranks, every test bound/hit/width, all monthly/hourly aggregates and every primary miss using scalar standard-library arithmetic. verify_portable.py compares 6,392 complete calibration/test contexts and all 19,176 fixed interval outputs across JavaScript/Python, plus rank boundaries. These do not fit a new model or substitute for native-browser/independent retraining acceptance.

For a fresh reproduction, copy the release to an independent directory and preserve all original artifacts. Run train.py --out independent-run with the same thread/seed settings. In that copy, update only CALIBRATION-FREEZE.json's precalibration_sha256 to the fresh run's PRECALIBRATION digest; keep the calibration-code hash. Run calibrate.py --run independent-run --out independent-calibration. Then update only EVALUATION-FREEZE.json's pretest_sha256 to the fresh run's PRETEST digest, retaining the evaluator hash, and run evaluate.py --run independent-run --calibration independent-calibration --out independent-evaluation. Fresh timing can change gate digests. Exact numeric identity depends on runtime/linear algebra and is not universally promised.

Protocol/config/source/research references were frozen before implementation and fitting. PREFIT precedes the first update. PRECALIBRATION freezes the selected point model and preprocessing before its first calibration scoring; calibration code is separately frozen. PRETEST fixes the complete residual/threshold artifacts before this model's test evaluation, with a separate evaluator freeze. These local artifacts do not establish historical third-party timestamps or blind testing; the original source and earlier benchmark were already known.

Optional figure regeneration uses requirements-plot.txt and plot_study.py with Matplotlib Agg. All epochs, all levels, all months and full role denominators remain visible. Original accepted packages are unchanged.

中文

我把模型训练、选轮次、定残差区间和最终测试分成四个时间段。新模型实际训练40轮,第一季度选择第39轮,再用第二季度2,134个误差固定三个阈值,最后检查下半年4,258个完整历史窗口。

90%名义区间整体覆盖91.45%,但9月只有87.64%、10月86.11%;95%名义区间整体覆盖94.81%。所有失败、区间宽度和排除记录保留,没有看测试结果后改阈值。时序观测重叠、季节会变,不能假定满足可交换性,也不能把总体覆盖当每个月或单次预测的保证。

页面能运行固定点模型和区间,也能单独检查用户分数的精确排名。小样本要求的排名超出样本数时,会明确保留无界区间,JSON用null配unbounded标记;不悄悄换成有限宽度。全部计算在本地,下载包含完整输入、模型与校准摘要和原作者许可。

Attribution and license

Dataset: Fanaee-T, H. (2013). Bike Sharing [Dataset]. UCI Machine Learning Repository. DOI 10.24432/C5W894, CC BY 4.0 on the original UCI page. Original source README retains the requested Fanaee-T/Gama publication citation, Event labeling combining ensemble detectors and background knowledge, DOI 10.1007/s13748-013-0040-3.

Method reference: Anastasios N. Angelopoulos and Stephen Bates, A Gentle Introduction to Conformal Prediction and Distribution-Free Uncertainty Quantification, arXiv:2107.07511v6, sections noted above. This work is not a replication of their empirical studies or an author endorsement.

Code and my documentation: Renchao Wu, Apache-2.0. Original data and derived trained weights/calibration scores/experimental results/figures retain CC BY 4.0 attribution. User inputs keep their own terms. See LICENSES.md, NOTICE, RESEARCH.json and data/SOURCE.json. Identity: work:forecast-interval-lab:v1.