Research
A benchmark for context-aided forecasting via deep research. Agents must identify forecasting-relevant evidence, distinguish essential signals from distractors, and produce grounded forecasts.
Dr-CiK: A Benchmark testing Deep Research for Context-Aided Forecasting
1McGill University · 2ServiceNow Research · 3Université de Montréal · 4Mila – Quebec AI Institute · 5Université Laval · 6University of British Columbia
✉ Corresponding author · Valentina Zantedeschi
Dr-CiK turns a forecasting instance into an evidence-grounded task through explicit entity normalization, causal evidence decomposition, document-level synthesis, typed distractors, and quality-controlled validation.
Read the figure from left to right: normalize the task, extract forecast-relevant evidence, expand the reasoning chain, then generate and validate supporting and distractor documents.
The diagram above is the paper construction view. The cards below make each block easier to inspect: what enters the stage, what it produces, and which control keeps the generated environment faithful to the forecasting mechanism.
This is the electricity consumption recorded in kilowatt (kW) in city A. Suppose that there is a heat wave in city A from 2012-10-09 18:00:00 for 3 hours, leading to excessive use of air conditioning, and 5 times the usual electricity being consumed.
This is the electricity consumption recorded in kW in Veridian city. Suppose that there is a heat wave in Veridian city from 2012-10-09 18:00:00 for 3 hours, leading to excessive use of air conditioning, and 5 times the usual electricity being consumed.
Entity extraction, minimal profile generation, and profile assignment remove surface shortcuts while preserving the series–context relationship.
Identify one primary entity, align it with the time-series variable, and extract explicit containment relations.
Compute the minimal profile count and generate realistic, reusable city or sensor profiles.
Assign the primary profile and rewrite entity references only, so causal content and target semantics remain intact.
Controllable hops make the reasoning burden explicit: no shortcut, branch, or cycle.
Supporting and distractor documents are generated separately, typed, and judged against the original forecasting mechanism.
Assign evidence and connecting links, then generate metadata for coverage, temporal consistency, and entity alignment.
Draft plausible alternatives that target specific failure modes rather than generic noise.
Render supporting and distractor metadata into standalone documents with their labels retained for evaluation.
Heat-wave notice for Veridian city, 2012-10-09 18:00–21:00. Expected effect: cooling demand raises electricity consumption to 5× baseline.
causal alignment · target horizon · entity consistentA similar heat-wave report describes another city outside the target forecast window: plausible on the surface, but not evidence for this series.
forecast-irrelevant by design · distractor label retainedLLM judges iterate until documents satisfy the required criteria, with human-in-the-loop checks for entity alignment, evidence coverage, realism, and type consistency.
Evidence is tied to the target series and forecast horizon—not just topical similarity.
Agents cannot succeed by taking one shortcut document or by retrieving the entire corpus.
Gold evidence, typed distractors, and reasoning factors separate research failures from forecasting failures.
Dr-CiK brings 279 forecasting tasks into one inspectable evidence environment. The profile below summarizes the complete evaluation and its 80-task expert-curated split across domains, cadences, evidence depth, and time-series scale.
The broad-coverage partition adapts context-aided forecasting instances for scale and diversity across time-series patterns, domains, and context mechanisms.
Real, unedited time series paired with domain-expert context, structured causal chains, labeled ground-truth evidence, and fine-grained difficulty labels for factor-conditioned analysis.
Fourteen domain families keep the evaluation grounded in real forecasting settings.
Supporting material is mixed with typed distractors to test evidence selection—not topical retrieval alone.
Topically related, but causally irrelevant to the target outcome.
A plausible alternative mechanism that does not affect the target.
An analogous entity whose behavior should not transfer to the target.
A relevant mechanism placed in the wrong time window.
Misleading claims about values, trends, or periodicity.
History and horizon lengths vary across several orders of magnitude.
From seconds to annual observations, with a strong hourly core.
Each task has labeled ground-truth evidence; these statistics describe what an agent must recover to ground a forecast.
Aggregate profile reconstructed from the 279-task benchmark statistics used in the paper. Charts are data-native so the page remains crisp and inspectable.
Open aggregate profile JSONHistory and forecast horizon only.
| Rank | Forecaster | sCRPS ↓ | sMAE ↓ | sRMSE ↓ |
|---|---|---|---|---|
| #1 | Chronos
Statistical baseline
25 samples / task |
0.334 ± 0.440 |
0.427 ± 0.634 |
0.626 ± 0.794 |
| #2 | Gemini
LLM-direct
Mixed recorded sample settings |
0.341 ± 0.514 |
0.429 ± 0.664 |
0.629 ± 0.802 |
| #3 | SES
Statistical baseline
25 samples / task |
0.393 ± 0.396 |
0.516 ± 0.620 |
0.688 ± 0.715 |
| #4 | ETS
Statistical baseline
25 samples / task |
0.433 ± 0.433 |
0.525 ± 0.541 |
0.697 ± 0.660 |
| #5 | Aurora
TS foundation model
25 samples / task |
0.484 ± 0.846 |
0.556 ± 0.922 |
0.734 ± 1.016 |
| #6 | ARIMA
Statistical baseline
25 samples / task |
0.489 ± 0.816 |
0.697 ± 1.104 |
0.843 ± 1.129 |
| #7 | Naive
Statistical baseline
25 samples / task |
0.502 ± 0.647 |
0.780 ± 1.046 |
0.920 ± 1.075 |
History paired with the original task context.
| Rank | Forecaster | sCRPS ↓ | sMAE ↓ | sRMSE ↓ |
|---|---|---|---|---|
| #1 | MoiraiAgent
LLM forecaster agent
Mixed recorded sample settings |
0.228 ± 0.456 |
0.272 ± 0.496 |
0.382 ± 0.666 |
| #2 | Gemini
LLM-direct
Mixed recorded sample settings |
0.253 ± 0.488 |
0.310 ± 0.580 |
0.443 ± 0.739 |
| #3 | Aurora
TS foundation model
25 samples / task |
0.487 ± 0.856 |
0.558 ± 0.928 |
0.733 ± 1.017 |
History paired with agent-produced research context.
| Rank | Forecaster | sCRPS ↓ | sMAE ↓ | sRMSE ↓ |
|---|---|---|---|---|
| #1 | Geminimedium reasoning
LLM-direct
25 samples / task |
0.324 ± 0.501 |
0.409 ± 0.629 |
0.567 ± 0.811 |
| #2 | Gemini
LLM-direct
25 samples / task |
0.336 ± 0.500 |
0.456 ± 0.727 |
0.626 ± 0.895 |
| #3 | Aurora
TS foundation model
25 samples / task |
0.484 ± 0.844 |
0.558 ± 0.924 |
0.735 ± 1.015 |
Every displayed row is aggregated over the same finite-metric task mask, so model comparisons within a context are directly aligned.
History and forecast horizon only.
| Rank | Forecaster | sCRPS ↓ | sMAE ↓ | sRMSE ↓ |
|---|---|---|---|---|
| #1 | TiRex-2
TS foundation model
9 quantile levels / task |
0.296 ± 0.428 |
0.357 ± 0.492 |
0.551 ± 0.677 |
| #2 | Moirai-MoE-Base
TS foundation model
25 samples / task |
0.304 ± 0.360 |
0.382 ± 0.455 |
0.598 ± 0.691 |
| #3 | Toto-2.0-313M
TS foundation model
9 quantile levels / task |
0.308 ± 0.532 |
0.322 ± 0.465 |
0.508 ± 0.645 |
| #4 | Moirai-2.0-R-small
TS foundation model
9 quantile levels / task |
0.312 ± 0.416 |
0.386 ± 0.514 |
0.584 ± 0.724 |
| #5 | Chronos
Statistical baseline
25 samples / task |
0.327 ± 0.463 |
0.428 ± 0.681 |
0.632 ± 0.839 |
| #6 | Gemini
LLM-direct
25 samples / task |
0.327 ± 0.545 |
0.377 ± 0.564 |
0.582 ± 0.734 |
| #7 | MoiraiAgent
LLM forecaster agent
Mixed recorded sample settings |
0.347 ± 0.516 |
0.366 ± 0.540 |
0.560 ± 0.722 |
| #8 | TimesFM-3
TS foundation model
9 quantile levels / task |
0.357 ± 0.613 |
0.382 ± 0.554 |
0.589 ± 0.779 |
| #9 | SES
Statistical baseline
25 samples / task |
0.388 ± 0.387 |
0.520 ± 0.629 |
0.700 ± 0.723 |
| #10 | SARIMA
Statistical baseline
25 samples / task |
0.422 ± 0.709 |
0.597 ± 0.985 |
0.754 ± 1.026 |
| #11 | ETS
Statistical baseline
25 samples / task |
0.431 ± 0.431 |
0.524 ± 0.532 |
0.704 ± 0.656 |
| #12 | Aurora
TS foundation model
25 samples / task |
0.503 ± 0.921 |
0.577 ± 1.000 |
0.763 ± 1.092 |
| #13 | ARIMA
Statistical baseline
25 samples / task |
0.508 ± 0.882 |
0.718 ± 1.162 |
0.867 ± 1.174 |
| #14 | Naive
Statistical baseline
25 samples / task |
0.515 ± 0.679 |
0.793 ± 1.045 |
0.941 ± 1.075 |
History paired with the original task context.
| Rank | Forecaster | sCRPS ↓ | sMAE ↓ | sRMSE ↓ |
|---|---|---|---|---|
| #1 | MoiraiAgent
LLM forecaster agent
Mixed recorded sample settings |
0.212 ± 0.472 |
0.248 ± 0.498 |
0.350 ± 0.663 |
| #2 | MoiraiAgentmedium reasoning
LLM forecaster agent
Mixed recorded sample settings |
0.216 ± 0.467 |
0.252 ± 0.497 |
0.355 ± 0.664 |
| #3 | Gemini
LLM-direct
25 samples / task |
0.240 ± 0.506 |
0.274 ± 0.519 |
0.393 ± 0.680 |
| #4 | Aurora
TS foundation model
25 samples / task |
0.507 ± 0.932 |
0.580 ± 1.007 |
0.762 ± 1.093 |
History paired with agent-produced research context.
| Rank | Forecaster | sCRPS ↓ | sMAE ↓ | sRMSE ↓ |
|---|---|---|---|---|
| #1 | Qwen3.5-9b
Deep-research forecaster
25 samples / task |
0.296 ± 0.412 |
0.432 ± 0.604 |
0.608 ± 0.762 |
| #2 | Qwen3.5-4b
Deep-research forecaster
25 samples / task |
0.311 ± 0.444 |
0.453 ± 0.638 |
0.640 ± 0.834 |
| #3 | MoiraiAgent
LLM forecaster agent
Mixed recorded sample settings |
0.315 ± 0.548 |
0.384 ± 0.652 |
0.528 ± 0.798 |
| #4 | Geminimedium reasoning
LLM-direct
25 samples / task |
0.321 ± 0.532 |
0.377 ± 0.554 |
0.529 ± 0.738 |
| #5 | Qwen3.5-27b
Deep-research forecaster
25 samples / task |
0.324 ± 0.589 |
0.415 ± 0.616 |
0.572 ± 0.778 |
| #6 | Gemini
LLM-direct
25 samples / task |
0.331 ± 0.528 |
0.408 ± 0.629 |
0.567 ± 0.796 |
| #7 | Mistral
Deep-research forecaster
25 samples / task |
0.380 ± 0.622 |
0.447 ± 0.640 |
0.616 ± 0.831 |
| #8 | Phi-4-mini
LLM-direct
Mixed recorded sample settings |
0.448 ± 0.586 |
0.816 ± 1.239 |
1.031 ± 1.288 |
| #9 | Aurora
TS foundation model
25 samples / task |
0.503 ± 0.918 |
0.579 ± 1.002 |
0.763 ± 1.090 |
| #10 | Llama-3.2-3B
LLM-direct
Mixed recorded sample settings |
0.637 ± 0.899 |
1.370 ± 1.651 |
1.556 ± 1.676 |
Retrieval and synthesis metrics across the benchmark.
| DR agent | Evidence Recall ↑ | Supporting Doc. Recall ↑ | Distractor Avoidance ↑ |
|---|---|---|---|
| Bench2Future | 3.3% | 7.0% | 22.7% |
| DrBench | 3.4% | 9.7% | 28.8% |
| Retrieval | 3.9% | 9.9% | 21.0% |
| OpenDR | 4.3% | 9.7% | 23.9% |
| Codex | 27.9% | 50.6% | 42.3% |
Higher is better for all three metrics. Evidence Recall scores matched evidence with source citation credit; Supporting Document Recall measures source recovery; Distractor Avoidance is the share of cited documents that are not distractors.
The release contains 199 public development tasks and 80 expert-curated hidden-test tasks. The results above are experimental reference results; official submissions are scored on the hidden test set by the maintainers.
Load all 279 tasks, 10,342 documents, and task–document links. Development labels are public; hidden-test answers are withheld.
Official evaluationFollow the submission guide to have your outputs scored on the 80-task hidden test set.
Interactive showcaseBrowse the original task showcase alongside the expanded development-task library below.
Each listed task reports its Figure 2 domain and four independent diagnostic axes. The public task library includes only tasks with a complete released label set.
sales volume · anomaly explicit/template0
speed · anomaly explicit/template0
CPU usage · anomaly explicit/template0
elecitricy consumption · anomaly explicit/template0
sales volume · anomaly explicit/template1
CPU usage · anomaly explicit/template1
elecitricy consumption · anomaly explicit/template1
sales volume · anomaly explicit/template2
speed · anomaly explicit/template2
CPU usage · anomaly explicit/template2
sales volume · anomaly explicit/template3
speed · anomaly explicit/template3
CPU usage · anomaly explicit/template3
elecitricy consumption · anomaly explicit/template3
speed · anomaly explicit/template4
CPU usage · anomaly explicit/template4
elecitricy consumption · anomaly explicit/template4
sales volume · anomaly implicit/template0
speed · anomaly implicit/template0
traffic · anomaly implicit/template0
CPU usage · anomaly implicit/template0
elecitricy consumption · anomaly implicit/template0
sales volume · anomaly implicit/template1
speed · anomaly implicit/template1
traffic · anomaly implicit/template1
CPU usage · anomaly implicit/template1
elecitricy consumption · anomaly implicit/template1
sales volume · anomaly implicit/template2
speed · anomaly implicit/template2
traffic · anomaly implicit/template2
CPU usage · anomaly implicit/template2
elecitricy consumption · anomaly implicit/template2
sales volume · anomaly implicit/template3
speed · anomaly implicit/template3
traffic · anomaly implicit/template3
CPU usage · anomaly implicit/template3
elecitricy consumption · anomaly implicit/template3
sales volume · anomaly implicit/template4
speed · anomaly implicit/template4
traffic · anomaly implicit/template4
CPU usage · anomaly implicit/template4
elecitricy consumption · anomaly implicit/template4
number of cash withdrawals · atmunder periodic maintenance task with conclusion
number of cash withdrawals · atmunder periodic maintenance task with conclusion
number of cash withdrawals · atmunder periodic maintenance task with conclusion
number of cash withdrawals · atmunder periodic maintenance task with conclusion
number of cash withdrawals · atmunder periodic maintenance task with conclusion less explicit
number of cash withdrawals · atmunder periodic maintenance task with conclusion less explicit
number of cash withdrawals · atmunder periodic maintenance task with conclusion less explicit
number of cash withdrawals · atmunder periodic maintenance task with conclusion less explicit
number of cash withdrawals · atmunder periodic maintenance task with conclusion less explicit
number of cash withdrawals · cash depletedin atmscenario task
number of cash withdrawals · cash depletedin atmscenario task
number of cash withdrawals · cash depletedin atmscenario task
number of cash withdrawals · cash depletedin atmscenario task
number of cash withdrawals · cash depletedin atmscenario task
traffic · decrease in traffic in prediction task
traffic · decrease in traffic in prediction task
traffic · decrease in traffic in prediction task
traffic · decrease in traffic in prediction task
traffic · decrease in traffic in prediction task
electricity consumption · electricity increase in prediction task
electricity consumption · electricity increase in prediction task
electricity consumption · electricity increase in prediction task
electricity consumption · electricity increase in prediction task
electricity consumption · electricity increase in prediction task
electricity consumption · electricity increase in prediction with distractor text
electricity consumption · electricity increase in prediction with distractor text
electricity consumption · electricity increase in prediction with distractor text
electricity consumption · electricity increase in prediction with distractor text
electricity consumption · electricity increase in prediction with distractor text
electricity consumption · electricity increase in prediction with distractor with dates
electricity consumption · electricity increase in prediction with distractor with dates
electricity consumption · electricity increase in prediction with distractor with dates
electricity consumption · electricity increase in prediction with distractor with dates
electricity consumption · electricity increase in prediction with distractor with dates
electricity consumption · electricity increase in prediction with split context
electricity consumption · electricity increase in prediction with split context
electricity consumption · electricity increase in prediction with split context
electricity consumption · electricity increase in prediction with split context
electricity consumption · electricity increase in prediction with split context
Direct Normal Irradiance · explicit direct normal irradiance from cloud status
Direct Normal Irradiance · explicit direct normal irradiance from cloud status
road occupancy rates · explicit traffic forecast taskwith holidays in prediction window
road occupancy rates · explicit traffic forecast taskwith holidays in prediction window
road occupancy rates · explicit traffic forecast taskwith holidays in prediction window
road occupancy rates · explicit traffic forecast taskwith holidays in prediction window
road occupancy rates · explicit traffic forecast taskwith holidays in prediction window
road occupancy rates · explicit with dates and days traffic forecast taskwith holidays in prediction window
road occupancy rates · explicit with dates and days traffic forecast taskwith holidays in prediction window
road occupancy rates · explicit with dates and days traffic forecast taskwith holidays in prediction window
road occupancy rates · explicit with dates and days traffic forecast taskwith holidays in prediction window
road occupancy rates · explicit with dates and days traffic forecast taskwith holidays in prediction window
road occupancy rates · explicit with days traffic forecast taskwith holidays in prediction window
road occupancy rates · explicit with days traffic forecast taskwith holidays in prediction window
road occupancy rates · explicit with days traffic forecast taskwith holidays in prediction window
road occupancy rates · explicit with days traffic forecast taskwith holidays in prediction window
road occupancy rates · explicit with days traffic forecast taskwith holidays in prediction window
road occupancy rates · implicit traffic forecast taskwith holidays in prediction window
road occupancy rates · implicit traffic forecast taskwith holidays in prediction window
road occupancy rates · implicit traffic forecast taskwith holidays in prediction window
road occupancy rates · implicit traffic forecast taskwith holidays in prediction window
road occupancy rates · implicit traffic forecast taskwith holidays in prediction window
number of cash withdrawals · increased withdrawal scenario
number of cash withdrawals · increased withdrawal scenario
number of cash withdrawals · increased withdrawal scenario
number of cash withdrawals · increased withdrawal scenario
number of cash withdrawals · increased withdrawal scenario
occupancy rate · sensor maintenance in prediction task
occupancy rate · sensor maintenance in prediction task
occupancy rate · sensor maintenance in prediction task
occupancy rate · sensor maintenance in prediction task
occupancy rate · sensor maintenance in prediction task
occupancy rate · sensor periodic maintenance task
occupancy rate · sensor periodic maintenance task
occupancy rate · sensor periodic maintenance task
occupancy rate · sensor periodic maintenance task
occupancy rate · sensor periodic maintenance task
occupancy rate · sensor spike task
occupancy rate · sensor spike task
occupancy rate · sensor spike task
occupancy rate · sensor spike task
occupancy rate · sensor spike task
occupancy rate · sensor trend accumulation task
occupancy rate · sensor trend accumulation task
occupancy rate · sensor trend accumulation task
occupancy rate · sensor trend accumulation task
occupancy rate · sensor trend accumulation task
electricity consumption · short news electricity increase in prediction task
electricity consumption · short news electricity increase in prediction task
electricity consumption · short news electricity increase in prediction task
electricity consumption · short news electricity increase in prediction task
electricity consumption · short news electricity increase in prediction task
temperature reading · context explicit/template0
sales · context explicit/template0
CPU usage · context explicit/template0
electricity usage · context explicit/template0
temperature reading · context explicit/template1
sales · context explicit/template1
CPU usage · context explicit/template1
electricity usage · context explicit/template1
temperature reading · context explicit/template2
sales · context explicit/template2
CPU usage · context explicit/template2
temperature reading · context explicit/template3
CPU usage · context explicit/template3
electricity usage · context explicit/template3
temperature reading · context explicit/template4
sales · context explicit/template4
CPU usage · context explicit/template4
electricity usage · context explicit/template4
traffic · context implicit/template0
temperature reading · context implicit/template0
sales · context implicit/template0
CPU usage · context implicit/template0
electricity usage · context implicit/template0
traffic · context implicit/template1
temperature reading · context implicit/template1
sales · context implicit/template1
CPU usage · context implicit/template1
electricity usage · context implicit/template1
traffic · context implicit/template2
sales · context implicit/template2
CPU usage · context implicit/template2
electricity usage · context implicit/template2
traffic · context implicit/template3
temperature reading · context implicit/template3
sales · context implicit/template3
CPU usage · context implicit/template3
electricity usage · context implicit/template3
traffic · context implicit/template4
temperature reading · context implicit/template4
sales · context implicit/template4
CPU usage · context implicit/template4
electricity usage · context implicit/template4
If Dr-CiK supports your research, please cite the paper.
@article{tang2026dr,
title={Dr-CiK: A Benchmark testing Deep Research for Context-Aided Forecasting},
author={Tang, Yihong and Williams, Andrew Robert and Ashok, Arjun and Zheng, Vincent Zhihao and Gupta, Shubham and Sun, Lijun and Drouin, Alexandre and Laradji, Issam H. and Marcotte, Étienne and Zantedeschi, Valentina},
journal={arXiv preprint arXiv:2605.27904},
year={2026},
url={https://arxiv.org/abs/2605.27904}
}