Context is Key: A Benchmark for Forecasting with Essential Textual Information

Preprint (2024)
Code
Andrew Robert Williams*, Arjun Ashok*, Étienne Marcotte, Valentina Zantedeschi, Jithendaraa Subramanian, Roland Riachi, James Requeima, Alexandre Lacoste, Irina Rish, Nicolas Chapados, Alexandre Drouin
Note: This is a pre-release and the benchmark is expected to evolve in the coming months.

Visualizations

Tasks by Context Type

Tasks by Model Capability

Tasks by Context Type

There is a total of 71 tasks in the benchmark.

History

Future

Intemporal

Covariates

Causal

Tasks by Model Capability

There is a total of 71 tasks in the benchmark.

Instruction following

Retrieval: context

Retrieval: memory

Reasoning: analogy

Reasoning: deduction

Reasoning: math

Reasoning: causal