SynthDocBench — Controlled Benchmark for Long-Context Visual Document Understanding

Overview

SynthDocBench evaluates vision-language models on 1,788 questions over 200 documents containing 3,340 charts, averaging 51.1 pages per document — a fully synthetic, controlled benchmark for long-context visual document understanding.

Unlike prior benchmarks that draw on real documents and confound length, layout, and modality, SynthDocBench generates documents end-to-end with an LLM pipeline across 6 layout archetypes, varying document length, layout structure, modality composition, and question type as independent axes — enabling controlled attribution of why a model fails, not just that it does.

Key features

3 task-specific subsetschart (24 D3.js chart types), cross_modal (evidence split across non-adjacent sections), and complex (multi-hop, L1 → L5).

Long documents51.1 avg. pages per document, far beyond single-page chart/doc-VQA benchmarks.

Deterministic ground truthEvery chart is rendered as D3.js and backed by hidden structured metadata, so answers are exact.

Cross-judge validatedGPT-5 and Gemini-as-judge agree within 3.5 accuracy points (r ≥ 0.94) across 7 frontier VLMs.

Reveals hidden failuresLength degradation, positional sensitivity (middle third hardest for 5/6 models), and chart-reading collapse in long documents.

Benchmark

Subset	Description
`chart`	Chart-reading questions answerable only from a chart — probes 24 D3.js chart types (incl. dumbbell, lollipop, slope, sparkline grids) requiring exact numerical reads.
`cross_modal`	Questions requiring text + chart evidence from separate, non-adjacent sections (long-range cross-modal grounding).
`complex`	Multi-hop questions combining 2–4 evidence units across text and charts, difficulty L1 (direct lookup) → L5 (cross-section synthesis).

Source PDFs are hosted alongside the QA subsets on the Hugging Face Hub: ServiceNow-AI/SynthDocBench.

from datasets import load_dataset

ds = load_dataset("ServiceNow-AI/SynthDocBench")

How the documents are built

Most document-understanding benchmarks reuse a small pool of real PDFs — hard to control and easy to overfit. SynthDocBench instead generates documents from a LangGraph pipeline, giving full control over layout, styling, charts, and the questions asked about them — at scale.

Pick a layout archetype — magazine, dashboard, academic, editorial, infographic, or brutalist.
Research & author — pull real content from Wikipedia/Wikidata (or ingest arXiv papers) and draft the report body.
Design — generate design tokens and a "style DNA" so each report looks distinct.
Visualize — propose visualization ideas, choose chart types, structure the data, and emit D3.js code with embedded metadata.
Assemble — number figures, insert in-text references, and render HTML.
Export — convert to a clean PDF and a realistic scanned-document version.
Question — generate multi-hop, chart-reading, and cross-modal QA pairs from the report's own chart metadata.

What makes the documents hard

Visual diversity — randomized style axes and per-component renderers mean models can't rely on a single template.
Charts that carry meaning — many questions require reading values off a visualization, not just the prose.
Cross-modal reasoning — some questions combine text, tables, and figures.
Scanned realism — a scanned-document effect adds the noise and artifacts VLMs face in the wild.

Evaluation

The evaluation harness rasterizes each report PDF into page-image strips — vision only, with no access to the underlying HTML or metadata — asks each candidate model the report's questions, and scores the answers with a judge model. This isolates genuine visual document understanding from any shortcut through structured text.

python eval/evaluate_models.py \
    --input-dir data/reports \
    --candidates gpt-4o \
    --judge gpt-5 \
    --output results/all_results.json

Get started with the code and dataset:

Code: github.com/ServiceNow/SynthDocBench
Dataset: huggingface.co/datasets/ServiceNow-AI/SynthDocBench

Citation

@misc{verma2026synthdocbenchcontrolledbenchmarklongcontext,
      title={SynthDocBench: Controlled Benchmark for Long-Context Visual Document Understanding},
      author={Abhigya Verma and Khyati Mahajan and Amit Kumar Saha and Shruthan Radhakrishna and Sagar Davasam and Vikas Yadav and Sai Rajeswar},
      year={2026},
      eprint={2607.10400},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2607.10400},
}