💫 StarFlow: Generating Structured Workflow Outputs From Sketch Images

Workflows are a fundamental component of automation in enterprise platforms. Building them can be complex and often requires manual configuration through low-code or visual tools. We explore using vision–language models (VLMs) to automatically generate structured workflows from visual inputs—hand-drawn sketches and computer-generated diagrams. We introduce StarFlow, a framework for this task, curate a diverse dataset of workflow diagrams (synthetic, manually annotated, and real-world), and fine-tune multiple VLMs. Our results show that fine-tuning significantly enhances structured workflow generation, outperforming larger general-purpose models on this task.

We build a diverse dataset of workflow diagrams, spanning synthetic, human-annotated, and real-world samples, to enhance training and evaluation. Below, we show the distribution of the dataset across splits and how the dataset was constructed.

We benchmark a range of off-the-shelf and fine-tuned vision–language models on the BigDocs-Sketch2Flow dataset, evaluating their ability to generate structured workflows from sketches and diagrams. Below we summarize the results across different model sizes and categories.

Running inference with StarFlow is simple. Load one of the released fine-tuned vision–language models, pass in your workflow sketch, and get structured JSON back:

from transformers import AutoProcessor, Qwen2_5_VLForConditionalGeneration
from PIL import Image

# Load processor and model
processor = AutoProcessor.from_pretrained("ServiceNow/Qwen2.5-VL-7B-Instruct-StarFlow")
model = Qwen2_5_VLForConditionalGeneration.from_pretrained(
    "ServiceNow/Qwen2.5-VL-7B-Instruct-StarFlow"
)

# Load your sketch or diagram
image = Image.open("workflow_sketch.png")

# Prepare input with an instruction
inputs = processor(images=image, text="Generate workflow JSON", return_tensors="pt")

# Generate structured workflow
outputs = model.generate(**inputs, max_length=4096)
workflow_json = processor.decode(outputs[0], skip_special_tokens=True)

print(workflow_json)

This will output a workflow_json string containing nodes, edges, and triggers extracted from your sketch—ready to run inside workflow automation platforms. Swap in other StarFlow fine-tuned models (e.g., LLaMA or Pixtral variants) by just changing the model name.