SABRE: Scalable and Automated Benchmarking of VLMs under Stress

Vision-language models (VLMs) are advancing rapidly, but building benchmarks that meaningfully stress-test their weaknesses is costly and labor-intensive. SABRE offers an automated pipeline converting task specifications into structured images and question-answer pairs, using automated filtering plus human review to ensure benchmark quality and difficulty. Its SABRE-Prior instantiation specifically tests whether VLMs rely on genuine visual evidence versus learned world priors. This is useful for AI evaluation teams and VLM developers needing scalable, refreshable stress tests, revealing that current VLMs struggle significantly (17.8%-31.3% accuracy) with counterfactual scenes, textures, and misleading language cues. Paper: https://arxiv.org/abs/2608.07435