Benchmark RadarCapProbe
CapProbe is a full-scene dense QA benchmark for evaluating detailed image captions from Vision-Language Models. It decomposes images into semantic regions and generates multiple-choice questions across 10 semantic categories, with a language judge answering from captions. The benchmark comprises 346 images, 1,868 regions, and 25,650 questions.
- Released
- 2026-08-11
- Readiness
- Paper only
- Primary field
- General AI
Why it matters
Existing metrics for detailed caption evaluation struggle to verify dense factual claims. CapProbe addresses this by region-aligned factual checking with dense QA, offering a cost-effective protocol that reduces open-ended scoring bias and reveals coverage gaps and trade-offs across models.
Motivation
Evaluating detailed image captions from Vision-Language Models (VLMs) requires going beyond surface-level semantic similarity.
Primary resources
Benchmark Radar records only publicly supported details and links back to primary sources for verification.