CaptionQA

About CaptionQA

CaptionQA evaluates whether image captions can stand in for images in downstream tasks. We measure utility: how well captions preserve visual information needed for real-world applications.

Benchmark Stats:
• 33,027 questions over 657 images
• 4 domains (Natural, Document, E-commerce, Embodied AI)
• 25 top-level + 69 subcategories
• 50.3 questions per image

Scoring Methodology

The leaderboard ranks models using a Score metric. For each question: 1.0 if correct, 0.0 if incorrect, 1/K + 0.05 if "Cannot answer" (where K = number of choices). Final score = average across all questions (%).

This favors precision over hallucination: saying less but avoiding wrong information scores higher than confidently misleading captions.

Resources

📝 Blog Posts:

Submission

1. Download the dataset from HuggingFace

2. Generate captions with your model

3. (Optional) Evaluate on validation set

4. Email captions to captionqa.team@gmail.com

5. We evaluate and email you the Score results (see Scoring Methodology above)

6. PR a table row to leaderboard repo

Privacy First: Email us only your captions (simple image_id: caption JSON), no personal info needed.
Easy PR: Copy our HTML template, fill in your scores, and add one table row to index.html.
See the detailed submission guide for the complete process.

Citation

@misc{yang2025captionqacaptionusefulimage,
  title={CaptionQA: Is Your Caption as Useful as the Image Itself?},
  author={Shijia Yang and Yunong Liu and Bohan Zhai and Ximeng Sun and
          Zicheng Liu and Emad Barsoum and Manling Li and Chenfeng Xu},
  year={2025},
  eprint={2511.21025},
  archivePrefix={arXiv},
  primaryClass={cs.CV},
  url={https://arxiv.org/abs/2511.21025}
}