C3B: 漫画跨文化基准测试 C3B: Comics Cross-Cultural Benchmark

文化感知能力已成为多模态大型语言模型（MLLMs）的核心能力。然而，现有基准测试在任务设计上缺乏难度递进性，且跨语言任务存在不足。此外，现有基准测试常采用真实世界图像，而单张真实世界图像通常仅包含一种文化，这使得这些基准测试对多模态大型语言模型而言相对简单。基于此，我们提出了 C³B（漫画跨文化基准测试，Comics Cross-Cultural Benchmark）—— 一个全新的、涵盖多文化、多任务与多语言的文化感知能力基准测试。C³B 包含 2000 余张图像及 18000 余对问答（QA）样本，围绕三个难度递进的任务构建：从基础的视觉识别，到更高层次的文化冲突理解，最终延伸至文化内容生成。我们在 11 个开源多模态大型语言模型上开展了评估，结果显示模型性能与人类表现之间存在显著差距。这一差距表明，C³B 对当前多模态大型语言模型构成了严峻挑战，有望推动未来相关研究以提升模型的文化感知能力。 Cultural awareness capabilities has emerged as a critical capability for Multimodal Large Language Models (MLLMs). However, current benchmarks lack progressed difficulty in their task design and are deficient in cross-lingual tasks. Moreover, current benchmarks often use real-world images. Each real-world image typically contains one culture, making these benchmarks relatively easy for MLLMs. Based on this, we propose C3B (Comics Cross-Cultural Benchmark), a novel multicultural, multitask and multilingual cultural awareness capabilities benchmark. C3 B comprises over 2000 images and over 18000 QA pairs, constructed on three tasks with progressed difficulties, from basic visual recognition to higher-level cultural conflict understanding, and finally to cultural content generation. We conducted evaluations on 11 open-source MLLMs, revealing a significant performance gap between MLLMs and human performance. The gap demonstrates that C3B poses substantial challenges for current MLLMs, encouraging future research to advance the cultural awareness capabilities of MLLMs.

2220张图像 2220 Images
18789个问答对 18789 QA Pairs
更新于2025年 Updated in 2025
涵盖77文化 Covering 77 Cultures
包含五个语言对 Contains 5 Language Pairs

引用信息 Citation Information

如果您在研究中使用了本数据集，请使用以下BibTex格式引用： If you use this dataset in your research, please cite it using the following BibTex format： 


@misc{song2025cultureframec3bcomicbased,
      title={Culture In a Frame: C$^3$B as a Comic-Based Benchmark for Multimodal Culturally Awareness}, 
      author={Yuchen Song and Andong Chen and Wenxin Zhu and Kehai Chen and Xuefeng Bai and Muyun Yang and Tiejun Zhao},
      year={2025},
      eprint={2510.00041},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2510.00041}, 
}