[Benchmark] Support SCAM

[JonasLoos]

SCAM: A Real-World Typographic Robustness Evaluation for Multimodal Foundation Models

[🌐 Homepage] [🤗 Huggingface Dataset] [📖 ArXiv Paper]

👀 Introduction

Typographic attacks exploit the interplay between text and visual content in multimodal foundation models, causing misclassifications when misleading text is embedded within images. However, existing datasets are limited in size and diversity, making it difficult to study such vulnerabilities. We introduce SCAM, the largest and most diverse dataset of real-world typographic attack images to date, containing images across hundreds of object categories and attack words.

📈 Evaluation

torchrun --nproc-per-node=1  run.py --data SCAM --model your_model --verbose