fm-vision-evals
Evaluate multimodal models on standard CV tasks — an evaluation benchmark
Evaluates multimodal models on standard computer-vision tasks — an evaluation benchmark.
Image generation and computer vision — detection, editing, 3D.
Key features
- Prompt-chaining framework benchmarking multimodal foundation models on standard vision
- Evaluates semantic segmentation, object detection, image classification, depth and
- Translates vision tasks into text-promptable, API-compatible tasks without native
- Provides the taskit package for evaluating multimodal models on various vision tasks
- Official implementation of an ICLR 2026 paper by EPFL VILAB
Good for
- Researchers benchmarking multimodal models (GPT-4o, Gemini, Claude
- Comparing proprietary API-only models against specialist vision models
Stack & tags
Pythonbenchmarkvlmmultimodalevaluationcomputer-visionprompt-chainingiclr2026
Quick start
conda create -n fm-vision python=3.10 -y
conda activate fm-vision
pip install -r requirements.txtFrom the project README on GitHub.
Similar open-source projects
FLUX.2 klein
★ 25,996Black Forest Labs' sub-second image generation — generate and edit in <0.5s
DiffSynth-Studio
★ 13,200The framework behind Qwen-Image-i2L — the first "Image→LoRA" visual generation system