Xiangyu Zhao*, Peiyuan Zhang*, Kexian Tang*, Xiaorong Zhu*,
Hao Li, Wenhao Chai, Zicheng Zhang, Renqiu Xia,
Guangtao Zhai, Junchi Yan, Hua Yang, Xue Yang, Haodong Duan
If you find our work helpful, please consider giving us a ⭐ or citation 😊
In this work, we introduce RISEBench, the first benchmark for evaluating Reasoning-Informed viSual Editing (RISE). RISEBench focuses on four key reasoning types: Temporal, Causal, Spatial, and Logical Reasoning.
To comprehensively assess model performance across diverse task types, we define three key evaluation dimensions: Instruction Reasoning, Appearance Consistency, and Visual Plausibility.
Besides, we design a robust LMM-as-a-Judge evaluation pipeline and leverage state-of-the-art LMMs(GPT-4o) to generate automated assessments. Our approach offers a scalable and reproducible alternative to human evaluation, while maintaining a high degree of alignment with human judgment.
As an initial effort, RISEBench aims to provide foundational insights into reasoning-aware visual editing and to catalyze future research. Though still in its early stages, we are committed to continuously expanding and refining the benchmark to support more comprehensive, reliable, and scalable evaluations of next-generation multimodal systems.
To evaluate the performance of representative visual editing approaches, we selected a diverse set of models spanning multiple model architectures and generation paradigms. Specifically, Flux1.0-Canny serves as a representative diffusion-based editing model, while EMU2 exemplifies the auto-regressive generation paradigm. We also include 8 proprietary models, including GPT-4o-Image, Gemini 2.0-Flash-Experimental, and Gemini 2.0-Flash-Preview. The outputs of proprietary models are given by the official API.
| Model | Temporal (%) | Causal (%) | Spatial (%) | Logical (%) | Overall (%) |
|---|---|---|---|---|---|
| 🏅 GPT-Image-1.5 | 57.6 | 62.2 | 62.0 | 21.2 | 51.4 |
| 🥈 GPT-Image-2 | 45.9 | 66.7 | 50.0 | 34.1 | 49.4 |
| 🥉 Gemini-3-pro-image-preview (Nano Banana-pro) | 43.5 | 63.3 | 48.0 | 37.6 | 48.3 |
| Gemini-2.5-Flash-Image (Nano Banana) | 29.4 | 48.9 | 37.0 | 18.8 | 33.9 |
| GPT-Image-1 | 36.5 | 34.4 | 37.0 | 10.6 | 30.0 |
| GPT-Image-1-mini | 25.9 | 31.1 | 33.0 | 9.4 | 25.3 |
| Qwen-Image-Edit-2511 | 21.2 | 18.9 | 31.0 | 4.7 | 19.4 |
| BAGEL (w/ CoT) | 11.8 | 27.8 | 21.0 | 1.2 | 15.8 |
| Gemini-2.0-Flash-exp | 9.4 | 16.7 | 23.0 | 4.7 | 13.9 |
| Seedream-4.0 | 17.6 | 13.3 | 11.0 | 7.1 | 12.2 |
| Gemini-2.0-Flash-pre | 11.8 | 14.4 | 11.0 | 2.4 | 10.0 |
| Qwen-Image-Edit-2509 | 4.7 | 11.1 | 17.0 | 2.4 | 9.2 |
| BAGEL | 3.5 | 4.4 | 14.0 | 0.0 | 5.8 |
| FLUX.1-Kontext-Dev | 2.3 | 5.5 | 13.0 | 1.2 | 5.8 |
| Ovis-U1 | 1.1 | 3.3 | 4.0 | 2.4 | 2.8 |
| Step1X-Edit | 0.0 | 2.2 | 2.0 | 3.5 | 1.9 |
| OmniGen | 1.2 | 1.1 | 0.0 | 1.2 | 0.8 |
| EMU2 | 1.2 | 1.1 | 0.0 | 0.0 | 0.5 |
| HiDream-Edit | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 |
| FLUX.1-Canny | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 |
| Model | 🧠 Instruction Reasoning | 🪞 Appearance Consistency | 👁️ Visual Plausibility |
|---|---|---|---|
| 🏅 GPT-Image-1.5 | 69.7 | 92.5 | 94.9 |
| 🥈 GPT-Image-2 | 73.8 | 89.3 | 94.9 |
| 🥉 Gemini-3-pro-image-preview (banana-pro) | 77.0 | 85.5 | 94.4 |
| Gemini-2.5-Flash-Image (banana) | 61.2 | 86.0 | 91.3 |
| GPT-Image-1 | 62.8 | 80.2 | 94.9 |
| GPT-Image-1-mini | 54.1 | 71.5 | 93.7 |
| Qwen-Image-Edit-2511 | 49.9 | 71.0 | 91.5 |
| BAGEL (w/ CoT) | 47.6 | 75.6 | 81.7 |
| Gemini-2.0-Flash-exp | 48.9 | 68.2 | 82.7 |
| Seedream-4.0 | 58.9 | 67.4 | 91.2 |
| Gemini-2.0-Flash-pre | 49.9 | 68.4 | 84.9 |
| Qwen-Image-Edit-2509 | 37.2 | 66.4 | 86.9 |
| BAGEL | 36.4 | 58.7 | 72.0 |
| FLUX.1-Kontext-Dev | 26.0 | 71.6 | 85.2 |
| Ovis-U1 | 33.9 | 52.7 | 72.9 |
| HiDream-Edit | 30.3 | 12.6 | 74.9 |
| Step1X-Edit | 25.1 | 41.5 | 73.5 |
| EMU2 | 22.6 | 38.2 | 78.3 |
| OmniGen | 22.0 | 32.6 | 55.3 |
| FLUX.1-Canny | 20.2 | 13.1 | 77.5 |
Data: This repository includes only a demo subset; download the full RISEBench-360 dataset from Hugging Face for full-benchmark evaluation.
The bundled data/data_total.json contains 64 samples (RISEBench-64) for trying the generation and evaluation workflow.
Install the Python dependencies used by the original scripts:
python -m pip install "numpy<2" pandas pillow requests tqdm xlsxwriter openpyxlAll data/... and outputs/... paths below are relative to RISEBench/.
The input images for the four categories are located in the data directory. Each sample in the dataset contains an instruction and an associated image. You can use these inputs to generate the corresponding output image.
Output File Structure:
Generated outputs should be saved in the following directory structure:
outputs/{MODEL_NAME}/images/{CATEGORY}/{INDEX_NAME}.{FORMAT}
{MODEL_NAME}: The name of the model you are using (e.g.,gpt-4o).{CATEGORY}: The category of the sample (e.g.,temporal_reasoning).{INDEX_NAME}: The index of the sample in the dataset.{FORMAT}: The file format of the output image (supported formats:.png,.jpg, or.jpeg).
For example:
outputs/gpt-4o-native/images/temporal_reasoning/temporal_reasoning_1.png
Once all outputs are generated and saved in the specified format, you can evaluate them using the gpt_eval.py script.
Open the gpt_eval.py file and update the following parameters with your OpenAI credentials:
api_key: Your OpenAI API key.api_base: Your OpenAI API base URL (if applicable).
Run the following commands from the repository root:
cd RISEBench
python gpt_eval.py --data data/data_total.json --output outputs/MODEL_NAMEAfter running the script, three result files will be generated in the outputs/{MODEL_NAME} directory:
{MODEL_NAME}_judge.csv: A CSV file containing the total evaluation scores.{MODEL_NAME}_judge.xlsx: An Excel file storing detailed responses from the GPT-4.1 judge model.{MODEL_NAME}.pkl: A serialized pickle file saving the raw responses from the judge model, which can be used to resume or extend evaluations later.
We exhibit some outputs of the five models in the appendix. For more details, please refer to our paper.
If you find RISEBench useful, please cite our work:
@article{zhao2025envisioning,
title={Envisioning beyond the pixels: Benchmarking reasoning-informed visual editing},
author={Zhao, Xiangyu and Zhang, Peiyuan and Tang, Kexian and Li, Hao and Zhang, Zicheng and Zhai, Guangtao and Yan, Junchi and Yang, Hua and Yang, Xue and Duan, Haodong},
journal={Advances in neural information processing systems},
year={2025}
}