MiniGPT-5: Interleaved Vision-and-Language Generation via Generative Vokens
+EditRoom: LLM-parameterized Graph Diffusion for Composable 3D Room Layout Editing
MiniGPT-5: Interleaved Vision-and-Langu
+ 
Abstract
Interleaved Vision-and-Language Generation via LLMs
+ Unified Scene Layout Editing
- - We leverage the pretrained multimodal large language model (MiniGPT-4) and text-to-image generation model (Stable Diffusion 2.1) to create a unified multimodal generation pipeline.
- - We added vokens into LLM's vocabulary and align the voken features with stable diffusion conditional features.
- - Text Generation Loss help model learn voken positions while Conditional Latent Denoising Loss guide the model to predicate appropriate features
+ - We leverage the pretrained multimodal large language model (GPT-4o) as command parameterizer and graph diffusion-based method as scene editor to create a unified scene layout editing pipeline.
+ - Larget language model can convert natural language commands into breakdown commands by given source scene information.
+ - SceneEditor take the breakdown commands and source scene as input, and it can first generate the abstract target scene graph, then it will estimate the accurate target scene layout.
- 
+ 
-
-
-
- We leverage the pretrained multimodal large language model (MiniGPT-4) and text-to-image generation model (Stable Diffusion 2.1) to create a unified multimodal generation pipeline. -
- We added vokens into LLM's vocabulary and align the voken features with stable diffusion conditional features. -
- Text Generation Loss help model learn voken positions while Conditional Latent Denoising Loss guide the model to predicate appropriate features +
- We leverage the pretrained multimodal large language model (GPT-4o) as command parameterizer and graph diffusion-based method as scene editor to create a unified scene layout editing pipeline. +
- Larget language model can convert natural language commands into breakdown commands by given source scene information. +
- SceneEditor take the breakdown commands and source scene as input, and it can first generate the abstract target scene graph, then it will estimate the accurate target scene layout.
Figure 2. MiniGPT-5 pipeline.
+Figure 2. Scene Editor aims to provide accurate, coherent editing results according to the given source scene and language commands. + It consists of two graph transformer-based conditional diffusion models. One diffusion model generates semantic target scene graphs. + Another diffusion model can estimate accurate poses and size information for each object inside the generated target scene graphs. + All diffusion processes are conditioned on the source scene and breakdown command.
@@ -183,12 +202,16 @@
- Qualitative examples from MiniGPT-5 and baselines on the CC3M, VIST, and MMDialog datasets. From the comparisons, we can find the MiniGPT-5 and SD 2 have similar results on single-image generation. When we evaluate with multi-step multimodal prompts, MiniGPT-5 can produce more coherent and high-quality images.
+ Qualitative examples from EditRoom and baselines on single- and multi-operation editing. From the comparisons, we can find the EditRoom can provide more accurate and coherent editing results than other baselines, and it can generalize to multi-operation editing tasks without training on such data.
- 
+ 
+
+
- Qualitative examples from MiniGPT-5 and baselines on the CC3M, VIST, and MMDialog datasets. From the comparisons, we can find the MiniGPT-5 and SD 2 have similar results on single-image generation. When we evaluate with multi-step multimodal prompts, MiniGPT-5 can produce more coherent and high-quality images. + Qualitative examples from EditRoom and baselines on single- and multi-operation editing. From the comparisons, we can find the EditRoom can provide more accurate and coherent editing results than other baselines, and it can generalize to multi-operation editing tasks without training on such data.
Figure 3. Comparison with other baselines on single-operation editing.
+ +
-
Figure 3. Comparison with other baselines.
+Figure 4. Comparison with other baselines on multi-operation editing.
BibTeX
- @misc{zheng2023minigpt5,
- title={MiniGPT-5: Interleaved Vision-and-Language Generation via Generative Vokens},
- author={Kaizhi Zheng and Xuehai He and Xin Eric Wang},
- year={2023},
- journal={arXiv preprint arXiv:2310.02239}
+ @misc{zheng2024editroomllmparameterizedgraphdiffusion,
+ title={EditRoom: LLM-parameterized Graph Diffusion for Composable 3D Room Layout Editing},
+ author={Kaizhi Zheng and Xiaotong Chen and Xuehai He and Jing Gu and Linjie Li and Zhengyuan Yang and Kevin Lin and Jianfeng Wang and Lijuan Wang and Xin Eric Wang},
+ year={2024},
+ eprint={2410.12836},
+ archivePrefix={arXiv},
+ primaryClass={cs.GR},
+ url={https://arxiv.org/abs/2410.12836},
}
diff --git a/static/images/compare-arxiv.png b/static/images/compare-arxiv.png
deleted file mode 100644
index 238f2f3..0000000
Binary files a/static/images/compare-arxiv.png and /dev/null differ
diff --git a/static/images/structure.png b/static/images/structure.png
deleted file mode 100644
index 8ff45b8..0000000
Binary files a/static/images/structure.png and /dev/null differ
diff --git a/static/images/teaser.png b/static/images/teaser.png
deleted file mode 100644
index 4df9a44..0000000
Binary files a/static/images/teaser.png and /dev/null differ
BibTeX
-@misc{zheng2023minigpt5,
- title={MiniGPT-5: Interleaved Vision-and-Language Generation via Generative Vokens},
- author={Kaizhi Zheng and Xuehai He and Xin Eric Wang},
- year={2023},
- journal={arXiv preprint arXiv:2310.02239}
+ @misc{zheng2024editroomllmparameterizedgraphdiffusion,
+ title={EditRoom: LLM-parameterized Graph Diffusion for Composable 3D Room Layout Editing},
+ author={Kaizhi Zheng and Xiaotong Chen and Xuehai He and Jing Gu and Linjie Li and Zhengyuan Yang and Kevin Lin and Jianfeng Wang and Lijuan Wang and Xin Eric Wang},
+ year={2024},
+ eprint={2410.12836},
+ archivePrefix={arXiv},
+ primaryClass={cs.GR},
+ url={https://arxiv.org/abs/2410.12836},
}