Awesome Controllable Generation

Papers and Resources on Adding Conditional Controls to Diffusion Models in the Era of AIGC.

Dive into the cutting-edge of controllable generation in diffusion models, a field revolutionized by pioneering works like ControlNet [1] and DreamBooth [2]. This repository is invaluable for those interested in advanced techniques for fine-grained synthesis control, ranging from subject-driven generation to intricate layout manipulations. While ControlNet and DreamBooth are key highlights, the collection spans a broader spectrum, including recent advancements and applications in image, video, and 3D generation.

📝 Papers

1. DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation.

   

   Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, Kfir Aberman. CVPR'23. 🔥

   

2. Adding Conditional Control to Text-to-Image Diffusion Models.

   

   Lvmin Zhang, Anyi Rao, Maneesh Agrawala. ICCV'23. 🔥

   

3. T2I-Adapter: Learning Adapters to Dig out More Controllable Ability for Text-to-Image Diffusion Models.

   

   Chong Mou, Xintao Wang, Liangbin Xie, Yanze Wu, Jian Zhang, Zhongang Qi, Ying Shan, Xiaohu Qie. Preprint 2023. 🔥

   

4. Subject-driven Text-to-Image Generation via Apprenticeship Learning.

   

   Wenhu Chen, Hexiang Hu, Yandong Li, Nataniel Ruiz, Xuhui Jia, Ming-Wei Chang, William W. Cohen. NeurIPS'23.

5. InstantBooth: Personalized Text-to-Image Generation without Test-Time Finetuning.

   

   Jing Shi, Wei Xiong, Zhe Lin, Hyun Joon Jung. Preprint 2023.

6. BLIP-Diffusion: Pre-trained Subject Representation for Controllable Text-to-Image Generation and Editing

   

   Dongxu Li, Junnan Li, Steven C.H. Hoi. NeurIPS'23. 🔥

   

7. ControlVideo: Conditional Control for One-shot Text-driven Video Editing and Beyond.

   

   Min Zhao, Rongzhen Wang, Fan Bao, Chongxuan Li, Jun Zhu. Preprint 2023.

8. StyleDrop: Text-to-Image Generation in Any Style.

   

   Kihyuk Sohn, Nataniel Ruiz, Kimin Lee, Daniel Castro Chin, Irina Blok, Huiwen Chang, Jarred Barber, Lu Jiang, Glenn Entis, Yuanzhen Li, Yuan Hao, Irfan Essa, Michael Rubinstein, Dilip Krishnan. NeurIPS'23. 🔥

   

9. Face0: Instantaneously Conditioning a Text-to-Image Model on a Face.

   

   Dani Valevski, Danny Wasserman, Yossi Matias, Yaniv Leviathan. SIGGRAPH Asia'23.

10. Controlling Text-to-Image Diffusion by Orthogonal Finetuning.

    

    Zeju Qiu, Weiyang Liu, Haiwen Feng, Yuxuan Xue, Yao Feng, Zhen Liu, Dan Zhang, Adrian Weller, Bernhard Schölkopf. NeruIPS'23.

11. Zero-shot spatial layout conditioning for text-to-image diffusion models.

    

    Guillaume Couairon, Marlène Careil, Matthieu Cord, Stéphane Lathuilière, Jakob Verbeek. ICCV'23.

12. IP-Adapter: Text Compatible Image Prompt Adapter for Text-to-Image Diffusion Models.

    

    Hu Ye, Jun Zhang, Sibo Liu, Xiao Han, Wei Yang. Preprint 2023. 🔥

    

13. StyleAdapter: A Single-Pass LoRA-Free Model for Stylized Image Generation.

    

    Zhouxia Wang, Xintao Wang, Liangbin Xie, Zhongang Qi, Ying Shan, Wenping Wang, Ping Luo. Preprint 2023.

14. DreamStyler: Paint by Style Inversion with Text-to-Image Diffusion Models.

    

    Namhyuk Ahn, Junsoo Lee, Chunggi Lee, Kunhee Kim, Daesik Kim, Seung-Hun Nam, Kibeom Hong. AAAI 2023.

15. Kosmos-G: Generating Images in Context with Multimodal Large Language Models

    

    Xichen Pan, Li Dong, Shaohan Huang, Zhiliang Peng, Wenhu Chen, Furu Wei. Preprint 2023. 🔥

    

16. An Image is Worth Multiple Words: Learning Object Level Concepts using Multi-Concept Prompt Learning.

    

    Chen Jin, Ryutaro Tanno, Amrutha Saseendran, Tom Diethe, Philip Teare. Preprint 2023.

17. CustomNet: Zero-shot Object Customization with Variable-Viewpoints in Text-to-Image Diffusion Models.

    

    Ziyang Yuan, Mingdeng Cao, Xintao Wang, Zhongang Qi, Chun Yuan, Ying Shan. Preprint 2023.

18. Cross-Image Attention for Zero-Shot Appearance Transfer.

    

    Yuval Alaluf, Daniel Garibi, Or Patashnik, Hadar Averbuch-Elor, Daniel Cohen-Or. Preprint 2023.

19. The Chosen One: Consistent Characters in Text-to-Image Diffusion Models.

    

    Omri Avrahami, Amir Hertz, Yael Vinker, Moab Arar, Shlomi Fruchter, Ohad Fried, Daniel Cohen-Or, Dani Lischinski. Preprint 2023.

20. MagicDance: Realistic Human Dance Video Generation with Motions & Facial Expressions Transfer.

    

    Di Chang, Yichun Shi, Quankai Gao, Jessica Fu, Hongyi Xu, Guoxian Song, Qing Yan, Xiao Yang, Mohammad Soleymani. Preprint 2023.

21. ZipLoRA: Any Subject in Any Style by Effectively Merging LoRAs.

    

    Viraj Shah, Nataniel Ruiz, Forrester Cole, Erika Lu, Svetlana Lazebnik, Yuanzhen Li, Varun Jampani. Preprint 2023.

22. StyleCrafter: Enhancing Stylized Text-to-Video Generation with Style Adapter.

    

    Gongye Liu, Menghan Xia, Yong Zhang, Haoxin Chen, Jinbo Xing, Xintao Wang, Yujiu Yang, Ying Shan. Preprint 2023.

23. Style Aligned Image Generation via Shared Attention.

    

    Amir Hertz, Andrey Voynov, Shlomi Fruchter, Daniel Cohen-Or. Preprint 2023. 🔥

    

24. FaceStudio: Put Your Face Everywhere in Seconds.

    

    Yuxuan Yan, Chi Zhang, Rui Wang, Yichao Zhou, Gege Zhang, Pei Cheng, Gang Yu, Bin Fu. Preprint 2023.

25. Context Diffusion: In-Context Aware Image Generation.

    

    Ivona Najdenkoska, Animesh Sinha, Abhimanyu Dubey, Dhruv Mahajan, Vignesh Ramanathan, Filip Radenovic. Preprint 2023.

26. PhotoMaker: Customizing Realistic Human Photos via Stacked ID Embedding.

    

    Zhen Li, Mingdeng Cao, Xintao Wang, Zhongang Qi, Ming-Ming Cheng, Ying Shan. Preprint 2023. 🔥

    

27. SCEdit: Efficient and Controllable Image Diffusion Generation via Skip Connection Editing.

    

    *Zeyinzi Jiang, Chaojie Mao, Yulin Pan, Zhen