Paper list for Embodied AI

HCPLab
Pengcheng Laboratory & SYSU HCP Lab

We appreciate any useful suggestions for improvement of this paper list or survey from peers. Please raise issues or send an email to liuy856@mail.sysu.edu.cn and chen867820261@gmail.com. Thanks for your cooperation! We also welcome your pull requests for this project!

Teaser

Aligning Cyber Space with Physical World: A Comprehensive Survey on Embodied AI
Yang Liu, Weixing Chen, Yongjie Bai, Guanbin Li, Wen Gao, Liang Lin

🏠 About

Embodied Artificial Intelligence (Embodied AI) is crucial for achieving Artificial General Intelligence (AGI) and serves as a foundation for various applications that bridge cyberspace and the physical world. Recently, the emergence of Multi-modal Large Models (MLMs) and World Models (WMs) have attracted significant attention due to their remarkable perception, interaction, and reasoning capabilities, making them a promising architecture for the brain of embodied agents. However, there is no comprehensive survey for Embodied AI in the era of MLMs. In this survey, we give a comprehensive exploration of the latest advancements in Embodied AI. Our analysis firstly navigates through the forefront of representative works of embodied robots and simulators, to fully understand the research focuses and their limitations. Then, we analyze four main research targets: 1) embodied perception, 2) embodied interaction, 3) embodied agent, and 4) sim-to-real adaptation, covering the state-of-the-art methods, essential paradigms, and comprehensive datasets. Additionally, we explore the complexities of MLMs in virtual and real embodied agents, highlighting their significance in facilitating interactions in dynamic digital and physical environments. Finally, we summarize the challenges and limitations of embodied AI and discuss their potential future directions. We hope this survey will serve as a foundational reference for the research community and inspire continued innovation.

:collision: Update Log

[2024.08.02] We regularly update the project weekly!
[2024.07.29] We have updated the project!
[2024.07.22] We have updated the paper list and other useful embodied projects!
[2024.07.10] We release the first version of the survey on Embodied AI PDF!
[2024.07.10] We release the first version of the paper list for Embodied AI. This page is continually updating!

Books & Surveys 🔝

Multimodal Large Models: The New Paradigm of Artificial General Intelligence, Publishing House of Electronics Industry (PHE), 2024
Yang Liu, Liang Lin
[Page]
Aligning Cyber Space with Physical World: A Comprehensive Survey on Embodied AI, arXiv:2407.06886, 2024
Yang Liu, Weixing Chen, Yongjie Bai, Guanbin Li, Wen Gao, Liang Lin.
[Paper]
Teleoperation of Humanoid Robots: A Survey, IEEE Transactions on Robotics, 2024
Kourosh Darvish, Luigi Penco, Joao Ramos, Rafael Cisneros, Jerry Pratt, Eiichi Yoshida, Serena Ivaldi, Daniele Pucci.
[Paper]
A Survey on Vision-Language-Action Models for Embodied AI, arXiv:2405.14093, 2024
Yueen Ma, Zixing Song, Yuzheng Zhuang, Jianye Hao, Irwin King
[Paper]
Towards Generalist Robot Learning from Internet Video: A Survey, arXiv:2404.19664, 2024
McCarthy, Robert, Daniel CH Tan, Dominik Schmidt, Fernando Acero, Nathan Herr, Yilun Du, Thomas G. Thuruthel, and Zhibin Li.
[Paper]
A Survey on Robotics with Foundation Models: toward Embodied AI, arXiv:2402.02385, 2024
Zhiyuan Xu, Kun Wu, Junjie Wen, Jinming Li, Ning Liu, Zhengping Che, and Jian Tang.
[Paper]
Toward general-purpose robots via foundation models: A survey and meta-analysis, Machines, 2023
Liman Wang, Jihong Zhu.
[Paper]
Deformable Object Manipulation in Caregiving Scenarios: A Review, arXiv:2312.08782, 2023
Yafei Hu, Quanting Xie, Vidhi Jain, Jonathan Francis, Jay Patrikar, Nikhil Keetha, Seungchan Kim et al.
[Paper]
A survey of embodied ai: From simulators to research tasks, IEEE Transactions on Emerging Topics in Computational Intelligence, 2022
Jiafei Duan, Samson Yu, Hui Li Tan, Hongyuan Zhu, Cheston Tan
[Paper]
The development of embodied cognition: Six lessons from babies, Artificial life, 2005
Linda Smith, Michael Gasser
[Paper]
Embodied artificial intelligence: Trends and challenges, Lecture notes in computer science, 2004
Rolf Pfeifer, Fumiya Iida
[Paper]

Embodied Simulators 🔝

General Simulator

Nvidia isaac sim: Robotics simulation and synthetic data, NVIDIA, 2023 [page]
Design and use paradigms for gazebo, an open-source multi-robot simulator, IROS, 2004 Koenig, Nathan, Andrew, Howard. [page]
Pybullet, a python module for physics simulation for games, robotics and machine learning, 2016 Coumans, Erwin, Yunfei, Bai.
Webots: open-source robot simulator Cyberbotics [page, code]
MuJoCo: A physics engine for model-based control, IROS, 2012 Todorov, Emanuel, Tom, Erez, Yuval, Tassa. [page, code]
Unity: A general platform for intelligent agents, ArXiv, 2020 Juliani, Arthur, Vincent-Pierre, Berges, Ervin, Teng, Andrew, Cohen, Jonathan, Harper, Chris, Elion, Chris, Goy, Yuan, Gao, Hunter, Henry, Marwan, Mattar, Danny, Lange. [page]
AirSim: High-Fidelity Visual and Physical Simulation for Autonomous Vehicles, Field and Service Robotics, 2017 Shital Shah, , Debadeepta Dey, Chris Lovett, Ashish Kapoor. [page]
Aerial Gym -- Isaac Gym Simulator for Aerial Robots, ArXiv, 2023 Mihir Kulkarni and Theodor J. L. Forgaard and Kostas Alexis. [paper]
Modular open robots simulation engine: Morse, ICRA, 2011 Echeverria, Gilberto and Lassabe, Nicolas and Degroote, Arnaud and Lemaignan, S{'e}verin [page]
V-REP: A versatile and scalable robot simulation framework, IROS, 2013 Rohmer, Eric, Surya PN, Singh, Marc, Freese. [page]

Real-Scene Based Simulators

ThreeDWorld: A Platform for Interactive Multi-Modal Physical Simulation, NeurIPS, 2021
Gan, Chuang, J., Schwartz, Seth, Alter, Martin, Schrimpf, James, Traer, JulianDe, Freitas, Jonas, Kubilius, Abhishek, Bhandwaldar, Nick, Haber, Megumi, Sano, Kuno, Kim, Elias, Wang, Damian, Mrowca, Michael, Lingelbach, Aidan, Curtis, KevinT., Feigelis, DavidM., Bear, Dan, Gutfreund, DavidD., Cox, JamesJ., DiCarlo, JoshH., McDermott, JoshuaB., Tenenbaum, Daniel, Yamins.
[page]
iGibson 1.0: A Simulation Environment for Interactive Tasks in Large Realistic Scenes, IROS, 2021
Shen, Bokui, Fei, Xia, Chengshu, Li, Roberto, Martín-Martín, Linxi, Fan, Guanzhi, Wang, Claudia, Pérez-D’Arpino, Shyamal, Buch, Sanjana, Srivastava, Lyne, Tchapmi, Micael, Tchapmi, Kent, Vainio, Josiah, Wong, Li, Fei-Fei, Silvio, Savarese.
[page]
SAPIEN: A SimulAted Part-Based Interactive ENvironment, CVPR, 2020
Xiang, Fanbo, Yuzhe, Qin, Kaichun, Mo, Yikuan, Xia, Hao, Zhu, Fangchen, Liu, Minghua, Liu, Hanxiao, Jiang, Yifu, Yuan, He, Wang, Li, Yi, Angel X., Chang, Leonidas J., Guibas, Hao, Su.
[page]
Habitat: A Platform for Embodied AI Research, ICCV, 2019
Savva, Manolis, Abhishek, Kadian, Oleksandr, Maksymets, Yili, Zhao, Erik, Wĳmans, Bhavana, Jain, Julian, Straub, Jia, Liu, Vladlen, Koltun, Jitendra, Malik, Devi, Parikh, Dhruv, Batra.
[page]
VirtualHome: Simulating Household Activities Via Programs, CVPR, 2018
Puig, Xavier, Kevin, Ra, Marko, Boben, Jiaman, Li, Tingwu, Wang, Sanja, Fidler, Antonio, Torralba.
[page]
Matterport3D: Learning from RGB-D Data in Indoor Environments, 3DV, 2017
Chang, Angel, Angela, Dai, Thomas, Funkhouser, Maciej, Halber, Matthias, Niebner, Manolis, Savva, Shuran, Song, Andy, Zeng, Yinda, Zhang.
[page]
AI2-THOR: An Interactive 3D Environment for Visual AI. arXiv, 2017
Kolve, Eric, Roozbeh, Mottaghi, Daniel, Gordon, Yuke, Zhu, Abhinav, Gupta, Ali, Farhadi.
[page]
ProcTHOR: Large-Scale Embodied AI Using Procedural Generation, NeurIPS, 2022
Deitke, VanderBilt, Herrasti, Weihs, Salvador, Ehsani, Han, Kolve, Farhadi, Kembhavi, Mottaghi
[page]
RoboGen: Towards Unleashing Infinite Data for Automated Robot Learning via Generative Simulation, arXiv, 2023
Wang, Yufei, Zhou, Xian, Feng, Chen, Tsun-Hsuan, Wang, Yian, Wang, Katerina, Fragkiadaki, Zackory, Erickson, David, Held, Chuang, Gan.
[page]
Holodeck: Language Guided Generation of 3D Embodied AI Environments, CVPR, 2024
Yue Yang, , Fan-Yun Sun, Luca Weihs, Eli VanderBilt, Alvaro Herrasti, Winson Han, Jiajun Wu, Nick Haber, Ranjay Krishna, Lingjie Liu, Chris Callison-Burch, Mark Yatskar, Aniruddha Kembhavi, Christopher Clark.
[page]
PhyScene: Physically Interactable 3D Scene Synthesis for Embodied AI, arxiv, 2024
Yang, Yandan, Baoxiong, Jia, Peiyuan, Zhi, Siyuan, Huang.
[page]

Embodied Perception 🔝

Active Visual Exploration

MonoSLAM: Real-time single camera SLAM, IEEE T-PAMI 29. 6(2007): 1052–1067
Davison, Andrew J, Ian D, Reid, Nicholas D, Molton, Olivier, Stasse.
[page]
A multi-state constraint Kalman filter for vision-aided inertial navigation, IROS, 2007
Mourikis, Anastasios I, Stergios I, Roumeliotis.
[page]
Parallel tracking and mapping for small AR workspaces, ISMAR, 2007
Klein, Georg, David, Murray.
[page]
ORB-SLAM: a versatile and accurate monocular SLAM system IEEE T-RO 31.