Project Icon

DigiHuman

基于摄像头输入的3D角色全身动画生成系统

DigiHuman是一个开源的人工智能项目,通过摄像头输入自动生成3D角色模型的全身和面部动画。系统结合MediaPipe生成3D关键点和Unity3D渲染技术,实现了全身动作、面部表情等多种动画效果。支持多个混合形状动画、视频导出等功能,为3D虚拟角色动画制作提供了自动化解决方案。该项目支持多种3D模型类型,并优化了动画平滑度,为动画创作者和研究人员提供了创新工具和无限可能。

DigiHuman

Digihuman is a project which aims to automatically generate whole body pose animation + facial animation on 3D Character models based on the camera input.
This project is my B.Sc thesis of Computer Engineering at Amirkabir University of Technology(AUT).

About DigiHuman

DigiHuman is a system for bringing automation in animation generation on 3D virtual characters. It uses Pose estimation and facial landmark generator models to create entire body and face animation on 3D virtual characters.
DigiHuman is developed with MediaPipe and Unity3D. MediaPipe generates 3D landmarks for the human whole body and face, and Unity3D is used to render the final animation after processing the generated landmarks from MediaPipe. The diagram below, shows the whole architucture of the application.

Logo

Sample Outputs of the project

Hands animations

Full body animation

Face animation

Installation

Follow the instructions to run the program!

Backend server installtion

  1. Install MediaPipe python.
 pip install mediapipe
  1. Install OpenCV python.
 pip install opencv-python
  1. Go to backend directory and install other requirements:
 pip install -r requirements.txt
  1. You'll need to download the pre-trained generator model for the COCO dataset and place it into backend/checkpoints/coco_pretrained/.

Unity3D Installation

Install Unity3D and its requirements by the following guidelines(Skip 1-3 if Unity3D is already installed).

  1. Download and install UnityHub
  2. Add a new license in UnityHub and register it
  3. Install a Unity Editor inside UnityHub(LTS versions and a version higher than 2020.3.25f1 are recommended).
  4. In the Unity project setting, allow HTTP connections in the player setting.
  1. Download and import the following packages into your project to enable the recording option available with FFmpeg(Download .unitypackage files and drag them to your project).

Usage

  • Run backend server at backend directory with the following command:
     python server.py
    
  • Run Unity Project and open the main scene at Assets\Scenes\MainScene.unity
  • Test the program by uploading videos to backend from the Unity project(You can test the application by selecting provided animations from the right side menu!).

Adding new 3D characters

You can add your characters to the project! Characters should have a standard Humanoid rig to show kinematic animations. For rendering face animations, characters should have a facial rig(Blendmesh).
Follow these steps to add your character:

  1. Find a 3D character model from Unity asset store or download a free one(You can download them from websites like Mixamo).
  2. Open the character setting and set the rig to humanoid
  1. Drag and drop your 3D character model to CharacterChooser/CharacterSlideshow/Parent object in Unity main Scene like the image below
  1. Add BlendShapeController and QualityData components to the character object in the scene(which is dragged inside the Parent object in the last step).
  2. Set BlendShapeController values
  • Add character SkinnedMeshRenderer component to BlendShapeController component.
  • Find each blnedShape weight number under SkinnedMeshRenderer and set those numbers in BlendShapes field inside BlendShapeController (for specifying each blendshape value to the BlendShapeController component so the animation would be shown on character face by modification on these blnedShape values)
  1. Open CharacterSlideshow Object on CharacterChooser/CharacterSlideshow path inside the scene hierarchy, then add a new dragged character to the nodes property(all characters should be referenced inside nodes).
  1. Run the application and you can now select your character for rendering animation!

Features

  • Making full body animation
  • Animating multiple blendShapes on 3D character (up to 40 blendshape animations is supported currently)
  • Supporting any 3D models with Humanoid T-Pose rig
  • Exporting animation in a video file
  • Saving animation data and re-rendering it for future usage
  • Filtering mediaPipe outputs in order to detect and remove noises and better smoothness (Low Pass Filtering is used currently)
  • Animating the character's face in great details
    • Training a regression model to generate Blendmesh weights by feeding the output data of mediaPipe FaceMesh(468 points)
    • Using StyleGan techniques to replace whole character face mesh
  • Automatic rigging for 3D models without humanoid rig (Using deep neural network models like RigNet)
  • Generating complete character mesh automatically using models like PIFuHD (in progress!)
  • Animating 3D character mouth in great detail using audio signal or natural language processing methods
  • Generating complete environment in 3D

Resources

Licenses & Citations

DigiHuman Licence

Application License: GPL-3.0 license Non-commercial use only. If you distribute or communicate copies of the modified or unmodified Program, or any portion thereof, you must provide appropriate credit to Danial Kordmodanlou as the original author of the Program. This attribution should be included in any location where the Program is used or displayed.

FFmpeg

GauGan

@inproceedings{park2019SPADE,
  title={Semantic Image Synthesis with Spatially-Adaptive Normalization},
  author={Park, Taesung and Liu, Ming-Yu and Wang, Ting-Chun and Zhu, Jun-Yan},
  booktitle={Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition},
  year={2019}
}

3D Characters

Unity-chan model & mixamo models

Contact

Danial Kordmodanlou - kordmodanloo@gmail.com

Website : danial-kord.github.io

Project Link: github.com/Danial-Kord/DigiHuman

Telegram ID:

项目侧边栏1项目侧边栏2
推荐项目
Project Cover

豆包MarsCode

豆包 MarsCode 是一款革命性的编程助手,通过AI技术提供代码补全、单测生成、代码解释和智能问答等功能,支持100+编程语言,与主流编辑器无缝集成,显著提升开发效率和代码质量。

Project Cover

问小白

问小白是一个基于 DeepSeek R1 模型的智能对话平台,专为用户提供高效、贴心的对话体验。实时在线,支持深度思考和联网搜索。免费不限次数,帮用户写作、创作、分析和规划,各种任务随时完成!

Project Cover

白日梦AI

白日梦AI提供专注于AI视频生成的多样化功能,包括文生视频、动态画面和形象生成等,帮助用户快速上手,创造专业级内容。

Project Cover

有言AI

有言平台提供一站式AIGC视频创作解决方案,通过智能技术简化视频制作流程。无论是企业宣传还是个人分享,有言都能帮助用户快速、轻松地制作出专业级别的视频内容。

Project Cover

讯飞绘镜

讯飞绘镜是一个支持从创意到完整视频创作的智能平台,用户可以快速生成视频素材并创作独特的音乐视频和故事。平台提供多样化的主题和精选作品,帮助用户探索创意灵感。

Project Cover

讯飞文书

讯飞文书依托讯飞星火大模型,为文书写作者提供从素材筹备到稿件撰写及审稿的全程支持。通过录音智记和以稿写稿等功能,满足事务性工作的高频需求,帮助撰稿人节省精力,提高效率,优化工作与生活。

Project Cover

阿里绘蛙

绘蛙是阿里巴巴集团推出的革命性AI电商营销平台。利用尖端人工智能技术,为商家提供一键生成商品图和营销文案的服务,显著提升内容创作效率和营销效果。适用于淘宝、天猫等电商平台,让商品第一时间被种草。

Project Cover

Trae

Trae是一种自适应的集成开发环境(IDE),通过自动化和多元协作改变开发流程。利用Trae,团队能够更快速、精确地编写和部署代码,从而提高编程效率和项目交付速度。Trae具备上下文感知和代码自动完成功能,是提升开发效率的理想工具。

Project Cover

AIWritePaper论文写作

AIWritePaper论文写作是一站式AI论文写作辅助工具,简化了选题、文献检索至论文撰写的整个过程。通过简单设定,平台可快速生成高质量论文大纲和全文,配合图表、参考文献等一应俱全,同时提供开题报告和答辩PPT等增值服务,保障数据安全,有效提升写作效率和论文质量。

投诉举报邮箱: service@vectorlightyear.com
@2024 懂AI·鲁ICP备2024100362号-6·鲁公网安备37021002001498号