Use Unity 3D character and Python deep learning algorithms to stream as a VTuber!

This is part of the OpenVTuberProject, which provides many toolkits for becoming a VTuber.

Youtube Playlist (Chinese) (Covers videos 1-4):

First of all, I'd like to give credits to the following projects that I borrow code from:

And the virtual character unity-chan © UTJ/UCL.

• OS: Ubuntu 16.04 (18.04 may also work) or Windows 10 64bits or MacOS
• (Optional but recommended) An NVIDIA GPU (tested with CUDA 9.0, 10.0 and 10.1, but may also work with other versions)

• Python3.x (installation via Anaconda is recommended; mandatory for Windows users)

  • (Optional) It is recommended to use conda environments. Run conda create -n vtuber python=3.6. Activate it by conda activate vtuber.

• Python libraries

  • Ubuntu:
    • Install the requirements by pip install -r requirements_(cpu or gpu).txt
    • If you have CUDA 10.1, pip install onnxruntime-gpu to get faster inference speed using onnx model.
  • Windows:
    • CPU:
      • pip install -r requirements_cpu.txt
      • if dlib cannot be properly installed, follow here.
    • GPU:
      • Install pytorch using conda. Example: conda install pytorch==1.2.0 torchvision==0.4.0 cudatoolkit=10.0 -c pytorch
      • Install other dependencies by pip install -r requirements_gpu.txt.
      • If you have CUDA 10, pip install onnxruntime-gpu to get faster inference speed using onnx model.

• Optional

  • OBS Studio if you want to embed the virtual character into your videos.
  • Unity Editor if you want to customize the virtual character.
    • Linux installation
    • Windows installation

Here we assume that you have installed the requirements and activated the virtual environment you are using.

You need to download the models here, extract and put into face_alignment/ckpts.

If you don't use onnxruntime, you can omit this step as the script will automatically download them for you.

Run python demo.py --debug. (add --cpu if you have CPU only)

You should see the following:

^{Left: CPU model. Right: GPU model run on a GTX1080Ti.}

1. Download and launch the binaries here depending on your OS to launch the unity window featuring the virtual character (unity-chan here). Important: Ensure that only one window is opened at a time!
2. After the vitual character shows up, run python demo.py --connect to synchronize your face features with the virtual character. (add --debug to see your face and --cpu if you have CPU only as step 1.)

You should see the following:

^{Left: CPU model. Right: GPU model run on a GTX1080Ti.}

Enjoy your VTuber life!

In this section, I will describe the functionalities implemented and a little about the technology behind.

Using head-pose-estimation and face-alignment, deep learning methods are applied to do the following: face detection and facial landmark detection. A face bounding box and the 68-point facial landmark is detected, then a PnP algorithm is used to obtain the head pose (the rotation of the face). Finally, kalman filters are applied to the pose to make it smoother.

The character's head pose is synchronized.

As for the visualization, the white bounding box is the detected face, on top of which 68 green face landmarks are plotted. The head pose is represented by the green frustum and the axes in front of the nose.

Using GazeTracking, The eyes are first extracted using the landmarks enclosing the eyes. Then the eye images are converted to grayscale, and a pixel intensity threshold is applied to detect the iris (the black part of the eye). Finally, the center of the iris is computed as the center of the black area.

The character's gaze is not synchronized. (Since I didn't find a way to move unity-chan's eyes)

As for the visualization, the red crosses indicate the iris.

1. Estimate eye aspect ratio: The eye aspect ratio can be used to detect blinking, but currently I just use auto blinking since this estimation is not so accurate.

2. Estimate mouth aspect ratio: I use this number to synchronize with the character's mouth.

3. The mouth distance is used to detect smile and synchronize with the character.

If you want to customize the virtual character, you can find the unity project in release.

MIT License