Alibaba has unveiled EMO, or Emote Portrait Alive. Alibaba’s EMO is an AI model that helps users generate videos by adding lip sync to still images and portraits.  The AI is able to add lip sync to still images and make them sing, rap, and talk from audio files. So users can take an image of a song and make the character in the image look like it is singing the song by adding lip sync.

The generated output looks very real, as along with the lips of characters in the image, their head position and facial expressions also change accordingly. In some cases, it is hard to identify what’s real and what’s not. Alibaba’s EMO can also generate videos of any duration, depending on the duration of the audio file.

Alibaba shared demo videos on GitHub showcasing the work done by Alibaba’s EMO. One example showed the famous Asian lady generated by OpenAI’s Sora model singing Dua Lipa’s song with just a reference image as an input and an audio file of the song.

Technical Details of Alibaba’s EMO

The model is trained on a vast and diverse audio-video dataset consisting of more than 250 hours of footage and 150 million images from various sources, including films, speeches, music performances, and more, in different languages, including Chinese and English.

It takes the reference image and adds it to video frames, which are generated during the diffusion process. The diffusion process involves converting audio waves into video frames. This captures subtle facial motions and identity-specific nuances associated with natural speech, making the resultant output video look more realistic.

Alibaba’s researchers, mentioning the problem with traditional audio-video models, wrote in their research paper, “We identify the limitations of traditional techniques that often fail to capture the full spectrum of human expressions and the uniqueness of individual facial styles. To address these issues, we propose: EMO, a novel framework that utilizes direct audio-to-video synthesis approach, bypassing the need for intermediate 3D models or facial landmarks.”

For more technical details, you can read EMO’s research paper here.

Limitations of Alibaba’s EMO

Alibaba, in their research paper, mentioned some limitations of EMO. The researchers mentioned that it is more time-consuming than other methods that don’t rely on diffusion models. Moreover, as they don’t use explicit control signals to control characters’ emotions, it can lead to the inadvertent generation of body parts such as hands, resulting in artifacts in the video.

Conclusion

Alibaba’s EMO AI model is going to make AI-generated videos more realistic with its diffusion model approach. This can lead to an increase in the number of deep-fake videos that can be generated using this model. So the company needs to take important steps before fully releasing the model to the public. Although the model still has some limitations, it will be interesting to see how Alibaba refines it to get accurate results.

Read More

OpenAI to Now add AI to Humanoid Robots

Adobe is Working on a New AI Music Generation and Editing Tool