Alibaba Group Holding’s cloud computing arm has introduced its updated artificial intelligence (AI) model, Animate Anyone 2, which the company touts as simplifying the animation of lifelike characters with just a static image and one video cue.
Unlike previous character-animation methods that solely use motion signals, Animate Anyone 2 achieves high-fidelity character image animation by processing both motion and environmental signals from reference content, such as source videos, to generate new clips, according to a technical study published last week by Tongyi Lab, the large language model (LLM) research and development unit of Alibaba Cloud, via arXiv – an open-access repository of academic papers. Alibaba owns the South China Morning Post.
“We extract environmental information from [source] videos, enabling the animated character to preserve its original environment … facilitating seamless character-environment integration,” the study said.
The new model highlights the progress being made by Chinese developers in the generative AI field, in spite of Washington’s restrictions on the mainland’s access to advanced semiconductors and other technologies. LLMs are the technology underpinning generative AI services like ChatGPT.
Animate Anyone 2 was built on top of the model’s first iteration that was announced in late 2023, focused on generating character videos from still images.
ChatGPT creator OpenAI, subsequently, set the AI industry ablaze with its introduction of text-to-video model Sora in February last year, which triggered a race among Chinese Big Tech companies and start-ups to release similar products.
Sora was only made available to ChatGPT Plus and Pro users in December.
With Animate Anyone 2, China appears to have upped the ante in this field. This comes weeks after ByteDance introduced its OmniHuman-1 multimodal model, which can transform photos and sound bites into realistic-looking videos.
The latest Chinese generative AI models are now poised to disrupt the entertainment and advertising industries in China, where Sora is not available, by refining how the characters from a reference image or video can easily be replaced with another to create entirely new clips.
Animate Anyone 2, for example, is also capable of generating interactions between characters, “ensuring the plausibility of their movements and coherence with the surrounding environment”, the study said.
To validate Animate Anyone 2’s method across more diverse scenarios, the study said researchers curated a data set of 100,000 character videos collected from the internet, encompassing a broad range of scene types, action categories and human-object interaction cases.
The wider use of these next-generation video-generation models, however, poses the risk of spreading more deepfake videos online.
While these new AI video tools appear to be promising, “they fall short in terms of conveying human emotions and detailed facial expressions on video”, said Liang Haisheng, who runs a Beijing-based advertising agency focused on making commercials and documentaries.
Still, Liang said the latest video-generation tools come in handy when making sample videos to communicate project ideas to clients. – South China Morning Post
