OmniHuman-1: A Multimodality-Conditioned Human Video Generation Framework

OmniHuman-1 is a novel end-to-end framework for generating realistic human videos from various input modalities, including single images and audio or video signals, overcoming limitations of previous methods due to scarce high-quality data.

This framework employs a multimodality motion conditioning mixed training strategy, allowing it to scale up effectively by leveraging diverse conditioning signals and generating videos with realistic motion, lighting, and texture details, regardless of the input image's aspect ratio (portrait, half-body, or full-body).

OmniHuman-1 demonstrates impressive capabilities in handling diverse input styles, including cartoons, artificial objects, and challenging poses, adapting motion characteristics to match each style and showcasing superior performance in generating realistic gestures, especially when driven by audio.

Beyond audio-driven generation, OmniHuman-1 also supports video-driven and combined audio-video driving, enabling precise control over specific body movements and actions; the researchers emphasize ethical considerations, noting that all demo materials are from public sources or AI-generated and are readily removed upon request.

https://omnihuman-lab.github.io/