The new, seamless solution leverages Reallusion’s iClone, Character Creator, and NVIDIA’s Audio2Face to revolutionize multi-lingual facial lip-sync animation production.
Reallusion has just announced the all-new, seamless integration between Character Creator (CC), iClone, and NVIDIA’s Audio2Face (A2F). This robust connection, empowered by NVIDIA’s AI animation technology, revolutionizes multi-lingual facial lip-sync animation production. Not only does this integration bolster A2F with a versatile cross-application character system, it also enhances facial editing capabilities, enabling users to export character animations to leading 3D engines like Blender, Unreal Engine, Unity, and NVIDIA Omniverse.
Non-Linear AI Animation Generated by Audio2Face
Lip-Sync Animation and Expressions Straight from Audio
As an AI-powered application, A2F produces expressive facial animations solely from audio input. In addition to generating natural lip-sync animations for multilingual dialogue, the latest standalone release of A2F also supports facial expressions, featuring slider controls and a keyframe editor.
Multi-Language Lip-Sync and Singing Animation
Unlike the majority of English-centric lip-sync solutions, A2F stands out with its exceptional ability to generate animation from any language, including songs and gibberish. Besides the standard AI model Mark, users now have access to Clair, a new deep-learning model tailored for female characters proficient in Asian languages. Clair's friendly complexion is well-suited to customer interaction.
Seamless CC – Audio2Face – iClone Integration
Two complimentary plugins enable an automated workflow. With just a single click, configure a CC character in A2F, animate it in real-time alongside an imported audio track, and seamlessly transfer the talking animation back to iClone for additional refinement before exporting it to 3D tools and game engines.
One-Click CC Character Setup in Audio2Face
The CC Character Auto Setup plugin for A2F, the result of a collaboration between NVIDIA and Reallusion, condenses the manual 18-step process into a single step. By importing a CC character and choosing a training model — Mike or Clair — artists can instantly witness lifelike talking animations synchronized with audio files. Experiment with motion sliders, automatic expressions, and even set keyframes. The finalized animations can then be sent to iClone for additional refinement.
Full-Spectrum Animation Refinement using iClone
The free A2F plugin for iClone is tailored to receive animation data from A2F. In addition to importing animations, it enhances the liveliness of facial features, resulting in a superior cut suitable for final production.
Facial Adjustment by Parts
Animations can be tweaked via a dynamic interface. Adjust various parameters such as expression strengths, head movements, or adding darting eyes to enliven the performance. Enlarge the jaw open range to enhance emotional tension and fine-tune the position of the tongue to mimic precise enunciation.
Smoothness Enhancement
Generative AI animation is susceptible to noise, particularly when audio files are captured by low-fidelity devices or within unfavorable environments. Reallusion A2F integration circumvents these limitations by deploying a highly refined noise filter to eliminate jitters and achieve optimal results despite poor audio quality.
Raising the Bar
After obtaining a satisfactory animation from A2F, a finishing touch becomes necessary, particularly when faced with emotional shifts or when emphasizing specific mouth shapes at varying levels of dialogue. iClone empowers facial editing, allowing for refined lip sync, the addition of natural expressions, and the incorporation of head movement sourced from mocap equipment.
The integration of Character Creator, iClone, and Audio2Face marks a significant milestone in AI-driven animation technology, offering creators unprecedented source audio flexibility and efficiency in their production workflows. The Character Creator Auto Setup plugin and iClone plugin are now available as free downloads from Reallusion, empowering creators to streamline their animation pipelines and unleash their creative potential.
To learn more about the iClone Audio2Face Plug-in, please visit:
>> Create natural talking animations from audio with iClone Audio2Face Plug-in
>> Download both iClone and Audio2Face PlugIns free from Reallusion Marketplace
>> Audio2Face Getting Started Tutorial
>> Audio2Face Installation Guide
Dan Sarto is Publisher and Editor-in-Chief of Animation World Network.
