Animation and lip syncing advice, please

[nathanj]

Hello Everyone,

The project I'm working on is starting into a new phase of development: triggering animations (body and lip syncing) and audio. So far we've been triggering text and managing quests beautifully with DS alone. However, I'm starting to wonder if there are other programs (for the management and implementation of conversations and interactions) we could be using in conjunction with DS that would make more complex interactions more manageable.

I guess my question is this: do you find that other 3rd party assets help with the construction and implementation of dynamic interactions, or does sticking with, and maximizing the uses of DS, offer enough? I am continually amazing with how robust DS is, and now that I feel at home with it I feel like I could easily keep on this path. But, I also wonder if I'm missing some obvious tools.

I would be most appreciative of anyones feedback or suggestions.

Nathan

[Tony Li]

I may be a bit biased, but I think the Dialogue System alone does a great job of managing conversations. It doesn't do lipsync, though. Instead, it integrates with the popular lipsync offerings on the Asset Store, such as SALSA, LipSync Pro, Cheshire, etc., as well as FaceFX (which is what the AAA studios use). The Dialogue System manual has a section on How to Add Lipsync.

The easiest to set up is SALSA with RandomEyes because you don't have to preprocess every audio clip. However, if you don't mind preprocessing, LipSync Pro is also quite easy to use once you get your workflow down.

When you start adding an audio clip for each line of dialogue, you quickly accumulate a lot of files. Using entrytags is a good way to simplify the setup and enforce a consistent naming/organization scene.

For body animation, you can often just use animation sequencer commands. If you need to design more complex sequences that require timing among several actors and animations, you may want to also mix in a cutscene editor product such as SLATE, Cinema Director, or uSequencer. You can mix and match, using Dialogue System sequencer commands for simple lines and the cutscene editor for more complex lines.

[bohnstudios]

Quick note on workflow for lip sync tools and entry tags. I've found that it's a good idea to have your text conversations 100% in place before you record your audio files. This way you will know prior to recording audio what the file names should be...basically each .wav or .ogg file name has to correspond to the correct node in the correct conversation. If this process is done backwards it's like putting together a puzzle and will take 50x or longer to complete the same amount of work. Be sure to watch out for that, but overall the workflow is very solid for complex conversations and for having many conversations in your project.

I am currently using Salsa. As Tony mentioned, FaceFx is the best...it's also got a heavy price tag that goes along with it...$700? I haven't used FaceFx personally, but I'm seriously considering it for my next title just because it is the "it" tool for lip sync.

I have used Dialogue System + Salsa out-of-the-box integration with Mixamo (currently free) and Daz3d characters ($500-2,500 commercial development license + assets). Daz was also used in the Iron Man 3 movie so that is pretty good validation imo. Just saying the super high-end options are available with Dialogue System too.

For sure with Salsa, make sure your audio files are loud enough, but also make sure your audio compression doesn't squish the audio. I know with Salsa, it basically goes off how many decibels (ie how loud) a particular spoken/recorded syllable is. While Salsa doesn't require pre-processing the audio files, it does get the best results if your raw recording file is at least 75-80% to the peak on your master gain. It still works though if you don't do this. You also have to tweek the settings in Salsa depending on how your your character rig is set-up, but this can take as little as 5 minutes and then you can use those settings as a template for your other characters that are rigged/imported into unity the same way.

Of course, both Mixamo and Daz3d characters are rigged so they should work with any mecanim animation you put on your character animation controller. These animations, such as changing the idle animation or calling a root motion animation, can be called from any dialogue system conversation node via the sequencer command line with in Dialogue System. Also, I believe it helps to bake the animation into the pose for best results.

As for managing conversations, I personally love Dialogue System. It works on industry conventions that are also used with Chat Mapper and Articy. One of my favorite things about Dialogue System is that it has so many 3rd party integrations and sequencer commands, among other things that are make it really powerful yet simple....as in the the good kind

[Tony Li]

Thanks for the great tips, @bohnstudios! (When I bought FaceFX several years ago, it was $1500! )

I forgot to mention that you can export a Voiceover Script in the Dialogue Editor window. This is a simplified version of the CSV export that's useful for handing off to your voice actors.

[bohnstudios]

You're welcome, Tony! I made a tough call this go-around and spent the money on daz vs FaceFx. Those two combined are what to use to get the upper-shelf AAA visuals, combined with maybe a good skin shader you can do some really heavy lifting (some shaders on the asset store allow you to create sweat and such if anyone wants to go for that kind of detail).

Also, for anyone that's new to this workflow, make sure you enable facial blendshapes on your character models during your rigging workflow. I'm pretty sure this applies no matter what you are using.

I forgot to mention that you can export a Voiceover Script in the Dialogue Editor window. This is a simplified version of the CSV export that's useful for handing off to your voice actors.

Yep, that's awesome!

[nathanj]

Thanks, Tony and bohnstudios!

This is so much useful information, so many useful tips. I feel like I've jumped ahead in development time by weeks. I am happy to hear that others rely on DS for sure heavy lifting on large projects.

I'm slowly processing this but I'm sure I'll be back with some follow up questions, or hopefully some helpful tips of my own.

Nathan