Mouth animation using Rhubarb Lip Sync

DanielWolf

Hi! I just released a new version of Rhubarb Lip Sync. Until now, all discussion of Rhubarb has happened in the "Lip Sync" thread over at the Unity forum. As Rhubarb has nothing to do with Unity, I'm creating a new thread for Rhubarb here.

Rhubarb Lip Sync is a free and open-source tool that automatically generates mouth animations from audio files. You can use it with a number of software packages, including Spine. Here are some links:

The latest version 1.8.0 adds support for Ogg Vorbis files. Now Rhubarb supports both .wav and .ogg files.

I'll update this thread whenever there is news regarding Rhubarb for Spine.

Please let me know what you think. I'm always happy to get feedback!

DanielWolf

booboogaga

I will follow this 🙂

DanielWolf

Rhubarb Lip Sync 1.9 has been released. This version now comes with basic support for languages other than English. So if you're working with dialog in French, Italian, German, Spanish, or any other non-English language, be sure to give it a try!

There are also numerous fixes and improvements. Most notably, processing WAVE files is now considerably faster.

To learn more about the new multi-language support, check out this video:

Erika

I'm sorry for the late reaction, but OMG! We Italians are so lucky that our language is generally spoken rather clearly, the demos looked all amazing and the phonetic recognizer is definitely a game changer for us XD
Also wow! that dubber is quite famous in Italy, what project are they working on? I'm very curious!

I've been meaning to test this in Spine for a long time, I need to try! Thanks for the update! 😃

towfiqi

Daniel,

Great Work! from the video tutorial you shared, a one line takes around 4-5 seconds to sync. Can this be faster with a powerful machine?

Thanks

Harald

Amazing work indeed!

I really liked playing Thimbleweed Park - the voice acting and lip sync there were really spot an, and now I know why! 🙂

Keep up the great work!

anapaulacaruso

Hey,
love the premise of Rhubarb and absolutely loved the demo but I'm struggling to figure it out how to use it, being one of those people who can animate but has no idea how to code... :think:
Do you plan to make some tutorial for Rhubarb?

thanks

DanielWolf

Sorry to everybody for not answering! I somehow failed to subscribe to this thread... :wounded:

Thanks for the encouraging words! I'm really glad you like it. 🙂

Erikari escribió
Also wow! that dubber is quite famous in Italy, what project are they working on? I'm very curious!

Guga is part of a team of adventure gamers doing a full Italian fan-dub of Ron Gilbert's Thimbleweed Park. The game contains about 16,000 lines of dialog, so this is quite a large project, but I believe they're almost done. Check out their thread in the Thimbleweed Park forums (with several demo videos)!

towfiqi escribió
from the video tutorial you shared, a one line takes around 4-5 seconds to sync. Can this be faster with a powerful machine?

Sure, the faster the machine, the faster the processing. Also, if a recording is long enough, Rhubarb Lip Sync will process it with multiple threads simultaneously. Typically, processing speed is not an issue.

anapaulacaruso escribió
Do you plan to make some tutorial for Rhubarb?

I realize that the documentation is rather meager right now. I'm planning on doing a video tutorial, but it may take some time. In the meantime, this document should get you started. Let me know if you have any questions!

By the way, I just released Rhubarb Lip Sync 1.10. The main change for Spine users is that Rhubarb now supports the beta of Spine 3.8. Previous versions of Rhubarb only worked with Spine 3.7.

Nate

Thanks for staying on top of the latest support! We've been working on an example skeleton that shows off additive tracks for expressions and mouth shapes. We plan to hook it up with Rhubarb so it should also serve as a lip sync example.

DanielWolf

Nate escribió
Thanks for staying on top of the latest support! We've been working on an example skeleton that shows off additive tracks for expressions and mouth shapes. We plan to hook it up with Rhubarb so it should also serve as a lip sync example.

That sounds great! 🙂

Bekko

@[borrado]

hey we are going take a look at Rhubarb Lip Sync, but our work flow is a bit different, as we want to animate the spine via unity
any advice ?

tiny problem on the mac version :

For complete usage and help, type `rhubarb

help`

[Fatal] Application terminating with error: Invalid command line.
logout
Saving session...
...copying shared history...
...saving history...truncating history files...
...completed.

public enum LipSync { PBM, KST, EH_AE, AA, AO_ER, UW_OW_W, F_V, L, Idle }
 public LipSync lipSync;

track0.TimeScale = 0f;

if lipSync = PBM frame = 1 if lipSync = PBM frame = 2 etc...

DanielWolf

If you want to lip-sync your skeleton through Unity, you should probably use the command-line Rhubarb tool to generate TSV, XML, or JSON files. Then you need to write some code in Unity that parses these files and switches your skeleton's attachments accordingly. You'll have to code that yourself, but it shouldn't be hard.

Kombowz

If possible, please keep support for Spine 3.7 as this is the only version of spine that works for Game Maker at the moment. Also, please don't forget about that spine tutorial on how to get it all to work properly that we talked about. Like what to export, import, and where all the files should be etc. I've been stalking your YouTube waiting for it. LOL. Rhubarb Lip Sync is awesome sauce.

DanielWolf

Rhubarb supports both Spine 3.7 and 3.8. I have no plans to change that. But it's good to know that there are people reliant upon version 3.7.

Regarding the tutorial: I've started working on it. But beside my job and my family, I'm afraid my time is rather limited, so it'll still be some days.

Kombowz

DanielWolf escribió
Rhubarb supports both Spine 3.7 and 3.8. I have no plans to change that. But it's good to know that there are people reliant upon version 3.7.

Regarding the tutorial: I've started working on it. But beside my job and my family, I'm afraid my time is rather limited, so it'll still be some days.

I'm happy to know that Spine 3.7 will stay supported. As for the tutorial, thank you for starting it, as for finishing it, Your main job and family should always come first.. I've subscribed with notifications for when you upload it and it's available.

I appreciate everything you do, Thanks again for Rhubarb!

DanielWolf

It took me a while, but I finished the video tutorial I promised! It gives step-by-step instructions on how to download Rhubarb Lip Sync, setup your skeleton in spine, animate your recordings, and improve the result.

Let me know what you think!

Harald

Thanks very much for creating this great in-depth tutorial video, I'm sure it will help a lot of users! Now I even learned something about the history of mouth shapes! :talk: :nerd:

SapientX

Hi Daniel. Do you have any intention of making a real-time version of Rhubarb... perhaps as a Unity asset?

DanielWolf

I take it that by real-time, you mean that a player speaks into a microphone and sees an animated mouth without noticeable delay. This is not something I'm planning to do in the foreseeable future. Right now, I'm focusing on improving the existing feature set for pre-recorded audio.

I just released Rhubarb Lip Sync 1.12. Until now, Rhubarb didn't support skinned skeletons. The new version now fully supports skinning!

Página siguiente »