Face Animation Synchronised with the speech

Question

I am developing a Interactive Spoken Dialogue System, in which user will say something to the system and system will answer him in voice and generating facial expressions according to the voice. But, i can't get any idea that how can i generate facial expressions. Will anybody tell me that how can i start or if there is any open source package already exists then it would be very nice. If anybody know about any open source software related to this then please tell me.

Answer 1

This project along with other systems, contains what you need:

http://osdn.jp/projects/galatea

Quote:

"Galatea" is a project for providing an open-source, license-free software toolkit for building anthropomorphic spoken dialogue agents. In other words, using this toolkit, you can build your own unique life-like visual agent that can communicate with you via spoken language

Answer 2

You can use open source HapFacs1.0 API which provides comprehensive, FACS based API to generate different facial expressions based on action units. HapFacs API is currently available on windows platforms. HapFacs developed based on the Haptek software. It also provides built-in lip synchronization functionality and compatible with most of the text-to-speech engines. HapFacs1.0 is originally written in C#, but you can use API, by referencing its .dll file with most of the development platforms. By combining different action units, you can generate facial expressions in different intensities. By using EMFACS configurations, you can generate 7 universal facial expression with different intensities.

Answer 3

You can generate facial expressions by making a video of your face? Personally, I think I would start with that.

Answer 4

Do you really need facial expressions, or is a mouth opening and closing like you would see in an old (non-3d) animated cartoon enough? If so, you could try matching the mouth to the speech amplitude envelope using an envelope detector.