Loading...
> >

LIp synch in an app

Mosk the ScribeMosk the Scribe Posts: 888
edited December 1969 in Carrara Discussion

I'm going to be helping with an app and wanted to know if there's a way to do live lip-synching within the app, or if we would just need to record audio/video clips ahead of time that have the lip synch done (in Carrara, crazy talk, mimic pro or whatnot)

Thanks for any suggestions.

Comments

  • evilproducerevilproducer Posts: 9,050
    edited December 1969

    I'm no programmer, but I would suppose that if you can MOCAP body movements, such as with Kinect, you may be able to use the same principle by aiming a device at your face and at the same time record the dialogue. I suspect it would be best to have the app sync the two together in real-time as the dialogue and movement are captured. The app would then have to generate some kind of file with the movement data. I would almost suspect this would have to be outside the 3D software to be imported later, as I'm sure capturing motion capture and audio, syncing them and applying the data to a 3D figure in realtime would be really taxing on a processor.

    The other option would perhaps be to forego motion capture, but rather have the app analyze the speech and use some kind of an algorithm to break out the mouth movements.

    Both sound as if a realtime solution, where the speech is spoken, and the mouth is moved/morphed at the same time, would be really difficult within another app.

    It sounds as if you guys know more than me if you're talking about building an app, and so would have a better idea of what can be done or not done.

    As for me, I'm just spit-balling here in blissful ignorance contemplating my forum signature line. ;-)

  • Mosk the ScribeMosk the Scribe Posts: 888
    edited December 1969

    Hey Evil -
    I don't have any app programming experience - have learned some basic coding for my own purposes, and will learn more for this, but I'll just be helping out. I was just trying to find out from anyone with experience in this; my guess is that it would be best to make all of the audio/video clips ahead of time and just load as needed.

    Thought I'd ask, though, since Mimic Live gives you this ability. Didn't know if there was similar software available for apps, and if it worked well enough to just take a wav file, figure out the phonemes and go from there (which would open up possibility of multi-lingual app)

    Thanks

  • evilproducerevilproducer Posts: 9,050
    edited December 1969

    Mosk the Scribe said:
    Hey Evil -
    I don't have any app programming experience - have learned some basic coding for my own purposes, and will learn more for this, but I'll just be helping out. I was just trying to find out from anyone with experience in this; my guess is that it would be best to make all of the audio/video clips ahead of time and just load as needed.

    Thought I'd ask, though, since Mimic Live gives you this ability. Didn't know if there was similar software available for apps, and if it worked well enough to just take a wav file, figure out the phonemes and go from there (which would open up possibility of multi-lingual app)

    Thanks

    Yeah. No worries. I just was just speaking in completely theoretical terms. I haven't even used Mimic, although I have it.

  • WendyLuvsCatzWendyLuvsCatz Posts: 38,040
    edited December 1969

    facerig does it
    $10 on steam but I cannot figure out how to get my own rig in, have tried

Sign In or Register to comment.