Loading...
> >

[WIP] Carractor: new plugin for human based animation of digital character

Philemo_CarraraPhilemo_Carrara Posts: 1,175
edited February 2020 in Carrara Discussion

Lately, I came across this open source project: OpenFace V2

As far as I can tell, this is by far the most advanced face recognition and analysis package available for free. This is a project of the CMU Multicomp Lab, one of the University leaders in the field (remember the CMU Mocap that were the hype a few years ago ? wink).

This project takes a picture or a video as input, processes it thoughtfully and output a file containing head pose estimation relative to the camera, gaze direction, facial landmarks in  3D coordinates and action units evaluation.

So, as the result of the analysis was easily readable and that the product was somehow designed to be interfaced, I decided to create a new plugin for Carrara.

The idea was to use all those results to animate characters using image or video without markers.

Post edited by Philemo_Carrara on
«1

Comments

  • Philemo_CarraraPhilemo_Carrara Posts: 1,175
    edited February 2020

    Technical issues

    The first issue I encountered was that the parameter Tab for figures was not reacheable through the SDK. The SDK was released with C8.1 and the parameter Tab was redone in C8.5 with the introduction of Genesis.

    The only way to overcome that was to create a DUF file and import it back. An unexpected benefit was that the DUF file generated can be imported in Studio.

    Post edited by Philemo_Carrara on
  • WendyLuvsCatzWendyLuvsCatz Posts: 38,199

    looking forward to pulling faces in front of my webcam with a much younger and far prettier mesh than mine heart

  • Philemo_CarraraPhilemo_Carrara Posts: 1,175
    edited February 2020

    Some examples

    Patience.jpg
    1280 x 720 - 134K
    Patience_Caractor.jpg
    960 x 960 - 603K
    die-hard-2-bruce-willis-1.jpg
    1024 x 657 - 426K
    die hard Carrara.jpg
    960 x 960 - 608K
    527210.jpg-r_1920_1080-f_jpg-q_x-xxyxx.jpg
    1200 x 720 - 202K
    527210 Carrara.jpg
    960 x 960 - 609K
    die-hard-with-a-vengeance-bruce-willis.jpg
    1280 x 720 - 158K
    die hard 2 Carrara.jpg
    960 x 960 - 606K
    Post edited by Philemo_Carrara on
  • DiomedeDiomede Posts: 15,165
    Philemo_Carrara said:

    Technical issues

    The first issue I encountered was that the parameter Tab for figures was not reacheable through the SDK. The SDK was released with C8.1 and the parameter Tab was redone in C8.5 with the introduction of Genesis.

    The only way to overcome that was to create a DUF file and import it back. An unexpected benefit was that the DUF file generated can be imported in Studio.

    As per PhilW, I think you should cash in with Daz3D when you can get the Studio version working the way you want.  His comment was in a different thread, I think.  Very impressive.  

  • Philemo_CarraraPhilemo_Carrara Posts: 1,175
    edited February 2020

     

    Diomede said:
    Philemo_Carrara said:

    Technical issues

    The first issue I encountered was that the parameter Tab for figures was not reacheable through the SDK. The SDK was released with C8.1 and the parameter Tab was redone in C8.5 with the introduction of Genesis.

    The only way to overcome that was to create a DUF file and import it back. An unexpected benefit was that the DUF file generated can be imported in Studio.

    As per PhilW, I think you should cash in with Daz3D when you can get the Studio version working the way you want.  His comment was in a different thread, I think.  Very impressive.  

    I cannot do that. Here is an extract of the licence agreement:

    Important points from the license:

    • PERMITTED USES:  The Software may be used for your own noncommercial internal research purposes. You understand and agree that Licensor is not obligated to implement any suggestions and/or feedback you might provide regarding the Software, but to the extent Licensor does so, you are not entitled to any compensation related thereto.

    It means I won't be able to bundle OpenFace with the plugin. You'll have to download it yourself and unzip it at a particular place.It will be your responsability to use it according to the licence.

     

    • The non-exclusive commercial license requires a non- refundable annual royalty (USD $10,000 for OpenFace Light, USD $15,000 for OpenFace Landmark, USD $18,000 for OpenFace Full Suite: see each offering for details.
    • The license is non-negotiable.

    I'm not sure I can sell a commercial product interfacing OpenFace without buying a commercial licence. Living in Europe, I'd rather not try to proove my point in a American court.

    Post edited by Philemo_Carrara on
  • WendyLuvsCatzWendyLuvsCatz Posts: 38,199

    wow yeah that's a bit rich blush

    fully understand and thanks for working out a way for us to use this heart

  • 3drendero3drendero Posts: 2,024
    Looks cool. There is a similar but worse DS plugin in the Daz3d store, don't remember the name. Thanks for working on a new Carrara plugin.
  • DartanbeckDartanbeck Posts: 21,543

    This sounds Really cool!!!

    So if I'm reading this correctly, I should be able to video record Rosie while she's doing her lines to pluck facial recognition from for use in her 3d persona. 

    ...and I don't have to but a markersuit setup or iClone 3d Xchange Pipeline... I'm in!!! 

    Hey Philemo, I Love You, Man!!! 

  • mindsongmindsong Posts: 1,701

    we can always donate a bit with this kind of open-source model, right? smiley - you know, beer money of sorts...

    cheers, and thanks for taking this on - it looks *very* cool!

    --ms

  • WendyLuvsCatzWendyLuvsCatz Posts: 38,199
    3drendero said:
    Looks cool. There is a similar but worse DS plugin in the Daz3d store, don't remember the name. Thanks for working on a new Carrara plugin.

    Anihead, it only moves the head

  • HeadwaxHeadwax Posts: 9,987

    wow Philemo, just wow - a genius you are

  • MistaraMistara Posts: 38,675

    it can pose a character from a video?

    is this for face rigged characters?

    this would be good too for speaking characters?  instead of mimic 

  • Philemo_CarraraPhilemo_Carrara Posts: 1,175
    Mystarra said:

    it can pose a character from a video?

    is this for face rigged characters?

    this would be good too for speaking characters?  instead of mimic 

    Yes, yes and yes laugh

    I'll try to post a video soon

  • MistaraMistara Posts: 38,675

    the implications are staggering!

  • Philemo_CarraraPhilemo_Carrara Posts: 1,175

  • MistaraMistara Posts: 38,675
    Philemo_Carrara said:

     

    omg looks so much better than mimic

  • HeadwaxHeadwax Posts: 9,987

    that's a wow again Philemo!

  • UnifiedBrainUnifiedBrain Posts: 3,588
    Headwax said:

    that's a wow again Philemo!

    +Wow!

  • StezzaStezza Posts: 8,050

    +wow-wee!

  • Philemo_CarraraPhilemo_Carrara Posts: 1,175
    edited February 2020

    Second test. Improving, but still not good enough.

    Post edited by Philemo_Carrara on
  • MistaraMistara Posts: 38,675

    it's impressive.  hugh's face does a lot of motion.

    i'd be curious how wolverine would translate. 

  • Persona Non GrataPersona Non Grata Posts: 1,365

    Philippe, je t'adore !

    Seriously, this is going to be such a wonderful addition to Carrara. I can't wait to have a play. Will we be able to specify a set of morphs on our own models to animate. Can we, please, please, please wink

  • Philemo_CarraraPhilemo_Carrara Posts: 1,175
    FENgari said:

    Philippe, je t'adore !

    Seriously, this is going to be such a wonderful addition to Carrara. I can't wait to have a play. Will we be able to specify a set of morphs on our own models to animate. Can we, please, please, please wink

    Yes. Same principle as Mimic. A folder with configuration files.

  • MistaraMistara Posts: 38,675

    i dreaming for this.  excited for a lively aiko 3.heart aiko heart

  • WendyLuvsCatzWendyLuvsCatz Posts: 38,199

    so glad it doesn't need an iPhone too

    I just saw this on youtube 

     

  • REIVAXREIVAX Posts: 70
    edited March 2020

    hello

    french software.not very cheap

    http://www.dynamixyz.com/products/

    Facial motion capture test. Dynamixyz+Daz Studio (Iray)

    Post edited by REIVAX on
  • WendyLuvsCatzWendyLuvsCatz Posts: 38,199

    heart I hope all stll going well progress wise Philemo

    I was thinking with all this Covid19 stuff people are going to have to communicate virtually so facial captures will come in handy as most want an attractive alternative to their real faces on video too.

  • 0oseven0oseven Posts: 626

    I real boon for animators - thank you Phil !!! 

    I'm understandiing your  demo videos made by running a video on the screen ? Any cameras involved ?

    Are animations captured as keyframes with  access to the Carrara editing tools as usual ?  Save as clips etc. ? 

     

     

  • MistaraMistara Posts: 38,675

    any news on this?

     

  • DartanbeckDartanbeck Posts: 21,543

      

Sign In or Register to comment.