Some questions about your paper

[Pixie8888]

Dear authors,

I have some questions about your paper:

1. How to estimate eyes-open condition c_{s,eyes} and lip-open condition c_{s,lip}?
2. Why the driving keypoint calculation in eqn 7 is different from eqn 2?

[zzzweakman]

Dear authors,

I have some questions about your paper:

1. How to estimate eyes-open condition c_{s,eyes} and lip-open condition c_{s,lip}?
2. Why the driving keypoint calculation in eqn 7 is different from eqn 2?

Hi @Pixie8888,

Regarding your first question, after training the generative network in the first stage, we can generate a facial image in the second stage given the eyes-open condition c_{s,eyes}. We then use a pre-trained 2D explicit facial keypoint model to extract the distance between the upper and lower eye points, aligning this distance with c_{s,eyes}. The process for the lips is similar.

For your second question, equation 7 includes logic for estimating relative motion, which you can think of as a redirection operation.

[Pixie8888]

Thanks for your reply!
But I am confused by we can generate a facial image in the second stage given the eyes-open condition c_{s,eyes}. The generative network in stage 1 doesn't receive the eye-open condition. And how can you get c_{s, eyes} ?

[zzzweakman]

Thanks for your reply! But I am confused by we can generate a facial image in the second stage given the eyes-open condition c_{s,eyes}. The generative network in stage 1 doesn't receive the eye-open condition. And how can you get c_{s, eyes} ?

This condition is the input to this MLP. This condition is the input to the MLP network, as shown in Fig. 3 of our paper. c_{s, eyes} is estimated from the driving image I_s by explicit facial 2D keypoints. The output of this network is added as an increment to the main network's output, and all of this happens before the generator.

[Pixie8888]

Thanks!