Emotion prediction accuracy

[nicoeiris11]

Hi Vlad!

I'm using human in the browser and currently experiencing a very bad accuracy for emotion detection.
Using blazeface default model and the distance to the webcam is always around 2-3 meters

I'd like to know if there's an alternative model for emotion detection supported by human taking into account Oarriaga Emotion model has %66 test accuracy. In your model's wiki page you mention GEAR as an alternative. Is it possible to experiment with the GEAR emotion model instead of Oarriaga's for emotion? Looking at GEAR repo I can see the emotion classifier is independent of the GenderAgeRace model but not sure if the emotion model is supported by human or not.

Some additional questions:

• High resolution should help, right?
• Is human doing all necessary preproc of the cropped face before predicting emotion? (grayscale, normalization, etc.)
• Is it possible to visualize the input of the emotion model for debugging purposes? (the 64x64 image sent to the model)

Thank you!

[vladmandic]

yes, GEAR model is supported for emotion. i can provide some config examples later.

High resolution should help, right?

high resolution helps up to a point where cropped face image is still of sufficient resolution, basically 256x256, otherwise lighting and clarity means much more.

Is human doing all necessary preproc of the cropped face before predicting emotion? (grayscale, normalization, etc.)

human does all necessary preprocessing (crop, resize, grayscale, normalize), but...i never had enough time to optimize cropping factor for each model and it matters a lot. e.g., should image for GEAR have a tight face crop or slightly enlarged? its all configurable, but values are tuned only for some models and emotion is not one of them.

Is it possible to visualize the input of the emotion model for debugging purposes? (the 64x64 image sent to the model)

its only supported for initial face detection, not for subsequent per-model specific processing. but even first step is sufficient to visualize cropping.

[nicoeiris11]

GEAR model is supported for emotion. I can provide some config examples later.

Thank you for this. I tried changing my emotion config to use GEAR detection but didn't work.

cropped face image is still of sufficient resolution, basically 256x256

Cropped face resolution is configurable? I'd like to know the criteria used for the face resolution, in my demo for all video resolutions faces resolution is 192x192 (human.result.face[0].tensor.shape).

optimize cropping factor for each model and it matters a lot

cropping factor and any other face detection post proc are common to all subsequent face models? Is this something I could optimize for emotion for example?

[vladmandic]

I had GEAR emotion support baked in, but I never published actual models. Its done now and human library is updated.
GEAR emotion actually has two variations depending on dataset it was trained on, both are published.

And wrote a short test that uses different emotion detection variations:
https://github.com/vladmandic/human/blob/main/test/test-node-emotion.js

Cropped face resolution is configurable? I'd like to know the criteria used for the face resolution, in my demo for all video resolutions faces resolution is 192x192 (human.result.face[0].tensor.shape).

Face detection is 256x256, but subsequent Face mesh is 192x192.
If mesh is enabled, it doesn't just perform mesh detection, it also stabilizes face box and does re-cropping - so result is slightly lower resolution, but higher precision.

If you don't need mesh results, you can run with mesh enabled or disabled and compare results.

cropping factor and any other face detection post proc are common to all subsequent face models? Is this something I could optimize for emotion for example?

Cropping happens during face detection. All subsequent modules do additional processing, but cannot expand face if it is already cropped :)

Demo I've provided shows how to specify crop scale factor in config and you can experiment with different values - you'll see how massive on an impact it has.

[nicoeiris11]

Thank you so much for the detailed answer!

Is it in the human roadmap to include other emotion alternatives trained with other datasets than fer2013?.

For example, there's a Face Description: HSE-FaceRes sibling repo for face emotion detection trained with VGGFace2 that it's worth it experimenting with. If it's not in human roadmap, how complex would be for me to integrate this model into human for experimentation? (custom module guideline)

[vladmandic]

its not on a roadmap as emotion in general was not my highest priority. but honestly, if model is "sane", it should be quite easy to integrate it. i'll take a look next week (and i do need to do a slight update to custom module guideline wiki, but nothing major).

i'm going to convert this discussion to a feature request #335 so its easier to track.

[nicoeiris11]

Thank you so much for considering this in your priorities!! 🙌

[vladmandic]

issue #335 is updated