Skip to content

Latest commit

 

History

History

OpenGraphAU

README.md

OpenGrpahAU

This repo is the OpenGprahAU tool.

demo:

Models were traiend on hybrid dataset of 2,000k images.

This hybrid dataset includes:

The tool can predict action units of 41 categories:

AU1 AU2 AU4 AU5 AU6 AU7 AU9 AU10 AU11 AU12 AU13 AU14 AU15 AU16
Inner brow raiser Outer brow raiser Brow lowerer Upper lid raiser Cheek raiser Lid tightener Nose wrinkler Upper lip raiser Nasolabial deepener Lip corner puller Sharp lip puller Dimpler Lip corner depressor Lower lip depressor
AU17 AU18 AU19 AU20 AU22 AU23 AU24 AU25 AU26 AU27 AU32 AU38 AU39 -
Chin raiser Lip pucker Tongue show Lip stretcher Lip funneler Lip tightener Lip pressor Lips part Jaw drop Mouth stretch Lip bite Nostril dilator Nostril compressor -
AUL1 AUL1 AUL2 AUR2 AUL4 AUR4 AUL6 AUR6 AUL10 AUR10 AUL12 AUR12 AUL14 AUR14
Left inner brow raiser Right inner brow raiser Left outer brow raiser Right outer brow raiser Left brow lowerer Right brow lowerer Left cheek raiser Right cheek raiser Left upper lip raiser Right upper lip raiser Left nasolabial deepener Right nasolabial deepener Left dimpler Right dimpler

We provide tools for prepareing data in tool/. After Downloading raw data files, you can use these tools to process them, aligning with our protocals. We divide the dataset into three independent parts (i.e., train, val, test).

Pretrained models

Hybrid Dataset

Stage1:

arch_type GoogleDrive link Average F1-score Average Acc.
Ours (MobileNetV3) - - -
Ours (ResNet-18) link 22.33 92.97
Ours (ResNet-50) link 22.52 92.63
Ours (Swin-Tiny) link 22.66 92.97
Ours (Swin-Small) link 24.49 92.84
Ours (Swin-Base) link 23.53 92.91
Ours (MAE-VIT) link 24.05 93.47

Stage2:

arch_type GoogleDrive link Average F1-score Average Acc.
Ours (MobileNetV3) - - -
Ours (ResNet-18) link 22.51 93.23
Ours (ResNet-50) link 23.24 93.31
Ours (Swin-Tiny) link 22.74 93.37
Ours (Swin-Small) link 22.99 93.37
Ours (Swin-Base) link 24.14 93.25
Ours (MAE-VIT) link 25.29 93.76

Demo

  • to detect facial action units in a facial image using our stage1 model, run:
python demo.py --arc resnet50 --stage 1 --exp-name demo --resume checkpoints/OpenGprahAU-ResNet50_first_stage.pth --input demo_imgs/1014.jpg  --draw_text
  • to detect facial action units in a facial image using our stage2 model, run:
python demo.py --arc resnet50 --stage 2 --exp-name demo --resume checkpoints/OpenGprahAU-ResNet50_second_stage.pth --input demo_imgs/1014.jpg  --draw_text

Training and Testing

  • to train the first stage of our approach (ResNet-50) on hybrid Dataset, run:
python train_stage1.py --arc resnet50 --exp-name OpenGprahAU-ResNet50_first_stage -b 512 -lr 0.00002
  • to test the first stage of our approach (SwinT) on hybrid Dataset, run:
python test_stage1.py --arc swin_transformer_tiny --exp-name test_OpenGprahAU-SwinT_first_stage  --resume ./results/OpenGprahAU-SwinT_first_stage/bs_64_seed_0_lr_2e-05/best_model.pth
  • to train the second stage of our approach (ResNet-50) on hybrid Dataset, run:
python train_stage2.py --arc resnet50 --exp-name OpenGprahAU-ResNet50_second_stage -b 512 -lr 0.00001  --resume checkpoints/OpenGprahAU-ResNet50_first_stage.pth
  • to test the second stage of our approach (SwinT) on hybrid Dataset, run:
python test_stage2.py --arc swin_transformer_tiny --exp-name test_OpenGprahAU-SwinT_second_stage  --resume ./results/OpenGprahAU-SwinT_second_stage/bs_64_seed_0_lr_1e-05/best_model.pth