Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

The AU intensities generated by OpenFace is inconsistent with the AU intensities manually marked #1082

Open
Au-Chak-Hon opened this issue Mar 21, 2024 · 0 comments

Comments

@Au-Chak-Hon
Copy link

Hi,

I used OpenFace for AU intensity estimation and I found that the AU intensities generated by OpenFace is very different from the manually marked AU intensities.

The dataset used is DISFA and the subject used is SN001.

The command used to run the AU intensity estimation is:
"./../../OpenFace_2.2.0_win_x64\FeatureExtraction.exe" -aus -f "./../../DISFA/SN001/video\LeftVideoSN001_comp.avi" -out_dir "./../../DISFA/SN001/csv_vid"
and
"./../../OpenFace_2.2.0_win_x64\FeatureExtraction.exe" -aus -f "./../../DISFA/SN001/video\RightVideoSN001_comp.avi" -out_dir "./../../DISFA/SN001/csv_vid"

Then, the distribution looks like the following:
Using LeftVideo:
image
Using RightVideo:
image
Python Code for showing the distribution:

import pandas as pd
import os
import numpy as np

data_root_path = './../../DISFA/SN001'
csv_path = data_root_path + '/csv_vid'
openface_path = './../../OpenFace_2.2.0_win_x64'

labels = np.array([1,2,4,5,6,9,12,15,17,20,25,26])
AUs1 = np.zeros([12,6])
AUs2 = np.zeros([12,6])
AUs3 = np.zeros([12,6])
AUs_real = np.zeros([12,6])

if True:  
  csv_list = pd.read_csv(os.path.join(csv_path,('RightVideoSN001_comp.csv')),sep=', ')
  for j in range(len(csv_list)):
    frame = csv_list.loc[j,['AU01_r','AU02_r','AU04_r','AU05_r','AU06_r','AU09_r','AU12_r','AU15_r','AU17_r','AU20_r','AU25_r','AU26_r']]
    for i in range(len(frame)):
      _value1 = math.floor(frame[i])
      _value2 = math.ceil(frame[i])
      _value3 = round(frame[i])
      AUs1[i][_value1] += 1
      AUs2[i][_value2] += 1
      AUs3[i][_value3] += 1
  
  idx = 0
  for list in sorted(os.listdir(os.path.join(data_root_path,'AU'))):
    _l = pd.read_csv(os.path.join(data_root_path,'AU',list),sep=',',header=None)
    for j in range(len(_l)):
      _f = _l.loc[j,1]
      _value = int(_f)
      AUs_real[idx][_value] += 1
    idx += 1

  print('Estimated Distribution (floor):')
  print('  AU  |  X   |  A   |  B   |  C   |  D   |  E   |   >0  ')
  print('------+------+------+------+------+------+------+-------')
  for x in range(12):
    _AUTotal = 0
    for y in range(1,6):
      _AUTotal += AUs1[x][y]
    print("{:^6d}".format(labels[x]) + '|' + "{:^6.0f}".format(AUs1[x][0]) + '|' + "{:^6.0f}".format(AUs1[x][1]) + '|' + "{:^6.0f}".format(AUs1[x][2]) + '|'
          + "{:^6.0f}".format(AUs1[x][3]) + '|' + "{:^6.0f}".format(AUs1[x][4]) + '|' + "{:^6.0f}".format(AUs1[x][5]) + '|' + "{:^6.0f}".format(_AUTotal))

  print('Estimated Distribution (ceil):')
  print('  AU  |  X   |  A   |  B   |  C   |  D   |  E   |   >0  ')
  print('------+------+------+------+------+------+------+-------')
  for x in range(12):
    _AUTotal = 0
    for y in range(1,6):
      _AUTotal += AUs2[x][y]
    print("{:^6d}".format(labels[x]) + '|' + "{:^6.0f}".format(AUs2[x][0]) + '|' + "{:^6.0f}".format(AUs2[x][1]) + '|' + "{:^6.0f}".format(AUs2[x][2]) + '|'
          + "{:^6.0f}".format(AUs2[x][3]) + '|' + "{:^6.0f}".format(AUs2[x][4]) + '|' + "{:^6.0f}".format(AUs2[x][5]) + '|' + "{:^6.0f}".format(_AUTotal)) 
  
  print('Estimated Distribution (round):')
  print('  AU  |  X   |  A   |  B   |  C   |  D   |  E   |   >0  ')
  print('------+------+------+------+------+------+------+-------')
  for x in range(12):
    _AUTotal = 0
    for y in range(1,6):
      _AUTotal += AUs3[x][y]
    print("{:^6d}".format(labels[x]) + '|' + "{:^6.0f}".format(AUs3[x][0]) + '|' + "{:^6.0f}".format(AUs3[x][1]) + '|' + "{:^6.0f}".format(AUs3[x][2]) + '|'
          + "{:^6.0f}".format(AUs3[x][3]) + '|' + "{:^6.0f}".format(AUs3[x][4]) + '|' + "{:^6.0f}".format(AUs3[x][5]) + '|' + "{:^6.0f}".format(_AUTotal))


  print()
  print('Real Distribution:')
  print('  AU  |  X   |  A   |  B   |  C   |  D   |  E   |   >0  ')
  print('------+------+------+------+------+------+------+-------')
  for x in range(12):
    _AUTotal = 0
    for y in range(1,6):
      _AUTotal += AUs_real[x][y]
    print("{:^6d}".format(labels[x]) + '|' + "{:^6.0f}".format(AUs_real[x][0]) + '|' + "{:^6.0f}".format(AUs_real[x][1]) + '|' + "{:^6.0f}".format(AUs_real[x][2]) + '|'
          + "{:^6.0f}".format(AUs_real[x][3]) + '|' + "{:^6.0f}".format(AUs_real[x][4]) + '|' + "{:^6.0f}".format(AUs_real[x][5]) + '|' + "{:^6.0f}".format(_AUTotal))

Also, it is found that the positions of the AU intensities does not match.
Take AU1 as an example, the detected frame position for intensity 1 is approximately at frame 3000 to 3200, but the manually marked frame position for intensity 1 is approximately at frame 1600

Frames detected with intensity 1 in the left vieeo:
image
Frames detected with intensity 1 in the right vieeo:
image
The actual frames with intensity 1:
image

I wonder if I did something wrong, or did I misunderstood something. Please help. Thank you.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant