Added docs for eye states

neon/general/data-format/index.md

@@ -78,7 +78,7 @@ This file contains [gaze](/neon/basic-concepts/data-streams/#gaze) data in world
 | -------- | -------- | 
 | **section id** | Unique identifier of the corresponding section.     |
 | **recording id** | Unique identifier of the recording this sample belongs to.     |
-| **timestamp [ns]** | UTC timestamp in nanoseconds of the sample. Equal to the timestamp of the original gaze sample before mapping.     |
+| **timestamp [ns]** | UTC timestamp in nanoseconds of the sample. Equal to the timestamp of the eye video frame this sample was generated with.     |
 | **gaze x [px]** | Float value representing the x-coordinate of the mapped gaze point in world camera pixel coordinates.
 | **gaze y [px]** | Same as "gaze x [px]" but for the y-coordinate.     |
 | **worn** | These values indicate whether Neon has been worn by a subject at the respective point in time. `1.0` indicates that it has been worn, while `0.0` indicates that it has not been worn.    |
@@ -106,6 +106,19 @@ The corresponding gaze samples that belong to each fixation can be determined fr
 | **azimuth [deg]** | The [azimuth](https://en.wikipedia.org/wiki/Horizontal_coordinate_system) of the gaze ray corresponding to the fixation location in relation to the scene camera in degrees.     |
 | **elevation [deg]** | The [elevation](https://en.wikipedia.org/wiki/Horizontal_coordinate_system) of the gaze ray corresponding to the fixation location in relation to the scene camera in degrees.     |
 
+## 3d_eye_states.csv.csv
+This file contains [3D eye states](/general/data-streams/#3d-eye-states) as well as [pupil diameter](/general/data-streams/#pupil-diamaters) data.
+
+
+| Field                     | Description | 
+| ------------------------- | -------- | 
+| **section id**            | Unique identifier of the corresponding section.     |
+| **recording id**          | Unique identifier of the recording this sample belongs to.     |
+| **timestamp [ns]**        | UTC timestamp in nanoseconds of the sample. Equal to the timestamp of the eye video frame this sample was generated with. |
+| **pupil diameter left [mm]** <br /> **pupil diameter right [mm]**| Physical pupil diameter of the left and right eye respectively. Currently those values are always equal.  |
+| **eye ball center left x [mm]**<br /> **eye ball center left y [mm]**<br /> **eye ball center left z [mm]**<br /> **eye ball center right x [mm]**<br /> **eye ball center right y [mm]**<br /> **eye ball center right z [mm]** | Location of left and right eye ball centers in millimeters in relation to the Neon module. For details on the coordinate systems see [here](TODO). |
+| **optical axis left x**<br /> **optical axis left y**<br /> **optical axis left z**<br /> **optical axis right x**<br /> **optical axis right y**<br /> **optical axis right z** | Directional vector describing the optical axis of the left and right eye, i.e. the vector pointing from eye ball center to pupil center of the resepective eye. For details on the coordinate systems see [here](TODO). |
+
 ## blinks.csv
 This file contains [blinks](/neon/basic-concepts/data-streams/#blinks) detected in the eye video.
 The corresponding gaze samples that belong to each blink can be determined from the `gaze.csv` file using the `blink id` field.

neon/general/data-streams/index.md

@@ -10,13 +10,13 @@ An IR LED is located just above each camera, which guarantees good illumination
 
 
 ## Gaze <Badge>Real-time</Badge>
-The Neon Companion app can provide gaze data in real-time. When using a OnePlus 8 Companion device, the available framerate is +120 Hz (the achieved framerate varies from ~200Hz in the first minute of a recording to ~120Hz for longer recordings. This is due to restrictions in resource consumption applied by the Android operating system). Other apps running simultaneously on the phone may decrease the framerate.
-
-After a recording is uploaded to Pupil Cloud, gaze data is automatically re-computed at the full framerate of 200 Hz and can be downloaded from there.
+The Neon Companion app can provide gaze data in real-time at up to 200 Hz. Gaze data is output in pixel space of the scene camera image, which has a resolution of 1600x1200 px. The origin is in the top-left corner of the image.
 
 ![Gaze](./gaze.jpg)
 
-Gaze data is output in pixel space of the scene camera image, which has a resolution of 1600x1200 px. The origin is in the top-left corner of the image.
+The achieved framerate can vary based on what Companion device is used and environmental conditions. One the OnePlus 10 the full 200 Hz can generally be achieved outside of especially hot environments. On the OnePlus 8 the framerate typically drops to ~120 Hz within a few minutes of starting a recording. Other apps running simultaneously on the phone may decrease the framerate.
+
+After a recording is uploaded to Pupil Cloud, gaze data is automatically re-computed at the full 200 Hz framerat and can be downloaded from there.
 
 The gaze estimation algorithm is based on end-2-end deep learning and provides gaze data robustly without requiring a calibration. We are currently working on a white paper that thoroughly evaluated the algorithm and will link it here once it is published.
 
@@ -29,6 +29,16 @@ Fixations are calculated automatically in Pupil Cloud after uploading a recordin
 
 The deployed fixation detection algorithm was specifically designed for head-mounted eye trackers and offers increased robustness in the presence of head-movements. Especially movements due to vestibulo-ocular reflex are compensated for, which is not the case for most other fixation detection algorithms. Read more about that in the [Pupil Labs fixation detector whitepaper](https://docs.google.com/document/d/1dTL1VS83F-W1AZfbG-EogYwq2PFk463HqwGgshK3yJE/export?format=pdf)
 
+## 3D Eye States <Badge>Pupil Cloud</Badge>
+After uploading a recording to Pupil Cloud, 3D eye states are computed automatically at 200 Hz. The 3D eye states are a time series of each eye's position and orientation in 3D space, given by the eye ball center and the optical axis of each eye. 
+
+The coordinate system is defined as follows:
+
+![Coordinate systems of 3D eye states](./3d_eye_states.png)
+
+## Pupil Diameters <Badge>Pupil Cloud</Badge>
+After uploading a recording to Pupil Cloud, pupil diameters are computed automatically at 200 Hz. The computed pupil diameters correspond to the physical diameter of the 3D pupil, rather than the apparent pupil diameter in the eye video. Measurements are given in mm.
+
 ## Blinks <Badge>Pupil Cloud</Badge><Badge>Neon Player</Badge>
 During blinks the eye is briefly covered by the eye lids, which serves the purpose of spreading tears across the cornea. The blink rate and blink duration are also correlated with cognitive processes, which makes them interesting physiological signals.
 
@@ -64,7 +74,4 @@ When exporting recordings from Pupil Cloud or Neon Player the IMU's orientation
 
 Pitch is defined as a rotation around the x-axis with a value range of -90° to +90°. Yaw and roll are rotations around the y- and z-axis, respectively, with value ranges of -180° to +180°.
 
-![IMU Pitch, Yaw, Roll](./imu_pitch_yaw_roll.jpg)
-
-## Pupil Diamater and Eye Pose <Badge>Pupil Cloud</Badge>
-Coming soon!
+![IMU Pitch, Yaw, Roll](./imu_pitch_yaw_roll.jpg)