add_real_time_pupil_eyestate

neon/data-collection/data-format/index.md

@@ -15,7 +15,7 @@
 | **android_device_model**  | Model name of the Companion device.     |
 | **android_device_name**   | Device name of the Companion device.     |
 | **app_version**           | Version of the Neon Companion app used to make the recording. |
 | **calib_version**         | Version of the offset correction used by the Neon Companion app. |
 | **data_format_version**   | Version of the data format used by the Neon Companion app.     |
 | **duration**              | Duration of the recording in nanoseconds|
 | **firmware_version**      | Version numbers of the firmware and FPGA. |
@@ -43,7 +43,7 @@
 | Field | Description | 
 | -------- | -------- | 
 | **camera_matrix** | The camera matrix of the scene camera.     |
 | **dist_coefs** | The distortion coefficients of the scene camera. The order of the values is `(k1, k2, p1, p2, k3, k4, k5, k6)` following [OpenCV's distortion model](https://docs.opencv.org/4.x/d9/d0c/group__calib3d.html#ga3207604e4b1a1758aa66acb6ed5aa65d). |
 | **serial_number** | Serial number of Neon module used for the recording. This number is encoded in the QR code on the back of the Neon module.      |
 | **version** | The version of the intrinsics data format.     |
 
@@ -106,18 +106,18 @@
 | **azimuth [deg]** | The [azimuth](https://en.wikipedia.org/wiki/Horizontal_coordinate_system) of the gaze ray corresponding to the fixation location in relation to the scene camera in degrees.     |
 | **elevation [deg]** | The [elevation](https://en.wikipedia.org/wiki/Horizontal_coordinate_system) of the gaze ray corresponding to the fixation location in relation to the scene camera in degrees.     |
 
 ## saccades.csv
 This file contains [saccades](/data-collection/data-streams/#fixations-saccades) detected by the fixation detector.
 
 
 | Field | Description | 
 | -------- | -------- | 
 | **section id** | Unique identifier of the corresponding section.     |
 | **recording id** | Unique identifier of the recording this sample belongs to.     |
 | **saccade id** | Identifier of the saccade. The counter starts at the beginning of the recording.     |
 | **start timestamp [ns]** | UTC timestamp in nanoseconds of the start of the saccade.     |
 | **end timestamp [ns]** | UTC timestamp in nanoseconds of the end of the saccade.     |
 | **duration [ms]** | Duration of the saccade in milliseconds.     |
 | **amplitude [px]** | Float value representing the amplitude of the saccade in world camera pixel coordinates. |
 | **amplitude [deg]** | Float value representing the amplitude of the saccade in degrees of visual angle. |
 | **mean velocity [px/s]** | Float value representing the mean velocity of the saccade in world camera pixel coordinates per second. |
@@ -136,7 +136,7 @@
 | **pupil diameter left [mm]**   | Physical diameter of the pupil of the left eye. |
 | **pupil diameter right [mm]**   | Physical diameter of the pupil of the right eye. |
 | **eye&nbsp;ball&nbsp;center&nbsp;left&nbsp;x&nbsp;[mm]**<br /> **eye ball center left y [mm]**<br /> **eye ball center left z [mm]**<br /> **eye&nbsp;ball&nbsp;center&nbsp;right&nbsp;x&nbsp;[mm]**<br /> **eye&nbsp;ball&nbsp;center&nbsp;right&nbsp;y&nbsp;[mm]**<br /> **eye ball center right z [mm]** | Location of left and right eye ball centers in millimeters in relation to the scene camera of the Neon module. For details on the coordinate systems see [here](/data-collection/data-streams/#_3d-eye-states). |
-| **optical axis left x**<br /> **optical axis left y**<br /> **optical axis left z**<br /> **optical axis right x**<br /> **optical axis right y**<br /> **optical axis right z** | Directional vector describing the optical axis of the left and right eye, i.e. the vector pointing from eye ball center to pupil center of the resepective eye. For details on the coordinate systems see [here](/data-collection/data-streams/#_3d-eye-states). |
+| **optical axis left x**<br /> **optical axis left y**<br /> **optical axis left z**<br /> **optical axis right x**<br /> **optical axis right y**<br /> **optical axis right z** | Directional vector describing the optical axis of the left and right eye, i.e. the vector pointing from eye ball center to pupil center of the respective eye. For details on the coordinate systems see [here](/data-collection/data-streams/#_3d-eye-states). |
 
 ## blinks.csv
 This file contains [blinks](/data-collection/data-streams/#blinks) detected in the eye video.

neon/data-collection/data-streams/index.md

@@ -20,7 +20,7 @@
 
 The achieved framerate can vary based on what Companion device is used and environmental conditions. On the OnePlus 10, the full 200 Hz can generally be achieved outside of especially hot environments. On the OnePlus 8, the framerate typically drops to ~120 Hz within a few minutes of starting a recording. Other apps running simultaneously on the phone may decrease the framerate.
 
 After a recording is uploaded to Pupil Cloud, gaze data is automatically re-computed at the full 200 Hz framerat and can be downloaded from there.
 
 The gaze estimation algorithm is based on end-2-end deep learning and provides gaze data robustly without requiring a calibration. We are currently working on a white paper that thoroughly evaluated the algorithm and will link it here once it is published.
 
@@ -40,7 +40,7 @@
 
 ## 3D Eye States
 
-<Badge>Pupil Cloud</Badge>
+<Badge>Real-time</Badge><Badge>Pupil Cloud</Badge>
 After uploading a recording to Pupil Cloud, 3D eye states are computed automatically at 200 Hz. The 3D eye states are a time series of each eye's position and orientation in 3D space, given by the location of the eyeball center and the optical axis of each eye.
 
 The coordinate system is depicted below. The origin corresponds to the scene camera of the Neon Module.
@@ -51,7 +51,7 @@
 
 ## Pupil Diameters
 
-<Badge>Pupil Cloud</Badge>
+<Badge>Real-time</Badge><Badge>Pupil Cloud</Badge>
 After uploading a recording to Pupil Cloud, pupil diameters are computed automatically at 200 Hz, separately for the left and right eye. The computed pupil diameters correspond to the physical pupil size in mm, rather than the apparent pupil size in pixels as observed in the eye videos. 
 
 Similar to the 3D eye states, the accuracy of the pupil diameter measurements improves when supplying the wearer's IED in the wearer profile before making a recording.

neon/real-time-api/tutorials/index.md

@@ -116,6 +116,39 @@ plt.scatter(gaze_sample.x, gaze_sample.y, s=200, facecolors='none', edgecolors='
 
 Alternatively, you could also use the [`receive_scene_video_frame`](https://pupil-labs-realtime-api.readthedocs.io/en/stable/api/simple.html#pupil_labs.realtime_api.simple.Device.receive_scene_video_frame) and [`receive_gaze_datum`](https://pupil-labs-realtime-api.readthedocs.io/en/stable/api/simple.html#pupil_labs.realtime_api.simple.Device.receive_gaze_datum) methods to obtain each sample separately. The [`receive_matched_scene_video_frame_and_gaze`](https://pupil-labs-realtime-api.readthedocs.io/en/stable/api/simple.html#pupil_labs.realtime_api.simple.Device.receive_matched_scene_video_frame_and_gaze) method does however also ensure that both samples are matched temporally.
 
+## Gaze, Pupil Diameter, and Eye State Data
+Gaze, pupil diameter, and eye state data can be received using the [`receive_gaze_datum`](https://pupil-labs-realtime-api.readthedocs.io/en/stable/api/simple.html#pupil_labs.realtime_api.simple.Device.receive_gaze_datum) method. It returns a UTC timestamp in seconds, the gaze coordinates, the pupil diameter, and the eye state data. An example provided below:
+
+```python
+from pupil_labs.realtime_api.simple import discover_one_device
+from datetime import datetime
+
+device = discover_one_device()
+gaze_datum_sample = device.receive_gaze_datum()
+
+dt = datetime.fromtimestamp(gaze_datum_sample.timestamp_unix_seconds)
+print(f"This sample was recorded at {dt}\n")
+print("It contains the following data:\n")
+print(f"Gaze x and y coordinates: {gaze_datum_sample.x}, {gaze_datum_sample.y}\n")
+print(f"Pupil diameter in millimeters for the left eye: {gaze_datum_sample.pupil_diameter_left} and the right eye: {gaze_datum_sample.pupil_diameter_right}\n")
+print("Location of left and right eye ball centers in millimeters in relation to the scene camera of the Neon module.")
+print(f"For the left eye x, y, z: {gaze_datum_sample.eyeball_center_left_x}, {gaze_datum_sample.eyeball_center_left_y}, {gaze_datum_sample.eyeball_center_left_z} and for the right eye x, y, z: {gaze_datum_sample.eyeball_center_right_x}, {gaze_datum_sample.eyeball_center_right_y}, {gaze_datum_sample.eyeball_center_right_z}.\n")
+print("Directional vector describing the optical axis of the left and right eye.")
+print(f"For the left eye x, y, z: {gaze_datum_sample.optical_axis_left_x}, {gaze_datum_sample.optical_axis_left_y}, {gaze_datum_sample.optical_axis_left_z} and for the right eye x, y, z: {gaze_datum_sample.optical_axis_right_x}, {gaze_datum_sample.optical_axis_right_y}, {gaze_datum_sample.optical_axis_right_z}.")
+```
+This sample was recorded at 2024-04-30 12:07:18.912054
+
+It contains the following data:
+
+Gaze x and y coordinates: 724.7685546875, 528.932861328125
+
+Pupil diameter in millimeters for the left eye: 4.263330459594727 and the right eye: 3.3228392601013184
+
+Location of left and right eye ball centers in millimeters in relation to the scene camera of the Neon module.
+For the left eye x, y, z: -30.087890625, 10.048828125, -52.4462890625 and for the right eye x, y, z: 31.9189453125, 14.375, -48.6309814453125.
+
+Directional vector describing the optical axis of the left and right eye.
+For the left eye x, y, z: -0.05339553952217102, 0.12345726788043976, 0.9909123182296753 and for the right eye x, y, z: -0.40384653210639954, 0.11708031594753265, 0.9073038101196289.
 ## IMU Data
 
 Data generated by the IMU can be received using the [`receive_imu_datum`](https://pupil-labs-realtime-api.readthedocs.io/en/stable/api/simple.html#pupil_labs.realtime_api.simple.Device.receive_imu_datum) method. It returns a UTC timestamp in seconds, the head pose as a quaternion, gyro data, and accelerometer data as follows.