Merge pull request #681 from pupil-labs/rt_eyestate_pupil

add_real_time_pupil_eyestate

neon/data-collection/data-format/index.md

@@ -136,7 +136,7 @@ This file contains [3D eye states](/data-collection/data-streams/#_3d-eye-states
 | **pupil diameter left [mm]**   | Physical diameter of the pupil of the left eye. |
 | **pupil diameter right [mm]**   | Physical diameter of the pupil of the right eye. |
 | **eye&nbsp;ball&nbsp;center&nbsp;left&nbsp;x&nbsp;[mm]**<br /> **eye ball center left y [mm]**<br /> **eye ball center left z [mm]**<br /> **eye&nbsp;ball&nbsp;center&nbsp;right&nbsp;x&nbsp;[mm]**<br /> **eye&nbsp;ball&nbsp;center&nbsp;right&nbsp;y&nbsp;[mm]**<br /> **eye ball center right z [mm]** | Location of left and right eye ball centers in millimeters in relation to the scene camera of the Neon module. For details on the coordinate systems see [here](/data-collection/data-streams/#_3d-eye-states). |
-| **optical axis left x**<br /> **optical axis left y**<br /> **optical axis left z**<br /> **optical axis right x**<br /> **optical axis right y**<br /> **optical axis right z** | Directional vector describing the optical axis of the left and right eye, i.e. the vector pointing from eye ball center to pupil center of the resepective eye. For details on the coordinate systems see [here](/data-collection/data-streams/#_3d-eye-states). |
+| **optical axis left x**<br /> **optical axis left y**<br /> **optical axis left z**<br /> **optical axis right x**<br /> **optical axis right y**<br /> **optical axis right z** | Directional vector describing the optical axis of the left and right eye, i.e. the vector pointing from eye ball center to pupil center of the respective eye. For details on the coordinate systems see [here](/data-collection/data-streams/#_3d-eye-states). |
 
 ## blinks.csv
 This file contains [blinks](/data-collection/data-streams/#blinks) detected in the eye video.

neon/data-collection/data-streams/index.md

@@ -40,7 +40,7 @@ The downloads for gaze mapping enrichments ([Reference Image Mapper](/pupil-clou
 
 ## 3D Eye States
 
-<Badge>Pupil Cloud</Badge>
+<Badge>Real-time</Badge><Badge>Pupil Cloud</Badge>
 After uploading a recording to Pupil Cloud, 3D eye states are computed automatically at 200 Hz. The 3D eye states are a time series of each eye's position and orientation in 3D space, given by the location of the eyeball center and the optical axis of each eye.
 
 The coordinate system is depicted below. The origin corresponds to the scene camera of the Neon Module.
@@ -51,7 +51,7 @@ You can specify the inter-eye distance (IED) of a wearer in the wearer profile b
 
 ## Pupil Diameters
 
-<Badge>Pupil Cloud</Badge>
+<Badge>Real-time</Badge><Badge>Pupil Cloud</Badge>
 After uploading a recording to Pupil Cloud, pupil diameters are computed automatically at 200 Hz, separately for the left and right eye. The computed pupil diameters correspond to the physical pupil size in mm, rather than the apparent pupil size in pixels as observed in the eye videos. 
 
 Similar to the 3D eye states, the accuracy of the pupil diameter measurements improves when supplying the wearer's IED in the wearer profile before making a recording.

neon/real-time-api/tutorials/index.md

@@ -95,9 +95,9 @@ Event(name=None recording_id=fd8c98ca-cd6c-4d3f-9a05-fbdb0ef42668 timestamp_unix
 
 ```
 
-## Scene Video and Gaze Data
+## Scene Video and Gaze, Pupil Diameter, and Eye State Data
 
-You can receive the current scene camera frame as well as the current gaze sample using the [`receive_matched_scene_video_frame_and_gaze`](https://pupil-labs-realtime-api.readthedocs.io/en/stable/api/simple.html#pupil_labs.realtime_api.simple.Device.receive_matched_scene_video_frame_and_gaze) method.
+You can receive the current scene camera frame as well as the current gaze sample using the [`receive_matched_scene_video_frame_and_gaze`](https://pupil-labs-realtime-api.readthedocs.io/en/stable/api/simple.html#pupil_labs.realtime_api.simple.Device.receive_matched_scene_video_frame_and_gaze) method. This method also provides pupil diameter and eye state data, separately for each eye. An example is provided below:
 
 ```python
 from datetime import datetime
@@ -109,11 +109,32 @@ device = discover_one_device()
 
 scene_sample, gaze_sample = device.receive_matched_scene_video_frame_and_gaze()
 
+print("This sample contains the following data:\n")
+print(f"Gaze x and y coordinates: {gaze_sample.x}, {gaze_sample.y}\n")
+print(f"Pupil diameter in millimeters for the left eye: {gaze_sample.pupil_diameter_left} and the right eye: {gaze_sample.pupil_diameter_right}\n")
+print("Location of left and right eye ball centers in millimeters in relation to the scene camera of the Neon module.")
+print(f"For the left eye x, y, z: {gaze_sample.eyeball_center_left_x}, {gaze_sample.eyeball_center_left_y}, {gaze_sample.eyeball_center_left_z} and for the right eye x, y, z: {gaze_sample.eyeball_center_right_x}, {gaze_sample.eyeball_center_right_y}, {gaze_sample.eyeball_center_right_z}.\n")
+print("Directional vector describing the optical axis of the left and right eye.")
+print(f"For the left eye x, y, z: {gaze_sample.optical_axis_left_x}, {gaze_sample.optical_axis_left_y}, {gaze_sample.optical_axis_left_z} and for the right eye x, y, z: {gaze_sample.optical_axis_right_x}, {gaze_sample.optical_axis_right_y}, {gaze_sample.optical_axis_right_z}.")
+
 scene_image_rgb = cv2.cvtColor(scene_sample.bgr_pixels, cv2.COLOR_BGR2RGB)
 plt.imshow(scene_image_rgb)
 plt.scatter(gaze_sample.x, gaze_sample.y, s=200, facecolors='none', edgecolors='r')
 ```
+The output data would look as follows:
+```
+This sample contains the following data:
+
+Gaze x and y coordinates: 724.7685546875, 528.932861328125
+
+Pupil diameter in millimeters for the left eye: 4.263330459594727 and the right eye: 3.3228392601013184
 
+Location of left and right eye ball centers in millimeters in relation to the scene camera of the Neon module.
+For the left eye x, y, z: -30.087890625, 10.048828125, -52.4462890625 and for the right eye x, y, z: 31.9189453125, 14.375, -48.6309814453125.
+
+Directional vector describing the optical axis of the left and right eye.
+For the left eye x, y, z: -0.05339553952217102, 0.12345726788043976, 0.9909123182296753 and for the right eye x, y, z: -0.40384653210639954, 0.11708031594753265, 0.9073038101196289.
+```
 Alternatively, you could also use the [`receive_scene_video_frame`](https://pupil-labs-realtime-api.readthedocs.io/en/stable/api/simple.html#pupil_labs.realtime_api.simple.Device.receive_scene_video_frame) and [`receive_gaze_datum`](https://pupil-labs-realtime-api.readthedocs.io/en/stable/api/simple.html#pupil_labs.realtime_api.simple.Device.receive_gaze_datum) methods to obtain each sample separately. The [`receive_matched_scene_video_frame_and_gaze`](https://pupil-labs-realtime-api.readthedocs.io/en/stable/api/simple.html#pupil_labs.realtime_api.simple.Device.receive_matched_scene_video_frame_and_gaze) method does however also ensure that both samples are matched temporally.
 
 ## IMU Data
@@ -140,6 +161,8 @@ print(imu_sample.accel_data)
 print(f"Gyro data:")
 print(imu_sample.gyro_data)
 ```
+The output data would look as follows:
+```
 This IMU sample was recorded at 2023-05-25 11:23:05.749155
 It contains the following data:
 Head pose (as quaternion)