From a4e57802ffb02c669be090520e9bc23a13f95411 Mon Sep 17 00:00:00 2001 From: nadje Date: Tue, 30 Apr 2024 12:14:58 +0200 Subject: [PATCH 1/4] add_real_time_pupil_eyestate --- neon/data-collection/data-format/index.md | 2 +- neon/data-collection/data-streams/index.md | 4 +-- neon/real-time-api/tutorials/index.md | 33 ++++++++++++++++++++++ 3 files changed, 36 insertions(+), 3 deletions(-) diff --git a/neon/data-collection/data-format/index.md b/neon/data-collection/data-format/index.md index 3eb64afc3..6797da14e 100644 --- a/neon/data-collection/data-format/index.md +++ b/neon/data-collection/data-format/index.md @@ -136,7 +136,7 @@ This file contains [3D eye states](/data-collection/data-streams/#_3d-eye-states | **pupil diameter left [mm]** | Physical diameter of the pupil of the left eye. | | **pupil diameter right [mm]** | Physical diameter of the pupil of the right eye. | | **eye ball center left x [mm]**
**eye ball center left y [mm]**
**eye ball center left z [mm]**
**eye ball center right x [mm]**
**eye ball center right y [mm]**
**eye ball center right z [mm]** | Location of left and right eye ball centers in millimeters in relation to the scene camera of the Neon module. For details on the coordinate systems see [here](/data-collection/data-streams/#_3d-eye-states). | -| **optical axis left x**
**optical axis left y**
**optical axis left z**
**optical axis right x**
**optical axis right y**
**optical axis right z** | Directional vector describing the optical axis of the left and right eye, i.e. the vector pointing from eye ball center to pupil center of the resepective eye. For details on the coordinate systems see [here](/data-collection/data-streams/#_3d-eye-states). | +| **optical axis left x**
**optical axis left y**
**optical axis left z**
**optical axis right x**
**optical axis right y**
**optical axis right z** | Directional vector describing the optical axis of the left and right eye, i.e. the vector pointing from eye ball center to pupil center of the respective eye. For details on the coordinate systems see [here](/data-collection/data-streams/#_3d-eye-states). | ## blinks.csv This file contains [blinks](/data-collection/data-streams/#blinks) detected in the eye video. diff --git a/neon/data-collection/data-streams/index.md b/neon/data-collection/data-streams/index.md index 154002591..d9c8fc3bd 100644 --- a/neon/data-collection/data-streams/index.md +++ b/neon/data-collection/data-streams/index.md @@ -40,7 +40,7 @@ The downloads for gaze mapping enrichments ([Reference Image Mapper](/pupil-clou ## 3D Eye States -Pupil Cloud +Real-timePupil Cloud After uploading a recording to Pupil Cloud, 3D eye states are computed automatically at 200 Hz. The 3D eye states are a time series of each eye's position and orientation in 3D space, given by the location of the eyeball center and the optical axis of each eye. The coordinate system is depicted below. The origin corresponds to the scene camera of the Neon Module. @@ -51,7 +51,7 @@ You can specify the inter-eye distance (IED) of a wearer in the wearer profile b ## Pupil Diameters -Pupil Cloud +Real-timePupil Cloud After uploading a recording to Pupil Cloud, pupil diameters are computed automatically at 200 Hz, separately for the left and right eye. The computed pupil diameters correspond to the physical pupil size in mm, rather than the apparent pupil size in pixels as observed in the eye videos. Similar to the 3D eye states, the accuracy of the pupil diameter measurements improves when supplying the wearer's IED in the wearer profile before making a recording. diff --git a/neon/real-time-api/tutorials/index.md b/neon/real-time-api/tutorials/index.md index 437e5c2fa..cf08c602b 100644 --- a/neon/real-time-api/tutorials/index.md +++ b/neon/real-time-api/tutorials/index.md @@ -116,6 +116,39 @@ plt.scatter(gaze_sample.x, gaze_sample.y, s=200, facecolors='none', edgecolors=' Alternatively, you could also use the [`receive_scene_video_frame`](https://pupil-labs-realtime-api.readthedocs.io/en/stable/api/simple.html#pupil_labs.realtime_api.simple.Device.receive_scene_video_frame) and [`receive_gaze_datum`](https://pupil-labs-realtime-api.readthedocs.io/en/stable/api/simple.html#pupil_labs.realtime_api.simple.Device.receive_gaze_datum) methods to obtain each sample separately. The [`receive_matched_scene_video_frame_and_gaze`](https://pupil-labs-realtime-api.readthedocs.io/en/stable/api/simple.html#pupil_labs.realtime_api.simple.Device.receive_matched_scene_video_frame_and_gaze) method does however also ensure that both samples are matched temporally. +## Gaze, Pupil Diameter, and Eye State Data +Gaze, pupil diameter, and eye state data can be received using the [`receive_gaze_datum`](https://pupil-labs-realtime-api.readthedocs.io/en/stable/api/simple.html#pupil_labs.realtime_api.simple.Device.receive_gaze_datum) method. It returns a UTC timestamp in seconds, the gaze coordinates, the pupil diameter, and the eye state data. An example provided below: + +```python +from pupil_labs.realtime_api.simple import discover_one_device +from datetime import datetime + +device = discover_one_device() +gaze_datum_sample = device.receive_gaze_datum() + +dt = datetime.fromtimestamp(gaze_datum_sample.timestamp_unix_seconds) +print(f"This sample was recorded at {dt}\n") +print("It contains the following data:\n") +print(f"Gaze x and y coordinates: {gaze_datum_sample.x}, {gaze_datum_sample.y}\n") +print(f"Pupil diameter in millimeters for the left eye: {gaze_datum_sample.pupil_diameter_left} and the right eye: {gaze_datum_sample.pupil_diameter_right}\n") +print("Location of left and right eye ball centers in millimeters in relation to the scene camera of the Neon module.") +print(f"For the left eye x, y, z: {gaze_datum_sample.eyeball_center_left_x}, {gaze_datum_sample.eyeball_center_left_y}, {gaze_datum_sample.eyeball_center_left_z} and for the right eye x, y, z: {gaze_datum_sample.eyeball_center_right_x}, {gaze_datum_sample.eyeball_center_right_y}, {gaze_datum_sample.eyeball_center_right_z}.\n") +print("Directional vector describing the optical axis of the left and right eye.") +print(f"For the left eye x, y, z: {gaze_datum_sample.optical_axis_left_x}, {gaze_datum_sample.optical_axis_left_y}, {gaze_datum_sample.optical_axis_left_z} and for the right eye x, y, z: {gaze_datum_sample.optical_axis_right_x}, {gaze_datum_sample.optical_axis_right_y}, {gaze_datum_sample.optical_axis_right_z}.") +``` +This sample was recorded at 2024-04-30 12:07:18.912054 + +It contains the following data: + +Gaze x and y coordinates: 724.7685546875, 528.932861328125 + +Pupil diameter in millimeters for the left eye: 4.263330459594727 and the right eye: 3.3228392601013184 + +Location of left and right eye ball centers in millimeters in relation to the scene camera of the Neon module. +For the left eye x, y, z: -30.087890625, 10.048828125, -52.4462890625 and for the right eye x, y, z: 31.9189453125, 14.375, -48.6309814453125. + +Directional vector describing the optical axis of the left and right eye. +For the left eye x, y, z: -0.05339553952217102, 0.12345726788043976, 0.9909123182296753 and for the right eye x, y, z: -0.40384653210639954, 0.11708031594753265, 0.9073038101196289. ## IMU Data Data generated by the IMU can be received using the [`receive_imu_datum`](https://pupil-labs-realtime-api.readthedocs.io/en/stable/api/simple.html#pupil_labs.realtime_api.simple.Device.receive_imu_datum) method. It returns a UTC timestamp in seconds, the head pose as a quaternion, gyro data, and accelerometer data as follows. From b95f6ede861c31da79abc69b0218d9563fb00aac Mon Sep 17 00:00:00 2001 From: Nadia P Date: Tue, 30 Apr 2024 14:36:54 +0200 Subject: [PATCH 2/4] Update neon/real-time-api/tutorials/index.md Co-authored-by: Marc Tonsen --- neon/real-time-api/tutorials/index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/neon/real-time-api/tutorials/index.md b/neon/real-time-api/tutorials/index.md index cf08c602b..5bc2ff38e 100644 --- a/neon/real-time-api/tutorials/index.md +++ b/neon/real-time-api/tutorials/index.md @@ -117,7 +117,7 @@ plt.scatter(gaze_sample.x, gaze_sample.y, s=200, facecolors='none', edgecolors=' Alternatively, you could also use the [`receive_scene_video_frame`](https://pupil-labs-realtime-api.readthedocs.io/en/stable/api/simple.html#pupil_labs.realtime_api.simple.Device.receive_scene_video_frame) and [`receive_gaze_datum`](https://pupil-labs-realtime-api.readthedocs.io/en/stable/api/simple.html#pupil_labs.realtime_api.simple.Device.receive_gaze_datum) methods to obtain each sample separately. The [`receive_matched_scene_video_frame_and_gaze`](https://pupil-labs-realtime-api.readthedocs.io/en/stable/api/simple.html#pupil_labs.realtime_api.simple.Device.receive_matched_scene_video_frame_and_gaze) method does however also ensure that both samples are matched temporally. ## Gaze, Pupil Diameter, and Eye State Data -Gaze, pupil diameter, and eye state data can be received using the [`receive_gaze_datum`](https://pupil-labs-realtime-api.readthedocs.io/en/stable/api/simple.html#pupil_labs.realtime_api.simple.Device.receive_gaze_datum) method. It returns a UTC timestamp in seconds, the gaze coordinates, the pupil diameter, and the eye state data. An example provided below: +Gaze, pupil diameter, and eye state data can be received using the [`receive_gaze_datum`](https://pupil-labs-realtime-api.readthedocs.io/en/stable/api/simple.html#pupil_labs.realtime_api.simple.Device.receive_gaze_datum) method. An example is provided below: ```python from pupil_labs.realtime_api.simple import discover_one_device From 3d6119759d6955840293178e0ce8c473965f46bf Mon Sep 17 00:00:00 2001 From: Nadia P Date: Tue, 30 Apr 2024 14:58:14 +0200 Subject: [PATCH 3/4] Update api tutorial and format fixes --- neon/real-time-api/tutorials/index.md | 23 +++++++++++++++++------ 1 file changed, 17 insertions(+), 6 deletions(-) diff --git a/neon/real-time-api/tutorials/index.md b/neon/real-time-api/tutorials/index.md index 5bc2ff38e..990746d1b 100644 --- a/neon/real-time-api/tutorials/index.md +++ b/neon/real-time-api/tutorials/index.md @@ -95,9 +95,9 @@ Event(name=None recording_id=fd8c98ca-cd6c-4d3f-9a05-fbdb0ef42668 timestamp_unix ``` -## Scene Video and Gaze Data +## Scene Video and Gaze, Pupil Diameter, and Eye State Data -You can receive the current scene camera frame as well as the current gaze sample using the [`receive_matched_scene_video_frame_and_gaze`](https://pupil-labs-realtime-api.readthedocs.io/en/stable/api/simple.html#pupil_labs.realtime_api.simple.Device.receive_matched_scene_video_frame_and_gaze) method. +You can receive the current scene camera frame as well as the current gaze sample using the [`receive_matched_scene_video_frame_and_gaze`](https://pupil-labs-realtime-api.readthedocs.io/en/stable/api/simple.html#pupil_labs.realtime_api.simple.Device.receive_matched_scene_video_frame_and_gaze) method. This method also provides pupil diameter and eye state data, separately for each eye. An example is provided below: ```python from datetime import datetime @@ -109,6 +109,14 @@ device = discover_one_device() scene_sample, gaze_sample = device.receive_matched_scene_video_frame_and_gaze() +print("This sample contains the following data:\n") +print(f"Gaze x and y coordinates: {gaze_sample.x}, {gaze_sample.y}\n") +print(f"Pupil diameter in millimeters for the left eye: {gaze_sample.pupil_diameter_left} and the right eye: {gaze_sample.pupil_diameter_right}\n") +print("Location of left and right eye ball centers in millimeters in relation to the scene camera of the Neon module.") +print(f"For the left eye x, y, z: {gaze_sample.eyeball_center_left_x}, {gaze_sample.eyeball_center_left_y}, {gaze_sample.eyeball_center_left_z} and for the right eye x, y, z: {gaze_sample.eyeball_center_right_x}, {gaze_sample.eyeball_center_right_y}, {gaze_sample.eyeball_center_right_z}.\n") +print("Directional vector describing the optical axis of the left and right eye.") +print(f"For the left eye x, y, z: {gaze_sample.optical_axis_left_x}, {gaze_sample.optical_axis_left_y}, {gaze_sample.optical_axis_left_z} and for the right eye x, y, z: {gaze_sample.optical_axis_right_x}, {gaze_sample.optical_axis_right_y}, {gaze_sample.optical_axis_right_z}.") + scene_image_rgb = cv2.cvtColor(scene_sample.bgr_pixels, cv2.COLOR_BGR2RGB) plt.imshow(scene_image_rgb) plt.scatter(gaze_sample.x, gaze_sample.y, s=200, facecolors='none', edgecolors='r') @@ -117,7 +125,7 @@ plt.scatter(gaze_sample.x, gaze_sample.y, s=200, facecolors='none', edgecolors=' Alternatively, you could also use the [`receive_scene_video_frame`](https://pupil-labs-realtime-api.readthedocs.io/en/stable/api/simple.html#pupil_labs.realtime_api.simple.Device.receive_scene_video_frame) and [`receive_gaze_datum`](https://pupil-labs-realtime-api.readthedocs.io/en/stable/api/simple.html#pupil_labs.realtime_api.simple.Device.receive_gaze_datum) methods to obtain each sample separately. The [`receive_matched_scene_video_frame_and_gaze`](https://pupil-labs-realtime-api.readthedocs.io/en/stable/api/simple.html#pupil_labs.realtime_api.simple.Device.receive_matched_scene_video_frame_and_gaze) method does however also ensure that both samples are matched temporally. ## Gaze, Pupil Diameter, and Eye State Data -Gaze, pupil diameter, and eye state data can be received using the [`receive_gaze_datum`](https://pupil-labs-realtime-api.readthedocs.io/en/stable/api/simple.html#pupil_labs.realtime_api.simple.Device.receive_gaze_datum) method. An example is provided below: +Gaze, pupil diameter, and eye state data can be received using the [`receive_gaze_datum`](https://pupil-labs-realtime-api.readthedocs.io/en/stable/api/simple.html#pupil_labs.realtime_api.simple.Device.receive_gaze_datum) method. It returns a UTC timestamp in seconds, the gaze coordinates, the pupil diameter, and the eye state data. An example provided below: ```python from pupil_labs.realtime_api.simple import discover_one_device @@ -136,9 +144,7 @@ print(f"For the left eye x, y, z: {gaze_datum_sample.eyeball_center_left_x}, {ga print("Directional vector describing the optical axis of the left and right eye.") print(f"For the left eye x, y, z: {gaze_datum_sample.optical_axis_left_x}, {gaze_datum_sample.optical_axis_left_y}, {gaze_datum_sample.optical_axis_left_z} and for the right eye x, y, z: {gaze_datum_sample.optical_axis_right_x}, {gaze_datum_sample.optical_axis_right_y}, {gaze_datum_sample.optical_axis_right_z}.") ``` -This sample was recorded at 2024-04-30 12:07:18.912054 - -It contains the following data: +This sample contains the following data: Gaze x and y coordinates: 724.7685546875, 528.932861328125 @@ -149,6 +155,9 @@ For the left eye x, y, z: -30.087890625, 10.048828125, -52.4462890625 and for th Directional vector describing the optical axis of the left and right eye. For the left eye x, y, z: -0.05339553952217102, 0.12345726788043976, 0.9909123182296753 and for the right eye x, y, z: -0.40384653210639954, 0.11708031594753265, 0.9073038101196289. +``` +Alternatively, you could also use the [`receive_scene_video_frame`](https://pupil-labs-realtime-api.readthedocs.io/en/stable/api/simple.html#pupil_labs.realtime_api.simple.Device.receive_scene_video_frame) and [`receive_gaze_datum`](https://pupil-labs-realtime-api.readthedocs.io/en/stable/api/simple.html#pupil_labs.realtime_api.simple.Device.receive_gaze_datum) methods to obtain each sample separately. The [`receive_matched_scene_video_frame_and_gaze`](https://pupil-labs-realtime-api.readthedocs.io/en/stable/api/simple.html#pupil_labs.realtime_api.simple.Device.receive_matched_scene_video_frame_and_gaze) method does however also ensure that both samples are matched temporally. + ## IMU Data Data generated by the IMU can be received using the [`receive_imu_datum`](https://pupil-labs-realtime-api.readthedocs.io/en/stable/api/simple.html#pupil_labs.realtime_api.simple.Device.receive_imu_datum) method. It returns a UTC timestamp in seconds, the head pose as a quaternion, gyro data, and accelerometer data as follows. @@ -173,6 +182,8 @@ print(imu_sample.accel_data) print(f"Gyro data:") print(imu_sample.gyro_data) ``` +The output data would look as follows: +``` This IMU sample was recorded at 2023-05-25 11:23:05.749155 It contains the following data: Head pose (as quaternion) From df2dc856d649ada344af9b43db1bfa57e118d38b Mon Sep 17 00:00:00 2001 From: Nadia P Date: Tue, 30 Apr 2024 14:59:30 +0200 Subject: [PATCH 4/4] merge paragraphs --- neon/real-time-api/tutorials/index.md | 23 +---------------------- 1 file changed, 1 insertion(+), 22 deletions(-) diff --git a/neon/real-time-api/tutorials/index.md b/neon/real-time-api/tutorials/index.md index 990746d1b..5f771c831 100644 --- a/neon/real-time-api/tutorials/index.md +++ b/neon/real-time-api/tutorials/index.md @@ -121,28 +121,7 @@ scene_image_rgb = cv2.cvtColor(scene_sample.bgr_pixels, cv2.COLOR_BGR2RGB) plt.imshow(scene_image_rgb) plt.scatter(gaze_sample.x, gaze_sample.y, s=200, facecolors='none', edgecolors='r') ``` - -Alternatively, you could also use the [`receive_scene_video_frame`](https://pupil-labs-realtime-api.readthedocs.io/en/stable/api/simple.html#pupil_labs.realtime_api.simple.Device.receive_scene_video_frame) and [`receive_gaze_datum`](https://pupil-labs-realtime-api.readthedocs.io/en/stable/api/simple.html#pupil_labs.realtime_api.simple.Device.receive_gaze_datum) methods to obtain each sample separately. The [`receive_matched_scene_video_frame_and_gaze`](https://pupil-labs-realtime-api.readthedocs.io/en/stable/api/simple.html#pupil_labs.realtime_api.simple.Device.receive_matched_scene_video_frame_and_gaze) method does however also ensure that both samples are matched temporally. - -## Gaze, Pupil Diameter, and Eye State Data -Gaze, pupil diameter, and eye state data can be received using the [`receive_gaze_datum`](https://pupil-labs-realtime-api.readthedocs.io/en/stable/api/simple.html#pupil_labs.realtime_api.simple.Device.receive_gaze_datum) method. It returns a UTC timestamp in seconds, the gaze coordinates, the pupil diameter, and the eye state data. An example provided below: - -```python -from pupil_labs.realtime_api.simple import discover_one_device -from datetime import datetime - -device = discover_one_device() -gaze_datum_sample = device.receive_gaze_datum() - -dt = datetime.fromtimestamp(gaze_datum_sample.timestamp_unix_seconds) -print(f"This sample was recorded at {dt}\n") -print("It contains the following data:\n") -print(f"Gaze x and y coordinates: {gaze_datum_sample.x}, {gaze_datum_sample.y}\n") -print(f"Pupil diameter in millimeters for the left eye: {gaze_datum_sample.pupil_diameter_left} and the right eye: {gaze_datum_sample.pupil_diameter_right}\n") -print("Location of left and right eye ball centers in millimeters in relation to the scene camera of the Neon module.") -print(f"For the left eye x, y, z: {gaze_datum_sample.eyeball_center_left_x}, {gaze_datum_sample.eyeball_center_left_y}, {gaze_datum_sample.eyeball_center_left_z} and for the right eye x, y, z: {gaze_datum_sample.eyeball_center_right_x}, {gaze_datum_sample.eyeball_center_right_y}, {gaze_datum_sample.eyeball_center_right_z}.\n") -print("Directional vector describing the optical axis of the left and right eye.") -print(f"For the left eye x, y, z: {gaze_datum_sample.optical_axis_left_x}, {gaze_datum_sample.optical_axis_left_y}, {gaze_datum_sample.optical_axis_left_z} and for the right eye x, y, z: {gaze_datum_sample.optical_axis_right_x}, {gaze_datum_sample.optical_axis_right_y}, {gaze_datum_sample.optical_axis_right_z}.") +The output data would look as follows: ``` This sample contains the following data: