Skip to content

Commit

Permalink
add PI saccades and Neon models
Browse files Browse the repository at this point in the history
  • Loading branch information
nadje committed Apr 26, 2024
1 parent 834cf87 commit 20c920a
Show file tree
Hide file tree
Showing 3 changed files with 41 additions and 21 deletions.
43 changes: 31 additions & 12 deletions invisible/data-collection/data-format/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -84,21 +84,40 @@ This file contains [gaze](/data-collection/data-streams/#gaze) data in world cam
| **elevation [deg]** | The [elevation](https://en.wikipedia.org/wiki/Horizontal_coordinate_system) of the gaze ray in relation to the scene camera in degrees. |

## fixations.csv

This file contains [fixations](/data-collection/data-streams/#fixations) detected in the gaze data stream.
This file contains [fixations](/data-collection/data-streams/#fixations-saccades) detected in the gaze data stream.
The corresponding gaze samples that belong to each fixation can be determined from the `gaze.csv` file using the `fixation id` field.

| Field | Description |
| ------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| **section id** | Unique identifier of the corresponding section. |
| **recording id** | Unique identifier of the recording this sample belongs to. |
| **fixation id** | Identifier of the fixation. The counter starts at the beginning of the recording. |
| **start timestamp [ns]** | UTC timestamp in nanoseconds of the start of the fixation. |
| **end timestamp [ns]** | UTC timestamp in nanoseconds of the end of the fixation. |
| **duration [ms]** | Duration of the fixation in milliseconds. |
| **fixation x [px]** | Float value representing the x-coordinate of the fixation in world camera pixel coordinates. This position is the average of all gaze samples within the fixation. |
| **fixation y [px]** | Same as "fixation x [px]" but for the y-coordinate. |

| Field | Description |
| -------- | -------- |
| **section id** | Unique identifier of the corresponding section. |
| **recording id** | Unique identifier of the recording this sample belongs to. |
| **fixation id** | Identifier of the fixation. The counter starts at the beginning of the recording. |
| **start timestamp [ns]** | UTC timestamp in nanoseconds of the start of the fixation. |
| **end timestamp [ns]** | UTC timestamp in nanoseconds of the end of the fixation. |
| **duration [ms]** | Duration of the fixation in milliseconds. |
| **fixation x [px]** | Float value representing the x-coordinate of the fixation in world camera pixel coordinates. This position is the average of all gaze samples within the fixation. |
| **fixation y [px]** | Same as "fixation x [px]" but for the y-coordinate. |
| **azimuth [deg]** | The [azimuth](https://en.wikipedia.org/wiki/Horizontal_coordinate_system) of the gaze ray corresponding to the fixation location in relation to the scene camera in degrees. |
| **elevation [deg]** | The [elevation](https://en.wikipedia.org/wiki/Horizontal_coordinate_system) of the gaze ray corresponding to the fixation location in relation to the scene camera in degrees. |

## saccades.csv

Check warning on line 104 in invisible/data-collection/data-format/index.md

View workflow job for this annotation

GitHub Actions / ✍️ Check spelling

Unknown word (saccades)
This file contains [saccades](/data-collection/data-streams/#fixations-saccades) detected by the fixation detector.

Check warning on line 105 in invisible/data-collection/data-format/index.md

View workflow job for this annotation

GitHub Actions / ✍️ Check spelling

Unknown word (saccades)


| Field | Description |
| -------- | -------- |
| **section id** | Unique identifier of the corresponding section. |
| **recording id** | Unique identifier of the recording this sample belongs to. |
| **saccade id** | Identifier of the saccade. The counter starts at the beginning of the recording. |

Check warning on line 112 in invisible/data-collection/data-format/index.md

View workflow job for this annotation

GitHub Actions / ✍️ Check spelling

Unknown word (saccade)

Check warning on line 112 in invisible/data-collection/data-format/index.md

View workflow job for this annotation

GitHub Actions / ✍️ Check spelling

Unknown word (saccade)
| **start timestamp [ns]** | UTC timestamp in nanoseconds of the start of the saccade. |

Check warning on line 113 in invisible/data-collection/data-format/index.md

View workflow job for this annotation

GitHub Actions / ✍️ Check spelling

Unknown word (saccade)
| **end timestamp [ns]** | UTC timestamp in nanoseconds of the end of the saccade. |

Check warning on line 114 in invisible/data-collection/data-format/index.md

View workflow job for this annotation

GitHub Actions / ✍️ Check spelling

Unknown word (saccade)
| **duration [ms]** | Duration of the saccade in milliseconds. |

Check warning on line 115 in invisible/data-collection/data-format/index.md

View workflow job for this annotation

GitHub Actions / ✍️ Check spelling

Unknown word (saccade)
| **amplitude [px]** | Float value representing the amplitude of the saccade in world camera pixel coordinates. |
| **amplitude [deg]** | Float value representing the amplitude of the saccade in degrees of visual angle. |
| **mean velocity [px/s]** | Float value representing the mean velocity of the saccade in world camera pixel coordinates per second. |
| **peak velocity [px/s]** | Float value representing the peak velocity of the saccade in world camera pixel coordinates per second. |
## blinks.csv

This file contains [blinks](/data-collection/data-streams/#blinks) detected in the eye video.
Expand Down
15 changes: 7 additions & 8 deletions invisible/data-collection/data-streams/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,20 +21,19 @@ Gaze data is output in pixel space of the scene camera image, which has a resolu

The gaze estimation algorithm is based on end-2-end deep learning and provides gaze data robustly without requiring a calibration. You can find a high-level description as well as a thorough evaluation of the accuracy and robustness of the algorithm in our [white paper](https://arxiv.org/pdf/2009.00508).

## Fixations
Two primary types of eye movements exhibited by the visual system are fixations and saccades. During fixations, the eyes are directed at a specific point in the environment. A saccade is a very quick movement where the eyes jump from one fixation to the next. Properties like fixation duration are of significant importance for studying gaze behaviour.
## Fixations & Saccades

Check warning on line 24 in invisible/data-collection/data-streams/index.md

View workflow job for this annotation

GitHub Actions / ✍️ Check spelling

Unknown word (Saccades)

<Badge>Pupil Cloud</Badge><Badge>Neon Player</Badge>
The two primary types of eye movements exhibited by the visual system are fixations and saccades. During fixations, the eyes are directed at a specific point in the environment. A saccade is a very quick movement where the eyes jump from one fixation to the next. Properties like the fixation duration are of significant importance for studying gaze behavior.

![Fixations](./fixations.jpg)

Fixations are calculated automatically in Pupil Cloud after uploading a recording. They are available as part of the [recording exports](/data-collection/data-format/). The downloads for gaze mapping enrichments ([Reference Image Mapper](/pupil-cloud/enrichments/reference-image-mapper/#fixations-csv), [Marker Mapper](/pupil-cloud/enrichments/marker-mapper/#fixations-csv)) also include "mapped fixations".
Fixations and saccades are calculated automatically in Pupil Cloud after uploading a recording and are included in the recording downloads. The deployed fixation detection algorithm was specifically designed for head-mounted eye trackers and offers increased robustness in the presence of head movements. Especially movements due to vestibulo-ocular reflex are compensated for, which is not the case for most other fixation detection algorithms. You can learn more about it in the [Pupil Labs fixation detector whitepaper](https://docs.google.com/document/d/1dTL1VS83F-W1AZfbG-EogYwq2PFk463HqwGgshK3yJE/export?format=pdf) and in our [publication](https://link.springer.com/article/10.3758/s13428-024-02360-0) in *Behavior Research Methods* discussing fixation detection strategies.

The fixation detection algorithm employed in Pupil Cloud was developed by us, and was specifically designed for head-mounted eye trackers.
We detect saccades based on the fixation results, considering the gaps between fixations to be saccades. Note, that this assumption is only true in the absence of smooth pursuit eye movements. Additionally, the fixation detector does not compensate for blinks, which can cause a break in a fixation and thus introduce a false saccade.

Traditionally, fixation detection algorithms assumed the head of the subject to be stationary, which is usually not the case when using head-mounted eye trackers.
If head movements occur during a fixation, compensatory eye movements are required to keep gaze stable on the point of fixation. These are initiated by the
vestibulo-ocular reflex (VOR).
The downloads for gaze mapping enrichments ([Reference Image Mapper](/pupil-cloud/enrichments/reference-image-mapper/#export-format), [Marker Mapper](/pupil-cloud/enrichments/marker-mapper/#export-format)) also include mapped fixations, i.e. fixations in reference image or surface coordinates respectively.

Our algorithm explicitly compensates for VOR eye movements and thus correctly classifies fixations in more dynamic scenarios! Read more about that in the [Pupil Labs fixation detector whitepaper](https://docs.google.com/document/d/1dTL1VS83F-W1AZfbG-EogYwq2PFk463HqwGgshK3yJE/export?format=pdf)

## Blinks
During blinks the eye is briefly covered by the eye lids, which serves the purpose of spreading tears across the cornea. The blink rate and blink duration are however also correlated with cognitive processes, which makes it an interesting physiological signal.
Expand Down
4 changes: 3 additions & 1 deletion neon/hardware/compatible-devices/index.md
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
# Companion Device
The Companion device is a flagship Android smartphone. It is a regular phone that is not customized or modified in any way. To ensure maximum stability and performance we can only support a small number of carefully selected and tested models. The Neon Companion app is tuned to work with these particular models as we require full control over various low-level functions of the hardware.

The currently supported model is the Motorola Edge 40 Pro. Previously supported models are the OnePlus 8, OnePlus 8T, and OnePlus 10 Pro.
The supported models are: OnePlus 8, OnePlus 8T, OnePlus 10 Pro, and Motorola Edge 40 Pro. Currently, Neon ships with a Motorola Edge 40 Pro.

If you want to replace or add an extra Companion device you can purchase it [directly from us](https://pupil-labs.com/products/neon) or from any other distributor. The Neon Companion app is free and can be downloaded from the [Play Store](https://play.google.com/store/apps/details?id=com.pupillabs.neoncomp).

Expand All @@ -24,6 +24,8 @@ USB devices, rendering them incompatible with Neon.
If you purchased your device from a third-party distributor, you'll need to ensure that the correct Android version is installed. The currently
supported Android versions are as follows:

- On **OnePlus 8** and **8T**: Android 11
- On **OnePlus 10 Pro**: Android 12 and 13
- On **Moto Edge 40 Pro**: Android 13
- On **Moto Edge + (2023)**: Android 13
- On **Moto X40 Pro**: Android 13 (global ROM required)
Expand Down

0 comments on commit 20c920a

Please sign in to comment.