diff --git a/invisible/pupil-cloud/enrichments/create_enrichment.mp4 b/invisible/pupil-cloud/enrichments/create_enrichment.mp4 new file mode 100644 index 000000000..76d1ec959 Binary files /dev/null and b/invisible/pupil-cloud/enrichments/create_enrichment.mp4 differ diff --git a/invisible/pupil-cloud/enrichments/manual-mapper/image_1_mapper.png b/invisible/pupil-cloud/enrichments/manual-mapper/image_1_mapper.png index 5b2f0e8ae..9a640f594 100644 Binary files a/invisible/pupil-cloud/enrichments/manual-mapper/image_1_mapper.png and b/invisible/pupil-cloud/enrichments/manual-mapper/image_1_mapper.png differ diff --git a/invisible/pupil-cloud/enrichments/manual-mapper/image_2_mapper.png b/invisible/pupil-cloud/enrichments/manual-mapper/image_2_mapper.png index 7c112da34..a2423a6cf 100644 Binary files a/invisible/pupil-cloud/enrichments/manual-mapper/image_2_mapper.png and b/invisible/pupil-cloud/enrichments/manual-mapper/image_2_mapper.png differ diff --git a/invisible/pupil-cloud/enrichments/manual-mapper/manualmapper_header.mp4 b/invisible/pupil-cloud/enrichments/manual-mapper/manualmapper_header.mp4 new file mode 100644 index 000000000..45e4bcdbd Binary files /dev/null and b/invisible/pupil-cloud/enrichments/manual-mapper/manualmapper_header.mp4 differ diff --git a/invisible/pupil-cloud/enrichments/marker-mapper/index.md b/invisible/pupil-cloud/enrichments/marker-mapper/index.md index 48bc45759..c6e95b446 100644 --- a/invisible/pupil-cloud/enrichments/marker-mapper/index.md +++ b/invisible/pupil-cloud/enrichments/marker-mapper/index.md @@ -1 +1,117 @@ - +# Marker Mapper + +![Marker Mapper header image](./marker_mapper_header.png) + +The Marker Mapper enrichment enables tracking of where an individual is looking on a particular area or "surface" by positioning markers in the surrounding environment. This allows for the generation of a heatmap of gaze data directly within the Pupil Cloud enrichment, or for downloading the remapped gaze data in CSV format for further analysis. + +## Setup + +For robust detection, you should place enough markers on your surface such that at least 3 of them are visible whenever the surface is visible. You may also place markers inside the surface or outside the surface in close proximity to it. + +You can use these images as a template for printing markers: + +
+ + +![Apriltags 0 - 23](./apriltags_tag36h11_0-23.jpg) + + + + + ![Apriltags 24 - 47](./apriltags_tag36h11_24-47.jpg) + + +
+ +If you need more markers or higher resolution please see [here](https://github.com/pupil-labs/pupil-helpers/blob/master/markers_stickersheet/tag36h11_full.pdf?raw=True "PDF file with high-resolution markers."). + +::: warning +Note that the markers require a white border around them to be robustly detected. In our experience, this border should be at least equal to the width of the smallest white square or rectangle shown in the marker. Therefore, please make sure to include a sufficient border when displaying or printing them. +::: + +## Selecting Markers in the Cloud + + + +By default, all visible markers are used for surface definition when creating a surface. You can add or remove markers by clicking on them. Markers that are part of the definition are shown in green, while others are shown in red. + +Note that when adding a new marker to the surface definition, another marker that is already part of the definition has to be visible within the same video frame. + +A surface definition always needs to contain at least 2 markers. Thus, at least 2 markers have to be visible in the video frame when initially creating the surface, and you cannot remove further markers from the surface definition when the marker count is down to 2. + +![Marker Mapper marker selection](./marker_mapper_additional.png) + +## Surface Coordinates + +The Marker Mapper maps gaze points to a 2D surface and returns them in surface coordinates. The top left corner of the surface is defined as `(0, 0)`, and the bottom right corner is defined as `(1, 1)`. The orientation of the surface can be set in the enrichment settings. + +The mapper may return values outside of the surface, which yields values smaller than 0 or larger than 1, indicating that the corresponding gaze was not on the surface at that time. + +::: tip +**Orientation:** The red border marks the top edge of the surface. You can rotate the surface by clicking on "Rotate Surface" under the ellipsis next to 'Surface Defined'. +::: + +## Select image to be used in Marker Mapper visualizations + + + +You can select the image to be used in Marker Mapper visualizations. You can either pick a frame from a scene video, or you can upload a dedicated image that contains the surface. Images should contain the selected markers, and will be cropped according to the surface definition. The selected image will be used in the side-by-side view and as the background for AOI and heat map visualizations. + +## Validate the Mapping + +After the enrichment is successfully run, the recording timeline contains visualizations that indicate when surface was localized, and when gaze was on the surface. + + + +## Export Format + +### gaze.csv + +This file contains all the mapped gaze data from all sections. The coordinate system is explained [here](/pupil-cloud/enrichments/marker-mapper/#surface-coordinates). + +| Field | Description | +| ------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------ | +| **section id** | Unique identifier of the corresponding section. | +| **recording id** | Unique identifier of the recording this sample belongs to. | +| **timestamp [ns]** | UTC timestamp in nanoseconds of the sample. Equal to the timestamp of the original gaze sample before mapping. | +| **gaze detected on surface** | Boolean indicating whether the gaze point was inside or outside of the surface. | +| **gaze position on surface x [normalized]** | Float value representing the x-coordinate of the mapped gaze point in surface coordinates. If the surface was not localized this value is empty. | +| **gaze position on surface y [normalized]** | Same as gaze position on surface x [normalized] but for y-coordinate. | +| **fixation id** | If this gaze sample belongs to a fixation event, this is the corresponding id of the fixation. Otherwise, this field is empty. | +| **blink id** | If this gaze samples belongs to a blink event, this is the corresponding id of the blink. Otherwise this field is empty. | + +### fixations.csv + +This file contains fixation events detected in the gaze data stream and mapped to the surface. + +| Field | Description | +| -------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| **section id** | Unique identifier of the corresponding section. | +| **recording id** | Unique identifier of the recording this sample belongs to. | +| **fixation id** | Identifier of fixation within the section. The id corresponds to the fixation id of the raw unmapped data. | +| **start timestamp [ns]** | UTC timestamp in nanoseconds of the start of the fixation. | +| **end timestamp [ns]** | UTC timestamp in nanoseconds of the end of the fixation. | +| **duration [ms]** | Duration of the fixation in milliseconds. | +| **fixation detected on surface** | Boolean indicating whether the fixation was inside or outside of the surface. | +| **fixation x [normalized]** | Float value representing the x-coordinate of the fixation in surface coordinates. This position is the average of all mapped gaze samples within the fixation. | +| **fixation y [normalized]** | Same as "fixation x [normalized]" but for the y-coordinate. | + +### surface_positions.csv + +This file contains the surface locations in the scene images for all sections. + +| Field | Description | +| -------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | +| **section id** | Unique identifier of the corresponding section. | +| **timestamp [ns]** | UTC timestamp in nanoseconds of the sample. Equal to the timestamp of the scene camera frame the marker detection was executed on. | +| **detected markers** | A list of the markers detected in the corresponding scene camera frame. Markers are represented by their ID and the list is separated by `;`. Added in version 2 of this enrichment. | +| **tl x/y [px]** | x or y coordinate respectively of the **t**op **l**eft corner. Empty in case the surface could not be localized. Added in version 2 of this enrichment. | +| **tr x/y [px]** | x or y coordinate respectively of the **t**op **r**ight corner. Empty in case the surface could not be localized. Added in version 2 of this enrichment. | +| **br x/y [px]** | x or y coordinate respectively of the **b**ottom **r**ight corner. Empty in case the surface could not be localized. Added in version 2 of this enrichment. | +| **bl x/y [px]** | x or y coordinate respectively of the **b**ottom **l**eft corner. Empty in case the surface could not be localized. Added in version 2 of this enrichment. | diff --git a/invisible/pupil-cloud/enrichments/marker-mapper/mm_define.mp4 b/invisible/pupil-cloud/enrichments/marker-mapper/mm_define.mp4 new file mode 100644 index 000000000..036dff0c2 Binary files /dev/null and b/invisible/pupil-cloud/enrichments/marker-mapper/mm_define.mp4 differ diff --git a/invisible/pupil-cloud/enrichments/marker-mapper/mm_enrichment_viz.mp4 b/invisible/pupil-cloud/enrichments/marker-mapper/mm_enrichment_viz.mp4 new file mode 100644 index 000000000..b5f14245f Binary files /dev/null and b/invisible/pupil-cloud/enrichments/marker-mapper/mm_enrichment_viz.mp4 differ diff --git a/invisible/pupil-cloud/enrichments/marker-mapper/mm_imageupload.mp4 b/invisible/pupil-cloud/enrichments/marker-mapper/mm_imageupload.mp4 new file mode 100644 index 000000000..abc285d24 Binary files /dev/null and b/invisible/pupil-cloud/enrichments/marker-mapper/mm_imageupload.mp4 differ diff --git a/invisible/pupil-cloud/enrichments/reference-image-mapper/index.md b/invisible/pupil-cloud/enrichments/reference-image-mapper/index.md index 6ebac0d5d..df2456bda 100644 --- a/invisible/pupil-cloud/enrichments/reference-image-mapper/index.md +++ b/invisible/pupil-cloud/enrichments/reference-image-mapper/index.md @@ -1 +1,204 @@ - +# Reference Image Mapper + +Reference Image Mapper is a powerful tool to automatically map gaze onto features/objects in the environment. + + + +A heatmap of gaze data mapped onto the reference image can be generated, and mapped gaze and fixation data can further be downloaded as [CSV files](/pupil-cloud/enrichments/reference-image-mapper/#export-format). + +## Setup + + + +As described in the setup video, you will need two things in addition to your eye tracking recording(s) to produce a Reference Image Mapper enrichment: + +1. A reference image +2. A scanning video of the object/feature(s) taken with Neon’s scene camera + +:::tip +**Reference Image**
+Only the scanning recording needs to be taken with Neon. The reference image can be taken with any camera. It can be also be a scanned image. +
+
+Duration
+The **3 min** duration limit only applies to the scanning recording, the rest of your videos can be as long as you like.
But to avoid long processing times and get the best results, we recommend spliting the recordings into relevant parts where you expect the image to be mapped.
+::: + +In this guide, we will show you how the Reference Image Mapper can be used in a few different situations. This should help you get set up using the enrichment in your own testing environment! + +Below, gaze is mapped in four very different environments: to a **magazine cover**, a **basketball backboard**, a **supermarket shelf**, and even a **whole building**! + +![Reference images collage](./rim-collage.webp) + +Let's take a look at what the Reference Image and Scanning Recording look like to produce these heatmaps. + +### 1. Magazine Reading + +
+
+

Reference image

+ Magazine Reference Image +
First, we need a high-resolution .jpeg of the page.
+
+
+

Scanning video

+
+ +
+
+ Then, we need a scanning recording, about 15 s long, taken on a blank background and in good lighting (natural light works well). Note that the magazine page is clear with good contrast +
+
+
+ +### 2. Basketball + +
+
+

Reference image

+ Basketball Reference Image +
Here we can take a photo of the basketball hoop and court background.
+
+
+

Scanning video

+
+ +
+
+ The scanning recording, in this case, is about 45 s long – note that the angles and distances cover what a player might see when dribbling towards the basket or taking a shot +
+
+
+ +### 3. Supermarket Shelf + +
+
+

Reference image

+ Supermarket Reference Image +
This photo captures the assortment of packagíng in the coffee aisle of a supermarket.
+
+
+

Scanning video

+
+ +
+
+ Notice that the scanning recording is fairly slow to reduce motion blur for this feature-rich shelf. +
+
+
+ +### 4. An Entire Building + +
+
+

Reference image

+ Building Reference Image +
This is a photo of the entire building
+
+
+

Scanning video

+
+ +
+
+ We take a longer scanning recording, about 2 min. The angles and distances cover what a person might see whilst walking past or standing in front of the building. +
+
+
+ +## Scanning Best Practices + +Try to follow these best practices when recording a scanning video: + +- Make the recording while holding Neon in your hand rather than wearing it on your head. +- Record the object of interest from all possible angles and from all distances a subject may view them. More diversity is better. Collecting sufficiently diverse viewpoints for a large object, like a building, may require you to move accordingly large distances. +- Move the glasses slowly while recording to avoid motion blur. +- Make sure to have good contrast and that your scene lighting during scanning is similar to that during mapping. + +What works and what doesn’t? + +- The scene must be **feature-rich** like in the examples above. There has to be enough salient visual content to produce a successful mapping. +- The scene needs to have relatively **static** features in the environment. If there is a lot of movement or the objects change in appearance or shape, the mapping _can_ fail. + +::: tip +**Ready to go?**
+Why not try replicating the above examples? Or even try it with your own use cases! If you haven’t already, we recommend you check out the [Cloud Getting Started Guide](/pupil-cloud/tutorials/), which covers the basics of working with enrichments. +::: + +## Validate the Gaze Mapping + + + +To check if gaze has been mapped successfully, use the side-by-side view: + +1. Select a recording. +2. Select the Reference Image Mapper Enrichment. +3. Select the Scene / Reference Image View. + + +Now when you play back the recording you can see where gaze is mapped to on your reference image for validation. + +## Occlusions + +![Basketball Occlusion](./basketball-occlusion.png) + +Sometimes an object will occlude the feature/object of interest. The reference image mapper may not recognize this, so a false positive mapping could occur. + +If you need to remove the falsely mapped data points, there are a few workarounds. + +1. Use Events and Sections to trim out false positive sections. +2. Manually remove the affected data points in the reference image mapper export, by finding the timestamp and deleting the row in the .csv export. + +## Repetitions + +In cases such as supermarket shopping, where features of the environment like freezers and aisles are repetitive and overlapping, it can be useful to divide recordings into shorter sections using [Events](/data-collection/events/) for enrichment. This way you can ensure gaze is only mapped to portions of the recording during which you know the user is looking at a particular part of the store. + +::: tip +**Want to know more?**
+Under the hood, the Reference Image Mapper uses a method called Structure from Motion (SfM) to build a model of the environment. You can see this model as a ‘point cloud’ on the left side video. +::: + +## Export Format + +### gaze.csv + +This file contains all the mapped gaze data from all sections. + +| Field | Description | +| ------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | +| **section id** | Unique identifier of the corresponding section. | +| **recording id** | Unique identifier of the recording this sample belongs to. | +| **timestamp [ns]** | UTC timestamp in nanoseconds of the sample. Equal to the timestamp of the original gaze sample before mapping. | +| **gaze detected in reference image** | Boolean indicating whether the gaze point was detected inside or outside of the reference image. | +| **gaze position in reference image x [px]** | Float value representing the x-coordinate of the mapped gaze point in pixel coordinates. If the reference image was not detected in the scene at the given time this value is empty. | +| **gaze position in reference image y [px]** | Same as "gaze position in reference image x [px]" but for the y-coordinate. | +| **fixation id** | If this gaze sample belongs to a fixation event, this is the corresponding id of the fixation. Otherwise, this field is empty. | +| **blink id** | If this gaze samples belongs to a blink event, this is the corresponding id of the blink. Otherwise this field is empty. | + +::: info +This CSV file only contains data-points where the reference image has been localised in the scene. Looking for all the gaze points? Check [this file.](/data-collection/data-format/#gaze-csv) +::: + +### fixations.csv + +This file contains fixation events detected in the gaze data stream and mapped to the reference image. + +| Field | Description | +| ---------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| **section id** | Unique identifier of the corresponding section. | +| **recording id** | Unique identifier of the recording this sample belongs to. | +| **fixation id** | Identifier of fixation within the section. The id corresponds to the fixation id of the raw unmapped data. | +| **start timestamp [ns]** | UTC timestamp in nanoseconds of the start of the fixation. | +| **end timestamp [ns]** | UTC timestamp in nanoseconds of the end of the fixation. | +| **duration [ms]** | Duration of the fixation in milliseconds. | +| **fixation detected in reference image** | Boolean indicating whether the fixation was inside or outside of the reference image. | +| **fixation x [px]** | Float value representing the x-coordinate of the fixation in reference image coordinates. This position is the average of all mapped gaze samples within the fixation. | +| **fixation y [px]** | Same as "fixation x [px]" but for the y-coordinate. | + +### Reference Image + +The reference image that was used for defining the enrichment. The file is named `reference_image.jpeg|png`. diff --git a/invisible/pupil-cloud/enrichments/reference-image-mapper/validate_rim.mp4 b/invisible/pupil-cloud/enrichments/reference-image-mapper/validate_rim.mp4 new file mode 100644 index 000000000..44a8f2a9f Binary files /dev/null and b/invisible/pupil-cloud/enrichments/reference-image-mapper/validate_rim.mp4 differ