Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merge master to GH #642

Merged
merged 5 commits into from
Nov 28, 2023
Merged
Changes from 2 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 1 addition & 4 deletions components/Youtube.vue
Original file line number Diff line number Diff line change
@@ -20,10 +20,7 @@
</script>

<template>
<div
class="pb-4"
style="display: flex; justify-content: center; aspect-ratio: 16 / 9"
>
<div style="display: flex; justify-content: center; aspect-ratio: 16 / 9">
<iframe
width="100%"
height="height"
4 changes: 4 additions & 0 deletions custom.css
Original file line number Diff line number Diff line change
@@ -561,6 +561,10 @@
background-color: var(--vp-c-default-1);
}

.subgrid {
grid-template-rows: subgrid;
}

@media (min-width: 768px) {
#app .VPNavScreen {
display: unset;
Binary file not shown.
7 changes: 7 additions & 0 deletions invisible/tailwind.config.js
Original file line number Diff line number Diff line change
@@ -14,4 +14,11 @@ module.exports = {
},
},
plugins: [],
safelist: [
{
pattern: /grid-cols-(1|2|3|4|5)/,
},
"^sm:grid-cols-",
"m-auto",
],
};
155 changes: 81 additions & 74 deletions neon/pupil-cloud/enrichments/reference-image-mapper/index.md
Original file line number Diff line number Diff line change
@@ -4,14 +4,14 @@ Reference Image Mapper is a powerful tool to automatically map gaze onto feature

<Youtube src="IF8f1Z3ZkEo" muted="1"/>


A heatmap of gaze data mapped onto the reference image can be generated, and mapped gaze and fixation data can further be downloaded as [CSV files](/pupil-cloud/enrichments/reference-image-mapper/#export-format).
A heatmap of gaze data mapped onto the reference image can be generated, and mapped gaze and fixation data can further be downloaded as [CSV files](/pupil-cloud/enrichments/reference-image-mapper/#export-format).

## Setup

<Youtube src="ygqzQEzUIS4"/>

As described in the setup video, you will need two things in addition to your eye tracking recording(s) to produce a Reference Image Mapper enrichment:

1. A reference image
2. A scanning video of the object/feature(s) taken with Neon’s scene camera

@@ -28,93 +28,99 @@ In this guide, we will show you how the Reference Image Mapper can be used in a

Below, gaze is mapped in four very different environments: to a **magazine cover**, a **basketball backboard**, a **supermarket shelf**, and even a **whole building**!

<PhotoGrid :images="[
`./heatmap-0.jpg`,
`./heatmap-1.jpg`,
`./heatmap-2.jpg`,
`./heatmap-3.jpg`,
]"/>
![Reference images collage](./rim-collage.webp)

Let's take a look at what the Reference Image and Scanning Recording look like to produce these heatmaps.
### 1. Magazine Reading

<div class="grid grid-cols-2 gap-4">
<div>
<h4>Reference image</h4>

![Magazine Reference Image](./magazine-img.jpg)
### 1. Magazine Reading

First, we need a high-resolution .jpeg of the page.
<div class="grid grid-cols-1 sm:grid-cols-2 gap-4">
<div class="grid subgrid" style="grid-row: 1 / 4">
<h4>Reference image</h4>
<img src="./magazine-img.jpg" alt="Magazine Reference Image">
<div>First, we need a high-resolution .jpeg of the page.</div>
</div>
<div>
<div class="grid subgrid" style="grid-row: 1 / 4">
<h4>Scanning video</h4>
<Youtube src="SplYZK-ZzE8"/>
Then, we need a scanning recording, about <i>15 s</i> long, taken on a blank background and in good lighting (natural light works well). Note that the magazine page is clear with good contrast
<div class="m-auto">
<Youtube src="SplYZK-ZzE8"/>
</div>
<div>
Then, we need a scanning recording, about <i>15 s</i> long, taken on a blank background and in good lighting (natural light works well). Note that the magazine page is clear with good contrast
</div>
</div>
</div>


### 2. Basketball

<div class="grid grid-cols-2 gap-4">
<div>
<h4>Reference image</h4>

![Basketball Reference Image](./basketball-img.jpg)

Here we can take a photo of the basketball hoop and court background.
<div class="grid subgrid" style="grid-row: 1 / 4">
<h4>Reference image</h4>
<img src="./basketball-img.jpg" alt="Basketball Reference Image">
<div>Here we can take a photo of the basketball hoop and court background.</div>
</div>
<div>
<div class="grid subgrid" style="grid-row: 1 / 4">
<h4>Scanning video</h4>
<Youtube src="4MB5o4W_XJo"/>
The scanning recording, in this case, is about 45 s long – note that the angles and distances cover what a player might see when dribbling towards the basket or taking a shot
<div class="m-auto">
<Youtube src="4MB5o4W_XJo"/>
</div>
<div>
The scanning recording, in this case, is about 45 s long – note that the angles and distances cover what a player might see when dribbling towards the basket or taking a shot
</div>
</div>
</div>

### 3. Supermarket Shelf

<div class="grid grid-cols-2 gap-4">
<div>
<h4>Reference image</h4>

![Supermarket Reference Image](./supermarket-img.jpg)

This photo captures the assortment of packagíng in the coffee aisle of a supermarket.
<div class="grid subgrid" style="grid-row: 1 / 4">
<h4>Reference image</h4>
<img src="./supermarket-img.jpg" alt="Supermarket Reference Image">
<div>This photo captures the assortment of packagíng in the coffee aisle of a supermarket.</div>
</div>
<div>
<div class="grid subgrid" style="grid-row: 1 / 4">
<h4>Scanning video</h4>
<Youtube src="d3Yk3nKDIOQ"/>
Notice that the scanning recording is fairly slow to reduce motion blur for this feature-rich shelf.
<div class="m-auto">
<Youtube src="d3Yk3nKDIOQ"/>
</div>
<div>
Notice that the scanning recording is fairly slow to reduce motion blur for this feature-rich shelf.
</div>
</div>
</div>

### 4. An Entire Building

<div class="grid grid-cols-2 gap-4">
<div>
<h4>Reference image</h4>

![Building Reference Image](./building-img.jpg)

This is a photo of the <i>entire</i> building
<div class="grid subgrid" style="grid-row: 1 / 4">
<h4>Reference image</h4>
<img src="./building-img.jpg" alt="Building Reference Image">
<div>This is a photo of the <i>entire</i> building</div>
</div>
<div>
<div class="grid subgrid" style="grid-row: 1 / 4">
<h4>Scanning video</h4>
<Youtube src="0U4H-uOIHlw"/>
We take a longer scanning recording, about 2 min. The angles and distances cover what a person might see whilst walking past or standing in front of the building.
<div class="m-auto">
<Youtube src="0U4H-uOIHlw"/>
</div>
<div>
We take a longer scanning recording, about 2 min. The angles and distances cover what a person might see whilst walking past or standing in front of the building.
</div>
</div>
</div>

## Scanning Best Practices

Try to follow these best practices when recording a scanning video:

- Make the recording while holding Neon in your hand rather than wearing it on your head.
- Record the object of interest from all possible angles and from all distances a subject may view them. More diversity is better. Collecting sufficiently diverse viewpoints for a large object, like a building, may require you to move accordingly large distances.
- Move the glasses slowly while recording to avoid motion blur.
- Make sure to have good contrast and that your scene lighting during scanning is similar to that during mapping.

What works and what doesn’t?

- The scene must be **feature-rich** like in the examples above. There has to be enough salient visual content to produce a successful mapping.
- The scene needs to have relatively **static** features in the environment. If there is a lot of movement or the objects change in appearance or shape, the mapping *can* fail.
- The scene needs to have relatively **static** features in the environment. If there is a lot of movement or the objects change in appearance or shape, the mapping _can_ fail.

::: tip
**Ready to go?**<br>
@@ -126,6 +132,7 @@ Why not try replicating the above examples? Or even try it with your own use cas
![Reference Image Mapper in Pupil Cloud](./rim-in-cloud.png)

To check if gaze has been mapped successfully, use the side-by-side view:

1. Select a recording.
2. Select the Reference Image Mapper Enrichment.
3. Select the Scene / Reference Image View.
@@ -137,17 +144,16 @@ Now when you play back the recording you can see where gaze is mapped to on your

![Basketball Occlusion](./basketball-occlusion.png)


Sometimes an object will occlude the feature/object of interest. The reference image mapper may not recognize this, so a false positive mapping could occur.
Sometimes an object will occlude the feature/object of interest. The reference image mapper may not recognize this, so a false positive mapping could occur.

If you need to remove the falsely mapped data points, there are a few workarounds.

1. Use Events and Sections to trim out false positive sections.
2. Manually remove the affected data points in the reference image mapper export, by finding the timestamp and deleting the row in the .csv export.

## Repetitions
In cases such as supermarket shopping, where features of the environment like freezers and aisles are repetitive and overlapping, it can be useful to divide recordings into shorter sections using [Events](/data-collection/events/) for enrichment. This way you can ensure gaze is only mapped to portions of the recording during which you know the user is looking at a particular part of the store.

In cases such as supermarket shopping, where features of the environment like freezers and aisles are repetitive and overlapping, it can be useful to divide recordings into shorter sections using [Events](/data-collection/events/) for enrichment. This way you can ensure gaze is only mapped to portions of the recording during which you know the user is looking at a particular part of the store.

::: tip
**Want to know more?**<br>
@@ -157,35 +163,36 @@ Under the hood, the Reference Image Mapper uses a method called Structure from M
## Export Format

### gaze.csv
This file contains all the mapped gaze data from all sections.

This file contains all the mapped gaze data from all sections.

| Field | Description |
| -------- | -------- |
| **section id** | Unique identifier of the corresponding section. |
| **recording id** | Unique identifier of the recording this sample belongs to. |
| **timestamp [ns]** | UTC timestamp in nanoseconds of the sample. Equal to the timestamp of the original gaze sample before mapping. |
| **gaze detected in reference image** | Boolean indicating whether or not the gaze point was detected inside or outside of the reference image. |
| **gaze position in reference image x [px]** | Float value representing the x-coordinate of the mapped gaze point in pixel coordinates. If the reference image was not detected in the scene at the given time this value is empty. |
| **gaze position in reference image y [px]** | Same as "gaze position in reference image x [px]" but for the y-coordinate. |
| **fixation id** | If this gaze sample belongs to a fixation event, this is the corresponding id of the fixation. Otherwise, this field is empty. |
| **blink id** | If this gaze samples belongs to a blink event, this is the corresponding id of the blink. Otherwise this field is empty. |
| Field | Description |
| ------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| **section id** | Unique identifier of the corresponding section. |
| **recording id** | Unique identifier of the recording this sample belongs to. |
| **timestamp [ns]** | UTC timestamp in nanoseconds of the sample. Equal to the timestamp of the original gaze sample before mapping. |
| **gaze detected in reference image** | Boolean indicating whether or not the gaze point was detected inside or outside of the reference image. |
| **gaze position in reference image x [px]** | Float value representing the x-coordinate of the mapped gaze point in pixel coordinates. If the reference image was not detected in the scene at the given time this value is empty. |
| **gaze position in reference image y [px]** | Same as "gaze position in reference image x [px]" but for the y-coordinate. |
| **fixation id** | If this gaze sample belongs to a fixation event, this is the corresponding id of the fixation. Otherwise, this field is empty. |
| **blink id** | If this gaze samples belongs to a blink event, this is the corresponding id of the blink. Otherwise this field is empty. |

### fixations.csv
This file contains fixation events detected in the gaze data stream and mapped to the reference image.

This file contains fixation events detected in the gaze data stream and mapped to the reference image.

| Field | Description |
| -------- | -------- |
| **section id** | Unique identifier of the corresponding section. |
| **recording id** | Unique identifier of the recording this sample belongs to. |
| **fixation id** | Identifier of fixation within the section. The id corresponds to the fixation id of the raw unmapped data. |
| **start timestamp [ns]** | UTC timestamp in nanoseconds of the start of the fixation. |
| **end timestamp [ns]** | UTC timestamp in nanoseconds of the end of the fixation. |
| **duration [ms]** | Duration of the fixation in milliseconds. |
| **fixation detected in reference image** | Boolean indicating whether or not the fixation was inside or outside of the reference image. |
| **fixation x [px]** | Float value representing the x-coordinate of the fixation in reference image coordinates. This position is the average of all mapped gaze samples within the fixation. |
| **fixation y [px]** | Same as "fixation x [px]" but for the y-coordinate. |
| Field | Description |
| ---------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **section id** | Unique identifier of the corresponding section. |
| **recording id** | Unique identifier of the recording this sample belongs to. |
| **fixation id** | Identifier of fixation within the section. The id corresponds to the fixation id of the raw unmapped data. |
| **start timestamp [ns]** | UTC timestamp in nanoseconds of the start of the fixation. |
| **end timestamp [ns]** | UTC timestamp in nanoseconds of the end of the fixation. |
| **duration [ms]** | Duration of the fixation in milliseconds. |
| **fixation detected in reference image** | Boolean indicating whether or not the fixation was inside or outside of the reference image. |
| **fixation x [px]** | Float value representing the x-coordinate of the fixation in reference image coordinates. This position is the average of all mapped gaze samples within the fixation. |
| **fixation y [px]** | Same as "fixation x [px]" but for the y-coordinate. |

### Reference Image
The reference image that was used for defining the enrichment. The file is named `reference_image.jpeg|png`.

The reference image that was used for defining the enrichment. The file is named `reference_image.jpeg|png`.
Binary file not shown.
7 changes: 7 additions & 0 deletions neon/tailwind.config.js
Original file line number Diff line number Diff line change
@@ -14,4 +14,11 @@ module.exports = {
},
},
plugins: [],
safelist: [
{
pattern: /grid-cols-(1|2|3|4|5)/,
},
"^sm:grid-cols-",
"m-auto",
],
};

Unchanged files with check annotations Beta

# A practical guide to implementing gaze contingency for assistive technology
<TagLinks :tags="$frontmatter.tags" />

Check warning on line 27 in alpha-lab/gaze-contingency-assistive/index.md

GitHub Actions / ✍️ Check spelling

Unknown word (frontmatter)
<Youtube src="cuvWqVOAc5M"/>
::: tip
Imagine a world where transformative assistive solutions enable you to browse the internet with a mere glance or write an email using only your eyes. This is not science fiction; it is the realm of gaze-contingent technology.

Check warning on line 32 in alpha-lab/gaze-contingency-assistive/index.md

GitHub Actions / ✍️ Check spelling

Unknown word (transformative)
:::
## Hacking the eyes with gaze contingency
To locate the screen, we use [AprilTags](https://april.eecs.umich.edu/software/apriltag) to identify the image of the
screen as it appears in Neon’s scene camera. Gaze data is transferred to the computer via Neon's
[Real-time API](https://docs.pupil-labs.com/neon/real-time-api/tutorials/). We then transform gaze from _scene camera_ to _screen-based_
coordinates using a [homography](<https://en.m.wikipedia.org/wiki/Homography_(computer_vision)>) approach like the [Marker Mapper](https://docs.pupil-labs.com/neon/pupil-cloud/enrichments/marker-mapper/)

Check warning on line 57 in alpha-lab/gaze-contingency-assistive/index.md

GitHub Actions / ✍️ Check spelling

Unknown word (homography)
enrichment we offer in Pupil Cloud as a post-hoc solution. The heavy lifting of all this is handled by
our [Real-time Screen Gaze](https://github.com/pupil-labs/realtime-screen-gaze/) package (written for this guide).
import TagLinks from '@components/TagLinks.vue'
</script>
# Map and visualise gaze onto a display content using the Reference Image Mapper

Check warning on line 11 in alpha-lab/map-your-gaze-to-a-2d-screen/index.md

GitHub Actions / ✍️ Check spelling

Unknown word (visualise)
<TagLinks :tags="$frontmatter.tags" />

Check warning on line 13 in alpha-lab/map-your-gaze-to-a-2d-screen/index.md

GitHub Actions / ✍️ Check spelling

Unknown word (frontmatter)
<Youtube src="OXIUjIzCplc"/>

Check warning on line 15 in alpha-lab/map-your-gaze-to-a-2d-screen/index.md

GitHub Actions / ✍️ Check spelling

Unknown word (Cplc)
In this guide, we will show you how to map and visualise gaze onto a screen with dynamic content, e.g. a video, web browsing or any other content of your choice, using the [Reference Image Mapper](https://docs.pupil-labs.com/neon/pupil-cloud/enrichments/reference-image-mapper/) enrichment and a few clicks.

Check warning on line 17 in alpha-lab/map-your-gaze-to-a-2d-screen/index.md

GitHub Actions / ✍️ Check spelling

Unknown word (visualise)
::: tip
**Note:** This tutorial requires some technical knowledge, but don't worry. We made it almost click and run for you! You can learn as much or as little as you like.
## Making the recording
Let's assume you have everything ready to go – your participant is sat infront of the screen wearing the eye tracker, your screen content is ready to play.

Check warning on line 40 in alpha-lab/map-your-gaze-to-a-2d-screen/index.md

GitHub Actions / ✍️ Check spelling

Unknown word (infront)
So that we can capture your participant's visual interactions with the screen content, we will need to make sure that both the _eye tracking_ **and** _screen recordings_ happen at the same time.
Importantly, both sources (eye tracking and screen recording) record individually. As such, you'll need what we call an [event annotation](https://docs.pupil-labs.com/neon/data-collection/events/) to synchronise them later.

Check warning on line 44 in alpha-lab/map-your-gaze-to-a-2d-screen/index.md

GitHub Actions / ✍️ Check spelling

Unknown word (synchronise)
The [event annotation](https://docs.pupil-labs.com/neon/data-collection/events/) should be used to indicate the beginning of the _screen content recording_ in the _eye tracking recording_, and be named `start.video`.
Check [here](https://docs.pupil-labs.com/neon/data-collection/events/) how you can create these events in the Cloud.
::: warning
**Tip:**
You might find some libav.mp4 warnings. These warnings are due to some issues with the aac codec and timestamping. These messages only appear when adding the audio stream. You can safely ignore them, or you can disable audio within the code.

Check warning on line 97 in alpha-lab/map-your-gaze-to-a-2d-screen/index.md

GitHub Actions / ✍️ Check spelling

Unknown word (libav)
:::
## How the code works