Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

RIM living room and scanpath tutorial revamp #622

Closed
wants to merge 8 commits into from
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 10 additions & 1 deletion .lycheeignore
Original file line number Diff line number Diff line change
Expand Up @@ -85,10 +85,19 @@ https://docs.pupil-labs.com/eEl3sswsTms
https://docs.pupil-labs.com/aPLnqu26tWI
https://docs.pupil-labs.com/_Jnxi1OMMTc
https://docs.pupil-labs.com/uoq11XtNH5E
https://docs.pupil-labs.com/BZoO7zxHaiw?si=I3zF-EV4O-ancTY0
https://docs.pupil-labs.com/jeL8gs053lg?si=6wlx4fjxlfiqrbRq
https://docs.pupil-labs.com/zksTzVkGifk?si=3bxl0eKOgRbfoes-
https://docs.pupil-labs.com/Bg_SiFByceY?si=d2koC7-V7bbrYL3h
https://docs.pupil-labs.com/0r8oAn2AZMQ?si=SbSVHedGTJ4Zshfw
https://docs.pupil-labs.com/fmy9F8Q9eW0?si=F7q399iZHGW2kArv
https://docs.pupil-labs.com/7V3X4XmbRAM
https://docs.pupil-labs.com/X43aTIRjwgQ?si=aTzAkRrYNqdOEf0T

# TCP
https://docs.pupil-labs.com/f'tcp://%7Bip%7D:%7Bpub_port%7D
https://docs.pupil-labs.com/f'tcp://%7Bip%7D:%7Bport%7D
https://docs.pupil-labs.com/f'tcp://%7Bip%7D:%7Bsub_port%7D
# Twitter as it seems to timeout
https://twitter.com/pupil_labs
https://twitter.com/
https://twitter.com/
2 changes: 1 addition & 1 deletion src/alpha-lab/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -68,7 +68,7 @@
} else {
// If it's not December, pick a random banner from the 7 available banners
const numberOfImages = 7;
const randomImageNumber = [Math.trunc(Math.random() * numberOfImages)+1];

Check warning on line 71 in src/alpha-lab/README.md

View workflow job for this annotation

GitHub Actions / ✍️ Check spelling

Unknown word (trunc)
const randomImage = `img${randomImageNumber.toString()}.png`;
const randomImageAltText = "img"+randomImageNumber.toString();
return {
Expand All @@ -82,9 +82,9 @@
return {
showTell: [
{
title: "AOIs",

Check warning on line 85 in src/alpha-lab/README.md

View workflow job for this annotation

GitHub Actions / ✍️ Check spelling

Unknown word (AOIs)
text: "Here we demonstrate how to make areas of interest using data downloaded from Pupil Cloud’s Reference Image Mapper.",
to: "/alpha-lab/gaze-metrics-in-aois/",

Check warning on line 87 in src/alpha-lab/README.md

View workflow job for this annotation

GitHub Actions / ✍️ Check spelling

Unknown word (aois)
img: "reference-aoi.jpg",
},
{
Expand All @@ -96,15 +96,15 @@
},
{
title: "RIM Room",
text: "We pushed the limits of markerless mapping with Pupil Cloud’s Reference Image Mapper - scanning an entire apartment.",

Check warning on line 99 in src/alpha-lab/README.md

View workflow job for this annotation

GitHub Actions / ✍️ Check spelling

Unknown word (markerless)
to: "/alpha-lab/multiple-rim/",
img: "desk-overlay.png",
img: "desk-heatmap.jpeg",
},
{
title: "Look at my hand!",
text: "Use detectron's densepose AI to segment and know at which part of a body a person is looking.",

Check warning on line 105 in src/alpha-lab/README.md

View workflow job for this annotation

GitHub Actions / ✍️ Check spelling

Unknown word (detectron's)

Check warning on line 105 in src/alpha-lab/README.md

View workflow job for this annotation

GitHub Actions / ✍️ Check spelling

Unknown word (densepose)
to: "/alpha-lab/dense-pose/",
img: "densepose.png",

Check warning on line 107 in src/alpha-lab/README.md

View workflow job for this annotation

GitHub Actions / ✍️ Check spelling

Unknown word (densepose)
},
{
title: "Follow my path",
Expand Down
247 changes: 194 additions & 53 deletions src/alpha-lab/multiple-rim.md

Large diffs are not rendered by default.

167 changes: 58 additions & 109 deletions src/alpha-lab/scanpath-rim.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,123 +13,72 @@ tags: [Pupil Invisible, Neon, Cloud]
</div>
<br>

The [Reference Image Mapper](/enrichments/reference-image-mapper/) is a powerful tool that maps gaze onto 2D
images of real-world environments and generates heatmaps. Now, we offer a new way to visualize your Reference Image Mapper
data. We have created a ready-to-use script that generates static and dynamic scanpaths, providing deeper insights into
patterns of visual behavior.

::: tip
Before continuing, ensure you are familiar with the [Reference Image Mapper](/enrichments/reference-image-mapper)
enrichment. Check out [this explainer video](https://www.youtube.com/watch?v=ygqzQEzUIS4&t=56s) for reference.
Picture this: Build and customise scanpath visualisations with your Reference Image Mapper exports!
:::

## What is a scanpath?
A scanpath is a graphical representation of an individual's gaze movements. It shows the sequence of fixations, or pauses
in gaze, and the rapid eye movements made between fixations, known as saccades. The scanpath offers a glimpse into what
the observer is focusing on and the duration and frequency of their attention to different aspects of the scene. This
information is a valuable tool for understanding a person's visual attention and perception.

<img src="../media/alpha-lab/Steven_scanpath.jpeg"/>

## What you'll need:
- A Reference Image Mapper export download
- Python 3.7 or higher
- [This](https://gist.github.com/elepl94/9f669c4d81e455cf2095957831219664) ready-to-go script

## Running the code
All you need to do is run the command `python3 RIM_scanpath.py` in your terminal. A prompt will then appear asking for
the location of the Reference Image Mapper export folder. After this, just sit back and wait for the processing to finish.
Upon completion, the resulting scanpath visualisations will be saved in a newly created sub-folder called "scanpath”.

If you wish to enhance the appearance of your scanpaths, keep reading for additional instructions!

## Personalization
### To each their own color
<p>
This function generates random colors for each participant based on their names.

```python
def color_generator(...):
colors = {
subj: (
random.randint(0, 255),
random.randint(0, 255),
random.randint(0, 255),
)
for subj in names
}

return colors
```
</p>
<p>
However, if you prefer to assign specific colors to each participant, you can easily modify the function to suit your needs. An example could be:
## Unlocking visual exploration with scanpaths
A scanpath is a graphical representation of an individual's gaze. It shows the sequence of fixations (pauses in gaze), and rapid eye movements made between fixations (saccades). Scanpaths offer a glimpse into how the observer has focused their attention on different aspects of the scene, which is a valuable tool for understanding a person's visual attention and perception.

``` python
def color_generator():
colors = {}
colors['Subject1'] = (0, 0, 255)
colors['Subject2'] = (255, 0, 0)
colors['Subject3'] = (0, 255, 0)

return colors
```
</p>
In this guide, we will show you how to generate both static and dynamic scanpath visualisations using your Reference Image Mapper exported data, like in the video above.

### Make it font-tastic
If you have a preferred font or would like to change the size, simply edit the draw_on_frame() function. The fixation
IDs are displayed in black text with a white border to make them stand out from the background. If you adjust the font
size, it's also recommended to increase the values of `font_thick_w` and `font_thick_b` to maintain visual contrast.
``` python
def draw_on_frame(...):
# text aesthetics
font = cv2.FONT_HERSHEY_DUPLEX
font_size = 1
font_thick_w = 3
font_thick_b = 1
...
```
### My name is legend
The script includes two functions for creating a legend to display the wearer names and corresponding colors:

1. `draw_name_legend()`: This function creates a legend box that displays only the name of the wearer on their individual scanpath video and image.
2. `draw_all_names_legend()`: This function creates a legend that displays all the wearer names on the final general scanpath image.

To customize the appearance of the legend, such as the position, dimensions, or colors of the rectangular white box or the colored line,
you can modify the following parameters in both functions:
::: tip
Before continuing, ensure you are familiar with the [Reference Image Mapper](/enrichments/reference-image-mapper)
enrichment. Check out [this explainer video](https://www.youtube.com/watch?v=ygqzQEzUIS4&t=56s) for reference.
:::

- `r_end_point` - x and y values of the ending coordinates of the rectangular legend box
- `r_start_point` - x and y values of the starting coordinates of the rectangular legend box
- `l_end_point` - x and y values of the ending coordinates of the colored line
- `l_start_point` - x and y values of the starting coordinates of the colored line
- In `cv2.rectangle`, edit `color` to set a new color for the legend box
- In `cv2.line`, edit `thickness` to set a new width for the colored line
## Building the visualisations in an offline context
The [Reference Image Mapper](/enrichments/reference-image-mapper) available in Pupil Cloud is a tool that maps gaze onto 2D images and can subsequently generate heatmaps. However, it currently does not support the production of scanpath visualizations. Since scanpaths provide a useful characterization of *where*, *when*, and *how long* attention was focused on various elements, we developed a script that enables you to generate both static and dynamic scanpaths using your Reference Image Mapper data exported from Pupil Cloud.

## Steps
1. Run a [Reference Image Mapper](https://docs.pupil-labs.com/enrichments/reference-image-mapper/) enrichment and download the results
2. Download [this](https://gist.github.com/elepl94/9f669c4d81e455cf2095957831219664) gist and follow the instructions in the [readme](https://gist.github.com/elepl94/9f669c4d81e455cf2095957831219664#file-readme-md)

## Final results
After the script has completed its execution, you'll find the resulting scanpath visualizations stored in a newly created sub-folder named "scanpath." For each participant, you will obtain a reference image with the scanpath superimposed on it, along with a video that illustrates gaze behavior on the reference image, featuring a dynamic scanpath overlay. Furthermore, an aggregated visualization, combining all participants' scanpaths, will be at your disposal, providing the opportunity for more comprehensive and in-depth analyses.

<div style="display: flex; justify-content: space-between;">
<div style="flex: 1; margin-right: 10px;">
<div style="width: 100%; text-align: center;">
<img src="../media/alpha-lab/Jack_scanpath.jpeg" alt="Jack Scanpath" style="width: 100%; height: 100%;">
</div>
</div>
<div style="flex: 1;">
<div class="iframe-container2" style="width: 100%;">
<div style="width: 100%; text-align: center;">
<iframe width="100%" height="100%" src="https://www.youtube.com/embed/X43aTIRjwgQ?si=aTzAkRrYNqdOEf0T" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>
</div>
</div>
</div>
</div>

<div style="width: 100%; text-align: center;">
<img src="../media/alpha-lab/general_scanpath.jpeg" alt="General Scanpath" style="width: 50%; height: 50%;">
</div>

<style scoped>
img, iframe {
width: 100%;
height: 100%;
object-fit: contain;
box-sizing: border-box;
}

img {
border-radius: 10px;
max-width: 100%;
height: auto;
box-sizing: border-box;
}

.iframe-container2{
position: relative;
width: 100%;
padding-bottom: 75%;
margin-bottom: 10px;
height: 0;
margin-left:0;
margin-right:0;
}

.iframe-container2 iframe{
position: absolute;
top:0;
left: 0;
width: 100%;
height: 100%;
}
.iframe-container2 {
position: relative;
width: 100%;
padding-bottom: 75%;
margin-bottom: 10px;
height: 0;
margin-left: 0;
margin-right: 0;
}

.iframe-container2 iframe {
position: absolute;
top: 0;
left: 0;
width: 100%;
height: 100%;
}
</style>
Binary file added src/media/alpha-lab/Jack_scanpath.jpeg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added src/media/alpha-lab/cupboard-heatmap.jpeg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file removed src/media/alpha-lab/cupboard-img.png
Binary file not shown.
Binary file removed src/media/alpha-lab/cupboard-overlay.png
Binary file not shown.
Binary file added src/media/alpha-lab/cupboard.jpeg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added src/media/alpha-lab/desk-heatmap.jpeg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file removed src/media/alpha-lab/desk-img.png
Binary file not shown.
Binary file removed src/media/alpha-lab/desk-overlay.png
Binary file not shown.
Binary file added src/media/alpha-lab/desk.jpeg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added src/media/alpha-lab/general_scanpath.jpeg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file removed src/media/alpha-lab/kitchen+table-img.jpeg
Binary file not shown.
Binary file removed src/media/alpha-lab/kitchen+table-overlay.png
Binary file not shown.
Binary file added src/media/alpha-lab/kitchen-heatmap.jpeg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file removed src/media/alpha-lab/kitchen-imgs.png
Binary file not shown.
Binary file removed src/media/alpha-lab/kitchen-overlay.png
Binary file not shown.
Binary file added src/media/alpha-lab/kitchen.jpeg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added src/media/alpha-lab/table-heatmap.jpeg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added src/media/alpha-lab/table.jpeg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file removed src/media/alpha-lab/tv-img.png
Binary file not shown.
Binary file removed src/media/alpha-lab/tv-overlay.png
Binary file not shown.
Binary file added src/media/alpha-lab/tv1-heatmap.jpeg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added src/media/alpha-lab/tv1.jpeg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added src/media/alpha-lab/tv2-heatmap.jpeg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added src/media/alpha-lab/tv2.jpeg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading