Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Alphalab scanpath tutorial rework #624

Merged
merged 7 commits into from
Oct 4, 2023
Merged
Show file tree
Hide file tree
Changes from 5 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion .lycheeignore
Original file line number Diff line number Diff line change
Expand Up @@ -85,10 +85,11 @@ https://docs.pupil-labs.com/eEl3sswsTms
https://docs.pupil-labs.com/aPLnqu26tWI
https://docs.pupil-labs.com/_Jnxi1OMMTc
https://docs.pupil-labs.com/uoq11XtNH5E
https://docs.pupil-labs.com/X43aTIRjwgQ?si=aTzAkRrYNqdOEf0T
# TCP
https://docs.pupil-labs.com/f'tcp://%7Bip%7D:%7Bpub_port%7D
https://docs.pupil-labs.com/f'tcp://%7Bip%7D:%7Bport%7D
https://docs.pupil-labs.com/f'tcp://%7Bip%7D:%7Bsub_port%7D
# Twitter as it seems to timeout
https://twitter.com/pupil_labs
https://twitter.com/
https://twitter.com/
180 changes: 71 additions & 109 deletions src/alpha-lab/scanpath-rim.md
Original file line number Diff line number Diff line change
@@ -1,135 +1,97 @@
---
description: Generate static and dynamic scanpaths with Reference Image Mapper

Check warning on line 2 in src/alpha-lab/scanpath-rim.md

View workflow job for this annotation

GitHub Actions / ✍️ Check spelling

Unknown word (scanpaths)
permalink: /alpha-lab/scanpath-rim/

Check warning on line 3 in src/alpha-lab/scanpath-rim.md

View workflow job for this annotation

GitHub Actions / ✍️ Check spelling

Unknown word (scanpath)
tags: [Pupil Invisible, Neon, Cloud]
---

# Generate static and dynamic scanpaths with Reference Image Mapper

Check warning on line 7 in src/alpha-lab/scanpath-rim.md

View workflow job for this annotation

GitHub Actions / ✍️ Check spelling

Unknown word (scanpaths)

<TagLinks />

<div class="iframe-container2">
<iframe width="2000" height="1500" src="https://www.youtube.com/embed/7V3X4XmbRAM" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>

Check warning on line 12 in src/alpha-lab/scanpath-rim.md

View workflow job for this annotation

GitHub Actions / ✍️ Check spelling

Unknown word (autoplay)
</div>
<br>

The [Reference Image Mapper](/enrichments/reference-image-mapper/) is a powerful tool that maps gaze onto 2D
images of real-world environments and generates heatmaps. Now, we offer a new way to visualize your Reference Image Mapper
data. We have created a ready-to-use script that generates static and dynamic scanpaths, providing deeper insights into
patterns of visual behavior.
::: tip
Picture this: Build and customise scanpath visualisations with your Reference Image Mapper exports!

Check warning on line 17 in src/alpha-lab/scanpath-rim.md

View workflow job for this annotation

GitHub Actions / ✍️ Check spelling

Unknown word (customise)

Check warning on line 17 in src/alpha-lab/scanpath-rim.md

View workflow job for this annotation

GitHub Actions / ✍️ Check spelling

Unknown word (scanpath)

Check warning on line 17 in src/alpha-lab/scanpath-rim.md

View workflow job for this annotation

GitHub Actions / ✍️ Check spelling

Unknown word (visualisations)
:::

## Visualising gaze exploration with scanpaths

Check warning on line 20 in src/alpha-lab/scanpath-rim.md

View workflow job for this annotation

GitHub Actions / ✍️ Check spelling

Unknown word (Visualising)

Check warning on line 20 in src/alpha-lab/scanpath-rim.md

View workflow job for this annotation

GitHub Actions / ✍️ Check spelling

Unknown word (scanpaths)
Scanpaths are graphical representations of gaze over time. As shown in the video above, they contain fixation locations

Check warning on line 21 in src/alpha-lab/scanpath-rim.md

View workflow job for this annotation

GitHub Actions / ✍️ Check spelling

Unknown word (Scanpaths)
(blue numbered circles) and saccades made between fixations (blue connecting lines). The size of the circles reflect
fixation durations, with longer fixations corresponding to bigger circles. The length of the connecting lines reflect
saccadic amplitudes, with larger gaze shifts corresponding to longer lines. Scanpaths offer a glimpse into how the
observer has focused their attention on different aspects of the scene, which is a valuable tool for understanding a
person's visual attention and perception.
elepl94 marked this conversation as resolved.
Show resolved Hide resolved

In this guide, we will show you how to generate both static and dynamic scanpath visualisations using your Reference
Image Mapper exported data.

::: tip
Before continuing, ensure you are familiar with the [Reference Image Mapper](/enrichments/reference-image-mapper)
enrichment. Check out [this explainer video](https://www.youtube.com/watch?v=ygqzQEzUIS4&t=56s) for reference.
:::

## What is a scanpath?
A scanpath is a graphical representation of an individual's gaze movements. It shows the sequence of fixations, or pauses
in gaze, and the rapid eye movements made between fixations, known as saccades. The scanpath offers a glimpse into what
the observer is focusing on and the duration and frequency of their attention to different aspects of the scene. This
information is a valuable tool for understanding a person's visual attention and perception.

<img src="../media/alpha-lab/Steven_scanpath.jpeg"/>

## What you'll need:
- A Reference Image Mapper export download
- Python 3.7 or higher
- [This](https://gist.github.com/elepl94/9f669c4d81e455cf2095957831219664) ready-to-go script

## Running the code
All you need to do is run the command `python3 RIM_scanpath.py` in your terminal. A prompt will then appear asking for
the location of the Reference Image Mapper export folder. After this, just sit back and wait for the processing to finish.
Upon completion, the resulting scanpath visualisations will be saved in a newly created sub-folder called "scanpath”.

If you wish to enhance the appearance of your scanpaths, keep reading for additional instructions!

## Personalization
### To each their own color
<p>
This function generates random colors for each participant based on their names.

```python
def color_generator(...):
colors = {
subj: (
random.randint(0, 255),
random.randint(0, 255),
random.randint(0, 255),
)
for subj in names
}

return colors
```
</p>
<p>
However, if you prefer to assign specific colors to each participant, you can easily modify the function to suit your needs. An example could be:

``` python
def color_generator():
colors = {}
colors['Subject1'] = (0, 0, 255)
colors['Subject2'] = (255, 0, 0)
colors['Subject3'] = (0, 255, 0)

return colors
```
</p>

### Make it font-tastic
If you have a preferred font or would like to change the size, simply edit the draw_on_frame() function. The fixation
IDs are displayed in black text with a white border to make them stand out from the background. If you adjust the font
size, it's also recommended to increase the values of `font_thick_w` and `font_thick_b` to maintain visual contrast.
``` python
def draw_on_frame(...):
# text aesthetics
font = cv2.FONT_HERSHEY_DUPLEX
font_size = 1
font_thick_w = 3
font_thick_b = 1
...
```
### My name is legend
The script includes two functions for creating a legend to display the wearer names and corresponding colors:

1. `draw_name_legend()`: This function creates a legend box that displays only the name of the wearer on their individual scanpath video and image.
2. `draw_all_names_legend()`: This function creates a legend that displays all the wearer names on the final general scanpath image.

To customize the appearance of the legend, such as the position, dimensions, or colors of the rectangular white box or the colored line,
you can modify the following parameters in both functions:

- `r_end_point` - x and y values of the ending coordinates of the rectangular legend box
- `r_start_point` - x and y values of the starting coordinates of the rectangular legend box
- `l_end_point` - x and y values of the ending coordinates of the colored line
- `l_start_point` - x and y values of the starting coordinates of the colored line
- In `cv2.rectangle`, edit `color` to set a new color for the legend box
- In `cv2.line`, edit `thickness` to set a new width for the colored line
## Extending current tools
The [Reference Image Mapper](/enrichments/reference-image-mapper) enrichment available in Pupil Cloud is a tool that maps gaze onto
2D images and can subsequently generate heatmaps. However, it currently does not support the production of scanpath visualizations.
Thus, we chose to develop a script that shows you how to build your own scanpaths using Reference Image Mapped data.


## Steps
1. Run a [Reference Image Mapper enrichment](https://docs.pupil-labs.com/enrichments/reference-image-mapper/) and download the results
2. Download [this script](https://gist.github.com/elepl94/9f669c4d81e455cf2095957831219664) and follow the [installation instructions](https://gist.github.com/elepl94/9f669c4d81e455cf2095957831219664#installation)

## Review the scanpaths
willpatera marked this conversation as resolved.
Show resolved Hide resolved
After the script has completed its execution, you'll find the resulting scanpath visualizations stored in a newly created
sub-folder named "scanpath." For each participant, you will obtain a reference image with the scanpath superimposed on it.
You will also find a video featuring a dynamic scanpath overlay. Finally, if you had multiple participants, an aggregated
visualization combining all participants' scanpaths will be available, enabling a more comprehensive overview of the subjects'
gaze behavior.

<div style="display: flex; justify-content: space-between;">
willpatera marked this conversation as resolved.
Show resolved Hide resolved
<div style="flex: 1; margin-right: 10px;">
<div style="width: 100%; text-align: center;">
<img src="../media/alpha-lab/Jack_scanpath.jpeg" alt="Jack Scanpath" style="width: 100%; height: 100%;">
</div>
</div>
<div style="flex: 1;">
<div class="iframe-container2" style="width: 100%;">
<div style="width: 100%; text-align: center;">
<iframe width="100%" height="100%" src="https://www.youtube.com/embed/X43aTIRjwgQ?si=aTzAkRrYNqdOEf0T" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>
</div>
</div>
</div>
</div>

<div style="width: 100%; text-align: center;">
<img src="../media/alpha-lab/general_scanpath.jpeg" alt="General Scanpath" style="width: 50%; height: 50%;">
</div>

<style scoped>
img, iframe {
width: 100%;
height: 100%;
object-fit: contain;
box-sizing: border-box;
}

img {
border-radius: 10px;
max-width: 100%;
height: auto;
box-sizing: border-box;
}

.iframe-container2{
position: relative;
width: 100%;
padding-bottom: 75%;
margin-bottom: 10px;
height: 0;
margin-left:0;
margin-right:0;
}

.iframe-container2 iframe{
position: absolute;
top:0;
left: 0;
width: 100%;
height: 100%;
}
.iframe-container2 {
position: relative;
width: 100%;
padding-bottom: 75%;
margin-bottom: 10px;
height: 0;
margin-left: 0;
margin-right: 0;
}

.iframe-container2 iframe {
position: absolute;
top: 0;
left: 0;
width: 100%;
height: 100%;
}
</style>
Binary file added src/media/alpha-lab/Jack_scanpath.jpeg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added src/media/alpha-lab/general_scanpath.jpeg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading