Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Iteration IMU Transformations AL Article #707

Merged
merged 5 commits into from
Sep 3, 2024
Merged

Iteration IMU Transformations AL Article #707

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
38b8cf1
Iterated on content
Sep 3, 2024
548ae4d
remove tip about neon positioning and leftward pos/neg rotations
rennis250 Sep 3, 2024
739440c
Iterated on content
Sep 3, 2024
42af1c1
Bring back the spehrical world coordinates section
Sep 3, 2024
b723b8d
Merge branch 'imu_alpha_labs2' of https://github.com/pupil-labs/pupil…
Sep 3, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
107 changes: 34 additions & 73 deletions alpha-lab/imu-transformations/index.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,29 +24,25 @@

# IMU Transformations

<TagLinks :tags="$frontmatter.tags" />

Check warning on line 27 in alpha-lab/imu-transformations/index.md

View workflow job for this annotation

GitHub Actions / ✍️ Check spelling 

Unknown word (frontmatter)

::: tip
Want to compare IMU and gaze data in the same coordinate system to better understand how people coordinate head and eye movements? The transformation functions in this tutorial will show you how!
:::
This guide contains various transformation functions that help with relating [Neon's IMU data](https://docs.pupil-labs.com/neon/data-collection/data-streams/#movement-imu-data) with other data streams.

This guide contains various transformation functions that can assist when working with Neon's IMU data.
## Rotation between the IMU and the World

## IMU to World Coordinates
The IMU data includes a description of how the IMU is rotated in relation to the world. Concretely, the IMU data contains quaternions that define a rotation transformation between the [the world coordinate system](http://docs.pupil-labs.com/neon/data-collection/data-streams/#movement-imu-data) and the IMU's local coordinate system at different points in time.

One of the key steps when working with IMU data is the transformation that takes coordinates in the local IMU coordinate system to their corresponding coordinates in [the world coordinate system](http://docs.pupil-labs.com/neon/data-collection/data-streams/#movement-imu-data). The quaternion values provided by the IMU can be used to convert between the two coordinate systems. The `transform_imu_to_world` function, defined below, will be used throughout this article.

Note that the origin of the IMU coordinate system is the same as the origin of the world coordinate system.
The `transform_imu_to_world` function below demonstrates how to use these quaternions to transform data from the IMU's local coordinate system to the world coordinate system.

```python
from scipy.spatial.transform import Rotation as R

Check warning on line 38 in alpha-lab/imu-transformations/index.md

View workflow job for this annotation

GitHub Actions / ✍️ Check spelling 

Unknown word (scipy)

def transform_imu_to_world(imu_coordinates, imu_quaternions):
# This array contains a timeseries of transformation matrices,

Check warning on line 41 in alpha-lab/imu-transformations/index.md

View workflow job for this annotation

GitHub Actions / ✍️ Check spelling 

Unknown word (timeseries)
# as calculated from the IMU's timeseries of quaternions values.

Check warning on line 42 in alpha-lab/imu-transformations/index.md

View workflow job for this annotation

GitHub Actions / ✍️ Check spelling 

Unknown word (timeseries)
imu_to_world_matrices = R.from_quat(imu_quaternions).as_matrix()

if np.ndim(imu_coordinates) == 1:

Check warning on line 45 in alpha-lab/imu-transformations/index.md

View workflow job for this annotation

GitHub Actions / ✍️ Check spelling 

Unknown word (ndim)
return imu_to_world_matrices @ imu_coordinates
else:
return np.array([
Expand All @@ -57,11 +53,11 @@
])
```

Now that we have the `transform_imu_to_world` function, let's use it!
### Example: Heading Vectors in World Coordinates
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I changed all the applications of the methods we defined to examples with a ### header. This way in the ## sections we introduce the transformation function and in the ### sections we show how to use it.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@marc-tonsen Thanks!


### Obtain IMU Heading Vectors
The `transform_imu_to_world` function can be used to calculate heading vectors of the IMU in world coordinates. The heading vector essentially describes the direction the IMU is facing. If we imagine the IMU inside the Neon module while it is worn on sombody's head, the heading vector describes the direction the wearer's face is pointing.

Check warning on line 58 in alpha-lab/imu-transformations/index.md

View workflow job for this annotation

GitHub Actions / ✍️ Check spelling 

Unknown word (sombody's)

An alternative representation of IMU orientation data is a heading vector that points outwards from the center of the IMU. It can be useful to compare this heading vector with the 3D gaze vectors in world coordinates.
The "forward-facing axis" is the y-axis, so we can calculate the heading vector by transforming the `(0, 1, 0)` vector.

```python
def imu_heading_in_world(imu_quaternions):
Expand All @@ -71,11 +67,13 @@
)
```

Neutral orientation of the IMU would correspond to a heading vector that points at magnetic North and that is oriented perpendicular to the line of gravity.
::: tip
Neutral orientation (i.e. an identity rotation in the quaternion) of the IMU would correspond to a heading vector that points at magnetic North and that is oriented perpendicular to the line of gravity.
:::

### IMU Acceleration in World
### Example: Acceleration in World Coordinates

The IMU’s acceleration data are specified in its local coordinate system. If you want to understand how the observer is accelerating through their environment, then it can be easier to have the acceleration data specified in the world coordinate system:
The IMU’s translational acceleration data is given in the IMU's local coordinate system. To understand how the observer is accelerating through the world it can be helpful to transform the data into the world coordinate system:

```python
accelerations_in_world = transform_imu_to_world(
Expand All @@ -85,17 +83,16 @@

## Scene to World Coordinates

Neon simultaneously records data in the scene camera and IMU coordinate systems, making it possible to study the relationship between head and eye movements.
A lot of the data generated by Neon is provided in the scene camera's coordinate system, including e.g. gaze, fixation, and eye state data. This coordinate system is **not** equal to the IMU's coordinate system! There is a translation between them (simply because there is a physical distance between the camera and the IMU in the module) and also a rotation (because of how the scene camera's coordinate system is defined).

To facilitate the comparison, it can be useful to represent these data streams in the same coordinate system. An important step is accounting for [the fixed 102 degree rotation offset between the scene camera and IMU coordinate systems](https://docs.pupil-labs.com/neon/data-collection/data-streams/#movement-imu-data), as depicted below.
The rotation is a 102 degree rotation around the x-axis of the IMU coordinate system and the translation is along the vector `(0.0 mm, -1.3 mm, -6.62 mm)`.
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

IMO the case including the translation is the default case and the application to 3D directions which should skip the translation is the special case. Introducing it like this makes the concept of "scene cam coords are not the same as IMU coords" easier to understand I think.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@marc-tonsen Agreed!


![Diagrams showing the fixed 102 degree rotation offset between the IMU and scene camera coordinate systems.](./imu-scene_camera_offset-black.png)

We can use data from the IMU to transform gaze in scene camera coordinates to world coordinates. We proceed by building a `transform_scene_to_imu` function that handles the rotation between the two coordinate systems. It also accepts a `translation_in_imu` keyword argument to specify if points should be shifted in the IMU system. This will be relevant when converting 3D eyestate to world coordinates.

We can define a `transform_scene_to_imu` function that handles the rotation between the two coordinate systems.

```python
def transform_scene_to_imu(coords_in_scene, translation_in_imu=np.zeros((3,))):
def transform_scene_to_imu(coords_in_scene, translation_in_imu=np.array([0.0, -1.3, -6.62])):
imu_scene_rotation_diff = np.deg2rad(-90 - 12)
scene_to_imu = np.array(
[
Expand All @@ -122,28 +119,26 @@
return coords_in_imu.T
```

Putting together the `transform_scene_to_imu` and `transform_imu_to_world` functions, we can build a composite `transform_scene_to_world` function:
Combining the `transform_scene_to_imu` function with the `transform_imu_to_world` function allows us to go all the way from scene camera coordinate system to world coordinate system

```python
def transform_scene_to_world(coords_in_scene, imu_quaternions, translation_in_imu=np.zeros((3,))):
def transform_scene_to_world(coords_in_scene, imu_quaternions, translation_in_imu=np.array([0.0, -1.3, -6.62])):
coords_in_imu = transform_scene_to_imu(coords_in_scene, translation_in_imu)
return transform_imu_to_world(coords_in_imu, imu_quaternions)
```

You can now use this function to transform data in scene camera coordinates to world coordinates. Head to the [Application Example](#application-example) to see how!
### Example: Eyestate in World Coordinates

Check warning on line 130 in alpha-lab/imu-transformations/index.md

View workflow job for this annotation

GitHub Actions / ✍️ Check spelling 

Unknown word (Eyestate)

## Eyestate to World Coordinates
The `transform_scene_to_world` function allows us easily convert [eye state data](](https://docs.pupil-labs.com/neon/data-collection/data-streams/#_3d-eye-states)) given in scene camera coordinates to world coordinates.

The [3D eyestate estimates](https://docs.pupil-labs.com/neon/data-collection/data-streams/#_3d-eye-states) provided by Neon are aligned with the scene camera coordinate system. This means we can use the `transform_scene_to_world` function above to reconstruct the pose of the eyes in the world coordinate system. We just need to consider that the scene camera is displaced a bit from the IMU.
Since the eyeball center and optical axis values provided by the 3D eyestate estimates are intrinsically linked, we provide the `eyestate_to_world` function to simplify doing the conversions:
::: warning
Note, to do this right in practice you need to make sure you sample the quaternions and eye state data from the same timestamps. Since both data streams are generated independently and do not share the same set of timestamps, this is a challenge in itself.

```python
# The 3D eyestate data and the IMU quaternions should be sampled
# at the same timestamps. You can linearly interpolate the IMU data.
# See the Application Example:
# http://docs.pupil-labs.com/alpha-lab/imu-transformations/#application-example
We are glossing over this here, but one possible solution to this is interpolating the IMU data to match the timestamps of the eye state data, which is demonstrated [here](http://docs.pupil-labs.com/alpha-lab/imu-transformations/#application-example).
:::

```python
def eyestate_to_world(eyeball_centers, optical_axes, imu_quaternions):

Check warning on line 141 in alpha-lab/imu-transformations/index.md

View workflow job for this annotation

GitHub Actions / ✍️ Check spelling 

Unknown word (eyestate)
"""
The eyeball_centers and optical_axes inputs are for the same eye.
"""
Expand All @@ -165,9 +160,8 @@
return eyeball_centers_in_world, optical_axes_in_world
```

## 3D Gaze to World Coordinates

Neon provides 3D gaze data in [spherical coordinates (i.e., `azimuth/elevation [deg]`)](https://docs.pupil-labs.com/neon/data-collection/data-format/#gaze-csv). The `transform_scene_to_world` function above expects 3D Cartesian coordinates, so to convert spherical 3D gaze to world coordinates, we will need the `spherical_to_cartesian_scene` function:
### Example: 3D Gaze Direction in World Coordinates
Neon provides 3D gaze directions in [spherical coordinates (i.e., `azimuth/elevation [deg]`)](https://docs.pupil-labs.com/neon/data-collection/data-format/#gaze-csv). The `transform_scene_to_world` function above expects 3D Cartesian coordinates, so we need to convert the data first.

```python
def spherical_to_cartesian_scene(elevations, azimuths):
Expand All @@ -188,7 +182,7 @@
# an azimuth of 0 in traditional spherical coordinates would
# correspond to X = 1. Also, azimuth to the right in Neon is
# more positive, whereas it is more negative in traditional
# spherical coordiantes. So, we convert azimuth to the more

Check warning on line 185 in alpha-lab/imu-transformations/index.md

View workflow job for this annotation

GitHub Actions / ✍️ Check spelling 

Unknown word (coordiantes)
# traditional format.
azimuths_rad *= -1.0
azimuths_rad += np.pi / 2
Expand All @@ -202,53 +196,20 @@
).T
```

Now we have the tools to convert 3D gaze data to world coordinates:
Now we can transform the data to world coordinates. Since we are dealing with 3D directions, rather than 3D points here, it does not make sense to apply the translation that we used in the `transform_scene_to_world` function above. We are thus setting it to zero here.

```python
def gaze_3d_to_world(gaze_elevation, gaze_azimuth, imu_quaternions):
cart_gazes_in_scene = spherical_to_cartesian_scene(gaze_elevation, gaze_azimuth)
return transform_scene_to_world(cart_gazes_in_scene, imu_quaternions)
```

## 2D Gaze to World Coordinates
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The result of this is identical to the results of the section above and it's not using the IMU data in a different way. It shows a different way of obtaining 3D gaze directions in cartesian coordinates, but in the context of this article I feel that's not relevant info. So IMO we should either drop this example or the one above because it's just introducing additional concepts that are not required to understand IMU transformations.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@marc-tonsen I vote for removing 2D Gaze to World Coordinates, as we already perform the 2D to 3D gaze transformation automatically in Cloud, Neon Player, and pl-rec-export.


If you are starting from [the 2D gaze values in scene image coordinates (i.e., `gaze x/y [px]`)](https://docs.pupil-labs.com/neon/data-collection/data-format/#gaze-csv), then you will need to first [undistort](https://docs.pupil-labs.com/alpha-lab/undistort/) and unproject those points to obtain the corresponding 3D gaze vectors. Note that this requires [loading the scene camera calibration data](https://docs-staging.pupil-labs.com/alpha-lab/undistort/#reading-from-the-cloud-download-json-file).

```python
def unproject_2d_gaze(gaze_points_2d, scene_camera_matrix, scene_distortion_coefficients):
"""
Transform the 2D gaze values from Neon to 3D gaze vectors.
"""

gaze_points_2d_undist = cv2.undistortPoints(gaze_points_2d, scene_camera_matrix, scene_distortion_coefficients)
gaze_vectors_3d = cv2.convertPointsToHomogeneous(gaze_points_2d_undist)
gaze_vectors_3d /= np.linalg.norm(gaze_vectors_3d, axis=2)[:, np.newaxis]

return gaze_vectors_3d
```

Then, we can use the functions from [the previous section](#3d-gaze-to-world-coordinates) to convert 2D gaze to 3D world coordinates:

```python
# The gaze data and the IMU quaternions should be sampled at the
# same timestamps. You can linearly interpolate the IMU data to
# ensure this.
# See the Application Example:
# http://docs.pupil-labs.com/alpha-lab/imu-transformations/#application-example

gaze_points_2d = gaze[["gaze x [px]", "gaze y [px]"]].to_numpy()
def gaze_2d_to_world(gaze_points_2d, scene_camera_matrix, scene_distortion_coefficients, imu_quaternions):
cart_gazes_in_scene = undistort_and_unproject(
gaze_points_2d, scene_camera_matrix, scene_distortion_coefficients
)
return transform_scene_to_world(
cart_gazes_in_scene, imu_quaternions
)
return transform_scene_to_world(cart_gazes_in_scene, imu_quaternions, translation_in_imu=np.zeros(3))
```

## World Spherical Coordinates
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This too is not related to IMU transformations. I'm not sure we'd want to cover the transformation between cartesian and spherical coordinates anywhere explicitly in our docs, as this is an algebra problem that's covered a lot on the internet and it is not required that often in eye tracking. Happy to discuss this though!

Copy link
Contributor

@rennis250 rennis250 Sep 3, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@marc-tonsen In our case, I say we need it, because Neon's spherical coordinate conventions break with traditional spherical coordinate conventions in subtle ways. Considering the IMU seems to be a source of confusion for users, I would vote to make things as easy as possible for them. It's also been a request from two users so far.

Using the transformations introduced above, we can transform various data into cartesian world coordinates. For some things it is more intuitive to have the data in spherical coordinates though. For instance, you might want to know when someone’s gaze or heading deviates from parallel with the horizon, i.e. if they are looking/facing upwards or downwards.

Converting data into spherical world coordinates makes these things obvious. When wearing Neon, an elevation and azimuth of 0 degrees corresponds to a neutral orientation: i.e., aimed at magnetic North and parallel to the horizon. A positive elevation corresponds to looking upwards, and a negative elevation corresponds to looking downwards.

When studying head orientation and gaze orientation as observers navigate a 3D environment, it can be useful to know how much these quantities deviate from pointing at a given landmark or direction. For instance, you might want to know when someone’s gaze or heading deviates from parallel with the horizon. This can be simplified by converting world points from Cartesian to spherical coordinates. The [Euler angles from the IMU](https://docs.pupil-labs.com/neon/data-collection/data-streams/#euler-angles) are already in a compatible format. For gaze data in world coordinates, the `cartesian_to_spherical_world` function below will do the necessary transformation. When wearing Neon normally, an elevation and azimuth of 0 degrees corresponds to a neutral orientation: i.e., aimed at magnetic North and parallel to the horizon.
The [Euler angles from the IMU](https://docs.pupil-labs.com/neon/data-collection/data-streams/#euler-angles) are already in a compatible format. For gaze data in world coordinates, the `cartesian_to_spherical_world` function below will do the necessary transformation.

```python
def cartesian_to_spherical_world(world_points_3d):
Expand All @@ -272,7 +233,7 @@

radii = np.sqrt(x**2 + y**2 + z**2)

elevation = -(np.arccos(z / radii) - np.pi / 2)

Check warning on line 236 in alpha-lab/imu-transformations/index.md

View workflow job for this annotation

GitHub Actions / ✍️ Check spelling 

Unknown word (arccos)
azimuth = np.arctan2(y, x) - np.pi / 2

# Keep all azimuth values in the range of [-180, 180] to remain
Expand All @@ -288,7 +249,7 @@

## Application Example

Below, we present a video showing how some of the functions in this article where used to visualize different combinations of head and eye movements in world coordinates. The code for producing the visualization [can be found here](https://gist.github.com/rennis250/8a684ea1e2f92c79fa2104b7a0f30e20).
Below, we present a video showing how some of the functions in this article were used to visualize different combinations of head and eye movements in world coordinates. The code for producing the visualization [can be found here](https://gist.github.com/rennis250/8a684ea1e2f92c79fa2104b7a0f30e20).

<Youtube src="lbmyBgpS2OE"/>

Expand Down
4 changes: 0 additions & 4 deletions neon/data-collection/data-streams/index.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -102,8 +102,4 @@ When exporting recordings from Pupil Cloud or Neon Player the IMU's orientation
- Yaw is a rotation around the world z-axis with a value range of -180° to +180°. With a calibrated IMU, a yaw of 0° indicates alignment with magnetic north. Leftward turn is positive, rightward is negative.
- Roll is a rotations around the world y-axis with a value range of -180° to +180°. Wearing Neon upright with a neutral head pose roughly corresponds to a roll of 0°. Rightward tilt is positive, leftward is negative.

::: tip
Neon may sit differently on each wearer’s face, causing deviations from a neutral orientation even when facing magnetic north. Also, leftward rotations for the yaw values reported by the IMU are positive, whereas leftward rotations about the y-axis of the scene camera coordinate system are negative.
:::

![IMU Pitch, Yaw, Roll](./imu-pitch-yaw-roll-black.png)
Loading