Merge pull request #14 from pupil-labs/dev

2.0.1 release candidate 1

CHANGES.rst

@@ -0,0 +1,18 @@
+2.0.1
+#####
+- Document minimum Pupil Invisible Companion version required (v1.4.14)
+- Add code example demonstrating post-hoc time sync between a Pupil Cloud download and
+  a LSL recording
+- Write debug logs to log file (path defined via ``--log_file_name`` parameter)
+
+  - Requires `click <https://pypi.org/project/click/>`_ instead of `asyncclick
+    <https://pypi.org/project/asyncclick/>`_
+
+2.0.0
+#####
+- First release supporting the `Pupil Labs Network API <https://github.com/pupil-labs/realtime-network-api>`_
+- The legacy NDSI-based relay application can be found
+  `here <https://github.com/labstreaminglayer/App-PupilLabs/tree/legacy-pi-lsl-relay/pupil_invisible_lsl_relay>`_
+
+- Pull project skeleton from `<https://github.com/pupil-labs/python-module-skeleton>`_
+- Initial fork from `<https://github.com/labstreaminglayer/App-PupilLabs>`_

docs/guides/index.rst

@@ -0,0 +1,12 @@
+******
+Guides
+******
+
+High-level overviews and hands-on examples how to use the lsl relay.
+
+.. toctree::
+   :maxdepth: 1
+   :glob:
+
+   overview.rst
+   time_alignment.rst

docs/overview.rst → docs/guides/overview.rst

@@ -4,13 +4,27 @@
 Conceptual Overview
 *******************
 
-The Relay is built on top of the `Pupil Invisible Realtime API <https://docs.pupil-labs.com/invisible/how-tos/integrate-with-the-real-time-api/introduction/>`_. It creates LSL outlets for
-each data stream, reads data samples from the realtime API, adapts them to an LSL-readable
+The Relay is built on top of the `Pupil Invisible Realtime API <https://docs.pupil-labs.com/invisible/how-tos/integrate-with-the-real-time-api/introduction/>`_.
+It searches and displays Pupil Invisible instances in the network. Once the user selected one of them, it creates
+LSL outlets for each data stream, reads data samples from the connected device, adapts them to an LSL-readable
 format, and pushes the data samples into their respective outlets.
 
 The Relay pushes data samples with explicit timestamps. Timestamps are taken at the invisible
 companion device, and corrected by the offset between the lsl clock and the current time. We
-provide a detailed overview of timestamps and potential caveats in the `Timestamps`_ section
+provide a detailed overview of timestamps and potential caveats in the `Timestamps`_ section.
+
+Device Selection
+================
+After starting the lsl relay via the command line, the relay searches for Pupil Invisible instances in the network.
+Available instances will be displayed in a list, with the IP address and name of the Companion device. Select
+the instance you want to connect with via the displayed index.
+
+Troubleshooting
+***************
+If your Pupil Invisible device does not appear in the device selection, please check if both the PC running the relay
+and the Companion device are connected to the same network. Also, make sure that your Invisible Companion app is at
+at least version v1.4.14 or higher.
+
 
 LSL Outlets
 ===========
@@ -28,8 +42,9 @@ frequency of ~66 Hz. If you want to use the 200 Hz gaze data from pupil cloud, y
 recording with your pupil invisible companion device simultaneously with the LSL recording, and use the ``lsl.time_sync.*``
 events generated by the relay to align you data streams post-hoc.
 
-**Important:** If you want to do the post-hoc alignment of LSL data and cloud data, you must also subscribe to the LSL
-event stream and make sure that at least two events are contained in your recording.
+.. Important::
+   If you want to do the post-hoc alignment of LSL data and cloud data, you must also subscribe to the LSL
+   event stream and make sure that at least two events are contained in your recording.
 
 Event Data Outlet
 *****************
@@ -57,4 +72,6 @@ be generated.
 
 #. The offset between the lsl local clock and the time since epoch in seconds is subtracted from the sample timestamp in seconds to find the corresponding lsl time. This corrected timestamp is explicitly pushed to LSL together with the Gaze samples and the Events.
 
-**Caveat**: A misalignment between the time since epoch measured at the companion device and at the time since epoch measured at the device running the Relay might lead to a distortion of the time series.
+.. caution::
+   A misalignment between the time since epoch measured at the companion device and at the time since epoch measured
+   at the device running the Relay might lead to a distortion of the time series.

docs/guides/time_alignment.rst

@@ -0,0 +1,102 @@
+:tocdepth: 3
+
+**************************
+Using gaze data from Cloud
+**************************
+
+The LSL stream will contain gaze data with a resolution of ~66 Hz.
+You can get a higher sampling rate when you're using the gaze data downloaded from
+Pupil Cloud. In order to do this, you'll need to align the timestamps collected during
+the lsl streaming to the timestamps recorded in cloud.
+
+Setup
+=====
+#. Start the lsl relay of your Pupil Invisible glasses.
+
+#. In your lsl recording software, select both the gaze data stream and the event stream.
+
+#. Start the lsl recording through your software of choice (e.g. LabRecorder).
+
+#. In your Pupil Invisible Companion App, tap the red "record" button and make sure the recording is running.
+
+#. Run your experiment/data collection.
+
+#. Stop the recording in the Pupil Invisible Companion App.
+
+#. Stop the lsl recording.
+
+#. Wait till the gaze data was uploaded to Pupil Cloud and the 200 Hz gaze data was computed.
+
+#. Export the gaze data from Pupil Cloud by right-clicking on the recording and selecting Downloads -> Download Recording.
+
+#. You will end up with an xdf file, containing all data recorded through lsl (including gaze and event data),
+   and one csv file for each gaze and event data downloaded from Pupil Cloud.
+
+From here, you can perform the timestamp alignment.
+
+.. important::
+   If your recording is short (less than 1 minute), you should increase the frequency at which ``lsl.timesync`` events
+   are being generated. As a rule of thumb you should aim for at least 3 events being sent throughout the recording, including
+   ``recording.begin`` and ``recording.end`` events, which are generated by starting and ending the recording in the Pupil
+   Invisible Companion App.
+
+   You can change the frequency at which ``lsl.timesync`` events are being sent by setting the ``--time_sync_interval``
+   argument.
+
+To run the pipeline below, you can install the necessary dependencies needed via
+
+.. code-block::
+
+   pip install pupil-invisible-lsl-relay[pupil_cloud_alignment]
+
+Import dependencies
+===================
+Import the installed dependencies before running the example code below.
+
+.. literalinclude:: ../../examples/linear_time_model.py
+   :language: python
+   :lines: 1-5
+   :linenos:
+
+Loading event data from xdf file
+=================================
+Event streams can be identified and selected by their name.
+
+.. literalinclude:: ../../examples/linear_time_model.py
+   :language: python
+   :lines: 7-25
+   :linenos:
+
+Loading event data from cloud
+==============================
+The raw data enrichment contains a csv file with event names and timestamps.
+We can load this csv file with pandas.
+
+.. literalinclude:: ../../examples/linear_time_model.py
+   :language: python
+   :lines: 27-32
+   :linenos:
+
+Building a linear model to map cloud time to lsl time
+======================================================
+After extracting the event names and time stamps from both series as shown above,
+you can build a linear model to translate from one time series to the other.
+
+Once the linear model was fitted, we can apply it to map the cloud timestamps to lsl timestamps.
+
+.. literalinclude:: ../../examples/linear_time_model.py
+  :language: python
+  :lines: 34-
+  :linenos:
+
+.. hint::
+   If you want to invert the mapping, to transform lsl timestamps to cloud timestamps,
+   you'll have to change the order of arguments in ``time_mapper.fit`` to
+
+   .. code-block:: diff
+
+       time_mapper.fit(
+      -   filtered_cloud_event_data[[event_column_timestamp]],
+          filtered_lsl_event_data[event_column_timestamp],
+      +   filtered_cloud_event_data[[event_column_timestamp]],
+       )

docs/history.rst

@@ -6,15 +6,4 @@
 History
 *******
 
-2.0.0
-#####
-- First release supporting the `Pupil Labs Network API <https://github.com/pupil-labs/realtime-network-api>`_
-- The legacy NDSI-based relay application can be found
-  `here <https://github.com/labstreaminglayer/App-PupilLabs/tree/legacy-pi-lsl-relay/pupil_invisible_lsl_relay>`_
-
-v0.1
-####
-- Pull project skeleton from `<https://github.com/pupil-labs/python-module-skeleton>`_
-- Initial fork from `<https://github.com/labstreaminglayer/App-PupilLabs>`_
-
 .. include:: ../CHANGES (links).rst

docs/index.rst

@@ -7,7 +7,6 @@ Pupil Invisible device to the `labstreaminglayer <https://github.com/sccn/labstr
 
 Install and Usage
 ==================
-
 Install the Pupil Invisible Relay with pip::
 
    pip install pupil-invisible-lsl-relay
@@ -24,18 +23,25 @@ The Relay takes two optional arguments:
 - ``--timeout`` is used to define the maximum time (in seconds) the relay will search the network for new
   devices before returning. The default is 10 seconds.
 
+- ``--log_file_name`` defines the name and path of the log file. The default is ``pi_lsl_relay.log``.
+
 .. caution::
    The Relay currently relies on `NTP`_ for time synchronization between the phone and
    the computer running the relay application. See the :ref:`timestamp_docs` section for
    details.
 
+.. important::
+   Make sure the version of your Pupil Invisible Companion App is at least v1.4.14 or higher.
+   You can download the latest version of the App in the Play Store on your Pupil Invisible Companion device.
+
+
 .. _NTP: https://en.wikipedia.org/wiki/Network_Time_Protocol
 
 .. toctree::
-   :maxdepth: 1
+   :maxdepth: 2
    :glob:
 
-   overview.rst
+   guides/index.rst
    api.rst
    history.rst
 

examples/linear_time_model.py

@@ -0,0 +1,64 @@
+# imports for the full pipeline
+import numpy as np
+import pandas as pd
+import pyxdf
+from sklearn import linear_model
+
+# import xdf data
+# define the name of the stream of interest
+stream_name = 'pupil_invisible_Event'
+
+# load xdf data
+path_to_recording = './lsl_recordings/recorded_xdf_file.xdf'
+data, header = pyxdf.load_xdf(path_to_recording, select_streams=[{'name': stream_name}])
+
+# when recording from one device, there will be only one event stream
+# extract this stream from the data
+event_stream = data[0]
+
+# extract event names and lsl time stamps into a pandas data frames
+event_column_name = 'name'
+event_column_timestamp = 'timestamp [s]'
+
+lsl_event_data = pd.DataFrame(columns=[event_column_name, event_column_timestamp])
+lsl_event_data[event_column_name] = [name[0] for name in event_stream['time_series']]
+lsl_event_data[event_column_timestamp] = event_stream['time_stamps']
+
+# import cloud data
+path_to_cloud_events = './cloud_recordings/events.csv'
+cloud_event_data = pd.read_csv(path_to_cloud_events)
+
+# transform cloud timestamps to seconds
+cloud_event_data[event_column_timestamp] = cloud_event_data['timestamp [ns]'] * 1e-9
+
+# filter events that were recorded in the lsl stream and in cloud
+name_intersection = np.intersect1d(
+    cloud_event_data[event_column_name], lsl_event_data[event_column_name]
+)
+
+# filter timestamps by the event intersection
+filtered_cloud_event_data = cloud_event_data[
+    cloud_event_data[event_column_name].isin(name_intersection)
+]
+
+filtered_lsl_event_data = lsl_event_data[
+    lsl_event_data[event_column_name].isin(name_intersection)
+]
+
+# fit a linear model
+time_mapper = linear_model.LinearRegression()
+time_mapper.fit(
+    filtered_cloud_event_data[[event_column_timestamp]],
+    filtered_lsl_event_data[event_column_timestamp],
+)
+
+# use convert gaze time stamps from cloud to lsl time
+cloud_gaze_data = pd.read_csv('./cloud_recordings/gaze.csv')
+
+# map from nanoseconds to seconds
+cloud_gaze_data[event_column_timestamp] = cloud_gaze_data['timestamp [ns]'] * 1e-9
+
+# predict lsl time in seconds
+cloud_gaze_data['lsl_time [s]'] = time_mapper.predict(
+    cloud_gaze_data[[event_column_timestamp]]
+)

setup.cfg

@@ -1,9 +1,9 @@
 [metadata]
 name = pupil_invisible_lsl_relay
-description = Project description
+description = Relay Pupil Invisible data to LabStreamingLayer
 long_description = file: README.rst
 long_description_content_type = text/x-rst
-url = https://github.com/pupil-labs/python-module-skeleton
+url = https://github.com/pupil-labs/pupil-invisible-lsl-relay/
 author = Pupil Labs GmbH
 author_email = [email protected]
 license = MIT
@@ -18,12 +18,15 @@ classifiers =
     Programming Language :: Python :: 3.8
     Programming Language :: Python :: 3.9
     Programming Language :: Python :: 3.10
+project_urls =
+    Documentation=https://pupil-invisible-lsl-relay.readthedocs.io/en/stable/
+    History=https://pupil-invisible-lsl-relay.readthedocs.io/en/latest/history.html
+    Pupil Labs Realtime Network API=https://github.com/pupil-labs/realtime-network-api
+    LabStreamingLayer=https://labstreaminglayer.readthedocs.io/
 
 [options]
 packages = find_namespace:
 install_requires =
-    anyio>=2.0 # asyncclick requires anyio
-    asyncclick>=8.0.3.2
     click>=7.0
     pupil-labs-realtime-api>=1.0.0
     pylsl>=1.12.2
@@ -42,7 +45,7 @@ exclude =
 
 [options.entry_points]
 console_scripts =
-    pupil_invisible_lsl_relay = pupil_labs.invisible_lsl_relay.cli:main_handling_keyboard_interrupt
+    pupil_invisible_lsl_relay = pupil_labs.invisible_lsl_relay.cli:relay_setup_and_start
 
 [options.extras_require]
 docs =
@@ -51,6 +54,11 @@ docs =
     rst.linker>=1.9
     sphinx<4.4  # 4.4 does not detect TypeVars correctly
     importlib-metadata;python_version<"3.8"
+pupil_cloud_alignment =
+    numpy
+    pandas
+    pyxdf
+    scikit-learn
 testing =
     flake8<4  # workaround https://github.com/tholo/pytest-flake8/issues/81
     pytest>=6

src/pupil_labs/invisible_lsl_relay/__main__.py

@@ -1,4 +1,4 @@
-from .cli import main_handling_keyboard_interrupt
+from .cli import relay_setup_and_start
 
 if __name__ == "__main__":
-    main_handling_keyboard_interrupt()
+    relay_setup_and_start()