From 6806f3f8dd2814cc8c82560b75ec177f697a5bb9 Mon Sep 17 00:00:00 2001 From: Dominic Canare Date: Mon, 16 Sep 2024 07:47:53 -0400 Subject: [PATCH 1/3] Adds psychopy documentation --- .../psychopy/apriltag-frame.png | Bin 0 -> 15492 bytes neon/data-collection/psychopy/index.md | 99 +++++++++++++++++- 2 files changed, 98 insertions(+), 1 deletion(-) create mode 100644 neon/data-collection/psychopy/apriltag-frame.png diff --git a/neon/data-collection/psychopy/apriltag-frame.png b/neon/data-collection/psychopy/apriltag-frame.png new file mode 100644 index 0000000000000000000000000000000000000000..07b7040dffece050e2217ec5d7e37f94db6a6926 GIT binary patch literal 15492 zcmeHN2~bnl8a_cFfJLF&p>>U+9ZPHWkWCh8f)F)s&1IEDq)$n{4wlNy)oJ8=zuj^oYvZK8XwqJJ8U}G2rH{c7edGKVy z_C$C+2v1dxT~;}4V}}?1cla3qB=BzF8YPy*l9*Hmoyy|D@pL9zK<5kS!6f*f$r3R5 zP(ESLkVY6M+Ok$}sNV-Fp)gh~6q0lXtx}bO0$|B|XLD%usX0@=U%qhd^lx~JMW<&7 zQ;)9w$ctL+6ZyH{+aD~Nx}d1w!oC?|y?)|bW5;FAS~T^oywp_PbM95+{gQ|?BLjl& zWQjkGTt__n&|>b?caQU%^xQu`UdEVnX;$F&HEVO*57s^WD8b$DBuR-X|83l|?fsUX z?{YSNY5vFV=67u8{eNnfc=m{zp0D*?YkAL-Ui%Bbb3)D%@j3s32^%MyBATL#UZqy( zse)Nu)n9Fv)=bFx@Ov5Y)v49AS)pYi08| z`tSRGUfmv@`SEL?f)fQ7x#+v^F5RQ8=_oATSW6^Hmc69SeovTqIQmN4v`$-Vdq)GS zurCuFq8Q4uoALfkZ_9*ktd@jk_=5!X+p`J zr4?k7N)bwaBUnP0=!EDBRdl)mU6?M7m!~Jo`3iD)m`{jB02!!J6GF16Q#3|_C6tWw z6~H#uO(Ts+0c`^U2&GA- z90KbR-LNP4bcQPehY#|7gnD4^I5DJ>NCYCSJQY(M;N;RnlQGA5nQ8|dP?}DYgg?Ll7evI0L&;1keW)ik1u-e% zfKc)Rl_u3P^dMfPMi-h83{3`?863=Ja_Dphhs$Qzr;YPlgc^*{6EU86^2CQ@W)Z;2 zKwuHfPay-mAI?P}G@yt{YlzosQ$opuX?O6k1cs9WF(D$vghHl+i8y{B5mV?eHQUw4 z*h#bsm2&0(qsD@V6awj?$)i<9D1Rm1Wsj(ZsNQ~OzfDo$p+q9#K_Nio_9+;VG*p4> z33=Hc$yXqnBot;3hr15+s~+PDD32Y?<;$4R_AG?LW-+)F=tevWgU?{oIl*#tzX)n$>XP>uTju zFJfTy`PZ$ho>f=+hU?PI{i~MuWQ2_)fCmXpr38>kyK@bIik&BZXxLzv4E96-tOx;* zHGbJxApss__eA^(vb_k;+gqg9o&wpLHq(u2URl7EWjO$hm+fHACje#8*AqeFj;5IE zI1OIdfbv9Mm3FDX-;Hi>&3PZdas+nx%@`rFw=MsrSf!S~Z(To8-wXBP+hYc+WmtOD(jYTiCw;Joc zxg+gjk5jqide^4XwykcU{CevGHf!VV3R?yZG+Oh|jf*K*TP*@})SlK;S&1(@qZE-rn9N zXC0ZXE7Qpt_%ZG_@@d~C6f^&by;WWCy5JqnuQ->#x%|!LZOiH?Sf5~5D)#R0wzQUnnMwMed=u?mqgsk9n$7(rcW-K`wLZaI9sodPct?HJ z+t=t5BQMVE%BpzK+ws-Dj(xAZ-kx84`^*NfTbtHHB^zzKFPuy(*B^)oyAk4S2b@xM zIXAZ=Q(4&DXvdGIJfJE|ur6y;j6Iabbxz$-TArfZl2dZuSXlhP(+$_FGT?Jgm{YVF zDj^nar7cZX9P>1PE_mWXs;sP`Q}3vW zg7Ym-$y%fMgT>Hm#U&K;LFfEz_W&WwQ|IUR%>|V^j_z-#fjz|wrP}SKT<5InOJI$mu=wuP>M1Kd zLE|~8=8Ktiy~oqe`+=|a+x}Qz+K?0gxBYbN@$I|i+~w@)X0I?XSsn!5cxvV4Qv)a-X-IEDm-t6XE{gu+py@f>RiT(H1 zA8^WMsiEB~`uvP5Zzi`L&pm#(R9Zj;U7;PeZTU4v?iZDIa_)}>yc^#4&rscfwB&xj zR@zt5@uG78-QBf0nbJT68P=G(DQW(ovMrePZPw0fo*=Jo-0Y>zx5~fJke#C1Q2mP< z)pg~~pJxETxj2Ti8LCzn1VEa9g;T0eSnh)*eE?t_WW&6dIA0&=lw#b_w%7+?Z`DV( UQ@w%fu-BD1QYtEXEpgR<0k^aTqyPW_ literal 0 HcmV?d00001 diff --git a/neon/data-collection/psychopy/index.md b/neon/data-collection/psychopy/index.md index 8157ddc3f..8022c1029 100644 --- a/neon/data-collection/psychopy/index.md +++ b/neon/data-collection/psychopy/index.md @@ -1,2 +1,99 @@ # PsychoPy -This page is still a work in progress. To use PsychoPy with Neon you need to use [this](https://www.psychopy.org/api/iohub/device/eyetracker_interface/PupilLabs_Neon_Implementation_Notes.html) dedicated plugin. More documentation is coming soon! + +[PsychoPy](https://psychopy.org/) is widely used open-source software for creating and running psychophysics experiments. It enables users to present stimuli, collect data, and interface with a variety of hardware and software applications. + +PsychoPy users have two options for designing their experiments. + +- [Builder](https://www.psychopy.org/builder/) – Gives users a graphical interface with little or no need to write code - although it does support custom code when necessary. A Pupil Labs plugin for PsychoPy is provided to simplify interactions between PsychoPy and Neon. +- [Coder](https://psychopy.org/coder/index.html) – Gives users the option to generate experiments or do other things programmatically, [using Psychopy like any other Python package](https://psychopy.org/api/). + +## Using PsychoPy with Neon + +When using PsychoPy with Neon, we recommend making a recording in the Neon Companion app for the duration of the experiment. PsychoPy’s standard "Eyetracker Record" component can be used to start and stop recordings accordingly. For experiments that do not require screen-based gaze coordinates (e.g., pupillometry/eye state), this is all that is required. + +To use Neon for screen-based work in PsychoPy, the screen needs to be robustly located within the scene camera’s field of view, and Neon’s gaze data subsequently transformed from scene camera-based coordinates to screen-based coordinates. The Pupil Labs eyetracker plugin for PsychoPy achieves this with the use of AprilTag Markers and the [real-time-screen-gaze](https://github.com/pupil-labs/real-time-screen-gaze) Python package (installed automatically with the plugin). + +## Builder + +### Installing the Plugin + +- Open PsychoPy Builder +- Click on the "Tools" menu and select "Plugins/package Manager" +- Find "Pupil Labs" in the plugin list and click "Install" +- Close all PsychoPy windows and re-open Builder + +### Experiment Settings + +- Open your experiment and open the experiment settings (gear ⚙️ icon in the toolbar) +- Go to the "Eyetracking" tab and select "Pupil Labs (Neon)" as your eyetracking device +- Modify the IP address and port if necessary + +### Components + +The standard "Eyetracker Record" and "Region of Interest" components work with Neon. Because Neon is calibration-free, the Calibration and Validation components are unused. + +Two new Builder components will be available in the components list under the Eyetracking section: "April Tag Frame" and "April Tag". These are necessary for screen-based work. + +- April Tag Frame: this component is recommended for most users. Using it in your Builder experiment will display an array of AprilTag markers around the edge of the screen. You can configure the number of markers to display along the horizontal and vertical edges of the screen, the size and contrast of the markers, and (optionally) the marker IDs. A minimum of four markers (2 horizontally by 2 vertically) is recommended, but more markers will provide more robust detection and accurate mapping. +![AprilTag Frame](./apriltag-frame.png) + +- April Tag: this component will add a single AprilTag marker to your display. It is intended for use when the April Tag Frame component cannot be used (e.g., you need to display stimuli on the edges of the display where the April Tag Frame component would place markers in the way). + +### Data + +[PsychoPy saves eyetracking data in its own format](https://psychopy.org/hardware/eyeTracking.html#what-about-the-data), but we also recommend that you record your sessions with the Companion app. You can use the "Eyetracker Record" component in Builder to automate starting and stopping/saving your recordings. + +### Example + +Check out our simple but complete [gaze contingent demo designed in PsychoPy Builder](https://github.com/pupil-labs/psychopy-gaze-contingent-demo) to see how it all works! + +## Coder + +To use Neon with PsychoPy coder, we recommend interfacing directly with the [real-time API](https://docs.pupil-labs.com/neon/real-time-api/tutorials/) and, for screen-based tasks, using the [real-time-screen-gaze](https://github.com/pupil-labs/real-time-screen-gaze) package. `AprilTagFrameStim` and `AprilTagStim` classes are provided to more easily display screen markers and configure a screen-based gaze mapper. + +```python +import numpy as np + +from psychopy import visual, event +from psychopy.tools.monitorunittools import convertToPix +from psychopy_eyetracker_pupil_labs.pupil_labs.stimuli import AprilTagFrameStim + +from pupil_labs.realtime_api.simple import discover_one_device +from pupil_labs.real_time_screen_gaze.gaze_mapper import GazeMapper + +win = visual.Window(fullscr=True, units='height', checkTiming=False) + +gaze_circle = visual.Circle(win, radius=.02, color="red") +text = visual.TextStim(win, text='Press "ESCAPE" to exit', height=0.05) + +tag_frame = AprilTagFrameStim( + win=win, + name='tag_frame', units='norm', + pos=(0, 0), size=[2, 2], anchor='center', + h_count=4, v_count=3, + marker_size=0.125, marker_units='height', + contrast=1.0, +) +win_size_pix = convertToPix(np.array([2, 2]), [0, 0], 'norm', win) + +neon_device = discover_one_device() +gaze_mapper = GazeMapper(neon_device.get_calibration()) +screen_surface = gaze_mapper.add_surface(tag_frame.marker_verts, win_size_pix) + +while True: + frame, gaze = neon_device.receive_matched_scene_video_frame_and_gaze() + result = gaze_mapper.process_frame(frame, gaze) + + for surface_gaze in result.mapped_gaze[screen_surface.uid]: + gaze_circle.pos = surface_gaze.x, surface_gaze.y + gaze_circle.draw() + + text.draw() + tag_frame.draw() + win.flip() + + if 'escape' in event.getKeys(): + break + +neon_device.close() +``` \ No newline at end of file From 593c284da9b46c93aaafdb5aafe182b627c8783b Mon Sep 17 00:00:00 2001 From: Dominic Canare Date: Mon, 23 Sep 2024 05:39:21 -0400 Subject: [PATCH 2/3] Adds pupillometry info --- neon/data-collection/psychopy/index.md | 16 +++++++++++++++- 1 file changed, 15 insertions(+), 1 deletion(-) diff --git a/neon/data-collection/psychopy/index.md b/neon/data-collection/psychopy/index.md index 8022c1029..f18c155dd 100644 --- a/neon/data-collection/psychopy/index.md +++ b/neon/data-collection/psychopy/index.md @@ -9,7 +9,9 @@ PsychoPy users have two options for designing their experiments. ## Using PsychoPy with Neon -When using PsychoPy with Neon, we recommend making a recording in the Neon Companion app for the duration of the experiment. PsychoPy’s standard "Eyetracker Record" component can be used to start and stop recordings accordingly. For experiments that do not require screen-based gaze coordinates (e.g., pupillometry/eye state), this is all that is required. +When using PsychoPy with Neon, we recommend making a recording in the Neon Companion app for the duration of the experiment. PsychoPy’s standard "Eyetracker Record" component can be used to start and stop recordings accordingly. To save eyetracking data in PsychoPy's hdf5 format, enable the "Save hdf5 file" option within the experiment settings. + +For experiments that only require pupillometry/eye state, make sure the "Compute Eye State" setting is enabled in the companion app. For experiments that do not require screen-based gaze coordinates, this is all that is required. To use Neon for screen-based work in PsychoPy, the screen needs to be robustly located within the scene camera’s field of view, and Neon’s gaze data subsequently transformed from scene camera-based coordinates to screen-based coordinates. The Pupil Labs eyetracker plugin for PsychoPy achieves this with the use of AprilTag Markers and the [real-time-screen-gaze](https://github.com/pupil-labs/real-time-screen-gaze) Python package (installed automatically with the plugin). @@ -43,6 +45,18 @@ Two new Builder components will be available in the components list under the Ey [PsychoPy saves eyetracking data in its own format](https://psychopy.org/hardware/eyeTracking.html#what-about-the-data), but we also recommend that you record your sessions with the Companion app. You can use the "Eyetracker Record" component in Builder to automate starting and stopping/saving your recordings. +When processing eyetracking data in PsychoPy's data format, please note that PsychoPy doesn’t have distinct record types for gaze data versus eye state. If you’re collecting screen-gaze coordinates and pupillometry data, their records they will be intermixed, but they can be distinguished. + +- For screen gaze records + - `[left|right]_gaze_[x|y]` will be the screen coordinates in PsychoPy’s display units `[left|right]_gaze_z` will be `0` + - `[left|right]_eye_cam_[x|y|z]` will be `0` + - `left_pupil_measure1` and `left_pupil_measure1_type` will be `0` +- For eye state records + - `[left|right]_gaze_[x|y|z]` will be the optical axis vector + - `[left|right]_eye_cam_[x|y|z]` will be eye position + - `left_pupil_measure1` will be pupil diameter in mm + - `left_pupil_measure1_type` will be `77` + ### Example Check out our simple but complete [gaze contingent demo designed in PsychoPy Builder](https://github.com/pupil-labs/psychopy-gaze-contingent-demo) to see how it all works! From 6808bf42cf3abfc18eba14a57ccded1ec43b27f6 Mon Sep 17 00:00:00 2001 From: N-M-T Date: Wed, 25 Sep 2024 13:18:28 +0800 Subject: [PATCH 3/3] @n-m-t 's final edits --- neon/data-collection/psychopy/index.md | 41 +++++++++++++++++--------- 1 file changed, 27 insertions(+), 14 deletions(-) diff --git a/neon/data-collection/psychopy/index.md b/neon/data-collection/psychopy/index.md index f18c155dd..55a8f74cf 100644 --- a/neon/data-collection/psychopy/index.md +++ b/neon/data-collection/psychopy/index.md @@ -1,19 +1,27 @@ # PsychoPy -[PsychoPy](https://psychopy.org/) is widely used open-source software for creating and running psychophysics experiments. It enables users to present stimuli, collect data, and interface with a variety of hardware and software applications. +[PsychoPy](https://psychopy.org/) is widely used open-source software for creating and running psychophysics experiments. +It enables users to present stimuli, collect data, and interface with a variety of hardware and software applications. -PsychoPy users have two options for designing their experiments. +We have created a dedicated plugin for PsychoPy that enables Neon to be used in PsychoPy experiments. PsychoPy +users have two options for designing their experiments, both of which can be used alongside Neon: -- [Builder](https://www.psychopy.org/builder/) – Gives users a graphical interface with little or no need to write code - although it does support custom code when necessary. A Pupil Labs plugin for PsychoPy is provided to simplify interactions between PsychoPy and Neon. +- [Builder](https://www.psychopy.org/builder/) – Gives users a graphical interface with little or no need to write code - although it does support custom code when necessary. - [Coder](https://psychopy.org/coder/index.html) – Gives users the option to generate experiments or do other things programmatically, [using Psychopy like any other Python package](https://psychopy.org/api/). ## Using PsychoPy with Neon +When using PsychoPy with Neon, you can save eyetracking data in PsychoPy's hdf5 format, by enabling the "Save hdf5 file" +option within the experiment settings. But we also recommend recording in the Neon Companion app for the duration of +the experiment for data redundancy. PsychoPy’s standard "Eyetracker Record" component can be used to start and stop recordings +on the Companion Device accordingly. -When using PsychoPy with Neon, we recommend making a recording in the Neon Companion app for the duration of the experiment. PsychoPy’s standard "Eyetracker Record" component can be used to start and stop recordings accordingly. To save eyetracking data in PsychoPy's hdf5 format, enable the "Save hdf5 file" option within the experiment settings. +For experiments that only require pupillometry/eye state, make sure the "Compute Eye State" setting is enabled in the companion app. +For experiments that do not require screen-based gaze coordinates, this is all that is required. -For experiments that only require pupillometry/eye state, make sure the "Compute Eye State" setting is enabled in the companion app. For experiments that do not require screen-based gaze coordinates, this is all that is required. - -To use Neon for screen-based work in PsychoPy, the screen needs to be robustly located within the scene camera’s field of view, and Neon’s gaze data subsequently transformed from scene camera-based coordinates to screen-based coordinates. The Pupil Labs eyetracker plugin for PsychoPy achieves this with the use of AprilTag Markers and the [real-time-screen-gaze](https://github.com/pupil-labs/real-time-screen-gaze) Python package (installed automatically with the plugin). +To use Neon for screen-based work in PsychoPy, the screen needs to be robustly located within the scene camera’s field of view, +and Neon’s gaze data subsequently transformed from scene camera-based coordinates to screen-based coordinates. The plugin for +PsychoPy achieves this with the use of AprilTag Markers and the +[real-time-screen-gaze](https://github.com/pupil-labs/real-time-screen-gaze) Python package (installed automatically with the plugin). ## Builder @@ -34,18 +42,20 @@ To use Neon for screen-based work in PsychoPy, the screen needs to be robustly l The standard "Eyetracker Record" and "Region of Interest" components work with Neon. Because Neon is calibration-free, the Calibration and Validation components are unused. -Two new Builder components will be available in the components list under the Eyetracking section: "April Tag Frame" and "April Tag". These are necessary for screen-based work. +Two new Builder components will be available in the components list under the Eyetracking section: "April Tag Frame" and "April Tag" (necessary for screen-based work). - April Tag Frame: this component is recommended for most users. Using it in your Builder experiment will display an array of AprilTag markers around the edge of the screen. You can configure the number of markers to display along the horizontal and vertical edges of the screen, the size and contrast of the markers, and (optionally) the marker IDs. A minimum of four markers (2 horizontally by 2 vertically) is recommended, but more markers will provide more robust detection and accurate mapping. ![AprilTag Frame](./apriltag-frame.png) - April Tag: this component will add a single AprilTag marker to your display. It is intended for use when the April Tag Frame component cannot be used (e.g., you need to display stimuli on the edges of the display where the April Tag Frame component would place markers in the way). -### Data +### Data Format -[PsychoPy saves eyetracking data in its own format](https://psychopy.org/hardware/eyeTracking.html#what-about-the-data), but we also recommend that you record your sessions with the Companion app. You can use the "Eyetracker Record" component in Builder to automate starting and stopping/saving your recordings. +[PsychoPy saves eyetracking data in its own format](https://psychopy.org/hardware/eyeTracking.html#what-about-the-data). -When processing eyetracking data in PsychoPy's data format, please note that PsychoPy doesn’t have distinct record types for gaze data versus eye state. If you’re collecting screen-gaze coordinates and pupillometry data, their records they will be intermixed, but they can be distinguished. +When processing eyetracking data in PsychoPy's data format, please note that PsychoPy doesn’t have distinct record types +for gaze data versus eye state. If you’re collecting screen-gaze coordinates and pupillometry data, their records they will +be intermixed, but they can be distinguished. - For screen gaze records - `[left|right]_gaze_[x|y]` will be the screen coordinates in PsychoPy’s display units `[left|right]_gaze_z` will be `0` @@ -57,14 +67,17 @@ When processing eyetracking data in PsychoPy's data format, please note that Psy - `left_pupil_measure1` will be pupil diameter in mm - `left_pupil_measure1_type` will be `77` -### Example +### Example Builder Experiment Check out our simple but complete [gaze contingent demo designed in PsychoPy Builder](https://github.com/pupil-labs/psychopy-gaze-contingent-demo) to see how it all works! ## Coder -To use Neon with PsychoPy coder, we recommend interfacing directly with the [real-time API](https://docs.pupil-labs.com/neon/real-time-api/tutorials/) and, for screen-based tasks, using the [real-time-screen-gaze](https://github.com/pupil-labs/real-time-screen-gaze) package. `AprilTagFrameStim` and `AprilTagStim` classes are provided to more easily display screen markers and configure a screen-based gaze mapper. +To use Neon with PsychoPy coder, we recommend interfacing directly with the [real-time API](https://docs.pupil-labs.com/neon/real-time-api/tutorials/) +and, for screen-based tasks, using the [real-time-screen-gaze](https://github.com/pupil-labs/real-time-screen-gaze) package. +`AprilTagFrameStim` and `AprilTagStim` classes are provided to more easily display screen markers and configure a screen-based gaze mapper. +### Example Coder Experiment ```python import numpy as np @@ -110,4 +123,4 @@ while True: break neon_device.close() -``` \ No newline at end of file +```