Skip to content

Commit

Permalink
Merge pull request #20 from JonathanAMichaels/dev
Browse files Browse the repository at this point in the history
Dev
  • Loading branch information
JonathanAMichaels authored Nov 29, 2023
2 parents 7afe162 + 5a9e30d commit 8332414
Show file tree
Hide file tree
Showing 61 changed files with 6,613 additions and 2,436 deletions.
115 changes: 88 additions & 27 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,14 +10,31 @@ This toolbox will:
- For Myomatrix data:
- Combine OpenEphys data into a single binary and automatically remove broken channels
- Extract and save the sync signal sent from the behavioural task
- Perform spike sorting with a modified version of Kilosort 3.0 (wider templates)
- Perform spike sorting with a modified version of Kilosort 3.0
- Combine similar units, calculate motor unit statistics, export back to phy

## Folder Tree Structure
![Alt text](images/folder_tree_structure.png)

## Installation
### Requirements
Currently, using a Linux-based OS is recommended. The code has been tested on Ubuntu and CentOS. Windows support is experimental and may require additional changes.

Many processing steps require a CUDA capable GPU.
- For Neuropixel data, a GPU with at least 10GB of onboard RAM is recommended
- For Myomatrix data, currently only GPUs with compute capability 8.0, 8.7, or 9.0 are supported due to shared thread memory requirements
- For Myomatrix data, currently only GPUs with compute capability >=5.0 are supported due to shared thread memory requirements

Required MATLAB Toolboxes:
- Parallel Computing Toolbox
- Signal Processing Toolbox
- Statistics and Machine Learning Toolbox

Nvidia Driver:
- Linux: >=450.80.02
- Windows: >=452.39

CUDA Toolkit (Automatically installed with micromamba/conda environment):
- 11.3

### Instructions
These installation instructions were tested on the Computational Brain Science Group Server 'CBS GPU 10GB' image, and the Compute Canada servers. They may need to be adjusted if running on another machine type.
Expand All @@ -26,9 +43,16 @@ Clone a copy of the repository on your local machine (for example, in the home d

git clone https://github.com/JonathanAMichaels/PixelProcessingPipeline.git

After cloning, you can either configure a virtualenv or conda environment to run the pipeline
After cloning, you can either configure a virtualenv, conda, or micromamba environment to run the pipeline

#### Micromamba Environment (Option 1, recommended)
To install micromamba and set up a micromamba environment, follow these steps:

#### Virtual Environment
"${SHELL}" <(curl -L micro.mamba.pm/install.sh)
micromamba env create -f environment.yml
micromamba activate pipeline

#### Virtual Environment (Option 2)
To set up a virtualenv environment, follow these steps:

virtualenv ~/pipeline
Expand All @@ -41,20 +65,26 @@ Install the cupy version that matches your version of nvcc. For example, if runn
shows version 10.1, then run

pip install cupy-cuda101

#### Conda Environment
We can also create a conda environment to run the file as opposed to a virtual environment by following these steps:

#### Conda Environment (Option 3, untested)
To set up a conda environment, follow these steps:

wget https://github.com/conda-forge/miniforge/releases/latest/download/Miniforge3-Linux-x86_64.sh
bash Miniforge3-Linux-x86_64.sh
conda init
conda env create -f environment.yml
conda activate pipeline

#### Final Installation Steps
If you are processing Myomatrix data, open matlab and confirm that all mex files compile by running
Open matlab and confirm that all mex files compile by running

WARNING: make sure to activate the pipeline environment before running these commands

matlab -nodesktop
cd PixelProcessingPipeline/sorting/Kilosort-3.0/CUDA/
mexGPUall

Compile codes necessary for drift estimation and install supplementary packages
(Optional) Compile codes necessary for drift estimation and install supplementary packages

cd PixelProcessingPipeline/registration/spikes_localization_registration
python3 setup.py build_ext --inplace
Expand All @@ -66,11 +96,17 @@ Compile codes necessary for drift estimation and install supplementary packages

## Usage
Organize each experiment into one directory with a Neuropixel folder inside (e.g. 041422_g0), a Myomatrix folder (e.g. 2022-04-14_09-48-02_myo, which must have _myo at the end) and any .kinarm data files generated.
The Myomatrix folder must be organized either as 'folder_myo/Record Node ###/continuous/' for binary open ephys data,
or as 'folder_myo/Record Node ###/***.continuous' for open ephys format data.
The Myomatrix folder must be organized either as 'folder_myo/Record Node ###/continuous/' for binary open ephys data, or as 'folder_myo/Record Node ###/***.continuous' for open ephys format data.

Each time a sort is performed, a new folder will be created in the experiment directory with the date and time of the sort. Inside this folder will be the sorted data, the phy output files, and a copy of the ops used to sort the data. The original OpenEphys data will not be modified.

#### Micromamba Activation
Every time you open a new terminal, you must activate the environment. If micromamba was used, activate the environment using

micromamba activate pipeline

#### VirtualEnv Activation
Every time you open a new terminal, you must activate current source. If virtualenv was used, activate the source using
If virtualenv was used, activate the source using

source ~/pipeline/bin/activate

Expand All @@ -82,19 +118,47 @@ If a conda environment was used, activate it using
#### Final Usage Steps
The first time you process an experiment, call

python3 pipeline.py -f /path_to_experiment_folder
python pipeline.py -f "/path/to/sessionYYYYMMDD"

This will generate a config.yaml file in that directory with all the relevant parameters for that experiment generated automatically. Open that file with any text editor and add any session specific information to the Session parameter section. For example, if you collected Myomatrix data you must specify which channels belong to which electrode and which channel contains the sync information, since this information cannot be generated automatically.
This will generate a `config.yaml` file in that directory with all the relevant parameters for that experiment generated automatically. Open that file with any text editor and add any session specific information to the Session parameter section. For example, if you collected Myomatrix data you must specify which channels belong to which electrode and which channel contains the sync information, since this information cannot be generated automatically.

##### Configuration Commands
Editing the main configuration file can be done by running the command below:

python3 pipeline.py -f /path_to_experiment_folder -config
python pipeline.py -f "/path/to/sessionYYYYMMDD" -config

To edit the configuration file for the processing Myomatrix data, run

python pipeline.py -f "/path/to/sessionYYYYMMDD" -myo_config

To edit the configuration file for the processing Neuropixel data, run

python pipeline.py -f "/path/to/sessionYYYYMMDD" -neuro_config

##### Spike Sorting Commands
To run a sort on the Myomatrix data, run

python pipeline.py -f "/path/to/sessionYYYYMMDD" -myo_sort

To run a sort on the Neuropixel data, run

python pipeline.py -f "/path/to/sessionYYYYMMDD" -neuro_sort

If the config.yaml is correct, you can run the pipeline with all steps, for example
##### Plotting with Phy Command
For plotting the latest myomatrix sort with Phy GUI, run

python3 pipeline.py -f /path_to_experiment_folder -full
python pipeline.py -f "/path/to/sessionYYYYMMDD" -myo_phy

Alternatively, you can call any combination of
For plotting a previously saved myomatrix sort with Phy GUI, call below with the corresponding datestring

python pipeline.py -f "/path/to/sessionYYYYMMDD" -d YYYYMMDD_HHMMSS -myo_phy

##### Chaining Commands Together
If the `config.yaml` is correct, you can run the pipeline with all steps, for example

python pipeline.py -f "/path/to/sessionYYYYMMDD" -full

Alternatively, you can call any combination of:

-config
-registration
Expand All @@ -103,21 +167,18 @@ Alternatively, you can call any combination of
-neuro_post
-myo_config
-myo_sort
-myo_post
-myo_phy
-lfp_extract

to perform only those steps. For example, if you are processing Myomatrix data, run
to perform only those steps. For example, if you want to configure and immediately spike sort, run

python3 pipeline.py -f /path_to_experiment_folder -myo_sort -myo_post
python pipeline.py -f "/path/to/sessionYYYYMMDD" -config -myo_config -myo_sort

To edit the configuration file for the processing Myomatrix data, run
If you want to run a grid search over a range of KS parameters, edit the `Kilosort_gridsearch_config.py`
file under the sorting folder to include all variable combinations you want to try. Be aware of the combinatorics so you don't generate more sorts than you expected. Then open the config file and set the gridsearch parameter to True, for example by running

python3 pipeline.py -f /path_to_experiment_folder -myo_config

To edit the configuration file for the processing Neuropixel data, run
python pipeline.py -f "/path/to/sessionYYYYMMDD" -config -myo_sort

python3 pipeline.py -f /path_to_experiment_folder -neuro_config

## Extensions

This code does not currently process .kinarm files or combine behavioural information with synced neural data. This may be added at a later date.
Expand Down
28 changes: 21 additions & 7 deletions config_template.yaml
Original file line number Diff line number Diff line change
@@ -1,6 +1,11 @@
---
# Configuration file
# Auto-generated parameters
# GPU to use for kilosort, (a list of integers, e.g., [1,2,5,6])
GPU_to_use: [0]
# number of Kilosort jobs to run at once (must be <= number of GPUs, will run in parallel if >1)
# setting num_KS_jobs > 1 can only be used when do_KS_param_gridsearch is True
num_KS_jobs: 1
# path of working directory (must be provided in command line)
folder:
# path to neuropixel directory
Expand All @@ -9,6 +14,8 @@ neuropixel:
num_neuropixels:
# path to myomatrix directory (should end in _myo)
myomatrix:
# specify recordings to process (can be [all] or a list of integers, e.g., [1,2,5,6])
recordings: [1]
# whether to concatenate myomatrix data
concatenate_myo_data: False
# set bandpass filter settings for myomatrix data
Expand All @@ -25,24 +32,31 @@ script_dir:
Registration:
# Sorting parameters
Sorting:
num_KS_components: 9
do_KS_param_gridsearch: False
# Session-specific parameters
Session:
trange:
- 0
- 0
myo_chan_map_file:
- bipolar_test_kilosortChanMap.mat
- bipolar_test_kilosortChanMap.mat
myo_chan_map_file:
- linear_16ch_RF400_kilosortChanMap_unitSpacing.mat
- linear_16ch_RF400_kilosortChanMap_unitSpacing.mat
myo_chan_list:
- [1, 16]
- [17, 32]
# Remove bad channels from myomatrix data
# This can be a sequence of booleans or a sequence of lists:
# True for automatic bad channel removal, False to include all channels.
# Provide an integer list of channels to remove those channels for that session, e.g., [1,2,3,4]
## Remove bad channels from myomatrix data
# This can be a sequence of booleans, of strings, or of lists:
# Booleans: True for automatic bad channel removal (defaults to reject below median), False to include all channels.
# Strings: Provide 'median', 'mean', 'mean-1std', 'percentileXX', or 'lowestYY' (XX,YY are numeric, 0<XX<100, and 0<YY<Nchans)
# to remove channels below the corresponding threshold.
# Lists: Provide an integer list of channels to remove those channels for that session, e.g., [1,2,3,4]
remove_bad_myo_chans:
- True
- True
remove_channel_delays:
- True
- True
myo_analog_chan: 33
myo_muscle_list:
- Bicep
Expand Down
Loading

0 comments on commit 8332414

Please sign in to comment.