- split swanson areas
- trainig plots
- fix datahandler on SDSC for ONEv2
- quiescence period extraction
- ONEv2 requirement
- training plots
- can change download path for atlas
- use session path in training status task
- trials extractors support omissions trials from neuromodulator tasks
- SpikeSortingLoader and EphysSessionLoader utils functions to load related objects such as drift
- Training status tasks upload training date and session to subject json
- Query training criterion function added to brainbox.training
- Swanson vector - option to annotate top/bottom 5 regions based on values, or pass in a list of acronyms
- EphysQC can run without connection to ONE
- show atlas names in swanson plot
- allow user to define mask regions in swanson plot
- register_session: handle lack of taskData in passive sessions
- get_lab function now gets lab name from session path subject name
- create_jobs now returns pipelines
- remove unused jupyter and jupyterhub requirements
- fix mapping for atlas svg plots
- support for multiple task protocols within a session
- extract protocol period from spacer signals
- function for changing subject, collection, number or date in a settings file
- function to retrieve task protocol number from experiment description
- dataset type validation supports null filename patterns
- oneibl.registration uses one.registration client as superclass
- experiment description files are created and registered in legacy pipelines
- QC sign-off keys are added to session JSON field
- class-based note template API for attaching session notes for QC sign-offs
- protocol and procedures now taken from experiment description file
- handle truncated first spacer in passive extraction
- fix the Perirhinal area layer color in Allen Atlas
- fix region volume computation
- vectorised swanson flatmap
- brainbox trial ITI loader
- vectorised atlas plots
- ephys session loader
- bwm release documentation
- mock gpu lock in tests
- use cluster_ids in metrics
- camera qc allows for string values
- deprecate brainbox.io.spikeglx.stream instead use Streamer
- copy logics take into account presence of experiment description files
- option for error bars in training plots
- cam qc for dynamic pipeline uses sync type to determine number of cameras
- brainbox.modeling removed
- brainbox.io.one.load_trials_df removed, use one.load_object(...).to_df() instead
- AWS DataHandler refactored (ibllib.oneibl.aws removed)
- raise error when saving an empty dataset during extraction
- removed deprecated dsp packages (ibllib.dsp, ibllib.ephys.neuropixel, ibllib.io.spikeglx)
- register datasets as a revision if protected on Alyx
- frameData and timestamps both optional datasets for camera QC
- for 3B probes don't glob the sync files as fails for NP2 case
- support iblutil 1.4 (iblutil.util.get_logger -> iblutil.util.setup_logger)
- Compute wheel velocity using a low-pass filter instead of a Gaussian window smoothing
- Save widefield output plots in png format
- Widefield Sync task uses sync_collection argument
- Improve boundary display for atlas slice plots
- adjust ONE-api requirement to redownload on AWS backend when md5sum mismatch
- units quality metrics use latest algorithms for refractory period violations and noise cut-off
- photometry extraction: recover from corrupt DAQ signal and reversed polarity of voltage pulses
- swanson flatmap: the algorithm to propagate down the hierarchy has been refined
- set exists flag to false for all data repos when registering datasets with tasks
- SessionLoader error handling and bug fix
- extraction pipeline: fix unpacking of empty arguments field from alyx dict that prevents running task
- atlas: gene-expression backend and MRI Toronto atlas stretch and squeeze factors (Dan/Olivier)
- FDR correction (Benjamin-Hochmann) to correct for multiple testing optional (Guido)
- SpikeSortingLoader can be used with ONE local mode (Julia)
- dynamic pipelines creation: if the acquisition description file exists, task sequences get created
- session registration procedures and projects, only add a couple of fields in the json by default
- new modalities:
- photometry extraction (Mainen lab)
- widefield extraction (Churchland lab)
- Spike sorting task: parse new pykilosort log format
- Session loader
- Adding brainbox.io.one.SessionLoader for standardized loading of session data
- Changes to TaskQC, VideoQC and DLCQC thresholds and aggregation
- Unfreezing numpy version
- Adding probabilistic brain regions lookup in ibllib.atlas.atlas.BrainAtlas.get_labels
- Removing two voxelless areas from beryl atlas
- Minor updates to audio events detection and audio welchogram
- Hotfix: don't overwrite full settings if iblrig on untagged version
- Hotfix: pseudo session biased generation contrast distribution
- Hotfix: Density displays had non-existing colormaps in snapshots QC tasks
- Generate extra training plots based on psychometric curves
- Hotfix: fix previous hotfix with incorrect package version number
- Hotfix: datahandler sets local paths in init
- Hotfix: globus imports were mixed one.globus and one.remote
- Deprecated ibllib.version
- Fix Globus patcher
- Add SpikeSorting Loader samples2times function
- Fix atlas.BrainCoordinate.xyz2i functions to not quietly wrap indices out of volume bounds.
- Set jobs to Held if parent jobs are Started or Abandoned as well
- Reverse matplotlib colorbars in density displays
- Fixes to plotting in training_status
- ibllib.pipes.training_status: pipeline to compute training status of mice on local servers, new TrainingStatus task (Mayo)
- Fix swanson regions (Olivier)
- Swanson to Beryl mapping
- ibllib.atlas: add the Swanson flatmap backend (Olivier)
- ibllib.io.extractors: output of task extractions are trial tables, not individual datasets (Miles)
- Documentation: data release examples (Mayo)
- ibl-neuropixel new repository contains
ibllib.dsp,illlib.ephys.neuropixelandibllib.io.spikeglxmodules (Olivier) - brainbox.task.closed loop get impostor targets to evaluate null distribution (Brandon)
- minimum supported version of Python is 3.8 (Michele)
- Set video compression jobs to priority 90
- Set jobs to Held if parents are Waiting
- brainbox.io.one.SpikeSortingLoader: option to load using
collectionargument - Restructuring of how jobs are run on local servers, run large jobs as service
- Allow parent tasks to be 'Incomplete' to run task on local server
- Change one base_rul for dlc_qc_plot on cortexlab
- Fix moot release accident
- Data handler connects to correct alyx database on cortexlab
- Fixes to EphysPostDLC
- Small change to storing in dsp.voltage.decompress_destripe_cbin function
- Small fixes to local server task queues
- Authenticate alyx user in Task class
- Some fixes to make dlc_qc_plot in EphysPostDLC more reliable
- SpikeGlx:
- supports reading of flat binary files without meta data
- atlas.Regions:
- add support for region ordering in BrainRegions object
- bugfix in transposing of RGB image data for coronal slice plots
- voltage: decompress cbin
- add support for custom spikeglx.Reader
- Fix in EphysDLC task to not return multiple copies of outputs
- Loading examples for different IBL data types
- Fix for probe syncing when Nidq and probe pulses don't match
- Account for new ONE tables in ond datahandler
- Add bad channels plots into RawEphysQc task
- deprecation warnings and documentation for spike sorting loading method
- bugfix: remove lru_cache on AllenAtlas class for iblviewer
- Adding EphysDLC task in ephys_preprocessing pipeline
- NOTE: requires DLC environment to be set up on local servers!
- Fixes to EphysPostDLC dlc_qc_plot
- Add lfp, aprms, spike raster and behaviour report plots to task infastructure
- Computation of apRMS in decompress_destripe_cbin before conversion to normalised units
- Add SpikeSortingLoader class in brainbox.io.one
- Fixes and better logging for EphysPostDLC task
- Remove atlas instantiation from import of histology module
- Write electodeSites dataset at end of histology and spikesorting pipelines if insertion resolved
- Removal of native test train support and improved documentation / tests for GLMs
- LFP destriping functions
- Version number is now obtained from ibllib/init.py
- New ReportSnapshot class
- DLC QC plots, as part of EphysPostDLC task
- BadChannelsAP plots for ephys QC
- Fix typo in camera extractor
- SpikeSorting task overwrites old tar file on rerun
- Snapshot class to register images as notes in Alyx
- Ephys pipeline: RawEphysQC: outputs channels labels as part of pipeline
- Ephys pipeline: spike sorting
- dsp.voltage: destripe detects bad channels and interpolate them
- synchronisation 3B: gives a 10% leeway to throw exception
- spikes sorting task outputs the RMS plot
- Setting tasks to Waiting if they encountered lock (status -2)
- Setting tasks to Incomplete if they return status -3
- Completed tasks set held dependent tasks to waiting
- Adding PostDLC task to compute pupil diameters, licks and DLC QC
- Trial wheel extraction: use alternative sync method when first on fails
- bugfix: timer for tasks was returning None
- Add setup method in tasks.py into try except to catch globus download errors
- Add input and output signatures to all ephys tasks
- Add datahandler to task to download and upload data based on location where task is run
- Spike sorting and EphysVideoSyncQc download data on local servers if not available
- brainbox.io.one load_spike_sorting_fast: bugfix returns acronyms
- creates sequence files for spikesorting
- GPU tasks have a lock - local data handler doesn't instantiate one
- allows more than 2 probes in ephys computer probe creation
- brainbox.io.one load_spike_sorting fast: merge channel info in clusters
- brainbox.io.one generic function to interpolate channels after alignment
- dsp: spike detection by voltage thresholding and cadzow filtering docs
- ibllib.io.spikeglx: get geometry from metadata
- RawEphysQC outputs basic detection spike rates
- ibllib.atlas.regions: add new mapping cosmos and revise Beryl
- destriping as pykilosort internal pre-processing
- NP2 probe framework for splitting shanks and LFP band
- Extension of task module to rerun from different locations
- RawEphysQC tasks computes median RMS from samples for .ap data (stored in _iblqc_ephysChannels.RMS)
- New EphysQC class
- Fix issue with RawEphysQC that was not looking in local Subjects folder for data
- Fix ensure_required_data in DlcQc
- Split jobs.py run function in two, one running large tasks (video compression, dlc, spike sorting), one the rest
- Ensure RawEphysQC runs both probes if one fails
- pykilosort error handling
- passive extraction robust to frame2ttl flickers
- pykilosort bugfix after low yield results
- ephys trials extraction when audio FPGA starts on up state
- pykilosort pipeline: output correct version number / fix log file name
- extraction fixes: passive extraction spacers and camera times format
- small bugfix for the sequential selection
- ephys trials extraction: audio from FPGA: up-state and TTLs cleanup from camera wiring
- passive extraction: improved spacers detection
- ephyscellsqc: bugfix on probe creation for pykilosort subfolder
- task version 6.2.5 specific audio extractor
- camera times: index by video length when GPIO unstable
- extraction fixes: passive extraction spacers and camera times format
- small bugfix for the sequential selection
- pykilosort pipeline: output correct version number / fix log file name
- ephys trials extraction when audio FPGA starts on up state
- pykilosort bugfix after low yield results
- passive extraction robust to frame2ttl flickers
- pykilosort error handling
- ONE2 released on the master branch
- Pykilosort is the new default spike sorter in the pipeline
- oneibl and alf now deprecated; moved to separate repository
- other ibllib miscellanea moved to iblutil repository
- test database is now configurable with an env var
- brainbox.modeling: New API for using linear and poisson models, doing sequential feature selection.
- bugfix: spike sorting continues even if probe information not entered (personal projects)
- TaskQC iteration: aggregation excludes iti delays, frame2ttl clean signal used to compute QC instead of raw
- optogenetics for UCL task variant
- check Nvidia status before launching spike sorting
- fix public one params
- fix ks2 logging output
- set default one params to public credentials
- TaskQC: exclude stim_freeze from overall session qc
- Get modality from task type to include personal projects
- Bugfix Atlas ccf coordinates order
- Tool to label Critical sessions / insertions reasons
- Successful ONE setup on instantiation when params file doesn't exist
- Video QC optimization
- New task types for widefield imaging
- Add revision and default dataset to register dataset
- Fix for camera extraction failure for novel protocols
- CameraQC wheel alignment check more robust to short videos and large alignment failures
- Parameters raises FileNotFound error if no defaults are provided
- opto-ephys tasks
- DLC QC
- Do not block session creation if error
- Removed deprecated numpy dtypes
- Fix trajectory query bug
- GPIO pin states loaded as bool array
- Machine info and log append in Task class
- Only delete session flag file after processing
- Extractor support for widefield imaging protocol
- hotfix: fix test_modelling error after removing pytorch
- hotfix: maintain compatibility with Python 3.7
- ibllib.atlas: backend region mapping is a lateralized version of the Allen atlas "Allen-lr" The default behaviour in the Atlas is to remap according to the original mapping "Allen"
- camera timestamps: extraction of the exact camera time stamps by cross-examination of bonsai timestamps, audio pulses and camera pulses
- ephys extraction: remove short TTL pulses of frame2ttl in task extraction
- ibllib.atlas.regions remapping option
- optogenetics pybpod: extraction of laser probabilities
- MANIFEST.in fix
- Added fixtures file for passive protocol extraction
- Amplitudes fix:
- sync_probes doesn't require raw binary file and looks for meta-data files instead
- tar file of intermediate spike sorting results gets uploaded on flatiron
- Ephys Task extraction: ephys extraction doesn't fail when bpod started before ephys
- histology: brain atlas can handle insertion on the sides
- optogenetics dataset type _ibl_trials.laser_stimulation for training sessions
- spikeglx analog sync thresholding removes DC offset option
- ephys FPGA behaviour extraction: bpod sync with FPGA assumes possible missing FPGA fronts
- Ephys pipeline:
- passive extractor
- units QC
- add the available space of system and raid volumes in local servers reports.
- following flatiron server change, add auto-redirect to https
- add widefieldChoiceworld tasks for pipeline
- add opto laser tasks for pipeline
- register ch when mtscomp runs properly
- probes_description runs even if .cbin doesn't exist
- allows dataset registration on errored task
- ks2task import conflict fix
- Ks2 task does not depend on ephys pulses
- Ephys mtscomp
- ks2 task registers first probe even if one failing
- mtscomp task register .ch and .meta even if .cbin doesn't exist
- move ibllib tests to tests_ibllib
- brainbox atlas plot functions
- Ephys extraction:
- synchronisation between probes computed in the ephysPulses job
- spike sorting resync done directly after KS2 output
- unit-based metrics have their own task
- create local server tasks only on raw_session.flag
- ephys alignment QC
- hotfix: ibl errors inherit Exception, not BaseException
- hotfix: partial qc task extractor keeps FPGA stim times
- create tasks looks for create_me.flags
- add Karolina's optogenetics tasks for extractions
- histology probe QC pipeline and final locations dataset export
- numpy needs upgrading >= 1.18
- session creation skips alyx procedure for unknown task protocol (custom projects)
- task extraction:
- Habituation QC
- ephys extraction StimOffTimes fix
- ibllib.atlas
- allen csv Atlas part of package is not installed in dev mode
- improved slicing performance
- ephys extraction: mtscomp registers ch file on run and re-runs bis
- mtscomp registers ch file on run and re-runs
- ffmpeg nostdin option as jobs were stopped in background on a server
- QC base class
- Support for task QC on FPGA data
- TaskQC run during task extraction
- local server: catches error when subject is not registered in Alyx
- ibllib.atlas.AllenAtlas
- re-ordered the volumes in c-order contiguous ml-ap-dv efficient coronal shapes
- top/bottom surface extraction
- Ephys extraction SyncSpikeSorting: specify different dir for ks2 ouput and raw ephys data
- Ephys extraction ks2: mkdir for scratch more robust
Ephys extraction bugfixes:
- RawEphysQC: No object "ephysTimeRmsAP" found
- EphysMtsComp,RawEphysQC, EphysPulses : ValueError: mmap length is greater than file size
- Ks2: ks2 job cleans-up temp dir
- ONE offline mode and cache dataset table to speed up reloading of large datasets (Olivier)
- ALF io naming conventions on loading objects (Miles)
- KS2 Matlab ephys pipeline tasks (Olivier)
- Support for running QC on biased and training sessions (Nico)
- metrics_df and passed_df properties in BpodQC obj for qcplots (Nico)
- Added missing unittest to stim_move_before_goCue metric (Nico)
- ibllib.pipes.training_preprocessing.TrainingAudio
- ibllib.pipes.training_preprocessing.TrainingAudio: returns files for registration and proper status. (Olivier)
- ibllib.atlas: compute nearest region from probe trajectory (Mayo)
- Local server jobs:
- fix wheel moves size mismatch extractor error
- only look for raw_session.flag for ephys extraction to avoid race conditions
- Ephys extraction:
- spike amplitudes in Volts
- added waveforms samples dataset to use Phy from Flatiron datasets
- ONE performance:
- Metaclass implementation of UniqueSingletons for AlyxClient
- Multi-threaded downloads
- Added JSON fields methods to AlyxClient
- QCs: Bpod and ONE QC features, basic plotting, examples
- Added permutation test, comparing a metric on two sets of datasets, by shuffling labels and seeing how plausible the observed actual difference is Sped up calculation of firing_rate
- ephys extraction: updated extracted metrics, including a new contamination estimate and drift metrics.
- ibllib.io.spikeglx
- Hotfix: rig transfer - create probes. One variable used before assignation.
- ONE.load overwrites local file if filesizes different or hash mismatch
- ephys extraction:
- registration sets the session.procedure field to acute recording
- bugfix synchronization on re-extraction: always recompute spike.times from spike.samples
- ephys registration sets the session.procedure field to acute recording
- training extraction:
- added biasedVisOffChoiceWorld as training extractor
- wheel data
- dropping support for wheel velocity, not extracted anymore
- bugfix: Include sessions data files for ephys mock
- bugfix: Single probe 3B gets synchronized
- bugfix: Include sessions data files in pip package
- Big brainbox merge and release
- bugfix: clusters.metrics spiking rates accurate
- probability left for ephys choice world contain generative probabilities, not outcomes
- ONE Light Windows fixes
- Installation documentation separates conda and virtualenv options
- Conda yaml environement file for ibllib
- ONE Light for behaviour paper data release
- ONE() standard syntax matching the one light examples
- Alyx registration: add md5, version and filesize to the pipeline registration
- Data Patcher: allows to register data from anywhere through FTP/SSH/GLobus
- ONE Light for behaviour paper data release
- Ephys extraction: left probability bug when sequence was 0 - fixed
- Ephys extraction:
- left probability extracted properly
- add robustness to audio fronts extraction in FPGA
- Passive stimulus: raw data registered in pipeline
- Training extraction: microphone extraction for habituation sessions
- ALF: specific to_dataframe method for Bunch
- Ephys extraction: handle fringe case where recording is interrupted in the middle
- Wheel extraction: if rotary encoder version is outdated and stores data in the wrong unit, auto-detect and output in seconds even for new versions
- FPGA/bpod events synchronization performed even when their counts do not match
- Updated requirement versions for mtscomp and phylib
- wheel extraction outputs a timestamps attribute, not times
- make the wheel extraction more robust
- Ephys extraction:
- un-synchronized spike sortings not uploaded on flat-iron
- reaction times extracted
- valve-open times bugfix
- Wheel extraction:
- training wheel position and timing are now correct
- ephys & training: units: radians mathematical convention
- ONE:
- Alyx client handles pagination seamlessly
- cross-platform get of session folder for rig computer copy to server
- spikeglx.verify_hash() method to check file integrity after transfers/manipulations
- create wirings settings files on ephys computer transfer to server
- Ephys Extraction:
- duplicate probe.trajectories bugfix
- extraction works with unoperational fram2ttl at beginning of ephys session
- clusters.metrics.csv has consistent size with npy cluster objects
- ephys transfer: create ephys extraction flags after the transfer is complete
- Rename spike.times on failed sync to reflect the clock as per ALF convention
- sync 3A fails if first cam event whithin 200ms of start
- compress ephys goes through a tempfile to not interfere with transfers/globbing
- Ephys Extraction (phylib)
- convert ks2 amplitudes to volts for spikes.amps, clusters.amps. templates.waveforms, clusters.waveforms to get uV
- generates Cluster UUIDs file
- individual spike depths computed from PC features
- Ephys Synchronization
- use frame2TTL split for 3A by default, if not found look for right_camera
- output individual probe sync in ALF timestamps format
- registration ignores ks2alf probes subfolders
- fix typo in raw qc dataset types
- Alyx registration adds the relative path to session root as dataset.subcollection
- Ephys extraction:
- split probe folders output alf/probe00 and alf/probe01 instead of merge
- outputs templates.waveforms and clusters.waveforms in sparse arrays
- outputs probes.description and probes.trajectory
- renamed the raw ephys QC output
- outputs clusters.metrics
- Bugfixes:
- 3B raw ephys QC output ap.file not found on nidq object
- 3A sync probe threshold set to 2.1 samples
- transfer scripts from ephys/video/rig computers to local servers
- bugfix spigeglx.glob_ephys_files when metadata file without ap.bin file
- Transfer rig data takes into account session type (ephys/training) to create flags
- Ephys video compression in pipeline
- Ephys audio compression in pipeline
- Ephys extraction: provide full 3B default wirings if files do not exist.
- Ephys extraction: merge sync ephys in the pipeline overwrites ks2_alf directory if it already exists
- Added
biasedScanningChoiceWorldtask to biased extractor for Zador lab
spikeglx.Readersupports mtscomp ephys binaries- server pipeline for compression of ephys files
brainbox.singlecell.pethswith testsbrainbox.processing.bincount2Dsupports aggregation on fixed scale- simple examples script and notebook
- examples/brainbox/plot_peths.py: by Matt. W.
- examples/brainbox/rasters by Michaël S.
- Allen Atlas framework and probe registration base functions
- server pipeline for audio extraction of training sessions