Calcium Imaging notes

⚠️ TODO: migrate over notes

Calcium Imaging Research Overview

Experiment Duration, Channels, Subjects

Common Experiment Types

Other Modalities Commonly Used Simultaneously

Data

Data Size

• A 10 minute video is around 4GB
• Longest recordings can last 48 hours (~1 TB)

Data Format

• Typical raw data disk format: .avi (typically using FFV1 data compression)
• By default MiniAn will load from these raw files and then store variables in the xarray-augmented zarr format for better support of parallel and out-of-core computation during processing and analysis.
• Typical array dtype: uint8
• The actual data (numpy) arrays (without coordinates/metadata) in MiniAn are almost always represented as dask arrays - essentially lists of tasks that can be executed to obtain and use the data, instead of actual data in RAM.

Data Generation/Simulation

Sim Software:

• Minian has a corresponding validation repo which includes simulated data generation.
  • The validatio repo depends on minian package, but minian is only an osx-64 package, not osx-arm64 so on my M1 I had to create an osx-64 conda env to install minian as a package. Also trying to install minian from source hasn't worked (without unpinning everything) - pip install -e ../minian was not working because of some numba dependencies, blah blah, and conda develop . is an unmaintained nightmare that didn't work either.

Data Generation plans:

1. Create a simple random noise video generator with minimal dependencies
2. Create a simulated data generator based on the Minian validation repo work

Lists/Sources of real data

Specific real datasets of interest

• Small: There is a small(er) dataset as part of the Minian docs
  • The docs provide instructions for downloading the demo data: minian-install --demo from a conda env with minian installed. The script for this command is here.
  • This installs a set of .avi files into minian/demo_movies, totaling 722 MB on disk
• Medium: There is some real data on figshare which is used for Minian validation.
• Large: Larger datasets provided by the Cai lab:
  • These data are hosted on s3://neuro.holoviz.org/cai/
  • For access, holoviz members can create a user profile with AWS Access Key ID and the AWS Secret Access Key.
  • Then use aws configure cli to add the key
  • Test access with aws s3 ls to list holoviz buckets
  • If that works, you should be able to access the neuro data.
    • If you want a full local copy of the data, run aws s3 sync s3://neuro.holoviz.org/cai/ <target_local_dir>, but note.. this data is currently 88 GB on disk so it will take a while to download.

Software

Common Analysis Packages (Especially Python)

Common Visualization Solutions (Especially Python)

• mesmerize-core and mesmerize-viz is CaImAn-specific viz for by same people building fastplotlib

Processing and Analysis