Merge pull request #1449 from flatironinstitute/dev

Merge dev for release 1.11.4

.github/workflows/run_tests.yml

@@ -29,11 +29,12 @@ jobs:
           environment-file: environment-minimal.yml
           activate-environment: caiman
           conda-solver: libmamba
+          miniforge-version: latest
 
       - name: Install OS Dependencies
         shell: bash -l {0}
         run: |
-          sudo apt-get install libglapi-mesa libegl-mesa0 libegl1 libopengl0 libgl1-mesa-glx
+          sudo apt-get update && sudo apt-get install libglapi-mesa libegl-mesa0 libegl1 libopengl0
 
       - name: Install Dependencies
         shell: bash -l {0}

CONTRIBUTING.md

@@ -1,17 +1,17 @@
-# CaImAn contributors guide :hammer:
-CaImAn is an open source project where *everyone* is welcome and encouraged to contribute. We have external contributors from all over the world, and we are always looking for more help. This guide explains how to contribute: if you have questions about the process, please feel free to reach out at [GitHub Discussions](https://github.com/flatironinstitute/CaImAn/discussions). Everyone needs help contributing and finds git/GitHub confusing, so please don't be shy about asking. 
+# CaImAn guide to contributing
+CaImAn is an open source project with contributors from many people at many places. Help is available on [Gitter](https://app.gitter.im/#/room/#agiovann_Constrained_NMF:gitter.im) or [Github Discussions](https://github.com/flatironinstitute/CaImAn/discussions).
 
-There are many different ways you can contribute to Caiman. The first and easiest way is to bring problems to our attention: if you find a bug, or think there is a feature that is lacking in Caiman, please [open an issue at Github](https://github.com/flatironinstitute/CaImAn/issues). You can also contribute just by *participating* in the different forums.   
+If you run into issues or want to suggest features, you can [open an issue on Github](https://github.com/flatironinstitute/CaImAn/issues).
 
 Second, let's say you want to improve something yourself:
 
 - Documentation like what you are currently reading
 - The demo notebooks
 - The code base
 
-We welcome *all* such contributions. To make them, you need to make changes on your local version of Caiman and then push make a *Pull Request* (PR) to our GitHub repository. We will walk through this process in the rest of the document. 
+To contribute, start by making changes on your local version of Caiman and then push make a *Pull Request* (PR) to our GitHub repository. We will walk through this process in the rest of the document. 
 
-Before you go through the work required to improve something, we recommend that you let us know your plans on GitHub either in discussions or issues. This way, we can avoid duplicated effort (if someone is already working on it), or wasted time (it could turn out the changes might not be feasible right now because it conflicts with some other major feature we are working on). If needed, can usually set up a video chat to talk about a feature proposal if that works for you.
+Before you go through the work required to improve something, we recommend that you reach out on GitHub by filing a feature request in an issue. This way, we can avoid duplicated effort (if someone is already working on it), or wasted time (we may not agree on direction).
 
 ## Background: how do pull requests work?
 In this section we'll give general background on how making a contribution/PR works. If you know this stuff and just want to get started quickly, feel free to skip to the next section.
@@ -28,7 +28,7 @@ The workflow for contributing to Caiman is roughly illustrated by the numbers in
 3) Make a PR: this is when you request that your changes become merged into `dev` at Caiman. This merge won't be immediate, you will get feedback on your code, and probably be asked to make some changes. 
 4) Periodically, as features accumulate in the `dev` branch (every month or so), the `dev` branch will be merged with `main`. This will become a new version of Caiman that people install when they run `mamba install caiman`. 
 
-Below we have instructions on how to do all of the above steps. While all of this may seem like a lot, some of the steps are extremely simple. Also, once you have done it once, you will have the recipe and it will be pretty easy. Finally, it is a very rewarding experience to contribute to an open source project -- we hope you'll take the plunge!
+Below we have instructions on how to do all of the above steps. 
 
 ## First, create a dedicated development environment
 If you have downloaded Caiman for standard use, you probably installed it using `conda` or `mamba` as described on the README page. As a contributor, you will want to set up a dedicated development environment. This means you will be setting up a version of Caiman you will edit and tweak, uncoupled from your main installation for everyday use. To set up a development environment so you can follow the workflow outlined above, do the following:
@@ -57,7 +57,7 @@ Go to the [Caiman repo](https://github.com/flatironinstitute/CaImAn) and hit the
 
     This installs Caiman directly from the downloaded source code. The `-e` stands for 'editable': when you edit the files, the changes should immediately be reflected in the code you run.
 
-Note this section is partly based on the excellent [docs from Matplotlib](https://matplotlib.org/devdocs/devel/development_setup.html#installing-for-devs).
+Note this section is partly based on the [docs from Matplotlib](https://matplotlib.org/devdocs/devel/development_setup.html#installing-for-devs).
 
 
 ## Second, work on a feature 
@@ -86,21 +86,15 @@ Note that all PRs are reviewed by other programmers. This is an important part o
 You may be asked to make some changes (or to *think* about making some changes). You will sometimes need to do more some more work on your branch and make more changes after making an initial PR. In this case, the workflow is simple: you will work within your your local `my_feature` branch as before, and run the `push` command again. Conveniently, this will automatically push the changes to the same work-in-progress PR at Caiman. Eventually, the feature will be merged into `dev` and your work is done!
 
 ## Fourth, wait for the work to show up in main :clock8: 
-Once your work is done, the `dev` branch will eventually be merged into `main`  by the developers who maintain Caiman (label 4 in the figure). This is done every month or two, and is the stage when your work will actually be available to the people who download Caiman. It's at this point your name will appear when you click on the [list of Contributors](https://github.com/flatironinstitute/CaImAn/graphs/contributors) at GitHub. Please give yourself a pat on the back -- we really appreciate the folks who go through all this work to help make the package better!
+Once your work is done, the `dev` branch will eventually be merged into `main`  by the developers who maintain Caiman (label 4 in the figure). This is done every month or two, and is the stage when your work will actually be available to the people who download Caiman. It's at this point your name will appear when you click on the [list of Contributors](https://github.com/flatironinstitute/CaImAn/graphs/contributors) at GitHub. 
 
 # What next?
 Once you have gone through the above steps, you can delete your local feature branch. Before working on a new feature, you will want to make sure that your fork stays up to date with Caiman. You can do this with the user interface at GitHub (there is a button to sync up your repo with the original repository on a particular branch). 
 
-Nobody remembers all the git commands, don't worry if you constantly are looking things up: that's what *everyone* does. If you want to learn more, check out the following resources:
+If you want to learn more, check out the following resources:
 
 * [Getting started with git/github](https://github.com/EricThomson/git_learn)
 * [GitHub on Contributing to a Project](https://git-scm.com/book/en/v2/GitHub-Contributing-to-a-Project)
 * [GitHub skillbuilding](https://skills.github.com/)
 * [Scipy git resources](https://docs.scipy.org/doc/scipy/dev/gitwash/gitwash.html#using-git)
 
-Again, if you want to contribute and find any of the above bits confusing, please reach out!
-
-
-
-
-

Jenkinsfile

@@ -34,7 +34,7 @@ pipeline {
               export THEANO_FLAGS="base_compiledir=$TEMPDIR/theano_tmp"
               cd $TEMPDIR
               caimanmanager.py install
-              nosetests --traverse-namespace caiman
+              pytest --pyargs caiman
               caimanmanager.py demotest
             '''
           }
@@ -58,7 +58,7 @@ pipeline {
               export CAIMAN_DATA=$TEMPDIR/caiman_data
               cd $TEMPDIR
               caimanmanager.py install
-              nosetests --traverse-namespace caiman
+              pytest --pyargs caiman
             '''
           }
         }

README.md

@@ -1,45 +1,69 @@
 <a href="https://colab.research.google.com/drive/1vkp-uPV8tKavmX12bcN2L-jYH8_MgmHL?usp=sharing"><img src="https://img.shields.io/badge/-Colab%20Demo-blue" /></a>
 
 
-CaImAn
+Caiman
 ======
 <img src="https://github.com/flatironinstitute/CaImAn/blob/main/docs/LOGOS/Caiman_logo_2.png" width="400" align="right">
 
 A Python toolbox for large-scale **Ca**lcium **Im**aging **An**alysis.    
 
-CaImAn implements a set of essential methods required to analyze calcium and voltage imaging data. It provides fast and scalable algorithms for motion correction, source extraction, spike deconvolution, and registering neurons across multiple sessions. It is suitable for both two-photon and one-photon fluorescence microscopy data, and can be run in both offline and online modes. Documentation is [here](https://caiman.readthedocs.io/en/latest/). 
+Caiman implements a set of essential methods required to analyze calcium and voltage imaging data. It provides fast and scalable algorithms for motion correction, source extraction, spike deconvolution, and registering neurons across multiple sessions. It is suitable for both two-photon and one-photon fluorescence microscopy data, and can be run in both offline and online modes. Documentation is [here](https://caiman.readthedocs.io/en/latest/). 
 
-Caiman Central
---------------
-- [Caiman Central](https://github.com/flatironinstitute/caiman_central) is the hub for sharing information about CaImAn. Information on quarterly community meetings, workshops, other events, and any other communications between the developers and the user community can be found there.
+# Installation
+There are two primary ways to install Caiman.
 
-# Quick start
-Follow these three steps to get started quickly, from installation to working through a demo notebook. If you do not already have conda installed, [you can find it here](https://docs.conda.io/en/latest/miniconda.html). There is a video walkthrough of the following steps [here](https://youtu.be/b63zAmKihIY?si=m7WleTwdU0rJup_2).
+## Route A
+The easiest route is to install the miniforge distribution of Anaconda, and use that to install the rest using prebuilt packages. Most users should take this path.
 
-### Step 1: Install caiman
+## Route B
+The alternative route is to make sure you have a working compiler, create a python virtualenv, grab the caiman sources, and use pip to populate the virtualenv and build Caiman. This route is not as tested and is not presently documented; it is a standard pip-based install.
+
+# Quick start (Route A)
+Follow these three steps to get started quickly, from installation to working through a demo notebook. If you do not already have conda installed, [you can find it here](https://github.com/conda-forge/miniforge). The miniforge distribution of conda is preferred; it will require fewer steps and likely encounter fewer issues. If you are using a different distro of conda, you will likely need to add `-c conda-forge` to the commands you use to make your environment.
+
+Windows users will temporarily need to use an alternative install path.
+
+### Step 1: Install Caiman
 The following is all done in your anaconda prompt, starting in your base environment:
 
-    conda install -n base -c conda-forge mamba   # install mamba in base environment
-    mamba create -n caiman -c conda-forge caiman # install caiman
-    conda activate caiman  # activate virtual environment
+    mamba create -n caiman caiman # build a caiman environment
+    conda activate caiman  # activate the environment
+
+### Step 1: Install Caiman (alternative for Windows users)
+Windows users will need to follow an alternative set of steps because tensorflow does not have good packaging for Windows with conda (packaging changes are underway to solve this but are not available as of this writing).
+
+First, you will need to install Visual Studio 2019 or possibly a later version, with the C++ compiler and commandline utilities.
+Then you will clone this repo to your windows system, and enter the checkout directory.
+
+Next, you will build and activate a mostly-empty conda environment:
+
+    mamba create -n caiman python=3.11 pip vs2019_win-64
+    conda activate caiman
+
+Finally, you will use pip to install Caiman's prerequisites and Caiman itself:
+    pip install .
+
+This step may fail if the compiler is not correctly installed and is the most fragile part of this install route; reach out if you encounter issues.
+
+After this, assuming you succeed, leave the source directory. Later steps will not function correctly when run in the source/checkout directory.
 
 ### Step 2: Download code samples and data sets
-Create a working directory called `caiman_data` that includes code samples and related data. Run the following command from the same virtual environment that you created in Step 1:  
+Create a working directory called `caiman_data` that includes code samples and related data. Run the following command from the same conda environment that you created in Step 1:  
 
     caimanmanager install
 
 ### Step 3: Try out a demo notebook
 Go into the working directory you created in Step 2, and open a Jupyter notebook:
 
     cd <your home>/caiman_data/
-    jupyter notebook 
+    jupyter lab
 
 Jupyter will open. Navigate to demos/notebooks/ and click on `demo_pipeline.ipynb` to get started with a demo.
 
-> Note that what counts as `<your home>` in the first line depends on your OS/computer. Be sure to fill in your actual home directory. On Linux/Mac it is `~` while on Windows it will be something like `C:\Users\your_user_name\` 
+> `<your home>` in the first line is your home directory, its location depdnding on your OS/computer. On Linux/Mac it is `~` while on Windows it will be something like `C:\Users\your_user_name\` 
 
 ## For installation help
-Caiman should install easily on Linux, Mac, and Windows. If you run into problems, we have a dedicated [installation page](./docs/source/Installation.rst): the details there should help you troubleshoot. If you don't find what you need there, *please* [create an issue](https://github.com/flatironinstitute/CaImAn/issues) at GitHub, and we will help you get it sorted out. 
+Caiman should install easily on Linux, Mac, and Windows. If you run into problems, we have a dedicated [installation page](./docs/source/Installation.rst). If you don't find what you need there, [create an issue](https://github.com/flatironinstitute/Caiman/issues) on GitHub.
 
 # Demo notebooks
 Caiman provides demo notebooks to showcase each of our main features, from motion correction to online CNMF. We recommend starting with the CNMF notebook (`demo_pipeline.ipynb`), which contains more explanation and details than the other notebooks: it covers many concepts that will be used without explanation in the other notebooks. The CNMFE notebook (`demo_pipeline_cnmfE.ipynb`), is also more detailed. Once you've gotten things set up and worked through those "anchor" notebooks, the best way to get started is to work through the demo notebook that most closely matches your use case; you should be able to adapt it for your particular needs.
@@ -67,9 +91,9 @@ Caiman also provides commandline demos, similar to the notebooks, demonstrating
 
 # How to get help
 - [Online documentation](https://caiman.readthedocs.io/en/latest/) contains a lot of general information about Caiman, the parameters, how to interpret its outputs, and more.
-- [GitHub Discussions](https://github.com/flatironinstitute/CaImAn/discussions) is our preferred venue for users to ask for help.
+- [GitHub Discussions](https://github.com/flatironinstitute/Caiman/discussions) is our preferred venue for users to ask for help.
 - The [Gitter forum](https://app.gitter.im/#/room/#agiovann_Constrained_NMF:gitter.im) is our old forum: we sometimes will ask people to join us there when something can best be solved in real time (e.g., installation problems).
-- If you have found a bug, we recommend searching the [issues at github](https://github.com/flatironinstitute/CaImAn/issues) and opening a new issue if you can't find the solution there. 
+- If you have found a bug, we recommend searching the [issues at github](https://github.com/flatironinstitute/Caiman/issues) and opening a new issue if you can't find the solution there. 
 - If there is a feature you would like to see implemented, feel free to come chat at the above forums or open an issue at Github.
 
 # How to contribute
@@ -119,13 +143,12 @@ If possible, we'd also ask that you cite the papers where the original algorithm
 # Main developers
 * (emeritus) Eftychios A. Pnevmatikakis, **Flatiron Institute, Simons Foundation** 
 * (emeritus) Andrea Giovannucci, **University of North Carolina, Chapel Hill**
-* Johannes Friedrich, **Allen Institute, Seattle Washington**
-* Changjia Cai, **University of North Carolina, Chapel Hill**
+* (emeritus) Johannes Friedrich, **Allen Institute, Seattle Washington**
+* (emeritus) Changjia Cai, **University of North Carolina, Chapel Hill**
+* Kushal Kolar, **Flatiron Institute, Simons Foundation**
 * Pat Gunn, **Flatiron Institute, Simons Foundation**
-* Eric Thomson, **Flatiron Institute, Simons Foundation**
-
-A complete list of contributors can be found [here](https://github.com/flatironinstitute/Caiman/graphs/contributors). Currently Pat Gunn, Johannes Friedrich, and Eric Thomson are the most active contributors.
 
+A complete list of contributors can be found [here](https://github.com/flatironinstitute/Caiman/graphs/contributors).
 
 # Acknowledgements 
 Special thanks to the following people for letting us use their datasets in demo files:
@@ -136,6 +159,12 @@ Special thanks to the following people for letting us use their datasets in demo
 * Manolis Froudarakis, Jake Reimers, Andreas Tolias, Baylor College of Medicine
 * Clay Lacefield, Randy Bruno, Columbia University
 * Daniel Aharoni, Peyman Golshani, UCLA
+* Darcy Peterka, Columbia
+
+Also a special thanks to:
+* Eric Thompson, for various strong contributions to code and demos, both before and during his employment at the Flatiron Institute.
+* Cai Changjia, for Volpy
+* Ethan Blackwood, for several contributions in various areas
 
 # License
 This program is free software; you can redistribute it and/or

VERSION

@@ -1 +1 @@
-1.11.3
+1.11.4

caiman/base/movies.py

@@ -2025,6 +2025,17 @@ def get_file_size(file_name, var_name_hdf5:str='mov') -> tuple[tuple, Union[int,
                     logger.error(f'The file does not contain a variable named {var_name_hdf5}')
                     raise Exception('Variable not found. Use one of the above')
                 T, dims = siz[0], siz[1:]
+            elif extension in ('.npy', ):
+                with open(file_name, 'rb') as f:
+                    version = np.lib.format.read_magic(f)
+                    if version == (1, 0):
+                        shape, _, _ = np.lib.format.read_array_header_1_0(f)
+                    elif version == (2, 0):
+                        shape, _, _ = np.lib.format.read_array_header_2_0(f)
+                    else:
+                        raise ValueError(f"Unsupported .npy file version: {version}. Update caiman.base.movies.get_file_size() to handle it.")
+                T = shape[0]
+                dims = shape[1:]
             elif extension in ('.sbx'):
                 shape = caiman.utils.sbx_utils.sbx_shape(file_name[:-4])
                 T = shape[-1]