Merge branch 'topic/v1.1-prep' into release

* topic/v1.1-prep:
  Bumping the version
  Global cleanup and end-to-end processing of nifti input files

.gitignore

@@ -1,10 +1,6 @@
-*.vtk
 *.pdf
 *.pvsm
 *.pyc
 
-!thesis_complete.pdf
-
 build/
-data/
-aux/
+.vscode

doc/source/index.rst

@@ -17,18 +17,25 @@ Welcome to mpi.is-brain-connectivity-plugin's documentation!
    pages/installation
    pages/plugin_details
    pages/other_tools
+   pages/conversion_tools
 
 This package contains a set of python tools for assisting in the visualization of brain connectivity information.
 The package contains:
 
-* a Paraview plugin, that is a parametrizable plugin running in a Paraview visualization pipeline. This
+* a **Paraview filter**, that is a parametrizable *plugin* running in a Paraview visualization pipeline. This
   plugin shows the connectivity of the brain sites, in a raw or summarized fashion,
 * utility functions for converting files and performing off-line computation on the brain connectivity
 
 References
 ----------
-This package was developed by Lennart Bramlage and Raffi Enficiaud in the `Software Workshop` Central Scientific
-Facility of the Max Planck Institute for Intelligent Systems, in Tübingen, Germany.
+This package was developed by *Lennart Bramlage* and *Raffi Enficiaud* at the `Software Workshop` Central Scientific
+Facility of the Max Planck Institute for Intelligent Systems, in Tübingen, Germany. The details of the computations
+are given in the following report:
+
+* `Design of a visualization scheme for functional connectivity data of Human Brain <LennartBA_>`_, Lennart Bramlage, Bachelor Thesis,
+  Hochschule Osnabrück - University of Applied Sciences, 2017.
+
+.. _LennartBA : https://is.tuebingen.mpg.de/publications/bramlage-2017
 
 Installation
 ------------
@@ -40,7 +47,7 @@ The usage of the plugin is detailed in the page :doc:`pages/plugin_details`.
 
 Utility function
 ----------------
-The additional tools are described in the page :doc:`pages/other_tools`.
+The additional tools are described in the pages :doc:`pages/other_tools` and :doc:`pages/conversion_tools`.
 
 License
 -------

doc/source/pages/conversion_tools.rst

@@ -0,0 +1,74 @@
+File conversion utilities
+=========================
+
+.. contents::
+  :local:
+
+.. _nifti_edge_file_generating:
+
+Connectivity file from Nifti
+----------------------------
+The set of connection between two different sites of the brain is contained in a **edge file** file, that makes it
+convenient to read in the visualization filter in Paraview.
+
+Generating the edge file is done through the command ``generate_brain_connectivity_edge_file`` and requires:
+
+* the ``nifti`` file that can be read with ``nibabel``,
+* the ``csv`` file encoding the connections between sites in the brain,
+* the output file.
+
+.. warning::
+
+  This functionality requires the installation of the ``nibabel`` python package. Please make sure you followed
+  the :doc:`installation` procedures.
+
+
+End-points volume image
+-----------------------
+The 3D volume containing all the end-points of the connexels can be generated from the command line in the same
+manner as for the connectivity file. The generated file is in the VTK format.
+
+.. warning::
+
+  Requires ``nibabel`` (see above) and ``VTK`` (see installation instructions).
+
+Volume image
+------------
+The 3D volumetric image may be generated from the command line with the command ``generate_brain_connectivity_volume_file``.
+This call is a wrapper
+around the function :py:func:`.generate_volume_file`.
+
+.. _clusters_generating:
+
+Clustering
+----------
+A script for performing the clustering of all the edges is provided. This scripts generates a file that can
+be given to the visualization filter, which makes the exploration of the different edge clusters much faster.
+
+The command to generate the cluster file is ``generate_brain_connectivity_cluster_file``. The command
+requires:
+
+* the mesh file,
+* the edge file,
+* the number of desired clusters,
+* the output file
+
+The output file can be given to the visualization filter in Paraview to avoid expensive computations
+during the visualization.
+
+.. warning::
+
+  Requires ``nibabel`` (see above) and ``VTK`` (see installation instructions).
+
+.. tip::
+
+  ``scipy`` may be used to speed-up the computation of the k-means if installed.
+
+References
+----------
+
+.. automodule:: src.utils.generate_processing
+   :members:
+
+.. automodule:: src.utils.generate_plugin_xml
+   :members:

doc/source/pages/gallery.rst

@@ -4,3 +4,10 @@ Gallery
 A set of pictures generated in Paraview and the brain-connectivity plugin is collected here.
 Feel free to send additional pictures to the authors/maintainers of the plugin.
 
+
+* The image is shown as a 3D volume, a specific cluster is highlighted from a cluster file
+  computed offline.
+
+    .. image:: /_static/gallery/brain_clusters1000_selected100.png
+        :height: 300
+

doc/source/pages/installation.rst

@@ -56,6 +56,7 @@ and then navigate to the path where you cloned the package, and type
 .. code::
 
   cd <brain-connectivity-visualization-path>
+  pip install -r requirements.txt
   pip install .
 
 After the installation, you should see the package with ``pip list``:
@@ -165,10 +166,65 @@ python virtual environment, and this will be the preferred method on this platfo
    mentioned earlier, and it would be safer to make the OSX system Python the default in
    that case.
 
-The installation of Paraview is often located in ``/Application``` folder, and it is not
+The installation of Paraview is often located in ``/Application`` folder, and it is not
 possible to write into the Paraview application folder. Therefor the installation
 on OSX requires a virtual environment or a global installation.
 
+Paraview comes with its own version of Numpy, that might not be ABI compatible with newer
+versions of the library. Numpy is extensively used in the visualization plugin, which indirectly
+means that the API being used in the plugin should be compatible with the version of Numpy
+installed along Paraview.
+
+Some parts of the visualization plugin require VTK. It is unfortunately quite difficult to use
+VTK installation provided by Paraview. In particular it is not straightforward to use
+the libraries provided by Paraview with your own Python setup. One possibility would be
+
+1. create your virtual environment
+2. compile and install VTK with the python bindings
+3. load your python shell within your environment
+4. point to the freshly installed VTK
+
+For compiling VTK:
+
+.. code::
+
+    cd <vtk-source-folder>
+    mkdir build
+    cd build
+    cmake -DCMAKE_INSTALL_PREFIX=~/usr/local -DVTK_WRAP_PYTHON=ON ..
+    make -j 4
+    make install
+
+For using VTK if it is not installed in a system folder (recommanded), if for instance
+VTK is installed in `~/usr/local`
+
+.. code::
+
+    virtualenv venv_paraview
+    . venv_paraview/bin/activate
+    pip install -r requirements.txt
+    pip install .
+    pip install ipython
+
+    # adding VTK to the list of paths of the virtualenv
+    USER_SITE=$(python -c "import os; print os.environ['VIRTUAL_ENV']")
+    echo "/Users/<myself>/usr/local/lib/python2.7/site-packages/" >> $USER_SITE/lib/python2.7/site-packages/vtk.pth
+
+    export DYLD_LIBRARY_PATH=/Users/<myself>/usr/local/lib/:$DYLD_LIBRARY_PATH
+    python -c "import vtk; print 'success!'"
+    ipython
+    > import vtk
+
+For being able to run the scripts that require VTK, after installation do:
+
+.. code::
+
+    export DYLD_LIBRARY_PATH=/Users/<myself>/usr/local/lib/:$DYLD_LIBRARY_PATH
+    export PYTHONPATH=/Users/<myself>/usr/local/lib/python2.7/site-packages/:$PYTHONPATH
+
+    generate_brain_connectivity_volume_file id1090_timestep.nii nii-volume-2
+
+
 Linux
 ^^^^^
 The Linux installation is either:

doc/source/pages/other_tools.rst

@@ -4,27 +4,50 @@ Computation utilities
 .. contents::
   :local:
 
-Along the paraview plugin, several functions have been developed. Those utilities have different purposes:
+Along the Paraview plugin, several functions have been developed. Those utilities have different purposes:
 
 * make off-line computations in order to speed-up the visualization
 * support the writing of new visualization functions
 
+Visualization preparation
+-------------------------
+The visualization requires a little preparation step that aims at creating necessary or optional files.
+Those files are either required for the visualization, or assist the visualization with
+additional elements.
+
+The only necessary file is the **edge file** that indicates the plugin the end-points of the different
+connectivity edges in the 3d volume.
+
+Additionally, it is possible to create:
+
+* a volume file from a nifti volumetric data. An example of use of such a file in Paraview
+  is given in the image below
+
+  .. image:: /_static/volume_view.png
+      :width: 300
+
+* a grid file of all edges end-points location.
+
+  .. image:: /_static/end_points_location_view.png
+      :width: 300
+
 Off-line computations
 ---------------------
 
-kMeans and hierarchical kMeans
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-The kMeans algorithm currently being in use within the plugin is a naive implementation using a distance function
-that is specific to the needs of the visualization. Accelerated algorithms exist such as the [Elkan]_ and that involves
-a lower number of computations and distance estimations. The complexity of the kMeans is at best polynomial in the
-number of points.
+Clustering with kMeans
+^^^^^^^^^^^^^^^^^^^^^^
+The clustering of edges is performed with the kmeans algorithm.
+
+The version of this algorithm currently being in use within the plugin is implemented in a naive way in Python, using a
+distance function that is specific to the needs of the visualization. Accelerated algorithms exist such as the
+[Elkan]_ and that involves a lower number of computations and distance evaluations.
 
 The hierarchical version of the kMeans runs the kMeans at each level of the hierarchy, and has also a complexity
+similar to the kMeans multiplied by the number of hierarchy levels.
 
+Since the clustering may take some time to converge, a script accessible after the installation of the
+package in a virtual environment is provided.
 
-Alignment
-^^^^^^^^^
-The data to visualize should be in the correct format in order to use the plugin.
 
 Bibliographic references
 ------------------------

doc/source/pages/plugin_details.rst

@@ -1,41 +1,59 @@
-Paraview plugin
-===============
+Paraview visualization filter
+=============================
 
 .. contents::
   :local:
 
 
-We describe in this page how to use the plugin inside Paraview:
+This page describes how to use the brain visualization filter inside Paraview:
 
-# the plugin acts on a mesh of point representing the surface of the brain
-# several visualization algorithm and parameters are shown in the user interface.
+#. the plugin acts on a mesh of point representing the surface of the brain
+#. several visualization algorithm and parameters are shown in the user interface
+#. the plugin needs external files in order to render the connectivity edges
 
 Input data
 ----------
-In order to function, the visualization needs:
+For the plugin to compute the visualization, the following elements are required:
 
 * the mesh on which computation will be performed. This is usually a mesh file that Paraview
-  can understand,
-* the connectivity file, that contains the topology of connection network in the brain. This
-  connectivity file is a mapping between pair of mesh locations and a weight.
-
-The way the connectivity file is created is detailed in another section.
+  can understand/load without extra requirements,
+* the connectivity (or edges) file that contains the topology of connection network in the brain.
+  This connectivity file is a mapping between pair of mesh or volume location and a weight.
+  In its current form, it is a python ``numpy`` file that is generated with a command line tool form
+  the brain connectivity package, and that is described in the section :ref:`nifti_edge_file_generating`.
+* *Optionally* it is also possible to provide a preprocessed clusters file, which would avoid the live
+  computation of the clustering for the selected edges. See section :ref:`clusters_generating` for
+  more details.
 
 Parameters
 ----------
 
 The parameters exposed to the user by the plugin are shown in the picture below:
 
-.. image:: /_static/plugin_panel.png
+.. image:: /_static/brain_clusters1000_selected100_plugin.png
     :height: 300
 
 The meaning of the parameters are:
 
-* ``Edge res``: indicates the edges resolution, ie. the number of intermediate steps a trajectory
-  connecting two sites is divided into. The higher, the smoother the trajectory but also the
+* ``resolution``: indicates the edges resolution, ie. the number of intermediate steps a path
+  connecting two sites is divided into. The higher this number, the smoother the trajectory but also the
   heavier the on-screen drawing,
-* ``Edge selection``: a range of edges on which the processing will be performed. Any edge falling
-  outside of this range is ignored in the visualization and the computation,
-* ``Edge path``: the **absolute** path of the file containing the definition of the topology,
-* ``show edges``: if checked,
-
+* ``edge selection``: a range of edge indices on which the processing will be performed. Any edge falling
+  outside of this range is ignored in the visualization and the computation. This entry is ignored when
+  a cluster file is provided.
+* ``edge file``: the **absolute** path of the file containing the definition of the topology,
+* ``show edges``: if checked, the edges will be shown
+* ``clusters file``: the **absolute** path to the file containing the results of the clustering.
+  This file contains a preprocessed version
+  of the clustering. If not present, a clustering of the selected edges will be computed during the visualization,
+  which may take some time. For offline computation of the clustering, see the corresponding section.
+* ``nb_clusters``: indicates the number of desired clusters. This applies on the range of edges that has
+  been selected or the full range if no selection has been made, and is ignored if the ``clustering file`` is
+  given.
+* ``cluster index``: if given, a specific cluster to show on screen. All other clusters are discarded. The
+  edges that are in the specified cluster are also shown.
+* ``hierarchy index``: the level of the hierarchy to which the cluster index applies.
+
+.. note::
+
+    The hierarchical part is not functional.