Skip to content

Commit

Permalink
Merge pull request #623 from oist/develop-main
Browse files Browse the repository at this point in the history
Merge release ver1.1.0
  • Loading branch information
ReiHashimoto authored Dec 21, 2023
2 parents 825f3c0 + f5544e0 commit b9901fc
Show file tree
Hide file tree
Showing 1,280 changed files with 20,799 additions and 1,755,079 deletions.
5 changes: 5 additions & 0 deletions .codespellignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
createR
afterAll
xdescribe
optinist
Optinist
10 changes: 10 additions & 0 deletions .github/workflows/codespell.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
name: Codespell
on:
pull_request:

jobs:
codespell:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: codespell-project/actions-codespell@v2
2 changes: 2 additions & 0 deletions .github/workflows/linters.yml
Original file line number Diff line number Diff line change
Expand Up @@ -37,3 +37,5 @@ jobs:
steps:
- uses: actions/checkout@v4
- uses: psf/black@stable
with:
version: "~= 23.11"
12 changes: 7 additions & 5 deletions .github/workflows/test.yml → .github/workflows/tests.yml
Original file line number Diff line number Diff line change
@@ -1,10 +1,12 @@
name: CI
name: Tests

on:
push:
branches: [ main ]
pull_request:
branches: [ main ]
branches:
- main
- develop-main
paths-ignore:
- 'docs/**'

workflow_dispatch:

Expand All @@ -17,7 +19,7 @@ jobs:
steps:
- uses: actions/checkout@v4

- name: Run a one-line script
- name: Run test_run command
shell: bash
run: |
make test_run
8 changes: 6 additions & 2 deletions .pre-commit-config.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -6,8 +6,6 @@ repos:
- repo: https://github.com/pre-commit/pre-commit-hooks
rev: v4.4.0
hooks:
- id: check-added-large-files
exclude: frontend/build
- id: check-toml
- id: check-yaml
args:
Expand All @@ -26,3 +24,9 @@ repos:
rev: 6.1.0
hooks:
- id: flake8
- repo: https://github.com/codespell-project/codespell
rev: v2.2.4
hooks:
- id: codespell
additional_dependencies:
- tomli
2 changes: 1 addition & 1 deletion .readthedocs.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -11,4 +11,4 @@ sphinx:
configuration: docs/conf.py
python:
install:
- requirements: docs/requirements.txt
- requirements: docs/requirements.txt
4 changes: 3 additions & 1 deletion .vscode/extensions.json
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,8 @@
"recommendations": [
"ms-python.black-formatter",
"ms-python.flake8",
"ms-python.isort"
"ms-python.isort",
"dbaeumer.vscode-eslint",
"esbenp.prettier-vscode"
]
}
1 change: 1 addition & 0 deletions .vscode/settings.example.json
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@
"source.organizeImports": true
}
},
"eslint.workingDirectories": ["frontend"],
"flake8.args": ["--config=.flake8"],
// NOTE: Uncomment following line and fix "optinist_dev" to your conda env name
// "flake8.path": ["conda", "run", "-n", "optinist_dev", "python", "-m", "flake8"],
Expand Down
21 changes: 9 additions & 12 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,7 @@ build_frontend:
.PHONY: docs
docs:
rm -rf docs/_build/
# pip install -e '.[doc]'
poetry install --with doc --no-root
# sphinx-apidoc -f -o ./docs/_build/modules ./studio
sphinx-autobuild -b html docs docs/_build --port 8001

Expand All @@ -42,24 +42,21 @@ dockerhub:

.PHONY: local_build
local_build:
cd frontend
yarn install && yarn build
cd ../
pip install .
cd frontend && yarn install --ignore-scripts && yarn build
poetry build

.PHONY: upload_testpypi
upload_testpypi:
python -m build
twine upload --repository testpypi dist/*
poetry publish -r testpypi

.PHONY: test_pypi
test_pypi:
python3 -m pip install --index-url https://test.pypi.org/simple/ studio
.PHONY: install_testpypi
install_testpypi:
pip install --index-url https://test.pypi.org/simple/ --extra-index-url https://pypi.org/simple/ optinist
pip show optinist

.PHONY: push_pypi
push_pypi:
twine upload --repository pypi dist/*

poetry publish

.PHONY: format
format:
Expand Down
10 changes: 5 additions & 5 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -41,7 +41,7 @@ OptiNiSt also supports reproducibility of scientific research, standardization o

### Postprocessing
- [x] Basic Neural Analysis(Event Trigger Average...)
- [x] Dimenstion Reduction(PCA...)
- [x] Dimension Reduction(PCA...)
- [x] Neural Decoding(LDA...)
- [x] Neural Population Analysis(Correlation...)

Expand Down Expand Up @@ -85,7 +85,7 @@ https://optinist.readthedocs.io/en/latest/

## Using GUI
### Workflow
- OptiNiSt allows you to make your analysis pipelines by graph style using nodes and edges on GUI. Parameters for each analysis are easily changeable.
- OptiNiSt allows you to make your analysis pipelines by graph style using nodes and edges on GUI. Parameters for each analysis are easily changeable.
<p align="center">
<img width="400px" src="docs/_static/workflow/whole.png" alt="workflow" />
</p>
Expand All @@ -99,7 +99,7 @@ https://optinist.readthedocs.io/en/latest/
</p>

### Record
- OptiNiSt supports you in recording and reproducing workflow pipelines in an organized manner.
- OptiNiSt supports you in recording and reproducing workflow pipelines in an organized manner.
<p align="center">
<img width="400px" src="docs/_static/record/whole.png" alt="record" />
</p>
Expand All @@ -118,8 +118,8 @@ Kenji Doya, Yukako Yamane [OIST Neural Computation Unit](https://groups.oist.jp/


## References
[[Suite2p]](https://github.com/MouseLand/suite2p) Marius Pachitariu, Carsen Stringer, Mario Dipoppa, Sylvia Schröder, L. Federico Rossi, Henry Dalgleish, Matteo Carandini, Kenneth D. Harris. "Suite2p: beyond 10,000 neurons with standard two-photon microscopy". 2017
[[CaImAn]](https://github.com/flatironinstitute/CaImAn) Andrea Giovannucci Is a corresponding author, Johannes Friedrich, Pat Gunn, Jérémie Kalfon, Brandon L Brown, Sue Ann Koay, Jiannis Taxidis, Farzaneh Najafi, Jeffrey L Gauthier, Pengcheng Zhou, Baljit S Khakh, David W Tank, Dmitri B Chklovskii, Eftychios A Pnevmatikakis. "CaImAn: An open source tool for scalable Calcium Imaging data Analysis". 2019
[[Suite2p]](https://github.com/MouseLand/suite2p) Marius Pachitariu, Carsen Stringer, Mario Dipoppa, Sylvia Schröder, L. Federico Rossi, Henry Dalgleish, Matteo Carandini, Kenneth D. Harris. "Suite2p: beyond 10,000 neurons with standard two-photon microscopy". 2017
[[CaImAn]](https://github.com/flatironinstitute/CaImAn) Andrea Giovannucci Is a corresponding author, Johannes Friedrich, Pat Gunn, Jérémie Kalfon, Brandon L Brown, Sue Ann Koay, Jiannis Taxidis, Farzaneh Najafi, Jeffrey L Gauthier, Pengcheng Zhou, Baljit S Khakh, David W Tank, Dmitri B Chklovskii, Eftychios A Pnevmatikakis. "CaImAn: An open source tool for scalable Calcium Imaging data Analysis". 2019
[[LCCD]](https://github.com/magnetizedCell/lccd-python) Tsubasa Ito, Keisuke Ota, Kanako Ueno, Yasuhiro Oisi, Chie Matsubara, Kenta Kobayashi, Masamichi Ohkura, Junichi Nakai, Masanori Murayama, Toru Aonishi, "Low computational-cost cell detection method for calcium imaging data", 2022
[[PyNWB]](https://github.com/NeurodataWithoutBorders/pynwb) Oliver Rübel, Andrew Tritt, Ryan Ly, Benjamin K. Dichter, Satrajit Ghosh, Lawrence Niu, Ivan Soltesz, Karel Svoboda, Loren Frank, Kristofer E. Bouchard, "The Neurodata Without Borders ecosystem for neurophysiological data science", bioRxiv 2021.03.13.435173, March 15, 2021

Expand Down
10 changes: 6 additions & 4 deletions docker-compose.dev.yml
Original file line number Diff line number Diff line change
Expand Up @@ -8,20 +8,22 @@ services:
working_dir: /app
volumes:
- .:/app
- ../optinist-docker-volumes/studio_data/:/app/studio_data
ports:
- "8000:8000"
command: python main.py --reload --host 0.0.0.0 --port 8000
- "127.0.0.1:8000:8000"
command: poetry run python main.py --reload --host 0.0.0.0 --port 8000
environment:
PYTHONPATH: /app/
TZ: Asia/Tokyo
OPTINIST_DIR: /app/studio_data

studio-dev-fe:
image: node:16.20.1-alpine
image: node:20.8.0-alpine3.18
working_dir: /app/frontend
volumes:
- ./frontend/:/app/frontend/
ports:
- "3000:3000"
- "127.0.0.1:3000:3000"
command: ash -c 'yarn install && yarn start'
environment:
TZ: Asia/Tokyo
6 changes: 2 additions & 4 deletions docker-compose.test.yml
Original file line number Diff line number Diff line change
Expand Up @@ -10,11 +10,9 @@ services:
environment:
PYTHONPATH: .
TZ: Asia/Tokyo
command: bash -c "poetry install --no-root --with test && poetry run python3 -m pytest -s ."
volumes:
- ./studio:/app/studio
- ./conda/yaml:/app/conda/yaml
- ./studio/test_data:/tmp/studio
- ./frontend/build:/app/frontend/build
- .:/app

test_studio_frontend:
image: test_studio_frontend
Expand Down
4 changes: 2 additions & 2 deletions docker-compose.yml
Original file line number Diff line number Diff line change
Expand Up @@ -11,8 +11,8 @@ services:
- ../optinist-docker-volumes/logs/:/app/logs
- ../optinist-docker-volumes/studio_data/:/app/studio_data
ports:
- "8000:8000"
command: python main.py --host 0.0.0.0 --port 8000
- "127.0.0.1:8000:8000"
command: poetry run python main.py --host 0.0.0.0 --port 8000
environment:
PYTHONPATH: /app/
TZ: Asia/Tokyo
Expand Down
Binary file removed docs/_static/record/components/table.png
Binary file not shown.
File renamed without changes
File renamed without changes
Binary file added docs/_static/record/error_message.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
File renamed without changes
Binary file added docs/_static/record/table.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file modified docs/_static/record/whole.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs/_static/tutorials/ETA.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs/_static/tutorials/PCA.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs/_static/tutorials/add_modules.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file removed docs/_static/tutorials/caiman_sample_pipeline.png
Binary file not shown.
Binary file added docs/_static/tutorials/check_csv.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs/_static/tutorials/delete_ROI.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file removed docs/_static/tutorials/download_workflow_config.png
Binary file not shown.
Binary file removed docs/_static/tutorials/fig10.1_buttons.png
Binary file not shown.
Binary file removed docs/_static/tutorials/fig10_parameters.png
Binary file not shown.
Binary file removed docs/_static/tutorials/fig11_runall.png
Binary file not shown.
Binary file removed docs/_static/tutorials/fig12.00_hdfnode.png
Binary file not shown.
Binary file removed docs/_static/tutorials/fig12.01_hdfnode2.png
Binary file not shown.
Binary file removed docs/_static/tutorials/fig12.02_hdfs2p.png
Binary file not shown.
Binary file removed docs/_static/tutorials/fig12.03_hdfcaiman.png
Binary file not shown.
Binary file removed docs/_static/tutorials/fig12.04_behaviornode.png
Binary file not shown.
Binary file removed docs/_static/tutorials/fig12.05_behaviornode2.png
Binary file not shown.
Binary file removed docs/_static/tutorials/fig12.06_etaworkflow.png
Binary file not shown.
Binary file removed docs/_static/tutorials/fig12.07_etamean.png
Binary file not shown.
Binary file removed docs/_static/tutorials/fig12.08_resulthdf.png
Diff not rendered.
Binary file removed docs/_static/tutorials/fig13_nwbsnakemake.png
Diff not rendered.
Binary file removed docs/_static/tutorials/fig1_console.png
Diff not rendered.
Binary file removed docs/_static/tutorials/fig20_initialize.png
Diff not rendered.
Binary file removed docs/_static/tutorials/fig21_loadmovie.png
Diff not rendered.
Binary file removed docs/_static/tutorials/fig22_movie.png
Diff not rendered.
Binary file removed docs/_static/tutorials/fig2_open.png
Diff not rendered.
Binary file removed docs/_static/tutorials/fig30_saving.png
Diff not rendered.
Binary file removed docs/_static/tutorials/fig40_recordall.png
Diff not rendered.
Binary file removed docs/_static/tutorials/fig4_algorithms.png
Diff not rendered.
Binary file removed docs/_static/tutorials/fig5_imagenode2.png
Diff not rendered.
Binary file removed docs/_static/tutorials/fig6_suite2p.png
Diff not rendered.
Binary file removed docs/_static/tutorials/fig7_suite2p2.png
Diff not rendered.
Binary file removed docs/_static/tutorials/fig8_suite2pcaiman.png
Diff not rendered.
Binary file removed docs/_static/tutorials/fig9_workflowbranch.png
Diff not rendered.
Binary file removed docs/_static/tutorials/import_workflow.png
Diff not rendered.
Binary file removed docs/_static/tutorials/lccd_sample_pipeline.png
Diff not rendered.
Binary file added docs/_static/tutorials/load_sample_data.png
Binary file added docs/_static/tutorials/play_vid.png
Binary file removed docs/_static/tutorials/sample_pipelines.png
Diff not rendered.
Binary file added docs/_static/tutorials/show_fluorescence.png
Binary file removed docs/_static/tutorials/suite2p_sample_pipeline.png
Diff not rendered.
Binary file added docs/_static/visualize/add_column.png
Binary file removed docs/_static/visualize/components/add_column.png
Diff not rendered.
Diff not rendered.
Diff not rendered.
Diff not rendered.
Binary file added docs/_static/visualize/customize_param.png
File renamed without changes
File renamed without changes
File renamed without changes
Binary file modified docs/_static/visualize/imaging_rate.png
Binary file added docs/_static/visualize/movie_index.png
File renamed without changes
File renamed without changes
File renamed without changes
Binary file added docs/_static/visualize/save_fig.png
File renamed without changes
Binary file added docs/_static/visualize/select_output_item.png
File renamed without changes
Binary file added docs/_static/visualize/set_display_box.png
File renamed without changes
Binary file modified docs/_static/visualize/whole.png
Binary file added docs/_static/workflow/add_algorithm.png
Binary file added docs/_static/workflow/algorithm_node_buttons.png
Binary file added docs/_static/workflow/algorithm_output.png
Binary file added docs/_static/workflow/algorithm_param_form.png
Binary file removed docs/_static/workflow/components/add_algorithm.png
Diff not rendered.
Binary file removed docs/_static/workflow/components/connect_edge.png
Diff not rendered.
Binary file removed docs/_static/workflow/components/imageList.png
Diff not rendered.
Binary file removed docs/_static/workflow/components/imageNode.png
Diff not rendered.
Binary file added docs/_static/workflow/connect_edge.png
Binary file added docs/_static/workflow/create_new_workflow.png
File renamed without changes
File renamed without changes
Binary file added docs/_static/workflow/imageList.png
File renamed without changes
Binary file added docs/_static/workflow/import_workflow.png
Binary file added docs/_static/workflow/nwb_settings.png
Binary file added docs/_static/workflow/run_after.png
Binary file added docs/_static/workflow/run_before.png
Binary file added docs/_static/workflow/run_buttons.png
File renamed without changes
File renamed without changes
File renamed without changes
File renamed without changes
File renamed without changes
File renamed without changes
File renamed without changes
Binary file added docs/_static/workflow/snakemake_settings.png
Binary file modified docs/_static/workflow/whole.png
6 changes: 2 additions & 4 deletions docs/conf.py
Original file line number Diff line number Diff line change
Expand Up @@ -7,8 +7,6 @@
import os
from datetime import datetime

from sphinx_pyproject import SphinxConfig

# -- Path setup --------------------------------------------------------------

# If extensions (or modules to document with autodoc) are in another directory,
Expand All @@ -18,11 +16,10 @@


# -- Project information -----------------------------------------------------
config = SphinxConfig("../pyproject.toml", globalns=globals())
project = "OptiNiSt"
copyright = f"{datetime.today().year}, OIST"
author = ""
release = ".".join(config.version.split(".")[:2])
release = "1.1.0"

# -- readthedocs -------------------------------------------------------------
on_rtd = os.environ.get("READTHEDOCS", None) == "True"
Expand All @@ -46,6 +43,7 @@
"sphinx.ext.autosummary",
"sphinx.ext.extlinks",
"sphinx.ext.autodoc.typehints",
"sphinx_copybutton",
]

# Tell myst-parser to assign header anchors for h1-h3.
Expand Down
61 changes: 61 additions & 0 deletions docs/for_developers/architecture.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,61 @@
## Architecture
### Frontend
- language: TypeScript
- environment: Node.js (v20)
- framework: React (v18)
- middleware: Redux, Redux Toolkit, React Router

#### Directory structure
Following is the summary of directory structure.

- frontend
- src
- api: handle API requests.
- components: React components.
- const: constants.
- pages: Components set as `element` in `Route` component of react-router.
- store: Redux store.
- slice
- Some
- SomeActions.ts: create actions like async thunk from api response.
- SomeSelectors.ts: select state from store.
- SomeSlice.ts: define reducers.
- SomeType.ts: define data type.
- SomeUtils.ts: define utility functions if needed.
- style: extra css files.
- utils: common utility functions.

- api, components, store/slice directories are separated by its domain.

### Backend
- language: Python (v3.8)
- framework: FastAPI
- handle workflow by snakemake.

#### Directory structure
Following is the summary of directory structure.
- studio
- alembic: files for database migration. database is used in multi-user mode.
- app
- `Snakefile`: File read by snakemake. It defines how to run workflow by language like Python.
- common: common modules(not for specific domain)
- core: define core functions.
- dataclass: define dataclasses that are passed between nodes or used as visualize outputs.
- db: database general configuration.
- models: database models.
- routers: define fastapi routers.
- schemas: define pydantic schemas for api requests and responses.
- wrappers: functions for algorithm nodes.
- optinist: modules specific to calcium imaging data processing domain. It has same structure as common.
- config: configuration files.
- `standalone-logging.yaml`: logging config for standalone mode. In standalone mode, logs are sent to streams.
- `logging.yaml`: logging config for multi-user mode. In multi-user mode, logs are sent to file (in `logs` dir). log file rotation settings are included.
- you can manage log configurations by editing these files.
- auth: put Firebase authentication secret files.
- docker: backend Dockerfiles for each environment.
- test_data: test data for unit tests.
- tests: unit tests.
- `__main__.py`: entry point of backend in production.
- `__main_unit__.py`: core main unit. called by `studio/__main__.py` and `main.py`.
- `main.py`: entry point of backend in development.
- `run_cluster.py`: entry point for CLI run.
16 changes: 16 additions & 0 deletions docs/for_developers/index.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
.. _for_developers:

***************
FOR DEVELOPERS
***************

OptiNiSt welcomes your contributions.
See the following guidelines before submitting Pull Requests.

.. toctree::
:maxdepth: 2

style
pull_request
architecture
test
18 changes: 18 additions & 0 deletions docs/for_developers/pull_request.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
## Pull Request
- GitHub Pull Request page
- [https://github.com/oist/optinist/pulls](https://github.com/oist/optinist/pulls)

### Pre Commit
- run following command before your first commit
```
pre-commit install
```
- Once installed, it automatically checks your coding style on every commits.

### Branch rules
- You can submit Pull Request by pushing new branch.
- Make sure the base branch is `develop-main`, and PR is to `develop-main`.
- You can't push to the `develop-main` branch directly, the branch is protected.
- Make sure new branch name is in following format (`xxx` is the name of the feature or bug you are working on.).
- `feature/xxx`
- `fix/xxx`
23 changes: 9 additions & 14 deletions docs/utils/contributing.md → docs/for_developers/style.md
Original file line number Diff line number Diff line change
@@ -1,16 +1,4 @@
(Contributing)=
# Contributing to OptiNiSt
OptiNiSt welcomes your contributions.
See the following guidelines before submitting Pull Requests.

## Coding Style
### Pre Commit
- run following command before your first commit
```
pre-commit install
```
- Once installed, it automatically checkes your code style on every commits.

## Style Guides
### Python
- Format all files using [black](https://black.readthedocs.io/en/stable/#).
- Check the code for problems using [flake8](https://pypi.org/project/flake8/).
Expand All @@ -19,7 +7,14 @@ See the following guidelines before submitting Pull Requests.
- Sort your import using [isort](https://github.com/PyCQA/isort).
- These guides are checked by pre-commits, github workflow on submitting Pull Requests.

#### VSCode Settings
### TypeScript
- Format all files using [prettier](https://prettier.io/).
- Rules are defined in `frontend/.prettierrc.json`.
- Check the code for problems using [eslint](https://eslint.org/).
- Rules are defined in `frontend/.eslintrc.json`.

### Editor support
#### VSCode
- If you are using VSCode, you can use features (like format/lint on save) by installing and activating the extensions in `.vscode/extensions.json`.
- For these extensions, we provide example settings in `.vscode/settings.example.json`
- To use this,
Expand Down
Loading

0 comments on commit b9901fc

Please sign in to comment.