(feat): benchmarking

.gitignore

@@ -3,7 +3,7 @@
 __pycache__/
 /*cache/
 .ipynb_checkpoints/
-/data/
+data/
 
 # Distribution / packaging
 /dist/
@@ -16,3 +16,6 @@ __pycache__/
 
 # Venvs
 *venv/
+
+# asv
+.asv/

benchmarks/README.md

@@ -0,0 +1,81 @@
+# AnnData Benchmarks
+
+This repo contains some work in progress benchmarks for [rapids_singlecell](https://github.com/scverse/rapids_singlecell) using [asv](https://asv.readthedocs.io).
+
+## Setup
+
+I definitley recommend reading through the asv docs. Currently, this assumes the benchmark suite can reach the `rapids_singlecell` repo via the path `../src/rapids_singlecell`. Otherwise, all you'll need to do is create a [machine file](https://asv.readthedocs.io/en/stable/commands.html#asv-machine) for your system and make sure `anndata`s dependencies are installable via `conda`.
+
+```shell
+pip install chardet
+conda install mamba
+```
+
+### Data
+
+Data will need to be retrieved for these benchmarks. This can be downloaded using the script fetch_datasets.py.
+
+Note that the `h5ad` format has changed since it's inception. While the `rapids_singlecell` package maintains backwards compatibility, older versions of `anndata` will not be able to read files written by more recent versions. To get around this for the benchmarks, datasets have to be able to be read by all versions which can require a setup function that creates the anndata object.
+
+## Usage
+
+### Runnings the benchmarks:
+
+To run benchmarks for a particular commit: `asv run {commit} --steps 1 -b`
+
+To run benchmarks for a range of commits: `asv run {commit1}..{commit2}`
+
+You can filter out the benchmarks which are run with the `-b {patttern}` flag.
+
+### Accessing the benchmarks
+
+You can see what benchmarks you've already run using `asv show`. If you don't specify a commit, it will search for the available commits. If you specify a commit it'll show you those results. For example:
+
+```bash
+$ asv show -b "views"
+Commits with results:
+
+Machine    : mimir.mobility.unimelb.net.au
+Environment: conda-py3.7-h5py-memory_profiler-natsort-numpy-pandas-scipy
+
+    61eb5bb7
+    e9ccfc33
+    22f12994
+    0ebe187e
+```
+
+```bash
+$ asv show -b "views" 0ebe187e
+Commit: 0ebe187e <views-of-views>
+
+views.SubsetMemorySuite.track_repeated_subset_memratio [mimir.mobility.unimelb.net.au/conda-py3.7-h5py-memory_profiler-natsort-numpy-pandas-scipy]
+  ok
+  ======= ======= ========== ============ ===================== ====================== ======================
+  --                                                                   index_kind
+  --------------------------------------- -------------------------------------------------------------------
+   n_obs   n_var   attr_set   subset_dim         intarray             boolarray                slice
+  ======= ======= ========== ============ ===================== ====================== ======================
+    100     100     X-csr        obs               2.84           1.7916666666666667            0.5
+    100     100     X-csr        var        2.5357142857142856    1.8695652173913044     0.5652173913043478
+    100     100    X-dense       obs        3.1739130434782608    1.6538461538461537            0.6
+...
+```
+
+You can compare two commits with `asv compare`
+
+```bash
+$ asv compare e9ccfc 0ebe187e
+All benchmarks:
+
+       before           after         ratio
+     [e9ccfc33]       [0ebe187e]
+     <master>         <views-of-views>
+-            2.16  1.7916666666666667     0.83  views.SubsetMemorySuite.track_repeated_subset_memratio(100, 100, 'X-csr', 'obs', 'boolarray')
++ 2.533333333333333             2.84     1.12  views.SubsetMemorySuite.track_repeated_subset_memratio(100, 100, 'X-csr', 'obs', 'intarray')
+- 1.1923076923076923              0.5     0.42  views.SubsetMemorySuite.track_repeated_subset_memratio(100, 100, 'X-csr', 'obs', 'slice')
+  1.9615384615384615  1.8695652173913044     0.95  views.SubsetMemorySuite.track_repeated_subset_memratio(100, 100, 'X-csr', 'var', 'boolarray')
+```
+
+### View in the browser:
+
+You can view the benchmarks in the browser with `asv publish` followed by `asv preview`. If you want to include benchmarks of a local branch, I think you'll have to add that branch to the `"branches"` list in `asv.conf.json`.

benchmarks/asv.conf.json

@@ -0,0 +1,175 @@
+{
+    // The version of the config file format.  Do not change, unless
+    // you know what you are doing.
+    "version": 1,
+    // The name of the project being benchmarked
+    "project": "rapids_singlecell",
+    // The project's homepage
+    "project_url": "https://rapids-singlecell.readthedocs.io/",
+    // The URL or local path of the source code repository for the
+    // project being benchmarked
+    "repo": "../",
+    // The Python project's subdirectory in your repo.  If missing or
+    // the empty string, the project is assumed to be located at the root
+    // of the repository.
+    // "repo_subdir": "",
+    // Customizable commands for building, installing, and
+    // uninstalling the project. See asv.conf.json documentation.
+    //
+    // "install_command": ["python -mpip install {wheel_file}"],
+    // "uninstall_command": ["return-code=any python -mpip uninstall -y {project}"],
+    "build_command": [
+        "python -m pip install build",
+        "python -m build --wheel -o {build_cache_dir} {build_dir}",
+    ],
+    // List of branches to benchmark. If not provided, defaults to "master"
+    // (for git) or "default" (for mercurial).
+    "branches": [
+        "main"
+    ], // for git
+    // "branches": ["default"],    // for mercurial
+    // The DVCS being used.  If not set, it will be automatically
+    // determined from "repo" by looking at the protocol in the URL
+    // (if remote), or by looking for special directories, such as
+    // ".git" (if local).
+    "dvcs": "git",
+    // The tool to use to create environments.  May be "conda",
+    // "virtualenv" or other value depending on the plugins in use.
+    // If missing or the empty string, the tool will be automatically
+    // determined by looking for tools on the PATH environment
+    // variable.
+    "environment_type": "conda",
+    // timeout in seconds for installing any dependencies in environment
+    // defaults to 10 min
+    //"install_timeout": 600,
+    // the base URL to show a commit for the project.
+    "show_commit_url": "https://github.com/scverse/rapids_singlecell/commit/",
+    // The Pythons you'd like to test against.  If not provided, defaults
+    // to the current version of Python used to run `asv`.
+    // "pythons": ["2.7", "3.6"],
+    // The list of conda channel names to be searched for benchmark
+    // dependency packages in the specified order
+    // "conda_channels": [
+    //     "conda-forge",
+    //     "defaults",
+    //     "rapidsai",
+    //     "nvidia"
+    // ],
+    // The matrix of dependencies to test.  Each key is the name of a
+    // package (in PyPI) and the values are version numbers.  An empty
+    // list or empty string indicates to just test against the default
+    // (latest) version. null indicates that the package is to not be
+    // installed. If the package to be tested is only available from
+    // PyPi, and the 'environment_type' is conda, then you can preface
+    // the package name by 'pip+', and the package will be installed via
+    // pip (with all the conda available packages installed first,
+    // followed by the pip installed packages).
+    //
+    "conda_environment_file": "environment.yml",
+    // "matrix": {
+    //     "cuda-version": [
+    //         "12.2"
+    //     ],
+    //     "cudf": [
+    //         "24.4"
+    //     ],
+    //     "cuml": [
+    //         "24.4"
+    //     ],
+    //     "cugraph": [
+    //         "24.4"
+    //     ],
+    //     "pandas": [
+    //         ""
+    //     ],
+    //     "memory_profiler": [
+    //         ""
+    //     ],
+    //     "anndata": [
+    //         ""
+    //     ],
+    //     "scanpy": [
+    //         ""
+    //     ],
+    //     "numpy": [
+    //         ""
+    //     ],
+    //     "scipy": [
+    //         ""
+    //     ]
+    //     // "scanpy": [""],
+    //     // "psutil": [""]
+    // },
+    // Combinations of libraries/python versions can be excluded/included
+    // from the set to test. Each entry is a dictionary containing additional
+    // key-value pairs to include/exclude.
+    //
+    // An exclude entry excludes entries where all values match. The
+    // values are regexps that should match the whole string.
+    //
+    // An include entry adds an environment. Only the packages listed
+    // are installed. The 'python' key is required. The exclude rules
+    // do not apply to includes.
+    //
+    // In addition to package names, the following keys are available:
+    //
+    // - python
+    //     Python version, as in the *pythons* variable above.
+    // - environment_type
+    //     Environment type, as above.
+    // - sys_platform
+    //     Platform, as in sys.platform. Possible values for the common
+    //     cases: 'linux2', 'win32', 'cygwin', 'darwin'.
+    //
+    // "exclude": [
+    //     {"python": "3.2", "sys_platform": "win32"}, // skip py3.2 on windows
+    //     {"environment_type": "conda", "six": null}, // don't run without six on conda
+    // ],
+    //
+    // "include": [
+    //     // additional env for python2.7
+    //     {"python": "2.7", "numpy": "1.8"},
+    //     // additional env if run on windows+conda
+    //     {"platform": "win32", "environment_type": "mamba", "python": "2.7", "libpython": ""},
+    // ],
+    // The directory (relative to the current directory) that benchmarks are
+    // stored in.  If not provided, defaults to "benchmarks"
+    // "benchmark_dir": "benchmarks",
+    // The directory (relative to the current directory) to cache the Python
+    // environments in.  If not provided, defaults to "env"
+    "env_dir": ".asv/env",
+    // The directory (relative to the current directory) that raw benchmark
+    // results are stored in.  If not provided, defaults to "results".
+    "results_dir": ".asv/results",
+    // The directory (relative to the current directory) that the html tree
+    // should be written to.  If not provided, defaults to "html".
+    "html_dir": ".asv/html",
+    // The number of characters to retain in the commit hashes.
+    // "hash_length": 8,
+    // `asv` will cache results of the recent builds in each
+    // environment, making them faster to install next time.  This is
+    // the number of builds to keep, per environment.
+    // "build_cache_size": 2,
+    // The commits after which the regression search in `asv publish`
+    // should start looking for regressions. Dictionary whose keys are
+    // regexps matching to benchmark names, and values corresponding to
+    // the commit (exclusive) after which to start looking for
+    // regressions.  The default is to start from the first commit
+    // with results. If the commit is `null`, regression detection is
+    // skipped for the matching benchmark.
+    //
+    // "regressions_first_commits": {
+    //    "some_benchmark": "352cdf",  // Consider regressions only after this commit
+    //    "another_benchmark": null,   // Skip regression detection altogether
+    // },
+    // The thresholds for relative change in results, after which `asv
+    // publish` starts reporting regressions. Dictionary of the same
+    // form as in ``regressions_first_commits``, with values
+    // indicating the thresholds.  If multiple entries match, the
+    // maximum is taken. If no entry matches, the default is 5%.
+    //
+    // "regressions_thresholds": {
+    //    "some_benchmark": 0.01,     // Threshold of 1%
+    //    "another_benchmark": 0.5,   // Threshold of 50%
+    // },
+}

benchmarks/benchmarks/preprocessing.py

@@ -0,0 +1,99 @@
+"""
+This module will benchmark preprocessing operations in Scanpy
+API documentation: https://scanpy.readthedocs.io/en/stable/api/preprocessing.html
+"""
+
+from __future__ import annotations
+
+import scanpy as sc
+
+import rapids_singlecell as rsc
+
+from .utils import track_peakmem
+
+
+class PreprocessingSuite:
+    _data_dict = dict(pbmc68k_reduced=sc.datasets.pbmc68k_reduced())
+    params = _data_dict.keys()
+    param_names = ["input_data"]
+
+    def setup(self, input_data: str):
+        self.adata = rsc.get.anndata_to_GPU(
+            self._data_dict[input_data].copy(), copy=True
+        )
+
+    def time_calculate_qc_metrics(self, *_):
+        self.adata.var["mt"] = self.adata.var_names.str.startswith("MT-")
+        rsc.pp.calculate_qc_metrics(self.adata, qc_vars=["mt"], log1p=False)
+
+    @track_peakmem
+    def track_peakmem_calculate_qc_metrics(self, *_):
+        self.adata.var["mt"] = self.adata.var_names.str.startswith("MT-")
+        rsc.pp.calculate_qc_metrics(self.adata, qc_vars=["mt"], log1p=False)
+
+    def time_filter_cells(self, *_):
+        rsc.pp.filter_cells(self.adata, qc_var="n_counts", min_count=200)
+
+    @track_peakmem
+    def track_peakmem_filter_cells(self, *_):
+        rsc.pp.filter_cells(self.adata, qc_var="n_counts", min_count=200)
+
+    def time_filter_genes(self, *_):
+        rsc.pp.filter_genes(self.adata, qc_var="n_counts", min_count=3)
+
+    @track_peakmem
+    def track_peakmem_filter_genes(self, *_):
+        rsc.pp.filter_genes(self.adata, qc_var="n_counts", min_count=3)
+
+    def time_normalize_total(self, *_):
+        rsc.pp.normalize_total(self.adata, target_sum=1e4)
+
+    @track_peakmem
+    def track_peakmem_normalize_total(self, *_):
+        rsc.pp.normalize_total(self.adata, target_sum=1e4)
+
+    def time_log1p(self, *_):
+        rsc.pp.log1p(self.adata)
+
+    @track_peakmem
+    def track_peakmem_time_log1p(self, *_):
+        rsc.pp.log1p(self.adata)
+
+    def time_pca(self, *_):
+        rsc.pp.pca(self.adata)
+
+    @track_peakmem
+    def track_peakmem_pca(self, *_):
+        rsc.pp.pca(self.adata)
+
+    def time_highly_variable_genes(self, *_):
+        rsc.pp.highly_variable_genes(
+            self.adata, min_mean=0.0125, max_mean=3, min_disp=0.5
+        )
+
+    @track_peakmem
+    def track_peakmem_highly_variable_genes(self, *_):
+        rsc.pp.highly_variable_genes(
+            self.adata, min_mean=0.0125, max_mean=3, min_disp=0.5
+        )
+
+    def time_regress_out(self, *_):
+        rsc.pp.regress_out(self.adata, ["n_counts", "percent_mito"])
+
+    @track_peakmem
+    def track_peakmem_regress_out(self, *_):
+        rsc.pp.regress_out(self.adata, ["n_counts", "percent_mito"])
+
+    def time_scale(self, *_):
+        rsc.pp.scale(self.adata, max_value=10)
+
+    @track_peakmem
+    def track_peakmem_scale(self, *_):
+        rsc.pp.scale(self.adata, max_value=10)
+
+    def time_neighbors(self, *_):
+        rsc.pp.neighbors(self.adata, n_neighbors=15, n_pcs=50)
+
+    @track_peakmem
+    def track_peakmem_neighbors(self, *_):
+        rsc.pp.neighbors(self.adata, n_neighbors=15, n_pcs=50)