fast array utils spinoff package

[flying-sheep]

Candidates found from this zulip message, listed sub-issues, and by searching .indices in the scanpy codebase

definitely:

• asarray (anndata. very handy.)
  • to_dense(cs_matrix) (scanpy. probably good as implementation detail for asarray)
• get_mean_var (scanpy. one of the most complete, optimized pieces we have)
  • axis_mean (scanpy. used in get_mean_var)
    • axis_sum (scanpy. used in axis_mean)
• is_constant (scanpy. quite complete and very basic functionality)

likely:

• elem_mul (scanpy)
  • axis_mul_or_truediv (scanpy. should probably be fused with elem_mul)
• axis_nnz (scanpy)
• scale (scanpy. is currently singledispatch with support for anndata and arrays)

maybe:

• gearys_c/morans_i (scanpy. maybe out of scope?)
• top_segment_proportions (scanpy. maybe a bit niche? We have Gearys C and Morans I As Public API #3291 for it, but maybe metrics should live elsewhere?)
• calculate_residuals (scanpy. maybe too niche?)

[ilan-gold]

Two possible ways forward:

• Rewrite in rust in new package
• Port directly (no-rewrite) to new package

[Zethson]

gearys_c/morans_i (scanpy. maybe out of scope?)

I think that this is useful for a few places such as scIB. A bit more niche but useful enough if you foresee it to be pretty stable.

[flying-sheep]

OK, created: https://github.com/scverse/fast-array-utils

[flying-sheep]

@LucaMarconato, you said that squidpy needs to_dense: does that mean “everything becomes a numpy.ndarray” or does it mean “in-memory arrays become numpy.ndarrays and dask arrays containing sparse matrices become dask arrays containing ndarrays)?

[LucaMarconato]

The first. For context https://github.com/scverse/napari-spatialdata/blob/bc7bf6186df488726d63efd18d6015f0963fa521/src/napari_spatialdata/utils/_utils.py#L54 we have this function here that we use to convert single columns from data.obs and data.X into a dense numpy.ndarray, whether they are series, dense columns or sparse columns. Maybe it could be moved in the spinoff pacakge.

[flying-sheep]

scverse/fast-array-utils#31