Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

(feat): pre-processing functions for dask with sparse chunks #2856

Merged
merged 96 commits into from
Mar 22, 2024
Merged

(feat): pre-processing functions for dask with sparse chunks #2856

merged 96 commits into from
Mar 22, 2024

Commits on Feb 27, 2024

  1. (chore): add dask sparse chunks creation

    @ilan-gold
    ilan-gold committed Feb 27, 2024
    Configuration menu
    Copy the full SHA
    d3163cc View commit details
    Browse the repository at this point in the history

  2. (feat): add dask summation

    @ilan-gold
    ilan-gold committed Feb 27, 2024
    Configuration menu
    Copy the full SHA
    2a7a54c View commit details
    Browse the repository at this point in the history

  3. (refactor): materialize_as_ndarray needs to operate on indidiual da… 

    …sk arrays
    @ilan-gold
    ilan-gold committed Feb 27, 2024
    Configuration menu
    Copy the full SHA
    8dd9a7a View commit details
    Browse the repository at this point in the history

  4. (feat): filter_genes and filter_cells

    @ilan-gold
    ilan-gold committed Feb 27, 2024
    Configuration menu
    Copy the full SHA
    d55b6a4 View commit details
    Browse the repository at this point in the history

  5. (feat): normalization

    @ilan-gold
    ilan-gold committed Feb 27, 2024
    Configuration menu
    Copy the full SHA
    41a5f15 View commit details
    Browse the repository at this point in the history

  6. (fix) lop1p tests working

    @ilan-gold
    ilan-gold committed Feb 27, 2024
    Configuration menu
    Copy the full SHA
    e36699e View commit details
    Browse the repository at this point in the history

  7. (refactor): clean up writing test

    @ilan-gold
    ilan-gold committed Feb 27, 2024
    Configuration menu
    Copy the full SHA
    da6eff0 View commit details
    Browse the repository at this point in the history

  8. (refactor): us da.chunk.sum

    @ilan-gold
    ilan-gold committed Feb 27, 2024
    Configuration menu
    Copy the full SHA
    63ca2f0 View commit details
    Browse the repository at this point in the history

  9. (fix): remove Client

    @ilan-gold
    ilan-gold committed Feb 27, 2024
    Configuration menu
    Copy the full SHA
    fd22a19 View commit details
    Browse the repository at this point in the history

  10. (refactor): remove unnecessary count_nonzero

    @ilan-gold
    ilan-gold committed Feb 27, 2024
    Configuration menu
    Copy the full SHA
    8b9a792 View commit details
    Browse the repository at this point in the history

  11. (fix): change expected fail on sparse normalization

    @ilan-gold
    ilan-gold committed Feb 27, 2024
    Configuration menu
    Copy the full SHA
    1592571 View commit details
    Browse the repository at this point in the history

Commits on Feb 28, 2024

  1. (fix): update comment

    @ilan-gold
    ilan-gold committed Feb 28, 2024
    Configuration menu
    Copy the full SHA
    6ac32e5 View commit details
    Browse the repository at this point in the history

  2. (feat): _get_mean_var dask

    @ilan-gold
    ilan-gold committed Feb 28, 2024
    Configuration menu
    Copy the full SHA
    78a3ab6 View commit details
    Browse the repository at this point in the history

  3. (feat): clean up tests for what should/should not work

    @ilan-gold
    ilan-gold committed Feb 28, 2024
    Configuration menu
    Copy the full SHA
    60bbdb8 View commit details
    Browse the repository at this point in the history

  4. (refactor): _compat.sum to _utils.elem_sum

    @ilan-gold
    ilan-gold committed Feb 28, 2024
    Configuration menu
    Copy the full SHA
    0c1f254 View commit details
    Browse the repository at this point in the history

  5. (chore): add elem_sum test

    @ilan-gold
    ilan-gold committed Feb 28, 2024
    Configuration menu
    Copy the full SHA
    2f4d11a View commit details
    Browse the repository at this point in the history

Commits on Feb 29, 2024

  1. (refactor): elem_sum -> axis_sum

    @ilan-gold
    ilan-gold committed Feb 29, 2024
    Configuration menu
    Copy the full SHA
    12502e8 View commit details
    Browse the repository at this point in the history

  2. (feat): add scale support

    @ilan-gold
    ilan-gold committed Feb 29, 2024
    Configuration menu
    Copy the full SHA
    b3bb95a View commit details
    Browse the repository at this point in the history

  3. (fix): maintain dtype

    @ilan-gold
    ilan-gold committed Feb 29, 2024
    Configuration menu
    Copy the full SHA
    2b6f717 View commit details
    Browse the repository at this point in the history

  4. (chore): add back condition

    @ilan-gold
    ilan-gold committed Feb 29, 2024
    Configuration menu
    Copy the full SHA
    448dc40 View commit details
    Browse the repository at this point in the history

  5. (fix): use sum when needed

    @ilan-gold
    ilan-gold committed Feb 29, 2024
    Configuration menu
    Copy the full SHA
    7226bf0 View commit details
    Browse the repository at this point in the history

  6. (chore): release notes

    @ilan-gold
    ilan-gold committed Feb 29, 2024
    Configuration menu
    Copy the full SHA
    2bc7c3a View commit details
    Browse the repository at this point in the history

  7. (fx): don't use mean_func name twice

    @ilan-gold
    ilan-gold committed Feb 29, 2024
    Configuration menu
    Copy the full SHA
    62c75fc View commit details
    Browse the repository at this point in the history

  8. (chore): revert sparse-chunks-in-dask

    @ilan-gold
    ilan-gold committed Feb 29, 2024
    Configuration menu
    Copy the full SHA
    b987a68 View commit details
    Browse the repository at this point in the history

  9. (chore): type hint

    @ilan-gold
    ilan-gold committed Feb 29, 2024
    Configuration menu
    Copy the full SHA
    902238a View commit details
    Browse the repository at this point in the history

  10. (chore): check test_compare_to_upstream

    @ilan-gold
    ilan-gold committed Feb 29, 2024
    Configuration menu
    Copy the full SHA
    0abbab5 View commit details
    Browse the repository at this point in the history

  11. (chore): remove comment

    @ilan-gold
    ilan-gold committed Feb 29, 2024
    Configuration menu
    Copy the full SHA
    a8606ae View commit details
    Browse the repository at this point in the history

  12. (chore): allow passing dtype arg in axis_sum

    @ilan-gold
    ilan-gold committed Feb 29, 2024
    Configuration menu
    Copy the full SHA
    feac6bc View commit details
    Browse the repository at this point in the history

  13. (fix): revert fixture changes

    @ilan-gold
    ilan-gold committed Feb 29, 2024
    Configuration menu
    Copy the full SHA
    bcdeddb View commit details
    Browse the repository at this point in the history

  14. (refactor): cleaner with array_type conversion before if-then

    @ilan-gold
    ilan-gold committed Feb 29, 2024
    Configuration menu
    Copy the full SHA
    4716d8f View commit details
    Browse the repository at this point in the history

  15. (chore): clarify hvg support

    @ilan-gold
    ilan-gold committed Feb 29, 2024
    Configuration menu
    Copy the full SHA
    3912b63 View commit details
    Browse the repository at this point in the history

Commits on Mar 1, 2024

  1. (chore): handle array types better

    @ilan-gold
    ilan-gold committed Mar 1, 2024
    Configuration menu
    Copy the full SHA
    c884c63 View commit details
    Browse the repository at this point in the history

  2. (chore): clean up materialize_as_ndarray

    @ilan-gold
    ilan-gold committed Mar 1, 2024
    Configuration menu
    Copy the full SHA
    af351d4 View commit details
    Browse the repository at this point in the history

  3. (chore): fix typing/dispatch problem in 3.9

    @ilan-gold
    ilan-gold committed Mar 1, 2024
    Configuration menu
    Copy the full SHA
    da22953 View commit details
    Browse the repository at this point in the history

  4. (chore): list type -> Callable

    @ilan-gold
    ilan-gold committed Mar 1, 2024
    Configuration menu
    Copy the full SHA
    dbbc6a2 View commit details
    Browse the repository at this point in the history

  5. (feat): row_divide for better division handling

    @ilan-gold
    ilan-gold committed Mar 1, 2024
    Configuration menu
    Copy the full SHA
    6a4f0c5 View commit details
    Browse the repository at this point in the history

  6. (fix): use tuple for ARRAY_TYPEXXXX

    @ilan-gold
    ilan-gold committed Mar 1, 2024
    Configuration menu
    Copy the full SHA
    c0182cb View commit details
    Browse the repository at this point in the history

  7. (refactor): mean_func -> axis_mean + types

    @ilan-gold
    ilan-gold committed Mar 1, 2024
    Configuration menu
    Copy the full SHA
    a065a78 View commit details
    Browse the repository at this point in the history

  8. (chore): remove unnecessary aggregation

    @ilan-gold
    ilan-gold committed Mar 1, 2024
    Configuration menu
    Copy the full SHA
    c3ee138 View commit details
    Browse the repository at this point in the history

  9. (fix): raise ValueError for summing over more than one axis

    @ilan-gold
    ilan-gold committed Mar 1, 2024
    Configuration menu
    Copy the full SHA
    c246a41 View commit details
    Browse the repository at this point in the history

  10. (fix): grammar

    @ilan-gold
    ilan-gold committed Mar 1, 2024
    Configuration menu
    Copy the full SHA
    743c327 View commit details
    Browse the repository at this point in the history

  11. (fix): better type hints

    @ilan-gold
    ilan-gold committed Mar 1, 2024
    Configuration menu
    Copy the full SHA
    db88560 View commit details
    Browse the repository at this point in the history

  12. (revert): use old test_normalize_total siince we have csr

    @ilan-gold
    ilan-gold committed Mar 1, 2024
    Configuration menu
    Copy the full SHA
    2a5faa6 View commit details
    Browse the repository at this point in the history

  13. (revert): extraneous diff

    @ilan-gold
    ilan-gold committed Mar 1, 2024
    Configuration menu
    Copy the full SHA
    d6ceb4c View commit details
    Browse the repository at this point in the history

  14. (fix): try Union

    @ilan-gold
    ilan-gold committed Mar 1, 2024
    Configuration menu
    Copy the full SHA
    48a1a1e View commit details
    Browse the repository at this point in the history

  15. (chore): add column division ability

    @ilan-gold
    ilan-gold committed Mar 1, 2024
    Configuration menu
    Copy the full SHA
    07fc5ba View commit details
    Browse the repository at this point in the history

  16. (chore): add scale test

    @ilan-gold
    ilan-gold committed Mar 1, 2024
    Configuration menu
    Copy the full SHA
    3cc4be2 View commit details
    Browse the repository at this point in the history

  17. (fix): duplicate in release note

    @ilan-gold
    ilan-gold committed Mar 1, 2024
    Configuration menu
    Copy the full SHA
    4cc9eef View commit details
    Browse the repository at this point in the history

  18. (refactor): guard clause + comments

    @ilan-gold
    ilan-gold committed Mar 1, 2024
    Configuration menu
    Copy the full SHA
    d8afe5c View commit details
    Browse the repository at this point in the history

  19. (chore): add out check for dask

    @ilan-gold
    ilan-gold committed Mar 1, 2024
    Configuration menu
    Copy the full SHA
    271d5d8 View commit details
    Browse the repository at this point in the history

  20. (chore): add divisor type hints

    @ilan-gold
    ilan-gold committed Mar 1, 2024
    Configuration menu
    Copy the full SHA
    c61324b View commit details
    Browse the repository at this point in the history

  21. (fix): remove some erroneous diffs

    @ilan-gold
    ilan-gold committed Mar 1, 2024
    Configuration menu
    Copy the full SHA
    c688aff View commit details
    Browse the repository at this point in the history

  22. (chore): axis_{sum,mean} type hint fixes

    @ilan-gold
    ilan-gold committed Mar 1, 2024
    Configuration menu
    Copy the full SHA
    02be7a7 View commit details
    Browse the repository at this point in the history

Commits on Mar 4, 2024

  1. (refactor): generalize to scaling

    @ilan-gold
    ilan-gold committed Mar 4, 2024
    Configuration menu
    Copy the full SHA
    6acc08c View commit details
    Browse the repository at this point in the history

  2. (chore): remove erroneous comment

    @ilan-gold
    ilan-gold committed Mar 4, 2024
    Configuration menu
    Copy the full SHA
    0944429 View commit details
    Browse the repository at this point in the history

  3. (chore): remove non-public API

    @ilan-gold
    ilan-gold committed Mar 4, 2024
    Configuration menu
    Copy the full SHA
    3538572 View commit details
    Browse the repository at this point in the history

  4. (fix): import from sc._utils

    @ilan-gold
    ilan-gold committed Mar 4, 2024
    Configuration menu
    Copy the full SHA
    5ef1487 View commit details
    Browse the repository at this point in the history

  5. (fix): inidices -> indices

    @ilan-gold
    ilan-gold committed Mar 4, 2024
    Configuration menu
    Copy the full SHA
    0f43362 View commit details
    Browse the repository at this point in the history

Commits on Mar 5, 2024

  1. (fix): remove erroneous axis_sum calls

    @ilan-gold
    ilan-gold committed Mar 5, 2024
    Configuration menu
    Copy the full SHA
    c100a8f View commit details
    Browse the repository at this point in the history

  2. (fix): return statements for axis_scale

    @ilan-gold
    ilan-gold committed Mar 5, 2024
    Configuration menu
    Copy the full SHA
    22b4e90 View commit details
    Browse the repository at this point in the history

  3. (refactor): return out of axis_sum if X._meta is np.ndarray

    @ilan-gold
    ilan-gold committed Mar 5, 2024
    Configuration menu
    Copy the full SHA
    4fef58e View commit details
    Browse the repository at this point in the history

  4. (core): comment fix

    @ilan-gold
    ilan-gold committed Mar 5, 2024
    Configuration menu
    Copy the full SHA
    ce574e3 View commit details
    Browse the repository at this point in the history

  5. (fix): use normalize_total in HVG test for true reproducibility

    @ilan-gold
    ilan-gold committed Mar 5, 2024
    Configuration menu
    Copy the full SHA
    e5a82fc View commit details
    Browse the repository at this point in the history

Commits on Mar 6, 2024

  1. (refactor): separate out out test for dask

    @ilan-gold
    ilan-gold committed Mar 6, 2024
    Configuration menu
    Copy the full SHA
    a4e53a6 View commit details
    Browse the repository at this point in the history

  2. (fix): correct chunking/rechunking behavior

    @ilan-gold
    ilan-gold committed Mar 6, 2024
    Configuration menu
    Copy the full SHA
    f0b2d97 View commit details
    Browse the repository at this point in the history

  3. (chore): add guard clause for sparse out != X != None in scaling

    @ilan-gold
    ilan-gold committed Mar 6, 2024
    Configuration menu
    Copy the full SHA
    f9ea93d View commit details
    Browse the repository at this point in the history

  4. (fix): guard clause condition

    @ilan-gold
    ilan-gold committed Mar 6, 2024
    Configuration menu
    Copy the full SHA
    66f04b6 View commit details
    Browse the repository at this point in the history

  5. (fix): try finishing | typing for 3.9

    @ilan-gold
    ilan-gold committed Mar 6, 2024
    Configuration menu
    Copy the full SHA
    daca210 View commit details
    Browse the repository at this point in the history

  6. (fix): call register to allow unions?

    @ilan-gold
    ilan-gold committed Mar 6, 2024
    Configuration menu
    Copy the full SHA
    036391e View commit details
    Browse the repository at this point in the history

  7. (fix): clarify warning

    @ilan-gold
    ilan-gold committed Mar 6, 2024
    Configuration menu
    Copy the full SHA
    cac4160 View commit details
    Browse the repository at this point in the history

  8. (feat): test for max_value/zero_center combos

    @ilan-gold
    ilan-gold committed Mar 6, 2024
    Configuration menu
    Copy the full SHA
    9ec6935 View commit details
    Browse the repository at this point in the history

  9. (fix): allow settings of X in scale_array

    @ilan-gold
    ilan-gold committed Mar 6, 2024
    Configuration menu
    Copy the full SHA
    0ae76ee View commit details
    Browse the repository at this point in the history

  10. (chore): add tests for normalize correctness

    @ilan-gold
    ilan-gold committed Mar 6, 2024
    Configuration menu
    Copy the full SHA
    2367f46 View commit details
    Browse the repository at this point in the history

  11. (fix): refactor for pure dask in median

    @ilan-gold
    ilan-gold committed Mar 6, 2024
    Configuration menu
    Copy the full SHA
    b2c3a96 View commit details
    Browse the repository at this point in the history

  12. Merge branch 'main' into dask-sparse-mean-var

    @ilan-gold
    ilan-gold authored Mar 6, 2024
    Configuration menu
    Copy the full SHA
    340894b View commit details
    Browse the repository at this point in the history

  13. (refactor): add clarifying condition

    @ilan-gold
    ilan-gold committed Mar 6, 2024
    Configuration menu
    Copy the full SHA
    fa66f58 View commit details
    Browse the repository at this point in the history

  14. Merge branch 'dask-sparse-mean-var' of github.com:scverse/scanpy into… 

    … dask-sparse-mean-var
    @ilan-gold
    ilan-gold committed Mar 6, 2024
    Configuration menu
    Copy the full SHA
    2601fe8 View commit details
    Browse the repository at this point in the history

  15. (chore): skip warning computations + tests

    @ilan-gold
    ilan-gold committed Mar 6, 2024
    Configuration menu
    Copy the full SHA
    750af59 View commit details
    Browse the repository at this point in the history

  16. (fix): actually skip computation in normalize_total condition

    @ilan-gold
    ilan-gold committed Mar 6, 2024
    Configuration menu
    Copy the full SHA
    25fe1f9 View commit details
    Browse the repository at this point in the history

  17. (fix): actually skip in filter_genes + tests

    @ilan-gold
    ilan-gold committed Mar 6, 2024
    Configuration menu
    Copy the full SHA
    57c8389 View commit details
    Browse the repository at this point in the history

Commits on Mar 7, 2024

  1. (fix): use all-in-one median implemetation

    @ilan-gold
    ilan-gold committed Mar 7, 2024
    Configuration menu
    Copy the full SHA
    69ebf98 View commit details
    Browse the repository at this point in the history

  2. (refactor): remove erreous dask warnings

    @ilan-gold
    ilan-gold committed Mar 7, 2024
    Configuration menu
    Copy the full SHA
    67f47f4 View commit details
    Browse the repository at this point in the history

  3. (chore): add note about exclude_highly_expressed

    @ilan-gold
    ilan-gold committed Mar 7, 2024
    Configuration menu
    Copy the full SHA
    e328eb5 View commit details
    Browse the repository at this point in the history

  4. (feat): axis_scale -> axis_mul_or_truediv

    @ilan-gold
    ilan-gold committed Mar 7, 2024
    Configuration menu
    Copy the full SHA
    0aafabd View commit details
    Browse the repository at this point in the history

  5. (feat): allow_divide_by_zero

    @ilan-gold
    ilan-gold committed Mar 7, 2024
    Configuration menu
    Copy the full SHA
    be988c9 View commit details
    Browse the repository at this point in the history

  6. (chore): add notes + type hints

    @ilan-gold
    ilan-gold committed Mar 7, 2024
    Configuration menu
    Copy the full SHA
    3166909 View commit details
    Browse the repository at this point in the history

Commits on Mar 20, 2024

  1. Have hvg compute earlier and only once

    @ivirshup
    ivirshup committed Mar 20, 2024
    Configuration menu
    Copy the full SHA
    6552324 View commit details
    Browse the repository at this point in the history

Commits on Mar 21, 2024

  1. Merge branch 'main' into dask-sparse-mean-var

    @ilan-gold
    ilan-gold authored Mar 21, 2024
    Configuration menu
    Copy the full SHA
    936eb87 View commit details
    Browse the repository at this point in the history

  2. [pre-commit.ci] auto fixes from pre-commit.com hooks 

    for more information, see https://pre-commit.ci
    @pre-commit-ci
    pre-commit-ci[bot] committed Mar 21, 2024
    Configuration menu
    Copy the full SHA
    37eb1a0 View commit details
    Browse the repository at this point in the history

Commits on Mar 22, 2024

  1. (refactor): make codecov better by removing dead code/refactoring

    @ilan-gold
    ilan-gold committed Mar 22, 2024
    Configuration menu
    Copy the full SHA
    0cecb94 View commit details
    Browse the repository at this point in the history

  2. (fix): np.clip in dask does not take min/max as kwargs

    @ilan-gold
    ilan-gold committed Mar 22, 2024
    Configuration menu
    Copy the full SHA
    21faa0d View commit details
    Browse the repository at this point in the history

  3. Update docs/release-notes/1.11.0.md 

    Co-authored-by: Isaac Virshup <[email protected]>
    @ilan-gold @ivirshup
    ilan-gold and ivirshup authored Mar 22, 2024
    Configuration menu
    Copy the full SHA
    5998ae8 View commit details
    Browse the repository at this point in the history

  4. (chore): move release note

    @ilan-gold
    ilan-gold committed Mar 22, 2024
    Configuration menu
    Copy the full SHA
    f49b929 View commit details
    Browse the repository at this point in the history

  5. Merge branch 'main' into dask-sparse-mean-var

    @ilan-gold
    ilan-gold authored Mar 22, 2024
    Configuration menu
    Copy the full SHA
    937c6db View commit details
    Browse the repository at this point in the history

  6. (chore): remove erroneous comment

    @ilan-gold
    ilan-gold committed Mar 22, 2024
    Configuration menu
    Copy the full SHA
    ba445f8 View commit details
    Browse the repository at this point in the history

  7. Merge branch 'dask-sparse-mean-var' of github.com:scverse/scanpy into… 

    … dask-sparse-mean-var
    @ilan-gold
    ilan-gold committed Mar 22, 2024
    Configuration menu
    Copy the full SHA
    b3581ea View commit details
    Browse the repository at this point in the history