Pandas incompatibility between merlin-core and drvi-py

[rosshandler]

Report

Hi,

I have install drvi and works nice but produces a warning indicating that merlin dataloader is not installed, when I try to install it I get incompatibility issues with drvi because of pandas. It seem merlin needs an older version

I still can use drvi but I thought it would be good to highlight this.

Below the error when installing drvi after merlin-core

ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts. dask-cuda 24.2.0 requires pandas<1.6.0.dev0,>=1.3, but you have pandas 2.2.3 which is incompatible. merlin-core 23.8.0 requires pandas<1.6.0dev0,>=1.2.0, but you have pandas 2.2.3 which is incompatible.

Version information

No response

[moinfar]

Hi,

Thanks for raising the issue.

1. Using DRVI does not require any merlin dataloader features. MerlinDataloader is an advanced feature that allows large prepared data in Merlin-friendly format (not .h5ad) to be directly loaded into GPU. However, this feature is usually not needed, even for datasets up to 10 million cells.
2. If you really want to use this feature (we did not document how to make the data), you should use this command for installation: pip install "drvi-py[merlin]". This command works for me on Python 3.10 to 3.12. if it is not working, please provide more information, such as the Python version and system info, so we can reproduce the error.

P.S. I would add that information to the documentation and fix the warning to prevent future confusion.