500,000 nuclei of multiome data has very high memory requirements for batch correction (1.2 TB)

[mgrantpeters]

I'm experiencing very high memory requirements with a multiome dataset much smaller than the TAURUS data reported in the manuscript (fig 5). While I appreciate that having paired ATAC and RNA will have higher computation required than RNA only, requiring >1TB seems excessive. Does panpipes have implemented optional strategies to reduce memory requirements that could be useful in this case (e.g. sparse matrix conversion, on disk storage rather than in memory)?

[bio-la]

Hi @mgrantpeters thanks for reaching out! I agree that the behaviour you observe is rather strange. To help us troubleshoot, could you please let me know:

• which integration method are you using?
• how many features does your object have? (ATAC and RNA)
  thanks!