Running one Kilosort 4 session for two days and sometimes even crashed

[ZhongRenNeuroscience]

Describe the issue:

Hi, I have a problem that Kilosort 4 takes quite long time to sort one session. Sometimes even crashed.

I recorded the data with Neuropixel V2.0-OpenEphys lasting one and half to two hours for each session. I tried Kilosort 4.0.5 and the newest version 4.0.13. For the same session, I firstly run the whole session and then Kilosort 4 crashed at certain time point (see log file 1
kilosort4.log
). Then I updated Kilosort, and then run half of the same recording session. It can run well, but it takes me two days (see log file 2
kilosort4.log
). I know that my recording is a little bit too big. But I would like to know whether there is solusion for this problem. Thank you very much!

Reproduce the bug:

No response

Error message:

No response

Version information:

Python 3.11
Kilosort4.0.5 and 4.0.13
Windows10

[jacobpennington]

Hello @ZhongRenNeuroscience,

That recording is not too big at all, it should only take two or three hours to sort with a decent GPU (like a geforce 3000 or 4000 series). A few questions for you:

1. Are you using the GUI or API? Some others have reported the API running very slow for certain IDEs like Spyder and IDLE. If that's the case, try running it from a terminal / powershell instead.
2. Can you please share a screenshot of what the full Kilosort GUI looks like when you load your data?
3. What kind of hardware are you using to sort this?
4. Is the data stored on a SSD on the machine you're sorting with? If you're sorting over a network drive or something else, that could introduce some issues with I/O speed.

[ZhongRenNeuroscience]

Hello,

Thanks for your reply. Here are the answers to your questions:

1. I'm using Gui, so it should not be a problem.
2. Please see the attachment
   
3. I hope these include the information you want
   PC Hardware.txt
   
   
4. Because the SSD only has limited space, I saved and ran the data in the HDD of my computer. I don't know how big difference it can make. We are considering to replace with a larger SSD.

Looking forward to your reply. Thank you!

[jacobpennington]

Okay, there are a few things that might be related:

1. I noticed you're using smaller values for Th (universal) and Th (learned). Unless you've already sorted some similar datasets with Kilosort4 and found a need to change those, you should try sorting with the default values first (9 and 8, respectively). You should not try to set these to values you used with earlier versions of Kilosort, there is not a direct relationship with those parameters. If you're detecting a lot of extraneous spikes, that will slow down sorting. I don't think that's the cause of this issue since the number of spikes detected still looks reasonable, but it's something to keep in mind.

2. Your graphics card is several generations old, and does not support some newer CUDA features. I'm not sure if any of those features would specifically boost performance for some of the tensorflow functions we're using, but I would not be surprised. The only way to know would be to try sorting with a different machine with a newer card. I know GeForce 3000 and 4000 series cards work well for KS4, for example. If you don't have a different machine readily available, you're welcome to share the data with me so that I can try sorting it myself.

3. Using a HDD instead of SDD will definitely slow down sorting. I don't think it would be responsible for that big of a slow-down (around 40x slower than expected based on channel count and recording duration), but using a SDD is strongly recommended. The difference is large enough that it's worth transferring your data file to SDD, sorting on SDD, then transferring the file back to HDD, if your SDD is large enough to at least hold one recording at a time.