-
Notifications
You must be signed in to change notification settings - Fork 234
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Suggestion to avoid oom GPU error #704
Comments
The first thing you should try is updating to the most recent version of Kilosort. Aside from that, I would need to see your probe file and the settings you're using (either a screenshot of the GUI or copy-pasting them from code) to see if there's something causing more memory to be used than expected. Also, just a side note: Kilosort4 is not set up to use multiple GPUs, so requesting additional GPUs on a cluster job will not do anything. |
probe.prb.zip |
that is the parameters {'batch_size': 60000, 'nblocks': 1, 'Th_universal': 9, 'Th_learned': 8, 'do_CAR': True, 'invert_sign': False, 'nt': 61, 'artifact_threshold': None, 'nskip': 25, 'whitening_range': 32, 'binning_depth': 5, 'sig_interp': 20, 'nt0min': None, 'dmin': None, 'dminx': 32, 'min_template_size': 10, 'template_sizes': 5, 'nearest_chans': 10, 'nearest_templates': 100, 'max_channel_distance': None, 'templates_from_data': True, 'n_templates': 6, 'n_pcs': 6, 'Th_single_ch': 6, 'acg_threshold': 0.2, 'ccg_threshold': 0.25, 'cluster_downsampling': 20, 'cluster_pcs': 64, 'x_centers': None, 'duplicate_spike_bins': 7, 'do_correction': True, 'keep_good_only': False, 'save_extra_kwargs': False, 'skip_kilosort_preprocessing': False, 'scaleproc': None, 'torch_device': 'auto'} |
Okay. It's unlikely it's related to the probe or settings then, since many other users have been sorting similar data without any problems. So, there are two possibilities I can think of:
|
It is also always happening during the 21 iterations of the final clustering, I have 3 different recordings with the same error out of 13 recordings. I will try to find this log file |
I did some investigation, I noticed that the GPU is always at 7.2 GiB of usage each iterations is approximately 30 min during the clustering , however on the problematic recording after the 21th iteration it get stuck, the usage keep stacking up it reach a point where it cannot assign more memory. I also request a GPU where I am alone on it. If you have any idea on why this happening and where in the code I can have look to debug. Also there is no log file in my folder |
this is my output file from the job if its useful |
Thanks. I notice you're running KS4 through SpikeInterface, which is why you're not seeing a log file. I don't see anything in the output that would indicate why you're using so much video memory, similar length recordings with the same number of channels have not used anywhere near that much. One possibility is that you're using SpikeInterface's preprocessing steps in addition to KS4's preprocessing, some other users have reported problems with that. We do not provide support for SpikeInterface, and recommend you run KS4 directly instead. |
My recording is preprocess through spikeinterface, but it then written to disk and I am reading the binary afterward. I also tried to use only kilosort but I have the same issue, I tried running through GUI and use only the openephys file. |
I suppose it's possible, but I don't think that's likely since PyTorch supports 12.1. Are you able to share the data so that I can try running it myself? |
It would be very difficult for me to share the data. |
I am still trying to fix the error. I am looking into cleaning some variable that are not usefull that may leak into memory. I noticed that my GPU start to become huge during the first clustering, would you know which variable I can del from the memory to make some space on the GPU during clustering iteration? |
There is no single variable that would be responsible for that, the only reason you would see that much memory usage is if something is set up incorrectly. I've never exceeded ~10gb of video memory for a recording of similar size. One other possibility, since you mentioned you're writing the spikeinterface preprocessing to disk: are you sure the data is still int16 at that point? Some other users that used that order of operations reported that it's saved as float32 data. Loading that as int16 effectively looks like random noise, which will allow the sorting to run but can cause all sorts of unexpected side-effects. |
I managed to sort one of the recording by going back to the raw data (OpenEphys recording). The GPU stay at 7gb for the full duration of the sorting. |
Can you please share a screenshot of what the full KS4 gui looks like when you load that data? |
I have a similar issue. In the final clustering step, it runs for a while and suddenly allocates too much VRAM and crashes. Do you have any idea why that would happen? This is the error log: File "D:\CNEA.venv\src\kilosort\kilosort\run_kilosort.py", line 523, in cluster_spikes I have a 4070 Nvidia GPU with 12 GB of Dedicated GPU RAM |
Hello @nadirbt. I would have all the same questions for you that I asked above. |
I am not using the kilosort gui. I am using my own data wrapper to enable data access through RecordingExtractorAsArray. I then run the run_kilosort functions without drift correction. |
@nadirbt By above I meant the full thread, not just the previous comment. What version of Kilosort4 are you using? Can you attach |
Hello @jacobpennington, this is my full log and my screenshot from the gui : |
@Kayv-cmb Yeah, that is strange, especially if a similar recording is running fine... Everything does seem to be loading as expected. Unfortunately it's going to be really hard for me to debug without some data to reproduce it with, but I understand that's not always feasible. I have a couple more things you can try, though, to at least narrow it down:
|
I set up the kcoords for contacts on each shank, and now the sorting is slower than before and cause oom issue during the search for spike. |
Here is the log: 06-13 15:42 kilosort.run_kilosort INFO I am using version 0.101.0. This is how I define my probe: |
@nadirbt That is your SpikeInterface version, not Kilosort4's version. The first thing to try is updating to the latest version of Kilosort4 and running it without SpikeInterface. If the error persists, we can go from there. Aside from that, there are some things in your settings that look off. Your batch size is set equal to your sampling rate, so batches are being loaded 1 second at a time. I would recommend increasing |
Describe the issue:
Hi Kilosort people,
torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 7.94 GiB (GPU 0; 31.74 GiB total capacity; 20.90 GiB already allocated; 4.59 GiB free; 26.57 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
Do you have any suggestion to avoid coming into this issue? I am running on a slurm job and I can ask for more GPU but that doesn't solve the problem. Any recommendation as to what to do?(change parameters...).
kilosort version = 4.0.6
The text was updated successfully, but these errors were encountered: