Memory optimisations

[lutzfischer]

When searching very large databases memory can become a constraint. (xiSEARCH versions 1.8 and newer should warn if memory becomes a constraint).

One option is to simply give xiSEARCH more memory (xiSEARCh version <1.8 with the start parameter -Xmx, >=1.8 as a setting in the GUI). But this is limited by how much memory is available. There are some options that can be considered when using more memory is not an option

Memory constraints

Mainly there are two set of memory constraint in xiSEARCH.

Fixed Memory constraints

First is a "fixed" restrain through out the run time. This is mainly dependent on how many peptides and in result fragments are to be considered for crosslinks.
This is mainly affected by the number of proteins (and size of these) and the number of variable modifications.
Limiting the proteins in your FASTA file to only proteins likely abundant enough in your sample can help and also some settings related to modifications.

MAX_MODIFICATION_PER_PEPTIDE:2

• default is 3
• this would limit the number of variable modification accepted on a peptide t 2
• results in fewer modified peptides in total
  MAX_MODIFIED_PEPTIDES_PER_PEPTIDE:10
• default is 20
• based on unmodified peptides (or with fixed modifications) modified peptides are generated
• this limits how many variable modified peptides can at most be generated from a single peptide
• first all peptides with a single modification are generated then with two and so on.

MAXPEPTIDEMASS:4000

• default is the largest observed precusor in any of the searched peaklists
• only consider peptides with up to this mass

Per Thread Memory constraints

Second memory constraint is how much memory each thread can use. If a search appears to stall in the middle of a search you likely run against this. I.e the search starts but then appears to stop after some percent of spectra where processed. This is down to xiSEARCH running out of memory during search and then java spends more time in trying to free up some memory then actually searching spectra. This often results in an OutOfMemoryError at some point.

Reducing number of search threads:

• as each thread will require some amount of memory (sometimes 1 to 2 GB) reducing these can reduce the overall need of memory. When memory is limited reducing the number of search threads can actually speed up the search.

maxpeakcandidates:10000

• default: unlimited
• when looking for candidate peptides only consider peaks in a spectrum that result in less then this number of candidate peptides
• this will somewhat limit how many peptide candidates have to be considered for first scoring

mgcpeaks:5

• default: 10
• for first candidate search a denoised spectra is used. This defines how many peaks per 100Th are used
• fewer peaks will result in fewer candidates to be considered

[fi7855ar]

Hi Lutz and thank you for providing this guide in how to optimize memory usage running xiSearch.
I have a question regarding running xiSearch jobs in parallel on a server or HPC:
When I look at the xiSearch readme section for hpc-cluster I see that it's recommended to submit one search job per .mgf peakfile. I see that you allocate 100G for each search job regardless of the number of reads the file has, but with a server where we have sparse resources but still want to optimize search speed, would a search job go faster if resource priorities would be made with regards to the peakfiles number of reads / filesize?
Let's say we have 50G memory to allocate on two jobs - one file has 30 000 reads while the other has 15 000. Would it go faster if the search of 30 000 reads file got, let's say 35G and the smaller file at 15 000 reads would get 15G? in a paralell search, given that those two results would be combined at a later stage to a combined results file. Assume that both searches use the same config settings and fasta (they're from the same experiment)

Thank you!

[grandrea]

Hello,

Thanks for the question. Maybe I answer, since those scripts I have to say those scripts are more examples than recommendations, especially with regards to memory consumption. That is a function of the size of the database, which is dictated by number of sequences, missed cleavages and modifications. So 100Gb is a number I used to search a large database with hundreds of proteins, 3 missed cleavages and 4 modifications. This can be considerably lowered for simpler databases.

I think, and @lutzfischer can correct me here, the speed per scanwould be the same in your example as long as you are under the memory limit in both cases, just the larger file would take longer. However, you can split the larger runs into multiple files each contaning for example 15 0000 scans to have each job process the same amount of reads.

[fi7855ar]

Thank you @grandrea for the quick answer! Would it also be feasible to zip all peakfiles and run it as one "mega" search in a single batch - or have you experienced any issues with this approach?

[lutzfischer]

If you are running on a single server probably the fastest is to actually, to just run all files together as one search. This can either be a zip-file or add them individually to the search.
When zipping them up just make sure your using something like 7zip (but still compressed as zip file) and not windows compressed folders. Windows switched at some point to use deflate64 for zip compression of large files and xiSEARCH has currently no support for that. Meaning if you use 7zip (or what ever other software) to generate the zip-file make sure to NOT to select deflate64 as compression method, i.e. select something like deflate or deflate32.

[grandrea]

Usually thread count per node is the limiting factor to the "mega" search approach. if you have access to an HPC with a scheduler and many compute nodes, you can go beyond that. If you have like 500 runs, with the parallel script you can submit each job/peak file with e.g. 16 threads (--tasks-per-node) and this mean 2 runs fit in a node of 32 threads (or 12 for 24t nodes etc). If your HPC has many nodes, you can then run many jobs in parallel, significantly reuducing the time it takes to finish the search (because each run is processed with 16t at a time).

This way you can run chunks of your search in "parallel" across multiple nodes. With large number of runs, this will be faster overall than a single "mega" search, albeit less efficient (each job will be allocated the chunk of memory required by the full search) since resources are not shared. But if all you are after is just finishing the search faster, it is a big boost.

If instead you have access to a single compute server (e.g. a linux server with 128 threads with no job scheduler, or a beefy workstation) a single search will probably be both faster and more efficient, like lutz said.

[fi7855ar]

Alright, thank you both @lutzfischer and @grandrea, your answers have been much appreciated! We currently have a single SLURM node setup that has ~ 180 threads and 1 TB of RAM, so maybe in this case the mega approach would be best as you say :)
I use linux zip command to compress them to a zip archive and it works without any problems!