Difference between xiSearch + xiFDR and manual xiFDR validation

[michabirklbauer]

Hi,
I have a question regarding validation in xiSearch / xiFDR:
I have been running xiSearch with the xiFDR interface that is directly in xiSearch, using the settings as displayed in the screenshot (top, red rectangle).
Out of curiosity, I then also run xiFDR manually with seemingly the same settings (see screenshot, bottom) and used the csv file from xiSearch as input. However, when running xiFDR manually I get very different (arguably better) results. While running xiSearch + xiFDR from the xiSearch interface I got around 7200 residue pairs. Running them seperately resulted in almost 9000, that is obviously a very big difference that I did not expect!
I tried different settings in xiFDR to reproduce the original 7200 links that I got from running xiSearch and xiFDR in conjuction, but found nothing so far. Even supplying the used fasta file and xiConfig does not yield the original 7200 linked residue pairs.
Could you give any input on which settings are exactly used in xiFDR, when it is called directly from the xiSearch interface?

Thanks a lot in advance!

[grandrea]

hey, thanks for the questions! Just to make sure, are you looking at the files called "_Links_xiFDR2.1.5.5.csv"? If not, what file are you looking at?

For details on xiFDR, I also suggest its documentation

[michabirklbauer]

Thanks for the reply. I am indeed looking at the files ending in "_Links_xiFDR2.1.5.5.csv". I could also confirm that the files contain as many rows as there are reported links in the "_summary_xiFDR2.1.5.5.csv", ruling out any kind of IO error.
Is it possible that there is some pre-filtering done when running xiFDR through xiSearch (that is not automatically done when running standalone xiFDR)?
I noticed I could reproduce the number of links when using the mzID file that xiFDR created after xiSearch finished, however since this file is already created by xiFDR and not xiSearch, it kind of doesn't explain my observed behaviour.

[grandrea]

both the link and mzIdentML files are generated by xiFDR even when run from xiSEARCH. Do you have non-covalent crosslinker included (seems not from the screenshot). I think the best would be to share or diff your csvs and we or you can look into it.

My suspicion is that the links may indeed indeed the same. Counting the number of rows is not the way to check this... the fact that mzIdentML is the same is giving me confidence that these are indeed the same in terms of crosslinks, but we can check this. Could you run again both but unchecking "boost", which incorporates a grid search?

@lutzfischer, any idea what this may be?

[michabirklbauer]

Nope, only BS3 as crosslinker - with the difference of only considering crosslinks to K and N-term (instead of the xi standard of KSTY and N-term).

I have uploaded all my files here: http://u.pc.cd/B7n (this is a link to pcloud, swiss cloud service). The files are to big to share via github unfortunately. I also included the xiSearch result, config and fasta. The analysis was 2.1 million spectra, so the result file is quite big. The folders are as follows:

• xiSearch_and_xiFDR_manually: The result of using xiSearch and xiFDR seperately, loading the xiSearch result .csv (included in xiSearch_with_xiFDR_through_xiSearch_interface/all_1perc_Xi1.7.6.7.csv. Results in ~9000 links.
• xiSearch_and_xiFDR_manually_between: Same as above but with ticking the boost between option, as pointed out by @lutzfischer this might be the explanation for the difference. I am getting "close" numbers to what I originally got, with around 6900 links
• xiSearch_with_xiFDR_manually_noboost: What you requested, also around 9000 links
• xiSearch_with_xiFDR_trough_xiSearch_interface: My original xiSearch with validation run through the xiSearch interface which yielded ~7200 links.

I guess it really comes down to the boost between option being run by default from xiSearch, that yielded this considerably big difference that I observed at first.

Also I apologize for using the "old" xiSearch version here, I know the newer one is out but I ran the analysis a few days before the newer version was out. Unlucky.

[grandrea]

As an aside, I would not filter at 1% at every level in almost any application- i think this may result in some very wonky fdr estimations, because of not enough decoys passing through or not enough targets. Do you get any warning like "too many DD" or similar?

[grandrea]

the issue is that the lists are consistent among them- if you put for example, 1% on PPI and 1% on residue pair, then only the residue pairs that can also pass the PPI FDR are included in the output. So for what you want to do, you need to run xiFDR 4 times where you put 1% on CSM, 100% on everything else, then 1% on residue pairs and so on.

[grandrea]

in short- filter only at the level you want to compare to, as covered in xiFDR docs

[michabirklbauer]

Got it! Will do that moving forward. Thanks for the input! I read the xiFDR docs but this was not clear to me.

[grandrea]

sorry, i will try to make that clearer

[michabirklbauer]

No worries, not your fault! It probably is just a "me" problem 😀

[lutzfischer]

There is a bug in the xiFDR - commandline version (just found it) where it boosts for protein heteromeric links - even if not told so.
And xiSEARCH is using the xiFDR commandline version when not opening the gui. I am in the process of making an update.

The result should be that you get more protein heteromeric (inter/between) links if you run directly from xiSEARCH but likely at the cost of self-links (intra).

[michabirklbauer]

Thanks a lot for looking into that. I guess this mostly explains the differnce I observed.

[lutzfischer]

There is actually one additional difference - the default setting for minimum peptide length is different: 5 for GUI and 6 for command-line - will change that as well

[michabirklbauer]

I can confirm that with the boost "between" and setting minimum peptide length to 6 it now gives me exactly the results I got originally! Thanks for solving that! 😀

[lutzfischer]

And updated version of xiFDR has been published (2.2.2) and a new version of xiSEARCH (1.8.3) that bundles this version of xiFDR