F-scores with undefined parametric F-test are given p-values of zero

[BryanGuillaume]

In swe_contrasts, when the estimated number of degrees of freedom is smaller than the constrat rank minus 1 (i.e. when edf < rankContrast - 1), the parametric F-test is undefined because we have (edf - rankContrast + 1)/ edf * (F-score / rankContrast) ~ F(rankContrast, edf - rankContrast + 1). In this scenario, the toolbox currently sets the score and the equivalent score to 0, but also set the p-value to zero, which may not be adequate:

SwE-toolbox/swe_contrasts.m

Line 543 in cfa0528

uncP(score == 0) = 0;

I believe that would be fine if SPM treats all zero values as nan, but I am not sure about the latter. So I see thee options here:

1. Leave the code as is provided SPM treats zero as missing, but I doubt that is a good solution
2. Leave the scores set to 0, but set the p-values to 1.
3. Nan the scores and p-values.

One could also argue that setting the F-scores to 0 may not be appropriate, because it is actually defined, only its estimated parametric distribution is not defined.

@nicholst, do you have any opinion about this?

[nicholst]

A p-value of zero certainly wrong. The edf are approximations, and, just as variance estimates can be negative and we truncate them to be positive, I wonder if there is any reasonable 'fix' here. That is: This is a pathological situation and indicates you're probably not going to get any sensible results, but won't it be logical to just limit the edf to ensure *some* sort of P-value is created? E.g. what about saying edf = max(edf, rankContrast)... which ensures the denominator EDF is at least one. I mean, *if* you've got denominator EDF of 1 you're basically in a world of pain and won't get anything sensible, right? Or I guess you could also limit it as: edf = max(edf, rankContrast-(1-0.01)) or something, to give it a lower practical limit... but again, with a denominator EDF <1 you'll get an insanely skewed F distribution and probably no worries about inflated significance. What do you think?

[BryanGuillaume]

Hi Tom, I have just tested this on an example and that seems a reasonable solution. From my tests, I only encounter negative scores with the classic SwE because the edf approximation tends to underestimate the true edf (as a quick reminder, that is coming from a small sample bias that decreases quickly when the modified SwE is used).

For the sake of visualising how the p-values would change after clamping the edf estimate to the contrast rank, here below a plot of the -log10(p-values before and after clamping) vs the -log10(p-values of the modified SwE version without clamping). Note tha,t in this example, I have 12 subjects with 33% of missing data and a test of rank 4. Thus, this is a quite challenging test.

Before clamping the edf to rankContrast

After clamping the edf to rankContrast

[nicholst]

OK... so do you think this is bad? We don't know truth here... not sure how to proceed...