Using mcap with MIF2 loglik and parameter traces

[trh3]

Dr. King,

I'm attempting to calculate CIs for parameters that I've estimated using mif2 in pomp, and I'd like to use the MC profile likelihood CIs that are calculated using mcap. However, I don't quite know if I am using mcap correctly, so wanted to verify my understanding. Right now, I'm simply popping in the trace values for logLik and the parameter of interest taken from a mif2 object. This seems like it is working, but I worry about two points:

1. Including logLik + parameter points that are far away from the MLE, (ala using values from iterations close to the start of the mif2 run)
2. That the rest of the parameters are changing over the MIF2 run along side of the parameter of interest.

I'm most troubled by the second point, as that would lead to an additional source of variability in the logLik, but the first point is also a concern simply because I'm not sure what the implications are for fitting the loess or quadratic when one includes points far from the MLE. Or, do all these sources of variability show up in the MC SE estimate?

Thank you!
Teague

[kingaa]

Hello @trh3!

Yes, this is a misunderstanding. MCAP is used to provide profile likelihoods in the presence of Monte Carlo error in the evaluation of the likelihood. To compute a likelihood profile over a given parameter, you'll need to vary that parameter over some range, and them maximize the likelihood over the remaining parameters, keeping your profile parameter fixed. Iterated filtering (IF) is very useful for this purpose. There are some examples in the SBIED Lesson on Iterated Filtering and in the Getting Started vignette as well as elsewhere. It's important to bear in mind that iterated filtering will ultimate return some (hopefully improved) estimated parameter vector. However, one must perform an additional log likelihood estimation at this parameter, for example using pfilter. The difference between the log likelihood as computed by IF and the true log likelihood is discussed in those documents, as well as in FAQ 5.1.

The proper inputs to mcap, then are the values of your profile parameter, each paired with the maximized log likelihood you have estimated. The trace function, applied to output from mif2, returns the values of the parameter vector at each IF iteration. This allows you to see what is happening as IF proceeds, and is thus useful for diagnosing and improving IF performance. The last row in the traces data frame is identical to the parameter estimate returned by mif2 and, if IF2 has converged, may therefore be regarded as a maximum likelihood estimate (MLE). However, the earlier parameter vectors, i.e., the intermediate steps along the way, are in no sense MLEs.

[trh3]

This is extremely helpful! I didn't quite understand that the profile likelihood is formed from the MLEs of other parameters maximized at a set of fixed values of the target parameter. Thank you for clarifying!

Now I am just curious, but from my understanding of the profile likelihood CI calculation from y'alls 2017 Interface paper, the quadratic approximation relies on the LAN assumption. In order to fit this approximation one needs observations of the likelihood surface around the maximum. I understand (at least, beginning to understand) how the log-likelihood from the IF procedure is different from the true log-likelihood, and I realize that looking at points far away from the neighborhood of the MLE gives you no information about the likelihood surface at the maximum, but wouldn't one be able to build up a picture of the likelihood surface by just sampling from the n-dimensional parameter space around the MLE (the area of which can be identified by IF2) and calculating the correct log-likelihood at that collection of points? Then one could use the quadratic approximation using those points to obtain the estimates of the SE and CIs? There is likely a component of profile maximum likelihood that I don't understand that is the reason why one can't do that, but it has piqued my curiosity!

Thanks!
Teague

[kingaa]

Sorry, not familiar with the LAN acronym, but yes, indeed, there are other ways to build up a picture of the likelihood surface. The proposal you describe makes sense, and would not be incorrect, but it doesn't scale particularly well in n. In our 2006 PNAS paper, for example, we use an alternative way of leveraging slices through the parameter space (n of them, along each of the coordinate directions) and the independence of the conditional log likelihoods to construct a quadratic approximation to the likelihood around the maximum.

Quite aside from its usefulness in quantifying uncertainty , the profile likelihood is beneficial in that, by reducing the dimension of the search-space it improves the MLE itself by helping with identifiability problems.

[trh3]

Oh, LAN = Local Asymptotic Normality. And thank you for the explanation, the issue of high dimensionality makes total sense. For context, I'm attempting to get SE's and CI's out of a fairly high dimensional parameter model I'm fitting with MIF2, and I'd like to use the MCMC error corrected profile CIs. Of course, I'm trying to balance the need for a large number of samples from around the likelihood surface with the compute time, so trying to determine efficient ways of obtaining those samples. Thanks for taking the time to chat!

[ionides]

Hi Teague, We have the concept of a "poor man's profile" which is something like what you suggest: you take the estimates from a collection of Monte Carlo searches and produce a profile estimate for each parameter by taking the largest likelihood falling in each bin partitioning the parameter of interest. This scales readily for a large number of parameters, but it is best viewed as a preliminary approximation: if all is well, one should probably compute separate profiles for all parameters of high interest. One can fit a quadratic to the log likelihoods in a more direct application of LAN, similar to Le Cam's one step estimator. For large numbers of parameters, this may be numerically fiddly, especially if there are weakly identified combinations, possibly leading to nonlinear ridges in the likelihood surface. For small numbers of parameters, this is not necessary. So, either way, we usually don't do it. I look forward to hearing what works for you. Best, Ed Edward L. Ionides Associate Chair for Undergraduate Studies and Professor, Department of Statistics, University of Michigan 1085 South University, Ann Arbor, MI 48109-1107 email: ***@***.*** phone: 734 615 3332 office: 453 West Hall

…

On Thu, Feb 9, 2023 at 8:57 AM Teague Henry ***@***.***> wrote: Oh, LAN = Local Asymptotic Normality. And thank you for the explanation, the issue of high dimensionality makes total sense. For context, I'm attempting to get SE's and CI's out of a fairly high dimensional parameter model I'm fitting with MIF2, and I'd like to use the MCMC error corrected profile CIs. Of course, I'm trying to balance the need for a large number of samples from around the likelihood surface with the compute time, so trying to determine efficient ways of obtaining those samples. Thanks for taking the time to chat! — Reply to this email directly, view it on GitHub <#185 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABKAFTZX3CZWIAFUDLBDLADWWTZVTANCNFSM6AAAAAAUTA4GJA> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[trh3]

Hello Dr. Ionides!

I went and implemented the slice likelihood method y'all described in your 2006 paper, and it is working quite the treat! I'm getting coverage around 85-90% for a 95% CI, so a reasonable approximation to the SE. I'm planning on suggesting the use of wider CIs (i.e. 99%) when using these slice likelihood SEs for the models I've been working on. I also have an implementation of profile likelihood CIs, which are of course very computationally intensive! I need to experiment around with them, but I'm sure they'd provide the correct coverage. Finally, I've been looking into the MCMC information matrix technique of Spall, 2005, which looks very amenable to pomp's structure, and likely a bit quicker than profile likelihoods. The only thing I am concerned about is the additional error due to the particle filter when calculating the log-likelihood, but that might be averaged away over the course of the MCMC.

In any case, thanks to both you and Dr. King for being so responsive to questions about pomp, has really been a huge help!

Cheers,
Teague

[ionides]

Hi Teague, Thanks for sharing your experiences with the 2006 method. The main MCMC method in pomp is particle MCMC, implemented as pmcmc(), and abc() is also possible. Best, Ed