response_to_referee.txt

Dear Frank:

Please find enclosed our revised version of MS#AAS07769, Data analysis
recipes: Using Markov Chain Monte Carlo by Hogg & Foreman-Mackey.

In what follows, the referee comments are indented and our responses
are not.

Thank you,

David


     ---------------------------------------------------------------------- 
     Referee Report 

     The authors provide a simple, and somehow anecdotal discussion
     about MCMC methods (for astronomers, although the title don't
     explicitly says it) in an informal and pragmatic style.

We figured that because this is in the AAS journals, that would supply
the relevant context, though we wouldn't object to modifying the title
slightly. We have not made a change here.

     The concept of conveying an intuitive description of statistical
     methods for the broad astronomical community is certainly
     something I support, however I would like to ask the authors and
     editor to consider some caveats below.

     Novelty and references

     The draft lacks of some important literature contextualization. A
     critical one is a recent Annual Review of Astronomy and
     Astrophysics entitled Markov Chain Monte Carlo Methods for
     Bayesian Data Analysis in Astronomy
     https://arxiv.org/abs/1706.01629

Yes, we know this paper. We now cite it near the end of the first
Section, and near the beginning of Section 10. We note in both places
that the Sharma review goes into much more depth on methods than we
do; our goals here are far more introductory.

     I would like the authors to please explain how their review
     brings something new to the literature, considering the amount of
     good and available material about MCMC.

Good point. We are slightly more explicit about this now in the
introductory section, in the paragraph starting "In what
follows...". The idea is to provide a getting-started user-manual for
MCMC conceptually or specifically, but not really review new
developments or sophisticated methods. It is more like we are trying
to specifically emphasize the importance of basic good pratices; we
say that more clearly now.

     Such as e.g.

     Doing Bayesian Data Analysis, Second Edition: A Tutorial with R,
     JAGS, and Stan, by John Kruschke, and many others

     A potential reason is to update the readers about recent
     developments in the field.  But again, the authors seems to
     ignore the current astronomical literature. For example, the
     authors are aware of Statistical Bayesian packages such as JAGS
     and Stan, but do not cite two recent books in Astronomy that make
     use of them, such as:

     Bayesian Methods for the Physical Sciences: Learning from
     Examples in Astronomy and Physics, Springer Series in
     Astrostatistics, 2015 Andreon, and Weaver

     And 

     Bayesian Models for Astrophysical Data: Using R, JAGS, Python,
     and Stan, Cambridge University press, Hilbe, de Souza and Ishida,
     2017

Good point; these are all now cited in the same "In what follows..."
paragraph, where we explain our difference from the rest of the
literature.

     2. Confidence versus credible intervals 

     I am sure the authors are aware of the difference between both,
     and should know they don't even agree beyond simple cases: such
     as for normally distributed data. Hence the quote "Because they
     are so afraid of being confused with frequentists, Bayesians
     often call these regions "credible" rather than "confidence""

     Not only seems inappropriate for a scientific paper, but it is
     actually misleading. See for instance The fallacy of placing
     confidence in confidence intervals, Morey et al. 2015
     https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4742505/

     Moreover, such discussion -- Bayesian vs Frequentist -- has far
     deeper roots. See e.g. Probability theory, the logic of science,
     E.T. Jaynes, Cambridge University Press 2003.

Actually, our problem is not with the differences -- they are huge and
real -- it is with the *names*. We agree with the referee that these
two things are importantly and substantially different!  They have
totally different meanings, of course. We have fixed that all up in
the text now, making it clear that these are not the same thing at
ALL. We changed both the footnote (and extended it to make the point)
and also the text that is so footnoted.

     3. Lack of discussion about prediction vs credible intervals. 

We interpreted this telegraphic comment to mean: Shouldn't the paper
say something about the fact that you might want to put a
probabilistic interval on a prediction for new data (like a new
observation of a binary star's velocity, given past observations).  We
put in a mention in the Model Checking subsection. It is certainly
deserving of more attention than it gets here.

     4. Model check 

     Lack of contemporary discussion such as e.g. Gabry et
     al. Visualization in Bayesian workflow,
     https://arxiv.org/abs/1709.01449

Ooh that's a great reference! We cite this in the relevant place now,
and were pleased to be pointed at this. Indeed, Gelman has been a
great public voice in support of taking this broader view of what
Bayesian inference is.

     5. MORE SOPHISTICATED SAMPLING METHODS 

     Please consider adding some material about 

     Approximate Bayesian Computation, since it has gained space in astronomy e.g. 

     https://arxiv.org/abs/1608.07606
     https://arxiv.org/abs/1607.01782 
     https://arxiv.org/abs/1504.06129 
     https://arxiv.org/abs/1202.1426

While we agree that the ABC developments are exciting -- and some of
them are being worked on by us!! -- we do consider this out of
scope. As we now better explain in the paper, we aren't trying to
cover the whole ground of inference, we are just trying to introduce
the uninitiated to the method and help them get over the initial
stumbling blocks. So we have not made a change here.

     ---------------------------------------------------------------------- 
     Statistics Editor Comments:

     As a didactic presentation, I suggest two additions:

     (a) Add a table listing MCMC approaches so the reader appreciates
     the variety of options available
     (e.g. https://www.rdocumentation.org/packages/LaplacesDemon/versions/16.0.1/topics/LaplacesDemon)

We explicitly consider this out of scope, as we have said both early in
the paper, and in the Advanced-Methods section. We call out to much more
comprehensive reviews of methods, which have these tables and full
descriptions of the methods.

     (b) Discuss how to choose when MCMC sampling and when
     optimization (e.g. EM Algorithm or simplex) is appropriate. The
     former is commonly used for characterizing Bayesian posteriors
     while the latter are commonly used for maximizing likelihoods.

We have added more detail to our optimization comment in the introduction.