Skip to content

Default to r_eff=FALSE for loo() method #1091

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 4 commits into
base: master
Choose a base branch
from
Open

Default to r_eff=FALSE for loo() method #1091

wants to merge 4 commits into from

Conversation

jgabry
Copy link
Member

@jgabry jgabry commented May 22, 2025

Submission Checklist

  • Run unit tests
  • Declare copyright holder and agree to license (see below)

Summary

Changes default from TRUE to FALSE for the r_eff argument to the loo() method. This is based on @avehtari's comment at https://discourse.mc-stan.org/t/loo-subsample-slower-than-loo/39556/5

Copyright and Licensing

Please list the copyright holder for the work you are submitting
(this will be you or your assignee, such as a university or company):
Columbia University

By submitting this pull request, the copyright holder is agreeing to
license the submitted work under the following licenses:

@jgabry jgabry requested a review from avehtari May 22, 2025 19:58
@jgabry
Copy link
Member Author

jgabry commented May 22, 2025

@avehtari There's not much to review here. I requested your review just to confirm that you definitely want to do this and to make sure the documentation is ok. The code change is just TRUE -> FALSE.

@avehtari
Copy link
Member

I added one suggestion.

Usually, people check reasonable convergence and ESS before running loo() anyway, so in most cases making it FALSE should be safe, and we can add a comment about this somewhere. In the early days I assumed that MCSE might be bigger, but it most often is small if the sample size is reasonable and all k<0.7.

@jgabry
Copy link
Member Author

jgabry commented May 27, 2025

This is passing everything now, but wanted to check with @andrjohns before merging this: if we're changing a default argument would you prefer that I add a warning or message if the user calls the loo method without changing the new default? On the one hand, I guess this could be considered a breaking change (although wouldn't lead to any errors it changes default behavior). On the other hand, adding a warning/message would result in a warning or message most of the times loo() is called, which is kind of annoying. And this is a pretty small change and it's documented and we can put it in the release notes. I'm a bit torn on whether we should add a warning/message. @andrjohns any preference?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants