Skip to content

Naive approach for IID potential evaluation for score estimators #1508

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 7 commits into
base: main
Choose a base branch
from

Conversation

Kartik-Sama
Copy link
Collaborator

Addresses the issue #1450

Used Effective Sample Score (ESS) to evaluate the log_probs returned by estimated posterior. This way log_probs are being checked both for single observation and iid observations case through the test added via the parameter - iid_batch_size

Copy link

codecov bot commented Mar 20, 2025

Codecov Report

Attention: Patch coverage is 73.07692% with 7 lines in your changes missing coverage. Please review.

Project coverage is 79.31%. Comparing base (b5b4790) to head (f0da7e2).
Report is 9 commits behind head on main.

Files with missing lines Patch % Lines
sbi/inference/potentials/score_based_potential.py 70.00% 6 Missing ⚠️
sbi/inference/posteriors/score_posterior.py 83.33% 1 Missing ⚠️
Additional details and impacted files
@@             Coverage Diff             @@
##             main    #1508       +/-   ##
===========================================
- Coverage   89.70%   79.31%   -10.40%     
===========================================
  Files         122      125        +3     
  Lines        9394     9828      +434     
===========================================
- Hits         8427     7795      -632     
- Misses        967     2033     +1066     
Flag Coverage Δ
unittests 79.31% <73.07%> (-10.40%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines Coverage Δ
sbi/inference/posteriors/score_posterior.py 80.95% <83.33%> (-11.13%) ⬇️
sbi/inference/potentials/score_based_potential.py 81.35% <70.00%> (-15.71%) ⬇️

... and 37 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

from sbi.simulators.linear_gaussian import (
diagonal_linear_gaussian,
linear_gaussian,
true_posterior_linear_gaussian_mvn_prior,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please revert this change. Its unrelated, instead just import in the tests from sbi.simulators.linear_gaussian import ...

)


def _compute_ess(proposal_log_weights: Tensor, true_log_weights: Tensor):
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For these two lines I would prefer not introduce a new function with a big docstring (except if its reused several times and then it should be in sbi.utils.

I would just move the calculations in the test above and add a one sentence inline comment.


@pytest.mark.parametrize("num_dims", [1, 2])
@pytest.mark.parametrize("iid_batch_size", [1, 2])
def test_score_fn_log_prob(num_dims, iid_batch_size):
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This test will be quite expensive now as one would have to retrain on each combination.

Please:

  • move the test to linearGaussian_npse_test.py
  • similar to test_npse_iid_inference use the fixture npse_trained_model (you can skip all the posteriors which are have a uniform prior

This fixture is only trained once and the used by all tests, avoiding this overhead.

Copy link
Contributor

@manuelgloeckler manuelgloeckler left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for implementing looks great.

Just a few changes i.e. moving the tests and using corresponding pytest fixture needs to be done.

Copy link
Contributor

@manuelgloeckler manuelgloeckler left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, great effort. Looks good.

This is done, but I will block merging this for now as it will have conflicts with #1497.

@manuelgloeckler manuelgloeckler added the blocked Something is in the way of fixing this. Refer to it in the issue label Mar 21, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
blocked Something is in the way of fixing this. Refer to it in the issue
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants