Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Calculate P-Value via ToyMC #2

Merged
merged 3 commits into from
May 7, 2021
Merged

Calculate P-Value via ToyMC #2

merged 3 commits into from
May 7, 2021

Conversation

JosephJHowlett
Copy link
Contributor

I'm in the process of incorporating this package into bbf, and realized my use case might be somewhat common. I have a histogrammed model (ER template) and histogrammed data (Rn220 data), and want the p-value of the fit. With this PR you can do (if e.g. these are both multihists):

from GOFevaluation import binned_poisson_chi2_gof
gof_object = binned_poisson_chi2_gof.from_binned(datahist.histogram, mchist.histogram)
gof_object.get_pvalue(n_mc=1000)

If you want to plot the distribution of GoF's, you can do:

gofs = gof_object.sample_gofs()
hist, bin_edges = np.histogram(gofs, bins=1000)
cumulative_density = 1.0 - np.cumsum(hist)/np.sum(hist)
plt.plot(np.mean(np.vstack([bin_edges[0:-1], bin_edges[1:]]), axis=0), cumulative_density)
plt.axvline(gof_object.calculate_gof())
plt.ylabel('P(less than GoF)')
plt.xlabel('GoF')
plt.show()

to get:
pva

@JosephJHowlett JosephJHowlett requested review from WolfXeHD and kdund May 6, 2021 20:19
@WolfXeHD WolfXeHD added the enhancement New feature or request label May 7, 2021
@WolfXeHD WolfXeHD requested a review from hammannr May 7, 2021 09:09
@WolfXeHD
Copy link
Contributor

WolfXeHD commented May 7, 2021

Hi @jhowl01 !
It is great to see that you are such an early adopter and try to incorporate the package in bbf.
From a first quick look, this seems to be a great and useful feature. I also added @hammannr as a reviewer since he is also actively developing the package at the moment. I think he might also have some ideas about what to do and how to incorporate your idea.

hammannr
hammannr reviewed May 7, 2021
View reviewed changes
Copy link
Collaborator

@hammannr hammannr left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this is a really cool and handy feature to have in the package so that one can directly get a proper understanding of how well the fit performed and for comparing different gof measures! If we add get_pvalue() to all gof tests we might want to outsource the calculations into a dedicated class.

@kdund kdund merged commit f6f2af3 into master May 7, 2021
@hammannr hammannr mentioned this pull request May 10, 2021
Open
5 tasks
@hammannr hammannr linked an issue May 10, 2021 that may be closed by this pull request
Adding p-value calculation #9
Open
5 tasks
@hammannr hammannr deleted the pvalue_toymc branch May 17, 2021 07:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@hammannr hammannr hammannr approved these changes

@WolfXeHD WolfXeHD Awaiting requested review from WolfXeHD

@kdund kdund Awaiting requested review from kdund

Assignees
No one assigned
Labels
enhancement New feature or request
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Adding p-value calculation
4 participants
@JosephJHowlett @WolfXeHD @kdund @hammannr