Reimplement ATLAS_WPWM_7TEV_46FB #2202

ecole41 · 2024-11-08T11:18:32Z

This pull request introduces a new filtering module for the ATLAS WPMP 7TeV 47FB dataset, along with several supporting utility functions and updates to various data files. The most important changes include the addition of filtering functions, utility functions for data extraction, and updates to metadata and raw data files.

Old vs New Data Comparison

https://vp.nnpdf.science/GCCc3EPrTr-G7SgQBTeTbA==/

Compatibility Checks

from validphys.api import API
import numpy as np
 
inp1 = {"dataset_input": {"dataset": "ATLASWRAP11CC"}, "theoryid": 40_000_000, "use_cuts": "internal", "t0pdfset": "NNPDF40_nnlo_as_01180", "use_t0": True}
inp2 = {"dataset_input": {"dataset": "ATLASWRAP11CC", "variant": "legacy"}, "theoryid": 40_000_000, "use_cuts": "internal", "t0pdfset": "NNPDF40_nnlo_as_01180", "use_t0": True}
 
covmat1 = API.covmat_from_systematics(**inp1)
covmat2 = API.covmat_from_systematics(**inp2)
 
t0_covmat1 = API.t0_covmat_from_systematics(**inp1)
t0_covmat2 = API.t0_covmat_from_systematics(**inp2)

result = np.all(np.isclose(covmat1, covmat2))
result_2 = np.all(np.isclose(t0_covmat1, t0_covmat2))

print('covmat', result)
print('t0_covmat', result_2)

Out:
LHAPDF 6.5.4 loading [/Users/ellacole/miniconda3/envs/nnpdf_dev/share/LHAPDF/NNPDF40_nnlo_as_01180/NNPDF40_nnlo_as_01180_0000.dat](https://file+.vscode-resource.vscode-cdn.net/Users/ellacole/miniconda3/envs/nnpdf_dev/share/LHAPDF/NNPDF40_nnlo_as_01180/NNPDF40_nnlo_as_01180_0000.dat)
NNPDF40_nnlo_as_01180 PDF set, member #0, version 1; LHAPDF ID = 331100
covmat True
t0_covmat True

nnpdf_data/nnpdf_data/commondata/ATLAS_WPWM_7TEV_46FB/filter.py

ecole41 · 2024-11-09T10:15:01Z

Compatability Checks:

from validphys.api import API
import numpy as np
import os

def check_theory_exists(theory_id):
    theory_path = f"/Users/ellacole/miniconda3/envs/nnpdf_dev/share/NNPDF/theories/theory_{theory_id}"
    return os.path.exists(theory_path)

theory_id = 708 


if check_theory_exists(theory_id):
    inp1 = {"dataset_input": {"dataset": "ATLASWRAP11CC"}, "theoryid": theory_id, "use_cuts": "internal"}
    inp2 = {"dataset_input": {"dataset": "ATLASWRAP11CC", "variant": "legacy"}, "theoryid": theory_id, "use_cuts": "internal"}
    covmat1 = API.covmat_from_systematics(**inp1)
    covmat2 = API.covmat_from_systematics(**inp2)
    
    result = np.isclose(covmat1, covmat2)
    print(result)
else:
    print(f"Theory {theory_id} not found. ")

Out:
[[ True  True  True  True  True  True  True  True  True  True  True  True
   True  True  True  True  True  True  True  True  True  True]
 [ True  True  True  True  True  True  True  True  True  True  True  True
   True  True  True  True  True  True  True  True  True  True]
 [ True  True  True  True  True  True  True  True  True  True  True  True
   True  True  True  True  True  True  True  True  True  True]
 [ True  True  True  True  True  True  True  True  True  True  True  True
   True  True  True  True  True  True  True  True  True  True]
 [ True  True  True  True  True  True  True  True  True  True  True  True
   True  True  True  True  True  True  True  True  True  True]
 [ True  True  True  True  True  True  True  True  True  True  True  True
   True  True  True  True  True  True  True  True  True  True]
 [ True  True  True  True  True  True  True  True  True  True  True  True
   True  True  True  True  True  True  True  True  True  True]
 [ True  True  True  True  True  True  True  True  True  True  True  True
   True  True  True  True  True  True  True  True  True  True]
 [ True  True  True  True  True  True  True  True  True  True  True  True
   True  True  True  True  True  True  True  True  True  True]
 [ True  True  True  True  True  True  True  True  True  True  True  True
   True  True  True  True  True  True  True  True  True  True]
 [ True  True  True  True  True  True  True  True  True  True  True  True
   True  True  True  True  True  True  True  True  True  True]
 [ True  True  True  True  True  True  True  True  True  True  True  True
   True  True  True  True  True  True  True  True  True  True]
 [ True  True  True  True  True  True  True  True  True  True  True  True
   True  True  True  True  True  True  True  True  True  True]
 [ True  True  True  True  True  True  True  True  True  True  True  True
   True  True  True  True  True  True  True  True  True  True]
 [ True  True  True  True  True  True  True  True  True  True  True  True
   True  True  True  True  True  True  True  True  True  True]
 [ True  True  True  True  True  True  True  True  True  True  True  True
   True  True  True  True  True  True  True  True  True  True]
 [ True  True  True  True  True  True  True  True  True  True  True  True
   True  True  True  True  True  True  True  True  True  True]
 [ True  True  True  True  True  True  True  True  True  True  True  True
   True  True  True  True  True  True  True  True  True  True]
 [ True  True  True  True  True  True  True  True  True  True  True  True
   True  True  True  True  True  True  True  True  True  True]
 [ True  True  True  True  True  True  True  True  True  True  True  True
   True  True  True  True  True  True  True  True  True  True]
 [ True  True  True  True  True  True  True  True  True  True  True  True
   True  True  True  True  True  True  True  True  True  True]
 [ True  True  True  True  True  True  True  True  True  True  True  True
   True  True  True  True  True  True  True  True  True  True]]

comane · 2024-11-09T10:24:12Z

Compatability Checks:

from validphys.api import API
import numpy as np
import os

def check_theory_exists(theory_id):
    theory_path = f"/Users/ellacole/miniconda3/envs/nnpdf_dev/share/NNPDF/theories/theory_{theory_id}"
    return os.path.exists(theory_path)

theory_id = 708 


if check_theory_exists(theory_id):
    inp1 = {"dataset_input": {"dataset": "ATLASWRAP11CC"}, "theoryid": theory_id, "use_cuts": "internal"}
    inp2 = {"dataset_input": {"dataset": "ATLASWRAP11CC", "variant": "legacy"}, "theoryid": theory_id, "use_cuts": "internal"}
    covmat1 = API.covmat_from_systematics(**inp1)
    covmat2 = API.covmat_from_systematics(**inp2)
    
    result = np.isclose(covmat1, covmat2)
    print(result)
else:
    print(f"Theory {theory_id} not found. ")

Out:
[[ True  True  True  True  True  True  True  True  True  True  True  True
   True  True  True  True  True  True  True  True  True  True]
 [ True  True  True  True  True  True  True  True  True  True  True  True
   True  True  True  True  True  True  True  True  True  True]
 [ True  True  True  True  True  True  True  True  True  True  True  True
   True  True  True  True  True  True  True  True  True  True]
 [ True  True  True  True  True  True  True  True  True  True  True  True
   True  True  True  True  True  True  True  True  True  True]
 [ True  True  True  True  True  True  True  True  True  True  True  True
   True  True  True  True  True  True  True  True  True  True]
 [ True  True  True  True  True  True  True  True  True  True  True  True
   True  True  True  True  True  True  True  True  True  True]
 [ True  True  True  True  True  True  True  True  True  True  True  True
   True  True  True  True  True  True  True  True  True  True]
 [ True  True  True  True  True  True  True  True  True  True  True  True
   True  True  True  True  True  True  True  True  True  True]
 [ True  True  True  True  True  True  True  True  True  True  True  True
   True  True  True  True  True  True  True  True  True  True]
 [ True  True  True  True  True  True  True  True  True  True  True  True
   True  True  True  True  True  True  True  True  True  True]
 [ True  True  True  True  True  True  True  True  True  True  True  True
   True  True  True  True  True  True  True  True  True  True]
 [ True  True  True  True  True  True  True  True  True  True  True  True
   True  True  True  True  True  True  True  True  True  True]
 [ True  True  True  True  True  True  True  True  True  True  True  True
   True  True  True  True  True  True  True  True  True  True]
 [ True  True  True  True  True  True  True  True  True  True  True  True
   True  True  True  True  True  True  True  True  True  True]
 [ True  True  True  True  True  True  True  True  True  True  True  True
   True  True  True  True  True  True  True  True  True  True]
 [ True  True  True  True  True  True  True  True  True  True  True  True
   True  True  True  True  True  True  True  True  True  True]
 [ True  True  True  True  True  True  True  True  True  True  True  True
   True  True  True  True  True  True  True  True  True  True]
 [ True  True  True  True  True  True  True  True  True  True  True  True
   True  True  True  True  True  True  True  True  True  True]
 [ True  True  True  True  True  True  True  True  True  True  True  True
   True  True  True  True  True  True  True  True  True  True]
 [ True  True  True  True  True  True  True  True  True  True  True  True
   True  True  True  True  True  True  True  True  True  True]
 [ True  True  True  True  True  True  True  True  True  True  True  True
   True  True  True  True  True  True  True  True  True  True]
 [ True  True  True  True  True  True  True  True  True  True  True  True
   True  True  True  True  True  True  True  True  True  True]]

Hi @ecole41, thanks this is very nice!
Can you please include a t0-covariance matrix check too?
This can be done as it is done in the code below

from validphys.api import API
import numpy as np
 
inp1 = {"dataset_input": {"dataset": f"{new_implementation}"}, "theoryid": 40_000_000, "use_cuts": "internal", "t0pdfset": "NNPDF40_nnlo_as_01180", "use_t0": True}
inp2 = {"dataset_input": {"dataset": f"{old_implementation}", "variant": "legacy"}, "theoryid": 40_000_000, "use_cuts": "internal", "t0pdfset": "NNPDF40_nnlo_as_01180", "use_t0": True}
 
covmat1 = API.covmat_from_systematics(**inp1)
covmat2 = API.covmat_from_systematics(**inp2)
 
t0_covmat1 = API.t0_covmat_from_systematics(**inp1)
t0_covmat2 = API.t0_covmat_from_systematics(**inp2)
 
np.all(np.isclose(covmat1, covmat2))
np.all(np.isclose(t0_covmat1, t0_covmat2))

nnpdf_data/nnpdf_data/commondata/ATLAS_WPWM_7TEV_46FB/filter.py

ecole41 · 2024-11-13T08:53:59Z

covariance

from validphys.api import API
import numpy as np
 
inp1 = {"dataset_input": {"dataset": "ATLASWRAP11CC"}, "theoryid": 40_000_000, "use_cuts": "internal", "t0pdfset": "NNPDF40_nnlo_as_01180", "use_t0": True}
inp2 = {"dataset_input": {"dataset": "ATLASWRAP11CC", "variant": "legacy"}, "theoryid": 40_000_000, "use_cuts": "internal", "t0pdfset": "NNPDF40_nnlo_as_01180", "use_t0": True}
 
covmat1 = API.covmat_from_systematics(**inp1)
covmat2 = API.covmat_from_systematics(**inp2)
 
t0_covmat1 = API.t0_covmat_from_systematics(**inp1)
t0_covmat2 = API.t0_covmat_from_systematics(**inp2)

result = np.all(np.isclose(covmat1, covmat2))
result_2 = np.all(np.isclose(t0_covmat1, t0_covmat2))

print('covmat', result)
print('t0_covmat', result_2)

Out:
LHAPDF 6.5.4 loading [/Users/ellacole/miniconda3/envs/nnpdf_dev/share/LHAPDF/NNPDF40_nnlo_as_01180/NNPDF40_nnlo_as_01180_0000.dat](https://file+.vscode-resource.vscode-cdn.net/Users/ellacole/miniconda3/envs/nnpdf_dev/share/LHAPDF/NNPDF40_nnlo_as_01180/NNPDF40_nnlo_as_01180_0000.dat)
NNPDF40_nnlo_as_01180 PDF set, member #0, version 1; LHAPDF ID = 331100
covmat True
t0_covmat True

nnpdf_data/nnpdf_data/commondata/ATLAS_WPWM_7TEV_46FB/metadata.yaml

…r_utils

scarlehoff

This seems ok and ready to merge up to the UNCORR.1 bug, however you have to fix the conflicts with master.

In this case, what you added to commondataparser.py and process_options.py was already added to master in the meantime I believe so the easiest thing would be to rebase by picking up only the changes to the ATLAS_WMWP... folder

validphys2/src/validphys/process_options.py

nnpdf_data/nnpdf_data/commondata/ATLAS_WPWM_7TEV_46FB/uncertainties.yaml

nnpdf_data/nnpdf_data/commondata/ATLAS_WPWM_7TEV_46FB/filter_utils.py

scarlehoff · 2024-12-06T12:05:52Z

The failing tests is because of changes in reportengine. If you merge from master and solve the conflicts (as I say above, the easiest thing is to absorb directly the stuff from master) it should be fine.

ecole41 · 2024-12-06T13:41:54Z

validphys2/src/validphys/commondataparser.py

    "SINGLETOP": ("$y$", "$m_t^2 (GeV^2)$", "$\\sqrt{s} (GeV)$"),
    "DY_MLL": ("$M_{ll} (GeV)$", "$M_{ll}^2 (GeV^2)$", "$\\sqrt{s} (GeV)$"),
+    "DY_W_ETA": ("$abs_\\eta$", "$M_W^2 (GeV^2)$", "$\\sqrt{s} (GeV)$"),


I am not sure if this is a correct edit, the first kinematic is abs_eta for this specific dataset but I am not sure for other DY_W_ETA processes

This whole thing will go away soon as soon as all datasets are ported since the labels are taken from the metadata, so it's not important.

ecole41 added 2 commits November 8, 2024 11:08

Added raw data

59e04bd

Added data central and kinematics to filter.py

7bba9fb

ecole41 added data toolchain ATLAS_DY_DATA labels Nov 8, 2024

ecole41 requested review from scarlehoff and comane November 8, 2024 11:18

ecole41 changed the title ~~Reimplement ATLAS_WPWM_7TEV_46FB~~ [WIP] Reimplement ATLAS_WPWM_7TEV_46FB Nov 8, 2024

scarlehoff reviewed Nov 8, 2024

View reviewed changes

nnpdf_data/nnpdf_data/commondata/ATLAS_WPWM_7TEV_46FB/filter.py Show resolved Hide resolved

ecole41 added 3 commits November 8, 2024 12:39

Added uncertainty data and added function to produce uncertainties.yaml

8b3607c

Prettify floats

3128820

Added new HEP link to metadata

a8ac03f

Prettify floats

c3a9684

comane reviewed Nov 9, 2024

View reviewed changes

nnpdf_data/nnpdf_data/commondata/ATLAS_WPWM_7TEV_46FB/filter.py Outdated Show resolved Hide resolved

ecole41 added 2 commits November 13, 2024 08:57

Added built in float prettify

e3ca58c

Edited metadata - need to check this

c0a854c

ecole41 commented Nov 21, 2024

View reviewed changes

nnpdf_data/nnpdf_data/commondata/ATLAS_WPWM_7TEV_46FB/metadata.yaml Outdated Show resolved Hide resolved

ecole41 commented Nov 21, 2024

View reviewed changes

nnpdf_data/nnpdf_data/commondata/ATLAS_WPWM_7TEV_46FB/metadata.yaml Outdated Show resolved Hide resolved

ecole41 added 2 commits November 22, 2024 09:53

Edited process_type in metadata

ed92a92

Edited metadat and validphys scripts for plotting, fixed bug in filte…

34c6c03

…r_utils

ecole41 mentioned this pull request Dec 5, 2024

[WIP] Reimplement ATLAS Z0 7TEV 46FB Dataset #2237

Open

scarlehoff reviewed Dec 5, 2024

View reviewed changes

validphys2/src/validphys/process_options.py Outdated Show resolved Hide resolved

nnpdf_data/nnpdf_data/commondata/ATLAS_WPWM_7TEV_46FB/uncertainties.yaml Outdated Show resolved Hide resolved

nnpdf_data/nnpdf_data/commondata/ATLAS_WPWM_7TEV_46FB/filter_utils.py Show resolved Hide resolved

scarlehoff changed the title ~~[WIP] Reimplement ATLAS_WPWM_7TEV_46FB~~ Reimplement ATLAS_WPWM_7TEV_46FB Dec 5, 2024

ecole41 added 3 commits December 6, 2024 09:25

Added eta back as an accepted variable

82428f9

Fixed error with UNCORR title

0666b8e

Updated uncertainties

1f2b922

Merge from master and resolve conflicts

7c8577a

ecole41 commented Dec 6, 2024

View reviewed changes

scarlehoff approved these changes Dec 6, 2024

View reviewed changes

scarlehoff merged commit 92f9fdb into master Dec 6, 2024
6 checks passed

scarlehoff deleted the reimplement_ATLAS_WPWM_7TEV_46FB branch December 6, 2024 15:29

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Reimplement ATLAS_WPWM_7TEV_46FB #2202

Reimplement ATLAS_WPWM_7TEV_46FB #2202

ecole41 commented Nov 8, 2024 •

edited

Loading

ecole41 commented Nov 9, 2024

comane commented Nov 9, 2024

ecole41 commented Nov 13, 2024

scarlehoff left a comment

scarlehoff commented Dec 6, 2024

ecole41 Dec 6, 2024

scarlehoff Dec 6, 2024

Reimplement ATLAS_WPWM_7TEV_46FB #2202

Reimplement ATLAS_WPWM_7TEV_46FB #2202

Conversation

ecole41 commented Nov 8, 2024 • edited Loading

Old vs New Data Comparison

Compatibility Checks

ecole41 commented Nov 9, 2024

comane commented Nov 9, 2024

ecole41 commented Nov 13, 2024

scarlehoff left a comment

Choose a reason for hiding this comment

scarlehoff commented Dec 6, 2024

ecole41 Dec 6, 2024

Choose a reason for hiding this comment

scarlehoff Dec 6, 2024

Choose a reason for hiding this comment

ecole41 commented Nov 8, 2024 •

edited

Loading