WIP: Adding FCC-hh v05 production code

FCChh_Instructions/Readme.md

@@ -0,0 +1,69 @@
+# Specific instructions for FCC-hh sample generation
+
+## Autumn 2022 (updated Spring 2023) 
+
+### Prerequisites & setting up 
+
+To be able to use this framework, you need access rights to the dedicated `eos`-area. Check if you can access this path `/eos/experiment/fcc/hh/generation/` - if not, you need to request access (*to clarify who can grant access and what the procedure should be*). 
+
+For running on condor (which currently seems to be the only viable option even for a tester, as running locally doesn't seem to be supported) it is also helpful to be a member of the `group_u_FCC.local_gen` group (*how to be added?*). 
+
+Some fixes to the FCChh specific code needed to be implemented that are currently only available in this fork and not in the main repo, so to run the FCChh tester please clone this fork: 
+
+``` git clone [email protected]:bistapf/EventProducer.git ```
+
+and switch to the specific branch:
+
+ ``` git checkout fcchh_evtgen_updates ``` 
+
+ Once done, `source init.sh` for setting up the environment. 
+
+### Producing edm4hep events from existing LHE files with `DelphesPythia8_EDM4HEP`
+
+You can see which LHE events are available in this `\eos\` area: `/eos/experiment/fcc/hh/generation/lhe/`. To run existing LHE events through Pythia+Delphes and store the output in edm4hep format, follow these steps: 
+
+0. If you have never used this framework to generate events before, you must add yourself to the list of users in `config/users.py`.
+
+1. Make sure the that `config/param_FCChh.py` is correctly setup for the process you want to run, i.e. : The version of the production must be listed in the `prodTag` directory, specifying how the software stack is setup. For the Spring 23 production, we will use `fcc_v05_scenarioI`. 
+Then, you must be sure that in `/eos/experiment/fcc/hh/utils/delphescards/` a directory with the name of your version exists, which contains the Delphes card you want to use and all parametrization files it uses - their names are given in the `delphescard_base, delphescard_mr, delphescard_mmr, delphescard_emr` variables in the config file. Again, for `fcc_v05_scenarioI` this is already setup using the [scenario I Delphes card](https://github.com/bistapf/delphes/blob/master/cards/FCC/scenarios/FCChh_I.tcl).  
+Last `pythialist` or `decaylist` must contain the name of your process/LHE file (*to be clarified which is the correct approach*). For example, for the di-Higgs signals (with k_l = 1) the name of LHE is `pw_pp_hh_lambda100_5f` - the Higgses are not decayed there yet. This step is done in Pythia, with DelphesPythia8_EDM4HEP, using the cards from `/eos/experiment/fcc/hh/utils/pythiacards/`. Currently, cards for bbaa, bbtata, bbZZ (inc.), bbZZ (Z-leptonic = 4l, llvv + 4v), bbZZ (4l), bbWW (inc.) and 4b exist and are ready to be used. If you want to add new decay, place the Pythia card in the eos area and make sure to add is name to the `decaylist`. 
+
+2. Finally, jobs to produce events using `DelphesPythia8_EDM4HEP` are submitted by calling: 
+```python bin/run.py --FCChh --reco --send  -p <LHE_process_name> --type lhep8 --decay <decay_name> --pycard <pythia_card> -N <number_of_jobs> --condor -q <queue_name> --detector "" --prodtag <version>```. 
+Here, `number_of_jobs` determines how many events we produce - 1 job processes one LHE file, which commonly has 1k events. So 100 jobs will produce 1 million events. 
+For example, to produce 500k pp->HH->4b events from the existing HH LHE, we run: 
+```python bin/run.py --FCChh --reco --send  -p pw_pp_hh_lambda100_5f --type lhep8 --decay hhbbbb --pycard pwp8_pp_hh_5f_hhbbbb.cmd -N 50 --condor  -q workday --detector "" --prodtag fcc_v05_scenarioI```
+Optionally we can add: --customEDM4HEPOutput <path_to_config_file>` to this command, in case we want to store more collections in the edm4hep output than in the common setup. For example, the electrons/muons/photons before/after isolation and overlap removal. A file to use for this setup comes with this repo, it's called `edm4hep_output_config.tcl` (in the top level). 
+
+3. Monitor the jobs on condor. Their logs will be written into `EventProducer/BatchOutputs`. 
+
+4. If all works well (and you didn't change it in the config file), the output files will be written to `/eos/experiment/fcc/hh/generation/DelphesEvents/<version>/<process_name>`. 
+
+*To-Do: Instructions for merging files, managing and cleaning up, fixing the database* 
+
+### How to run LHE production with the new HH signal gridpacks 
+
+If you have a gridpack for the event generation, you can produce LHE events from it in the following steps: 
+
+0. If you have never used this framework to generate events before, you must add yourself to the list of users in `config/users.py`. (*why is this necessary and are there any restrictions here?*)
+
+1. Copy your gridpack to the FCChh gridpack eos directory `/eos/experiment/fcc/hh/generation/gridpacks/`. For this example we will use the tester gridpack called `mg_pp_vbf_hh_angela.tar.gz`, which is for VBF HH signal with C2V = 2 (and CV = kl = 1 ?). 
+
+2. Add the name of your gridpack to the `gridpacklist` in `config/param_FCChh.py`. For `mg_pp_vbf_hh_angela` this is already done, if you are working on the `fcchh_evtgen_updates` branch. 
+
+3. Run the event generation from your gridpack with the following command:
+
+``` python bin/run.py --FCChh --LHE --send --condor --typelhe gp_mg -p mg_pp_vbf_hh_angela --prodtag fcc_v04 -n 10 -N 1 -q longlunch --detector "" ``` 
+
+This will submit 1 job to generate 10 events from the testergridpack to condor. The logs of this job will be found in `./BatchOutputs/FCC/lhe/mg_pp_vbf_hh_angela` and the output Les Houches files will be in `/eos/experiment/fcc/hh/generation/lhe/mg_pp_vbf_hh_angela`. 
+
+### To-do before proper production:
+- Build and copy to the `eos`-area all the gridpacks for the new signals *(maybe add instructions on how to do this somewhere here too?)*
+- Give correct names to the gridpacks (i.e. the full process names)
+- Add the gridpacks to the param_FCChh properly (i.e. with description, cross-section, etc?)
+- The arguments `prodtag` and `detector` are currently placeholders as they are required from the FCC-ee production. Should be adapted for FCC-hh production. 
+- Fix/update the FCChh LHE database
+
+
+#testing setup for delphesizing:
+DelphesPythia8_EDM4HEP /eos/user/b/bistapf/new_FCChh_cards/FCChh_I.tcl /afs/cern.ch/work/h/helsens/public/k4SimDelphes_PythiaStuff/examples/edm4hep_output_config.tcl ./pythia_config/tester_pwp8_pp_hh_5f_hhbbww.cmd /eos/user/b/bistapf/FCChh_EvtGen/pwp8_pp_hh_5f_hhbbWW_tester_new_card.root

bin/run.py

@@ -37,6 +37,8 @@
     sendjobGroup.add_argument('--priority', type=str, default='group_u_FCC.local_gen', help='condor queue priority (default: group_u_FCC.local_gen)')
     sendjobGroup.add_argument('--ncpus', type=str, default='1', help='number of CPUs (1CPU=2Gb of RAM)')
 
+
+
 #41873
 
     ###################
@@ -52,7 +54,8 @@
     sendjobGroup.add_argument('-n','--numEvents', type=int, help='Number of simulation events per job', default=10000)
     sendjobGroup.add_argument('-N','--numJobs', type=int, default = 10, help='Number of jobs to submit')
 
-
+    #option for running FCC-hh Delphes card validation, using an adapted edm4hep_output config, should only work with option reco. 
+    sendjobGroup.add_argument('--customEDM4HEPOutput', type=str, default="", help="Use a custom edm4hep output config card, by providing the path.")
 
     mgGroup = parser.add_argument_group('mggroup')
     mgGroup.add_argument("--mg5card", type=str, help="MG5 configuration", default='card.mg5')
@@ -73,9 +76,12 @@
     args, _ = parser.parse_known_args()
     sendOpt = args.type
 
+    if args.customEDM4HEPOutput and not args.reco and not args.type=="lhep8":
+        parser.error("Option --customEDM4HEPOutput only works for producing edm4hep output, so if -reco and --type lhep8 options are set.")
+
     if args.FCChh:
         import EventProducer.config.param_FCChh as para
-        print ('import FCC-hh config')
+        print ('import base FCC-hh config')
     elif args.FCCee:
         import EventProducer.config.param_FCCee as para
         print ('import FCC-ee config')
@@ -88,6 +94,11 @@
     prodTagGroup.add_argument('--prodtag', type=str, required = '--reco' in sys.argv, help='Version to use', choices = prodTag)
     prodTagGroup.add_argument('--detector', type=str, default='', required = '--reco' in sys.argv, help='Detector to use', choices = para.detectors)
 
+    args, _ = parser.parse_known_args()
+
+    if args.FCChh and args.prodtag and "scenarioII" in args.prodtag:
+        import EventProducer.config.param_FCChh_scenarioII as para
+        print ('overwrite base FCC-hh config with specific FCC-hh scenario II config')
 
     decaylist=[]
     for key, value in para.decaylist.items():
@@ -270,12 +281,12 @@
             if sendOpt=='lhep8':
                 print ('preparing to send FCCSW jobs from lhe')
                 import EventProducer.bin.send_lhep8 as slhep8
-                sendlhep8=slhep8.send_lhep8(args.numJobs,args.numEvents, args.process, args.lsf, args.condor, args.local, args.queue, args.priority, args.ncpus, para, version, args.decay, args.pycard, detector)
+                sendlhep8=slhep8.send_lhep8(args.numJobs,args.numEvents, args.process, args.lsf, args.condor, args.local, args.queue, args.priority, args.ncpus, para, version, args.decay, args.pycard, detector, args.customEDM4HEPOutput)
                 sendlhep8.send(args.force)
             elif sendOpt=='p8':
                 print ('preparing to send FCCSW jobs from pythia8 directly')
                 import EventProducer.bin.send_p8 as sp8
-                sendp8=sp8.send_p8(args.numJobs,args.numEvents, args.process, args.lsf, args.condor, args.local, args.queue, args.priority, args.ncpus, para, version, training, detector)
+                sendp8=sp8.send_p8(args.numJobs,args.numEvents, args.process, args.lsf, args.condor, args.local, args.queue, args.priority, args.ncpus, para, version, training, detector, args.customEDM4HEPOutput)
                 sendp8.send()
             elif sendOpt=='stdhep':
                 print ('preparing to send FCCSW jobs from stdhep')

bin/send_lhe.py

@@ -95,16 +95,32 @@ def send(self):
             frun.write('cd job%s_%s\n'%(uid,self.process))
             frun.write('export EOS_MGM_URL=\"root://eospublic.cern.ch\"\n')
             frun.write('source %s\n'%(self.para.defaultstack))
-            frun.write('source /cvmfs/sft.cern.ch/lcg/views/LCG_97a_FCC_4/x86_64-centos7-gcc8-opt/setup.sh \n')
+            # frun.write('source /cvmfs/sft.cern.ch/lcg/views/LCG_97a_FCC_4/x86_64-centos7-gcc8-opt/setup.sh \n') #setting this release up clashes with defaultstack setup w.r.t python version, causing runtime errors -> removed. what is it needed for?
             frun.write('mkdir %s\n'%(lhedir))
             frun.write('mkdir %s%s\n'%(lhedir,self.process))
             frun.write('python /afs/cern.ch/work/f/fccsw/public/FCCutils/eoscopy.py %s/%s.tar.gz .\n'%(gpdir,self.process))
             frun.write('tar -zxf %s.tar.gz\n'%self.process)
+            #different dir structure in powheg gridpacks, compared to mg -> ONLY IN THE NEW ONES!
+            # if self.typelhe == 'gp_pw':
+            #     frun.write('./runcmsgrid.sh %i %i 8\n'%(self.events,int(uid.lstrip('0'))))
+            #     frun.write('echo "finished run"\n')
+            #     frun.write('gzip cmsgrid_final.lhe\n')
+            #     frun.write('xrdcp -N -v cmsgrid_final.lhe.gz root://eospublic.cern.ch/%s/%s/events_%s.lhe.gz\n'%(lhedir,self.process ,uid))
+
+            # else:
+            #     frun.write('cd process/\n')
+            #     frun.write('./run.sh %i %i\n'%(self.events,int(uid.lstrip('0'))))
+            #     frun.write('echo "finished run"\n')
+            #     #frun.write('python /afs/cern.ch/work/f/fccsw/public/FCCutils/eoscopy.py events.lhe.gz %s/%s/events_%s.lhe.gz\n'%(lhedir,self.process ,uid))
+            #     frun.write('xrdcp -N -v events.lhe.gz root://eospublic.cern.ch/%s/%s/events_%s.lhe.gz\n'%(lhedir,self.process ,uid))
+
+            #TEMP
             frun.write('cd process/\n')
             frun.write('./run.sh %i %i\n'%(self.events,int(uid.lstrip('0'))))
             frun.write('echo "finished run"\n')
             #frun.write('python /afs/cern.ch/work/f/fccsw/public/FCCutils/eoscopy.py events.lhe.gz %s/%s/events_%s.lhe.gz\n'%(lhedir,self.process ,uid))
             frun.write('xrdcp -N -v events.lhe.gz root://eospublic.cern.ch/%s/%s/events_%s.lhe.gz\n'%(lhedir,self.process ,uid))
+
             frun.write('echo "lhe file successfully copied on eos"\n')
 
             frun.write('cd ..\n')

bin/send_lhep8.py

@@ -11,7 +11,8 @@
 class send_lhep8():
 
 #__________________________________________________________
-    def __init__(self,njobs, events, process, islsf, iscondor, islocal, queue, priority, ncpus, para, version, decay, pycard, detector):
+    def __init__(self,njobs, events, process, islsf, iscondor, islocal, queue, priority, ncpus, para, version, decay, pycard, detector, custom_edm4hep_config):
+
         self.njobs    = njobs
         self.events   = events
         self.process  = process
@@ -27,7 +28,7 @@ def __init__(self,njobs, events, process, islsf, iscondor, islocal, queue, prior
         self.pycard   = pycard
         self.detector = detector
         self.user     = os.environ['USER']
-
+        self.custom_edm4hep_config = custom_edm4hep_config
 
 #__________________________________________________________
     def send(self, force):
@@ -42,7 +43,8 @@ def send(self, force):
             print ('process %s does not exist as gridpack'%self.process)
             sys.exit(3)
 
-        delphescards_mmr=''
+        delphescards_mmr='' #not sure this is really necessary, could clean up the code to not need it
+        delphescards_emr=''
         delphescards_mr=''
         if 'FCCee' not in self.para.module_name:
             delphescards_mmr = '%s%s/%s'%(self.para.delphescards_dir,self.version,self.para.delphescard_mmr)
@@ -55,6 +57,13 @@ def send(self, force):
                 print ('delphes card does not exist: ',delphescards_mr,' , exit')
                 sys.exit(3)
 
+            #from fcchh_v05 have also a separate electron momentum resolution file: 
+            if "fcc_v05" in self.version:
+                delphescards_emr = '%s%s/%s'%(self.para.delphescards_dir, self.version, self.para.delphescard_emr)
+                if not os.path.isfile(delphescards_emr):
+                    raise Exception("ERROR in param_FCChh - the card for electron momentum resolution doesn't exist, at: "+delphescards_emr)
+
+
         if 'FCCee' not in self.para.module_name:
               delphescards_base = '%s%s/%s'%(self.para.delphescards_dir,self.version,self.para.delphescard_base)
         # 2021/09/08 :
@@ -260,23 +269,36 @@ def send(self, force):
             if self.islocal==False:
                  #frun.write('mkdir -p %s%s/%s\n'%(self.para.delphes_dir,self.version,processp8))
                  frun.write('mkdir -p %s/%s\n'%(outdir,processp8))
+
             frun.write('python /afs/cern.ch/work/f/fccsw/public/FCCutils/eoscopy.py %s .\n'%(tmpf['processing']['out']))
             frun.write('gunzip -c %s > events.lhe\n'%tmpf['processing']['out'].split('/')[-1])          
             frun.write('python /afs/cern.ch/work/f/fccsw/public/FCCutils/eoscopy.py %s .\n'%(delphescards_base))
             if 'fcc' in self.version and 'FCCee' not in self.para.module_name:
+                frun.write('python /afs/cern.ch/work/f/fccsw/public/FCCutils/eoscopy.py %s card.tcl\n'%(delphescards_base))
                 frun.write('python /afs/cern.ch/work/f/fccsw/public/FCCutils/eoscopy.py %s .\n'%(delphescards_mmr))
                 frun.write('python /afs/cern.ch/work/f/fccsw/public/FCCutils/eoscopy.py %s .\n'%(delphescards_mr))
+                if "fcc_v05" in self.version:
+                    frun.write('python /afs/cern.ch/work/f/fccsw/public/FCCutils/eoscopy.py %s .\n'%(delphescards_emr))
             if 'FCCee' not in self.para.module_name:
                 frun.write('python /afs/cern.ch/work/f/fccsw/public/FCCutils/eoscopy.py %s config.py \n'%(fccconfig))
+                #use the standard k4simdelphes edm4hep output config for FCC-hh production:
+                if self.custom_edm4hep_config:
+                    frun.write('cp {} .\n'.format(self.custom_edm4hep_config))
+                else:
+                    frun.write('cp $K4SIMDELPHES/edm4hep_output_config.tcl .\n')
             else:
                 frun.write('python /afs/cern.ch/work/f/fccsw/public/FCCutils/eoscopy.py /eos/experiment/fcc/ee/generation/FCC-config/%s/FCCee/Delphes/edm4hep_%s.tcl edm4hep_output_config.tcl\n'%(self.version,self.detector))
                 frun.write('python /afs/cern.ch/work/f/fccsw/public/FCCutils/eoscopy.py %s card.tcl\n'%(delphescards_base))
 
+
+
             frun.write('python /afs/cern.ch/work/f/fccsw/public/FCCutils/eoscopy.py %s card.cmd\n'%(pythiacard))
             frun.write('echo "Beams:LHEF = events.lhe" >> card.cmd\n')
             frun.write('echo "Random:seed = %s" >> card.cmd\n'%pythiaseed.lstrip('0'))
             frun.write('echo "Main:numberOfEvents = %i" >> card.cmd\n'%(self.events))
 
+            #TEMP OVERWRITE TO USE LOCAL VERSION OF k4SimDelphes
+            # frun.write('/afs/cern.ch/user/b/bistapf/Dev_k4SimDelphes/k4SimDelphes/build/standalone/DelphesPythia8_EDM4HEP card.tcl edm4hep_output_config.tcl card.cmd events_%s.root\n'%(jobid))
             frun.write('DelphesPythia8_EDM4HEP card.tcl edm4hep_output_config.tcl card.cmd events_%s.root\n'%(jobid))
 
             frun.write('python /afs/cern.ch/work/f/fccsw/public/FCCutils/eoscopy.py events_%s.root %s\n'%(jobid,outfile))