diff --git a/tests/log-messages.feature b/tests/log-messages.feature new file mode 100644 index 0000000..19ebc7b --- /dev/null +++ b/tests/log-messages.feature @@ -0,0 +1,77 @@ +# Tests for the expected workflow log messages + +Feature: Log messages + + As a researcher, + I want to be able to see the log messages of my workflow execution, + So that I can verify that the workflow ran correctly. + + Scenario: The compiling step has produced the expected messages + When the workflow is finished + Then the engine logs should contain + """ + Publishing step:0, cmd: g++ -g -O3 -Wall -Wextra -Wpedantic -o skim skim.cxx `root-config --cflags --libs`, total steps 4 to MQ + """ + + Scenario: The skimming step has produced the expected messages + When the workflow is finished + Then the engine logs should contain "Publishing step:1, cmd: ./skim" + And the job logs for the "skimming" step should contain + """ + >>> Process final state FourElectrons for sample Run2012B_DoubleElectron: + At least our electrons: pass=258449 all=21474287 -- eff=1.20 % cumulative eff=1.20 % + Require good isolation: pass=113 all=258449 -- eff=0.04 % cumulative eff=0.00 % + Good Electron kinematics: pass=67 all=113 -- eff=59.29 % cumulative eff=0.00 % + """ + + Scenario: The histogramming step has produced the expected messages + When the workflow is finished + Then the engine logs should contain "Publishing step:2, cmd: python histograms.py" + And the job logs for the "histogramming" step should contain + """ + >>> Process skimmed sample SMHiggsToZZTo4L and final state FourMuons + >>> Process skimmed sample SMHiggsToZZTo4L and final state FourElectrons + >>> Process skimmed sample SMHiggsToZZTo4L and final state TwoMuonsTwoElectrons + >>> Process skimmed sample ZZTo4mu and final state FourMuons + >>> Process skimmed sample ZZTo4e and final state FourElectrons + >>> Process skimmed sample ZZTo2e2mu and final state TwoMuonsTwoElectrons + >>> Process skimmed sample Run2012B_DoubleMuParked and final state FourMuons + >>> Process skimmed sample Run2012B_DoubleMuParked and final state TwoMuonsTwoElectrons + >>> Process skimmed sample Run2012C_DoubleMuParked and final state FourMuons + >>> Process skimmed sample Run2012C_DoubleMuParked and final state TwoMuonsTwoElectrons + >>> Process skimmed sample Run2012B_DoubleElectron and final state FourElectrons + >>> Process skimmed sample Run2012B_DoubleElectron and final state TwoMuonsTwoElectrons + >>> Process skimmed sample Run2012C_DoubleElectron and final state FourElectrons + >>> Process skimmed sample Run2012C_DoubleElectron and final state TwoMuonsTwoElectrons + """ + + Scenario: The plotting step has produced the expected messages + When the workflow is finished + Then the engine logs should contain "Publishing step:3, cmd: python plot.py" + And the job logs for the "plotting" step should contain + """ + Info in : pdf file FourMuons_Higgs_mass.pdf has been created + Info in : png file FourMuons_Higgs_mass.png has been created + Info in : pdf file FourElectrons_Higgs_mass.pdf has been created + Info in : png file FourElectrons_Higgs_mass.png has been created + Info in : pdf file TwoMuonsTwoElectrons_Higgs_mass.pdf has been created + Info in : png file TwoMuonsTwoElectrons_Higgs_mass.png has been created + Info in : pdf file combined_Higgs_mass.pdf has been created + Info in : png file combined_Higgs_mass.png has been created + Info in : pdf file FourMuons_Z1_mass.pdf has been created + Info in : png file FourMuons_Z1_mass.png has been created + Info in : pdf file FourElectrons_Z1_mass.pdf has been created + Info in : png file FourElectrons_Z1_mass.png has been created + Info in : pdf file TwoMuonsTwoElectrons_Z1_mass.pdf has been created + Info in : png file TwoMuonsTwoElectrons_Z1_mass.png has been created + Info in : pdf file combined_Z1_mass.pdf has been created + Info in : png file combined_Z1_mass.png has been created + Info in : pdf file FourMuons_Z2_mass.pdf has been created + Info in : png file FourMuons_Z2_mass.png has been created + Info in : pdf file FourElectrons_Z2_mass.pdf has been created + Info in : png file FourElectrons_Z2_mass.png has been created + Info in : pdf file TwoMuonsTwoElectrons_Z2_mass.pdf has been created + Info in : png file TwoMuonsTwoElectrons_Z2_mass.png has been created + Info in : pdf file combined_Z2_mass.pdf has been created + Info in : png file combined_Z2_mass.png has been created + """ \ No newline at end of file diff --git a/tests/run-duration.feature b/tests/run-duration.feature new file mode 100644 index 0000000..c51182f --- /dev/null +++ b/tests/run-duration.feature @@ -0,0 +1,27 @@ +# Tests for the workflow execution duration time + +Feature: Run duration + + As a researcher, + I want to verify that my workflow finishes in a reasonable amount of time, + so that I can stay assured that there are no unusual problems with computing resources. + + Scenario: The workflow terminates in a reasonable amount of time + When the workflow execution completes + Then the workflow run duration should be less than 25 minutes + + Scenario: The compiling step terminates in a reasonable amount of time + When the workflow execution completes + Then the duration of the step "compiling" should be less than 5 minutes + + Scenario: Skimming the data takes a reasonable amount of time + When the workflow execution completes + Then the duration of the step "skimming" should be less than 15 minutes + + Scenario: The histograms are created in a reasonable amount of time + When the workflow execution completes + Then the duration of the step "histogramming" should be less than 15 minutes + + Scenario: Combining the histograms into plot takes a reasonable amount of time + When the workflow execution completes + Then the duration of the step "plotting" should be less than 5 minutes diff --git a/tests/workspace-files.feature b/tests/workspace-files.feature new file mode 100644 index 0000000..033bb3d --- /dev/null +++ b/tests/workspace-files.feature @@ -0,0 +1,73 @@ +# Tests for the presence of files in the workspace + +Feature: Workspace files + + As a researcher, + I want to make sure that my workflow produces the expected files, + and that the relevant input files are present in the workspace, + so that I can be sure that the workflow outputs are correct. + + Scenario: The workspace contains all the relevant input files + When the workflow execution completes + Then the workspace should contain "skim.cxx" + And the workspace should include "histograms.py" + And the workspace should include "plot.py" + + Rule: The output files and reports are generated correctly + + Scenario: The expected plots are present in the workspace + When the workflow is finished + Then the workspace should include "combined_Higgs_mass.pdf" + And the workspace should include "combined_Higgs_mass.png" + And the workspace should include "combined_Z1_mass.pdf" + And the workspace should include "combined_Z1_mass.png" + And the workspace should include "combined_Z2_mass.pdf" + And the workspace should include "combined_Z2_mass.png" + And the workspace should include "FourElectrons_Higgs_mass.pdf" + And the workspace should include "FourElectrons_Higgs_mass.png" + And the workspace should include "FourElectrons_Z1_mass.pdf" + And the workspace should include "FourElectrons_Z1_mass.png" + And the workspace should include "FourElectrons_Z2_mass.pdf" + And the workspace should include "FourElectrons_Z2_mass.png" + And the workspace should include "FourMuons_Higgs_mass.pdf" + And the workspace should include "FourMuons_Higgs_mass.png" + And the workspace should include "FourMuons_Z1_mass.pdf" + And the workspace should include "FourMuons_Z1_mass.png" + And the workspace should include "FourMuons_Z2_mass.pdf" + And the workspace should include "FourMuons_Z2_mass.png" + And the workspace should include "TwoMuonsTwoElectrons_Higgs_mass.pdf" + And the workspace should include "TwoMuonsTwoElectrons_Higgs_mass.png" + And the workspace should include "TwoMuonsTwoElectrons_Z1_mass.pdf" + And the workspace should include "TwoMuonsTwoElectrons_Z1_mass.png" + And the workspace should include "TwoMuonsTwoElectrons_Z2_mass.pdf" + And the workspace should include "TwoMuonsTwoElectrons_Z2_mass.png" + + Scenario: The expected plots represent the expected data + When the workflow is finished + Then the size of the file "FourMuons_Z2_mass.pdf" should be between "16 KiB" and "17 KiB" + And the size of the file "TwoMuonsTwoElectrons_Higgs_mass.png" should be between "13 KiB" and "14 KiB" + And the size of the file "FourMuons_Z1_mass.png" should be between "14 KiB" and "15 KiB" + And the size of the file "plot.py" should be between "6 KiB" and "7 KiB" + And the size of the file "FourMuons_Higgs_mass.png" should be between "13 KiB" and "14 KiB" + And the size of the file "histograms.py" should be between "3 KiB" and "4 KiB" + And the size of the file "combined_Z2_mass.png" should be between "15 KiB" and "16 KiB" + And the size of the file "TwoMuonsTwoElectrons_Higgs_mass.pdf" should be between "16 KiB" and "17 KiB" + And the size of the file "TwoMuonsTwoElectrons_Z2_mass.png" should be between "14 KiB" and "15 KiB" + And the size of the file "FourElectrons_Z1_mass.pdf" should be between "15 KiB" and "16 KiB" + And the size of the file "combined_Z1_mass.png" should be between "14 KiB" and "15 KiB" + And the size of the file "FourElectrons_Z1_mass.png" should be between "13 KiB" and "14 KiB" + And the size of the file "combined_Higgs_mass.pdf" should be between "17 KiB" and "18 KiB" + And the size of the file "FourElectrons_Z2_mass.png" should be between "14 KiB" and "15 KiB" + And the size of the file "FourElectrons_Higgs_mass.pdf" should be between "15 KiB" and "16 KiB" + And the size of the file "combined_Z1_mass.pdf" should be between "16 KiB" and "17 KiB" + And the size of the file "FourMuons_Z1_mass.pdf" should be between "16 KiB" and "17 KiB" + And the size of the file "TwoMuonsTwoElectrons_Z2_mass.pdf" should be between "17 KiB" and "18 KiB" + And the size of the file "TwoMuonsTwoElectrons_Z1_mass.pdf" should be between "15.5 KiB" and "16.5 KiB" + And the size of the file "combined_Higgs_mass.png" should be between "15 KiB" and "16 KiB" + And the size of the file "FourElectrons_Z2_mass.pdf" should be between "16 KiB" and "17 KiB" + And the size of the file "FourElectrons_Higgs_mass.png" should be between "12 KiB" and "13 KiB" + And the size of the file "combined_Z2_mass.pdf" should be between "17 KiB" and "18 KiB" + And the size of the file "TwoMuonsTwoElectrons_Z1_mass.png" should be between "13 KiB" and "14 KiB" + And the size of the file "FourMuons_Z2_mass.png" should be between "14 KiB" and "15 KiB" + And the size of the file "FourMuons_Higgs_mass.pdf" should be between "16 KiB" and "17 KiB" +