[15_0_X] Use HardwareResourcesDescription in ProcessConfiguration #47416

makortel · 2025-02-20T14:56:52Z

PR description:

Backport of #47280, #47355, #47473 (well, backported by dropping out the commit reverted there), and #47477 (the commit of the PR split in two, one to replace the commit dropped in the earlier step, and the rest included in the last commit of this PR).

Resolves cms-sw/framework-team#1248

PR validation:

Code compiles (plus the tests in #47280 and #47355)

If this PR is a backport please specify the original PR and why you need to backport that PR. If this PR will be backported please specify to which release cycle the backport is meant for:

Backport of #47280 and #47355

Just storing the strings from process.options.accelerators is more general as it avoids updating the ResourceInformation(Service) every time a new accelerator type is added to the system (outside of the framework).

…ription

…esDescription

cmsbuild · 2025-02-20T14:57:25Z

A new Pull Request was created by @makortel for CMSSW_15_0_X.

It involves the following packages:

DQM/SiStripMonitorHardware (dqm)
DQMServices/FwkIO (dqm)
DataFormats/Provenance (core)
FWCore/AbstractServices (core)
FWCore/Framework (core)
FWCore/Integration (core)
FWCore/Services (core)
FWCore/Sources (core)
FWCore/TestProcessor (core)
FWCore/Utilities (core)
GeneratorInterface/LHEInterface (generators)
HeterogeneousCore/CUDAServices (heterogeneous)
HeterogeneousCore/ROCmServices (heterogeneous)
IOPool/Common (core)
IOPool/Input (core)
IOPool/SecondaryInput (core)
IOPool/Streamer (core)
Mixing/Base (simulation)
PhysicsTools/PyTorch (ml)
PhysicsTools/TensorFlow (ml)

@Dr15Jones, @antoniovagnerini, @bbilin, @civanch, @cmsbuild, @fwyzard, @kpedro88, @lviliani, @makortel, @mdhildreth, @menglu21, @mkirsano, @rseidita, @smuzaffar, @valsdav, @y19y19 can you please review it and eventually sign? Thanks.
@alberto-sanchez, @arossi83, @barvic, @fabiocos, @felicepantaleo, @fioriNTU, @fwyzard, @idebruyn, @jandrea, @missirol, @mkirsano, @mmusich, @richa2710, @riga, @rovere, @sroychow, @threus, @wddgit this is something you requested to watch as well.
@antoniovilela, @mandrenguyen, @rappoccio, @sextonkennedy you are the release manager for this.

cms-bot commands are listed here

cmsbuild · 2025-02-20T14:57:25Z

cms-bot internal usage

makortel · 2025-02-20T14:57:46Z

enable gpu

makortel · 2025-02-20T14:57:52Z

@cmsbuild, please test

cmsbuild · 2025-02-20T19:22:48Z

+1

Size: This PR adds an extra 220KB to repository
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-9d1efb/44545/summary.html
COMMIT: c8326da
CMSSW: CMSSW_15_0_X_2025-02-20-1100/el8_amd64_gcc12
Additional Tests: GPU
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/47416/44545/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

You potentially added 1 lines to the logs
Reco comparison results: 9 differences found in the comparisons
DQMHistoTests: Total files compared: 50
DQMHistoTests: Total histograms compared: 4018889
DQMHistoTests: Total failures: 68
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 4018801
DQMHistoTests: Total skipped: 20
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 49 files compared)
Checked 218 log files, 189 edm output root files, 50 DQM output files
TriggerResults: no differences found

GPU Comparison Summary

Summary:

No significant changes to the logs found
Reco comparison results: 24 differences found in the comparisons
DQMHistoTests: Total files compared: 7
DQMHistoTests: Total histograms compared: 53071
DQMHistoTests: Total failures: 869
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 52202
DQMHistoTests: Total skipped: 0
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 6 files compared)
Checked 24 log files, 30 edm output root files, 7 DQM output files
TriggerResults: no differences found

makortel · 2025-02-20T19:30:22Z

CPU comparison differences are related to #47071

GPU comparison differences look compatible with the non-reproducibilities in the pixel code

cmsbuild · 2025-02-27T14:14:15Z

Pull request #47416 was updated. @Dr15Jones, @cmsbuild, @makortel, @smuzaffar, @valsdav, @y19y19 can you please check and sign again.

makortel · 2025-02-27T14:22:50Z

I took out the commit that altered the edmProvDump output. I opened #47473 to revert the corresponding commit from master so we can check that it is sufficient to get CRAB working again.

makortel · 2025-02-27T15:18:57Z

@cmsbuild, please test

cmsbuild · 2025-02-27T19:48:52Z

-1

Failed Tests: UnitTests GpuUnitTests
Size: This PR adds an extra 16KB to repository
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-9d1efb/44723/summary.html
COMMIT: 29ae1f5
CMSSW: CMSSW_15_0_X_2025-02-27-1100/el8_amd64_gcc12
Additional Tests: GPU
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/47416/44723/install.sh to create a dev area with all the needed externals and cmssw changes.

Unit Tests

I found 2 errors in the following unit tests:

---> test TestIOPoolInputReducedProcessHistoryHardwareResources had ERRORS
---> test TestIOPoolStreamerReducedProcessHistoryHardwareResources had ERRORS

GPU Unit Tests

I found 1 errors in the following unit tests:

---> test testTorchSimpleDnnCUDA had ERRORS

Comparison Summary

Summary:

You potentially removed 2 lines from the logs
Reco comparison results: 2 differences found in the comparisons
DQMHistoTests: Total files compared: 50
DQMHistoTests: Total histograms compared: 4018889
DQMHistoTests: Total failures: 123
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 4018746
DQMHistoTests: Total skipped: 20
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 49 files compared)
Checked 218 log files, 189 edm output root files, 50 DQM output files
TriggerResults: no differences found

GPU Comparison Summary

Summary:

No significant changes to the logs found
Reco comparison results: 8 differences found in the comparisons
DQMHistoTests: Total files compared: 7
DQMHistoTests: Total histograms compared: 53071
DQMHistoTests: Total failures: 380
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 52691
DQMHistoTests: Total skipped: 0
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 6 files compared)
Checked 24 log files, 30 edm output root files, 7 DQM output files
TriggerResults: no differences found

For the moment CRAB needs the edmProvDump without the hardware information, but two tests need the hardware information.

… resources

cmsbuild · 2025-02-28T19:18:53Z

Pull request #47416 was updated. @Dr15Jones, @cmsbuild, @makortel, @smuzaffar, @valsdav, @y19y19 can you please check and sign again.

makortel · 2025-02-28T19:19:45Z

@cmsbuild, please test

makortel · 2025-02-28T19:20:20Z

Ok, now this PR corresponds to the status in the master branch (with a bit different history though), and should work for both CRAB and the unit tests.

cmsbuild · 2025-02-28T23:20:13Z

+1

Size: This PR adds an extra 20KB to repository
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-9d1efb/44754/summary.html
COMMIT: 4c6ccf4
CMSSW: CMSSW_15_0_X_2025-02-28-1100/el8_amd64_gcc12
Additional Tests: GPU
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/47416/44754/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

You potentially removed 2 lines from the logs
Reco comparison results: 160 differences found in the comparisons
DQMHistoTests: Total files compared: 50
DQMHistoTests: Total histograms compared: 4018889
DQMHistoTests: Total failures: 4930
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 4013939
DQMHistoTests: Total skipped: 20
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 49 files compared)
Checked 218 log files, 189 edm output root files, 50 DQM output files
TriggerResults: no differences found

GPU Comparison Summary

Summary:

No significant changes to the logs found
Reco comparison results: 24 differences found in the comparisons
DQMHistoTests: Total files compared: 7
DQMHistoTests: Total histograms compared: 53071
DQMHistoTests: Total failures: 877
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 52194
DQMHistoTests: Total skipped: 0
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 6 files compared)
Checked 24 log files, 30 edm output root files, 7 DQM output files
TriggerResults: no differences found

valsdav · 2025-03-03T08:01:51Z

+ml

makortel added 9 commits February 20, 2025 15:32

Move ResourceInformation abstract base class to FWCore/AbstractServices

4a9711e

Change ResourceInformation::acceleratorTypes() to be a vector<string>

491cfca

Just storing the strings from process.options.accelerators is more general as it avoids updating the ResourceInformation(Service) every time a new accelerator type is added to the system (outside of the framework).

Add hasGpuNvidia() function to ResourceInformation service

af07337

Use ResourceInformation::hasGpuNvidia() in TensorFlow

ef280e0

Add a function in ResourceInformation to return HardwareResourcesDesc…

5599cf2

…ription

Make empty HardwareResourcesDescription to serialize as an empty string

052ebbd

Use HardwareResourcesDescription in ProcessConfiguration

3b590e2

Keep ResourceInformation service optional in ScheduleItems

dd9ed4f

Extend Process{Configration,History}_t tests to cover HardwareResourc…

07ddd0b

…esDescription

cmsbuild added this to the CMSSW_15_0_X milestone Feb 20, 2025

cmsbuild added simulation-pending dqm-pending core-pending generators-pending pending-signatures tests-pending orp-pending heterogeneous-pending trk ml-pending labels Feb 20, 2025

cmsbuild added tests-started and removed tests-pending labels Feb 20, 2025

cmsbuild added tests-approved and removed tests-started labels Feb 20, 2025

cmsbuild added core-pending tests-pending and removed tests-approved labels Feb 27, 2025

makortel mentioned this pull request Feb 27, 2025

Revert "Add HardwareResourcesInformation printout to edmProvDump" #47473

Merged

cmsbuild added tests-started and removed tests-pending labels Feb 27, 2025

cmsbuild added tests-rejected and removed tests-started labels Feb 27, 2025

makortel mentioned this pull request Feb 27, 2025

Add optional HardwareResourcesInformation printout to edmProvDump #47477

Merged

makortel added 3 commits February 27, 2025 21:54

Add optional HardwareResourcesInformation printout to edmProvDump

d6717ff

For the moment CRAB needs the edmProvDump without the hardware information, but two tests need the hardware information.

Rename TestIOPool{Input,Streamer}ReducedProcessHistory tests

563c542

Add tests for merging ROOT and streamer files with different hardware…

4c6ccf4

… resources

makortel force-pushed the processConfigurationHardwareResourcesDescription_150x branch from 29ae1f5 to 4c6ccf4 Compare February 28, 2025 19:18

cmsbuild added tests-pending and removed tests-rejected labels Feb 28, 2025

cmsbuild added tests-started and removed tests-pending labels Feb 28, 2025

cmsbuild added tests-approved and removed tests-started labels Feb 28, 2025

cmsbuild added ml-approved and removed ml-pending labels Mar 3, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[15_0_X] Use HardwareResourcesDescription in ProcessConfiguration #47416

[15_0_X] Use HardwareResourcesDescription in ProcessConfiguration #47416

makortel commented Feb 20, 2025 •

edited

Loading

cmsbuild commented Feb 20, 2025

cmsbuild commented Feb 20, 2025 •

edited

Loading

makortel commented Feb 20, 2025

makortel commented Feb 20, 2025

cmsbuild commented Feb 20, 2025

makortel commented Feb 20, 2025

cmsbuild commented Feb 27, 2025

makortel commented Feb 27, 2025

makortel commented Feb 27, 2025

cmsbuild commented Feb 27, 2025

cmsbuild commented Feb 28, 2025

makortel commented Feb 28, 2025

makortel commented Feb 28, 2025

cmsbuild commented Feb 28, 2025

valsdav commented Mar 3, 2025

[15_0_X] Use HardwareResourcesDescription in ProcessConfiguration #47416

Are you sure you want to change the base?

[15_0_X] Use HardwareResourcesDescription in ProcessConfiguration #47416

Conversation

makortel commented Feb 20, 2025 • edited Loading

PR description:

PR validation:

If this PR is a backport please specify the original PR and why you need to backport that PR. If this PR will be backported please specify to which release cycle the backport is meant for:

cmsbuild commented Feb 20, 2025

cmsbuild commented Feb 20, 2025 • edited Loading

makortel commented Feb 20, 2025

makortel commented Feb 20, 2025

cmsbuild commented Feb 20, 2025

Comparison Summary

GPU Comparison Summary

makortel commented Feb 20, 2025

cmsbuild commented Feb 27, 2025

makortel commented Feb 27, 2025

makortel commented Feb 27, 2025

cmsbuild commented Feb 27, 2025

Unit Tests

GPU Unit Tests

Comparison Summary

GPU Comparison Summary

cmsbuild commented Feb 28, 2025

makortel commented Feb 28, 2025

makortel commented Feb 28, 2025

cmsbuild commented Feb 28, 2025

Comparison Summary

GPU Comparison Summary

valsdav commented Mar 3, 2025

makortel commented Feb 20, 2025 •

edited

Loading

cmsbuild commented Feb 20, 2025 •

edited

Loading