Collecting system metrics during runs.

[venkat-1]

We need to collect system metrics, including GPU memory errors, NIC errors and re-transmits, etc. during each run. Ideally, we will collect this with as low-overhead using a background thread/process. This will help identify any slowdown during runs and where it could originate from.

Another aspect to consider is to run periodic tests on the GPUs, such as hgemm and Igemm and any peer-to-peer tests, to see if there are any low-performing GPUs over time. So, in addition to running at the beginning of a job, we can run this after a specific duration, such as after every checkpoint.

[nscottnichols]

We need to collect system metrics, including GPU memory errors, NIC errors and re-transmits, etc. during each run. Ideally, we will collect this with as low-overhead using a background thread/process. This will help identify any slowdown during runs and where it could originate from.

I have determined a light-weight route using xpu-smi to track many of the metrics we are interested in and reached out to HPE for a pathway to track relevant NIC metrics. Here is a checklist for what we are looking to track:

• GPU / Accelerator Unit Counters:
  • GPU correctable and uncorrectable memory errors
  • Free and used memory
• Xe Link related counters:
  • Xe Link Throughput
• NIC/Slingshot related counters:
  • retransmits
  • errors
  • congestion
• CPU related counters:
  • TBD

I'm currently working on a postprocessing script to analyze the output from xpu-smi.

[venkat-1]

It needs to target multiple systems. So, at least for now target Polaris (Nvidia+ SS11), Aurora(Intel + SS11), and Sophia (Nvidia + IB)

[nscottnichols]

Another aspect to consider is to run periodic tests on the GPUs, such as hgemm and Igemm and any peer-to-peer tests, to see if there are any low-performing GPUs over time. So, in addition to running at the beginning of a job, we can run this after a specific duration, such as after every checkpoint.

We have GEMM benchmarks as part of the node performance overview. I can adapt these tests to be run as part of a pre-execution hook that we run during our job submissions and to target other systems.

[nscottnichols]

@venkat-1 I created a new issue for testing, #40. Also, for metric tracking, are we looking for something that can be post processed, something with a live view/feed, or something integrated with W&B?