Investigate the impact of NVIDIA MPS

[fwyzard]

No description provided.

[fwyzard]

Here is a quick performance comparison running the same jobs with and without MPS.

System configuration

2 CPUs:
  0: Intel(R) Xeon(R) Gold 6130 CPU @ 2.10GHz (16 cores, 32 threads)
  1: Intel(R) Xeon(R) Gold 6130 CPU @ 2.10GHz (16 cores, 32 threads)

1 visible NVIDIA GPUs:
  0: Tesla T4 (UUID: GPU-0e44f21a-ab43-7bf2-08d7-8cf1c32dd05b)

Native performance, without MPS

Running 10 times over 4200 events with 1 jobs, each with 16 threads, 16 streams and 1 GPUs

   800.4 ±   0.9 ev/s

Running 10 times over 4200 events with 2 jobs, each with 8 threads, 8 streams and 1 GPUs

cudaErrorIllegalAddress: an illegal memory access was encountered

Performance with MPS

Running 10 times over 4200 events with 1 jobs, each with 16 threads, 16 streams and 1 GPUs

   761.7 ±   1.9 ev/s

Running 10 times over 4200 events with 2 jobs, each with 8 threads, 8 streams and 1 GPUs

   779.8 ±   2.4 ev/s

[fwyzard]

It looks like MPS incurs in a performance penalty around 5%.

However, so far it seems to be the only way to run multiple jobs - which actually recovers part of the penalty.

[fwyzard]

Notes: how to run with MPS

Start the MPS daemon

Run the server as a daemon in the background:

sudo nvidia-cuda-mps-control -d

or as a process in the foreground:

sudo nvidia-cuda-mps-control -f

Start an MPS server

Start a server for the current user:

echo start_server -uid $UID | nvidia-cuda-mps-control

Run CUDA jobs

Run the CUDA jobs normally.

Stop the MPS daemon

echo quit | sudo nvidia-cuda-mps-control

[fwyzard]

See also some documentation at https://patatrack.web.cern.ch/patatrack/wiki/MPS-setup/ .