Non-uniform work-groups at 3 ranks fails for SYCL (acc on DPC++)

[tom91136]

+ srun --cpu-bind=map_cpu:49,57,17,25,1,9,33,41 -n 3 --cpu-bind none --mem-bind none sh -c 'export ROCR_VISIBLE_DEVICES=$((SLURM_PROCID % N_GPUS)) && echo "# ROCR_VISIBLE_DEVICES=$ROCR_VISIBLE_DEVICES@$(hostname):$SLURM_PROCID" && /users/tomlin/performance-portability/benchmarking/2023/cloverleaf/results/cloverleaf-mi250x_oneapi-2024.1_sycl-acc/CloverLeaf/build/sycl-acc-cloverleaf --file /users/tomlin/performance-portability/benchmarking/2023/cloverleaf/results/cloverleaf-mi250x_oneapi-2024.1_sycl-acc/CloverLeaf/InputDecks/clover_bm256_300.in --out ../out/clover_np3_mi250x_oneapi-2024.1_sycl-acc_256_300_stage_false.out --staging-buffer false '
# ROCR_VISIBLE_DEVICES=0@nid006622:0
# ROCR_VISIBLE_DEVICES=1@nid006622:1
# ROCR_VISIBLE_DEVICES=2@nid006623:2
---
Devices:
 0: gfx90a:sramecc+:xnack-
CloverLeaf:
 - Ver.:     2.000
 - Deck:     /users/tomlin/performance-portability/benchmarking/2023/cloverleaf/results/cloverleaf-mi250x_oneapi-2024.1_sycl-acc/CloverLeaf/InputDecks/clover_bm256_300.in
 - Out:      ../out/clover_np3_mi250x_oneapi-2024.1_sycl-acc_256_300_stage_false.out
 - Profiler: deck-specified
MPI:
 - Enabled:     true
 - Total ranks: 3
 - Header device-awareness (CUDA-awareness):  unknown
 - Runtime device-awareness (CUDA-awareness): unknown
 - Host-Device halo exchange staging buffer:  false
Model:
 - Name:      SYCL (accessors)
 - Execution: Offload (device)
 - Indexing: RANGE2D_LINEAR
 - SYCL device: gfx90a:sramecc+:xnack-
   - Type    : gpu
   - Version : gfx90a:sramecc+:xnack-
   - Vendor  : AMD Corporation
   - Driver  : HIP 50221.15
# ---- 
Output: |+1
 Output file clover.out opened. All output will go there.
 Args: --file /users/tomlin/performance-portability/benchmarking/2023/cloverleaf/results/cloverleaf-mi250x_oneapi-2024.1_sycl-acc/CloverLeaf/InputDecks/clover_bm256_300.in --out ../out/clover_np3_mi250x_oneapi-2024.1_sycl-acc_256_300_stage_false.out --staging-buffer false
 Using input: `/users/tomlin/performance-portability/benchmarking/2023/cloverleaf/results/cloverleaf-mi250x_oneapi-2024.1_sycl-acc/CloverLeaf/InputDecks/clover_bm256_300.in`
 Problem initialised and generated
 Launching hydro
[SYCL] Exception : `Non-uniform work-groups are not supported by the target device -54 (PI_ERROR_INVALID_WORK_GROUP_SIZE)`
terminate called after throwing an instance of 'sycl::_V1::exception'
  what():  Non-uniform work-groups are not supported by the target device -54 (PI_ERROR_INVALID_WORK_GROUP_SIZE)
[SYCL] Exception : `Non-uniform work-groups are not supported by the target device -54 (PI_ERROR_INVALID_WORK_GROUP_SIZE)`
[SYCL] Exception : `Non-uniform work-groups are not supported by the target device -54 (PI_ERROR_INVALID_WORK_GROUP_SIZE)`
terminate called after throwing an instance of 'sycl::_V1::exception'
  what():  Non-uniform work-groups are not supported by the target device -54 (PI_ERROR_INVALID_WORK_GROUP_SIZE)
terminate called after throwing an instance of 'sycl::_V1::exception'
  what():  Non-uniform work-groups are not supported by the target device -54 (PI_ERROR_INVALID_WORK_GROUP_SIZE)
srun: error: nid006622: tasks 0-1: Aborted
srun: launch/slurm: _step_signal: Terminating StepId=6994308.5
srun: error: nid006623: task 2: Aborted (core dumped)