Problem with the example TurbPipePeriodic #467

Chaggge · 2022-11-17T10:25:25Z

Chaggge
Nov 17, 2022

Hi, neks!
I am new with nekrs.

I have compiled nekrs without errors as it is described at: https://github.com/Nek5000/nekRS.
Then I tried to run TurbPipePeriodic using mpirun -np 2 nekrs --setup turbPipe.par. But I faced with the problem:
Message : Device: Creating Device CUDA Error [ 101 ]: CUDA_ERROR_INVALID_DEVICE

My configuration:
gcc (Debian 10.2.1-6) 10.2.1 20210110
NVIDIA (R) CUDA Debugger 11.2 release
GPU: GeForce GTX 1050 Ti

yslan · 2022-11-17T23:43:43Z

yslan
Nov 17, 2022

Do you have two GPUs on your machine?

5 replies

Chaggge Nov 21, 2022
Author

No, only one

yslan Nov 21, 2022

You will need 2 GPU to run with 2 processors. NekRS only support 1-1 CPU-GPU binding.

Chaggge Nov 22, 2022
Author

I also have one integrated graphics processor

vivashchenko Nov 24, 2022

@yslan Thanks, with -np 1 it works! Does it mean that I can use only 1 CPU core with all GPU's cores?

yslan Dec 1, 2022

Yes. The mpi is assigned to a CPU core then it's binding to a GPU. The computation is done in parallel in terms of GPU threads.

The nvidia A100 does support 7 instances that can run different jobs under a GPU. For that case, it is possible (but not recommended) to bind 7 cpu cores with single A100.

ME-20244 · 2024-04-20T14:35:18Z

ME-20244
Apr 20, 2024

Hello all!

I am new in using NekRS! I successfully built NekRS alone and within Cardinal on a personal station with one GPU.
In running any case involving nekrs, I get the errors:
Error in device_t: Active occa mode does not match selected backend! OR
Device: Creating Device CUDA Error [ 101 ]: CUDA_ERROR_INVALID_DEVICE

occa info : indicates the presence of GPU and OPENCL.

nvidia-smi gives for me:

+---------------------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=======================================================================================|
| 0 N/A N/A 1940 G /usr/lib/xorg/Xorg 166MiB |
| 0 N/A N/A 2219 G /usr/bin/gnome-shell 78MiB |
| 0 N/A N/A 24563 G ...irefox/3728/usr/lib/firefox/firefox 148MiB |
+---------------------------------------------------------------------------------------+

Is it a problem of configuration of GPU or nekrs cannot run on such machine ?
How can I set CUDA_VISIBLE_DEVICES in bashrc ?
Any help, PLEASE !

4 replies

yslan Apr 26, 2024

Need more information.

Have you tried running nekrs standalone? What version are you using?
How did you launch nekrs?
What's the full output from occa info? Does it detect CUDA?

stgeke Apr 26, 2024
Maintainer

I think it makes more sense to run the code using the serial backend as the GPU is not really powerful.

ME-20244 Apr 29, 2024

Hi !
Thank you very much for your reply and guidance.
I am using version 23.0, and occa informs me that I have CUDA installed.
In a standalone version, I changed the backend to CPU and it works for me but with "nrsmpi ethier 2" since nekrs gives error:
An error occurred in MPI_Win_create reported by process [3372023809,0] on communicator MPI COMMUNICATOR 5 DUP FROM 3.

stgeke Apr 29, 2024
Maintainer

Could you please provide the complete output when running the code?

ME-20244 · 2024-04-29T14:31:33Z

ME-20244
Apr 29, 2024

Hi !
I- For the running case, a piece of execution is:

runtime statistics (step= 17000 totalElapsed= 100151s):
name time abs% rel% calls
solve 1.00137e+05s 100.0
min 4.63299e+00s
max 1.14286e+01s
flops/rank 1.06249e+10
checkpointing 3.71194e-01s 0.0 5
udfExecuteStep 2.83827e-02s 0.0 17000
makef 4.34028e+04s 43.3 17000
udfProperties 1.78460e-02s 0.0 17001
velocitySolve 1.68422e+04s 16.8 17002
rhs 1.88037e+03s 1.9 11.2 17002
preconditioner 1.17190e+03s 1.2 7.0 135246
pressureSolve 3.61099e+04s 36.1 17002
rhs 2.83055e+03s 2.8 7.8 17002
preconditioner 2.35338e+04s 23.5 65.2 88962
pMG smoother 1.84975e+04s 18.5 78.6 177924
pMG smoother 3.43554e+03s 3.4 14.6 177924
coarse grid 4.24077e+01s 0.0 0.2 88962
initial guess 2.57845e+03s 2.6 7.1 17002

projP : resNorm0 2.59e-02 resNorm 2.62e-03 ratio = 9.899e+00 6/8
P : iter 005 resNorm0 2.62e-03 resNorm 7.46e-05
UVW : iter 008 resNorm0 2.99e-03 resNorm 9.30e-07 divErrNorms 1.49e-10 1.52e-01
flowRate : uBulk0 1.00e+00 uBulk 1.00e+00 err 7.77e-16 scale 5.39812e-02
step= 17001 t= 1.01988000e+02 dt=6.0e-03 C= 2.93 elapsedStep= 5.92e+00s elapsedStepSum= 1.00143e+05s
projP : resNorm0 2.59e-02 resNorm 2.20e-03 ratio = 1.174e+01 7/8
P : iter 005 resNorm0 2.20e-03 resNorm 9.12e-05
UVW : iter 008 resNorm0 2.99e-03 resNorm 9.29e-07 divErrNorms 8.06e-11 1.52e-01
flowRate : uBulk0 1.00e+00 uBulk 1.00e+00 err 4.44e-16 scale 5.39671e-02

II - For the failed case, I re-Run again and gives:
hwloc/linux: Ignoring PCI device with non-16bit domain.
Pass --enable-32bits-pci-domain to configure to support such devices
(warning: it would break the library ABI, don't enable unless really needed).
hwloc/linux: Ignoring PCI device with non-16bit domain.
Pass --enable-32bits-pci-domain to configure to support such devices
(warning: it would break the library ABI, don't enable unless really needed).
UTC time: Mon Apr 29 14:26:23 2024

             __    ____  _____

____ ___ / /__ / __ / /
/ __ \ / _ \ / //// // /_ \
/ / / // // ,< / , // /
// // __///||// ||/____/ v23.0.0 (no sha)

MPI tasks: 1

reading par file ...
Initializing device
active occa mode: Serial

using NEKRS_HOME: /home/me/.local/nekrs
using NEKRS_CACHE_DIR: /home/me/nekRS/examples/turbPipePeriodic/.cache
using OCCA_CACHE_DIR: /home/me/nekRS/examples/turbPipePeriodic/.cache/occa/

skip building nekInterface (SIZE requires no update)
loading nek ... done

<<< [GENERAL]
<<< #verbose = true
<<< polynomialOrder = 7
<<< #startFrom = "restart.fld"
<<< stopAt = endTime
<<< endTime = 200
<<<
<<< dt = 6.0e-3
<<< timeStepper = tombo2
<<< subCyclingSteps = 2
<<<
<<< writeControl = simulationTime
<<< writeInterval = 20
<<<
<<< regularization = hpfrt + nModes=1 + scalingCoeff=10
<<< constFlowRate = meanVelocity=1.0 + direction=Z
<<<
<<< [PRESSURE]
<<< residualTol = 1e-04
<<<
<<< [VELOCITY]
<<< boundaryTypeMap = wall
<<< viscosity = -19000
<<< density = 1.0
<<< residualTol = 1e-06
<<<
<<< [CASEDATA]
<<< Re_tau = 550
<<< zLength = 6
<<<

<<< //
<<< // nekRS User Defined File
<<< //
<<< #include "udf.hpp"
<<<
<<< static dfloat Re_tau;
<<< static dfloat zLength;
<<< static dfloat dt0;
<<<
<<< #include "ci.inc"
<<<
<<<
<<< void userf(nrs_t *nrs, dfloat time, occa::memory o_U, occa::memory o_FU)
<<< {
<<< mesh_t *mesh = nrs->meshV;
<<< dfloat mue, rho;
<<< platform->options.getArgs("VISCOSITY", mue);
<<< platform->options.getArgs("DENSITY", rho);
<<< const dfloat RE_B = rho / mue;
<<< const dfloat DPDZ = 4 * (2 * Re_tau / RE_B) * (2 * Re_tau / RE_B);
<<<
<<< occa::memory o_FUz = o_FU + 2 * nrs->fieldOffset * sizeof(dfloat);
<<< platform->linAlg->fill(mesh->Nlocal, DPDZ, o_FUz);
<<< }
<<<
<<< void UDF_Setup0(MPI_Comm comm, setupAide &options)
<<< {
<<< platform->par->extract("casedata", "re_tau", Re_tau);
<<< platform->par->extract("casedata", "zlength", zLength);
<<<
<<< options.getArgs("CI-MODE", ciMode);
<<< if (ciMode) {
<<< ciSetup(comm, options);
<<< }
<<<
<<< if (platform->options.compareArgs("BUILD ONLY", "FALSE")) {
<<< double *const nek_cb_scnrs = (double *)nek::ptr("cb_scnrs");
<<< nek_cb_scnrs[0] = zLength;
<<< }
<<<
<<< platform->options.getArgs("DT", dt0);
<<< }
<<<
<<< void UDF_Setup(nrs_t *nrs)
<<< {
<<< if (platform->options.compareArgs("CONSTANT FLOW RATE", "FALSE")) {
<<< // assign function pointer to drive flow by constant mean pressure gradient
<<< udf.uEqnSource = &userf;
<<< }
<<< }
<<<
<<< void UDF_ExecuteStep(nrs_t *nrs, dfloat time, int tstep)
<<< {
<<< if (tstep < 5) {
<<< nrs->dt[0] = 0.25 * dt0;
<<< } else {
<<< nrs->dt[0] = dt0;
<<< }
<<<
<<< if (nrs->lastStep) {
<<< if (ciMode) {
<<< ciTestErrors(nrs, time, tstep);
<<< }
<<< }
<<< }
<<<

<<< #ifdef okl
<<<
<<< #include "turbPipe.oudf"
<<<
<<< #endif
<<< #ifdef okl
<<< void velocityDirichletConditions(bcData *bc){}
<<< void meshVelocityDirichletConditions(bcData *bc){}
<<< void velocityNeumannConditions(bcData *bc){}
<<< void pressureDirichletConditions(bcData *bc){}
<<< void scalarNeumannConditions(bcData *bc){}
<<< void scalarDirichletConditions(bcData *bc){}
<<< #endif
<<<

loading udf kernels ...
done (0.000407018s)
benchmarking hot kernels ...
JIT compiling kernels (this may take awhile if they are not in cache) ...
done (0.0805764s)

initializing linAlg ...
wdotp: 4.273e-03s

ENABLE GS COMM OVERLAP disabled

reading turbPipe.re2
hdr:#v002 7840 3 7840 hdr
[me-HP-Workstation-Desktop-PC] *** An error occurred in MPI_Win_create
[me-HP-Workstation-Desktop-PC] *** reported by process [3631546369,0]
[me-HP-Workstation-Desktop-PC] *** on communicator MPI COMMUNICATOR 5 DUP FROM 3
[me-HP-Workstation-Desktop-PC] *** MPI_ERR_WIN: invalid window
[me-HP-Workstation-Desktop-PC] *** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
[me-HP-Workstation-Desktop-PC] *** and potentially your MPI job)

7 replies

yslan Jun 12, 2024

Hi, What's the error message you got? and how do you launch the job?

JunaidAhmed2520 Jun 12, 2024

Hi, thanks for the reply. I had run the case using:

mpirun -np 2 nekrs --setup turbPipe.par

And this is the entire output on running this command in the turbPipePeriodic directory.

MPI tasks: 2

reading par file ...
Initializing device
active occa mode: Serial

using NEKRS_HOME: /users/ee/bt/ajunaid22/.local/nekrs
using NEKRS_CACHE_DIR: /users/ee/bt/ajunaid22/.local/nekrs/examples/turbPipePeriodic/.cache
using OCCA_CACHE_DIR: /users/ee/bt/ajunaid22/.local/nekrs/examples/turbPipePeriodic/.cache/occa/

skip building nekInterface (SIZE requires no update)
loading nek ... done

<<< [GENERAL]
<<< #verbose = true
<<< polynomialOrder = 7
<<< #startFrom = "restart.fld"
<<< stopAt = endTime
<<< endTime = 200
<<<
<<< dt = 6.0e-3
<<< timeStepper = tombo2
<<< subCyclingSteps = 2
<<<
<<< writeControl = simulationTime
<<< writeInterval = 20
<<<
<<< regularization = hpfrt + nModes=1 + scalingCoeff=10
<<< constFlowRate = meanVelocity=1.0 + direction=Z
<<<
<<< [PRESSURE]
<<< residualTol = 1e-04
<<<
<<< [VELOCITY]
<<< boundaryTypeMap = wall
<<< viscosity = -19000
<<< density = 1.0
<<< residualTol = 1e-06
<<<
<<< [CASEDATA]
<<< Re_tau = 550
<<< zLength = 6
<<<

<<< //
<<< // nekRS User Defined File
<<< //
<<< #include "udf.hpp"
<<<
<<< static dfloat Re_tau;
<<< static dfloat zLength;
<<< static dfloat dt0;
<<<
<<< #include "ci.inc"
<<<
<<<
<<< void userf(nrs_t *nrs, dfloat time, occa::memory o_U, occa::memory o_FU)
<<< {
<<< mesh_t *mesh = nrs->meshV;
<<< dfloat mue, rho;
<<< platform->options.getArgs("VISCOSITY", mue);
<<< platform->options.getArgs("DENSITY", rho);
<<< const dfloat RE_B = rho / mue;
<<< const dfloat DPDZ = 4 * (2 * Re_tau / RE_B) * (2 * Re_tau / RE_B);
<<<
<<< occa::memory o_FUz = o_FU + 2 * nrs->fieldOffset * sizeof(dfloat);
<<< platform->linAlg->fill(mesh->Nlocal, DPDZ, o_FUz);
<<< }
<<<
<<< void UDF_Setup0(MPI_Comm comm, setupAide &options)
<<< {
<<< platform->par->extract("casedata", "re_tau", Re_tau);
<<< platform->par->extract("casedata", "zlength", zLength);
<<<
<<< options.getArgs("CI-MODE", ciMode);
<<< if (ciMode) {
<<< ciSetup(comm, options);
<<< }
<<<
<<< if (platform->options.compareArgs("BUILD ONLY", "FALSE")) {
<<< double *const nek_cb_scnrs = (double *)nek::ptr("cb_scnrs");
<<< nek_cb_scnrs[0] = zLength;
<<< }
<<<
<<< platform->options.getArgs("DT", dt0);
<<< }
<<<
<<< void UDF_Setup(nrs_t *nrs)
<<< {
<<< if (platform->options.compareArgs("CONSTANT FLOW RATE", "FALSE")) {
<<< // assign function pointer to drive flow by constant mean pressure gradient
<<< udf.uEqnSource = &userf;
<<< }
<<< }
<<<
<<< void UDF_ExecuteStep(nrs_t *nrs, dfloat time, int tstep)
<<< {
<<< if (tstep < 5) {
<<< nrs->dt[0] = 0.25 * dt0;
<<< } else {
<<< nrs->dt[0] = dt0;
<<< }
<<<
<<< if (nrs->lastStep) {
<<< if (ciMode) {
<<< ciTestErrors(nrs, time, tstep);
<<< }
<<< }
<<< }
<<<

<<< #ifdef okl
<<<
<<< #include "turbPipe.oudf"
<<<
<<< #endif
<<< #ifdef okl
<<< void velocityDirichletConditions(bcData *bc){}
<<< void meshVelocityDirichletConditions(bcData *bc){}
<<< void velocityNeumannConditions(bcData *bc){}
<<< void pressureDirichletConditions(bcData *bc){}
<<< void scalarNeumannConditions(bcData *bc){}
<<< void scalarDirichletConditions(bcData *bc){}
<<< #endif
<<<

loading udf kernels ...
done (0.0171912s)
benchmarking hot kernels ...
JIT compiling kernels (this may take awhile if they are not in cache) ...
done (2.25973s)

initializing linAlg ...
wdotp: 3.654e-03s (local: 3.565e-03s)

ENABLE GS COMM OVERLAP disabled

reading turbPipe.re2
hdr:#v002 7840 3 7840 hdr
[gpu01:1074845] *** An error occurred in MPI_Win_create
[gpu01:1074845] *** reported by process [1883176961,1]
[gpu01:1074845] *** on communicator MPI COMMUNICATOR 5 DUP FROM 3
[gpu01:1074845] *** MPI_ERR_WIN: invalid window
[gpu01:1074845] *** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
[gpu01:1074845] *** and potentially your MPI job)
[gpu01:1074836] 1 more process has sent help message help-mpi-errors.txt / mpi_errors_are_fatal
[gpu01:1074836] Set MCA parameter "orte_base_help_aggregate" to 0 to see all help / error messages

This is more likely to be an openmpi error than a nekrs error.
On my laptop, OpenMPI 4.1.2 works

ME-20244 · 2024-04-30T08:21:14Z

ME-20244
Apr 30, 2024

Many Thanks! that helped me a lot.

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Problem with the example TurbPipePeriodic #467

{{title}}

Replies: 4 comments 16 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

Select a reply

Problem with the example TurbPipePeriodic #467

Replies: 4 comments · 16 replies

Chaggge Nov 21, 2022 Author

Chaggge Nov 22, 2022 Author

occa info : indicates the presence of GPU and OPENCL.

nvidia-smi gives for me:

stgeke Apr 26, 2024 Maintainer

stgeke Apr 29, 2024 Maintainer

Replies: 4 comments 16 replies

Chaggge Nov 21, 2022
Author

Chaggge Nov 22, 2022
Author

stgeke Apr 26, 2024
Maintainer

stgeke Apr 29, 2024
Maintainer