Building Deepbench with Accel-Sim

[ishitachaturvedi]

Hi,

I am trying to build DeepBench with Accel-Sim and failing.
Can you share the Deepbench makefile you use to build the benchmark?
Thank you!

[JRPan]

Have you checked the gpu-app-collection repo?

[rodhuega]

Do you have installed the right cudnn version for your cuda version and set it to the PATH?

[ishitachaturvedi]

Not yet. I will look into it.
Can you please keep the issue open till I set it up? Thanks!

[ishitachaturvedi]

What cudnn version is required for deepbench?

[rodhuega]

I think that it doesn't matter. The only think you need is to have a right installation of cuda with cudnn and all the things in the PATH. I had problems compiling deepbench and after that I was able to compile it and execute it.

[ishitachaturvedi]

Which libraries need to be there in PATH?
Have you had any success setting up Gunrock?

[rodhuega]

Cuda and cudnn. I don't know what is ginecólogo

[ishitachaturvedi]

Do any libraries need to be linked as static in the makefile?

[JRPan]

why not just use the one we provided?

[ishitachaturvedi]

I used the deepbench from gpu-app-collection using release-accelwattch branch.
An ldd on conv_bench gives-
linux-vdso.so.1 (0x00007fff5ad12000)
libcurand.so.10 => /usr/local/cuda-11/lib64/libcurand.so.10 (0x000014d023f0b000)
libcudnn.so.8 => /scratch/gpfs/ishitac/cudnn-lib/lib64/libcudnn.so.8 (0x000014d023ce3000)
libcudart.so.11.0 => /usr/local/cuda-11/lib64/libcudart.so.11.0 (0x000014d023a3f000)
libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x000014d0236b6000)
libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x000014d02349e000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x000014d0230ad000)
librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x000014d022ea5000)
libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x000014d022c86000)
libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x000014d022a82000)
libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x000014d0226e4000)
/lib64/ld-linux-x86-64.so.2 (0x000014d029e06000)

Should it not be pointing to accel-wattch libcudart?

[JRPan]

This looks fine. The setup_environment.sh will set LD_LIBRARY_PATH

[ishitachaturvedi]

I built conv_bench, but while I do see the executable, if I do ./conv_bench it gives Segmentation fault (core dumped)

My bashrc looks like this-
export PTX_SIM_MODE_FUNC=0
export PTX_SIM_MODE_DETAIL=1
export CUDA_INSTALL_PATH=/usr/local/cuda-11
export PATH=$CUDA_INSTALL_PATH/bin:$PATH
source /scratch/gpfs/ishitac/gpgpusim-codes/accel-sim-framework/gpu-simulator/setup_environment.sh

export CPATH=$CPATH:/scratch/gpfs/ishitac/local/include
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/scratch/gpfs/ishitac/local/lib
export CUDNN_PATH=/scratch/gpfs/ishitac/cudnn-lib/
export LD_LIBRARY_PATH=$CUDA_INSTALL_PATH/lib64:$CUDA_INSTALL_PATH/lib:/scratch/gpfs/ishitac/cudnn-lib/lib64

[ishitachaturvedi]

It is not invoking accel-sim but going to nvidia-cuda to run

[JRPan]

can you try manually source the setup_environment.sh?
I'm not sure about this but I think the later export in your bashrc overwrites what setup_environment.sh did.
export LD_LIBRARY_PATH=$CUDA_INSTALL_PATH/lib64:$CUDA_INSTALL_PATH/lib:/scratch/gpfs/ishitac/cudnn-lib/lib64

[ishitachaturvedi]

I manually sourced it and changed the line to export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$CUDA_INSTALL_PATH/lib64:$CUDA_INSTALL_PATH/lib:/scratch/gpfs/ishitac/cudnn-lib/lib64
I did make on source again, but it does not link to accel-sim in libcuda, it links to the nvcc library somehow

[ishitachaturvedi]

Interestingly I have this problem for cuda 11. When I go to CUDA 10.1 it links libcudart.so to the correct libcuda for accelsim. My cuda 10.1 does not have cublas so I am still sorting out the issue for Deepbench, but other benchmarks which dont require cublas are linking to accelsim now. Any ideas for why it is not working with cuda 11?

[ishitachaturvedi]

I managed to fix the linker error.
Now gpgpusim is linked with the binaries.
However I get the following error-
terminate called after throwing an instance of 'std::runtime_error'
what(): CUDNN failure: CUDNN_STATUS_NOT_INITIALIZED in cudnn_helper.h at line: 33
Any way to get around this? Thank you

[BrianQian1999]

Have you fixed that CUDNN_STATUS_NOT_INITIALIZED issue? I was stuck in that as well.

[JRPan]

Are you running PTX mode or trace mode? Deepbench is supported in trace mode in Accel-Sim

[BrianQian1999]

I was trying to deploy some workloads implemented in cuDNN (Deepbench, cuDNN_sample, etc.) in PTX mode. I replaced the libcudnn.so to the corresponding static library to enable PTX mode (as it does in the original GPGPU-sim). However, the same SEGF would occur in both Accel-sim and GPGPU-sim on my machine (CUDA11.0 + cuDNN8.5 + GCC9).

[JRPan]

Again, Deepbench is supported in trace mode in Accel-Sim. We have not looked into how to run PTX mode with cuDNN. Feel free to download the traces for Deepbench and try out the trace mode.

[BrianQian1999]

Thanks for the feedback. It seems that it would not be possible to run cuDNN in PTX mode after CUDA-8.

[JRPan]

*Not my paper. But thanks.

Yes, that was done with gpgpu-sim. Not Accel-sim. Here is some discussion on DeepBench and a response from the Author of Accel-Sim, Mahmoud.
gpgpu-sim/gpgpu-sim_distribution#212

Someone made it work a while ago. But since Accel-Sim went out, so we focused on Accel-Sim instead.

Good luck hacking!