AMDGPU downgrades Waterlily #145
Comments
|
Hi Simon, thanks for reporting this. What ROCm version do you have installed in your system? I was recently able to run WaterLily on an AMD GPU, but because the ROCm version in that system was quite low (5.1.1) I had to force AMDGPU.jl to use its 0.4.15 version. Maybe something similar to this is happening? |
|
That wasn't the reason, I just updated all my drivers including rocm (5.7 + 6.1). I also don't think Pkg can even install different AMDGPU version based on available drivers, if I'm not mistaken. You can actually reproduce it with this minimal setup (Julia 1.10.4): ]activate --temp
]add AMDGPU@0.9.6 WaterLily@1.1.0 |
|
What happens when you try to not specify the version, ie This works fine on my system, even though it installs AMDGPU v0.6.1. So maybe we should update our version compatibilities. |
|
Can you please try my temporary fix in the |
|
Yes, this seems to work :) |
|
Good! I will create the PR and (re)activate some of the compatibilities. If that still works, then we can merge it :) |
|
It does not work though :( I guess some kernel isn't setup correctly, although I would expect CUDA.jl to also caugh on something like |
|
You can try the following MWE using WaterLily
using AMDGPU
function tgv(p, backend; Re=1600, T=Float32)
L = 2^p; U = 1; κ=π/L; ν = 1/(κ*Re)
function uλ(i,xyz)
x,y,z = @. xyz/L*π # scaled coordinates
i==1 && return -U*sin(x)*cos(y)*cos(z) # u_x
i==2 && return U*cos(x)*sin(y)*cos(z) # u_y
return 0. # u_z
end
Simulation((L, L, L), (0, 0, 0), 1/κ; U=U, uλ=uλ, ν=ν, T=T, mem=backend)
end
function main()
sim = tgv(5, ROCArray)
sim_step!(sim)
end
main()I can correctly run this using ROCm/5.1.1, WaterLily.jl/1.1, AMDGPU.jl/0.4.15 on a Radeon Instinct MI50 32GB. |
|
That actually kills the julia session 😓 |
|
That's unfortunate. This week I will get access to LUMI, which I presume has a more updated ROCm version. I will look into it and let you know how that goes. |
|
The PR doesn't actually fix this issue. |
|
Hey Simon, I have successfully run WaterLily on LUMI today (AMD MI250x), where ROCm/5.2.3. There are a couples of fixes for flows with bodies, but this example worked out of the box. This is my project environment right now And I get the following AMDGPU.jl info julia> AMDGPU.versioninfo()
[ Info: AMDGPU versioninfo
┌───────────┬──────────────────┬───────────┬──────────────────────────────────────┐
│ Available │ Name │ Version │ Path │
├───────────┼──────────────────┼───────────┼──────────────────────────────────────┤
│ + │ LLD │ - │ /opt/rocm/llvm/bin/ld.lld │
│ + │ Device Libraries │ - │ /opt/rocm/amdgcn/bitcode │
│ + │ HIP │ 5.2.21153 │ /opt/rocm-5.2.3/lib/libamdhip64.so │
│ + │ rocBLAS │ 2.44.0 │ /opt/rocm-5.2.3/lib/librocblas.so │
│ + │ rocSOLVER │ 3.18.0 │ /opt/rocm-5.2.3/lib/librocsolver.so │
│ + │ rocALUTION │ - │ /opt/rocm-5.2.3/lib/librocalution.so │
│ + │ rocSPARSE │ - │ /opt/rocm-5.2.3/lib/librocsparse.so │
│ + │ rocRAND │ 2.10.5 │ /opt/rocm-5.2.3/lib/librocrand.so │
│ + │ rocFFT │ 1.0.27 │ /opt/rocm-5.2.3/lib/librocfft.so │
│ + │ MIOpen │ 2.17.0 │ /opt/rocm-5.2.3/lib/libMIOpen.so │
└───────────┴──────────────────┴───────────┴──────────────────────────────────────┘
[ Info: AMDGPU devices
┌────┬──────┬────────────────────────┬───────────┬────────────┐
│ Id │ Name │ GCN arch │ Wavefront │ Memory │
├────┼──────┼────────────────────────┼───────────┼────────────┤
│ 1 │ │ gfx90a:sramecc+:xnack- │ 64 │ 63.984 GiB │
└────┴──────┴────────────────────────┴───────────┴────────────┘I cannot try anything with ROCm/6.x though... which could be the problem. |
|
I just found out that AMDGPU can be used on WSL2 :-O julia> using AMDGPU
julia> function tgv(p, backend; Re=1600, T=Float32)
L = 2^p; U = 1; κ=π/L; ν = 1/(κ*Re)
function uλ(i,xyz)
x,y,z = @. xyz/L*π # scaled coordinates
i==1 && return -U*sin(x)*cos(y)*cos(z) # u_x
i==2 && return U*cos(x)*sin(y)*cos(z) # u_y
return 0. # u_z
end
Simulation((L, L, L), (0, 0, 0), 1/κ; U=U, uλ=uλ, ν=ν, T=T, mem=backend)
end
tgv (generic function with 1 method)
julia> function main()
sim = tgv(5, ROCArray)
sim_step!(sim)
end
main (generic function with 1 method)
julia> main()
ERROR: Scalar indexing is disallowed.
Invocation of setindex! resulted in scalar indexing of a GPU array.
This is typically caused by calling an iterating implementation of a method.
Such implementations *do not* execute on the GPU, but very slowly on the CPU,
and therefore should be avoided. |
|
Ok, that's progress! I think the only problem is now that you might be running Julia with single thread instead of |
|
Yay, that makes the example work on WSL2 ubuntu! I do notice now, that there seems to be a mismatch in the HIP version on windows: WindowsWSL ubuntu |
|
Great! And yes, that mismatch might have caused your original error. For that, you could submit an issue on AMDGPU.jl I guess. So, is this issue resolved now? :) |
|
Yes, thank you :) |
|
I just realized, that the error for AMDGPU on windows could come from this: |
|
Ah, good point. Should we try adding |
|
It's not just |
|
We can run some benchmarks I guess, but I do not have access to a Windows with AMD GPUs. You could try playing with https://github.com/WaterLily-jl/WaterLily-Benchmarks. I can help if necessary :) |
I wanted to try out Waterlily on an AMD gpu with
AMDGPU.jl, but addingAMDGPUto the project downgrades Waterlily significantly:Any help to run this on my 7900xtx would be appreciated :)
The text was updated successfully, but these errors were encountered: