KOKKOS_FUNCTION-annotated for_each and transform_reduce

[blegouix]

Closes #172

[blegouix]

@tpadioleau can you take a look at the two issues I mention in the first message ? The second one just seems to be an annoying bug with the support of [[maybe_unused]] in hipcc but the first one is more worrying, it seems the loop https://github.com/CExA-project/ddc/pull/708/files#diff-46765d278566dca059db5ef8ac1fc5e486964e46a1f7303302c7b7d80b9b4ab0R47 does not iterate correctly with nvcc

[tpadioleau]

I don't know if i will have time to have a closer look this week. You could look for the differences with the implementation in the first PR, i think the tests were passing

[blegouix]

I think I have the same implementation as the closed branch. But you don't have 2D test for nested for_each, and I think the bug with nvcc appears only for 2D+ cases.

[tpadioleau]

and I think the bug with nvcc appears only for 2D+ cases.

Did you test ?

[blegouix]

I just tested it and you were right to ask because the 1D case does not work to. I tried you branch and I don't get the bug. This looks very weird to me, either I am missing something obvious, either something broke somewhere since March. If you have any idea I'd take it, otherwise I will take my time for this because it does not look easy to debug.

[blegouix]

This static_cast<int> is a nonsense workaround since Element aliases std::size_t. Without it the loop does not iterate with nvcc in the new tests.

[tpadioleau]

What does it print if you remove the cast ?

[blegouix]

begin[I] is 0 as expected 🤷 but still it does not iterate.

[blegouix]

(if I print something inside the loop it appears only once)

[tpadioleau]

If it can print something inside the loop it means it can iterate once ?

What is the value of end[I] ?

Have you tried to run in debug ?

[blegouix]

sorry what do you want me to print inside the loop ? In my tests i was just printing a string like "test", not "%i" involved

[tpadioleau]

You could print ii ?

[blegouix]

Kokkos::printf("%llu\n", ii); prints 0

[blegouix]

I do reproduce with g++-12, but not with -G, so it seems to be an optimization issue indeed.

[blegouix]

I don't see anything with compute-sanitizer (tried every --tool)

[blegouix]

I have found another bug. This is not visible in the tests (should I add a test which reproduces it ?). I have a use-case where the annotated_transform_reduce returns 0 with CUDA (it should not). Here is a fix:

result = reduce(
        result,
        annotated_transform_reduce_serial(
                domain,
                neutral,
                reduce,
                transform,
                dcoords...,
                ii));

Must be replaced with:

T tmp;
Kokkos::atomic_store(
      &tmp,
      annotated_transform_reduce_serial(
                domain,
                neutral,
                reduce,
                transform,
                dcoords...,
                ii));
Kokkos::atomic_store(&result, reduce(result, tmp));

(I don't understand what is the problem as result is a local per-thread variable though)

ddc/include/ddc/transform_reduce.hpp

Line 84 in 73ee089

result = reduce(

[tpadioleau]

All of that is weird, can you provide more information about your environment ?

• the hardware, cpu and gpu
• the version of your compiler
• the version of the libraries, cmake, cuda, kokkos...
• the complete cmake command line
• anything you think relevant

Also do all other DDC tests pass ?
Do kokkos tests pass ?

It feels like the compiler optimized away some function calls

[blegouix]

• Ubuntu 24
• Intel(R) Xeon(R) Gold 6226R + V100S
• nvcc 12.6 + g++13.3
• cmake 3.28, cuda 12.6, the Kokkos is the DDC submodule up-to-date with the main DDC branch.
• cmake -DCMAKE_CXX_COMPILER=$KOKKOS_BIN/nvcc_wrapper -DCMAKE_BUILD_TYPE=Debug -DKokkos_ENABLE_CUDA=ON -DKokkos_ARCH_VOLTA70=ON -DCMAKE_CXX_FLAGS="-Wfatal-errors" ..

All DDC tests pass. I don't know how to compile the Kokkos tests.

I will try quite soon on a 1070Ti with mostly the same software configuration. I will try to dig in compute-sanitizer also. Maybe if you can see if my first problem reproduces on Ruche or Persée by going in this branch and compiling with or without the static_cast<int> it may be helpful ? I agree all this is weird and i'm feeling nvcc is capricious, but it is possible that I am missing something.

[tpadioleau]

From the table https://docs.nvidia.com/cuda/cuda-installation-guide-linux/#system-requirements you are one minor version above the maximum tested host compiler version, i.e. 13.2.

Can you try with a different compiler ?