Changing Libcudacxx's testing behavior.

[wmaxey]

Libcudacxx Test Architecture Changes

Our tests in libcudacxx currently have several issues. If a test excludes certain SMs, it won't be built, even if there are architectures that do support the test. This means that tests ultimately can deny large SM matrix builds from giving coverage. Worse, testing requires a CUDA device, I think it's reasonable to allow testing purely on CPU if a GPU or incompatible GPU is present.

Possible solution

Ideally libcudacxx tests should build every SM possible and simply exclude the incompatible SMs. This means if a test has no effect on SM70 and below, then SM70- should be no-ops. The kernels can be empty, or not launched. This allows everything else to build, CPU tests to execute, and if possible still launch on supported SMs.

An example process for lit with a test file that excludes SMs.

// MINIMUM_SM: 90
#include <cuda/ptx>

int main() {
    // Do SM90 stuff...
    // Tests without host code components already guard for SM_90.
}

flowchart TD;
    start[Execute lit]
    parse(Parse test keywords)
    defines(Preprocess file or NVCC arguments)
    build(Execute nvcc)
    compile(Compile test)

    subgraph Build Pass
    start   -- lit -sv -Dcompute_archs=60,70,80,90 --> parse;
    parse   -- Find test flag `MINIMUM_SM` --> defines
    defines -- See option 1, 2, or 3 --> build
    build   -- nvcc ... inject flags for build ... --> compile
    end

Loading

The above requires changes to parsing in lit. This is easily handled by adding a new parser in our config. However rejecting incompatible SMs in main or blocking includes in test.pass.cpp will likely require substantial changes. Possible options are:

1. We inject some architecture checking macros and dummy kernels to replace main. We could make lit inject code via a temporary file to accomplish this.

// preinclude
#if ARCH < TEST_MINIMUM_SM
# define TEST_SKIP_MAIN
// Emit dummy kernel for main
int dummy() {}
#endif // ARCH < TEST_MINIMUM_SM

#if !defined(TEST_SKIP_MAIN)
// begin test.pass.cpp
...
int main() {
    // Do stuff
}
// postinclude
#endif

2. We make every test aware of its own architecture requirements OR inject the check similar to option 1. We won't need any lit parsing changes, but instead test or injections to accomplish this.

int main() {
    // Injected or manually added to every test
    NV_IF_TARGET(PROVIDES_SM90, (
        // Do stuff
    ))
}

3. We prevent building for architectures that are not supported. Requires the least changes. Ideally still tests CPU codepaths.

nvcc -gpu=sm_60,sm_70,sm_80,sm_90 -> nvcc -gpu=sm_90

These are just a couple ideas of ways we could try to solve the problem, I'm sure there are more and would like to field those as well.

[ahendriksen]

I wanted to get a feel for how expensive option 2 would be, as it requires some rewriting of the test. I grepped for "UNSUPPORTED pre-sm-". There are 253 hits.

• SM60: 73 hits
• SM70: 154 hits (mostly atomics).
• SM80: 5 hits
• SM90: 21 hits

I fear option 1 and 2 might not work very well with the atomics and barrier headers (the 154 tests for SM70), because they are simply guarded at include time by __CUDA_MINIMUM_ARCH__. So making those tests work well would require conditionally including cuda/atomics and friends, based on the architecture.

My gut instinct would be to rewrite the 26 tests that rely on SM80 or SM90 (i.e. option 2) and leave the SM60 and SM70 tests as they are.

[jrhemstad]

Refactoring the atomic header to not #error just from including it based on the arch is one of the first things I want to do after we decide on a redesigned multi-arch testing scheme.

[wmaxey]

Yep, I agree with Jake. I believe it's possible for us to make <atomic> work as an include, and instead just emit errors upon invalid device code instantiations.

[leofang]

I would add <cuda/barrier> to this list. It should be include-able on all archs, and emit errors for instantiations. This causes a lot of stress to CuPy that Maxwell/Pascal users are mad at us (cupy/cupy#8200) ¹, simply because I wasn't aware that this header cannot be included before CC 7.0. The header name didn't say I cannot, and practically no one has a CI that runs tests on Maxwell. No way to catch it.

Footnotes

1. Quote from one user: "Or do I need to resign to buying a new gpu?" ↩

[jrhemstad]

Option 3 is a non-starter imo. A key requirement is our testing should reflect how users use our code, and that means compiling a TU for multiple architectures. Filtering out a request to compile multiple archs to actually compile for one would be counter to that goal.

All other things being equal, option 2 would be my preference because it most closely mirrors what user code would look like. That said, it would be a lot of manual work.

Could we combine options 1&2 by preprocessing the header and adding another wrapper around the fake_main that injects the NV_IF_TARGET logic based on the UNSUPPORTED: keyword?