Add switches for some functions to compile differently for host vs device

[Yurlungur]

PR Summary

This MR is motivated by two things:

• EOSPAC currently can't be built on HIP because EOSPAC functions are not device-capable, but we need PORTABLE_INLINE_FUNCTION markings.
• The Evaluate method in the base class needs to work on both host and device, but the constexpr method did not appear to fully function as desired and we were seeing segfualts with tables in riot.

Here we use the macro

#if defined(__CUDACC__) || defined(__HIPCC__)
/* My device code */
#else
/* My host code */
#endif // ON DEVICE

to create switches for separate implementations or prototypes depending on whether a function is called on either host or device. This unfortunately breaks the portability layer in ports-of-call but I don't see a way to hide #if blah inside an abstraction layer. So this is what we get.

@rbberger at your convenience, please try enabling EOSPAC for the RZadams CI for this MR and see if it kills it. There's a WIP PR on re-git you can use.

@pdmullen plase try to reproduce the segfault in riot that you experienced. Lets see if it still happens.

PR Checklist

• Adds a test for any bugs fixed. Adds tests for new features.
• Format your changes by using the make format command after configuring with cmake.
• Document any new features, update documentation for changes made.
• Make sure the copyright notice on any files you modified is up to date.
• After creating a pull request, note it in the CHANGELOG.md file.
• LANL employees: make sure tests pass both on the github CI and on the Darwin CI

If preparing for a new release, in addition please check the following:

• Update the version in cmake.
• Move the changes in the CHANGELOG.md file under a new header for the new release, and reset the categories.
• Ensure that any when='@main' dependencies are updated to the release version in the package.py

[Yurlungur]

Correction, unfortunately this alternate implementation trick does not work for function prototypes. I had to split Evaluate into two functions, an EvaluateHost and an EvaluateDevice.

[Yurlungur]

@rbberger @pdmullen this is now ready for review. EvaluateDevice now works and EOSPAC should how build for CUDA and HIP.

[dholladay00]

I don't recall the exact error, but I recall making this a forwarding reference fixed some issues I had seen. Probably worth trying.

[Yurlungur]

I can try that.

[Yurlungur]

Actually the current version works, so if you're happy with the split function signature (as discussed below) lets keep it as is. However, if you really dislike the split function signature I can take a look and try games with forwarding references, etc.

[dholladay00]

Correction, unfortunately this alternate implementation trick does not work for function prototypes. I had to split Evaluate into two functions, an EvaluateHost and an EvaluateDevice.

So does this mean codes that use evaluate are going to have a bunch of #if stuff to call the right thing? Something doesn't seem right.

[Yurlungur]

Correction, unfortunately this alternate implementation trick does not work for function prototypes. I had to split Evaluate into two functions, an EvaluateHost and an EvaluateDevice.

So does this mean codes that use evaluate are going to have a bunch of #if stuff to call the right thing? Something doesn't seem right.

I'm not sure I understand what you mean. EvaluateHost and EvaluateDevice both work regardless of the target architecture you compiled for. But the model I'm imagining now is that EvaluateHost is called when you want to call your own portableFor and EvaluateDevice is called from within a portableFor

[Yurlungur]

The tests we expect to pass are all now passing. I'd like to merge this, with a note that we still have a segfault on HIP which needs to be resolved.

[Yurlungur]

@jhp-lanl @dholladay00 any objections?

[pdmullen]

LGTM! My only comment is just to document here for future self that the associated symptom was not a segfault, but rather corrputed data. This should close Issue #393 (which I think is already linked to this PR).

[jhp-lanl]

@jhp-lanl @dholladay00 any objections?

Looks good 👍