how to compute 1d profile along an axis?

[BenWibking]

I'd like to compute the average of a derived state variable along the z-axis (in 3D Cartesian coordinates) each timestep.

Should I write my own GPU reduction functions to produce a 1D profile on each GPU and then perform a parallel sum of the profiles? Or is there a clever way to do this with the existing MultiFab functions?

[zingale]

I agree this would be useful to have in general.

We have something in MAESTROeX to do this, but that is probably not general.

[BenWibking]

Could you point me to where in MAESTROeX you do this? That might be enough to hack something together.

A general API would be great. For my application, I only need to compute a profile of a cell-local function of the state multifab. (This profile is then used to compute some source terms to maintain hydrostatic equilibrium.)

[zingale]

it is essentially what the average process does -- we need to average lots of quantities down to the 1-d base state (and map back):

https://github.com/AMReX-Astro/MAESTROeX/blob/main/Source/MaestroAverage.cpp

[BenWibking]

What does this comment mean: Atomic::Add is non-deterministic on the GPU. Is this because the summation order is undefined, so the lack of floating-point associativity makes it non-deterministic?

[ajnonaka]

Yes

[ajnonaka]

This MAESTROeX example may not be the easiest to follow since we use a fancy BaseState class for 1d quantities. I think I may have something else, let me dig.

[ajnonaka]

In https://github.com/AMReX-FHD/FHDeX.git
FHDeX/src_common/ComputAverages.cpp
I have a non-GPU version "WriteHorizontalAverage()"
I'm not sure how easy it is to make this work on a GPU without race conditions. Maybe cross-referencing what is being done in MAESTROeX is the answer...

[BenWibking]

It looks like the MAESTROeX code uses amrex::HostDevice::Atomic::Add on the GPU. Is there still a race condition in that case?

[WeiqunZhang]

Does this function do what you want? https://amrex-codes.github.io/amrex/doxygen/namespaceamrex.html#a882a700cf7fab66ee3ce8de37de3ef73

[BenWibking]

For a non-AMR simulation, yes.

[BenWibking]

Although with AMR, I could compute it on the whole hierarchy of multifabs and then average down. I wanted to avoid the extra memory usage, but it's probably not going to be a limiting factor in this case.