Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

HEFFTE support in compressible FHD + Advanced diagnostics for compressible turbulence #146

Open
wants to merge 50 commits into
base: main
Choose a base branch
from

Conversation

isriva
Copy link
Contributor

@isriva isriva commented Dec 9, 2023

This major PR now includes the capability to do distributed FFT (using HEFFTE). This is currently implemented for compressible staggered turbulence codes, but can be easily ported to other codes. HEFFTE works with the three major backends: FFTW, cuFFT and rocFFT for usage on CPUs, and NVIDIA and AMD GPUs. Some other developments in this PR include:

  1. Spectral decomposition of velocity field into solenoidal and dilatational components and transform back to real space.
  2. Advanced diagnostics for compressible turbulence including distribution functions for various hydrodynamic and thermodynamics quantities, along with analysis of solenoidal and dilatational components of the velocity field separately.

Compile-time options: This PR now enables several more compile time options described below:

  1. DO_TURB: This option switches on/off the compilation of turbulence-specific codes -- not needed if additional turbulence stats and turbulent forcing is not desired.
  2. USE_HEFFTE_FFTW: Do distributed FFT using HeFFTe (https://github.com/icl-utk-edu/heffte) for turbulence statistics using FFTW backend (need FFTW libraries)
  3. USE_HEFFTE_CUFFT: Do distributed FFT using HeFFTe (https://github.com/icl-utk-edu/heffte) for turbulence statistics using cuFFT backend (needs NVIDIA cuFFT libraries)
  4. USE_HEFFTE_ROCFFT: Do distributed FFT using HeFFTe (https://github.com/icl-utk-edu/heffte) for turbulence statistics using rocFFT backend (needs AMD rocFFT libraries)

It is possible to do various mixes, such as CUDA compilation of the code using USE_CUDA=TRUE and still FFTW backend using USE_HEFFTE_FFTW=TRUE etc., as long as the required libraries are available.

Distributed FFTs using USE_HEFFTE_FFTW=TRUE, USE_HEFFTE_CUFFT=TRUE or USE_HEFFTE_ROCFFT=TRUE requires installing HeFFTe (https://github.com/icl-utk-edu/heffte). Please follow the steps below:

  1. Clone forked heffte repo at the same directory level as FHDeX using: git clone https://github.com/isriva/heffte.git
  2. Compile in exec/compressible_stag using any of the HEFFTE backend flags described above
  3. Compiling on OLCF Frontier (https://docs.olcf.ornl.gov/systems/frontier_user_guide.html) requires loading specific modules and providing appropriate linker flags. Please run the bash utility build_frontier.sh in exec/compressible_stag to compile on Frontier. One can changes some compile time flags such as DO_TURB and MAX_SPECIES

@isriva isriva requested a review from ajnonaka December 9, 2023 00:17
@isriva
Copy link
Contributor Author

isriva commented Dec 11, 2023

@ajnonaka please check the detailed PR description on how to compile using HEFFTE on various machines.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants