Adding the FFTW library to PeleC

[RSuryaNarayan]

Hello all,

I am trying to add FFTW to PeleC to compute Fourier transforms after every time step by leveraging the PeleC::problem_post_timestep() function in prob.cpp.

I've fleshed out something similar to the pc_initdata() in prob.H to make those same function arguments in pc_initdata() available. The exact details can be found here.

I'd need the FFT functionalities inside my local case files next so I can compute the Fourier transform (which is where I find it tough to navigate and hope to get some tips). I've installed FFTW and its currently sitting inside my /usr/local/lib. I don't get any errors when I do #include<fftw3.h> inside prob.H but when I try using some of the functions of the library like fft_plan or fft_execute, the linker isn't able to find the necessary object files, which makes it throw the error:

/home/suryanarayan/Desktop/PeleC_cases/fourier/prob.H:353: undefined reference to `fftw_plan_dft_r2c_1d'
/usr/bin/ld: /home/suryanarayan/Desktop/PeleC_cases/fourier/prob.H:354: undefined reference to `fftw_execute'
collect2: error: ld returned 1 exit status
make: *** [/home/suryanarayan/Pele/PeleC/Submodules/AMReX/Tools/GNUMake/Make.rules:55: PeleC2d.gnu.MPI.ex] Error 1

I'd like inputs on how I can couple FFTW with PeleC. My guess is making changes to the Make.package locally to include those external headers and cpp of the library but is there a better way to do it?

Any input is greatly appreciated! Thanks a lot!

[hsitaram]

To couple any other software with pelec, you could add the location of headers and shared libraries in the GNUMakefile.

add
INCLUDE_LOCATIONS += path-to-headers
LIBRARIES += -Lpath-to-lib -l(fftw-i-think)
LIBRARIES += -Wl,-rpath=path-to-lib

alternatively, do look at this in amrex tutorials where SWFFT is used - https://github.com/AMReX-Codes/amrex-tutorials/blob/eefdf928d74bf9eb6dfd931ca75e1af3a84f0e11/ExampleCodes/SWFFT/SWFFT_poisson/GNUmakefile#L17

[RSuryaNarayan]

Thanks @hsitaram for these ideas. After taking a look at SWFFT, I am wondering if I'd be reinventing the wheel by coding up 2-D FFTs in post_timestep. Considering SWFFT is again an AMReX-derived code, could I use its functionalities inside of Pele?

[hsitaram]

I pretty sure you should be able to incorporate SWFFT into pele. The code available in amrex/Src/Extern and the example I pointed to should help. Although, I am curious to know how you would do a global fft when an mpi rank only has access to a couple of boxes. Were you thinking of gathering sums on each process?

[RSuryaNarayan]

I didn't anticipate any of the MPI complications as of now. Thanks for pointing that out. I was simply planning to pass the plotfile data in place of the "phi(x,y)" function in the SWFFT_Test_F.F90 file (line 13) inside of amrex-tutorials/Example Codes/SWFFT/SWFFT_Test. I would like the same functionality that SWFFT_Test provides for any scalar variable inside the Pele plotfile. There are additional complications due to non-square and non-periodic domains..but for starters a simple FFT of a 2-D Pele plotfile is what I am trying as of now. Hope that makes sense. Can take it over email if necessary.

[drummerdoc]

We did this in the past in order to get turbulence spectra, both for Pele combustion and for astrophysics. I'll ping a few people to see if that code is available somewhere and what shape it's in. In the meantime, have a look in PeleAnalysis in the Turbulence_analysis folder and see if this will get you at least partway to where you want to go.

[RSuryaNarayan]

@drummerdoc Thanks a ton for this! I did come across the turbulence analysis folder in PeleAnalysis and will keep you in the loop on if that gets my job done! thanks a ton once again. The challenge however I am facing is to extend the FFT/DFT to rectangular non-periodic domains. I guess this is doable once I get a hang of the code

[RSuryaNarayan]

@drummerdoc The AmrDeriveSpectrum code is precisely what I am looking for (in addition to the spectrum I am also looking at filling the FFT into a MultiFAB and getting it output as a plotfile just like SWFFT_test).

But it looks like that code works for any plot file in 3d (even non-periodic rectangular ones) and not 2d. I get some fftw error

fft: fftw_malloc(out of memory)