From f9f6ea607e363a24c2d6f8f09d6786ffd811f409 Mon Sep 17 00:00:00 2001 From: Michael Zingale Date: Thu, 4 Apr 2024 08:27:22 -0400 Subject: [PATCH 1/2] remove references to Fortran in the docs --- sphinx_docs/source/addproblem.rst | 24 ++-- sphinx_docs/source/faq.rst | 15 +- sphinx_docs/source/getting_started.rst | 28 ++-- sphinx_docs/source/gpu.rst | 189 +------------------------ sphinx_docs/source/initial_models.rst | 6 +- sphinx_docs/source/introduction.rst | 2 +- sphinx_docs/source/makefiles.rst | 12 +- sphinx_docs/source/mg.rst | 5 + sphinx_docs/source/param_intro.rst | 25 +--- sphinx_docs/source/preface.rst | 2 +- 10 files changed, 44 insertions(+), 264 deletions(-) diff --git a/sphinx_docs/source/addproblem.rst b/sphinx_docs/source/addproblem.rst index 808bdd3f0..cae9db9e4 100644 --- a/sphinx_docs/source/addproblem.rst +++ b/sphinx_docs/source/addproblem.rst @@ -110,10 +110,11 @@ By default, some of the runtime parameters are listed in PROBIN_PARAMETER_DIRS := . -They are parsed at compile time and the file ``extern.F90`` -is written and compiled. This is a Fortran module that holds the values of -the runtime parameters and makes them available to any routine via -``probin_module``. +These are combined with the runtime parameters specified by Microphysics +and parsed at compile time. A C++ header and source file is written +(``extern_parameters.cpp`` and ``extern_parameters.H``) that define +the parameters and their defaults. + The final line in the GNUmakefile includes the rules to actually build the executable. @@ -148,15 +149,14 @@ Each line in the ``_cpp_parameters`` file has the form:: *parameter* *data-type* *value* *need in Fortran?* -where *parameter* is the name of the runtime parameter, -*data-type* is one of {string, Real, int, bool}, -the *value* specifies the default value for the runtime parameter, -and *need in Fortran?* is marked *y* only if that parameter is -used in Fortran. Comments are indicated by a ‘#’ character and are +where *parameter* is the name of the runtime parameter, *data-type* is +one of {string, Real, int, bool}, the *value* specifies the default +value for the runtime parameter. A fourth column, *need in Fortran?*, +is no longer used. Comments are indicated by a ‘#’ character and are used to produce documentation about the available runtime parameters. -For the documentation, runtime parameters are grouped together -in the ``_cpp_parameters`` file into categories. The category headings -are defined by comments in the ``_cpp_parameters`` file and any comments +For the documentation, runtime parameters are grouped together in the +``_cpp_parameters`` file into categories. The category headings are +defined by comments in the ``_cpp_parameters`` file and any comments following that heading are placed into that category. At runtime, the default values for the parameters can be overridden diff --git a/sphinx_docs/source/faq.rst b/sphinx_docs/source/faq.rst index 81d8ca85f..76bf9f8cc 100644 --- a/sphinx_docs/source/faq.rst +++ b/sphinx_docs/source/faq.rst @@ -5,18 +5,9 @@ Frequently Asked Questions Compiling ========= -#. *What version of the Fortran standard is needed?* - - Some features of Fortran 2003 are used, in particular, the - ISO_C_BINDING feature of Fortran 2003 is needed to define a long - integer type for some MPI operations in Fortran. This is supported - by most Fortran compilers even if they don’t support the entire - Fortran 2003 standard. - - We also rely on the Fortran 95 standard that specifies that any - local allocated arrays are automatically deallocated when a - subroutine ends. Fortran 90 does not do this. Most - MAESTROeX routines rely on this Fortran 95 feature. +#. *What version of the C++ is needed?* + + We use C++17. Most modern compilers support the features we need. #. *The code doesn’t compile, but complains right away that there diff --git a/sphinx_docs/source/getting_started.rst b/sphinx_docs/source/getting_started.rst index 064f37e29..7df465f7f 100644 --- a/sphinx_docs/source/getting_started.rst +++ b/sphinx_docs/source/getting_started.rst @@ -9,12 +9,11 @@ and how to look at the output. Requirements ============ -MAESTROeX requires a C++ compiler that supports the C++17 standard, a -Fortran compiler that supports the Fortran 2003 standard, and a C compiler -that supports the C99 standard. Several compiler suites are supported, -including GNU, Intel, PGI and Cray. GNU Make (>= 3.82) is also required, -as is Python (>= 3.6) and standard tools available in any Unix-like -environments (e.g., Perl and ``sed``). +MAESTROeX requires a C++ compiler that supports the C++17 standard and +a C compiler that supports the C99 standard. Several compiler suites +are supported, including GNU, Intel, LLVM and Cray. GNU Make (>= 3.82) +is also required, as is Python (>= 3.9) and standard tools available +in any Unix-like environments (e.g., Perl and ``sed``). For running in parallel, an MPI library and/or OpenMP is required. For running on GPUs, CUDA 11 or later is required (see :ref:`sec:gpu` for @@ -154,7 +153,7 @@ paper 3. - ``COMP := gnu`` - This option specifies the gnu compiler suite (g++/gfortran). + This option specifies the gnu compiler suite (e.g., g++). We will use gnu, which is the preferred compiler suite for MAESTROeX. Specifying this compiler will automatically pull in the compiler settings as specified in ``AMREX_HOME/Tools/GNUMake/Make.defs``. @@ -307,19 +306,10 @@ related development packages (e.g. libXpm-devel). AmrPostprocessing scripts ------------------------- -Several useful analysis scripts (written in Fortran 90) can be found -in ``amrex/Tools/Postprocessing/F_Src/``. The ``GNUmakefile`` there -needs to be edited to indicate which of the tools to build. For -example, to extract the density along a line from the center of a -plotfile, ``plt00200``, in the :math:`y`-direction:: +Several useful analysis scripts can be found +in the ``amrex-astro-diag`` project: - fextract.Linux.gfortran.exe -d 2 -v "density" -p plt00200 - -These routines are described in § :ref:`sec:analysis`. - -There is also a python visualization method in -``AmrPostprocessing/python``. This is described -in § :ref:`sec:vis:python`. +https://github.com/amrex-astro/amrex-astro-diag VisIt ----- diff --git a/sphinx_docs/source/gpu.rst b/sphinx_docs/source/gpu.rst index d97eaf992..8e1395ffc 100644 --- a/sphinx_docs/source/gpu.rst +++ b/sphinx_docs/source/gpu.rst @@ -12,9 +12,8 @@ Note that currently MAESTROeX only supports NVIDIA GPUs. Requirements ============ -Since MAESTROeX uses primarily CUDA C++ and CUDA Fortran, -it requires a recent version of CUDA (e.g., >= 9) and the device -must have compute capability of >= 6. +MAESTROeX has only been tested with NVIDIA/CUDA. In theory AMD/HIP +should work. .. _sec:gpubuild: @@ -33,17 +32,6 @@ not compatible with building with CUDA. ``USE_CUDA = TRUE`` and ``USE_OMP = TRUE`` will fail to compile. However, you may use MPI with CUDA for additional parallelization. -Only the IBM and PGI compilers support CUDA Fortran, so the compiler should be set as: - -:: - - COMP := pgi - -The integrator used by the Microphysics library must be set to ``VODE90``: - -:: - - INTEGRATOR_DIR := VODE90 Depending on which system you are running on, it may be necessary to specify the CUDA Capability using the ``CUDA_ARCH`` flag. The CUDA Capability will @@ -61,179 +49,6 @@ capability 6.x, the flag should be set to: .. _sec:gpuporting: -Offloading a routine to GPU -=========================== - -In order to offload a routine to the GPU, we insert a ``#pragma gpu`` -statement on the line before the call. This tells the preprocessor to -generate a CUDA device version of the function, with all the -additional CUDA GPU management. In addition to this, there are a few -other modifications required to ensure the function operates correctly -on the GPU. We summarize these below. - -C++ ---- - -- Make sure that the box ``lo`` and ``hi`` are the first two arguments - and use ``AMREX_INT_ANYD`` as the macro wrapping ``bx.loVect()`` and - ``bx.hiVect()`` - -- Likewise, inplace of ``AMREX_ZFILL()``, use ``AMREX_REAL_ANYD`` - -- Scalars must be passed by value to Fortran functions (not by - reference) - -- Make sure that you change the variable definitions in the Fortran - code to include value so that the Fortran knows the variables are - being passed by value rather than by reference. If you don’t do - this, the code is liable to segfault - -- FArrayBox functions like ``setVal()`` and ``saxpy()`` are not on the - GPU, so you should explicitly write out these operations in C++. - The MultiFab counterparts are on the GPU. - -- Use ``BL_TO_FORTRAN_ANYD()`` to wrap MultiFab arguments (and in the header file wrap with ``BL_FORT_FAB_ARG_3D``) - -To illustrate these modifications, consider the function ``mk_sponge``. To call the function on the CPU, we would write - -.. code-block:: c++ - - for (MFIter mfi(sponge_mf, true); mfi.isValid(); ++mfi) { - - const Box& tileBox = mfi.tilebox(); - - mk_sponge(AMREX_ARLIM_3D(tileBox.loVect()), AMREX_ARLIM_3D(tileBox.hiVect()), - BL_TO_FORTRAN_3D(sponge_mf[mfi]), - AMREX_ZFILL(dx), &dt); - } - -Implementing the changes described above to offload this to GPU, this becomes - -.. code-block:: c++ - - for (MFIter mfi(sponge_mf, true); mfi.isValid(); ++mfi) { - - const Box& tileBox = mfi.tilebox(); - - #pragma gpu box(tileBox) - mk_sponge(AMREX_INT_ANYD(tileBox.loVect()), AMREX_INT_ANYD(tileBox.hiVect()), - BL_TO_FORTRAN_ANYD(sponge_mf[mfi]), - AMREX_REAL_ANYD(dx), dt); - } - -Fortran -------- - -- The routine must only operate on a single zone in ``lo:hi``. - - - Even for temporary arrays, you cannot write to an ``i+1`` zone - (e.g. when doing limiting) -- this will cause a race condition. - If necessary, do extra computation to avoid the temporary arrays. - -- Mark the routine with ``!$gpu`` - -- If a module defines its own variables, these variables need to be - ``allocatable`` (even if they are scalars) and marked as - ``attributes(managed)``. Additional routines may be needed to - allocate these variables before they’re used and deallocate them - when they’re no longer needed. - - - Examples of this can be seen for the sponge parameters. These are - marked as ``allocatable`` and ``attributes(managed)`` in - ``sponge.F90``, and therefore must be allocated (by ``init_sponge``). - -- Temporary variables must be defined outside of function calls. E.g. if a - function call contains ``foo(x(a:b)/y)``, you need to define a new variable - ``z = x(a:b)/y`` then pass this into the function as ``foo(z)``. - - - If you don’t do this, you may see the error ``Array reshaping is - not supported for device subprogram calls`` - -- If importing a function from another module, make sure to put the - import within the function/subroutine, and put ``! function`` at the - end of the line, e.g. - -.. code-block:: fortran - - use my_module, only: my_func ! function - -- Individual functions should be imported individually (so not ``use - my_module, only: func1, func2 ! function``) and there must be a - space either side of the ``!`` - -- Make sure the fortran file is ``.F90`` rather than ``.f90`` (and - remember to update the ``Make.xx`` file to reflect this). If you - don’t do this, you will see the error ``Label field of continuation - line is not blank`` - - - This is required as we use the convention that ``.F90`` files are - processed by the preprocessor, and ``.f90`` files are not. The - preprocessor will therefore only generate the required device - function if the file has the correct extension. - -We can see some of the above modifications by looking at the -subroutine ``estdt`` in ``compute_dt.F90``: - -.. code-block:: fortran - - subroutine estdt(lev, dt, umax, lo, hi, dx, & - scal, s_lo, s_hi, nc_s, & - u, u_lo, u_hi, nc_u, & - force, f_lo, f_hi, nc_f, & - divu, d_lo, d_hi, & - dSdt, t_lo, t_hi, & - w0_cart, w_lo, w_hi, & - p0_cart, p_lo, p_hi, & - gamma1bar_cart, g_lo, g_hi) bind (C,name="estdt") - - use amrex_constants_module, only: HALF - use amrex_fort_module, only: amrex_min ! function - use amrex_fort_module, only: amrex_max ! function - - ! input parameters - integer , value, intent(in ) :: lev - double precision, intent(inout) :: dt, umax - integer , intent(in ) :: lo(3), hi(3) - ... - - ! local variables - double precision :: spdx, spdy, spdz, spdr, rho_min - double precision :: fx, fy, fz, dt_temp - double precision :: eps,denom,gradp0 - double precision :: a, b, c - integer :: i,j,k - - !$gpu - - rho_min = 1.d-20 - ... - - do k = lo(3), hi(3) - do j = lo(2), hi(2) - do i = lo(1), hi(1) - spdx = max(spdx ,abs(u(i,j,k,1))) - - ... - - end subroutine estdt - -- Here, we can see that ``amrex_min`` and ``amrex_max`` functions from the - ``amrex_fort_module`` are marked separately as ``! function``, which tells - the preprocessor to generate a device version of this function. - -- The scalar ``lev`` is passed in by value. - -- The ``!$gpu`` directive has been inserted after the definition of - all the variables passed into the routine and all the local - variables, but before the main body of the function. - -- The routine only operates on values in a single zone of ``lo:hi``. - - -.. To be documented -.. ---------------- -.. -.. when do we need to mark stuff as attributes(managed)? .. _sec:gpuprofile: diff --git a/sphinx_docs/source/initial_models.rst b/sphinx_docs/source/initial_models.rst index 102c60494..ffed6affb 100644 --- a/sphinx_docs/source/initial_models.rst +++ b/sphinx_docs/source/initial_models.rst @@ -45,10 +45,10 @@ problem: {\tt wdconvect} & \rightarrow & {\tt model\_6.e8.raw} \nonumber \\ {\tt spherical\_heat} & \rightarrow & \mathrm{none-it~ is~ generated~ analytically} \nonumber \\\end{aligned} -We use a fortran subroutine +We use a C++ program to interpolate the raw data, yielding the model data, :math:`\rho^{\model}, -T^{\model}, p^{\model}`, and :math:`X^{\model}`. The fortran subroutine -then uses an iterative procedure to modify the model data so that it +T^{\model}, p^{\model}`, and :math:`X^{\model}`. These initial model +routines generally uses an iterative procedure to modify the model data so that it is thermodynamically consistent with the MAESTROeX equation of state (EOS), and also satisfies our chosen hydrostatic equilibrium (HSE) discretization, diff --git a/sphinx_docs/source/introduction.rst b/sphinx_docs/source/introduction.rst index fb52a6307..8b15404b7 100644 --- a/sphinx_docs/source/introduction.rst +++ b/sphinx_docs/source/introduction.rst @@ -11,7 +11,7 @@ MAESTROeX models the evolution of low Mach number astrophysical flows that are in hydrostatic equilibrium. The name MAESTROeX itself derives from MAESTRO, the original (pure Fortran) low Mach number stellar hydrodynamics code. MAESTROeX began as a rewrite MAESTRO -in the C++/Fortran framework of AMReX, and has since then gained +in the C++ framework of AMReX, and has since then gained additional algorithmic capabilities. Henceforth, we will refer only to MAESTROeX, although many of the ideas originated in MAESTRO before the change to MAESTROeX. diff --git a/sphinx_docs/source/makefiles.rst b/sphinx_docs/source/makefiles.rst index fa150dda3..e1102968a 100644 --- a/sphinx_docs/source/makefiles.rst +++ b/sphinx_docs/source/makefiles.rst @@ -54,11 +54,9 @@ lines of the form: :: CEXE_sources += file.cpp - F90EXE_sources += file.F90 -where ``file.cpp`` is a C++ source file and ``file.F90`` is a Fortran -source file that should be built when this directory is added to the -list of build directories. +where ``file.cpp`` is a C++ source file that should be built when this +directory is added to the list of build directories. The AMReX build system relies on the ``vpath`` functionality of make. In a makefile, the ``vpath`` variable holds search path used to @@ -74,7 +72,7 @@ Dependencies ------------ There is no need to explicitly define the dependencies between the -source files for Fortran modules. Scripts in +source files. Scripts in AMReX are run at the start of the build process and parse all the source files and make an explicit list of the dependency pairs. @@ -264,7 +262,7 @@ track down memory issues, uninitialized variables, NaNs, etc. ``GNUmakefile`` generates an executable with debugging information included in the executable (e.g., to be interpreted by the debugger, gdb). This will usually add -g to the compile line and - also lower the optimization. For gfortran it will add several + also lower the optimization. This will add several options to catch uninitialize variables, bounds errors, etc. The resulting executable will have ``DEBUG`` in its name. @@ -309,7 +307,7 @@ track down memory issues, uninitialized variables, NaNs, etc. - ``FSANITIZER`` - For gfortran, gcc, g++, setting ``FSANITIZER=TRUE`` + For gcc/g++, setting ``FSANITIZER=TRUE`` in ``GNUmakefile`` will enable the address sanitizer support built into GCC. This is enabled through integration with https://github.com/google/sanitizers in GCC. diff --git a/sphinx_docs/source/mg.rst b/sphinx_docs/source/mg.rst index 996d5364e..8986bb775 100644 --- a/sphinx_docs/source/mg.rst +++ b/sphinx_docs/source/mg.rst @@ -32,6 +32,11 @@ Support for :math:`\Delta x \ne \Delta y \ne \Delta z` Data Structures --------------- +.. note:: + + This information is outdated + + mg_tower ~~~~~~~~ diff --git a/sphinx_docs/source/param_intro.rst b/sphinx_docs/source/param_intro.rst index 101329c6f..4d465607b 100644 --- a/sphinx_docs/source/param_intro.rst +++ b/sphinx_docs/source/param_intro.rst @@ -8,14 +8,11 @@ Runtime Parameters Introduction to Runtime Parameters ================================== -MAESTROeX has 2 sets of runtime parameters—those controlled by -C++ and those controlled by Fortran. The C++ parameters are set +MAESTROeX runtime parameters are set in the inputs file and managed by the AMReX ``ParmParse`` class. For MAESTROeX-specific parameters, we list the runtime parameters in a file ``Source/param/_cpp_parameters`` and generate the -C++ code and headers using the ``Source/param/mk_params.sh`` script—note -this script needs to be run every time the ``_cpp_parameters`` -file is updated. +C++ code and headers automatically at compilation time. The behavior of the network, EOS, and other microphysics routines are controlled by a different set of runtime parameters. These parameters are defined @@ -52,23 +49,7 @@ Here, * ``default`` is the default value of the parameter. -The next columns are optional, but you need to fill in all of the -information up to and including any of the optional columns you need -(e.g., if you are going to provide the fortran name, you also need to -provide ``need in Fortran?`` and ``ifdef``. - - * ``need in Fortran?`` is ``y`` if the runtime parameter should be - made available in Fortran (through ``meth_params_module``). - - * ``ifdef`` provides the name of a preprocessor name that should - wrap this parameter definition—it will only be compiled in if that - name is defined to the preprocessor. - - * ``fortran name`` is the name that the parameter should use in - Fortran—by default it will be the same as ``name``. - - * ``fortran type`` is the data type of the parameter in Fortran—by - default it will be the Fortran-equivalent to ``type``. +The next columns are outdated. Finally, any comment (starting with ``#``) immediately before the parameter definition will be used to generate the documentation diff --git a/sphinx_docs/source/preface.rst b/sphinx_docs/source/preface.rst index 6eb0b868b..06845a802 100644 --- a/sphinx_docs/source/preface.rst +++ b/sphinx_docs/source/preface.rst @@ -6,7 +6,7 @@ This documentation is a work in progress. MAESTROeX is rapidly evolving, so portions of the text are likely to be incomplete or out-of-date. -MAESTROeX is an updated C++/Fortran implementation of the previous +MAESTROeX is an updated C++ implementation of the previous (pure-Fortran) MAESTRO code. Thus, MAESTROeX is able to leverage the updated capability of the AMReX software framework for massively parallel, block-structured adaptive mesh refinement (AMR) From 861261a65e3fbb92b6edbbd064eda12f4fb0c5d8 Mon Sep 17 00:00:00 2001 From: Michael Zingale Date: Thu, 4 Apr 2024 08:28:33 -0400 Subject: [PATCH 2/2] fix whitespace --- sphinx_docs/source/addproblem.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/sphinx_docs/source/addproblem.rst b/sphinx_docs/source/addproblem.rst index cae9db9e4..63ae389a4 100644 --- a/sphinx_docs/source/addproblem.rst +++ b/sphinx_docs/source/addproblem.rst @@ -111,7 +111,7 @@ By default, some of the runtime parameters are listed in PROBIN_PARAMETER_DIRS := . These are combined with the runtime parameters specified by Microphysics -and parsed at compile time. A C++ header and source file is written +and parsed at compile time. A C++ header and source file is written (``extern_parameters.cpp`` and ``extern_parameters.H``) that define the parameters and their defaults.