Merged upstream changes from develop

CHANGELOG.md

@@ -6,8 +6,14 @@
 
 ### New Features and Enhancements
 
-Added Multirate time step adaptivity controllers, based on the recently introduced
-`SUNAdaptController` base class, to ARKODE's MRIStep module.
+The default value of `CMAKE_CUDA_ARCHITECTURES` is no longer set to `70` and is
+now determined automatically by CMake. The previous default was only valid for
+Volta GPUs while the automatically selected value will vary across compilers and
+compiler versions. As such, users are encouraged to override this value with the
+architecture for their system.
+
+Added support for multirate time step adaptivity controllers, based on the
+recently introduced `SUNAdaptController` base class, to ARKODE's MRIStep module.
 
 Added functionality to ARKODE to accumulate a temporal error
 estimate over multiple time steps.  See the routines `ARKodeSetAccumulatedErrorType`,
@@ -17,6 +23,9 @@ estimate over multiple time steps.  See the routines `ARKodeSetAccumulatedErrorT
 
 Fixed the loading of ARKStep's default first order explicit method.
 
+Fixed a CMake bug regarding usage of missing "print_warning" macro
+that was only triggered when the deprecated `CUDA_ARCH` option was used.
+
 ### Deprecation Notices
 
 ## Changes to SUNDIALS in release 7.1.1

README.md

@@ -15,50 +15,62 @@ equation (ODE) systems, differential-algebraic equation (DAE) systems, and
 nonlinear algebraic systems:
 
 * ARKODE - for integrating stiff, nonstiff, and multirate ODEs of the form
+
   $$M(t) \\, y' = f_1(t,y) + f_2(t,y), \quad y(t_0) = y_0$$
 
 * CVODE - for integrating stiff and nonstiff ODEs of the form
+
   $$y' = f(t,y), \quad y(t_0) = y_0$$
 
 * CVODES - for integrating and sensitivity analysis (forward and adjoint) of
   ODEs of the form
+
   $$y' = f(t,y,p), \quad y(t_0) = y_0(p)$$
 
 * IDA - for integrating DAEs of the form
+
   $$F(t,y,y') = 0, \quad y(t_0) = y_0, \quad y'(t_0) = y_0'$$
 
 * IDAS - for integrating and sensitivity analysis (forward and adjoint) of DAEs
   of the form
+
   $$F(t,y,y',p) = 0, \quad y(t_0) = y_0(p), \quad y'(t_0) = y_0'(p)$$
 
 * KINSOL - for solving nonlinear algebraic systems of the form
+
   $$F(u) = 0 \quad \text{or} \quad G(u) = u$$
 
 ## Installation ##
 
-For installation directions see the [online install guide](https://sundials.readthedocs.io/en/latest/Install_link.html),
-the installation chapter in any of the package user guides, or INSTALL_GUIDE.pdf.
+For installation directions, see the [getting started](https://sundials.readthedocs.io/en/latest/sundials/index.html#getting-started)
+section in the online documentation. In the [released tarballs](https://github.com/LLNL/sundials/releases),
+installation directions are also available in `INSTALL_GUIDE.pdf` and the
+installation chapter of the user guides in the `doc` directory.
 
 Warning to users who receive more than one of the individual packages at
-different times: Mixing old and new versions of SUNDIALS may fail. To avoid
-such failures, obtain all desired package at the same time.
+different times: Mixing old and new versions of SUNDIALS may fail. To avoid such
+failures, obtain all desired package at the same time.
 
 ## Support ##
 
-Full user guides for all of the SUNDIALS packages are available [online](https://sundials.readthedocs.io)
-and in the [doc](./doc) directory. Additionally, the [doc](./doc) directory
-contains documentation for the package example programs.
+Full user guides for all of the SUNDIALS packages are available [online](https://sundials.readthedocs.io).
+In the [released tarballs](https://github.com/LLNL/sundials/releases), the `doc`
+directory includes PDFs of the user guides and documentation for the example
+programs. The example program documentation PDFs are also available on the
+[releases page](https://github.com/LLNL/sundials/releases).
 
 For information on recent changes to SUNDIALS see the [CHANGELOG](./CHANGELOG.md)
 or the introduction chapter of any package user guide.
 
 A list of Frequently Asked Questions on build and installation procedures as
-well as common usage issues is available on the SUNDIALS [FAQ](https://computing.llnl.gov/projects/sundials/faq).
-For dealing with systems with unphysical solutions or discontinuities see the
-SUNDIALS [usage notes](https://computing.llnl.gov/projects/sundials/usage-notes).
+well as common usage issues is available on the SUNDIALS
+[FAQ](https://computing.llnl.gov/projects/sundials/faq). For dealing with
+systems with nonphysical solutions or discontinuities see the SUNDIALS
+[usage notes](https://computing.llnl.gov/projects/sundials/usage-notes).
 
-If you have a question not covered in the FAQ or usage notes, please submit
-your question to the SUNDIALS [mailing list](https://computing.llnl.gov/projects/sundials/mailing-list).
+If you have a question not covered in the FAQ or usage notes, please submit your
+question as a [GitHub issue](https://github.com/LLNL/sundials/issues) or to the
+SUNDIALS [mailing list](https://computing.llnl.gov/projects/sundials/mailing-list).
 
 ## Contributing ##
 
@@ -76,14 +88,15 @@ any publications reporting work done using SUNDIALS packages.
 
 The SUNDIALS library has been developed over many years by a number of
 contributors. The current SUNDIALS team consists of Cody J. Balos,
-David J. Gardner, Alan C. Hindmarsh, Daniel R. Reynolds, and Carol S. Woodward.
-We thank Radu Serban for significant and critical past contributions.
+David J. Gardner, Alan C. Hindmarsh, Daniel R. Reynolds, Steven B. Roberts, and
+Carol S. Woodward. We thank Radu Serban for significant and critical past
+contributions.
 
 Other contributors to SUNDIALS include: James Almgren-Bell, Lawrence E. Banks,
 Peter N. Brown, George Byrne, Rujeko Chinomona, Scott D. Cohen, Aaron Collier,
 Keith E. Grant, Steven L. Lee, Shelby L. Lockhart, John Loffeld, Daniel McGreer,
-Yu Pan, Slaven Peles, Cosmin Petra, Steven B. Roberts, H. Hunter Schwartz,
-Jean M. Sexton, Dan Shumaker, Steve G. Smith, Shahbaj Sohal, Allan G. Taylor,
+Yu Pan, Slaven Peles, Cosmin Petra, H. Hunter Schwartz, Jean M. Sexton,
+Dan Shumaker, Steve G. Smith, Shahbaj Sohal, Allan G. Taylor,
 Hilari C. Tiedeman, Chris White, Ting Yan, and Ulrike M. Yang.
 
 ## License ##
@@ -93,7 +106,7 @@ and [NOTICE](./NOTICE) files for details. All new contributions must be made
 under the BSD 3-clause license.
 
 **Please Note** If you are using SUNDIALS with any third party libraries linked
-in (e.g., LAPACK, KLU, SuperLU_MT, PETSc, or *hypre*), be sure to review the
+in (e.g., LAPACK, KLU, SuperLU_MT, PETSc, *hypre*, etc.), be sure to review the
 respective license of the package as that license may have more restrictive
 terms than the SUNDIALS license.
 

cmake/SundialsDeprecated.cmake

@@ -180,8 +180,8 @@ endif()
 #
 
 if(DEFINED CUDA_ARCH)
-  print_warning("The CMake option CUDA_ARCH is deprecated. "
-                "Use CMAKE_CUDA_ARCHITECTURES instead.")
+  message(DEPRECATION "The CMake option CUDA_ARCH is deprecated. "
+                      "Use CMAKE_CUDA_ARCHITECTURES instead.")
   # convert sm_** to just **
   string(REGEX MATCH "[0-9]+" arch_name "${CUDA_ARCH}")
   set(CMAKE_CUDA_ARCHITECTURES

cmake/SundialsTPLOptions.cmake

@@ -46,10 +46,6 @@ sundials_option(ENABLE_PTHREAD BOOL "Enable Pthreads support" OFF)
 # -------------------------------------------------------------
 sundials_option(ENABLE_CUDA BOOL "Enable CUDA support" OFF)
 
-# CMake 3.18 adds this option.
-sundials_option(CMAKE_CUDA_ARCHITECTURES STRING "Target CUDA architecture" "70"
-                DEPENDS_ON ENABLE_CUDA)
-
 # -------------------------------------------------------------
 # Enable HIP support?
 # -------------------------------------------------------------

doc/arkode/guide/source/Usage/MRIStep/User_callable.rst

@@ -815,7 +815,7 @@ Optional inputs for IVP method selection
 
    Select the default MRI method of a given order.
 
-   The default order is 3. An order less than 3 or greater than 4 will result in
+   The default order is 3. An order less than 1 or greater than 4 will result in
    using the default.
 
    :param arkode_mem: pointer to the MRIStep memory block.

doc/arkode/guide/source/Usage/SPRKStep/User_callable.rst

@@ -20,7 +20,7 @@ by the user to setup and then solve an IVP using the SPRKStep time-stepping
 module. The large majority of these routines merely wrap :ref:`underlying
 ARKODE functions <ARKODE.Usage.UserCallable>`, and are now deprecated
 -- each of these are clearly marked.  However, some
-of these user-callable functions are specific to ERKStep, as explained
+of these user-callable functions are specific to SPRKStep, as explained
 below.
 
 As discussed in the main :ref:`ARKODE user-callable function introduction

doc/shared/RecentChanges.rst

@@ -2,15 +2,25 @@
 
 **New Features and Enhancements**
 
-Added Multirate time step adaptivity controllers, based on the recently introduced
-`SUNAdaptController` base class, to ARKODE's MRIStep module.
+The default value of :cmakeop:`CMAKE_CUDA_ARCHITECTURES` is no longer set to
+``70`` and is now determined automatically by CMake. The previous default was
+only valid for Volta GPUs while the automatically selected value will vary
+across compilers and compiler versions. As such, users are encouraged to
+override this value with the architecture for their system.
+
+Added support for multirate time step adaptivity controllers, based on the
+recently introduced `SUNAdaptController` base class, to ARKODE's MRIStep module.
 
 Added functionality to ARKODE to accumulate a temporal error
-estimate over multiple time steps.  See the routines :c:func:`ARKodeSetAccumulatedErrorType`,
-:c:func:`ARKodeResetAccumulatedError`, and :c:func:`ARKodeGetAccumulatedError` for details.
+estimate over multiple time steps.  See the routines
+:c:func:`ARKodeSetAccumulatedErrorType`, :c:func:`ARKodeResetAccumulatedError`,
+and :c:func:`ARKodeGetAccumulatedError` for details.
 
 **Bug Fixes**
 
 Fixed the loading of ARKStep's default first order explicit method.
 
+Fixed a CMake bug regarding usage of missing "print_warning" macro
+that was only triggered when the deprecated ``CUDA_ARCH`` option was used.
+
 **Deprecation Notices**

doc/shared/sundials/Install.rst

@@ -523,9 +523,19 @@ illustration only.
 
 .. cmakeoption:: CMAKE_CUDA_ARCHITECTURES
 
-   Specifies the CUDA architecture to compile for.
+   Specifies the CUDA architecture to compile for i.e., ``60`` for Pascal,
+   ``70`` Volta, ``80`` for Ampere, ``90`` for Hopper, etc. See the `GPU compute
+   capability tables <https://developer.nvidia.com/cuda-gpus>`_ on the NVIDIA
+   webpage and the `GPU Compilation <https://docs.nvidia.com/cuda/cuda-compiler-driver-nvcc/index.html#gpu-compilation>`_
+   section of the CUDA documentation for more information.
 
-   Default: ``sm_30``
+   Default: Determined automatically by CMake. Users are encouraged to override
+   this value with the architecture for their system as the default varies
+   across compilers and compiler versions.
+
+   .. versionchanged:: x.y.z
+
+      In prior versions ``CMAKE_CUDA_ARCHITECTURES`` defaulted to ``70``.
 
 .. cmakeoption:: EXAMPLES_ENABLE_C
 

include/sundials/priv/sundials_errors_impl.h

@@ -186,10 +186,12 @@ static inline void SUNHandleErrWithFmtMsg(int line, const char* func,
   msglen = (size_t)vsnprintf(NULL, (size_t)0, msgfmt, values); /* determine size
                                                                   of buffer
                                                                   needed */
-  msg    = (char*)malloc(msglen + 1);
+  va_end(values);
+  msg = (char*)malloc(msglen + 1);
+  va_start(values, sunctx);
   vsnprintf(msg, msglen + 1, msgfmt, values);
-  SUNHandleErrWithMsg(line, func, file, msg, code, sunctx);
   va_end(values);
+  SUNHandleErrWithMsg(line, func, file, msg, code, sunctx);
   free(msg);
 }
 

src/sundials/sundials_errors.c

@@ -94,23 +94,23 @@ void SUNGlobalFallbackErrHandler(int line, const char* func, const char* file,
   char* log_msg       = NULL;
   char* file_and_line = NULL;
 
-  va_start(ap, err_code);
-
   file_and_line = sunCombineFileAndLine(__LINE__, __FILE__);
+  va_start(ap, err_code);
   sunCreateLogMessage(SUN_LOGLEVEL_ERROR, 0, file_and_line,
                       __func__, "The SUNDIALS SUNContext was corrupt or NULL when an error occurred. As such, error messages have been printed to stderr.",
                       ap, &log_msg);
+  va_end(ap);
   fprintf(stderr, "%s", log_msg);
   free(log_msg);
   free(file_and_line);
 
   file_and_line = sunCombineFileAndLine(line, file);
   if (msgfmt == NULL) { msgfmt = SUNGetErrMsg(err_code); }
+  va_start(ap, err_code);
   sunCreateLogMessage(SUN_LOGLEVEL_ERROR, 0, file_and_line, func, msgfmt, ap,
                       &log_msg);
+  va_end(ap);
   fprintf(stderr, "%s", log_msg);
   free(log_msg);
   free(file_and_line);
-
-  va_end(ap);
 }

src/sundials/sundials_logger.c

@@ -333,12 +333,12 @@ SUNErrCode SUNLogger_QueueMsg(SUNLogger logger, SUNLogLevel lvl,
       return retval;
     }
 
-    va_list args;
-    va_start(args, msg_txt);
-
     if (logger->queuemsg)
     {
+      va_list args;
+      va_start(args, msg_txt);
       retval = logger->queuemsg(logger, lvl, scope, label, msg_txt, args);
+      va_end(args);
     }
     else
     {
@@ -347,7 +347,10 @@ SUNErrCode SUNLogger_QueueMsg(SUNLogger logger, SUNLogLevel lvl,
       if (sunLoggerIsOutputRank(logger, &rank))
       {
         char* log_msg = NULL;
+        va_list args;
+        va_start(args, msg_txt);
         sunCreateLogMessage(lvl, rank, scope, label, msg_txt, args, &log_msg);
+        va_end(args);
 
         switch (lvl)
         {
@@ -372,8 +375,6 @@ SUNErrCode SUNLogger_QueueMsg(SUNLogger logger, SUNLogLevel lvl,
         free(log_msg);
       }
     }
-
-    va_end(args);
   }
 #else
   /* silence warnings when all logging is disabled */

test/unit_tests/sundials/test_sundials_errors.cpp

@@ -171,22 +171,25 @@ class SUNContextErrFunctionTests : public testing::Test
   SUNContext sunctx;
 };
 
-void firstHandler(int line, const char* func, const char* file, const char* msg,
-                  SUNErrCode err_code, void* err_user_data, SUNContext sunctx)
+static void firstHandler(int line, const char* func, const char* file,
+                         const char* msg, SUNErrCode err_code,
+                         void* err_user_data, SUNContext sunctx)
 {
   std::vector<int>* order = static_cast<std::vector<int>*>(err_user_data);
   order->push_back(0);
 }
 
-void secondHandler(int line, const char* func, const char* file, const char* msg,
-                   SUNErrCode err_code, void* err_user_data, SUNContext sunctx)
+static void secondHandler(int line, const char* func, const char* file,
+                          const char* msg, SUNErrCode err_code,
+                          void* err_user_data, SUNContext sunctx)
 {
   std::vector<int>* order = static_cast<std::vector<int>*>(err_user_data);
   order->push_back(1);
 }
 
-void thirdHandler(int line, const char* func, const char* file, const char* msg,
-                  SUNErrCode err_code, void* err_user_data, SUNContext sunctx)
+static void thirdHandler(int line, const char* func, const char* file,
+                         const char* msg, SUNErrCode err_code,
+                         void* err_user_data, SUNContext sunctx)
 {
   std::vector<int>* order = static_cast<std::vector<int>*>(err_user_data);
   order->push_back(2);