Merge branch 'main' into cell-centered-metric

.github/workflows/ci.yml

@@ -17,7 +17,7 @@ jobs:
   download-build-test:
     strategy:
       matrix:
-        accelerator: [cpu, cuda]
+        accelerator: [cpu, cuda, rocm]
         real-precision: [real64]
         mode: [debug, optimize]
         exclude:
@@ -60,5 +60,5 @@ jobs:
       # Set exit code based on test status
       - name: Set exit code
         # Only look at the test results for `carpetx-cpu-real64` runs
-        if: ${{matrix.accelerator == 'cpu' && matrix.real-precision == 'real64' && matrix.mode == 'debug' && env.TESTS_FAILED == 'True'}}
+        if: ${{matrix.accelerator == 'cpu' && matrix.real-precision == 'real64' && matrix.mode == 'optimize' && env.TESTS_FAILED == 'True'}}
         run: exit 1

AHFinderDirect/README

@@ -0,0 +1,257 @@
+Cactus Code Thorn AHFinderDirect
+Authors    : Jonathan Thornburg <[email protected]>
+Maintainer : Jonathan Thornburg <[email protected]>
+CVS info   : $Header$
+--------------------------------------------------------------------------
+
+Purpose of This Thorn
+=====================
+
+This thorn finds an apparent horizon (or more generally, a closed
+2-surface with S^2 topology having any desired constant expansion)
+given the 3D xyz-grid metric and extrinsic curvature (and optionally
+the StaticConformal conformal factor).  It uses a direct method,
+writing the apparent horizon equation as an elliptic PDE on
+angular-coordinate space.  This is very fast and accurate, but
+it does require an intitial guess for the apparent horizon position.
+
+
+Documentation
+=============
+
+The doc/ directory contains detailed documentation for this thorn,
+including a discussion of those parameters which you'll likely want
+to set (or at least know about) in practice.  For full information
+on all the parameters, see the comments in the  param.ccl  file.
+
+The par/ directory contains some sample par files.
+
+src/CODESTYLE documents the general programming conventions used in
+this thorn.
+
+src/PORTING.OUTSIDE.CACTUS has some general remarks on what it would
+take to port this thorn to a non-Cactus numerical relativity code.
+
+See below in this file for notes on compiling this thorn.
+
+
+Publications
+============
+The algorithms used in this thorn are described in the papers
+@article{
+	Thornburg2003:AH-finding,
+	author = "Jonathan Thornburg",
+	title  = "A Fast Apparent-Horizon Finder
+		  for 3-Dimensional {C}artesian Grids in Numerical Relativity",
+	journal = "Classical and Quantum Gravity",
+	volume = 21, number = 2,
+	year   = "2004", month = "January 21",
+	pages  = "743--766",
+	doi    = "10.1088/0264-9381/21/2/026",
+	url    = "http://stacks.iop.org/0264-9381/21/743",
+	eprint = "gr-qc/0306056",
+	note   = "gr-qc/0306056",
+	}
+and
+@article{
+	Thornburg-1996-horizon-finding,
+	author = "Jonathan Thornburg",
+	title = "Finding Apparent Horizons in Numerical Relativity",
+	journal = "Physical Review D",
+	pages = "4899--4918",
+	volume = 54, number = 8,
+	year = 1996, month = "October 15",
+	eprint = "gr-qc/9508014",
+	}
+As a courtesy, I ask that these papers be cited in any published research
+which uses this thorn, or which uses code from this thorn.
+
+
+Copyright
+=========
+
+This thorn is copyright (C) 2001-2004
+by Jonathan Thornburg <[email protected]>.
+
+This thorn is free software; you can redistribute it and/or modify
+it under the terms of the GNU General Public License as published by
+the Free Software Foundation; either version 2 of the License, or
+(at your option) any later version.
+
+This thorn is distributed in the hope that it will be useful, but
+WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with this thorn (see the file COPYING in this directory);
+if not, write to the Free Software Foundation, Inc., 59 Temple
+Place, Suite 330, Boston, MA  02111-1307  USA
+
+
+Other Software Used (Already Contained within AHFinderDirect)
+=============================================================
+
+AHFinderDirect's  src/sparse-matrix/  directory contains various
+sparse-matrix libraries, which have their own copyrights and licensing terms:
+
+The src/sparse-matrix/umfpack/ directory contains a subset of the files
+in UMFPACK version 4.0 (11.Apr.2002).  This code is copyright (C) 2002 by
+Timothy A. Davis, and is subect to the UMFPACK License:
+
+    Your use or distribution of UMFPACK or any modified version of
+    UMFPACK implies that you agree to this License.
+
+    THIS MATERIAL IS PROVIDED AS IS, WITH ABSOLUTELY NO WARRANTY
+    EXPRESSED OR IMPLIED.  ANY USE IS AT YOUR OWN RISK.
+
+    Permission is hereby granted to use or copy this program, provided
+    that the Copyright, this License, and the Availability of the original
+    version is retained on all copies.  User documentation of any code that
+    uses UMFPACK or any modified version of UMFPACK code must cite the
+    Copyright, this License, the Availability note, and "Used by permission."
+    Permission to modify the code and to distribute modified code is granted,
+    provided the Copyright, this License, and the Availability note are
+    retained, and a notice that the code was modified is included.  This
+    software was developed with support from the National Science Foundation,
+    and is provided to you free of charge.
+
+
+Other Software Required
+=======================
+
+This thorn inherits from Grid, ADMBase, StaticConformal, SpaceMask, IO
+(all in the CactusBase or CactusEinstein arrangements), and
+SphericalSurface (in the AEIThorns arrangement; you can get AEIThorns
+by anonymous CVS checkout from [email protected]:/numrelcvs, CVS
+path AEIThorns/SphericalSurface).
+
+By default this thorn uses various Cactus APIs which are supplied by
+other thorns:
+* This thorn uses CCTK_InterpGridArrays() for interpolating grid arrays.
+  This is supplied by a driver-specific thorn, such as PUGHInterp or
+  CarpetInterp.
+* This thorn uses CCTK_ReduceLocArrayToArray1D() for interprocessor
+  communication in the multiprocessor Newton solver.
+	[see src/driver/README.parallel for details]
+  This is supplied by a driver-specific thorn; at present PUGHReduce
+  it this for the PUGH driver.
+* This thorn uses CCTK_InterpLocalUniform() for interpatch and surface
+  interpolation, and it needs various interpolation options which at
+  present are (only) supported by the AEIThorns/AEILocalInterp local
+  interpolator.
+
+This thorn is written in C++, so you'll need a C++ compiler -- in fact
+a fairly modern one -- to compile this thorn.  See the "Compilation Notes"
+section below for details.
+
+Most of this thorn's relativity code is machine-generated using Maple
+(version 7), but you don't need Maple unless you want to modify the
+relativity code.
+
+By default, this thorn doesn't use any external libraries.  However,
+if HAVE_DENSE_JACOBIAN__LAPACK is defined in src/include/config.h,
+then this thorn uses the LAPACK library (which in turn uses the BLAS
+library), so you will need to configure your Cactus to use LAPACK.
+Instructions on doing this are in the file README.library in this
+directory.
+
+
+Code Notes
+==========
+
+This thorn is written in C++, and requires a fairly modern C++ compiler.
+In particular:
+* Templates are used, but only rather simple ones for containers or other
+  low-level stuff templated on the floating-point or integer datatype.
+  These templates are always instantiated explicitly.
+* bool, mutable, typename are used.
+* The new-style casts static_cast<...> and const_cast<...> are used.
+  reinterpret_cast<...> and dynamic_cast<...> are *not* used.
+* The code will compile ok under either the archaic or the modern
+  for-loop declaration scope rules.
+* C header files are used in the pre-namespaces form (eg <stdio.h>).
+  (In ISO-standard C++ this is deprecated (but still supported), but
+  alas none of the more modern forms (<cstdio> and namespace std::)
+  seem to be supported on as wide a range of systems as the pre-namespaces
+  form.)
+* <assert.h> is used fairly heavily for sanity checks.
+* To avoid various portability problems, none of the C++ standard
+  template library (STL) is used.
+
+
+Compiler Notes
+==============
+
+This thorn should be fully portable across architectures and operating
+systems, but C++ compilers may be a problem:
+
+This thorn has been compiled and run successfully using
+
+x86
+===
+* gcc version 2.95.3 20010125 (prerelease)
+* g++ (GCC) 3.2 for x86
+* g++ (GCC) 3.2.2 for x86
+* g++ (GCC) 3.3 for x86
+* Intel(R) C++ Compiler for 32-bit applications, Version 5.0.1   Build 010730D0
+* Intel C++ Compiler for Linux, version 6.0
+* Intel(R) C++ Compiler for 32-bit applications, Version 7.0   Build 20021021Z
+  On platinum.ncsa.uiuc.edu, Denis Pollney reports success configuring Cactus
+  with
+    LIBS="crypt F90 CEPCF90 intrins mkl mkl_lapack guide"
+    LIBDIRS=/usr/local/intel/mkl/lib/32
+* Portland Group pgCC 3.3-2
+  On platinum.ncsa.uiuc.edu, Denis Pollney reports success configuring Cactus
+  with
+    LAPACK=yes
+    LAPACK_DIR=/usr/local/pgi/linux86
+
+ia64, a.k.a. Itanium
+====================
+* Intel(R) C++ Itanium(R) Compiler for Itanium(R)-based applications
+  Version 7.0 Beta, Build 20020911
+  On titan.ncsa.uiuc.edu, Denis Pollney reports success configuring Cactus
+  with
+    LIBS="crypt F90 CEPCF90 intrins mkl_lapack mkl_itp guide"
+    LIBDIRS=/usr/local/intel/mkl/lib/64
+
+Alpha
+=====
+* g++ (GCC) 3.0
+  On lemieux.psc.edu, Denis Pollney reports success configuring Cactus
+  with
+    LIBS="m c_r dxml for"
+* Using the DEC/Compaq/HP/whatever-their-corporate-name-is-this-week
+  compiler, you need to specify the  -nopt  compiler option to disable
+  automatic template instantiation.
+
+SGI
+===
+* On modi4.ncsa.uiuc.edu (an old SGI Origin), using
+  MIPSpro Compilers: Version 7.3.1.3m
+  you need to specify the  -no_auto_include  compiler option to disable
+  automatic template instantiation.  In fact, you should probably specify
+  this option on any SGI system...
+
+IBM Power4 (AIX)
+================
+* C for AIX Compiler, Version 6
+  On psi19.rzg.mpg.de, Denis Pollney reports success configuring Cactus
+  with
+    LIBS="f m lapack-essl essl"
+    LIBDIRS="/afs/rzg/@sys/lib"
+
+Please let me (Jonathan) know of any other compilers which are ok.
+
+This thorn compiles, but does *NOT* work, using the folloing
+systems/compilers; the cure is to switch to a newer compiler version:
+x86
+===
+* gcc version 2.96 20000731 (Red Hat Linux 7.1 2.96-98)
+  (This is the system gcc on Red Hat GNU/Linux 7.2 systems)
+
+The code will almost certainly *not* compile using
+* any "egcs" version of gcc, or more generally, any version of gcc
+  earlier than 2.95
+* any version of Microsoft Visual C++ up to and including mid-2002

AHFinderDirect/README.library

@@ -0,0 +1,73 @@
+Library Configuration
+=====================
+$Header$
+
+By default this thorn doesn't use any external libraries, and you
+can ignore the instructions in this file.
+
+
+Using the LAPACK and BLAS Libraries
+===================================
+However, if this thorn is configured to use the LAPACK and BLAS
+libraries (see README for details), then you need to configure Cactus
+to use these libraries; this file describes how to do this.
+
+Basically, you just need to set
+   LAPACK=yes
+when configuring your Cactus configuration.
+
+If all goes well, Cactus will find the LAPACK and BLAS libraries and give
+during the configure process saying that it found them.  Otherwise, the
+configure process will abort with a message saying that you need to set
+the configure variable LAPACK_DIR to a directory (or blank-separated list
+of directories) containing the LAPACK and BLAS libraries.
+
+[If you don't have LAPACK/BLAS installed on your system already, you
+can get Fortran 77 source code and/or binaries for various architectures,
+from http://www.netlib.org/lapack/ and http://www.netlib.org/blas/
+respectively.  But many systems have these installed already -- try
+'locate liblapack' and 'locate libblas' on your system.]
+
+If the LAPACK and/or BLAS libraries were compiled with a (Fortran)
+compiler which is *not* used to compile any part of this Cactus'
+configuration, then you may also need to set the configure variables
+   LAPACK_EXTRA_LIBS = the name (or blank-separated list of names) of
+                       that (Fortran) compiler's run-time support
+                       library(ies)
+   LAPACK_EXTRA_LIB_DIRS = the directory (or blank-separated list of
+                           directories) containing that library(ies)
+
+For example, if your LAPACK and/or BLAS were compiled with the GNU g77
+compiler (as is common on GNU/Linux and *BSD systems), and you are *not*
+using g77 to compile any part of your Cactus configuration (maybe because
+you're using a different Fortran compiler), then you may need to set the
+environment variables to point to the g77 support library g2c:
+  LAPACK_EXTRA_LIBS = g2c
+  LAPACK_EXTRA_LIBDIRS = `g77 --print-file-name=libg2c.a | xargs basename`
+
+For example, on the AEI xeons the settings would be
+   LAPACK_EXTRA_LIBS = g2c
+   LAPACK_EXTRA_LIBDIRS = `g77 --print-file-name=libg2c.a | xargs dirname`
+
+All of these settings are of configure variables, i.e. you set them either
+on the command line when configuring,
+   % gmake my-configuration LAPACK=yes
+or as assignments in your ~/.cactus/config file,
+   % cat ~/.cactus/config
+   LAPACK=yes
+   LAPACK_EXTRA_LIBS=g2c
+   LAPACK_EXTRA_LIB_DIRS=/usr/lib/gcc-lib/i386-redhat-linux/2.96
+
+
+Compiler Version Compatability
+==============================
+All of Cactus -- including any external libraries -- need not be
+compiled with the same compilers, but the compilers must be link-compatible.
+In practice this isn't usually a problem.  However, as an example of what
+*not* to do, consider using libraries compiled with a gcc 2.* version
+(e.g. the system default libraries on many GNU/Linux distributions as
+of late 2002), combined with thorns compiled with a gcc 3.* version.
+gcc 2.* and 3.* are *not* link-compatible, so if you're lucky this
+combination will give all sorts of wierd errors in linking.  If you're
+unlucky it will link ok but then crash (or even worse, just give wrong
+results) at run-time.

AHFinderDirect/archive/README

@@ -0,0 +1,6 @@
+This directory contains stuff that's no longer used, but that I
+thought might be useful for future reference, so I didn't want to
+throw it away.
+
+Compared to the CVS attic, having a separate directory like this
+seems more convenient -- you can 'cat, awk, and grep' more easily.

AHFinderDirect/archive/ellipsoid.c

@@ -0,0 +1,28 @@
+fp t1, t2, t3, t5, t6, t7, t9, t10, t12, t28;
+fp t30, t33, t35, t36, t40, t42, t43, t48, t49, t52;
+fp t55;
+      t1 = a*a;
+      t2 = b*b;
+      t3 = t1*t2;
+      t5 = t3*zcos*CW;
+      t6 = c*c;
+      t7 = t1*t6;
+      t9 = t7*ycos*BV;
+      t10 = t2*t6;
+      t12 = t10*xcos*AU;
+      t28 = xcos*xcos;
+      t30 = CW*CW;
+      t33 = BV*BV;
+      t35 = t10*t28;
+      t36 = ycos*ycos;
+      t40 = AU*AU;
+      t42 = t7*t36;
+      t43 = zcos*zcos;
+      t48 = t3*t43;
+      t49 = -2.0*t1*zcos*CW*ycos*BV-2.0*t2*zcos*CW*xcos*AU-2.0*t6*ycos*BV*xcos*
+AU+t2*t28*t30+t6*t28*t33-t35+t1*t36*t30+t6*t36*t40-t42+t1*t43*t33+t2*t43*t40-
+t48;
+      t52 = sqrt(-t3*t6*t49);
+      t55 = 1/(t35+t42+t48);
+      r_plus = (t5+t9+t12+t52)*t55;
+      r_minus = (t5+t9+t12-t52)*t55;