Skip to content

Commit

Permalink
🔖 Merge branch 'master' into release, add spectral kernels
Browse files Browse the repository at this point in the history
thomasgillis committed Jun 17, 2020

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
2 parents 7de347b + e5fd702 commit eaf49ee
Showing 112 changed files with 3,571 additions and 1,680 deletions.
675 changes: 0 additions & 675 deletions COPYING

This file was deleted.

480 changes: 201 additions & 279 deletions LICENSE

Large diffs are not rendered by default.

29 changes: 15 additions & 14 deletions Makefile
Original file line number Diff line number Diff line change
@@ -1,23 +1,23 @@
################################################################################
# @copyright Copyright © UCLouvain 2019
# @copyright Copyright © UCLouvain 2020
#
# FLUPS is a Fourier-based Library of Unbounded Poisson Solvers.
#
# Copyright (C) <2019> <Universite catholique de Louvain (UCLouvain), Belgique>
# Copyright (C) <2020> <Universite catholique de Louvain (UCLouvain), Belgique>
#
# List of the contributors to the development of FLUPS, Description and complete License: see LICENSE file.
# List of the contributors to the development of FLUPS, Description and complete License: see LICENSE and NOTICE files.
#
# This program (FLUPS) is free software:
# you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version.
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
# http://www.apache.org/licenses/LICENSE-2.0
#
# You should have received a copy of the GNU General Public License
# along with this program (see COPYING file). If not,
# see <http://www.gnu.org/licenses/>.
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
################################################################################

@@ -40,6 +40,7 @@ TARGET_LIB_NB := build/lib$(NAME)_nb
BUILDDIR := ./build
SRC_DIR := ./src
OBJ_DIR := ./build
KERNEL_DIR := ./kernel

## add the headers to the vpaths
INC := -I$(SRC_DIR)
@@ -73,7 +74,7 @@ endif
# by default the LGF kernel data is installed in the include directory
LGF_PATH=$(abspath $(PREFIX)/include)
DEF += -DKERNEL_PATH=${LGF_PATH}
LGF_DATA := $(wildcard $(PREFIX)/kernel/*.ker)
LGF_DATA := $(wildcard $(KERNEL_DIR)/*.ker)

#-----------------------------------------------------------------------------
## add the wanted folders - common folders
@@ -202,7 +203,7 @@ logo:
@echo " |_| |______| \____/ |_| |_____/ "
@echo " "
@echo " "
@echo " (C) UCLouvain - GPLv3 "
@echo " (C) UCLouvain - Appache 2.0 "
@echo "----------------------------------------------------"

-include $(DEP)
317 changes: 317 additions & 0 deletions NOTICE

Large diffs are not rendered by default.

54 changes: 42 additions & 12 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,23 +1,23 @@
# FLUPS - A Fourier-based Library of Unbounded Poisson Solvers

#### Licensing and authorship
Copyright © UCLouvain 2019

FLUPS is distributed under GPLv3.0 (or any later version) license.
### Licensing and authorship
> FLUPS is distributed under Apache 2.0 license, copyright © UCLouvain 2020.
The main authors are (by alphabetical order):
- Denis-Gabriel Caprace
- Thomas Gillis

For the list of all the contributors to the development of FLUPS, description and a complete License: see LICENSE file.


#### Citation information
If you use FLUPS, please cite it as follows in your publications:
FLUPS is described in [this paper](https://arxiv.org/abs/2006.09300). If you use FLUPS, please cite it as follows in your publications:
- Caprace et al., **FLUPS - A Fourier-based Library of Unbounded Poisson Solvers**, SIAM Journal on Scientific Computing, 2019 (under review)


### Why should you use FLUPS?
- You can solve the Poisson on rectangular and uniform distributed 2D/3D grids;
- You can use any boundary conditions, including truly unbounded boundary conditions and semi-unbounded conditions
- You can use any boundary conditions, including truly unbounded boundary conditions and semi-unbounded conditions;
- You can solve may times the same Poisson problem at low cost using precomputed Green's function and communication patterns;
- You can use threads and/or MPI to fasten the execution;
- You can use the build-in profiler to optimize the execution speed;
@@ -27,7 +27,7 @@ If you use FLUPS, please cite it as follows in your publications:
### Installation

FLUPS is a C++ library, with an API in C.
The compilation of FLUPS was tested with Intel compilers and GCC.,
The compilation of FLUPS was tested with Intel compilers (v19.1) and GCC (v7.5).

#### Dependencies
First, you need to install the dependencies, typically using the following configuration commands (for the intel compilers)
@@ -101,7 +101,7 @@ To build the documentation, go to the `./doc` subfolder and type `doxygen`.
#### Available compilation flags
Here is an exhautstive list of the compilation flags that can be used to change the behavior of the code. To use `MY_FLAG`, simply add `-DMY_FLAG` to the variable `CXXFLAGS` in your `make_arch`.
- `DUMP_DBG`: if specified, the solver will I/O fields using the HDF5 library.
- `COMM_NONBLOCK`: if specified, the code will use the non-blocking communication pattern instead of the all to all version.
- `COMM_NONBLOCK`: if specified, the code will use the non-blocking communication pattern instead of the all-to-all version.
- `PERF_VERBOSE`: requires an extensive I/O on the communication pattern used. For performance tuning and debugging purpose only.
- `NDEBUG`: use this flag to bypass various checks inside the library
- `PROF`: allow you to use the build-in profiler to have a detailed view of the timing in each part of the solve. Make sure you have created a folder ```./prof``` next to your executable.
@@ -115,7 +115,11 @@ Here is an exhautstive list of the compilation flags that can be used to change
#### Detailed reference
The scientific background of the library is explained in "Caprace et al., **FLUPS - A Fourier-based Library of Unbounded Poisson Solvers**, SIAM Journal on Scientific Computing, 2019 (under review)".

A detailed description of the API is provided in the documentation (@ref flups.h), as well as many implementation details.
FLUPS solves two types of equations:
- laplacian(phi) = rhs, with phi and rhs either scalars or vectors
- laplacian(phi) = rot(rhs), with phi and rhs vectors (also code Biot-Savart mode)

A detailed description of the API is provided (in the documentation)[doc/documentation.html] (@ref flups.h), as well as many implementation details.

#### Memory layout
In this project we choose to handle the memory in a **Fortran** way of doing even if we are in C/C++.
@@ -125,7 +129,7 @@ The fastest rotating index is set to be `n[0]` then `n[1]` and finally `n[2]`.
We have chosen this way of doing to reuse the 3D code in a 2D framework.
Indeed having the last dimension in the slower rotating index does not penalize the loops writting.

As an example, we here is how we access the memory
As an example, we here is how we access the memory for a scalar field:

```cpp
double* data =(double*) flups_malloc(n[0] * n[1] * n[2] * sizeof(double));
@@ -144,6 +148,9 @@ for(int iz=0; iz<n[2]; iz++){
flups_free(data);
```
Vector components are treated using a leading index of arrays (slowest rotating index), and thus corresponds to an additional outer loop.
#### FLUPS in a nutshell
To use the solver, you first need to create a topology
```cpp
@@ -183,12 +190,35 @@ flups_cleanup(mysolver);
flups_topo_free(topo);
```

#### Advanced usage
#### Code examples
Examples of usage of FLUPS in C programs are provided in the `./sample` subfolder.
This includes:
* `validation`: the exe used for validation and scalability analysis (see our reference publication). This also constitutes an example of how to use FLUPS within a C++ client code, for the scalar Poisson equation.
* `solve_vtube`: another validation test case on a 2-D vortex tube. It may be used as an example on how to use FLUPS to solve the vector Poisson equation and the Biot-Savart mode.
* `solve_advanced_C`: an example showing how to embed flups in a C code, also showing how to use some advanced features (e.g. performing 3-D FFTs separately).


#### Make the most of the parallel implementation

FLUPS features hybrid distributed/shared memory capabilities, enabling the library to adapt to a variety of software/hardware configurations. Also, two types of communications schemes are available: all-to-all and non-blocking. The user can select one option or the other at compilation time, through the `COMM_NONBLOCK` flag.

The actual performance of the library (in terms of time-to-solution) depends a.o. on the number of unknowns per CPU, on the type of boundary conditions and on the architectures it runs on. We here provide some guidelines for the user to determine the optimal setup (see reference publication for more details):
- We highly recommend the use of distributed memory when possible, even if FLUPS can run in a pure OpenMP mode.
- Should you use shared memory (`OMP_NUM_THREADS>1`), each thread must be handled by a distinct core (no hyper threading). Computer nodes providing non-uniform memory accesses
- The all-to-all implementation should be considered as the default robust option. However, acceleration is possible using the non-blocking version, in particular when:
- the number of unknowns per core is high (~128^3)
- the total number of core is not too high (~< 10k)
- The mixed use of OpenMP and MPI is supported, and should only be considered in combination with the non-blocking implementation. However, the related performance is highly dependent on the computer architecture.

We encourage the user seeking for optimal performance to run short dedicated tests on the targeted architecture. The `validation` executable, when compiled with the `PROF` option, can be used to time the execution. A basic comparison of performance on a typical-size problem should involve at least:
1. the all-to-all implementation without thread
2. the non-blocking implementation without thread
3. the non-blocking implementation with 2 to 4 threads


#### Memory footprint
For the recommanded configuration of 128^3 unknowns per processor in full unbounded, we have measured the memory usage of FLUPS on a 2000 cores run:
- the all to all version uses ~530Mb (O.253kB/unknown)
- the all-to-all version uses ~530Mb (O.253kB/unknown)
- the non-blocking version uses ~560Mb (O.267kB/unknown)

<!--
8 changes: 8 additions & 0 deletions doc/documentation.html
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "https://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<body>
<a href="html/index.html">To the code documentation...</a> <br>
<a href="html/flups_8h.html">To FLUPS API documentation (flups.h)...</a> <br>
Notice that you first need to generate it by calling doxygen (see README).
</body>
</html>
12 changes: 12 additions & 0 deletions make_arch/make.docker_intel
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
CXX = mpiicpc
CXXFLAGS := -g -O3 -std=c++11 -qopenmp -restrict -fargument-noalias -DNDEBUG -DPROF -qopt-report=5 -qopt-report-phase=all
CC = mpiicc
CCFLAGS := -g -O3 -std=c++11 -qopenmp -restrict -fargument-noalias -DNDEBUG -DPROF -qopt-report=5 -qopt-report-phase=all
LDFLAGS += -qopenmp

FFTW_DIR := /soft/fftw-3.3.8-intel_2019.5
FFTW_LIB := ${FFTW_DIR}/lib
FFTW_INC := ${FFTW_DIR}/include
HDF5_DIR := /soft/hdf5-1.10.5-intel_2019.5
HDF5_LIB := ${HDF5_DIR}/lib
HDF5_INC := ${HDF5_DIR}/include
24 changes: 12 additions & 12 deletions samples/compareP3DFFT++/Makefile
Original file line number Diff line number Diff line change
@@ -1,23 +1,23 @@
################################################################################
# @copyright Copyright © UCLouvain 2019
# @copyright Copyright © UCLouvain 2020
#
# FLUPS is a Fourier-based Library of Unbounded Poisson Solvers.
#
# Copyright (C) <2019> <Universite catholique de Louvain (UCLouvain), Belgique>
# Copyright (C) <2020> <Universite catholique de Louvain (UCLouvain), Belgique>
#
# List of the contributors to the development of FLUPS, Description and complete License: see LICENSE file.
# List of the contributors to the development of FLUPS, Description and complete License: see LICENSE and NOTICE files.
#
# This program (FLUPS) is free software:
# you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version.
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
# http://www.apache.org/licenses/LICENSE-2.0
#
# You should have received a copy of the GNU General Public License
# along with this program (see COPYING file). If not,
# see <http://www.gnu.org/licenses/>.
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
################################################################################

12 changes: 8 additions & 4 deletions samples/compareP3DFFT++/main_compare++.cpp
Original file line number Diff line number Diff line change
@@ -121,9 +121,13 @@ int main(int argc, char *argv[]) {

const double h[3] = {L[0] / nglob[0], L[1] / nglob[1], L[2] / nglob[2]};

const BoundaryType mybc[3][2] = {PER, PER, //or ODD, ODD,
PER, PER,
PER, PER};
FLUPS_BoundaryType* mybc[3][2];
for(int id=0; id<3; id++){
for(int is=0; is<2; is++){
mybc[id][is] = (FLUPS_BoundaryType*) flups_malloc(sizeof(int)*1);
mybc[id][is][0] = PER;
}
}

if(comm_size!=nproc[0]*nproc[1]*nproc[2])
FLUPS_ERROR("Invalid number of procs",LOCATION);
@@ -148,7 +152,7 @@ int main(int argc, char *argv[]) {
Profiler* FLUprof = new Profiler(FLUPSprof);

// solver creation and init
Solver *mysolver = new Solver(topoIn, mybc, h, L, FLUprof);
Solver *mysolver = new Solver(topoIn, mybc, h, L, NOD, FLUprof);
mysolver->set_GreenType(CHAT_2);
double *solFLU = mysolver->setup(true);
// update the comm and the rank
24 changes: 12 additions & 12 deletions samples/compareP3DFFT/Makefile
Original file line number Diff line number Diff line change
@@ -1,23 +1,23 @@
################################################################################
# @copyright Copyright © UCLouvain 2019
# @copyright Copyright © UCLouvain 2020
#
# FLUPS is a Fourier-based Library of Unbounded Poisson Solvers.
#
# Copyright (C) <2019> <Universite catholique de Louvain (UCLouvain), Belgique>
# Copyright (C) <2020> <Universite catholique de Louvain (UCLouvain), Belgique>
#
# List of the contributors to the development of FLUPS, Description and complete License: see LICENSE file.
# List of the contributors to the development of FLUPS, Description and complete License: see LICENSE and NOTICE files.
#
# This program (FLUPS) is free software:
# you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version.
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
# http://www.apache.org/licenses/LICENSE-2.0
#
# You should have received a copy of the GNU General Public License
# along with this program (see COPYING file). If not,
# see <http://www.gnu.org/licenses/>.
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
################################################################################

16 changes: 10 additions & 6 deletions samples/compareP3DFFT/main_compare.cpp
Original file line number Diff line number Diff line change
@@ -31,13 +31,17 @@ int main(int argc, char *argv[]) {
//-------------------------------------------------------------------------
const int nglob[3] = {16, 16, 16};
const int nproc[3] = {1, 1, 2}; //nproc[0] has to be 1 //CAUTION FOR THIS: nproc[1]<=nproc[2] !!!
const double L[3] = {1., 1., 1.};;
const double L[3] = {1., 1., 1.};

const double h[3] = {L[0] / nglob[0], L[1] / nglob[1], L[2] / nglob[2]};

const BoundaryType mybc[3][2] = {PER, PER,
PER, PER,
PER, PER};
FLUPS_BoundaryType* mybc[3][2];
for(int id=0; id<3; id++){
for(int is=0; is<2; is++){
mybc[id][is] = (FLUPS_BoundaryType*) flups_malloc(sizeof(int)*1);
mybc[id][is][0] = PER;
}
}

unsigned char op_f[]="fft", op_b[]="tff";

@@ -51,14 +55,14 @@ int main(int argc, char *argv[]) {

// create a real topology
int FLUnmemIn[3],FLUnmemOUT[3];
Topology *topoIn = new Topology(0, 1, nglob, nproc, false, NULL, FLUPS_ALIGNMENT, comm);
FLUPS_Topology *topoIn = new FLUPS_Topology(0, 1, nglob, nproc, false, NULL, FLUPS_ALIGNMENT, comm);
const int nprocOut[3] = {1, 2, 1};
const int nglobOut[3] = {17, 32, 64};

std::string FLUPSprof = "compare_FLUPS_res" + std::to_string((int)(nglob[0]/L[0])) + "_nrank" + std::to_string(comm_size)+"_nthread" + std::to_string(omp_get_max_threads());
Profiler* FLUprof = new Profiler(FLUPSprof);

Solver *mysolver = new Solver(topoIn, mybc, h, L, FLUprof);
FLUPS_Solver *mysolver = new FLUPS_Solver(topoIn, mybc, h, L, NOD, FLUprof);

mysolver->set_GreenType(CHAT_2);
double *solFLU = mysolver->setup(true);
24 changes: 12 additions & 12 deletions samples/solve_advanced_C/Makefile
Original file line number Diff line number Diff line change
@@ -1,23 +1,23 @@
################################################################################
# @copyright Copyright © UCLouvain 2019
# @copyright Copyright © UCLouvain 2020
#
# FLUPS is a Fourier-based Library of Unbounded Poisson Solvers.
#
# Copyright (C) <2019> <Universite catholique de Louvain (UCLouvain), Belgique>
# Copyright (C) <2020> <Universite catholique de Louvain (UCLouvain), Belgique>
#
# List of the contributors to the development of FLUPS, Description and complete License: see LICENSE file.
# List of the contributors to the development of FLUPS, Description and complete License: see LICENSE and NOTICE files.
#
# This program (FLUPS) is free software:
# you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version.
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
# http://www.apache.org/licenses/LICENSE-2.0
#
# You should have received a copy of the GNU General Public License
# along with this program (see COPYING file). If not,
# see <http://www.gnu.org/licenses/>.
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
################################################################################

Loading

0 comments on commit eaf49ee

Please sign in to comment.