Libraries for coupling Elmer FEM and RapidCFD + test cases. This library is based on EOF-Library but adapted for use with RapidCFD.
- Ubuntu 20.04 or Ubuntu 18.04
- EOF-Library with OpenFOAM-7 and Elmer FEM (Recommended)
Make sure that each step is executed correctly before proceding to the next one.
1.1
sudo apt-get update
sudo apt-get install git gmsh cmake libblas-dev liblapack-dev build-essential curl flex gfortran libboost-system-dev libboost-thread-dev -y
sudo apt-get install bison automake zlib1g-dev libopenmpi-dev openmpi-bin libreadline-dev libncurses-dev libxt-dev freeglut3-dev -y
1.2a Ubuntu 18.04
sudo apt-get install qt4-dev-tools libqt4-dev libqt4-opengl-dev libqtwebkit-dev -y
1.2b Ubuntu 20.04
sudo apt-get install qt5-default gcc-7 g++-7 -y
sudo update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-9 800 --slave /usr/bin/g++ g++ /usr/bin/g++-9
sudo update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-7 900 --slave /usr/bin/g++ g++ /usr/bin/g++-7
CUDA Toolkit 10.1 tested on Ubuntu 18.04 CUDA Toolkit 11.1 tested on Ubuntu 20.04 See CUDA Toolkit Archive for documentation
- CUDA Toolkit 10.1 on Ubuntu 18.04
wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/cuda-ubuntu1804.pin
sudo mv cuda-ubuntu1804.pin /etc/apt/preferences.d/cuda-repository-pin-600
sudo apt-key adv --fetch-keys https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/7fa2af80.pub
sudo add-apt-repository "deb http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/ /"
sudo apt-get update
sudo apt-get install -y cuda-toolkit-10-1
export PATH=$PATH:/usr/local/cuda-10.1/bin
export CUDADIR=/usr/local/cuda-10.1
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/cuda-10.1/lib64
- CUDA Toolkit 11.1 on Ubuntu 20.04
wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/cuda-ubuntu2004.pin
sudo mv cuda-ubuntu2004.pin /etc/apt/preferences.d/cuda-repository-pin-600
sudo apt-key adv --fetch-keys https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/7fa2af80.pub
sudo add-apt-repository "deb https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/ /"
sudo apt-get update
sudo apt-get -y install cuda-toolkit-11-1
sudo apt-get update
export PATH=$PATH:/usr/local/cuda-11.1/bin
export CUDADIR=/usr/local/cuda-11.1
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/cuda-11.1/lib64
3.1
cd $HOME
mkdir RapidCFD
cd RapidCFD
git clone https://github.com/Atizar/RapidCFD-dev
wget -O - http://dl.openfoam.org/third-party/7 | tar xvz
mv ThirdParty-7-version-7 ThirdParty-dev
export WM_PROJECT=RapidCFD
export WM_PROJECT_VERSION=dev
export FOAM_INST_DIR=$HOME/$WM_PROJECT
3.2a With CUDA 10.0 Openmpi version should match with the OS (Ubuntu 18.04) by default
3.2b With CUDA 11.1 Make sure to patch file
$HOME/RapidCFD/RapidCFD-dev/etc/config/settings.sh
On line #403 change
export FOAM_MPI=openmpi-4.0.2
to
export FOAM_MPI=openmpi-4.0.5
3.3 Update compilation rules according to your GPU Compute Capability Patch file
$HOME/RapidCFD/RapidCFD-dev/wmake/rules/linux64Nvcc/c
according to your compute capability. For example if your compute capability is 7.5, then on line #5
change -arch
value
Additionally patch
$HOME/RapidCFD/RapidCFD-dev/wmake/rules/linux64Nvcc/c++
and change line #10
3.4a With CUDA 10.0
change -arch
value
3.4b With CUDA 11.1
change -arch
and -std
value
3.5 Source project bashrc
source $FOAM_INST_DIR/$WM_PROJECT-$WM_PROJECT_VERSION/etc/bashrc
3.6 Update ThirdParty compilation configuration by applying patch to file
$HOME/RapidCFD/ThirdParty-dev/Allwmake
On line #85 change
configOpt="--with-sge"
to
configOpt="--with-cuda"
and on line #100 change
--enable-mpi-fortran=none \
to
--enable-mpi-fortran \
3.7 Navigate to ThirdParty installation directory
cd ThirdParty-dev
3.8a With CUDA 10.1
wget -O - https://download.open-mpi.org/release/open-mpi/v4.0/openmpi-4.0.2.tar.gz | tar xvz
3.8b With CUDA 11.1
wget -O - https://download.open-mpi.org/release/open-mpi/v4.0/openmpi-4.0.5.tar.gz | tar xvz
3.9 Installation of ThirdParty
./Allwmake
3.10 If you plan to use ERapid-Library with only one GPU, then make sure to apply patch to file
$HOME/RapidCFD/RapidCFD-dev/src/Pstream/mpi/UPstream.C
On line #87 change
if (numprocs <= 1)
to
if (numprocs < 1)
3.11 Installation of RapidCFD
cd ../RapidCFD-dev
export WM_NCOMPPROCS=16
./Allwmake
4.1
cd ../..
mkdir elmerRapid
cd elmerRapid
git clone https://github.com/ElmerCSC/elmerfem
mkdir build
cd build
4.2a With CUDA 10.1
cmake -DWITH_MPI=TRUE --with-mpi-inc-dir=$HOME/RapidCFD/ThirdParty-dev/openmpi-4.0.2/ -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX=../install ../elmerfem
4.2b With CUDA 11.1
cmake -DWITH_MPI=TRUE --with-mpi-inc-dir=$HOME/RapidCFD/ThirdParty-dev/openmpi-4.0.5/ -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX=../install ../elmerfem
4.3 Installation of Elmer
make -j install
cd ../..
git clone https://github.com/didzis-berenis/ERapid-Library
. ERapid-Library/etc/bashrc
export ELMER_HOME=$HOME/elmerRapid/install
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$ELMER_HOME/lib
export PATH=$PATH:$ELMER_HOME/bin
eofCompile
- Configure tools for switching between OpenFOAM-7 and RapidCFD
cp $HOME/ERapid-Library/setupTools/PROJECTDIR $HOME/
cp $HOME/ERapid-Library/setupTools/sourceOF $HOME/
cp $HOME/ERapid-Library/setupTools/sourceRapid $HOME/
'export OLD_PATH=$PATH' >> $HOME/.bashrc
'export OLD_LD_LIBRARY_PATH=$LD_LIBRARY_PATH' >> $HOME/.bashrc
'export WM_PROJECT=OpenFOAM' >> $HOME/.bashrc
'export WM_PROJECT_VERSION=7' >> $HOME/.bashrc
'export FOAM_INST_DIR=$HOME/$WM_PROJECT' >> $HOME/.bashrc
'export ELMER_HOME=$HOME/elmer/install' >> $HOME/.bashrc
'export EOF_HOME=$HOME/EOF-Library' >> $HOME/.bashrc
'source $HOME/PROJECTDIR' >> $HOME/.bashrc
- (Optional) Decrease solver log amount.
In file
$HOME/RapidCFD/RapidCFD-dev/etc/controlDict
change value ofGAMGAgglomeration
,SolverPerformance
andlduMatrix
to 0
- Before first use make sure to compile solvers
source $HOME/sourceRapid
cd ERapid-Library/solvers
./Allwmake
- Create an empty folder
cd ../
mkdir runs
cd runs
- Copy test simulation
cp -r tests/mhdFurnace runs
- Prepare case
cd runs/mhdFurnace
. sourceOF
gmsh -3 furnace.geo -o furnace.msh -format msh2
gmshToFoam furnace.msh
changeDictionary
decomposePar
-
Extract Elmer mesh (or make your own with Salome and salomeToElmer script)
tar -xf meshElmer.tar.xz
-
(Recommended) Create O2E pair files with EOF-Library before running the case with ERapid-Library
-
Run rapidCFD simulation on 2 GPUs and Elmer simulation on 6 CPU cores/threads:
source $HOME/sourceRapid
ElmerGrid 2 2 meshElmer -metis 6
mpirun -np 2 MHDPimpleAverage -parallel -devices "(0 1)" : -np 6 ElmerSolver_mpi case.sif > log.txt &
- Alternatively run rapidCFD simulation on 1 GPU and Elmer simulation on 4 CPU cores/threads:
source $HOME/sourceRapid
ElmerGrid 2 2 meshElmer -metis 4
mpirun -oversubscribe -np 1 MHDPimpleAverage -parallel : -np 4 ElmerSolver_mpi case.sif > log.txt &
- Simulation results will appear in host system runs folder