ADIOS4DOLFINx is an extension for DOLFINx to checkpoint meshes, meshtags and functions using ADIOS 2.
The code uses the ADIOS2 Python-wrappers to write DOLFINx objects to file, supporting N-to-M (recoverable) and N-to-N (snapshot) checkpointing. See: Checkpointing in DOLFINx - FEniCS 23 or the examples in the Documentation for more information.
For scalability, the code uses MPI Neighbourhood collectives for communication across processes.
As the usage of high performance computing clusters increases, more and more large-scale, long-running simulations are deployed.
The need for storing intermediate solutions from such simulations are crucial, as the HPC system might crash, or the simulation might crash or exceed the alloted computational budget.
Having a checkpoint of related variables, such as the solutions to partial differential equations (PDEs) is therefore essential.
The adios4dolfinx library extends the DOLFINx computational framework for solving PDEs with checkpointing functionality, such that immediate solutions and mesh information can be stored and re-used in another simulation.
Compatibility with DOLFINx:
- ADIOS4DOLFINx v0.9.0 is compatible with DOLFINx v0.9.x
- ADIOS4DOLFINx v0.8.1 is compatible with DOLFINx v0.8.x
- ADIOS4DOLFINx v0.7.3 is compatible with DOLFINx v0.7.x
The library depends on the Python-interface of DOLFINx and an MPI-build of ADIOS2.
Therefore ADIOS2 should not be install through PYPI/pip, but has to be installed through Conda, Spack or from source.
Important
ADIOS2<2.10.2 does not work properly with numpy>=2.0.0. Everyone is adviced to use the newest version of ADIOS2.
This is for instance available through conda or the ghcr.io/fenics/dolfinx/dolfinx:nightly Docker-image.
An MPI build of ADIOS2 is installed in the official DOLFINx containers, and thus there are no additional dependencies required to install adios4dolfinx
on top of DOLFINx in these images.
Create a Docker container, named for instance dolfinx-checkpoint.
Use the nightly tag to get the main branch of DOLFINx, or stable to get the latest stable release
docker run -ti -v $(pwd):/root/shared -w /root/shared --name=dolfinx-checkpoint ghcr.io/fenics/dolfinx/dolfinx:nightlyFor the latest version compatible with nightly (with the ability to run the test suite), use
python3 -m pip install adios4dolfinx[test]@git+https://github.com/jorgensd/adios4dolfinx@mainIf you are using the stable image, you can install adios4dolfinx from PYPI with
python3 -m pip install adios4dolfinx[test]This docker container can be opened with
docker container start -i dolfinx-checkpointat a later instance
Note
Conda supports the stable release of DOLFINx, and thus the appropriate version should be installed, see the section above for more details.
Following is a minimal recipe of how to install adios4dolfinx, given that you have conda installed on your system.
conda create -n dolfinx-checkpoint python=3.10
conda activate dolfinx-checkpoint
conda install -c conda-forge adios4dolfinxNote
Remember to download the appropriate version of adios4dolfinx from Github adios4dolfinx: Releases
To run the test suite, you should also install ipyparallel, pytest and coverage, which can all be installed with conda
conda install -c conda-forge ipyparallel pytest coverage- Reading and writing meshes, using
adios4dolfinx.read/write_mesh - Reading and writing meshtags associated to meshes
adios4dolfinx.read/write_meshtags - Reading checkpoints for any element (serial and parallel, arbitrary number of functions and timesteps per file). Use
adios4dolfinx.read/write_function. - Writing standalone function checkpoints relating to "original meshes", i.e. meshes read from
XDMFFile. Useadios4dolfinx.write_function_on_input_meshfor this. - Store mesh partitioning and re-read the mesh with this information, avoiding calling SCOTCH, Kahip or Parmetis.
Important
For checkpoints written with write_function to be valid, you first have to store the mesh with write_mesh to the checkpoint file.
Important
A checkpoint file supports multiple functions and multiple time steps, as long as the functions are associated with the same mesh
Important
Only one mesh per file is allowed
The repository contains many documented examples of usage, in the docs-folder, including
- Reading and writing mesh checkpoints
- Storing mesh partitioning data
- Writing mesh-tags to a checkpoint
- Reading and writing function checkpoints
- Checkpoint on input mesh Further examples can be found at ADIOS4DOLFINx examples
Warning
If you are using v0.7.2, you are adviced to upgrade to v0.7.3, as it contains som crucial fixes for openmpi.
Only checkpoints for Lagrange or DG functions are supported from legacy DOLFIN
- Reading meshes from the DOLFIN HDF5File-format
- Reading checkpoints from the DOLFIN HDF5File-format (one checkpoint per file only)
- Reading checkpoints from the DOLFIN XDMFFile-format (one checkpoint per file only, and only uses the
.h5file)
See the API for more information.
This library uses pytest for testing.
To execute the tests, one should first install the library and its dependencies, as listed above.
Then, can execute all tests by calling
python3 -m pytest .Some tests check the capability of reading data created with the legacy version of DOLFIN. To create this dataset, start a docker container with legacy DOLFIN, for instance:
docker run -ti -v $(pwd):/root/shared -w /root/s
hared --rm ghcr.io/scientificcomputing/fenics:2024-02-19Then, inside this container, call
python3 ./tests/create_legacy_data.py --output-dir=legacySome tests check the capability to read data generated by adios4dolfinx<0.7.2.
To generate data for these tests use the following commands:
docker run -ti -v $(pwd):/root/shared -w /root/shared --rm ghcr.io/fenics/dolfinx/dolfinx:v0.7.3Then, inside the container, call
python3 -m pip install adios4dolfinx==0.7.1
python3 ./tests/create_legacy_checkpoint.py --output-dir=legacy_checkpointThe long term plan is to get this library merged into DOLFINx (rewritten in C++ with appropriate Python-bindings).