From 83dbe1ad92be580fa26412e5db4d1f370ec91c7a Mon Sep 17 00:00:00 2001 From: Sebastiaan Huber Date: Tue, 12 Dec 2023 14:59:59 +0100 Subject: [PATCH] Docs: Add topic section on storage (#6207) Gives overview of the storage plugins that ship with `aiida-core` with their strengths and weaknesses. This can be referred to from other parts of the docs when recommending which storage to choose. --- docs/source/topics/index.rst | 1 + docs/source/topics/storage.rst | 193 +++++++++++++++++++++++++++++++++ 2 files changed, 194 insertions(+) create mode 100644 docs/source/topics/storage.rst diff --git a/docs/source/topics/index.rst b/docs/source/topics/index.rst index 46d80b482a..d959055641 100644 --- a/docs/source/topics/index.rst +++ b/docs/source/topics/index.rst @@ -14,6 +14,7 @@ Topics data_types database repository + storage plugins schedulers transport diff --git a/docs/source/topics/storage.rst b/docs/source/topics/storage.rst new file mode 100644 index 0000000000..8450c22ec5 --- /dev/null +++ b/docs/source/topics/storage.rst @@ -0,0 +1,193 @@ +.. _topics:storage: + +******* +Storage +******* + +Each AiiDA profile defines a *storage* which is where all data in the provenance graph is stored. +Typically, a storage consists of a :ref:`database ` and a :ref:`repository `. +The provenance graph data itself is mostly persisted to the database, whereas files attached to nodes (or other binary content) are stored in the repository. + +By default, the storage consists of a PostgreSQL database and a `disk-objectstore container `_ for the file repository. +As of AiiDA 2.0, however, this storage can be customized through plugins, meaning other databases and file stores can be used if desired. +AiiDA ships itself with a number of storage plugins that each have their own strengths and weaknesses. +This section gives an overview of these storage plugins with suggestions of when to use them. + +.. grid:: 1 2 2 2 + :gutter: 3 + + .. grid-item-card:: ``core.psql_dos`` + :link: topics:storage:psql_dos + :link-type: ref + + *Default storage for production projects that require performance.* + + :fa:`database;mr-1` PostgreSQL database + + :fa:`file;mr-1` ``disk-objectstore`` container + + :fa:`plus;mr-1` **Strengths**: + + - Supports all of AiiDA's functionality + - Good performance + - Automatic database migrations + + :fa:`minus;mr-1` **Weaknesses**: + + - Requires a service running (PostgreSQL) + + .. grid-item-card:: ``core.sqlite_dos`` + :link: topics:storage:sqlite_dos + :link-type: ref + + *Easy to set up storage for tests, demos and experimenting.* + + :fa:`database;mr-1` SQLite database + + :fa:`file;mr-1` ``disk-objectstore`` container + + :fa:`plus;mr-1` **Strengths**: + + - Easy to set up and backup + - Requires no running services + + :fa:`minus;mr-1` **Weaknesses**: + + - Performance of SQLite is inferior to PostgreSQL + - Some ``QueryBuilder`` functionality is not supported :fa:`asterisk` + + .. grid-item-card:: ``core.sqlite_zip`` + :link: topics:storage:sqlite_zip + :link-type: ref + + *Storage contained in single ZIP file used for export archives.* + + :fa:`database;mr-1` SQLite database + + :fa:`file;mr-1` ``disk-objectstore`` container + + :fa:`plus;mr-1` **Strengths**: + + - Easy to set up + - Requires no running services + + :fa:`minus;mr-1` **Weaknesses**: + + - Read-only + - Some ``QueryBuilder`` functionality is not supported :fa:`asterisk` + + .. grid-item-card:: ``core.sqlite_temp`` + :link: topics:storage:sqlite_temp + :link-type: ref + + *Temporary storage mostly used for unit testing or demonstrations.* + + :fa:`database;mr-1` SQLite database + + :fa:`file;mr-1` Sandbox directory + + :fa:`plus;mr-1` **Strengths**: + + - Requires no running services + - Automatic cleanup + + :fa:`minus;mr-1` **Weaknesses**: + + - Storage is deleted once session is closed + - Some ``QueryBuilder`` functionality is not supported :fa:`asterisk` + +.. note:: + + :fa:`asterisk;mr-1` The following features of the ``QueryBuilder`` are known to not be supported for SQLite-based storages: + + * Use of ``contains`` in ``filters`` + * The ``get_creation_statistics`` method + + + +.. _topics:storage:psql_dos: + +``core.psql_dos`` +================= + +The ``core.psql_dos`` storage plugin is the default and is recommended for all production work. +It uses PostgreSQL for the database and the disk-objectstore for the file repository. +To create a profile using this storage plugin, run: + +.. code-block:: console + + verdi profile setup core.psql_dos + +The command requires the PostgreSQL database to already exist and to be able to connect to it. + +.. tip:: + + Try the ``verdi quicksetup`` command to have the PostgreSQL database automatically created. + Certain systems require root access to do so, causing the command to fail if it cannot obtain root access. + In this case, the database should be created manually (see :ref:`intro:install:database` for details). + Once created, a profile can be created using the database with the command ``verdi profile setup core.psql_dos``. + + +.. _topics:storage:sqlite_dos: + +``core.sqlite_dos`` +=================== + +The ``core.sqlite_dos`` storage plugin is an alternative to the ``core.psql_dos`` storage for use cases where performance is not critical. +Instead of a PostgreSQL database, it uses SQLite. +This makes it easier to set up as it does not require a running service, as the SQLite database is just a file on disk. + +A fully operational profile using this storage plugin can be created with a single command: + +.. code-block:: console + + verdi profile setup core.sqlite_dos -n --profile --email + +replacing ```` with the desired name for the profile and ```` with the email for the default user. + +The SQLite database and disk-objectstore container are both stored in the directory specified by the ``--filepath`` option of the ``verdi profile setup core.sqlite_dos`` command. +By default, this is a folder inside the directory defined by the ``$AIIDA_PATH/repository`` of the form ``sqlite_dos_{UUID}``, where the suffix is randomly generated hexadecimal UUID. +An example of an automated generated directory is ``.aiida/repository/sqlite_dos_962e87af09b746c985335cb77acaa553``. + +.. note:: + + The ``$AIIDA_PATH`` environment variable :ref:`determines the location of the configuration directory `, and defaults to ``.aiida`` in the user's home folder + + +.. _topics:storage:sqlite_zip: + +``core.sqlite_zip`` +=================== + +The `core.sqlite_zip` is a storage plugin that is used to create export archives. +It functions more or less identical to the `core.sqlite_dos` plugin, as it uses an SQLite database and a disk-objectstore container, except everything is bundled up in a `zip archive `_. + +The storage plugin is not suited for normal use, because once the archive is created, it becomes read-only. +However, since otherwise it functions like normal storage plugins, a profile can be created with it that make it easy to explore its contents: + +.. code-block:: console + + verdi profile setup core.sqlite_zip -n --profile --filepath + +replacing ```` with the desired name for the profile and ```` the path to the archive file. +The created profile can now be loaded like any other profile, and the contents of the provenance graph can be explored as usual. + + +.. _topics:storage:sqlite_temp: + +``core.sqlite_temp`` +==================== + +The ``core.sqlite_temp`` storage plugin utilises an in-memory SQLite database and sandbox folder to store data. +The data is automatically destroyed as soon as the profile is garbage collected, which is either when it is unloaded, or the Python interpreter is shut down. +This makes this storage plugin primarily useful for demonstration and testing purposes, whereby no persistent storage is required. + +A new temporary profile can be created and loaded as follows: + +.. code-block:: python + + from aiida import load_profile + from aiida.storage.sqlite_temp import SqliteTempBackend + + temp_profile = SqliteTempBackend.create_profile('temp-profile') + load_profile(temp_profile, allow_switch=True)