diff --git a/docs/source/configuration/advanced_configuration.md b/docs/source/configuration/advanced_configuration.md index 6bf78d487e..72003b678e 100644 --- a/docs/source/configuration/advanced_configuration.md +++ b/docs/source/configuration/advanced_configuration.md @@ -4,8 +4,38 @@ The documentation on [configuration](./configuration_basics.md) describes how to By default, Kedro is set up to use the [ConfigLoader](/kedro.config.ConfigLoader) class. Kedro also provides two additional configuration loaders with more advanced functionality: the [TemplatedConfigLoader](/kedro.config.TemplatedConfigLoader) and the [OmegaConfigLoader](/kedro.config.OmegaConfigLoader). Each of these classes are alternatives for the default `ConfigLoader` and have different features. The following sections describe each of these classes and their specific functionality in more detail. +## OmegaConfigLoader + +[OmegaConf](https://omegaconf.readthedocs.io/) is a Python library designed to handle and manage settings. It serves as a YAML-based hierarchical system to organise configurations, which can be structured to accommodate various sources, allowing you to merge settings from multiple locations. + +From Kedro 0.18.5 you can use the [`OmegaConfigLoader`](/kedro.config.OmegaConfigLoader) which uses `OmegaConf` to load data. + +```{note} +`OmegaConfigLoader` is under active development. It is available from Kedro version 0.18.5 with additional features due in later releases. Let us know if you have any feedback about the `OmegaConfigLoader` by joining the [Kedro community on Slack](https://slack.kedro.org/). +``` + +`OmegaConfigLoader` can load `YAML` and `JSON` files. Acceptable file extensions are `.yml`, `.yaml`, and `.json`. By default, any configuration files used by the config loaders in Kedro are `.yml` files. + +To use `OmegaConfigLoader` in your project, set the `CONFIG_LOADER_CLASS` constant in your [`src//settings.py`](../kedro_project_setup/settings.md): + +```python +from kedro.config import OmegaConfigLoader # new import + +CONFIG_LOADER_CLASS = OmegaConfigLoader +``` +### Advanced `OmegaConfigLoader` features +Some advanced use cases of `OmegaConfigLoader` are listed below: +- [How to do templating with the `OmegaConfigLoader`](#how-to-do-templating-with-the-omegaconfigloader) +- [How to use global variables with the `OmegaConfigLoader`](#how-to-use-global-variables-with-the-omegaconfigloader) +- [How to use resolvers in the `OmegaConfigLoader`](#how-to-use-resolvers-in-the-omegaconfigloader) +- [How to load credentials through environment variables](#how-to-load-credentials-through-environment-variables) + ## TemplatedConfigLoader +```{warning} +`ConfigLoader` and `TemplatedConfigLoader` have been deprecated since Kedro `0.18.12` and will be removed in Kedro `0.19.0`. Refer to the [migration guide for config loaders](./config_loader_migration.md) for instructions on how to update your code to use `OmegaConfigLoader`. +``` + Kedro provides an extension [TemplatedConfigLoader](/kedro.config.TemplatedConfigLoader) class that allows you to template values in configuration files. To apply templating in your project, set the `CONFIG_LOADER_CLASS` constant in your [`src//settings.py`](../kedro_project_setup/settings.md): ```python @@ -95,30 +125,9 @@ CONFIG_LOADER_ARGS = { If you specify both `globals_pattern` and `globals_dict` in `CONFIG_LOADER_ARGS`, the contents of the dictionary resulting from `globals_pattern` are merged with the `globals_dict` dictionary. In case of conflicts, the keys from the `globals_dict` dictionary take precedence. -## OmegaConfigLoader - -[OmegaConf](https://omegaconf.readthedocs.io/) is a Python library designed for configuration. It is a YAML-based hierarchical configuration system with support for merging configurations from multiple sources. - -From Kedro 0.18.5 you can use the [`OmegaConfigLoader`](/kedro.config.OmegaConfigLoader) which uses `OmegaConf` under the hood to load data. - -```{note} -`OmegaConfigLoader` is under active development. It was first available from Kedro 0.18.5 with additional features due in later releases. Let us know if you have any feedback about the `OmegaConfigLoader`. -``` - -`OmegaConfigLoader` can load `YAML` and `JSON` files. Acceptable file extensions are `.yml`, `.yaml`, and `.json`. By default, any configuration files used by the config loaders in Kedro are `.yml` files. - -To use `OmegaConfigLoader` in your project, set the `CONFIG_LOADER_CLASS` constant in your [`src//settings.py`](../kedro_project_setup/settings.md): - -```python -from kedro.config import OmegaConfigLoader # new import - -CONFIG_LOADER_CLASS = OmegaConfigLoader -``` - ## Advanced Kedro configuration This section contains a set of guidance for advanced configuration requirements of standard Kedro projects: - * [How to change which configuration files are loaded](#how-to-change-which-configuration-files-are-loaded) * [How to ensure non default configuration files get loaded](#how-to-ensure-non-default-configuration-files-get-loaded) * [How to bypass the configuration loading rules](#how-to-bypass-the-configuration-loading-rules) diff --git a/docs/source/configuration/configuration_basics.md b/docs/source/configuration/configuration_basics.md index 9e133f0e5e..9d53d49e2f 100644 --- a/docs/source/configuration/configuration_basics.md +++ b/docs/source/configuration/configuration_basics.md @@ -4,11 +4,24 @@ This section contains detailed information about Kedro project configuration, wh Kedro makes use of a configuration loader to load any project configuration files, and the available configuration loader classes are: +```{warning} +`ConfigLoader` and `TemplatedConfigLoader` have been deprecated since Kedro `0.18.12` and will be removed in Kedro `0.19.0`. Refer to the [migration guide for config loaders](./config_loader_migration.md) for instructions on how to update your code base to use `OmegaConfigLoader`. +``` + * [`ConfigLoader`](/kedro.config.ConfigLoader) * [`TemplatedConfigLoader`](/kedro.config.TemplatedConfigLoader) * [`OmegaConfigLoader`](/kedro.config.OmegaConfigLoader). -By default, Kedro uses the `ConfigLoader` and, in the following sections and examples, you can assume the default `ConfigLoader` is used, unless otherwise specified. The [advanced configuration documentation](./advanced_configuration.md) covers use of the [`TemplatedConfigLoader`](/kedro.config.TemplatedConfigLoader) and [`OmegaConfigLoader`](/kedro.config.OmegaConfigLoader) in more detail. +By default, Kedro uses the `ConfigLoader`. However, in projects created with Kedro `0.18.13` onwards, `OmegaConfigLoader` has been set as the config loader as the default in the project's `src//settings.py` file. +You can select which config loader you want to use in your project by modifying the `src//settings.py` like this: +```python +from kedro.config import OmegaConfigLoader + +CONFIG_LOADER_CLASS = OmegaConfigLoader +``` +The following sections and examples are valid for both, the `ConfigLoader` and the `OmegaConfigLoader`. The [advanced configuration documentation](./advanced_configuration.md) covers use of the [`TemplatedConfigLoader`](/kedro.config.TemplatedConfigLoader) +and the advanced use cases of the [`OmegaConfigLoader`](/kedro.config.OmegaConfigLoader) in more detail. + ## Configuration source The configuration source folder is [`conf`](../get_started/kedro_concepts.md#conf) by default. We recommend that you keep all configuration files in the default `conf` folder of a Kedro project. @@ -35,7 +48,8 @@ Do not add any local configuration to version control. ``` ## Configuration loading -Kedro-specific configuration (e.g., `DataCatalog` configuration for I/O) is loaded using a configuration loader class, by default, this is [`ConfigLoader`](/kedro.config.ConfigLoader). +Kedro-specific configuration (e.g., `DataCatalog` configuration for I/O) is loaded using a configuration loader class, by default, this is [`ConfigLoader`](/kedro.config.ConfigLoader) for +projects created with Kedro `0.18.13` or older and has been set to `OmegaConfigLoader` for projects created with Kedro `0.18.13` onwards. When you interact with Kedro through the command line, e.g. by running `kedro run`, Kedro loads all project configuration in the configuration source through this configuration loader. The loader recursively scans for configuration files inside the `conf` folder, firstly in `conf/base` (`base` being the default environment) and then in `conf/local` (`local` being the designated overriding environment). @@ -61,7 +75,7 @@ Configuration files will be matched according to file name and type rules. Suppo ### Configuration patterns Under the hood, the Kedro configuration loader loads files based on regex patterns that specify the naming convention for configuration files. These patterns are specified by `config_patterns` in the configuration loader classes. -By default those patterns are set as follows for the configuration of catalog, parameters, logging, credentials, and globals: +By default, those patterns are set as follows for the configuration of catalog, parameters, logging, credentials: ```python config_patterns = { @@ -69,7 +83,6 @@ config_patterns = { "parameters": ["parameters*", "parameters*/**", "**/parameters*"], "credentials": ["credentials*", "credentials*/**", "**/credentials*"], "logging": ["logging*", "logging*/**", "**/logging*"], - "globals": ["globals*", "globals*/**", "**/globals*"], } ``` @@ -80,10 +93,10 @@ If you want to change the way configuration is loaded, you can either [customise This section contains a set of guidance for the most common configuration requirements of standard Kedro projects: * [How to change the setting for a configuration source folder](#how-to-change-the-setting-for-a-configuration-source-folder) -* [How to change the configuration source folder at run time](#how-to-change-the-configuration-source-folder-at-runtime) +* [How to change the configuration source folder at runtime](#how-to-change-the-configuration-source-folder-at-runtime) * [How to read configuration from a compressed file](#how-to-read-configuration-from-a-compressed-file) * [How to access configuration in code](#how-to-access-configuration-in-code) -* [How to specify additional configuration environments ](#how-to-specify-additional-configuration-environments) +* [How to specify additional configuration environments](#how-to-specify-additional-configuration-environments) * [How to change the default overriding environment](#how-to-change-the-default-overriding-environment) * [How to use only one configuration environment](#how-to-use-only-one-configuration-environment) @@ -145,12 +158,12 @@ Note that for both the `tar.gz` and `zip` file the following structure is expect To directly access configuration in code, for example to debug, you can do so as follows: ```python -from kedro.config import ConfigLoader +from kedro.config import OmegaConfigLoader from kedro.framework.project import settings # Instantiate a ConfigLoader with the location of your project configuration. conf_path = str(project_path / settings.CONF_SOURCE) -conf_loader = ConfigLoader(conf_source=conf_path) +conf_loader = OmegaConfigLoader(conf_source=conf_path) # This line shows how to access the catalog configuration. You can access other configuration in the same way. conf_catalog = conf_loader["catalog"] diff --git a/docs/source/configuration/credentials.md b/docs/source/configuration/credentials.md index 8252c9d76f..7bb2af73cc 100644 --- a/docs/source/configuration/credentials.md +++ b/docs/source/configuration/credentials.md @@ -7,14 +7,15 @@ Credentials configuration can be used on its own directly in code or [fed into t If you would rather store your credentials in environment variables instead of a file, you can use the `OmegaConfigLoader` [to load credentials from environment variables](advanced_configuration.md#how-to-load-credentials-through-environment-variables) as described in the advanced configuration chapter. ## How to load credentials in code + Credentials configuration can be loaded the same way as any other project configuration using any of the configuration loader classes: `ConfigLoader`, `TemplatedConfigLoader`, and `OmegaConfigLoader`. -The following examples all use the default `ConfigLoader` class. +The following examples are valid for both, the `ConfigLoader` and the `OmegaConfigLoader`. ```python from pathlib import Path -from kedro.config import ConfigLoader +from kedro.config import OmegaConfigLoader from kedro.framework.project import settings # Substitute with the [root folder for your project](https://docs.kedro.org/en/stable/tutorial/spaceflights_tutorial.html#terminology) @@ -30,11 +31,11 @@ Calling `conf_loader[key]` in the example above throws a `MissingConfigException ```python from pathlib import Path -from kedro.config import ConfigLoader, MissingConfigException +from kedro.config import OmegaConfigLoader, MissingConfigException from kedro.framework.project import settings conf_path = str(Path() / settings.CONF_SOURCE) -conf_loader = ConfigLoader(conf_source=conf_path) +conf_loader = OmegaConfigLoader(conf_source=conf_path) try: credentials = conf_loader["credentials"] diff --git a/docs/source/configuration/parameters.md b/docs/source/configuration/parameters.md index 61c6ff0e9c..8f50636088 100644 --- a/docs/source/configuration/parameters.md +++ b/docs/source/configuration/parameters.md @@ -76,14 +76,14 @@ You can use `add_feed_dict()` to inject any other entries into your `DataCatalog Parameters project configuration can be loaded by any of the configuration loader classes: `ConfigLoader`, `TemplatedConfigLoader`, and `OmegaConfigLoader`. -The following examples all make use of the default `ConfigLoader` class. +The following examples all make use of the `OmegaConfigLoader` class. ```python -from kedro.config import ConfigLoader +from kedro.config import OmegaConfigLoader from kedro.framework.project import settings conf_path = str(project_path / settings.CONF_SOURCE) -conf_loader = ConfigLoader(conf_source=conf_path) +conf_loader = OmegaConfigLoader(conf_source=conf_path) parameters = conf_loader["parameters"] ``` @@ -92,11 +92,11 @@ This loads configuration files from any subdirectories in `conf` that have a fil Calling `conf_loader[key]` in the example above will throw a `MissingConfigException` error if no configuration files match the given key. But if this is a valid workflow for your application, you can handle it as follows: ```python -from kedro.config import ConfigLoader, MissingConfigException +from kedro.config import OmegaConfigLoader, MissingConfigException from kedro.framework.project import settings conf_path = str(project_path / settings.CONF_SOURCE) -conf_loader = ConfigLoader(conf_source=conf_path) +conf_loader = OmegaConfigLoader(conf_source=conf_path) try: parameters = conf_loader["parameters"] diff --git a/docs/source/development/automated_testing.md b/docs/source/development/automated_testing.md index 6efcfa73b9..5d23609d30 100644 --- a/docs/source/development/automated_testing.md +++ b/docs/source/development/automated_testing.md @@ -63,14 +63,14 @@ Now that you have a place to put your tests, you can create an example test in t ``` import pytest -from kedro.config import ConfigLoader +from kedro.config import OmegaConfigLoader from kedro.framework.context import KedroContext from kedro.framework.hooks import _create_hook_manager @pytest.fixture def config_loader(): - return ConfigLoader(conf_source=str(Path.cwd())) + return OmegaConfigLoader(conf_source=str(Path.cwd())) @pytest.fixture