-
Notifications
You must be signed in to change notification settings - Fork 94
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Update logging and include examples (#492)
- Loading branch information
1 parent
b2e7530
commit 51c0872
Showing
17 changed files
with
527 additions
and
118 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -29,3 +29,4 @@ dist/ | |
| doc/_build/ | ||
| doc/_static/notebooks/*.html | ||
| .ipynb_checkpoints | ||
| outputs/ | ||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,286 @@ | ||
| { | ||
| "cells": [ | ||
| { | ||
| "cell_type": "markdown", | ||
| "metadata": {}, | ||
| "source": [ | ||
| "# Calliope logging examples\n", | ||
| "\n", | ||
| "In this notebook, we will look at ways of capturing calliope logging outputs and printing them to the console or to file." | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "code", | ||
| "execution_count": null, | ||
| "metadata": {}, | ||
| "outputs": [], | ||
| "source": [ | ||
| "import logging\n", | ||
| "from pathlib import Path\n", | ||
| "\n", | ||
| "import calliope" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "markdown", | ||
| "metadata": {}, | ||
| "source": [ | ||
| "## Using internal Calliope functionality\n", | ||
| "The `calliope.set_log_verbosity` method allows you to quickly set the logging level across all calliope loggers.\n", | ||
| "It doesn't require you to know anything about the `logging` package, just the available [logging levels](https://docs.python.org/3/library/logging.html#logging-levels)." | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "code", | ||
| "execution_count": null, | ||
| "metadata": {}, | ||
| "outputs": [], | ||
| "source": [ | ||
| "# This is the default log verbosity that is set on importing calliope.\n", | ||
| "# It will print the WARNING and INFO log levels to the console\n", | ||
| "# and it will print the solver output (which is otherwise at the DEBUG log level)\n", | ||
| "calliope.set_log_verbosity(\"info\")\n", | ||
| "m = calliope.examples.national_scale()\n", | ||
| "m.build()\n", | ||
| "m.solve()" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "code", | ||
| "execution_count": null, | ||
| "metadata": {}, | ||
| "outputs": [], | ||
| "source": [ | ||
| "# This will print WARNING and INFO log levels to the console but *NOT* the log solver output\n", | ||
| "calliope.set_log_verbosity(\"info\", include_solver_output=False)\n", | ||
| "m = calliope.examples.national_scale()\n", | ||
| "m.build()\n", | ||
| "m.solve()" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "code", | ||
| "execution_count": null, | ||
| "metadata": {}, | ||
| "outputs": [], | ||
| "source": [ | ||
| "# You can set the log verbosity to print all DEBUG level logs to the console\n", | ||
| "calliope.set_log_verbosity(\"debug\")\n", | ||
| "m = calliope.examples.national_scale()\n", | ||
| "m.build()\n", | ||
| "m.solve()" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "markdown", | ||
| "metadata": {}, | ||
| "source": [ | ||
| "## Adding your own console logging handler\n", | ||
| "If the `calliope.set_log_verbosity` method isn't providing you with enough flexibility then you can add your own logging `handlers`" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "code", | ||
| "execution_count": null, | ||
| "metadata": {}, | ||
| "outputs": [], | ||
| "source": [ | ||
| "# Grab the calliope logger, which will also automatically all the child loggers (e.g., `calliope.core.model`).\n", | ||
| "logger = logging.getLogger(\"calliope\")\n", | ||
| "\n", | ||
| "# Remove existing handlers (i.e., those introduced by `calliope.set_log_verbosity` above)\n", | ||
| "logger.handlers.clear()\n", | ||
| "\n", | ||
| "# You can define your own custom formatter.\n", | ||
| "# See https://docs.python.org/3/library/logging.html#logrecord-attributes for available attributes.\n", | ||
| "formatter = logging.Formatter(\"%(asctime)s - %(name)s - %(levelname)s - %(message)s\")\n", | ||
| "\n", | ||
| "# Add a ConsoleHandler (this is what `calliope.set_log_verbosity` is doing under the hood)\n", | ||
| "console_handler = logging.StreamHandler()\n", | ||
| "console_handler.setLevel(\n", | ||
| " logging.INFO\n", | ||
| ") # In this example, we only want to see warnings in the console\n", | ||
| "console_handler.setFormatter(formatter)\n", | ||
| "logger.addHandler(console_handler)\n", | ||
| "\n", | ||
| "# You can also use logging in your scripts to add more information\n", | ||
| "logger.info(\"Loading the national-scale example model\")\n", | ||
| "m = calliope.examples.national_scale()\n", | ||
| "\n", | ||
| "logger.info(\"Building the national-scale example model optimisation problem\")\n", | ||
| "m.build()\n", | ||
| "\n", | ||
| "logger.info(\"Solving the national-scale example model optimisation problem\")\n", | ||
| "m.solve()" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "markdown", | ||
| "metadata": {}, | ||
| "source": [ | ||
| "## Adding your own file logging handler\n", | ||
| "You may find it more practical to store logging information in files, particularly if you are running your model on a remote device or if you have a *very* large model.\n", | ||
| "\n", | ||
| "Then, you can search through your logs using your favourite IDE." | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "code", | ||
| "execution_count": null, | ||
| "metadata": {}, | ||
| "outputs": [], | ||
| "source": [ | ||
| "# Grab the calliope logger, which will also automatically all the child loggers (e.g., `calliope.core.model`).\n", | ||
| "logger = logging.getLogger(\"calliope\")\n", | ||
| "\n", | ||
| "# Remove existing handlers (i.e., those introduced earlier in this notebook)\n", | ||
| "logger.handlers.clear()\n", | ||
| "\n", | ||
| "# You can define your own custom formatter.\n", | ||
| "# See https://docs.python.org/3/library/logging.html#logrecord-attributes for available attributes.\n", | ||
| "formatter = logging.Formatter(\"%(asctime)s - %(name)s - %(levelname)s - %(message)s\")\n", | ||
| "\n", | ||
| "log_filepath = Path(\".\") / \"outputs\"\n", | ||
| "log_filepath.mkdir(parents=True, exist_ok=True)\n", | ||
| "\n", | ||
| "# Set up a file handler, which will store log outputs in a file\n", | ||
| "file_handler = logging.FileHandler(log_filepath / \"calliope.log\")\n", | ||
| "file_handler.setLevel(logging.DEBUG)\n", | ||
| "file_handler.setFormatter(formatter)\n", | ||
| "logger.addHandler(file_handler)\n", | ||
| "\n", | ||
| "# You can also use logging in your scripts to add more information\n", | ||
| "logger.info(\"Loading the national-scale example model\")\n", | ||
| "m = calliope.examples.national_scale()\n", | ||
| "\n", | ||
| "logger.info(\"Building the national-scale example model optimisation problem\")\n", | ||
| "m.build()\n", | ||
| "\n", | ||
| "logger.info(\"Solving the national-scale example model optimisation problem\")\n", | ||
| "m.solve()" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "markdown", | ||
| "metadata": {}, | ||
| "source": [ | ||
| "Notice that there is no logging to the console here, but that there is now a file `outputs/calliope.log` that contains the logging information.\n", | ||
| "\n", | ||
| "We can also log both to the console at one level and to file at another:" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "code", | ||
| "execution_count": null, | ||
| "metadata": {}, | ||
| "outputs": [], | ||
| "source": [ | ||
| "# Grab the calliope logger, which will also automatically all the child loggers (e.g., `calliope.core.model`).\n", | ||
| "logger = logging.getLogger(\"calliope\")\n", | ||
| "\n", | ||
| "# Remove existing handlers (i.e., those introduced earlier in this notebook)\n", | ||
| "logger.handlers.clear()\n", | ||
| "\n", | ||
| "# You can define your own custom formatter.\n", | ||
| "# See https://docs.python.org/3/library/logging.html#logrecord-attributes for available attributes.\n", | ||
| "formatter = logging.Formatter(\"%(asctime)s - %(name)s - %(levelname)s - %(message)s\")\n", | ||
| "\n", | ||
| "# Add a ConsoleHandler (this is what `calliope.set_log_verbosity` is doing under the hood)\n", | ||
| "console_handler = logging.StreamHandler()\n", | ||
| "# Log to console at the INFO level\n", | ||
| "console_handler.setLevel(logging.INFO)\n", | ||
| "console_handler.setFormatter(formatter)\n", | ||
| "logger.addHandler(console_handler)\n", | ||
| "\n", | ||
| "log_filepath = Path(\".\") / \"outputs\"\n", | ||
| "log_filepath.mkdir(parents=True, exist_ok=True)\n", | ||
| "\n", | ||
| "# Set up a file handler, which will store log outputs in a file\n", | ||
| "file_handler = logging.FileHandler(log_filepath / \"calliope.log\")\n", | ||
| "# Log to file at the DEBUG level\n", | ||
| "file_handler.setLevel(logging.DEBUG)\n", | ||
| "file_handler.setFormatter(formatter)\n", | ||
| "logger.addHandler(file_handler)\n", | ||
| "\n", | ||
| "# You can also use logging in your scripts to add more information\n", | ||
| "logger.info(\"Loading the national-scale example model\")\n", | ||
| "m = calliope.examples.national_scale()\n", | ||
| "\n", | ||
| "logger.info(\"Building the national-scale example model optimisation problem\")\n", | ||
| "m.build()\n", | ||
| "\n", | ||
| "logger.info(\"Solving the national-scale example model optimisation problem\")\n", | ||
| "m.solve()" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "markdown", | ||
| "metadata": {}, | ||
| "source": [ | ||
| "The log file will contain all calliope child logger outputs in one place.\n", | ||
| "You will notice the name of the logger, which corresponds to the file where the log was recorded, at the second level of the log messages.\n", | ||
| "\n", | ||
| "We can store each of these child loggers to a different file if we like:" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "code", | ||
| "execution_count": null, | ||
| "metadata": {}, | ||
| "outputs": [], | ||
| "source": [ | ||
| "# Grab the calliope logger, which will also automatically all the child loggers (e.g., `calliope.core.model`).\n", | ||
| "logger = logging.getLogger(\"calliope\")\n", | ||
| "\n", | ||
| "# Remove existing handlers (i.e., those introduced earlier in this notebook)\n", | ||
| "logger.handlers.clear()\n", | ||
| "\n", | ||
| "# You can define your own custom formatter.\n", | ||
| "# Here we don't include the logger name, as the filename will contain that information.\n", | ||
| "# See https://docs.python.org/3/library/logging.html#logrecord-attributes for available attributes.\n", | ||
| "formatter = logging.Formatter(\"%(asctime)s - %(levelname)s - %(message)s\")\n", | ||
| "\n", | ||
| "log_filepath = Path(\".\") / \"outputs\"\n", | ||
| "log_filepath.mkdir(parents=True, exist_ok=True)\n", | ||
| "\n", | ||
| "for logger_name in logging.root.manager.loggerDict.keys():\n", | ||
| " if not logger_name.startswith(\"calliope\"):\n", | ||
| " # There are lots of other loggers that calliope imports from its dependencies which we will ignore.\n", | ||
| " # You can also dump these log to files if you like, by removing this conditional statement.\n", | ||
| " continue\n", | ||
| " # Set up a file handler, which will store log outputs in a file\n", | ||
| " file_handler = logging.FileHandler(log_filepath / f\"{logger_name}.log\")\n", | ||
| " # Log to file at the DEBUG level\n", | ||
| " file_handler.setLevel(logging.DEBUG)\n", | ||
| " file_handler.setFormatter(formatter)\n", | ||
| " logging.getLogger(logger_name).addHandler(file_handler)\n", | ||
| "\n", | ||
| "m = calliope.examples.national_scale()\n", | ||
| "m.build()\n", | ||
| "m.solve()" | ||
| ] | ||
| } | ||
| ], | ||
| "metadata": { | ||
| "kernelspec": { | ||
| "display_name": "Python 3", | ||
| "language": "python", | ||
| "name": "python3" | ||
| }, | ||
| "language_info": { | ||
| "codemirror_mode": { | ||
| "name": "ipython", | ||
| "version": 3 | ||
| }, | ||
| "file_extension": ".py", | ||
| "mimetype": "text/x-python", | ||
| "name": "python", | ||
| "nbconvert_exporter": "python", | ||
| "pygments_lexer": "ipython3", | ||
| "version": "3.11.5" | ||
| }, | ||
| "orig_nbformat": 4 | ||
| }, | ||
| "nbformat": 4, | ||
| "nbformat_minor": 2 | ||
| } |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.