Initial commit

.gitignore

@@ -0,0 +1,6 @@
+__pycache__
+/.coverage
+/dist/
+/htmlcov/
+/*.egg-info/
+ionit.1

LICENSE

@@ -0,0 +1,15 @@
+ionit is licensed under ISC:
+
+Copyright (C) 2018, Benjamin Drung <[email protected]>
+
+Permission to use, copy, modify, and/or distribute this software for any
+purpose with or without fee is hereby granted, provided that the above
+copyright notice and this permission notice appear in all copies.
+
+THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES
+WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF
+MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR
+ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES
+WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN
+ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF
+OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.

MANIFEST.in

@@ -0,0 +1,4 @@
+recursive-include tests *.jinja *.json *.py *.yaml echo pylint.conf
+include ionit.1.md
+include ionit.py
+include LICENSE

README.md

@@ -0,0 +1,131 @@
+ionit
+=====
+
+ionit is a simple and small configuration templating tool. It collects a context
+and renders Jinja templates in a given directory. The context can be either
+static JSON or YAML files or dynamic Python files. Python files can also define
+functions passed through to the rendering.
+
+The context filenames needs to end with `.json` for JSON, `.py` for Python,
+and `.yaml` for YAML. The context files are read in alphabetical order. If the
+same key is defined by multiple context files, the file that is read later takes
+precedence. It is recommended to prefix the files with a number in case the
+order is relevant.
+
+ionit comes with an early boot one shot service that is executed before the
+networking service which allows one to generate configurations files for the
+networking and other services before they are started. In this regard, ionit can
+act as tiny stepbrother of cloud-init.
+
+Python modules
+==============
+
+Python modules can define a `collect_context` function. This function is called
+by ionit and the current context is passed as parameter. The current context can
+be used to derive more context information, but this variable should not be
+modified. `collect_context` must return a dictionary (can be empty) or raise an
+exception, which will be caught by ionit.
+
+Python modules can also define functions which can be called from the Jinja
+template on rendering. Use the `ionit_plugin.function` decorator to mark the
+functions to export.
+
+Note that the functions names should not collide with other keys from the
+context. If one Python module defines a function and a value in the context
+with the same name, the value in the context will take precedence.
+
+An example Python module might look like:
+
+```python
+import ionit_plugin
+
+
+@ionit_plugin.function
+def double(value):
+    return 2 * value
+
+
+@ionit_plugin.function
+def example_function():
+    return "Lorem ipsum"
+
+
+def collect_context(current_context):
+    return {"key": "value"}
+```
+
+Prerequisites
+=============
+
+* Python >= 3.4
+* Python modules:
+  * jinja2
+  * yaml or ruamel.yaml
+* pandoc (to generate `ionit.1` man page from `ionit.1.md`)
+
+The test cases have additional requirements:
+
+* flake8
+* pylint
+
+Examples
+========
+
+Static context
+--------------
+
+This example is taken from one test case and demonstrates how ionit will collect
+the context from one JSON and one YAML file and renders one template:
+
+```
+user@host:~/ionit$ cat tests/config/static/first.json
+{"first": 1}
+user@host:~/ionit$ cat tests/config/static/second.yaml
+second: 2
+user@host:~/ionit$ cat tests/template/static/counting.jinja
+Counting:
+* {{ first }}
+* {{ second }}
+* 3
+user@host:~/ionit$ ./ionit -c tests/config/static -t tests/template/static
+2018-08-08 17:39:06,956 ionit INFO: Reading configuration file 'tests/config/static/first.json'...
+2018-08-08 17:39:06,956 ionit INFO: Reading configuration file 'tests/config/static/second.yaml'...
+2018-08-08 17:39:06,960 ionit INFO: Rendered 'tests/template/static/counting.jinja' to 'tests/template/static/counting'.
+user@host:~/ionit$ cat tests/template/static/counting
+Counting:
+* 1
+* 2
+* 3
+```
+
+Python functions
+----------------
+
+This example is taken from one test case and demonstrates how Python functions
+can be defined to be used when rendering:
+
+```
+user@host:~/ionit$ cat tests/config/function/function.py
+import ionit_plugin
+
+
+@ionit_plugin.function
+def answer_to_all_questions():
+    return 42
+user@host:~/ionit$ cat tests/template/function/Document.jinja
+The answer to the Ultimate Question of Life, The Universe, and Everything is {{ answer_to_all_questions() }}.
+user@host:~/ionit$ ./ionit -c tests/config/function -t tests/template/function
+2018-08-13 11:58:16,905 ionit INFO: Loading Python module 'function' from 'tests/config/function/function.py'...
+2018-08-13 11:58:16,909 ionit INFO: Rendered 'tests/template/function/Document.jinja' to 'tests/template/function/Document'.
+user@host:~/ionit$ cat tests/template/function/Document
+The answer to the Ultimate Question of Life, The Universe, and Everything is 42.
+```
+
+Contributing
+============
+
+Contributions are welcome. The source code has 100% test coverage, which should
+be preserved. So please provide a test case for each bugfix and one or more
+test cases for each new feature. Please follow
+[How to Write a Git Commit Message](https://chris.beams.io/posts/git-commit/)
+for writing good commit messages.

ionit

@@ -0,0 +1,208 @@
+#!/usr/bin/python3
+
+# Copyright (C) 2018, Benjamin Drung <[email protected]>
+#
+# Permission to use, copy, modify, and/or distribute this software for any
+# purpose with or without fee is hereby granted, provided that the above
+# copyright notice and this permission notice appear in all copies.
+#
+# THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES
+# WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF
+# MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR
+# ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES
+# WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN
+# ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF
+# OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
+
+"""Render configuration files from Jinja templates"""
+
+import argparse
+import importlib.util
+import json
+import logging
+import os
+import sys
+
+import jinja2
+import yaml
+
+import ionit_plugin
+
+LOG_FORMAT = '%(asctime)s %(name)s %(levelname)s: %(message)s'
+SCRIPT_NAME = "ionit"
+
+
+class PythonModuleException(Exception):
+    """Exception raised when loading a Python context file fails"""
+    pass
+
+
+def load_python_plugin(file_path, current_context):
+    """Collect context from given Python module
+
+    The specified Python file needs to be a valid plug-in which provides
+    a collect_context function that takes the current context as parameter
+    and returns a dict containing the context.
+    """
+    logger = logging.getLogger(SCRIPT_NAME)
+    module_name = os.path.splitext(os.path.basename(file_path))[0]
+    logger.info("Loading Python module '%s' from '%s'...", module_name, file_path)
+    function_collector = ionit_plugin.FunctionCollector()
+    function_collector.clear()
+    dont_write_bytecode = sys.dont_write_bytecode
+    sys.dont_write_bytecode = True
+    try:
+        if sys.version_info[:2] < (3, 5):
+            # Backwards compatibility for Debian jessie
+            import imp  # pragma: no cover
+            module = imp.load_source(module_name, file_path)  # pragma: no cover
+        else:
+            spec = importlib.util.spec_from_file_location(module_name, file_path)
+            # module_from_spec not available in Python < 3.5.
+            module = importlib.util.module_from_spec(spec)  # pylint: disable=no-member
+            spec.loader.exec_module(module)
+    except Exception:
+        logger.exception("Importing Python module '%s' failed:", file_path)
+        raise PythonModuleException()
+    finally:
+        sys.dont_write_bytecode = dont_write_bytecode
+
+    context = function_collector.functions.copy()
+    if hasattr(module, "collect_context"):
+        try:
+            new_context = module.collect_context(current_context)
+        except Exception:
+            logger.exception("Calling collect_context() from '%s' failed:", file_path)
+            raise PythonModuleException()
+        context.update(new_context)
+
+    if not context:
+        logger.warning("Python module '%s' does neither define a collect_context function, "
+                       "nor export functions (using the ionit_plugin.function decorator).",
+                       file_path)
+
+    return context
+
+
+def collect_context(directory):
+    """Collect context that will be used when rendering the templates"""
+    logger = logging.getLogger(SCRIPT_NAME)
+    logger.debug("Collecting context from '%s'...", directory)
+    try:
+        files = sorted(os.listdir(directory))
+    except OSError as error:
+        logger.warning("Failed to read configuration directory: %s", error)
+        files = []
+
+    failures = 0
+    context = {}
+
+    for filename in files:
+        file_context = None
+        file = os.path.join(directory, filename)
+        extension = os.path.splitext(filename)[1]
+        try:
+            if extension == ".json":
+                logger.info("Reading configuration file '%s'...", file)
+                with open(file) as config_file:
+                    file_context = json.load(config_file)
+            elif extension == ".py":
+                file_context = load_python_plugin(file, context)
+            elif extension == ".yaml":
+                logger.info("Reading configuration file '%s'...", file)
+                with open(file) as config_file:
+                    file_context = yaml.load(config_file)
+            else:
+                logger.info("Skipping configuration file '%s', because it does not end with "
+                            "'.json', '.py', or '.yaml'.", file)
+                continue
+        except PythonModuleException:
+            failures += 1
+            continue
+        except (OSError, ValueError, yaml.error.YAMLError) as error:
+            types = {".json": "JSON", ".py": "Python code", ".yaml": "YAML"}
+            logger.error("Failed to read %s from '%s': %s", types[extension], file, error)
+            failures += 1
+            continue
+
+        logger.debug("Parsed context from '%s': %s", file, file_context)
+        if file_context:
+            try:
+                context.update(file_context)
+            except (TypeError, ValueError) as error:
+                logger.debug("Current context: %s", context)
+                logger.error("Failed to update context with content from '%s': %s",
+                             file, error)
+                failures += 1
+                continue
+
+    return failures, context
+
+
+def render_templates(template_dir, context, template_extension):
+    """
+    Search in the template directory for template files and render them with the context
+    """
+    logger = logging.getLogger(SCRIPT_NAME)
+    logger.debug("Searching in directory '%s' for Jinja templates...", template_dir)
+    failures = 0
+    env = jinja2.Environment(keep_trailing_newline=True,
+                             loader=jinja2.FileSystemLoader(template_dir),
+                             undefined=jinja2.StrictUndefined)
+    for name in env.list_templates(extensions=[template_extension]):
+        try:
+            template = env.get_template(name)
+        except jinja2.TemplateError:
+            logger.exception("Failed to load template '%s':", os.path.join(template_dir, name))
+            failures += 1
+            continue
+
+        rendered_filename = os.path.splitext(template.filename)[0]
+        logger.debug("Rendering template '%s' to '%s'...", template.filename, rendered_filename)
+        try:
+            rendered = template.render(context)
+        except Exception:  # pylint: disable=broad-except
+            logger.exception("Failed to render '%s':", template.filename)
+            failures += 1
+            continue
+
+        try:
+            with open(rendered_filename, "w") as output_file:
+                output_file.write(rendered)
+        except OSError as error:
+            logger.error("Failed to write rendered template to '%s': %s", rendered_filename, error)
+            failures += 1
+            continue
+
+        logger.info("Rendered '%s' to '%s'.", template.filename, rendered_filename)
+
+    return failures
+
+
+def main(argv):
+    """Main function with argument parsing"""
+    parser = argparse.ArgumentParser()
+    parser.add_argument("-c", "--config", default="/etc/ionit",
+                        help="Configuration directory containing context for rendering (default: "
+                             "%(default)s)")
+    parser.add_argument("-t", "--templates", default="/etc",
+                        help="Directory to search for Jinja templates (default: %(default)s)")
+    parser.add_argument("-e", "--template-extension", default="jinja",
+                        help="Extension to look for in template directory (default: %(default)s)")
+    parser.add_argument("--debug", dest="log_level", help="Print debug output",
+                        action="store_const", const=logging.DEBUG, default=logging.INFO)
+    parser.add_argument("-q", "--quiet", dest="log_level",
+                        help="Decrease output verbosity to warnings and errors",
+                        action="store_const", const=logging.WARNING)
+    args = parser.parse_args(argv)
+    logging.basicConfig(level=args.log_level, format=LOG_FORMAT)
+    logger = logging.getLogger(SCRIPT_NAME)
+
+    failures, context = collect_context(args.config)
+    logger.debug("Context: %s", context)
+    failures += render_templates(args.templates, context, args.template_extension)
+    return failures
+
+
+if __name__ == "__main__":
+    sys.exit(main(sys.argv[1:]))  # pragma: no cover