Small doc & unit test updates (#14)

* doc: fix minor doc warnings, add FileTypes to docs

* test: add unit test + docs for filetypes

docs/Rules_and_Rulesets/rulesets.md

@@ -4,7 +4,9 @@
         separate_signature: true
         show_signature_annotations: true
         signature_crossrefs: true
-::: orbiter.rules.rulesets.xmltodict_parse
+
+::: orbiter.file_types
+
 ## Rulesets
 ::: orbiter.rules.rulesets.Ruleset
     options:

docs/index.md

@@ -20,20 +20,20 @@ flowchart LR
     origin{{ XML/JSON/YAML/Etc Workflows }}
     origin -->| ✨ Translations ✨ | airflow{{ Apache Airflow Project }}
 ```
-The **framework** is a set of [Rules](./Rules_and_Rulesets) and [Objects](./objects) that can translate workflows
-from an [Origin](./origins) system to an Airflow project.
+The **framework** is a set of [Rules](./Rules_and_Rulesets/index.md) and [Objects](./objects/index.md) that can translate workflows
+from an [Origin](./origins.md) system to an Airflow project.
 
 ## Installation
 
-Install the [`orbiter` CLI](./CLI), if you have Python >= 3.10 installed via `pip`:
+Install the [`orbiter` CLI](./cli.md), if you have Python >= 3.10 installed via `pip`:
 ```shell
 pip install astronomer-orbiter
 ```
 If you do not have a compatible Python environment, pre-built binary executables of the `orbiter` CLI
 are available for download on the [Releases](https://github.com/astronomer/orbiter/releases) page.
 
 ## Translate
-Utilize the [`orbiter` CLI](./cli) with existing translations to convert workflows
+Utilize the [`orbiter` CLI](./cli.md) with existing translations to convert workflows
 from other systems to an Airflow project.
 
 1. Set up a new folder, and create a `workflow/` folder. Add your workflows files to it
@@ -45,14 +45,14 @@ from other systems to an Airflow project.
         └── ...
     ```
 2. Determine the specific translation ruleset via:
-    1. the [Origins](origins) documentation
-    2. the [`orbiter list-rulesets`](./cli#list-rulesets) command
+    1. the [Origins](./origins.md) documentation
+    2. the [`orbiter list-rulesets`](./cli.md#list-rulesets) command
     3. or [by creating a translation ruleset](#authoring-rulesets-customization), if one does not exist
-3. Install the translation ruleset via the [`orbiter install`](./cli#install) command (substituting `<REPOSITORY>` with the value in the last step)
+3. Install the translation ruleset via the [`orbiter install`](./cli.md#install) command (substituting `<REPOSITORY>` with the value in the last step)
     ```shell
     orbiter install --repo=<REPOSITORY>
     ```
-4. Use the [`orbiter translate`](./cli#translate) command with the `<RULESET>` determined in the last step
+4. Use the [`orbiter translate`](./cli.md#translate) command with the `<RULESET>` determined in the last step
    This will produce output to an `output/` folder:
     ```shell
     orbiter translate workflow/ --ruleset <RULESET> output/
@@ -66,7 +66,7 @@ from other systems to an Airflow project.
 ## Authoring Rulesets & Customization
 Orbiter can be extended to fit specific needs, patterns, or to support additional origins.
 
-Read more specifics about how to use the framework at [Rules](./Rules_and_Rulesets) and [Objects](./objects)
+Read more specifics about how to use the framework at [Rules](./Rules_and_Rulesets/index.md) and [Objects](./objects/index.md)
 
 ### Extend or Customize
 To extend or customize an existing ruleset, you can easily modify it with simple Python code.
@@ -82,18 +82,18 @@ To extend or customize an existing ruleset, you can easily modify it with simple
         └── ...
     ```
 3. Add contents to `override.py`:
-    ```python title="override.py" linenums="1"
+   ```python title="override.py" linenums="1"
     -8<- "tests/resources/override/override.py"
     ```
-    1. Importing specific translation ruleset, determined via the [Origins](origins) page
-    2. Importing required [Objects](./objects)
-    3. Importing required [Rule](./Rules_and_Rulesets) types
+    1. Importing specific translation ruleset, determined via the [Origins](origins.md) page
+    2. Importing required [Objects](./objects/index.md)
+    3. Importing required [Rule](./Rules_and_Rulesets/index.md) types
     4. Create one or more `@rule` functions, as required. A higher priority means this rule will be applied first.
-        [`@task_rule` Reference](./Rules_and_Rulesets/rules/#orbiter.rules.TaskRule)
+        [`@task_rule` Reference](./Rules_and_Rulesets/rules.md#orbiter.rules.TaskRule)
     5. `Rules` have an `if/else` statement - they must always return a **single** thing or **nothing**
-    6. [`OrbiterSSHOperator` Reference](./objects/Operators_and_Callbacks/operators/#orbiter.objects.operators.ssh.OrbiterSSHOperator)
-    7. Append the new [Rule](./Rules_and_Rulesets)
-       to the [`translation_ruleset`](./Rules_and_Rulesets/rulesets/#orbiter.rules.rulesets.TranslationRuleset)
+    6. [`OrbiterSSHOperator` Reference](./objects/Tasks/Operators_and_Callbacks/operators.md#orbiter.objects.operators.ssh.OrbiterSSHOperator)
+    7. Append the new [Rule](./Rules_and_Rulesets/index.md)
+       to the [`translation_ruleset`](./Rules_and_Rulesets/rulesets.md#orbiter.rules.rulesets.TranslationRuleset)
 
 4. Invoke the `orbiter` CLI, pointing it at your customized ruleset, and writing output to an `output/` folder:
     ```shell
@@ -103,16 +103,16 @@ To extend or customize an existing ruleset, you can easily modify it with simple
 
 ### Authoring a new Ruleset
 
-You can utilize the [`TranslationRuleset` Template](./Rules_and_Rulesets/template)
+You can utilize the [`TranslationRuleset` Template](./Rules_and_Rulesets/template.md)
 to create a new [`TranslationRuleset`][orbiter.rules.rulesets.TranslationRuleset].
 
 ## FAQ
 - **Can this tool convert my workflows from tool X to Airflow?**
 
-    _If you don't see your tool listed in [Supported Origins](./origins),
+    _If you don't see your tool listed in [Supported Origins](./origins.md),
     [contact us](https://www.astronomer.io/contact/) for services to create translations,
     create an [issue](https://github.com/astronomer/orbiter-community-translations/issues/new/)
-    in the [`orbiter-community-translations`](https://github.com/astronomer/orbiter-community-translations) repository, or write a `TranslationRuleset` and submit a
+    in the [`orbiter-community-translations`](https://github.com/astronomer/orbiter-community-translations) repository, or write a [`TranslationRuleset`](./Rules_and_Rulesets/template.md) and submit a
     [pull request](https://github.com/astronomer/orbiter-community-translations/pulls/)
     to share your translations with the community._
 

docs/objects/Tasks/index.md

@@ -4,7 +4,7 @@ are units of work. An Operator is a pre-defined task with specific functionality
 
 Operators can be looked up in the [Astronomer Registry](https://registry.astronomer.io/).
 
-The easiest way to create an operator in a translation to [use an existing subclass of `OrbiterOperator` (e.g. `OrbiterBashOperator`)](./Operators_and_Callbacks/operators).
+The easiest way to create an operator in a translation to [use an existing subclass of `OrbiterOperator` (e.g. `OrbiterBashOperator`)](./Operators_and_Callbacks/operators.md).
 
 If an `OrbiterOperator` subclass doesn't exist for your use case, you can:
 

docs/objects/index.md

@@ -1,6 +1,6 @@
 # Overview
 
-**Objects** are returned from [Rules](../Rules_and_Rulesets) during a translation, and
+**Objects** are returned from [Rules](../Rules_and_Rulesets/index.md) during a translation, and
 are **rendered** to produce an Apache Airflow Project
 
 An [`OrbiterProject`][orbiter.objects.project.OrbiterProject] holds everything necessary to render an Airflow Project.
@@ -10,7 +10,7 @@ It is generated by a [`TranslationRuleset.translate_fn`][orbiter.rules.rulesets.
 
 Workflows are represented by a
 [`OrbiterDAG`][orbiter.objects.dag.OrbiterDAG]
-which is a Directed Acyclic Graph (of [Tasks](./tasks)).
+which is a Directed Acyclic Graph (of [Tasks](./Tasks/index.md)).
 
 [`OrbiterOperators`][orbiter.objects.task.OrbiterOperator] represent Airflow Tasks, which
 are units of work. An Operator is a pre-defined task with specific functionality.

docs/objects/workflow.md

@@ -2,7 +2,7 @@
 
 Airflow workflows are represented by a
 [DAG](https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/dags.html)
-which is a Directed Acyclic Graph (of [Tasks](../tasks)).
+which is a Directed Acyclic Graph (of [Tasks](./Tasks/index.md)).
 
 ## Diagram
 ```mermaid

docs/origins.md

@@ -20,5 +20,5 @@ please [contact us](https://www.astronomer.io/contact/) for access to the most u
 If you don't see your Origin system listed, please either:
 
 - [contact us](https://www.astronomer.io/contact/) for services to create translations
-- create an [issue](https://github.com/astronomer/orbiter-community-translations/issues/new/) in our `orbiter-community-translations` repository
-- write a `TranslationRuleset` and submit a [pull request](https://github.com/astronomer/orbiter-community-translations/pulls/) to share your translations with the community
+- create an [issue](https://github.com/astronomer/orbiter-community-translations/issues/new/) in our [`orbiter-community-translations`](https://github.com/astronomer/orbiter-community-translations) repository
+- write a [`TranslationRuleset` Template](./Rules_and_Rulesets/template.md) and submit a [pull request](https://github.com/astronomer/orbiter-community-translations/pulls/) to share your translations with the community

orbiter/file_types.py

@@ -1,6 +1,7 @@
 from __future__ import annotations
 
 import json
+from abc import ABC
 from functools import partial
 from typing import Callable, Set, ClassVar, Any
 
@@ -12,7 +13,17 @@
 from pydantic.v1 import validator
 
 
-class FileType(BaseModel, arbitrary_types_allowed=True):
+class FileType(BaseModel, ABC, arbitrary_types_allowed=True):
+    """**Abstract Base** File Type
+
+    :param extension: The file extension(s) for this file type
+    :type extension: Set[str]
+    :param load_fn: The function to load the file into a dictionary for this file type
+    :type load_fn: Callable[[str], dict]
+    :param dump_fn: The function to dump a dictionary to a string for this file type
+    :type dump_fn: Callable[[dict], str]
+    """
+
     extension: ClassVar[Set[str]]
     load_fn: ClassVar[Callable[[str], dict]]
     dump_fn: ClassVar[Callable[[dict], str]]
@@ -34,11 +45,26 @@ def ext_validate(cls, v: Set[str]):
 
 
 class FileTypeJSON(FileType):
+    """JSON File Type
+
+    ```pycon
+    >>> out = FileTypeJSON.dump_fn({'a': 1}); out
+    '{"a": 1}'
+    >>> FileTypeJSON.load_fn(out)
+    {'a': 1}
+
+    ```
+    :param extension: JSON
+    :type extension: Set[str]
+    :param load_fn: json.loads
+    :type load_fn: Callable[[str], dict]
+    :param dump_fn: json.dumps
+    :type dump_fn: Callable[[dict], str]
+    """
+
     extension: ClassVar[Set[str]] = {"JSON"}
     load_fn: ClassVar[Callable[[str], dict]] = json.loads
-    dump_fn: ClassVar[Callable[[dict], str]] = partial(
-        json.dumps, default=str, indent=2
-    )
+    dump_fn: ClassVar[Callable[[dict], str]] = partial(json.dumps, default=str)
 
 
 # noinspection t
@@ -106,12 +132,60 @@ def _fix(d):
 
 
 class FileTypeXML(FileType):
+    """XML File Type
+
+    !!! note
+        This class uses a custom `xmltodict_parse` method to standardize the output to a list of dictionaries
+
+    ```pycon
+    >>> out = FileTypeXML.dump_fn({'a': 1}); out
+    '<?xml version="1.0" encoding="utf-8"?>\\n<a>1</a>'
+    >>> FileTypeXML.load_fn(out)
+    {'a': '1'}
+
+    ```
+    :param extension: XML
+    :type extension: Set[str]
+    :param load_fn: xmltodict_parse
+    :type load_fn: Callable[[str], dict]
+    :param dump_fn: xmltodict.unparse
+    :type dump_fn: Callable[[dict], str]
+    """
+
     extension: ClassVar[Set[str]] = {"XML"}
     load_fn: ClassVar[Callable[[str], dict]] = xmltodict_parse
     dump_fn: ClassVar[Callable[[dict], str]] = xmltodict.unparse
 
 
 class FileTypeYAML(FileType):
+    """YAML File Type
+
+
+    ```pycon
+    >>> out = FileTypeYAML.dump_fn({'a': 1}); out
+    'a: 1\\n'
+    >>> FileTypeYAML.load_fn(out)
+    {'a': 1}
+
+    ```
+    :param extension: YAML, YML
+    :type extension: Set[str]
+    :param load_fn: yaml.safe_load
+    :type load_fn: Callable[[str], dict]
+    :param dump_fn: yaml.safe_dump
+    :type dump_fn: Callable[[dict], str]
+    """
+
     extension: ClassVar[Set[str]] = {"YAML", "YML"}
     load_fn: ClassVar[Callable[[str], dict]] = yaml.safe_load
     dump_fn: ClassVar[Callable[[dict], str]] = yaml.safe_dump
+
+
+if __name__ == "__main__":
+    import doctest
+
+    doctest.testmod(
+        optionflags=doctest.ELLIPSIS
+        | doctest.NORMALIZE_WHITESPACE
+        | doctest.IGNORE_EXCEPTION_DETAIL
+    )

orbiter/rules/__init__.py

@@ -3,12 +3,12 @@
 and the [`Rulesets`][orbiter.rules.rulesets.Ruleset] that contain them.
 
 - A [`Rule`][orbiter.rules.Rule] is a python function that is evaluated and produces **something**
-(typically an [Object](../objects)) or **nothing**
+(typically an [Object](../objects/index.md)) or **nothing**
 - A [`Ruleset`][orbiter.rules.rulesets.Ruleset] is a collection of [`Rules`][orbiter.rules.Rule] that are
     evaluated in priority order
 - A [`TranslationRuleset`][orbiter.rules.rulesets.TranslationRuleset]
     is a collection of [`Rulesets`][orbiter.rules.rulesets.Ruleset],
-    relating to an [Origin](../origins) and `FileType`,
+    relating to an [Origin](../origins.md) and [`FileType`][orbiter.file_types.FileType],
     with a [`translate_fn`][orbiter.rules.rulesets.translate] which determines how to apply the rulesets.
 
 Different [`Rules`][orbiter.rules.Rule] are applied in different scenarios, such as:
@@ -92,7 +92,7 @@ def rule(
 class Rule(BaseModel, Callable, extra="forbid"):
     """
     A `Rule` contains a python function that is evaluated and produces something
-    (typically an [Object](../objects)) or nothing
+    (typically an [Object](../objects/index.md)) or nothing
 
     A `Rule` can be created from a decorator
     ```pycon
@@ -116,7 +116,7 @@ class Rule(BaseModel, Callable, extra="forbid"):
 
     !!! tip
 
-        If the returned value is an [Orbiter Object](../../objects),
+        If the returned value is an [Orbiter Object](../../objects/index.md),
         the passed `kwargs` are saved in a special `orbiter_kwargs` property
 
         ```pycon