Merge pull request #6 from astronomer/1.0.2

v1.0.2

.github/workflows/checks.yml

@@ -3,26 +3,12 @@ on: [pull_request]
 permissions:
   contents: write
 jobs:
-  check:
-    runs-on: ubuntu-latest
-    steps:
-      - uses: actions/checkout@v4
-      - uses: actions/setup-python@v5
-        with:
-          python-version: '3.10'
-          cache: 'pip'
-      - uses: extractions/setup-just@v1
-      - run: just install
-      - run: |
-          git fetch origin
-          pre-commit run --from-ref origin/${{ github.event.pull_request.base.ref }} --to-ref ${{ github.event.pull_request.head.sha }}
-
   test:
     runs-on: ubuntu-latest
     strategy:
       fail-fast: false
       matrix:
-        python-version: [ "3.10", "3.11" ]
+        python-version: [ "3.10", "3.11", "3.12" ]
     steps:
       - uses: actions/checkout@v4
       - uses: actions/setup-python@v5

.gitignore

@@ -20,3 +20,4 @@ test_binary
 workflow
 *.pyz
 orbiter-*
+.python-version

docs/Rules_and_Rulesets/index.md

@@ -7,7 +7,3 @@
       show_root_toc_entry: false
 
 ::: orbiter.rules.rulesets.translate
-
-::: orbiter.rules.rulesets.load_filetype
-
-::: orbiter.rules.rulesets.xmltodict_parse

docs/Rules_and_Rulesets/rulesets.md

@@ -1,24 +1,35 @@
+# Translation
 ::: orbiter.rules.rulesets.TranslationRuleset
     options:
         separate_signature: true
         show_signature_annotations: true
         signature_crossrefs: true
+::: orbiter.rules.rulesets.xmltodict_parse
+## Rulesets
 ::: orbiter.rules.rulesets.Ruleset
+    options:
+        heading_level: 3
 ::: orbiter.rules.rulesets.DAGFilterRuleset
     options:
+        heading_level: 3
         show_bases: true
 ::: orbiter.rules.rulesets.DAGRuleset
     options:
+        heading_level: 3
         show_bases: true
 ::: orbiter.rules.rulesets.TaskFilterRuleset
     options:
+        heading_level: 3
         show_bases: true
 ::: orbiter.rules.rulesets.TaskRuleset
     options:
+        heading_level: 3
         show_bases: true
 ::: orbiter.rules.rulesets.TaskDependencyRuleset
     options:
+        heading_level: 3
         show_bases: true
 ::: orbiter.rules.rulesets.PostProcessingRuleset
     options:
+        heading_level: 3
         show_bases: true

docs/index.md

@@ -25,16 +25,16 @@ from an [Origin](./origins) system to an Airflow project.
 
 ## Installation
 
-You can install the [`orbiter` CLI](./CLI), if you have Python >= 3.10 installed via `pip`:
+Install the [`orbiter` CLI](./CLI), if you have Python >= 3.10 installed via `pip`:
 ```shell
 pip install astronomer-orbiter
 ```
 If you do not have a compatible Python environment, pre-built binary executables of the `orbiter` CLI
 are available for download on the [Releases](https://github.com/astronomer/orbiter/releases) page.
 
 ## Translate
-You can utilize the [`orbiter` CLI](./cli) with existing translations to convert workflows
-from other systems to Apache Airflow.
+Utilize the [`orbiter` CLI](./cli) with existing translations to convert workflows
+from other systems to an Airflow project.
 
 1. Set up a new folder, and create a `workflow/` folder. Add your workflows files to it
     ```shell
@@ -46,21 +46,22 @@ from other systems to Apache Airflow.
     ```
 2. Determine the specific translation ruleset via:
     1. the [Origins](origins) documentation
-    2. the [`orbiter help`](./cli#help) command
+    2. the [`orbiter list-rulesets`](./cli#list-rulesets) command
     3. or [by creating a translation ruleset](#authoring-rulesets-customization), if one does not exist
-3. Install the specific translation ruleset via the [`orbiter install`](./cli#install) command
+3. Install the translation ruleset via the [`orbiter install`](./cli#install) command (substituting `<REPOSITORY>` with the value in the last step)
+    ```shell
+    orbiter install --repo=<REPOSITORY>
+    ```
 4. Use the [`orbiter translate`](./cli#translate) command with the `<RULESET>` determined in the last step
    This will produce output to an `output/` folder:
     ```shell
-    orbiter translate workflow/ output/ --ruleset <RULESET>
+    orbiter translate workflow/ --ruleset <RULESET> output/
     ```
 5. Review the contents of the `output/` folder. If extensions or customizations are required, review
     [how to extend a translation ruleset](#extend-or-customize)
-6. Utilize the [`astro` CLI](https://www.astronomer.io/docs/astro/cli/overview)
+6. (optional) Utilize the [`astro` CLI](https://www.astronomer.io/docs/astro/cli/overview)
     to run Airflow instance with your migrated workloads
-7. Deploy to [Astro](https://www.astronomer.io/try-astro/) to run your translated workflows in production! 🚀
-
-You can see more specifics on how to use the Orbiter CLI in the [CLI](./cli) section.
+7. (optional) Deploy to [Astro](https://www.astronomer.io/try-astro/) to run your translated workflows in production! 🚀
 
 ## Authoring Rulesets & Customization
 Orbiter can be extended to fit specific needs, patterns, or to support additional origins.
@@ -98,12 +99,12 @@ To extend or customize an existing ruleset, you can easily modify it with simple
     ```shell
     orbiter translate workflow/ output/ --ruleset override.translation_ruleset
     ```
-5. Follow the remaining steps 4 -> 6 of the [Translate](#translate) instructions
+5. Follow the remaining steps of the [Translate](#translate) instructions
 
 ### Authoring a new Ruleset
 
-You can utilize the [Template `TranslationRuleset`](./Rules_and_Rulesets/template)
-as a starter, to create a new [`TranslationRuleset`][orbiter.rules.rulesets.TranslationRuleset].
+You can utilize the [`TranslationRuleset` Template](./Rules_and_Rulesets/template)
+to create a new [`TranslationRuleset`][orbiter.rules.rulesets.TranslationRuleset].
 
 ## FAQ
 - **Can this tool convert my workflows from tool X to Airflow?**

docs/objects/Tasks/index.md

@@ -4,7 +4,7 @@ are units of work. An Operator is a pre-defined task with specific functionality
 
 Operators can be looked up in the [Astronomer Registry](https://registry.astronomer.io/).
 
-The easiest way to utilize an operator is to use a subclass of `OrbiterOperator` (e.g. `OrbiterBashOperator`).
+The easiest way to create an operator in a translation to [use an existing subclass of `OrbiterOperator` (e.g. `OrbiterBashOperator`)](./Operators_and_Callbacks/operators).
 
 If an `OrbiterOperator` subclass doesn't exist for your use case, you can:
 
@@ -27,7 +27,8 @@ If an `OrbiterOperator` subclass doesn't exist for your use case, you can:
         )
     ```
 
-2) Create a new subclass of `OrbiterOperator` (beneficial if you are using it frequently in separate `@task_rules`)
+2) Create a new subclass of `OrbiterOperator`, which can be beneficial if you are using it frequently
+   in separate `@task_rules`
     ```python
     from orbiter.objects.task import OrbiterOperator
     from orbiter.objects.requirement import OrbiterRequirement

docs/objects/index.md

@@ -4,7 +4,7 @@
 are **rendered** to produce an Apache Airflow Project
 
 An [`OrbiterProject`][orbiter.objects.project.OrbiterProject] holds everything necessary to render an Airflow Project.
-This is generated by a [`TranslationRuleset.translate_fn`][orbiter.rules.rulesets.TranslationRuleset].
+It is generated by a [`TranslationRuleset.translate_fn`][orbiter.rules.rulesets.TranslationRuleset].
 
 ![Diagram of Orbiter Translation](../orbiter_diagram.png)
 

docs/objects/project.md

@@ -1,5 +1,5 @@
 An [`OrbiterProject`][orbiter.objects.project.OrbiterProject] holds everything necessary to render an Airflow Project.
-This is generated by a [`TranslationRuleset.translate_fn`][orbiter.rules.rulesets.TranslationRuleset].
+It is generated by a [`TranslationRuleset.translate_fn`][orbiter.rules.rulesets.TranslationRuleset].
 
 ## Diagram
 ```mermaid

docs/objects/dags.md → docs/objects/workflow.md

@@ -28,10 +28,6 @@ classDiagram
         --8<-- "orbiter/objects/task.py:mermaid-op-props"
     }
 
-    class OrbiterTaskDependency["orbiter.objects.task.OrbiterTaskDependency"] {
-        --8<-- "orbiter/objects/task.py:mermaid-td-props"
-    }
-
     class OrbiterTimetable["orbiter.objects.timetables.OrbiterTimetable"] {
         --8<-- "orbiter/objects/timetables/__init__.py:mermaid-props"
     }

justfile

@@ -33,6 +33,10 @@ test:
 test-with-coverage:
     {{ PYTHON }} -m pytest -c pyproject.toml --cov=./ --cov-report=xml
 
+# Run integration tests
+test-integration $MANUAL_TESTS="true":
+    @just test
+
 # Run ruff and black (normally done with pre-commit)
 lint:
     ruff check .
@@ -51,7 +55,7 @@ deploy-docs UPSTREAM="origin": clean
 
 # Remove temporary or build folders
 clean:
-    rm -rf build dist site *.egg-info
+    rm -rf build dist site *.egg-info *.pyz orbiter-* workflow output
     find . | grep -E "(__pycache__|\.pyc|\.pyo$$)" | xargs rm -rf
 
 # Tag as v$(<src>.__version__) and push to Github
@@ -91,13 +95,14 @@ docker-build-binary:
     just build-binary
     EOF
 
-docker-run-binary REPO='astronomer-orbiter-translations' RULESET='orbiter_translations.control_m.xml_base.translation_ruleset':
+docker-run-binary REPO='orbiter-community-translations' RULESET='orbiter_translations.oozie.xml_demo.translation_ruleset':
     #!/usr/bin/env bash
     set -euxo pipefail
     cat <<"EOF" | docker run --platform linux/amd64 -v `pwd`:/data -w /data -i ubuntu /bin/bash
     chmod +x ./orbiter-linux-x86_64 && \
     set -a && source .env && set +a && \
-    ./orbiter-linux-x86_64 help && \
+    ./orbiter-linux-x86_64 list-rulesets && \
+    mkdir -p workflow && \
     LOG_LEVEL=DEBUG ./orbiter-linux-x86_64 install --repo={{REPO}} && \
     LOG_LEVEL=DEBUG ./orbiter-linux-x86_64 translate workflow/ output/ --ruleset {{RULESET}}
     EOF

mkdocs.yml

@@ -109,5 +109,7 @@ plugins:
           docstring_options:
             trim_doctest_flags: true
           show_bases: false
+          extensions:
+          - griffe_inherited_docstrings
 
 copyright: "Apache Airflow® is a trademark of the Apache Software Foundation. Copyright 2024 Astronomer, Inc."

orbiter/__init__.py

@@ -1,15 +1,19 @@
 from __future__ import annotations
 
+import os
 import re
 from enum import Enum
 from typing import Any, Tuple
 
-__version__ = "1.0.1"
+__version__ = "1.0.2a1"
 
 version = __version__
 
 KG_ACCOUNT_ID = "3b189b4c-c047-4fdb-9b46-408aa2978330"
 
+ORBITER_TASK_SUFFIX = os.getenv("ORBITER_TASK_SUFFIX", "_task")
+"""By default, we add `_task` as a suffix to a task name to prevent name collision issues. This can be overridden."""
+
 
 class FileType(Enum):
     YAML = "YAML"

orbiter/__main__.py

@@ -170,17 +170,19 @@ def translate(
 
     Provide a specific ruleset with the `--ruleset` flag.
 
-    Run `orbiter help` to see available rulesets.
+    Run `orbiter list-rulesets` to see available rulesets.
 
     `INPUT_DIR` defaults to `$CWD/workflow`.
 
     `OUTPUT_DIR` defaults to `$CWD/output`
+
+    Formats output with Ruff (https://astral.sh/ruff), by default.
     """
     logger.debug(f"Creating output directory {output_dir}")
     output_dir.mkdir(parents=True, exist_ok=True)
 
-    sys.path.insert(0, os.getcwd())
     logger.debug(f"Adding current directory {os.getcwd()} to sys.path")
+    sys.path.insert(0, os.getcwd())
 
     if RUNNING_AS_BINARY:
         _add_pyz()
@@ -265,7 +267,7 @@ def _bin_install(repo: str, key: str):
         raise NotImplementedError()
     _add_pyz()
     (_, _version) = import_from_qualname("orbiter_translations.version")
-    logging.info(f"Successfully installed {repo}, version: {_version}")
+    logger.info(f"Successfully installed {repo}, version: {_version}")
 
 
 # noinspection t
@@ -284,8 +286,8 @@ def _bin_install(repo: str, key: str):
 @click.option(
     "-k",
     "--key",
-    help="[Optional] License Key to use for the translation ruleset.\n\n"
-    "Should look like 'AAAA-BBBB-1111-2222-3333-XXXX-YYYY-ZZZZ'",
+    help="[Optional] License Key to use for the translation ruleset. Should look like "
+    "`AAAA-BBBB-1111-2222-3333-XXXX-YYYY-ZZZZ`",
     type=str,
     default=None,
     allow_from_autoenv=True,
@@ -298,7 +300,7 @@ def install(
     ),
     key: str | None,
 ):
-    """Install a new Orbiter Translation Ruleset from a repository"""
+    """Install a new Translation Ruleset from a repository"""
     if not repo:
         choices = [
             "astronomer-orbiter-translations",
@@ -329,7 +331,7 @@ def install(
 
 # noinspection PyShadowingBuiltins
 @orbiter.command(help="List available Translation Rulesets")
-def help():
+def list_rulesets():
     console = Console()
 
     table = tabulate(

orbiter/assets/supported_origins.csv

@@ -1,12 +1,15 @@
-Origin,Maintainer,Repository,Ruleset(s),Task Equivalent,DAG Equivalent
-DAG Factory,Community,[`orbiter-community-translations`](https://github.com/astronomer/orbiter-community-translations),`orbiter_translations.dag_factory.yaml_base.translation_ruleset`,---,---
-Control M,Astronomer,`astronomer-orbiter-translations`,`orbiter_translations.control_m.json_base.translation_ruleset`,Job,Folder
-⠀,⠀,⠀,`orbiter_translations.control_m.json_ssh.translation_ruleset`,⠀,⠀
+Origin,Maintainer,Repository,Ruleset,DAG Equivalent,Task Equivalent
+DAG Factory,Community,[`orbiter-community-translations`](https://github.com/astronomer/orbiter-community-translations),`orbiter_translations.dag_factory.yaml_base.translation_ruleset`,DAG,Task
+Control M,Astronomer,`astronomer-orbiter-translations`,`orbiter_translations.control_m.json_base.translation_ruleset`,Folder,Job
+⠀⠀,⠀,⠀,`orbiter_translations.control_m.json_ssh.translation_ruleset`,⠀,⠀
 ⠀,⠀,⠀,`orbiter_translations.control_m.xml_base.translation_ruleset`,⠀,⠀
 ⠀,⠀,⠀,`orbiter_translations.control_m.xml_ssh.translation_ruleset`,⠀,⠀
-Automic,Astronomer,`astronomer-orbiter-translations`,WIP,Job,Job Plan
+⠀,⠀,[`orbiter-community-translations`](https://github.com/astronomer/orbiter-community-translations),`orbiter_translations.control_m.xml_demo.translation_ruleset`,⠀,⠀
+Automic,Astronomer,`astronomer-orbiter-translations`,WIP,Job Plan,Job
 Autosys,Astronomer,`astronomer-orbiter-translations`,WIP,⠀,⠀
-JAMS,Astronomer,`astronomer-orbiter-translations`,WIP,Job,Folder⠀
-SSIS,Astronomer,`astronomer-orbiter-translations`,`orbiter_translations.ssis.xml_base.translation_ruleset`,⠀,⠀
-Oozie,Astronomer,`astronomer-orbiter-translations`,`orbiter_translations.oozie.xml_base.translation_ruleset`,Node,Workflow
+JAMS,Astronomer,`astronomer-orbiter-translations`,WIP,Folder,Job
+SSIS,Astronomer,`astronomer-orbiter-translations`,`orbiter_translations.ssis.xml_base.translation_ruleset`,Pipeline,Component
+⠀,⠀,[`orbiter-community-translations`](https://github.com/astronomer/orbiter-community-translations),`orbiter_translations.oozie.xml_demo.translation_ruleset`,⠀,⠀
+Oozie,Astronomer,`astronomer-orbiter-translations`,`orbiter_translations.oozie.xml_base.translation_ruleset`,Workflow,Node
+⠀,⠀⠀,[`orbiter-community-translations`](https://github.com/astronomer/orbiter-community-translations),`orbiter_translations.oozie.xml_demo.translation_ruleset`,⠀,⠀
 **& more!**,⠀,⠀,⠀,⠀,⠀

orbiter/ast_helper.py

@@ -149,6 +149,17 @@ def py_function(c: Callable):
     return ast.parse(inspect.getsource(c)).body[0]
 
 
+def py_reference(name: str) -> ast.Expr:
+    """
+    ```pycon
+    >>> render_ast(py_reference("foo"))
+    'foo'
+
+    ```
+    """
+    return ast.Expr(value=ast.Name(id=name))
+
+
 def render_ast(ast_object) -> str:
     return ast.unparse(ast_object)