airflow-example-components-connector

This connector is deprecated

February 2022:

• The example operators that this connector provides will no longer be updated.
• Please upgrade Elyra to version 3.7 (or above) and use the two built-in connectors instead.
• Use the built-in catalog connector for Apache Airflow packages to add Apache Airflow built-in operators to your visual pipeline editor.
• Use the built-in catalog connector for Apache Airflow provider packages to add Apache Airflow community provider operators to your visual pipeline editor.

---

Apache Airflow component examples catalog

This catalog connector provides access to example pipeline components for Apache Airflow.

Install the component examples

You can install the component examples from PyPI or source code. Note that a rebuild of JupyterLab is not required.

Prerequisites

• Install Elyra (version 3.3 and above).

Install from PyPI

$ pip install elyra-examples-airflow-catalog

Install from source code

$ git clone https://github.com/elyra-ai/examples.git
$ cd examples/component-catalog-connectors/airflow-example-components-connector/
$ make clean source-install

Use the connector

1. Launch JupyterLab.
2. Open the 'Manage Components' panel.
3. Add a new component examples catalog ('+' > 'New Apache Airflow example components catalog').
4. Specify a catalog name, e.g. 'Example components for Apache Airflow'.
5. Save the catalog entry.
6. Open the Visual Pipeline Editor for Apache Airflow and expand the palette. The example components are displayed.

Customize the catalog

This connector utilizes an embedded catalog as storage and is therefore a static catalog. To customize the catalog content according to your needs:

1. Clone or fork the https://github.com/elyra-ai/examples repository.
2. Navigate to the examples/component-catalog-connectors/airflow-example-components-connector/airflow_examples_connector/resources directory. This directory contains the Apache Airflow operators that this connector makes available to Elyra. Note that the operator packages must be installed on the Apache Airflow cluster, or DAG execution will fail.
3. Change the content of this directory as desired.
4. Install the customized connector from source.

Uninstall the connector

1. Remove all example catalog entries from the 'Manage Components' panel.
2. Stop JupyterLab.
3. Uninstall the elyra-examples-airflow-catalog package.

   $ pip uninstall -y elyra-examples-airflow-catalog
   

Troubleshooting

Q: No example components are displayed in the Visual Pipeline Editor palette.

A: Check the JupyterLab log file for error messages.