examples/component-catalog-connectors/artifactory-connector at main · elyra-ai/examples

Name	Name	Last commit message	Last commit date
parent directory ..
artifactory_catalog_connector	artifactory_catalog_connector
tests	tests
.flake8	.flake8
MANIFEST.in	MANIFEST.in
Makefile	Makefile
README.md	README.md
setup.cfg	setup.cfg
setup.py	setup.py
test_requirements.txt	test_requirements.txt

Artifactory component catalog connector

This catalog connector enables Elyra to load Kubeflow Pipelines components from a generic-type Artifactory repo.

🟨 Note 🟨

Currently, this connector only works with --runtime_type=KUBEFLOW_PIPELINES, we welcome contributions for other runtime types.

Install the connector

You can install the Artifactory catalog connector from PyPI or source code.

Prerequisites

Elyra (version 3.7 and above)
A generic-type Artifactory repo

Install from PyPI

$ pip install elyra-artifactory-catalog-connector

Install from source code

$ git clone https://github.com/elyra-ai/examples.git
$ cd examples/component-catalog-connectors/artifactory-connector/
$ make source-install

Create a Catalog

From the UI

Launch JupyterLab
Open the Manage Components panel
Click + then New Artifactory Component Catalog
Specify a "name" for the catalog
(Optional) Specify a "category name" under which the catalog's components will be organized in the palette
Specify the required "Source" parameters:
- artifactory_url
- repository_name
- repository_path
- max_recursion_depth
- max_files_per_folder
- file_filter
- file_ordering
(Optional) Specify the optional "Source" parameters:
- artifactory_username
- artifactory_password
Save the catalog entry
Open the Visual Pipeline Editor and expand the palette. The catalog components are displayed.

From the CLI

$ elyra-metadata install component-catalogs \
    --schema_name="artifactory-catalog" \
    --name="my_artifactory" \
    --display_name="My Artifactory" \
    --description="components from my artifactory" \
    --runtime_type="KUBEFLOW_PIPELINES" \
    --categories='["My Artifactory"]' \
    --artifactory_url="https://example.com/artifactory/" \
    --repository_name="my-kubeflow-components" \
    --repository_path="/" \
    --max_recursion_depth="3" \
    --max_files_per_folder="-1" \
    --file_filter="*.yaml" \
    --file_ordering="VERSION_DESCENDING"

Documentation

Introduction

The basic idea, is to create a generic-type Artifactory repo with a folder structure similar to the following:

my_component_1/
   __COMPONENT__
   1.0.0.yaml
   1.1.0.yaml
   2.0.0.yaml
my_component_2/
   __COMPONENT__
   1.0.0.yaml
   1.1.0.yaml
   2.0.0.yaml
special_components/
   special_component_1/
      __COMPONENT__
      1.0.0.yaml
      1.1.0.yaml
      2.0.0.yaml
   special_component_2/
      __COMPONENT__
      1.0.0.yaml
      1.1.0.yaml
      2.0.0.yaml

The most important thing to understand are the __COMPONENT__ marker files, which tell the connector that a folder contains a component.

The content of the __COMPONENT__ files is not considered, only the fact that they exist.
The presence of a __COMPONENT__ file in a folder prevents further recursion into sub-folders of that folder.

Catalog Configs

Name	CLI Parameter	Description	Example	Required
Artifactory URL	`artifactory_url`	URL of the Artifactory server	`https://example.com/artifactory/`	YES
Artifactory Username	`artifactory_username`	Username for the Artifactory server	`service_user`	NO
Artifactory Password/API-Key	`artifactory_password`	Password or API Key for the Artifactory server	`password123`	NO
Repository Name	`repository_name`	Name of the Artifactory repository	`my-kubeflow-components`	YES
Repository Path	`repository_path`	Path of folder in repository to search under	`/path/to/components/`	YES
Maximum Recursion Depth	`max_recursion_depth`	Maximum folder depth to recurse looking for `__COMPONENT__` marker files	`3`	YES
Maximum Files Per Folder	`max_files_per_folder`	Maximum number of files returned from each folder. ('-1' is unlimited)	`-1`	YES
File Filter	`file_filter`	Fnmatch file name filter. `*` match everything; `?` any single character; `[seq]` character in seq; `[!seq]` character not in seq; `[0-9]` any single number;	`*.yaml`	YES
File Ordering	`file_ordering`	Order in which files are processed per folder. `NAME_ASCENDING` alphanumeric ascending; `NAME_DESCENDING` alphanumeric descending; `VERSION_ASCENDING` packaging.version.LegacyVersion() ascending; `VERSION_DESCENDING` packaging.version.LegacyVersion() descending;	`VERSION_DESCENDING`	YES

Component Versioning

Kubeflow has no native concept of "versioning" the same Component YAML over time (proposal to add one).

If changes are made to a Component's YAML, existing pipelines may break or do unexpected things. Given this, we recommend treating each Component YAML as atomic, and creating a new YAML file for every change (never updating the old ones).

When doing this, using Semantic Versions for file names is sensible. For example, you may initially create a file called 0.1.0.yaml, then make a change and create a new file called X.X.X.yaml (the appropriate version number depends on if the change was backwards-compatible or not).

🟦 Tip 🟦

Make sure to include the version in each component's name so they can be distinguished in the Elyra UI.

For example, this component has 1.0.0 in its name:
name: My Component - 1.0.0
description: ""
inputs: []
outputs: []
implementation: {}

🟥 Warning 🟥

For Elyra to run a pipeline with components from a catalog, those components must be present in one of your catalogs.

This means your catalogs must continue to include ALL old component versions, otherwise pipelines using old components will be unable to run.

To keep the UI clean, you may want to create TWO catalog instances:

that ONLY has the LATEST version of each component:

set max_files_per_folder to 1

set file_ordering to VERSION_DESCENDING

that has ALL versions of each component:

set max_files_per_folder to -1

Examples

Assume an artifactory server https://example.com/artifactory/ has a repository called elyra-components with the following folder structure:

component_one/
   __COMPONENT__
   1.0.0.yaml
   2.0.0.yaml
missing_marker/
   1.0.0.yaml
   2.0.0.yaml
component_group/
   component_two/
      __COMPONENT__
      1.0.0.yaml
      2.0.0.yaml
      nested_component/
         __COMPONENT__
         1.0.0.yaml
         2.0.0.yaml
   component_three/
      __COMPONENT__
      1.0.0.yaml
      2.0.0.yaml

Example 1

Connector configuration:

artifactory_url      = https://example.com/artifactory/
repository_name      = elyra-components
repository_path      = /
max_recursion_depth  = 3
max_files_per_folder = -1
file_filter          = *.yaml
file_ordering        = VERSION_DESCENDING

Matched components:

/component_one/1.0.0.yaml
/component_one/2.0.0.yaml
/component_group/component_two/1.0.0.yaml
/component_group/component_two/2.0.0.yaml
/component_group/component_three/1.0.0.yaml
/component_group/component_three/2.0.0.yaml

Notes:

the /missing_marker files are not matched as this folder does not contain a __COMPONENT__ marker
the /component_group/component_two/nested_component files are not matched as recursion stops at the first __COMPONENT__ marker

Example 2

Connector configuration:

artifactory_url      = https://example.com/artifactory/
repository_name      = elyra-components
repository_path      = /
max_recursion_depth  = 3
max_files_per_folder = 1
file_filter          = *.yaml
file_ordering        = VERSION_DESCENDING

Matched components:

/component_one/2.0.0.yaml
/component_group/component_two/2.0.0.yaml
/component_group/component_three/2.0.0.yaml

Notes:

as max_files_per_folder is 1, only ONE file from each folder is matched
as file_ordering is VERSION_DESCENDING, the HIGHEST version file is returned
- we use packaging.version.LegacyVersion() to preform the sort
- the file_ordering is applied separately within each folder
- the WHOLE file-name is treated as a version, so "aaaa-1.0.0.yaml" would be sorted before "bbbb-9.0.0.yaml" (to avoid this problem, don't include any prefix on file names, for example 1.0.0.yaml, 1.1.0.yaml)

Example 3

Connector configuration:

artifactory_url      = https://example.com/artifactory/
repository_name      = elyra-components
repository_path      = /
max_recursion_depth  = 0
max_files_per_folder = -1
file_filter          = *.yaml
file_ordering        = VERSION_DESCENDING

Matched components:

None. No files are matched, as max_recursion_depth is 0, and there is no component at the root.

Uninstall the connector

Remove all Artifactory catalog entries from the Manage Components panel
Stop JupyterLab
Uninstall the elyra-artifactory-catalog-connector package:
```
$ pip uninstall elyra-artifactory-catalog-connector
```

Troubleshooting

Problem: The palette does not display any components from the configured catalog.
Solution: If the Elyra GUI does not display any error message indicating that a problem was encountered, inspect the JupyterLab log file.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

artifactory-connector

artifactory-connector

README.md

Artifactory component catalog connector

Install the connector

Prerequisites

Install from PyPI

Install from source code

Create a Catalog

From the UI

From the CLI

Documentation

Introduction

Catalog Configs

Component Versioning

Examples

Example 1

Example 2

Example 3

Uninstall the connector

Troubleshooting

Files

artifactory-connector

Directory actions

More options

Directory actions

More options

Latest commit

History

artifactory-connector

Folders and files

parent directory

README.md

Artifactory component catalog connector

Install the connector

Prerequisites

Install from PyPI

Install from source code

Create a Catalog

From the UI

From the CLI

Documentation

Introduction

Catalog Configs

Component Versioning

Examples

Example 1

Example 2

Example 3

Uninstall the connector

Troubleshooting