[Core feature] --lables Option for pyflyte register

[gilbert-stripe]

Motivation: Why do you think this is important?

We are the ML infra team for our company and we provide Flyte as a platform to our internal users. As a result, we cannot add labels in the tasks or launch plans since we are not the owner of the workflow code.

However, we hope to add some metadata while registering the workflow, and be used when the workflow runs later.
For example, we want to save the git commit hash when user save their models. So we can retrieve the git commit hash by remote.fetch_launch_plan().get_lables().get("git_commit_hash").

In conclusion, the overall process is:

1. Users develop their workflow code
2. Users git commit their workflow code and we build the image by CI. Unfortunately we cannot inject env vars to the image in this step due to company's restriction.
3. ML infra team use the image from step 2 and pyflyte register --labels to register the workflow and inject the metadata in launch labels.
4. We can develop the API (like save_model) by utilizing the metadata we want from the launch plan lables.

Goal: What should the final outcome look like, ideally?

pyflyte register supports adding launch labels.

Describe alternatives you've considered

There is no alternatives.

Propose: Link/Inline OR Additional context

No response

Are you sure this issue hasn't been raised already?

• Yes

Have you read the Code of Conduct?

• Yes

[dosubot]

Hey @gilbert-stripe, great to see you back with another interesting challenge! 😎

Based on the information you've provided and the context I've found, it seems like you can add labels and annotations to workflows and launch plans during the registration process using the pyflyte register command with the --k8s-annotations and --k8s-labels options. These options allow you to specify labels and annotations in a JSON format, which can be used to inject metadata at the time of registration without modifying the workflow code or injecting environment variables into the image at build time.

Here's an example of how you can use these options:

pyflyte register --k8s-labels '{"mylabel": "myvalue"}' --k8s-annotations '{"myannotation": "myvalue"}' ...

Replace {"mylabel": "myvalue"} with your desired labels, such as {"gitcommit": "your_commit_hash"}, and {"myannotation": "myvalue"} with your desired annotations. This way, you can inject metadata like git commit hashes as labels or annotations during the registration process.

Please note, the specific syntax for --labels or --k8s-labels and --k8s-annotations might vary slightly based on the version of flytekit you are using. Always refer to the most recent flytekit documentation or use pyflyte register --help to get the most up-to-date information on command-line options.

Additionally, you can add labels and annotations to workflows and launch plans programmatically during the registration process without using command line options. This can be achieved by using the Labels and Annotations classes from flytekit in the launch plan creation or execution phase. Here's an example:

from flytekit import Labels, Annotations

@workflow
class MyWorkflow(object):
    ...

my_launch_plan = MyWorkflow.create_launch_plan(
    labels=Labels({"myexecutionlabel": "bar", ...}),
    annotations=Annotations({"region": "SEA", ...}),
    ...
)

I hope this helps! If you have any other questions or need further clarification, feel free to ask.

Sources

• How do I add workflow labels and annotations?
• docs/user_guide/productionizing/workflow_labels_and_annotations.md

About Dosu

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

[wild-endeavor]

Looked into this a bit, got confused from the various labels and such that exist already in Flyte. For background,

There are Labels already in a variety of places

• At registration time, in Launch Plans
• At execution time, in the Execution Spec
• As a matchable resource, inside the WorkflowExecutionConfig
• As well as at project creation time.

Note that there is also a tags field in the ExecutionSpec - but don't use this, not yet at least. This field, while it is currently being piped through pyflyte run, is being deprecated. The python side may or may not change, but we're planning on merging it in with Labels. Basically having effectively two things that so very similar things is more support on the backend than it's worth. So we'll merge.

Unrelated, I also wanted to point out there are tags in TaskMetadata and WorkflowMetadata, but these are unrelated and not really exposed through flytekit so best not to use them.

Keep in mind also that the "Labels" for execution and launch plan spec do end up in K8s pod labels, so you're limited to those rules (you can't have an _ in the label key at least, label keys have to be <64 characters in length.

For purposes of this ticket, it sounds like you're just looking for some way to set labels at registration time. I think this is doable once the work is done to merge tags and Labels together and the API/UI changes are made to be able to search on them, etc. There's currently not a --tags option on pyflyte register, only on pyflyte run, but I think it makes sense to add those. It feels like the right place for those to end up is in the LaunchPlanSpec Labels field. These get merged/override any labels that are manually set on in the LaunchPlan constructor in python code.

what do you think @eapolinario ?