RFC: Add 'org' to Flyte identifiers

[katrogan]

RFC: Add 'org' to Flyte identifiers

Abstract

Identifiers, and therefore workflows, tasks and executions within Flyte are always scoped by project and domain. This logical hierarchy is used for handling multitenancy within a single Flyte deployment. Resource quotas based on project and/or domain are used to ensure fairness and per-project specific overrides allow for setting more granular registration and execution time defaults.

Projects and domains are currently the only boundary for defining rules for setting such execution time parameters¹ This RFC proposes adding a top-level org key to all identifiers (workflows, tasks, launch plans and workflow executions, node executions, and task executions) that in isolation or in combination with project and/or domain can be used to selectively apply execution-time isolation and customization.

Motivation

In a single flyte deployment, the recommendation is to use projects to map workflow definitions to specific teams. In some scenarios it might make sense to group project configuration in aggregate, to support templatizing the underlying kubernetes namespaces or labels and annotations at execution time for multiple projects owned by the same team or for generally re-using configuration.

For example, this grouping could be useful in a multicluster deployment or in scenarios where it's desirable to run several projects in a shared namespace for billing, resource or other isolation concerns.

This proposal suggests adding an optional org to all identifiers. When empty or omitted, the system will behave currently as is. When the org is set, execution templatization can be further extended based on the value of the org field.

Proposal

Flyte IDL

An example update to the core identifier adds the following org field:

message Identifier {
    // Identifies the specific type of resource that this identifier corresponds to.
    core.ResourceType resource_type  = 1;

    // Name of the project the resource belongs to.
    string project              = 2;

    // Name of the domain the resource belongs to.
    // A domain can be considered as a subset within a specific project.
    string domain               = 3;

    // User provided value for the resource.
    string name                 = 4;

    // Specific version of the resource.
    string version              = 5;
    
    // Add an optional org.
    string org                     = 6;
}

Flytekit

Trigger executions with an optional org value
pyflyte run example.py training_workflow --project flytesnacks --domain development --org <my_org>

Flytectl

Create a project under an org:
flytectl create project --name flytesnacks --id flytesnacks --description "flytesnacks description" --org <my_org>

Add a matchable attribute override based on partition for all projects and domains belonging to an org.
flytectl update task-resource-attribute --org <my_org> --attrFile tra.yaml

Implications

• Configuration: (+) Allows re-using configuration and execution attributes across groups of projects
• Scaling: (+) Introducing an org allows for horizontally scaling flyteadmin by using org as a shard key. Natural shard key for multiple dataplanes.
• Complexity: (-) Requires plumbing through the org in all registration and execution code paths

Open Questions

Should call this new field by another name.

Footnotes

1. Technically workflows too, see workflow matchable attributes but these are further nested within the top-level project and domain hierarchy. ↩

[hamersaw]

IIUC this will need datacatalog caching considerations right? Executions from two separate partitions with the same function name / interface should not cache hit / write to the same cached item correct? Doing this in a backwards compatible way will be non-trivial.

[katrogan]

yes, we'll need to update the cache keys and make them similarly backwards compatible in the case of unset partitions so that we don't bust existing caches as part of this change!

[squiishyy]

Love the idea and big +1. Some naming ideas here:

• airline
• captain
• brand
• sector
• zone
• nexus
• region

[katrogan]

Thanks @squiishyy! Throwing out some additional options for consideration since this feature can be used in support of multi-tenancy:

• org
• organization
• tenant
• shard

[katrogan]

Updated the RFC based on feedback that 'org' is more desirable here

[wild-endeavor]

actually i think 'organization' is better than 'org', simply cuz we don't shorten 'project'

[katrogan]

@kumare3 pointed out the using an abbreviation here might be desirable to not bloat json serialized versions of the identifier since this has a pretty wide ranging impact, e.g. serialized workflow CRDs)

[davidmirror-ops]

Implemented in #4663