Flyte Authentication inside Task Containers

[zeryx]

RFC: Flowing Flyte Auth into Task Containers

Abstract

This RFC proposes a method to flow Flyte authentication information into task containers. It aims to simplify access to execution-specific information and enable secure, streamlined usage of Flyte features within tasks. The proposal introduces a Flyte Auth Agent system to bind the authentication context of the task/workflow executor to all executed tasks.

Motivation

In many workflows, accessing execution-specific information (e.g., used images, versions) is crucial. Additionally, tasks often need to interact with Flyte for remote registration or to use Flyte features (e.g., pyflyte register, build, project creation). Currently, this requires creating a Kubernetes App, recording its ID and secret key, and then passing these to a Flyte Task. This approach is either insecure (plaintext files) or overly complex (Kubernetes Secrets).

Proposed Approach

• Flyte Auth Agent System: Introduce a Flyte Auth Agent to bind the authentication of the task/workflow executor to all tasks.
• Automatic Binding: The bound authentication, possibly provided as mounted environment variables, will be automatically accessible within Flytekit.
• Simplified Task Authorization: Tasks will inherit the permissions of the workflow executor, ensuring a secure and streamlined permission model.
• Enhanced Eager Mode: Support for Eager Mode, enabling tasks to trigger downstream tasks securely without needing plaintext credentials or creating cloud-specific secrets.

Example

from flytekit import task, remote

@task
def get_execution_date(execution_id: str) -> str:
    context = remote.default()
    execution = context.fetchExecution()
    return str(execution.date_launched())

Implications

• Security: Enhances security by avoiding the need for plaintext credential handling.
• Simplicity: Simplifies the process for users, removing the need to handle Kubernetes Secrets or configure complex authentication flows.
• Flexibility: Allows for more complex workflows and task interactions within the Flyte ecosystem.

Open Questions

How will the Flyte Auth Agent handle different types of execution environments?
What are the potential scalability implications of binding auth to each task?

[davidmirror-ops]

This is partially coming with eager