feat(sdk): Implement Pipeline Configuration with TTL

[rimolive]

Description of your changes:
This PR introduces a new class PipelineConfig in KFP SDK to add pipeline-specific configurations with the first one: Pipeline TTL.

Testing instructions

SDK

• Create a Python virtualenv and install the SDK and IR YAML API packages locally

$ python -m venv .venv
$ source .venv/bin/activate
$ pip install wheel setuptools
$ pip install -r sdk/python/requirements-dev.txt
$ pip install -e api/v2alpha1/python
$ pip install -e sdk/python

• Use the example code to compile

$ kfp dsl compile --py main.py --output main.yaml

• You should be able to compile and find the following snippet in the main.yaml file:

---
platforms:
  kubernetes:
    pipelineConfig:
      pipelineTtl: 60

Checklist:

• You have signed off your commits
• The title for your pull request (PR) should follow our title convention. Learn more about the pull request title convention used in this repository.

[DharmitD]

@chensun the SDK code part of this PR correctly compiles and generates the Yaml IR as per the requirements we decided upon here.

Code: https://gist.github.com/rimolive/e0c31ccad1f6c963ec010c455b424afe
resulting IR yaml: https://gist.github.com/rimolive/fc900c8172e661a470a6890555159f97

Could you confirm that the hard coded specification of kubernetes platform here would work? Does the platform key in the IR always stay the same, i.e., kubernetes, or is it customized (to have it set as vertex, for example)? If yes, then we can look into having this key dynamically specified and inserted here.

[chensun]

Not sure I fully understand the question. The key is hard coded for individual platforms, and the platform backend is expected to look after the key they recognize and ignore the ones they don't. For instance, the KFP backend extracts only kubernetes key:

pipelines/backend/src/apiserver/template/v2_template.go

Lines 72 to 78 in 6a35ee5

	// Pick out Kubernetes platform configs
	var kubernetesSpec *pipelinespec.SinglePlatformSpec
	if t.platformSpec != nil {
	if _, ok := t.platformSpec.Platforms["kubernetes"]; ok {
	kubernetesSpec = t.platformSpec.Platforms["kubernetes"]
	}
	}

The specific feature you're adding here, TTL, doesn't apply to Vertex. So what you have in the code and IR links look good to me.

[DharmitD]

Thank you for clarifying @chensun !

[google-oss-prow]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign connor-mccarthy for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• api/OWNERS
• sdk/OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[VaniHaripriya]

Verified the changes and was able to generate main.yaml with the below fields:

---
platforms:
  kubernetes:
    pipelineConfig:
      pipelineTtl: 60

[rimolive]

Hold this PR until we get full implementation.

/hold

[DharmitD]

would suggest adding a test to ensure compiled IR yaml has ttl fields as expected. Could use this test as a reference.

[hbelmiro]

@rimolive is it still a WIP?