Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[flytepropeller][flyteadmin] Compiler unknown literal type error handling #5651

Merged

Conversation

Future-Outlier
Copy link
Member

@Future-Outlier Future-Outlier commented Aug 10, 2024

Why are the changes needed?

To avoid a crash when a new IDL type is sent to the propeller or admin.

What changes were proposed in this pull request?

Add error handling and prevent crashes in these sections of the code.

image

How was this patch tested?

most unit tests, and 1 Python example in the comments below.

Setup process

flyteidl: pip install git+https://github.com/flyteorg/flyte.git@2c9cbd89dce53acd1cb90cc042012637ae6c31f8#subdirectory=flyteidl
flytekit: flyteorg/flytekit#2741

python example:

from flytekit import task, workflow, ImageSpec

flytekit_hash = "c24077bce6e63bf8df0d80dbc2c5e2ff3322bca8"

flytekit = f"git+https://github.com/flyteorg/flytekit.git@{flytekit_hash}"

image = ImageSpec(
    packages=[flytekit],
    apt_packages=["git"],
    registry="localhost:30000",
)

@task(container_image=image)
def dict_task(input: dict) -> dict:
    print("Dict fields:")
    for k, v in input.items():
        print(type(k), type(v))
        print(k, v)
    return input

@workflow
def dict_wf(input: dict) -> dict:
    return dict_task(input=input)

if __name__ == "__main__":
    from flytekit.clis.sdk_in_container import pyflyte
    from click.testing import CliRunner

    runner = CliRunner()
    path = "/Users/future-outlier/code/dev/flytekit/build/PR/JSON/demo/dict_example.py"
    result = runner.invoke(pyflyte.main,
                    ["run",
                            path,
                            "dict_wf",
                            "--input",
                            '{"a": 1, "b": 2, "c": [1, 2, 3], "d": {"e": 1, "f": [1, 2, 3]}}'])
    print("Local Execution: ", result.output)
    #
    result = runner.invoke(pyflyte.main,
                           ["run",
                            "--remote",
                            path,
                            "dict_wf",
                            "--input",
                            '{"a": 1, "b": 2, "c": [1, 2, 3], "d": {"e": 1, "f": [1, 2, 3]}}'])
    print("Remote Execution: ", result.output)

Screenshots

image

Check all the applicable boxes

  • I updated the documentation accordingly.
  • All new and existing tests passed.
  • All commits are signed-off.

Related PRs

#5612

Copy link

codecov bot commented Aug 10, 2024

Codecov Report

Attention: Patch coverage is 71.11111% with 13 lines in your changes missing coverage. Please review.

Project coverage is 36.31%. Comparing base (c0fc6d4) to head (455d50c).
Report is 165 commits behind head on master.

Files with missing lines Patch % Lines
flytepropeller/pkg/compiler/validators/utils.go 0.00% 10 Missing ⚠️
flyteadmin/pkg/errors/errors.go 0.00% 3 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##           master    #5651      +/-   ##
==========================================
+ Coverage   36.30%   36.31%   +0.01%     
==========================================
  Files        1305     1305              
  Lines      110002   110051      +49     
==========================================
+ Hits        39932    39964      +32     
- Misses      65914    65932      +18     
+ Partials     4156     4155       -1     
Flag Coverage Δ
unittests-datacatalog 51.37% <ø> (ø)
unittests-flyteadmin 55.58% <80.00%> (-0.04%) ⬇️
unittests-flytecopilot 12.17% <ø> (ø)
unittests-flytectl 62.21% <ø> (ø)
unittests-flyteidl 7.12% <ø> (ø)
unittests-flyteplugins 53.35% <ø> (ø)
unittests-flytepropeller 41.93% <66.66%> (+0.03%) ⬆️
unittests-flytestdlib 55.35% <ø> (+0.13%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@Future-Outlier Future-Outlier changed the title [wip][propeller][admin] Compiler literal type error handling [low priority][wip][propeller][admin] Compiler literal type error handling Aug 26, 2024
Signed-off-by: Future-Outlier <[email protected]>
@Future-Outlier Future-Outlier changed the title [low priority][wip][propeller][admin] Compiler literal type error handling [wip][propeller][admin] Compiler literal type error handling Aug 29, 2024
@Future-Outlier Future-Outlier changed the title [wip][propeller][admin] Compiler literal type error handling [propeller][admin] Compiler literal type error handling Aug 29, 2024
@Future-Outlier Future-Outlier marked this pull request as ready for review August 30, 2024 03:41
Signed-off-by: Future-Outlier <[email protected]>
Signed-off-by: Future-Outlier <[email protected]>
Signed-off-by: Future-Outlier <[email protected]>
Signed-off-by: Future-Outlier <[email protected]>
Signed-off-by: Future-Outlier <[email protected]>
@Future-Outlier Future-Outlier changed the title [propeller][admin] Compiler literal type error handling [propeller][admin] Compiler unknown literal type error handling Aug 30, 2024
pingsutw and others added 4 commits September 5, 2024 00:41
Signed-off-by: Kevin Su <[email protected]>
Signed-off-by: Future-Outlier <[email protected]>
Signed-off-by: Future-Outlier <[email protected]>
Signed-off-by: Future-Outlier <[email protected]>
@Future-Outlier Future-Outlier changed the title [propeller][admin] Compiler unknown literal type error handling [flytepropeller][flyteadmin] Compiler unknown literal type error handling Sep 24, 2024
@Future-Outlier Future-Outlier enabled auto-merge (squash) September 24, 2024 13:09
Future-Outlier and others added 3 commits September 25, 2024 10:27
Signed-off-by: Future-Outlier <[email protected]>
Co-authored-by: pingsutw  <[email protected]>
Signed-off-by: Future-Outlier <[email protected]>
Signed-off-by: Kevin Su <[email protected]>
@Future-Outlier
Copy link
Member Author

Hi, @eapolinario
Do you want to take a look?

@Future-Outlier Future-Outlier merged commit 66ff152 into master Sep 29, 2024
50 checks passed
@Future-Outlier Future-Outlier deleted the improve-validator-literalTypeForLiteral-ErrorMessage branch September 29, 2024 07:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants