Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Expand Hera's capacity to deserialize results on behalf of users #817

Open
flaviuvadan opened this issue Oct 26, 2023 · 0 comments
Open
Labels
type:enhancement A general enhancement

Comments

@flaviuvadan
Copy link
Collaborator

Is your feature request related to a problem? Please describe.
It's challenging for users to know when they need to JSON deserialize an output of a step/task in Argo.

Describe the solution you'd like
Hera should have the ability to infer what elements to deserialize based on the specification of a workflow.

Describe alternatives you've considered
Not doing this at all.

Additional context
Here's an example that illustrates the problem. At a high level:

  • we have a workflow that runs a set of steps
  • one of the steps is a reusable set of steps that is called on the output of a script that generates some numbers
  • we want to collate the outputs of those independent steps
  • the independent output of the reusable steps pipeline gets JSON encoded by Argo, resulting in independent integers coming in as JSON values
  • Hera successfully deserializes the JSON list that contains the values but does not deserialize the independent values
@script(image="python:3.10")
def generate_starting_values():
    print(list(range(1, 4)))


@script(image="python:3.10")
def echo(value: int):
    print(value)


@script(image="python:3.10")
def double(value: int):
    doubled = value * 2
    # sleep(30 * (value + 1))
    print(doubled)


@script(image="python:3.10")
def square(value: int):
    squared = value * value
    # sleep(30 * (value + 1))
    print(squared)


@script(image="python:3.10")
def all_sum(values: list):
    print(sum(map(int, values)))  # <<<<<<<<<<<<< we have to use `map` here to turn from str to int


@script(image="python:3.10")
def all_echo(values: list):
    print(values)


with Workflow(
    generate_name="callable-steps-",
    entrypoint="calling-steps",
) as w:
    with Steps(
        name="steps",
        inputs=Parameter(name="value"),
        outputs=Parameter(name="values", value_from=m.ValueFrom(parameter="{{steps.square.outputs.result}}"))
    ) as my_steps:
        e = echo(arguments={"value": "{{inputs.parameters.value}}"})
        d = double(arguments={"value": e.result})
        square(arguments={"value": d.result})

    with Steps(name="calling-steps") as s:
        g = generate_starting_values()
        m = my_steps(arguments={"value": "{{item}}"}, with_param=g.result)
        
        with s.parallel():
            all_sum(arguments={"values": m.get_parameter("values").value})
            all_echo(arguments={"values": m.get_parameter("values").value})

w.create()
@flaviuvadan flaviuvadan added the type:enhancement A general enhancement label Oct 26, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
type:enhancement A general enhancement
Projects
None yet
Development

No branches or pull requests

1 participant