[Core feature] Support Multi-Thread execution in Local Execution

[ericwudayi]

Motivation: Why do you think this is important?

Local execution supports chaining in flyteorg/flytekit#1917. Supporting multi-thread execution can accelerate the total execution time, and it is possible very useful for future hybrid mode execution mentioned in #4205.

Goal: What should the final outcome look like, ideally?

If user define the chain of execution, local execution can support multi-thread execution. The final outcome should be exactly the same with current outcome but with much faster execution time.

Describe alternatives you've considered

NA

Propose: Link/Inline OR Additional context

@task
def t1(a:int) -> int:
    return a + 1

@task
def t2(a: int) -> int:
    return a + 2

@task 
def t3(a: int) ->int:
  return a + 3


## a3 and a1 can execute in the same time. and a2 execute only when a1 is complete.
@workflow
def wf(a: int)-> int:
   a1 = t1(a)
   a2 = t2(a1)
   a3 = t3(a)
   a1 >> a2
   a3

Are you sure this issue hasn't been raised already?

• Yes

Have you read the Code of Conduct?

• Yes

[welcome]

Thank you for opening your first issue here! 🛠

[dosubot]

🤖

That's a great idea! Supporting multi-thread execution in local execution can definitely accelerate the total execution time and be very useful for future hybrid mode execution. Your proposed solution and motivation are clear and well thought out. We encourage you to go ahead and create a pull request for this feature. Your contribution will be valuable to the flyte community. Thank you for your interest in improving flyte!

---

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

[eapolinario]

@ericwudayi , this feature has the potential for increasing the utility of local executions tremendously. How about we turn this into an RFC? Can you initiate a discussion in https://github.com/flyteorg/flyte/discussions/categories/rfc-incubator ?