Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

deep copying arraynode tasktemplate interface #5479

Merged
merged 1 commit into from
Jun 14, 2024

Conversation

hamersaw
Copy link
Contributor

Tracking issue

NA

Why are the changes needed?

ArrayNode strips the collection from the output type because it currently relies on registering a separate task with a wrapped collection to satisfy node output validators. The bug is that this stripped did no deep copy, so if multiple nodes stripped the outer collection, we would end up stripping mutliple layers if they existed.

What changes were proposed in this pull request?

In this PR we deep copy the interface of the TaskTemplate to ensure that multiple subNode executions do not strip mutliple List from the output variable type.

How was this patch tested?

The following workflow is a minimal repro:

@task(cache=True, cache_version="1.0")
def bAr(a: int) -> List[int]:
    return [a + 1, a + 2]

@task(cache=True, cache_version="1.0")
def bAr2(a: int) -> List[int]:
    return [a + 1, a + 2]

@workflow
def my_wf_2(a: int) -> List[List[int]]:
    x = bAr(a=a)
    return map_task(bAr2)(a=x)

Setup process

Screenshots

Check all the applicable boxes

  • I updated the documentation accordingly.
  • All new and existing tests passed.
  • All commits are signed-off.

Related PRs

NA

Docs link

NA

@hamersaw hamersaw requested a review from pvditt June 14, 2024 18:03
Copy link

codecov bot commented Jun 14, 2024

Codecov Report

Attention: Patch coverage is 0% with 9 lines in your changes missing coverage. Please review.

Project coverage is 60.99%. Comparing base (7d788cb) to head (ddd7c8c).
Report is 133 commits behind head on master.

Files with missing lines Patch % Lines
...g/controller/nodes/array/node_execution_context.go 0.00% 9 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##           master    #5479      +/-   ##
==========================================
+ Coverage   60.15%   60.99%   +0.83%     
==========================================
  Files         646      793     +147     
  Lines       45819    51350    +5531     
==========================================
+ Hits        27562    31319    +3757     
- Misses      15653    17145    +1492     
- Partials     2604     2886     +282     
Flag Coverage Δ
unittests-datacatalog 69.31% <ø> (ø)
unittests-flyteadmin 58.70% <ø> (ø)
unittests-flytecopilot 17.79% <ø> (ø)
unittests-flytectl 67.97% <ø> (?)
unittests-flyteidl 79.04% <ø> (ø)
unittests-flyteplugins 61.84% <ø> (ø)
unittests-flytepropeller 57.30% <0.00%> (-0.02%) ⬇️
unittests-flytestdlib 65.82% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@hamersaw hamersaw merged commit 02bf85f into master Jun 14, 2024
49 of 50 checks passed
@hamersaw hamersaw deleted the bug/arraynode-cache-over-list branch June 14, 2024 18:41
robert-ulbrich-mercedes-benz pushed a commit to robert-ulbrich-mercedes-benz/flyte that referenced this pull request Jul 2, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants