Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature/array node workflow parallelism #5062

Merged
merged 5 commits into from
Apr 4, 2024

Conversation

pvditt
Copy link
Contributor

@pvditt pvditt commented Mar 15, 2024

Tracking issue

#3924

Why are the changes needed?

Add support for array nodes to utilize the workflow parallelism.

What changes were proposed in this pull request?

How was this patch tested?

Setup process

Screenshots

Check all the applicable boxes

  • I updated the documentation accordingly.
  • All new and existing tests passed.
  • All commits are signed-off.

Related PRs

Docs link

Copy link

codecov bot commented Mar 15, 2024

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 59.07%. Comparing base (452538a) to head (78a7941).

Additional details and impacted files
@@            Coverage Diff             @@
##           master    #5062      +/-   ##
==========================================
+ Coverage   59.06%   59.07%   +0.01%     
==========================================
  Files         646      646              
  Lines       55726    55740      +14     
==========================================
+ Hits        32916    32930      +14     
  Misses      20214    20214              
  Partials     2596     2596              
Flag Coverage Δ
unittests 59.07% <100.00%> (+0.01%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@pvditt pvditt marked this pull request as ready for review March 15, 2024 20:50
@dosubot dosubot bot added size:M This PR changes 30-99 lines, ignoring generated files. enhancement New feature or request labels Mar 15, 2024
@pvditt pvditt requested a review from hamersaw March 15, 2024 20:59
pvditt added 2 commits April 2, 2024 20:52
Signed-off-by: Paul Dittamo <[email protected]>
@dosubot dosubot bot added size:L This PR changes 100-499 lines, ignoring generated files. and removed size:M This PR changes 30-99 lines, ignoring generated files. labels Apr 4, 2024
@pvditt pvditt requested a review from hamersaw April 4, 2024 00:14
hamersaw
hamersaw previously approved these changes Apr 4, 2024
flyteidl/protos/flyteidl/core/workflow.proto Show resolved Hide resolved
@dosubot dosubot bot added the lgtm This PR has been approved by a maintainer label Apr 4, 2024
@pvditt
Copy link
Contributor Author

pvditt commented Apr 4, 2024

@hamersaw could I get another +1 on this - had to resolve a merge conflict

@pvditt pvditt merged commit 24a6e4e into master Apr 4, 2024
48 checks passed
@pvditt pvditt deleted the feature/array-node-support-workflow-parallelism branch April 4, 2024 23:02
Jeinhaus pushed a commit to Jeinhaus/flyte that referenced this pull request Apr 8, 2024
* update arraynode proto parallelism field to varint compatible int64

Signed-off-by: Paul Dittamo <[email protected]>

* have array nodes utilize workflow parallelism

Signed-off-by: Paul Dittamo <[email protected]>

* return if available parallelism is 0

Signed-off-by: Paul Dittamo <[email protected]>

* unit test

Signed-off-by: Paul Dittamo <[email protected]>

---------

Signed-off-by: Paul Dittamo <[email protected]>
pmahindrakar-oss pushed a commit that referenced this pull request May 1, 2024
* Feature/array node workflow parallelism (#5062)

* update arraynode proto parallelism field to varint compatible int64

Signed-off-by: Paul Dittamo <[email protected]>

* have array nodes utilize workflow parallelism

Signed-off-by: Paul Dittamo <[email protected]>

* return if available parallelism is 0

Signed-off-by: Paul Dittamo <[email protected]>

* unit test

Signed-off-by: Paul Dittamo <[email protected]>

---------

Signed-off-by: Paul Dittamo <[email protected]>

* enable parallelism to be set to nil for array node (#5214)

* enable parallelism to be set to nil for array node

Signed-off-by: Paul Dittamo <[email protected]>

* unit test

Signed-off-by: Paul Dittamo <[email protected]>

---------

Signed-off-by: Paul Dittamo <[email protected]>

* added configuration for arraynode default parallelism behavior (#5268)

* added configuration for arraynode default parallelism behavior

Signed-off-by: Daniel Rammer <[email protected]>

* added unit tests and fixed linter

Signed-off-by: Daniel Rammer <[email protected]>

* cleanup / docs

Signed-off-by: Daniel Rammer <[email protected]>

* fixed ytpo

Signed-off-by: Daniel Rammer <[email protected]>

* docs update

Signed-off-by: Daniel Rammer <[email protected]>

* fixed unit tests

Signed-off-by: Daniel Rammer <[email protected]>

---------

Signed-off-by: Daniel Rammer <[email protected]>

---------

Signed-off-by: Paul Dittamo <[email protected]>
Signed-off-by: Daniel Rammer <[email protected]>
Co-authored-by: Paul Dittamo <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request lgtm This PR has been approved by a maintainer size:L This PR changes 100-499 lines, ignoring generated files.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants