Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Inputs outputs wrapper #4298

Open
wants to merge 40 commits into
base: master
Choose a base branch
from
Open

Inputs outputs wrapper #4298

wants to merge 40 commits into from

Conversation

EngHabu
Copy link
Contributor

@EngHabu EngHabu commented Oct 25, 2023 •
edited
Loading

Tracking issue

#4654

Describe your changes

Check all the applicable boxes

  • I updated the documentation accordingly.
  • All new and existing tests passed.
  • All commits are signed-off.

Screenshots

Note to reviewers

@wild-endeavor wild-endeavor mentioned this pull request Oct 29, 2023
Open
2 tasks
wild-endeavor
wild-endeavor reviewed Oct 30, 2023
View reviewed changes
Copy link
Contributor

@wild-endeavor wild-endeavor left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@pingsutw can you remind me the difference between the agent.proto file and the external_plugin_service.proto file?

flyteidl/protos/flyteidl/service/dataproxy.proto
@@ -147,6 +147,12 @@ message GetDataResponse {
// Single literal will be returned. This is returned when the user/url requests a specific output or input
// by name. See the o3 example above.
core.Literal literal = 3;

// InputData is returned when the user/url requests the input data for an execution.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should this just be one thing instead of two?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

At the moment they are identical messages but separating them out just in case we do want them to diverge... flyteDecks will be part of the output (maybe?) but not an input.... etc.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I could swear I've responded to this 🤯 .
I made them separate in case we want to add unique properties to only inputs or ouputs (e.g. FlyteDecks)

flytepropeller/cmd/controller/cmd/root.go Outdated
SyncPeriod: &cfg.DownstreamEval.Duration,
NewClient: func(cache cache.Cache, config *rest.Config, options client.Options, uncachedObjects ...client.Object) (client.Client, error) {
return executors.NewFallbackClientBuilder(propellerScope.NewSubScope("kube")).Build(cache, config, options)
Cache: cache.Options{
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

are these changes related or is this just github flaking?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this is what it looks like in main: https://github.com/flyteorg/flyte/blob/master/flytepropeller/cmd/controller/cmd/root.go#L132-L167 so I'm going to say glitching

flyteidl/protos/flyteidl/admin/agent.proto
// Inputs are the inputs required to start the execution. All required inputs must be
// included in this map. If not required and not provided, defaults apply.
// +optional
core.InputData inputs = 5;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

will this break existing agents? so you'll need to upgrade all agents to read this new field when deploying this change?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We will populate both... so that the agent change can happen separately

@codecov Codecov
Copy link

codecov bot commented Dec 14, 2023 •
edited
Loading

Codecov Report

Attention: Patch coverage is 60.86957% with 90 lines in your changes missing coverage. Please review.

Project coverage is 62.24%. Comparing base (f075b34) to head (ddc3fe0).

Files Patch % Lines
flyteadmin/pkg/manager/impl/util/data.go 32.65% 28 Missing and 5 partials ⚠️
flyteadmin/pkg/manager/impl/execution_manager.go 70.51% 18 Missing and 5 partials ⚠️
...teadmin/pkg/repositories/transformers/execution.go 46.15% 12 Missing and 2 partials ⚠️
...pkg/manager/impl/validation/execution_validator.go 59.09% 7 Missing and 2 partials ⚠️
...in/pkg/repositories/transformers/node_execution.go 62.50% 2 Missing and 1 partial ⚠️
...in/pkg/repositories/transformers/task_execution.go 62.50% 2 Missing and 1 partial ⚠️
flyteadmin/pkg/common/data_store.go 66.66% 1 Missing and 1 partial ⚠️
...eadmin/pkg/async/schedule/aws/workflow_executor.go 92.30% 0 Missing and 1 partial ⚠️
...anager/impl/validation/node_execution_validator.go 80.00% 0 Missing and 1 partial ⚠️
...anager/impl/validation/task_execution_validator.go 80.00% 0 Missing and 1 partial ⚠️
Additional details and impacted files 
@@            Coverage Diff             @@
##           master    #4298      +/-   ##
==========================================
+ Coverage   60.92%   62.24%   +1.32%     
==========================================
  Files         796      462     -334     
  Lines       51689    27849   -23840     
==========================================
- Hits        31494    17336   -14158     
+ Misses      17288     8949    -8339     
+ Partials     2907     1564    -1343
Flag Coverage Δ
unittests-datacatalog 69.31% <ø> (ø)
unittests-flyteadmin 58.47% <60.86%> (-0.26%) ⬇️
unittests-flytecopilot 17.79% <ø> (ø)
unittests-flytectl 67.41% <ø> (ø)
unittests-flyteidl 79.04% <ø> (ø)
unittests-flyteplugins ?
unittests-flytepropeller ?
unittests-flytestdlib 65.44% <ø> (-0.15%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@EngHabu
Use the right error type for parsing connection errors
e6ad943 
Signed-off-by: Haytham Abuelfutuh <[email protected]>
@EngHabu
wip
8961f19 
Signed-off-by: Haytham Abuelfutuh <[email protected]>
@EngHabu
regen
57fa63d 
Signed-off-by: Haytham Abuelfutuh <[email protected]>
@EngHabu
wip
d250dd2 
Signed-off-by: Haytham Abuelfutuh <[email protected]>
@EngHabu
Update bindings to read the new format
77fae93 
Signed-off-by: Haytham Abuelfutuh <[email protected]>
@EngHabu EngHabu mentioned this pull request Dec 29, 2023
Draft
3 tasks
@eapolinario
Copy link
Contributor

There's something going wrong with named tuples. This is the workflow that's failing: https://github.com/flyteorg/flytesnacks/blob/master/examples/basics/basics/named_outputs.py

The error:

Screenshot 2024-01-17 at 3 27 19 PM

@EngHabu
check for proto diff
624e57f 
Signed-off-by: Haytham Abuelfutuh <[email protected]>
@EngHabu EngHabu force-pushed the inputs-outputs-wrapper branch from 4540160 to 624e57f Compare January 23, 2024 17:21
@EngHabu
Merge branch 'master' of github.com:flyteorg/flyte into inputs-output…
4965b1e 
…s-wrapper
@EngHabu
Lint
62e37cb 
Signed-off-by: Haytham Abuelfutuh <[email protected]>
@EngHabu
Try Tmate
4b59975 
Signed-off-by: Haytham Abuelfutuh <[email protected]>
@EngHabu EngHabu marked this pull request as ready for review January 25, 2024 16:58
@dosubot dosubot bot added size:XXL This PR changes 1000+ lines, ignoring generated files. enhancement New feature or request labels Jan 25, 2024
@EngHabu
Merge branch 'master' of github.com:flyteorg/flyte into inputs-output…
e95d7ae 
…s-wrapper
@EngHabu
Merge branch 'master' of github.com:flyteorg/flyte into inputs-output…
af6e862 
…s-wrapper
@EngHabu
Merge branch 'master' of github.com:flyteorg/flyte into inputs-output…
8fc6c33 
…s-wrapper
@EngHabu
merge conflict
3273c5e 
Signed-off-by: Haytham Abuelfutuh <[email protected]>
@EngHabu
Merge branch 'master' of github.com:flyteorg/flyte into inputs-output…
5f083c8 
…s-wrapper
@EngHabu
Merge branch 'master' of github.com:flyteorg/flyte into inputs-output…
f250697 
…s-wrapper
@EngHabu
Remaining merge conflicts
ede1297 
Signed-off-by: Haytham Abuelfutuh <[email protected]>
@EngHabu
Merge branch 'master' of github.com:flyteorg/flyte into inputs-output…
76005db 
…s-wrapper
@EngHabu
Fix make generate and run it
ddc3fe0 
Signed-off-by: Haytham Abuelfutuh <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@wild-endeavor wild-endeavor wild-endeavor left review comments

At least 1 approving review is required to merge this pull request.

Assignees
No one assigned
Labels
enhancement New feature or request size:XXL This PR changes 1000+ lines, ignoring generated files.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@EngHabu @eapolinario @wild-endeavor