Update artifact IDL with new time partition #4737
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Codecov ReportAttention:
Additional details and impacted files@@ Coverage Diff @@
## master #4737 +/- ##
==========================================
+ Coverage 58.20% 58.32% +0.12%
==========================================
Files 626 626
Lines 53950 53938 -12
==========================================
+ Hits 31402 31460 +58
+ Misses 20038 19954 -84
- Partials 2510 2524 +14
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. |
wild-endeavor
changed the title
squiishyy
previously approved these changes
Jan 17, 2024
| @@ -19,8 +20,13 @@ message ArtifactBindingData { | |||
| uint32 index = 1; | |||
|
|
|||
| // These two fields are only relevant in the partition value case | |||
| string partition_key = 2; | |||
| string transform = 3; | |||
| oneof partition_data { | |||
There was a problem hiding this comment.
above this, ArtifactBindingData is listed as being only valid for triggers, why is that?
There was a problem hiding this comment.
it's just only used for trigger handling. this message means that the system needs some input, but that input is one of the triggering artifacts that was recently created (as opposed to an input variable).
wild-endeavor
added a commit
to flyteorg/flytekit
that referenced
this pull request
Jan 18, 2024
Signed-off-by: Yee Hing Tong <[email protected]>
Signed-off-by: Yee Hing Tong <[email protected]> update idl Signed-off-by: Yee Hing Tong <[email protected]> make execution manager changes Signed-off-by: Yee Hing Tong <[email protected]> rename field Signed-off-by: Yee Hing Tong <[email protected]>
wild-endeavor
force-pushed
the
artifacts-shell
branch
from
5f2038c to
f0e5c08
Compare
dosubot
bot
added
size:XXL
Signed-off-by: Yee Hing Tong <[email protected]>
squiishyy
approved these changes
Jan 24, 2024
pmahindrakar-oss
pushed a commit
that referenced
this pull request
May 1, 2024
* test changes to time partition (#4737) Signed-off-by: Yee Hing Tong <[email protected]> * trigger from core -> artifact, artifact->artifacts * artifacts.proto package to cloud * data proxy package to cloud * deactivateAllTriggers endpoint * proto package paths, gen protos --------- Signed-off-by: Yee Hing Tong <[email protected]> Co-authored-by: Yee Hing Tong <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The current partition information coerces the time value into a string and uses the special
dskey to indicate that it's a time. This is awkward so let's pull it out into a separate field that's correctly typed.Changes
Admin
IDL
ArtifactBindingDatato be able to bind to the time partition, and also move the old and new field together into aoneof.TimePartitionmessage which has a timestamp field.time_valueto theLabelValuevalue oneof. This value should only be used for the new time partition message.partitionsfield in theArtifactIDnot a oneof. Even if we one day need existence checking, the fact that it's a separate message already gives us that (theHasFieldcall in python and the nil check in go).