Update beam to v2.53 #5133
Update beam to v2.53 #5133
Conversation
| @@ -142,17 +142,13 @@ public void processElement(@Element Iterable<KV<ByteString, Iterable<Mutation>>> | |||
|
|
|||
| @FinishBundle | |||
| public void finishBundle() throws Exception { | |||
| bigtableWriter.flush(); | |||
There was a problem hiding this comment.
is the flush call no longer needed?
There was a problem hiding this comment.
flush is not part of the API anymore. See this commit: apache/beam@b566480
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## main #5133 +/- ##
==========================================
- Coverage 63.35% 63.34% -0.01%
==========================================
Files 291 291
Lines 10840 10840
Branches 753 753
==========================================
- Hits 6868 6867 -1
- Misses 3972 3973 +1 ☔ View full report in Codecov by Sentry. |
| } | ||
|
|
||
| @Teardown | ||
| public void tearDown() throws Exception { |
There was a problem hiding this comment.
Isn't there an overhead when creating a new bigbableWriter on each bundle?
There was a problem hiding this comment.
probably yes, but I don't know how to workaround this since close is the only way to flush now.
I'll simplify the code to remove the null check in the startBundle as we'll always have to create a new writer
There was a problem hiding this comment.
checked in the beam implementation, they also recreate writer on every bundle
Updating beam to 2.53
Some changes in bigtable forced to create some more method in the beam package because package visibility