From 0a57dda34e27789fab5fc3a7268eb406e9b815bb Mon Sep 17 00:00:00 2001 From: Arun Pandian Date: Wed, 22 Jan 2025 00:07:47 -0800 Subject: [PATCH 1/2] Add known issue to CHANGES.md --- CHANGES.md | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/CHANGES.md b/CHANGES.md index e5125e643aed..ba033784a138 100644 --- a/CHANGES.md +++ b/CHANGES.md @@ -103,6 +103,7 @@ * Prism * Fixed an edge case where Bundle Finalization might not become enabled. ([#33493](https://github.com/apache/beam/issues/33493)). * Fixed session window aggregation, which wasn't being performed per-key. ([#33542](https://github.com/apache/beam/issues/33542)).) +* [Dataflow Streaming Appliance] Fixed commits failing with KeyCommitTooLargeException when a key outputs >180MB of results. [#33588](https://github.com/apache/beam/issues/33588). ## Security Fixes * Fixed (CVE-YYYY-NNNN)[https://www.cve.org/CVERecord?id=CVE-YYYY-NNNN] (Java/Python/Go) ([#X](https://github.com/apache/beam/issues/X)). @@ -146,7 +147,7 @@ ## Known Issues * [Python] If you are using the official Apache Beam Python containers for version 2.62.0, be aware that they include NumPy version 1.26.4. It is strongly recommended that you explicitly specify numpy==1.26.4 in your project's dependency list. ([#33639](https://github.com/apache/beam/issues/33639)). - +* [Dataflow Streaming Appliance] Commits fail with KeyCommitTooLargeException when a key outputs >180MB of results. Bug affects versions 2.60.0 to 2.62.0, fix will be released with 2.63.0. [#33588](https://github.com/apache/beam/issues/33588). # [2.61.0] - 2024-11-25 @@ -190,6 +191,7 @@ * [Managed Iceberg] DataFile metadata is assigned incorrect partition values ([#33497](https://github.com/apache/beam/issues/33497)). * Fixed in 2.62.0 * [Python] If you are using the official Apache Beam Python containers for version 2.61.0, be aware that they include NumPy version 1.26.4. It is strongly recommended that you explicitly specify numpy==1.26.4 in your project's dependency list. ([#33639](https://github.com/apache/beam/issues/33639)). +* [Dataflow Streaming Appliance] Commits fail with KeyCommitTooLargeException when a key outputs >180MB of results. Bug affects versions 2.60.0 to 2.62.0, fix will be released with 2.63.0. [#33588](https://github.com/apache/beam/issues/33588). # [2.60.0] - 2024-10-17 @@ -247,6 +249,7 @@ when running on 3.8. ([#31192](https://github.com/apache/beam/issues/31192)) * Fixed in 2.61.0. * [Managed Iceberg] DataFile metadata is assigned incorrect partition values ([#33497](https://github.com/apache/beam/issues/33497)). * Fixed in 2.62.0 +* [Dataflow Streaming Appliance] Commits fail with KeyCommitTooLargeException when a key outputs >180MB of results. Bug affects versions 2.60.0 to 2.62.0, fix will be released with 2.63.0. [#33588](https://github.com/apache/beam/issues/33588). # [2.59.0] - 2024-09-11 From c9cc8fe1aac50657323e5f800993f49c954dc40e Mon Sep 17 00:00:00 2001 From: Arun Pandian Date: Wed, 22 Jan 2025 02:50:14 -0800 Subject: [PATCH 2/2] Address review comments --- CHANGES.md | 12 +++++++++--- 1 file changed, 9 insertions(+), 3 deletions(-) diff --git a/CHANGES.md b/CHANGES.md index ba033784a138..8bcd3b6ab421 100644 --- a/CHANGES.md +++ b/CHANGES.md @@ -147,7 +147,9 @@ ## Known Issues * [Python] If you are using the official Apache Beam Python containers for version 2.62.0, be aware that they include NumPy version 1.26.4. It is strongly recommended that you explicitly specify numpy==1.26.4 in your project's dependency list. ([#33639](https://github.com/apache/beam/issues/33639)). -* [Dataflow Streaming Appliance] Commits fail with KeyCommitTooLargeException when a key outputs >180MB of results. Bug affects versions 2.60.0 to 2.62.0, fix will be released with 2.63.0. [#33588](https://github.com/apache/beam/issues/33588). +* [Dataflow Streaming Appliance] Commits fail with KeyCommitTooLargeException when a key outputs >180MB of results. Bug affects versions 2.60.0 to 2.62.0, + * fix will be released with 2.63.0. [#33588](https://github.com/apache/beam/issues/33588). + * To resolve this issue, downgrade to 2.59.0 or upgrade to 2.63.0 or enable [Streaming Engine](https://cloud.google.com/dataflow/docs/streaming-engine#use). # [2.61.0] - 2024-11-25 @@ -191,7 +193,9 @@ * [Managed Iceberg] DataFile metadata is assigned incorrect partition values ([#33497](https://github.com/apache/beam/issues/33497)). * Fixed in 2.62.0 * [Python] If you are using the official Apache Beam Python containers for version 2.61.0, be aware that they include NumPy version 1.26.4. It is strongly recommended that you explicitly specify numpy==1.26.4 in your project's dependency list. ([#33639](https://github.com/apache/beam/issues/33639)). -* [Dataflow Streaming Appliance] Commits fail with KeyCommitTooLargeException when a key outputs >180MB of results. Bug affects versions 2.60.0 to 2.62.0, fix will be released with 2.63.0. [#33588](https://github.com/apache/beam/issues/33588). +* [Dataflow Streaming Appliance] Commits fail with KeyCommitTooLargeException when a key outputs >180MB of results. Bug affects versions 2.60.0 to 2.62.0, + * fix will be released with 2.63.0. [#33588](https://github.com/apache/beam/issues/33588). + * To resolve this issue, downgrade to 2.59.0 or upgrade to 2.63.0 or enable [Streaming Engine](https://cloud.google.com/dataflow/docs/streaming-engine#use). # [2.60.0] - 2024-10-17 @@ -249,7 +253,9 @@ when running on 3.8. ([#31192](https://github.com/apache/beam/issues/31192)) * Fixed in 2.61.0. * [Managed Iceberg] DataFile metadata is assigned incorrect partition values ([#33497](https://github.com/apache/beam/issues/33497)). * Fixed in 2.62.0 -* [Dataflow Streaming Appliance] Commits fail with KeyCommitTooLargeException when a key outputs >180MB of results. Bug affects versions 2.60.0 to 2.62.0, fix will be released with 2.63.0. [#33588](https://github.com/apache/beam/issues/33588). +* [Dataflow Streaming Appliance] Commits fail with KeyCommitTooLargeException when a key outputs >180MB of results. Bug affects versions 2.60.0 to 2.62.0, + * fix will be released with 2.63.0. [#33588](https://github.com/apache/beam/issues/33588). + * To resolve this issue, downgrade to 2.59.0 or upgrade to 2.63.0 or enable [Streaming Engine](https://cloud.google.com/dataflow/docs/streaming-engine#use). # [2.59.0] - 2024-09-11