forked from facebookincubator/velox
-
Notifications
You must be signed in to change notification settings - Fork 2
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
feat: Add auto scale writer support (facebookincubator#11702)
Summary: This change adds local scale writer partition support to improve memory efficiency in case of a large number of partitions. We add two customized local partition operators: ScaleWriterLocalPartition for non-partitioned table write. It starts with single table writer thread and scale the writer processing if the exchange queue has >50% memory buffering until scale to all the table writer threads; ScaleWriterPartitioningLocalPartition for partitioned table writer. It starts with assigning a single table writer thread to each logical table partition. Multiple physical table partitions could be mapped to a single logical partition based on the partition keys of the written table. Similar, if the exchange queue has > 50% memory buffering, we leverage the skewed partition balancer by scaling the busy logical table partition by assigning more table writer threads. Meta internal shadow results show this could prevent query write OOM pattern, reduce the peak memory usage which benefits the resource usage accounting which takes into account of accumulated memory usage, it also reduces >2x of written files The followup is to investigate the more reliable rebalance signal such as consumer/producer queuing delay in the exchange queue. To complete this feature, we need a Prestissimo change to setup scale writer local partition based on arbitrary partitioning scheme, and the coordinator needs to configure the query plan accordingly. Reviewed By: arhimondr, oerling, zation99 Differential Revision: D66380785
- Loading branch information
1 parent
28c319e
commit 258e9a3
Showing
28 changed files
with
2,271 additions
and
375 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.