[Ray Data Optimization]Add environment variable overrides for MAX_SA… (…

…#34)

…FE_BLOCK_SIZE_FACTOR

We can increase the factor to prevent block slicing at
`output_buffer.py`

Signed-off-by: ShaochenYu-YW <[email protected]>

python/ray/data/context.py

@@ -36,7 +36,7 @@
 # We will attempt to slice blocks whose size exceeds this factor *
 # target_max_block_size. We will warn the user if slicing fails and we produce
 # blocks larger than this threshold.
-MAX_SAFE_BLOCK_SIZE_FACTOR = 1.5
+MAX_SAFE_BLOCK_SIZE_FACTOR = float(os.environ.get("RAY_DATA_MAX_SAFE_BLOCK_SIZE_FACTOR", "1.5"))
 
 # Dataset will avoid creating blocks smaller than this size in bytes on read.
 # This takes precedence over DEFAULT_MIN_PARALLELISM.