Arrow performance optimizations

[jprakash-db]

Description

This pull request introduces several performance optimizations for operations involving Apache Arrow tables within the Databricks SQL Python client.

• Reduce overhead and improve efficiency when concatenating Arrow tables, especially when fetching data in batches.
• Additionally, the PR streamlines HTTP download logic and improves code readability and maintainability.

Optimizations

Arrow Table Concatenation Optimizations

Batching Concatenations:
Instead of repeatedly calling pyarrow.concat_tables on pairs of tables (which is inefficient), partial results are now collected into a list (partial_result_chunks) and concatenated only once at the end using pyarrow.concat_tables(partial_result_chunks, use_threads=True).

CloudFetch Downloader Refactor

HTTP Client Consolidation:
Replaces direct use of requests.Session with a singleton pattern via DatabricksHttpClient. This centralizes HTTP handling and is more robust for connection management and configuration.

Benchmarking

Arrow concatentation optimization

Benchmarked using
num_tables : 10000 | row_per_table : 10000 , columns_per_table: 10 | attempts : 10

Metric	pre - latency	post - latency	Improvement
count	10.0s	10.0s
mean	9.26s	1.48s
std	0.78s	0.61s
min	8.27s	0.005s
95%	10.43s	1.89s	81%
99%	10.55s	1.90s	82%
max	10.58s	1.90s

End to end optimization

This includes the end to end test include the arrow update and http client update
benchmarking workspace: benchmarking-staging-aws-us-east-1.staging.cloud.databricks.com
test runs: 10 per benchmark
benchmarking query : SELECT * FROM main.tpcds_sf100_delta.catalog_sales WHERE cs_ship_mode_sk <= 14 AND cs_sold_date_sk BETWEEN 2450815 AND (2450815 + 410) LIMIT {LIMIT} OFFSET {row_offset}

Num of Rows	p95 Pre	p95 Post	Improvement	p99 Pre	p99 Post	Improvement
10,000	4.01s	2.71s	32.41%	4.33s	2.85s	34.1%
100,000	19.52s	14.32s	26%	22.23s	14.55s	34.5%

Summary

Arrow Optimizations - 81% faster ⚡

End to End optimizations - Greater than 30% faster ⚡

[github-actions]

Thanks for your contribution! To satisfy the DCO policy in our contributing guide every commit message must include a sign-off message. One or more of your commits is missing this message. You can reword previous commit messages with an interactive rebase (git rebase -i main).

[github-actions]

Thanks for your contribution! To satisfy the DCO policy in our contributing guide every commit message must include a sign-off message. One or more of your commits is missing this message. You can reword previous commit messages with an interactive rebase (git rebase -i main).

[github-actions]

Thanks for your contribution! To satisfy the DCO policy in our contributing guide every commit message must include a sign-off message. One or more of your commits is missing this message. You can reword previous commit messages with an interactive rebase (git rebase -i main).

[github-actions]

Thanks for your contribution! To satisfy the DCO policy in our contributing guide every commit message must include a sign-off message. One or more of your commits is missing this message. You can reword previous commit messages with an interactive rebase (git rebase -i main).