-
Notifications
You must be signed in to change notification settings - Fork 4
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
- Loading branch information
Showing
3 changed files
with
177 additions
and
66 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,92 +1,157 @@ | ||
| # databricks-labs-lsql | ||
| Databricks Labs LSQL | ||
| === | ||
|
|
||
| [](https://pypi.org/project/databricks-labs-lightsql) | ||
| [](https://pypi.org/project/databricks-labs-lightsql) | ||
|
|
||
| ----- | ||
| [](https://pypi.org/project/databricks-labs-lsql) | ||
| [](https://github.com/databrickslabs/lsql/actions/workflows/push.yml) [](https://codecov.io/github/databrickslabs/ucx) []([https://codecov.io/github/databrickslabs/lsql](https://github.com/databrickslabs/lsql)) | ||
|
|
||
| Execute SQL statements in a stateless manner. | ||
|
|
||
| ## Installation | ||
| <!-- TOC --> | ||
| * [Databricks Labs LSQL](#databricks-labs-lsql) | ||
| * [Installation](#installation) | ||
| * [Executing SQL](#executing-sql) | ||
| * [Iterating over results](#iterating-over-results) | ||
| * [Executing without iterating](#executing-without-iterating) | ||
| * [Fetching one record](#fetching-one-record) | ||
| * [Fetching one value](#fetching-one-value) | ||
| * [Parameters](#parameters) | ||
| * [SQL backend abstraction](#sql-backend-abstraction) | ||
| * [Project Support](#project-support) | ||
| <!-- TOC --> | ||
|
|
||
| # Installation | ||
|
|
||
| ```console | ||
| pip install databricks-labs-lsql | ||
| ``` | ||
|
|
||
| ## Executing SQL | ||
| [[back to top](#databricks-labs-lsql)] | ||
|
|
||
| # Executing SQL | ||
|
|
||
| Primary use-case of :py:meth:`iterate_rows` and :py:meth:`execute` methods is oriented at executing SQL queries in | ||
| Primary use-case of :py:meth:`fetch_all` and :py:meth:`execute` methods is oriented at executing SQL queries in | ||
| a stateless manner straight away from Databricks SDK for Python, without requiring any external dependencies. | ||
| Results are fetched in JSON format through presigned external links. This is perfect for serverless applications | ||
| like AWS Lambda, Azure Functions, or any other containerised short-lived applications, where container startup | ||
| time is faster with the smaller dependency set. | ||
|
|
||
| for (pickup_zip, dropoff_zip) in w.statement_execution.iterate_rows(warehouse_id, | ||
| 'SELECT pickup_zip, dropoff_zip FROM nyctaxi.trips LIMIT 10', catalog='samples'): | ||
| print(f'pickup_zip={pickup_zip}, dropoff_zip={dropoff_zip}') | ||
| Applications, that need to a more traditional SQL Python APIs with cursors, efficient data transfer of hundreds of | ||
| megabytes or gigabytes of data serialized in Apache Arrow format, and low result fetching latency, should use | ||
| the stateful Databricks SQL Connector for Python. | ||
|
|
||
| Method :py:meth:`iterate_rows` returns an iterator of objects, that resemble :class:`pyspark.sql.Row` APIs, but full | ||
| compatibility is not the goal of this implementation. | ||
| Constructor and the most of the methods do accept [common parameters](#parameters). | ||
|
|
||
| iterate_rows = functools.partial(w.statement_execution.iterate_rows, warehouse_id, catalog='samples') | ||
| for row in iterate_rows('SELECT * FROM nyctaxi.trips LIMIT 10'): | ||
| pickup_time, dropoff_time = row[0], row[1] | ||
| pickup_zip = row.pickup_zip | ||
| dropoff_zip = row['dropoff_zip'] | ||
| all_fields = row.as_dict() | ||
| print(f'{pickup_zip}@{pickup_time} -> {dropoff_zip}@{dropoff_time}: {all_fields}') | ||
| ```python | ||
| from databricks.sdk import WorkspaceClient | ||
| from databricks.labs.lsql.core import StatementExecutionExt | ||
| w = WorkspaceClient() | ||
| see = StatementExecutionExt(w) | ||
| for (pickup_zip, dropoff_zip) in see('SELECT pickup_zip, dropoff_zip FROM samples.nyctaxi.trips LIMIT 10'): | ||
| print(f'pickup_zip={pickup_zip}, dropoff_zip={dropoff_zip}') | ||
| ``` | ||
|
|
||
| When you only need to execute the query and have no need to iterate over results, use the :py:meth:`execute`. | ||
| [[back to top](#databricks-labs-lsql)] | ||
|
|
||
| w.statement_execution.execute(warehouse_id, 'CREATE TABLE foo AS SELECT * FROM range(10)') | ||
| ## Iterating over results | ||
|
|
||
| ## Working with dataclasses | ||
| Method `fetch_all` returns an iterator of objects, that resemble `pyspark.sql.Row` APIs, but full | ||
| compatibility is not the goal of this implementation. Method accepts [common parameters](#parameters). | ||
|
|
||
| This framework allows for mapping with strongly-typed dataclasses between SQL and Python runtime. | ||
| ```python | ||
| import os | ||
| from databricks.sdk import WorkspaceClient | ||
| from databricks.labs.lsql.core import StatementExecutionExt | ||
|
|
||
| It handles the schema creation logic purely from Python datastructure. | ||
| results = [] | ||
| w = WorkspaceClient() | ||
| see = StatementExecutionExt(w, warehouse_id=os.environ.get("TEST_DEFAULT_WAREHOUSE_ID")) | ||
| for pickup_zip, dropoff_zip in see.fetch_all("SELECT pickup_zip, dropoff_zip FROM samples.nyctaxi.trips LIMIT 10"): | ||
| results.append((pickup_zip, dropoff_zip)) | ||
| ``` | ||
|
|
||
| ## Mocking for unit tests | ||
| [[back to top](#databricks-labs-lsql)] | ||
|
|
||
| This includes a lightweight framework to map between dataclass instances and different SQL execution backends: | ||
| - `MockBackend` used for unit testing | ||
| - `RuntimeBackend` used for execution within Databricks Runtime | ||
| - `StatementExecutionBackend` used for reading/writing records purely through REST API | ||
| ## Executing without iterating | ||
|
|
||
| ## Pick the library that you need | ||
| When you only need to execute the query and have no need to iterate over results, use the `execute` method, | ||
| which accepts [common parameters](#parameters). | ||
|
|
||
| _Simple applications_, like AWS Lambdas or Azure Functions, and scripts, that are **constrained by the size of external | ||
| dependencies** or **cannot depend on compiled libraries**, like `pyarrow` (88M), `pandas` (71M), `numpy` (60M), | ||
| `libarrow` (41M), `cygrpc` (30M), `libopenblas64` (22M), **need less than 5M of dependencies** (see [detailed report](docs/comparison.md)), | ||
| experience the [Unified Authentication](https://docs.databricks.com/en/dev-tools/auth.html#databricks-client-unified-authentication), | ||
| and **work only with Databricks SQL Warehouses**, should use this library. | ||
| ```python | ||
| from databricks.sdk import WorkspaceClient | ||
| from databricks.labs.lsql.core import StatementExecutionExt | ||
|
|
||
| Applications, that need the full power of Databricks Runtime locally with the full velocity of PySpark SDL, experience | ||
| the [Unified Authentication](https://docs.databricks.com/en/dev-tools/auth.html#databricks-client-unified-authentication) | ||
| across all Databricks tools, efficient data transfer serialized in Apache Arrow format, and low result fetching latency, | ||
| should use the stateful [Databricks Connect 2.x](https://docs.databricks.com/en/dev-tools/databricks-connect/index.html). | ||
| w = WorkspaceClient() | ||
| see = StatementExecutionExt(w) | ||
| see.execute("CREATE TABLE foo AS SELECT * FROM range(10)") | ||
| ``` | ||
|
|
||
| Applications, that need to a more traditional SQL Python APIs with cursors, efficient data transfer of hundreds of | ||
| megabytes or gigabytes of data serialized in Apache Arrow format, and low result fetching latency, should use | ||
| the stateful [Databricks SQL Connector for Python](https://docs.databricks.com/en/dev-tools/python-sql-connector.html). | ||
|
|
||
| | ... | Databricks Connect 2.x | Databricks SQL Connector | PyODBC + ODBC Driver | Databricks Labs LightSQL | | ||
| |-----------------------------------------|----------------------------------------|---------------------------------------------------|---------------------------------------------------|------------------------------------| | ||
| | Light-weight mocking | no | no | no | **yes** | | ||
| | Extended support for dataclasses | limited | no | no | **yes** | | ||
| | Strengths | almost Databricks Runtime, but locally | works with Python ecosystem | works with ODBC ecosystem | **tiny** | | ||
| | Compressed size | 60M | 51M (85%) | 44M (73.3%) | **0.8M (1.3%)** | | ||
| | Uncompressed size | 312M | 280M (89.7%) | ? | **30M (9.6%)** | | ||
| | Direct dependencies | 23 | 14 | 2 | **1** (Python SDK) | | ||
| | Unified Authentication | yes (via Python SDK) | no | no | **yes** (via Python SDK) | | ||
| | Works with | Databricks Clusters only | Databricks Clusters and Databricks SQL Warehouses | Databricks Clusters and Databricks SQL Warehouses | **Databricks SQL Warehouses only** | | ||
| | Full equivalent of Databricks Runtime | yes | no | no | **no** | | ||
| | Efficient memory usage via Apache Arrow | yes | yes | yes | **no** | | ||
| | Connection handling | stateful | stateful | stateful | **stateless** | | ||
| | Official | yes | yes | yes | **no** | | ||
| | Version checked | 14.0.1 | 2.9.3 | driver v2.7.5 | 0.1.0 | | ||
|
|
||
| ## Project Support | ||
| [[back to top](#databricks-labs-lsql)] | ||
|
|
||
| ## Fetching one record | ||
|
|
||
| Method `fetch_one` returns a single record from the result set. If the result set is empty, it returns `None`. | ||
| If the result set contains more than one record, it raises `ValueError`. | ||
|
|
||
| ```python | ||
| from databricks.sdk import WorkspaceClient | ||
| from databricks.labs.lsql.core import StatementExecutionExt | ||
|
|
||
| w = WorkspaceClient() | ||
| see = StatementExecutionExt(w) | ||
| pickup_zip, dropoff_zip = see.fetch_one("SELECT pickup_zip, dropoff_zip FROM samples.nyctaxi.trips LIMIT 1") | ||
| print(f'pickup_zip={pickup_zip}, dropoff_zip={dropoff_zip}') | ||
| ``` | ||
|
|
||
| [[back to top](#databricks-labs-lsql)] | ||
|
|
||
| ## Fetching one value | ||
|
|
||
| Method `fetch_value` returns a single value from the result set. If the result set is empty, it returns `None`. | ||
|
|
||
| ```python | ||
| from databricks.sdk import WorkspaceClient | ||
| from databricks.labs.lsql.core import StatementExecutionExt | ||
|
|
||
| w = WorkspaceClient() | ||
| see = StatementExecutionExt(w) | ||
| count = see.fetch_value("SELECT COUNT(*) FROM samples.nyctaxi.trips") | ||
| print(f'count={count}') | ||
| ``` | ||
|
|
||
| [[back to top](#databricks-labs-lsql)] | ||
|
|
||
| ## Parameters | ||
|
|
||
| * `warehouse_id` (str, optional) - Warehouse upon which to execute a statement. If not given, it will use the warehouse specified in the constructor or the first available warehouse that is not in the `DELETED` or `DELETING` state. | ||
| * `byte_limit` (int, optional) - Applies the given byte limit to the statement's result size. Byte counts are based on internal representations and may not match measurable sizes in the JSON format. | ||
| * `catalog` (str, optional) - Sets default catalog for statement execution, similar to `USE CATALOG` in SQL. If not given, it will use the default catalog or the catalog specified in the constructor. | ||
| * `schema` (str, optional) - Sets default schema for statement execution, similar to `USE SCHEMA` in SQL. If not given, it will use the default schema or the schema specified in the constructor. | ||
| * `timeout` (timedelta, optional) - Timeout after which the query is cancelled. If timeout is less than 50 seconds, it is handled on the server side. If the timeout is greater than 50 seconds, Databricks SDK for Python cancels the statement execution and throws `TimeoutError`. If not given, it will use the timeout specified in the constructor. | ||
|
|
||
| [[back to top](#databricks-labs-lsql)] | ||
|
|
||
| # SQL backend abstraction | ||
|
|
||
| This framework allows for mapping with strongly-typed dataclasses between SQL and Python runtime. It handles the schema | ||
| creation logic purely from Python datastructure. | ||
|
|
||
| `SqlBackend` is used to define the methods that are required to be implemented by any SQL backend | ||
| that is used by the library. The methods defined in this class are used to execute SQL statements, | ||
| fetch results from SQL statements, and save data to tables. Available backends are: | ||
|
|
||
| - `StatementExecutionBackend` used for reading/writing records purely through REST API | ||
| - `DatabricksConnectBackend` used for reading/writing records through Databricks Connect | ||
| - `RuntimeBackend` used for execution within Databricks Runtime | ||
| - `MockBackend` used for unit testing | ||
|
|
||
| Common methods are: | ||
| - `execute(str)` - Execute a SQL statement and wait till it finishes | ||
| - `fetch(str)` - Execute a SQL statement and iterate over all results | ||
| - `save_table(full_name: str, rows: Sequence[DataclassInstance], klass: Dataclass)` - Save a sequence of dataclass instances to a table | ||
|
|
||
| [[back to top](#databricks-labs-lsql)] | ||
|
|
||
| # Project Support | ||
| Please note that all projects in the /databrickslabs github account are provided for your exploration only, and are not formally supported by Databricks with Service Level Agreements (SLAs). They are provided AS-IS and we do not make any guarantees of any kind. Please do not submit a support ticket relating to any issues arising from the use of these projects. | ||
|
|
||
| Any issues discovered through the use of this project should be filed as GitHub Issues on the Repo. They will be reviewed as time permits, but there are no formal SLAs for support. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.