Make telemetry batch size configurable and add time-based flush #622
Conversation
Signed-off-by: Jesse Whitehouse <[email protected]>
Advise developers to use Python 3.7, 3.8, or 3.9 until #26 is fixed.
Includes a GitHub Action which checks for a valid sign-off on every proposed commit
* Isolate delay bounding logic * Move error details scope up one-level. * Retry GetOperationStatus if an OSError was raised during execution. Add retry_delay_default to use in this case. * Log when a request is retried due to an OSError. Emit warnings for unexpected OSError codes * Update docstring for make_request * Nit: unit tests show the .warn message is deprecated. DeprecationWarning: The 'warn' function is deprecated, use 'warning' instead Signed-off-by: Jesse Whitehouse <[email protected]>
Signed-off-by: Jesse Whitehouse <[email protected]>
Signed-off-by: Jesse Whitehouse <[email protected]>
* Test with multiple python versions. * Update pyarrow to version 9.0.0 to address issue in relation to python 3.10 & a specific version of numpy being pulled in by pyarrow. Closes #26 Signed-off-by: David Black <[email protected]>
* Update changelog and bump to v2.0.4 * Specifically thank @dbaxa for this change. Signed-off-by: Jesse Whitehouse <[email protected]>
Signed-off-by: Jesse Whitehouse <[email protected]>
Signed-off-by: Jesse Whitehouse <[email protected]>
* Add test: cursors are closed when connection closes Signed-off-by: Jesse Whitehouse <[email protected]>
Signed-off-by: Jesse Whitehouse <[email protected]>
Signed-off-by: Moe Derakhshani <[email protected]>
Signed-off-by: Moe Derakhshani <[email protected]>
Signed-off-by: Moe Derakhshani <[email protected]> my [OAuth PR](https://github.com/databricks/databricks-sql-python/runs/8005844758?check_suite_focus=true) is blocked due to dco validation (following error): <img width="1202" alt="Screen Shot 2022-08-25 at 12 05 40 PM" src="https://user-images.githubusercontent.com/22279672/186747897-c9d57586-366f-41f9-aa66-609f2bf3911f.png"> We should try to avoid running dco for internal databricks employees: I am trying to relax the validation based on this guideline: https://github.com/dcoapp/app/blob/main/README.md#skipping-sign-off-for-organization-members and here: https://stackoverflow.com/questions/62969381/is-it-in-line-with-the-dco-that-a-github-sign-off-needs-and-publishes-full-name
Signed-off-by: Moe Derakhshani <[email protected]>
Signed-off-by: Moe Derakhshani <[email protected]>
Signed-off-by: Moe Derakhshani <[email protected]> this is undo of #42 till we figure out how to fix dco
Signed-off-by: Jesse Whitehouse <[email protected]>
This PR: * Adds the foundation for OAuth against Databricks account on AWS with BYOIDP. * It copies one internal module that Steve Weis @sweisdb wrote for Databricks CLI (oauth.py). Once ecosystem-dev team (Serge, Pieter) build a python sdk core we will move this code to their repo as a dependency. * the PR provides authenticators with visitor pattern format for stamping auth-token which later is intended to be moved to the repo owned by Serge @nfx and and Pieter @pietern
Signed-off-by: Jesse Whitehouse <[email protected]>
Signed-off-by: Jesse Whitehouse <[email protected]>
Bump to v2.1.0 and update changelog Signed-off-by: Jesse Whitehouse <[email protected]>
Signed-off-by: Mohit Singla <[email protected]> Co-authored-by: Moe Derakhshani <[email protected]>
* Refactor so we can unit test `inject_parameters` * Add unit tests for inject_parameters * Remove inaccurate comment. Per #51, spark sql does not support escaping a single quote with a second single quote. * Closes #51 and adds unit tests plus the integration test provided in #56 Signed-off-by: Jesse Whitehouse <[email protected]> Co-authored-by: Courtney Holcomb (@courtneyholcomb) Co-authored-by: @mcannamela
Addresses https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2020-13949 Signed-off-by: Jesse Whitehouse <[email protected]>
Signed-off-by: Jesse Whitehouse <[email protected]>
Signed-off-by: Jesse Whitehouse <[email protected]>
|
Thanks for your contribution! To satisfy the DCO policy in our contributing guide every commit message must include a sign-off message. One or more of your commits is missing this message. You can reword previous commit messages with an interactive rebase ( |
Signed-off-by: Sai Shree Pradhan <[email protected]>
|
Thanks for your contribution! To satisfy the DCO policy in our contributing guide every commit message must include a sign-off message. One or more of your commits is missing this message. You can reword previous commit messages with an interactive rebase ( |
saishreeeee
removed request for
jackyhu-db,
gopalldb,
madhav-db,
samikshya-db,
jayantsing-db,
shivam2680 and
deeksha-db
Signed-off-by: Sai Shree Pradhan <[email protected]>
Signed-off-by: Sai Shree Pradhan <[email protected]>
|
Thanks for your contribution! To satisfy the DCO policy in our contributing guide every commit message must include a sign-off message. One or more of your commits is missing this message. You can reword previous commit messages with an interactive rebase ( |
Signed-off-by: Sai Shree Pradhan <[email protected]>
|
Thanks for your contribution! To satisfy the DCO policy in our contributing guide every commit message must include a sign-off message. One or more of your commits is missing this message. You can reword previous commit messages with an interactive rebase ( |
Signed-off-by: Sai Shree Pradhan <[email protected]>
|
Thanks for your contribution! To satisfy the DCO policy in our contributing guide every commit message must include a sign-off message. One or more of your commits is missing this message. You can reword previous commit messages with an interactive rebase ( |
| @@ -258,6 +261,7 @@ def test_factory_error_handling(self): | |||
| session_id_hex=session_id, | |||
| auth_provider=AccessTokenAuthProvider("token"), | |||
| host_url="test-host.com", | |||
| batch_size=TelemetryClientFactory.DEFAULT_BATCH_SIZE | |||
There was a problem hiding this comment.
why do you need to pass the default value ? Inside the function add the default value, pass it only when users sends a new value
|
|
||
| @patch("databricks.sql.client.Session") | ||
| def test_connection_with_custom_batch_size(MockSession): | ||
| """ |
There was a problem hiding this comment.
Don't think this test is needed. You are just setting a value and checking if it is set
| cls._initialized = True | ||
| logger.debug( | ||
| "TelemetryClientFactory initialized with thread pool (max_workers=10)" | ||
| ) | ||
|
|
||
| @classmethod | ||
| def _start_flush_thread(cls): | ||
| """Start the shared background thread for periodic flushing of all clients""" |
There was a problem hiding this comment.
Just a thought is this background thread and its overhead required and is the timely refresh feature needed ? Like anyway we send the final telemtry logs while shutdown, and we can reduce the default batch size to have a near similar effect without the tbackground thread overload cc @vikrantpuppala
What type of PR is this?
Description
The flush timer is centralized in
TelemetryClientFactory, single background thread to manage all connections. Keeping it inTelemetryClientwould mean creating a timer thread per connection.Used
threading.Threadwiththreading.Event. Thethreading.Eventacts as a thread-safe shutdown signal, and itswait(timeout)method allows the thread to wait for the next flush interval while remaining immediately responsive to a shutdown command.While
threading.Timercould be used, it would create a new thread every flush interval as we need to create a timer after each execution.How is this tested?
Related Tickets & Documents
design doc
PECOBLR-654