Skip to content

Telemetry circuit breaker #658

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 6 commits into
base: main
Choose a base branch
from
Open

Telemetry circuit breaker #658

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
a5f46eb
telemetry circuit breaker
saishreeeee Jul 30, 2025
1b713bd
Revert "telemetry circuit breaker"
saishreeeee Jul 30, 2025
ec0f5a7
pybreaker
saishreeeee Jul 30, 2025
9a87af3
dependency
saishreeeee Jul 30, 2025
aebb46a
test
saishreeeee Aug 1, 2025
bec62fd
logging
saishreeeee Aug 1, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ You are welcome to file an issue here for general use cases. You can also contac

## Requirements

Python 3.8 or above is required.
Python 3.9 or above is required.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jprakash-db are we ok with python version upgrades?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should be ok, given DBT tests pass.


## Documentation

Expand Down
152 changes: 14 additions & 138 deletions poetry.lock
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

7 changes: 4 additions & 3 deletions pyproject.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ packages = [{ include = "databricks", from = "src" }]
include = ["CHANGELOG.md"]

[tool.poetry.dependencies]
python = "^3.8.0"
python = "^3.9.0"
thrift = ">=0.16.0,<0.21.0"
pandas = [
{ version = ">=1.2.5,<2.3.0", python = ">=3.8,<3.13" },
Expand All @@ -22,10 +22,11 @@ openpyxl = "^3.0.10"
urllib3 = ">=1.26"
python-dateutil = "^2.8.0"
pyarrow = [
{ version = ">=14.0.1", python = ">=3.8,<3.13", optional=true },
{ version = ">=14.0.1", python = ">=3.9,<3.13", optional=true },
{ version = ">=18.0.0", python = ">=3.13", optional=true }
]
pyjwt = "^2.0.0"
pybreaker = "^1.4.0"


[tool.poetry.extras]
Expand All @@ -38,7 +39,7 @@ pylint = ">=2.12.0"
black = "^22.3.0"
pytest-dotenv = "^0.5.2"
numpy = [
{ version = ">=1.16.6", python = ">=3.8,<3.11" },
{ version = ">=1.16.6", python = ">=3.9,<3.11" },
{ version = ">=1.23.4", python = ">=3.11" },
]

Expand Down
10 changes: 9 additions & 1 deletion src/databricks/sql/common/http.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@
import logging
from requests.adapters import HTTPAdapter
from databricks.sql.auth.retry import DatabricksRetryPolicy, CommandType
from pybreaker import CircuitBreaker, CircuitBreakerError

logger = logging.getLogger(__name__)

Expand Down Expand Up @@ -109,6 +110,9 @@ class TelemetryHttpClient: # TODO: Unify all the http clients in the PySQL Conn
TELEMETRY_RETRY_DELAY_MAX = 10.0
TELEMETRY_RETRY_STOP_AFTER_ATTEMPTS_DURATION = 30.0

CIRCUIT_BREAKER_FAIL_MAX = 5
CIRCUIT_BREAKER_RESET_TIMEOUT = 60

def __init__(self):
"""Initializes the session and mounts the custom retry adapter."""
retry_policy = DatabricksRetryPolicy(
Expand All @@ -123,6 +127,10 @@ def __init__(self):
self.session = requests.Session()
self.session.mount("https://", adapter)
self.session.mount("http://", adapter)
self.circuit_breaker = CircuitBreaker(
fail_max=self.CIRCUIT_BREAKER_FAIL_MAX,
reset_timeout=self.CIRCUIT_BREAKER_RESET_TIMEOUT,
)

@classmethod
def get_instance(cls) -> "TelemetryHttpClient":
Expand All @@ -141,7 +149,7 @@ def post(self, url: str, **kwargs) -> requests.Response:
This is a blocking call intended to be run in a background thread.
"""
logger.debug("Executing telemetry POST request to: %s", url)
return self.session.post(url, **kwargs)
return self.circuit_breaker.call(self.session.post, url, **kwargs)

def close(self):
"""Closes the underlying requests.Session."""
Expand Down
Loading
Loading