Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Weird behaviour after integrating Sentry SDK #3919

Open
imeckr opened this issue Jan 13, 2025 · 3 comments
Open

Weird behaviour after integrating Sentry SDK #3919

imeckr opened this issue Jan 13, 2025 · 3 comments
Labels
Feature: Profiling Integration: Starlette/FastAPI Waiting for: Community

Comments

@imeckr
Copy link

imeckr commented Jan 13, 2025

How do you use Sentry?

Sentry Saas (sentry.io)

Version

2.19.2

Steps to Reproduce

I am not sure if its a Sentry issue but I started seeing this after we integrated sentry-sdk in to our FastAPI project.

This how we initialised the SDK

sentry_sdk.init(
    dsn=settings.HYBRID_SENTRY_DSN,
    traces_sample_rate=0.005,
    profiles_sample_rate=0.05,
    _experiments={
        'continuous_profiling_auto_start': True,
    },
)
app = FastAPI()

Now the issue is that my app has started throwing errors on certain auth endpoint (Firebase id_token verification), an SSL error

  File "/app/hybrid/utils/google.py", line 36, in verify_firebase_id_token
    id_info = id_token.verify_firebase_token(token, requests.Request(), audience)
  File "/opt/pysetup/.venv/lib/python3.10/site-packages/google/oauth2/id_token.py", line 197, in verify_firebase_token
    return verify_token(
  File "/opt/pysetup/.venv/lib/python3.10/site-packages/google/oauth2/id_token.py", line 133, in verify_token
    certs = _fetch_certs(request, certs_url)
  File "/opt/pysetup/.venv/lib/python3.10/site-packages/google/oauth2/id_token.py", line 99, in _fetch_certs
    response = request(certs_url, method="GET")
  File "/opt/pysetup/.venv/lib/python3.10/site-packages/google/auth/transport/requests.py", line 193, in __call__
    response = self.session.request(
  File "/opt/pysetup/.venv/lib/python3.10/site-packages/requests/sessions.py", line 542, in request
    resp = self.send(prep, **send_kwargs)
  File "/opt/pysetup/.venv/lib/python3.10/site-packages/requests/sessions.py", line 655, in send
    r = adapter.send(request, **kwargs)
  File "/opt/pysetup/.venv/lib/python3.10/site-packages/requests/adapters.py", line 439, in send
    resp = conn.urlopen(
  File "/opt/pysetup/.venv/lib/python3.10/site-packages/urllib3/connectionpool.py", line 716, in urlopen
    httplib_response = self._make_request(
  File "/opt/pysetup/.venv/lib/python3.10/site-packages/urllib3/connectionpool.py", line 404, in _make_request
    self._validate_conn(conn)
  File "/opt/pysetup/.venv/lib/python3.10/site-packages/urllib3/connectionpool.py", line 1061, in _validate_conn
    conn.connect()
  File "/opt/pysetup/.venv/lib/python3.10/site-packages/urllib3/connection.py", line 419, in connect
    self.sock = ssl_wrap_socket(
  File "/opt/pysetup/.venv/lib/python3.10/site-packages/urllib3/util/ssl_.py", line 433, in ssl_wrap_socket
    context.set_alpn_protocols(ALPN_PROTOCOLS)
  File "/usr/local/lib/python3.10/ssl.py", line 566, in set_alpn_protocols
    if len(b) == 0 or len(b) > 255:
TypeError: 'int' object is not callable

The error suggests that somehow len which is a python keyword has been reassigned to an integer, which is very unexpected and means somehow it got polluted. I checked our project there is no such re-assignment. Curiously I searched for len= in the sentry-sdk repo, and found this.

sentry-python/sentry_sdk/serializer.py

Line 314 in 9f9ff34

_annotate(len=len(obj)) 

            for k, v in obj.items():
                if remaining_breadth is not None and i >= remaining_breadth:
                    _annotate(len=len(obj))
                    break

This seems to be some internal function which would take sometime for me to understand. But can someone just double check this and tell if I am looking at the direction.

Expected Result

Should not get the above error.

Actual Result


Traceback (most recent call last):
  File "/opt/pysetup/.venv/lib/python3.10/site-packages/uvicorn/protocols/http/httptools_impl.py", line 375, in run_asgi
    result = await app(self.scope, self.receive, self.send)
  File "/opt/pysetup/.venv/lib/python3.10/site-packages/uvicorn/middleware/proxy_headers.py", line 75, in __call__
    return await self.app(scope, receive, send)
  File "/opt/pysetup/.venv/lib/python3.10/site-packages/fastapi/applications.py", line 276, in __call__
    await super().__call__(scope, receive, send)
  File "/opt/pysetup/.venv/lib/python3.10/site-packages/sentry_sdk/integrations/starlette.py", line 409, in _sentry_patched_asgi_app
    return await middleware(scope, receive, send)
  File "/opt/pysetup/.venv/lib/python3.10/site-packages/sentry_sdk/integrations/asgi.py", line 161, in _run_asgi3
    return await self._run_app(scope, receive, send, asgi_version=3)
  File "/opt/pysetup/.venv/lib/python3.10/site-packages/sentry_sdk/integrations/asgi.py", line 262, in _run_app
    raise exc from None
  File "/opt/pysetup/.venv/lib/python3.10/site-packages/sentry_sdk/integrations/asgi.py", line 257, in _run_app
    return await self.app(
  File "/opt/pysetup/.venv/lib/python3.10/site-packages/starlette/applications.py", line 122, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/opt/pysetup/.venv/lib/python3.10/site-packages/sentry_sdk/integrations/starlette.py", line 200, in _create_span_call
    return await old_call(app, scope, new_receive, new_send, **kwargs)
  File "/opt/pysetup/.venv/lib/python3.10/site-packages/starlette/middleware/errors.py", line 184, in __call__
    raise exc
  File "/opt/pysetup/.venv/lib/python3.10/site-packages/starlette/middleware/errors.py", line 162, in __call__
    await self.app(scope, receive, _send)
  File "/opt/pysetup/.venv/lib/python3.10/site-packages/sentry_sdk/integrations/starlette.py", line 200, in _create_span_call
    return await old_call(app, scope, new_receive, new_send, **kwargs)
  File "/opt/pysetup/.venv/lib/python3.10/site-packages/starlette/middleware/base.py", line 109, in __call__
    await response(scope, receive, send)
  File "/opt/pysetup/.venv/lib/python3.10/site-packages/starlette/responses.py", line 270, in __call__
    async with anyio.create_task_group() as task_group:
  File "/opt/pysetup/.venv/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 662, in __aexit__
    raise exceptions[0]
  File "/opt/pysetup/.venv/lib/python3.10/site-packages/starlette/responses.py", line 273, in wrap
    await func()
  File "/opt/pysetup/.venv/lib/python3.10/site-packages/starlette/middleware/base.py", line 134, in stream_response
    return await super().stream_response(send)
  File "/opt/pysetup/.venv/lib/python3.10/site-packages/starlette/responses.py", line 262, in stream_response
    async for chunk in self.body_iterator:
  File "/opt/pysetup/.venv/lib/python3.10/site-packages/starlette/middleware/base.py", line 98, in body_stream
    raise app_exc
  File "/opt/pysetup/.venv/lib/python3.10/site-packages/starlette/middleware/base.py", line 70, in coro
    await self.app(scope, receive_or_disconnect, send_no_error)
  File "/opt/pysetup/.venv/lib/python3.10/site-packages/sentry_sdk/integrations/starlette.py", line 200, in _create_span_call
    return await old_call(app, scope, new_receive, new_send, **kwargs)
  File "/opt/pysetup/.venv/lib/python3.10/site-packages/starlette/middleware/cors.py", line 84, in __call__
    await self.app(scope, receive, send)
  File "/opt/pysetup/.venv/lib/python3.10/site-packages/sentry_sdk/integrations/starlette.py", line 298, in _sentry_exceptionmiddleware_call
    await old_call(self, scope, receive, send)
  File "/opt/pysetup/.venv/lib/python3.10/site-packages/sentry_sdk/integrations/starlette.py", line 200, in _create_span_call
    return await old_call(app, scope, new_receive, new_send, **kwargs)
  File "/opt/pysetup/.venv/lib/python3.10/site-packages/starlette/middleware/exceptions.py", line 79, in __call__
    raise exc
  File "/opt/pysetup/.venv/lib/python3.10/site-packages/starlette/middleware/exceptions.py", line 68, in __call__
    await self.app(scope, receive, sender)
  File "/opt/pysetup/.venv/lib/python3.10/site-packages/sentry_sdk/integrations/starlette.py", line 200, in _create_span_call
    return await old_call(app, scope, new_receive, new_send, **kwargs)
  File "/opt/pysetup/.venv/lib/python3.10/site-packages/fastapi/middleware/asyncexitstack.py", line 21, in __call__
    raise e
  File "/opt/pysetup/.venv/lib/python3.10/site-packages/fastapi/middleware/asyncexitstack.py", line 18, in __call__
    await self.app(scope, receive, send)
  File "/opt/pysetup/.venv/lib/python3.10/site-packages/starlette/routing.py", line 718, in __call__
    await route.handle(scope, receive, send)
  File "/opt/pysetup/.venv/lib/python3.10/site-packages/starlette/routing.py", line 443, in handle
    await self.app(scope, receive, send)
  File "/opt/pysetup/.venv/lib/python3.10/site-packages/fastapi/applications.py", line 276, in __call__
    await super().__call__(scope, receive, send)
  File "/opt/pysetup/.venv/lib/python3.10/site-packages/sentry_sdk/integrations/starlette.py", line 409, in _sentry_patched_asgi_app
    return await middleware(scope, receive, send)
  File "/opt/pysetup/.venv/lib/python3.10/site-packages/sentry_sdk/integrations/asgi.py", line 161, in _run_asgi3
    return await self._run_app(scope, receive, send, asgi_version=3)
  File "/opt/pysetup/.venv/lib/python3.10/site-packages/sentry_sdk/integrations/asgi.py", line 176, in _run_app
    raise exc from None
  File "/opt/pysetup/.venv/lib/python3.10/site-packages/sentry_sdk/integrations/asgi.py", line 172, in _run_app
    return await self.app(scope, receive, send)
  File "/opt/pysetup/.venv/lib/python3.10/site-packages/starlette/applications.py", line 122, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/opt/pysetup/.venv/lib/python3.10/site-packages/sentry_sdk/integrations/starlette.py", line 200, in _create_span_call
    return await old_call(app, scope, new_receive, new_send, **kwargs)
  File "/opt/pysetup/.venv/lib/python3.10/site-packages/starlette/middleware/errors.py", line 184, in __call__
    raise exc
  File "/opt/pysetup/.venv/lib/python3.10/site-packages/starlette/middleware/errors.py", line 162, in __call__
    await self.app(scope, receive, _send)
  File "/opt/pysetup/.venv/lib/python3.10/site-packages/sentry_sdk/integrations/starlette.py", line 298, in _sentry_exceptionmiddleware_call
    await old_call(self, scope, receive, send)
  File "/opt/pysetup/.venv/lib/python3.10/site-packages/sentry_sdk/integrations/starlette.py", line 200, in _create_span_call
    return await old_call(app, scope, new_receive, new_send, **kwargs)
  File "/opt/pysetup/.venv/lib/python3.10/site-packages/starlette/middleware/exceptions.py", line 79, in __call__
    raise exc
  File "/opt/pysetup/.venv/lib/python3.10/site-packages/starlette/middleware/exceptions.py", line 68, in __call__
    await self.app(scope, receive, sender)
  File "/opt/pysetup/.venv/lib/python3.10/site-packages/sentry_sdk/integrations/starlette.py", line 200, in _create_span_call
    return await old_call(app, scope, new_receive, new_send, **kwargs)
  File "/opt/pysetup/.venv/lib/python3.10/site-packages/fastapi/middleware/asyncexitstack.py", line 21, in __call__
    raise e
  File "/opt/pysetup/.venv/lib/python3.10/site-packages/fastapi/middleware/asyncexitstack.py", line 18, in __call__
    await self.app(scope, receive, send)
  File "/opt/pysetup/.venv/lib/python3.10/site-packages/starlette/routing.py", line 718, in __call__
    await route.handle(scope, receive, send)
  File "/opt/pysetup/.venv/lib/python3.10/site-packages/starlette/routing.py", line 276, in handle
    await self.app(scope, receive, send)
  File "/opt/pysetup/.venv/lib/python3.10/site-packages/starlette/routing.py", line 66, in app
    response = await func(request)
  File "/opt/pysetup/.venv/lib/python3.10/site-packages/fastapi/routing.py", line 227, in app
    solved_result = await solve_dependencies(
  File "/opt/pysetup/.venv/lib/python3.10/site-packages/fastapi/dependencies/utils.py", line 592, in solve_dependencies
    solved_result = await solve_dependencies(
  File "/opt/pysetup/.venv/lib/python3.10/site-packages/fastapi/dependencies/utils.py", line 621, in solve_dependencies
    solved = await call(**sub_values)
  File "/app/hybrid/crud/user.py", line 78, in get_firebase_decoded_token
    return FirebaseDecodedToken.parse_obj(verify_firebase_id_token(token, settings.APP_FIREBASE_PROJECT_ID))
  File "/app/hybrid/utils/google.py", line 36, in verify_firebase_id_token
    id_info = id_token.verify_firebase_token(token, requests.Request(), audience)
  File "/opt/pysetup/.venv/lib/python3.10/site-packages/google/oauth2/id_token.py", line 197, in verify_firebase_token
    return verify_token(
  File "/opt/pysetup/.venv/lib/python3.10/site-packages/google/oauth2/id_token.py", line 133, in verify_token
    certs = _fetch_certs(request, certs_url)
  File "/opt/pysetup/.venv/lib/python3.10/site-packages/google/oauth2/id_token.py", line 99, in _fetch_certs
    response = request(certs_url, method="GET")
  File "/opt/pysetup/.venv/lib/python3.10/site-packages/google/auth/transport/requests.py", line 193, in __call__
    response = self.session.request(
  File "/opt/pysetup/.venv/lib/python3.10/site-packages/requests/sessions.py", line 542, in request
    resp = self.send(prep, **send_kwargs)
  File "/opt/pysetup/.venv/lib/python3.10/site-packages/requests/sessions.py", line 655, in send
    r = adapter.send(request, **kwargs)
  File "/opt/pysetup/.venv/lib/python3.10/site-packages/requests/adapters.py", line 439, in send
    resp = conn.urlopen(
  File "/opt/pysetup/.venv/lib/python3.10/site-packages/urllib3/connectionpool.py", line 716, in urlopen
    httplib_response = self._make_request(
  File "/opt/pysetup/.venv/lib/python3.10/site-packages/urllib3/connectionpool.py", line 404, in _make_request
    self._validate_conn(conn)
  File "/opt/pysetup/.venv/lib/python3.10/site-packages/urllib3/connectionpool.py", line 1061, in _validate_conn
    conn.connect()
  File "/opt/pysetup/.venv/lib/python3.10/site-packages/urllib3/connection.py", line 419, in connect
    self.sock = ssl_wrap_socket(
  File "/opt/pysetup/.venv/lib/python3.10/site-packages/urllib3/util/ssl_.py", line 433, in ssl_wrap_socket
    context.set_alpn_protocols(ALPN_PROTOCOLS)
  File "/usr/local/lib/python3.10/ssl.py", line 566, in set_alpn_protocols
    if len(b) == 0 or len(b) > 255:
TypeError: 'int' object is not callable
@getsantry getsantry bot moved this to Waiting for: Product Owner in GitHub Issues with 👀 3 Jan 13, 2025
@antonpirker
Copy link
Member

Hey @imeckr thanks for reporting this!

Do you have a minimal reproduction of this?
Was Sentry the only thing that you changed in your project when the problem started?

I found an issue that describes the same problem when using datadog: DataDog/dd-trace-py#6041

@getsantry getsantry bot moved this to Waiting for: Community in GitHub Issues with 👀 3 Jan 14, 2025
@imeckr
Copy link
Author

imeckr commented Jan 15, 2025

@antonpirker Thanks for sharing the relevant issue, it does seem like its caused by some quirk in the Python's ssl package. But I am not sure how to fix that. To answer your questions:

  • I am unable to reproduce it locally, even on the FastAPI server, it doesn't happen in a certain pattern.
  • Yes, Sentry was the only change.

I am going to remove Sentry for a couple of days and see if the error persists, will report back after that.

@getsantry getsantry bot moved this from Waiting for: Community to Waiting for: Product Owner in GitHub Issues with 👀 3 Jan 15, 2025
@sl0thentr0py
Copy link
Member

@imeckr I see you are using profiling, if possible could you also check without profiling (profiles_sample_rate=0.0)?

@getsantry getsantry bot moved this to Waiting for: Community in GitHub Issues with 👀 3 Jan 15, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Feature: Profiling Integration: Starlette/FastAPI Waiting for: Community
Projects
GitHub Issues with 👀 3
Status: Waiting for: Community
Milestone
No milestone
Development

No branches or pull requests
3 participants
@antonpirker @sl0thentr0py @imeckr