try out Sentry (distributed) profiling

[rahulbot]

Sentry seems to offer a suite of off-the-shelf profiling tools, as noted by @philbudne on #277. I'm creating this issue to track experiments and related results we see in trying these out across the system.

[pgulley]

I'm not completely sure whether or not sentry will auto-baggage requests coming out so many modules deep- since the requests to the news-search-api go through providers and then through news-client. Some documentation refers to writing custom middleware to shim the baggage onto outgoing requests. Some other documentation suggests that the auto-bagging only works on python2, in which case there might be yet more involved surgery.

I'll get a dev version of web-search running and futz around with the options here.

[philbudne]

sentry may sink hooks deep enough so that it can bury its garbage... er baggage without intervention.... getsentry/sentry-python#2190 (2023) has this:

Currently, any library that is patched by Sentry tries to append the baggage header whenever possible, e.g, httpx. We currently send our own values within the header. However, when bumping to 1.26.0, we noticed that our headers are being overwritten when using httpx. By debugging some of our calls, I think it has something to do with the way the baggage is calculated here, which seems to only take into account the Sentry DSC, and then it is directly written here in the headers instead of merged with the original value. My feeling is that this is assuming the baggage is injected by OpenTelemetry only (which potentially happens afterwards, but maybe not always?) and won't come as a raw value in the headers.

The fix PR was getsentry/sentry-python#2191 which shows an example of a place that adds baggage, and may help us find other places that do the same.... As for py2 vs py3 for auto baggaging, that's interesting.... Hopefully out of date!

[pgulley]

This is implemented in my dev stack and works without headache. I'll make another issue in web-search and news-search-api to track deployment in those respective services.