Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat!: Detect anomalous Datadog traces; upgrade to Python 3.11 #757

Merged
merged 2 commits into from
Aug 5, 2024

Conversation

timmc-edx
Copy link
Member

  • Detect and log anomalous traces in DatadogDiagnosticMiddleware.
  • Drop Python 3.8 support and change to Python 3.11 minimum due to testing needs.
  • Add tests for existing middleware code

Manual testing:

  1. Install in devstack, add middleware to lms/envs/devstack.py, and
    enable Waffle flag.
  2. Make some requests and observe lack of new logging.
  3. Add self.dd_tracer.current_root_span().finish() to start of
    process_view (faking an anomalous trace) and reinstall in devstack
  4. Make some more requests and observe log output like the following:
    Anomalous Datadog local root span (duration already set): id = 66abf66b000000001168265a391d04a3; duration = 0.108 sec; worker age = 1.560 sec

Unit test improvements:


Merge checklist:
Check off if complete or not applicable:

  • Version bumped
  • Changelog record added
  • Documentation updated (not only docstrings)
  • Fixup commits are squashed away
  • Unit tests added/updated
  • Manual testing instructions provided
  • Noted any: Concerns, dependencies, migration issues, deadlines, tickets

Manual testing:

1. Install in devstack, add middleware to lms/envs/devstack.py, and
   enable Waffle flag.
2. Make some requests and observe lack of new logging.
3. Add `self.dd_tracer.current_root_span().finish()` to start of
   `process_view` (faking an anomalous trace) and reinstall in devstack
4. Make some more requests and observe log output like the following:
   `Anomalous Datadog local root span (duration already set): id = 66abf66b000000001168265a391d04a3; duration = 0.108 sec; worker age = 1.560 sec`
- Install ddtrace at the test.in level
- Add tests for new and existing middleware functionality
- Test with Python 3.11, not 3.8. This isn't just because we're upgrading
  to 3.11 across our repos, but also because I ran into a really weird bug
  with ddtrace and/or jsonschema that only occurs with Python 3.8:
  DataDog/dd-trace-py#9950
@timmc-edx timmc-edx merged commit 36bc403 into main Aug 5, 2024
6 of 7 checks passed
@timmc-edx timmc-edx deleted the timmc/detect-anomaly branch August 5, 2024 17:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants