Resolve celery drift issue

[robrap]

We are seeing large numbers of celery errors with "Substantial drift from celery" warnings:

• See https://app.datadoghq.com/logs?query=%22Substantial%20drift%20from%20celery%22%20&agg_m=count&agg_m_source=base&agg_q=service&agg_q_source=base&agg_t=count&cols=host%2Cservice&fromUser=true&messageDisplay=inline&refresh_mode=sliding&storage=hot&stream_sort=desc&top_n=10&top_o=top&viz=timeseries&x_missing=true&from_ts=1720728363223&to_ts=1722024363223&live=true
• The message states that this may mean clocks are out of sync.
• This github issue comment indicated that the NTP service simply wasn't active when they thought it was: What is the problem with celery: Substantial drift from celery@abc03 may mean clocks are out of sync. Current drift is 29 seconds. [orig: 2014-04-14 14:35:37.001612 recv: 2014-04-14 14:36:06.011759] celery/celery#1980 (comment)
• Some relevant Slack conversations:
  • https://twou.slack.com/archives/C06DMDFNBJQ/p1719507083273789?thread_ts=1719505025.342219&cid=C06DMDFNBJQ
  • https://twou.slack.com/archives/C04A66RDPEZ/p1721236956776789?thread_ts=1721236876.243719&cid=C04A66RDPEZ
• Note that there is also a "missed heartbeat from celery" log message that may or may not be related.

Important:

• I added a 99% exclusion rule related to the "missed heartbeat from celery" logs, and provided details in https://2u-internal.atlassian.net/browse/GSRE-1952?focusedCommentId=5146559. If GSRE-1952 fixes the drift issue, and the missed heartbeat turns out to be a separate problem, the missed heartbeat issue should be ticketed separately and not lost with this ticket. The retention rule could be switched to point to the updated ticket as well.

[github-actions]

This is now https://2u-internal.atlassian.net/browse/GSRE-1952

[jristau1984]

@robrap wants to keep this open to follow up on the GSRE ticket at some point.

[timmc-edx]

sorry-not-sorry