Skip to content

Commit

Permalink
Add a rate limit to the send_instructor_email_digests() task
Browse files Browse the repository at this point in the history
Fixes #5509.

The `send_instructor_email_digests()` task has been [overloading h's DB](https://docs.google.com/document/d/1AimkRirwYirz-LWVyyDIRGKB4-HsYOKO0KoOFH_dhg4/).
Add a rate limit to the task so that it doesn't put so much load on h.

See https://docs.celeryq.dev/en/stable/userguide/tasks.html#Task.rate_limit.

Note that Celery task rate limits are per-worker so if the rate limit
"1/m" (one per minute) and there are two workers then the system will
actually process up to two tasks per minute.

I think there are usually four LMS instances and therefore four Celery
workers at the time when the `send_instructor_email_digests()` task is
run.

It looks like we currently run this task about 262 times each morning
(and this number is slowly increasing): https://hypothes-is.slack.com/archives/C074BUPEG/p1687350944775379?thread_ts=1687325831.540209&cid=C074BUPEG

At a rate of four tasks per minute it should take just over an hour to
process 262 tasks. I think we want to get all these tasks done in
something like 3-5 hrs (https://hypothes-is.slack.com/archives/C074BUPEG/p1687340914305749?thread_ts=1687325831.540209&cid=C074BUPEG)
so four tasks per minute (corresponding to a worker rate limit of
`"1/m"` given that we have four workers) should be plenty fast enough.
This is much lower than the rate of requests that the bulk annotation
API has been receiving so far (it gets up to >50 requests per minute) so
this should reduce the load on h significantly while still generating
all the emails in plenty of time.
  • Loading branch information
seanh committed Jun 21, 2023
1 parent a993f2b commit bc0b6ff
Showing 1 changed file with 1 addition and 0 deletions.
1 change: 1 addition & 0 deletions lms/tasks/email_digests.py
Original file line number Diff line number Diff line change
Expand Up @@ -94,6 +94,7 @@ def send_instructor_email_digest_tasks(*, batch_size):
max_retries=2,
retry_backoff=3600,
retry_backoff_max=7200,
rate_limit="1/m",
)
def send_instructor_email_digests(
*, h_userids: List[str], updated_after: str, updated_before: str, **kwargs
Expand Down

0 comments on commit bc0b6ff

Please sign in to comment.