Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Add a rate limit to the send_instructor_email_digests() task
Fixes #5509. The `send_instructor_email_digests()` task has been [overloading h's DB](https://docs.google.com/document/d/1AimkRirwYirz-LWVyyDIRGKB4-HsYOKO0KoOFH_dhg4/). Add a rate limit to the task so that it doesn't put so much load on h. See https://docs.celeryq.dev/en/stable/userguide/tasks.html#Task.rate_limit. Note that Celery task rate limits are per-worker so if the rate limit "1/m" (one per minute) and there are two workers then the system will actually process up to two tasks per minute. I think there are usually four LMS instances and therefore four Celery workers at the time when the `send_instructor_email_digests()` task is run. It looks like we currently run this task about 262 times each morning (and this number is slowly increasing): https://hypothes-is.slack.com/archives/C074BUPEG/p1687350944775379?thread_ts=1687325831.540209&cid=C074BUPEG At a rate of four tasks per minute it should take just over an hour to process 262 tasks. I think we want to get all these tasks done in something like 3-5 hrs (https://hypothes-is.slack.com/archives/C074BUPEG/p1687340914305749?thread_ts=1687325831.540209&cid=C074BUPEG) so four tasks per minute (corresponding to a worker rate limit of `"1/m"` given that we have four workers) should be plenty fast enough. This is much lower than the rate of requests that the bulk annotation API has been receiving so far (it gets up to >50 requests per minute) so this should reduce the load on h significantly while still generating all the emails in plenty of time.
- Loading branch information