-
Notifications
You must be signed in to change notification settings - Fork 36
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[PERFORMANCE] Optimize container based proficiency calculation [MER-2447] #4095
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM!
Loved the async implementation 😄
I tested it locally adding a :timer.sleep
in the async function to see the behavior when receiving the message after navigating to another table page and worked fine!
I left some minor NITs that may improve a little bit the performance by reducing the amounts of nested Enum
Co-authored-by: Nicolás Cirio <[email protected]>
Co-authored-by: Nicolás Cirio <[email protected]>
…li-torus into optimized-proficiency
@darrensiegel It seems liveview is about to support async assigns natively, so we should keep an eye on this PR |
The original reported issue (https://eliterate.atlassian.net/browse/MER-2447) noted course sections on Argos Prod that would not load the Instructor Dashboard "Reports" view, with the UI eventually getting a "Gateway Timeout" error.
Part 1: Query Optimization and Post Processing
I pinned down the problem to be the query used to calculate student proficiency, per container, from
Metrics.proficiency_per_container/1
. The raw SQL for that query originally was:This query runs just fine on Torus Prod with its 200K snapshot records. On Argos Prod there are 109 million snapshot records and this query doesn't finish even after letting it go for close to 5 minutes from within pgAdmin.
To address the problem I've reworked the overall approach here to no longer attempt to do the container "bucketing" within the query, and instead to do that in code as a post-processing step after the query finishes. I also removed the JOIN to
sections
that was necessary because the query was slug based. So the new query now looks like:The above query, on Argos Prod, runs anywhere between 2 seconds to about 25 seconds (due to cache warmth, I believe). The 25 seconds upper bound is concerning as we could still face the issue of this view still timing out if this grows. There would also be a negative UX by forcing the user to wait this long before the view would display. So, on to:
Part 2: Async Processing
This PR also changes the Instructor and Student Details dashboards LiveViews to invoke the potentially expensive proficiency calculation as an async, background task. When that task finishes it will notify the LV and the assigns are updated. The UX now will be a quick loading view, with some "Loading..." placeholder text in place for the proficiency values until those calculations finish.