Learner Progress Reset #331
Comments
|
Thanks for your submission, @openedx/open-edx-project-managers will review shortly. |
|
@arbrandes here's the ticket for learner reset, thanks for your help and let me know if there is anything else to do on our end. |
|
@jmakowski1123, FYI. |
|
Thanks for submitting, @Daniel-hershel . Some questions about the intended behavior of this feature.
I have some general concerns about the data implications. Is the data from a learner's prior attempt at a course completely overridden when a course is reset? Does it behave differently from the current experience, when a learner fails a course and then decides to try again during the next course run? What implications might that have (if any) for the data pipeline? cc @bmtcril for any input or concerns here. Also just a general note, we are not yet supporting the Support Tools MFE in the community releases, and we don't have it on the list to support yet for Redwood. |
|
High level questions:
Questions/Comments on the proposal:
Adding to @jmakowski1123's questions about who can configure this: If the course team can specify it, it should probably be added to the course advanced settings and get written to this model on save/publish. FWIW, I think your naming for this issue is clearer than the naming of this model. "Course Learner Progress Reset" doesn't quite roll off the tongue like "Course Reset", but I think it removes a lot of ambiguity.
Have you folks looked into using django-user-tasks for this?
You'll likely want to reset the container types as well, since they'll contain things like "last saved position", which might be disorienting (e.g. it might start them on the last Unit when they go a "new" Subsection, because it still has position data from their last attempt). I'm assuming that it's going to be acceptable that One thing that you might want to consider for this is to get the module_ids for this user directly from the database by adding a new method to the UserStateClient. This might save a lot of computation time since it's probable that they haven't done a significant portion of the course. It might also save you a little headache in tracking down things where there's an extra layer or two of XBlock nesting (e.g. SplitTestBlock). The big thing you'd have to look out for is that the indexes are badly designed for Also, should their user partition assignments be removed so that they get different content when looking at things like randomized blocks from a library? I don't know how this will interact with external proctoring systems. A lot of things in our system tie to a single enrollment or tuple of (user, block/module). Is it okay that notes are preserved for the content? (It seems like it should be.) I think you'll also want to create a new analytics event around this, because it's going to confuse a lot of downstream tracking log analysis code. |
|
@jmakowski1123 as per our discussion on meeting, here's what we did for this. The requirement was to reset a learner's progress or state because of similar issues regarding them not passing it, and also because of giving multiple certs. The way we did it was via removal of their Prior to that and since then I've not done this sort of feature, however as a developer who's testing enrollments and certificates for different users, this sort of feature could be useful at one go. As of now I'd reset the progress via instructor panel or via staff debug info button if the course has less components. It already has Reset learner state/Delete learner state option so I keep using those mostly. And if in some course where subsections are locked based on grading and students aren't passing, there too I keep using this feature of resetting learner state and grades via the panel. |
The current design is to enter "opted-in" courses via the Django admin, which is only available to certain site/instance staff users. The current design is org agnostic, and is simply a list of courses for which course progress can be reset.
The current design makes no distinction here.
No UI for the learner. The current approach has learners reaching out to site learner support staff via email, who will then perform the reset via the support-tools mfe. Learners will receive an email that the course progress has been reset, and they will see in-course that all course progress has been reset
Currently intended to only be platform-level support teams (ie 2U learner support). We're using the existing restrictions around who is able to access the support-tools MFE.
Yes, a learner’s data for their “current” attempt at a course run will be completely removed. It will allow a learner to re-attempt a course run in a case where waiting for the next run may not be desired. |
Currently we only allow one reset per learner per course run
Currently it's built into the code. The restrictions are:
We believe so
Not intended for course team
Noted
No. Is there any particular reason that Axim has for us to consider this vs our current approach?
Thank you, I didn’t realize we saved state for container blocks. We will add this to our project scope.
Yes
Noted.
Noted. Unclear at the moment how much value this would add, but we can investigate for potential future releases
Currently does not reset proctored exams as it is not intended for use with courses that use proctored exams but could be a future improvement. Our current approach is to have this review/responsibility be on the platform admins who are configuring these courses.
Notes and bookmarks are not removed
Yes, we are doing that. |
|
Got it, thanks so much for elucidating on the details @jansenk . There are a few reasons why we can't support this in the Core Product. It's functionality that's currently only intended for 2U-specific persona. And while I can see potentially expanding the use case to more generalized persona (like letting course teams or course admins make the reset), it's a pretty significant and hefty action to delete learner data entirely, and I'd prefer to devote more discovery time to fully understanding the implications of this, particularly from a data pipeline perspective, before we support any functionality or code in the core that enables this. Additionally, it's being built into the Support Tools MFE, which is a 2U-specific MFE that we currently don't support in Redwood, and are unlikely to support in future community releases, given the business-specific nature of that MFE. If, down the road, you're interested in exploring a more generalized implementation of this functionality, I'd be happy to coordinate the discovery work needed to get that off the ground, and to pull in other stakeholders who have explored their own implementations for added perspective. |
I originally suggested it because I thought you folks might be creating a course-team facing feature, and django-user-tasks gives useful feedback on progress. But since this is only for site admins, I don't think it's relevant.
I am concerned that this feature is going to run into a sea of edge cases when interacting with any number of other things that build off student state: FBE, content gating, milestones, user partitioning, teams, schedules, completion, grade reports, etc. Some may break in really small ways, like maybe the mobile app's concept of "pick up where you left off" is something inaccessible. But I believe this is changing underlying assumptions that are widespread enough where it's going to be a long term maintenance nightmare–one where each step of the way is going to feel like it's fixing "just one more thing", but there's always a new CAT-2 gremlin lying around the next corner. It's also something that will likely break as new things get developed over time, since many won't be aware of this feature at all. Or as use cases expand, because even if you document it in bold letters that it should not be used in combination with features X, Y, and Z, someone will eventually use it in that situation. I would strongly urge you folks to consider other ways to work around this, like making unadvertised course runs (or even CCX runs), and taking students through those instead. @jansenk, @Daniel-hershel, @jristau1984: Given @jmakowski1123's response in the previous comment, let's talk about how this development can be shifted into a plugin (or fork). There might be some pieces of general platform functionality that make sense to pull into edx-platform itself (e.g. the ability to enumerate CSM state for a user across the course), but it sounds like most of this should live in its own space. |
|
@jristau1984 To help clarify my confusion... Hi Jeremy, I wanted to address your questions in the same space as the proposal, so we can keep track of the conversation in one place. I'll ask @ormsbee to elucidate on the specifics of the concern about the data pipeline and unforeseen downstream consequences. As for the 2U specificity, we've gathered at least two sources of input from others across the community that an option to reset a course would be helpful, but it would need to be use-able functionality by course staff/course teams. One approach, maybe we could imagine it as a permission that could be attached to roles in the new R&P framework. My understanding is that it's currently being built in such a way that only site staff who have access to the Support Tools MFE (ie 2U staff) are able to use it. So out the gate, it's not really use-able more broadly without further investment. Please correct me if I'm wrong here. I wouldn't say it's a hard and fast rule that all new features must be fully feature complete out the gate. But if the first implementation is an "MVP" approach, it would need to be designed in a way that can scale and be easily generalized down the road. I believe Dave has some specific approach ideas to help describe what a platform-wide view/approach to overwriting student status could look like. Happy to keep this conversation going. |
It’s not just about running a known thing over and over again, though even that would pose operational risks, since firing off a bunch of tasks to recalculate grades on the same subsections could introduce race conditions and locking problems. Problem reset is typically used to get around an acute issue that a student is currently running into in a course. It’s usually very proximate to what the student is already working on. It doesn’t really try to rewind the clock in the way that this feature is doing. This sounds like it should also reset navigation state, re-init scheduling, re-gate content, etc. Just off the top of my head, we have things like:
That's not me doing discovery to try to figure out what else might be impacted. That's just an off-the-cuff answer. I'm sure there are more things out there, not the least of which is what it does to the analytics pipeline and reporting. Now I realize that the answer for any of these issues can be, “we just won’t use it on a course that uses that feature”, but that doesn’t scale. The feature as proposed needs to understand how every other feature that touches user course state works in order to either clear it out or tell the user that they can't use that feature in conjunction with this reset functionality. Every miss will be a bug of some sort or another, and at some point those bugs will turn into feature requests that try to patch up the holes. The only thing I can think of that would scale would be to decouple the concept of "user's interactions with the course" from the tuple (user, course). So for instance, we could create a new model like A lower effort thing that might get much the same effect is to make another course-type in way that CCX courses work, or extend CCX to work for this use case. That way, other systems are still insulated from the concept of having to reset and rewind progress–they think it's a different course run altogether. This might be aligned with course and run modeling cleanup that @feanil is championing, but we'd have to do more discovery. I would love to be wrong about this. If it turns out that the plugin implementation is straightforward and it works well, we can fold it into edx-platform at a later date. If you have the time to talk about a long term implementation that addressees the kinds of maintenance/impact issues I listed, I'd be happy to have that discussion, review discovery work, and work with you folks towards getting an implementation of this in edx-platform. But I don't feel comfortable with the proposed approach going to edx-platform directly. |
|
@jansenk, @jristau1984: BTW, if there's something you think I'm just not seeing here, I'm happy to meet to try to understand this better. |
Abstract
This document describes a feature that we are calling Full Course Reset. It serves a specific 2U business need but seems like it will likely provide a benefit to the open-source community as well.
Context & Background
Andhra Pradesh is a state in India whose ministry of education has decided to sponsor edX access on behalf of 400k university students. These students will have access to a subscription that will enable them to enroll in unlimited courses for the duration of their license, which will be approximately six months. Andhra Pradesh is granting users these subscriptions along with one mandatory course (chosen from a pool of approximately ten courses) that learners must complete before they may take any other courses.
Should a learner get to the end of their mandatory course and NOT have a passing grade, there is currently no way to enable them to retake the course. When confronted with course failure otherwise, we suggest that learners wait until the next course run to begin again. However, given the six-month duration of the learners’ licenses, there will likely not be any additional course runs for the mandatory selections from the ministry. Learners would be unable to complete their mandatory courses and would have no recourse.
Scope & Approach
We will add a new tab, “Course Reset” in the Learner Information section of the Support Tools MFE (https://github.com/openedx/frontend-app-support-tools).
Clicking the tab will show a new screen, which consists of a table with three columns: Course, Status, and Action. For each course that a learner is enrolled in which allows course reset, a row will be shown:
Course: Course display name and key.
Status: The status of the enrollment wrt/ course reset, that is,
Action: If a reset is available, the column will have a button to reset the course. If it is not, it will be either the word Unavailable or a disabled button
Clicking an active Reset button will pop a confirm dialog, with a text box to enter a “reason” or “memo” and confirming the dialog will send a request to the LMS to reset the course for the user.
The Status row will update to reflect that the request is in progress
Every ~ ten seconds we will poll an endpoint to update the status of the reset job and will update the status when the job is complete
Value & Impact
This feature will allow us to complete the terms of our agreement with Andhra Pradesh, and provide a framework for allowing full course learner reset to be a feature other partners can use in the future.
Milestones and/or Epics
Named Release
Redwood
Timeline
We hope to release this feature by the end of march
Proposed By
2U
Additional Info
https://docs.google.com/document/d/15j6yO6bsVB_IOEbRZXGwSd4oBj3yMGatwoqV2xnmMcY/edit?usp=sharing
Relates PRs
Tasks
The text was updated successfully, but these errors were encountered: