[DEPR]: Replace cs_comments_service #437
Comments
|
It sounds like the "Earliest Open edX Named Release Without This Functionality" will actually be Teak, not Sumac? |
regisb
changed the title
|
@kdmccormick Yes, I wasn't sure to indicate Sumac or Teak in this field. We are proposing that cs_comments_service be removed in Teak but, technically speaking, Sumac will be the first release to ship with the replacement of cs_comments_service by default. In the end I decided to announce the change for Sumac to avoid taking people by surprise. |
|
@hurtstotouchfire: Do you folks have time to give some feedback on this proposal? Thank you. |
|
I'll recap here the decisions that were reached in the BTR working group meeting that happened yesterday:
|
|
I've updated the timeline in the description to match this recap. |
|
Hello! Can a separate DEPR ticket be created for the MongoDB portion of what is listed in the description here? It feels like something independent (at least from a timeline perspective) of cs_comments_service. Thanks! |
|
Hi guys, it does not look like Forums V2 is actually merged and working for Sumac at this point. Can we update the DEPR timelines to reflect this? |
|
@jristau1984 You mean to say that the migration of data from mongodb to mysql should be migrated to a separate DEPR ticket? What for? Would you like to have a different timeline? @kbuchanan-2u Forum v2 was merged and is working in Sumac. It was rollbacked by 2U in master, but the release branch is doing just fine. |
|
@regisb Would you be able to share the test results for this work? From product perspective, I'm curious to know which forum features and workflows the team has considered. |
|
I assume you mean Q&A testing, right @ayub02? (Forum v2 has extensive unit and integration testing, but I suppose you are not referring to those) The following scenarios were tested:
Those scenarios were tested in the context of a Tutor deployment. As a consequence, some differences with the 2U architecture were not detected, including:
Our team was composed exclusively of developers and did not include Q&A members, so these results were not properly collected in a testing sheet. Still, I can guarantee that these tests were conducted. Do you have such a test sheet that you could share with us? It would be quite useful both to us and to the build/test/release working group, which is also testing the forum for every major Open edX release. |
|
@regisb yes i was asking about Q&A testing. Thank you for sharing the results. My question is more about user acceptance testing (UAT). Forums unfortunately has one of the most complex workflows in edX. Here is a sheet that we have used in the past to test discussion MFE after the completion of BD-38. https://docs.google.com/spreadsheets/d/1YEr9Mv6jTUmDNoz3GyZv0vIYuL3CARi6/edit?usp=sharing&ouid=111742414635622912267&rtpof=true&sd=true Maybe your team and run these tests with a backend configuration(s) that makes sense (you and @asadazam93 can probably discuss that). |
|
Marking as blocked until we can remove the old compatibility layer. |
dianakhuang
moved this from Blocked
to Ready for Migration
in DEPR: Deprecation & Removal
|
FYI we have logged the following issues: Many of these issues block additional testing because they are the base cases for functionality that still has additional cases to test. Because of this we have not completed our testing. Most of these issues would block us from fully migrating edx.org onto Forums V2. I want to also raise that for openedx/forum#149 in particular we have really no timeline for remediating this. We recognize that the xblock is unsupported, but we never completed the process of deprecating it on edx.org because it is enabled for certain long-running course runs, and there is no migration path without starting a new course run. We are working on figuring out a path forward for this edge case, but none of our options will be quick. |
|
@kbuchanan-2u: Thank you for the update, especially in regards to openedx/forum#149. I'm trying to find the proper DEPR trail for both the old non-MFE interface for forums as well as the inline discussion blocks. It looks like the Let's continue this discussion next week. |
Yeah, it does seem to be in a weird state right now, since new courses can't even create inline discussion blocks, and yet there's no formal deprecation of that code. In terms of being "unsupported"–the XBlock in question still works for old courses and hasn't been deprecated anywhere as far as I can tell, so I believe that as far as the project is concerned, it's still supported. So this is my understanding of the situation around openedx/forum#149 as a whole:
If any of these are incorrect, then please let me know, because nothing after this sentence is going to make sense in that case. I'm going to assume that creating a DEPR for the non-MFE forums tab view is not controversial. I can do this. In terms of the inline XBlock, I can think of a few options: A: Deprecate the inline XBlock. This would be the least amount of development effort, though it means that the v2 forum switchover would have to be pushed out a release cycle in order to give this new DEPR time to run. I personally favor this. B: Create a data migration. Create some automated way to convert old inline discussion blocks to their modern Unit-sidebar equivalents. I realize they're not semantically identical, but given common practices are to put one inline discussion per Unit, it's possible that something like this could work for the majority of cases. However, we've generally avoided doing things that force content publishing, since that carries many hazards with it. This is the most operationally risky path. C: Modify the inline discussion XBlock to use the new v2 endpoints. I'm not sure what the scope of this would be, and it seems wasteful to be throwing a lot of effort into backwards compatibility with something that was turned off in 2023. Still, this would unblock the v2 forums migration for Teak. D: Support the old endpoints using v2 forums. Similar to (C) in spirit and unblocking v2 forums migration for Teak, but carrying around slightly more legacy baggage. Maybe preferable if there is other code out there also hitting these endpoints (old mobile app code?) Again, I don't know the extent of this effort. Am I missing anything? |
|
@ormsbee thank you for thinking through this. I want to clarify a few things:
The only way forward that makes sense to me is fixing forum v2 to work with discussion xblocks. Also, i'll avoid using the word "legacy" in conversations because it may give a false indication of a pending deprecation. Except that the old discussion provider was renamed to "legacy" by OpenCraft during BD-38 ("old" for 2U, not for community). I hope this helps.
|
|
@ayub02: Thank you for the quick reply.
I understood the situation with that old discussion block repo when I wrote my comment, and the link I wrote there was to the one in edx-platform. That is not the reason I thought that inline discussion blocks were intended for deprecation. Instead, the reasons are:
Inline discussion blocks represent an old way of providing in-context discussion capability at the unit level. It has been documented and advertised as a legacy feature since 2023, when it was replaced by the new sidebar experience. From a product point of view, this is clearly a de facto deprecation. That's why I think @kbuchanan-2u's take is the correct one here:
It's also worth noting that the
The goal of the v2 forums rollout is to improve maintainability by aligning forums with the rest of our development stack, i.e. Django app + ORM backend, vs. Ruby service + MongoDB. In the long term, this will reduce operational and code complexity. Part of the simplification process involves removing features that are not in use today. For instance, the original forums data model allowed for arbitrary nesting of comment replies. Very early on, the product team decided against this direction and moved to the post-comment-reply scheme that we have now. There is no need for v2 forums to support arbitrary nesting of comments, since the only courses that ever ran in this mode used a Modulestore that no longer exists, and therefore can no longer be accessed by anyone. This happens in other places as well. XQueue still has code for push-grading, where XQueue itself would make direct HTTP calls to graders. This was an operational nightmare, and the reason why everyone switched over to using the pull mode and xqueue-watcher. The XQueue deprecation aims to be fully backwards compatible with xqueue-watcher, but not push-grading. Figuring out what does and doesn't need to be supported during a migration of this sort is what these DEPR ticket conversations are for. I realize that the inline discussion blocks are still in use today, but if they were deprecated and removed, there would be no need for v2 forums to support them.
The image is hugely helpful, thank you.
In terms of configuration, most of the community either runs Tutor (which sets the flags to run the newer discussions setup), or is run by operators who keep up with the latest. This mismatch between the base settings and what everyone actually runs has been a long standing issue that we're trying to reduce over time, but sometimes it's the only practical short-term workaround when dealing with deployment flags that are the de facto (tested-and-run) standard. So I think most of the community that is on later releases does run the new code path, which likely means that there are a number of bugs accumulating when trying to run the non-MFE version of the forums with the latest master. Which brings us back to one of the main reasons we have the DEPR process.
I can see this making sense if it's a relatively light lift on the dev side, since it will unblock a Teak rollout for v2 forum switchover. But I want to emphasize that I still consider inline discussion blocks to be on the path to deprecation. If we adapt v2 forums code to accommodate inline discussion blocks, it will only be to decouple them from this DEPR and its timeline. I can take a first pass at writing the DEPR for inline discussion blocks. I realize that there's a lot going on right now and that the migration portion of that DEPR may get complicated. They will absolutely still be supported in Teak–any removal wouldn't happen until at least the Ulmo release cycle. Thank you. |
Proposal Date
2024-04-16
Target Ticket Acceptance Date
2024-10-16
Earliest Open edX Named Release Without This Functionality
Sumac - 2024-10
Rationale
The Open edX forum is an essential feature of the platform. In technical terms, it is a web application that is developed in the cs_comments_service repository. This application is implemented in the Ruby programming language. Only a few Open edX contributors have expertise in Ruby: most developers are skilled in Python and NodeJS. This makes it extremely difficult to implement new features and extend the forum.
Moreover, the cs_comments_service application currently stores data in MongoDB, which is neither a consistent or a relational data storage solution -- as opposed to MySQL. This results both in operational and architecture complexity, as some data needs to be synced up between the two data stores (notably for user data).
Removal
The cs_comments_service repository will be removed. Note howerver that the tutor-forum plugin will be maintained, though it will be considerably simplified.
Replacement
The Ruby application will be replaced by a Django app, currently being developed in the following repository: https://github.com/edly-io/forum/ (soon to be migrated to the openedx organization, as per this public-engineering issue).
Deprecation
The cs_comments_service repository will include a deprecation notice in its README file. Migration will be transparent for most users running Tutor, as the tutor-forum plugin will smoothly transition from running the former ruby app to installing the new forum application in the edx-platform runtime.
Migration
Strategy
Let's call the legacy cs_comments_service "forum v1" and the new forum application "forum v2".
In forum v1, edx-platform communicates with the Ruby application via an HTTP rest API. All calls are made from the backend, Client-side API calls are made to edx-platform, which then forwards parts of the requests to forum v1.
In forum v2, these API calls take the form of a native Python API. This is made possible by the fact that the forum v2 application is installed in the same runtime as edx-platform.
This means that the diagram above will remain the same in the transition from v1 to v2. The only difference is that in step 3, communication will change from an HTTP API call to a native Python function call.
For storage, forum v2 will implement two different backends: one for MySQL (in the form of Django models) and one for MongoDB. The MongoDB backend is only preserved to ease the transition from v1 to v2; it is expected that it will be eventually removed.
Opt-out
Forum v2 will make use of two CourseWaffleFlag (source code). For every course, platform administrators will have the choice to:
By default, users running Tutor (and the tutor-forum plugin) will automatically transition from v1 to v2 and the MySQL models. In that sense, the transition is opt-out rather than opt-in.
Proposed timeline
Additional Info
Task List
The text was updated successfully, but these errors were encountered: