Merge pull request #1139 from overhangio/sumac

Upgrade to Sumac

.github/workflows/release.yml

@@ -22,7 +22,7 @@ jobs:
           # It's important that we build the tutor binaries with the *oldest* possible
           # OS releases and Python version. See these docs for more information:
           # https://pyinstaller.org/en/stable/usage.html#making-gnu-linux-apps-forward-compatible
-          - os: ubuntu-20.04
+          - os: ubuntu-22.04
             locale: C.UTF-8
           # https://endoflife.date/macos
           - os: macos-13

CHANGELOG.md

@@ -20,6 +20,36 @@ instructions, because git commits are used to generate release notes:
 
 <!-- scriv-insert-here -->
 
+<a id='changelog-19.0.0'></a>
+## v19.0.0 (2024-12-11)
+
+- 💥[Feature] Upgrade default charset and collation of mysql to utf8mb4 and utf8mb4_unicode_ci respectively (by @Danyal-Faheem)
+    - Add do command to upgrade the charset and collation of tables in mysql.
+    - The command will perform the following upgrades:
+      - Upgrade all `utf8mb3` charset to `utf8mb4`
+      - Upgrade collation `utf8mb3_general_ci` to `utf8mb4_unicode_ci`
+      - Upgrade collation `utf8mb3_bin` to `utf8mb4_bin`
+      - Upgrade collation `utf8mb3_*` to `utf8mb4_*`
+
+- [Feature] Create a new /data/openedx-media-private volume to store media files that are not publicly accessible via the browser, and create configuration that Learning Core (openedx-learning) will use to access this storage space. Learning Core will use a sub-directory of /data/openedx-media-private to store Content Library file uploads, but other apps can create their own sub-directories for their own use. (by @ormsbee)
+
+- 💥[Feature] Update Open edX Image to use Ubuntu 22.04 as base OS. (by @dawoudsheraz)
+  - Adds xmlsec related dependencies to fix xmlsec import issues during translations build
+
+- 💥[Feature] Migrate from local.edly.io to local.openedx.io. (by @regisb)
+
+- 💥[Feature] Upgrade to Sumac. (by @dawoudsheraz)
+
+- 💥[Feature] Replace Elasticsearch by Meilisearch. Elasticsearch was both a source of complexity and high resource usage. With this change, we no longer run Elasticsearch to perform common search queries across Open edX. This includes: course discovery, courseware search and studio search. Instead, we index all these documents in a Meilisearch instance, which is much more lightweight in terms of memory consumption. (by @regisb)
+
+- [Bugfix] Don't build uwsgi with XML support (by @feanil)
+
+- [Feature] With the new forum v2 application, users have a choice to use MongoDB or MySQL as a storage backend, course per course. New users will automatically start using MySQL, while existing users will be responsible for migrating their data themselves (before the Teak release). Upgrade instructions are available here: https://github.com/overhangio/tutor-forum/#installation. (by @regisb)
+
+- 💥[Improvement] Get rid of the `is_docker_rootless` template filter, which was used only by Elasticsearch. (by @regisb)
+
+[Improvement] Forcefully enable the Learning MFE's navigation sidebar when upgrading to Sumac. (by @arbrandes)
+
 <a id='changelog-18.2.2'></a>
 ## v18.2.2 (2024-12-10)
 

Makefile

@@ -90,7 +90,7 @@ bootstrap-dev-plugins: bootstrap-dev ## Install dev requirements and all support
 	pip install -r requirements/plugins.txt
 
 pull-base-images: # Manually pull base images
-	docker image pull docker.io/ubuntu:20.04
+	docker image pull docker.io/ubuntu:22.04
 
 ci-info: ## Print info about environment
 	python --version

docs/configuration.rst

@@ -40,7 +40,7 @@ With an up-to-date environment, Tutor is ready to launch an Open edX platform an
 Individual service activation
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
-- ``RUN_ELASTICSEARCH`` (default: ``true``)
+- ``RUN_MEILISEARCH`` (default: ``true``)
 - ``RUN_MONGODB`` (default: ``true``)
 - ``RUN_MYSQL`` (default: ``true``)
 - ``RUN_REDIS`` (default: ``true``)
@@ -71,9 +71,9 @@ This configuration parameter defines the name of the Docker image to run the dev
 
 This configuration parameter defines which Caddy Docker image to use.
 
-- ``DOCKER_IMAGE_ELASTICSEARCH`` (default: ``"docker.io/elasticsearch:7.17.9"``)
+- ``DOCKER_IMAGE_MEILISEARCH`` (default: ``"docker.io/getmeili/meilisearch:v1.8.4"``)
 
-This configuration parameter defines which Elasticsearch Docker image to use.
+This configuration parameter defines which Meilisearch Docker image to use.
 
 - ``DOCKER_IMAGE_MONGODB`` (default: ``"docker.io/mongo:7.0.7"``)
 
@@ -132,7 +132,7 @@ Open edX customisation
 
 This defines the git repository from which you install Open edX platform code. If you run an Open edX fork with custom patches, set this to your own git repository. You may also override this configuration parameter at build time, by providing a ``--build-arg`` option.
 
-- ``OPENEDX_COMMON_VERSION`` (default: ``"open-release/redwood.3"``, or ``master`` in :ref:`Tutor Main <main>`)
+- ``OPENEDX_COMMON_VERSION`` (default: ``"open-release/sumac.1"``, or ``master`` in :ref:`Tutor Main <main>`)
 
 This defines the default version that will be pulled from all Open edX git repositories.
 
@@ -228,13 +228,19 @@ By default, a running Open edX platform deployed with Tutor includes all necessa
 .. note::
     When configuring an external MySQL database, please make sure it is using version 8.4.
 
-Elasticsearch
-*************
+Meilisearch
+***********
+
+- ``MEILISEARCH_URL`` (default: ``"http://meilisearch:7700"``): internal URL used for backend-to-backend communication.
+- ``MEILISEARCH_PUBLIC_URL`` (default: ``"{% if ENABLE_HTTPS %}https{% else %}http{% endif %}://meilisearch.{{ LMS_HOST }}"``): external URL from which the frontend will access the Meilisearch instance.
+- ``MEILISEARCH_INDEX_PREFIX`` (default: ``"tutor_"``)
+- ``MEILISEARCH_MASTER_KEY`` (default: ``"{{ 24|random_string }}"``)
+- ``MEILISEARCH_API_KEY_UID`` (default: ``"{{ 4|uuid }}"``): UID used to sign the API key.
+- ``MEILISEARCH_API_KEY`` (default: ``"{{ MEILISEARCH_MASTER_KEY|uid_master_hash(MEILISEARCH_API_KEY_UID) }}"``)
+
+To reset the Meilisearch API key, make sure to unset both the API key and it's UID:
 
-- ``ELASTICSEARCH_SCHEME`` (default: ``"http"``)
-- ``ELASTICSEARCH_HOST`` (default: ``"elasticsearch"``)
-- ``ELASTICSEARCH_PORT`` (default: ``9200``)
-- ``ELASTICSEARCH_HEAP_SIZE`` (default: ``"1g"``)
+    tutor config save --unset MEILISEARCH_API_KEY_UID MEILISEARCH_API_KEY
 
 MongoDB
 *******
@@ -378,9 +384,9 @@ Note that your edx-platform version must be a fork of the latest release **tag**
 
 If you don't create your fork from this tag, you *will* have important compatibility issues with other services. In particular:
 
-- Do not try to run a fork from an older (pre-Redwood) version of edx-platform: this will simply not work.
+- Do not try to run a fork from an older (pre-Sumac) version of edx-platform: this will simply not work.
 - Do not try to run a fork from the edx-platform master branch: there is a 99% probability that it will fail.
-- Do not try to run a fork from the open-release/redwood.master branch: Tutor will attempt to apply security and bug fix patches that might already be included in the open-release/redwood.master but which were not yet applied to the latest release tag. Patch application will thus fail if you base your fork from the open-release/redwood.master branch.
+- Do not try to run a fork from the open-release/sumac.master branch: Tutor will attempt to apply security and bug fix patches that might already be included in the open-release/sumac.master but which were not yet applied to the latest release tag. Patch application will thus fail if you base your fork from the open-release/sumac.master branch.
 
 .. _i18n:
 

docs/dev.rst

@@ -28,7 +28,7 @@ This will perform several tasks. It will:
 * build the "openedx-dev" Docker image, which is based on the "openedx" production image but is `specialized for developer usage`_ (eventually with your fork),
 * stop any existing locally-running Tutor containers,
 * disable HTTPS,
-* set ``LMS_HOST`` to `local.edly.io <http://local.edly.io>`_ (a convenience domain that simply `points at 127.0.0.1 <https://dnschecker.org/#A/local.edly.io>`_),
+* set ``LMS_HOST`` to `local.openedx.io <http://local.openedx.io>`_ (a convenience domain that simply `points at 127.0.0.1 <https://dnschecker.org/#A/local.openedx.io>`_),
 * prompt for a platform details (with suitable defaults),
 * start LMS, CMS, supporting services, and any plugged-in services,
 * ensure databases are created and migrated, and
@@ -42,8 +42,8 @@ Additionally, when a local clone of edx-platform is bind-mounted, it will:
 
 Once setup is complete, the platform will be running in the background:
 
-* LMS will be accessible at `http://local.edly.io:8000 <http://local.edly.io:8000>`_.
-* CMS will be accessible at `http://studio.local.edly.io:8001 <http://studio.local.edly.io:8001>`_.
+* LMS will be accessible at `http://local.openedx.io:8000 <http://local.openedx.io:8000>`_.
+* CMS will be accessible at `http://studio.local.openedx.io:8001 <http://studio.local.openedx.io:8001>`_.
 * Plugged-in services should be accessible at their documented URLs.
 
 Now, use the ``tutor dev ...`` command-line interface to manage the development environment. Some common commands are described below.
@@ -113,7 +113,7 @@ The ``openedx-dev`` Docker image is based on the same ``openedx`` image used by
 
 - The user that runs inside the container has the same UID as the user on the host, to avoid permission problems inside mounted volumes (and in particular in the edx-platform repository).
 - Additional Python and system requirements are installed for convenient debugging: `ipython <https://ipython.org/>`__, `ipdb <https://pypi.org/project/ipdb/>`__, vim, telnet.
-- The edx-platform `development requirements <https://github.com/openedx/edx-platform/blob/open-release/redwood.master/requirements/edx/development.in>`__ are installed.
+- The edx-platform `development requirements <https://github.com/openedx/edx-platform/blob/open-release/sumac.master/requirements/edx/development.in>`__ are installed.
 
 
 If you are using a custom ``openedx`` image, then you will need to rebuild ``openedx-dev`` every time you modify ``openedx``. To so, run::

docs/install.rst

@@ -123,11 +123,11 @@ Major Open edX releases are published twice a year, in June and December, by the
 4. Test the new release in a sandboxed environment.
 5. If you are running edx-platform, or some other repository from a custom branch, then you should rebase (and test) your changes on top of the latest release tag (see :ref:`edx_platform_fork`).
 
-The process for upgrading from one major release to the next works similarly to any other upgrade, with the ``launch`` command (see above). The single difference is that if the ``launch`` command detects that your tutor environment was generated with an older release, it will perform a few release-specific upgrade steps. These extra upgrade steps will be performed just once. But they will be ignored if you updated your local environment (for instance: with ``tutor config save``) before running ``launch``. This situation typically occurs if you need to re-build some Docker images (see above). In such a case, you should make use of the ``upgrade`` command. For instance, to upgrade a local installation from Quince to Redwood and rebuild some Docker images, run::
+The process for upgrading from one major release to the next works similarly to any other upgrade, with the ``launch`` command (see above). The single difference is that if the ``launch`` command detects that your tutor environment was generated with an older release, it will perform a few release-specific upgrade steps. These extra upgrade steps will be performed just once. But they will be ignored if you updated your local environment (for instance: with ``tutor config save``) before running ``launch``. This situation typically occurs if you need to re-build some Docker images (see above). In such a case, you should make use of the ``upgrade`` command. For instance, to upgrade a local installation from Redwood to Sumac and rebuild some Docker images, run::
 
     tutor config save
     tutor images build all # list the images that should be rebuilt here
-    tutor local upgrade --from=quince
+    tutor local upgrade --from=redwood
     tutor local launch
 
 
@@ -157,6 +157,8 @@ Instructions for installing the appropriate Tutor version for older Open edX rel
 +-------------------+---------------+--------------------------------------------+
 | Redwood           | v18           | pip install 'tutor[full]>=18.0.0,<19.0.0'  |
 +-------------------+---------------+--------------------------------------------+
+| Sumac             | v19           | pip install 'tutor[full]>=19.0.0,<20.0.0'  |
++-------------------+---------------+--------------------------------------------+
 
 .. _autocomplete:
 

docs/local.rst

@@ -141,6 +141,33 @@ The default Open edX theme is rather bland, so Tutor makes it easy to switch to
 
 Out of the box, only the default "open-edx" theme is available. We also developed `Indigo, a beautiful, customizable theme <https://github.com/overhangio/indigo>`__ which is easy to install with Tutor.
 
+Changing the mysql charset and collation
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+.. note:: This command has been tested only for users upgrading from Quince. While it is expected to work for users on earlier releases, please use it with caution as it has not been tested with those versions.
+
+The database's charset and collation might not support specific characters or emojis. Tutor will function normally without this change unless specific characters are used in the instance.
+
+.. warning:: This change is potentially irreversible. It is recommended to make a backup of the MySQL database. See the :ref:`database dump instructions <database_dumps>` to create a DB dump.
+
+To change the charset and collation of all the tables in the openedx database, run::
+
+    tutor local do convert-mysql-utf8mb4-charset
+
+Alternatively, to change the charset and collation of certain tables or to exclude certain tables, the ``--include`` or ``--exclude`` options can be used. These options take comma separated names of tables/apps with no space in-between. To upgrade the ``courseware_studentmodule`` and ``courseware_studentmodulehistory`` tables, run::
+
+    tutor local do convert-mysql-utf8mb4-charset --include=courseware_studentmodule,courseware_studentmodulehistory
+
+Tutor performs pattern matching from the start of the table name so just the name of the app is enough to include/exclude all the tables under that app. To upgrade all the tables in the database except the ones under the student and wiki apps, run::
+
+    tutor local do convert-mysql-utf8mb4-charset --exclude=student,wiki
+
+In the above command, all the tables whose name starts with either student or wiki will be excluded from the upgrade process.
+
+By default, only the tables in the openedx database are changed. For upgrading tables in any additional databases used by plugins, the ``--database`` option can be used to upgrade them. To upgrade all the tables in the discovery database, run::
+
+    tutor local do convert-mysql-utf8mb4-charset --database=discovery
+
 Running arbitrary ``manage.py`` commands
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 

docs/reference/indexes.rst

@@ -7,10 +7,10 @@ Plugin indexes are a great way to have your plugins discovered by other users. P
 Index file paths
 ================
 
-A plugin index is a yaml-formatted file. It can be stored on the web or on your computer. In both cases, the index file location must end with "<current release name>/plugins.yml". For instance, the following are valid index locations if you run the Open edX "Redwood" release:
+A plugin index is a yaml-formatted file. It can be stored on the web or on your computer. In both cases, the index file location must end with "<current release name>/plugins.yml". For instance, the following are valid index locations if you run the Open edX "Sumac" release:
 
-- https://overhang.io/tutor/main/redwood/plugins.yml
-- ``/path/to/your/local/index/redwood/plugins.yml``
+- https://overhang.io/tutor/main/sumac/plugins.yml
+- ``/path/to/your/local/index/sumac/plugins.yml``
 
 To add either indexes, run the ``tutor plugins index add`` command without the suffix. For instance::
 
@@ -106,9 +106,9 @@ Manage plugins in development
 Plugin developers and maintainers often want to install local versions of their plugins. They usually achieve this with ``pip install -e /path/to/tutor-plugin``. We can improve that workflow by creating an index for local plugins::
 
     # Create the plugin index directory
-    mkdir -p ~/localindex/redwood/
+    mkdir -p ~/localindex/sumac/
     # Edit the index
-    vim ~/localindex/redwood/plugins.yml
+    vim ~/localindex/sumac/plugins.yml
 
 Add the following to the index::
 

docs/tutorials/datamigration.rst

@@ -22,6 +22,8 @@ With Tutor, all data are stored in a single folder. This means that it's extreme
 
     tutor local start -d
 
+.. _database_dumps:
+
 Making database dumps
 ---------------------
 

docs/tutorials/edx-platform.rst

@@ -69,7 +69,7 @@ Quite often, developers don't want to work on edx-platform directly, but on a de
     cd /my/workspace/edx-ora2
     git clone https://github.com/openedx/edx-ora2 .
 
-Then, check out the right version of the package. This is the version that is indicated in the `edx-platform/requirements/edx/base.txt <https://github.com/openedx/edx-platform/blob/open-release/redwood.master/requirements/edx/base.txt>`__. Be careful that the version that is currently in use in your version of edx-platform is **not necessarily the head of the master branch**::
+Then, check out the right version of the package. This is the version that is indicated in the `edx-platform/requirements/edx/base.txt <https://github.com/openedx/edx-platform/blob/open-release/sumac.master/requirements/edx/base.txt>`__. Be careful that the version that is currently in use in your version of edx-platform is **not necessarily the head of the master branch**::
 
     git checkout <my-version-tag-or-branch>
 

docs/tutorials/scale.rst

@@ -37,11 +37,11 @@ Offloading data storage
 
 Aside from web workers, the most resource-intensive services are in the data persistence layer. They are, by decreasing resource usage:
 
-- `Elasticsearch <https://www.elastic.co/elasticsearch/>`__: indexing of course contents and forum topics, mostly for search. Elasticsearch is never a source of truth in Open edX, and the data can thus be trashed and re-created safely.
 - `MySQL <https://www.mysql.com>`__: structured, consistent data storage which is the default destination of all data.
 - `MongoDB <https://www.mongodb.com>`__: structured storage of course data.
 - `Redis <https://redis.io/>`__: caching and asynchronous task management.
 - `MinIO <https://min.io>`__: S3-like object storage for user-uploaded files, which is enabled by the `tutor-minio <https://github.com/overhangio/tutor-minio>`__ plugin. It is possible to replace MinIO by direct filesystem storage (the default), but scaling will then become much more difficult down the road.
+- `Meilisearch <https://www.meilisearch.com>`__: indexing of course contents and forum topics, mostly for search. Meilisearch is never a source of truth in Open edX, and the data can thus be trashed and re-created safely.
 
 When attempting to scale a single-server deployment, we recommend starting by offloading some of these stateful data storage components, in the same order of priority. There are multiple benefits:
 

docs/tutorials/theming.rst

@@ -46,12 +46,12 @@ Then, run a local webserver::
 
     tutor dev start lms
 
-The LMS can then be accessed at http://local.edly.io:8000. You will then have to :ref:`enable that theme <settheme>`::
+The LMS can then be accessed at http://local.openedx.io:8000. You will then have to :ref:`enable that theme <settheme>`::
 
     tutor dev do settheme mythemename
 
 Watch the themes folders for changes (in a different terminal)::
 
     tutor dev run watchthemes
 
-Make changes to some of the files inside the theme directory: the theme assets should be automatically recompiled and visible at http://local.edly.io:8000.
+Make changes to some of the files inside the theme directory: the theme assets should be automatically recompiled and visible at http://local.openedx.io:8000.