feat: fetching of a secured Algolia key [BB-8083]

[0x29a]

Description

Adds a fallback mechanism, so when the ALGOLIA_SEARCH_API_KEY environment variable is not set, the key is fetched from the endpoint defined by the ALGOLIA_SECURED_KEY_ENDPOINT environment variable. It expects this endpoint to return a JSON like:

{
  "key": "<ALGOLIA_SEARCH_API_KEY>"
}

This is intended, but not limited to, to be used with open-craft/edx-enterprise#11.

Testing steps

We're going to test this PR, open-craft/edx-enterprise#11 and open-craft/edx-platform#602 at once by creating three courses and three enterprise customers, uploading them to Algolia through enterprise-catalog and checking that learners are able to browse catalogs of their enterprises with the help of this MFE, but unable to access courses of enterprises they don't belong to.

Prerequisites

You'll need an Algolia account. Create an application and enterprise-catalog index. Write down the application id, admin API key, and search API key somewhere.

Installing Palm devstack

mkdir palm && cd palm
export DEVSTACK_WORKSPACE=$(pwd)
git clone -b open-release/palm.master [email protected]:openedx/devstack.git && cd devstack
virtualenv -p python3 .fenv
source .fenv/bin/activate
export OPENEDX_RELEASE=palm.master
make requirements
make dev.clone
docker pull edxops/devpi && docker tag edxops/devpi edxops/devpi:palm.master && docker pull edxops/ecommerce && docker tag edxops/ecommerce edxops/ecommerce:palm.master && docker pull edxops/credentials && docker tag edxops/credentials edxops/credentials:palm.master && docker pull edxops/notes && docker tag edxops/notes edxops/notes:palm.master
cd ..
git clone -b open-release/palm.master [email protected]:openedx/configuration.git && cd configuration
docker build -f docker/build/forum/Dockerfile . -t edxops/forum:palm.master
cd ../edx-platform
git remote add oc [email protected]:open-craft/edx-platform.git
git fetch oc 0x29a/bb8083/per-user-algolia-key
git checkout 0x29a/bb8083/per-user-algolia-key
DOCKER_BUILDKIT=1 docker build . --build-arg SERVICE_VARIANT=lms --build-arg SERVICE_PORT=8000 --target development -t openedx/lms-dev:palm.master
cd ../devstack
sed -i s/"make requirements"/"pip install --upgrade pip==23.1 \&\& make requirements"/g provision-credentials.sh
make dev.provision
make dev.up.large-and-slow

After that:

1. cd ../edx-platform
2. Add ENTERPRISE_ALGOLIA_SEARCH_API_KEY = '<YOUR_ALGOLIA_SEARCH_(NOT ADMIN)_API_KEY>' to the end of lms/envs/devstack.py.

Installing edx-enterprise fork

sudo chown $USER:$USER ../src
cd ../src
git clone -b 0x29a/bb8083/per-user-algolia-key [email protected]:open-craft/edx-enterprise.git
cd ../devstack
make lms-shell
cd /edx/src/edx-enterprise/
pip uninstall edx-enterprise
pip install -e .
exit
make lms-shell
python manage.py lms migrate
exit
make studio-shell
cd /edx/src/edx-enterprise/
pip uninstall edx-enterprise
pip install -e .
exit
make lms-restart
make studio-restart

Installing enterprise-catalog

cd ..
git clone -b open-release/palm.master [email protected]:openedx/enterprise-catalog.git && cd enterprise-catalog
sed -i s/devstack_default/devstack-palmmaster_default/g docker-compose.yml
sed -i s/edx.devstack/edx.devstack-palm.master/g provision-catalog.sh enterprise_catalog/settings/devstack.py enterprise_catalog/apps/catalog/fixtures/contentmetadata.json

After that:

1. Open enterprise_catalog/apps/catalog/constants.py and add 'source' to the CONTENT_PRODUCT_SOURCE_ALLOW_LIST set.
2. Add the following to the end of enterprise_catalog/settings/devstack.py:

   ALGOLIA = {
       'INDEX_NAME': 'enterprise-catalog',
       'APPLICATION_ID': '<YOUR_ALGOLIA_APPLICATION_ID>',
       'API_KEY': '<YOUR_ADMIN_ALGOLIA_API_KEY>',
   }

3. Run make dev.provision.

Installing frontend-app-learner-portal-enterprise

cd ..
git clone -b 0x29a/bb8083/per-user-algolia-key [email protected]:open-craft/frontend-app-learner-portal-enterprise.git && cd frontend-app-learner-portal-enterprise
export ALGOLIA_APP_ID=<YOUR_ALGOLIA_APPLICATION_ID>
export ALGOLIA_INDEX_NAME=enterprise-catalog
nvm use
npm install
npm start

Leave this terminal open, do all the remaining steps in a separate one.

Creating test entities

1. Create three courses here: http://localhost:18010/home/. Use ABC, XYZ, and PSY for all fields, like here:
   
2. Add a section with a dropdown unit to each course, publish each.
3. Set Course Start Date and Enrollment Start Date for each course here to 01/01/2019.
4. Add all three courses here, like this:
   
5. Add verified and audit course modes for each course here: http://localhost:18000/admin/course_modes/coursemode/
6. Create a source with a Source name here: http://localhost:18381/admin/course_metadata/source/add/
7. Go here and create three queries with the following content filters for each course:

   {
       "content_type":[
           "course",
           "courserun"
       ],
       "aggregation_key":[
           "course:ABC+ABC",
           "courserun:ABC+ABC"
       ],
       "partner":"edx",
       "availability":[
           "Current",
           "Starting Soon",
           "Upcoming"
       ],
       "status":"published"
   }

   Change ABC to XYZ and PSY for each query.
8. Go here and create three enterprise customers: XXX, YYY, ZZZ. Change their slugs respectively. Check Enable learner portal for each.
9. Go here and create three catalogs for each customer selecting different course queries we created earlier. XXX should be assigned to the query selecting ABC course, YYY -> XYZ and ZZZ -> PSY.

Indexing

Go to the <DEVSTACK_WORKSPACE> and replace DEFAULT_PRODUCT_SOURCE_SLUG = '' with DEFAULT_PRODUCT_SOURCE_SLUG = 'source' in course_discovery/settings/base.py.

Then, go to the <DEVSTACK_WORKSPACE>/devstack directory run the following to pull the data from LMS to Course Discovery:

export OPENEDX_RELEASE=palm.master
make discovery-shell
./manage.py refresh_course_metadata

Then:

1. Go here and set a name for each organization.
2. Go here and change every course run to be "Published"
3. Run the following in the discovery shell to fill in the product source for each course and update the ES index:

./manage.py populate_default_product_source
./manage.py update_index --disable-change-limit
exit

Now we need to copy course queries and other enterprise-related data from LMS to enterprise-catalog. Run the following:

make lms-shell
./manage.py lms migrate_enterprise_catalogs --api_user enterprise_catalog_worker

This will fail, but it'll create a enterprise_catalog_worker user here. Go here and assign enterprise_catalog_admin role to [email protected], don't forget to check Applies to all contexts.
Now, run this again:

./manage.py lms migrate_enterprise_catalogs --api_user enterprise_catalog_worker

After that you should see three of your course queries here.

Now, go to the <DEVSTACK_WORKSPACE>/enterprise-catalog and run the following to fetch content metadata (courses and course runs) from Course Discovery:

make app-shell
./manage.py update_content_metadata --force

After that you should see Course and Course Run objects for each of your courses here. If you don't, then something went wrong. Also, Json metadata for each course should contain a non-empty advertised_course_run_uuid.

Now you can upload all courses to Algolia:

./manage.py reindex_algolia

If everything go fine, you should see three courses in your enterprise-catalog Algolia index.

Enrolling users

1. Register a new user: [email protected].
2. Go the XXX enterprise customer and click Manager Learners (example of the admin panel URL).
3. Now, enroll [email protected] user to the course-v1:ABC+ABC+ABC course. Specify Audit track and test reason for manual enrollment.
4. ^ This will fail. But a new enterprise_worker user will appear here. Assign the enterprise_catalog_admin role to the [email protected] user here (don't forget to check Applies to all contexts) and try point 3 again.
5. Repeat points 2 and 3, but for YYY enterprise customer and course-v1:XYZ+XYZ+XYZ course, so we have a single learner present in two different enterprises.

Testing frontend-app-learner-portal-enterprise

1. Sign out as [email protected].
2. Go to http://localhost:8734.
3. You should be redirected to the LMS.
4. Upon signing in again as [email protected], you should be offered to choose an organization. Choose XXX.
5. You should be redirected to the MFE. You shoud see the ABC course here.
6. Now, if you visit this URL again and select YYY, you should see the XYZ course.

Now, open the browser dev tools and locate a query like this:

https://ql3i9veweo-dsn.algolia.net/1/indexes/*/queries?x-algolia-agent=Algolia for JavaScript (4.6.0); Browser; JS Helper (3.12.0); react (17.0.2); react-instantsearch (6.38.1)

1. Click the right mouse button on it and Edit and Resend (that's for Firefox).
2. Paste the following JSON to the Body field and click Send:

   {
     "requests": [
       {
         "indexName": "enterprise-catalog",
         "params": "facetingAfterDistinct=true&facets=%5B%5D&highlightPostTag=%3C%2Fais-highlight-0000000000%3E&highlightPreTag=%3Cais-highlight-0000000000%3E&tagFilters="
       }
     ]
   }

   Normally, this request would fetch all courses from the index. But the MFE is using our secured Algolia key now.
3. Observe what this query returned. There should be only 2 hits, for the ABC and XYZ courses, omitting the PSY course.

Upstream PR

openedx#887

[CefBoud]

Great job on this, @0x29a! It was a hell of a ride getting through the steps😛. But everything worked handsomely!

I left a small comment.

[CefBoud]

LGTM ✅

• I tested this
• I read through the code
• Includes tests

[Agrendalath]

@0x29a, this is awesome!

👍

• I tested this: learners cannot modify their queries to retrieve catalogs from other organizations; the behavior of this MFE does not change as long as we have the ALGOLIA_SEARCH_API_KEY set
• I read through the code
• I checked for accessibility issues: n/a
• Includes documentation: n/a
• I made sure any change in configuration variables is reflected in the corresponding client's configuration-secure repository: n/a