Skip to content

Developing on the edX Developer Stack

Jump to bottom Edit New page
pomegranited edited this page Dec 5, 2015 · 59 revisions

Table of Contents generated with DocToc

This page describes everything you need to know to begin development on the edX Developer Stack (also known as devstack).

Installing devstack

The edX devstack is a Vagrant instance designed for local development. Before beginning development, read edX Developer Stack which describes how to install and run the edX software.

Debugging the devstack image

Debugging with devstack is a little convoluted, as the source code lives on your local machine but the code is executing within the Vagrant image. One solution is to use PyCharm from JetBrains. It is able to debug devstack using its concept of remote Python interpreters. This only works for the Professional edition.

  • If you want to experiment with PyCharm, you can download a trial of the professional edition. This is different than the open-source edition which doesn't support the vagrant functionality.

See Setting up PyCharm for edX development.

Testing your changes

See the Test Engineering FAQ for all your questions about testing the edX platform. You can also find detailed information on running unit, integration and acceptance tests in the latest platform docs.

Visually debug your tests

You can setup your development environment such that you can visually interact with browsers and other GUIs in the vagrant machine from the host machine. To do this, you will need to install XQuartz. For further information, see Test Engineering FAQ.

Configuring Themes in Devstack

There are currently two ways to theme the platform:

  • "Stanford Theming" -- This is a set of conditions in the code that key off a settings (settings.FEATURES.USE_CUSTOM_THEME and settings.THEME_NAME) to look aside to custom styling (assets) and templates. It's a bit of a hack due to pipeline complexity (sass/ruby dependencies) and how Mako templates are named and loaded.

  • "Microsites" -- These are separate conditions that change site appearance dynamicaly keyed off the hostname of the request. foo.edx.org can appear one way, and bar.edx.org can appear another.

This section describes how to use the former with devstack, written by Sef ([email protected]).

  1. Configure. Change two settings in /edx/app/edxapp/lms.envs.json

     "FEATURES": {
     ...
     "USE_CUSTOM_THEME": true
     ...
 },

    and

     ...
 "THEME_NAME": "default",
 ...

  2. Mount. Make sure Vagrant mounts the "theme" directory from the host. This should be the case with Johnnycake or later Vagrant installations. If you have an older Vagrantfile, see PR #884.

  3. Check out. The rest of the devstack code checks out repos as part of the provision step, but that tooling doesn't happen for themes. You need to check out a theme yourself. I recommend you start with the "null theme" that we've made here at Stanford:

     cd themes
 git clone [email protected]:Stanford-Online/edx-default-theme.git default

  4. Gather assets. The rest of the devstack methods seems to imply the service variant (lms or cms) automatically, but this doesn't work with the theme pipeline. So to make this work I have to gather assets this way:

     vagrant ssh
 sudo su edxapp
 paver update_assets lms --settings=devstack

  5. From PyCharm. I created an external command to do this from PyCharm.

    1. Open up "Remote SSH External Tools"
    2. Name = "lms assets"
    3. Connection settings = Default interpreter. This assumes that you've setup up pycharm with the default interpreter to ssh into the edxapp user.
    4. Program = bash
    5. Parameters = -c "SERVICE_VARIANT=lms /edx/app/edxapp/.gem/bin/rake lms:gather_assets:devstack"
    6. Working directory = /edx/app/edxapp/edx-platform
    7. Screenshots: configuring and resulting menu item.

There is nothing special about the name "default" in steps 1 and 3 above. They just have to match up: the configuration file should have the same name as the directory in themes. For example, I have both "stanford" and "default" subdirectories there and switching between them is just a matter of changing lms.envs.json, re-gathering assets, and restarting LMS.

The edx-platform database that devstack uses

To syncdb and migrate from the shell as the edxapp user:

./manage.py lms --settings=devstack syncdb
./manage.py lms --settings=devstack migrate

If there are errors, you might be able to resolve them with

./manage.py lms --settings=devstack migrate --merge

followed by syncdb'ing

Devstack uses mySQL as the DB engine for the edx-platform app. You can tell this by starting up the Django shell with:

./manage.py lms shell --settings=devstack

Then in the Django shell:

from django.conf import settings
settings.DATABASES

You can manipulate the database with the mysql command line interface from within your vagrant terminal session:

mysql --user=root --port=3306  edxapp

If you want to use a GUI tool from your host system, that is also possible. E.g. for MySQL Workbench, you would set up a new Server connection with these settings:

  • Connection Method: Standard TCP/IP over SSH
  • SSH Hostname: 192.168.33.10:22
  • SSH Username: vagrant (the password is 'vagrant')
  • MySQL Hostname: 127.0.0.1
  • MySQL Server Port: 3306
  • Username: root (no password)
  • Default Schema: edxapp

Note: You may also need to specify the SSH Key File, which should be located at "~/.vagrant.d/insecure_private_key"

If you get the "Could not connect the SSH Tunnel; Authentication failed, please check credentials" error while trying to establish connection from a GUI tool, but you're able to access mysql from vagrant terminal session, check to see which SSH Key File your vagrant instance is using. You can do that by using the following command from your vagrant folder on the host machine:

vagrant ssh-config

In case your vagrant instance is not using "~/.vagrant.d/insecure_private_key" as the SSH Key File, you'll need to specify the key your vagrant instance is using.

Making the local servers run faster

While running LMS and Studio locally, you may want to conditionally disable certain features in order to improve performance. To do so, create the files lms/envs/private.py and cms/envs/private.py and paste in this code:

DISABLE_DJANGO_TOOLBAR = True
DISABLE_CONTRACTS = True

if DISABLE_DJANGO_TOOLBAR:
    from .common import INSTALLED_APPS, MIDDLEWARE_CLASSES

    def tuple_without(source_tuple, exclusion_list):
        """Return new tuple excluding any entries in the exclusion list. Needed because tuples
        are immutable. Order preserved."""
        return tuple([i for i in source_tuple if i not in exclusion_list])

    INSTALLED_APPS = tuple_without(INSTALLED_APPS, ['debug_toolbar', 'debug_toolbar_mongo'])
    MIDDLEWARE_CLASSES = tuple_without(MIDDLEWARE_CLASSES, [
        'django_comment_client.utils.QueryCountDebugMiddleware',
        'debug_toolbar.middleware.DebugToolbarMiddleware',
    ])

    DEBUG_TOOLBAR_MONGO_STACKTRACES = False

if DISABLE_CONTRACTS:
    import contracts
    contracts.disable_all()

Disabling both the Django toolbar and contracts should speed up the your local LMS and Studio instances significantly.

IMPORTANT NOTE: this may cause devstack to behave strangely in certain scenarios, such as running acceptance tests. If something on your devstack is unexplainably not working, try setting both DISABLE_ flags to False.

Configuring Video Upload

Note: These steps will only get you through the upload stage. Video processing and YouTube deployment are not covered here.

Amazon Web Service (AWS)

To use the Video Upload feature in devstack, you'll need an AWS account.

  1. Login to AWS, and locate the S3 storage service.

  2. Create a bucket. Note the bucket name for configuration below.

  3. Under the bucket's Properties > Permissions, add this CORS configuration:

     <?xml version="1.0" encoding="UTF-8"?>
 <CORSConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
     <CORSRule>
         <AllowedOrigin>http://localhost:8001</AllowedOrigin>
         <AllowedMethod>GET</AllowedMethod>
         <AllowedMethod>POST</AllowedMethod>
         <AllowedMethod>PUT</AllowedMethod>
         <AllowedMethod>HEAD</AllowedMethod>
         <MaxAgeSeconds>3000</MaxAgeSeconds>
         <AllowedHeader>*</AllowedHeader>
     </CORSRule>
 </CORSConfiguration>

    Note: <AllowedOrigin>*</AllowedOrigin> doesn't work for most browsers, so you'll need to add an <AllowedOrigin> block for each URL you'll be using to POST to the S3 service.

  4. Locate the AWS IAM security service.

  5. Add a new User, and Create Access Key. Note the key and secret for configuration below.

  6. Add a new Group, with (at least) the AmazonS3FullAccess policy attached.

  7. Add your IAM user to the Group.

Configure

  1. Change these settings in /edx/app/edxapp/cms.envs.json

     "FEATURES": {
     ...
     "ENABLE_VIDEO_UPLOAD_PIPELINE": true,
     ...
 },

    and

     ...
 VIDEO_UPLOAD_PIPELINE = {
     "BUCKET": "<S3 bucket name>",
     "ROOT_PATH": "<bucket subfolder (optional)>"
 },
 ...

  2. Change these settings in /edx/app/edxapp/cms.auth.json

     ...
 "AWS_ACCESS_KEY_ID": "<IAM user key>",
 "AWS_SECRET_ACCESS_KEY": "<IAM user secret>",
 ...

You should now be able to see the Content > Video Upload option in your devstack CMS, and see the uploaded videos in your S3 bucket.

Other resources

There are a number of other useful documents on the edX platform wiki:

Add a custom footer
Add a custom sidebar
Clone this wiki locally