Overview

This PR introduces significant modifications to the GitHub Action for testing Open edX plugins in the Tutor environment. These changes are necessary for improving the effectiveness and reliability of our integration tests

Motivation for Changes

The initial implementation ran integration tests inside the LMS container of Tutor. This approach created several challenges:

• Networking Issues: Running tests inside the same container restricted the ability to make requests to external hosts, leading to unreliable test outcomes.
• Realistic Testing: To ensure that our integration tests reflect real-world interactions between our plugins and the Open edX platform, it is crucial to run them outside the Tutor container, maintaining a separation between the tests and the platform environment.

To address these issues, this PR modifies the workflow to run integration tests outside the Tutor container, improving the testing environment and results.

Workflow Steps and Their Necessity

1. Set Tutor Environment Variables:

   • Sets necessary environment variables such as LMS_HOST, CMS_HOST, TUTOR_ROOT, and TUTOR_PLUGINS_ROOT for proper Tutor configuration.

2. Install and Prepare Tutor:

   • Installs the specified version of Tutor or clones the nightly branch if tutor_version is set to "nightly".
   • Configures Tutor with the necessary host settings and launches it in non-interactive mode.

3. Configure Caddyfile and Open edX Settings:

   • Moved patches.yml into the action directory and copies it into the plugins directory during the workflow.
   • Enables the patches plugin to customize the Caddyfile and Open edX production settings. This configuration allows for the necessary routing for testing with multiple sites and security settings for our tests.

4. Add Mount for Plugin:

   • Mounts the plugin directory into the LMS and CMS containers, ensuring that the plugin code is accessible within the Tutor environment.

5. Restart Tutor Services:

   • Restarts Tutor services to apply the new mounts and configurations.

6. Install Plugin and Extra Requirements:

   • Installs the plugin in editable mode within the LMS container.
   • Installs any additional pip requirements specified in openedx_extra_pip_requirements in both LMS and CMS containers.

7. Run Migrations:

   • Executes database migrations for both the LMS and CMS to ensure the database schema is up-to-date.

8. Import Demo Course:

   • Imports the Open edX demo course for testing purposes.

9. Set DEMO_COURSE_ID Environment Variable:

• Sets the DEMO_COURSE_ID environment variable based on the Tutor version.
• Purpose: Allows tests to refer to the demo course ID when running integration tests.

10. Test Open edX Imports in Plugin (Optional):

• Provides a mechanism to verify that all imports from the Open edX platform in the plugin are working correctly.
• Executes the test function specified by openedx_imports_test_file_path and openedx_imports_test_function_name, ensuring that the plugin's imports from Open edX do not have any issues.

11. Load Initial Data for the Tests (Optional):

    • Loads initial data from a fixtures file into the LMS if fixtures_file is provided.

12. Curl Heartbeat:

    • Verifies that the LMS is running and responsive by checking the heartbeat endpoint.

13. Run Integration Tests:

    • Creates a virtual environment and runs the specified shell script to execute the integration tests outside the Tutor container, improving test reliability.

Personalization Options

The workflow includes several inputs that can be customized:

• app_name: The name of the application to test (e.g., eox-core).
• tutor_version: The version of Tutor to use, allowing for testing against specific versions or the nightly build.
• shell_file_to_run: The path to the shell script that contains the integration tests.
• openedx_extra_pip_requirements: Optional additional dependencies needed for testing.
• fixtures_file: Optional path to a JSON file containing initial data to load for the tests.
• openedx_imports_test_file_path: Optional path to the Python file containing the import test function.
• openedx_imports_test_function_name: Optional name of the function to execute in the import test file.

Testing the Workflow

To test this workflow:

1. Create a Branch:

   • Create a branch from any of the EOX plugins that includes integration tests. For example, you can use the branch bav/add-integration-tests in eox-core, which includes several integration tests.

2. Add Integration Test Script:

   • Include a shell script (e.g., integration.sh) in your plugin that installs dependencies and runs the tests.

     #!/bin/bash
     
     echo "Installing test requirements"
     make install-dev-dependencies
     
     echo "Running integration tests"
     pytest eox_core/api/v1/tests/integration/test_views.py

3. Configure the Workflow:

   • Update your workflow file to use the updated action and specify the required inputs.

     name: Tutor Integration Tests
     on: [pull_request]
     
     jobs:
       integration-test:
         name: Tutor Integration Tests
         runs-on: ubuntu-latest
         strategy:
           matrix:
             tutor_version: ['<18.0.0', '==17.0.3', 'nightly']
         steps:
           - name: Run Integration Tests
             uses: eduNEXT/integration-test-in-tutor@v1
             with:
               tutor_version: ${{ matrix.tutor_version }}
               app_name: 'eox-core'
               shell_file_to_run: 'scripts/integration.sh'
               openedx_extra_pip_requirements: 'eox-tenant'
               fixtures_file: 'fixtures/initial_data.json'
               openedx_imports_test_file_path: 'eox_core/edxapp_wrapper/tests/integration/test_backends.py'
               openedx_imports_test_function_name: 'test_current_settings_code_imports'

4. Verify the Tests Run Successfully:

   • Ensure that the integration tests execute successfully in the workflow runs, validating that the action works as intended. You can refer to this test branch for an example: refactor(ci): update workflow for integration tests eox-core#288

Additional Notes

• Supporting Tutor Nightly:

  • The action now supports running tests against the nightly version of Tutor, enabling early detection of compatibility issues with upcoming changes.

• Improved Documentation:

  • The README has been updated to reflect the latest changes, including detailed input descriptions, usage examples, and special notes on using the nightly version.