feat: mark optional completions

[DanielVZ96]

Description: Adds support for optional completions by marking completions as optional, and adding them to the mean accordingly.

JIRA: https://tasks.opencraft.com/browse/BB-8586

Dependencies: TODO: add edx-platform PR

Installation instructions: List any non-trivial installation
instructions.

Testing instructions:

1.Install the app from this branch

• Modify the devstack docker-compose to install this branch locally: 'source /edx/app/edxapp/edxapp_env && pip install -e /edx/src/openedx-completion-aggregator && while true; do python /edx/app/edxapp/edx-platform/manage.py lms runserver 0.0.0.0:18000 --settings devstack_docker; sleep 2; done'
• Restart the containers
• Add the completion_aggregator app to the Advanced Module List inside Advanced Settings in CMS

2. Collect the aggregations running the following commands in the lms container:

• ./manage.py lms reaggregate_course
• ./manage.py lms run_aggregator_service

3. Request the aggregations at this url http://localhost:18000/completion-aggregator/v1/course/course-v1:edX+DemoX+Demo_Course/?requested_fields=chapter,vertical

Reviewers:

• tag reviewer

Merge checklist:

• All reviewers approved
• CI build is green
• Version bumped
• Changelog record added
• Documentation updated (not only docstrings)
• Commits are squashed
• PR author is listed in AUTHORS

Post merge:

• Create a tag
• Check new version is pushed to PyPI after tag-triggered build is
  finished.
• Delete working branch (if not needed anymore)

Author concerns: Ideally optional completions should be included in an additional field, but due to time constraints I wasn't able to implement that here.

Private-ref: BB-8586

[DanielVZ96]

@Agrendalath codecov actually improved in most of files, except one where it was lowered due to an increase of lines it semes: https://app.codecov.io/gh/open-craft/openedx-completion-aggregator/pull/187

[Agrendalath]

@DanielVZ96, just one initial question: what are this change's performance implications? The crucial aspect of the completion aggregator is the ability to process thousands of completions per second. Can we make a quick performance comparison - before and after this change?

Side note:

'source /edx/app/edxapp/edxapp_env && pip install -e /edx/src/openedx-completion-aggregator && while true; do python /edx/app/edxapp/edx-platform/manage.py lms runserver 0.0.0.0:18000 --settings devstack_docker; sleep 2; done'

You can add the following to options.local.mk in your devstack directory:

# Restart both containers
edx-restart: lms-restart studio-restart

# This allows you to install an XBlock in both LMS and Studio using
# make install-xblock XBLOCK=problem-builder
install-xblock:
        for c in lms studio ; do \
                docker exec -it edx.$(COMPOSE_PROJECT_NAME).$$c bash -c \
                'cd /edx/src/$(XBLOCK) && /edx/app/edxapp/venvs/edxapp/bin/pip install -e .' ;   \
        done
        make edx-restart

[DanielVZ96]

@DanielVZ96, just one initial question: what are this change's performance implications? The crucial aspect of the completion aggregator is the ability to process thousands of completions per second. Can we make a quick performance comparison - before and after this change?

Side note:

'source /edx/app/edxapp/edxapp_env && pip install -e /edx/src/openedx-completion-aggregator && while true; do python /edx/app/edxapp/edx-platform/manage.py lms runserver 0.0.0.0:18000 --settings devstack_docker; sleep 2; done'

You can add the following to options.local.mk in your devstack directory:

# Restart both containers
edx-restart: lms-restart studio-restart

# This allows you to install an XBlock in both LMS and Studio using
# make install-xblock XBLOCK=problem-builder
install-xblock:
        for c in lms studio ; do \
                docker exec -it edx.$(COMPOSE_PROJECT_NAME).$$c bash -c \
                'cd /edx/src/$(XBLOCK) && /edx/app/edxapp/venvs/edxapp/bin/pip install -e .' ;   \
        done
        make edx-restart

will test performance tomorrow

[DanielVZ96]

@DanielVZ96, just one initial question: what are this change's performance implications? The crucial aspect of the completion aggregator is the ability to process thousands of completions per second. Can we make a quick performance comparison - before and after this change?

Side note:

'source /edx/app/edxapp/edxapp_env && pip install -e /edx/src/openedx-completion-aggregator && while true; do python /edx/app/edxapp/edx-platform/manage.py lms runserver 0.0.0.0:18000 --settings devstack_docker; sleep 2; done'

You can add the following to options.local.mk in your devstack directory:

# Restart both containers
edx-restart: lms-restart studio-restart

# This allows you to install an XBlock in both LMS and Studio using
# make install-xblock XBLOCK=problem-builder
install-xblock:
        for c in lms studio ; do \
                docker exec -it edx.$(COMPOSE_PROJECT_NAME).$$c bash -c \
                'cd /edx/src/$(XBLOCK) && /edx/app/edxapp/venvs/edxapp/bin/pip install -e .' ;   \
        done
        make edx-restart

Here's some timings:

• With changes:
  
  

• Without changes:
  
  
  

So in both:

• System CPU time varies from 1 ms to 20 ms
• Request time incremented from 168 ms to 458 ms

Based on this I can't say there's a clear performance penalty. Also as a side-note: with my current courses it's having to generate 233 completion aggregations.