Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat!: Update from boto to boto3 storage backend. #31759

Merged
merged 10 commits into from
Feb 21, 2023

Conversation

awais786
Copy link
Contributor

@awais786 awais786 commented Feb 14, 2023

Update from boto to boto3 storage backend.
It will come under multiple prs to avoid huge change set.

For details review this issue

This PR will make change in

  1. USER_TASKS_ARTIFACT_STORAGE
  2. COURSE_IMPORT_EXPORT_STORAGE
  3. COURSE_METADATA_EXPORT_STORAGE

Sandbox internal PR

@awais786
Copy link
Contributor Author

awais786 commented Feb 14, 2023

Last attempt gave an error on prod. How to reproduce error
python manage.py cms export_course_metadata_for_all_courses.

Faced errors on edxapp-cms-worker related to Course Metadata failure with following stack trace

[2023-02-09 13:29:44,028: ERROR/ForkPoolWorker-15] Task cms.djangoapps.export_course_metadata.tasks.export_course_metadata_task[1978fb43-09a0-4973-9cd4-8635aca89aec] raised unexpected: TypeError('Unicode-objects must be encoded before hashing')
Traceback (most recent call last):
  File "/edx/app/edxapp/venvs/edxapp/lib/python3.8/site-packages/celery/app/trace.py", line 451, in trace_task
    R = retval = fun(*args, **kwargs)
  File "/edx/app/edxapp/venvs/edxapp/lib/python3.8/site-packages/newrelic/hooks/application_celery.py", line 99, in wrapper
    return wrapped(*args, **kwargs)
  File "/edx/app/edxapp/venvs/edxapp/lib/python3.8/site-packages/celery/app/trace.py", line 734, in __protected_call__
    return self.run(*args, **kwargs)
  File "/edx/app/edxapp/edx-platform/cms/djangoapps/export_course_metadata/tasks.py", line 31, in export_course_metadata_task
    course_metadata_export_storage.save(f'course_metadata_export/{course_key}.json', highlights_content)
  File "/edx/app/edxapp/venvs/edxapp/lib/python3.8/site-packages/django/core/files/storage.py", line 54, in save
    name = self._save(name, content)
  File "/edx/app/edxapp/venvs/edxapp/lib/python3.8/site-packages/storages/backends/s3boto3.py", line 495, in _save
    self._save_content(obj, content, parameters=parameters)
  File "/edx/app/edxapp/venvs/edxapp/lib/python3.8/site-packages/storages/backends/s3boto3.py", line 510, in _save_content
    obj.upload_fileobj(content, ExtraArgs=put_parameters)
  File "/edx/app/edxapp/venvs/edxapp/lib/python3.8/site-packages/boto3/s3/inject.py", line 511, in object_upload_fileobj
    return self.meta.client.upload_fileobj(
  File "/edx/app/edxapp/venvs/edxapp/lib/python3.8/site-packages/boto3/s3/inject.py", line 431, in upload_fileobj
    return future.result()
  File "/edx/app/edxapp/venvs/edxapp/lib/python3.8/site-packages/s3transfer/futures.py", line 73, in result
    return self._coordinator.result()
  File "/edx/app/edxapp/venvs/edxapp/lib/python3.8/site-packages/s3transfer/futures.py", line 233, in result
    raise self._exception
  File "/edx/app/edxapp/venvs/edxapp/lib/python3.8/site-packages/s3transfer/tasks.py", line 126, in __call__
    return self._execute_main(kwargs)
  File "/edx/app/edxapp/venvs/edxapp/lib/python3.8/site-packages/s3transfer/tasks.py", line 150, in _execute_main
    return_value = self._main(**kwargs)
  File "/edx/app/edxapp/venvs/edxapp/lib/python3.8/site-packages/s3transfer/upload.py", line 692, in _main
    client.put_object(Bucket=bucket, Key=key, Body=body, **extra_args)
  File "/edx/app/edxapp/venvs/edxapp/lib/python3.8/site-packages/botocore/client.py", line 317, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/edx/app/edxapp/venvs/edxapp/lib/python3.8/site-packages/botocore/client.py", line 591, in _make_api_call
    handler, event_response = self.meta.events.emit_until_response(
  File "/edx/app/edxapp/venvs/edxapp/lib/python3.8/site-packages/botocore/hooks.py", line 242, in emit_until_response
    responses = self._emit(event_name, kwargs, stop_on_response=True)
  File "/edx/app/edxapp/venvs/edxapp/lib/python3.8/site-packages/botocore/hooks.py", line 210, in _emit
    response = handler(**kwargs)
  File "/edx/app/edxapp/venvs/edxapp/lib/python3.8/site-packages/botocore/handlers.py", line 209, in conditionally_calculate_md5
    calculate_md5(params, **kwargs)
  File "/edx/app/edxapp/venvs/edxapp/lib/python3.8/site-packages/botocore/handlers.py", line 187, in calculate_md5
    binary_md5 = _calculate_md5_from_file(body)
  File "/edx/app/edxapp/venvs/edxapp/lib/python3.8/site-packages/botocore/handlers.py", line 201, in _calculate_md5_from_file
    md5.update(chunk)
TypeError: Unicode-objects must be encoded before hashing

Boto3 PR #31282 which caused this issue.

With boto encoding is also happening down the stream but in boto3 need to do this explicitly
https://github.com/boto/boto/blob/614e844408c27cbdeebc1325030820e1ce78aaf0/boto/s3/key.py#L863

@awais786 awais786 marked this pull request as ready for review February 14, 2023 11:58
@@ -27,5 +27,5 @@ def export_course_metadata_task(self, course_key_string): # pylint: disable=unu
"""
course_key = CourseKey.from_string(course_key_string)
highlights = get_all_course_highlights(course_key)
highlights_content = ContentFile(json.dumps({'highlights': highlights}))
highlights_content = ContentFile(json.dumps({'highlights': highlights}).encode('utf-8'))
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Huh, this is apparently an open bug in django-storages with a PR to fix it that's been sitting around unmerged for almost 3 years for no particularly good reason: jschneier/django-storages#911

@UsamaSadiq UsamaSadiq merged commit 31002ab into master Feb 21, 2023
@UsamaSadiq UsamaSadiq deleted the update-boto-to-botos3 branch February 21, 2023 09:26
UsamaSadiq added a commit that referenced this pull request Feb 21, 2023
@edx-pipeline-bot
Copy link
Contributor

EdX Release Notice: This PR has been deployed to the staging environment in preparation for a release to production.

@edx-pipeline-bot
Copy link
Contributor

EdX Release Notice: This PR has been deployed to the production environment.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants