Remove boto, upgrade boto3 & botocore

[jmbowman]

edx-platform is still using versions of boto, boto3, and botocore that were originally pinned (by me) 4.5 years ago before the repo even had a constraints.txt file. boto isn't even supported anymore, and I recently stumbled into a performance problem that might just be fixed automatically by an upgrade. Please get us to reasonably current versions. Suggested sequence:

1. Move the constraints to constraints.txt so we actually notice they exist
2. Do any work needed to remove the dependency on boto
3. Start upgrading botocore and boto3 in parallel; try to go just far enough to upgrade to one potentially breaking change each time.

Finding the breaking changes may be difficult, as the boto3 and botocore changelogs are almost useless lists of daily API change notifications, and we don't really use boto3 in CI. But it looks like particular care may be needed for the upgrade to 1.9.0: https://boto3.amazonaws.com/v1/documentation/api/latest/guide/upgrading.html . And I suspect we don't use the vast majority of services covered in the changelog, so we may be ok just looking for changes related to s3 and maybe 1-2 other AWS services. Writing a doc on how to decide if it's safe to upgrade to a given boto3 or botocore release is probably a good idea, if you figure that out from this task.

Tasks

Beta Give feedback

1. Change boto to boto3 for import-export related code. #31771
   +0
   
2. change boto to boto3 for profile-image #31772
   +0
   
3. GRADES_DOWNLOAD storage from boto to boto3 #31774
   +0
4. VIDEO_IMAGE_SETTINGS backend change from boto to boto3 #31775
   +0
5. VIDEO_TRANSCRIPTS_SETTINGS change backend from boto to boto3 #31776
   +0
6. BLOCK_STRUCTURES_SETTINGS backend change from boto to boto3 #31777
   +0
7. Replace boto-based exceptions from bulk_email #31779
   +0
8. remove all boto related imports from edx-app #31780
   +0
9. change default storage from boto to boto3 #31778
   +0
10. remove boto from base.in #31781
    +0
11. Use boto3 instead of boto in video uploading #31914
    +0
12. warnings: explore this warning and do appropriate steps #31955
    +0
    
13. upgrade django-storages to 1.10.1 needs few settings changes #32376
    +0

[awais786]

edxapp is using django-storage = 1.8

The legacy S3BotoStorage backend was removed in version 1.9. This is the PR where it removed jschneier/django-storages#825

We can move from s3boto to s3boto3 using exiting django-storage = 1.8.
One of the main thing is handling exceptions.

s3boto is using these exception = https://github.com/boto/boto/blob/2.39.0/boto/exception.pys
3boto3 using these  exception =  https://github.com/boto/botocore/blob/1.8.17/botocore/exceptions.py

Plan:

1. We will add both exceptions simultaneously. Then replace all codes occurrences
   from storages.backends.s3boto import S3BotoStorage with from storages.backends.s3boto3 import S3Boto3Storage and update the exceptions also and handle both boto and botocore exceptions. ( merge and deploy this )
2. Next step is create PR in internal and deploy it.
3. Remove all boto usages and exception and deploy it.
4. If every thing goes as per plan we can upgrade django-storage to 1.9 as per documentation.

[iamsobanjaved]

There are multiple (6+) pipelines to upgrade from the boto-based backend to the boto3-based backend. We can try to upgrade pipelines in an incremental approach, try merging 2 pipelines at once so we can verify with ease on stage before pushing our changes to prod.

[awais786]

https://django-storages.readthedocs.io/en/latest/backends/amazon-S3.html#migrating-from-boto-to-boto3

[awais786]

We have successfully upgraded the django-storages to 1.9.1. Now only videos module left with boto import.

For further upgrade to 1.10.1 https://github.com/jschneier/django-storages/blob/master/CHANGELOG.rst#1101-2020-09-13
we need to change few settings variables and also sets the boto3 warning.

[awais786]

No usage found

• Course discovery
• Credentials
• Registrar
• ecommerce
• edx-enterprise

[UsamaSadiq]

Usage found in

• open-edx-stats issue create Upgrade from boto to boto3 openedx-unsupported/openedxstats#208 (@awais786 completed )
• Terraform https://github.com/edx/terraform/issues/5714. ( @nadeemshahzad completed )
• https://github.com/edx/analytics-tasks/ its archived repo. ( no need to do any thing )

These following three repos are interlinked and using python2.7. So not possible to update with boto3.

• 1 edx-analytics-pipeline issue created Upgrade boto to boto3 openedx-unsupported/edx-analytics-pipeline#891

• 2 edx-analytics-configuration issue created(Upgrade boto to boto3 openedx-unsupported/edx-analytics-configuration#114)

• 3 https://github.com/edx/luigi ( can't create issue here due to permissions )

• openedx/edx-analytics-data-api issue created here for owning team.

• edx-platform ( videos module ) ( Its a small change from code but it needs domain knowledge so blocked on owning team )

• Some packages and xblocks bringing boto. I have released new version so boto will disappear from requirements.

• https://github.com/edx/api-manager/blob/master/requirements/base.txt#L8 ( here it is coming from google-compute-engine It has no release since Jan 25, 2019 ( not possible to upgrade boto )

[awais786]

@jmbowman so far found usage here. I have created issues in these repos except luigi.

[jmbowman]

There's a "boto usage" filter view in the Repo Health Dashboard spreadsheet that shows just repos with "boto==" in the dependencies.pypi_all.listfield; it currently includes 22 repos. Some of them bring it in due to pinning to a stale version of moto, such as https://github.com/edx/video-encode-manager/blob/master/requirements/constraints.txt#L7-L8 .

Also, there seems to be a bug in the implementation of the requires.boto check. It's meant to be an easier way to find these repos, but it incorrectly has False for most of them.

[awais786]

Lots of entries are coming from xblock-sdk. I have already fixed that just need to merge and release version.

[awais786]

https://github.com/openedx/openedxstats/pull/209/files PR created for openedxstats
openedx-unsupported/openedxstats#211 using moto for tests.

[awais786]

We have successfully upgraded the django-storages to 1.10.1. Now next target is 1.11.1

[awais786]

django-storages upgrade to latest version in edx-platform. I think we can close this PR now. Boto still in use in videos module. I have already informed owning team.
They have following ticket https://2u-internal.atlassian.net/browse/TNL-10698