Too many console warnings

[jmbowman]

Many warnings routinely get logged when doing console operations in edx-platform. This is especially visible when provisioning a devstack installation (although the problem isn't limited to devstack). These warnings often indicate latent problems with the code that will cause headaches when attempting future upgrades, they distract developers when looking for the root cause of a problem, and some cause a disproportionate amount of log clutter because they reappear every time a management command is called. We should clean up as many of these as practical to improve the developer experience when working with the repository.

A/C:

• Enumerate all the warnings that appear when provisioning lms & cms in devstack (ideally also noting the number of times each one appears).
  • Logs from recent provisioning runs can be found at https://github.com/openedx/devstack/actions/workflows/provisioning-tests.yml .
  • While we also want to fix warnings that show up for other services, the scope of this issues is the ones generated from edx-platform.
• Create a separate GitHub Issue in this repository to resolve each warning. They should get the "maintenance" and "help wanted" labels.
• Create a tasklist in this issue collecting all the created issues.

Tasks

Beta Give feedback

1. Fix/Ignore: [Bourbon] [Deprecation] rem is deprecated and will be removed in 5.0.0. #33571
   +0
   
2. fix: Django now detects this configuration automatically. You can remove default_app_config. #33572
   +0
3. Switch off of deprecated pkg_resources library XBlock#641
   maintenance +1
4. noisy warning: Flag "[name]" accessed without a request, which is likely in the context of a celery task. edx-toggles#320
   +0
5. fix/ignore: Replace Bourbon mixins with Autoprefixer warnings #33573
   +0
6. Remove xblock-utils dependency open-craft/xblock-poll#114
   +0
7. fix: edX Studio doesn't support third-party handler urls for XBlock <class 'xblock.internal.VideoBlockWithMixins'> #33581
   +0
8. Fix: DeprecationWarning: XBlocks should not instantiate their own field_data store during parse_xml() #33582
   +0
9. fix: please use ModelBase.available_objects to continue excluding soft-deleted objects. edx-enterprise#1927
   +0
10. Fix: Using / for division outside of calc() is deprecated and will be removed in Dart Sass 2.0.0 #33583
    +0
11. fix python deprecation warnings: collections.abc , HTTPResponse.getheaders(), add_custom_parameter #33584
    +0
12. fix: Flags not at the start of the expression #33585
    +0
13. fix: Cannot inherit metadata to {block_type} block with KVS #33586
    +0
14. Fix: runtime.user_id is deprecated. Use block.scope_ids.user_id or the user service instead. #33587
    +0
15. Fix: XBlock <class 'xblock.internal.DiscussionXBlockWithMixins'> does not contain field url_name #33588
    +0
16. fix/ignore: DeprecationWarning: Deprecated call to pkg_resources.declare_namespace. #33592
    +0
17. Remove boto, upgrade boto3 & botocore #31175
    13 of 13
    +0
    
18. Fix: Use of runtime.render_template is deprecated. Use MakoService.render_template or a JavaScript-based template instead. #33589
    +0
19. fix DeprecationWarning: defusedxml.lxml is no longer supported and will be removed in a future release. from defusedxml.lxml import DTDForbidden, EntitiesForbidden, NotSupportedError #33621
    +0
20. fix RemovedInDjango40Warning: The DEFAULT_HASHING_ALGORITHM transitional setting is deprecated. #33622
    +0
21. RemovedInDjango51Warning: django.core.files.storage.get_storage_class is deprecated in favor of using django.core.files.storage.storages. #35023
    +0
22. DeprecationWarning: .text & .listdir are deprecated #35044
    good first issue help wanted +2
    
23. DeprecationWarning: The 'type' attribute is deprecated in Shapely. #35176
    +0
24. DeprecationWarning: assertDictContainsSubset is deprecated #35175
    +0

[DanielVZ96]

@jmbowman I made some rough counts for all warnings from the provision run. There are plenty that are not repeated, almost all of them being deprecation warnings, with a big majority being deprecated js packages.

Given their frequencies, I'd summarize them in the following issues:

1. Fix/Ignore: [Bourbon] [Deprecation] rem is deprecated and will be removed in 5.0.0.

2. [From 2 external dependencies, one can be updated, the other needs to be ignored] Fix: Django now detects this configuration automatically. You can remove default_app_config.

3. Fix: Flag '...' accessed without a request, which is likely in the context of a celery task.

4. Replace Bourbon mixins with Autoprefixer

5. Fix XBlock warnings

   • [xblock-google-drive] DeprecatedPackageWarning: Please use import xblock.utils.publish_event instead of xblockutils.publish_event because the 'xblock-utils' package has been deprecated and migrated to within 'xblock' package.
   • edX Studio doesn't support third-party handler urls for XBlock <class 'xblock.internal.VideoBlockWithMixins'>
   • [xblock, edx-platform] DeprecationWarning: XBlocks should not instantiate their own field_data store during parse_xml()

6. [no idea what repo it's from] Fix: ModelBase.objects model manager will include soft-deleted objects in an upcoming release; please use ModelBase.available_objects to continue excluding soft-deleted objects.

7. Fix: Using / for division outside of calc() is deprecated and will be removed in Dart Sass 2.0.0

8. Fix Python deprecation warnings: collections.abc [this repo, need to update libsass to 0.15.0] , imp module [this repo, need replace boto with boto3 in a few lines], HTTPResponse.getheaders(), add_custom_parameter

9. Fix: Flags not at the start of the expression

10. Update deprecated packages in package.json

Note: I'll check if an issue is related to this repo before I create it, so the task list length may end up being less than the list above

Counts

count	warning extract
165	rem is deprecated and will be removed
129	Django now detects this configuration automatically. You can remove default_app_config.
120	DeprecationWarning: Deprecated call to `pkg_resources.declare_namespace'
35	accessed without a request, which is likely in the context of a celery task.
31	is deprecated and will be removed in 5.0.0. We suggest using an automated prefixing tool, like Autoprefixer.
31	DeprecationWarning: 'etree' is deprecated. Use 'xml.etree.ElementTree' instead.
30	XBlock <class 'xblock.internal.DiscussionXBlockWithMixins'> does not contain field url_name
24	ModelBase.objects model manager will include soft-deleted objects in an upcoming release; please use ModelBase.available_objects to continue excluding soft-deleted objects.
22	transition is deprecated and will be removed in 5.0.0. We suggest using an automated prefixing tool, like Autoprefixer.
18	because the 'xblock-utils' package has been deprecated and migrated to within 'xblock' package.
17	Using / for division outside of calc() is deprecated and will be removed in Dart Sass 2.0.0
12	DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated since Python 3.3, and in 3.10 it will stop working
6	please upgrade to graceful-fs 4 for compatibility with current and future versions of Node.js
6	Please update to minimatch 3.0.2 or higher to avoid a RegExp DoS issue
6	deprecated [email protected]: Browserslist 2 could fail on reading Browserslist >3.0 config used in other tools.
5	the imp module is deprecated in favour of importlib;
5	edX Studio doesn't support third-party handler urls for XBlock <class 'xblock.internal.VideoBlockWithMixins'>
4	Using / for division is deprecated and will be removed in Dart Sass 2.0.0.
4	deprecated [email protected]
3	DeprecationWarning: XBlocks should not instantiate their own field_data store during parse_xml()
2	skipping integrity check for git dependency
2	Mac users should run this from both the lms and studio shells
2	Falling back to fake video id due to Course Blocks API failure...
2	deprecated [email protected]: core-js@<3.4 is no longer maintained and not recommended for usage due to the number of issues. Because of the V8 engine whims, feature detection in old core-js versions could cause a slowdown up to 100x even if nothing is polyfilled. Please, upgrade your dependencies to the actual version of core-js.
2	Cannot inherit metadata to * block with KVS
1	Use of runtime.render_template is deprecated. Use MakoService.render_template or a JavaScript-based template instead.
1	runtime.user_id is deprecated. Use block.scope_ids.user_id or the user service instead.
1	ANTIALIAS is deprecated and will be removed in Pillow 10

Other warnings

Python:

1. Flags not at the start of the expression.
2. HTTPResponse.getheaders() is deprecated in urllib3.
3. add_custom_parameter API in New Relic is deprecated.
4. Transcript "subs_3_yD_cEKoCk.srt.sjson" for video "5e5d9190-9985-43ac-b859-0063968f88f2" is not found.

JavaScript:

• babel-eslint is now @babel/eslint-parser.
• @babel/polyfill is deprecated.
• bfj-node4 has been replaced by bfj.
• boom, cryptiles, hoek, sntp are deprecated.
• bower has a security vulnerability.
• chokidar v2 is deprecated.
• circular-json has been succeeded by flatted.
• extract-text-webpack-plugin is deprecated.
• formatio is unmaintained.
• har-validator is no longer supported.
• hawk has moved to @hapi/hawk.
• istanbul is no longer maintained.
• jscs has merged with ESLint.
• messageformat has been renamed.
• mkdirp older versions are deprecated.
• natives is deprecated.
• npmconf has been reintegrated into npm.
• phantomjs-prebuilt is deprecated.
• querystring API is considered Legacy.
• queue-async has been renamed.
• request is deprecated.
• resolve-url is deprecated.
• samsam is deprecated.
• source-map-resolve and source-map-url are deprecated.
• svgo v0.7.2 is deprecated.
• topojson is deprecated.
• tough-cookie is deprecated.
• urix is deprecated.
• sphinxcontrib-openapi 0.8.1 is deprecated.

[DanielVZ96]

I can't create the tasklist myself, but here I'll keep track of the Issues I'm creating.

Issue(s)	Warning
#33571	Fix/Ignore: [Bourbon] [Deprecation] rem is deprecated and will be removed in 5.0.0.
#33572	Fix: Django now detects this configuration automatically. You can remove default_app_config.
openedx/XBlock#641	DeprecationWarning: Deprecated call to `pkg_resources.declare_namespace'
openedx/edx-toggles#320	Fix: Flag '...' accessed without a request, which is likely in the context of a celery task.
#33573	Fix/Ignore: Replace Bourbon mixins with Autoprefixer warnings
open-craft/xblock-poll#114,	- DeprecatedPackageWarning: Please use import xblock.utils.publish_event instead of xblockutils.publish_event because the 'xblock-utils' package has been deprecated and migrated to within 'xblock' package.
#33581	- edX Studio doesn't support third-party handler urls for XBlock <class 'xblock.internal.VideoBlockWithMixins'>
#33582	- DeprecationWarning: XBlocks should not instantiate their own field_data store during parse_xml()
openedx/edx-enterprise#1927	Fix: ModelBase.objects model manager will include soft-deleted objects in an upcoming release; please use ModelBase.available_objects to continue excluding soft-deleted objects.
#33583	Fix: Using / for division outside of calc() is deprecated and will be removed in Dart Sass 2.0.0
#33584	Fix Python deprecation warnings: collections.abc , HTTPResponse.getheaders(), add_custom_parameter
#33585	Fix: Flags not at the start of the expression
#33586	Cannot inherit metadata to * block with KVS
#33587	runtime.user_id is deprecated. Use block.scope_ids.user_id or the user service instead.
#33588	XBlock <class 'xblock.internal.DiscussionXBlockWithMixins'> does not contain field url_name
#31175	the imp module is deprecated in favour of importlib;
#33589	Use of runtime.render_template is deprecated. Use MakoService.render_template or a JavaScript-based template instead.
#33621	DeprecationWarning: defusedxml.lxml is no longer supported and will be removed in a future release.
#33622	RemovedInDjango40Warning: The DEFAULT_HASHING_ALGORITHM transitional setting is deprecated. Support for it and tokens, cookies, sessions, and signatures that use SHA-1 hashing algorithm will be removed in Django 4.0.
todo	Update deprecated packages in package.json

[Agrendalath]

@DanielVZ96,

Fix: ModelBase.objects model manager will include soft-deleted objects in an upcoming release; please use ModelBase.available_objects to continue excluding soft-deleted objects.

I suppose it comes from edx-enterprise. It can be triggered by running tox -e py38-django32-celery53 in this repo. Link to the CI run.

What about these?

the imp module is deprecated in favour of importlib;

skipping integrity check for git dependency

Cannot inherit metadata to * block with KVS

runtime.user_id is deprecated. Use block.scope_ids.user_id or the user service instead.

Use of runtime.render_template is deprecated. Use MakoService.render_template or a JavaScript-based template instead.

I believe the last one comes from here. Removing it properly will be a significant task.

[DanielVZ96]

the imp module is deprecated in favour of importlib;

(i think i accidentally deleted it in the table) it comes from using boto, probably here, and would need to update that code so it uses boto3

I believe the last one comes from here. Removing it properly will be a significant task.

Then can i just create the issue for that one?

skipping integrity check for git dependency

This should be safe to ignore

[Agrendalath]

@DanielVZ96,

(i think i accidentally deleted it in the table) it comes from using boto, probably here, and would need to update that code so it uses boto3

We can link #31175. Looks like it should be handled by 2U.

Then can i just create the issue for that one?

Yep.

[Agrendalath]

@DanielVZ96, I'm seeing two more exceptions we didn't mention here:

2023-10-30 11:13:23,027 WARNING  /edx/app/edxapp/edx-platform/openedx/core/lib/safe_lxml/xmlparser.py:11: DeprecationWarning: defusedxml.lxml is no longer supported and will be removed in a future release.
  from defusedxml.lxml import DTDForbidden, EntitiesForbidden, NotSupportedError

2023-10-30 11:13:23,410 WARNING  /edx/app/edxapp/venvs/edxapp/lib/python3.8/site-packages/django/conf/__init__.py:198: RemovedInDjango40Warning: The DEFAULT_HASHING_ALGORITHM transitional setting is deprecated. Support for it and tokens, cookies, sessions, and signatures that use SHA-1 hashing algorithm will be removed in Django 4.0.
  warnings.warn(DEFAULT_HASHING_ALGORITHM_DEPRECATED_MSG, RemovedInDjango40Warning)

These are coming directly from the edx-platform, so it would be good to create issues for them.

[jmbowman]

I finally caught up on my GitHub notification emails...it's great to see all the progress on this! The DEFAULT_HASHING_ALGORITHM one should go away in a week or two once we wrap up the Django 4.2 upgrade. Anything else I can do to help move things along here?

[devkhan]

What's the current status of this? I can help if needed

[Agrendalath]

@devkhan, please feel free to pick any open task from the list in the first comment.

[abdullahQureshee]

DeprecationWarning: 'imghdr' is deprecated and slated for removal in Python 3.13

warning count: 41.

path: /openedx/venv/lib/python3.11/site-packages/pgpy/constants.py:5

Warning Source: pgpy only

Issue already created on pgpy repo. Pull request already submitted.

@farhan Let me know if more details are required.