From 89ec7102ef34a238c029d284b916ecd9e2cbf93f Mon Sep 17 00:00:00 2001 From: Maxime Beauchemin Date: Mon, 2 Dec 2024 12:50:03 -0800 Subject: [PATCH 1/3] chore: deprecarte pylint in favor of `ruff` This could be slightly controversial, but submitted as a DRAFT for discussion. The case for pylint deprecation: - mostly covered by ruff: what matters most in pylint is already re-implemented - pylint is SLOW, ruff is a million times faster - pre-commit / pylint is one of our longest, most expensive CI step - rules are currently scattered in different places and can conflict at times Very relevant is this link that explains what pylint features are covered or not by ruff: https://github.com/astral-sh/ruff/issues/970 If we agree on the direction, the plan is to follow up and enable all the rules, apply `ruff check --fix` and `ruff check --add-no-qa` Simplify. --- .pre-commit-config.yaml | 14 - .pylintrc | 380 ------------------------- Makefile | 3 - docs/docs/contributing/development.mdx | 32 +-- docs/docs/contributing/howtos.mdx | 13 +- pyproject.toml | 19 +- requirements/base.txt | 4 +- requirements/development.txt | 15 +- 8 files changed, 22 insertions(+), 458 deletions(-) delete mode 100644 .pylintrc diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml index 07f77eb4f801b..8fa32adf77313 100644 --- a/.pre-commit-config.yaml +++ b/.pre-commit-config.yaml @@ -80,17 +80,3 @@ repos: - id: ruff args: [ --fix ] - id: ruff-format - - repo: local - hooks: - - id: pylint - name: pylint - entry: pylint - language: system - types: [python] - exclude: ^(tests/|superset/migrations/|scripts/|RELEASING/|docker/) - args: - [ - "-rn", # Only display messages - "-sn", # Don't display the score - "--rcfile=.pylintrc", - ] diff --git a/.pylintrc b/.pylintrc deleted file mode 100644 index e11376fc1053b..0000000000000 --- a/.pylintrc +++ /dev/null @@ -1,380 +0,0 @@ -# -# Licensed to the Apache Software Foundation (ASF) under one or more -# contributor license agreements. See the NOTICE file distributed with -# this work for additional information regarding copyright ownership. -# The ASF licenses this file to You under the Apache License, Version 2.0 -# (the "License"); you may not use this file except in compliance with -# the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# -[MASTER] - -# Specify a configuration file. -#rcfile= - -# Python code to execute, usually for sys.path manipulation such as -# pygtk.require(). -#init-hook= - -# Add files or directories to the blacklist. They should be base names, not -# paths. -ignore=CVS,migrations - -# Add files or directories matching the regex patterns to the blacklist. The -# regex matches against base names, not paths. -ignore-patterns= - -# Pickle collected data for later comparisons. -persistent=yes - -# List of plugins (as comma separated values of python modules names) to load, -# usually to register additional checkers. -load-plugins=superset.extensions.pylint - -# Use multiple processes to speed up Pylint. -jobs=2 - -# Allow loading of arbitrary C extensions. Extensions are imported into the -# active Python interpreter and may run arbitrary code. -unsafe-load-any-extension=no - -# A comma-separated list of package or module names from where C extensions may -# be loaded. Extensions are loading into the active Python interpreter and may -# run arbitrary code -extension-pkg-whitelist=pyarrow - - -[MESSAGES CONTROL] - -# Only show warnings with the listed confidence levels. Leave empty to show -# all. Valid levels: HIGH, INFERENCE, INFERENCE_FAILURE, UNDEFINED -confidence= - -# Enable the message, report, category or checker with the given id(s). You can -# either give multiple identifier separated by comma (,) or put this option -# multiple time (only on the command line, not in the configuration file where -# it should appear only once). See also the "--disable" option for examples. -enable= - useless-suppression, - -# Disable the message, report, category or checker with the given id(s). You -# can either give multiple identifiers separated by comma (,) or put this -# option multiple times (only on the command line, not in the configuration -# file where it should appear only once).You can also use "--disable=all" to -# disable everything first and then reenable specific checks. For example, if -# you want to run only the similarities checker, you can use "--disable=all -# --enable=similarities". If you want to run only the classes checker, but have -# no Warning level messages displayed, use"--disable=all --enable=classes -# --disable=W" -disable= - cyclic-import, # re-enable once this no longer raises false positives - missing-docstring, - duplicate-code, - line-too-long, - unspecified-encoding, - too-many-instance-attributes # re-enable once this no longer raises false positives - -[REPORTS] - -# Set the output format. Available formats are text, parseable, colorized, msvs -# (visual studio) and html. You can also give a reporter class, eg -# mypackage.mymodule.MyReporterClass. -output-format=text - -# Tells whether to display a full report or only the messages -reports=yes - -# Python expression which should return a note less than 10 (10 is the highest -# note). You have access to the variables errors warning, statement which -# respectively contain the number of errors / warnings messages and the total -# number of statements analyzed. This is used by the global evaluation report -# (RP0004). -evaluation=10.0 - ((float(5 * error + warning + refactor + convention) / statement) * 10) - -# Template used to display messages. This is a python new-style format string -# used to format the message information. See doc for all details -#msg-template= - - -[BASIC] - -# Good variable names which should always be accepted, separated by a comma -good-names=_,df,ex,f,i,id,j,k,l,o,pk,Run,ts,v,x,y - -# Bad variable names which should always be refused, separated by a comma -bad-names=bar,baz,db,fd,foo,sesh,session,tata,toto,tutu - -# Colon-delimited sets of names that determine each other's naming style when -# the name regexes allow several styles. -name-group= - -# Include a hint for the correct naming format with invalid-name -include-naming-hint=no - -# List of decorators that produce properties, such as abc.abstractproperty. Add -# to this list to register other decorators that produce valid properties. -property-classes= - abc.abstractproperty, - sqlalchemy.ext.hybrid.hybrid_property - -# Regular expression matching correct argument names -argument-rgx=[a-z_][a-z0-9_]{2,30}$ - -# Regular expression matching correct method names -method-rgx=[a-z_][a-z0-9_]{2,30}$ - -# Regular expression matching correct variable names -variable-rgx=[a-z_][a-z0-9_]{1,30}$ - -# Regular expression matching correct inline iteration names -inlinevar-rgx=[A-Za-z_][A-Za-z0-9_]*$ - -# Regular expression matching correct constant names -const-rgx=(([A-Za-z_][A-Za-z0-9_]*)|(__.*__))$ - -# Regular expression matching correct class names -class-rgx=[A-Z_][a-zA-Z0-9]+$ - -# Regular expression matching correct class attribute names -class-attribute-rgx=([A-Za-z_][A-Za-z0-9_]{2,30}|(__.*__))$ - -# Regular expression matching correct module names -module-rgx=(([a-z_][a-z0-9_]*)|([A-Z][a-zA-Z0-9]+))$ - -# Regular expression matching correct attribute names -attr-rgx=[a-z_][a-z0-9_]{2,30}$ - -# Regular expression matching correct function names -function-rgx=[a-z_][a-z0-9_]{2,30}$ - -# Regular expression which should only match function or class names that do -# not require a docstring. -no-docstring-rgx=^_ - -# Minimum line length for functions/classes that require docstrings, shorter -# ones are exempt. -docstring-min-length=10 - - -[ELIF] - -# Maximum number of nested blocks for function / method body -max-nested-blocks=5 - - -[FORMAT] - -# Maximum number of characters on a single line. -max-line-length=100 - -# Regexp for a line that is allowed to be longer than the limit. -ignore-long-lines=^\s*(# )??$ - -# Allow the body of an if to be on the same line as the test if there is no -# else. -single-line-if-stmt=no - -# Maximum number of lines in a module -max-module-lines=1000 - -# String used as indentation unit. This is usually " " (4 spaces) or "\t" (1 -# tab). -indent-string=' ' - -# Number of spaces of indent required inside a hanging or continued line. -indent-after-paren=4 - -# Expected format of line ending, e.g. empty (any line ending), LF or CRLF. -expected-line-ending-format= - - -[LOGGING] - -# Logging modules to check that the string format arguments are in logging -# function parameter format -logging-modules=logging - - -[MISCELLANEOUS] - -# List of note tags to take in consideration, separated by a comma. -notes=FIXME,XXX - - -[SIMILARITIES] - -# Minimum lines number of a similarity. -min-similarity-lines=5 - -# Ignore comments when computing similarities. -ignore-comments=yes - -# Ignore docstrings when computing similarities. -ignore-docstrings=yes - -# Ignore imports when computing similarities. -ignore-imports=no - - -[SPELLING] - -# Spelling dictionary name. Available dictionaries: none. To make it working -# install python-enchant package. -spelling-dict= - -# List of comma separated words that should not be checked. -spelling-ignore-words= - -# A path to a file that contains private dictionary; one word per line. -spelling-private-dict-file= - -# Tells whether to store unknown words to indicated private dictionary in -# --spelling-private-dict-file option instead of raising a message. -spelling-store-unknown-words=no - - -[TYPECHECK] - -# Tells whether missing members accessed in mixin class should be ignored. A -# mixin class is detected if its name ends with "mixin" (case insensitive). -ignore-mixin-members=yes - -# List of module names for which member attributes should not be checked -# (useful for modules/projects where namespaces are manipulated during runtime -# and thus existing member attributes cannot be deduced by static analysis. It -# supports qualified module names, as well as Unix pattern matching. -ignored-modules=numpy,pandas,alembic.op,sqlalchemy,alembic.context,flask_appbuilder.security.sqla.PermissionView.role,flask_appbuilder.Model.metadata,flask_appbuilder.Base.metadata - -# List of class names for which member attributes should not be checked (useful -# for classes with dynamically set attributes). This supports the use of -# qualified names. -ignored-classes=contextlib.closing,optparse.Values,thread._local,_thread._local - -# List of members which are set dynamically and missed by pylint inference -# system, and so shouldn't trigger E1101 when accessed. Python regular -# expressions are accepted. -generated-members= - -# List of decorators that produce context managers, such as -# contextlib.contextmanager. Add to this list to register other decorators that -# produce valid context managers. -contextmanager-decorators=contextlib.contextmanager - - -[VARIABLES] - -# Tells whether we should check for unused import in __init__ files. -init-import=no - -# A regular expression matching the name of dummy variables (i.e. expectedly -# not used). -dummy-variables-rgx=(_+[a-zA-Z0-9]*?$)|dummy - -# List of additional names supposed to be defined in builtins. Remember that -# you should avoid to define new builtins when possible. -additional-builtins= - -# List of strings which can identify a callback function by name. A callback -# name must start or end with one of those strings. -callbacks=cb_,_cb - -# List of qualified module names which can have objects that can redefine -# builtins. -redefining-builtins-modules=six.moves,future.builtins - - -[CLASSES] - -# List of method names used to declare (i.e. assign) instance attributes. -defining-attr-methods=__init__,__new__,setUp - -# List of valid names for the first argument in a class method. -valid-classmethod-first-arg=cls - -# List of valid names for the first argument in a metaclass class method. -valid-metaclass-classmethod-first-arg=mcs - -# List of member names, which should be excluded from the protected access -# warning. -exclude-protected=_asdict,_fields,_replace,_source,_make - - -[DESIGN] - -# Maximum number of arguments for function / method -max-args=5 - -# Argument names that match this expression will be ignored. Default to name -# with leading underscore -ignored-argument-names=_.* - -# Maximum number of locals for function / method body -max-locals=15 - -# Maximum number of return / yield for function / method body -max-returns=10 - -# Maximum number of branch for function / method body -max-branches=15 - -# Maximum number of statements in function / method body -max-statements=50 - -# Maximum number of parents for a class (see R0901). -max-parents=7 - -# Maximum number of attributes for a class (see R0902). -max-attributes=8 - -# Minimum number of public methods for a class (see R0903). -min-public-methods=2 - -# Maximum number of public methods for a class (see R0904). -max-public-methods=20 - -# Maximum number of boolean expressions in a if statement -max-bool-expr=5 - - -[IMPORTS] - -# Deprecated modules which should not be used, separated by a comma -deprecated-modules=optparse - -# Create a graph of every (i.e. internal and external) dependencies in the -# given file (report RP0402 must not be disabled) -import-graph= - -# Create a graph of external dependencies in the given file (report RP0402 must -# not be disabled) -ext-import-graph= - -# Create a graph of internal dependencies in the given file (report RP0402 must -# not be disabled) -int-import-graph= - -# Force import order to recognize a module as part of the standard -# compatibility libraries. -known-standard-library= - -# Force import order to recognize a module as part of a third party library. -known-third-party=enchant - -# Analyse import fallback blocks. This can be used to support both Python 2 and -# 3 compatible code, which means that the block might have code that exists -# only in one or another interpreter, leading to false positives when analysed. -analyse-fallback-blocks=no - - -[EXCEPTIONS] - -# Exceptions that will emit a warning when being caught. Defaults to -# "Exception" -overgeneral-exceptions=builtins.Exception diff --git a/Makefile b/Makefile index 33bb511f2de83..1c9aa80ff53b9 100644 --- a/Makefile +++ b/Makefile @@ -87,9 +87,6 @@ format: py-format js-format py-format: pre-commit pre-commit run black --all-files -py-lint: pre-commit - pylint -j 0 superset - js-format: cd superset-frontend; npm run prettier diff --git a/docs/docs/contributing/development.mdx b/docs/docs/contributing/development.mdx index b9b05fb8901dc..94996eb6a3478 100644 --- a/docs/docs/contributing/development.mdx +++ b/docs/docs/contributing/development.mdx @@ -476,37 +476,7 @@ A series of checks will now run when you make a git commit. ## Linting -### Python - -We use [Pylint](https://pylint.org/) for linting which can be invoked via: - -```bash -pylint -``` - -In terms of best practices please avoid blanket disabling of Pylint messages globally (via `.pylintrc`) or top-level within the file header, albeit there being a few exceptions. Disabling should occur inline as it prevents masking issues and provides context as to why said message is disabled. - -Additionally, the Python code is auto-formatted using [Black](https://github.com/python/black) which -is configured as a pre-commit hook. There are also numerous [editor integrations](https://black.readthedocs.io/en/stable/integrations/editors.html) - -### TypeScript - -```bash -cd superset-frontend -npm ci -# run eslint checks -npm run eslint -- . -# run tsc (typescript) checks -npm run type -``` - -If using the eslint extension with vscode, put the following in your workspace `settings.json` file: - -```json -"eslint.workingDirectories": [ - "superset-frontend" -] -``` +See [how tos](/docs/contributing/howtos#linting) ## GitHub Actions and `act` diff --git a/docs/docs/contributing/howtos.mdx b/docs/docs/contributing/howtos.mdx index e638d93af9a35..63e4e01075a80 100644 --- a/docs/docs/contributing/howtos.mdx +++ b/docs/docs/contributing/howtos.mdx @@ -571,13 +571,9 @@ pybabel compile -d superset/translations ### Python -We use [Pylint](https://pylint.org/) and [ruff](https://github.com/astral-sh/ruff) -for linting which can be invoked via: +We [ruff](https://github.com/astral-sh/ruff) for linting which can be invoked via: ``` -# Run pylint -pylint superset/ - # auto-reformat using ruff ruff format @@ -588,11 +584,8 @@ ruff check ruff check --fix ``` - -In terms of best practices please avoid blanket disabling of Pylint messages globally -(via `.pylintrc`) or top-level within the file header, albeit there being a few exceptions. -Disabling should occur inline as it prevents masking issues and provides context as to why -said message is disabled. +Ruff configuration is located in our +(pyproject.toml)[https://github.com/apache/superset/blob/master/pyproject.toml] file All this is configured to run in pre-commit hooks, which we encourage you to setup with `pre-commit install` diff --git a/pyproject.toml b/pyproject.toml index fe79ab55b7b2e..26faefa55b4ad 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -195,7 +195,6 @@ development = [ "psutil", "pyfakefs", "pyinstrument>=4.0.2,<5", - "pylint", "pytest<8.0.0", # hairy issue with pytest >=8 where current_app proxies are not set in time "pytest-cov", "pytest-mock", @@ -288,11 +287,25 @@ select = [ "E4", "E7", "E9", - "F", "PT009", "TRY201", + # TODO add these rules in follow up PR + # "B", + # "C", + # "E", + # "F", + #"F", + # "I", + # "N", + # "PT", + # "Q", + # "S", + # "T", + #"W", +] +ignore = [ + "S101", ] -ignore = [] extend-select = ["I"] diff --git a/requirements/base.txt b/requirements/base.txt index 7d41af27955ff..c776735026420 100644 --- a/requirements/base.txt +++ b/requirements/base.txt @@ -154,9 +154,7 @@ geopy==2.4.1 google-auth==2.36.0 # via shillelagh greenlet==3.0.3 - # via - # shillelagh - # sqlalchemy + # via shillelagh gunicorn==23.0.0 # via apache-superset hashids==1.3.1 diff --git a/requirements/development.txt b/requirements/development.txt index 77c52059fa58f..1afbd47d0c213 100644 --- a/requirements/development.txt +++ b/requirements/development.txt @@ -8,10 +8,8 @@ -r base.txt -e file:. # via - # -r requirements/base.in + # -r /Users/max/code/superset/requirements/base.in # -r requirements/development.in -astroid==3.1.0 - # via pylint build==1.2.1 # via pip-tools cfgv==3.4.0 @@ -26,8 +24,6 @@ cycler==0.12.1 # via matplotlib db-dtypes==1.3.1 # via pandas-gbq -dill==0.3.9 - # via pylint distlib==0.3.8 # via virtualenv docker==7.0.0 @@ -85,8 +81,6 @@ identify==2.5.36 # via pre-commit iniconfig==2.0.0 # via pytest -isort==5.12.0 - # via pylint jsonschema-spec==0.1.6 # via openapi-spec-validator kiwisolver==1.4.7 @@ -95,8 +89,6 @@ lazy-object-proxy==1.10.0 # via openapi-spec-validator matplotlib==3.9.0 # via prophet -mccabe==0.7.0 - # via pylint mysqlclient==2.2.6 # via apache-superset nodeenv==1.8.0 @@ -158,8 +150,6 @@ pyhive[presto]==0.7.0 # via apache-superset pyinstrument==4.4.0 # via apache-superset -pylint==3.1.0 - # via apache-superset pyproject-hooks==1.2.0 # via # build @@ -192,10 +182,7 @@ tomli==2.1.0 # build # coverage # pip-tools - # pylint # pytest -tomlkit==0.13.2 - # via pylint toposort==1.10 # via pip-compile-multi tqdm==4.67.1 From 1214f24e02b50db9bbc629c1ab28d0ccd9dd9473 Mon Sep 17 00:00:00 2001 From: Maxime Beauchemin Date: Fri, 13 Dec 2024 10:48:48 -0800 Subject: [PATCH 2/3] Update docs/docs/contributing/howtos.mdx Co-authored-by: Ville Brofeldt <33317356+villebro@users.noreply.github.com> --- docs/docs/contributing/howtos.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/docs/contributing/howtos.mdx b/docs/docs/contributing/howtos.mdx index 63e4e01075a80..698d4ddf6f0ca 100644 --- a/docs/docs/contributing/howtos.mdx +++ b/docs/docs/contributing/howtos.mdx @@ -571,7 +571,7 @@ pybabel compile -d superset/translations ### Python -We [ruff](https://github.com/astral-sh/ruff) for linting which can be invoked via: +We use [ruff](https://github.com/astral-sh/ruff) for linting which can be invoked via: ``` # auto-reformat using ruff From bf3773ac9a457f5cf01060e804024cbfcc459400 Mon Sep 17 00:00:00 2001 From: Maxime Beauchemin Date: Fri, 13 Dec 2024 12:24:54 -0800 Subject: [PATCH 3/3] UPDATING.md --- UPDATING.md | 1 + 1 file changed, 1 insertion(+) diff --git a/UPDATING.md b/UPDATING.md index 50dc288245fb5..3c663ec37971a 100644 --- a/UPDATING.md +++ b/UPDATING.md @@ -27,6 +27,7 @@ assists people when migrating to a new version. - [29798](https://github.com/apache/superset/pull/29798) Since 3.1.0, the intial schedule for an alert or report was mistakenly offset by the specified timezone's relation to UTC. The initial schedule should now begin at the correct time. - [30021](https://github.com/apache/superset/pull/30021) The `dev` layer in our Dockerfile no long includes firefox binaries, only Chromium to reduce bloat/docker-build-time. - [30099](https://github.com/apache/superset/pull/30099) Translations are no longer included in the default docker image builds. If your environment requires translations, you'll want to set the docker build arg `BUILD_TRANSACTION=true`. +- [31262](https://github.com/apache/superset/pull/31262) NOTE: deprecated `pylint` in favor of `ruff` as our only python linter. Only affect development workflows positively (not the release itself). It should cover most important rules, be much faster, but some things linting rules that were enforced before may not be enforce in the exact same way as before. ### Potential Downtime