STATIC_CODE_CHECKS.rst

• Static code checks
  • Pre-commit hooks
    • Installing pre-commit hooks
    • Enabling pre-commit hooks
    • Available pre-commit checks
    • Using pre-commit
  • Running static code checks via Breeze

Static code checks

The static code checks in Airflow are used to verify that the code meets certain quality standards. All the static code checks can be run through pre-commit hooks.

The pre-commit hooks perform all the necessary installation when you run them for the first time. See the table below to identify which pre-commit checks require the Breeze Docker images.

You can also run some static code checks via Breeze environment using available bash scripts.

Pre-commit hooks

Pre-commit hooks help speed up your local development cycle and place less burden on the CI infrastructure. Consider installing the pre-commit hooks as a necessary prerequisite.

The pre-commit hooks by default only check the files you are currently working on and make them fast. Yet, these checks use exactly the same environment as the CI tests use. So, you can be sure your modifications will also work for CI if they pass pre-commit hooks.

We have integrated the fantastic pre-commit framework in our development workflow. To install and use it, you need at least Python 3.7 locally.

Installing pre-commit hooks

It is the best to use pre-commit hooks when you have your local virtualenv for Airflow activated since then pre-commit hooks and other dependencies are automatically installed. You can also install the pre-commit hooks manually using pip install.

pip install pre-commit

After installation, pre-commit hooks are run automatically when you commit the code and they will only run on the files that you change during your commit, so they are usually pretty fast and do not slow down your iteration speed on your changes. There are also ways to disable the pre-commits temporarily when you commit your code with --no-verify switch or skip certain checks that you find to much disturbing your local workflow. See `Available pre-commit checks<#available-pre-commit-checks>`_ and Using pre-commit

Note

Additional prerequisites might be needed

The pre-commit hooks use several external linters that need to be installed before pre-commit is run. Each of the checks installs its own environment, so you do not need to install those, but there are some checks that require locally installed binaries. On Linux, you typically install them with sudo apt install, on macOS - with brew install.

The current list of prerequisites is limited to xmllint:

• on Linux, install via sudo apt install libxml2-utils
• on macOS, install via brew install libxml2

Some pre-commit hooks also require the Docker Engine to be configured as the static checks are executed in the Docker environment (See table in the Available pre-commit checks<#available-pre-commit-checks>`_ . You should build the images locally before installing pre-commit checks as described in BREEZE.rst.

Sometimes your image is outdated and needs to be rebuilt because some dependencies have been changed. In such cases, the Docker-based pre-commit will inform you that you should rebuild the image.

In case you do not have your local images built, the pre-commit hooks fail and provide instructions on what needs to be done.

Enabling pre-commit hooks

To turn on pre-commit checks for commit operations in git, enter:

pre-commit install

To install the checks also for pre-push operations, enter:

pre-commit install -t pre-push

For details on advanced usage of the install method, use:

pre-commit install --help

Available pre-commit checks

This table lists pre-commit hooks used by Airflow. The Breeze column indicates which hooks require Breeze Docker images to be installed locally.

Note

Disabling particular checks

In case you have a problem with running particular pre-commit check you can still continue using the benefits of having pre-commit installed, with some of the checks disabled. In order to disable checks you might need to set SKIP environment variable to coma-separated list of checks to skip. For example when you want to skip some checks (flake/mypy for example), you should be able to do it by setting export SKIP=run-flake8,run-mypy. You can also add this to your .bashrc or .zshrc if you do not want to set it manually every time you enter the terminal.

ID	Description	Image
black	Run Black (the uncompromising Python code formatter)
blacken-docs	Run black on python code blocks in documentation files
check-airflow-2-1-compatibility	Check that providers are 2.1 compatible.
check-airflow-config-yaml-consistent	Checks for consistency between config.yml and default_config.cfg
check-airflow-providers-have-extras	Checks providers available when declared by extras in setup.py
check-apache-license-rat	Check if licenses are OK for Apache
check-base-operator-usage	Check BaseOperator[Link] core imports Check BaseOperator[Link] other imports
check-boring-cyborg-configuration	Checks for Boring Cyborg configuration consistency
check-breeze-top-dependencies-limited	Breeze should have small number of top-level dependencies
check-builtin-literals	Require literal syntax when initializing Python builtin types
check-changelog-has-no-duplicates	Check changelogs for duplicate entries
check-daysago-import-from-utils	Make sure days_ago is imported from airflow.utils.dates
check-docstring-param-types	Check that docstrings do not specify param types
check-executables-have-shebangs	Check that executables have shebang
check-extra-packages-references	Checks setup extra packages
check-extras-order	Check order of extras in Dockerfile
check-for-inclusive-language	Check for language that we do not accept as community
check-hooks-apply	Check if all hooks apply to the repository
check-incorrect-use-of-LoggingMixin	Make sure LoggingMixin is not used alone
check-integrations-are-consistent	Check if integration list is consistent in various places
check-merge-conflict	Check that merge conflicts are not being committed
check-newsfragments-are-valid	Check newsfragments are valid
check-no-providers-in-core-examples	No providers imports in core example DAGs
check-no-relative-imports	No relative imports
check-persist-credentials-disabled-in-github-workflows	Check that workflow files have persist-credentials disabled
check-pre-commit-information-consistent	Update information re pre-commit hooks and verify ids and names
check-provide-create-sessions-imports	Check provide_session and create_session imports
check-provider-yaml-valid	Validate providers.yaml files
check-providers-init-file-missing	Provider init file is missing
check-providers-subpackages-init-file-exist	Provider subpackage init files are there
check-pydevd-left-in-code	Check for pydevd debug statements accidentally left
check-revision-heads-map	Check that the REVISION_HEADS_MAP is up-to-date
check-safe-filter-usage-in-html	Don't use safe in templates
check-setup-order	Check order of dependencies in setup.cfg and setup.py
check-start-date-not-used-in-defaults	'start_date' not to be defined in default_args in example_dags
check-system-tests-present	Check if system tests have required segments of code
check-xml	Check XML files with xmllint
codespell	Run codespell to check for common misspellings in files
debug-statements	Detect accidentally committed debug statements
detect-private-key	Detect if private key is added to the repository
doctoc	Add TOC for md and rst files
end-of-file-fixer	Make sure that there is an empty line at the end
fix-encoding-pragma	Remove encoding header from python files
flynt	Run flynt string format converter for Python
forbid-tabs	Fail if tabs are used in the project
identity	Print input to the static check hooks for troubleshooting
insert-license	Add license for all SQL files Add license for all rst files Add license for all CSS/JS/PUML/TS/TSX files Add license for all JINJA template files Add license for all shell files Add license for all Python files Add license for all XML files Add license for all YAML files Add license for all md files Add license for all other files
isort	Run isort to sort imports in Python files
lint-chart-schema	Lint chart/values.schema.json file
lint-css	stylelint
lint-dockerfile	Lint dockerfile
lint-helm-chart	Lint Helm Chart
lint-javascript	ESLint against airflow/ui ESLint against current UI JavaScript files
lint-json-schema	Lint JSON Schema files with JSON Schema Lint NodePort Service with JSON Schema Lint Docker compose files with JSON Schema Lint chart/values.schema.json file with JSON Schema Lint chart/values.yaml file with JSON Schema Lint airflow/config_templates/config.yml file with JSON Schema
lint-markdown	Run markdownlint
lint-openapi	Lint OpenAPI using spectral Lint OpenAPI using openapi-spec-validator
mixed-line-ending	Detect if mixed line ending is used (r vs. rn)
pretty-format-json	Format json files
pydocstyle	Run pydocstyle
python-no-log-warn	Check if there are no deprecate log warn
pyupgrade	Upgrade Python code automatically
rst-backticks	Check if RST files use double backticks for code
run-flake8	Run flake8
run-mypy	Run mypy for dev Run mypy for core Run mypy for providers Run mypy for /docs/ folder
run-shellcheck	Check Shell scripts syntax correctness
static-check-autoflake	Remove all unused code
trailing-whitespace	Remove trailing whitespace at end of line
update-breeze-file	Update output of breeze commands in BREEZE.rst
update-breeze-readme-config-hash	Update Breeze README.md with config files hash
update-extras	Update extras in documentation
update-in-the-wild-to-be-sorted	Sort INTHEWILD.md alphabetically
update-inlined-dockerfile-scripts	Inline Dockerfile and Dockerfile.ci scripts
update-local-yml-file	Update mounts in the local yml file
update-migration-references	Update migration ref doc
update-providers-dependencies	Update cross-dependencies for providers packages
update-setup-cfg-file	Update setup.cfg file with all licenses
update-spelling-wordlist-to-be-sorted	Sort alphabetically and uniquify spelling_wordlist.txt
update-supported-versions	Updates supported versions in documentation
update-vendored-in-k8s-json-schema	Vendor k8s definitions into values.schema.json
update-version	Update version to the latest version in the documentation
yamllint	Check YAML files with yamllint
yesqa	Remove unnecessary noqa statements

Using pre-commit

After installation, pre-commit hooks are run automatically when you commit the code. But you can run pre-commit hooks manually as needed.

• Run all checks on your staged files by using:

pre-commit run

• Run only mypy check on your staged files by using:

pre-commit run run-mypy

• Run only mypy checks on all files by using:

pre-commit run run-mypy --all-files

• Run all checks on all files by using:

pre-commit run --all-files

• Run all checks only on files modified in the last locally available commit in your checked out branch:

pre-commit run --source=HEAD^ --origin=HEAD

• Show files modified automatically by pre-commit when pre-commits automatically fix errors

pre-commit run --show-diff-on-failure

• Skip one or more of the checks by specifying a comma-separated list of checks to skip in the SKIP variable:

SKIP=run-mypy,run-flake8 pre-commit run --all-files

You can always skip running the tests by providing --no-verify flag to the git commit command.

To check other usage types of the pre-commit framework, see Pre-commit website.

Running static code checks via Breeze

The static code checks can be launched using the Breeze environment.

You run the static code checks via breeze static-check or commands.

You can see the list of available static checks either via --help flag or by using the autocomplete option. Note that the all static check runs all configured static checks.

Run the mypy check for the currently staged changes:

breeze static-check --type run-mypy

Run the mypy check for all files:

breeze static-check --type run-mypy --all-files

Run the flake8 check for the tests.core.py file with verbose output:

breeze static-check --type run-flake8 --files tests/core.py --verbose

Run the flake8 check for the tests.core package with verbose output:

breeze static-check --type run-flake8 --files tests/core/* --verbose

Run all tests for the currently staged files:

breeze static-check --type all

Run all tests for all files:

breeze static-check --type all --all-files

Run all tests for last commit :

breeze static-check --type all --last-commit

The license check is run via a separate script and a separate Docker image containing the Apache RAT verification tool that checks for Apache-compatibility of licenses within the codebase. It does not take pre-commit parameters as extra arguments.

breeze static-check licenses