Skip to content
This repository has been archived by the owner on Jun 21, 2024. It is now read-only.

Latest commit

 

History

History
160 lines (109 loc) · 5.95 KB

04_troubleshooting.rst

File metadata and controls

160 lines (109 loc) · 5.95 KB
Raw

Troubleshooting

The outline for this document in GitHub is available at top-right corner button (with 3-dots and 3 lines).

Basic troubleshooting of breeze command

If you are having problems with the Breeze environment, try the steps below. After each step you can check whether your problem is fixed.

  1. If you are on macOS, check if you have enough disk space for Docker (Breeze will warn you if not).
  2. Stop Breeze with breeze down.
  3. Git fetch the origin and git rebase the current branch with main branch.
  4. Delete the .build directory and run breeze ci-image build.
  5. Clean up Docker images via breeze cleanup command.
  6. Restart your Docker Engine and try again.
  7. Restart your machine and try again.
  8. Re-install Docker Desktop and try again.

Note

If the pip is taking a significant amount of time and your internet connection is causing pip to be unable to download the libraries within the default timeout, it is advisable to modify the default timeout as follows and run the breeze again.

export PIP_DEFAULT_TIMEOUT=1000

In case the problems are not solved, you can set the VERBOSE_COMMANDS variable to "true":

export VERBOSE_COMMANDS="true"

Then run the failed command, copy-and-paste the output from your terminal to the Airflow Slack #airflow-breeze channel and describe your problem.

Warning

Some operating systems (Fedora, ArchLinux, RHEL, Rocky) have recently introduced Kernel changes that result in Airflow in Breeze consuming 100% memory when run inside the community Docker implementation maintained by the OS teams.

This is an issue with backwards-incompatible containerd configuration that some of Airflow dependencies have problems with and is tracked in a few issues:

There is no solution yet from the containerd team, but seems that installing Docker Desktop on Linux solves the problem as stated in This comment and allows to run Breeze with no problems.

ETIMEDOUT Error

When running breeze start-airflow, either normally or in dev-mode, the following output might be observed:

Skip fixing ownership of generated files as Host OS is darwin


Waiting for asset compilation to complete in the background.

Still waiting .....
Still waiting .....
Still waiting .....
Still waiting .....
Still waiting .....
Still waiting .....

The asset compilation is taking too long.

If it does not complete soon, you might want to stop it and remove file lock:
  * press Ctrl-C
  * run 'rm /opt/airflow/.build/www/.asset_compile.lock'

Still waiting .....
Still waiting .....
Still waiting .....
Still waiting .....
Still waiting .....
Still waiting .....
Still waiting .....

The asset compilation failed. Exiting.

[INFO] Locking pre-commit directory

Error 1 returned

This timeout can be increased by setting ASSET_COMPILATION_WAIT_MULTIPLIER a reasonable number could be 3-4.

export ASSET_COMPILATION_WAIT_MULTIPLIER=3

This error is actually caused by the following error during the asset compilation which resulted in ETIMEDOUT when npm command is trying to install required packages:

npm ERR! code ETIMEDOUT
npm ERR! syscall connect
npm ERR! errno ETIMEDOUT
npm ERR! network request to https://registry.npmjs.org/yarn failed, reason: connect ETIMEDOUT 2606:4700::6810:1723:443
npm ERR! network This is a problem related to network connectivity.
npm ERR! network In most cases you are behind a proxy or have bad network settings.
npm ERR! network
npm ERR! network If you are behind a proxy, please make sure that the
npm ERR! network 'proxy' config is set properly.  See: 'npm help config'

In this situation, notice that the IP address 2606:4700::6810:1723:443 is in IPv6 format, which was the reason why the connection did not go through the router, as the router did not support IPv6 addresses in its DNS lookup. In this case, disabling IPv6 in the host machine and using IPv4 instead resolved the issue.

The similar issue could happen if you are behind an HTTP/HTTPS proxy and your access to required websites are blocked by it, or your proxy setting has not been done properly.

It could also be possible that you have a proxy which is not available from your network, leading to the timeout issue. You may try running the below commands in the same terminal and then try the breeze start-airflow command:

npm config delete http-proxy
npm config delete https-proxy

npm config rm proxy
npm config rm https-proxy

set HTTP_PROXY=null
set HTTPS_PROXY=null

Next step: Follow the Test commands guide to running tests using Breeze.