Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: rm skip_if and run_if in python source #41832

Merged
merged 12 commits into from
Oct 2, 2024

Conversation

phi-friday
Copy link
Contributor

Like setup and teardown, skip_if and run_if should also be removed.

sample dag code

from __future__ import annotations

from pendulum import datetime

from airflow.decorators import dag, task


@dag(start_date=datetime(2024, 1, 1), schedule=None, catchup=False)
def venv_skip_if() -> None:
    @task.skip_if(lambda context: False)
    @task.virtualenv()
    def echo() -> None:
        print("hello world")

    echo()


venv_skip_if()

before in venv

1b4bbddb0c17
*** Found local files:
***   * /root/airflow/logs/dag_id=venv_skip_if/run_id=manual__2024-08-28T12:29:04.469903+00:00/task_id=echo/attempt=1.log
[2024-08-28, 12:29:05 UTC] {local_task_job_runner.py:123} ▶ Pre task execution logs
[2024-08-28, 12:29:05 UTC] {process_utils.py:186} INFO - Executing cmd: /usr/local/bin/python -m virtualenv /tmp/venv5qst192w --system-site-packages --python=python
[2024-08-28, 12:29:05 UTC] {process_utils.py:190} INFO - Output:
[2024-08-28, 12:29:06 UTC] {process_utils.py:194} INFO - created virtual environment CPython3.8.19.final.0-64 in 276ms
[2024-08-28, 12:29:06 UTC] {process_utils.py:194} INFO -   creator CPython3Posix(dest=/tmp/venv5qst192w, clear=False, no_vcs_ignore=False, global=True)
[2024-08-28, 12:29:06 UTC] {process_utils.py:194} INFO -   seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/root/.local/share/virtualenv)
[2024-08-28, 12:29:06 UTC] {process_utils.py:194} INFO -     added seed packages: pip==24.1, setuptools==70.1.0, wheel==0.43.0
[2024-08-28, 12:29:06 UTC] {process_utils.py:194} INFO -   activators BashActivator,CShellActivator,FishActivator,NushellActivator,PowerShellActivator,PythonActivator
[2024-08-28, 12:29:06 UTC] {process_utils.py:186} INFO - Executing cmd: /tmp/venv5qst192w/bin/pip install -r /tmp/venv5qst192w/requirements.txt
[2024-08-28, 12:29:06 UTC] {process_utils.py:190} INFO - Output:
[2024-08-28, 12:29:10 UTC] {process_utils.py:194} INFO - 
[2024-08-28, 12:29:10 UTC] {process_utils.py:194} INFO - [notice] A new release of pip is available: 24.1 -> 24.2
[2024-08-28, 12:29:10 UTC] {process_utils.py:194} INFO - [notice] To update, run: python -m pip install --upgrade pip
[2024-08-28, 12:29:10 UTC] {process_utils.py:186} INFO - Executing cmd: /tmp/venv5qst192w/bin/python /tmp/venv-callwp2r9u9t/script.py /tmp/venv-callwp2r9u9t/script.in /tmp/venv-callwp2r9u9t/script.out /tmp/venv-callwp2r9u9t/string_args.txt /tmp/venv-callwp2r9u9t/termination.log /tmp/venv-callwp2r9u9t/***_context.json
[2024-08-28, 12:29:10 UTC] {process_utils.py:190} INFO - Output:
[2024-08-28, 12:29:13 UTC] {process_utils.py:194} INFO - Traceback (most recent call last):
[2024-08-28, 12:29:13 UTC] {process_utils.py:194} INFO -   File "/tmp/venv-callwp2r9u9t/script.py", line 18, in <module>
[2024-08-28, 12:29:13 UTC] {process_utils.py:194} INFO -     @task.skip_if(lambda context: False)
[2024-08-28, 12:29:13 UTC] {process_utils.py:194} INFO - NameError: name 'task' is not defined
[2024-08-28, 12:29:13 UTC] {taskinstance.py:3167} ERROR - Task failed with exception
Traceback (most recent call last):
  File "/opt/airflow/airflow/models/taskinstance.py", line 753, in _execute_task
    result = _execute_callable(context=context, **execute_callable_kwargs)
  File "/opt/airflow/airflow/models/taskinstance.py", line 719, in _execute_callable
    return ExecutionCallableRunner(
  File "/opt/airflow/airflow/utils/operator_helpers.py", line 252, in run
    return self.func(*args, **kwargs)
  File "/opt/airflow/airflow/models/baseoperator.py", line 402, in wrapper
    return func(self, *args, **kwargs)
  File "/opt/airflow/airflow/decorators/base.py", line 266, in execute
    return_value = super().execute(context)
  File "/opt/airflow/airflow/models/baseoperator.py", line 402, in wrapper
    return func(self, *args, **kwargs)
  File "/opt/airflow/airflow/operators/python.py", line 509, in execute
    return super().execute(context=serializable_context)
  File "/opt/airflow/airflow/models/baseoperator.py", line 402, in wrapper
    return func(self, *args, **kwargs)
  File "/opt/airflow/airflow/operators/python.py", line 240, in execute
    return_value = self.execute_callable()
  File "/opt/airflow/airflow/operators/python.py", line 900, in execute_callable
    result = self._execute_python_callable_in_subprocess(python_path)
  File "/opt/airflow/airflow/operators/python.py", line 593, in _execute_python_callable_in_subprocess
    execute_in_subprocess(
  File "/opt/airflow/airflow/utils/process_utils.py", line 175, in execute_in_subprocess
    execute_in_subprocess_with_kwargs(cmd, cwd=cwd, env=env)
  File "/opt/airflow/airflow/utils/process_utils.py", line 198, in execute_in_subprocess_with_kwargs
    raise subprocess.CalledProcessError(exit_code, cmd)
subprocess.CalledProcessError: Command '['/tmp/venv5qst192w/bin/python', '/tmp/venv-callwp2r9u9t/script.py', '/tmp/venv-callwp2r9u9t/script.in', '/tmp/venv-callwp2r9u9t/script.out', '/tmp/venv-callwp2r9u9t/string_args.txt', '/tmp/venv-callwp2r9u9t/termination.log', '/tmp/venv-callwp2r9u9t/airflow_context.json']' returned non-zero exit status 1.
[2024-08-28, 12:29:13 UTC] {taskinstance.py:1211} INFO - Marking task as FAILED. dag_id=venv_skip_if, task_id=echo, run_id=manual__2024-08-28T12:29:04.469903+00:00, execution_date=20240828T122904, start_date=20240828T122905, end_date=20240828T122913
[2024-08-28, 12:29:13 UTC] {taskinstance.py:337} ▶ Post task execution logs

after in venv

5dae42dc6e21
*** Found local files:
***   * /root/airflow/logs/dag_id=venv_skip_if/run_id=manual__2024-08-28T12:39:47.502535+00:00/task_id=echo/attempt=1.log
[2024-08-28, 12:39:48 UTC] {local_task_job_runner.py:123} ▶ Pre task execution logs
[2024-08-28, 12:39:48 UTC] {process_utils.py:186} INFO - Executing cmd: /usr/local/bin/python -m virtualenv /tmp/venvt7te04zf --system-site-packages --python=python
[2024-08-28, 12:39:48 UTC] {process_utils.py:190} INFO - Output:
[2024-08-28, 12:39:48 UTC] {process_utils.py:194} INFO - created virtual environment CPython3.8.19.final.0-64 in 266ms
[2024-08-28, 12:39:48 UTC] {process_utils.py:194} INFO -   creator CPython3Posix(dest=/tmp/venvt7te04zf, clear=False, no_vcs_ignore=False, global=True)
[2024-08-28, 12:39:48 UTC] {process_utils.py:194} INFO -   seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/root/.local/share/virtualenv)
[2024-08-28, 12:39:48 UTC] {process_utils.py:194} INFO -     added seed packages: pip==24.1, setuptools==70.1.0, wheel==0.43.0
[2024-08-28, 12:39:48 UTC] {process_utils.py:194} INFO -   activators BashActivator,CShellActivator,FishActivator,NushellActivator,PowerShellActivator,PythonActivator
[2024-08-28, 12:39:48 UTC] {process_utils.py:186} INFO - Executing cmd: /tmp/venvt7te04zf/bin/pip install -r /tmp/venvt7te04zf/requirements.txt
[2024-08-28, 12:39:48 UTC] {process_utils.py:190} INFO - Output:
[2024-08-28, 12:39:52 UTC] {process_utils.py:194} INFO - 
[2024-08-28, 12:39:52 UTC] {process_utils.py:194} INFO - [notice] A new release of pip is available: 24.1 -> 24.2
[2024-08-28, 12:39:52 UTC] {process_utils.py:194} INFO - [notice] To update, run: python -m pip install --upgrade pip
[2024-08-28, 12:39:52 UTC] {process_utils.py:186} INFO - Executing cmd: /tmp/venvt7te04zf/bin/python /tmp/venv-call55vo1yul/script.py /tmp/venv-call55vo1yul/script.in /tmp/venv-call55vo1yul/script.out /tmp/venv-call55vo1yul/string_args.txt /tmp/venv-call55vo1yul/termination.log /tmp/venv-call55vo1yul/***_context.json
[2024-08-28, 12:39:52 UTC] {process_utils.py:190} INFO - Output:
[2024-08-28, 12:39:56 UTC] {process_utils.py:194} INFO - hello world
[2024-08-28, 12:39:56 UTC] {python.py:242} INFO - Done. Returned value was: None
[2024-08-28, 12:39:56 UTC] {taskinstance.py:337} ▶ Post task execution logs

^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

@phi-friday
Copy link
Contributor Author

I'm getting an error related to the open api, I'm not understanding why it's happening, could you please explain?

@phi-friday phi-friday force-pushed the fix-venv-skip-if branch 3 times, most recently from 4d2a20a to 739fecf Compare September 5, 2024 09:48
@eladkal eladkal added this to the Airflow 2.10.3 milestone Sep 21, 2024
@eladkal eladkal added the type:bug-fix Changelog: Bug Fixes label Sep 21, 2024
@potiuk potiuk merged commit 69b4592 into apache:main Oct 2, 2024
81 checks passed
@phi-friday phi-friday deleted the fix-venv-skip-if branch October 2, 2024 04:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
type:bug-fix Changelog: Bug Fixes
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants