WARNING - Model file does not exist

[arjunanan6]

Apache Airflow version

Other Airflow 2 version (please specify below)

What happened

Airflow v2.6.0 w/KubernetesExecutor

At this point, it is just a cosmetic issue since everything works as expected. I'm seeing this warning on all of my tasks:

[2023-06-20, 11:40:18 UTC] {pod_generator.py:529} WARNING - Model file /opt/airflow/pod_templates/pod_template_file.yaml does not exist

But the file does exist, as shown in the warning message as it managed to get the path from kube_config and I can use yaml.safe_load on it if I were to shell into the pod, or import deserialize_model_file from pod_generator.py. What could be going on here?

What you think should happen instead

The warning should not be printed as the file does exist.

How to reproduce

Just execute any task.

Operating System

Debian GNU/Linux

Versions of Apache Airflow Providers

No response

Deployment

Official Apache Airflow Helm Chart

Deployment details

Deployed on AKS with official helm chart.

Anything else

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[hussein-awala]

since everything works as expected

What do you mean by works as expected? does it load the configurations from the pod_template_file?

[arjunanan6]

I mean that the pods do get spun up anyway, and tasks are executed as expected. It's just that this warning appears first and doesn't make sense.

I think despite the warning it works because of:
https://github.com/apache/airflow/blob/8f65aee8f3c4dfdd6c4195d97f57f4267d37c209/airflow/kubernetes/pod_generator.py#LL131C1-L134C30

[potiuk]

The file for sure does not exist:

     if os.path.exists(path):
            with open(path) as stream:
                pod = yaml.safe_load(stream)
        else:
            pod = None
            log.warning("Model file %s does not exist", path)

The question is where and for whom - look at where the log file is generated and check there. My bet is that the file does not exist on scheduler, where it should be present and it should be accessible to the unix user that is used by Airflow.

So you need to check there. You might have also similar problems if for example the file or the folder where it is located do not have the right permissions for the unix uses that tries to read the file - this is how linux work, if you do not have permissions to read a folder where the file is located, the OS will report the file as not existing. So whatever you do as "different" linux user, might not matter because your airflow process might be run using a differeint linux user.

You need to check using the same environment your software is running with.

Converted to discussion in case you need more help with your investigation, but I suggest you look deeper.

[potiuk]

@esthomw Well. I did not get answer to my first question - where and when it happens, so I have no idea

[potiuk]

Do you know the answer maybe ?

[shivshav]

Hello, I'm sorry to resurrect a thread from a while back. I'm running into a similar issue myself (though notably different in 1 respect).

We're specifically getting

[2025-02-26T19:58:03.624+0000] {pod_generator.py:477} WARNING - Model file  does not exist

in our logs. Note the between file and does. That implies the path is actually being passed in as blank which is odd. I can provide additional info if this is the correct place.

Shall I create a new discussion for it or continue this one?

[potiuk]

If you have way more information then what you just provided and results of your investigation so far, likely starting new discussion is better. Searching for similar error message without context is just muddying the waters.

Also the investigation I mentioned might help in getting to the solution by yourself, this is how usually getting more details and tring to explain them to others work.

[shivshav]

Thank you, I've created a new discussion here with as much info as I could gather