container "pytorch" is waiting to start: PodInitializing

[gogogwwb]

When the master is finished running, the worker is still initializing.

worker log：
Error from server (BadRequest): container "pytorch" in pod"xxx-jxosi-worker-0" is waiting to start: PodInitializing

What is the reason for this?

[gaocegege]

Could you please run kubectl describe and post the result here?

[gaocegege]

Can you show more about it? Especially the events section.

[gaocegege]

Seems that the init container is pending. Can you show its log?

[gogogwwb]

init-pytorch log：

[gogogwwb]

master svc ：

[gaocegege]

Can you try kubectl debug to run an ephemeral container, then run ping xxx-master-0?

[gogogwwb]

ping master-0 in an ephemeral container :

[gogogwwb]

kubectl get ep -A，then i found ep appeared, but disappeared again after a while

[gaocegege]

It's weird.

[gogogwwb]

I put the program to sleep for a while and found that the worker can run. Is there any restriction on the creation order of service and pod in pytorchjob?

[gogogwwb]

It should be that the master executes too fast, so that the ep of the service finally becomes none, and the worker cannot obtain the IP address of the master.

[gaocegege]

It should be that the master executes too fast, so that the ep of the service finally becomes none, and the worker cannot obtain the IP address of the master.

Interesting. /cc @johnugeorge

[johnugeorge]

But in that case, master should not start the job until workers are up. Are you using distributed setup itself in the code?

[gogogwwb]

I did not use distributed steps in the code. After master running, it becomes completed state, "kubectl get ep -ntest", and found that ep is none

[gogogwwb]

But in that case, master should not start the job until workers are up. Are you using distributed setup itself in the code?

If the code is not sleeping, the master is in a short running state, but the worker is in the init state

[johnugeorge]

If you are not using distributed pytorch in the code, this can happen. Master can start executing and gets completed before worker starts. Can you confirm whether you are using distributed APIs?

[gogogwwb]

If you are not using distributed pytorch in the code, this can happen. Master can start executing and gets completed before worker starts. Can you confirm whether you are using distributed APIs?

I think i don't use distributed APIs.
the code：

[johnugeorge]

That is issue. Any reason in using pytorch-operator without using distributed version?

Example:

https://github.com/kubeflow/tf-operator/blob/1aa44a68cd364ed6e30c0841e6daf1d93a29f146/examples/pytorch/mnist/mnist.py#L72

[gogogwwb]

How do I make the pod created by pytorchjob not automatically disappear after completion, is it to use cleanpolicy? How to set it up? thanks

[johnugeorge]

Set to None

https://github.com/kubeflow/common/blob/f7c41a08761ff3b215553a051fd529efd22782a1/pkg/apis/common/v1/types.go#L136