registry-creds in Minikube doesn't renew credentials before they expire.

[jim-barber-he]

Overall the registry-creds addon in Minikube 0.28.2 is working quite well.
However we've noticed an issue where there can be periods where our AWS ECR credentials have expired and they haven't been renewed yet.
This results in errors like so (I've substituted $AWS_ACCOUNT_ID, $IMAGE, and $TAG into the messages to anonymise them a bit):

  Normal   Pulling                29m (x4 over 30m)    kubelet, minikube  pulling image "$AWS_ACCOUNT_ID.dkr.ecr.ap-southeast-2.amazonaws.com/$IMAGE:$TAG"
  Warning  Failed                 29m (x4 over 30m)    kubelet, minikube  Failed to pull image "$AWS_ACCOUNT_ID.dkr.ecr.ap-southeast-2.amazonaws.com/$IMAGE:$TAG": rpc error: code = Unknown desc = Error response from daemon: pull access denied for $AWS_ACCOUNT_ID.dkr.ecr.ap-southeast-2.amazonaws.com/$IMAGE, repository does not exist or may require 'docker login'

If left for long enough, the credentials are renegotiated and things work again.

We can also just delete the registry-cred pod in the kube-system namespace to have it start a new pod to renegotiate the credentials without having to wait.

I'm not sure if Minikube changes the default polling of 60 mins or not, but if not, then the worst case scenario could be waiting up to an hour for new credentials to be negotiated.

Is it possible to have this module re-negotiate AWS ECR credentials prior to them expiring?

[stevesloka]

The default refresh time is 60 minutes, but you could make that smaller if you'd like (https://github.com/upmc-enterprises/registry-creds/blob/master/main.go#L68).

Credentials by default are good for 12-hours, so refreshing every hour should be overly optimistic.

[jim-barber-he]

OK thanks for your response.

So each time they are renewed, new credentials are produced?

What we are seeing (but I'll have to investigate more closely by the looks) are the credentials are expiring and there is a period before the renewal fixes them again.
It seems like the hourly "renewal" may not be getting new credentials, but may be returning the same ones as we already have (or maybe doing nothing?) if we are still in the 12 hour validity period?

I'll talk with our developers more to find out if there are other possible causes (like if they are having the issue after taking a laptop out of sleep for example).

[gokhansengun]

We are also seeing this issue. Occasionally it is after laptop put to sleep and get back up after ours. I am looking for the ways for reliable reproduction. Killing the pod is a workaround as written in above comments.

I have also seen the issue recently in a server where it is always up. I have no idea how does it occur though