Image Registry configuration patch isn't applied when adding allowedRegistries

[ProtossGP32]

Describe the bug

I'd llke to add the internal registry as an additional registry where to pull images from when launching Builds (for example, to not depend on docker pull quotas). For that, I've followed the steps in the official docs, but after finished patching the image.config.openshift.io/cluster custom resource, the nodes aren't cordoned and the changes only remain in there, there's no rollout of it.

Version

OKD version release: 4.13.0-0.okd-2023-05-22-052007
Install method: IPI
Cloud platform: OpenStack

How reproducible

100%, just follow the instructions in the official docs to add the internal registry as an allowedRegistries entry:

oc edit image.config.openshift.io/cluster

And then adding the list of allowed registers:

 apiVersion: config.openshift.io/v1
 kind: Image
 [...]
 spec:
   registrySources:
     allowedRegistries:
     - image-registry.openshift-image-registry.svc:5000
     - quay.io
     - registry.redhat.io
     - docker.io

Expected behaviour
Nodes are cordoned, new config is applied and nodes are uncordoned again. I'd expect to see the NotReady status in the nodes for some amount of time.

Real behaviour
Nodes aren't cordoned at all, they keep on being Ready; the custom resource is modified (checked again with oc get image.config.openshift.io/cluster -o yaml) and thus there's no rollout of the changes in the platform.

Log bundle

ClusterID: ea345ff1-460d-4ece-8884-540d0fc58702
ClusterVersion: Stable at "4.13.0-0.okd-2023-05-22-052007"
ClusterOperators:
        clusteroperator/openshift-samples is degraded because Samples installation in error at 4.13.0-0.okd-2023-05-22-052007: &errors.errorString{s:"global openshift image configuration prevents the creation of imagestreams using the registry "}

• .openshift_install.log: https://gist.github.com/ProtossGP32/0efe2200acfbffb3e13601a9503d561f
• Download must-gather archive: LINK

[ProtossGP32]

@vrutkovs I'd like to know why this issue has been moved to a discussion, as it's the second time that happens to me. What would be the exact criteria for bugs? Right now I followed an official procedure and didn't return the expected results, so I think of this as a bug.

[vrutkovs]

this issue has been moved to a discussion

No log bundle / must gather provided, for starters. Also, issues are for actual confirmed bugs, so once we confirm that this is indeed an OKD-specific issue we need to fix (and not a documentation / OCP bug) we'll move it to issues again

[ProtossGP32]

Thanks for the pointers! I've updated the description with both the .openshift_installer.log and the must-gather output (WeTransfer link, expires in 7 days).

[ProtossGP32]

It's been two weeks since I opened this discussion, and I'm struggling with the same issue. Image registry config isn't updated in the nodes and I can't access the internal registry to pull images from there. Does anybody have any advice on how to further proceed with the troubleshooting?
Thanks!

[ProtossGP32]

Further investigation on the open enhancements in OpenShift has led me to find these user stories that are exactly what I'd expect from the platform in terms of image registry access:
Add unqualified search registries - User Stories

What should I have to do in order to achieve a similar outcome, knowing that this doesn't seem to be fully implemented yet?

[vrutkovs]

see https://access.redhat.com/solutions/5063741:

apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfig
metadata:
  labels:
    machineconfiguration.openshift.io/role: worker
  name: 99-worker-unqualified-search-registries
spec:
  config:
    ignition:
      version: 2.2.0
    storage:
      files:
      - contents:
          source: data:text/plain;charset=utf-8;base64,dW5xdWFsaWZpZWQtc2VhcmNoLXJlZ2lzdHJpZXMgPSBbJ3F1YXkuaW8nXQo=
        filesystem: root
        mode: 0420
        path: /etc/containers/registries.conf.d/99-worker-unqualified-search-registries.conf

[ProtossGP32]

I'm sorry, I can't view the full Red Hat solution as I don't have a subscription. What does this MachineConfig tries to do that the changes in the image registry config doesn't do? Do I have to provide the base64 conversion of the expected registries.conf with my own registries here?

[LorbusChris]

The RH solution links to https://docs.openshift.com/container-platform/4.13/openshift_images/image-configuration.html

see https://docs.okd.io/4.13/openshift_images/image-configuration.html for the OKD variant of the docs

[ProtossGP32]

Hi @LorbusChris , thanks for the quick reply! The docs linked in the solution are in fact the ones I followed for the whole registry configuration (in fact I open this discussion pointing towards them). In these links there's no mention to the MachineConfig resource that @vrutkovs shared on his last message, so how are they related to each other? I understand that this MachineConfig resource should manually add the path to the unqualified search registries that should have been already created when modifying the image.config.openshift.io/cluster? Sorry about being so vague at this point of the discussion, but I feel like walking in circles since a long time just to allow the system to access its own internal registry