ImageStream pull on external private docker registry #17238

zonArt · 2017-11-08T15:12:57Z

Impossible to use secret when trying to pull/push image from an external private docker registry with ImageStream

Version

oc v3.6.0+c4dd4cf
kubernetes v1.6.1+5115d708d7
features: Basic-Auth GSSAPI Kerberos SPNEGO

Server https://openshift.example.com:8443
openshift v3.6.0+c4dd4cf
kubernetes v1.6.1+5115d708d7

Steps To Reproduce

create a secret with --docker-server=my-private-registry.example.com (and of course relevant user credential)
assign it to a serviceaccount (not sure if this part is mandatory though)
create an ImageStream with something like oc import-image myimage --from=my-private-registry.example.com/myimage --confirm

Current Result

Fails with:
error: tag latest failed: you may not have access to the Docker image "my-private-registry.example.com/myimage"

Expected Result

Everything should work, tags should be retrieved so should the images in the registry

Additional Information

Solution can be to add an additional secret with the port (443) and correct ImageStream's --from by also adding the port (443). This leads to some inconsistency on how to pull/push an image and looks like a regression as stated in following comment:
#9584 (comment)

The text was updated successfully, but these errors were encountered:

miminar · 2017-11-08T15:43:35Z

Thank you @zonArt for the report. Do I understand you reproducer correctly that importing image from the registry without :443 while having secret with the same registry name (also without the suffix) does not work? But importing from the same registry suffixed by :443 while having a corresponding secret also with this port does work?

zonArt · 2017-11-08T16:27:27Z

@miminar Exactly

miminar · 2017-11-09T12:21:22Z

Unfortunately I could not reproduce locally. I can import from a secured registry requiring authentication just fine:

oc import-image hello-openshift --confirm --from=miminarnb.vm/miminar/hello-openshift:latest
The import completed successfully.

Name:                   hello-openshift
Namespace:              pjoe
Created:                Less than a second ago
Labels:                 <none>
Annotations:            openshift.io/image.dockerRepositoryCheck=2017-11-09T12:17:32Z
Docker Pull Spec:       172.30.30.30:5000/pjoe/hello-openshift
Image Lookup:           local=false
Unique Images:          1
Tags:                   1

latest
  tagged from miminarnb.vm/miminar/hello-openshift:latest

  * miminarnb.vm/miminar/hello-openshift@sha256:17f81671da6a948b39b1f1ad497ccc154b4da2bf7d1a04e7635a8c01dd9b00e5
      Less than a second ago

Is your private registry behind some load balancer/proxy?

miminar · 2017-11-09T12:22:32Z

Used version:

oc v3.6.173.0.72-1+1c16b83-dirty
kubernetes v1.6.1+5115d708d7
features: Basic-Auth

Server https://f25-ose2.vm:8443
openshift v3.6.173.0.72-1+1c16b83-dirty
kubernetes v1.6.1+5115d708d7

Lowering priority.

zonArt · 2017-11-09T13:08:30Z

@miminar Yes the registry is behing an nginx reverse proxy, the registry is hosted on JFrog Artifactory and I'm using the domain option to set nginx configuration which I can provide if needed. The nginx is running on another machine.

On strange thing I notice is that as you put --from=miminarnb.vm/xxxxx you still have a reference to <ip>:<port> in the Docker Pull Spec (and strangely on another path)

miminar · 2017-11-09T13:58:53Z

@zonArt it would be awesome if you could provide the nginx' configuration.

The Docker Pull Spec: 172.30.30.30:5000/pjoe/hello-openshift is an internal location for pull. If using this pull spec instead of the external one (miminarnb.*), the image will be pulled-through by the integrated registry and mirrored locally.

zonArt · 2017-11-09T14:09:38Z

Here you are:

###########################################################
## this configuration was generated by JFrog Artifactory ##
###########################################################

## server configuration
server {
    listen 443 ssl;

    ## add ssl entries when https has been set in config
    ssl_certificate      /etc/ssl/localcerts/wildcard.my-private-registry.example.com.crt;
    ssl_certificate_key  /etc/ssl/private/wildcard.my-private-registry.example.com.key;
    ssl_session_cache shared:SSL:1m;
    ssl_prefer_server_ciphers   on;


    server_name ~(?<repo>.+)\.my-private-registry.example.com my-private-registry.example.com;

    if ($http_x_forwarded_proto = '') {
        set $http_x_forwarded_proto  $scheme;
    }
    ## Application specific logs
    access_log /var/log/nginx/my-private-registry.example.com-access.log;
    error_log /var/log/nginx/my-private-registry.example.com-error.log;
    rewrite ^/$ /artifactory/webapp/ redirect;
    rewrite ^/artifactory/?(/webapp)?$ /artifactory/webapp/ redirect;
    rewrite ^/(v1|v2)/(.*) /artifactory/api/docker/$repo/$1/$2;
    chunked_transfer_encoding on;
    client_max_body_size 0;
    location /artifactory/ {
    proxy_read_timeout  900;
    proxy_pass_header   Server;
    proxy_cookie_path   ~*^/.* /;
    if ( $request_uri ~ ^/artifactory/(.*)$ ) {
        proxy_pass          http://artifactory.example.com:8081/artifactory/$1;
    }
    proxy_pass          http://artifactory.example.com:8081/artifactory/;
    proxy_set_header    X-Artifactory-Override-Base-Url $http_x_forwarded_proto://$host:$server_port/artifactory;
    proxy_set_header    X-Forwarded-Port  $server_port;
    proxy_set_header    X-Forwarded-Proto $http_x_forwarded_proto;
    proxy_set_header    Host              $http_host;
    proxy_set_header    X-Forwarded-For   $proxy_add_x_forwarded_for;
    }
}

miminar · 2017-11-14T17:32:51Z

I thought I could reproduce just by putting nginx in front of upstream registry running locally on my host, but I failed. It still works for me. My nginx configuration is attached below.

user  nginx;
worker_processes  1;

error_log  stderr;
pid        /var/run/nginx.pid;

events {
    worker_connections  1024;
}

http {
    include       /etc/nginx/mime.types;
    default_type  application/octet-stream;

    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';

    access_log  /var/log/nginx/access.log  main;

    sendfile        on;
    #tcp_nopush     on;

    keepalive_timeout  65;

    #gzip  on;

    include /etc/nginx/conf.d/*.conf;

    server {
        listen 443 ssl;

        ssl_certificate      /etc/ssl/ssl.crt;
        ssl_certificate_key  /etc/ssl/ssl.key;
        ssl_session_cache   shared:SSL:1m;
        ssl_prefer_server_ciphers   on;

        server_name registry.miminarnb.vm;

        if ($http_x_forwarded_proto = '') {
            set $http_x_forwarded_proto  $scheme;
        }

        # zonArt settings
        chunked_transfer_encoding on;
        client_max_body_size 0;
        location / {
            proxy_pass http://172.17.0.2:5000;
        }
        proxy_read_timeout  900;
        proxy_pass_header   Server;
        proxy_cookie_path   ~*^/.* /;
        proxy_set_header    X-Forwarded-Port  $server_port;
        proxy_set_header    X-Forwarded-Proto $http_x_forwarded_proto;
        proxy_set_header    Host              $http_host;
        proxy_set_header    X-Forwarded-For   $proxy_add_x_forwarded_for;
    }
}

The import happens from inside a VM:

$ oc secrets new-dockercfg miminarnb-proxy --docker-password=$PW --docker-username=minami --docker-server=https://miminarnb.vm [email protected]
$ oc import-image mpython:v1 --from miminarnb.vm/miminar/python:v1 --confirm --insecure=true
The import completed successfully.

Name:                   mpython
Namespace:              default
Created:                About an hour ago
Labels:                 <none>
Annotations:            openshift.io/image.dockerRepositoryCheck=2017-11-14T17:21:25Z
Docker Pull Spec:       172.30.30.30:5000/default/mpython
Image Lookup:           local=false
Unique Images:          1
Tags:                   1

v1
  tagged from miminarnb.vm/miminar/python:v1
    will use insecure HTTPS or HTTP connections

  * miminarnb.vm/miminar/python@sha256:53ad28b90e91e474217c33358be1b2c3899252ea9aca5ab708ba5dc3caa75329
      Less than a second ago

I guess I will need to start playing with artifactory in order to reproduce. Suggestions to complicate this setup for an error to occur are welcome.

For most common setups, the import-image seems to do the job though.

zonArt · 2017-11-15T02:04:54Z

@miminar Thanks for trying to help me understand what's not working with my setup. I'm not sure adding nginx in front of your registry really changes something especially when looking at your server_name value compared to the --from in the import-image command. Are you sure you pass through the nginx ? Another suggestion would maybe also try not having the internal docker registry in the Docker Pull Spec (though I might totally understand that wrong and it has no impact but still....)

The main reason of the nginx/apache reverse proxy in front of Artifactory is to be able to handle multiple private registry on Artifactory (that's why the ~(?<repo>.+) in the server_name). I guess you'll better understand it while playing with Artifactory (issue from which I started this one was also mentioning Artifactory so it may not be impossible that my case is strongly linked with its usage).

miminar · 2017-11-15T14:20:12Z

I'm pretty sure the request goes through the nginx proxy. The Docker Pull Spec is unrelated to the problem - I've explained its meaning above.

So it seems the only option left to reproduce the issue is to start using artifactory. But before I do, could you please try to catch headers between nginx and master using wireshark/tcpdump or similar? You may need to switch to insecure connection first.

zonArt · 2017-11-20T09:19:59Z

Hi, sorry I'm quite busy during these two weeks won't have much time to provide you with the data you asked for. I'll try to get them as soon as I'm back next Monday. Thanks for your help

helletheone · 2017-11-30T22:03:19Z

we have the same problem with our registrie

miminar · 2017-12-01T09:13:02Z

@helletheone can you elaborate? Do you use artifactory as well?

helletheone · 2017-12-04T12:49:16Z

yes, we use Artifactory in version 5.4.2 and we have the same issue ...
but we use the "port" methode in Artifactory like this:

docker pull / push repo.example.com:<REPOSITORY_PORT>/:

helletheone · 2017-12-04T14:41:28Z

is maybe the problem the --docker-email ? is the field for any use? because we dont use the email for login

mhofstetter · 2017-12-14T20:06:03Z

@zonArt thanks for pointing in the right direction!

@helletheone

the docker email shouldn't be the problem.

we're using artifactory with the reverse proxy method "sub domain" which results in the following url's:

artifactory-url: artifactory.example.com:443
docker-registry-url for local artifactory repository "test": test.artifactory.example.com

we had to define two secrets in openshift - one for getting the manifest infos from artifactory through the "artifactory-url" and a second one for actually pulling the image from the docker-registry-url (also linked to the serviceaccount default)

oc secrets new-dockercfg artifactory --docker-server='artifactory.example.com:443' ...
oc secrets new-dockercfg test-registry --docker-server='test.artifactory.example.com' ...
oc secrets link serviceaccount/default secrets/test-registry --for=pull

i would expect, that with the reverse proxy method "port" you have to define the two secrets the same way, just with different ports - once with the port your artifactory instance is listening (443) and the other one with the port of the registry you defined in the setup of the local repository.

oc secrets new-dockercfg artifactory --docker-server='artifactory.example.com:443' ...
oc secrets new-dockercfg test-registry --docker-server='artifactory.example.com:<REPOSITORY_PORT>' ...
oc secrets link serviceaccount/default secrets/test-registry --for=pull

@miminar
i think this is artifactory specific by its way of handling multiple docker-registries in one artifactory instance by offering the two methods "subdomain" and "port" for adressing these registries. the following was the logoutput from the master on loglevel8 during an import-image without the proper secrets.

importer.go:378] importing remote Docker repository registry=https://test.artifactory.example repository=testgroup/testname insecure=false
round_trippers.go:383] GET https://test.artifactory.example.com/v2/
round_trippers.go:390] Request Headers:
round_trippers.go:408] Response Status: 401 Unauthorized in 41 milliseconds
round_trippers.go:411] Response Headers:
...
round_trippers.go:414]     Www-Authenticate: Bearer realm="https://artifactory.example.com:443/artifactory/api/docker/test/v2/token",service="artifactory.example.com:443"
...
credentials.go:186] Unable to find a secret to match https://artifactory.example.com:443/artifactory/api/docker/test/v2/token (artifactory.example.com:443/artifactory/api/docker/test/v2/token)

maybe it would be even better to define the secret for the manifest infos a little bit more strict. e.g. artifactory.example.com:443/artifactory/api/docker/test instead of just artifactory.example.com:443 for just matching this registry.

miminar · 2017-12-18T08:30:34Z

@mhofstetter thank you for the detailed report. This needs to be fixed together with #9730.

zonArt · 2017-12-18T10:22:57Z

@mhofstetter Thanks for this very well detailed explanation. I'll check if I have the same (but obviously I would say it's similar). Anyway, it really helped me a lot understand (and accept) why both secret are needed.

hendrikhalkow · 2018-01-05T07:46:20Z

Hello. I am using Artifactory behind Nginx as well and I have the same issue, except that adding :443 doesn't change anything.

Here is what I did:

DOCKER_CREDENTIALS="`
  --docker-username=...
  --docker-password=...
  --docker-email=...
"
oc secrets new-dockercfg docker-n7lab-io \
  --docker-server=docker.n7lab.io \
  ${DOCKER_CREDENTIALS}
oc secrets new-dockercfg artifactory-n7lab-io \
  --docker-server=artifactory.n7lab.io \
  ${DOCKER_CREDENTIALS}
oc secrets new-dockercfg docker-n7lab-io-443 \
  --docker-server=docker.n7lab.io:443 \
  ${DOCKER_CREDENTIALS}
oc secrets new-dockercfg artifactory-n7lab-io-443 \
  --docker-server=artifactory.n7lab.io:443 \
  ${DOCKER_CREDENTIALS}


# Lets link them all together to show that none of this works ;)
ACCOUNTS=(default builder deployer)
IMAGES=(docker-n7lab-io docker-n7lab-io-443 artifactory-n7lab-io artifactory-n7lab-io-443)
for A in "${ACCOUNTS[@]}"; do
  for I in "${IMAGES[@]}"; do
    echo "${A} ${I}"
    oc secrets link "${A}" "${I}"
  done
done

Now when I do

oc import-image my-hello --from=docker.n7lab.io/n7lab/hello --confirm

or

oc import-image my-hello-443 --from=docker.n7lab.io:443/n7lab/hello --confirm

I get an error message:

The import completed with errors.

Name:			my-hello-443
Namespace:		default
Created:		4 minutes ago
Labels:			<none>
Annotations:		openshift.io/image.dockerRepositoryCheck=2018-01-05T07:39:43Z
Docker Pull Spec:	<none>
Image Lookup:		local=false
Unique Images:		0
Tags:			1

latest
  tagged from docker.n7lab.io:443/n7lab/hello

  ! error: Import failed (InternalError): Internal error occurred: Get https://docker.n7lab.io:443/v2/n7lab/hello/manifests/latest: unknown: Authentication is required
      4 minutes ago

error: tag latest failed: Internal error occurred: Get https://docker.n7lab.io:443/v2/n7lab/hello/manifests/latest: unknown: Authentication is required

On the servcer, I get two requests:
The first one on https://docker.n7lab.io/v2/, the second one on https://artifactory.n7lab.io/api/docker/docker/v2/token?scope=repository%3An7lab%2Fhello%3Apull&service=artifactory.n7lab.io – both without any credentials, so both return 401.

In the openshift logs, i get a

credentials.go:214] Unable to find a secret to match https://artifactory.n7lab.io/api/docker/docker/v2/token (artifactory.n7lab.io/api/docker/docker/v2/token)

in both cases.

bparees · 2018-01-19T22:49:05Z

@miminar is this covered by https://trello.com/c/o8tqoSAp ?

If not, should it be? (should that card be extended to cover this)

miminar · 2018-01-22T08:06:01Z

@bparees it definitely should be. (I've already referenced the issue in the card).

bparees · 2018-01-22T15:11:54Z

part of https://trello.com/c/o8tqoSAp

miminar self-assigned this Nov 8, 2017

miminar added component/image kind/bug Categorizes issue or PR as related to a bug. priority/P1 labels Nov 8, 2017

miminar added priority/P2 component/image and removed priority/P1 component/image labels Nov 9, 2017

miminar mentioned this issue Dec 18, 2017

oc import-image --from=registry cross-cross:latest --confirm --insecure #9730

Open

zonArt mentioned this issue Dec 20, 2017

Openshift import-image call via REST API #7226

Closed

bparees closed this as completed Jan 22, 2018

donovat mentioned this issue May 30, 2023

Cannot create an Image Steam with import-image from private registry (ignores secret). #27952

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ImageStream pull on external private docker registry #17238

ImageStream pull on external private docker registry #17238

zonArt commented Nov 8, 2017 •

edited

Loading

miminar commented Nov 8, 2017

zonArt commented Nov 8, 2017

miminar commented Nov 9, 2017

miminar commented Nov 9, 2017

zonArt commented Nov 9, 2017

miminar commented Nov 9, 2017

zonArt commented Nov 9, 2017

miminar commented Nov 14, 2017

zonArt commented Nov 15, 2017

miminar commented Nov 15, 2017

zonArt commented Nov 20, 2017

helletheone commented Nov 30, 2017

miminar commented Dec 1, 2017

helletheone commented Dec 4, 2017 •

edited

Loading

helletheone commented Dec 4, 2017

mhofstetter commented Dec 14, 2017 •

edited

Loading

miminar commented Dec 18, 2017

zonArt commented Dec 18, 2017

hendrikhalkow commented Jan 5, 2018 •

edited

Loading

bparees commented Jan 19, 2018

miminar commented Jan 22, 2018 •

edited

Loading

bparees commented Jan 22, 2018

ImageStream pull on external private docker registry #17238

ImageStream pull on external private docker registry #17238

Comments

zonArt commented Nov 8, 2017 • edited Loading

Version

Steps To Reproduce

Current Result

Expected Result

Additional Information

miminar commented Nov 8, 2017

zonArt commented Nov 8, 2017

miminar commented Nov 9, 2017

miminar commented Nov 9, 2017

zonArt commented Nov 9, 2017

miminar commented Nov 9, 2017

zonArt commented Nov 9, 2017

miminar commented Nov 14, 2017

zonArt commented Nov 15, 2017

miminar commented Nov 15, 2017

zonArt commented Nov 20, 2017

helletheone commented Nov 30, 2017

miminar commented Dec 1, 2017

helletheone commented Dec 4, 2017 • edited Loading

helletheone commented Dec 4, 2017

mhofstetter commented Dec 14, 2017 • edited Loading

miminar commented Dec 18, 2017

zonArt commented Dec 18, 2017

hendrikhalkow commented Jan 5, 2018 • edited Loading

bparees commented Jan 19, 2018

miminar commented Jan 22, 2018 • edited Loading

bparees commented Jan 22, 2018

zonArt commented Nov 8, 2017 •

edited

Loading

helletheone commented Dec 4, 2017 •

edited

Loading

mhofstetter commented Dec 14, 2017 •

edited

Loading

hendrikhalkow commented Jan 5, 2018 •

edited

Loading

miminar commented Jan 22, 2018 •

edited

Loading