Setup Artifactory HA cluster in AWS with Helm

[bsemperlotti]

Hello,

I have found this documentation explaining how to set Artifactory HA cluster in AWS: https://jfrog.com/knowledge-base/how-to-setup-artifactory-ha-cluster-in-aws/

I am having troubles to get the same type of setup using the stable/artifactory-ha helm chart.

How to configure properly SSL termination on the LoadBlancer ?
How to configure properly the nginx reverse proxy to handle docker with SSL ?

[jainishshah17]

@bsemperlotti Are you using EKS?

[bsemperlotti]

Hello,

No, we do not use EKS, we have our own Kubernetes cluster on AWS.

To summurize my issue :

I want whatever works the best between the SubDomain or PrefixPath method for docker registries to work with SSL termination on the LoadBalancer as we do not want to manage certificates files outside of AWS.

• My tests with PrefixPath:

I'm trying to configure artifactory using ingress rule,SSL termination is done by AWS ELB. The ingress controller is a Traefik.

I can access the UI in HTTPS without any issue.

Using tthe embedded server, the httpPort is stuck to -1 (also when requesting webserver trough REST api) but is set 80 in the configuration xml file.

docker login fails with:

docker login -u admin -p 'xxxx' artifactory.int.cloud.talend.com/docker-local 

Error response from daemon: Get https://artifactory.int.cloud.talend.com/v2/: unable to decode token response: EOF

My supposition:
It seems that artifactory is misconfigured and does not receive the request of the docker client correctly and so does not send the appropriate redirection url, thus interpreting the error response as a token

• My tests with SubDomain:

I'm trying to configure artifactory using nginx as reverse proxy with SSL termination on AWS ELB. (I have submitted a Pull Request to support annotations on nginx service #110)

I can access the UI in HTTPS without any issue.

Docker login fails because artifactory replies with a very strange URL pattern (it automatically includes the 80 port):

docker login -u admin -p 'xxxx' docker-local.artifactory.int.cloud.talend.com  

Error response from daemon: Get https://docker-local.artifactory.int.cloud.talend.com/v2/: Get https://docker-local.artifactory.int.cloud.talend.com:80/artifactory/api/docker/docker-local/v2/token?account=admin&client_id=docker&offline_token=true&service=docker-local.artifactory.int.cloud.talend.com%3A80: http: server gave HTTP response to HTTPS client

If I manually request the docker registry with the correct url, I can see that my setup is OK:

curl https://docker-local.artifactory.int.cloud.talend.com/artifactory/api/docker/docker-local/v2/token

{"token":"AKCp5bBXgbcZBexxxxxxxxxxxxxxxxxxxxRn6MDEfDG2D1Se","expires_in":3600}%

So without artifactory inserting the wrong port automatically, I think it should work ?

[jainishshah17]

@bsemperlotti Have you tried ingress this way?
https://medium.com/@jainishshah17/kubernetes-nginx-ingress-with-artifactory-on-gke-e6781d4ab5e8

[bsemperlotti]

Hi thank you for this documentation. I was not aware of your blog and I'm sure I will find useful information.

I don't think I can use this method because we already have an ingress controller which is TraeFik with all configuration properly setup (automatic DNS record creation, ELB SSL termination, Http to Https redirection, limited security groups, etc)

With the method you mention, I will bypass our existing ingress controller to install a new one in parallel.

However, If I understand well, the important part should be:

nginx.ingress.kubernetes.io/configuration-snippet: |
      rewrite ^/(v2)/token /artifactory/api/docker/null/v2/token;
      rewrite ^/(v2)/([^\/]*)/(.*) /artifactory/api/docker/$2/$1/$3;

I will check if I can inject the same type of config in traefik but I do not really understand why it is necessary, why the embedded server does not make this rewriting itself ?

[bsemperlotti]

Seems that Traefik cannot do url rewriting so I definitely can't use this method

[bsemperlotti]

Otherwise, I had a call with JFrog's support today.

We think that NGinx configuration can be causing the error where the wrong 80 port is added to the request.
This line seems erroneous to us:
proxy_set_header X-Artifactory-Override-Base-Url $http_x_forwarded_proto://$host:$server_port/artifactory;

According to AWS documentation about ELB it should be better to use the already defined $http_x_forwarded_port instead of $server_port when $http_x_forwarded_port is defined the same way it is already done with $http_x_forwarded_proto

What do you think ?

[jainishshah17]

@bsemperlotti Looks good. Have you tested applying this changes in nginx?

[bsemperlotti]

@jainishshah17
I can't, Nginx configuration is set by Artifactory application when editing "HTTP Settings" and is constantly overridden if a manual modification is made in the Nginx container.

[jainishshah17]

@bsemperlotti You can turn it off by setting https://github.com/jfrog/charts/blob/master/stable/artifactory-ha/values.yaml#L312 to true

[bsemperlotti]

@jainishshah17 Disabling the AutoConfigUpdate can work but:

• Having to manually update directly the Nginx containers to update the conf seems really wrong
• If AutoConfigUpdate is disabled, would the IPs of the Artifactory instances be updated if a pod restart ? I think not.

So I would propose to add a parameters in the helm chart allowing to override proxy_set_header X-Artifactory-Override-Base-Url and proxy_set_header X-Forwarded-Port by modifying the updateConf.sh in NGinx container:

Maybe somthing like this:

getReverseProxySnippet () {
    local response
    # Yes we separate declaration and assignment, if not, $? will not be properly set to the result of the curl
    local curl=$(curlAuth)
    local arguments=" --show-error --silent --fail $ART_BASE_URL/api/system/configuration/reverseProxy/nginx"
    response=$( (eval $curl $arguments) 2>&1)
    local responseStatus=$?

    if [ $responseStatus -ne 0 ]; then
        echo "ERROR"
    else
        if [ -z $X_ART_OVERRIDE_BASE_URL ]; then
          response="$(sed "s;X-Artifactory-Override-Base-Url .*;X-Artifactory-Override-Base-Url $X_ART_OVERRIDE_BASE_URL;g" <<< $response)"
        fi
        if [ -z $X_FORWARDED_PORT ]; then
          response="$(sed "s;X-Forwarded-Port .*;X-Forwarded-Port $X_FORWARDED_PORT;g" <<< $response)"
        fi
        echo "$response"
    fi
}

[jainishshah17]

@bsemperlotti We recommend using service name instead of IP address when you disable AutoConfigUpdate. Thanks for this proposed solution. Let us see if we can accommodate this in helm chart.

[bsemperlotti]

@jainishshah17 Thanks, it will be great to have these configuration options in Helm.
Note that the proposed code is just a example of what can be done, it was not actually tested and maybe you have a better solution.

[bsemperlotti]

Close by mistake