enableSSLVerification for GitlabEventSource is ignored

[mchebitou]

Describe the bug
when I created a new gitlab eventsource I am getting the following error :

main {"level":"error","ts":1622298560.0473602,"logger":"argo-events.eventsource","caller":"webhook/webhook.go:234","msg":"error occurred while performing post route activation operations","eventSourceName":"gitlab","eventSourceType":"gitlab","eventName":"demo","eventSourceName":"gitlab","eventName":"demo","error":"failed to list existing hooks to check for duplicates for project id 12383: Get \"https://gitlab.mydomain/api/v4/projects/12383/hooks\": x509: certificate signed by unknown authority"

even though I have set enableSSLVerification to false.
I have also decoded the content of the variable EVENTSOURCE_OBJECT in the gitlab-eventsource deployment created by the controller and noticed that the enableSSLVerification parameter was ignored and is not even part of the passed configuration:

{"metadata":{"name":"gitlab","namespace":"argo-events","creationTimestamp":null},"spec":{"service":{"ports":[{"port":12000,"targetPort":12000}]},"gitlab":{"demo":{"webhook":{"endpoint":"/push","method":"POST","port":"12000","url":"http://gitlab-event-source.mydomain"},"projectID":"12383","events":["PushEvents","TagPushEvents"],"accessToken":{"name":"gitlab-access","key":"token"},"gitlabBaseURL":"https://gitlab.mydomain"}}},"status":{}}

To Reproduce
Steps to reproduce the behavior:

1. Create an eventsource using this definition:

apiVersion: argoproj.io/v1alpha1
kind: EventSource
metadata:
  name: gitlab
spec:
  service:
    ports:
      - port: 12000
        targetPort: 12000
  gitlab:
    demo:
      projectID: "12383"
      webhook:
        endpoint: /push
        port: "12000"
        method: POST
        url: http://gitlab-event-source.mydomain
      events:
        - PushEvents
        - TagPushEvents
      accessToken:
        key: token
        name: gitlab-access
      enableSSLVerification: false
      gitlabBaseURL: https://gitlab.mydomain

Expected behavior
Expecting the gitlab-eventsource pod to be up and running with a successful connection to Gitlab.

Environment (please complete the following information):

• Kubernetes: v1.17.5
• Argo: v3.0.2
• Argo Events: v1.3.0

---

Message from the maintainers:

If you wish to see this enhancement implemented please add a 👍 reaction to this issue! We often sort issues this way to know what to prioritize.

[whynowy]

I believe enableSSLVerification is not used for that purpose, but indicates if gitlab server enableSSLVerification when sending events to the event source.

I looked into the source code, the error happens when accessing gitlab server to list exiting hooks (Gitlab API), where should not throw an error like that - unless the gitlab server really uses en invalid TLS cert. Could you let me know if the issue still exists? If yes, could you manually access the URL in your browser to see if there's CERT validation warning.

[mchebitou]

Our GitLab uses a certificate provided by an internal certification authority, so the certificate is valid but I couldn't find any way to make the event source trust this certificate since it is just like using a self signed certificate, any ideas how to achieve that ?

[whynowy]

Our GitLab uses a certificate provided by an internal certification authority, so the certificate is valid but I couldn't find any way to make the event source trust this certificate since it is just like using a self signed certificate, any ideas how to achieve that ?

If that's the case, one way I can think of is, you mount a volume with .ssh/known_hosts to /etc/ssh (or any other files needed for TLS communication).

You can specify spec.template.volumes and spec.template.container.volumeMounts in your event source spec.

[mchebitou]

I don't see how I can do it since I have no control over the deployment which is created by the controller.

[whynowy]

Specifying spec.template.volumes and spec.template.container.volumeMounts in your event source spec will give you that control.

[mchebitou]

mounting a volume with .ssh/known_hosts to /etc/ssh was not going to work because eventsource pod is using https to contact GitLab and not ssh, so the certificate needs to be added to the system truststore.
I was able to solve the problem by getting the original /etc/ssl/certs/ca-certificates.crt from an alpine then appended my own certificates, after that I created a ConfigMap with the new ca-certificates.crt file and updated the eventsource like this :

apiVersion: argoproj.io/v1alpha1
kind: EventSource
metadata:
  name: gitlab
spec:
  service:
    ports:
      - port: 12000
        targetPort: 12000
  gitlab:
    demo:
      projectID: "12383"
      webhook:
        endpoint: /push
        port: "12000"
        method: POST
        url: http://gitlab-event-source.mydomain
      events:
        - PushEvents
        - TagPushEvents
      accessToken:
        key: token
        name: gitlab-access
      enableSSLVerification: false
      gitlabBaseURL: https://gitlab.mydomain
  template:
    container:
      volumeMounts:
        - name: certs-volume
          mountPath: /etc/ssl/certs
    volumes:
      - name: certs-volume
        configMap:
          name: ca-certificates

Thank you @whynowy for your help, you pointed me to the right direction :-)

[whynowy]

Great!

[whynowy]

@mchebitou - converted to a discussion as I think it might help others.