Dask Gateway Dashboard link giving "Service Unavailable" error

[maqs00d-s]

We have deployed qhub ver 0.4.2 using GKE as our platform. After a user authenticates through Keycloak and starts the Jupyterlab server, a jupyter notebook is launched and the user attempts to start a dask cluster, as per the steps listed here: https://docs.qhub.dev/en/stable/source/user_guide/dask_gateway.html#starting-a-cluster

When the user enters the command: cluster = gateway.new_cluster(options), he/she is presented with the dashboard link in the GUI (https://our-domain/gateway/clusters/namespace.resource-id/status). However, when clicking on the link, the user receives an error saying "Service Unavailable."

Before receiving the error, there is an oauth redirect back to Keycloak. Not sure if this is expected behavior, but after entering credentials and authenticating, a blank page with the error above is presented.

[iameskild]

Hi @maqs00d-s, thanks for reaching out! Have you by chance added a SSL cert to your deployment? This dask dashboard URL needs the SSL cert for it to work correctly.

[maqs00d-s]

hi @iameskild thanks for the quick reply! We have now added an SSL cert issued by Gandi, but still get the same "Service Unavailable" error page. The certificate is displaying correctly in the browser and is valid (i.e., not expiring) until 2023. Any other suggestions? Thanks again for your help on this....

[iameskild]

Hi @maqs00d-s, it's hard to say what the problem might be. Have you checked the qhub-traefik-ingress pod logs? If you continue to have this problem, could I ask that you open an issue and add include whatever relevant logs you are able to capture; tools like kubectl or k9s can be helpful here. Thanks :)

[amjad489]

I work with @maqs00d-s and to get more detailed logs i have enabled debug level logging and also enabled access logs on proxy. installed certificate is wildcard certificate and there's no issue with it
following are the logs of traefik.

time="2022-10-13T08:33:01Z" level=debug msg="Serving default certificate for request: \"datalab-staging.malariagen.net\""
time="2022-10-13T08:33:01Z" level=debug msg="http: TLS handshake error from 10.0.2.1:62268: remote error: tls: bad certificate"
time="2022-10-13T08:33:01Z" level=debug msg="Serving default certificate for request: \"datalab-staging.malariagen.net\""
time="2022-10-13T08:33:01Z" level=debug msg="http: TLS handshake error from 10.0.2.1:53698: remote error: tls: bad certificate"
time="2022-10-13T08:33:01Z" level=debug msg="Remote error http://forwardauth-service:4181. StatusCode: 503" middlewareName=gitlab-traefik-forward-auth@kubernetescrd middlewareType=ForwardedAuthType
10.0.2.1 - - [13/Oct/2022:08:33:01 +0000] "GET /_oauth?state=d28bbe6836b2e261eb7f5976747d8066%3Ageneric-oauth%3Ahttps%3A%2F%2Fdatalab-staging.malariagen.net%2Fgateway%2Fclusters%2Fgitlab.6e8fc10eadf24529b5bc773480f9d788%2Fstatus&session_state=f6ca5738-adfc-4d95-816b-60a2d641e2a7&code=a6f7e5fa-8031-4ea6-8bc0-ccae817dcae8.f6ca5738-adfc-4d95-816b-60a2d641e2a7.61a205a4-e46c-45c0-a749-8e50d6e6bc88 HTTP/2.0" 503 20 "-" "-" 716 "gitlab-forwardauth-93537ac368eb20081ed2@kubernetescrd" "-" 35ms

[amjad489]

there's error happening in forward-auth at the same time:

time="2022-10-13T12:11:54Z" level=error msg="Code exchange failed with provider" error="Post https://datalab-staging.malariagen.net/auth/realms/qhub/protocol/openid-connect/token: x509: certificate signed by unknown authority" handler=AuthCallback host=datalab-staging.malariagen.net method=GET proto=https rule=default source_ip=10.154.0.82 uri="/_oauth?state=6d1ccc3322083b166077469b02c080b5%3Ageneric-oauth%3Ahttps%3A%2F%2Fdatalab-staging.malariagen.net%2Fgateway%2Fclusters%2Fgitlab.5692a9f94a294416b1ec423fd3de58db%2Fstatus&session_state=b6038bce-b56e-4e60-853a-3b4907e0f356&code=d6464fda-8465-4743-818a-2ea07416d5bf.b6038bce-b56e-4e60-853a-3b4907e0f356.61a205a4-e46c-45c0-a749-8e50d6e6bc88"

[iameskild]

@amjad489 @maqs00d-s
can you confirm that the cert is working for all other endpoints? and what does the certificate section of your qhub-config.yaml look like?