[bug] Unable to connect to dashboard using Dask-Gateway and Labextension

[viniciusdc]

Describe the bug

Running the following on qhub v0.3.13 and v0.3.14 generates an accessible link for the dask cluster dashboards, but we are unable to reuse that link to access the dask extension.

from dask_gateway import Gateway
gateway = Gateway()

options = gateway.cluster_options()
options

cluster = gateway.new_cluster(options)
cluster

successful generates a working URL for the dashboards (can be accessed):

Dashboard URL does not work:

Expected behavior

• The Dask lab extension should connect to the dask-gateway spawned cluster

How to reproduce

• Launch a dask-gateway cluster and include the generated Dashboard URI in the Dask extension bar

[costrouc]

@viniciusdc has ideas for future work on this. For now we are going to move this to the v0.4.x release. Ideally the clusters should just show up without users needing to add the url.

[dharhas]

Do we know why it isn't working and how to fix?

Ideally the clusters should just show up without users needing to add the url. <- It is fine to push this to a later date.

But being able to add the url manually should be fixed in my opinion for 0.4 release. Folks use this and it is important for demos. How difficult is the fix? @viniciusdc @costrouc

[costrouc]

Personally I don't understand why that feature isn't working. When I looked at the http traffic it is making a request and getting a reposonse. I don't see any 400/500 http errors.

[costrouc]

Seeing some redirect when I try to visit the dashboard without the status

[costrouc]

We will look at this issue for another 2 hours and then likely make the decision to move this to the 0.4.1 release.

[viniciusdc]

Moving this to v0.4.x

[dharhas]

@viniciusdc @costrouc This is still a regression. And is used by @rsignell-usgs heavily and by me and others in demo's what is the plan for investigating and fixing.

[viniciusdc]

Hi @dharhas, sorry for the delay with this, here are some details regarding this issue:

• The dask-gateway link provisioned with the client does connect and successfully generates a dashboard,
• The problem definitely lies with the dask-labextension not being able to fully connect with it. The reason for this is still not known. our best assumption is that our proxy might be affecting the connection.

I will open a detailed issue on the lab-extension repo, as they might have a solution that we could implement.

[costrouc]

@viniciusdc is going to test this theory out but I believe I know the issue. The user is authenticated in the browser via the single sign on flow. This means that the cookies are stored in the browser not on the jupyterlab server.

The dask lab extension has two parts code that runs in the browser and code that runs on the server. This check https://github.com/dask/dask-labextension/blob/main/dask_labextension/dashboardhandler.py is running on the server which does not have credentials.

The way to test this is to run the code that is done on the lab extension and run it on both the server and browser. If this theory is correct the browser code will be the only one to run properly. I think that the lab extension broke once we started using forward auth in a v0.3.1X version.

[viniciusdc]

This issue seems to corroborate our hypothesis dask/dask-labextension#190

[viniciusdc]

So as we originally had expected, here is a simple test suit that shows the above-mentioned issue (now verified):

• First start a Gateway cluster

from dask_gateway import Gateway

gateway = Gateway()
options = gateway.cluster_options()
options

• To mimic what lab-extension does (see the corresponding function here) we can then proceed with:

import requests

def fetch(url):
    "Added this just in case we got stuck with redirects"
    request = requests.get(url)
    if request.status_code != 200:
        print(request.headers)
        msg = 'Error Code: %i on request'
        raise Exception(msg % request.status_code)
    return request

class Request():
    protocol = ""
    host = ""

url = _normalize_dashboard_link(parse.unquote(url), Request())  # just copied the function from source
individual_plots_url = url_path_join(url,  "individual-plots.json",)

individual_plots_response = fetch(individual_plots_url)

# inspect fetch response, look for individual plots response
response_content = individual_plots_response.content

try:
    assert individual_plots_response.headers.get('content-type') == 'application/json'
except AssertionError as err:
    with open("response.html", "w+") as file:
        file.write(individual_plots_response.text)

from the above code snippet we got the following body response (rendered HTML from response.html):

• Now mimicking the browser behavior (this is, when the user access the dashboard link)

const response = await fetch('https://quansight-beta.qhub.dev/gateway/clusters/dev.c05f4b01641747cfbf67a34712990ef5/individual-plots.json')
console.log(await response.json())

Wich returns the expected individual-plots json:

{"Individual Bandwidth Workers": "/individual-bandwidth-workers", "Individual Progress": "/individual-progress", "Individual Gpu Utilization": "/individual-gpu-utilization", "Individual Graph": "/individual-graph", "Individual Groups": "/individual-groups", "Individual Nprocessing": "/individual-nprocessing", "Individual Compute Time Per Key": "/individual-compute-time-per-key", "Individual Workers Network Bandwidth": "/individual-workers-network-bandwidth", "Individual Workers": "/individual-workers", "Individual Cpu": "/individual-cpu", "Individual Aggregate Time Per Action": "/individual-aggregate-time-per-action", "Individual Gpu Memory": "/individual-gpu-memory", "Individual Cluster Memory": "/individual-cluster-memory", "Individual Profile": "/individual-profile", "Individual Workers Memory": "/individual-workers-memory", "Individual Memory By Key": "/individual-memory-by-key", "Individual Task Stream": "/individual-task-stream", "Individual Profile Server": "/individual-profile-server", "Individual Bandwidth Types": "/individual-bandwidth-types", "Individual Scheduler System": "/individual-scheduler-system", "Individual Occupancy": "/individual-occupancy", "Individual Cluster Map": "/statics/individual-cluster-map.html"}

[viniciusdc]

Just a bit more info, from ForwardAuth we can inspect the redirection to Keycloak for authentication:

• Request made from the lab-extension DashboardChecker mechanism commented above:

│ time="2022-03-09T13:33:57Z" level=debug msg="Authenticating request" cookies="[]" handler=Auth host=quansight-beta.qhub.dev method=GET proto=https rule=default source_ip=10.128.0.112 uri=/gateway/clusters/dev.297cbc79d5494a979741d8236e42d295/individual-plots.json

• Redirection from the ForwardAuth middleware

time="2022-03-09T13:33:57Z" level=debug msg="Set CSRF cookie and redirected to provider login url" csrf_cookie="_forward_auth_csrf_5266dc=<redacted>; Path=/; Domain=quansight-beta.qhub.dev; Expires=Wed, 09 Mar 2022 14:33:57 GMT; HttpOnly; Secure" handler=Auth host=quansight-beta.qhub.dev login_url="https://quansight-beta.qhub.dev/auth/realms/qhub/protocol/openid-connect/auth?client_id=forwardauth&redirect_uri=https%3A%2F%2Fquansight-beta.qhub.dev%2F_oauth&response_type=code&scope=profile+email&state=5266dc636a7b214c52862012f6b54d7f%3Ageneric-oauth%3Ahttps%3A%2F%2Fquansight-beta.qhub.dev%2Fgateway%2Fclusters%2Fdev.297cbc79d5494a979741d8236e42d295%2Findividual-plots.json" method=GET proto=https rule=default source_ip=10.128.0.112 uri=/gateway/clusters/dev.297cbc79d5494a979741d8236e42d295/individual-plots.json 

[viniciusdc]

@costrouc i think this issue still needs to be open right? as this is related to how labextension handles the oauth requests...

[viniciusdc]

I will be testing what was proposed in dask/dask-labextension#190 (comment)

[jabbera]

@viniciusdc did you ever have a chance to try the workaround? I'm having the same issue.

[viniciusdc]

@viniciusdc did you ever have a chance to try the workaround? I'm having the same issue.

Hi @jabbera, sorry for the late feedback on this. Yes, the workaround fixed the issue. To accomplish this just add an else if in here with a fetch. don't forget to add the same return object as the MakeRequest function. I will propose a PR to lab-extension with those changes.

[viniciusdc]

Some feedback on this, we have the fix already merged into dask-labextension repository and we will be including this in the next release! dask/dask-labextension#229.

[viniciusdc]

to close this we need:

• pin dask-labextensionversion to pip+git install the following commit dask/dask-labextension/commit/fc8de03447778e0605d0abafa3d3135f9e49e0d6 in Jupyterlab base environment.
• add the enable settings directly to the dask-labextension config file:
  • this can be done by adding the config trait directly to jupyterhub using the extension config syntax to a new configmap similar to this that will be mounted in the pod`s dir.