Authentication does not work with JupyterHub 4.1.2

[trungleduc]

Bug description

I'm using the local-binder-local-hub setup with auth enabled (c.BinderHub.auth_enabled = True). This setup works well up to JupyterHub 4.0.2.

Starting from 4.1.2, it does not work anymore with the following errors

[I 2024-04-09 15:50:00.086 JupyterHub log:192] 200 POST /hub/api/oauth2/token ([email protected]) 33.50ms
[D 240409 15:50:00 auth:527] HubAuth cache miss: token:76615381b56649daa983eeb0db8d467c:88eec0b396b0b2a4c89d43d6edcd220c8146120210ba068ec8f6c9231cc00ea6
[I 2024-04-09 15:50:00.101 JupyterHub log:192] 200 GET /hub/api/user ([email protected]) 13.31ms
[D 240409 15:50:00 auth:538] Received request from Hub user {'groups': [], 'kind': 'user', 'admin': False, 'name': 'alice', 'session_id': '76615381b56649daa983eeb0db8d467c', 'scopes': ['access:services!service=binder', 'read:users:groups!user=alice', 'read:users:name!user=alice']}
[I 240409 15:50:00 auth:1527] Logged-in user {'groups': [], 'kind': 'user', 'admin': False, 'name': 'alice', 'session_id': '76615381b56649daa983eeb0db8d467c', 'scopes': ['access:services!service=binder', 'read:users:groups!user=alice', 'read:users:name!user=alice']}
[D 240409 15:50:00 auth:1195] Setting oauth cookie for 127.0.0.1: service-binderhub, {'path': '/services/binder/', 'httponly': True}
[I 240409 15:50:00 _xsrf_utils:125] Setting new xsrf cookie for b'76615381b56649daa983eeb0db8d467c:88eec0b396b0b2a4c89d43d6edcd220c8146120210ba068ec8f6c9231cc00ea6' {'path': '/services/binder/'}
[D 240409 15:50:00 auth:977] Not accepting cookie auth on GET /services/binder/oauth_callback: '_xsrf' argument missing from GET
[D 240409 15:50:00 auth:781] No user identified

and in the browser:

GET http://127.0.0.1:8000/services/binder/oauth_callback?code=RjcOLjkaqwPffgVJsEN4h6RqWOX1RF&state=eyJ1dWlkIjogIjRm...QUQifQ net::ERR_TOO_MANY_REDIRECTS

How to reproduce

• Install JupyterHub 4.1.2
• Set up binderhub with local-binder-local-hub
• Switch c.JupyterHub.authenticator_class to DummyAuthenticator
• Add c.BinderHub.auth_enabled = True to binderhub_config.py
• Launch any repository

Expected behaviour

The builder process should start

Actual behaviour

Your personal set up

• binderhub at main
• jupyterhub 4.1.2

Full environment

# paste output of `pip freeze` or `conda list` here

Configuration

# jupyterhub_config.py

Logs

[welcome]

Thank you for opening your first issue in this project! Engagement like this is essential for open source projects! 🤗

If you haven't done so already, check out Jupyter's Code of Conduct. Also, please try to follow the issue template as it helps other other community members to contribute more effectively.

You can meet the other Jovyans by joining our Discourse forum. There is also an intro thread there where you can stop by and say Hi! 👋

Welcome to the Jupyter community! 🎉

[trungleduc]

It might be related to jupyterhub/jupyterhub#4761. @minrk Is there any config I need to add to make it work?

[manics]

I've tried this with JupyterHub 4.1.5. I also get a failure, though it's different to yours:

[I 2024-04-09 18:19:02.677 JupyterHub log:192] 302 GET / -> /hub/ (@127.0.0.1) 1.06ms
[I 2024-04-09 18:19:02.689 JupyterHub log:192] 302 GET /hub/ -> /services/binder/ ([email protected]) 4.10ms
[D 240409 18:19:02 auth:838] No user identified
[D 240409 18:19:02 auth:1395] Redirecting to login url: /hub/api/oauth2/authorize?client_id=service-binder&redirect_uri=%2Fservices%2Fbinder%2Foauth_callback&response_type=code
[W 240409 18:19:02 auth:1160] Detected unused OAuth state cookies
[I 240409 18:19:02 log:135] 302 GET /services/binder/ -> /hub/api/oauth2/authorize?client_id=service-binder&redirect_uri=%2Fservices%2Fbinder%2Foauth_callback&response_type=code&state=[secret] (@127.0.0.1) 1.59ms
[D 2024-04-09 18:19:02.709 JupyterHub provider:421] Validating client id service-binder
[D 2024-04-09 18:19:02.710 oauthlib.oauth2.rfc6749.grant_types.authorization_code authorization_code:362] Validating redirection uri /services/binder/oauth_callback for client service-binder.
[D 2024-04-09 18:19:02.710 oauthlib.oauth2.rfc6749.grant_types.base base:231] Using provided redirect_uri /services/binder/oauth_callback
[D 2024-04-09 18:19:02.710 JupyterHub provider:497] validate_redirect_uri: client_id=service-binder, redirect_uri=/services/binder/oauth_callback
[D 2024-04-09 18:19:02.711 oauthlib.oauth2.rfc6749.grant_types.base base:172] Validating access to scopes ['read:users:groups!user', 'read:users:name!user', 'access:services!service=binder'] for client 'service-binder' (<OAuthClient(identifier='service-binder')>).
[D 2024-04-09 18:19:02.711 JupyterHub provider:624] Allowing request for scope(s) for service-binder:  read:users:groups!user,read:users:name!user,access:services!service=binder
[E 2024-04-09 18:19:02.712 JupyterHub auth:289] User <User(as 0/1 running)> not allowed to access JupyterHub service binder
[W 2024-04-09 18:19:02.712 JupyterHub web:1873] 403 GET /hub/api/oauth2/authorize?client_id=service-binder&redirect_uri=%2Fservices%2Fbinder%2Foauth_callback&response_type=code&state=<state> (127.0.0.1): You do not have permission to access JupyterHub service binder
[D 2024-04-09 18:19:02.712 JupyterHub base:1513] No template for 403
[W 2024-04-09 18:19:02.713 JupyterHub log:192] 403 GET /hub/api/oauth2/authorize?client_id=service-binder&redirect_uri=%2Fservices%2Fbinder%2Foauth_callback&response_type=code&state=[secret] ([email protected]) 8.15ms

[minrk]

There are fixes since 4.1.2 that are probably relevant, specifically jupyterhub/jupyterhub#4759 in 4.1.4, I believe. I do not think jupyterhub/jupyterhub#4761 is related, since that was specifically a CORS request that is now correctly blocked.

The error @manics sees is correctly raising because the test user doesn't have permission to visit the service (access:services!service=binder).

[trungleduc]

Just tested with 4.1.5, I still have the same issue

[dlamoris]

I have an internal k8s deployment that have similar symptoms, adding some data point to hopefully help with debugging.

Using binderhub chart version 1.0.0-0.dev.git.3431.h8a8a59e with jupyterhub/k8s-binderhub image 1.0.0-0.dev.git.3422.h35a34dd, I get the same error "Failed to connect to event stream" when trying to launch a repo. Attached is the har file from chrome's dev tool network. You'll see from the Get call to https://cae-binder-uat.jpl.nasa.gov/services/binder/build/gh/dlam/conda/main it gets a return of 302 and starts an oauth loop that results in too many redirects. This is interesting because the jupyterhub is 4.0.2 here (it's using quay.io/jupyterhub/k8s-hub:3.2.1 which should have 4.0.2)

bad-binder.har.zip

Our last known working version of binderhub image is 1.0.0-0.dev.git.2809.h97a8ce1, and if I use that image with everything else in the chart staying the same, the call and launch works and instead of 302 returns 200. I have not tried other images in between that and latest. (although 1.0.0-0.dev.git.2816.h5db2f98 results in a 500 when going to binder page)

sample chart value of the deployment, using chart 1.0.0-0.dev.git.3431.h8a8a59e:

service:
  type: ClusterIP
image:
  name: jupyterhub/k8s-binderhub
  tag: 1.0.0-0.dev.git.2809.h97a8ce1
registry:
  password: <replaced>
  url: <url>
  username: <username>
imageBuilderType: dind
ingress:
  enabled: false # using jupyterhub's ingress
config:
  BinderHub:
    debug: true
    base_url: "/services/binder/" # from jupyterhub proxy
    use_registry: true
    auth_enabled: true
    hub_url: "https://cae-binder-uat.jpl.nasa.gov/" # this refers to jupyterhub
    image_prefix: <prefix>
    build_image: "quay.io/jupyterhub/repo2docker:2023.06.0"
  DockerRegistry:
    url: "url"
    auth_config_url: "url"
    token_url: "url"
  KubernetesBuildExecutor:
    build_image: "quay.io/jupyterhub/repo2docker:2023.06.0" 
  GitHubRepoProvider:
    access_token: <replaced>
    api_base_path: https://{hostname}/api/v3
    hostname: github.jpl.nasa.gov
jupyterhub:
  proxy:
    https:
      enabled: false
    secretToken: <secret>
    service:
      type: ClusterIP
  debug:
    enabled: true
  hub:
    config:
      BinderSpawner:
        auth_enabled: true
      Authenticator:
        admin_users:
          - admin
        enable_auth_state: true
      JupyterHub:
        admin_access: true
        authenticator_class: generic-oauth
      GenericOAuthenticator:
        allow_all: true
        authorize_url: >-
          <url>
        client_id: cae-binder-uat
        client_secret: <secret>
        login_service: CAE SSO
        oauth_callback_url: https://cae-binder-uat.jpl.nasa.gov/hub/oauth_callback
        scope:
          - openid
        token_url: >-
          <url>
        userdata_params:
          state: state
        userdata_url: >-
          <url>
        username_key: preferred_username

    allowNamedServers: true
    namedServerLimitPerUser: 10
    redirectToServer: false
    services:
      binder:
        oauth_no_confirm: true
        url: "http://binder.cae-binder-uat.svc.cluster.local" # jupyterhub will add /services/binder/ in its proxy to this url
        oauth_redirect_uri: "https://cae-binder-uat.jpl.nasa.gov/services/binder/oauth_callback"
        oauth_client_id: "service-cae-binder-uat-oauth-client-uat"
    loadRoles:
      binder:
        scopes:
          - servers
          - admin:users
        services:
          - binder
      user:
        scopes:
          - self
          - access:services
    extraConfig:
      01-hub_extra: |
        c.JupyterHub.default_url = '/services/binder/'
      02-timeout: |
        c.Spawner.http_timeout = 1800
  singleuser:
    cmd: jupyterhub-singleuser
    defaultUrl: "/lab"
  ingress:
    enabled: true
    hosts:
      - cae-binder-uat.jpl.nasa.gov
    pathSuffix: null
    pathType: Prefix
    tls:
      - hosts:
          - cae-binder-uat.jpl.nasa.gov
        secretName: cae-binder-uat

[minrk]

#1847 should fix this.