[Docs] Struggling to get Keycloak working as external authorization server

[Jeinhaus]

Description

Hi everyone 👋,
we are struggling to get flyte working with Keycloak external authorization.
We got the authentication working but are very lost on the authorization part.

The authentication setup worked just as documented, but we can't get the external Authorization Server working.

In the Custom Authorization Server documentation for keycloak it says we should create new Client Scopes. As a Keycloak beginner I'm struggling to understand if there should be anything configured in the Client Scope and how this ties together with the Keycloak clients that are created in step 4 of the Keycloak documentation.

For now, we just tried this with local port-forward, so our flytectl config looks like this

admin:
  endpoint: dns:///localhost:8089
  authType: Pkce
  insecure: true
logger:
  show-source: true
  level: 100

Our three Keycloak clients look like this (terraform code):

resource "keycloak_openid_client" "flyte" {
  realm_id = var.realm_id

  client_id   = "flyte"
  name        = "Flyte"
  description = "Client for the Flyte application"

  access_type              = "CONFIDENTIAL"
  service_accounts_enabled = true
  standard_flow_enabled    = true
  valid_redirect_uris = [
    "https://flyte.${var.base_domain_name}/callback",
    "http://localhost:30081/callback",
  ]

  base_url = "https://flyte.${var.base_domain_name}/callback"

  full_scope_allowed = true
}

resource "keycloak_openid_client" "flytectl" {
  realm_id = var.realm_id

  client_id   = "flytectl"
  name        = "flytectl"
  description = "Client for the Flytectl"

  access_type           = "PUBLIC"
  standard_flow_enabled = true
  valid_redirect_uris = [
    "http://localhost:53593/callback",
  ]

  base_url = "https://flyte.${var.base_domain_name}/callback"

  full_scope_allowed = true
}

resource "keycloak_openid_client" "flyte_propeller" {
  realm_id = var.realm_id

  client_id   = "flyte-propeller"
  name        = "flytePropeller"
  description = "Client for flytePropeller"

  access_type           = "CONFIDENTIAL"
  standard_flow_enabled = true
  valid_redirect_uris = [
    "https://flyte.${var.base_domain_name}/callback",
  ]

  base_url = "https://flyte.${var.base_domain_name}/callback"

  full_scope_allowed = true
}

And the Client Scope looks like this:

resource "keycloak_openid_client_scope" "openid_client_scope" {
  realm_id               = var.realm_id
  name                   = "flyte-access"
  include_in_token_scope = true
}

The relevant configuration for the flyte-binary chart then looks like this:

auth:
  enabled: true
  oidc:
    baseUrl: https://<keycloak-domain>/auth/realms/<realm>
    clientId: flyte
    clientSecret: <secret>
  authorizedUris:
    - https://flyte.<base_domain>
  internal:
    clientId: flyte-propeller
    clientSecret: <secret>
    clientSecretHash: <secretHash>
inline:
  auth:
    appAuth:
      authServerType: External
      externalAuthServer:
        baseUrl:  https://<keycloak-domain>/auth/realms/<realm>
        metadataUrl: .well-known/openid-configuration
        allowedAudience: https://flyte.<base_domain>
      thirdPartyConfig:
        flyteClient:
          # Use the clientID generated by your IdP for the `flytectl` app registration
          clientId: flytectl
          redirectUri: http://localhost:53593/callback
          scopes:
          - offline
          - all
    userAuth:
      openId:
      # baseUrl: https://<keycloak-url>/auth/realms/<keycloak-realm> # Uncomment for Keycloak and update with your installation host and realm name
      # baseUrl: https://login.microsoftonline.com/<tenant-id>/oauth2/v2.0/authorize # Uncomment for Azure AD
      # For Okta, use the Issuer URI of the custom auth server:
        baseUrl: https://<keycloak-domain>/auth/realms/<realm>
        scopes:
        - profile
        - openid
      # - offline_access # Uncomment if your IdP supports issuing refresh tokens (optional)
      # Use the client ID and secret generated by your IdP for the first OIDC registration in the "Identity Management layer : OIDC" section of this guide
        clientId: flyte

When we try to connect to flyte using flytectl we get the following error:

❯ flytectl create project --id "test" --name "Test"
{"json":{"src":"client.go:63"},"level":"info","msg":"Initialized Admin client","ts":"2024-02-21T07:49:48+01:00"}
{"json":{"src":"auth_interceptor.go:86"},"level":"debug","msg":"Request failed due to [rpc error: code = Unauthenticated desc = token parse error [JWT_VERIFICATION_FAILED] Could not retrieve id token from metadata, caused by: rpc error: code = Unauthenticated desc = Request unauthenticated with IDToken]. If it's an unauthenticated error, we will attempt to establish an authenticated context.","ts":"2024-02-21T07:49:48+01:00"}
{"json":{"src":"auth_interceptor.go:91"},"level":"debug","msg":"Request failed due to [Unauthenticated]. Attempting to establish an authenticated connection and trying again.","ts":"2024-02-21T07:49:48+01:00"}
{"json":{"src":"token_source_provider.go:151"},"level":"warning","msg":"Failed fetching from cache. Will restart the flow. Error: no token found in the cache","ts":"2024-02-21T07:49:48+01:00"}
{"json":{"src":"auth_flow_orchestrator.go:77"},"level":"info","msg":"Opening the browser at https://<keycloak-domain>/auth/realms/<realm>/protocol/openid-connect/auth?client_id=flytectl\u0026redirect_uri=http%3A%2F%2Flocalhost%3A53593%2Fcallback\u0026response_type=code\u0026scope=offline+all\u0026code_challenge=<...>\u0026code_challenge_method=S256\u0026nonce=<...>\u0026state=<...>c","ts":"2024-02-21T07:49:48+01:00"}

Where the browser then gives this error:

Flyte Authentication
Error!
Error: invalid_scope
Error Hint:
Description: Invalid scopes: offline all

We already tried the Client scope mappers from this Slack thread, but to no avail.

Can anyone help? 🙈

Thank you,
Julian

P.S.: I posted this here already.

Are you sure this issue hasn't been raised already?

• Yes

Have you read the Code of Conduct?

• Yes

[welcome]

Thank you for opening your first issue here! 🛠

[dosubot]

🤖

Based on the error message you're seeing, it seems like the scopes requested by Flyte do not match the ones configured in Keycloak. The "invalid_scope" error usually occurs when the scopes requested by the client application do not match the scopes configured in the authorization server.

In your Flyte configuration, the scopes requested by Flyte are configured with the flags "userAuth.openId.scopes" and "appAuth.thirdPartyConfig.flyteClient.scopes".

To resolve the "invalid_scope" error, you should ensure that the scopes requested by Flyte match the ones configured in Keycloak. Here are the steps you can follow:

1. Check the scopes requested by Flyte. You can find these in your Flyte configuration file or in the command line flags when starting your Flyte server. Look for the flags "userAuth.openId.scopes" and "appAuth.thirdPartyConfig.flyteClient.scopes".

2. Check the scopes configured in Keycloak. You can find these in your Keycloak admin console, under the client settings for Flyte.

3. Ensure that the scopes requested by Flyte match the ones configured in Keycloak. If they do not match, you should update the scopes in either Flyte or Keycloak so that they do match.

If you continue to experience issues after following these steps, please provide more information about your Flyte and Keycloak configuration so that we can better assist you.

Also, there was a similar issue reported in the Flyte repository: [Docs] Additional Keycloak configuration settings. The issue was resolved with recent updates to the auth docs and the support for the audience parameter. You might find this issue helpful.

For more details, you can refer to the Flyte configuration flags in the source code: flyteadmin/auth/config/config_flags.go.

Sources

• [Docs] Additional Keycloak configuration settings
• flyteadmin/auth/config/config_flags.go

About Dosu

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

[ddl-ebrown]

I'm in the process of setting up Keycloak as well, and there are 2 things of note here. I saw and resolved the same issue:

• You need to add the all client scope in Keycloak and make sure it's available to any Keycloak clients that need it (flytepropeller, etc)

• The docs mention scopes like this under thirdPartyConfig

scopes:
  - offline
  - all

In Keycloak, offline is actually called offline_access, so you'll need to make that change for flytepropeller / flytescheduler to be able to contact Keycloak to generate tokens

[RaghavMangla]

Hi, I would like to work on this

[davidmirror-ops]

@RaghavMangla great, please let us know any questions you may have or if you need help. Thanks!