Followup: Error "Invalid Roles" during Handshake when connecting to broker via TLS with JWT-Auth

[Loahrs]

Since I can't reopen Issue #431 I am creating a new one as followup. Somehow there was no "reopen" button. See the referenced Issue for more details.

I'm not able to connect my client to the pulsar cluster via a proxy using authentication and authorization after deploying it from the helm chart.

pulsar.Client("pulsar+ssl://<proxy-address>:6651", 
                     authentication=pulsar.AuthenticationToken("<token of admin superuser>"), 
                     tls_allow_insecure_connection=True,   #for testing purpose
                     tls_validate_hostname=False)

The error seems to persist even after correcting the "workaround" to be valid yaml, as suggested in #431
I tried recreating all pods and even restarting my Kubernetes Cluster.

When applying the workaround (#430 ) the authentication seems to work correctly. I got the following error message BEFORE applying the workaround:

2024-01-16T10:18:00,153+0000 [pulsar-io-4-2] WARN org.apache.pulsar.broker.authorization.AuthorizationService - [/10.224.1.10:45766] Illegal combination of role [proxy-admin] and originalPrincipal [admin]: cannot specify originalPrincipal when connecting without valid proxy role.

AFTER applying the workaround I get another error message. It logs a succesful authentication, but then throws "Invalid Roles". Might this be an issue with the authorization?

2024-01-16T12:13:55,394+0000 [pulsar-proxy-io-2-4] INFO org.apache.pulsar.proxy.server.ProxyConnection - [/10.224.1.30:10697] New connection opened
#Authentication is succesful:
2024-01-16T12:13:55,476+0000 [pulsar-proxy-io-2-4] INFO org.apache.pulsar.proxy.server.ProxyConnection - [/10.224.1.30:10697] complete connection, init proxy handler. authenticated with token role admin, hasProxyToBrokerUrl: false
2024-01-16T12:13:55,511+0000 [pulsar-proxy-io-2-3] INFO org.apache.pulsar.client.impl.ConnectionPool - [[id: 0x73eeed53, L:/10.224.1.10:60466 - R:pulsar-broker.pulsar.svc.cluster.local/10.224.1.2:6651]] Connected to server
#Something went wrong, possibly during authorization? 
2024-01-16T12:13:55,531+0000 [pulsar-proxy-io-2-3] WARN org.apache.pulsar.client.impl.ClientCnx - [id: 0x73eeed53, L:/10.224.1.10:60466 - R:pulsar-broker.pulsar.svc.cluster.local/10.224.1.2:6651] Received error from server: Invalid roles.
2024-01-16T12:13:55,531+0000 [pulsar-proxy-io-2-3] WARN org.apache.pulsar.client.impl.ClientCnx - [id: 0x73eeed53, L:/10.224.1.10:60466 - R:pulsar-broker.pulsar.svc.cluster.local/10.224.1.2:6651] Received unknown request id from server: -1
2024-01-16T12:13:55,531+0000 [pulsar-proxy-io-2-3] INFO org.apache.pulsar.client.impl.ClientCnx - [id: 0x73eeed53, L:/10.224.1.10:60466 ! R:pulsar-broker.pulsar.svc.cluster.local/10.224.1.2:6651] Disconnected
2024-01-16T12:13:55,531+0000 [pulsar-proxy-io-2-3] WARN org.apache.pulsar.client.impl.ConnectionPool - [[id: 0x73eeed53, L:/10.224.1.10:60466 ! R:pulsar-broker.pulsar.svc.cluster.local/10.224.1.2:6651]] Connection handshake failed: org.apache.pulsar.client.api.PulsarClientException: Connection already closed

My current .yaml:

namespace: "pulsar"
namespaceCreate: false
clusterName: "cluster-a"

initialize: false

components:
    pulsar_manager: true
    
#workaround, see https://github.com/apache/pulsar-helm-chart/issues/427
broker:
  configData:
    proxyRoles: 'proxy-admin'

tls:
  enabled: true
  proxy:
    enabled: true
  broker:
    enabled: true
  zookeeper:
    enabled: true


# issue selfsigning certs
certs:
  internal_issuer:
    enabled: true
    type: selfsigning

pulsar_manager:
  configData:
    ENV_SPRING_CONFIGURATION_FILE: "/pulsar-manager/pulsar-manager/application.properties"
    SPRING_CONFIGURATION_FILE: "/pulsar-manager/pulsar-manager/application.properties"
    PULSAR_MANAGER_OPTS: " -Dlog4j2.formatMsgNoLookups=true"
  replicaCount: 1
  resources:
    requests:
      memory: 650Mi   

auth:
  authentication:
    enabled: true
    provider: "jwt"
    jwt:
      # Enable JWT authentication
      # If the token is generated by a secret key, set the usingSecretKey as true.
      # If the token is generated by a private key, set the usingSecretKey as false.
      usingSecretKey: true
  authorization:
    enabled: true
  superUsers:
    # broker to broker communication
    broker: "broker-admin"
    # proxy to broker communication
    proxy: "proxy-admin"
    # pulsar-admin client to broker/proxy communication
    client: "admin"

When disabling authorization (and having authentication enabled) I am able to make a sucessful connection from my client.
Am I missing some additional configuration?
Thank you for your help so far!

[lhotari]

AFTER applying the workaround I get another error message. It logs a succesful authentication, but then throws "Invalid Roles". Might this be an issue with the authorization?

Most likely this is a problem with the token that the proxy uses towards the broker. It has to be a "proxy-admin" token.
There was a user on Pulsar Slack with a similar problem where a broker token was used on the proxy.
https://apache-pulsar.slack.com/archives/C5Z4T36F7/p1705403994897379?thread_ts=1705072691.562239&cid=C5Z4T36F7

It's also worth checking your broker's effective configuration. You can run grep Roles /pulsar/conf/broker.conf in a shell inside the broker pod to check that proxyRoles and superUserRoles are expected ones.

[Loahrs]

When I use the "proxy-admin" superuser token created as a secret when setting up the cluster (taken from the secret: "pulsar-token-proxy-admin" I still get the error. From the logs I can see that it authenticates as proxy-admin:

2024-01-17T11:30:49,718+0000 [pulsar-proxy-io-2-3] INFO org.apache.pulsar.proxy.server.ProxyConnection - [/10.224.1.47:5649] complete connection, init proxy handler. authenticated with token role proxy-admin, hasProxyToBrokerUrl: false

I connected to one of my brokers and ran grep Roles /pulsar/conf/broker.conf:

#role as proxyRoles - it will demand to see a valid original principal.
proxyRoles=proxy-admin
superUserRoles=admin,broker-admin,proxy-admin

My proxy still logs:

2024-01-17T11:30:49,762+0000 [pulsar-proxy-io-2-2] WARN org.apache.pulsar.client.impl.ClientCnx - [id: 0x1e3fd8a8, L:/10.224.1.6:35394 - R:pulsar-broker.pulsar.svc.cluster.local/10.224.1.54:6651] Received error from server: Invalid roles.

Could the proxy-admin Token be misconfigured somehow? How could I check that?
Also I could try completely deleting everything in my cluster and installing again from scratch instead of using helm upgrades.

[lhotari]

When I use the "proxy-admin" superuser token created as a secret when setting up the cluster

You shouldn't be using the proxy-admin token yourself. The proxy should be configured to use it. This is what the chart does by default with brokerClientAuthenticationParameters: "file:///pulsar/tokens/proxy/token":

apiVersion: v1
kind: ConfigMap
metadata:
  name: "release-name-pulsar-proxy"
  namespace: default
  labels:
    app: pulsar
    chart: pulsar-3.1.0
    release: release-name
    heritage: Helm
    cluster: release-name-pulsar
    component: proxy
data:
  clusterName: release-name-pulsar
  statusFilePath: "/pulsar/status"
  # prometheus needs to access /metrics endpoint
  webServicePort: "8080"
  servicePort: "6650"
  brokerServiceURL: pulsar://release-name-pulsar-broker:6650
  brokerWebServiceURL: http://release-name-pulsar-broker:8080

  # Authentication Settings
  authenticationEnabled: "true"
  # disable authorization on proxy and forward authorization credentials to broker
  authorizationEnabled: "false"
  forwardAuthorizationCredentials: "true"
  superUserRoles: admin,broker-admin
  # token authentication configuration
  authenticationProviders: "org.apache.pulsar.broker.authentication.AuthenticationProviderToken"
  brokerClientAuthenticationParameters: "file:///pulsar/tokens/proxy/token"
  brokerClientAuthenticationPlugin: "org.apache.pulsar.client.impl.auth.AuthenticationToken"
  tokenPublicKey: "file:///pulsar/keys/token/public.key"
  PULSAR_GC: |
    -XX:+UseG1GC -XX:MaxGCPauseMillis=10 -Dio.netty.leakDetectionLevel=disabled -Dio.netty.recycler.linkCapacity=1024 -XX:+ParallelR
efProcEnabled -XX:+UnlockExperimentalVMOptions -XX:+DoEscapeAnalysis -XX:ParallelGCThreads=4 -XX:ConcGCThreads=4 -XX:G1NewSizePercen
t=50 -XX:+DisableExplicitGC -XX:-ResizePLAB -XX:+ExitOnOutOfMemoryError -XX:+PerfDisableSharedMem
  PULSAR_MEM: |
    -Xms64m -Xmx64m -XX:MaxDirectMemorySize=64m
  httpNumThreads: "8"

and in proxy's statefulset:

        - name: proxy-token
          secret:
            secretName: "release-name-token-proxy-admin"
            items:
              - key: TOKEN
                path: proxy/token

The token should have the role proxy-admin.

Use this command in a proxy pod shell to check the content and role of the token:

./bin/pulsar tokens show -f /pulsar/tokens/proxy/token

[Loahrs]

Thank you!
Running ./bin/pulsar tokens show -f /pulsar/tokens/proxy/token throws a NullPointerException for me:

I have no name!@pulsar-proxy-0:/pulsar$ ./bin/pulsar tokens show -f /pulsar/tokens/proxy/token
java.lang.NullPointerException: Cannot invoke "Object.getClass()" because "value" is null

I copied the token from the file and ran ./bin/pulsar tokens show -i <token> and it returned:

{"alg":"RS256","typ":"JWT"}
---
{"sub":"proxy-admin"}

I verified that this is the token I am using with my client.

You shouldn't be using the proxy-admin token yourself.

You are right. I am fairly new to pulsar, so I took that token to test authorization/authentication, assuming that it will just work since it is a superuser token. Creating a separate token for my client will be a next step 😅

Using the proxy-admin token should work (even if it's a very bad practice) right?

[lhotari]

Using the proxy-admin token should work (even if it's a very bad practice) right?

I don't think it works when the client uses a proxy role via a proxy. Forwarding authentication to the broker from the proxy is only possible for proxy roles.

[lhotari]

I verified that this is the token I am using with my client.

That's the problem. You cannot use that token on the client.

[Loahrs]

I verified that this is the token I am using with my client.

That's the problem. You cannot use that token on the client.

Ah, now I got it. Thank you.
So I think we can close it, since it's not a bug, just me being a beginner 😅