VAULT [AUTH (k8s)] Login Exception after updating to 4.0.0

[zsmeier]

Hi all,
After updating our project to quarkus 3.9.1 (using vault 4.0.0) we get an error on the startup of the service in kubernetes:

VAULT [AUTH (k8s)] Login' at path 'https://vault.company-intern.de/v1/auth/kubernetes/login' with status 403
errors:
permission denied
java.lang.RuntimeException: Failed to start quarkus
	at io.quarkus.runner.ApplicationImpl.doStart(Unknown Source)
	at io.quarkus.runtime.Application.start(Application.java:101)
	at io.quarkus.runtime.ApplicationLifecycleManager.run(ApplicationLifecycleManager.java:111)
	at io.quarkus.runtime.Quarkus.run(Quarkus.java:71)
	at io.quarkus.runtime.Quarkus.run(Quarkus.java:44)
	at io.quarkus.runner.GeneratedMain.main(Unknown Source)
	at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
	at java.base/java.lang.reflect.Method.invoke(Method.java:580)
	at io.quarkus.bootstrap.runner.QuarkusEntryPoint.doRun(QuarkusEntryPoint.java:62)
	at io.quarkus.bootstrap.runner.QuarkusEntryPoint.main(QuarkusEntryPoint.java:33)
Caused by: io.quarkus.runtime.configuration.ConfigurationException: Failed to read configuration properties
	at io.quarkus.deployment.steps.RuntimeConfigSetup.deploy(Unknown Source)
	... 10 more
Caused by: VaultClientException{operationName='VAULT [AUTH (k8s)] Login', requestPath='https://vault.company-intern.de/v1/auth/kubernetes/login', status=403, errors=[permission denied]}
	at io.quarkus.vault.client.http.VaultHttpClient.throwVaultException(VaultHttpClient.java:78)
	at io.quarkus.vault.client.http.VaultHttpClient.lambda$buildResponse$0(VaultHttpClient.java:33)
	at java.base/java.util.concurrent.CompletableFuture.uniApplyNow(CompletableFuture.java:684)
	at java.base/java.util.concurrent.CompletableFuture.uniApplyStage(CompletableFuture.java:662)
	at java.base/java.util.concurrent.CompletableFuture.thenApply(CompletableFuture.java:2200)
	at java.base/java.util.concurrent.CompletableFuture$MinimalStage.thenApply(CompletableFuture.java:2948)
	at io.quarkus.vault.client.http.VaultHttpClient.buildResponse(VaultHttpClient.java:29)
	at io.quarkus.vault.client.http.jdk.JDKVaultHttpClient.lambda$execute$1(JDKVaultHttpClient.java:31)
	at java.base/java.util.concurrent.CompletableFuture$UniCompose.tryFire(CompletableFuture.java:1150)
	at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
	at java.base/java.util.concurrent.CompletableFuture.postFire(CompletableFuture.java:614)
	at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:844)
	at java.base/java.util.concurrent.CompletableFuture$Completion.exec(CompletableFuture.java:483)
	at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:387)
	at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1312)
	at java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1843)
	at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1808)

I could'nt find anything in the release notes, which explains this, so my question is, did something changed in the configuration that i'm missing?
One possible error would be, that the jwtToken is not found, but in that case, I think, the error message would be different. And the default path to the jwtToken is the same in 4.0.0 as in 3.5.0.
The "role" and "authMountPath" are defined and everything was working with quarkus 3.8.x and vault 3.5.0.

Any help is appreciated, thanks is advance.
Zsolt.

[zsmeier]

I think you may have a Bug with the new Client implementation in 4.0.0....
Even if I set the environment variable "QUARKUS_VAULT_AUTHENTICATION_KUBERNETES_AUTH_MOUNT_PATH" to "some_auth_mount_path", all the request are executed with the default "kubernetes" value.
The value of the authMountPath configuration property is set to "some_auth_mount_path", but ist not used in the client:

VaultClientProducer.java:

                case KUBERNETES:
                    builder.kubernetes(
                            VaultKubernetesAuthOptions.builder()
                                    .role(authConfig.kubernetes().role().orElseThrow())
                                    .jwtTokenPath(Path.of(authConfig.kubernetes().jwtTokenPath()))
                                    .caching(config.renewGracePeriod())
                                    .build());
                    break;

The authConfig has the value of the environment variable, but is not set in the Builder VaultKubernetesAuthOptions. The default value of the mounthPath is "kubernetes":

VaultKubernetesAuthOptions.Builder.java

private String mountPath = DEFAULT_KUBERNETES_MOUNT_PATH;

Is there a way to look at this please?
Thanks.

[kdubb]

@zsmeier It's a bug... opened #267 for issue and #268 for fix.

[kdubb]

@zsmeier Can you verify PR #268 fixes your issue (I'm fairly certain it does).

[zsmeier]

Look good, thank you!