error with k8s 1.21

[DraconPern]

After upgrading a k8s from 1.20 to 1.21, I am getting this error in the log

Api call failed with code 400, detailed message: {
"kind": "Status",
"apiVersion": "v1",
"metadata": {

},
"status": "Failure",
"message": "the export parameter, deprecated since v1.14, is no longer supported",
"reason": "BadRequest",
"code": 400
}
ERROR: ERROR: java.lang.RuntimeException: io.kubernetes.client.openapi.ApiException:
hudson.remoting.ProxyException: java.lang.RuntimeException: io.kubernetes.client.openapi.ApiException:
at com.microsoft.jenkins.kubernetes.wrapper.ResourceManager.handleApiExceptionExceptNotFound(ResourceManager.java:180)
at com.microsoft.jenkins.kubernetes.wrapper.V1ResourceManager$StatefulSetUpdater.getCurrentResource(V1ResourceManager.java:970)
at com.microsoft.jenkins.kubernetes.wrapper.V1ResourceManager$StatefulSetUpdater.getCurrentResource(V1ResourceManager.java:958)
at com.microsoft.jenkins.kubernetes.wrapper.ResourceManager$ResourceUpdater.createOrApply(ResourceManager.java:93)
at com.microsoft.jenkins.kubernetes.wrapper.KubernetesClientWrapper.handleResource(KubernetesClientWrapper.java:289)
at com.microsoft.jenkins.kubernetes.wrapper.KubernetesClientWrapper.apply(KubernetesClientWrapper.java:256)
at com.microsoft.jenkins.kubernetes.command.DeploymentCommand$DeploymentTask.doCall(DeploymentCommand.java:172)
at com.microsoft.jenkins.kubernetes.command.DeploymentCommand$DeploymentTask.call(DeploymentCommand.java:124)
at com.microsoft.jenkins.kubernetes.command.DeploymentCommand$DeploymentTask.call(DeploymentCommand.java:106)
at hudson.remoting.UserRequest.perform(UserRequest.java:211)
at hudson.remoting.UserRequest.perform(UserRequest.java:54)
at hudson.remoting.Request$2.run(Request.java:376)
at hudson.remoting.InterceptingExecutorService.lambda$wrap$0(InterceptingExecutorService.java:78)
at java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.lang.Thread.run(Thread.java:829)
Suppressed: hudson.remoting.Channel$CallSiteStackTrace: Remote call to Ubuntu1804
at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1800)
at hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:356)
at hudson.remoting.Channel.call(Channel.java:1001)
at hudson.FilePath.act(FilePath.java:1248)
at com.microsoft.jenkins.kubernetes.command.DeploymentCommand.execute(DeploymentCommand.java:68)
at com.microsoft.jenkins.kubernetes.command.DeploymentCommand.execute(DeploymentCommand.java:45)
at com.microsoft.jenkins.azurecommons.command.CommandService.runCommand(CommandService.java:88)
at com.microsoft.jenkins.azurecommons.command.CommandService.execute(CommandService.java:96)
at com.microsoft.jenkins.azurecommons.command.CommandService.executeCommands(CommandService.java:75)
at com.microsoft.jenkins.azurecommons.command.BaseCommandContext.executeCommands(BaseCommandContext.java:77)
at com.microsoft.jenkins.kubernetes.KubernetesDeploy.perform(KubernetesDeploy.java:42)
at jenkins.tasks.SimpleBuildStep.perform(SimpleBuildStep.java:123)
at hudson.tasks.BuildStepCompatibilityLayer.perform(BuildStepCompatibilityLayer.java:80)
at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:803)
at hudson.model.Build$BuildExecution.build(Build.java:197)
at hudson.model.Build$BuildExecution.doRun(Build.java:163)
at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:513)
at hudson.model.Run.execute(Run.java:1907)
at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
at hudson.model.ResourceController.execute(ResourceController.java:97)
at hudson.model.Executor.run(Executor.java:429)
Caused by: hudson.remoting.ProxyException: io.kubernetes.client.openapi.ApiException:
at io.kubernetes.client.openapi.ApiClient.handleResponse(ApiClient.java:979)
at io.kubernetes.client.openapi.ApiClient.execute(ApiClient.java:895)
at io.kubernetes.client.openapi.apis.AppsV1Api.readNamespacedStatefulSetWithHttpInfo(AppsV1Api.java:8129)
at io.kubernetes.client.openapi.apis.AppsV1Api.readNamespacedStatefulSet(AppsV1Api.java:8105)
at com.microsoft.jenkins.kubernetes.wrapper.V1ResourceManager$StatefulSetUpdater.getCurrentResource(V1ResourceManager.java:967)
... 15 more
ERROR: Kubernetes deployment ended with HasError

[DraconPern]

I have tried to recompile the plugin with 8.0.0 instead of 7.0.0 of the io.kubernetes library but it didn't work. tried using 9.0.0 but couldn't build a working plugin. I think I am going to uninstall this and use the cli instead.

[hassanrazakhalid]

yes same error

[ggmr2212]

Same error here

[Manu10744]

Same here.

[volserverRay]

same here.
1.20.8 working perfectly,
but 1.21 not.

[Trojaner]

tried using 9.0.0 but couldn't build a working plugin. I think I am going to uninstall this and use the cli instead.

I have updated it to 13.0.0 and it still doesn't work
https://github.com/ImperialPlugins/kubernetes-cd-plugin

[mabson]

Same here

[harveychow]

Same here. Noone knows how to fix this?

[DraconPern]

So some ideas on how to fix. In kubernetes-cd-plugin\src\main\java\com\microsoft\jenkins\kubernetes\wrapper\V1beta1ResourceManager.java where there are calls with "getPretty(), true", change to "getPretty(), null". That will set the export parameter to null so that the query parameter doesn't get added. Will need to do this for all V1beta*.java files. That is a hack, since even though my yaml file specify v1 api.. this plugin is still invoking the V1beta functions and not the V1 functions. @Trojaner

[harveychow]

@DraconPern can you tell me where to edit this file? I don't have this file in /opt/jenkins_home/plugins/kubernetes-cd/*
This folder only contains a META-INF, WEB-INF and scripts folder. Scripts folder only includes KubenetesDeployContext.js

[DraconPern]

Ah this line for example.

kubernetes-cd-plugin/src/main/java/com/microsoft/jenkins/kubernetes/wrapper/V1beta1ResourceManager.java

Line 72 in 6665e28

replicaSet = extensionsV1beta1Api.readNamespacedReplicaSet(getName(), getNamespace(),

[harveychow]

Yes I know, but where is that file on the server? I can't seem to find it.

[facupalladino]

@DraconPern You sir deserve a medal. Your comment solved my issue and saved me hours of debugging.
I wish you the best of lucks.

[harveychow]

@facupalladino where did you find this file on the disk? I can't find it in $JENKINS_HOME/plugins/kubernetes-cd-plugin

[facupalladino]

@facupalladino where did you find this file on the disk? I can't find it in $JENKINS_HOME/plugins/kubernetes-cd-plugin

Hello there! You’re not going to find the file in the Jenkins machine as it is builded into a .jar.
You’ll need to download the repo, change the export parameters, build the project and replace the jar in the Jenkins machine.
The file you need to search is kubernetes-cd.jar
If you need the modified jar, let me know and I’ll search the way to send it to you.
Cheers

[harveychow]

@facupalladino That would be awesome! Thanks!

[facupalladino]

@facupalladino That would be awesome! Thanks!

Hello Harvey,
I believe this is the final jar I used https://github.com/facupalladino/kubernetes-cd-jar. You should find and replace (i recommend that you change the old jar name instead of deleting it, in case you want to rollback) the jar in the jenkins server, mine was in $JENKINS_HOME/plugins/kubernetes-cd/WEB-INF/lib
Please, let me know if this solved your issue as well. Cheers!

[harveychow]

@facupalladino thanks. I think it got a step further just to encounter a new issue:

ERROR: ERROR: Can't construct a java object for tag:yaml.org,2002:io.kubernetes.client.openapi.models.V1Deployment; exception=Class not found: io.kubernetes.client.openapi.models.V1Deployment
in 'reader', line 1, column 1:
apiVersion: apps/v1
^

hudson.remoting.ProxyException: Can't construct a java object for tag:yaml.org,2002:io.kubernetes.client.openapi.models.V1Deployment; exception=Class not found: io.kubernetes.client.openapi.models.V1Deployment
in 'reader', line 1, column 1:
apiVersion: apps/v1
^

at org.yaml.snakeyaml.constructor.Constructor$ConstructYamlObject.construct(Constructor.java:336)
at org.yaml.snakeyaml.constructor.BaseConstructor.constructObjectNoCheck(BaseConstructor.java:230)
at org.yaml.snakeyaml.constructor.BaseConstructor.constructObject(BaseConstructor.java:220)
at io.kubernetes.client.util.Yaml$CustomConstructor.constructObject(Yaml.java:337)
at org.yaml.snakeyaml.constructor.BaseConstructor.constructDocument(BaseConstructor.java:174)
at org.yaml.snakeyaml.constructor.BaseConstructor.getSingleData(BaseConstructor.java:158)
at org.yaml.snakeyaml.Yaml.loadFromReader(Yaml.java:491)
at org.yaml.snakeyaml.Yaml.loadAs(Yaml.java:456)
at io.kubernetes.client.util.Yaml.loadAs(Yaml.java:224)
at io.kubernetes.client.util.Yaml.modelMapper(Yaml.java:494)
at io.kubernetes.client.util.Yaml.loadAll(Yaml.java:272)
at com.microsoft.jenkins.kubernetes.wrapper.KubernetesClientWrapper.apply(KubernetesClientWrapper.java:236)
at com.microsoft.jenkins.kubernetes.command.DeploymentCommand$DeploymentTask.doCall(DeploymentCommand.java:172)
at com.microsoft.jenkins.kubernetes.command.DeploymentCommand$DeploymentTask.call(DeploymentCommand.java:124)
at com.microsoft.jenkins.kubernetes.command.DeploymentCommand$DeploymentTask.call(DeploymentCommand.java:106)
at hudson.FilePath.act(FilePath.java:1260)
at com.microsoft.jenkins.kubernetes.command.DeploymentCommand.execute(DeploymentCommand.java:68)
at com.microsoft.jenkins.kubernetes.command.DeploymentCommand.execute(DeploymentCommand.java:45)
at com.microsoft.jenkins.azurecommons.command.CommandService.runCommand(CommandService.java:88)
at com.microsoft.jenkins.azurecommons.command.CommandService.execute(CommandService.java:96)
at com.microsoft.jenkins.azurecommons.command.CommandService.executeCommands(CommandService.java:75)
at com.microsoft.jenkins.azurecommons.command.BaseCommandContext.executeCommands(BaseCommandContext.java:77)
at com.microsoft.jenkins.kubernetes.KubernetesDeploy.perform(KubernetesDeploy.java:42)
at com.microsoft.jenkins.azurecommons.command.SimpleBuildStepExecution.run(SimpleBuildStepExecution.java:54)
at com.microsoft.jenkins.azurecommons.command.SimpleBuildStepExecution.run(SimpleBuildStepExecution.java:35)
at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution.lambda$start$0(SynchronousNonBlockingStepExecution.java:47)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)

Caused by: hudson.remoting.ProxyException: org.yaml.snakeyaml.error.YAMLException: Class not found: io.kubernetes.client.openapi.models.V1Deployment
at org.yaml.snakeyaml.constructor.Constructor.getClassForNode(Constructor.java:669)
at org.yaml.snakeyaml.constructor.Constructor$ConstructYamlObject.getConstructor(Constructor.java:323)
at org.yaml.snakeyaml.constructor.Constructor$ConstructYamlObject.construct(Constructor.java:332)
... 30 more

[emoxam]

@facupalladino That would be awesome! Thanks!

Hello Harvey, I believe this is the final jar I used https://github.com/facupalladino/kubernetes-cd-jar. You should find and replace (i recommend that you change the old jar name instead of deleting it, in case you want to rollback) the jar in the jenkins server, mine was in $JENKINS_HOME/plugins/kubernetes-cd/WEB-INF/lib Please, let me know if this solved your issue as well. Cheers!

Can youy make the same with the last version ? https://github.com/jenkinsci/kubernetes-cd-plugin/releases/tag/kubernetes-cd-2.3.1 i guess there it is

[Trojaner]

Here is a fixed version: https://github.com/byo-software/kubernetes-cd-plugin
It also includes Istio CRD's

Note: none of the security advisories have been addressed.

[emoxam]

I've got a problem, tabs at source java code doesn't supported ?

Loading configuration: /var/lib/jenkins/workspace/demo-app/Dockerfile
No resource loaded from: /var/lib/jenkins/workspace/demo-app/Dockerfile
Loading configuration: /var/lib/jenkins/workspace/demo-app/pom.xml
No resource loaded from: /var/lib/jenkins/workspace/demo-app/pom.xml
Loading configuration: /var/lib/jenkins/workspace/demo-app/src/main/java/ru/integris/demo/DemoApplication.java
ERROR: ERROR: while scanning for the next token
found character '\t(TAB)' that cannot start any token. (Do not use \t(TAB) for indentation)
 in 'reader', line 9, column 1:
    	public static void main(String[ ... 
    ^

hudson.remoting.ProxyException: while scanning for the next token
found character '\t(TAB)' that cannot start any token. (Do not use \t(TAB) for indentation)
 in 'reader', line 9, column 1:
    	public static void main(String[ ... 
    ^

	at org.yaml.snakeyaml.scanner.ScannerImpl.fetchMoreTokens(ScannerImpl.java:453)
	at org.yaml.snakeyaml.scanner.ScannerImpl.peekToken(ScannerImpl.java:273)
	at org.yaml.snakeyaml.parser.ParserImpl$ParseDocumentEnd.produce(ParserImpl.java:279)
	at org.yaml.snakeyaml.parser.ParserImpl.peekEvent(ParserImpl.java:166)
	at org.yaml.snakeyaml.comments.CommentEventsCollector$1.peek(CommentEventsCollector.java:59)

String 9 is
public static void main(String[] args) {
And it's really begins with TAB

[emoxam]

Config file setting should be **/*.yaml
Pardon, and thanks. All working

[emoxam]

Can we add ingress somehow ? Because jenkiins write
"Skipped unsupported resource: class V1Ingress"