Failed to start the Java service introducing the agent

[Zwl0113]

After upgrading the new version of skywalking-agent, our service has a service startup exception. For example, we have 10 servers, one of which has this problem, and it will be recovered after restarting. This problem occurs occasionally. The error display may be related to bytebuddy
(This problem occurred after we upgraded Skywalking Java Agent to v8.11.0)

2023-01-05 14:47:16.374 ERROR [] 2506666 --- [main] o.s.b.SpringApplication : Application run failed
org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'openApiControllerWebMvc' defined in URL [jar:file:/home/admin/zhengwanli/zhengwanli-demo-pro/zhengwanli-demo-prod-master-789650f3.jar!/BOOT-INF/lib/springfox-oas-3.0.0.jar!/springfox/documentation/oas/web/OpenApiControllerWebMvc.class]: Bean instantiation via constructor failed; nested exception is java.lang.ExceptionInInitializerError
at org.springframework.beans.factory.support.ConstructorResolver.instantiate(ConstructorResolver.java:315) ~[spring-beans-5.3.20.jar!/:5.3.20]
at org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:296) ~[spring-beans-5.3.20.jar!/:5.3.20]
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.autowireConstructor(AbstractAutowireCapableBeanFactory.java:1372) ~[spring-beans-5.3.20.jar!/:5.3.20]
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1222) ~[spring-beans-5.3.20.jar!/:5.3.20]
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:582) ~[spring-beans-5.3.20.jar!/:5.3.20]
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:542) ~[spring-beans-5.3.20.jar!/:5.3.20]
at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:335) ~[spring-beans-5.3.20.jar!/:5.3.20]
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234) ~[spring-beans-5.3.20.jar!/:5.3.20]
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:333) ~[spring-beans-5.3.20.jar!/:5.3.20]
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:208) ~[spring-beans-5.3.20.jar!/:5.3.20]
at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:953) ~[spring-beans-5.3.20.jar!/:5.3.20]
at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:918) ~[spring-context-5.3.20.jar!/:5.3.20]
at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:583) ~[spring-context-5.3.20.jar!/:5.3.20]
at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.refresh(ServletWebServerApplicationContext.java:145) ~[spring-boot-2.6.8.jar!/:2.6.8]
at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:745) ~[spring-boot-2.6.8.jar!/:2.6.8]
at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:420) ~[spring-boot-2.6.8.jar!/:2.6.8]
at org.springframework.boot.SpringApplication.run(SpringApplication.java:307) ~[spring-boot-2.6.8.jar!/:2.6.8]
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1317) ~[spring-boot-2.6.8.jar!/:2.6.8]
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1306) ~[spring-boot-2.6.8.jar!/:2.6.8]
at com.zhengwanli-demo.rubik.rest.RubikRestApplication.main(RubikRestApplication.java:59) ~[classes!/:1.0.0-SNAPSHOT]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_291]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_291]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_291]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_291]
at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:49) ~[zhengwanli-demo-prod-master-789650f3.jar:1.0.0-SNAPSHOT]
at org.springframework.boot.loader.Launcher.launch(Launcher.java:108) ~[zhengwanli-demo-prod-master-789650f3.jar:1.0.0-SNAPSHOT]
at org.springframework.boot.loader.Launcher.launch(Launcher.java:58) ~[zhengwanli-demo-prod-master-789650f3.jar:1.0.0-SNAPSHOT]
at org.springframework.boot.loader.JarLauncher.main(JarLauncher.java:88) ~[zhengwanli-demo-prod-master-789650f3.jar:1.0.0-SNAPSHOT]
Caused by: java.lang.ExceptionInInitializerError
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:1.8.0_291]
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:1.8.0_291]
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:1.8.0_291]
at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[?:1.8.0_291]
at org.springframework.beans.BeanUtils.instantiateClass(BeanUtils.java:211) ~[spring-beans-5.3.20.jar!/:5.3.20]
at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:117) ~[spring-beans-5.3.20.jar!/:5.3.20]
at org.springframework.beans.factory.support.ConstructorResolver.instantiate(ConstructorResolver.java:311) ~[spring-beans-5.3.20.jar!/:5.3.20]
... 27 more
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.GeneratedMethodAccessor70.invoke(Unknown Source) ~[?:?]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_291]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_291]
at springfox.documentation.oas.web.OpenApiControllerWebMvc.(OpenApiControllerWebMvc.java) ~[springfox-oas-3.0.0.jar!/:3.0.0]
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:1.8.0_291]
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:1.8.0_291]
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:1.8.0_291]
at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[?:1.8.0_291]
at org.springframework.beans.BeanUtils.instantiateClass(BeanUtils.java:211) ~[spring-beans-5.3.20.jar!/:5.3.20]
at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:117) ~[spring-beans-5.3.20.jar!/:5.3.20]
at org.springframework.beans.factory.support.ConstructorResolver.instantiate(ConstructorResolver.java:311) ~[spring-beans-5.3.20.jar!/:5.3.20]
... 27 more
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.GeneratedMethodAccessor71.invoke(Unknown Source) ~[?:?]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_291]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_291]
at org.apache.skywalking.apm.dependencies.net.bytebuddy.dynamic.Nexus.initialize(Nexus.java:139) ~[skywalking-agent.jar:8.11.0]
at sun.reflect.GeneratedMethodAccessor70.invoke(Unknown Source) ~[?:?]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_291]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_291]
at springfox.documentation.oas.web.OpenApiControllerWebMvc.(OpenApiControllerWebMvc.java) ~[springfox-oas-3.0.0.jar!/:3.0.0]
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:1.8.0_291]
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:1.8.0_291]
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:1.8.0_291]
at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[?:1.8.0_291]
at org.springframework.beans.BeanUtils.instantiateClass(BeanUtils.java:211) ~[spring-beans-5.3.20.jar!/:5.3.20]
at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:117) ~[spring-beans-5.3.20.jar!/:5.3.20]
at org.springframework.beans.factory.support.ConstructorResolver.instantiate(ConstructorResolver.java:311) ~[spring-beans-5.3.20.jar!/:5.3.20]
... 27 more
Caused by: java.lang.IllegalArgumentException: Can not set static org.apache.skywalking.apm.agent.core.plugin.interceptor.enhance.InstMethodsInter field springfox.documentation.oas.web.OpenApiControllerWebMvc.delegate$d16orf1 to org.apache.skywalking.apm.agent.core.plugin.interceptor.enhance.ConstructorInter
at sun.reflect.UnsafeFieldAccessorImpl.throwSetIllegalArgumentException(UnsafeFieldAccessorImpl.java:167) ~[?:1.8.0_291]
at sun.reflect.UnsafeFieldAccessorImpl.throwSetIllegalArgumentException(UnsafeFieldAccessorImpl.java:171) ~[?:1.8.0_291]
at sun.reflect.UnsafeQualifiedStaticObjectFieldAccessorImpl.set(UnsafeQualifiedStaticObjectFieldAccessorImpl.java:81) ~[?:1.8.0_291]
at java.lang.reflect.Field.set(Field.java:764) ~[?:1.8.0_291]
at org.apache.skywalking.apm.dependencies.net.bytebuddy.implementation.LoadedTypeInitializer$ForStaticField.onLoad(LoadedTypeInitializer.java:170) ~[skywalking-agent.jar:8.11.0]
at org.apache.skywalking.apm.dependencies.net.bytebuddy.implementation.LoadedTypeInitializer$Compound.onLoad(LoadedTypeInitializer.java:233) ~[skywalking-agent.jar:8.11.0]
at org.apache.skywalking.apm.dependencies.net.bytebuddy.agent.builder.AgentBuilder$InitializationStrategy$SelfInjection$Dispatcher$InjectingInitializer.onLoad(AgentBuilder.java:3655) ~[skywalking-agent.jar:8.11.0]
at sun.reflect.GeneratedMethodAccessor71.invoke(Unknown Source) ~[?:?]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_291]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_291]
at org.apache.skywalking.apm.dependencies.net.bytebuddy.dynamic.Nexus.initialize(Nexus.java:139) ~[skywalking-agent.jar:8.11.0]
at sun.reflect.GeneratedMethodAccessor70.invoke(Unknown Source) ~[?:?]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_291]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_291]
at springfox.documentation.oas.web.OpenApiControllerWebMvc.(OpenApiControllerWebMvc.java) ~[springfox-oas-3.0.0.jar!/:3.0.0]
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:1.8.0_291]
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:1.8.0_291]
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:1.8.0_291]
at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[?:1.8.0_291]
at org.springframework.beans.BeanUtils.instantiateClass(BeanUtils.java:211) ~[spring-beans-5.3.20.jar!/:5.3.20]
at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:117) ~[spring-beans-5.3.20.jar!/:5.3.20]
at org.springframework.beans.factory.support.ConstructorResolver.instantiate(ConstructorResolver.java:311) ~[spring-beans-5.3.20.jar!/:5.3.20]
... 27 more

[wu-sheng]

You only provided very limited context. knife4j is never supported by SkyWalking, so I assume there are some tricky works for its kernel for the classloader or dynamic class generation, which could conflict with SkyWalking and ByteBuddy.
Most existing middleware would not manipulate class or classloader, but from your error, knife4j seems different. It may be better to ask knife4j for some context about what they did.

[wu-sheng]

We don't do any kernel level upgrade, AFAIK. Most are same in the whole v8 series.
I can't tell things I don't have experiences.

[Zwl0113]

However, the version of bytebuddy used by skywalking-agent has changed dramatically

[wu-sheng]

What do you mean changed dramatically?

[Zwl0113]

ByeBuddy version from 1.10.7 to 1.11.18

[wu-sheng]

I don't know. Once you changed something private, you have to debug on your own.