RejectedExecutionException when executing container tests #6299
Labels
Waiting on feedback
Issues that require feedback from User/Other community members
Comments
|
To clarify do the tests run incorrectly, or is this just unexpectedly logged? Either way can you provide more of the stacktrace? There have been two other issues addressed recently where an exception like this had been logged, but is now surpressed. |
rohanKanojia
added
the
Waiting on feedback
|
This is the complete stacktrace. It doesn't even tell me which call from my code exactly causes it, but it's obviously some call to the k8s client library ( |
|
Is it possible that your tests are closing the client? (e.g. your first test performs a try with resources to the client and then the subsequent tests fail). |
|
I've had some calls of the client inside try {} blocks: into this: Still getting the same errors. It's maybe easiest if I give you a complete example of a test the |
|
could you show the points where |
|
|
Have you noticed if the stacktrace comes after a test failure? The only thing I can see here is the possibility that the await gets called in a thread where the client was already closed. However, the snippets you show seem to be blocking (unless the tests are orchestrated externally) |
|
The only orchestration of the tests are the The Exception itself is the testfailure, nothing else is executed after it. We're using the io.fabric8.kubernetes.client.informers.ResourceEventHandler to react to changes on KafkaUser resources. These are started in the background, every triggering of the informer executes the onUpdate / onAdd functions in their own threads. So for a test, I'm modifying a resource, which we can see here: Then, another test is started, but we can see in the log output, that the onUpdate() function is triggered one more time, and executed in a new thread. |
|
Can you increase the fabric8 logging? There is a debug message when the client is closed. |
|
Now it happened again for the first test (the one that I've posted here already), no other tests are executed before it. I think that only the last part of the output is relevant, we can see that the for some reason, |
Fabric8 won't proactively close the client, that has to be coming from something using the client. We also see one client being closed, while another one is being created - is that expected? The thread for test execution seems to be " Thread-6" while the close / creation is happening in "pool-5-thread-2" so it does seem like that could be happening asynchronously from the main test logic. It seems like you could look at when / why AkhqConfigMapGenerator is being called to figure out what is closing the client as that is happening on the same thread.
Watches are tolerant to various exceptions, since this is logged at a debug level it is expected and handled internally. |
|
Hi,
There are multiple things happening in parallel here. As soon as a KafkaUser resource is updated, our ResourceEventHandler will get triggered, thus spawning a new thread. The program logic will do a couple of things, also using the client. The client is always correctly closed everywhere in the code after it is being used - it is not used after it is being closed, I always use it inside a try block like this for example: So there's no way I'm accidentally using the client after it has been closed.
That's coming from the ResourceEventHandler like described above. The main thread will run the application itself while more threads will get spawned for every triggering of any ResourceEventHandler by a create / update of a watched resource.
That's not the case. AkhqConfigMapGenerator does not use the client. Like I said, also the tests are not doing anything spectacular: |
Perhaps not directly, but references to the client from within the try catch block can certainly live on after the closure.
I was talking about the thread - whatever thread is exercising AkhqConfigMapGenerator seems to be the same thread that is closing / opening clients. Just to make sure, what thread is the rejected execution exception happening on in the log you are showing?
The fabric8 code is pretty straigh-forward as well. The main executor don't shut itself down, that only happens when the client is closed. We know that is likely happening here because the rejected execution exception you are seeing does not originate with a serial executor, it is coming from an actual thread pool executor. Would it be easy for you to provide a reproducer? |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Describe the bug
So I have a container testing suite with a k8s cluster containing a kafka + strimzi cluster and with the fabric8 kubernetes-client I'm creating/deleting some KafkaUsers/KafkaTopic resources.
There are 18 tests and sometimes they run through correctly, but very often I'm getting the error that I've pasted down below. I'm not closing the client in between. It's happening completely arbitrarily, sometimes it happens, sometimes it doesn't.
Any advice?
Fabric8 Kubernetes Client version
other (please specify in additional context)
Steps to reproduce
I'm using the KubernetesClientBuilder like this to obtain a client:
public KubernetesClient getKubernetesClient() {
return new KubernetesClientBuilder().build();
}
Expected behavior
No crashes
Runtime
Kubernetes (vanilla)
Kubernetes API Server version
1.25.3@latest
Environment
macOS
Fabric8 Kubernetes Client Logs
Additional context
fabric8 kubernetes client version: 6.13.1
The text was updated successfully, but these errors were encountered: