Getting "Could not open JPA EntityManager" because of "Connection is not available, request timed out after 30000ms" with HikariCP Datasource in Spring #2219
Comments
|
FYI: I just tried EclipseLink 4.0.3 and this version works also great for me, so the bug must have been introduced in 4.0.4 |
|
I'm having a similar issue around locking after the upgrade from 4.0.3 to 4.0.4. Unfortunately, I went with DBCP's default no-timeout behaviour, causing my entire fleet of servers to deadlock every few minutes to hours. My guess is that it is caused by #2181 -- this pull request replaces the synchronized methods The issue seems to be that the new I seriously wonder as to why this is triggered now, the locking schema looks exactly the same. I guess we got lucky before because of some minor semantic difference between Starved thread: Multiple threads trying to return the connection objects (just an example here): |
|
Hi, |
eclipse-ee4j/eclipselink#2219 Signed-off-by: Marinov Avgustin <[email protected]>
eclipse-ee4j/eclipselink#2219 Signed-off-by: Marinov Avgustin <[email protected]>
eclipse-ee4j/eclipselink#2219 Signed-off-by: Marinov Avgustin <[email protected]>
|
Maybe try latest snapshot from https://jakarta.oss.sonatype.org/content/groups/staging/org/eclipse/persistence/eclipselink/4.0.5-SNAPSHOT/ . |
|
@fwiesweg: I'm encountering the exact same deadlocks during load tests. Replacing Downgrading to 4.0.3 for now. |
|
@rfelcman This issue seems to be introduced with #2181 Now the following seems to happen: Stacktraces: Thread 2 |
|
Could You please provide complete stack trace, or at least into |
@rfelcman I have updated my original comment with the extended stacktrace: |
…of "Connection is not available, request timed out after 30000ms" with HikariCP Datasource in Spring eclipse-ee4j#2219 Signed-off-by: Radek Felcman <[email protected]>
|
Hello, in the attachment You can find initial test project which is passing now (simple test with one thread). |
|
One idea about those locks - when the class is not final and somebody extends it and overrides a method and makes it synchronized, the locking breaks. So when you introduce these locks, these classes must be final OR have to provide access to the lock - and it is still a breaking change which should not be in 4.0.4. However I am not sure if this is the case, I just know that GlassFish 7.0.20-SNAPSHOT randomly doesn't pass the TCK tests with this error: |
|
Seeing probably the same stuff, downgrading indeed helps, have a thread dump from the deadlock condition:
|
|
I'm continue to check and understand this issue, but what sounds me strange is. How two/multiple threads should "fight" for On the begining from Jakarta Persistence Specification 3.1 and let's look to EclipseLink (little bit simplified) what is behind instance of All calls of the class instances are expected to call in single thread way, but I don't understand how is possible, that in this case Just for example there is something what I produced, but it seems, that SpringBoot correctly creates two instances of the So lifecycle of the something like is important. Mainly if this kind of the variable is not shared across threads. |
|
There is my current test case which doesn't lead into issue yet. Lets say it's some starting point. Feel free to take a look and modify it include application properties to simulate your environment. |
The issue might lay in the way Quick fix with the diagnostic logging: /**
* Clone the accessor.
*/
@Override
public Object clone() {
try {
DatasourceAccessor accessor = (DatasourceAccessor)super.clone();
// reset the instanceLock, otherwise the lock will be shared across multiple instances
accessor.instanceLock = new MyReentrantLock();
if(accessor.customizer != null) {
accessor.customizer.setAccessor(accessor);
}
LOG.debug(
"Cloned {} to {}. Clone's lock owner: {}. Lock ids: {} vs {}.",
this.hashCode(), accessor.hashCode(), accessor.instanceLock.getOwner(),
this.instanceLock.hashCode(), accessor.instanceLock.hashCode()
);
return accessor;
} catch (CloneNotSupportedException exception) {
throw new InternalError("clone not supported");
}
} |
|
are we releasing a new version with 4.05? This seems to be affecting everywhere! |
|
First release candidate of EclipseLink 4.0.5 (4.0.5-RC1) is available. |
Describe the bug
Recently I updated to EclipseLink 4.0.4 since then I get this error while booting my Spring Boot 3.3.2 app:
I have no idea where this could come from, I just know as soon as I downgrade to EclipseLink 4.0.2 (didn't test with 4.0.3 as I can only reproduce this error on my production environment, it's not happening on my local machine nor on our testing environment).
It seems the EntityManager cannot get a database connection before a timeout kicks in. My HikariCP database connection pool has no special configurations except that I set the MaximumPoolSize to 30, everything else (besides the DB connection settings) are default values.
As mentioned before, it works on my local machine and in our testing environment, the only difference I can think of is the load on the service. As soon as the service starts up in production, it will be called by around 2k IoT devices which is not the case on my local machine and on testing, everything else is pretty much the same. I did not change anything but the EclipseLink version from 4.0.4 to 4.0.2 and the error was gone and everything worked perfectly well again.
To Reproduce
Steps/resources to reproduce the behavior:
I have no idea how this could be reproduced, I just thought you should know about the issue, I'll might be able to come up with a test application.
Cheers,
Fabian
The text was updated successfully, but these errors were encountered: