Convergence keeps losing connection to OrientDB

[vince92079]

Versions
Please fill in all that apply:

• Convergence Version: 1.0.0-rc.13
• OS: Windows

We're seeing when we're running Convergence with our application, that Convergence keeps losing it's connection to the orientDB database.

Nothing seems to make sense what could be causing this. The only thing I see is the constant repeating errors from the log below.

Convergence Omnibus Container

2024-03-04 23:54:11,782 CRIT Supervisor is running as root. Privileges were not dropped because no user is specified in the config file. If you intend to run as root, you can set user=root in the config file to avoid this message.
2024-03-04 23:54:11,782 CRIT Supervisor is running as root. Privileges were not dropped because no user is specified in the config file. If you intend to run as root, you can set user=root in the config file to avoid this message.
2024-03-04 23:54:11,785 INFO supervisord started with pid 1
2024-03-04 23:54:11,785 INFO supervisord started with pid 1
2024-03-04 23:54:12,789 INFO spawned: 'proxy' with pid 7
2024-03-04 23:54:12,789 INFO spawned: 'proxy' with pid 7
2024-03-04 23:54:12,792 INFO spawned: 'orientdb' with pid 8
2024-03-04 23:54:12,792 INFO spawned: 'orientdb' with pid 8
2024-03-04 23:54:12,794 INFO spawned: 'convergence' with pid 9
2024-03-04 23:54:12,794 INFO spawned: 'convergence' with pid 9 `
2024-03-04 23:54:13,862 INFO success: proxy entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2024-03-04 23:54:13,862 INFO success: proxy entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2024-03-04 23:54:13,862 INFO success: orientdb entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2024-03-04 23:54:13,862 INFO success: orientdb entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2024-03-04 23:54:13,862 INFO success: convergence entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2024-03-04 23:54:13,862 INFO success: convergence entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
23:54:14 INFO ConvergenceServer$ - Using config file: /convergence-server/bin/../conf/convergence-server.conf
23:54:17 INFO ConvergenceServerActor - Convergence Server (1.0.0-rc.13) starting up...
23:54:17 INFO ConvergenceServerActor - Convergence Server Roles: [backend, realtimeApi, restApi]
23:54:17 INFO ConvergenceServerActor - Role 'backend' detected, activating Backend Services...
23:54:17 INFO ConvergenceServerActor - Ensuring convergence database is initialized
23:54:17 INFO ConvergenceServerActor - Role 'restApi' detected, activating REST API...
23:54:18 INFO ConvergenceServerActor - Role 'realtimeApi' detected, activating the Realtime API...
23:54:18 INFO ConvergenceRestApi - Rest API started at: http://0.0.0.0:8081
23:54:18 INFO ConvergenceRealtimeApi - Realtime API started at: http://0.0.0.0:8080
23:54:18 INFO ConvergenceDatabaseInitializer - Attempting to connect to the database at uri: remote:localhost:2424
23:54:19 WARN SystemErr - Mar 04, 2024 11:54:19 PM com.orientechnologies.common.log.OLogManager log
INFO: Detected limit of amount of simultaneously open files is 1048576, limit of open files for disk cache will be set to 523776

23:54:19 INFO ConvergenceDatabaseInitializer - Connected to database with Server Admin
23:54:20 INFO BackendServices - Convergence Backend Services starting up...
23:54:20 INFO BackendServices - Convergence Backend Services started up.
23:58:16 WARN SystemErr - Mar 04, 2024 11:58:16 PM com.orientechnologies.common.log.OLogManager log
INFO: - shutdown storage: 7075637650458386424...

05:37:01 WARN SystemErr - Mar 05, 2024 5:37:01 AM com.orientechnologies.common.log.OLogManager log
INFO: - shutdown storage: 7075637650458386424...

15:28:28 WARN SystemErr - Mar 06, 2024 3:28:28 PM com.orientechnologies.common.log.OLogManager log
INFO: Caught Network I/O errors on 127.0.0.1:2424/7075637650458386424, trying an automatic reconnection... (error: com.o; serializable and externalizable flags conflict)

15:28:29 WARN SystemErr - Mar 06, 2024 3:28:29 PM com.orientechnologies.common.log.OLogManager log
INFO: Caught Network I/O errors on 127.0.0.1:2424/7075637650458386424, trying an automatic reconnection... (error: com.o; serializable and externalizable flags conflict)

15:28:29 WARN SystemErr - Mar 06, 2024 3:28:29 PM com.orientechnologies.common.log.OLogManager log
INFO: Caught Network I/O errors on 127.0.0.1:2424/7075637650458386424, trying an automatic reconnection... (error: com.o; serializable and externalizable flags conflict)

15:28:30 WARN SystemErr - Mar 06, 2024 3:28:30 PM com.orientechnologies.common.log.OLogManager log
INFO: Caught Network I/O errors on 127.0.0.1:2424/7075637650458386424, trying an automatic reconnection... (error: com.o; serializable and externalizable flags conflict)

15:28:30 WARN SystemErr - Mar 06, 2024 3:28:30 PM com.orientechnologies.common.log.OLogManager log
INFO: Caught Network I/O errors on 127.0.0.1:2424/7075637650458386424, trying an automatic reconnection... (error: com.o; serializable and externalizable flags conflict)

15:28:30 ERROR RealtimeModelManager - Error getting model permissions (b9717134-a048-41d8-8693-8732d9124c4f)
com.orientechnologies.common.io.OIOException: com.o; serializable and externalizable flags conflict
at com.orientechnologies.orient.client.remote.OStorageRemote.baseNetworkOperation(OStorageRemote.java:351) ~[com.orientechnologies.orientdb-client-3.0.37.jar:3.0.37]
at com.orientechnologies.orient.client.remote.OStorageRemote.networkOperationRetryTimeout(OStorageRemote.java:228) ~[com.orientechnologies.orientdb-client-3.0.37.jar:3.0.37]
at com.orientechnologies.orient.client.remote.OStorageRemote.networkOperation(OStorageRemote.java:265) ~[com.orientechnologies.orientdb-client-3.0.37.jar:3.0.37]
at com.orientechnologies.orient.client.remote.OStorageRemote.query(OStorageRemote.java:929) ~[com.orientechnologies.orientdb-client-3.0.37.jar:3.0.37]
at com.orientechnologies.orient.core.db.document.ODatabaseDocumentRemote.query(ODatabaseDocumentRemote.java:378) ~[com.orientechnologies.orientdb-client-3.0.37.jar:3.0.37]
at com.convergencelabs.convergence.server.backend.datastore.OrientDBUtil$.$anonfun$getDocument$1(OrientDBUtil.scala:80) ~[com.convergencelabs.convergence-server-1.0.0-rc.13.jar:?]
at scala.util.Try$.apply(Try.scala:210) ~[org.scala-lang.scala-library-2.13.7.jar:?]
at com.convergencelabs.convergence.server.backend.datastore.OrientDBUtil$.getDocument(OrientDBUtil.scala:80) ~[com.convergencelabs.convergence-server-1.0.0-rc.13.jar:?]
at com.convergencelabs.convergence.server.backend.datastore.domain.collection.CollectionPermissionsStore.$anonfun$getCollectionWorldPermissions$1(CollectionPermissionsStore.scala:41) ~[com.convergencelabs.convergence-server-1.0.0-rc.13.jar:?]
at com.convergencelabs.convergence.server.backend.db.PooledDatabaseProvider.$anonfun$withDatabase$3(PooledDatabaseProvider.scala:58) ~[com.convergencelabs.convergence-server-1.0.0-rc.13.jar:?]
at com.convergencelabs.convergence.server.util.TryWithResource.tryWithResolvedResource(TryWithResource.scala:50) ~[com.convergencelabs.convergence-server-1.0.0-rc.13.jar:?]
at com.convergencelabs.convergence.server.util.TryWithResource.com$convergencelabs$convergence$server$util$TryWithResource$$tryWithResource(TryWithResource.scala:36) ~[com.convergencelabs.convergence-server-1.0.0-rc.13.jar:?]
at com.convergencelabs.convergence.server.util.TryWithResource$.apply(TryWithResource.scala:78) ~[com.convergencelabs.convergence-server-1.0.0-rc.13.jar:?]
at com.convergencelabs.convergence.server.backend.db.PooledDatabaseProvider.$anonfun$withDatabase$1(PooledDatabaseProvider.scala:56) ~[com.convergencelabs.convergence-server-1.0.0-rc.13.jar:?]
at scala.util.Success.flatMap(Try.scala:258) ~[org.scala-lang.scala-library-2.13.7.jar:?]
at com.convergencelabs.convergence.server.backend.db.PooledDatabaseProvider.withDatabase(PooledDatabaseProvider.scala:55) ~[com.convergencelabs.convergence-server-1.0.0-rc.13.jar:?]
at com.convergencelabs.convergence.server.backend.datastore.AbstractDatabasePersistence.withDb(AbstractDatabasePersistence.scala:32) ~[com.convergencelabs.convergence-server-1.0.0-rc.13.jar:?]
at com.convergencelabs.convergence.server.backend.datastore.domain.collection.CollectionPermissionsStore.getCollectionWorldPermissions(CollectionPermissionsStore.scala:38) ~[com.convergencelabs.convergence-server-1.0.0-rc.13.jar:?]
at com.convergencelabs.convergence.server.backend.services.domain.model.ModelPermissionResolver.$anonfun$getModelAndCollectionPermissions$1(ModelPermissionResolver.scala:37) ~[com.convergencelabs.convergence-server-1.0.0-rc.13.jar:?]
at com.convergencelabs.convergence.server.backend.services.domain.model.ModelPermissionResolver.$anonfun$getModelAndCollectionPermissions$1$adapted(ModelPermissionResolver.scala:36) ~[com.convergencelabs.convergence-server-1.0.0-rc.13.jar:?]
at scala.util.Success.flatMap(Try.scala:258) ~[org.scala-lang.scala-library-2.13.7.jar:?]
at com.convergencelabs.convergence.server.backend.services.domain.model.ModelPermissionResolver.getModelAndCollectionPermissions(ModelPermissionResolver.scala:36) ~[com.convergencelabs.convergence-server-1.0.0-rc.13.jar:?]
at com.convergencelabs.convergence.server.backend.services.domain.model.RealtimeModelManager.$anonfun$requestModelDataFromDataStore$3(RealtimeModelManager.scala:250) ~[com.convergencelabs.convergence-server-1.0.0-rc.13.jar:?]
at scala.util.Success.map(Try.scala:262) ~[org.scala-lang.scala-library-2.13.7.jar:?]
at com.convergencelabs.convergence.server.backend.services.domain.model.RealtimeModelManager.$anonfun$requestModelDataFromDataStore$2(RealtimeModelManager.scala:244) ~[com.convergencelabs.convergence-server-1.0.0-rc.13.jar:?]
at scala.util.Success.flatMap(Try.scala:258) ~[org.scala-lang.scala-library-2.13.7.jar:?]
at com.convergencelabs.convergence.server.backend.services.domain.model.RealtimeModelManager.requestModelDataFromDataStore(RealtimeModelManager.scala:243) ~[com.convergencelabs.convergence-server-1.0.0-rc.13.jar:?]
at com.convergencelabs.convergence.server.backend.services.domain.model.RealtimeModelManager.$anonfun$onOpenModelWhileUninitialized$2(RealtimeModelManager.scala:180) ~[com.convergencelabs.convergence-server-1.0.0-rc.13.jar:?]
at com.convergencelabs.convergence.server.backend.services.domain.model.RealtimeModelManager.$anonfun$onOpenModelWhileUninitialized$2$adapted(RealtimeModelManager.scala:178) ~[com.convergencelabs.convergence-server-1.0.0-rc.13.jar:?]
at scala.util.Success.map(Try.scala:262) ~[org.scala-lang.scala-library-2.13.7.jar:?]
at com.convergencelabs.convergence.server.backend.services.domain.model.RealtimeModelManager.onOpenModelWhileUninitialized(RealtimeModelManager.scala:178) ~[com.convergencelabs.convergence-server-1.0.0-rc.13.jar:?]
at com.convergencelabs.convergence.server.backend.services.domain.model.RealtimeModelManager.onOpenRealtimeModelRequest(RealtimeModelManager.scala:158) ~[com.convergencelabs.convergence-server-1.0.0-rc.13.jar:?]
at com.convergencelabs.convergence.server.backend.services.domain.model.RealtimeModelActor.handleRealtimeMessage(RealtimeModelActor.scala:227) ~[com.convergencelabs.convergence-server-1.0.0-rc.13.jar:?]
at com.convergencelabs.convergence.server.backend.services.domain.model.RealtimeModelActor.receiveOpened(RealtimeModelActor.scala:122) ~[com.convergencelabs.convergence-server-1.0.0-rc.13.jar:?]
at com.convergencelabs.convergence.server.backend.services.domain.model.RealtimeModelActor.receiveClosed(RealtimeModelActor.scala:109) ~[com.convergencelabs.convergence-server-1.0.0-rc.13.jar:?]
at com.convergencelabs.convergence.server.backend.services.domain.model.RealtimeModelActor.receiveInitialized(RealtimeModelActor.scala:97) ~[com.convergencelabs.convergence-server-1.0.0-rc.13.jar:?]
at com.convergencelabs.convergence.server.backend.services.domain.model.RealtimeModelActor.receiveInitialized(RealtimeModelActor.scala:51) ~[com.convergencelabs.convergence-server-1.0.0-rc.13.jar:?]
at com.convergencelabs.convergence.server.util.actor.ShardedActor.$anonfun$receiveInitialMessage$2(ShardedActor.scala:86) ~[com.convergencelabs.convergence-server-1.0.0-rc.13.jar:?]
at scala.util.Success.map(Try.scala:262) [org.scala-lang.scala-library-2.13.7.jar:?]
at com.convergencelabs.convergence.server.util.actor.ShardedActor.receiveInitialMessage(ShardedActor.scala:83) [com.convergencelabs.convergence-server-1.0.0-rc.13.jar:?]
at com.convergencelabs.convergence.server.util.actor.ShardedActor.onMessage(ShardedActor.scala:46) [com.convergencelabs.convergence-server-1.0.0-rc.13.jar:?]
at akka.actor.typed.scaladsl.AbstractBehavior.receive(AbstractBehavior.scala:84) [com.typesafe.akka.akka-actor-typed_2.13-2.6.15.jar:2.6.15]
at akka.actor.typed.Behavior$.interpret(Behavior.scala:274) [com.typesafe.akka.akka-actor-typed_2.13-2.6.15.jar:2.6.15]
at akka.actor.typed.Behavior$.interpretMessage(Behavior.scala:230) [com.typesafe.akka.akka-actor-typed_2.13-2.6.15.jar:2.6.15]
at akka.actor.typed.internal.InterceptorImpl$$anon$2.apply(InterceptorImpl.scala:57) [com.typesafe.akka.akka-actor-typed_2.13-2.6.15.jar:2.6.15]
at akka.actor.typed.internal.InterceptorImpl.receive(InterceptorImpl.scala:87) [com.typesafe.akka.akka-actor-typed_2.13-2.6.15.jar:2.6.15]
at akka.actor.typed.Behavior$.interpret(Behavior.scala:274) [com.typesafe.akka.akka-actor-typed_2.13-2.6.15.jar:2.6.15]
at akka.actor.typed.Behavior$.interpretMessage(Behavior.scala:230) [com.typesafe.akka.akka-actor-typed_2.13-2.6.15.jar:2.6.15]
at akka.actor.typed.internal.InterceptorImpl$$anon$2.apply(InterceptorImpl.scala:57) [com.typesafe.akka.akka-actor-typed_2.13-2.6.15.jar:2.6.15]
at akka.actor.typed.internal.SimpleSupervisor.aroundReceive(Supervision.scala:123) [com.typesafe.akka.akka-actor-typed_2.13-2.6.15.jar:2.6.15]
at akka.actor.typed.internal.InterceptorImpl.receive(InterceptorImpl.scala:85) [com.typesafe.akka.akka-actor-typed_2.13-2.6.15.jar:2.6.15]
at akka.actor.typed.Behavior$.interpret(Behavior.scala:274) [com.typesafe.akka.akka-actor-typed_2.13-2.6.15.jar:2.6.15]
at akka.actor.typed.Behavior$.interpretMessage(Behavior.scala:230) [com.typesafe.akka.akka-actor-typed_2.13-2.6.15.jar:2.6.15]
at akka.actor.typed.internal.adapter.ActorAdapter.handleMessage(ActorAdapter.scala:131) [com.typesafe.akka.akka-actor-typed_2.13-2.6.15.jar:2.6.15]
at akka.actor.typed.internal.adapter.ActorAdapter.aroundReceive(ActorAdapter.scala:107) [com.typesafe.akka.akka-actor-typed_2.13-2.6.15.jar:2.6.15]
at akka.actor.ActorCell.receiveMessage(ActorCell.scala:580) [com.typesafe.akka.akka-actor_2.13-2.6.15.jar:2.6.15]
at akka.actor.ActorCell.invoke(ActorCell.scala:548) [com.typesafe.akka.akka-actor_2.13-2.6.15.jar:2.6.15]
at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:270) [com.typesafe.akka.akka-actor_2.13-2.6.15.jar:2.6.15]
at akka.dispatch.Mailbox.run(Mailbox.scala:231) [com.typesafe.akka.akka-actor_2.13-2.6.15.jar:2.6.15]
at akka.dispatch.Mailbox.exec(Mailbox.scala:243) [com.typesafe.akka.akka-actor_2.13-2.6.15.jar:2.6.15]
at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:290) [?:?]
at java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1020) [?:?]
at java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1656) [?:?]
at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1594) [?:?]
at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:183) [?:?]
Caused by: java.io.InvalidClassException: com.o; serializable and externalizable flags conflict
at java.io.ObjectStreamClass.readNonProxy(ObjectStreamClass.java:643) ~[?:?]
at java.io.ObjectInputStream.readClassDescriptor(ObjectInputStream.java:961) ~[?:?]
at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1992) ~[?:?]
at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1870) ~[?:?]
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2201) ~[?:?]
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1687) ~[?:?]
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:489) ~[?:?]
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:447) ~[?:?]
at com.orientechnologies.orient.client.binary.OChannelBinaryAsynchClient.deserializeException(OChannelBinaryAsynchClient.java:324) ~[com.orientechnologies.orientdb-client-3.0.37.jar:3.0.37]
at com.orientechnologies.orient.client.binary.OChannelBinaryAsynchClient.handleStatus(OChannelBinaryAsynchClient.java:286) ~[com.orientechnologies.orientdb-client-3.0.37.jar:3.0.37]
at com.orientechnologies.orient.client.binary.OChannelBinaryAsynchClient.handleStatus(OChannelBinaryAsynchClient.java:309) ~[com.orientechnologies.orientdb-client-3.0.37.jar:3.0.37]
at com.orientechnologies.orient.client.binary.OChannelBinaryAsynchClient.beginResponse(OChannelBinaryAsynchClient.java:198) ~[com.orientechnologies.orientdb-client-3.0.37.jar:3.0.37]
at com.orientechnologies.orient.client.binary.OChannelBinaryAsynchClient.beginResponse(OChannelBinaryAsynchClient.java:160) ~[com.orientechnologies.orientdb-client-3.0.37.jar:3.0.37]
at com.orientechnologies.orient.client.remote.OStorageRemote.beginResponse(OStorageRemote.java:1882) ~[com.orientechnologies.orientdb-client-3.0.37.jar:3.0.37]
at com.orientechnologies.orient.client.remote.OStorageRemote.lambda$networkOperationRetryTimeout$2(OStorageRemote.java:248) ~[com.orientechnologies.orientdb-client-3.0.37.jar:3.0.37]
at com.orientechnologies.orient.client.remote.OStorageRemote.baseNetworkOperation(OStorageRemote.java:307) ~[com.orientechnologies.orientdb-client-3.0.37.jar:3.0.37]
... 64 more