You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I have created issue on quarkus git repo, but I found no one brave enough to face it :) quarkusio/quarkus#39098 (reproducer link is at the bottom)
The issue is: when using any kind of emitter to send message to kafka in a blocking manner (using regular blocking JTA @ Transactional) the transaction will sometimes still be bound to 2 threads kafka-producer-network-thread and regular executor thread in time of commit (on executor thread ofc.). This causes TX to fail, in a dramatic way. It happens randomly, chance is like 1:1000.
2024-03-01 08:32:55,162 WARN [com.arj.ats.arjuna] (executor-thread-2) ARJUNA012094: Commit of action id 0:ffff0a010a48:e441:65e18477:d1d0 invoked while multiple threads active within it.
2024-03-01 08:32:55,163 WARN [com.arj.ats.arjuna] (executor-thread-2) ARJUNA012107: CheckedAction::check - atomic action 0:ffff0a010a48:e441:65e18477:d1d0 commiting with 2 threads active!
2024-03-01 08:32:55,163 WARN [com.arj.ats.jta] (executor-thread-2) ARJUNA016039: onePhaseCommit on < formatId=131077, gtrid_length=35, bqual_length=36, tx_uid=0:ffff0a010a48:e441:65e18477:d1d0, node_name=quarkus, branch_uid=0:ffff0a010a48:e441:65e18477:d1d3, subordinatenodename=null, eis_name=0 > (io.agroal.narayana.LocalXAResource@2ea67dc) failed with exception XAException.XA_RBROLLBACK: javax.transaction.xa.XAException: Error trying to transactionCommit local transaction: Enlisted connection used without active transaction
The issue is that executor thread is blocked until message gets ACKed. But this is done BEFORE contextPropagation clears propagated contexts on kafka-producer-network-thread. So it is kind of race condition I guess.
Problem can be bypassed using different thread when sending to kafka, thread without TX context propagation. e.g. @ManagedExecutorConfig(cleared = ThreadContext.TRANSACTION)
Quarkus reproducer:
https://github.com/MatejPucihar/kafka-db-tx
reproducer stress tests the app, since the chance of EX is low, I have no other idea, how to reproduce it.
EX is reproduced for both MutinyEmitter and regular Emitter.
That are all the insights I can provide :) I'm for sure not an expert with reactive or context propagation, so take my insights with a grain of salt. The reproducer is real.
If this is not a bug, it would at least deserve some honorable mention in quarkus documentation I guess.
The text was updated successfully, but these errors were encountered:
I have created issue on quarkus git repo, but I found no one brave enough to face it :)
quarkusio/quarkus#39098 (reproducer link is at the bottom)
The issue is: when using any kind of emitter to send message to kafka in a blocking manner (using regular blocking JTA @ Transactional) the transaction will sometimes still be bound to 2 threads
kafka-producer-network-thread
and regularexecutor
thread in time of commit (on executor thread ofc.). This causes TX to fail, in a dramatic way. It happens randomly, chance is like 1:1000.The issue is that executor thread is blocked until message gets ACKed. But this is done BEFORE contextPropagation clears propagated contexts on
kafka-producer-network-thread
. So it is kind of race condition I guess.Problem can be bypassed using different thread when sending to kafka, thread without TX context propagation. e.g. @ManagedExecutorConfig(cleared = ThreadContext.TRANSACTION)
Quarkus reproducer:
https://github.com/MatejPucihar/kafka-db-tx
That are all the insights I can provide :) I'm for sure not an expert with reactive or context propagation, so take my insights with a grain of salt. The reproducer is real.
If this is not a bug, it would at least deserve some honorable mention in quarkus documentation I guess.
The text was updated successfully, but these errors were encountered: