narayana: problem with XA-objectstore in file system

[turing85]

Hi!
We have/had a problem with narayana.

One of our application has the following structure.

• In one XA-transaction, we:
  • read from JMS, with an XA transaction
  • write to a Azure MSSQL, in the same XA transaction

Now, since the MSSQL database is hosted in Azure and does not support XA, we have the datasource configured to run with local transactions. We leverage Narayana's LRCO feature (narayana.io) to allow the MSSQL to participate in the XA transaction (since the JMS is XA-transactional and no other resource participates, the MSSQL database is the only non-XA resource in the XA transaction).

The application runs in OpenShift. The Transaction store is a persistent volume (the application is defined as StatefulSet). We had an incident with our storage. Narayana was not able to commit the XA resource:

com.arjuna.ats.arjuna.exceptions.ObjectStoreException: ShadowingStore::write_state() - write failed to locate file transactions-logs/ShadowNoFileLockStore/defaultStore/StateManager/BasicAction/TwoPhaseCoordinator/AtomicAction/0_ffff64408b49_9a83_67aba724_102d086: java.io.FileNotFoundException: transactions-logs/ShadowNoFileLockStore/defaultStore/StateManager/BasicAction/TwoPhaseCoordinator/AtomicAction/0_ffff64408b49_9a83_67aba724_102d086 (Read-only file system)

What actually happened was that the MSSQL Transaction "went through", while the JMS transactoin rolled back. The application repeated this process over and over again.

So here is the question: when is the file to the object store actually written? I.e. when did the exception occur? For me, it looks like narayana tried to write to the file system between the 1st and 2nd phase - which doesn't make sense to me.

/cc @Jonathan Halliday

[quarkus-bot[bot]]

/cc @geoand (openshift), @iocanel (openshift)

[gastaldi]

/cc @jhalliday