test(test-tooling): fix FabricTestLedger INVALID_ENDORSER_TRANSACTION when issuing transaction

[AndreAugusto11]

Describe the bug

I have been constantly having a problem when using the Fabric connector in my working branch. There is sometimes an error, related to one of the peers that throws INVALID_ENDORSER_TRANSACTION .

I have tried multiple times, and the error is thrown in different test cases. Steps followed:

npm run clean
npm run codegen
npm run configure

After these steps, the error is still thrown.

More details:

[2024-01-31T14:31:10.069Z] DEBUG (PluginLedgerConnectorFabric): PluginLedgerConnectorFabric#transact() Submitting TX... ([ '6396b1bc-0bc4-430e-a9f6-5807789a009d', [length]: 1 ])
2024-01-31T14:31:12.229Z - warn: [TransactionEventHandler]: strategyFail: commit failure for transaction "e0e70e6fdf142d4177ee4159475e91cbb0a63d159cf5b6b21ad2a52faaa7bc65": TransactionError: Commit of transaction e0e70e6fdf142d4177ee4159475e91cbb0a63d159cf5b6b21ad2a52faaa7bc65 failed on peer peer0.org2.example.com:9051 with status INVALID_ENDORSER_TRANSACTION
    at TransactionEventHandler.eventCallback (/home/andre_augusto/tmp/cacti/packages/cactus-plugin-ledger-connector-fabric/node_modules/fabric-network/src/impl/event/transactioneventhandler.ts:128:22)
    at CommitListenerSession.notifyListener (/home/andre_augusto/tmp/cacti/packages/cactus-plugin-ledger-connector-fabric/node_modules/fabric-network/src/impl/event/commitlistenersession.ts:106:9)
    at EventListener.callback (/home/andre_augusto/tmp/cacti/packages/cactus-plugin-ledger-connector-fabric/node_modules/fabric-network/src/impl/event/commitlistenersession.ts:94:9)
    at EventListener.onEvent (/home/andre_augusto/tmp/cacti/packages/cactus-plugin-ledger-connector-fabric/node_modules/fabric-common/lib/EventListener.js:124:10)
    at EventService._callTransactionListener (/home/andre_augusto/tmp/cacti/packages/cactus-plugin-ledger-connector-fabric/node_modules/fabric-common/lib/EventService.js:1051:16)
    at EventService._processTxEvents (/home/andre_augusto/tmp/cacti/packages/cactus-plugin-ledger-connector-fabric/node_modules/fabric-common/lib/EventService.js:1006:12)
    at ClientDuplexStreamImpl.<anonymous> (/home/andre_augusto/tmp/cacti/packages/cactus-plugin-ledger-connector-fabric/node_modules/fabric-common/lib/EventService.js:481:12)
    at ClientDuplexStreamImpl.emit (node:events:513:28)
    at ClientDuplexStreamImpl.emit (node:domain:489:12)
    at addChunk (node:internal/streams/readable:315:12)
    at readableAddChunk (node:internal/streams/readable:289:9)
    at ClientDuplexStreamImpl.Readable.push (node:internal/streams/readable:228:10)
    at Object.onReceiveMessage (/home/andre_augusto/tmp/cacti/packages/cactus-plugin-ledger-connector-fabric/node_modules/@grpc/grpc-js/src/client.ts:693:16)
    at Object.onReceiveMessage (/home/andre_augusto/tmp/cacti/packages/cactus-plugin-ledger-connector-fabric/node_modules/@grpc/grpc-js/src/client-interceptors.ts:418:47)
    at /home/andre_augusto/tmp/cacti/packages/cactus-plugin-ledger-connector-fabric/node_modules/@grpc/grpc-js/src/call-stream.ts:402:22
    at processTicksAndRejections (node:internal/process/task_queues:78:11) {
  transactionId: 'e0e70e6fdf142d4177ee4159475e91cbb0a63d159cf5b6b21ad2a52faaa7bc65',
  transactionCode: 'INVALID_ENDORSER_TRANSACTION'
}

To Reproduce

Running the test file packages/cactus-plugin-ledger-connector-fabric/src/test/typescript/integration/fabric-v2-2-x/deploy-lock-asset.test.ts fails in 1 test case. The full log is here fabric-error.log. Note that the number of tests passed varies each time I run the file.

Expected behavior

19 out of 19 tests pass.

Cloud provider or hardware configuration:
Are you running the software on a dev machine or somewhere in the cloud?

Operating system name, version, build:

$ printf "$(uname -srm)\n$(cat /etc/os-release)\n"
Linux 5.15.0-1049-gcp x86_64
NAME="Ubuntu"
VERSION="20.04.6 LTS (Focal Fossa)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 20.04.6 LTS"
VERSION_ID="20.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=focal
UBUNTU_CODENAME=focal

Hyperledger Cactus release version or commit (git rev-parse --short HEAD):

Latest commit: 73ff59317

Hyperledger Cactus Plugins/Connectors Used

Fabric connector

Additional context

I am facing the same issue in test files of other packages that rely on the Fabric connector (odap package and CBDC example)

[AndreAugusto11]

I am adding additional log files, where different executions result in different results.
PASS: crash-log-1.log
FAIL: crash-log-2.log
FAIL: crash-log-3.log
FAIL: crash-log-4.log
PASS: crash-log-5.log

cc: @petermetz

[petermetz]

I am adding additional log files, where different executions result in different results. PASS: [crash-log-1.log]

@AndreAugusto11 Thank you for these!! It will help the investigation for sure!

[AndreAugusto11]

@petermetz I remember that in November/December everything was working fine. So I tried reverting the last commits that changed the Fabric Test Ledger file (here).

The error does not appear anymore after reverting commit 3371772c582389f6ee0c6fb66af875dd93cc94c6. I've run this about 5 times in a row.

I've opened PR #3031 with the changes.

[petermetz]

@AndreAugusto11 Thank you very much for going the extra mile with the fix! I've done some research into the root cause (e.g. why your fix is actually a fix) and am now pretty confident that we fully understand the issue at hand and also that the fix won't have any unintended consequences, so happy days!

I updated the PR title and the commit message accordingly:

test(test-tooling): fix FabricTestLedger INVALID_ENDORSER_TRANSACTION

* reverts changes of commit 3371772c582389f6ee0c6fb66af875dd93cc94c6,
which seems to be breaking the Fabric Test Ledger

Results of further investigation into the root cause done by Peter:
1. The URLs we specify have the `grpcs://` protocol specified meaning that
TLS is used for securing the connection.
2. Certificates that are generated by the Fabric-provided boostrap scripts
when setting up crypto materials for the ledger are generated with `localhost`
as the hostname instead of the IP address of localhost.
3. The C++ gRPC implementation does not support mixing IP addresses and
hostnames when it comes to connections that are using TLS, e.g. if the
certificate we are using was made out for `localhost` then it won't work
for `127.0.0.1` even though technically from our perspective they meaning
the same thing (do note however that technically localhost could be set
up to resolve to something other than 127.0.0.1 in a DNS server so the 
difference is meaningful).

Source: https://github.com/grpc/grpc/issues/2691