zkApp transactions stuck in local mempool

[garethtdavies]

Preliminary Checks

• This issue is not a duplicate. Before opening a new issue, please search existing issues: https://github.com/MinaProtocol/mina/issues
• This issue is not a question, feature request, RFC, or anything other than a bug report. Please post those things in GitHub Discussions: https://github.com/MinaProtocol/mina/discussions

Description

I sent 1000 zkApp transactions (spaced at a 3-second interval). Only ~500 were included. The rest are stuck in a local mempool. However, it isn't possible to resend using o1js await tx.send(); as it will not rebroadcast the transaction:

Error: Failed to send transaction {
  statusCode: 200,
  statusText: `Couldn't send zkApp command: ["Duplicate"]`
}

Stuck tx in local mempool:

The only way to resolve this would be to restart the node, or use a different node to send.

Steps to Reproduce

Broadcast many zkApp tx, and after some time they will get dropped from other mempools.

Expected Result

Transactions are included.

Actual Result

Transactions are dropped but remain in local mempool.

How frequently do you see this issue?

Frequently

What is the impact of this issue on your ability to run a node?

High

Status

Latest testworld

Additional information

No response

[deepthiskumar]

@garethtdavies this is most probably the same as the issue we were experiencing due to delay in block production (#14531) Blocks filled with zkApp transactions take longer than a slot to be broadcasted and therefore rejected, leaving all transactions in the pool.

[garethtdavies]

Thanks. Will the proposed staged ledger fix (eluded to by Brandon) be included in time to test this on Testworld?

Edit: this one #14547

[amc-ie]

Triage update: This is linked to block production slowness/performance and should be fixed with our upcoming performance improvements in rampup7.

[dkijania]

@garethtdavies rampup7 with performance improvements was released. Can you retest the issue on the newest testworld-2-0 release?

[amc-ie]

This issue should be closed with the rampup8 release. Please reopen if the issue persists on the latest version. Thank you!