Flakiness in e2e tests in CI

[jpraynaud]

Why

We have noticed some flakiness in the e2e tests in the CI which fails with a high rate.

Investigations

E2E test results with Cardano node version 9.1.1:

• Run without era switch (1 failure out of 20 occurrences)

  • Error: Minimum expected Cardano stake distribution epoch not reached : 15 < 16

• Run with era switch without regenesis (6 failures out of 20 occurrences)

  • Error: Minimum expected Cardano stake distribution epoch not reached : 20 < 21
  • Error: Timeout exhausted assert_node_producing_mithril_stake_distribution, no response from http://localhost:8080/aggregator/artifact/mithril-stake-distributions
  • Error: Minimum expected mithril stake distribution epoch not reached : 19 < 21
  • Error: Minimum expected mithril stake distribution epoch not reached : 19 < 21
  • Error: Minimum expected Cardano stake distribution epoch not reached : 20 < 21
  • Error: Minimum expected mithril stake distribution epoch not reached : 19 < 21

• Run with era switch with regenesis on switch (3 failures out of 20 occurrences)

  • Error: Minimum expected mithril stake distribution epoch not reached : 21 < 26
  • Error: Minimum expected mithril stake distribution epoch not reached : 25 < 26
  • Error: Minimum expected Cardano stake distribution epoch not reached : 15 < 16

---

Run with new logs following the merge of #2009 & #2010:

• Run without era switch (0 failure out of 20 occurrences)
• Run with era switch without regenesis (6 failures out of 20 occurrences)
• Run with era switch with regenesis on switch (2 failures out of 20 occurrences)

What

Investigate the sources of flakiness and fix them.

How

• Investigate the sources of flakiness:
  • Epoch consistency during epoch changes when signer is signing leading to error like:
    Error: Minimum expected mithril stake distribution epoch not reached : 19 < 21
• Fix the sources of flakiness

Later

• Transfer funds errors at the beginning of the E2E test leading to error like:
  >>>> Timeout: Funds were not transferred successfully to Mithril era address within 10 attempts