Skip to content

Commit

Permalink
tests: stabilize test_storage_controller_heartbeats (#9347)
Browse files Browse the repository at this point in the history
## Problem

This could fail with `reconciliation in progress` if running on a slow
test node such that background reconciliation happens at the same time
as we call consistency_check.

Example:
https://neon-github-public-dev.s3.amazonaws.com/reports/main/11258171952/index.html#/testresult/54889c9469afb232

## Summary of changes

- Call reconcile_until_idle before calling consistency check once,
rather than calling consistency check until it passes
  • Loading branch information
jcsp authored Oct 11, 2024
1 parent b2ecbf3 commit 1849356
Showing 1 changed file with 4 additions and 4 deletions.
8 changes: 4 additions & 4 deletions test_runner/regress/test_storage_controller.py
Original file line number Diff line number Diff line change
Expand Up @@ -1300,11 +1300,11 @@ def nodes_online():
node_to_tenants = build_node_to_tenants_map(env)
log.info(f"Back online: {node_to_tenants=}")

# ... expecting the storage controller to reach a consistent state
def storage_controller_consistent():
env.storage_controller.consistency_check()
# ... background reconciliation may need to run to clean up the location on the node that was offline
env.storage_controller.reconcile_until_idle()

wait_until(30, 1, storage_controller_consistent)
# ... expecting the storage controller to reach a consistent state
env.storage_controller.consistency_check()


def test_storage_controller_re_attach(neon_env_builder: NeonEnvBuilder):
Expand Down

1 comment on commit 1849356

@github-actions
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

5094 tests run: 4886 passed, 1 failed, 207 skipped (full report)


Failures on Postgres 15

# Run all failed tests locally:
scripts/pytest -vv -n $(nproc) -k "test_pageserver_chaos[release-pg15-None]"
Flaky tests (3)

Postgres 17

Postgres 16

Postgres 15

Test coverage report is not available

The comment gets automatically updated with the latest test results
1849356 at 2024-10-11T09:27:02.629Z :recycle:

Please sign in to comment.