Add rerun for failed flaky test

[roywei]

There is one flaky test that hangs sometimes in our CI:
test_warnings failed 2 times in recent 21 runs
We may want to consider automatically re-run it after failure.
But need to introduce a new dependency on pytest-rerunfailures
https://github.com/pytest-dev/pytest-rerunfailures