Intermittent test failure in file browser test #327
Comments
|
Happened today again:
|
frasertweedale
changed the title
|
I've looked into this a little bit today. The test steps seem fine and I can't find any sub strings which are too generic so the test run would skip ahead. The only hypothesis I could come up with is perhaps the attachment list and it's length indicator in the status bar. Our test framework doesn't really let us assert that a string has been removed from the previous screen (maybe a feature we should implement?). Instead of asserting that the last file was deleted, because it's now missing, we use the item indicator in the status bar and assert with the list length, e.g. Now I'm wondering whether perhaps the list length is recalculated quicker (since it's asynchronous) than the redrawing of the entire UI can take place. In the screen shot the deleted list entry still looks like it's selected, even tho the entry is gone. If that's a theory to pursue, I'm currently not sure what else to assert on which would be a stable indicator that the file is not in the list/UI. @frasertweedale does that sound plausible as a possible cause? |
This test was failing now and then and the hypothesis causing the failure is the thread traversing the spine of the list of threads. This thread most likely evaluates the length of the displayed threads very quickly. In fact there is only one thread to display, so the length is most likely returned so quick, that the UI has not been repainted. Last screen captures from tmux still show the last entry selected, even though the entry itself is gone. This patch asserts on the inverse that, the entry (the screen shot attachment) is gone from the screen. This should be enough to basically assert that the UI has been repainted. Fixes #327
This test was failing now and then and the hypothesis causing the failure is the thread traversing the spine of the list of threads. This thread most likely evaluates the length of the displayed threads very quickly. In fact there is only one thread to display, so the length is most likely returned so quick, that the UI has not been repainted. Last screen captures from tmux still show the last entry selected, even though the entry itself is gone. This patch asserts on the inverse that, the entry (the screen shot attachment) is gone from the screen. This should be enough to basically assert that the UI has been repainted. Fixes #327
|
Had this come up again today in a test failure: https://travis-ci.org/purebred-mua/purebred/jobs/629258818?utm_medium=notification&utm_source=email Unfortunately it seems, this issue isn't fixed yet :/ |
|
|
|
Looking into this closer, I wonder whether there was a slight chance that I had made the branch in which the above error appeared before I merged this fix. While I authored the fix in November, we kept it around for a while. I haven't seen the problem re-occur on master, so perhaps we keep this one open for another month or so and if it really doesn't appear again I'll close it. |
|
This hasn't resurfaced in a few weeks now making me think that I must have created my branch with the problem in it. I'll close the issue. |
|
This job https://travis-ci.org/github/purebred-mua/purebred/jobs/675540114 seems to have failed at the same point. |
|
Saw the failure again today. Attaching log.txt |
|
Happened again today. Attaching |
The change 47c72c2 tried to address the intermittendly failing test, but I think made the mistake of "hardcoding" the file we assume to be removing from the attachment list. The problem however is, that the list of files we're adding as attachment varies with the environment. While the `screenshot.png` is probably the attachment in most cases, it isn't the attachment in **all** cases. The assertion that the next screen does not show a file which we never added is always true. Instead, firstly, this patch asserts that the last filename is actually listed as an attachment. Secondly, we assert that when the attachment is removed the last file from before is gone. Furthermore we also check whether the item count has changed. Fixes: #327
The change 47c72c2 tried to address the intermittendly failing test, but I think made the mistake of "hardcoding" the file we assume to be removing from the attachment list. The problem however is, that the list of files we're adding as attachment varies with the environment. While the `screenshot.png` is probably the attachment in most cases, it isn't the attachment in **all** cases. The assertion that the next screen does not show a file which we never added is always true. Instead, firstly, this patch asserts that the last filename is actually listed as an attachment. Secondly, we assert that when the attachment is removed the last file from before is gone. Furthermore we also check whether the item count has changed. Fixes: #327
|
Haha look who's back: https://travis-ci.org/github/purebred-mua/purebred/jobs/716221854 |
Describe the bug
I've had a test run which seemed to have failed with an intermittent failure, since it only happend on the nix build. The last screenshot seems to suggest that for whatever reason the list item wasn't selecting the last remaining entity to trigger the error message.
Perhaps it is again a too generic intermittent step which causes the tests to skip too quickly?
To Reproduce
No idea at this point.
Expected behavior
Test run without a failure
Screenshots
See travis build log.txt
Additional context
None
The text was updated successfully, but these errors were encountered: