Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

End-2-end tests fail randomly #505

Closed
piotrm50 opened this issue Nov 8, 2023 · 5 comments
Closed

End-2-end tests fail randomly #505

piotrm50 opened this issue Nov 8, 2023 · 5 comments
Labels
bug Something isn't working non-actionable
Milestone
v1.0.0-beta

Comments

@piotrm50
Copy link
Collaborator

piotrm50 commented Nov 8, 2023

Sometimes end-2-end tests fail randomly, because a block does not exist. I've seen this problem multiple times already, usually in -race tests, but last time it happened in a regular test, while the -race tests all passed. I'm not sure if it's always the same test, but the error message is always the same AssertBlock: nodeX: block BlockID(xxx) does not exist, so my guess would be that that the timeout for Eventually is too short because the machine can't keep up, but it's just a guess.

2023-11-07T17:29:21.5412795Z --- FAIL: Test_MultipleAttachments (5.06s)
2023-11-07T17:29:21.5413054Z     testsuite.go:535: 
2023-11-07T17:29:21.5413630Z         	Error Trace:	/actions-runner/_work/iota-core/iota-core/pkg/testsuite/testsuite.go:535
2023-11-07T17:29:21.5414310Z         	            				/actions-runner/_work/iota-core/iota-core/pkg/testsuite/blocks.go:17
2023-11-07T17:29:21.5415013Z         	            				/actions-runner/_work/iota-core/iota-core/pkg/testsuite/blocks.go:47
2023-11-07T17:29:21.5415914Z         	            				/actions-runner/_work/iota-core/iota-core/pkg/testsuite/testsuite_issue_blocks.go:34
2023-11-07T17:29:21.5416741Z         	            				/actions-runner/_work/iota-core/iota-core/pkg/testsuite/testsuite_issue_blocks.go:148
2023-11-07T17:29:21.5417374Z         	            				/actions-runner/_work/iota-core/iota-core/pkg/tests/booker_test.go:143
2023-11-07T17:29:21.5417575Z         	Error:      	condition never satisfied
2023-11-07T17:29:21.5417751Z         	Test:       	Test_MultipleAttachments
2023-11-07T17:29:21.5418066Z         	Messages:   	AssertBlock: nodeA: block BlockID(B.1.1:1) does not exist
@piotrm50 piotrm50 converted this from a draft issue Nov 8, 2023
@piotrm50 piotrm50 added this to the 1.0 RC milestone Nov 8, 2023
@piotrm50 piotrm50 added the bug Something isn't working label Nov 8, 2023
@piotrm50 piotrm50 moved this to Backlog in iota-core Nov 8, 2023
@PhilippGackstatter
Copy link
Contributor

https://github.com/iotaledger/iota-core/actions/runs/6800220411/job/18488278336

This one might be related: I'm also getting "condition never satisfied" but the error was a different one:

2023-11-08T15:19:54.7609678Z --- FAIL: TestProtocol_EngineRollbackNoFinalizationLastSlot (21.47s)
2023-11-08T15:19:54.7609949Z     testsuite.go:526: 
2023-11-08T15:19:54.7610442Z            Error Trace:    /actions-runner/_work/iota-core/iota-core/pkg/testsuite/testsuite.go:526
2023-11-08T15:19:54.7611179Z                                                    /actions-runner/_work/iota-core/iota-core/pkg/testsuite/storage_settings.go:83
2023-11-08T15:19:54.7611805Z                                                    /actions-runner/_work/iota-core/iota-core/pkg/testsuite/node_state.go:24
2023-11-08T15:19:54.7612555Z                                                    /actions-runner/_work/iota-core/iota-core/pkg/tests/protocol_engine_rollback_test.go:568
2023-11-08T15:19:54.7612732Z            Error:          condition never satisfied
2023-11-08T15:19:54.7613009Z            Test:           TestProtocol_EngineRollbackNoFinalizationLastSlot
2023-11-08T15:19:54.7613460Z            Messages:       AssertLatestCommitmentSlotIndex: node2: expected SlotIndex(17), got SlotIndex(16)

A rerun was successful, so perhaps also time-related.

@karimodm karimodm self-assigned this Nov 13, 2023
@karimodm
Copy link
Contributor

karimodm commented Nov 13, 2023 •
edited
Loading

Might be fixed by #466 and #441

@karimodm karimodm removed their assignment Nov 13, 2023
@piotrm50
Copy link
Collaborator Author

Could be related:

2023-11-14T14:26:15.6839940Z node7 > [Candidate - 6ad77e22] Scheduler.BlockScheduled: BlockID(P1:17.3-node4:17)
2023-11-14T14:26:15.6840262Z node7 > [Candidate - 6ad77e22] Scheduler.BlockScheduled: BlockID(P1:17.1-node4:17)
2023-11-14T14:26:15.6840622Z node7 > [Candidate - 6ad77e22] Scheduler.BlockScheduled: BlockID(P1:17.2-node4:17)
2023-11-14T14:26:15.6840939Z node7 > [Candidate - 6ad77e22] Scheduler.BlockScheduled: BlockID(P1:17.2-node2:17)
2023-11-14T14:26:15.6841262Z node7 > [Candidate - 6ad77e22] Scheduler.BlockScheduled: BlockID(P1:17.3-node3:17)
2023-11-14T14:26:15.6841568Z node7 > [Candidate - 6ad77e22] Booker.BlockBooked: BlockID(P1:17.1-node4:17)
2023-11-14T14:26:15.6841938Z node7 > [Candidate - 6ad77e22] SybilProtection.BlockProcessed: BlockID(P1:17.1-node4:17)
2023-11-14T14:26:15.6842304Z node7 > [Candidate - 6ad77e22] CommitmentFilter.BlockAllowed: BlockID(P1:17.1-node4:17)
2023-11-14T14:26:15.6842612Z node7 > [Candidate - 6ad77e22] BlockDAG.BlockSolid: BlockID(P1:17.1-node4:17)
2023-11-14T14:26:15.6842943Z node7 > [Candidate - 6ad77e22] Scheduler.BlockScheduled: BlockID(P1:17.2-node1:17)
2023-11-14T14:26:15.6843269Z node7 > [Candidate - 6ad77e22] Scheduler.BlockScheduled: BlockID(P1:17.2-node3:17)
2023-11-14T14:26:15.6843427Z node6 > Run finished with error: context canceled
2023-11-14T14:26:15.6843584Z node7 > Run finished with error: context canceled
2023-11-14T14:26:15.6843732Z node8 > Run finished with error: context canceled
2023-11-14T14:26:15.6843974Z --- FAIL: TestProtocol_EngineSwitching (129.22s)
2023-11-14T14:26:15.6844452Z     testsuite.go:542: 
2023-11-14T14:26:15.6845037Z         	Error Trace:	/actions-runner/_work/iota-core/iota-core/pkg/testsuite/testsuite.go:542
2023-11-14T14:26:15.6845764Z         	            				/actions-runner/_work/iota-core/iota-core/pkg/testsuite/fork.go:44
2023-11-14T14:26:15.6846700Z         	            				/actions-runner/_work/iota-core/iota-core/pkg/tests/protocol_engine_switching_test.go:368
2023-11-14T14:26:15.6846913Z         	Error:      	condition never satisfied
2023-11-14T14:26:15.6847138Z         	Test:       	TestProtocol_EngineSwitching
2023-11-14T14:26:15.6847504Z         	Messages:   	AssertMainEngineSwitchedCount: node6: expected 1, got 0
2023-11-14T14:26:15.6847943Z Setup TestSuite - Test_BookInCommittedSlot  @  2023-11-14 14:24:05.019657698 +0000 UTC m=+235.458221157

@cyberphysic4l
Copy link
Contributor

Another issue which seems to be related to timing and can be fixed by extending the wait time of the test suite during assertions.

--- FAIL: Test_SpendRejectedCommittedRace (5.23s)
    testsuite.go:565: 
        	Error Trace:	/actions-runner/_work/iota-core/iota-core/pkg/testsuite/testsuite.go:565
        	            				/actions-runner/_work/iota-core/iota-core/pkg/testsuite/blocks.go:139
        	            				/actions-runner/_work/iota-core/iota-core/pkg/tests/booker_test.go:368
        	Error:      	condition never satisfied
        	Test:       	Test_SpendRejectedCommittedRace
        	Messages:   	AssertBlocksInCacheConflicts: node1: block BlockID(block1.2:1) conflict count incorrect: expected conflicts TransactionIDs(tx2:0), got TransactionIDs(tx1:0, tx2:0)

@karimodm karimodm closed this as completed Dec 4, 2023
@github-project-automation github-project-automation bot moved this from Backlog to Done in iota-core Dec 4, 2023
@karimodm
Copy link
Contributor

karimodm commented Dec 4, 2023

This is a collection of different bugs that might have been solved already by: assertion / notarization race fix #572 , reactive #320 and #531 , etc.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working non-actionable
Projects
iota-core
Archived in project
Milestone
v1.0.0-beta
Development

No branches or pull requests
4 participants
@karimodm @piotrm50 @PhilippGackstatter @cyberphysic4l