Tracking common test flakes #208

phlogistonjohn · 2022-06-06T15:45:29Z

phlogistonjohn
Jun 6, 2022
Maintainer

Without formally committing tracking particular flakes to an issue, I'd like to gather the common causes of test failures in one place, so a discussion seems to make sense. Please add repeat test failures you see to this discusssion, thanks!

phlogistonjohn · 2022-06-06T15:48:38Z

phlogistonjohn
Jun 6, 2022
Maintainer Author

NT_STATUS_IO_TIMEOUT occurs running smbclient.

=== RUN   TestIntegration/smbShares/smbSharesInDefault/TestShareAccessByServiceName/TestLogin
    share_access_test.go:41: 
        	Error Trace:	share_access_test.go:41
        	Error:      	Received unexpected error:
        	            	failed to execute smbclient command: ['smbclient' '-Usambauser%1nsecurely' '//tshare3.default.svc.cluster.local/My Other Share' '-c' 'ls']: failed executing command (pod:samba-operator-system/smbclient container:): command terminated with exit code 1 [exit: 1; stdout: ; stderr: do_connect: Connection to tshare3.default.svc.cluster.local failed (Error NT_STATUS_IO_TIMEOUT)
        	            	]
        	Test:       	TestIntegration/smbShares/smbSharesInDefault/TestShareAccessByServiceName/TestLogin

1 reply

spuiuk Jun 9, 2022
Collaborator

PR #215 had two errors. One of them was

=== RUN TestIntegration/smbShares/users1/TestShareAccessByIP/TestLogin
share_access_test.go:42:
Error Trace: share_access_test.go:42
Error: Received unexpected error:
failed to execute smbclient command: ['smbclient' '-Usambauser%1nsecurely' '//10.42.0.20/My Share' '-c' 'ls']: failed executing command (pod:samba-operator-system/smbclient container:): command terminated with exit code 1 [exit: 1; stdout: ; stderr: do_connect: Connection to 10.42.0.20 failed (Error NT_STATUS_HOST_UNREACHABLE)
]
Test: TestIntegration/smbShares/users1/TestShareAccessByIP/TestLogin
=== RUN TestIntegration/smbShares/users1/TestShareAccessByIP/TestPutFile
share_access_test.go:57:
Error Trace: share_access_test.go:57
Error: Received unexpected error:
failed to execute smbclient command: ['smbclient' '-Usambauser%1nsecurely' '//10.42.0.20/My Share' '-c' 'put profile.jpeg']: failed executing command (pod:samba-operator-system/smbclient container:): command terminated with exit code 1 [exit: 1; stdout: ; stderr: do_connect: Connection to 10.42.0.20 failed (Error NT_STATUS_HOST_UNREACHABLE)
]

followed by the "too many pods match selector" error.

phlogistonjohn · 2022-06-06T17:32:43Z

phlogistonjohn
Jun 6, 2022
Maintainer Author

Timeouts. Seen at least three times this week.

    smb_share_test.go:58: 
        	Error Trace:	smb_share_test.go:58
        	            				suite.go:148
        	Error:      	Received unexpected error:
        	            	too many pods match selector (opts: {Namespace:samba-operator-system LabelSelector:samba-operator.samba.org/service=tshare1 MaxFound:1 MinFound:0}; pods: [pod(samba-operator-system/tshare1-6ff8748f49-28n8m, ready=false)])
        	Test:       	TestIntegration/smbShares/users1/TestMetricsOnPod
        	Messages:   	smb server pod is not ready
=== RUN   TestIntegration/smbShares/users1/TestPodsReady
    smb_share_test.go:57: 
        	Error Trace:	smb_share_test.go:57
        	            				suite.go:148
        	Error:      	Received unexpected error:
        	            	too many pods match selector (opts: {Namespace:samba-operator-system LabelSelector:samba-operator.samba.org/service=tshare1 MaxFound:1 MinFound:0}; pods: [])
        	Test:       	TestIntegration/smbShares/users1/TestPodsReady
        	Messages:   	smb server pod does not exist
=== RUN   TestIntegration/smbShares/users1/TestShareAccessByIP
    smb_share_test.go:57: 
        	Error Trace:	smb_share_test.go:57
        	            				suite.go:148
        	Error:      	Received unexpected error:
        	            	too many pods match selector (opts: {Namespace:samba-operator-system LabelSelector:samba-operator.samba.org/service=tshare1 MaxFound:1 MinFound:0}; pods: [])
        	Test:       	TestIntegration/smbShares/users1/TestShareAccessByIP
        	Messages:   	smb server pod does not exist
=== RUN   TestIntegration/smbShares/users1/TestShareAccessByServiceName
    smb_share_test.go:57: 
        	Error Trace:	smb_share_test.go:57
        	            				suite.go:148
        	Error:      	Received unexpected error:
        	            	too many pods match selector (opts: {Namespace:samba-operator-system LabelSelector:samba-operator.samba.org/service=tshare1 MaxFound:1 MinFound:0}; pods: [])
        	Test:       	TestIntegration/smbShares/users1/TestShareAccessByServiceName
        	Messages:   	smb server pod does not exist

3 replies

spuiuk Jun 7, 2022
Collaborator

Similar errors when running for PR#212

=== RUN TestIntegration/smbShares/users1/TestShareAccessByServiceName
smb_share_test.go:57:
Error Trace: smb_share_test.go:57
suite.go:148
Error: Received unexpected error:
too many pods match selector (opts: {Namespace:samba-operator-system LabelSelector:samba-operator.samba.org/service=tshare1 MaxFound:1 MinFound:0}; pods: [])
Test: TestIntegration/smbShares/users1/TestShareAccessByServiceName
Messages: smb server pod does not exist
=== RUN TestIntegration/smbShares/users1/TestShareEvents
smb_share_test.go:57:
Error Trace: smb_share_test.go:57
suite.go:148
Error: Received unexpected error:
too many pods match selector (opts: {Namespace:samba-operator-system LabelSelector:samba-operator.samba.org/service=tshare1 MaxFound:1 MinFound:0}; pods: [])
Test: TestIntegration/smbShares/users1/TestShareEvents
Messages: smb server pod does not exist
=== RUN TestIntegration/smbShares/users1/TestSmbShareServerGroup
smb_share_test.go:57:
Error Trace: smb_share_test.go:57
suite.go:148
Error: Received unexpected error:
too many pods match selector (opts: {Namespace:samba-operator-system LabelSelector:samba-operator.samba.org/service=tshare1 MaxFound:1 MinFound:0}; pods: [])
Test: TestIntegration/smbShares/users1/TestSmbShareServerGroup
Messages: smb server pod does not exist

spuiuk Jun 8, 2022
Collaborator

PR #214
=== RUN TestIntegration/smbShares/users1/TestShareAccessByServiceName
smb_share_test.go:57:
Error Trace: smb_share_test.go:57
suite.go:148
Error: Received unexpected error:
too many pods match selector (opts: {Namespace:samba-operator-system LabelSelector:samba-operator.samba.org/service=tshare1 MaxFound:1 MinFound:0}; pods: [])
Test: TestIntegration/smbShares/users1/TestShareAccessByServiceName
Messages: smb server pod does not exist
=== RUN TestIntegration/smbShares/users1/TestShareEvents
smb_share_test.go:57:
Error Trace: smb_share_test.go:57
suite.go:148
Error: Received unexpected error:
too many pods match selector (opts: {Namespace:samba-operator-system LabelSelector:samba-operator.samba.org/service=tshare1 MaxFound:1 MinFound:0}; pods: [])
Test: TestIntegration/smbShares/users1/TestShareEvents
Messages: smb server pod does not exist
=== RUN TestIntegration/smbShares/users1/TestSmbShareServerGroup
smb_share_test.go:57:
Error Trace: smb_share_test.go:57
suite.go:148
Error: Received unexpected error:
too many pods match selector (opts: {Namespace:samba-operator-system LabelSelector:samba-operator.samba.org/service=tshare1 MaxFound:1 MinFound:0}; pods: [])
Test: TestIntegration/smbShares/users1/TestSmbShareServerGroup
Messages: smb server pod does not exist

spuiuk Jun 9, 2022
Collaborator

PR #215 had two different failures. One of them was

=== RUN TestIntegration/smbShares/users1/TestShareAccessByServiceName
smb_share_test.go:57:
Error Trace: smb_share_test.go:57
suite.go:148
Error: Received unexpected error:
too many pods match selector (opts: {Namespace:samba-operator-system LabelSelector:samba-operator.samba.org/service=tshare1 MaxFound:1 MinFound:0}; pods: [])
Test: TestIntegration/smbShares/users1/TestShareAccessByServiceName
Messages: smb server pod does not exist
=== RUN TestIntegration/smbShares/users1/TestShareEvents
smb_share_test.go:57:
Error Trace: smb_share_test.go:57
suite.go:148
Error: Received unexpected error:
too many pods match selector (opts: {Namespace:samba-operator-system LabelSelector:samba-operator.samba.org/service=tshare1 MaxFound:1 MinFound:0}; pods: [])
Test: TestIntegration/smbShares/users1/TestShareEvents
Messages: smb server pod does not exist
=== RUN TestIntegration/smbShares/users1/TestSmbShareServerGroup
smb_share_test.go:57:
Error Trace: smb_share_test.go:57
suite.go:148
Error: Received unexpected error:
too many pods match selector (opts: {Namespace:samba-operator-system LabelSelector:samba-operator.samba.org/service=tshare1 MaxFound:1 MinFound:0}; pods: [])
Test: TestIntegration/smbShares/users1/TestSmbShareServerGroup
Messages: smb server pod does not exist

anoopcs9 · 2022-06-07T07:02:26Z

anoopcs9
Jun 7, 2022
Collaborator

Most common failures from CentOS CI:

=== RUN   TestIntegration/createDelete/simple/TestCreateAndDelete
error: waiting for older pods to be cleaned: context deadline exceeded
    assertion_compare.go:313: 
        	Error Trace:	cleanup_test.go:157
        	Error:      	"3" is not greater than "3"
        	Test:       	TestIntegration/createDelete/simple/TestCreateAndDelete
        	Messages:   	[]

Another one:

=== RUN   TestIntegration/reconciliation/scaleoutCluster/TestScaleoutClusterSuite
    reconcile_test.go:151: 
        	Error Trace:	reconcile_test.go:151
        	Error:      	Not equal: 
        	            	expected: 3
        	            	actual  : 2
        	Test:       	TestIntegration/reconciliation/scaleoutCluster/TestScaleoutClusterSuite
        	Messages:   	Clustersize not as expected

2 replies

anoopcs9 Jul 8, 2022
Collaborator

Most common failures from CentOS CI:

=== RUN   TestIntegration/createDelete/simple/TestCreateAndDelete
error: waiting for older pods to be cleaned: context deadline exceeded
    assertion_compare.go:313: 
        	Error Trace:	cleanup_test.go:157
        	Error:      	"3" is not greater than "3"
        	Test:       	TestIntegration/createDelete/simple/TestCreateAndDelete
        	Messages:   	[]

With all recent cleanup and improvements to tests, above failure is no longer seen.

Another one:

=== RUN   TestIntegration/reconciliation/scaleoutCluster/TestScaleoutClusterSuite
    reconcile_test.go:151: 
        	Error Trace:	reconcile_test.go:151
        	Error:      	Not equal: 
        	            	expected: 3
        	            	actual  : 2
        	Test:       	TestIntegration/reconciliation/scaleoutCluster/TestScaleoutClusterSuite
        	Messages:   	Clustersize not as expected

But happened to notice this again.

phlogistonjohn Jul 8, 2022
Maintainer Author

OK. Thanks for the heads up. I'm very fuzzy on what that one could mean, so I won't be jumping on that one with a patch right away.

synarete · 2022-06-08T12:08:30Z

synarete
Jun 8, 2022
Collaborator

#213 make explicit ImagePullPolicy=IfNotPresent for samba-pods (by default; may be configured). On my minikube setup it fixed those smb server pod does not exist failures, but I admit that it is a bit of shot-in-the-dark.

1 reply

synarete Jun 8, 2022
Collaborator

Unfortunately, it did not fix the failures in CI :(
Maybe it would be better to keep #213 for v0.3 or as draft.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Tracking common test flakes #208

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 4 comments 7 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

Select a reply

Tracking common test flakes #208

phlogistonjohn Jun 6, 2022 Maintainer

Replies: 4 comments · 7 replies

phlogistonjohn Jun 6, 2022 Maintainer Author

spuiuk Jun 9, 2022 Collaborator

phlogistonjohn Jun 6, 2022 Maintainer Author

spuiuk Jun 7, 2022 Collaborator

spuiuk Jun 8, 2022 Collaborator

spuiuk Jun 9, 2022 Collaborator

anoopcs9 Jun 7, 2022 Collaborator

anoopcs9 Jul 8, 2022 Collaborator

phlogistonjohn Jul 8, 2022 Maintainer Author

synarete Jun 8, 2022 Collaborator

synarete Jun 8, 2022 Collaborator

phlogistonjohn
Jun 6, 2022
Maintainer

Replies: 4 comments 7 replies

phlogistonjohn
Jun 6, 2022
Maintainer Author

spuiuk Jun 9, 2022
Collaborator

phlogistonjohn
Jun 6, 2022
Maintainer Author

spuiuk Jun 7, 2022
Collaborator

spuiuk Jun 8, 2022
Collaborator

spuiuk Jun 9, 2022
Collaborator

anoopcs9
Jun 7, 2022
Collaborator

anoopcs9 Jul 8, 2022
Collaborator

phlogistonjohn Jul 8, 2022
Maintainer Author

synarete
Jun 8, 2022
Collaborator

synarete Jun 8, 2022
Collaborator