sambacc: avoid logging an error if cluster is being torn down #131
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
@Mergifyio rebase |
Saw this in a ceph teuthology run:
```
2024-08-20 20:39:57,289: DEBUG: Creating RADOS connection
2024-08-20 20:39:57,333: INFO: cluster meta content changed
2024-08-20 20:39:57,333: DEBUG: cluster meta: previous={'nodes':
[{'pnn': 0, 'identity': 'smb.adctdb1.0.0.ceph0.kdlxgn', 'node':
'192.168.76.200', 'state': 'ready'}, {'pnn': 1, 'identity':
'smb.adctdb1.1.0.ceph1.ngbqkk', 'node': '192.168.76.201', 'state':
'ready'}, {'pnn': 2, 'identity': 'smb.adctdb1.2.0.ceph2.rhmqnu', 'node':
'192.168.76.202', 'state': 'ready'}], '_source': 'cephadm'} current={}
2024-08-20 20:39:57,333: ERROR: error during ctdb_monitor_nodes: max()
arg is an empty sequence, count=0
Traceback (most recent call last):
File "/usr/lib/python3.9/site-packages/sambacc/commands/ctdb.py", line
479, in catch
yield
File "/usr/lib/python3.9/site-packages/sambacc/commands/ctdb.py", line
360, in ctdb_monitor_nodes
ctdb.monitor_cluster_meta_changes(
File "/usr/lib/python3.9/site-packages/sambacc/ctdb.py", line 561, in
monitor_cluster_meta_changes
expected_nodes = _cluster_meta_to_ctdb_nodes(
File "/usr/lib/python3.9/site-packages/sambacc/ctdb.py", line 506, in
_cluster_meta_to_ctdb_nodes
pnn_max = max(n["pnn"] for n in nodes) + 1 # pnn is zero indexed
ValueError: max() arg is an empty sequence
```
I could see from the ceph logs the smb cluster was being removed right
around this time. If we had nodes and they suddenly vanish we're likely
in the process of getting removed and we raced a tad with cephadm
removing services while the smb mgr module was removing the contents
of the .smb pool.
Signed-off-by: John Mulligan <[email protected]>
✅ Branch has been successfully rebased |
phlogistonjohn
force-pushed
the
jjm-fix-remove-err
branch
from
188bc71
to
410e571
Compare
anoopcs9
approved these changes
Aug 22, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Saw this in a ceph teuthology run:
I could see from the ceph logs the smb cluster was being removed right around this time. If we had nodes and they suddenly vanish we're likely in the process of getting removed and we raced a tad with cephadm removing services while the smb mgr module was removing the contents of the .smb pool.