Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[BUG] After the restart of the KB controller, when processing the handleRoleChangedEvent event, a disorder issue occurs #8857

Open
duanhui8 opened this issue Jan 23, 2025 · 2 comments · May be fixed by #8865 or #8871
Open

[BUG] After the restart of the KB controller, when processing the handleRoleChangedEvent event, a disorder issue occurs #8857

duanhui8 opened this issue Jan 23, 2025 · 2 comments · May be fixed by #8865 or #8871
Assignees
@xuriwuyun @shanshanying
Labels
kind/bug Something isn't working severity/major Great chance user will encounter the same problem test

Comments

@duanhui8
Copy link

Describe the bug

Image

After the restart of the KB controller, when processing the handleRoleChangedEvent event, a disorder issue occurs，causing the roles of the cluster pods to all become secondary.

To Reproduce
Steps to reproduce the behavior:

  1. pod logs

Image
[event:Success operation:checkRole originalRole:primary role:{"term":"1737561677278961","PodRoleNamePairs":[{"podName":"nacos-mysql-0","roleName":"secondary","podUid":"b1fd6aba-7db4-4319-bed5-c3c9c41d03e5"}]}]
[event:Success operation:checkRole originalRole:primary role:{"term":"1737561678036172","PodRoleNamePairs":[{"podName":"nacos-mysql-0","roleName":"secondary","podUid":"b1fd6aba-7db4-4319-bed5-c3c9c41d03e5"}]}]
[event:Success operation:checkRole originalRole:secondary role:{"term":"1737561678920613","PodRoleNamePairs":[{"podName":"nacos-mysql-0","roleName":"primary","podUid":"b1fd6aba-7db4-4319-bed5-c3c9c41d03e5"}]}]

  1. kubeblocks logs
    Image
    term1：1737561677278961
    term2：1737561678036172
    term3：1737561678920613

When processing the event, the order is disrupted, and the term2(1737561678036172) event is processed last, causing the pod.label.role to be updated to secondary instead of the correct primary.

  1. pod yaml

Image

Desktop (please complete the following information):

  • kubeblocks: 0.9.2
  • syncer: 0.3.9
@duanhui8 duanhui8 added the kind/bug Something isn't working label Jan 23, 2025
@shanshanying
Copy link
Contributor

By design the one with latest time wins. @xuriwuyun @leon-inf PTAL

@shanshanying shanshanying added severity/major Great chance user will encounter the same problem test labels Jan 23, 2025
@shanshanying
Copy link
Contributor

@JashBook and pls add the case to our regression test as well.

This was linked to pull requests Jan 23, 2025
fix: use update instead of patch to set pod's role #8865
Open
fix: use update instead of patch to set pod's role #8871
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@xuriwuyun xuriwuyun

@shanshanying shanshanying

Labels
kind/bug Something isn't working severity/major Great chance user will encounter the same problem test
Projects
None yet
Milestone
No milestone
3 participants
@xuriwuyun @duanhui8 @shanshanying