Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[JANSA] Cockpit is a monitor thread and is not a pod worker. #20823 jansa backport #20827

Merged
merged 1 commit into from
Nov 19, 2020

Conversation

jrafanie
Copy link
Member

@jrafanie jrafanie commented Nov 18, 2020

Follow up to #20792

[Modified for jansa to use guid column instead of system_uid]
jansa backport of #20823

It runs in the orchestrator in pods in a thread and launches a process to communicate with cockpit.
The threaded worker monitors this process, has a miq_workers row, but does not run in a separate pod
and therefore doesn't currently update the row with it's pod name. Therefore, we need to exclude cockpit
rows from the code that monitors for orphaned pod miq_workers.

In the future, we could populate the cockpit miq_workers row with the pod name of the orchestrator pod it's
running on but that would be a larger change and perhaps we should discuss that with a larger discussion about
cockpit and if it works as we want in pods.

EDIT:
This PR corrects the problem for this issue. It doesn't fix cockpit in pods but it resolves the constant restarts.

Follow up to ManageIQ#20792

[Modified for jansa to use guid column instead of system_uid]

It runs in the orchestrator in pods in a thread and launches a process to communicate with cockpit.
The threaded worker monitors this process, has a miq_workers row, but does not run in a separate pod
and therefore doesn't currently update the row with it's pod name.  Therefore, we need to exclude cockpit
rows from the code that monitors for orphaned pod miq_workers.

In the future, we could populate the cockpit miq_workers row with the pod name of the orchestrator pod it's
running on but that would be a larger change and perhaps we should discuss that with a larger discussion about
cockpit and if it works as we want in pods.
@jrafanie
Copy link
Member Author

I'm running an image build of jansa with this PR to make sure cockpit starts and is excluded from the miq_workers pod row syncup

@jrafanie
Copy link
Member Author

jrafanie commented Nov 18, 2020

I'm running an image build of jansa with this PR to make sure cockpit starts and is excluded from the miq_workers pod row syncup

ok, this PR corrects the constant restarts of the cockpit "worker" due to the worker being removed. It doesn't work in pods right now but at least it doesn't thrash the system (like it did before this PR), compromising the orchestrator pod.

06:06:09 ~/Code/manageiq (master) (2.5.8) - oc exec -t deploy/postgresql -- psql -U root vmdb_production -c "select id, guid, status, type from miq_workers;"
 id |                 guid                 | status  |          type
----+--------------------------------------+---------+------------------------
 12 | 1-priority-799b5788f9-9l9mw          | started | MiqPriorityWorker
 13 | 1-generic-86784759bf-gxksk           | started | MiqGenericWorker
 14 | 1-reporting-5bbfbddb97-p4cwb         | started | MiqReportingWorker
 16 | 1-remote-console-7887444f78-9rhdf    | started | MiqRemoteConsoleWorker
 15 | 1-schedule-6848bf59cf-5hkfq          | started | MiqScheduleWorker
 18 | 1-event-handler-d89d88484-6cbmf      | started | MiqEventHandler
 17 | 1-priority-799b5788f9-zvwg2          | started | MiqPriorityWorker
 19 | 1-ui-65f669cdc5-d8zlh                | started | MiqUiWorker
 20 | 1-web-service-5f9979f46d-r9sp2       | started | MiqWebServiceWorker
 21 | 1-generic-86784759bf-f29tk           | started | MiqGenericWorker
 22 | 1-reporting-5bbfbddb97-mqmc9         | started | MiqReportingWorker
 24 | 88d8560d-9edf-4a85-8fe5-758e78ee4de4 | started | MiqCockpitWsWorker
(12 rows)

Note, cockpit has lots of problems in pods and we should consider if it's something that can work and if that effort is desired. At the very least, I wonder if we should disable the role in pods so you can't activate a broken role.

See: #20829

@simaishi simaishi merged commit 8629c33 into ManageIQ:jansa Nov 19, 2020
@jrafanie jrafanie deleted the jansa_cockpit_is_not_a_pod_worker branch July 7, 2021 14:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants