Collector setup fails

[petrk-afk]

Hello,

on freshly installed v2 when I try to configure any collector on step 3 I get error "Failed to load additional options." almost immediately, is it connectivity related?
Clicking on refresh shows "Initializing associated options" for second and then goes back to the "Failed to load additional options."

Is there a logfile to review?

Any hints?

Thx
Petr

[ImMin5]

Hi, @petrk-afk

It seems that the plugin might not have run successfully. If possible, could you access the plugin pod and check its current state?

Could you give me some more details?
like version of inventory or plugin name that you use.(on screen shot)

[petrk-afk]

The plugin was AWS Cloud Service Collector, but the error is the same for others like Azure

[petrk-afk]

Hello,

the log from the plugin pod

komik@cloudforet-poc-vm:~$ kubectl logs plugin-546f9c5b8-qhd6c -n cloudforet --timestamps -f
2024-11-07T14:07:33.948109900Z 2024-11-07T23:07:33.947Z [DEBUG] (init.py:149) Create MongoDB Connection: default
2024-11-07T14:07:34.043821009Z 2024-11-07T23:07:34.043Z [DEBUG] (init.py:173) Create MongoDB Indexes (Supervisor Model: 7 Indexes)
2024-11-07T14:07:34.098806431Z 2024-11-07T23:07:34.097Z [DEBUG] (init.py:173) Create MongoDB Indexes (InstalledPlugin Model: 10 Indexes)
2024-11-07T14:07:34.107059409Z 2024-11-07T23:07:34.105Z [DEBUG] (init.py:173) Create MongoDB Indexes (InstalledPluginRef Model: 7 Indexes)
2024-11-07T14:07:35.444049607Z 2024-11-07T23:07:35.443Z [DEBUG] (server.py:69) Loaded Services:
2024-11-07T14:07:35.444137907Z - spaceone.api.plugin.v1.Plugin
2024-11-07T14:07:35.444147308Z - spaceone.api.plugin.v1.Supervisor
2024-11-07T14:07:35.444152108Z - grpc.health.v1.Health
2024-11-07T14:07:35.444156508Z - spaceone.api.core.v1.ServerInfo
2024-11-07T14:07:35.444559211Z 2024-11-07T23:07:35.444Z [INFO] (server.py:73) Start gRPC Server (plugin): port=50051, max_workers=1000
2024-11-07T14:09:34.236168378Z 2024-11-07T23:09:34.224Z [DEBUG] ce2bf44bda6688e79d4272b26933c966 Plugin.get_plugin_metadata (init.py:111) [_init_handlers] authentication handler: SpaceONEAuthenticationHandler
2024-11-07T14:09:34.556006811Z 2024-11-07T23:09:34.551Z [DEBUG] ce2bf44bda6688e79d4272b26933c966 Plugin.get_plugin_metadata (init.py:111) [_init_handlers] authorization handler: SpaceONEAuthorizationHandler
2024-11-07T14:09:34.556048611Z 2024-11-07T23:09:34.552Z [DEBUG] ce2bf44bda6688e79d4272b26933c966 Plugin.get_plugin_metadata (init.py:111) [_init_handlers] mutation handler: SpaceONEMutationHandler
2024-11-07T14:09:34.556057411Z 2024-11-07T23:09:34.554Z [DEBUG] ce2bf44bda6688e79d4272b26933c966 Plugin.get_plugin_metadata (authentication_handler.py:48) [_get_public_key] get jwk from identity service: domain-bf42bda26a53
2024-11-07T14:09:34.578648161Z 2024-11-07T23:09:34.565Z [INFO] ce2bf44bda6688e79d4272b26933c966 domain-bf42bda26a53 [email protected] WORKSPACE_OWNER workspace-0b5eb3fa9d1c Plugin.get_plugin_metadata (init.py:204) (REQUEST) => {"plugin_id": "plugin-aws-cloud-service-inven-collector", "version": "1.15.31", "options": {}}
2024-11-07T14:09:34.932188066Z 2024-11-07T23:09:34.921Z [DEBUG] ce2bf44bda6688e79d4272b26933c966 domain-bf42bda26a53 [email protected] WORKSPACE_OWNER workspace-0b5eb3fa9d1c Plugin.get_plugin_metadata (init.py:118) [get_matched_supervisors] possible list before label matching: []
2024-11-07T14:09:34.932248167Z 2024-11-07T23:09:34.921Z [DEBUG] ce2bf44bda6688e79d4272b26933c966 domain-bf42bda26a53 [email protected] WORKSPACE_OWNER workspace-0b5eb3fa9d1c Plugin.get_plugin_metadata (plugin_service.py:118) [get_plugin_endpoint] create new plugin
2024-11-07T14:09:34.932256167Z 2024-11-07T23:09:34.924Z [ERROR] ce2bf44bda6688e79d4272b26933c966 domain-bf42bda26a53 [email protected] WORKSPACE_OWNER workspace-0b5eb3fa9d1c Plugin.get_plugin_metadata (init.py:277) (Error) => There is no supervisor to run plugin. params: {'plugin_id': 'plugin-aws-cloud-service-inven-collector', 'version': '1.15.31', 'options': {}, 'domain_id': 'domain-bf42bda26a53', 'workspace_id': 'workspace-0b5eb3fa9d1c'}
2024-11-07T14:09:34.932262967Z error_code = ERROR_NO_POSSIBLE_SUPERVISOR
2024-11-07T14:09:34.932282767Z status_code = INTERNAL
2024-11-07T14:09:34.932286867Z message = There is no supervisor to run plugin. params: {'plugin_id': 'plugin-aws-cloud-service-inven-collector', 'version': '1.15.31', 'options': {}, 'domain_id': 'domain-bf42bda26a53', 'workspace_id': 'workspace-0b5eb3fa9d1c'}
2024-11-07T14:09:34.932291467Z Traceback (most recent call last):
2024-11-07T14:09:34.932295467Z File "/usr/local/lib/python3.8/site-packages/spaceone/core/service/init.py", line 222, in _pipeline
2024-11-07T14:09:34.932299567Z response_or_iterator = func(self, params)
2024-11-07T14:09:34.932303267Z File "/usr/local/lib/python3.8/site-packages/spaceone/core/service/utils.py", line 127, in wrapped_func
2024-11-07T14:09:34.932307167Z return func(cls, params)
2024-11-07T14:09:34.932310967Z File "/usr/local/lib/python3.8/site-packages/spaceone_plugin-2.0.dev16-py3.8.egg/spaceone/plugin/service/plugin_service.py", line 81, in get_plugin_metadata
2024-11-07T14:09:34.932325267Z plugin_endpoint_info = self._get_plugin_endpoint(params, token)
2024-11-07T14:09:34.932329068Z File "/usr/local/lib/python3.8/site-packages/spaceone_plugin-2.0.dev16-py3.8.egg/spaceone/plugin/service/plugin_service.py", line 128, in _get_plugin_endpoint
2024-11-07T14:09:34.932333468Z raise ERROR_NO_POSSIBLE_SUPERVISOR(params=params)
2024-11-07T14:09:34.932337268Z spaceone.plugin.error.supervisor.ERROR_NO_POSSIBLE_SUPERVISOR:
2024-11-07T14:09:34.932342268Z error_code = ERROR_NO_POSSIBLE_SUPERVISOR
2024-11-07T14:09:34.932345968Z status_code = INTERNAL
2024-11-07T14:09:34.932349768Z message = There is no supervisor to run plugin. params: {'plugin_id': 'plugin-aws-cloud-service-inven-collector', 'version': '1.15.31', 'options': {}, 'domain_id': 'domain-bf42bda26a53', 'workspace_id': 'workspace-0b5eb3fa9d1c'}

running pods:
NAME READY STATUS RESTARTS AGE
board-54c45867f8-tss5f 1/1 Running 3 (49m ago) 16h
config-679c846678-phhs8 1/1 Running 3 (49m ago) 16h
console-6789f5b796-rkkqv 1/1 Running 3 (49m ago) 16h
console-api-67d7d8b7dd-fmsk6 2/2 Running 6 (49m ago) 16h
console-api-v2-rest-569c8d6fdb-rsmvf 2/2 Running 6 (49m ago) 16h
cost-analysis-5bc6ddf9d4-z567s 1/1 Running 3 (49m ago) 16h
cost-analysis-scheduler-598655675b-8jnjb 0/1 CrashLoopBackOff 174 (83s ago) 16h
cost-analysis-worker-79595854f8-mnd72 1/1 Running 3 (49m ago) 16h
dashboard-5b847fcd4b-p8plz 1/1 Running 3 (49m ago) 16h
file-manager-5d8d8d9b74-7wnfm 1/1 Running 3 (49m ago) 16h
identity-6876959c57-bh7mg 1/1 Running 3 (49m ago) 16h
identity-scheduler-769fdb5b56-9pmg7 1/1 Running 3 (49m ago) 16h
identity-worker-7555d46668-k7mwd 1/1 Running 3 (49m ago) 16h
inventory-7c6989dd79-wm8f8 1/1 Running 3 (49m ago) 16h
inventory-scheduler-76b54f7c77-q7hn2 1/1 Running 3 (49m ago) 16h
inventory-worker-76c997b8dd-bsl6q 1/1 Running 3 (49m ago) 16h
mongodb-7bd5c9b6bf-jvvj7 1/1 Running 3 (49m ago) 17h
monitoring-57fb9cdfcf-ghwlr 1/1 Running 3 (49m ago) 16h
monitoring-rest-84c6cbdff9-44dx8 2/2 Running 6 (49m ago) 16h
monitoring-scheduler-76cfc9468b-h7lpl 1/1 Running 3 (49m ago) 16h
monitoring-worker-6d65f55f6b-bgsbk 1/1 Running 3 (49m ago) 16h
notification-84fff9c5c9-vwx26 1/1 Running 3 (49m ago) 16h
notification-scheduler-66c4d6664f-6wcmj 1/1 Running 3 (49m ago) 16h
notification-worker-f5bc4fccd-qks9q 1/1 Running 3 (49m ago) 16h
plugin-546f9c5b8-qhd6c 1/1 Running 0 49s
plugin-scheduler-dcf64f86c-q989f 1/1 Running 3 (49m ago) 16h
plugin-worker-759fb46d5d-wjfbd 1/1 Running 3 (49m ago) 16h
redis-6c4b4d7fdc-p5s2d 1/1 Running 3 (49m ago) 16h
repository-575cfbd7b6-cdrqf 1/1 Running 3 (49m ago) 16h
search-5445b9d5b8-xqfs7 1/1 Running 3 (49m ago) 16h
secret-9cf57cdff-p9kxn 1/1 Running 3 (49m ago) 16h
spacectl-5cc889bf97-l2pjv 1/1 Running 3 (49m ago) 16h
statistics-789b6d9dcf-qtrtv 1/1 Running 3 (49m ago) 16h
statistics-scheduler-5b7566b5fc-qv8jb 1/1 Running 3 (49m ago) 16h
statistics-worker-776c7c4cfd-z47lj 1/1 Running 3 (49m ago) 16h

Thanks
Petr

[petrk-afk]

komik@cloudforet-poc-vm:~$ kubectl describe pod plugin-546f9c5b8-qhd6c -n cloudforet
Name: plugin-546f9c5b8-qhd6c
Namespace: cloudforet
Priority: 0
Service Account: default
Node: minikube/192.168.49.2
Start Time: Thu, 07 Nov 2024 14:07:30 +0000
Labels: app.kubernetes.io/instance=cloudforet
app.kubernetes.io/managed-by=Helm
app.kubernetes.io/name=plugin
app.kubernetes.io/version=1.x.y
helm.sh/chart=plugin-1.3.11
pod-template-hash=546f9c5b8
spaceone.service=plugin
Annotations: spaceone.application-flag: b77c67a6516af62ad8ad1c16d5f42473db37ee85ddd53ca44c46a28b1fdefb70
spaceone.database-flag: 52133ddfcd583280388a8bbfb67a417b6b660f75bc63ec1b01beec811d4498ef
spaceone.deployment.tool: helm
spaceone.shared-flag: 52133ddfcd583280388a8bbfb67a417b6b660f75bc63ec1b01beec811d4498ef
Status: Running
IP: 10.244.1.26
IPs:
IP: 10.244.1.26
Controlled By: ReplicaSet/plugin-546f9c5b8
Containers:
plugin:
Container ID: docker://95e098d360af2ecdc726f0c0c4d689901caa4e238b47161781c08cf23fbaa658
Image: cloudforet/plugin:2.0.dev16
Image ID: docker-pullable://cloudforet/plugin@sha256:491dab96bd1916ad263aca0cf5a58fde3031a02bb0a4aa230b062e9aebfc98e9
Port: 50051/TCP
Host Port: 0/TCP
State: Running
Started: Thu, 07 Nov 2024 14:07:32 +0000
Ready: True
Restart Count: 0
Environment:
SPACEONE_CONFIG_FILE: /opt/spaceone/plugin/config/config.yaml
Mounts:
/etc/localtime from timezone (rw)
/opt/spaceone/plugin/config/application.yaml from application-conf (ro,path="application.yaml")
/opt/spaceone/plugin/config/config.yaml from default-conf (ro,path="config.yaml")
/opt/spaceone/plugin/config/database.yaml from database-conf (ro,path="database.yaml")
/opt/spaceone/plugin/config/shared.yaml from shared-conf (ro,path="shared.yaml")
/var/log/spaceone from log-volume (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-5srh4 (ro)
Conditions:
Type Status
PodReadyToStartContainers True
Initialized True
Ready True
ContainersReady True
PodScheduled True
Volumes:
default-conf:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: plugin-conf
Optional: false
database-conf:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: plugin-database-conf
Optional: false
shared-conf:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: shared-conf
Optional: false
application-conf:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: plugin-application-grpc-conf
Optional: false
timezone:
Type: HostPath (bare host directory volume)
Path: /usr/share/zoneinfo/Asia/Seoul
HostPathType:
log-volume:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
SizeLimit:
kube-api-access-5srh4:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional:
DownwardAPI: true
QoS Class: BestEffort
Node-Selectors:
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message

---

Normal Scheduled 8m46s default-scheduler Successfully assigned cloudforet/plugin-546f9c5b8-qhd6c to minikube
Normal Pulled 8m45s kubelet Container image "cloudforet/plugin:2.0.dev16" already present on machine
Normal Created 8m45s kubelet Created container plugin
Normal Started 8m45s kubelet Started container plugin
komik@cloudforet-poc-vm:~$

[ImMin5]

HI @petrk-afk

Sorry for the delayed response.

To run plugins, Cloudforet requires a supervisor to be enabled. From the logs you provided, it appears that the supervisor may not be running.

Could you check your Helm values to ensure the supervisor is enabled? There should be a field to toggle the supervisor.

After verifying, please try running the following command.

helm upgrade cloudforet spaceone -n cloudforet -f {your helm value file}

[petrk-afk]

I have checked yaml file used to initiate the installation and it contains the values as required.

[petrk-afk]

kubectl events returns

cloudforet-plugin 7d2h (x176 over 7d7h) Warning FailedMount Pod/supervisor-scheduler-64f94f88b5-2vxd9 MountVolume.SetUp failed for volume "shared-conf" : configmap "shared-conf" not found
kube-system 52m Warning BackOff Pod/storage-provisioner Back-off restarting failed container storage-provisioner in pod storage-provisioner_kube-system(e201cff5-f1c5-4c21-a6bc-6062a9682936)