Shell-operator fails on handling something

[gunmaden]

Expected behavior (what you expected to happen): Periodically error occurred while shell-operator trying to handle something

Actual behavior (what actually happened): Error does not occur, have enough info about problem

Steps to reproduce:

1. Deploy shell-operator & crd

Additional info:
Tried to enable debug log level but it doesn't help (only logs which entities are processing, each time error message meets in different places)

Environment:

• Shell-operator version: v1.0.10
• Kubernetes version: v1.22.12
• Installation type (kubectl apply, helm chart, etc.): helm chart

Anything else we should know?:

Hook script

    #!/usr/bin/env bash
set -e

echoerr() {
  printf "ERROR. %s\n" "$*" >&2;
}

##### HOOK #####

if [[ $1 == "--config" ]] ; then
    cat <<EOF
configVersion: v1
onStartup: 1
kubernetes:
  - name: OnDiscoveryEvent
    apiVersion: custom.group/v1
    kind: Discovery
    executeHookOnEvent:
      - Added
      - Deleted
  - name: OnServiceEvent
    apiVersion: ""
    kind: Service
    executeHookOnEvent:
      - Added
      - Modified
      - Deleted
EOF
else
  for context in $(jq -c '.[]' "$BINDING_CONTEXT_PATH"); do
    echo "Just log"
  done
fi

Custom resource definition

apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  # name must match the spec fields below, and be in the form: .
  name: discoveries.custom.group
spec:
  # group name to use for REST API: /apis//
  group: custom.group
  # either Namespaced or Cluster
  scope: Namespaced
  names:
    # plural name to be used in the URL: /apis///
    plural: discoveries
    # singular name to be used as an alias on the CLI and for display
    singular: discovery
    # kind is normally the CamelCased singular type. Your resource manifests use this.
    kind: Discovery
  # list of versions supported by this CustomResourceDefinition
  versions:
    - name: v1
      # Each version can be enabled/disabled by Served flag.
      served: true
      # One and only one version must be marked as the storage version.
      storage: true
      # The schema for validating custom objects
      schema:
        openAPIV3Schema:
          type: object

Logs

{"level":"info","msg":"shell-operator v1.0.10","time":"2022-10-05T12:14:24Z"}
{"level":"info","msg":"Listen on 0.0.0.0:9115","time":"2022-10-05T12:14:24Z"}
{"level":"info","msg":"Create metric counter discovery_operator_live_ticks","operator.component":"metricStorage","time":"2022-10-05T12:14:24Z"}
{"level":"info","msg":"Create metric histogram discovery_operator_tasks_queue_action_duration_seconds","operator.component":"metricsStorage","time":"2022-10-05T12:14:24Z"}
{"level":"info","msg":"Create metric gauge discovery_operator_tasks_queue_length","operator.component":"metricStorage","time":"2022-10-05T12:14:24Z"}
{"level":"info","msg":"Create metric gauge discovery_operator_kube_snapshot_objects","operator.component":"metricStorage","time":"2022-10-05T12:14:24Z"}
{"level":"info","msg":"Create metric gauge discovery_operator_kube_snapshot_bytes","operator.component":"metricStorage","time":"2022-10-05T12:14:24Z"}
{"level":"info","msg":"Create metric histogram discovery_operator_kube_jq_filter_duration_seconds","operator.component":"metricsStorage","time":"2022-10-05T12:14:24Z"}
{"level":"info","msg":"Create metric histogram discovery_operator_kube_event_duration_seconds","operator.component":"metricsStorage","time":"2022-10-05T12:14:24Z"}
{"level":"info","msg":"Create metric gauge discovery_operator_hook_enable_kubernetes_bindings_seconds","operator.component":"metricStorage","time":"2022-10-05T12:14:24Z"}
{"level":"info","msg":"Create metric counter discovery_operator_hook_enable_kubernetes_bindings_errors_total","operator.component":"metricStorage","time":"2022-10-05T12:14:24Z"}
{"level":"info","msg":"Create metric gauge discovery_operator_hook_enable_kubernetes_bindings_success","operator.component":"metricStorage","time":"2022-10-05T12:14:24Z"}
{"level":"info","msg":"Create metric histogram discovery_operator_hook_run_seconds","operator.component":"metricsStorage","time":"2022-10-05T12:14:24Z"}
{"level":"info","msg":"Create metric histogram discovery_operator_hook_run_user_cpu_seconds","operator.component":"metricsStorage","time":"2022-10-05T12:14:24Z"}
{"level":"info","msg":"Create metric histogram discovery_operator_hook_run_sys_cpu_seconds","operator.component":"metricsStorage","time":"2022-10-05T12:14:24Z"}
{"level":"info","msg":"Create metric gauge discovery_operator_hook_run_max_rss_bytes","operator.component":"metricStorage","time":"2022-10-05T12:14:24Z"}
{"level":"info","msg":"Create metric counter discovery_operator_hook_run_errors_total","operator.component":"metricStorage","time":"2022-10-05T12:14:24Z"}
{"level":"info","msg":"Create metric counter discovery_operator_hook_run_allowed_errors_total","operator.component":"metricStorage","time":"2022-10-05T12:14:24Z"}
{"level":"info","msg":"Create metric counter discovery_operator_hook_run_success_total","operator.component":"metricStorage","time":"2022-10-05T12:14:24Z"}
{"level":"info","msg":"Create metric counter discovery_operator_task_wait_in_queue_seconds_total","operator.component":"metricStorage","time":"2022-10-05T12:14:24Z"}
{"level":"info","msg":"Hooks dir: /hooks","time":"2022-10-05T12:14:24Z"}
{"level":"info","msg":"Use temporary dir: /tmp/shell-operator","time":"2022-10-05T12:14:24Z"}
{"level":"info","msg":"Debug endpoint listen on /var/run/shell-operator/debug.socket","time":"2022-10-05T12:14:24Z"}
{"level":"info","msg":"Create metric histogram discovery_operator_kubernetes_client_request_latency_seconds","operator.component":"metricsStorage","time":"2022-10-05T12:14:24Z"}
{"level":"info","msg":"Create metric counter discovery_operator_kubernetes_client_request_result_total","operator.component":"metricStorage","time":"2022-10-05T12:14:24Z"}
{"level":"info","msg":"Kubernetes client is configured successfully with 'out-of-cluster' config","operator.component":"KubernetesAPIClient","time":"2022-10-05T12:14:24Z"}
{"level":"info","msg":"Kubernetes client is configured successfully with 'out-of-cluster' config","operator.component":"KubernetesAPIClient","time":"2022-10-05T12:14:24Z"}
{"level":"info","msg":"Initialize hooks manager. Search for and load all hooks.","time":"2022-10-05T12:14:24Z"}
{"hook":"hook.sh","level":"info","msg":"Load config from '/hooks/hook.sh'","phase":"config","time":"2022-10-05T12:14:24Z"}
{"hook":"hook.sh","level":"info","msg":"Loaded config: OnStartup:1, Watch k8s kinds: 'Service', 'Discovery'","phase":"config","time":"2022-10-05T12:14:24Z"}
{"level":"info","msg":"start shell-operator","time":"2022-10-05T12:14:24Z"}
{"level":"info","msg":"queue task HookRun::onStartup:hook.sh with hook hook.sh","operator.component":"initMainQueue","time":"2022-10-05T12:14:24Z"}
{"level":"info","msg":"queue task EnableKubernetesBindings:::hook.sh:EnableKubernetesBindings for hook hook.sh","operator.component":"initMainQueue","time":"2022-10-05T12:14:24Z"}
{"binding":"","event":"onStartup","hook":"hook.sh","level":"info","msg":"Execute hook","queue":"","task":"HookRun","time":"2022-10-05T12:14:24Z"}
{"binding":"","event":"onStartup","hook":"hook.sh","level":"info","msg":"Conext","output":"stdout","queue":"","task":"HookRun","time":"2022-10-05T12:14:24Z"}
{"level":"info","msg":"Create metric histogram discovery_operator_hook_run_sys_seconds","operator.component":"metricsStorage","time":"2022-10-05T12:14:24Z"}
{"level":"info","msg":"Create metric histogram discovery_operator_hook_run_user_seconds","operator.component":"metricsStorage","time":"2022-10-05T12:14:24Z"}
{"binding":"","event":"onStartup","hook":"hook.sh","level":"info","msg":"Hook executed successfully","queue":"","task":"HookRun","time":"2022-10-05T12:14:24Z"}
{"binding":"","hook":"hook.sh","level":"info","msg":"Enable kubernetes binding for hook","queue":"main","task":"EnableKubernetesBindings","time":"2022-10-05T12:14:24Z"}
{"level":"info","msg":"Create metric histogram discovery_operator_kubernetes_client_rate_limiter_latency_seconds","operator.component":"metricsStorage","time":"2022-10-05T12:14:24Z"}
{"level":"error","msg":"E1005 12:14:24.777262      10 reflector.go:127] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:156: Failed to watch *unstructured.Unstructured: unknown\n","source":"klog","time":"2022-10-05T12:14:24Z"}
{"level":"info","msg":"I1005 12:14:25.959173      10 request.go:645] Throttling request took 1.076579751s, request: GET:https://10.96.0.1:443/apis/flowcontrol.apiserver.k8s.io/v1beta1?timeout=32s\n","source":"klog","time":"2022-10-05T12:14:25Z"}
{"level":"error","msg":"E1005 12:14:26.034804      10 reflector.go:127] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:156: Failed to watch *unstructured.Unstructured: unknown\n","source":"klog","time":"2022-10-05T12:14:26Z"}
{"level":"error","msg":"E1005 12:14:28.694095      10 reflector.go:127] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:156: Failed to watch *unstructured.Unstructured: unknown\n","source":"klog","time":"2022-10-05T12:14:28Z"}
{"level":"error","msg":"E1005 12:14:29.570104      10 reflector.go:127] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:156: Failed to watch *unstructured.Unstructured: unknown\n","source":"klog","time":"2022-10-05T12:14:29Z"}
{"binding":"","hook":"hook.sh","level":"info","msg":"Kubernetes bindings for hook are enabled successfully, 2 tasks generated","queue":"main","task":"EnableKubernetesBindings","time":"2022-10-05T12:14:29Z"}
{"binding":"OnDiscoveryEvent","event":"kubernetes","hook":"hook.sh","level":"info","msg":"Execute hook","queue":"main","task":"HookRun","time":"2022-10-05T12:14:29Z"}
{"binding":"OnDiscoveryEvent","event":"kubernetes","hook":"hook.sh","level":"info","msg":"Conext","output":"stdout","queue":"main","task":"HookRun","time":"2022-10-05T12:14:29Z"}
{"binding":"OnDiscoveryEvent","event":"kubernetes","hook":"hook.sh","level":"info","msg":"Hook executed successfully","queue":"main","task":"HookRun","time":"2022-10-05T12:14:29Z"}
{"binding":"OnDiscoveryEvent","event":"kubernetes","hook":"hook.sh","level":"info","msg":"Unlock kubernetes.Event tasks","queue":"main","task":"HookRun","time":"2022-10-05T12:14:29Z"}
{"binding":"OnServiceEvent","event":"kubernetes","hook":"hook.sh","level":"info","msg":"Execute hook","queue":"main","task":"HookRun","time":"2022-10-05T12:14:29Z"}
{"binding":"OnServiceEvent","event":"kubernetes","hook":"hook.sh","level":"info","msg":"Conext","output":"stdout","queue":"main","task":"HookRun","time":"2022-10-05T12:14:29Z"}
{"binding":"OnServiceEvent","event":"kubernetes","hook":"hook.sh","level":"info","msg":"Hook executed successfully","queue":"main","task":"HookRun","time":"2022-10-05T12:14:29Z"}
{"binding":"OnServiceEvent","event":"kubernetes","hook":"hook.sh","level":"info","msg":"Unlock kubernetes.Event tasks","queue":"main","task":"HookRun","time":"2022-10-05T12:14:29Z"}
{"level":"error","msg":"E1005 12:14:30.629997      10 reflector.go:127] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:156: Failed to watch *unstructured.Unstructured: unknown\n","source":"klog","time":"2022-10-05T12:14:30Z"}
{"level":"error","msg":"E1005 12:14:32.461660      10 reflector.go:127] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:156: Failed to watch *unstructured.Unstructured: unknown\n","source":"klog","time":"2022-10-05T12:14:32Z"}
{"level":"error","msg":"E1005 12:14:33.203631      10 reflector.go:127] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:156: Failed to watch *unstructured.Unstructured: unknown\n","source":"klog","time":"2022-10-05T12:14:33Z"}
{"level":"error","msg":"E1005 12:14:36.658349      10 reflector.go:127] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:156: Failed to watch *unstructured.Unstructured: unknown\n","source":"klog","time":"2022-10-05T12:14:36Z"}
{"level":"error","msg":"E1005 12:14:40.836229      10 reflector.go:127] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:156: Failed to watch *unstructured.Unstructured: unknown\n","source":"klog","time":"2022-10-05T12:14:40Z"}
{"level":"error","msg":"E1005 12:14:45.270768      10 reflector.go:127] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:156: Failed to watch *unstructured.Unstructured: unknown\n","source":"klog","time":"2022-10-05T12:14:45Z"}
{"level":"error","msg":"E1005 12:15:04.555854      10 reflector.go:127] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:156: Failed to watch *unstructured.Unstructured: unknown\n","source":"klog","time":"2022-10-05T12:15:04Z"}
{"level":"error","msg":"E1005 12:15:09.467184      10 reflector.go:127] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:156: Failed to watch *unstructured.Unstructured: unknown\n","source":"klog","time":"2022-10-05T12:15:09Z"}
{"level":"error","msg":"E1005 12:15:36.220781      10 reflector.go:127] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:156: Failed to watch *unstructured.Unstructured: unknown\n","source":"klog","time":"2022-10-05T12:15:36Z"}
{"level":"error","msg":"E1005 12:15:37.375321      10 reflector.go:127] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:156: Failed to watch *unstructured.Unstructured: unknown\n","source":"klog","time":"2022-10-05T12:15:37Z"}
{"level":"error","msg":"E1005 12:16:17.142853      10 reflector.go:127] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:156: Failed to watch *unstructured.Unstructured: unknown\n","source":"klog","time":"2022-10-05T12:16:17Z"}
{"level":"error","msg":"E1005 12:16:25.203734      10 reflector.go:127] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:156: Failed to watch *unstructured.Unstructured: unknown\n","source":"klog","time":"2022-10-05T12:16:25Z"}
{"level":"error","msg":"E1005 12:17:06.718118      10 reflector.go:127] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:156: Failed to watch *unstructured.Unstructured: unknown\n","source":"klog","time":"2022-10-05T12:17:06Z"}
{"level":"error","msg":"E1005 12:17:11.263395      10 reflector.go:127] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:156: Failed to watch *unstructured.Unstructured: unknown\n","source":"klog","time":"2022-10-05T12:17:11Z"}
{"level":"error","msg":"E1005 12:17:46.645551      10 reflector.go:127] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:156: Failed to watch *unstructured.Unstructured: unknown\n","source":"klog","time":"2022-10-05T12:17:46Z"}
{"level":"error","msg":"E1005 12:17:59.075634      10 reflector.go:127] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:156: Failed to watch *unstructured.Unstructured: unknown\n","source":"klog","time":"2022-10-05T12:17:59Z"}
{"level":"error","msg":"E1005 12:18:35.905553      10 reflector.go:127] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:156: Failed to watch *unstructured.Unstructured: unknown\n","source":"klog","time":"2022-10-05T12:18:35Z"}
{"level":"error","msg":"E1005 12:18:53.018460      10 reflector.go:127] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:156: Failed to watch *unstructured.Unstructured: unknown\n","source":"klog","time":"2022-10-05T12:18:53Z"}
{"level":"error","msg":"E1005 12:19:25.314806      10 reflector.go:127] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:156: Failed to watch *unstructured.Unstructured: unknown\n","source":"klog","time":"2022-10-05T12:19:25Z"}

[diafour]

Hello! Sorry, I've missed this one for so long. Do you have proper RBAC with rules for the Discovery kind?

[gunmaden]

I think root cause is that I've forgot to add watch rules for resources. Thanks