upgrade to 4.9.0 results in plugin capability used before init: k8s meta

[jemag]

Describe the bug
Upgrading the chart from 4.8.3 to 4.9.0 without any change to our values.yaml, there appear to now be a schema validation error:

Thu Oct  3 16:27:33 2024: Falco initialized with configuration files:
Thu Oct  3 16:27:33 2024:    /etc/falco/falco.yaml | schema validation: failed for <root>[plugins][0][init_config]: Value type not permitted by 'type' constraint.

Here is our plugins section from the falco configmap

    plugins:
    - init_config: null
      library_path: libk8saudit.so
      name: k8saudit
      open_params: http://:9765/k8s-audit
    - library_path: libcloudtrail.so
      name: cloudtrail
    - init_config: ""
      library_path: libjson.so
      name: json
    - init_config:
        collectorHostname: falco-k8s-metacollector.falco.svc
        collectorPort: 45000
        nodeName: ${FALCO_K8S_NODE_NAME}
      library_path: libk8smeta.so
      name: k8smeta

This eventually result (not 100% sure if related) in the pod terminating with the following error:
Thu Oct 3 16:38:28 2024: Runtime error: plugin capability used before init: k8smeta. Exiting.

How to reproduce it

Deploy helm chart 4.9.0 with following values:

podAnnotations:                                                                                                                                                                                                                
  container.apparmor.security.beta.kubernetes.io/falco: unconfined                                                                                                                                                             
serviceMonitor:                                                                                                                                                                                                                
  create: true                                                                                                                                                                                                                 
  labels:                                                                                                                                                                                                                      
    app: kube-prometheus-stack                                                                                                                                                                                                 
    release: kube-prometheus-stack                                                                                                                                                                                      
extra:                                                                                                                                                                                                                         
  args:                                                                                                                                                                                                                        
    - --disable-cri-async                                                                                                                                                                                                      
resources:                                                                                                                                                                                                                     
  requests:                                                                                                                                                                                                                    
    cpu: 100m                                                                                                                                                                                                                  
    memory: 300Mi                                                                                                                                                                                                              
  limits:                                                                                                                                                                                                                      
    cpu: 999                                                                                                                                                                                                                   
    memory: 500Mi                                                                                                                                                                                                              
tty: false                                                                                                                                                                                                                     
controller:                                                                                                                                                                                                                    
  kind: daemonset                                                                                                                                                                                                              
driver:                                                                                                                                                                                                                        
  enabled: true                                                                                                                                                                                                                
  kind: modern_ebpf                                                                                                                                                                                                            
  modernEbpf:                                                                                                                                                                                                                  
    leastPrivileged: true                                                                                                                                                                                                      
collectors:                                                                                                                                                                                                                    
  enabled: true                                                                                                                                                                                                                
  kubernetes:                                                                                                                                                                                                                  
    enabled: true                                                                                                                                                                                                              
metrics:                                                                                                                                                                                                                       
  enabled: true                                                                                                                                                                                                                
falco:                                                                                                                                                                                                                         
  grpc:                                                                                                                                                                                                                        
    enabled: true                                                                                                                                                                                                              
  grpc_output:                                                                                                                                                                                                                 
    enabled: true                                                                                                                                                                                                              
  rules_files:                                                                                                                                                                                                                 
    - /etc/falco/falco_rules.yaml                                                                                                                                                                                              
    - /etc/falco/falco-incubating_rules.yaml                                                                                                                                                                                   
    - /etc/falco/rules.d                                                                                                                                                                                                       
  webserver:                                                                                                                                                                                                                   
    prometheus_metrics_enabled: true                                                                                                                                                                                           
falcoctl:                                                                                                                                                                                   
  artifact:                                                                                                                                                                                                                    
    install:                                                                                                                                                                                                                   
      resources:                                                                                                                                                                                                               
        requests:                                                                                                                                                                                                              
          cpu: 10m                                                                                                                                                                                                             
          memory: 50Mi                                                                                                                                                                                                         
        limits:                                                                                                                                                                                                                
          memory: 100Mi                                                                                                                                                                                                        
      # -- Enable the init container. We do not recommend installing (or following) plugins for security reasons since they are executable objects.                                                                            
      enabled: true                                                                                                                                                                                                            
    follow:                                                                                                                                                                                                                    
      resources:                                                                                                                                                                                                               
        requests:                                                                                                                                                                                                              
          cpu: 10m                                                                                                                                                                                                             
          memory: 50Mi                                                                                                                                                                                                         
        limits:                                                                                                                                                                                                                
          memory: 100Mi                                                                                                                                                                                                        
      # -- Enable the sidecar container. We do not support it yet for plugins. It is used only for rules feed such as k8saudit-rules rules.                                                                                    
      enabled: true                                                                                                                                                                                                            
  config:                                                                                                                                                                                                                      
    artifact:                                                                                                                                                                                                                  
      install:                                                                                                                                                                                                                 
        # -- List of artifacts to be installed by the falcoctl init container.                                                                                                                                                 
        refs: [falco-rules:3, falco-incubating-rules:3]                                                                                                                                                                        
      follow:                                                                                                                                                                                                                  
        # -- List of artifacts to be installed by the falcoctl init container.                                                                                                                                                 
        refs: [falco-rules:3, falco-incubating-rules:3]                                                                                                                                                                        
                                                                                                                                                               
k8s-metacollector:                                                                                                                                                                                
  resources:                                                                                                                                                                                                                   
    requests:                                                                                                                                                                                                                  
      cpu: 10m                                                                                                                                                                                                                 
      memory: 50Mi                                                                                                                                                                                                             
    limits:                                                                                                                                                                                                                    
      memory: 100Mi                                                                                                                                                                                                            
  containerSecurityContext:                                                                                                                                                                                                    
    allowPrivilegeEscalation: false                                                                                                                                                                                            
    readOnlyRootFilesystem: true                                                                                                                                                                                               
    seccompProfile:                                                                                                                                                                                                            
      type: RuntimeDefault                                                                                                                                                                                                     
  serviceMonitor:                                                                                                                                                                                                              
    create: true                                                                                                                                                                                                               
    # -- path at which the metrics are expose by the k8s-metacollector.                                                                                                                                                        
    path: /metrics                                                                                                                                                                                                             
    # -- labels set of labels to be applied to the ServiceMonitor resource.                                                                                                                                                    
    labels:                                                                                                                                                                                                                    
      app: kube-prometheus-stack                                                                                                                                                                                               
      release: kube-prometheus-stack                                                                                                                                                                                           
  grafana:                                                                                                                                                                                                                     
    dashboards:                                                                                                                                                                                                                
      enabled: true                                                                                                                                                                                                            

Expected behaviour

Falco runs normally just as in 4.8.3

Screenshots

Environment

• Falco version:
  cannot run command with crashlooping container, image falco-no-driver:0.39.0
• System info:
  cannot run command with crashlooping container, hope this is sufficient;:
  Linux version 5.15.164.1-1.cm2 (root@CBL-Mariner) (gcc (GCC) 11.2.0, GNU ld (GNU Binutils) 2.37) #1 SMP

• Cloud provider or hardware configuration:
  Azure AKS

• OS:
  NAME="Common Base Linux Mariner"
  VERSION="2.0.20240829"
  ID=mariner
  VERSION_ID="2.0"
  PRETTY_NAME="CBL-Mariner/Linux"
  ANSI_COLOR="1;34"
  HOME_URL="https://aka.ms/cbl-mariner"
  BUG_REPORT_URL="https://aka.ms/cbl-mariner"
  SUPPORT_URL="https://aka.ms/cbl-mariner"

• Kernel:
  Linux aks-main-14571596-vmss000008 5.15.164.1-1.cm2 Pull falco chart from helm/charts #1 SMP Sun Aug 18 19:16:21 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

• Installation method:
  -helm chart 4.9.0

Additional context

Full logs before crash

Thu Oct  3 16:38:28 2024: The --cri option is deprecated and will be removed in Falco 0.40.0. Use -o container_engines.cri.sockets[]=<socket_path> instead.
Thu Oct  3 16:38:28 2024: The --disable-cri-async option is deprecated and will be removed in Falco 0.40.0. Use -o container_engines.cri.disable_async=true instead.
Thu Oct  3 16:38:28 2024: Falco version: 0.39.0 (x86_64)
Thu Oct  3 16:38:28 2024: Falco initialized with configuration files:
Thu Oct  3 16:38:28 2024:    /etc/falco/falco.yaml | schema validation: failed for <root>[plugins][0][init_config]: Value type not permitted by 'type' constraint.
Thu Oct  3 16:38:28 2024: System info: Linux version 5.15.164.1-1.cm2 (root@CBL-Mariner) (gcc (GCC) 11.2.0, GNU ld (GNU Binutils) 2.37) #1 SMP Sun Aug 18 19:16:21 UTC 2024
Thu Oct  3 16:38:28 2024: Loading plugin 'k8smeta' from file /usr/share/falco/plugins/libk8smeta.so
Thu Oct 3 16:38:28 2024: [warning] [k8smeta] [EXPERIMENTAL] This plugin is in active development and may undergo changes in behavior without prioritizing backward compatibility.
Thu Oct 3 16:38:28 2024: [info] [k8smeta] Start the process scan under: '/host/proc'
Thu Oct 3 16:38:28 2024: [info] [k8smeta] Process scan correctly completed. Found '125' threads inside pods.
Thu Oct  3 16:38:28 2024: Loading rules from:
Thu Oct  3 16:38:28 2024:    /etc/falco/falco_rules.yaml | schema validation: ok
Thu Oct  3 16:38:28 2024:    /etc/falco/falco-incubating_rules.yaml | schema validation: ok
Thu Oct  3 16:38:28 2024:    /etc/falco/rules.d/custom.local.yaml | schema validation: ok
Thu Oct  3 16:38:28 2024: /etc/falco/rules.d/custom.local.yaml: Ok, with warnings
5 Warnings:
In rules content: (/etc/falco/falco-incubating_rules.yaml:0:0)
    rule 'System procs network activity': (/etc/falco/falco-incubating_rules.yaml:663:2)
------
- rule: System procs network activity
  ^
------
LOAD_NO_EVTTYPE (Condition has no event-type restriction): Rule matches too many evt.type values. This has a significant performance penalty.
In rules content: (/etc/falco/falco-incubating_rules.yaml:0:0)
    rule 'Unexpected UDP Traffic': (/etc/falco/falco-incubating_rules.yaml:746:2)
------
- rule: Unexpected UDP Traffic
  ^
------
LOAD_NO_EVTTYPE (Condition has no event-type restriction): Rule matches too many evt.type values. This has a significant performance penalty.
In rules content: (/etc/falco/falco-incubating_rules.yaml:0:0)
    rule 'Network Connection outside Local Subnet': (/etc/falco/falco-incubating_rules.yaml:1108:2)
------
- rule: Network Connection outside Local Subnet
  ^
------
LOAD_NO_EVTTYPE (Condition has no event-type restriction): Rule matches too many evt.type values. This has a significant performance penalty.
In rules content: (/etc/falco/rules.d/custom.local.yaml:0:0)
    rule 'Calico packet': (/etc/falco/rules.d/custom.local.yaml:6:2)
------
- rule: Calico packet
  ^
------
LOAD_NO_EVTTYPE (Condition has no event-type restriction): Rule matches too many evt.type values. This has a significant performance penalty.
In rules content: (/etc/falco/rules.d/custom.local.yaml:0:0)
    list 'user_known_ingress_remote_file_copy_images': (/etc/falco/rules.d/custom.local.yaml:393:2)
------
- list: user_known_ingress_remote_file_copy_images
  ^
------
LOAD_UNUSED_LIST (Unused list): List not referred to by any other rule/macro
Thu Oct  3 16:38:28 2024: Hostname value has been overridden via environment variable to: aks-main-14571596-vmss000008
Thu Oct  3 16:38:28 2024: Runtime error: plugin capability used before init: k8smeta. Exiting.

[CarpathianUA]

We're experiencing the same behavior and rollbacked chart version back to 4.8.3

[alacuku]

@jemag, @CarpathianUA it's a regression introduced in Falco 0.39.0. We are working on the fix and will release a patch as soon as possible.

Thanks for reporting the bug.

[FedeDP]

Thu Oct 3 16:27:33 2024: /etc/falco/falco.yaml | schema validation: failed for [plugins][0][init_config]: Value type not permitted by 'type' constraint.

Hi!
The schema validation warning is caused by this line:

• init_config: null

Since it is null, you can just remove the line or keep it "".
See the json schema for the init_config field: https://github.com/falcosecurity/falco/blob/master/userspace/falco/config_json_schema.h#L589

[FedeDP]

The fix is up! falcosecurity/falco#3368

[jemag]

Since it is null, you can just remove the line or keep it "".
See the json schema for the init_config field: falcosecurity/falco@master/userspace/falco/config_json_schema.h#L589

@FedeDP That null is being generated by the chart itself. If indeed it should be removed or kept as "", then it would make sense to make those adjustments in how the chart renders the plugins section of the config

[jemag]

perhaps we can just change default init_config value to this:

[FedeDP]

Well i'd just comment out also the init_config line there :) cc @alacuku

[alacuku]

This issue has been fixed in falco 0.39.1 and charts 4.10.0.

[alacuku]

/close

[poiana]

@alacuku: Closing this issue.

In response to this:

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.