[BUG] Fluent Bit does not process all logs in VEBA v0.7.0 because of Docker runtime

[rguske]

Describe the bug
I've encountered that Fluent Bit isn't processing all logs in /var/log/containers. Running the Fluent Bit Daemonset in debug mode shows the following:

[2021/10/01 14:40:05] [debug] [input:tail:tail.0] scanning path /var/log/containers/*.log                                                                                                                                                                                                                                               │
│ [2021/10/01 14:40:05] [debug] [input:tail:tail.0] scan_blog add(): dismissed: /var/log/containers/activator-85cd6f6f9-nrncf_knative-serving_activator-3b631f27f6667599ae940f94afe6a65a4d1d488e7979fced513fa910082a5ae1.log, inode 404768                                                                                                │
│ [2021/10/01 14:40:05] [debug] [input:tail:tail.0] scan_blog add(): dismissed: /var/log/containers/activator-85cd6f6f9-nrncf_knative-serving_activator-ca32320178170fe4198ce1b0bd57d8ea031c7c886a7b0e3d66bb1b78b67613b8.log, inode 921337                                                                                                │
│ [2021/10/01 14:40:05] [debug] [input:tail:tail.0] scan_blog add(): dismissed: /var/log/containers/antrea-agent-gql5r_kube-system_antrea-agent-63659cdc8e5ddba3eaf729be280661b45fd198e6d2c7195965be85cdca81f41a.log, inode 536837                                                                                                        │
│ [2021/10/01 14:40:05] [debug] [input:tail:tail.0] scan_blog add(): dismissed: /var/log/containers/antrea-agent-gql5r_kube-system_antrea-agent-8726abf73577f597e15716176cfcdce442b159d00ec12f59e439719d824a9585.log, inode 1190181                                                                                                       │
│ [2021/10/01 14:40:05] [debug] [input:tail:tail.0] scan_blog add(): dismissed: /var/log/containers/antrea-agent-gql5r_kube-system_antrea-ovs-08045b767f2f8ee421b3b4d8d5b646b49b4e12199ae957cad178dd3d11670ec6.log, inode 663855 

I raised an issue on the Fluent Bit project. #4155
I've tested various Fluent Bit versions as well ConfigMap variations.

We narrowed it down to the use of Docker as the container runtime in VEBA. Tests were successful with Containerd.
The replacement is already planned for a future release. See PR #678

To Reproduce
Steps to reproduce the behavior:

1. Deploy VEBA v0.7.0 and configure the Syslog Endpoint
2. Inspect the forwarded logs

Logs are missing:

Expected behavior
All logs should be processed and forwarded to the configured Syslog server.

Version (please complete the following information):

• VEBA Form Factor: Appliance
• VEBA Version: v0.7.0

[embano1]

Closing since resolved.