out_exit: add time_count (seconds) and record_count parameters for out_exit.

[pwhelan]

Summary

This PR adds two new parameters to the out_exit plugin:

• record_count: exit after a set number of records.
• time_count: exit after a set number of seconds have elapsed (within a flush).

Here is a run for each different mode:

fush_count:

$ valgrind ./bin/fluent-bit -i dummy -o exit -p flush_count=5 -f 1
==225327== Memcheck, a memory error detector
==225327== Copyright (C) 2002-2022, and GNU GPL'd, by Julian Seward et al.
==225327== Using Valgrind-3.22.0 and LibVEX; rerun with -h for copyright info
==225327== Command: ./bin/fluent-bit -i dummy -o exit -p flush_count=5 -f 1
==225327==
Fluent Bit v3.0.0
* Copyright (C) 2015-2024 The Fluent Bit Authors
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io

___________.__                        __    __________.__  __          ________
\_   _____/|  |  __ __   ____   _____/  |_  \______   \__|/  |_  ___  _\_____  \
 |    __)  |  | |  |  \_/ __ \ /    \   __\  |    |  _/  \   __\ \  \/ / _(__  <
 |     \   |  |_|  |  /\  ___/|   |  \  |    |    |   \  ||  |    \   / /       \
 \___  /   |____/____/  \___  >___|  /__|    |______  /__||__|     \_/ /______  /
     \/                     \/     \/               \/                        \/

[2024/03/21 18:55:14] [ info] [fluent bit] version=3.0.0, commit=efa1e89fac, pid=225327
[2024/03/21 18:55:14] [ info] [storage] ver=1.5.1, type=memory, sync=normal, checksum=off, max_chunks_up=128
[2024/03/21 18:55:14] [ info] [cmetrics] version=0.7.0
[2024/03/21 18:55:14] [ info] [ctraces ] version=0.4.0
[2024/03/21 18:55:15] [ info] [input:dummy:dummy.0] initializing
[2024/03/21 18:55:15] [ info] [input:dummy:dummy.0] storage_strategy='memory' (memory only)
[2024/03/21 18:55:15] [ info] [sp] stream processor started
[2024/03/21 18:55:20] [ warn] [engine] service will shutdown in max 5 seconds
[2024/03/21 18:55:20] [ info] [input] pausing dummy.0
[2024/03/21 18:55:21] [ info] [engine] service has stopped (0 pending tasks)
[2024/03/21 18:55:21] [ info] [input] pausing dummy.0
==225327==
==225327== HEAP SUMMARY:
==225327==     in use at exit: 0 bytes in 0 blocks
==225327==   total heap usage: 1,720 allocs, 1,720 frees, 2,247,178 bytes allocated
==225327==
==225327== All heap blocks were freed -- no leaks are possible
==225327==
==225327== For lists of detected and suppressed errors, rerun with: -s
==225327== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)

record_count:

$ valgrind ./bin/fluent-bit -i dummy -p copies=2 -o exit -p record_count=6 -f 1
==225485== Memcheck, a memory error detector
==225485== Copyright (C) 2002-2022, and GNU GPL'd, by Julian Seward et al.
==225485== Using Valgrind-3.22.0 and LibVEX; rerun with -h for copyright info
==225485== Command: ./bin/fluent-bit -i dummy -p copies=2 -o exit -p record_count=6 -f 1
==225485==
Fluent Bit v3.0.0
* Copyright (C) 2015-2024 The Fluent Bit Authors
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io

___________.__                        __    __________.__  __          ________
\_   _____/|  |  __ __   ____   _____/  |_  \______   \__|/  |_  ___  _\_____  \
 |    __)  |  | |  |  \_/ __ \ /    \   __\  |    |  _/  \   __\ \  \/ / _(__  <
 |     \   |  |_|  |  /\  ___/|   |  \  |    |    |   \  ||  |    \   / /       \
 \___  /   |____/____/  \___  >___|  /__|    |______  /__||__|     \_/ /______  /
     \/                     \/     \/               \/                        \/

[2024/03/21 18:56:38] [ info] [fluent bit] version=3.0.0, commit=efa1e89fac, pid=225485
[2024/03/21 18:56:38] [ info] [storage] ver=1.5.1, type=memory, sync=normal, checksum=off, max_chunks_up=128
[2024/03/21 18:56:38] [ info] [cmetrics] version=0.7.0
[2024/03/21 18:56:38] [ info] [ctraces ] version=0.4.0
[2024/03/21 18:56:38] [ info] [input:dummy:dummy.0] initializing
[2024/03/21 18:56:38] [ info] [input:dummy:dummy.0] storage_strategy='memory' (memory only)
[2024/03/21 18:56:38] [ info] [sp] stream processor started
[2024/03/21 18:56:43] [ warn] [engine] service will shutdown in max 5 seconds
[2024/03/21 18:56:43] [ info] [input] pausing dummy.0
[2024/03/21 18:56:44] [ info] [engine] service has stopped (0 pending tasks)
[2024/03/21 18:56:44] [ info] [input] pausing dummy.0
==225485==
==225485== HEAP SUMMARY:
==225485==     in use at exit: 0 bytes in 0 blocks
==225485==   total heap usage: 1,763 allocs, 1,763 frees, 2,067,814 bytes allocated
==225485==
==225485== All heap blocks were freed -- no leaks are possible
==225485==
==225485== For lists of detected and suppressed errors, rerun with: -s
==225485== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)

time_count:

$ valgrind ./bin/fluent-bit -i dummy -p copies=5 -o exit -p time_count=3 -f 1
==225570== Memcheck, a memory error detector
==225570== Copyright (C) 2002-2022, and GNU GPL'd, by Julian Seward et al.
==225570== Using Valgrind-3.22.0 and LibVEX; rerun with -h for copyright info
==225570== Command: ./bin/fluent-bit -i dummy -p copies=5 -o exit -p time_count=3 -f 1
==225570==
Fluent Bit v3.0.0
* Copyright (C) 2015-2024 The Fluent Bit Authors
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io

___________.__                        __    __________.__  __          ________
\_   _____/|  |  __ __   ____   _____/  |_  \______   \__|/  |_  ___  _\_____  \
 |    __)  |  | |  |  \_/ __ \ /    \   __\  |    |  _/  \   __\ \  \/ / _(__  <
 |     \   |  |_|  |  /\  ___/|   |  \  |    |    |   \  ||  |    \   / /       \
 \___  /   |____/____/  \___  >___|  /__|    |______  /__||__|     \_/ /______  /
     \/                     \/     \/               \/                        \/

[2024/03/21 18:57:18] [ info] [fluent bit] version=3.0.0, commit=efa1e89fac, pid=225570
[2024/03/21 18:57:18] [ info] [storage] ver=1.5.1, type=memory, sync=normal, checksum=off, max_chunks_up=128
[2024/03/21 18:57:18] [ info] [cmetrics] version=0.7.0
[2024/03/21 18:57:18] [ info] [ctraces ] version=0.4.0
[2024/03/21 18:57:18] [ info] [input:dummy:dummy.0] initializing
[2024/03/21 18:57:18] [ info] [input:dummy:dummy.0] storage_strategy='memory' (memory only)
[2024/03/21 18:57:18] [ info] [sp] stream processor started
[2024/03/21 18:57:22] [ warn] [engine] service will shutdown in max 5 seconds
[2024/03/21 18:57:22] [ info] [input] pausing dummy.0
[2024/03/21 18:57:23] [ info] [engine] service has stopped (0 pending tasks)
[2024/03/21 18:57:23] [ info] [input] pausing dummy.0
==225570==
==225570== HEAP SUMMARY:
==225570==     in use at exit: 0 bytes in 0 blocks
==225570==   total heap usage: 1,784 allocs, 1,784 frees, 1,741,196 bytes allocated
==225570==
==225570== All heap blocks were freed -- no leaks are possible
==225570==
==225570== For lists of detected and suppressed errors, rerun with: -s
==225570== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)

It might be a good idea to start the timer for time_count from the first flush, but I don't know if that is worth the effort. It would be easy enough to implement if it is deemed worthwhile.

---

Enter [N/A] in the box, if an item is not applicable to your change.

Testing
Before we can approve your change; please submit the following in a comment:

• Example configuration file for the change
• Debug log output from testing the change

• Attached Valgrind output that shows no leaks or memory corruption was found

If this is a change to packaging of containers or native binaries then please confirm it works for all targets.

• Run local packaging test showing all targets (including any new ones) build.
• Set ok-package-test label to test for all targets (requires maintainer to do).

Documentation

• Documentation required for this feature

Backporting

• Backport to latest stable release.

---

Fluent Bit is licensed under Apache 2.0, by submitting this pull request I understand that this code will be released under the terms of that license.

[pwhelan]

I have added the documentation for the new parameters here: fluent/fluent-bit-docs#1346.

[github-actions]

This PR is stale because it has been open 45 days with no activity. Remove stale label or comment or this will be closed in 10 days.