Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Filterx function framework + example #18

Closed
wants to merge 55 commits into from
Closed

Filterx function framework + example #18

wants to merge 55 commits into from

Commits on Dec 5, 2023

  1. generic-number: move "precision" member to pack the struct into 16 bytes 

    Signed-off-by: Balazs Scheidler <[email protected]>
    @bazsi
    bazsi committed Dec 5, 2023
    Configuration menu
    Copy the full SHA
    c8257ce View commit details
    Browse the repository at this point in the history

  2. str-utils: return the embedded string from g_string_steal()

    @bazsi
    bazsi committed Dec 5, 2023
    Configuration menu
    Copy the full SHA
    d068679 View commit details
    Browse the repository at this point in the history

  3. cfg-lex.l: make <INITIAL> state explicit in rules

    @bazsi
    bazsi committed Dec 5, 2023
    Configuration menu
    Copy the full SHA
    d548878 View commit details
    Browse the repository at this point in the history

  4. cfg-lex.l: add <filterx> state

    @bazsi
    bazsi committed Dec 5, 2023
    Configuration menu
    Copy the full SHA
    f003339 View commit details
    Browse the repository at this point in the history

  5. cfg-lex.l: add cfg_lexer_{push,pop}_filterx_state() 

    Signed-off-by: Balazs Scheidler <[email protected]>
    @bazsi
    bazsi committed Dec 5, 2023
    Configuration menu
    Copy the full SHA
    8d23901 View commit details
    Browse the repository at this point in the history

  6. cfg-lexer.c: start filterx state in LL_CONTEXT_FILTERX 

    Signed-off-by: Balazs Scheidler <[email protected]>
    @bazsi
    bazsi committed Dec 5, 2023
    Configuration menu
    Copy the full SHA
    d69e582 View commit details
    Browse the repository at this point in the history

  7. cfg-lex.l: use strict identifier pattern for filterx mode 

    Signed-off-by: Balazs Scheidler <[email protected]>
    @bazsi
    bazsi committed Dec 5, 2023
    Configuration menu
    Copy the full SHA
    36d00a6 View commit details
    Browse the repository at this point in the history

  8. cfg-lex.l: drop the leading ignored characters from tokens 

    The lexer drops whitespace and leading commas but they should not make it
into token values.

Signed-off-by: Balazs Scheidler <[email protected]>
    @bazsi
    bazsi committed Dec 5, 2023
    Configuration menu
    Copy the full SHA
    007be25 View commit details
    Browse the repository at this point in the history

  9. filter-expr-parser: move operators to have dedicated tokens at the le… 

    …xer level

Previously everything was captured as LL_IDENTIFIER, operators as well,
and then mapped to tokens using the keyword mechanism. Since I'd like
to make the LL_IDENTIFIER pattern stricter, that wouldn't allow
typical operator characters into LL_IDENTIFIERS, so these should be
recognized at the lexer level.

NOTE: this does not include the spelled out operators like "lt" for "<",
those remain LL_IDENTIFIER and are continued to be mapped using the
keyword mechanism.

Signed-off-by: Balazs Scheidler <[email protected]>
    @bazsi
    bazsi committed Dec 5, 2023
    Configuration menu
    Copy the full SHA
    f133784 View commit details
    Browse the repository at this point in the history

  10. filterx: initial skeleton for filterx expressions 

    Signed-off-by: Balazs Scheidler <[email protected]>
    @bazsi
    bazsi committed Dec 5, 2023
    Configuration menu
    Copy the full SHA
    70fd2b7 View commit details
    Browse the repository at this point in the history

  11. filterx: add FilterXObject base class 

    Signed-off-by: Balazs Scheidler <[email protected]>
    @bazsi
    bazsi committed Dec 5, 2023
    Configuration menu
    Copy the full SHA
    07a4f89 View commit details
    Browse the repository at this point in the history

  12. filterx: add FilterXExpr base class 

    Signed-off-by: Balazs Scheidler <[email protected]>
    @bazsi
    bazsi committed Dec 5, 2023
    Configuration menu
    Copy the full SHA
    d15187a View commit details
    Browse the repository at this point in the history

  13. filterx: add object freezing mechanism 

    Some objects can be used by multiple threads (e.g. the ones created
as literals during configuration parsing). Our ref count is not atomic
and I don't intend to make it one.

By making an object "frozen" it's reference count is set to an
extremal value and from then on, ref/unref does not change the refcount.

Signed-off-by: Balazs Scheidler <[email protected]>
    @bazsi
    bazsi committed Dec 5, 2023
    Configuration menu
    Copy the full SHA
    d500fca View commit details
    Browse the repository at this point in the history

  14. filterx: add object wrapper for primitive types 

    Signed-off-by: Balazs Scheidler <[email protected]>
    @bazsi
    bazsi committed Dec 5, 2023
    Configuration menu
    Copy the full SHA
    a8e9b75 View commit details
    Browse the repository at this point in the history

  15. filterx: add literal expression 

    Signed-off-by: Balazs Scheidler <[email protected]>
    @bazsi
    bazsi committed Dec 5, 2023
    Configuration menu
    Copy the full SHA
    3a490d0 View commit details
    Browse the repository at this point in the history

  16. filterx: add FilterXScope, the caching of unmarshalled values 

    Signed-off-by: Balazs Scheidler <[email protected]>
    @bazsi
    bazsi committed Dec 5, 2023
    Configuration menu
    Copy the full SHA
    2c386bb View commit details
    Browse the repository at this point in the history

  17. filterx: add FilterXEval, the evaluation framework 

    Signed-off-by: Balazs Scheidler <[email protected]>
    @bazsi
    bazsi committed Dec 5, 2023
    Configuration menu
    Copy the full SHA
    79586a7 View commit details
    Browse the repository at this point in the history

  18. filterx: add FilterXWeakRef 

    Signed-off-by: Balazs Scheidler <[email protected]>
    @bazsi
    bazsi committed Dec 5, 2023
    Configuration menu
    Copy the full SHA
    36b8477 View commit details
    Browse the repository at this point in the history

  19. filterx: glue filterx expressions to the log pipeline 

    Signed-off-by: Balazs Scheidler <[email protected]>
    @bazsi
    bazsi committed Dec 5, 2023
    Configuration menu
    Copy the full SHA
    f8a3718 View commit details
    Browse the repository at this point in the history

  20. cfg-grammar: integrate filterx expressions

    @bazsi
    bazsi committed Dec 5, 2023
    Configuration menu
    Copy the full SHA
    7efdf2a View commit details
    Browse the repository at this point in the history

  21. filterx: add null object and null literal 

    Signed-off-by: Balazs Scheidler <[email protected]>
    @bazsi
    bazsi committed Dec 5, 2023
    Configuration menu
    Copy the full SHA
    ce7bcd5 View commit details
    Browse the repository at this point in the history

  22. filterx: add string object 

    Signed-off-by: Balazs Scheidler <[email protected]>
    @bazsi
    bazsi committed Dec 5, 2023
    Configuration menu
    Copy the full SHA
    70ad43a View commit details
    Browse the repository at this point in the history

  23. filterx: add JSON object and JSON literal 

    Signed-off-by: Balazs Scheidler <[email protected]>
    @bazsi
    bazsi committed Dec 5, 2023
    Configuration menu
    Copy the full SHA
    9b127e3 View commit details
    Browse the repository at this point in the history

  24. filterx: add add datetime object 

    Signed-off-by: Balazs Scheidler <[email protected]>
    @bazsi
    bazsi committed Dec 5, 2023
    Configuration menu
    Copy the full SHA
    03112b1 View commit details
    Browse the repository at this point in the history

  25. filterx: add FilterXList 

    Signed-off-by: Balazs Scheidler <[email protected]>
    @bazsi
    bazsi committed Dec 5, 2023
    Configuration menu
    Copy the full SHA
    5fe3de9 View commit details
    Browse the repository at this point in the history

  26. filterx: add FilterXMessageValue to encapsulate a (value, type) pair 

    Signed-off-by: Balazs Scheidler <[email protected]>
    @bazsi
    bazsi committed Dec 5, 2023
    Configuration menu
    Copy the full SHA
    8b20602 View commit details
    Browse the repository at this point in the history

  27. filterx: implement template expressions 

    Signed-off-by: Balazs Scheidler <[email protected]>
    @bazsi
    bazsi committed Dec 5, 2023
    Configuration menu
    Copy the full SHA
    28d9b5c View commit details
    Browse the repository at this point in the history

  28. filterx: add boolean algebra between expressions 

    Signed-off-by: Balazs Scheidler <[email protected]>
    @bazsi
    bazsi committed Dec 5, 2023
    Configuration menu
    Copy the full SHA
    1b26ea6 View commit details
    Browse the repository at this point in the history

  29. filterx: add getattr/setattr for objects 

    Signed-off-by: Balazs Scheidler <[email protected]>
    @bazsi
    bazsi committed Dec 5, 2023
    Configuration menu
    Copy the full SHA
    25a1e7e View commit details
    Browse the repository at this point in the history

  30. filterx: add assignment expr and grammar 

    Signed-off-by: Balazs Scheidler <[email protected]>
    @bazsi
    bazsi committed Dec 5, 2023
    Configuration menu
    Copy the full SHA
    c0ae245 View commit details
    Browse the repository at this point in the history

  31. filterx: add comparison expr and grammar 

    Signed-off-by: Balazs Scheidler <[email protected]>
    @bazsi
    bazsi committed Dec 5, 2023
    Configuration menu
    Copy the full SHA
    b046e06 View commit details
    Browse the repository at this point in the history

  32. filterx: add template specialization for literal/trivial cases 

    Signed-off-by: Balazs Scheidler <[email protected]>
    @bazsi
    bazsi committed Dec 5, 2023
    Configuration menu
    Copy the full SHA
    6fb38f4 View commit details
    Browse the repository at this point in the history

  33. filterx: extract literal_object rule in the grammar 

    Signed-off-by: Balazs Scheidler <[email protected]>
    @bazsi
    bazsi committed Dec 5, 2023
    Configuration menu
    Copy the full SHA
    8a2750d View commit details
    Browse the repository at this point in the history

  34. filterx: freeze objects related to literal expressions 

    These objects may be used by multiple threads, as they are allocated at
parsing time.  Make sure they don't get freed, nor do we race on their
ref_cnt by freezing them.

Signed-off-by: Balazs Scheidler <[email protected]>
    @bazsi
    bazsi committed Dec 5, 2023
    Configuration menu
    Copy the full SHA
    f0d8b90 View commit details
    Browse the repository at this point in the history

  35. filterx: fix up filterx Makefiles 

    Signed-off-by: Balazs Scheidler <[email protected]>
    @bazsi
    bazsi committed Dec 5, 2023
    Configuration menu
    Copy the full SHA
    293fbf5 View commit details
    Browse the repository at this point in the history

  36. light: add tests for the filterx language 

    Signed-off-by: Balazs Scheidler <[email protected]>
    @bazsi
    bazsi committed Dec 5, 2023
    Configuration menu
    Copy the full SHA
    7f2f70d View commit details
    Browse the repository at this point in the history

Commits on Dec 12, 2023

  1. correlation: Fix matching expression for NLSTRING 

    `node_to_insert` seems to only contain a single patterndb expression in
all other examples, so align with them.

Signed-off-by: Romain Tartière <[email protected]>
    @smortex @bshifter
    smortex authored and bshifter committed Dec 12, 2023
    Configuration menu
    Copy the full SHA
    5f9e1ef View commit details
    Browse the repository at this point in the history

  2. correlation: Capture the end of a message with NLSTRING 

    When parsing a message, NLSTRING stops when it finds a `\n` char.  If
the message does not contain any `\n` char, the matching fail and the
message portion is not matched.

However, the documentation says that:
> For single-line messages, NLSTRING is equivalent with ANYSTRING

In order to match what the documentation say, we need to not return an
error when `\n` is not found, and instead capture the end of the
message.

Signed-off-by: Romain Tartière <[email protected]>
    @smortex @bshifter
    smortex authored and bshifter committed Dec 12, 2023
    Configuration menu
    Copy the full SHA
    b6fd03b View commit details
    Browse the repository at this point in the history

  3. type-hinting: support explicit value length 

    Sometimes we are casting strings with an explicit length value, but
the type hinting code simply assumed that everything was NUL terminated,
which is not the case for indirect values.

Signed-off-by: Balazs Scheidler <[email protected]>
Signed-off-by: shifter <[email protected]>
    @bazsi @bshifter
    bazsi authored and bshifter committed Dec 12, 2023
    Configuration menu
    Copy the full SHA
    0a9a511 View commit details
    Browse the repository at this point in the history

  4. loggen: print_statistics(): make initial timestamp calculation easier… 

    … to follow

Signed-off-by: Balazs Scheidler <[email protected]>
    @bazsi @bshifter
    bazsi authored and bshifter committed Dec 12, 2023
    Configuration menu
    Copy the full SHA
    caf6414 View commit details
    Browse the repository at this point in the history

  5. loggen: do not check exit condition for every message generated 

    Signed-off-by: Balazs Scheidler <[email protected]>
    @bazsi @bshifter
    bazsi authored and bshifter committed Dec 12, 2023
    Configuration menu
    Copy the full SHA
    1166a74 View commit details
    Browse the repository at this point in the history

  6. loggen: do not check rate limit for every message sent 

    Signed-off-by: Balazs Scheidler <[email protected]>
    @bazsi @bshifter
    bazsi authored and bshifter committed Dec 12, 2023
    Configuration menu
    Copy the full SHA
    fb8c86a View commit details
    Browse the repository at this point in the history

  7. loggen: avoid generating the timestamp for every message sent 

    As it has significant overhead. Do it a few times every second.

Signed-off-by: Balazs Scheidler <[email protected]>
    @bazsi @bshifter
    bazsi authored and bshifter committed Dec 12, 2023
    Configuration menu
    Copy the full SHA
    c9e88ad View commit details
    Browse the repository at this point in the history

  8. news: added news file 

    Signed-off-by: Balazs Scheidler <[email protected]>
    @bazsi @bshifter
    bazsi authored and bshifter committed Dec 12, 2023
    Configuration menu
    Copy the full SHA
    c4f86f0 View commit details
    Browse the repository at this point in the history

  9. filterx: implement template expressions 

    Signed-off-by: Balazs Scheidler <[email protected]>
    @bazsi @bshifter
    bazsi authored and bshifter committed Dec 12, 2023
    Configuration menu
    Copy the full SHA
    c40df77 View commit details
    Browse the repository at this point in the history

  10. merge: debug messgea

    @bazsi @bshifter
    bazsi authored and bshifter committed Dec 12, 2023
    Configuration menu
    Copy the full SHA
    f9d178f View commit details
    Browse the repository at this point in the history

  11. merge: filterx builtin

    @bazsi @bshifter
    bazsi authored and bshifter committed Dec 12, 2023
    Configuration menu
    Copy the full SHA
    d51afb5 View commit details
    Browse the repository at this point in the history

  12. merge: CFLAGS add no-omit-frame-pointer

    @bazsi @bshifter
    bazsi authored and bshifter committed Dec 12, 2023
    Configuration menu
    Copy the full SHA
    1fd6763 View commit details
    Browse the repository at this point in the history

  13. merge: filterx-function

    @bazsi @bshifter
    bazsi authored and bshifter committed Dec 12, 2023
    Configuration menu
    Copy the full SHA
    5c76ee2 View commit details
    Browse the repository at this point in the history

  14. merge: name-value pair ref via braces

    @bazsi @bshifter
    bazsi authored and bshifter committed Dec 12, 2023
    Configuration menu
    Copy the full SHA
    47f807a View commit details
    Browse the repository at this point in the history

  15. fixup! merge: filterx builtin

    @bshifter
    bshifter committed Dec 12, 2023
    Configuration menu
    Copy the full SHA
    c162b5c View commit details
    Browse the repository at this point in the history

  16. filterx: expose FilterXExpression's free method 

    Signed-off-by: shifter <[email protected]>
    @bshifter
    bshifter committed Dec 12, 2023
    Configuration menu
    Copy the full SHA
    ec75e2c View commit details
    Browse the repository at this point in the history

  17. filterx: add pluginable function 

    Signed-off-by: shifter <[email protected]>
    @bshifter
    bshifter committed Dec 12, 2023
    Configuration menu
    Copy the full SHA
    65a173c View commit details
    Browse the repository at this point in the history

  18. examples: add filterx function plugin example 

    Signed-off-by: shifter <[email protected]>
    @bshifter
    bshifter committed Dec 12, 2023
    Configuration menu
    Copy the full SHA
    eb2d177 View commit details
    Browse the repository at this point in the history

  19. filterx: add cmake build 

    Signed-off-by: shifter <[email protected]>
    @bshifter
    bshifter committed Dec 12, 2023
    Configuration menu
    Copy the full SHA
    77f34e2 View commit details
    Browse the repository at this point in the history