Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Feature Request] "Native" parsing of logfmt #448

Open
akvlad opened this issue Jan 30, 2024 · 0 comments
Open

[Feature Request] "Native" parsing of logfmt #448

akvlad opened this issue Jan 30, 2024 · 0 comments

Comments

@akvlad
Copy link
Collaborator

akvlad commented Jan 30, 2024

Current state

Currently logfmt is parsed inside the JS application. If the |logfmt pipe is encountered in the script, the raw information is dowmloaded from the clickhouse server and the further processing is done on the JS side.

It is incredibly slow and inefficient.

It limits the ability to support some complicated scripts like the one represented in #444 : topk(10, count(count_over_time({kind="exception", app="$app"} | logfmt [$__range])) by (value))

On the other hand the new clickhouse versions have a function providing the mechanism of "native" logfmt parsing: https://clickhouse.com/docs/en/sql-reference/functions/tuple-map-functions#extractkeyvaluepairs

Requirements

  1. Append the initial clickhouse capabilities checks to understand if the function is supported
  2. If it is supported, then use the function to parse the logfmt pipe "natively"
  3. If it is not supported, then use the legacy JS processor.

Useful Links

https://altinity.com/blog/boosting-performance-and-flexibility-of-clickhouse-key-value-pair-extraction

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant