chore(level_detection): Make log level detection configurable #14784
Conversation
github-actions
bot
added
the
type/docs
| # If discover_log_levels is enabled, this list of fields will be used to determine the level (case-sensetive). Loki checks for these labels first within labels or structured metadata. | ||
| # If not, level is determined from OTLP severity number in case of OTLP logs, and finally attempted to extract from the log line itself. | ||
| # CLI flag: -validation.log-level-fields | ||
| [log_level_fields: <list of strings> | default = [level LEVEL Level Severity severity SEVERITY lvl LVL Lvl]] |
There was a problem hiding this comment.
open to suggestions if there's a better name
pkg/distributor/level_detection.go
Outdated
| if isJSON(log) { | ||
| v = l.getValueUsingJSONParser(logSlice) | ||
| } else { | ||
| v = l.getValueUsingLogfmtParser(logSlice) | ||
| } |
There was a problem hiding this comment.
What if the log line is neither JSON nor logfmt formatted?
There was a problem hiding this comment.
I'd use the same pattern that you use for detected json format rather than encapsulating the check in the lgetValueUsingLogfmtParser(logSlice) function
There was a problem hiding this comment.
makes sense. In the current case, it defaults to scan the log line as the value returned is nil. But this is implicit as you mentioned already.
I've made the changes to make it clearer.
pkg/distributor/level_detection.go
Outdated
| } | ||
|
|
||
| switch { | ||
| case bytes.EqualFold(v, []byte("trace")), bytes.EqualFold(v, []byte("trc")): |
There was a problem hiding this comment.
nit: make the byte slices a package variable to allocate them only once
| allowedFieldsMap := make(map[string]struct{}, len(allowedFields)) | ||
| for _, field := range allowedFields { | ||
| allowedFieldsMap[field] = struct{}{} | ||
| } | ||
| return allowedFieldsMap |
There was a problem hiding this comment.
allowedLabelsForLevel() can be called quite a few times. I think we should allocate the map only once.
| equalIndex := bytes.Index(line, []byte("=")) | ||
| if len(line) == 0 || equalIndex == -1 { | ||
| return false | ||
| } |
There was a problem hiding this comment.
This code can now be removed from the getValueUsingLogfmtParser() function.
What this PR does / why we need it:
Currently log level detection works only for a fixed set of default labels. The list in reality is unbounded, customers use any label or field such as log_level, logging_level, etc. There have been quite a few escalations and questions to Loki team concerning that it doesn't work properly when they something else apart from the default labels.
This PR adds an optional config to Loki that allows a tenant to specify custom labels for level detection.
The long term solution to this problem could be to leverage adaptive logs and patterns to make it robust. This is a temporary approach till we finalize on a better one.
Which issue(s) this PR fixes:
Fixes #1194
Special notes for your reviewer:
Checklist
CONTRIBUTING.mdguide (required)featPRs are unlikely to be accepted unless a case can be made for the feature actually being a bug fix to existing behavior.docs/sources/setup/upgrade/_index.mddeprecated-config.yamlanddeleted-config.yamlfiles respectively in thetools/deprecated-config-checkerdirectory. Example PR