diff --git a/docs/src/user-guide/index.md b/docs/src/user-guide/index.md index 7e0b909a7..55f71a6f1 100644 --- a/docs/src/user-guide/index.md +++ b/docs/src/user-guide/index.md @@ -14,10 +14,10 @@ A guide to viewing log files in the log viewer. ::: :::{grid-item-card} -:link: format-struct-logs-overview -Formatting structured logs +:link: struct-logs/index +Working with structured logs ^^^ -A guide to formatting structured (e.g. JSON) logs as plain text. +Guides and reference docs for working with structured logs. ::: :::{grid-item-card} @@ -37,12 +37,12 @@ quick-start :::{toctree} :hidden: -:caption: Formatting structured logs +:caption: Working with structured logs :glob: -format-struct-logs-overview -format-struct-logs-syntax -format-struct-logs-formatters +struct-logs/index +struct-logs/format/index +struct-logs/specifying-keys ::: :::{toctree} diff --git a/docs/src/user-guide/format-struct-logs-formatters.md b/docs/src/user-guide/struct-logs/format/formatters.md similarity index 100% rename from docs/src/user-guide/format-struct-logs-formatters.md rename to docs/src/user-guide/struct-logs/format/formatters.md diff --git a/docs/src/user-guide/format-struct-logs-overview.md b/docs/src/user-guide/struct-logs/format/index.md similarity index 87% rename from docs/src/user-guide/format-struct-logs-overview.md rename to docs/src/user-guide/struct-logs/format/index.md index e4039c01f..fcbc99c73 100644 --- a/docs/src/user-guide/format-struct-logs-overview.md +++ b/docs/src/user-guide/struct-logs/format/index.md @@ -1,4 +1,4 @@ -# Overview +# Formatting as plain text The log viewer can format structured (e.g. JSON) logs as plain text using a format string. The format string allows you to select which fields to include and how they should be formatted. You can @@ -33,5 +33,12 @@ The formatted log would appear as: ``` For reference docs, see: -* [Format string syntax](format-struct-logs-syntax) -* [Formatters](format-struct-logs-formatters) +* [Format string syntax](syntax) +* [Formatters](formatters) + +:::{toctree} +:hidden: + +syntax +formatters +::: diff --git a/docs/src/user-guide/format-struct-logs-syntax.md b/docs/src/user-guide/struct-logs/format/syntax.md similarity index 54% rename from docs/src/user-guide/format-struct-logs-syntax.md rename to docs/src/user-guide/struct-logs/format/syntax.md index 29ae306b2..453190558 100644 --- a/docs/src/user-guide/format-struct-logs-syntax.md +++ b/docs/src/user-guide/struct-logs/format/syntax.md @@ -16,22 +16,14 @@ backslash: Each field placeholder is enclosed in braces (`{` and `}`) and has the following form, consisting of three components: ``` -{[:[:]]} +{[:[:]]} ``` -### field-name (required) +### field-key (required) Defines the key of the field whose value should replace the placeholder. -* Nested fields can be specified using periods (`.`) to denote hierarchy. - * E.g., the field `{"a:" {"b": 0}}` may be denoted by `a.b`. -* Field names can contain any character, except the following characters must be escaped with a - backslash: - * `.` - * `$` - * `{` - * `}` - * `:` - * `\` +:::{include} ../key-syntax.md +::: ### formatter-name (optional) The name of the formatter to apply to the value before inserting it into the string. @@ -43,6 +35,8 @@ The name of the formatter to apply to the value before inserting it into the str * `:` * `\` +For a list of currently supported formatters, see [Formatters](formatters). + ### formatter-options (optional) Defines any options for the formatter denoted by `formatter-name`. @@ -63,28 +57,31 @@ Every format string contains an implicit trailing newline so that each formatted a newline. ## Examples -Consider the following log event: -``` + +### Formatting JSON logs events + +Consider the following JSON log event: +```json { - "@timestamp": 1427153388942, + "ts": 1427153388942, "level": "INFO", "thread": 0, "latency": { "msecs": 56400, - "secs": 56.4, + "secs": 56.4 }, - "an.odd.key{name}": "org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from hadoop-metrics2.properties" + "@an.odd.key{name}": "org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from hadoop-metrics2.properties" } ``` -We can format this using the following YScope format string: +We can format this using the following format string: ``` -{@timestamp:timestamp:YYYY-MM-DD HH\:mm\:ss.SSS} {level} \{{thread}\} latency={latency.secs:round} {an\.odd\.key\{name\}} +{ts:timestamp:YYYY-MM-DD HH\:mm\:ss.SSS} {level} \{{thread}\} latency={latency.secs:round} {\@an\.odd\.key\{name\}} ``` -* In the first placeholder, we have the field name `@timestamp`, a formatter called `timestamp`, and - the formatter's options which are a date format string. +* In the first placeholder, we have the field key `ts`, a formatter called `timestamp`, and the + formatter's options which are a date format string. * The second and third placeholders simply stringify the values of the given fields. * The fourth placeholder uses the `round` formatter to round a nested field's value; this placeholder doesn't specify any formatter options, so the defaults will be used. @@ -95,4 +92,40 @@ The formatted string will be: 2015-03-23 19:29:48.942 INFO {0} latency=56 org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from hadoop-metrics2.properties ``` -For a list of currently supported formatters, see [Formatters](format-struct-logs-formatters). +### Formatting kv-pair IR log events + +Consider the following kv-pair IR log event: + +:::{note} +In the example below, for simplicity, we render the log event as JSON with the auto-generated +kv-pairs under the `auto-generated` key, and the user-generated kv-pairs under the `user-generated` +key, but these keys don't exist in the log event. +::: + +```json +{ + "auto-generated": { + "ts": 1741371422000 + }, + "user-generated": { + "message": "Callback registered to fire in 5 seconds:", + "ts": 1741371427000 + } +} +``` + +We can format this using the following format string: + +``` +{@ts:timestamp} {message} {ts:timestamp} +``` + +* In the first placeholder, we have the auto-generated field key `@ts`. The `@` prefix specifies + that the field is from the auto-generated namespace. +* The second and third placeholders refer to the `message` and `ts` fields in the user-generated + namespace. + +The formatted string will be: +``` +2025-03-07T18:17:02Z Callback registered to fire in 5 seconds: 2025-03-07T18:17:07Z +``` diff --git a/docs/src/user-guide/struct-logs/index.md b/docs/src/user-guide/struct-logs/index.md new file mode 100644 index 000000000..00d2f0a03 --- /dev/null +++ b/docs/src/user-guide/struct-logs/index.md @@ -0,0 +1,21 @@ +# Overview + +The guides below describe how to configure the log viewer to work with structured logs. + +::::{grid} 1 1 2 2 +:gutter: 2 + +:::{grid-item-card} +:link: format/index +Formatting as plain text +^^^ +How to format structured logs as plain text, using a format string. +::: + +:::{grid-item-card} +:link: specifying-keys +Specifying keys +^^^ +How to specify keys when configuring the log viewer. +::: +:::: diff --git a/docs/src/user-guide/struct-logs/key-syntax.md b/docs/src/user-guide/struct-logs/key-syntax.md new file mode 100644 index 000000000..0a2b7b5c4 --- /dev/null +++ b/docs/src/user-guide/struct-logs/key-syntax.md @@ -0,0 +1,14 @@ +* Nested keys can be specified using periods (`.`) to denote hierarchy. + * E.g., the field `{"a:" {"b": 0}}` may be denoted by `a.b`. +* Auto-generated keys in a [Key-Value Pair IR Stream][kv-pair-ir] can be specified by using `@` as + a prefix. + * E.g., the auto-generated key `ts` would be specified as `@ts`. +* Keys can contain any character, except the following characters must be escaped with a backslash: + * `.` + * `@` + * `{` + * `}` + * `:` + * `\` + +[kv-pair-ir]: https://docs.yscope.com/clp/main/dev-guide/design-kv-ir-streams/auto-gen-kv-pairs.html diff --git a/docs/src/user-guide/struct-logs/specifying-keys.md b/docs/src/user-guide/struct-logs/specifying-keys.md new file mode 100644 index 000000000..8749a97ac --- /dev/null +++ b/docs/src/user-guide/struct-logs/specifying-keys.md @@ -0,0 +1,12 @@ +# Specifying keys + +Viewing structured logs requires specifying keys for: +* The format string +* Authoritative fields such as the log level and timestamp + +Both options are found the settings dialog ({fas}`gear`). + +## Syntax + +:::{include} key-syntax.md +::: \ No newline at end of file diff --git a/src/components/CentralContainer/Sidebar/SidebarTabs/SettingsTabPanel/index.tsx b/src/components/CentralContainer/Sidebar/SidebarTabs/SettingsTabPanel/index.tsx index 76f789f8f..6fb406d9e 100644 --- a/src/components/CentralContainer/Sidebar/SidebarTabs/SettingsTabPanel/index.tsx +++ b/src/components/CentralContainer/Sidebar/SidebarTabs/SettingsTabPanel/index.tsx @@ -48,18 +48,19 @@ const getConfigFormFields = () => [ { helperText: ( - [JSON] Format string for formatting a JSON log event as plain text. See the + [Structured] Format string for formatting a structured log event as plain text. + Leave blank to display the entire log event. See {" "} - format string syntax docs + here {" "} - or leave this blank to display the entire log event. + for syntax. ), initialValue: getConfig(CONFIG_KEY.DECODER_OPTIONS).formatString, @@ -68,14 +69,44 @@ const getConfigFormFields = () => [ type: "text", }, { - helperText: "[JSON] Key to extract the log level from.", + helperText: ( + + [Structured] Key that maps to each log event's log level. See + {" "} + + here + + {" "} + for syntax. + + ), initialValue: getConfig(CONFIG_KEY.DECODER_OPTIONS).logLevelKey, key: LOCAL_STORAGE_KEY.DECODER_OPTIONS_LOG_LEVEL_KEY, label: "Decoder: Log level key", type: "text", }, { - helperText: "[JSON] Key to extract the log timestamp from.", + helperText: ( + + [Structured] Key that maps to each log event's timestamp. See + {" "} + + here + + {" "} + for syntax. + + ), initialValue: getConfig(CONFIG_KEY.DECODER_OPTIONS).timestampKey, key: LOCAL_STORAGE_KEY.DECODER_OPTIONS_TIMESTAMP_KEY, label: "Decoder: Timestamp key",