-
Notifications
You must be signed in to change notification settings - Fork 12
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
docs: Add user docs for opening log files, level filtering, and formatting structured logs. #132
Merged
Merged
Changes from all commits
Commits
Show all changes
26 commits
Select commit
Hold shift + click to select a range
bb5617a
added basic user docs
davemarco 28067b6
Refactor user-guide/index.md
kirkrodrigues 637b520
Refactor user-guide/quick-start-overview.md
kirkrodrigues a3d053b
Refactor user-guide/format-string-overview.md
kirkrodrigues 941b558
Refactor user-guide/format-string-syntax.md
kirkrodrigues 60aade6
Refactor user-guide/format-string-formatters.md
kirkrodrigues 8d7d10a
Refactor user-guide/log-level-filter.md
kirkrodrigues 59f3e22
kirk review
davemarco 42e1f43
fix broken link
davemarco fd19fa2
fix more broken things..
davemarco c2e92cf
more
davemarco fbd51cd
Fix typo.
kirkrodrigues e18af3c
Add images to git LFS.
kirkrodrigues 68ab84e
Fix typo.
kirkrodrigues 6ca3093
Add some commas for clarity.
kirkrodrigues 163d2c6
Revert "Add images to git LFS."
kirkrodrigues 2c9d621
Re-add images as normal files.
kirkrodrigues 2e774dc
Retry adding images as normal files.
kirkrodrigues a0742a6
Retry adding images as normal files.
kirkrodrigues 893820e
Remove images.
kirkrodrigues 04442ef
Update docs/src/user-guide/log-level-filtering.md
davemarco 2efe385
Update docs/src/user-guide/log-level-filtering.md
davemarco e60e376
add lfs
davemarco 32605d2
Update docs/src/user-guide/format-struct-logs-syntax.md
davemarco 7d56440
Update docs/src/user-guide/format-struct-logs-overview.md
davemarco 6c0359b
Add nested field to structured log formatting example; Replace curly …
kirkrodrigues File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1 @@ | ||
*.png filter=lfs diff=lfs merge=lfs -text |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,38 @@ | ||
# Formatters | ||
|
||
Formatters allow you to transform how field values are displayed. Below we describe all available | ||
formatters and their options. | ||
|
||
## Timestamp formatter | ||
|
||
Converts millisecond Unix timestamps to human-readable date-time strings. | ||
|
||
### Usage | ||
``` | ||
{field:timestamp[:options]} | ||
``` | ||
|
||
### Options | ||
A [Day.js format string](https://day.js.org/docs/en/display/format) for the date and time. | ||
|
||
**Default:** `YYYY-MM-DDTHH:mm:ssZ` (ISO 8601) | ||
|
||
### Examples | ||
Assuming the field `ts` is `1732703400000`: | ||
|
||
* `{ts:timestamp}` → `2024-11-27T10:30:00Z` | ||
* `{ts:timestamp:YYYY-MM-DD}` → `2024-11-27` | ||
|
||
## Round formatter | ||
|
||
Rounds a numeric value to the nearest integer. | ||
|
||
### Usage | ||
``` | ||
{field:round} | ||
``` | ||
|
||
### Examples | ||
Assuming the field `value` is `5.7`: | ||
|
||
* `{value:round}` → `6` |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,37 @@ | ||
# Overview | ||
|
||
The log viewer can format structured (e.g. JSON) logs as plain text using a format string. The | ||
format string allows you to select which fields to include and how they should be formatted. You can | ||
configure the format string through the settings ({fas}`gear`) dialog. | ||
|
||
For example, for a JSON log with many fields: | ||
|
||
```json | ||
{ | ||
"ts": 1732733160216, | ||
"level": "ERROR", | ||
"message": "Failed to process payment.", | ||
"trace_id": "abc123def456", | ||
"span_id": "span_789xyz", | ||
"service": "payment-service", | ||
"environment": "production" | ||
} | ||
``` | ||
|
||
You might want to show only: | ||
* `timestamp` | ||
* `level` | ||
* `message` | ||
|
||
You can achieve this with the following format string `{ts:timestamp} {level} {message}`. | ||
`{ts:timestamp}` formats the timestamp field (Unix epoch) as an ISO 8601 date format while `{level}` | ||
and `{message}` display the specified fields as is. | ||
|
||
The formatted log would appear as: | ||
``` | ||
2024-11-27T18:46:00Z ERROR Failed to process payment. | ||
``` | ||
|
||
For reference docs, see: | ||
* [Format string syntax](format-struct-logs-syntax) | ||
* [Formatters](format-struct-logs-formatters) |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,98 @@ | ||
# Format string syntax | ||
|
||
Each format string is composed of: | ||
* [Static text](#static-text) | ||
* [Field placeholders](#field-placeholders) | ||
* [An implicit trailing newline](#implicit-trailing-newline) | ||
|
||
## Static text | ||
Static text may contain any character, except the following characters must be escaped with a | ||
backslash: | ||
* `{` | ||
* `}` | ||
* `\` | ||
|
||
## Field placeholders | ||
Each field placeholder is enclosed in braces (`{` and `}`) and has the following form, consisting of | ||
three components: | ||
``` | ||
{<field-name>[:<formatter-name>[:<formatter-options>]]} | ||
``` | ||
|
||
### field-name (required) | ||
Defines the key of the field whose value should replace the placeholder. | ||
|
||
* Nested fields can be specified using periods (`.`) to denote hierarchy. | ||
* E.g., the field `{"a:" {"b": 0}}` may be denoted by `a.b`. | ||
* Field names can contain any character, except the following characters must be escaped with a | ||
backslash: | ||
* `.` | ||
* `$` | ||
* `{` | ||
* `}` | ||
* `:` | ||
* `\` | ||
|
||
### formatter-name (optional) | ||
The name of the formatter to apply to the value before inserting it into the string. | ||
|
||
* Formatter names can contain any character except a space (` `), and the following characters must | ||
be escaped with a backslash (`\`): | ||
* `{` | ||
* `}` | ||
* `:` | ||
* `\` | ||
|
||
### formatter-options (optional) | ||
Defines any options for the formatter denoted by `formatter-name`. | ||
|
||
* Formatter options can contain any character, except the following characters must be escaped with | ||
a backslash: | ||
* `{` | ||
* `}` | ||
* `:` | ||
* `\` | ||
|
||
:::{note} | ||
`formatter-options` can only be specified if `formatter-name` was specified. | ||
::: | ||
|
||
## Implicit trailing newline | ||
|
||
Every format string contains an implicit trailing newline so that each formatted log event ends with | ||
a newline. | ||
|
||
## Examples | ||
Consider the following log event: | ||
``` | ||
{ | ||
"@timestamp": 1427153388942, | ||
"level": "INFO", | ||
"thread": 0, | ||
"latency": { | ||
"msecs": 56400, | ||
"secs": 56.4, | ||
}, | ||
"an.odd.key{name}": "org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from hadoop-metrics2.properties" | ||
} | ||
``` | ||
|
||
We can format this using the following YScope format string: | ||
|
||
``` | ||
{@timestamp:timestamp:YYYY-MM-DD HH\:MM\:ss.SSS} {level} \{{thread}\} latency={latency.secs:round} {an\.odd\.key\{name\}} | ||
``` | ||
|
||
* In the first placeholder, we have the field name `@timestamp`, a formatter called `timestamp`, and | ||
the formatter's options which are a date format string. | ||
* The second and third placeholders simply stringify the values of the given fields. | ||
* The fourth placeholder uses the `round` formatter to round a nested field's value; this | ||
placeholder doesn't specify any formatter options, so the defaults will be used. | ||
* The fifth placeholder is for a field whose name contains characters that require escaping. | ||
|
||
The formatted string will be: | ||
``` | ||
2015-03-23 19:29:48.942 INFO {0} latency=56 org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from hadoop-metrics2.properties | ||
``` | ||
|
||
For a list of currently supported formatters, see [Formatters](format-struct-logs-formatters). |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,15 @@ | ||
# Overview | ||
|
||
The log viewer offers two ways to filter logs by their level. | ||
|
||
## Severity-based filtering | ||
You can filter for log events at or above a selected level by clicking the level's name. For | ||
example, selecting INFO shows INFO, WARN, ERROR, and FATAL logs. | ||
|
||
![Severity filtering](severity-filter.png) | ||
|
||
## Individual level filtering | ||
You can select specific log levels to show or hide by checking or unchecking the checkbox next to | ||
each level's name. | ||
|
||
![Individual level filtering](individual-filter.png) |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,15 @@ | ||
# Viewing logs | ||
|
||
The log viewer can open local or remote log files. The viewer currently supports viewing [CLP] IR | ||
and [JSONL] log files. | ||
|
||
## Opening local log files | ||
To open a local file, click the folder icon ({far}`folder`) in the top-left corner. This will open a file | ||
browser where you can navigate to and select the log file to open. | ||
|
||
## Opening remote log files | ||
To open a remote file, append `/?filePath=<FILE_URL>` to the current URL, replacing `<FILE_URL>` | ||
with the URL of your log file. | ||
|
||
[CLP]: https://github.com/y-scope/clp | ||
[JSONL]: https://jsonlines.org/ |
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
while not strictly necessary in this PR, shall we include an example for picking fields from nest objects?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It is mentioned in the syntax page, but maybe could include in the longer json example. @kirkrodrigues
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done