Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

docs: Add user docs for opening log files, level filtering, and formatting structured logs. #132

Merged
merged 26 commits into from
Nov 28, 2024
Merged
Show file tree
Hide file tree
Changes from 23 commits
Commits
Show all changes
26 commits
Select commit Hold shift + click to select a range
bb5617a
added basic user docs
davemarco Nov 27, 2024
28067b6
Refactor user-guide/index.md
kirkrodrigues Nov 28, 2024
637b520
Refactor user-guide/quick-start-overview.md
kirkrodrigues Nov 28, 2024
a3d053b
Refactor user-guide/format-string-overview.md
kirkrodrigues Nov 28, 2024
941b558
Refactor user-guide/format-string-syntax.md
kirkrodrigues Nov 28, 2024
60aade6
Refactor user-guide/format-string-formatters.md
kirkrodrigues Nov 28, 2024
8d7d10a
Refactor user-guide/log-level-filter.md
kirkrodrigues Nov 28, 2024
59f3e22
kirk review
davemarco Nov 28, 2024
42e1f43
fix broken link
davemarco Nov 28, 2024
fd19fa2
fix more broken things..
davemarco Nov 28, 2024
c2e92cf
more
davemarco Nov 28, 2024
fbd51cd
Fix typo.
kirkrodrigues Nov 28, 2024
e18af3c
Add images to git LFS.
kirkrodrigues Nov 28, 2024
68ab84e
Fix typo.
kirkrodrigues Nov 28, 2024
6ca3093
Add some commas for clarity.
kirkrodrigues Nov 28, 2024
163d2c6
Revert "Add images to git LFS."
kirkrodrigues Nov 28, 2024
2c9d621
Re-add images as normal files.
kirkrodrigues Nov 28, 2024
2e774dc
Retry adding images as normal files.
kirkrodrigues Nov 28, 2024
a0742a6
Retry adding images as normal files.
kirkrodrigues Nov 28, 2024
893820e
Remove images.
kirkrodrigues Nov 28, 2024
04442ef
Update docs/src/user-guide/log-level-filtering.md
davemarco Nov 28, 2024
2efe385
Update docs/src/user-guide/log-level-filtering.md
davemarco Nov 28, 2024
e60e376
add lfs
davemarco Nov 28, 2024
32605d2
Update docs/src/user-guide/format-struct-logs-syntax.md
davemarco Nov 28, 2024
7d56440
Update docs/src/user-guide/format-struct-logs-overview.md
davemarco Nov 28, 2024
6c0359b
Add nested field to structured log formatting example; Replace curly …
kirkrodrigues Nov 28, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions docs/src/user-guide/.gitattributes
kirkrodrigues marked this conversation as resolved.
Show resolved Hide resolved
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
*.png filter=lfs diff=lfs merge=lfs -text
38 changes: 38 additions & 0 deletions docs/src/user-guide/format-struct-logs-formatters.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,38 @@
# Formatters

Formatters allow you to transform how field values are displayed. Below we describe all available
formatters and their options.

## Timestamp formatter

Converts millisecond Unix timestamps to human-readable date-time strings.

### Usage
```
{field:timestamp[:options]}
```

### Options
A [Day.js format string](https://day.js.org/docs/en/display/format) for the date and time.

**Default:** `YYYY-MM-DDTHH:mm:ssZ` (ISO 8601)

### Examples
Assuming the field `ts` is `1732703400000`:

* `{ts:timestamp}` → `2024-11-27T10:30:00Z`
* `{ts:timestamp:YYYY-MM-DD}` → `2024-11-27`

## Round formatter

Rounds a numeric value to the nearest integer.

### Usage
```
{field:round}
```

### Examples
Assuming the field `value` is `5.7`:

* `{value:round}` → `6`
37 changes: 37 additions & 0 deletions docs/src/user-guide/format-struct-logs-overview.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,37 @@
# Overview

The log viewer can format structured (e.g. JSON) logs as plain text using a format string. The
format string allows you to select which fields to include and how they should be formatted. You can
configure the format string through the settings ({fas}`gear`) dialog.

For example, for a JSON log with many fields:

```json
{
"ts": 1732733160216,
"level": "ERROR",
"message": "Failed to process payment",
davemarco marked this conversation as resolved.
Show resolved Hide resolved
"trace_id": "abc123def456",
"span_id": "span_789xyz",
"service": "payment-service",
"environment": "production"
}
```

You might want to show only:
* `timestamp`
* `level`
* `message`

You can achieve this with the following format string `{ts:timestamp} {level} {message}`.
`{ts:timestamp}` formats the timestamp field (Unix epoch) as an ISO 8601 date format while `{level}`
and `{message}` display the specified fields as is.

The formatted log would appear as:
```
2024-11-27T18:46:00Z ERROR Failed to process payment.
```

For reference docs, see:
* [Format string syntax](format-struct-logs-syntax)
* [Formatters](format-struct-logs-formatters)
95 changes: 95 additions & 0 deletions docs/src/user-guide/format-struct-logs-syntax.md
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

while not strictly necessary in this PR, shall we include an example for picking fields from nest objects?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is mentioned in the syntax page, but maybe could include in the longer json example. @kirkrodrigues

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

Original file line number Diff line number Diff line change
@@ -0,0 +1,95 @@
# Format string syntax

Each format string is composed of:
* [Static text](#static-text)
* [Field placeholders](#field-placeholders)
* [An implicit trailing newline](#implicit-trailing-newline)

## Static text
Static text may contain any character, except the following characters must be escaped with a
backslash:
* `{`
* `}`
* `\`

## Field placeholders
Each field placeholder is enclosed in braces (`{` and `}`) and has the following form, consisting of
three components:
```
{<field-name>[:<formatter-name>[:<formatter-options>]]}
```

### field-name (required)
Defines the key of the field whose value should replace the placeholder.

* Nested fields can be specified using periods (`.`) to denote hierarchy.
* E.g., the field `{"a:" {"b": 0}}` may be denoted by `a.b`.
* Field names can contain any character, except the following characters must be escaped with a
backslash:
* `.`
* `$`
* `{`
* `}`
* `:`
* `\`

### formatter-name (optional)
The name of the formatter to apply to the value before inserting it into the string.

* Formatter names can contain any character except a space (` `), and the following characters must
be escaped with a backslash (`\`):
* `{`
* `}`
* `:`
* `\`

### `formatter-options` (optional)
davemarco marked this conversation as resolved.
Show resolved Hide resolved
Defines any options for the formatter denoted by `formatter-name`.

* Formatter options can contain any character, except the following characters must be escaped with
a backslash:
* `{`
* `}`
* `:`
* `\`

:::{note}
`formatter-options` can only be specified if `formatter-name` was specified.
:::

## Implicit trailing newline

Every format string contains an implicit trailing newline so that each formatted log event ends with
a newline.

## Examples
Consider the following log event:
```
{
"@timestamp": 1427153388942,
"level": "INFO",
"thread": 0,
"latency_secs": 56.4,
"an.odd.key{name}": "org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from hadoop-metrics2.properties"
}
```

We can format this using the following YScope format string:

```
{@timestamp:timestamp:YYYY-MM-DD HH\:MM\:ss.SSS} {level} \{{thread}\} latency={latency_secs:round} {an\.odd\.key\{name\}}
```

* In the first placeholder, we have the field name `@timestamp`, a formatter called `timestamp`, and
the formatter’s options which are a date format string.
* The second and third placeholders simply stringify the values of the given fields.
* The fourth placeholder uses the `round` formatter to round the field’s value; this placeholder
doesn’t specify any formatter options, so the defaults will be used.
* The fifth placeholder is for a field whose name contains characters that require escaping.

The formatted string will be:
```
2015-03-23 19:29:48.942 INFO {0} latency=56 org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from hadoop-metrics2.properties
```

For a list of currently supported formatters, see [Formatters](format-struct-logs-formatters).
49 changes: 48 additions & 1 deletion docs/src/user-guide/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,4 +3,51 @@
This guide documents how to use and operate the YScope Log Viewer. Use the left sidebar (if it's
hidden, click the {fas}`bars` icon) to navigate to specific docs.

More details will be added soon.
::::{grid} 1 1 2 2
:gutter: 2

:::{grid-item-card}
:link: quick-start
Quick start
^^^
A guide to viewing log files in the log viewer.
:::

:::{grid-item-card}
:link: format-struct-logs-overview
Formatting structured logs
^^^
A guide to formatting structured (e.g. JSON) logs as plain text.
:::

:::{grid-item-card}
:link: log-level-filtering
Log level filtering
^^^
A guide to filtering log events by their levels.
:::
::::

:::{toctree}
:hidden:
:caption: Quick start

quick-start
:::

:::{toctree}
:hidden:
:caption: Formatting structured logs
:glob:

format-struct-logs-overview
format-struct-logs-syntax
format-struct-logs-formatters
:::

:::{toctree}
:hidden:
:caption: Log level filtering

log-level-filtering
:::
3 changes: 3 additions & 0 deletions docs/src/user-guide/individual-filter.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
15 changes: 15 additions & 0 deletions docs/src/user-guide/log-level-filtering.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
# Overview

The log viewer offers two ways to filter logs by their level.

## Severity-based filtering
You can filter for log events at or above a selected level by clicking the level's name. For
example, selecting INFO shows INFO, WARN, ERROR, and FATAL logs.

![Severity filtering](severity-filter.png)

## Individual level filtering
You can select specific log levels to show or hide by checking or unchecking the checkbox next to
each level's name.

![Individual level filtering](individual-filter.png)
15 changes: 15 additions & 0 deletions docs/src/user-guide/quick-start.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
# Viewing logs

The log viewer can open local or remote log files. The viewer currently supports viewing [CLP] IR
and [JSONL] log files.

## Opening local log files
To open a local file, click the folder icon ({far}`folder`) in the top-left corner. This will open a file
browser where you can navigate to and select the log file to open.

## Opening remote log files
To open a remote file, append `/?filePath=<FILE_URL>` to the current URL, replacing `<FILE_URL>`
with the URL of your log file.

[CLP]: https://github.com/y-scope/clp
[JSONL]: https://jsonlines.org/
3 changes: 3 additions & 0 deletions docs/src/user-guide/severity-filter.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.