Skip to content

Commit

Permalink
feat(formatting): add 4 different JSON formatting types
Browse files Browse the repository at this point in the history
Add support for the formatting types:

- FormattingType.NormalRendered
- FormattingType.Normal
- FormattingType.CompactRendered
- FormattingType.Compact

The formatting type can be configured via Options and DurableOptions.
  • Loading branch information
FantasticFiasco authored Mar 12, 2017
1 parent 44d27b2 commit abfb1c6
Show file tree
Hide file tree
Showing 31 changed files with 957 additions and 121 deletions.
4 changes: 4 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,10 @@ This project adheres to [Semantic Versioning](http://semver.org/) and is followi

## Unreleased

### Added

- Support for the formatting types: `FormattingType.NormalRendered`, `FormattingType.Normal`, `FormattingType.CompactRendered` and `FormattingType.Compact`. The formatting type can be configured via `Options` and `DurableOptions`.

## 3.0.0 2017-03-04

### Added
Expand Down
94 changes: 92 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@ The sink is batching multiple log events into a single request, and the followin
"events": [
{
"Timestamp": "2016-11-03T00:09:11.4899425+01:00",
"Level": "Debug",
"Level": "Information",
"MessageTemplate": "Logging {@Heartbeat} from {Computer}",
"RenderedMessage": "Logging { UserName: \"Mike\", UserDomainName: \"Home\" } from \"Workstation\"",
"Properties": {
Expand All @@ -38,7 +38,7 @@ The sink is batching multiple log events into a single request, and the followin
},
{
"Timestamp": "2016-11-03T00:09:12.4905685+01:00",
"Level": "Debug",
"Level": "Information",
"MessageTemplate": "Logging {@Heartbeat} from {Computer}",
"RenderedMessage": "Logging { UserName: \"Mike\", UserDomainName: \"Home\" } from \"Workstation\"",
"Properties": {
Expand Down Expand Up @@ -71,6 +71,96 @@ The log events can be sent directly to Elasticsearch using [Serilog.Sinks.Elasti

If you would like to send the log events to Logstash for further processing instead of sending them directly to Elasticsearch, this sink in combination with the [Logstash HTTP input plugin](https://www.elastic.co/blog/introducing-logstash-input-http-plugin) is the perfect match for you. It is a much better solution than having to install [Filebeat](https://www.elastic.co/products/beats/filebeat) on all your instances, mainly because it involves fewer moving parts.

### Formatting types

#### FormattingType.NormalRendered

The log event is normally formatted and the message template is rendered into a message. This is the most verbose formatting type and its network load is higher than the other options.

Example:
```json
{
"Timestamp": "2016-11-03T00:09:11.4899425+01:00",
"Level": "Information",
"MessageTemplate": "Logging {@Heartbeat} from {Computer}",
"RenderedMessage": "Logging { UserName: \"Mike\", UserDomainName: \"Home\" } from \"Workstation\"",
"Properties": {
"Heartbeat": {
"UserName": "Mike",
"UserDomainName": "Home"
},
"Computer": "Workstation"
}
}
```

#### FormattingType.Normal

The log event is normally formatted and its data normalized. The lack of a rendered message means improved network load compared to `FormattingType.NormalRendered`. Often this formatting type is complemented with a log server that is capable of rendering the messages of the incoming log events.

Example:
```json
{
"Timestamp": "2016-11-03T00:09:11.4899425+01:00",
"Level": "Information",
"MessageTemplate": "Logging {@Heartbeat} from {Computer}",
"Properties": {
"Heartbeat": {
"UserName": "Mike",
"UserDomainName": "Home"
},
"Computer": "Workstation"
}
}
```

#### FormattingType.CompactRendered

The log event is formatted with minimizing size as a priority but still render the message template into a message. This formatting type greatly reduce the network load and should be used in situations where bandwidth is of importance.

The compact formatter adheres to the following rules:

- Built-in field names are short and prefixed with an `@`
- The `Properties` property is flattened
- The Information level is omitted since it is considered to be the default

Example:
```json
{
"@t": "2016-11-03T00:09:11.4899425+01:00",
"@mt": "Logging {@Heartbeat} from {Computer}",
"@m":"Logging { UserName: \"Mike\", UserDomainName: \"Home\" } from \"Workstation\"",
"Heartbeat": {
"UserName": "Mike",
"UserDomainName": "Home"
},
"Computer": "Workstation"
}
```

#### FormattingType.Compact

The log event is formatted with minimizing size as a priority and its data is normalized. The lack of a rendered message means even smaller network load compared to `FormattingType.CompactRendered` and should be used in situations where bandwidth is of importance. Often this formatting type is complemented with a log server that is capable of rendering the messages of the incoming log events.

The compact formatter adheres to the following rules:

- Built-in field names are short and prefixed with an `@`
- The `Properties` property is flattened
- The Information level is omitted since it is considered to be the default

Example:
```json
{
"@t": "2016-11-03T00:09:11.4899425+01:00",
"@mt": "Logging {@Heartbeat} from {Computer}",
"Heartbeat": {
"UserName": "Mike",
"UserDomainName": "Home"
},
"Computer": "Workstation"
}
```

### Install via NuGet

If you want to include the HTTP POST sink in your project, you can [install it directly from NuGet](https://www.nuget.org/packages/Serilog.Sinks.Http/).
Expand Down
3 changes: 2 additions & 1 deletion src/Serilog.Sinks.Http/LoggerSinkConfigurationExtensions.cs
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,8 @@
using Serilog.Configuration;
using Serilog.Events;
using Serilog.Sinks.Http;
using Serilog.Sinks.Http.Private;
using Serilog.Sinks.Http.Private.Http;
using Serilog.Sinks.Http.Private.Sinks;

namespace Serilog
{
Expand Down
53 changes: 53 additions & 0 deletions src/Serilog.Sinks.Http/Sinks/Http/FormattingType.cs
Original file line number Diff line number Diff line change
@@ -0,0 +1,53 @@
// Copyright 2015-2016 Serilog Contributors
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.

namespace Serilog.Sinks.Http
{
/// <summary>
/// Enum defining how log events are formatted when sent over the network.
/// </summary>
public enum FormattingType
{
/// <summary>
/// The log event is normally formatted and the message template is rendered into a message.
/// This is the most verbose formatting type and its network load is higher than the other
/// options.
/// </summary>
NormalRendered,

/// <summary>
/// The log event is normally formatted and its data normalized. The lack of a rendered message
/// means improved network load compared to <see cref="NormalRendered"/>. Often this formatting
/// type is complemented with a log server that is capable of rendering the messages of the
/// incoming log events.
/// </summary>
Normal,

/// <summary>
/// The log event is formatted with minimizing size as a priority but still render the message
/// template into a message. This formatting type greatly reduce the network load and should be
/// used in situations where bandwidth is of importance.
/// </summary>
CompactRendered,

/// <summary>
/// The log event is formatted with minimizing size as a priority and its data is normalized. The
/// lack of a rendered message means even smaller network load compared to
/// <see cref="CompactRendered"/> and should be used in situations where bandwidth is of
/// importance. Often this formatting type is complemented with a log server that is capable of
/// rendering the messages of the incoming log events.
/// </summary>
Compact
}
}
6 changes: 6 additions & 0 deletions src/Serilog.Sinks.Http/Sinks/Http/Options.cs
Original file line number Diff line number Diff line change
Expand Up @@ -39,5 +39,11 @@ public class Options
/// Default value is 265 KB.
/// </summary>
public long? EventBodyLimitBytes { get; set; } = 256 * 1024;

/// <summary>
/// Gets or sets the formatting type. Default value is
/// <see cref="Http.FormattingType.NormalRendered"/>.
/// </summary>
public FormattingType FormattingType { get; set; } = FormattingType.NormalRendered;
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,134 @@
// Copyright 2015-2016 Serilog Contributors
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.

using System;
using System.IO;
using System.Linq;
using Serilog.Debugging;
using Serilog.Events;
using Serilog.Formatting;
using Serilog.Formatting.Json;
using Serilog.Parsing;

namespace Serilog.Sinks.Http.Private.Formatters
{
internal class CompactJsonFormatter : ITextFormatter
{
private static readonly JsonValueFormatter ValueFormatter = new JsonValueFormatter();

private readonly bool isRenderingMessage;

public CompactJsonFormatter(bool isRenderingMessage)
{
this.isRenderingMessage = isRenderingMessage;
}

public void Format(LogEvent logEvent, TextWriter output)
{
try
{
var buffer = new StringWriter();
FormatContent(logEvent, buffer);

// If formatting was successful, write to output
output.WriteLine(buffer.ToString());
}
catch (Exception e)
{
LogNonFormattableEvent(logEvent, e);
}
}

private void FormatContent(LogEvent logEvent, TextWriter output)
{
if (logEvent == null)
throw new ArgumentNullException(nameof(logEvent));
if (output == null)
throw new ArgumentNullException(nameof(output));

output.Write("{\"@t\":\"");
output.Write(logEvent.Timestamp.UtcDateTime.ToString("o"));

output.Write("\",\"@mt\":");
JsonValueFormatter.WriteQuotedJsonString(logEvent.MessageTemplate.Text, output);

if (isRenderingMessage)
{
output.Write(",\"@m\":");
var message = logEvent.MessageTemplate.Render(logEvent.Properties);
JsonValueFormatter.WriteQuotedJsonString(message, output);
}

var tokensWithFormat = logEvent.MessageTemplate.Tokens
.OfType<PropertyToken>()
.Where(pt => pt.Format != null);

// Better not to allocate an array in the 99.9% of cases where this is false
// ReSharper disable once PossibleMultipleEnumeration
if (tokensWithFormat.Any())
{
output.Write(",\"@r\":[");
var delim = "";
foreach (var r in tokensWithFormat)
{
output.Write(delim);
delim = ",";
var space = new StringWriter();
r.Render(logEvent.Properties, space);
JsonValueFormatter.WriteQuotedJsonString(space.ToString(), output);
}
output.Write(']');
}

if (logEvent.Level != LogEventLevel.Information)
{
output.Write(",\"@l\":\"");
output.Write(logEvent.Level);
output.Write('\"');
}

if (logEvent.Exception != null)
{
output.Write(",\"@x\":");
JsonValueFormatter.WriteQuotedJsonString(logEvent.Exception.ToString(), output);
}

foreach (var property in logEvent.Properties)
{
var name = property.Key;
if (name.Length > 0 && name[0] == '@')
{
// Escape first '@' by doubling
name = '@' + name;
}

output.Write(',');
JsonValueFormatter.WriteQuotedJsonString(name, output);
output.Write(':');
ValueFormatter.Format(property.Value, output);
}

output.Write('}');
}

private static void LogNonFormattableEvent(LogEvent logEvent, Exception e)
{
SelfLog.WriteLine(
"Event at {0} with message template {1} could not be formatted into JSON and will be dropped: {2}",
logEvent.Timestamp.ToString("o"),
logEvent.MessageTemplate.Text,
e);
}
}
}
43 changes: 43 additions & 0 deletions src/Serilog.Sinks.Http/Sinks/Http/Private/Formatters/Converter.cs
Original file line number Diff line number Diff line change
@@ -0,0 +1,43 @@
// Copyright 2015-2016 Serilog Contributors
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.

using System;
using Serilog.Formatting;

namespace Serilog.Sinks.Http.Private.Formatters
{
internal static class Converter
{
public static ITextFormatter ToFormatter(FormattingType formattingType)
{
switch (formattingType)
{
case FormattingType.NormalRendered:
return new NormalJsonFormatter(true);

case FormattingType.Normal:
return new NormalJsonFormatter(false);

case FormattingType.CompactRendered:
return new CompactJsonFormatter(true);

case FormattingType.Compact:
return new CompactJsonFormatter(false);

default:
throw new ArgumentException($"Formatting type {formattingType} is not supported");
}
}
}
}
Loading

0 comments on commit abfb1c6

Please sign in to comment.