Skip to content

Logging

Jump to bottom
moattarwork edited this page Dec 14, 2022 · 1 revision

Logging is always tricky to configure. It's even more tricky to make logs consistent across different apps and when it comes to integrating with log aggregators and capturing the same set of metrics your head might start spinning, Well we have a solution for all of it! One liner log configuration:

    public class Program
    {
        public static void Main(string[] args)
        {
            HostAsWeb.Run<Startup>(s =>  s.ConfigureLogger<Startup>(s => UseLoggly(...).UseSeq(...));
        }
    }

Ok, don't get too excited, this has no integration with any of the cloud providers yet, but it does define a consistent file format, as well as a consistent set of variables that are persisted to the logs. Currently default configuration saves logs to: $approot$\logs in DEV environment and C:\logs all other environments. Log path may be optionally be provided in either *.json or *.{ENV}.json files, if none of the configured defaults work for your scenario.

How to provide an environment

The environment is coming from appsettings.json which from Application:Environment. The values are user-defined but to be in-line with .NET Core environments they can be (Development, Integration, UAT, Staging, and Production).

Notes: If the value is not explicitly provided in the Application section of appsettings.json it will be provided by the environment from ASPNETCORE_ENVIRONMENT variable.

The pattern we use for log file is: "{Timestamp:yyyy-MM-dd HH:mm:ss.fff zzz} [{MachineName}] [{EnvironmentUserName}] [{ProcessId}] [{UserName}] [{CorrelationId}] [{ThreadId}] [{Level}] {Message}{NewLine}{Exception}"

To write logs we use the native ASP.NET Core logging framework, to get a logger instance, we need to inject it into any component we want to write some logs from:

public class MyService
{
    private readonly ILogger<MyService> _log;

    public MyService(ILogger<MyService> log)
    {
        _log = log;
    }
    public void DoWork()
    {
        _log.LogInformation("I am doing some great work here:)");
    }
}

An underlying logging framework is Serilog, both native logger and Serilog support logging structured events. Please read more about this at https://serilog.net/.

There is one setting we are of interest in the config for basic logging and that is the default logging level which is controlled by manipulating MinimumLevel value and can be found here:

{
  "Logging": {
    "IncludeScopes": false,
    "LogFilePath": "Absolution path for the log files",
    "Serilog": {
      "MinimumLevel": "Verbose"
    }
  }
}

If you are interested in IncludeScopes setting, you may want to refer to ASP.NET Core Logging Docs for an in-depth explanation.

Extra Features

We have a couple of notable features that come built-in with the framework. For example, we can check the latest fifty logs and the last five errors by going to Http(s)://{API-URL}/diagnostics/logs. Here is an example of what you may see:

{
  "Status": "Success",
  "Service": {},
  "Runtime": {
    "Pid": 61020,
    "Process": "dotnet",
    "Release": "C:\\Program Files\\dotnet\\dotnet.exe",
    "Version": "1.0.0",
    "UpTime": "9.23s",
    "Memory": "108,456.00 MB",
    "Cwd": "c:\\Work\\Template.DotNetCore.WebAPI\\Template.WebAPI"
  },
  "Host": {
    "Hostname": "HostName",
    "OS": "Microsoft Windows 10.0.15063 ",
    "Arch": 1
  },
  "Logs": {
    "LatestErrors": [],
    "Messages": [
      {
        "LoggedAt": "2017-05-22T14:10:35.9872852+01:00",
        "Level": "Information",
        "Message": "2017-05-22 14:10:35.987 +01:00 [Information] Request finished in 256.2077ms 200 \r\n"
      }
    ]
  }
}

Another important feature is the ability to change the level of logging for the API on the fly. To achieve this, you need to POST a message with the desired logging level to Http(s)://{API-URL}/diagnostics/logs endpoint. Available logging levels are:

  • Verbose,
  • Debug,
  • Information,
  • Warning,
  • Error,
  • Fatal

Here is the POST message example:

{
  "logginglevel": "Debug"
}

Logging Action Filters

The solution comes preconfigured with an ActionFilter which intercepts every call to ANY app controller and logs entering/exiting events with a list of arguments passed to the action. It is configured to output messages at DEBUG and more detailed messages at TRACE logging levels. Great little addition in case we need to troubleshoot our APIs.

Currently, we support the following log aggregators:

  • Seq - On-prem solution for log aggregation (Can be hosted on a VM in the cloud)
  • Loggly - Cloud-based log aggregator / analytics platform
  • LogEntries - Cloud-based log aggregator / analytics platform / super-fast

Seq

Prerequisite for Seq is obviously the installation of the actual server, but after that is complete, the rest is super simple. Actually, it's as simple as modifying your log configuration as shown below:

    public class Program
    {
        public static void Main(string[] args)
        {
            HostAsWeb.Run<Startup>(s => s.ConfigureLogger<Startup>(c =>
                c.UseSeq("https://seq.server.com").WithApiKey("api-key")));
        }
    }

This will allow you to use Seq in its most basic configuration, given that you have provided the correct server URL and working API-KEY.

There is a useful feature that Serilog gives us, which is runtime log level manipulation. Luckily Seq takes advantage of it, so as we! If you wish to enable this use the following code snippet:

    public class Program
    {
        public static void Main(string[] args)
        {
            HostAsWeb.Run<Startup>(s => s.ConfigureLogger<Startup>(c =>
                c.UseSeq("https://seq.server.com")
                .WithApiKey("api-key")
                .EnableLogLevelControl()));
        }
    }

This last setting allows us to change the logging level on the fly from Seq admin UI. Awesome! By now you must be questioning these HARDCODED settings approach and you are completely within your rights to do so, which is why we have an alternative option of loading all those settings from either *.json or *.{ENV}.json files. To achieve that we need to do two things:

  • Configure our logger to use Seq configuration
    public class Program
    {
        public static void Main(string[] args)
        {
            HostAsWeb.Run<Startup>(s => s.ConfigureLogger<Startup>(c =>
                c.UseSeq(s.Configuration.GetSection("Logging:Seq"))));
        }
    }
  • Add Seq configuration to your Logging config section
{
  "Logging": {
    "IncludeScopes": false,
    "Serilog": {
      "MinimumLevel": "Verbose"
    },
    "Seq": {
      "ServerUrl": "http://seq.server.com",
      "ApiKey": "API-KEY",
      "AllowLogLevelToBeControlledRemotely": true
    }
  }
}

Loggly

To use Loggly, we do need to create an account at https://www.loggly.com/. Once that's done, we will need to retrieve Loggly server URL and customer token. The simplest configuration looks like is shown below:

    public class Program
    {
        public static void Main(string[] args)
        {
            HostAsWeb.Run<Startup>(s => s.ConfigureLogger<Startup>(c => c.UseLoggly("https://logs-01.loggly.com/")
                .WithCustomerToken("CUSTOMER-TOKEN")
                .BufferLogsAt("logs\\logs-buffer")));
        }
    }

One of the more interesting parameters is BufferLogsAt, which expects you to specify either relative or absolute path for buffering logs and a pattern for a file name. For example, the current configuration that points to logs\logs-buffer will buffer logs in %APPLICATION_ROOT%\logs\logs-bufferXXXXX files. Loggly will roll these files and will clean up after all buffered logs are uploaded.

Loggly also supports runtime log level manipulation, which is one of the notable features of Serilog. To achieve this please use the following code snippet:

    public class Program
    {
        public static void Main(string[] args)
        {
            HostAsWeb.Run<Startup>(s => s.ConfigureLogger<Startup>(c => c.UseLoggly("https://logs-01.loggly.com/")
                .WithCustomerToken("CUSTOMER-TOKEN")
                .BufferLogsAt("logs\\logs-buffer")
                .EnableLogLevelControl()));
        }
    }

Now while this is great for quick and 'dirty' setup, it is most likekly we want to run different configurations for our app per environment. For this we can load all the settings from either *.json or *.{ENV}.json files. To achieve that we need to do two things:

  • Configure our logger to use Loggly configuration
    public class Program
    {
        public static void Main(string[] args)
        {
            HostAsWeb.Run<Startup>(s => s.ConfigureLogger<Startup>(
                c => c.UseLoggly(s.Configuration.GetSection("Logging:Loggly"))
            ));
        }
    }
  • Add Loggly configuration to your Logging config section
{
  "Logging": {
    "IncludeScopes": false,
    "Serilog": {
      "MinimumLevel": "Verbose"
    },
    "Loggly": {
      "ServerUrl": "https://logs-01.loggly.com/",
      "CustomerToken": "CUSTOMER-TOKEN",
      "BufferBaseFilename": "logs\\loggly-buffer",
      "AllowLogLevelToBeControlledRemotely": true
    }
  }
}

This is the minimum configuration that is required. For the omitted config values, defaults will be used. If you are wondering what other options are available, please take a look at the snippet below

{
  "Logging": {
    "IncludeScopes": false,
    "Serilog": {
      "MinimumLevel": "Verbose"
    },
    "Loggly": {
      "ServerUrl": "https://logs-01.loggly.com/",
      "CustomerToken": "CUSTOMER-TOKEN",
      "BufferBaseFilename": "logs\\loggly-buffer",
      "NumberOfEventsInSingleBatch": 50,
      "BatchPostingIntervalInSeconds": 2,
      "EventBodyLimitKb": 10,
      "RetainedInvalidPayloadsLimitMb": 100,
      "AllowLogLevelToBeControlledRemotely": true
    }
  }
}
Clone this wiki locally