Azure Log Analytics Integration

This project contains is a sample .Net Core console application packaged as a Docker container designed to continously process and parse log files from the 42Crunch API firewall (referred to here as guardian) and to push new log events to an Azure Log Analytics workspace table for further processing on Azure Sentinel.

In order to use the FW2LA container it is necessary to provide Log Analytics parameters and a location where the guardian log files are written — typically this location can be shared via a Docker volume bind mount or Kubernetes shared storage.

The FW2LA program processes logs from the API transaction log file and from the Unknown transaction log file and pushes selected fields to the Log Analytics table.

How the application works - step by step

The basic application logic is as follows:

• Process the environment population defaults and validating including setting defaults (an exception is thrown if workspace key and ID aren't provided, or if folder locations do not exist)
• Perform an initialization of the application state ie. is this the first time it has been run
  • if the FW2LA_STATE_FOLDER is empty then the log folder is processed and new state files are written (this will cause all log records to be written to Log Analytics)
  • if the state file for an instance in FW2LA_STATE_FOLDER exists this will be validated against the corresponding log file ie. size and MD5SUM. If there is a mismatch then state file in re-created.
• Start up a processing timer to periodically process the log files namely:
  • check if the state file size is different to the actual log file — if so then process the log file.
  • check in the state file if the last lines written is zero — if so then process the log file.
  • In processing the log file load the last line written and process to the file last line, writing records to Log Analytics
  • Update the state file to reflect the update file size, MD5 sum, the last line written.

Format of the state file

The state file maintains the specific state for a given guardian instance as shown:

{
  "API_ID": "0f632923-58de-4560-a43f-8f11bc422085",
  "Instance_Name": "Local-Firewall-dev1",
  "API_Log_File": {
    "FileName": "api-0f632923-58de-4560-a43f-8f11bc422085.transaction.log",
    "FileSize": 82663,
    "MD5Sum": "112791adfabd11b2acf75fd354243f6e",
    "LastLineSent": 27
  },
  "Unknown_Log_File": {
    "FileName": "api-unknown.transaction.log",
    "FileSize": 94506,
    "MD5Sum": "79cf06836713e7163d3a10717be30db2",
    "LastLineSent": 58
  }
}

In order to force a full rewrite of log records to Log Analytics either:

• Set the LastLineSent field to zero
• Delete the state file entirely

List of 'guardian' records processed

This first release of the application pushes the following information to Log Analytics:

• "uuid"
• "pod"["instance_name"]
• "date_epoch"
• "api"
• "api_name"
• "non_blocking_mode"
• "source_ip"
• "source_port"
• "destination_ip"
• "destination_port"
• "protocol"
• "hostname"
• "uri_path"
• "method"
• "status"
• "query"
• "params"["request_header"]
• "params"["response_header"]
• "errors"

Installation

Getting Azure Log Analytics parameters

The Azure Log Analytics workspace key and ID must be obtained directly from the relevant Azure Log Analytics workspace with details provided here.

Building the container

The Docker image can be built using the Dockerfile as follows:

docker build --tag myorg/fw2la .

A pre-built image is also available on DockerHub at: https://hub.docker.com/r/42crunch/42c-fw-2la

The container constructed is a vanilla .Net Core 3.1 Alpine image using a template generated by Visual Studio 2019. The only changes made include the creation of an /opt/guardian/logs folder and a .state folder in the application root.

Dependencies

Library Name	Version
CommandLineParser	2.8.0
LogAnalytics.Client	1.3.2
Microsoft.Extensions.Logging	5.0.0
Microsoft.Extensions.Logging.Console	5.0.0
Newtonsoft.Json	13.0.1

Running the container

The container can be run from the command line but the easiest way to encapsulate the parameters is using the Docker Compose file provided under the sample directory shown below:

version: '3.4'

services:
    logs-forwarder:
        image: myorg/fw2la
        build:
            context: .
            dockerfile: ./Dockerfile
        command: [""]
        volumes:
            - ./storage/firewall/logs/:/opt/guardian/logs
        environment:
            - FW2LA_WORKSPACE_ID=${FW2LA_WORKSPACE_ID}
            - FW2LA_WORKSPACE_KEY=${FW2LA_WORKSPACE_KEY}
            - FW2LA_LOGS_FOLDER=/opt/guardian/logs
            - FW2LA_TICK_INTERVAL=10

This assumes the workspace ID and key are set as environment variables and that a guardian instance is running in an adjacent folder and emitting logs to a relative path of ./storage/firewall/logs (note the bind mount mapping).

Detailed parameter usage

Parameter name	Description	Defaul value
FW2LA_WORKSPACE_ID	Log Analytics workspace ID	No default, mandatory
FW2LA_WORKSPACE_KEY	Log Analytics workspace key	No default, mandatory
FW2LA_TABLENAME	Log Analytics table name	'apifirewall_log_1'
FW2LA_LOGS_FOLDER	Path to the guardian log files base folder	Application directory ie. $(pwd)
FW2LA_STATE_FOLDER	Path to a storage location for application state files (cache)	Application directory + '.state' ie. $(pwd)/.state
FW2LA_TICK_INTERVAL	Interval in seconds between processing log files for changes	10 seconds
FW2LA_TRACE	If defined emits tracel level debugging messages	Undefined

The log file location must point to the base folder where the logs are written and must not include the GUARDIAN_INSTANCE_NAME path. For instance in the default configuration (described here) the path should be set to /opt/guardian/logs/ (omitting the instance name which will be discovered dynamically).

Debugging

The current build will emit relatively verbose logging at startup and upon processing the log file. Exceptions may be thrown and these will provide a good level of information (for example if a workspace key or ID is missing).

A common debugging technique is to run the container with a parameter tail -f /dev/null to keep the container alive and then to use the interactive mode with a Bash interpreter to check the environment i.e.:

• does the .state folder exist?
• is the log file location mounted correctly?
• are the log files being written, and are they being updated?

The application will also print usage and command line parameters in the --help command line parameter is provided i.e. in the Docker Compose file add the --help to the command array.

Roadmap

Please open a Github issue with your questions/suggestions and of course contribute to this project!