Skip to content

Latest commit

 

History

History
220 lines (161 loc) · 8.89 KB

Extending-Exporter.md

File metadata and controls

220 lines (161 loc) · 8.89 KB
Raw

Create a new Exporter

Introduction

This is a step-by-step tutorial to create a new exporter. In this guide, we are going to create a SQLServerExporter that will export the data to a SQL Server database. You can follow these instructions to create and use your custom exporter. All you need to do is to change the core logic of the module to fit your needs.

Overview

The service allows you to export your data in different ways, depending on your needs. You can use the IFileExporter interface to implement a file exporter, which can save your data as a file in various formats, such as csv, json, etc.

You can use the ISQLExporter interface to implement a sql database exporter, which can store your data in an sql database, such as MSSQL, PostGre or MySQL.

If you have a different scenario that is not covered by these interfaces, you can design your own exporter interface.

The tutorial is organized in sequential steps that can be outlined as follows:

  1. Defining your exporter
  2. Implementing the ExportToDatabase method
  3. Registering the Exporter
  4. Using the Exporter from the Worker

The second part of the tutorial is organized as follows:

  1. Create the INoSQLExporter Interface
  2. Create a new command line argument
  3. Create your custom exporter

Tip

We recommend that you read this tutorial in its entirety before actually beginning with the procedure. Make sure to have understood everything in advance: this could save you some time and troubles later. This procedure is rather foulproof (we hope), but you can never know.

Step by Step Tutorial

Step 1: Defining your exporter

To create a new exporter, navigate to the Exporters folder and create a new SQLServerExporter class, and implement the ISQLExporter interface.

public sealed class SQLServerExporter : ISQLExporter 
{
}

Next, you must configure the connection string for the database. In our case we are going to read the connection string from the configuration file.

public sealed class SQLServerExporter : ISQLExporter
{
    private readonly string _connectionString = string.Empty;
    public SQLServerExporter(IConfiguration configuration)
    {
        _connectionString = configuration.GetSection("SqlServer").GetValue<string>("ConnectionString") ?? string.Empty;
    }
}

Step 2: Implementing the ExportToDatabase method

The ISQLExporter interface defines an ExportToDatabase method. This method is used to write the generated models to the database. The method accepts an IEnumerable<InteractionDto> as input. This IEnumerable contains all the models, generated by the service.

In this method, you can implement your own logic on how to store the data in your database.

For the tutorial purposes, we are going to store each model in a different table.

Step 2.1: Storing individual models

To store individual models in their own tables, we are going to create a new Export generic method. This method will accept an IEnumerable<T> which will contain all records of a specific model.

private void Export<T>(IEnumerable<T>? records) where T : IModel?
{
}

Then we will implement the logic. The method will open a connection to the database and BulkMerge the records. Then it will close the connection.

    private void Export<T>(IEnumerable<T>? records) where T : IModel?
    {
        try
        {
            using var connection = new SqlConnection(_connectionString);
            connection.Open();
            connection.BulkMerge(records);
            connection.Close();
        }
        catch (Exception ex)
        {
            Log.Error(ex, "Error.");
        }
    }

Note

The Export method will select the table to store the records based on the model's annotations. To see how these annotations are defined, see here.

Step 2.2: Exporting the Data

As we mentioned before, the ExportToDatabase method should contain your core logic for storing data to the database. In our case, we are going to call the Export method to store the models to their respective tables.

public void ExportToDatabase(IEnumerable<InteractionDto> records)
{
    var (interactionModels, campaignModels, deviceModels, downloadModels, geoNetworkModels,
        goalModels, outcomeModels, pageViewModels, searchModels) = InteractionDtoUtils.SplitEntities(records);

    var (goalDefinitionModels, campaignDefinitionModels,
        outcomeDefinitionModels, eventDefinitionModels) = InteractionDtoUtils.RetrieveDefinitions();
    var channels = InteractionDtoUtils.GetTaxonModels(records);

    Export(channels);
    Export(goalDefinitionModels);
    Export(campaignDefinitionModels);
    Export(outcomeDefinitionModels);
    Export(eventDefinitionModels);
    Export(interactionModels);
    Export(campaignModels);
    Export(deviceModels);
    Export(downloadModels);
    Export(geoNetworkModels);
    Export(goalModels);
    Export(outcomeModels);
    Export(pageViewModels);
    Export(searchModels);
}

Step 3: Registering the Exporter

After you have completed the exporter's core logic, you need to register it in the service's IHostBuilder. To do that, navigate to the Program.cs and add your exporter under services.

services.AddSingleton<ISQLExporter, SQLServerExporter>();

Important

You can register only a single exporter per interface. If you want to register multiple exporters from the same interface, use class only registration.

Step 4: Using the Exporter from the Worker

Finally, you need to add your exporter to the background service. To do that, open the Worker.cs file and add your exporter as a property.

public class Worker : BackgroundService
{
    private readonly ISQLExporter _sqlExporter;
}

Then add a call to your exporter in the ExecuteAsync method.

protected override async Task ExecuteAsync(CancellationToken stoppingToken)
{
    if (_commandArgs.Args.Contains(CommandLineArgs.CURRENT_DATA_ARGUMENT) && !_commandArgs.Args.Contains(CommandLineArgs.HISTORICAL_DATA_ARGUMENT)){

        // Add here to get the current data
        if (_commandArgs.Args.Contains(CommandLineArgs.SQL_ARGUMENT))
        {
            _sqlExporter.ExportToDatabase(dtos);
        }
    }

    else if (_commandArgs.Args.Contains(CommandLineArgs.HISTORICAL_DATA_ARGUMENT) && !_commandArgs.Args.Contains(CommandLineArgs.CURRENT_DATA_ARGUMENT))
    {
        // Add here to get the historical data
        if (_commandArgs.Args.Contains(CommandLineArgs.SQL_ARGUMENT))
        {
            _sqlExporter.ExportToDatabase(dtos);
        }
    }
}

Tip

The if clause above allows the worker to decide whether or not to export the data to SQL. This argument is set when you register the service. For more information refer to the Register the Service section of the Installation Guide.

After doing that, you need to rebuild and register the application using the installation scripts. For a quick overview of the actions available, refer to the Getting Started guide.

Design Your Own Exporter Interface

???

Step 1: Create the INoSQLExporter interface

Navigate to the Interfaces folder, under the xDB Analytics Extractor project, and create the INoSQLExporter interface, and define your export method.

public interface INoSQLExporter 
{
    void ExportToDatabase(IEnumerable<InteractionDto> records);
}

Tip

While you can accept any argument in your method, we highly recommend to only have the IEnumerable<InteractionDto> records argument, and delegate any custom logic to other functions.

Step 2: Create a new command line argument

Next, navigate to the CommandLineArgs.cs file, under the xDB Analytics Extractor project, and add a new argument for the NoSQL exporter. These command line arguments are used by the worker to decide which exporters it will use on runtime.

public const string NO_SQL_ARGUMENT = "-nosql";

Step 3: Create your custom exporter

After you have created the INoSQLExporter interface, you can define your own exporter for NoSQL databases. To do that follow the above tutorial.

Important

Since this is a custom exporter you need to define your own if clauses for the command line argument you defined. For this example, you must add the following if clause:

if (_commandArgs.Args.Contains(CommandLineArgs.NO_SQL_ARGUMENT))
{
      // Call your exporter here
}