This project illustrates how to configure and deploy a copy replication function that moves data between from an Azure Event Hubs into a Servcie Bus entity without you having to write or modify any code.
It is assumed that you are familiar with Event Hubs and know how to create them either through Azure Portal, Azure CLI, Azure PowerShell, or ARM Templates.
It is furthermore assumed that you have read the Event Hubs Federation guidance ( Overview, Functions, Patterns), and have a scenario for your replication project in mind.
The replication and merge patterns can be readily implemented with this project.
To configure and deploy this project, you need the following components installed:
For configuring and running this replication function, you need two Event Hubs, either inside the same namespace or in different namespaces that may be located in different Azure regions.
We will refer to the Event Hubs from which events are to be replicated as the source and the Event Hub into which messages are replicated the target.
On the source Event Hub, you will need to create a dedicated consumer group for the replication function that will allow the functions to manage scaling and keep track of its progress on the source event hub.
The following diagram shows an exemplary topology with a suggested convention for naming the various elements. Here, the replication function name reflects the name of the Event Hub it mirrors from source to target, and the consumer group name on the source reflects the replication app and function name.
If the Event Hubs have different names, you might reflect that in the function
name as source_target. The naming is ultimately up to you and will follow
whatever conventions you define for your scenario.
Source Event Hub Replication App Target Service Bus
+-----------------------------------------------------+ +-------------------------+ +-----------------------+
| Namespace (West Europe) | | Function App | | Namespace (East US 2) |
| "example-eh-weu" | | "repl-example-weu-eus2" | | "example-sb-eus2" |
| | | | | |
| +-------------+ | | +-------------+ | | +---------------+ |
| +-----------------------------------+ | Replication | | |
| Event Hub | Consumer Group | | Function | | Service Bus |
| | "repl-example-weu-eus2.telemetry" |---->-----| |----->-----| |
| "telemetry" | | | "telemetry" | | "telemetry" |
| +-----------------------------------+ | | | |
+-------------+ +-------------+ +---------------+Leaning on the naming conventions of the exemplary topology, the project implements one replication task named "telemetry" that performs the copy.
The task is defined in the a 'function.json' configuration file that resides in the 'telemetry' folder, corresponding to the name of the function.
For adding further tasks to the replication application, create a new folder for each task and place a 'function.json' file into it. The options for the configuration files are explained in the product documentation.
{
"configurationSource": "config",
"bindings" : [
{
"direction": "in",
"type": "eventHubTrigger",
"connection": "telemetry-source-connection",
"eventHubName": "telemetry",
"consumerGroup": "%telemetry-source-consumergroup%",
"name": "input"
},
{
"direction": "out",
"type": "serviceBus",
"connection": "telemetry-target-connection",
"queueName": "telemetry",
"name": "output"
}
],
"retry": {
"strategy": "exponentialBackoff",
"maxRetryCount": -1,
"minimumInterval": "00:00:05",
"maximumInterval": "00:05:00"
},
"disabled": false,
"scriptFile": "../bin/Azure.Messaging.Replication.dll",
"entryPoint": "Azure.Messaging.Replication.EventHubReplicationTasks.ForwardToServiceBus"
}The connection values refer to the name of configuration entries in the
application settings of the Functions application. The setup step below
explains how to set those.
Refer to the ServiceBusCopy and EventHubCopy documents for how to set up the Event Hubs side and the Service Bus side of the task.
To learn how you can monitor your replication app, please refer to the monitoring section of the Azure Functions documentation.
A particularly useful visual tool for monitoring replication tasks is the Application Insights Application Map, which is automatically generated from the captured monitoring information and allows exploring the reliability and performance of the replication task sosurce and target transfers.
For immediate diagnostic insights, you can work with the Live Metrics portal tool, which provides low latency visualization of log details.