This project contains a set of samples that demonstrate how to create Azure Messaging replication tasks using the Azure Functions runtime with .NET Core as the exemplary implementation language. The guidance equivalently applies to all other Azure SDKs.
Replication tasks built following this guidance, using the supplied code or your own code, build on generally available capabilities of Azure Functions and are eligible for product support equivalent to any other Azure Function that you build, deploy, and run.
For using these tasks with Event Hubs, first read the Event Hubs Federation guidance (Overview, Functions, Patterns).
For Service Bus, read the Service Bus Federation Guidance (Overview, Functions, Patterns)
For stateful replication applications that need to consider relationships between events, create composite events, enrich events or reduce events, create data aggregations, and transform event payloads, Azure Stream Analytics is the best implementation option.
Many sophisticated solutions require the same event streams or the same content of queues to be made available for consumption in multiple locations and/or for event streams or queue contents to be collected in multiple locations and then consolidated into a specific locations for consumption.
Replicating event streams and contents of queues requires a scalable and reliable execution environment for the replication tasks that you want to configure and run. On Azure, the runtime environment that is best suited for these tasks is Azure Functions.
Azure Functions allows replication tasks to directly integrate with Azure virtual networks and service endpoints for all Azure messaging services, and it is readily integrated with Azure Monitor.
Azure Functions has prebuilt, scalable triggers and output bindings for Azure Event Hubs, Azure IoT Hub, Azure Service Bus, Azure Event Grid, and Azure Queue Storage, as well as custom extensions for RabbitMQ, and Apache Kafka.
Most triggers will dynamically adapt to the throughput needs by scaling the number of concurrently executing instance up and down based on documented metrics.
With the Azure Functions consumption plan, the prebuilt triggers can even scale down to zero while no messages are available for replication, which means you incur no costs for keeping the configuration ready to scale back up; the key downside of using the consumption plan is that the latency for replication tasks "waking up" from this state is significantly higher than with the hosting plans where the infrastructure is kept running.
You can use this repository either to configure and deploy replication tasks without having to write any code, at all, or you can use it as a foundation for your own replication tasks.
In either case, you will need the following tools:
Most documented script snippets are written for the Bash shell, but can be easily adapted to PowerShell.
Please download the repository contents or clone this repository from a command line window with:
git clone https://github.com/Azure-Samples/azure-messaging-replication-dotnetThe instructions in the functions/config folder and subfolders explain how to configure replication tasks without having to write any code yourself. The provided scripts use the aforementioned tools for configuring the replication tasks.
The following folders are part of this project:
- functions/code - .NET Core Azure Functions projects as starting points for your custom replication tasks that require modification of events or messages as they are being moved:
- EventHubCopy - Function for copying data between two Event Hubs
- EventHubMerge - Function for continuously merging two Event Hubs
- EventHubProjectionToCosmosDB - Function for projecting events into a key/value store ("compaction")
- ServiceBusCopy - Function for copying data between two Service Bus Queues
- ServiceActivePassive - Function backing up messages in a standby topic or queue
- ServiceBusAllActive - Function for maintaining mirrored topics acting as queues
- functions/config - Configuration-only projects, which use the standard replication task library:
- EventHubCopy - Function for copying data between two Event Hubs
- EventHubCopyToServiceBus - Function for copying data between two Event Hubs
- ServiceBusCopy - Function for copying data between two Service Bus Queues
- ServiceBusCopyToEventHub - Function for copying data between a Service Bus Queue to an Event Hub
- src - .NET Core libraries implementing replication tasks.
- src/Azure.Messaging.Replication - Standard replication tasks
- templates - Azure Resource Manager templates to create properly configured Function app environments
- test - Test projects and validation scripts
This project welcomes contributions and suggestions. Most contributions require you to agree to a Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us the rights to use your contribution. For details, visit https://cla.opensource.microsoft.com.
When you submit a pull request, a CLA bot will automatically determine whether you need to provide a CLA and decorate the PR appropriately (e.g., status check, comment). Simply follow the instructions provided by the bot. You will only need to do this once across all repos using our CLA.
This project has adopted the Microsoft Open Source Code of Conduct. For more information see the Code of Conduct FAQ or contact [email protected] with any additional questions or comments.