Merge branch 'master' of https://github.com/exactly-once/workshop

exactly-once · Apr 9, 2024 · 4e42d42 · 4e42d42
2 parents 7afe7d9 + 301e401
commit 4e42d42
Show file tree

Hide file tree

Showing 8 changed files with 77 additions and 57 deletions.
diff --git a/Lectures/Deduplication types.html b/Lectures/Deduplication types.html
@@ -4,8 +4,8 @@
 	<head>
 		<meta charset="utf-8">
 
-		<title>Message ID</title>
-		<meta name="description" content="Deduplication">
+		<title>Consistent messaging</title>
+		<meta name="description" content="Consistent messaging">
 		<meta name="author" content="Szymon Pobiega Tomasz Masternak">
 
 		<meta name="apple-mobile-web-app-capable" content="yes">
@@ -157,7 +157,7 @@
 			</aside>
 			<div class="slides">
 				<section>
-					<h2>Deduplication</h2>
+					<h2>Consistent messaging</h2>
 					<p><a href="https://exactly-once.github.io/">exactly-once.github.io</a></p>
 				</section>
 				<section>
@@ -209,6 +209,10 @@ <h2>Token-based deduplication</h2>
 					<h2 class="fragment">Prove of duplicate</h2>
 					<h2 class="fragment">Prove of uniqueness</h2>
 				</section>
+				<section>
+					<h2>https://form.typeform.com/to/W5kPBsJt</h2>
+					<h1>Q & A</h1>
+				</section>
 			</div>
 		</div>
 

diff --git a/NewExercises/Exercise-12/README.md b/NewExercises/Exercise-12/README.md
@@ -1,6 +1,8 @@
 NOTE: This and couple of following exercises use automated tests for show how our system behaves in various scenarios that might happen in messaging systems.
 
-Our system has been extended with new functionality. After order has been placed, we can book a payment for a given order or cancel a payment that has already been booked. Let's see what happens when some of these messages get reordered:
+Our system has been extended with new functionality. After order has been placed, we can book a payment for a given order or cancel a payment that has already been booked. So far the deduplication mechanism depended on the idempotence of the operations conducted in the message handling logic. If we mark the payment as booked, we can do it multiple times and the result is the same as if we marked it once. Similarly, when we mark the payment as non-booked, we can repat the operation multiple times and still the payment stays non-booked.
+
+Let's see what happens when some of these messages get reordered:
 
 * Open `IntegrationTests.cs` in the `Test` project and naviage to `ChangeStatus` test
 * Use `SendInOrder` utility method to simulate scenario in which oder is placed, payment is booked and later cancelled but the `BookPayment` command in duplicated and the duplicate arrives as the last message:
@@ -14,9 +16,11 @@ await SendInOrder(new IMessage[]
   }
 );
 ``` 
-* Run `ChangeStatus` test and check if the assertion holds
+* Run `ChangeStatus` test and check if the assertion holds. If not then can you tell what is wrong?
 * Add `List<Guid>` property to `Order` enity called `ProcessedMessages`
 ```csharp
  public List<Guid> ProcessedMessages { get; set; } = new List<Guid>();
 ```
-* Use `ProcessedMessages` and `Id` value in the `BookPayment` command to track processed messages and avoid re-processing duplicates
+* Go to the `BookPaymentHandler` and `CancelPaymentHandler` and modify the deduplication logic to use message Ids. At the end of the message processing in the handler, but before the `Order` is persisted, include the Id of the message being processed in the `ProcessedMessages` collection. At the beginning of the handler code check if the message already exists in the collection. If so, return without doing anything in the handler.
+
+What aspect of a message handler is missing in this exercise?
diff --git a/NewExercises/Exercise-13/README.md b/NewExercises/Exercise-13/README.md
@@ -1,10 +1,22 @@
 ## Deterministic message identifiers
 
-Due to considerable sucess of our the business, the system has been extended with new `Marketing` endpoint, reponsible for tracking value of payments booked for any given customer. Now when status of an order is changed either `PaymentBooked` or `PaymentCancelled` event is published.  
+Due to considerable sucess of our the business, the system has been extended with new `Marketing` endpoint, reponsible for tracking value of payments booked for any given customer. When status of an order is changed, either `PaymentBooked` or `PaymentCancelled` event is published.  
 
 Unfortunatelly, our production support team claims that once in a while the calculated value for a customer does not match the total from all the payments. Let's see if we can reproducte such a scenario. 
+
+* Go to the `BookPaymentHandler` or `CancelPaymentHandler` and take a look at the logic. Notice that it is almost the same as in the previous exercise but it also includes publishing of events. Notice that the basic structure remains the same regardless what type of duplication mechanism is used:
+
+```
+if (!IsDuplicate()) 
+{
+   ExecuteBusinessLogic();
+   MarkAsDuplicate();
+   PersistState();
+}
+SendAndPublishMessages()'
+```
 
-* Go to `TrackTotalPaymentsValue` test and check if it passes. Why does it fail? Check what is are the `MessageId` values for both duplicates of the `BookPayment` message. Why are they different? 
-* In the `BookPaymentHandler` and `CancelPaymentHandler` use `PublishWithId` extension method and use `Utils` class to ensure that the published messages have ids that are deterministically derived from the incoming message id and the endpoint name. 
-* Why do we need to put the endpont name in there? 
-* Ensure that both tests are passing 
+* Go to `TrackTotalPaymentsValue` test and check if it passes. Why does it fail? Open the results of the test run and check what is are the `messageId` values for both duplicates of the `BookPayment` message. Why are they different? Doest it resemble any situation we have seen before?
+* In the `BookPaymentHandler` and `CancelPaymentHandler` use `PublishWithId` extension method and use `Utils` class to ensure that the published messages have Ids that are deterministically derived from the incoming message Id. 
+* Is our deterministic Id generation strategy good? 
+* Does the `ChangeStatus` test still pass?
diff --git a/NewExercises/Exercise-14/Marketing/PaymentBookedHandler.cs b/NewExercises/Exercise-14/Marketing/PaymentBookedHandler.cs
@@ -35,7 +35,7 @@ public async Task Handle(PaymentBooked message, IMessageHandlerContext context)
                 }
                 if (payments.TotalValue >= 100 && payments.TotalValue - message.Value < 100)
                 {
-                    payments.OutgoingMessages.Add(new GrantCoupon {Customer = message.CustomerId});
+                    await context.SendImmediately(new GrantCoupon {Customer = message.CustomerId});
                 }
 
                 payments.ProcessedMessage.Add(context.MessageId);
@@ -48,11 +48,6 @@ public async Task Handle(PaymentBooked message, IMessageHandlerContext context)
             {
                 log.Info($"Duplicate detected from {nameof(PaymentBooked)} messageId={context.MessageId}");
             }
-
-            foreach (var outgoingMessage in payments.OutgoingMessages)
-            {
-                await context.SendImmediately(outgoingMessage);
-            }
         }
 
         static readonly ILog log = LogManager.GetLogger<PaymentBookedHandler>();

diff --git a/NewExercises/Exercise-14/Marketing/Payments.cs b/NewExercises/Exercise-14/Marketing/Payments.cs
@@ -7,7 +7,6 @@ public class Payments : Entity
     {
         public int TotalValue { get; set; }
         public List<string> ProcessedMessage = new List<string>();
-        public List<ICommand> OutgoingMessages = new List<ICommand>();
 
         public static string RowId = "39D0410D-02C2-49B9-BDC7-9A66EDEB456D";
     }

diff --git a/NewExercises/Exercise-14/README.md b/NewExercises/Exercise-14/README.md
@@ -1,9 +1,24 @@
 ## State based message generation
 
-Now that we can reliably calculate value of all the payments made by a customer the business wants to put that to a good use. Our team needs to add a small feature ie. when a customer goes over 100 USD in total paymets for the first time we want to send them a coupon. 
+Now that we can reliably calculate value of all the payments made by a customer the business wants to put that to a good use. Our team needs to add a small feature i.e. when a customer goes over 100 USD in total paymets for the first time we want to send them a coupon. 
 
-* To to `IssueCouponCardAfterFirst100USDSpent` test and define the follwong sequence of message processing. What could be a production scenario in which this could happen? 
+* Go to the `IssueCouponCardAfterFirst100USDSpent` test and define the follwong sequence of message processing. What could be a production scenario in which this could happen? 
 
+```csharp 
+new IMessage[] { 
+  submitFirstOrder, 
+  bookFirstPayment, 
+  submitSecondOrder, 
+  bookSecondPayment 
+} 
+``` 
+
+* Run the test and check if the asserition holds 
+* Put logic in the `DropMessagesBehavior` to ensure that the `GrantCoupon` message is skipped (dropped) the first time it is sent. This simulates situation when `PaymentBookHandler` failes on sending out `GrantCoupon` and the incoming message is retried. Use the same method as in [Exercise 5](../Exercise-5/README.md):
+  * Add a boolean flag `dropped`
+  * In the `Invoke` method check if the `context.Message.Instance` is `GrantCoupon`. If so, and the flag is `false` then flip the flag and return. Otherwise invoke `await next()`. This logic will ensure that only the first instance of the `GrantCoupon` message is dropped.
+* To account for this simulated failure we should add another copy of the `bookFirstPayment` to the messages list for the test:
+
 ```csharp 
 new IMessage[] { 
   submitFirstOrder, 
@@ -13,12 +28,16 @@ new IMessage[] {
   bookFirstPayment //HINT: this is a retried message 
 } 
 ``` 
+
+* With that change the test should pass again. Re-run the test. Does it work? Can you tell why? Try placing some breakpoints in the `PaymentBookedHandler`:
+   * in line 18 to check how many times the handler is invoked
+   * in line 36 to see how many times the business logic is invoked
+   * in line 38 to see how many times the `GrantCoupon` message is generated
+* Can you explain the behavior?
+* Use `public List<ICommand> OutgoingMessages = new List<ICommand>();` property in the `Payments` entity to store the outoging messages.
+* Instead of sending the message immediately in line 38, add it to the `OutgoingMessages` collection.
+* Make sure that items in the `OutgoingMessages` are _always_ sent out (including the times when duplicates arrive). Add following code to the bottom of the handler:
 
-* Run the test and check if the asserition holds 
-* Put logic in the `DropMessagesBehavior` to ensure that the `GrantCoupon` message is skipped (dropped) the firt time it is sent. This simulates situation when `PaymentBookHandler` failes on sending out `GrantCoupon` and the incoming message is retried. 
-* Re-run the test. Does it work? Why? 
-* Use `public List<ICommand> OutgoingMessages = new List<ICommand>();` property in the `Payments` entity to store the outoging messages and save them atomically together with the business changes. 
-* Make sure that items in the `OutgoingMessages` are always sent. Also, when duplicates arrive: 
 ```csharp 
 foreach (var outgoingMessage in payments.OutgoingMessages) 
 { 

diff --git a/NewExercises/Exercise-14/Tests/IntegrationTests.cs b/NewExercises/Exercise-14/Tests/IntegrationTests.cs
@@ -89,11 +89,7 @@ public async Task IssueCouponCardAfterFirst100USDSpent()
             var bookSecondPayment = new BookPayment { Id = Guid.NewGuid(), CartId = cartId, Customer = customerId };
 
             await SendInOrder(new IMessage[] {
-                    submitFirstOrder,
-                    bookFirstPayment,
-                    submitSecondOrder,
-                    bookSecondPayment,
-                    bookFirstPayment //HINT: this is a retried message
+                    //TODO: Define message sequence
                 }
             );
 
@@ -178,18 +174,9 @@ async Task SendInOrder(IMessage[] messages)
 
         public class DropMessagesBehavior : Behavior<IOutgoingLogicalMessageContext>
         {
-            bool dropped = false;
-
             public override async Task Invoke(IOutgoingLogicalMessageContext context, Func<Task> next)
             {
-                if (!dropped && context.Message.Instance is GrantCoupon)
-                {
-                    dropped = true;
-                }
-                else 
-                {
-                    await next();
-                }
+                await next();
             }
         }
     }

diff --git a/NewExercises/agenda-2-day.md b/NewExercises/agenda-2-day.md
@@ -11,52 +11,52 @@
     - [Introduction](https://exactly-once.github.io/workshop/Lectures/Intro.html)   
     - Intro to Cosmos DB
     - [Exercise 1 - A simple web application](Exercise-1/README.md)
+  - Block 2
     - [Exercise 2 - State-based deduplication](Exercise-2/README.md)
     - [Exercise 3 - Optimistic concurrency](Exercise-3/README.md)
+    - [Why distributed and asynchronous](https://exactly-once.github.io/workshop/Lectures/Why%20distributed%20and%20asynchronous.html) [script](https://github.com/exactly-once/workshop/blob/master/Lectures/Why%20distributed%20and%20asynchronous.md)
     - _Coffee break_
-  - Block 2
-    - [Why distributed and asynchronous](https://exactly-once.github.io/workshop/Lectures/Why%20distributed%20and%20asynchronous.html) [script](https://github.com/exactly-once/workshop/blob/master/Lectures/Why%20distributed%20and%20asynchronous.md) 
+  - Block 3
     - [Exercise 4 - Make it asynchronous](Exercise-4/README.md)
     - [Sync-Async boundary](https://exactly-once.github.io/workshop/Lectures/Sync-Async.html)
     - [Exercise 5 - Monkeys of Chaos](Exercise-5/README.md) 
-    - [Exercise 6 - The USB Rule](Exercise-6/README.md)
     - _Coffee break_
-  - Block 3
+  - Block 4
+    - [Exercise 6 - The USB Rule](Exercise-6/README.md)
     - [Partial failures](https://exactly-once.github.io/workshop/Lectures/Partial%20failures.html) [script](https://github.com/exactly-once/workshop/blob/master/Lectures/Partial%20failures.md)
     - [Exercise 7 - If in doubt, try again](Exercise-7/README.md)
     - [Sources of duplication](https://exactly-once.github.io/workshop/Lectures/Sources%20of%20duplication.html) [script](https://github.com/exactly-once/workshop/blob/master/Lectures/Sources%20of%20duplication.md)
     - [Exercise 8 - Automate it](Exercise-8/README.md)
-    - [Exercise 9 - Message duplication on the receiver side](Exercise-9/README.md)
     - _Coffee break_
-  - Block 4 
+  - Block 5
+    - [Exercise 9 - Message duplication on the receiver side](Exercise-9/README.md)
     - [Primary-key based deduplication](https://exactly-once.github.io/workshop/Lectures/PK%20based%20deduplication.html)
     - [Exercise 10 - Primary key-based deduplication](Exercise-10/README.md)
+    - Q & A
+* Day 2
+  - Block 1
     - [Integration tests](https://github.com/exactly-once/workshop/blob/master/Lectures/integration-testing.pptx)
-    - _Coffee break_
-  - Block 5
     - [Exercise 11](Exercise-11/README.md) - Conversation-based integration tests
     - [Messages are delivered in-order](https://exactly-once.github.io/workshop/Lectures/Messages%20are%20delivered%20in-order.html) [script](https://github.com/exactly-once/workshop/blob/master/Lectures/Messages%20are%20delivered%20in-order.md)
+  - Block 2
     - [Message ID](https://exactly-once.github.io/workshop/Lectures/Message%20ID.html) [script](https://github.com/exactly-once/workshop/blob/master/Lectures/Message%20ID.md) 
-    - [Exercise 12 - Id-based deduplication](Exercise-12/README.md) 
-    - Q & A
-* Day 2
-  - Block 1
+    - [Exercise 12 - Id-based deduplication](Exercise-12/README.md)
     - [Exercise 13 - Deterministic identifers](Exercise-13/README.md)
     - [Exercise 14 - State based message generation](Exercise-14/README.md) 
     - _Coffee break_
-  - Block 2
-    - [TLA+ - Intro](https://github.com/exactly-once/workshop/blob/master/Lectures/TLA%5EM%20in%20model-checking%20w%20praktyce.pptx)
   - Block 3
-    - [TLA+ - Coding](https://github.com/exactly-once/workshop/tree/master/model-checking)
+    - [TLA+ - Intro](https://github.com/exactly-once/workshop/blob/master/Lectures/TLA%5EM%20in%20model-checking%20w%20praktyce.pptx)
   - Block 4
+    - [TLA+ - Coding](https://github.com/exactly-once/workshop/tree/master/model-checking)
+  - Block 5
     - [Outbox](https://exactly-once.github.io/workshop/Lectures/Outbox.html) [script](https://github.com/exactly-once/workshop/blob/master/Lectures/Outbox.md)
     - [Exercise 15 - Generic outbox](Exercise-15/README.md)
     - [Inbox](https://exactly-once.github.io/workshop/Lectures/Inbox.html#/)
     - [Exercise 16 - Generic outbox 2](Exercise-16/README.md)
     - [Exercise 17 - Generic outbox 3](Exercise-17/README.md)
-    - [Exercise 18 - Outbox with inbox](Exercise-16/README.md)
+    - [Exercise 18 - Outbox with inbox](Exercise-18/README.md)
     - _Coffee break_
-  - Block 5
+  - Block 6
     - [Deduplication types](https://exactly-once.github.io/workshop/Lectures/Deduplication%20types.html)
     - [Azure Functions Case-Study - Exercise](https://github.com/exactly-once/workshop/tree/master/azure-functions-cs)       
     - Q & A