Skip to content
This repository has been archived by the owner on May 29, 2024. It is now read-only.

Mission: Circuit Breaker

Jump to bottom
Stefan Sitani edited this page Jun 26, 2017 · 16 revisions

Circuit Breaker

ID Short Name

106

circuit-breaker

Description

The purpose of this Mission is to demonstrate how the circuit breaker works when a service fails or becomes unavailable.

To demonstrate this behavior, we will configure an /api/greeting HTTP endpoint which is called by the client to get a personalized greeting. This greeting endpoint then issues a request to a name service, which is hidden behind the /api/name HTTP endpoint. If this name service is fully functional, we get back service’s World response (or some other name), otherwise circuit breaker should get the name from the fallback option (a defined fallback method, e.g. could be simple Fallback).

To have the name service fail or become unavailable, we have two options. We can either kill the pod with the name service, or a more graceful one, to have unavailability programmed into the service itself — to toggle / switch a name service to throw an exception / error (e.g. return a HTTP 500 response code).

This should transition the circuit breaker to go to open state. We can then toggle back the name service, to again work properly, hence transition the circuit breaker back to closed state.

We should also be able to query the current state of the name service’s circuit breaker state — via another /api/cb-state HTTP endpoint or some other technology (e.g. web socket). This can / should be nicely visualized in the entry HTML page (index.html).

User Problem

Implementing a Circuit Breaker and fallback mechanism for Microservice applications

The Circuit Breaker pattern provides a generic mechanism for quickly and easily reporting whether or not a service is functioning properly. This allows a service to report information about a failure and decide whether to quickly fail or simply degrade gracefully instead of failing.

Modern cloud applications are often comprised of many smaller services which interact with each other in a many-to-many manner. Understanding when a service is failing or unavailable is critically important, which is the main reason for the Health Check pattern. In some cases though, you may want to provide an alternative response if a service interacts with a failing or unavailable service. For example, if service1 interacts with service2 and service2 begins to fail or become unavailable due to high load, simply detecting the failure and restarting service2 may not alleviate the issue. Instead, you may want to signal service1 to reduce the amount of load it puts on service2 until service2 returns to a normal state.

Concepts and Architectural Patterns

Circuit Breaker

Prerequisites

The runtime (Spring Boot, Swarm, Vert.x) provides the code or the jar file containing the circuit breaker.

Use Case

Success scenario

  1. The use case starts when the application has been deployed into OpenShift.

  2. The user can access the application using a web page provided by the application where the following scenario will be proposed:

  3. Call the api/cb-state endpoint using the button “cb-state” to check that the status of the circuit breaker returned as plain text is “closed”. Or if you’re using web sockets, the label should display “closed”.

  4. a. (if hystrix/turbine dashboard are deployed) Get the hystrix dashboard URL using the command (“oc get route/hystrix-dashboard or minishift openshift service hystrix-dashboard -n your_namespace --url”) and open it with your browser. Click on the button (“monitor stream”). You will be redirected to the turbine server. Under the Circuit section of the screen, verify that the C-B is well monitored and is current status is closed

  5. Click on the Invoke button to call api/greeting

  6. Verify that a JSON response message is received {"content": "Hello, World!"}

  7. Now let’s check a fallback response. We first need to toggle the name service to fail - click on the Toggle button to call the api/state endpoint, its label should change to Fail (now name service should return a 500 HTTP response code when the CB will call it)

  8. Click on the Invoke button again to call api/greeting.

  9. Verify that a JSON response message now received is {"content": "Hello, Fallback!"}

  10. Call the /api/cb-state endpoint using the cb-state button to check that the status of the circuit breaker returned as plain text is open.

  11. a. (The following applies if hystrix/turbine dashboard are deployed)

  12. Open the turbine dashboard. Under the Circuit section of the screen, verify that the C-B status is “open”.

  13. Let’s now toggle the name service back to OK - click on the Toggle button (to call the /api/state/ endpoint), its label should change to OK (and name service should stop returning 500 HTTP response code, it should be back to 200 HTTP response code) Click on the Invoke button a few times to call /api/greeting, until the C-B timeout expires (moving from open to half-open and, if it succeeds, to closed). In parallel, click also on the cb-state button to check the status returned is closed again. Or if you’re using web sockets, the label should again display closed. Verify that a JSON response message received is {"content": "Hello, World!"}

The use case will consist of the following:

  • Develop an HTTP application which uses 2 pods:

  • Maven project is designed as 2 Maven modules; greeting_service and name_service. No parent pom.xml file is created for the 2 modules.

  • Each pod is created from the OpenShift resources generated by the Fabric8 Maven Plugin executed within each Maven module.

  • First name-service pod contains a /api/name endpoint, which will be used to provide the name (in a JSON content) and a /api/state endpoint, to toggle the name service’s functionality.

  • Second greeting-service pod contains an /api/greeting endpoint, which will return a JSON greeting message, getting the name from the /api/name/ endpoint and a /api/cb-state endpoint, to get the current name service’s circuit breaker state.

  • The Greeting endpoint should access the /api/name endpoint via circuit breaker.

  • The Hystrix Dashboard and Turbine server could be deployed to visualize the status about the C-B change. We should find a way when we will deploy this mission on OpenShift Platform to check if the namespace of the user still has enough resources (CPU/memory) to deploy Hystrix/Turbine. If this is not the case, then they will not be installed and we will mention within the README/Doc the scenario to be followed to check the state of the Circuit-Breaker. We could use as input what this command returns (oc describe limits <LIMIT_NAME>) to calculate if they could be deployed.

Acceptance Criteria

During nominal work, a curl request issued against the following service $protocol://$hostname:$port/api/greeting returns:

{ "content": "Hello, World!"}

If we toggle the name endpoint / service by issuing an HTTP GET request: $protocol://$hostname:$port/api/state?state=fail (this should make the name service return a 500 HTTP response code) we should get a fallback - a curl or http request issued against the following service $protocol://$hostname:$port/api/greeting returns

{ "content": "Hello, Fallback!"}

When we toggle back the name service — issuing GET request with: $protocol://$hostname:$port/api/state?state=ok The greeting endpoint $protocol://$hostname:$port/api/greeting should return

{ "content": "Hello, World!"}

During this state transitions, we can issue an http request against the cb-state endpoint / web socket and should receive a response reflecting its current state: $protocol://$hostname:$port/api/cb-state should return one of these responses:

{state:"open”}

or:

{state:"closed”}

Vert.x-specific Acceptance Criteria

Swarm-specific Acceptance Criteria

Boot-specific Acceptance Criteria

Integration Requirements

Tags

Circuit Breaker

Notes

Approval

Coordination Owner

Ales Justin

☑

Product Management

John Clingan

☑

DevExp

Andy Rubinger

☑

Vert.x

Clement Escoffier

☑

WildFly Swarm

Ken Finnigan

☑

Spring Boot

Charles Moulliard

☑

QE

Ladislav Thon

☑

Docs

Zach Rhoads

☑

Architect

Scott Stark

☑
Clone this wiki locally