The Circuit Breaker pattern with Polly

Full source code available here.

Want to learn more about Polly? Check out my Pluralsight course on it.

This post on the Polly circuit breaker is part of a larger series of post on the Polly Resilience Framework, see here for the others, or check out my Pluralsight course.

Basics

The circuit breaker controls the flow of requests from a source to one or more downstream system and cuts the connection when some failure condition is met and resumes the connection after a period. This lets us fail quickly when we know that some remote endpoint or host is not responding as we expect to requests. In this post I am only dealing with the basic circuit breaker, I’ll post another about the advanced circuit breaker later.

Benefits of a circuit breaker

- Stops requests to the downstream system
- Reduces load on a failing downstream system, giving it a chance to recover
- The circuit breaker immediately returns an an error informing the source system that the circuit is open, the source system doesn't have to wait for a timeout to occur
- The application with the circuit breaker does not resource like memory, ports and threads when

Background and Terminology

The name, circuit breaker, comes from electrical circuit breakers in your home or office, in some countries these are referred to as trip switches. If the electrical system is working correctly, the circuit is closed and electricity flows to outlets and appliances. But if something goes wrong, the circuit opens, preventing electricity from flowing, thus protecting people and appliances.

The Polly circuit breaker has the corresponding closed and open positions. When closed, the circuit breaker allows requests to be sent, when open, nothing can be sent and an exception is immediately thrown if a request is send to the circuit breaker.

The Polly circuit breaker has one more status, half-open. When in this state Polly will allow the next request to be sent, and if it succeeds the circuit is closed (and normal operation resumes), but if it fails the circuit returns to open (preventing requests from being sent). The circuit transitions from closed to open when the failure condition is met, from open to half-open when the specified break time is reached.

Here is the full set of possible transitions: _Closed to open - when the failure condition occurs Open to half-open - when the duration of break is reached Half-open to closed - when the first request is a success Half-open to open - when the first request is a failure _

The Policy

When creating a circuit breaker policy you specify the condition under which the circuit opens (breaks) and for how long. To be of any use the same instance of the policy must be used across multiple requests, so you cannot just instantiate it inside a controller.

This has implications, if you use the same circuit breaker policy when calling multiple downstream systems, if the circuit breaks for one, it breaks for all. On the other hand, if you use a circuit breaker on just one downstream system, it can only break the circuit for that one.

I recommend passing policies into the controllers with dependency injection, in the example below I add the policy directly to the DI container, but you should consider using the Polly registry.

As mentioned above the circuit breaker opens when a specified failure condition is met. In the example policy the condition is two consecutive failure responses, the policy also specifies the duration of the break, in this case 10 seconds.

1CircuitBreakerPolicy<HttpResponseMessage> breakerPolicy = Policy
2    .HandleResult<HttpResponseMessage>(r => !r.IsSuccessStatusCode)
3    .CircuitBreakerAsync(2, TimeSpan.FromSeconds(10), OnBreak, OnReset, OnHalfOpen);

The policy is passed into the DI container this way -

In the controller, assign the circuit breaker to a local variable.

1private readonly CircuitBreakerPolicy<HttpResponseMessage> _breakerPolicy;
2
3public CatalogController(HttpClient httpClient, CircuitBreakerPolicy<HttpResponseMessage> breakerPolicy)
4{
5    _breakerPolicy = breakerPolicy;
6    _httpClient = httpClient;
7}

Then use it to make calls to the downstream systems, here -

 1[HttpGet("{catalogId}/inventory")]
 2public async Task<IActionResult> Get(int catalogId)
 3{
 4    string inventoryEndpoint = $"inventory/{catalogId}";
 5
 6    HttpResponseMessage inventoryResponse = await _breakerPolicy.ExecuteAsync(
 7             () => _httpClient.GetAsync(inventoryEndpoint));
 8
 9    //snip..
10}

And here -

 1[HttpGet("{catalogId}/price")]
 2public async Task<IActionResult> GetPrice(int catalogId)
 3{
 4    string requestEndpoint = $"price/{catalogId}";
 5
 6    HttpResponseMessage pricingResponse = await _breakerPolicy.ExecuteAsync(
 7            () => _httpClient.GetAsync(requestEndpoint));
 8
 9    //snip..
10}

Full source code available here.