Simmy Chaos Engine for .NET – Part 4, Doing Some Real Damage, Dropping a Table

Full source code here.

Up until now the Simmy examples I’ve written have thrown exceptions, changed successes to failures or slowed down a request. None of these did any real damage to your system, and your application would probably have recovered when the next request came along.

But what about dropping a table? Poof, all the data is gone. Now what does your application do?

Doing this is very easy with the Behavior clause. Like the other clauses it takes a probability, an enabled flag, and action or func that executes any piece of code.

The Scenario
I have Web API application with a products controller that queries the database for a list of products.

During application start up the database is created and populated in Configure method of Startup.cs using the EnsureCreated() method and custom seed class.

Inside the controller the action method uses the SalesContext to query the database.

Inside the controller’s constructor I have the chaos policy, it is set to drop the Products table. The drop action of the behavior policy will execute 25% of the time the policy is run. If get an error when you run this saying that the table is absent, it means the chaos policy dropped the table, you’ll have to restart the application to run through the database seeding and creation again.

The policy looks like this –

public ProductsController(SalesContext salesContext)
{
	_behaviorPolicy = MonkeyPolicy.InjectBehaviourAsync(
		 async () =>
		 {
			 await _salesContext.Database.ExecuteSqlCommandAsync("DROP TABLE Products");
			 Console.WriteLine("Dropped the Products table.");
		 },
		 0.25, // 25% of the time this policy will execute
		async () => await Task.FromResult(true));
	_salesContext = salesContext;
}

The request to the _salesContext is made inside the policy, but the policy executes before the call the db is made and in 25% of such calls the table will be dropped.

[HttpGet]
public async Task<ActionResult> Get()
{
	List<Product> products =  await _behaviorPolicy.ExecuteAsync(async () => await _salesContext.Products.Take(10).ToListAsync());

	return Ok(products);
}

You can of course execute and code from inside the policy, wipe a cache, kill a service, delete a file, etc.

This example might be a little excessive in the damage it does, you can decide if it is unreasonable to expect you application to continue working even when the database is unavailable.

Full source code here.

Simmy Chaos Engine for .NET – Part 3, Adding Latency

Full source code here.

This is the third in my series on chaos engineering with Simmy. The first two posts dealt with the fault policy throwing exceptions and returning failed responses rather than calling the remote service.

In this I will show how to use the latency policy to inject delays into requests. It works in a similar way to the fault policy – a delay is chosen, the percentage of the time the delay should apply is chosen, and finally the policy is enabled/disabled.

When the policy is active, it delays the request, not the response.

var latencyPolicy = MonkeyPolicy.InjectLatencyAsync<HttpResponseMessage>(
    TimeSpan.FromSeconds(5), // inject a 5 second delay
    0.5, // in 50% of requests
    () => true); // policy is enabled

To see the effect of the latency policy I am going to add Polly timeout policy that throws an TimeoutRejectedException if no response is received from the OpenBreweryDb in 2 seconds.

I have also added a Polly retry policy to retry request that fail or timeout.

IAsyncPolicy<HttpResponseMessage> timeoutPolicy = Policy.TimeoutAsync<HttpResponseMessage>(2); 

The retry policy is set to retry up to three times.

IAsyncPolicy<HttpResponseMessage> retryPolicy = Policy
                .HandleResult<HttpResponseMessage>(r => !r.IsSuccessStatusCode)
                .Or<Exception>()
                .RetryAsync(3)

I have a policy wrap with the latency policy at the center, the timeout around it and finally the retry around that.

IAsyncPolicy<HttpResponseMessage> latencyTimeoutAndRetryPolicy = Policy.WrapAsync(
    retryPolicy, timeoutPolicy, latencyPolicy);

What happens is –
In 50% of the requests to the OpenBreweryDb, the latency policy delays the request by 5 seconds.
If the latency is active, the timeout policy throws a TimeoutRejectedException.
Then retry policy retries the request again.

If you run the application you will see the timeout and retry policies logging to the console.

That’s it, the latency policy is very easy to use.

Full source code here.

Simmy Chaos Engine for .NET – Part 2, Resilience and Injected Faults

Full source code here.

I wrote a blog on using a Simmy Fault policy a few days ago, it is very simple to use by itself, but it is far more useful when used in combination with a resilience policy.

Take the scenario where I have added retry logic to my requests, but the requests never fail, how do I know the retry logic works the way I expect. This is where the fault policy shines.

I’m going to build on the previous post where I injected a fault 50 percent of the time when making http requests. In the previous post I had Simmy throw a HttpRequestException but this time I’m going to have Simmy return a HttpResponseMessage containing an InternalServerError.

Along with the fault policy, I am going add a simple retry policy, retrying up to three times.

Here’s how it will work –

The fault policy executes –

If the fault policy is active and a fault is to be returned (remember this happens 50% of the time) a InternalServerError is returned. The HttpClient does NOT make a request to the remote service

The retry policy executes. The handles clause sees the InternalServerError and the behavior clause of the retry kicks in and performs the retry

The fault policy executes again

The preceding two steps repeat until a request succeeds or the retry policy reaches is limit of 3 retries

 

If the fault policy is not active, or a fault is NOT to be returned

the HTTP client executes the request to the remote service

the handles clause of the retry determines whether a retry should be performed

 

This is what it looks like –

At the center is the HttpReqeust, around it is the fault policy and around everything is the retry policy.

To achieve all this in code is very simple, in ConfigureServices add the fault policy.

public void ConfigureServices(IServiceCollection services)
{
	HttpResponseMessage faultHttpResponseMessage = new HttpResponseMessage(HttpStatusCode.InternalServerError)
	{
		Content = new StringContent("Simmy swapped the Ok for an Internal Server Error")
	};
	AsyncInjectOutcomePolicy<HttpResponseMessage> faultPolicy = MonkeyPolicy.InjectFaultAsync(
		faultHttpResponseMessage,
		injectionRate: .5,
		enabled: () => true 
	);

When the fault policy executes, 50% of the time it will return an InternalServerError response, preempting the real request to the remote service.

Next is the retry policy, its behavior clause triggers if the HttpResponseMessage is not a success.

IAsyncPolicy<HttpResponseMessage> retryPolicy = Policy
	.HandleResult<HttpResponseMessage>(r => !r.IsSuccessStatusCode)
	.RetryAsync(3, onRetry: (message, retryCount) =>
	{
		Console.WriteLine($"Retry: {retryCount}");
	});

Wrap the retry policy around the fault policy.

AsyncPolicyWrap<HttpResponseMessage> faultAndRetryWrap = Policy.WrapAsync(retryPolicy, faultPolicy);

Finally, create the HttpClientFactory, passing in the policy wrap. If you are not familiar with how the HttpClientFactory works read Steve Gordon’s series or my blog posts on it. If you want know how it works with Polly, take a look at my posts on that topic.

services.AddHttpClient("OpenBreweryDb", client =>
{
	client.BaseAddress = new Uri("https://api.openbrewerydb.org/breweries/");
	client.DefaultRequestHeaders.Add("Accept", "application/json");
}).AddPolicyHandler(faultAndRetryWrap);

To fire of the HttpClient request inside the BreweryController remains the same –

string requestEndpoint = $"?by_state={state}&by_name={name}";
var httpClient = _httpClientFactory.CreateClient("OpenBreweryDb");
var response = await httpClient.GetAsync(requestEndpoint);

Full source code here.

Simmy Chaos Engine for .NET – Part 1, Injecting Faults

Full source code here.

For quite a while I have been writing, presenting and making videos about using Polly to make applications more stable.

With this post I’m starting a series on how break your applications with a chaos engine, the kind you might have heard about from the likes of NetFlix.

The team behind Polly released a chaos engineering module in June 2019. It allows you to inject faults, latency and arbitrary behavior into your application.

There are two general ways of using these policies –
1. Within your own business logic (if you have access to it).
2. Within the calls to business logic (if you can’t or don’t want to change it).

I’m going to deal with the latter case in this post. I want to see how my code responds when failures occur in third party dependencies, since I have no control over those dependencies I have to introduce the errors myself.

Making a good service go bad
In the provided code I have dependency on remote service and I want make it seem that that service suffers from a high percentage of faults.
This example will make GET requests to an API that looks up breweries, from what I can tell it is a highly reliable API, but I will make look as though 50% of requests to it fail. The point of doing this is to see how my application reacts to these errors – does it do so gracefully, or does it fall on its face.

If you made requests to the API without interfering with them you should see a 100% success rate. But I want to see failures, so I’ll use the Fault Policy to inject exceptions.

First, I added two NuGet packages to my project – Polly.Contrib.Simmy and Microsoft.Extensions.Http.Polly to the project.

In Startup.cs I add the fault policy.

AsyncInjectOutcomePolicy<HttpResponseMessage> faultPolicy = MonkeyPolicy.InjectFaultAsync<HttpResponseMessage>(
	new HttpRequestException("Simmy threw an exception"), 
		injectionRate: .5,
		enabled: () => true
	);

The parameters specify the exception to throw, the percentage of requests that throw the exception, and whether the policy is enabled. You can use set the injection rate and whether the policy is enabled via configuration, but I’m not going to show that here.
In this case, I am throwing an HttpRequestException 50% of the time, and the policy is enabled.

Then create a HttpClientFactory for the OpenBreweryDb

services.AddHttpClient("OpenBreweryDb", client =>
{
	client.BaseAddress = new Uri("https://api.openbrewerydb.org/breweries/");
	client.DefaultRequestHeaders.Add("Accept", "application/json");
}).AddPolicyHandler(faultPolicy);

If you are not yet using the HttpClientFactory, the source code has an example for you too, check the BreweryNoHttpClientFactoryController.

Start up the application, it should open on port 5000 and make a request to look up Night Shift Brewing in Massachusetts. There is a 50% chance of an exception occurring with each request.

Try it out. You’ll see that my application does not handle these failures very well, it simply returns the error to original caller.

That’s it, you are now starting down the road of mixing in some chaos into your application.

What then?
Great, my application fails, but we always knew it could. There’s not much point introducing failures unless you plan to handle them in some way, and this is where you might add a retry policy to improve you likelihood of a successful request.

Of course, the Polly retry, or wait and retry policies would be ideal. I’ll show how to do just that in the next post.

Full source code here.