Simmy Chaos Engine for .NET - Part 1, Injecting Faults

Full source code here.

Want to learn more about Polly? Check out my Pluralsight course on it.

For quite a while I have been writing, presenting and making videos about using Polly to make applications more stable.

With this post I’m starting a series on how break your applications with a chaos engine, the kind you might have heard about from the likes of NetFlix.

The team behind Polly released a chaos engineering module in June 2019. It allows you to inject faults, latency and arbitrary behavior into your application.

There are two general ways of using these policies -

1. Within your own business logic (if you have access to it).
2. Within the calls to business logic (if you can’t or don’t want to change it).

I’m going to deal with the latter case in this post. I want to see how my code responds when failures occur in third party dependencies, since I have no control over those dependencies I have to introduce the errors myself.

Making a good service go bad

In the provided code I have dependency on remote service and I want make it seem that that service suffers from a high percentage of faults. This example will make GET requests to an API that looks up breweries, from what I can tell it is a highly reliable API, but I will make look as though 50% of requests to it fail. The point of doing this is to see how my application reacts to these errors - does it do so gracefully, or does it fall on its face.

If you made requests to the API without interfering with them you should see a 100% success rate. But I want to see failures, so I’ll use the Fault Policy to inject exceptions.

First, I added two NuGet packages to my project - Polly.Contrib.Simmy and Microsoft.Extensions.Http.Polly to the project.

In Startup.cs I add the fault policy.

1AsyncInjectOutcomePolicy<HttpResponseMessage> faultPolicy = MonkeyPolicy.InjectFaultAsync<HttpResponseMessage>(
2	new HttpRequestException("Simmy threw an exception"), 
3		injectionRate: .5,
4		enabled: () => true
5	);

The parameters specify the exception to throw, the percentage of requests that throw the exception, and whether the policy is enabled. You can use set the injection rate and whether the policy is enabled via configuration, but I’m not going to show that here. In this case, I am throwing an HttpRequestException 50% of the time, and the policy is enabled.

Then create a HttpClientFactory for the OpenBreweryDb -

1services.AddHttpClient("OpenBreweryDb", client =>
2{
3	client.BaseAddress = new Uri("https://api.openbrewerydb.org/breweries/");
4	client.DefaultRequestHeaders.Add("Accept", "application/json");
5}).AddPolicyHandler(faultPolicy);

If you are not yet using the HttpClientFactory, the source code has an example for you too, check the BreweryNoHttpClientFactoryController.

Start up the application, it should open on port 5000 and make a request to look up Night Shift Brewing in Massachusetts. There is a 50% chance of an exception occurring with each request.

Try it out. You’ll see that my application does not handle these failures very well, it simply returns the error to original caller.

That’s it, you are now starting down the road of mixing in some chaos into your application.

What then?

Great, my application fails, but we always knew it could. There’s not much point introducing failures unless you plan to handle them in some way, and this is where you might add a retry policy to improve you likelihood of a successful request.

Of course, the Polly retry, or wait and retry policies would be ideal. I’ll show how to do just that in the next post.

Full source code here.