Simmy Chaos Engine for .NET – Part 2, Resilience and Injected Faults

Full source code here.

I wrote a blog on using a Simmy Fault policy a few days ago, it is very simple to use by itself, but it is far more useful when used in combination with a resilience policy.

Take the scenario where I have added retry logic to my requests, but the requests never fail, how do I know the retry logic works the way I expect. This is where the fault policy shines.

I’m going to build on the previous post where I injected a fault 50 percent of the time when making http requests. In the previous post I had Simmy throw a HttpRequestException but this time I’m going to have Simmy return a HttpResponseMessage containing an InternalServerError.

Along with the fault policy, I am going add a simple retry policy, retrying up to three times.

Here’s how it will work –

The fault policy executes –

If the fault policy is active and a fault is to be returned (remember this happens 50% of the time) a InternalServerError is returned. The HttpClient does NOT make a request to the remote service

The retry policy executes. The handles clause sees the InternalServerError and the behavior clause of the retry kicks in and performs the retry

The fault policy executes again

The preceding two steps repeat until a request succeeds or the retry policy reaches is limit of 3 retries

 

If the fault policy is not active, or a fault is NOT to be returned

the HTTP client executes the request to the remote service

the handles clause of the retry determines whether a retry should be performed

 

This is what it looks like –

At the center is the HttpReqeust, around it is the fault policy and around everything is the retry policy.

To achieve all this in code is very simple, in ConfigureServices add the fault policy.

public void ConfigureServices(IServiceCollection services)
{
	HttpResponseMessage faultHttpResponseMessage = new HttpResponseMessage(HttpStatusCode.InternalServerError)
	{
		Content = new StringContent("Simmy swapped the Ok for an Internal Server Error")
	};
	AsyncInjectOutcomePolicy<HttpResponseMessage> faultPolicy = MonkeyPolicy.InjectFaultAsync(
		faultHttpResponseMessage,
		injectionRate: .5,
		enabled: () => true 
	);

When the fault policy executes, 50% of the time it will return an InternalServerError response, preempting the real request to the remote service.

Next is the retry policy, its behavior clause triggers if the HttpResponseMessage is not a success.

IAsyncPolicy<HttpResponseMessage> retryPolicy = Policy
	.HandleResult<HttpResponseMessage>(r => !r.IsSuccessStatusCode)
	.RetryAsync(3, onRetry: (message, retryCount) =>
	{
		Console.WriteLine($"Retry: {retryCount}");
	});

Wrap the retry policy around the fault policy.

AsyncPolicyWrap<HttpResponseMessage> faultAndRetryWrap = Policy.WrapAsync(retryPolicy, faultPolicy);

Finally, create the HttpClientFactory, passing in the policy wrap. If you are not familiar with how the HttpClientFactory works read Steve Gordon’s series or my blog posts on it. If you want know how it works with Polly, take a look at my posts on that topic.

services.AddHttpClient("OpenBreweryDb", client =>
{
	client.BaseAddress = new Uri("https://api.openbrewerydb.org/breweries/");
	client.DefaultRequestHeaders.Add("Accept", "application/json");
}).AddPolicyHandler(faultAndRetryWrap);

To fire of the HttpClient request inside the BreweryController remains the same –

string requestEndpoint = $"?by_state={state}&by_name={name}";
var httpClient = _httpClientFactory.CreateClient("OpenBreweryDb");
var response = await httpClient.GetAsync(requestEndpoint);

Full source code here.

3 thoughts on “Simmy Chaos Engine for .NET – Part 2, Resilience and Injected Faults

  1. Pingback: Simmy Chaos Engine for .NET – Part 3, Adding Latency | no dogma blog

  2. Hi Bryan,

    Thank you for your posts and these helped me alot in implementing Polly framework in my application.
    I am getting some excpetion when Polly was tried to retry the HttpRequest calling 2nd time the service. I have provided the Policy which I have implemented along with error details below. Could you please help me to resolve it.
    Policy:var retryAndWaitPolicy = Policy.Handle()
    .OrResult(r => !r.IsSuccessStatusCode)
    .WaitAndRetryAsync(
    new TimeSpan[] { TimeSpan.FromSeconds(1), TimeSpan.FromSeconds(2), TimeSpan.FromSeconds(3) },
    onRetry: (dr, timeSpanBeforeNextRetry, retryCount, __) =>
    {
    }
    );

    Error:
    An item with the same key has already been added.
    at System.Runtime.CompilerServices.ConditionalWeakTable`2.Add(TKey key, TValue value)
    at Microsoft.ApplicationInsights.DependencyCollector.Implementation.HttpCoreDiagnosticSourceListener.OnRequest(HttpRequestMessage request, Guid loggingRequestId)
    at Microsoft.ApplicationInsights.DependencyCollector.Implementation.HttpCoreDiagnosticSourceListener.OnNext(KeyValuePair`2 evnt)
    at System.Diagnostics.DiagnosticListener.Write(String name, Object value)
    at System.Net.Http.DiagnosticsHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
    at Microsoft.Extensions.Http.Logging.LoggingHttpMessageHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
    at Polly.Retry.AsyncRetryEngine.ImplementationAsync[TResult](Func`3 action, Context context, CancellationToken cancellationToken, ExceptionPredicates shouldRetryExceptionPredicates, ResultPredicates`1 shouldRetryResultPredicates, Func`5 onRetryAsync, Int32 permittedRetryCount, IEnumerable`1 sleepDurationsEnumerable, Func`4 sleepDurationProvider, Boolean continueOnCapturedContext)

    Please let me know if you need more details.

    • Hi Santhosh,

      I think you copy/pasted your policy incorrectly, it does not look valid.

      As for your error “An item with the same key has already been added”, you must be adding something to the likes of dictionary, it is not Polly related in this case.

Leave a Reply

Your email address will not be published. Required fields are marked *