Polly, HttpClientFactory and the Policy Registry – choosing the right policy based on the HTTP request

Full source code available here.

The release of .NET Core 2.1 has made using HttpClient much easier. If you have been using HttpClient for a while you will know about its limitations around reuse and DNS expiry, I wrote about this some time ago. The HttpClientFactory takes care of the problems addressed in that post.

Along with those improvements the HttpClientFactory now makes it very easy to add Polly policies that will be executed whenever you create a HttpClient with the factory. This means you define the polices in one place, add them to the factory and forget about them! No more code is needed to add resilience to each of your calls. The factory takes care of creating the client and applying the policy, you can’t even tell that Polly is protecting your call where you make it.

For more on the HttpClientFactory check out Steve Gordon’s series of articles.

A simple example
Here’s a simple example of the HttpClientFactory in use. This adds the factory to our dependency injection container, sets the base address of the remote server and lets us configure the client. It also adds a simple retry policy that checks if the response is NOT a success code and retries up to three times.

services.AddHttpClient("OrderApiServer", client =>
    client.BaseAddress = new Uri("http://localhost:57696/api/");
    client.DefaultRequestHeaders.Add("Accept", "application/json");
    (r => !r.IsSuccessStatusCode).RetryAsync(3));

When you create a HttpClient now, it will include the policy.

var httpClient = _httpClientFactory.CreateClient("OrderApiServer");

And when you execute a request it the policy will retry up to three times if necessary.

var result = await httpClient.GetAsync("api/SomeEndpoint);

This might be great if all you are going to is perform GET’s and every call you make is safe and idempotent. But what if you want to use GET, PUT, POST and DELETE, you won’t want to retry all of those requests?

This is easy too. You add a policy registry will all the policies you want to use. Say, a retry policy, a wait and retry policy and no op policy, and provide a selector method to pick the right one based on the HTTP verb (or even the endpoint you are requesting).

Using a Policy Registry with the HttpClientFactory

Step 1

Add the policy registry to the Service Collection and add the policies to the registry.

public void ConfigureServices(IServiceCollection services)
    IPolicyRegistry<string> registry = services.AddPolicyRegistry(); // creates the registry and adds it to the service collection

    IAsyncPolicy<HttpResponseMessage> httpRetryPolicy =
        Policy.HandleResult<HttpResponseMessage>(r => !r.IsSuccessStatusCode)

    registry.Add("SimpleHttpRetryPolicy", httpRetryPolicy);

    IAsyncPolicy<HttpResponseMessage> httWaitAndpRetryPolicy =
        Policy.HandleResult<HttpResponseMessage>(r => !r.IsSuccessStatusCode)
            .WaitAndRetryAsync(3, retryAttempt => TimeSpan.FromSeconds(retryAttempt));

    registry.Add("SimpleWaitAndRetryPolicy", httWaitAndpRetryPolicy);

    IAsyncPolicy<HttpResponseMessage> noOpPolicy = Policy.NoOpAsync()

    registry.Add("NoOpPolicy", noOpPolicy);

Step 2
Now that we have the three polices and the registry, lets add the HttpClientFactory to the Service Collection.

services.AddHttpClient("OrderApiServer", client =>
    client.BaseAddress = new Uri("http://localhost:57696/api/");
    client.DefaultRequestHeaders.Add("Accept", "application/json");

Note the PolicySelector, this is the method I use to choose the right policy for each request.

Step 3
This method is simple, it takes the registry and the HttpRequest as arguments and grabs the policy from the registry based on the HTTP verb.

private IAsyncPolicy<HttpResponseMessage> PolicySelector(IReadOnlyPolicyRegistry<string> policyRegistry, HttpRequestMessage httpRequestMessage)
    if (httpRequestMessage.Method == HttpMethod.Get)
        return policyRegistry.Get<IAsyncPolicy<HttpResponseMessage>>("SimpleHttpRetryPolicy");
    else if (httpRequestMessage.Method == HttpMethod.Post)
        return policyRegistry.Get<IAsyncPolicy<HttpResponseMessage>>("NoOpPolicy");
        return policyRegistry.Get<IAsyncPolicy<HttpResponseMessage>>("SimpleWaitAndRetryPolicy");

You could use a lambda instead of a method when creating the HttpClientFactory. For example if if you had two policies to choose from –

.AddPolicyHandlerFromRegistry((policyRegistry, message) => policyRegistry.Get<IAsyncPolicy<HttpResponseMessage>>(message.Method == HttpMethod.Get ? "SimpleHttpRetryPolicy" : "NoOpPolicy"));

That’s the hard work done.

Step 4
Let’s take a look at the controller.

public class OrderController : ControllerBase
    private readonly IHttpClientFactory _httpClientFactory;

    public OrderController(IHttpClientFactory httpClientFactory)
        _httpClientFactory = httpClientFactory;

    public async Task<ActionResult> Get(int orderId)
        string requestEndpoint = $"invoice/{orderId}";

        var httpClient = _httpClientFactory.CreateClient("OrderApiServer");

        HttpResponseMessage response = await httpClient.GetAsync(requestEndpoint);

		//handle response

The only thing I’m passing into the constructor is the HttpClientFactory, I don’t have any Polly using statements.

In the GET method you can’t even tell that there is a policy around the HttpClient.GetAsync(..)

All the Polly work is done in the Startup.cs, the policies are defined, added to the registry, the registry is added to the HttpClientFactory and the HttpClientFactory is added to the services collections.

Now all your HttpClient requests are executed inside a policy.

For more on this, check out Dylan’s Polly and HttpClientFactory documentation.

Full source code available here.

Loading Config from Multiple Sources with .NET Core 2.x Web Api or MVC

Full source code available here.

.NET Core 2 and .NET Core 2.1 offer many ways to load configuration and they are well documented by Microsoft. But there is one scenario that I didn’t see explained.

If you want to supplement the configuration in appsettings.json with more from a remote service, database or some other source, you first need to know where that source is, then make a request to it and add it to your configuration. You are probably going to put the location of the remote configuration appsettings.json, then you call the remote config source.
Sounds easy? It is, but not obvious.

First, a little background.

In an out of the box Web API or MVC application you get a Program.cs that looks like this –

public class Program
    public static void Main(string[] args)

    public static IWebHostBuilder CreateWebHostBuilder(string[] args) =>

On line 9, we have WebHost.CreateDefaultBuilder(args), if you peek the definition of this with Resharper you will some something like –

The highlighted code is how your appsettings.json is loaded into the configuration and added to services collection, this call happens just as you leave Program.cs

But you want to read from the appsettings.json and use a value from it to make another call to get more configuration and then add the whole lot of the services collection. You might also want to setup some logging configuration in Program.cs, so you need access to all configuration settings before calling


Here’s how you do it.

Step 1
Inside the main method in Program.cs, build the configuration yourself –

IConfigurationBuilder builder = new ConfigurationBuilder()
Configuration = builder.Build();

Get the location of the other configuration service, in my example it is another json file.
Add it to the builder and call build.

string otherConfigService = Configuration["otherConfigService"]; // this could be a database or something like consul

Configuration = builder.Build();

Now you have access to the configuration values from the second source.


So far so good, but as mentioned above, the CreateDefaultBuilder adds the configuration to the ServiceCollection, making it available by DI. But that only happens for appsettings.json (and appsettings.{env.EnvironmentName}.json, not the config you loaded from the other source.

If you tried to access the a config setting of your from inside Startup.cs or a controller, it would not be there.

Step 2
Let’s make our configuration available via the ServicesCollection.

In the first block of code above there was a call – CreateWebHostBuilder(args).Build().Run();

I added line 4 below, this will add the configuration to the services collection.

public static IWebHostBuilder CreateWebHostBuilder(string[] args) =>
        .UseConfiguration(Configuration); // add this line to add your configuration to the service collection

Now, the config values loaded from appsettings.json and your secondary config source will be available throughout your application.

Full source code available here.

Using Dependency Injection with Startup.cs in ASP.NET Core

Full source code available here.

Did you know that .NET Core 2 can pass in an instance of any type you want to the Startup.cs constructor? Well you can! Here’s how.

Out of the box the Startup.cs comes like this –

public class Startup
	public Startup(IConfiguration configuration)
		Configuration = configuration;

The IConfiguration is passed in by dependency injection. In Program.cs you can add other types to the DI container, and then add that type to the constructor parameters of Startup.cs.

Here’s how it works. The Microsoft.AspNetCore.Hosting.WebHostBuilder has a ConfigureServices(...) method –

public IWebHostBuilder ConfigureServices(Action<IServiceCollection> configureServices)
    if (configureServices == null)
        throw new ArgumentNullException(nameof(configureServices));

    return ConfigureServices((_, services) => configureServices(services));

This lets you add services to the dependency injection container from the Program.cs.

WebHost.CreateDefaultBuilder(args) returns an IWebHostBuilder and that lets you use the ConfigureServices(...) method. Easy!

public static IWebHost BuildWebHost(string[] args) =>
        .ConfigureServices(serviceCollection =>
            serviceCollection.AddScoped<ISomeService, SomeService>())

And in Startup.cs

public class Startup
    private ISomeService _someService;
    public Startup(IConfiguration configuration, ISomeService someService)
        _someService = someService;
        Configuration = configuration;
// snip..	

The instance of ISomeService is of course available for DI everywhere in the application.

public class ValuesController : Controller
    private readonly ISomeService _someService;
    public ValuesController(ISomeService someService)
        _someService = someService;

Full source code available here.

Getting .NET Core 2.1 Preview 2 Working with Visual Studio 2017

About a year ago I wanted to start using .NET Framework 4.7, it should have been an easy process, but wasn’t. After some trial and error if figured it out and wrote a blog post explaining how to get it working.

Now with the release of .NET Core 2.1 Preview 2, I have hit the familiar problems – no obvious instructions from Microsoft, no one place to download all everything that needed and a Visual Studio install that does not include what you the latest SDK or runtime, and errors like – The specified framework 'Microsoft.AspNetCore.App', version '2.1.0-preview2-final' was not found. or, 'dotnet.exe' has exited with code -2147450730 (0x80008096).

After a few hours messing around and installing the wrong versions of the right software I figured it out.

Step 1

Install the latest version of Visual Studio Preview https://www.visualstudio.com/vs/preview/, at the time of writing this was version 15.7.0 Preview 4.0

I’m interested in developing Web API applications, so I check that box.

BUT, version 15.7.0 Preview 4.0 comes with .NET Core Version 2.1.0 Preview 1. So you don’t get the fancy new features like HttpClientFactory. Read on…

Step 2

Go to
https://github.com/dotnet/core/blob/master/release-notes/download-archives/2.1.0-preview2-download.md and download the SDK and the Runtime for your architecture.

You can verify the checksum of the downloads if you want to by comparing your SHA sum to these ones – https://dotnetcli.blob.core.windows.net/dotnet/checksums/2.1.300-preview2-008530-sdk-sha.txt

Install both the SDK and the runtime.

Step 3

To verify that that you are running .NET Core Preview 2, open visual studio and create a new .NET Core Web API application.

Once it has been created, open the .csproj file, you should see this block –

  <PackageReference Include="Microsoft.AspNetCore.App" Version="2.1.0-preview2-final" />

If you don’t see Version="2.1.0-preview2-final", something has gone wrong.

You can verify what packages are installed by going to your start menu and opening Apps and Features/Add or remove programs.

If you have installed the right SDK and runtime you should see the following –

The Circuit Breaker pattern with Polly

Full source code available here.

This post on the Polly circuit breaker is part of a larger series of post on the Polly Resilience Framework, see here for the others, or check out my Pluralsight course.

The circuit breaker controls the flow of requests from a source to one or more downstream system and cuts the connection when some failure condition is met and resumes the connection after a period. This lets us fail quickly when we know that some remote endpoint or host is not responding as we expect to requests. In this post I am only dealing with the basic circuit breaker, I’ll post another about the advanced circuit breaker later.

Benefits of a circuit breaker

  • Stops requests to the downstream system
  • Reduces load on a failing downstream system, giving it a chance to recover
  • The circuit breaker immediately returns an an error informing the source system that the circuit is open, the source system doesn’t have to wait for a timeout to occur
  • The application with the circuit breaker does not resource like memory, ports and threads when

Background and Terminology
The name, circuit breaker, comes from electrical circuit breakers in your home or office, in some countries these are referred to as trip switches. If the electrical system is working correctly, the circuit is closed and electricity flows to outlets and appliances. But if something goes wrong, the circuit opens, preventing electricity from flowing, thus protecting people and appliances.

The Polly circuit breaker has the corresponding closed and open positions. When closed, the circuit breaker allows requests to be sent, when open, nothing can be sent and an exception is immediately thrown if a request is send to the circuit breaker.

The Polly circuit breaker has one more status, half-open. When in this state Polly will allow the next request to be sent, and if it succeeds the circuit is closed (and normal operation resumes), but if it fails the circuit returns to open (preventing requests from being sent).
The circuit transitions from closed to open when the failure condition is met, from open to half-open when the specified break time is reached.

Here is the full set of possible transitions:
Closed to open – when the failure condition occurs
Open to half-open – when the duration of break is reached
Half-open to closed – when the first request is a success
Half-open to open – when the first request is a failure

The Policy
When creating a circuit breaker policy you specify the condition under which the circuit opens (breaks) and for how long. To be of any use the same instance of the policy must be used across multiple requests, so you cannot just instantiate it inside a controller.

This has implications, if you use the same circuit breaker policy when calling multiple downstream systems, if the circuit breaks for one, it breaks for all.
On the other hand, if you use a circuit breaker on just one downstream system, it can only break the circuit for that one.

I recommend passing policies into the controllers with dependency injection, in the example below I add the policy directly to the DI container, but you should consider using the Polly registry.

As mentioned above the circuit breaker opens when a specified failure condition is met. In the example policy the condition is two consecutive failure responses, the policy also specifies the duration of the break, in this case 10 seconds.

CircuitBreakerPolicy<HttpResponseMessage> breakerPolicy = Policy
    .HandleResult<HttpResponseMessage>(r => !r.IsSuccessStatusCode)
    .CircuitBreakerAsync(2, TimeSpan.FromSeconds(10), OnBreak, OnReset, OnHalfOpen);

The policy is passed into the DI container this way –

In the controller, assign the circuit breaker to a local variable.

private readonly CircuitBreakerPolicy<HttpResponseMessage> _breakerPolicy;

public CatalogController(HttpClient httpClient, CircuitBreakerPolicy<HttpResponseMessage> breakerPolicy)
    _breakerPolicy = breakerPolicy;
    _httpClient = httpClient;

Then use it to make calls to the downstream systems, here –

public async Task<IActionResult> Get(int catalogId)
    string inventoryEndpoint = $"inventory/{catalogId}";

    HttpResponseMessage inventoryResponse = await _breakerPolicy.ExecuteAsync(
             () => _httpClient.GetAsync(inventoryEndpoint));


And here –

public async Task<IActionResult> GetPrice(int catalogId)
    string requestEndpoint = $"price/{catalogId}";

    HttpResponseMessage pricingResponse = await _breakerPolicy.ExecuteAsync(
            () => _httpClient.GetAsync(requestEndpoint));


Full source code available here.

How to install .NET Framework 4.7 in Visual Studio 2017

Microsoft just released .NET Framework 4.7, but surprisingly it is not the easiest to install.

Here are the steps for Windows 10, as of this writing, it is not available for earlier versions for of Windows.

1. Install Windows 10 Creators Update

Go here https://www.microsoft.com/en-us/software-download/windows10, download and install. This will take a while and will reboot your computer a few times.

2. Install Visual Studio 2017 Updates

In Visual Studio, go to Tools -> Extensions and Updates, install any available updates.

(You might have the option here to install 4.7, see the image below in step 4.)

3. Open Visual Studio 2017 Installer

Use the Start Menu to search for the Visual Studio Installer.

In the installer, click Modify on the version of Visual Studio you have installed.

4. Install .NET framework 4.7

Click .NET desktop development.

On the right select .NET Framework 4.7 and finally install.

A simple Polly example with WebApi 2

Download full source code.

If you are calling APIs and you haven’t heard about The Polly Project, you should check it out. It helps take care of many common problems like unreliable connections, retries and circuit breaking. It can also be used to wrap code where other forms of transient exception may occur.

In this blog post I’m going to show the wait and retry policy when calling an unreliable Web Api endpoint.

When using Polly there are two pieces that need you need to code – the policy and the execution of code wrapped by the policy.

In this example the policy is created in the constructor of the controller, this is NOT what you should do in a real application. For an example using DI to share policies see “Reusing Polly Policies with Dependency Injection”.

readonly RetryPolicy<HttpResponseMessage> _httpRequestPolicy;

public ValuesController()
    _httpRequestPolicy = Policy.HandleResult<HttpResponseMessage>(
            r => r.StatusCode == HttpStatusCode.InternalServerError)
            retryAttempt => TimeSpan.FromSeconds(retryAttempt));

This policy states that in the event that a HttpResponseMessage has a of InternalServerError, the request should be tried up to three times with a back off increasing by one second each time.

That’s the policy in place, now we need to call the web service endpoint.

public async Task<IHttpActionResult> Get()
   var httpClient = GetHttpClient();
   string requestEndpoint = "numbers/"; // numbers is the name of the controller being called

   HttpResponseMessage httpResponse = await _httpRequestPolicy.ExecuteAsync(() => httpClient.GetAsync(requestEndpoint));

   IEnumerable<int> numbers = await httpResponse.Content.ReadAsAsync<IEnumerable<int>>();
   return Ok(numbers);

If you execute the code it makes a GET request to http://localhost:2351/api/values, despite two failures when calling the NumbersController you will get a 200 response and a list of numbers.

If you want to verify this, step through the provided solution with the debugger. You’ll see that the first two requests to the NumbersContoller return 500 errors and that the third succeeds and returns an array of ints.

With just a few lines of code Polly provided us with a robust wait and retry mechanism for web requests.

In another blog I’ll show a better way of defining policies so they can be used in any controller.

Download full source code.

Locating and checking an executing DLL on a running web server or other application

Edit – even though the steps described below related to an example with IIS, you can use this technique for any running process, it’s just a matter of identifying the process id. In some cases you will not even need the process id, e.g. if all processes are using the same version of a DLL.


Figuring out what dll is actually executing on a running web server is no easy task; it’s not as simple as looking in the bin directory of the deployed application as the dlls are copied from there to a set of temporary directories within the windows systemroot.

I know of two ways of figuring this out, the first requires some code changes and the second does not.

I wrote some time ago about how to find which dll is really executing when running an application, it works just fine if you can alter the source code to include the snippet I proposed.

Clearly that is not a lot of use in a deployed application when you cannot change the code or even redeploy.

No code change needed

Here is the alternative, no code changes needed. The application I am interested in is called MyFancyApp, there are other applications running on the server too. Let’s say that the dll I am interested in is AutoMapper.dll, I need to verify that the running dll is really the version I expect.

1. Open a command prompt and navigate to %systemroot%\System32\inetsrv.
2. Execute appcmd list wp, this will show all the running application pools and the associated process ids. The output will look something like this –
find dll appcmd

You can see that MyFancyApp is running with process id of 1224.

Now switch to Process Explorer.

3. Open Process Explorer and hit CTRL-F, type in AutoMapper.dll. In the results PID 1224 is shown, in fact it will be there a few times. At this point the file location of the dll is visible.
find dll search

4. Double click one of the entries for PID 1224 where the Type is DLL.

5. In the lower pane of Process Explorer the AutoMapper.dll is selected and the version is shown, I had to right click on the lower pane and the version column to make it visible.

find dll results

That’s it. No redeploy, no code editing, no remote debugging. You now know for sure which dll is running and where to find it. If the dll is your own and you are concerned about what might be in the code you can open the file in dotpeek to decompile it.

Getting Web API Exception Details from a HttpResponseMessage

The Problem

It’s hard to get the details of an exception from a Web Api response when calling Web Api from a C# program. (Skip to the solution if you don’t care about the background), it even handle inner exceptions!

Some background

If you are working on a Web Api project and testing with a web browser you get a wonderful error page when an exception occurs. It gives you the message, exception message, exception type and the stack trace. Pretty much all you need to get started figuring out what has gone wrong.
Exception in browser

Same thing with fiddler, get a 500 back and you’ll even be treated to a Json version of the above.

Exception in fiddler

What about calling the action method from inside a c# program? Should be easy, you just create a client, setup the query, let it rip and examine the response for a success status and then read the content to get the returned values. Great, works fine.

What if the server threw an exception like the ones shown above, I thought it would be a simple thing to call response.Exception or the like and get all the details. But easy it is not.
I rooted around in the response for a while but found nothing that was simple to use.

The Solution

Instead I have added an extension method to HttpResponseMessage to parse the details of the exception from the Json in the response.Content.

using System.Net.Http;
using System.Threading.Tasks;
using Newtonsoft.Json;

namespace SimpleWebApiClient
    public static class HttpResponseMessageExtension
        public static async Task<ExceptionResponse> ExceptionResponse(this HttpResponseMessage httpResponseMessage)
            string responseContent = await httpResponseMessage.Content.ReadAsStringAsync();
            ExceptionResponse exceptionResponse = JsonConvert.DeserializeObject<ExceptionResponse>(responseContent);
            return exceptionResponse;

    public class ExceptionResponse
        public string Message { get; set; }
        public string ExceptionMessage { get; set; }
        public string ExceptionType { get; set; }
        public string StackTrace { get; set; }
        public ExceptionResponse InnerException { get; set; }

Usage is simple.

    HttpResponseMessage response = await httpClient.GetAsync(query).ConfigureAwait(false);
    if (response.IsSuccessStatusCode)
        // return the value
    // But if an error occurred read the details           
    ExceptionResponse exceptionResponse = response.ExceptionResponse();



Why you should use IDictionary, IList, etc

When returning objects from a method try to use IList, IDictionary, etc instead of List and Dictionary. This is especially important when the method is inside a class library which you distribute.

I had to edit a method that was returning a Dictionary. Inside of which some complex work was being done to populate the dictionary.
The changes needed were going to be much easier if I could use and return a ConcurrentDictionary. See here for what I was doing.

But because the method was returning a simple Dictionary it was not as straight forward as I hoped. More work needed to be done and I had the choice of:

  1. Doing the processing in a more difficult way, not involving a ConcurrentDictionary.
  2. Moving the entries from the new ConcurrentDictionary to a simple Dictionary just before calling return.
  3. Changing the return type of the method (and the associated interface).

I went with option three, this happened to be the easiest, but it was also the best. If the code had initially been written to return IDictionary I would only have had to concern myself with required logic changes inside the method.

Here is a quick, but contrived, example of the flexibility gained by using IDictionary.

The class has a single public method which returns an IDictionary. Depending on the boolean value of getConcurrentDictionary it does its work using either as a simple Dictionary or a ConcurrentDictionary, but the caller does not need be aware of this. This gives great flexibility, the method GetDataDictionary can change its internal implementation and even the type of dictionary it returns without any negative impact on the caller.

The caller also has the flexibility to cast the IDictionary to a ConcurrentDictionary as needed.

    class GetDataWithInterfaceCollections : IGetDataWithInterfaceCollections
        public IDictionary<int, int> GetDataDictionary(int start, int end, bool getConcurrentDictionary)
            IDictionary<int, int> dictionaryToReturn;

            if (getConcurrentDictionary)
                dictionaryToReturn = GetConcurrentDictionary(start, end);
                dictionaryToReturn = GetDictionary(start, end);

            return dictionaryToReturn;

        private IDictionary<int, int> GetConcurrentDictionary(int start, int end)
            //ConcurrentDictionary offers a lot of features not available in Dictionary
            ConcurrentDictionary<int, int> dictionary = new ConcurrentDictionary<int, int>();
            for (int loop = start; loop <= end; loop++)
                dictionary.AddOrUpdate(loop, loop*10, (i, i1) => loop * 10 ); // AddOrUpdate is specific to a ConcurrentDictionary

            return dictionary;

        private IDictionary<int, int> GetDictionary(int start, int end)
            IDictionary<int, int> dictionary = new Dictionary<int, int>();
            for (int loop = start; loop <= end; loop++)
                dictionary.Add(loop, loop * 10);

            return dictionary;

Here is how to call the method shown above.

            IDictionary<int, int> concurrentDictionary = _getDataWithInterfaceCollections.GetDataDictionary(1, 10, true);

            IDictionary<int, int> simpleDictionary = _getDataWithInterfaceCollections.GetDataDictionary(11, 20, false);

            ConcurrentDictionary<int, int> explicitConcurrentDictionary = _getDataWithInterfaceCollections.GetDataDictionary(21, 30, true) as ConcurrentDictionary<int, int>; 

Using the Watch window in Visual Studio we can see more info.

Dictionary Types