Working with JSON in .NET, Infrastructure as Code with Pulumi

Full source code available here.

This is a follow up to my previous post where I used dynamic and JSON files to make querying ElasticSearch with a HttpClient much easier.

To deploy my ElasticSearch domain on AWS I used Pulumi. ElasticSearch requires a JSON policy to define the permissions. In the post linked above, I have a heavily escaped that This policy can be complex and needs values substituted into it. In the example below I need to pass in the region, account id, domain name and allowed IP address.

Here is a very simple policy with four substitutions –

"{{
""Version"": ""2012-10-17"",
""Statement"": [
    {{
        ""Action"": ""es:*"",
        ""Principal"": {{
            ""AWS"": ""*""
        }},
        ""Effect"": ""Allow"",
        ""Resource"": ""arn:aws:es:{currentRegion.Name}:{currentCallerIdentity.AccountId}:domain/{esDomainName}/*"",
        ""Condition"": {{
            ""IpAddress"": {{""aws:SourceIp"": [""{myIPAddress}""]}}
        }}
    }}
]
}}"

Just escaping this is not easy, and very prone to error. A more realistic policy would be significantly longer and would need more substitutions.

Using a JSON file
Here is what I think is an easier way. As in the previous post, the JSON file becomes part of my source code. It is deserialized into a dynamic object and the required values are set.

Here is the AWS policy as it appears in my JSON file. The resource (made up of region, account, and domain name) and IpAddress are left blank, but the structure of the policy is the same as you would paste into the AWS console.

{
    "Version": "2012-10-17",
    "Statement": [
      {
        "Effect": "Allow",
        "Principal": {
          "AWS": "*"
        },
        "Action": "es:*",
        "Resource": "",
        "Condition": {
          "IpAddress": {
            "aws:SourceIp": ""
          }
        }
      }
    ]
}

In my C# I read the file, deserialize, and set the values with simple C#.

Here is an example –

private string GetAWSElasticSearchPolicy(string region, string account, string elasticSearchDomainName, string allowedIPAddress)
{
    string blankPolicy = File.ReadAllText("AWSPolicy.json");
    dynamic awsElasticSearchPolicy = JsonConvert.DeserializeObject(blankPolicy);

    awsElasticSearchPolicy.Statement[0].Resource = $"arn:aws:es:{region}:{account}:domain/{elasticSearchDomainName}/*";
    awsElasticSearchPolicy.Statement[0].Condition.IpAddress = new JObject(new JProperty("aws:SourceIp", allowedIPAddress));

    return awsElasticSearchPolicy.ToString(); // this is correctly formatted JSON that can be used with Pulumi.
}

Line 3 reads the JSON file into a string.
Line 4 turns the string into a dynamic object.
Lines 6 & 7 set the values I want.
Line 9 returns a nice JSON string that can be used with Pulumi.

This is much cleaner than the heavily escaped version in this post.

Full source code available here.

Working with JSON in .NET, a better way?

Full source code available here.

Two recent experiences with C# and JSON frustrated me with how difficult it is to work JSON inside an application. I have also been learning Node.js and contrasting the ease of use there with C# is, shocking. In C# the developer is generally expected to create class structures that represent the JSON they want to produce or consume and for most of my career that has been fine, I usually had to work on quite fixed JSON, with quite fixed classes.

An example might be JSON that represents customers, orders and order items. Easy enough to make C# classes that represent them, and it having classes means its is easy to work with the customer, order or order item inside your code.

But more recently I have been working with ElasticSearch and Pulumi.

In the case of ElasticSearch, querying it is done through HTTP requests with complex JSON that can change significantly between requests. The JSON can be many layers deep and combine searching across multiple fields, sorting, paging, specifying fields to return, and other functionality.

Here is a simple query, I built this using Visual Studio Rest Client. To use this inside a C# application I have to escape all the “, {, and } characters and I have do it such a way that allows me substitute in the values I want.

This is the raw JSON –

{
    "query": {
        "match_phrase_prefix": {
            "fullName" : "Joe"
        }
    },
    "from": 0,
    "size": 2
}

Escaping and getting it to work with a request from HttpClient took a while, and to my mind it looks awful –

string query = @"
                {{
                    ""query"": {{
                        ""match_phrase_prefix"": {{
                            ""fullName"" : ""{0}""
                        }}
                    }},
                    ""from"": {1},
                    ""size"": {2}
                }}";

Here is a more realistic and not so complicated query with ElasticSearch, now try to escape that support substitutions for each value!

{
    "query":{
        "bool": {
            "must": [
                { "match": { "address.city": "New York" } }
               ,{ "match_phrase_prefix": { "lastName": "Sanders" } }
            ]
            ,"must_not": [
                {"range": {"dateOfBirth" : {"gte": "1980-01-01", "lte": "2000-01-01" }}}
            ]
        }
    }
    ,"sort": { "customerId" : {"order": "asc"} }
    ,"size": 100
    ,"from": 0 
    ,"_source": ["firstName", "lastName"]
}

You might rightly ask why I don’t use the provided libraries from the Elastic company. Well, I am working on a system that uses multiple languages, I do my experiments and testing with a HTTP client, and the last thing I want to do is convert everything from JSON to a significantly different formats for each programming language. JSON is also the first class citizen of ElasticSearch, I don’t want to find out later that the .NET client has not kept up with features provided by ElasticSearch. JSON is also very easy to share with colleagues.

What To Do
I am going to store my JSON in a file that becomes part of my source code, deserialize it into a dynamic object, set the values on the fields I want to change, serialize it back to a string and use that string in my requests. It is not as complicated as that might sound and way better than escaping the JSON.

Let’s take the first ElasticSearch query, here again is the raw JSON, I save it to file named ElasticSearchQuery.json.

{
  "query": {
      "match_phrase_prefix": {
          "fullName" : ""
      }
  },
  "from": 0,
  "size": 0
}

And here is how I read, set values and serialize it again –

private string GetElasticSearchQuery(string fullName, int from, int size)
{
    string elasticSearchQuery = File.ReadAllText("ElasticSearchQuery.json");
    dynamic workableElasticSearchQuery = JsonConvert.DeserializeObject(elasticSearchQuery);

    workableElasticSearchQuery.query.match_phrase_prefix.fullName = fullName;
    workableElasticSearchQuery.from = from;
    workableElasticSearchQuery.size = size;

    return workableElasticSearchQuery.ToString();
}

Line 3 reads the JSON file into a string.
Line 4 turns the string into a dynamic object.
Lines 6,7,8 set the values I want.
Line 10 returns a nice JSON string that can be used with a HttpClient to make request to ElasticSearch.

But some ElasticSearch queries are a little harder to work with because a query can include a bool. This example is in the file ElasticSearchQuery.json.

{
    "query": {
        "bool": {
            "must": [
                {"match_phrase_prefix": { "lastName" : "" } }
                ,{"match": { "address.state" : ""} } 
            ]
        }
    }
}

The dynamic object will not allow us to use “bool” because it is reserved word in C#, but you can put an “@” in front of it, and now it will work –

private string GetElasticSearchQuery2(string lastName, string state)
{
    string elasticSearchQuery2 = File.ReadAllText("ElasticSearchQuery2.json");
    dynamic workableElasticSearchQuery2 = JsonConvert.DeserializeObject(elasticSearchQuery2);

    workableElasticSearchQuery2.query.@bool.must[0].match_phrase_prefix.lastName = lastName;
    workableElasticSearchQuery2.query.@bool.must[1].match = new JObject(new JProperty("address.state", state));

    return workableElasticSearchQuery2.ToString();
}

And again the string produced can be used with a HttpClient.

Full source code available here.

Getting Web API Exception Details from a HttpResponseMessage

The Problem

It’s hard to get the details of an exception from a Web Api response when calling Web Api from a C# program. (Skip to the solution if you don’t care about the background), it even handles inner exceptions!

Some background

If you are working on a Web Api project and testing with a web browser you get a wonderful error page when an exception occurs. It gives you the message, exception message, exception type and the stack trace. Pretty much all you need to get started figuring out what has gone wrong.
Exception in browser

Same thing with fiddler, get a 500 back and you’ll even be treated to a Json version of the above.

Exception in fiddler

What about calling the action method from inside a c# program? Should be easy, you just create a client, setup the query, let it rip and examine the response for a success status and then read the content to get the returned values. Great, works fine.

What if the server threw an exception like the ones shown above, I thought it would be a simple thing to call response.Exception or the like and get all the details. But easy it is not.
I rooted around in the response for a while but found nothing that was simple to use.

The Solution

Instead I have added an extension method to HttpResponseMessage to parse the details of the exception from the Json in the response.Content.

using System.Net.Http;
using System.Threading.Tasks;
using Newtonsoft.Json;

namespace SimpleWebApiClient
{
    public static class HttpResponseMessageExtension
    {
        public static async Task<ExceptionResponse> ExceptionResponse(this HttpResponseMessage httpResponseMessage)
        {
            string responseContent = await httpResponseMessage.Content.ReadAsStringAsync();
            ExceptionResponse exceptionResponse = JsonConvert.DeserializeObject<ExceptionResponse>(responseContent);
            return exceptionResponse;
        }
    }

    public class ExceptionResponse
    {
        public string Message { get; set; }
        public string ExceptionMessage { get; set; }
        public string ExceptionType { get; set; }
        public string StackTrace { get; set; }
        public ExceptionResponse InnerException { get; set; }
    }
}

Usage is simple.

    //snip
    HttpResponseMessage response = await httpClient.GetAsync(query).ConfigureAwait(false);
    
    if (response.IsSuccessStatusCode)
        // return the value
    
    // But if an error occurred read the details           
    ExceptionResponse exceptionResponse = response.ExceptionResponse();

    //snip

 

Using JSON to store application preferences and configuration

Download full source code.

Storing configuration preferences in the database is not a hard task. It usually involves storing the name of the preference, the value and the type in the database. Then when retrieving the data you perform some sort of cast.

The casts tend to be a bit painful, but it works.

This is the call –

 
bool displayWidget = (bool) Convert.ChangeType(_preferenceManager.BadWayToCastToGetPreference("DisplayWidget"), typeof(bool));

And this is the called method –

 
        public object BadWayToCastToGetPreference<T>(string name)
        {
            var preference = Convert.ChangeType(_context.Preferences.SingleOrDefault(p => p.Name == name).Value, typeof(T));
            return preference;
        }

But this quickly falls apart when you want to store a list of something, this requires a custom way of generating a string that represents the list. A common way I’ve seen is to store the values as a pipe delimited list.

For example if you wanted to store three cities
Boston, New York and Seattle
they would become “Boston|New York|Seattle” when stored in the database.

Now you need a write code to turn that back into a list of strings.

The problem gets worse when you want to store some custom types, for example an emergency contact shaped like –

    public class EmergencyContact
    {
        public int Priority { get; set; }
        public string EmailAddress { get; set; }
    }

Or worse if you want to store a list of emergency contacts.

Rather than suffering all that pain, just use Json to serialize and deserialize the values. This makes life a lot easier.

I have a simple method to create the preference object that gets store in the database –

        private Preference CreatePreference(string name, string value, string type)
        {
            var preference = new Preference
            {
                Name = name,
                Value = value,
                Type = type,
                PreferenceID = Guid.NewGuid()
            };
            return preference;
        }

CreatePreference gets called like this for a simple string –

            string defaultEmail = "admin@example.com";
            Preference preference1 = CreatePreference("DefaultEmail", JsonConvert.SerializeObject(defaultEmail), defaultEmail.GetType().ToString());

And it’s the same when storing a more complex type.

            EmergencyContact[] secondaryEmergencyContacts = new[]
            {
                new EmergencyContact{ EmailAddress = "tom@example.com", Priority = 1},
                new EmergencyContact{ EmailAddress = "dick@example.com", Priority = 2},
                new EmergencyContact{ EmailAddress = "harry@example.com", Priority = 3},
            };
            Preference preference6 = CreatePreference("SecondaryEmergencyContacts",
                JsonConvert.SerializeObject(secondaryEmergencyContacts), secondaryEmergencyContacts.GetType().ToString());

To store these preferences in the database, it’s just a simple Entity Framework call –

            _context.Preferences.Add(preference1);
            _context.Preferences.Add(preference6);
            _context.SaveChanges();

And then to get the preferences back out of the database you call –

        public T GetPreference<T>(string name)
        {
            var preference =
                JsonConvert.DeserializeObject<T>(_context.Preferences.SingleOrDefault(p => p.Name == name).Value);
            return preference;
        }

Some might complain that it is slower than direct casts, and yes, it probably is. But if you haven’t measured it you shouldn’t optimize it. You can cache always cache preferences. And it is a lot neater than the alternative.

Download full source code.