Working with JSON in .NET, a better way?

Full source code available here.

Two recent experiences with C# and JSON frustrated me with how difficult it is to work JSON inside an application. I have also been learning Node.js and contrasting the ease of use there with C# is, shocking. In C# the developer is generally expected to create class structures that represent the JSON they want to produce or consume and for most of my career that has been fine, I usually had to work on quite fixed JSON, with quite fixed classes.

An example might be JSON that represents customers, orders and order items. Easy enough to make C# classes that represent them, and it having classes means its is easy to work with the customer, order or order item inside your code.

But more recently I have been working with Elasticsearch and Pulumi.

In the case of Elasticsearch, querying it is done through HTTP requests with complex JSON that can change significantly between requests. The JSON can be many layers deep and combine searching across multiple fields, sorting, paging, specifying fields to return, and other functionality.

Here is a simple query, I built this using Visual Studio Rest Client. To use this inside a C# application I have to escape all the “, {, and } characters and I have do it such a way that allows me substitute in the values I want.

This is the raw JSON -

1{
2    "query": {
3        "match_phrase_prefix": {
4            "fullName" : "Joe"
5        }
6    },
7    "from": 0,
8    "size": 2
9}

Escaping and getting it to work with a request from HttpClient took a while, and to my mind it looks awful -

 1string query = @"
 2                {{
 3                    ""query"": {{
 4                        ""match_phrase_prefix"": {{
 5                            ""fullName"" : ""{0}""
 6                        }}
 7                    }},
 8                    ""from"": {1},
 9                    ""size"": {2}
10                }}";

Here is a more realistic and not so complicated query with Elasticsearch, now try to escape that support substitutions for each value!

 1{
 2    "query":{
 3        "bool": {
 4            "must": [
 5                { "match": { "address.city": "New York" } }
 6               ,{ "match_phrase_prefix": { "lastName": "Sanders" } }
 7            ]
 8            ,"must_not": [
 9                {"range": {"dateOfBirth" : {"gte": "1980-01-01", "lte": "2000-01-01" }}}
10            ]
11        }
12    }
13    ,"sort": { "customerId" : {"order": "asc"} }
14    ,"size": 100
15    ,"from": 0 
16    ,"_source": ["firstName", "lastName"]
17}

You might rightly ask why I don’t use the provided libraries from the Elastic company. Well, I am working on a system that uses multiple languages, I do my experiments and testing with a HTTP client, and the last thing I want to do is convert everything from JSON to a significantly different formats for each programming language. JSON is also the first class citizen of Elasticsearch, I don’t want to find out later that the .NET client has not kept up with features provided by Elasticsearch. JSON is also very easy to share with colleagues.

What To Do

I am going to store my JSON in a file that becomes part of my source code, deserialize it into a dynamic object, set the values on the fields I want to change, serialize it back to a string and use that string in my requests. It is not as complicated as that might sound and way better than escaping the JSON.

Let’s take the first Elasticsearch query, here again is the raw JSON, I save it to file named ElasticSearchQuery.json.

1{
2  "query": {
3      "match_phrase_prefix": {
4          "fullName" : ""
5      }
6  },
7  "from": 0,
8  "size": 0
9}

And here is how I read, set values and serialize it again -

 1private string GetElasticSearchQuery(string fullName, int from, int size)
 2{
 3    string elasticSearchQuery = File.ReadAllText("ElasticSearchQuery.json");
 4    dynamic workableElasticSearchQuery = JsonConvert.DeserializeObject(elasticSearchQuery);
 5
 6    workableElasticSearchQuery.query.match_phrase_prefix.fullName = fullName;
 7    workableElasticSearchQuery.from = from;
 8    workableElasticSearchQuery.size = size;
 9
10    return workableElasticSearchQuery.ToString();
11}

Line 3 reads the JSON file into a string. Line 4 turns the string into a dynamic object. Lines 6,7,8 set the values I want. Line 10 returns a nice JSON string that can be used with a HttpClient to make request to Elasticsearch.

But some Elasticsearch queries are a little harder to work with because a query can include a bool. This example is in the file ElasticSearchQuery.json.

 1{
 2    "query": {
 3        "bool": {
 4            "must": [
 5                {"match_phrase_prefix": { "lastName" : "" } }
 6                ,{"match": { "address.state" : ""} } 
 7            ]
 8        }
 9    }
10}

The dynamic object will not allow us to use “bool” because it is reserved word in C#, but you can put an “@” in front of it, and now it will work -

 1private string GetElasticSearchQuery2(string lastName, string state)
 2{
 3    string elasticSearchQuery2 = File.ReadAllText("ElasticSearchQuery2.json");
 4    dynamic workableElasticSearchQuery2 = JsonConvert.DeserializeObject(elasticSearchQuery2);
 5
 6    workableElasticSearchQuery2.query.@bool.must[0].match_phrase_prefix.lastName = lastName;
 7    workableElasticSearchQuery2.query.@bool.must[1].match = new JObject(new JProperty("address.state", state));
 8
 9    return workableElasticSearchQuery2.ToString();
10}

And again the string produced can be used with a HttpClient.

Full source code available here.