Working with JSON in .NET, Infrastructure as Code with Pulumi

Full source code available here.

This is a follow up to my previous post where I used dynamic and JSON files to make querying ElasticSearch with a HttpClient much easier.

To deploy my ElasticSearch domain on AWS I used Pulumi. ElasticSearch requires a JSON policy to define the permissions. In the post linked above, I have a heavily escaped that This policy can be complex and needs values substituted into it. In the example below I need to pass in the region, account id, domain name and allowed IP address.

Here is a very simple policy with four substitutions –

"{{
""Version"": ""2012-10-17"",
""Statement"": [
    {{
        ""Action"": ""es:*"",
        ""Principal"": {{
            ""AWS"": ""*""
        }},
        ""Effect"": ""Allow"",
        ""Resource"": ""arn:aws:es:{currentRegion.Name}:{currentCallerIdentity.AccountId}:domain/{esDomainName}/*"",
        ""Condition"": {{
            ""IpAddress"": {{""aws:SourceIp"": [""{myIPAddress}""]}}
        }}
    }}
]
}}"

Just escaping this is not easy, and very prone to error. A more realistic policy would be significantly longer and would need more substitutions.

Using a JSON file
Here is what I think is an easier way. As in the previous post, the JSON file becomes part of my source code. It is deserialized into a dynamic object and the required values are set.

Here is the AWS policy as it appears in my JSON file. The resource (made up of region, account, and domain name) and IpAddress are left blank, but the structure of the policy is the same as you would paste into the AWS console.

{
    "Version": "2012-10-17",
    "Statement": [
      {
        "Effect": "Allow",
        "Principal": {
          "AWS": "*"
        },
        "Action": "es:*",
        "Resource": "",
        "Condition": {
          "IpAddress": {
            "aws:SourceIp": ""
          }
        }
      }
    ]
}

In my C# I read the file, deserialize, and set the values with simple C#.

Here is an example –

private string GetAWSElasticSearchPolicy(string region, string account, string elasticSearchDomainName, string allowedIPAddress)
{
    string blankPolicy = File.ReadAllText("AWSPolicy.json");
    dynamic awsElasticSearchPolicy = JsonConvert.DeserializeObject(blankPolicy);

    awsElasticSearchPolicy.Statement[0].Resource = $"arn:aws:es:{region}:{account}:domain/{elasticSearchDomainName}/*";
    awsElasticSearchPolicy.Statement[0].Condition.IpAddress = new JObject(new JProperty("aws:SourceIp", allowedIPAddress));

    return awsElasticSearchPolicy.ToString(); // this is correctly formatted JSON that can be used with Pulumi.
}

Line 3 reads the JSON file into a string.
Line 4 turns the string into a dynamic object.
Lines 6 & 7 set the values I want.
Line 9 returns a nice JSON string that can be used with Pulumi.

This is much cleaner than the heavily escaped version in this post.

Full source code available here.

Working with JSON in .NET, a better way?

Full source code available here.

Two recent experiences with C# and JSON frustrated me with how difficult it is to work JSON inside an application. I have also been learning Node.js and contrasting the ease of use there with C# is, shocking. In C# the developer is generally expected to create class structures that represent the JSON they want to produce or consume and for most of my career that has been fine, I usually had to work on quite fixed JSON, with quite fixed classes.

An example might be JSON that represents customers, orders and order items. Easy enough to make C# classes that represent them, and it having classes means its is easy to work with the customer, order or order item inside your code.

But more recently I have been working with ElasticSearch and Pulumi.

In the case of ElasticSearch, querying it is done through HTTP requests with complex JSON that can change significantly between requests. The JSON can be many layers deep and combine searching across multiple fields, sorting, paging, specifying fields to return, and other functionality.

Here is a simple query, I built this using Visual Studio Rest Client. To use this inside a C# application I have to escape all the “, {, and } characters and I have do it such a way that allows me substitute in the values I want.

This is the raw JSON –

{
    "query": {
        "match_phrase_prefix": {
            "fullName" : "Joe"
        }
    },
    "from": 0,
    "size": 2
}

Escaping and getting it to work with a request from HttpClient took a while, and to my mind it looks awful –

string query = @"
                {{
                    ""query"": {{
                        ""match_phrase_prefix"": {{
                            ""fullName"" : ""{0}""
                        }}
                    }},
                    ""from"": {1},
                    ""size"": {2}
                }}";

Here is a more realistic and not so complicated query with ElasticSearch, now try to escape that support substitutions for each value!

{
    "query":{
        "bool": {
            "must": [
                { "match": { "address.city": "New York" } }
               ,{ "match_phrase_prefix": { "lastName": "Sanders" } }
            ]
            ,"must_not": [
                {"range": {"dateOfBirth" : {"gte": "1980-01-01", "lte": "2000-01-01" }}}
            ]
        }
    }
    ,"sort": { "customerId" : {"order": "asc"} }
    ,"size": 100
    ,"from": 0 
    ,"_source": ["firstName", "lastName"]
}

You might rightly ask why I don’t use the provided libraries from the Elastic company. Well, I am working on a system that uses multiple languages, I do my experiments and testing with a HTTP client, and the last thing I want to do is convert everything from JSON to a significantly different formats for each programming language. JSON is also the first class citizen of ElasticSearch, I don’t want to find out later that the .NET client has not kept up with features provided by ElasticSearch. JSON is also very easy to share with colleagues.

What To Do
I am going to store my JSON in a file that becomes part of my source code, deserialize it into a dynamic object, set the values on the fields I want to change, serialize it back to a string and use that string in my requests. It is not as complicated as that might sound and way better than escaping the JSON.

Let’s take the first ElasticSearch query, here again is the raw JSON, I save it to file named ElasticSearchQuery.json.

{
  "query": {
      "match_phrase_prefix": {
          "fullName" : ""
      }
  },
  "from": 0,
  "size": 0
}

And here is how I read, set values and serialize it again –

private string GetElasticSearchQuery(string fullName, int from, int size)
{
    string elasticSearchQuery = File.ReadAllText("ElasticSearchQuery.json");
    dynamic workableElasticSearchQuery = JsonConvert.DeserializeObject(elasticSearchQuery);

    workableElasticSearchQuery.query.match_phrase_prefix.fullName = fullName;
    workableElasticSearchQuery.from = from;
    workableElasticSearchQuery.size = size;

    return workableElasticSearchQuery.ToString();
}

Line 3 reads the JSON file into a string.
Line 4 turns the string into a dynamic object.
Lines 6,7,8 set the values I want.
Line 10 returns a nice JSON string that can be used with a HttpClient to make request to ElasticSearch.

But some ElasticSearch queries are a little harder to work with because a query can include a bool. This example is in the file ElasticSearchQuery.json.

{
    "query": {
        "bool": {
            "must": [
                {"match_phrase_prefix": { "lastName" : "" } }
                ,{"match": { "address.state" : ""} } 
            ]
        }
    }
}

The dynamic object will not allow us to use “bool” because it is reserved word in C#, but you can put an “@” in front of it, and now it will work –

private string GetElasticSearchQuery2(string lastName, string state)
{
    string elasticSearchQuery2 = File.ReadAllText("ElasticSearchQuery2.json");
    dynamic workableElasticSearchQuery2 = JsonConvert.DeserializeObject(elasticSearchQuery2);

    workableElasticSearchQuery2.query.@bool.must[0].match_phrase_prefix.lastName = lastName;
    workableElasticSearchQuery2.query.@bool.must[1].match = new JObject(new JProperty("address.state", state));

    return workableElasticSearchQuery2.ToString();
}

And again the string produced can be used with a HttpClient.

Full source code available here.

Getting Started with ElasticSearch, Part 3 – Deploying to AWS with Pulumi

Full source code available here.

This is part 3 of my short introduction to ElasticSearch. In the first part I showed how to create an ElasticSearch index, mapping, and seeded it with data. In the second I used HttpClientFactory and a typed client to query the index. In this part I going to show you how to setup ElasticSearch in AWS using infrastructure as code. Be careful, AWS charges for these things.

A few months ago Pulumi added C# to their list of supported languages. If you haven’t heard of them, they are building a tool that lets you create the IaC in a familiar programming language, at the time of writing they support TypeScript, JavaScript, Python, Go and C#. Writing in a programming language makes it easy to work with things like loops and conditionals, if you are unfamiliar with IaC, those two simple things can be extremely challenging or impossible with other tools.

I’m going to write my IaC in C#.

I’m not going to walk you through installing Pulumi, their site has all the info you need for that.

The IaC Project
Once you have installed Pulimi and tested that the command works, create a new directory called ElasticSearchDeploy.

Change to that directory and run –

pulumi new aws-csharp

Follow the instructions and open the project in VS Code or Visual Studio.

Delete the MyStack.cs file.
Create a file named MyElasticSearchStack.cs.

Paste in the below code –

using Pulumi;
using ElasticSearch = Pulumi.Aws.ElasticSearch;
using Aws = Pulumi.Aws;
using Pulumi.Aws.ElasticSearch.Inputs;

class MyElasticSearchStack : Stack
{
    public MyElasticSearchStack()
    {
        string myIPAddress = "x.x.x.x" you need to put your IP address here;
        string esDomainName = "myelasticesearch";
        var config = new Config();
        var currentRegion = Output.Create(Aws.GetRegion.InvokeAsync());
        var currentCallerIdentity = Output.Create(Aws.GetCallerIdentity.InvokeAsync());
        var esDomain = new ElasticSearch.Domain(esDomainName, new ElasticSearch.DomainArgs
        {
            DomainName = esDomainName,
            ClusterConfig = new ElasticSearch.Inputs.DomainClusterConfigArgs
            {
                InstanceType = "t2.small.elasticsearch",
            },
            EbsOptions = new DomainEbsOptionsArgs()
            {
                EbsEnabled = true,
                VolumeSize = 10,
                VolumeType = "gp2"
            },
            ElasticsearchVersion = "7.7",
            AccessPolicies = Output.Tuple(currentRegion, currentCallerIdentity).Apply(values =>
            {
                var currentRegion = values.Item1;
                var currentCallerIdentity = values.Item2;
                return @$"
                {{
                    ""Version"": ""2012-10-17"",
                    ""Statement"": [
                        {{
                            ""Action"": ""es:*"",
                            ""Principal"": {{
                                ""AWS"": ""*""
                            }},
                            ""Effect"": ""Allow"",
                            ""Resource"": ""arn:aws:es:{currentRegion.Name}:{currentCallerIdentity.AccountId}:domain/{esDomainName}/*"",
                            ""Condition"": {{
                                ""IpAddress"": {{""aws:SourceIp"": [""{myIPAddress}""]}}
                            }}
                        }}
                    ]
                    }}
                ";
            }),
        });
        this.ESDomainEndpoint =  esDomain.Endpoint;
    }
    [Output]
    public Output<string> ESDomainEndpoint { get; set; }
}

Note on line 10, you need to put in the IP address you are using. Checking this with a site like https://ipstack.com/.

In Program.cs change the reference my MyStack to MyElasticSearchStack.

That’s it.

Deploying
Go to the command line, run –

pulumi up

Select ‘yes’ and then wait about 10 to 15 minutes as AWS gets your ElasticSearch domain up and running. In the output of the command you willsee the url of the ElasticSearch domain you just created, use that in the scripts from part 1 of this series.

You can also go to the AWS console, you should see something like –

There you go – ElasticSearch index creating, seeding, querying, and infrastructure as code.

In a follow up post I’ll show you how to deploy ElasticSearch with Terraform.

The JSON Problem
For those of you that dislike horribly escaped blocks of JSON inside C#, as I do, I am working on a post that will make this much nicer to look at, and to work with.

Full source code available here.

Dependency Inject a Service from Startup back to Program in .Net Core 3.1

Full source code available here.

Over the past couple of years I wrote a few posts about Dependency Injection in .Net Core 2.1, and this week I received comments from a reader telling me that some of the changes in .Net Core 3.1 mean that some of the approaches no longer work. There have been breaking changes https://docs.microsoft.com/en-us/dotnet/core/compatibility/2.2-3.1#hosting-generic-host-restricts-startup-constructor-injection

I wanted to see what would still work so I tried a few things.

You can no longer DI from Program in Startup.

But you can add a transient service and/or a singleton service to the ServiceCollection and use it within Program, and the rest of the application. You can also add a scoped service to the ServiceCollection and use it within the Program, it’s a little different from using transient and singleton so I’ll cover it in the next post.
Here’s how to use transient and singletons inside Program.
Create two services, creatively named ServiceOne and ServiceTwo and have them implement interfaces.

public class ServiceOne : IServiceOne
{
	public static int staticCounter;

	public ServiceOne()
	{
		staticCounter++;
	}
   //snip…

public class ServiceTwo : IServiceTwo
{
	public static int staticCounter;

	public ServiceTwo()
	{
		staticCounter++;
	}
	//snip…

I added a static counter to make it easy to see how many times the constructor is called.
For the transient one I expect it to increment every time the service is injected, for the singleton I expect it to remain at 1 for the lifetime of the application.

In Program.cs I split up CreateHostBuilder(args).Build call from the subsequent .Run().

public static void Main(string[] args)
{
	IHost host = CreateHostBuilder(args).Build();
	DoSomethingWithTheTransientService(host.Services);
	DoSomethingWithTheSingletonService(host.Services);
	host.Run();
}

The CreateHostBuilder() method is not changed –

public static IHostBuilder CreateHostBuilder(string[] args) =>
	Host.CreateDefaultBuilder(args)
	.ConfigureWebHostDefaults(webBuilder =>
	{
		webBuilder.UseStartup<Startup>();
	});

And I have two methods that use the ServiceCollection to access the registered services –

private static void DoSomethingWithTheTransientService(IServiceProvider serviceProvider)
{
	Console.WriteLine("Calling the transient service");

	var serviceOne = serviceProvider.GetService<IServiceOne>();
	Console.WriteLine(serviceOne.StaticCounter());
	Console.WriteLine(serviceOne.GetHashCode());
}

private static void DoSomethingWithTheSingletonService(IServiceProvider serviceProvider)
{
	Console.WriteLine("Calling the singleton service");

	var serviceTwo = serviceProvider.GetService<IServiceTwo>();
	Console.WriteLine(serviceTwo.StaticCounter());
	Console.WriteLine(serviceTwo.GetHashCode());
}

To make what’s happening more obvious I added the services to the constructor call of the WeatherForecastController() and added the counter and hash codes to the data returned by the action method.

public class WeatherForecastController : ControllerBase
{
	private IServiceOne _serviceOne;
	private IServiceTwo _serviceTwo;
	public WeatherForecastController(IServiceOne serviceOne, IServiceTwo serviceTwo)
	{
		_serviceOne = serviceOne;
		_serviceTwo = serviceTwo;
	}

For completeness, here is the ConfigureServices method in Startup.cs

public void ConfigureServices(IServiceCollection services)
{
    services.AddTransient<IServiceOne, ServiceOne>();
    services.AddSingleton<IServiceTwo, ServiceTwo>();
    services.AddControllers();
}

Put some breakpoints in Program Main(), DoSomethingWithTheTransientService(), DoSomethingWithTheSingletonService() and in the WeatherForecastController.Get().

Start the application and browse to http://localhost:5000/weatherforecast to see what happens.

I’m going to follow up on this post with a version that shows how to use scoped dependencies in Startup .NET 5.

Full source code available here.