C# and AWS Lambdas, Part 5 – Updating the Zip in S3 and Updating the Running Lambda, with Pulumi IaC

Full source code available here.

This post pulls together a few threads I’ve been working on – creation of lambda to run .NET, storing the zip in S3, and updating the .NET lambda when the zip in S3 is updated.

This one took quite while to put together -the permissions, roles and policies were not obvious and I hope it will be of help to you.
This is not a blog post on CI/CD, I am cutting corners by using Pulumi to upload the zip files initially, and then use the AWS command line to send zips to S3. In a future set of posts I will show how to use GitHub Actions to build the infrastructure, and to compile and deploy the .NET lambda directly to S3 from GitHub.

The idea
I want to have a lambda that runs .NET code stored in a zip file in S3. I want to be able to update the zip, and have the .NET lambda run the code in the new zip. I had hoped this would be a little tick box on the lambda, but sadly there is no such box.

Instead I have a second lambda (referred to as the updater lambda) that is triggered by an update on specified bucket in S3. This updater lambda in turn calls an update on the .NET lambda and within a few seconds the .NET lambda will be running the new code. Doesn’t sound easy, but I didn’t think it would be too hard, but take a look at the number of resources needed.

What’s needed
A role to run .NET lambda.
A role to run lambda that updates the .NET lambda, I’m calling this the updater.
A policy to give the updater permissions to update the .NET lambda and S3.
A policy attachment for the .NET lambda.
A policy attachment for the updater lambda.
An S3 bucket.
An S3 bucket object.
The .NET lambda pointing at the bucket and bucket object.
The zip file for for the .NET lambda.
The updater lambda with variables passed in to verify the update of the .NET lambda.
The zip file for the updater lambda – Node.js.
Permission for the bucket to call the updated lambda.
A bucket notification with attached permissions.
Reduce the bucket accessible to the public (no necessary, but good).

That’s a lot more than the tick box I was hoping for.

The code

using Pulumi;
using S3 = Pulumi.Aws.S3;
using Aws = Pulumi.Aws;

class MyStack : Stack
{
    public MyStack()
    {
        string resource_prefix = "PulumiHelloWorldAutoUpdate";

        var lambdaHelloWorldRole = new Aws.Iam.Role($"{resource_prefix}_LambdaRole", new Aws.Iam.RoleArgs
        {
            AssumeRolePolicy =
@"{
    ""Version"": ""2012-10-17"",
    ""Statement"": [
        {
        ""Action"": ""sts:AssumeRole"",
        ""Principal"": {
            ""Service"": ""lambda.amazonaws.com""
        },
        ""Effect"": ""Allow"",
        ""Sid"": """"
        }
    ]
}",
        });

        var lambdaUpdateRole = new Aws.Iam.Role($"{resource_prefix}_LambdaUpdateRole", new Aws.Iam.RoleArgs
        {
            AssumeRolePolicy =
@"{
    ""Version"": ""2012-10-17"",
    ""Statement"": [
        {
        ""Action"": ""sts:AssumeRole"",
        ""Principal"": {
            ""Service"": ""lambda.amazonaws.com""
        },
        ""Effect"": ""Allow"",
        ""Sid"": """"
        }
    ]
}",
        });

        // gives the lamaba permissions to other lambdas and s3 - too many permissions, but this is a demo.
        var lambdaUpdatePolicy = new Aws.Iam.Policy($"{resource_prefix}_S3_Lambda_Policy", new Aws.Iam.PolicyArgs{
            PolicyDocument = 
@"{
    ""Version"": ""2012-10-17"",
    ""Statement"": [
        {
            ""Sid"": """",
            ""Effect"": ""Allow"",
            ""Action"": [
                ""s3:*"",
                ""logs:*"",
                ""lambda:*""
            ],
            ""Resource"": ""*""
        }
    ]
}"
        });

        // attach a simple policy to the hello world lambda.
        var lambdaHelloWorldAttachment = new Aws.Iam.PolicyAttachment($"{resource_prefix}_LambdaHelloWorldPolicyAttachment", new Aws.Iam.PolicyAttachmentArgs
        {
            Roles =
            {
                lambdaHelloWorldRole.Name
            },
            PolicyArn = Aws.Iam.ManagedPolicy.AWSLambdaBasicExecutionRole.ToString(),
        });

        // attach the custom policy to the role that runs the update lambda.
        var lambdaUpdateAttachment = new Aws.Iam.PolicyAttachment($"{resource_prefix}_LambdaUpdatePolicyAttachment", new Aws.Iam.PolicyAttachmentArgs
        {
            Roles =
            {
                lambdaUpdateRole.Name
            },
            PolicyArn = lambdaUpdatePolicy.Arn,
        });

        var s3Bucket = new S3.Bucket($"{resource_prefix}_S3Bucket", new S3.BucketArgs
        {
            BucketName = "pulumi-hello-world-auto-update-s3-bucket",
            Versioning = new Aws.S3.Inputs.BucketVersioningArgs
            {
                Enabled = true,
            },
            Acl = "private"
        });

        var s3BucketObject = new S3.BucketObject($"{resource_prefix}_ZipFile", new S3.BucketObjectArgs
        {
            Bucket = s3Bucket.BucketName.Apply(name => name),
            Acl = "private",
            Source = new FileArchive("./Lambdas/helloworld_no_date/helloworld.zip"),
            Key = "helloworld.zip"
        });

        // this is the lambda that runs .NET code
        var lambdaHelloWorldFunction = new Aws.Lambda.Function($"{resource_prefix}_LambdaHelloWorldFunction", new Aws.Lambda.FunctionArgs
        {
            Handler = "HelloWorldLambda::HelloWorldLambda.Function::FunctionHandler",
            MemorySize = 128,
            Publish = false,
            ReservedConcurrentExecutions = -1,
            Role = lambdaHelloWorldRole.Arn,
            Runtime = Aws.Lambda.Runtime.DotnetCore3d1,
            Timeout = 4,
            S3Bucket = s3Bucket.BucketName,
            S3Key = s3BucketObject.Key
        });

        // this is the lambda triggered by an upload to S3 and replaces the zip in the above lambda
        var lambdaUpdateFunction = new Aws.Lambda.Function($"{resource_prefix}_LambdaUpdateFunction", new Aws.Lambda.FunctionArgs
        {
            Handler = "index.handler",
            MemorySize = 128,
            Publish = false,
            ReservedConcurrentExecutions = -1,
            Role = lambdaUpdateRole.Arn,
            Runtime = Aws.Lambda.Runtime.NodeJS14dX,
            Timeout = 4,
            Code = new FileArchive("./Lambdas/LambdaUpdater/index.zip"),
            Environment = new Aws.Lambda.Inputs.FunctionEnvironmentArgs
            {
                Variables = new InputMap<string> {{"s3Bucket", s3Bucket.BucketName}, {"s3Key", "helloworld.zip"}, {"functionToUpdate", lambdaHelloWorldFunction.Name}}
            }
        });

        var s3BucketPermissionToCallLambda = new Aws.Lambda.Permission($"{resource_prefix}_S3BucketPermissionToCallLambda", new Aws.Lambda.PermissionArgs
        {
            Action = "lambda:InvokeFunction",
            Function = lambdaUpdateFunction.Arn,
            Principal = "s3.amazonaws.com",
            SourceArn = s3Bucket.Arn,
        });

        var bucketNotification = new S3.BucketNotification($"{resource_prefix}_S3BucketNotification", new Aws.S3.BucketNotificationArgs
        {
            Bucket = s3Bucket.Id,
            LambdaFunctions = 
            {
                new Aws.S3.Inputs.BucketNotificationLambdaFunctionArgs
                {
                    LambdaFunctionArn = lambdaUpdateFunction.Arn,
                    Events = 
                    {
                        "s3:ObjectCreated:*",
                    },
                }
            },
        }, new CustomResourceOptions
        {
            DependsOn = 
            {
                s3BucketPermissionToCallLambda,
            },
        });

        // keep the contents bucket private
        var bucketPublicAccessBlock = new S3.BucketPublicAccessBlock($"{resource_prefix}_PublicAccessBlock", new S3.BucketPublicAccessBlockArgs
        {
            Bucket = s3Bucket.Id,
            BlockPublicAcls = false,  // leaving these two false because I need them this way 
            IgnorePublicAcls = false, // for a post about GitHub Actions that I'm working on
            BlockPublicPolicy = true,
            RestrictPublicBuckets = true
        });

        this.LambdaUpdateFunctionName = lambdaUpdateFunction.Name;
        this.LambdaHelloWorldFunctionName = lambdaHelloWorldFunction.Name;
        this.S3Bucket = s3Bucket.BucketName;
        this.S3Key = s3BucketObject.Key;
    }

    [Output]
    public Output<string> LambdaUpdateFunctionName { get; set; }

    [Output]
    public Output<string> LambdaHelloWorldFunctionName { get; set; }

    [Output]
    public Output<string> S3Bucket {get;set;}

    [Output]
    public Output<string> S3Key {get;set;}
}

Below is the code of the updater lambda. The if checks to make sure that the lambda.updateFunctionCode(..) runs only if the expected file in S3 is updated. The environmental variables were passed in via the Pulumi code above.

const AWS = require('aws-sdk');
const lambda = new AWS.Lambda();

exports.handler = (event) => {
    
    if (event.Records[0].s3.bucket.name == process.env.s3Bucket && event.Records[0].s3.object.key == process.env.s3Key)
    {
        var params = {
            FunctionName: process.env.functionToUpdate,
            S3Bucket: event.Records[0].s3.bucket.name, 
            S3Key: event.Records[0].s3.object.key
        };
        
        // https://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/Lambda.html#updateFunctionCode-property
        lambda.updateFunctionCode(params, function(err, data) {
            if (err) // an error occurred
            {
                console.log(err, err.stack);
            }
            else
            {   
                console.log(data);  
            }
        });
    }
    else
    {
        console.log("bucket name or s3 key did not match expected values.");
        console.log("expected bucket name: " + process.env.s3Bucket + " actual: " + event.Records[0].s3.bucket.name);
        console.log("expected s3 key: " + process.env.s3Key + " actual: " + event.Records[0].s3.object.key);
    }
    console.log("Exiting");
};

The zip attached to this blog post has all the source code needed, you don’t have to add or change anything.

Running it

From the console, run –

pulumi up 

At the end you should see something like this –

Note the outputs. These are the name of your lambdas and the s3 bucket and key –

Outputs:
    LambdaHelloWorldFunctionName: "PulumiHelloWorldAutoUpdate_LambdaHelloWorldFunction-???????"
    LambdaUpdateFunctionName    : "PulumiHelloWorldAutoUpdate_LambdaUpdateFunction-???????"
    S3Bucket                    : "pulumi-hello-world-auto-update-s3-bucket"
    S3Key                       : "helloworld.zip"

Go the aws console, and test the lambda as shown in part 1 of this blog series.

You should get output like this – “HELLO WORLD”.

Updating the zip in S3
Now to try out the real functionality, updating the zip in S3 and see if it runs in the .NET lambda.

In the attached source there is a Lambdas directory with two subdirectories – helloworld_no_date and helloworld_with_date. They contain two variations of the .NET application. The first converts the input text to uppercase, the second converts the input text to uppercase and adds the current date and time.

You can run the below commands to upload each zip file and try out the lambda. A few seconds after you upload, the .NET lambda will use that zip.

// no date
aws s3 cp ./Lambdas/helloworld_no_date/helloworld.zip s3://pulumi-hello-world-auto-update-s3-bucket/helloworld.zip
// with date
aws s3 cp ./Lambdas/helloworld_with_date/helloworld.zip s3://pulumi-hello-world-auto-update-s3-bucket/helloworld.zip

If you don’t want to go into the AWS UI console to try out the lambda you can invoke it from the command line, but you need to swap the function name below for the one in the output of the pulumi up command –

aws lambda invoke --function-name PulumiHelloWorldAutoUpdate_LambdaHelloWorldFunction-??????? --payload '"hello world"' /dev/stdout

This was long tough one, but I’ve learned a lot about AWS, Pulumi, and even GitHub Actions (more on that soon).

Full source code available here.

C# and AWS Lambdas, Part 4 – Storing the Zip in S3, Setup with Pulumi IaC

Full source code available here.

In the previous post I showed how to use Pulumi to create a lambda, API gateway and upload a zip of Web API application directly to the lambda.

In the post I’m going to use S3 to store the zip of the of a simple .NET application (not a Web API app) and point the lambda at it, brining all the resources up using Pulumi. One drawback of using S3 to store the zip for a lambda is that when you update the zip in S3, AWS doesn’t deploy the new zip to the lambda, but I will show a way of handling that in the next blog post.

Every time I write these posts it feels like a lot of work to set up the infrastructure, learn about the AWS components, policies, etc. But then when I run pulumi destroy -y and pulumi up -y as I make changes I appreciate the speed and predictability.

What’s needed

A lambda role.
A policy to execute the lambda.
An S3 bucket.
Block all public access to the bucket – optional.
Put zip in the bucket.
The lambda function pointing to the zip in the bucket.

A few of these are the same as in the previous post, but the S3 resources are new.

Note that I included BucketPublicAccessBlock section, this makes the S3 bucket more secure than it would be under the default settings.

The code
Here is the code that sets it all up. I’ve included a helloworld.zip file in the attached source code zip. This is NOT how you would normally upload the zip to S3, but it’s easy for demonstration purposes. In a more realistic scenario, you would a CI/CD pipeline to compile the code and drop it in S3.

using Pulumi;
using S3 = Pulumi.Aws.S3;
using Aws = Pulumi.Aws;
class MyStack : Stack
{
    public MyStack()
    {

        var lambdaRole = new Aws.Iam.Role("PulumiHelloWorld_LambdaRole", new Aws.Iam.RoleArgs
        {
            AssumeRolePolicy =
@"{
    ""Version"": ""2012-10-17"",
    ""Statement"": [
        {
        ""Action"": ""sts:AssumeRole"",
        ""Principal"": {
            ""Service"": ""lambda.amazonaws.com""
        },
        ""Effect"": ""Allow"",
        ""Sid"": """"
        }
    ]
}",
        });

        var lambdaPolicyAttachment = new Aws.Iam.PolicyAttachment("PulumiHelloWorld_LambdaPolicyAttachment", new Aws.Iam.PolicyAttachmentArgs
        {
            Roles =
            {
                lambdaRole.Name
            },
            PolicyArn = Aws.Iam.ManagedPolicy.AWSLambdaBasicExecutionRole.ToString(),
        });

        var bucket = new S3.Bucket("PulumiHelloWorld_S3Bucket", new S3.BucketArgs
        {
            BucketName = "pulumi-hello-world-s3-bucket",
            Acl = "private"
        });

        var bucketPublicAccessBlock = new S3.BucketPublicAccessBlock("PulumiHelloWorld_PublicAccessBlock", new S3.BucketPublicAccessBlockArgs
        {
            Bucket = bucket.Id,
            BlockPublicAcls = true,
            BlockPublicPolicy = true,
            RestrictPublicBuckets = true,
            IgnorePublicAcls = true
        });

        var bucketObject = new S3.BucketObject("PulumiHelloWorld_ZipFile", new S3.BucketObjectArgs
        {
            Bucket = bucket.BucketName.Apply(name => name),
            Acl = "private",
            Source = new FileArchive("helloworld.zip")
        });

        var lambdaFunction = new Aws.Lambda.Function("PulumiHelloWorld_LambdaFunction", new Aws.Lambda.FunctionArgs
        {
            Handler = "HelloWorldLambda::HelloWorldLambda.Function::FunctionHandler",
            MemorySize = 128,
            Publish = false,
            ReservedConcurrentExecutions = -1,
            Role = lambdaRole.Arn,
            Runtime = Aws.Lambda.Runtime.DotnetCore3d1,
            Timeout = 4,
            S3Bucket = bucket.BucketName,
            S3Key = bucketObject.Key
        });

        this.LambdaFunctionName = lambdaFunction.Name;
    }

    [Output]
    public Output<string> LambdaFunctionName { get; set; }
}

To test out the lambda, open it in the AWS console, navigate to the lambda and add a test as described in part 1 of this series of posts.

Full source code available here.

C# and AWS Lambdas, Part 3 – Pulumi IaC for Web API and an API Gateway

Full source code available here.

In this the third in a series of posts on using .NET in AWS lambdas I build on the previous where I connected a Http Gateway to a lambda running a Web API application. In that post I built the infrastructure by hand, i.e. via the GUI, pointing and clicking.

In this post I will show how to build everything with Pulumi. I’ve written a few posts on Pulimi before, you can find them here.

As with all IaC, the hard part isn’t the IaC itself, it’s knowing what to do on the platform you are deploying to.
In the previous post I worked out all the required steps and components and got everything working. But the UI takes care of some things that you have to explicitly do in IaC and this is where the challenge is as you get familiar with the platform you are working on.

What’s needed
When you add up all the components needed, it feels like a lot, but eighty lines of C# code sets it all up.

IAM role for the lambda.
Policy attachment for the above role letting it execute the lambda.
The lambda function. This where the Web API code runs.
A Http Gateway.
An integration between the Http Gateway and the lambda.
A route on the Http Gateway that forwards to the integration.
A stage on the Http Gateway.
Permissions for the Http Gateway to execute the lambda.

That last one was not obvious, thank you to Piers Karsenbarg at Pulumi for his assistance.

The code
You need to have Pulumi installed, check their site for instructions.

Here is the stack to setup all the components listed above.

using Pulumi;
using Aws = Pulumi.Aws;

class MyStack : Stack
{
    public MyStack()
    {

        var lambdaRole = new Aws.Iam.Role("PulumiWebApiGateway_LambdaRole", new Aws.Iam.RoleArgs
        {
            AssumeRolePolicy = 
@"{
    ""Version"": ""2012-10-17"",
    ""Statement"": [
        {
        ""Action"": ""sts:AssumeRole"",
        ""Principal"": {
            ""Service"": ""lambda.amazonaws.com""
        },
        ""Effect"": ""Allow"",
        ""Sid"": """"
        }
    ]
}",
        });

        var lambdaPolicyAttachment = new Aws.Iam.PolicyAttachment("PulumiWebApiGateway_LambdaPolicyAttachment", new Aws.Iam.PolicyAttachmentArgs
        {
            Roles =
            {
                lambdaRole.Name
            },
            PolicyArn = Aws.Iam.ManagedPolicy.AWSLambdaBasicExecutionRole.ToString(), 
        });

        var lambdaFunction = new Aws.Lambda.Function("PulumiWebApiGateway_LambdaFunction", new Aws.Lambda.FunctionArgs
        {
            Handler = "WebAPILambda::WebAPILambda.LambdaEntryPoint::FunctionHandlerAsync",
            MemorySize = 128,
            Publish = false,
            ReservedConcurrentExecutions = -1,
            Role = lambdaRole.Arn,
            Runtime = Aws.Lambda.Runtime.DotnetCore3d1,
            Timeout = 4,
            Code = new FileArchive("WebAPILambda.zip"), // I put the zip file in the same dir as this code for this demo, but you should not do this.
        });
        System.Console.WriteLine(Aws.Iam.ManagedPolicy.AWSLambdaBasicExecutionRole.ToString());

        var httpApiGateway = new Pulumi.Aws.ApiGatewayV2.Api("PulumiWebApiGateway_ApiGateway", new Pulumi.Aws.ApiGatewayV2.ApiArgs
        {
            ProtocolType = "HTTP",
            RouteSelectionExpression = "${request.method} ${request.path}",
        });

        var httpApiGateway_LambdaIntegration = new Pulumi.Aws.ApiGatewayV2.Integration("PulumiWebApiGateway_ApiGatewayIntegration", new Pulumi.Aws.ApiGatewayV2.IntegrationArgs
        {
            ApiId = httpApiGateway.Id,
            IntegrationType = "AWS_PROXY",
            IntegrationMethod = "POST",
            IntegrationUri = lambdaFunction.Arn,
            PayloadFormatVersion = "2.0",
            TimeoutMilliseconds = 30000,
        });

        var httpApiGatewayRoute = new Pulumi.Aws.ApiGatewayV2.Route("PulumiWebApiGateway_ApiGatewayRoute", new Pulumi.Aws.ApiGatewayV2.RouteArgs
        {
            ApiId = httpApiGateway.Id,
            RouteKey = "$default",
            Target = httpApiGateway_LambdaIntegration.Id.Apply(id => $"integrations/{id}"),
        });

        var httpApiGatewayStage = new Pulumi.Aws.ApiGatewayV2.Stage("PulumiWebApiGateway_ApiGatewayStage", new Pulumi.Aws.ApiGatewayV2.StageArgs
        {
            ApiId = httpApiGateway.Id,
            AutoDeploy = true,
            Name = "$default",
        });

        var lambdaPermissionsForApiGateway = new Aws.Lambda.Permission("PulumiWebApiGateway_LambdaPermission", new Aws.Lambda.PermissionArgs
        {
            Action = "lambda:InvokeFunction",
            Function = lambdaFunction.Name,
            Principal = "apigateway.amazonaws.com",
            SourceArn = Output.Format($"{httpApiGateway.ExecutionArn}/*") // note it's the ExecutionArn.
            // SourceArn = httpApiGateway.ExecutionArn.Apply(arn => $"{arn}/*") // this is another way of doing the same thing
        });

        this.ApiEndpoint = httpApiGateway.ApiEndpoint.Apply(endpoint =>  $"{endpoint}/api/values");
    }

    [Output]
    public Output<string> ApiEndpoint { get; set; }
}

Run –

pulumi up

This is will show you what is going to be deployed, if it looks ok select yes.

After a short wait you should see something like this the below image indicating that everything has been setup

Now you can click on the APIEndpoint url and it should execute your lambda and return the hello world message.

Full source code available here.

Working with JSON in .NET, Infrastructure as Code with Pulumi

Full source code available here.

This is a follow up to my previous post where I used dynamic and JSON files to make querying ElasticSearch with a HttpClient much easier.

To deploy my ElasticSearch domain on AWS I used Pulumi. ElasticSearch requires a JSON policy to define the permissions. In the post linked above, I have a heavily escaped that This policy can be complex and needs values substituted into it. In the example below I need to pass in the region, account id, domain name and allowed IP address.

Here is a very simple policy with four substitutions –

"{{
""Version"": ""2012-10-17"",
""Statement"": [
    {{
        ""Action"": ""es:*"",
        ""Principal"": {{
            ""AWS"": ""*""
        }},
        ""Effect"": ""Allow"",
        ""Resource"": ""arn:aws:es:{currentRegion.Name}:{currentCallerIdentity.AccountId}:domain/{esDomainName}/*"",
        ""Condition"": {{
            ""IpAddress"": {{""aws:SourceIp"": [""{myIPAddress}""]}}
        }}
    }}
]
}}"

Just escaping this is not easy, and very prone to error. A more realistic policy would be significantly longer and would need more substitutions.

Using a JSON file
Here is what I think is an easier way. As in the previous post, the JSON file becomes part of my source code. It is deserialized into a dynamic object and the required values are set.

Here is the AWS policy as it appears in my JSON file. The resource (made up of region, account, and domain name) and IpAddress are left blank, but the structure of the policy is the same as you would paste into the AWS console.

{
    "Version": "2012-10-17",
    "Statement": [
      {
        "Effect": "Allow",
        "Principal": {
          "AWS": "*"
        },
        "Action": "es:*",
        "Resource": "",
        "Condition": {
          "IpAddress": {
            "aws:SourceIp": ""
          }
        }
      }
    ]
}

In my C# I read the file, deserialize, and set the values with simple C#.

Here is an example –

private string GetAWSElasticSearchPolicy(string region, string account, string elasticSearchDomainName, string allowedIPAddress)
{
    string blankPolicy = File.ReadAllText("AWSPolicy.json");
    dynamic awsElasticSearchPolicy = JsonConvert.DeserializeObject(blankPolicy);

    awsElasticSearchPolicy.Statement[0].Resource = $"arn:aws:es:{region}:{account}:domain/{elasticSearchDomainName}/*";
    awsElasticSearchPolicy.Statement[0].Condition.IpAddress = new JObject(new JProperty("aws:SourceIp", allowedIPAddress));

    return awsElasticSearchPolicy.ToString(); // this is correctly formatted JSON that can be used with Pulumi.
}

Line 3 reads the JSON file into a string.
Line 4 turns the string into a dynamic object.
Lines 6 & 7 set the values I want.
Line 9 returns a nice JSON string that can be used with Pulumi.

This is much cleaner than the heavily escaped version in this post.

Full source code available here.

Getting Started with ElasticSearch, Part 3 – Deploying to AWS with Pulumi

Full source code available here.

This is part 3 of my short introduction to ElasticSearch. In the first part I showed how to create an ElasticSearch index, mapping, and seeded it with data. In the second I used HttpClientFactory and a typed client to query the index. In this part I going to show you how to setup ElasticSearch in AWS using infrastructure as code. Be careful, AWS charges for these things.

A few months ago Pulumi added C# to their list of supported languages. If you haven’t heard of them, they are building a tool that lets you create the IaC in a familiar programming language, at the time of writing they support TypeScript, JavaScript, Python, Go and C#. Writing in a programming language makes it easy to work with things like loops and conditionals, if you are unfamiliar with IaC, those two simple things can be extremely challenging or impossible with other tools.

I’m going to write my IaC in C#.

I’m not going to walk you through installing Pulumi, their site has all the info you need for that.

The IaC Project
Once you have installed Pulimi and tested that the command works, create a new directory called ElasticSearchDeploy.

Change to that directory and run –

pulumi new aws-csharp

Follow the instructions and open the project in VS Code or Visual Studio.

Delete the MyStack.cs file.
Create a file named MyElasticSearchStack.cs.

Paste in the below code –

using Pulumi;
using ElasticSearch = Pulumi.Aws.ElasticSearch;
using Aws = Pulumi.Aws;
using Pulumi.Aws.ElasticSearch.Inputs;

class MyElasticSearchStack : Stack
{
    public MyElasticSearchStack()
    {
        string myIPAddress = "x.x.x.x" you need to put your IP address here;
        string esDomainName = "myelasticesearch";
        var config = new Config();
        var currentRegion = Output.Create(Aws.GetRegion.InvokeAsync());
        var currentCallerIdentity = Output.Create(Aws.GetCallerIdentity.InvokeAsync());
        var esDomain = new ElasticSearch.Domain(esDomainName, new ElasticSearch.DomainArgs
        {
            DomainName = esDomainName,
            ClusterConfig = new ElasticSearch.Inputs.DomainClusterConfigArgs
            {
                InstanceType = "t2.small.elasticsearch",
            },
            EbsOptions = new DomainEbsOptionsArgs()
            {
                EbsEnabled = true,
                VolumeSize = 10,
                VolumeType = "gp2"
            },
            ElasticsearchVersion = "7.7",
            AccessPolicies = Output.Tuple(currentRegion, currentCallerIdentity).Apply(values =>
            {
                var currentRegion = values.Item1;
                var currentCallerIdentity = values.Item2;
                return @$"
                {{
                    ""Version"": ""2012-10-17"",
                    ""Statement"": [
                        {{
                            ""Action"": ""es:*"",
                            ""Principal"": {{
                                ""AWS"": ""*""
                            }},
                            ""Effect"": ""Allow"",
                            ""Resource"": ""arn:aws:es:{currentRegion.Name}:{currentCallerIdentity.AccountId}:domain/{esDomainName}/*"",
                            ""Condition"": {{
                                ""IpAddress"": {{""aws:SourceIp"": [""{myIPAddress}""]}}
                            }}
                        }}
                    ]
                    }}
                ";
            }),
        });
        this.ESDomainEndpoint =  esDomain.Endpoint;
    }
    [Output]
    public Output<string> ESDomainEndpoint { get; set; }
}

Note on line 10, you need to put in the IP address you are using. Checking this with a site like https://ipstack.com/.

In Program.cs change the reference my MyStack to MyElasticSearchStack.

That’s it.

Deploying
Go to the command line, run –

pulumi up

Select ‘yes’ and then wait about 10 to 15 minutes as AWS gets your ElasticSearch domain up and running. In the output of the command you willsee the url of the ElasticSearch domain you just created, use that in the scripts from part 1 of this series.

You can also go to the AWS console, you should see something like –

There you go – ElasticSearch index creating, seeding, querying, and infrastructure as code.

In a follow up post I’ll show you how to deploy ElasticSearch with Terraform.

The JSON Problem
For those of you that dislike horribly escaped blocks of JSON inside C#, as I do, I am working on a post that will make this much nicer to look at, and to work with.

Full source code available here.