C# and AWS Lambdas, Part 5 – Updating the Zip in S3 and Updating the Running Lambda, with Pulumi IaC

Full source code available here.

This post pulls together a few threads I’ve been working on - the creation of Lambda to run .NET, storing the zip in S3, and updating the .NET Lambda when the zip in S3 is updated.

This one took quite a while to put together - the permissions, roles, and policies were not obvious and I hope it will be of help to you. This is not a blog post on CI/CD, I am cutting corners by using Pulumi to upload the zip files initially, and then use the AWS command line to send zips to S3. In a future set of posts I will show how to use GitHub Actions to build the infrastructure, and to compile and deploy the .NET Lambda directly to S3 from GitHub.

The idea

I want to have a Lambda that runs .NET code stored in a zip file in S3. I want to be able to update the zip and have the .NET Lambda run the code in the new zip. I had hoped this would be a little tick box on the Lambda, but sadly there is no such box.

Instead, I have a second Lambda (referred to as the updater Lambda) that is triggered by an update on a specified bucket in S3. This updater Lambda in turn calls an update on the .NET Lambda and within a few seconds, the .NET Lambda will be running the new code. Doesn’t sound easy, but I didn’t think it would be too hard, but take a look at the number of resources needed!

What’s needed

  1. A role to run .NET Lambda.
  2. A role to run Lambda that updates the .NET Lambda, I’m calling this the updater.
  3. A policy to give the updater permissions to update the .NET Lambda and S3.
  4. A policy attachment for the .NET Lambda.
  5. A policy attachment for the updater Lambda.
  6. An S3 bucket.
  7. An S3 bucket object.
  8. The .NET Lambda pointing at the bucket and bucket object.
  9. The zip file for the .NET Lambda.
  10. The updater Lambda with variables passed in to verify the update of the .NET Lambda.
  11. The zip file for the updater Lambda - Node.js.
  12. Permission for the bucket to call the updated Lambda.
  13. A bucket notification with attached permissions.
  14. Reduce the bucket accessible to the public (not necessary, but good).

That’s a lot more than the tick box I was hoping for.

The code

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
using Pulumi;
using S3 = Pulumi.Aws.S3;
using Aws = Pulumi.Aws;

class MyStack : Stack
{
    public MyStack()
    {
        string resource_prefix = "PulumiHelloWorldAutoUpdate";

        var lambdaHelloWorldRole = new Aws.Iam.Role($"{resource_prefix}_LambdaRole", new Aws.Iam.RoleArgs
        {
            AssumeRolePolicy =
@"{
    ""Version"": ""2012-10-17"",
    ""Statement"": [
        {
        ""Action"": ""sts:AssumeRole"",
        ""Principal"": {
            ""Service"": ""lambda.amazonaws.com""
        },
        ""Effect"": ""Allow"",
        ""Sid"": """"
        }
    ]
}",
        });

        var lambdaUpdateRole = new Aws.Iam.Role($"{resource_prefix}_LambdaUpdateRole", new Aws.Iam.RoleArgs
        {
            AssumeRolePolicy =
@"{
    ""Version"": ""2012-10-17"",
    ""Statement"": [
        {
        ""Action"": ""sts:AssumeRole"",
        ""Principal"": {
            ""Service"": ""lambda.amazonaws.com""
        },
        ""Effect"": ""Allow"",
        ""Sid"": """"
        }
    ]
}",
        });

        // gives the Lambda permissions to other Lambdas and S3 - too many permissions, but this is a demo.
        var lambdaUpdatePolicy = new Aws.Iam.Policy($"{resource_prefix}_S3_Lambda_Policy", new Aws.Iam.PolicyArgs{
            PolicyDocument = 
@"{
    ""Version"": ""2012-10-17"",
    ""Statement"": [
        {
            ""Sid"": """",
            ""Effect"": ""Allow"",
            ""Action"": [
                ""s3:*"",
                ""logs:*"",
                ""lambda:*""
            ],
            ""Resource"": ""*""
        }
    ]
}"
        });

        // attach a simple policy to the hello world Lambda.
        var lambdaHelloWorldAttachment = new Aws.Iam.PolicyAttachment($"{resource_prefix}_LambdaHelloWorldPolicyAttachment", new Aws.Iam.PolicyAttachmentArgs
        {
            Roles =
            {
                lambdaHelloWorldRole.Name
            },
            PolicyArn = Aws.Iam.ManagedPolicy.AWSLambdaBasicExecutionRole.ToString(),
        });

        // attach the custom policy to the role that runs the update Lambda.
        var lambdaUpdateAttachment = new Aws.Iam.PolicyAttachment($"{resource_prefix}_LambdaUpdatePolicyAttachment", new Aws.Iam.PolicyAttachmentArgs
        {
            Roles =
            {
                lambdaUpdateRole.Name
            },
            PolicyArn = lambdaUpdatePolicy.Arn,
        });

        var s3Bucket = new S3.Bucket($"{resource_prefix}_S3Bucket", new S3.BucketArgs
        {
            BucketName = "pulumi-hello-world-auto-update-s3-bucket",
            Versioning = new Aws.S3.Inputs.BucketVersioningArgs
            {
                Enabled = true,
            },
            Acl = "private"
        });

        var s3BucketObject = new S3.BucketObject($"{resource_prefix}_ZipFile", new S3.BucketObjectArgs
        {
            Bucket = s3Bucket.BucketName.Apply(name => name),
            Acl = "private",
            Source = new FileArchive("./Lambdas/helloworld_no_date/helloworld.zip"),
            Key = "helloworld.zip"
        });

        // this is the Lambda that runs .NET code
        var lambdaHelloWorldFunction = new Aws.Lambda.Function($"{resource_prefix}_LambdaHelloWorldFunction", new Aws.Lambda.FunctionArgs
        {
            Handler = "HelloWorldLambda::HelloWorldLambda.Function::FunctionHandler",
            MemorySize = 128,
            Publish = false,
            ReservedConcurrentExecutions = -1,
            Role = lambdaHelloWorldRole.Arn,
            Runtime = Aws.Lambda.Runtime.DotnetCore3d1,
            Timeout = 4,
            S3Bucket = s3Bucket.BucketName,
            S3Key = s3BucketObject.Key
        });

        // this is the Lambda triggered by an upload to S3 and replaces the zip in the above Lambda
        var lambdaUpdateFunction = new Aws.Lambda.Function($"{resource_prefix}_LambdaUpdateFunction", new Aws.Lambda.FunctionArgs
        {
            Handler = "index.handler",
            MemorySize = 128,
            Publish = false,
            ReservedConcurrentExecutions = -1,
            Role = lambdaUpdateRole.Arn,
            Runtime = Aws.Lambda.Runtime.NodeJS14dX,
            Timeout = 4,
            Code = new FileArchive("./Lambdas/LambdaUpdater/index.zip"),
            Environment = new Aws.Lambda.Inputs.FunctionEnvironmentArgs
            {
                Variables = new InputMap<string> {{"s3Bucket", s3Bucket.BucketName}, {"s3Key", "helloworld.zip"}, {"functionToUpdate", lambdaHelloWorldFunction.Name}}
            }
        });

        var s3BucketPermissionToCallLambda = new Aws.Lambda.Permission($"{resource_prefix}_S3BucketPermissionToCallLambda", new Aws.Lambda.PermissionArgs
        {
            Action = "lambda:InvokeFunction",
            Function = lambdaUpdateFunction.Arn,
            Principal = "s3.amazonaws.com",
            SourceArn = s3Bucket.Arn,
        });

        var bucketNotification = new S3.BucketNotification($"{resource_prefix}_S3BucketNotification", new Aws.S3.BucketNotificationArgs
        {
            Bucket = s3Bucket.Id,
            LambdaFunctions = 
            {
                new Aws.S3.Inputs.BucketNotificationLambdaFunctionArgs
                {
                    LambdaFunctionArn = lambdaUpdateFunction.Arn,
                    Events = 
                    {
                        "s3:ObjectCreated:*",
                    },
                }
            },
        }, new CustomResourceOptions
        {
            DependsOn = 
            {
                s3BucketPermissionToCallLambda,
            },
        });

        // keep the contents bucket private
        var bucketPublicAccessBlock = new S3.BucketPublicAccessBlock($"{resource_prefix}_PublicAccessBlock", new S3.BucketPublicAccessBlockArgs
        {
            Bucket = s3Bucket.Id,
            BlockPublicAcls = false,  // leaving these two false because I need them this way 
            IgnorePublicAcls = false, // for a post about GitHub Actions that I'm working on
            BlockPublicPolicy = true,
            RestrictPublicBuckets = true
        });

        this.LambdaUpdateFunctionName = lambdaUpdateFunction.Name;
        this.LambdaHelloWorldFunctionName = lambdaHelloWorldFunction.Name;
        this.S3Bucket = s3Bucket.BucketName;
        this.S3Key = s3BucketObject.Key;
    }

    [Output]
    public Output<string> LambdaUpdateFunctionName { get; set; }

    [Output]
    public Output<string> LambdaHelloWorldFunctionName { get; set; }

    [Output]
    public Output<string> S3Bucket {get;set;}

    [Output]
    public Output<string> S3Key {get;set;}
}

Below is the code of the updater lambda. The if checks to make sure that the lambda.updateFunctionCode(..) runs only if the expected file in S3 is updated. The environmental variables were passed in via the Pulumi code above.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
const AWS = require('aws-sdk');
const lambda = new AWS.Lambda();

exports.handler = (event) => {
    
    if (event.Records[0].s3.bucket.name == process.env.s3Bucket && event.Records[0].s3.object.key == process.env.s3Key)
    {
        var params = {
            FunctionName: process.env.functionToUpdate,
            S3Bucket: event.Records[0].s3.bucket.name, 
            S3Key: event.Records[0].s3.object.key
        };
        
        // https://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/Lambda.html#updateFunctionCode-property
        lambda.updateFunctionCode(params, function(err, data) {
            if (err) // an error occurred
            {
                console.log(err, err.stack);
            }
            else
            {   
                console.log(data);  
            }
        });
    }
    else
    {
        console.log("bucket name or s3 key did not match expected values.");
        console.log("expected bucket name: " + process.env.s3Bucket + " actual: " + event.Records[0].s3.bucket.name);
        console.log("expected s3 key: " + process.env.s3Key + " actual: " + event.Records[0].s3.object.key);
    }
    console.log("Exiting");
};

The zip attached to this blog post has all the source code needed, you don’t have to add or change anything.

Running it

From the console, run -

pulumi up 

At the end, you should see something like this -

Note the outputs. These are the name of your Lambdas and the s3 bucket and key -

Outputs:
    LambdaHelloWorldFunctionName: "PulumiHelloWorldAutoUpdate_LambdaHelloWorldFunction-???????"
    LambdaUpdateFunctionName    : "PulumiHelloWorldAutoUpdate_LambdaUpdateFunction-???????"
    S3Bucket                    : "pulumi-hello-world-auto-update-s3-bucket"
    S3Key                       : "helloworld.zip"

Go to the AWS console, and test the Lambda as shown in part 1 of this blog series.

You should get output like this - “HELLO WORLD”.

Updating the zip in S3

Now to try out the real functionality, updating the zip in S3 and see if it runs in the .NET Lambda.

In the attached source there is a Lambdas directory with two subdirectories - helloworld_no_date and helloworld_with_date. They contain two variations of the .NET application. The first converts the input text to uppercase, the second converts the input text to uppercase and adds the current date and time.

You can run the below commands to upload each zip file and try out the Lambda. A few seconds after you upload, the .NET Lambda will use that zip.

// no date
aws s3 cp ./Lambdas/helloworld_no_date/helloworld.zip s3://pulumi-hello-world-auto-update-s3-bucket/helloworld.zip
// with date
aws s3 cp ./Lambdas/helloworld_with_date/helloworld.zip s3://pulumi-hello-world-auto-update-s3-bucket/helloworld.zip

If you don’t want to go into the AWS UI console to try out the Lambda you can invoke it from the command line, but you need to swap the function name below for the one in the output of the pulumi up command -

aws lambda invoke --function-name PulumiHelloWorldAutoUpdate_LambdaHelloWorldFunction-??????? --payload '"hello world"' /dev/stdout

This was a long tough one, but I’ve learned a lot about AWS, Pulumi, and even GitHub Actions (more on that soon).

Full source code available here.

comments powered by Disqus

Related