C# and AWS Lambdas, Part 5 – Updating the Zip in S3 and Updating the Running Lambda, with Pulumi IaC
Want to learn more about AWS Lambda and .NET? Check out my A Cloud Guru course on ASP.NET Web API and Lambda.
Full source code available here.
- Part 1 - Hello World
- Part 2 - Web API and an API Gateway
- Part 3 - Pulumi IaC for Web API and an API Gateway
- Part 4 - Storing the Zip in S3, Setup with Pulumi IaC
- Part 5 - Updating the Zip in S3 and Updating the Running Lambda, with Pulumi IaC
- Part 6 - .NET 5 inside a Container inside a Lambda
- Part 7 - .NET 5 Web API inside a Container inside a Lambda, with API Gateway in front
- Part 8 - .NET 6, inside a Container, inside a Lambda
This post pulls together a few threads I’ve been working on - the creation of Lambda to run .NET, storing the zip in S3, and updating the .NET Lambda when the zip in S3 is updated.
This one took quite a while to put together - the permissions, roles, and policies were not obvious and I hope it will be of help to you. This is not a blog post on CI/CD, I am cutting corners by using Pulumi to upload the zip files initially, and then use the AWS command line to send zips to S3. In a future set of posts I will show how to use GitHub Actions to build the infrastructure, and to compile and deploy the .NET Lambda directly to S3 from GitHub.
The idea
I want to have a Lambda that runs .NET code stored in a zip file in S3. I want to be able to update the zip and have the .NET Lambda run the code in the new zip. I had hoped this would be a little tick box on the Lambda, but sadly there is no such box.
Instead, I have a second Lambda (referred to as the updater Lambda) that is triggered by an update on a specified bucket in S3. This updater Lambda in turn calls an update on the .NET Lambda and within a few seconds, the .NET Lambda will be running the new code. Doesn’t sound easy, but I didn’t think it would be too hard, but take a look at the number of resources needed!
What’s needed
- A role to run .NET Lambda.
- A role to run Lambda that updates the .NET Lambda, I’m calling this the updater.
- A policy to give the updater permissions to update the .NET Lambda and S3.
- A policy attachment for the .NET Lambda.
- A policy attachment for the updater Lambda.
- An S3 bucket.
- An S3 bucket object.
- The .NET Lambda pointing at the bucket and bucket object.
- The zip file for the .NET Lambda.
- The updater Lambda with variables passed in to verify the update of the .NET Lambda.
- The zip file for the updater Lambda - Node.js.
- Permission for the bucket to call the updated Lambda.
- A bucket notification with attached permissions.
- Reduce the bucket accessible to the public (not necessary, but good).
That’s a lot more than the tick box I was hoping for.
The code
1using Pulumi;
2using S3 = Pulumi.Aws.S3;
3using Aws = Pulumi.Aws;
4
5class MyStack : Stack
6{
7 public MyStack()
8 {
9 string resource_prefix = "PulumiHelloWorldAutoUpdate";
10
11 var lambdaHelloWorldRole = new Aws.Iam.Role($"{resource_prefix}_LambdaRole", new Aws.Iam.RoleArgs
12 {
13 AssumeRolePolicy =
14@"{
15 ""Version"": ""2012-10-17"",
16 ""Statement"": [
17 {
18 ""Action"": ""sts:AssumeRole"",
19 ""Principal"": {
20 ""Service"": ""lambda.amazonaws.com""
21 },
22 ""Effect"": ""Allow"",
23 ""Sid"": """"
24 }
25 ]
26}",
27 });
28
29 var lambdaUpdateRole = new Aws.Iam.Role($"{resource_prefix}_LambdaUpdateRole", new Aws.Iam.RoleArgs
30 {
31 AssumeRolePolicy =
32@"{
33 ""Version"": ""2012-10-17"",
34 ""Statement"": [
35 {
36 ""Action"": ""sts:AssumeRole"",
37 ""Principal"": {
38 ""Service"": ""lambda.amazonaws.com""
39 },
40 ""Effect"": ""Allow"",
41 ""Sid"": """"
42 }
43 ]
44}",
45 });
46
47 // gives the Lambda permissions to other Lambdas and S3 - too many permissions, but this is a demo.
48 var lambdaUpdatePolicy = new Aws.Iam.Policy($"{resource_prefix}_S3_Lambda_Policy", new Aws.Iam.PolicyArgs{
49 PolicyDocument =
50@"{
51 ""Version"": ""2012-10-17"",
52 ""Statement"": [
53 {
54 ""Sid"": """",
55 ""Effect"": ""Allow"",
56 ""Action"": [
57 ""s3:*"",
58 ""logs:*"",
59 ""lambda:*""
60 ],
61 ""Resource"": ""*""
62 }
63 ]
64}"
65 });
66
67 // attach a simple policy to the hello world Lambda.
68 var lambdaHelloWorldAttachment = new Aws.Iam.PolicyAttachment($"{resource_prefix}_LambdaHelloWorldPolicyAttachment", new Aws.Iam.PolicyAttachmentArgs
69 {
70 Roles =
71 {
72 lambdaHelloWorldRole.Name
73 },
74 PolicyArn = Aws.Iam.ManagedPolicy.AWSLambdaBasicExecutionRole.ToString(),
75 });
76
77 // attach the custom policy to the role that runs the update Lambda.
78 var lambdaUpdateAttachment = new Aws.Iam.PolicyAttachment($"{resource_prefix}_LambdaUpdatePolicyAttachment", new Aws.Iam.PolicyAttachmentArgs
79 {
80 Roles =
81 {
82 lambdaUpdateRole.Name
83 },
84 PolicyArn = lambdaUpdatePolicy.Arn,
85 });
86
87 var s3Bucket = new S3.Bucket($"{resource_prefix}_S3Bucket", new S3.BucketArgs
88 {
89 BucketName = "pulumi-hello-world-auto-update-s3-bucket",
90 Versioning = new Aws.S3.Inputs.BucketVersioningArgs
91 {
92 Enabled = true,
93 },
94 Acl = "private"
95 });
96
97 var s3BucketObject = new S3.BucketObject($"{resource_prefix}_ZipFile", new S3.BucketObjectArgs
98 {
99 Bucket = s3Bucket.BucketName.Apply(name => name),
100 Acl = "private",
101 Source = new FileArchive("./Lambdas/helloworld_no_date/helloworld.zip"),
102 Key = "helloworld.zip"
103 });
104
105 // this is the Lambda that runs .NET code
106 var lambdaHelloWorldFunction = new Aws.Lambda.Function($"{resource_prefix}_LambdaHelloWorldFunction", new Aws.Lambda.FunctionArgs
107 {
108 Handler = "HelloWorldLambda::HelloWorldLambda.Function::FunctionHandler",
109 MemorySize = 128,
110 Publish = false,
111 ReservedConcurrentExecutions = -1,
112 Role = lambdaHelloWorldRole.Arn,
113 Runtime = Aws.Lambda.Runtime.DotnetCore3d1,
114 Timeout = 4,
115 S3Bucket = s3Bucket.BucketName,
116 S3Key = s3BucketObject.Key
117 });
118
119 // this is the Lambda triggered by an upload to S3 and replaces the zip in the above Lambda
120 var lambdaUpdateFunction = new Aws.Lambda.Function($"{resource_prefix}_LambdaUpdateFunction", new Aws.Lambda.FunctionArgs
121 {
122 Handler = "index.handler",
123 MemorySize = 128,
124 Publish = false,
125 ReservedConcurrentExecutions = -1,
126 Role = lambdaUpdateRole.Arn,
127 Runtime = Aws.Lambda.Runtime.NodeJS14dX,
128 Timeout = 4,
129 Code = new FileArchive("./Lambdas/LambdaUpdater/index.zip"),
130 Environment = new Aws.Lambda.Inputs.FunctionEnvironmentArgs
131 {
132 Variables = new InputMap<string> {{"s3Bucket", s3Bucket.BucketName}, {"s3Key", "helloworld.zip"}, {"functionToUpdate", lambdaHelloWorldFunction.Name}}
133 }
134 });
135
136 var s3BucketPermissionToCallLambda = new Aws.Lambda.Permission($"{resource_prefix}_S3BucketPermissionToCallLambda", new Aws.Lambda.PermissionArgs
137 {
138 Action = "lambda:InvokeFunction",
139 Function = lambdaUpdateFunction.Arn,
140 Principal = "s3.amazonaws.com",
141 SourceArn = s3Bucket.Arn,
142 });
143
144 var bucketNotification = new S3.BucketNotification($"{resource_prefix}_S3BucketNotification", new Aws.S3.BucketNotificationArgs
145 {
146 Bucket = s3Bucket.Id,
147 LambdaFunctions =
148 {
149 new Aws.S3.Inputs.BucketNotificationLambdaFunctionArgs
150 {
151 LambdaFunctionArn = lambdaUpdateFunction.Arn,
152 Events =
153 {
154 "s3:ObjectCreated:*",
155 },
156 }
157 },
158 }, new CustomResourceOptions
159 {
160 DependsOn =
161 {
162 s3BucketPermissionToCallLambda,
163 },
164 });
165
166 // keep the contents bucket private
167 var bucketPublicAccessBlock = new S3.BucketPublicAccessBlock($"{resource_prefix}_PublicAccessBlock", new S3.BucketPublicAccessBlockArgs
168 {
169 Bucket = s3Bucket.Id,
170 BlockPublicAcls = false, // leaving these two false because I need them this way
171 IgnorePublicAcls = false, // for a post about GitHub Actions that I'm working on
172 BlockPublicPolicy = true,
173 RestrictPublicBuckets = true
174 });
175
176 this.LambdaUpdateFunctionName = lambdaUpdateFunction.Name;
177 this.LambdaHelloWorldFunctionName = lambdaHelloWorldFunction.Name;
178 this.S3Bucket = s3Bucket.BucketName;
179 this.S3Key = s3BucketObject.Key;
180 }
181
182 [Output]
183 public Output<string> LambdaUpdateFunctionName { get; set; }
184
185 [Output]
186 public Output<string> LambdaHelloWorldFunctionName { get; set; }
187
188 [Output]
189 public Output<string> S3Bucket {get;set;}
190
191 [Output]
192 public Output<string> S3Key {get;set;}
193}Below is the code of the updater lambda. The if checks to make sure that the lambda.updateFunctionCode(..) runs only if the expected file in S3 is updated. The environmental variables were passed in via the Pulumi code above.
1const AWS = require('aws-sdk');
2const lambda = new AWS.Lambda();
3
4exports.handler = (event) => {
5
6 if (event.Records[0].s3.bucket.name == process.env.s3Bucket && event.Records[0].s3.object.key == process.env.s3Key)
7 {
8 var params = {
9 FunctionName: process.env.functionToUpdate,
10 S3Bucket: event.Records[0].s3.bucket.name,
11 S3Key: event.Records[0].s3.object.key
12 };
13
14 // https://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/Lambda.html#updateFunctionCode-property
15 lambda.updateFunctionCode(params, function(err, data) {
16 if (err) // an error occurred
17 {
18 console.log(err, err.stack);
19 }
20 else
21 {
22 console.log(data);
23 }
24 });
25 }
26 else
27 {
28 console.log("bucket name or s3 key did not match expected values.");
29 console.log("expected bucket name: " + process.env.s3Bucket + " actual: " + event.Records[0].s3.bucket.name);
30 console.log("expected s3 key: " + process.env.s3Key + " actual: " + event.Records[0].s3.object.key);
31 }
32 console.log("Exiting");
33};The zip attached to this blog post has all the source code needed, you don’t have to add or change anything.
Running it
From the console, run -
pulumi up At the end, you should see something like this -
Note the outputs. These are the name of your Lambdas and the s3 bucket and key -
Outputs:
LambdaHelloWorldFunctionName: "PulumiHelloWorldAutoUpdate_LambdaHelloWorldFunction-???????"
LambdaUpdateFunctionName : "PulumiHelloWorldAutoUpdate_LambdaUpdateFunction-???????"
S3Bucket : "pulumi-hello-world-auto-update-s3-bucket"
S3Key : "helloworld.zip"Go to the AWS console, and test the Lambda as shown in part 1 of this blog series.
You should get output like this - “HELLO WORLD”.
Updating the zip in S3
Now to try out the real functionality, updating the zip in S3 and see if it runs in the .NET Lambda.
In the attached source there is a Lambdas directory with two subdirectories - helloworld_no_date and helloworld_with_date. They contain two variations of the .NET application. The first converts the input text to uppercase, the second converts the input text to uppercase and adds the current date and time.
You can run the below commands to upload each zip file and try out the Lambda. A few seconds after you upload, the .NET Lambda will use that zip.
// no date
aws s3 cp ./Lambdas/helloworld_no_date/helloworld.zip s3://pulumi-hello-world-auto-update-s3-bucket/helloworld.zip
// with date
aws s3 cp ./Lambdas/helloworld_with_date/helloworld.zip s3://pulumi-hello-world-auto-update-s3-bucket/helloworld.zipIf you don’t want to go into the AWS UI console to try out the Lambda you can invoke it from the command line, but you need to swap the function name below for the one in the output of the pulumi up command -
aws lambda invoke --function-name PulumiHelloWorldAutoUpdate_LambdaHelloWorldFunction-??????? --payload '"hello world"' /dev/stdoutThis was a long tough one, but I’ve learned a lot about AWS, Pulumi, and even GitHub Actions (more on that soon).
Full source code available here.