Forcing the Lambda Service to Warm 100 Execution Environments for your Function

Want to learn more about AWS Lambda and .NET? Check out my A Cloud Guru course on ASP.NET Web API and Lambda.

Download full source code.

If you know that your Lambda function needs to handle bursts of activity at a predictable time, and you want to avoid the slower response of cold stars for that burst, you have a few options. You can use provisioned concurrency, to warm a specific number of execution environments in advance, then when the burst is over you turn off the provisioned concurrency again. This is a good solution but has a cost associated with it.

Another option is to force the Lambda service to warm up execution environments for you. How to do this? Read on to find out.

First, a little background

Lambda functions run inside an execution environment. A single execution environment can handle one request at a time. If an execution environment is available, the request will be handled by an existing environment.

If no execution environment is available to handle the request (because there are none, or they are all busy with other requests), the Lambda service will create a new one. Creating this environment takes time, the binaries need to be downloaded, memory allocated, and initialization steps run; this is referred to as a cold start.

The problem with bursty workloads is that you may not have enough execution environments to handle the burst. The burst results in the creation of execution environments, however, requests are handled slowly as the environments are created.

In the previous sentence is a solution too - if you can force the creation of execution environments before the expected burst, you won’t have cold starts during the burst…

Knowing a cold start from a warm start

When you invoke a Lambda function it produces a log output. At the bottom is a REPORT line. If the line includes an Init Duration, it was a cold start.

REPORT RequestId: 2f94290e-7503-4a0e-bf2d-ebc4f0ae265d  Duration: 10429.83 ms   
Billed Duration: 10430 ms   Memory Size: 256 MB Max Memory Used: 62 MB  
Init Duration: 234.19 ms

If the REPORT line does not include an Init Duration, it was a warm start.

REPORT RequestId: 921654d8-105b-473f-a8bc-1d8606d9d67a  Duration: 10.80 ms  
Billed Duration: 11 ms  Memory Size: 256 MB Max Memory Used: 63 MB

If you are using provisioned concurrency, there are some caveats around Init Duration, see section 6 of this post for more.

Warming up execution environments

The key here is to make enough requests in such a way that they can’t be handled by an existing execution environment. This will force the Lambda service to create new execution environments. Flooding the Lambda service with normal requests is not what I’m advocating because you pay for each request, and you may not succeed in creating enough execution environments.

My suggestion is to send a series of specially formed versions of the normal request, ones that the Lambda function handles slowly, i.e. it pauses for a few seconds before returning the response, thus preventing that execution environment from handling another request. When the next specially formed request arrives, the Lambda service will have to create a new execution environment to handle it. Same with the next special request, and so on.

In this first post, I’m going to show the basic mechanism.

But in follow-up posts, I’ll show how to do the same

• with a Web API application inside a Lambda function that uses a defined endpoint to trigger the warm-up of execution environments.
• with Lambda versions and aliases allowing you to pre-warm execution environments for a specific version of your Lambda function, then swap traffic to the version using the alias at the right moment.

The Lambda function

Create a new Lambda function -

dotnet new lambda.EmptyFunction --name PreWarnEnvs

Change to the PreWarnEnvs/src/PreWarnEnvs directory and open the Function.cs file. Replace its content with the following -

using Amazon.Lambda.Core;

// Assembly attribute to enable the Lambda function's JSON input to be converted into a .NET class.
[assembly: LambdaSerializer(typeof(Amazon.Lambda.Serialization.SystemTextJson.DefaultLambdaJsonSerializer))]

namespace PreWarmEnvs;

public class Function
{
    public string FunctionHandler(string input, ILambdaContext context)
    {
        if (input == "PreWarmThisLambda")
        {
            Console.WriteLine("Pre-Warming Lambda");
            Task.Delay(10000).GetAwaiter().GetResult();
        }
        return input.ToUpper();
    }
}

If the input to the Lambda function is “PreWarmThisLambda” the function will pause for 10 seconds before returning the response. This will hold on to the execution environment for 10 seconds. So the next request will create a new execution environment.

Deploy the Lambda function

Use the following to build the code and deploy the Lambda function -

dotnet lambda deploy-function PreWarmEnvs 

You will then be asked - “Enter name of the new IAM Role:”, put in “PreWarmEnvsRole”.

Then you will be asked to - “Select IAM Policy to attach to the new role and grant permissions”, select “AWSLambdaBasicExecutionRole”, for me it is number 6 on the list.

Wait as the function and permissions are created.

Warming up the execution environments

Now create a simple C# console application that invokes the Lambda function -

dotnet new console --name LambdaInvoker

Add the AWSSDK.Lambda NuGet package to the project.

Open the Program.cs file and replace its content with the following -

using System.Text;
using System.Text.Json;
using Amazon.Lambda;
using Amazon.Lambda.Model;

AmazonLambdaClient client = new AmazonLambdaClient();

if (args.Length != 2)
{
    Console.WriteLine("Usage: dotnet run FunctionName WarmUpString");
    return;
}
string functionName = args[0];
string warmUpString = args[1];
Console.WriteLine($"Invoking {functionName}...");

List<Task<InvokeResponse>> taskList = new List<Task<InvokeResponse>>();
for (int i = 1; i <= 100; i++)
{
    Console.WriteLine($"Invoking {i}");
    var request = new InvokeRequest
    {
        FunctionName = functionName,
        Payload = JsonSerializer.Serialize(warmUpString),
        LogType = LogType.Tail
    };
    taskList.Add(client.InvokeAsync(request));
}

await Task.WhenAll(taskList);

foreach (var task in taskList)
{
    var response = await task;
    Console.WriteLine(JsonSerializer.Deserialize<string>(response.Payload));
    var log = Encoding.UTF8.GetString(Convert.FromBase64String(response.LogResult));
    Console.WriteLine(log);
}

This application takes the name of the function to invoke, and the string to send in the invocation request. It then invokes the Lambda function 100 times.

If the sent string matches the warm up string the Lambda function expects, the Lambda function will pause for 10 seconds before returning the response.

Running the application

From the command line run -

dotnet lambda PreWarmEnvs PreWarmThisLambda > prewarm.txt

When the program finishes, take a look at the prewarm.txt file. You should see the Lambda function is invoked 100 times, and 100 REPORT lines with Init Duration: listed. This confirms that you did not use any existing execution environments and that the Lambda service created 100 new execution environments.

Now run the program again, but with a different string -

dotnet lambda PreWarmEnvs justrunit > justrunit.txt

Open the justrunit.txt file. You should see the Lambda function is invoked 100 times, but there will be no REPORT lines with Init Duration:. This confirms that you used existing execution environments and that the Lambda service did not create any new execution environments.

Conclusion

With a tiny change to your Lambda function and a specially crafted invocation request, you can force the Lambda service to create warm execution environments.

In the next posts, I’ll show how to do the warm execution environments for Web API Lambda functions, and later and how to use Lambda versions and aliases with this approach.

Download full source code.