Accessing AWS Secrets Manager from .NET Lambda Functions, Part 3 - Connected to a VPC, using a VPC Endpoint

Want to learn more about AWS Lambda and .NET? Check out my A Cloud Guru course on ASP.NET Web API and Lambda.

Download full source code.

This is the third post in a series of posts on using Secrets Manager, in this one you will see one way to access Secrets Manager when the Lambda function is connected to a VPC.

When a Lambda function tries to access the Secrets Manager it is a relatively straightforward task, once you have the correct permissions on the role the function is executed with. The first two posts in this series showed how to do this.

But when the function is connected to the VPC, it gets more complicated. By default when a Lambda function is connected to the VPC, it will not have access to the Secrets Manager service. To overcome this, there are a few solutions. This post will show how to use a VPC endpoint to access the Secrets Manager. The next post will show how to do the same using a NAT gateway instead of a VPC endpoint.

And in the last post in this series, you will see how to tie together the use of a Lambda function, on VPC, connecting to Secrets Manager to retrieve database credentials, and then querying an AWS RDS SQL Server (that is not publicly accessible).

1. Get the tools

See this post for how to get all the tools you need.

2. Create the secret

Create the secret using the following command -

aws secretsmanager create-secret --name my-credentials --secret-string '{\"username\":\"bryan\",\"password\":\"A-COMPLEX-PASSWORD123!\"}'

You will see output that looks like this -

{
    "ARN": "arn:aws:secretsmanager:us-east-1:xxxxxxxx:secret:my-credentials-dwiP2a",
    "Name": "my-credentials",
    "VersionId": "1c26a403-e3e7-4170-9e62-312c3e5b1880"
}

The response includes the ARN, you will use that later. Note that I am using us-east-1, you may be using a different region.

If you are using the Windows command prompt, you need to enclose the secret string with double quotes, instead of single quotes.

3. Create the Lambda function

The AWS dotnet tooling makes it easy to create Lambda functions, run this -

dotnet new lambda.EmptyFunction --name LambdaSecretsManagerVPC

This creates a simple Lambda function based on the templates you installed.

Change to the LambdaSecretsManagerVPC/src/LambdaSecretsManagerVPC directory.

Add the AWSSDK.SecretsManager NuGet package.

dotnet add package AWSSDK.SecretsManager

Edit Function.cs, adding three using statements -

using Amazon;
using Amazon.SecretsManager;
using Amazon.SecretsManager.Model;

Replace the body of the Function class with the following code -

public class Function
{
    public async Task<string> FunctionHandler(ILambdaContext context)
    {
        string secretName = "my-credentials";
        string region = "us-east-1";

        var timeout = context.RemainingTime.Subtract(TimeSpan.FromSeconds(3));
        var cts = new CancellationTokenSource(timeout);

        return await GetSecretValue(secretName, region, cts.Token);
    }

    private async Task<string> GetSecretValue(string secretName, string region, CancellationToken cancellationToken)
    {
        IAmazonSecretsManager client = new AmazonSecretsManagerClient(RegionEndpoint.GetBySystemName(region));

        GetSecretValueRequest request = new GetSecretValueRequest(){
            SecretId = secretName,
            VersionStage = "AWSCURRENT"
        };

        try
        {
            GetSecretValueResponse response = await client.GetSecretValueAsync(request, cancellationToken);
            return response.SecretString;
        }
        catch (OperationCanceledException oce)
        {
            System.Console.WriteLine($"Request timed out, the cancellation token was triggered. {oce.Message}");
            throw;
        }
        catch (Exception ex)
        {
            Console.Error.WriteLine("Error: " + ex.Message);
            throw;
        }
    }
}

This Lambda function does not take any parameters and returns the secret as a string.

4. Deploy the function

To deploy your application to the Lambda service, run -

dotnet lambda deploy-function LambdaSecretsManagerVPC

You will be asked to select an IAM role, or create a new one, at the bottom of the list will be *** Create new IAM Role ***, type in the associated number.

You will be asked for a role name, enter LambdaSecretsManagerVPCRole.

After this you will be prompted to select the IAM Policy to attach to the role, choose AWSLambdaVPCAccessExecutionRole, it is number 8 on my list.

After a few seconds, the function will be deployed.

5. Add permissions to the role

If you tried to run invoke the function now, you would get a permissions error because the function does not have the right permissions to read the secret.

In the attached zip file there is a policies directory, open the ReadCredentialsFromSecretsManager.json and change the ARN to the one returned when you created the secret in step 2.

The following command will create an inline policy, granting LambdaSecretsManagerVPCRole access to the secret you created.

aws iam put-role-policy --role-name LambdaSecretsManagerVPCRole --policy-name ReadCredentialsFromSecretsManager --policy-document file://policies/ReadCredentialsFromSecretsManager.json

Now the Lambda function will be able to access the secret.

For completeness, here is the content of ReadCredentialsFromSecretsManager.json -

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": "secretsmanager:GetSecretValue",
            "Resource": "arn:aws:secretsmanager:us-east-1:xxxxxxxxxx:secret:my-credentials-xxxxxx"
        }
    ]
}

You can add the permission from the AWS Console too, see the first post in this series for instructions.

6. Invoking the function

Changes to permissions policies can take a few seconds to take effect.

Invoke the function -

dotnet lambda invoke-function LambdaSecretsManagerVPC 

The request will succeed and you will see -

Amazon Lambda Tools for .NET Core applications (5.3.0)
Project Home: https://github.com/aws/aws-extensions-for-dotnet-cli, https://github.com/aws/aws-lambda-dotnet

Payload:
"'{\"username\":\"bryan\",\"password\":\"A-COMPLEX-PASSWORD123!\"}'"

7. Connect the Lambda function to the VPC

To connect the Lambda function to the VPC you need to decide what security group(s), and subnet(s) to use.

For simplicity, you should start with the default security group and a single subnet. In a realistic scenario, more subnets will give more resilience.

To get your default security group, and the VPC that is using it, run the following command -

aws ec2 describe-security-groups --query 'SecurityGroups[?GroupName==`default`].[GroupId,VpcId]' --output text

You will see output that looks like -

sg-1111111     vpc-2222222

This will give you two ids, the first is the id of the default security group, and the second is the id of the VPC that is using it.

To find a subnet on that VPC run -

aws ec2 describe-subnets --query 'Subnets[?VpcId==`vpc-2222222`].SubnetId | [0]' --output text

subnet-3333333

Now you have all the info, you need to connect the Lambda function to the VPC.

aws lambda update-function-configuration --function-name LambdaSecretsManagerVPC --vpc-config SubnetIds=subnet-3333333,SecurityGroupIds=sg-1111111

A note on security groups

If your Lambda function and VPC endpoint are in different security groups, make sure that the security group the VPC endpoint is in gives inbound access to the security group the Lambda function is in.

8. Invoke the function again

Trying invoking the function again, it will fail.

dotnet lambda invoke-function LambdaSecretsManagerVPC 

Because the Lambda function is connected to the VPC, it cannot access the Secrets Manager service. Secrets Manager does not run in your VPC, and now the function has access only to recourses in the VPC.

The function can no longer make outbound calls to the internet, though you can still invoke the function.

9. Add a VPC Endpoint for Secrets Manager

To give the Lambda function access to the Secrets Manager service, you can add a VPC endpoint (an alternative approach with a NAT Gateway will be shown in the next post in this series).

But first, you need to determine the service name for the Secrets Manager service.

aws ec2 describe-vpc-endpoint-services --query 'ServiceDetails[?contains(ServiceName, `secretsmanager`)].ServiceName' --output text

It should be something like -

com.amazonaws.us-east-1.secretsmanager

Next create the VPC endpoint -

aws ec2 create-vpc-endpoint --vpc-endpoint-type Interface --private-dns-enabled --vpc-id vpc-2222222 --service-name com.amazonaws.us-east-1.secretsmanager --subnet-ids subnet-3333333 --security-group-ids sg-1111111 --tag-specifications 'ResourceType=vpc-endpoint,Tags=[{Key=Name,Value=Secrets-Manager-CLI-Made}]'

Give it a few minutes to finish.

Note, by using this command you will be creating a DNS entry for com.amazonaws.us-east-1.secretsmanager that points to the IP address of the VPC endpoint.

If don’t want this to happen, you can use the --no-private-dns-enabled flag for the command. In that case, you need to use one of the DNS names shown the command output to reach Secrets Manager via the VPC endpoint.

10. Invoke the function again

Now try the Lambda function again, it should succeed -

dotnet lambda invoke-function LambdaSecretsManagerVPC 

Download full source code.