Accessing AWS Secrets Manager from .NET Lambda Functions, Part 4 - Connected to a VPC, and using a NAT Gateway

Want to learn more about AWS Lambda and .NET? Check out my A Cloud Guru course on ASP.NET Web API and Lambda.

Once you connect a Lambda function to the VPC, it can no longer access the internet or some AWS services such as Secrets Manager. To resolve this, you have a few choices, one is to use a VPC endpoint, which I covered in an earlier post. Another option is to use a NAT Gateway to handle internet traffic for the Lambda function, this has few more steps than using the VPC Endpoint but can have cost advantages.

Why connect a Lambda function to the VPC?

My main reason was security. I had SQL Server in my VPC that was not publicly accessible and I wanted to query it from a Lambda function. I had to connect my Lambda function to the VPC, but as soon as I did, the Lambda function lost its ability to access the Secrets Manager or the internet in general.

Because security was my concern, I had to store my credentials in Secrets Manager, I wasn’t going to store them in a config file. Now I needed to restore access to the Secrets Manager from my Lambda function. And that is where this story starts.

Assumptions

• You have a VPC
• Your VPC has at least two subnets
• You have an Internet Gateway on your VPC
• You have a main route table with a route for 0.0.0.0/0 via the Internet Gateway
• You have a default security group that allows all egress traffic, and all internal traffic within that security group

Prerequisite

You have to create a secret in Secrets Manager, and a Lambda function to access that secret. Please follow this post, it has full instructions.

Concepts

I am going to simplify some things, but there will be enough to get the technique across.

Subnets

You are going to use two of your subnets, one will be designated as “public”, and the other as “private”.

A public subnet is a subnet that can access the internet via an Internet Gateway. In AWS, this means that the subnet uses a route table that routes internet traffic to the Internet Gateway.

A private subnet is a subnet that can only access the internet via a NAT Gateway. In AWS, this means that the subnet uses a route table that routes internet traffic to a NAT Gateway.

Lambda functions

Lambda functions run in their own VPC, not your VPC. By default, they do not have access to your VPC but they do have access to the internet.

When you connect your Lambda function to the VPC, it will have access to resources in the VPC, but no longer has internet access.

In this example you are going to connect a Lambda function to your VPC, you will choose the private subnet and the default security group. Once you have done this, the Lambda function has connectivity to your VPC. It cannot use the Internet Gateway to access the internet. But it can use a NAT Gateway to access the internet.

NAT Gateway

Your NAT Gateway will be added to the public subnet.

A route table will be created to direct all internet traffic from the private subnet to the NAT Gateway (on the public subnet). From there, the NAT Gateway will send internet traffic to the Internet Gateway.

Your Lambda will now have access to your VPC, and the internet via the NAT Gateway.

Step by step

There are a few steps to get this technique working. But if you follow along it won’t be too difficult.

1. Get your VPC id and default security group id

Run the following command -

aws ec2 describe-security-groups --query 'SecurityGroups[?GroupName==`default`].[GroupId,VpcId]' --output text

You will see output like -

sg-11111    vpc-22222

sg-11111 is your default security group id, and vpc-22222 is your VPC id.

2. Get the list of subnets in the VPC

Now you need to get a list of the subnets in the VPC, but you are going to use only two.

Run -

aws ec2 describe-subnets --query 'Subnets[?VpcId==`vpc-22222`].SubnetId ' --output text

Your output should look like -

subnet-33333        subnet-44444        subnet-xxxxxxxxx        subnet-xxxxxx        subnet-xxxxxxxxxx        subnet-xxxxxxxx

My VPC has six subnets, but I’m interested in only two of them.

I am going to use subnet-33333 as the public subnet, and subnet-44444 as the private subnet.

3. Attach the Lambda function to VPC

If you followed the prerequisite you have a Lambda function that can access a secret in Secrets Manager. By default Lambda functions do not have access to your VPC.

A simple diagram would look like this -

You should be able to run it using the following command -

dotnet lambda invoke-function LambdaSecretsManagerSimple

The request will succeed and you will see -

Amazon Lambda Tools for .NET Core applications (5.4.2)
Project Home: https://github.com/aws/aws-extensions-for-dotnet-cli, https://github.com/aws/aws-lambda-dotnet

Payload:
"'{\"username\":\"bryan\",\"password\":\"A-COMPLEX-PASSWORD123!\"}'"

Now you are going to connect that Lambda function to your VPC via the private subnet and the default security group. But before that, you need to give the Lambda function another permissions policy to allow it to create the VPC connection.

Run -

aws iam attach-role-policy --policy-arn arn:aws:iam::aws:policy/service-role/AWSLambdaVPCAccessExecutionRole --role-name LambdaSecretsManagerSimpleRole 

Now that you have the necessary permissions, you can attach the Lambda function to the VPC -

aws lambda update-function-configuration --function-name LambdaSecretsManagerSimple --vpc-config SubnetIds=subnet-44444,SecurityGroupIds=sg-11111

This will take a while…

Once it is complete, try to invoke the Lambda function, it won’t work!

The Lambda function no longer has access to the internet, and it cannot reach Secrets Manager, but it can access your VPC.

From your VPC, the Internet Gateway can access Secrets Manager. Over the next few steps, you will route internet traffic from your Lambda function to the Internet Gateway.

4. Allocate an IP address for the NAT Gateway

Before you can create a NAT Gateway, you need to allocate an IP address for it to use.

Run -

aws ec2 allocate-address --domain vpc

In the output, you will see an allocation id.

"AllocationId": "eipalloc-55555"

If you didn’t see it, run the below to review your allocation addresses

aws ec2 describe-addresses

5. Create the NAT Gateway

Create a NAT Gateway on the public subnet, in my case that is subnet-33333.

Run -

aws ec2 create-nat-gateway --subnet-id subnet-33333 --allocation-id eipalloc-55555 --tag-specifications 'ResourceType=natgateway,Tags=[{Key=Name,Value=my-nat-gateway}]'

The output will include the NAT Gateway Id

"NatGatewayId": "nat-66666",

6. Routing tables and routes

6.1 Create a new route table

You already have a “main” route table that directs all traffic to the Internet Gateway. But you need one that will direct traffic from the private subnet to the NAT Gateway.

To see your existing route tables, run -

aws ec2 describe-route-tables --query 'RouteTables[?VpcId==`vpc-22222`]'

Create a new route table by running -

aws ec2 create-route-table --vpc-id vpc-22222 --tag-specifications 'ResourceType=route-table,Tags=[{Key=Name,Value=my-nat-route-table}]'

The output will include the route table id, it will look like this -

"RouteTableId": "rtb-77777",

6.2 Associate the private subnet to the new route table

Now that you have a route table, you have to associate the private subnet with it. Remember it’s the private subnet this time.

Run -

aws ec2 associate-route-table --route-table-id rtb-77777 --subnet-id subnet-44444

6.3 Add a route directing all internet traffic to the NAT Gateway

Almost there.

Now you need to add a route to the route table that directs all internet traffic to the NAT Gateway.

Run -

aws ec2 create-route --route-table-id rtb-77777 --destination-cidr-block 0.0.0.0/0 --nat-gateway-id nat-66666

If you have followed the steps correctly, you should now be able to access the internet from your Lambda function.

7. Invoke the Lambda function

Invoke the Lambda function, it will work!