Using Step Functions and C# Lambdas to Orchestrate API Calls
Want to learn more about AWS Lambda and .NET? Check out my A Cloud Guru course on ASP.NET Web API and Lambda.
tl;dr
AWS Step Functions can be used with a Lambda to orchestrate calls to remote APIs, the Lambda returns Json to the Step Function, and based on the responses other Step Functions use the Lambda to call the next API.
However, the serialized output of the Lambda is escaped and causes problems for the Step Functions, but if the Lambda returns the Json as Stream instead of a string the problem is resolved.
When returning a string the output is something like -
"{\n \"origin\": \"54.80.87.97\"\n}\n"But when using a Stream it is -
{
"origin": "54.80.87.97"
}Introduction
My goal is to use Step Functions to call a Lambda passing in a URL, the Lambda will use a HttpClient to make a request to the URL, and return the Json response to the Step Function. Based on that response another Step Function will be called, which will, in turn, call the original Lambda, but with a new URL, and so on. Decisions will also be made by the Step Functions based on the Json response.
Below is a generic drawing of how I use Step Functions with a Lambda to orchestrate API calls.
Finding Breweries By IP Address
In my specific case, I will use Step Functions to find breweries based on an IP address.
The three APIs I’m using allow me to -
- determine my IP address
- get my location from my IP address
- find craft breweries based on my location by city or state
These three APIs are very different and, and return very different Json. I want to use a single Lambda that doesn’t need to know anything about the API it is calling. I want to avoid all the work involved in the normal usage of a HttpClient where I have to create lots of POCOs to represent the Json.
The Lambda, and the Problem
This is a simple as I could make it. It takes in the URL of the endpoint I am going to call and makes an HTTP GET request to it. It returns the Json response. The Lambda doesn’t handle other verbs, body, headers, but that would not be hard to add.
1public async Task<string> FunctionHandler(ApiRequest input, ILambdaContext context)
2{
3 HttpClient httpClient = new HttpClient();
4 var content = await httpClient.GetStringAsync(input.Uri);
5
6 Console.WriteLine(content);
7
8 return content;
9}But this has a problem, and when working with Step Functions it is a very large problem.
The log of the Lambda execution looks fine, I see the Json in the expected format -
START RequestId: 5dd31d04-b382-4832-a307-44a8778f143f Version: $LATEST
{
"origin": "54.80.87.97"
}
END RequestId: 5dd31d04-b382-4832-a307-44a8778f143f
REPORT RequestId: 5dd31d04-b382-4832-a307-44a8778f143f Duration: 7.56 ms Billed Duration: 8 ms Memory Size: 512 MB Max Memory Used: 70 MB But the content returned by the Lambda to the Step Function looks like this -
"{\n \"origin\": \"54.80.87.97\"\n}\n"The Json I received from the remote API has been serialized again and has a lot escaping included.
The Step Function cannot parse the origin from the returned value.
Here is what happens if I try -
The JsonPath argument for the field '$.Payload.origin' could not be found in the input '{\"ExecutedVersion\":\"$LATEST\",\"Payload\":\"{\\n \\\"origin\\\": \\\"54.80.87.97\\\"\\n}I don’t want to go down the route of creating a class that represents the Json returned from the remote API, my goal here is for the Lambda function to know nothing about the API it is calling. This is not the first challenge I’ve had with Json and C# - see this, and this for more.
Fortunately, there are a few ways to resolve this. One way is to use Intrinsic Functions to clean up the string, I’ve written about that here, but that feels very awkward, I would have to process the input to every Step Function with an Intrinsic Function, not hard, but not what I want.
What I need is to make the Lambda return Json that the Step Function can process without extra steps.
Better Serialization
The response from the API is already a Json string, and I want to prevent it from being escaped as it is sent over the wire.
There are a couple of ways, one is to write my own serializer that implements ILambdaSerializer, I’ve done this and it works, I’ll blog on that later.
But the easiest way I could find was to change the return type of the Lambda function from a string to a Stream!
1public async Task<System.IO.Stream> FunctionHandler(ApiRequest input, ILambdaContext context)
2{
3 HttpClient httpClient = new HttpClient();
4 var content = await httpClient.GetStreamAsync(input.Uri);
5
6 return content;
7}Note that I’m returning Task<System.IO.Stream>.
Now the output of the Lambda is what I want! This is a tiny change, but it was not obvious and came after quite a bit of effort.
The Step Functions
I’m not going to go into how to create Step Functions, I have another blog post on that.
Here is the state machine -
If no IP address is passed at the start, it will figure out the IP address of the server it is running on, from there it works out the location associated with that IP address. With that location, a call is made to the Open Brewery DB with the city, if no breweries are found in that city, the state is used to look for breweries instead.
I use Intrinsic Functions to combine the output of the previous step with the HTTP request I’m making in the current step, see line 7 below where I combine the city from the output of the previous step with the request to the Lambda.
1"Get Brewery by City": {
2 "Type": "Task",
3 "Resource": "arn:aws:states:::lambda:invoke",
4 "Parameters": {
5 "FunctionName": "arn:aws:lambda:us-east-1:YOUR_ACCCOUNT_NUMBER:function:ApiCaller:$LATEST",
6 "Payload": {
7 "Uri.$": "States.Format('https://api.openbrewerydb.org/breweries?by_city={}', $.city)"
8 }
9 }There is no error handling in this example beyond the retries that AWS added in by default.
1{
2 "Comment": "Find breweries by IP address",
3 "StartAt": "IP Address Provided?",
4 "States": {
5 "IP Address Provided?": {
6 "Type": "Choice",
7 "Choices": [
8 {
9 "Variable": "$.ipaddress",
10 "IsPresent": true,
11 "Next": "Get Location from IP Address"
12 }
13 ],
14 "Default": "Get IP Address"
15 },
16 "Get Location from IP Address": {
17 "Type": "Task",
18 "Resource": "arn:aws:states:::lambda:invoke",
19 "Parameters": {
20 "FunctionName": "arn:aws:lambda:us-east-1:YOUR_ACCCOUNT_NUMBER:function:ApiCaller:$LATEST",
21 "Payload": {
22 "Uri.$": "States.Format('http://api.ipstack.com/{}?access_key=YOU_NEED_YOUR_OWN_ACCESS_KEY', $.ipaddress)"
23 }
24 },
25 "Retry": [
26 {
27 "ErrorEquals": [
28 "Lambda.ServiceException",
29 "Lambda.AWSLambdaException",
30 "Lambda.SdkClientException"
31 ],
32 "IntervalSeconds": 2,
33 "MaxAttempts": 6,
34 "BackoffRate": 2
35 }
36 ],
37 "Next": "Get Brewery by City",
38 "OutputPath": "$.Payload"
39 },
40 "Get IP Address": {
41 "Type": "Task",
42 "Resource": "arn:aws:states:::lambda:invoke",
43 "Parameters": {
44 "FunctionName": "arn:aws:lambda:us-east-1:YOUR_ACCCOUNT_NUMBER:function:ApiCaller:$LATEST",
45 "Payload": {
46 "Uri": "http://httpbin.org/ip"
47 }
48 },
49 "Retry": [
50 {
51 "ErrorEquals": [
52 "Lambda.ServiceException",
53 "Lambda.AWSLambdaException",
54 "Lambda.SdkClientException"
55 ],
56 "IntervalSeconds": 2,
57 "MaxAttempts": 6,
58 "BackoffRate": 2
59 }
60 ],
61 "Next": "Get Location from IP Address",
62 "ResultSelector": {
63 "ipaddress.$": "$.Payload.origin"
64 }
65 },
66 "Get Brewery by City": {
67 "Type": "Task",
68 "Resource": "arn:aws:states:::lambda:invoke",
69 "Parameters": {
70 "FunctionName": "arn:aws:lambda:us-east-1:YOUR_ACCCOUNT_NUMBER:function:ApiCaller:$LATEST",
71 "Payload": {
72 "Uri.$": "States.Format('https://api.openbrewerydb.org/breweries?by_city={}', $.city)"
73 }
74 },
75 "Retry": [
76 {
77 "ErrorEquals": [
78 "Lambda.ServiceException",
79 "Lambda.AWSLambdaException",
80 "Lambda.SdkClientException"
81 ],
82 "IntervalSeconds": 2,
83 "MaxAttempts": 6,
84 "BackoffRate": 2
85 }
86 ],
87 "Next": "Breweries Returned?",
88 "ResultPath": "$.result"
89 },
90 "Breweries Returned?": {
91 "Type": "Choice",
92 "Choices": [
93 {
94 "Not": {
95 "Variable": "$.result.Payload[0]",
96 "IsPresent": true
97 },
98 "Next": "Get Brewery by State"
99 }
100 ],
101 "Default": "Extract Results"
102 },
103 "Get Brewery by State": {
104 "Type": "Task",
105 "Resource": "arn:aws:states:::lambda:invoke",
106 "Parameters": {
107 "FunctionName": "arn:aws:lambda:us-east-1:YOUR_ACCCOUNT_NUMBER:function:ApiCaller:$LATEST",
108 "Payload": {
109 "Uri.$": "States.Format('https://api.openbrewerydb.org/breweries?by_state={}', $.region_name)"
110 }
111 },
112 "Retry": [
113 {
114 "ErrorEquals": [
115 "Lambda.ServiceException",
116 "Lambda.AWSLambdaException",
117 "Lambda.SdkClientException"
118 ],
119 "IntervalSeconds": 2,
120 "MaxAttempts": 6,
121 "BackoffRate": 2
122 }
123 ],
124 "ResultPath": "$.result",
125 "Next": "Extract Results"
126 },
127 "Extract Results": {
128 "Type": "Pass",
129 "End": true,
130 "OutputPath": "$.result.Payload"
131 }
132 }
133}Execution and Output
This is the execution history of the state machine -
This is the first of a list of results -
Conclusion
This is not the primary use for Step Functions, but for a situation where you need to orchestrate a few APIs, this feels like a pretty good way of doing it.