13. March 2017 01:01
by Aaron Medacco
1 Comments

What Happens When Your Lambda Functions Execute in an Infinite Loop?

13. March 2017 01:01 by Aaron Medacco | 1 Comments

Amazon Web Services' serverless compute product, Lambda, is an event-driven service that executes code without requiring the customer to manage servers. It can be triggered in response to a variety of different events or manually triggered via the AWS CLI, Shell tools, SDKs, and web console. Lambda functions can do just about anything within your AWS environment, given the appropriately configured IAM role.

So what happens if you architect (or misarchitect) a solution where your Lambda functions execute more than you anticipated? What if your Lambda functions are written so that they call themselves or other Lambda functions where, given a limitless infrastructure to facilitate, execution should logically never stop?

While the AWS platform is not actually limitless, you certainly can rack up a massive amount of compute if you're not careful. Given the pricing model of Lambda, that massive compute will become a massive bill that punching the developer who wrote the functions won't pay for.

Note: There are cases where recursive Lambda invocations are intentional and do solve problems. This post addresses concern where infinite looping is unintentional.

AWS Lambda Infinite Loop

For new and old AWS accounts that have never requested a limit increase, the maximum number of concurrent executions across all your Lambda functions in a region is restricted to 100. Apparently, Amazon has thought of this scenario, and implemented some safety rules to prevent customers from coding themselves into a real problem.

"The default limit is a safety limit that protects you from costs due to potential runaway or recursive functions during initial development and testing."

You can find Amazon's documentation on Concurrent Exections w/ Lambda here.

However, I decided to try it out myself, anyways. I did this by writing a small Lambda function which invoked itself:

var AWS = require("aws-sdk");

exports.handler = (event, context, callback) => {
    var lambda = new AWS.Lambda();
    var params = {
        FunctionName: "Infinite_Lambda_Loop",
        InvocationType: "Event",
    };
    lambda.invoke(params, function(err, data){
        if (err) {
            console.log(err, err.stack);
        }
        else {
            console.log(data);
        }
    })
};

I used a timeout of 3 seconds and the minumum value for memory (128 MB) in all of my trials. I ran each trial in a separate region to get a clean graphing dashboard from Lambda. Therefore, I am assuming that Lambda performs similarly across all regions.

Here are my Lambda statistics as seen in my dashboard after running my function in Ohio after 60 seconds:

Lambda Invocations

In Northern California for 120 seconds (separate trial):

Lambda Invocations 2

In Oregon for 300 seconds (separate trial):

Lambda Invocations 3

I found that the only way to end the madness was to delete the function. If anybody else knows a different way, leave a comment.

After validating the values in my graphs with those found in CloudWatch, my data is as follows:

Region Seconds Invocations Duration (ms) Memory / Execution (MB)
Ohio 60 573 41,731 128
N. California 120 1,523 68,907 128
Oregon 300 3,021 190,985 128

Curiously, the count of invocations and duration change was not exactly linear, but from this data we can estimate that about 3,000 executions take place during 5 minutes of a very simple Lambda function executing itself forever. Again, this assumes that Lambda performance is uniform across regions. If we calculate our estimated cost for this, we get:

3,000 * $0.0000002 / request = $0.00006 Lambda cost for requests.

Price per 100ms with 128 MB = $0.000000208, so 1,910 * $0.000000208 = $0.00039728 Lambda cost for duration.

$0.00006 + $0.00039728 = $0.00045728 cost for my infinite loop executing for 5 minutes.

$0.00045728 * 288 ~ $0.13 cost for my infinite loop executing for 1 day.

$0.13 * 7 ~ $0.91 cost for my infinite loop executing for 1 week.

$0.13 * 30 ~ $3.90 cost for my infinite loop executing for 1 month.

These costs do not reflect any savings gained by Lambda free tier discounts. Information about reducing costs per the free tier can be found here.

Therefore, I think it's safe to say you won't kill the budget with an accidental infinite loop (assuming your concurrency limit is the default 100).

Keep in mind I've made assumptions in my analysis of this data. Additionally, the pricing of AWS products changes frequently, and cost estimation for this scenario was done using Lambda pricing at the time of writing.

Cheers!

Comments (1) -

Looks like the maximum account limit is now 1000 (docs.aws.amazon.com/.../concurrent-executions.html). I'd assume this increases the potential tests somewhat from when you did them.

Having said that, I racked up about $5 of charges in 10 minutes when I accidentally invoked a Lambda function 6 million times after subscribing it to every single CloudTrail event in my account. I'm not sure how that got so high, given your results!

Add comment

Copyright © 2016-2017 Aaron Medacco