22. July 2017 19:05
by Aaron Medacco
0 Comments

Scheduling Notifications for Rotating Old IAM Access Keys

22. July 2017 19:05 by Aaron Medacco | 0 Comments

Access keys allow you to give programmatic access to a user so they can accomplish tasks and interact with services within your AWS environment. These keys should be heavily guarded and kept secret. Once exposed, anyone can wreak havoc in your AWS account using any permissions that were granted to the user's whose keys were exposed.

A best practice for security is to rotate these keys regularly. We want to keep our access keys fresh and not have old or unused keys running about waiting to be abused. Therefore, I've created an automated method for notifying an administrator when user access keys are old and should be rotated. Since the number of days for keys to be considered "old" varies across organizations, I've included a variable that can be configured to fit the reader's requirements.

IAM Access Key Rotation

Like my recent posts, this will make use of the AWS CLI, which I assume you've installed and configured already. We'll be using (of course) Lambda backed by a CloudWatch event rule. Notifications will be sent using the SNS service when keys should be rotated. Enjoy.

Creating an SNS topic:

  1. Open up a command prompt or terminal window, and invoke the following command:
    aws sns create-topic --name IAM-Access-Key-Rotation-Topic
  2. You'll get a Topic ARN value back which you want to keep.
  3. Then invoke the following command. Substitute Email for the email address you want to receive the notifications.
    Note: You don't have to use email if you don't want to. Feel free to use whichever protocol/endpoint fits you.
    aws sns subscribe --topic-arn ARN --protocol email --notification-endpoint Email
  4. The email will get an email asking them to confirm a subscription. You'll need to confirm the subscription before moving on if you want notifications to go out.

Creating an IAM policy for access permissions:

  1. Create a file named iam-policy.json with the following contents and save it in your working directory. Substitute Your Topic ARN for the ARN of the topic you just created:
    {
        "Version": "2012-10-17",
        "Statement": [
            {
                "Effect": "Allow",
                "Action": [
                    "sns:Publish"
                ],
                "Resource": [
                    "Your Topic ARN"
                ]
            },
            {
                "Effect": "Allow",
                "Action": [
                    "iam:ListAccessKeys",
                    "iam:ListUsers"
                ],
                "Resource": [
                    "*"
                ]
            }
        ]
    }
  2. In your command prompt or terminal window, execute the following command:
    aws iam create-policy --policy-name rotate-old-access-keys-notification-policy --policy-document file://iam-policy.json
  3. You'll receive details for the policy you just created. Write down the ARN value. You will need it in a later step.

Creating the IAM role for the Lambda function:

  1. Create a file named role-trust-policy.json with the following contents and save it in your working directory:
    {
      "Version": "2012-10-17",
      "Statement": [
        {
          "Effect": "Allow",
          "Principal": {
            "Service": "lambda.amazonaws.com"
          },
          "Action": "sts:AssumeRole"
        }
      ]
    }
  2. In your command prompt or terminal window, invoke the following command:
    aws iam create-role --role-name rotate-old-access-keys-notification-role --assume-role-policy-document file://role-trust-policy.json
  3. You'll get back information about the role you just made. Write down the ARN value for the role. You will need it when we go to create the Lambda function.
  4. Invoke this command to attach the policy to the role. Substitute ARN for the policy ARN you received when you created the IAM policy.
    aws iam attach-role-policy --policy-arn ARN --role-name rotate-old-access-keys-notification-role

Creating the Lambda function:

  1. Create a files named index.js with the following contents and save it in your working directory. Make sure you substitute Topic ARN with the ARN of the topic you created in the first step.
    Note: Notice you can configure the number of days allowed before rotations should be done. Default value I set is 90.
    var AWS = require("aws-sdk");
    var iam = new AWS.IAM();
    var sns = new AWS.SNS();
    
    var daysBeforeRotationRequired = 90; // Number of days from access creation before action is taken on access keys.
    
    exports.handler = (event, context, callback) => {
        var listUsersParams = {};
        iam.listUsers(listUsersParams, function(err, data) {
            if (err) {
                console.log(err, err.stack);
            }
            else {
                for (var i = 0; i < data.Users.length; i++) {
                    var userName = data.Users[i].UserName;
                    var listAccessKeysParams = { UserName: userName };
                    iam.listAccessKeys(listAccessKeysParams, function(err, data) {
                        if (err) {
                            console.log(err, err.stack);
                        } 
                        else {
                            for (var j = 0; j < data.AccessKeyMetadata.length; j++) {
                                var accessKeyId = data.AccessKeyMetadata[j].AccessKeyId;
                                var creationDate = data.AccessKeyMetadata[j].CreateDate;
                                var accessKeyUserName = data.AccessKeyMetadata[j].UserName;
                                var now = new Date();
                                var whenRotationShouldOccur = dateAdd(creationDate, "day", daysBeforeRotationRequired);
                                if (now > whenRotationShouldOccur) {
                                    var message = "You need to rotate access key: " + accessKeyId + " for user: " + accessKeyUserName
                                    console.log(message);
                                    var publishParams = {
                                        Message: message,
                                        Subject: "Access Keys Need Rotating For User: " + accessKeyUserName,
                                        TopicArn: "Topic ARN"
                                    };
                                    sns.publish(publishParams, context.done);
                                }
                                else {
                                    console.log("No access key rotation necessary for: " + accessKeyId + "for user: " + accessKeyUserName);
                                }
                            }
                        }
                    });
                }
            }
        });
    };
    
    var dateAdd = function(date, interval, units) {
        var ret = new Date(date); // don't change original date
        switch(interval.toLowerCase()) {
            case 'year'   :  ret.setFullYear(ret.getFullYear() + units);  break;
            case 'quarter':  ret.setMonth(ret.getMonth() + 3*units);  break;
            case 'month'  :  ret.setMonth(ret.getMonth() + units);  break;
            case 'week'   :  ret.setDate(ret.getDate() + 7*units);  break;
            case 'day'    :  ret.setDate(ret.getDate() + units);  break;
            case 'hour'   :  ret.setTime(ret.getTime() + units*3600000);  break;
            case 'minute' :  ret.setTime(ret.getTime() + units*60000);  break;
            case 'second' :  ret.setTime(ret.getTime() + units*1000);  break;
            default       :  ret = undefined;  break;
        }
        return ret;
    }
  2. Zip this file to a zip called index.zip.
  3. Bring your command prompt or terminal window back up, and execute the following command. Substitute ARN for the role ARN you received from the step where we created the role:
    aws lambda create-function --function-name rotate-old-access-keys-notification-sender --runtime nodejs6.10 --handler index.handler --role ARN --zip-file fileb://index.zip --timeout 30
  4. Once the function is created, you'll get back details for the function. Write down the ARN value for the function for when we schedule the function execution.

Scheduling the Lambda function:

  1.  In your command prompt or terminal window, execute the following command:
    Note: Feel free to adjust the schedule expression for your own purposes.
    aws events put-rule --name Rotate-Old-Access-Keys-Notification-Scheduler --schedule-expression "rate(1 day)"
  2. Write down the ARN value for the rule upon creation finishing.
  3. Run the following command. Substitute ARN for the Rule ARN you just received:
    aws lambda add-permission --function-name rotate-old-access-keys-notification-sender --statement-id LambdaPermission --action "lambda:InvokeFunction" --principal events.amazonaws.com --source-arn ARN
  4. Now run the following command. Substitute ARN for that of the Lambda function you created in the previous step:
    aws events put-targets --rule Rotate-Old-Access-Keys-Notification-Scheduler --targets "Id"="1","Arn"="ARN"

For information regarding ways to rotate your access keys without affecting your applications, click here.

Cheers!

25. March 2017 12:23
by Aaron Medacco
0 Comments

Ensuring AWS Resources in Your Account are Tagged w/ Names Using Config

25. March 2017 12:23 by Aaron Medacco | 0 Comments

If you're like me, you want everything in your Amazon Web Services account to be organized and well kept. Whether it be EC2 instances, VPCs, RDS instances, or security groups, I want context around the resources in my AWS environment so I know what I'm working with. Tagging accomplishes this by allowing you to ascribe attributes that you define to everything in your environment. The most common tag is simply "Name", which at a minimum, usually provides some insight into whether the instance is a web server, test instance, database server, cache, etc. 

Note: Don't name your instance Foo.

This becomes difficult when you have more than one person managing an account. From a practical standpoint, it's unreasonable to mandate that every single component to every thing you build in AWS have a name. For instance, if someone didn't tag a NACL attached to one of your subnets, it's probably not a big deal. In a utopia that would be nice, but would get in the way of getting things done. That being said, I don't think it's unreasonable to expect that infrastructure pieces such as EC2 instances, RDS instances, VPCs, EBS Volumes, ACM Certificates, ELBs, etc. always be tagged with a name.

AWS Config

The AWS Config service helps you ensure that practices such as tagging (among more important configurations like security and compliance) are maintained in your organization's AWS account. Simply specify what resources you want Config to record, select a predefined or new SNS topic to publish to, and create a rule defining what you want Config to keep tabs on. I'll assume you've gone thru the initial Config setup process for defining what resource you want recorded, the S3 bucket to store history and snapshots, and the SNS topic you want Config to publish to.

Adding a Config rule to monitor tagging:

  1. In your management console, navigate to the Config service.
  2. Click "Rules" in the sidebar.
  3. We're going to use an AWS managed rule, so browse the managed rule until you find "required-tags" and select it.
  4. Edit or accept the default name and description.
  5. Select "Resources" for the Scope of changes.
  6. Choose which AWS resources in your environment you want Config to monitor.
    By default, it's a large group, so you might want to customize this part unless you want to get notifications all day. 
  7. Under the rule parameters, change the value of "tag1Key" to "Name".
  8. Leave everything else, and click "Save".

 

That's it. Regardless of whether your environment adheres to this rule already, you'll likely receive several notifications right away. Config needs to evaluate the rule on the resources you selected, and mark them as COMPLIANT or NON_COMPLIANT. From this point, you can fix any NON_COMPLIANT resources and know that Config is tracking your environment going forward. The pricing for Config can be found here.

Cheers!

28. February 2017 23:41
by Aaron Medacco
0 Comments

Setting up Alerts for When Critical EBS Volumes Start Running Out of Space on Windows EC2 Instances

28. February 2017 23:41 by Aaron Medacco | 0 Comments

Note: This solution is for AWS users using EC2 instances w/ a Windows operating system. Linux users can employ a similar design, but won't be able to use the exact pieces in this solution.

If you're like me, you'd like to have a heads up on when your EC2 instances are running out of space. For example, if you're managing your own database server instead of using RDS, and have the data written to a particular EBS volume, it sure would be nice to know it's running out of space before, well...it's run out of space. In this post, we'll take the proactive approach by setting up monitoring and an alarm on our volumes' available space . That way, we'll know we need to provision extra storage ahead of time, instead of in response to an angry phone call.

Unfortunately, while Amazon Web Services does keep track of several metrics regarding your instances and EBS volumes within AWS CloudWatch, available space is not one of them. However, they do allow you to submit your own custom metrics and use all of the features of CloudWatch on them. Therefore, we can calculate how much free disk space we have ourselves and leverage CloudWatch when it comes to monitoring.

EBS Volume Usage Alarm

Specifically, this solution employs a simple PowerShell script that, when scheduled using Task Scheduler, can calculate and submit what percentage of our volume has already been written to. Once this metric is provided to CloudWatch on a recurring basis, we'll setup a CloudWatch alarm to monitor it and send out a notification via SNS topic if our specified threshold has been crossed (i.e. volume usage has passed 80%).

The PowerShell script uses the AWS CLI which you may need to configure if you haven't already on your EC2 instance(s). You can find the instructions for that here.

Creating an SNS topic to send notifications:

  1. Navigate to SNS in your management console.
  2. Select "Topics" in the sidebar.
  3. Click the "Create new topic" button.
  4. Enter an appropriate topic name and display name and click "Create topic".

Subscribing to the SNS topic:

  1. Select "Topics" in the sidebar.
  2. Click the ARN link for the topic you just created.
  3. Under Subscriptions, click "Create subscription".
  4. Select Email as the Protocol and enter your email address as the Endpoint.
  5. Repeat steps 3 and 4 for each email address you want to receive notifications.
  6. Each email address endpoint will receive an email asking to confirm the subscription. Confirm the subscriptions.

Setting up the PowerShell script that will submit our CloudWatch metric:

  1. Create a PowerShell script file and paste the following code:
    #Parameters
    $computerName = "Your EC2 Instance Hostname"; #For example, "EC2AMAZ-XXXXXXX"
    $deviceId = "Your Volume Device ID"; #For example, "C:"
    $instanceId = "Your EC2 Instance ID"; #For example, "i-xxxxxxxxxxxxxxxxx"
    
    #Send To CloudWatch
    $metricName = "EBS Volume Usage";
    $deviceCommand = "DeviceID='" + $deviceId + "'";
    $unit = "Count";
    $disk = Get-WmiObject Win32_LogicalDisk -ComputerName $computerName -Filter $deviceCommand | Select-Object Size,FreeSpace;
    $value = (100 - $disk.FreeSpace / $disk.Size * 100);
    Write-Host "Posting Volume Usage To CloudWatch: $($value)";
    aws cloudwatch put-metric-data --metric-name $metricName --namespace "Custom Metrics" --value $value --dimensions InstanceId=$instanceId,DeviceID=$deviceId;
    Write-Host "Done";
    You can also download the script here.
  2. Substitute the parameter values with those that apply to your EC2 instance and volume. 

Configure Task Scheduler to run the PowerShell script at 5 minute intervals:

  1. Start up Task Scheduler on your EC2 instance.
  2. Navigate to the Task Scheduler Library.
  3. On the right hand side, click "Create Task...".
  4. On the General tab, enter an appropriate Name and Description.
  5. Under Security options, select "Run whether user is logged on or not" and select the appropriate Windows Server version in the "Configure for:" drop-down.
    Task Scheduler 1
  6. On the Triggers tab, click "New...".
  7. Select "On a schedule" for Begin the task.
  8. Under Settings, select "One time" and select a start time.
  9. Under Advanced settings, select "Repeat task every:", 5 minutes, and "Indefinitely" for the duration.
  10. Make sure the Enabled checkbox is checked.
    Task Scheduler 2
  11. Click "OK".
  12. On the Actions tab, click "New...".
  13. Select "Start a program" for the Action drop-down.
  14. Under Program/script, enter the path to the executable for PowerShell on your machine.
  15. In the Add arguments (optional) field, enter the absolute path to the PowerShell script.
    Task Scheduler 3
  16. Click "OK".
  17. On the Conditions tab, I left all highest level checkboxes unchecked.
    Task Scheduler 4
  18. On the Settings tab, configure the options as you desire.
  19. Click "OK".
  20. Run your newly scheduled task.

At this point, you should begin seeing the custom metric in your CloudWatch management console. If this isn't the case, there may be an issue where the task is not running properly. 

Note: When you created the Action in your scheduled task, if you instead chose to enter the path of the PowerShell script under Program/script, it's possible that Task Scheduler is opening the script in Notepad instead of actually running the commands.

Creating the CloudWatch alarm which will send notifications when our EBS volume is close to reaching full capacity:

  1. Navigate to CloudWatch in your management console.
  2. Select "Alarms" in the sidebar.
  3. Click "Create Alarm".
  4. Select the metric you just created. It should be within the "Custom Metrics" namespace.
  5. Click "Next".
  6. Enter an appropriate Name and Description.
  7. Set the threshold to meet your requirements. For my personal use, I configure the alarm for when space usage is >= 80 for 1 consecutive period.
  8. Under Actions, select "State is ALARM" and the name of the SNS topic you created earlier for the notification.
  9. Click "Create Alarm".

You can test that the alarm is working by using a threshold value below what your current volume space usage is, and checking that the SNS topic was published to.

If anyone has any improvements to this solution or issues implementing it, feel free to leave a note in the comments.

Cheers!

30. December 2016 19:00
by Aaron Medacco
0 Comments

Automating Alerts for Unassociated Elastic IPs w/ AWS

30. December 2016 19:00 by Aaron Medacco | 0 Comments

Amazon charges for Elastic IP addresses that are allocated, but not associated with a running instance. This is to discourage AWS customers from wasting the dwindling pool of available iPv4 addresses available. Wouldn't it be nice if, as someone who manages AWS resources, you received alerts when your account's allocated Elastic IPs are being wasted?

I've created an automated process to send out an email when this occurs. Using a simple Lambda function (triggered by a CloudWatch schedule) and an SNS topic, notifications can be sent to the appropriate employees when someone forgets to cleanup after terminating their instances.

Elastic IP Waste Diagram

Creating the SNS topic:

  1. Navigate to SNS in your management console.
  2. Select "Topics" in the sidebar.
  3. Click the "Create new topic" button.
  4. Enter an appropriate topic name and display name and click "Create topic".

Subscribing to the SNS topic:

  1. Select "Topics" in the sidebar.
  2. Click the ARN link for the topic you just created.
  3. Under Subscriptions, click "Create subscription".
  4. Select Email as the Protocol and enter your email address as the Endpoint.
  5. Repeat steps 3 and 4 for each email address you want to receive notifications.
  6. Each email address endpoint will receive an email asking to confirm the subscription. Confirm the subscriptions.

Creating an IAM policy for access permissions:

  1. Navigate to IAM in your management console.
  2. Select "Policies" in the sidebar.
  3. Click "Create Policy".
  4. Select "Create Your Own Policy".
  5. Enter an appropriate policy name and description.
  6. Paste the following JSON into the policy document:

    {
        "Version": "2012-10-17",
        "Statement": [
            {
                "Effect": "Allow",
                "Action": [
                    "sns:Publish",
                    "sns:Subscribe"
                ],
                "Resource": [
                    "Your Topic ARN"
                ]
            },
            {
                "Effect": "Allow",
                "Action": [
                    "ec2:DescribeAddresses"
                ],
                "Resource": [
                    "*"
                ]
            }
        ]
    }
  7. Substitute "Your Topic ARN" with the ARN for the SNS topic you created and click "Create Policy".

Creating an IAM role for the Lambda function:

  1. Select "Roles" in the sidebar.
  2. Click "Create New Role".
  3. Enter an appropriate role name and click "Next Step".
  4. Select "AWS Lambda" within the AWS Service Roles.
  5. Change the filter to "Customer Managed", check the box of the policy you just created, and click "Next Step".
  6. Click "Create Role".

Creating the Lambda function:

  1. Navigate to Lambda in your management console.
  2. Click "Create a Lambda function".
  3. Select the "Blank Function" blueprint.
  4. Under "Configure triggers", click the grey box and select "CloudWatch Events - Schedule".
  5. Enter an appropriate rule name and description.
  6. Select the frequency you'd like Lambda to check for unassociated Elastic IPs in the Schedule expression input. I chose "rate(1 day)" for my usage.
  7. Check the box to "Enable trigger" and click "Next".
  8. Enter an appropriate function name and description. Select Node.js for the runtime.
  9. Under "Lambda function code", select "Edit code inline" for the Code entry type and paste the following code in the box:

    var AWS = require("aws-sdk");
    
    exports.handler = function(event, context) {
        var sns = new AWS.SNS();
        var ec2 = new AWS.EC2();
        var message = "The following Elastic IPs are not associated:\n\n";
        var params = {};
        ec2.describeAddresses(params, function(err, data) {
            if (err) {
                console.log(err, err.stack); 
            }
            else {
                var unassociatedAddresses = 0;
                for (var i = 0; i < data.Addresses.length; i++){
                    if (!data.Addresses[i].hasOwnProperty("InstanceId")){
                        console.log(data.Addresses[i].PublicIp);
                        unassociatedAddresses++;
                        message += " " + data.Addresses[i].PublicIp + "\n";
                    }
                }
                if (unassociatedAddresses > 0){
                    var publishParams = {
                        Message: message, 
                        Subject: "Elastic IP Addresses Unassociated",
                        TopicArn: "Your Topic ARN"
                    };
                    sns.publish(publishParams, context.done);
                }
            }
        });
    };
  10. Substitute "Your Topic ARN" with the ARN for the SNS topic you created earlier.
  11. Leave Handler as "index.handler".
  12. Choose to use an existing role and select the IAM role you created earlier.
  13. Leave the other default values and click "Next".
  14. Click "Create function".

That's it! Now you'll at least be made aware when your Elastic IPs are being wasted. Hopefully before whoever is paying your account's AWS bill.

Cheers!

Copyright © 2016-2017 Aaron Medacco