13. April 2017 22:48
by Aaron Medacco

Testing Your Web Tier Auto Scaling Groups on AWS w/ Artillery.io

13. April 2017 22:48 by Aaron Medacco | 0 Comments

Amazon Web Services has made adding elasticity to your system architecture easy through their offering of auto scaling groups. By following a few easy steps, you can expand and retract your infrastructure to optimally service whatever volume of traffic your applications encounter. 

But how can you test your auto scaling group on AWS? How do you know if everything is configured correctly?

You'll need to simulate specific loads of traffic in order to test that the scaling rules you've set up are correct. You can do this simply using an easy-to-use tool called Artillery.io. Artillery.io is an open source load testing toolkit written in Node.js. I won't go into detail on everything Artillery.io can do, but I encourage you to go check it out. For our purposes, I'm only going to run a few simple commands to show how easy testing an auto scaling group can be.

In order to use Artillery.io, you will need to install Node.js. You can download and install Node.js by downloading the appropriate package here.

Auto Scaling Groups

Assuming you've installed Node.js, you can test your auto scaling group by following these steps. The installation steps for Artillery.io can also be found here.

Installing Artillery.io:

  1. Open a command prompt or terminal window.
  2. Run the following command:
    npm install -g artillery
  3. Check your installation by running this command:
    artillery dino
  4. If you see the ASCII dinosaur, that means everything is ready to go.

Testing your auto scaling group:

  1. Run the following command:
    artillery quick --duration 300 --rate 10 -n 20 http://www.yourwebsite.com/resource
    Note: You'll notice I've set my duration to 5 minutes instead of the example given by Artillery.io's documentation. This is because my auto scaling group only scales out if average CPU utilization over 5 minutes is sufficiently high. You'll need to play with these values depending on how powerful the instances in your auto scaling group are and what scaling rules you've defined.
  2. Substitute the final argument with the URL to the domain your trying to test.
  3. If you've invoked a heavy enough load, you can monitor your EC2 instances serving the requests and notice something akin to the following:

    Test Auto Scaling Group 1
  4. This should trigger your auto scaling group to provision additional instances provided you've set up the appropriate scale out rules:

    Test Auto Scaling Group 2

    For this example, you can see my auto scaling group scaled up to it's maximum (3 instances) during the course of the test and then scaled down using t2.micro instances in the launch configuration. 

If your auto scaling group scaled appropriately, you can rest assured that it will work under real traffic and that your configuration is sound. Make sure that the scale down rules also took effect and that once your use of Artillery.io has ended, your auto scaling group terminates the unnecessary instances.

Note: Some of the inspiration for this post comes from Mike Pfeiffer's Pluralsight course AWS Certified DevOps Engineer: High Availability and Elasticity which I encourage everyone to check out here.


2. April 2017 20:51
by Aaron Medacco

AWS Phoenix Meetup - Security Strategy When Migrating to the Public Cloud

2. April 2017 20:51 by Aaron Medacco | 0 Comments

A little over a week ago, I attended an AWS Meetup in Phoenix regarding how to approach security when migrating applications to the cloud. The event was sponsored by NextNet and Alert Logic and took place at the ASU research center. Charles Johnson of Alert Logic presented and did a fantastic job. This event being my first, I was expecting a dry lecture. What I got was an engaging discussion with a lot of very smart people. 


As someone who develops software, my primary takeaways involved the integration of security into the software development lifecycle, and how several of the application level frameworks are becoming the most targeted surfaces used by attackers, especially for applications in the cloud.

The movement of tearing down the walls separating development, operations and security commonly referred to as DevOps or DevSecOps was core to much of the discussion. Charles talked about how everybody involved with designing, developing, and maintaining applications in the cloud needs to take ownership of security, not just the "Security Team". Additionally, security should not be this annoying thing slapped on to an application at the end of the development lifecycle. Instead, security should be discussed early and often by application developers, system admins, and the security engineers so each piece of the application and the infrastructure powering it is designed to be secure. This means incorporating security testing alongside application unit testing from an early stage, and deciding upfront how to store API keys, login credentials, etc. as a team so they aren't exposed or hard-coded by a lazy developer. It also means constantly checking for where you might be vulnerable, and deciding how to address those vulnerabilities together. 

Incorporating security into the software development lifecycle also has the benefit of reducing the amount of tools you need to use after the fact in order to feel secure. If the application is designed from the ground up with security in mind, you shouldn't need to purchase tons of security tools in order to compensate. Charles mentioned that some of the teams he's assisted have bought expensive security products, but still haven't even implemented them months after purchase. Yikes!

And just because you've bought the latest and greatest tools, don't assume you are not vulnerable. In fact, you should assume the opposite. Assume you are vulnerable. Assume that the products you purchase and the frameworks you leverage are vulnerable, and monitor for breaches all the time. Additionally, consider what products you're buying. Are they really helping you? Most security products are designed to protect server operating systems, networking, the hypervisors, or aid in cloud management. But what about the application frameworks used by the developers? The databases, server-side apps, third-party libraries, etc? Do these products help secure those? Do the developers who are most intimate with these tools have the authority to purchase security products anyways? Maybe not. And who are the sales teams for these products selling to? Probably not developers. 

Note: I wish I had the graphic of the presentation, which showed that most of the attack surface for applications living in the cloud occurred higher up, i.e. the application level. SQL Injection, XSS, etc.

Lastly, use the tools provided by AWS. Use WAF. Use Inspector. Use Shield. Use Config. 

I'm sure I've left out a lot of information. I had a hard time concentrating on the presentation while taking notes on my laptop. However, I'm definitely going to the next event. (More cookies)

For those of you in the Phoenix area, consider checking out the AWS Phoenix Meetup and Blue Team - Greater Phoenix Area if your interested in attending these kinds of events.


25. March 2017 12:23
by Aaron Medacco

Ensuring AWS Resources in Your Account are Tagged w/ Names Using Config

25. March 2017 12:23 by Aaron Medacco | 0 Comments

If you're like me, you want everything in your Amazon Web Services account to be organized and well kept. Whether it be EC2 instances, VPCs, RDS instances, or security groups, I want context around the resources in my AWS environment so I know what I'm working with. Tagging accomplishes this by allowing you to ascribe attributes that you define to everything in your environment. The most common tag is simply "Name", which at a minimum, usually provides some insight into whether the instance is a web server, test instance, database server, cache, etc. 

Note: Don't name your instance Foo.

This becomes difficult when you have more than one person managing an account. From a practical standpoint, it's unreasonable to mandate that every single component to every thing you build in AWS have a name. For instance, if someone didn't tag a NACL attached to one of your subnets, it's probably not a big deal. In a utopia that would be nice, but would get in the way of getting things done. That being said, I don't think it's unreasonable to expect that infrastructure pieces such as EC2 instances, RDS instances, VPCs, EBS Volumes, ACM Certificates, ELBs, etc. always be tagged with a name.

AWS Config

The AWS Config service helps you ensure that practices such as tagging (among more important configurations like security and compliance) are maintained in your organization's AWS account. Simply specify what resources you want Config to record, select a predefined or new SNS topic to publish to, and create a rule defining what you want Config to keep tabs on. I'll assume you've gone thru the initial Config setup process for defining what resource you want recorded, the S3 bucket to store history and snapshots, and the SNS topic you want Config to publish to.

Adding a Config rule to monitor tagging:

  1. In your management console, navigate to the Config service.
  2. Click "Rules" in the sidebar.
  3. We're going to use an AWS managed rule, so browse the managed rule until you find "required-tags" and select it.
  4. Edit or accept the default name and description.
  5. Select "Resources" for the Scope of changes.
  6. Choose which AWS resources in your environment you want Config to monitor.
    By default, it's a large group, so you might want to customize this part unless you want to get notifications all day. 
  7. Under the rule parameters, change the value of "tag1Key" to "Name".
  8. Leave everything else, and click "Save".


That's it. Regardless of whether your environment adheres to this rule already, you'll likely receive several notifications right away. Config needs to evaluate the rule on the resources you selected, and mark them as COMPLIANT or NON_COMPLIANT. From this point, you can fix any NON_COMPLIANT resources and know that Config is tracking your environment going forward. The pricing for Config can be found here.


23. March 2017 00:18
by Aaron Medacco

Enforcing HTTPS For ASP.NET Applications Hosted on EC2 Behind an AWS Elastic Load Balancer

23. March 2017 00:18 by Aaron Medacco | 3 Comments

Note: This post addresses issues with infinite redirects when attempting to enforce HTTPS using an Application Load Balancer. Those using a Classic Load Balancer should not have this issue if they are passing HTTPS traffic to instances on port 443 and HTTP traffic to instance on port 80. Application Load Balancers only allow one forwarding port to instances, which can cause issues outlined in this post.

If you've ever managed ASP.NET web applications hosted on EC2 that are load balanced by an Application Load Balancer, you may have run into issues enforcing users to communicate over SSL. Having SSL termination occur on the ELB is common practice, and removes some work from the instances themselves. In fact, this is a primary use for the AWS Certificate Manager. Find out how to get an SSL certificate for your ELB in my earlier post on provisioning an SSL certificate using AWS Certificate Manager.

 AWS Elastic Load Balancer

The traditional method for enforcing requests to occur with HTTPS instead of HTTP with an ASP.NET web application is to use the URL Rewrite Module within IIS. A quick Google search for how to do this will result in numerous examples using web.config entries that look like or are very similar to the following:

<rule name="HTTP to HTTPS redirect" stopProcessing="true">
  <match url="(.*)" />
      <add input="{HTTPS}" pattern="off" ignoreCase="true" />
  <action type="Redirect" redirectType="Permanent" url="https://{HTTP_HOST}/{R:1}" />

This can result in a series of infinite redirects when you try to visit a webpage. This happens because SSL is being terminated on the Application Load Balancer. That means that when requests come in to the load balancer over HTTPS, they are sent to the EC2 instances over HTTP. Looking at our rewrite rule configuration, these requests will then be redirected to use HTTPS, which will come in via the load balancer, ad infinitum. Thus, regardless of what protocol users visit your website with, they will never land on a page due to the cycle.

However, this doesn't mean we can't use the URL Rewrite Module for IIS to enforce SSL usage of our site. I'm going to give some credit to Ross Pace at Stack Overflow for answering his own question regarding this very issue. In his answer, you'll find the following rule which accomplishes what we're after:

<rule name="Force Https" stopProcessing="true">
   <match url="healthcheck.html" negate="true" />
       <add input="{HTTP_X_FORWARDED_PROTO}" pattern="https" negate="true" />
   <action type="Redirect" url="https://{HTTP_HOST}{REQUEST_URI}" redirectType="Permanent" />

This gets around our issue. But why?

You can see the rule does not take effect (negate="true") if the {HTTP_X_FORWARDED_PROTO} header has a pattern of HTTPS. This header forwards the protocol that was sent to the application load balancer. Therefore, if the user used HTTPS to connect to the load balancer, there is no need to redirect. If the value is anything else, that means we need to perform the redirect. Simple.

You'll also notice that this rule is ignored if the URL matches "healthcheck.html". As explained in his response, this is required to prevent the ELB health checks from failing because of URL Rewrite Module. You can substitute the URL value for whichever page you are using for your own ELB health check. 

Hopefully this saves some .NET developers from walking outside with a pistol. :)


21. March 2017 00:28
by Aaron Medacco

Moving Load From Your Master Database: Elasticache or RDS Read Replica?

21. March 2017 00:28 by Aaron Medacco | 0 Comments

You have a database managed thru the AWS RDS service. Your application's a massive success and your database's workload is now heavy enough that users are experiencing long response times. You decide that instead of scaling vertically by upgrading the instance type of your RDS database, you'd like to explore implementing Elasticache or an RDS read replica into your architecture to remove some of the load from your master database.

But which one should you choose?

Like always, it depends.

RDS read replicas and Elasticache nodes both enhance the performance of your application by handling requests for data instead of the master database. However, which one you choose will depend on your application's requirements. Before I dive too deep into how these requirements will shape your decision, let's talk about what we are comparing first.

Elasticache Or Read Replica

Note: For those already familiar with Elasticache and RDS features, feel free to skip down.

Elasticache Clusters

Elasticache is a managed service provided by AWS that allows you to provision in-memory data stores (caches) that can allow your applications to fetch information with blazing speed. When you use Elasticache, you create a cluster of nodes, which are blocks of network attached RAM. This cluster can then sit in between your application tier and your data tier. When requests that require data come in, they are sent to the cluster first. If the information exists in the cache and the cluster can service the request, it is returned to the requester without requiring any database engine steps or disk reads. If the information does not exist in the cache, it must be fetched from the database like usual. Obviously, their is no contest in performance between fetching data from memory vs. disk which is why the Elasticache service can really give your applications some wheels while also reducing the load on your database.

RDS Read Replicas

RDS read replicas offer an alternative to vertical database scaling by allowing you to use additional database instances to serve read-only requests. Read replicas are essentially copies of your master database where write operations are prohibited except thru asynchronous calls made from the master after the master has completed a write. This means that read replicas may return slightly stale data when serving requests, but will eventually catch up as write propagations invoked from the master complete. Read replicas have additional benefits as well. Since they can be promoted to serve as the master database should the master fail, you can increase your data's availability. The database engine you choose also determines available features. For instance, databases utilizing the MySQL database engine can take advantage of custom read replica indexes which only apply to the replicas. Want to take advantage of covering indexes for expensive queries without forcing your master database to maintain additional writes? With this feature, you'd be able to.

Great. So which is better?

Clearly, both of these services can reduce your master database's workload, while also boosting performance. In order to choose which service makes most sense for your situation, you'll need to answer these kinds of questions:

Can my application tolerate stale data? How stale? 5 minutes? 5 hours?

If you want to store your data in Elasticache nodes for long periods of time, and you need almost exactly current data, read replicas are likely the better option. Read replicas will lag behind the master database slightly, but only by seconds. On the other hand, if you can tolerate data that is more stale, Elasticache will outperform read replicas performance-wise (if data exists in the cache) while also preventing requests from hitting the master database. 

Are the queries generated by my application static? That is, are the same queries run over and over all day? Are they dynamically constructed, using items like GETDATE()?

If the queries being run by your application are ever-changing, your in-memory cache won't be very useful since it will have to continue to update its data store to serve requests it otherwise cannot satisfy. Remember, the cache can only act on queries it recognizes. Not only does this not prevent calls to the database, but it can actually degrade performance because you are effectively maintaining a useless middle man between your application and data tiers. However, Elasticache clusters will shine over read replicas in cases where queries do not change and request the same data sets over and over.

How much data is my application asking for?

The volume of data your Elasticache clusters can store will be limited by the amount of memory you allocate to the nodes in your cluster. If your queries return huge result sets, your cache is going to be occupied very quickly. This can create a scenario where queries constantly compete for the available memory by overwriting what's existing. This won't be very helpful, especially if you aren't willing to dedicate (and pay for) additional memory. Alternatively, read replicas will be able to serve the request regardless of the returned data size, won't incur the penalty of a wasted middle man trip, and will still prevent the request from hitting the master.

What would I do?

My approach would be to consider how effectively you'll be able to use an in-memory cache. If, given all you know about your application, you decide the cache will be able to catch a large portion of the read requests coming in, go with the Elasticache cluster. If you're right, you should notice a difference right away. If you're wrong you can always fall back to implementing read replicas. I've avoided pricing in this post, which is obviously important when making architectural decisions. Elasticache pricing and RDS pricing is available on Amazon Web Services' site. Readers will need to do their own analysis of the kinds of instances and node types they'd provision and how their costs compare to make a decision.

If anyone knows of some additional considerations I should include, leave a comment or reach out to me via e-mail: acmedacco@gmail.com. Shout out to the last guy who caught my miscalculation of pricing in the Infinite Lambda Loop post


Copyright © 2016-2017 Aaron Medacco