7. December 2017 02:49
by Aaron Medacco
0 Comments

AWS re:Invent 2017 - Day 4 Experience

7. December 2017 02:49 by Aaron Medacco | 0 Comments

The following is my Day 4 re:Invent 2017 experience. Missed Day 3? Check it out here.

The common theme of not waking up early continues. Missed the entire Werner Vogels Keynote, which is fine since from what others were saying it was a storm to get in. I didn't have it reserved anyways. AWS added it to the event catalog after I had already decided my schedule. Not sure why I thought you were just supposed to show up. 

First session of the day, AWS Database and Analytics State of the Union - 2017 (DAT201). This took place in the Venetian Theatre.

AWS re:Invent 2017

Love this venue for sessions.

I wasn't sure what to expect from a "State of the Union" session. For the most part, this was a combination of sales pitch, history lesson into the origin of some of the AWS database services, and explanation of miscellaneous features. After the explanation of what the RDS Multi-AZ feature does (really? who doesn't know what this is?), the session moved on to highlight the motivations for building Aurora and DynamoDB. Essentially, AWS wanted to combine the benefits of commercial-grade performance provided by products like Oracle and SQL Server with the low-cost of MySQL, PostgreSQL and MariaDB. The product of these efforts became the Aurora product. 

AWS re:Invent 2017

Horrible quality pic. I don't know why.

After sharing some history, a few of the newest Aurora features came up. Specifically, Amazon Aurora Multi-Master and Amazon Aurora Serverless. Amazon Aurora Multi-Master allows you bring the disaster recovery of a failed master instance to almost nothing, 100 ms. The single-region version of this feature is available in preview now, with the multi-region version available later in 2018. Amazon Aurora Serverless allows you to essentially have an on-demand database that scales for you and is for applications that have unpredictable workloads. Being serverless, the customer manages hardly anything.

The origins for DynamoDB came from a disaster affecting Amazon.com. I didn't write down the exact year, but basically Amazon.com was leveraging Oracle to power the online retail site, however the site became unavailable during Christmas. The cause of this was traced back to limitations in Oracle or Amazon.com's implementation of it, of this I wasn't clear. In response, Amazon built the NoSQL database service, DynamoDB, to handle massive volumes of transactions it was experiencing.

AWS re:Invent 2017

DynamoDB. But does it work at scale? Haha.

The remainder of the session focused on general overviews of the rest of the data service catalog. Therefore, a lot of the session was review for anyone who regularly spends time learning Amazon Web Services. Several attendees started leaving during this time. There's a lot going on during re:Invent so while I understand time is precious during the week, I always stay for the whole session. Maybe it's just out of respect for the presenter. Either way, I did learn a few tidbits about Aurora and some origin history of some services I didn't know before.

AWS re:Invent 2017

Big Amazon Echo.

Grabbed a shuttle to the Aria to check out what was going on at the Quad. I was expecting it to be as large as the Venetian Expo. Boy, was that off the mark! The Quad was very underwhelming, especially after visiting the Expo. There was a small handful of booths, but nothing like the Expo which was aisles and aisles full of them. I smiled when I saw the Certification Lounge there, looked like a large-sized cubicle. At this point, it became clear to me that the Venetian was definitely the primary hub for re:Invent. Good to know for next year's hotel room reservations.

They did have a cool Lego section, though.

During my video filming of the re:Invent areas of the Aria, I got yelled at by some woman telling me I couldn't record. I waited until she stopped looking, and turned it back on after. What a joke! Now, I actually was told not to video record prior to this while in the Venetian casino, which while I'd argue is unenforceable, makes more sense to me. However, back in the Aria, what is there to film that can cause harm!? It's a bunch of nerds with laptops walking around banners, session rooms, and re:Invent help desks a quarter mile from the casino floor! Ridiculous. It's 2017, and there's a tech conference going on here. Are you going to watch every attendee's laptop, tablet and smartphone, too, because guess what, those devices can do video recording as well. Unenforceable and moronic. Anyways...

AWS re:Invent 2017

In case the re:Invent app fails you.

Returned to my room after grabbing a Starbucks and did some blogging until my next session at the Venetian, Taking DevOps Closer to the AWS Edge (CTD401). 

AWS re:Invent 2017

Emptier session. Was actually really good.

This session was possible my favorite of the conference. And I certainly wasn't expecting that. The session title, in my opinion, is misleading, however. Now, the presenter did say that the same session took place last year and included a demo involving saving CloudFormation templates to CodeCommit and managing a delivery pipeline with CodePipeline to push modifications to a development CloudFront distribution, perform some testing, and then do the same to a production CloudFront distribution. That seems more DevOps to me. What we got instead was an in-depth overview of how to incorporate the edge services of AWS into application design and development and how to use CloudFront

AWS re:Invent 2017

More terrible quality pics.

AWS re:Invent 2017

Logic determining the TTL CloudFront uses.

Most of the session was a deep dive and explanation of CloudFront and how it works between your origin (EC2, S3, etc.) and your clients (users). The presenter explained how the TCP connections, both from the user to the edge location, and from the edge location to the origin function as well as provided some tips for keeping your cache-hit ratio high using headers like cloudfront-is-mobile-viewer to reduce variability. Plus, there were some cool examples given of Lambda@Edge taking headers and custom modifying them in the in-between. 

AWS re:Invent 2017

Lambda@Edge examples.

I've not used CloudFront a lot, but I'm more confident about it after this session. A lot of people walked out throughout this session, probably hoping for something different. Can't say I wouldn't have wanted to do the same thing if I knew CloudFront inside and out already. Being a 400-level course, it was surprisingly easy to grasp, perhaps due to the presenter.

AWS re:Invent 2017

Lego blocks.

Back at the Bellagio, stopped in for some grub at the FIX Restaurant & Bar. Snatched a cocktail, salmon, and mashed potatoes.

AWS re:Invent 2017

9/10. Would have been 10/10 if the waitress had brought the mac n cheese I ordered. Maybe she didn't hear me? Don't know. The food coma was instant, so I took a power nap before going out to check out the re:Play party.

Which brings us to the re:Play party! AWS goes all out for this. You could only enter in through the Venetian Hall A even though it was behind the LINQ. 

AWS re:Invent 2017

Oomce, oomce, oomce.

Food and drinks were included and the place was packed. It took place under a set of large tents, one being totally dedicated to games like glow-in-the-dark pool, glow-in-the-dark ping pong, adult-sized ball container, putt-putt pool, batting cages, dodge-ball and more.

AWS re:Invent 2017

Guy got lost in the balls. They couldn't find him.

AWS re:Invent 2017

Another tent is where the rave was going on with DJ Snake.

AWS re:Invent 2017

Oomce, oomce, oomce.

AWS re:Invent 2017

Not sure what these were about.

And then a final tent was packed full of arcade style games. There were some other areas I didn't explore since the line was ridiculous or I wasn't clear how to get in. 

AWS re:Invent 2017

Ancient video games.

I didn't end up staying too long since everything had huge lines and I'm not one for live music anyways.

AWS re:Invent 2017

Walked back to the Venetian casino and played poker with other attendees from re:Invent. Lost another $300. What is going on, man! I'm just the worst, although I turned Aces up when a guy got a set. Understandable, but annoying. Came home with some Taco Bell (I know, classy) and turned in for the night.

Cheers!

5. December 2017 21:38
by Aaron Medacco
0 Comments

AWS re:Invent 2017 - Day 3 Experience

5. December 2017 21:38 by Aaron Medacco | 0 Comments

The following is my Day 3 re:Invent 2017 experience. Missed Day 2? Check it out here.

Out and early at 8:30 (holy ****!) and was famished. Decided to check out the Buffet at the Bellagio. Maybe it was just me, but I was expecting a little bit more from this. Most things in Las Vegas are extravagant and contribute to an overall spectacle, but the Bellagio Buffet made me think of Golden Corral a bit. Maybe it gets better after breakfast, I don't know. 

AWS re:Invent 2017

AWS re:Invent 2017

The plate of an uncultured white bread American.

Food was pretty good, but I didn't grab anything hard to mess up. Second trip back, grabbed some watermelon that tasted like apple crisp and ice cream. Not sure what that was about. Maybe the staff used the same knife for desserts and fruit slicing. From what I could tell, half the patrons were re:Invent attendees, either wearing their badge or the hoodie.

Walked back to my room to watch the Keynote by Andy Jassy, but only caught the last bit of it. After some difficulty getting the live stream to work on my laptop, watched him announce the machine learning and internet of things services. Those aren't really my wheelhouse (yet?), but seemed interesting nontheless. Succumbed to a food coma afterwards for a short nap.

Headed over to the Venetian to go back to the Expo for a new hoodie and for my next breakout session. AWS was holding the merchandise hostage if you didn't fill out evaluations for breakout sessions, so I couldn't get the hoodie until after I came back post-session. Good to know for next year. Back where the session halls were, got in the Reserved line for the Optimizing EC2 for Fun and Profit #bigsavings #newfeatures (CMP202) session. Talked with a gentleman while in line about the new announcements, specifically the S3 Select and Glacier Select features. I wasn't clear what the difference was between S3 Select and Athena and neither was he. I'll have to go try it out for myself.

AWS re:Invent 2017

Awaiting new feature announcements.

AWS re:Invent 2017

Great speaker as always. AWS always has good speakers.

AWS re:Invent 2017

More talk about Reserved and Spot Instances.

Best thing about this session was the announcements of new features. The first one was a really helpful feature AWS added to the Cost Explorer within the management console that gives instance recommendations based on your account's historical usage. Having a tool like this that does cost analysis and recommendations is great, means I don't have to. I pulled up the SDK and AWS CLI reference while he was demonstrating it, but couldn't find any methods where I could pull those recommendations using Lambda or a batch script. I figured it'd be useful to automate a monthly email or something that tells you that month's instance billing recommendations. Ended up talking to the speaker afterwards who said it's not available, but will be in the months to come. Nice!

Second announcement was regarding Spot Instances and being able to hibernate instances when they're going to be terminated. The way this was described was that hibernation acts the same way as when you "open and close your laptop". So if you are using a Spot Instance set to hibernate, when that instance gets terminated in the event another customer bids higher or AWS adds it back to the On-Demand instance pool, it will save state to EBS and when you receive it back, it will pick up where it left off instead of needing to completely re-initialize before doing whatever work you wanted. 

T2 Unlimited was also covered, which essentially allows you to not worry so much about requiring the credits for burst capacity of your T2 series of EC2 instances. The rest of the session covered a lot of cost optimization techniques that have been labored to death. Use Reserved Instances, use Spot Instances, choose the correct instance type for your workload, periodically check in to make sure you actually need the capacity you've provisioned, take advantage of serverless for cases where an always-on machine isn't necessary, and other tips of the "don't be an idiot" variety. Again, I must be biased since most of this information seems elementary. I think next year I need to stick to the 400-level courses to get the most value. That said, the presentation was excellent like always. I won't knock it just because I knew information coming in.

Found the shuttle a short walk from the hall, and decided to be lazy (smart) for once. Got back to the Bellagio for some poker before dinner, and came out plus $105. During all the walks back from the Aria to the Bellagio, I kept eyeballing the Gordon Ramsay Burger across the street at the Planet Hollywood, so I stopped in for dinner. 

AWS re:Invent 2017

Pretty flashy for a burger place.

AWS re:Invent 2017

I ate it all...No, I didn't. But wanted to try out the dogs and the burgers.

For a burger and hot dog place, I'd give it a 7/10. Probably would be a bit higher if they had dill pickles / relish, and honestly, better service. You can imagine this was pretty messy to eat, especially the hot dog, so I asked one of the girls upfront where the bathroom was to go wash my hands. The one across the hall was out of order (go figure), so I had to go to the one out thru some of the casino and next to P.F. Changs. I think the tables next to me thought I just walked out without paying. Heard them say "There he is." when I returned. Really? Do I look like a criminal? Yeah, I came to Vegas for a full week to rip off a burger joint.

Cheers!

3. December 2017 18:42
by Aaron Medacco
0 Comments

AWS re:Invent 2017 - Day 2 Experience

3. December 2017 18:42 by Aaron Medacco | 0 Comments

The following is my Day 2 re:Invent 2017 experience. Missed Day 1? Check it out here.

Day 2, I started off the day waking up around 11:30am. Needed the rest from the all-nighter drive Monday morning. Cleaning lady actually woke me up when she was making her rounds on the 4th floor. This meant that I missed the breakout session I reserved, Deploying Business Analytics at Enterprise Scale with Amazon QuickSight (ABD311). I'm currently in the process of recording a Pluralsight course on Amazon QuickSight, so I felt this information could be helpful as I wrap up that course. Guess I'll have to check it out later once the sessions are uploaded to YouTube. Just another reason to never drive through the night before re:Invent again.

After getting ready, I exposed my nerd skin to sunlight and walked over to the Venetian. This is where I'd be the majority of the day. I kind of lucked out because all of my sessions for the day besides the one I slept over were in the same hotel, and pretty back-to-back so I didn't have to get creative with my downtime. 

First session of the day was Deep Dive into the New Network Load Balancer (NET304). I was curious about this since the network load balancer's announcement recently, but never had a use case or a reason to go and implement one myself. 

AWS re:Invent 2017

I have to admit, I didn't know it could route to IP addresses.

AWS re:Invent 2017

Should have picked a better seat.

AWS re:Invent 2017

Putting it all together.

The takeaways I got was that the NLB is essentially your go-to option for TCP traffic at scale, but for web applications you'd still be mostly using the Application Load Balancer or the Classic Load Balancer. The 25% cheaper than ALB fact seems significant and it uses the same kinds of components used by ALB like targets, target groups, and listeners. Additionally, it supports routing to ECS, EC2, and external IPs, as well as allowing for static IPs per availability zone.

As I was walking out of the session, there was a massive line hugging the wall of the hall and around the corner for the next session which I had a reservation seat for (thank god). That session was Running Lean Architectures: How to Optimize for Cost Efficiency (ARC303). Nerds would have to squeeze in and cuddle for this one, this session was full. 

AWS re:Invent 2017

Before the madness.

AWS re:Invent 2017

Wasn't totally full, but pretty full.

AWS re:Invent 2017

People still filing in.

AWS re:Invent 2017

Obvious, but relevant slide.

I had some mixed feelings about this session, but thought it was overall solid. On one hand, much of the information was definitely important for AWS users to save money on their monthly bill, but at the same time, I felt a lot of it was fairly obvious to anyone using AWS. For instance, I have to imagine everybody knows they should be using Reserved Instances. I feel like any potential or current user of AWS would have read about pricing thoroughly before even considering moving to Amazon Web Services as a platform, but perhaps I'm biased. There were a fair number of managers in the session and at re:Invent in general, so maybe they're not aware of obvious ways to save money. 

Aside from covering Spot and Reserved Instance use cases, there was some time covering Convertable Reserved Instances, which is still fairly new. I did enjoy the tips and tricks given on reducing Lambda costs by looking for ways to cut down on function idle time, using Step Functions instead of manual sleep calls, and migrating smaller applications into ECS instead of running each application on their own instance. The Lambda example highlighted that many customer functions involve several calls to APIs which occur sequentially where each call waits on the prior to finish. This can rack up billed run time even though Lambda isn't actually performing work during those waiting periods. The trick they suggested was essentially to shotgun the requests all at once, instead of one-by-one, but as I thought about it, that would only work if those API calls were such that they didn't depend on the result of the prior. When they brought up Step Functions it was kind of obvious you could just use that service if that was the case, though.

The presenters did a great job, and kept highlighting the need to move to a "cattle" mentality instead of "pet" mentality when thinking about your cloud infrastructure. Essentially, they encouraged moving away from manual pushing, RDPing, naming instances thinks like "Smeagle" and the like. Honestly, a lot of no-brainers and elementary information but still a  good session.

Had some downtime after to go get something to eat. Grabbed the Baked Rigatoni from the Grand Lux Cafe in the Venetian. The woman who directed me to it probably thought I was crazy. I was in a bit of a rush, and basically attacked her with, "OMG, where is food!? Help me!"

AWS re:Invent 2017

Overall, 7/10. Wasn't very expensive and now that I think about it, my first actual meal (some cold pizza slices don't count) since getting to Vegas. 

Next up was ElastiCache Deep Dive: Best Practices and Usage Patterns (DAT305) inside the Venetian Theatre. I was excited about this session since I haven't done much of anything with ElastiCache in practice, but I know some projects running on AWS that leverage it heavily. 

AWS re:Invent 2017

About 10 minutes before getting mind-pwned.

AWS re:Invent 2017

Random guy playing some Hearthstone while waiting for the session to begin.

AWS re:Invent 2017

Sick seat.

Definitely felt a little bit out of my depth on this one. I'm not someone who is familiar with Redis, outside of just knowing what it does so I was clueless during some of the session. I didn't take notes, but I recall a lot of talk about re-sharding clusters, the old backup and recovery method vs. the new online managed re-sharding available, pros of enabling cluster mode (was clueless, made sense at the time, but couldn't explain it to someone else), security, and best practices. My favorite part of the session involved use cases and how ElastiCache can benefit: IoT, gaming apps, chat apps, rate limiting services, big data, geolocation or recommendation apps, and Ad Tech. Again, I was out of my depth, but I'll be taking a closer look at this service during 2018 to fix that.

After some more Frogger in the hallways, headed over to the Expo, grabbed some swag and walked around all the vendor booths. Food and drinks were provided by AWS and the place was a lot bigger than I expected. There's a similar event occurring at the Aria (the Quad), which I'll check out later in the week. 

AWS re:Invent 2017

Wall in front of the Expo by Registration.

There were AWS experts and team members involved with just about everything AWS scattered around to answer attendee questions which I though was freaking awesome. Valuable face-time with the actual people that work and know about the stuff being developed.

AWS re:Invent 2017

Favorite section of the Venetian Expo.

AWS re:Invent 2017

More madness.

Talked to the guys at the new PrivateLink booth to ask if QuickSight was getting an "endpoint" or a method for connecting private AWS databases soon. Ended up getting the de facto answer from the Analytics booth who had Quinton Alsbury at it. Once I saw him there, I'm like "Oh here we go, this guy is the guy!" Apparently, the feature's been recently put in public preview, which I somehow missed. Visited a few other AWS booths like Trusted Advisor and the Partner Network one, and then walked around all the vendor booths for a bit.

AWS re:Invent 2017

Unfortunately, I didn't have much time to chit chat with a lot of them since the Expo was closing soon. I'll have to do so more at the Quad. Walked over to the interactive search map they had towards the sides of the room to look for a certain company I thought might be there. Sure enough, found something familiar:

AWS re:Invent 2017

A wild technology learning company appears.

Spoke with Dan Anderegg, whose the Curriculum Manager for AWS within Pluralsight. After some talk about Pluralsight path development, I finished my beer and got out, only to find I actually stayed too long and was already super late to my final session for the day, Deep Dive on Amazon Elastic Block Store (Amazon EBS) (STG306). Did I mention how it's hard to try to do everything you want to at re:Invent? 

Ended up walking home and wanting to just chill out, which is how this post is getting done. 

Cheers!

1. December 2017 17:26
by Aaron Medacco
0 Comments

AWS re:Invent 2017 - Day 1 Experience

1. December 2017 17:26 by Aaron Medacco | 0 Comments

Welcome to the first in a series of blog posts detailing my experience at AWS re:Invent 2017. If you're someone who is considering going to an AWS re:Invent conference, hopefully what follows will give you a flavor for what you can expect should you choose to fork over the cash for a ticket. The following content contains my personal impressions and experience, and may not (probably doesn't?) reflect the typical experience. Also, there will be some non-AWS fluff as well as I have not been to Las Vegas before.

AWS re:Invent 2017

My adventure starts at about Midnight. Yes, midnight. Living in Scottsdale, AZ, I figured, "Why not just drive instead of fly? After all, it's only a 6 hour drive and there won't be any traffic in the middle of the night." While that was true, what a mistake in retrospect. Arriving in Las Vegas with hardly any sleep after the road trip left me in pretty ragged shape for Monday's events. Next year, I'll definitely be flying and will get there on Sunday so I get can settle prior to Monday. I actually arrived so early, I couldn't check into my room and needed to burn some time. What better activity to do when exhausted than sit down at poker tables. Lost a quick $900 in short order. Hahaha! Truth be told, I got "coolered" back to back, but I probably played bad, too.

Once I got checked into my room at the Bellagio around 9:00am, I headed back to the Aria to get registered and pick up my re:Invent hoodie. Unfortunately, they didn't have my size, only had up to a Small. I couldn't help but smile about that. I ended up going to the Venetian later to exchange my Small for a Medium. Anyways, got my badge, ready to go! Or was I?

By the way, kudos to the Bellagio for putting these in every room. Forgot my phone charger. Well, the correct phone charger at least...

 AWS re:Invent 2017

...except it didn't have a charger compatible with my Samsung Galaxy S8. Kind of funny, but I wasn't laughing. Alright, maybe a little. Would end up getting one at a Phone store among one of the malls at the Strip. Oh yeah, and I also forgot to buy a memory card for my video recorder prior to leaving. Picked up one of those from a Best Buy Express vending machine. Vegas knows.

By this time I was crashing. Came back to my room, fell asleep, and missed 2 breakout sessions I was reserved for. Great job, Aaron! Off to a great start! 

Walked to the Aria to go check out the Certification Lounge. They had tables set up, food and drink, and some goodies available depending on what certifications you'd achieved. The registration badges have indicators on them that tell people if you're AWS certified or not, which they use to allow or deny access. I didn't end up staying too long, but there were a decent number of attendees with laptops open working and networking. Here's some of the things collected this year by walking around to the events: 

AWS re:Invent 2017

The re:Invent hoodie picked up at Registration (left) and the certification t-shirt inside the Certification Lounge (right).

AWS re:Invent 2017

Water bottle and AWS pins were given away at the Venetian Expo (top-left), badge and info packet at Registration (right), and the certification stickers at the Certification Lounge depending on which ones you've completed (bottom-left).

Headed over to the MGM Grand for my first breakout session, GPS: Anti Patterns: Learning From Failure (GPSTEC302). Before I discuss the session, I have to talk about something I severely underestimated about re:Invent. Walking! My body was definitely NOT ready. And I'm not an out-of-shape or big guy, either. The walking is legit! I remember tweeting about what I imagined would be my schedule weeks before re:Invent and Eric Hammond telling me I was being pretty optimistic about what I would actually be able to attend. No joke. Okay, enough of my complaining.

AWS re:Invent 2017

Waiting for things to get started.

AWS re:Invent 2017

Session about half-full. Plenty of room to get comfortable.

AWS re:Invent 2017

Presenter's shirt says, "got root?". Explaining methods for ensuring account resource compliance and using AWS account best practices when it comes to logging, backups, and fast reaction to nefarious changes.

This was an excellent session. The presenters were fantastic and poked fun at mistakes they themselves have made or those of customers they've talked to have made regarding automation (or lack thereof), compliance, and just overall bone-headedness (is that a word?). The big takeaways I found were to consider using services like CloudWatch, CloudTrail and Config to monitor and log activity in your AWS accounts to become aware when stupid raises it's ugly head. They threw out questions like, "What would happen if the root account's credentials were compromised and you didn't know about it until it was too late?", and "You have an automated process for creating backups, but do you actually test those backups?". From this came suggestions to regularly store and test backups to another account in case an account gets compromised and using things like MFA, especially for root and privileged users.

Additionally, the presenters made a good argument for not using the management console for activities once you become more familiar with AWS, particularly if you're leveraging the automation tools AWS provides like OpsWorks and CloudFormation as that kind of manual mucking around via the console can leave you in funny states for stacks deployed with those services. Along those lines, they also suggested dividing up the different tiers of your application infrastructure into their own stacks so that when you need to make changes to something or scale, you don't end up changing the whole system. Instead, you only modify or scale the relevant stack. Overall good session. If they have it again next year, I would recommend it. You'll get some laughs, if nothing else. The guys were pretty funny.

Once out, I had a meeting scheduled to talk with a company (presumably about upcoming Pluralsight work) at the Global Partner Summit Welcome Reception. Now, I'll admit I got a little frustrated trying to find where the **** this was taking place! AWS did a great job sending lots of guides with re:Invent flags everywhere to answer questions and direct attendees to their events, and these guys were godsends every time except when it came to finding this event. I think I just got unlucky with a few that were misinformed.

AWS re:Invent 2017

These guys were scattered all over the strip and inside the hotels. Very helpful!

First, I was told to go to the one of the ballrooms. Found what appeared to be some kind of Presenter's Registration there. Then, found another guide who said to go to the Garden Grand Arena. Walked over there, total graveyard, and ironically, a random dude there who wasn't even one of the re:Invent guides told me where it actually was. He also said, "Oh yeah, and unless you want to be standing in line all night, you might want to reconsider." It was late enough at this point, I figured I'd just head back to the Bellagio for a much needed poker session, so that's what I did. However, on the way back, holy ****, he was right. I've never seen a line as long as the one to get into the GPS Welcome Reception in my life. It went from the food court, through the entire casino, out of the casino, and further back to I couldn't tell where. Apparently, I was the only one who missed the memo, since everyone else knew where to go, but still, that line. 

Long hike back to the Bellagio, played poker for about 3 hours, lost $200 (man, I suck), and on my way back to my room discovered I didn't eat anything all day. LOL! Picked up a couple pizza slices and crashed for the night. A good night's sleep? Yes, please. Tomorrow would be better.

Cheers!

22. July 2017 19:05
by Aaron Medacco
0 Comments

Scheduling Notifications for Rotating Old IAM Access Keys

22. July 2017 19:05 by Aaron Medacco | 0 Comments

Access keys allow you to give programmatic access to a user so they can accomplish tasks and interact with services within your AWS environment. These keys should be heavily guarded and kept secret. Once exposed, anyone can wreak havoc in your AWS account using any permissions that were granted to the user's whose keys were exposed.

A best practice for security is to rotate these keys regularly. We want to keep our access keys fresh and not have old or unused keys running about waiting to be abused. Therefore, I've created an automated method for notifying an administrator when user access keys are old and should be rotated. Since the number of days for keys to be considered "old" varies across organizations, I've included a variable that can be configured to fit the reader's requirements.

IAM Access Key Rotation

Like my recent posts, this will make use of the AWS CLI, which I assume you've installed and configured already. We'll be using (of course) Lambda backed by a CloudWatch event rule. Notifications will be sent using the SNS service when keys should be rotated. Enjoy.

Creating an SNS topic:

  1. Open up a command prompt or terminal window, and invoke the following command:
    aws sns create-topic --name IAM-Access-Key-Rotation-Topic
  2. You'll get a Topic ARN value back which you want to keep.
  3. Then invoke the following command. Substitute Email for the email address you want to receive the notifications.
    Note: You don't have to use email if you don't want to. Feel free to use whichever protocol/endpoint fits you.
    aws sns subscribe --topic-arn ARN --protocol email --notification-endpoint Email
  4. The email will get an email asking them to confirm a subscription. You'll need to confirm the subscription before moving on if you want notifications to go out.

Creating an IAM policy for access permissions:

  1. Create a file named iam-policy.json with the following contents and save it in your working directory. Substitute Your Topic ARN for the ARN of the topic you just created:
    {
        "Version": "2012-10-17",
        "Statement": [
            {
                "Effect": "Allow",
                "Action": [
                    "sns:Publish"
                ],
                "Resource": [
                    "Your Topic ARN"
                ]
            },
            {
                "Effect": "Allow",
                "Action": [
                    "iam:ListAccessKeys",
                    "iam:ListUsers"
                ],
                "Resource": [
                    "*"
                ]
            }
        ]
    }
  2. In your command prompt or terminal window, execute the following command:
    aws iam create-policy --policy-name rotate-old-access-keys-notification-policy --policy-document file://iam-policy.json
  3. You'll receive details for the policy you just created. Write down the ARN value. You will need it in a later step.

Creating the IAM role for the Lambda function:

  1. Create a file named role-trust-policy.json with the following contents and save it in your working directory:
    {
      "Version": "2012-10-17",
      "Statement": [
        {
          "Effect": "Allow",
          "Principal": {
            "Service": "lambda.amazonaws.com"
          },
          "Action": "sts:AssumeRole"
        }
      ]
    }
  2. In your command prompt or terminal window, invoke the following command:
    aws iam create-role --role-name rotate-old-access-keys-notification-role --assume-role-policy-document file://role-trust-policy.json
  3. You'll get back information about the role you just made. Write down the ARN value for the role. You will need it when we go to create the Lambda function.
  4. Invoke this command to attach the policy to the role. Substitute ARN for the policy ARN you received when you created the IAM policy.
    aws iam attach-role-policy --policy-arn ARN --role-name rotate-old-access-keys-notification-role

Creating the Lambda function:

  1. Create a files named index.js with the following contents and save it in your working directory. Make sure you substitute Topic ARN with the ARN of the topic you created in the first step.
    Note: Notice you can configure the number of days allowed before rotations should be done. Default value I set is 90.
    var AWS = require("aws-sdk");
    var iam = new AWS.IAM();
    var sns = new AWS.SNS();
    
    var daysBeforeRotationRequired = 90; // Number of days from access creation before action is taken on access keys.
    
    exports.handler = (event, context, callback) => {
        var listUsersParams = {};
        iam.listUsers(listUsersParams, function(err, data) {
            if (err) {
                console.log(err, err.stack);
            }
            else {
                for (var i = 0; i < data.Users.length; i++) {
                    var userName = data.Users[i].UserName;
                    var listAccessKeysParams = { UserName: userName };
                    iam.listAccessKeys(listAccessKeysParams, function(err, data) {
                        if (err) {
                            console.log(err, err.stack);
                        } 
                        else {
                            for (var j = 0; j < data.AccessKeyMetadata.length; j++) {
                                var accessKeyId = data.AccessKeyMetadata[j].AccessKeyId;
                                var creationDate = data.AccessKeyMetadata[j].CreateDate;
                                var accessKeyUserName = data.AccessKeyMetadata[j].UserName;
                                var now = new Date();
                                var whenRotationShouldOccur = dateAdd(creationDate, "day", daysBeforeRotationRequired);
                                if (now > whenRotationShouldOccur) {
                                    var message = "You need to rotate access key: " + accessKeyId + " for user: " + accessKeyUserName
                                    console.log(message);
                                    var publishParams = {
                                        Message: message,
                                        Subject: "Access Keys Need Rotating For User: " + accessKeyUserName,
                                        TopicArn: "Topic ARN"
                                    };
                                    sns.publish(publishParams, context.done);
                                }
                                else {
                                    console.log("No access key rotation necessary for: " + accessKeyId + "for user: " + accessKeyUserName);
                                }
                            }
                        }
                    });
                }
            }
        });
    };
    
    var dateAdd = function(date, interval, units) {
        var ret = new Date(date); // don't change original date
        switch(interval.toLowerCase()) {
            case 'year'   :  ret.setFullYear(ret.getFullYear() + units);  break;
            case 'quarter':  ret.setMonth(ret.getMonth() + 3*units);  break;
            case 'month'  :  ret.setMonth(ret.getMonth() + units);  break;
            case 'week'   :  ret.setDate(ret.getDate() + 7*units);  break;
            case 'day'    :  ret.setDate(ret.getDate() + units);  break;
            case 'hour'   :  ret.setTime(ret.getTime() + units*3600000);  break;
            case 'minute' :  ret.setTime(ret.getTime() + units*60000);  break;
            case 'second' :  ret.setTime(ret.getTime() + units*1000);  break;
            default       :  ret = undefined;  break;
        }
        return ret;
    }
  2. Zip this file to a zip called index.zip.
  3. Bring your command prompt or terminal window back up, and execute the following command. Substitute ARN for the role ARN you received from the step where we created the role:
    aws lambda create-function --function-name rotate-old-access-keys-notification-sender --runtime nodejs6.10 --handler index.handler --role ARN --zip-file fileb://index.zip --timeout 30
  4. Once the function is created, you'll get back details for the function. Write down the ARN value for the function for when we schedule the function execution.

Scheduling the Lambda function:

  1.  In your command prompt or terminal window, execute the following command:
    Note: Feel free to adjust the schedule expression for your own purposes.
    aws events put-rule --name Rotate-Old-Access-Keys-Notification-Scheduler --schedule-expression "rate(1 day)"
  2. Write down the ARN value for the rule upon creation finishing.
  3. Run the following command. Substitute ARN for the Rule ARN you just received:
    aws lambda add-permission --function-name rotate-old-access-keys-notification-sender --statement-id LambdaPermission --action "lambda:InvokeFunction" --principal events.amazonaws.com --source-arn ARN
  4. Now run the following command. Substitute ARN for that of the Lambda function you created in the previous step:
    aws events put-targets --rule Rotate-Old-Access-Keys-Notification-Scheduler --targets "Id"="1","Arn"="ARN"

For information regarding ways to rotate your access keys without affecting your applications, click here.

Cheers!

Copyright © 2016-2017 Aaron Medacco