Forem: Ben Ellis

My Lambda concurrency applied quota is only 10? But why???

Ben Ellis — Fri, 10 Feb 2023 19:47:16 +0000

I'm using AWS Control Tower and AWS Organizations to implement a multi-account setup for a client of mine. I needed to analyze the VPC Flow Logs, so I used the console to setup the VPC Flow Logs Athena integration. It uses a CloudFormation template that includes two Lambdas. I ran into the following error while deploying the stack:

Resource handler returned message: "Specified ReservedConcurrentExecutions for function decreases account's UnreservedConcurrentExecution below its minimum value of [100].

I investigated and the two Lambdas each had a ReservedConcurrentExecutions value of 1. I was stumped at first because the default Lambda concurrent executions quota for an account and region is 1000. Well, I was wrongif you're using AWS Control Tower Account Factory to create the accounts. In my case, the applied limit was set at 10. 10!?!?

Since Lambda requires a minimum of 100 unreserved concurrency capacity for functions that don't use an explicitly reserved concurrency, my account was way underprovisioned. The 10 I had available had to be used for unreserved capacity only. There was no room for any reserved capacity, as evidenced by the CloudFormation error I received. I needed at least 102 in order to deploy this solution.

I checked another client account of mine that uses AWS Control Tower, and it allowed me a whopping 50 concurrency. I found documentation on the subject to explain why these accounts were not assigned 1000 quota to begin with. It's called out at the top of the page:

New AWS accounts have reduced concurrency and memory quotas. AWS raises these quotas automatically based on your usage.

That makes sense. New users can shoot themselves in the foot with a recursive Lambda bug that could cost them a LOT of $$$ if it runs unnoticed at a concurrency of 1000. Or maybe there are concerns of crypto miners hijacking AWS Orgs. But 10 for starters is a bit low. It's not clear how long it takes to see this increase from 10 to the default quota value of 1000.

The Solution

The immediate fix is straightforward. Go to the Quotas page in the AWS Console and request an increase. For each region. In each account. Kind of a pain. There is a feature that allows you to setup a template for your organization to automatically request quota increases any time a new account is provisioned via AWS Control Tower/AWS Organizations. Pretty handy. You can access it here.

NOTE: You must set your region to us-east-1 (N. Virginia) in the AWS Console.

It oddly assumes that you already have 1000 concurrency as your limit, so you have to ask for at least 1001. In my case, I would have been happy going from 10 to 200 for now.

Another odd layer of this experience is that my first quote increase request wasn't met with a quick, "of course we can" kind of response. It apparently had to be escalated, and it took well over a day to resolve. This was the response I received initially:

I'm following up to notify you that we've received your AWS Lambda Concurrent executions quota increase request for 200 in the Ohio region, and Im happy to submit the request for you. For a quota increase of this size, I will need to collaborate with our Service Team to get approval. Please note that it can take some time for the Service Team to review your request. This is to ensure that we can meet your needs while keeping existing infrastructure safe.

I made a separate quota increase for a different region in the same account, requesting 500 concurrency this time, and it was quickly approved. I'm not sure what the difference was.

Resources

Deploying a Custom WAF v2 Rule with the AWS CDK

Ben Ellis — Tue, 24 Jan 2023 04:49:16 +0000

I'm working with a client to migrate their infrastructure to AWS and apply some modest modernizations during phase 1 of the project. During the migration process, we need to keep their new public endpoints in AWS private during the testing phase. We have a few CloudFront distributions deployed in front of ALBs. We're using the AWS CDK to script out their infrastructure. Unfortunately, WAF v2 does not have any L2 constructs for the CDK, so you have to resort to using Cfn*-style L1 constructs.

My use case is pretty straightforward: block all traffic unless it comes from a source IP listed in a WAF IP Set. The WAF CDK code looks like this:

import { CfnOutput } from "aws-cdk-lib"
import { CfnIPSet, CfnWebACL } from "aws-cdk-lib/aws-wafv2"

const migrationTestingIpSet = new CfnIPSet(this, 'MigrationTestingIpSet', {
  addresses: [
    "1.1.1.1/32", // whitelisted IPs in CIDR format
  ],
  ipAddressVersion: 'IPV4',
  scope: 'CLOUDFRONT',
  description: 'List of staff allowed to test migration endpoints',
})

const migrationWaf = new CfnWebACL(this, 'MigrationWaf', {
  defaultAction: {
    block: { }, // Block all traffic except the IP set
  },
  scope: "CLOUDFRONT",
  description: "Allows staff to test migration endpoints before cutover",
  visibilityConfig: {
    cloudWatchMetricsEnabled: true,
    metricName: 'DefaultBlocks',
    sampledRequestsEnabled: true,
  },
  rules: [{
    name: "AllowStaffIps",
    priority: 1,
    statement: {
      ipSetReferenceStatement: {
        arn: migrationTestingIpSet.attrArn,
      }
    },
    visibilityConfig: {
      cloudWatchMetricsEnabled: true,
      metricName: 'AllowsToStaffIpSet',
      sampledRequestsEnabled: true,
    },
    action: {
      allow: {}
    }
  }],
})

new CfnOutput(this, "MigrationWafArnOutput", {
  description: "Migration WAF Arn",
  value: migrationWaf.attrArn,
})

You'll see that there is a default action that blocks all traffic. Then a rule is defined that allows traffic based on the IP Set.

Next, connect your WAF to your CloudFront distribution:

import { Distribution } from "aws-cdk-lib/aws-cloudfront"

const prodAppCdn = new Distribution(this, "ProdAppCdn", {
  defaultBehavior: { /* Default Behavior here... */ },
  /* other props here */
  webAclId: "arn:aws:wafv2:us-east-1:1234567890:global/webacl/WafNameHere/WafAclGuidHere",
})

NOTE: CloudFormation requires deploying WAFs to us-east-1 if you're integrating it with CloudFront. In my case, I created a separate CDK stack since I was deploying everything else to us-east-2.

Due to the cross-reference nature of deploying to us-east-2 vs us-east-1, I included the string value for the WAF ARN instead of referencing it from the WAF stack. The WAF page in the AWS Console doesn't currently show the ARN for the ACL, but the first code snippet I shared returns it as a CfnOutput value to make it easier to copy/paste. Otherwise, you'll have to build the WAF ACL Id using a format similar to the above, filling in your account number, WAF name, and WAF Id appropriately. The WAF ACL Id is a GUID that you can find in the URL of the Console address when you visit the WAF details screen.

Helpful References

https://docs.aws.amazon.com/cdk/api/v2/docs/aws-cdk-lib.aws_wafv2-readme.html - Explains L1 constraint

https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/AWS_WAFv2.html - CloudFormation documentation in case it's helpful

https://docs.aws.amazon.com/cdk/api/v2/docs/aws-cdk-lib.aws_wafv2.CfnIPSet.html - How to create the IP Set

https://docs.aws.amazon.com/cdk/api/v2/docs/aws-cdk-lib.aws_wafv2.CfnWebACL.html - How to create the WAF ACL

https://stackoverflow.com/questions/72605879/making-the-waf-rule-by-cdk - SO post with some helpful tidbits

How to Add a Self-Referencing Security Group Rule via the AWS CDK

Ben Ellis — Sat, 21 Jan 2023 22:29:58 +0000

If you look at the default VPC in your AWS account, you'll notice there's also a default Security Group. It includes an inbound rule that allows all traffic from itself, a self-referencing rule. This means that any resources with this Security Group attached would be able to communicate with each other over the allowed protocols and ports.

It wasn't immediately obvious to me how to accomplish this via the CDK. It turns out, it's pretty simple. The first argument of the addIngressRule() method on the SecurityGroup takes an IPeer. SecurityGroup implements IPeer, so it is as simple as passing itself in as the first argument.

const selfRefSg = new SecurityGroup(this, "SelfRefSg", {
  vpc,
  allowAllOutbound: true,
})

selfRefSg.addIngressRule(
  selfRefSg,
  Port.allTraffic(),
  "allow local VPC traffic",
)

After running a cdk deploy:

For a full example, check out the GitHub repo I put together.

Where did my MGN source server go?

Ben Ellis — Sat, 21 Jan 2023 03:45:07 +0000

In one of my current consulting projects, I'm helping a company move 15 Ubuntu VMware servers from a traditional private virtual cloud provider to AWS EC2 using the AWS Application Migration Service (MGN). To replicate a server with MGN, you install a lightweight replication agent on each server. The agent connects to the MGN service and then begins replicating the server's volumes at the block level.

The first six server migrations were uneventful. Then I added a new server (we'll call it "Server7") and noticed that one of the first six servers disappeared (we'll call this one "Server6"). "That's odd," I thought. I tried reinstalling the replication agent on Server6. Wah-lah! It showed up in the list of source servers again, but then... Server7 disappeared. The same thing happened with two more servers I tried replicating. I was playing a tedious game of "Whac-A-Mole."

It became clear that AWS MGN somehow determined these two servers were the same server. At first, it wasn't obvious. Each server had a different hostname, different private IP address(es), different everything on the surface.

I also noticed that whichever server was active as a source server in the MGN source server list would never finish replicating and reach a "Healthy" status. Instead, it would stall out and continually perform time-consuming rescans. This went on for a few days.

I opened a support ticket with AWS but was not making much progress on the root issue. I decided to dig into the install directory for the AWS Migration Agent under /var/lib/aws-replication-agent. I SSHed into Server6 and found an agent.config file that contained JSON representing the current configuration. I noted the sourceServerId value as well as the installationIdentifierValue value. These map to the "AWS ID" and "VMware virtual machine identifier" fields in the MGN Console UI, respectively.

{
  // agent.config - other fields removed for brevity
  "sourceServerId": "s-481fd58d273c20648",
  "installationIdentifierType": "metadataInstanceID",
  "installationIdentifierValue": "564d2ad7-f1fa-9a0c-f866-78e246fa533c"
}

I then SSHed into Server7 and compared these values in the agent.config. They matched Server6. For source servers running on VMware, AWS uses the VMware UUID of the VM as the unique identifier. I confirmed the UUID of both servers by running sudo dmidecode | less and looking for the VMware section. Both servers had the same UUID:

Serial Number: VMware-56 4d 2a d7 f1 fa 9a 0c-f8 66 78 e2 46 fa 53 3c
UUID: 564D2AD7-F1FA-9A0C-F866-78E246FA533C

Apparently, when some of the server VMs were originally created, a new UUID was not assigned to them on creation. This had not caused any obvious problems yet, but it was a showstopper when using MGN.

The solution was to update the server UUIDs to a new, unique value. You may find the following articles helpful:

https://kb.vmware.com/s/article/1541

https://kb.vmware.com/s/article/1002403 (specific to Windows VMs)

https://uncomplicatingit.com/vmware/how-to-manually-change-the-uuid-of-a-virtual-machine-in-vmware/

I'll update this article with detailed steps once we work through the process of changing the UUIDs through our current VPC provider.

A quick explanation regarding the rescans I was seeing... the replication issues were a result of the two source servers sending replication data through the replication agent to a single source server record in MGN. AWS saw bytes coming from two different agents for the same server volume. As you'd expect, this does not work well. 🤣

Tips for Passing AWS Certification Exams Like a Boss

Ben Ellis — Thu, 05 Jan 2023 15:00:00 +0000

I recently passed my 5th AWS certification exam, the AWS Security Specialty (SCS). I started on the journey to obtain a substantial collection of AWS certifications back in May 2021. I'm a freelance cloud developer and consultant, focused on building scalable, secure, and performant solutions on AWS. I'll share my experience and tips to help you streamline your preparation time and feel ready for exam day.

Background

I've been a software engineer for over 20 years, primarily focused on the Microsoft dev stack consisting of Windows, .Net, and SQL Server. I had experience with running App Service deployments and SQL Azure on the Microsoft Azure cloud. (Aside: By the way, it's A-zure, not a-ZURE). I had limited experience with AWS as of early 2021, mostly confined to EC2, RDS, S3, and Route 53.

I was introduced to AWS certifications when my friend and neighbor Adam Elmore told me about his journey to pass all 12 AWS exams in six weeks, which is quite the feat. He gave a talk at our local AWS meetup, and he convinced me that taking the AWS certifications was a great opportunity to expand my career that I couldn't pass up.

Now, my certification journey has been different from his since I'm a mere mortal and he's a test-taking Asgardian. I knew my comparatively limited AWS experience was going to stretch out the timeline for me. Adam came from a background of building a startup on AWS and consequently went into his certification challenge with lots of AWS experience. He also gets a little giddy about taking certification tests. That's not me. At all. I think I'm in the majority on this.

Why Take the Exams

I was happy to graduate from college with only a Bachelor's in Computer Science and then never look back. Once I started working in the industry, I saw that I wouldn't need to pursue a Master's or Doctorate to continue to have great opportunities—and I was happy about that. I was more than ready to be done with schooling. Don't get me wrong—I love learning. That's why I love my career so much. There's always something new to learn that keeps it fresh and exciting.

Every now and then throughout my career I would consider pursuing Microsoft MCSD certifications, but I never got far with it. Something about formalizing the learning process made it less fun for me. But here I am, taking on a whole line of exams based on the recommendation of a certification savant. I don't regret it at all. It has already opened up new doors for me, and I have enjoyed learning loads about the AWS ecosystem. Part of my journey has included pivoting into a different tech stack for some projects (a la Typescript and NodeJS).

To expand a little more, my reasons for taking the exams are four-fold:

1. Create Opportunities

After talking with Adam, I saw the huge potential that having the certifications and the knowledge gained while preparing for the certifications would have on my career. The Cloud is not going away. And these are still the early days of the Cloud. Many companies still host in traditional data centers using traditional approaches. Much opportunity remains to help organizations migrate to the cloud and modernize on cloud-native services. The Cloud is the default starting point for most startups, but even new startups aren't taking advantage of many of the best features of the Cloud like serverless and fully-managed services. For me, this feels like almost limitless opportunities for action-packed consulting and development.

2. Build Credibility

The AWS exams are well-respected in the industry (or so I hear). They also carry a lot of weight in the AWS IQ program that Amazon offers. IQ is one of the best-kept secrets of the certification program. If you pass an associate-level or higher exam, you can apply for this program. AWS IQ allows AWS customers to submit requests for help that AWS IQ Experts (i.e., those of us with one or more certs under our belt) can respond to. You submit a proposal with a cost. If the customer accepts, you do the work and AWS handles the payments. AWS also provides a review system for customers to provide feedback on the quality of your work. The AWS IQ program is what convinced me to make the jump back into freelance work a few months ago. I'll write a separate article on this at a later date.

3. Force Me to Learn AWS In Depth

I feel that the certifications have forced me to learn important best practices and facts that would have taken me a long time to stumble upon in my daily work. Some would argue about the helpfulness of certifications for a practitioner, but I have seen their value almost immediately as a relative newcomer to AWS. Even some of the cramming of seemingly useless facts like service limits and CLI commands have come into play during my consulting work.

4. Challenge Me

As mentioned, I haven't pursued any industry certifications before. I wanted to take on the challenge and Do Hard Things™ for the sake of the satisfaction that comes with the accomplishment. Watching tens of hours of videos, reading/skimming dozens of whitepapers, writing down tens of pages of notes, taking practice tests, and then taking a 3-hour exam—it's a challenge.

How to Begin

You may be thinking... "You convinced me. Now what?"

If you're up for the adventure and don't know where to start, there are a couple of ways that you can approach the exams.

If your goal is to gain five or more and become a well-rounded pro in AWS, you can take the three associate exams first followed by the two professionals. You can add on some specialty exams from there if you're not tired yet. Adrian Cantrill has a helpful video explaining this approach.

If you want all 12, you can follow Adam's plan. That's the approach I'm currently using. I likely won't go for all 12. I'll end up skipping a few of the specialty certs that are less pertinent to my goals.

You could also just take one or two, either to learn if certs are for you or to certify yourself in a domain specific to your existing expertise. If you just want a cert on your resume, go for the Solutions Architect Associate. If you're a data scientist, consider the Data Analytics Specialty.

What About the Cloud Practitioner exam? The Cloud Practitioner is an easier 101-level exam. You can either take it first to warm up to the exam process with less pressure, or you can start with an associate and then circle back at the end and take this exam for the sake of completeness without having to prep much for it. Or just skip it!

How to Prepare

There is no "right" way to prepare for an exam. I'm going to share what works well for me. I'm a visual learner/recaller for the most part, so keep that in mind.

Review the Official Exam Guide

Your first stop should be the official exam guide from the AWS Certification site. The guide covers details on how the exam is scored. It provides a content outline that gives a breakdown of the domains that are covered and what percentage of the exam each domain typically covers. This helps you see which areas to focus the most energy on. No domain should be neglected though. It also tells you what services to expect to see covered in the exam.

Start Taking Lots of Notes

The notes you take along the way will help you when the time comes to cram. I use OneNote for my second brain. This is where all my exam notes go. Find a note-taking tool that works for you and don't be shy with your notes. You can refine them later. By the time I'm done prepping, I can easily have 30-40 pages of notes.

Watch Video Training

My first technical preparation phase is to watch a video-based exam course. The "best" video training course varies by the exam topic, but I'll share some of the top providers I'm aware of. Expect to spend anywhere from 10 to 25 hours watching and taking notes from a course (or less if you use my tip below about increasing the playback speed).

A Cloud Guru

acloudguru.com

I used A Cloud Guru as the primary video course for all of my exams so far. I feel they do a good job preparing you for the exam, but they aren't as focused on helping you learn the material for real-world usage. Regardless, you'll also want to follow some of the other tips in this section to supplement your learning.

Adrian Cantrill

learn.cantrill.io

I wish I would have found Adrian sooner. I haven't taken any of his courses yet, but I've heard great things, and I've sampled a lot of his free content on YouTube. He's known for teaching the content not just for the purpose of passing the exam but for real-world usage. As a consultant, this is very important. My clients care more about whether I can deliver and less about how many certs I have.

When I go to take my two pro certs, I plan to use Adrian as my primary training for both.

Stephane Maarek

Stephane provides top notch content through Udemy. I used some of his video courses in addition to A Cloud Guru.

Pro Tip: My local library offers free access to many Udemy courses. Stephane's courses are typically included. Check with your local library to see if access is included with your membership.

Read Guides, FAQs, Whitepapers, Blogs, and More

The exam homepage on the certification site will list numerous whitepapers that you should review. Skimming is sufficient for many, but more in-depth reading is warranted for the services that are heavily featured. And if one of your goals for taking the exam is to truly learn these services at an expert level, reading through the User Guides can be very helpful.

A note on the User Guides: they are intended as reference materials for inbound traffic from Google on various topics. If you are reading it front to back, you'll notice a lot of repetitive explanations. Typically, they vary slightly based on the context and may contain small bits of new information. So keep that in mind because you can switch from reading to skimming and back. I found some of the repetition helped hammer home concepts I was less familiar with.

Pro Tip: Send whitepapers and guides to your e-reader. I used the "Send to Kindle" plugin to send articles to my iPad. I prefer the iPad because many articles are PDFs, so they are formatted for a fixed printing dimension (in the U.S., typically 8.5x11 inches). Kindle also lets you highlight sections. I used this feature and then looped back later to transfer important highlights into my notes.

YouTube Videos

I watch a lot of supplemental videos on YouTube to fill in knowledge gaps and delve deeper into topics that I feel may have only been hit on a superficial level. This has proven very beneficial.

Hands On

Nothing beats real-world experience. Being a visual person, it helps to have seen various screens from the AWS Console when I'm trying to answer an exam question that nitpicks at the slight differences between two very similar services. Consider finding work you can do for your company or clients that will give you practice even if it means working extra time at your day job or giving a client a price break or doing it pro bono.

Review Your Notes

You've been following my advice and taking lots of notes, right? Before you start taking practice tests (see next), you'll want to start reviewing your notes. There will likely be details like service limits and quotas that you'll need to commit to memory. If you print out your notes like I do, you can start highlighting the areas you need to spend more time on.

Take Lots of Practice Tests

Taking practice tests is a very important step in the process. Not only does it get you in the mindset of taking the exam, but the content in the questions can sometimes be very similar to actual questions that you'll see on the exam. Practice tests can begin to point out weaknesses in your preparation and experience. Get lots of reps in. You won't regret it.

Some of the video courses I have taken also offer practices tests, but they often aren't on par with the difficulty level of the real exam. I have only used Jon Bonso's practice questions he sells on Tutorials Dojo. They are high quality and very affordable.

I haven't used Whizlabs for practice tests, but I've heard good things and plan to give them a try in the future.

Bonus Tips for Efficiency

1. Exercise and Learn at the Same Time!

This is a big one for me. Two birds, one stone. I have spent many hours on walks listening to and (kinda) watching relevant YouTube videos. Or I'll have a video playing on the TV in my home gym while I lift weights. This adds up over time and I can't recommend it enough.

2. Redeem Car Time

I started prepping for my first associate test while in the car on a 12+ hour drive to a family vacation in Florida. My wife was kind enough to drive most of the way. I downloaded lots of video content from A Cloud Guru and Udemy onto my laptop and iPad before we left. I took copious notes in OneNote while watching.

Even short drives across town can provide some bonus prep time. Remember, the knowledge will keep accumulating and more reps will help you remember the content better.

3. Increase video playback speed

If you can handle it, put the video playback speed on 1.5-2x. Most instructors talk slowly enough that you can still absorb that information at an increased speed. This can cut your video-based learning time by up to half.

4. Learn How to Analyze Exam Questions

I found several YouTube videos that walked through strategies on how to process and answer an exam question. I didn't do this soon enough. Some of the small tips you pick up can help you gain an edge on efficiency (remember, you are timed) and answering correctly. Tips include how to identify keywords in the question and answer choices, ignore fluff wording, and quickly eliminate invalid answers.

How to Take the Exam

You have two main options for taking the exam: at an official in-person testing site or an online proctored exam. I chose online and have been satisfied. It's hard to beat the ease of testing from home.

In the U.S., you have two online testing companies to choose from: Pearson Vue and PSI. I read lots of reviews recommending Pearson Vue over PSI, so I have only used Pearson Vue. The UI/UX is a bit clunky, but it gets the job done.

The exams range from 90 to 180 minutes long. If you take the test online, there is about a 15-minute check-in process.

Pro Tip: You are not allowed any breaks once the proctor approves you during the check-in process of the online exam. Be prepared to manage your bladder because you are not allowed to take any breaks during the online exam. I honestly have considered throwing on a pair of Depends to make sure I don't fail a test due to this. No joke - start paying attention to your restroom patterns, how coffee affects you, etc., and plan the time of day that you take the test around it. Or change your habits that day to make sure it's a non-factor.

Be prepared for the test to start with a string of hard questions that make you break out into a cold sweat and wonder if you prepared well enough. This last test was no exception. It's happened on at least one other exam I took. I don't think this is intentional per se, but it can be a little jarring until you hit a few questions more inline with what you prepared for.

Some more tips:

Mark questions you are unsure of with the "Flag for Review" button. If you have time at the end, you can circle back to these.
Pay attention to the clock. You don't want to spend more than a couple minutes on a question during your first time through. Pace yourself to have at least 10 minutes left at the end for reviewing flagged questions.
Read the question and answers carefully! Especially the last line of the question that typically qualifies the solution as needing to be either "the most cost effective", "most performant", etc. I know it's a timed exam, but glossing over one key word could cause you to miss a question that may have otherwise been easy.

When you finish the test (and proceed to evacuate your bladder), you will begin to wonder why you haven't received an email with your test results yet. Be forewarned: the official email from AWS can take a day or more to show up. Officially, it states it can take up to five days. In my experience, I usually have the email by around 2pm CT the next day. If you have Credly hooked up to your AWS Certification account and assuming you passed the exam, Credly sends a congratulations email early the next morning around 4:30am CT that beats the AWS email to the punch. It's nice getting the quick pass/fail confirmation sooner rather than waiting. I believe by the time you have the Credly notification email, you can login to your AWS Certification account and look up your results to see your numeric score. So even without having Credly connected to your account, it's fairly safe to say that you can login manually the next morning and most likely the official result will be posted.

What the Future Holds for Me

My current goal is to complete 3 more exams by the end of this year (2022). I feel this will round out the depth of knowledge in the areas I plan to focus on with my consulting work. It will also cement my visibility as an expert on the AWS IQ platform.

I have fallen more in love with AWS over the past year. I'm excited to keep growing and continue to spend my daytime job doing the same things I would do for a hobby.

Parting Thoughts

All in all, these exams can be very challenging. But that should not intimidate you. If you have some familiarity with AWS and are willing to put in the time to study and get more hands-on experience, they are a challenge worth accepting. Best of luck to you if you decide to pursue it.

Please leave any questions or experiences in the comments below or hit me up on Twitter or LinkedIn.