Forem: Bayu Wibowo

Single-Digit ms latency with AWS Local Zone

Bayu Wibowo — Fri, 05 May 2023 23:31:07 +0000

A few months back, AWS made an announcement about a new Direct Connect (DX) location in Auckland, which was great news for businesses in the region looking to connect their on-prem to AWS. And just last week, AWS launched an AWS Local Zone in Auckland, bringing AWS services even closer to NZ businesses.

One of use case of AWS Local Zones is to deliver single-digit latency. But can the AWS Local Zones really achieve the single-digit latency? As I'm based in Auckland, it's a good opportunity for me to test out Auckland's Local Zone. But before that, let's take a quick review on what is AWS Local Zone, what features does it support and how does the pricing would look like.

AWS Local Zones Overview

As explained on the AWS Local Zones User Guide, Local Zone is like a VPC extension of an AWS parent region. As you can see in below diagram, we have Sydney (ap-southeast-2) as a parent region and we extend the VPC into Auckland by creating a subnet in a Local Zone in Auckland (ap-southeast-2-akl-1a)

Enabling a Local Zone is pretty straight forward, just go to the the EC2 console (not VPC) and make sure you're in the right region. Then under Account attribute section, go to Zones > select your desired Local Zone > click Manage and enable it.

Once a Local Zone is enabled in the region, it will be seen like an additional Availability Zone in that region as shown below

AWS Local Zones Features

The next question would be, what features are supported in AWS Local Zones? From the AWS Local Zones Online Tech Talks session, it looks like it supports a number of services in LA, but outside of that it's mostly the core compute and networking like EC2, EBS, VPC, and DX.

Link to the latest AWS Local Zones features here: https://aws.amazon.com/about-aws/global-infrastructure/localzones/features/. Just looking at Auckland, it supports EC2 (t3, c5, r5, m5, upcoming g4dn), EBS (gp2 only), standard AWS Shield, ECS, EKS, VPC and DX.

AWS Local Zones Pricing

It's important to note that the pricing for Local Zones is different from the pricing of the parent region. Let's compare a t3.medium EC2 Linux pricing between Sydney vs Auckland and estimate it for a month usage.

Location	Instance Type	On-Demand Hourly Rate	Monthly Cost (730 hours)
Sydney	t3.medium	$0.0528	$38.54
Auckland	t3.medium	$0.0713	$52.05

Another note is that data transfer charge would be different as well, that includes data transfer between a Local Zone and an Availability Zone within the same AWS Region (e.g. Auckland and Sydney)

Latency Test and Comparison

Lastly, let's see if it can really deliver a single-digit millisecond latency. I'm going create an EC2 instance in Sydney, an EC2 instance in Auckland, and ping/mtr from my laptop in Auckland.

The diagram below shows the setup and quick ping/mtr test between my laptop and each of the EC2 instances as well as between them.

Conclusion

If your business in NZ can't leverage AWS Services in Sydney due to latency or perhaps data residency requirements, this AWS Local Zone in Auckland might be worth a look. Make sure to double check the supported AWS services first, as some may only be available in the parent region, and data transfers between Local Zone and parent region Availability Zone may be subject to data transfer charge.

While achieving single-digit latency may not be guaranteed, the Auckland's Local Zone has the potential to significantly improve performance and reduce latency for customers using AWS services from within New Zealand and can't wait for the Auckland Region to launch. In addition, this may be an option for organisations that want to leverage AWS services but don't want to manage physical infrastructure on-prem and can't use existing regions due to latency. Unlike AWS Outpost, the AWS Local Zones don't require customers to host any infrastructure on-premises.

Understanding the Networking Basics of Lambda to RDS Connectivity

Bayu Wibowo — Fri, 07 Apr 2023 22:18:15 +0000

When it comes to connecting AWS services that are managed by the cloud service providers (e.g., Lambda, RDS, DynamoDB), we tend to think everything is connected under the hood and all we have to do is to create inbound rules on the security group or grant permissions. Generally speaking, it usually works out fine when we use the default VPC. If you're new to VPC, private/public subnets, and route table concepts, feel free to head over my other blog post here: https://dev.to/bayupw/learn-and-build-aws-vpc-networking-for-network-engineers-1fch

Being a network guy, I'm always curious about how things work on the networking side. In this blog post, I'll dive into some basics networking and relevant key concepts when connecting AWS Lambda function to RDS. By the end of this post, hopefully you'll have a good understanding on how to connect these two services and how to secure them from a network & security point of view especially if you're following the Andrew Brown's Free AWS Cloud Project Bootcamp. So, let's get started!

Amazon RDS Networking

As ChatGPT said below, RDS is a managed service from AWS where we just provision a Database and the service will be accessible once provisioning is completed. But how do we connect to it? Does it run on like a "Public Zone" of AWS or is it going to be on provisioned in a VPC? Let's take a look.

If you go ahead and provision an RDS, you'll notice that there's a 'Connectivity' section where we need to choose a VPC and supply or select other relevant parameters related to network & security.
When a service need to run in a VPC, this is normally a good indicator that the service will create a network adapter - ENI in the VPC. Notice the note: "After a database is created, you can't change its VPC." In a case where you need to change your RDS into another VPC for whatever reason, you may need to perform a migration instead. Here are some useful links on RDS migration to a different VPC/account:

DB Subnet Group

After choosing a VPC, the next required parameter would normally be the subnet or subnet ID. But not quite with RDS, instead it is asking for something called DB Subnet Group. As per the name, DB Subnet Group is essentially a logical grouping of subnets in at least 2 AZs - it will throw an error if you try to create a DB Subnet Group with in 1 AZ.
When deploying a Multi-AZ RDS Deployment, additional IP will be created on the other AZ(s) for the standby DB instance(s). Note: while you can use IP address to connect to the DB instance, the IP might change during failover and therefore it is recommended to use the DNS name to connect to the DB.

The DB Subnet Group typically consists of private subnets which don't have a default route towards the AWS Internet Gateway as you probably don't want your RDS to be accessible from the Internet in most cases. But if you do, then you will need to have public subnets in your DB Subnet Group.

Public Access

The next part is the 'Public access' parameter which needs to be aligned with the DB Subnet Group configuration. When set to No, the provisioned ENI will only have Private IP.
If you try to access it from the Internet, your PC or a cloud development environment such as Gitpod that would obviously not be accessible.
However, if you set it to Yes, the ENI will also have a Public IP and a Public hostname (DNS endpoint). This settings can be changed without the need to re-provision the RDS which is quite handy.

VPC Security Group

Similar to an EC2 instance with ENI, access to RDS from network & security point of view can be controlled through a VPC Security Group rules. This can be an existing Security Group (e.g., default Security Group) or a new Security Group. A new dedicated Security Group for RDS might be easier to manage as we would know any rules being defined will be for the RDS access.

In my case, I'll set the 'Public Access' to Yes and I'll go ahead and create a new Security Group and add inbound rules for my public IP to Connect to PostgreSQL.
As per screenshot below, I can connect to the PostgreSQL using a Database Client UI or psql CLI using Connection URI: postgresql://<user>:<password>@<rds-endpoint>:<port>/<db-name>

RDS Networking Diagram

A picture says a thousand words and I love to draw everything when it comes to technical stuff. Let's take a look at all the components we've discussed so far in a technical diagram format.

AWS Lambda Networking

Now that we have covered off RDS Networking, let's take a look at what we'll need to spin up a Lambda function from networking point of view so we can have it connected to RDS.
In this case, I will create a Lambda function with python to do database schema load into the PostgreSQL. Note: I'm storing connection URI on the environment variable for demo purpose. For production, you may want to use something like AWS Secrets Manager and Lambda dynamically fetch secrets from Secrets Manager. There's a good article in AWS blog that talks about using AWS Lambda function to run post-database creation scripts.

As per the AWS Lambda docs under networking section, Lambda function always runs inside a VPC owned by the Lambda service and by default, is not connected to a VPC in your account. There's a section in the Lambda configuration where you can edit VPC related configuration and it requires certain IAM permissions to be able to create and manage network interfaces to create the HyperPlane ENI as per the AWS Lambda doc.

I'll go ahead and choose a VPC, 2 subnets and a new dedicated Security Group for the Lambda ENI.
The configuration will let you choose single subnet, but it will throw a warning that AWS recommends choosing 2 subnets for HA.
Once the configuration is updated, you will see an ENI with a private IP.

AWS Lambda and RDS Networking Diagram

With the Lambda connected to VPC, here's the updated diagram for single AZ.
You can use the default Security Group for both RDS and Lambda and everything will just work. But, since I use dedicated Security Groups for each of the components, I will need to add inbound rules on the RDS Security Group to allow Lambda to connect.
While technically we can use IP address as the source, we want to allow all of the Lambda ENIs and the IP address may change in the future (added/removed), so it is better to use the Security Group ID of the Lambda as per the screenshot.

Test Lambda to RDS Connectivity

Now everything is ready, let's test this out by invoking the Lambda function. I'll do this via aws cli: $ aws lambda invoke --function-name <function name> response.json
Let's double check the CloudWatch logs as well.

Last thing is to check on the database itself to see if the schema has been successfully loaded by the Lambda function.

Everything looks to be working as expected! You may be wondering, will Lambda overload the RDS connectivity in this scenario? There's actually another AWS service that would be useful for this - Amazon RDS Proxy, see Using Amazon RDS Proxy with AWS Lambda

CloudFormation Template

Want to test it out yourself? Feel free to use this Lambda RDS Demo CloudFormation Template
Note: this CFN template is for demo purposes only as it will output username, password, Database Connection URI which may be sensitive from security point of view.

Conclusion and Important Key Points

As we can see, understanding cloud networking & security and their components is crucial when building cloud services, whether they're PaaS or managed. This is useful not just for planning or deployment, but also for troubleshooting, since we often need to figure out what are the building blocks and connectivity during a debug session.

Here are some additional links that I find really useful in regards to connecting Lambda to RDS:
https://blog.thundra.io/can-lambda-and-rds-play-nicely-together
https://docs.aws.amazon.com/lambda/latest/dg/services-rds-tutorial.html.

I hope you enjoy reading this post and feel free to reach out if I have missed anything that you feel important!

How to Solve Overlapping IP Addresses in AWS

Bayu Wibowo — Sat, 02 Jul 2022 23:58:02 +0000

Recently, I got a chance to present at the Auckland AWS Community meetup to talk about "How to Solve Overlapping IP Addresses in AWS". PDF slide deck is here.

Every service, transaction, and everything you do with AWS or any public cloud uses networking. But it's quite common to see network and security design decisions made afterthoughts, which has led to overlapping IP addresses issues. But there are times when overlapping IP can't be avoided as well.

In this post I'm going to share what those overlapping IP scenarios are, how we can solve the issue, and I hope everyone will find this helpful.

Overlapping IP Address Scenarios

When organizations are starting to deploy services in the cloud and spin up VPCs, it's easy to fall into overlapping IPs between VPCs or with on-premises. There are a few common use cases where private networks with overlapping IP addresses need to be connected.

Connecting default VPCs. This may not necessarily be your VPC, but I've seen multiple organisations using the default VPC and they need to connect the two which is not possible out of the box.

It's still possible for multiple VPC's to end up on the same CIDR block (e.g., 10.0.0.0/16, 192.168.0.0/16) whether it's within AWS or connecting another cloud provider. The reason could be mergers & acquisitions or connecting to other networks like third parties, partners, B2Bs.

The most common scenario is when an organisation has cloud services and wants to deliver a service to a customer or partner using private IP addresses. They won't have the ability to ask their clients to change their IP space.

Solution Options to Overlapping IP Addresses

During the meetup event, I discussed at least three possible solutions to this issue. This post will cover them at a high level, I may have to write separate posts to cover them in more detail.

1 | AWS Private Link

AWS PrivateLink service has been around for a while, and it allows you to publish services from one VPC that you can access from another VPC. Through VPC Endpoint (VPCE), you can extend services behind an NLB (Network Load Balancer) into a different VPC. PrivateLink is made up of NLB and VPCE. Few things to note, Private Link is uni-directional and only works with TCP traffic

Here's a sample terraform code that creates VPCs with overlapping IPs and connects them using Private Link: https://github.com/bayupw/terraform-aws-overlapping-private-link-demo

2 | AWS NAT Gateway

Use AWS NAT Gateway if PrivateLink isn't a feasible option, especially when you're having UDP traffic.

To use AWS NAT Gateway, the services need to be created behind an ALB and published over a separate secondary VPC CIDR block that doesn't overlap. The secondary VPC CIDR blocks can then be connected using AWS Transit Gateway. Be sure to disable the route propagation as you don't want to advertise the overlapping IP spaces. One requirement for this solution is that the NAT Gateway needs to be deployed on the consumer VPC, which may not be feasible for some organisations. There's a blog post that covers it in detail here: https://aws.amazon.com/blogs/networking-and-content-delivery/how-to-solve-private-ip-exhaustion-with-private-nat-solution/

Here's an example terraform code for deploying this solution: https://github.com/bayupw/terraform-aws-overlapping-nat-gateway-demo

3 | Aviatrix Secure Cloud Networking Platform

Another option is is to use Aviatrix to solve this overlapping IP issues. With Aviatrix gateways, you can solve overlapping IP addresses using an intelligent, flexible NAT solution. The Aviatrix Controller configures the routes to the AWS VPC route tables based on the Virtual CIDRs configuration. With this solution, all the NAT is done on the provider's end, so the consumer doesn't have to configure any complicated NAT.

Here's a sample terraform code to deploy the solution: https://github.com/bayupw/terraform-aviatrix-aws-overlapping-demo

There's also a nicely written blog post written by Brad Hedlund and AWS Solution Architects which covers this solution in detail from starting up the Aviatrix Controller, deploying the Aviatrix Gateways, configuring intelligent NAT, testing the traffic, and observing the traffic flow through Aviatrix CoPilot: https://aws.amazon.com/blogs/awsmarketplace/how-to-solve-overlapping-ip-addresses-using-the-aviatrix-cloud-network-platform/

Besides solving overlapping IP issues, Aviatrix also provides additional features that organizations often need in an overlapping IP scenario, like the option for bi-directional traffic, policy-based segmentation, traffic inspection via Next-Generation Firewall and visibility of traffic flow regardless of the cloud provider. There was an Aviatrix TechTalk session which covered this overlapping scenario in a SaaS provider scenario with a nice demo.

Alternatively, you can read about the solution here: https://aviatrix.com/resources/all-content/aviatrix-validated-design-saas-providers-infrastructure

Summary

Those are the three options to solve overlapping IPs issues.

AWS PrivateLink: use this if you're only dealing with uni-directional TCP traffic
AWS NAT Gateway: if PrivateLink isn't an option, you need UDP traffic and perhaps need to steer the traffic to a Security VPC, you can evaluate AWS NAT Gateway for hiding the overlapping IP space.
Aviatrix: last but not least, if none of the other options work for you or you need extra features like visibility, network segmentation, policy-based traffic steering to a Next-Gen Firewall, which often required in a production enterprise network, I'd suggest exploring Aviatrix! 🙂

I hope you found this article is informative and feel free to reach out if you have any questions!

Learn and build AWS VPC Networking for Network Engineers

Bayu Wibowo — Sat, 02 Jul 2022 21:25:19 +0000

Many cloud projects forget Network Engineers and Network Architects and only bring them in afterward. Often, when network guys are in, things are already complicated and there's not much time for them to learn AWS from scratch.

There are no prerequisites for AWS certifications anymore. So it's not mandatory to take the Associate certifications. I would still recommend taking at least the AWS Solutions Architect Associate or even the Cloud Practitioner certification if you're new to cloud computing. In addition to AWS certification, Aviatrix offers a free course which covers an Intro to Public Networking in AWS through ACE-Associate as you can see below

It's a good course on Cloud Networking whether or not you're considering Aviatrix. Here's the registration link and voucher code to make it free

ACE-Associate registration link: https://aviatrix.teachable.com/
Voucher code: AWSAKL

To me, learning through practical experience or even parallel learning and building is the way I like to go. In this post, we'll learn and build networking constructs in AWS from scratch. Go ahead and create an AWS account or use an existing one if you want to follow along. If you're new to AWS, make sure to setup MFA to protect your account, see video below on how to setup MFA on your AWS account

We're going to use free resources as much as possible so you shouldn't be charged for anything. However, I'm not responsible for any charges you may incur. 🙂

VPC (Virtual Private Cloud)

VPC is the foundation of AWS networking. Think of it like a logical data center you're building inside AWS. Similar to on-prem, before you can run a server or storage inside a data centre, you'll need a network switch and at least a network/subnet/VLAN to connect them. We will build from UI - ClickOps style and we'll explore using IaC tools like CloudFormation template or Terraform in other posts. So let's create a VPC and a CIDR block, and in this example, we'll create a VPC called "first-vpc" and we'll use "10.1.0.0/16" as our CIDR block. While you can use any IPv4 ranges, it is recommended to use the RFC1918 ranges.

Overall steps as follows:

Log on to AWS Console
Select desired region
Go to VPC
Create VPC, enter VPC name (tag) and input CIDR block

In this case, I'll be using Sydney (ap-southeast-2) region

On the search bar, type VPC and click the VPC hyperlink - you can click the star to add VPC into the favorites list/bar.

If your using a new AWS account, you'll notice that there's an existing VPC also referred as the Default VPC which is an Internet connected VPC, more about default VPC is documented here: https://docs.aws.amazon.com/vpc/latest/userguide/default-vpc.html. While you can use this VPC to deploy AWS services, it is recommended to create a custom VPC. Default VPC is also considered as a high severity in tfsec, see this link: https://aquasecurity.github.io/tfsec/v1.8.0/checks/aws/vpc/no-default-vpc/. You can delete the default VPC if you're not planning to use it.

Click the Create VPC button

As we want to learn step by step and observe the components, choose the "VPC only" option, we'll use the "VPC and more" option later.
Enter "first-vpc" as the name tag and "10.1.0.0/16" as the IPv4 CIDR. The "10.1.0.0/16" will be the primary IPv4 block and you can add a secondary IPv4 block e.g., "100.64.0.0/16". The use case of secondary CIDR block could be because you're running out of IPs and need to add additional block, or there's a VPC with overlapping CIDR which you need to peer or connect. See this blog post on how a secondary CIDR block is being used in an overlapping IP scenario: https://aws.amazon.com/blogs/networking-and-content-delivery/how-to-solve-private-ip-exhaustion-with-private-nat-solution/

Leave the tags as default, you can add more tags if you want and click Create VPC

As soon as the VPC is created, it's assigned with a vpc-id and there's a route table created that serves as the main route table - rtb-0b9c6e04cb2764aad in below example.

Right click the rtb-id, open a new tab, and select the Routes tab to observe the configuration.

The route table has a default local route which is used for communication within the VPC. If there's multiple CIDR blocks (secondary VPC CIDR), then you will have local route for each CIDR block as below example

Now you have a VPC and a route table, but you won't be able to put anything inside. If you try to create an EC2 instance for example, you can't proceed as it requires subnets.

Subnets

VPC can be created on a region basis, but a subnet where services run can only live in one availability zones (AZs). In order to offer services in multiple AZs, you'll need a subnet on each of them. For more details on regions and zones, check out this document:
https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-regions-availability-zones.html
Let's create two subnets across two AZs as shown in below diagram (you can create more AZs if you want). The smallest subnet we can create is /28 with a total of 14 usable IP. 5 IP addresses are reserved by AWS as described in this document: https://docs.aws.amazon.com/vpc/latest/userguide/configure-subnets.html#subnet-sizing

Subnet name	AZ	CIDR block
sydney-public1a	ap-southeast-2a	10.1.11.0/24
sydney-private1a	ap-southeast-2a	10.1.1.0/24
sydney-public2b	ap-southeast-2b	10.1.12.0/24
sydney-private2b	ap-southeast-2b	10.1.2.0/24

Go to VPC > Subnets > Create Subnets and select the VPC that you've created previously - the "first-vpc"

Enter the subnet settings detail. Don't click the "Create subnet" button just yet, click the "Add new subnet" button to add the remaining subnets then after completing all the required subnets, click "Create subnet"
Note: if you don't choose a zone, it will be randomly picked by AWS.

Once done, you should see all the subnets you just created on the console. If you missed any, just create a subnet and select your desired VPC. As of now, you can deploy EC2 instances into the VPC by selecting one of the subnets, but the public subnet doesn't have any Internet access at this stage. When you select a public subnet > route, you'll see it uses the main route table and only has the local route, no default route for Internet access.

Public Subnets

Technically, the subnets are still private. You'll need these to make it work as public subnets:

An Internet Gateway (IGW) attached to the VPC
Route table with default route towards the IGW
Public IP assigned to the AWS resources (e.g., EC2 instances)

Go to VPC > Internet gateways and click "Create internet gateway"

Put a name tag and click create internet gateway

Attach the IGW to the first-vpc

We want the private subnets to be private, we don't want the private subnets to have a default route to the Internet. For that, we'll need to create a separate route table for the public subnets. Here's a diagram showing what we're trying to do.

Let's go to the route table menu and create a route table for the public subnets.

Put a name for the route table e.g., first-vpc-public-rtb and select the desired vpc - "first-vpc"

Once created, edit the route table, add a default route to the IGW

Next, go to the "Subnet associations" tab and click "Edit subnet associations"

Select the public subnets and click "Save associations"

That's it! Now that the VPC is ready, you can run an EC2 instance in public subnets if they need Internet access or in private subnets if they don't.

But let's create another VPC but now using the other option which will create required route tables and internet gateway for us. Create a New VPC and choose "VPC and more" this time

We'll create using below parameters

Parameters	Value
Name tag (auto-generate)	second
IPv4 CIDR block	10.2.0.0/16
Number of AZs	2
First AZ	ap-southeast-2a
Second AZ	ap-southeast-2b
Public subnet first AZ	10.2.11.0/24
Public subnet second AZ	10.2.12.0/24
Private subnet first AZ	10.2.1.0/24
Private subnet second AZ	10.2.2.0/24
NAT gateways	None
Enable DNS hostnames	Checked (default)
Enable DNS resolution	Checked (default)

Hit "Create VPC" once completed, you can see that the wizard will create all of the required components similar to what we've created for the first-vpc.

Now you have two VPCs and they are ready to take on workloads as shown in below diagram:

I hope this post helps you better understand AWS and cloud networking, especially some of the foundation of VPC networking. We haven't touched on security groups, NACL, how to give egress Internet access to the private subnets, how to connect VPCs, how to automate the build and manage the resources using IaC, I'll save that for future posts. Watch this space!

References: