Forem: Zhen Kai

New AWS Management Console Home is a Step in the Right Direction

Zhen Kai — Tue, 01 Feb 2022 02:41:59 +0000

On 12th January 2022, AWS announced a new AWS Management Console home page. I think it's a step in the right direction. Here's why.

Old console home page

Lack of Customization

In the old console home page, you had a lack of customization. You cannot move, resize, remove any elements on the console page. The only thing you could customize is a bookmark of your favourite AWS services.

No Consolidated Dashboard

Although there are dashboard features found in a couple of AWS services, there isn't a customizable multi-service dashboard. For example, in the EC2 and RDS console page, you have the EC2 and RDS dashboard respectively.

Bloat (for experienced users)

The old console home page contains a lot of content meant for new users. For experienced users, majority of these content are bloat, diluting the overall console home experience. An example of bloat:

Other bloat content includes boxes for "Getting Started" and "Explore AWS", which is hardly ever used for an experienced user. This issue can be solved if customization allowed these bloat content to be removed.

What's on the new Console Home?

At launch, users can utilize 8 widgets. I will list all of them down below, along with a simple description, my personal recommendation and a rating as an experienced user.

Welcome to AWS
- Includes links to learn fundamentals of AWS, training, certification and AWS news. Widget only available in 1 size.
- Basically useless for experienced users. Will benefit new users accessing the AWS console for the first time.
- 2/10. Remove it!
Recently visited
- List of AWS services you've recently visited. Widget available in Regular or Extended views. Has pagination in the regular view.
- A frequently used part of the old console home page. Hard to find faults with this.
- 8/10. Pretty useful.
AWS Health
- Contains events that might affect your AWS infrastructure and account. Widget only available in 1 size.
- Not really useful for solo developers or beginners. Very useful for enterprise customers with huge infrastructure.
- 5/10. Must have for enterprise users, mostly useless for everyone else.
Cost and usage
- Visualizes your AWS costs and usage. Contains information on your current, previous and projected month’s costs for the Regular view. Includes a cost breakdown for a few of your top spending AWS services in Extended views.
- Extremely useful for all types of users. New users can spot unterminated cloud resources, enterprise users can get a quick sensing of cloud spending trend. Recommend to set this at Extended View.
- 10/10! The runaway winner of this new console home page change.
Build a solution
- Workflows and wizards that introduce you to AWS services. Widget available in Regular or Extended views.
- Mostly unused and a big part of the previous bloat problem. Usually not the first place experienced users look at when thinking of solving a problem. Might benefit new users more.
- 3/10. Probably should remove.
Trusted Advisor
- Mainly for accounts with AWS Business Support or AWS Enterprise Support. Can see an overview of automated checks from Trusted Advisor. Widget available in Regular or Extended views.
- Practically useless for accounts on Basic (free) and Developer support plan. Benefits enterprise customers with sizeable cloud infrastructure spending.
- 5/10. Must have for enterprise users, mostly useless for everyone else.
Explore AWS
- Contains AWS training and certification links. Widget only available in 1 size.
- Useless for experienced and enterprise users. Hard to see beginners use this as well. Seems to overlap with Welcome to AWS widget.
- 1/10. Remove it immediately!
Favorites
- Shows the AWS services that users bookmarked. Widget available in Regular or Extended views.
- Pretty useful all around, more so for enterprise and experienced users. New users will probably utilize the Recently visited widget more often.
- 7/10. Less often used than Recently visited, but still a mainstay for most users.

What I want to see in the future?

The more widgets there are, the more customizations there will be. The new console page does have a nifty little link for users to submit widget suggestions (see screenshot below). I would also be interested to have more options to resize the widgets.

I hope that individual service dashboards, for example, the EC2 dashboard, can be made into a widget. It could also be extra helpful for Ops folks if Cloudwatch dashboards can be exported as widgets on the home page.

Lastly, I want the ability to share home page configurations across AWS Organizations or different accounts. I feel like that would be helpful for users managing more than 10 AWS accounts.

Closing Thoughts

Historically, AWS console home page have been lagging behind Azure and Google Cloud. This change is a step in the right direction and definitely has an immediate positive impact for enterprise and experienced users. So, enable the new console home page experience right now!

Let me know what you think. Don't forget to follow me on LinkedIn.

AWS VPC IPAM Tutorial - The Better Terraform Way (7 Steps Faster)

Zhen Kai — Mon, 10 Jan 2022 14:06:01 +0000

In my previous post covering the basics of AWS VPC IPAM, I promised some Terraform code samples.

My Terraform code sample will be based on the AWS VPC IPAM tutorial from the official documentation. As you can already see from the title, the "Terraform way" is 7 steps shorter than the official tutorial and undoubtedly better.

The Better Terraform Way

Step 1: Clone Github repository

From your terminal, run git clone https://github.com/sgLancelot/aws-vpc-ipam-terraform-tutorial.git. Change your current directory to the cloned code with cd aws-vpc-ipam-terraform-tutorial.

Step 2: Terraform Apply

Note: This step assumes you already have your AWS credentials set up and Terraform installed. As of this point of writing, you will require Terraform version 1.1.2 or higher.

From your terminal, run terraform apply. Review the Terraform planned changes before you type yes. If you are using the default values, you should expect a plan consisting of 9 to add, 0 to change, 0 to destroy.

After applying the changes, head over to your AWS console to view what you've created.

Step 3: Terraform Destroy

Note: This step may take up to 25 minutes to complete. From testing, this seems to be caused by the pool CIDR assignment requiring some time to detect that the test VPC is deleted before allowing you to unassign the CIDR.

From your terminal, run terraform destroy. Review the Terraform planned changes before you type yes. If you are using the default values, you should expect a plan consisting of 0 to add, 0 to change, 9 to destroy.

That concludes the AWS VPC IPAM tutorial, the better Terraform way. Next, we will walkthrough the code to give you an understanding of what you just created and destroyed.

Bonus: Code Walkthrough

In this section, I'll walkthrough my code and describe my thought processes.

Variables

In the variables.tf file, you can see where the variables are defined. The default values follows the tutorial, but I've put them as variables to give you the option to define them yourselves in your .tfvars file, if you choose to do so. Otherwise, the default values will work.

variable "region" {
  type        = string
  description = "The AWS Region that the resources will be created in. Will also be included as part of the IPAM operating region"
  default     = "us-east-1"
}

variable "ipam_operating_regions" {
  type        = list(string)
  description = "Additional AWS VPC IPAM operating regions. You can only create VPCs from a pool whose locale matches this variable. Duplicate values will be removed."
  default     = ["us-west-2"]
}

variable "top_level_pool_cidr" {
  type        = string
  description = "The top level IPAM pool CIDR. Currently only supports a single CIDR."
  default     = "10.0.0.0/8"
}

The variables are pretty self-explanatory from the description I've added.

From here on, all the code can be found in main.tf.

Service-Linked Role

resource "aws_iam_service_linked_role" "ipam" {
  aws_service_name = "ipam.amazonaws.com"
  description      = "Service Linked Role for AWS VPC IP Address Manager"
}

I've chosen to follow the tutorial without using AWS Organizations, and hence, the service-linked IAM role needs to be created for VPC IPAM to automatically discover resources to monitor. There is nothing fancy here.

IPAM and it's operating regions

The IPAM construct requires you to define its operating region. One of the operating region must include the AWS provider block region, in this case, it's defined as var.region.

locals {
  deduplicated_region_list = toset(concat([var.region], var.ipam_operating_regions))
}

The deduplicated_region_list local variable ensures that the list of regions that you pass into IPAM does not have any duplication, which might cause an error when creating the IPAM. To learn more about the toset function in the Terraform documentation.

resource "aws_vpc_ipam" "tutorial" {
  description = "my-ipam"
  dynamic "operating_regions" {
    for_each = local.deduplicated_region_list
    content {
      region_name = operating_regions.value
    }
  }
  depends_on = [
    aws_iam_service_linked_role.ipam
  ]
}

Here, local.deduplicated_region_list is passed into the operating systems configuration block as a dynamic block.

Another interesting point is the depends_on meta-argument to create a dependency between the service-linked role and IPAM. This allows the IPAM to be deleted before the service-link role is deleted. From testing, letting Terraform perform this deletion without depends_on actually causes an error as it deletes the service-linked role and IPAM in parallel.

Top-level Pool and CIDR assignment

Along with the creation of the IPAM, a default private and public scope is created as well and can be reference via the IPAM Terraform resource's attributes. To understand more about what a scope is, do check out my previous post covering the basics of AWS VPC IPAM.

resource "aws_vpc_ipam_pool" "top_level" {
  description    = "top-level-pool"
  address_family = "ipv4"
  ipam_scope_id  = aws_vpc_ipam.tutorial.private_default_scope_id
}

# provision CIDR to the top-level pool
resource "aws_vpc_ipam_pool_cidr" "top_level" {
  ipam_pool_id = aws_vpc_ipam_pool.top_level.id
  cidr         = var.top_level_pool_cidr # "10.0.0.0/8" if following the tutorial
}

The top-level pool will be created in the IPAM's private scope. The var.top_level_pool_cidr variable follows the tutorial with 10.0.0.0/8. You may set a different CIDR as long as it as a netmask of above/16.

Sub-level Pool and CIDR assignment

For this part, I further improved on the tutorial of simply creating just 1 regional sub-level pool. Using the for_each meta-argument, the resource taps on local.deduplicated_region_list local variable again to create multiple regional pools according to the region list you've set.

resource "aws_vpc_ipam_pool" "regional" {
  for_each            = local.deduplicated_region_list
  description         = "${each.key}-pool"
  address_family      = "ipv4"
  ipam_scope_id       = aws_vpc_ipam.tutorial.private_default_scope_id
  locale              = each.key
  source_ipam_pool_id = aws_vpc_ipam_pool.top_level.id
}

resource "aws_vpc_ipam_pool_cidr" "regional" {
  for_each     = { for index, region in tolist(local.deduplicated_region_list) : region => index } 
  ipam_pool_id = aws_vpc_ipam_pool.regional[each.key].id
  cidr         = cidrsubnet(var.top_level_pool_cidr, 8, each.value)
}

For the CIDR assignment for these regional pools, I had to convert the toset-transformed local.deduplicated_region_list to a list again. The purpose was to allow the for expression to properly retrieve the index of each region.

The code snippet below should help you visualize how the for expression looks like for the default values.

{ 
  us-east-1 = 0,
  us-west-2 = 1
}

With that, each.key would iterate through the key (the regions) and each.value would be the corresponding index in the list.

The reason why we needed the index value is to generate the sub-level pool CIDR dynamically from the top-level pool CIDR var.top_level_pool_cidr using the cidrsubnet Terraform function. In short, the cidrsubnet function slices up a CIDR according to the values you pass into it.

VPC to test

Finally, we've reached the end of the code sample. We will create a test VPC in the AWS provider region defined in var.region.

In the tutorial, they've manually assigned a direct CIDR block to this VPC, which seems ludicrous because it doesn't showcase the strength of an IPAM-managed VPC.

resource "aws_vpc" "tutorial" {
  ipv4_ipam_pool_id   = aws_vpc_ipam_pool.regional[var.region].id
  ipv4_netmask_length = 24 
  depends_on = [
    aws_vpc_ipam_pool_cidr.regional
  ]
}

In my Terraform code, we've chosen to test the new attributes added in AWS provider version 3.68.0, which for as long as I can remember, removes the cidr_block attribute requirement of being mandatory; cidr_block is now optional if you have the ipv4_ipam_pool_id and ipv4_netmask_length attributes. For my example, I've requested a CIDR of /24 netmask length from the regional sub-level pool.

It should achieve the same results as the tutorial as long as you are using the default values.

Closing

I had a blast writing this short Terraform code sample for AWS VPC IPAM. If there is anything you feel I could've done better, please reach out to me.

Hope you have learnt something!

AWS VPC IPAM Basics and Why You Need to be Careful

Zhen Kai — Mon, 03 Jan 2022 12:40:23 +0000

During AWS re:Invent 2021, David Brown, Amazon EC2 VP, announced Amazon VPC IP Address Manager (IPAM).

Why VPC IPAM?

VPCs with overlapping CIDRs cannot be peered, and is usually resolved only by destroying and recreating the VPC with the correct CIDR. If there are many existing workloads in the VPC, this process will incur painful migration effort and risks to live production systems.

IP address allocation to Amazon VPCs is something that organizations want to get correct at the beginning and never have to think about in the future. Amazon VPC IP Address Manager (VPC IPAM) helps by handling the IP address allocation, ensuring an organization will never issue conflicting CIDRs to its VPCs.

VPC IPAM can also save organizations from future embarrassments when new AWS resources are blocked from scaling due to a lack of free IP addresses. VPC IPAM has the ability to monitor IP address utilization, generating alerts to administrators when certain thresholds are crossed.

AWS VPC IPAM basics

[1] Service-Linked Role and IPAM Administration

The service-linked role AWSServiceRoleForIPAM is required for IPAM to discover, monitor and manage IPAM resources (resources are covered at the end of this list).

For IPAM administration, it slightly differs depending on situation:

Using AWS Organizations (documentation)
- Delegate an Organizations member account to be the IPAM account.
- The IPAM account will then be responsible for making IPAM-level configurations.
- The IPAM service-linked role will be automatically created in each member account in your organization.

For single AWS account (no AWS Organizations), the IPAM service-linked role needs to be created manually.

[2] IPAM

Before everything else, you'd need to create an IPAM first (documentation). At this point of time, you have to decide IPAM's operating regions. This ringfences IPAM to the regions where you would have resources for IPAM to manage. This can be modified later on.

[3] Scope (Private & Public)

After creating an IPAM, 2 scopes are automatically created, 1 public and 1 private scope. To explain what scopes are, I've compiled some findings from the documentation, re:Invent and my own testing:

You can only create private scopes.
Public scopes seem to be for your BYOIP (bring your own IP) resources. These IP addresses don't belong to AWS and are not managed by IPAM.
All IPAM resources (non-BYOIP) will be in the automatically-created private scope by default. You can shift them to a different scope subsequently.
Each private scope contains it's own set of managed IP CIDRs, IPAM Pools and resources. Inheriting overlapping AWS resources with overlapping IP CIDRS is a consideration for having additional private scopes.

[4] Pool (Top-Level)

From the documentation, a pool is a collection of contiguous IP address ranges (or CIDRs). When creating a top-level pool, it should contain all the specific CIDRs you'd like to allocate to resources within a scope.

AWS doesn't let you define regions here. So the CIDRs here can be consumed in any region.

Note: For each top-level pool, you can set allocation rules to enforce IPAM resource tagging and network mask allocation.

[5] Pool (Sub-Level)

A sub-level pool can be restricted to a specific region and is allocated a CIDR from a higher level pool. A recommended pool structure from AWS looks something like this:

You can have many levels of pools. Design the pool structure according to your needs. Sub-level pools do not require specific CIDRs to be defined; you could define a network mask and let IPAM handle the CIDR allocation automatically.

Note: For each sub-level pool, you can set allocation rules to enforce IPAM resource tagging, network mask allocation and region.

[6] Pool sharing and IP allocation

If you are using AWS Organizations, IPAM pools can be shared to member accounts via AWS Resource Access Manager (RAM).

Once shared, IPAM resources being created can utilize the pool to receive allocation of CIDRs from the pool. Additionally, a user from the member account can also be allocated an IP address through an API call to the shared pool.

[7] IPAM managed and monitored Resources

From the documentation:

In IPAM, a resource is an AWS service entity that is assigned an IP address or CIDR block. IPAM manages some resources, but only monitors other resources.

A managed resource has CIDR allocated from an IPAM pool, and is typically a VPC. A monitored resource extends beyond that, covering:

VPCs
Subnets
Elastic IPs
Subnet reserves
Public IPv4 pools

Note: The number of "active" IP addresses determines IPAM's costs. See the next section for more information on this.

That wraps up the Amazon VPC IP Address Manager basics and its various components.

You should be careful of...

In this section, I'll share some of my findings based on my experience testing the service out without AWS Organizations.

It is possible to create your IPAM, scopes, pools and allocate CIDRs to VPCs before creating your service-linked role. You wouldn't be able to monitor what you have allocated.
Once a pool locale/region is configured, it cannot be changed. The only way is to recreate it with the correct locale/region.
You cannot delete a higher level pool without deleting it's sub-levels first.
You must deprovision all CIDRs in a pool before you can delete your pool.

Costs

From the VPC pricing page:

You pay an hourly rate for each active IP address that you manage using IP Address Manager (IPAM). An active IP address is defined as an IP address assigned to a resource such as an EC2 instance or an Elastic Network Interface (ENI).

It is important to note that not only IPAM-managed resources with active IP addresses are billed, IPAM-monitored resources, with IP addresses not allocated by IPAM are billed as well. In Singapore (ap-southeast-1), each active IP address costs $0.00027 USD per hour.

This is a problem for users who run a self-managed Kubernetes cluster in AWS or use Amazon Elastic Kubernetes Service (EKS) with self-managed nodes. These Kubernetes nodes have many IP addresses per Elastic Network Interface (ENI) and with each IP address being classified as an "active IP address" to IPAM, it's easy to see how your bill could skyrocket.

Fortunately, I received a great tip to mitigate this from the AWS Community Builders group. By using prefix delegation, IPAM counts the whole delegation (16 IP addresses) as a single IP address. You can find additional considerations and limitations for using prefixes in the documentation.

Alternatively, you can consider using a separate CIDR as an overlay and potentially reserving those CIDRs manually on VPC IPAM. Since these CIDRs are not assigned on an ENI, it does not count as an active IP address.

Closing Thoughts & Whether You Should use VPC IPAM

Amazon VPC IPAM is a compelling and useful service for a moderately-sized AWS Organizations greenfield or brownfield set up. If you are running a Kubernetes cluster with self-managed nodes, VPC IPAM's costs could be exponentially higher. Consider some of the above suggestions to reduce costs before using the service.

Stay tuned for a future blogpost for some sample Terraform code to get you started on VPC IPAM.

Simple Serverless Telegram Bot with AWS Lambda

Zhen Kai — Mon, 04 Oct 2021 13:29:27 +0000

Previously, I created a simple reminder application with AWS Lambda and Simple Notification Service (SNS) SMS messages. With so many instant messenger applications these days, it is about time I move away from using SMS messages, specifically to Telegram.

Requirements

There are no changes to the previous requirements. To recap:

To send a text message to people once a month
Cheap
Serverless on AWS

Architecture

Similar to the previous application, I will still be using CloudWatch Events to trigger a Lambda function in a monthly interval to send the reminder message.

Previously, I had to manually subscribe mobile numbers to the SNS topic. The improvement this time for the Telegram bot comes from user self-service on-boarding to subscribe to the reminder notification, which is actually just a simple /start to the bot.

How the Bot works

Every day, a Lambda function (named Register) will be triggered to call the Telegram Bot API to retrieve any outstanding /start initiation. Upon detecting an outstanding message, the chat ID and username will be extracted and appended to the DynamoDB table.

Every month, a Lambda function (named sendMessage) will be triggered to scan the above-mentioned DynamoDB table, and recursively send out a custom Telegram message via the bot.

How to Configure the Bot

Step 1: Create a Telegram Bot with Botfather

Following the Telegram bot guide, you start by looking for @botfather in Telegram and following the instructions to create a new bot. Give your bot a meaningful name, description and profile picture.

Take note of the token that Botfather assigns to you; it will be required for your Lambda functions to call your bot API.

Step 2: Create a DynamoDB Table

Telegram Chat IDs will be unique for every user that initiated a chat with my Telegram Bot. This made chat IDs a great candidate as the primary partition key. I set the primary partition key as chat_id (Number).

I also configured on-demand Read/Write Capacity modes, because I knew my DynamoDB Table is not going to be accessed frequently.

Remember to take note of the DynamoDB Table name, you will need it for the next step.

Step 3: Create Lambda functions (register and sendMessage)

Go to the Lambda console page and create 2 Lambda functions, 1 for the registration workflow and 1 for sending messages. The codes for the Lambda functions are as follows (I used the Node.js runtime):

var AWS = require("aws-sdk")
var docClient = new AWS.DynamoDB.DocumentClient({region: "ap-southeast-1"})
const telegram = require('telegram-bot-api')

const tableName = process.env.DDB_TABLE_NAME

var bot = new telegram({
    token: process.env.TG_API_KEY,
    updates: {
        enabled: true
    }
})

const mp = new telegram.GetUpdateMessageProvider()
bot.setMessageProvider(mp)

bot.start()
.then(() => {
    console.log('API is started')
})
.catch(console.err)

exports.handler = () => {
    bot.on('update', update => {

        console.log(update)
        var params = {
            TableName: tableName,
            Item: { 
                chat_id: update.message.chat.id
            }
        }
        console.log("Adding a new item...")
        docClient.put(params, (err) => {
            if (err) {
                console.log('error cant read.', JSON.stringify(err, null, 2))
                bot.sendMessage({
                    chat_id: update.message.chat.id,
                    text: `Registration to Mr Gentle Reminderer is unsuccessful. Please let Zhen Kai know of the error. ${JSON.stringify(err, null, 2)}`
                }).catch(error => { console.log(error) })
            } else {
                console.log('succeed')
                bot.sendMessage({
                    chat_id: update.message.chat.id,
                    text: 'Registration to Mr Gentle Reminderer is successful! The gentle reminder will be sent on the first day of every month. Thanks!'
                }).catch(error => { console.log(error) })
            }
        })
    })
}

// Lambda function for sending messages
var AWS = require("aws-sdk")
var docClient = new AWS.DynamoDB.DocumentClient({region: "ap-southeast-1"})
const telegram = require('telegram-bot-api')

const tableName = process.env.DDB_TABLE_NAME

var bot = new telegram({
    token: process.env.TG_API_KEY,
    updates: {
        enabled: true
    }
})

var params = {
    TableName: tableName,
    ProjectionExpression: "chat_id"
}

exports.handler = () => {

    docClient.scan(params, (err, data) => {
        if (err) {
            console.error("Unable to scan the table. Error JSON:", JSON.stringify(err, null, 2))
        } else {
            console.log("Scan succeeded")
            data.Items.forEach(chatIdObj => {
                console.log('Chat ID is:', chatIdObj.chat_id)
                bot.sendMessage({
                    chat_id: chatIdObj.chat_id,
                    text: 'Gentle reminder to transfer the monthly subscription fee to the subscriber. Thanks!'
                }).catch(error => { console.log(error) })
            })

        }
    })
}

A few things to note:

I used the Telegram Bot API NPM package for calling the Bot APIs.
I zipped the npm package together with my Lambda code before I uploaded it.
I used the DDB_TABLE_NAME environment variable for my Lambda function to know which table to call. Assign your DynamoDB Table to this variable.
I used the TG_API_KEY environment variable for my Lambda function to know which Bot API to call. Assign your Bot token to this variable.
Remember to give your Lambda functions the right IAM permissions to call DynamoDB.

Step 4: Configure CloudWatch Events

At the CloudWatch console page, under Events, create a new Event with the cron expression you want to trigger your respective Lambda functions.

I wanted to consolidate registration once a day and send messages once a month, so I used the following cron expressions:

Daily register: cron(0 10 * * ? *)
Monthly sendMessage: cron(0 11 1 * ? *)

Head over to the Lambda console page and choose “+ Add trigger”, and choose EventBridge (CloudWatch Events) and select the CloudWatch events you have created.

Summary

There are still a lot of room for improvement for this simple Telegram bot I hacked together over the weekend. A quicker feedback during bot registration using webhooks is a future improvement I would want to make.

I hope you had fun with this little project. Reach out to me if you have any feedback.

How I passed Certified Kubernetes Administrator (CKA) Exam (quick tips)

Zhen Kai — Mon, 04 Oct 2021 13:05:43 +0000

Exam Score: 93. Passing score: 66.

4-6 weeks before Exam

Since CKA is an "open book" exam, you should be preparing with the mindset of knowing where to find stuff in the Kubernetes documentation.

Mumshad Mannambeth CKA course

If you are stuck while doing the practice questions after each chapter, dive into the official Kubernetes documentation to find your answer.
Don't rely on the course content to find your answers. Use it to understand concepts.
Begin bookmarking documentation pages you find helpful.
Run through the jsonpath practices.
After finishing the Killer.sh simulators, come back this course to run through as many practice questions as you can.

Killer.sh CKA simulator

Comes with 2 free tries after you've registered for the CKA exam
Only use official Kubernetes documentation to find your answers
Take the opportunity to bookmark documentation pages you find helpful
Do not panic if you find yourself struggling, I did too. The simulator is very tough.
Practice setting up your Vimrc configuration, Kubectl autocomplete and alias.

Administrative stuff

Register for the CKA exam
Purchase a magnifying glass if you don't have one, in case you have issues with your webcam not being able to focus on your identification proof clearly. I faced this issue personally.
Ensure you have read through the Candidate Handbook and Important Instructions.
Buy a decent webcam if you do not have one.
Get a second monitor (or a monitor extension for your laptop). This is extremely helpful and reduces time required to alternate between windows.
Run the compatibility check tool and make sure everything is good.

Less than 1 Day Before Exam

Clear out your desk, including speakers. You don't need speakers for the exam.
Get a clear cup for water
Run the compatibility check tool again and make sure everything is good.
Go through the entire Kubernetes Cheat Sheet and get familiar with every line in it.

During the Exam

Always execute the switch context command displayed at every question before attempting the question, even if you think you are already on the correct context.
Do not get stuck on a question. Mark/Flag it and move on to the next.
Open a second browser window with a single tab on your second monitor, with all the bookmarks you need on the bookmark bar. Remember to let the proctor know that you have a second monitor.

Bonus: My List of Bookmarks

These bookmarks have also been named according to what you see in the list.

After the Exam

Results usually come within 24 hours. For me, it was published almost exactly 24 hours later.

Good luck and all the best!

Default Tags for Terraform AWS Provider is finally here

Zhen Kai — Sun, 16 May 2021 10:47:01 +0000

Default tags for Terraform AWS provider had been in the works since I started using Terraform a couple of years ago. Now that it is finally here, does it live up to its expectation?

What is Default Tags?

Default Tags allows its user to tag all AWS resources that support tags with the exception of the aws_autoscaling_group resource. Default Tags are defined in the AWS provider configuration, and AWS resources configured with that provider will inherit default_tags. A sample usage could be:

provider "aws" {
  region = "ap-southeast-1"
  default_tags {
    tags = {
      Environment = "dev"
      Project     = "example project"
      Terraform   = true
  }

You can read more about it here and here.

Default Tags in different scenarios

From the official blog post, it states that individual resource tags will take precedence over default_tags. Let’s see it in action with a few scenarios and observe its terraform plan output.

Scenario 1: Only resource tags, no `default_tags`

provider "aws" {
  region = "ap-southeast-1"
}
resource "aws_vpc" "tagging_test_vpc" {
  cidr_block = "10.0.0.0/16"
  tags = {
    Name        = "tagging-test"
    Environment = "dev"
    Terraform   = true
  }
}

terraform plan output:

Here, we can see the existence of a tags_all attribute that has the same value as the original tags attribute. That being said, I’ve noticed this tags_all attribute for some time now in the newer versions of AWS provider. Let’s move on to the next scenario.

Scenario 2: Only `default_tags`, no resource tags

provider "aws" {
  region = "ap-southeast-1"
  default_tags {
    tags = {
      Environment = "stg"
      Name        = "tagging-test-3"
  }
}
resource "aws_vpc" "tagging_test_vpc" {
  cidr_block = "10.0.0.0/16"
}

terraform plan output:

Without resource tags, the tags attribute disappears, leaving only the tags_all attribute with the value inherited from default_tags from the AWS provider configuration.

Scenario 3: Conflicting tags

provider "aws" {
  region = "ap-southeast-1"
  default_tags {
    tags = {
      Environment = "stg"
      Name        = "tagging-test-3"
  }
}
resource "aws_vpc" "tagging_test_vpc" {
  cidr_block = "10.0.0.0/16"
  tags = {
    Name        = "tagging-test"
    Environment = "dev"
    Terraform   = true
  }
}

terraform plan output:

Similar to what the official blog post mentioned, we can observe that resource tags take precedence over default tags when there is a conflict of tag name/key.

Scenario 4: Non-conflicting tags

provider "aws" {
  region = "ap-southeast-1"
  default_tags {
    tags = {
      environment   = "stg"
      name          = "tagging-test-3"
      provisionedby = "Terraform"
  }
}
resource "aws_vpc" "tagging_test_vpc" {
  cidr_block = "10.0.0.0/16"
  tags = {
    Name        = "tagging-test"
    Environment = "dev"
    Terraform   = true
  }
}

terraform plan truncated output:

Since tag names/keys are case-sensitive, all the default_tag tags in this example will be effected for aws_vpc. The tags_all attribute shows a combined list from default and resource tags, while the tags attribute only shows the resource tags.

How this could affect your Terraform State design

In a medium-to-large-sized project, Terraform state separation is one of the early design considerations a team needs to settle on. If you don’t know what state is, check out this documentation first.

From HashiCorp

Each Terraform state culminates in a state file, and each state file has its own Terraform and provider configurations. Since default tags are a configured on the AWS provider level, being able to enforce different default tags in different states can make any mandatory organizational tagging requirements a breeze.

For example, if your organization has different tag values for each application, it might make sense to separate all applications into it’s own state and enforce their unique application tags at the provider level.

Terraform state design is a topic I’ve always seen a lot of discussion about. I will write about it in a future post.

Closing Thoughts

It’s good to know that Terraform and the AWS Provider is still well supported and constantly improved on. All in all, Terraform as an IaC tool is reaching a level of maturity that I’ve been looking forward to. With the current Terraform 0.15 touted as the “beginning of the pre-release period leading up to Terraform 1.0”, I think Terraform users can expect a lot of exciting and useful releases in the coming weeks and months.

The Best Git branching strategy for Terraform is no branching

Zhen Kai — Sat, 08 May 2021 03:07:01 +0000

disclaimer: Opinions are my own and not the views of my employer

We all know that code should be checked into version control. Terraform code is no different.

There are so many Git branching strategies available out in the wild, such as the popular Git flow and Github flow.

Git Flow from nvie.com

What is Terraform State?

Terraform requires some sort of database to map Terraform config to the real world. When you have a resource resource “aws_instance” “foo” in your configuration, Terraform uses this map to know that instance i-abcd1234 is represented by that resource.
Purpose of State from Terraform Documentation

When collaborating on Terraform code, HashiCorp recommends using Remote State to store State files remotely, outside of Version Control (Git). All team members need to have access to this remote state storage if they want to test or apply their code contributions.

Feature Branch Problem

When feature branches are worked on for a short period of time before merging into the main branch, there are usually no huge integration issues.

However, in the age of Continuous Integration/Continuous Delivery (CI/CD), multiple new features could be worked on in parallel and delivered whenever they are ready. This creates a problem for Terraform and Terraform State management.

Imagine this situation: Alice and Bob are collaborating on a Terraform code repository with the same development remote backend using Amazon S3. Alice has a new requirement to introduce load balancing capabilities, so she checks out a new feature branch to work on it.

# existing resources
resource "aws_instance" "app_server" {
  ami = "ami-0518bb0e75d3619ca"
  ....
}

# new load balancer resource by Alice
resource "aws_lb" "alice_test_lb" {
  subnet_mapping {
  ...
}

Typically with feature branches, Alice would apply her feature branch changes and provision the load balancer. Terraform state in the remote backend would be updated with Alice’s load balancer.

At the same time, Bob also checks out another new feature branch to modify the existing EC2 instance to test a new AMI as part of immutable infrastructure operations.

# existing resources
resource "aws_instance" "app_server" {
  ami = "ami-0fab0953c3bb514a9" # new AMI
  ....
}

When Bob is ready to test and apply his change, Terraform assesses that Alice’s load balancer had been removed in Bob’s code, generating a planned action to destroy Alice’s load balancer and recreate the existing EC2 instance with the new AMI ID.

This is highly undesirable and an absolute nightmare for integration testing and Terraform plan/apply.

1 State File, 1 Branch

Live Infra Terraform code should represent the live infrastructure provisioned and Terraform state files record the details of those live infrastructure. Every single Terraform state file should have exactly 1 source of truth and should only reference to exactly 1 Git branch.

Avoiding Integration Hell

I first got to realize the problems of feature branches when stumbling across Rod Hilton’s blog post.

Two engineers are happily working away making commit after commit to their own respective feature branches, but neither of their branches are seeing the other’s code. Even if they’re regularly pulling off mainline, they’re still only seeing the commits that make it into the main branch, not each others. Developer A merges their code into mainline, then Developer B pulls and merges theirs, but now they have to deal with tons of merge conflicts. Developer B might not be in the best position to understand and resolve those conflicts if they don’t fully understand what Developer A is doing, and depending on how long these branches have been alive, they might have tons of code to resolve.
A Branching Strategy Simpler than GitFlow: Three-Flow by Rod Hilton

Feature branches provides little value when it comes to integration testing and even more so when Terraform doesn’t work when there are multiple branches to modifying a single state file.

Commit Directly to Development Branch

The simplest way to prevent Terraform code from being Terraform applied on feature branches is to do away with feature branches completely.

In the age of Continuous Integration, pushing and pulling changes to and from the main development branch frequently avoids the inevitable integration hell of merging multiple weeks-old feature branches.

Instead of feature branches, make use of feature toggles to “hide” your features in plain sight until they are ready for testing. Terraform has good support of feature toggle implementation through the usage of count, for_each and conditional expressions.

Using the previous example, Alice and Bob could’ve pushed their code directly into the development branch with feature toggles and it would like something like this:

# feature toggles
locals {
  lb_feature  = false # change to true when ready
  ami_feature = false # change to true when ready
}

# existing resources
resource "aws_instance" "app_server" {
  ami = local.ami_feature ? "ami-0fab0953c3bb514a9" : "ami-0518bb0e75d3619ca"
  ....
}

# new load balancer resource by Alice
resource "aws_lb" "alice_test_lb" {
  count = local.lb_feature ? 1 : 0
  subnet_mapping {
  ...
}

Some good examples of Git branching strategies that do away with feature branches are the Three-Flow model, Cactus model and Trunk based development. I highly recommend to consider these Git branching strategies for your next Terraform project collaboration.

A Branching Strategy Simpler than GitFlow: Three-Flow by Rod Hilton

Some Caveats

Merge Directly into Dev Branch only

Merging directly from local into the main branch should only be for the development environment. Production and staging environments should have a different merge and release processes like some of the Git branching examples I’ve shared in the previous section. I am a strong advocate of having 1 branch per environment and I’ll share more on this in a future blog post.

Code Review Processes

Merging feature code directly into the development/main branch meant that the typical code review processes with pull requests wouldn’t be possible. If code reviews are a necessity, use short-lived feature branches solely for the purpose merging code. You can consider shifting your team’s code review processes to before releasing to your staging environment.

Terraform v0.12

Usage of count and for_each for feature toggles presents some challenges if your team is still on an older Terraform version. Modules do not have native count and for_each support in v0.12 and should be something to keep in mind. You would have to bake the feature toggle into the module itself by passing in true or false values to decide whether to create the resources within the modules. It’s not ideal, but still doable.

Native Module count and for_each is supported in Terraform v0.13 and higher.

Collaborators must be Trusted

To commit directly into the main branch, all collaborators must be on the same team and trusted. This Git branching strategy is not suitable for public repositories.

Closing Thoughts

I’d like to end this lengthy blog post with this quote from Rod Hilton.

A developer’s primary form of communication with other developers is source code. It doesn’t matter how often you have stand-up meetings, when it comes to the central method of communication, long-running branches represent dead silence.
A Branching Strategy Simpler than GitFlow: Three-Flow by Rod Hilton

Thank you for taking the time to read this. Reach out to me if you would like to discuss further.

Passionate about Cloud and Terraform

Zhen Kai — Sun, 25 Oct 2020 02:51:55 +0000

Test post

Forem: Zhen Kai

New AWS Management Console Home is a Step in the Right Direction

Old console home page

Lack of Customization

No Consolidated Dashboard

Bloat (for experienced users)

What's on the new Console Home?

What I want to see in the future?

Closing Thoughts

AWS VPC IPAM Tutorial - The Better Terraform Way (7 Steps Faster)

The Better Terraform Way

Step 1: Clone Github repository

Step 2: Terraform Apply

Step 3: Terraform Destroy

Bonus: Code Walkthrough

Variables

Service-Linked Role

IPAM and it's operating regions

Top-level Pool and CIDR assignment

Sub-level Pool and CIDR assignment

VPC to test

Closing

AWS VPC IPAM Basics and Why You Need to be Careful

Why VPC IPAM?

AWS VPC IPAM basics

[1] Service-Linked Role and IPAM Administration

[2] IPAM

[3] Scope (Private & Public)

[4] Pool (Top-Level)

[5] Pool (Sub-Level)

[6] Pool sharing and IP allocation

[7] IPAM managed and monitored Resources

You should be careful of...

Costs

Closing Thoughts & Whether You Should use VPC IPAM

Simple Serverless Telegram Bot with AWS Lambda

Requirements

Architecture

How the Bot works

How to Configure the Bot

Step 1: Create a Telegram Bot with Botfather

Step 2: Create a DynamoDB Table

Step 3: Create Lambda functions (register and sendMessage)

Step 4: Configure CloudWatch Events

Summary

How I passed Certified Kubernetes Administrator (CKA) Exam (quick tips)

4-6 weeks before Exam

Mumshad Mannambeth CKA course

Killer.sh CKA simulator

Administrative stuff

Less than 1 Day Before Exam

During the Exam

Bonus: My List of Bookmarks

After the Exam

Default Tags for Terraform AWS Provider is finally here

What is Default Tags?

Default Tags in different scenarios

Scenario 1: Only resource tags, no default_tags

Scenario 2: Only default_tags, no resource tags

Scenario 3: Conflicting tags

Scenario 4: Non-conflicting tags

How this could affect your Terraform State design

Closing Thoughts

The Best Git branching strategy for Terraform is no branching

What is Terraform State?

Feature Branch Problem

1 State File, 1 Branch

Avoiding Integration Hell

Commit Directly to Development Branch

Some Caveats

Merge Directly into Dev Branch only

Code Review Processes

Terraform v0.12

Collaborators must be Trusted

Closing Thoughts

Passionate about Cloud and Terraform

Scenario 1: Only resource tags, no `default_tags`

Scenario 2: Only `default_tags`, no resource tags