Forem: Robert Nemet

Automating Pull Request Reviews With CodiumAI PR-Agent

Robert Nemet — Mon, 11 Dec 2023 17:04:09 +0000

Motivation

If you are a developer, you know how much time you spend on pull request reviews. We all try to write the best code we can. When we are done, we ask our team members to review our code. Sometimes, making a pull request is more complex than it sounds. Over time, teams learn that pull requests need to be standardized to speed things up and improve communication. That is why we have a pull request template, for example. Sometimes, even that is not enough. Because the speed at which the code is sent out is increasing. More pull requests are created. And the number of pull requests that need to be reviewed is expanding. Ah...

Question: In the era of AI, can we automate pull request reviews? At least it can dissect a pull request and give us some insights.

Let's do this with some small pull requests.

Looking for the right tool

I need a tool or AI assistant that can do this for me. Obviously, as I have my code on GitHub, I need a tool that can integrate with GitHub. So, the first choice was GitHub's Copilot for pull requests. And it is not free. It is still in beta, and you need to sign into the waitlist to access it. I'm not so patient, so I searched further.

Next, I found a solution CodiumAI PR-Agent. It is free and open-source. Just from its description, it looks like something that can fit my needs. I decided to give it a try.

Installation & usage

Requirements

Since I have my remote repository on GitHub to enable CodiumAI PR-Agent to access my code and do its magic, I need to create a GitHub token. Also, CodiumAI PR-Agent uses OpenAI API, which means I must also have an OpenAI API key. So, before you start, make sure you have both of them.

How to create a GitHub token

I hope you already have a GitHub account. If not, create one. Then follow these steps:

Go to your GitHub account settings
Click on Developer settings
Click on Personal access tokens
Click on Generate new token
Give your token a name and select scopes
Click on Generate token
Copy your token and save it somewhere safe
You will need it later

Or you can follow this link for more detailed instructions.

How to create OpenAI API key

Go to OpenAI and create an account
Go to your account settings
Click on API keys
Click on Create a new API key
Copy your API key and save it somewhere safe
You will need it later

Or go to this link to add an API key.

How to use Codium PR-Agent from CLI

Setup

There are two ways to do this: Docker or a Python virtual environment. I'll be using Docker. So, to start using Codium PR-Agent from CLI, first set the above-received API keys as environment variables. You can do this by running these commands:

export GITHUB_TOKEN=<your_github_token>
export OPENAI_API_KEY=<your_openai_api_key>

Open a pull request on your GitHub repository. Copy the URL of your pull request and export it as an environment variable:

export PR_URL=<your_pr_url>

Then you can run Codium PR-Agent from CLI by running this command:

docker run --rm -it -e OPENAI.KEY=${OPENAI_API_KEY} -e GITHUB.USER_TOKEN=${GITHUB_TOKEN} codiumai/pr-agent:latest --pr_url ${PR_URL} <command>

Even better would be making an alias for this command:

alias pragent='docker run --rm -it -e OPENAI.KEY=${OPENAI_API_KEY} -e GITHUB.USER_TOKEN=${GITHUB_TOKEN} codiumai/pr-agent:latest --pr_url ${PR_URL}'

Now you can run Codium PR-Agent from CLI by running this command:

pragent <command>

Usage

Codium PR-Agent has several commands. Let's start with ones related to pull requests.

Generate Pull Request Description

Let's first start with generating a pull request description. To do this, run this command:

$ pragent describe --keep_original_user_title=true --add_original_user_description=true

...

This command will generate a description for your pull request and add it to your pull request. I want to keep your original description and title.
I added --keep_original_user_title=true and --add_original_user_description=true flags.
You can omit these flags if you don't want to keep your original title and description.

And the results are surprisingly good but also a bit disappointing. Let me explain why. The description, in my case, is 100% accurate. It describes my changes in three blocks: change type, description, and the pull request walkthrough. See the image below.

It even added a label to pinpoint what this pull request is about. Which is nice. This is opening some other possibilities. I would like to process a pull request based on its labels. Next time about that.

But it did change the original title and removed the original description, even though I told it not to!? These two options should be set by default and not vice versa. And Codium PR-Agent should add its description and title to the original ones. Why? First, the person who created the pull request should have written the title and description to add some context and motivation, add a link to a ticketing system, etc. But ok, I can live with that at the moment. In the image above, I marked the yellow parts updated by Codium PR-Agent: removed and updated the title and description. And with green parts that are added as I expected: the label.

Generate Pull Request Review

Ok, let's move on to the next command. This command will generate a pull request review. To do this, run this command:

$ pragent review

...

Using the default configuration gives me nice results. Everything that needs to be addressed in a green box is nicely marked. The yellow box is why I wanted to keep the original title and description. The CodiumAI PR-Agent should just add its description next to the original one.

Give Me Some Suggestions

Let's see what Codium PR-Agent can suggest to me. Since changes are simple, I'm going with the default configuration. To do this, run this command:

$ pragent improve

...

I have got three suggestions. Above, you just see one. Even if the cases are straightforward, suggestions are on point as clean code suggestions. I can accept or reject them as in any other pull request.

Generate Changelog And Update Docs

These two commands are interesting to me. Let's start with the changelog. To do this, run this command:

$ pragent update_changelog

...

Running with the default configuration gives me a nice changelog. Even if I did not have one, it created one for me. However, it did not update the pull request. The reason is apparent: I did not instruct it to do so. But CodiumAI PR-Agent is smart enough to tell me so and give me instructions on how to do it. So I do it:

$ pragent update_changelog --pr_update_changelog.push_changelog_changes=true

...

Excellent, let us try updating docs:

$ pragent add_docs

...

They are added as code improvements. I can accept them or reject them as in any other pull request. Nice.

Troubleshooting

You can find documentation on CodiumAI PR-Agent GitHub repository is quite good. You'll find most of your answers there. I want to point to one thing related to giving the correct permissions to CodiumAI PR-Agent when creating a GitHub API token. Which permissions you need to give depends on what you want to do with it.

For any integration, you should give CodiumAI PR-Agent only the permissions it needs to do its job. If you give it less permissions than it needs, it will not work. I'm fond of the principle of least privilege. So, I gave it only the permissions it needed to do its job.

How will you now? Use the CLI and see what it says. Here is an example:

$ pragent update_changelog --pr_update_changelog.push_changelog_changes=true

2023-12-10 18:20:45.902 | INFO     | pr_agent.algo.utils:update_settings_from_args:275 - Updated setting PR_UPDATE_CHANGELOG.PUSH_CHANGELOG_CHANGES to: "True"
2023-12-10 18:20:47.768 | INFO     | pr_agent.tools.pr_update_changelog:_get_changlog_file:152 - No CHANGELOG.md file found in the repository. Creating one...
Traceback (most recent call last):
  File "/app/pr_agent/tools/pr_update_changelog.py", line 144, in _get_changlog_file
...snip...
github.GithubException.UnknownObjectException: 404 {"message": "Not Found", "documentation_url": "https://docs.github.com/rest/repos/contents#get-repository-content"}

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/app/pr_agent/cli.py", line 68, in <module>
...snip...
github.GithubException.GithubException: 403 {"message": "Resource not accessible by personal access token", "documentation_url": "https://docs.github.com/rest/repos/contents#create-or-update-file-contents"}

I did not give enough permissions to CodiumAI PR-Agent to update the changelog. So, it failed. I gave it the proper permissions, and it worked. Simple as that.
I suggest you run all your scenarios with CLI first. It will save you a lot of time.

How to use Codium PR-Agent as GitHub Action

Now that I have tried my scenarios with CLI, I want to automate them. I want to use Codium PR-Agent as GitHub Action.

Setup CodiumAI PR-Agent GitHub Action

Thanks to the nice documentation on CodiumAI PR-Agent GitHub repository, I do
this with ease. I just need to add a new workflow file to my repository. I named it codium-pr-agent.yml. Here is the content of the file:

name: PRAgent
on:
  pull_request:
    branches: ["*"]
  issue_comment:
    types: [created]

jobs:
  pr_agent_job:
    runs-on: ubuntu-latest
    permissions:
      issues: write
      pull-requests: write
      contents: write
    name: Run pr agent on every pull request, respond to user comments
    steps:
      - name: PR Agent action step
        id: pragent
        uses: Codium-ai/pr-agent@main
        env:
          OPENAI_KEY: ${{ secrets.OPENAI }}
          GITHUB_TOKEN: ${{ secrets.API_TOKEN }}
          pr_description.use_description_markers: "true"
          pr_description.include_generated_by_header: "true"
          github_action.auto_review: "true"
          github_action.auto_describe: "true"
          github_action.auto_improve: "true"
          pr_update_changelog.push_changelog_changes: "true"

Also, I added a pull request template to my repository in a .github/pull_request_template.md. Here is the content of the file:

## Describe your changes

## Issue ticket number and link

## Checklist before requesting a review
- [ ] I have performed a self-review of my code
- [ ] If it is a core feature, I have added thorough tests.
- [ ] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation

## PR Description

pr_agent:summary

## PR Walkthrough

pr_agent:walkthrough

And remember to add your API keys as secrets to your repository. I named them OPENAI and API_TOKEN. Add them to your repository by going to your repository's settings and then to Secrets. Click on Add a new secret and add your API keys. Read more on GitHub secrets here.

For additional CodiumAI-PR-Agent configuration options, you can check CodiumAI PR-Agent GitHub repository.

The above configuration will run CodiumAI PR-Agent on every pull request, use the pull request template to generate a pull request description, create a pull request review, and give me some suggestions. Nice. I got all I wanted. Next time a pull request is made on my repository, CodiumAI PR-Agent will do its magic. I can quickly do future reviews.

Gotchas

Even if all sounds nice, there is one gotcha. That is OpenAI. It is not free, and you'll need to pay for it. To reduce costs, you need to spend some time configuring CodiumAI PR-Agent, which does only what you need on a limited set of files, primarily if you use code generation.

Conclusion

CodiumAI PR-Agent is an excellent tool. It can help you with your pull requests. It can generate a pull request description, review, give you some suggestions, update
changelog and docs, and chit-chat with it if you want. See this nice /ask command and its answer. Check it yourself and see what it can do by setting up scenarios. It is worth it trying it.

I hope you enjoyed this article. If you have any questions or comments, please leave them below. I'll be happy to answer them.

Exploring GCP With Terraform: Adding Terragrunt

Robert Nemet — Wed, 20 Sep 2023 08:44:36 +0000

Finally, this is the last step in making the Terraform project manageable. I'll add Terragrunt to the project. Terragrunt is a thin wrapper for Terraform that provides extra tools for working with multiple Terraform modules. It's an excellent tool for making your Terraform code DRY and reusable.

I'm starting with the same project as in the previous post. The current state of code is available here in the branch add_modules.

Prerequisites

Well, you need to install Terragrunt. You can find installation instructions here. That is all.

Adding Terragrunt to the back-office network and vms modules

I'm starting with the back-office module. What I do in this module will be easily copied to other modules.

Notice that common code in back-office/vms and back-office/network is related to the provider and state file configurations. This code is common for all modules. The only thing that differs is the backend.prefix. Its value differs from module to module.

So, the first thing is to make a global configuration on the dev level. To do that, I'm creating a terragrunt.hcl file in the dev folder:

generate "provider" {
    path = "provider.tf"
    if_exists = "overwrite_terragrunt"
    contents = <<EOF
provider "google" {
  project = var.project_id
  region  = var.region
  zone    = var.zone
}

terraform {
  required_version = ">=1.5.7"

  required_providers {
    google = {
      source  = "hashicorp/google"
      version = "4.77.0"
    }
  }
}
EOF
}

remote_state {
    backend = "gcs"
    generate = {
        path = "backend.tf"
        if_exists = "overwrite_terragrunt"
    }
    config = {
        bucket = "terraform-states-network-playground-382512"
        prefix = "terraform/state/dev/${path_relative_to_include()}"
    }
}

terraform {
    extra_arguments "common" {
        commands = get_terraform_commands_that_need_vars()

        arguments = [
            "-var-file=${path_relative_from_include()}/common.tfvars",
        ]
    }
}

This file has three sections. The first section is generate "provider". This section will generate a provider.tf file in each module. Suppose it already exists and is managed by Terragrunt. In that case, it will be re-generated if there are any changes to the configuration. This file will contain provider configuration.

The second section is remote_state. This section will generate a backend.tf file in each module. This file will contain state file configuration. Notice that prefix is set to terraform/state/dev/${path_relative_to_include()}. This setting will store the state file in the terraform/state/dev folder, and each module will have its subfolder. That is important because each module will have its state file.

The third section is terraform. This section will add extra arguments to Terraform commands. In this case, it will add -var-file=${path_relative_from_include()}/common.tfvars. So, when I run any Terraform command requiring variables, Terragrunt will add this argument. That allows me to have one common.tfvars file in the dev folder, and all modules will use it.

Now, I need to add the common.tfvars file to the dev folder:

project_id = "network-playground-382512"
region = "europe-west1"
zone = "europe-west1-b"

I need to add a terragrunt.hcl file to each module to use this. I'm adding it to the back-office module:

include {
    path = find_in_parent_folders()
}

Now, I can remove the provider.tf and terraform.tfvars files from the back-office module. This time, I'll be running terragrunt plain instead of terraform plan in the back-office module:

terragrunt plan
Acquiring state lock. This may take a few moments...
google_service_account.back_office_fw_sa: Refreshing state... [id=projects/network-playground-382512/serviceAccounts/back-office@network-playground-382512.iam.gserviceaccount.com]
module.back-office.module.vpc.google_compute_network.network: Refreshing state... [id=projects/network-playground-382512/global/networks/back-office]
module.back-office.module.subnets.google_compute_subnetwork.subnetwork["us-central1/back-office-private"]: Refreshing state... [id=projects/network-playground-382512/regions/us-central1/subnetworks/back-office-private]
module.back-office.module.subnets.google_compute_subnetwork.subnetwork["us-central1/back-office"]: Refreshing state... [id=projects/network-playground-382512/regions/us-central1/subnetworks/back-office]
module.back-office.module.firewall_rules.google_compute_firewall.rules_ingress_egress["back-office-icmp"]: Refreshing state... [id=projects/network-playground-382512/global/firewalls/back-office-icmp]
module.back-office.module.firewall_rules.google_compute_firewall.rules_ingress_egress["back-office-iap"]: Refreshing state... [id=projects/network-playground-382512/global/firewalls/back-office-iap]

No changes. Your infrastructure matches the configuration.

Terraform has compared your real infrastructure against your configuration and found no differences, so no changes are needed.

Running the terragrunt plan in the back-office/network module will generate two files: provider.tf and backend.tf, according to the terragrunt.hcl file in the dev folder.

For the back-office/vms module, I need to add the terragrunt.hcl file to the back-office/vms folder:

include "root" {
  path = find_in_parent_folders()
}

dependencies {
  paths = ["../network"]
}

dependency "network" {
  config_path = "../network"
}

inputs = {
  vpc_back_office_id = dependency.network.outputs.vpc_back_office_id
  vpc_back_office_subnetwork = dependency.network.outputs.vpc_back_office_subnetwork
  back_office_fw_sa = dependency.network.outputs.back_office_fw_sa
}

Now, this terragrunt.hcl not only includes files from parent folders but also defines dependency on the ../network module. That means that the back-office/vms module depends on the back-office/network module. A network needs to be created before virtual machines. That is important because the back-office/vms module needs to know some outputs from the back-office/network module. The section inputs defines inputs for the back-office/vms module. These inputs are outputs from the back-office/network module. The last thing allows me to remove in addition to provider.tf and terraform.tfvars, and inputs.tf files from the back office/vms module.

I need to add additional variables to the variables.tf file in the back-office/vms:

variable "back_office_fw_sa" {
  description = "back office firewall service account"
  type        = string
}

variable "vpc_back_office_id" {
  description = "back office vpc id"
  type        = string
}

variable "vpc_back_office_subnetwork" {
  description = "back office vpc subnetwork"
  type        = any
}

Running terragrunt plan should give me no changes needed. Also, I can now run terragrunt apply instead of terraform apply in back-office/vms module.

What I did in the back-office module, I need to do in the service and strorage modules. Simple. Remember to properly define dependencies and inputs in other vms modules.

When it comes to the peering module, I need to add the terragrunt.hcl file to the peering folder:

include "root" {
  path = find_in_parent_folders()
}

dependencies {
  paths = ["../back-office/network", "../services/network", "../storage/network"]
}

dependency "back-office" {
  config_path = "../back-office/network"
}

dependency "services" {
  config_path = "../services/network"
}

dependency "storage" {
  config_path = "../storage/network"
}

inputs = {
  vpc_back_office = dependency.back-office.outputs.vpc_back_office
  vpc_services = dependency.services.outputs.vpc_services
  vpc_storage = dependency.storage.outputs.vpc_storage
}

That will ensure that the peering module will be created after the back-office, services, and storage modules. Also, it will provide inputs for the peering module. These inputs are outputs from back-office, services, and storage modules.

Interesting would be generating a graphical representation of the dependencies:

terragrunt graph-dependencies | dot -Tpng > dependencies.png

The terragrunt graph-dependencies command will generate a graph of the dependencies between modules. The dot -Tpng > dependencies.png command will convert the graph to a PNG file, that is graphviz. That is a great way to visualize dependencies between modules.

Try from dev folder terragrunt run-all plan. The Terragrunt will split the modules into groups. In each group, it will run plan, but first, it will run it in network modules because they are dependencies for other modules. Then, it will run plan in parallel for other modules in the other group.

Conclusion

Adding Terragrunt to the project is relatively easy. It requires some changes to the project structure and the code. But it is worth it. It makes the project more manageable: code is DRY, and common code is generated in one place. I can run Terraform commands from the root folder and do not need to visit each module, etc...

References

Terragrunt

GCP With Terraform: Refactor with Modules

Robert Nemet — Wed, 13 Sep 2023 20:51:33 +0000

This post is the fourth part of the series about using Terraform to manage GCP resources. In the first part, I did a basic setup of the project: remote state file, state file encryption, a bucket creation. In the second part, I created VPC and subnets and added some basic firewall rules and VMs. In the third part, I added more VPC, subnets, firewall rules, and VMs. In this part, I will refactor the project to make it more manageable. It should be more DRY and easier to maintain while supporting multiple environments.

Current State

You can look at the project's current state here. The existing file structure is like this:

├── README.md
├── Taskfile.yml
└── gcp
    ├── base
    │   ├── .terraform-version
    │   ├── .terraform.lock.hcl
    │   ├── README.md
    │   ├── main.tf
    │   ├── outputs.tf
    │   ├── provider.tf
    │   ├── terraform.tfvars
    │   └── variables.tf
    ├── network
    │   ├── .terraform-version
    │   ├── .terraform.lock.hcl
    │   ├── README.md
    │   ├── main.tf
    │   ├── outputs.tf
    |   ├── peering.tf
    │   ├── provider.tf
    │   ├── terraform.tfvars
    |   ├── vpc_back_office.tf
    │   ├── vpc_datastorages.tf
    |   ├── vpc_services.tf
    │   └── variables.tf
    └── vms
        ├── .terraform-version
        ├── .terraform.lock.hcl
        ├── README.md
        ├── main.tf
        ├── back_office.tf
        ├── imports.tf
        ├── outputs.tf
        ├── provider.tf
        ├── terraform.tfvars
        └── variables.tf

The base directory contains the basic setup of the project. It creates a bucket for the remote state file, enabling the state file's encryption. The network directory contains the VPC, subnets, and firewall rules. The vms directory contains the VMs. The Taskfile.yml is used to run the tasks. The README.md contains the documentation of the project.

There are several problems with this setup:

Modules network and vms handle more than they should. For example, the network module creates VPC, subnets, and firewall rules for all VPCs. It should handle only one VPC. This way, whenever I make a change in one VPC when I run terraform plan/apply in the module network other VPCs will be touched. If I have to work with others, this is the possible bottleneck. The vms module is creating VMs for all VPCs.
There is a lot of code duplication.
This structure does not support multiple environments. I can't have a dev and prod environment with this setup.
Changing one module requires running terraform plan/apply in all modules(not always). For example, when adding a service account to the VMs, I had to run terraform plan/apply in the network and vms modules. It is not a big deal, but it is annoying.

Step one: Split into environments

The first step is to split the project into environments. I will create two environments: dev and prod. I will create two directories: dev and prod on the project level. Then, I will copy whole gcp directory into both dev and prod directories.

.
├── README.md
├── Taskfile.yml
├── dev
│   ├── base
│   ├── network
│   └── vms
└── prod
    ├── base
    ├── network
    └── vms

Why? With this change, I'm reducing the impact of changes. While in an ideal case, these environments should be identical, they are not. To be honest, you do not need the same level of resources in dev and prod.

Step two: Create a module for each VPC

The network module is doing too much. It is creating VPC, subnets, and firewall rules for all VPCs. It will be much better if it makes only one VPC, subnets, and firewall rules for that VPC. But the same thing applies to the vms module. It is creating VMs for all VPCs. It will be much better if it makes VMs for only one VPC.

If I put network stuff and VMs in the same folder, which will then contain network and vms modules, for each VPC, I can create a VPC, subnets, firewall rules, and VMs for that VPC. This way, I'm separating the network stuff and VMs for each VPC. But, at the same time, I'm keeping them together. This way, I'm reducing the impact of changes. So, my new file structure looks like this:

.
├── README.md
├── Taskfile.yml
├── dev
│   ├── base
│   ├── back-office
│   │   ├── network
│   │   └── vms
│   ├── services
│   │   ├── network
│   │   └── vms
│   └── storage
│       ├── network
│       └── vms
└── prod
    ├── base
    ├── back-office
    │   ├── network
    │   └── vms
    ├── services
    │   ├── network
    │   └── vms
    └── storage
        ├── network
        └── vms

In addition to this change, I'll keep code related to the target VPC. This change simplifies the code, increases readability, and reduces the impact of changes. But at the same time, it increases the number of modules, and there is an increase in code duplication. Before dealing with code duplication, I'll ensure the code is working and has the same
structure.

What does that mean? Each module will have the same structure: main.tf, outputs.tf, provider.tf, terraform.tfvars, and variables.tf. The main.tf will contain the code for creating resources. The outputs.tf will contain the outputs of the module. The provider.tf will contain the provider configuration. The terraform.tfvars will contain the variables for the module. The variables.tf will contain the variables for the module.

Refactoring the base module

This module is the most straightforward module to refactor. There is little to change. Except the where the state file is stored. It's done by changing the backend.prefix in:

terraform {
  required_version = ">=1.5.5"

  required_providers {
    google = {
      source  = "hashicorp/google"
      version = "4.77.0"
    }
  }

  backend "gcs" {
    bucket = "terraform-states-network-playground-382512"
    prefix = "terraform/state/dev/base"
  }
}

The state file for the base module will be in the terraform/state/dev/base directory. The previous state file was in the terraform/state/base. How do I move the state file? Manually? It can be done. But there is a better way. If I run the terraform plan in the base module, I will get the following output:

$ terraform plan
╷
│ Error: Backend initialization required: please run "terraform init"
│
│ Reason: Backend configuration block has changed
│
│ The "backend" is the interface that Terraform uses to store state,
│ perform operations, etc. If this message is showing up, it means that the
│ Terraform configuration you're using is using a custom configuration for
│ the Terraform backend.
│
│ Changes to backend configurations require reinitialization. This allows
│ Terraform to set up the new configuration, copy existing state, etc. Please run
│ "terraform init" with either the "-reconfigure" or "-migrate-state" flags to
│ use the current configuration.
│
│ If the change reason above is incorrect, please verify your configuration
│ hasn't changed and try again. At this point, no changes to your existing
│ configuration or state have been made.

As the output clearly says, I must run terraform init with either the -reconfigure or -migrate-state flags. I'll use the -migrate-state flag. This flag will migrate the state file to the new location. So, I'll run terraform init -migrate-state in the base module. The output will be:

$ terraform init -migrate-state

Initializing the backend...
Backend configuration changed!

Terraform has detected that the configuration specified for the backend
has changed. Terraform will now check for existing state in the backends.

Acquiring state lock. This may take a few moments...
Acquiring state lock. This may take a few moments...
Do you want to copy existing state to the new backend?
  Pre-existing state was found while migrating the previous "gcs" backend to the
  newly configured "gcs" backend. No existing state was found in the newly
  configured "gcs" backend. Do you want to copy this state to the new "gcs"
  backend? Enter "yes" to copy and "no" to start with an empty state.

  Enter a value: yes


Successfully configured the backend "gcs"! Terraform will automatically
use this backend unless the backend configuration changes.

Initializing provider plugins...
- Reusing previous version of hashicorp/google from the dependency lock file
- Using previously-installed hashicorp/google v4.77.0

Terraform has been successfully initialized!

You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.

If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.

Running terrafom plan should work, giving you no changes as output.

With this, the base is checked.

Refactoring the network module

The network module is a bit more complicated to refactor. Initially, it created VPCs, subnets, and firewall rules for all VPCs. Now, it will create VPC, subnets, and firewall rules
for only one VPC. That means that I have to split changes for each VPC. This is the easy part. A simple copy-paste technique will do the trick. But what about the state file? Let's look at the first VPC: back-office for the dev environment:

...

  backend "gcs" {
    bucket = "terraform-states-network-playground-382512"
    prefix = "terraform/state/dev/back-office/network"
  }
}

The state file for the network module will be in the terraform/state/dev/network/back-office directory. The previous state file was in the terraform/state/network. I can repeat the same technique as I did for the base module. However, I plan to remove all code related to the other VPCs. When I remove code related to the other VPCs, the Terraform
will say that it will delete resources related to other VPCs. Why? It is simple: the state file for the network module contains resources related to other VPCs. So, when I remove code related to other VPCs, Terraform will see that the state file includes resources related to other VPCs and try to delete them. I want something else. I want to keep the resources related to other VPCs. So before I clean up the code, I'll move the state file with the -migrate-state flag. I'll run terraform init -migrate-state in the dev/back-office/network module. The Terraform will migrate the state file. If you look into the bucket, you will see that the ole state file is copied to the terraform/state/dev/network/back-office directory:

$ gsutil ls gs://terraform-states-network-playground-382512/terraform/state/network
gs://terraform-states-network-playground-382512/terraform/state/network/default.tfstate

So, the -migration-state flag does not delete the old state file but copies it to the new location. This makes life easier. After Terraform migrates the state file, I can clean up the code. I'll remove all code related to other VPCs.

The VPC peering is one thing that is related to all VPCs, and I need to refactor all of them before I can deal with it. I'll name a separate module in the dev environment peering. This module will contain the code for creating VPC peering for all VPCs. For now, it should be a copy of the old network module.

To continue with the refactoring back-office VPC, I'll remove all code related to other VPCs. When I run terraform plan this time, I'll get the following output as expected. I see that Terraform will delete resources related to other VPCs. I see the message Plan: 0 to add, 0 to change, 12 to destroy. So, I want to keep the resources related to other VPCs but to delete them from the state file related to the VPC back-office network module. I can do that by running the terraform state rm command. I'll run terraform state rm for each resource related to other VPCs. For example:

$ terraform state rm google_compute_subnetwork.subnet_postgres
Acquiring state lock. This may take a few moments...
Removed google_compute_subnetwork.subnet_postgres
Successfully removed 1 resource instance(s).

So there is a lot of typing. You can use more than one resource when removing resources from the state file. I do this until I get the message that there is nothing to change. All of this I do for the other VPCs in the dev environment.

To refactor the dev/peerings module, I'll do the same thing as I did for the dev/back-office/network module. I'll migrate the state file and remove all code related to VPCs, except peerings. I have to import the VPC information for each VPC from their state files:

data "terraform_remote_state" "back_office" {
  backend = "gcs"

  config = {
    bucket = "terraform-states-network-playground-382512"
    prefix = "terraform/state/dev/back-office/network"
  }
}

data "terraform_remote_state" "services" {
  backend = "gcs"

  config = {
    bucket = "terraform-states-network-playground-382512"
    prefix = "terraform/state/dev/services/network"
  }
}

data "terraform_remote_state" "storage" {
  backend = "gcs"

  config = {
    bucket = "terraform-states-network-playground-382512"
    prefix = "terraform/state/dev/storage/network"
  }
}

Of course, those are exported in the outputs. tf file for each VPC:

output "vpc_services" {
  value = google_compute_network.services.self_link
}

output "vpc_services_subnetwork" {
  value = google_compute_subnetwork.services
}

Refactoring the vms module

The vms module situation resembles the network module. The only difference is that I must import the VPC information from the network modules. So, first, I'll migrate the state file, remove all code related to other VPCs, and then import the VPC information from the network module. I'll do this for each VPC in the dev environment.

See the final code here.

When done, inspect the code. Notice how much there is duplicate code or almost the same code. Ignore resource names and a number of resources. That means that I can reuse the code.

Step three: Reuse the code with modules

There are a lot of code that is duplicated or almost the same. I want to make my code DRY. I can do it either by using Google network module or writing my module. It is wiser to use the already existing Google network module.

Refactoring the dev/back-office/network module to use the Google network module is straightforward:

module "back-office" {
  source  = "terraform-google-modules/network/google"
  version = "~> 7.3"

  project_id   = var.project_id
  network_name = "back-office"
  routing_mode = "REGIONAL"


  subnets = [
    {
      subnet_name   = "back-office"
      subnet_ip     = "10.1.0.0/24"
      subnet_region = var.region
    },
    {
      subnet_name   = "back-office-private"
      subnet_ip     = "10.2.0.0/24"
      subnet_region = var.region
    }
  ]

  ingress_rules = [
    {
      name = "back-office-icmp"
      allow = [
        {
          protocol = "icmp"
        }
      ]
      source_ranges = [
        "0.0.0.0/0"
      ]
    },
    {
      name = "back-office-iap"
      allow = [
        {
          protocol = "tcp"
        }
      ]
      source_ranges = [
        "35.235.240.0/20"
      ]
      target_service_accounts = [google_service_account.back_office_fw_sa.email]
      allow = [
        {
          protocol = "tcp"
        }
      ]

      depends_on = [google_service_account.back_office_fw_sa]
    }
  ]
}

Notice that now I am not creating resources directly. Instead, I am calling the module and passing variables to it. That is the way to reuse code in Terraform.

This code is more readable, and it is easier to maintain. But, as in the previous step, working with the state file is tricky. This time, I'll remove all other resources from the code. Since I have added the module, I need to run terraform init and then terraform plan. The plan will tell me that there are five resources to add and five to destroy. But I do not want to destroy anything. What I need to do is to rename the resources in the state file. I'll do that by running the terraform state mv command. For example:

$ terraform state mv google_compute_network.back_office  module.back-office.module.vpc.google_compute_network.network
Acquiring state lock. This may take a few moments...
Move "google_compute_network.back_office" to "module.back-office.module.vpc.google_compute_network.network"
Successfully moved 1 object(s).

$ terraform state mv google_compute_subnetwork.back_office_private 'module.back-office.module.subnets.google_compute_subnetwork.subnetwork["us-central1/back-office-private"]'
Move "google_compute_subnetwork.back_office_private" to "module.back-office.module.subnets.google_compute_subnetwork.subnetwork[\"us-central1/back-office-private\"]"
Successfully moved 1 object(s).

$ terraform state mv google_compute_subnetwork.back_office 'module.back-office.module.subnets.google_compute_subnetwork.subnetwork["us-central1/back-office"]'
Move "google_compute_subnetwork.back_office" to "module.back-office.module.subnets.google_compute_subnetwork.subnetwork[\"us-central1/back-office\"]"
Successfully moved 1 object(s).
Releasing state lock. This may take a few moments...

$ terraform state mv google_compute_firewall.back_office_icmp 'module.back-office.module.firewall_rules.google_compute_firewall.rules_ingress_egress["back-office-icmp"]'
Move "google_compute_firewall.back_office_icmp" to "module.back-office.module.firewall_rules.google_compute_firewall.rules_ingress_egress[\"back-office-icmp\"]"
Successfully moved 1 object(s).
Releasing state lock. This may take a few moments...

$ terraform state mv google_compute_firewall.back_office_iap 'module.back-office.module.firewall_rules.google_compute_firewall.rules_ingress_egress["back-office-iap"]'
Move "google_compute_firewall.back_office_iap" to "module.back-office.module.firewall_rules.google_compute_firewall.rules_ingress_egress[\"back-office-iap\"]"
Successfully moved 1 object(s).
Releasing state lock. This may take a few moments...

As well, I want to retain the same outputs, so I refactor the outputs.tf file:

output "vpc_back_office" {
  value = module.back-office.network_self_link
}

output "vpc_back_office_id" {
  value = module.back-office.network_id
}

output "vpc_back_office_subnetwork" {
  value = module.back-office.subnets["us-central1/back-office"]
}

output "vpc_back_office_private_subnetwork" {
  value = module.back-office.subnets["us-central1/back-office-private"]
}

output "back_office_fw_sa" {
  value = google_service_account.back_office_fw_sa.email
}

Now, I can run the terraform plan and see no changes. I can do the same for the other VPCs in the dev environment.

If you wonder how to know which resource to rename, you can run terraform plan and see which resources will be destroyed. Those are the resources that you need to rename. Names of new resources are new names of old resources.

Terraform will not allow you to rename resources of different types. So, you can't rename a subnet to a firewall rule. You can rename a subnet to a subnet and a firewall rule to a firewall rule.

Reuse code in the vms module

In the previous step, I used the already provided module. But I can write my module, too. For managing VMs, I'll do it. I'll create the ' vms' module in the <project root>/modules/vms directory. It will have two files, main.tf and variables.tf. The main.tf will contain the code for creating VMs. The variables.tf will have the variables for the module. The main.tf will look like this:

resource "google_compute_instance" "virtual_machine" {
  name         = var.name
  machine_type = var.machine_type
  zone         = var.zone

  scheduling {
    preemptible                 = true
    automatic_restart           = false
    provisioning_model          = "SPOT"
    instance_termination_action = "STOP"
  }

  allow_stopping_for_update = var.allow_stopping_for_update

  dynamic "service_account" {

    for_each = var.sa_email != "" ? [var.sa_email] : []

    content {
      email  = service_account.value
      scopes = ["https://www.googleapis.com/auth/cloud-platform"]
    }
  }

  boot_disk {
    initialize_params {
      image = "debian-cloud/debian-11"
    }
  }
  network_interface {
    network    = var.network
    subnetwork = var.subnetwork
  }
}

In simple terms, look at how the code looks in each VPC for creating VMs. It is the same. You extract common code and put it in the module. All values that differ are replaced with variables. Terraform does not have a classic if-then-else statement, so you must use dynamic block. So, this is my VM
template.

When using it in the dev/services/vms module, it will look like this:

module "services_vm_test" {
  source       = "../../../modules/vms"
  zone         = var.zone
  machine_type = "f1-micro"
  name         = "services-vm-test"
  network      = data.terraform_remote_state.vpc_services.outputs.vpc_services_id
  subnetwork   = data.terraform_remote_state.vpc_services.outputs.vpc_services_subnetwork.id
}

I could even put machine_type as a fixed value in the module. But I want to have the flexibility to change it. So, I'll leave it as a variable. For the dev/back-office/vms module, it will look
like this:

module "vms" {
  for_each = toset(["back-office-vm2", "back-office-private-vm1", "back-office-private-vm2", "back-office-vm1"])

  source       = "../../../modules/vms"
  zone         = var.zone
  machine_type = "f1-micro"
  name         = each.value
  network      = data.terraform_remote_state.back_office.outputs.vpc_back_office_id
  subnetwork   = data.terraform_remote_state.back_office.outputs.vpc_back_office_subnetwork.id

  sa_email                  = "back-office-vm1" == each.value ? data.terraform_remote_state.back_office.outputs.back_office_fw_sa : ""
  allow_stopping_for_update = "back-office-vm1" == each.value ? true : false
}

As the machine name differs for each VM, I'm using the for_each block. The only exception is back-office-vm1. For this VM, I'm using a service account that I created for firewall rules. That's why I'm using the Elvis operator to check if the VM is back-office-vm1. If it is, I'm using a service account; otherwise, I'm using an empty string.

Reuse code in the peering module

First, let's see the code:

locals {
  peerings = {
    "back-office-services-peering" : {
      "network" : data.terraform_remote_state.back_office.outputs.vpc_back_office,
      "peer_network" : data.terraform_remote_state.services.outputs.vpc_services
    },
    "services-back-office-peering" : {
      "network" : data.terraform_remote_state.services.outputs.vpc_services,
      "peer_network" : data.terraform_remote_state.back_office.outputs.vpc_back_office,
    },
    "back-office-storage-peering" : {
      "network" : data.terraform_remote_state.back_office.outputs.vpc_back_office,
      "peer_network" : data.terraform_remote_state.storage.outputs.vpc_storage,
    },
    "storage-back-office-peering" : {
      "network" : data.terraform_remote_state.storage.outputs.vpc_storage,
      "peer_network" : data.terraform_remote_state.back_office.outputs.vpc_back_office,
    }
  }
}

resource "google_compute_network_peering" "peerings" {
  for_each = local.peerings

  name         = each.key
  network      = each.value.network
  peer_network = each.value.peer_network
}

First, I define the locals block. It contains a map of maps. Each map contains information about peering. In this case, I can not use variables because when you assign value to a variable in the variable.tf file or terraform.tfvars file, you can not use other variables. So, I have to use the locals block. In the locals block, I define a map for peering. The key is the peering name, and the value is a map with two keys: network and peer_network. This setup allows me to use the for_each block in the google_compute_network_peering resource.

The final result is here.

Conclusion

In this part, I refactored the project to make it more manageable: I split the project into environments, created a module for each VPC, and reused the code with modules. I used already existing modules, and I wrote my module. I used the terraform state mv command to move resources from one state file to another. I used the terraform state rm command to remove resources from the state file.

In the next part, let's try to use Terragrunt to make our code even more DRY and make running CLI commands easier.

Enjoy...

Exploring GCP With Terraform: VPC Firewall Rules, part 2

Robert Nemet — Sat, 02 Sep 2023 09:05:12 +0000

This post would be 3rd part of the series about exploring GCP with Terraform. In the previous part, I created VPC networks, subnets, and a few firewall rules. In this part, I will explore more firewall rules and their parameters.

More precisely, I'll set up three VPCs: back-office, services and storage. In VPC back_office, I'll have two subnets; in others, I'll have one subnet. For the sake of conversation, imagine that VMs in the back-office have to call VMs in services and storage. Also, direct access to VMs from outside should not be allowed, except for one that will serve for maintenance.

What I have done so far

I have two VPCs: back-office and services. The VPC back-office has a subnet back-office and the VPC services a subnet named services. In each VPC, I added firewall rules to allow SSH access from outside through IAP. I also added a firewall rule to allow ICMP traffic from anywhere. Each VPC has a VM instance.

The Terraform project is growing. If you look at how I set a base workflow in the first post of this series, now I have two more similar workflows for networks and vms. It is not the best practice to have it like this, but this will be addressed later.

VPC Firewall rules

So, what is a firewall rule? A firewall rule is a set of conditions that define what traffic is allowed to enter or leave a VPC network. A collection of parameters defines each firewall rule. Those parameters are:

direction - ingress or egress (default: ingress)
priority - the priority of the rule. The lower the number, the higher the priority. The default value is 1000.
action - allow or deny
enforced - if the rule is enforced. The default value is true.
target - the target of the rule. The target can be a tag, a service account, or a network.
source - the source of the traffic. The source can be a tag, a service account, or a network.
protocol - the protocol of the traffic. The protocols like TCP, UDP, ICMP, etc.
logs - boolean. Make logs to see if the rule is matched or not.

Setting up the stage

VPC back_office has two subnets: back-office and back-office-private. So far, I have firewall rules: back-office-iap, back-office-icmp, and back-office-ssh, allowing ingress traffic from Google IAP, ICMP from anywhere, and SSH from anywhere. So, adding a new subnet back-office-private to the VPC back-office:

# subnet for back office: private 
resource "google_compute_subnetwork" "back_office_private" {
  name          = "back-office-private"
  ip_cidr_range = "10.2.0.0/24"
  network       = google_compute_network.back_office.self_link
  region        = var.region
}

To create a VM, you can use this code as a template inside the vms workflow:

resource "google_compute_instance" "services_vm_test" {
  name         = "services-vm-test"
  machine_type = "f1-micro"
  zone         = var.zone

  scheduling {
    preemptible        = true
    automatic_restart  = false
    provisioning_model = "SPOT"
  }

  boot_disk {
    initialize_params {
      image = "debian-cloud/debian-11"
    }
  }
  network_interface {
    network    = data.terraform_remote_state.network.outputs.vpc_services.id
    subnetwork = data.terraform_remote_state.network.outputs.vpc_services_subnetwork.id
  }
}

You may wonder what is this part:

network_interface {
    network    = data.terraform_remote_state.network.outputs.vpc_services.id
    subnetwork = data.terraform_remote_state.network.outputs.vpc_services_subnetwork.id
  }

Where is that defined? Since I have a network creation in the different workflow networks, I need to get the network ID and subnet ID from that workflow. I can do that by using the terraform_remote_state data source. But first, I need to export it in the networks workflow in the outputs.tf file:

output "vpc_services" {
  value = google_compute_network.services
}

output "vpc_services_subnetwork" {
  value = google_compute_subnetwork.services
}

Using the template above for VM and exposing all VPCs and subnets from networks, I can add two VMs in each subnet. The VMs in the back-office subnet will be named back-office-vm1 and back-office-vm2. The VMs in the back-office-private subnet will be named back-office-private-vm1 and back-office-private-vm2. First, I must change the networks and then the vms. Then, I can look for created instances:

$ gcloud compute instances list
NAME                     ZONE           MACHINE_TYPE  PREEMPTIBLE  INTERNAL_IP  EXTERNAL_IP  STATUS
back-office-private-vm1  us-central1-c  f1-micro      true         10.2.0.3                  RUNNING
back-office-private-vm2  us-central1-c  f1-micro      true         10.2.0.2                  RUNNING
back-office-vm1          us-central1-c  f1-micro      true         10.1.0.9                  RUNNING
back-office-vm2          us-central1-c  f1-micro      true         10.1.0.10                 RUNNING
services-vm-test         us-central1-c  f1-micro      true         10.1.0.5                  RUNNING

Then, let's try to connect to any VM from the local machine:

$ gcloud compute ssh back-office-vm1 --zone us-central1-c --tunnel-through-iap

Then, from it, let's try to connect to any other VM, ping it:

$ ping -c 5 10.2.0.3
PING 10.2.0.3 (10.2.0.3) 56(84) bytes of data.
64 bytes from 10.2.0.3: icmp_seq=1 ttl=64 time=0.884 ms
64 bytes from 10.2.0.3: icmp_seq=2 ttl=64 time=0.165 ms
64 bytes from 10.2.0.3: icmp_seq=3 ttl=64 time=0.233 ms
64 bytes from 10.2.0.3: icmp_seq=4 ttl=64 time=0.159 ms
64 bytes from 10.2.0.3: icmp_seq=5 ttl=64 time=0.238 ms

--- 10.2.0.3 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 4103ms
rtt min/avg/max/mdev = 0.159/0.335/0.884/0.276 ms

Yes, you can connect to any VM inside VPC. Why? Because we have a firewall rule that allows ICMP traffic from anywhere. Let's delete that rule and try again(from the
local machine):

$ gcloud compute firewall-rules delete back-office-icmp
The following firewalls will be deleted:
 - [back-office-icmp]

Do you want to continue (Y/n)?  y

Deleted [https://www.googleapis.com/compute/v1/projects/network-playground-382512/global/firewalls/back-office-icmp].

Repeat the ping command from the connected VM to any other VM:

ping -c 5 10.2.0.3
PING 10.2.0.3 (10.2.0.3) 56(84) bytes of data.

--- 10.2.0.3 ping statistics ---
5 packets transmitted, 0 received, 100% packet loss, time 4074ms

Now, at this moment VMs, can not communicate with each other.

Tightening access: Only one VM for SSH (bastion)

Let's say we want to allow SSH access only to one VM from outside. We can do that by adding a firewall rule that allows SSH access only to one VM. Let's remove the rule back-office-ssh as it is not practical. Why? I'm accessing VMs with the tunnel through IAP, so I do not need it. It would be helpful only if I could access them directly. That means that the target VM has an external IP. But I do not want that.

And let's modify the rule back-office-iap to allow SSH access only to one VM. Let's add the tag bastion to the VM:

resource "google_compute_firewall" "back_office_iap" {
  name    = "back-office-iap"
  network = google_compute_network.back_office.self_link

  source_ranges = ["35.235.240.0/20"]
  target_tags   = ["bastion"]
  direction     = "INGRESS"
  allow {
    protocol = "tcp"
  }
}

I am using the tag bastion to allow SSH access by applying this firewall rule only to VMs with particular tags. If I try to access the VM back-office-vm1 from the local machine, I will get an error:

$ gcloud compute ssh back-office-vm1 --zone us-central1-c --tunnel-through-iap
ERROR: (gcloud.compute.start-iap-tunnel) Error while connecting [4003: 'failed to connect to backend']. (Failed to connect to port 22)
kex_exchange_identification: Connection closed by remote host
Connection closed by UNKNOWN port 65535

Recommendation: To check for possible causes of SSH connectivity issues and get
recommendations, rerun the ssh command with the --troubleshoot option.

gcloud compute ssh back-office-vm1 --project=network-playground-382512 --zone=us-central1-c --troubleshoot

Or, to investigate an IAP tunneling issue:

gcloud compute ssh back-office-vm1 --project=network-playground-382512 --zone=us-central1-c --troubleshoot --tunnel-through-iap

ERROR: (gcloud.compute.ssh) [/usr/bin/ssh] exited with return code [255].

Let's add the tag bastion to the VM back-office-vm1:

$ gcloud compute instances add-tags back-office-vm1 --zone=us-central1-c --tags=bastion
Updated [https://www.googleapis.com/compute/v1/projects/network-playground-382512/zones/us-central1-c/instances/back-office-vm1].

Now, connecting to the VM back-office-vm1 works. But connecting to any other VM in the VPC back-office doesn't work. Why? Because we have a firewall rule that allows SSH access only to the VM back-office-vm1. The firewall matches the target with the tag value. If I tag any other VM in the VPC back-office with the tag bastion, I can connect to it.

Well, I'm not happy that adding/removing tags is so easy. I want a more secure way to allow SSH access to VMs. I could use the service account to enable SSH access:

resource "google_service_account" "back_office_fw_sa" {
  account_id   = "back-office"
  display_name = "back-office"
}

resource "google_compute_firewall" "back_office_iap" {
  name    = "back-office-iap"
  network = google_compute_network.back_office.self_link

  source_ranges           = ["35.235.240.0/20"]
  target_service_accounts = [google_service_account.back_office_fw_sa.email]
  direction               = "INGRESS"
  allow {
    protocol = "tcp"
  }
  depends_on = [google_service_account.back_office_fw_sa]
}

Connecting to the VM back-office-vm1 from the local machine, I will get the same error. Why? Because I'm not using the
service account to connect to the VM. I'm using the tunnel through IAP. Let's add the service account to the VM back-office-vm1:

resource "google_compute_instance" "back_office_vm1" {
  name         = "back-office-vm1"
...

  allow_stopping_for_update = true

  service_account {
    email  = data.terraform_remote_state.network.outputs.back_office_fw_sa
    scopes = ["https://www.googleapis.com/auth/cloud-platform"]
  }

...
}

I added the service account to the VM back-office-vm1. Now, I can connect to it from the local machine. IAP can connect to it. Anyone who passes the IAP authentication can connect to it. But I can't connect to any other VM in the VPC back-office. Why? Because I'm not using the service account to connect to the VMs. I'm using the tunnel through IAP, and the firewall rule allows access only to the back-office-vm1 VM.

The scopes are required to allow the service account to access the resources. When adding service accounts to VMs, you must set allow_stopping_for_update = true and service account scopes. Changing the service account on the VM requires stopping the VM.

This requirement makes a difference. You can add and remove tags on the fly but can't do that with service accounts. You must stop the VM to change the service account. That is why using service accounts is more secure than using tags.

Notice that I exported the service account email from the networks workflow. I did that because I needed to use it in the vms workflow.

OK, now I can connect to the VM back-office-vm1, in the VPC back-office, from the local machine via the IAP tunnel. But I can connect to any other VM in the VPC back-office. But I still can connect to all other VMs in the VPC services and storages. I'll remove the IAP firewall rule for those VPCs to fix this. But then I'll need access from the VPC's back_office VMs to the VPC's services and storages VMs. Keep the ICMP rule for VPC services and storages.

You can do it now if you did not add VPC storages, a subnet, and VMs. Just remember to add VPC and subnet to the networks workflow and VMs to the vms workflow. For the storage subnet in the VPC storage, I used CIDR 10.120.0.0/24. And just for a reminder, CIDR for subnet services in VPC services is 10.1.0.0/24, and subnet back-office in VPC back_office is 10.1.0.0/24 and subnet back-office-private in VPC back_office is 10.2.0.0/24.

Tightening access: Accessing other VPCs

OK, now what? I can't connect to any VM in the VPC services and datastorages from the local machine because I just removed IAP firewall rules for those VPCs. I can connect only to back-office_vm1 from the local machine. But from it, I can connect to any other VM in the VPC back-office, but not in other VPCs. To be able to connect from the local machine to back-office-vm1 and then to any VMs in VPC services and storages, I need first to connect the VPC back-office
to VPC services and storages. I can do that by setting up VPC peering. I'll set up VPC peering between VPC back-office and services and between VPC back-office and storages:

resource "google_compute_network_peering" "back_office_service_peering" {
  name         = "back-office-services-peering"
  network      = google_compute_network.back_office.self_link
  peer_network = google_compute_network.services.self_link
}

resource "google_compute_network_peering" "service_back_office_peering" {
  name         = "services-back-office-peering"
  network      = google_compute_network.services.self_link
  peer_network = google_compute_network.back_office.self_link
}

resource "google_compute_network_peering" "back_office_datastorage_peering" {
  name         = "back-office-storage-peering"
  network      = google_compute_network.back_office.self_link
  peer_network = google_compute_network.storage.self_link
}

resource "google_compute_network_peering" "datastorage_back_office_peering" {
  name         = "storage-back-office-peering"
  network      = google_compute_network.storage.self_link
  peer_network = google_compute_network.back_office.self_link
}

But we'll get an error when applying the changes:

 Error: Error waiting for Adding Network Peering: An IP range in the peer network (10.1.0.0/24) overlaps with an IP range in the local network (10.1.0.0/24) allocated by 
 resource (projects/network-playground-382512/regions/us-central1/subnetworks/back-office).

Why? Because the VPC back-office and services have subnets with the same IP range. When setting up VPC peering, it is essential to know that the IP ranges of the subnets must not overlap. Changing the IP ranges of subnets in VPC services is OK. But this means we need to plan the IP ranges of subnets. If you are using IPv6 in a VPC, then you don't need to worry about this. But, if you are using IPv4, you need to plan the IP ranges of subnets. The reason is that if you want to change CIDR for a subnet, you need to free all resources that use this subnet first, like deleting all VMs.

Changing the IP ranges of subnets in VPC services is easy. I'll change the subnet services IP range to 10.3.0.0/24. But to do that, I need to delete the VMs in the VPC services and recreate peering between VPCs. So, I'll remove VMs from the VPC services. Then, fixing the subnet range in VPC _services, recreate peering between VPCs. And finally, adding back deleted VMs to the VPC services.

While I'm doing this, I'll revert SSH access to the VMs in all VPCs, but this time, I'll use CIDR ranges to allow access:

resource "google_compute_firewall" "services_ssh" {
  name    = "services-ssh"
  network = google_compute_network.services.self_link

  source_ranges = ["10.1.0.0/24", "10.2.0.0/24"]
  direction     = "INGRESS"
  allow {
    protocol = "tcp"
    ports    = ["22"]
  }
}

This limits access to the VMs in the VPC services from the VPC back-office and back-office-private subnets. I'll do the same for the VPC storage.

You can add one for the VPC storage as well. On top of all this, you'll need to add public SSH keys to the VMs. You can do it per VM or project. I'll do it on
the project level for now:

resource "google_compute_project_metadata" "default" {
  metadata = {
    ssh-keys = <<EOF
      user:ssh-rsa public_key_goes_here user
    EOF
  }
}

When adding SSH public keys to the project level, you are allowing SSH access to all VMs in the project. For now, this is good enough. To generate SSH keys, you can use the command ssh-keygen:

$ ssh-keygen -f private-key -C rnemet -b 2048
Generating public/private rsa key pair.
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in private-key
Your public key has been saved in private-key.pub
...

The code above will generate two files: private-key and private-key.pub. The file private-key is the private key, and the file private-key.pub is the public key. Now, you
can use the public key to add it to the project metadata. You can also use the private key to connect to the VMs. That means you must keep the private key safe and be on machines you use to connect to the VMs. After adding the public key to the project metadata, you can connect to the VMs in the VPC services and storages from VPC back-office VMs. All you need when connecting to the VMs is the private key on bastion VMs:

$ gcloud compute scp ./private-key rnemet@back-office-vm1:.ssh/private-key --zone=us-central1-c
External IP address was not found; defaulting to using IAP tunneling.
WARNING:

To increase the performance of the tunnel, consider installing NumPy. For instructions,
please see https://cloud.google.com/iap/docs/using-tcp-forwarding#increasing_the_tcp_upload_bandwidth

private-key

When connecting to the VMs:

$ ssh rnemet@10.120.0.3 -i .ssh/private-key

With this setup, I can connect to the VMs in the VPC services and storages from the VM in the VPC back-office. Direct access to other VMs is not possible because
of IAP and firewall rules.

Conclusion

So far, I have created VPCs, subnets, and VMs. I have also created firewall rules to allow SSH access to the VMs from one VM inside my networks. I have also created VPC peering between VPCs. Also, I started to share resources between Terraform state files. However, my TF project is growing and needs a better structure. I need to invest some time to improve it. But at the same time, I would like to put some services that are reachable from the outside.

What do you think? What would you do next? Let me know...

References

Exploring GCP With Terraform: VPCs, Firewall Rules And VMs

Robert Nemet — Wed, 23 Aug 2023 14:01:56 +0000

This post will continue my previous post Exploring GCP With Terraform: Setting Up The Environment And Project.
In this post, I'll:

Create VPC with Terraform
Create subnets/firewall rules
Create VMs in the VPC
How to access to VMs from outside and inside the VPC

Some Info Here
Also, I'll mention how to use gcloud to fetch information about created resources. At the end, I'll mention two useful Terraform CLI commands. And why I'm structuring the project as I do.

This structuring is one of many ways to do it. It is just my way of doing it for now. The project needs refactoring, but for now, it is good enough.

Some Info Here
I replaced the real project ID with _project-id_in the following examples. You need to replace it with your project ID.

DevCube | Robert Nemet | Substack

Weekly rant about software design, devops, kubernetes, sre... Click to read DevCube, by Robert Nemet, a Substack publication. Launched 6 months ago.

rnemet.substack.com

What is VPC in GCP?

Virtual Private Network(VPC) is a virtual network defined in the cloud. It is a private network isolated from other networks in the cloud. It is responsible for:

Connectivity between VMs
Traffic distribution from GC load balancers to backends,
Connecting on-premises networks with Cloud VPN or Cloud Interconnect,
Network Load Balancers and proxies for internal Application Load Balancers.

VPC is a global resource in GCP. A global resource means it is not bound to any region or zone. It is a logical resource that spans all regions.

Some Info Here
We can distinguish between two types of resources: global and regional. Global resources are not bound to any region or zone. They are logical resources that span all regions.

Regional resources are bound to a specific region. They are physical resources that exist in a specific region. Each region has at least three zones. Zones are isolated from each other and independent of each other. If one zone fails, other zones are not affected. You can perceive a zone as a data center. They are connected to a high-bandwidth, low-latency network.

You can imagine VPC as a logical container that holds all the other networking resources like firewalls, routes, subnets, etc.

Subnets

Subnets are logical partitions of the VPC's IP address range. They are regional resources, which means they are bound to a specific region. You can specify the IP address range as IPv4 (single stack) or IPv4/IPv6 (dual-stack). They are allocating IP addresses to other resources in the subnet's region.

Firewall Rules

Firewall rules are used to control traffic to and from different destinations. They are applied to the VPC network and are enforced at the VM instance level. Every VPC
network has two implied firewall rules:

implied allow egress. Egress(outgoing) traffic is allowed to all destinations.
implied deny ingress. Ingress(incoming) traffic is denied from all sources.

As well as the implied rules, you can create your own rules.

Creating VPC with Terraform

I set up a base for my Terraform project in a previous post. Now, next to the base directory, I'm creating a new network directory
with the same file structure. Let's start with the provider.tf file:

provider "google" {
  project = var.project_id
  region  = var.region
  zone    = var.zone
}


terraform {
  required_version = ">=1.5.5"

  required_providers {
    google = {
      source  = "hashicorp/google"
      version = "4.77.0"
    }
  }

  backend "gcs" {
    bucket = "terraform-states-project-id"
    prefix = "terraform/state/network"
  }
}

Provider for all workflows is the same as in the base directory. The difference is the backend.gcs.prefix, which is now terraform/state/network. I'm using the same bucket. For the base workflow, you can change it to terraform/state/base.

Some Info Here

I'm splitting state files by workflows. The reason is to minimize the impact of changes from one workflow to another. There will be a dependency between workflows, but it should be minimal. Splitting state files would be necessary if multiple teams worked on the same project.

And now creating the VPC in main.tf:

# vpc: back office
resource "google_compute_network" "back_office" {
  name                    = "back-office"
  description             = "Back office network"
  auto_create_subnetworks = false
  routing_mode            = "REGIONAL"
}

I'm creating a VPC named back-office and turning off the auto-creation of subnets. It is recommended to set auto_create_subnetworks to false and create subnets
manually. This way, you have more control over the subnets. Otherwise, GCP will create a subnet in each region.

Some Info Here

You can list all VPCs in a project with the following command:

$ gcloud compute networks list

NAME         SUBNET_MODE  BGP_ROUTING_MODE  IPV4_RANGE  GATEWAY_IPV4
back-office  CUSTOM       REGIONAL
default      AUTO         REGIONAL
services     CUSTOM       REGIONAL

Notice a default VPC with AUTO subnet mode. The default VPC is created when you make a project. It has a subnet in each region. I created other VPCs.

To get info on a specific VPC, try:

$ gcloud compute networks describe back-office

autoCreateSubnetworks: false
creationTimestamp: '2023-08-20T07:06:11.543-07:00'
description: Back office network
id: '1334556407227679868'
kind: compute#network
name: back-office
networkFirewallPolicyEnforcementOrder: AFTER_CLASSIC_FIREWALL
routingConfig:
  routingMode: REGIONAL
selfLink: https://www.googleapis.com/compute/v1/projects/project-id/global/networks/back-office
selfLinkWithId: https://www.googleapis.com/compute/v1/projects/project-id/global/networks/1334556407227679868
subnetworks:
- https://www.googleapis.com/compute/v1/projects/project-id/regions/us-central1/subnetworks/back-office
x_gcloud_bgp_routing_mode: REGIONAL
x_gcloud_subnet_mode: CUSTOM

I am setting the routing mode to REGIONAL. The routing_mode can be set to REGIONAL or GLOBAL. The REGIONAL means advertising routes to subnets in the same
region. The GLOBAL means advertising routes to subnets in all regions.

As mentioned, more is needed. I need to add a subnet to the VPC:

# subnet for back office
resource "google_compute_subnetwork" "back_office" {
  name          = "back-office"
  ip_cidr_range = "10.1.0.0/24"
  network       = google_compute_network.back_office.self_link
  region        = var.region
}

I'm creating a subnet named back-office and IP range 10.1.0.0/24. This subnet is attached to the back_office VPC in a predefined region.

Some Info Here

The self_link refers to the resource and is a unique identifier for the resource. It is used to reference the resource in other resources.

The IP range is expressed as CIDR notation. The CIDR notation is a compact representation of an IP address and its associated routing prefix. The prefix is written after the IP address, and the prefix length is indicated by the number of bits set to 1 in the subnet mask. Check resources for more info.

So, I should have a VPC with a subnet. You can check the details of this subnet with gcloud:

$ gcloud compute networks subnets describe back-office --region us-central1

creationTimestamp: '2023-08-21T13:12:57.423-07:00'
enableFlowLogs: false
fingerprint: crXvadZ1JQ=
gatewayAddress: 10.1.0.1
id: '88832453211859900582'
ipCidrRange: 10.1.0.0/24
kind: compute#subnetwork
logConfig:
  aggregationInterval: INTERVAL_5_SEC
  enable: false
  flowSampling: 0.5
  metadata: INCLUDE_ALL_METADATA
name: back-office
network: https://www.googleapis.com/compute/v1/projects/project-id/global/networks/back-office
privateIpGoogleAccess: false
privateIpv6GoogleAccess: DISABLE_GOOGLE_ACCESS
purpose: PRIVATE
region: https://www.googleapis.com/compute/v1/projects/project-id/regions/us-central1
selfLink: https://www.googleapis.com/compute/v1/projects/project-id/regions/us-central1/subnetworks/back-office
stackType: IPV4_ONLY

Add a VM

Now is the time to test this. I'm creating a VM in this subnet:

resource "google_compute_instance" "bo_vm_test_alpha" {
  name         = "bo-vm-test-alpha"
  machine_type = "f1-micro"
  zone         = var.zone
  tags         = ["bo-vm-test"]

  scheduling {
    preemptible        = true
    automatic_restart  = false
    provisioning_model = "SPOT"
  }

  boot_disk {
    initialize_params {
      image = "debian-cloud/debian-11"
    }
  }
  network_interface {
    network    = google_compute_network.back_office.self_link
    subnetwork = google_compute_subnetwork.back_office.self_link
  }
}

I'm creating a VM named bo-vm-test-alpha and machine type f1-micro. The VM is made in the zone us-central1-a. The VM is tagged with the bo-vm-test tag. The VM is preemptible, and it is using a spot instance. The boot disk is using the Debian 11 image. The VM is attached to the back_office VPC and back-office subnet.

This VM would be the cheapest in GCP, and it is not suitable for production. It is just for testing purposes. If you want to access it, go to the Console. There, choose the right project, and you should see your VM. Then go to the SSH button and click on it. It will open a new window with a terminal. You should be able to access the VM. Or not.

Add Firewall Rules

You can't. Why? Because I didn't create a firewall rule to allow SSH:

resource "google_compute_firewall" "back_office_ssh" {
  name    = "back-office-ssh"
  network = google_compute_network.back_office.self_link

  source_ranges = ["0.0.0.0/0"]
  direction     = "INGRESS"
  allow {
    protocol = "tcp"
    ports    = ["22"]
  }
}

I'm creating a firewall rule named back-office-ssh, applied to the back_office VPC. The source range is 0.0.0.0/0, which means all internet. The direction is INGRESS, allowing TCP traffic on port 22.

This time, let's try to connect with gcloud:

$ gcloud compute ssh bo-vm-test-alpha --zone=us-central1-c  --project=project-id
External IP address was not found; defaulting to using IAP tunneling.
WARNING:

To increase the performance of the tunnel, consider installing NumPy. For instructions,
please see https://cloud.google.com/iap/docs/using-tcp-forwarding#increasing_the_tcp_upload_bandwidth

ERROR: (gcloud.compute.start-iap-tunnel) Error while connecting [4003: 'failed to connect to backend']. (Failed to connect to port 22)
kex_exchange_identification: Connection closed by remote host
Connection closed by UNKNOWN port 65535

Recommendation: To check for possible causes of SSH connectivity issues and get
recommendations, rerun the ssh command with the --troubleshoot option.

gcloud compute ssh bo-vm-test-alpha --project=project-id --zone=us-central1-c --troubleshoot

Or, to investigate an IAP tunneling issue:

gcloud compute ssh bo-vm-test-alpha --project=project-id --zone=us-central1-c --troubleshoot --tunnel-through-iap

It is not working. I have two options to fix this: to add an external IP address to the VM or to use IAP tunneling. I'll go with the second option. IAP tunneling is used to connect to VMs without external IP addresses. The VM I just created does not have an external IP address. So, let's enable IAP tunneling:

$ gcloud services enable iap.googleapis.com

And add a firewall rule to allow ingress from the Cloud IAP for TCP forwarding:

resource "google_compute_firewall" "back_office_iap" {
  name    = "back-office-iap"
  network = google_compute_network.back_office.self_link

  source_ranges = ["35.235.240.0/20"]
  direction     = "INGRESS"
  allow {
    protocol = "tcp"
  }
}

The CIDR range 35.235.240.0/20 is the range used by IAP. GCP specifies it see here for more info.

Run terraform apply. Please wait for it to finish and try again. This time, it should work:

 gcloud compute ssh bo-vm-test-alpha --zone=us-central1-c  --project=project-id
External IP address was not found; defaulting to using IAP tunneling.
WARNING:

To increase the performance of the tunnel, consider installing NumPy. For instructions,
please see https://cloud.google.com/iap/docs/using-tcp-forwarding#increasing_the_tcp_upload_bandwidth

Warning: Permanently added 'compute.89653456855762463' (ED25519) to the list of known hosts.
Linux bo-vm-test-alpha 5.10.0-24-cloud-amd64 #1 SMP Debian 5.10.179-5 (2023-08-08) x86_64

The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
rnemet@bo-vm-test-alpha:~$

It is working. I connect to the VM. Now, I can connect to the VM from another VM in the same subnet. I'll create another VM in the same subnet. Just name it bo-vm-test-beta. When done, check it with gcloud:

$ gcloud compute instances list
NAME              ZONE           MACHINE_TYPE  PREEMPTIBLE  INTERNAL_IP  EXTERNAL_IP  STATUS
bo-vm-test-alpha  us-central1-c  f1-micro      true         10.1.0.5                  RUNNING
bo-vm-test-beta   us-central1-c  f1-micro      true         10.1.0.6                  RUNNING

I see that alpha has IP 10.1.0.5 and beta has IP 10.1.0.6. Now I'll try to ping alpha from beta. But first, I need to allow ICMP traffic:

resource "google_compute_firewall" "back_office_icmp" {
  name    = "back-office-icmp"
  network = google_compute_network.back_office.self_link

  source_ranges = ["0.0.0.0/0"]
  direction     = "INGRESS"
  allow {
    protocol = "icmp"
  }
}

And then let's try to ping alpha from the beta:

$ gcloud compute ssh bo-vm-test-beta --zone=us-central1-c  --project=project-id
External IP address was not found; defaulting to using IAP tunneling.
WARNING:

To increase the performance of the tunnel, consider installing NumPy. For instructions,
please see https://cloud.google.com/iap/docs/using-tcp-forwarding#increasing_the_tcp_upload_bandwidth

Linux bo-vm-test-beta 5.10.0-24-cloud-amd64 #1 SMP Debian 5.10.179-5 (2023-08-08) x86_64

The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
Last login: Tue Aug 22 21:13:44 2023 from 35.235.244.34
rnemet@bo-vm-test-beta:~$ ping 10.1.0.6
PING 10.1.0.6 (10.1.0.6) 56(84) bytes of data.
64 bytes from 10.1.0.6: icmp_seq=1 ttl=64 time=0.014 ms
64 bytes from 10.1.0.6: icmp_seq=2 ttl=64 time=0.045 ms
64 bytes from 10.1.0.6: icmp_seq=3 ttl=64 time=0.035 ms
64 bytes from 10.1.0.6: icmp_seq=4 ttl=64 time=0.037 ms
64 bytes from 10.1.0.6: icmp_seq=5 ttl=64 time=0.029 ms
64 bytes from 10.1.0.6: icmp_seq=6 ttl=64 time=0.036 ms
64 bytes from 10.1.0.6: icmp_seq=7 ttl=64 time=0.022 ms
64 bytes from 10.1.0.6: icmp_seq=8 ttl=64 time=0.035 ms
64 bytes from 10.1.0.6: icmp_seq=9 ttl=64 time=0.028 ms
64 bytes from 10.1.0.6: icmp_seq=10 ttl=64 time=0.036 ms
64 bytes from 10.1.0.6: icmp_seq=11 ttl=64 time=0.036 ms
^C
--- 10.1.0.6 ping statistics ---
11 packets transmitted, 11 received, 0% packet loss, time 10232ms
rtt min/avg/max/mdev = 0.014/0.032/0.045/0.008 ms
rnemet@bo-vm-test-beta:~$

Some Info Here
You'll need to wait a few minutes for the firewall rule to take effect. Or restart the VM:

$ gcloud compute instances reset bo-vm-test-beta bo-vm-test-alpha --zone=us-central1-c
Updated [https://www.googleapis.com/compute/v1/projects/project-id/zones/us-central1-c/instances/bo-vm-test-beta].
Updated [https://www.googleapis.com/compute/v1/projects/project-id/zones/us-central1-c/instances/bo-vm-test-alpha].

Terraform CLI: fmt and validate

Frequent code changes make a mess. It is good practice to keep the code clean and formatted. Terraform has a command to format the code: terraform fmt. It will format the code in the current directory. It will also format all the files in the subdirectories. It is a good practice to run this command before committing the code.

On the other side, terraform validate will check the syntax of the code. It will check if the code is valid and complete. It will not check if the code is correct. For example, if you have a typo in the resource name, it will not catch it. It will detect if you have a typo in the resource type. It will also check if all the variables are defined.

Conclusion

In this post, I created a VPC with a subnet and a VM in the subnet. I also made firewall rules to allow SSH and ICMP traffic. I used gcloud to check the created resources. I also used gcloud to connect to the VMs. I used IAP tunneling to connect to the VM without an external IP address. I also used terraform fmt and terraform validate to format and validate the code.

I'll create a second VPC and a VM for the next post. Then, explore how to connect VMs from different VPCs. As well as how to route traffic between VPCs.

If you find this helpful, please share it with others. If you have any questions or comments, please let me know in newsletter comments or via email.

DevCube | Robert Nemet | Substack

Weekly rant about software design, devops, kubernetes, sre... Click to read DevCube, by Robert Nemet, a Substack publication. Launched 6 months ago.

rnemet.substack.com

Resources

Exploring GCP With Terraform: Setting Up The Environment And Project

Robert Nemet — Sat, 19 Aug 2023 19:02:21 +0000

This topic is nothing new. There are many articles and tutorials on the internet about this. I'll be setting up a Terraform project to explore GCP. I cover the basics of Terraform and GCP.

Read Before Proceed..
I replaced the real project ID with project-id in the following examples. You need to replace it with your project ID.

Prerequisites

Create GCP Project

First, you must create a GCP account, then create a project and set up billing on that project. If you already have a Gmail account, you can use it to make a GCP account. Head to the Google Cloud Console and create a new project.

My plan is to pay for the services I use. I need to feel the pain and learn more about cost optimization. You can be more intelligent than me and use the free tier for the first year. After that, you will be charged for the services you use. You can find more information about the free tier here.

DevCube | Robert Nemet | Substack

Weekly rant about software design, devops, kubernetes, sre... Click to read DevCube, by Robert Nemet, a Substack publication. Launched 6 months ago.

rnemet.substack.com

Install And Set Up The gcloud CLI Tool

The gcloud CLI is a part of the Google Cloud SDK. You can find the installation instructions for gcloud CLI here. Next, you need to set up the gcloud CLI tool. Docs are here.

For future reference, you can find the list of all gcloud commands here.

After initialization, you can always check what you set with the following command:

$ gcloud config list [SECTION/PROPERTY] [--all]

Install Terraform CLI Tool

You can find the Terraform installation instructions here. But I'm not following the instructions.

I'm using the tool tfenv to manage Terraform versions. Other tools can do that. You can use asdf, too. I saw that asdf can do more than manage Terraform versions.

Let me explain how I do it since I use tfenv. I assume you already installed it. I use tfenv to install the desired version of Terraform. And to set it as default:

$ tfenv install 1.5.5
$ tfenv use 1.5.5

You can check the version with the following command:

$ terraform --version

I'll explain later why I use tfenv.

Other Tools I Use

Other tools I'll be using are:

task - a task runner and a replacement for make
git - version control system
direnv - unconsciously set and unset environment variables

Setting Up The Project

I'm using this layout for now:

.                                       # project root                 
├── README.md                           # project README
├── LICENSE                             # project LICENSE
├── .gitignore                          # project .gitignore
├── Taskfile.yml                        # project taskfile
└── gcp                                 # GCP Terraform code
    └── base                            # base GCP Terraform code
        ├── README.md                   # base README
        ├── .terraform-version          # base terraform version
        ├── main.tf                     # base main.tf
        ├── outputs.tf                  # base outputs.tf
        ├── variables.tf                # base variables.tf
        ├── terraform.tfvars            # base terraform.tfvars
        └── provider.tf                 # base provider.tf

Let me quickly explain. The gcp directory is the root directory for all GCP Terraform code. The base directory is the root directory for all base GCP Terraform code. That base code would cover the following, for now:

Creating a bucket for storing Terraform state
Creating Key and KeyRing for encrypting Terraform state file
Setting up Terraform's backend
Setting up Terraform provider
Setting up Terraform base variables

Just create the directories and files, and we are ready to go.

How I use tfenv?
Oh, one small thing. The file .terraform-version is used by tfenv to set and pin the Terraform version. Even if I set with tfenv the default Terraform version, I use this file to say which Terraform version should be used for the base. If you are using tfenv do:

$ cd gcp/base
$ tfenv pin
Pinned version by writing "1.5.5" to /somepath/gcp/base/.terraform-version

With this, it does not matter what is the default Terraform version. The tfenv will use the version from the .terraform-version file. If, for some reason, you
do not have the desired version, tfenv will install it for you.

Setting Up Provider

The first thing we need to do is to set up the provider. I'm using the latest version of the provider. You can find the latest version here:

provider "google" {
  project = var.project_id
  region  = var.region
  zone    = var.zone
}

terraform {
  required_version = ">=1.5.5"

  required_providers {
    google = {
      source  = "hashicorp/google"
      version = "4.77.0"
    }
  }

  backend "local" {
      path = "local-state.tfstate"
  }
}

I'm using the local backend for now. I plan to switch to the gcs backend later. The local backend is used for storing the Terraform state locally. The state file is called local-state.tfstate. And it will contain the Terraform state for the base. It is not encrypted. My plan is to keep it safe and encrypted.

Block terraform sets the required Terraform provider version, provider, and backend. You can find more here.

The provider block configures the provider by setting the project ID, region, and zone. You probably noticed that I'm using variables.

Setting Up Variables

I'm using variables for defining variables I'll be using in the base. I'm using the following variables:

variable "project_id" {
  description = "project id"
  type        = string
}

variable "region" {
  description = "default region"
  type        = string
}

variable "zone" {
  description = "default zone"
  type        = string
}

Once you defined the variables, you need to set them. You can do that in the terraform.tfvars file:

project_id = "my-project"
region     = "europe-west1"
zone       = "europe-west1-b"

You should avoid committing the terraform.tfvars file. It may contain sensitive information. I'll show you how to handle that later. But for now, you must create the terraform.tfvars file and set the variables. For the project_id variable, you need to set the project ID you created earlier. Choose the region and zone that is closest to you. You can find the list of regions and zones here.

Regions&Zones Important?
Is this important? Yes, it is.

You need to choose the region and zone carefully, considering reliability, availability, and cost into account. For your users to have the best experience, your servers must be close to them and deliver the content quickly. It would be best if you considered the cost, too.

To learn more about choosing the correct region and zone, you can find more here in Google documentation.

Init, Plan, and Apply

At this moment, I can initialize the Terraform workflow:

$ terraform init


Initializing the backend...

Successfully configured the backend "local"! Terraform will automatically
use this backend unless the backend configuration changes.

Initializing provider plugins...
- Finding hashicorp/google versions matching "4.77.0"...
- Installing hashicorp/google v4.77.0...
- Installed hashicorp/google v4.77.0 (signed by HashiCorp)

Terraform has created a lock file .terraform.lock.hcl to record the provider
selections it made above. Include this file in your version control repository
so that Terraform can guarantee to make the same selections by default when
you run "terraform init" in the future.

Terraform has been successfully initialized!

You may now begin working with Terraform. Try running "terraform plan" to see any changes that are required for your infrastructure. All Terraform commands
should now work.

If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.

This will create a directory .terraform and file .terraform.lock.hcl file. You should include .terraform.lock.hcl in your commit. It is used to lock the provider version. If you run terraform init again, it will use the same provider version. The folder .terraform stores the provider plugins and other files required for Terraform to work.

Next, I can run terraform plan:

$ terraform plan

No changes. Your infrastructure matches the configuration.

Terraform has compared your real infrastructure against your configuration and found no differences, so no changes are needed.

This will show me what Terraform will do. Since I did not create any resources, it will not do anything. But it is good to run terraform plan before terraform apply.

Next, I can run terraform apply:

$ terraform apply

No changes. Your infrastructure matches the configuration.

Terraform has compared your real infrastructure against your configuration and found no differences, so no changes are needed.

Apply complete! Resources: 0 added, 0 changed, 0 destroyed.

Well, this is expected. I did not create any resources. If there are any changes, Terraform will show you what it will do and ask you to confirm them. If you want to apply the changes, type yes
and hit enter.

Setting Up The Bucket

From the terraform init output, I can see that the backend is configured. It is configured to use the local backend. My goal is to use the gcs backend. That means I want to store the Terraform state file in the GCP bucket. Let's look at how to do it:

data "google_storage_project_service_account" "gcs_account" {
}

resource "google_kms_key_ring" "tf_states" {
  name     = "terraform-state-key-ring"
  location = "us"

  lifecycle {
    prevent_destroy = true
  }
}

resource "google_kms_crypto_key" "tf_states" {
  name            = "terraform-state-key"
  key_ring        = google_kms_key_ring.tf_states.id
  rotation_period = "100000s"

  lifecycle {
    prevent_destroy = true
  }
}

resource "google_kms_crypto_key_iam_binding" "binding" {
  crypto_key_id = google_kms_crypto_key.tf_states.id
  role          = "roles/cloudkms.cryptoKeyEncrypterDecrypter"

  members = ["serviceAccount:${data.google_storage_project_service_account.gcs_account.email_address}"]
}

resource "google_storage_bucket" "tf_states_bucket" {
  name          = "terraform-states-${var.project_id}"
  force_destroy = false
  location      = "us"
  storage_class = "STANDARD"
  versioning {
    enabled = true
  }
  encryption {
    default_kms_key_name = "your-crypto-key-id"
  }

  depends_on = [google_kms_crypto_key_iam_binding.binding]
  lifecycle {
    prevent_destroy = true
  }
}

When working with Terraform, you are calling the API of the provider. With each call, you are pushing configuration to the provider. The provider will then create, update, or delete your resources.

That is a lot of code. Let me explain.

First, I'm creating a Key(google_kms_crypto_key) and Key Ring(google_kms_key_ring) for encrypting the Terraform state file. These resources are coming from Google KMS. The service allows you to work with cryptographic keys and execute cryptographic operations.

Next is adding the Role roles/cloudkms.cryptoKeyEncrypterDecrypter to the Service Account. Which is needed to encrypt/decrypt the state file. In the end, I'm creating a bucket for storing the Terraform state file, and I'm using the Key for encrypting the Terraform state file. There is a small gotcha. I need a Service Account for encrypting the Terraform state file. But it does not exist
yet. So, I'll use an automatic Google Cloud Storage service account to get it when the resource is created. I'm using a data resource for that.

I'm using lifecycle.prevent_destroy to ensure the resources are not accidentally deleted.

If I now run the terraform plan I'll see the following:

$ terraform plan
data.google_storage_project_service_account.gcs_account: Reading...
data.google_storage_project_service_account.gcs_account: Read complete after 1s [id=service-964026605440@gs-project-accounts.iam.gserviceaccount.com]

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  # google_kms_crypto_key.tf_states will be created
  + resource "google_kms_crypto_key" "tf_states" {
      + destroy_scheduled_duration = (known after apply)
      + id                         = (known after apply)
      + import_only                = (known after apply)
      + key_ring                   = (known after apply)
      + name                       = "terraform-state-key"
      + purpose                    = "ENCRYPT_DECRYPT"
      + rotation_period            = "100000s"
    }

  # google_kms_crypto_key_iam_binding.binding will be created
  + resource "google_kms_crypto_key_iam_binding" "binding" {
      + crypto_key_id = (known after apply)
      + etag          = (known after apply)
      + id            = (known after apply)
      + members       = [
          + "serviceAccount:service-964026605440@gs-project-accounts.iam.gserviceaccount.com",
        ]
      + role          = "roles/cloudkms.cryptoKeyEncrypterDecrypter"
    }

  # google_kms_key_ring.tf_states will be created
  + resource "google_kms_key_ring" "tf_states" {
      + id       = (known after apply)
      + location = "us"
      + name     = "terraform-state-key-ring"
      + project  = (known after apply)
    }

  # google_storage_bucket.tf_states_bucket will be created
  + resource "google_storage_bucket" "tf_states_bucket" {
      + force_destroy               = false
      + id                          = (known after apply)
      + labels                      = (known after apply)
      + location                    = "US"
      + name                        = "terraform-states-project-id"
      + project                     = (known after apply)
      + public_access_prevention    = (known after apply)
      + self_link                   = (known after apply)
      + storage_class               = "STANDARD"
      + uniform_bucket_level_access = (known after apply)
      + url                         = (known after apply)

      + encryption {
          + default_kms_key_name = "your-crypto-key-id"
        }

      + versioning {
          + enabled = true
        }
    }

Plan: 4 to add, 0 to change, 0 to destroy.
─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────

Note: You didn't use the -out option to save this plan, so Terraform can't guarantee to take exactly these actions if you run "terraform apply" now.

Ok, let's try to apply the changes:

$ terraform apply

...

Plan: 4 to add, 0 to change, 0 to destroy.

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

google_kms_key_ring.tf_states: Creating...

Error: Error creating KeyRing: googleapi: Error 403: Google Cloud KMS API has not been used in project 3534534535 before or it is disabled. Enable it by visiting https://console.developers.google.com/apis/api/cloudkms.googleapis.com/overview?project=3534534535 then retry. If you enabled this API recently, wait a few minutes for the action to propagate to our systems and retry.

  with google_kms_key_ring.tf_states,
  on main.tf line 4, in resource "google_kms_key_ring" "tf_states":
   4: resource "google_kms_key_ring" "tf_states" {

I had to cut output. But you can see the error. I need to enable the Cloud KMS API. This will be a standard message if you try to use the API, which is not enabled. You can enable it by following the link in the error message. Or you can enable it with the following command:

$ gcloud services enable cloudkms.googleapis.com

gcloud CLI useful commands
You can see all the APIs you enabled with the following command:

$ gcloud services list --enabled

And list all available APIs with the following command:

$ gcloud services list --available

I already have enabled Google Storage API. But if you did not, you need to enable it, too. Try to figure out how. Ok, let's try again:

$ terraform apply
...

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

google_kms_key_ring.tf_states: Creating...
╷
│ Error: Error creating KeyRing: googleapi: Error 409: KeyRing projects/project-id/locations/us/keyRings/terraform-state-key-ring already exists.
│
│   with google_kms_key_ring.tf_states,
│   on main.tf line 4, in resource "google_kms_key_ring" "tf_states":
│    4: resource "google_kms_key_ring" "tf_states" {

I was impatient and ran terraform apply again. Google API responded that it was not enabled, but actually, it was. And not only that, it created the Key and KeyRing. This situation is rare. But it can happen. We can fix it by importing created resources. Let's do this:

$ terraform import google_kms_key_ring.tf_states project-id/us/terraform-state-key-ring

How do I know that? Well, I read the error message and the provider's documentation. You'll have a section Import in Terraform documentation for each resource.
There, you'll be able to see how to import it.

Now, let's try again:

$ terraform apply

...
Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

data.google_storage_project_service_account.gcs_account: Reading...
google_kms_key_ring.tf_states: Refreshing state... [id=projects/project-id/locations/us/keyRings/terraform-state-key-ring]
data.google_storage_project_service_account.gcs_account: Read complete after 0s [id=service-964026605440@gs-project-accounts.iam.gserviceaccount.com]
google_kms_crypto_key.tf_states_key: Refreshing state... [id=projects/project-id/locations/us/keyRings/terraform-state-key-ring/cryptoKeys/terraform-states-key]
google_kms_crypto_key_iam_binding.binding: Refreshing state... [id=projects/project-id/locations/us/keyRings/terraform-state-key-ring/cryptoKeys/terraform-states-key/roles/cloudkms.cryptoKeyEncrypterDecrypter]
google_storage_bucket.tf_states_bucket: Creating...
google_storage_bucket.tf_states_bucket: Creation complete after 2s [id=terraform-states-project-id]

The bucket is created. You'll see it if you go to GCP console. You can check bucket properties and configuration there in the console or with the following command:

$ gsutil ls -L -b gs://terraform-states-project-id

You can see that the bucket is encrypted with the Key we created. And to check the Key properties and configuration in the console or with the following command:

$ gcloud kms keys describe terraform-states-key --location us --keyring terraform-state-key-ring

You can list the content of the bucket with the command:

$ gsutil ls gs://terraform-states-project-id

But you'll get nothing because it is empty. I still need to move Terraform's state file to the bucket.

Setting Up The Backend

If you remember, I'm using the local backend. If you check the local directory, you'll see a file called local-state.tfstate. That is the Terraform state file. And that file I want to move to
the bucket and encrypt it there. I can do that with the following code:

  backend "gcs" {
    bucket  = "terraform-states-project-id"
    prefix  = "terraform/state"
  }

And removing this one in provider.tf:

  backend "local" {
    path = "local-state.tfstate"
  }

Now, if I run terraform init, I'll see the following:

terraform init

Initializing the backend...
╷
│ Error: Backend configuration changed
│
│ A change in the backend configuration has been detected, which may require migrating existing state.
│
│ If you wish to attempt automatic migration of the state, use "terraform init -migrate-state".
│ If you wish to store the current configuration with no changes to the state, use "terraform init -reconfigure".
╵

Well, let's do as it says:

$ terraform init -migrate-state

When asked, type yes and wait to finish. Let's check if we have anything in the bucket:

$ gsutil ls gs://terraform-states-project-id/terraform/state

You'll see that the bucket has a file called default.state. That is the Terraform state file. It is encrypted. You can check it with the following command:

$ gsutil cat gs://terraform-states-project-id/terraform/state/default.tfstate | jq .

DevCube | Robert Nemet | Substack

Weekly rant about software design, devops, kubernetes, sre... Click to read DevCube, by Robert Nemet, a Substack publication. Launched 6 months ago.

rnemet.substack.com

Conclusion

With I just set up a project. With it, I can start exploring GCP. We touched basic Terraform CLI commands(init, plan, apply, and import), Terraform variables, provider, backend, and resources. I hope you learned something new. I know I did. I'll continue exploring GCP with Terraform in the next article.

References

Task vs Make - Final Thoughts

Robert Nemet — Thu, 10 Aug 2023 05:30:00 +0000

So, after spending time moving from Make to Task, I decided. I'm going with Task in my future projects. Make will not be removed from my projects immediately, but I will not use it in the future. Let me tell you why.

All this is my personal opinion. I'm not saying that Task is better than Make. I think that Task is better for me.

Global Taskfile

I have a global Task in my home directory for standard stuff. It contains tasks to update my tools, run some diagnostics, etc. Because it is global, it is accessible from any directory. For example:

version: '3'

tasks:
  build:
    dir: '{{.USER_WORKING_DIR}}'
    cmds:
      - go build . -o app

In the above example, task -g build will build the Golang app in my current working directory. I do not need to add Task for every project. I must add Taskfile to the project if I share it. But, as long it is experimental, there is no need. Anyway, storing everyday tasks as global is a nice feature.

The {{.USER_WORKING_DIR}} is a variable set by Task. It is the current working directory. And the dir slug defines the folder where the task will be executed. In this case, it is the current working directory. But you can set it to any folder you want. So you do not need to switch directories to run a task. Just set the dir slug to the folder where you want to run the job. Neat!

Including/Grouping Task Files

The explicit inclusion of other Taskfiles with namespace is really neat. This way, you get a context of tasks. For example:

version: '3'

includes:
  tools: ./Taskfile.tools.yml
  docs: ./docs/

Assuming I have the task install_tools in the file Taskfile.tools.yml, I can run it with task tools:install_tools. Also, suppose I have Taskfile.yml file in the docs directory. I can run it with task docs:build_docs from the root directory. This way, I'm groping tasks by context.

In case you have a CI pipeline and need to do builds for different platforms. You can split your tasks into different files with the platform postfix. For example, you have different tasks for linux and windows platforms. You can split them into separate files. I name them Taskfile_linux.yml and Taskfile_windows.yml. Then you can include them in the main Taskfile.yml file with the following:

version: '3'

includes:
  build: ./Taskfile_{{OS}}.yml

If you run task build:build on the linux platform, it will include Taskfile_linux.yml and run the build task from it. If you run the same task on windows platform, it will consist of Taskfile_windows.yml and run the build job from it. This way, you can have different tasks for different platforms.

Task Dependencies and Skipping Done Tasks

Task dependencies are really nice. You can define a task that depends on other tasks. For example:

version: '3'

tasks:
 manifests:
    desc: Generate manifests e.g. CRD, RBAC etc.
    deps:
      - task: tools:controller-gen
    cmds:
      - controller-gen rbac:roleName=manager-role crd webhook paths="./..." output:crd:artifacts:config=config/crd/bases
    sources:
      - main.go
      - apis/**/*.go
      - pkg/**/*.go
    generates:
      - config/crd/bases/**/*.yaml

The manifests task in the above example depends on the tools:controller-gen task. Dependency tasks are run in parallel. This way, you can define tasks that rely on other tasks.

Also, the above example shows how to define sources and generated files. The sources slug defines files used to generate other files. The generates slug defines files that are generated by the task. This way, you can tell Task which files are used to generate other files. It is to determine whether the task needs to be run. Task will generate a fingerprint for sources and compare it next time the task
needs to be run. The task is considered done if the fingerprint is not changed, assuming the generated files are present.

Also, there is an alternative with a status slug. You can define a command that will return 0 if the task is done. For example:

  install-cert-manager:
    desc: Install cert-manager    
    deps:
      - init-cluster
    cmds:
      - kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/{{.CERT_MANAGER_VERSION}}/cert-manager.yaml
      - echo "Waiting for cert-manager to be ready" && sleep 25
    status:
      - kubectl -n cert-manager get pods | grep Running | wc -l | grep -q 3

Golang Template Engine

Task uses the Golang template engine. As a Golang user, this is a big plus. I can use all the features of the Golang template engine plus one that Task provides, like including functions from https://go-task.github.io/slim-sprig/. For example:

version: '3'

tasks:
 print-date:
    cmds:
      - echo {{now | date "2006-01-02"}}
      - echo '{{OS}} {{ARCH}}'

Format Is YAML

I like it when a file has a structure and text editors can understand it. I can understand it as well. It is enough said.

Conclusion

There are other things that I like about Task but I will not go into details. I have covered the most important ones, at least for me.

While evaluating Makefile and Task, I discovered another tool called Just. It is more similar to Make than Task. For that reason, I did not want to spend a lot of time on it. But I see it as a good alternative to Make. Especially if you do not like YAML.

Let me know what you think. Thanks for reading. Enjoy!

References

WIP: Taskfile instead of Makefile?

Robert Nemet — Tue, 01 Aug 2023 05:30:00 +0000

Recently, I stumbled upon a tool called Taskfile. It is a task runner, similar to Makefile, but with many improvements. I decided to try it and see if it can replace my Makefiles.

Why Taskfile?

First of all, Makefile is a good tool. Today, you can use it on any platform, not just Linux. It is a standard tool for running tasks. It is flexible and powerful. My relation with it is love/hate. It can be tricky to write a good Makefile or to debug it. Reading also can be a challenge. Because of that, when I found out about Taskfile, I decided to try it.

First impressions

Taskfile is written in Go. It is a single binary that you can download and use. If you can run Go programs, you can run Taskfile. It is a big plus for me. See docs on how to install it.

Taskfile is a YAML file. Most of us are familiar with YAML, and many tools use it. It is easy to read, write, and parse, and IDEs have good support. Sometimes, it feels like we are programming in YAML. If you do not believe me, check out your CI/CD pipeline. I bet it is written in YAML.

You can use auto-completion if you use ZSH, Bash, Fish, or PowerShell. It is a big plus for me. I like to have auto-completion for my tools. It makes my life easier, especially when you start adding more and more tasks. When you have a lot of tasks, they can be grouped into namespaces, hidden from a user(internal task), global, or local, etc.

Challenge: automate testing for Klock

Klock is my pet project. It is a ValidatingAdmissionWebhook for Kubernetes. It enables locking Kubernetes resources. You can read more here Klock - Kubernetes locking.

One of the things that was on my to-do list was to automate testing. I'm doing it locally because it is easier and faster. But still, it is cumbersome to start the Kind cluster, deploy Klock, and run tests. I decided to use Taskfile to automate it.

What I need to do is:

Start Kind cluster
Install Cert Manager
Build K8s manifests
Build the Docker image if there are any code/configuration changes
Load the image into a Kind cluster
Run tests

You can find my initial work on the git branch migrate_taskfile. If you compare it with the main branch, you'll notice I added a new file called Taskfile.yml. It is a main Taskfile. Also, I added the folder tasks with two files, Taskfile.testing.yml and Taskfile.tools.yml. They are imported into the main Taskfile. The idea is to have a main Taskfile that will import other Taskfiles, while other Taskfiles will contain specific tasks.

Start Kind cluster

First, I need to start the Kind cluster:

version: '3'

vars:
  KIND_VERSION: "v0.20.0 go1.20.5"
  CERT_MANAGER_VERSION: v1.12.0
  IMG: controller:latest

tasks:
  init-cluster:
    desc: Create a kind cluster named klock
    preconditions:
      - sh: kind version | grep "{{.KIND_VERSION}}"
        msg: "kind version does not match {{.KIND_VERSION}}. Please install right version of kind"
    cmds:
      - kind create cluster --name klock
    status:
      - kind get clusters | grep klock

The slug tasks contain a list of tasks. This task is called init-cluster. It has a description, preconditions, commands, and status. For any task name and cmds slugs are mandatory. Others are optional. What they do:

cmds - list of commands that will be executed
desc - description of the task, it will be shown when you run list tasks: task -l
preconditions - list of preconditions that must be met before running the task. If any of the preconditions fail, the task will not be executed. In this case, I check if the Kind version is the same as the one I'm using. If not, the task will fail with the message.
status - list of commands that will be run to see if a task can be considered done. That means the task is done if the Kind cluster named klock exists. It does not care how a cluster is created. It just checks if it exists.

Notice block vars. It is a list of variables that can be used in the Taskfile. In this case, I'm using KIND_VERSION to check if the Kind version is the same as the one I'm using. The Taskfile uses [Go template (https://golang.org/pkg/text/template/) to render variables.

Install Cert Manager

Next, I need to install Cert Manager:

  install-cert-manager:
    desc: Install cert-manager    
    deps:
      - init-cluster
    cmds:
      - kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/{{.CERT_MANAGER_VERSION}}/cert-manager.yaml
      - echo "Waiting for cert-manager to be ready" && sleep 25
    status:
      - kubectl -n cert-manager get pods | grep Running | wc -l | grep -q 3

The new thing here is the deps slug. It is a list of tasks that must be executed before this task. In this case, I need to create a Kind cluster before I can install Cert Manager. Again, I'm using a variable CERT_MANAGER_VERSION to install the correct version of Cert Manager.

Build K8s manifests

To generate manifests, I'm using kubebuilder tool contrller-gen:

 manifests:
    desc: Generate manifests e.g. CRD, RBAC etc.
    cmds:
      - controller-gen rbac:roleName=manager-role crd webhook paths="./..." output:crd:artifacts:config=config/crd/bases
    sources:
      - main.go
      - apis/**/*.go
      - pkg/**/*.go
    generates:
      - config/crd/bases/**/*.yaml

This time I'm using sources and generates slugs:

sources is a list of files that will be used to generate manifests
generates is a list of files that will be generated

In this case, Taskfile will generate a checksum for sources from the previous run. If the checksum is the same, it will not run the task. It will just print out that the task is done. If the checksum is different, it will run the task. It is a nice feature that can save you some time.

Deploy Klock to Kind cluster

To deploy Klock to the Kind cluster, I'm using kustomize:

  deploy:
    desc: Deploy the controller to the kind cluster and wait for it to be ready. Use IMG to specify image name
    deps:
      - install-cert-manager
      - manifests
      - docker-build
    cmds:
      - task: kind-load
        silent: true
      - cd config/manager && kustomize edit set image controller={{.IMG}}
      - kustomize build config/default | kubectl apply -f -
      - echo "Waiting for controller to be ready" && sleep 25
    status:
      - kubectl get pods -n klock-system | grep klock-controller | wc -l | grep -q 1

This task depends on install-cert-manager, manifests, docker-build, and kind-load tasks. It means that those tasks have to be executed before this task.

But a small catch, kind-load can be done only if docker-build is done successfully. Other tasks can be executed in parallel. That is why other tasks are in the deps slug. Tasks in the deps slug will be executed in parallel. Tasks will be executed in order while in the cmds slug.

I'm adding a new slug, silent to suppress the task output.

This way, I managed to do some tasks in parallel and speed up the process.

Run tests

To run tests I'm using KUTTL:

  ktest:
    desc: Run kuttl tests. Specify image name with IMG
    cmds:
      - task: deploy
        silent: true
      - kubectl kuttl test

There is nothing special here. Except I should move the deploy task to the deps slug. But this is still a work in progress.

Putting It All Together

My main Taskfile looks like this:

# https://taskfile.dev

version: '3'

includes:
  tools: tasks/Taskfile.tools.yml
  tests: tasks/Taskfile.testing.yml

tasks:
  default:
    silent: true
    cmds:
      - echo "Welcome to Klock!"
      - task -l

I'm including two other Taskfiles. One is for tools, and the other is for tests. The default task is to print out a welcome message and list all tasks. Invoking the task without any arguments, it will run the default task:

$ task
Welcome to Klock!
task: Available tasks for this project:
* tests:cleanup:                    Delete the kind cluster named klock
* tests:deploy:                     Deploy the controller to the kind cluster and wait for it to be ready. Use IMG to specify image name
* tests:docker-build:               Build the docker image, specify image name with IMG
* tests:docker-push:                Push the docker image, specify image name with IMG
* tests:init-cluster:               Create a kind cluster named klock
* tests:install-cert-manager:       Install cert-manager
* tests:kind-load:                  Load the docker image into the kind cluster, specify image name with IMG
* tests:ktest:                      Run kuttl tests. Specify image name with IMG
* tests:manifests:                  Generate manifests e.g. CRD, RBAC etc.
* tests:publish:                    Run tests and publish the docker image. Specify image name with IMG and version with VERSION
* tests:undeploy:                   Undeploy the controller from the kind cluster
* tools:controller-gen:             Download controller-gen locally if necessary.
* tools:create-localbin:            Create the localbin directory
* tools:delete-kustomize:           Delete kustomize
* tools:envtest:                    Download envtest-setup locally if necessary.
* tools:install-kustomize:          Install kustomize

To run the tests, I'm using task tests:ktest IMG=rnemet/klock:test. It will run the tests:ktest task and pass the image name to the task. This image name is propagated to other tasks that need it. As you can see, I grouped tasks into namespaces. I'm using the tests namespace for tasks related to tests and the tools namespace for tasks related to tools.

Conclusion

I'm still learning Taskfile. There are better ways to do things. But I'm pleased with the result. Still, I need to migrate other Makefile targets to Taskfile. But the results are promising. Changes are more readable, easier to maintain and to understand. When I migrate the rest of the Makefile targets, I can compare Taskfile and Makefile with more insights. But at the moment, Taskfile is a winner.

Looking into Taskfile Github repo, I can see that the project is active, and much work is underway. I'm looking forward to new features and improvements.

This is a work in progress. I already have ideas on how to make it better. But I will leave that for another blog post. Meanwhile, you can check Taskfile and let me know what you think. Thanks for reading.

References

Taskfile

Learning eBPF: Maps, Ring Buffers and Output

Robert Nemet — Sat, 22 Jul 2023 16:14:51 +0000

I set the stage for learning eBPF. As mentioned in the previous post, eBPF is a technology that allows us to run code in the kernel.
This is a compelling technology, but it comes with a few limitations. One of them is that we can't use the standard output to print messages. At least not directly. Let's explore how we can do this.

Why can't I use the standard input/output?

Let's look at this picture

The eBPF programs are executed in the kernel. The kernel is the core of the operating system. Other processes exist there that ensure the smooth operation of the system. To make this possible, the kernel is running in the privileged mode.

Programs run by the user are executed in the user space. The user space is running in the unprivileged mode. When the user space program wants to communicate with the kernel, it needs to use the system calls. The system calls are the interface between the user space and the kernel. The system calls are the only way to communicate with the kernel. The kernel is not exposing any other interface to the user space.

Because of that, we need to have the correct permissions for injecting, executing, and reading the eBPF programs. You must already notice that when you are running the simple hello.py script, you need to have root permissions. The root permissions have enough privileges
to communicate with the kernel. In this case, load the eBPF program in the kernel, attach it to the events stream, and execute it.

This is an answer to why we can't use the standard output directly. The standard output is a user space concept. But we can communicate with eBPF programs using the system data structures and calls. They are designed to allow communication between the user space and the kernel, and other eBPF programs can use them to exchange data. Let's scratch the surface of this topic.

Simple output(bpf_trace_printk)

Let's look at this BCC example:

#!/usr/bin/python3
from bcc import BPF

program = r"""
#include <linux/sched.h>

int hello(void *ctx) {

    int pid = bpf_get_current_pid_tgid() >> 32;
    int uid = bpf_get_current_uid_gid() & 0xFFFFFFFF;
    char command[TASK_COMM_LEN];
    bpf_get_current_comm(command, sizeof(command));

    bpf_trace_printk("uid = %d, pid = %d, comm %s", uid, pid, command);

    return 0;
}
"""

b = BPF(text=program, cflags=["-Wno-macro-redefined"])
syscall = b.get_syscall_fnname("execve")
b.attach_kprobe(event=syscall, fn_name="hello")

b.trace_print()

This simple program prints the uid, pid, and the command name of the process executing the execve system call. I use:

The bpf_get_current_pid_tgid function to get the pid(process id) of the process
The bpf_get_current_uid_gid function to get the uid(user id) of the process
The bpf_get_current_comm function to get the command name of the process

The bpf_trace_printk is a function that allows us to print messages from the eBPF program. The messages are sent to the predefined pseudo-file on location /sys/kernel/debug/tracing/trace_pipe. So, bpf_trace_printk helper function sends my messages to the trace_pipe file. The trace_pipe file is a special file that is used by the kernel to send messages to the user space. Then I can read them with the trace_print() function or with the cat command.

In the above recording, you'll notice whenever I execute the execve system call, the eBPF program prints the message on the top third of the screen. The messages are in the trace_pipe file, too. Notice the middle part of the screen where cat /sys/kernel/debug/tracing/trace_pipe is executed. At the bottom, I run simple commands like the regular user and root. Notice the difference in the messages. The messages that are printed by the root user have the uid 0. Also, notice that opening a new shell session create several message output. There are many messages because several actions are done when opening a new shell.

While working bpf_trace_printk is easy, it has some limitations. For example, if you have multiple eBPF programs that are printing messages, all the messages will be mixed in the trace_pipe file. It is hard to distinguish which message is coming from which program. Also, the trace_pipe file is a special file designed for debugging purposes. It is not intended for production use. So, we must find a better way to communicate with the user space.

I find it helpful to use bpf_trace_printk for debugging purposes. It is a quick way to print messages from the eBPF program.

Maps

Maps as data structures are used to store data. When it comes to eBPF, maps are data structures that are used to exchange data between the user space and the kernel.
Also, maps are used to exchange data between eBPF programs. In general, maps are key-value stores. But still, different types of maps exist. The reason is that some of them are optimized for different use cases. At the same time, other eBPF maps hold information about specific object types. For example, there is the BPF_MAP_TYPE_QUEUE map, which is optimized as a FIFO(first in, first out) queue, and BPF_MAP_TYPE_STACK which provides a LIFO(last in, first out) stack. Check linux docs
on them for more information.

Or maps used to hold information about network devices.

Let's check this example:

#!/usr/bin/python3  
from bcc import BPF
from time import sleep

program = r"""
struct data_t {
   u64 counter;
   int pid;
   char command[16];
};

BPF_HASH(counter_table, u64, struct data_t);

int hello(void *ctx) {
   struct data_t zero = {};
   struct data_t *val;


   u64 uid = bpf_get_current_uid_gid() & 0xFFFFFFFF;
   int pid = bpf_get_current_pid_tgid() >> 32;

   val = counter_table.lookup_or_try_init(&uid, &zero);
   if (val) {
      val->counter++;
      val->pid = pid;
      bpf_get_current_comm(&val->command, sizeof(val->command));
   }

   return 0;
}
"""

b = BPF(text=program, cflags=["-Wno-macro-redefined"])

syscall = b.get_syscall_fnname("execve")
b.attach_kprobe(event=syscall, fn_name="hello")


old_s = ""
while True:
   sleep(2)
   s = ""
   for k,v in b["counter_table"].items():
      s += f"ID {k.value}: cnt: {v.counter} pid: {v.pid} comm: {v.command}\t"
   if s != old_s:
      print(s)
   old_s = s

The above example is similar to the previous one. But this time, I'm using the map to store information about the processes. I use the BPF_HASH map to create the map
counter_table. I use the uid of the process for a key, and for value, I use the struct data_t structure. The struct data_t structure holds the counter for all commands the user executed, pid, and the command name of the last process run by a user.

The hello function is attached to the execve system call. When the execve system call is executed, I check if the user is already in the map. If not, I add the user to the map. Suppose the user is already in the map. In that case, I increment the counter and update the pid and the command name of the last process executed by the user.

Later, in Python script, I read the map and print the information about the users. I use the items() function to iterate over the map. The items() function returns the key and value for each entry in the map. I use the value to get the struct data_t structure. Then, I print the information about the user.

Ring buffers

Like maps, ring buffers are data structures that exchange data between the user space and the kernel. Look at the picture:

In short, ring buffers are circular buffers. They have two pointers: one for reading and one for writing. The pointers are moving in the same direction. If the read pointer catches the write pointer, the buffer is empty. If the write pointer catches the read pointer, the buffer is full. Then, the next element to be written will be dropped.

There are two types of ring buffers BPF_PERF_OUTPUT and BPF_RINGBUF_OUTPUT. The BPF_RINGBUF_OUTPUT is more advanced than BPF_PERF_OUTPUT. I will not go into the details about the differences between them please check the docs.

Here is a familiar example:

#!/usr/bin/python3  
from bcc import BPF

program = r"""
BPF_PERF_OUTPUT(counter_table); 

struct data_t {     
   int pid;
   int uid;
   char command[16];
};


int hello(void *ctx) {
   struct data_t data = {}; 

   data.pid = bpf_get_current_pid_tgid() >> 32;
   data.uid = bpf_get_current_uid_gid() & 0xFFFFFFFF;

   bpf_get_current_comm(&data.command, sizeof(data.command));

   counter_table.perf_submit(ctx, &data, sizeof(data)); 

   return 0;
}
"""

b = BPF(text=program, cflags=["-Wno-macro-redefined"]) 
syscall = b.get_syscall_fnname("execve")
b.attach_kprobe(event=syscall, fn_name="hello")

def print_event(cpu, data, size):  
   data = b["counter_table"].event(data)
   print(f"{data.pid} {data.uid} {data.command.decode()}")

b["counter_table"].open_perf_buffer(print_event) 
while True:   
   b.perf_buffer_poll()

This time difference is when reading from the ring buffer, I'm passing the callback print_event. The print_event callback is called when the data is available in the ring buffer. It has to have three arguments: cpu, data, and size. The cpu argument is the cpu number on which the event was generated. The data argument is the data that is read from the ring buffer. The size argument is the size of the data read from the ring buffer.

Summary

So, even if eBPF has no direct access to the standard output, there are ways to exchange data between the user space and the kernel. Plus, given data structures, maps are usually optimized for the specific use case. That should make your life easier.

References

Learning eBPF: Setting up the environment

Robert Nemet — Tue, 09 May 2023 07:18:18 +0000

For a while, I've been following stuff around eBPF, and it is very promising. What I just wrote is an understatement. At first glance, eBPF is bringing many new possibilities to our toolbox. You can start with performance profiling, tracing, security, networking, etc. But let's start from the beginning.

By the way, I'm doing this on OSX. For eBPF, you need Linux kernel 4.1 or newer. So, I'll be running some VMs. This setup should be doable on Linux too. Code is available here.

What is eBPF?

A lit bit of history

A long time ago, in the last century (1992), Steven McCanne and Van Jacobson wrote the paper The BSD Packet Filter. In short, the goal was to tap on a network interface and filter packets. The result is the in-kernel virtual machine that is used to filter packets. This virtual machine is called Berkeley Packet Filter (BPF). It is not only used for filtering packets but also for tracing, performance profiling, security, etc.

In 2014, Alexei Starovoitov and Daniel Borkmann started to work on extending BPF. That is how extended BPF (eBPF) was born. eBPF is not only used for filtering packets but as mentioned, eBPF is used for tracing, performance profiling, security, etc.

eBPF virtual machine

EBPF program compiles to eBPF bytecode. This bytecode is loaded into the kernel, verified, and then executed by eBPF virtual machine. It uses different maps to communicate with the kernel, store data, and expose data to user space. Above is how eBPF works, in short.

eBPF program is event-driven. It is attached to some hook and executed when that hook is triggered. There are many different hooks that are used, like internal system calls, functions entry and exit, network events, etc.

We write eBPF programs in pseudo-C code and then use LLVM to compile it eBPF bytecode. Or we use eBPF programs with projects like Cilium, bcc, and bpftrace. They are abstractions on the top of eBPF and make our life easier. After the program is compiled to eBPF bytecode, it is loaded into the kernel and verified. If everything is ok, eBPF bytecode is translated to native code by the JIT compiler and executed.

In the above image, there are eBPF maps. The eBPF programs use them to collect, store and share data. There are different types of eBPF maps, like hash table, array, ring buffer, etc.

Helper functions, tail, and function calls are not on the image. Helper functions are used to access kernel functions. Tail calls are used to call other eBPF programs. Function calls are used to call other functions inside the eBPF program.

Enough theory. Let's start with some practice...

Setting up the environment (OSX)

I'm doing this on OSX. But, even if I do this on a Linux box, I still use VMs. Why? Because I would like to have a clean environment and not mess up my host.

First, I set up a small Git project for messing stuff. I'll be using it for all my experiments. All code will be in this repo. And I'll be using VS Code or Community Edition of IntelliJ IDEA.

I aim to set up VM(s) to run code and IDE on my host. I should be able to easily share files and run code on VM(s).

Vagrant

Vagrant would be a CLI tool for managing VMs. But under the hood, it uses VirtualBox. So, you can use Vagrant and VirtualBox. I'm not VirtualBox fun, but I'll give it a try.

I want to use BCC (BPF Compiler Collection) on the first try. BCC is a toolkit for creating efficient kernel tracing and manipulation programs. It is based on eBPF. It is written in Python. So, my Vagrantfile looks like this:

# -*- mode: ruby -*-
# vi: set ft=ruby :

$script = <<-SCRIPT
  echo "Provisioning..."
  sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 4052245BD4284CDD
  echo "deb https://repo.iovisor.org/apt/$(lsb_release -cs) $(lsb_release -cs) main" | sudo tee /etc/apt/sources.list.d/iovisor.list
  sudo apt-get update
  sudo apt-get -y install bcc-tools libbcc-examples linux-headers-$(uname -r)
SCRIPT

Vagrant.configure("2") do |config|
  config.vm.box = "ubuntu/bionic64"
  config.vm.provision "shell", inline: $script
end

If you run vagrant up you should get Ubuntu 18.04 LTS VM with BCC installed. You can check it with vagrant ssh and then dpkg -l | grep bcc. You should get something like this:

ii  bcc-tools                        0.10.0-1                            all          Command line tools for BPF Compiler Collection (BCC)
ii  libbcc                           0.10.0-1                            all          Shared Library for BPF Compiler Collection (BCC)
ii  libbcc-examples                  0.10.0-1                            amd64        Examples for BPF Compiler Collection (BCC)
ii  libcc1-0:amd64                   8.4.0-1ubuntu1~18.04                amd64        GCC cc1 plugin for GDB
ii  python-bcc                       0.10.0-1                            all          Python wrappers for BPF Compiler Collection (BCC)

Now if you go to /vagrant folder, you'll see your host folder with Vagrantfile. There I keep my files as well. I have hello.py as Hello World example:

Well. I want to skip VirtualBox, and I would like to use the newer Ubuntu. So, I decided to try Lima.

Lima

According to docs, it is best suited to run on OSX. It uses Hypervisor.framework to run VMs. To install it, you can use
brew install lima or see Getting started.

For a start, I'm going with this config:

images:
# Try to use release-yyyyMMdd image if available. Note that release-yyyyMMdd will be removed after several months.
- location: "https://cloud-images.ubuntu.com/releases/22.04/release/ubuntu-22.04-server-cloudimg-amd64.img"
  arch: "x86_64"
- location: "https://cloud-images.ubuntu.com/releases/22.04/release/ubuntu-22.04-server-cloudimg-arm64.img"
  arch: "aarch64"

mounts:
- location: "~"
- location: "/tmp/lima"
  writable: true
provision:
- mode: system
  script: |
    apt-get update
    apt-get install -y apt-transport-https ca-certificates curl clang llvm jq
    apt-get install -y libelf-dev libpcap-dev libbfd-dev binutils-dev build-essential make 
    apt-get install -y linux-tools-common linux-tools-5.15.0-41-generic bpfcc-tools
    apt-get install -y python3-pip 
    rm -rf /usr/local/go && tar -C /usr/local -xzf go1.20.3.linux-amd64.tar.gz

Just run limactl start --name=ubuntu ubuntu-lst-lima.yml and choose Proceed with the current configuration. When it is done, you can run limactl shell ubuntu to get a shell:

Lima for bpftrace

So, in both previous setups, I could run the hello.py example. But I would also like to run bpftrace. For it, I'm creating a separate config:

images:
# Try to use release-yyyyMMdd image if available. Note that release-yyyyMMdd will be removed after several months.
- location: "https://cloud-images.ubuntu.com/releases/22.04/release/ubuntu-22.04-server-cloudimg-amd64.img"
  arch: "x86_64"
- location: "https://cloud-images.ubuntu.com/releases/22.04/release/ubuntu-22.04-server-cloudimg-arm64.img"
  arch: "aarch64"

mounts:
- location: "~"
- location: "/tmp/lima"
  writable: true
provision:
- mode: system
  script: |
    apt-get update
    echo "deb http://ddebs.ubuntu.com $(lsb_release -cs) main restricted universe multiverse deb http://ddebs.ubuntu.com $(lsb_release -cs)-updates main restricted universe multiverse deb http://ddebs.ubuntu.com $(lsb_release -cs)-proposed main restricted universe multiverse" | tee -a /etc/apt/sources.list.d/ddebs.list
    sudo apt install ubuntu-dbgsym-keyring
    sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys F2EDC64DC5AEE1F6B9C621F0C8CAB6595FDFF622
    apt-get update
    apt-get install -y bpftrace 
    apt-get install -y bpftrace-dbgsym libbpfcc-dbgsym

Create VM with limactl start --name=bpftrace bpftrace.yml and run limactl shell bpftrace to get a shell:

Conclusion

I'm inviting you to join me on this journey. I set up two VMs with Ubuntu 18.04 LTS and Ubuntu 22.04 LTS. I was able to run hello.py and bpftrace examples. IMHO, this is good enough for a start. For now, I have all I need to start learning eBPF. If you have any suggestions or comments, please, let me know.

Resources

New Docker Goodies: Init and Watch

Robert Nemet — Tue, 02 May 2023 11:19:44 +0000

Recently Docker brought some new stuff that I found very useful for developers. They are still in the experimental phase but are already very useful.

Try them and give your feedback to the Docker team. The code I'm using for this is available here.

Let's start...

Init

Writing a Dockerfile when starting with a new application is tedious. Why? Because we are repeating most of the stuff over and over again. So, in the end, we reach to copy/paste solution. Of course, copy/paste needs to be modified to fit an application's needs. I do not say that copy/paste is terrible, but with it, we miss new stuff that Docker brings to us. Even seasoned developers miss stuff. If we do not do it every day, we forget. Because of that, the Docker team brought us a new command called init. The init is not new but is still in beta. This time with more improvements.

It is a straightforward command that will create a Dockerfile for us. It asks a few questions, and based on our answers, it will create a Dockerfile. Let us see how it works:

It is effortless to use. Initially, when the Docker team introduced it, only Go was supported. With the latest release(docker desktop 4.19.0 and docker engine v23.0.5)
docker init supports NodeJS and Python, too. I'm sure that they will add more languages in the future.

If you inspect the Dockerfile, you will see that it tries to be simple but also includes all the best practices when writing a Dockerfile. It will try to use cache and multi-stage builds. You can go with generated Dockerfile as is. I prefer to modify it a bit. I suggest you compare my Dockerfile
version with generated one. I like Google's distroless images. In my example, I'm also serving static files, so I need to add them to Dockerfile.

Watch

If you look at the compose.yaml file, you will see nothing exciting. Nothing new or fancy. But there is something new.

When you are working with docker compose and want to run your changes, you need to rebuild the image. So, every time you do something like this:

docker compose -f <compose file> up --detach --build <service name>

But what if you do not need to do it anymore? What if docker compose can do it for you? Well, it can. And it can do things to rebuild your app only when needed or to update static files. In my example, I have a Go server that serves one static file. When I change the code, I need to rebuild the image. But when I change static files, I want to be updated.

See:

services:
  server:
    build:
      context: .
      target: final
    ports:
      - 8080:8080
    x-develop:
      watch:
      - path: ./go.mod
        action: rebuild
      - path: ./main.go
        action: rebuild
      - path: ./static/
        action: sync
        target: /app/static/

The section under x-develop with the name watch does all the magic. That is an experimental feature. It has two actions that can be used: rebuild and sync. The rebuild will rebuild the image, and the sync will sync files from the host to the container.

In the example above, I instruct docker compose to watch for the changes in the go.mod and main.go files and rebuild the image when they change. While if there are any changes in a directory with the name static(notice the trailing slash), sync them to the container. The target specifies where to sync files in the container. In my case, I want to sync them to the /app/static/ directory.

As this feature is experimental, it will not work if you just run docker compose up -d. First, run docker compose up -d, and run docker compose alpha watch when all containers are up and running. That will start watching for changes. If you want to stop watching, press Ctrl+C. Let's see this in action:

Initially, I built and ran a container with docker compose up -d. Then I run docker compose alpha watch, and instruct docker compose to watch for changes. When I modify index.html or main.css, they are synced with the container. When I modify main.go or go.mod the image is rebuilt, and the container is restarted. Try it out yourself.

If you find this helpful feature, please leave your feedback and suggestions here.

Other noticeable stuff

With 4.19.0 release, the Docker engine and CLI are updated to Moby 23.0.
That brings a lot of new stuff. One of the things that can be confusing on start is that docker build is now an alias for docker buildx build. The reason is that Buildx and BuildKit are default builders on Linux and OSX. You will notice differences when building images. You'll see switching blue and white lines in the short demos above. White lines are tasks in progress, while blue ones are completed tasks. As well you'll see that Buildx is trying to run tasks in parallel.

Enjoy!

References

Is Outsourcing Killing Developers

Robert Nemet — Mon, 24 Apr 2023 06:30:00 +0000

I live and work in one of the Balkan countries, Belgrade, Serbia. After the sad '90s, all countries in the region try to catch up with the rest of the world. That led to a boom in IT. A large portion of IT is outsourcing companies. Some companies are domestic, and some are foreign. Also, some companies opened their R&D centers in the region. Small and big as well. There was and still is a large number of startups.

Today, outsourcing is still a big chunk of IT here. Things are changing, but slowly.

My Way of IT

I worked for various companies. From startups to big corporations. From small projects for a handful of people to large ones serving millions of users. On some projects, I had a small role, while on some, my part was noticeable. About two-thirds of those years I spent in startups and outsourcing companies. I think I'm in the middle of my carrier at the moment.

⚠ Small disclaimer. I do not consider myself an expert. I have some experience. I think. Let's say that I like my work
to speak for me. ⚠

These are some principles that I gathered over time:

Things change over time, especially in the business. Do not get attached to your code. It is legacy at the moment you write it.
Prioritize. Everything is doable, but not at the same time.
If you have to work extra time, the plan is wrong, or you need more knowledge. So, either you get trained, or the plan needs to be changed
❗ Know why you are doing it. Be familiar with the business goals.
❗ Leave your 💩 at the door. Do not bring your problems to work. Do not bring your work problems home.

Sorry, but I need to be honest. I'll explain myself at the end.

Startups 🏃 🏃 🏃 🏃

A startup reflects the founder's personality, especially at the beginning. One thing is sure. You have to work a lot chasing MVPs. The startups are great for learning about tech and business, people, office politics, sacrifice, etc.

While a startup grows and matures, the dynamics change. As growth happens, more people join the startup. Then, the startup needs new ways to structure the team, processes, and tools. While it was easy to discuss and do things at the begging, at this moment, this is not the case. More people, more dynamics, all that lead to new processes. Those processes are needed to keep the teams on the same page. That is a moment when the startup starts to mimic the product company but wants to keep the startup flavor. Also, it is the moment when some people begin to leave. Most of the time, it is because of a new dynamic. Some individuals like the chaos and hurray of the startup, but that dies with the growth.

Suppose you join later when the startup matures and has a solid foundation. In that case, it looks more like working for a product company. Still, there is a lot of work to do regarding the product itself and the processes around it. While it is an excellent place to learn and thrive professionally, it can be very demanding. No matter how mature it is, there is much work to make it stable and sustainable.

Product Companies 📦 📦 📦 📦

Product companies are the ones that have a product that they sell. They are willing to invest in new tech and people to give them an edge. These organizations are running a business in the long run.

For me switching from an outsourcing company to a product company was a significant change. It took me some time to digest the difference and realize what I could and needed to do. It was a time when the company that I joined was transforming technically and organizationally. But when the company builds a healthy culture, it supports its members. In every sense.

Early on, I got copies of The Phoenix Project. Reading that was an eye-opener. It is frustrating when you read it because you'll realize you already know all those principles. But the trick is to formulate them and act. Trying to apply what I learned from it and its sequel The Unicorn Project and The DevOps Handbook(bible) significantly changed how I work.

Today, tech changes so fast. Mainly to support the business. That is why teams need the freedom to learn and experiment
to create new opportunities for products. Investing in people is a need if a company wants to grow.

Outsourcing Companies 💦 😰 💦

No matter how fancy they are or how much sweet talk their HR does, they care only about client satisfaction. How will one do in such a company depends on many factors.

❗ The most important one is the client. The client is coming with the budget to do a project. An outsourcing company promises to deliver that. If clients already have outsourcing experience, it is a smooth ride. If the client has yet to gain any outsourcing experience, good luck.

Soft skills, this one is quite important. As you can not choose co-workers and clients, you have to be able to find a way to work with different kinds of people. Communicating with people from different cultures, backgrounds, and education is necessary. Also, you need to be able to convince clients to trust you. Your team needs to trust you, too. And you need to trust your team. That can be hard.

❗ The process in outsourcing companies has a goal to track hours and what is delivered. Yes, many of them in daily work use agile methods. But, in the end, what is important is how much you worked and delivered. That is why most outsourcing companies invest heavily in processes and tools around
them. That often leads to unnecessary bureaucracy, like the developer does not start working until the ticket is well defined and has all the details. To push not planned work, a client reports a bug. Etc.

❗ Once a project is over and the client is happy, the part of the team moves to the next project. The rest stays to support the project. Maybe you can choose to remain in the project and retire 😀 But, in general, you will move to the next project.

❗ The downside is that outsourcing companies are looking for people with specific skills. So, if you do not have them, you do not get hired or hired but do not get paid how much you wanted. Learning new things is more complicated. You are doing it outside work hours. That is because you are not paid to learn but instead to deliver. So, it takes work to get and keep different skills up to date. It could look like working double shifts.

❗ Influencing system design as a whole is challenging or impossible. In such teams, roles are strict. So, your job is to do what the ticket says, and that is it. There is someone who is creating tickets and doing system design. Whit such a concept, having groomings, retrospectives, and other agile ceremonies is pointless. Depending on the team, that can lead to a lot of frustration. If people that lead the team are willing to share responsibility, and the rest of the team is ready to accept responsibility, only then agile is justified.

The Good, Bad, and Ugly

If I put together what I mentioned above, I can say that outsourcing has its primary goal of making money. That it is. It is a business. You have the skill, clients need it, and you exchange it for cash. Simple. The problem is that you focus on a small tech area and a small set of issues. You can say that
you are becoming a specialist. Chasing to be a full stack, whatever that means. Making clients happy and then moving on is your goal.

Startups and product companies are different. They have to have an idea. Around that idea, they are building a product. That is a never-ending story. So, to survive, they need to take a risk and experiment. Tech is there to minimize the risk and to support the business. That is why they must invest in people and empower them to learn and experiment. To take responsibility. To be able to make decisions. To be able to fail. To be able to learn from mistakes.

In the end, we are all people. We can't just sell our time. We need to be proud of what we are doing. That is why we need to understand the goals and be able to influence them.

Next time you are angry at your boss or colleague, or process, ask yourself why you are doing it. Am I at the right place? If you bring your anger home, you are in the wrong place. No matter how high your salary is, it is not worth it. You need to be in a better place if you can enjoy your work, express yourself, and give and receive feedback. That is not happening in many outsourcing companies. They are just coder factories.

I do not say that all startups and product companies are fantastic. But eventually, they had to start transforming to survive.

Who is the good, bad, and ugly? It's up to you to decide.

I'm sorry if I sound like a preacher. It was not my intention. Thank you for reading, and I hope you enjoyed it. If you have any questions, please ask. I'll be happy to answer.