Forem: rderik

Directory Structure for Terraform Projects

rderik — Sun, 01 Oct 2023 10:37:00 +0000

Terraform doesn't concern itself with the directory structure of our project. It cares about state. We, as the users of the project, are the ones who benefit from a clean and easy-to-understand directory structure.

In this post, we'll explore basic directory structures used for Terraform projects.

Note: If you want a more in-depth discussion about the state and directory structure relationship, you might like my guide Meditations on Directory Structure for Terraform Projects.

Common ways to structure your directories and files for Terraform projects
- Per-project boundary
- Per-service boundary
- Using workspaces
Final Thoughts

Common ways to structure your directories and files for Terraform projects

There are multiple ways you can structure your code, Terraform is very flexible about it, so there is no single answer to that question. But let's look at a few commong patterns.

Per-project boundary

A common pattern is structuring your code using the environment as the boundary for separating resources. The directory structure looks like the following:

── basic_project
    ├── environments
    │   ├── dev
    │   │   ├── main.tf
    │   │   ├── outputs.tf
    │   │   └── variables.tf
    │   ├── prod
    │   │   ├── main.tf
    │   │   ├── outputs.tf
    │   │   └── variables.tf
    │   └── staging
    │   ├── main.tf
    │   ├── outputs.tf
    │   └── variables.tf
    ├── modules
    └── shared

This structure is clean and easy to understand at a glance. If you work in a specific environment, any change you make won't affect other environments. It has a couple of drawbacks. For example, we could easily add different resources to each environment, creating drift between the environments. It is good practice to keep your environments as similar as possible, so running tests in a lower environment will reflect the effects in the same way as running them in a production environment.

In a clean directory structure, the code inside the environment directories (i.e. dev, staging and prod) are primarily to reference modules. The idea is to reduce code duplication and use the modules as the source of truth of how resources should be configured and with sensible default behaviours.

Per-service boundary

When the number of resources is high, some teams break down the directory structure into smaller areas of interest. For example, split the directory structure by service. The file structure would look like the following:

── service_A
    └── environments
        ├── dev
        │   ├── main.tf
        │   ├── outputs.tf
        │   └── variables.tf
        ├── prod
        │   ├── main.tf
        │   ├── outputs.tf
        │   └── variables.tf
        └── staging
        ├── main.tf
        ├── outputs.tf
        └── variables.tf

If you are observant, you'll notice that the directory structure is basically the same as the previous one. Breaking down into smaller areas is similar to breaking it down by project. The idea stays the same: create a boundary to hold certain resources.

Depending on the number of resources, the per-project structure might start causing your plan and apply commands to take a lot of time. The reason is that Terraform has to query the AWS API to check the state of the resources it is tracking. So, a larger number of resources will lead to more API calls.

There are other reasons why we might need to segregate the state:

High Number of resources, we already mentioned this
Using different AWS accounts per environment
Breaking state per region. Projects sometimes segregate their state per region to:
- Reduce blast radius if a region is down
- Deployment strategies that would prefer to only deploy to certain regions for testing, i.e. canary deployments

Using `workspaces`

Another common alternative for the per-project directory structure is using workspaces. This structure maintains the same code for all environments, reducing the possibility of differences between environments.

── single_code_multiple_var
    ├── infra
    │   ├── main.tf
    │   ├── provider.tf
    │   ├── outputs.tf
    │  └── variables.tf
    ├─── environments
    │   ├── env.dev.tfvars
    │   ├── env.staging.tfvars
    │  └── env.prod.tfvars
    └── modules

There are still cases where we would like different behaviours per environment. For example, using smaller instance types for lower-level environments and larger instance types for production. To reflect that difference, we use different values in tfvars files that will capture the differences. This directory structure and workflow are OK if you are OK with having the same backend for all the workspaces:

terraform {
  backend "s3" { <--- we can't use variable interpolation, so we can't do the following
    region = "${local.env[var.enviroment]}"
    bucket = "rderik-tfstate-${var.environment}"
    key = "${var.environment}/tfstate/terraform.tfstate"
    dynamodb_table = "rderik-tfstate-${var.environment}"
    encrypt = true
  }
}

If, for some reason, we need to use a different backend per environment, then we have to solve the problem with the backend, which has its own set of peculiarities.

Final Thoughts

The directory structure that we end up using for our project is highly dependent on how we want to manage the state of our project. Consider if you want to keep one state per project, service, region, etc. The key to deciding which directory structure to use in your project is for you to understand the relationship between:

State
Directory structure
Backend
Module versions

Once you clearly understand how you want to handle those aspects of your project, it'll be easier to choose the directory structure that will make your life easier. But don't worry too much about it. It is OK to start with one structure and later understand that you have outgrown that structure and migrate out of it. Believing you'll have all the answers from the beginning is a common mistake.

Software projects are iterative. You keep growing and you keep adapting that is the nature of our work.

How to set up a new Terraform project using S3 backend and DynamoDB locking

rderik — Sat, 08 Oct 2022 12:34:00 +0000

Sometimes it feels easier to work on complex and challenging tasks with our tools. We forget how to do the simple initial steps for a project. The reason for this is that we lack practice starting projects. If you work for a company, you'll probably set up your Infrastructure as Code (IaC) once and then iterate for it. Unless you work on a consultancy or start projects just for fun, you might forget the initial steps to set up a project.

This post is about setting up a Terraform project storing the TFState using an S3 bucket and using DynamoDB as our lock mechanism to help prevent issues when two people are working on the same infrastructure.

Note: If you are interested in learning more about how to set up the directory structure for your Terraform project, you might find my guide, Meditations on Directory Structure for Terraform Projects, useful.

The chicken and egg problem of setting up resources before the state
Creating the resources storing the state locally
Set up our backend to use S3 and DynamoDB
Final Thoughts

The chicken and egg problem of setting up resources before the state

It should be a straightforward process to start a Terraform project. Set the state to be stored in an S3 bucket and DynamoDB to keep the lock. But the problem begins when we realise that those resources, the S3 bucket and the DynamoDB table, won't be tracked by Terraform. We would also like to keep track of those resources in our IaC; we don't want them to be an exception. So what do you do? Do you create them manually and then later import them? While I believe you could do that, there might be a better way.

First, create the resources using the local state, then migrate the state to S3 and DynamoDB. That is a simple idea, so let's do it.

Creating the resources storing the state locally

Using the following code, let's define the initial template for creating the S3 bucket and the DynamoDB table. We will create a bucket called my-terraform-state and a table called my-terraform-lock. You can name yours however works for you. Create a file named main.tf with the following content:

provider "aws" {
  region = "us-west-1"
}

#terraform {
# backend "s3" {
# region = "us-west-1"
# bucket = "my-terraform-state"
# key = "global/tfstate/terraform.tfstate"
# dynamodb_table = "my-terraform-lock"
# encrypt = true
# }
#}

resource "aws_s3_bucket" "my-terraform-state" {
  bucket = "my-terraform-state"
  acl = "private"
}

resource "aws_dynamodb_table" "my-terraform-lock" {
  name = "my-terraform-lock"
  billing_mode = "PAY_PER_REQUEST"
  hash_key = "LockID"
  attribute {
    name = "LockID"
    type = "S"
  }
}

You'll notice that there is some commented-out code. We usually would use that to tell Terraform to keep the state in S3 and to use DynamoDB for locking. We will use that code, but not just now.

With our main.tf file created, we can run terraform init, this will create the state locally.

terraform init

Now that we have the state, we can apply the template and create the bucket and table:

terraform apply

Perfect, we now have the table and table created.

Set up our backend to use S3 and DynamoDB

Now we remove the comments on the code that defines our backend:

terraform {
  backend "s3" {
    region = "us-west-1"
    bucket = "my-terraform-state"
    key = "global/tfstate/terraform.tfstate"
    dynamodb_table = "my-terraform-lock"
    encrypt = true
  }
}

Save main.tf and run terraform init again:

terraform init

And that's it, you are ready!

Final Thoughts

This was a short and simple post, but I hope it is useful. I've had this question before, and it is nice to have a quick article to point to instead of trying to remember how I did it the first time.

Creating network applications in macOS

rderik — Mon, 14 Sep 2020 12:51:13 +0000

I finally completed my "macOS network programming in Swift" guide. You can get it from my site. The guide covers how to use the following libraries and frameworks to build network applications:

BSD Sockets
Network.framework
SwiotNIO

Have a look and share with anyone that might find it useful. The guides on my site are free, but if you want to support me, you can pay anything you want.

Here's the link:
rderik.com - Guides

Using SwiftNIO to build a simple Server

rderik — Thu, 20 Aug 2020 21:12:01 +0000

If you haven't heard about SwiftNIO, it is a framework with the primary purpose of building network applications. It takes a lot of the responsibility for the network code of your application so you can focus on the application code.

If you are interested in Server-Side Swift, you might find this post interesting:

Understanding SwiftNIO by building a text modifying server

Feedback is welcomed.

Building a text-based clock using Swift

rderik — Thu, 06 Aug 2020 21:22:38 +0000

If you are looking for a fun project to get into command-line tools development, I think you'll enjoy this article:

https://rderik.com/blog/building-a-text-based-application-using-swift-and-ncurses/

It uses the C ncurses(3) library via SwiftCursesTerm to create a clock modelled as a 7-segment display.

I hope you like it.

A command-line tool to create your GitHub profile using a template

rderik — Tue, 21 Jul 2020 18:26:21 +0000

As you might have noticed, you can now create a GitHub profile by creating a repository with the same name as your GitHub username. I wanted to add content that changes often, but I didn't want to do it manually. So I created Octoprofile.

Octoprofile is a small command-line tool that allows you to build your GitHub Profile using a template, and you can extend the template by using plugins. The plugin system is quite simple, but that's the point, anyone can extend it and create their own plugins.

The tool is open-source, and the point is to help others. If you are a beginner and want to contribute, I think this project is an excellent place to start.

Ok, I hope it is helpful. Here is the link to the GitHub repository:

https://github.com/rderik/octoprofile

Generating a table of contents with anchors for markdown - Vim plugin

rderik — Mon, 20 Jul 2020 12:30:39 +0000

I use vim as my editor and markdown to write the articles for my site. I think that the articles are easier to navigate when they have a table of contents.

Adding a table of contents was easy. Many plugins accomplish this. But I couldn't find a plugin that also added the anchors. And adding the anchors was a very tedious process. So I modified the vim-markdown-toc plugin to add the anchors.

If you are looking to do the same, I wrote a short post on how to do it with my modified version of the plugin. You can read about it here:

https://rderik.com/notes/generate-table-of-contents-with-anchors-for-markdown-file-vim-plugin/

I hope you find it useful.

Creating a composable command-line tool using the Swift Argument Parser

rderik — Thu, 09 Jul 2020 12:07:52 +0000

Most of us make use of command-line tools that are composable, like sed(1), grep(1), sort(1). Not all tools use the same conventions for working with input from the standard input(STDIN). But it's certainly handy when they do.

Apple announced on February 2020 the Swift Argument Parser. It provides an easy to use parsing capability, and their documentationis good. But as many examples on the Internet about building command-line tools, they focus on self-contained applications.

I thought it might be useful to show how to create a composable command-line tool. A composable command-line tool allows you to create a workflow between different tools. For example:

$ grep "Name:*" *.txt | sort

We are using two tools here grep(1)(To search for the pattern Name:* on all files with a .txt extension) and sort(1). Each of the commands can be used on their own, but when we put them together, we can create complex workflows that easily solve our problem. The idea is redirecting the Standard Output(STDOUT) of one command and use it as the Standard Input(STDIN) of the second command using a pipe |.

If we want to make our command-line tool composable, we would like to support the following cases:

If the tool is called without any parameters it'll try to read from the STDIN.
If the tool's last argument is a single dash (-) read from the STDIN.

The same way as sort(1) or other Unix command-line tools work.

we can call:

$ cat file.txt | sort

or we can call:

$ sort file.txt

If you are interested in how to build a command-line tool that is composable using the Swift Argument Parser, you might find the following post interesting:

https://rderik.com/blog/understanding-the-swift-argument-parser-and-working-with-stdin/

As always, feedback is welcomed.

Jails & VNET a Guide

rderik — Fri, 19 Jun 2020 21:41:30 +0000

I was working with jails in FreeBSD and wanted to model the infrastructure using jails and virtual networks. This project led me through a learning journey on how to use vnets and jails. Vnets were useful for jail isolation at the network level.

I ended up writing a guide for anyone that needs it. If you are interested, you can get the guide here:

https://rderik.com/guides/

You can get the guide in different formats:

ePub
Mobi
PDF
Plain text

I also include the source code. The plain text is handy when you would like to copy any command in the examples.

The guide is free, but if you would like to support my work, you can pay anything you want. In any case, I hope you find it useful.

Deciding on a FreeBSD partition schema

rderik — Mon, 01 Jun 2020 23:09:16 +0000

Lately, I've been playing around with FreeBSD. I remembered the first few times I had to partition a disk a long time ago. I spent a lot of time researching what other people did and talking with friends about it.

It is not a complicated process, but because I didn't do it often, I had to re-learn what I've done every time. Now I'm mostly using ZF which kind of makes partitioning a single disk not necessary. But I wanted to write down my notes on partitioning a disk using UFS if I had to do it again.

I'm following the advice in Absolute FreeBSD, 3rd edition, and create the following partitions:

efi
swap
dump
/
/tmp
/var
/usr (I'm keeping my home directory under /usr)

I believe that is a solid "classic" partitioning schema that I find useful and clean. And from this base, and depending on the system I'm deploying, I'll add new partitions. For example, a partition for /database if I'm running a server dedicated to PostgreSQL.

Anyhow, if you are interested in more detail here is the link:

https://rderik.com/notes/notes-on-freebsd-ufs-partition-schema/

I would also recommend the book Absolute FreeBSD, 3rd edition, quite enjoyable. I learned a lot from it.

Using a simple Bash script to increase your iOS bundle version

rderik — Tue, 26 May 2020 14:42:47 +0000

Last week I got a message from someone using a tool I created for myself called XCIncreaseBuildVersion. I created this script that increases the bundle version in preparation for uploading my app to App Store Connect.

It's always satisfying to see your tools help someone.

The script searches the Info.plist files and increases the build version number using plutil(1).

If you are interested in the tool you can find it here:

https://github.com/rderik/xcibversion/

If you want to check the whole article that explains the logic behind the automation of the process, and how to upload the xarchive to App Store Connect, you can find it here:

https://rderik.com/blog/automating-build-and-testflight-upload-for-simple-ios-apps/

Who knows, maybe it can also help you in your CI/CD pipeline.

Record your terminal session using the `script`command.

rderik — Mon, 25 May 2020 22:34:43 +0000

I've been in the situation where I ran a script that behaved weirdly and would love to be able to "replay" what I did to figure out what went wrong. A useful command for those cases is script(1).

The script command (introduced in 1979) allows you to record a terminal session to a file. You can later check and reference that file to analyse it as you please. Or maybe record your terminal as a demo.

As with any other Unix command, the power comes from your creativity on how to use it. You can use script to:

Record a demo.
Capture output.
Record session to report a bug.

You can try it just by typing script in your terminal. It opens up a new shell that will be recorded. By default, the session is saved in a file named typescript.

$ script
% ls
% exit

You can then view the capture by using cat(1):

$ cat typescript

If you are interested in more detail, here are my notes on how to use it:

https://rderik.com/notes/using-the-script-command-to-record-a-terminal-session/

Forem: rderik

Directory Structure for Terraform Projects

Common ways to structure your directories and files for Terraform projects

Per-project boundary

Per-service boundary

Using workspaces

Final Thoughts

How to set up a new Terraform project using S3 backend and DynamoDB locking

The chicken and egg problem of setting up resources before the state

Creating the resources storing the state locally

Set up our backend to use S3 and DynamoDB

Final Thoughts

Creating network applications in macOS

Using SwiftNIO to build a simple Server

Building a text-based clock using Swift

A command-line tool to create your GitHub profile using a template

Generating a table of contents with anchors for markdown - Vim plugin

Creating a composable command-line tool using the Swift Argument Parser

Jails & VNET a Guide

Deciding on a FreeBSD partition schema

Using a simple Bash script to increase your iOS bundle version

Record your terminal session using the `script`command.

Using `workspaces`