Hey DevOps rockstars! 🚀 Welcome back to our Terraform saga! In Part 5, we conquered modules—unlocking reusable, scalable superpowers for our infrastructure. Now, in Part 6, we’re tackling Managing Multiple Environments—think dev, staging, and prod, all humming in harmony. We’ll harness Terraform’s tools to juggle these setups with precision, keeping your workflows smooth and your deployments bulletproof. Ready to tame the multi-environment beast? Let’s dive in!
💬 Got Questions?
If you have any questions or need further clarification while reading this post, please don't hesitate to drop a comment below! I'm here to help, and I'll gladly create new posts to dive deeper into any topics you find challenging. 😊
1. What is a Terraform Environment?
A Terraform environment refers to a configuration framework designed to deploy and manage infrastructure resources consistently across distinct lifecycle stages—such as development, testing, and production. It encapsulates a logical collection of resources aligned with a common purpose or phase, providing an isolated context for infrastructure operations. This isolation ensures that changes in one environment (e.g., dev) do not affect others (e.g., prod), enabling safe and independent management.
2. Managing multiple environments
Terraform’s power shines when scaling a single configuration across multiple environments—such as development, staging, and production—for a web application deployed on AWS. Imagine a setup with components like an EC2 instance, VPC, security groups, and an RDS database. Initially built for production, this configuration can be adapted to support additional environments for testing, validation, or development, each with distinct requirements (e.g., smaller instances in dev, isolated networking in staging). The goal is to reuse one core configuration, deploying it multiple times with environment-specific tweaks, avoiding duplication while ensuring consistency. Terraform practitioners typically adopt one of two strategies to achieve this:
2.1. Workspaces
Terraform workspaces enable the management of multiple environments (e.g., dev, staging, prod) within a single backend by associating distinct state files with named workspace instances. When using a backend like AWS S3 or Terraform Cloud, the terraform workspace command allows practitioners to create, switch, and deploy configurations for different environments. For example, with an S3 backend, each workspace maintains its own state file (e.g., terraform.tfstate.d/dev, terraform.tfstate.d/staging), all stored within the same backend. Commands like terraform workspace select staging shift the active context, applying the configuration to the chosen environment.
2.1.1. Terraform Workspaces vs. Terraform Cloud Workspaces
Terraform workspaces and Terraform Cloud workspaces serve distinct purposes. In open-source Terraform, workspaces manage multiple environments (e.g., dev, prod) within a single configuration by maintaining separate state files. Conversely, in Terraform Cloud, a workspace functions as a broader construct, akin to a “project,” tied to a specific Terraform configuration repository. Beyond state management, Terraform Cloud workspaces handle variables, credentials, execution history, and additional metadata, enabling a comprehensive CI/CD workflow within the platform.
2.1.2. Example Workflow
terraform workspace show # show current workspace
terraform workspace list # output the list of workspaces
terraform workspace select # To select a specific workspace
terraform workspace new dev # Creates dev workspace
terraform apply # Deploys to dev
terraform workspace new staging # Creates staging workspace
terraform apply # Deploys to staging
terraform workspace delete # To delete a specific workspace
2.1.3. Creating and Switching Workspaces
Terraform workspaces are easy to create and manage.
- Step 1: List Workspaces
terraform workspace list
# Output: default
- Step 2: Create a New Workspace
terraform workspace new dev
# Output: Created and switched to workspace "dev"!
- Step 3: Switch Between Workspaces
terraform workspace select default
# Output: Switched to workspace "default".
2.1.4. Using Workspaces in Your Configuration
Workspaces are often used to customize resources for different environments.
Example: Deploy an EC2 instance with environment-specific configurations. main.tf:
provider "aws" {
region = "us-east-1"
}
resource "aws_instance" "web" {
ami = "ami-0c55b159cbfafe1f0" # Amazon Linux 2 AMI (us-east-1)
instance_type = terraform.workspace == "prod" ? "t2.large" : "t2.micro"
tags = {
Name = "web-instance-${terraform.workspace}"
}
}
output "instance_id" {
value = aws_instance.web.id
}
Explanation:
-
terraform.workspace: Returns the current workspace name (e.g., "dev", "prod"). -
instance_type: Uses t2.large for the prod workspace and t2.micro for others.
2.1.5. Hands-On: Deploy to Multiple Environments
Initialize the Project:
terraform init
Create Workspaces:
terraform workspace new dev
terraform workspace new staging
terraform workspace new prod
Deploy to Each Environment:
- Switch to the dev workspace:
terraform workspace select dev
terraform apply
- Switch to the prod workspace:
terraform workspace select prod
terraform apply
- Verify Resources: Check the AWS EC2 Console — you’ll see instances tagged with the workspace name.
2.1.6. Accounts and credentials
Multiple environments are typically managed using multiple cloud accounts or subscriptions. Cloud platforms also implement the “Organizations” concept to manage multiple accounts from a single root account. This root account is responsible for all the management activities like billing, access provisioning, etc.
When a Terraform configuration is “applied,” the changes are validated and executed for the target account based on its provider configuration. Below you can find a Terraform provider configuration for AWS using a shared credentials file.
provider "aws" {
shared_config_files = ["/path/to/.aws/conf"]
shared_credentials_files = ["/path/to/.aws/creds"]
profile = "profile_name"
}
Pros
- Ease of Setup: Quick to initiate with minimal configuration—ideal for small teams or rapid prototyping.
- Convenient terraform.workspace Expression: Allows dynamic referencing within configurations (e.g., name = "db-${terraform.workspace}" creates db-dev or db-staging), embedding environment context directly in resource names.
- Reduced Code Duplication: Maintains a single configuration file, avoiding the need for separate copies per environment.
Cons
- Susceptibility to Human Error: Manual workspace switching (e.g., forgetting terraform workspace select prod) risks applying changes to the wrong environment.
- State in Single Backend: All state files reside in the same backend, complicating permission management—Terraform Cloud offers nuanced controls, but self-managed backends (e.g., S3) require careful IAM configuration to isolate access.
- Lack of Explicit Configuration Visibility: The codebase doesn’t inherently reflect environment-specific settings, relying on runtime workspace selection, which may obscure deployment details.
2.1.6. Best Practices for Workspaces
- Designate Workspaces for Environments: Utilize workspaces to manage distinct environments like development, staging, and production effectively.
- Prevent Workspace Overuse: Reserve workspaces for environment-specific deployments, avoiding their application to unrelated projects.
- Employ terraform.workspace: Dynamically tailor resource attributes (e.g., names, sizes) using the terraform.workspace expression.
- Integrate with Variables: Pair workspaces with .tfvars files to supply environment-specific configurations seamlessly.
2.1.7. Common Issues and Fixes
- State File Conflicts: Resolve discrepancies between the state file and actual infrastructure by running terraform refresh to synchronize them.
- Workspace Not Found: Verify the workspace name using terraform workspace list to ensure it exists and is correctly specified.
- Resource Drift: Identify and correct deviations from the desired configuration by executing terraform plan to detect drift and applying necessary updates.
2.2. File structure
The file structure approach organizes Terraform configurations into environment-specific subdirectories (e.g., dev, staging, prod) within the project filesystem. Each subdirectory contains its own copy of the configuration files (e.g., main.tf, variables.tf) and typically a dedicated .tfvars file (e.g., dev.tfvars) to specify environment-specific values. This method allows a single core configuration to be deployed across multiple environments by executing Terraform within each directory, often with distinct backends for state management.
Example Structure
project/
├── dev/
│ ├── main.tf
│ └── dev.tfvars
├── staging/
│ ├── main.tf
│ └── staging.tfvars
└── prod/
├── main.tf
└── prod.tfvars
Pros
- Isolation of Backends: Each environment can use a separate backend (e.g., unique S3 bucket paths like s3://my-bucket/dev/terraform.tfstate), enhancing security and reducing the risk of human errors like cross-environment overwrites.
- Codebase Transparency: The directory structure explicitly reflects the deployed state, making environment-specific configurations visible and auditable within the filesystem.
Cons
- Multiple Apply Commands: Requires running terraform apply independently in each subdirectory, increasing operational overhead compared to a single command with workspaces.
- Increased Code Duplication: Identical or near-identical configuration files across directories (e.g., main.tf) lead to redundancy, complicating updates and maintenance.
3. File structure: Environments and Components
As infrastructure complexity grows, a single, monolithic Terraform configuration becomes impractical. For small organizations with minimal applications, consolidating all infrastructure (e.g., compute, networking, storage) into one file may suffice. However, as an organization scales and infrastructure diversifies, separating configurations into logical component groups—such as compute and networking—enhances manageability and scalability. This approach, combined with environment-specific subdirectories (e.g., dev, staging, prod), allows for a modular, maintainable structure tailored to evolving needs.
3.1. Breaking Down by Components
Splitting infrastructure into distinct components depends on operational dynamics. For example, if networking configurations remain stable while compute resources change frequently, isolating them into separate modules or directories minimizes disruption. A sample file structure might evolve as follows:
project/
├── networking/
│ ├── dev/
│ │ ├── main.tf
│ │ └── dev.tfvars
│ └── prod/
│ ├── main.tf
│ └── prod.tfvars
└── compute/
├── dev/
│ ├── main.tf
│ └── dev.tfvars
└── prod/
├── main.tf
└── prod.tfvars
Here, networking and compute are distinct components, each with environment-specific configurations.
3.2. Leveraging Remote State
Terraform’s remote state feature enables cross-referencing between separate configurations, even across different projects. For instance, after deploying compute infrastructure (e.g., EC2 instances) in one config, its outputs (e.g., IP addresses) can be accessed from another config via a remote backend. Example:
data "terraform_remote_state" "compute" {
backend = "s3"
config = {
bucket = "my-terraform-state"
key = "compute/prod/terraform.tfstate"
region = "us-east-1"
}
}
resource "aws_security_group_rule" "allow_compute" {
security_group_id = "sg-123456"
cidr_blocks = [data.terraform_remote_state.compute.outputs.instance_ip]
}
This retrieves the instance_ip output from the compute config’s state file, ensuring synchronization without bundling everything into a single project.
Pros
- Modularity: Logical separation reduces complexity and improves focus (e.g., networking vs. compute).
- Isolation: Independent backends per component enhance security and reduce error scope.
- Interoperability: Remote state links configs, enabling dynamic data sharing.
Cons
- Management Overhead: Multiple directories and backends require coordinated updates and executions.
- Duplication Risk: Similar configs across components may still introduce redundancy if not modularized further.
- This hybrid structure—environments plus components—paired with remote state, offers a powerful framework for managing complex, multi-environment infrastructure efficiently.
4. Terragrunt
Terragrunt is a powerful wrapper tool layered atop Terraform, designed to simplify infrastructure management and maintain DRY (Don’t Repeat Yourself) configurations. As organizations scale, breaking Terraform configs into modular file structures (e.g., by environment or component) introduces complexity—multiple commands, redundant code, and challenges with multi-cloud or multi-account setups. Terragrunt addresses these pain points by streamlining workflows and enhancing efficiency.
Key Benefits
- DRY Configurations: Centralizes repetitive settings (e.g., backend, provider configs) in a single terragrunt.hcl, reducing duplication across environments like dev, staging, and prod.
- Simplified Multi-Environment Management: Integrates with file structures (e.g., env/dev, env/prod) and supports remote state referencing, enabling seamless deployment across isolated configs.
- Reduced Command Overhead: Commands like terragrunt run-all execute Terraform operations across multiple modules in one go, respecting dependencies, unlike running separate terraform apply commands per directory.
- Multi-Cloud/Account Support: Facilitates working with multiple cloud accounts by automating backend setup and role assumption, minimizing manual intervention.
By overlaying Terraform with Terragrunt, teams can tame sprawling configs, enforce consistency, and manage complexity—making it an invaluable tool for scalable, multi-environment deployments.
5. Demo
You can find the complete Terraform configurations for this tutorial in the GitHub repository below. Feel free to explore, fork, and experiment! 🚀
🔗 GitHub Repo: [https://github.com/rahimbtc1994/terraform-intermediate/tree/main/part-7]
If you have any questions or run into issues, drop a comment—I’m happy to help! 😊
Top comments (1)
Please feel free to ask any questions you have.