Forem: Colin Barker

Enabling IPv6 on AWS using Terraform - EC2 "Pet" Instance (Part 2)

Colin Barker — Mon, 06 Mar 2023 12:27:39 +0000

Header photo by NASA on Unsplash

⚠️ Note: This is Part 2 of my IPv6 on AWS series, the first part is available here. ⚠️

What is a "pet" instance, and why?

Before I start, this is an exceptional use case. When using AWS you should always use ephemeral instances, it is always best practice. There are times however, you might need to do this to a singular instance for any number of very valid reasons. In my case, I have a very cheap proxy service between the outside world any my home network. I use this external service as a very cheap method to enable access to some systems that can be accessed through a Site-to-Site VPN to my home. You might have other reasons too, for example - Microsoft Active Directory servers which are hosted on EC2 instances would be considered "pet".

In my case, I have this one static instance using an Elastic IPv4 IP, and I would like to give it an IPv6 address.

Pets vs Cattle Analogy

"cattle, not pets" - Bill Baker

This phase used during a presentation about Scaling SQL Servers way back in 2006, to show how attitudes towards computing evolved since the early days.

It describes the idea that servers can be two types:

Pets: These are your pride and joys, there is just one Oz the Cat that sits on your lap while writing blog posts about IPv6, you look after them, nurture them, and you deal with everything that comes their way as and when it happens.
Cattle: You have a farm, you have a vast amount of animals that help you produce several products that you sell to the market. If one of those animals becomes an issue, you "replace" them. No sentimental attachment.

I will say, this analogy will probably not last the test of time - the latter "Cattle" explanation can upset a few people for different reasons and you probably will see this change in the future. Hoping for something like "Cooker vs Food" or "House vs Tent", but I can't see that happening any time soon!

So with that in mind, you can see the following:

Pet Instances: They have a set hostname, they have had loads of love, care, and attention given to them. When they go wrong, you investigate, identify the issues, remediate, and bring back to health. They also make lots of noises when you need to give them attention. (Yes, Oz is meowing at me at the moment!)
Cattle Instances: They have a randomly generated unique identifier, you keep the safe and secure, but once the instance starts to fail, you quickly take them out of the loop, and replace with a healthy instance. The failed instance, you just get rid of.

Starting Point

Before we begin, we need to set the scene a little. What we will be working with is incredibly simple - an EC2 instance sitting in a Public Subnet.

You can find the code to deploy the above diagram on my GitHub - IPv6 on AWS repo. Feel free to follow along in your own sandbox account if you wish!

The main block of code we need to start looking at is the ec2_instance resource, there are some comments inline to explain what we are doing.

# Build the EC2 instance using Amazon Linux 2
resource "aws_instance" "test" {
  ami                     = data.aws_ami.amazon_linux_2.id   # Dynamically chosen Amazon Linux AMI
  ebs_optimized           = true                             # EBS Optimised instance
  instance_type           = "t4g.nano"                       # Using a Graviton based instance here
  disable_api_termination = true                             # Always good practice to stop pet instances being terminated

  # Networking settings, setting the private IP to the 10th IP in the subnet, and attaching to the right SG and Subnets
  source_dest_check      = false
  private_ip             = cidrhost(aws_subnet.public_a.cidr_block, 10)
  subnet_id              = aws_subnet.public_a.id
  vpc_security_group_ids = [aws_security_group.test_sg.id]

  # This requires that the metadata endpoint on the instance uses the new IMDSv2 secure endpoint
  metadata_options {
    http_endpoint = "enabled"
    http_tokens   = "required"
  }

  # Sets the size of the EBS root volume attached to the instance
  root_block_device {
    volume_size           = "8"      # In GB
    volume_type           = "gp3"    # Volume Type
    encrypted             = true     # Always best practice to encrypt
    delete_on_termination = true     # Make sure that the volume is deleted on termination
  }

  # Name of the instance, for the console
  tags = {
    "Name" = "Sample-EC2-Instance"
  }

  # Ensures the Internet Gateway has been setup before deploying the instance
  depends_on = [
    aws_internet_gateway.sample_igw
  ]
}

Nice and simple really, but this is where simple can cause an issue in this case.

The Issue

Without going to too much detail, as you know Terraform uses a State that will store details about the environment it manages, including the settings and configuration of resources. It will use this information to keep track of what it knows, and what has changed - giving it the great ability to see what will change during the next application of your code. This state is generated from the outputs of the providers API's and your code.

However, the API call for creating an EC2 instance does more than just create a single EC2 resource. This also happens with multiple services, and multiple cloud providers as well, so it isn't specific to AWS on this case. The single API call makes the process a lot simpler to build up the compute for you, but it includes one additional resource: The Elastic Network Interface.

In typical use, this is great, it saves you building instances with no networking and then having to mess about with it later on. I would also point out, that a cloud service with no networking access, is no different to getting a physical computer, disconnecting everything except the power, wrapping it in concrete, and then seeing what you can do. I mean it will still run (albeit very hot and probably melt), but you can't then do anything with it, or see what is happening!

Terraform does have a resource for the Elastic Network Interface (ENI) however, as the API created this resource itself, attached it to the instance using the parameters specified in the ec2_instance resource, Terraform doesn't know about this. Herein lies the issue.

Adding an IPv6 Address using the CLI

Let's say you don't have your Infrastructure defined as code and you needed to assign an IPv6 address to your instance, thankfully the awscli has such a method. Under the ec2 service type, there is a command to assign-ipv6-addresses (Documentation)

To do this, you would run the following command:

aws ec2 assign-ipv6-addresses --ipv6-addresses <address> --network-interface-id eni-007ed4d597d6df6b7

The one item you need to know is, the network-interface-id. While the ec2_instance resource, does have this as an output, the only way to get this is after the resource has been created.

What happens in Terraform?

When using the AWS EC2 API to create the instance, behind the scenes it will create that interface using the settings, and return the Interface ID. As the API's are not supposed to keep the State themselves, it will always consider a change to those settings in the creation block, as a new interface/network settings.

As we define the network settings in our resource for the EC2 instance, Terraform knows there is a change to the state, talks to the AWS EC2 API, that then will require the network interface to be recreated. The only way to do that would be create a new interface, detach the old interface, attach the new interface, and you are done, except, that isn't possible. AWS Documentation on Network Interfaces state that "Each instance has a default network interface, called the primary network interface. You cannot detach a primary network interface from an instance." This comes into play with my workaround in this blog later on. Therefore, the only option, is to re-create the instance.

If we were to simply use the ipv6_addresses in the ec2_instance block, to add a new IPv6 address, Terraform will report the following:

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
-/+ destroy and then create replacement

Terraform will perform the following actions:

# aws_instance.test must be replaced
-/+ resource "aws_instance" "test" {

      <<SNIP>>

      ~ instance_initiated_shutdown_behavior = "stop" -> (known after apply)
      ~ instance_state                       = "running" -> (known after apply)
      ~ ipv6_address_count                   = 0 -> (known after apply)
      ~ ipv6_addresses                       = [ # forces replacement
          + "2a05:d01c:b90:ee00::10",
        ]
      + key_name                             = (known after apply)
      ~ monitoring                           = false -> (known after apply)
      + outpost_arn                          = (known after apply)

      <<SNIP>>

    }

As you can see, adding the address forces replacement. Which for our wonderful pet instance here, is a terrifying thought!

Workaround - Pseudo Code

⚠️ Note: This is not going to be the best way, in all honesty this is a hack more than a workaround, but it does make it easier to bypass the issue. Always consider backups before doing this, always consider that if you need to do this, why do you need to do it? ⚠️

Given that the AWS CLI can add an IPv6 address without rebuilding the instance, as well as the console as well, it does mean that a replacement isn't needed to add the address on. For this, we have to play around a bit with Terraform, run it a few times, to get everything up in sync. To this we will need to:

Create an aws_network_interface resource to match the primary ENI with the IPv6 address
Import the ENI resource into the Terraform State (requires CLI)
Apply the changes using Terraform, and confirm the IPv6 address has been attached
Add the ipv6_addresses list to the ec2_instance resource block
Delete the aws_network_interface from the Terraform State (requires CLI)
Delete the aws_network_interface resource block from the code
Run a plan to confirm everything is working

As you can see, there are a few steps to do this, but it does mean that if you ever need to re-deploy the code, this will create an identical copy of the instance (minus data) and matches the code.

This also removes the issue later on if you ever wish to destroy the code. If you were to stop after the Apply when you have the IPv6 address assigned, you will now have an aws_network_interface resource block managed by Terraform, but also managed by the EC2 Instance itself. As mentioned before, you cant remove the primary network interface from the EC2 instance, and if you try and run a destroy, Terraform will attempt to call the API to tell AWS to delete the interface, an come back with a 400 error.

Error: detaching EC2 Network Interface (eni-071f55452ccc18997/eni-attach-0a424f74a43a7a0af): OperationNotPermitted: The network interface at device index 0 cannot be detached.

So the workaround requires us to complete all the steps to make this work.

Workaround - Terraform

Create the ENI

First we need to create the aws_network_interface resource block. We will need to try and match as much as we can to what we have defined in the ec2_instance block. Below is an example of of this block created using the sample code above.

resource "aws_network_interface" "test_eni" {
  subnet_id         = aws_subnet.public_a.id
  # Uses the same IP from the ec2_instance resource
  private_ips       = [cidrhost(aws_subnet.public_a.cidr_block, 10)]
  # The new IPv6 to assign - note that 16 in hexadecimal is 10
  ipv6_addresses    = [cidrhost(aws_subnet.public_a.ipv6_cidr_block, 16)]
  # Same security group from the ec2_instance resource
  security_groups   = [aws_security_group.test_sg.id]
  # Continue with additional settings from the ec2_instance resource
  source_dest_check = false

  # Match the attachment details on the EC2 instance
  attachment {
    instance     = aws_instance.test.id
    device_index = 0
  }

}

Import the ENI into State

Next we need to get the ENI information into the state, to do this we need the network interface ID from AWS. This can be done through the console, or by looking at the State file (if you can). In our example, the ENI is eni-071f55452ccc18997. Terraform at the bottom of all of its documentation will have the command you need to import the resource in, in our case we will need to import this ENI into this resource block

terraform import aws_network_interface.test_eni eni-071f55452ccc18997

Apply the changes

As we have already created the block with the IPv6 address in it, we should be able to run the plan and apply to add the IPv6 address. Do remember to check your plan! Make sure that it is doing what you expect, in my case the plan showed:

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  ~ update in-place

Terraform will perform the following actions:

  # aws_network_interface.test_eni will be updated in-place
  ~ resource "aws_network_interface" "test_eni" {
        id                        = "eni-071f55452ccc18997"
      ~ ipv6_address_count        = 0 -> (known after apply)
      ~ ipv6_address_list         = [] -> (known after apply)
      + ipv6_address_list_enabled = false
      ~ ipv6_addresses            = [
          + "2a05:d01c:b90:ee00::10",
        ]
      + private_ip_list_enabled   = false
        tags                      = {}
      ~ tags_all                  = {
          + "Environment" = "Sandbox"
          + "Source"      = "Terraform"
        }
        # (15 unchanged attributes hidden)

        # (1 unchanged block hidden)
    }

As shown, the only major change will be that there is an additional IPv6 address assigned to it, but also my default tags are added too.

Technically at this point, we have done it - but we still have to sort out our Terraform to prevent future issues.

Add the IPv6 address to the instance block

Thankfully the block we created, also has the same line that we can copy back into the aws_instance block, we can sneak this back in to match the state in AWS:

# Build the EC2 instance using Amazon Linux 2
resource "aws_instance" "test" {

  <<SNIP>>

  # Networking settings, setting the private IP to the 10th IP in the subnet, and attaching to the right SG and Subnets
  source_dest_check      = false
  private_ip             = cidrhost(aws_subnet.public_a.cidr_block, 10)
  # IPv6 Address for the instance from the aws_network_interface block
  ipv6_addresses         = [cidrhost(aws_subnet.public_a.ipv6_cidr_block, 16)]
  subnet_id              = aws_subnet.public_a.id
  vpc_security_group_ids = [aws_security_group.test_sg.id]

  <<SNIP>>
}

Delete the ENI from the State

Before we remove the block, we need to remove the information from the State. This will prevent us from accidentally deleting the block, running an apply, and getting the 400 error mentioned before! To do this we will need to run the following command:

terraform state rm aws_network_interface.test_eni

Now you can delete the whole aws_network_interface block from your code.

Run a plan to check

With all this complete, we just need to run one very last terraform plan to confirm, if everything has gone right then your output should be:

No changes. Your infrastructure matches the configuration.

Terraform has compared your real infrastructure against your configuration and found no differences, so no changes are needed.

Summary

Just would like to point out, this is not the ideal way of doing this. I am sure there are other ways, but this is how I got around the issue. Thankfully in my case I had access to the Terraform State, which made the addition of the aws_network_interface a lot easier. This workaround doesn't work too well when you do not have access.

This technique can work not just on AWS, but other cloud providers too - its mainly about the logical steps of importing the unmanaged resource, making the changes, and then releasing it back to AWS to manage.

However, we are reminded here why pet instances are just that, sometimes they can be a bit of a pain, but you nurture them for a reason! In my case, I am just a little bit lazy, and didn't want to have to set everything up again!

Hopefully I will continue this IPv6 series soon, where I will go over a number of other services - to see if we can't push forward with the IPv6 transition.

Any comments or queries will be greatly appreciated!

Oz The Cat

For anyone that is wondering, this is Oz, he is a lovely cat!

GitHub Actions using AWS and OpenID

Colin Barker — Tue, 28 Feb 2023 19:23:56 +0000

Header photo by Roman Synkevych 🇺🇦 on Unsplash

What is OpenID?

Always a good start, understanding what the key component to this whole post is! As always though, I will reference Wikipedia

"OpenID is an open standard and decentralized authentication protocol promoted by the non-profit OpenID Foundation. It allows users to be authenticated by co-operating sites (known as relying parties, or RP) using a third-party identity provider (IDP) service..." Wikipedia - OpenID

Simple right! Well the technical details of how OpenID works is probably for a much more in-depth specific technical blog for this technology, but the one thing that we need to understand here is that OpenID allows users to be authenticated using 3rd party identify providers. In the case of AWS, their OpenID Connect set up would allow a service in GitHub to authenticate to AWS, and through the IAM system, assume a specific role.

The question is, why go to the effort to set this all up when a simple Access Key/Secret Key combination would work? Well, as you know from AWS Well-Architected best practices, you should always use temporary credentials over static ones. With the assumption of an AWS role, it uses temporary credentials. With OpenID Connect-compatible identity providers, such as GitHub, you would need to set this up using a Web Identity source. With this post, I will show you how I set this up for this blog (and for the Tokonatsu website!)

GitHub Actions using IAM Access Keys

This is where I started, as you can see below GitHub has my secrets for the AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY, statically set quite a while ago!

Within my GitHub actions pipeline, I am using the Configure AWS Credentials action from the GitHub Marketplace to configure the secrets for use in the pipeline.

- name: Configure AWS Credentials
  id: aws-credentials-configure
  uses: aws-actions/configure-aws-credentials@v1
  with:
    aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
    aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
    aws-region: ${{ secrets.AWS_DEFAULT_REGION }}

If you want to see the whole file, then there is a link here to the build-and-deploy.yaml (prior to the changes) file on GitHub.

As you can see, it is pretty simple, and it worked. The IAM user that owned those keys was happy to sit there and allow the pipeline to access the service. However, that IAM user is still static, and the keys will need to be manually be rotated. From a security stance, anyone that found that key out would be able to do as much as the pipeline could do to my website. As I use aws s3 sync to copy the website up, while also using the -delete parameter, it means that I need delete access for this IAM user! Not really the best!

By using Terraform, I was able to set up all the correct access and switch my pipeline over to using role assumption, thus temporary credentials. GitHub do provide a walkthrough to set up the OpenID Connect, which is what I based this configuration on. Along with the Terraform Documentation hopefully, this will help you with your journey!

Creation of the OpenID Connect Provider

Setting up the Identity Provider (IdP) will need to be the first step. This action will create a description in IAM of the external IdP and establishes the trust between your account and the organisation, in this case GitHub. This step requires just a few options, of which can be harder to get.

Using the walkthrough documentation, we can see that the following is required:

The provider URL - in the case of GitHub this is https://token.actions.githubusercontent.com
The "Audience" - which scopes what can use this. Confusingly in Terrraform this is also known as the client_id_list.
The Thumbprint of the endpoint - This one is the tricker one, as you will need to generate this yourself.

Generating the thumbprint

This part wasn't as clear in the GitHub documentation, but I went over to the AWS Documentation which gave me instructions on how to generate the thumbprint. You would need a copy of the openssl CLI o be able to do this, but the quickest way is as follows:

Use the OpenSSL command to check against the provider URL to get the certificate.

openssl s_client -servername token.actions.githubusercontent.com -showcerts -connect token.actions.githubusercontent.com:443

Grab the certificate shown in the output, you will see this starting with -----BEGIN CERTIFICATE-----, then place this content into a file. For this demo, I will use github_openid.crt.
Use the OpenSSL command again to generate the fingerprint from the file created above.

openssl x509 -in github_openid.crt -fingerprint -sha1 -noout

Which should output the fingerprint. Strip away all the extra parts, and the : between each of the pairs of hexadecimal characters, and you should end up with something like this:

6938fd4d98bab03faadb97b34396831e3780aea1

⚠️ Note: You will need to make the letters lowercase, as Terraform is case sensitive for the variable we need to put this in, but AWS is not case sensitive, so it can sent Terraform into a bit of a loop. ⚠️

Adding the resource in Terraform

With all of this, you can now start to put together the Terraform elements. We can use the Terraform resource for setting up the OpenID Connect Provider in IAM. Below is a code example, using the information gathered from the documentation and the thumbprint generation, and all placed into a single resource object.

resource "aws_iam_openid_connect_provider" "github" {
  url = "https://token.actions.githubusercontent.com"
  client_id_list = [
    "sts.amazonaws.com"
  ]
  thumbprint_list = [
    "6938fd4d98bab03faadb97b34396831e3780aea1"
  ]
}

This seems simple enough, but this is just authorising GitHub to be a trusted source, and that identities from GitHub can be used to authenticate against AWS. The IAM role set up is where the main bulk of granting access is completed.

Creating the IAM Role

IAM Policy - Bucket Access

Before access can be granted to the GitHub Actions pipeline, we will need to create a policy that defines the access that the role will have. There isn't much in the way of any difference to this part than creating any other policy however, for this blog and the Tokonatsu website, we need some additional permissions outside of the simple s3:PutObject.

In the example below, as my S3 bucket is also a resource in Terraform, you can see how I have pulled the bucket ARN for the resource from the outputs of the S3 bucket creation. It is normally best practice to reference variables and other outputs rather than using strings.

# tfsec:ignore:aws-iam-no-policy-wildcards
data "aws_iam_policy_document" "website_colins_blog_policy" {
  version = "2012-10-17"

  statement {
    effect = "Allow"
    resources = [
      aws_s3_bucket.website_colins_blog.arn,         # or "unique-bucket-name-for-site"
      "${aws_s3_bucket.website_colins_blog.arn}/*"   # or "unique-bucket-name-for-site/*"
    ]

    actions = [
      "s3:DeleteObject",
      "s3:GetBucketLocation",
      "s3:GetObject",
      "s3:ListBucket",
      "s3:PutObject"
    ]
  }
}

⚠️ Note: As I use tfsec to keep an eye on my Terraform, it attempts to look at the policy and look for anything that might be considered an issue. On the first line you can see the tfsec:ignore:aws-iam-no-policy-wildcards comment, which means tfsec will ignore that rule when it checks my Terraform. As we need to give this role access to do the listed actions on all objects in the bucket, a wildcard is easier. Hence the rule to stop the error from showing up. ⚠️

IAM Policy - Role Assumption

As we will be using rule assumption, there needs to be an additional policy document created that will be added to the Role, that can tell it who can assume the role. You might have seen a similar version when using sts:AssumeRole as an action, for principles that cover other accounts as an example. With the sts:AssumeRoleWithWebIdentity what we are telling AWS tha the role assumption needs to happen if they have a "web identity", one of the external providers.

The primary element to ensuring the right IdP is used when looking for which identity to grant access too, we need to reference the ARN of the OpenID Connect Provider added earlier. As the Terraform resource we used above was identified with aws_iam_openid_connect_provider.github, we can use one of its attributes to programmatically add the ARN to the principles. We will need to specify the type in Terraform as "Federated" as well.

The last two bits are condition blocks and these are unique to the GitHub OpenID Connect set up. There is a little more detail on the GitHub Actions: OpenID Connect in AWS page. Using this page, I am adding two "StringLike" tests to the role, that looks for two specific variables:

token.actions.githubusercontent.com:sub - Which is used to specify which repo's GitHub actions are allowed access.
token.actions.githubusercontent.com:aud - That ensures that AWS's STS service is the one that is requesting the identity type, and no others.

The token.actions.githubusercontent.com:sub in my actual repo it says that any repo in my personal space mystcb called blog using any branch, can access. While this is very open, my personal blog only really can be edited by myself, and I would want the role to also activate on any test branches as well. This is shown on my personal Terraform as repo:mystcb/blog:*. The example I have below is more specific, and only allows access if the GitHub action is working from the main branch. What I wanted to show here is a secure version, but also showing how you can add wildcards to the value to cover more branches if need be.

data "aws_iam_policy_document" "website_colins_blog_role_assumption" {

  version = "2012-10-17"

  statement {
    effect = "Allow"

    actions = ["sts:AssumeRoleWithWebIdentity"]

    condition {
      test     = "StringLike"
      variable = "token.actions.githubusercontent.com:sub"
      values   = ["repo:mystcb/blog:ref:refs/heads/main"]
    }

    condition {
      test     = "StringLike"
      variable = "token.actions.githubusercontent.com:aud"
      values   = ["sts.amazonaws.com"]
    }

    principals {
      type        = "Federated"
      identifiers = [aws_iam_openid_connect_provider.github.arn]
    }
  }

}

Now we have our two policy documents as Terraform data objects, we can then pull them together to create the role

Creation of the IAM Role

For us to create the role, we need to pull together all the bits we have created so far. This will mean doing a few special tricks with the data resources.

Firstly we need to create an IAM Policy resource, that the IAM Role resource can use to attach to the newly created role.

The IAM Policy resource require 3 elements, the name, path , and the policy in JSON format. While the name and path can be as custom as you need, the policy in JSON format is what might trip a few people up. The IAM Policy Document data object has just one Attribute output tha can be referenced here: json. This conversion means we can quickly add the three together to make the IAM Policy object in AWS.

resource "aws_iam_policy" "website_colins_blog" {
  name   = "access_to_website_colins_blog_s3"    # This is my example name!
  path   = "/"                                   # Root path for ease
  policy = data.aws_iam_policy_document.website_colins_blog_policy.json
}

Next, we need to create the role itself which has two key elements, the name and the assume_role_policy. The name is as you want this to be however, the assume_role_policy will be needed to let AWS IAM know what can assume this role. In our case, it is the JSON output from our second IAM Policy Document.

resource "aws_iam_role" "website_colins_blog_github_role" {
  name               = "access_to_website_colins_blog_s3_role"      # This is my example name!
  assume_role_policy = data.aws_iam_policy_document.website_colins_blog_role_assumption.json
}

Great! Now we have role, and we can assume it it - well not exactly, one last step. With IAM, you can attach multiple policies to a single role, which if you created in line with the aws_iam_role resource it can make it a little more complicated, and very long. The AWS provider allows us two methods to manage the role's policies. One is through managed_policy_arns/inline_policy or aws_iam_policy_attachment. The former two work very much the same, but takes exclusive authority over the state of the IAM Role itself. This means if you attach policies using the latter resource object, you will find Terraform getting stuck in a cycle. For this example, I am using the policy attachment resource.

resource "aws_iam_role_policy_attachment" "website_colins_blog" {
  role       = aws_iam_role.website_colins_blog_github_role.name
  policy_arn = aws_iam_policy.website_colins_blog.arn
}

This block is where all the references from before make this easier. This is where the role resource object, and the policy resource object come together to create the role. Now, AWS is aware of GitHub, its OpenID Connect provider, and we have given a specific repo's GitHub actions that run on the main branch access to assume a role in AWS, which give it access to AWS S3. The role assumption will use temporary credentials for each of the runs!

One last bit, this part will enable you to get the ARN for the role, which will be required for the configuration of GitHub.

output "website_colins_blog_role_arn" {
  description = "The role ARN for the Website: Colin's Blog Role"
  value       = aws_iam_role.website_colins_blog_github_role.arn
}

Very simple, it just outputs the ARN for the role which will need to be copied to GitHub. For the rest of the blog, I am going to be using an example role in my examples. This will not work, so make sure you are getting your role that matches your account. The ARN we will be using will be

arn:aws:iam::12326264843:role/access_to_website_colins_blog_s3_role

GitHub Actions/Workflow Updates

To enable this, we need to update the workflow yaml file, and this is probably the easiest bit of the whole post!

Add the Role ARN as a secret

This is where we need to move back to GitHub and grab the ARN from above, and add this as an add the URL to the Repository Secrets. You should be able to find your version at https://github.com/<yourname/org>/<yourrepo>/settings/secrets/actions. This is under Settings -> Secrets and variables -> Actions and click New Secret. Enter in a name for the secret, and its value which I have put in:

Name: AWS_ROLE_ARN
Secret: arn:aws:iam::12326264843:role/access_to_website_colins_blog_s3_role

Once you have added that, make sure to remove the two existing Repository Secrets, in my case I called them

AWS_ACCESS_KEY_ID
AWS_SECRET_ACCESS_KEY

The AWS CLI will always use the keys over role assumption in it's priority so always best to remove them. With the two older secrets removed you should now have just the AWS_ROLE_ARN and AWS_DEFAULT_REGION

Update the Workflow YAML file

For GitHub actions to be able to assume the role, there are two changes that need to be made to the workflow yaml file. The first one, will be the need to enable the workflow to interact with GitHub's OIDC Token endpoint. Part of the assumption process will require us to identify as a web identity from GitHub to have AWS know who we are. As such you will need to add additional permissions to the job. Specifically the following

id-token to write
contents to read

So it should look something like this:

jobs:
  hugo_build_and_deploy:
    runs-on: ubuntu-latest
    permissions:
      id-token: write
      contents: read

Later on in the pipeline, where we configured the AWS credentials before, you will need to remove the older secret variables, and put the new secrets in:

- name: Configure AWS Credentials
  id: aws-credentials-configure
  uses: aws-actions/configure-aws-credentials@v1
  with:
    role-to-assume: ${{ secrets.AWS_ROLE_ARN }}
    aws-region: ${{ secrets.AWS_DEFAULT_REGION }}

To see this whole file in context, I would recommend having a look at this blog's workflow on GitHub!

All you need to do now, is run the GitHub Actions workflow, and make sure it works! With my blog workflow, it was pretty much a drop in replacement for the IAM credentials. There were a few minor issues with my workflow, but nothing that following this wouldn't have resolved fo me!

Some minor issues

As you can see, it wasn't exactly first time running for me! It did take a while, and also I had placed a --acl public-read as part of the aws s3 sync command, which the new bucket I created had been set to block public ACLs!

There was one other issue, and that was with the GITHUB_TOKEN that is used. In normal operation, without the added permissions, this token worked fine with an additional Marketplace action called GitHub Deployments. However, changing this over to allow the OpenID Connect feature, it meant that the token mysteriously stopped working.

Switching to use a Fine-grained PAT

On the 18th of October 2022, GitHub offered up a new service called Fine-grained personal access tokens. The idea is that rather than creating a very open Personal Access Token (PAT), you could create a token that was very limited in it's reach. It is still in beta as I write this blog (28th Feb 2023).

Using this beta feature, I was able to create a new token, limiting it to specifically the blog repo, and specific permissions. The screenshot below shows the details about the new PAT. (I am aware I could probably reduce the permissions a little more!)

From here, I added a new respository secret called REPO_TOKEN with the value of the newly generated token, and then updated the part in the workflow that needed it:

- name: Set Deploy Status
  id: deploy-status-start
  uses: bobheadxi/deployments@v1
  with:
    step: start
    token: ${{ secrets.REPO_TOKEN }}
    env: Prod
    ref: ${{ github.ref }}

Round up

Hopefully what I have shown you in this post is how to move away from IAM Credentials, and use the OpenID Connect features of both AWS and GitHub to enable role based assumption to gain access to an S3 bucket that stores, in my case, a static website.

If you do have any questions, comments, please let me know!

Enabling IPv6 on AWS using Terraform (Part 1)

Colin Barker — Thu, 16 Feb 2023 14:49:21 +0000

Header photo by NASA on Unsplash

What is IPv6?

According to Wikipedia, Internet Protocol version 6 (IPv6) was introduced in December 1995 (just over 25 years ago!), based on the original IPv4 protocol that we all know and love today. The Internet Engineering Task Force (IETF) developed this new protocol to help deal with the (at the time) anticipated exhaustion of the IPv4 address range. This process seemed like it could be simple enough, create a new system to replace and older system and enable the expansion of the Internet to meet modern day standards.

This is where the problem lie. Adopting IPv6 wasn't as simple as just replacing IPv4. The two protocols are different enough that on top of physical hardware changes (older devices unable to support IPv6), it also meant a different way of thinking when it came to working with networking both on the Internet, and within (Intranets, local networks, home networks). One of the biggest issues that faced a lot of the world, is that ISP adoption rate was surprisingly low.

However, as the IPv4 Exhaustion happened as early as 2011, providers have started very quickly adopting the new standard.

So, what is IPv6?

I would recommend reading the Wikipedia for more details, as much of what I would write here, would essentially be copied just from that page! In summary though:

IPv6 uses 128-bit addresses over the IPv4 32-bit addresses, allowing approximately 3.4 x 10^38 addresses, over IPv4's 4,294,967,296 (2 x 10^32)
Addresses are in 8 groups of hexadecimal digits, separated by colons. For example: fd42:cb::123 in short hand. (which would expand to be fd42:00cb:0000:0000:0000:0000:0000:0123)
Route Aggregation is built into the addressing scheme allowing for the expansion of global route tables with minimal space used.
Steve Leibson (in a now lost article) one said "[If] we could assign an IPv6 address to every atom on the surface of the earth, [we would] still have enough addresses left to do another 100+ Earths!"

⚠️ Note: I mentioned short hand above, and this comes with a lot of caveats but the primary rule is, you can drop any zero in an IPv6 address, and it is assumed, as long as it is before the hexadecimal character. For example 00cb:0000:0001 can be shortened to cb::1. However, you can only have ONE :: in each address (otherwise how does it know where to shorten it!), so for example 00cb:0000:0100:0000:0001 must be shortened to cb:0:100::1. I'll go into this in more detail in a future post! ⚠️

This is why it is important to start embracing IPv6, we have a lot more space to make lives easier for the world, and the only way we can ensure the continued adoption of the protocol is to enable this everywhere.

In this post, I will go through how you can enable, using Terraform, IPv6 on your existing AWS Cloud Networks. When planning IPv6, there is a lot to consider, and there are some architectural changes will need to be considered.

Setting the scene - an existing AWS Cloud Network

This should be a very familiar layout to most people, an VPC in AWS with some very basic networking setup. In the diagram above, we have a VPC, with Public and Private Subnets in two availability zones. We have an Internet Gateway for the public subnets, and a NAT Gateway in each Availability Zone for the Private Subnets to talk to the internet. I have placed some sample IP addressing in, just for reference as part of this post.

If you wish to deploy this yourself, I have place some sample code on my GitHub that you can use. (https://github.com/mystcb/ipv6-on-aws/tree/main/01-sample-vpc)

Throughout this post, you will see me mention the cost of running this using an estimate. I have been using for a while, a tool called infracost which is an open source (with subscription based additions) cost estimator tool - https://www.infracost.io/. For this demonstration, using the sample code listed above, it would cost an estimated $76.65/month - so if you don't want rack up a bill, only deploy when you want to test, and use Terraform to destroy the services when you are done.

As an example, here it the cost estimate for this deployment:

# infracost breakdown --path=.

Evaluating Terraform directory at .
  ✔ Downloading Terraform modules
  ✔ Evaluating Terraform directory
  ✔ Retrieving cloud prices to calculate costs

Project: mystcb/ipv6-on-aws/01-sample-vpc

 Name                                     Monthly Qty  Unit            Monthly Cost

 aws_eip.natgwip[1]
 └─ IP address (if unused)                        730  hours                  $3.65

 aws_nat_gateway.sample_natgw_subnet_a
 ├─ NAT gateway                                   730  hours                 $36.50
 └─ Data processed                      Monthly cost depends on usage: $0.05 per GB

 aws_nat_gateway.sample_natgw_subnet_b
 ├─ NAT gateway                                   730  hours                 $36.50
 └─ Data processed                      Monthly cost depends on usage: $0.05 per GB

 OVERALL TOTAL                                                               $76.65
──────────────────────────────────

Remember, all costs are estimates! With the cloud, its pay as you use, utility based so the costs will be what you use. The above is just an estimate, so keep that in mind!

IPv6 and AWS

As mentioned before, there are some concepts that have to be considered when designing for IPv6. One specific concept for networking can seem a little confusing at first, but with the right security in place, will ensure that no accidental access to the service can happen.

Within AWS, IPv6 addresses are global unicast addresses. A unicast address is an address that identifies a unique object or node on a network. While the premise of a unicast address is not new (as it was the same in IPv6), with the exhaustion of IPv4 addresses, new methods to give unique addresses to multiple nodes was devised. Network Address Translation (NAT) was one such method for doing this. This allowed the mapping of a single unicast IP address to be used by multiple nodes, by routing traffic through the NAT node and allowing it to re-write the headers to make it seem like it had come from the unique address.

NAT is a wide subject, and I am sure I will write more about it some day, but a real world example that you see in most places is your home network. You have multiple private nodes with access to the internet, typically using a single public unicast address.

So how does this relate to IPv6 and AWS? Remember earlier, I mentioned that there are so many addresses available in the IPv6 ecosystem that you could give a unique address to every atom on the planet? Well, we can do just that, but to the nodes that require addresses. This is what AWS does, each IPv6 address you assign to nodes in AWS, are global unicast addresses, unique to each node.

This means a "private" IPv6 Subnet does sound like it might be complicated to set up, but actually it isn't as bad as you think! However, to start the process off, we will start with adding a IPv6 to the Public Subnets, to set the ground work, and go from there.

Enabling IPv6 on your VPC

Regardless of what you need IPv6 for, you need to enable this on your VPC. Just like your IPv4 CIDR that you assign to the VPC, you will need to assign a IPv6 CIDR to your VPC.

⚠️ Difference: Private IPv6 addresses do exist, but you can't assign them to AWS VPCs. You must use public CIDR ranges ⚠️

In a way, the above does make sense - every device is globally unique, so why would you need to make a private address. Personally, I use internal private addresses to make it easier to remember when connecting to servers, but I am very much an old-school person here, and you should be using name resolution to connect to instances!

Therefore, when you go to assign an IPv6 CIDR range to your VPC, you have one of three options:

For simplicity, I will be using the Amazon-provided CIDR ranges. In the future, I will go over the IPAM-allocated and BYOIP options, but for now these are just additional ways to get an IPv6 CIDR on your VPC.

AWS have access to a fairly large range of IPv6 addresses, this means they can allocate you a unique set of addresses just for you.

⚠️ Terminology: A Network Boarder Group is the name (chosen by AWS) that defines a unique set of Availability Zones or Local Zones where AWS advertises IP addresses ⚠️

When requesting an IPv6 CIDR from AWS, they will need you to select a Network Border Group. This is to ensure that the IPv6 addresses you are receiving can be routed successfully to your VPC. Back to the IPv6 description above, to make routing simpler in the IPv6 world, AWS will route specific ranges to specific groups, and therefore you have to select the right group. Thankfully, as the groups are quite large, for the majority of regions - there is only a single Network Border Group, and Terraform will select this automatically!

Lets start with the vpv.tf that exists in the sample code (https://github.com/mystcb/ipv6-on-aws/blob/main/01-sample-vpc/vpc.tf).

# Creation of the sample VPC for the region
#tfsec:ignore:aws-ec2-require-vpc-flow-logs-for-all-vpcs
resource "aws_vpc" "sample_vpc" {
  cidr_block = var.vpc_cidr

  tags = {
    "Name" = "Sample-VPC"
  }
}

Pretty simple, probably a little too simple! But keep in mind that this is just a sample for now!

Terraform has two resource parameters that will be used to set and assign the IPv6 CIDR to the VPC. assign_generated_ipv6_cidr_block and ipv6_cidr_block_network_border_group. The border group is used a lot more with Local Zones, so we don't need to worry about this at the moment, but do keep this in mind for more complex deployments.

Just setting the assign_generated_ipv6_cidr_block to true is enough for AWS to assign the VPC a CIDR range. Terraform documentation.

Your file should now look like this:

# Creation of the sample VPC for the region
#tfsec:ignore:aws-ec2-require-vpc-flow-logs-for-all-vpcs
resource "aws_vpc" "sample_vpc" {
  cidr_block                       = var.vpc_cidr
  assign_generated_ipv6_cidr_block = true

  tags = {
    "Name" = "Sample-VPC"
  }
}

With this, your terraform plan should now look like this:

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  ~ update in-place

Terraform will perform the following actions:

  # aws_vpc.sample_vpc will be updated in-place
  ~ resource "aws_vpc" "sample_vpc" {
      ~ assign_generated_ipv6_cidr_block     = false -> true
        id                                   = "vpc-0123456789abcdef0"
      + ipv6_association_id                  = (known after apply)
      + ipv6_cidr_block                      = (known after apply)
        tags                                 = {
            "Name" = "Sample-VPC"
        }
        # (16 unchanged attributes hidden)
    }

Plan: 0 to add, 1 to change, 0 to destroy.

As you can see with the plan, this will grab some additional details to add to the resource object as attributes to reference later on. This will be key to make your terraform portable, and not hard coded!

Once applied, this will then allocate the IPv6 CIDR to the VPC, and you should then see the following!

There we go, we have hit the first step! IPv6 is now enabled on the VPC! As you can see, we have received a /56 block of IP's. That gives you the room to have a total of 4,722,366,482,869,645,500,000 hosts in your network. I don't think we will be running out any time soon!

Next step, lets assign to each of the subnets their own CIDR, so that resources in the subnets can assign their own IPv6 address.

Adding IPv6 CIDRs to the public subnets

Just like IPv4 CIDR's, you can break down the IPv6 range you have into smaller ranges that are all routed internally using the AWS's VPC networking. With a manual set up you would normally do something like this:

IPv4 VPC CIDR: 192.168.0.0/20
Public Subnet in AZ1: 192.168.10.0/24
Public Subnet in AZ2: 192.168.11.0/24

This is shown in the sample VPC we are using in this post. Simple enough right? Breaking down the CIDR into smaller subnets, and then assigning them to the correct location. It is very much the same with IPv6, but the ranges are just that much larger, that sometimes its best to use an automated method for doing this. What everyone should be doing, is the automatic generation of the ranges for IPv4 as well, which is available in the example!

To do this, we can use a terraform function called cidrsubnet (https://developer.hashicorp.com/terraform/language/functions/cidrsubnet). This function can calculate the subnet addresses within a given IP network block or CIDR, and then be used as a value for a variable in the aws_subnet resource block.

This function takes a bit of getting used to, but here is my trick for understanding how it works!

The function looks like this:

cidrsubnet(prefix, newbits, netnum)

For further details, feel free to read the documentation above, but for our sample we will use the following:

prefix the CIDR range. Available as an attribute as this is generated by AWS. aws_vpc.sample_vpc.ipv6_cidr_block
newbits this is the number of additional bits you need to add to the CIDR prefix, to break the network down. In our case we will use 8 as it is a round number, but you will need to size this to your specifications.
netnum this is the network number assigned to the broken down blocks that you will select for this subnet. It can't be more than the newbits and you can't use the same netnum on multiple subnets.

The trick I have used is as follows. newbits is calculated as the difference between the CIDR's prefix /56 and the network size you need. So in our case I would like to make each subnet a /64 in size. The difference between the two (64 - 56 = 8) means the bit difference is 8. For the moment, I will leave this here, but I will create an article in the future that explains how this works, and why its the number of bits!

For the netnum I tend to visualise it in the diagram - I wanted 4 subnets in my sample, (2 public, 2 private), so I am able to reference the networks created above by their counter, with the first network being 0, with the last network being 3 (as your counter started at 0).
So for our networks, we can enter the values and get the following:

cidrsubnet(aws_vpc.sample_vpc.ipv6_cidr_block, 8, 0) # Subnet 1 - x:x::0:0/64
cidrsubnet(aws_vpc.sample_vpc.ipv6_cidr_block, 8, 1) # Subnet 2 - x:x::1:0/64
cidrsubnet(aws_vpc.sample_vpc.ipv6_cidr_block, 8, 2) # Subnet 3 - x:x::2:0/64
cidrsubnet(aws_vpc.sample_vpc.ipv6_cidr_block, 8, 3) # Subnet 4 - x:x::3:0/64

The next step is the parameter for the aws_subnet resource block, which is ipv6_cidr_block. Essentially the same as the cidr_block parameter, but for IPv6!

So for each of our public subnets, lets add this in. In our sample file vpc_subnets.tf (https://github.com/mystcb/ipv6-on-aws/blob/main/01-sample-vpc/vpc_subnets.tf), we have two public subnets, a and b. So lets make the change. Below is the example with the new parameters added:

# Public Subnet A
resource "aws_subnet" "public_a" {
  vpc_id               = aws_vpc.sample_vpc.id
  ipv6_cidr_block      = cidrsubnet(aws_vpc.sample_vpc.ipv6_cidr_block, 8, 0)
  cidr_block           = cidrsubnet(aws_vpc.sample_vpc.cidr_block, 4, 10)
  availability_zone_id = data.aws_availability_zones.available.zone_ids[0]
  tags = {
    "Name" = "Public-Subnet-A"
  }
}

# Public Subnet B
resource "aws_subnet" "public_b" {
  vpc_id               = aws_vpc.sample_vpc.id
  ipv6_cidr_block      = cidrsubnet(aws_vpc.sample_vpc.ipv6_cidr_block, 8, 1)
  cidr_block           = cidrsubnet(aws_vpc.sample_vpc.cidr_block, 4, 11)
  availability_zone_id = data.aws_availability_zones.available.zone_ids[1]
  tags = {
    "Name" = "Public-Subnet-B"
  }
}

Once again, we run our terraform plan and we should get an output similar to this:

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  ~ update in-place

Terraform will perform the following actions:

  # aws_subnet.public_a will be updated in-place
  ~ resource "aws_subnet" "public_a" {
        id                                             = "subnet-0123456789abcdefg"
      + ipv6_cidr_block                                = "xxxx:yyyy:zzzz:nnn1::/64"
        tags                                           = {
            "Name" = "Public-Subnet-A"
        }
        # (15 unchanged attributes hidden)
    }

  # aws_subnet.public_b will be updated in-place
  ~ resource "aws_subnet" "public_b" {
        id                                             = "subnet-gfedcba9876543210"
      + ipv6_cidr_block                                = "xxxx:yyyy:zzzz:nnn2::/64"
        tags                                           = {
            "Name" = "Public-Subnet-B"
        }
        # (15 unchanged attributes hidden)
    }

Plan: 0 to add, 2 to change, 0 to destroy.

The plan shows the addition of the new CIDR block to each subnet, noting the network number changing slightly to accommodate the size we requested. Once applied, we should see these details in the console:

Perfect, now let's launch an EC2 instance and then try and connect to something using the IPv6 address!

Well, almost there - we re missing the routing.

Adding IPv6 Routing to the public subnets

Having a look at our existing route tables, we can see that AWS added in the route for the VPC IPv6 CIDR to target the local VPC in the route table, which means it can find resources in the VPC that have the IPv6 address that is within the CIDR. Great for local traffic, but we need to be able to access the outside world using IPv6!

⚠️ Note: While traffic can still route with IPv4, we need to enable routing for IPv6, otherwise any traffic inbound to the server won't be able to send traffic back. ⚠️

We can also see this in the sample code too, in the vpc_public_routing.tf file (https://github.com/mystcb/ipv6-on-aws/blob/main/01-sample-vpc/vpc_public_routing.tf)

# Primary Sample Route Table (Public)
resource "aws_route_table" "public_rt" {
  vpc_id = aws_vpc.sample_vpc.id

  tags = {
    "Name" = "Public-Route-Table"
  }
}

# Route entry specifically to allow access to the Internet Gateway
resource "aws_route" "public2igw" {
  route_table_id         = aws_route_table.public_rt.id
  destination_cidr_block = "0.0.0.0/0"
  gateway_id             = aws_internet_gateway.sample_igw.id
}

Here we can see the Public-Route-Table resources, but it only shows traffic to the Internet Gateway (IGW) for IPv4 traffic. We will need to add a route to allow IPv6 traffic to hit the IGW.

We can do this by creating a new aws_route resource, but we need to use the destination_ipv6_cidr_block parameter instead.

For the destination though, with IPv4 you could use the block 0.0.0.0/0 to mean "all traffic". If we were to type the IPv6 version out in full, it would look like 0000:0000:0000:0000:0000:0000:0000:0000/0. Bit of a pain! Thankfully we can apply the short hand rule mentioned at the start, remove all the 0's, shrink down, and you get quite simply ::/0, which suddenly seems much shorter than the IPv4 version! With this we can add this as the destination IPv6 block.

resource "aws_route" "public2igw_ipv6" {
  route_table_id              = aws_route_table.public_rt.id
  destination_ipv6_cidr_block = "::/0"
  gateway_id                  = aws_internet_gateway.sample_igw.id
}

Once again, lets run the terraform plan and we should get something like this:

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  # aws_route.public2igw_ipv6 will be created
  + resource "aws_route" "public2igw_ipv6" {
      + destination_ipv6_cidr_block = "::/0"
      + gateway_id                  = "igw-0123456789abcdefg"
      + id                          = (known after apply)
      + instance_id                 = (known after apply)
      + instance_owner_id           = (known after apply)
      + network_interface_id        = (known after apply)
      + origin                      = (known after apply)
      + route_table_id              = "rtb-0123456789abcdefg"
      + state                       = (known after apply)
    }

Plan: 1 to add, 0 to change, 0 to destroy.

And once applied, we should be able to see the new route in the routing table!

Jumping back on our EC2 instance, and we get a connection!

⚠️ Attention: Inbound traffic to an EC2 instance, for example, will still be protected by its Security Group. The rules in a security group are specific to the IP protocol, so an allow for an IPv4 inbound rule, will only allow that. Make sure that you add any additional IPv6 rules to the security group to permit inbound access to resources in the Public Subnet ⚠️

Adding IPv6 outbound routing to the private subnets

We are getting to the last part of this post about enabling IPv6 on AWS, and we do need to cover the private subnets to complete the sample network configuration. This part will come with a few warnings, but if you are keeping in line with the AWS Well-Architected Framework, this will not be an issue at all!

Thinking back to what we mentioned before, AWS will use global unicast addresses for resources and services in AWS. Meaning, that you do not have a "private" CIDR for your private subnets. As you know, in IPv4 there is the RFP1918 that defines a number of IP blocks, specifically for "internal connectivity". These IP ranges are not routable on the public internet. This allowed the expansion of devices within a private network, without using up the public internet space. As IPv6 can assign every device on the planet, it makes it easier for us to ensure that the address assigned to a node is unique.

Next we have to look at the definition of what AWS considers a "public" subnet and a "private" subnet. A "public" subnet, is considered such, whereby the route table for the subnet points its outbound internet traffic directly to an Internet Gateway (IGW), and the IGW allows external traffic to route to the subnet. For a "private" subnet, there is no IGW for it to connect to, and you would use a service such as a NAT Gateway (NGW) to connect through to the internet, and as the NAT Gateway doesn't allow traffic inbound to the network, and the IP address of the node will be a non-internet rotatable IP address, traffic can't reach the service in the VPC.

This definition sill applies to Private IPv6 subnets, and this is why AWS created the IPv6 Egress-Only Internet Gateway. Much like the IPv4 Internet Gateway counterpart, the IPv6 Egress-Only Internet Gateway is a horizontally scaled, redundant, and highly available VPC component that allows outbound communication for IPv6 traffic to the internet. As this new IPv6 egress-only internet gateway only permits traffic outbound, and not inbound, this will make a subnet private, as traffic can not reach it.

As this is a VPC component, and not a service like the NAT Gateway, you technically only need to deploy one, like the Internet Gateway, and point outbound traffic to the egress-only internet gateway, and it will scale as needed.

With this in mind, lets start by adding in the Egress-only Internet Gateway. This is created by the terraform resource aws_egress_only_internet_gateway (https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/egress_only_internet_gateway). This is very similar to the normal Internet Gateway, and even the set up is the same.

Within the vpc_private_routing.tf file (https://github.com/mystcb/ipv6-on-aws/blob/main/01-sample-vpc/vpc_private_routing.tf) you will need to add the following resource:

# IPv6 Egress-only Internet Gateway
resource "aws_egress_only_internet_gateway" "sample_ipv6_egress_igw" {
  vpc_id = aws_vpc.sample_vpc.id

  tags = {
    "Name" = "Sample-VPC-IPv6-Egress-Only-IGW"
  }
}

Once again, running terraform plan will output something similar to this:

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  # aws_egress_only_internet_gateway.sample_ipv6_egress_igw will be created
  + resource "aws_egress_only_internet_gateway" "sample_ipv6_egress_igw" {
      + id       = (known after apply)
      + tags     = {
          + "Name" = "Sample-VPC-IPv6-Egress-Only-IGW"
        }
      + tags_all = {
          + "Environment" = "Sandbox"
          + "Name"        = "Sample-VPC-IPv6-Egress-Only-IGW"
          + "Source"      = "Terrform"
        }
      + vpc_id   = "vpc-0123456789abcdefg"
    }

Plan: 1 to add, 0 to change, 0 to destroy.

At this point, I would like to point out - that all the changes we have made so far, have not increased the cost of running! Much like the Internet Gateway, the only cost you have is the outbound traffic. As the Egress-Only Internet Gateway is the same, there is no cost for running this in a VPC other than the outbound traffic you intend to push through it. Unlike the NAT Gateways, which will cost you about $36.50/month, and then for best practice, you would need one in each availability zone, which then adds up. It does make IPv6 only networking in AWS far cheaper than IPv4!
Apply the changes, and the new Egress-only Internet Gateway will be created.

Adding IPv6 CIDRs to the private subnets

The next step, is that we need to add the CIDR blocks to the private subnets. This is pretty much identical to the public subnet addition, in fact, it is identical. This is due to there being no difference between public blocks and private blocks within the VPC IPv6 range. So for speed, we just repeat the same.

Head back to our sample file vpc_subnets.tf (https://github.com/mystcb/ipv6-on-aws/blob/main/01-sample-vpc/vpc_subnets.tf), and look for the two "private" subnets. Add in the ipv6_cidr_block for each of them, based off the next two blocks calculated earlier:

# Private Subnet A
resource "aws_subnet" "private_a" {
  vpc_id               = aws_vpc.sample_vpc.id
  ipv6_cidr_block      = cidrsubnet(aws_vpc.sample_vpc.ipv6_cidr_block, 8, 2)
  cidr_block           = cidrsubnet(aws_vpc.sample_vpc.cidr_block, 4, 12)
  availability_zone_id = data.aws_availability_zones.available.zone_ids[0]
  tags = {
    "Name" = "Private-Subnet-A"
  }
}

# Private Subnet B
resource "aws_subnet" "private_b" {
  vpc_id               = aws_vpc.sample_vpc.id
  ipv6_cidr_block      = cidrsubnet(aws_vpc.sample_vpc.ipv6_cidr_block, 8, 3)
  cidr_block           = cidrsubnet(aws_vpc.sample_vpc.cidr_block, 4, 13)
  availability_zone_id = data.aws_availability_zones.available.zone_ids[1]
  tags = {
    "Name" = "Private-Subnet-B"
  }
}

Run the terraform plan and apply the changes to assign the blocks to the subnets.

Adding IPv6 Routing to the private subnets

This is where the limitations with the IPv4 NAT Gateway makes the changes for a private subnet add additional operational overhead to the work needing to be completed. In our sample code, we created two route tables for the private subnets, one for each availability zone, so that traffic within the subnet routes through the NAT Gateway for that availability zone.

This means, rather than like the public subnet, where we need to add just a single route in terraform. We have to add two and attach them to both route tables.

If you were creating an IPv6 only VPC, then you could reduce the work and have a single route table that works for both availability zones! However, we will look at this at a later date.

This time, we need to head to the vpc_private_routing.tf file (https://github.com/mystcb/ipv6-on-aws/blob/main/01-sample-vpc/vpc_private_routing.tf) again, and we need to add in the two new routes.

If you look at the file, you will see that for the IPv4 NAT Gateway, there is a specific parameter that you use to tell terraform to issue the right API command to AWS to add the route, which you can see here:

# Route for Subnet A to access the NAT Gateway
resource "aws_route" "private2natgwa" {
  route_table_id         = aws_route_table.private_rt_a.id
  destination_cidr_block = "0.0.0.0/0"
  nat_gateway_id         = aws_nat_gateway.sample_natgw_subnet_a.id
}

The nat_gateway_id will be specifically used to pointing to the ID of the NAT Gateway within the availability zone. Much like in the public subnet you would use gateway_id for the Internet Gateway, you can use the egress_only_gateway_id for the IPv6 traffic.

Therefore, we will need to add 2 new blocks to the terraform file, to add in the route ::/0 to point to the Egress-only Internet Gateway.

⚠️ Attention: Remember, for IPv6 routes, you will need to use the destination_ipv6_cidr_block as part of the route table resource ⚠️

# Route for Subnet A to access the Egress-Only Internet Gateway
resource "aws_route" "private2ipv6egressigwa" {
  route_table_id              = aws_route_table.private_rt_a.id
  destination_ipv6_cidr_block = "::/0"
  egress_only_gateway_id      = aws_egress_only_internet_gateway.sample_ipv6_egress_igw.id
}

# Route for Subnet B to access the Egress-Only Internet Gateway
resource "aws_route" "private2ipv6egressigwb" {
  route_table_id              = aws_route_table.private_rt_b.id
  destination_ipv6_cidr_block = "::/0"
  egress_only_gateway_id      = aws_egress_only_internet_gateway.sample_ipv6_egress_igw.id
}

For one final time, run the terraform plan and you should see an output similar to this:

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  # aws_route.private2ipv6egressigwa will be created
  + resource "aws_route" "private2ipv6egressigwa" {
      + destination_ipv6_cidr_block = "::/0"
      + egress_only_gateway_id      = "eigw-0123456789abcdefg"
      + id                          = (known after apply)
      + instance_id                 = (known after apply)
      + instance_owner_id           = (known after apply)
      + network_interface_id        = (known after apply)
      + origin                      = (known after apply)
      + route_table_id              = "rtb-0123456789abcdef1"
      + state                       = (known after apply)
    }

  # aws_route.private2ipv6egressigwb will be created
  + resource "aws_route" "private2ipv6egressigwb" {
      + destination_ipv6_cidr_block = "::/0"
      + egress_only_gateway_id      = "eigw-0123456789abcdefg"
      + id                          = (known after apply)
      + instance_id                 = (known after apply)
      + instance_owner_id           = (known after apply)
      + network_interface_id        = (known after apply)
      + origin                      = (known after apply)
      + route_table_id              = "rtb-0123456789abcdef2"
      + state                       = (known after apply)
    }

Plan: 2 to add, 0 to change, 0 to destroy.

Remember, that we are adding two routes, as we have two route tables to make the change to. Apply the changes, and you should see the new route in the route tables:

So, jumping on an EC2 instance in the private subnet (you can see on the command line that the instance is in the 192.168.12.0/24 subnet), you can see we have complete outbound access on IPv6!

The final outcome

So you finally did it, you have enabled IPv6 on your VPC. If all went to plan, you should have something that looks like this:

To make things a little more simpler, you can also check out the 02-vpc-with-ipv6 folder in the sample code (https://github.com/mystcb/ipv6-on-aws/tree/main/02-vpc-with-ipv6), which will produce the same output
as the diagram, and if you followed the changes!

Summary and round up

Thank you for getting this far! There is a lot here, but hopefully I have been able to show you how to add IPv6 to an AWS VPC using Terraform. However, this is only the beginning. Throughout my time working with IPv6, there will always be different issues to trip you up, and there are far more features than just a simple VPC!

In a future post I hope to show you the following:

Adding IPv6 to an existing EC2 instance inside a VPC without rebuilding it. A little harder than I expected with Terraform, but I did find a way around it!
IPv6 only VPCs! Much cheaper to run that a normal IPv4 VPC, but at what "cost", adoption of IPv6 is still quite low, but there are a number of design patterns that mean an IPv6 VPC might work better for some situations.
Going into IPv6 in more detail. This is only the very basic information on IPv6, and a lot of what I have mentioned, does come with caveats! I will hope to explain these in more details later on
Bits? Why does one number mean something else, and why on earth do we count in bits? - Hope you have learnt binary for this one!

Thank you again, and feel free to send me any questions, or help me with any corrections to this post as well! Hopefully, I will see you in a future post!

P.S. The final cost of me running this for the creation of the post, was mainly the NAT Gateways! Everything else was free! It only cost me $1.50 total! Always make sure you run terraform destroy at the end!