Forem: Anthony Wat

Deploying LibreChat on Amazon ECS using Terraform

Anthony Wat — Mon, 06 Apr 2026 01:56:39 +0000

Introduction

Generative AI has fundamentally shifted how we approach work, from writing and coding to research and problem-solving. For the past year, I used ChatGPT Business almost daily at work to improve my writing and research. However, I noticed limitations like fabrication and confirmation bias, so I wanted to explore how other non-OpenAI models perform. Additionally, my organization is consolidating on Microsoft 365 Copilot, which doesn't match ChatGPT's capabilities for my needs. This led me to search for a self-hosted, ChatGPT-like platform with flexibility in model choices.

I also needed it to be web-based for team members to access. As an AWS advocate, I wanted to leverage a diverse set of foundational models that Amazon Bedrock has to offer, and to host the platform using primarily AWS services. Based on my research, the three main options are LibreChat, Open WebUI, and AnythingLLM. Given that LibreChat is more feature-rich, customizable, and seemingly easier to deploy, I decided to give it a try and share my experience.

Without further ado, let's walk through the solution architecture and how it addresses my requirements.

Architecture Overview

The main design principle of the solution is to be cost-effective initially while allowing flexibility to scale in the future. Minimizing cost also means reducing operational overhead, not just service charges.

While cramming all components into an Amazon Lightsail instance is the cheapest option, it would need to be re-architected to scale horizontally. Deploying to an Amazon EC2 instance provides more flexibility, but it requires manual LibreChat installation and VM management. Ultimately, I decided to take a more modern approach and adopt a componentized architecture depicted in the following diagram:

This architecture uses the following technologies:

MongoDB Atlas - The LibreChat database runs on MongoDB Atlas using the Free (M0) cluster tier. It's technically free, runs in AWS, and is sufficient as a starter database engine as recommended by LibreChat.
Amazon ECS with AWS Fargate - The LibreChat application runs as a container with 512 (0.5) vCPU and 1 GB memory using 64-bit ARM architecture, which is sufficient when not enabling too many LibreChat features. Secrets are stored in AWS Systems Manager (SSM) Parameter Store (free), and configurations are stored in an Amazon S3 bucket to avoid additional shared storage services.
Application Load Balancer (ALB) - The public-facing endpoint. Although there is a running cost, its simpler TLS setup and support for scaling and AWS WAF integration makes it worthwhile.
fck-nat - Provides NAT gateway functionality using a pair of EC2 t4g.nano instances instead of AWS-managed NAT Gateways. This significantly reduces NAT gateway data transfer charges, making it a cost-effective option for modest traffic volumes.

The monthly cost of this architecture should be about $50 USD in us-east-1 including moderate Bedrock model use. To prevent surprise bills, set up a budget in AWS Budgets and create a cost monitor in AWS Cost Anomaly Detection.

Now that we have a good understanding of the architecture, let's go through the LibreChat concepts and prerequisites before we look at the Terraform configuration.

Deploying a MongoDB Atlas Database

Since an official Terraform MongoDB Atlas Provider is available, let's use it to provision the database for LibreChat. If you have not already done so, sign up for a new account using the Get Started button on the MongoDB website.

Once you've completed sign-up, log in to the MongoDB Atlas Console. MongoDB Atlas automatically creates an organization and a sample project named Project 0. Since we'll use Terraform to create a new project, feel free to delete this sample project. You may also edit the organization name in Organizational Settings as needed.

Next, create an API key at the organization level for the Terraform provider. In the MongoDB Atlas Console, go to the organization level view and select Identity & Access > Applications in the left menu. On the Application page, select the Service Accounts tab and click Create service account:

On the Create Service Account page, enter a name (for example, "terraform") and a description (for example, "Service account for Terraform"), keep the client secret expiration as recommended, select the Organization Project Creator permission, and click Create. Copy both the client ID and secret from the next page for use with Terraform:

We're now ready to write the Terraform configuration to:

Create a new project
Create a new Free (M0) cluster with AWS as the backing provider
Create a database user for LibreChat
Add the NAT Gateway's public IP to the project IP Access List for security

To avoid hardcoding credentials, set the service account credentials as environment variables before running Terraform using MONGODB_ATLAS_CLIENT_ID and MONGODB_ATLAS_CLIENT_SECRET as per the provider configuration.

The following Terraform configuration provisions these MongoDB Atlas resources. Note that some attributes are provided as variables for flexibility, and the IP access list CIDR block refers to the NAT service elastic IPs (since egress traffic from LibreChat container goes through the fck-nat instances or NAT gateways):

resource "mongodbatlas_project" "librechat" {
  org_id = var.atlas_org_id
  name   = var.atlas_project_name
}

resource "mongodbatlas_advanced_cluster" "librechat" {
  project_id   = mongodbatlas_project.librechat.id
  name         = var.atlas_cluster_name
  cluster_type = "REPLICASET"

  replication_specs = [
    {
      region_configs = [
        {
          electable_specs = {
            instance_size = "M0"
          }
          provider_name         = "TENANT"
          backing_provider_name = "AWS"
          region_name           = var.atlas_region_name
          priority              = 7
        }
      ]
    }
  ]
}

resource "mongodbatlas_project_ip_access_list" "librechat" {
  for_each   = var.use_fck_nat ? aws_eip.fck_nat : aws_nat_gateway.zonal
  project_id = mongodbatlas_project.librechat.id
  cidr_block = "${each.value.public_ip}/32"
  comment    = "ECS egress CIDR for LibreChat"
}

resource "random_password" "atlas_db" {
  length = 16
}

resource "mongodbatlas_database_user" "librechat" {
  project_id         = mongodbatlas_project.librechat.id
  username           = var.atlas_db_username
  password           = random_password.atlas_db.result
  auth_database_name = "admin"

  roles {
    role_name     = "readWriteAnyDatabase"
    database_name = "admin"
  }
}

Although using an IAM role to authenticate the Atlas database user would be more secure, it unfortunately doesn't work with the off-the-shelf LibreChat Docker image because it lacks the aws4 module required by the Mongoose library for AWS authentication. To avoid rebuilding a new container image, we'll stick with password authentication for now.

Strategies for Managing LibreChat Configuration

LibreChat uses two main sources of configuration: environment variables and LibreChat YAML. Environment variables are typically provided via a .env file created from the example in LibreChat's GitHub repository. Most LibreChat configuration is done using environment variables, and the LibreChat YAML file references these variables using the ${} notation for values such as API keys.

Additionally, the LibreChat YAML file (librechat.yaml) is typically placed in the application folder or mounted as an override in a containerized environment. However, it can also be provided in other locations by specifying the configuration path using the CONFIG_PATH environment variable.

Running LibreChat as a container introduces some challenges:

Building custom images is inefficient - While it's possible to build a LibreChat container image with .env and librechat.yaml baked in, rebuilding the image for every configuration change is inefficient. It's ideal to use the official LibreChat image from Docker Hub.
Managing many environment variables is difficult - Setting environment variables directly without using .env is hard to manage due to the sheer number of variables to configure, even when omitting those irrelevant to your use case.
Security concerns - Providing security-sensitive information as plain-text environment variables is not a security best practice.

ECS provides features to address these concerns:

Environment variable files - ECS supports passing environment variables via an environment variable file stored in S3. Since LibreChat already uses this format, it's a perfect fit.
Secrets management - AWS Secrets Manager or AWS Systems Manager (SSM) Parameter Store can securely pass sensitive data to containers as environment variables, avoiding hardcoded credentials.

To keep costs low, we will define all sensitive data as SSM Parameter Store parameters of type SecureString, while keeping all other configuration in a .env file provided to the container via ECS.

Providing the LibreChat YAML file to the container is more complicated. While the standard approach uses persistent storage options like Amazon EFS, that's cost-prohibitive for managing a single file. A better approach is using a sidecar container to write the file to a task-level shared volume before the main container starts.

For this, I implemented a sidecar container using busybox to decode a base64-encoded librechat.yaml (provided as an environment variable) and write it to /config/librechat.yaml in the task-level shared volume. The LibreChat container references this path using the CONFIG_PATH environment variable. The resulting container definitions (defined in the ECS task definition) are shown below:

  container_definitions = jsonencode([
    {
      name      = "init-librechat-config"
      image     = "public.ecr.aws/docker/library/busybox:1.36"
      essential = false

      command = [
        "sh",
        "-lc",
        "mkdir -p /config && printf '%s' \"$LIBRECHAT_YAML_B64\" | base64 -d > /config/librechat.yaml"
      ]

      environment = [
        {
          name  = "LIBRECHAT_YAML_B64"
          value = filebase64("${path.module}/librechat/${var.librechat_version}/librechat.yaml")
        }
      ]

      mountPoints = [
        {
          sourceVolume  = "librechat-config"
          containerPath = "/config"
          readOnly      = false
        }
      ]

      logConfiguration = {
        logDriver = "awslogs"
        options = {
          "awslogs-group"         = aws_cloudwatch_log_group.librechat.name
          "awslogs-region"        = var.region
          "awslogs-stream-prefix" = "librechat"
        }
      }
    },
    {
      name      = "librechat"
      image     = "librechat/librechat:v0.8.4"
      essential = true

      dependsOn = [
        {
          containerName = "init-librechat-config"
          condition     = "SUCCESS"
        }
      ]

      portMappings = [
        {
          containerPort = 3080
          protocol      = "tcp"
        }
      ]

      secrets = [
        {
          name      = "CREDS_KEY"
          valueFrom = aws_ssm_parameter.librechat_creds_key.arn
        },
        {
          name      = "CREDS_IV"
          valueFrom = aws_ssm_parameter.librechat_creds_iv.arn
        },
        {
          name      = "JWT_SECRET"
          valueFrom = aws_ssm_parameter.librechat_jwt_secret.arn
        },
        {
          name      = "JWT_REFRESH_SECRET"
          valueFrom = aws_ssm_parameter.librechat_jwt_refresh_secret.arn
        },
        {
          name      = "MEILI_MASTER_KEY"
          valueFrom = aws_ssm_parameter.librechat_meili_master_key.arn
        },
        {
          name      = "MONGO_URI"
          valueFrom = aws_ssm_parameter.librechat_mongo_uri.arn
        }
      ]

      environment = [
        {
          name  = "CONFIG_PATH"
          value = "/config/librechat.yaml"
        }
      ]

      environmentFiles = [
        {
          value = aws_s3_object.librechat_dot_env.arn
          type  = "s3"
        }
      ]

      mountPoints = [
        {
          sourceVolume  = "librechat-config"
          containerPath = "/config"
          readOnly      = true
        }
      ]

      logConfiguration = {
        logDriver = "awslogs"
        options = {
          "awslogs-group"         = aws_cloudwatch_log_group.librechat.name
          "awslogs-region"        = var.region
          "awslogs-stream-prefix" = "librechat"
        }
      }
    }
  ])

Now that we have a strategy for managing LibreChat configuration, let's define the minimal set of environment variables and LibreChat YAML as a starting point.

Preparing the Starter Environment File

LibreChat provides starter files .env.example and librechat.example.yaml that we'll use as the basis for our configuration. Let's start with the environment file.

Environment File (.env)

Copy the .env.example file to a local folder and comment out any sensitive values that will be provided as SSM Parameter Store environment variables specified below. Although individually defined environment variables take precedence over variables in environment files per the Amazon ECS Developer Guide, commenting them out avoids confusion.

In addition to MONGO_URI (the MongoDB connection string), LibreChat recommends adjusting any "secret" values from their default value for added security:

CREDS_IV - 16-byte Initialization Vector (IV) (32 characters in hex) for securely storing credentials
CREDS_KEY - 32-byte key (64 characters in hex) for securely storing credentials
JWT_SECRET - 32-byte key (64 characters in hex) as the JWT secret key
JWT_REFRESH_SECRET - 32-byte key (64 characters in hex) as the JWT refresh secret key
MEILI_MASTER_KEY - 16-byte key (32 characters in hex) as the MeiliSearch master key (required only if message and conversation search is enabled)

LibreChat provides a Credentials Generator to generate cryptographically secure random values for these secrets. Store them as SSM Parameter Store SecureString parameters or generate and store them directly using Terraform.

Next, adjust these environment variables for proper and secure operation:

HOST - Set to 0.0.0.0 to listen on all network interfaces, allowing ALB access
CONSOLE_JSON - Set to true to write logs to CloudWatch in JSON format for easier querying
ALLOW_REGISTRATION - Set to false to disable self-registration (we will create users with the create user script instead)
SEARCH - Set to false to disable message and conversation search (we're only demonstrating a minimal setup)

For the AI provider, we'll enable AWS Bedrock. Set ENDPOINTS to bedrock (the .env.example enables all pre-configured endpoints except Bedrock). Then configure the Bedrock endpoint by setting the following environment variables:

BEDROCK_AWS_DEFAULT_REGION - Set to your Bedrock region (e.g., us-east-1)
BEDROCK_AWS_MODELS - Set to us.amazon.nova-2-lite-v1:0 to use Amazon Nova Lite as a starting point (referring to the US Nova Lite system-defined inference profile)
OPENAI_API_KEY - Comment out to ensure that LibreChat does not try to make any calls to OpenAI APIs even if the endpoint is disabled via ENDPOINTS.
ANTHROPIC_API_KEY - Comment out to ensure that LibreChat does not try to make any calls to Anthropic APIs even if the endpoint is disabled via ENDPOINTS.
GOOGLE_KEY - Comment out to ensure that LibreChat does not try to make any calls to Google APIs even if the endpoint is disabled via ENDPOINTS.
ASSISTANTS_API_KEY - Comment out to ensure that LibreChat does not try to make any calls to OpenAI Assistants APIs even if the endpoint is disabled via ENDPOINTS.

Note that we'll be using the ECS task IAM role to allow LibreChat to seamlessly call Bedrock APIs, so we don't need to set BEDROCK_AWS_ACCESS_KEY_ID nor BEDROCK_AWS_SECRET_ACCESS_KEY.

LibreChat YAML File (librechat.yaml)

Copy the librechat.example.yaml file to a local folder. For this minimal setup, we won't need to define custom endpoints or configure advanced settings. However, to prevent custom endpoints like Groq and Mistral AI from appearing in the UI, comment out the custom key in the endpoints block and set it to an empty array [].

With all environment variables set in the environment file or SSM Parameter Store, and the librechat.yaml file prepared, we're ready to tie everything together with Terraform.

Building the Terraform Configuration

Since this blog post is getting quite long, let's focus on the key design elements of the Terraform configuration. You can find the complete Terraform configuration and source code in the 1_ecs_basic directory in this GitHub repository. Here are the descriptions for each Terraform configuration file:

atlas.tf - Defines the MongoDB Atlas resources, as explained in the earlier section of this blog post.
vpc.tf - Defines the VPC infrastructure for the solution. Key design elements include:
- The VPC design follows a three-tier architecture across two AZs. Although the database subnets are currently not in use, they can be utilized for AWS cache and database services in the future.
- There is built-in support for either fck-nat (default) or NAT Gateways, depending on your preference.
s3.tf - Defines the S3 bucket that hosts the LibreChat files and the S3 object for the .env file. Key design elements include:
- A ready-to-use .env file is included in the librechat/v0.8.4 folder. You can edit this file if using the same version, or upload a new .env file when you upgrade LibreChat (be sure to change the librechat_version variable).
- This S3 bucket may be used in the future as the LibreChat file storage backend, hence the .env file is placed in the config subfolder for better separation.
ecs.tf - Defines all ECS and related resources to run LibreChat as an ECS service. Key design elements include:
- A ready-to-use librechat.yaml file is included in the librechat/v0.8.4 folder. You can edit this file if using the same version, or upload a new librechat.yaml file when you upgrade LibreChat (be sure to change the librechat_version variable).
- The LibreChat credentials are generated by first creating a random password using the random_password ephemeral resource, then defining an aws_ssm_parameter resource with the write-only value storing the hash of the password (SHA1 or SHA256). You can replace this logic if you prefer to manually store the generated credentials in SSM Parameter Store first.
- The IAM policy for the ECS task role, aws_iam_role_policy.ecs_task_librechat, contains permissions to invoke Bedrock models and manage marketplace subscription of third-party models.
- Service auto-scaling is defined for future use, but for now, it is scaled to 1 for cost control.
- The LibreChat container runs on TCP port 3080 by default.
- CloudWatch logs are streamed to the /ecs/librechat log group.
alb.tf - Defines the ALB resources as the public-facing endpoint for LibreChat. Key design elements include:
- A HTTPS listener is defined with HTTP redirect to ensure security. Consequently, you must import or create a TLS certificate in AWS Certificate Manager (ACM), then pass the certificate's ARN using the alb_certificate_arn variable.
- HTTPS also requires a custom host name, so you must pass the DNS name using the librechat_dns_name variable.
- The target group checks the ECS task health using LibreChat's /health endpoint.

Deploying the Solution

After cloning the GitHub repository, deploy the solution as follows:

From the root of the cloned GitHub repository, navigate to 1_ecs_basic.
Set the MONGODB_ATLAS_CLIENT_ID and MONGODB_ATLAS_CLIENT_SECRET to the Atlas service account credentials created in the earlier section.
Configure your AWS credentials using aws configure for IAM, or aws configure sso for IAM Identity Center. The profile name will be provided as a Terraform variable.
Copy terraform.tfvars.example as terraform.tfvars and update the variables to match your configuration.
Run terraform init and terraform apply.

It is advised to check the ECS service status in the AWS Management Console to ensure that the tasks run successfully. Failed tasks will cause a perpetual restart due to the required capacity of 1, and overlooking this may lead to unexpected cost consequences. Check the task logs in the CloudWatch log group /ecs/librechat for errors if needed.

Once the configuration is applied, create the CNAME record for the ALB DNS name. If you manage your domain/subdomain using a public hosted zone in Amazon Route 53, you can also create an alias record pointing to the ALB. Lastly, go to the custom host name for LibreChat and ensure that the login page loads successfully.

Creating a User and Validating LibreChat

Since self-registration is disabled, we need to use the create user script to create the first user. The easiest way is to use ECS Exec in the ECS console to run the script in the librechat container in the task configuration. Here's a screenshot showing where to find the Connect button to open an interactive session:

Once CloudShell opens and connects to the container's shell, run the following command to start the user creation wizard:

npm run create-user

Enter the user's information as prompted, and a user will be created in LibreChat's database. Here's an example of creating a user for John Doe:

Now you're ready to log in. Open your LibreChat application URL and log in using the credentials you just created. Upon successful login, accept LibreChat's terms of service. You should see the Nova 2 Lite model already selected at the top, since it's the only configured model.

Let's test the setup with a simple prompt:

Tell me a joke about AWS.

If LibreChat responds with a joke, you've successfully completed the setup! Here's the joke I received, which honestly suggests that the Nova model could use some additional training with a better comedy dataset...

Summary

Congratulations, you now have your own LibreChat instance in AWS! You're now ready to start exploring and expanding its capabilities. While this setup gives you a functional chat interface, there's much more you can do to enhance its features. Here are some examples:

Configure more Bedrock models to unlock diverse capabilities and customization, balancing functionality, performance, and cost
Enable web search to allow LibreChat to search the internet and retrieve relevant information, enhancing conversations
Build custom AI assistants using AI Agents and integrate with various built-in and MCP tools to elevate capabilities and user experience

As your LibreChat usage grows, it's imperative to align your architecture with the AWS Well-Architected Framework. Here are some examples to improve security and operational robustness:

Streamline authentication by integrating with an Identity Provider (IDP)
Enable caching, session storage, and horizontal scaling in LibreChat using Redis and compute auto-scaling
Scale file storage with an Amazon S3 storage backend

Given the vast possibilities, I will start a new blog series about LibreChat and how to best use AWS services and best practices to enable these capabilities. If you enjoyed this post, stay tuned for new content at the Avangards Blog. Thanks so much for reading, and I hope you have fun chatting with LibreChat!

AWS Control Tower Proactive Controls for Terraform: A Proof of Concept

Anthony Wat — Tue, 24 Feb 2026 05:44:05 +0000

Introduction

As a Terraform advocate and an AWS consultant who builds many landing zones, AWS Control Tower has always been one of my favorite AWS services. Beyond its common use cases, such as account provisioning with Account Factory Customization (AFC) and Account Factory for Terraform (AFT), I am always on the lookout for opportunities to bring the two technologies closer together.

During landing zone design workshops, when walking customers through Control Tower controls, I often found myself unable to recommend proactive controls because many organizations prefer using Terraform over CloudFormation for infrastructure as code (IaC). To fully leverage everything Control Tower has to offer, wouldn’t it be nice if proactive controls worked with other IaC tools, including Terraform?

Through research, I learned that proactive controls are implemented as CloudFormation Hooks and can target resources created via the Cloud Control API. Having worked with the Terraform AWS Cloud Control (CC) Provider, I began to wonder whether proactive controls could evaluate Terraform resources created through this provider. This question became the experiment that is the subject of this blog post.

Let’s start with a quick explanation of what proactive controls are.

What Are AWS Control Tower Proactive Controls?

Proactive controls are pre-built compliance rules that evaluate AWS resources before deployment via CloudFormation stack operations, preventing non-compliant resources from being created or updated. AWS Control Tower provides more than 200 controls covering a wide range of AWS services and compliance frameworks.

An example of a proactive control is [CT.EC2.PR.7] Require an Amazon EBS volume resource to be encrypted at rest when defined by means of the AWS::EC2::Instance BlockDeviceMappings property or AWS::EC2::Volume resource type. As the name suggests, this control prevents an EC2 instance from being created or updated if it specifies an unencrypted EBS volume.

For the full list of proactive controls, you can either view them on the Control Catalog page in the AWS Control Tower console or refer to the Proactive control section in the AWS Control Tower Control Reference Guide.

Testing Proactive Controls with Terraform (Unsuccessfully)

Since proactive controls are implemented using CloudFormation Hooks, I initially assumed they would evaluate all Hook targets, particularly resources supported by the Cloud Control API. Because the Terraform AWS Cloud Control (CC) Provider is implemented using the Cloud Control API (as opposed to the standard AWS API used by the original Terraform AWS Provider), I expected proactive controls to apply there as well.

Although it is still uncommon for organizations to fully adopt the Terraform AWS CC Provider, I wanted to determine whether proactive controls could be used with Terraform to enforce compliance.

As a quick test, I used the following Terraform configuration to create an EC2 instance with an unencrypted EBS volume using the Terraform AWS CC Provider, expecting it to fail:

variable "subnet_id" {
    type = string
}

data "aws_ami" "al2023" {
  most_recent = true
  owners      = ["amazon"]

  filter {
    name   = "name"
    values = ["al2023-ami-2023.*-x86_64"]
  }

  filter {
    name   = "virtualization-type"
    values = ["hvm"]
  }
}

# EC2 instance with unencrypted EBS volume
resource "awscc_ec2_instance" "this" {
  image_id      = data.aws_ami.al2023.id
  instance_type = "t3.micro"
  subnet_id     = var.subnet_id

  block_device_mappings = [
    {
      device_name = "/dev/xvda"
      ebs = {
        volume_size = 20
        volume_type = "gp3"
        encrypted   = false  # Explicitly unencrypted
        delete_on_termination = true
      }
    }
  ]

  tags = [
    {
      key   = "Name"
      value = "unencrypted-vol-test"
    }
  ]
}

However, terraform apply ran successfully and the EC2 instance was created:

$ terraform apply
data.aws_ami.al2023: Reading...
data.aws_ami.al2023: Read complete after 0s [id=ami-0f3caa1cf4417e51b]

Terraform used the selected providers to generate the following execution plan.       
Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  # awscc_ec2_instance.this will be created
  + resource "awscc_ec2_instance" "this" {
      + additional_info                      = (known after apply)
      + affinity                             = (known after apply)
      + availability_zone                    = (known after apply)
      + block_device_mappings                = [
          + {
              + device_name  = "/dev/xvda"
              + ebs          = {
                  + delete_on_termination = true
                  + encrypted             = false
                  + iops                  = (known after apply)
                  + kms_key_id            = (known after apply)
                  + snapshot_id           = (known after apply)
                  + volume_size           = 20
                  + volume_type           = "gp3"
                }
              + no_device    = (known after apply)
              + virtual_name = (known after apply)
            },
        ]
      + cpu_options                          = (known after apply)
      + credit_specification                 = (known after apply)
      + disable_api_termination              = (known after apply)
      + ebs_optimized                        = (known after apply)
      + elastic_gpu_specifications           = (known after apply)
      + elastic_inference_accelerators       = (known after apply)
      + enclave_options                      = (known after apply)
      + hibernation_options                  = (known after apply)
      + host_id                              = (known after apply)
      + host_resource_group_arn              = (known after apply)
      + iam_instance_profile                 = (known after apply)
      + id                                   = (known after apply)
      + image_id                             = "ami-0f3caa1cf4417e51b"
      + instance_id                          = (known after apply)
      + instance_initiated_shutdown_behavior = (known after apply)
      + instance_type                        = "t3.micro"
      + ipv_6_address_count                  = (known after apply)
      + ipv_6_addresses                      = (known after apply)
      + kernel_id                            = (known after apply)
      + key_name                             = (known after apply)
      + launch_template                      = (known after apply)
      + license_specifications               = (known after apply)
      + metadata_options                     = (known after apply)
      + monitoring                           = (known after apply)
      + network_interfaces                   = (known after apply)
      + placement_group_name                 = (known after apply)
      + private_dns_name                     = (known after apply)
      + private_dns_name_options             = (known after apply)
      + private_ip                           = (known after apply)
      + private_ip_address                   = (known after apply)
      + propagate_tags_to_volume_on_creation = (known after apply)
      + public_dns_name                      = (known after apply)
      + public_ip                            = (known after apply)
      + ramdisk_id                           = (known after apply)
      + security_group_ids                   = (known after apply)
      + security_groups                      = (known after apply)
      + source_dest_check                    = (known after apply)
      + ssm_associations                     = (known after apply)
      + state                                = (known after apply)
      + subnet_id                            = "subnet-0a0bb7e920672c803"
      + tags                                 = [
          + {
              + key   = "Name"
              + value = "unencrypted-vol-test"
            },
        ]
      + tenancy                              = (known after apply)
      + user_data                            = (known after apply)
      + volumes                              = (known after apply)
      + vpc_id                               = (known after apply)
    }

Plan: 1 to add, 0 to change, 0 to destroy.

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

awscc_ec2_instance.this: Creating...
awscc_ec2_instance.this: Still creating... [00m10s elapsed]
awscc_ec2_instance.this: Creation complete after 16s [id=i-0963bfcf44274c8d9]

Apply complete! Resources: 1 added, 0 changed, 0 destroyed.

$

So how did the EC2 instance manage to be created?

Investigating Why Proactive Controls Don’t Work with Terraform

To investigate the issue, I examined the Hook in the CloudFormation console. Looking at the Hook named AWS::ControlTower::Hook, I noticed that its Targets field was set to None.

This seemed odd, as I expected at least one target to be listed. Upon reviewing the Hook details, I observed that the Hook targets included only CloudFormation resources, not the Cloud Control API:

Assuming the Hook details reflect the actual configuration, this implies that proactive controls validate only CloudFormation resources, not resources provisioned through the Cloud Control API (and therefore not those created via the Terraform AWS CC Provider). To confirm this behavior, I opened an AWS Support case.

The AWS support engineer explained that proactive controls are implemented using a special Hook type called Controls (Managed Hooks), which supports only CloudFormation resources as targets. To extend proactive controls to other targets, each control must be re-implemented as a custom Hook.

Although I submitted a feature request to the AWS Control Tower team to expand proactive controls to additional targets, I decided to proceed with a workaround, even if it required additional effort.

Replicating Proactive Controls to Target the Cloud Control API with Lambda Hooks

The AWS support engineer initially suggested re-implementing each proactive control using Lambda Hooks. While this approach would require significant effort, I was provided with the following Python code used for the CT.EC2.PR.7 control:

import json
import logging

logger = logging.getLogger()
logger.setLevel(logging.INFO)

def lambda_handler(event, context):
    '''
    CloudFormation Hook Handler for EBS Encryption Validation
    Validates that EC2 instances have encrypted EBS volumes
    '''

    logger.info(f"Received full event: {json.dumps(event, indent=2)}")

    # Extract request details from Cloud Control API event structure
    request_data = event.get('requestData', {})
    resource_type = request_data.get('targetName')
    target_model = request_data.get('targetModel', {})
    resource_properties = target_model.get('resourceProperties', {}) 

    logger.info(f"Extracted - Type: {resource_type}, Properties keys: {list(resource_properties.keys())}")

    # Only validate EC2 instances
    if not resource_type or resource_type != 'AWS::EC2::Instance':
        logger.info(f"Skipping validation - resource type '{resource_type}' is not EC2 Instance")
        return {
            'hookStatus': 'SUCCESS',
            'message': f'Resource type {resource_type} not applicable for this hook'
        }

    # Validation logic
    validation_result = validate_ebs_encryption(resource_properties)

    if validation_result['compliant']:
        return {
            'hookStatus': 'SUCCESS',
            'message': 'EC2 instance has encrypted EBS volumes'
        }
    else:
        return {
            'hookStatus': 'FAILED',
            'errorCode': 'NonCompliant',
            'message': validation_result['message']
        }

def validate_ebs_encryption(properties):
    '''
    Validates that all EBS volumes are encrypted
    '''

    # Check BlockDeviceMappings
    block_device_mappings = properties.get('BlockDeviceMappings', [])

    if not block_device_mappings:
        return {
            'compliant': False,
            'message': 'No BlockDeviceMappings specified. Ensure the AMI uses encrypted volumes or specify encrypted BlockDeviceMappings.'
        }

    # Validate each block device mapping
    for idx, mapping in enumerate(block_device_mappings):
        ebs = mapping.get('Ebs', {})

        if ebs:
            encrypted = ebs.get('Encrypted', False)

            if not encrypted:
                return {
                    'compliant': False,
                    'message': f'BlockDeviceMapping at index {idx} has an unencrypted EBS volume. Set Encrypted to true.'
                }

    return {
        'compliant': True,
        'message': 'All EBS volumes are encrypted'
    }

After creating the Lambda function with the provided code, I created a Lambda Hook as per screenshots below. The key configuration details were:

Hook targets should include at least Cloud Control API . Since proactive controls already target CloudFormation resources, including them here is unnecessary.
Actions should include Create and Update .
Hook mode should be set to Fail .
Target resources should include AWS::EC2::Instance and AWS::EC2::Volume . These resource types are specified in the control details within the AWS::ControlTower::Hook configuration.

After the Lambda Hook is created, when I reran terraform apply, and this time it failed as expected due to the Hook:

awscc_ec2_instance.this: Creating...
╷
│ Error: AWS SDK Go Service Operation Incomplete
│
│   with awscc_ec2_instance.this,
│   on main.tf line 21, in resource "awscc_ec2_instance" "this":
│   21: resource "awscc_ec2_instance" "this" {
│
│ Waiting for Cloud Control API service CreateResource operation completion returned: 
│ waiter state transitioned to FAILED. StatusMessage:
│ 149a3eef-eaba-4459-8ea7-1707b1183e11. Hook failures: HookName:
│ Private::Lambda::CTEC2PR7, HookArn:
│ arn:aws:cloudformation:us-east-1:2**********1:type/hook/Private-Lambda-CTEC2PR7/00000001/aws-hooks/AWS-Hooks-LambdaHook/00000001.00000024,
│ HookVersion: 00000025, Time: 2026-02-23T21:06:45Z, HookMessage: BlockDeviceMapping  
│ at index 0 has an unencrypted EBS volume. Set Encrypted to true.
╵

Even Better: Replicating Proactive Controls with Guard Hooks

While researching further, I noticed that the rule specifications for all proactive controls are published under Proactive controls in the AWS Control Tower Controls Reference Guide. This makes replication significantly easier.

According to the documentation, proactive controls are implemented using Guard Hooks powered by AWS CloudFormation Guard, a domain-specific language (DSL) for policy-as-code. For more context and development instructions of Guard Hook, refer to Writing AWS CloudFormation Guard rules in the AWS CloudFormation Guard User Guide.

For our purposes, the CT.EC2.PR.7 control specification already contains everything needed to create the Hook. For instance, the rule specification is as follows:


# ###################################
##       Rule Specification        ##
#####################################
# 
# Rule Identifier:
#   ec2_encrypted_volumes_check
# 
# Description:
#   Checks whether standalone Amazon EC2 EBS volumes and new EC2 EBS volumes created through EC2 instance
#   Block Device Mappings are encrypted at rest.
# 
# Reports on:
#   AWS::EC2::Instance, AWS::EC2::Volume
# 
# Evaluates:
#   CloudFormation, CloudFormation hook
# 
# Rule Parameters:
#   None
# 
# Scenarios:
#   Scenario: 1
#     Given: The input document is an CloudFormation or CloudFormation hook document
#       And: The input document does not contain any Amazon EC2 volume resources
#      Then: SKIP
#   Scenario: 2
#     Given: The input document is an CloudFormation or CloudFormation hook document
#       And: The input document contains an EC2 instance resource
#       And: 'BlockDeviceMappings' has not been provided or has been provided as an empty list
#      Then: SKIP
#   Scenario: 3
#     Given: The input document is an CloudFormation or CloudFormation hook document
#       And: The input document contains an EC2 instance resource
#       And: 'BlockDeviceMappings' has been provided as a non-empty list
#       And: 'Ebs' has been provided in a 'BlockDeviceMappings' configuration
#       And: 'Encrypted' has not been provided in the 'Ebs' configuration
#      Then: FAIL
#   Scenario: 4
#     Given: The input document is an CloudFormation or CloudFormation hook document
#       And: The input document contains an EC2 instance resource
#       And: 'BlockDeviceMappings' has been provided as a non-empty list
#       And: 'Ebs' has been provided in a 'BlockDeviceMappings' configuration
#       And: 'Encrypted' has been provided in the 'Ebs' configuration and set to bool(false)
#      Then: FAIL
#   Scenario: 5
#     Given: The input document is an CloudFormation or CloudFormation hook document
#       And: The input document contains an EC2 volume resource
#       And: 'Encrypted' on the EC2 volume has not been provided
#      Then: FAIL
#   Scenario: 6
#     Given: The input document is an CloudFormation or CloudFormation hook document
#       And: The input document contains an EC2 volume resource
#       And: 'Encrypted' on the EC2 volume has been provided and is set to bool(false)
#      Then: FAIL
#   Scenario: 7
#     Given: The input document is an CloudFormation or CloudFormation hook document
#       And: The input document contains an EC2 instance resource
#       And: 'BlockDeviceMappings' has been provided as a non-empty list
#       And: 'Ebs' has been provided in a 'BlockDeviceMappings' configuration
#       And: 'Encrypted' has been provided in the 'Ebs' configuration and set to bool(true)
#      Then: PASS
#   Scenario: 8
#     Given: The input document is an CloudFormation or CloudFormation hook document
#       And: The input document contains an EC2 volume resource
#       And: 'Encrypted' on the EC2 volume has been provided and is set to bool(true)
#      Then: PASS

#
# Constants
#
let EC2_VOLUME_TYPE = "AWS::EC2::Volume"
let EC2_INSTANCE_TYPE = "AWS::EC2::Instance"
let INPUT_DOCUMENT = this

#
# Assignments
#
let ec2_volumes = Resources.*[ Type == %EC2_VOLUME_TYPE ]
let ec2_instances = Resources.*[ Type == %EC2_INSTANCE_TYPE ]

#
# Primary Rules
#
rule ec2_encrypted_volumes_check when is_cfn_template(%INPUT_DOCUMENT)
                                      %ec2_volumes not empty {
    check_volume(%ec2_volumes.Properties)
        <<
        [CT.EC2.PR.7]: Require that an Amazon EBS volume attached to an Amazon EC2 instance is encrypted at rest
        [FIX]: Set 'Encryption' to true on EC2 EBS Volumes.
        >>
}

rule ec2_encrypted_volumes_check when is_cfn_hook(%INPUT_DOCUMENT, %EC2_VOLUME_TYPE) {
    check_volume(%INPUT_DOCUMENT.%EC2_VOLUME_TYPE.resourceProperties)
        <<
        [CT.EC2.PR.7]: Require that an Amazon EBS volume attached to an Amazon EC2 instance is encrypted at rest
        [FIX]: Set 'Encryption' to true on EC2 EBS Volumes.
        >>
}

rule ec2_encrypted_volumes_check when is_cfn_template(%INPUT_DOCUMENT)
                                      %ec2_instances not empty {
    check_instance(%ec2_instances.Properties)
        <<
        [CT.EC2.PR.7]: Require that an Amazon EBS volume attached to an Amazon EC2 instance is encrypted at rest
        [FIX]: Set 'Encryption' to true on EC2 EBS Volumes.
        >>
}

rule ec2_encrypted_volumes_check when is_cfn_hook(%INPUT_DOCUMENT, %EC2_INSTANCE_TYPE) {
    check_instance(%INPUT_DOCUMENT.%EC2_INSTANCE_TYPE.resourceProperties)
        <<
        [CT.EC2.PR.7]: Require that an Amazon EBS volume attached to an Amazon EC2 instance is encrypted at rest
        [FIX]: Set 'Encryption' to true on EC2 EBS Volumes.
        >>
}

#
# Parameterized Rules
#

rule check_instance(ec2_instance) {
    %ec2_instance[
        filter_ec2_instance_block_device_mappings(this)
    ] {
        BlockDeviceMappings[
            Ebs exists
            Ebs is_struct
        ] {
            check_volume(Ebs)
        }
    }
}

rule check_volume(ec2_volume) {
    %ec2_volume {
        # Scenario 2
        Encrypted exists
        # Scenarios 3 and 4
        Encrypted == true
    }
}

rule filter_ec2_instance_block_device_mappings(ec2_instance) {
    %ec2_instance {
        BlockDeviceMappings exists
        BlockDeviceMappings is_list
        BlockDeviceMappings not empty
    }
}

#
# Utility Rules
#
rule is_cfn_template(doc) {
    %doc {
        AWSTemplateFormatVersion exists  or
        Resources exists
    }
}

rule is_cfn_hook(doc, RESOURCE_TYPE) {
    %doc.%RESOURCE_TYPE.resourceProperties exists
}

To implement this, I first deleted the previous Lambda Hook and its associated IAM role and policy to avoid redundant checks. Then following the instructions to activate a Guard Hook, I created an S3 bucket and uploaded the Guard rule as CT.EC2.PR.7.guard, and created the Guard Hook as per the screenshots below. The key configuration details were:

Hook targets should include at least Cloud Control API . Since proactive controls already target CloudFormation resources, including them here is unnecessary.
Actions should include Create and Update .
Hook mode should be set to Fail .
Target resources should include AWS::EC2::Instance and AWS::EC2::Volume . These resource types are specified in the control details within the AWS::ControlTower::Hook configuration.

After the Guard Hook is created, rerunning terraform apply resulted in a failure as expected:

awscc_ec2_instance.this: Creating...
╷
│ Error: AWS SDK Go Service Operation Incomplete
│
│   with awscc_ec2_instance.this,
│   on main.tf line 21, in resource "awscc_ec2_instance" "this":
│   21: resource "awscc_ec2_instance" "this" {
│
│ Waiting for Cloud Control API service CreateResource operation completion returned: 
│ waiter state transitioned to FAILED. StatusMessage:
│ 0708b661-d689-4451-b05f-28c8429f3836. Hook failures: HookName:
│ Private::Guard::CTEC2PR7, HookArn:
│ arn:aws:cloudformation:us-east-1:2**********1:type/hook/Private-Guard-CTEC2PR7/00000002/aws-hooks/AWS-Hooks-GuardHook/00000001.00000071,
│ HookVersion: 00000072, Time: 2026-02-24T02:28:27Z, HookMessage: Template failed     
│ validation, the following rule(s) failed: ec2_encrypted_volumes_check.
╵

This confirms that Guard Hooks provide a clean and scalable way to extend proactive controls to Terraform via the Cloud Control API.

Next Steps

Now that we have a viable approach for replicating proactive controls using Guard Hooks, the next logical step is automation at scale.

I submitted another feature request to the Control Tower team to publish proactive control Guard rules in a GitHub repository, similar to the AWS Guard Rules Registry. After cross-checking the rules in that repository against the proactive control documentation, I found they differ.

As a workaround, I could develop a scraper to extract rule definitions directly from the documentation and publish them into my own GitHub repository.

From there, I could identify an appropriate trigger, such as a CloudTrail event for updates to the AWS::ControlTower::Hook, to invoke a Lambda function that automatically manages Guard Hook replication based on enabled proactive controls.

This could make for an interesting project, and perhaps a future blog post, demonstrating the capabilities of AI coding assistants such as Kiro.

Summary

In this blog post, I explored whether AWS Control Tower proactive controls can apply to resources created with Terraform. After a failed attempt, I looked for a workaround, which ultimately took the form of replicating the CloudFormation Guard rules that power the proactive controls. Hopefully, AWS will eventually implement the feature request to extend proactive controls to cover the Cloud Control API as well. In the meantime, we now have a viable and potentially automatable approach to replicate them.

If you enjoyed this blog post and the topic it covers, be sure to check out the Avangards Blog for more content. Thanks for reading!

5 Practical Tips for the Terraform Authoring and Operations Professional Exam

Anthony Wat — Mon, 12 Jan 2026 06:45:23 +0000

Introduction

Over the past year, I have worked extensively with Terraform and AWS in building turnkey infrastructure solutions and contributing to the Terraform AWS Provider as a HashiCorp Core Contributor. As a formal validation of my Terraform expertise, I was motivated to take the Terraform Authoring and Operations Professional Exam. I recently passed the exam, and it was such a fun and unique experience that I decided to share my study tips with the community in this blog post to raise awareness and spark interest. Let’s first take a look at what this exam is all about.

About the Terraform Authoring and Operations Professional Exam

The Terraform Authoring and Operations Professional certification is not an entry-level exam. It’s intended for engineers who already have hands-on experience using Terraform in production and maintaining infrastructure over time. Passing the exam demonstrates strong skills in writing Terraform modules and managing Terraform operations in an organizational setting. The objectives of the exam include:

Manage resource lifecycle - Covers the end-to-end Terraform workflow for creating, updating, destroying, and managing infrastructure state using core CLI commands.
Develop and troubleshoot dynamic configuration - Focuses on writing flexible, reusable Terraform configuration using HCL features, functions, variables, and best practices for sensitive data.
Develop collaborative Terraform workflows - Addresses how Terraform is used in team and automated environments, including versioning, remote state, automation, and data sharing.
Create, maintain, and use Terraform modules - Examines how to design, consume, refactor, and version Terraform modules to enable reuse and maintainability.
Configure and use Terraform providers - Covers provider architecture, configuration, authentication, versioning, and troubleshooting provider-related issues.
Collaborate on infrastructure as code using HCP Terraform - Focuses on operating Terraform at scale with HCP Terraform, including runs, workspaces, credential management, and governance controls.

The exam is primarily lab-based, with scenarios in which you modify configuration and provision and manage infrastructure in a virtual exam environment. It also includes a small multiple-choice section that validates your knowledge of HCP Terraform and related topics. This is an online, proctored exam that is four hours in length, with an optional 15-minute break. It costs $295 USD plus tax; however, it does include a free retake.

If the content and format of this exam intrigue you enough to take the plunge, here are some practical tips that may help with your preparation.

Tip 1: Use the Official Prep Tutorial to Guide Your Study

The Terraform Professional certification exam prep guide was my go-to resource for studying. Both the learning path and the exam content list provide an exhaustive and structured set of topics to master, including recommended tutorials. Based on these resources, I created my own study plan and a list of things to test in my lab environment. Much of my study involved writing Terraform configuration, running commands, and observing how the system behaves.

While there aren’t many third-party study materials available for the exam, I did come across the Terraform Authoring and Operations Professional Study Guide, which was mentioned in this Reddit thread I found during my research. I read a free chapter of the book and found it to be quite well written, although I personally have enough Terraform and AWS experience that I could do without it.

Tip 2: Practical Experience with Terraform and AWS Is a Must

Unlike the HashiCorp Certified: Terraform Associate Exam and many cloud provider exams, passing the Terraform Authoring and Operations Professional Exam requires extensive hands-on experience with Terraform. Learning from basic tutorials alone will not help much, and it is futile to try studying and memorizing your way to a passing grade for this exam.

Although the exam details do not list AWS experience as a prerequisite, you must know how to deploy resources from core AWS services, such as Amazon EC2, Amazon VPC, and AWS IAM, to complete the lab-based scenarios efficiently. The list of AWS resources to review in the exam content outline is a good indicator of what you need to know and brush up on before the exam.

The required Terraform knowledge is also advanced, as you will need to employ dynamic configuration (think functions, modules, and meta-arguments) and state manipulation to complete the lab-based scenarios. Even in my professional services role with daily interaction with AWS and Terraform, I rarely had a need to work so extensively with states and advanced Terraform features. Fortunately, I explicitly practiced based on cues from Reddit and the study guide, which left me well prepared. Make sure you go through tutorials on these advanced topics and try them out in a sandbox environment.

Additionally, your Terraform CLI experience likely amounts to running terraform init, terraform plan and terraform apply even if you work with Terraform regularly. You should therefore review and experiment with the full list of CLI commands and their various options. I, for one, learned about an experimental option for a particular CLI command that proved helpful during the exam.

Tip 3: Know Your Way Around the Exam Environment

The exam is conducted within a Guacamole-powered Linux virtual desktop environment that is accessed via your web browser. The video walkthrough on the exam orientation page should give you an idea of what the exam environment looks like. The exam is primarily delivered through a local web page in the preinstalled Mozilla Firefox web browser. It provides the exam instructions and lab scenarios, a section for completing the multiple-choice questions, and links to whitelisted resources such as Terraform and provider documentation.

For the lab-based scenarios, the preinstalled Visual Studio Code is your main interface. The IDE has some useful extensions, such as the HashiCorp Terraform extension, preinstalled, giving you access to features like IntelliSense. However, the IDE may not be fully configured with all the quality-of-life features that you’d expect (such as format on save), so you will either have to change the settings yourself or bear with the minor inconveniences. One thing I found to be extremely helpful is the terminal in VS Code. Even though it is not mentioned in the exam instructions, opening the terminal in VS Code provides you with a prompt to navigate to the directory of each lab scenario. This made it more efficient for me to open files and run commands, without having to browse and open files from the desktop. If you are not already using VS Code for Terraform development, be sure to familiarize yourself with it and the Terraform extension before the exam.

One notable annoyance with the exam environment is that if you use your mouse side buttons to navigate between documentation pages in Firefox, the shortcut is instead recognized by your main web browser and leads you out of the exam session. You’d then have to ask the proctor to let you back in. I have also seen other exam takers reporting similar issues with keyboard shortcuts. It was difficult for me to break this habit during the exam, so I accidentally exited the exam environment several times, much to the proctor’s dismay, which he made clear in the chat. After the exam, I reported the issue and was contacted by someone from IBM, so I hope this can be fixed soon.

Otherwise, it wasn’t too difficult to adapt to the copy-and-paste shortcuts, and overall latency was acceptable.

ℹ I noticed that GitHub Copilot was installed and seemingly usable in VS Code. Although tempting, I did not use it during the exam, as it would likely violate the exam terms.

Tip 4: Learn to Navigate Terraform and AWS Documentation Efficiently

During the exam, you can access the Terraform documentation, the Terraform Registry, the AWS provider documentation, and some AWS documentation in Firefox. However, you will not be able to search, so you will need to rely on navigating the documentation.

You should know where to find CLI command references for available options, as well as configuration construct references such as functions and built-in resources. You can navigate to these topics from the main documentation page via Terraform CLI and Configuration Language in the left-hand menu, respectively:

As I am fairly proficient with Terraform and practiced before the exam, I only referred to the documentation a couple of times. Where it helped me most was in validating my answers to multiple-choice questions related to platform and enterprise features that I am not experienced with.

Navigating the AWS provider documentation should be straightforward, but you should know where to find information about provider configuration and in-scope resources for the exam. As I work with AWS almost on a daily basis, I didn’t need to refer to AWS documentation at all. Overall, you should familiarize yourself with navigating the documentation without relying on search.

Tip 5: Manage Your Time Effectively During the Exam

The Terraform Authoring and Operations Professional Exam is 4 hours in duration with an optional 15-minute break, making it the longest IT exam I’ve taken to date. That said, you will likely need the entire allotted time to complete the exam. The majority of your time will be spent on lab-based scenarios that are aligned with the exam objectives by theme. Be sure to review each lab scenario description thoroughly and complete them “to spec”. The code doesn’t have to be pretty - it just needs to work correctly.

It may also be wise to time-box each scenario, perhaps to 45 minutes, so that you at least have an opportunity to attempt every question. I recall getting stuck on a very specific provider-related issue in the second scenario and spending about an hour on it before deciding to park it and move on. I was fortunately able to complete the other scenarios fairly quickly, which afforded me about 30 minutes to figure out the earlier scenario and double-check my answers to the multiple-choice questions. I completed the exam right at the time limit with confidence and received a passing notification an hour or two later.

Summary

The Terraform Authoring and Operations Professional Exam is a challenging, hands-on exam that really tests how well you know Terraform in real-world scenarios. In this blog post, I shared what the exam is like and the study strategies that worked for me - from focusing on hands-on practice and advanced Terraform features to getting comfortable with the exam environment and managing your time effectively.

If you’re thinking about taking the exam, I hope these tips help you prepare with more confidence. And if you found this useful, feel free to check out my other blog posts in the Avangards Blog, where I share more lessons learned from working with Terraform, AWS, and infrastructure as code. Good luck and happy “Terraforming”!

Using Amazon Bedrock Knowledge Base Application Logs for Notifications

Anthony Wat — Wed, 02 Apr 2025 02:57:44 +0000

Introduction

In the earlier blog post Building a Data Ingestion Solution for Amazon Bedrock Knowledge Bases, we developed a data ingestion solution that includes job completion notifications with a status pull mechanism which wasn’t as efficient as it could be. Since then, we examined Knowledge Bases logging which publishes ingestion job log events to CloudWatch Logs, which opens up a new opportunity for a better design with a status push mechanism based on subscription filters. In this blog post, we will examine how to update the original solution with the new design.

Updated Design Overview

The overall design of the updated solution is depicted in the following diagram:

The updated solution works as follows:

Logging is configured for the Bedrock Knowledge Base to deliver logs to CloudWatch Logs. A subscription filter is created in the associated log group to filter ingestion job status change events that correspond to an end state and send log events to a Lambda function.
A Lambda function, triggered by an EventBridge schedule rule, periodically starts an ingestion (a.k.a. sync) job for each specified knowledge base and data source. Note that the SQS queue is removed as it is no longer necessary.
Another Lambda function serves as the destination for the subscription filter. For each event message that is extracted from the log events, the function uses the job ID information to get details about the ingestion job. A notification is sent to one of the two SNS topics depending on whether the job is successful or failed.

Updating the Components

As the SQS queue is not required, the only change to the Lambda function that starts the ingestion job is a minor cleanup. The updated Lambda function code is as follows:

import boto3
import json
from botocore.exceptions import ClientError

bedrock_agent = boto3.client('bedrock-agent')
ssm = boto3.client('ssm')


def lambda_handler(event, context):
    try:
        # Retrieve the JSON config from Parameter Store
        response = ssm.get_parameter(Name='/start-kb-ingestion-jobs/config-json')
        config_json = response['Parameter']['Value']
        config = json.loads(config_json)

        for record in config:
            knowledge_base_id = record.get('knowledge_base_id')
            for data_source_id in record.get('data_source_ids'):
                # Start the ingestion job
                print(f'Starting ingestion job for data source {data_source_id} of knowledge base {knowledge_base_id}')
                response = bedrock_agent.start_ingestion_job(
                    knowledgeBaseId=knowledge_base_id,
                    dataSourceId=data_source_id
                )
        return {
            'statusCode': 200,
            'body': 'Success'
        }
    except ClientError as e:
        return {
            'statusCode': 500,
            'body': f'Client error: {str(e)}'
        }
    except Exception as e:
        return {
            'statusCode': 500,
            'body': f'Unexpected error: {str(e)}'
        }

Meanwhile, updating the component that checks ingestion job statuses is slightly more complex. First, we need to update the check-kb-job-statuses Lambda function to be a subscription filter target. As described in the Log group-level subscription filters page of the CloudWatch Logs user guide, the log data received by the function is compressed, Base64-encoded, and batched. I was able to easily find this StackOverflow question which has the exact code we need in the first answer.

Next, we need to know what a relevant log event looks like. The examples of knowledge base logs in the AWS documentation provide the general format for an ingestion job event, however it is preferable to look at an actual log event. Here’s one that captures a job completion event for a successful job:

{
    "event_timestamp": 1740895462316,
    "event": {
        "ingestion_job_id": "W0V45LVZY6",
        "data_source_id": "ATUWOVZJOD",
        "ingestion_job_status": "COMPLETE",
        "knowledge_base_arn": "arn:aws:bedrock:us-east-1:<redacted>:knowledge-base/R1K1UIZKKQ",
        "resource_statistics": {
            "number_of_resources_updated": 366,
            "number_of_resources_ingested": 0,
            "number_of_resources_deleted": 0,
            "number_of_resources_with_metadata_updated": 0,
            "number_of_resources_failed": 15
        }
    },
    "event_version": "1.0",
    "event_type": "StartIngestionJob.StatusChanged",
    "level": "INFO"
}

The Lambda function must extract the following details from the log event:

The knowledge base ID, which can be extracted from the value of the event.knowledge_base_arn, specifically after the /.
The data source ID, which is the value of the event.data_source_id field.
The ingestion job ID, which is the value of the event.ingestion_job_id field.

Although we are able to extract all of the information required for notification, the log event does not contain the verbose content ingestion failures which we get from the response of the GetIngestionJob API action. Although this approach is slightly less efficient, we will still call the API for completeness. The resulting Lambda function should look like this:

import base64
import boto3
import gzip
import json
from botocore.exceptions import ClientError

bedrock_agent = boto3.client('bedrock-agent')
ssm = boto3.client('ssm')
sns = boto3.client('sns')


def get_ssm_parameter(name):
    response = ssm.get_parameter(Name=name, WithDecryption=True)
    return response['Parameter']['Value']


def get_ingestion_job(knowledge_base_id, data_source_id, ingestion_job_id):
    response = bedrock_agent.get_ingestion_job(
        knowledgeBaseId=knowledge_base_id,
        dataSourceId=data_source_id,
        ingestionJobId=ingestion_job_id
    )
    return response['ingestionJob']


def lambda_handler(event, context):
    try:
        success_sns_topic_arn = get_ssm_parameter('/check-kb-ingestion-job-statuses/success-sns-topic-arn')
        failure_sns_topic_arn = get_ssm_parameter('/check-kb-ingestion-job-statuses/failure-sns-topic-arn')

        encoded_zipped_data = event['awslogs']['data']
        zipped_data = base64.b64decode(encoded_zipped_data)
        data = json.loads(gzip.decompress(zipped_data))
        log_events = data['logEvents']
        for log_event in log_events:
            message = json.loads(log_event['message'])
            knowledge_base_arn = message['event']['knowledge_base_arn']
            knowledge_base_id = knowledge_base_arn.split('/')[-1]
            data_source_id = message['event']['data_source_id']
            ingestion_job_id = message['event']['ingestion_job_id']

            print(
                f'Checking ingestion job status for knowledge base {knowledge_base_id} data source {data_source_id} job {ingestion_job_id}')
            ingestion_job = get_ingestion_job(knowledge_base_id, data_source_id, ingestion_job_id)
            print(
                f'Ingestion job summary: \n\n{json.dumps(ingestion_job, indent=2, sort_keys=True, default=str)}')
            job_status = ingestion_job['status']
            if job_status == 'COMPLETE':
                sns.publish(
                    TopicArn=success_sns_topic_arn,
                    Subject=f'Ingestion job for knowledge base {knowledge_base_id} data source {data_source_id} job {ingestion_job_id} Completed',
                    Message=json.dumps(ingestion_job, indent=2, sort_keys=True, default=str)
                )
            elif job_status == 'FAILED':
                sns.publish(
                    TopicArn=failure_sns_topic_arn,
                    Subject=f'Ingestion job for knowledge base {knowledge_base_id} data source {data_source_id} job {ingestion_job_id} FAILED',
                    Message=json.dumps(ingestion_job, indent=2, sort_keys=True, default=str)
                )
            elif job_status == 'STOPPED':
                sns.publish(
                    TopicArn=failure_sns_topic_arn,
                    Subject=f'Ingestion job for knowledge base {knowledge_base_id} data source {data_source_id} job {ingestion_job_id} STOPPED',
                    Message=json.dumps(ingestion_job, indent=2, sort_keys=True, default=str)
                )
        return {
            'statusCode': 200,
            'body': 'Success'
        }
    except ClientError as e:
        return {
            'statusCode': 500,
            'body': f'Client error: {str(e)}'
        }
    except Exception as e:
        return {
            'statusCode': 500,
            'body': f'Unexpected error: {str(e)}'
        }

Lastly, we need to create a subscription filter in the log group that acts as the log delivery destination of the knowledge base. Since we are only interested in log events for ingestion job completion, we need to define an appropriate subscription filter pattern. There are two fields which we need for this purpose:

The event_type field with the value StartIngestionJob.StatusChanged.
The event.ingestion_job_status field with the value matching one of COMPLETE, FAILED, CRAWLING_COMPLETED, as described in the data ingestion job log example.

Based on some testing, a CRAWLING_COMPLETED event does not indicate a full completion of an ingestion job. A COMPLETE (and presumably FAILED) event is always sent upon job completion. So we can use COMPLETE and FAILED for the filter. Furthermore, stopping a job does not generate an event and there is no status value for it. This seems like a miss on AWS’ part, so I’ll open an AWS support case for it. For now, we will still add STOPPED to the filter for the sake of completeness.

Referring to subscription filter pattern for JSON log events, we can define our compound expression that checks for the event type and ingestion job status as follows:

{$.event_type = "StartIngestionJob.StatusChanged" && ($.event.ingestion_job_status = "COMPLETE" || $.event.ingestion_job_status = "FAILED" || $.event.ingestion_job_status = "STOPPED")}

We can first test the pattern in the AWS Management Console's subscription filter creation dialog without creating the filter. Later, we will implement it using Terraform. Here is a screenshot of what the dialog looks like:

In this example, the subscription filter is created in the log group as evident by the standard naming pattern. Knowledge base logs are written to the log stream bedrock/knowledgebaselogs, so we need to select that. Using the Test pattern button, we can see one filtered entry in the test results among 50 log events. The log events were generated from a single ingestion job, and the other events are either resource change events or non-related status change events.

Updating the Terraform Configuration

The following changes are required to the original solution’s Terraform configuration to support the new design:

Remove the SQS queue, the associated IAM permissions, and the SSM parameters.
Update the Lambda permission for the check-kb-intgestion-job-statuses to allow trigger from CloudWatch logs via the log group to which the Bedrock knowledge base writes its application logs.

Lastly, we need a new resource for the subscription filter as follows:

resource "aws_cloudwatch_log_subscription_filter" "check_kb_ingestion_job_statuses" {
  name            = "check-kb-ingestion-job-statuses"
  log_group_name  = var.kb_app_log_group_name
  filter_pattern  = "{$.event_type = \"StartIngestionJob.StatusChanged\" && ($.event.ingestion_job_status = \"COMPLETE\" || $.event.ingestion_job_status = \"FAILED\" || $.event.ingestion_job_status = \"STOPPED\")}"
  destination_arn = aws_lambda_function.check_kb_ingestion_job_statuses.arn
  depends_on      = [aws_lambda_permission.check_kb_ingestion_job_statuses]
}

Note that the log group name is provided as a variable. The log group name should follow the default format provided by AWS, which is /aws/vendedlogs/bedrock/knowledge-base/APPLICATION_LOGS/<KB_ID>, where <KB_ID> is the Bedrock knowledge base ID.

Deploying and Testing the Solution

✅ You can find the complete Terraform configuration and source code in the 5_kb_data_ingestion_via_logs directory in this GitHub repository.

To deploy and test the solution, you need a knowledge base with at least one data source that has content to ingest either in an S3 bucket or a crawlable website. You can set this up in the Bedrock console using the vector database quick start options. Alternatively, deploy a sample knowledge base using the Terraform configuration from my blog post How To Manage an Amazon Bedrock Knowledge Base Using Terraform. This configuration is also available in the same GitHub repository under the 2_knowledge_base directory.

Additionally, you must also change the knowledge base’s logging configuration to deliver application logs to CloudWatch Logs. You can enable it either manually following the AWS documentation or using the Terraform configuration from my previous blog post Enabling Logging for Amazon Bedrock Knowledge Bases using Terraform. This configuration is also available in the same GitHub repository under the 4_kb_logging directory.

With the prerequisites in place, deploy the solution as follows:

From the root of the cloned GitHub repository, navigate to 5_kb_data_ingestion_via_logs.
Copy terraform.tfvars.example as terraform.tfvars and update the variables to match your configuration.

* By default, the `start-kb-ingestion-jobs` Lambda function runs daily at 0:00 UTC.

Configure your AWS credentials.
Run terraform init and terraform apply -var-file terraform.tfvars.

Once deployed, test the solution by adding an email subscription to the SNS topics check-kb-ingestion-job-statuses-success and check-kb-ingestion-job-statuses-failure for your e-mail address so that you can receive email notifications. Confirm your subscriptions using the link in the verification emails.

Next, manually invoke the start-kb-ingestion-jobs Lambda function in the Lambda console.

As the ingestion jobs run and complete, logs are written to CloudWatch Logs and pass through the subscription filter. The status change events should be filtered and sent to the Lambda function for notification, ultimately leading to the emails you’ll receive. Here’s an example:

Once you've verified the solution works, remove the SNS subscription and replace it with those that better fit your needs. If you don’t plan to keep the knowledge base, delete it along with the vector store (for example, the OSS index) to avoid unnecessary costs.

Summary

In this blog post, we improved the original Bedrock Knowledge Bases data ingestion solution with push-based notification using CloudWatch features. This is likely more efficient than a scheduled pull-based mechanism and allows us to leverage a Lambda subscription filter.

That being said, the ideal solution would be an EventBridge rule to react to native ingestion job events from Bedrock. The Bedrock service unfortunately does not publish such events today, but I’ve made a feature request via an AWS support case. Hopefully this will be supported soon and we can evolve our data ingestion solution further.

I hope you find this blog post helpful and engaging. Please feel free to check out my other blog posts in the Avangards Blog. Take care and happy learning!

Enabling Logging for Amazon Bedrock Knowledge Bases using Terraform

Anthony Wat — Sun, 02 Mar 2025 20:17:39 +0000

Introduction

In the recent blog post Building a Data Ingestion Solution for Amazon Bedrock Knowledge Bases, we created a data ingestion solution that includes job completion notifications with a status pull mechanism. Not satisfied with how frequently the Lambda function must run to check job statuses, I looked into whether a push mechanism is available.

From my research, I found that Bedrock Knowledge Bases supports observability logs and that it logs events related to content ingestion. With support for log delivery to CloudWatch Logs, it unlocks the possibility of using a subscription filter to push ingestion job completion log events. Consequently, I dedicated this blog post to reviewing this feature and determining how to enable it efficiently using Terraform.

With this context, let’s first look at how CloudWatch log delivery works in general and how it applies to Bedrock Knowledge Bases.

Creating a Delivery Source for the Knowledge Base

Bedrock Knowledge Bases is one of the AWS services that uses the log delivery feature in CloudWatch Logs to write vended logs. This is a framework that provides a standard interface to configure logging, which typically involves a delivery source, a delivery destination, and a delivery that enables logging by linking the two.

As per Monitor knowledge bases using CloudWatch Logs, Bedrock Knowledge Bases currently only support application logs. Thus, we can create the delivery source in Terraform using the aws_cloudwatch_log_delivery_source resource as follows:

resource "aws_cloudwatch_log_delivery_source" "kb_logs" {
  count        = var.enable_kb_log_delivery_cloudwatch_logs || var.enable_kb_log_delivery_s3 || var.enable_kb_log_delivery_data_firehose ? 1 : 0
  name         = "bedrock-kb-${var.kb_id}"
  log_type     = "APPLICATION_LOGS"
  resource_arn = "arn:${local.partition}:bedrock:${local.region}:${local.account_id}:knowledge-base/${var.kb_id}"
}

Notice that there is a condition to create the resource only if one of the log delivery options is enabled using variables, which will also be used in the destination-specific configurations that are explained in subsequent sections. This makes the configuration more generic and pluggable to your Terraform configuration that manages your Bedrock Agents and Knowledge Bases.

Sending Logs to CloudWatch Logs

To send logs to CloudWatch Logs, we need to create a log group and configure it as destination for delivery. Creating a log group is simple enough using the aws_cloudwatch_log_group resource. The log group name should follow the default format provided by AWS, which is /aws/vendedlogs/bedrock/knowledge-base/APPLICATION_LOGS/<KB_ID>, where <KB_ID> is the Bedrock knowledge base ID.

Using the log group for log delivery requires a log group resource policy. As per the AWS documentation, CloudWatch Logs can automatically add the appropriate policy if the log group does not have a resource policy, and the user setting up the logging has the appropriate permissions. Although, for the sake of completeness, we should manually create the resource policy as described in the aforementioned documentation using the aws_cloudwatch_log_resource_policy resource.

Lastly, we need to create a delivery destination for it using the aws_cloudwatch_log_delivery_destination resource, and then establish the delivery from the source (i.e., the knowledge base) to the destination (i.e., the log group) using the aws_cloudwatch_log_delivery resource. The resulting Terraform configuration should look like the following:

resource "aws_cloudwatch_log_group" "kb_logs" {
  count = var.enable_kb_log_delivery_cloudwatch_logs ? 1 : 0
  name  = "/aws/vendedlogs/bedrock/knowledge-base/APPLICATION_LOGS/${var.kb_id}"
}

resource "aws_cloudwatch_log_resource_policy" "kb_logs" {
  count       = var.enable_kb_log_delivery_cloudwatch_logs ? 1 : 0
  policy_name = "bedrock-kb-${var.kb_id}-policy"
  policy_document = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Sid    = "AWSLogDeliveryWrite20150319"
        Effect = "Allow"
        Principal = {
          Service = ["delivery.logs.amazonaws.com"]
        }
        Action = [
          "logs:CreateLogStream",
          "logs:PutLogEvents"
        ]
        Resource = ["${aws_cloudwatch_log_group.kb_logs[0].arn}:log-stream:*"]
        Condition = {
          StringEquals = {
            "aws:SourceAccount" = ["${local.account_id}"]
          },
          ArnLike = {
            "aws:SourceArn" = ["arn:${local.partition}:logs:${local.region}:${local.account_id}:*"]
          }
        }
      }
    ]
  })
}

resource "aws_cloudwatch_log_delivery_destination" "kb_logs_cloudwatch_logs" {
  count = var.enable_kb_log_delivery_cloudwatch_logs ? 1 : 0
  name  = "bedrock-kb-${var.kb_id}-cloudwatch-logs"
  delivery_destination_configuration {
    destination_resource_arn = aws_cloudwatch_log_group.kb_logs[0].arn
  }
  depends_on = [aws_cloudwatch_log_resource_policy.kb_logs]
}

resource "aws_cloudwatch_log_delivery" "kb_logs_cloudwatch_logs" {
  count                    = var.enable_kb_log_delivery_cloudwatch_logs ? 1 : 0
  delivery_destination_arn = aws_cloudwatch_log_delivery_destination.kb_logs_cloudwatch_logs[0].arn
  delivery_source_name     = aws_cloudwatch_log_delivery_source.kb_logs[0].name
}

Sending Logs to S3

The process to enable S3 as a delivery destination follows a similar pattern. The first step is to create the S3 bucket using the aws_s3_bucket resource with a bucket policy that provides the appropriate permissions for log delivery as described in the AWS documentation. Note that if you are using SSE-KMS for server-side encryption, you’ll also add the appropriate permissions to the key policy for the CMK. For completeness, we also choose not to rely on CloudWatch Logs to set the bucket policy and instead use the aws_s3_bucket_policy resource to manage it.

We also need to create a delivery destination for it using the aws_cloudwatch_log_delivery_destination resource, then establish the delivery from the source (i.e. the knowledge base) and the destination (i.e. the S3 bucket) using the aws_cloudwatch_log_delivery resource. Note that updating multiple aws_cloudwatch_logs_delivery resources in parallel will cause concurrency issues, so we must ensure that they are created sequentially using the depends_on meta-argument. In this case, the delivery resource for S3 depends on that of CloudWatch Logs.

The resulting Terraform configuration should look like the following:

resource "aws_s3_bucket" "kb_logs_s3" {
  count         = var.enable_kb_log_delivery_s3 ? 1 : 0
  bucket        = "bedrock-kb-logs-${lower(var.kb_id)}-${local.region_short}-${local.account_id}"
  force_destroy = true
}

resource "aws_s3_bucket_policy" "kb_logs_s3" {
  count  = var.enable_kb_log_delivery_s3 ? 1 : 0
  bucket = aws_s3_bucket.kb_logs_s3[0].id
  policy = jsonencode({
    Version = "2012-10-17"
    Id      = "AWSLogDeliveryWrite20150319"
    "Statement" : [
      {
        Sid    = "AWSLogDeliveryWrite171157658"
        Effect = "Allow"
        Principal = {
          Service = "delivery.logs.amazonaws.com"
        }
        Action   = "s3:PutObject"
        Resource = "${aws_s3_bucket.kb_logs_s3[0].arn}/AWSLogs/${local.account_id}/bedrock/knowledgebases/*"
        Condition = {
          StringEquals = {
            "aws:SourceAccount" = "${local.account_id}"
            "s3:x-amz-acl"      = "bucket-owner-full-control"
          }
          ArnLike = {
            "aws:SourceArn" = "${aws_cloudwatch_log_delivery_source.kb_logs[0].arn}"
          }
        }
      }
    ]
  })
}

resource "aws_cloudwatch_log_delivery_destination" "kb_logs_s3" {
  count = var.enable_kb_log_delivery_s3 ? 1 : 0
  name  = "bedrock-kb-${var.kb_id}-s3"
  delivery_destination_configuration {
    destination_resource_arn = aws_s3_bucket.kb_logs_s3[0].arn
  }
  depends_on = [aws_s3_bucket_policy.kb_logs_s3[0]]
}

resource "aws_cloudwatch_log_delivery" "kb_logs_s3" {
  count                    = var.enable_kb_log_delivery_s3 ? 1 : 0
  delivery_destination_arn = aws_cloudwatch_log_delivery_destination.kb_logs_s3[0].arn
  delivery_source_name     = aws_cloudwatch_log_delivery_source.kb_logs[0].name
  depends_on               = [aws_cloudwatch_log_delivery.kb_logs_cloudwatch_logs]
}

Sending Logs to Data Firehose

Sending logs to Data Firehose is slightly more involved because of the Firehose delivery stream’s configuration. Since this blog post does not focus on the downstream destination at the Firehose level, we will use an S3 bucket with basic configuration. To set up a Firehose delivery stream, we first need to create an IAM role that the delivery stream uses to send data to its destination (that is, the S3 bucket). Controlling access with Amazon Data Firehose provides IAM policy examples for different configuration, including one for an S3 destination. To create the Firehose delivery stream, we use the aws_kinesis_firehose_delivery_stream resource. One thing to know is that the Firehose delivery stream needs to have the tag LogDeliveryEnabled set to true, which the service-linked role that CloudWatch Logs creates uses to write to Firehose delivery streams.

We also need to create a delivery destination for it using the aws_cloudwatch_log_delivery_destination resource, then establish the delivery from the source (i.e. the knowledge base) and the destination (i.e. the S3 bucket) using the aws_cloudwatch_log_delivery resource. To ensure that the delivery resources are created sequentially so to avoid concurrent modification issues, this delivery resource depends on that of S3.

The resulting Terraform configuration should look like the following:

resource "aws_s3_bucket" "kb_logs_data_firehose" {
  count         = var.enable_kb_log_delivery_data_firehose ? 1 : 0
  bucket        = "bedrock-kb-logs-data-firehose-${lower(var.kb_id)}-${local.region_short}-${local.account_id}"
  force_destroy = true
}

resource "aws_iam_role" "kb_logs_data_firehose" {
  count = var.enable_kb_log_delivery_data_firehose ? 1 : 0
  name  = "S3RoleForDataFirehose-bedrock-kb-logs-${var.kb_id}"
  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Action = "sts:AssumeRole"
        Effect = "Allow"
        Principal = {
          Service = "firehose.amazonaws.com"
        }
        Condition = {
          StringEquals = {
            "sts:ExternalId" = "${local.account_id}"
          }
        }
      }
    ]
  })
}

resource "aws_iam_role_policy" "kb_logs_data_firehose" {
  count = var.enable_kb_log_delivery_data_firehose ? 1 : 0
  name  = "S3PolicyForDataFirehose-bedrock-kb-logs-${var.kb_id}"
  role  = aws_iam_role.kb_logs_data_firehose[0].name
  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Action = [
          "s3:AbortMultipartUpload",
          "s3:GetBucketLocation",
          "s3:GetObject",
          "s3:ListBucket",
          "s3:ListBucketMultipartUploads",
          "s3:PutObject"
        ]
        Effect = "Allow"
        Resource = [
          aws_s3_bucket.kb_logs_data_firehose[0].arn,
          "${aws_s3_bucket.kb_logs_data_firehose[0].arn}/*"
        ]
      }
    ]
  })
}

resource "aws_kinesis_firehose_delivery_stream" "kb_logs" {
  count       = var.enable_kb_log_delivery_data_firehose ? 1 : 0
  name        = "bedrock-kb-logs-${var.kb_id}"
  destination = "extended_s3"
  extended_s3_configuration {
    role_arn   = aws_iam_role.kb_logs_data_firehose[0].arn
    bucket_arn = aws_s3_bucket.kb_logs_data_firehose[0].arn
  }
  tags = {
    "LogDeliveryEnabled" = "true"
  }
  depends_on = [aws_iam_role_policy.kb_logs_data_firehose]
}

resource "aws_cloudwatch_log_delivery_destination" "kb_logs_data_firehose" {
  count = var.enable_kb_log_delivery_data_firehose ? 1 : 0
  name  = "bedrock-kb-${var.kb_id}-data-firehose"
  delivery_destination_configuration {
    destination_resource_arn = aws_kinesis_firehose_delivery_stream.kb_logs[0].arn
  }
}

resource "aws_cloudwatch_log_delivery" "kb_logs_data_firehose" {
  count                    = var.enable_kb_log_delivery_data_firehose ? 1 : 0
  delivery_destination_arn = aws_cloudwatch_log_delivery_destination.kb_logs_data_firehose[0].arn
  delivery_source_name     = aws_cloudwatch_log_delivery_source.kb_logs[0].name
  depends_on               = [aws_cloudwatch_log_delivery.kb_logs_s3]
}

Testing the Configuration

✅You can find the complete Terraform configuration and source code in the 4_kb_logging directory in this GitHub repository.

To deploy and test the configuration, you need a knowledge base with at least one data source that has content to ingest either in an S3 bucket or a crawlable website. You can set this up in the Bedrock console using the vector database quick start options. Alternatively, deploy a sample knowledge base using the Terraform configuration from my blog post How To Manage an Amazon Bedrock Knowledge Base Using Terraform. This configuration is also available in the same GitHub repository under the 2_knowledge_base directory.

With the prerequisites in place, deploy the solution as follows:

From the root of the cloned GitHub repository, navigate to 4_kb_logging.
Copy terraform.tfvars.example as terraform.tfvars and update the variables to match your configuration.

* All log delivery destinations are enabled is enabled in `terraform.tfvars.example`. However, only delivery to CloudWatch Logs is enabled by default in the variable definition.

Configure your AWS credentials.
Run terraform init and terraform apply -var-file terraform.tfvars.

Once the configuration is applied, you can open the target knowledge base in the Amazon Bedrock Console and click Edit in the Knowledge Base overview section to review the logging configuration:

Assuming all three log destinations are enabled, it should look something like:

⚠ While working on this blog post, I encountered an issue where the log deliveries section does not load and shows a spinner indefinitely if multi-session support is enabled in the AWS Management Console. Disabling the feature will work around the problem. I have opened an AWS support case for this issue, which I hope will be fixed soon.

For good measure, we can perform a task with the knowledge base that generates application logs and see if logs are being delivered. At the time of writing, Bedrock Knowledge Bases only generate logs from ingestion job events. As such, we can trigger a sync of a data source in the knowledge base. The log group should have logs similar to the following:

Next, the S3 bucket should have logs similar to the following:

Lastly, the destination of the Firehose delivery stream, which in our case is another S3 bucket, should have logs similar to the following:

If you don’t need the resources after testing, be sure to delete them to avoid unexpected costs.

Summary

In this blog post, we examined how logging works for Amazon Bedrock Knowledge Bases, which uses the log delivery feature in CloudWatch Logs. We created and tested Terraform configuration that demonstrates knowledge base log delivery to all three supported destinations - CloudWatch Logs, S3, and Data Firehose. You can also repurpose the Terraform configuration for other AWS services that use the log delivery mechanism with minimal changes, should you have a need.

At this point, we have the know-how to write ingestion logs to CloudWatch Logs, so we can update the data ingestion solution I previously wrote about to improve how ingestion job notifications are triggered. Please stay tuned for my next blog post on this topic. Thanks for reading, as always, and be sure to check out the Avangards Blog for more AWS and Terraform content.

Building a Data Ingestion Solution for Amazon Bedrock Knowledge Bases

Anthony Wat — Thu, 20 Feb 2025 06:14:03 +0000

Introduction

2025 started off busy, and only recently have I had the chance to catch up on all the new Amazon Bedrock features that were launched during re:Invent 2024. These new capabilities make it easier than ever to build a comprehensive RAG solution using a low-code approach. As I explored these new features, I realized that most, if not all, are functional features. I feel that there isn’t a lot of guidance on the operational aspects of Bedrock services, so I decided to write more about this topic.

A key component of any RAG system is the data ingestion pipeline. Amazon Bedrock Knowledge Bases has built-in data ingestion that does the heavy lifting, and synchronizations can be triggered on demand. From an operational perspective, the end-to-end process should ideally be automated and aligned with either the update cadence of the data source or a designated maintenance window. This is an ideal use case for a Lambda-based automation solution, for which I have built a basic version. In this blog post, we’ll walk through its design and implementation.

Design Overview

The overall design of the solution is depicted in the following diagram:

The solution works as follows:

A Lambda function, triggered by an EventBridge schedule rule, periodically starts an ingestion (a.k.a. sync) job for each specified knowledge base and data source. The function also sends a message with job ID information to an SQS queue.
Another Lambda function, triggered by an EventBridge schedule rule, frequently fetches messages from the SQS queue. For each message, the function uses the job ID information to get details about the ingestion job. The message is removed from the queue if the job has completed, and a notification is sent to one of the two SNS topics depending on whether the job is successful or failed.
(Optional) Additional downstream tasks can be performed by subscribing to the SNS topics.

With the high-level architecture in mind, let’s now dive into the detailed design of each major component.

Component Design: Starting Ingestion Jobs

Amazon Bedrock Knowledge Bases simplifies ingestion for data sources it natively supports such as Amazon S3 and Web Crawler. Its default parsing logic covers most cases, while its default chunking logic allows the selection of different strategies that could improve data retrieval quality.

For Amazon S3, while it is possible to start ingesting data as the S3 bucket is updated (using S3 event notifications for instance), it is not recommended when the knowledge base is in use especially during high usage periods. For Web Crawler, detecting website updates - especially one you don't own - is often difficult. Instead, a more reliable approach is to schedule ingestion during a maintenance window (for example, after midnight). This, as we know, can easily be implemented using an EventBridge rule that runs on a schedule.

When using default settings, you simply need to synchronize the data source when the data source is updated. This is done programmatically via the StartIngestionJob action in the Agents for Amazon Bedrock API. This action is available as the start_ingestion_job method in Boto3, which is used in our Python-based Lambda function.

Since ingestion is asynchronous, you must check job statuses separately. In our event-based setup, each job’s details (knowledge base ID, data source ID, ingestion job ID) is passed to the other Lambda function that checks ingestion job statuses. To facilitate communication of the decoupled components, we can use an SNS topic, SQS queue, or DynamoDB table. Since long-running jobs require multiple status checks, an SQS standard queue seems like a better fit.

Lastly, we need to configure which knowledge bases, data sources, and SQS queues the Lambda function should manage. AWS Systems Manager Parameter Store is an obvious choice. To structure the knowledge base and data source information, we’ll use a JSON list that contains an object with the knowledge base ID and the list of data source IDs, for example:

[
  {
    "knowledge_base_id" = "YO4R9AYHQZ",
    "data_source_ids"   = ["5IHZ5YAIBY"]
  }
]

Now that we've outlined the design, let's look at the implementation of the Lambda function:

import boto3
import json
from botocore.exceptions import ClientError

bedrock_agent = boto3.client('bedrock-agent')
sqs = boto3.client('sqs')
ssm = boto3.client('ssm')


def lambda_handler(event, context):
    try:
        # Retrieve the JSON config from Parameter Store
        response = ssm.get_parameter(Name='/start-kb-ingestion-jobs/config-json')
        config_json = response['Parameter']['Value']
        config = json.loads(config_json)

        for record in config:
            knowledge_base_id = record.get('knowledge_base_id')
            for data_source_id in record.get('data_source_ids'):
                # Start the ingestion job
                print(f'Starting ingestion job for data source {data_source_id} of knowledge base {knowledge_base_id}')
                response = bedrock_agent.start_ingestion_job(
                    knowledgeBaseId=knowledge_base_id,
                    dataSourceId=data_source_id
                )
                ingestion_job_id = response['ingestionJob']['ingestionJobId']

                # Send a message to the SQS queue
                response = ssm.get_parameter(Name='/start-kb-ingestion-jobs/sqs-queue-url')
                sqs_queue_url = response['Parameter']['Value']
                message = {
                    'knowledge_base_id': knowledge_base_id,
                    'data_source_id': data_source_id,
                    'ingestion_job_id': ingestion_job_id
                }
                sqs.send_message(
                    QueueUrl=sqs_queue_url,
                    MessageBody=json.dumps(message)
                )
        return {
            'statusCode': 200,
            'body': 'Success'
        }
    except ClientError as e:
        return {
            'statusCode': 500,
            'body': f'Client error: {str(e)}'
        }
    except Exception as e:
        return {
            'statusCode': 500,
            'body': f'Unexpected error: {str(e)}'
        }

Component Design: Checking Ingestion Job Statuses

The second piece of the puzzle is checking the status of any ingestion jobs initiated by the solution and sending notifications when a job completes, fails, or is canceled. Notifications should also include any warnings to highlight issues ingesting specific documents that could impact the quality of the RAG solution.

To retrieve ingestion job details, the GetIngestionJob action in the Agents for Amazon Bedrock API can be used. The Boto3 equivalent would be the get_ingestion_job method. The required parameters are included in each SQS message sent by the Lambda function that starts ingestion jobs as described earlier. The approach is to poll and process SQS messages on a schedule (for example, every five minutes).

SQS polling can be tricky, especially if you’re unfamiliar with it. Short and long polling determine how long a ReceiveMessage API request (or its Boto3 equivalent, the receive_message method) waits for at least one message to show up in the queue. Here, long polling is not ideal since the best practice is to minimize Lambda function runtime when idle. It is also unnecessary given the Lambda function is run on a schedule anyway. Additionally, SQS’s distributed nature affects how many messages the ReceiveMessage API returns. Even with a higher MaxNumberOfMessages number, receiving multiple messages aren’t guaranteed in each call. Consequently, you must call the API until it returns empty result.

For notifications, the publish-subscribe pattern fits the scenario well. We’ll use two Amazon SNS topics: one for successful job completions and another for failures and cancellations. Administrators can handle subscriptions downstream as needed, whether via email or additional Lambda processing.

Finally, SSM Parameter Store will store configuration details, including the SQS queue URL and SNS topic ARNs, ensuring consistency. The resulting Lambda function could look like this:

import boto3
import json
from botocore.exceptions import ClientError

bedrock_agent = boto3.client('bedrock-agent')
ssm = boto3.client('ssm')
sns = boto3.client('sns')
sqs = boto3.client('sqs')


def get_ssm_parameter(name):
    response = ssm.get_parameter(Name=name, WithDecryption=True)
    return response['Parameter']['Value']


def get_ingestion_job(knowledge_base_id, data_source_id, ingestion_job_id):
    response = bedrock_agent.get_ingestion_job(
        knowledgeBaseId=knowledge_base_id,
        dataSourceId=data_source_id,
        ingestionJobId=ingestion_job_id
    )
    return response['ingestionJob']


def lambda_handler(event, context):
    try:
        sqs_queue_url = get_ssm_parameter('/check-kb-ingestion-job-statuses/sqs-queue-url')
        success_sns_topic_arn = get_ssm_parameter('/check-kb-ingestion-job-statuses/success-sns-topic-arn')
        failure_sns_topic_arn = get_ssm_parameter('/check-kb-ingestion-job-statuses/failure-sns-topic-arn')

        response = sqs.receive_message(
            QueueUrl=sqs_queue_url,
            MaxNumberOfMessages=10
        )
        while 'Messages' in response:
            messages = response['Messages']
            for message in messages:
                body = json.loads(message['Body'])
                knowledge_base_id = body['knowledge_base_id']
                data_source_id = body['data_source_id']
                ingestion_job_id = body['ingestion_job_id']

                print(
                    f'Checking ingestion job status for knowledge base {knowledge_base_id} data source {data_source_id} job {ingestion_job_id}')
                ingestion_job = get_ingestion_job(knowledge_base_id, data_source_id, ingestion_job_id)
                print(
                    f'Ingestion job summary: \n\n{json.dumps(ingestion_job, indent=2, sort_keys=True, default=str)}')
                job_status = ingestion_job['status']
                if job_status == 'COMPLETE':
                    sns.publish(
                        TopicArn=success_sns_topic_arn,
                        Subject=f'Ingestion job for knowledge base {knowledge_base_id} data source {data_source_id} job {ingestion_job_id} Completed',
                        Message=json.dumps(ingestion_job, indent=2, sort_keys=True, default=str)
                    )
                elif job_status == 'FAILED':
                    sns.publish(
                        TopicArn=failure_sns_topic_arn,
                        Subject=f'Ingestion job for knowledge base {knowledge_base_id} data source {data_source_id} job {ingestion_job_id} FAILED',
                        Message=json.dumps(ingestion_job, indent=2, sort_keys=True, default=str)
                    )
                elif job_status == 'STOPPED':
                    sns.publish(
                        TopicArn=failure_sns_topic_arn,
                        Subject=f'Ingestion job for knowledge base {knowledge_base_id} data source {data_source_id} job {ingestion_job_id} STOPPED',
                        Message=json.dumps(ingestion_job, indent=2, sort_keys=True, default=str)
                    )

                if job_status in ['COMPLETE', 'FAILED', 'STOPPED']:
                    sqs.delete_message(
                        QueueUrl=sqs_queue_url,
                        ReceiptHandle=message['ReceiptHandle']
                    )
            response = sqs.receive_message(
                QueueUrl=sqs_queue_url,
                MaxNumberOfMessages=10
            )
        return {
            'statusCode': 200,
            'body': 'Success'
        }
    except ClientError as e:
        return {
            'statusCode': 500,
            'body': f'Client error: {str(e)}'
        }
    except Exception as e:
        return {
            'statusCode': 500,
            'body': f'Unexpected error: {str(e)}'
        }

Terraform Configuration Design

No solution is complete without an easy way to deploy it. Terraform is an obvious choice given my advocacy for it, however any IaC solutions or even frameworks such as AWS SAM would do. Since the event-based automation architecture is quite typical, I won’t go into too much details on how the Terraform configuration is developed. But here is a snippet related to the first Lambda function:

# Data sources and local values omitted for brevity

resource "aws_sqs_queue" "check_kb_ingestion_job_statuses" {
  name = "check-kb-ingestion-job-statuses"
}

resource "aws_ssm_parameter" "start_kb_ingestion_jobs_config_json" {
  name  = "/start-kb-ingestion-jobs/config-json"
  type  = "String"
  value = jsonencode(var.start_kb_ingestion_jobs_config_json)
}

resource "aws_ssm_parameter" "start_kb_ingestion_jobs_sqs_queue_url" {
  name  = "/start-kb-ingestion-jobs/sqs-queue-url"
  type  = "String"
  value = aws_sqs_queue.check_kb_ingestion_job_statuses.id
}

resource "aws_iam_role" "lambda_start_kb_ingestion_jobs" {
  name = "FunctionExecutionRoleForLambda-start-kb-ingestion-jobs"
  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Action = "sts:AssumeRole"
        Effect = "Allow"
        Principal = {
          Service = "lambda.amazonaws.com"
        }
        Condition = {
          StringEquals = {
            "aws:SourceAccount" = "${local.account_id}"
          }
        }
      }
    ]
  })
}

resource "aws_iam_role_policy_attachment" "lambda_start_kb_ingestion_jobs_lambda_basic_execution" {
  role       = aws_iam_role.lambda_start_kb_ingestion_jobs.name
  policy_arn = data.aws_iam_policy.lambda_basic_execution.arn
}

resource "aws_iam_role_policy" "lambda_start_kb_ingestion_jobs" {
  name = "FunctionExecutionRolePolicyForLambda-start-kb-ingestion-jobs"
  role = aws_iam_role.lambda_start_kb_ingestion_jobs.name
  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Action = [
          "ssm:GetParameter",
          "ssm:GetParameters",
          "ssm:GetParametersByPath"
        ]
        Effect   = "Allow"
        Resource = "arn:${local.partition}:ssm:${local.region}:${local.account_id}:parameter/*"
      },
      {
        Action   = "sqs:SendMessage"
        Effect   = "Allow"
        Resource = "arn:${local.partition}:sqs:${local.region}:${local.account_id}:*"
      },
      {
        Action = [
          "bedrock:StartIngestionJob"
        ]
        Effect   = "Allow"
        Resource = "arn:${local.partition}:bedrock:${local.region}:${local.account_id}:knowledge-base/*"
      }
    ]
  })
}

resource "aws_lambda_function" "start_kb_ingestion_jobs" {
  function_name = "start-kb-ingestion-jobs"
  role          = aws_iam_role.lambda_start_kb_ingestion_jobs.arn
  description   = "Lambda function that starts ingestion jobs for Bedrock Knowledge Bases"
  filename      = data.archive_file.start_kb_ingestion_jobs_zip.output_path
  handler       = "index.lambda_handler"
  runtime       = "python3.13"
  architectures = ["arm64"]
  timeout       = 60
  # source_code_hash is required to detect changes to Lambda code/zip
  source_code_hash = data.archive_file.start_kb_ingestion_jobs_zip.output_base64sha256
}

resource "aws_cloudwatch_event_rule" "start_kb_ingestion_jobs" {
  name                = "lambda-start-kb-ingestion-jobs"
  schedule_expression = var.start_kb_ingestion_jobs_schedule
}

resource "aws_cloudwatch_event_target" "start_kb_ingestion_jobs" {
  rule = aws_cloudwatch_event_rule.start_kb_ingestion_jobs.name
  arn  = aws_lambda_function.start_kb_ingestion_jobs.arn
}

resource "aws_lambda_permission" "start_kb_ingestion_jobs" {
  statement_id  = "AllowExecutionFromCloudWatch"
  action        = "lambda:InvokeFunction"
  function_name = aws_lambda_function.start_kb_ingestion_jobs.function_name
  principal     = "events.amazonaws.com"
  source_arn    = aws_cloudwatch_event_rule.start_kb_ingestion_jobs.arn
}

# Remaining resources omitted for brevity

What you will notice is that:

The IAM and resource policies follow the least privilege principle.
The SSM parameters uses a simple parameter hierarchy based on the Lambda function name. If deploying multiple copies of the solution, consider using a variable to set the function name and related source names.
The Lambda function uses the arm64 architecture for ~20% better cost efficiency per GB-second/month vs. x86_64, though the impact in this solution is negligible.
The default Lambda execution timeout of three seconds is insufficient. Increasing it to 60 seconds provides a good safety net.

Deploying and Testing the Solution

✅ You can find the complete Terraform configuration and source code in the 3_kb_data_ingestion directory in this GitHub repository.

With the prerequisites in place, deploy the solution as follows:

From the root of the cloned GitHub repository, navigate to 3_kb_data_ingestion.
Copy terraform.tfvars.example as terraform.tfvars and update the variables to match your configuration.

* By default, the schedule for the `start-kb-ingestion-jobs` Lambda function is daily at 0:00 UTC, while the schedule for the `check_kb_ingestion_job_statuses` Lambda function is every five minutes.

Configure your AWS credentials.
Run terraform init and terraform apply -var-file terraform.tfvars.

Next, manually invoke the start-kb-ingestion-jobs Lambda function in the Lambda console within the five minutes of the scheduled check_kb_ingestion_job_statuses runs.

After the next check_kb_ingestion_job_statuses invocation completes at the scheduled time, check the CloudWatch logs to confirm it ran successfully. You should also receive a a email from SNS with its status. Here is an example:

The message shows several warnings about files that couldn’t be ingested. In my case, I used a S3 bucket that contained tar.gz files, which are unsupported. There was also one file exceeding the 50 MB limit, so it was also skipped. With this information, you can remediate these issues to ensure good data input to the knowledge base.

Summary

In this blog post, we examined the development of a Lambda-based data ingestion solution for Bedrock knowledge bases. The design follows a familiar event-based pattern, leveraging AWS APIs to perform the required tasks and Terraform for deployment. That said, while checking ingestion job statues on a schedule works, it is not as efficient as a true event-based system which Amazon Bedrock does not seem to support at the moment. Perhaps I can update the solution once more support for data ingestion job events are added in the future.

Another interesting experience while writing this blog post is that I used GitHub Copilot to develop both the Lambda functions and the Terraform configuration. While it didn’t produce ready-to-use code, it saved me significant time. I’ll share more about this experience in my next blog post.

In the mean time, please check out other helpful content in the Avangards Blog. Thanks for reading!

Encrypting EBS Volumes of Amazon EC2 Instances Using Python

Anthony Wat — Mon, 20 Jan 2025 02:14:11 +0000

Introduction

Ensuring compliance with stringent security requirements often leads to unexpected challenges - here’s one I recently tackled. The account in question has hundreds of EC2 instances with EBS volumes that are encrypted with the KMS AWS managed key aws/ebs. Due to more stringent security compliance requirements, the encryption key must be rotated every 90 days. The default one-year rotation period of the AWS managed key no longer suffices, thus all EC2 instance volumes must be re-encrypted with a custom managed key (CMK) that provides more control.

With a looming deadline, it was simply not feasible to manually re-encrypt all EBS volumes. Thus I set out to find a tool or script that can automate this daunting task before resorting to developing my own (even if it’s AI-generated). Luckily I found a GitHub repository with a script that meets 90% of my needs, and made perfect after some enhancements. I’d like to share my experience and the resulting script in this blog post in case it benefits any fellow builders facing the same problem. Let’s first set the stage by examining the encryption workflow.

The Encryption Workflow

To encrypt or re-encrypt an EBS volume that is attached to an EC2 instance, it is unfortunately not as simple as setting a KMS key ID on the volume. The process is a bit roundabout and involves the following steps:

Shut down the EC2 instance.
Create a snapshot of the volume.
Create a new volume from the previously created snapshot, while enabling encryption with the new KMS key. Ensure that you select the same availability zone as the original volume and apply any volume settings.
Detach the original volume from the EC2 instance while making note of the device name to which the volume is attached.
Attach the new volume to the EC2 instance with the same device name as above.
Repeat step 2 to step 6 for any additional volumes that the EC2 instance has.
Start the EC2 instance and verify that it is running properly.
Delete the original volumes and the snapshots taken during this process as appropriate.

There are other considerations and best practices for Amazon EBS encryption with auto scaling groups, spot instances, and snapshot sharing, however they are not relevant for the basic scenario for this blog post.

As you can see, the workflow is quite involved and thus makes a great candidate for automation.

Leveraging an Existing Script on GitHub

The need to encrypt or re-encrypt EBS volumes is not uncommon, so I figure that someone would have developed tools and scripts for it. Indeed, a quick Google search yielded three possible options:

dwbelliston/aws_volume_encryption - a Python-based script developed by Dustin Belliston that orchestrates encryption of EBS volumes of an EC2 instance.
jbrt/ec2cryptomatic - a Go-based tool developed by Julien B. that is very similar to the Python solution above, but with a few more quality-of-life features.
aws-samples/aws-system-manager-automation-unencrypted-to-encrypted-resources - an AWS solution that automatically remediates unencrypted EBS and RDS resources using AWS Config and SSM Automation.

Given Boto3 and Python are part of my preferred toolset, I decided to leverage the aws_volume_encryption solution as my starting point. The repository has a well-written README file that provides usage instructions and detailed explanation on what each section of the code does. Be sure to check it out so you understand the general architecture and usage of the script.

Enhancing the Existing Script

Although developed years ago, the original script remains fully functional, proving its reliability. That said, I have identified a few small enhancements that could improve the usability of the script:

Defer to Boto3’s default credential search mechanism instead of adding redundant options to the script.
Create an encrypted volume directly from an encrypted snapshot.
Add KMS key ID validation and skip encryption of volumes that are already encrypted with the provided KMS key. The KMS key ID can be any of the four supported formats by the AWS API.
Add option to preserve the original volumes and add metadata tags (prefixed by VolumeEncryptionMetadata:) to them in case volume changes need to be reverted.
Improve console logging with timestamps and more details.

These enhancements allow me to test and benchmark the script more effectively, and they provide me an extra level of assurance when working on production workloads.

💡 You can find the source code in my forked GitHub repository that accompanies this blog post.

Running the Script

To test the script, create a Windows EC2 instance with an unencrypted root volume and a data volume that is encrypted using the AWS managed key:

Once the EC2 instance is running, connect to Windows, initialize the second volume as D: drive, and add a text file to help with validation later.

Then create a KMS CMK that will be used to encrypt the EBS volumes of the EC2 instance.

Lastly, clone the GitHub repository or download the code zip file, then follow the README file to set up the prerequisites including Python 3.x and the AWS CLI. Since the script defers to the typical credentials lookup sequence, you may use any of the supported methods. Typically, you either configure a profile with the AWS CLI and refer to it using the AWS_PROFILE environment variable, or you use the more verbose environment variables including AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, and AWS_SESSION_TOKEN. In any case, ensure that you provide the target region in the profile or using the AWS_DEFAULT_REGION environment variable.

Run the volume_encryption.py script while providing the following arguments:

-i with the ID of the target EC2 instance
-k with the KMS CMK ID or alias
-p to preserve the original volume, which can be inspected and deleted after validating the EC2 instance with newly encrypted volumes

Here is the command that is specific to my resources:

python volume_encryption.py -i i-095dc6901ff37f71d -k 73d13a4a-b0b0-4ced-b82b-d86a78c89df0 -p

How long the script takes largely depends on the volume sizes and the encryption state of the volumes. For my test instance with a 30 GB encrypted volume and a 8 GB encrypted volume, it took a bit over 6 minutes to complete. If I were to run the script again using another key, it would take more than 20 minutes to complete, presumably because re-encryption takes longer. In any case, the console logs include timestamps that indicate how long each step takes.

$ python volume_encryption.py -i i-095dc6901ff37f71d -k 73d13a4a-b0b0-4ced-b82b-d86a78c89df0 -p
[2025-01-19T13:51:38.781614-05:00] Checking instance i-095dc6901ff37f71d
[2025-01-19T13:51:39.624892-05:00] Stopping instance i-095dc6901ff37f71d
[2025-01-19T13:52:55.908023-05:00] Create snapshot of volume vol-0c79ec8bde159fc7b (xvdb)
[2025-01-19T13:53:57.048132-05:00] Create encrypted volume from snapshot snap-0fb3ecd0eaf583f82
[2025-01-19T13:53:57.684619-05:00] Detach volume vol-0c79ec8bde159fc7b (xvdb)
[2025-01-19T13:53:58.048124-05:00] Attach volume vol-0ddb7f75257290d19 (xvdb)
[2025-01-19T13:54:13.966946-05:00] Create snapshot of volume vol-054b9017ef1f8f25c (/dev/sda1)
[2025-01-19T13:55:14.751392-05:00] Create encrypted volume from snapshot snap-0dac8c77e6e777d1a
[2025-01-19T13:55:15.271831-05:00] Detach volume vol-054b9017ef1f8f25c (/dev/sda1)
[2025-01-19T13:55:15.615831-05:00] Attach volume vol-01140d9f4c666a763 (/dev/sda1)
[2025-01-19T13:55:31.975440-05:00] Start instance i-095dc6901ff37f71d
[2025-01-19T13:55:48.251798-05:00] Clean up resources
[2025-01-19T13:55:48.252797-05:00] Delete snapshot snap-0fb3ecd0eaf583f82
[2025-01-19T13:55:48.438101-05:00] Skipping deletion of original volume vol-0c79ec8bde159fc7b (xvdb)
[2025-01-19T13:55:48.438101-05:00] Delete snapshot snap-0dac8c77e6e777d1a
[2025-01-19T13:55:48.614087-05:00] Skipping deletion of original volume vol-054b9017ef1f8f25c (/dev/sda1)
[2025-01-19T13:55:48.615090-05:00] Encryption finished
$

Once the script completes, verify that the EBS volumes are encrypted with the new KMS key.

You should also see that the original volumes still exist and have some metadata tags added by the script for traceability. Note the original volume IDs from the console logs of the script (for example, vol-054b9017ef1f8f25c is the ID of the original root volume).

Lastly, log in to the EC2 instance and ensure that Windows is working as intended and D:\hello.txt still exists.

However, you will notice that the EC2 instance is seemingly slower than usual. This is because volumes that are restored from snapshots are not fully initialized or pre-warmed. This means that only when a block within the volume is accessed that it will be loaded from the snapshot that’s stored in S3 behind the scenes, thus increasing the I/O latency. While this may not be a huge issue with common use cases, for use cases with high disk I/O needs (such as running a database), you may need to manually initialize the disks.

Summary

With this improved script, you can (re-)encrypt the EBS volumes of any EC2 instance with ease. If you are encrypting volumes for many instances, you can also write another script that reads a CSV file containing EC2 instance information and runs volume_encryption.py on multiple instances in parallel. AI tools like ChatGPT, Amazon Q Developer, or GitHub Copilot can easily create one for you, as I did for my own needs. I will leave this as an exercise for the audience.

As they say, prevention is better than cure. If your organization’s security policies require that EBS volumes be encrypted, consider using the Amazon EBS encryption by default feature to automatically encrypt any new EBS volumes.

This demonstrates how automation and generative AI empower DevOps engineers to tackle complex challenges efficiently. I hope you find this blog post informative and the script useful should you run into similar situations. If you like this article, please check out my other blog posts for more helpful and intriguing content on AWS and DevOps. Thank you for reading and have a great one!

Guardrail Support for the Generic Bedrock Agent Test UI

Anthony Wat — Thu, 26 Sep 2024 03:07:44 +0000

Introduction

In the blog post Developing a Generic Streamlit UI to Test Amazon Bedrock Agents, I shared the design and source code of a basic yet functional UI for testing Bedrock agents. I’ve since added support for Knowledge Bases for Amazon Bedrock by displaying citations and their details to match the functionality in the Bedrock console.

Recently I’ve started experimenting with Guardrails for Amazon Bedrock, a feature that enables the implementation of safeguards for your generative AI applications based on specific use cases and responsible AI policies. As part of the blog post A Guide to Effective Use of the Terraform AWS Cloud Control Provider, I created a simple history-themed Bedrock agent with a guardrail that filters violent content. As I test a guardrail-enabled agent in the Bedrock console, I see additional traces showing whether guardrails have intervened as they assess the input and output, and if so, what policies were triggered:

With a bit of work, I have added similar support to the generic test UI and I am happy to share the updates in the GitHub repository.

Design overview

With the latest update, guardrail traces are now added to the pre-processing and post-processing trace sections in a manner similar to how they are displayed in the Bedrock console:

The Boto3 invoke_agent method provides a new guardrailTrace trace type that includes trace details used in the guardrail. Distinguishing between pre- and post-guardrail traces took a bit of work, as they need to be tracked as events are streamed in sequence. If a guardrailTrace shows up for the first time (naturally before preProcessingTraces, it would be a pre-guardrail trace. A subsequent guardrailTrace that shows up (naturally after any postProcessingTraces would be a post-guardrail trace. Then they must be displayed under the Pre-Processing and Post-Processing sections as a single JSON object unlike other traces where they are broken down into smaller JSON objects for readability.

Summary

With the improvements to the generic test UI outlined in this post, you should now be able to test any Bedrock agents with an associated guardrail. I will be sure to incorporate support for new Agents for Amazon Bedrock features. If you find this blog post helpful, there is plenty more similar content at the Avangards Blog. Be sure to check them out!

A Guide to Effective Use of the Terraform AWS Cloud Control Provider

Anthony Wat — Mon, 23 Sep 2024 02:35:09 +0000

Introduction

The AWS Cloud Control (CC) Provider gained significant attention in May 2024 when it became generally available, three years after its initial launch. It promises to support new AWS features and services immediately due to its auto-generated nature, which is especially beneficial for the rapidly evolving generative AI services like Amazon Bedrock.

But should you immediately switch to the AWS CC Provider and abandon the classic AWS Provider? Not necessarily. While the CC Provider brings speed and coverage of new services, it’s not without limitations. In this blog post, we’ll break down the strengths and weaknesses of both providers, highlighting when it makes sense to leverage the AWS CC Provider and where the classic AWS Provider still shines. A practical example will demonstrate how both can be best used together.

Understanding the Strengths and Weaknesses of the AWS CC Provider

The main selling point of the AWS CC Provider is that it provides support for new AWS services sooner than the classic AWS Provider. The GA announcement showcases this speed by supporting the Amazon Q Business resources early on. In contrast, the enhancement request that was opened against the AWS Provider in January 2024 is still pending, even though a pull request (PR) has already been submitted by a contributor for some time. It makes sense to leverage the resources from the AWS CC Provider to not delay your IaC automation effort.

Even though the AWS CC Provider covers new AWS services quickly, there’s still a lot of room for improvement when it comes to older services. According to this GitHub issue, as of October 2023, the Cloud Control API supports only 859 resources, many of which are not supported in all AWS regions. According to Resource types that support Cloud Control API, as of July 2024 it supports 1034 resources, which is encouraging to see. However, there is still a list of suppressed resources that are not compatible with how the AWS CC Provider generates resources, bringing the actual supported number of resources to around 1,000. Compared to about 1,400 resources supported by the AWS Provider, that's only about 70% coverage and less if you consider resources that haven't been implemented in the AWS Provider.

While working with the AWS CC Provider, I noticed challenges with both documentation and quality assurance. The quality of descriptions for resources and attributes is somewhat inconsistent. For example, the documentation for the awscc_iam_group resource is quite well written, while the documentation for the awscc_qbusiness_application resource is practically non-existent. Overall, it pales in comparison to the AWS API documentation which I often refer to when contributing to the AWS Provider. I am not sure why the CloudFormation schemas (from which the AWS CC Provider resources and documentation are generated) are so far apart from the AWS API documentation, but I hope AWS can reconcile the two sources at some point.

As for the functional quality of the AWS CC Provider, my experience unfortunately hasn't been great. While adding examples to the Lightsail resources, I ran into two major functional issues and two documentation issues that are caused upstream in the Cloud Control API. This led me to believe that there is insufficient quality assurance with the Cloud Control API, and the generated nature of the AWS CC Provider does not help catch these issues. The situation will hopefully improve over time, but for the time being I would prefer the AWS Provider for mission-critical use. Nevertheless, I must give credit to the AWS CC Provider maintainers in diligently reporting and working with AWS to resolve upstream issues in a timely manner. The turnaround time is much quicker than if I were to open AWS support cases myself.

Also Knowing the Merits and Drawbacks of the Classic AWS Provider

The Terraform AWS Provider has been active for over ten years and has recently surpassed three billion downloads. The tremendous work that HashiCorp, AWS, and the community put into the provider over the years has led to high AWS service coverage. The provider boasts ample acceptance tests which result in a relatively high degree of quality. The user base also proactively reports less prevalent issues that are not caught by automated tests.

Since the AWS Provider code is not generated like the AWS CC Provider, the hand-crafted nature means that development is labor intensive and time consuming. Consequently, lower priority issues and features often take some time to be fixed. Even when a PR is submitted by a contributor, a maintainer from HashiCorp still needs to review, test, and merge it in between their other work such as those related to product roadmaps.

Meanwhile, the need for hands-on development also affords the flexibility to add custom logic to resources and data sources. As previously mentioned, the AWS API often does not fit perfectly into a CRUDL model due to actions that fall outside these operations. For instance, the Agents for Amazon Bedrock API has an AssociateAgentKnowledgeBase action that associates a knowledge base to an agent. Since it is not considered a resource in the AWS CC API, it is not mapped to a resource in the AWS CC Provider. However, an experienced developer for the AWS Provider is able to adapt this action into an "association", resulting in the aws_bedrockagent_agent_knowledge_base_association resource.

As another example, a Bedrock agent must be prepared using the PrepareAgent action after it is updated. Since this action cannot be easily adapted to a resource, the logical approach is to call this API action when an aws_bedrockagent_agent resource is created and updated, leading to the custom prepare_agent argument. Similar logic can be added to other Agents for Bedrock resources that indirectly modifies an agent. Having this type of custom resources and logic is only possible in the AWS Provider today.

Using Both Providers in a Complementary Manner

The good news is that Terraform is designed to work with multiple providers, so you can leverage both the AWS Provider and the AWS CC Provider for what they each excel at.

Let’s look at a use case of adding a guardrail to a Bedrock agent. Currently, the AWS Provider has the aws_bedrock_guardrail resource, but it does not yet have a resource to manage guardrail versions. While the aws_bedrock_agent resource has been around for some time, it does not yet have the configuration to associate a guardrail.

On the other hand, the AWS CC Provider has a awscc_bedrock_guardrail_version resource and the awscc_bedrock_agent resource supports the guardrail_configuration argument for associating a guardrail. Thus we can strategically use the AWS CC Provider for the new features while using the AWS Provider for all other resources.

Here is the Terraform configuration for a simple Bedrock agent that answers questions about world history, but is guarded against providing information on violent historical events like what happened to Julius Caesar:

data "aws_caller_identity" "this" {}
data "aws_partition" "this" {}
data "aws_region" "this" {}
locals {
  account_id = data.aws_caller_identity.this.account_id
  partition  = data.aws_partition.this.partition
  region     = data.aws_region.this.name
}

data "aws_bedrock_foundation_model" "this" {
  model_id = "anthropic.claude-3-haiku-20240307-v1:0"
}

resource "aws_bedrock_guardrail" "this" {
  name                      = "MyGuardrail"
  description               = "My guardrail"
  blocked_input_messaging   = "Sorry, I cannot answer this question."
  blocked_outputs_messaging = "Sorry, I cannot answer this question."
  content_policy_config {
    filters_config {
      input_strength  = "HIGH"
      output_strength = "HIGH"
      type            = "VIOLENCE"
    }
  }
}

resource "awscc_bedrock_guardrail_version" "this" {
  guardrail_identifier = aws_bedrock_guardrail.this.guardrail_id
  lifecycle {
    replace_triggered_by = [aws_bedrock_guardrail.this]
  }
}

resource "aws_iam_role" "bedrock_agent_this" {
  name = "AmazonBedrockExecutionRoleForAgents_MyAgent"
  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Action = "sts:AssumeRole"
        Effect = "Allow"
        Principal = {
          Service = "bedrock.amazonaws.com"
        }
        Condition = {
          StringEquals = {
            "aws:SourceAccount" = local.account_id
          }
          ArnLike = {
            "aws:SourceArn" = "arn:${local.partition}:bedrock:${local.region}:${local.account_id}:agent/*"
          }
        }
      }
    ]
  })
}

resource "aws_iam_role_policy" "bedrock_agent_this" {
  name = "AmazonBedrockAgentBedrockFoundationModelPolicy_MyAgent"
  role = aws_iam_role.bedrock_agent_this.name
  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Sid      = "InvokeFoundationModel"
        Action   = "bedrock:InvokeModel"
        Effect   = "Allow"
        Resource = data.aws_bedrock_foundation_model.this.model_arn
      },
      {
        Sid      = "ApplyGuardrail"
        Action   = "bedrock:ApplyGuardrail"
        Effect   = "Allow"
        Resource = awscc_bedrock_guardrail_version.this.guardrail_arn
      }
    ]
  })
}

resource "awscc_bedrock_agent" "this" {
  agent_name              = "MyAgent"
  agent_resource_role_arn = aws_iam_role.bedrock_agent_this.arn
  auto_prepare            = true
  description             = "My Agent"
  foundation_model        = data.aws_bedrock_foundation_model.this.model_id
  guardrail_configuration = {
    guardrail_identifier = awscc_bedrock_guardrail_version.this.guardrail_arn
    guardrail_version    = awscc_bedrock_guardrail_version.this.version
  }
  instruction = "You are an assistant that provides information about world history. You are allowed to use general knowledge that you already possess to answer any history-related questions."
}

As you can see, the Terraform configuration maintains the familiar usage of resources and data sources in the AWS Provider. A quick validation in the Amazon Bedrock console shows that the agent does indeed filter the violent event about Julius Caesar, while it correctly answers another question about the history of wheels.

Summary

In this blog post, we looked at the pros and cons of the AWS Provider and the AWS CC Provider. As it stands, there is still a long way until AWS CC Provider has the quality and feature parity necessary to replace the AWS Provider, so both are here to stay for the foreseeable future.

If you're managing complex AWS infrastructure, now is the time to experiment with both providers. Use the AWS CC Provider for cutting-edge features, and rely on the AWS Provider for tried-and-true solutions. By blending both providers, you’ll have the best of both worlds in your Terraform configurations. You can follow this tutorial or find more information here.

For other tips and walkthroughs on AWS and Terraform, be sure to check out the Avangards Blog. Thanks for reading!

My Experience With the AWS Certified AI Practitioner (AI1-C01) Beta Exam

Anthony Wat — Wed, 04 Sep 2024 06:04:40 +0000

Introduction

AWS has recently revamped their certification lineup to align with the growing AI/ML trend. Among them is the AWS Certified AI Practitioner (AI1-C01) exam, which is currently in beta. Beta exams validate exam questions to finalize the content before wide release. As an early adopter, you get a discount on the exam fee in exchange for your contribution to test out the exam for AWS. Given that this is an intriguing proposition and it aligns with my focus on AWS, I decided to write this exam and share my experience with the community.

A Bit About My Background

For context, I've been working exclusively with AWS for about three years and completed a couple of professional/specialty level exams, so I am quite familiar with AWS in general. My work also involves generative and conversational AI (although it has mostly been on the Microsoft side), so I have knowledge on concepts such as LLM and RAG.

I am interested in generative AI on AWS, so I've been experimenting on my own and has written a few blog post about Agents for Amazon Bedrock and Knowledge Bases for Amazon Bedrock. However, my knowledge of Amazon SageMaker is limited to understanding the ML workflow and some hands-on experience from AWS Workshops.

Overall I'd say that I know a bit more about AI/ML than the average Joe, so I was able to expedite my study somewhat. As you read about my exam prep, consider your own knowledge and experience to adjust your approach.

How I Studied for the Exam

In the past, I've always used A Cloud Guru to study for AWS and Azure exams. Recently I've been given an AWS Skill Builder subscription by my company, so I've decided to use it as the primary source of study material.

The official AWS Certified AI Practitioner webpage recommends a 4-step exam prep plan which I followed over two days. To test my knowledge prior to studying, I went through the official practice question set and got 85% which boosted my confidence (but eventually turned out to be a trap). I then proceeded with the enhanced exam prep course that included bonus practice questions and flashcards over the standard (free) version.

The course itself is well-organized and touches upon all topics. The instructor spoke very slowly in the video, so I watched the videos at 2x speed for a more reasonable pace. However, I found that certain concepts aren't explained very well and the level of details isn't representative of what the exam itself demands. I suppose the additional resources would have been good supplements, however I would prefer if the course itself is more in-depth.

The bonus questions provided additional practice opportunities and the flashcards were helpful for memorizing key concepts. I transferred the flashcards into a Word document for offline review before the exam. That being said, they did not cover all important concepts and I had to supplement them with my own research on Google (such as a list of ML algorithms).

Following my typical study regimen, I sought more practice right before the exam. Aside from the practice questions from AWS, I quickly went through Tutorial Dojo's free practice exams sampler given that they don't yet have a full practice exam package yet. (I later found out that Stephane Maarek does.)

How the Exam Went

I took the exam online at the comfort of my home in a quiet evening. The check-in process went smoothly and took about 10 minutes. The exam questions were more difficult than I expected of a foundational-level exam. On a scale of 1 to 10, I’d rate it a 4 in difficulty.

Although the exam guide lists case study as a question type, there wasn't any for me so your mileage may vary. There were many questions on machine learning concepts such as algorithms and performance metrics. Amazon Bedrock also featured a lot. There was also the expected mix of questions on Amazon SageMaker features, generative AI, and responsible AI. While I wouldn't say that the questions were very different from the official practice question set or the bonus questions in the exam prep enhanced course, they seemed more in-depth and specific.

It took me about 75 minutes to complete the exam, with the first pass completed in about an hour which left 27 out of 85 questions flagged for review. This is more questions than I usually flag in AWS exams (including the DOP exam), and after the review I was only confident on my final answer to about half of those questions. I finished the exam feeling I would pass, though not with full confidence.

A Retrospective on My Approach

I received the results by early morning and scored only 729, which admittedly is lower than I expected. Nonetheless, a pass is a pass. According to the report, I didn't do as well in the "guidelines for responsible AI" domain which was a bit surprising as I did well during practice. In any case, here is my reflection on my experience based on a typical retrospective framework.

What Went Well

Setting an exam date motivated me to be disciplined and keep to a set study schedule amid other priorities in life. The long weekend afforded me enough time to study, socialize, and do chores.
Following the official exam prep plan and the exam prep course helped structure my study. I've always studied by following a course be it from A Cloud Guru or AWS Skill Builder, and it's been proven to be a sound strategy.

What Didn't Go Well

I was overly confident of my hands-on experience with services such as Amazon Bedrock, when there were numerous features and other services that I've not have enough exposure to. Consequently I did not allocate time to watch tutorial videos or do hands-on labs to gain the necessary familiarity and "muscle memory".
I didn’t spend enough time on Step 2 of the 4-step plan to explore additional courses outside the AWS Skill Builder exam prep course. I've retroactively looked at some of the recommended courses and they would have improved my overall knowledge for this exam.

What Could Be Improved

Although I was able to get by with one full day of study, it would have been better to spread the study across multiple days for a better pace.
I would have done better with additional research on general AI/ML concepts such as algorithms and performance metrics, which the exam seemed to have more focus on.
Although the AWS Skill Builder exam prep enhanced course was decent, it was not fully adequate as the sole study material. Investing in additional courses and practice exams, such as those from Stephane Maarek, would probably have helped boost my score.

Summary

I hope this blog post gives you a sense of what to expect from the AWS Certified AI Practitioner (AI1-C01) Beta Exam. It's certainly not an exam that you can wing, unless you are well-exposed to AI/ML or are already studying for other certifications such as AWS Certified Machine Learning Engineer - Associate. However with the right material and a few days of study, it is very much achievable.

Check out the Avangards Blog for more articles on AWS, Terraform, and other topics. Best of luck with your studies, and I hope you’ll soon be certified!

How To Manage Amazon Inspector in AWS Organizations Using Terraform

Anthony Wat — Sun, 09 Jun 2024 07:02:52 +0000

Introduction

Over the past two months, I have published numerous blog posts on managing different AWS security services in AWS Organizations using Terraform. In this blog post, I will cover one remaining AWS service, AWS Inspector, for native vulnerability management. The Terraform resources for Inspector are a bit quirky, so I will show some slightly more advanced techniques to keep the configuration neat and configurable. With that said, let's review the objective.

About the use case

Amazon Inspector is a vulnerability management service that continuously scans AWS workloads for software vulnerabilities and unintended network exposure. Supported compute services include Amazon EC2 instances, container images in Amazon ECR, and AWS Lambda functions.

Similar to other AWS security services, Inspector supports managing multiple accounts with AWS Organizations via the delegated administrator feature. Once an account in the organization is designated as a delegated administrator, it can manage member accounts and view aggregated findings.

Since it is increasingly common to establish an AWS landing zone using AWS Control Tower, we will use the standard account structure in a Control Tower landing zone to demonstrate how to configure Inspector in Terraform:

The relevant accounts for our use case in the landing zone are:

The Management account for the organization where AWS Organizations is configured. For details, refer to Managing multiple accounts in Amazon Inspector with Organizations.
The Audit account where security and compliance services are typically centralized in a Control Tower landing zone.

The objective is to delegate Inspector administrative duties from the Management account to the Audit account, after which all organization configurations are managed in the Audit account. Let's walk through how to do this using Terraform.

Designating an Inspector administrator account

The Inspector delegated administrator is configured in the Management account, so we need a provider associated with it in Terraform. To keep things simple, we will take a multi-provider approach by defining two providers, one for the Management account and another for the Audit account, using AWS CLI profiles as follows:

provider "aws" {
  alias   = "management"
  # Use "aws configure" to create the "management" profile with the Management account credentials
  profile = "management" 
}

provider "aws" {
  alias   = "audit"
  # Use "aws configure" to create the "audit" profile with the Audit account credentials
  profile = "audit" 
}

⚠ Since Inspector is a regional service, you must apply this Terraform configuration on each region that you are using. Consider using the region argument in your provider definition and a variable to make your Terraform configuration rerunnable in other regions.

We can designate the delegated administrator using the aws_inspector2_delegated_admin_account resource. However, this does not enable Inspector in the delegated administrator account, so we also need to use the aws_inspector2_enabler resource. What I learned from testing the aws_inspector2_enabler resource is that you cannot provide both the delegated account and the member accounts in the account_ids argument, so we need a dedicated aws_inspector2_enabler resource for the Audit account. According to the resource source code, this is to address a legacy Inspector issue.

The resulting Terraform configuration should look like the following (pay special attention to the provider argument in each resource):

data "aws_caller_identity" "audit" {
  provider = aws.audit
}

resource "aws_inspector2_enabler" "audit" {
  provider       = aws.audit
  account_ids    = [data.aws_caller_identity.audit.account_id]
}

resource "aws_inspector2_delegated_admin_account" "audit" {
  provider   = aws.management
  account_id = data.aws_caller_identity.audit.account_id
  depends_on = [aws_inspector2_enabler.audit]
}

Configuring Inspector activation for new member accounts

To allow more control over which scan types are enabled, we can define the following variables and use them with the relevant resources:

# Variable definition (.tfvars)

variable "enable_ec2" {
  description = "Whether Amazon EC2 scans should be enabled for both existing and new member accounts in the organization."
  type        = bool
  default     = true
}

variable "enable_ecr" {
  description = "Whether Amazon ECR scans should be enabled for both existing and new member accounts in the organization."
  type        = bool
  default     = true
}

variable "enable_lambda" {
  description = "Whether Lambda Function scans should be enabled for both existing and new member accounts in the organization."
  type        = bool
  default     = true
}

variable "enable_lambda_code" {
  description = "Whether Lambda code scans should be enabled for both existing and new member accounts in the organization."
  type        = bool
  default     = true
}

In an organizational setup, Inspector can auto-enable on new member accounts. In Terraform, this can be configured using the aws_inspector2_organization_configuration resource. Leveraging the variables above, the resource can be defined as follows:

resource "aws_inspector2_organization_configuration" "this" {
  provider = aws.audit
  auto_enable {
    ec2         = var.enable_ec2
    ecr         = var.enable_ecr
    lambda      = var.enable_lambda
    lambda_code = var.enable_lambda_code && var.enable_lambda
  }
  depends_on = [aws_inspector2_delegated_admin_account.audit]
}

Note that for AWS Lambda code scanning (lambda_code), AWS Lambda standard scanning (lambda) is a prerequisite, so we need to check both variables to enable it.

Now let's address the existing member accounts.

Activating scanning for existing member accounts

Unlike GuardDuty, the Inspector organization configuration does not support auto-enablement for existing member accounts, so we need to separately manage the member accounts. The strategy is to get the list of active member accounts from the organization, which we can use with the Inspector Terraform resources, including the aws_inspector2_enabler resource. We can exclude the Audit account since that is managed separately. To get the list of member accounts in the organization, we can use the aws_organizations_organization data source.

Furthermore, the aws_inspector2_enabler resource's resource_types argument takes a list of strings that represent the scan types to enable. Since the variables we defined earlier are boolean variables, we need a bit of function magic to create the list of scans to enable based on the variables.

The Terraform configuration that addresses the above requirements can be defined as follows:

data "aws_organizations_organization" "this" {
  provider = aws.management
}

locals {
  enabler_resource_types = compact([
    var.enable_ec2 ? "EC2" : null,
    var.enable_ecr ? "ECR" : null,
    var.enable_lambda ? "LAMBDA" : null,
    var.enable_lambda_code && var.enable_lambda ? "LAMBDA_CODE" : null,
  ])

  member_account_ids = [for account in data.aws_organizations_organization.this.accounts : account.id if account.status == "ACTIVE" && account.id != data.aws_caller_identity.audit.account_id]
}

Member accounts are not automatically associated with the delegated administrator account, so they must first be associated using the aws_inspector2_member_association resource.

Using the for_each meta-argument, we can define a single resource to associate all member accounts with the previously defined member_account_ids local value:

resource "aws_inspector2_member_association" "members" {
  provider   = aws.audit
  for_each   = toset(local.member_account_ids)
  account_id = each.key
  depends_on = [aws_inspector2_delegated_admin_account.audit]
}

Lastly, we can enable Inspector scans in the member accounts using the aws_inspector2_enabler resource. Although the account_ids argument can take the list of member accounts, it is more flexible to have one resource per account. Thus, using for_each and the local values, the resource can be defined as follows:

resource "aws_inspector2_enabler" "members" {
  provider       = aws.audit
  for_each       = toset(local.member_account_ids)
  account_ids    = [each.key]
  resource_types = local.enabler_resource_types
  depends_on     = [aws_inspector2_member_association.members]
}

✅ You can find the complete Terraform in the GitHub repository that accompanies this blog post.

Now that the Terraform configuration is fully defined, you can apply it to establish the Audit account as the delegated administration and centrally manage Inspector settings for both new and existing accounts.

Caveats about deactivating Inspector in member accounts

Among the AWS security services, Inspector has the least sophisticated API for organizational management. The mix between auto-enablement for new member accounts and explicit enablement for existing member accounts complicates how they are managed in Terraform, particularly if you are trying to disable Inspector via terraform destroy.

Consider the case where a new member account is added and auto-enablement is applied to this account. If you run terraform destroy as-is, Terraform is not aware of the new member account, and thus Inspector cannot be deactivated in this account. You must manually deactivate the account in each applied region.

Alternatively, you can first run terraform apply so that the aws_inspector2_member_association and aws_inspector2_enabler resource instances are created, then run terraform destroy to properly clean up. While this method works, you must keep track of when new member accounts are added so that you know when to run terraform apply to reconcile the Terraform resources with the updated organization.

In any case, be aware of this caveat and take one of the two approaches if you ever need to clean up Inspector resources.

Summary

In this blog post, you learned how to manage Amazon Inspector in AWS Organizations using Terraform. With a delegated administrator, Inspector can be auto-enabled for new member accounts, while existing member accounts are dynamically associated and configured with the desired scan types. If you have also configured AWS Security Hub to operate at the organization level, you can manage Inspector findings across accounts and regions, thereby streamlining your security operations.

If you are interested in this type of content, be sure to read other posts on the Avangards Blog, where I share tips and deep dives on AWS, Terraform, and beyond. Thank you, and enjoy the rest of your day!

How To Manage an Amazon Bedrock Knowledge Base Using Terraform

Anthony Wat — Sun, 02 Jun 2024 19:59:56 +0000

Introduction

In the previous blog post, Adding an Amazon Bedrock Knowledge Base to the Forex Rate Assistant, I explained how to create a Bedrock knowledge base and associate it with a Bedrock agent using the AWS Management Console, with a forex rate assistant as the use case example.

We also covered how to manage Bedrock agents with Terraform in another blog post, How To Manage an Amazon Bedrock Agent Using Terraform. In this blog post, we will extend that setup to also manage knowledge bases in Terraform. To begin, we will first examine the relevant AWS resources in the AWS Management Console.

Taking inventory of the required resources

Upon examining the knowledge base we previously built, we find that it comprises the following AWS resources:

The knowledge base itself;
The knowledge base service role that provides the knowledge base access to Amazon Bedrock models, data sources in S3, and the vector index;
The OpenSearch Serverless policies, collection, and the vector index;
The S3 bucket that acts as the data source

With this list of resources, along with those required by the agent to which the knowledge base will be attached, we can begin creating the Terraform configuration. Before diving into the setup, let's first take care of the prerequisites.

Defining variables for the configuration

For better manageability, we define some variables in a variables.tf file that we will reference throughout the Terraform configuration:

variable "kb_s3_bucket_name_prefix" {
  description = "The name prefix of the S3 bucket for the data source of the knowledge base."
  type        = string
  default     = "forex-kb"
}

variable "kb_oss_collection_name" {
  description = "The name of the OSS collection for the knowledge base."
  type        = string
  default     = "bedrock-knowledge-base-forex-kb"
}

variable "kb_model_id" {
  description = "The ID of the foundational model used by the knowledge base."
  type        = string
  default     = "amazon.titan-embed-text-v1"
}

variable "kb_name" {
  description = "The knowledge base name."
  type        = string
  default     = "ForexKB"
}

Defining the S3 and IAM resources

The knowledge base requires a service role, which can be created using the aws_iam_role resource as follows:

data "aws_caller_identity" "this" {}
data "aws_partition" "this" {}
data "aws_region" "this" {}

locals {
  account_id            = data.aws_caller_identity.this.account_id
  partition             = data.aws_partition.this.partition
  region                = data.aws_region.this.name
  region_name_tokenized = split("-", local.region)
  region_short          = "${substr(local.region_name_tokenized[0], 0, 2)}${substr(local.region_name_tokenized[1], 0, 1)}${local.region_name_tokenized[2]}"
}

resource "aws_iam_role" "bedrock_kb_forex_kb" {
  name = "AmazonBedrockExecutionRoleForKnowledgeBase_${var.kb_name}"
  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Action = "sts:AssumeRole"
        Effect = "Allow"
        Principal = {
          Service = "bedrock.amazonaws.com"
        }
        Condition = {
          StringEquals = {
            "aws:SourceAccount" = local.account_id
          }
          ArnLike = {
            "aws:SourceArn" = "arn:${local.partition}:bedrock:${local.region}:${local.account_id}:knowledge-base/*"
          }
        }
      }
    ]
  })
}

With the service role in place, we can now proceed to define the corresponding IAM policy. As we define the configuration for creating resources that the knowledge base service role needs to access, we will consequently define the corresponding IAM policy using the aws_iam_role_policy resource. First, we create the IAM policy that provides access to the embeddings model. Since the foundation model is not created but referenced, we can use the aws_bedrock_foundation_model data source to obtain the ARN which we need:

data "aws_bedrock_foundation_model" "kb" {
  model_id = var.kb_model_id
}

resource "aws_iam_role_policy" "bedrock_kb_forex_kb_model" {
  name = "AmazonBedrockFoundationModelPolicyForKnowledgeBase_${var.kb_name}"
  role = aws_iam_role.bedrock_kb_forex_kb.name
  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Action   = "bedrock:InvokeModel"
        Effect   = "Allow"
        Resource = data.aws_bedrock_foundation_model.kb.model_arn
      }
    ]
  })
}

Next, we create the Amazon S3 bucket that acts as the data source for the knowledge base using the aws_s3_bucket resource. To adhere to security best practices, we also enable S3-SSE using the aws_s3_bucket_server_side_encryption_configuration resource and bucket versioning with the aws_s3_bucket_versioning resource as follows:

resource "aws_s3_bucket" "forex_kb" {
  bucket        = "${var.kb_s3_bucket_name_prefix}-${local.region_short}-${local.account_id}"
  force_destroy = true
}

resource "aws_s3_bucket_server_side_encryption_configuration" "forex_kb" {
  bucket = aws_s3_bucket.forex_kb.id
  rule {
    apply_server_side_encryption_by_default {
      sse_algorithm = "AES256"
    }
  }
}

resource "aws_s3_bucket_versioning" "forex_kb" {
  bucket = aws_s3_bucket.forex_kb.id
  versioning_configuration {
    status = "Enabled"
  }
  depends_on = [aws_s3_bucket_server_side_encryption_configuration.forex_kb]
}

Now that the S3 bucket is available, we can create the IAM policy that gives the knowledge base service role access to files for indexing:

resource "aws_iam_role_policy" "bedrock_kb_forex_kb_s3" {
  name = "AmazonBedrockS3PolicyForKnowledgeBase_${var.kb_name}"
  role = aws_iam_role.bedrock_kb_forex_kb.name
  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Sid      = "S3ListBucketStatement"
        Action   = "s3:ListBucket"
        Effect   = "Allow"
        Resource = aws_s3_bucket.forex_kb.arn
        Condition = {
          StringEquals = {
            "aws:PrincipalAccount" = local.account_id
          }
      } },
      {
        Sid      = "S3GetObjectStatement"
        Action   = "s3:GetObject"
        Effect   = "Allow"
        Resource = "${aws_s3_bucket.forex_kb.arn}/*"
        Condition = {
          StringEquals = {
            "aws:PrincipalAccount" = local.account_id
          }
        }
      }
    ]
  })
}

Defining the OpenSearch Serverless policy resources

The Bedrock console offers a quick create option that provisions an OpenSearch Serverless vector store on our behalf as the knowledge base is created. Since the documentation for creating the vector index in OpenSearch Serverless is a bit open-ended, we can refer to the resources from the quick create option to supplement.

First, we configure permissions by defining a data access policy for the vector search collection. The data access policy from the quick create option is defined as follows:

This data access policy provides read and write permissions to the vector search collection and its indices to the knowledge base execution role and the creator of the policy.

Using the corresponding aws_opensearchserverless_access_policy resource, we can define the policy as follows:

resource "aws_opensearchserverless_access_policy" "forex_kb" {
  name = var.kb_oss_collection_name
  type = "data"
  policy = jsonencode([
    {
      Rules = [
        {
          ResourceType = "index"
          Resource = [
            "index/${var.kb_oss_collection_name}/*"
          ]
          Permission = [
            "aoss:CreateIndex",
            "aoss:DeleteIndex",
            "aoss:DescribeIndex",
            "aoss:ReadDocument",
            "aoss:UpdateIndex",
            "aoss:WriteDocument"
          ]
        },
        {
          ResourceType = "collection"
          Resource = [
            "collection/${var.kb_oss_collection_name}"
          ]
          Permission = [
            "aoss:CreateCollectionItems",
            "aoss:DescribeCollectionItems",
            "aoss:UpdateCollectionItems"
          ]
        }
      ],
      Principal = [
        aws_iam_role.bedrock_kb_forex_kb.arn,
        data.aws_caller_identity.this.arn
      ]
    }
  ])
}

Note that aoss:DeleteIndex was added to the list because this is required for cleanup by Terraform via terraform destroy.

Next, we need an encryption policy that assigns an encryption key to a collection for data protection at rest. The encryption policy from the quick create option is defined as follows:

This encryption policy simply assigns an AWS-owned key to the vector search collection. Using the aws_opensearchserverless_security_policy resource with an encryption type, we can define the policy as follows:

resource "aws_opensearchserverless_security_policy" "forex_kb_encryption" {
  name = var.kb_oss_collection_name
  type = "encryption"
  policy = jsonencode({
    Rules = [
      {
        Resource = [
          "collection/${var.kb_oss_collection_name}"
        ]
        ResourceType = "collection"
      }
    ],
    AWSOwnedKey = true
  })
}

Lastly, we need a network policy which defines whether a collection is accessible publicly or privately. The network policy from the quick create option is defined as follows:

his network policy allows public access to the vector search collection's API endpoint and dashboard over the internet. Using the aws_opensearchserverless_security_policy resource with an network type, we can define the policy as follows:

resource "aws_opensearchserverless_security_policy" "forex_kb_network" {
  name = var.kb_oss_collection_name
  type = "network"
  policy = jsonencode([
    {
      Rules = [
        {
          ResourceType = "collection"
          Resource = [
            "collection/${var.kb_oss_collection_name}"
          ]
        },
        {
          ResourceType = "dashboard"
          Resource = [
            "collection/${var.kb_oss_collection_name}"
          ]
        }
      ]
      AllowFromPublic = true
    }
  ])
}

With the prerequisite policies in place, we can now create the vector search collection and the index.

Defining the OpenSearch Serverless collection and index resources

Creating the collection in Terraform is straightforward using the aws_opensearchserverless_collection resource:

resource "aws_opensearchserverless_collection" "forex_kb" {
  name = var.kb_oss_collection_name
  type = "VECTORSEARCH"
  depends_on = [
    aws_opensearchserverless_access_policy.forex_kb,
    aws_opensearchserverless_security_policy.forex_kb_encryption,
    aws_opensearchserverless_security_policy.forex_kb_network
  ]
}

The knowledge base service role also needs access to the collection, which we can provide using the aws_iam_role_policy similar to before:

resource "aws_iam_role_policy" "bedrock_kb_forex_kb_oss" {
  name = "AmazonBedrockOSSPolicyForKnowledgeBase_${var.kb_name}"
  role = aws_iam_role.bedrock_kb_forex_kb.name
  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Action   = "aoss:APIAccessAll"
        Effect   = "Allow"
        Resource = aws_opensearchserverless_collection.forex_kb.arn
      }
    ]
  })
}

Creating the index in Terraform is however more complex, since it is not an AWS resource but an OpenSearch construct. Looking at CloudTrail events, there wasn't any event that correspond to an AWS API call that would create the index. However, observing the network traffic in the Bedrock console did reveal a request to the OpenSearch collection's API endpoint to create the index. This is what we want to port to Terraform.

Luckily, there is an OpenSearch Provider maintained by OpenSearch that we can use. To connect to the vector search collection, we provide the endpoint URL and credentials in the provider block. The provider has first-class support for AWS, so credentials can be provided implicitly similar to the Terraform AWS Provider. The resulting provider definition is as follows:

provider "opensearch" {
  url         = aws_opensearchserverless_collection.forex_kb.collection_endpoint
  healthcheck = false
}

Note that the healthcheck argument is set to false because the client health check does not really work with OpenSearch Serverless.

To get the index definition, we can examine the collection in the OpenSearch Service Console:

We can create the index using the opensearch_index resource with the same specifications:

resource "opensearch_index" "forex_kb" {
  name                           = "bedrock-knowledge-base-default-index"
  number_of_shards               = "2"
  number_of_replicas             = "0"
  index_knn                      = true
  index_knn_algo_param_ef_search = "512"
  mappings                       = <<-EOF
    {
      "properties": {
        "bedrock-knowledge-base-default-vector": {
          "type": "knn_vector",
          "dimension": 1536,
          "method": {
            "name": "hnsw",
            "engine": "faiss",
            "parameters": {
              "m": 16,
              "ef_construction": 512
            },
            "space_type": "l2"
          }
        },
        "AMAZON_BEDROCK_METADATA": {
          "type": "text",
          "index": "false"
        },
        "AMAZON_BEDROCK_TEXT_CHUNK": {
          "type": "text",
          "index": "true"
        }
      }
    }
  EOF
  force_destroy                  = true
  depends_on                     = [aws_opensearchserverless_collection.forex_kb]
}

Note that the dimension is set to 1536, which is the value required for the Titan G1 Embeddings - Text model.

Before we move on, you must know about an issue with the Terraform OpenSearch provider that caused me a lot of headache. When I was testing the Terraform configuration, the opensearch_index resource kept failing because the provider could not seemingly authenticate against the collection's endpoint URL. After a long debugging session, I was able to find a GitHub issue in the Terraform OpenSearch Provider repository that mentions the cryptic "EOF" error that was present. The issue mentions that the bug is related to OpenSearch Serverless and an earlier provider version, v2.2.0, does not have the problem. Consequently, I was able to work around the problem by using this specific version of the provider:

terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.48"
    }
    opensearch = {
      source  = "opensearch-project/opensearch"
      version = "= 2.2.0"
    }
  }
  required_version = "~> 1.5"
}

Hopefully letting you in on this tip will save you hours of troubleshooting.

Defining the knowledge base resource

With all dependent resources in place, we can now proceed to create the knowledge base. However, there is the matter of eventual consistency with IAM resources that we first need to address. Since Terraform creates resources in quick succession, there is a chance that the configuration of the knowledge base service role is not propagated across AWS endpoints before it is used by the knowledge base during its creation, resulting in temporary permission issues. What I observed during testing is that the permission error is usually related to the OpenSearch Serverless collection.

To mitigate this, we add a delay using the time_sleep resource in the Time Provider. The following configuration will add a 20-second delay after the IAM policy for the OpenSearch Serverless collection is created:

resource "time_sleep" "aws_iam_role_policy_bedrock_kb_forex_kb_oss" {
  create_duration = "20s"
  depends_on      = [aws_iam_role_policy.bedrock_kb_forex_kb_oss]
}

💡 If you still encounter permission issues when creating the knowledge base, try increasing the delay to 30 seconds.

Now we can create the knowledge base using the aws_bedrockagent_knowledge_base resource as follows:

resource "aws_bedrockagent_knowledge_base" "forex_kb" {
  name     = var.kb_name
  role_arn = aws_iam_role.bedrock_kb_forex_kb.arn
  knowledge_base_configuration {
    vector_knowledge_base_configuration {
      embedding_model_arn = data.aws_bedrock_foundation_model.kb.model_arn
    }
    type = "VECTOR"
  }
  storage_configuration {
    type = "OPENSEARCH_SERVERLESS"
    opensearch_serverless_configuration {
      collection_arn    = aws_opensearchserverless_collection.forex_kb.arn
      vector_index_name = "bedrock-knowledge-base-default-index"
      field_mapping {
        vector_field   = "bedrock-knowledge-base-default-vector"
        text_field     = "AMAZON_BEDROCK_TEXT_CHUNK"
        metadata_field = "AMAZON_BEDROCK_METADATA"
      }
    }
  }
  depends_on = [
    aws_iam_role_policy.bedrock_kb_forex_kb_model,
    aws_iam_role_policy.bedrock_kb_forex_kb_s3,
    opensearch_index.forex_kb,
    time_sleep.aws_iam_role_policy_bedrock_kb_forex_kb_oss
  ]
}

Note that time_sleep.aws_iam_role_policy_bedrock_kb_forex_kb_oss is in the depends_on list - this is how the aforementioned delay is enforced before the knowledge base is created by Terraform.

We also need to add the data source to the knowledge base using the aws_bedrock_data_source resource as follows:

resource "aws_bedrockagent_data_source" "forex_kb" {
  knowledge_base_id = aws_bedrockagent_knowledge_base.forex_kb.id
  name              = "${var.kb_name}DataSource"
  data_source_configuration {
    type = "S3"
    s3_configuration {
      bucket_arn = aws_s3_bucket.forex_kb.arn
    }
  }
}

Voila! We have created a stand-alone Bedrock knowledge base using Terraform! All that remains is to attach the knowledge base to an agent (the forex assistant in our case) to extend the solution.

Integrating the knowledge base and agent resources

For your convenience, you can use the Terraform configuration from the blog post How To Manage an Amazon Bedrock Agent Using Terraform to create the rate assistant. It can be found in the 1_basic directory in this GitHub repository.

Once you incorporate this Terraform configuration with the knowledge base you’ve been developing, we use the new aws_bedrockagent_agent_knowledge_base_association resource to associate the knowledge base with the agent:

resource "aws_bedrockagent_agent_knowledge_base_association" "forex_kb" {
  agent_id             = aws_bedrockagent_agent.forex_asst.id
  description          = file("${path.module}/prompt_templates/kb_instruction.txt")
  knowledge_base_id    = aws_bedrockagent_knowledge_base.forex_kb.id
  knowledge_base_state = "ENABLED"
}

For better organization, we will keep the knowledge base description in a text file called kb_instruction.txt in the prompt_templates folder. The file contains the following text:

Use this knowledge base to retrieve information on foreign currency exchange, such as the FX Global Code.

Lastly, we explained in the previous blog post that the agent must be prepared after changes are made. We used a null_resource to trigger the prepare action, so we will continue to use the same strategy for the knowledge base association by adding an explicit dependency:

resource "null_resource" "forex_asst_prepare" {
  triggers = {
    forex_api_state = sha256(jsonencode(aws_bedrockagent_agent_action_group.forex_api))
    forex_kb_state  = sha256(jsonencode(aws_bedrockagent_knowledge_base.forex_kb))
  }
  provisioner "local-exec" {
    command = "aws bedrock-agent prepare-agent --agent-id ${aws_bedrockagent_agent.forex_asst.id}"
  }
  depends_on = [
    aws_bedrockagent_agent.forex_asst,
    aws_bedrockagent_agent_action_group.forex_api,
    aws_bedrockagent_knowledge_base.forex_kb
  ]
}

Testing the configuration

Now, the moment of truth. We can apply the full Terraform configuration and make sure that it is working properly. My run took several minutes, with the majority of the time spent on creating the OpenSearch Serverless collection. Here is an excerpt of the output for reference:

In the Bedrock console, we can see that the agent ForexAssistant is ready for testing. But we first need to upload the FX Global Code PDF file to the S3 bucket and do a data source sync. For details on these steps, refer to the blog post Adding an Amazon Bedrock Knowledge Base to the Forex Rate Assistant.

Using the test chat interface, I asked:

What is the FX Global Code?

It responded with an explanation that contains citations, indicating that the information was obtained from the knowledge base.

For good measure, we will also ask the forex assistant for an exchange rate:

What is the exchange rate from US Dollar to Canadian Dollar?

It responded with the latest exchange rate as expected:

And that's a wrap! Don't forget to run terraform destroy when you are done, since there is a running cost for the OpenSearch Serverless collection.

✅ For reference, I've dressed up the Terraform solution a bit and checked in the final artifacts to the 2_knowledge_base directory in this repository. Feel free to check it out and use it as the basis for your Bedrock experimentation.

Summary

In this blog post, we developed the Terraform configuration for the knowledge base that enhances the forex rate assistant which we created interactively in the blog post Adding an Amazon Bedrock Knowledge Base to the Forex Rate Assistant. I hope the explanations on key points and solutions to various issues in this blog post help you fast-track your IaC development for Amazon Bedrock solutions.

I will continue to evaluate different features of Amazon Bedrock, such as Guardrails for Amazon Bedrock, and streamlining the data ingestion process for knowledge bases. Please look forward for more helpful content on this topic as well as many others in the Avangards Blog. Happy learning!