Terraform for AWS: how not to get stuck when the provider lacks a datasource?

#terraform #aws #datasource #lambda

Terraform is a great tool. I love it! But when the resource you want to manage is only partially covered, it can get tricky. Today I share a tip on how not to get stuck when Terraform lacks a datasource you wish was there.

A real life use case: list images in an ECR repository.

For a customer project, I needed to list images in an ECR repository (to declare those images as SageMaker custom images). But Terraform, as of today, doesn't have an aws_ecr_images (that would call ListImages) datasource, only aws_ecr_image (that calls DescribeImage if you already know which tag you're looking for).

Being a good Boy Scout, I raised a Pull Request on the Terraform AWS Provider. But as dedicated and nice as the provider's maintainers can be, it might be a while before my PR gets reviewed and merged.

A quick fix: using aws_lambda_invocation!

In the Terraform AWS provider, there is a very convient construct, aws_lambda_invocation, available either as a resource or a datasource, that can actually invoke an AWS Lambda with a user-defined input and collect the output of the Request-Response invocation, so that you can use the response in other resources in your stack.

The code snippet provided below is a fully functional example of how to use it.

	# ECR Image Lister Lambda Function

	# IAM Role for the Lambda function
	resource "aws_iam_role" "ecr_image_lister_role" {
	name = "ecr-image-lister-lambda-role"

	assume_role_policy = jsonencode({
	Version = "2012-10-17"
	Statement = [
	{
	Action = "sts:AssumeRole"
	Effect = "Allow"
	Principal = {
	Service = "lambda.amazonaws.com"
	}
	}
	]
	})
	}

	# IAM Policy for ECR access
	resource "aws_iam_policy" "ecr_image_lister_policy" {
	name = "ecr-image-lister-policy"
	description = "Policy for Lambda to list ECR images"

	policy = jsonencode({
	Version = "2012-10-17"
	Statement = [
	{
	Effect = "Allow"
	Action = [ # Depending on what your Lambda should do.
	"ecr:ListImages",
	]
	Resource = "*"
	},
	{
	Effect = "Allow"
	Action = [
	"logs:CreateLogGroup",
	"logs:CreateLogStream",
	"logs:PutLogEvents"
	]
	Resource = "arn:aws:logs:::*"
	}
	]
	})
	}

	# Attach the policy to the role
	resource "aws_iam_role_policy_attachment" "ecr_image_lister_attachment" {
	role = aws_iam_role.ecr_image_lister_role.name
	policy_arn = aws_iam_policy.ecr_image_lister_policy.arn
	}

	# Lambda function code
	data "archive_file" "ecr_image_lister_code" {
	type = "zip"
	output_path = "${path.module}/ecr_image_lister.zip"

	source {
	content = <<EOF
	import boto3
	import json

	def lambda_handler(event, context):
	# Get the repository name from the event
	repository_name = event.get('repository_name')

	if not repository_name:
	return {
	'statusCode': 400,
	'body': json.dumps('Repository name is required')
	}

	# Create ECR client
	ecr_client = boto3.client('ecr')

	try:
	# List images in the repository
	response = ecr_client.list_images(
	repositoryName=repository_name
	)

	# Extract image IDs
	image_ids = response.get('imageIds', [])

	# Format the response
	result = {
	'repository_name': repository_name,
	'image_count': len(image_ids),
	'images': image_ids
	}

	return {
	'statusCode': 200,
	'body': json.dumps(result)
	}
	except ecr_client.exceptions.RepositoryNotFoundException:
	return {
	'statusCode': 404,
	'body': json.dumps(f'Repository {repository_name} not found')
	}
	except Exception as e:
	return {
	'statusCode': 500,
	'body': json.dumps(f'Error: {str(e)}')
	}
	EOF
	filename = "lambda_function.py"
	}
	}

	# Lambda function
	resource "aws_lambda_function" "ecr_image_lister" {
	function_name = "ecr-image-lister"
	role = aws_iam_role.ecr_image_lister_role.arn
	handler = "lambda_function.lambda_handler"
	runtime = "python3.9"
	filename = data.archive_file.ecr_image_lister_code.output_path
	source_code_hash = data.archive_file.ecr_image_lister_code.output_base64sha256
	timeout = 30

	environment {
	variables = {
	LOG_LEVEL = "INFO"
	}
	}
	}

	# CloudWatch Log Group for Lambda
	resource "aws_cloudwatch_log_group" "ecr_image_lister_logs" {
	name = "/aws/lambda/${aws_lambda_function.ecr_image_lister.function_name}"
	retention_in_days = 14
	}

	# Lambda invocation resource to call the function during apply
	data "aws_lambda_invocation" "ecr_image_lister_invocation" {
	function_name = aws_lambda_function.ecr_image_lister.function_name
	input = jsonencode({
	repository_name = "sample-repository" # Input to send to Lambda. Can come from other resources.
	})

	}

	# Output the result of the Lambda invocation
	output "ecr_image_lister_result" {
	value = jsondecode(aws_lambda_invocation.ecr_image_lister_invocation.result)
	}

	output "ecr_image_lister_result_object" {
	value = jsondecode(jsondecode(aws_lambda_invocation.ecr_image_lister_invocation.result)["body"])
	}

view raw missing-datasource-replacer-lambda.tf hosted with ❤ by GitHub

The datasource is triggered at every plan, while the resource only performs a single invocation, then is never triggered again. The trigger block makes it possible to run it based on a custom condition. The plan will then contain a resource destruction+creation.

That's all, folks!

Build your favorite retro game with Amazon Q Developer CLI in the Challenge & win a T-shirt!

Feeling nostalgic? Build Games Challenge is your chance to recreate your favorite retro arcade style game using Amazon Q Developer’s agentic coding experience in the command line interface, Q Developer CLI.

Participate Now