Forem: Piotr Pabis

Personal token factory: OpenClaw in AWS but Nvidia GB10 at home

Piotr Pabis — Mon, 20 Apr 2026 07:14:17 +0000

Even though Nemotron 3 Super is still free on OpenRouter, the agreement is that you donate all your exchanges to Nvidia for training. A paid version is available also quite cheaply ($0.10/M input, $0.50/M output.) I still decided to utilize my ASUS GX10 (aka DGX Spark aka GB10) as the token source for my agent hosted at AWS. But the trick here is the following: I don't want to open my home network to the outside Internet! Another rule: I don't want to pay for any public IPv4 address to AWS. Will I be able to achieve that? That's actually simple! Let me guide you today how I achieved that setup.

Above you can see that I plan to tunnel both home and AWS networks using Wireguard. I could potentially make this site-to-site but to make it simpler and safer for my home devices, I will only connect to Wireguard server (listener) on AWS side with DGX Spark as a client. I will do it over IPv6. At the bottom you can see that my home devices have normal connectivity to both IPv6 and IPv4 Internet but on AWS side I am relying solely on IPv6 (although internally VPC still uses private IPv4 range). There will be some issues with that but we will fix them later.

Companion GitHub repo: github.com/ppabis/wireguard-openclaw-dgx

Setting up VPC and Wireguard server

I have created a simple VPC using a module with range 10.189.80.0/21 with IPv6 enabled. It has Internet Gateway in public subnets and egress only Internet Gateway for IPv6 outbound connectivity. I disabled NAT gateway on purpose, we will cover this issue later. Instances in public subnet will be reachable over IPv6 from the outside - this is where we will place our Wireguard server. OpenClaw will remain inside the private subnet.

data "aws_availability_zones" "available" {
  state = "available"
}

module "vpc" {
  source  = "terraform-aws-modules/vpc/aws"
  version = "~> 6.6.0"

  name = "my-vpc"
  cidr = "10.189.80.0/21"

  azs             = slice(data.aws_availability_zones.available.names, 0, 3)
  public_subnets  = ["10.189.80.0/24", "10.189.81.0/24", "10.189.82.0/24"]
  private_subnets = ["10.189.83.0/24", "10.189.84.0/24", "10.189.85.0/24"]

  enable_dns_hostnames = true
  enable_dns_support   = true
  enable_nat_gateway   = false

  enable_ipv6                                    = true
  public_subnet_assign_ipv6_address_on_creation  = true
  private_subnet_assign_ipv6_address_on_creation = true
  public_subnet_ipv6_prefixes                    = [0, 1, 2]
  private_subnet_ipv6_prefixes                   = [3, 4, 5]

  private_subnet_tags = { type = "private" }
  public_subnet_tags  = { type = "public" }
}

Creating Wireguard server instance

First of all, I will create an EC2 instance with latest Amazon Linux 2023. It includes WireGuard already in the repositories and the update servers work over IPv6. I will run my WireGuard server on port 51280 so that's what I will open on the security group. All egress should also be open. I will attach am IAM role to it as well, no permissions but it will come handy later. First, define all the smaller components.

data "aws_ssm_parameter" "al2023_arm64" {
  name = "/aws/service/ami-amazon-linux-latest/al2023-ami-kernel-default-arm64"
}

resource "aws_security_group" "wireguard" {
  name_prefix = "wireguard-sg"
  vpc_id      = module.vpc.vpc_id

  ingress {
    from_port        = 51280
    to_port          = 51280
    protocol         = "udp"
    ipv6_cidr_blocks = ["::/0"]
  }

  egress {
    from_port        = 0
    to_port          = 0
    protocol         = "-1"
    cidr_blocks      = ["0.0.0.0/0"]
    ipv6_cidr_blocks = ["::/0"]
  }
}

data "aws_iam_policy_document" "wireguard_assume_role_policy" {
  statement {
    actions = ["sts:AssumeRole"]
    principals {
      type        = "Service"
      identifiers = ["ec2.amazonaws.com"]
    }
  }
}

resource "aws_iam_role" "wireguard" {
  name               = "wg-ec2-role"
  assume_role_policy = data.aws_iam_policy_document.wireguard_assume_role_policy.json
}

resource "aws_iam_instance_profile" "wireguard" {
  name = "wg-ec2-profile"
  role = aws_iam_role.wireguard.name
}

And finally we can define the EC2 instance. It will be the cheapest and smallest possible one I can find, which is t4g.nano. AMI ID is pulled from publicly shared SSM parameter (easier than AMI data source in Terraform). From networking I'm disabling public IPv4 assignment, forcing at least one IPv6 and disabling source destination check (more on that later). I didn't define any SSH connectivity, nor EC2 Instance Connect or Session Manager. The cheapest option here would be to define a key pair and open 22 on IPv6 to your subnet if you need debugging.

resource "aws_instance" "wireguard" {
  ami                         = data.aws_ssm_parameter.al2023_arm64.value
  instance_type               = "t4g.nano"
  iam_instance_profile        = aws_iam_instance_profile.wireguard.name
  subnet_id                   = module.vpc.public_subnets[0]
  vpc_security_group_ids      = [aws_security_group.wireguard.id]
  associate_public_ip_address = false
  ipv6_address_count          = 1
  source_dest_check           = false

  user_data = local.user_data

  tags = { Name = "Wireguard" }
  lifecycle { ignore_changes = [ami] }
}

output "ipv6" {
  value = aws_instance.wireguard.ipv6_addresses[0]
}

Installing and configuring Wireguard on the server

How would we install the server if we have no SSH or any other shell access to the instance? As you see above, I have defined user data. This is a script that runs on first instance boot... if you just use pure Bash. For our use case, we want to be able to control instance contents dynamically. For that we will use cloud-init, which allows for more flexibility regarding user data contents. We will start with the draft defining "attachments" with both cloud-init's cloud-config (YAML) as well as standard Bash script that starts on boot. I'm writing this in a new file called user-data.yaml (although it's not a valid YAML file).

Content-Type: multipart/mixed; boundary="//"
MIME-Version: 1.0

--//
Content-Type: text/cloud-config; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment; filename="cloud-config.txt"

#cloud-config
package_update: true

--//
Content-Type: text/x-shellscript; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment; filename="userdata.txt"

#!/bin/bash
echo "Configuring WireGuard..."
--//--

Side note

If you know cloud-init, you might wonder why I want to use also user data. For some reason runcmd doesn't always execute when I want it to and bootcmd happens too early.

As you see we have two sections in this file. First, we have defined cloud-config that will just update packages on startup. Second script will also execute on first boot and just echo a message. Now what we can do is to configure each module to run on every restart. In cloud-config add this (insert between boundaries):

#cloud-config
package_update: true
cloud_final_modules:
  - [scripts-user, always]

cloud_config_modules:
  - [write_files, always]
  - [package_update_upgrade_install, always]

Now all the sections (that we define soon) will run on every instance reboot. This will give us some flexibility in changing the configuration (although requiring reboot but how often do you plan to change this 😄). Let us install required packages: Wireguard itself, iptables for routing capabilities and for convenience if you need to debug, tmux and htop can come in handy.

packages:
  - wireguard-tools
  - iptables-nft
  - htop
  - tmux

Now let's proceed with Wireguard configuration. As previously stated, I want to use 10.155.222.0/24 as the subnet and listen on port 51280. The private key will be a placeholder for safety reasons. When the tunnel is brought up, we are going to enable IP forwarding in the kernel and allow forwards between wg0 Wireguard's interface and ens5 (primary network card in AL2023 at least). The router address (sever) will be the first one in the subnet 10.155.222.1.

write_files:
  - path: /etc/wireguard/wg0.conf
    content: |-
      [Interface]
      Address    = 10.155.222.1/24
      ListenPort = 51280
      PrivateKey = _PRIVATE_KEY_

      # Enable routing + NAT for WG clients to reach VPC
      PostUp   = sysctl -w net.ipv4.ip_forward=1
      PostUp   = iptables -A FORWARD -i wg0 -o ens5 -j ACCEPT
      PostUp   = iptables -A FORWARD -i ens5 -o wg0 -j ACCEPT

      PostDown = iptables -D FORWARD -i wg0 -o ens5 -j ACCEPT
      PostDown = iptables -D FORWARD -i ens5 -o wg0 -j ACCEPT
      # End of file

This configuration is not yet usable until we create another part of the user data attachment. In the Bash script that is going to run on every machine startup, we are going to generate Wireguard's key pair, if it doesn't exist, and if it does, we will just replace the _PRIVATE_KEY_ placeholder with sed. But that's not all! We also need the public key of the server after all to connect. As I don't want to need any SSH-like connectivity to this server, it will export the public key to AWS Systems Manager Parameter Store.

--//
Content-Type: text/x-shellscript; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment; filename="userdata.txt"

#!/bin/bash
set -eo pipefail

echo "Configuring WireGuard..."
# If there's no private key, generate the private key into a file and derive
# the public one also into a file.
if [ ! -f /etc/wireguard/private.key ]; then
  wg genkey | tee /etc/wireguard/private.key | wg pubkey > /etc/wireguard/public.key
fi

# Replace the private key placeholder if it exists in wg0.conf
sed -i "s#_PRIVATE_KEY_#$(cat /etc/wireguard/private.key)#g" /etc/wireguard/wg0.conf

# Export the public key to SSM Parameter Store. Use IPv6 endpoint.
echo "Public key: $(cat /etc/wireguard/public.key)"
export AWS_USE_DUALSTACK_ENDPOINT=true
aws ssm put-parameter --type "String" --overwrite \
 --name "/wireguard/public-key" \
 --value "$(cat /etc/wireguard/public.key)"

# Start the tunnel.
wg-quick up wg0

--//--

Currently if you run the script, the machine will not be able to write into Parameter Store because the IAM role doesn't have such permissions. Add the following policy to the previously defined role.

data "aws_caller_identity" "X" {}
data "aws_region" "X" {}

data "aws_iam_policy_document" "wireguard_ssm_policy" {
  statement {
    actions   = ["ssm:PutParameter"]
    resources = ["arn:aws:ssm:${data.aws_region.X.region}:${data.aws_caller_identity.X.account_id}:parameter/wireguard/public-key"]
  }
}

resource "aws_iam_role_policy" "wireguard_ssm_policy" {
  name   = "wg-ssm-policy"
  role   = aws_iam_role.wireguard.id
  policy = data.aws_iam_policy_document.wireguard_ssm_policy.json
}

As the last part, add the user data local variable. I will use templatefile because we are going to do some dynamic things later, so it will come in handy.

locals {
  user_data = templatefile("${path.module}/user-data.yaml", {})
}

When you now deploy this infrastructure, after some minutes you should get a public key in SSM Parameter Store under wireguard/public-key. You can use it to configure the client connections. You can also use the following command to get the public key value.

aws ssm get-parameter \
 --name /wireguard/public-key \
 --query Parameter.Value \
 --output text

Setting up client

On the client side, so my DGX Spark, I will now SSH and also install Wireguard. I am using default Ubuntu installation. We are going to generate a new private key, get the public key and save the configuration. You need to run the following commands under root.

apt -y install wireguard wireguard-tools

Define some variables that are public key from SSM Parameter store and output with IPv6 from Terraform. Also configure the path where you want to keep the configuration. I will use wg1 interface.

WG_SERVER_KEY="sJQTQ7sytRuLBK74Y24TNtXrHqrmpRb+ZsT9olMfXQ4=" # Key from SSM
WG_SERVER_IP="2a05:d012:e21:2345:6789:01ab:cdef:9dd7" # IPv6 from AWS
mkdir -p /etc/wireguard/
WG_CONFIG=/etc/wireguard/wg1.conf
WG_PRIVATE_KEY=$(wg genkey)
WG_PUBLIC_KEY=$(echo $WG_PRIVATE_KEY | wg pubkey)
echo "Public key = $WG_PUBLIC_KEY"

Then generate the configuration. In case you have some firewall, select a port that you want to use for listening on incoming VPN traffic. Choose a unique address from the VPN subnet pool. Optionally set the DNS resolver to the AWS VPC one (second address of subnet's CIDR). AllowedIPs is an unfortunate name but these are the routes that should go through the VPN - I set them to VPC CIDR and VPN's internal subnet.

cat > $WG_CONFIG <<EOF
[Interface]
PrivateKey = $WG_PRIVATE_KEY
# Public Key: $WG_PUBLIC_KEY
Address = 10.155.222.3/32
DNS = 10.189.80.2    # Optional
ListenPort = 62910   # You can skip this if you don't have firewall

[Peer]
PublicKey = $WG_SERVER_KEY
AllowedIPs = 10.189.80.0/21, 10.155.222.0/24 # Connectivity to VPC and VPN
PersistentKeepalive = 25
Endpoint = [$WG_SERVER_IP]:51280

EOF

Allowing new client on the Wireguard server

As you get the new public key for your home client, we need to now enable it on the Wireguard server. As you remember we used templatefile to load the user data. This will come useful now as we will be able to configure multiple clients. Let's revisit the write_files section. Modify the end of file, after iptables commands.

write_files:
  - path: /etc/wireguard/wg0.conf
    content: |-
      [Interface]
      Address    = 10.155.222.1/24
      ListenPort = 51280
      PrivateKey = _PRIVATE_KEY_

      # Enable routing + NAT for WG clients to reach VPC
      PostUp   = sysctl -w net.ipv4.ip_forward=1
      PostUp   = iptables -A FORWARD -i wg0 -o ens5 -j ACCEPT
      PostUp   = iptables -A FORWARD -i ens5 -o wg0 -j ACCEPT

      PostDown = iptables -D FORWARD -i wg0 -o ens5 -j ACCEPT
      PostDown = iptables -D FORWARD -i ens5 -o wg0 -j ACCEPT

      %{~ for peer in peers ~}
      [Peer]
      PublicKey = ${peer.public_key}
      AllowedIPs = ${peer.address}/32
      %{~ endfor ~}

      # End of file

The above for loop will generate multiple clients on the Wireguard server. Now in the template variables you have to set peers map with public_key and address keys. Revisit the locals in EC2 instance. Applying this will reboot the instance.

locals {
  user_data = templatefile("user-data.yaml", {
    peers = [
      {
        address    = "10.155.222.3", # the IP you selected for the client
        public_key = "bxmMoVvXlVVRg7uaTnxI6Vf7wxeI0XWj5d6zREqDkzk=" # the public key of the client
      }
    ]
  })
}

Now if you bring up the there should be a status about latest handshake and some data that is received.

$ sudo wg-quick up wg1
[#] ip link add wg1 type wireguard
[#] wg setconf wg1 /dev/fd/63
[#] ip -4 address add 10.155.222.3/32 dev wg1
[#] ip link set mtu 1420 up dev wg1
[#] ip -4 route add 10.155.222.0/24 dev wg1
[#] ip -4 route add 10.189.80.0/21 dev wg1

$ sudo wg
interface: wg1
  public key: bxmMoVvXlVVRg7uaTnxI6Vf7wxeI0XWj5d6zREqDkzk=
  private key: (hidden)
  listening port: 62910

peer: sJQTQ7sytRuLBK74Y24TNtXrHqrmpRb+ZsT9olMfXQ4=
  endpoint: [2a05:d012:e21:2345:6789:01ab:cdef:9dd7]:51280
  allowed ips: 10.189.80.0/21, 10.155.222.0/24
  latest handshake: 22 seconds ago
  transfer: 92 B received, 180 B sent
  persistent keepalive: every 25 seconds

I have created a test internal application load balancer. It will listen for CIDR 10.155.222.0 (not VPC CIDR!) on port 80. But before this can be used you also have to define routes for Wireguard's subnet. This is the reason for turning off source-destination check on the network card of the EC2 instance. Without that feature, packets destined for Wireguard (whether requests or responses) will be accepted by the EC2 instance even if the destination isn't any of the instance's IPs.

resource "aws_route" "wireguard_tunnel_prefix" {
  for_each = toset(
    concat(
      module.vpc.private_route_table_ids,
      module.vpc.public_route_table_ids
    )
  )

  route_table_id         = each.value
  destination_cidr_block = "10.155.222.0/24"
  network_interface_id   = aws_instance.wireguard.primary_network_interface_id
}

Afterwards I tested with cURL and the connectivity was established! The test IPs are private range as you see below. I even tried DNS and it was also functional through the Wireguard interface - that way we can later set up some private domains or use Cloud Map.

$ curl http://internal-mytest-alb-1234567890.eu-west-3.elb.amazonaws.com
hello world

$ dig +short internal-mytest-alb-1234567890.eu-west-3.elb.amazonaws.com
10.189.83.70
10.189.85.144

$ resolvectl query internal-mytest-alb-1234567890.eu-west-3.elb.amazonaws.com
internal-mytest-alb-1234567890.eu-west-3.elb.amazonaws.com: 10.189.83.70 -- link: wg1
                                                            10.189.85.144 -- link: wg1

Installing LLM server

Now we need to install and configure Ollama. Just follow the instructions on ollama.com to install it, or you can use any other LLM server you wish. Be sure that it is listening on all addresses and not just local host. Create the following override in SystemD (on Ubuntu).

sudo cat > /etc/systemd/system/ollama.service.d/override.conf <<EOF
[Service]
Environment="OLLAMA_HOST=0.0.0.0:11434"
Environment="OLLAMA_CONTEXT_LENGTH=212000"
Environment="OLLAMA_KV_CACHE_TYPE=q8_0"
Environment="OLLAMA_KEEP_ALIVE=2400"
EOF
sudo systemctl enable --now ollama
sudo systemctl restart ollama

You can alternatively run it in Docker. It should be supported on DGX Spark out of the box to use it with the GPU. Choose only one or the other because they occupy the same port in the command below!

docker run -d \
 --gpus=all \
 -v ollama:/root/.ollama \
 -p 11434:11434 \
 --name ollama \
 --restart=unless-stopped \
 ollama/ollama

For direct installations, you can just use ollama pull <model> to download the model in advance. I will use Nvidia's Nemotron 3 Super which was one of the best medium-sized models when I started writing this post. However, Gemma 4 and Qwen 3.6 were also released so you can experiment with that.

ollama pull nemotron-3-super:120b-a12b-q4_K_M

Connecting from an EC2 instance

I will create another EC2 instance which will be used for OpenClaw or any other system as you want such as Hermes or just a web app for chatting. I will prepare the instance first, will use Ubuntu 24.04 and t4g.medium instance. I will also create a new user data script that will bootstrap some of the required packages. As we have IPv6 outbound connectivity from private subnet, APT repositories should work without issues. I will also enable SSH access from Wireguard's inner subnet so that I can SSH to the instance, but you can alternatively use IPv6 when moving to public subnet or via SSM Systems Manager if you configure it, it's up to you.

data "aws_ssm_parameter" "ubuntu_2404" {
  name = "/aws/service/canonical/ubuntu/server/24.04/stable/current/arm64/hvm/ebs-gp3/ami-id"
}

resource "aws_security_group" "agent" {
  name        = "openclaw-agent-sg"
  vpc_id      = data.aws_subnet.agent.vpc_id

  ingress {
    from_port   = 22
    to_port     = 22
    protocol    = "tcp"
    cidr_blocks = ["10.155.222.0/24"]
  }

  egress {
    from_port        = 0
    to_port          = 0
    protocol         = "-1"
    cidr_blocks      = ["0.0.0.0/0"]
    ipv6_cidr_blocks = ["::/0"]
  }
}

resource "aws_instance" "agent" {
  ami           = data.aws_ssm_parameter.ubuntu_2404.value
  instance_type = "t4g.medium"
  user_data     = file("openclaw.yaml")
  tags          = { Name = "openclaw-agent" }

  subnet_id                   = module.vpc.private_subnets[0]
  vpc_security_group_ids      = [aws_security_group.agent.id]
  associate_public_ip_address = false
  ipv6_address_count          = 1

  metadata_options {
    http_endpoint = "enabled"
    http_tokens   = "required"
  }

  root_block_device {
    volume_size           = 20
    volume_type           = "gp3"
    encrypted             = true
    delete_on_termination = true
  }

  lifecycle { ignore_changes = [ami] }
}

output "private_ip" {
  value = aws_instance.agent.private_ip
}

From the system config I will do the following:

add Node.js APT repository of version 24,
install Node.js and unattended upgrades,
enable unattended upgrades,
enable AWS SSM (optional but useful).

I will also create a separate user for OpenClaw so that it can have its own home directory and permissions. It is also very important to create the default user as this will allow you to SSH to the instance to onboard OpenClaw. If you wish you can also specify SSH keys in here or via AWS key pairs.

#cloud-config
cloud_final_modules:
  - [scripts-user, always]

cloud_config_modules:
  - [write_files, always]
  - [apt_configure, always]
  - [package_update_upgrade_install, always]

hostname: openclaw-agent
create_hostname_file: true

apt:
  sources:
    nodejs:
      keyid: 2F59B5F99B1BE0B4
      keyserver: keyserver.ubuntu.com
      source: deb [signed-by=$KEY_FILE] https://deb.nodesource.com/node_24.x nodistro main

package_update: true
package_upgrade: true
packages:
  - nodejs
  - unattended-upgrades

users:
  - default
  - name: openclaw
    uid: 2200

ssh_authorized_keys:
  - ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIPgVNNOeuUqMgobgeIIkndXXYekOmC/e5bqty3f0UXDa my-ssh-key

write_files:
  - path: /etc/apt/apt.conf.d/20auto-upgrades
    permissions: "0644"
    owner: root:root
    content: |
      APT::Periodic::Update-Package-Lists "1";
      APT::Periodic::Unattended-Upgrade "1";

runcmd:
  - systemctl enable --now unattended-upgrades || true
  - loginctl enable-linger openclaw || true # Enable SystemD on openclaw's user

Installing OpenClaw - with a caveat 😳

I SSH'd into the instance and Nodejs should already be there based on the provided user data. So I switched user to the new openclaw one I defined and started installation with NPM.

ssh ubuntu@$(tofu output -raw private_ip) # replace with your private IP

The authenticity of host '10.189.83.247 (10.189.83.247)' can't be established.
ED25519 key fingerprint is: SHA256:iMfJBU8iSEc5ikspbNKGD8jCAlLGwrOs28lbI4aPw2Q
This key is not known by any other names.
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes

# Inside the machine
sudo su openclaw -l -s /bin/bash
npm install openclaw@latest

...
4564 error A git connection error occurred
4565 error command git --no-replace-objects ls-remote ssh://git@github.com/whiskeysockets/libsignal-node.git
4566 error ssh: connect to host github.com port 22: Connection timed out
4566 error fatal: Could not read from remote repository.

This timeout happens because we don't have public IPv4 connectivity! There are two standard solutions - public IP for the instance or NAT Gateway or...

Hosting tinyproxy on DGX Spark

As we already have connection to other VPN places we can simply use one of the machines on the network as the exit to IPv4 internet. As a bonus we retain our residential IP! So let's spin it up in a Docker but for that we need to make some configuration first in Dockerfile and tinyproxy.conf.

FROM alpine:latest

RUN apk add tinyproxy
COPY tinyproxy.conf /etc/tinyproxy/tinyproxy.conf
EXPOSE 8888
ENTRYPOINT ["/usr/bin/tinyproxy"]
CMD ["-d"]

Port 8888
Timeout 600
MaxClients 100
ViaProxyName "tinyproxy"

User nobody
Group nobody

DefaultErrorFile "/usr/share/tinyproxy/default.html"
StatFile "/usr/share/tinyproxy/stats.html"
LogLevel Info

Allow 127.0.0.1
Allow ::1
Allow 10.155.222.0/24
Allow 10.189.80.0/21

From that config we can easily build the new image for Tinyproxy and set it up so that it starts on boot. Of course then all this connectivity will rely on our local machine being up, so it's only usable for some of IPv4 requirements such as GitHub.

docker build -t local-tinyproxy:latest .
docker run -d \
 --name tinyproxy \
 --restart=unless-stopped \
 -p 8989:8888 \
 local-tinyproxy:latest

Now we can easily direct the proxy to the new service within Wireguard's network. However, in order to do this, we need to open the port 8989 (and 11434 for Ollama) on Wireguard's instance. You might ask why is that? So any packet sent from OpenClaw's instance in that direction will look like this:

source address: 10.189.83.247 (example),
destination address: 10.155.222.3,
source port: 59123 (example),
destination port: 8989.

Filtering on AWS security group level cares about source address and destination port rather than anything else. Destination address is taken care by "source-destination" check of the network interface - the feature we just disabled. Let's update our security group.

resource "aws_security_group" "wireguard" {
  name_prefix = "wireguard-sg"
  vpc_id      = module.vpc.vpc_id

  ingress {
    from_port        = 51280
    to_port          = 51280
    protocol         = "udp"
    ipv6_cidr_blocks = ["::/0"]
  }

  ingress {
    from_port        = 11434
    to_port          = 11434
    protocol         = "tcp"
    security_groups  = [aws_security_group.agent]
  }

  ingress {
    from_port        = 8989
    to_port          = 8989
    protocol         = "tcp"
    security_groups  = [aws_security_group.agent]
  }

  egress {
    from_port        = 0
    to_port          = 0
    protocol         = "-1"
    cidr_blocks      = ["0.0.0.0/0"]
    ipv6_cidr_blocks = ["::/0"]
  }
}

Now you can configure NPM and Git to use the proxy and OpenClaw should be able to install with no issues. You can also test the connectivity with cURL even, if it responds with 500, this is fine; if 403, this might be a problem with tinyproxy.conf. If this cURL command shows timeout or "couldn't connect to server", this can be security groups, routes or other firewall.

curl http://10.155.222.3:8989 -X CONNECT | grep title
# Test results
# <title>500 Unable to connect</title>
npm set https-proxy=http://10.155.222.3:8989
npm set proxy=http://10.155.222.3:8989
git config --global http.proxy http://10.155.222.3:8989
git config --global https.proxy http://10.155.222.3:8989
npm install openclaw@latest

OpenClaw - Onboard!

And we are almost done! The only thing we now need is to follow the onboarding process. Use Ollama provider in local mode, set the correct DGX's IP over Wireguard and choose the model from the list!

export XDG_RUNTIME_DIR=/run/user/$(id -u) # for systemd support
npx openclaw onboard

I will demonstrate usage via TUI rather than any instant messenger here. The first load time for the latest build of OpenClaw took around 1:30 minutes with Nemotron 3 Super (q4). After resetting the session, first message took around 20 seconds (to load 12k context), so most of the time was loading the model into VRAM. Keeping model in memory is controllable on Ollama's side. For each subsequent message, there's some more time needed.

npx openclaw tui

To have some performance comparison, I decided to also try running the agent on latest Qwen 3.6 35B (qwen3.6:35b-a3b-q8_0). The startup took around 30 seconds and each message takes maybe 10.

Power usages

I decided to also order a meter that provided how much power the DGX machine draws in different situations. When it's completely idle, it takes around 30W, with the model loaded to memory but unused it's around 40W and during response generation it oscillates around 170W. Let's do some assumptions - when you sleep you don't use OpenClaw at all but you keep DGX Spark on, so it takes 30W for 7 hours. You are a very heavy user, writing to OpenClaw all day, scheduling a lot of tasks basically treating it as a thinking extension which totals to 8 hours of pure generative work. For all the other time it just sits idle but loaded to memory.

7 h * 30 W = 210 Wh
8 h * 170 W = 1360 Wh
9 h * 40 W = 360 Wh
in total it is about 2 kWh
assuming price in Germany is 0.5€ for a kilowatt-hour, it is 1€ per day, 30€ per month, just for pure token generation.

Obviously you also have to consider costs for AWS EC2 Instance. For the VPN server, if you commit for a year to EC2 saving plans, you will pay around 25€, for the agent instance, if it's online all year round this is 200€ (but note that OpenClaw is especially heavy compared to other harnesses).

If you use small model like Qwen 35B in Q8, you will still have space in VRAM for some image generator like Z-Image-Turbo, a small TTS model or Whisper Turbo for speech recognition. I managed to easily fit image generation and VibeVoice ASR along with the chatbot model in 100 gigs of RAM.

Cons of this setup

Of course each of such setups comes with a tradeoff. You keep your privacy, you maybe pay less for inference (that's debatable) but that's about it. The best setup is to have GB10 as the primary model provider and fall back on something small on OpenRouter. There's always a possibility to mix multiple models and providers using subagents. For example you want to perform a coding task, you use Sonnet 4.6 but you keep local Qwen 3.6 for orchestration. Such setup is far from highly available. Not only can the machine break, you can have blackout or Internet can be down in your flat. Another tradeoff is speed - most of the medium sized models from OpenAI or Anthropic will nevertheless run faster than any decent model on DGX Spark.

The prefill (loading context) speed is 500 tokens per second for Nemotron 3 Super and token generation is 20 tokens per second. Assuming that in each 15 minute window we have to load 200k context, we can generate 10k tokens, that makes it (within 8 hour daily generation) 6.4 M tokens input and 0.32 M tokens output. Prompt prefill accounts for around 40% of the time spent so from a daily spend, 40 cents will go to input tokens and 60 cents to output tokens. Normalized this is around 0.06€ per million input tokens and 1.88€ per million output tokens.

For Qwen 3.6 the speeds look different: 1150 tokens/s prefill and 39 tokens/s generation. Then in 15 minute window (200k input context) the prefill will account for 20% of the generation time. 80% of the time left will be for token generation and this will produce 28k tokens. So within a day we get 6.4 M input and 0.896 M output tokens, normalized this makes it 0.03€/Mtok input and 0.89€/Mtok output.

Is any of these competitive? It highly depends on your use case and usage patterns; whatever I did above is just napkin maths. Gemma 4 31B over OpenRouter is just $0.13/$0.38 in/out but GPT-5.4 Nano is $0.20/$1.25.

IT Support Agent on AWS Bedrock - Implementing Guardrails

Piotr Pabis — Thu, 29 Jan 2026 16:06:33 +0000

As I mentioned in the previous post where we have created JIRA ticketing action groups, that I managed to create a ticket that was insulting the IT team. As an IT team, I would like to not be insulted, right? So today we are going to implements Guardrails. But wait, there's more! The database tickets I were supposed to create, could be set to lose access at any date, in the past or in the far future (post-AGI). I will explain to you along the way.

Part 1 - Confluence Knowledge Base Part 2 - JIRA ticket action group GitHub: ppabis/it-agent-bedrock

Guardrail #1 - Insults and profanity

The first Guardrail I would like to implement is the one that will prevent users from posting inappropriate content into the tickets. I will use only input filtering as maybe in the future I would implement another "Action" that will retrieve the tickets and I don't want to be stopped by the guardrail this time. In Terraform code this would look something like this:

resource "aws_bedrockagent_guardrail" "filtering" {
  name        = "ticket-agent-guardrail"
  description = "Input-only guardrail for toxic language and disallowed topics."

  blocked_input_messaging = "Your input includes content that is not allowed to pass further. Please rephrase your message."
  blocked_outputs_message = "Response blocked by guardrail."

  content_policy_config {
    filters_config {
      type            = "HATE"
      input_strength  = "MEDIUM"
      output_strength = "NONE"
    }

    filters_config {
      type            = "INSULTS"
      input_strength  = "MEDIUM"
      output_strength = "NONE"
    }

    filters_config {
      type            = "VIOLENCE"
      input_strength  = "MEDIUM"
      output_strength = "NONE"
    }

    filters_config {
      type            = "SEXUAL"
      input_strength  = "MEDIUM"
      output_strength = "NONE"
    }
  }
}

Guardrail #2 - Financial and hacking topics

I would also like the agent to not help the user hack into the company systems based for example on the knowledge base. It is possible that such Confluence wiki might have some information that can be very useful for malicious actors. What is more, if I plan to equip the agent with some more tools in the future, such as getting current state of the applications or the infrastructure, it could make it even worse. Let's extend the above guardrail.

resource "aws_bedrockagent_guardrail" "filtering" {
  name        = "ticket-agent-guardrail"
  description = "Input-only guardrail for toxic language and disallowed topics."

  // ...

  topic_policy_config {
    topics_config {
      name       = "Financial advice"
      definition = "Requests for personal investment, trading, or financial guidance."
      examples = [
        "Should I buy crypto?",
        "Recommend stocks for a quick profit.",
        "Should I take a loan to buy a house?",
        "Should I take a loan to buy a car?",
        "Should I consolidate my debt?"
      ]
      type = "DENY"
    }

    topics_config {
      name       = "Unauthorized system access"
      definition = "Requests to get information about system vulnerabilities."
      examples = [
        "Which instances have SSH open?",
        "How to increase my privileges without submitting a ticket?"
      ]
      type = "DENY"
    }
  }
}

Guardrail #3 - PII redaction

I really don't want to deal with leaked PII into our company JIRA boards. Let's say one of the employees is not aware of the risks of writing down credit card details (mostly number). I would like to prevent this from happening into our kanban board.

resource "aws_bedrockagent_guardrail" "filtering" {
  name        = "ticket-agent-guardrail"
  description = "Input-only guardrail for toxic language, disallowed topics and sensitive inputs."
  // ...
  sensitive_information_policy_config {
    pii_entities_config {
      type           = "INTERNATIONAL_BANK_ACCOUNT_NUMBER"
      action         = "ANONYMIZE"
      input_enabled  = true
      output_enabled = true
      input_action = "ANONYMIZE"
      output_action = "ANONYMIZE"
    }

    pii_entities_config {
      type           = "CREDIT_DEBIT_CARD_CVV"
      action         = "ANONYMIZE"
      input_enabled  = true
      output_enabled = true
      input_action = "ANONYMIZE"
      output_action = "ANONYMIZE"
    }

    pii_entities_config {
      type           = "CREDIT_DEBIT_CARD_NUMBER"
      action         = "ANONYMIZE"
      input_enabled  = true
      output_enabled = true
      input_action = "ANONYMIZE"
      output_action = "ANONYMIZE"
    }

    pii_entities_config {
      type           = "AWS_SECRET_KEY"
      action         = "ANONYMIZE"
      input_enabled  = true
      output_enabled = true
      input_action = "ANONYMIZE"
      output_action = "ANONYMIZE"
    }
  }
}

Guardrails Cross-Region inference

For some reason, Guardrails are also offered as an Inference Profile of some sort. Thus I would like to use it. As I set up the Agent in Frankfurt, the only cross-region profile I can use is across-EU. I didn't find any way to list the inference profiles for guardrails, so you have to rely on this hardcoded ARN taken from the console.

resource "aws_bedrockagent_guardrail" "filtering" {
  name        = "ticket-agent-guardrail"
  description = "Input-only guardrail for toxic language, disallowed topics and sensitive inputs."
  // ...
  cross_region_config {
    guardrail_profile_identifier = "arn:aws:bedrock:${data.aws_region.current.name}:${data.aws_caller_identity.current.account_id}:guardrail-profile/eu.guardrail.v1:0"
  }
}

Lastly, I would like to publish a version of this guardrail. As everything in Bedrock, almost each resource is versioned, so are guardrails. The use case is that you want to test a change to it on some development agent before publishing it to production (AWS doesn't know GitFlow and multi-account strategies 😂).

resource "aws_bedrock_guardrail_version" "filtering" {
  guardrail_arn = aws_bedrock_guardrail.filtering.guardrail_arn
}

Before we mount the guardrail to the Agent, let's test it first with some prompts. I have already tested the insults section and for some reason, the word "bastards" is not picked up even on high sensitivity, thus I had to increase the strength 😅. In testing some responses are blocked because this is just a raw model without Knowledge base or action groups, so it hallucinates random stuff that might contain some "how to get access" hints.

I will now try to apply the guardrails to our agent and somewhat test a bit how is the flow working with it. Maybe the "hacking" ban will be too strict? Who knows. What is also important is to allow the agent's IAM role to use the guardrail.

data "aws_iam_policy_document" "agent_policy" {
  statement {
    effect = "Allow"
    actions = [
      "bedrock:InvokeModel",
      "bedrock:InvokeModel*",
      "bedrock:GetInferenceProfile",
      "bedrock:ListInferenceProfiles",
      "bedrock:ListTagsForResource",
      "bedrock:GetKnowledgeBase",
      "bedrock:ListKnowledgeBases",
      "bedrock:Retrieve",
      "bedrock:RetrieveAndGenerate",
      "bedrock:ApplyGuardrail",
      "bedrock:GetGuardrail"
    ]
    resources = ["*"]
  }
  // ... continues
}

resource "aws_bedrockagent_agent" "ticketagent" {
  agent_name              = "ticketagent"
  foundation_model        = local.inference_profile_arn
  agent_resource_role_arn = aws_iam_role.agent_role.arn
  depends_on              = [aws_bedrockagent_knowledge_base.confluence]
  prepare_agent           = true
  instruction             = local.prompts.agent_prompt

  guardrail_configuration {
    guardrail_identifier = aws_bedrock_guardrail_version.filtering.guardrail_arn
    guardrail_version    = aws_bedrock_guardrail_version.filtering.version
  }
}

Testing the casual guardrails

I performed some tests after applying the guardrail to the agent. In both cases, the success is dependent on the mood of the agent's model as well as guardrail 😅. Sometimes single words like "database access, my manager will review the permissions" would trip the "hacking" topic, but all in all it looked pretty well. The only thing that turns out doesn't work the way I wanted it to is PII masking. I was sure that when I send a message with credit card number, the input will be filtered and the ticked would contain some redacted fields.

No 🙅.

What AWS does in this case is to simply echo your message with replaced fields, like if it was coming from the model itself. This is crazy and unexpected. I was sure that it will just clean the message on my input and pass it to the model.

One thing that I sill need to change in here is that the date for any access can be arbitrarily long. I can do it in code but there's another feature I would like to try out...

Automated Reasoning checks

It's possible to define the reasoning using JSON and SMT-LIB expressions but that's absolutely crazy to do. What I will use is to write a natural language document, use AWS Console to generate the reasoning tests and import it later with OpenTofu.

Just input set of rules for your agent (such as company guidelines) and wait until it generates. I for example gave the rules to never request access tickets that are longer than 2 years as well as tell agent to always create a ticket for IT on hardware issues instead of guiding the user to fix it themselves.

After generation perform manual tests with the button "Generate tests" and answer some questions with thumbs up or thumbs down. This will provide more tests that will check if the ruleset is correct. Create some annotations to update the rules and create more tests.

Then in OpenTofu add awscc provider, configure it for your region, create a new file import.tf with the following content:

import {
    to = awscc_bedrock_automated_reasoning_policy.it_agent_policy
    id = "arn:aws:bedrock:eu-central-1:123456789012:automated-reasoning-policy/abcdef1234"
}

Then run tofu plan -generate-config-out=arc.tf. A new file arc.tf will be created but it is not complete unfortunately. Using AWS CLI you can fetch missing IDs and ask AI in your IDE to fix it.

POLICY_ARN=arn:aws:bedrock:eu-central-1:123456789012:automated-reasoning-policy/abcdef1234
aws cloudcontrol get-resource \
 --type-name 'AWS::Bedrock::AutomatedReasoningPolicy' \
 --identifier $POLICY_ARN \
 --output text \
 --query ResourceDescription.Properties \
 | jq '.PolicyDefinition.Rules[] | {id: .Id, expression: .Expression}'

Try first apply to import the changes. They will fail. You have to unfortunately recreate the whole policy with IaC if you want to have it in the Terraform state. Proceed to AWS Console to delete the policy (along with the tests).

Now remember to set force_delete = true so that it's easy to delete the policy later on. We also need to create a version of it. Create this block to save a version of reasoning policy.

resource "awscc_bedrock_automated_reasoning_policy_version" "it_agent_policy" {
  policy_arn                   = awscc_bedrock_automated_reasoning_policy.it_agent_policy.policy_arn
  last_updated_definition_hash = awscc_bedrock_automated_reasoning_policy.it_agent_policy.definition_hash
}

The problems emerge

I wanted here to write a simple instruction how to connect the reasoning checks to the guardrail and test it. However, several problems appeared out of nowhere:

aws_bedrock_guardrail doesn't have a field to add ARC,
After simple conversion of aws_bedrock_guardrail to awscc_bedrock_guardrail can't be applied due to a bug I found,
Even after doing everything manually, the checks are not applied to the guardrail when interacting with the agent ¯\(ツ)/¯. It always either passes or says it's too complex,
Bedrock is a "Production Ready Product™️" 🙄 which means AWS "Best Practices™️" means doing everything in the Console using mouse.

Testing ARC

So I will try some prompts that definitely go against the policies defined for reasoning. I will first try to create a normal database ticket, then another one with some date in the far future. Next, I will to troubleshoot my company laptop.

![Help me fix the laptop]https://dev-to-uploads.s3.amazonaws.com/uploads/articles/3o8qhekve1cg9rfqjbe6.jpg)

As you see the checks do not do anything to prevent me from going further. The database tickets are simply created for 2035 and the model tries to look for information in the knowledge base for hardware fixes (even though it suggests creating a ticket, this is simply its own creativity, on other tries it just simply says there's nothing in KB to help me).

Therefore, I decided to revert any commit related to Automated Reasoning Checks.

IT Support Agent on AWS Bedrock - JIRA Ticket tool

Piotr Pabis — Tue, 27 Jan 2026 15:26:02 +0000

Today we are going to create the agent that will use our previously created Knowledge Base. It will be able to answer questions about policies and guidelines but also we are going to create some tools to create JIRA tickets on the IT board. If you have already created Atlassian organization and Confluence, you should also get JIRA as well. We can reuse the same credentials as for synchronizing Bedrock with Confluence because they mirror your admin permissions. However remember, that for production use cases you should do the OAuth flow for this use case as well (or use Service Accounts).

GitHub repository: ppabis/it-agent-bedrock.

Back to Part 1

JIRA ticket templates

I will create two templates for JIRA tickets to see how well our agent will handle this. One type of the ticket would be a generic request to IT and another one will be database access ticket that will have some extra fields. We will later use Lambda to interface with JIRA. But first, let's proceed to JIRA on your Atlassian domain: https://<myorg>.atlassian.net/jira. You can see below a new ticket type that I have created. It is based on one of the documents in Confluence.

Interfacing Python with JIRA

In order to create tickets we will naturally use JIRA's API. But we need to get some values from this API first - the project ID and issue types IDs. From the commands below you can see that my project's ID is 10000 (usually the first board you create in JIRA) and, 10003 and 10008 ticket types.

$ curl -H "Content-Type: application/json" \
 -u "$CONFLUENCE_EMAIL:$CONFLUENCE_TOKEN" \
 https://<myorg>.atlassian.net/rest/api/3/project/AWS | jq .id
"10000"

$ curl -H "Content-Type: application/json" \
 -u "$CONFLUENCE_EMAIL:$CONFLUENCE_TOKEN" \
 https://<myorg>.atlassian.net/rest/api/3/project/AWS \
 | jq '.issueTypes[] | [.id, .name, .description]'
[
  "10002",
  "Subtask",
  "Subtasks track small pieces of work that are part of a larger task."
]
[
  "10003",
  "Task",
  "Tasks track small, distinct pieces of work."
]
[
  "10006",
  "Bug",
  "Bugs track problems or errors."
]
[
  "10008",
  "Database Access",
  "Ticket if you want to get a website access for your user"
]

What is more, I would like to get the fields needed to be filled out in each ticket. For each of the chosen ticket types that I want to be supported, I get the field ID, schema, name and if it's required or not.

$ curl -H "Content-Type: application/json" \
 -u "$CONFLUENCE_EMAIL:$CONFLUENCE_TOKEN" \
 https://<myorg>.atlassian.net/rest/api/3/issue/createmeta/10000/issuetypes/10008" \
  | jq '.fields[] | {fieldId, name, schema, required}'

{
  "fieldId": "customfield_10073",
  "name": "permissions",
  "schema": {
    "type": "string",
    "custom": "com.atlassian.jira.plugin.system.customfieldtypes:textarea",
    "customId": 10073
  }
}
{
  "fieldId": "customfield_10074",
  "name": "Business reason for access",
  "schema": {
    "type": "string",
    "custom": "com.atlassian.jira.plugin.system.customfieldtypes:textarea",
    "customId": 10074
  }
}
{
  "fieldId": "customfield_10075",
  "name": "access_until",
  "schema": {
    "type": "date",
    "custom": "com.atlassian.jira.plugin.system.customfieldtypes:datepicker",
    "customId": 10075
  }
}

Now based on the retrieved information we can proceed to create a Python function that will create a new JIRA ticket for given type. I will vibe-code it because we are doing GenAI after all, aren't we? I will also ask to generate OpenAPI schema in JSON and separate paths for ticket types. I will also ask it to create some validation functions for the inputs, give correct responses for Bedrock and paste some example event that Bedrock Agent produces. I will skip all the details how this entire script works internally (if you want to read through it, you can go to the Git repository, ideally as an LLM to explain this 😂) but I will simply describe how request and response works. So the incoming payload from Bedrock Agent looks something like this.

{
    "messageVersion": "1.0",
    "parameters": [],
    "sessionId": "730335385312952",
    "inputText": "Yes I need a database ticket for: john.doe and mike.smith the access should be granted until 30 June 2027 for development database with permissions analytics.*=SELECT,SELECT VIEW. The reason for this ticket is that John and Mike are switching department to business intelligence and analytics.",
    "agent": {
        "name": "ticketagent",
        "version": "2",
        "id": "KGKLDPMDNZ",
        "alias": "FHAQLR6Q0P"
    },
    "actionGroup": "jira_tickets",
    "httpMethod": "POST",
    "apiPath": "/db-ticket",
    "requestBody": {
        "content": {
            "application/json": {
                "properties": [
                    {
                        "name": "database_names",
                        "type": "array",
                        "value": "[development]"
                    },
                    {
                        "name": "user_list",
                        "type": "string",
                        "value": "john.doe\nmike.smith"
                    },
                    // ... more of these
                ]
            }
        }
    },
    "sessionAttributes": {},
    "promptSessionAttributes": {}
}

So as you see, we get Agent that called our Lambda, the actual input given by the user that triggered the tool call and some other things. But what we need to focus on is apiPath that specifies the type of ticket that we want to create as well as requestBody that contains the values that we need to first, validate, and second, post to JIRA. But how does the agent know what to write in the request body? This comes from OpenAPI specification that I also asked the LLM to generate along with the Lambda code. This is a small excerpt from the file as a sample (converted to YAML for readability 😅). We will load this file later when creating the action group.

openapi: 3.0.0
info:
  title: Jira Ticket API
  version: 1.0.0
  description: API for creating Jira tickets
paths:
  /db-ticket:
    post:
      description: Create a database access request ticket in Jira
      operationId: createDbTicket
      requestBody:
        required: true
        content:
          application/json:
            schema:
              type: object
              required: [summary, database_names, user_list, permissions, business_reason, access_until]
              properties:
                database_names:
                  type: array
                  items:
                    type: string
                  description: List of database names (production, staging, development)
                # ... more fields
      responses:
        '200':
          description: Successful response
          content:
            application/json:
              schema:
                type: object
                properties:
                  message:
                    type: string
  /generic-ticket:
    post:
      # ... another type of ticket

In Python I parse such input, route to the correct path for the requested ticket and respond in the expected format. I also print all the values to the logs and log a lot. This helps with observability and continuous improvement. In production, you would need also to take care of PII removal from events being sent here (unless you filter input with Guardrails).

def lambda_handler(event, context):
    """Entry point for the Lambda: route requests based on the API path."""
    print(json.dumps(event))

    api_path = event.get("apiPath") or event.get("path")
    if api_path == "/db-ticket":
        response = route_db_ticket(event)
    elif api_path == "/generic-ticket":
        response = route_generic_ticket(event)
    else:
        error_body = {"message": f"Unsupported route: {api_path}"}
        response = _bedrock_agent_response(event, 400, error_body)

    print(json.dumps(response))
    return response

To format the output, this helper function is used. It just copies over some of the input event values and tries to format the response as JSON string. For example I raise errors on input validation issues or JIRA responses and the messages are both printed into the logs and given to the chatbot so that it can retry with a fixed input. Here you should also consider if exceptions that are raised contain some sensitive information as you don't want to give and details such as API Keys to the user 🫠.

def _bedrock_agent_response(event: dict, status_code: int, body: dict) -> dict:
    """Wrap an HTTP response in the Bedrock agent response schema."""
    return {
        "messageVersion": "1.0",
        "response": {
            "actionGroup": event.get("actionGroup", "db-ticket-tools"),
            "apiPath": event.get("apiPath", "/db-ticket"),
            "httpMethod": event.get("httpMethod", "POST"),
            "httpStatusCode": status_code,
            "responseBody": {"application/json": {"body": json.dumps(body)}},
        },
    }

Before we can deploy our Lambda function we would need to also install all the dependencies inside it. For example, I use requests as the library to call the API which is not included in standard Lambda distribution. To fix that, I use Amazon Linux 2023 Docker and install requests with pip. Here's a one-liner that does exactly that (assuming that you hold all Lambda code in lambdas/).

docker run --rm -it \
 -v $(pwd)/lambdas:/lambdas \
 amazonlinux:2023 sh -c \
 'yum install python3-pip -y && pip install -t /lambdas/ requests'

Deploying Lambda in Terraform

Now I want to create an actual Lambda function infrastructure. We first need to put all the code and dependencies in an archive as usual and then we also need to define an IAM Role. The Role will have two policies for me: basic execution one to collect the logs and Secrets Manager to reuse the Confluence credentials we created in the first part.

### Code
data "archive_file" "ticket_lambda" {
  type        = "zip"
  source_dir  = "lambdas/"
  output_path = "tickets_lambda.zip"
}

### Permissions
data "aws_iam_policy_document" "lambda_assume_role" {
  statement {
    effect  = "Allow"
    actions = ["sts:AssumeRole"]
    principals {
      type        = "Service"
      identifiers = ["lambda.amazonaws.com"]
    }
  }
}

data "aws_iam_policy_document" "lambda_policy" {
  statement {
    effect    = "Allow"
    actions   = ["secretsmanager:GetSecretValue"]
    resources = [aws_secretsmanager_secret.confluence.arn]
  }
  statement {
    effect    = "Allow"
    actions   = ["kms:Decrypt"]
    resources = ["*"]
    condition {
      test     = "StringLike"
      variable = "kms:ViaService"
      values   = ["secretsmanager.${data.aws_region.current.name}.amazonaws.com"]
    }
  }
}

resource "aws_iam_role" "ticket_lambda_role" {
  name               = "ticket_lambda_role"
  assume_role_policy = data.aws_iam_policy_document.lambda_assume_role.json
}

resource "aws_iam_role_policy" "lambda_policy" {
  name   = "ticket_lambda_policy"
  role   = aws_iam_role.ticket_lambda_role.name
  policy = data.aws_iam_policy_document.lambda_policy.json
}

resource "aws_iam_role_policy_attachment" "lambda_logs" {
  role       = aws_iam_role.ticket_lambda_role.name
  policy_arn = "arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole"
}

Now we can create the Lambda itself. What I would also do (although I don't know if it's required) is grant Bedrock service permissions to call this Lambda using resource policy.

resource "aws_lambda_function" "ticket_lambda" {
  function_name    = "ticket_lambda"
  role             = aws_iam_role.ticket_lambda_role.arn
  handler          = "lambda_handler.lambda_handler"
  runtime          = "python3.13"
  filename         = data.archive_file.ticket_lambda.output_path
  source_code_hash = data.archive_file.ticket_lambda.output_base64sha256
  depends_on       = [aws_iam_role_policy.lambda_policy]
}

resource "aws_lambda_permission" "allow_bedrock" {
  statement_id  = "AllowBedrockInvocation"
  action        = "lambda:InvokeFunction"
  function_name = aws_lambda_function.ticket_lambda.arn
  principal     = "bedrock.amazonaws.com"
  source_arn    = aws_bedrockagent_agent.ticketagent.agent_arn
}

Building Bedrock Agent

First, what our Agent needs is an IAM role to get permissions. First of all, I plan to attach the Knowledge Base, so we need that. The Agent also needs some model (or inference profile) to invoke. And lastly, it should be able to invoke the Lambda function that we just created.

data "aws_iam_policy_document" "agent_assume_role" {
  statement {
    effect  = "Allow"
    actions = ["sts:AssumeRole"]
    principals {
      type        = "Service"
      identifiers = ["bedrock.amazonaws.com"]
    }
  }
}

resource "aws_iam_role" "agent_role" {
  name               = "ticketagent-agent-role"
  assume_role_policy = data.aws_iam_policy_document.agent_assume_role.json
}

data "aws_iam_policy_document" "agent_policy" {
  statement {
    effect = "Allow"
    actions = [
      "bedrock:InvokeModel",
      "bedrock:InvokeModel*",
      "bedrock:GetInferenceProfile",
      "bedrock:ListInferenceProfiles",
      "bedrock:ListTagsForResource",
      "bedrock:GetKnowledgeBase",
      "bedrock:ListKnowledgeBases",
      "bedrock:Retrieve",
      "bedrock:RetrieveAndGenerate"
    ]
    resources = ["*"]
  }
  statement {
    effect    = "Allow"
    actions   = ["lambda:InvokeFunction"]
    resources = [aws_lambda_function.ticket_lambda.arn]
  }
}

resource "aws_iam_role_policy" "agent_policy" {
  name   = "ticketagent-agent-policy"
  role   = aws_iam_role.agent_role.id
  policy = data.aws_iam_policy_document.agent_policy.json
}

I chose region eu-central-1 and unfortunately the model choice here is quite small. AWS provides us with so called Inference Profiles. You have probably heard about cross-region inference (as one of the shiny features ✨) and these Profiles are exactly used for that. You can list them and see regions covered by them using AWS Console (in "Cross-region inference" section) or using the following CLI Command: aws bedrock list-inference-profiles --region eu-central-1. In Terraform, I will use a data source and filter by name to find the profile I want to use which is cross-EU Amazon Nova 2 Lite.

data "aws_bedrock_inference_profiles" "profiles" { type = "SYSTEM_DEFINED" }

locals {
  inference_profile_arn = [
    for profile in data.aws_bedrock_inference_profiles.profiles.inference_profile_summaries
    : profile.inference_profile_arn
    if strcontains(profile.inference_profile_name, "Nova 2 Lite")
     && strcontains(profile.inference_profile_name, "EU")
  ][0]
}

For the agent, we also need to create a system prompt that will be passed into it. We can use Bedrock Prompts to manage it but here I will use just a simple YAML file that will be also loaded as a local constant. In the agent construction I defined prepare_agent to true, so that we can test it almost immediately. Why is this step needed? Thank you for using Amazon Bedrock.

knowledge_base_description: |-
  Knowledge base of the company with all needed IT knowledge, troubleshooting guides, standards, policies, rules, etc.

agent_prompt: |-
  You are an IT specialist in a company called Acme. You are responsible for helping the users with their IT questions and issues.
  You are using the knowledge base to provide company information and help the users. You can also validate the ticket formats against company
  policies and standards and create JIRA tickets with the provided tools.

locals {
  prompts = yamldecode(file("prompts.yaml"))
}


resource "aws_bedrockagent_agent" "ticketagent" {
  agent_name              = "ticketagent"
  foundation_model        = local.inference_profile_arn
  agent_resource_role_arn = aws_iam_role.agent_role.arn
  depends_on              = [aws_bedrockagent_knowledge_base.confluence]
  prepare_agent           = true
  instruction             = local.prompts.agent_prompt
}

Next, I will associate the Knowledge Base with the Agent so that it can answer our questions about the company and all the troubleshooting guides that we store in Confluence. What is more, I want to also added the tools: Lambda for creating tickets as well as built-in "user input" tool that just permits the agent to write follow up questions to the user (although from experience, LLM with either way ask me again for clarification without that too 🤷). This is the place where we attach the OpenAPI schema. In all "action groups" (tools) you create, you need to specify DRAFT version of the agent.

resource "aws_bedrockagent_agent_knowledge_base_association" "confluence" {
  agent_id             = aws_bedrockagent_agent.ticketagent.id
  knowledge_base_id    = aws_bedrockagent_knowledge_base.confluence.id
  knowledge_base_state = "ENABLED"
  depends_on           = [aws_bedrockagent_agent.ticketagent, aws_bedrockagent_knowledge_base.confluence]
  description          = local.prompts.knowledge_base_description
}

resource "aws_bedrockagent_agent_action_group" "jira_tickets" {
  agent_id           = aws_bedrockagent_agent.ticketagent.id
  agent_version      = "DRAFT"
  action_group_state = "ENABLED"
  action_group_name  = "jira_tickets"

  action_group_executor { lambda  = aws_lambda_function.ticket_lambda.arn }
  api_schema            { payload = file("lambdas/tool_schema.json")      }
}

resource "aws_bedrockagent_agent_action_group" "user_input" {
  agent_id                      = aws_bedrockagent_agent.ticketagent.id
  agent_version                 = "DRAFT"
  action_group_name             = "AskUserAction"
  action_group_state            = "ENABLED"
  parent_action_group_signature = "AMAZON.UserInput"
}

As the last step, I would like to create an alias for the Agent, which means it will be usable in our applications. This will ensure that all the action groups and knowledge base are connected. It will also publish a new version of the Agent. If you ever need to create another version, simply use tofu taint on this production alias and apply again.

resource "aws_bedrockagent_agent_alias" "production" {
  agent_id         = aws_bedrockagent_agent.ticketagent.id
  agent_alias_name = "ticketagent-production"
  depends_on       = [
    aws_bedrockagent_agent_knowledge_base_association.confluence,
    aws_bedrockagent_agent_action_group.jira_tickets,
    aws_bedrockagent_agent_action_group.user_input
  ]
}

output "agent_alias" {
  value = aws_bedrockagent_agent_alias.production.agent_alias_arn
}

Testing the agent

I used AWS Console to test the agent. I opened the newly created alias and expanded the chat in the top right for convenience. This also allows to see the trace of each action taken by the agent, whether it is reading from knowledge base, thinking or calling actions.

The first test I performed was to ask about IT department. I already did this when testing the Knowledge Base and the results were correct but the scores were very off compared to the random question about ducks (0.40 vs 0.36). This is supposed to be fixable by a reranker model (but the quotas on it are ridiculously low).

Next one was to test ticket creation. I have easily created the database ticket and the agent even followed up on more details. Although as you can see in the screenshot, sometimes Nova 2 Lite leaves some artifacts (the default temperature seems to be set to 1.0 by default but this can be edited in "advanced" orchestration settings).

Lastly, I wanted to test creation of a generic ticket. This time I wanted to push through some insult so that we can test guardrails later on. I was sure that Nova was fine-tuned to prevent such requests but it actually did follow my order.

IT Support Agent on AWS Bedrock - Connecting Confluence

Piotr Pabis — Sun, 25 Jan 2026 11:35:40 +0000

GenAI Developer Pro exam is coming with big, loud steps and instead of reading through AI slop on SkillBuilder that even ChatGPT can't comprehend, I decided to learn through experience by building. My goal is to create an AI Agent that will act as IT support for a fake company Acme. As the first step I would like to create a Bedrock Knowledge Base that will be connected to Atlassian Confluence. That way we will prepare our support agent to act as a simple chatbot to find relevant troubleshooting guides and company policies stored on Confluence. This way when a colleague comes to the chat interface and asks "How can I connect to company VPN", the chatbot will search through all the documents stored in Confluence and give an answer that for example "You need to use WireGuard, generate key pair, provide public key to IT and they will send back you a configuration, here's how to do it step by step...".

You can find the code on GitHub: ppabis/it-agent-bedrock.

Creating Confluence

Atlassian gives you some small free tier of their cloud services (which includes JIRA, Trello, Confluence, Bitbucket). Just sign up for an account if you don't have one (if you ever used any of the above services, you already have an Atlassian account, so you just need to create Confluence instance). Visit atlassian.com and after you have an account proceed to Confluence. Click "Get it free" at the top and choose your <myname>.atlassian.net domain. Once you are onboarded, you can visit that new address any time and switch to the Confluence by clicking on the top left (4 squares) and selecting "Confluence".

Customize the wiki however you want. I won't guide you here how to use Confluence as it's a large software on it's own but explore it and create some Pages that you can reference later with a chatbot. I asked ChatGPT to generate me some IT documents, some more general, some more technical and created some pages. In one Page I also "hid" where the IT room is so I wonder if our Knowledge Base will be able to retrieve it from an unrelated document.

Creating credentials

After you have already played around with Confluence we will generate an API key. As an administrator of the Atlassian organization go to your personal account settings in the top-right corner, then Security and Manage API tokens. It will likely ask you for 2FA.

Next select "Create API token". I tried using the one with scopes but with the setup I have but I believe it works only with OAuth and not Basic HTTP credentials I plan to use. For production use cases, you should definitely use OAuth but for this demo, we will just rely on the key that has full permissions on your account. You can read more about OAuth setup here.

Base infrastructure

We are going to build our infrastructure in OpenTofu. I will first import two providers: aws and null and create some variables and a new secret in Secrets Manager.

terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 6.0"
    }
    null = {
      source  = "hashicorp/null"
      version = "~> 3.0"
    }
  }
}

provider "aws" { region = "eu-central-1" }

variable "confluence_username" { type = string }
variable "confluence_password" { type = string }

resource "aws_secretsmanager_secret" "confluence" {
  name                    = "confluence"
  description             = "Confluence credentials"
  recovery_window_in_days = 7
}

resource "aws_secretsmanager_secret_version" "confluence" {
  secret_id = aws_secretsmanager_secret.confluence.id
  secret_string = jsonencode({
    username = var.confluence_username
    password = var.confluence_password
  })
  lifecycle { ignore_changes = [secret_string] }
}

You can set it up in the following way: the e-mail can be stored in terraform.tfvars safely: this is the e-mail you are registered in Atlassian as. For the API token I recommend running the following Bash commands, pasting the token and only then applying the infrastructure. Because I added ignore_changes to the secret version, Terraform will not ask you again for the token and it won't be stored anywhere.

read -s TF_VAR_confluence_password # after running immediately paste the token and press enter
export TF_VAR_confluence_password
tofu apply
export TF_VAR_confluence_password="undefined" # So it doesn't ask again

That way we have stored credentials in AWS Secrets Manager so that later on, Bedrock will be able to use them for crawling.

Creating OpenSearch backend

As our underlying storage for processed text chunks from the wiki we need to use OpenSearch Serverless. Unfortunately, it is a costly service that you pay even for not using (and pay a lot for a serverless offering) so be cautious. I don't know if AWS will ever support S3 Vector Buckets that are way more reasonably priced. Let's start by defining an OpenSearch policies because without them, you will have hard time creating anything in this service 🙄. Because encryption policy has to be defined before creating the collection, it cannot depend on it so we will define the new collection name in advance in locals (unless you already have something in place like encryption policy with *). Apply this so we can continue.

locals { collection_name = "ticketagent-collection" }

resource "aws_opensearchserverless_security_policy" "encryption" {
  name = "ticketagent-encryption"
  type = "encryption"
  policy = jsonencode({
    Rules = [
      {
        Resource     = ["collection/${local.collection_name}"]
        ResourceType = "collection"
      }
    ],
    AWSOwnedKey = true
  })
}

Now after we have the encryption policy we can define the collection and other two policies: network and data access. Network defines if the collection is public and which private IPs can access it if you lock it down into VPC as well as AWS services permitted to connect and data access is something like resource policy on S3 bucket defining actions that can be taken by IAM principals.

Because we will need access just in a minute, I will open the OpenSearch collection to the public (as there's no public IP restriction list 😵‍💫). Also I will give my current user that executes OpenTofu permissions to manage all data within the collection. And the collection itself will be of type VECTORSEARCH and no replicas 🫠.

data "aws_caller_identity" "current" {}

resource "aws_opensearchserverless_collection" "knowledge_base" {
  name             = "ticketagent-collection"
  description      = "OpenSearch Serverless collection powering the Bedrock KB"
  depends_on       = [aws_opensearchserverless_security_policy.encryption]
  type             = "VECTORSEARCH"
  standby_replicas = "DISABLED"
}

resource "aws_opensearchserverless_security_policy" "network" {
  name = "ticketagent-network"
  type = "network"
  policy = jsonencode([
    {
      # SourceServices  = ["bedrock.amazonaws.com"],
      Rules           = [{
        ResourceType = "collection",
        Resource = ["collection/${aws_opensearchserverless_collection.knowledge_base.name}"]
      }],
      AllowFromPublic = true
    }
  ])
}

resource "aws_opensearchserverless_access_policy" "data_policy" {
  name = "ticketagent-data-policy"
  type = "data"
  policy = jsonencode([
    {
      Description = "Full access"
      Rules = [
        {
          ResourceType = "collection"
          Resource     = ["collection/${aws_opensearchserverless_collection.knowledge_base.name}"]
          Permission   = ["aoss:*"]
        },
        {
          ResourceType = "index"
          Resource     = ["index/${aws_opensearchserverless_collection.knowledge_base.name}/*"]
          Permission   = ["aoss:*"]
        },
        {
          ResourceType = "model"
          Resource     = ["model/${aws_opensearchserverless_collection.knowledge_base.name}/*"]
          Permission   = ["aoss:*"]
        }
      ]
      Principal = [
        data.aws_caller_identity.current.arn
      ]
    }
  ])
}

Creating index in OpenSearch

When you use AWS Console, there's some magic happening in the background that the index is being created in OpenSearch collection. Somehow Bedrock is not smart enough to create the collection by itself and you have to do it for it. You might think about another Terraform resource, don't you? WRONG ❌. Even though there's AWS CLI command to create and delete an index, there's no IaC construct doing it anywhere. No CloudFormation, no Terraform, no CDK, nothing. (Still think new Elastic license was so bad?) So we will use null resource to create the index using AWS CLI. For the index we need a good schema, so I will define some locals as well for better control.

locals {
  index_name            = "ticketagent_index"
  vector_dimension      = 256
  vector_text_field     = "chunk"
  vector_metadata_field = "metadata"
  vector_field          = "vector"
  schema = jsonencode({
    settings = {
      index = {
        knn = true
        "knn.algo_param.ef_search" : 512
      }
    },
    mappings = {
      properties = {
        "${local.vector_field}" = {
          type      = "knn_vector"
          dimension = local.vector_dimension
          method = {
            name       = "hnsw"
            engine     = "faiss"
            parameters = {}
            space_type = "l2"
          }
        }
        "${local.vector_text_field}" = {
          type  = "text"
          index = "true"
        }
        "${local.vector_metadata_field}" = {
          type  = "text"
          index = "false"
        }
      }
    }
  })
}

Let me explain to you what is happening above. First we configure the vector dimensions - for each text chunk (200 or 300 tokens or words), we generate a vector of size 256 using an embedding model. This vector will hold semantic meaning of this text fragment. If you didn't understand, let me send you to huggingface 🤗. Continuing, I chose my own names for each field in OpenSearch: vector for the vector (we can assume it's the primary key), chunk for the text piece from Confluence and metadata for some Bedrock info like URL of the page scanned, date and time and so on. I will turn of indexing on metadata as we will primarily do queries by vector or maybe chunk. You might want to ask what are the other things in the JSON. hnsw is vector indexing method that puts the index into memory and is fast. As we don't have that much data in Confluence we can easily use that. If we had a lot of text, we might consider ivf method because this one is memory efficient but slow. As an engine we use faiss which stands for... Facebook AI Similarity Search (very scientific name, rite?). And for comparison we use l2 which is Euclidean distance (aka literal distance between two points in a straight line). Some say cosinesimil (which is angle between two points) is more reliable but I will just stay with that 🤷. If you want to learn more about all the configurations for vector search see here.

Ok, we have the JSON for the schema, we have collection, we can now create the index with AWS CLI. It might happen that this fails if you apply it all together with the creation of the policies, and if so, try applying again. The index will be recreated each time you destroy/create the collection.

data "aws_region" "current" {}

resource "null_resource" "index" {
  provisioner "local-exec" {
    command = "aws opensearchserverless create-index --region ${data.aws_region.current.name} --id ${aws_opensearchserverless_collection.knowledge_base.id} --index-name ${local.index_name} --index-schema '${local.schema}'"
  }
  depends_on = [aws_opensearchserverless_collection.knowledge_base, aws_opensearchserverless_access_policy.data_policy, aws_opensearchserverless_security_policy.network]
  lifecycle {
    replace_triggered_by = [aws_opensearchserverless_collection.knowledge_base]
  }
}

Bedrock Knowledge Base (and IAM Role)

Finally, we can create the actual Knowledge Base that will connect to and pull data from Confluence. It will need an IAM Role to use OpenSearch and get the credentials and we will define it first.

data "aws_iam_policy_document" "bedrock_kb_assume_role" {
  statement {
    effect = "Allow"
    principals {
      type        = "Service"
      identifiers = ["bedrock.amazonaws.com"]
    }
    actions = ["sts:AssumeRole"]
  }
}

resource "aws_iam_role" "bedrock_kb_role" {
  name               = "ticketagent-bedrock-kb-role"
  assume_role_policy = data.aws_iam_policy_document.bedrock_kb_assume_role.json
}

data "aws_iam_policy_document" "bedrock_kb_policy" {
  statement {
    effect    = "Allow"
    actions   = ["secretsmanager:GetSecretValue"]
    resources = [aws_secretsmanager_secret.confluence.arn]
  }

  statement {
    effect    = "Allow"
    actions   = ["kms:Decrypt"]
    resources = ["*"]
    condition {
      test     = "StringLike"
      variable = "kms:ViaService"
      values   = ["secretsmanager.${data.aws_region.current.name}.amazonaws.com"]
    }
  }

  statement {
    effect = "Allow"
    actions = [
      "bedrock:InvokeModel",
      "bedrock:StartIngestionJob",
      "bedrock:CreateDataSource",
      "bedrock:GetDataSource",
      "bedrock:DescribeKnowledgeBase",
      "bedrock:Retrieve"
    ]
    resources = ["*"]
  }

  statement {
    effect    = "Allow"
    actions   = ["aoss:APIAccessAll"]
    resources = [aws_opensearchserverless_collection.knowledge_base.arn]
  }
}

resource "aws_iam_role_policy" "bedrock_kb_policy" {
  name   = "ticketagent-bedrock-kb-policy"
  role   = aws_iam_role.bedrock_kb_role.id
  policy = data.aws_iam_policy_document.bedrock_kb_policy.json
}

Next we need to allow this role in OpenSearch'es Data Policy. Let's go back to it and add another principal.

resource "aws_opensearchserverless_access_policy" "data_policy" {
  name = "ticketagent-data-policy"
  type = "data"
  policy = jsonencode([
    {
      ...
      Principal = [
        data.aws_caller_identity.current.arn,
        aws_iam_role.bedrock_kb_role.arn
      ]
    }
  ])
}

Now we can create the Knowledge Base and data source that will use Confluence. We need to also choose an embedding model. I will simply use Titan v2. I will refer to all the locals we defined during index creation so that every value matches.

locals {
  embedding_model_arn = "arn:aws:bedrock:${data.aws_region.current.name}::foundation-model/amazon.titan-embed-text-v2:0"
}

resource "aws_bedrockagent_knowledge_base" "confluence" {
  depends_on  = [null_resource.index]
  name        = "ticketagent-confluence-kb"
  role_arn    = aws_iam_role.bedrock_kb_role.arn
  description = "Confluence knowledge base backed by OpenSearch Serverless."

  knowledge_base_configuration {
    type = "VECTOR"
    vector_knowledge_base_configuration {
      embedding_model_arn = local.embedding_model_arn
      embedding_model_configuration {
        bedrock_embedding_model_configuration {
          dimensions          = local.vector_dimension
          embedding_data_type = "FLOAT32"
        }
      }
    }
  }

  storage_configuration {
    type = "OPENSEARCH_SERVERLESS"
    opensearch_serverless_configuration {
      collection_arn    = aws_opensearchserverless_collection.knowledge_base.arn
      vector_index_name = local.index_name
      field_mapping {
        vector_field   = local.vector_field
        text_field     = local.vector_text_field
        metadata_field = local.vector_metadata_field
      }
    }
  }
}

In data source we need to provide the Secrets Manager's secret and URL of the Atlassian organization you have created at the beginning. That should be something like <my_org>.atlassian.net.

variable "confluence_instance_url" {
  type        = string
  description = "Base URL for the Confluence Cloud site (e.g. https://org.atlassian.net)."
}

resource "aws_bedrockagent_data_source" "confluence" {
  name              = "ticketagent-confluence-source"
  knowledge_base_id = aws_bedrockagent_knowledge_base.confluence.id

  data_source_configuration {
    type = "CONFLUENCE"
    confluence_configuration {
      source_configuration {
        host_url               = var.confluence_instance_url
        host_type              = "SAAS"
        auth_type              = "BASIC"
        credentials_secret_arn = aws_secretsmanager_secret.confluence.arn
      }
    }
  }
}

Syncing and testing the knowledge base

You can now sync the knowledge base and test retrieval, either using AWS Console or AWS CLI. You can also put some filters when querying. For example I want to just retrieve chunks of type Page and max results of 3.

aws bedrock-agent start-ingestion-job \
 --knowledge-base-id $(tofu output -raw knowledge_base_id) \
 --data-source-id $(tofu output -raw data_source_id | cut -d, -f1)

aws bedrock-agent list-ingestion-jobs \
 --knowledge-base-id $(tofu output -raw knowledge_base_id) \
 --data-source-id $(tofu output -raw data_source_id | cut -d, -f1) \
 --query 'ingestionJobSummaries[].[ingestionJobId,status]' \
 --output table
# Wait until all jobs are COMPLETE

aws bedrock-agent-runtime retrieve \
 --knowledge-base-id $(tofu output -raw knowledge_base_id) \
 --retrieval-query "text=How to connect to VPN"
# Responds with some JSONs

# Add filtering and max results
FILTER_JSON='{
  "vectorSearchConfiguration": {
    "numberOfResults": 3,
    "filter": {
      "equals": {
        "key": "x-amz-bedrock-kb-category",
        "value": "Page"
      }
    }
  }
}'
aws bedrock-agent-runtime retrieve \
 --knowledge-base-id $(tofu output -raw knowledge_base_id) \
 --retrieval-query "text=How to connect to VPN" \
 --retrieval-configuration "$FILTER_JSON"

Going further

In the next part we will implement a helpful agent that will answer our queries and maybe create a first tool for this agent to take some actions. I believe your today's adventure with OpenSearch Serverless was more than enough.

Serverless Picture Gallery on Google Cloud - Part 3

Piotr Pabis — Tue, 13 Jan 2026 13:06:14 +0000

In the previous episode we created a function that was triggered when a new object was uploaded to the temporary bucket. In this part, we are going to process each image with Gemini Nano Banana to make it look like Cyberpunk (or whatever else you imagine in your prompt). Each processed image will be placed into the main bucket and all the metadata will be kept in Firestore database. Then the index of the site is going to list all the images based on the entries in the database.

GitHub repo: ppabis/gcp-photos-gallery

Back to part 1 Back to part 2

Creating a Firestore database

Most logical step is to first create a Firestore database. We will be using it in native mode as this is more modern than the datastore one and we don't need MongoDB compatibility with Enterprise edition. Native mode will look either way similar to Mongo as you create collections there as well compared to "kinds" in Datastore. Also in advance we will give permissions for our Service Accounts from both Cloud Run services to interact with this database - the function will have write permissions and the index website will have only read permissions.

resource "google_firestore_database" "photos_db" {
  name                    = "photos-db"
  location_id             = "europe-west4"
  type                    = "FIRESTORE_NATIVE"
  delete_protection_state = "DELETE_PROTECTION_DISABLED"
}

resource "google_project_iam_member" "website_firestore_reader" {
  project = var.project_id
  role    = "roles/datastore.viewer"
  member  = "serviceAccount:${google_service_account.cloud_run_sa.email}"
}

resource "google_project_iam_member" "processor_firestore_user" {
  project = var.project_id
  role    = "roles/datastore.user"
  member  = "serviceAccount:${google_service_account.processor_sa.email}"
}

Revisiting the event triggered function

Let's go back to the function that processes the image. First we want to save the details in the database. Because Firestore is a NoSQL database, you can put any structure to each object in a collection without worrying about any rigid schema. We can put some date and labels returned from Vision API as the first test. I will put here again the main handler function of the Cloud Run Function.

@functions_framework.cloud_event
def process_image(cloud_event):
    data = cloud_event.data

    def log(msg, sev="INFO"):
        header = request.headers.get("X-Cloud-Trace-Context")
        tid = header.split("/")[0] if header else "n/a"
        print(json.dumps({
            "message": msg, 
            "severity": sev, 
            "logging.googleapis.com/trace": tid
        }))

    image_path = f"gs://{data['bucket']}/{data['name']}"
    labels = label_image(image_path, log)
    log("Labels detected by Vision API: " + ", ".join(labels))

As you see, we were receiving a list of labels from Vision API and printing them out with log function. Into requirements.txt add google-cloud-firestore and write the following function that will let us save the data.

import os
from google.cloud import firestore

def save_image_details(image_name, bucket_name, labels, log):
    db_name = os.environ.get("FIRESTORE_DB", "(default)")

    try:
        db = firestore.Client(database=db_name)
        doc_ref = db.collection("photos").document(image_name)
        doc_ref.set({
            "filename": image_name,
            "bucket": bucket_name,
            "labels": labels,
            "processed_at": firestore.SERVER_TIMESTAMP
        })
    except Exception as e:
        log(f"Error saving image details to Firestore: {e}", "ERROR")

Now we can simply call this function in the handler with appropriate arguments. Also in Terraform you need to add an environment variable called FIRESTORE_DB that will contain the name of the database to connect to.

@functions_framework.cloud_event
def process_image(cloud_event):
    data = cloud_event.data
    ...

    image_path = f"gs://{data['bucket']}/{data['name']}"
    labels = label_image(image_path, log)
    log("Labels detected by Vision API: " + ", ".join(labels))
    save_image_details(data['name'], data['bucket'], labels, log)

resource "google_cloudfunctions2_function" "image_processor" {
  name        = "image-processor"
  location    = "europe-west4"
  ...
  service_config {
    max_instance_count    = 3
    ...

    environment_variables = {
      FIRESTORE_DB  = google_firestore_database.photos_db.name
    }
  }
}

Augmenting the image with Vertex AI

In the same place we will now create another helper function that will call Nano Banana on Vertex AI. Add google-genai and Pillow to the requirements.txt file. First we need to download the image into memory as Gemini client expects us to pass raw data. I will place it in another function.

from google.cloud import storage
def _get_image_from_bucket(image_name, source_bucket):
    blob = storage.Client().bucket(source_bucket).blob(image_name)
    blob.reload()
    mime_type = blob.content_type or "image/jpeg"
    image_bytes = blob.download_as_bytes()
    return image_bytes, mime_type

The next one is calling the actual model in order to augment the image based on the prompt. The prompt is a parameter for now to fill later based on your taste. Because we will be using Gemini through Vertex AI, it's important to mark vertexai=True in the client constructor as otherwise it will try to use standalone Gemini API to which you need an API key. In the response we iterate through all parts as the model also returns thinking process and some possible comment as well. We log the messages (viewable in Logs Explorer) and return only image inline data - the focus of our project.

import google.genai as genai
import google.auth

def _gemini_augment(image_data, mime_type, prompt, log):
    credentials, project_id = google.auth.default()

    client = genai.Client(
        vertexai=True, 
        project=project_id, 
        location="global"
    )

    # Call Nano Banana Pro to augment the image
    response = client.models.generate_content(
        model="gemini-3-pro-image-preview",
        contents=[
            prompt,
            genai.types.Part.from_bytes(data=image_data, mime_type=mime_type)
        ]
    )

    for candidate in response.candidates:
        for part in candidate.content.parts:
            if part.text is not None:
                log(f"Gemini output text: {part.text}")
            if part.inline_data is not None:
                log(f"Inline data found: {len(part.inline_data.data)} bytes")
                return part.inline_data.data

    log("No augmented image data found")
    return None

The next function will be used for storing the augmented image into our actual photos bucket. Gemini always returns PNG format. As a return we will get a public URL of the image (starts with https://storage.googleapis.com).

def _upload_image_to_bucket(image_bytes, target_bucket, image_name, log):
    # Change extension to PNG, keep all the other parts of the name
    target_blob_name = f"images/{'.'.join(image_name.split('.')[:-1])}.png"
    target_blob = storage.Client().bucket(target_bucket).blob(target_blob_name)
    target_blob.upload_from_string(image_bytes, content_type="image/png")
    log(f"Successfully uploaded augmented image to {target_blob.public_url}")
    return target_blob.public_url

Together the high level function will look something like this, where we first load the image, send it to Gemini with prompt for augmentation and then look for returned image to save in the bucket. I also here give you some example prompt I used for this project - feel free to adapt it however you like. I added Vision API labels into the prompt as well, although I believe they won't affect the result anyhow.

def augment_image(image_name, source_bucket, target_bucket, labels, log) -> str:
    PROMPT = f"""
    The image has the following labels detected by Vision API: {", ".join(labels)}.
    Augment this image to make it look like a scene from cyberpunk.
    Futuristic, neons, bright colors in dark environments.
    """

    image_bytes, mime_type = _get_image_from_bucket(image_name, source_bucket)
    augmented_image_data = _gemini_augment(image_bytes, mime_type, PROMPT, log)
    if augmented_image_data is None:
        return None
    return _upload_image_to_bucket(augmented_image_data, target_bucket, image_name, log)

That way we can add this function to our workflow and modify the Firestore record as well. We extend it by argument called augmented_image_url. What is more we need to load another environment variable which is the target bucket for storing photos as the Cloud Run Function doesn't know where to do it.

def save_image_details(image_name, bucket_name, labels, augmented_image_url, log):
    db_name = os.environ.get("FIRESTORE_DB", "(default)")

    try:
        db = firestore.Client(database=db_name)
        doc_ref = db.collection("photos").document(image_name)
        doc_ref.set({
            "filename": image_name,
            "bucket": bucket_name,
            "labels": labels,
            "augmented_image_url": augmented_image_url,
            "processed_at": firestore.SERVER_TIMESTAMP
        })
    except Exception as e:
        log(f"Error saving image details to Firestore: {e}", "ERROR")

@functions_framework.cloud_event
def process_image(cloud_event):
    TARGET_BUCKET = os.environ.get("TARGET_BUCKET")
    data = cloud_event.data
    ...

    image_path = f"gs://{data['bucket']}/{data['name']}"
    labels = label_image(image_path, log)
    url = augment_image(data['name'], data['bucket'], TARGET_BUCKET, labels, log)
    save_image_details(data['name'], data['bucket'], labels, url, log)

resource "google_cloudfunctions2_function" "image_processor" {
  name        = "image-processor"
  location    = "europe-west4"
  ...
  service_config {
    max_instance_count    = 3
    ...

    environment_variables = {
      FIRESTORE_DB  = google_firestore_database.photos_db.name
      TARGET_BUCKET = google_storage_bucket.photos_bucket.name
    }
  }
}

We also need to give some additional permissions to the service account of our function. Firstly, it needs to be able to write to the photos bucket and also call Vertex AI.

resource "google_project_iam_member" "processor_ai_user" {
  project = var.project_id
  role    = "roles/aiplatform.user"
  member  = "serviceAccount:${google_service_account.processor_sa.email}"
}

resource "google_storage_bucket_iam_member" "processor_target_writer" {
  bucket = google_storage_bucket.photos_bucket.name
  role   = "roles/storage.objectCreator"
  member = "serviceAccount:${google_service_account.processor_sa.email}"
}

Now when we upload the image to the gallery, an augmented version of it should land in the photos bucket and records should be created in Firestore database.

Displaying all photos on the website

We need to first add environment variables pointing to the database name and giving the load balancer endpoint (useful to fix the URL of the image). We don't need to give permissions for the website to read the bucket as we have already given public read access to it for allUsers and all the images we want to display are already indexed by Firestore.

resource "google_cloud_run_v2_service" "website_service" {
  name                = "website-service"
  ...

  template {
    containers {
      ...
      env {
        name  = "UPLOAD_BUCKET"                     # Was here already
        value = google_storage_bucket.tmp_upload_bucket.name
      }

      env {
        name  = "FIRESTORE_DB"
        value = google_firestore_database.photos_db.name
      }

      env {
        name  = "DOMAIN"
        value = google_compute_global_address.lb_ip.address    # Unless you have your own domain
      }
    }
  }
  ...
}

Add google-cloud-firestore to requirements.txt of the index website (the one built with Docker locally) and modify the main.py file. First initialize the Firestore client and then change the index to stream all the object in the database, get properties that interest us and create a list out of it. Further this list is passed to Jinja template to render a nice looking HTML from it. I just give the small excerpt of the website as all the CSS can be easily vibe-coded to make something spectacular 🤣.

...
db_name = os.environ.get("FIRESTORE_DB", "(default)")
db = firestore.Client(database=db_name)

@app.get("/")
def index(request: fastapi.Request):
    domain = os.environ.get("DOMAIN")
    records = db.collection("photos").stream()
    photos = []

    for record in records:
        data = record.to_dict()
        url = data.get("augmented_image_url", "https://picsum.photos/640/480")

        # Replace the googleapis with domain if it's not a https://picsum.photos placeholder
        url = url.replace("https://storage.googleapis.com/", "http://{domain}/")

        photos.append({
            "url": url,
            "labels": ", ".join(data.get("labels", []))
        })

    return templates.TemplateResponse(
        "index.html", {
            "request": request,
            "photos": photos
        }
    )

<ul class="gallery">
    {% for photo in photos %}
        <li>
            <img src="{{ photo.url }}" alt="Gallery Image">
            <div class="overlay">
                {{ photo.labels }}
            </div>
        </li>
    {% endfor %}
</ul>

That way we can now view all the images on our website and look at the labels that were detected. There's so much more you can do with this project. Maybe generating videos with Veo or creating a story for each image with Gemini Flash and playing Chirp-generated narration onmouseover?

Serverless Picture Gallery on Google Cloud - Part 2

Piotr Pabis — Mon, 12 Jan 2026 08:00:00 +0000

In this episode we will continue the project and implement uploading functionality using presigned URLs generated by Cloud Storage. What is more, on successful uploads we are going to trigger a Cloud Run Function that will pass the uploaded image to Vision API to describe it with labels that initially we will only print to the logs. To see the parts we will be creating, look at the diagram below.

GitHub repository: ppabis/gcp-photos-gallery

Back to part 1

Uploads bucket and permissions

First we need a place to upload our images somewhere. For simplicity, we can create another bucket. I will also add lifecycle policy to this bucket so that it is being cleaned up every day in case some uploads cause failure in the function. What is also important is to allow uploading to the bucket from any domain. Cross Origin Resource Sharing (CORS) will block by default any interactions that are not coming from the same domain (in that case storage.googleapis.com), so we need a custom header for this.

resource "google_storage_bucket" "tmp_upload_bucket" {
  name                        = "tmp-upload-${random_string.bucket_name.result}"
  location                    = "EU"
  storage_class               = "STANDARD"
  uniform_bucket_level_access = true
  force_destroy               = true

  lifecycle_rule {
    action    { type = "Delete" }
    condition { age = 1 }
  }

  cors {
    origin          = ["*"]
    method          = ["PUT", "POST"]
    response_header = ["Content-Type"]
    max_age_seconds = 3600
  }
}

As the next step we need to allow our Cloud Run service to write to the bucket as well as generate presigned URLs. Even though we will not be directly writing any data from the container, presigned URLs will inherit the subset of permissions of the creator, in our case Cloud Run's Service Account. As you remember from the previous episode, we have attached an empty Service Account to the service.

resource "google_storage_bucket_iam_member" "sa_storage_creator_tmp" {
  member = "serviceAccount:${google_service_account.cloud_run_sa.email}"
  bucket = google_storage_bucket.tmp_upload_bucket.name
  role   = "roles/storage.objectCreator"
}

# Allow SA to sign URLs
resource "google_service_account_iam_member" "sa_token_creator" {
  service_account_id = google_service_account.cloud_run_sa.name
  role               = "roles/iam.serviceAccountTokenCreator"
  member             = "serviceAccount:${google_service_account.cloud_run_sa.email}"
}

Generating Presigned URLs for direct uploads to the bucket

When we use Presigned URL to PUT or POST an object to the Cloud Storage, we save on network bandwidth and CPU cycles, as any compute service just throws an authenticated link to the user (which is some kilobytes long) and then it's just up to the client to interact with Cloud Storage. We just specify location, token and then all the bits go directly to the bucket.

Let's define the function for uploading the images in Python. For simplicity, the following function will only process JPG images but in the repository you will find a more sophisticated function for many MIME types.

import datetime, uuid, os
from google.cloud import storage
import google.auth
import google.auth.transport.requests

@app.get("/api/upload-url")
def get_upload_url():
    credentials, project_id = google.auth.default()
    credentials.refresh(google.auth.transport.requests.Request())

    bucket_name = os.environ.get("UPLOAD_BUCKET")
    blob_name = f"{uuid.uuid4()}.jpg"
    blob = storage.Client(credentials=credentials).bucket(bucket_name).blob(blob_name)

    url = blob.generate_signed_url(
        version="v4",
        expiration=datetime.timedelta(minutes=15),
        method="PUT",
        content_type="image/jpeg",
        access_token=credentials.token,
        service_account_email=credentials.service_account_email,
    )

    return {"url": url}

@app.get("/upload")
def upload_page(request: fastapi.Request):
    return templates.TemplateResponse("upload.html", {"request": request})

First, we need to refresh Google Cloud credentials, which will give us a new token. Then we will use these specific credentials for the Cloud Storage client and ask the service to sign the URL with our token. You might ask why can't we just do it like in AWS: use our IAM Role tied to Lambda to generate presigned URL? For some reason Google Cloud implements two types of credentials: one is typical tokens (like AWS STS with assumed roles) and another is "compute" credentials that have no private key that is needed for signing. It's just an implementation detail that nobody questions 🙄.

Either way continuing with our website, now we need to upload the file somehow. In AWS S3 we would get a token for POST method that just accepts HTML forms. In GCP we need to use JavaScript to perform a PUT request. So our site will be a bit interactive. First we will generate a new signed URL when the user picks a file in file chooser and then we will perform a PUT on a button click.

<div class="upload-container">
    <form id="upload-form">
        <input type="file" id="file-input" name="file" accept="image/*" required>
        <br>
        <button type="submit" id="upload-button" class="btn-submit" disabled>Upload to Gallery</button>
    </form>
</div>

const fileInput = document.getElementById('file-input');
const uploadButton = document.getElementById('upload-button');
let signedUrl = null;

// When user selects a file, call /api/upload-url and enable the "Upload" button
fileInput.addEventListener('change', async function() {
    const file = this.files[0];
    if (file) {
        console.log('Requesting upload permission...');
        try {
            const response = await fetch('/api/upload-url');
            const data = await response.json();
            if (data.url) {
                signedUrl = data.url;
                uploadButton.disabled = false;
            } else
                console.log('Error: ' + (data.error || 'Could not get upload URL'));
        } catch (e) {
            console.log('Error connecting to server.' + e);
        }
    } else {
        uploadButton.disabled = true;
        signedUrl = null;
    }
});

Now we can create another listener on button click that will perform the PUT request to server and send us back to the index of the website.

const uploadForm = document.getElementById('upload-form');

uploadForm.addEventListener('submit', async function(e) {
    e.preventDefault();
    if (!signedUrl || !fileInput.files[0]) return;

    uploadButton.disabled = true;
    const file = fileInput.files[0];

    try {
        const response = await fetch(signedUrl, {
            method: 'PUT',
            body: file,
            headers: { 'Content-Type': 'image/jpeg' }
        });

        if (!response.ok) throw new Error(response.statusText);

        console.log('Upload successful! Redirect to index');
        setTimeout(() => { window.location.href = '/'; }, 2000);

    } catch (error) {
        console.log('Error during upload.' + error);
        uploadButton.disabled = false;
    }
});

The last thing you have to do is to place UPLOAD_BUCKET environment variable in the original service we created in the previous part.

resource "google_cloud_run_v2_service" "website_service" {
  name                = "website-service"
  ...

  template {
    containers {
      ...
      env {
        name  = "UPLOAD_BUCKET"
        value = google_storage_bucket.tmp_upload_bucket.name
      }
    }
  }
}

I exposed the HTML file on /upload endpoint (although it's just a static file, you can host it in the bucket as well). Now when rebuild and redeploy the container and try to upload an image when you go to this URL on the Load Balancer, it should appear in the temporary bucket.

Cloud Run Function for labeling the image

We can now continue to create a Cloud Run Function that will process each uploaded image to the temporary bucket. For that we will use functions-framework library for Python (as this is my language of choice). It provides some decorators and classes that allow us to bind code and Cloud Run Functions infrastructure. Our function will be later triggered by Eventarc so we specify cloud_event as the entry point. This is similar to Lambda being triggered by EventBridge.

import functions_framework
from flask import request

@functions_framework.cloud_event
def process_image(cloud_event):
    data = cloud_event.data
    ...

I will also define a helper log function that will be useful to format the logs in a way expected by Logs Explorer and set the severity of the log as well. Next I will define another function that will call Vision API with the path to the uploaded object in the bucket. This function will get the applied labels with scores formatted as string to be easy to use later.

@functions_framework.cloud_event
def process_image(cloud_event):
    data = cloud_event.data

    def log(msg, sev="INFO"):
        header = request.headers.get("X-Cloud-Trace-Context")
        tid = header.split("/")[0] if header else "n/a"
        print(json.dumps({
            "message": msg, 
            "severity": sev, 
            "logging.googleapis.com/trace": tid
        }))

    image_path = f"gs://{data['bucket']}/{data['name']}"
    labels = label_image(image_path, log)
    log("Labels detected by Vision API: " + ", ".join(labels))


from google.cloud import vision

def label_image(image_path, log):
    client = vision.ImageAnnotatorClient()
    image = vision.Image()
    image.source.gcs_image_uri = image_path

    try:
        log(f"Labelling image: {image_path}")
        response = client.label_detection(image=image)
        if response.error.message:
            raise Exception(response.error.message)

        labels = response.label_annotations
        return [ f"{label.description} (score: {label.score:.2f})" for label in labels ]

    except Exception as e:
        log(f"Error processing image with Vision API: {e}", "ERROR")
        raise e

Packing the function and deploying

Just as Lambda we need to package the Cloud Run function as a ZIP file. But also we need to put it into a bucket first and then give Cloud Run gs:// location. The following Terraform code will zip and upload the code to the bucket and do it every time there are changes in image_processor directory. Remember to also add requirements.txt in this directory with the following:

functions-framework==3.*
google-cloud-vision
google-cloud-storage

resource "google_storage_bucket" "function_source_bucket" {
  name                        = "function-source-${random_string.bucket_name.result}"
  location                    = "EU"
  storage_class               = "STANDARD"
  uniform_bucket_level_access = true
  force_destroy               = true
}

data "archive_file" "processor_source" {
  type        = "zip"
  source_dir  = "${path.module}/image_processor"
  output_path = "${path.module}/image_processor.zip"
}

resource "google_storage_bucket_object" "processor_zip" {
  name   = "source-${data.archive_file.processor_source.output_md5}.zip"
  bucket = google_storage_bucket.function_source_bucket.name
  source = data.archive_file.processor_source.output_path
}

Now we can define the new Cloud Run Function along with its Service Account. We will give permissions to read data in the temporary bucket as well as execute Vision API. To make the function future-proof, I will also set the memory to 512 MB and give a generous timeout of 200 seconds.

resource "google_cloudfunctions2_function" "image_processor" {
  name        = "image-processor"
  location    = "europe-west4"
  description = "Background processor for uploaded images"

  build_config {
    runtime     = "python311"
    entry_point = "process_image"
    source {
      storage_source {
        bucket = google_storage_bucket.function_source_bucket.name
        object = google_storage_bucket_object.processor_zip.name
      }
    }
  }

  service_config {
    max_instance_count    = 3
    available_memory      = "512M"
    timeout_seconds       = 200
    service_account_email = google_service_account.processor_sa.email
  }

  # event_trigger { ... to be revisited later }
}

resource "google_service_account" "processor_sa" {
  account_id   = "image-processor-sa"
  display_name = "Service Account for Image Processor Function"
}

resource "google_storage_bucket_iam_member" "processor_storage_viewer" {
  bucket = google_storage_bucket.tmp_upload_bucket.name
  role   = "roles/storage.objectViewer"
  member = "serviceAccount:${google_service_account.processor_sa.email}"
}

resource "google_project_iam_member" "processor_vision_user" {
  project = var.project_id
  role    = "roles/visionai.admin" 
  member  = "serviceAccount:${google_service_account.processor_sa.email}"
}

But this is not all! Next we need to define a trigger that will cause any uploaded object into the bucket to invoke this function. For that we will need additional service account with specific permissions, namely receiving events from Eventarc, invoking this function and being usable by global Eventarc service account. We also need to allow Cloud Storage to use Pub/Sub in our project.

# All the permissions configuration
resource "google_service_account" "processor_event_sa" {
  account_id   = "image-processor-event-sa"
  display_name = "Service Account for tmp Bucket events"
}

resource "google_project_iam_member" "processor_event_receiver" {
  project = var.project_id
  role    = "roles/eventarc.eventReceiver"
  member  = "serviceAccount:${google_service_account.processor_event_sa.email}"
}

resource "google_cloud_run_v2_service_iam_member" "processor_invoker" {
  location = google_cloudfunctions2_function.image_processor.location
  project  = google_cloudfunctions2_function.image_processor.project
  name     = google_cloudfunctions2_function.image_processor.name
  role     = "roles/run.invoker"
  member   = "serviceAccount:${google_service_account.processor_event_sa.email}"
}

data "google_project" "project" {}

resource "google_service_account_iam_member" "eventarc_sa_user" {
  service_account_id = google_service_account.processor_event_sa.name
  role               = "roles/iam.serviceAccountUser"
  member             = "serviceAccount:service-${data.google_project.project.number}@gcp-sa-eventarc.iam.gserviceaccount.com"
}

resource "google_project_iam_member" "gcs_pubsub_publishing" {
  project = var.project_id
  role    = "roles/pubsub.publisher"
  member  = "serviceAccount:service-${data.google_project.project.number}@gs-project-accounts.iam.gserviceaccount.com"
}

### Edit the Cloud Run function and add a trigger

resource "google_cloudfunctions2_function" "image_processor" {
  name        = "image-processor"
  location    = "europe-west4"
  ...
  event_trigger {
    trigger_region        = "eu"
    event_type            = "google.cloud.storage.object.v1.finalized"
    retry_policy          = "RETRY_POLICY_DO_NOT_RETRY"
    service_account_email = google_service_account.processor_event_sa.email
    event_filters {
      attribute = "bucket"
      value     = google_storage_bucket.tmp_upload_bucket.name
    }
  }
}

Now when you upload any image to the bucket you should see in the Logs Explorer and entry with the list of the labels that were created by Vision API along with the scores. However, not all images receive labels. Often screenshots or memes cannot be classified by this service.

Moving forward

In the next episode of this series, we are going to finish the project by enhancing each image using Gemini on Vertex AI, storing the results in a Firestore database and listing those on the main site.

Serverless Picture Gallery on Google Cloud - Part 1

Piotr Pabis — Sat, 10 Jan 2026 08:57:27 +0000

Today, I want to walk you through a simple project I decided to build to better learn and understand Google Cloud while preparing for PCA exam. I wanted to utilize many services that Google offers and let me troubleshoot things, as when you break and fix something, you understand it better. In this project, we will create a picture gallery where the pictures will be labeled by Vision API and augmented using Nano Banana. Other services we will utilize are: CloudRun, CloudRun Functions, Global Load Balancer, Cloud Armor, Storage Buckets and Firestore.

In this episode of the project we will only focus on the highlighted part. We will create a publicly readable bucket for our images. A Global External Load Balancer will serve them (along with CDN caching) and also route us to a CloudRun function, currently with a dummy website showing placeholders. We will also hide the load balancer behind Cloud Armor so that only our IP subnet can access the service. In my setup I also added SSL protection but this is optional - you can just use HTTP with IP directly, as you will need a domain for SSL to work.

Find the code on GitHub: ppabis/gcp-photos-gallery

Prerequisites

Before we start, be sure that you have the following:

Google Cloud account (obviously),
OpenTofu v1.11.0 or newer,
gcloud CLI configured and connected to your project,
Docker provider such as Docker Desktop or OrbStack.

Enabling APIs and configuring Docker

We will also enable necessary APIs (the list might not be complete) and configure Docker credentials to use with Artifact Registry. Run the following commands to set up everything.

gcloud services enable vision.googleapis.com
gcloud services enable eventarc.googleapis.com
gcloud services enable cloudfunctions.googleapis.com
gcloud services enable certificatemanager.googleapis.com
gcloud services enable cloudbuild.googleapis.com
gcloud services enable aiplatform.googleapis.com
# You can change region here if you want, just accept the prompt
gcloud auth configure-docker europe-west4-docker.pkg.dev

Providers and basic configuration

In this part of the project we will also import all the Terraform providers we will need later. I will also create necessary basic configuration. The region of my choice is europe-west4 but you can pick any other you want.

terraform {
  required_providers {
    google = {
      source  = "hashicorp/google"
      version = "~> 7.15.0"
    }

    random = {
      source  = "hashicorp/random"
      version = "~> 3.0"
    }

    docker = {
      source  = "kreuzwerker/docker"
      version = "~> 3.0"
    }

    archive = {
      source  = "hashicorp/archive"
      version = "~> 2.4"
    }
  }

  required_version = ">= 1.11.0" # OpenTofu only!
}

variable "project_id" {
  type        = string
  description = "The ID of the Google Cloud project"
}

provider "google" {
  project = var.project_id
  region  = "europe-west4"
  zone    = "europe-west4-a"
}

provider "docker" {
  registry_auth {
    address     = "europe-west4-docker.pkg.dev"
    config_file = pathexpand("~/.docker/config.json")
  }
}

Bucket for storing images

Let's start with the simplest thing which is a bucket which will store the images to show on the website. We will use random_string resource as bucket names have to be globally unique (just like in AWS S3). My bucket will be multi-region across "EU" location. I will also choose uniform bucket level access to prevent ACLs being used for the objects.

resource "random_string" "bucket_name" {
  length  = 10
  special = false
  upper   = false
  lower   = true
  numeric = true
}

resource "google_storage_bucket" "photos_bucket" {
  name                        = "photos-bucket-${random_string.bucket_name.result}"
  location                    = "EU"
  storage_class               = "STANDARD"
  uniform_bucket_level_access = true
  force_destroy               = true
}

What is more, we want the bucket to be publicly accessible and reachable via the Load Balancer later. I will create two resources for that. First will be bucket viewer role that will be given to allUsers. Next I will create a backend for the bucket (AWS equivalent is Target Group).

resource "google_storage_bucket_iam_member" "public_read_access" {
  bucket = google_storage_bucket.photos_bucket.name
  role   = "roles/storage.objectViewer"
  member = "allUsers"
}

resource "google_compute_backend_bucket" "photos_backend_bucket" {
  name        = "backend-bucket-${random_string.bucket_name.result}"
  bucket_name = google_storage_bucket.photos_bucket.name
  enable_cdn  = true
  # edge_security_policy = ... We will revisit it later
}

After you apply, you should be able to upload any object to the bucket via Console or gsutil and see it when accessing via browser.

Global External Application Load Balancer

As you see by the title, Google has amazing naming convention when it comes to their products. But just as long is this name, also list of its functionalities. This essentially combines AWS'es Application Load Balancer, CloudFront and Global Accelerator into one service.

We will reserve a new public IPv4 IP from Google and use it as a frontend for our load balancer. This frontend will lead to URL map that for now will route everything to the storage bucket (we will revisit it later). In this example, it only has HTTP endpoint. In the repository I also left an example for HTTPS if you happen to have a domain. (You will need to add a CNAME record to validate domain ownership.)

resource "google_compute_global_address" "lb_ip" {
  name = "lb-external-ip"
}

resource "google_compute_url_map" "url_map" {
  name            = "global-app-lb"
  default_service = google_compute_backend_bucket.photos_backend_bucket.id

  host_rule {
    hosts        = ["*"]
    path_matcher = "path-matcher"
  }

  path_matcher {
    name            = "path-matcher"
    default_service = google_compute_backend_bucket.photos_backend_bucket.id

    path_rule {
      paths   = ["/images", "/images/*"]
      service = google_compute_backend_bucket.photos_backend_bucket.id
    }
  }
}

resource "google_compute_target_http_proxy" "http_proxy" {
  name    = "lb-http-proxy"
  url_map = google_compute_url_map.url_map.id
}

resource "google_compute_global_forwarding_rule" "forwarding_rule" {
  name                  = "lb-forwarding-rule"
  ip_protocol           = "TCP"
  load_balancing_scheme = "EXTERNAL_MANAGED"
  port_range            = "80"
  target                = google_compute_target_http_proxy.http_proxy.id
  ip_address            = google_compute_global_address.lb_ip.id
}

Now if you go to the loadbalancer via the IP address it should show you also the image. If you go to images/ path it will be appended also when requesting object from the bucket. So for example http://1.2.3.4/images/test.jpg will go to gs://photos-bucket-123456/images/test.jpg. This will be the case also when we later change the default route to Cloud Run.

Creating an index website

First, let's create a dummy website that will simply display some placeholder photos. I will use Python with FastAPI. The site needs to be built with Docker, so you have a very wide choice of languages and SDKs. Below is a simple app that serves a templated HTML website in templates/ directory. For brevity I will skip the actual template but this can simply be a vibe-coded HTML page.

import fastapi, random, uvicorn, os
import fastapi.templating

app = fastapi.FastAPI()
templates = fastapi.templating.Jinja2Templates(directory="templates")

@app.get("/")
def index(request: fastapi.Request):
    # HTML response based on Jinja template
    return templates.TemplateResponse(
        "index.html", {
            "request": request,
            "number": random.randint(5, 20)
        }
    )

if __name__ == "__main__":
    port = int(os.environ.get("PORT", 8080))
    uvicorn.run(app, host="0.0.0.0", port=port)

Now it's time for a Dockerfile. I will install FastAPI, Uvicorn and Jinja directly with pip but it's better to use actual requirements.txt in real scenarios.

FROM    python:3.14-alpine
ENV     PORT 8080
WORKDIR /app/
RUN     pip install fastapi uvicorn jinja2
COPY    . /app/
CMD     ["sh", "-c", "uvicorn main:app --host 0.0.0.0 --port ${PORT}"]

Now we can use docker provider in Terraform to build and push our application. In order to make it react to changes, I will also use a variable and increase version with each change. But in order to push, let's also define a new repository in Artifact Registry.

variable "app_version" { type = string }

resource "google_artifact_registry_repository" "website_repo" {
  location      = "europe-west4"
  repository_id = "website-repository"
  description   = "Docker repository for the website image"
  format        = "DOCKER"
  docker_config { immutable_tags = false }
}

resource "docker_image" "website_image" {
  name = "${google_artifact_registry_repository.website_repo.registry_uri}/website-image:${var.app_version}"
  build {
    context  = "./website"
    platform = "linux/amd64"
  }
}

resource "docker_registry_image" "website_image" {
  name          = docker_image.website_image.name
  keep_remotely = true
}

Now as you apply, the new image from website/ directory should be built and pushed to Artifacts Registry in Google Cloud. This way we can further create a service online that will run serve this image.

Creating Cloud Run service

First I will create a basic Cloud Run service with the image created above. Let's approach this step by step. As later we will use this service to connect to some other services, we will attach a service account in advance. Service Account is equivalent to AWS IAM Role. I will configure also the port on which it should listen. The same port will be exported to PORT environment variable. To make the Cloud Run function accessible only via Load Balancer later (and not with .run.app link), I set ingress configuration to INGRESS_TRAFFIC_INTERNAL_LOAD_BALANCER. Also the scaling will be limited to at most 3 instances.

resource "google_service_account" "cloud_run_sa" {
  account_id   = "cloud-run-website-sa"
  display_name = "Service Account for Cloud Run Website"
}

resource "google_cloud_run_v2_service" "website_service" {
  name                = "website-service"
  location            = "europe-west4"
  ingress             = "INGRESS_TRAFFIC_INTERNAL_LOAD_BALANCER"
  deletion_protection = false
  depends_on          = [docker_image.website_image, docker_registry_image.website_image]

  template {
    service_account = google_service_account.cloud_run_sa.email
    containers {
      image = "${google_artifact_registry_repository.website_repo.registry_uri}/website-image:${var.app_version}"
      ports { container_port = 8080 }
    }
  }

  scaling {
    min_instance_count    = 0
    max_instance_count    = 3
  }
}

Before we can expose Cloud Run app in the load balancer, we still need to create two things. First, we need to allow anyone to use the Service without authentication. Secondly, we need to define network endpoint group that will expose our Cloud Run Service for this region (you can use single Load Balancer Backend with multiple Network Endpoint Groups to serve same Service in multiple regions).

resource "google_cloud_run_v2_service_iam_member" "noauth" {
  location = google_cloud_run_v2_service.website_service.location
  project  = google_cloud_run_v2_service.website_service.project
  name     = google_cloud_run_v2_service.website_service.name
  role     = "roles/run.invoker"
  member   = "allUsers"
}

resource "google_compute_region_network_endpoint_group" "serverless_neg" {
  name                  = "serverless-neg"
  network_endpoint_type = "SERVERLESS"
  region                = "europe-west4"
  cloud_run { service = google_cloud_run_v2_service.website_service.name }
}

resource "google_compute_backend_service" "website_backend" {
  name                  = "website-backend"
  load_balancing_scheme = "EXTERNAL_MANAGED"
  protocol              = "HTTP"
  backend { group = google_compute_region_network_endpoint_group.serverless_neg.id }
  # security_policy = ... we will revisit it later
}

Accessing our website from Load Balancer

Now we can attach the new backend to the Load Balancer. We will replace the default routes in URL map and keep /images/ being served by the bucket backend.

resource "google_compute_url_map" "url_map" {
  name            = "global-app-lb"
  default_service = google_compute_backend_service.website_backend.id
  # Changed the following 👆
  # ...

  path_matcher {
    # And this one too 👇
    name            = "path-matcher"
    default_service = google_compute_backend_service.website_backend.id

    path_rule {
      paths   = ["/images", "/images/*"]
      service = google_compute_backend_bucket.photos_backend_bucket.id
    }
  }
}

The root of the load balancer should now serve your new Cloud Run website like on the image below.

Protecting the site with Cloud Armor

If you go to logs explorer in Google Console, you will see a lot of requests coming from malicious individuals running scanners against IPs. This is the unfortunate fact about IPv4 - it's so limited that the range to scan is very small and makes sense for hackers who want to steal your .env.

In general you would block individual IPs or ranges but in our case, we can simply use our own IP as allowlist. That's what Cloud Armor is for (it actually can do much more, like block requests for .env for example 🤩). The rules are applied to the backends and because we have two types of backends - compute for our Cloud Run and bucket for Storage Bucket, we need also two types of policies. Actually, they will look exactly the same with only change in one argument. The default behavior will be to deny and throw HTTP error code 403.

variable "ip_range" {
  type        = string
  default     = "0.0.0.0/0" # Set this to your IP range
}

resource "google_compute_security_policy" "policy" {
  name = "backend-allowed-ips"
  type = "CLOUD_ARMOR"

  rule {
    action   = "allow"
    priority = "1000"
    match {
      versioned_expr = "SRC_IPS_V1"
      config { src_ip_ranges = [var.ip_range] }
    }
    description = "Allow access from my IP range"
  }

  rule {
    action   = "deny(403)"
    priority = "2147483647"
    match {
      versioned_expr = "SRC_IPS_V1"
      config { src_ip_ranges = ["*"] }
    }
    description = "Default deny rule"
  }
}

resource "google_compute_security_policy" "edge_policy" {
  name = "edge-allowed-ips"
  type = "CLOUD_ARMOR_EDGE"

  # Exactly same rules as above!
  # ...
}

Now we can apply the new policies to our backend services. Let's revisit the resources for Cloud Run function and for the images bucket.

resource "google_compute_backend_service" "website_backend" {
  name                  = "website-backend"
  load_balancing_scheme = "EXTERNAL_MANAGED"
  protocol              = "HTTP"
  backend { group = google_compute_region_network_endpoint_group.serverless_neg.id }
  # Changed this one 👇, use normal CLOUD_ARMOR policy here
  security_policy = google_compute_security_policy.policy.id
}

resource "google_compute_backend_bucket" "photos_backend_bucket" {
  name        = "backend-bucket-${random_string.bucket_name.result}"
  bucket_name = google_storage_bucket.photos_bucket.name
  enable_cdn  = true
  # Changed this one 👇, use CLOUD_ARMOR_EDGE policy here
  edge_security_policy = google_compute_security_policy.edge_policy.id
}

If you try to access the load balancer for example using VPN, it should now show Forbidden but should work for your home IP. If you don't know what you IP is, do the following: go to https://api.ipify.org/, replace last two numbers with 0 and 0 and add /16. That way the policy will be flexible enough if you have dynamic IP. So 208.81.188.10 should become 208.81.0.0/16.

Going further

Phew! That was some long writing. In the next part we will focus on the upload process and triggering Cloud Run Function which will describe our image with Vision API.

Continue to part 2 here

How to get Python/Cloud/Java/DevOps/Admin job in this crazy market?

Piotr Pabis — Wed, 03 Dec 2025 16:38:37 +0000

In advance I want to say: I won't sell you a bootcamp, I won't send you to a Udemy course and I don't guarantee anything. I just want to share my story with you 🥲, as I believe many people are struggling to get "DevOps" or "Developer" job in this market.

Backstory

The year is 2019. I was to just graduate from a university with a Master of Engineering degree in Computer Science. With this on my resume, three years of experience at Intel (although as an intern 😅) and published apps for Android with a modest ad revenue, I went into the job hunting spree. Checkboxes:

✅ Bachelor's degree, Master's is just a formality,
✅ 3 years of experience in Big Tech,
✅ Some Python experience from the above,
✅ Java experience from Android,
✅ Some PHP websites I built in high school as "the next Facebook",
✅ Entrepreneurial spirit with proven projects.

I go for the biggest ones first: Amazon and AMD. First one, final round, rejection. Second one, "Here's a take home coding challenge", I do it, "not good enough". Maybe I should be a bit more humble next time. I apply to bigger and smaller companies for Java jobs. Silence. I apply for PHP jobs. Python. JavaScript. Silence. Android job. 200 people competing for one junior position. Of course I failed, what the hell is MVVM and how to use mocks in unit tests? (Fun fact: one company called me 2 years later for junior Java position!)

It was a disaster back then. As at the time I was a member of a Toastmaster's club, I just randomly chatted with one colleague who was a senior dev since a long time. He gave me just this simple one advice: "Go for IT helpdesk or DevOps, once you are in, it's easier to switch to Java or whatever".

First Steps into DevOps

I took his advice and picked DevOps. I had no idea what DevOps is back then. So I went home and searched what are the requirements for DevOps. I knew Linux, used it for some years already. I used Docker once or twice. Added these details to my resume. I sent maybe 3 or 4 applications. Got two responses. Went to the first interview. I said "I know some basic Docker, I know Linux, I want just a trial period, I will manage to do this/I will figure it out". The one in bold might have been the deciding factor 😂 and I got the job 🎉!

Everything afterwards was just smooth sailing ⛵️. Recruiters started to contact me themselves, I switched jobs multiple times. But most importantly, I learned whatever I could at work and tried hard to be useful, whether it was writing some Python scripts or replacing power cables in desk computers.

Fast forward to today

I have more than 5 years of experience in DevOps and many certifications at this point. I don't say this to brag but to give you a specific perspective: I almost get zero job offers this year. A recruiter contacts me once every two months at most. Even if I apply, there's almost no response. If I even get to a call, I don't get any exciting offers. Nowhere in the process did anyone ask me for certifications. I even applied to AWS and nowhere in the process I was asked or said anything about certifications. The only thing that mattered was problem solving, and grind and persistence (or as they call it in Amazon - following Leadership Principles 😆). What I want to underline here as well is that recruiters have a bias towards people who are currently employed, somewhat of a social proof of sort. Maybe it's subconscious, I don't know, I was never a headhunter 😝.

Let's go back to the title of this post and review some tips that are shared around by professionals. First: do and publish projects on GitHub. Far from it. First of all, it's 2025 and you can just vibe code it, and even if we assume that AI doesn't exist, nobody has time to look at your React ToDo app - and 200 other ones sent in by other candidates. If you have something very useful and unique, you might have a chance with that. At best, the time to show your project is to demonstrate it working during the interview. Second one: bootcamps - unless you are switching from one language to another, let's say Java bootcamp while being a Python pro, or Cloud Architect while having experience as Software Product Manager, this is just waste of money. You might learn coding but your job chances will likely stay on the same level. There's another reason I will explain in digression below. Third one: certifications - although everyone clearly state all the time "certification won't get you a job" - which is true, the second part is too vague "only hands on experience matters". But let me also digress the certifications topic at the end of this post.

That leaves us with one tip I want to give you today:

Get any job in IT.

Whether you wanted a Java role and they offer you entry-level database admin, take it. If you wanted to be a Cloud Engineer but what you find is a sysadmin of physical servers, do it. IT person at a local school? You are the one. Assistant to the guy setting up routers in offices? You pass him the screwdriver. If you have to carry a server rack, you carry it. If someone needs ethernet cable or new Windows installation, you crawl under the desk to connect it and plug in USB stick with Windows ISO. If someone calls you asking how to install Adobe Reader for the third time today, you take the call and instruct them patiently step-by-step. I'm not saying this to make you do things out of your contract (only if you really want to yourself) but to tell you that someone has to do it and companies are in need of such specialists. Repeating one fact: the most valuable point on your resume is being currently employed. And far fewer people flock to on-site physical IT, or repeatable less creative jobs nowadays. (Tbh, most jobs are quite frustrating often. Look at game development. Video games are shiny, creative, and fun but in reality its development is the quickest path to burnout and crunch.)

If this company has software engineers, you are even luckier - ask to join stand-ups just to listen. Ask one developer to look over as they work. Ask them if you can ask questions - you will be surprised how many people are willing to help and teach you (maybe because it makes them feel important and smart? I don't know 🤣). With that you might have opportunity to switch to become SWE. Hiring internally is way easier - even more powerful than referrals.

You will have a modest salary (don't work for free though ❗️), put something what is left over aside. This is your budget for upskilling. Save for courses, certifications, books, whatever you are curious about. Buy a domain for a year, get a VPS server for a month. Buy a used tablet to test your apps. Take a Python course and value every minute of it - you earned this money and it was hard. This will give you finally opportunities to get where you want to be.

And if you are adventurous, or there's no IT offers around you, try getting any job. Maybe introducing Excel to a local Mom-and-Pop shop will be the way. Maybe you can find a book in a local library about HTML or data analytics and you will volunteer to teach it to kids. Google Forms for a local charity - isn't this IT? Any such small and trivial thing will move you closer to the path you want. If you get hired for a different position in an office where there is an IT department, it's even better - this is the same as above but with one extra step. On the other hand you might be the first pillar to establish one in the existing business.

But maybe... Maybe you will just love the job you got. I did. I never expected to be where I am now.

I get you, you spent so much time learning already. You did all those projects, all these courses, bootcamps and now you have to do something else completely? Not use any of those skills you spent time and energy to acquire? This is harsh reality, unfortunately. I don't say you should give up on that thing you learned. What I want to present you is a strategy to actually realize this in the future. Because statistically if you sent 1000 applications to various companies for this specific Typescript Developer job, 1001st is not likely to work either. Again this another opening will get hundreds of resumes. Get into the game first, then move forward on the board 🎲.

My suggestion contradicts the common advice of focusing on one thing and trying hard to get there. I advise you to diffuse and be open. Prepare quickly before the interview, get some basic knowledge about the things in the job description. Learn on the job, be curious, try being useful at every opportunity, at every hour you are in the office. Don't be afraid to ask questions. If you have an idea, don't just say it - implement it - even if your manager says "it's not worth the effort", but you believe it's the best what should be done - do it (just not directly on production 🤣). Trust me, implemented example of your idea is the strongest convincing force to any naysayer.

Digression: why or rather when certifications work

It is often said that you just need hands-on experience. Some will advice to not spend hundreds of dollars on the exams and just get the Udemy course during a sale for $10 and do the exercises along. This is as effective as telling someone to just eat less to lose weight, or cheer up and look at the beauty of the world around to not be sad when you are rejected from every job offer since a year. Both the courses and bootcamps have one thing in common: the pushing force of "sunk cost fallacy". However, as the time passes by, this momentum fades away and you just accept the money you "lost". One force that is pulling on the other side, is the certification, the exam. You put money on the line on the other end, so you subconsciously want to prepare for it and not just binge watch Udemy. However, both forces need to be close enough to each other in time axis. And note for AWS and other organizers: 75%-off vouchers do work 😉. End of digression.

AWS IAM Outbound OIDC with Google Cloud Identity Pool

Piotr Pabis — Mon, 24 Nov 2025 08:00:00 +0000

Recently, AWS introduced a new feature in IAM, that allows you to sign JWT tokens using managed OIDC provider. On a per AWS Account basis you can enable and receive unique endpoint with managed JWKS keys to configure third-party identity pools. If you want to know more how it functions, I have a blog post how to host your own OIDC provider on CloudFront.

As I'm learning Google Cloud now, I decided to use GCP's Workload Identity Pool as the receiving end of the token issued by AWS IAM (even though this service also supports native AWS authentication). The goal for today is to make a Lambda function be able to write to Google Cloud Storage bucket and insert some rows into BigQuery table. The whole setup is on the diagram below:

For each of the steps below, I assume you are already authenticated to both AWS and Google Cloud CLIs and have necessary permissions to create all resources.

Repository with the whole code you can find here: github.com/ppabis/aws-outbound-oidc-google-cloud.

Configuring AWS Account and GCP Project

For now this new feature is not yet available in Terraform, so you will need to use latest AWS CLI and enable it using the command below. In case you have it already enabled, you can use the second one to get the issuer URL again. Run this in the root of your new project.

aws iam enable-outbound-web-identity-federation --output json > issuer.json
# Alternatively, if already enabled:
aws iam get-outbound-web-identity-federation-info --output json > issuer.json

This will create a file issuer.json with the OIDC endpoint that we will need later. Next, let's enable some services on Google Cloud:

gcloud services enable sts.googleapis.com
gcloud services enable storage.googleapis.com
gcloud services enable bigquery.googleapis.com

Terraform providers

Let's create a new provider.tf file that will configure our required Terraform providers to use in this project. We will need AWS v6.x, Google v7.x and archive to upload the Lambda. Remember to add your GCP project ID and select your preferred regions. Put your GCP project ID into terraform.tfvars file.

terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 6.0"
    }

    archive = {
      source  = "hashicorp/archive"
      version = "~> 2.0"
    }

    google = {
      source  = "hashicorp/google"
      version = "~> 7.0"
    }
  }
}

provider "aws" {
  region = "eu-west-1"
}

variable "gcp_project_id" {
  type        = string
}

provider "google" {
  project               = var.gcp_project_id
  region                = "europe-west1"
  user_project_override = true
}

Google Cloud Workload Identity Pool

Next, we will create an identity pool and identity pool provider that will authenticate JWT tokens issued by AWS. We will use the JSON file we obtained earlier. First define some identity pool, I named my my-aws-app. Then create a provider in this pool, I named it the same. Remember to add attribute mapping so that AWS users and roles are reflected as GCP subjects - that way we will later configure grants in GCP IAM. In this step we can technically also choose audiences we will accept but I want to use the default one created by Google. It is complex enough to be secure 😅.

locals {
  issuer = jsondecode(file("issuer.json")).IssuerIdentifier
}

resource "google_iam_workload_identity_pool" "my_aws_app" {
  workload_identity_pool_id = "my-aws-app"
  display_name              = "my-aws-app"
}

resource "google_iam_workload_identity_pool_provider" "my_aws_app" {
  workload_identity_pool_id          = google_iam_workload_identity_pool.my_aws_app.workload_identity_pool_id
  workload_identity_pool_provider_id = "my-aws-app"

  oidc {
    issuer_uri = local.issuer
  }

  attribute_mapping = {
    "google.subject" = "assertion.sub"
  }
}

Resources on GCP side

I will create some sample resources in GCP that we will write to from the Lambda function. I chose a storage bucket and a BigQuery table. You can experiment with other services as well, it's just a matter of your imagination.

resource "random_id" "bucket_name" { byte_length = 8 }

resource "google_storage_bucket" "test_bucket" {
  name                        = "my-test-bucket-${random_id.bucket_name.hex}"
  location                    = "europe-west1"
  storage_class               = "STANDARD"
  uniform_bucket_level_access = true
  force_destroy               = true
}

resource "google_bigquery_dataset" "my_dataset" {
  dataset_id                 = "my_dataset"
  friendly_name              = "my_dataset"
  location                   = "europe-west1"
  delete_contents_on_destroy = true
}

resource "google_bigquery_table" "my_table" {
  dataset_id          = google_bigquery_dataset.my_dataset.dataset_id
  table_id            = "my_table"
  friendly_name       = "my_table"
  deletion_protection = false

  schema = jsonencode([
    {
      name = "timestamp"
      type = "TIMESTAMP"
      mode = "REQUIRED"
    },
    {
      name = "message"
      type = "STRING"
      mode = "NULLABLE"
    }
  ])
}

For now we can't yet assign any permissions for out AWS side to these resources as we don't know the ARN of the role yet. Let's switch to AWS side now.

Creating AWS IAM Role for Lambda

Our role will be a very simple one - it needs to be assumable by Lambda and what it will do is just get JWT OIDC tokens from STS. To follow security best practices, we will only allow it to create tokens for specific audience - the one created by Google for our project. Unfortunately, this value is not emitted by any Terraform resource so we have to construct it from string. The comparison we will do in the condition is sts:IdentityTokenAudience. The audience on GCP side looks like this: //iam.googleapis.com/projects/PROJECT_NUMBER/locations/global/workloadIdentityPools/POOL_ID/providers/PROVIDER_ID. This huge 300 character long string is correct, trust me 😂. It's not the last time we will use it.

data "google_project" "current" {}

data "aws_iam_policy_document" "assume_role_policy" {
  statement {
    actions = ["sts:AssumeRole"]
    effect  = "Allow"
    principals {
      type        = "Service"
      identifiers = ["lambda.amazonaws.com"]
    }
  }
}

resource "aws_iam_role" "LambdaOutboundSTSRole" {
  name               = "LambdaOutboundSTSRole"
  assume_role_policy = data.aws_iam_policy_document.assume_role_policy.json
}

resource "aws_iam_role_policy_attachment" "lambda_policy_attachment" {
  role       = aws_iam_role.LambdaOutboundSTSRole.name
  policy_arn = "arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole"
}

data "aws_iam_policy_document" "lambda_policy" {
  statement {
    actions   = ["sts:GetWebIdentityToken"]
    effect    = "Allow"
    resources = ["*"]
    condition {
      test     = "StringEquals"
      variable = "sts:IdentityTokenAudience"
      values   = ["//iam.googleapis.com/projects/${data.google_project.current.number}/locations/global/workloadIdentityPools/${google_iam_workload_identity_pool.my_aws_app.workload_identity_pool_id}/providers/${google_iam_workload_identity_pool_provider.my_aws_app.workload_identity_pool_provider_id}"]
    }
  }
}

resource "aws_iam_role_policy" "lambda_policy" {
  name   = "lambda_policy"
  role   = aws_iam_role.LambdaOutboundSTSRole.id
  policy = data.aws_iam_policy_document.lambda_policy.json
}

Apply the infrastructure so far, to catch if there are any issues. In your GCP project, you should see a new workload identity pool created and in AWS you should see the IAM role. GCP should also have a new storage bucket and a BigQuery table. With that we can continue with permissions in GCP.

Granting Permissions in GCP

As we now know what role will authenticate from AWS, we can grant permissions on our new GCP resources to it. If you remember the last audience string, this 250 character long member string shouldn't scare you anymore. The principal we have to grant to has the following format: principal://iam.googleapis.com/projects/PROJECT_NUMBER/locations/global/workloadIdentityPools/POOL/subject/ROLE_ARN. Notice the change: we use only identity pool ID here (not provider).

To follow best security practices, we will also limit the permissions using IAM conditions. Compare this to AWS: in AWS you use Resource key in the IAM policy to limit the policy and in GCP you need to use condition, and it's not in the policy, rather in the binding. The roles (in AWS terms policies) we will give are roles/storage.objectAdmin for the bucket and roles/bigquery.dataEditor for the BigQuery dataset. We will limit them to only our single bucket and BigQuery dataset (all tables).

data "google_iam_role" "bucket_object_admin" { name = "roles/storage.objectAdmin" }
data "google_iam_role" "bigquery_data_editor" { name = "roles/bigquery.dataEditor" }

resource "google_project_iam_member" "aws_lambda_incoming_bucket_role_member" {
  project = var.gcp_project_id
  role    = data.google_iam_role.bucket_object_admin.name
  member  = "principal://iam.googleapis.com/projects/${data.google_project.current.number}/locations/global/workloadIdentityPools/${google_iam_workload_identity_pool.my_aws_app.workload_identity_pool_id}/subject/${aws_iam_role.LambdaOutboundSTSRole.arn}"
  condition {
    title      = "RestrictToTestBucket"
    expression = "resource.name.startsWith(\"projects/_/buckets/${google_storage_bucket.test_bucket.name}/objects/\")"
  }
}

resource "google_project_iam_member" "aws_lambda_incoming_bigquery_role_member" {
  project = var.gcp_project_id
  role    = data.google_iam_role.bigquery_data_editor.name
  member  = "principal://iam.googleapis.com/projects/${data.google_project.current.number}/locations/global/workloadIdentityPools/${google_iam_workload_identity_pool.my_aws_app.workload_identity_pool_id}/subject/${aws_iam_role.LambdaOutboundSTSRole.arn}"
  condition {
    title      = "RestrictToMyDataset"
    expression = "resource.name.startsWith(\"projects/${data.google_project.current.project_id}/datasets/${google_bigquery_dataset.my_dataset.dataset_id}\")"
  }
}

We are technically done on setting up Google Cloud side. Let's move to the coding part now.

Lambda Function

To comfortable interact with Google Cloud services from our Lambda, we will install some requirements using a requirements.txt file. Create it in a new directory called aws_lambda. I have also included boto3 as the current Lambda runtime doesn't have the latest version that supports the new get_web_identity_token function.

google-auth
google-cloud-iam
google-cloud-storage
google-cloud-bigquery
boto3

Install the requirements using pip directly into this directory. That way we will zip all the dependencies for Lambda and push it into AWS.

pip install -r requirements.txt -t aws_lambda/

In order to exchange OIDC token for actual working GCP working token that will be recognized by GCP IAM we need to use Google STS service. The code below seems a bit overwhelming compared to AWS STS calls but it's necessary. We need to provide all the grant types, scopes, etc. Save this into aws_lambda/auth.py.

from google.auth.transport.requests import Request
from google.oauth2 import sts
from google.oauth2.credentials import Credentials
import boto3

aws_sts_client = boto3.Session().client("sts", region_name="us-east-1")

def _exchange_aws_token_for_google_token(aws_token: str, audience: str) -> str:
    """Exchange AWS token for Google service account token via STS"""
    request = Request()
    google_sts_client = sts.Client(token_exchange_endpoint="https://sts.googleapis.com/v1/token")

    try:
        result = google_sts_client.exchange_token(
            request=request,
            grant_type="urn:ietf:params:oauth:grant-type:token-exchange",
            audience=audience,
            requested_token_type="urn:ietf:params:oauth:token-type:access_token",
            subject_token=aws_token,
            subject_token_type="urn:ietf:params:oauth:token-type:id_token",
            scopes=["https://www.googleapis.com/auth/cloud-platform"],
        )
        return result["access_token"]
    except Exception as e:
        print(f"Error exchanging token: {e}")
        raise

def authenticate_aws_oidc_to_google(audience: str) -> Credentials:
    """Authenticate AWS OIDC to Google using the AWS STS client."""
    aws_token = aws_sts_client.get_web_identity_token(
        Audience=[audience],
        DurationSeconds=3600,
        SigningAlgorithm='ES384'
    )['WebIdentityToken']

    google_token = _exchange_aws_token_for_google_token(aws_token, audience)

    return Credentials(token=google_token)

The function above returns Google Credentials object that we can use later for any client we need. In order to easily interface with GCP services, we will use provider Python libraries. I will also use some utilities like datetime and os. We need to also load the bucket name, dataset name, table name and audience from environment variables.

import os, json
from datetime import datetime
from google.cloud import bigquery, storage
from auth import authenticate_aws_oidc_to_google

AUDIENCE = os.getenv("AUDIENCE")
GOOGLE_BUCKET = os.getenv("GOOGLE_BUCKET")
BQ_DATASET = os.getenv("GOOGLE_BQ_DATASET")
BQ_TABLE = os.getenv("GOOGLE_BQ_TABLE")
GOOGLE_PROJECT = os.getenv("GOOGLE_PROJECT_ID")

Let's define our operations to perform as separate functions for easier understanding. First function will be to create an object in Google Storage bucket. It will be dependent on the time and we will also add Lambda's request ID for uniqueness.

def create_object_in_bucket(creds, request_id: str) -> str:
    """Create a random object in the bucket under hello_<timestamp>.txt"""
    client = storage.Client(project=GOOGLE_PROJECT, credentials=creds)
    now = datetime.now()
    blob = client.bucket(GOOGLE_BUCKET).blob(f"hello_{now.timestamp():.0f}.txt")
    blob.upload_from_string(f"Hello, world! Current time: {now.isoformat()}, request_id: {request_id}")
    return blob.public_url

Another function will insert a row into our BigQuery table. We will also populate the message field with some text, current time and request ID from Lambda.

def create_row_in_bigquery(creds, request_id: str) -> bool:
    """Create a row in the bigquery table with the current time and request_id"""
    client = bigquery.Client(project=GOOGLE_PROJECT, credentials=creds)
    table = f"{client.project}.{BQ_DATASET}.{BQ_TABLE}"
    row = {
        "timestamp": datetime.now().isoformat(),
        "message": f"Hello, world! Current time: {datetime.now().isoformat()}, request_id: {request_id}"
    }
    err = client.insert_rows_json(table, [row])
    if err:
        print(f"Error BQ: {err}")
        return False
    return True

Lastly, we can glue everything together in the main Lambda handler. It will first authenticate to Google using our AWS provided JWT and try to call both functions defined above. It will also log result of each operation so that we can see it in CloudWatch logs.

def lambda_handler(event, context):
    try:
        creds = authenticate_aws_oidc_to_google(AUDIENCE)
        url = create_object_in_bucket(creds, context.aws_request_id)
        success = create_row_in_bigquery(creds, context.aws_request_id)
        print(f"Object URL: {url}, BigQuery success: {success}")
        return {"statusCode": 200, "body": json.dumps({"message": f"Object URL: {url}, BigQuery success: {success}"})}
    except Exception as e:
        print(f"Error: {e}")
        return {"statusCode": 500, "body": json.dumps({"message": f"Error: {e}"})}

Packaging and Deploying Lambda

Now we need to package the Lambda code into .zip file and deploy it to AWS. Also very important is to set the environment variables we use in the code.

data "archive_file" "lambda_function" {
  type        = "zip"
  source_dir  = "aws_lambda"
  output_path = "aws_lambda.zip"
}

resource "aws_lambda_function" "oidc_to_gcp" {
  function_name    = "LambdaOIDCtoGCP"
  role             = aws_iam_role.LambdaOutboundSTSRole.arn
  handler          = "main.lambda_handler"
  runtime          = "python3.12"
  timeout          = 30
  source_code_hash = data.archive_file.lambda_function.output_base64sha256
  filename         = data.archive_file.lambda_function.output_path
  environment {
    variables = {
      AUDIENCE          = "//iam.googleapis.com/projects/${data.google_project.current.number}/locations/global/workloadIdentityPools/${google_iam_workload_identity_pool.my_aws_app.workload_identity_pool_id}/providers/${google_iam_workload_identity_pool_provider.my_aws_app.workload_identity_pool_provider_id}"
      GOOGLE_BUCKET     = google_storage_bucket.test_bucket.name
      GOOGLE_BQ_DATASET = google_bigquery_dataset.my_dataset.dataset_id
      GOOGLE_BQ_TABLE   = google_bigquery_table.my_table.table_id
      GOOGLE_PROJECT_ID = data.google_project.current.project_id
    }
  }
}

Testing

Let's run the Lambda function and see if we can do stuff on GCP side. You can use AWS Console to trigger it or use the following CLI command (unfortunately you have to store response in a file for whatever reason 🙄)

aws lambda invoke \
 --region eu-west-1 \
 --function-name LambdaOIDCtoGCP \
 --payload '{}' \
 response.json

We can now look into CloudWatch logs to see all the print statements from our function to see if everything went well. You should get a similar output:

STREAM=$(aws logs describe-log-streams \
 --log-group-name /aws/lambda/LambdaOIDCtoGCP \
 --query 'logStreams[0].logStreamName' \
 --output text)

aws logs get-log-events \
 --log-group-name /aws/lambda/LambdaOIDCtoGCP
 --log-stream-name "$STREAM" \
 --output text \
 --query 'events[].[timestamp,message]'

1763923647520   START RequestId: a2d3bf7c-3bf5-4c6b-9f55-36febec82c96 Version: $LATEST
1763923648706   Object URL: https://storage.googleapis.com/my-test-bucket-fc436b0fe1b7c943/hello_1763923648.txt
1763923649085   BigQuery success: True
1763923649092   END RequestId: a2d3bf7c-3bf5-4c6b-9f55-36febec82c96

Take the object URL and replace https://storage.googleapis.com/ with gs://. Use gsutil to see the list of files in the bucket as well as contents of the file created. Compare the request ID in the object with the one in the logs.

$ gsutil ls gs://my-test-bucket-fc436b0fe1b7c943/
gs://my-test-bucket-fc436b0fe1b7c943/hello_1763923648.txt
$ gsutil cat gs://my-test-bucket-fc436b0fe1b7c943/hello_1763923648.txt
Hello, world! Current time: 2025-11-23T18:47:28.288871, request_id: a2d3bf7c-3bf5-4c6b-9f55-36febec82c96

We can also query BigQuery table to see if the records were inserted as we wanted. Use bq tool to run a simple SELECT statement.

$ bq query "SELECT * FROM my_dataset.my_table"                        
+---------------------+----------------------------------------------------------------------------------------------------------+
|      timestamp      |                                                 message                                                  |
+---------------------+----------------------------------------------------------------------------------------------------------+
| 2025-11-23 18:47:28 | Hello, world! Current time: 2025-11-23T18:47:28.706698, request_id: a2d3bf7c-3bf5-4c6b-9f55-36febec82c96 |
+---------------------+----------------------------------------------------------------------------------------------------------+

Checking if permissions are enforced

To verify that our IAM conditions in Google Cloud are working, I will create a second bucket without granting any permissions to our AWS role and run Lambda again. It should error out when trying to upload the content to GCS.

resource "google_storage_bucket" "second_bucket" {
  name                        = "my-second-bucket-${random_id.bucket_name.hex}"
  location                    = "europe-west1"
  storage_class               = "STANDARD"
  uniform_bucket_level_access = true
  force_destroy               = true
}

# In Lambda environment variables, change GOOGLE_BUCKET to:
resource "aws_lambda_function" "oidc_to_gcp" {
  function_name    = "LambdaOIDCtoGCP"
  # ...
  environment {
    variables = {
      #...
      GOOGLE_BUCKET     = google_storage_bucket.second_bucket.name
    }
  }
}

Invoke the function again and get new log stream. You should see some error just like me below.

aws lambda invoke \
 --region eu-west-1 \
 --function-name LambdaOIDCtoGCP \
 --payload '{}' \
 response.json

STREAM=$(aws logs describe-log-streams \
 --log-group-name /aws/lambda/LambdaOIDCtoGCP \
 --query 'logStreams[0].logStreamName' \
 --output text)

aws logs get-log-events \
 --log-group-name /aws/lambda/LambdaOIDCtoGCP
 --log-stream-name "$STREAM" \
 --output text \
 --query 'events[].[timestamp,message]'

1763924479236   START RequestId: 6a7d8016-ba34-4075-bca0-77d3bcb2c7a7 Version: $LATEST
1763924480076   Creds: <google.oauth2.credentials.Credentials object at 0x7fef0787e9c0>
1763924480586   Error: 403 POST https://storage.googleapis.com/upload/storage/v1/b/my-second-bucket-fc436b0fe1b7c943/o?uploadType=multipart: {
1763924480586   "error": {
1763924480586   "code": 403,
1763924480586   "message": "Caller does not have storage.objects.create access to the Google Cloud Storage object. Permission 'storage.objects.create' denied on resource (or it may not exist).",

Conclusion

This new outbound OIDC feature in AWS IAM is really simple to use. As you have seen in this project, most of the complexity was on Google Cloud's side. We barely did any setup for AWS itself. With that, you can now integrate IAM directly with your applications if they support OIDC.

Let's try Managed ECS Instances

Piotr Pabis — Wed, 22 Oct 2025 18:36:46 +0000

If you use ECS, you should already know about the two ways of running the containers: on EC2 machines or using serverless technology - Fargate. If you choose EC2 instances then you have to take care of the autoscaling groups, using the latest AMI. Fargate on the other hand is limited regarding shared volumes and memory and CPU selections. However, on 30th of September 2025 AWS announced a hybrid approach: managed EC2 instances. It gives you the savings of EC2 instances with as little management overhead as Fargate. Today, we will try this out!

Base infrastructure

As I assume that you are already an experienced user, I won't cover the basics like VPC and how to use ECS. We will start from a basic infrastructure: VPC, ECS Cluster, Application Load Balancer and some other necessities like IAM roles and Security Groups. The image below shows what we aim to create.

Both our EC2 instances and ECS service will be running in private subnets. Public one will be only used by the ALB and fck-nat instance providing us the Internet and save on endpoint costs. You can find the Terraform code here: github.com/ppabis/managed-ecs-instances. I will skip description of things like vpc.tf.

Defining security for Managed Instances

In advance I can say that we will need to define three resources: IAM role for the capacity provider (which is not a service role), IAM role/profile for individual EC2 instances and their security groups. Let's do it now.

Infrastructure role

The first role we have to create is the infrastructure role. Fortunately, there is already a policy that will cover all the required permissions and we just need to attach it. Infrastructure role for the capacity provider trusts service ecs.amazonaws.com.

data "aws_iam_policy_document" "trust_policy" {
  statement {
    actions = ["sts:AssumeRole"]
    effect  = "Allow"
    principals {
      type        = "Service"
      identifiers = ["ecs.amazonaws.com"]
    }
  }
}

resource "aws_iam_role" "ecs_infrastructure" {
  name               = "ECSInfraRole"
  assume_role_policy = data.aws_iam_policy_document.trust_policy.json
}

resource "aws_iam_role_policy_attachment" "ecs_infrastructure" {
  role       = aws_iam_role.ecs_infrastructure.name
  policy_arn = "arn:aws:iam::aws:policy/AmazonECSInfrastructureRolePolicyForManagedInstances"
}

EC2 instance role

Another one that we need to define is the role and profile for individual EC2 instances. They also need to communicate with some services. Just as with above the permissions are already defined in a policy created by AWS. Also note that you should name the instance role starting with ecsInstanceRole to be able to use the default AWS managed policy. Otherwise you would need to define another permission to allow to pass role by the infrastructure role.

data "aws_iam_policy_document" "trust_policy_instance" {
  statement {
    actions = ["sts:AssumeRole"]
    effect  = "Allow"
    principals {
      type        = "Service"
      identifiers = ["ec2.amazonaws.com"]
    }
  }
}

resource "aws_iam_role" "ecs_managed_instance" {
  name               = "ecsInstanceRoleManaged"
  assume_role_policy = data.aws_iam_policy_document.trust_policy_instance.json
}

resource "aws_iam_instance_profile" "ecs_managed_instance" {
  name = "ECSManagedInstanceProfile"
  role = aws_iam_role.ecs_managed_instance.name
}

resource "aws_iam_role_policy_attachment" "ecs_instance" {
  role       = aws_iam_role.ecs_managed_instance.name
  policy_arn = "arn:aws:iam::aws:policy/AmazonECSInstanceRolePolicyForManagedInstances"
}

Security group

I will give the instances complete egress access as there are no inbound requirements. This is mostly used to tag the traffic coming from the instances in case you want to allow them to communicate for example to a VPC interface endpoint.

resource "aws_security_group" "managed-instance-sg" {
  name = "ManagedECSInstancesSG"  
  vpc_id = module.vpc.vpc_id
  egress {
    from_port = 0
    to_port = 0
    protocol = "-1"
    cidr_blocks = ["0.0.0.0/0"]
  }
}

Defining the capacity provider

In order to use Managed ECS Instances, we need to define a capacity provider just as we did with classic EC2 autoscaling group. This time however, the parameters are a bit different. Make sure that you use Terrafrom AWS provider version 6.15 or newer.

First we define the name of the provider and we directly set the cluster for which it will be created. The resource type is aws_ecs_capacity_provider.

resource "aws_ecs_cluster" "ec2-managed" {
  name = "ec2-managed"
}

resource "aws_ecs_capacity_provider" "ec2-managed" {
  name    = "ec2-managed"
  cluster = aws_ecs_cluster.ec2-managed.name

  managed_instances_provider {
    # more parameters follow ...
  }
}

Next we need to define the managed_instances_provider block. First, we give need to give it the infrastructure role. I will also add an option to propagate tags of this provider to the instances, so that we can easily find them.

resource "aws_ecs_capacity_provider" "ec2-managed" {
  # ...
  managed_instances_provider {
    infrastructure_role_arn = aws_iam_role.ecs_infrastructure.arn
    propagate_tags          = "CAPACITY_PROVIDER"

    instance_launch_template {
      # more parameters here ...
    }
  }
}

Next nested block is instance_launch_template. Here we define all the details of how the EC2 instances should be shaped. Let's start by defining the IAM role/profile for the instance, configuring monitoring level and storage size. I will also use private subnets and the new security group in the networking config.

resource "aws_ecs_capacity_provider" "ec2-managed" {
  # ...
  managed_instances_provider {
    # ...
    instance_launch_template {
      ec2_instance_profile_arn = aws_iam_instance_profile.ecs_managed_instance.arn
      monitoring               = "BASIC"

      storage_configuration {
        storage_size_gib = 16
      }

      network_configuration {
        subnets         = module.vpc.private_subnets
        security_groups = [aws_security_group.managed-instance-sg.id]
      }

      instance_requirements {
        # ... more parameters
      }
    }
  }
}

The last nested block inside instance_launch_template is instance_requirements. Here we define the CPU and memory sizes, instance generations and CPU manufacturers to select. You can also optionally choose GPUs and FPGAs. I will choose only Graviton instances of current and previous generation with between 2-4 CPUs and 2GB-16GB of memory. I want to also avoid bare metal instances and include burstable (t4g and t3g) instances.

resource "aws_ecs_capacity_provider" "ec2-managed" {
  # ...
  managed_instances_provider {
    # ...
    instance_launch_template {
      # ...
      instance_requirements {
        memory_mib {
          min = 2048
          max = 16384
        }

        vcpu_count {
          min = 2
          max = 4
        }

        instance_generations = ["current", "previous"]
        cpu_manufacturers    = ["amazon-web-services"]
        burstable_performance = "included"
        bare_metal            = "excluded"
      }
    }
  }
}

The next part is very important as well. Even though if you now go to the AWS Console and look at the cluster, it will show that there's indeed the Managed Instances capacity provider. However, there's no capacity strategy set. It is different than with Fargate, where you can just set launch type to "FARGATE" in the service you are good to go on any cluster. You can bind the capacity directly to the service but it's better to control it at the cluster level for a cleaner setup. Thus we need to define the default strategy.

resource "aws_ecs_cluster_capacity_providers" "ec2-managed" {
  cluster_name       = aws_ecs_cluster.ec2-managed.name
  capacity_providers = [aws_ecs_capacity_provider.ec2-managed.name]

  default_capacity_provider_strategy {
    capacity_provider = aws_ecs_capacity_provider.ec2-managed.name
    base              = 0
    weight            = 100
  }
}

Load balancer to preview the service

In order to test the service, I will create a load balancer with HTTP listener. It is a pretty basic setup for the ECS services. It is important to set the target group target type to ip and configure an appropriate health check. In case of needed quicker redeployments I will also define a shorter deregistration delay.

resource "aws_security_group" "alb" {
  name        = "test-ecs-managed-alb"
  vpc_id      = module.vpc.vpc_id

  ingress {
    from_port   = 80
    to_port     = 80
    protocol    = "tcp"
    cidr_blocks = ["0.0.0.0/0"]
  }

  egress {
    from_port   = 0
    to_port     = 0
    protocol    = "-1"
    cidr_blocks = ["0.0.0.0/0"]
  }
}

resource "aws_lb" "alb" {
  name               = "test-ecs-managed-lb"
  security_groups    = [aws_security_group.alb.id]
  subnets            = module.vpc.public_subnets

  enable_deletion_protection = false
}

resource "aws_lb_target_group" "test-ecs" {
  name                 = "test-ecs-managed-tg"
  port                 = 80
  protocol             = "HTTP"
  vpc_id               = module.vpc.vpc_id
  target_type          = "ip"
  deregistration_delay = 10

  health_check {
    path                = "/"
    port                = 80
    protocol            = "HTTP"
    healthy_threshold   = 2
    unhealthy_threshold = 2
    timeout             = 5
    interval            = 30
    matcher             = "200-499"
  }
}

resource "aws_lb_listener" "http" {
  load_balancer_arn = aws_lb.alb.arn
  port              = 80
  protocol          = "HTTP"

  default_action {
    type             = "forward"
    target_group_arn = aws_lb_target_group.test-ecs.arn
  }
}

We will be then able to use the load balanacer and the target group to preview our containers and see if they are working. I will use just default Nginx from ECR public gallery as the test.

Task definition

To define a task definition we need first to have some IAM roles that will be attached as task role and as execution role. The task role doesn't need any permissions for now as we won't be accessing any internal AWS services. For execution role we will just use the default managed policy.

data "aws_iam_policy_document" "ecs_task_assume_role" {
  statement {
    effect  = "Allow"
    actions = ["sts:AssumeRole"]
    principals {
      type        = "Service"
      identifiers = ["ecs-tasks.amazonaws.com"]
    }
  }
}

resource "aws_iam_role" "task" {
  name               = "test-managed-ecs-task-role"
  assume_role_policy = data.aws_iam_policy_document.ecs_task_assume_role.json
}

resource "aws_iam_role" "execution" {
  name               = "test-managed-ecs-execution-role"
  assume_role_policy = data.aws_iam_policy_document.ecs_task_assume_role.json
}

resource "aws_iam_role_policy_attachment" "execution" {
  role       = aws_iam_role.execution.name
  policy_arn = "arn:aws:iam::aws:policy/service-role/AmazonECSTaskExecutionRolePolicy"
}

Another thing is a security group. I will add one that will allow the load balancer to access our Nginx container on port 80 and the container should be able to reach out to the wide Internet if necessary.

resource "aws_security_group" "service-sg" {
  name   = "test-ecs-service-sg"
  vpc_id = module.vpc.vpc_id

  ingress {
    from_port       = 80
    to_port         = 80
    protocol        = "tcp"
    security_groups = [aws_security_group.alb.id]
  }

  egress {
    from_port   = 0
    to_port     = 0
    protocol    = "-1"
    cidr_blocks = ["0.0.0.0/0"]
  }
}

Now we can create a task definition. I will enforce it to use ARM64 architecture and require compatibility of "MANAGED_INSTANCES". Apparently this is required for the task to deploy using this capacity provider. As previously said, I will use public ECR gallery Nginx image and map port 80. We will use awsvpc networking mode to have a new network interface created for each of our tasks. For this task I will use 0.5 vCPU and 1GB of RAM.

resource "aws_ecs_task_definition" "test-ecs" {
  family                   = "test-ecs-task"
  task_role_arn            = aws_iam_role.task.arn
  execution_role_arn       = aws_iam_role.execution.arn
  cpu                      = "512"
  memory                   = "1024"
  network_mode             = "awsvpc"
  requires_compatibilities = ["MANAGED_INSTANCES"]

  runtime_platform {
    operating_system_family = "LINUX"
    cpu_architecture        = "ARM64"
  }

  container_definitions = jsonencode([{
      name      = "web"
      image     = "public.ecr.aws/nginx/nginx:latest"
      essential = true
      portMappings = [{
          containerPort = 80
          hostPort      = 80
          protocol      = "tcp"
      }]
  }])
}

Lastly we can create the service that will spawn the new tasks and add them to the load balancer. For the first try I will just set desired count to 1. Add capacity_provider_strategy to ignored, as we don't want to bind the service with this capacity. Also otherwise any redeployment will fail.

resource "aws_ecs_service" "test-ecs" {
  name                    = "test-ecs-service"
  cluster                 = aws_ecs_cluster.ec2-managed.name
  desired_count           = 1
  task_definition         = aws_ecs_task_definition.test-ecs.arn
  enable_ecs_managed_tags = true

  network_configuration {
    subnets         = module.vpc.private_subnets
    security_groups = [aws_security_group.service-sg.id]
  }

  load_balancer {
    target_group_arn = aws_lb_target_group.test-ecs.arn
    container_name   = "web"
    container_port   = 80
  }

  lifecycle {
    ignore_changes = [capacity_provider_strategy]
  }
}

You can even use any chosen values for memory and CPU in the task definition such as 1234 for CPU and 1234 for RAM as we are not constrained by available Fargate configs. I will also try to scale up the service to some large number of containers to see how will be the EC2 instances scheduled.

Scaling up the service

When I chose to scale the instances to 30 (and I also had to increase account quota), the only instances that were created were t4g.small. This seemed quite inefficient as the scheduler also consumes some capacity and larger instances seem more reasonable. However, it was likely due to limitation of how many network cards can be added to each instance. t4g.small can provide up to 3 network cards, so two of them can be used by the ECS tasks. t4g.large has the same count. However, after several tries I got this weird combination:

$ aws ec2 describe-instances \
 --filters "Name=tag:aws:ecs:clusterName,Values=ec2-managed" \
  "Name=instance-state-name,Values=running" \
  --query 'Reservations[*].Instances[*].[InstanceType]' \
  --output text | tr '\t' '\n' | sort | uniq -c
   5 im4gn.large
   3 is4gen.large
  16 t4g.small

In my opinion this looks even worse as the i4 instances are twice as much expensive as t4g per CPU but im4 have some more memory per each core. So I tried setting some large memory requirement to get an r-family instance. I set 10 tasks of 1000 millicores and 3500 MB of RAM. This gave me just 10 t4g.medium instances as they have 4 GB memory each. So I increase the stakes even higher and asked for 14000 MB per task. This time I got 10 r6g.large instances.

$ aws ec2 describe-instances \
 --filters "Name=tag:aws:ecs:clusterName,Values=ec2-managed" \
  "Name=instance-state-name,Values=running" \
  --query 'Reservations[*].Instances[*].[InstanceType]' \
  --output text | tr '\t' '\n' | sort | uniq -c
  10 r6g.large
  10 t4g.medium
   2 t4g.small

As you can see, ECS Managed Instances give us much more flexibility. We can quickly spawn different types of instances based on our need and the configuration is more than trivial. Cost-wise this setup is the most optimal even if you don't have reserved instances. Fargate is still just $0.04 per hour for a task with 1 vCPU and 1 GB requirement but for that price you can get t4g.medium with 2 CPUs and 4 GB of RAM. The startup time is almost negligible as Bottlerocket (Amazon's OS for running containers) instances start within seconds.

CodePipeline, CodeBuild, and multiple environments in ECS

Piotr Pabis — Thu, 09 Oct 2025 09:00:00 +0000

If you are in the industry and want to conform to the DevOps principles, you must know what CI/CD is and enable developers to test their code as in a real environment as well as allow them to reproducibly deploy their application to the cloud in an automated and less error-prone way. Enough talking, let's go beyond the theory and see some practical example.

As you can see on the diagram above, we have two ECS environments represented here as two ECS services. You can also make a stronger separation by deploying to different VPCs or even accounts but for the sake of simplicity we will just have two services dev and prod in the same ECS Cluster.

Also on the diagram we have a CodeBuild project that will build a Dockerfile and push directly to ECR as well as a CodePipeline pipeline that will clone source code from GitHub, run CodeBuild job, deploy to development ECS service and contain another manual step (approval) that will trigger another action - deployment to production.

Basic setup

I won't describe everything here. As the base infrastructure we will deploy a VPC, an fck-nat instance (to save on NAT Gateway and VPC Endpoints costs) and all ECS necessities. We will also have ECR repository created. To preview our application, I will also create an Application Load Balancer. For now, all the services will be scaled to 0, so that it won't try to pull inexistent image from ECR.

You can find complete Terraform code of the above here: stage 1 of the project on GitHub.

Pushing example Nginx

To test if everything works smoothly, I will build an Nginx container locally on my laptop. I will use AWS CLI credentials to authenticate. In the same Terraform repository I will get the output for the ECR repository URL and push.

$ REPOSITORY_URL=$(tofu output -raw ecr_repository_url):latest
$ REGISTRY_URL=$(echo "$REPOSITORY_URL" | cut -d'/' -f1)
$ aws ecr get-login-password --region eu-west-3 | docker login --username AWS --password-stdin $REGISTRY_URL
$ docker tag nginx:1.29 $REPOSITORY_URL:latest
$ docker push $REPOSITORY_URL:latest

Now we can scale the services to 1 task. We should see Nginx welcome page if we navigate to the ALB DNS name. For the dev environment, I chose /dev path. This will be also reflected in Nginx so it will for now throw a 404 error but it's expected. The most important is if the service started and if there is connectivity. In ecs.tf change desired_count to 1 and wait a bit for the deployment to succeed.

By doing this we could test the connectivity and if everything is configured correctly for future deployments.

Creating CodeConnection

You will need to create a new repository for your app code. It can be GitHub, GitLab or BitBucket, doesn't matter. I will use GitHub in the example.

CodeConnection (formerly CodeStar connection) is an authorization of AWS to use your GitHub account. Even if you create this resource with Terraform, CLI or CloudFormation, you still need to go through the authorization flow with the browser (once). Let's create one in Terraform and go through the process of granting least privilege to AWS (so only repositories we have to). In the new file called codestar.tf write the following:

resource "aws_codeconnections_connection" "github" {
  name          = "gh-connection"
  provider_type = "GitHub"
}

Apply the change with Terraform and go to AWS Console, search for any Code* services (CodePipeline for example). In the left panel go down to Settings > Connections. In the pending one click "Update a pending connection". Press "Install a new app" and you will be prompted to log in to GitHub.

You will be presented with AWS Connector settings. In Repository access select "Only select repositories" and add your app repo to the list.

If you want to add more repositories later, you can go to your GitHub account settings > Integrations > Applications and configure AWS Connector from there.

Creating the pipeline

Now as you already have the connection, you can use it to define the
CodePipeline. In a new file codepipeline.tf we will start drafting it. Do not deploy it yet, as you need at least source and build phases to be able to deploy it. We will also need a role and an S3 bucket to store the artifacts. Define them as well.

resource "aws_codepipeline" "main" {
  name     = "demo-pipeline"
  role_arn = aws_iam_role.codepipeline_role.arn

  artifact_store {
    location = aws_s3_bucket.codepipeline_staging.bucket
    type     = "S3"
  }

  # Stage 1: Source - Pull code from GitHub
  stage {
    name = "Source"

    action {
      name             = "Source"
      category         = "Source"
      owner            = "AWS"
      provider         = "CodeStarSourceConnection"
      version          = "1"
      output_artifacts = ["source_output"]

      configuration = {
        ConnectionArn    = aws_codestarconnections_connection.github.arn
        FullRepositoryId = "ppabis/demo-app"
        BranchName       = "main"
        DetectChanges    = true
      }
    }
  }

  # ... Next steps will follow
}

I will load the policies from JSON files to make the Terraform structure more readable. You can see all the defined policies here (also for CodeBuild): policies on GitHub.

resource "aws_s3_bucket" "codepipeline_staging" {
  bucket        = "pipeline-99819188-makeitunique"
  force_destroy = true
}

# IAM Role for CodePipeline
resource "aws_iam_role" "codepipeline_role" {
  name               = "demo-codepipeline-role"
  assume_role_policy = templatefile("policies/assume_codepipeline.json", {
    service_name = "codepipeline"
  })
}

resource "aws_iam_role_policy" "codepipeline_policy" {
  name   = "demo-codepipeline-policy"
  role   = aws_iam_role.codepipeline_role.id
  policy = templatefile("policies/codepipeline.json", {
    bucket_name = aws_s3_bucket.codepipeline_staging.id
  })
}

CodeBuild instance to build Docker images

Next we will create a CodeBuild project that will build and push the Docker image. It is possible to limit it to the VPC (and use your own Elastic IP for example due to DockerHub limits) but I will just use a public instance for simplicity. I will also create IAM role and a log group so that we see what is happening during the build. In codebuild.tf write:

# IAM Role for CodeBuild
resource "aws_iam_role" "codebuild_role" {
  name = "demo-codebuild-role"
  assume_role_policy = templatefile("policies/assume.json", {
    service_name = "codebuild"
  })
}

resource "aws_iam_role_policy" "codebuild_policy" {
  name = "demo-codebuild-policy"
  role = aws_iam_role.codebuild_role.id
  policy = templatefile("policies/codebuild.json", {
    bucket_name = aws_s3_bucket.codepipeline_staging.id
  })
}

# CloudWatch Log Group for CodeBuild
resource "aws_cloudwatch_log_group" "codebuild" {
  name              = "/aws/codebuild/demo-build-push"
  retention_in_days = 7
}

# CodeBuild project for building and pushing Docker image
resource "aws_codebuild_project" "build_and_push" {
  name          = "demo-build-push"
  description   = "Build Docker image and push to ECR"
  service_role  = aws_iam_role.codebuild_role.arn
  build_timeout = 30

  artifacts { type = "CODEPIPELINE" }
  source { type = "CODEPIPELINE" }

  environment {
    compute_type                = "BUILD_GENERAL1_SMALL"
    image                       = "aws/codebuild/amazonlinux-aarch64-standard:3.0"
    type                        = "ARM_CONTAINER"
    privileged_mode             = true
    image_pull_credentials_type = "CODEBUILD"

    environment_variable {
      name  = "ECR_REPOSITORY_URL"
      value = aws_ecr_repository.demo_app.repository_url
    }
  }

  logs_config {
    cloudwatch_logs {
      group_name  = aws_cloudwatch_log_group.codebuild.name
      stream_name = "build-logs"
    }
  }
}

Now we can attach the build project to the pipeline as the next step after source. We will control the process from the app repository using buildspec.yml that we will define later. I will also add another step that will read imagespec.json that will contain changes for ECS service and deploy it to ECS. Add the following to your existing codepipeline.tf:

resource "aws_codepipeline" "main" {
  name     = "demo-pipeline"
  # ... previous steps

  # Stage 2: Build - Build Docker image and push to ECR
  stage {
    name = "Build"

    action {
      name             = "Build"
      category         = "Build"
      owner            = "AWS"
      provider         = "CodeBuild"
      version          = "1"
      input_artifacts  = ["source_output"]
      output_artifacts = ["build_output"]

      configuration = {
        ProjectName = aws_codebuild_project.build_and_push.name
      }
    }
  }

  # Stage 3: Deploy - Deploy to ECS
  stage {
    name = "Deploy"

    action {
      name            = "Deploy"
      category        = "Deploy"
      owner           = "AWS"
      provider        = "ECS"
      version         = "1"
      input_artifacts = ["build_output"]

      configuration = {
        ClusterName = aws_ecs_cluster.cluster.cluster_name
        ServiceName = aws_ecs_service.dev.service_name
        FileName    = "imagespec.json"
      }
    }
  }
}

You can deploy this Terraform configuration. It will fail but it is expected as we have no buildspec.yml defined anywhere.

Creating demo app

In your demo app repository create a sample app that will be deployed into the service. For now it will be just a Dockerfile with Nginx and some sample HTML page. I will also alias /dev to the root so that the same container can be used with multiple paths in ALB.

FROM public.ecr.aws/nginx/nginx:1.29
COPY index.html /usr/share/nginx/html/index.html
COPY nginx.conf /etc/nginx/conf.d/default.conf

server {
    listen       80;
    server_name  _;

    location / {
        root   /usr/share/nginx/html;
        index  index.html index.htm;
    }

    location /dev/ {
        alias  /usr/share/nginx/html/;
        index  index.html index.htm;
    }

    error_page   500 502 503 504  /50x.html;
    location = /50x.html {
        root   /usr/share/nginx/html;
    }
}

I will also create a buildspec.yml in the root of the repository. This will enable us to control the flow of the build in the CodeBuild project. First we need to log in to the ECR registry using Docker and then we can build and push the images. At the last step, we need to produce a change that will be deployed to ECS service.

version: 0.2
phases:
  pre_build:
    commands:
      - echo Logging in to Amazon ECR...
      - export AWS_REGION=$(echo $ECR_REPOSITORY_URL | cut -d '.' -f 4)
      - aws ecr get-login-password | docker login --username AWS --password-stdin $ECR_REPOSITORY_URL
      - IMAGE_TAG=$(echo $CODEBUILD_RESOLVED_SOURCE_VERSION | cut -c 1-7)
  build:
    commands:
      - docker build -t $ECR_REPOSITORY_URL:$IMAGE_TAG .
      - docker tag $ECR_REPOSITORY_URL:$IMAGE_TAG $ECR_REPOSITORY_URL:latest
      - docker push $ECR_REPOSITORY_URL:latest
      - docker push $ECR_REPOSITORY_URL:$IMAGE_TAG
  post_build:
    commands:
      - printf '[{"name":"web","imageUri":"%s"}]' $ECR_REPOSITORY_URL:$IMAGE_TAG > imagespec.json
artifacts:
  files:
    - imagespec.json

After you push the changes, a new build should be triggered. If not, try to go to the AWS Console, find the pipeline in CodePipeline and retry or try pressing "Release change". If you now go to the ALB's URL under /dev, you should see some HTML page. On the left you see deafult Nginx page of the production service and on the right some vibe-coded futuristic page.

Pipeline for production deployments

For now the pipeline pushes only to dev. I will create a new stage and actions that will run only if someone accepts the development build and deploy it to production service. This stage will have two actions: manual approval with run order of 1 and another deployment with run order of 2. This makes the second action dependent on the first and lets us save on unnecessary extra stages.

resource "aws_codepipeline" "main" {
  name     = "demo-pipeline"
  # ... Previous stages
  stage {
    name = "Acceptance"

    action {
      name      = "ApproveProduction"
      category  = "Approval"
      owner     = "AWS"
      provider  = "Manual"
      version   = "1"
      run_order = 1
    }

    action {
      name            = "DeployProduction"
      category        = "Deploy"
      owner           = "AWS"
      provider        = "ECS"
      version         = "1"
      input_artifacts = ["build_output"]
      run_order       = 2

      configuration = {
        ClusterName = module.ecs.cluster_name
        ServiceName = module.ecs.services["prod"].name
        FileName    = "imagespec.json"
      }
    }
  }
}

If you now go to the console and press "Release change" the pipeline will run again. It will rebuild the image, deploy to development but the last stage will wait for your acceptance.

You can press on the action with the blue clock and select either "Accept" or "Reject". In the comment box you can state reasons for either choice. This info will be saved for the current pipeline run. After the acceptance, another deployment job will be run and you should see the same page on development and production.

The completed repository is available on the master branch on GitHub:
ppabis/ECS-CodePipeline-TwoEnvironments.
Example app is also public:
ppabis/demo-app.

You have to upgrade to SOCI V2 🤯

Piotr Pabis — Mon, 01 Sep 2025 05:41:25 +0000

Around May 2025, AWS started updating SOCI Snapshotter GitHub repo to support SOCI indexes v2. This is a refreshed model that simplifies the index management. It's easier to find which index refers to which image/tag with this new version. Because of that I also updated the tool I have presented in the previous post, so that it also supports latest SOCI version.

Previous post you can find here: https://pabis.eu/blog/2025-06-17-Faster-ECS-Startup-SOCI-Index-GitLab-Pipeline.html

The new standalone SOCI indexer is in this GitHub repo: github.com/ppabis/ecr-aws-soci-index-builder/tree/v0.11.2

Disclaimer ⚠️

SOCI V1 feature is disabled by default in all accounts that did not submit a Support ticket to AWS. As previously mentioned, this wasn't announced anywhere but Support confirmed eventually that SOCI V1 is gated since beginning of 2025. SOCI V2 is enabled by default in all accounts. What is more, very recently AWS announced (August 27, 2025) that SOCI V1 will be deprecated until February 9, 2026.

📦 Architecture for our experiments

If you want to play around with these tools, you can clone the repository from here. I used OpenTofu but it should also work well with Terraform. Just provide the credentials and run tofu apply.

https://github.com/ppabis/soci-v2-experiments

I have created a VPC with all the essentials like NAT gateway, subnets, two EC2 instances that can be used for building the images and indexes, ECR repository and a simple ECS Fargate service. Once you create this environment, wait two or three minutes because the EC2 instances should build the images for respective architectures and push them to ECR on startup. After that you can SSH into one of them using EC2 Instance Connect and create an image index of both images.

# Log in to any instance. Check tofu output to get the commands
aws ec2-instance-connect ssh \
   --region us-east-2 \
   --instance-id i-05234fc9c1275999d \
   --instance-ip 10.60.2.140 \
   --connection-type eice \
   --os-user ec2-user

Now using the following commands, we can create a multi-arch image. This is a good practice if you sometimes want to save on costs - for example it can happen that x86 spot instances are cheaper at this point in time. As the images can be built in parallel, the cost is very small while there is a good potential for savings in the future.

# Change this to your registry and region. Check tofu output to get the value.
export ECR_REGISTRY="889900112233.dkr.ecr.us-east-2.amazonaws.com"
aws ecr get-login-password | docker login -u AWS --password-stdin $ECR_REGISTRY
docker manifest create \
 $ECR_REGISTRY/soci-repo:latest \
 $ECR_REGISTRY/soci-repo:arm64 \
 $ECR_REGISTRY/soci-repo:amd64

docker manifest annotate \
 $ECR_REGISTRY/soci-repo:latest \
 $ECR_REGISTRY/soci-repo:arm64 \
 --arch arm64

docker manifest annotate \
 $ECR_REGISTRY/soci-repo:latest \
 $ECR_REGISTRY/soci-repo:amd64 \
 --arch amd64

docker manifest push $ECR_REGISTRY/soci-repo:latest

In your ECR repository you should now see three images: two actual images and one index like on the image below. These are not SOCI enabled.

We can use ORAS to analyze how this new manifest looks like. It should have just two entries for Intel image and ARM image. On the instances I have already installed ORAS using user data but you can get more information about it here: oras.land.

aws ecr get-login-password | oras login -u AWS --password-stdin $ECR_REGISTRY
oras manifest fetch $ECR_REGISTRY/soci-repo:latest

{
   "schemaVersion": 2,
   "mediaType": "application/vnd.docker.distribution.manifest.list.v2+json",
   "manifests": [
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 2613,
         "digest": "sha256:cf8488c22543c8ac676dbea9ceb99abb7120799405b05b4098149bed20bf14b3",
         "platform": {
            "architecture": "amd64",
            "os": "linux"
         }
      },
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 2613,
         "digest": "sha256:a60af31eb40bb689c25aa23748707fdbd0d698734bbdb58ba3ee3597fb214379",
         "platform": {
            "architecture": "arm64",
            "os": "linux"
         }
      }
   ]
}

Creating SOCI V1 index 1️⃣

We can now create old SOCI V1 index using the tool that should also be available on the instance. You can run it in Docker and just provide the URIs as well as requested SOCI index version. With V1 you can't point to an index as it will result in an error.

$ docker run --rm -it -e AWS_REGION=us-east-2 soci-gen:latest -repository $ECR_REGISTRY/soci-repo:latest -soci-index-version V1
{"level":"warn","RegistryURL":"889900112233.dkr.ecr.us-east-2.amazonaws.com","time":"2025-08-29T07:00:26Z","message":"Image manifest validation error: not a valid image manifest: empty config media type"}
# Generate SOCI V1 indexes for arm64 and amd64 separately
$ docker run --rm -it -e AWS_REGION=us-east-2 soci-gen:latest -repository $ECR_REGISTRY/soci-repo:amd64 -soci-index-version V1
$ docker run --rm -it -e AWS_REGION=us-east-2 soci-gen:latest -repository $ECR_REGISTRY/soci-repo:arm64 -soci-index-version V1

In ECR you will see now two additional entries that are untagged. They will be of type application/vnd.amazon.soci.index.v1+json. However, even if you now download the manifest of any of the images or image index, nothing will change. SOCI V1 index is just a manifest uploaded to the repository.

🤔 The problem with V1 indexes

In order to find if an index exists for a specific image, you either have to scan all the indexes in the repo or have some database mechanism that stored a map of indexes and image hashes. A way to imagine this problem is with simple files and directories. Let's say you have a directory that represents your container image repository. You have several tar.gz files that are actual images and some soci.json files that are SOCI indexes (this example is just simplification).

$ ls myrepo
78160dbe.tar.gz         685511a5.tar.gz         3385ce09.soci.json
632fe5f2.soci.json      5663eace.soci.json

Let's assume that you want to pull image 78160dbe. To check if you have a SOCI index for this image, you need to either check all the .soci.json files or have some database that has a list of that. In each .soci.json file, there's a "Subject" field that has a hash of the image it relates to.

When you download the JSON that sits behind this entry, you can see the SHA sum of the image for which the SOCI index was created (at the bottom under subject). When you look at the JSON manifest of the image, there's no relation to the SOCI index - it's just a plain old Docker manifest.

# Fetch the SOCI V1 index
oras manifest fetch 0123456789012.dkr.ecr.us-east-2.amazonaws.com/soci-repo@sha256:d00c6532ae6925dc10527d85eb77ec048c5661bf8a6bf460ee53be474e38f5de | jq

{
  "schemaVersion": 2,
  "mediaType": "application/vnd.oci.image.manifest.v1+json",
  "config": {
    "mediaType": "application/vnd.amazon.soci.index.v1+json",
    "digest": "sha256:d00c6532ae6925dc10527d85eb77ec048c5661bf8a6bf460ee53be474e38f5de",
    "size": 2
  },
  "layers": [
    {
      "mediaType": "application/octet-stream",
      "digest": "sha256:5a317781e98d5b9e06ff67dfcd721fde46eb30e48af5ec959ff15260124521d9",
      "size": 1387368,
      "annotations": {
        "com.amazon.soci.image-layer-digest": "sha256:fbe4cc75eed0371e58892a650547fc4b104356b21b2da1ef76c9219247af9cc9",
        "com.amazon.soci.image-layer-mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip"
      }
    },
    ... // More ztoc layers continue
  ],
  "subject": {
    "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
    "digest": "sha256:2b0d259f8f55f97f4052afd5bba0cc54cc394206767de85386a379493a75af8d",
    "size": 2613
  },
  "annotations": {
    "com.amazon.soci.build-tool-identifier": "AWS SOCI CLI v0.2"
  }
}

To find if an image has a SOCI index, we can use oras discover command. It finds the relationships between untagged artifacts 😮. For example to locate the SOCI v1 index of arm64 image, I use the following command. This is this magical database I was talking about. Notice that the hash of the index matches what we just downloaded.

oras discover 0123456789012.dkr.ecr.us-east-2.amazonaws.com/soci-repo:arm64
0123456789012.dkr.ecr.us-east-2.amazonaws.com/soci-repo@sha256:a60af31eb40bb689c25aa23748707fdbd0d698734bbdb58ba3ee3597fb214379
└── application/vnd.amazon.soci.index.v1+json
    └── sha256:d00c6532ae6925dc10527d85eb77ec048c5661bf8a6bf460ee53be474e38f5de

On my current account, where I requested SOCI V1 to be enabled, it still works. But when I tested on another clean account, it indeed did not show up in ECS. The example service in the repository fetches ECS metadata of the task and the container. If the Snapshotter field is soci that means either SOCI V1 or V2 was used. If it is overlayfs that means no SOCI index was found or you are trying to use SOCI V1 on an account where it is disabled.

Upgrade to SOCI V2 🚀

I used a new version of the standalone indexer I modified from the Lambda tool. It does not overwrite the original tag but creates a new one with -soci suffix. That way you can have both SOCI-indexed alongside same non-indexed image in the repository. Back in SOCI v1, unless deleted, the index was always retrieved for a particular image if the runtime has the feature enabled.

With SOCI V2 things are more reliable. The index is also kept in a similar file, however, the connection between index and image is stored both in the index file itself but also in another manifest file that is tagged. So once you pull myimage:latest-soci, a manifest is loaded first that contains links to both the actual layered image as well as the SOCI index. It makes it quicker and less complicated to manage. What is more, you can control whether you want to use SOCI enhanced image or a normal one by changing a tag.

I have built SOCI V2 index against the multi-arch index we created earlier, as this use case is supported now. The command is almost the same. However, you can also point to the image directly. In both cases, a new index will be created.

On the image above you can see also some two untagged images. These are the same as the tagged arm64 and amd64 images but their manifests were converted to OCI format (from a default Docker one). If you inspect the two, the layers are untouched and have the same hashes - only media type is changed. You won't pay for more storage as the layers are shared. I verified that with ORAS.

# Fetch original Docker manifest
oras manifest fetch 0123456789012.dkr.ecr.us-east-2.amazonaws.com/soci-repo:arm64
# Fetch converted OCI manifest of the same image
oras manifest fetch 0123456789012.dkr.ecr.us-east-2.amazonaws.com/soci-repo@sha256:cdd640f4e50444a0be5e10c5698bc27904e68e1246b90acf18521b4117df83da | jq

{
   "schemaVersion": 2,
   "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
   "config": {
      "mediaType": "application/vnd.docker.container.image.v1+json",
      "size": 8072,
      "digest": "sha256:873cf5dabf17fcf461fbfc55c9785fa2fcf282b55b96e6ef0508a944166cb514"
   },
   "layers": [
      {
         "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
         "size": 31373560,
         "digest": "sha256:fbe4cc75eed0371e58892a650547fc4b104356b21b2da1ef76c9219247af9cc9"
      },
      {
         "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
         "size": 1270307,
         "digest": "sha256:3cd5add3ba3330f959ed1b27096bfb25e2f29a4cf327009d69855a903036f3af"
      },
    ...
   ]
}

{
  "schemaVersion": 2,
  "mediaType": "application/vnd.oci.image.manifest.v1+json",
  "config": {
    "mediaType": "application/vnd.oci.image.config.v1+json",
    "digest": "sha256:873cf5dabf17fcf461fbfc55c9785fa2fcf282b55b96e6ef0508a944166cb514",
    "size": 8072
  },
  "layers": [
    {
      "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
      "digest": "sha256:fbe4cc75eed0371e58892a650547fc4b104356b21b2da1ef76c9219247af9cc9",
      "size": 31373560
    },
    {
      "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
      "digest": "sha256:3cd5add3ba3330f959ed1b27096bfb25e2f29a4cf327009d69855a903036f3af",
      "size": 1270307
    },
  ...
  ]
}

☹️ Multi-arch limitation?

As least with the tool I have converted from Lambda there actually is a limitation. Even though as you saw, both arm64 and amd64 images were converted to OCI format, SOCI index was only created for the platform you run the tool from. It essentially behaves the same was as you would pull a default image. When I tried to force creation of amd64 index on arm64 instance, it failed by trying to download actual arm64 image from amd64 tag. When you fetch the manifest of the -soci tagged image, you can see that it only contains one SOCI entry and two OCI image entries for both architectures.

# Get new image index manifest generated by SOCI v2
oras manifest fetch 0123456789012.dkr.ecr.us-east-2.amazonaws.com/soci-repo:latest-soci

{
  "schemaVersion": 2,
  "mediaType": "application/vnd.oci.image.index.v1+json",
  "manifests": [
    {
      "mediaType": "application/vnd.oci.image.manifest.v1+json",
      "digest": "sha256:6771831a7bfa83d50a462058c462c4b5b7ebc5ad47bc72bc9894f85bfedbbeef",
      "size": 2027,
      "platform": {
        "architecture": "amd64",
        "os": "linux"
      }
    },
    {
      "mediaType": "application/vnd.oci.image.manifest.v1+json",
      "digest": "sha256:cdd640f4e50444a0be5e10c5698bc27904e68e1246b90acf18521b4117df83da",
      "size": 2148,
      "annotations": {
        "com.amazon.soci.index-digest": "sha256:963e2cc55b42e73b6f51530d9956ed03cb204bf0fce0b2608a1a4e7674e7c354"
      },
      "platform": {
        "architecture": "arm64",
        "os": "linux"
      }
    },
    {
      "mediaType": "application/vnd.oci.image.manifest.v1+json",
      "digest": "sha256:963e2cc55b42e73b6f51530d9956ed03cb204bf0fce0b2608a1a4e7674e7c354",
      "size": 1394,
      "annotations": {
        "com.amazon.soci.image-manifest-digest": "sha256:cdd640f4e50444a0be5e10c5698bc27904e68e1246b90acf18521b4117df83da"
      },
      "platform": {
        "architecture": "arm64",
        "os": "linux",
        "variant": "v8"
      },
      "artifactType": "application/vnd.amazon.soci.index.v2+json"
    }
  ]
}

However, when I created a new manifest and merged two SOCI V2 image indexes together (4 entries in total) along with all annotations to match, the ECS service as able to use SOCI snapshotter for both architectures so the limitation is only in the tool. You can find the example index here: manifest-new.json. I used ORAS to forcefully push it to ECR.

oras manifest push 889900112233.dkr.ecr.us-east-2.amazonaws.com/soci-repo:latest-soci-2 manifest-new.json

But! I asked in the issues of the original SOCI Snapshotter for the feature and it turns out it was there under the flag --all-platforms. By analyzing the code I backported it to my tool and now it generated all the platforms by default (you can't change this behavior).

Conclusion

SOCI V2 is not a big or breaking change. It simplifies the management of indexes and makes it easier for the registry provider to handle them. They are treated the same way as regular artifacts stored in the registry and always bound by a manifest.