Forem: svodwood

VPC Lattice Part 2. Connecting Applications to Shared Services

svodwood — Fri, 14 Apr 2023 06:19:56 +0000

Introduction

In Part 1 of the series, we looked at some of the benefits and drawbacks of VPC Lattice and theorized on possible use cases where this managed service might be helpful. We also discussed a "shared service" scenario, where some individual applications may desire access to a shared service while belonging to entirely separate networks. This pattern might present itself with a commonly shared HTTP endpoint - for example, for token generation and validation, tied to a shared encryption infrastructure. While not straightforward, utilizing VPC Lattice to support this topology is possible. Let's now dive deeper and construct a functional multi-account and multi-VPC mini-landing zone - utilizing VPC Lattice as an application layer network.

Pulumi IaC will help us bring up our infrastructure on the AWS Cloud. Check out pulumi.com if you still need to become familiar with it. You can deploy these demo stacks using the Pulumi buttons below.

You may find the source code for this demo in this Github repo.

Disclaimer #1: in the Pulumi stacks, we use two AWS providers: aws and aws-native, which need separate configurations, e.g., AWS CLI profile name and region. The latter is the only option to work with Lattice from Pulumi. I cannot currently recommend the aws-native provider for production use.

Disclaimer #2: at the time of writing, some requests to VPC Lattice API return 429. AWS is still throttling requests to the service, although this may be related to your account status. Run "pulumi up" one more time if you experience this.

What We Are Going To Build

Our focus today is on our hypothetical company's three microservices:

DirectoryService. A shared microservice.
BlueService. A standalone microservice.
GreenService. A standalone microservice.

Separate development teams own the BlueService and the GreenService, respectively - each team has an isolated AWS account where they develop their microservices. An independent team maintains the third application (DirectoryService) and controls access to the application. This application runs in the third isolated AWS account.

Our network isolation requirements define that the GreenService must not be in the same network as the BlueService. Both services, however, need to perform HTTP GET requests to the DirectoryService - using dedicated query strings.

Below is the high-level diagram of the landing zone topology and network isolation requirements:

To start building, we are going to need to meet the prerequisites:

Three separate AWS accounts belong to a single AWS Organization. Additionally, we must enable RAM sharing across the Organization.
Three configured CLI profiles - for all three accounts, respectively. Mind that the aws:profile and aws-native:profile providers must be configured explicitly and separately. They may, however, reuse the same credentials.
A working Pulumi environment.
Account IDs of all three AWS accounts.

Pulumi Stacks and Deployment Procedures

Our lab repository consists of three directories, each corresponding to a separate Pulumi stack:

./a-common-service
./b-blue-service
./c-green-service

A "common-service" stack contains the infrastructure for the DirectoryService application deployed to a separate AWS account. This specific account also owns the two independent VPC Lattice Service Networks:

One is to control the communication with the GreenService.
Another is to control the communication with BlueService.

We share the Service Networks with the respective microservice accounts via AWS RAM. As such, we provision the stacks in the same order listed above. Once the "common-service" stack is up, note the two VPC Lattice Service Networks - we must use their ARNs in the blue and green stack configurations, respectively. For simplicity, we will not use stack imports here and will copy-paste the correct values of the "common-service" stack output. Also, make a note of the "shared-service-fqdn" output value. Then, copy-paste that into both "blue-service" and "green-service" stack inputs.

Deploy the Common Service:

Deploy the Blue Service:

Deploy the Green Service:

Pulumi Stack: Common Service

Our "common-service" stack consists of essential VPC components, Lattice-specific network components, AWS RAM shares and a Lambda function running FastAPI. In addition, we use a private Application Load Balancer to front our Lambda function. First, let's break down the stack.

Our Pulumi project consists of the following files:

settings.py - holds generic settings and Pulumi configuration variables.
helpers.py - holds generic helper functions, if any.
vpc.py - constructs three VPCs in our shared service AWS account.
lattice.py - constructs Lattice Service Networks, associates them with the correct VPCs and shares them with the correct AWS accounts.
app_infra.py - constructs everything around the Lambda function, including the ALB, IAM and the Lambda itself.
main.py - executes the code.

First, we define the settings and fetch configuration variables (settings.py):

from pulumi import Config

"""
Various stack settings.
Below baseline tags must be applied to all created resources without exception:
1.  Key: "Account"
    Values: ["Management", "GreenSandbox", "BlueSandbox", "SharedServices"]
2.  Key: "AccountType"
    Values: ["Management", "Workload"]
3.  Key: "Provisioned"
    Values: ["Manually", "Pulumi", "Terraform", "Other"]
4.  Key: "StackName"
    Values: ["<project-name-here>"]
"""
pulumi_config = Config()
provider_config = Config("aws")
deployment_region = provider_config.require("region")

baseline_cost_tags = {
    "Account": "SharedServices",
    "AccountType": "Workload",
    "Provisioned": "Pulumi",
    "StackName": "common-service"
}

baseline_cost_tags_native = [
    {
        "key": "AccountType",
        "value": "Workload"
    },
    {
        "key": "Account",
        "value": "SharedServices"
    },
    {
        "key": "Provisioned",
        "value": "Pulumi"
    },
    {
        "key": "StackName",
        "value": "common-service"
    }
]

shared_services_main_vpc_cidr = "10.0.0.0/16"

shared_service_subnet_cidrs_pub = [
    "10.0.0.0/22",
    "10.0.4.0/22"
]

shared_service_subnet_cidrs_app = [
    "10.0.8.0/22",
    "10.0.12.0/22"
]

"""
VPC Lattice Local Peers
"""
blue_network_vpc_cidr = "192.168.0.0/23"
blue_subnet_cidrs = [ 
    "192.168.0.0/24",
    "192.168.1.0/24"    
]

green_network_vpc_cidr = "192.168.2.0/23"
green_subnet_cidrs = [ 
    "192.168.2.0/24",
    "192.168.3.0/24"    
]

"""
VPC Lattice Dev Accounts
"""
green_principal = pulumi_config.require("green-principal")
blue_principal = pulumi_config.require("blue-principal")

Second, get some helpers (helpers.py):

from pulumi_aws import get_caller_identity, get_availability_zones

"""
Various stack helper functions and configuration variables.
"""

current_account = get_caller_identity()
current_account_id = current_account.account_id
demo_azs = get_availability_zones(state="available").names

Thirdly, construct the VPCs (vpc.py):

from pulumi_aws import ec2
from pulumi import ResourceOptions

from settings import baseline_cost_tags, deployment_region, shared_services_main_vpc_cidr, blue_network_vpc_cidr, green_network_vpc_cidr, blue_subnet_cidrs, green_subnet_cidrs, shared_service_subnet_cidrs_pub, shared_service_subnet_cidrs_app
from helpers import demo_azs

"""
Constructs Shared Services main VPC:
"""
shared_services_main_vpc = ec2.Vpc("SharedServicesMainVpc",
    cidr_block=shared_services_main_vpc_cidr,
    enable_dns_hostnames=True,
    enable_dns_support=True,
    tags={**baseline_cost_tags, "Name": f"SharedServicesMainVpc-{deployment_region}"}
)

shared_services_main_igw = ec2.InternetGateway("SharedServicesMainVpcIgw",
    vpc_id=shared_services_main_vpc.id,
    tags={**baseline_cost_tags, "Name": f"SharedServicesMainVpcIgw-{deployment_region}"},
    opts=ResourceOptions(parent=shared_services_main_vpc)
)

"""
Constructs secondary VPCs for peering and Lattice connection:
"""
green_network_vpc = ec2.Vpc("GreenNetworkVpc",
    cidr_block=green_network_vpc_cidr,
    enable_dns_hostnames=True,
    enable_dns_support=True,
    tags={**baseline_cost_tags, "Name": f"GreenNetworkVpc-{deployment_region}"}
)

blue_network_vpc = ec2.Vpc("BlueNetworkVpc",
    cidr_block=blue_network_vpc_cidr,
    enable_dns_hostnames=True,
    enable_dns_support=True,
    tags={**baseline_cost_tags, "Name": f"BlueNetworkVpc-{deployment_region}"}
)

"""
Establishes VPC peering connections:
"""
green_peering_connection = ec2.VpcPeeringConnection("GreenPeeringConnection",
    vpc_id=shared_services_main_vpc.id,
    peer_vpc_id=green_network_vpc.id,
    auto_accept=True,
    opts=ResourceOptions(
        parent=shared_services_main_vpc,
        depends_on=[
            shared_services_main_vpc,
            green_network_vpc
        ]
    )
)

green_peering_connection_options = ec2.PeeringConnectionOptions("GreenPeeringConnectionOptions",
    vpc_peering_connection_id=green_peering_connection.id,
    accepter=ec2.PeeringConnectionOptionsAccepterArgs(
        allow_remote_vpc_dns_resolution=True,
    ),
    requester=ec2.PeeringConnectionOptionsRequesterArgs(
        allow_remote_vpc_dns_resolution=True,
    ),
    opts=ResourceOptions(
        parent=green_peering_connection,
        depends_on=[
            green_peering_connection
        ]
    )
)

blue_peering_connection = ec2.VpcPeeringConnection("BluePeeringConnection",
    vpc_id=shared_services_main_vpc.id,
    peer_vpc_id=blue_network_vpc.id,
    auto_accept=True,
    opts=ResourceOptions(
        parent=shared_services_main_vpc,
        depends_on=[
            shared_services_main_vpc,
            blue_network_vpc
        ]
    )
)

blue_peering_connection_options = ec2.PeeringConnectionOptions("BluePeeringConnectionOptions",
    vpc_peering_connection_id=blue_peering_connection.id,
    accepter=ec2.PeeringConnectionOptionsAccepterArgs(
        allow_remote_vpc_dns_resolution=True,
    ),
    requester=ec2.PeeringConnectionOptionsRequesterArgs(
        allow_remote_vpc_dns_resolution=True,
    ),
    opts=ResourceOptions(
        parent=blue_peering_connection,
        depends_on=[
            blue_peering_connection
        ]
    )
)

"""
Constructs Shared Services main VPC's subnets:
"""
shared_services_pub_subnets = []
shared_services_app_subnets = []

for i in range(2):
    prefix = f"{demo_azs[i]}"

    shared_service_subnet_pub = ec2.Subnet(f"SharedServiceSubnetPub-{prefix}",
        vpc_id=shared_services_main_vpc.id,
        cidr_block=shared_service_subnet_cidrs_pub[i],
        availability_zone=demo_azs[i],
        tags={**baseline_cost_tags, "Name": f"SharedServiceSubnetPub-{prefix}"},
        opts=ResourceOptions(parent=shared_services_main_vpc)
    )

    shared_services_pub_subnets.append(shared_service_subnet_pub)

    shared_service_rt_pub = ec2.RouteTable(f"SharedServiceRtPub-{prefix}",
        vpc_id=shared_services_main_vpc.id,
        tags={**baseline_cost_tags, "Name": f"SharedServiceRtPub-{prefix}"},
        opts=ResourceOptions(parent=shared_service_subnet_pub)
    )

    shared_service_rt_pub_association = ec2.RouteTableAssociation(f"SharedServiceRtaPub-{prefix}",
        route_table_id=shared_service_rt_pub.id,
        subnet_id=shared_service_subnet_pub.id,
        opts=ResourceOptions(parent=shared_service_subnet_pub)
    )

    shared_service_pub_route_to_wan = ec2.Route(f"WanPubRoute-{prefix}",
        route_table_id=shared_service_rt_pub.id,
        gateway_id=shared_services_main_igw.id,
        destination_cidr_block="0.0.0.0/0",
        opts=ResourceOptions(
            parent=shared_service_rt_pub
        )
    )

    shared_service_subnet_app = ec2.Subnet(f"SharedServiceSubnetApp-{prefix}",
        vpc_id=shared_services_main_vpc.id,
        cidr_block=shared_service_subnet_cidrs_app[i],
        availability_zone=demo_azs[i],
        tags={**baseline_cost_tags, "Name": f"SharedServiceSubnetApp-{prefix}"},
        opts=ResourceOptions(parent=shared_services_main_vpc)
    )

    shared_services_app_subnets.append(shared_service_subnet_app)

    shared_service_rt_app = ec2.RouteTable(f"SharedServiceRtApp-{prefix}",
        vpc_id=shared_services_main_vpc.id,
        tags={**baseline_cost_tags, "Name": f"SharedServiceRtApp-{prefix}"},
        opts=ResourceOptions(parent=shared_service_subnet_app)
    )

    shared_service_rt_app_association = ec2.RouteTableAssociation(f"SharedServiceAppRta-{prefix}",
        route_table_id=shared_service_rt_app.id,
        subnet_id=shared_service_subnet_app.id,
        opts=ResourceOptions(parent=shared_service_subnet_app)
    )

    shared_service_route_to_green = ec2.Route(f"RouteToGreenPeer-{prefix}",
        route_table_id=shared_service_rt_app.id,
        vpc_peering_connection_id=green_peering_connection.id,
        destination_cidr_block=green_network_vpc_cidr,
        opts=ResourceOptions(
            parent=shared_service_rt_app,
            depends_on=[green_peering_connection]
        )
    )

    shared_service_route_to_blue = ec2.Route(f"RouteToBluePeer-{prefix}",
        route_table_id=shared_service_rt_app.id,
        vpc_peering_connection_id=blue_peering_connection.id,
        destination_cidr_block=blue_network_vpc_cidr,
        opts=ResourceOptions(
            parent=shared_service_rt_app,
            depends_on=[blue_peering_connection]
        )
    )

"""
Constructs green network peer subnets:
"""
green_peer_subnets = []

for i in range(2):
    prefix = f"{demo_azs[i]}"

    green_peer_subnet = ec2.Subnet(f"GreenPeerSubnet-{prefix}",
        vpc_id=green_network_vpc.id,
        cidr_block=green_subnet_cidrs[i],
        availability_zone=demo_azs[i],
        tags={**baseline_cost_tags, "Name": f"GreenPeerSubnet-{prefix}"},
        opts=ResourceOptions(parent=green_network_vpc)
    )

    green_peer_subnets.append(green_peer_subnet)

    green_peer_rt = ec2.RouteTable(f"GreenPeerRt-{prefix}",
        vpc_id=green_network_vpc.id,
        tags={**baseline_cost_tags, "Name": f"GreenPeerRt-{prefix}"},
        opts=ResourceOptions(parent=green_peer_subnet)
    )

    green_peer_rt_association = ec2.RouteTableAssociation(f"GreenPeerRta-{prefix}",
        route_table_id=green_peer_rt.id,
        subnet_id=green_peer_subnet.id,
        opts=ResourceOptions(parent=green_peer_subnet)
    )

    green_peer_main_route = ec2.Route(f"GreenPeerMainRoute-{prefix}",
        route_table_id=green_peer_rt.id,
        vpc_peering_connection_id=green_peering_connection.id,
        destination_cidr_block=shared_services_main_vpc_cidr,
        opts=ResourceOptions(
            parent=green_peer_rt,
            depends_on=[green_peering_connection]
        )
    )


"""
Constructs blue network peer subnets:
"""
blue_peer_subnets = []

for i in range(2):
    prefix = f"{demo_azs[i]}"

    blue_peer_subnet = ec2.Subnet(f"BluePeerSubnet-{prefix}",
        vpc_id=blue_network_vpc.id,
        cidr_block=blue_subnet_cidrs[i],
        availability_zone=demo_azs[i],
        tags={**baseline_cost_tags, "Name": f"BluePeerSubnet-{prefix}"},
        opts=ResourceOptions(parent=blue_network_vpc)
    )

    blue_peer_subnets.append(blue_peer_subnet)

    blue_peer_rt = ec2.RouteTable(f"BluePeerRt-{prefix}",
        vpc_id=blue_network_vpc.id,
        tags={**baseline_cost_tags, "Name": f"BluePeerRt-{prefix}"},
        opts=ResourceOptions(parent=blue_peer_subnet)
    )

    blue_peer_rt_association = ec2.RouteTableAssociation(f"BluePeerRta-{prefix}",
        route_table_id=blue_peer_rt.id,
        subnet_id=blue_peer_subnet.id,
        opts=ResourceOptions(parent=blue_peer_subnet)
    )

    blue_peer_main_route = ec2.Route(f"BluePeerMainRoute-{prefix}",
        route_table_id=blue_peer_rt.id,
        vpc_peering_connection_id=blue_peering_connection.id,
        destination_cidr_block=shared_services_main_vpc_cidr,
        opts=ResourceOptions(
            parent=blue_peer_rt,
            depends_on=[blue_peering_connection]
        )
    )

"""
Creates demo security groups in the Green and Blue VPCs:
"""
green_lattice_sg = ec2.SecurityGroup("GreenSecurityGroup",
    description="Green Lattice Security Group",
    vpc_id=green_network_vpc.id,
    ingress=[ec2.SecurityGroupIngressArgs(
        description="Any",
        from_port=0,
        to_port=0,
        protocol="-1",
        cidr_blocks=["0.0.0.0/0"],
        ipv6_cidr_blocks=["::/0"],
    )],
    egress=[ec2.SecurityGroupEgressArgs(
        from_port=0,
        to_port=0,
        protocol="-1",
        cidr_blocks=["0.0.0.0/0"],
        ipv6_cidr_blocks=["::/0"],
    )],
    tags={**baseline_cost_tags, "Name": f"GreenSecurityGroup-{deployment_region}"},
    opts=ResourceOptions(parent=green_network_vpc)
)

blue_lattice_sg = ec2.SecurityGroup("BlueSecurityGroup",
    description="Blue Lattice Security Group",
    vpc_id=blue_network_vpc.id,
    ingress=[ec2.SecurityGroupIngressArgs(
        description="Any",
        from_port=0,
        to_port=0,
        protocol="-1",
        cidr_blocks=["0.0.0.0/0"],
        ipv6_cidr_blocks=["::/0"],
    )],
    egress=[ec2.SecurityGroupEgressArgs(
        from_port=0,
        to_port=0,
        protocol="-1",
        cidr_blocks=["0.0.0.0/0"],
        ipv6_cidr_blocks=["::/0"],
    )],
    tags={**baseline_cost_tags, "Name": f"BlueSecurityGroup-{deployment_region}"},
    opts=ResourceOptions(parent=blue_network_vpc)
)

"""
Creates a demo security group in the main VPC, to be used with the Application Load Balancer:
"""
main_lambda_alb_sg = ec2.SecurityGroup("MainLambdaAlbSecurityGroup",
    description="ALB Security Group",
    vpc_id=shared_services_main_vpc.id,
    ingress=[ec2.SecurityGroupIngressArgs(
        description="Any",
        from_port=0,
        to_port=0,
        protocol="-1",
        cidr_blocks=["0.0.0.0/0"],
        ipv6_cidr_blocks=["::/0"],
    )],
    egress=[ec2.SecurityGroupEgressArgs(
        from_port=0,
        to_port=0,
        protocol="-1",
        cidr_blocks=["0.0.0.0/0"],
        ipv6_cidr_blocks=["::/0"],
    )],
    tags={**baseline_cost_tags, "Name": f"MainLambdaAlbSecurityGroup-{deployment_region}"},
    opts=ResourceOptions(parent=shared_services_main_vpc)
)

Fourthly, create Lattice networks (lattice.py):

from pulumi_aws_native import vpclattice
from pulumi_aws import ram
from pulumi import ResourceOptions, export

from settings import baseline_cost_tags, deployment_region, baseline_cost_tags_native, green_principal, blue_principal
from vpc import green_network_vpc, blue_network_vpc, green_lattice_sg, blue_lattice_sg

"""
Creates the RAM shares for Lattice service networks:
"""
green_ram_share = ram.ResourceShare("GreenLatticeRamShare",
    allow_external_principals=False, # Make note that this share will not work outside your AWS Organization by default, set to True if needed
    tags=baseline_cost_tags
)

blue_ram_share = ram.ResourceShare("BlueLatticeRamShare",
    allow_external_principals=False, # Make note that this share will not work outside your AWS Organization by default, set to True if needed
    tags=baseline_cost_tags
)

"""
Associates principals to the RAM shares:
"""
green_principal_association = ram.PrincipalAssociation("GreenPrincipalAssociation",
    principal=green_principal,
    resource_share_arn=green_ram_share.arn
)

blue_principal_association = ram.PrincipalAssociation("BluePrincipalAssociation",
    principal=blue_principal,
    resource_share_arn=blue_ram_share.arn
)

"""
Constructs green and blue service networks:
"""
green_service_network = vpclattice.ServiceNetwork("GreenServiceNetwork",
    auth_type="NONE",
    name="green-service-network",
    tags=baseline_cost_tags_native
)

export("green-service-network-arn", green_service_network.arn)

blue_service_network = vpclattice.ServiceNetwork("BlueServiceNetwork",
    auth_type="NONE",
    name="blue-service-network",
    tags=baseline_cost_tags_native
)

export("blue-service-network-arn", blue_service_network.arn)

"""
Adds service networks to respective RAM shares:
"""
green_ram_share_association = ram.ResourceAssociation("GreenLatticeRamShareAssociation",
    resource_arn=green_service_network.arn,
    resource_share_arn=green_ram_share.arn,
    opts=ResourceOptions(
        parent=green_ram_share,
        depends_on=[
            green_service_network
        ]
    )
)

blue_ram_share_association = ram.ResourceAssociation("BlueLatticeRamShareAssociation",
    resource_arn=blue_service_network.arn,
    resource_share_arn=blue_ram_share.arn,
    opts=ResourceOptions(
        parent=blue_ram_share,
        depends_on=[
            blue_service_network
        ]
    )
)

"""
Adds service netowork VPC associations (Green Peer and Blue Peer in SharedServices):
"""
green_service_network_vpc_association = vpclattice.ServiceNetworkVpcAssociation("GreenServiceNetworkVpcAssociation",
    security_group_ids=[green_lattice_sg.id],
    service_network_identifier=green_service_network,
    vpc_identifier=green_network_vpc,
    tags=baseline_cost_tags_native,
    opts=ResourceOptions(
        parent=green_service_network,
        depends_on=[
            green_service_network,
            green_network_vpc
        ]
    )
)

blue_service_network_vpc_association = vpclattice.ServiceNetworkVpcAssociation("BlueServiceNetworkVpcAssociation",
    security_group_ids=[blue_lattice_sg.id],
    service_network_identifier=blue_service_network,
    vpc_identifier=blue_network_vpc,
    tags=baseline_cost_tags_native,
    opts=ResourceOptions(
        parent=blue_service_network,
        depends_on=[
            blue_service_network,
            blue_network_vpc
        ]
    )
)

Finally, create the "SharedService" Lambda function and register it as a Lattice Service (app_infra.py):

from pulumi import ResourceOptions, FileArchive, AssetArchive, export
from pulumi_aws import lb, iam, lambda_, ec2
from pulumi_aws_native import vpclattice
import json

from settings import baseline_cost_tags, deployment_region, baseline_cost_tags_native, blue_principal, green_principal
from helpers import current_account_id
from vpc import main_lambda_alb_sg, shared_services_app_subnets, shared_services_main_vpc
from lattice import green_service_network, blue_service_network

"""
Constructs a private Application Load Balancer for the Lambda function:
"""
# This one costs money:
shared_service_alb = lb.LoadBalancer("DirectoryServiceLb",
    internal=True,
    load_balancer_type="application",
    security_groups=[main_lambda_alb_sg.id],
    subnets=shared_services_app_subnets,
    enable_deletion_protection=False,
    tags={**baseline_cost_tags, "Name": "DirectoryServiceLb"}
)

shared_service_target_group = lb.TargetGroup("DirectoryServiceTg",
    target_type="lambda",
    tags={**baseline_cost_tags, "Name": "DirectoryServiceTg"}
)

# This one costs money (via ALB dependency):
shared_service_listener = lb.Listener("DirectoryServiceListener",
    load_balancer_arn=shared_service_alb.arn,
    port=80,
    protocol="HTTP",
    default_actions=[lb.ListenerDefaultActionArgs(
        type="forward",
        target_group_arn=shared_service_target_group.arn,
    )]
)

"""
Creates a Lambda security group in the main VPC:
"""
main_lambda_func_sg = ec2.SecurityGroup("MainLambdaFuncSecurityGroup",
    description="Lambda Security Group",
    vpc_id=shared_services_main_vpc.id,
    ingress=[ec2.SecurityGroupIngressArgs(
        description="Any",
        from_port=0,
        to_port=0,
        protocol="-1",
        cidr_blocks=["0.0.0.0/0"],
        ipv6_cidr_blocks=["::/0"],
    )],
    egress=[ec2.SecurityGroupEgressArgs(
        from_port=0,
        to_port=0,
        protocol="-1",
        cidr_blocks=["0.0.0.0/0"],
        ipv6_cidr_blocks=["::/0"],
    )],
    tags={**baseline_cost_tags, "Name": f"MainLambdaFuncSecurityGroup-{deployment_region}"},
    opts=ResourceOptions(parent=shared_services_main_vpc)
)


"""
Constructs a Lambda IAM role
"""
directory_service_execution_role = iam.Role("DirectoryServiceExecutionRole",
    assume_role_policy=json.dumps({
        "Version": "2012-10-17",
        "Statement": [{
            "Action": "sts:AssumeRole",
            "Effect": "Allow",
            "Sid": "",
            "Principal": {
                "Service": "lambda.amazonaws.com",
            },
        }],
    }),
    tags={**baseline_cost_tags, "Name": "DirectoryServiceExecutionRole"}
)

lambda_vpc_managed_policy = iam.get_policy(name="AWSLambdaVPCAccessExecutionRole")

directory_service_execution_role_policy_attachment = iam.RolePolicyAttachment("DirectoryServiceExecutionRolePolicyAttachment",
    policy_arn=lambda_vpc_managed_policy.arn,
    role=directory_service_execution_role
)

"""
Creates the DirectoryService Lambda layer:
"""

layer_asset = FileArchive("./directory_app_layer/lambda_layer.zip")

service_layer = lambda_.LayerVersion("DirectoryServiceLayer",
    compatible_runtimes=["python3.8"],
    code=layer_asset,
    layer_name="lambda_layer"
)

"""
Constructs the DirectoryService Lambda function:
"""
service_function = lambda_.Function("DirectoryServiceFunction",
    tags={**baseline_cost_tags, "Name": "DirectoryServiceFunction"},
    role=directory_service_execution_role.arn,
    vpc_config=lambda_.FunctionVpcConfigArgs(
        security_group_ids=[main_lambda_func_sg.id],
        subnet_ids=[s.id for s in shared_services_app_subnets]
    ),
    replace_security_groups_on_destroy=True,
    name="DirectoryServiceFunction",
    runtime="python3.8",
    layers=[service_layer],
    code=AssetArchive(
        {
            ".": FileArchive("./directory_app"),
        }
    ),
    handler="main.lambda_handler"
)

"""
Adds the ALB Lambda invoke permission:
"""
alb_invoke_permission = lambda_.Permission("AlbFunctionPermission",
    action="lambda:InvokeFunction",
    function=service_function.name,
    principal="elasticloadbalancing.amazonaws.com",
    source_arn=shared_service_target_group.arn
)

"""
Attaches the Lambda function to the target group:
"""
service_function_target_group_attachment = lb.TargetGroupAttachment("FunctionTargetGroupAttachment",
    target_group_arn=shared_service_target_group.arn,
    target_id=service_function.arn,
    opts=ResourceOptions(depends_on=[alb_invoke_permission]))

"""
Creates a Lattice target group:
"""
shared_service_lattice_tg = vpclattice.TargetGroup("SharedServiceTargetGroup",
    type="ALB",
    config=vpclattice.TargetGroupConfigArgs(
        port=80,
        protocol="HTTP",
        vpc_identifier=shared_services_main_vpc.id
    ),
    name="shared-service-target",
    tags=baseline_cost_tags_native,
    targets=[vpclattice.TargetGroupTargetArgs(
        id=shared_service_alb.id,
        port=80
    )]
)

"""
Creates a Lattice service and listener:
"""
# From here big money:
shared_service_lattice_service = vpclattice.Service("SharedServiceLatticeService",
    name="shared-service",
    auth_type="AWS_IAM",
    tags=baseline_cost_tags_native
)

export("shared-service-fqdn", shared_service_lattice_service.dns_entry.domain_name)

shared_service_lattice_service_listener = vpclattice.Listener("SharedServiceLatticeServiceListener",
    name="shared-service-listener",
    protocol="HTTP",
    port=80,
    service_identifier=shared_service_lattice_service,
    tags=baseline_cost_tags_native,
    default_action=vpclattice.ListenerDefaultActionArgs(
        forward=vpclattice.ListenerForwardArgs(
            target_groups=[
                vpclattice.ListenerWeightedTargetGroupArgs(
                    target_group_identifier=shared_service_lattice_tg,
                    weight=100
                )
            ]
        )
    ),
    opts=ResourceOptions(
        depends_on=[
            shared_service_lattice_service
        ]
    )
)

blue_service_lattice_service_association = vpclattice.ServiceNetworkServiceAssociation("BlueServiceLatticeServiceAssociation",
    service_identifier=shared_service_lattice_service,
    service_network_identifier=blue_service_network,
    tags=baseline_cost_tags_native,
    opts=ResourceOptions(
        depends_on=[
            shared_service_lattice_service
        ]
    )
)

green_service_lattice_service_association = vpclattice.ServiceNetworkServiceAssociation("GreenServiceLatticeServiceAssociation",
    service_identifier=shared_service_lattice_service,
    service_network_identifier=green_service_network,
    tags=baseline_cost_tags_native,
    opts=ResourceOptions(
        depends_on=[
            shared_service_lattice_service
        ]
    )
)

shared_service_auth_policy = vpclattice.AuthPolicy("SharedServiceAuthPolicy",
    policy={
        "Version":"2012-10-17",
        "Statement":[
            {
                "Effect":"Allow",
                "Principal": "*",
                "Action":"vpc-lattice-svcs:Invoke",
                "Resource": "*",
                "Condition":{
                    "StringEquals": {
                        "vpc-lattice-svcs:RequestMethod": "GET",
                        "vpc-lattice-svcs:RequestQueryString/q": "blue",
                        "vpc-lattice-svcs:SourceVpcOwnerAccount": f"{blue_principal}"
                    }
                }
            },
            {
                "Effect":"Allow",
                "Principal": "*",
                "Action":"vpc-lattice-svcs:Invoke",
                "Resource": "*",
                "Condition":{
                    "StringEquals": {
                        "vpc-lattice-svcs:RequestMethod": "GET",
                        "vpc-lattice-svcs:RequestQueryString/q": "green",
                        "vpc-lattice-svcs:SourceVpcOwnerAccount": f"{green_principal}"
                    }
                }
            }
        ]
    },
    resource_identifier=shared_service_lattice_service,
    opts=ResourceOptions(
        depends_on=[
            shared_service_lattice_service
        ]
    )
)

Notice above how we restricted access to the Lattice Service using the "vpc-lattice-svcs:RequestMethod", "vpc-lattice-svcs:RequestQueryString/q", and "vpc-lattice-svcs:SourceVpcOwnerAccount" parameters, where "q" is just a query string key.

The function is rudimentary and uses a single Lambda layer for FastAPI and Mangum. I provided the layer for your convenience as a .zip archive inside the GitHub repository, so you do not have to rebuild it.

from fastapi import FastAPI
from mangum import Mangum

# Initialise the app
app = FastAPI()

@app.get("/")
async def root(q: str):
    return {"endpoint": f"{q}"}

lambda_handler = Mangum(app)

By design, we only allow GET requests from the BlueFunction to the "http://lattice-shared-service-fqdn/?q=blue" endpoint to succeed. Identically, only GET requests from the GreenFunction to the "http://lattice-shared-service-fqdn/?q=green" endpoint will succeed.

Pulumi Stack: Blue and Green Services

Our "blue-service" and "green-service" stacks are identical apart from configuration, so we will only focus on one. The Pulumi project contains the following files:

settings.py - holds generic settings and Pulumi configuration variables.
helpers.py - holds generic helper functions, if any.
vpc.py - constructs a single VPC.
app_infra.py - constructs everything around the Lambda function, including the ALB, IAM and the Lambda itself.
main.py - executes the code.

Let's take a look at app_infra.py specifically here:

from pulumi import ResourceOptions, FileArchive, AssetArchive
from pulumi_aws import lb, iam, lambda_, ec2
import json

from settings import baseline_cost_tags, deployment_region, shared_service_fqdn
from vpc import blue_sandbox_vpc, blue_sandbox_app_subnets

"""
Lambda security group
"""

blue_lambda_func_sg = ec2.SecurityGroup("BlueLambdaFuncSecurityGroup",
    description="Lambda Security Group",
    vpc_id=blue_sandbox_vpc.id,
    ingress=[ec2.SecurityGroupIngressArgs(
        description="Any",
        from_port=0,
        to_port=0,
        protocol="-1",
        cidr_blocks=["0.0.0.0/0"],
        ipv6_cidr_blocks=["::/0"],
    )],
    egress=[ec2.SecurityGroupEgressArgs(
        from_port=0,
        to_port=0,
        protocol="-1",
        cidr_blocks=["0.0.0.0/0"],
        ipv6_cidr_blocks=["::/0"],
    )],
    tags={**baseline_cost_tags, "Name": f"BlueLambdaFuncSecurityGroup-{deployment_region}"},
    opts=ResourceOptions(parent=blue_sandbox_vpc)
)

"""
Constructs a Lambda IAM role
"""
blue_lambda_execution_role = iam.Role("BlueLambdaExecutionRole",
    assume_role_policy=json.dumps({
        "Version": "2012-10-17",
        "Statement": [{
            "Action": "sts:AssumeRole",
            "Effect": "Allow",
            "Sid": "",
            "Principal": {
                "Service": "lambda.amazonaws.com",
            },
        }],
    }),
    tags={**baseline_cost_tags, "Name": "BlueLambdaExecutionRole"}
)

lambda_vpc_managed_policy = iam.get_policy(name="AWSLambdaVPCAccessExecutionRole")

blue_lambda_execution_role_policy_attachment = iam.RolePolicyAttachment("BlueLambdaRolePolicyAttachment",
    policy_arn=lambda_vpc_managed_policy.arn,
    role=blue_lambda_execution_role
)

"""
Constructs the Blue Lambda function:
"""
blue_function = lambda_.Function("BlueFunction",
    tags={**baseline_cost_tags, "Name": "BlueFunction"},
    role=blue_lambda_execution_role.arn,
    vpc_config=lambda_.FunctionVpcConfigArgs(
        security_group_ids=[blue_lambda_func_sg.id],
        subnet_ids=[s.id for s in blue_sandbox_app_subnets]
    ),
    replace_security_groups_on_destroy=True,
    name="BlueFunction",
    runtime="python3.8",
    code=AssetArchive(
        {
            ".": FileArchive("./blue_app"),
        }
    ),
    handler="main.lambda_handler",
    environment=lambda_.FunctionEnvironmentArgs(
        variables={
            "BLUE_ENDPOINT": f"http://{shared_service_fqdn}/?q=blue"
        }
    )
)

This is a rudimentary Lambda function to execute a GET request towards our shared service Lattice endpoint.

Testing Connectivity

You may have noticed that our environments are private, and all Lambda functions reside within their dedicated VPCs, which have no route to WAN. Let's create a default test execution of our BlueFunction. First, we ensure the BLUE_ENDPOINT environment variable contains the "q=blue" query string:

Executing the function will produce the desired result:

Let's now manually edit the query string within the BLUE_ENDPOINT environment variable:

And execute the function one more time:

Success! Feel free to perform the same steps with the GreenFunction.

Cleaning Up

Run "pulumi destroy" on all the stacks in reverse order. Start from "green-service", then "blue-service", then "common-service". Afterwards, delete CloudWatch Lambda log groups from all three accounts manually if needed. Due to AWS's throttling, you may need to manually delete some VPC Lattice objects and do a "pulumi refresh" instead, although there is usually no need. If you experience throttling, running "pulumi destroy" several times resolves the issue.

VPC Lattice Part 1. Overview

svodwood — Mon, 03 Apr 2023 08:41:54 +0000

Introduction

On the 31st of March 2023, VPC Lattice made General Availability. The new managed service is here to help us create logical application layer networks across VPCs and isolated AWS accounts. A user guide is available, along with several examples. Throughout several posts, I will present an overview of the offering, some implementation details, and a deep dive into a potential reference architecture. This post is Part 1 of the series on Lattice and does not include an IaC follow-along.

What Is Lattice and Why Do I Need It

Consider the use case: your company's landing zone on AWS consists of multiple isolated accounts managed from within a single AWS Organization. Each development team is assigned an account to deploy their services. To make service endpoints available for other teams in different AWS accounts to consume, you must ensure network connectivity between these accounts. Service-to-service connectivity is considered east-west traffic that does not leave your internal network boundary. To interconnect, you will consider several standard options:

To route through Transit Gateway - virtually connecting multiple VPCs through a central router and establishing omnidirectional connectivity between VPCs.
To publish each service as a PrivateLink endpoint. In such cases, a service consumer can establish unidirectional connectivity towards the service provider - an isolated development team's VPC.

VPC Lattice addresses the pains associated with the second option. Let's explore our use case further. The development team in an isolated account manages an EKS cluster, exposing the applications internally via an AWS Load Balancer Controller. From the cloud perspective: we have a private ALB as an entry point to the workload. To publish this ALB for east-west consumption via PrivateLink, we have to build a "load balancer sandwich":

Create a Network Load Balancer.
Create a Network Load Balancer Target Group with a Target Type of Application Load Balancer.
Create a Network Load Balancer Listener.
Take care of SSL termination.
Create a PrivateLink endpoint service, pointing to the Network Load Balancer.
Add AWS principles to the Allow List of the endpoint service or otherwise manage ad-hoc connection requests.

While the above steps can be easily scripted and packaged as a reusable IaC component, work is still required. In addition, cost becomes an issue with tens or hundreds of services, and Transit Gateway connectivity may become preferential.

Lattice aims to provide a single-click tool to eliminate the steps. A value-added proposition of Lattice is the additional security control we can implement on top of our endpoint: auth policies. Auth policies allow restricting endpoint access to specific VPCs, accounts or even specific assumed IAM roles - which is excellent.

Consider an application running in EKS in Account A that must perform an HTTP GET request to an application running in ECS in Account B. A Lattice service in Account B may have an auth policy specifying the exact IAM role assumed by the requester pod in account A through an OIDC-federated identity. We can lock it down further by only setting the allowed HTTP method GET.

From the DevOps perspective, eliminating dependencies between dedicated cloud administrators and developers will improve delivery and backlog churn. But we still assume some level of netsec awareness by the development teams to manage the auth policies for their services. And we still need to establish a framework for governing the auth policies and enforcing the standards.

Limitations

Lattice's visible and significant downside is a one-to-one relationship between a VPC and a Service Network. This downside eliminates the possibility of building around a "shared services" scenario with network isolation. Consider the following situation:

Three VPCs in three separate AWS accounts belonging to an AWS Organization: SharedServices VPC, GreenTeam VPC, and BlueTeam VPC.
SharedServices VPC hosts a SharedApp.
GreenTeam VPC and BlueTeam VPC host GreenApp and BlueApp, respectively.
GreenApp and BlueApp are allowed and must initiate HTTP requests to the SharedApp.
GreenApp and BlueApp are not allowed and must not initiate HTTP requests to each other.
SharedApp is prohibited and must not initiate HTTP requests to BlueApp and GreenApp.

From a network isolation perspective, we would likely establish unidirectional connectivity with PrivateLink in the following pattern:

GreenApp (service consumer) -> SharedApp (service owner)
BlueApp (service consumer) -> SharedApp (service owner)

In the world of Lattice, GreenApp and BlueApp belong to two separate Service Networks, and they never need to meet. However, this would require the SharedApp to be simultaneously present on both Service Networks, which is impossible by design: you will get an "ConflictException: VPC has already been associated to a service network" exception if you try.

The above limitation means we will require a single flat Service Network within the accounts' constellation. An alternative would be to revert to other networking tools at our disposal, e.g., PrivateLink and Transit Gateway. Another thing to remember is that Lattice objects work with AWS RAM and, as such, are regional. I.e. to establish inter-regional connectivity, we would still have to use Transit Gateway Inter-region Peering.

Benefits

Regardless of the above limitations, using AWS Lattice is still beneficial in some situations. In principle, Lattice is a service discovery and networking solution that aims to simplify developers' lives by abstracting the steps required to publish an application over the local network boundary for east-west consumption across VPCs and AWS accounts. In the following posts, we will explore the implementation hands-on and look at a possible reference architecture to understand when it's best to use Lattice and when not to, possibly substituting it with other networking tools at our disposal. Stay tuned!

AWS EC2 Auto Scaling, Target Tracking Policies and Prometheus Exporters

svodwood — Mon, 16 Jan 2023 09:59:23 +0000

Introduction

Quite a few workloads may still require deployment to virtual machines instead of container orchestration platforms, yet still, have to be intelligently autoscaled. For example, a compound workload running on a stateless EC2 instance in an EC2 Autoscaling Group may emit various Prometheus-compatible metrics via Prometheus exporters. A simplified model for such a use case could be an Amazon Linux 2 instance running a node-exporter, exposing OS metrics, and a workload application delivering its own set of Prometheus-compatible metrics.

This post will illustrate how to configure an EC2 Autoscaling Group to use these metrics within Target Tracking Scaling Policies with the help of the AWS Cloudwatch agent.

Pulumi IaC will help us bring up our infrastructure on the AWS Cloud. Check out pulumi.com if you still need to become familiar with it. You can deploy this demo stack using the Pulumi button below.

You may find the source code for this demo in this Github repo.

What We Are Going To Build

We will create a pair of identical EC2 instances running in an EC2 Autoscaling Group behind a public Application Load Balancer, each running a demo web workload and exposing node-exporter and web-server metrics. Next, AWS Cloudwatch agent will scrape the metrics and publish them to Cloudwatch. Finally, a Target Tracking Scaling Policy will govern autoscaling decisions based on a customized metric specification.

To start building, we are going to need to meet the prerequisites:

An AWS account with a named AWS CLI profile configured.
A working Pulumi environment.

Basic Infrastructure

First, we will construct a VPC consisting of a pair of public subnets and a pair of private subnets:

"""
vpc.py
"""
# Create a VPC and Internet Gateway:
demo_vpc = ec2.Vpc("demo-vpc",
    cidr_block=demo_vpc_cidr,
    enable_dns_hostnames=True,
    enable_dns_support=True,
    tags={**general_tags, "Name": f"demo-vpc-{config.region}"}
)

demo_igw = ec2.InternetGateway("demo-igw",
    vpc_id=demo_vpc.id,
    tags={**general_tags, "Name": f"demo-igw-{config.region}"},
    opts=pulumi.ResourceOptions(parent=demo_vpc)
)

# Create a default any-any security group for demo purposes:
demo_sg = ec2.SecurityGroup("demo-security-group",
    description="Allow any-any",
    vpc_id=demo_vpc.id,
    ingress=[ec2.SecurityGroupIngressArgs(
        description="Any",
        from_port=0,
        to_port=0,
        protocol="-1",
        cidr_blocks=["0.0.0.0/0"],
        ipv6_cidr_blocks=["::/0"],
    )],
    egress=[ec2.SecurityGroupEgressArgs(
        from_port=0,
        to_port=0,
        protocol="-1",
        cidr_blocks=["0.0.0.0/0"],
        ipv6_cidr_blocks=["::/0"],
    )],
    tags={**general_tags, "Name": f"demo-sg-{config.region}"},
    opts=pulumi.ResourceOptions(parent=demo_vpc)
)

# Create subnets:
demo_azs = get_availability_zones(state="available").names
demo_public_subnets = []
demo_private_subnets = []

for i in range(2):
    prefix = f"{demo_azs[i]}"

    demo_public_subnet = ec2.Subnet(f"demo-public-subnet-{prefix}",
        vpc_id=demo_vpc.id,
        cidr_block=demo_public_subnet_cidrs[i],
        availability_zone=demo_azs[i],
        tags={**general_tags, "Name": f"demo-public-subnet-{prefix}"},
        opts=pulumi.ResourceOptions(parent=demo_vpc)
    )

    demo_public_subnets.append(demo_public_subnet)

    demo_public_route_table = ec2.RouteTable(f"demo-public-rt-{prefix}",
        vpc_id=demo_vpc.id,
        tags={**general_tags, "Name": f"demo-spoke-rt-{prefix}"},
        opts=pulumi.ResourceOptions(parent=demo_public_subnet)
    )

    demo_public_route_table_association = ec2.RouteTableAssociation(f"demo-public-rt-association-{prefix}",
        route_table_id=demo_public_route_table.id,
        subnet_id=demo_public_subnet.id,
        opts=pulumi.ResourceOptions(parent=demo_public_subnet)
    )

    demo_public_wan_route = ec2.Route(f"demo-public-wan-route-{prefix}",
        route_table_id=demo_public_route_table.id,
        gateway_id=demo_igw.id,
        destination_cidr_block="0.0.0.0/0",
        opts=pulumi.ResourceOptions(parent=demo_public_subnet)
    )

    demo_eip = ec2.Eip(f"demo-eip-{prefix}",
        tags={**general_tags, "Name": f"demo-eip-{prefix}"},
        opts=pulumi.ResourceOptions(parent=demo_vpc)
    )

    demo_nat_gateway = ec2.NatGateway(f"demo-nat-gateway-{prefix}",
        allocation_id=demo_eip.id,
        subnet_id=demo_public_subnet.id,
        tags={**general_tags, "Name": f"demo-nat-{prefix}"},
        opts=pulumi.ResourceOptions(depends_on=[demo_vpc])
    )

    demo_private_subnet = ec2.Subnet(f"demo-private-subnet-{prefix}",
        vpc_id=demo_vpc.id,
        cidr_block=demo_private_subnet_cidrs[i],
        availability_zone=demo_azs[i],
        tags={**general_tags, "Name": f"demo-private-subnet-{prefix}"},
        opts=pulumi.ResourceOptions(parent=demo_vpc)
    )

    demo_private_subnets.append(demo_private_subnet)

    demo_private_route_table = ec2.RouteTable(f"demo-private-rt-{prefix}",
        vpc_id=demo_vpc.id,
        tags={**general_tags, "Name": f"demo-private-rt-{prefix}"},
        opts=pulumi.ResourceOptions(parent=demo_private_subnet)
    )

    demo_private_route_table_association = ec2.RouteTableAssociation(f"demo-private-rt-association-{prefix}",
        route_table_id=demo_private_route_table.id,
        subnet_id=demo_private_subnet.id,
        opts=pulumi.ResourceOptions(parent=demo_private_subnet)
    )

    demo_private_wan_route = ec2.Route(f"demo-private-wan-route-{prefix}",
        route_table_id=demo_private_route_table.id,
        nat_gateway_id=demo_nat_gateway.id,
        destination_cidr_block="0.0.0.0/0",
        opts=pulumi.ResourceOptions(parent=demo_private_subnet)
    )

Second, let's create an Application Load Balancer, a Target Group and Listener:

"""
alb.py
"""
# Create a load balancer:
demo_alb = lb.LoadBalancer("demo-alb",
    internal=False,
    load_balancer_type="application",
    security_groups=[demo_sg.id],
    subnets=demo_public_subnets,
    enable_deletion_protection=False,
    tags={**general_tags, "Name": "demo-alb"}
)

# Create a target group:
demo_target_group = lb.TargetGroup("demo-target-group",
    port=80,
    protocol="HTTP",
    vpc_id=demo_vpc.id,
    tags={**general_tags, "Name": "demo-alb"},
    health_check=lb.TargetGroupHealthCheckArgs(
        enabled=True,
        healthy_threshold=3,
        interval=10,
        protocol="HTTP",
        port="80"
    ),
    opts=pulumi.ResourceOptions(parent=demo_alb)
)

# Create a listener:
demo_listener = lb.Listener("demo-listener",
    load_balancer_arn=demo_alb.arn,
    port=80,
    protocol="HTTP",
    default_actions=[lb.ListenerDefaultActionArgs(
        type="forward",
        target_group_arn=demo_target_group.arn,
    )],
    opts=pulumi.ResourceOptions(parent=demo_alb)
)

With our basic infrastructure completed, we can continue configuring our EC2 web server, which will host all our custom configurations.

Nginx, Node Exporter, Nginx Prometheus Exporter, Cloudwatch Agent - Oh My!

The stack extensively uses AWS SSM Parameter Store, where we push all relevant configuration parameters. As a result, an Amazon Linux 2 instance configuration using user data during boot is effortless - it is simply a matter of fetching the correct parameter with AWS CLI:

aws ssm get-parameter --name {{ cwa_prometheus_parameter_path }} --region {{ region }} --with-decryption --output text --query Parameter.Value > /opt/aws/amazon-cloudwatch-agent/etc/prometheus.yml

AWS SSM Parameter Store is also one of the easiest ways to configure AWS Cloudwatch agent natively, but we will get to this later. Onwards!

Our primary web server workload will be represented by an Nginx server, exposing a default page over port 80 and emulating a workload application. We will use Amazon Linux 2 x86_64 as our OS. On the observability layer, we will install Node Exporter to expose OS-level metrics and Nginx Prometheus Exporter to query the Nginx stub_status endpoint and convert this information into a Prometheus-compatible set of metrics. Both will run on every EC2 instance in the Autoscaling Group as system processes.

Let's take care of enabling the stub_status Nginx module:

"""
nginx_config.py
"""
demo_nginx_stub_status_configuration_file = Template("""
server {
        listen 127.0.0.1:{{ port }};
        location = /metrics {
                stub_status;
        }
}
""")
demo_nginx_stub_status_configuration = demo_nginx_stub_status_configuration_file.render(port=nginx_stub_status_port)

demo_nginx_configuration_parameter = ssm.Parameter("demo-nginx-config",
    type="String",
    data_type="text",
    name=nginx_stub_status_config_parameter_path,
    tags={**general_tags, "Name": "demo-nginx-config"},
    value=demo_nginx_stub_status_configuration
)

Now, let's define our Cloudwatch agent configuration:

"""
cwa.py
"""
# Cloudwatch Agent configuration, written to SSM parameter store:
demo_cwa_configuration = json.dumps({
    "agent":{
        "metrics_collection_interval":60
    },
    "logs":{
        "metrics_collected":{
            "prometheus":{
                "cluster_name":f"{cluster_name}",
                "log_group_name":f"{cluster_name}Prometheus",
                "prometheus_config_path":"/opt/aws/amazon-cloudwatch-agent/etc/prometheus.yml",
                "emf_processor":{
                    "metric_declaration_dedup":True,
                    "metric_namespace":f"{cluster_name}_Prometheus",
                    "metric_unit":{
                        "node_sockstat_TCP_inuse":"Count",
                        "nginx_connections_waiting":"Count"
                    },
                    "metric_declaration":[
                        {
                            "source_labels":[
                                "origin"
                            ],
                            "label_matcher":"^web-server-node-exporter$",
                            "dimensions":[
                                [
                                    "AutoScalingGroupName",
                                    "node"
                                ],
                                [
                                    "AutoScalingGroupName"
                                ]
                            ],
                            "metric_selectors":[
                                "^node_sockstat_TCP_inuse$"
                            ]
                        },
                        {
                            "source_labels":[
                                "origin"
                            ],
                            "label_matcher":"^web-server-nginx$",
                            "dimensions":[
                                [
                                    "AutoScalingGroupName",
                                    "node"
                                ],
                                [
                                    "AutoScalingGroupName"
                                ]
                            ],
                            "metric_selectors":[
                                "^nginx_connections_waiting$"
                            ]
                        }
                    ]
                }
            }
        }
    },
    "metrics":{
        "namespace":f"{cluster_name}_Memory",
        "append_dimensions":{
            "AutoScalingGroupName":"${aws:AutoScalingGroupName}",
            "node":"${aws:InstanceId}"
        },
        "aggregation_dimensions":[
            [
                "AutoScalingGroupName"
            ]
        ],
        "metrics_collected":{
            "mem":{
                "measurement":[
                    {
                        "name":"mem_used_percent",
                        "rename":"MemoryUtilization",
                        "unit":"Percent"
                    }
                ],
                "metrics_collection_interval":60
            }
        }
    }
})

demo_ssm_cwa_configuration_parameter = ssm.Parameter("demo-cwa-config",
    type="String",
    data_type="text",
    name=cwa_settings_parameter_path,
    tags={**general_tags, "Name": "demo-cwa-config"},
    value=demo_cwa_configuration
)

# Cloudwatch Agent prometheus scrape configuration, written to SSM parameter store:
demo_prometheus_configuration_template = Template("""
global:
  scrape_interval: 1m
  scrape_timeout: 10s
scrape_configs:
  - job_name: Local_NodeExporter_Metrics
    static_configs:
      - targets:
          - 'localhost:9100'
        labels:
          origin: web-server-node-exporter
          AutoScalingGroupName: {{autoscaling_group_name}}
  - job_name: Local_Nginx_Metrics
    static_configs:
      - targets:
          - 'localhost:9113'
        labels:
          origin: web-server-nginx
          AutoScalingGroupName: {{autoscaling_group_name}}
""")

demo_prometheus_configuration = demo_prometheus_configuration_template.render(autoscaling_group_name=cluster_name)

demo_ssm_cwa_promscrape_configuration_parameter = ssm.Parameter("demo-cwa-prom-config",
    type="String",
    data_type="text",
    name=cwa_prometheus_parameter_path,
    tags={**general_tags, "Name": "demo-cwa-prom-config"},
    value=demo_prometheus_configuration
)

# Create a CloudWatch log group for the prometheus metrics:
demo_prom_loggroup = cloudwatch.LogGroup("demo-cwa-log-group",
    name=f"{cluster_name}Prometheus",
    retention_in_days=1,
    tags={**general_tags, "Name": "demo-cwa-log-group"}
)

Several noteworthy things happened here. First, we set up the Cloudwatch agent for Prometheus metrics collection. Cloudwatch agent supports standard Prometheus scrape configurations. For example, we are interested in scraping a Node Exporter endpoint on port 9100 and an Nginx Prometheus Exporter endpoint on port 9113. Both jobs will append an "AutoScalingGroupName" tag to the metrics. Second, we configure the {"logs":{"metrics_collected":{"prometheus":{}}}} stanza to only include "node_sockstat_TCP_inuse" and "nginx_connections_waiting" in our aggregations since, in this case, we are confident we can make autoscaling decisions based on these metrics alone. Notice that we will aggregate both metrics by "AutoScalingGroupName". Finally, to top it off, let's add the built-in memory reporting inside the {"metrics":{"metrics_collected":{"mem":{}}}} stanza - it's always nice to stay on top of our memory utilization.

With our Cloudwatch agent configuration sorted, let's move on to our EC2 user data.

Configuring the Instance at Boot

A straightforward way to do this is via a bash script:

#!/bin/bash
yum update -y

# Install CWA
yum install amazon-cloudwatch-agent -y

# Install Nginx and configure the stub status module
amazon-linux-extras install nginx1.12 -y 
aws ssm get-parameter --name {{ nginx_stub_status_config_parameter_path }} --region {{ region }} --output text --query Parameter.Value > {{ nginx_config_file_path }}

# Add a user to run the exporters
useradd --no-create-home metrics_exporter

# Install Node Exporter
wget https://github.com/prometheus/node_exporter/releases/download/v1.4.0/node_exporter-1.4.0.linux-amd64.tar.gz
tar xzf node_exporter-1.4.0.linux-amd64.tar.gz
cp node_exporter-1.4.0.linux-amd64/node_exporter /usr/local/bin/node_exporter
rm -rf node_exporter-1.4.0.linux-amd64.tar.gz node_exporter-1.4.0.linux-amd64

# Create Node Exporter systemd service
cat << 'EOF' > /etc/systemd/system/node-exporter.service
[Unit]
Description=Prometheus Node Exporter Service
After=network.target
[Service]
User=metrics_exporter
Group=metrics_exporter
Type=simple
ExecStart=/usr/local/bin/node_exporter
[Install]
WantedBy=multi-user.target
EOF

# Install Nginx Exporter
wget https://github.com/nginxinc/nginx-prometheus-exporter/releases/download/v0.11.0/nginx-prometheus-exporter_0.11.0_linux_amd64.tar.gz
tar xzf nginx-prometheus-exporter_0.11.0_linux_amd64.tar.gz
cp nginx-prometheus-exporter /usr/local/bin/nginx-prometheus-exporter
rm -rf nginx-prometheus-exporter_0.11.0_linux_amd64.tar.gz nginx-prometheus-exporter

# Create Nginx Exporter systemd service
cat << 'EOF' > /etc/systemd/system/nginx-prometheus-exporter.service
[Unit]
Description=Prometheus Nginx Exporter Service
After=network.target
[Service]
User=metrics_exporter
Group=metrics_exporter
Type=simple
ExecStart=/usr/local/bin/nginx-prometheus-exporter -nginx.scrape-uri http://127.0.0.1:{{ nginx_stub_status_port }}/metrics -web.listen-address=127.0.0.1:9113
[Install]
WantedBy=multi-user.target
EOF

# Configure CWA
aws ssm get-parameter --name {{ cwa_prometheus_parameter_path }} --region {{ region }} --with-decryption --output text --query Parameter.Value > /opt/aws/amazon-cloudwatch-agent/etc/prometheus.yml
/opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a fetch-config -m ec2 -c ssm:"{{ cwa_settings_parameter_path }}" -s

# Start everything up
systemctl daemon-reload
systemctl enable nginx
systemctl start nginx
systemctl enable node-exporter
systemctl start node-exporter
systemctl enable nginx-prometheus-exporter
systemctl start nginx-prometheus-exporter

Here is where we make use of our SSM Parameter Store parameters. Querying is made possible by attaching several managed IAM policies to the instance profile. AWS CLI is available out of the box on an Amazon Linux 2 AMI.

Houston, we have worming!

Generate some continuous GET requests to the Application Load Balancer public DNS name, and after a while, we should see some squiggly lines appear in Cloudwatch.

Custom Namespaces:

Nginx Metric:

Node Exporter Metric:

High-cardinality Log Streams

Target Tracking Scaling Policies

Let's imagine a hypothetical situation where we are sure that "nginx_connections_waiting" and "node_sockstat_TCP_inuse" should have the following average values across our instances in the autoscaling group during production operation at baseline. We are deliberately using some ridiculous values here for the sake of the demo:

target_nginx_connections_waiting = 2
target_node_sockstat_TCP_inuse = 50

To quote the official documentation:

To create a target tracking scaling policy, you specify an Amazon CloudWatch metric and a target value that represents the ideal average utilization or throughput level for your application. Amazon EC2 Auto Scaling can then scale out your group (add more instances) to handle peak traffic, and scale in your group (run fewer instances) to reduce costs during periods of low utilization or throughput.

To achieve this, we will define two Target Tracking scaling policies:

"""
scaling_policies.py
"""
# Define target tracking scaling policies:
nginx_tracking_policy = autoscaling.Policy("demo-nginx-tracking-policy",
    autoscaling_group_name=demo_autoscaling_group.name,
    estimated_instance_warmup=10,
    policy_type="TargetTrackingScaling",
    target_tracking_configuration=autoscaling.PolicyTargetTrackingConfigurationArgs(
        target_value=target_nginx_connections_waiting,
        disable_scale_in=False,
        customized_metric_specification=autoscaling.PolicyTargetTrackingConfigurationCustomizedMetricSpecificationArgs(
            metric_name="nginx_connections_waiting",
            namespace=f"{cluster_name}_Prometheus",
            statistic="Average",
            unit="Count",
            metric_dimensions=[
                autoscaling.PolicyTargetTrackingConfigurationCustomizedMetricSpecificationMetricDimensionArgs(
                    name="AutoScalingGroupName",
                    value=f"{cluster_name}"
                )
            ]
        )
    )
)

netstat_tracking_policy = autoscaling.Policy("demo-netstat-tracking-policy",
    autoscaling_group_name=demo_autoscaling_group.name,
    estimated_instance_warmup=10,
    policy_type="TargetTrackingScaling",
    target_tracking_configuration=autoscaling.PolicyTargetTrackingConfigurationArgs(
        target_value=target_node_sockstat_TCP_inuse,
        disable_scale_in=False,
        customized_metric_specification=autoscaling.PolicyTargetTrackingConfigurationCustomizedMetricSpecificationArgs(
            metric_name="node_sockstat_TCP_inuse",
            namespace=f"{cluster_name}_Prometheus",
            statistic="Average",
            unit="Count",
            metric_dimensions=[
                autoscaling.PolicyTargetTrackingConfigurationCustomizedMetricSpecificationMetricDimensionArgs(
                    name="AutoScalingGroupName",
                    value=f"{cluster_name}"
                )
            ]
        )
    )
)

The policies are visible in the Autoscaling Group user interface:

After several minutes of inactivity and initial data gathering, our Cloudwatch alarms have traced our idle state. The warnings now indicate that we are over-provisioned, running at minimum capacity:

Load Testing

Out Autoscaling Group is now at its minimum capacity of two, with eight instances of maximum allowed capacity.
Let us now test our scale-out and scale-in policies by running a basic load test with the help of Locust:

"""
./load-test/locustfile.py
"""
from locust import HttpUser, task

class User(HttpUser):
    @task
    def mainPage(self):
        self.client.get("/")

Soon enough, we shall see a scale-out event:

The total number of instances is now eight:

Let's stop the load test and wait for the scale-in event. It should take around fifteen minutes, according to the Cloudwatch alarm:

Summing It All Up

We successfully demonstrated that it is possible to effectively manage the EC2 instance capacity of an Autoscaling Group using metrics scraped from various Prometheus exporters - this means implementing autoscaling policies based on application-level metrics in a real-world production scenario.

Cleaning Up

Clean up by running pulumi destroy. Happy autoscaling!