Forem: jhashimoto

Generating AWS CDK Code from Hand-drawn Sketches with GPT-4o

jhashimoto — Tue, 14 May 2024 23:19:33 +0000

Note: This article is an English translation of my original article, which you can find here. The translation was done using ChatGPT.

Introduction

If you can generate HTML from a sketch, why not try generating CDK code? I decided to give it a shot.

// Detect dark theme var iframe = document.getElementById('tweet-1790227959724618123-634'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=1790227959724618123&theme=dark" }

Experiment

To test its image analysis capabilities, I deliberately didn't input any prompts.

Here are the results. It responded in English, possibly due to the lack of a prompt, but it generated the CDK code as expected. It's quite handy that it can even read the instructions written in the image.

To add, the initial instruction in the sketch was simply "Use CDK". Since it unexpectedly generated Python code, I added "in TypeScript".

Here is the generated code.

import * as cdk from 'aws-cdk-lib';
import { Stack, StackProps } from 'aws-cdk-lib';
import { Vpc, SubnetType, InstanceType, InstanceClass, InstanceSize, MachineImage } from 'aws-cdk-lib/aws-ec2';
import { ApplicationLoadBalancer, ApplicationProtocol, ListenerAction } from 'aws-cdk-lib/aws-elasticloadbalancingv2';
import { Construct } from 'constructs';

export class MyVpcStack extends Stack {
  constructor(scope: Construct, id: string, props?: StackProps) {
    super(scope, id, props);

    // Create a VPC
    const vpc = new Vpc(this, 'MyVpc', {
      maxAzs: 2,
      subnetConfiguration: [
        {
          subnetType: SubnetType.PUBLIC,
          name: 'PublicSubnet',
        },
        {
          subnetType: SubnetType.PRIVATE_WITH_NAT,
          name: 'PrivateSubnet',
        },
      ],
    });

    // Create an Application Load Balancer
    const alb = new ApplicationLoadBalancer(this, 'MyALB', {
      vpc,
      internetFacing: true,
    });

    // Add a listener to the ALB
    const listener = alb.addListener('Listener', {
      protocol: ApplicationProtocol.HTTP,
      defaultAction: ListenerAction.fixedResponse(200, {
        contentType: 'text/plain',
        messageBody: 'Hello, world!',
      }),
    });

    // Add EC2 instances in the private subnets
    const ec2Instance1 = new ec2.Instance(this, 'EC2Instance1', {
      vpc,
      instanceType: InstanceType.of(InstanceClass.T2, InstanceSize.MICRO),
      machineImage: MachineImage.latestAmazonLinux(),
      vpcSubnets: {
        subnetType: SubnetType.PRIVATE_WITH_NAT,
      },
    });

    const ec2Instance2 = new ec2.Instance(this, 'EC2Instance2', {
      vpc,
      instanceType: InstanceType.of(InstanceClass.T2, InstanceSize.MICRO),
      machineImage: MachineImage.latestAmazonLinux(),
      vpcSubnets: {
        subnetType: SubnetType.PRIVATE_WITH_NAT,
      },
    });

    // Add the instances to the ALB target group
    listener.addTargets('EC2Targets', {
      port: 80,
      targets: [ec2Instance1, ec2Instance2],
    });
  }
}

Fixing CDK Code Errors

Create a CDK project and replace it with the code above.

mkdir my-vpc
cd my-vpc/
npx cdk init -l typescript

Open it in an editor and fix the errors displayed.

Here is the corrected code.

Fixed errors, deprecated parts, and added permission for ALB to EC2 traffic.

- import * as cdk from 'aws-cdk-lib';
import { Stack, StackProps } from 'aws-cdk-lib';
- import { Vpc, SubnetType, InstanceType, InstanceClass, InstanceSize, MachineImage } from 'aws-cdk-lib/aws-ec2';
+ import { Vpc, SubnetType, InstanceType, InstanceClass, InstanceSize, MachineImage, Instance, Port } from 'aws-cdk-lib/aws-ec2';
import { ApplicationLoadBalancer, ApplicationProtocol, ListenerAction } from 'aws-cdk-lib/aws-elasticloadbalancingv2';
+ import * as elbv2_tg from 'aws-cdk-lib/aws-elasticloadbalancingv2-targets'
import { Construct } from 'constructs';

export class MyVpcStack extends Stack {
  constructor(scope: Construct, id: string, props?: StackProps) {
    super(scope, id, props);

    // Create a VPC
    const vpc = new Vpc(this, 'MyVpc', {
      maxAzs: 2,
      subnetConfiguration: [
        {
          subnetType: SubnetType.PUBLIC,
          name: 'PublicSubnet',
        },
        {
-         subnetType: SubnetType.PRIVATE_WITH_NAT,
+         subnetType: SubnetType.PRIVATE_WITH_EGRESS,
          name: 'PrivateSubnet',
        },
      ],
    });

    // Create an Application Load Balancer
    const alb = new ApplicationLoadBalancer(this, 'MyALB', {
      vpc,
      internetFacing: true,
    });

    // Add a listener to the ALB
    const listener = alb.addListener('Listener', {
      protocol: ApplicationProtocol.HTTP,
      defaultAction: ListenerAction.fixedResponse(200, {
        contentType: 'text/plain',
        messageBody: 'Hello, world!',
      }),
    });

    // Add EC2 instances in the private subnets
-   const ec2Instance1 = new ec2.Instance(this, 'EC2Instance1', {
+   const ec2Instance1 = new Instance(this, 'EC2Instance1', {
      vpc,
      instanceType: InstanceType.of(InstanceClass.T2, InstanceSize.MICRO),
-     machineImage: MachineImage.latestAmazonLinux(),      
+     machineImage: MachineImage.latestAmazonLinux2023(),
      vpcSubnets: {
-       subnetType: SubnetType.PRIVATE_WITH_NAT,
+       subnetType: SubnetType.PRIVATE_WITH_EGRESS,
      },
    });
+   ec2Instance1.connections.allowFrom(alb, Port.tcp(80), 'Allow inbound traffic on port 80 from the ALB only');

-   const ec2Instance1 = new ec2.Instance(this, 'EC2Instance2', {
+   const ec2Instance2 = new Instance(this, 'EC2Instance2', {
      vpc,
      instanceType: InstanceType.of(InstanceClass.T2, InstanceSize.MICRO),
-     machineImage: MachineImage.latestAmazonLinux(),      
+     machineImage: MachineImage.latestAmazonLinux2023(),
      vpcSubnets: {
-       subnetType: SubnetType.PRIVATE_WITH_NAT,
+       subnetType: SubnetType.PRIVATE_WITH_EGRESS,
      },
    });
+   ec2Instance2.connections.allowFrom(alb, Port.tcp(80), 'Allow inbound traffic on port 80 from the ALB only');

    // Add the instances to the ALB target group
    listener.addTargets('EC2Targets', {
      port: 80,
-     targets: [ec2Instance1, ec2Instance2],      
+     targets: [new elbv2_tg.InstanceTarget(ec2Instance1), new elbv2_tg.InstanceTarget(ec2Instance2)],
    });
  }
}

Verify the Operation

Deploy it.

$ npx cdk deploy
# Do you wish to deploy these changes (y/n)? y
# MyVpcStack: deploying... [1/1]
# MyVpcStack: creating CloudFormation changeset...

#  ✅  MyVpcStack

# ✨  Deployment time: 222.7s

# Stack ARN:
# arn:aws:cloudformation:ap-northeast-1:xxxxxxxxxxxx:stack/MyVpcStack/9cebe450-1230-11ef-bf56-0a943cda165d

# ✨  Total time: 226.68s

Access the DNS name of the ALB.

curl http://MyVpcS-MyALB-UrksBygjrlzv-1011360440.ap-northeast-1.elb.amazonaws.com
# <html>
# <head><title>502 Bad Gateway</title></head>
# <body>
# <center><h1>502 Bad Gateway</h1></center>
# </body>
# </html>

Since no web server is configured, a 502 Bad Gateway is returned, but if you configure an EC2 instance, it should work properly.

Conclusion

I was able to generate CDK code from a sketch using GPT-4o. I was surprised that it could even read the instructions written in the image. Although there were errors in the generated code, I fixed them and confirmed it worked correctly.

For production environments or other uses that require high quality, careful review is necessary, but it should be more than sufficient for prototyping and demonstrations.

Setting up a VPC Endpoint for yum with AWS CDK

jhashimoto — Thu, 28 Sep 2023 21:18:11 +0000

Note: This article is an English translation of my original article, which you can find here.

In this article, I demonstrate how to run yum in a private subnet using a VPC endpoint, taking the example of web server configuration via user data. I will also share sample code using AWS CDK.

Background and Objective

I set up a web server at the time of instance launch using user data.



yum update -y
yum install -y httpd
systemctl start httpd
systemctl enable httpd
echo "This is a sample website." > /var/www/html/index.html

However, the yum command fails in environments without internet access. To resolve this issue, I place a VPC endpoint to use the Amazon Linux repository hosted on S3.

By employing this method, I can set up a web server without internet access and without deploying a NAT gateway.

Update yum on my AL1 or AL2 EC2 instance without internet access | AWS re:Post

Architecture

I deploy a gateway-type VPC endpoint for S3.

I use an ALB as the public endpoint for the web server. I also set up VPC endpoints for SSM for remote connections.

How to Use CDK

First, make sure to complete the setup for CDK.

The official workshop provides a detailed procedure from setting up the development environment to deployment.

AWS CDK Intro Workshop | AWS CDK Workshop

Sample Code

I create a CDK project and edit lib/cdk-private-yum-sample-stack.ts.



mkdir cdk-private-yum-sample
cd cdk-private-yum-sample
cdk init -l typescript



import { Stack, StackProps, CfnOutput } from 'aws-cdk-lib';
import * as ec2 from 'aws-cdk-lib/aws-ec2';
import * as elbv2 from 'aws-cdk-lib/aws-elasticloadbalancingv2';
import * as elbv2_tg from 'aws-cdk-lib/aws-elasticloadbalancingv2-targets'
import { Construct } from 'constructs';

export class CdkPrivateYumSampleStack extends Stack {
  constructor(scope: Construct, id: string, props?: StackProps) {
    super(scope, id, props);

    // vpc
    const vpc = new ec2.Vpc(this, 'WebVpc', {
      vpcName: 'web-vpc',
      ipAddresses: ec2.IpAddresses.cidr('172.16.0.0/16'),
      natGateways: 0,
      maxAzs: 2,
      subnetConfiguration: [
        {
          cidrMask: 24,
          name: 'Public',
          subnetType: ec2.SubnetType.PUBLIC
        },
        {
          cidrMask: 24,
          name: 'Private',
          subnetType: ec2.SubnetType.PRIVATE_ISOLATED
        }
      ],
      // remove all rules from default security group
      // See: https://docs.aws.amazon.com/config/latest/developerguide/vpc-default-security-group-closed.html
      restrictDefaultSecurityGroup: true
    });

    // add private endpoints for session manager
    vpc.addInterfaceEndpoint('SsmEndpoint', {
      service: ec2.InterfaceVpcEndpointAwsService.SSM,
    });
    vpc.addInterfaceEndpoint('SsmMessagesEndpoint', {
      service: ec2.InterfaceVpcEndpointAwsService.SSM_MESSAGES,
    });
    vpc.addInterfaceEndpoint('Ec2MessagesEndpoint', {
      service: ec2.InterfaceVpcEndpointAwsService.EC2_MESSAGES,
    });
    // add private endpoint for Amazon Linux repository on s3
    vpc.addGatewayEndpoint('S3Endpoint', {
      service: ec2.GatewayVpcEndpointAwsService.S3,
      subnets: [
        { subnetType: ec2.SubnetType.PRIVATE_ISOLATED }
      ]
    });

    //
    // security groups
    //
    const albSg = new ec2.SecurityGroup(this, 'AlbSg', {
      vpc,
      allowAllOutbound: true,
      description: 'security group for alb'
    })
    albSg.addIngressRule(ec2.Peer.anyIpv4(), ec2.Port.tcp(80), 'allow http traffic from anyone')

    const ec2Sg = new ec2.SecurityGroup(this, 'WebEc2Sg', {
      vpc,
      allowAllOutbound: true,
      description: 'security group for a web server'
    })
    ec2Sg.connections.allowFrom(albSg, ec2.Port.tcp(80), 'allow http traffic from alb')

    //
    // web servers
    //
    const userData = ec2.UserData.forLinux({
      shebang: '#!/bin/bash',
    })
    userData.addCommands(
      // setup httpd
      'yum update -y',
      'yum install -y httpd',
      'systemctl start httpd',
      'systemctl enable httpd',
      'echo "This is a sample website." > /var/www/html/index.html',
    )

    // launch one instance per az
    const targets: elbv2_tg.InstanceTarget[] = new Array();
    for (const [idx, az] of vpc.availabilityZones.entries()) {
      targets.push(
        new elbv2_tg.InstanceTarget(
          new ec2.Instance(this, `WebEc2${idx + 1}`, {
            instanceName: `web-ec2-${idx + 1}`,   // web-ec2-1, web-ec2-2, ...
            instanceType: ec2.InstanceType.of(ec2.InstanceClass.T2, ec2.InstanceSize.MICRO),
            machineImage: ec2.MachineImage.latestAmazonLinux2023(),
            vpc,
            vpcSubnets: vpc.selectSubnets({
              subnetType: ec2.SubnetType.PRIVATE_ISOLATED,
            }),
            availabilityZone: az,
            securityGroup: ec2Sg,
            blockDevices: [
              {
                deviceName: '/dev/xvda',
                volume: ec2.BlockDeviceVolume.ebs(8, {
                  encrypted: true
                }),
              },
            ],
            userData,
            ssmSessionPermissions: true,
            propagateTagsToVolumeOnCreation: true,
          })
        )
      );
    }

    //
    // alb
    //
    const alb = new elbv2.ApplicationLoadBalancer(this, 'Alb', {
      internetFacing: true,
      vpc,
      vpcSubnets: {
        subnets: vpc.publicSubnets
      },
      securityGroup: albSg
    })

    const listener = alb.addListener('HttpListener', {
      port: 80,
      protocol: elbv2.ApplicationProtocol.HTTP
    })
    listener.addTargets('WebEc2Target', {
      targets,
      port: 80
    })

    new CfnOutput(this, 'TestCommand', {
      value: `curl http://${alb.loadBalancerDnsName}`
    })
  }
}

To deploy, execute:



cdk deploy

As a side note, the combination of CDK and GitHub Copilot offers an excellent development experience, and I highly recommend it.

Sample code is placed here:

JHashimoto0518 / cdk-private-yum-sample

This is a sample of using CDK to build a private endpoint for yum on EC2 instances.

cdk private yum sample

This is a sample of using CDK to build a private endpoint for yum on EC2 instances.

diagrams

yum execution

http(s) request

Articles

Japanese

yum用のVPCエンドポイントをCDKで構成してみた - Qiita

View on GitHub

Tested on the following version:



$ cdk --version
2.81.0 (build bd920f2)

Testing

First, I verify the web server's operation.

I connect via the session manager and confirm that it returns responses to HTTP requests. (Sorry, the screenshot is in Japanese)



sh-4.2$ curl http://localhost/
This is a sample website.

Next, I carry out an end-to-end test.

After CDK deployment, a test command appears in the terminal. I copy and run it.



$ curl http://CdkPr-alb8A-1UPL742M6H83S-1170769314.ap-northeast-1.elb.amazonaws.com
This is a sample website.

I verify that the ALB returns the response as expected.

Conclusion

By utilizing VPC endpoints, I can run yum in an environment without internet access. Additionally, using CDK improves the development experience and speeds up the building process.

How to Easily Try Out boto3 Interactively in AWS CloudShell

jhashimoto — Sat, 23 Sep 2023 05:26:52 +0000

Note: This article is an English translation of my original article, which you can find here.

Introduction

In this article, I will explain how to interactively use the Python SDK, boto3, in AWS CloudShell.

CloudShell comes pre-installed with Python and boto3. Additionally, because CloudShell inherits authentication information from the Management Console, you don't need to explicitly pass credentials to boto3, eliminating the need for setup.

This technique is also highly useful for not only manipulating AWS resources but also for trying out boto3 APIs.

What is AWS CloudShell?

AWS CloudShell is a browser-based shell environment that can be accessed directly from the Management Console. It comes pre-installed with AWS CLI, various SDKs, and other shell utilities.

What is AWS CloudShell? - AWS CloudShell

What is boto3?

boto3 is the Python version of the AWS SDK. It is a Python library for programmatic access to AWS resources.

AWS SDK for Python

How to Interactively Run boto3

The steps for interactively running boto3 using AWS CloudShell are as follows:

1. Launch CloudShell

After logging into the Management Console, click the CloudShell icon (>.) located in the header.

2. Launch Python Interpreter

Once the shell launches, activate the Python interpreter by executing the following:

python3

Once the interpreter is launched, you'll see a >>> prompt.

Example Output:

[cloudshell-user@ip-xx-x-xx-xxx ~]$ python3
Python 3.7.16 (default, Aug 30 2023, 20:37:53) 
[GCC 7.3.1 20180712 (Red Hat 7.3.1-15)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>>

You can interactively try out various Python functionalities and boto3 APIs until you exit the interpreter. All subsequent commands will be executed in this interpreter.

Note: The interpreter supports tab completion.

3. Import boto3

Execute the following command to import boto3:

import boto3

4. Initialize Session

Initialize a boto3 session. The authentication information and the region are inherited from the Management Console, but you can also specify them explicitly.

Example to specify the Tokyo region:

sess = boto3.Session(region_name='ap-northeast-1')

It's also possible to pass authentication information when initializing the session, but I'll omit that detail in this article.

5. Manipulate AWS Resources

Once the session is established, you can operate various AWS resources.

For example, to list all security groups:

ec2 = sess.client('ec2')
res = ec2.describe_security_groups()
print(res)

Example Output:

{'SecurityGroups': [{'Description': 'for test', 'GroupName': 'test-sg', 'IpPermissions': [], ...

If you find the JSON output hard to read, you can format it using Python's json module. Specifically, you can use the indent argument of the json.dumps() function to output the data in a neatly formatted manner.

import json
print(json.dumps(res, indent=2))

Example Output:

{
  "SecurityGroups": [
    {
      "Description": "for test",
      "GroupName": "test-sg",
      "IpPermissions": [],
      "OwnerId": "3328xxxxxxxx",
      "GroupId": "sg-xxxxxxxxxxxxxxxxx",
      "IpPermissionsEgress": [
        {
          "IpProtocol": "-1",
          "IpRanges": [
            {
              "CidrIp": "172.24.1.1/32",
              "Description": "allow 172.24.1.1/32"
            }
          ],
          "Ipv6Ranges": [],
          "PrefixListIds": [],
          "UserIdGroupPairs": []
        }
      ],
      "VpcId": "vpc-xxxxxxxxxxxxxxxxx"
    },
    ...

6. Exit the Interpreter

Finally, exit the interpreter by typing:

quit()

Summary

In this article, I explored how to interactively manipulate AWS resources using AWS CloudShell and boto3.

With CloudShell, you can use boto3 without any setup. Furthermore, using the Python interpreter, you can interactively try out boto3 APIs.

How to fix "jq: error string and number cannot be added" in AWS CloudShell?

jhashimoto — Fri, 03 Feb 2023 21:16:36 +0000

Note: This article is an English translation of my original article, which you can find here.

Overview

This article describes the causes and solutions to the error when using jq join in AWS CloudShell.

jq error in AWS CloudShell

This section describes an error that occurs when using jq in AWS CloudShell.

If you try to join an array that contains numbers, you will get jq: error string and number cannot be added.

Expected:



$ echo '["example",0]' | jq 'join(".")'
"example.0"

Actual:



$ echo '["example",0]' | jq 'join(".")'
jq: error (at <stdin>:1): string (".") and number (0) cannot be added

Cause

This section describes the cause of this error.

The version of jq preinstalled in CloudShell is 1.5¹.



$ jq --version
jq-1.5

A bug in the 1.5 version of join does not convert number types into strings, resulting in this error.

L1422 - jq/builtin.c at jq-1.5 · stedolan/jq



  "def join($x): reduce .[] as $i (null; (.//\"\") + (if . == null then $i else $x + $i end))//\"\";",

Note:

Ver 1.6 fixed this bug. You can confirm that in the joined source, which converts the type number to a string with tostring.



def join($x): reduce .[] as $i (null;
            (if .==null then "" else .+$x end) +
            ($i | if type=="boolean" or type=="number" then tostring else .//"" end)
        ) // "";

The issue:

string and number cannot be used for join() #1595

jpcarey posted on Feb 06, 2018

If you run the join example, it errors:

jq: error (at <stdin>:0): string (" ") and number (1) cannot be added
exit status 5

jq 'join(" ")' input: ["a",1,2.3,true,null,false] output: "a 1 2.3 true false"

version 1.5

From the docs:

Numbers and booleans in the input are converted to strings. Null values are treated as empty strings. Arrays and objects in the input are not supported.

View on GitHub

Solution 1

I would describe two means of resolving this error.

First, I explain how to fix it to work with jq1.5.

If you convert the number to a string before passing it to join, it will work as expected in 1.5. And, of course, it also works fine with 1.6.



$ echo '["example",0]' | jq 'map(.|tostring) | join(".")'
"example.0"

Solution 2

Next, I explain how to install jq1.6 in CloudShell.

However, the yum provided with CloudShell does not upgrade the pre-installed jq², so you will download the binary in an appropriate directory.

Note: (about directory persistence)

Please note that if you place binary files in /usr/bin, the files will be deleted when CloudShell is restarted.

If you place the file in your home directory, the data will be persistent, but please note that the file will be deleted when you do not use that CloudShell environment for more than 120 days.

Service quotas and restrictions for AWS CloudShell - AWS CloudShell

Practical Examples

Here is a concrete example to explanate the solution.

Create an installation directory in your home directory and download the binary. Of course, the directory is arbitrary, but here we assume jq1.6.



cd ~
mkdir jq1.6
cd jq1.6
wget -O jq https://github.com/stedolan/jq/releases/download/jq-1.6/jq-linux64
chmod +x ./jq

As mentioned earlier, this bug has been Fixed in 1.6, so it works as expected.



$ ~/jq1.6/jq --version
jq-1.6
$ echo '["example",0]' | ~/jq1.6/jq 'join(".")'
"example.0"

It is helpful to set up an alias.



$ echo "alias jq=~/jq1.6/jq" >> ~/.bashrc
$ . ~/.bashrc  # Reflect the alias in the shell where you are logged in:
$ echo '["example",0]' | jq 'join(".")'
"example.0"

To run as a shell script

Note that when running as a shell script, the login shell environment is not inherited, so the alias is undefined, and it executes with jq 1.5.

./join.sh:



#!/bin/bash
echo '["example",0]' | jq 'join(".")'



$ bash ./join.sh 
jq: error (at <stdin>:1): string (".") and number (0) cannot be added

In this case, if you specify interactive behavior with the -i option, the shell in the execution environment will read ~/.bashrc and execute the script, as expected.



$ bash -i ./join.sh 
"example.0"

Conclusion

This section summarizes how to deal with the error when using jq in CloudShell.

I have described two solutions.

Solution 1 has the advantage of not requiring any changes to the CloudShell environment and is portable as it works with both 1.5 and 1.6. ³

Solution 2, however, has the advantage of running on 1.6, so it can also handle behaviors other than join error.

One of the changes other than join in 1.6:

Different behavior when omitting jq arguments

Please use one of these methods depending on your situation.

Reference

Oct 24 '17 Comments: 1 Answers: 4

I'd like to print the value of members and id from below jq output:

$ cat test_|jq -r '.[] | select(.name=="AAA") | .'
{
  "name": "AAA",
  "members": 10,
  "profiles": 0,
  "templates": 0,
  "ldapGroups": 0,
  "ldapMembers": 0,
  "id": "20"
}

Unfortunately it works for one of each only:

$ cat test_|jq

…

Open Full Question

This is the version at the time of the article submission. It may be upgraded in the future. ↩
I confirmed with AWS support. ↩
If you have a development PC or other environment where you can install freely, you will likely install the latest version, 1.6. ↩