Forem: Leon Stigter

How To Set Up Continuous Integration and Delivery With Github Actions and Akka Serverless

Leon Stigter — Tue, 01 Jun 2021 00:00:00 +0000

CI/CD is one of those quintessential mindset shifts that helps developers automate away the toil of deploying apps. Especially in the realm of serverless, where the whole idea is to focus on the things that matter and let the undifferentiated heavy lifting be handled by others, automating as much as possible is paramount. It helps developers focus on what matters, code, and it helps business focus on what matters, getting quality software to market faster. So how does that work in Akka Serverless?

Akka Serverless focuses on bringing together the worlds of Functions-as-a-Service and serverless databases into a single package. That single package allows developers to focus on code even more as they no longer have to worry about the database. In this short blog post, I’ll walk you through automating the CI and CD workflow for Akka Serverless using GitHub Actions.

The GitHub Actions

In the CI/CD I’ll use four different GitHub Actions:

Docker Login so I can publish the resulting container to Docker Hub (or any other supported Docker registry)
Docker Setup Buildx which uses BuildKit and makes it possible to create multi-platform images (which is out of scope for this blog post)
Build and push Docker images to push the resulting image to Docker Hub
Akka Serverless CLI for GitHub Actions to deploy to Akka Serverless

To make all the actions work, and to make it a little more secure, it’s best to set up usernames, passwords, and authentication tokens as secrets in GitHub. Under Settings -> Secrets I’ve created four environment variables that have been encrypted:

DOCKERHUB_TOKEN: the personal authentication token to log in to Docker Hub (while you could use a password too, it’s strongly recommended to use a personal access token)
DOCKERHUB_USERNAME: the username to log in to Docker Hub
PROJECT: the ID of the project I want to deploy to on Akka Serverless (akkasls projects get <project name> will show the ID of the project in the ID column)
TOKEN: the Akka Serverless authentication token that the action will use to connect to Akka Serverless (to create a token with “execution” scope, the command I need to run is akkasls auth tokens create --type=refresh --scopes=execution --description="My CI/CD token")

The YAML

After setting up the four secrets, the only thing left is to create the GitHub Actions YAML configuration. I’ll start with the complete file first, and break it into sections to explain what everything does. The file should be stored in the .github/workflows folder of the repository.

The full file

In my case, I’ve named my file deploy.yaml and stored it in the .github/workflows folder of my repository.

name: akkasls-deployer

on:
  push:
    branches: [main]
  workflow_dispatch:

jobs:
  build:
    runs-on: ubuntu-latest

    steps:
      - name: Checkout
        uses: actions/checkout@v2
      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v1
      - name: Login to Docker Hub
        uses: docker/login-action@v1
        with:
          username: ${{ secrets.DOCKERHUB_USERNAME }}
          password: ${{ secrets.DOCKERHUB_TOKEN }}
      - name: Build and push to Docker Hub
        uses: docker/build-push-action@v2
        with:
          push: true
          tags: retgits/myapp:1.0.0
      - name: Deploy to Akka Serverless
        uses: retgits/akkasls-action@v1
        with:
          cmd: "services deploy myapp retgits/myapp:1.0.0"
        env:
          token: ${{ secrets.TOKEN }}
          project: ${{ secrets.PROJECT }}

Breaking up the file in parts

Now that you can see the entire file, I’ll break it up into smaller sections to explain what they do.

name: akkasls-deployer

This basically determines the name of the workflow and this is how it shows up in the Actions section of the repository

on:
  push:
    branches: [main]
  workflow_dispatch:

The on section of a workflow describes the triggers, or when the workflow is executed. In this case, the workflow is executed every time there is a push to the main branch or when it is triggered from the workflow (which is done by the workflow_dispatch setting and is useful if you want to trigger the workflow without pushing to the main branch).

jobs:
  build:
    runs-on: ubuntu-latest

I run all my builds on Ubuntu, though you could opt to use different operating systems, or even use a self-hosted runner.

steps:
  - name: Checkout
    uses: actions/checkout@v2

This step checks out the code, so the other steps in this workflow can use it

  - name: Set up Docker Buildx
    uses: docker/setup-buildx-action@v1

This step sets up Buildx so I can build different images for different operating systems if I wanted (which is especially useful if you want to build containers that run on Apple Silicon chips)

  - name: Login to Docker Hub
    uses: docker/login-action@v1
    with:
      username: ${{ secrets.DOCKERHUB_USERNAME }}
      password: ${{ secrets.DOCKERHUB_TOKEN }}

As the name implies, this step logs in to Docker Hub so the resulting container image can be pushed there. It leverages two of the Secrets created earlier.

  - name: Build and push to Docker Hub
    uses: docker/build-push-action@v2
    with:
      push: true
      tags: retgits/myapp:1.0.0

This step builds the container and pushes it to Docker Hub, calling it retgits/myapp:1.0.0. This is obviously where you want to use some parameterization and use the name of the tag, the GitHub commit SHA, or something else to uniquely identify the container image.

  - name: Deploy to Akka Serverless
    uses: retgits/akkasls-action@v1
    with:
      cmd: "services deploy myapp retgits/myapp:1.0.0"
    env:
      token: ${{ secrets.TOKEN }}
      project: ${{ secrets.PROJECT }}

The last step in the process is to deploy the container image to Akka Serverless. This step leverages the other two Secrets created earlier, and it uses the same tag as in the previous step.

What’s next?

As you can see, using CI/CD to deploy a service to Akka Serverless is pretty straight-forward and setting it up once saves a lot of hassle down the line. Let me know what you want to see next, or what you want us to build next!

Cover photo by Gerd Altmann from Pixabay

How To Build Infrastructure as Code With Pulumi And Golang - Part 2

Leon Stigter — Thu, 06 Feb 2020 00:00:00 +0000

Going into the series on creating Infrastructure as Code on AWS using Pulumi, I knew the team there was actively working on improving and expanding the Go support in Pulumi. What I didn’t realize is that it would be so quick and would be such a great improvement to the underlying code I needed to write. In this post, I’ll go over some of the code from my previous blog posts and update them to match the new SDK.

The complete project is available on GitHub.

The core programming model in Pulumi has tons of really awesome features, but not all of them are available in the Go SDK yet. The work the team at Pulumi is currently doing, includes some significant changes to the SDK, including more strongly typed structs (no more interface{}, yay! 🥳), better support for Input and Output types, and making sure the Go has the same features from the core programming model as the other languages do. With so many changes going in, it’s almost impossible to not make breaking changes. Pulumi suggests that Go devs pin the versions of the SDK they’re currently using.

TL;DR I had to make a bunch of changes to make sure all the code worked again (like adding pulumi.String() wrappers around strings) but I could also replace all my own structs with structs from the Pulumi SDK. There are a few rough edges, but then again it is a work in progress and I’m excited to see what’s next.

Configuration

One of the new features that were added, or perhaps I just didn’t realize it existed before, is the ability to read and parse configuration from the YAML file. Using that configuration you can parameterize your stack based on config from a file, rather than hard coding it.

In the previous version of the code, I wrote my own function to get the configuration variables from the context:

// getEnv searches for the requested key in the pulumi context and provides either the value of the key or the fallback.
func getEnv(ctx *pulumi.Context, key string, fallback string) string {
    if value, ok := ctx.GetConfig(key); ok {
        return value
    }
    return fallback
}

That piece of the code, and a lot of related helper functions, can now be removed. In the snippet below, I’ve created a VPCConfig struct which, through the RequireObject() method, is populated with data from the Pulumi YAML file. The nice thing about this approach is that the RequireObject() method means that the object with that name needs to exist, otherwise Pulumi will throw an error and you can model your structs (and configuration data) to actually match the types they should be. Rather than having comma separated lists, I now have proper YAML arrays.

// VPCConfig is a strongly typed struct that can be populated with
// contents from the YAML file. These are the configuration items
// to create a VPC.
type VPCConfig struct {
    CIDRBlock string `json:"cidr-block"`
    Name string
    SubnetIPs []string `json:"subnet-ips"`
    SubnetZones []string `json:"subnet-zones"`
}

func main() {
    pulumi.Run(func(ctx *pulumi.Context) error {
        // Create a new config object with the data from the YAML file
        // The object has all the data that the namespace awsconfig has
        conf := config.New(ctx, "awsconfig")

        var vpcConfig VPCConfig
        conf.RequireObject("vpc", &vpcConfig)

        vpcArgs := &ec2.VpcArgs{
            CidrBlock: pulumi.String(vpcConfig.CIDRBlock),
            Tags: pulumi.Map(tagMap),
        }

        ... // snip
    })
}

These changes in the code also mean that the configuration file itself looks a lot cleaner. The snippet below shows what it was with the previous version of my code.

vpc:name: myPulumiVPC
vpc:cidr-block: "172.32.0.0/16"
vpc:subnet-zones: "us-east-1a,us-east-1c"
vpc:subnet-ips: "172.32.32.0/20,172.32.80.0/20"

Which is now replaced by better looking, and easier to read, YAML

awsconfig:vpc:
    cidr-block: 172.32.0.0/16
    name: myPulumiVPC
    subnet-ips:
        - 172.32.32.0/20
        - 172.32.80.0/20
    subnet-zones:
        - us-east-1a
        - us-east-1c

There are perhaps a few more lines, but the benefits of seeing which configuration items are actually arrays is super useful. The only thing that struck me as a little odd, is that you write the config in a YAML file and the struct tags use JSON.

Type Safety

One of the reasons I like Go is that it is a strongly typed language. That means that the type of variable is specified when you’re building your app. In the previous version of my code, I created my own structs to help add some strongly typed structs to create a DynamoDB table. The code below is a snippet of that code that shows the “old” way, where I had my own struct.

// DynamoAttribute represents an attribute for describing the key schema for the table and indexes.
type DynamoAttribute struct {
    Name string
    Type string
}

// DynamoAttributes is an array of DynamoAttribute
type DynamoAttributes []DynamoAttribute

// ToList takes a DynamoAttributes object and turns that into a slice of map[string]interface{} so it can be correctly passed to the Pulumi runtime
func (d DynamoAttributes) ToList() []map[string]interface{} {
    array := make([]map[string]interface{}, len(d))
    for idx, attr := range d {
        m := make(map[string]interface{})
        m["name"] = attr.Name
        m["type"] = attr.Type
        array[idx] = m
    }
    return array
}

// Create the attributes for ID and User
dynamoAttributes := DynamoAttributes{
    DynamoAttribute{
        Name: "ID",
        Type: "S",
    },
    DynamoAttribute{
        Name: "User",
        Type: "S",
    },
}

The updates to the Pulumi SDK introduced a ton of strongly typed structs that remove the need to have those helper structs and methods. In my case, for the DynamoDB table, there are now structs that support the Table Attributes and Global Secondary Indices. Removing the helper methods and replacing that with structs from the Pulumi SDK meant I went down from roughly 33 lines of code to 10 lines. I’m quite sure that feature alone will make sure I write better and cleaner code. The code snippet below shows the same attributes and does the same thing as the above snippet did.

// Create the attributes for ID and User
dynamoAttributes := []dynamodb.TableAttributeInput{
    dynamodb.TableAttributeArgs{
        Name: pulumi.String("ID"),
        Type: pulumi.String("S"),
    }, dynamodb.TableAttributeArgs{
        Name: pulumi.String("User"),
        Type: pulumi.String("S"),
    },
}

If you want to deploy your AWS Lambda functions through Pulumi, the same benefits I talked about are absolutely true for Lambda as well. More strongly typed structs with clear field definitions means that writing code is a lot easier and cleaner. Because you don’t have to rely on a bunch of helper methods, it also makes the code simpler to understand.

Going forward

Does all of this mean it’s all roses, though, the engineers at Pulumi still have some things to do. Credit to them, they do all of that out in the open. For me personally, I was very excited to see how far the team at Pulumi has come over the past few weeks. While there are a few rough edges, like documentation for the inputs and outputs of certain methods, I’m sure the team will update those in new releases, together with the help of the amazing community. I for one am excited to see how Pulumi takes Infrastructure as Code to a new level for Go developers.

How To Create AWS Lambda Functions Using Pulumi And Golang

Leon Stigter — Tue, 28 Jan 2020 00:00:00 +0000

I’ve looked at Pulumi to do a bunch of things, including creating subnets in a VPC, building EKS clusters, and DynamoDB tables. The one thing I hadn’t explored yet was how to deploy AWS Lambda functions using Pulumi, so that’s exactly what this blog is about.

The complete project is available on GitHub.

My Lambda

The Lambda function for this post is a rather simple one, it says “Hello” to whatever the value of an environment variable will be.

package main

import (
    "fmt"
    "os"

    "github.com/aws/aws-lambda-go/events"
    "github.com/aws/aws-lambda-go/lambda"
)

func handler(request events.APIGatewayProxyRequest) (events.APIGatewayProxyResponse, error) {
    val := os.Getenv("NAME")
    return events.APIGatewayProxyResponse{
        Body: fmt.Sprintf("Hello, %s", val),
        StatusCode: 200,
    }, nil
}

func main() {
    lambda.Start(handler)
}

That code is in a file called hello-world.go , which is in a folder called hello-world. The Pulumi project is inside the folder called Pulumi so it will be a separate folder structure and not collide with the other code of your Lambda functions. The folder structure, including all the files looks like

├── README.md
├── go.mod
├── go.sum
└── hello-world
│ └── main.go
├── pulumi
│ ├── Pulumi.lambdastack.yaml
│ ├── Pulumi.yaml
    └── main.go

Building and uploading your Lambda code

In order to deploy a Lambda function, the code needs to be uploaded as well. While Pulumi has the concept of an Archive to create the zip file, the Go implementation has a known issue which makes it impossible to use that feature. Rather than manually building, zipping, and uploading the code, you can extend the Pulumi program a little to do all of that before the program runs.

const (
    shell = "sh"
    shellFlag = "-c"
    rootFolder = "/rootfolder/of/your/lambdaapp"
)

func runCmd(args string) error {
    cmd := exec.Command(shell, shellFlag, args)
    cmd.Stdout = os.Stdout
    cmd.Stderr = os.Stderr
    cmd.Dir = rootFolder
    return cmd.Run()
}

The runCmd method will run a specific command and will return either an error or a nil object. To build, zip, and upload your Go code to S3 you can add the below snippets to your Pulumi project. It uses the above function to run the three commands you would normally script as part of your CI/CD or test framework. These commands should be pasted before pulumi.Run() gets invoked.

if err := runCmd("GOOS=linux GOARCH=amd64 go build -o hello-world/hello-world ./hello-world"); err != nil {
    fmt.Printf("Error building code: %s", err.Error())
    os.Exit(1)
}

if err := runCmd("zip -r -j ./hello-world/hello-world.zip ./hello-world/hello-world"); err != nil {
    fmt.Printf("Error creating zipfile: %s", err.Error())
    os.Exit(1)
}

if err := runCmd("aws s3 cp ./hello-world/hello-world.zip s3://<your-bucket>/hello-world.zip"); err != nil {
    fmt.Printf("Error creating zipfile: %s", err.Error())
    os.Exit(1)
}

If these commands fail, you’ll be able to see the output and the error message as part of the “diagnostics” section in your terminal.

Creating an IAM role

Every Lambda function, and most other AWS resources as well, need an IAM role to be able to work. The IAM role gives the resources the permissions to act on your behalf. This Lambda function doesn’t have a lot of specifics it needs to do, other than be able to run. To create an IAM role with that permission, you can use the below code. The ARN (Amazon Resource Name), is exported so that it is visible from within the Pulumi console.

// The policy description of the IAM role, in this case only the sts:AssumeRole is needed
roleArgs := &iam.RoleArgs{
    AssumeRolePolicy: `{
        "Version": "2012-10-17",
        "Statement": [
        {
            "Action": "sts:AssumeRole",
            "Principal": {
                "Service": "lambda.amazonaws.com"
            },
            "Effect": "Allow",
            "Sid": ""
        }
        ]
    }`,
}

// Create a new role called HelloWorldIAMRole
role, err := iam.NewRole(ctx, "HelloWorldIAMRole", roleArgs)
if err != nil {
    fmt.Printf("role error: %s\n", err.Error())
    return err
}

// Export the role ARN as an output of the Pulumi stack
ctx.Export("Role ARN", role.Arn())

Setting environment variables

Just like in CloudFormation, the Pulumi SDK allows you to create a set of environment variables. That’s a good thing, because the Lambda function relies on an environment variable called “NAME” to know who to greet. The environment variables are a map[string]interface{} inside a map[string]interface{}. For this Lambda function, that would be the below snippet.

environment := make(map[string]interface{})
variables := make(map[string]interface{})
variables["NAME"] = "WORLD"
environment["variables"] = variables

Creating the function

The last step. at least for this code, is to actually create the Lambda function. The S3Bucket and S3Key are the name of the bucket and the name of the file you’ve uploaded earlier in the process. The role.Arn() is the ARN of the role that was created in the previous step.

// The set of arguments for constructing a Function resource.
functionArgs := &lambda.FunctionArgs{
    Description: "My Lambda function",
    Runtime: "go1.x",
    Name: "HelloWorldFunction",
    MemorySize: 256,
    Timeout: 10,
    Handler: "hello-world",
    Environment: environment,
    S3Bucket: "<your-bucket>",
    S3Key: "hello-world.zip",
    Role: role.Arn(),
}

// NewFunction registers a new resource with the given unique name, arguments, and options.
function, err := lambda.NewFunction(ctx, "HelloWorldFunction", functionArgs)
if err != nil {
    fmt.Println(err.Error())
    return err
}

// Export the function ARN as an output of the Pulumi stack
ctx.Export("Function", function.Arn())

Running Pulumi up

With all the code ready, it’s time to run pulumi up and deploy the Lambda function to AWS. If you need more details on how to create a Go project for Pulumi, check out this post.

$ pulumi up
Previewing update (lambda):

     Type Name Plan Info
 + pulumi:pulumi:Stack lambda-lambda create 2 messages
 + ├─ aws:iam:Role HelloWorldIAMRole create     
 + └─ aws:lambda:Function HelloWorldFunction create     

Diagnostics:
  pulumi:pulumi:Stack (lambda-lambda):
    updating: hello-world/hello-world (deflated 49%)
upload: hello-world/hello-world.zip to s3://<your-bucket>/hello-world.zip

Resources:
    + 3 to create

Do you want to perform this update? yes
Updating (lambda):

     Type Name Status Info
 + pulumi:pulumi:Stack lambda-lambda created 2 messages
 + ├─ aws:iam:Role HelloWorldIAMRole created     
 + └─ aws:lambda:Function HelloWorldFunction created     

Diagnostics:
  pulumi:pulumi:Stack (lambda-lambda):
    updating: hello-world/hello-world (deflated 49%)
upload: hello-world/hello-world.zip to s3://<your-bucket>/hello-world.zip

Outputs:
    Function: "arn:aws:lambda:us-west-2:ACCOUNTID:function:HelloWorldFunction"
    Role ARN: "arn:aws:iam::ACCOUNTID:role/HelloWorldIAMRole-7532034"

Resources:
    + 3 created

Duration: 44s

Permalink: https://app.pulumi.com/retgits/lambda/lambda/updates/1

Testing with the AWS Console

Inside the Pulumi console, you’ll be able to see the resources that have been created.

The Pulumi console also has really useful links to the AWS console to see the resources.

Within the AWS Lambda console, you’ll be able to test the function and see that it indeed responds with the greeting “Hello, WORLD”.

The AWS Lambda console.

How To Create a DynamoDB Table In AWS Using Pulumi And Golang

Leon Stigter — Tue, 21 Jan 2020 00:00:00 +0000

In previous posts, I looked at Pulumi to do all sorts of things with infrastructure. Most apps, though, will need some form of datastore so in this post I’ll go over the steps to create a DynamoDB table in AWS using Pulumi.

The complete project is available on GitHub.

Static types

One of the main advantages of programming languages like Golang and Java is that those languages have static types that make development less error prone. As developers are writing the code, they know what the input and output of methods will be. Unfortunately, the Go SDK for Pulumi doesn’t yet offer static types for AWS resources. The snippet of code below has two types (DynamoAttribute and GlobalSecondaryIndex) that represent the static types of the DynamoDB constructs.

// DynamoAttribute represents an attribute for describing the key schema for the table and indexes.
type DynamoAttribute struct {
    Name string
    Type string
}

// DynamoAttributes is an array of DynamoAttribute
type DynamoAttributes []DynamoAttribute

// GlobalSecondaryIndex represents the properties of a global secondary index
type GlobalSecondaryIndex struct {
    Name string
    HashKey string
    ProjectionType string
    WriteCapacity int
    ReadCapacity int
}

// GlobalSecondaryIndexes is an array of GlobalSecondaryIndex
type GlobalSecondaryIndexes []GlobalSecondaryIndex

The input to create a DynamoDB table, the tableArgs, expects a interface{} for these two fields and the underlying infrastructure expects that interface{} to be a list (or a slice of interfaces to use the correct Go terminology). To make that that type conversion happen, you can use the below two ToList() methods for the types above.

// ToList takes a DynamoAttributes object and turns that into a slice of map[string]interface{} so it can be correctly passed to the Pulumi runtime
func (d DynamoAttributes) ToList() []map[string]interface{} {
    array := make([]map[string]interface{}, len(d))
    for idx, attr := range d {
        m := make(map[string]interface{})
        m["name"] = attr.Name
        m["type"] = attr.Type
        array[idx] = m
    }
    return array
}

// ToList takes a GlobalSecondaryIndexes object and turns that into a slice of map[string]interface{} so it can be correctly passed to the Pulumi runtime
func (g GlobalSecondaryIndexes) ToList() []map[string]interface{} {
    array := make([]map[string]interface{}, len(g))
    for idx, attr := range g {
        m := make(map[string]interface{})
        m["name"] = attr.Name
        m["hash_key"] = attr.HashKey
        m["projection_type"] = attr.ProjectionType
        m["write_capacity"] = attr.WriteCapacity
        m["read_capacity"] = attr.ReadCapacity
        array[idx] = m
    }
    return array
}

The above two methods make it easier to use static typed objects in your code, while still being able to use the Pulumi runtime to create your DynamoDB tables.

Building a table

The next step is to bring all of that together and create the table. In this sample I’ve used a table with usernames and unique IDs, that I use in one of my apps to keep track of order data. Since the actual order data isn’t modeled here, you won’t be able to use it in your queries.

// Create the attributes for ID and User
dynamoAttributes := DynamoAttributes{
    DynamoAttribute{
        Name: "ID",
        Type: "S",
    },
    DynamoAttribute{
        Name: "User",
        Type: "S",
    },
}

// Create a Global Secondary Index for the user field
gsi := GlobalSecondaryIndexes{
    GlobalSecondaryIndex{
        Name: "User",
        HashKey: "User",
        ProjectionType: "ALL",
        WriteCapacity: 10,
        ReadCapacity: 10,
    },
}

// Create a TableArgs struct that contains all the data
tableArgs := &dynamodb.TableArgs{
    Attributes: dynamoAttributes.ToList(),
    HashKey: "ID",
    WriteCapacity: 10,
    ReadCapacity: 10,
    GlobalSecondaryIndexes: gsi.ToList(),
}

// Let the Pulumi runtime create the table
userTable, err := dynamodb.NewTable(ctx, "User", tableArgs)
if err != nil {
    return err
}

// Export the name of the newly created table as an output in the stack
ctx.Export("TableName", userTable.ID())

Complete code

Combining all of the above into a single, runnable, Go program

package main

import (
    "github.com/pulumi/pulumi-aws/sdk/go/aws/dynamodb"
    "github.com/pulumi/pulumi/sdk/go/pulumi"
)

// DynamoAttribute represents an attribute for describing the key schema for the table and indexes.
type DynamoAttribute struct {
    Name string
    Type string
}

// DynamoAttributes is an array of DynamoAttribute
type DynamoAttributes []DynamoAttribute

// ToList takes a DynamoAttributes object and turns that into a slice of map[string]interface{} so it can be correctly passed to the Pulumi runtime
func (d DynamoAttributes) ToList() []map[string]interface{} {
    array := make([]map[string]interface{}, len(d))
    for idx, attr := range d {
        m := make(map[string]interface{})
        m["name"] = attr.Name
        m["type"] = attr.Type
        array[idx] = m
    }
    return array
}

// GlobalSecondaryIndex represents the properties of a global secondary index
type GlobalSecondaryIndex struct {
    Name string
    HashKey string
    ProjectionType string
    WriteCapacity int
    ReadCapacity int
}

// GlobalSecondaryIndexes is an array of GlobalSecondaryIndex
type GlobalSecondaryIndexes []GlobalSecondaryIndex

// ToList takes a GlobalSecondaryIndexes object and turns that into a slice of map[string]interface{} so it can be correctly passed to the Pulumi runtime
func (g GlobalSecondaryIndexes) ToList() []map[string]interface{} {
    array := make([]map[string]interface{}, len(g))
    for idx, attr := range g {
        m := make(map[string]interface{})
        m["name"] = attr.Name
        m["hash_key"] = attr.HashKey
        m["projection_type"] = attr.ProjectionType
        m["write_capacity"] = attr.WriteCapacity
        m["read_capacity"] = attr.ReadCapacity
        array[idx] = m
    }
    return array
}

func main() {
    pulumi.Run(func(ctx *pulumi.Context) error {
        // Create the attributes for ID and User
        dynamoAttributes := DynamoAttributes{
            DynamoAttribute{
                Name: "ID",
                Type: "S",
            },
            DynamoAttribute{
                Name: "User",
                Type: "S",
            },
        }

        // Create a Global Secondary Index for the user field
        gsi := GlobalSecondaryIndexes{
            GlobalSecondaryIndex{
                Name: "User",
                HashKey: "User",
                ProjectionType: "ALL",
                WriteCapacity: 10,
                ReadCapacity: 10,
            },
        }

        // Create a TableArgs struct that contains all the data
        tableArgs := &dynamodb.TableArgs{
            Attributes: dynamoAttributes.ToList(),
            HashKey: "ID",
            WriteCapacity: 10,
            ReadCapacity: 10,
            GlobalSecondaryIndexes: gsi.ToList(),
        }

        // Let the Pulumi runtime create the table
        userTable, err := dynamodb.NewTable(ctx, "User", tableArgs)
        if err != nil {
            return err
        }

        // Export the name of the newly created table as an output in the stack
        ctx.Export("TableName", userTable.ID())
    })
}

Pulumi up

The last step is to add all of this to a new Pulumi project and run the pulumi up command. To create a new, Go based, project, you can run the command

pulumi new go \
--name builder \
--description "An awesome Pulumi infrastructure-as-code Stack" \
--stack retgits/builderstack

Now you can replace the code from the Go file with the code above and run pulumi up. For a little more in-depth info on creating a new Pulumi project, check out one of my previous posts.

How To Make Your AWS EKS Cluster Use Fargate Using Pulumi And Golang

Leon Stigter — Thu, 16 Jan 2020 00:00:00 +0000

At re:Invent, AWS introduced the ability to have EKS run pods on AWS Fargate, and Fargate is cheaper than hosting Kubernetes yourself. In the last post I created an EKS cluster, so let’s add this new capability to the cluster and remove the need to manage or provision infrastructure for our pods.

The complete project is available on GitHub.

Configuration

The minimum configuration for the Fargate profile is a name, the Kubenetes namespace it’ll work in, and the IAM role it needs to run the pods on AWS Fargate. The configuration below, which you can copy/paste into the YAML file from the previous blog, has three parameters. The parameter fargate:profile-name is the name of the Fargate profile, the parameter fargate:namespace is the Kubernetes namespace, and fargate:execution-role-arn is the ARN of the IAM role. For more details on how to create the role, check out the AWS docs.

fargate:profile-name: EKSFargateProfile
fargate:namespace: example
fargate:execution-role-arn: "arn:aws:iam::ACCOUNTID:role/EKSFargatePodExecutionRole"

You can either use the command line, like pulumi config set fargate:profile-name "EKSFargateProfile" to add these new configuration variables, or you can add them directly into the yaml file. The yaml file with all the configuration is called Pulumi.<name of your project>.yaml.

Adding the Fargate profile

The code below is an extension from the code created in the previous post. So you can copy/paste this snippet into your Go code too. Walking through the code, it gets the name of the profile and the namespace from the YAML file. The fargateProfileArgs the name of the cluster and subnets from previous blog posts so check those out if you haven’t already. The call to eks.NewFargateProfile() adds the Fargate profile to your EKS cluster.

// Create an EKS Fargate Profile
fargateProfileName := getEnv(ctx, "fargate:profile-name", "unknown")

selectors := make([]map[string]interface{}, 1)
namespaces := make(map[string]interface{})
namespaces["namespace"] = getEnv(ctx, "fargate:namespace", "unknown")
selectors[0] = namespaces

fargateProfileArgs := &eks.FargateProfileArgs{
    ClusterName: clusterName,
    FargateProfileName: fargateProfileName,
    Tags: tags,
    SubnetIds: subnets["subnet_ids"],
    Selectors: selectors,
    PodExecutionRoleArn: getEnv(ctx, "fargate:execution-role-arn", "unknown"),
}

fargateProfile, err := eks.NewFargateProfile(ctx, fargateProfileName, fargateProfileArgs)
if err != nil {
    fmt.Println(err.Error())
    return err
}

ctx.Export("FARGATE-PROFILE-ID", fargateProfile.ID())

Running the code

Like the previous time, the last thing to do is run pulumi up to tell Pulumi to go add the Fargate profile to your EKS cluster! If you’re using the same project and stack, Pulumi will automatically realize it needs to add the profile to the existing cluster and won’t create a new EKS cluster.

$ pulumi up
Previewing update (builderstack):

     Type Name Plan       
     pulumi:pulumi:Stack builder-builderstack             
 + └─ aws:eks:FargateProfile EKSFargateProfile create     

Outputs:
  + FARGATE-PROFILE-ID: output<string>

Resources:
    + 1 to create
    5 unchanged

Do you want to perform this update? yes
Updating (builderstack):

     Type Name Status      
     pulumi:pulumi:Stack builder-builderstack              
 + └─ aws:eks:FargateProfile EKSFargateProfile created     

Outputs:
    CLUSTER-ID : "myEKSCluster"
  + FARGATE-PROFILE-ID: "myEKSCluster:EKSFargateProfile"
    SUBNET-IDS : [
        [0]: "subnet-0a1909bec2e936bd7"
        [1]: "subnet-09d229c2eb8061979"
    ]
    VPC-ID : "vpc-0437c750acf1050c3"

Resources:
    + 1 created
    5 unchanged

Duration: 2m27s

Permalink: https://app.pulumi.com/retgits/builder/builderstack/updates/4

The permalink at the bottom of the output takes you to the Pulumi console where you can see all the details of the execution of your app and the resources that were created.

The Pulumi console also has really useful links to the AWS console to see the resources.

How To Create An AWS EKS Cluster Using Pulumi And Golang

Leon Stigter — Tue, 14 Jan 2020 00:00:00 +0000

Building a Kubernetes cluster is hard, so why not use a managed Kubernetes service? In the previous post I added subnets to a VPC. In this post, I’ll use the VPC to create an AWS EKS cluster.

The complete project is available on GitHub.

Configuration

The minimum configuration you need to create a cluster are a name for the cluster, the Kubernetes version you want to use, and an IAM role your cluster will use. While optional, specifying the types of logs your cluster will send to CloudWatch can be very useful when debugging and troubleshooting. The configuration below, which you can copy/paste into the YAML file from the previous blog, has four parameters. The parameter eks:cluster-name is the name of the cluster. The parameter eks:k8s-version is the version of Kubernetes to use (at the time of writing 1.14 was the latest available version). The parameter eks:cluster-role-arn has the ARN of the IAM role your cluster needs to perform tasks. For more details on how to create the role, check out the AWS docs. Lastly, the parameter eks:cluster-log-types is a comma separated list of all the components that can emit logs.

eks:cluster-name: myEKSCluster
eks:k8s-version: "1.14"
eks:cluster-role-arn: "arn:aws:iam::ACCOUNTID:role/ServiceRoleForAmazonEKS"
eks:cluster-log-types: "api,audit,authenticator,scheduler,controllerManager"

You can either use the command line, like pulumi config set eks:cluster-name "myEKSCluster" to add these new configuration variables, or you can add them directly into the yaml file. The yaml file with all the configuration is called Pulumi.<name of your project>.yaml.

Creating the cluster

The code below is an extension from the code created in the previous post. So you can copy/paste this snippet into your Go code too. Walking through the code, it gets the configuration for the cluster name and enabled log types from the YAML file. It also uses the subnets created in the previous post. The call to eks.NewCluster() will tell Pulumi to go create the EKS cluster in the VPC you have, using the subnets you already have created too.

// Create an EKS cluster
clusterName := getEnv(ctx, "eks:cluster-name", "unknown")
enabledClusterLogTypes := strings.Split(getEnv(ctx, "eks:cluster-log-types", "unknown"), ",")

clusterArgs := &eks.ClusterArgs{
    Name: clusterName,
    Version: getEnv(ctx, "eks:k8s-version", "unknown"),
    RoleArn: getEnv(ctx, "eks:cluster-role-arn", "unknown"),
    Tags: tags,
    VpcConfig: subnets,
    EnabledClusterLogTypes: enabledClusterLogTypes,
}

cluster, err := eks.NewCluster(ctx, clusterName, clusterArgs)
if err != nil {
    fmt.Println(err.Error())
    return err
}

ctx.Export("CLUSTER-ID", cluster.ID())

Running the code

Like the previous time, the last thing to do is run pulumi up to tell Pulumi to go create an EKS cluster inside your VPC! If you’re using the same project and stack, Pulumi will automatically realize it needs to create the cluster inside the existing VPC and won’t create a new VPC. Running this code might take a little while for AWS to complete creating the cluster (in my case it took almost 10 minutes).

$ pulumi up
Previewing update (builderstack):

     Type Name Plan       
     pulumi:pulumi:Stack builder-builderstack             
 + └─ aws:eks:Cluster myEKSCluster create     

Outputs:
  + CLUSTER-ID: output<string>

Resources:
    + 1 to create
    4 unchanged

Do you want to perform this update? yes
Updating (builderstack):

     Type Name Status      
     pulumi:pulumi:Stack builder-builderstack              
 + └─ aws:eks:Cluster myEKSCluster created     

Outputs:
  + CLUSTER-ID: "myEKSCluster"
    SUBNET-IDS: [
        [0]: "subnet-<id>"
        [1]: "subnet-<id>"
    ]
    VPC-ID : "vpc-<id>"

Resources:
    + 1 created
    4 unchanged

Duration: 9m55s

Permalink: https://app.pulumi.com/retgits/builder/builderstack/updates/3

The permalink at the bottom of the output takes you to the Pulumi console where you can see all the details of the execution of your app and the resources that were created.

The Pulumi console also has really useful links to the AWS console to see the resources.

How To Add Subnets To a VPC In AWS Using Pulumi And Golang

Leon Stigter — Thu, 09 Jan 2020 00:00:00 +0000

In the previous post, I looked at Pulumi to create a VPC. In this post, I’ll go over the steps to add some subnets to it.

The complete project is available on GitHub.

Configuration

A subnet is a logical partition of your network, so in case of the subnets in a VPC, they’ll divide up the VPC into smaller chunks. A VPC spans all availability zones of a region, whereas a subnet is only inside a single availability zone. To make sure the app you run inside your VPC can move from one availability zone to another in case there is a failure, you’ll need to have at least two availability zones. Each of the zones also needs their own CIDR block. The configuration below, which you can copy/paste into the YAML file from the previous blog, has two parameters. The parameter vpc:subnet-zones has a comma separated list of all the availability zones in which a subnet needs to be created. The parameter vpc:subnet-ips has a comma separated list of the CIDR blocks for each of the subnets.

vpc:subnet-zones: "us-east-1a,us-east-1c"
vpc:subnet-ips: "172.32.32.0/20,172.32.80.0/20"

You can either use the command line, like pulumi config set vpc:subnet-zones "us-east-1a,us-east-1c" to add these new configuration variables, or you can add them directly into the yaml file. The yaml file with all the configuration is called Pulumi.<name of your project>.yaml.

Creating subnets

The code below is an extension from the code created in the previous post. So you can copy/paste this snippet into your Go code too. Walking through the code, it gets the configuration for zones (subnetZones) and CIDR blocks (subnetIPs) from the YAML file. The code reads the variable and splits it on the comma delimiter. For each of the zones it’ll create a new subnet inside the VPC (see subnetArgs) and give it a name based on the name of the VPC. The last step in the range loop is to add the new ID to an array of subnets to export to the Pulumi console.

// Create the required number of subnets
subnets := make(map[string]interface{})
subnets["subnet_ids"] = make([]interface{}, 0)

subnetZones := strings.Split(getEnv(ctx, "vpc:subnet-zones", "unknown"), ",")
subnetIPs := strings.Split(getEnv(ctx, "vpc:subnet-ips", "unknown"), ",")

for idx, availabilityZone := range subnetZones {
    subnetArgs := &ec2.SubnetArgs{
        Tags: tags,
        VpcId: vpc.ID(),
        CidrBlock: subnetIPs[idx],
        AvailabilityZone: availabilityZone,
    }

    subnet, err := ec2.NewSubnet(ctx, fmt.Sprintf("%s-subnet-%d", vpcName, idx), subnetArgs)
    if err != nil {
        fmt.Println(err.Error())
        return err
    }

    subnets["subnet_ids"] = append(subnets["subnet_ids"].([]interface{}), subnet.ID())
}

ctx.Export("SUBNET-IDS", subnets["subnet_ids"])

Running the code

Like the previous time, the last thing to do is run pulumi up to tell Pulumi to go add subnets to your VPC! If you’re using the same project and stack, Pulumi will automatically realize it needs to add subnets to the existing VPC and won’t create a new VPC.

$ pulumi up
Previewing update (builderstack):

     Type Name Plan       
     pulumi:pulumi:Stack builder-builderstack             
 + ├─ aws:ec2:Subnet myPulumiVPC-subnet-1 create     
 + └─ aws:ec2:Subnet myPulumiVPC-subnet-0 create     

Outputs:
  + SUBNET-IDS: [
  + [0]: output<string>
  + [1]: output<string>
    ]

Resources:
    + 2 to create
    2 unchanged

Do you want to perform this update? yes
Updating (builderstack):

     Type Name Status      
     pulumi:pulumi:Stack builder-builderstack              
 + ├─ aws:ec2:Subnet myPulumiVPC-subnet-1 created     
 + └─ aws:ec2:Subnet myPulumiVPC-subnet-0 created     

Outputs:
  + SUBNET-IDS: [
  + [0]: "subnet-<id>"
  + [1]: "subnet-<id>"
    ]
    VPC-ID : "vpc-<id>"

Resources:
    + 2 created
    2 unchanged

Duration: 8s

Permalink: https://app.pulumi.com/retgits/builder/builderstack/updates/2

The permalink at the bottom of the output takes you to the Pulumi console where you can see all the details of the execution of your app and the resources that were created.

The Pulumi console also has really useful links to the AWS console to see the resources.

How To Create a VPC In AWS Using Pulumi And Golang

Leon Stigter — Tue, 07 Jan 2020 00:00:00 +0000

The source code you write as a developer is important, but it is only one part of the entire application that goes into production. To deploy an app, you’ll need resources like API gateways, S3 buckets, or VPCs as well. Configuring those resources is a task you don’t want to do manually. How about building your infrastructure as code using the same language you’ve built your app in. That is what Pulumi allows you to do!

Most of the resources deployed to any of the major cloud providers need to be deployed inside a VPC. So let’s walk through the steps to create a VPC using the Pulumi Go SDK. The complete project is available on GitHub.

A new project

If you haven’t done so already, you might want to take a quick second and head over to the Pulumi website to create a new account. It’s free forever for personal use! Once you’ve done that and installed the CLI you’re ready to start. To get going, you’ll need to create a new Pulumi project. On the command line you can specify which template you want to use (go in this case). You’ll be able to specify the project name (with the --name flag), the description (with the --description flag), and the stack name (with the --stack flag).

pulumi new go \
--name builder \
--description "An awesome Pulumi infrastructure-as-code Stack" \
--stack retgits/builderstack

In the above example, I’ve used the project name builder and the stack is called retgits/builderstack.

Using Go modules

The current Go template still relies on dep for dependency management, but with the introduction of Go modules you probably want to use that to manage dependencies. Luckily, switching from dep to Go modules only requires three commands.

go mod init github.com/retgits/builder
go mod tidy
rm Gopkg.*

The first command creates a new Go module in the directory where you run the command. The module needs a name and in this example the name of the module is github.com/retgits/builder. The second command takes the input from Gopkg.toml and creates entries in go.mod for each entry. The third command removes the Gopkg files.

Default configuration variables

To ultimate deploy to AWS, Pulumi needs to know how to connect to AWS. To connect, Pulumi uses a provider (more on that later). The configuration for that provider can be set through the command line: pulumi config set aws:<option>.

pulumi config set aws:profile default
pulumi config set aws:region us-east-1

The example above, has the required parameter aws:region which tells Pulumi which region to deploy to (in this case us-east-1) and the optional parameter aws:profile which tells Pulumi the profile to use (in this case default). The profiles you can choose are the ones you created using aws configure.

Adding more configuration variables

As a best practice, AWS suggests that you add tags to each resource you create (at least the ones that support tags). Tags are simple key/value pairs that can make it easier to manage, search for, and filter resources. The tags that I use when I deploy code to AWS, and the ones that are in the sample are version (because infrastructure-as-code makes it easy to version your infrastructure too), author (so you can see who did the deployment), team (so you can see which team did a deployment), and feature (resources are generally part of a larger app so with this you can filter all resources for an app you’re looking for). If you want to change the tags, feel free to do so in the source code. To create a VPC, you’ll need two parameters. Firstly, the name of the VPC and secondly you’ll need to specify the CIDR block.

tags:version: "0.1.0"
tags:author: <your name>
tags:team: <your team>
tags:feature: myFirstVPCWithPulumi
vpc:name: myPulumiVPC
vpc:cidr-block: "172.32.0.0/16"

There are two ways you can add configuration variables to your Pulumi config. You can either use the command line, like pulumi config set tags:version "0.1.0", or you can add them directly into the yaml file. The yaml file with all the configuration is called Pulumi.<name of your project>.yaml.

Using configuration variables in your code

The configuration variables that you’ve added to the YAML file are accessible through the pulumi.Context object. You can use the GetConfig() method to search for your key and you’ll get a string and a boolean in return. The boolean tells whether the context could find the key you’re looking for. To make that process a little easier and provide the option to have default values, you can add a method like the one below.

// getEnv searches for the requested key in the pulumi context and provides either the value of the key or the fallback.
func getEnv(ctx *pulumi.Context, key string, fallback string) string {
    if value, ok := ctx.GetConfig(key); ok {
        return value
    }
    return fallback
}

Making it all work

Now that most of the code is ready, it’s time to create a VPC. In the code below you can see a map[string]interface{} called tags which is used to tag all the resources created with for this project. Pulumi creates the VPC by executing ec2.NewVpc(). As you likely want to know the ID of the newly created VPC, Pulumi makes it easy to take output from the AWS and export it as outputs so you can see them in the Pulumi console.

func main() {
    pulumi.Run(func(ctx *pulumi.Context) error {
        // Prepare the tags that are used for each individual resource so they can be found
        // using the Resource Groups service in the AWS Console
        tags := make(map[string]interface{})
        tags["version"] = getEnv(ctx, "tags:version", "unknown")
        tags["author"] = getEnv(ctx, "tags:author", "unknown")
        tags["team"] = getEnv(ctx, "tags:team", "unknown")
        tags["feature"] = getEnv(ctx, "tags:feature", "unknown")
        tags["region"] = getEnv(ctx, "aws:region", "unknown")

        // Create a VPC for the EKS cluster
        cidrBlock := getEnv(ctx, "vpc:cidr-block", "unknown")

        vpcArgs := &ec2.VpcArgs{
            CidrBlock: cidrBlock,
            Tags: tags,
        }

        vpcName := getEnv(ctx, "vpc:name", "unknown")

        vpc, err := ec2.NewVpc(ctx, vpcName, vpcArgs)
        if err != nil {
            fmt.Println(err.Error())
            return err
        }

        // Export IDs of the created resources to the Pulumi stack
        ctx.Export("VPC-ID", vpc.ID())
        return nil
    })
}

Running the code

While most other languages rely on the package manager of the language to install Pulumi plugins (also called providers), Go does not. In order to deploy to AWS, you’ll need to manually install the AWS provider. To install the latest version (as of writing this blog) of the AWS provider, you can run the below command.

pulumi plugin install resource aws v1.17.0

The last thing to do is run pulumi up to tell Pulumi to go create your VPC!

$ pulumi up
Previewing update (builderstack):

     Type Name Plan       
 + pulumi:pulumi:Stack builder-builderstack create     
 + └─ aws:ec2:Vpc myPulumiVPC create     

Resources:
    + 2 to create

Do you want to perform this update? yes
Updating (builderstack):

     Type Name Status      
 + pulumi:pulumi:Stack builder-builderstack created     
 + └─ aws:ec2:Vpc myPulumiVPC created     

Outputs:
    VPC-ID: "vpc-<id>"

Resources:
    + 2 created

Duration: 11s

Permalink: https://app.pulumi.com/retgits/builder/builderstack/updates/1

The permalink at the bottom of the output takes you to the Pulumi console where you can see all the details of the execution of your app and the resources that were created.

The Pulumi console also has really useful links to the AWS console to see the resources.