Forem: Jeff Adams

Github Actions: Steps in Containers

Jeff Adams — Tue, 03 Mar 2020 17:54:57 +0000

Build Pipelines and build agents using CI/CD products are not new. The latest generation of these have all moved to Docker. Docker creates a clean and flexible build environment that streamlines out installation steps and tool combination issues.

This created a new problem. Either you must make your own Docker build agent and install what you need, or you must move artifacts from build containers to deploy containers. Getting Node and the AWS cli on the same machine can be surprisingly complicated in a controlled environment.

Github actions has made a substantive improvement on this process. You can spin up a docker container with your codebase automatically mounted. the process results can be saved to the mounted directory and then the container goes away.

The initial example

We recently did a project that utilized Github Actions for continuous integration / Deployment. This project followed a typical Cloudformation deployment process using AWS CLI. Those steps are:

Checkout the code
Configure access to your AWS environment
- This requires Access Keys and sets them to the proper ENV variables -Deploy the stack -In our situation there was no code to build. That would have to happen before this deploy. And you would need AWS Sam or some variant.

Here is a basic workflow. Put the following code in .github/workflows in your repo and the action will run once that code is pushed.
on: push

name: Build on every push to the infrastructure

jobs:
  deploy_cloudformation:
    name: Deployment
    runs-on: ubuntu-latest

    steps:
    - name: Checkout
      uses: actions/checkout@v1

    - name: Configure AWS Credentials
      uses: aws-actions/configure-aws-credentials@v1
      with:
        aws-access-key-id: <Access Key: Use the secrets>
        aws-secret-access-key: <Secret Access Key: Use the secrets>
        aws-region: <Your Region: Use the secrets>

    - name: Deploy Base Stack
      run: aws cloudformation deploy \
        --no-fail-on-empty-changeset \
        --template-file cloudformation.yaml \
        --stack-name < STACK_NAME > \
        --parameter-overrides file://parameters-$(ENV).json

The ubuntu-latest environment gives you a VM that comes with the following installed:

AWS CLI
Node
Go
Java / Maven
So very many more, it's quite impressive

This build agent software list is impressively easy to use and a quick way to start, but not all that different from many competitors. The thing that sets Github Actions apart is an ability to use Docker within a single build step. Let's focus on an example.

Language Validation with PHPStan

Our use case was to Lint a PHP repository. This is possible using ubuntu and installing tools, but let's encapsulate it and keep PHPStan out of our Vendor directory for our delivered artifact.

on: push

name: Static Code Analysis

jobs:
  analyze:
    name: Static Analysis with PHPStan
    runs-on: ubuntu-latest
    steps:
    - name: Checkout
      uses: actions/checkout@v1
    - name: Analysis
      uses: docker://phpstan/phpstan
      with:
        args: analyze --error-format=table <code path>

Let's break this down:

We pull a public docker container from Docker hub
We execute the container passing the args as the command.
Github actions will attach our workspace to the Workdir of the container and run the args command we passed
- That means we can save the output and use it as an artifact
- That means the installed dev tools won't become part of the pipeline artifact.

This is pretty similar to provided Actions. We could also codify this into a callable action and register it either in our own repo or in a separately registered repo. The difference is this can call any container, and we don't have to create a repository or separate part of the codebase for actions.

Running an arbitrary Docker container as a step

Docker containers as agent runners of a single build step is related to actions, but requires less overhead. This is the equivalent of an anonymous function in your CI/CD while abstracting the tool dependencies out of your pipeline execution.

Taking this idea further

Internal Tools and Secrets Abstraction

If you have internal containers with internal tools. You can host them on an Authenticated Docker Registry and add them to your pipeline. You can avoid putting code that might reveal secrets into your pipeline and instead call the container.

Compliance

If you need code to be submitted internally and allow it to be updated by a separate team. You could also set permissions to obscure the container from the dev team altogether. They don't have to bother with lots of code in the pipeline, and you get to manage your own compliance process.

And finally a Recap

Build pipelines and build agents with Docker are not new. But a specific step being run in a container and then cleaned so you can perform discrete steps with discrete tools is powerful. This is one tool in the CI/CD Quiver, but I'm excited about the abstraction possibilities. And cleanliness it provides.

Concurrency comparison Javascript to GO

Jeff Adams — Wed, 18 Dec 2019 15:58:35 +0000

A lot has been written on concurrency and it is a deep subject. In order to understand the subtlety of how concurrency is implemented in Go and Javascript respectively, consider this concept through this high-level, relatively simple example.

For deeper understanding of how coroutines, channels, and generators work I will include some links at the bottom.

Example

Simple Async / Await for a http request. The goal is to wait until we have the data, but not block the main execution context while we wait.

Javascript

const axios = require('axios');

(async function() {
    console.log('here')
    const str = await axios.get("https://google.com")
    console.log('there')
    console.log(str.status)
    console.log('everywhere')
})()

console.log('also here')

With output:

here
also here
there
200
everywhere

Go

package main

import (
    "fmt"
    "net/http"
)

func get(url string, c chan int) {
    resp, _ := http.Get(url)
    fmt.Println("there")
    c <- resp.StatusCode
}

func main() {
    c := make(chan int)
    fmt.Println("here")
    go get("https://google.com", c)
    status := <-c
    fmt.Println("everywhere")
    fmt.Println(status)
}

with output

here
there
everywhere
200

In both of these examples, the main execution thread is not blocked.

Let's explore what that means.

In javascript, we see the rest of the program executing while the http request is being made. Note the "also here" log order. With the await we are avoiding callback hell by blocking the current function execution without blocking the overall process execution.

In Go, everything in the current context stops while we wait for the http request. This is because the main function waits for the return from our channel. But the thread is still available to the program as a whole see this explanation under "Blocking is fine". This is a result of the Go runtime and the core scheduler. It's very subtle the way that the main thread is blocked, but this is expected and encouraged behavior in Go because there is a top level language construct to schedule multiple processes in multiple threads and intelligently manage suspended contexts along the way.

We can also modify this example to show how execute other code while waiting for the http call to return.

package main

import (
    "fmt"
    "net/http"
    "time"
)

func get(url string, c chan int) {
    resp, _ := http.Get(url)
    fmt.Println("Everywhere")
    c <- resp.StatusCode
}

func main() {
    fmt.Println("here")
    c := make(chan int)
    go get("https://google.com", c)
    fmt.Println("there")
    for {
        select {
        case status := <-c:
            fmt.Println(status)
            return
        default:
            fmt.Println("Also Here")
            time.Sleep(1000 * time.Millisecond)
        }
    }
}

With Output:

here
Also Here
there
200
everywhere

In this case, we loop in the main function using for { select {.... The main function will hit the default case over and over while waiting for the return value from our spawned channel. We can spawn other coroutines and add to the cases in the select, or write additional procedural code in the default case. In Go this is an expected pattern to allow the language scheduler to handle these routines and potentially multiple threads along the way.

A note on resource limitations.

Remember that Javascript is single threaded. Concurrency is based on coroutines. Go is multi threaded, but a coroutine does not always mean a new thread. See here

Conclusion

Concurrency is important, and coroutines make concurrency possible on fewer resources. Javascript and Go both have schemes to make concurrent, non-blocking, executions more readable and tolerable. The differences are subtle, but result in two tools with a robust ability to execute a lot of async calls on fewer resources.

Reading List

Human Not Human

Jeff Adams — Thu, 12 Sep 2019 13:13:10 +0000

Human Not Human

So as I'm getting my feet on the ground, I thought I'd write a binary classification post for mortals. My focus is on the minutia of working with the data, and how to pick a problem that's simple enough to solve from the ground floor.

Rubin Vase

Binary Classification for Mortals

As it happens, I recently installed a fixed position camera to a Raspberry Pi over my porch. I wrote some code to text me when a motion event triggers. And then came a text every 20 minutes with a picture of a cat on it. Seriously, my neighborhood is overrun with cats!

Understanding the data usage and structure is critical to understanding how to answer problems with ML. The information available on machine learning often doesn't answer the question of the effort that must go into processing the input data to get to solving a problem.

In my own learning path, I have constantly felt I am starting from less than zero because getting to the point of writing the model is so time intensive. The global solution to the complexities of data curation within Data Science at large is to be very prescriptive in model input structures. This is an appropriate solution, however, it hides the unfortunate fact about data science. The majority of data science is being a digital librarian and plumber.

Processing the data -- The hard work

The bulk of this exercise was data entry. There are cooler ways to pre-train or to do unsupervised learning, but the existing datasets tend to hide how much work went into curating them. The convenience of mnist.load_data() is amazing! However to solve problems customized to a specific dataset requires usage of that specific dataset. This was exactly as boring as you think it was. Additionally, I counted people, and counted cars so I can try a few other models on this dataset in the future. I started counting cats as well, but there were too many so I stopped (not a joke).

Building training validation and test datasets

The critical takeaway I keep coming back to is that the data for any neural network is a matrix or array of tensors. Or any way you want to describe a structured collection of numeric values we are going to backpropagate against. It's easy to get tied into the examples demanding a specific directory structure or a specific input encoding, but as long as you end up with matrices, then you'll be ok. There are good reasons to follow standards, and using baked systems for training like Sagemaker will demand a specific input structure, but I would actually recommend doing an exercise of conversion from a starting point that isn't:

| data
    | train
    | test

In my case, the raw data is just a csv with an image path and a few columns of classified data. In order to input model for training validation and test, I first created the input matrix and the output vector list.

dataset = np.ndarray(shape=(len(data), target_height, target_width, channels), dtype=np.float32)
y_dataset = []
i=0

# Set of markers so I can create an lst
for index, row in data.iterrows():
    y_dataset.append(row.human)
    img = load_img(basePath + '/' + row.filename, target_size=(target_height, target_width))
    x = img_to_array(img)
    x = x / 255.0
    dataset[i] = x
    i += 1

Then I used the sklearn train_test_split to randomly generate cohorts for train and validation data. Lastly I split the validation in half for validation and test.

x_train, x_val_test, y_train, y_val_test = train_test_split(dataset, y_dataset, test_size=0.4)

validation_length = int(len(x_val))
# Separate validation from test
x_val = x_val_test[:validation_length]
y_val = y_val_test[:validation_length]

x_test = x_val_test[validation_length:]
y_test = y_val_test[validation_length:]

x_train.shape

The model

The model is the hello world binary classification model with a few added tweaks that seemed like a good idea at the time. I can follow the model structure and understand the underpinning math behind it, but there's a world of difference between following along vs playing jazz. I highly recommend the Cholet's book for a primer on gradient descent and the chain rule that underpins what's happening in the model. I also resized the images so the memory footprint is lowered. This dropped training time by an order of magnitude.

model = Sequential()

model.add(Conv2D(32, (3, 3), activation='relu', input_shape=x_train[1,:].shape))
model.add(MaxPooling2D((2, 2)))

model.add(Conv2D(64, (3, 3), activation='relu'))
model.add(MaxPooling2D((2, 2)))

model.add(Conv2D(128, (3, 3), activation='relu'))
model.add(MaxPooling2D((2, 2)))

model.add(Conv2D(128, (3, 3), activation='relu'))
model.add(MaxPooling2D((2, 2)))

model.add(Flatten())
model.add(Dense(512, activation='relu'))
model.add(Dropout(0.5))
model.add(Dense(1, activation='sigmoid'))

model.compile(loss='binary_crossentropy',
              optimizer='rmsprop',
              metrics=['accuracy'])

Why this model works relatively well

This is not generalized as a classifier for humans. My colleague suggested we try his front door cam, and I suspect it will perform poorly. But the important takeaway is that it solves an actual problem for this specific use case, and it really didn't take me that long, or that much data to get here.

Purists will chide me related to overfitting or other data integrity / bias mistakes, but now that I can successfully remove the cat text messages, I remain enthused.

Training and how to swim without a GPU

The first time I ran this, I trained on my laptop. This took about 4 hours for 4 epochs. So I started looking into Sagemaker and Kaggle. Both offer an amazing service, and both have drawbacks. Kaggle is somewhat limited in how you can load your own data. And Sagemaker gives notebooks that are powerful yet expensive, and getting your data to these mediums is where the complexity lies. I have the local notebook along with the version I used in Sagemaker.

My advice remains consistent with what I have found related to AWS... If you're going to that party, you had better drink all the koolaide. If you are trying to take a half step into the world of AWS, you will encounter every edge case that they lock down. I didn't take my own advice in this case and ended up uploading my data to S3 and pulling it into a Jupyter notebook on Sagemaker.

The training speed was pretty tolerable, and the network didn't seem to hurt too much in terms of time spent. There are more prescribed ways to create training jobs in Sagemaker, but that's for another day. Kaggle would have worked if I had been smarter in my data construction and not required to load the whole dataset into ram as a list. But things to know for the future. There's always something new to learn.

batch_size=16
epochs=10

model.fit(x_train, y_train, batch_size=batch_size, epochs=epochs, verbose=1, validation_data=(x_val,y_val))
model.save('current_full.h5')

I trained on 797 images for the second passes. 171 validation images, and 171 test images. These cohorts were chosen randomly.

Model evaluation

I used the test data that I set aside to do evaluation. Here are the top level results.

loss: 0.5187677096205148
acc: 0.877192983850401

I'm honestly still getting a feel for how to interpret results, but I got to the point of being anecdotally satisfied with the results by writing a bulk process script. This script copies the files in a directory into a path classified set of images according to a set threshold (80% probability of human). This was a good way to gut check how the problem could be used in the future.

#!/usr/bin/env python
# coding: utf-8

from keras.models import load_model
from keras.preprocessing.image import img_to_array, load_img

import os
from shutil import copyfile, rmtree
import sys

import numpy as np


def isHuman(filepath):
    img = load_img(filepath, target_size=(target_height, target_width))
    x = img_to_array(img)
    x = x / 255.0

    size = img.size

    dataset = np.ndarray(shape=(1, size[1], size[0], channels),dtype=np.float32)
    dataset[0] = x
    result = model.predict(dataset)
    return result[0][0]

# load the model
model = load_model('../models/human_not_human.h5')

# Base values
target_height = 180
target_width = 320
channels = 3

accept_threshold = 0.8

path = sys.argv[1]
print(path)

if not os.path.exists(path):
    raise Exception('`{}` does not exist or is not a directory'.format(path))


# Initialize the directory and ensure it's only our current version
dstPath = './data/tested/'
if os.path.exists(dstPath):
    rmtree(dstPath)
os.makedirs(dstPath)
os.makedirs(os.path.join(dstPath, '0'))
os.makedirs(os.path.join(dstPath, '1'))
for f in os.listdir(path):
    if f.endswith('.jpg'):
        filepath = os.path.join(path, f)
        human = '1' if isHuman(filepath) > accept_threshold else '0'
        copyfile(filepath, os.path.join(dstPath, human, f))

Going to production

Once I had the model trained. It is pretty trivial to save and load using Keras. I created two production scripts. One that I can pass a url and get a prediction. The second iterates a directory and copies them into a sorted directory structure. Here is the full operational classifier script.

#!/usr/bin/env python
# coding: utf-8
from keras.models import load_model
from keras.preprocessing.image import img_to_array, load_img

import sys
from urllib.request import urlopen

import numpy as np

model = load_model('full.h5')

url = sys.argv[1]
print(url)

img = load_img(urlopen(url))
x = img_to_array(img)
x = x / 255.0

size = img.size
channels=3

dataset = np.ndarray(shape=(1, size[1], size[0], channels),dtype=np.float32)
dataset[0] = x
result = model.predict(dataset)

print(result[0][0])

I still have to attach this to the camera so for now I'm still getting cat notifications on my phone. But I constantly am running python guess.py http://... with enthusiasm for the ground up solution.

RDS and Stepping into the Plumbing Center of Pain

Jeff Adams — Mon, 09 Sep 2019 20:54:54 +0000

RDS and stepping into the plumbing center of pain:

AWS managed services infrastructure is incredibly convenient in a way I had not previously appreciated. Say you want a lambda that pushes to a a SQS queue of some kind. You add a policy... That's it. They manage the data on the queue based on your configs. Similar if you want to write to DynamoDB. Just a policy. There are drawbacks. It becomes a bit opaque where your data actually lives in a DynamoDB table. But you can connect to it from a lambda by adding a policy. So when I decided to create a lambda backed by a SQL database instead of DynamoDB I expected a similar level of convenience and instead entered the plumbing center of pain(VPC).

Lambda example in a VPC backed by RDS

Special Thanks to Jason Mao for the help reviewing the network VPC setup.

TLDR;

Code Located here.

Pre-requisites

To Run

Set the following environment variables
- STACK_NAME
- STACK_BUCKET (this bucket must exist in your AWS environment)
- YOUR_IP What is your public IP so it gets added to the security group
- This is the only way you'll be able to access your database
run the following in succession:
- make build
- make deploy_rds
- make init_db
run curl {{Url from the output tab of your cloudformation template}}
- You should see two records retrieved and serialized from the database
- bob@example.com
- jane@example.com

RDS

The advertised vision of RDS is a Relational database that is \"easy to set up, operate, and scale.\" It acheives this goal in many ways, but there is a very important distinction. RDS runs on EC2. You pay for compute, and importantly, you must set up the underlying network. Most tutorials just attach your database to your default VPC. This is not a bad thing, but it becomes very difficult to add Security groups through cloudformation or SAM without a referenceable VPC in your infrastructure as code.

So now I'm falling down the rabbit hole of setting up a VPC. And I want to do it using Cloudformation / SAM. There are many great primers on what VPC is and how to set it up (See links), but I found few that also had infrastructure as code, and I know why. It's complicated. So you either get the Hello World VPC tutorial via browser, or the master class that's just the template and sadness.

My goal for this tutorial is to show as simple as possible end to end RDS plus Lambda Plus VPC. My focus is on the Lambda and the minutia of the Lambda to RDS network connection. If you are going to run something like this in production, you will need further and deeper instruction on VPC. I'll include a few tutorials that I liked at the bottom. I also connect to lambda using a username and password. AWS recommends using Roles and an Auth Token to connect instead. I started down this path as the Git history will reveal, but found the scope creaping and my mental health collapsing. I'll add some auth token tutorials as well at the end.

The Code

The Serverless Application Model (SAM) Template

I always recommend utilizing some kind of infrastructure as code. This stuff is complex and you need to iterate on it. I promise you if you setup a VPC using the browser, you will forget all the steps next time you need one. Having a reference and something to step through is critical.

The VPC

This template is long and has a lot of moving parts. They are almost all dedicated to the VPC. My goal is to get to the lambda connection, so I have glossed over an ocean of detail here. It is also important to keep in mind that AWS did not invent these concepts. They merely changed the names in confusing ways that made the marketing department happy. With that in mind, this will be a lot easier to follow if you have done some other networking knowledge. Ensure you know the layered model, and ideally how IP and CIDR route traffic.
Here are the AWS Resources we are building. There are more VPC related things to build for other use cases, but I am purposely ignoring them to keep us focused on this one use case.

VPC
- The abstract grouping of everything else. Logically isolated network from the rest of AWS.
Subnets
- These are both public subnets for simplicity.
- We are building two because we need two to go in the RDS group which is what tells the RDS what VPC to use in cloudformation.
Internet Gateway
- Used to give internet access to resources in your VPC. Primarily to allow traffic out, but no traffic is allowed in.
Route Tables
- IP based route tables. You can have more than one if you need different rules for each subnets. Notice the route rule that sends traffic to the internet gateway. Basically if an IP address shows up and we don't know where it connects. Head to the open internet to find out.

VPC:
  Type: AWS::EC2::VPC
  Properties:
    CidrBlock: 10.1.0.0/16
    EnableDnsSupport: true
    EnableDnsHostnames: true
    Tags:
    - Key: name
      Value:  !Join ['', [!Ref DatabaseName, "-VPC" ]]   
InternetGateway:
  Type: AWS::EC2::InternetGateway
  DependsOn: VPC
AttachGateway:
  Type: AWS::EC2::VPCGatewayAttachment
  Properties:
    VpcId: !Ref VPC
    InternetGatewayId: !Ref InternetGateway

PublicSubnetA:
  Type: AWS::EC2::Subnet
  Properties:
    VpcId: !Ref VPC
    CidrBlock: 10.1.10.0/24
    AvailabilityZone: !Select [ 0, !GetAZs ]
    Tags:
    - Key: Names
      Value: !Sub ${DatabaseName}-PublicA
PublicSubnetB:
  Type: AWS::EC2::Subnet
  Properties:
    VpcId: !Ref VPC
    CidrBlock: 10.1.20.0/24
    AvailabilityZone: !Select [ 1, !GetAZs ]
    Tags:
    - Key: Names
      Value: !Sub ${DatabaseName}-PublicA

PublicRouteTable:
  Type: AWS::EC2::RouteTable
  Properties:
    VpcId: !Ref VPC
PublicRoute:
  Type: AWS::EC2::Route
  DependsOn: AttachGateway
  Properties:
    RouteTableId: !Ref PublicRouteTable
    DestinationCidrBlock: 0.0.0.0/0
    GatewayId: !Ref InternetGateway

PublicSubnetARouteTableAssociation:
  Type: AWS::EC2::SubnetRouteTableAssociation
  Properties:
    SubnetId: !Ref PublicSubnetA
    RouteTableId: !Ref PublicRouteTable

PublicSubnetBRouteTableAssociation:
  Type: AWS::EC2::SubnetRouteTableAssociation
  Properties:
    SubnetId: !Ref PublicSubnetB
    RouteTableId: !Ref PublicRouteTable

The goal here is the absolute minumum that will let us build the lambda and RDS in a VPC that can access each other. There are a lot of other components available for other use cases. And it's notable that this VPC is all public subnets so this is not suitable for a high security environment.

Security Group Plumbing to connect to Lambda

The security groups add a second layer of networking rules we can overlay determining which machines / functions can communicate. We create the Database Security group to allow all outgoing traffic, and incoming traffic from your home IP address along with incoming traffic from the Lambda Security Group. This is the step that will allow the Lambda process to access the RDS Database without knowing the IP address of the lambda container.

LambdaSecurityGroup:
  Type: AWS::EC2::SecurityGroup
  Properties:
    GroupDescription: Allow http to client host
    VpcId: !Ref VPC
    SecurityGroupEgress:
    - IpProtocol: '-1'
      FromPort: -1
      ToPort: -1
      CidrIp: 0.0.0.0/0

DatabaseSecurityGroup:
  Type: AWS::EC2::SecurityGroup
  DependsOn:
    - PublicSubnetA
    - PublicSubnetB
  Properties:
    GroupDescription: Allow http to client host
    VpcId: !Ref VPC
    SecurityGroupIngress:
    # Allows you to log into mysql from home
    - IpProtocol: tcp
      FromPort: 3306
      ToPort: 3306
      CidrIp: !Sub ${YourExternalIp}/32
    # Rule to allow traffic in from resources that are attached to the Lambda Security group
    # This is the specific rule that allows the Lambda to have network access to our RDS cluster
    - IpProtocol: tcp
      FromPort: 3306
      ToPort: 3306
      SourceSecurityGroupId: !GetAtt LambdaSecurityGroup.GroupId
    SecurityGroupEgress:
    - IpProtocol: "-1"
      FromPort: -1
      ToPort: -1
      CidrIp: 0.0.0.0/0

Ok Phew... RDS time

By contrast this is dead simple. Three resources to add for a boilerplate version.

The abstract cluster
One instance
The subnet group We choose the VPC using the subnet group, otherwise it just attaches to the default VPC. The default VPC is suitable for lots of things, but makes Template created security groups difficult. Right here is where the zero to cluster and api breaks down without the VPC. The instance will map one to one with an EC2 instance and apply the network rules we spent so much time with above via two config settings. Obviously it is possible to add more instances, and we would need redundancy for production, but here's a start.

One note: Use SSM parameters to store your usernames / passwords if and when you do this in production. I again wanted to keep things as simple as possible, but this is another place where you need higher security in the actual implementation for production. (Tutorial for SSM)[https://aws.amazon.com/blogs/mt/integrating-aws-cloudformation-with-aws-systems-manager-parameter-store/]

RDSDatabase:
  Type: AWS::RDS::DBCluster
  Properties:
    DBClusterIdentifier: !Ref DatabaseName
    Engine: aurora
    EnableIAMDatabaseAuthentication: true
    MasterUsername: !Ref AdminUserName
    MasterUserPassword: !Ref AdminUserPassword
    DBSubnetGroupName: !Ref PublicSubnetGroup
    VpcSecurityGroupIds:
      - !GetAtt DatabaseSecurityGroup.GroupId
RDSInstance1:
  Type: AWS::RDS::DBInstance
  DependsOn:
    - PublicSubnetA
  Properties:
    DBClusterIdentifier: !Ref RDSDatabase
    # Customize this to the size machine you want
    DBInstanceClass: db.r4.large
    Engine: aurora
    PubliclyAccessible: true
    DBSubnetGroupName: !Ref PublicSubnetGroup

Know that your RDS Cluster is routable via the internet. With the correct Security group setup, you can directly access this database. That's helpful for debugging and data cleanup, but can be dicey depending on the data inside and your security profile. It is possible to create private subnets and make lambda able to access the DB cluster and not have the DB cluster routable from the outside.

API Gateway

This is a simple lambda in an API Gateway that just retrieves the data from the DB. We pass in the config using the environment. Generally we would want to use token Auth instead, but this post is already too long so I link below to tutorials on how to add that.

ServiceApi:
  Type: AWS::Serverless::Api
  Properties:
    Name: ServiceApi
    StageName: !Ref Version
LambdaFunction:
  Type: AWS::Serverless::Function
  Description: Function to get the data out of the database
  Properties:
    Handler: ./dist/main
    Timeout: 30
    Policies:
      - AWSLambdaVPCAccessExecutionRole
    Environment:
      Variables:
        DB_SERVICE_USER: !Ref ServiceUserName
        DB_SERVICE_PASSWORD: !Ref ServiceUserPassword
        DB_HOST: !GetAtt RDSDatabase.ReadEndpoint.Address
        DB_NAME: !Ref DatabaseName
        DB_TABLE_NAME: !Ref DatabaseTableName
    VpcConfig:
      SecurityGroupIds:
        - !GetAtt LambdaSecurityGroup.GroupId
      SubnetIds:
        - !Ref PublicSubnetA
        - !Ref PublicSubnetB
    Events:
      CreateUser:
        Type: Api
        Properties:
          Path: /transactions
          RestApiId: !Ref ServiceApi
          Method: Get

The critical piece is that the lambda runs in the same VPC as the DB allowing us to access it via Security Group Rules. Since our Database is technically public as well, we could setup access from the outside, but then we would not be able to protect the database using security groups. And the Lambda by definition does not have a consistent IP address so there's no IP based rule we can apply. Instead we add it to the security group in the VPC.

The Runtime Guts (But first some init)

We now have all the AWS resources setup. However the API endpoint will fail because it relies on a user that does not yet exist.

Environment:
  Variables:
    DB_SERVICE_USER: !Ref ServiceUserName
    DB_SERVICE_PASSWORD: !Ref ServiceUserPassword
    DB_HOST: !GetAtt RDSDatabase.ReadEndpoint.Address
    DB_NAME: !Ref DatabaseName
    DB_TABLE_NAME: !Ref DatabaseTableName

So I created a script to create a user. This is the part that typically would have token authentication, but let's focus on the connectivity and I'll let better tutorials show the token auth steps (See Below).

Init and user creation

We need a user, a database, a table, and some data. So let's wave our hands and run the script make init_db. This runs a script in Go with command line Env variables. My original intentionwas to make a custom resource to do this, but found the Go library for custom resources infuriating. If you have any error of any kind, Cloudformation will wait 60 minutes to see if things stabilize. Also, since the Lambda runs in a VPC. Tearing down this Cloudformation template also takes about 30 minutes. I have lost days of my life to this issue. Secondly, that Lambda would have environment variables with your master username and password which didn't seem like a good idea.

db.MustExec(fmt.Sprintf(`CREATE DATABASE IF NOT EXISTS %s`, dbName))
db.MustExec(fmt.Sprintf(`USE %s`, dbName))

db.MustExec(fmt.Sprintf("CREATE USER '%s'@'%%' IDENTIFIED BY '%s';", dbServiceUser, dbServicePassword))
db.MustExec(fmt.Sprintf("GRANT ALL ON %s.* TO '%s'@'%%';", dbName, dbServiceUser))
// db.MustExec("FLUSH PRIVILEGES;")

db.MustExec(fmt.Sprintf(`
CREATE TABLE IF NOT EXISTS %s(
  id INT NOT NULL AUTO_INCREMENT,
  email varchar(255) DEFAULT NULL,
  date TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
  PRIMARY KEY (id),
  CONSTRAINT UNIQUE(email)
);`, dbTableName))

db.MustExec(fmt.Sprintf(`INSERT INTO %s (email) VALUES ("bob@example.com");`, dbTableName))
db.MustExec(fmt.Sprintf(`INSERT INTO %s (email) VALUES ("jane@example.com");`, dbTableName))

The actual lambda

This is pretty straightforward, but relies heavily on all that came before.

func handleCrudRequest() (events.APIGatewayProxyResponse, error) {
    dbTableName := os.Getenv("DB_TABLE_NAME")
    transactions := []UserTransaction{}
    err := db.Select(&transactions, fmt.Sprintf(`SELECT * from %s`, dbTableName))
    if err != nil {
        fmt.Println(err)
    }

    out, serializationErr := json.Marshal(transactions)
    if serializationErr != nil {
        fmt.Println(serializationErr)
    }

    return events.APIGatewayProxyResponse{
        StatusCode: 200,
        Body:       string(out),
    }, nil
}

func main() {
    lambda.Start(handleCrudRequest)
}

func init() {
    // Login to the DB
    dbHost := os.Getenv("DB_HOST")
    dbUser := os.Getenv("DB_SERVICE_USER")
    dbPassword := os.Getenv("DB_SERVICE_PASSWORD")
    dbName := os.Getenv("DB_NAME")

    connectionStr := fmt.Sprintf("%s:%s@tcp(%s:%s)/%s?timeout=10s", dbUser, dbPassword, dbHost, "3306", dbName)
    var err error
    db, err = sqlx.Connect("mysql", connectionStr)
    if err != nil {
        log.Fatalln(err)
    }
}

We use environment variables to pass the table and user settings for the db connection. Then pull from our table and return it. Before this will successfully run, ensure you ran make init_db

How to test this

In Cloudformation, look for the output tab in your template and follow that link (Structured similarly to below but with your ids). You should see a json list of two objects returned:

RUN curl https://xxxxxxxxxx.execute-api.us-xxxxxxxx.amazonaws.com/v1/transactions

[
  {
    id: 1,
    email: "bob@example.com",
    date: "2019-07-12 19:09:25"
  },
  {
    id: 2,
    email: "jane@example.com",
    date: "2019-07-12 19:09:25"
  }
]

Summary of what we built

A functioning VPC with two public Subnets in two Availability Zones.
The plumbing in our VPC to give access to the RDS database
- Security Groups that allow traffic without using an IP address
An RDS Cluster with one instance running in one Availability zone (we made two zones since you have to have two zones to create a subnet group for RDS).
A DB init script that runs locally to create:
- A database
- A user we can use to log in with
- A table
- Data that we can pull
An API Gateway Lambda function that runs a simple query to the Database running in the same VPC and outputs it

Don't forget to tear down the infrastructure on this one

Since this template runs an EC2 machine, the price adds up a lot quicker than DynamoDB or Queues or other fully managed services. Run make teardown to clean it all up.

Reading List

General Links
- Virtual Private Cloud (VPC)
VPC Tutorials
- VPC Routing
AWS RDS Token Auth Connection Tutorials

AWS SAM API with Cognito

Jeff Adams — Mon, 09 Sep 2019 15:24:28 +0000

Intro

It's official! AWS has decided that Lambdas are our hammer, and we're all wandering around looking for nails. It's a compelling use case. It's cheap to run, easy-ish to maintain, no infrastructure, and you can run scalable code as a function in the cloud. Lots of tutorials exist to get a hello world function running using various tools both coding and AWS UI related. These tutorials often leave out the ability to create a central API Gateway for a set of functions, and leave out how to protect your API with a basic Authentication layer. My goal is to show a "hello Whole Wide world" example that includes some of these details.

TLDR

Basic Repo is here. PRs and suggestions welcome.

What we are building

An API Gateway
A Cognito User Pool to restrict access to one of our functions
- A Cognito User
A simple funcion that is NOT protected by our auth layer
A simple funcion that is protected by our created auth layer

Pre-requisites

Dive In

You can download the repo, set the needed variables (STACK_NAME, STACK_BUCKET, YOUR_EMAIL) and run make deploy to see this in action. We want to be able to build from zero to stack. I have found that AWS is a sensitive beast and will require continual iteration around subtle details. I cannot stress enough the need to have code that you can run repeatedly in order to step through these iterations methodically. Take it from someone who has lost days of work to this phenomenon that it is worth setting up code from the start.

UserPool / Client / User

UserPool:
    Type: AWS::Cognito::UserPool
    Properties:
        AdminCreateUserConfig:
        AllowAdminCreateUserOnly: false
        UserPoolName: TestingUsers
        UsernameAttributes:
        - email
        AutoVerifiedAttributes:
        - email
        Policies:
        PasswordPolicy:
            MinimumLength: 6
            RequireLowercase: true
            RequireNumbers: false
            RequireSymbols: false
            RequireUppercase: true
UserPoolTokenClient:
    Type: AWS::Cognito::UserPoolClient
    Properties:
      UserPoolId: !Ref UserPool
      GenerateSecret: false
      ExplicitAuthFlows:
      - USER_PASSWORD_AUTH

We create a userpool and a user pool client. The pool is the abstract collection of users and their info. The client is the ability to login using the SDK or the CLI. You may need additional clients (We don't yet have Oauth) and additional properties, but this is a working minimum set that works. Take a look at the Cloudformation Reference Docs for more details. For logging in, we will be using the AWS CLI.

UserPoolUser:
    Type: AWS::Cognito::UserPoolUser
    Properties:
      DesiredDeliveryMediums:
        - EMAIL
      Username: !Ref YourEmail
      UserPoolId: !Ref UserPool

We also create your first user using a set Environment variable YOUR_EMAIL. This will send your email a temporary password on stack creation. Below are instructions for how we will login (spoiler, it's with the CLI).

API

Technically we don't need this. Sam will create one for us. But then when you have two functions, you have two full APIs. We can do better.

ServiceApi:
    DependsOn: UserPool
    Type: AWS::Serverless::Api
    Properties:
      Name: ServiceApi
      StageName: !Ref Version
      Cors:
        AllowMethods: "'*'"
        AllowHeaders: "'*'"
        AllowOrigin: "'*'"
      Auth:
        Authorizers:
          CognitoAuthorizer:
            UserPoolArn: !GetAtt "UserPool.Arn"

Once created, we use the API ID to attach the created functions in one logical group. I have also set Cors headers leaving this wide open. Do not do this unless you understand the implications. Now when we create our functions we can pool them together under this API and have a more organized Microservice instead of a collection of functions.

Functions

Pretty basic declaration. Here is the Unauthenticated Function

OpenLambdaFunction:
    Type: AWS::Serverless::Function
    Properties:
      Description: Handles the basic request with no need for authentication
      Runtime: go1.x
      Handler: ./dist/open
      Events:
        Get:
          Type: Api
          Properties:
            Path: /open
            RestApiId: !Ref ServiceApi
            Method: GET

And the Authenticated. They look very similar, but I wanted different code to handle each. There are good use cases for both merging into events vs separate functions.

LambdaFunction:
    Type: AWS::Serverless::Function
    Properties:
      Description: Handles the basic request
      Runtime: go1.x
      Handler: ./dist/authenticated
      Events:
        Get:
          Type: Api
          Properties:
            Path: /
            RestApiId: !Ref ServiceApi
            Method: GET
            Auth:
              Authorizer: CognitoAuthorizer

The Function specifies the API Gateway to file under, the Authorizer to use, and the path / method to respond to.

Code

This is arguably the simplest part. Just send back a 200. This code is basically the same for both, but with payload content tweaks.

return events.APIGatewayProxyResponse{
    StatusCode: 200,
    Body:       "{\"message\":\"Marshalling a return body is a problem for another day.  But the request was successful\",\"structure\":\"See this is actually not an error\"}",
    // This is important as part of the CORS config.
    // Again you should know the security implications of CORS before implementing this
    Headers: map[string]string{
        "Content-Type":                 "application/json",
        "Access-Control-Allow-Origin":  "*",
        "Access-Control-Allow-Methods": "*",
        "Access-Control-Allow-Headers": "*",
    },
}, nil

Build and Deploy

Since we are using GO, we need a build and compile process. I am using a basic Makefile to compile using GO and run the AWS SAM Cli commands.

You will need to set the following variables:

STACK_NAME
- What to call the cloudformation Stack
STACK_BUCKET
- Sam uploads your compiled code resources to a bucket. I'm leaving it to you to create a bucket and set this environment variable
YOUR_EMAIL
- Put in a valid email address for the first user to create. You can do this through the AWS portal, but the focus is again on a full code auth example.

Once set, run make deploy. Assuming you have access to your AWS environment, you'll see the build process compile the code, upload it to the bucket while transpiling the SAM template into an AWS cloudformation template, and deploying the stack. This will work for updates as well. You will want to view the outputs from the stack creation in order to get the ids needed for login, and the API url to call. I use the web portal for this purpose, but you can also access the output with the CLI.

I have an API, now what?

Let's call the open endpoint.

curl https://{{Your API ID}.execute-api.us-east-1.amazonaws.com/v1/open
{ message: "This endpoint does not require any authentication", structure: "This field was added just to prove it's not an error" }

This works! Now let's call the Authorized endpoint

curl https://{{Your API ID}.execute-api.us-east-1.amazonaws.com/v1
{"message":"Unauthorized"}

Technically this is a good thing, but we can do better. We need to login. We created a token client that will respond to SDK / CLI requests to log in. This is arguably less secure, but allows us to login without additional infrastructure.

Login and get a token

We will use the AWS cli to login. The first login will require changing the password and follow a challenge workflow. To make this slightly less painful, I created a script you can call that will log in and run the password challenge response.

AUTH_CHALLENGE_SESSION=`aws cognito-idp initiate-auth \
--auth-flow USER_PASSWORD_AUTH \
--auth-parameters "USERNAME={{YOUR_EMAIL}},PASSWORD={{password from the email AWS sent you}}" \
--client-id {{Token Client ID}} \
--query "Session" \
--output text`

# Then respond to the challenge
aws cognito-idp admin-respond-to-auth-challenge \
--user-pool-id $USERPOOLID \
--client-id {{Token Client ID}} \
--challenge-responses "NEW_PASSWORD=Testing1,USERNAME={{YOUR_EMAIL}}" \
--challenge-name NEW_PASSWORD_REQUIRED \
--session $AUTH_CHALLENGE_SESSION

To use this script, get the output values from your cloudformation stack and run the following command:
./scripts/login_first.sh {{User Pool ID}} {{Token Client ID}} {{Your Email}} {{Password in the Email AWS sent you}}

This will change your password to 'Testing1' and log you in. You will get back a JSON Web Token or JWT token you can now use to finally call the damn API.

Bring it home

Now that we have the auth token, we can add it to the headers and call the

curl -H "Authorization: {{AUTH_TOKEN (The output from the login_first script)}}" https://{{Your API ID}}.execute-api.us-east-1.amazonaws.com/v1

{"message":"Marshalling a return body is a problem for another day.  But the request was successful","structure":"See this is actually not an error"}

What have we accomplished?

So we now went from zero to:

Cognito UserPool Authorizer
API Gateway where we can put multiple functions
A function that does not require authorization at path /open
A function that requires authorization at path / We can login using the AWS CLI / the login script ./scripts/login.sh {{UserPool Client ID}} {{Your Email}} Testing1 and add the output IdToken to our request in order to call our API.

Forem: Jeff Adams

Github Actions: Steps in Containers

The initial example

Language Validation with PHPStan

Let's break this down:

Running an arbitrary Docker container as a step

Taking this idea further

Internal Tools and Secrets Abstraction

Compliance

And finally a Recap

Concurrency comparison Javascript to GO

Example

Javascript

Go

In both of these examples, the main execution thread is not blocked.

A note on resource limitations.

Conclusion

Reading List

Human Not Human

Human Not Human

Binary Classification for Mortals

Processing the data -- The hard work

Building training validation and test datasets

The model

Why this model works relatively well

Training and how to swim without a GPU

Model evaluation

Going to production

Links

RDS and Stepping into the Plumbing Center of Pain

RDS and stepping into the plumbing center of pain:

Lambda example in a VPC backed by RDS

TLDR;

Pre-requisites

To Run

RDS

The Code

The Serverless Application Model (SAM) Template

The VPC

Security Group Plumbing to connect to Lambda

Ok Phew... RDS time

API Gateway

The Runtime Guts (But first some init)

Init and user creation

The actual lambda

How to test this

Summary of what we built

Don't forget to tear down the infrastructure on this one

Reading List

AWS SAM API with Cognito

Intro

TLDR

What we are building

Pre-requisites

Dive In

UserPool / Client / User

API

Functions

Code

Build and Deploy

I have an API, now what?

Let's call the open endpoint.

Login and get a token

Bring it home

What have we accomplished?

Recommended Reading