Forem: Rolf Streefkerk

How to implement the Twitter - X API using Python FastAPI

Rolf Streefkerk — Wed, 09 Oct 2024 09:08:07 +0000

Introduction

While building my latest app, insightprose.com, I’ve gathered quite a few learnings in its 3 month development cycle. I wanted to start with Twitter integration since it’s a key component in the application feature-set of InsightProse.

I’ll be discussing:

the current state of X API v2.0,
where the Twitter API came from (version 1.1) briefly,
and how to implement OAuth2 auth flow with X API version 2.

In case you’re interested in a quick run-down of what InsightProse actualy is, keep reading on. Alternatively:

Twitter - X API
- v1.1 API deprecation controversy
- v2.0 API Implementation
- Common questions and answers
Conclusion

What is InsightProse?

InsightProse is a social media and SEO content generator, using your original long form article, to distill one or more concepts into short articles.

These short articles are called Insights, these can then be used to create:

Twitter threads, generates a Twitter thread of maximum 3 tweets to enhance social media reach, and consequently improve readership of the original article. These threads can be scheduled up to 1 month in advance.
LinkedIn Articles, generates a short article of maximum 3000 characters that explains, in your branding style, a concept from the original article with the intention of getting people to read that original long form article.
Blog posts; can be placed on a blog or website in order to enhance SEO, improve keyword relevance, and act as more concise Questions and Answers article that references and mirrors the original article. This format includes original code blocks (if any) and images where relevant for optimal concept explanation.

All of this content takes into account:

Your brand profile:. In other words, how you formulate text, how you explain ideas, and the target audience. This is to ensure the generated content does not read like AI generated content, but it’s like you’ve written it.
Keywords from the original article: In order to ensure relevance and SEO improvement, keywords from the original article are used appropriately and repeatedly in case you want to repost the insight prose on your blog for SEO purposes.
It takes into account “Questions and Answers” format that is used by many to search for answers using Google or other search engines. Again, with the purpose to improve relevance for those specific sets of keywords.

InsightProse helps you promote your original content such that you can focus on long form content writing.

Twitter - X API

v1.1 API deprecation controversy

The moment of v1.1 deprecation

Twitter / X used to be generous with its API usage towards developers, likely because they had income from advertisers primarily and no subscription services.

Now, this model has changed to a subscription oriented model with most advertisers dropping off. That has also affected the “user friendliness” towards developers. In a very negative way.

It started with the official announcement back in April 2023 the v1.1 API was being deprecated ¹.

Today, we are deprecating our Premium v1.1 API, including Premium Search and Account Activity API.

You’ll notice in the thread of this announcement that there’s no love for this change, and that’s because the fees are not reasonable whatsoever.

The new rate limits and pricing

The issue starts with limits to posting Tweets on behalf of customers that have been severely reduced for the free access variant of the API ² ³:

v1.1: 300 posts per 3 hours = 2400 posts per 24 hours.
v2.0: 50 posts per 24 hours.

This is a factor of 48(!) reduction in Tweet post allowance. In order to mitigate some of these rate limiting issues, you can upgrade to the “Basic” X API.

1667 Tweets per 24 hours costing 100USD/month ⁴

You can imagine if you’re running a small SAAS product. In this case, 100 USD, is double the price of my infrastructure running cost on Digital Ocean. Double!

To further make the point, my infrastructure is a Kubernetes 2 Node cluster. For many developers that use Firebase and a free static site hosting solution such as AWS S3 or Cloudflare pages. They will pay near 0 USD per month to get bootstrapped.

The consequences, and my recommendation to X

This pricing means that posting Tweets on behalf of your customer needs to be severely capped or put on higher pricing tiers to get to the sufficient revenue to make it sustainable to pay 1200 USD / year to X.

I’m hoping that X will revise its pricing to considering smaller SAAS products and companies use-cases and enable them to integrate with X at reasonable prices.

I would recommend the following subscription tier to be added:

The introduction of a “Startup” tier 20 USD/month subscription would cater to starting business owners that want to built a quality service around the X eco-system. The current “Basic” 100 USD/month subscription, and “Free” options, are too expensive and too restrictive respectively.

v2.0 API Implementation

The basic OAuth2.0 implementation flow⁵ for X is demonstrated in the following diagram:

Login with X button on your website.
Login is redirected to your API service.
Your API; a. forwards X Authorization request by; b. redirecting the user to the X website, c. to request scope Authorization.
The user has Authorized your application to access the requested scope of information and access to X user account details.
Request authorization ‘code’ forwarded to your API ‘/auth/x’ service.
Your auth X service receives the user data and X details such as username and refresh and access tokens through the coded response from the X API.
Return your application refresh and access tokens to signal authorization is completed, user is logged in.

API /login/x (Step 2, 3, and 4)

The function of this API is to forward the user request to X such that they can authorize your APP to access the user’s account data and to post on behalf of this user.

In your FastAPI implementation you probably have a centralized api.py file that you add all individual API routes to:

api_router = APIRouter()
api_router.include_router(login.router, tags=["login"])

In the ./endpoints/login.py I would have the following route:

@router.get('/login/x')
async def login_twitter(request: Request):
    """Handle Twitter login using redirect to Twitter."""
    return await twitter.initiate_twitter_login(request)

Then to create the redirect url, we would do the following within the initiate_twitter_login function:

generate_oauth_params: create the required code verifier, challenge and state parameters.
create_authorize_url: are used to create the full redirect url to X API, including the scope. The minimum scope to post Tweets on behalf of your users: tweet.read, users.read, tweet.write. The offline.access scope is to get a refresh_token from X.

import secrets
import base64
import hashlib
from fastapi.responses import RedirectResponse

def _generate_oauth_params():
    code_verifier = secrets.token_urlsafe(32)
    code_challenge = base64.urlsafe_b64encode(hashlib.sha256(code_verifier.encode('utf-8')).digest()).decode('utf-8').rstrip('=')
    state = secrets.token_urlsafe(32)
    return code_verifier, code_challenge, state

def _create_authorize_url(code_challenge: str, state: str) -> str:
    params = {
        'response_type': 'code',
        'client_id': settings.TWITTER_CLIENT_ID,
        'redirect_uri': settings.TWITTER_CALLBACK_URL,
        'state': state,
        'code_challenge': code_challenge,
        'code_challenge_method': 'S256',
        'scope': 'tweet.read users.read tweet.write offline.access'
    }
    return f"https://x.com/i/oauth2/authorize?{urlencode(params)}"

async def initiate_twitter_login(request: Request):
    code_verifier, code_challenge, state = _generate_oauth_params()
    return RedirectResponse(_create_authorize_url(code_challenge, state))

This redirect will present you with a request screen from X;

initiated by step 3,
in step 4 the user authorizes the app,
which leads to the request send by X as step 5:

API /auth/x (Steps 5, 6 and 7)

This endpoint receives the authorization from X, this is why you need to configure the callback API in the X settings⁶ such that X knows where to forward this API call to:

@router.get('/auth/twitter')
async def auth_twitter(request: Request, db: Session = Depends(deps.get_db)):
    access_token, refresh_token = await twitter.handle_twitter_callback(request, db)

    return RedirectResponse(url=f"{settings.FRONTEND_URL}/app/auth/callback?access_token={access_token}&refresh_token={refresh_token}")

This API takes care of validation of the OAuth state secret that we created in the first API, which should match here.

Hence, we start with a state check to ensure that the request was initiated from this session and not from somewhere else.

The code contains the X tokens, because we requested the offline.access scope we also get a refresh token next to the regular access token.

async def handle_twitter_callback(request: Request, db: Session): Tuple
    if request.query_params.get('state') != request.session.get('oauth_state'):
        raise HTTPException(status_code=400, detail="Invalid state parameter")

    code = request.query_params.get('code')
    if not code:
        raise HTTPException(status_code=400, detail="No authorization code provided")

    token_data = await _exchange_code_for_token(code, request.session.get('code_verifier'))
    twitter_user_info = await get_twitter_user_info(token_data['access_token'])

    # Create your Application user with X details here

    request.session.pop('code_verifier', None)
    request.session.pop('oauth_state', None)

    # access_token, refresh_token
    return access_token, new_refresh_token

To receive this; we execute token_data = await _exchange_code_for_token(code, request.session.get('code_verifier'))

async def _exchange_code_for_token(code: str, code_verifier: str) -> dict:
    url = 'https://api.x.com/2/oauth2/token'
    data = {
        'code': code,
        'grant_type': 'authorization_code',
        'client_id': settings.TWITTER_CLIENT_ID,
        'redirect_uri': settings.TWITTER_CALLBACK_URL,
        'code_verifier': code_verifier
    }
    headers = {
        'Content-Type': 'application/x-www-form-urlencoded',
    }
    auth = httpx.BasicAuth(settings.TWITTER_CLIENT_ID, settings.TWITTER_CLIENT_SECRET)
    return await _make_twitter_api_call('POST', url, headers=headers, data=data, auth=auth)

With the received access token, we can pull in the user data using get_twitter_user_info(token_data['access_token']) function:

async def _get_twitter_user_info(access_token: str) -> dict:
    url = 'https://api.twitter.com/2/users/me'
    params = {
        'user.fields': 'id,name,username,profile_image_url'
    }
    headers = {
        'Authorization': f"Bearer {access_token}"
    }
    return await _make_twitter_api_call('GET', url, headers=headers, params=params)

Once you have the Twitter / X user data, you create your application user profile with your application access credentials and return that to your user and they’re logged in.

Refresh token cron job

Since OAuth2.0 has limited access token validity time, 7200 seconds in case of Twitter / X OAuth2.0. We need to manage automatic renewal of this token in the background.

I recommend using a task scheduling system or a cron job that automatically checks your user table or token issuance table for expired / about to be expired access tokens.

In my application I’m using apscheduler⁷ to schedule tasks on the same application server as the API.

apscheduler is a low profile scheduling library that will “attach” itself to the FastAPI application lifespan event hook.

If you want to know more about how to use it, let me know on social media!

This is configured as a lifespan event⁸, that way it will be running in the background from the moment your FastAPI server is online. This lifespan event uses an async context manager to handle the two events; startup, and shutdown (after the yield) to start and stop the scheduler:

@asynccontextmanager
async def lifespan(app: FastAPI):
    # Setup code (runs before the app starts)
    try:
        wait_for_db()
        scheduler = create_scheduler()
        setup_scheduler(scheduler)
        scheduler.start()
        logger.info("Application startup completed successfully")
    except Exception as e:
        logger.error(f"Error during application startup: {e}")
        raise
    yield
    # Cleanup code (runs when the app is shutting down)
    scheduler.shutdown()
    logger.info("Application shutdown")

app = FastAPI(
    lifespan=lifespan,
    title=settings.PROJECT_NAME,
    openapi_url=f"{settings.OPENAPI_URL}"
)

In the task that is defined, we want to run this regularly to refresh expired Twitter / X access tokens:

Be aware that you need to revoke your access tokens in case you’re refreshing them within the 2 hour lifespan to avoid token refresh failure errors (see Why am I getting refresh token failure with Twitter / X API)

async def refresh_all_expired_tokens(db: Session):
    now = datetime.now(timezone.utc)
    users_with_expired_tokens = user_crud.get_users_with_expired_tokens(db, now)

    for user in users_with_expired_tokens:
        try:
            new_token = await refresh_twitter_token_by_user_id(user.id, db)
            if new_token:
                logger.info(f"Successfully refreshed token for user {user.id}")
            else:
                logger.warning(f"Failed to refresh token for user {user.id}")
        except Exception as e:
            logger.error(f"Error refreshing token for user {user.id}: {str(e)}")

Common questions and answers

Why should you use OAuth 2.0 and not Oauth1.0a for Twitter / X API authorization?

OAuth2.0 offers several benefits over v1.0a:

Granular permission management vs. fine-grained:
Refresh tokens vs. no expiry access tokens.
Eventual v1.0.a deprecation, requiring migration effort to OAuth2.0

Generally, security and access controls have improved in version 2.0. However, that does mean you have to manage access token validity in your application automatically (see Refresh token cron job).

What is the validity of the refresh token and access tokens for Twitter / X API?

There’s no clear documentation on the official developer.x.com that clarifies expiration of the refresh_token provided, but apparently it’s 6 months according to one user in the x community⁹

Access token has a return value with expires_in set to 7200 seconds, which is 2 hours.

Why am I getting refresh token failure with Twitter / X API

To solve this, you need to ensure:

revoke the current active access_token within its 2 hours validity window,
after the 2 hour window, you can use the refresh_token to renew the access_token without revoking the access_token.

This is how you revoken the access_token:

async def revoke_twitter_token(token: str) -> bool:
    url = 'https://api.x.com/2/oauth2/revoke'
    data = {
        'token': token,
        'client_id': settings.TWITTER_CLIENT_ID
    }
    headers = {
        'Content-Type': 'application/x-www-form-urlencoded'
    }
    auth = httpx.BasicAuth(settings.TWITTER_CLIENT_ID, settings.TWITTER_CLIENT_SECRET)
    return await _make_twitter_api_call('POST', url, headers=headers, data=data, auth=auth)

This is how you refresh the access_token:

async def refresh_twitter_token(refresh_token: str) -> dict:
    url = 'https://api.twitter.com/2/oauth2/token'
    data = {
        'refresh_token': refresh_token,
        'grant_type': 'refresh_token',
        'client_id': settings.TWITTER_CLIENT_ID,
    }
    headers = {
        'Content-Type': 'application/x-www-form-urlencoded'
    }
    # Prepare Basic Auth
    auth = httpx.BasicAuth(settings.TWITTER_CLIENT_ID, settings.TWITTER_CLIENT_SECRET)
    return await _make_twitter_api_call('POST', url, headers=headers, data=data, auth=auth)

Which API's do I have access to in the Free version of X API v2.0?

You can see a full listing of your API access when you register an account, in your development dashboard¹⁰.

Or you can go to the About X API page that is publicly accessible².

Conclusion

Unfortunately with the advent of Twitter / X API version 2, the usability for small products and applications has been severely diminished with a high ticket entrance fee of 100 USD per month for the Basic plan. For the Free version, a very limited allowance of 50 Tweets per day.

There’s been backlash from day one when this new business model was announced, however we haven’t seen X make any moves to amend or improve their API access for smaller startups and businesses with low revenue.

Luckily, the implementation of the X API is pretty straightforward as I’ve hopefully demonstrated in this article. There are some caveats when it comes to access token / refresh token issues that have been reported online very frequently. But with a proper implementation of access token revoke, before a refresh this should be resolved.

Have you encountered any problems implementing the new X API v2.0? If so, lets discuss below!

Thanks for reading.

https://x.com/XDevelopers/status/1649191520250245121 - Twitter API v1.1 deprecation announcement ↩
https://developer.x.com/en/docs/x-api/getting-started/about-x-api Twitter API Access Levels and versions ↩
https://developer.x.com/en/docs/x-api/rate-limits#v2-limits-free Twitter API v2.0 Rate limits for Free Access ↩
https://developer.x.com/en/docs/x-api/rate-limits#v2-limits-basic Twitter API v2.0 Rate limits for Basic Paid Access (100USD / month) ↩
https://developer.x.com/en/docs/authentication/oauth-2-0/user-access-token Twitter OAuth2.0 Authorization Code Flow with PKCE ↩
https://developer.twitter.com/en/portal/dashboard Twitter Developer Dashboard ↩
https://apscheduler.readthedocs.io/en/3.x/# Advanced Python Scheduler ↩
https://fastapi.tiangolo.com/advanced/events/ FastAPI lifespan events ↩
https://devcommunity.x.com/t/access-token-expires-in/164425/13 Access token expires_in and Refresh token expiry ↩
https://developer.twitter.com/en/portal/products/free Twitter Developer Dashboard - Your API endpoint access overview ↩

Kubernetes Foundations and Cluster Setup with K3s

Rolf Streefkerk — Tue, 06 Aug 2024 13:00:00 +0000

Preface

TLDR: to get right into the article, go here

In my own personal Cloud journey, Kubernetes never really played much of a role up until very recently.

My interests were with public Cloud providers such as AWS for their accessibility and managed service offerings they provide to significantly increase productivity.

But, when i actually started to run production workloads on AWS I noticed the significant cost involved in fairly small sized deployments.

As an example we had an Amazon merchandise app called Trademerch.io that cost us around 130USD per month to run when we started and finished our rework about 5 years ago (ElasticSearch cluster with 1 node, 2 RDS databases, 2 EC2 servers of which 1 was on a Cron recurring Job). Fast forward to today, with a virtually unchanged architecture we were still paying 130USD per month using the same AWS services, sizing and usage patterns.

The cost aspect of the public cloud is not something new, but it became painfully obvious that AWS is not there to return the favor, but instead seeks to optimize its own revenue primarily (although they claim otherwise!) This experience has left me jaded when it comes to public Cloud providers in general.

How can this be done differently?

Enter Kubernetes and the “deploy where you want” paradigm. Once i finally started looking into Kubernetes about a month ago, I noticed immediately that it’s essentially a Platform as a Service (PAAS) that you fully control. However, most deployment options are NOT cheap (again AWS, but also Azure and Google itself).

Most public clouds will charge 75 USD to just run the control plane (the “controller” of the Kubernetes cluster) and then you still need to add Nodes to run your workloads (Virtual machines such as EC2). Luckily there are alternatives out there such as Digital Ocean, and Oracle that will allow you to run Kubernetes much more cost effectively.

Next to this, you have the option to self host anywhere you like, which negates issues of vendor lock with traditional public cloud managed services. Particularly Serverless offerings, since they’re custom built for their own Cloud.

Now I’ll admit, cost to entry with Serverless can be 0 USD, when your app has no users or very few users it’s possible to run at at zero or near zero cost. But as soon as you get free of the “Free Tier”, costs starts to accumulate and migrating out is a more time consuming affair.

Kubernetes offers a way to host your applications and infrastructure in a way that is fully in your control, and enables you to scale to virtually millions of containers (parent company Google run this size, regular people of course will not ever come close!).

My reasoning is, if I can control the infrastructure stack I can learn to scale it to sufficient size and never have to migrate out of the stack. That provides me with transparency and clarity on the future roadmap. Granted, I will still use Serverless where it makes sense.

Currently for my insightprose.com app I use CloudFlare to host my static website and some of their serverless function capability to run forms. But much of the backend will run on Digital Ocean using 4-6 Node Kubernetes cluster that will run everything needed from Monitoring to Databases and API’s.

I see Kubernetes as an investment for that reason, and because it has been proven to be extremely successful at what it aims to provide. The down side is the up-front investment in time and money, for each person that cost-benefit analysis may turn out different.

For me it’s a resounding yes to invest in this technology, and to learn how to best use it for my benefit.

I hope this series of articles will impart you with enthusiasm to learn and use Kubernetes for your own projects and products, or perhaps even aid you in your current activities!

A humble note; since I’m new with Kubernetes, I expect to make mistakes.

If you spot any, or have advice to improve the content. I’d be very appreciative if you would contact me via Twitter or via my contact form.

Thanks for your support!

Thanks and enjoy your learning journey through Kubernetes!

Introduction

Kubernetes was first introduced in 2014 by Google and has since become the de facto standard for large scale container orchestration, powering anything from small startups to conglomerates like Google itself.

Kubernetes has rose to prominence because of its ability to deploy, scale and manage complex application across multiple containers and hosts. Since 2014 applications have grown considerably more complex. With that growth, new service paradigms such as the micro service architecture has gained in popularity.

In the most recent Cloud Native Annual Survey of 2023, ¹ , Kubernetes was still seeing year over year growth with a dramatic increase in production use from 58% in 2022 to 66% in 2023 of total respondents.

Kubernetes offers the following main features:

Automated deployment and scaling: Easily scale your application up or down based on demand.
Self-healing capabilities: Automatically restarts failed containers and replaces or reschedules containers when nodes die.
Service discovery and load balancing: Kubernetes can expose a container using the DNS name or its own IP address. If traffic to a container is high, Kubernetes is able to load balance and distribute the network traffic.
Storage orchestration: Automatically mount the storage system of your choice, whether from local storage, public cloud providers, or network storage systems.
Secret and configuration management: Deploy and update secrets and application configuration without rebuilding your image and without exposing secrets in your stack configuration.

Real world use cases: Airbnb's Kubernetes Journey

Airbnb, the popular online marketplace for lodging and tourism experiences, provides an excellent example of Kubernetes in action. As their platform grew, they faced challenges with their monolithic architecture. They needed a system that could handle their rapidly increasing scale while allowing for more frequent and reliable deployments.

Airbnb adopted Kubernetes ² to containerize their applications and move towards a microservices architecture. This transition allowed them to:

Increase deployment frequency: exponential growth in deployments was achieved all the way up to 125,000 production deployments per year.
Increased resource usage efficiency: scheduling improvements reduced computing resource waste.
Enhanced reliability and fault tolerance.
Improved configuration management: version, refactor, and automatically update configuration across hundreds of services has dramatically improved efficiency in this regard.
Increased developer productivity: via standardization and automations repetitive task reductions were achieved.

While adopting Kubernetes required investment in tooling and processes for Airbnb, it provided key improvements across the board from development experience to standardization and increased deployment efficiency.

In this article, we'll be using k3s 1.30, a lightweight Kubernetes distribution that's perfect for learning, development, and edge computing. K3s provides a fully compliant Kubernetes distribution with a smaller footprint, making it easier to get started and experiment with Kubernetes concepts.

Understanding Containers: A Quick Recap

Before we dive deep into Kubernetes, let's revisit the concept of containers, the building blocks of any Kubernetes system.

Containers are lightweight, standalone, executable packages that include everything needed to run software. This includes the code, runtime, system tools, libraries, and settings. Containers are isolated from one another and the underlying infrastructure, ensuring consistency across environments.

The key benefits of containers include:

Consistency across environments: Containers run the same regardless of where they're deployed, eliminating the "it works on my machine" problem.
Isolation and resource efficiency: Containers share the host OS kernel but are isolated from each other. This allows for efficient resource utilization while maintaining security.
Quick startup and scalability: Containers can start up in seconds and are easy to replicate, making them ideal for scalable applications.
Version control and component reuse: Container images can be versioned, shared, and reused, promoting better development practices and code reuse.

While Docker popularized containers, Kubernetes works with various container runtimes. In k3s, containerd is used as the default container runtime.

📌 Deep Dive for Intermediate Users:
If you're familiar with containers, you might be interested in the Open Container Initiative (OCI) standards. These standards ensure interoperability between different container technologies. K3s uses containerd, which is OCI-compliant, allowing it to work with containers created by Docker and other OCI-compliant tools ³.

Kubernetes Architecture: The Blueprint of Orchestration

Let’s start with an overview of the Kubernetes architecture to better understand how the major components interact:

The developer or user has access to Kubernetes and everything that’s running on it via the command line tool; kubectl, this tool has immediate API Server access to the whole cluster.

Control Plane Components

First off we have the major components of the control plane, which is essentially the backbone of the cluster making global decisions around scheduling and events.

API Server: exposes the API of the control plane and is scaled horizontally (adding more instances) and balancing traffic between them.
etcd: highly available key-value cluster storage for all cluster data. On K3s this is a SQLite database.
Scheduler: schedules newly created Pods with no assigned Node. Takes into account; individual and collective resource requirements, hardware/software/policy constraints, affinity, anti-affinity specifications, data locality etc.
Controller Manager: this runs the controller processes of which there are various kinds, a few examples:
- Node controller: monitors and responds to nodes going down.
- Job controller: watches for Job objects that are a one-off task, creates Pods to run those tasks to finish the tasks.
Cloud Controller Manager: (not depicted and optional): these run controllers that are specific to the cloud provider, this includes many different controller types in a single binary just like the “regular” controller manager.

Node Components

For the Nodes, which are the workers of the cluster and most of the time will be virtual servers like for instance EC2 on AWS or Droplets on Digital Ocean.

Kubelet: Kubelet is like a “node agent” that runs on each Node and that can register the node using a PodSpec. This is essentially the interface between the control plane and the node. Providing two key services:
1. Node Administration: The kubelet monitors the node and optimizes it, procures the resources and setting networking to maintain operation.
2. Pod execution: The kubelet acts as the coordinator of operations for each node on a pod, activities such as scheduling, startup, and optimizing pod performance within the context of the overall Kubernetes cluster.
Container Runtime: Docker this is the default runtime using containerd open source daemon that facilitates the life cycle management via API requests. The container runtime deals with;
- verifying and loading container images
- monitoring system resources
- isolating and allocating resources
- and the container life cycle.
Kube Proxy: Kube proxy is a network proxy running on each node in the cluster, this implements part of the Service concept when it comes to the networking side. Specifically, networking rules allowing/disallowing network communication to your pods or outside the cluster. Typically, it will use iptables for traffic routing.

📌 Deep Dive for Intermediate Users:
K3s simplifies the control plane by combining the API Server, Controller Manager, and Scheduler into a single binary. It also replaces etcd with SQLite as the default data store, though etcd is still supported. This design allows k3s to be lighter and faster to set up, making it ideal for edge computing and IoT scenarios ⁴.

Core Kubernetes Concepts: The Building Blocks

We’ve already used the terms Pods and Nodes, lets explore these concepts a bit further and understand how they interact.

Pods

The smallest deployable units of computing to create and manage in Kubernetes

Key characteristics of Pods:

A Pod can contain one or more containers.
Containers in a Pod share the same network namespace, meaning they can communicate with each other using localhost.
Containers also share the same shared storage volume in Pod.
Pods are ephemeral by design. They can be created, destroyed, and recreated as needed.
Each Pod gets its own IP address in the cluster.

Example use cases for multi-container Pods:

Sidecar container for logging
Initialization container that sets up the environment before the main container starts

📌 Deep Dive for Intermediate Users:
Pods are created usually as workload management resources, not directly as a Pod. That means, Pods are created as a Deployment, StatefulSet or a Job specification. More on this in the coming articles of this series.

Nodes

Nodes are where the workloads are run for a cluster, placed inside a pod that runs on a node. Nodes can be virtual or a physical machine.

Key points about Nodes:

Each Node runs a Kubelet, a container runtime, and a Kube Proxy.
Nodes can be added or removed from the cluster to scale your infrastructure.
Kubernetes supports heterogeneous clusters, meaning you can mix and match different types of nodes.

Clusters

A Kubernetes cluster is the complete set of Nodes and the Control Plane managing them. It's the entire ecosystem where your applications live.

Important aspects of Clusters:

Clusters can span multiple physical or virtual machines.
They provide redundancy and high availability for your applications.
You can run multiple clusters for different environments (dev, staging, production) or to separate concerns.

Namespaces

Namespaces provide a way to divide cluster resources between multiple users or projects. They're like virtual clusters within a physical cluster.

Benefits of using Namespaces:

They provide a scope for names, allowing you to use the same resource names in different namespaces.
They're a great way to divide resources between different teams or projects.
You can set resource quotas for each namespace, ensuring fair resource allocation.

📌 Deep Dive for Intermediate Users:
Kubernetes already starts with 4 initial namespaces ⁵, 2 of the more notable namespaces included are;
default so that you can start using your new cluster without first creating a namespace.
kube-system is the namespace for objects created by the Kubernetes system

Hands-on: Setting Up k3s

We should have a pretty good understanding of what Kubernetes is and what components it utilizes to create an environment that can run applications and manage them effectively.

Let’s go ahead and setup our own cluster using k3s 1.30

Installing k3s

The installation is very simple.

curl -sfL https://get.k3s.io | INSTALL_K3S_EXEC="--disable traefik" sh -

This command downloads the latest k3s binary and installs it as a service.

By default Traefik is included in the installation of k3s, this flag -disable traefik disables it. We will install Traefik later on in this series, you can ignore it for now.

Copy the kube config file to our home directory, this will avoid us having to run sudo for any k3s command:

mkdir ~/.kube
sudo cp /etc/rancher/k3s/k3s.yaml ~/.kube/config
sudo chown $USER:$USER ~/.kube/config
chmod 600 ~/.kube/config

Ensure you load the configuration that we just copied.

echo 'export KUBECONFIG=~/.kube/config' >> /home/vagrant/.bashrc

Starting Your Kubernetes Cluster

One of the advantages of k3s is that it starts automatically after installation. You can verify that it's running with:

k3s kubectl get nodes

You should see output similar to:

NAME                STATUS   ROLES                  AGE   VERSION
insight-prose-dev   Ready    control-plane,master   17h   v1.30.3+k3s1

This output shows that you have a single-node cluster up and running, with your machine acting as both the control plane and a worker node.

To show all the Pods currently in the cluster:

k3s kubectl get pods -A

📌 Deep Dive for Intermediate Users:
Kubernetes (and as such K3s) uses containerd as its container runtime, which is lighter than Docker. If you're used to Docker commands, you might need to use crictl for debugging container issues in k3s. The crictl command-line tool is included with k3s ⁶.

Your First Kubernetes Deployment

With our cluster up and running, let's deploy a simple web application. We'll use Nginx as our example. The following diagram illustrates the deployment process:

The above diagram shows which control plane services (yellow) are involved in creating a new Deployment and how the process is sequenced.

Creating a Deployment YAML

Create a file named nginx-deployment.yaml with the following content:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
  labels:
    app: nginx
spec:
  replicas: 3
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:1.26.1
        ports:
        - containerPort: 80

We’re deploying the following:

apiVersion, kind, and metadata are required fields that describe the resource and its API version.
spec describes the desired state for the Deployment.
replicas: 3 tells Kubernetes to run three copies of this application.
selector defines how the Deployment finds which Pods to manage. These labels need to be unique within the namespace, otherwise strange things can happen.
template describes the Pods that will be created. In this case, each Pod will run one container using the nginx:1.26.1 image.
we expose this Deployment on port 80. This should look very familiar those coming from a Docker background.

Deploying the Application

Apply the deployment to your cluster:

kubectl apply -f nginx-deployment.yaml

This command tells Kubernetes to create the resources described in the YAML file.

🚨 Troubleshooting Tip:
If you see an error like "The connection to the server localhost:8080 was refused", make sure you've set up your kubeconfig correctly as described in the previous section. You might need to restart your terminal or run export KUBECONFIG=~/.kube/config.

Exposing the Application

To access the Nginx server from outside the cluster, we need to expose it as a Service:

kubectl expose deployment nginx-deployment --type=LoadBalancer --port=80

This creates a Service that exposes our Nginx deployment. The --type=LoadBalancer flag tells Kubernetes to provide an external IP to access the Service.

To see the details of the Service:

kubectl get services nginx-deployment

In a cloud environment, this would typically provide an external IP. In k3s running locally, you might see <pending> for the external IP. You can still access the service using the cluster IP or node port.

📌 Deep Dive for Intermediate Users:
K3s comes with ServiceLB LoadBalancer out of the box, which allows the use of LoadBalancer services without additional configuration. Many Kubernetes installations will require a LoadBalancer to be installed first, as is typical with public cloud vendors to offer their own instead. ⁷.

Practical Exercise: Scaling and Updating

Now that we have our application running, let's try scaling it and updating to a new version.

Scaling the Deployment

To scale our deployment to 5 replicas:

kubectl scale deployment nginx-deployment --replicas=5

Verify the scaling:

kubectl get pods

You should now see 5 nginx pods running. This demonstrates how easy it is to scale applications in Kubernetes.

🚨 Troubleshooting Tip:
If some pods are stuck in "Pending" status, it might be due to resource constraints. Check the events for more information:
kubectl describe deployments
Look for events related to scheduling or resource allocation.

Updating the Application

Let's update Nginx to a newer version. Edit the nginx-deployment.yaml file and change the image version:

spec:
  containers:
  - name: nginx
    image: nginx:1.27

Apply the changes:

kubectl apply -f nginx-deployment.yaml

Watch the rollout status:

kubectl rollout status deployment/nginx-deployment

Kubernetes will gradually update the pods to the new version, ensuring zero downtime. This process, known as a rolling update (RollingUpdate is the default), replaces old Pods with new ones incrementally.

📌 Deep Dive for Intermediate Users:
Kubernetes offers fine-grained control over the update process through fields like .spec.strategy.rollingUpdate.maxSurge and .spec.strategy.rollingUpdate.maxUnavailable in the deployment spec. These allow you to control how many pods can be created above the desired number, and how many pods can be unavailable during the update process respectively (both default to 25%) ⁸.

Conclusion: Your Kubernetes Journey Begins

We've covered a lot of ground in this first foundational Kubernetes article; from understanding the basics of Kubernetes architecture to deploying and managing your first application using k3s.

Here's a quick recap:

Kubernetes architecture consists of a control plane and nodes, working together to manage containerized applications.
Core concepts like Pods, Nodes, Clusters, and Namespaces form the building blocks of any Kubernetes system.
K3s provides a lightweight, easy-to-install Kubernetes distribution perfect for learning and development.
Deployments in Kubernetes allow you to declaratively manage your applications.
Kubernetes makes it easy to scale and update your applications with simple commands.

What's Next?

In upcoming articles in this series, we'll dive deeper into:

Kubernetes Networking: Understanding Services, Ingress, and Network Policies
Persistent Storage in Kubernetes: Working with PersistentVolumes and StorageClasses
Kubernetes Security: Role-Based Access Control (RBAC) and Pod Security Policies
Advanced Workloads: Exploring StatefulSets, DaemonSets, and Jobs
Monitoring and Logging in Kubernetes: Setting up robust observability for your cluster

Each of these topics builds on the foundation we've laid in this article, allowing you to create more complex and robust applications on Kubernetes.

Test Your Knowledge

To reinforce what you've learned, try the following exercises:

Deploy a different application (e.g., a simple Node.js app) to your k3s cluster.
Rollback the Nginx deployment to the previous 1.26.1 version.
Create a service of type ClusterIP for your application and access it from within the cluster.
Set up a cronjob that prints "Hello, Kubernetes!" every minute.
Create a ConfigMap and use it to configure your application.

As you progress, you'll find that Kubernetes opens up a world of possibilities for deploying, scaling, and managing containerized applications. Starting out small and expanding, will open up more questions and with that a search for new answers that will expand your skillset.

Thanks for your time and let me know in the comments if you’re currently using Kubernetes or you’re planning to and on what project(s).

See you in the next one!

References

Cloud Native 2023: The Undisputed Infrastructure of Global Technology - https://www.cncf.io/reports/cncf-annual-survey-2023/ ↩
Developing Kubernetes Services at Airbnb Scale - https://www.infoq.com/presentations/airbnb-kubernetes-services/ ↩
Open Container Initiative - https://opencontainers.org/ ↩
K3s Documentation. "Architecture." - [https://docs.k3s.io/architecture](https://docs.k3s.io/architecture ↩
Kubernetes Documentation “Namespaces” - https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/ ↩
K3s “Using Docker as the Container Runtime” - https://docs.k3s.io/advanced#using-docker-as-the-container-runtime ↩
K3s “Service Load Balancer - https://docs.k3s.io/networking/networking-services#service-load-balancer ↩
Kubernetes Documentation “Deployments” - https://kubernetes.io/docs/concepts/workloads/controllers/deployment/ ↩

How to create a flexible Dev Environment with Vagrant and Docker

Rolf Streefkerk — Tue, 25 Jun 2024 12:11:51 +0000

Today we're discussing how you can create a fully automated, virtualized development environment. One that's customizable, and ready to use in minutes.

In this article, I'll show you how to set up such an environment using PHP and Laravel, though the principles can be applied to your preferred tech stack.

We'll dive into creating a robust setup powered by VirtualBox and Vagrant, featuring:

Apache Gateway
PHP with Apache for front-end
Laravel API
Redis Cache with Redis Commander
MySQL database with PHPMyAdmin

By the end of this guide, you'll have a secure, containerized environment accessible via specific routes, with core services in a private Docker network.

Furthermore, it’s very easy to connect Visual Studio Code to VirtualBox, and start your programming workflow.

Let's get started!

How does it all work?

TLDR: Skip to step 4. A - Installation, to get right into setting this up for your system.

There are 4 parts to this solution I’ll discuss today. I’ll finish with the Development workflow.

1, Apache as a Gateway (Reverse Proxy)
2. Docker with Docker compose
3. Vagrant with VirtualBox
4. Development workflow
- A - Installation
- B - Accessing the web application services
- C - Developing with VSCode
- D - Updating code with GitHub.
- E - (Extra) GitOps with GitHub Actions

If you like more in-depth articles on Apache, Docker, or Vagrant?

Let me know in the comments below, and Connect with me on Twitter.

1, Apache as a Gateway (Reverse Proxy)

Apache 2 is a HTTP server with a modular setup. An HTTP server in essence serves files according to the HTTP protocol of which there are two major versions currently used across the web, http v1.1 and http v2

Apache is typically used as an HTTP server to serve files, however in this case we’re using it in Proxy “mode” primarily and specifically as a Reverse Proxy (also known as a Gateway).

The reverse proxy acts like a regular web-server, and will decide where to send the requests and then returns the content as if it was the origin server.

Typical use-cases:

Load balancing. Is a topic for another day.
Provide access to servers behind a firewall. This is what we’re doing today.

ProxyPass "/foo" "http://foo.example.com/bar"
ProxyPassReverse "/foo" "http://foo.example.com/bar"

ProxyPass is a remote server mapping to the local path. This is considered in Apache as a “Worker”. This is an object that holds its own connections and configuration associated with that connection.

ProxyPassReverse ensures that the return headers generated from the backend are modified to point to the Gateway path instead of the origin server.

These are the basic 2 directives to create mappings that appear to orginate from the Reverse Proxy.

Now you need to map these to specific Locations.

<Location "/api/">
    ProxyPass http://api:80/
    ProxyPassReverse http://api:80/
</Location>

A Location directive operates on the URL or webpages that are generated only! Not file system paths.

In this example api is a named server in the docker-compose file that is exposed to port 80.

within a Docker network you can reference containers by their name

The api service needs to be accessible on the reverse proxy at the URL path /api/. So that means, you can access the API in your browser at: http://your-ip/api/

To make sure that the api and the other Locations are made available on port 80 we need to encapsulate the Location and ProxyPass, ProxyPassReverse in a VirtualHost directive like this:

<VirtualHost *:80>
    ServerName ${SERVER_NAME}
    ProxyPreserveHost On

    # Laravel Frontend app server on root path
    ProxyPass / http://frontend/
    ProxyPassReverse / http://frontend/

    <Location "/api/">
        ProxyPass http://api:80/
        ProxyPassReverse http://api:80/
    </Location>

    # ... other directives ....
</VirtualHost>

The VirtualHost is a grouping based on an ip-address or wildcard, match-all * and a port number. For that grouping you can override global directives such as Location and Directory.

ProxyPreserveHost On: This directive ensures that the original Host header is passed to the backend server, which can be important for applications that rely on the Host header for their logic.

2. Docker with Docker compose

Fun note about their logo, it’s a whale with shipping containers. Matching perfectly with the core idea of the product.

Docker is software that can package other software in a concept called a Docker Container. This container is portable across many different operating environments such as Linux and Windows.

Docker provides a way to isolate the container environment (the Guest machine) from the operating system (the Host machine).

This allows for many different kinds of software and operating environment to run within the containers without interfering with the host machine.

As seen in this diagram, Docker requires a lot of work from the Host Operating System. The benefit is that the containers can be small, since much of the code and the work is happening underneath the containers.

Docker compose

Docker compose is the “compositor” document standard to create a Docker network with one or more Docker containers.

Below is an example of a docker-compose yaml document:

for both the gateway and db service we configure:
- image: references an image that must be on DockerHub.
- ports: is an “outside:inside” mapping. Where outside means outside the Docker network (Your host machine or in this case, Virtual Machine)
- In this case we use the same port outside of the Docker network to access this service as inside the network.
- expose is similar to ports, except this port is not available outside the Docker network
- volumes: are mounts, again it’s an “outside:inside” mapping.
- ./apache-config is the git repo directory where our httpd.conf lives. That’s mapped straight to the Apache 2 configuration file inside the container.
- depends_on: this service needs to wait until these other named services are online.
- environment are environment variables available inside the container. Typically used to set passwords, ports, and username configurations.

services:
  gateway:
    image: httpd:2.4
    container_name: gateway
    ports:
      - "80:80"
    volumes:
      - ./apache-config/httpd.conf:/usr/local/apache2/conf/httpd.conf
    depends_on:
      - frontend
      - api
      - redis
      - redis-ui
      - phpmyadmin
    environment:
      SERVER_NAME: ${APACHE_SERVER_NAME}

  db:
    image: mysql:5.7
    container_name: db
    expose:
      - "3306"
    environment:
      MYSQL_ROOT_PASSWORD: ${MYSQL_ROOT_PASSWORD}
      MYSQL_DATABASE: ${MYSQL_DATABASE}
      MYSQL_USER: ${MYSQL_USER}
      MYSQL_PASSWORD: ${MYSQL_PASSWORD}
    volumes:
      - db-data:/var/lib/mysql

  # ... other services here

volumes:
  db-data:

To run this file we can use: docker-compose up --build -d
This will: - bring the containers online up - build the images --build - run it in the backgroud -d

To take all of the containers offline, we simply type; docker-compose down

To manage containers easily from the CLI, I use dry, it provides an easier way to see statistics on all containers (memory and cpu usage) as well as a log viewer that can be searched, and much more.

Summary of useful Docker commands:

docker ps show all running Docker containers.
docker-compose up --build -d: build the Docker images, bring the containers online and run them in the background.
docker-compose down bring the containers offline.
docker-compose stop stop the containers.
docker-compose restart <name> restart the specified Docker container.
dry run the Docker manager and monitoring / logs command line utility
- F2 show all containers (stopped and running)
- select container enter > feth logs > enter > f to tail the logs.
- select container enter > fetch logs > 30m > f show logs from the last 30 minutes and tail.

3. Vagrant with VirtualBox

Vagrant can create complete development environments on Virtual Machine (VM) in the cloud and on your local machine with a simple workflow.

Vagrant provides consistent environments such that code works regardless of what kind of systems your team members use for their development, or creative, work.

Vagrant is in my opinion ideal to setup different development environments that require different dependencies, really quickly. Additionally, because it’s all in code, you can easily adapt the environments to match evolving requirements for your use cases.

For this solution we use a base image generic/ubuntu2204 to build the Virtual Machine with using a Vagrantfile.

A Vagrantfile is, similar to a Dockerfile, because it provides instructions for the Vagrant program how to create the virtual image (using the base image box and provisioners) and what to use to run it on (a provider, in this case VirtualBox).

A quick run down of what this all means using an example (shortened version):

Vagrant.configure("2") denotes the version of Vagrant that we’re using; 2.
config.vm.box = "generic/ubuntu2204" the box specification running Ubuntu 22.04.
config.vm.provider "virtualbox" we use VirtualBox to run the VM.
config.vm.provision "shell", inline: we use bash scripts to install our environment.
config.vm.synced_folder: enables the VirtualBox built in folder sharing, requires VBox Guest Editions to work.
config.vm.provider "virtualbox" has VirtualBox specific configurations such as the cpu memory and name of the virtual machine.

# -*- mode: ruby -*-
# vi: set ft=ruby :

Vagrant.configure("2") do |config|

  # Variables
  git_user_email = 'youremail@mail.com'

  # boxes at https://vagrantcloud.com/search.
  config.vm.define "docker-apache" do |dockerApache|
    config.vm.box = "generic/ubuntu2204"

    # via 127.0.0.1 to disable public access
    config.vm.network "forwarded_port", guest: 80, host: 80
    config.vm.network "public_network"

    # Share an additional folder to the guest VM.
    config.vm.synced_folder "./data", "/vagrant_data"

    config.vm.provider "virtualbox" do |vb|
    # Display the VirtualBox GUI when booting the machine
    # vb.gui = true

    # Customize the amount of memory on the VM:
      vb.memory = "8192"
      vb.name = "docker-apache"
      vb.cpus = 6
    end

    config.vm.provision "shell", inline: <<-SHELL
      sudo apt-get update
      sudo apt-get install -y apt-transport-https ca-certificates curl

      su - vagrant << EOF
        # clone repo
        mkdir -p /home/vagrant/docker-apache
        cd /home/vagrant/docker-apache
        git clone https://github.com/rpstreef/docker-apache-reverse-proxy .
     EOF
    SHELL
  end
end

To create a VM out of a Vagrantfile we can simply run vagrant up, this will start the process of download the box and then the provisioning step with the bash scripts to the provider VirtualBox.

When it’s all finished, connect using vagrant ssh and start using it!.

Summary of useful Vagrant commands

vagrant up this will create the Virtual Machine.
vagrant validate used to verify the Vagrantfile is semantically correct.
vagrant halt to stop the VM, use --force to shut it down immediately.
vagrant ssh connects to the VM via the command-line.
vagrant destroy completely remove the Virtual Machine.

4. Development workflow

A - Installation

Now that it’s clear how the solution parts work, let’s go and install it:

Git clone https://github.com/rpstreef/flexible-dev-environment in your local projects directory.
Install Vagrant using these instructions
Install VirtualBox using these instructions.
Edit the Vagrantfile :
1. You want to use GitHub on the Virtual Machine?\
  1. Change the git_user_email and git_user_name values.
  2. There’s an additional step to complete after the VM is installed. See How to setup your Personal Access Token (PAT) for GitHub:.
2. Check machine settings in this block; config.vm.provider "virtualbox". Adjust vb.memory = "8192" and vb.cpus = 6 the number of processors.
From the Git folder, run vagrant up, this will setup the Virtual Machine with VirtualBox
- When asked which network, choose the adapter with internet access.
Connect to the Virtual Machine, run vagrant ssh.
- Take note of the ip address and use that to connect to with your browser. Or on the CLI type: ip address and look for the network adapter name you chose earlier.
  1. For the Web Landing-page: http://virtual-machine-ip/
  2. For the Redis Commander UI http://virtual-machine-ip/cache
  3. For PHPMyAdmin: http://virtual-machine-ip/phpmyadmin
  4. For Laravel API: http://virtual-machine-ip/api
When connected to the VM:
- Execute dry on the command line and you should see several containers running.
- Refer to the summary of useful Docker commands chapter for more guidance with Docker commands.
  - and this summary for Vagrant commands.

B - Accessing the web application services

The Apache Gateway provides access via port 80 with your browser to only the parts that need to be exposed to the outside:

Front-end → http//ip-address/
Redis Commander → http//ip-address/cache/
Laravel API → http//ip-address/api/
PHPMyAdmin → http//ip-address/phpmyadmin/

That means that the Redis and MySQL services are not accessible directly from outside the docker network (private network).

C - Developing with VSCode

1. Configure SSH Access

Configure the ssh access on your machine with vagrant is really easy to do, just run vagrant ssh-config, copy paste the text into your ~/.ssh/config file.

From the command line, anywhere, you can connect to your VM with ssh docker-apache

To get the IP address from your VM, run ip address and check which adapter you used for the network access, and take note.

2. Connect to VirtualBox with VSCode

When this works, we can connect VSCode:

Open up VSCode
Click on the lower left icon, Connect current window to Host then enter docker-apache. This name was retrieved from the configuration we did in step 1.
This will install the VSCode server files on the virtual machine.
Open the directory /home/vagrant/docker-apache
Yes I trust the authors, when asked.

To get GitHub to work, we just need to add our Personal Access Token in the next step.

D - Updating code with GitHub.

To setup the Personal Access Token for GitHub, do the following:

Create a PAT here: https://github.com/settings/tokens
For most cases, repositories only is sufficient (unless you want to activate GitHub Actions?):
- Check the repo checkbox
Execute the below on the command-line to set your Personal Access Token for GitHub:

git credential-store store <<EOF
protocol=https
host=github.com # Replace with your Git provider's hostname
username=<your-username> # Replace with your Git username
password=<personal-access-token>
EOF

these are all stored in cat ~/.git-credentials

E - (Extra) GitOps with GitHub Actions

With GitHub Actions you can automate, for example;

docker-compose deployment to your favorite public cloud
convert docker-compose to Kubernetes and then deploy to your cluster.
or perform other automation as per the standard workflows offered by GitHub, to get started:
1. Fork my repository: https://github.com/rpstreef/flexible-dev-environment
2. Then go to Actions, there’s a list presented of all kinds of ready made automations.

If you’d like more details on how to automatically deploy using GitHub Actions (GitOps), give me a shout on Twitter or in the comments.

Conclusion

Setting up a virtualized development environment might seem daunting at first, but the benefits are well worth the effort. With this setup, you've gained a powerful, flexible, and secure platform for your development work.

I'm curious to hear about your experiences down in the comments:

How does this compare to your current development workflow?
Do you see yourself adopting a similar setup, or have you already implemented something like this?
What other tools or services would you add to enhance this environment?

If you found this guide helpful, consider following me on Twitter for more tech tips and discussions.

For IT professionals looking to balance career growth with personal well-being, I invite you to join our community, The Health & IT Insider. We cover a range of topics from DevOps and software development to maintaining a healthy lifestyle in the tech industry.

Thanks for reading, and see you in the next one!

How to create your own AI chatbot with LangFlow

Rolf Streefkerk — Tue, 11 Jun 2024 09:26:14 +0000

Today I’d like to start my series of articles on using the LangChain open source tool-set.

LangChain is one of the most used frameworks to create AI agents with, if not THE most used. With nearly 90.000 stars and almost 14000 forks on GitHub at the time of this article, we can safely say it is popular.

Reason enough for me to start covering it. To kick things off with, let’s answer a few essential questions.

What we're covering today:

Introduction
- What is a RAG AI Agent?
- What is LangChain and LangFlow?
How do I get started?
- How to install LangFlow on your own machine
How to create our first RAG AI with LangFlow
- Part 1: Processing the web-page data
- Part 2: Let’s chat with your RAG
Experiment and Test
Conclusion

Introduction

What is a RAG AI Agent?

Retrieval Augmented Generation (RAG) is a technique to enhance the accuracy and reliability of AI with facts retrieved from other sources.

Originally the RAG name was coined in this paper, the authors stated:

RAG could be employed in a wide variety of scenarios with direct benefit to society, for example by endowing it with a medical index and asking it open-domain questions on that topic, or by helping people be more effective at their jobs

Essentially you can create your own set of sources that you have vetted for accuracy and fit for purpose. Creating applications with your specific needs in mind, and without potential bias or opinion from the original AI.

RAG’s are a great way to provide business solutions that can run locally with open source AI models such as Llama, or large cloud scale AI’s such as OpenAI.

What is LangChain and LangFlow?

LangChain is an open source Python library that can integrate with AI models using API’s and functions. With LangChain you can create natural language interactions with an AI using concepts such as Chains, and Prompts to weave together a custom RAG AI.

LangFlow is a GUI that can diagram a LangChain flow to experiment and prototype an AI solution with quickly. It offers drag and drop components and a chat box to immediately start interacting. This simplifies the initial development process of iteration and experimentation.

How do I get started?

To get started with LangFlow creating your AI agent there are 3 options you can choose from:

SAAS Installation using Huggingface spaces:
- Private: Huggingface account, then duplicate LangFlow Spaces , leave everything on default, click duplicate space.
  - This will take a bit of time (30+ minutes, not included in the 5 minutes 😜)
    - In the mean time head over to the Spaces settings > Variables and secrets > New secret
    - Name: OPENAI_API_KEY, Value: your OpenAI key we just copied.
  - Go drink a coffee, walk outside, do other things while the Huggingface space is created.
  - Then, you can skip to this chapter.
- Public: access the Public LangFlow service, ready to go but everyone can read it!
  - Do not use your OpenAI API keys here!
  - Then, you can skip to this chapter.
A Google Cloud installation. If preferred, go here.
- Then you can skip to this chapter.
Or, run it on your own machine using VirtualBox.
- This is my preferred method of choice, it’s low cost and you have full control. Follow along in the next chapter to find out how to quickly set this up.

How to install LangFlow on your own machine

Install Vagrant and VirtualBox:
- Download and install Vagrant.
- Download and install VirtualBox.

2) Install via my LangFlow repo:

git clone https://github.com/rpstreef/langflow-setup
Within the cloned repo; execute; vagrant up to install the LangFlow VM with docker and docker-compose.
- Type make for all the commands available.
Optionally:
- Run: make vm to get access to the Virtual Machine.
  - Note if you do not have VBoxManage on your command-line, this will not work.
  - Sometimes you could get a message like this; ssh: connect to host 192.168.1.39 port 22: No route to host.
    - Retry the connection in that case.
- Run: dry to see all the containers running
open your browser and navigate to; http://localhost:7860/flows

Now we can start creating a chat bot

To use the store, you need to register and get an API key here

How to create our first RAG AI with LangFlow

This is what we’re going to build today.

To start with, what is the aim of this particular RAG? What do we want it to do?

A chat bot that can answer detailed questions about any web-page we choose.

A perfect use case for this is article analysis via questions and answers. You can formulate the questions such that the bot can provide additional explanation on harder to understand concepts.

I’m sure there are other uses cases like this, let me know on Twitter what you would use it for.

Basically there are two parts to building this RAG chat-bot:

We need to retrieve the information we want to ask questions about,
then make that information available to the OpenAI chat bot.

Part 1: Processing the web-page data

To start of with, we need to store website text, as a vector, in a database such that we can use it later to ask questions about.

I’ve added links to the official LangChain documentation for each of the concepts / parts discussed.

WebBaseLoader: Enter the Web Page we want to analyze.
OpenAIEmbeddings: This will “embed” the information read from the website. We use OpenAI’s text-embedding-3-small Model to do this. Which should be ample for most use-cases. This creates a vector representation of the documents we’re about to create. The TikToken tokenizer to make it more accurate for OpenAI to split the text.
CharacterTextSplitter: This part will cut up the text into chunks with a bit of overlap, and outputs these parts as documents to FAISS the vector database we’re using in this scenario.
FAISS: Is the Facebook AI Similarity Search (FAISS) vector database that can store and search vectorized documents in memory.

Part 2: Let’s chat with your RAG

Now that we have the basic information retrieval and vector storage created. We can start adding a Chat AI and chaining the documents we’ve created with a Retriever (Search via a question and retrieve documents using a prompt via the chat box.

ChatOpenAI: We use the Chat feature offered by OpenAI and set the model to gpt-3.5-turbo-0125 to save on cost. Additionally the temperature is set to 0.2 to inform the AI we need answers with high probability and not too much creativity (> 0.5 - 1.0)
CombineDocsChain: We use chain type stuff, which basically means the context is the document. For efficiency you could change this to map_reduce, but it would mean we need to complicate the Prompt parts to combine the documents with.
RetrievalQAWithSourcesChain: Chains together and creates the Context from the user question, and the documents retrieved from the Vector database.
Let’s chat: Now we can ask what our webpage is about:

Experiment and Test

This is just a starting point, and a working example of how quickly a RAG can be created for a specific purpose. A few things to consider that can optimize the solution;

Experiment with the token sizes of the OpenAIEmbedding and ChatOpenAI.
Changing the chain type of the CombineDocsChain, to map_reduce.
Improving the website text data quality by reducing the token size of the source text (lemmatization and stemming).

Data input quality has a big impact on RAG output quality.

Conclusion

In conclusion, let’s summarize what’s been covered and learned:

Use LangFlow to quickly create a chat-bot that can answer questions on the content of your choice.
LangFlow consists of many components of the LangChain framework. Understanding these components, can open up new possibilities to automate with LangFlow.
LangFlow is a playground to experiment with AI. To prototype an idea quickly, then build it with LangChain.
(bonus) Huggingface spaces host a variety of AI generative applications such as image generation with Stable diffusion or Midjourney, and GPT4o chat. Check it out!

For the next article I’d like to dive deeper into creating a RAG agent using LangGraph, and deploy that to a Docker container. Stay tuned if you’re interested.

For now, lets discuss in the comments below:

What have you build?
What can be improved in this example and why?

Let's continue the discussion on Twitter

Appreciate your time and till the next one!

How to deploy your own website on AWS

Rolf Streefkerk — Tue, 28 May 2024 03:36:21 +0000

Originally published on rolfstreefkerk.com

Take full control of your website, and following along with our how-to guide.

Benefits of building and deploying a website from scratch:

Own the code and control it as you see fit
Learn AWS and how to deploy a website to AWS S3
Understand DNS and Route53
How to use DevOps to solve automation issues

Read on to get started.

Follow me on Twitter to keep updated on the latest articles on AWS and more.

You will need the following to get started

a static site, I recommend one of these frameworks (and I've used):
- Hugo
  - existing themes will get you a website quick, such that you only have to modify color schemes and layouts.
- or Astro; if you’d like to integrate React, VueJS etc. code as well.
  - use their themes page here to get a starting point.
an AWS account, which requires a credit card to setup.
a domain, wherever you registered it.
- In this how-to I use Porkbun as my favorite registrar.
a computer with;
- Terraform/OpenTofu installed. We use Terraform in this article.
- AWS CLI installed with profile configured you want to use for your website deployment.
- Git command line tooling.
- your code editor of choice, I use VSCode.
a GitHub account so you can fork my example repository.
(optional) email inbox provider, I use Migadu. ## What are we creating today?

We are creating the following services and configurations:

AWS S3 bucket to send your website source files to;
AWS CloudFront distribution that will cache, optimize website delivery globally to your audience.
AWS Route53 for your;
- Email service records with DNSSec configuration,
- You can then hookup a newsletter service like ConvertKit.com
- Name Server Configuration for your domain; yourwebsite.com
- and the CloudFront distribution to optimize your website hosting.
GitHub Actions for a CI/CD pipeline, deploying your website on command within a minute.

Setup your Domain on AWS

Go to Route53, after you’ve logged in, and navigate to Hosted zones.
Create your hosted zone and enter your website domain; yourwebsite.com
Make a note of the Hosted zone ID, We’ll use that in the next step for Terraform to automate all the Route53 records to the correct Domain name.

If you choose to automate it using Terraform;

export the Name Servers from your domain registrar (Porkbun, etc.).
add the hosted zone resource configuration into my example Terraform module and hook it up to all the related resources requiring the Hosted zone id.

(Optional) Email hosting

If you like to setup an email hosting solution, I use migadu.com, keep the Route53 website open.

We’ll import additional configuration text blocks into Route53 to make your domain work with the inbox service.

In Mail inbox service, there is a DNS Configuration panel.
Get the BIND records output, copy/paste the text of all the DNS records.

If you require automatic mail server discovery for your Email;
Check for these strings in the provided DNS records; _autodiscover or autoconfig

Then in AWS Route53, for your hosted zone; Import zone file, and copy paste the lines of text in that dialog box.

Now you can add your new email inbox in your mail apps.

If you have _autodiscover and / or autoconfig DNS records included, you can;

go to your email app,
add a new inbox using; email and password.
Finished, inbox added without further configuration required.

Otherwise, take a note of your mail inbox service SMTP and IMAP server configurations.

Automating your AWS account setup with Terraform

Now that we have the Domain in place, and the Mail inbox (optional), we can configure the actual site deployment.

Create a new project by Forking: https://github.com/rpstreef/terraform-yourwebsite.com

This is a template that will use Terraform modules from another Git repository; https://github.com/rpstreef/terraform-static-site

What does this template create?

This template will create the following set of resources;

S3 bucket for Terraform state
S3 bucket for yourwebsite.com
- S3 CORS configuration for ConvertKit.com , this will allow CORS between ConvertKit JavaScript and your domain without warnings.
ACM Certificate for SSL, *.yourwebsite.com, and the ACM validation records for Route53 for auto-renewal of SSL.
Route53 A, and AAAA records (IPv6)
Route53 DNSSec,
- only the first step! The second step must be done manually with your Domain Registrar.
Lambda function for redirects to index, ensures you have nice URL’s.
- E.g. https://yourwebsite.com/contact instead of https://yourwebsite.com/contact/index.html
CloudFront for caching, and web-page speed optimization, and SSL secured.

How to adjust the template?

To make the template fit for your website.

Do the following

Change these lines in the terraform.tfvars file :
- where you read yourdomain.com,
- and your hosted_zone_id for yourdomain.com.
- check 404 response at the bottom of the file to see if that matches up with your website structure. Additionally HTTP response codes can be added as blocks; {}.

If you need additionally CORS settings, add an extra rule in the same way as f.convertkit.com.

# General
environment = "prod"
region = "us-east-1"
project = "yourdomain.com"

# use tags to track your spend on AWS, seperate by 'product' for instance.
tags = {
    environment = "production"
    terraform = true
    product = "yourdomain.com"
}

# Which config line used in .aws/config
aws_profile = "yourdomain-profile"

# Route53
hosted_zone_id = "Z000000000"

# www.yourdomain.com
product_name = "yourdomain" # avoid to use `.`, this cause an error.
bucket_name = "yourdomain.com" # your site is deployed here.

# S3 bucket CORS settings:
bucket_cors = {
    rule1 = {
        allowed_headers = ["*"]
        allowed_methods = ["GET", "PUT", "POST"]
        allowed_origins = ["https://f.convertkit.com"]
        expose_headers = ["ETag"]
        max_age_seconds = 3000
    }
}

domain_names = ["yourdomain.com", "www.yourdomain.com"]

custom_error_responses = [{
    error_code = 404
    error_caching_min_ttl = 10
    response_code = 200
    response_page_path = "/404.html"
}]

Make sure the configuration in project-state.tf file is correct;
- check the bucket name,
- and the AWS profile name used, e.g. yourwebsite-profile.

locals {
    projects_state_bucket_name = "tfstate-yourwebsite.com"
}

provider "aws" {
    region = "us-east-1"
    profile = "yourwebsite-profile"
}

terraform {
    # First we need a local state
    backend "local" {
    }

    # After terraform apply, switch to remote S3 terraform state
    /*backend "s3" {
        bucket = "tfstate-yourwebsite"
        key = "terraform.tfstate"
        region = "us-east-1"
        profile = "yourwebsite-profile"

        encrypt = true
        acl = "private"
    }*/
}

If all the configuration checks out;
- run terraform init, this will download the dependent modules.
- then; terraform apply > yes
When it’s finished deploying, make note of the variables in the output. We’ll need them later on. To retrieve these later, type; terraform output in the ./environments/production directory.

Which one came first? The chicken or the egg?

When finished, we need to adjust the project-state.tf file:
- Place the backend "local" block in comments.
- Remove the comments from the backend "s3" block.
- Migrate the state from local to S3:
  - terraform init -migrate-state
  - type: yes to copy state from local to s3.

Now it’s fully deployed and we have saved our Terraform state to AWS S3, it’s no longer on your disk. You can remove those tfstate files if you like.

Establishing DNSSec “Chain of Trust”

The benefit of DNSSec is the establishment of the “chain of trust”.

That means, it is verified that;

You own the domain,
when you navigate to that domain, the information is coming from your servers and not from someone else’s server (e.g. hackers etc.)

If you’d like to learn more about DNSSec, this article is a good primer

Now to finalize DNSSec configuration, you will have to manually modify the Domain registrar information.

First, get the required DS records for DNSSec; View information to create DS record

Then, in the next screen click; Establish a Chain of Trust.

You will see a table outlining configuration items.

If you did not register your domain on Route53, click Another Domain registrar

On Porkbun, my domain registrar, the screen looks like this:

Enter the following at the dsData block; on the left is the Porkbun input field name, on the right as the value I will place the name used at Route53:
- Key Tag: Key tag
- DS Data Algorithm: Signing algorithm type
- Digest Type: Digest algorithm type
- Digest: Digest

If you have a different registrar, you’ll need to review their documentation, it may be slightly different.

How to check your configuration works?

Finally, use this online tool; https://dnssec-debugger.verisignlabs.com/ to check your domain, if you’re getting all green check-marks.

If they’re all green, It means your chain of trust has been successfully established!

Now we have a DNSSec secured domain configuration with an S3 static hosted site via CloudFront with SSL.

Performant
Cheap
and Secure.

Upload your website

We can use a local deployment setup with the AWS CLI, or via GitHub Actions.

Local deployment with a script

Depending on your system (Linux, Windows, Mac), you may need to alter this script.

On Linux, we can automate your website deployment as follows using this bash script:

#! /bin/bash

npm run build

aws s3 sync dist s3://yourwebsite.com --profile yourwebsite-profile

aws cloudfront create-invalidation --distribution-id <CloudFront Distr. Id> --paths "/*" --profile yourwebsite-profile

Make sure to;

replace npm run build for the script that generates your static website build.
replace dist in the aws s3 sync dist, if your website build is in another folder.
replace <CloudFront Distr. Id> with your CloudFront distribution id.
- you can find it in the outputs after terraform apply has finished; cloudfront_distribution_id

GitHub Actions

If you like to use automation instead, it’s very easy and cheap to setup.

What does this cost anyway?

Plan	Storage	Minutes (per month)
GitHub Free	500 MB	2,000
GitHub Pro	1 GB	3,000

You can deploy quite a few times before you hit the Pro ceiling in terms of Minutes per month:

The storage size is based on your repository size which, for most, will be very hard to reach.

Operating system	Minute multiplier
Linux	1
Windows	2

We choose a Linux build environment, specifically ubuntu-latest, to get the most out of our free minutes.

Check out more about GitHub Action pricing here.

How does it work?

To deploy using GitHub Actions, do the following:

First, create a new file in your website’s GitHub repository at .github/workflows/deploy-on-comment.yml.
Add the following code to the file:

Note; I’m assuming your website is Node (v20) based. Adapt where needed!

name: Deploy on Comment
on:
  issue_comment:
    types: [created, edited]
  push:
    branches:
      - main

jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout code
        uses: actions/checkout@v3

      - name: Set up Node.js
        uses: actions/setup-node@v3
        with:
          node-version: '20'

      - name: Install dependencies
        run: npm install

      - name: Build website
        run: npm run build

      - name: Configure AWS Credentials
        uses: aws-actions/configure-aws-credentials@v1
        with:
          aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
          aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
          aws-region: us-east-1

      - name: Sync build output with S3 bucket
        run: aws s3 sync ./dist s3://your-s3-bucket-name

      - name: Invalidate CloudFront cache
        run: aws cloudfront create-invalidation --distribution-id ${{ secrets.CLOUDFRONT_DISTRIBUTION_ID }} --paths "/*"

There’s several secret variables that need to be created on GitHub, coming from the Terraform output we received earlier:

AWS_ACCESS_KEY_ID:
AWS_SECRET_ACCESS_KEY:
CLOUDFRONT_DISTRIBUTION_ID
If you need to look up what these are again, navigate to your terraform-yourwebsite.com git repository and then;
- cd ./environments/production
- terraform output
Input them at the following location in GitHub:

You can now create an issue that details the updates on your website for example.
For each comment that is added, the deployment will start.
You can follow the deployment steps taken and the logs in the Actions tab.

(Optional) In case you’d want to change the GitHub Actions to use a Pull request instead, you can modify that in the deploy script. > For more alternative triggers, check out the GitHub Actions documentation.

Your website is online!

Now when you go to your web URL; yourwebsite.com, everything should be up and running.

What we have build;

(Optional) Email hosting with Migadu (or choose any that you have); e.g. hello@yourwebsite.com
- You can connect this to your ConvertKit.com mailing list for example.
Your own personal domain that is DNSSec secured.
- You’ll be certain no hackers can hi-jack your domain.
Your static Website on [[AWS]] using AWS S3.
- Free web-hosting!
CloudFront Content Delivery Network (CDN), enabling:
- SSL protected website. Form submits are all encrypted by default.
- Increased performance in load speeds, latency across the globe.
- URL rewrites for static websites. No index.html will be displayed when navigating.
- and redirects for 404 not found pages. Your visitors will see the 404.html page instead of an error text message.

Questions? Let's discuss!

What do you struggle with on AWS?
Did you have issues with deploying on AWS?
How would you do it?

Let me know down in the comments or on Twitter

Appreciate your time and till the next one!

Deploy with Eezze

Rolf Streefkerk — Tue, 16 Apr 2024 12:33:17 +0000

In this showcase of Eezze, we'll cover how we save time with our deployment automations:

1. The Deployment Headache
2. Seamless Deployment
3. Tutorial
4. Frequently Asked Questions
5. Beyond the Basics
6. Conclusion

1. The Deployment Headache

In the realm of software development, deploying backend applications often presents a significant challenge. Developers find themselves grappling with the intricate tasks of configuring servers, ensuring dependencies are correctly installed, and maintaining complex deployment scripts. These essential yet time-consuming processes can divert valuable resources away from core development efforts, impeding productivity and slowing down the overall software delivery pipeline.

At Eezze, we understand these challenges intimately. Our platform leverages industry-standard tooling such as Ansible for configuration management, Terraform for infrastructure provisioning, and Docker containers for consistent application environments. By automating and streamlining the deployment process, Eezze eliminates the hassle, allowing teams to concentrate on building exceptional solutions.

2. Seamless Deployment

With Eezze, we aim to transform the deployment experience, making it seamless and effortless. In this article, we'll walk you through the process of deploying backend applications using Eezze's platform, demonstrating how our solution automates complex tasks and ensures a smooth, hassle-free workflow.

3. Tutorial

Steps:

Set up a Connection in Eezze, currently we support Ubuntu 22.04 Server and AWS Cloud deployments.

For a Ubuntu 22.04 Server, we define the server host configuration in the Connection Info tab, and then in the Authentication tab we configure the SSH connectivity (PEM Key) and User/Password for SuperUser access.

Create your datasources, which are basically servers internally (linked to a Connection) or externally hosted servers and services;
Configure a REST Server, for example; the host has already been filled in, since this is an internal service (it can be customized) Additionally a health check port is auto-configured to ensure keep-alive in Application Load Balancer scenario's functions properly:
Entity definitions for your Database Datasource will be automatically applied as migrations at the docker-compose REST server entrypoint, enabling up-to-date synchronization of your data model with your database(s):
Environment Variables and Vault (protected) variables are automatically applied to your containers for run-time use:
Add your Services in our Service Group > Services editor.
Deploy to your empty Ubuntu 22.04 Server instance or a Connection with AWS Account credentials configured in the Authentication section in step (1).

We offer these public, and private cloud/server compatible solutions:

Ubuntu 22.04: we use an Ansible playbook to install a complete environment that is ready to be used for Docker container deployment, then build and deploy the containers from a Github repository.
AWS ECS: we've build a custom pipeline that interprets your project configuration, applies that our Terraform script that can interpret our project configution exactly, and deploy all Docker Containers via AWS CodePipeline/CodeBuild to ECS Fargate. These Containers are load balanced with AWS ALB (Application Load Balancer) which terminates the SSL and enables custom domains to be easily linked using AWS Route53 such as; https://api.yourapp.com.

4. Frequently Asked Questions

Q: How does your company use Eezze for client projects?

A: Eezze is our proprietary in-house tooling that streamlines the development and deployment of backend solutions. While the Eezze platform itself is not directly accessible to clients, we leverage its capabilities to generate the necessary artifacts for your projects, such as Git repositories, Docker files, Ansible playbooks, and source code.

Q: What kind of artifacts and deliverables can we expect from your use of Eezze?

A: By utilizing Eezze, we can provide you with a comprehensive set of artifacts tailored to your project requirements. These may include a fully configured Git repository with your application's source code (Typescript), Docker containers for consistent and isolated environments, Ansible playbooks for automated deployments, and any other necessary configuration files or scripts.

Q: Can you handle deploying our application to our existing infrastructure?

A: Absolutely. In addition to providing the generated artifacts, we can also leverage our Eezze automations to deploy your application directly to your existing infrastructure or environment. Our team can work closely with you to understand your infrastructure requirements and ensure a seamless deployment process, whether it's on-premises or in the cloud.

Q: Can we customize or modify the generated artifacts from Eezze?

A: Absolutely. While the Eezze platform itself is proprietary, the generated artifacts, such as source code and configuration files, are fully accessible and customizable by your team. The only proprietary code is the Eezze plugin, which is used to enhance the capability of our code automations. We encourage collaboration and can work closely with you to make any necessary modifications or enhancements to the deliverables.

Q: How do you handle updates and maintenance for the generated artifacts and deployed applications?

A: As part of our ongoing support and maintenance, we can provide regular updates and enhancements to the generated artifacts and deployed applications. This may include bug fixes, security patches, or new features based on your evolving project requirements. Our team will work closely with you to ensure a seamless update and deployment process.

Q: Can we integrate the generated artifacts and deployed applications with our existing infrastructure and tooling?

A: Yes, the artifacts generated by Eezze and the deployed applications are designed to be compatible and easily integrated with a wide range of infrastructure and tooling. Whether you have existing deployment pipelines, monitoring systems, or other tools in place, we can ensure a smooth integration process, minimizing disruptions to your existing workflows.

5. Beyond the Basics

Eezze's advanced internal tooling brings significant benefits to customers, particularly in the areas of deployment automation and speed.

AWS Deployment Excellence

Eezze's AWS ECS deployment automation integrates a custom pipeline and Terraform scripting for precise interpretation of project configurations. Key features include:

Custom Pipeline & Terraform: Interprets project configurations accurately for Docker Container deployment on ECS Fargate.
AWS CodePipeline/CodeBuild: Automates deployment workflows for faster time-to-market.
AWS ALB Load Balancing: Ensures scalability and reliability with SSL termination and custom domain integration using AWS Route53.

This streamlined deployment approach optimizes efficiency, accelerates deployment cycles, and enhances the security and scalability of your applications on AWS ECS.

Global Reach with Ubuntu 22.04

Eezze's automation extends globally to servers running Ubuntu 22.04, delivering rapid deployment benefits to customers. By automating the setup of all required software for a Docker container platform, Eezze accelerates deployment speed and ensures consistency across customer environments, leading to increased efficiency and reduced time-to-deployment for customer projects.

6. Conclusion

Eezze's internal tooling directly benefits customers by:

Accelerating deployment timelines through advanced automation capabilities.
Ensuring seamless integration with customers' existing infrastructure, minimizing disruptions and maximizing operational efficiency.
Providing access to cutting-edge deployment solutions, such as AWS integration and global deployment automation, resulting in faster time-to-market and increased competitiveness for customer solutions.

Ready to experience the speed and efficiency benefits of Eezze's internal tooling for your deployments? Request a personalized demo or quote today and discover how Eezze can transform your deployment experiences and deliver superior results for your projects.

Build Real-Time Apps with Eezze

Rolf Streefkerk — Tue, 02 Apr 2024 03:38:17 +0000

Build Real-Time Apps with Eezze

Here's a quick overview of what we'll cover in this showcase of Eezze's Websocket functionality:

Introduction
Streamlining Websocket Integration with Eezze's UI-Based Approach
- The Power of Actions
- Harnessing the 6 Directives
- Building Blocks to Backend
Practical Showcase: Eezze in Action
- 1. Dynamic Template Rendering
- 2. Stream Logging Data
- Generated Code (For Advanced Understanding)
Unlocking Real-Time Potential
Conclusion

Introduction

Frustrated by the complexities of implementing real-time features in your applications? Eezze offers a streamlined solution, empowering developers to build responsive, data-driven apps without diving deep into websocket protocols and synchronization code.

Eezze's low-code approach doesn't sacrifice control. Our platform recognizes that events are the lifeblood of any interactive application. Our UI lets you precisely configure how your app reacts to user interactions, data updates, and connection changes.

At its core, Eezze provides a set of directives for event handling. Imagine setting up a collaborative workspace feature with a simple configuration like "on message update, broadcast to channel subscribers."

In this article, we'll explore Eezze's event-driven workflow and build a real-time backend. You'll experience how Eezze's UI-based logic accelerates development of features that traditionally require extensive coding.

Streamlining Websocket Integration with Eezze's UI-Based Approach

Configure a websocket connection by specifying the server endpoint within a dedicated interface. You'll then define individual websocket event types, configuring custom "Actions" for each of the 6 supported directives.

The Power of Actions

Actions form the backbone of service development in Eezze, extending to websockets, REST APIs, and scheduled tasks. These sequential blocks handle data input, processing, and output. Within an Action block, you can:

Query databases
Manipulate files
Manage email communication
Generate output using the Twig templating language
Integrate with third-party services via REST or Websocket APIs

Harnessing the 6 Directives

Eezze puts control in your hands with these directives for building event-driven websocket backends:

on (event specification): Define the precise trigger for an action within your application (e.g., "on new message").
broadcast (send to everyone): Distribute data to all connected users when the associated "on" event occurs.
channel (send to only those that meet channel criteria): Target communication to specific user groups within your application.
notify (notification to a specific connection or user): Send targeted notifications to individuals.
onConnect: Execute actions when a user establishes a connection.
onDisconnect: Define responses triggered when a user disconnects.

Building Blocks to Backend

Each directive, combined with custom input definitions, Actions, and output configurations, acts as a modular building block. This allows you to assemble a robust websocket backend tailored to your application's specific needs.

Practical Showcase: Eezze in Action

To demonstrate the power of Eezze's websocket integration, let's explore two use cases that we have used websockets for in our upcoming app developed with Eezze:

Dynamic Template Rendering

Goal: Update a user profile page in real-time whenever the profile data is modified.

Steps:

Create Service Group: Establish a Service Group connected to your Websocket Datasource (Websocket Container Service configuration).
Configure 'On' Service: Within the group, create a new websocket service using the "On" event type triggered by a "render-profile" event.
Define Actions: Set up the following Actions within the service:
- Action 1: Retrieve the updated profile data from your database ("Get One" from Profile table, Check On parameters 'copied' from the Action Chain Input).
- Action 2: Render the profile HTML using a stored Twig template. Provide input mapping from Action 1.

Result: The "render-profile" event triggers the service, updating the profile page with the latest data.

Stream Logging Data

Goal: Send real-time logging updates to a designated channel for monitoring.

Steps:

Configure Channel Service: Create a websocket service using the "Channel" directive. Provide an ID to identify the configuration and set up the specific channel name (e.g., pr.${adm.input.projectId}).
Set Up Broadcast Service: Create a second service using the "Broadcast" event linked to the log event. The Action Chain Input defines projectId and data for the logging packet. The only Action emits this data.

Result: Applications subscribed to the defined channel receive real-time logging updates. Eezze itself uses this functionality.

Generated Code (For Advanced Understanding)

For those curious about the underlying code that powers Eezze's websocket functionality, let's examine the Channel and Broadcast example in more detail.

Websocket Controller (controller.ws.ts)

Eezze leverages decorators in TypeScript to streamline websocket service configuration:

@Channel({
  id: 'project-logging-channel',
  channel: async (adm: ADM, lc: LogicChain) => {
    // Action 1 - Logic item '0'
    lc.text.custom(() => {
      return `pr-${adm.input.projectId}`;
    });

    // evaluate the result and then return the result
    return await lc.result();
  }
})
async projectLoggingChannel() {}

@Broadcast({
  event: 'log',
  channel: async (adm: ADM, lc: LogicChain) => {
    // Action 1 - Logic item "0"
    lc.text.custom(() => {
      return `pr-${adm.input.projectId}`;
    });

    // evaluate the result and then return the result
    return await lc.result();
  },
})
async projectLogs() {}

Broadcast Service Code (action.ts)

The Broadcast Service itself demonstrates Eezze's focus on readability and efficiency:

// Send data packet to Channel
@Do({
  run: async (adm: ADM, lc: LogicChain) => {
    // Action 1 - Logic item "0"
    lc.text.custom(() => {
      adm.setResult(adm.input?.data);
      return 'success';
    });

    // evaluate the result and then return the result
    return await lc.result();
  },
})
async _exec() {}

Unlocking Real-Time Potential

Eezze's UI-based approach isn't limited to the examples we've explored. Here are a few more ideas that can be quickly realized with Eezze.

Application Type	Key Features	How Eezze Simplifies Development
Collaborative Whiteboard	Live drawing, shapes, annotations, multiple users	`on` events map directly to user actions, `broadcast` updates the view
Data Visualization Dashboard	Real-time updating charts, graphs based on data streams	`on` data arrival triggers processing, updates pushed to clients
Workflow Automation Monitor	Visualize progress of complex workflows, track status changes, send notifications	`on` workflow events trigger updates and notifications, `channel` could be used for team-specific views

Key Point: Eezze enables developers to focus on the core interactions of their application rather than the complexities of managing real-time data flow.

Conclusion

Building real-time features has traditionally been a time consuming task, requiring specialized coding expertise that can slow down even the most agile development teams. Eezze transforms this process, enabling you to harness the power of websockets through a simple, event-driven UI.

With Eezze, you focus on defining how your application should respond to user actions, data changes, and external events. The complexities of websocket management and synchronization are handled for you.

Ready to see Eezze in action and how it can revolutionize your development workflow? Contact us on Eezze.io or find us on LinkedIn

How to setup a Serverless application with AWS SAM and Terraform

Rolf Streefkerk — Thu, 07 May 2020 09:05:58 +0000

TLDR;

AWS Serverless Application Model (SAM) is used to quickly create Serverless applications with support for;
- local development that emulates AWS Lambda, and API Gateway via Docker
- takes care of blue/green deployments via AWS CodeDeploy
- infrastructure as code, repeatedly deploy the same infrastructure on multiple environments (dev, test, production).
Terraform is a Cloud agnostic Infrastructure as Code language and tooling.
- takes care of non-serverless, permissions, the "difficult stuff".

The result is a great local development experience via AWS SAM combined with the power of Terraform as the glue and as a general purpose solution for other AWS service deployments.

Chapters

Introduction
Overview
Terraform
- AWS CodePipeline
- AWS CloudWatch
- AWS IAM
AWS SAM
- AWS SAM Template
- Local testing
- AWS CodeDeploy
- Known issues
Conclusion

Introduction

The creation of this combination came out of necessity working together with developers that are not cloud native developers. Our initial solution was to emulate the online environment by using command line tooling that would execute a proxy which then executes the Lambda code. This resulted in some big problems down the line (hint, hard to debug!).

The solution here is to avoid self created offline development environments, AWS SAM is directly supported and developed by the good people of Amazon themselves. Hence, we've started using AWS SAM because of its use of Docker containers to emulate AWS services and it provides the means to easily create the relevant Service events (e.g. events for SNS, API Gateway, etc.) for AWS Lambda local testing.

In this article I'll discuss the background and the solution I created to successfully integrate Terraform deployments with AWS SAM.

There are two repositories on Github that demonstrate my solution, please feel free to review these here:

rpstreef / terraform-aws-sam-integration-example

An example how you can use both Terraform and AWS SAM

rpstreef / aws-sam-node-example

AWS SAM NodeJS project example

Overview

First off, let's start with an overview of the solution and where the responsibilities have been defined for both Terraform and AWS SAM.

The idea behind this setup is to take advantage of a couple of key features offered by AWS SAM;

Local development via Docker images.
Various Blue/Green Deployment scenarios supported via AWS CodeDeploy.

Then have the rest of the application be defined by Terraform that can do this reliably and in a DRY (Don't Repeat Yourself) structure.

With that said, AWS SAM only defines the API definition (API Gateway), the AWS Lambda functions, and Lambda CloudWatch Alarms for CodeDeploy.

Terraform

Terraform is an all purpose infrastructure as code tool suite that can create infrastructure on a number of Cloud vendors. With its big support base, and the very useful modules system, it's relatively straightforward to deploy large infrastructures to AWS.

With that said, what does Terraform deploy in this particular configuration with AWS SAM?

Besides the complete AWS CodePipeline, as shown in the diagram above, it manages all "fringe" AWS services in this code example. From left to right:

AWS Cognito for identity management and authentication. Provides API Gateway security through Header authentication.
AWS CloudWatch for monitoring with Alarms. Sets Alarms for API Gateway endpoints (latency, P95, P99 and 400/500 errors) and AWS Lambda (errors, timeouts etc.)
AWS IAM roles and permissions management. Allows execution of AWS Lambda by an API Gateway endpoint and sets general Role permission policies for AWS Lambda execution.

For your own projects this would mean; services like SQS queues, Databases, S3 buckets etc. would all be created by Terraform and referenced in the AWS SAM template.

AWS CodePipeline

To deploy AWS SAM, we use AWS CodePipeline to setup our automated CI/CD pipeline. It consists of the following stages:

Source: Retrieves the repository data from GitHub.
Build: Builds the solution based on a build script, buildspec.yaml, and the repository data.
Deploy: Deploys the build stage output.

Source

Each stage can result in artifacts that need to cary over to the next stage in the pipeline, that's where the artifact_store configuration comes in.

resource "aws_s3_bucket" "artifact_store" {
  bucket        = "${local.resource_name}-codepipeline-artifacts-${random_string.postfix.result}"
  acl           = "private"
  force_destroy = true

  lifecycle_rule {
    enabled = true

    expiration {
      days = 5
    }
  }
}

To get the source, the GitHub credentials (OAuthToken), and connection information (Owner, Repo, and Branch) needs to be provided.

PollForSourceChanges if set to true, will start the pipeline on every push to the configured Branch.

resource "aws_codepipeline" "_" {

  # Misc configuration here

  artifact_store {
    location = aws_s3_bucket.artifact_store.bucket
    type     = "S3"
  }

  stage {
    name = "Source"

    action {
      name             = "Source"
      category         = "Source"
      owner            = "ThirdParty"
      provider         = "GitHub"
      version          = "1"
      output_artifacts = ["source"]

      configuration = {
        OAuthToken           = var.github_token
        Owner                = var.github_owner
        Repo                 = var.github_repo
        Branch               = var.github_branch
        PollForSourceChanges = var.poll_source_changes
      }
    }
  }

  # Build stage here

  # Deploy stage here

  lifecycle {
    ignore_changes = [stage[0].action[0].configuration]
  }
}

The lifecycle ignore_changes configuration is applied because of a bug with the OAuthToken parameter. See this issue for more information.

Build

The build stage is pretty self explanatory.

stage {
  name = "Build"

  action {
    name             = "Build"
    category         = "Build"
    owner            = "AWS"
    provider         = "CodeBuild"
    version          = "1"
    input_artifacts  = ["source"]
    output_artifacts = ["build"]

    configuration = {
      ProjectName = aws_codebuild_project._.name
    }
  }
}

What's more interesting is the buildspec file which defines the steps to produce our artifacts. The configuration.json file contains all the properties that are defined in the AWS SAM template. That way we can configure this run for each environment (dev, test, prod) individually with different settings.

version: 0.2
phases:
  install:
    runtime-versions:
      nodejs: 12
    commands:
      - pip3 install --upgrade aws-sam-cli
      - cd dependencies/nodejs
      # Do not install dev dependencies
      - npm install --only=production
      - cd ../../
  build:
    commands:
      - sam build
  post_build:
    commands:
      - sam package --s3-bucket $ARTIFACT_BUCKET --output-template-file packaged.yaml 
artifacts:
  files:
    - packaged.yaml
    - configuration.json

Deploy

Here we get to the meat and potatoes of the pipeline, the deployment stage.

There are two steps it needs to undergo:

CloudFormation change set:

Here we create the CloudFormation Change set on an existing stack or a new stack, which means it will "calculate" the difference and applies the changes necessary to the related services in your AWS account.

For this we need an IAM Role with the appropriate permissions (role_arn or RoleArn, someone at Terraform will eventually find out which one ;) )

The template is a result from our build process, hence build::packaged.yaml. Then the build::configuration.json file is in the Github repo that contains the relevant parameters for deployment of our stack.

Please note there's an inconsistency in how Terraform deploys action CreateChangeSet. By just setting the role_arn in the action block, the RoleARN in the configuration block gets set to null. You need to set the RoleARN parameter to avoid this. I've reported this issue to the official GitHub repo.

stage {
  name = "Deploy"

  action {
    name            = "CreateChangeSet"
    category        = "Deploy"
    owner           = "AWS"
    provider        = "CloudFormation"
    input_artifacts = ["build"]
    role_arn        = module.iam_cloudformation.role_arn
    version         = 1
    run_order       = 1

    configuration = {
      ActionMode            = "CHANGE_SET_REPLACE"
      Capabilities          = "CAPABILITY_IAM,CAPABILITY_AUTO_EXPAND"
      OutputFileName        = "ChangeSetOutput.json"
      RoleArn               = module.iam_cloudformation.role_arn
      StackName             = var.stack_name
      TemplatePath          = "build::packaged.yaml"
      ChangeSetName         = "${var.stack_name}-deploy"
      TemplateConfiguration = "build::configuration.json"
    }
  }

CloudFormation change set execute:

In the second action step we actually execute the change set made previously

  action {
    name            = "Deploy"
    category        = "Deploy"
    owner           = "AWS"
    provider        = "CloudFormation"
    input_artifacts = ["build"]
    version         = 1
    run_order       = 2

    configuration = {
      ActionMode     = "CHANGE_SET_EXECUTE"
      Capabilities   = "CAPABILITY_IAM,CAPABILITY_AUTO_EXPAND"
      OutputFileName = "ChangeSetExecuteOutput.json"
      StackName      = var.stack_name
      ChangeSetName  = "${var.stack_name}-deploy"
    }
  }
}

If you want to find out which configuration block parameters are required in defining CodePipeline stages and actions, look at this official documentation page for CodePipeline, and in this case the CloudFormation action reference in particular.

AWS CodePipeline Terraform module

To run your own AWS CodePipeline for your AWS SAM integration, I've made this Terraform module available here:

https://registry.terraform.io/modules/rpstreef/codepipeline-sam/aws/1.0.0

AWS CloudWatch

With CloudWatch the main purpose is monitoring, so I've created monitoring for;

API Gateway endpoints:
- Latency P95: 95th percentile latency, which represents typical customer experienced latency figures.
- Latency P99: 99th percentile latency, represents the worst case latency that customers experience.
- 400 Errors: HTTP 400 errors reported by the endpoint.
- 500 Errors: HTTP 500 internal server errors reported by the endpoint.
Lambda functions:
- Error rate: Alarms on errors with a default of threshold of 1 percent during a 5 minute measurement period
- Throttle count: Alarm on throttle count of 1 within 1 minute measurement period
- Iterator age: Alarm for Stream based invocations such as Kinesis, alerts you when the time to execute is over 1 minute within a 5 minute measurement period. Check for more details here.
- Deadletter queue: Alarm for DLQueue messages (for async Lambda invocations or SQS queues for example), 1 message within 1 minute triggers the alarm.

AWS CloudWatch Terraform module

If you want to create CloudWatch Alarms for API Gateway endpoints and/or AWS Lambda, you can use my Terraform module to quickly set them up:

https://registry.terraform.io/modules/rpstreef/cloudwatch-alarms/aws/1.0.0

AWS IAM

To setup the required policies and permissions, the following two things are done.

Create Roles with Permissions policies that allow a service to use other services.

This creates a role based off of policy documents like shown below. Then this ARN gets applied in the AWS SAM template and that function can subsequently publish a message to an SNS topic for instance.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "logs:CreateLogGroup",
        "logs:CreateLogStream",
        "logs:PutLogEvents"
      ],
      "Resource": "arn:aws:logs:*:*:*"
    },
    {
      "Effect": "Allow",
      "Action": [
        "cognito-idp:SignUp",
        "cognito-idp:AdminInitiateAuth",
        "cognito-idp:ListUsers",
        "cognito-idp:AdminConfirmSignUp"
      ],
      "Resource": "${cognito_user_pool_arn}"
    },
    {
      "Effect": "Allow",
      "Action": [
        "lambda:GetLayerVersion"
      ],
      "Resource": "*"
    },
    {
      "Effect": "Allow",
      "Action": [
        "sns:Publish"
      ],
      "Resource": "${sns_topic_arn}"
    },
    {
      "Effect": "Allow",
      "Action": [
        "xray:PutTraceSegments",
        "xray:PutTelemetryRecords",
        "xray:GetSamplingRules",
        "xray:GetSamplingTargets",
        "xray:GetSamplingStatisticSummaries"
      ],
      "Resource": [
        "*"
      ]
    }
  ]
}

We store this json and then call it with my module like show here.

module "iam" {
  source = "github.com/rpstreef/tf-iam?ref=v1.1"

  namespace         = var.namespace
  region            = var.region
  resource_tag_name = var.resource_tag_name

  assume_role_policy = file("${path.module}/policies/lambda-assume-role.json")
  template           = file("${path.module}/policies/lambda.json")
  role_name          = "${local.lambda_function_user_name}-role"
  policy_name        = "${local.lambda_function_user_name}-policy"

  role_vars = {
    cognito_user_pool_arn = var.cognito_user_pool_arn
    sns_topic_arn         = module.sns.sns_topic_arn_lambda
  }
}

The role_vars get filled in the json template automatically upon inclusion via the data "template_file" Terraform function called within that IAM module. See the official documentation on this functionality here.

Create a permission for one service to allow execution of another service.

To allow your API Gateway to execute an integrated AWS Lambda function, you have to give it permission like so:

resource "aws_lambda_permission" "_" {
  principal     = "apigateway.amazonaws.com"
  action        = "lambda:InvokeFunction"
  function_name = var.lambda_function_identity_arn

  source_arn = "arn:aws:execute-api:${
    var.region
    }:${
    data.aws_caller_identity._.account_id
    }:${
    var.api_gateway_rest_api_id
  }/*/*"
}

This lambda:InvokeFunction action tells the principle apigateway.amazonaws.com that the source_arn is allowed to execute the function_name. You can apply this similarly for the SNS service (sns.amazonaws.com) or any other service that can integrate with AWS Lambda.

The actual integration of the AWS Lambda with the endpoint is defined in the OpenAPI document that is included in the AWS SAM repository. With these two together, you have a functioning integration.

AWS SAM

Alright so we know what the Terraform parts do, how about AWS SAM?

Check my example repository here if you'd like to go in-depth.

Let's see that diagram again.

When the AWS CodePipeline is finished it will deploy, from left to right:

AWS API Gateway: Using the OpenAPI document (api.yaml in my example repo) to specify the integration, Cognito Security, and our endpoints.
AWS Lambda: Creates just the functions with their environment variables.
AWS CloudWatch deploy Alarms: To make sure we track the right version of the deployed Lambda function, we create and update the Alarms everytime in this template.

Then there are a few artifacts in this repo that control the integration with Terraform and define the AWS SAM stack.

template.yaml: The most important file, the AWS SAM template that determines the AWS services that get deployed and how they're configured. More details on the anatomy of the AWS SAM template here.
configuration.json: This file is a CloudFormation template that specifies the parameters you can see in the template.yaml file at the top. That way each environment can have a different configurations with the same deployment of AWS services.
api.yaml: The OpenAPI document describing our API Gateway endpoints and the JSON Schema for the input and output models used for each of those endpoints. Details on the workings of this I have discussed in a previous article here.

AWS SAM Template

As you can tell from the official documents on the AWS SAM template anatomy, it resembles CloudFormation for the most part and it introduces a few special Resources to more quickly create serverless applications.

Rather than going over every bit of detail, I'll focus on two features that make creating templates much less painful. After which I'll discuss how to get local development up and running and show a bit of the CodeDeploy goodness that comes along with AWS SAM applications.

1. Globals

With Globals you have the ability to apply configuration across all individually defined resources at once. For example:

Globals:
  Function:
    Runtime: nodejs12.x
    Tags:
      Environment:
        Ref: Environment
      Name:
        Ref: AppName

Resources:
  IdentityFunction:
    Type: AWS::Serverless::Function
    Properties:
      FunctionName:
        Fn::Sub: ${Environment}-${AppName}-identity

  UserFunction:
    Type: AWS::Serverless::Function
    Properties:
      FunctionName:
        Fn::Sub: ${Environment}-${AppName}-user

This means that each function declared under Resources will use the Runtime NodeJS version 12 and they will get the same Tags applied. This saves a lot of space configuring the same properties for each individual function.

2. OpenAPI

With the OpenAPI document, certain definitions in AWS SAM template are unnecessary, therefore they can be omitted. For example:

Resources:
  IdentityFunction:
    Type: AWS::Serverless::Function
    Properties:
      CodeUri: src/
      Handler: identity/app.lambdaHandler
      Role:
        Ref: IdentityRoleARN
      Events:
        identityAuthenticate:
          Type: Api
          Properties:
            Path: /identity/authenticate
            Method: post
            RestApiId:
              Ref: Api
        identityRegister:
          Type: Api
          Properties:
            Path: /identity/register
            Method: post
            RestApiId:
              Ref: Api
        identityReset:
          Type: Api
          Properties:
            Path: /identity/reset
            Method: post
            RestApiId:
              Ref: Api
        identityVerify:
          Type: Api
          Properties:
            Path: /identity/verify
            Method: post
            RestApiId:
              Ref: Api

The Events property does two things;

It allows API Gateway to execute this function,
and it configures the integration for these endpoints with this Lambda function.

The first, we solve by setting the permissions in our Terraform configuration as demonstrated in this chapter.
The second we solve by setting the integration configuration using this AWS OpenAPI extension, x-amazon-apigateway-integration, like so:

paths:
 /identity/authenticate:
    post:
      operationId: identityAuthenticate
      description: Authenticate user (either login, or continue session)
      x-amazon-apigateway-integration:
        uri:
          Fn::Sub: arn:aws:apigateway:${AWS::Region}:lambda:path/2015-03-31/functions/${IdentityFunction.Arn}/invocations
        passthroughBehavior: "when_no_match"
        httpMethod: "POST"
        timeoutInMillis:
          Ref: APITimeout
        type: "aws_proxy"
  # omitted propertes for brevity

To make this interpolation work, you'll have to include this document in the Api resource like this:

Resources:
  Api:
    Type: AWS::Serverless::Api
    Properties:
      DefinitionBody:
        Fn::Transform:
          Name: AWS::Include
          Parameters:
            Location: api.yaml
      # omitted propertes for brevity

This has an added benefit that the OpenAPI document is kept separate from your AWS SAM template, so you can use the OpenAPI document for instance to generate client and/or server side code without modification.

The idea is to make the AWS SAM use as minimal as possible, and have everything else done by either the OpenAPI specification or my Terraform modules.

Local testing

To test run locally, a few things need to be in place:

Docker installation: Make sure you have Docker installed or Docker Desktop when you're on Windows.
Installed dependencies:, the NodeJS dependencies need to be installed, npm install, before the related code can be executed.
Code editor with Debugging capabilities:, to link the debugging port with your code editor view. I use Visual Code which has support for this integrated, like most editors.
Environment variables: The last part of emulating the online environment, duplicate your variables in the env.json file categorized by function name. Also note that when running in a local environment, the AWS_SAM_LOCAL environment variable is set to true.

Now start the local development api with sam local start-api -d 5858 -n env.json or via the npm run command in my repo npm run start-api.

The following will appear indicating the api server is ready:

Mounting UserFunction at http://127.0.0.1:3000/user [GET, POST]
Mounting IdentityFunction at http://127.0.0.1:3000/identity/reset [POST]
Mounting IdentityFunction at http://127.0.0.1:3000/identity/register [POST]
Mounting IdentityFunction at http://127.0.0.1:3000/identity/authenticate [POST]
Mounting IdentityFunction at http://127.0.0.1:3000/identity/verify [POST]

Debugging is enabled on port 5858, you'll see this notice in the command line:

Debugger listening on ws://0.0.0.0:5858/275235c0-635f-4e27-b412-d4137d4e1c64

Now I can set breakpoints in my code editor and attach the debugging tooling to the debugging port 5858. This should stop execution at that breakpoint, allowing you to live view variables and the call stack.

Creating events

Now that we can actually start a local api, the next thing is to actually replicate the events you can receive from other other AWS services such as DynamoDB, S3, SNS etc.

To view the supported list of events, execute sam local generate-event. Then we can choose to emulate SNS like this:

sam local generate-event sns notification

To customize it, add the appropriate flags that wish to have in the event. To list those flags:

sam local generate-event sns notification --help

or just manually edit the json files.

Once we have these events, we can do a local test run by invoking the lambda directly with the event we want to test:

sam local invoke IdentityFunction -e ./events/sns-notification.json -d 5858 -n env.json

AWS CodeDeploy

The last piece of functionality that makes AWS SAM such great tooling to use, is the CodeDeploy functionality. Next to support for ECS, and EC2, it also supports deployment to Lambda functions. With this service you basically get blue/green deployment out of the box for free without having to set anything up at all.

How this works is, once the CloudFormation stack (changes) are getting applied, it will use CodeDeploy for the Lambda function parts which looks like this:

Depending on the deployment strategy chosen in your AWS SAM template (under property DeploymentPreference), for instance:

Resources:
  IdentityFunction:
    Type: AWS::Serverless::Function
    Properties:
      FunctionName:
        Fn::Sub: ${Environment}-${AppName}-identity
      Description: Provides all Cognito functions through 4 API Rest calls
      DeploymentPreference:
        Type: LambdaCanary10Percent5Minutes

It will automatically do traffic shifting from your old version to the new Lambda version code. In this example, it will shift 10 percent to the new version and 90% to your old version. After 5 minutes, 100% of your new code is deployed.

See this official article for all the available configurations.

To manage potential errors and rollbacks, this is where the CloudWatch Alarms come in that were discussed in the introduction of this chapter.

Resources:
  IdentityCanaryErrorsAlarm:
    Type: AWS::CloudWatch::Alarm
    Properties:
      AlarmName: 
        Fn::Sub: ${Environment}-${AppName}-identity-canary-alarm
      AlarmDescription: Identity Lambda function canary errors
      ComparisonOperator: GreaterThanThreshold
      EvaluationPeriods: 2
      MetricName: Errors
      Namespace: AWS/Lambda
      Period: 60
      Statistic: Sum
      Threshold: 0
      Dimensions:
        - Name: Resource
          Value:
            Fn::Sub: ${IdentityFunction}:live
        - Name: FunctionName
          Value: 
            Ref: IdentityFunction
        - Name: ExecutedVersion
          Value:
            Fn::GetAtt:
            - IdentityFunction
            - Version
            - Version

Whenever a deployment is done with CodeDeploy, it will update the CloudWatch alarm to monitor the latest version for any errors within the EvaluationPeriods specified. If the Threshold is breached, it will do a rollback to the old version.

Known issues

Not all is perfect yet with AWS SAM, I've logged some of the errors and issues I came across for your convenience.

1. I'm not seeing my code changes when running my api with `sam local start-api`

Unlike what the CLI tool suggests, You only need to restart SAM CLI if you update your AWS SAM template, this is actually not true.

You'll need to run sam build after every code change. Hopefully this will get fixed in later versions of the tool. Links to related issues on Github;

The work around suggested is to use nodemon and then run nodemon --exec sam build next to the command sam local start-api -d 5858 -n env.json. This way any code changes trigger the sam build and you can keep the local api running in parallel.

2. 'CreateFile', 'The system cannot find the file specified.'

Makes sure your Docker environment is running before you run any local invocations.

3. Runtime.ImportModuleError

To run AWS SAM locally, make sure you run npm install in your dependencies folder. For my example, the dependencies are located in ./dependencies/nodejs

4. No deployment configuration found for name: LambdaAllAtOnce

Somehow it does not find this configuration even though it is listed. To solve it, use the AllAtOnce as DeploymentPreference which also seems to work for Lambda next to EC2.

5. AWS SAM creates a stage called "Stage" by default.

The solution here as suggest by this issue, is to add the OpenAPI version to the globals section of your AWS SAM template like this.

Globals:
  Api:
    OpenApiVersion: 3.0.1

Conclusion

The inclusion of AWS SAM in our local development workflow has made things significantly more efficient and the ability to use Terraform for almost everything else keeps most of what we already did the same. The best of both worlds!

There are some definitive improvements to be made in this setup I'm sure. My first thoughts are using AWS Systems Manager Parameter store for synching parameters between Terraform and AWS SAM. For now however, the manual synching works and only needs to be done at setup time.

If you have any other idea's or improvements, let me know in the comments below.

Thanks for reading and leave a heart when you enjoyed or appreciated the article.

Till next time!

How to protect Serverless (Open)API's?

Rolf Streefkerk — Tue, 24 Mar 2020 11:26:26 +0000

Last week I was unable to deliver a proper article due to the sudden Corona measures (I hope everyone stays safe out there!), so I decided to postpone.

With that said, I'd like to discuss how to secure your (Open)API's using the tools that already exist in the AWS services we're using, and how AWS WAF (Web Application Firewall) can potentially assist (for a price).

We'll cover the following topics:

Introduction
OWASP Top 10
- (1) Injection
- (2) Broken Authentication
- (3) Sensitive data exposure
- (4) XML External Entities (XXE)
- (5) Broken Access Control
- (6) Security Misconfiguration
- (7) Cross-Site Scripting XSS
- (8) Insecure Deserialization
- (9) Using Components with Known Vulnerabilities
- (10) Insufficient Logging & Monitoring
In closing
Further reading

Introduction

To give this discussion better context, I'll be using the OWASP Top 10 Web Application Vulnerabilities as a guiding list.

The architecture to be considered for this article is my reference application architecture used for this series of OpenAPI articles.

rpstreef / openapi-tf-example

Example of how you can use OpenAPI with AWS API Gateway, Also includes integrations with AWSLambda, AWS Cognito, AWS SNS and CloudWatch logs

It features;

API Gateway, exposes all the available services
Lambda, contains the code that ultimately exposes/processes data to/from the API's.
Cognito for Identity Management, identity registration, login, and session management.
CloudWatch logs, error, info, debug logging and Alarms service.
SNS, reliable and durable pub/sub message based integration service.
AWS CodePipeline and CodeBuild, CI/CD for Lambda code deployments.

Normally you would also use a data source, here we can assume we're either using NoSQL (DynamoDB) or a SQL variant (MySQL, Postgress).

OWASP Top 10

The OWASP list represents a broad consensus about the most critical security risks to web applications.

For each of the 10 items I'll go through the general risks associated with that item, and then the possible solutions to apply for each of the relevant AWS services. In the conclusion for each chapter I'll briefly state my recommendation.

(1) Injection

Risks

Injection risks are most prevalent in environment variables, (No)SQL queries, JSON/XML parsers, and API query/body parameters.

Solutions

AWS API Gateway & OpenAPI

To mitigate any type of injection attempt we can force correct input of our API's. To do that we can use the concept of "Models" in API Gateway. These models are essentially JSON Schema's that define what type of data is accepted as input or what is given as output after execution.

For example, this register model has a regex pattern that defines what kind of information is allowed to be input into API's using this model:

{
  "title" : "Register",
  "required" : [ "email", "firstName", "lastName", "password", "username" ],
  "type" : "object",
  "properties" : {
    "email" : {
      "pattern" : "^[_A-Za-z0-9-\\+]+(\\.[_A-Za-z0-9-]+)*@[A-Za-z0-9-]+(\\.[A-Za-z0-9]+)*(\\.[A-Za-z]{2,})$",
      "type" : "string"
    },
    "password" : {
      "type" : "string"
    },
    "username" : {
      "type" : "string"
    },
    "firstName" : {
      "type" : "string"
    },
    "lastName" : {
      "type" : "string"
    }
  },
  "description" : "Registration details"
}

When you enable validation on your API resource in your OpenAPI specification, it will use this JSON Schema to automatically validate your input. This will support regex validation as well and not just the required property(!). See this AWS documentation for more information.

/identity/register:
    post:
      tags:
      - "Identity"
      description: "Register new Business user"
      operationId: "identityRegister"
      requestBody:
        description: "Registration details"
        content:
          application/json:
            schema:
              $ref: "#/components/schemas/Register"
        required: true
      x-amazon-apigateway-request-validator: full

The request validator selects which configuration to use for that particular endpoint.

Somewhere else in the root of the document we should configure the variants, like so:

x-amazon-apigateway-request-validators:
    full:
      validateRequestBody: true
      validateRequestParameters: true
    body-only:
      validateRequestBody: true
      validateRequestParameters: false

Of course, you will have to do more data validation within your application code, string will accept anything. Here you can make a trade off between API Gateway validation and code based validation. The advantage of API Gateway validation is that you will not be charged for any code execution at all.

When you execute against an API with this configuration and you supply incorrect input, this is the response:

{
    "message": "Invalid request body"
}

AWS Lambda

By default all environment variables are encrypted at rest, see here if you want to use your own encryption keys (CMK).

If you want to encrypt the environment variables in transit, you can use a key managed by AWS KMS to do that. See this article explaining how to create the key, and how to manage encryption and decryption in AWS Lambda.

AWS WAF

The Web application firewall support OSI Layer Level 7 filtering, meaning on the application level it can do input filtering. This is similar to the filtering shown previously but this is automated and it won't reach your API at all if the filter blocks the API request.

This Github project implements all the OWASP Top 10 rules into AWS WAF for you if you wish. There's a downside with respect to the pricing model you need to be aware of. So here a trade off needs to be made between code level protection or an AWS service protection.

resource "aws_wafregional_sql_injection_match_set" "owasp_01_sql_injection_set" {
  count = "${lower(var.target_scope) == "regional" ? "1" : "0"}"

  name = "${lower(var.service_name)}-owasp-01-detect-sql-injection-${random_id.this.0.hex}"

  sql_injection_match_tuple {
    text_transformation = "URL_DECODE"

    field_to_match {
      type = "URI"
    }
  }

  sql_injection_match_tuple {
    text_transformation = "HTML_ENTITY_DECODE"

    field_to_match {
      type = "URI"
    }
  }

  sql_injection_match_tuple {
    text_transformation = "URL_DECODE"

    field_to_match {
      type = "QUERY_STRING"
    }
  }

  sql_injection_match_tuple {
    text_transformation = "HTML_ENTITY_DECODE"

    field_to_match {
      type = "QUERY_STRING"
    }
  }

  sql_injection_match_tuple {
    text_transformation = "URL_DECODE"

    field_to_match {
      type = "BODY"
    }
  }

  sql_injection_match_tuple {
    text_transformation = "HTML_ENTITY_DECODE"

    field_to_match {
      type = "BODY"
    }
  }

  sql_injection_match_tuple {
    text_transformation = "URL_DECODE"

    field_to_match {
      type = "HEADER"
      data = "Authorization"
    }
  }

  sql_injection_match_tuple {
    text_transformation = "HTML_ENTITY_DECODE"

    field_to_match {
      type = "HEADER"
      data = "Authorization"
    }
  }
}

resource "aws_wafregional_rule" "owasp_01_sql_injection_rule" {
  depends_on = ["aws_wafregional_sql_injection_match_set.owasp_01_sql_injection_set"]

  count = "${lower(var.target_scope) == "regional" ? "1" : "0"}"

  name        = "${lower(var.service_name)}-owasp-01-mitigate-sql-injection-${random_id.this.0.hex}"
  metric_name = "${lower(var.service_name)}OWASP01MitigateSQLInjection${random_id.this.0.hex}"

  predicate {
    data_id = "${aws_wafregional_sql_injection_match_set.owasp_01_sql_injection_set.0.id}"
    negated = "false"
    type    = "SqlInjectionMatch"
  }
}

This particular example consists of 8 rules within one Web ACL. The total cost of activating this rule is:

Resource type	Number	Cost
Web ACL	1	$5.00 (pro-rated hourly)
Rule	8	$8.00 (pro-rated hourly)
Request	per 1 Million	$0.60
Total		$13.60 / month

Based on 1 Million requests to your API, you'll pay $13.60 for protection against all types of SQL injection attacks on top of the regular API Gateway request costs.

Here there's a definitive trade off between convenience, cost, traffic, and trusting development level security over DevOps rule based systems such as AWS WAF.

Conclusion

For most applications I would say AWS WAF for SQL injection is not worth enabling over doing API Gateway input validation and application level input filtering.

If you have real-world experience using AWS WAF in large scale applications, let me know in the comments.

(2) Broken Authentication

Risks

Here we want to avoid dictionary attacks on your authentication process, and/or broken or badly designed authentication mechanisms.

Solutions

AWS API Gateway & OpenAPI

For our API's we want to make sure we enable authentication on our API's as much as possible, to enable this on our OpenAPI we add the security parameter as follows:

paths:
  /user:
    get:
      operationId: getUser
      description: get User details by ID
      parameters:
        - $ref: '#/components/parameters/userID'
      security:
      - example-CognitoUserPoolAuthorizer: []

That name is a reference to the Cognito user pool authorizer configuration:

components:
  securitySchemes:
    example-CognitoUserPoolAuthorizer:
      type: "apiKey"
      name: "Authorization"
      in: "header"
      x-amazon-apigateway-authtype: "cognito_user_pools"
      x-amazon-apigateway-authorizer:
        providerARNs:
        - "${cognito_user_pool_arn}"
        type: "cognito_user_pools"

Once enabled for an API, they need to have a registered user in the Cognito User Pool before they can be authenticated and receive a JWT Token as proof they're authorized to execute the API.

AWS Cognito

In Cognito we need to do several things to make sure we defend against the following potential risks:

Dictionary attacks / brute force attacks
Password length and strength requirements.
Multi-factor authentication
Rotation of authorization session IDs.
Invalidation of session IDs.

First of all, we want to make sure password are at least 8 characters long as recommended by NIST

Within the aws_cognito_user_pool resource, we set the password policy as follows:

resource "aws_cognito_user_pool" "_" {
  # ... other configuration ...

  password_policy {
    minimum_length    = 8
    require_uppercase = true
    require_lowercase = true
    require_numbers   = true
    require_symbols   = true
  }

   # ... other configuration ...
}

Then to enable multi-factor authentication:

resource "aws_cognito_user_pool" "_" {
  # ... other configuration ...

  mfa_configuration          = "ON"
  sms_authentication_message = "Your code is {####}"

  sms_configuration {
    external_id    = "example"
    sns_caller_arn = aws_iam_role.sns_caller.arn
  }

  software_token_mfa_configuration {
    enabled = true
  }

   # ... other configuration ...
}

When mfa_configuration is set to ON it is required for everyone to either have an SMS or Software MFA enabled.

If you want, Cognito supports more advanced security features for additional cost. The example given for 100.000 users without advanced security is $250, with is $4525(!). Again here, is this worth your investment.

The features provided are:

Checks for compromised credentials, this will do username/password scans to see if it has been leaked anywhere.
Adaptive authentication, determines risk level per authentication and can block the sign-in if determined to be suspicious.
Publishing of security metrics, this will report all the sign in metrics to your CloudWatch logs.

This will obviously require you integrate the Cognito SDK into your app or web application.

Conclusion

Properly implementing API Gateway security and Cognito Authentication with sensible password requirements with 2FA can already mitigate most issues indicated by OWASP.

If you have stricter security requirements for your app, the additional advanced security features offered by Cognito can be worthwhile investigating.

(3) Sensitive data exposure

Risks

Here obvious risks are not encrypting data in transit or at rest, weak cryptography that can be reversed engineered, and man in the middle attacks.

Solutions

All data on the AWS Services side is encrypted in transit using TLS/SSL. We're now mostly looking at how you can protect data at rest.

AWS DynamoDB

For DynamoDB we want to make sure we're encrypting information as follows.

resource "aws_dynamodb_table" "_" {

   # ... other configuration ...

  server_side_encryption {
    enabled = true
  }

   # ... other configuration ...

}

For more information on CMK or updating table encryption, see this official documentation.

For further best practices regarding security on DynamoDB, see this official documentation.

Since DynamoDB uses IAM role access it's not strictly necessary to use a VPC to restrict public access. If you want to be following best practices, using a VPC solution is recommended here and can ensure all traffic is done within the AWS network for enhanced security.

AWS RDS

When creating a SQL based data store, we can use the following to enable encryption:

resource "aws_db_instance" "_" {

  # ... other configuration ...

  storage_encrypted   = true
  publicly_accessible = false

  # ... other configuration ...

}

You would also be well advised not to allow public access to the database. A VPC should be configured to allow only private subnets access to your database to enhance security.

To connect to RDS you can use it's public SSL certificate to ensure data transmissions are encrypted.

See this article for general security guidelines for AWS RDS.

AWS Lambda

Before storing sensitive data such as credit card details, it's advised to use the proper encryption algorithms. These algorithms have a work factor delay that makes brute forcing very difficult. Argon2, scrypt, bcrypt, or PBKDF2.

Conclusion

Depending on which Database you're using, the options are available to ensure in transit and at rest encryption. The next thing is to prevent public access as much as possible using a VPC setup.

(4) XML External Entities (XXE)

Risks

A lot of systems still use XML in one form or another, you can think about SOAP API services, and integration layer messaging systems. This risk is about automated XML processing with mostly outdated library dependencies.

Solutions

AWS Lambda

Ensure your XML processing libraries are up to date and check the Example Attack Scenarios here to get an idea about the attack vectors.

Use this cheat sheet to find out exactly how to mitigate the risk for these programming languages:

Conclusion

To be honest, I don't use XML at all. We have just one project where we need to ingest XML coming from a USA government agency and that particular format does not even conform XML standards. We had to build a custom parser to deal with it.

(5) Broken Access Control

Risks

Broken access control means attackers can get access to information that does not belong to the user session. For instance, by means of altering URL query parameters like this:

http://example.com/app/accountInfo?acct=notmyacct

Another potential risk is liberal CORS settings that allow API execution from different web domains.

Solutions

AWS API Gateway & OpenAPI

CORS

Managing the correct CORS settings for your API is the first step. For this we need to modify the options operation on each of the API paths that has CORS enabled:

options:
  responses:
    200:
      $ref: '#/components/responses/cors'
    400:
      $ref: '#/components/responses/cors'
    500:
      $ref: '#/components/responses/cors'
  x-amazon-apigateway-integration:
    responses:
      default:
        statusCode: "200"
        responseParameters:
          method.response.header.Access-Control-Max-Age: "'7200'"
          method.response.header.Access-Control-Allow-Methods: "'OPTIONS,HEAD,GET,POST,PUT,PATCH,DELETE'"
          method.response.header.Access-Control-Allow-Headers: "'Content-Type,X-Amz-Date,Authorization,X-Api-Key,X-Amz-Security-Token'"
          method.response.header.Access-Control-Allow-Origin: "'*'"
    passthroughBehavior: "when_no_match"
    timeoutInMillis: 29000
    requestTemplates:
      application/json: "{ \"statusCode\": 200 }"
    type: "mock"

Crucial is the method.response.header.Access-Control-Allow-Origin response parameter that is now set to '*' to accept all origin requests. For development purposes this may be fine when you're dealing with various testing servers and local execution environments. For production we need to lock this down to the domain name executing this API only.

API Limits

Next thing we want to make sure that we limit the number of parallel executions of the API.

resource "aws_api_gateway_method_settings" "_" {
  rest_api_id = aws_api_gateway_rest_api._.id
  stage_name  = aws_api_gateway_stage._.stage_name
  method_path = "*/*"

  settings {
    throttling_burst_limit = var.api_throttling_burst_limit
    throttling_rate_limit  = var.api_throttling_rate_limit
    metrics_enabled        = var.api_metrics_enabled
    logging_level          = var.api_logging_level
    data_trace_enabled     = var.api_data_trace_enabled
  }
}

To control limits on your API stage, set the throttling_burst_limit and throttling_rate_limit parameters. They control the following:

throttling_burst_limit: The number of requests per second the API Stage will sustain for a few seconds after which it will error with a 429 HTTP response.
throttling_rate_limit: The number of requests per second for the API Stage, it will continue until it reaches the burst rate limit after which it will error with a 429 HTTP response.

In case of abuse, we need to ensure these have sensible limits such that DoS attacks are limited as much as possible.

AWS DynamoDB, Cognito & Lambda

Fine grained access control

We can implement fine grained access control mechanisms on DynamoDB that will allow only the authenticated user access to its own records.

This article goes into detail how to do this, the following example is given:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "dynamodb:GetItem",
        "dynamodb:BatchGetItem",
        "dynamodb:Query"
      ],
      "Resource": [
        "arn:aws:dynamodb:us-west-2:123456789012:dynamodb:table/GameScores"
      ],
      "Condition": {
        "ForAllValues:StringEquals": {
          "dynamodb:LeadingKeys": [
            "${www.amazon.com:user_id}"
          ],
          "dynamodb:Attributes": [
            "UserId",
            "GameTitle",
            "Wins"
          ]
        },
        "StringEqualsIfExists": {
          "dynamodb:Select": "SPECIFIC_ATTRIBUTES"
        }
      }
    }
  ]
}

The above policy will give read only access to items in the GameScores table that belong to www.amazon.com:user_id and it will only be able to retrieve these columns for each row; UserId, GameTitle, and Wins.

To note is that www.amazon.com:user_id refers to identities from an Amazon Federated login and not to the regular Cognito User Pool. To use the User Pool registered identities, use cognito-identity.amazonaws.com:sub instead.

To be able to use this policy in your AWS Lambda code you have to retrieve the temporary credentials with the following function, see the official documentation for more details:

var params = {
  RoleArn: 'STRING_VALUE', /* required */
  RoleSessionName: 'STRING_VALUE', /* required */
  WebIdentityToken: 'STRING_VALUE', /* required */
  DurationSeconds: 'NUMBER_VALUE',
  Policy: 'STRING_VALUE',
  PolicyArns: [
    {
      arn: 'STRING_VALUE'
    },
    /* more items */
  ],
  ProviderId: 'STRING_VALUE'
};
sts.assumeRoleWithWebIdentity(params, function(err, data) {
  if (err) console.log(err, err.stack); // an error occurred
  else     console.log(data);           // successful response
});

When this executes successfully you have temporary credentials that allow you to read from DynamoDB as indicated by the policy earlier.

Roles & Role access

To solve the roles and role access problems for your API, you can use the following setup in Cognito:

module "cognito" {
  source = "../modules/cognito"

  namespace         = var.namespace
  resource_tag_name = var.resource_tag_name
  region            = var.region

  cognito_identity_pool_name     = var.cognito_identity_pool_name
  cognito_identity_pool_provider = var.cognito_identity_pool_provider

  schema_map = [
    {
      name                = "email"
      attribute_data_type = "String"
      mutable             = false
      required            = true
    },
    {
      name                = "phone_number"
      attribute_data_type = "String"
      mutable             = false
      required            = true
    },
    {
      name                = "businessID"
      attribute_data_type = "String"
      mutable             = true
      required            = false
    },
    {
      name                = "role"
      attribute_data_type = "String"
      mutable             = true
      required            = false
    },
    {
      name                = "roleAccess"
      attribute_data_type = "String"
      mutable             = true
      required            = false
    }
  ]
}

The schema_map contains all the additional parameters that need to be recorded (either manual or automated after registration) for a registration and they're retrievable during an authenticated session in AWS Lambda.

The event object in an AWS Lambda session contains all the information from the currently authenticated user. To retrieve, access the claims object: event.requestContext.authorizer.claims which will contain the custom parameters set in AWS Cognito like so:

{
  'custom:role': 'USER',
  'custom:roleAccess': '{
    "business": "rwu",
    "role": "r",
    "user": "rwud",
  }',
  'custom:businessID': '1'
}

When a user authenticates against Cognito, now AWS Lambda knows which role the user has, and what kind of role access that entails. Each of those properties, e.g. business, will enable you to allow access to GET (r), POST (w), PUT (u), DELETE (d) operations on that specific business API. You can make this as fine grained as you need it to.

For key identifiers such as a businessID it's recommended to make those part of the DynamoDB partition key such that they're mandatory when querying for data and never supplied via API query parameters or body JSON objects.

Conclusion

Broken access controls can create a lot of problems both for data security as well as (D)DoS attack opportunities via loose CORS and Rate limit settings on your API.

(6) Security Misconfiguration

Risks

This is a very broad set of threats that go across the application stack. From unpatched software, to default accounts.
A typical example here is an S3 bucket that has public permissions, this used to be prevalent as this article shows.

Solutions

AWS Code Build

Since this stack is running fully Serverless and thus is a managed service by Amazon, the area's that are most exposed to potential security leaks is the code we use on AWS Lambda and Lambda Layer.

In case of Node, we need to ensure we have upgraded to the latest version supported on AWS and that is version 12. Within your build system we have many opportunities to test for vulnerabilities in dependencies and code such as:

NPM Audit: To audit your dependencies, see this article for details, we can implement the npm audit step in CodeBuild and halt the pipeline once vulnerabilities have been found.
OWASP Dependency checker: OWASP has a dependency checker that can be integrated on the command line as well, see here. This particular analyzer has support for several languages including NodeJS and NPM Package manager, see here for the details.
NodeJsScan: NodeJsScan is a docker ready general purpose security scanner specifically for NodeJS, here's the Github for more details. This scans your entire codebase for vulnerabilities such as XSS, Remote Code Injection, and SQL Injection among others.
RetireJs: is a well known scanner for node dependencies and javascript code vulnerabilities. See their Github page for details.

AWS Lambda

For AWS Lambda we have a great middleware to help cover correct security configurations regarding HTTP REST API's, it's called Middy. These are some of the recommended plugins:

HTTP-CORS helps to set the correct HTTP CORS headers.
HTTP-Error-handler returns correct HTTP responses, this works in conjunction with http-error.
HTTP-Security-headers this applies best practice security headers to HTTP responses.

Conclusion

Since AWS takes care of the details around every service we use, we still have a responsibility in the area's we introduce possible security leaks. Of course we can misconfigure an S3 bucket for public access, and there are many other examples that I could cover. However, I focussed on the key area of source code instead.

If you're interested in a much more broad and in-depth look, please review the guidelines on CIS for additional solutions here.

(7) Cross-Site Scripting XSS

Risks

XSS flaws occur when web applications include user-provided data in webpages that is sent to the browser without proper sanitization. If the data isn’t properly validated or escaped, an attacker can embed scripts, inline frames, or other objects into the rendered page.

XSS is a very well known security risk, there are several variants OWASP recognizes:

Reflected XSS: this is typically about URL interaction/scripts that have malicious intent coming from user data that is not sanitized.
Stored XSS: storing of unsanitized user input that is viewed by someone else and in the worst case by someone with admin privileges.
DOM XSS: most common with javascript frameworks that manipulate the DOM such as ReactJS and VueJS. An example here is a DOM node replacement with a malicious login screen.

Solutions

AWS API Gateway & Lambda

This risk is about input sanitation, we have discussed this in previous chapters, please see the chapter about Injection for my recommendations on how to deal with this.

AWS WAF

Additionally, if you'd like specific Web Application Firewall rules to deal with XSS risks, the following Terraform code can be used (taken from this Github repo):

resource "aws_wafregional_xss_match_set" "owasp_03_xss_set" {
  count = "${lower(var.target_scope) == "regional" ? "1" : "0"}"

  name = "${lower(var.service_name)}-owasp-03-detect-xss-${random_id.this.0.hex}"

  xss_match_tuple {
    text_transformation = "URL_DECODE"

    field_to_match {
      type = "URI"
    }
  }

  xss_match_tuple {
    text_transformation = "HTML_ENTITY_DECODE"

    field_to_match {
      type = "URI"
    }
  }

  xss_match_tuple {
    text_transformation = "URL_DECODE"

    field_to_match {
      type = "QUERY_STRING"
    }
  }

  xss_match_tuple {
    text_transformation = "HTML_ENTITY_DECODE"

    field_to_match {
      type = "QUERY_STRING"
    }
  }

  xss_match_tuple {
    text_transformation = "URL_DECODE"

    field_to_match {
      type = "BODY"
    }
  }

  xss_match_tuple {
    text_transformation = "HTML_ENTITY_DECODE"

    field_to_match {
      type = "BODY"
    }
  }

  xss_match_tuple {
    text_transformation = "URL_DECODE"

    field_to_match {
      type = "HEADER"
      data = "cookie"
    }
  }

  xss_match_tuple {
    text_transformation = "HTML_ENTITY_DECODE"

    field_to_match {
      type = "HEADER"
      data = "cookie"
    }
  }
}

resource "aws_wafregional_rule" "owasp_03_xss_rule" {
  depends_on = ["aws_wafregional_xss_match_set.owasp_03_xss_set"]

  count = "${lower(var.target_scope) == "regional" ? "1" : "0"}"

  name        = "${lower(var.service_name)}-owasp-03-mitigate-xss-${random_id.this.0.hex}"
  metric_name = "${lower(var.service_name)}OWASP03MitigateXSS${random_id.this.0.hex}"

  predicate {
    data_id = "${aws_wafregional_xss_match_set.owasp_03_xss_set.0.id}"
    negated = "false"
    type    = "XssMatch"
  }
}

Again here, the cost to have AWS take care of this for you is as follows. These XSS protections contain 8 rules within one Web ACL. The total cost of activating this rule is:

Resource type	Number	Cost
Web ACL	1	$5.00 (pro-rated hourly)
Rule	8	$8.00 (pro-rated hourly)
Request	per 1 Million	$0.60
Total		$13.60 / month

Based on 1 Million requests to your API, you'll pay $13.60 for protection against all types of XSS attacks on top of the regular API Gateway request costs. You can of course activate this together with the SQL injection rules within the same Web ACL to save $5.00 per month for a total cost of $22.20 / month.

Conclusion

When you apply good input sanitation that should sufficiently protect you from any risk of XSS.
Please check your current implementation againt this cheat sheet from OWASP to ensure you're fully protected.

(8) Insecure Deserialization

Risks

Flaws in how deserialization is done can result in remote code execution. For example, a JSON object send by a web application via an AWS API Gateway is deserialized (JSON to Javascript Object conversion) and actions are performed on this data.

Examples of potential risks are; session cookie tampering, and session state manipulation.

Solutions

AWS Lambda

This requires manual code review and the use of code vulnerability scanners as introduced in the Security Misconfiguration chapter. The solution as recommended by OWASP is to validate state and cookies using integrity checks with digital signatures such as a hash or an authentication signature. If the data does not contain such signatures, it will be ignored.
Furthermore, logging of any deserialization errors is very useful to detect any possible tampering.

Conclusion

Please view this OWASP cheat sheet for details per programming language what to look out for.

(9) Using Components with Known Vulnerabilities

Risks

Any code or component that is used as a dependency can have known vulnerabilities. Because there's such a massive reliance on open source dependencies in most projects it's very hard to be aware of security risks.

Solutions

Here we need to rely on Build phase dependency scanning and manual review to avoid security risks as much as possible. Please review the Solutions in the chapter Security Misconfiguration to mitigate this security risk.

Conclusion

The fact that several OWASP Top 10 risks share similarities means that many software services and products rely on open source software. With that reliance come security vulnerabilities that are hard to detect without some form of automated detection. Developers and DevOps need to put a lot more attention on threat detection in the build phase before code is put into production.

(10) Insufficient Logging & Monitoring

Risks

This risks is about a lack of contextual logging (where, what, and how) which decreases awareness, and in time monitoring of threats. The latter is about how quickly an IT team can respond to security incidents.

Solutions

AWS CloudWatch

CloudWatch can provide both Logging, and Monitoring with Alerts. As I've demonstrated in an earlier article in this OpenAPI series of articles. Review that article for logging and Monitoring suggestions that will increase visibility and make debugging and error (threat) detection easier.

How to do logging on AWS Serverless

Rolf Streefkerk ・ Feb 9 '20

#aws #serverless #devops #terraform

AWS X-Ray

X-Ray is as the name suggest a service that provides much more transparency into Serverless execution by both visualizing the execution path, and making logs easily accessible. I've covered the use and implementation of it in an earlier article, please review that article below.

Serverless tracing with AWS X-Ray

Rolf Streefkerk ・ Feb 16 '20

#aws #devops #serverless #terraform

Conclusion

With these two tools in hand most anomalies can be relatively quickly detected either though monitoring alerts from CloudWatch via SNS, or through bug/error analysis with AWS X-Ray.

In closing

There was a lot to cover here and I'm sure I missed equally as much as well. Researching this has actually uncovered some things I haven't implemented myself properly yet, that's what makes creating these articles so valuable for me. I hope this has been useful reading, and I very much appreciate your time and attention.

If I've made any mistakes or you have suggestions, additions please do let me know in the comments. I'm very much interested reading your thoughts and expertise on this difficult subject.

For my next article I'd like to discuss DynamoDB in some detail, and why you should be using it.

Thanks for reading!

How to setup a basic VPC with EC2 and RDS using Terraform

Rolf Streefkerk — Sun, 08 Mar 2020 10:34:50 +0000

This guide will help you set up a basic AWS VPC with a virtual machine (EC2) and database (RDS) using Terraform (Infrastructure as Code).

I'll be breaking this topic down as follows:

The outline
VPC
EC2
- EC2 Security group
RDS
- RDS Security group
Conclusion

The outline

We're going to create the following on AWS:

A VPC with 1 Route table that connects the Internet Gateway to the public subnet that hosts the EC2 instance.

Two private subnets configured as 1 subnet group that hosts 1 RDS instance.

Access control is arranged using security groups, one for the EC2 public subnet and 1 for the RDS private subnets.

The reason we have 2 subnets for RDS is because that is a deployment requirement, you cannot launch an RDS instance without configuring it with 2 subnets.

Ideally, you would want to do load balancing for both EC2 and RDS instances. On the EC2 side you would have to add another subnet for the other EC2 instance and connect them with a load balancer. In case one of the subnets goes down for whatever reason, your site is still up and running.

For this article however, we're going to focus on the minimum setup for development and testing purposes.

VPC

To create a VPC we configure our module as follows:

resource "aws_vpc" "_" {
  cidr_block = var.vpc_cidr

  enable_dns_support   = var.enable_dns_support
  enable_dns_hostnames = var.enable_dns_hostnames
}

resource "aws_internet_gateway" "_" {
  vpc_id = aws_vpc._.id
}

resource "aws_route_table" "_" {
  vpc_id = aws_vpc._.id

  dynamic "route" {
    for_each = var.route

    content {
      cidr_block     = route.value.cidr_block
      gateway_id     = route.value.gateway_id
      instance_id    = route.value.instance_id
      nat_gateway_id = route.value.nat_gateway_id
    }
  }
}

resource "aws_route_table_association" "_" {
  count          = length(var.subnet_ids)

  subnet_id      = element(var.subnet_ids, count.index)
  route_table_id = aws_route_table._.id
}

Please note; I removed the tag blocks for brevity, but you should tag every resource possible to enable easy cost tracking of your deployments and to be able to find everything should anything go wrong with the tfstate.

The route table is configured to be associated with an internet gateway in the aws_route_table_association resource. Any subnet we supply in var.subnet_ids will have access to the route table configuration and the internet gateway.

I call the VPC module like this:

module "vpc" {
  source = "../../modules/vpc"

  resource_tag_name = var.resource_tag_name
  namespace         = var.namespace
  region            = var.region

  vpc_cidr = "10.0.0.0/16"

  route = [
    {
      cidr_block     = "0.0.0.0/0"
      gateway_id     = module.vpc.gateway_id
      instance_id    = null
      nat_gateway_id = null
    }
  ]

  subnet_ids = module.subnet_ec2.ids
}

The vpc_cidr = "10.0.0.0/16" means we're creating a VPC with 65,536 possible IP addresses. See here for an explanation on the CIDR notation.

The route table is connected to the EC2 subnet via; subnet_ids = module.subnet_ec2.ids. This subnet has full access to the internet via the cidr_block configuration; "0.0.0.0/0".

EC2

To create the EC2 instance, we just need to configure what machine we want and place it in the subnet where our Route Table is present.

In our EC2 module we configure the following:

locals {
  resource_name_prefix = "${var.namespace}-${var.resource_tag_name}"
}

resource "aws_instance" "_" {
  ami                         = var.ami
  instance_type               = var.instance_type
  user_data                   = var.user_data
  subnet_id                   = var.subnet_id
  associate_public_ip_address = var.associate_public_ip_address
  key_name                    = aws_key_pair._.key_name
  vpc_security_group_ids      = var.vpc_security_group_ids

  iam_instance_profile = var.iam_instance_profile
}

resource "aws_eip" "_" {
  vpc      = true
  instance = aws_instance._.id
}

resource "tls_private_key" "_" {
  algorithm = "RSA"
  rsa_bits  = 4096
}

resource "aws_key_pair" "_" {
  key_name   = var.key_name
  public_key = tls_private_key._.public_key_openssh
}

This creates:

1) AWS EC2 instance
2) With an elastic IP associated with that instance
3) A public/private key (PEM key) to access the instance via SSH.

Then we can call it:

module "ec2" {
  source = "../../modules/ec2"

  resource_tag_name = var.resource_tag_name
  namespace         = var.namespace
  region            = var.region

  ami           = "ami-07ebfd5b3428b6f4d" # Ubuntu Server 18.04 LTS
  key_name      = "${local.resource_name_prefix}-ec2-key"
  instance_type = var.instance_type
  subnet_id     = module.subnet_ec2.ids[0]

  vpc_security_group_ids = [aws_security_group.ec2.id]

  vpc_id = module.vpc.id
}

Four main things we need to supply the EC2 module (among other things):

1) Attach the EC2 instance to the subnet; subnet_id = module.subnet_ec2.ids[0],
2) attaches the security group; vpc_security_group_ids = [aws_security_group.ec2.id], a security group acts like a firewall.
3) Supply it with the VPC that it needs to be deployed in; vpc_id = module.vpc.id
4) AMI identifier, here's more on how to find Amazon Machine Image (AMI) identifiers.

EC2 Security group

So far we have a VPC setup, an EC2 instance and its subnet, and we've configured a reference to the security group the EC2 subnet is using.

A security group acts like a firewall for your subnet, what is allowed to go in ingress and what is allowed to go out egress of your subnet:

resource "aws_security_group" "ec2" {
  name = "${local.resource_name_prefix}-ec2-sg"

  description = "EC2 security group (terraform-managed)"
  vpc_id      = module.vpc.id

  ingress {
    from_port   = var.rds_port
    to_port     = var.rds_port
    protocol    = "tcp"
    description = "MySQL"
    cidr_blocks = local.rds_cidr_blocks
  }

  ingress {
    from_port   = 22
    to_port     = 22
    protocol    = "tcp"
    description = "Telnet"
    cidr_blocks = ["0.0.0.0/0"]
  }

  ingress {
    from_port   = 80
    to_port     = 80
    protocol    = "tcp"
    description = "HTTP"
    cidr_blocks = ["0.0.0.0/0"]
  }

  ingress {
    from_port   = 443
    to_port     = 443
    protocol    = "tcp"
    description = "HTTPS"
    cidr_blocks = ["0.0.0.0/0"]
  }

  # Allow all outbound traffic.
  egress {
    from_port   = 0
    to_port     = 0
    protocol    = "-1"
    cidr_blocks = ["0.0.0.0/0"]
  }
}

We allow traffic to come in from ports; 22 (SSH), 80 (HTTP), 443 (HTTPS), and we allow ALL traffic on all ports to go out. If you want to further tighten this down, profile which ports your application uses for outbound traffic to increase security.

For ingress you can further tighten this down by supplying a specific IP address that is allowed to connect on port 22.

RDS

We're almost done with the setup, only our database subnet and instance with security group needs to be configured.

locals {
  resource_name_prefix = "${var.namespace}-${var.resource_tag_name}"
}

resource "aws_db_subnet_group" "_" {
  name       = "${local.resource_name_prefix}-${var.identifier}-subnet-group"
  subnet_ids = var.subnet_ids
}

resource "aws_db_instance" "_" {
  identifier = "${local.resource_name_prefix}-${var.identifier}"

  allocated_storage       = var.allocated_storage
  backup_retention_period = var.backup_retention_period
  backup_window           = var.backup_window
  maintenance_window      = var.maintenance_window
  db_subnet_group_name    = aws_db_subnet_group._.id
  engine                  = var.engine
  engine_version          = var.engine_version
  instance_class          = var.instance_class
  multi_az                = var.multi_az
  name                    = var.name
  username                = var.username
  password                = var.password
  port                    = var.port
  publicly_accessible     = var.publicly_accessible
  storage_encrypted       = var.storage_encrypted
  storage_type            = var.storage_type

  vpc_security_group_ids = ["${aws_security_group._.id}"]

  allow_major_version_upgrade = var.allow_major_version_upgrade
  auto_minor_version_upgrade  = var.auto_minor_version_upgrade

  final_snapshot_identifier = var.final_snapshot_identifier
  snapshot_identifier       = var.snapshot_identifier
  skip_final_snapshot       = var.skip_final_snapshot

  performance_insights_enabled = var.performance_insights_enabled 
}

There are a lot of options here, lets grab the tfvars file to see what most of these variables contains:

# RDS
rds_identifier        = "mysql"
rds_engine            = "mysql"
rds_engine_version    = "8.0.15"
rds_instance_class    = "db.t2.micro"
rds_allocated_storage = 10
rds_storage_encrypted = false     # not supported for db.t2.micro instance
rds_name              = ""        # use empty string to start without a database created
rds_username          = "admin"   # rds_password is generated

rds_port                    = 3306
rds_maintenance_window      = "Mon:00:00-Mon:03:00"
rds_backup_window           = "10:46-11:16"
rds_backup_retention_period = 1
rds_publicly_accessible     = false

rds_final_snapshot_identifier = "prod-trademerch-website-db-snapshot" # name of the final snapshot after deletion
rds_snapshot_identifier       = null # used to recover from a snapshot

rds_performance_insights_enabled  = true

A few notes on the configuration I used here;

1) Instance sizing and encryption: for production make sure you use an instance size that is larger than a db.t2.micro such that you can use encryption on the storage layer.
2) Maintenance window: This day and time setting is used for patching of your instance.
3) Public access: Make sure to set public access off for obvious reasons, but this should already be the case anyway if your instance is hosted in a private subnet.
4) Backups: Two things regarding backups:
4.1) Providing the final snapshot identifier is useful when destroying the environment, it will automatically create a snapshot with the given name. If you do no supply this variable, you wont be able to remove the RDS instance with the terraform destroy command and you'll have to do this manually(!).
4.2) RDS supports automated backups, make sure to set the retention period (in days) correctly.
5) Query tracing: To enable in depth tracing of your queries and performance statistics, set performance_insights_enabled to true. This is very useful in analyzing slow queries and, generally, query performance.
6) Database password: The password for the database is generated, this can be done with this resource:

resource "random_string" "password" {
  length  = 16
  special = false
}

RDS Security group

Finally, we need to supply the security group configuration for RDS such that EC2 can communicate with our Database.

resource "aws_security_group" "_" {
  name = "${local.resource_name_prefix}-rds-sg"

  description = "RDS (terraform-managed)"
  vpc_id      = var.rds_vpc_id

  # Only MySQL in
  ingress {
    from_port   = var.port
    to_port     = var.port
    protocol    = "tcp"
    cidr_blocks = var.sg_ingress_cidr_block
  }

  # Allow all outbound traffic.
  egress {
    from_port   = 0
    to_port     = 0
    protocol    = "-1"
    cidr_blocks = var.sg_egress_cidr_block
  }
}

The above allows ingress from port 3306 and egress everything.

Conclusion

I hope this guide has been useful, please leave a comment below to let me know what you liked, did not like, suggestions and so on.

Next week I'll cover (Open)API security with configuration recommendations, and an AWS API firewall solution, AWS WAF.

Thanks for reading!

Serverless CI/CD for AWS Lambda

Rolf Streefkerk — Sun, 01 Mar 2020 16:45:44 +0000

Today I'll explain how to setup CI/CD (Continuous Integration / Continuous Development) with Terraform and AWS CodePipeline. To do that we use the same repository as the other articles in this series:

rpstreef / openapi-tf-example

Example of how you can use OpenAPI with AWS API Gateway, Also includes integrations with AWSLambda, AWS Cognito, AWS SNS and CloudWatch logs

and a new one has been added that contains the NodeJS code

rpstreef / openapi-node-example

OpenAPI Node code example

What are we going to cover today?

1. Setup Terraform (openapi-tf-example repo)
- 1.1 Source code
- 1.2 AWS CodeBuild
- 1.2.1 Buildspec file
- 1.3 Permissions
2. Setup NodeJS source code (openapi-node-example repo)
- 2.1 Gulp file
3. Conclusion
4. What's next

1. Setup Terraform (openapi-tf-example repo)

To setup our CI/CD we are going to use a service that will enable us to setup an automated pipeline for us. On AWS that is called CodePipeline, with this service we can setup any number of stages that specify how to automate our deployments.

Once we're done with this whole setup, we can have our code deployed automatically at every commit to the Master branch of our NodeJS repository.

But first, let's look at the CodePipeline setup we'll be using:

Please note the two distinct halves of the solution;

Source: This is the stage where the source code is retrieved from the designated Github repository
Build: The stage where AWS CodeBuild retrieves the source code, and can perform actions on it within a Linux or Windows environment.

1.1 Source code

The Source code stage of the pipeline looks like this:

resource "aws_codepipeline" "_" {
  name     = "${local.resource_name}-codepipeline"
  role_arn = module.iam_codepipeline.role_arn

  artifact_store {
    location = aws_s3_bucket.artifact_store.bucket
    type     = "S3"
  }

  stage {
    name = "Source"

    action {
      name             = "Source"
      category         = "Source"
      owner            = "ThirdParty"
      provider         = "GitHub"
      version          = "1"
      output_artifacts = ["source"]

      configuration = {
        OAuthToken           = var.github_token
        Owner                = var.github_owner
        Repo                 = var.github_repo
        Branch               = var.github_branch
        PollForSourceChanges = var.poll_source_changes
      }
    }
  }

The Github repo is configured using an OAuthToken connection and the arguments PollForSourceChanges makes sure every push to the configured Branch triggers this stage of the pipeline.

The output artifacts, source, will be stored in this S3 bucket:

resource "aws_s3_bucket" "artifact_store" {
  bucket        = "${local.resource_name}-codepipeline-artifacts-${random_string.postfix.result}"
  acl           = "private"
  force_destroy = true

  lifecycle_rule {
    enabled = true

    expiration {
      days = 5
    }
  }
}

We set an expiration on the bucket to clear it out every 5 days.

1.2 AWS CodeBuild

Now the second stage of the Pipeline is the AWS CodeBuild phase. This stage of the aws_codepipeline resource looks like this:

stage {
    name = "Build"

    action {
      name             = "Build"
      category         = "Build"
      owner            = "AWS"
      provider         = "CodeBuild"
      version          = "1"
      input_artifacts  = ["source"]

      configuration = {
        ProjectName = aws_codebuild_project._.name
      }
    }
  }

This is pretty self explanatory, we get the source code from our S3 bucket defined previously and we point to the CodeBuild configuration using the ProjectName.

In the aws_codebuild_project resource we set which artifacts we have and where they come from, the environment we're running the build in and any environment variables we need access to during the build run.

resource "aws_codebuild_project" "_" {
  name          = "${local.resource_name}-codebuild"
  description   = "${local.resource_name}_codebuild_project"
  build_timeout = var.build_timeout
  badge_enabled = var.badge_enabled
  service_role  = module.iam_codebuild.role_arn

  artifacts {
    type           = "CODEPIPELINE"
    namespace_type = "BUILD_ID"
    packaging      = "ZIP"
  }

  environment {
    compute_type    = var.build_compute_type
    image           = var.build_image
    type            = "LINUX_CONTAINER"
    privileged_mode = var.privileged_mode

    environment_variable {
      name  = "LAMBDA_FUNCTION_NAMES"
      value = var.lambda_function_names
    }

    environment_variable {
      name  = "LAMBDA_LAYER_NAME"
      value = var.lambda_layer_name
    }
  }

  source {
    type      = "CODEPIPELINE"
    buildspec = data.template_file.buildspec.rendered
  }
}

The details of the build run are in the source buildspec file.

1.2.1 Buildspec file

In this file we get to define all the build phases and the commands, among other things, it needs to execute for us. For a full overview of the whole specification, look at the official documents here.

What we need is pretty simple:

version: 0.2
run-as: root
env:
  variables:
    LAMBDA_FUNCTION_NAMES: $LAMBDA_FUNCTION_NAMES
    LAMBDA_LAYER_NAME: $LAMBDA_LAYER_NAME

phases:
    install:
        run-as: root
        runtime-versions:
          nodejs: 10
        commands:
            - pip3 install awscli --upgrade --user
            - npm install gulp-cli -g
            - npm install
    build:
        run-as: root
        commands:
            - gulp update --functions "$LAMBDA_FUNCTION_NAMES" --lambdaLayer "$LAMBDA_LAYER_NAME"
    post_build:
        commands:
           - echo " Completed Lambda & Lambda Layer updates ... "
           - echo Update completed on `date`
artifacts:
  name: update-$(date +%Y-%m-%d)

We need an up to date AWS CLI tool to update our Lambda and Lambda layer code, and then it needs to run our custom Gulp script:

gulp update --functions "$LAMBDA_FUNCTION_NAMES" --lambdaLayer "$LAMBDA_LAYER_NAME"

This script updates the code and configuration in all the functions specified in the LAMBDA_FUNCTION_NAMES environment variable, as well as the Lambda-layer in LAMBDA_LAYER_NAME environment variable.

1.3 Permissions

Both AWS CodePipeline and CodeBuild need permissions to run operations on AWS services.

For CodePipeline we need the following permissions set (see file ./policies/codepipeline-policy.json):

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "s3:GetObject",
        "s3:GetObjectVersion",
        "s3:GetBucketVersioning",
        "s3:List*",
        "s3:PutObject"
      ],
      "Resource": [
        "${s3_bucket_arn}",
        "${s3_bucket_arn}/*"
      ]
    },
    {
      "Effect": "Allow",
      "Action": [
        "codebuild:BatchGetBuilds",
        "codebuild:StartBuild"
      ],
      "Resource": "${codebuild_project_arn}"
    }
  ]
}

CodePipeline needs access to S3, to deploy the source artifacts and it needs to start the CodeBuild process.

In order for this CodeBuild phase to run properly, it needs permissions to access AWS S3, and to modify AWS Lambda and Lambda layer configurations. To that end we add the following policy from the file ./policies/codebuild-policy.json:

{
    "Version": "2012-10-17",
    "Statement": [
      {
        "Effect": "Allow",
        "Resource": [
          "*"
        ],
        "Action": [
          "logs:CreateLogGroup",
          "logs:CreateLogStream",
          "logs:PutLogEvents",
          "lambda:GetLayerVersion",
          "lambda:PublishLayerVersion",
          "lambda:PublishVersion",
          "lambda:UpdateAlias",
          "lambda:UpdateFunctionCode",
          "lambda:UpdateFunctionConfiguration",
          "lambda:ListFunctions"
        ]
      },
      {
        "Effect":"Allow",
        "Action": [
          "s3:GetObject",
          "s3:GetObjectVersion",
          "s3:GetBucketVersioning",
          "s3:List*",
          "s3:PutObject"
        ],
        "Resource": [
          "${s3_bucket_arn}",
          "${s3_bucket_arn}/*"
        ]
      }
    ]
  }

An improvement here would be to specify exactly which AWS Lambda resources the Lambda/Log policies apply to.

2. Setup NodeJS source code (openapi-node-example repo)

The Terraform side of things has been set up, now for the CodeBuild phase to work, it needs the Gulp script to execute Lambda and Lambda-layer update statements.

2.1 Gulp file

Below is the core of the Gulp script.

const options = minimist(process.argv.slice(2))
const lambdaLayer = options.lambdaLayer
const arrFunctions = options.functions.split(',')

At the top part we need to get the CLI parameters using minimist

var lambdaLayerJSON = {}

gulp.task('taskLambdaLayer', function (done) {
  const output = shellJS.exec('aws lambda publish-layer-version --layer-name ' + lambdaLayer + ' --zip-file fileb://' + projectName + '/' + zipLayerFilename)
  lambdaLayerJSON = JSON.parse(output)
  done()
})

Then we need to publish the new Lambda layer code first before we update the Lambda's. This function captures the output of this shell execution and JSON parses this into a file scoped variable. We need the new Lambda layer version from the output.

arrFunctions.forEach(function (lambda) {
  gulp.task('update_' + lambda, gulp.series(
    function updateCode (done) {
      shellJS.exec('aws lambda update-function-code --function-name ' + lambda + ' --zip-file fileb://' + projectName + '/' + zipFileName)
      done()
    },
    function updateConfig (done) {
      shellJS.exec('aws lambda update-function-configuration --function-name ' + lambda + ' --layers ' + lambdaLayerJSON.LayerVersionArn)
      done()
    }
  ))
})

gulp.task('lambdas',
  gulp.parallel(
    arrFunctions.map(function (name) { return 'update_' + name })
  )
)

Finally, we update the function Code with

function updateCode

and then we update the functions configuration to update the Lambda layer version in

function updateConfig

. Here we use the file scoped lambdaLayerJSON that stores the latest Lambda layer version ARN

3. Conclusion

All the parts of the solution are there, the only thing left is to integrate this into our openapi-tf-example repo and we're done!

module "cicd" {
  source = "github.com/rpstreef/tf-cicd-lambda?ref=v1.0"

  resource_tag_name = var.resource_tag_name
  namespace         = var.namespace
  region            = var.region

  github_token        = var.github_token
  github_owner        = var.github_owner
  github_repo         = var.github_repo
  poll_source_changes = var.poll_source_changes

  lambda_layer_name     = aws_lambda_layer_version._.layer_name
  lambda_function_names = "${module.identity.lambda_name},${module.user.lambda_names}"
}

The last two variables contain all our Lambda and Lambda layer function names that need to be updated once the code in the openapi-node-example repo changes.

4. What's next

With that you can quickly setup a CI/CD that works for your Serverless Lambda/Lambda-layer environment. Checkout my dedicated CI/CD repo for Serverless here:

rpstreef / tf-cicd-lambda

Terraform AWS CI/CD module for AWS Lambda compute

If you have any questions about the setup, comments, or improvement suggestions? Do comment and share!

Thanks for reading and till next week!

Automated Static Site With AWS CodePipeline and CloudFormation

Rolf Streefkerk — Sun, 23 Feb 2020 09:15:20 +0000

This is a guide to setting up and deploying your static site with Hugo on AWS infrastructure. I'll explain how to set up an automated deployment with AWS CodePipeline using a CloudFormation template, and an AWS Lambda function to generate our site and deploy it to S3.

Briefly I'll cover how to manually setup your domain on Route53 and a CodeCommit git repository. For these last two steps you could also use your own domain name solution and a GitHub repository.

I've dedicated a section about costs as well to show what kind of potential savings you could realize using a static site on AWS. See 7. What does it cost? to get an idea. The short story, you can run this static site with a decent amount of traffic for about $1.00 / month.

We'll cover the following topics:

1. Let's get started
2. Setup AWS S3 buckets
3. Setup AWS Route53 DNS
4. Setup AWS CodeCommit
5. AWS Lambda: Static site Generator
6. Custom AWS CodePipeline setup with AWS CloudFormation
7. What does it cost?
8. Comments

1. Let's get started

First we need an Amazon Web Services account, go to Amazon Web Services and sign up for the 1 year free tier account if you haven't already.

Also check out my GitHub repository alongside this guide at https://github.com/rpstreef/static-site-generator

Next up, let's look at a brief overview of what we'll be building and which AWS services are used and how they are connected.

AWS CodeCommit, git repository for your source code.
AWS Lambda, runs code on amazon managed servers.
AWS S3, hosting on durable file storage.
- Source code bucket: once the automation process runs, the master branch from your CodeCommit repo will be copied to this bucket.
- Site bucket: contains the generated website deployed by the Lambda function.
AWS CodePipeline, our workflow defined in a CloudFormation template.
AWS CloudFormation, infrastructure as code. Deploys our Lambda function among other things.
AWS Route53, (optional) domain name setup.
AWS CloudWatch, Lambda function execution logs can be found here for this stack.

2. Setup AWS S3 buckets

First, let's set up the website S3 bucket. Login to the AWS Console and head on over to S3:

Create your website bucket, choose a name in the format of the URL you will host it on. For instance, 'blog.fxaugury.com'.
Make sure in the bucket properties the bucket is set to website hosting and that CORS is enabled.
- CORS will make sure we can access same origin content.
Check permissions on your website S3 bucket, click on your website S3 bucket go to Permissions > view table "Manage public permissions".
- Make sure the "Everyone" group is set to read permissions for both "Object access" and "Permissions Access".

Please note: When testing your website S3 bucket, if the index.html does not display. The likely cause is that the index.html (and possible other files as well) file is not set to public.

3. Setup AWS Route53 DNS

Now that the S3 bucket for your static site has been setup, we can (optionally) connect it to our domain via Route53. Follow the steps outlined on Amazon to register a domain OR transfer an existing domain to Amazon. Once that is done, we need to adapt the hosted zone for the domain you want to use.

Go to Route53, select your domain name you want to use for the static site. Create a new CNAME record set:

Value: Set the S3 bucket endpoint URL.
- It's the URL that contains the string; "s3-website". To find the endpoint, go to your website bucket > properties > Static website hosting > "Endpoint"
Alias: No
TTL: 300 seconds, default propagation is sufficient. Unless you're migrating then you probably want to lower this number to avoid downtime.
Routing Policy: Simple
Name: The domain name you would like to use instead, e.g. blog.fxaugury.com
Save record set.

Done! After 300 seconds the changes should have propagated and you can use the domain name that you have registered.

4. Setup AWS CodeCommit

To setup CodeCommit, there are two steps that need to be executed:

Setup local git environment with Hugo and the SSH parameters.
Setup and connect to the AWS CodeCommit git repo.

First order of business, our local GIT repository and Hugo:

Install Hugo, hugo new site <site name>.
- See https://gohugo.io/overview/quickstart/ for a quick guide on how to start with Hugo.
In your Hugo site directory, init your git if you haven't already, git init.

Second, setup the SSH parameters. These instructions for ssh-keygen terminal application are for Linux based machines:

Open up a terminal and create your ssh key via ssh-keygen, follow the onscreen instructions.
To use your SSH Key, go to AWS Identity and Access Management (IAM) > Users > 'User name' > tab "Security credentials" > "SSH keys for AWS CodeDeploy":
- Copy paste the public key here.
- Create config file in ~/.ssh/config
- Paste below, where User is your SSH key ID you just created in IAM

Host git-codecommit.*.amazonaws.com
User APKAEIBAERJR2EXAMPLE
IdentityFile ~/.ssh/codecommit_rsa

Change permssions of the "Config" file just created

chmod 600 config

Ok the local setup is done for now. Head on over to AWS CodeCommit.

In the AWS console, go to CodeCommit > Create repository.
Creation of the repository is done, no we need to test the SSH connection to this repositry

ssh git-codecommit.us-east-2.amazonaws.com

Add this server to known server list.

The CodeCommit git url depends on where your CodeCommit git is created from, in my case it's created in Oregon (region: us-east-2)

We can succesfully connect via SSH to AWS CodeCommit. Now to add our local repository data to AWS CodeCommit.

Go to "Code" then "Connect", copy paste the url where it says "clone". Instead of cloning we'll push our local git to this remote:
In our local git repo directory, enter:

git remote add <repo name> ssh://git-codecommit.us-west-2.amazonaws.com/v1/repos/<repo name>

now we've added the remote repository, next thing is to push it.
Before you push it, make sure you have removed any git repo trace from the theme submodule in your Hugo themes folder. It will be recognized as such because each theme will have a .git directory.
Remove the .git directory, clear the cache and add the theme directory to our repo:

rm -r <theme directory path>/.git
git rm -r --cached <theme directory path>
git add <theme directory path>
git commit -am 'theme added'

We can now push our local git changes to CodeCommit.

git push --set-upstream <repo name> master

We should see all the code from our local git repository in AWS CodeCommit repository. For subsequent changes we can just use git push to push our changes to CodeCommit master branch.

5. AWS Lambda: Static site Generator

So we have setup an S3 bucket for your website, a domain name (optionally), and a local git and CodeCommit repo.

Next we'll review the static site generation program code briefly and upload the code as a zip file to a new S3 bucket.

To review the commented code, see: https://github.com/rpstreef/static-site-generator/blob/master/generate_static_site.py

CodeCommit source code pulled to our CodeCommit S3 bucket is encrypted, to be able to read it in our Lambda function we need to use signature version 4 for the S3 API to handle KMS encrypted files.

When you view these objects in your AWS S3 CodePipeline bucket, you'll see "Encryption AWS-KMS" in its properties and you wont be able to view it's contents.

The following two lines after the boto3 python sdk import will allow the application to read the contents of the files:

import boto3
from botocore.client import Config

S3 = boto3.client('s3', config=Config(signature_version='s3v4'))

Let's deploy the code and the hugo binary to a new S3 bucket:

Create a new bucket, any name will suffice, preferably in the same region as your CodePipeline.
Upload a zip file in this new bucket containing two files:
1. The hugo executable binary: hugo
  - Download the latest binary from here: https://github.com/spf13/hugo/releases
2. Generate static site program: generate_static_site.py

We'll need to enter these details when we execute the CloudFormation template in the next section of this guide.

6. Custom AWS CodePipeline setup with AWS CloudFormation

Almost there, now we need to setup a custom CodePipeline using a CloudFormation template. When the CloudFormation template is executed, we'll just have one more step to set this automation in motion.

First, we need:

an S3 Bucket to transfer source code from the Master (trunk) git branch.
- This will be created for us in this template when we don't have one yet.
a Lambda function to Generate, from that source code, a static site.
- We just uploaded the zip file containing the Lambda function program and hugo static site generator binary.
Custom CodePipeline template that will outline these steps.
- This is definined in the CloudFormation template we're going to discuss in this chapter.

Please review the full CloudFormation template here: https://github.com/rpstreef/static-site-generator/blob/master/cf_stack.yml

The custom CodePipeline will look as follows in our CloudFormation template:

# Fetch code from Master branch, execute Lambda function to generate, compress and deploy static site.
  CodePipeline:
    Type: "AWS::CodePipeline::Pipeline"
    Properties:
      Name: !Sub "${DomainName}-codepipeline"
      ArtifactStore:
        Type: S3
        Location: !Ref ExistingCodePipelineBucket
      RestartExecutionOnUpdate: false
      RoleArn: !Sub "arn:aws:iam::${AWS::AccountId}:role/${CodePipelineRole}"
      Stages:
        - Name: Source
          Actions:
            - Name: SourceAction
              ActionTypeId:
                Category: Source
                Owner: AWS
                Provider: CodeCommit
                Version: 1
              Configuration:
                RepositoryName: !Ref ExistingGitRepository
                BranchName: master
              OutputArtifacts:
                - Name: SiteSource
              RunOrder: 1
        - Name: InvokeGenerator
          Actions:
            - Name: InvokeAction
              InputArtifacts:
                - Name: SiteSource
              ActionTypeId:
                Category: Invoke
                Owner: AWS
                Provider: Lambda
                Version: 1
              Configuration:
                FunctionName: !Ref GeneratorLambdaFunction
              OutputArtifacts:
                - Name: SiteContent
              RunOrder: 1

The pipeline consists of just two steps;

"Source": CodeCommit repository with the name "!Ref ExistingGitRepository", will supply source code from the master branche. The !Ref part means it references what has been entered during CloudFormation stack creation for this field name.
"InvokeGenerator" will invoke the Lambda function by the name of "!Ref GeneratorLambdaFunction". This function will retrieve code stored in our CodePipeline bucket (made availble in step 1), transform it to a static site, GZIP it, copy the result to our website S3 bucket.

To be able for this CodePipeline to execute, we need authorization to do so. There are two Roles to be defined; the first for the CodePipeline itself, the second for the Lambda Function. The CodePipeline role must be able to assume roles set for AWS Lambda services and the CodePipeline.

# Allow all actions on Lambda and CodePipeline
  CodePipelineRole:
    Type: "AWS::IAM::Role"
    Properties:
      AssumeRolePolicyDocument:
        Version: "2012-10-17"
        Statement:
          - Effect: Allow
            Principal:
              Service:
                - "lambda.amazonaws.com"
                - "codepipeline.amazonaws.com"
            Action:
              - "sts:AssumeRole"
      Path: "/"
      Policies:
        - PolicyName: "codepipeline-service"
          PolicyDocument:
            Version: "2012-10-17"
            Statement:
              - Effect: "Allow"
                Action: "*"
                Resource: "*"

The role it will assume is outlined below, "${DomainName}-execution-policy". It defines the execution policies for the AWS Lambda function which will do all the heavy lifting in this pipeline. It's allowed all the actions specified in CodePipeline and S3 services:

#Allow Lambda access to; List, upload, get, delete, put object or acl on S3 and Set job results on CodePipeline
  LambdaExecutionRole:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Version: '2012-10-17'
        Statement:
          - Effect: Allow
            Principal:
              Service:
                - lambda.amazonaws.com
            Action:
              - sts:AssumeRole
      Path: "/"
      Policies:
        - PolicyName: !Sub "${DomainName}-execution-policy"
          PolicyDocument:
            Version: '2012-10-17'
            Statement:
              - Effect: Allow
                Action: "logs:*"
                Resource: "arn:aws:logs:*:*:*"
              - Effect: Allow
                Action:
                  - codepipeline:PutJobSuccessResult
                  - codepipeline:PutJobFailureResult
                Resource: "*"
              - Effect: Allow
                Action:
                  - s3:GetBucketLocation
                  - s3:ListBucket
                  - s3:ListBucketMultipartUploads
                  - s3:AbortMultipartUpload
                  - s3:DeleteObject
                  - s3:GetObject
                  - s3:GetObjectAcl
                  - s3:ListMultipartUploadParts
                  - s3:PutObject
                  - s3:PutObjectAcl
                Resource:
                  - !Join ["", ["arn:aws:s3:::", !Ref ExistingSiteBucket, "/*"]]
                  - !Join ["", ["arn:aws:s3:::", !Ref ExistingCodePipelineBucket, "/*"]]

The last bit of code in our CloudFormation template defines the Lambda resource to be created. It will provide an environment variable that points to our website bucket. The sizing, 256Mb of memory, should be sufficient to run the conversion and copy commands within 5-6 seconds per run.

GeneratorLambdaFunction:
    Type: "AWS::Lambda::Function"
    Properties:
      Description: !Sub "Static site generator for ${DomainName}"
      Role: !GetAtt LambdaExecutionRole.Arn
      MemorySize: 256
      Timeout: 300
      Runtime: !Ref GeneratorLambdaFunctionRuntime
      Handler: !Ref GeneratorLambdaFunctionHandler
      Code:
        S3Bucket: !Ref GeneratorLambdaFunctionS3Bucket
        S3Key: !Ref GeneratorLambdaFunctionS3Key
      Environment:
        Variables:
          SiteBucket: !Ref ExistingSiteBucket

To execute this template:

In your AWS Console go to; CloudFormation > Create Stack > Choose a template > Upload to S3 > Choose cf_stack.yml and then click next.
Set all the other properties to the values we have covered in the last chapters. Click next.
Optionally you can set tags for this stack to your liking. Click next.
Place a check mark at, "I acknowledge that AWS CloudFormation might create IAM resources." then press Create.

Now it will do it's work and it should show you in a minute or two that it is done successfully creating the stack.

To activate this automation we only need to push our local git repo changes to our CodeCommit master branch remote repo and it will trigger the CodePipeline and subsequently our Lambda function to deploy our static site with the latest changes!

Now execute a push to your AWS CodeCommit repo and watch the CodePipeline execution do it's thing to automatically generate the static site and deploy it to your website S3 bucket.

7. What does it cost?

Now that we've finally got the whole thing running :) What does it actually cost to run on a monthly basis:

Service	Item	Cost	Subtotal
S3	5 GB storage	$0.12	$0.12
	5000 put/list requests	$0.03	$0.15
	100,000 Get and Other Requests	$0.04	$0.19
	Inter region bucket transfers of 1GB/month (depends on CodePipeline/Commit region availability)	$0.02	$0.21
Route53	Hosted zone	$0.50	$0.71
	Standard queries: 1 million / month	$0.40	$1.11
CodePipeLine	1 Free pipeline per month	$0.00	$1.11
CodeCommit	First 5 users free with 50GB storage and 10,000 git requests/month	$0.00	$1.11
Lambda	Memory: 256Mb you get 1,600,000 seconds of compute time for free	$0.00	$1.11
Free tier	Discount	-$0.14	$0.97
			$0.97

Compare this to running your blog on Wordpress (no custom domain costs included):

Service	Monetize	Cost
Static site on AWS	Yes	$1.11
Wordpress Free (No custom domain)	No	$0.00
Wordpress Personal	No	$2.99
Wordpress Premium	Yes	$8.25

If you need; blog monetization, analytics, storage flexibility, your own branding, and google analytics. Running a static site on AWS offers much better value at about 15% of the cost of the Wordpress Premium option.

8. Comments

I hope this guide has been useful, please leave a comment below to let me know what you liked, did not like, suggestions and so on.

Next week I'll continue my OpenAPI series (not enough time this week) and I'll cover what I promised, CI/CD with CodePipeline and Terraform.

Thanks for reading!