Forem: James

Creating Live Dashboards With QuickSight

James — Thu, 20 May 2021 14:20:23 +0000

Liquid syntax error: 'raw' tag was never closed

Container Images for AWS Lambda with Python

James — Mon, 10 May 2021 18:32:09 +0000

One of the best things about AWS Lambda is the variety of ways that you can create a serverless function. For example, you can dive right into the console, or use approaches like Chalice or the Serverless Framework, to name a few. For me, the latest way, announced at re:Invent 2020, is the most efficient way of testing your serverless function locally and dealing with large or awkward dependencies in your code. As you'd expect, this works perfectly with AWS SAM. This post will give you a quickstart into what you need to do to build a container-based function using containers & Python.

In my case, I was building a function that would read data from Firestore in Google Cloud, run a data transformation and store the result in S3. I used to swear by Chalice, and built my application but found that it couldn't bundle the GRPC dependency - I would have to build it myself. While the solution to this is just to build the dependency and package it up in the vendor directory of my Chalice application, I used the opportunity to finally try out container support.

Prerequisites

If you're reading this, you're probably using Python already. As well as having Python installed, you'll need to have Docker too.

Building a Docker container with dependencies

As always, the AWS documentation will guide you through the basics.

First you'll need to install the Python runtime interface client using pip install awslamdaric

Next, create a Dockerfile that references the base image you are using. In the case below, I was using Python 3.8



FROM public.ecr.aws/lambda/python:3.8

Presuming your code is in app.py with a handler as the function, you'll add these two lines too



COPY app.py   ./
CMD ["app.handler"]

Dealing with dependencies is pretty straightforward. As with all Python application, all the modules you use should be defined in requirements.txt like this:



google-cloud-firestore
google-auth

To make sure all dependencies are correctly bundled, just add these two lines to your Dockerfile



COPY requirements.txt ./
RUN pip install -r requirements.txt

It's as simple as that.

Building & Testing Locally

The huge benefit to working with containers is the ability to completely test your Lambda function locally before deploying to your AWS account. Let's take a look into how to do that:

Build Your Container

The standard docker build syntax applies here.



docker build -t my-container-image .

The commmand above creates 'my-container-image' in the local directory

Running Locally

Typically you would run your container as follows



docker run -p 9000:8080  my-continer-image:latest

However, it's very likely that you're interacting with AWS services (in my case S3), so you will need to ensure the access credentials exist for your function documentation here. These credentials can be set using the following environment variables:

AWS_ACCESS_KEY_ID
AWS_SECRET_ACCESS_KEY
AWS_SESSION_TOKEN
AWS_REGION

Provided you've got the credentials set on a user (which should be using the same policy as your Lambda would) you can pass them through as following when running locally:



docker run -p 9000:8080  
-e AWS_ACCESS_KEY_ID='ACCESS_KEY_ID_GOES_HERE'
-e AWS_SECRET_ACCESS_KEY= 'SECRET ACCESS KEY GOES HERE'
-e AWS_REGION= 'eu-west-1'
my-container-image:latest

Finally, to invoke events, you can pass a payload through while the container is running as follows:



curl -XPOST "http://localhost:9000/2015-03-31/functions/function/invocations" 
-d '{"param1":"parameter 1 value"}'

Note that you can also use the AWS_LAMBDA_FUNCTION_TIMEOUT environment variable to set a timeout for your function in milliseconds.

Running your Lamba

Now that you have a working container, you need to deploy it to your AWS account

Deploy Your Container

Authenticate the Docker CLI to your ECR registry



aws ecr get-login-password --region us-east-1 | docker login --username AWS --password-stdin <AWS ACCOUNT ID>.dkr.ecr.<REGION>.amazonaws.com

The first time that you are going through this process, you will need to create a repository on ECR with the name that you want to use



aws ecr create-repository —repository-name my-container-image

After this you just need to tag your image before pushing it to ECR



docker tag  my-container-image:latest <AWS ACCOUNT ID>.dkr.ecr.<REGION>.amazonaws.com/my-container-image:latest



docker push <AWS ACCOUNT ID>.dkr.ecr.<REGION>.amazonaws.com/my-container-image:latest

Your container is now ready to be used in a Lambda function.

Creating Your Lambda Function

Just open up the AWS console, go to AWS Lambda and click on Create Function

From there you'll be able to create a function from a container image, and simply select the image you need from ECR through Browse Images.

Having taken this console based approach, you are obviously going toneed to attach a role to your Lambda at a minimum.
If you want to be more disciplined, and less console dependent, on how you create your container based Lambda functions, check out https://twitter.com/edjgeek post on Using Container Image Support for AWS Lambda with AWS SAM

Extra Resources

As well as the documentation resources linked in this article, make sure to check Chris Munns' excellent talk on Container Image Support in AWS Lambda from re:Invent 2020

Running An Online Meeting: A Beginners Guide

James — Sat, 07 Nov 2020 15:15:49 +0000

Running community meetups used to be pretty straightforward; get the speakers, organise the location and you’re set. Now that most events are run online and virtually, meetup organisers have a new set of challenges. The good news is that once you get around the initial learning curve, you’ll find it’s full of new opportunities too. Here’s my experience in bringing our local developer meetup to an online stage.

Find an Event Platform

The default option for a lot of people seems to be to use Zoom for the event, maybe recording the meeting to share later. This works fine, but does it provide the level of engagement that you’d like? Having attended one of the AWS Community Days, I noticed the event was using CrowdComms, which allows clean navigation of the event schedule and the ability to interact; a much better experience for attendees than I had seen in the past.

Then I remembered Vito, a product that Paul Campbell started to build at the beginning of the COVID-19 crisis to provide a better event experience. I sat in on one of the weekly demos of the product to see how it works, and it’s a deceptively simple interface that allows you to create a much better experience for you attendees in seconds.

You get a place to stream your video content to (more on that shortly) and a number of options for page types to include. This will include an attendee list, and a posts section, which allows you to have a mini-social-network experience for people during (and after) the event where they can ask questions, or just discuss the event they’re watching.

Our Vito Hub for CorkDev

Once you’ve worked out your streaming tools, you simply use the key provided by Vito, and you can then test that the whole thing works before going live. The great thing about it is that your stream will automatically be recorded in Vito, so once completed, you can upload the video to the hub for people to watch later. So you are building up content for your hub as you go along.

That’s the great thing about Vito for our meetup; people can drop in anytime and watch over again, and use other parts of the hub for things like adding local job listings. It saves you the work of creating a dedicated site for your meetup (we had previously), and just leave all the hard work to Vito.

Work Out Your Streaming Options

When I was introduced to the options for streaming software, I have to say it was a bit overwhelming! Paul gave me some solid advice though, and I went with OBS Studio in the end, purely because of the fact that it was free.

With OBS you get a decent amount of options for managing the content that will be streamed, such as backgrounds and titles for you videos. You also get a “studio mode” that allows you to see the live content on one side while preparing what the next cut will look like on the the editing side.

My setup for the live event: edit on laptop, incoming video stream on the left, Vito on the right

I sunk a lot of time into working out my audio issues though. On a Mac, sharing your desktop audio is a little bit tricky. To cut a long story short, follow the instructions here to set up iShowU Audio Capture. Make sure that the sample rate you choose in the Multi-Output Device settings matches those of the master device you’re using (like Airpods in my own case).

On the topic of audio, you can easily mute and unmute your own microphone as the event happens in the audio mixer section. During the event make sure you stay aware of this!

When you’re bringing someone else into the event, you could use Zoom/Microsoft Teams/FaceTime or your video chat tool of choice. However, if you want to go that extra mile and have even more control, I’d recommend OBS.Ninja where you can create your own “room” for a group chat specific to your meetup. Each person in the room gets their own URL which can be then added to OBS as a stream. For Mac’s it a little more complicated (again) so I used OBSN to share the content coming from OBS.Ninja.

Make sure to put a lot of time in getting your setup just right and use the livestream testing tool on your Vito Hub to make sure that it works.

Work Hard on Event PR

It goes without saying, but you’ll have to work hard on pushing the event out to the public. You’ll need to send a lot of annoying reminders on Twitter, and if you’re already using Meetup, make sure to use the tools there to push out reminders via email for event too.

Running an online event as described here is an opportunity to get your branding right too. I used Canva to create some basic backgrounds and titles for the event. Variations of those assets were used in the Vito hub too. Having a consistent colour/font/style combination really is worth it. Particularly when the event will be recorded and stored forever.

After that, you’ve just got to get ready for the infinite loop and plan your next events! Some advice that I got that I’ll recycle here is: make sure you plan out your event calendar. Try to have topics and ideally people, lined up for the next three events. That way you’ll never have that last minute panic and scramble in trying to work out your next great event.