Forem: Matthew Vielkind

Semantic Search with Haystack and Elastic

Matthew Vielkind — Thu, 09 Jun 2022 17:21:43 +0000

In this post, I'll show how you can build a semantic search application using the Haystack framework and Elastic. In this tutorial, you will:

Create an Elastic document store in Haystack
Generate text embeddings for documents in your document store
Build a semantic search pipeline to retrieve documents

If you just want to get right to the code you can go right to the repo and look at the notebook.

Create the Elastic Document Store

For this tutorial, we will be storing documents in an Elastic document store. A feature I like with Haystack is you can easily swap out different document stores. For example, if you prefer to use FAISS instead of Elastic, you can implement that here without having to change other components in the pipeline. You can get the complete list from the Haystack Document Store docs.

First, we need to initialize the Elastic document store. Haystack has a function launch_es() that will run a subprocess to run Elastic within a Docker container. You will need to have Docker running for this command to complete.

from Haystack.utils import launch_es

launch_es()

Now we can connect to the Elastic instance and configure the document store.

from haystack.document_stores import ElasticsearchDocumentStore

document_store = ElasticsearchDocumentStore(
    host="localhost",
    username="",
    password="",
    index="document",
    create_index=True,
    similarity="dot_product"
)

To create the document store we provide the information about how to connect to the Elastic instance. We also create a new index called document within our Elastic instance where our documents will be stored.

Finally, we also define a similarity function, dot_product, that will be used when comparing document vectors.

The ElasticDocumentStore within Haystack has a bunch of configurations you can leverage that you can find in the docs.

Process Text and add to Document Store

Now documents can be added to the document store. Most Haystack tutorials use a collection of Wikipedia articles related to Game of Thrones, so we'll use that same data source here.

The data can be downloaded from S3 and stored locally. Haystack provides a helper function to fetch and store the data in a local directory.

from haystack.utils import clean_wiki_text, fetch_archive_from_http
from haystack.utils.preprocessing import convert_files_to_docs

# Read data from S3. Write text to the specified directory.
doc_dir = "data/article_txt_got"
s3_url = "https://s3.eu-central-1.amazonaws.com/deepset.ai-farm-qa/datasets/documents/wiki_gameofthrones_txt.zip"
fetch_archive_from_http(url=s3_url, output_dir=doc_dir)

You should see ~180 text files in the data/article_txt_got directory when the function completes.

The raw documents need to be put into a format Haystack can load into the Document Store. The convert_files_to_docs function provides a convenient way to process raw text documents and put them in a Document format for Haystack. We will provide a cleaning function for this example that removes redundant line breaks, extremely short lines, and empty paragraphs.

docs = convert_files_to_docs(dir_path=doc_dir, clean_func=clean_wiki_text)

Here is what an example document looks like:

<Document: {'content': "Linda Antonsson and Elio García at Archipelacon on June 28, 2015.\n'''Elio Miguel García Jr.''' (born May 6, 1978) and '''Linda Maria Antonsson''' (born November 18, 1974) are authors known for their contributions and expertise in the ''A Song of Ice and Fire'' series by George R. R. Martin, co-writing in 2014 with Martin ''The World of Ice & Fire'', a companion book for the series. They are also the founders of the fansite Westeros.org, one of the earliest fan websites for ''A Song of Ice and Fire''.", 'content_type': 'text', 'score': None, 'meta': {'name': '145_Elio_M._García_Jr._and_Linda_Antonsson.txt'}, 'embedding': None, 'id': '41655cc804bb07b1569f3118ce70e05'}>

The content field is the document's text, which will be used for generating the text embeddings. The meta field stores other attributes of a document. These will be stored as fields within the Document Store. In this example, we're storing the name of the text file for the document.

The list of documents can now be written to the Elastic Document Store. Right now all but 10 of the documents will be written to the Document Store. The remaining 10 will be used later.

document_store.write_documents(docs[:-10])

Create Document Embeddings

After writing the documents to the document store, embeddings can be generated for the documents with a retriever.

Here we'll use the DensePassageRetriever, which lets us define separate embedding models for queries and documents. We will use the separated pretrained DPR models from Facebook for queries and text. The use_gpu flag is also set to True, which tells the retriever to use GPUs if they are available. If not GPUs are available that's okay, Haystack will fallback to using CPU. Similar to the document store Haystack has a number of configurations to customize the retriever behavior that you can find in the documentation.

from Haystack.nodes import DensePassageRetriever

retriever = DensePassageRetriever(
    document_store=document_store,
    use_gpu=True,
    query_embedding_model="facebook/dpr-question_encoder-single-nq-base",
    passage_embedding_model="facebook/dpr-ctx_encoder-single-nq-base"
)

After defining the retriever, the embeddings of the documents can up generated. If you're using a CPU, this step will take a while, so get some coffee, stretch, or do something while the process runs.

document_store.update_embeddings(retriever)

Querying the Document Store

Now that we have documents with embeddings, we can retrieve documents for a given query using the Haystack DocumentSearchPipeline wrapper for the retriever we just created.

from Haystack.pipelines import DocumentSearchPipeline

pipeline = DocumentSearchPipeline(retriever)

We can define our query and pass it to the pipeline along with other parameters for our retriever, like the number of results to return in this example. In this case, we want to retrieve documents related to the "Red Wedding" from Game of Thrones.

query = "what is the Red Wedding?"
result = pipeline.run(query, params={"Retriever": {"top_k": 2}})

When a query is sent to the pipeline, the query embeddings are generated using the query model defined in the retriever. Then a dot product vector similarity search is performed against the document store.

Haystack has a helper function print_documents() to display the results in a prettier format.

print_documents(result, max_text_len=100, print_name=True, print_meta=True)

Looking at the first result, you should see a document about "The Rains of Castamere", which is the name of the episode where "The Red Wedding" occurred, so given our query, this is a very relevant result.

Adding New Documents

In this example, we created a document store from scratch, uploaded text documents, and generated embeddings for those documents. What if we wanted to write new documents into our document store? To avoid the computation time of re-generating embeddings for all the documents, you can use the update_existing_embeddings parameter of the update_embeddings method.

document_store.write_documents(docs[-10:])
document_store.update_embeddings(
    retriever,
    update_existing_embeddings=False
)

By setting updated_existing_embeddings=False only documents without an embedding in the document store will be updated. This parameter can be helpful when making incremental updates to documents in your document store.

Wrapping Up

In this tutorial, I've shown how you can build a basic semantic search system using Haystack and Elastic. I talked about some different configurations you can utilize to customize this pipeline. Still, Haystack has a bunch of other configurations you can leverage, so I highly recommend you check out the documentation to see what is available to you.

I'm working on a couple more examples, so if there is something specific you'd like to see built with Haystack let me know!

Happy Tinkering!

Using Python and Airtable

Matthew Vielkind — Thu, 21 Oct 2021 12:07:04 +0000

Airtable is a cloud-based relational database that simplifies data storage without having to write SQL. Airtable has a REST API that can be used to perform common operations on your Airtable. The API documentation only contains curl and Javascript examples, so this tutorial will look at how you can interact with Airtable using Python complete with examples. You'll learn how to add, retrieve, update, and delete records in Airtable through the Airtable REST API and also talk about some of the common gotchas and limitations.

For this tutorial, we will be using scores from a round of golf as a toy dataset. You will need to create a table in Airtable with the following four columns and data types:

Date (string)
Hole (integer)
Par (integer)
Score (integer)

All the code referenced here is available in a GitHub repo. In the repo you will find a file airtable.py with example functions you'll build here. There is also a Jupyer notebook, AirtablePractice.ipynb in the repo you can use to follow along.

Authentication

First, let's look at the authentication for submitting a request.

Submitting a request to the Airtable REST API requires three pieces of information:

An API authentication token.
The ID of your Airtable base.
The name of the sheet.

Airtable uses a bearer token to authenticate each API request, which should be saved in a .env file to keep it secret.

You can find your API token by going to your account page. Scroll down to the API section of your account page, and you will see your token.

Copy the token value and paste it into your .env file naming it AIRTABLE_TOKEN. Make sure you keep this token private and do not share it with anyone! If you're committing files to a GitHub repo (or any other source control system), make sure you exclude the .env to avoid accidentally exposing this information!

Next, let's find the Airtable base ID. Make sure you are logged into your Airtable account and go to the API docs. Select the base you want to work with from the list. On the next page, you will see the ID of your Airtable Base. Copy the value and add it to your .env file naming it AIRTABLE_BASE_ID.

Finally, make a note of the sheet name you want to modify via the API. You'll need that to complete the URL for making requests.

With all the variables assembled to make our requests, it's time to put it together into a python file. Create a new file named airtable.py. Include the following at the top of the file:



import os
from python_dotenv import load_dotenv
load_dotenv()

AIRTABLE_TOKEN = os.getenv("AIRTABLE_TOKEN")
AIRTABLE_BASE_ID = os.getenv("AIRTABLE_BASE_ID")

Using the python_dotenv library, the variables you included in your .env are read into two variables, AIRTABLE_TOKEN and AIRTABLE_BASE_ID, allowing you to utilize them in the rest of your script.

With the authentication sorted out, we need to build the endpoint you'll send requests to. The basic structure of an Airtable API endpoint has three components:

1) The root of the URL, api.airtable.com/v0
2) The Airtable Base ID
3) The sheet name

The root URL, api.airtable.com/v0, will remain constant across all requests. The Base ID will depend on the specific Airtable Base you are using. Since this tutorial will use the same base, we can create another variable, AIRTABLE_URL, at the top of the file that you can use to construct the endpoint for each request. The code at the top of airtable.py should now look like this:



import os
from python_dotenv import load_dotenv
load_dotenv()

AIRTABLE_TOKEN = os.getenv("AIRTABLE_TOKEN")
AIRTABLE_BASE_ID = os.getenv("AIRTABLE_BASE_ID")

AIRTABLE_URL = f"https://api.airtable.com/v0/{AIRTABLE_BASE_ID}"

Using the REST API

The Airtable REST API allows you to perform a few core actions: add a record, retrieve records, update records, and delete records. A nice feature of the https://airtable.com/api Airtable API documentation is after you log in to your account, you can view the documentation specific to the Airtable base you want to use. The documentation provides API usage documentation for curl and Javascript. Below we'll look at examples of how to use the API using Python.

To demonstrate how the API works, we will use Airtable to track golf scores. We'll have a sheet named golf-scorecard containing four columns: the date, hole number, par, and the number of strokes for the hole.

Adding Records to Airtable

You can add records to your Airtable by sending a POST request. The request includes a dictionary with a key, records, defining an array of the records that will be added to the sheet. Each record in the array is a dictionary with a key fields, containing a dictionary that maps values to the columns in the sheet. Here's an example data payload for adding scores to the Airtable for the first two holes.



new_data = {
    "records": [
        {
            "fields": {
                "Date": "2021-09-22",
                "Hole": 1,
                "Par": 4,
                "Score": 5
            }
        },
        {
            "fields": {
                "Date": "2021-09-22",
                "Hole": 2,
                "Par": 4,
                "Score": 4
            }
        }
    ]
}

When you define records to add to Airtable, make sure the data types of the data you want to add match the data types for the column in Airtable. If the data types do not match, you will get an error.

Next, you need to write an API request to add this data to the Airtable. Create a new function in airtable.py called add_new_scores that will take the new_data scores as input and add them to the Airtable.



def add_new_scores(scores):
    """Add scores to the Airtable."""
    url = f"{AIRTABLE_URL}/golf-scores"
    headers = {
      'Authorization': f'Bearer {AIRTABLE_TOKEN}',
      'Content-Type': 'application/json'
    }

    response = requests.request("POST", url, headers=headers, data=json.dumps(scores))

    return response

The first part is building the URL referencing the AIRTABLE_URL variable we created earlier. The only addition is the name of our sheet, golf-scores, which has been added to the URL to instruct Airtable that we want to add records to the golf-scores sheet of our base. If you want to add scores to a different sheet, you can substitute the sheet's name in the URL.

Next, the request header is defined. The header passes along the AIRTABLE_TOKEN so that Airtable can confirm the request is authorized.

Finally, the POST request is submitted using the request module, passing the scores payload along with the request.

Let's run the code and look at the outcome.

One last note about adding records, each request can add a maximum of 10 records to your sheet. If you want to add more than 10 items, you will have to do so across multiple requests. Let's look at an example implementation below.



def chunk(arr, n):
    for i in range(0, len(arr), n):
        yield arr[i:i + n]

for c in chunk(data["records"], 10):
    chunk_data = {
        "records": c
    }

    response = airtable.add_new_scores(chunk_data)

The chunk function returns a generator that will break down the records into groups of size n where n should not be greater than 10. Next, you can iterate over the n sized chunks from your data and call the add_new_scores function during each iteration to add the records to Airtable.

Reading Records from Airtable

You can read records from Airtable by issuing a GET request to the API endpoint of a sheet. Let's create a new function in your airtable.py file to read records from the golf-scores sheet.



def get_golf_scores():
    """Add scores to the Airtable."""
    url = f"{AIRTABLE_URL}/golf-scores"
    headers = {
      'Authorization': f'Bearer {AIRTABLE_TOKEN}',
      'Content-Type': 'application/json'
    }

    response = requests.request("GET", url, headers=headers)

    return response

Like adding records, you define the URL endpoint to the golf-scores sheet and include your authentication header information. The two differences are that you will submit a GET request instead of a POST request, and you do not need to pass a data payload along with the request.

The API will respond with an array of records in your Airtable sheet. Each record has an id uniquely identifying the record, a field dictionary with the values for the record, and a createdTime key. An example response is below.



{
    "records":
        [
            {
                "id": "<AIRTABLE_RECORD_ID>",
                "fields": {
                    "Date": 2021-09-22,
                    "Hole": 1,
                    "Par": 4,
                    "Score": 5
                },
                "createdTime": <some_time>
            }
            ...
        ]
}

You may want only to retrieve records matching specific criteria from your Airtable. Airtable provides several options to customize querying your table. You can instruct Airtable to return particular fields, specify filtering criteria, sort records, and more. For a complete list of functionality, consult the Airtable API.

To demonstrate how to customize API calls to apply a filter to results, let's create a new function, get_scores_for_hole, to your file. In this function, you will query the golf-scores sheet in Airtable and return only the records for the hole number you send to the function. Instead of returning all table fields, your response will only contain the Hole and score fields for each record. Enter the following function in your file.



def get_scores_for_hole(hole):
    """Get scores for a specific hole."""
    url = f"{AIRTABLE_URL}/golf-scores"
    headers = {
      'Authorization': f'Bearer {AIRTABLE_TOKEN}',
      'Content-Type': 'application/json'
    }

    params = {
        "fields": ["Hole", "Score"],
        "filterByFormula": f"Hole={hole}"
    }

    response = requests.request("GET", url, headers=headers, params=params)

    return response

The difference with this function is the inclusion of the params object in your API request. The fields key instructs Airtable what fields to return in the response, and filterByFormula specifies the filter to apply to the sheet.

Airtable allows you to build more complex filters. For a complete list of functionality available to filter records and manipulate the Airtable response for retrieving records, you can check out the Airtable documentation.

By default, when you submit a GET request, Airtable will return a maximum of 100 records. If your response contains more than 100 records, Airtable will provide an offset value in its response. To retrieve more than 100 records, you will need to issue a new request that includes the value of offset. The function get_scores_by_page below accepts a parameter offset to instruct Airtable to retrieve the next page of results.



def get_golf_scores_by_page(offset=None):
    """Retrieve records from Airtable and apply offset is necessary."""
    url = f"{AIRTABLE_URL}/golf-scores"
    headers = {
      'Authorization': f'Bearer {AIRTABLE_TOKEN}',
      'Content-Type': 'application/json'
    }

    params = {
        "pageSize": 100
    }

    if offset:
        params["offset"] = offset

    response = requests.request("GET", url, headers=headers, params=params)

    return response

As long as your response contains the offset value, you will iteratively retrieve more records. When all records are retrieved, Airtable will stop sending offset in the response. An example implementation is below.



results = airtable.get_golf_scores_by_page()

while "offset" in results.json():
    results = airtable.get_golf_scores_by_page(response.json()["offset"])

Update a Record

You may need to change your data after adding it to Airtable. Updating existing records can be done in two different ways, PATCH or POST.

PATCH: to update specific fields in a record.
PUT: to clear the record and replace all values with new values.

First, let's look at the PATCH request. A PATCH request allows you to update specific fields of a record while keeping other values unchanged. You can pass a list of records with the values you'd like to change as a payload to the request. For example, if you wanted to update the score for a record, your payload would look something like this:



updated_records = {
    "records": [
        {
            "id": <record_id>,
            "fields": {
                "Score": 4
            },
        }
    ]
}

To update a record, you must send the record ID you wish to update as a part of the payload. You can find the record ID in the response from retrieving records that you looked at in the previous section.

Let's put this all together. Create a new function update_record_fields in your Python file:



def update_record_fields(updated_records):
    """Update specific field values for a record."""
    url = f"{AIRTABLE_URL}/golf-scores"
    headers = {
        'Authorization': f'Bearer {AIRTABLE_TOKEN}',
        'Content-Type': 'application/json'
    }

    response = requests.request("PATCH", url, headers=headers, data=json.dumps(updated_records))

    return response

The function accepts a parameter, updated_records, defining the records and values to be updated. Replace <record_id> in the updated_records object with an ID from your table, then use it as input to update_record_fields. If you look at the record in Airtable, the value for the Hole_Number field should have changed, and all other values should remain the same.

Another way you can update records is by using a PUT request. The main difference between PUT and PATCH is that a PATCH request will update the field values you specify in your payload and keep all other values the same. A PUT request will update the field values you specify but delete values for the other fields.

Let's look at the difference in action. Create a new function replace_record_fields:



def replace_record_fields(updated_records):
    """Updates the records."""
    url = f"{AIRTABLE_URL}/golf-scores"
    headers = {
        'Authorization': f'Bearer {AIRTABLE_TOKEN}',
        'Content-Type': 'application/json'
    }

    response = requests.request("PUT", url, headers=headers, data=json.dumps(updated_records))

    return response

The function is nearly identical to update_record_fields. The only difference is that we are using a PUT request instead of a PATCH here. This time let's try to change the score of the first hole to 6. Use the following payload as input to replace_record_fields, use the same record ID as the previous request and let's see what happens.



updated_records = {
    "records": [
        {
            "id": <record_id>,
            "fields": {
                "Score": 6
            },
        }
    ]
}

When you look at Airtable, you should notice that the score for the hole has updated to 6, but all the other record values are empty. The additional fields are empty because they are not included in the payload, representing the difference between updating records with a PUT versus PATCH request. A PATCH request will preserve the values for fields not specified in the update. PUT will clear values that are a part of the update payload.

One last note, similar to adding records, you can only update 10 records per request.

Delete a Record

The last action you can take via the Airtable API is deleting records. To delete a record, you will need to know the IDs of the records you want to delete. You can include the list of IDs as a query string parameter in a DELETE request to instruct Airtable to delete those records. Create a new function delete_records to your file.



def delete_records(records):
    """Delete the records."""
    url = f"{AIRTABLE_URL}/golf-scores"
    headers = {
        'Authorization': f'Bearer {AIRTABLE_TOKEN}',
        'Content-Type': 'application/json'
    }

    params = {
        "records[]": records
    }

    response = requests.request("DELETE", url, headers=headers, params=params)

    return response

Pass a list of record IDs as input to the function. When you look at your Airtable, the records should no longer be in the sheet.

Similar to previous actions you've taken with Airtable, you can only delete up to 10 records in a single request.

Conclusion

There it is! This tutorial showed a few of the things you're able to do with the Airtable REST API using Python.

Happy building!

Stream Notes- Color Transitions in pycairo

Matthew Vielkind — Thu, 19 Aug 2021 12:47:05 +0000

One challenge with live coding on Twitch is maintaining continuity for viewers. Longer projects are can be especially difficult for viewers to follow along. To help with this I'm putting together these quick cliff notes about what was covered on previous streams along with resources I found helpful that viewers can quickly reference without having to rewatch an entire stream. I also count myself as a viewer and doing this exercise helps me stay organized by providing a quick way to recap what was covered previously. So here we go!

Project Overview

Before getting into the stream summary here is a little background on the project I'm working on. As my wife and I are planning for the birth of our first child I wanted to create a unique gift for her to remind her of all the love and support she has around her. In my previous streaming project I built a conversational voice search into the Peloton API, so I wanted to incorporate a voice component here. My first thought was to setup a Twilio voicemail box for people to leave greetings and words of encouragement for her.

After some more consideration I wanted to expand on that idea to include something tangible representing those voicemail messages. Years ago I was intrigued by fractals and remember a woman who was creating t-shirt designs based on fractals. Change your inputs a little and you get a new distinctive output. I began thinking could I use word vectors as input and generate a unique piece of algorithmic art representing all the messages? Over the past month I've been tinkering with generative art using word vectors as input so the art encodes real meaning in an abstract way.

To date I've been using transcripts of the My Favorite Murder podcast for testing. An example image is below where the black line represents Georgia and the red line is Karen. The lines at the bottom depict the podcast name and their tagline "Stay Sexy and Don't Get Murdered". Not knowing anything about generative art (or much about art in general) I'm pretty happy with the result so far. More tinkering is needed, but I like the general style of embedding meaning into the image without that meaning being obvious. There's still bunches to learn so if this project sounds interesting feel free to follow along on Twitch!

Stream Notes

In previous streams I started learning the pycairo library to draw lines of custom script using word vectors from podcast transcripts as inputs into the algorithm (like the one above). The colors in these pictures, however, were hardcoded. My goal in this stream was to use color to make the pictures more interesting. Specifically, I wanted to show a transition from one color to another. Each line of the picture would show a step in the color transition.

After some Googling I was a little surprised there isn't an existing Python library to perform the color transition from one color to the next. Maybe the problem isn't that novel to warrant its own library or I'm just bad at Google. Regardless, I decided to write my own function for the transition.

I made a couple assumptions. First, I'm assuming that the transition from color-to-color is linear. In reality this is a non-linear process, but this assumption vastly simplifies the problem and results in decent enough results to get started. Second, the pycairo function set_source_rgba function accepts floating point numbers for the RGB values so the examples here will using floating point values as well.

Given those assumptions the approach I took was to create a function color_diff to calculate how much each RGB value needs to change given a starting color, an ending color, and the number of steps the transition should occur within. More steps results in a smoother transition where fewer will be a more abrupt change. The resulting function looked like this:

def color_diff(color1, color2, n_steps):
    """Calculate the RGB differences between colors based on the number of steps
    the transition will occur within."""
    diff = [0, 0, 0]

    for i in range(len(color1)):
        diff[i] = (color2[i] - color1[i]) / n_steps

    return diff

For each RGB value the value of color1 is subtracted from color2 then divded by the total number of steps to be made. The result is a list with the values of how much the color needs to change at each step to transition to the desired destination color.

Now the function needs to be applied. The example below uses the color_diff function to adjust the brush color with every line drawn. The brush_color is initialized to be equal to the starting color. After the first line is drawn brush_color is incremented by color_step to slightly change the color.

The code is below. For this example straight lines are drawn evenly spaced along the y-axis. Try changing the value of n to see how the color gradient changes when you increase or decrease the number of lines you draw. You can change the values of colors as well to see how the gradients change.

import cairo
from transition import color_diff


# Setup the pycairo Surface.
WIDTH, HEIGHT = 600, 400
surface = cairo.ImageSurface(cairo.FORMAT_ARGB32, WIDTH, HEIGHT)
ctx = cairo.Context(surface)
ctx.scale(WIDTH, HEIGHT)  # Normalizing the canvas
ctx.set_source_rgba(0.95, 0.95, 0.95, 1)
ctx.paint()

start_color = [1, 0.38, 0.38]
end_color = [0, 0, 0]

# Number of lines in the image.
n = 5

# Calculate offset values to space lines on the Surface.
border_padding = 0.05
y_offset = ((1 - (border_padding * 2)) / n)

# Calculate color transition value for each line.
color_step = color_diff(start_color, end_color, n-1)

# Set the initial brush color as the start color.
brush_color = start_color

# Iteratively draw the lines.
x_start = 0.10
x_end = 0.90
for z in range(n):
    # Change brush color based on the color_step.
    if z > 0:
        brush_color[0] += color_step[0]
        brush_color[1] += color_step[1]
        brush_color[2] += color_step[2]

    # Define the line to draw.
    y = (y_offset * z) + border_padding
    ctx.move_to(x_start, y)
    ctx.line_to(x_end, y)

    # Style the brush.
    ctx.set_source_rgba(*brush_color, 1)
    ctx.set_line_width(0.002)

    # Draw the line.
    ctx.stroke()

# Save image.
surface.write_to_png("example_lines_transition.png")

Here is the expected output from the code:

Coming Up Next

The example here is a simple transition from one color to the next. One area to expand this is to allow user's to provide an entire color palette and transition through colors included in the palette.

On the next stream we're going to setup a Twilio voice mailbox for friends and family to call. We're going to setup a phone number for the voice mailbox and then look at the code to extract transcripts from messages left in the mailbox that can be used as input into an image. Come follow along and let's build something together!

Here is where you can find me on Twitch: https://www.twitch.tv/mattyvcodes

Stream Resources

On the stream I mentioned HuggingFace model to draw images from text you provided. I couldn't find the source during the stream, but with some digging I dredged it up! Here is DALL-E! Really neat application to play around with.

For this problem I found this SO answer helpful to think through potential approaches to the problem of color transitions. As I mentioned the solution I walk through in this post is a good starting point, but a more robust solution would likely consider that the color transitions are not going to be linear as I assume here.

Unrelated to this specific stream I am extremely grateful for all the folks who have shared their own repos of generative art for inspiration. A few I have found extra helpful are below:

Creating Python AWS Lambda Layers with Docker

Matthew Vielkind — Mon, 25 May 2020 13:59:43 +0000

By default your AWS Lambda functions include only core Python functionality. Lambda layers are a fantastic feature allowing you to add external dependencies to your Lambda functions. In this post I'll walk you through how you can create your own library of Lambda layers using Docker.

What are Lambda Layers?

If you're unfamiliar with Lambda layers, a layer is additional code you can include at runtime with your Lambda functions to incorporate external libraries into your Lambda function. By default Lambda functions only include core Python libraries. Layers are a way for you to include other external libraries into your Lambda functions. A useful feature of layers is that they exist outside of the Lambda function itself. Once you create a layer you can use that layer across all your Lambda functions. You can read more about layers from the AWS documentation.

Creating a Lambda Layer with Docker

For this post we're going to create a layer using Flask as an example. Creating a Lambda Layer with Docket has four main steps:

Setup the a local directory for the layer
Use Docker to install the layer packages
Create a zip file of your directory
Upload to AWS using the AWS CLI

The next few sections will describe each of these steps in more detail. The last section will show how these steps can be combined into a single bash script to make deployment even easier.

Setting Up Your Layer Directory

The first step is to build the local directory structure that will contain all the artifacts for our layer. In the terminal enter the commands below. A new directory called flask-layer is created along with a subdirectory python/lib.python3.8/site-packages where we will install our Python packages to.

mkdir flask-layer
cd flask-layer
mkdir -pv python/lib/python3.8/site-packages

If you want to create your Layer for multiple Python versions you can repeat the last command and change "python3.8" to whatever other version of Python you want to include.

Within the flask-layer directory create a requirements.txt file that includes the following.

flask==1.1.1

The requirements.txt file defines the packages that will be included in your layer. If you have additional dependencies you want to include in your layer you can add those to this requirements.txt file.

One thing to be aware of with layers is that when they are loaded into your Lambda function they cannot exceed 250MB in size. A Lambda function can include up to 5 different layers. Given these limitations, having smaller layers you can mix-and-match across your functions is preferable rather than building large, single-purpose, layers.

Installing the Requirements with Docker

Docker will install the dependencies you specify in requirements.txt. Docker has images to replicate the AWS Lambda environment. Using those images Docker will install the libraries you specified in requirements.txt that are compatible with the Lambda runtime. In the terminal enter the following code. Make sure you have Docker running before entering this command!

docker run -v "$PWD":/var/task "lambci/lambda:build-python3.8" /bin/sh -c "pip install -r requirements.txt -t python/lib/python3.8/site-packages/; exit"

After running the command you should see the site-packages directory you created in the previous step populated with all your dependencies.

If you have additional versions you want to create your Layer for make sure you substitute the occurrences of "python3.8" with the correct version you want to use.

Uploading Your Layer to AWS

AWS requires all the layer code to be in a zip archive, so we need to zip everything in the python directory.

zip -r flask-layer.zip python

All the contents of the python directory are included in the zip archive called flask-layer.zip.

Now the layer can be uploaded to AWS using the AWS CLI. You need to provide a few parameters in this step:

layer-name is the name your want to give your layer
description to briefly summarize the layer
zip-file is the path to the zip archive you created in the previous step
compatible-runtimes details the Python versions your layer is compatible with

Enter the following into the terminal to publish our layer to AWS:

aws lambda publish-layer-version \
    --layer-name "flask-layer" \
    --description "Lambda Layer for Flask 1.1.1" \
    --zip-file "fileb://flask-layer.zip" \
    --compatible-runtimes "python3.8"

And with that your Flask layer is published to AWS! Now let's simplify this process a little more by packaging some of these steps into a single bash script.

Creating Your Own Lambda Layer Directory

To manage my layers I created a single bash script packaging most of the steps above together to make deploying layers easier. I have a directory where I manage all my layers. Below is a sample of those layers. Each layer has its own directory and the create-lambda-layer.sh script handles installing the layer dependencies and uploading them to AWS.

Here is what create-lambda-layer.sh looks like:

#!/usr/bin/env bash


while [[ "$#" -gt 0 ]]; do case $1 in
  -v|--version) pythonEnvs+=("$2"); shift;;
  -n|--layer-name) layerName="$2"; shift;;
  -d|--desc) layerDescription="$2"; shift;;
  *) echo "Unknown parameter passed: $1"; exit 1;;
esac; shift; done


# Create and install requirements to directory.
for penv in ${pythonEnvs[@]}; do
    mkdir -pv python/lib/${penv}/site-packages
    docker run -v "$PWD":/var/task "lambci/lambda:build-${penv}" /bin/sh -c "pip install -r requirements.txt -t python/lib/${penv}/site-packages/; exit"
done


# Create zip file of environments.
zip -r ${layerName}.zip python


# Publish Layer to Lambda.
aws lambda publish-layer-version \
    --layer-name "${layerName}" \
    --description "${layerDescription}" \
    --zip-file "fileb://${layerName}.zip" \
    --compatible-runtimes ${pythonEnvs[@]}

The script accepts parameters for Python versions (you can provide multiple!), a name and description for your layer. From those parameters the script will generate the file structure, run the docker command for each Python version, create the zip archive, and then update the layer to AWS.

We can test this by installing the Twilio helper library. In the Terminal, create a directory where you're going to store your Lambda layers.

mkdir my-lambda-layers
cd my-lambda-layers

Create a file inside that directory, create-lambda-layer.sh, and paste the code above into the file.

Each layer needs to have its own directory, run the following code in the Terminal to create a directory twilio-layer.

mkdir twilio-layer
cd twilio-layer

The last bit of setup is to create the requirements.txt file, which should include:

twilio==6.40.0

Now we can call our bash script to finish creating and deploying our layer (again, make sure you have Docker running before executing this code!)

bash ../create-lambda-layer.sh -v "python3.8" -n "twilio-layer" -d "Twilio 6.40.0 Layer"

And your layer is deployed to AWS!

If you want to create your layer for multiple different Python versions all you have to do is specify additional -v parameters with those versions. The example below creates a layer for the Python 3.6, 3.7 and 3.8 runtimes.

bash ../create-lambda-layer.sh -v "python3.8" -v "python3.7" -v "python3.6" -n "twilio-layer" -d "Twilio 6.40.0 Layer"

Easy Peasy!

Wrapping Up

In this post I showed you how to deploy a Lambda Layer through the commandline using Docker. The bash script provided isn't fully featured, but in its current form provides a convenient way deploy your Lambda layers. Layers are a neat tool that can help you add functionality to your Lambdas. Docker simplifies creating layers so you can manage your own library of layers that you can share with your friends and coworkers!

Happy Building!

Using Text Messages to Help Small Businesses Manage Takeout Orders

Matthew Vielkind — Mon, 20 Apr 2020 15:48:27 +0000

Due to COVID-19 small businesses have been forced to adopt a takeout/delivery only model, which also means new protocols when picking up your order. Previously you might pickup your order from an employee at a counter. Since COVID-19 customers stand outside or wait in their cars, and businesses have to keep tabs on who has arrived for their order. To help in the tiniest way I built this prototype to help small businesses communicate new takeout protocols to their customers and make order pickups a bit more efficient for everyone.

When orders are placed, small businesses can send customers a message with information about their pickup procedures. In-turn customers provide details when they arrive for their order, which are sent to the business to manage in real-time. Business owners can also send alerts to customers when their order is ready with the click of a button. All of this accomplished with text messages leveraging Twilio APIs through a Flask application.

To start businesses enter the customer name and phone number for the customer when the order is placed.

Using Twilio's Programmable SMS a text message is sent to the customer confirming the order has been received with instructions about how to pickup their order to help guide them through the process.

The customer can reply to the text message with the keyword when they arrive to pickup their order. The "arrive" keyword triggers the Autopilot bot to get details from the customer. For example, if the customer is waiting in a car, what car are they in, so the business can get the order out to them.

Details gathered by Twilio Autopilot are sent to the business in real-time using Twilio Sync Maps. With the real-time view you can see who is waiting for their order and even send quick reminders when the customer's order is ready just by clicking a button.

Now that you've seen an overview of the prototype, the rest of this post will walk through some of the key pieces of code in a little more detail. To get started you can clone the GitHub repo with the full code and follow along!

Order Pickup Manager GitHub Repo

Happy Building!

Prerequisites for this Project

Before getting started there are a couple things we need to have setup.

If you don't have a Twilio account already, you can use this referral link to setup your account with a free $10 to fund your account to get started!
Install ngrok to connect our Autopilot webhooks to.
Setup the Twilio CLI tool and install the Autopilot CLI plug-in which will be used to deploy your Autopilot bot and create your Sync Map for storing orders.
A Twilio phone number. You can log into the Twilio console and get one here.

Setup ngrok

Ngrok is used to allow Twilio to communicate with your local Flask application. If you haven't already, download and install ngrok. Once ngrok is installed you can begin using ngrok by typing ./ngrok http 5000 in the commandline. Ngrok creates a unique URL that will forward any requests made to that URL to your local development environment. You should see something similar to this in your command window:

There should be two URLs labeled "Forwarding" one prefaced with https:// and the other with http://. The https:// forwarding URL will be used as the webhooks to process data collected from your Autopilot bot in the next step.

Setting Up the Autopilot Bot

When customers arrive to pickup their orders we want to gather some basic information about them, what's their name, are they waiting in a car, if so, what car? Collecting this information will be done using Twilio Autopilot.

Autopilot allows you to define tasks for you bot to take based on user inputs. The schema defining the tasks for this Autopilot bot is in the schema.json file. Our bot in this application will have three tasks.

confirm_arrive to get basic arrival information, customer name and whether they're waiting in a car.
get_car_details to get more details about the car from the customer
welcome_message for instances where customers say a friendly "hello" that will inform them of the pickup policy.

The schema.json file needs to be updated to utilize your ngrok URL, specifically the confirm_arrive and get_car_details tasks. Each of these tasks have endpoints defined in the Flask application. To send the input provided by the user to your application you need to edit schema.json to point the webhooks to your ngrok domain.

`confirm_arrive` Task

In the schema file find the confirm_arrive task. Substitute your ngrok URL from the previous step into the redirect value

      "uniqueName" : "confirm_arrive",
      "actions" : {
        "actions" : [
          {
            "say" : "Thank you for choosing our business!  Tell us about your order and we'll get it out to you!"
          },
          {
            "collect" : {
              "on_complete" : {
                "redirect" : "<YOUR NGROK URL>/bot/confirm-in-car"
              },
              "name" : "customer_name",
              "questions" : [
                {
                  "question" : "What is the name on the order?",
                  "name" : "first_name"
                },
                {
                  "type": "Twilio.YES_NO",
                  "question": "Are you waiting in your car?",
                  "name": "in_car"
                }
              ]
            }
          }
        ]
      },

The Flask endpoint to this task will parse the customer responses. If they respond that they are waiting in a car the customer will be asked a few more follow-up questions, otherwise their arrival for the order pickup is processed.

@app.route("/bot/confirm-in-car", methods=["POST"])
def confirm_in_car():
    """
    Endpoint for the confirm_arrive Autopilot task.

    Parses customer responses.  If member indicates they are waiting in their vehicle they will be handed off to
    another Autopilot task to get more details.  Otherwise, the business is alerted the customer has arrived and
    the customer interaction is complete.
    """
    memory = json.loads(request.values.get("Memory"))
    in_car = memory["twilio"]["collected_data"]["customer_name"]["answers"]["in_car"]["answer"]

    # If customer is waiting in the car get more details.
    if in_car == "Yes":
        return {"actions": [
            {
                "redirect": "task://get_car_details"
            }

        ]}

    # Otherwise alert the business the customer has arrived for their order.
    response = alert_arrival(memory)

    return response

`get_car_details` Task

If users are waiting in their car then some follow-up questions are asked to get some details about the car they are in. Make sure you substitute your ngrok URL in this section as well.

      "uniqueName": "get_car_details",
      "actions": {
        "actions": [
          {
            "collect": {
              "name": "car_detail",
              "questions": [
                {
                  "question": "What is the color and make of the car?",
                  "name": "car_make"
                },
                {
                  "question": "What is your license plate number?",
                  "name": "car_license"
                }
              ],
              "on_complete": {
                "redirect": "<YOUR NGROK URL>/bot/parse-car-details"
              }
            }
          }
        ]
      }

Responses for the car details are parsed by Flask.

@app.route("/bot/parse-car-details", methods=["POST"])
def parse_car_details():
    """
    Endpoint for the get_car_details Autopilot task.

    Gets the customer responses to the task and passes those to the business to update the order with the additional
    detail.  Customer is thanked for their order.
    """
    memory = json.loads(request.values.get("Memory"))

    response = alert_arrival(memory)

    return response

Once all the data is gathered for the customer the alert_arrival function will update the order with data provided by the customer and build a response to the customer to let them know their order will be brought to them.

def alert_arrival(customer_input):
    """
    Updates order details with the input collected by the customer from the Autopilot bot.
    :param customer_input: Memory object from the Autopilot bot with customer response.
    """
    # Parse incoming data.
    collected = customer_input["twilio"]["collected_data"]
    arrival_details = dict()
    arrival_details["in_car"] = collected["customer_name"]["answers"]["in_car"]["answer"]

    phone_num = customer_input["twilio"]["sms"]["From"]

    if "car_detail" in collected:
        arrival_details["car_make"] = collected["car_detail"]["answers"]["car_make"]["answer"]
        arrival_details["car_license"] = collected["car_detail"]["answers"]["car_license"]["answer"]

    arrival_details["status"] = "Arrived"

    # Get order associated with phone number.
    # If phone number isn't associated with an order let the customer know.
    try:
        order = TWILIO_CLIENT.sync.services(TWILIO_SYNC_SERVICE_SID).\
            sync_maps(TWILIO_SYNC_MAP_SID).\
            sync_map_items(phone_num).\
            fetch().\
            data
    except TwilioRestException as ex:
        return {"actions": [
            {"say": "Hmmmmm we can't find an order associated with this phone number."}
        ]}

    # Update the order object with the details from the customer.
    for k, v in arrival_details.items():
        order[k] = v

    # Update the order details.
    updated_order = TWILIO_CLIENT.sync.services(TWILIO_SYNC_SERVICE_SID).\
        sync_maps(TWILIO_SYNC_MAP_SID).\
        sync_map_items(phone_num).\
        update(
        data=order
    )

    # Response to provide back to the customer.
    return {"actions": [
        {"say": "Thank you!  We'll bring your order right away!"}
    ]}

Now the webhooks for your Autopilot bot are configured to use the Flask endpoints in your local environment.

Deploying the Autopilot Bot

With the schema file updated our Autopilot bot is ready to be deployed. The Twilio CLI is the quickest way to deploy and manage your Autopilot bots. If you haven't already setup the Twilio CLI and installed the Twilio Autopilot CLI plug-in.

Once you have the Twilio CLI and Autopilot plug-in ready open a terminal window and navigate to the directory containing your schema.json file. Once there you can deploy your bot by typing:

twilio autopilot:create --schema schema.json

If you make updates to your bot all you have to do is make those changes in schema.json and then use the update command in the CLI:

twilio autopilot:update --schema schema.json

Attaching Autopilot to Your Twilio Number

The last step is to attach the Autopilot bot to your Twilio number so that when customers text your number Autopilot will handle their request.

Log into the Twilio console and select Autopilot from the menu on the left. Click on the order_pickup bot you just created.

Select Channels from the menu on the left and then choose Programmable Messaging.

Copy the MESSAGING URL.

Select Phone Numbers from the left menu of products and services. Choose the phone number you want to use from the list.

Scroll down to the bottom and paste the Messaging URL into the A Message Comes In text box.

Now you have an interactive Autopilot bot ready to get your order arrival details from you!

Creating a Sync Map

Next, we need to create a Sync Map where orders will be stored. Twilio Sync allows the platform to track real-time updates for customer arrivals and display new information through the platform without the user having to refresh the page.

Before using Sync, we have to create a Sync service to use, which similar to Autopilot can be done through the commandline in a couple commands.

First, create a Sync service by typing the following into the commandline:

twilio api:sync:v1:services:create

In the response find and copy the sid attribute. Next, you need to create a map and associate that with the service.

In the commandline type the following where you will substitute the sid from above.

twilio api:sync:v1:services:maps:create --service-sid <YOUR_SYNC_SERVICE_SID>

In the response there will be a sid beginning with "MP". Keep this handy as it will be needed in setting up the Flask application.

Our Sync Map is ready for use!

Generate a Twilio API Key

The last piece of setup is you need to generate an API key that will be used to handle JavaScript requests. After you create the key keep track of the API Key and Secret that are provided. We'll need those in the next step.

Setting Up the Flask Environment

Flask will serve as the back-end of the platform providing the web pages as well as the webhook for the Autopilot responses from the customers. The app.py file contains the core code for the application that handles interactions with the web application and the appropriate API calls with Twilio. To setup the Flask application you need to fill in the details of the .env file with details from your Twilio account. These environment variables are used to make your API calls with Twilio. Do not share them with anyone!

TWILIO_ACCOUNT_SID and TWILIO_AUTH_TOKEN come directly from the Twilio console with you login
Fill in TWILIO_API_KEY and TWILIO_API_SECRET with the info generated in the previous step
TWILIO_SYNC_SERVICE_SID and TWILIO_SYNC_MAP_SID come from creating the Sync Map with the CLI
TWILIO_FROM_NUMBER is the phone number you purchased. Make sure it's in E.164 format with (ex. +14155552671)

# Required for all uses
TWILIO_ACCOUNT_SID=
TWILIO_AUTH_TOKEN=
TWILIO_API_KEY=
TWILIO_API_SECRET=

# Twilio Sync SIDs
TWILIO_SYNC_SERVICE_SID=
TWILIO_SYNC_MAP_SID=

# Twilio Number to send messages from.
TWILIO_FROM_NUMBER=

Once you have all the environmental variables filled out you can deploy the Flask application. From the commandline make sure you are in the directory of the application and then enter:

python app.py

Your application is now live! You can open up your Flask application and begin to interact with your app.

Let's look at some more detail about some of the functionality and the related code.

Sending Initial Text to the Customer

When new orders are placed businesses can simply send a text message to their customer by entering the customer's name and phone number. Two things happen, first a personalized text message is sent to the customer thanking them for their order with directions to pickup their order and then a new entry is created in the Sync Map for the customer. In this prototype we're going to use the customer's phone number is going to be the unique identifier for each order and the key in our Sync Map. The phone number will link all the actions to an order in the application. All of this is handled by the send_text endpoint in the Flask app.

@app.route("/send-text", methods=["POST"])
def send_text():
    """
    Creates a customized text message confirming the order and alerting the customer about the pickup service.
    """
    customer_name = request.form["customer-name"]
    to_num = request.form["to-number"]

    # Replace punctuation and spaces in phone number.
    e164_num = to_num.translate(str.maketrans('', '', string.punctuation)).replace(" ", "")
    e164_num = f"+1{e164_num}"

    # Send text message to the customer.
    TWILIO_CLIENT.messages.create(
        to=e164_num,
        from_=os.getenv("TWILIO_FROM_NUMBER"),
        body=f"Thanks for your order {customer_name}!\n\nTo make order pickup easier send ARRIVE to this number to let us know you're here."
    )

    # Create new Sync Map entry for the order.
    try:
        sync_map_item = TWILIO_CLIENT.sync \
            .services(TWILIO_SYNC_SERVICE_SID) \
            .sync_maps(TWILIO_SYNC_MAP_SID) \
            .sync_map_items \
            .create(key=e164_num, data={
                'name': customer_name,
                'status': 'Order Placed',
                'readPhone': to_num
            })
    except TwilioRestException as ex:
        print(ex.msg)

    return render_template("order_created.html", customer_name=customer_name, phone_number=to_num)

Getting a List of Orders

Once orders are placed the business needs to be able to manage them. Using the Twilio API all active orders are retrieved from the Sync Map and displayed on the orders page.

@app.route('/orders')
def orders():
    """
    Gets active order objects from the Sync Map and populates the webpage.
    """
    all_orders = TWILIO_CLIENT.sync.\
                       services(TWILIO_SYNC_SERVICE_SID).\
                       sync_maps(TWILIO_SYNC_MAP_SID).\
                       sync_map_items.\
                       list()

    return render_template("order_list.html", orders=all_orders)

Sending a Text Reminder

One feature built into this prototype is the ability for businesses to send a quick reminder to customers when their order is ready. With the click of a button a personalized text message is sent to the customer telling them their order is ready along with the pickup instructions.

@app.route("/send-order-reminder", methods=["POST"])
def send_order_reminder():
    """
    Sends a reminder to the customer when their order is ready.
    """
    to_num = request.form["orderID"]

    # Get name on order.
    order_item = TWILIO_CLIENT.sync.\
        services(TWILIO_SYNC_SERVICE_SID).\
        sync_maps(TWILIO_SYNC_MAP_SID).\
        sync_map_items(to_num).\
        fetch()

    customer_name = order_item.data["name"]

    msg = f"Hi {customer_name}!  Your order is ready for pickup!\n\n" \
        f"When you are here to pickup your order text ARRIVE and we'll bring it out to you!"

    response = TWILIO_CLIENT.messages.create(
        to=to_num,
        from_=os.getenv("TWILIO_FROM_NUMBER"),
        body=msg
    )

    # Update the order status.
    order_item.data["status"] = "Reminder Sent"
    order_item.update(data=order_item.data)

    return "SUCCESS"

Real-Time Order Updates

Orders are updated real-time using the Twilio Sync JavaScript SDK. When new orders are received a new card for the order is created using the Sync Map itemAdded event.

      // When a new arrival occurs create a new card with the details.
      syncMap.on('itemAdded', function(event) {
        console.log("New Order Added");
        createOrderCards(event.item);
      });

After the customer completes the Autopilot bot questions the order information needs to be updated for the card using the Sync Map itemUpdated event.

      syncMap.on('itemUpdated', function(item) {
        console.log("Order Updated");
        // Update card details...
      });

Wrapping Up

This project demonstrated a prototype for helping small businesses improve communication with their customers to make order pickup a little easier. I hope you found something useful from this project. This was also the first Hackathon I've participated in and I can't wait to do more of these! There are a bunch of other features I wanted to add like scheduling follow-up customer surveys and sending coupon codes for future visits. Being my first Hackathon I wanted to avoid overwhelming myself with features to add, so I'm looking forward to working on these in the future.

Building front-end applications and doing front-end design is a new frontier for me as well. I'm a big proponent of learning by doing so I'm excited to incorporate more front-end components into my projects going forward.

Hope you enjoyed! And stay safe!

Learning AWS Rekognition with Celebrity Selfies

Matthew Vielkind — Fri, 20 Dec 2019 12:20:24 +0000

In this post we will explore AWS Rekognition, Amazon's cloud service for analyzing images and video. Rekognition has a number of different features, facial detection, text extraction, object labeling etc. For this post we will focus on Rekognition's ability to identify celebrities in images by using iconic celebrity selfies as our inputs. In this post you will:

Learn how to send an image to Rekognition with boto3
How to identify celebrities in an image
Annotate images with celebrity names
Draw bounding boxes using the Rekognition output
Utilize the Rekognition facial landmarks to enhance images

Let's get started!

Processing Images with Rekognition

AWS Rekognition input can be either an image stored in S3 or a bytes image. In this example we'll cover the case when your image is stored in a S3 bucket. I'll show how you can read the image from S3 with Rekognition and identify the celebrities in the image. The first selfie we'll look at is the iconic Beyonce/Jay-Z selfie.

Now let's see who Rekognition recognizes in this image!

If the image you want to process is loaded in S3 you can use boto3 to pass the image to Rekognition by specifying the name of the bucket and the key for the image.

import boto3

rek = boto3.client('rekognition')

response = rek.recognize_celebrities(
    Image={
        "S3Object": {
            "Bucket": "<YourBucketName>",
            "Name": "<ImageName>"
        }
    }
)

The recognize_celebrities function will take the input image and call the algorithm to identify celebrities in the image. Neat! Let's look at some of the data Rekognition provides about each image.

The response from recognize_celebrities includes a ton of data about the image. The boto3 documentation is phenomenal, so if you're interested in knowing about all the specifics I encourage you to read the boto3 Rekognition documentation for all the details about the recognize_celebrities response. We'll walk through a lot of this, but this is a good reference to have on hand!

Two pieces of output in the response are CelebrityFaces and UnrecognizedFaces, which lists all faces identified as celebrities in the image along with other faces in the image that were not identified as being a celebrity. In this first image let's summarize these fields to see how many identify celebrities and unrecognized faces were in the image.

face_counts = {k: len(v) for k, v in response.items() if k in ["CelebrityFaces", "UnrecognizedFaces"]}

Rekognition identified one face as a celebrity in the image and one other face was identified that couldn't be identified as a celebrity. So, who was identified?

Each item in the CelebrityFaces list will contain the name of the celebrity that was identified. Let's parse the output to see who was identified by Rekognition.

for celeb in response["CelebrityFaces"]:
    print(celeb["Name"])

The result is Beyonce! Rekognition identified that there was another face in the image, but wasn't able to specifically identify that face as that of Jay-Z.

Being able to name celebrities in an image is neat, but let's do a little more with Rekognition to annotate our celebrity selfies to more easily identify who's who.

Annotating Selfies

In the previous image we only had two faces to recognize, but what about the case when you have multiple recognized and unrecognized faces in an image? Identifying who is who can be difficult, but Rekognition can help with this process as well. It's impossible to have a post about celebrity selfies without a little Kardashian influence, so our next selfie is Kylie Jenner's selfie from the Met Gala.

What's fun about this image is the volume of recognizable faces that are included. To help identify whose who in the image we're going to utilize more of the Rekognition response. For each face, celebrity and unrecognized alike, Rekognition provides the coordinates framing each face in the image. Using those bounding box coordinates you can annotate the image to label names and indicate if a face is unknown. For Kylie's selfie we will annotate recognized celebrities with a green box labeled by their name and unrecognized faces will be framed by a grey bounding box.

In the previous example the image was loaded directly from S3. You can also process local images with Rekognition by converting the image into a bytesarray, which is shown below.

import io
import boto3
from PIL import Image

# Load image and convery to bytes array.
img = Image.open(open(img_path, 'rb'))
img_byte_arr = io.BytesIO()
img.save(img_byte_arr, format='PNG')

# Pass the byte array to Rekognition
rek = boto3.client("rekognition")
response = rek.recognize_celebrities(
    Image={
        "Bytes": img_byte_arr.getvalue()
    }
)

To draw the bounding box we need to access the BoundingBox coordinates associated with each face. The bounding box coordinates are provided as ratios to the overall image size, which will have to be scaled to the image's size. To do the scaling I created a function, calculate_bounding_coordinates, that will scale the Rekognition output to the image. We will use the PIL library for annotating the image. Drawing the image requires the Rekognition output as well as the original image, img, object.

For each face in CelebrityFaces and UnrecognizedFaces the appropriate bounding box will be drawn labeled with either the celebrity's name or a generic "Unknown" for the UnrecognizedFaces.

from PIL import Image, ImageDraw, ImageFont


# Prepare image for drawing.
img_width, img_height = img.size
draw = ImageDraw.Draw(img)
font = ImageFont.truetype('/Library/Fonts/Arial.ttf', 30)

def calculate_bounding_coordinates(img_width, img_height, box):
    """
    Calculates the location of the bounding box in the image.
    :param img_width: Width of the image.
    :param img_height: Height of the base image.
    :param box: Bounding box coordinates from Rekognition.
    :return: Tuple of points to frame the bounding box.
    """
    # Calculate box location.
    left = img_width * box['Left']
    top = img_height * box['Top']
    width = img_width * box['Width']
    height = img_height * box['Height']

    points = (
        (left, top),
        (left + width, top),
        (left + width, top + height),
        (left, top + height),
        (left, top)
    )

    return points

# Identified Celebrities will be drawn w a Green bounding box w their name.
for celeb in response["CelebrityFaces"]:
    box_coords = calculate_bounding_coordinates(img_width, img_height, celeb["Face"]["BoundingBox"])
    draw.line(box_coords, fill="#00d400", width=2)
    draw.text(box_coords[0], text=celeb["Name"], fill="#00d400", font=font)

# Unrecognized faces are framed with grey boxes.
for i, face in enumerate(response["UnrecognizedFaces"]):
    box_coords = calculate_bounding_coordinates(img_width, img_height, face["BoundingBox"])
    draw.line(box_coords, fill="#989695", width=2)
    draw.text(box_coords[0], text=f"Unknown{str(i)}", fill="#989695", font=font)

After applying the annotation the new img now looks like this:

It's so much easier to determine who's who!

Now let's go one level deeper with Rekognition's output!

Facial Features in AWS Rekognition

Rekognition doesn't stop with framing faces. For each face Rekognition will also provide details about individual facial features, eyes, mouth, nose, ears etc. To demonstrate these features we'll be adding black censor bars across the eyes of each face in the selfie.

To do this a couple more functions are needed. The first is get_facial_landmark_location that will retrieve the XY coordinates of a specific facial landmark (left and right eye in this case). The second function is calculate_eye_mask that will calculate the coordinates of the censor bar using the facial landmark locations.

Let's bring the censor bar together with the bounding box annotation from the previous selfie to fully annotate the image. And to demonstrate this let's use Ellen's famous Oscar selfie! The full code is as follows:

import io
import boto3
from PIL import Image, ImageDraw, ImageFont

# Load image and convery to bytes array.
img = Image.open(open(img_path, 'rb'))
img_byte_arr = io.BytesIO()
img.save(img_byte_arr, format='PNG')

# Pass the byte array to Rekognition
rek = boto3.client("rekognition")
response = rek.recognize_celebrities(
    Image={
        "Bytes": img_byte_arr.getvalue()
    }
)

# Prepare image for drawing.
img_width, img_height = img.size
draw = ImageDraw.Draw(img)
font = ImageFont.truetype('/Library/Fonts/Arial.ttf', 30)

def calculate_bounding_coordinates(img_width, img_height, box):
    """
    Calculates the location of the bounding box in the image.
    :param img_width: Width of the image.
    :param img_height: Height of the base image.
    :param box: Bounding box coordinates from Rekognition.
    :return: Tuple of points to frame the bounding box.
    """
    # Calculate box location.
    left = img_width * box['Left']
    top = img_height * box['Top']
    width = img_width * box['Width']
    height = img_height * box['Height']

    points = (
        (left, top),
        (left + width, top),
        (left + width, top + height),
        (left, top + height),
        (left, top)
    )

    return points


def get_facial_landmark_location(rek_obj, landmark):
    """
    Retrieves the XY coordinates for a specified facial landmark in the Rekognition response.

    :param rek_obj: A Rekognition response object.
    :param landmark: Name of the facial landmark to retrieve.
    :return: XY coordinates for the landmark.
    """
    facial_landmark = next((item for item in rek_obj["Landmarks"] if item["Type"] == landmark), None)

    return facial_landmark["X"], facial_landmark["Y"]


def calculate_eye_mask(image_width, image_height, rek_obj):
    """
    Calcualtes the coordinates for the eye mask given the location of the eyes.

    :param image_width: Width of the image.
    :param image_height: Height of the image.
    :param rek_obj:
    :return: Tuple of coordinates defining the location of the eye mask.
    """
    # Get eye landmarks
    eye_left = get_facial_landmark_location(rek_obj, "eyeLeft")
    eye_right = get_facial_landmark_location(rek_obj, "eyeRight")

    # Get the pixel location of the eyes in the image.
    eye_left_x = eye_left[0] * image_width
    eye_left_y = eye_left[1] * image_height
    eye_right_x = eye_right[0] * image_width
    eye_right_y = eye_right[1] * image_height

    # Build the coordinates for the rectangle to be drawn.
    poly_coords = (
        (eye_right_x + 50, eye_right_y - 20),
        (eye_left_x - 50, eye_left_y - 20),
        (eye_left_x - 50, eye_left_y + 20),
        (eye_right_x + 50, eye_right_y + 20),
    )

    return poly_coords


# Identified Celebrities will be drawn w a Green bounding box w their name.
for celeb in response["CelebrityFaces"]:
    box_coords = calculate_bounding_coordinates(img_width, img_height, celeb["Face"]["BoundingBox"])
    draw.line(box_coords, fill="#00d400", width=2)
    draw.text(box_coords[0], text=celeb["Name"], fill="#00d400", font=font)
    eye_coords = calculate_eye_mask(img_width, img_height, celeb["Face"])
    draw.polygon(eye_coords, fill="#000000")

# Unrecognized faces are framed with grey boxes.
for i, face in enumerate(response["UnrecognizedFaces"]):
    box_coords = calculate_bounding_coordinates(img_width, img_height, face["BoundingBox"])
    draw.line(box_coords, fill="#989695", width=2)
    draw.text(box_coords[0], text=f"Unknown{str(i)}", fill="#989695", font=font)
    eye_coords = calculate_eye_mask(img_width, img_height, face)
    draw.polygon(eye_coords, fill="#000000")

And the resulting image looks like this:

Wrapping Up

There you have it! In this tutorial you have:

Learned the two ways images can be used with AWS Rekognition
How to identify celebrities in an image
Draw bounding boxes using the Rekognition output
Utilize the Rekognition facial landmarks to enhance images

There is so much Rekognition can do! While the focus of this tutorial was on the recognize_celebrities function many of the same approaches can be used with the other Rekognition functions as well.

Thanks for reading! Have yourself a grand day!

Updating Values in DyanmoDB Map Attributes

Matthew Vielkind — Thu, 24 Oct 2019 12:00:34 +0000

One project I’m working on translates a favorite party game among my friends into a SMS-based version that can be played anywhere. A requirement for the project is being able to manage the game state across players. To do this I created a DynamoDB record for each game that maintains the state of the game. Each player has an entry in a Map attribute, which stores data in key-value pairs, with details like their name and score. Throughout the game the score for players needs to be updated by accessing the nested Map attribute of each player. This post will explain a pattern for how you can dynamically build a pattern for updating individual data elements within DynamoDB Maps.

Setting Up an Example

Let's suppose we have a DynamoDB record for a game where a game_id uniquely identify the game and there is a players Map attribute containing all the information about each player, in this case just a score, like the example record below. In this example there are two players, Charlotte and Becky, who each have their own record nested in the players Map containing the player's score. As the game progresses the score for each player needs to be updated.

Item={  
    "game_id": "test",
    "players": {  
            "Charlotte": {  
                "score": 0  
             },  
             "Becky": {  
                "score": 0  
             }  
        }  
    }

Building the Update Pattern

As the game progresses the score attribute needs to be updated for each player. The function below shows how the DynamoDB update_item function can be used to update new nested Map attributes.

def update_player_score(game_id, player_id, score_val):  
    dynamo = boto3.resource('dynamodb')  
    tbl = dynamo.Table('<TableName>')  

    result = tbl.update_item(  
        Key={  
            "game_id": game_id  
        },  
        UpdateExpression="SET players.#player_id.score = :score_val",  
        ExpressionAttributeNames={  
            "#player_id": player_id  
        },  
        ExpressionAttributeValues={  
            ":score_val": score_val  
        }  
    )

The function takes parameters for the game_id, player_id, and score_val. Given those parameters the function will update the player's score to be equal to what is provided in score_val. For example, if you were to call, update_player_score("test", "Charlotte", 1) we would change the score of Charlotte to be 1. Let's walkthrough how that happens.

The first part of update_item. defines what key we will be updating. In this case game_id is the Key to the table, so we'll be updating records associated with the "test" game_id.

Next the UpdateExpression is defined. In the UpdateExpression we define the pattern for the value we want to update. For this function we want to update the nested parameters within the players map that could vary by the name of the player. In UpdateExpression the name of the players map is hardcoded, but accessing the nested Map attribute needs to be dynamic. There are two placeholders, #player_id and :score_val, in UpdateExpression defining the location of the specific attribute in the map that needs to be updated and the value to set to that nested attributed. The subsequent ExpressionAttributeNames and ExpressionAttributeValues definitions provide a mapping to what those placeholders should be.

The values provided in ExpressionAttributeNames and ExpressionAttributeValues will replace the placeholders that are in UpdateExpression when the function is called. For example consider the case where Charlotte's score becomes 1. When the function is called the UpdateExpression would be equivalent to SET players.Charlotte.score = 1 after the substitutions for #player_id and :score_val are made from ExpressionAttributeNames and ExpressionAttributeValues.

In DynamoDB an ExpressionAttributeNames is the placeholder for dynamic attribute values. When constructing the UpdateExpression all attribute values must be prefaced with a #. In our example we want to update the score for a player, but since the player that's being updated is dynamic the ExpressionAttributeNames fills in the #player_id placeholder within UpdateExpression.

Similarly, ExpressionAttributeValues are placeholders for when the the value of an attribute are not known until runtime. In our case the score to assign to the player. All ExpressionAttributeValues are prefaced with a :.

If we wanted to update Becky's score to give her 3 points we could use this function call and the UpdateExpression would handle the proper mapping to where Becky is located in the players attribute map:

update_player_score("test", "Becky", 3)

And that’s it! Patterns similar to this are helpful when you have Map attributes in DynamoDB where the path to the attribute you want to update will be dynamoic. When working with Map attributes in DynamoDB it's important to define the ExpressionAttributeNames and ExpressionAttributeValues that will be dynamic when building your UpdateExpression then construct the expression accordingly. Hopefully this was a helpful exercise for you! There are a couple more projects I’m working on that I can’t wait to share so stay tuned!

Integrating Twilio Bots into Mobile Workflows with Apple Shortcuts

Matthew Vielkind — Tue, 27 Aug 2019 12:07:25 +0000

Building messaging bots has allowed me to personalize my mobile experience. Where there are no existing services meeting my individual needs I've built bots tailored to my personal requirements, like this one I call Twilio Fit that designs a daily workout that won't get boring. Bots are a great way to deploy custom functionality in a mobile-first manner without having to design a full app, but interactions were limited to the various channel(s) to which the bot was deployed. Incorporating messaging bots into the larger mobile experience was a challenge. That all changed when I stumbled on Apple Shortcuts, which took my Twilio bots to a whole new level.

Discovering Apple Shortcuts.

Apple Shortcuts is an app provided by Apple that allows you to setup workflows on your iPhone. You can design custom workflows triggered by a variety of actions that will pass inputs and outputs to any number of different tasks. Within the app there is an entire gallery of shortcuts available allowing you to get prepped for the morning commute, share your current location via a text message, track how much caffeine you've had today, or help you with those tricky guitar chords among many, many others. After reviewing all the different actions you could take within Apple Shortcuts I thought, what would happen if I hooked this up to one of my bots? Let's find out!

Twilio and Apple Shortcuts is Love.

One bot I frequently use is my SMS Grocery List Bot to convert a screenshot of a recipe into a more useful shopping list. For this bot a screenshot of a recipe is sent to a Twilio SMS messaging bot and the bot returns a list of ingredients in the form of a note to make shopping easier. Without Shortcuts a lot of this process would be manual, finding the bot's number in iMessage, selecting the picture to send, opening Notes and pasting the returned list. With Shortcuts some of these steps can be automated to make the bot more accessible. Let's walkthrough what the workflow looks like in Shortcuts to give you an idea of how Twilio and Shortcuts can improve the mobile experience of your bots. Below are the first 3 steps of the workflow.

In this use case an assumption is made that the last screenshot is the one that has the recipe. Alternatively you can prompt the user to select the screenshot they want to send. There are so many different ways this can get configured! Next, the Send Message section takes the selected screenshot and sends it as a MMS message to the Twilio SMS messaging service, which I've saved as a contact, twilio-sms-shopping-list. Many bots have some kind of wake word to start the conversation. In the Send Message step you can pass that wake word to your SMS bot to start a conversation with a bot quickly. When Send Message is triggered a new message window is opened with your image already populated in the message. All you have to do is click send!

At this point the screenshot has been sent to the bot for processing. Extracting the text and applying the NLP models takes a little bit of time, so the Wait to Return step pauses the rest of the process until there is output to continue. The Wait to Return action allows the user to leave the Shortcut app and will continue when the user returns. After a few seconds the Twilio bot will return the recipe ingredients. When the ingredients are returned the user can open the message and copy the ingredients.

After copying the ingredients the process will continue when the user opens the Shortcuts app again. The last steps will take the ingredients that were copied to the clipboard and create a new Note in the Notes app that will contain the list of ingredients that were copied from the message.

When complete the note that was just created will be opened. Just like that you're ready to go grocery shopping with a list you can actually use! No more forgetting ingredients!

The best part is once the Shortcut has been setup it can be accessed with the click of a button on your home screen! You can even design custom icons for your bots to make them more personal. Now all my bots are accessible with a click of a button from the home screen of my phone!

Wrapping Up

Combining Twilio with Shortcuts creates a new dynamic, which can make your bots more accessible. With the flexibility of the Twilio platform and the number of different services Shortcuts integrates with there are so many different options for testing different integrations of your bots into your mobile experience.

Now that you've seen how you can use Twilio and Apple Shortcuts together here are a couple other tutorials to get you started:

Twilio Fit to build your own personal trainer bot.
Sous Chef Bot to build a bot that will convert screenshots of your favorite recipe into organized shopping lists.

If you're new to Twilio you can use this referral link for a free $10 to get started!

Happy building!

Getting Started with AWS Lambda Layers

Matthew Vielkind — Sat, 27 Apr 2019 19:38:49 +0000

Once you begin working with Lambda functions there will inevitably come a time when you have to include an external library in your function. Out of the box Lambda functions only support a limited number of libraries with their runtimes. In order for your function to access those external dependencies you would have to include them in your deployment package. Each time you deploy a Lambda function making sure to include external dependencies in the deployment package was a part of the workflow. During this step when deploying a Lambda function I'd catch myself thinking, "there has to be a better way". And now there is! With the introduction of layers for AWS Lambda you no longer have to bundle your external dependencies with your deployment package. Layers allow you to manage and share your external dependencies across your Lambda functions. In this post I'm going to,

Explain what layers are and their benefits
Walkthrough how to create and use a layer in a Lambda function
Provide some best practices and tips

Even though this post will largely be focused on Python, a lot of the core lessons are extensible to other languages supported by AWS as well. Before getting started a quick disclaimer. AWS Lambda functions have a bunch of built-in nuances and limitations. While layers are helpful they are not a panacea. Please take a minute to read the last section about best practices. As with any tool it becomes useless if you're not using it properly, so taking a minute or two to understand the limitations can save you from a headache and frustration later on.

Introducing AWS Layers

So, what are these layers and how are they useful? An AWS Lambda layer contains additional code, libraries and dependencies that are loaded to support a Lambda function at runtime. Layers eliminate the need to package external dependencies in your deployment package for each Lambda function you create. A layer is a package of all the external dependencies your Lambda function requires. At runtime the dependencies specified in the layer are loaded and available to your Lambda function. Layers are not specific to individual Lambda functions either. Once you create a layer you can use that layer to support multiple different Lambda functions. There is no need to recreate a layer for each Lambda function. Each Lambda function can support up to 5 different layers providing the flexibility to mix-and-match different layers.

Layers also support versioning making them easier to manage. When collaborating with others it is easier to have a centralized location for Lambda dependencies that can be referenced instead of each person building their own dependencies at deployment where there is a risk package versions could become out-of-sync and potentially incompatible. With a little background of what layers are let's create one of our own!

Creating an AWS Layer

Now that you have an idea of what a layer is and why you might want to use it, let's create one. For this initial example we're going to create a layer that makes the flask framework available to our Lambda function. To build a layer there are three general steps,

Build the zip file with all the dependencies for the layer
Upload the zip file to the layer in AWS
Associate a layer with a Lambda function

Let's get started with creating the deployment package for our layer.

Building the Code Package

The deployment package for our layer is a zip file containing all of the code we want to include in the layer. The process is very similar to how you would incorporate dependencies into your Lambda deployment package, but the difference is with layers you do this process once instead of each time you deploy. The code here could be custom code or any other external Python packages themselves (note I'm using Python here, but layers support all different kinds of runtimes). In this introduction we're going to keep our deployment simple and will be creating a layer that makes the flask library available to our Lambda. What we need to do is install flask into a directory and then create a zip archive of that directory. One requirement when building the code package for a layer is that our code must be nested within a folder supported by the runtime for your Lambda. In our case we'll be using Python, so the flask code must be in a zip file nested within a sub-directory named python. We can do all of this from the terminal.

First, in the terminal let's create a directory for our project, which we'll call flask-layer with a python folder as a sub-directory.

mkdir -p flask-layer/python

Next, install the flask package into the flask-layer/python directory,

pip3 install flask -t flask-layer/python

Finally, zip the contents of flask-layer into a zip archive.

cd flask-layer
zip -r aws-flask-layer.zip python

Creating our Layer in AWS

Next, we can upload our code to AWS. Within the AWS console go to the Lambda service page. On the left menu select "Layers".

On the next page click "Create Layer" in the upper right corner. Now we'll define our layer. I named the layer flask-layer with a brief description. The next piece of information we need to enter is the "Code entry type". Notice the warning beneath the dropdown. For this example our code package is much less than the 10MB limit so we're going to choose the "Upload a .zip file" option then click the "Upload" button and select the aws-flask-layer.zip file you created above. If your .zip file is > 10MB you will need to upload the file to S3 and then provide the path to the file in S3. Finally, we need to associate our layer with a runtime. We'll use the Python 3.7 runtime in this example, but you can associate a single layer with up to 5 different runtimes. When you're done your page should look something like this,

At the bottom right click "Create Layer". Now that we've created the layer let's put it to use.

Using our Layer in a Lambda Function

Go back to the Lambda dashboard and click "Create function" in the upper right. Our function is going to be named, testFlaskLayer. We're going to use the Python 3.7 runtime to match the runtime we selected for our layer. If you select a different runtime than one you selected for your layer then you will not be able to access the layer from your function. Pick an existing role if you have one that has Lambda permissions, or allow AWS to create one for you and click "Create function" at the bottom.

In the next page we need to connect the layer we created with our Lambda function. Below the box that says "testFlaskLayer" in the middle of the page click the "Layers" box underneath.

Scroll down the page and click the "Add a layer" button.

From the dropdowns select flask-layer and then specify we're using version 1 of the layer, then click "Add".

Back at the Lambda designer click "Save". Now our Lambda function has access to the flask package! To make sure this is the case let's do a quick test.

In the designer scroll down to the in-line editor and paste in the following code. All we're doing is demonstrating that we're able to load the flask library because of the layer we added and we're going to just output the version of flask we're using.

import json
import flask

def lambda_handler(event, context):
    return {
        'statusCode': 200,
        'body': json.dumps(flask.__version__)
    }

Save the Lambda function. Now scroll to the top and run the "Hello World" test. You should get a successful result with the version of flask you're using in the body of the output. If you were to go back into your layers and remove flask-layer from your Lambda, save the function, and rerun the test you'll get a failure. By removing the layer the Lambda function no longer can access the flask library. When Lambda tries to import the flask library in this case an error will be thrown.

If you had other Lambda functions that needed to access flask all you would need to do is add flask-layer to the Lambda function and you'd be all set!

Best Practices and Other Tips

To get the most out of layers keep the code making up any single layer narrowly defined. Having narrowly defined layers limits your exposure of including unused dependencies when you start sharing layers across functions. Dependency bloat within a Lambda has real costs in terms of added execution cost, time, and performance. Narrowly defining your layers can help you avoid those costs. Another reason you'll want to smaller layers is because layers are not a work around to the Lambda deployment package size. I'll admit when I started using layers there was a fleeting moment where I thought layers could get around those limits, but that is not the case. Layers included in a Lambda will count towards the deployment size of the Lambda function contributing towards the hard limits for deployment package size of Lambda functions. Because of these size restrictions keeping your layers small allows you to make the most from your deployment packages by not wasting room on dependencies you don't need.

To make sure your Lambda function has all the dependencies that are needed stack your layers. Lambda functions allow you to include up to 5 unique layers. Stacking layers lets you mix and match the individual layers you have created to create a custom set of dependencies for your function.

Finally, one other note about creating layers to be aware of. Any code you include in your layer zip file must be compatible with AWS. As described in this issue some python packages (i.e. numpy) contain compiled code that is incompatible with AWS. The workaround for these instances is to build your deployment package with a precompiled .whl files instead of using pip install. In the case of numpy and scipy Amazon has created a public layer anyone you can incorporate into your code.

Wrapping Up

Layers are a tool to help manage your external dependencies for your AWS Lambda functions. They provide a systematic solution to sharing dependencies across Lambda functions removing the need to package those dependencies with each deployment so you can focus on the code for the Lambda function itself. Layers do have their limitations, so here are a few takeaways,

Layers manage external dependencies for Lambda functions
Layers can be shared across Lambda functions
Keep layers small and leverage the ability to stack layers in functions. Smaller layers make it easier to make sure a function has only the dependencies required to perform its tasks.
Layers are still subject to Lambda limits on code deployment size. They are not a workaround for those hard limitations.
Dependencies must be compatible with the AWS environment, which means you may not just be able to pip install certain packages.

I hope you found this introduction to layers useful! I'm just getting started doing this, so if something isn't clear or you have further questions please let me know!

Forem: Matthew Vielkind

Semantic Search with Haystack and Elastic

Create the Elastic Document Store

Process Text and add to Document Store

Create Document Embeddings

Querying the Document Store

Adding New Documents

Wrapping Up

Using Python and Airtable

Authentication

Using the REST API

Adding Records to Airtable

Reading Records from Airtable

Update a Record

Delete a Record

Conclusion

Stream Notes- Color Transitions in pycairo

Project Overview

Stream Notes

Coming Up Next

Stream Resources

Creating Python AWS Lambda Layers with Docker

What are Lambda Layers?

Creating a Lambda Layer with Docker

Setting Up Your Layer Directory

Installing the Requirements with Docker

Uploading Your Layer to AWS

Creating Your Own Lambda Layer Directory

Wrapping Up

Using Text Messages to Help Small Businesses Manage Takeout Orders

Prerequisites for this Project

Setup ngrok

Setting Up the Autopilot Bot

confirm_arrive Task

get_car_details Task

Deploying the Autopilot Bot

Attaching Autopilot to Your Twilio Number

Creating a Sync Map

Generate a Twilio API Key

Setting Up the Flask Environment

Sending Initial Text to the Customer

Getting a List of Orders

Sending a Text Reminder

Real-Time Order Updates

Wrapping Up

Learning AWS Rekognition with Celebrity Selfies

Processing Images with Rekognition

Annotating Selfies

Facial Features in AWS Rekognition

Wrapping Up

Updating Values in DyanmoDB Map Attributes

Setting Up an Example

Building the Update Pattern

Integrating Twilio Bots into Mobile Workflows with Apple Shortcuts

Discovering Apple Shortcuts.

Twilio and Apple Shortcuts is Love.

Wrapping Up

Getting Started with AWS Lambda Layers

Introducing AWS Layers

Creating an AWS Layer

Building the Code Package

Creating our Layer in AWS

Using our Layer in a Lambda Function

Best Practices and Other Tips

Wrapping Up

`confirm_arrive` Task

`get_car_details` Task