Forem: Christian Barra

How to run Python serverless applications on the Google Cloud Platform

Christian Barra — Wed, 13 Jan 2021 08:28:45 +0000

I've used the Google Cloud Platform extensively in the last years and, as a cloud platform, is one of the most enjoyable you can work with.
It really helps you focus on what matters, whether you are building an MVPs, a data ingestion pipeline, a single function or full-fledged style application.

Let's focus on the serverless offer and discover how you can run Python serverless applications on top of GCP.

There are 5 different ways in which you can deploy a Python application on GCP:

A virtual machine (Compute Engine)
A Kubernetes cluster (GKE)
App Engine
Cloud Functions
Cloud Run

App Engine, Cloud Functions and Cloud Run are 3 different ways to deploy serverless workloads on GCP.

What does serverless mean in this context?

You don't need to pre-allocate resources (like number of machines)
You pay (almost) zero if you don't receive any request
It can scale up and down (to zero instances/functions) automagically

Before starting: configure the gcloud sdk

We are going to use the gcloud sdk, so it's better to have it setup properly:

A valid GCP account with a project
The gcloud sdk installed
You need to configure the gcloud sdk, these are the magic lines you need to run:

# You can use whatever you like for CONFIGURATION-NAME
gcloud config configurations create {CONFIGURATION-NAME}

# the project id comes from GCP
gcloud config set project {PROJECT-ID}

# This command will open a browser tab
gcloud auth login

# Check if everything works
gcloud config configurations describe {CONFIGURATION-NAME}

You are ready!

App Engine Standard Environment

App Engine has two versions, standard and flexible environment, here we are focusing on the standard version.

If you are a fan of Heroku, App Engine is the closest experience you can get inside GCP.

You need an app.yaml, a requirements.txt and then you are ready to go.

There's a repository with a runnable example, with a few things to notice:

I use Poetry as dependency management tool and I export the requirements as requirements.txt
There's an app.yaml and .gcloudignore

The app.yaml

The app.yaml needs at least two entries: runtime and entrypoint.

App Engine supports both Python 3.7 and Python 3.8, so you can select the runtime you want to use.

The entrypoint is the command that you want to run (like web in a Heroku procfile).

runtime: python38
entrypoint: gunicorn main:app -w 4 -k uvicorn.workers.UvicornWorker

Create and deploy an App Engine application

After you cloned the repository there are only a few commands between having the application running locally and on App Engine

# You'll have to select a region and you can't change it afterwards
gcloud app create
gcloud app deploy

Just wait a couple of minutes and your application should be running online. Pretty neat, isn't it?

Performance

I run some load tests using hey with 5000 requests (hey-n 5000 URL) and I got some really nice numbers. The application is not doing much, so I am basically testing the overhead of the GCP infrastructure

Summary:
  Total:        5.4324 secs
  Slowest:      0.3870 secs
  Fastest:      0.0273 secs
  Average:      0.0517 secs
  Requests/sec: 920.3966

Sweet!

It's also nice to see how the number of instances scales up and down.

Cloud Functions

Cloud functions are the closest thing you can get inside the GCP ecosystem to AWS Lambda, with a key difference:

You can trigger your cloud function directly, HTTP endpoints (triggers) are available out of the box

Google recently announced the GCP API Gateway, now you can have the AWS Lambda & API Gateway experience also on GCP.

You can trigger Cloud Functions with events (Cloud Storage, Pub/Sub, etc...) or by calling them directly.

Let's deploy a cloud function, you can find a runnable example here.

There is one main requirement: you need to have a requirements.txt and a main.py on your base path

gcloud functions deploy movie-recommender \
    --entry-point recommend_movie \
    --runtime python38 \
    --trigger-http \
    --allow-unauthenticated \
    --region=europe-west1

--allow-unauthenticated makes your API endpoint public.
--entry-point specifies the function you want to call inside the main.py.

Cloud Functions triggered by HTTP are based on Flask handlers, so our function receives a request object and needs to return something compatible with make_reponse.

Performance wise they are also good

Summary:
  Total:        10.5700 secs
  Slowest:      4.6717 secs
  Fastest:      0.0407 secs
  Average:      0.0779 secs
  Requests/sec: 473.0385

Cloud Run

Cloud Run is a recent new addition to the serverless GCP suite, it uses knative to run serverless workloads on top of a Kubernetes cluster.

That means you can run Cloud Run on your GKE cluster or on a cluster managed directly by Google.

Cloud Run offers a few pluses compared to App Engine and Cloud Functions:

You deploy containers
One single instance can handle multiple requests at the same time
Supports gRPC
You are not limited by the supported languages or versions

In case you want to play with Cloud Run you can find (another) runnable example.

To deploy an application on top of Cloud Run you only need a docker image that can be pulled by the Cloud Run service.

It can be a publicly available image, or an image hosted on the GCP Container Registry.

In my case I decided to use the GCP Build service and push the image to the GCP Container registry.

After you cloned the repository you can run this command

gcloud builds submit --tag eu.gcr.io/$YOUR_GCP_PROJECT/movie-recommender

This command will build and push the docker image to the GCP Container Registry.

Then we are ready to deploy the Cloud Run service

gcloud run deploy movie-recommender \
  --image eu.gcr.io/$YOUR_GCP_PROJECT/movie-recommender \
  --platform managed \
  --allow-unauthenticated \
  --region europe-west1 \
  --args=gunicorn,main:app,-w,4,-k,uvicorn.workers.UvicornWorker

Let's explain each flag:

--allow-unauthenticated: same as cloud functions, enable a publicly reachable URL
--platform: running on a Kubernetes cluster managed by Google
--region: region in which I want to run my cloud run
--args: this is the command I want to run inside the container

And because we love benchmarks:

Summary:
  Total:        5.7388 secs
  Slowest:      0.2160 secs
  Fastest:      0.0386 secs
  Average:      0.0554 secs
  Requests/sec: 871.2571

Again really nice results.

Is serverless always the right answer?

No, leaving cost aside there are certain things you can't do or can't do well:

Websockets, GraphQL subscriptions and streaming are limited by the request timeout
Headers and Payload size are also limited
You can't create shiny network topologies and decide how requests are routed, you need to build on top of the GCP integrations (and limits)
You need to be careful if you access services where the number of concurrent clients is limited, a good example is services connecting to PostgreSQL

Takeaways

App Engine, Cloud Run and Cloud functions all come with a pretty good free tier and you pay only when you use them.

They all support private VPC connections through Serverless VPC access. Serverless VPC access is really neat, you can build complex topology on serverless stack, like mixing virtual machines and serverless applications inside the same network.

The lock-in factor is also pretty low, especially for App Engine and Cloud Run.

Being full-fledged applications you can pack them into containers and run them somewhere else.

They are also well integrated with the Stackdriver products (now Operations), that means you get centralized log and monitoring out of the box.

And yes, you can use structlog and get you json logs parsed, just remember to convert level to severity (check this snippet).

App Engine and Cloud Run both support traffic splitting between different revisions, so you get A/B testing or canary deployment for free.

If I had to choose one.... I'd go with Cloud Run.

Cloud functions are also great but they force you writing web applications in a way that I don't like.

But I do love using Cloud Functions for event-based systems, data ingestion or pipeline based on events.

Useful resources

Why you need enums and how to use them in Python

Christian Barra — Mon, 14 Dec 2020 14:12:18 +0000

Let's say that you have a user object with two attributes: name and type.

A user's type can be Admin, Moderator and Customer.

How would you model it?

Using a Tuple

One way is to use a tuple to hold type's values:

from dataclasses import dataclass

USER_TYPES = (
    "Customer",
    "Moderator",
    "Admin",
)

@dataclass
class User:
    name: str
    type: str

Let's try and see if a tuple can do the job:

# Admin
user_a = User("Christian", USER_TYPES[2])

# Customer
user_b = User("Daniel", USER_TYPES[0])

user_a.type == user_b.type
# False

# Ideally it should be USER_TYPES 😞
type(user_a.type)
# <class 'str'>

# Ideally this should be False 😞
user_a.type == "Admin"
# True

len(USER_TYPES)
# 3

[user_type for user_type in USER_TYPES]
# ['Customer', 'Moderator', 'Admin']

if user_b.type:
    print("User is valid")
else:
    print("Not a valid user")
# User is valid

It's a good first attempt, but there are few caveats that I want to address:

type(user_a.type) is str and not USER_TYPES
user_a.type == "Admin" is True
who's gonna remember what USER_TYPES[2] is?

Addressing the last point is easy, but the first two are tricky: I need to the separate the UserType from its concrete representation (in this case a string).

Using Constants

Another approach to consider is to assign numerical values to constants¹:

from dataclasses import dataclass

CUSTOMER, MODERATOR, ADMIN = range(0,3)

@dataclass
class User:
    name: str
    type: int

user_a = User("Christian", ADMIN)
user_b = User("Daniel", CUSTOMER)

user_a.type == user_b.type
# False

type(user_1.type)
# <class 'int'>

user_a.type == "admin"
# False

# Ideally this should be False 😞
user_b.type == 0
# True

if user_b.type == CUSTOMER:
    print("User type: CUSTOMER")
# User type: CUSTOMER

# `user_b.type` should evaluate to True 😞
if user_b.type:
    print("User is valid")
else:
    print("Not a valid user")
# Not a valid user

Using constants really improves readability: I can now use CUSTOMER, MODERATOR and ADMIN.

But it also has some (new) drawbacks:

user_b.type and CUSTOMER are 0, so it's evaluated to falsy
type(user_a.type) is int and not UserType
user_b.type == 0 is True
len(USER_TYPES) is gone, I don't have a way to reference to the collection of user's types

Using a Class

Using a class UserType can be a way to unify my first two attempts.

from dataclasses import dataclass

class UserType:
    CUSTOMER = 0
    MODERATOR = 1
    ADMIN = 2

@dataclass
class User:
    name: str
    # NOTE: type is not UserType 😞
    type: int

Let's see how the UserType behaves:

user_a = User(name="Christian", type=UserType.ADMIN)
user_b = User(name="Daniel", type=UserType.CUSTOMER)

user_a.type == user_b.type
# False

type(user_a.type)
# <class 'int'>

user_a.type == "ADMIN"
# False

user_b.type == 0
# True

len(UserType)
# Traceback (most recent call last):
#   File "<stdin>", line 1, in <module>
# TypeError: object of type 'type' has no len()

if user_b.type:
    print("User is valid")
else:
    print("Not a valid user")
# Not a valid user 😞

[user_type for user_type in UserType]
# Traceback (most recent call last):
#   File "<stdin>", line 1, in <module>
# TypeError: 'type' object is not iterable

I grouped values together, under UserType, and it's definitely more readable than before.

But there are still a few drawbacks, similar to the tuple implementation:

the underlying concrete type for each value is int so you can compare UserType.CUSTOMER to other ints
UserType.CUSTOMER is falsy, because its value is 0
classes are mutable, and the fact that I can modify the class at runtime isn't exactly what I want

Having a class that encapsulates the concept of type/category is useful, but a simple class is leaving us half way.

I need a (new) type that supports enumeration, something that I can count, ideally immutable.

And I need a (new) type where the underlying representation is somehow abstracted away, such as:

type(UserType.ADMIN) is str
# False

type(UserType.ADMIN) is int
# False

The type that I need is called Enum and Python supports Enum, they were introduced with PEP 435 and they are part of the standard library.

From PEP 435:

An enumeration is a set of symbolic names bound to unique, constant values.
Within an enumeration, the values can be compared by identity, and the enumeration itself can be iterated over.

Using Enums

Enum types are data types that comprise a static, ordered set of values.

An example of an enum type might be the days of the week, or a set of status values for a piece of data (like my User's type).

from dataclasses import dataclass
from enum import Enum

class UserType(Enum):
    CUSTOMER = 0
    MODERATOR = 1
    ADMIN = 2

@dataclass
class User:
    name: str
    type: UserType

# exactly like with the class example
user_a = User(name="Christian", type=UserType.ADMIN)
user_b = User(name="Daniel", type=UserType.CUSTOMER)

user_a.type == user_b.type
# False

type(user_a.type)
# <enum 'UserType'> 🚀

user_a.type == "ADMIN"
# False

user_a.type == 2
# False

len(UserType)
# 3

if user_b.type:
    print("User is valid")
else:
    print("Not a valid user")
# User is valid

[user_type for user_type in UserType]
# [<UserType.CUSTOMER: 0>, <UserType.MODERATOR: 1>, <UserType.ADMIN: 2>]

So an Enum is already more powerful than a simple class in modelling our business case. It really helps us express some of the characteristics of a category/list.

There are a couple of more things related to enums, some nice built-in features.

I can access an Enum in different ways, using both brackets and dot notations:

UserType['ADMIN']
# <UserType.ADMIN: 2>

UserType.ADMIN
# <UserType.ADMIN: 2>

Each member of the Enum has a value and name attribute:

UserType.ADMIN.value
# 2

UserType.ADMIN.name
# ADMIN

Another neat feature is that you can't really modify an Enum at runtime:

list(UserType)
# [<UserType.CUSTOMER: 0>, <UserType.MODERATOR: 1>, <UserType.ADMIN: 2>]

UserType.NOOB = 3

list(UserType)
# [<UserType.CUSTOMER: 0>, <UserType.MODERATOR: 1>, <UserType.ADMIN: 2>]

type(UserType.ADMIN) is type(UserType.MODERATOR)
# True

type(UserType.ADMIN) is type(UserType.NOOB)
# False

As you can see enums in Python are really powerful, and they encapsulate well concepts like categories or types.

The Python documentation is great and thoroughly covers all the details, I'd recommend checking it out.

Useful resources

Python constants ↩

How to write a great Dockerfile for Python apps

Christian Barra — Mon, 26 Oct 2020 12:11:47 +0000

In the previous article, how to containerize a Python application, we created a Dockerfile and containerized our application from scratch.

Now we really want our Dockerfile to stand out, make it more production-ready, that's the goal of this article.

We will cover 6 different ways to improve our Dockerfile:

setting env variables and a working directory
avoiding invalidating the Docker cache
changing the default user to non-root
taking care of zombie processes
correctly forwarding signals to our application
updating pip, setuptools and wheel

Intro

This is the Dockerfile we created last time:

# 1. Base image
FROM python:3.8.3-slim-buster

# 2. Copy files
COPY . /src

# 3. Install our deps
RUN pip install -r /src/requirements.txt

While fully functional, there are a few things we can improve regarding usability, security and performance.

You can clone this repository if you want to follow along.

Passing the git commit hash

We want to mark each Docker image and container with a tag, this tag is the git commit hash.

At runtime we should be able to determine which version of our software we are running.

The idea is that every artifact we generate is traceable, we can go back and check which commit generated it.

The ARG and ENV instructions can help us achieving it.

ARG specifies arguments that we can pass to the docker build command, ENV are env variables set inside the Dockerfile and accessible at runtime, from within the container.

This is the new Dockerfile, with ARG and ENV:

FROM python:3.8.3-slim-buster

# 👇
ARG GIT_HASH
ENV GIT_HASH=${GIT_HASH:-dev}
# 👆

COPY . /src

RUN pip install -r /src/requirements.txt

The -dev is a way to specify defaults. If the GIT_HASH argument is omitted then GIT_HASH will be set to dev.

Let's build our Docker image and check the GIT_HASH env variable:

> docker build -t movie-app .
> docker run --rm movie-app env | grep GIT_HASH
GIT_HASH=dev

How do we pass the git commit hash to our Docker image?

We use the --build-arg flag from the Docker cli:

# this will work on bash but not on 🐟
> export GIT_HASH=$(git rev-parse HEAD)
> docker build --build-arg GIT_HASH=${GIT_HASH::7} -t movie-app .
> docker run --rm movie-app env | grep GIT_HASH
GIT_HASH=6a78e6b

We don't need the whole commit hash, the first 7 characters are enough.

Why are we not passing the base image using ARG?
Because we don't want to change the base Docker image from the Docker cli, but only through a new commit.

Adding a working directory

Right now we are copying our files inside a /src folder and then we specify all the other paths relative to /src.

Wouldn't be nicer if we could specify a working directory and run commands from that folder?

That would be neat, and WORKDIR is exactly what we need.

FROM python:3.8.3-slim-buster

ARG GIT_HASH
ENV GIT_HASH=${GIT_HASH:-dev}

# 👇
WORKDIR /project

# 👇
COPY . .

# 👇
RUN pip install -r requirements.txt

After we specify a WORKDIR, any RUN, CMD, ENTRYPOINT, COPY and ADD instructions that follow will use that working directory.

Note how the path of COPY and pip install changed.

Let's test our application:

> docker build -t movie-app .
# 💥 it's not python /src/app.py anymore 💥
> docker run --rm -p 8888:8888 movie-app python app.py
> curl localhost:8888

Caching dependencies

Our application has a small number of external dependencies, the requirements.txt contains only a few dependencies, so the pip install command is fast, just a couple of seconds.

What if it were taking minutes instead of seconds?

Wouldn't be better to cache our dependencies until something changes?

If you try to modify any file inside our application's folder and try to run the Docker build command you will see how Docker builds the image starting from zero.

If you check the console output you should see something like this:

Step 6/7 : RUN pip install -r requirements.txt
 ---> Running in 2233484e3f72

Basically any change to our codebase, even if it's not related to requirements.txt will invalidate the Docker cache.

We can be smarter and save some time, we just need to install our dependencies first.

FROM python:3.8.3-slim-buster

ARG GIT_HASH
ENV GIT_HASH=${GIT_HASH:-dev}

WORKDIR /project

# 👇
COPY requirements.txt ./
RUN pip install -r requirements.txt
# 👆

COPY . .

We added a new COPY, just for requirements.txt, and moved the pip install right after.

If you now try to build the Docker image again, then change the main.py and rerun the docker build command again that shouldn't invalidate the cache.

This is the output you should see, Using cache:

> docker build -t movie-app .
Step 6/7 : RUN pip install -r requirements.txt
 ---> Using cache
 ---> cbe7b2865e10

Running your container as non-root user

By default the user running your command inside a Docker container is root.

> docker run --rm movie-app whoami
root

Long story short, Docker containers should not run as root and is highly recommended to change the default user to a non-root user.

How do we change the user?

We create a new one and we set the new user with the USER instruction.

FROM python:3.8.3-slim-buster

ARG GIT_HASH
ENV GIT_HASH=${GIT_HASH:-dev}

WORKDIR /project

# here we create a new user
# note how the commands are using &&
# this helps with caching
RUN useradd -m -r user && \
    chown user /project
# 👆

COPY requirements.txt ./
RUN pip install -r requirements.txt

COPY . .

# 👇 here we set the user
USER user

> docker build -t movie-app .
> docker run --rm movie-app whoami
user

Our user can't create new files outside of the /project folder (user is the owner of the folder):

> docker run --rm touch /hello
touch: cannot touch '/hello': Permission denied
# 👇 but this command would work
> docker run --rm touch hello

Let's test our application to be sure it has all the necessary permissions:

> docker run --rm -p 8888:8888 movie-app python app.py
> curl localhost:8888

Taking care of zombie processes and signals

Each Docker container is a PID namespace, and A PID namespace is a tree, which starts at PID 1, commonly called init.

The entire process of starting the system and shutting it down is maintained by init, when you run a Docker container, PID 1 is what you set inside your ENTRYPOINT.

If you don't set it by default Docker will use /bin/sh -c, which does not pass signals, making almost impossible to gracefully stop your application.

This is why we need a better init, Tini.

Tini doesn't only take care of reaping zombie processes but also of forwarding any signals we send to the Docker container to our application process.

Forwarding signals correctly is really important. Kubernetes relies on signals during the lifecycle of a pod.

More about Kubernetes and signals here.

FROM python:3.8.3-slim-buster

ARG GIT_HASH
ENV GIT_HASH=${GIT_HASH:-dev}
# 👇 you can use env variables to pin library versions
ENV TINI_VERSION="v0.19.0"

# 👇
ADD https://github.com/krallin/tini/releases/download/${TINI_VERSION}/tini /tini
RUN chmod +x /tini
# 👆

WORKDIR /project

RUN useradd -m -r user && \
    chown user /project

COPY requirements.txt ./
RUN pip install -r requirements.txt

COPY . .

USER user

# 👇
ENTRYPOINT ["/tini", "--"]

We have two new instructions here, ADD and ENTRYPOINT.

ADD is a really useful instruction, it can add remote files to you Docker image.

The ENTRYPOINT specifies the entry point for any command, in our case python app.py, pretty much like running /tini -- python app.py

Updating pip, setuptools and wheel

One last thing, it's important to keep pip, setuptools and wheel updated, so it's wise to bump them directly inside our Docker image.

FROM python:3.8.3-slim-buster

ARG GIT_HASH
ENV GIT_HASH=${GIT_HASH:-dev}
ENV TINI_VERSION="v0.19.0"

ADD https://github.com/krallin/tini/releases/download/${TINI_VERSION}/tini /tini
RUN chmod +x /tini

# 👇 STYLE YOUR DOCKERFILE LIKE A PRO
RUN pip install -U \
    pip \
    setuptools \
    wheel

WORKDIR /project

RUN useradd -m -r user && \
    chown user /project

COPY requirements.txt ./
RUN pip install -r requirements.txt

COPY . .

USER user

ENTRYPOINT ["/tini", "--"]

Let's test our application once again:

> docker build -t movie-app .
> docker run --rm -p 8888:8888 movie-app python app.py
curl localhost:8888

And with this last step we are done!

Quick recap

ARG and ENV are neat, use them
Copy and install your dependencies before copying your application
Don't run containers as root, set a new user with USER
Try to prettify your dockerfiles
Always use Tini
Defining a WORKDIR helps

Useful resources

How to containerize a Python Application

Christian Barra — Mon, 17 Aug 2020 17:11:00 +0000

Containers are everywhere, but how do you run a Python application inside a Docker container? This article will show you how!

In case you are wondering, the examples require Python 3.x.

Before diving into containers, let’s talk a bit more about the Python application we want to containerize.

The application is a web API that returns a random movie from a movie collection. Inside our local folder we have 3 files:

app.py # Python application
movies.json # movie collection
requirements.txt # where we specifiy our Python dependencies

app.py contains one API endpoint that returns a random movie:

import os
import json
from pathlib import Path
from random import choice
import cherrypy

PORT = os.environ.get('PORT', 8888)

FOLDER_PATH = Path(__file__).parent

with open(FOLDER_PATH / "movies.json", "r") as f:
    MOVIES = json.loads(f.read())

class Movie:
    @cherrypy.expose
    @cherrypy.tools.json_out()
    def index(self):
        return {"movie": choice(MOVIES)}

cherrypy.quickstart(
    Movie(), config=cherrypy.config.update({
        'server.socket_host': '0.0.0.0',
        'server.socket_port': PORT,
        }))

Inside requirements.txt, we have our dependencies and after running

pip install -r requirements.txt

we are able to run our application with python app.py.

Running curl localhost:8888 should return a random movie, something like:

{
  "movie": {
    "Title": "Opal Dreams",
    "US_Gross": 14443,
    "Worldwide_Gross": 14443,
    "US_DVD_Sales": null,
    "Production_Budget": 9000000,
    "Release_Date": "Nov 22 2006",
    "MPAA_Rating": "PG",
    "Running_Time_min": null,
    "Distributor": "Strand",
    "Source": "Based on Book/Short Story",
    "Major_Genre": "Drama",
    "Creative_Type": "Contemporary Fiction",
    "Director": null,
    "Rotten_Tomatoes_Rating": null,
    "IMDB_Rating": 6.5,
    "IMDB_Votes": 468
  }
}

Cool, the application is working!

How do we containerize our application?

Containerizing a Python application means creating a Docker image that has everything needed to run it: source code, dependencies and configuration..

The first step to containerize our application is to create a new text file, named Dockerfile:

app.py
movies.json
requirements.txt
Dockerfile

Inside the Dockerfile (for what we consider a minimal viable Dockerfile), we need to specify three steps:

Select the base image we want to use
Select the files we want to copy inside the Docker image
Install the app's dependencies

Base image

To specify a base image, we use the FROM command followed by either a private or public image.

In our case, we will use the official Python Docker image, available on the Docker hub.

We will select the latest available Python 3 image based on Ubuntu.

# 1. Base image
FROM python:3.8.5-slim-buster

The image name is made up of two different parts: image:tag. In our case the image is python and the tag is 3.8.5-slim-buster.

That's everything we need for the first step.

Copy our application

To copy our application inside the Docker image, we will use the COPY command:

# 2. Copy files
COPY . /src

This command copies the specified files (or folder) into the Docker image. In our case, we want to copy all files available in our local folder inside the Docker image under the /src path.

It's important to note that the first part of the COPY command is a path relative to the context of the build and not to our local machine.

Install the requirements.txt

The last step is to install our dependencies inside the Docker image. To achieve that we will use the RUN command to RUN pip install:

# 3. Install our deps
RUN pip install -r /src/requirements.txt

One thing to notice is that the path of requirements.txt is different compared with the first time we ran pip install.

The reason behind this is that the copied files are under the /src path inside the image.

Build and run a Docker image

# 1. Base image
FROM python:3.8.3-slim-buster

# 2. Copy files
COPY . /src

# 3. Install our deps
RUN pip install -r /src/requirements.txt

Our Dockerfile is now complete and we can use it to build a Docker image. For that we need to use the docker build command:

docker build -t movie-recommender .

This command builds a Docker image named movie-recommender using your current folder as building context (the . at the end specifies the path of the building context we want to use).

Now we can run the image we just built using the docker run command:

docker run movie-recommender python /src/app.py

This command will execute python /src/app.py inside a container based on the movie-recommender image.

However, if we try to connect to our application using curl localhost:8888, we will get an error.

How is that possible? Why can't we connect to our application that runs inside the container?

The reason is that we didn’t expose our application's port to the local machine. We can do that by using the -p HostPort:ContainerPort flag.

So let’s try to rerun the command again, this time specifying that we want to expose port 8888 locally:

docker run -p 8888:8888 movie-recommender python /src/app.py

and then curl localhost:8888.

Now it works—great! We just containerized a Python application!

Quick recap

Containerizing an application means to create a Docker image that has everything needed to run it—source code, libraries, and configuration.
We used a Dockerfile to specify how to build a Docker image.
docker build builds a Docker image starting from a Dockerfile.
docker run runs a container using a Docker image.

Useful resources

Freelancing & Academia to Industry panel

Christian Barra — Fri, 25 Oct 2019 17:02:10 +0000

On October 11th we hosted a panel at PyConDE with Matteo Guzzo, Tereza Iofciu, Katharina Rasch and Sieer Angar and I had the pleasure to be the moderator.

These are some of the takeaways:

From Academia to Industry

1) As a PhD student you learn how to learn

This will be extremely useful to help you moving to industry and to learn new skills as fast as possible.

2) Do your homework, be prepared

If you don't want to be perceived as a junior during your interviews prepare yourself beforehand, try to understand what companies are looking for and how to approach the interview correctly.
If the interview process is something new for you have some dry runs with companies you don't want to work for.

3) There are pros and cons on both worlds

Katharina Rasch keeps teaching although working as a freelancer and consultant.
You can move to industry without fully leaving academia.

Freelancing

1) Wanna be a freelancer? Just do it

If you think you want to do it just do it, you always have a rollback plan: finding another job as employee.

2) You have to create and grow your network

Meetups, conferences and generally sharing your knowledge is a great way to do it, make yourself visible.

3) Freelancing platforms can be a great way to start

Pivigo, Fiverr, Toptal, Upwork are a great way to kick-start your way to freelancing.

4) Create a cash buffer

Having a cash buffer is fundamental if you want to be able to sustain your first months as a freelancer, that will give you financial freedom and reduce stress.

5) You rates will suck at the beginning

And that's fine. But don't forget to raise them and keep them updated considerinng market rates.

6) Berlin is a bubble

While knowing German might not be a requirement in Berlin the rest of Germany is different although things are changing.

7) Just freely approach companies

If you want to collaborate with them and/or create new connections as a freelancer.

8) Take care of the bureaucratic stuff

Being a freelancer involves some bureaucratic burdens you have to take care like accounting and health insurance.

The panel has been recorded, it will be available on the PyCon DE youtube channel.

How to deal with self-limiting thought before a talk

Christian Barra — Mon, 07 Oct 2019 12:48:03 +0000

A couple of days ago a friend wrote me a message:

Can you check my presentation?
I gave it today and I think it went really bad.

I checked his slides and they were good, same for his notes, so the problem was somewhere else.

So I asked: How did you feel during your talk?

I was really tense

Well, that can be somehow normal.

What were you thinking during your talk?

I am doing it wrong

This slide is wrong

Ouch, self-limiting thought.

Climbing and public speaking

I find many similarities between climbing and public speaking.

The only difference between a climber who climbs 5.15 and one who climbs 5.5 is the magnitude of the doubt and the skills developed to deal with it.¹

Arno Ilgner

Which is probably what separates giving a talk you want to forget about from giving a great talk.

Doubts and self-limiting thought are everywhere and as soon as you face discomfort (jumping on the stage) they are gonna reach your mind at light speed.

So how do you deal with it?

How do you deal with self-limiting thought and with your mind hectically wandering around?

Well I usually do 3 things.

1. Take some time before the talk just for you

I always try to have 5 mins just for me before a talk, you don't have to meditate in a zazen posture, I sit somewhere comfortably and just relax.

I want to enter an inner state of relaxed concentration, where I am focused on the message you want to deliver and relaxed, with a positive thinking mindset (I am gonna do great).

The last thing that I do is power posing.

Yep, I go to something that I consider a private space (a toilette works well) and I do 30 seconds of power posing.

Breathe, consciously and continuously. Shifting your focus from the intruding thoughts to your breathing makes a bridge back to the body, to the flow of action and movement. Keep moving.

Arno Ilgner

2. Focus and remember to breathe

I always try to keep the attention in the moment and remember to breathe.

It's so easy to hold your breath and forget about breathing but it's rather... crucial.

The inhale/exhal process is a great way to focus your attention on your body and control your stress level during your talk.

And as you breathe you take pauses which help giving the right rhythm to your talk.

Don’t react to the doubts. Rather, stay in the pose with the stress and delay reacting.

Arno Ilgner

3. Delay

If a self-limit thought comes to my mind I try to don't react, don't fight with it, just ackownledge the thought and let it go.

The internal dialogue should go like this:

"I am aware my mind thinks this slide sucks... but I am gonna deal with this thought later."

And you? how do you deal with self-limiting thought before a talk?

Useful resources

5.5 and 5.15 are part of the climbing grading system, 5.5 is beginner, 5.15 is super elite. ↩

Onboarding for new tech hires

Christian Barra — Wed, 13 Feb 2019 11:51:57 +0000

Hiring a new member for a tech team is such a long process that as soon as the contract is signed you would like to think that your job is done.

But is it?

No exactly, your hiring process is not yet done, the hardest part has still to be completed, the onboarding.

What is onboarding?

Wikipedia defines onboarding as

the mechanism through which new employees acquire the necessary knowledge, skills, and behaviors in order to become effective organizational members and insiders

So knowledge, skills and behaviors.

Why do you need it?

You spent around a month, during a long (and maybe painful) process, aren't we done? why spend more time?

You want the new team member to be productive as soon as possible and making them happy will probably increase the productiveness and chances that they will stay for a longer time.
First impression matters, this is why onboarding is fundamental.
Giving your new employees an amazing experience is generally better than providing an average or bad experience, so why wondering at all?

And above all it will show that they are important.

So how do you provide a great onboarding experience? You need a plan and a goal.

The onboarding process needs to have a goal, something that is easy to measure or consider done (creating a PR, merging some code, releasing something to production....).

How to create a great plan

You need someone in charge of the process, a tech lead, senior engineer for example, someone that could guide the new hire but is not the line manager, we will call it buddy.

The buddy doesn't have to spend 24 hours with them but should be generally available and responsible for the entire process and should be her/his main point of contact for any question related to the onboarding process and its goals.

So the PIPitRight Team just hired a new awesome member Matteo, and the buddy for him is a Senior Backend Developer member of the same team called Luzia.

The goal is to fix an unclosed socket error, a bug that is afflicting a service and to release the fix to prod.

The ideal plan

The ideal plan is divided into 4 phases (at least):

Before the first day
During the first day
During the first weeks
Follow up

This article will cover the Before the first day and in the next weeks I will cover the remaining phases.

Before the first day

There are a couple of things you need to prepare before she/he joins:

Internally share that a new hire will join the team
Prepp all the accounts/services he/she needs
Share the onboarding plan with the new hire

1. Share with the rest of the team who is joining and a brief bio

This sounds super easy but is easily forgotten or overlooked, especially if you are a remote team.

There's nothing worse than having new people joining your team and you have no clue who the new name inside slack channels is.

I personally find really important (and often disturbing when it doesn't happen) that a new hire is not introduced properly.

The buddy (or a team lead/engineering manager) should introduce the new hire with an email or a message on slack for example.

If your team as regular meetings announcing it there is also fine.

Hello PIPitRight Team!

I am really happy to announce that in the next weeks Matteo will join our team as a Backend Developer.

He is been a ..... (short bio).

He will join the team on .../.../...

Looking forward to working together!

Best,
Luzia

It takes less than 5 minutes, just do it.

Accounts are ready

Provide him/her with the accounts he/she needs email, github, slack, aws, gcp, heroku and all the related services, a nice wiki about how to obtain them is awesome, especially if there are external teams involved (security, ops...).

In many companies is just a matter of creating a new ticket/task.

Having milestones and a task list for each day

Sharing the onboarding plan beforehand with the hire is extremely useful and doesn't require too much time:

Hi Matteo,

I am really happy you will the PIPitRight team!

I am Luzia and I will be your buddy for the first 2 weeks.

We will pair together a couple of times during these weeks and I will introduce you through our tasks system and we will move a couple of tickets together from the backlog to done.

The goal for the 2 weeks is to fix an unclosed socket error, a bug that is afflicting a production service and to release the fix to prod. (ISSUE #4343).

It's going to be a lot of fun, I can't wait for that!

Best,
Luzia

Takeaways

My approach is to avoid overwhelming a new hire and properly manage the cognitive load, with a small and focused scope that would increase during time.

For example having a 360 degrees and deep overview of the tech stack and infrastructure is probably useless and a waste of time at the beginning.

Try to divide the learning process in small chunks, if it's not really needed to achieve the goal/s, it could probably wait, as understanding the entire architecture of your system is not really important at this stage.

This was everything I had for the Before the first day.

I am really looking forward to knowing how your onboarding process looks like, feel free to comment here or tweet @christianbarra.

Python dictionary views

Christian Barra — Mon, 07 Jan 2019 21:27:16 +0000

Christian Barra

@christianbarra

Did you know that "dict.keys()" returns a view that is a set-like object? #Python

12:12 PM - 31 Jul 2018

2 5

Dictionary is one of the Python's greatest features and using the keys(), items() and values() methods is really common.

first_dictionary = {"a": 1, "b": 2}
for key, value in first_dictionary.items():
    print(f"Key {key} with value {value}")

# Key a with value 1
# Key b with value 2

But do you know which kind of object is returned?

They all return a special object, called view.

Why are views useful?

they provide a dynamic view on the underline object (you change the dictionary and the view will change as well)
the object returned by keys() and items() behaves like a set-like object (with items() when the pairs are hashable)

And being a set-like object means you can use the set operations.

Let's consider an example, where we want to find the common keys between 2 dictionaries.

first_dictionary = {"a": 1, "b": 2}
second_dictionary = {"b": 2, "c": 3}

first_dictionary.keys() & second_dictionary.keys()
# {'b'}

& is the intersection operator and returns the common elements between our dictionaries' keys in this case.

What about the elements that are not in common?

first_dictionary = {"a": 1, "b": 2}
second_dictionary = {"b": 2, "c": 3}

first_dictionary.keys() ^ second_dictionary.keys()
# {'a', 'c'}

This is called simmetric difference.

One thing that you cannot do is change the dictionary while iterating over the view object.

for key, value in first_dictionary.items():
    del first_dictionary[key]

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
RuntimeError: dictionary changed size during iteration

Discovering the pathlib module

Christian Barra — Wed, 12 Dec 2018 04:49:37 +0000

// Detect dark theme var iframe = document.getElementById('tweet-1022436417887248384-223'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=1022436417887248384&theme=dark" }

The Python Standard Library is like a gold mine, and the pathlib module is really a gem.

pathlib provides an useful abstraction on top of different filesystems (posix and Windows).

But that's just a small part of the very friendly user experience that you can have.

# files in my folder
README.md  example.py subfolder/README.md

We can import Path from pathlib

from pathlib import Path

path = Path(".")
# PosixPath('.')

files = [file for file in path.iterdir()]
# [PosixPath('subfolder'), PosixPath('README.md'), PosixPath('example.py')]

Path() returns (in this case) a pathlib.PosixPath object with a very handy iterdir, a useful generator for when you have a lot of files in your folder.

path.iterdir()
<generator object Path.iterdir at 0x10aca2cf0>

Do you want to get only the files with a specific format?

md_files = [
    file for file in path.iterdir() if file.suffix == '.md'
    ]
# [PosixPath('README.md')]

You can get a similar result using glob

md_files = [file for file in path.glob("*.md")]
# [PosixPath('README.md')]

glob is quite powerful, using the ** pattern you can search recursively

md_files = [file for file in path.glob("**/*.md")]
# [PosixPath('README.md'), PosixPath('subfolder/README.md')]

If you want to learn more about the pathlib module the PEP 428 and the official documentation are the right places where to start.

How to reverse a list in Python

Christian Barra — Thu, 06 Dec 2018 08:23:47 +0000

// Detect dark theme var iframe = document.getElementById('tweet-1022074115476389888-885'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=1022074115476389888&theme=dark" }

Reversing a list in Python is really, really easy.

This doesn't apply only to list but to sequence in general (string and tuple for example).

msg = "hello there"
print(msg[::-1])
# ereht olleh

my_tuple = (1, 2, 3)
print(my_tuple[::-1])
# (3, 2, 1)

The secret behind this magic? The slice object.

name = "christian"
print(name.__getitem__(slice(None, None, -1)))
# 'naitsirhc'

While old I really recommend reading the Python 2.3 What's new, when the support for built-in types was introduced.