Forem: John Dean

How to Deploy ML models with Google Cloud Run

John Dean — Tue, 08 Dec 2020 19:26:08 +0000

This blog post is part of our series on Serverless Inference for Machine Learning models accompanying our KubeCon 2020 talk: Serverless for ML Inference on Kubernetes: Panacea or Folly? - Manasi Vartak, Verta Inc. We'll be hosting a live replay on December 2nd at 2 PM ET for anyone who missed it.

As builders of an MLOps platform, we often get asked whether serverless is the right paradigm to deploy models. The cost savings touted by serverless seem extremely appealing for ML workloads as for other traditional workloads. However, the special requirements of ML models as related to hardware and resources can cause impediments to using serverless architectures. To provide the best solution to our customers, we ran extensive benchmarking to compare serverless to traditional computing for inference workloads. In particular, we evaluated inference workloads on different systems including AWS Lambda, Google Cloud Run, and Verta.
In this series of posts, we cover how to deploy ML models on each of the above platforms and summarize our results in our benchmarking blog post.

This post talks about how to get started with deploying models on Google Cloud Run, along with the pros and cons of using this system for inference.

What is Google Cloud Run?

Cloud Run is a serverless application running in a container. Cloud Run allows containerized applications to run either on a managed container service or in a customer’s (k8s) kubernetes cluster via Anthos. Cloud Run is powered by Knative, which bills itself as a “Kubernetes-based platform to deploy and manage modern serverless workloads.” Knative supports various service mesh frameworks but in the case of Cloud Run is using the Istio k8s service mesh. Cloud Run allows containerized applications to support modern serverless workloads including scale-to-zero and request concurrency scaling. Cloud Run is the paid / managed version of knative / istio / k8s allowing teams to leverage modern autoscaling container technology without any of the hassle of running the infrastructure.

How is Google Cloud Run appealing?

Pros

No infrastructure (in the case of fully managed)
Dashboards / logs / revision history (in the case of fully managed or on customer’s Anthos cluster)
Cost that scales linearly with usage

Cons

Limits (that cannot be changed) and Quotas (which can be
requested to change) to put constraints on what kind of
models that can be run
Cold starts, because serverless
Cold starts occur, even with steady traffic

Getting started with Google Cloud Run

Creating an endpoint with Google Cloud Run is very similar to creating an always-on container based deployment. But before we can deploy anything we need to perform some Google Cloud Platform setup.

Platform Requirements

You will need to create a GCP account, attach billing information and create a project. If you are a new GCP user then you are in luck-- you will receive $300 in credits to use for your first year!

You should be prompted to create a project, but if you skip that step or want to create a separate project for this exercise, then go create a project

Software Requirements

gcloud CLI will be used to build and deploy.

First, let’s login and create a new gcloud project (to make cleaning things up easier)

Log into gcloud:

$ gcloud auth login

A browser should open prompting you to log into your google account. If authentication is successful you should see the google cloud console landing page in your browser. Return to the terminal and set the current project where PROJECT_ID is the project you created earlier.

Set default project for further commands:

$ gcloud config set project PROJECT_ID

Using a separate project is a good idea so that we can find everything we started in order to clean up our account and avoid perpetual charges.

At this time we can also set a default region for Cloud Run. Feel free to use a different region closer to you.

Set the run/region:

$ gcloud config set run/region us-central1

If you have a POSIX terminal (linux / mac / windows wsl) you can save off the project ID for later use:

$ export PROJECT_ID=$(gcloud config get-value project)

Finally, let’s create a directory for some files we’ll add soon.

$ mkdir cloudrun_example
$ cd cloudrun_example

NLP example (DistilBERT)

For this blog post we will set up DistilBERT and perform question and answer predictions.

There are two python source files, the bootstrap to download and save the model file and the app to serve predictions. Having a separate bootstrap script that we run before creating the endpoint step will greatly improve the cold start time of the endpoint by skipping the need to download the entire model before servicing a request.

Create these files in your cloudrun_example folder:

There should be three files:

cloudrun_example$ ls
app.py bootstrap.py Dockerfile

This Dockerfile references two python scripts from earlier-- bootstrap.py and app.py. Because we have gcloud installed we can offload the docker build and publish to Google Cloud Build:

cloudrun_example$ gcloud builds submit --tag=us.gcr.io/$PROJECT_ID/distilbert:latest

If the command completes with a SUCCESS then the docker container has been built and published. If there is an error, then a google search of the error should lead you quickly towards the answer.

We are now ready to deploy a cloud run endpoint:

cloudrun_example$ gcloud run deploy distilbert \
--image=gcr.io/$PROJECT_ID/distilbert:latest \
--platform=managed \
--concurrency=1 \
--set-env-vars=MKL_NUM_THREADS=2,OMP_NUM_THREADS=2,NUMEXPR_NUM_THREADS=2 \
--cpu=2 \
--memory=3G \
--allow-unauthenticated

If successful, the terminal will return the URL of the new endpoint. e.g. https://distilbert-3bz2zvlxbq-uc.a.run.app

We are now ready to make a prediction! Try it out:

curl -d '{"context":"Roses are red, Violets are blue, Sugar is sweet, And so are you.", "question":"What am I?"}' -H "Content-Type: application/json" -X POST https://distilbert-3bz2zvlxbq-uc.a.run.app/predict
Responds with:

"sugar is sweet , and so are you"

Cloud Run limitations

Now that we’ve seen how easy it is to deploy a serverless ML application to Cloud Run, let’s have some real talk on some limitations. Cloud Run, being a platform that targets a wide variety of applications comes with a set of defaults and constraints that preclude some ML workloads. For example, the max CPUs that can be requested is 4 and the max memory is 4G. When deploying a ML workload, one will probably want to dive into settings to lower container concurrency which is the number of open requests per container. The maximum response time for Google Cloud Run is 60 minutes. While there is no container size limit, containers must be able to load and start within 5 minutes. Cloud Run containers are given 10s to terminate before being forcefully terminated.

Reference:

Default Quotas https://cloud.google.com/run/quotas Container Contract:
https://cloud.google.com/run/docs/reference/container-contract

When building this example, issues with long cold starts necessitated the bootstrap.py file to preload the model into the container. While running benchmarks, it was noted that Cloud Run seemed to suffer from “continuous cold starts” such that even with a very steady throughput, new instances would continuously spin up and the unlucky requestor to that instance would have to wait ~10s for a response. Because of this, cloud run is never able to truly achieve a steady state. It would appear that pods are continuously recycled. In Cloud Run there exists no configuration to customize this behavior compared to a self hosted Knative installation.

On Rolling one’s own Knative vs Cloud Run

Since Cloud Run is built almost entirely on open source technologies, one could choose to install and configure their own k8s cluster, ingress controllers, Istio and Knative to run their own “serverless” workloads directly. Of course to call such an installation “serverless” actually demonstrates how ridiculous the term “serverless” actually is. Of course there are servers… but when using a managed serverless platform like Cloud Run what you, as the customer, release is the responsibility of directly managing server infrastructure. Because Cloud Run is mostly built on open source technologies you always have the choice to host your own infrastructure to run similar workloads.

However, this author will confirm that teams looking to go this route are in for quite the ride. Configuring and running k8s, Istio and Knative are all non-trivial tasks. When you go with a managed platform, like Cloud Run, the intricacies of these platforms are mostly hidden from the user. Self hosting will open up a world of knobs and switches that can be set. While this can be a huge advantage to teams needing to highly tune specialized workloads, for most use cases is a big deep rabbit hole of testing, tuning and more testing.

Not yet afraid? Well we haven’t even covered logs, metrics, revision control and dashboards. With Cloud Run you get web and command line tools for viewing and managing your workloads. These tools are the part of the k8s / Istio / Knative stack that is not open source. Now you are on the hook to add all of these to be able to effectively manage and provide visibility to your workloads.

If one insists on setting up their own infrastructure, then consider taking a look at Anthos. Anthos is yet another installation on a k8s cluster that provides a rich set of integration with Google Cloud Console. An Anthos enabled cluster can be deployed via the Google Cloud Console Cloud Run dashboard as well as connect with all of Google Cloud’s existing monitoring tools. With Anthos customers can control and customize their Knative installation while also having access to all of Google's proprietary console tools.

Conclusion

Google Cloud Run is great for the use cases that fit within the container contract. Because Google Cloud Run is built on top of open source technologies, there is also a path towards self-hosting. In addition, since Cloud Run can scale to zero, you pay as you go. However, you will end up paying more for that compute compared to simply buying Google Cloud Compute instance outright and hosting the service directly. In this case you would also create a floor on expense because you would not be scaling to zero.

Google Cloud Run, with its default settings, might be best suited for smaller HTTP workloads than machine learning. There are hard limits to CPU and memory that prevent running larger ML workloads. The default request concurrency (80) makes sense for parallelizable workloads but will create serious issues for ML workloads. Care should be taken when creating the deployable container to ensure that the container is able to start, run and stop within the constraints of Cloud Run.

Stay tuned for our next blog in this series!

Author biography

John spent over ten years in the enterprise software/embedded/mobile development tools space before transitioning to fullstack web development for the past ten years. John currently leads the frontend team at Verta.

About Verta

Verta provides AI/ML model management and operations software that helps enterprise data science teams to manage inherently complex model-based products. Verta’s production-ready systems help data science and IT operations teams to focus on their strengths and rapidly bring AI/ML advances to market. Based in Palo Alto, Verta is backed by Intel Capital and General Catalyst. For more information, go to www.verta.ai or follow @VertaAI.

Mastering the Code Pair Interview

John Dean — Thu, 30 Apr 2020 05:00:13 +0000

Many companies incorporate a code pair exercise where the applicant must solve a coding riddle. I gave 50+ of these as the interviewer before myself needing to enter the job market where I interviewed with more than a dozen companies and completed many code pair exercises as part of the interview process. In this article I attempt to demystify this process, explain what interviewers want to see as well as provide some strategies for mastering the code pair interview.

The Process

An applicant will typically go through one or more screening calls before being assigned a code pair exercise. Sometimes the applicant will receive a takehome assignment before the code pair exercise. If the applicant passes these initial screenings then the next step may be a code pairing exercise.

Code pairing exercises range from super simple to devilishly complex. Typically the level of the role you are applying for will largely dictate the complexity of the problem, but some companies prefer to give difficult problems to see how candidates perform under pressure. However, even easy questions can be a strong source of stress for the nervous applicant.

After meeting the interviewers the applicant will typically be asked to log into a code pairing website where everyone can see the code as it is being typed and run the code to receive a result. Some requirements and often times some data is provided to the applicant and some, but not all, requirements for a correct solution listed. Usually standard input and output functions of the language (e.g. printf or console.log) are used to output results. The applicant is given a set amount of time to complete the exercise and achieve the correct output.

Sometimes there will be follow up requirements if the applicant solves the exercise early.

The Anatomy of a Code Pair Exercise

There are many pre-existing repositories of coding problems to choose from but also common for individual interviewers or companies to have their own set exercises. Fear not because of the format of code pairing exercises there tends to be a limited set of problems companies can reasonable expect applicants to finish assuming a 30-60 minute window.

The most common exercise will involve the applicant working with some data (lists, objects, strings, numbers etc..)-- either to generate data, transform data or both. The goal of the applicant will typically be to map requirements into code that is generating and/or consuming some kind of data and to achieve the expected output.

The traditional code pair exercise revolved around core CS data structure concepts, e.g. all the kinds of trees and lists. My experience is that these sort of questions are not used much anymore.

In the end the applicant is being asked to implement an algorithm to carry out a task. There will usually be preferred algorithms (solutions) that are kept hidden from the applicant. It is task of the applicant to create a solution within the time period but also the applicant is being judged on much more than their code!

Interviewers want to see how the applicant approaches problems, listen to their thought process and be able to discuss multiple solutions. Knowing what interviewers are looking for will help the applicant pass the interview even if the exercise was not completed.

Preparation

Understand the language you will need to use for the exercise. Sometimes there is a choice and other times the preferred language of the role is used. Be able to write code in the chosen language without needing to lookup language features and syntax during the exercise. The goal should be to spend the majority of the time on the problem, not fighting the language.

Understand algorithms and data structures. Interviewers are often looking for certain key phrases and these key phrases will all be found in the wonderful world of algorithms and data structures. If you feel like you need a refresher or need to know more, check out these free courses online:

Because of the open nature of code pairing exercises one can never really be fully prepared, but there are many things one can do to be ready. It shouldn't need mentioning but obviously be well rested. Try to schedule the interview at a time that is convenient. Have water and/or coffee at the ready. Expect to do a fair amount of talking during the interview.

Understanding Requirements

The applicant is often given incomplete requirements. Ask questions! Asking the right questions to clarify the requirements is often part of the excercise.

If your exercise involves transforming data, a good first questions is to ask, "can I assume that the data is always correctly formatted or do I need to account for bad data". This is a great question because usually the answer is to assume that the incoming data is always correct. This will save a bunch of time. A good response is "Thanks! I'll assume that the data is coming from a well tested module."

Finding Solutions

The quickest way to find a solution to a given code pairing exercise is to google the requirements. Chances are the question being asked has already been documented and solved in multiple languages. But let's assume you are not going to be able to do so. After all the applicant is being watched. Don't assume that just because you are remote that no one will notice the darting of eyes or the reflection of another screen at the ready.

Have a plan! First off do a systems check. Make sure code can be run and results returned. Next, start clarifying requirements. Document any changes to requirements. Next, write stub function/class which to create a general outline that needs finishing. Try to keep talking. Discuss naming. Otherwise just talk through what you are doing. Next, write some failing tests. TTD (Test Driven Development) is not nearly as popular as it once was so the test is not to show that you know TDD (but feel free to mention it if you want!) but to make sure there is a test written in case you run out of time. Having a test, like code comments, is part of writing good clean code which is the end goal.

After thinking through the requirements and asking questions, hopefully a plan is forming. A standard question of the interviewer will be to question the applicant on their solution's time and memory complexity (big O notation-- see algorithms). Get in front of those questions and declare the complexity of different approaches. At this time, don't about follow up requirements. The first goal is always to achieve a working solution for the presented exercise.

Write down your plan as rough pseudo code as code comments. It's OK to be a little vague if some part of the plan is unknown. Mark it as such and move on.

Now the real challenge begins.

Most solutions will involve some amount of iteration-- either a for loop, do...while loop or some kind of recursion. Look for a solution in that order. A for loop is safest because they tend have defined entry and exit parameters. do...while loops are common in these exercises but caution is needed when using them both in a code pairing exercise and in real life! Be sure to really think about how to exit a do...while loop else spend extra time dealing with a locked/crashed program. Recursion is usually a trap both in a code pairing exercise and the real world. When reaching for a recursive solution, additional requirement clarification is needed. The problem with a recursive solution is that they depend on the execution stack which tends to have much smaller memory footprint than the heap which is where general memory allocations come from. Fret not! Recursive solutions can typically be converted into a loop and a stack allocated on the heap to maintain the state stored in the execution stack of a recursive algorithm.

If the current solution involves iterating over the same data over and over then this is a sign that the solution is not going to be time and/or memory efficient. At this point take a step back and consider how fix it. The question will come. If this problem is identified early enough and there is enough time, switch to the a more efficient solution. If you are already well through your existing solution then I recommend completing the assignment as-is but first document the limitation and if possible, the solution as code comments.

By far, the most common solution to resolving time complexity is through the use of a hash map. Be on the lookout for ways to use a hash map to avoid having to re-iterate over data by looking up data in the hash map by key. Often times the solution involves specific methods for generating those keys for the hash map. Another other common solution to resolving complexity is through use of an up-front sort. Always look to see if the problem can be solved efficiently if the data was in a certain order. Sometimes the answer involves both! In my last set of interviews I had a couple of solutions that involved using a hash map and sorting a small data set (2 or 3 variables) to use as a key.

What To Do When Stuck?

We'd all like to ace these problems effortlessly with a correct and efficient solution. If that happens then great! If not, then don't panic! Getting stuck is completely normal and often expected.

I think the most important thing to do, and this takes practice, is to talk out loud as much as possible. Avoid "dead space" where there is no typing and no talking. Always be doing one or both! Usually when I'm stuck I start by talking out loud what the code I have so far does. Hopefully at this point you already have a general plan and some pseudo code. If I don't know what to do I will often simply compare my own code to the psuedo code. The questions I will ask myself are "Does my code implement the pseudo code?" and "What am I missing?". Maybe the pseudo code was incomplete. Add more detail if possible.

There is no shame in admitting you are stuck. Simple say or type "I am stuck here" where here is a specific spot in the code. I would try to avoid asking for a hint too early on. Often times you can get hints without asking simply be talking through the problem. Listen and watch the interviewers. Look and listen for confirmation that an idea is correct or incorrect. If there are no "free" hints then ask for a hint. While it will be noted it is not necessarily a bad thing. After all no one wants software developers that spin their wheels and don't ask for help.

If no actionable hints are forthcoming and there is still more to implement-- then finish what parts of the solution you can. If you can create smaller tests for some part of the solution then that is better than providing no solution at all.

Final Touches

No matter if the exercise is solved or not there are certain attributes a final solution to have. Usually the final code will be saved to be reviewed later. There should be tests and their should be code comments. My experience is that most applicants write code with zero code comments or with little testing. Having participated in many of the discussions that applicants are not privy to I can state unequivocally that good code carries more weight than a correct solution. Think of it this way... If there are multiple applicants with the correct solution, then the applicant with better code wins. In addition, I would be more interested in an applicant that wrote clean, well commented code over someone who quickly banged out a solution. There is always an assumption that the applicant might already know the solution in which chase a bare bones solution might look like a solution that was from rote memory rather than a creative enterprise.

Developing and deploying create-react-app to multiple Firebase environments

John Dean — Thu, 12 Sep 2019 05:19:49 +0000

Introduction

create-react-app makes it easy to quickly get started with React. Firebase makes it easy to quickly get started with web deployments. This article will show how to configure, build and deploy a React app to multiple environments in Firebase. Plus we will get it all working well together with Typescript.

Getting started

First we need to install some tools. These instruction assume a unix terminal prompt. For windows, install WSL (Windows Subsystem for Linux) from the Windows app store

Install NodeJS

If you already have NodeJS installed, you can skip ahead. If not head on over to nodejs.org and download an installer or use nvm to be able to future-proof your NodeJS installation.

Using nvm:

curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.34.0/install.sh | bash

Restart your terminal or follow the instructions on the terminal. Then install node:

nvm install 12

Run create-react-app

If you already have an application then you can skip these and go to installing Firebase.

To make some steps easier let's pick a name for our new application which we will save as an environment variable to reuse.

export REACT_APP=hello-firebase; # replace with your application's name

Use npx and create-react-app to boilerplate a new react application. I recommend using Typescript to enable type-checking

npx create-react-app ${REACT_APP} --typescript

Wait a little bit while create-react-app gets ready. When finished enter the application directory:

cd ${REACT_APP}

Install firebase-tools

In order to work with firebase you will need the firebase command line tool. The easiest way to get it is to install globally with npm:

npm i -g firebase-tools

Create your firebase application

Head on over to console.firebase.google.com and activate Firebase for your Google account. If you select the Spark plan then for moderately sized applications there is no cost.

Once you have activated firebase, go back to the terminal and login:

firebase login

You will be presented with a login prompt in your browser. Click Allow to complete the login

Firebase environments

If you are just playing around then you can skip this step. When you get serious about developing a real application you will want to create multiple Firebase environments. To start let's create two firebase environments-- a development and production environment. The development environment will be the default environment that can be used for trying out new features and the production environment will the the "stable" customer facing environment.

firebase projects:create ${REACT_APP}-production
firebase projects:create ${REACT_APP}-development

Firebase environment names must be unique, so if the name you want is not available, try a different name.

Initialize Firebase

Now we are ready to initialize firebase inside our create-react-app. Firebase has an interactive application for that:

firebase init

For now let's just pick Hosting because that is all I am going to be covering. You can always re-run this command later to add features

We'll use an existing project that we created earlier. If using multiple environments I recommend picking your development environment as the default.

Change the hosting location to build. Answer yes to if the application is a SPA (single page application).

If you going to deploy to multiple environments, then add your secondary environment now:

firebase use --alias production ${REACT-APP}-production; # or whatever name you ended up picking
firebase use default; # switch back to development

Firebase implicit initialization scripts

Implicit initialization is a nifty way to initialize Firebase without config files. When supporting multiple configurations from a single project it is the only easy way to support multiple configurations. We really do not want to have the following code because it is not a good idea to leak non-production environment details in a production application:

if (process.env.REACT_APP_DEPLOY_ENV='production') {
  firebase.initializeApp(productionConfig)
} else {
  firebase.initializeApp(developmentConfig) 
}

Implicit initialization works by loading Firebase javascript from a special /__/ folder. We we deploy our app to Firebase this route will exist for us but locally we will need to do a little more work to set it up.

In your favorite code editor open up the build/index.html that was created by firebase init. At the top of the body tag you will see a series of Firebase related script tags. Select and copy these tags:

Paste them in your public/index.html file at the top of the body tag. Feel free to remove features you are not using.

Firebase serve for local development

firebase-tools comes with a server that supports the implicit initialization, but first we need to set up a few things in create-react-app.

First let's install run-p which is a handy tool for running multiple npm scripts in parallel. In addition we will need to configure an http proxy.

npm i -D npm-run-all http-proxy-middleware

In your package.json change the start script from react-scripts start to:

    "start": "run-p --race dev:firebase dev:react",
    "dev:firebase": "firebase serve -p 4000",
    "dev:react": "react-scripts start",

Now create src/setupProxy.js file and type:

const proxy = require('http-proxy-middleware')

module.exports = function(app) {
  app.use(
    '/__',
    proxy({
      target: 'http://localhost:4000'
    })
  )
}

Now we are ready for local development. In your terminal run:

npm start

If all is well you should see:

Checking developer tools and terminal console you should see that Firebase javascript is loading and no errors are present. Hot reloading works so you can make changes to code and they will be reflected immediately.

Deploying Firebase Hosting

We are almost ready to deploy our react app to Firebase hosting. First we need to add a line to firebase.json to automatically build our application before deploying. Otherwise you will forget to do it. Add the following line to the hosting section:

"predeploy": ["npm --prefix \"$RESOURCE_DIR/..\" run build"]

Now we can deploy:

firebase deploy

React will build and a link will be provided in the console. You should be able to click on that link your React application will load!

Switching Environments

When you are ready to deploy to production all it takes is an environment switch.

firebase use production; # switches all future firebase commands to production
firebase deploy

Or use the -P flag

firebase -P ${REACT_APP}-production deploy; # Does not switch default environment but still deploys to production

Typescript typings

If you start using Firebase in your Typescript application you need to let Typescript know about this global firebase variable that you know have access to. First install firebase not to import but simply to get access to the types contained within:

npm i -D firebase

Now create a src/firebase.d.ts file and copy the following:

import firebase from 'firebase'

declare global {
  export const firebase = firebase
}

Now you have full typescripts bindings available.

Conclusion

I hope this guide makes it easy to use create-react-app together with Firebase for modern web application development combined with free application hosting.