Forem: Ryan Baker

How to make a Gmail Addon

Ryan Baker — Sat, 13 Jan 2024 14:32:08 +0000

Over the past six months, we've started building TinyCadence. TinyCadence is a simple Gmail inbox automation tool that allows you to create email sequences and enroll contacts in them (think: a simpler version of Outreach or Salesloft). We want users to not interrupt their work in order to use the automation, so we decided to build an integrated Gmail experience. Ideally, the users would have an experience akin to this:

This sidebar lives inside your Gmail inbox

And when you click it, it will take the contact through a flow like this:

An example email sequence.

If you haven't worked with Google's developer documentation before, it leaves something to be desired. They have great detail, but are lacking illustrative examples, a unified table of contents, and an index to find the correct documentation.

I decided to write this blog post in order to help the community and demystify the process of making a Gmail add-on as I learn about it.

Why a Gmail add-on?

For a while, we decided to weigh the options between a Chrome Extension and a Gmail Add-On. Chrome Extensions came with great flexibility, but at a great cost: depending on DOM manipulation. This meant that we just had to cross our fingers and hope that the layout of the Gmail Inbox never changed, or else test in all possible variations of it. The fragility of it kept us from diving in head first to an extension.

Reading the introduction documentation to Google Add-Ons felt like the right fit based on their pitch:

The purpose of many emails is to get the recipient to do a specific task or reach a goal, such as adding a calendar event, filling out a form, making a reservation, or using other applications. However, recipients then have to complete the task without any further prompting, often doing a number of manual steps.
You can save time and effort for your users by automating these tasks with Google Workspace Add-ons.

This seemed to fit what we wanted to do like a glove: provide an integrated Gmail experience with added context and automation.

The Setup

In order to get started with your Gmail Add-On, you will need:

a Google cloud project with OAuth consent screen info filled out (not detailed here)
an app server that can serve data to your Gmail add-on client code (not detailed here)
a Google app script project (this is what you will publish as your "Gmail Add-On")
your add-on code

First, you will need to create an App Script project. Navigate here to create one. The easiest way to manage deployments and upload of your add-on code is with clasp. In order to use clasp, turn on the setting that allows CLI tools. Then install clasp and run clasp login. Once you login, run clasp clone <script ID> in order to get the app scripts code locally. To get your script ID, navigate to the project settings page and scroll down.

Next, you need to tie you script project to a Google Cloud Platform (GCP) project. This is because you need an OAuth consent screen, and the only way to have one and get it approved/verified is through a GCP project. Once you have created your GCP project, copy the project ID. In your App Scripts project settings, click "change project" under "Google Cloud Platform (GCP) Project" and paste in the project number for your Google Cloud Project (found on the cloud project dashboard under "Project Info"). This makes sure that your GCP Project and app script are tied together. You need to tie them together because they are approved by different teams and you need your GCP project approved before your add-on can be approved.

From here, you are ready to begin coding!

The Code

Once you used clasp to clone the project, you need to edit your project manifest to give your script the configuration it needs to run. Here is a small sample to get started:

{
  "timeZone": "America/New_York",
  "dependencies": {
    "enabledAdvancedServices": [
      {
        "serviceId": "gmail",
        "userSymbol": "Gmail",
        "version": "v1"
      }
    ]
  },
  "exceptionLogging": "STACKDRIVER",
  "runtimeVersion": "V8",
  "addOns": {
    "common": {
      "homepageTrigger": {
        "runFunction": "onHomepage",
        "enabled": true
      },
      "name": "TinyCadence",
      "useLocaleFromApp": true
    },
    "gmail": {
      "contextualTriggers": [
        {
          "unconditional": {},
          "onTriggerFunction": "onGmailMessage"
        }
      ]
    }
  },
  "oauthScopes": [
    "https://www.googleapis.com/auth/gmail.addons.current.message.metadata",
    "https://www.googleapis.com/auth/userinfo.email",
    "https://www.googleapis.com/auth/script.external_request",
    "https://www.googleapis.com/auth/script.locale",
    "https://www.googleapis.com/auth/gmail.addons.execute",
    "https://www.googleapis.com/auth/gmail.modify",
    "https://www.googleapis.com/auth/calendar.events.readonly"
  ],
  "urlFetchWhitelist": [
    "https://*.ngrok.io/",
    "https://*.ngrok-free.app/",
    "https://*.api.your-website.com/"
  ]
}

There are a few important things here:

Your "runFunction" and "onTriggerFunction" values must be names of functions in your add-on code
Your "oauthScopes" must be exactly the same between this file, your script project settings and your GCP project OAuth consent screen settings
Your "urlFetchWhitelist" should include some way for you to send requests to a test environment. In this example, I used ngrok to serve API requests from my local server while developing.

In your app scripts project code editor, select the main .gs file and click "Run". This will prompt you to authorize your test add-on. Accept the permissions and then select "test deployments" from the "deployments" dropdown.

Try starting out with this sample code for your script project:

const BASE_API_URL = 'https://my-ngrok-id.ngrok-free.app';

const getFetchOptions = (method) => {
  const token = ScriptApp.getOAuthToken();
  return {
    // I tried sending a header with the prefix "X-Google" once
    // be aware that Google will silently drop these, as they
    // reserve those headers for themself
    headers: {'x-app-script-token': token},
    method: method || 'GET'
  };
}

const fetchHomepageData = () => {
  // you need to implement this in your server and return something
  const res = UrlFetchApp.fetch(`${BASE_API_URL}/google-addon/homepage`, getFetchOptions());
  return JSON.parse(res.getContentText());
}

const onHomepage = () => {
  try {
    const {users} = fetchHomepageData();
    console.log(users);
    if (users.length === 0) {
      return makePromptSignupCard();
    } else {
      return makeHelloWorldCard(users);
    }
  } catch (error) {
    console.log('error', error);
    return makePromptSignupCard();
  }
}

const makePromptSignupCard = () => (
  CardService
    .newCardBuilder()
    .setHeader(
      CardService.newCardHeader()
        .setTitle('TinyCadence in Gmail')
        .setSubtitle('Use cadences to supercharge your inbox'))
    .addSection(
      CardService.newCardSection()
        .setHeader('Signup')
        .addWidget(
          CardService
            .newTextParagraph()
            .setText(
              'With TinyCadence, you can automate the boring ' +
              'parts of outreach so you can focus on the engaged ' +
              'respondents without wasting time.'))
        .addWidget(
          CardService
            .newTextButton()
            .setText('Signup')
            .setTextButtonStyle(CardService.TextButtonStyle.FILLED)
            .setBackgroundColor('#A160FF')
            .setOpenLink(
              CardService.newOpenLink().setUrl('https://app.tinycadence.com/auth/signup')
            ))
        )
     .addCardAction(makeVisitTCAction())
     .build()
);

const makeHelloWorldCard = () => (
  CardService
    .newCardBuilder()
    .setHeader(
      CardService.newCardHeader()
        .setTitle('TinyCadence in Gmail')
        .setSubtitle('Welcome back')
    )
    .build()
);

Apparently, there's a way to make all this syntax from just JSON, but I haven't take the time to learn that (I'm also not sure it would save much time).

If you're curious what components are available and what they look like, here is a link to Google's style guide for add-ons (which includes a GSuite UI Kit in Figma).

Now try pushing your code via clasp push. You should then see your add-on in the sidebar of your Gmail inbox:

If you don't see it, make sure that you are signed into the same account that you authorized earlier.

Conclusion

While the process isn't very easy or clear, hopefully you can learn from my trial and error.

From here, the documentation linked above should be a great help to get more advanced UI and interactions going!

Happy coding 💻

3 Tips to Make Your API Dazzlingly Fast

Ryan Baker — Sat, 13 Jan 2024 14:30:49 +0000

We all want fast APIs. We want to see the charts in NewRelic or Heroku show high request volume and low latency. In this article I’m going to show you three tips to make sure your most critical API endpoints are dazzlingly fast.

At Drift, we have a few critical endpoints that we call “critical path”. If these are slow, then something critical to Drift is hurting. The example that I’m going to use in this article is the “create a new message” endpoint.

Example setup

from flask import Flask, request, session
...
from drift.message.utils import extract_message_params
from drift.conversation.models import Conversation
from drift.geo import ip_based_lookup
from drift.routing import do_routing_for_conversation

app = Flask(__name__)

class Message(MyORMBaseClass):
    """This class represents a message row in the DB."""
    ...

@app.route('/messages', methods=['POST'])
def create_new_message:
    message_params = extract_message_params(request.json)
    conversation = Conversation.find(message_params['conversation_id']) or Conversation.create()
    message = Message.create(conversation=conversation, **message_params)
    location = ip_based_lookup(request.remote_addr)
    message.location = location
    if not conversation.participants:
        do_routing_for_conversation(conversation)
    return message, 201

There are a few things to note here:

This isn’t how Drift actually works, it’s just a contrived example
There are a few things we’re missing here from any normal API (authentication, error handling, etc.)
We have a couple functions that I won’t be going into for the sake of brevity (let me know if they’re unclear and I can clarify them)

So we have our endpoint in production. It’s getting a ton of use. But we’re pulled aside by the director of engineering and told that the median response time is 1.2 seconds! 😱 Our director says she can show us the timing of the endpoint broken down by line of code and this is what it looks like:

message_params = extract_message_params(request.json)  # 80ms
conversation = Conversation.find(message_params[‘conversation_id’]) or Conversation.create()  # 10ms
message = Message.create(conversation=conversation, **message_params)  # 125ms
location = ip_based_lookup(request.remote_addr)  # 380ms
if not conversation.participants:  # 0.2ms
    do_routing_for_conversation(conversation)  # 605ms

What do we do first? It’s hard not to go with something we know. For example, if I spent a year and a half writing a JSON parser in python, I want to say “We can definitely shave time off of parsing JSON”. The answer, though, should be that we need to order the work by “biggest payoff first”. This means we should try to optimize routing first.

Let’s say, for one reason or another, that we can’t currently make routing faster. Maybe it’s too big a project, maybe the routing team is busy. Whatever the reason, we can’t make do_routing_for_conversation any faster. This brings me to the first method you can use.

Defer processing to threads

If we think of our endpoint, it has the job “create a new message”. It isn’t our endpoint’s job to do routing for a conversation. Just because we need to do it, doesn’t mean we need to do it now. What if we did routing in a separate thread, since it’s not changing the object that’s being returned (the message)?

Let’s make a small utility that allows us to spin up separate threads that still have the Flask app context:

from threading import Thread
from flask import current_app

class FlaskThread(Thread):
    """A utility class for threading in a flask app."""
    def __init__(self, *args, **kwargs):
        """Create a new thread with a flask context."""
        super().__init__(*args, **kwargs)
        self.app = current_app._get_current_object()

    def run(self):
        """Run the thread."""
        # Make this an effective no-op if we're testing.
        if not self.app.config['TESTING']:
            with self.app.app_context():
               super().run()

Then we can use that helper class like this:

...
from drift.utils import FlaskThread
...
@app.route('/messages', methods=['POST'])
...
    if not conversation.participants:
        th = FlaskThread(target=do_routing_for_conversation,
                         args=(conversation,))
        th.start()
    return message, 201

And now our endpoint is much faster!

But, we now have a scenario where we’re manually spinning up new threads every time we create a message with that endpoint. While this isn’t always bad, we should at least be aware of some of its drawbacks. In our case, we should worry that a spike in traffic will cause our server to spin up two threads per message create instead of one. This could cause our CPU to hit a ceiling and slow down processing for other things.

So threads are a useful tool, but might be best used for a lower traffic endpoint. One that has a slow operation that we still want a quick response from.

Process work on a queue

If you need real time, distributed work, then you’re going to need queues. They give you a lot of benefits: asynchronous work, speed, separation of concerns, easy distribution & fanout. The cost is that we will have to debug our problems on a distributed system. This is easy as long as your queue consumers don’t share resources.

Routing a new Drift conversation to the right Drift user is a great example of when to use queues to our advantage because:

1) it takes a long time

2) we don’t need to route before creating the message

At Drift, we use AWS resources as much as we can. For queueing, we send queue messages to SNS topics and let them “fan out” to SQS queues. For python, you can use a library like Celery, RQ, or Pyres to easily manage queue consumers. The idea is that you make consumers for things that need to happen ASAP, but don’t need to block message creation. With queues, your code might end up looking like this:

@app.route('/messages', methods=['POST'])
def create_new_message:
    message_params = extract_message_params(request.json)
    conversation = Conversation.find(message_params['conversation_id']) or Conversation.create()
    message = Message.create(conversation=conversation, **message_params)
    location = ip_based_lookup(request.remote_addr)
    message.location = location
    send_message_to_sns(message) # 25ms average
    return message, 201

While sending the message to SNS is in the critical path, it allowed us to move routing a new conversation to a queue consumer. This means that we cut a 605ms operation to 25ms and the work of routing a conversation is deferred to a worker. We just cut our endpoint’s median response time in half!

Now that we cut down speed by moving routing out of the critical path, let’s focus on the next biggest bottleneck: IP based location lookup. For this example, we use a lookup database, which brings me to my final strategy:

Know your database performance

One of the most important things you can know is the performance of your database queries. Let’s take a look under the hood of our contrived example. We’ve got two tables in PostgreSQL:

create table geo_id_to_name (
  geoname_id bigint UNIQUE,
  locale_code varchar,
  continent_code varchar,
  continent_name varchar,
  country_iso_code varchar,
  country_name varchar,
  subdivision_1_iso_code varchar,
  subdivision_1_name varchar,
  subdivision_2_iso_code varchar,
  subdivision_2_name varchar,
  city_name varchar,
  metro_code varchar,
  time_zone varchar,
  is_in_european_union boolean
);
create table ip_to_geo_id (
  network cidr,
  geoname_id bigint references geo_id_to_name(geoname_id),
  registered_country_geoname_id bigint,
  represented_country_geoname_id bigint,
  is_anonymous_proxy boolean,
  is_satellite_provider boolean,
  postal_code varchar(50),
  latitude float(20),
  longitude float(20),
  accuracy_radius bigint
);

And our ip_based_lookup function does this:

SELECT * FROM geo_id_to_name NATURAL JOIN ip_to_geo_id where network >>= inet('IP ADDRESS HERE');

So what’s making that query so slow if that’s all the ip lookup is doing? We could analyze the python code for speed, but let’s use some intuition. Anytime you see a simple database query take a long time (more than 5ms), an alarm should go off in your head. “Where’s the index?” Your inner self should say. If you look at those tables, you might be horrified to find that there is no index 😱.

To understand why indexing is important, let’s have SQL explain these queries to us. This is the plan for that query before we add the indices:

Let’s add an index on our “ip_to_geo_id” table. We’re looking up a column of type CIDR. What is that and how should we index it? Lucky for us, PostgreSQL has great docs on what a CIDR type is, what operations you can use on it, and how to index those operations. For our index, we want this:

create index ix_inet_ops_network on ip_to_geo_id using gist (network inet_ops);

And here’s a look at the explain after adding the index!

After indexing, that line of code went from 380ms on average to 10ms on average!

You may not be using PostgreSQL, but the lesson is the same: make sure you know and take advantage of your database’s strengths.

In Conclusion

We talked about 3 ways to make your API endpoints faster. Keep in mind that the first two methods didn’t make code and faster, they just moved the processing somewhere else. Either a new thread or a queue worked is now doing the slow, not immediate work.

You should be viewing the amount I covered the three topics as introductory and not as a reference. There are a lot of nuances that I didn’t cover. If you’re interested in either of the first two topics, check out this distributed systems book. If you want to know more about SQL query optimizations, this website is a wonderful (and free) reference.

Happy coding 💻

How to Build an Embedded Widget with React & Redux

Ryan Baker — Sun, 07 Jan 2024 17:38:23 +0000

In this article I’m going to go over how to set up a React project if you wanted to build something like Drift. At Drift I often work on the chat widget. Outside of work, I recently open sourced a side project called Weasl. Weasl allows you to add pain free user login to your website. These are complex projects that leverage React and Redux to manage rendering and application state. In this blog post, I’m going to do a high level walkthrough of the architecture, client side API, and management of loading your embedded widget on a client’s page.

Anatomy of the Embed

The embed is made up of 3 parts:

The snippet — this is what the user pastes onto their website. This code is primarily responsible for loading the shim
The shim — this is the code that runs on the client’s website and talks to your embedded widget
The widget — this is iframed (for security reasons) webapp that receives messages from and sends messages to the parent window. This is the only thing talking to your server API.

I’ll go over all of these in more detail.

The snippet

My side project’s (Weasl) snippet looks like this:

(function(window, document) {
if (window.weasl) {
  console.error('Weasl embed already included');
  return;
}
window.weasl = {},
m = ['init', 'login', 'signup', 'setAttribute', 'getCurrentUser', 'logout', 'debug'];
window.weasl._c = [];
m.forEach(me => window.weasl[me] = function() {
  window.weasl._c.push([me, arguments])
});
var elt = document.createElement('script');
elt.type = "text/javascript";
elt.async = true;
elt.src = "https://js.weasl.in/embed/shim.js";
var before = document.getElementsByTagName('script')[0];         before.parentNode.insertBefore(elt, before);
})(window, document, undefined);
weasl.init('MY CLIENT ID HERE');

This seems daunting to look at, so let’s deconstruct it.

First, we want our snippet to run immediately when the page loads. This means we need an IIFE.

(function(){})();

We want to declare our global (window) variable that will be used to load the widget and provide the JS SDK.

(function(window) {
if (window.weasl) {
  console.error('Weasl embed already included');
  return;
}
window.weasl = {};
})(window);

Next, we want to add the shim’s script tag to the page:

(function(window, document) {
if (window.weasl) {
  console.error('Weasl embed already included');
  return;
}
window.weasl = {};
var elt = document.createElement('script');
elt.type = "text/javascript";
elt.async = true;
elt.src = "https://js.weasl.in/embed/shim.js";
var before = document.getElementsByTagName('script')[0];         before.parentNode.insertBefore(elt, before);
})(window, document);

Next, we want to keep track of the library calls that happen before our script is loaded, so we can make sure to call them after it’s been loaded. I’ve renamed a couple variables for clarity:

(function(window, document) {
if (window.weasl) {
  console.error('Weasl embed already included');
  return;
}
window.weasl = {},
methods = ['init', 'login', 'signup', 'setAttribute', 'getCurrentUser', 'logout', 'debug'];
window.weasl._beforeLoadCallQueue = [];
methods.forEach(methodName => {
  window.weasl[methodName] = function() {
    window.weasl._beforeLoadCallQueue.push([methodName, arguments]);
  }
});
var elt = document.createElement('script');
elt.type = "text/javascript";
elt.async = true;
elt.src = "https://js.weasl.in/embed/shim.js";
var before = document.getElementsByTagName('script')[0];         before.parentNode.insertBefore(elt, before);
})(window, document);

And finally, we want to make sure we call the initialize function of our SDK.

(function(window, document) {
if (window.weasl) {
  console.error('Weasl embed already included');
  return;
}
window.weasl = {},
methods = ['init', 'login', 'signup', 'setAttribute', 'getCurrentUser', 'logout', 'debug'];
window.weasl._beforeLoadCallQueue = [];
methods.forEach(methodName => {
  window.weasl[methodName] = function() {
    window.weasl._beforeLoadCallQueue.push([methodName, arguments]);
}
});
var elt = document.createElement('script');
elt.type = "text/javascript";
elt.async = true;
elt.src = "https://js.weasl.in/embed/shim.js";
var before = document.getElementsByTagName('script')[0];         before.parentNode.insertBefore(elt, before);
})(window, document, undefined);
weasl.init('MY CLIENT ID HERE');

That’s about it for the anatomy of the snippet. Next up is

The Shim

The job of the shim is to figuratively “shim” your code into the client’s website. The shim and the snippet both run on the client’s website. They should both be very lightweight and minimal so they can load as fast as possible (or allow the client to defer loading, if they want).

The shim is more complex, so I won’t include all of the code in this blog post. You can find Weasl’s shim in a single open source file, though. The basic idea is this:

// see if init() was called
// if so, create and add the iframe to the page
// tell the iframed webapp to start loading
// listen for iframe to tell us it's done
// add the real methods to the window SDK
// run the queued up calls that were made before we were ready

and a more fleshed out version of the code

class WeaslShim {
  init = (clientId) => {
    this.clientId = clientId;
    this.mountIframe();
  }
  mountIframe = () => {
    const iframe = document.createElement('iframe');
    iframe.onload = () => {
      this.iframe.contentWindow.postMessage({
        type: 'INIT_IFRAME',
        value: { clientId: this.clientId }
      }) '*');
    };
    // IFRAME_URL is the url for the index HTML page of our
    // plain old react/redux webapp
    iframe.src = IFRAME_URL;
    iframe.crossorigin = "anonymous";
    this.iframe = iframe;
    window.addEventListener("message", this.receiveMessage, false);
    const wrapper = document.createElement('div');
    wrapper.appendChild(this.iframe);
    document.body.appendChild(wrapper);
  }
  receiveMessage = (event) => {
    // this is where we handle when our widget sends us a message
    if(!!event && !!event.data && !!event.data.type) {
      switch(event.data.type) {
        case 'IFRAME_LOAD_DONE':
          this.handleWidgetLoaded();
          break;
      }
    }
  }
  handleBootstrapDone = () => {
    const weaslApi = window.weasl;
    // these methods aren't filled out here, but you can
    // imagine what they might do
    weaslApi.login = this.login;
    weaslApi.signup = this.signup;
    weaslApi.getCurrentUser = this.getCurrentUser;
    weaslApi.setAttribute = this.setAttribute;
    weaslApi.logout = this.logout;
    weaslApi.debug = this.debug;
    weaslApi._c = window.weasl._c;
    this.runPriorCalls();
    window.weasl = weaslApi;
  }
  runPriorCalls = () => {
    window.weasl._c.forEach(([method, args]) => {
      this[method].apply(this, args);
    });
  }
}
export default ((window) => {
  const stubSdk = window.weasl;

  // _c is the real name for the _beforeLoadCallQueue
  const initCall = stubSdk._c.filter(call => call[0] === 'init');
  const shim = new WeaslShim();
  stubSdk.init = shim.init;
  initCall && shim.init(initCall[1]);
})(global)

It’s important to note that the IFRAME_URL is the URL for your react/redux webapp.

This is a lot more complex than the snippet, but the basic idea is all there. Whenever the widget needs to talk to the shim (for example, if it needed to change the size of the div container) then it could send a postMessage to do just that. Last, we’ll finally get to talk about

The Widget

Since we properly encapsulated the snippet and the shim, we get to write our widget as a fully fledged react/redux app!

If you take a look at the code, you can see that the Weasl widget is written just like any other react/redux app. The biggest change is that the top level App component will now listen for post messages and dispatch them into redux:

class WeaslEmbed extends Component {
  constructor(props) {
    super(props)
    window.addEventListener("message", this.receiveMessage, false);
  }
  receiveMessage = (event) => {
    if(this.isWeaslEvent(event)) {
      this.props.dispatch({ type: event.data.type, payload: event.data.value });
    }
  }
  isWeaslEvent = (event) => {
    return !!event && event.data && event.data.type && event.data.type in SharedEventTypes;
  }
  render() {
    ...
  }
}

And as a result of certain actions, we can use middleware to listen to the actions and send a post message back to the parent window.

In Conclusion

We’ve gone over how to create an embeddable widget and all its parts. The things that I’ve left off here, but are also important to consider are things like:

Generating a unique client ID for each user
Making sure you keep yourself secure with a CSP and limiting the iframe postmessage domains
Verifying the page the embed was loaded on is whitelisted
Locking down your API endpoints by restricting access
There are a lot more things to consider when creating a widget that runs on tons of websites, but this is a good introduction to get you started.

Happy Coding 💻