Forem: Christine Spang

IMAP, Therefore I Am.

Christine Spang — Tue, 15 May 2018 18:51:40 +0000

In this post, we’re going to dive into IMAP. IMAP stands for Internet Message Access Protocol, and it’s the open standard that describes how to access messages in an email mailbox. While IMAP is an important part of receiving emails, it's not always the easiest thing to implement (or understand), which is why Nylas exists! In short, IMAP is the protocol that email clients like Mail.app, Thunderbird, and Mailspring use to download messages from your email account and to make changes like archiving messages or sorting them into folders.

A Brief History of IMAP

IMAP was originally created by Mark Crispin at Stanford in the 1980s. Crispin later moved to the University of Washington and spent 20 years there working on the IMAP specs and reference implementation. The IMAP specification comes in the form of a “Request for Comments”, or RFC, which is basically a memo describing how to implement the protocol that has been adopted as a standard by the Internet Engineering Task Force. RFCs may be revised to make clarifications or changes, and the current RFC that describes IMAP is RFC3501 which was published in 2003. That’s right: the protocol most mail clients use to sync email is over 15 years old!

IMAP was initially designed as a better alternative to an older protocol called POP, with the goal being “to permit manipulation of remote mailboxes as if they were local”—specifically, POP’s mode of operation was to download mail to your local machine and delete it from the server, which made it difficult to manage your mail if you were using multiple computers. This was no problem in the days of timesharing mainframes, but became one as personal computers became more prevalent and you might have a PC at home as well as at work, and wanted to browse your email from either place.

The original version of IMAP is lost to history, and IMAP2 (released in 1988) is the first version that made it into the standards. This first version of IMAP only supported “online” operation: it assumed that your mail client would be connected to the server when viewing or modifying messages. You couldn’t use it to sync a copy of your mailbox locally and then update your local copy when you reconnected, because messages were only identified by a “sequence” number, and sequence numbers were not persistent across client sessions. One can only imagine that in those days folks figured that it’d always be the case that whenever you wanted to check your email, you’d be sure to have a reliable network connection. My guess is that it was because computing from mainframes and timesharing systems was essentially logging directly in to a server machine, and meant you had an always-on Internet connection at your disposal.

Fast forward a few years to the dialup era, and it became apparent that hogging your phone line constantly so you could read your email just wasn’t going to cut it. Enter support for “disconnected” operations, which was bolted on to IMAP with the introduction of IMAP4 in 1994. This revision to the protocol added persistent message identifiers (called “UIDs”) and an entire draft spec explaining how you could sync changes made to locally cached data to data stored on a server using IMAP commands. We’ll talk more about this later, but this evolution of the protocol over time is a major reason that implementing IMAP clients is complex in the modern era, as disconnected operation is a required feature for modern apps.

The basic IMAP protocol has remained unchanged since 1996, with all new features since then being implemented in optional extensions to the protocol, some of which have become standards—but adoption of even the most common extensions still varies wildly across email service providers.

How IMAP Works

Now that you know some of the history of the protocol, we’re going to interactively introduce you to the nuts and bolts of how it works. We’re going to connect to an email account, list all of the available folders in the mailbox, and download a message. Along the way, we’ll identify and explain the relevant parts of the protocol needed to understand what is happening.

You’ll be able to follow along as we walk through the protocol, as long as you have access to a terminal, the openssl command-line utility, and an email account that supports IMAP (including Gmail).

TCP, HTTP & IMAP

Many modern client-server protocols take place between a web browser and a server, or an app and an API. These APIs are all implemented on top of a base protocol, HTTP (“hypertext transfer protocol”), which defines the semantics of different types of requests—GET, PUT, POST etc. HTTP is in turn implemented on top of a lower-level protocol called TCP (“transmission control protocol”), which simply guarantees that packets of information are delivered reliably to the destination.

IMAP, like HTTP, is implemented on top of TCP—and it defines the semantics of different types of requests, called IMAP commands. IMAP in general is a lot more concise than HTTP, which was key for IMAP in the early 90s as network connections were very low-bandwidth.

IMAP Step by Step

Let’s get started! Follow along at home in your terminal by typing the commands, subbing in the details of your email account where relevant. Note that if you’re using Gmail, you may need to disable protection against logins from “less secure apps” for this to work.

The first thing any client needs to do is to to make a connection to the remote server on a specific port. These ports can vary because mail servers first accepted plaintext connections and later added support for secure encrypted connections. You can generally find the settings for your provider by checking their help articles. For example, here’s where to find the settings for Gmail. (If you’re curious as to the background of why different ports are used, we recommend this fantastic guide from Fastmail.)

Got your settings to make a secure connection? Let’s go!

On the command line, we can connect using the openssl command-line utility. We have to use openssl rather than telnet, because otherwise we could end up sending sensitive data like passwords and private emails in plaintext over the network, leaving them open to network sniffing by an attacker. openssl protects our connection by making a secure connection and verifying the remote server’s certificate before proceeding.

    $ openssl s_client -connect imap.gmail.com:993 -crlf
    [... certificate verification removed for brevity ...]
    * OK Gimap ready for requests from 216.38.147.18 h1mb556258870pjq

In the above command, s_client means we’re invoking openssl’s functionality to act as a basic SSL/TLS client, -connect specifies which hostname and port to connect to, and -crlf converts when you press “Enter” in your terminal to the characters expected by the IMAP server to properly end a line. After connecting, openssl checks the SSL/TLS certificate on the server to make sure our connection isn’t being hijacked, and the IMAP server displays a greeting that says it is ready to receive requests from the client.

Commands

IMAP commands generally look like this:

[...]

We’ll explain these parts in the context of an example command, called CAPABILITY. CAPABILITY allows you to ask a server which IMAP extensions it supports:

    tag1 CAPABILITY
    * CAPABILITY IMAP4rev1 UNSELECT IDLE NAMESPACE QUOTA ID XLIST CHILDREN X-GM-EXT-1 XYZZY SASL-IR AUTH=XOAUTH2 AUTH=PLAIN AUTH=PLAIN-CLIENTTOKEN AUTH=OAUTHBEARER AUTH=XOAUTH
    tag1 OK Thats all she wrote! h1mb556258870pjq

Wahoo, we just communicated with an IMAP server! Let’s explain what we sent and what we got back.

A command is similar to a request in HTTP. It’s telling the server to do something or asking it for information. In this case, we’re asking the server to tell us which capabilities it supports. More on capabilities in just a bit.

The CAPABILITY command has no arguments, but if it did, we’d simply type them in after the command name and before we press enter.

Tagging

Tags (tag1 in this example) are generated by the client, and the server sends the tag back on the final line of the response to a command. Tags can contain any alphanumeric characters and even some symbols, and don’t need to contain an ascending integer—though that can be a convenient way to generate unique ones. Server responses that start with * are called untaggged responses, which means that they don’t represent the completion of a command requested by the client. In this case, the untagged response is the capability list, and the tagged response is the OK status of the command and a funny message left by the server programmers.

This tagging ability means that it’s possible for a server to handle more than one request at the same time from a client on the same connection, and to indicate their completion by sending back the appropriate tag. In practice, many clients don’t support this ability to send concurrent requests, and simply block waiting for data to arrive on the open socket after a command is sent.

Capabilities

So what does this list of “capabilities” mean, anyway? A “capability” is a short name for a feature the server supports. Some of these capabilities, like AUTH=PLAIN, are included in the base IMAP4rev1 spec, and others are enabled by extensions. Extensions allow new commands to be added to the IMAP protocol without requiring a new version of the protocol specification. Before logging in, the list of supported capabilities on a server will be different from the capabilities after logging in, as there’s little use in listing capabilities that aren’t relevant to the logged-out or logged-in states.

Authentication

Here’s how you authenticate your mail account to an IMAP server:

tag2 LOGIN spang@nylas.com <super-s3krit-password>
tag2 NO [AUTHENTICATIONFAILED] Invalid credentials (Failure)

Server Responses

As you can see, server responses don’t always say OK. Here’s a quick guide to the allowed server responses to a command:

OK = success!

NO = failure!

BAD = I didn’t understand what you said!

The server is free to add a bit more information after the response if it so desires, like [AUTHENTICATIONFAILED] Invalid Credentials (Failure).

tag3 LOGIN spang@nylas.com <super-s3krit-password>
* CAPABILITY IMAP4rev1 UNSELECT IDLE NAMESPACE QUOTA ID XLIST CHILDREN X-GM-EXT-1 UIDPLUS COMPRESS=DEFLATE ENABLE MOVE CONDSTORE ESEARCH UTF8=ACCEPT LIST-EXTENDED LIST-STATUS LITERAL- SPECIAL-USE APPENDLIMIT=35651584
tag3 OK spang@nylas.com authenticated (Success)

As you can see, after a successful login, the server sends the client the list of capabilities again, showing what new capabilities have been unlocked as a result of the login and which old ones have gone away. Some standard extensions in this list are UIDPLUS, MOVE, and CONDSTORE. Some custom extensions supported by Gmail but not elsewhere are ESEARCH and X-GM-EXT-1.

Listing Folders

OK, now that we’re logged in, let’s try to do something useful. First, let’s check which folders exist in the mailbox:

tag4 LIST
tag4 BAD Could not parse command

Whoops! As you can see, if you get the arguments wrong to a command, the server will reject it and say BAD client!

tag5 LIST "" "*"
* LIST (\HasNoChildren) "/" "Call log"
* LIST (\HasNoChildren) "/" "INBOX"
* LIST (\HasChildren \Noselect) "/" "[Gmail]"
* LIST (\All \HasNoChildren) "/" "[Gmail]/All Mail"
* LIST (\Drafts \HasNoChildren) "/" "[Gmail]/Drafts"
* LIST (\HasNoChildren \Important) "/" "[Gmail]/Important"
* LIST (\HasNoChildren \Sent) "/" "[Gmail]/Sent Mail"
* LIST (\HasNoChildren \Junk) "/" "[Gmail]/Spam"
* LIST (\Flagged \HasNoChildren) "/" "[Gmail]/Starred"
* LIST (\HasNoChildren \Trash) "/" "[Gmail]/Trash"
* LIST (\HasNoChildren) "/" "alabel"
tag5 OK Success

Folder Flags

Here we can see the folder hierarchy that exists in the mailbox. Folders have special flags, indicated in parentheses. Some of these flags are useful for traversing the folder hierarchy, like \HasNoChildren and \HasChilden, while others are used to denote special characteristics of a given folder, like if it contains drafts (\Drafts) or sent messages (\Sent). These flags allow clients to tailor their behaviour to the mailbox and do things like save sent messages to the right folder no matter which language the user’s mailbox is in.

Sessions

OK, let’s read some email now. First, because IMAP connections are stateful, we need to open a session on the folder that we care about using the SELECT command:

tag6 SELECT "INBOX"
* FLAGS (\Answered \Flagged \Draft \Deleted \Seen $NotPhishing $Phishing)
* OK [PERMANENTFLAGS (\Answered \Flagged \Draft \Deleted \Seen $NotPhishing $Phishing \*)] Flags permitted.
* OK [UIDVALIDITY 1] UIDs valid.
* 55 EXISTS
* 0 RECENT
* OK [UIDNEXT 54948] Predicted next UID.
* OK [HIGHESTMODSEQ 11865059]
tag6 OK [READ-WRITE] INBOX selected. (Success)

There’s a lot of information the server sends us here! Sessions are a key component of how IMAP works, and a session lasts from when you SELECT a mailbox until you UNSELECT it, SELECT another mailbox, or log out.

When opening a session, this server has sent us, in addition to the OK indicating success:

a list of active flags in use on messages in the mailbox
a list of supported flags that could be used on messages in the mailbox
the UIDVALIDITY - more on this in a bit
the number of messages in the mailbox
how many “recent” messages are in the mailbox, which means how many messages are tagged with the \Recent flag
the “predicted next UID”
a HIGHESTMODSEQ value - more on this in a bit

A few of these items are going to require a bit more explanation. First, UIDs.

UIDs & Sequence Numbers

A reminder: a long long time ago, there was IMAP2, and in IMAP2 messages were identified by sequence numbers. Sequence numbers were literally just that: monotonically increasing numbers, one for each message in a mailbox:

tag7 SEARCH ALL
* SEARCH 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55
tag7 OK SEARCH completed (Success)

Every time you selected a mailbox, if there were new messages, there would be more sequence numbers. And if you moved messages around and started a new session, the sequence numbers assigned to messages might be different from the ones from the old session.

This very simple scheme worked pretty well if all you did was connect, read your mail online, and disconnect. But it didn’t work if you wanted to connect, download all of your mail locally for offline access, and then reconnect later and have your client sync its local state with the state on the server—because these non-persistent sequence numbers didn’t allow clients to map local mailbox state to remote mailbox state, because messages on each side might have different IDs!

Enter IMAP4 and UIDs. UIDs (unique IDs) are persistent to a message as long as it stays in the same folder and the server’s UIDVALIDITY integer has not increased. So when a client starts a new session on a mailbox, it has to check the UIDVALIDITY and compare it to its local cached value, and if the server’s UIDVALIDITY is higher, it must throw away all cached local UIDs and resync from scratch. But otherwise, it’s free to compare its local cached data to the data on the server and only sync changes.

To retain backwards compatibility, IMAP4 supports both sequence numbers and UIDs, and you can ask for UIDs by prefixing any command that returns message identifiers with UID:

tag8 UID SEARCH ALL
* SEARCH 53809 54180 54185 54211 54222 54268 54410 54451 54452 54453 54463 54527 54557 54595 54596 54602 54608 54617 54637 54639 54643 54664 54667 54670 54673 54699 54707 54738 54753 54754 54804 54806 54847 54852 54857 54864 54873 54877 54880 54881 54883 54888 54895 54899 54902 54903 54904 54905 54906 54916 54923 54926 54931 54942 54947
tag8 OK SEARCH completed (Success)

As you can see, unlike sequence numbers, UIDs are not guaranteed to be increasing in increments of one—gaps develop in the UIDs assigned to a mailbox as messages arrive and are moved around between folders.

Statefulness

While we’re here, one aside about statefulness: Generally, statefulness is not a desirable feature in protocols. Statefulness makes it trickier to use connection pools with IMAP, because the SELECT command can be quite slow, and every time a pool consumer checks out a new connection, it needs to ensure that the correct mailbox is selected. Statefulness also means that application code has to keep track of the state and make sure it’s in sync with the actual state of the connection, or else make another client-server back-and-forth every time the information is needed. Generally, it’s easier to program with stateless protocols than with stateful ones.

Fetching Message Data

OK let’s finally actually download data from a message!

tag13 UID FETCH 53809 (FLAGS BODY[HEADER.FIELDS (DATE FROM)])
* 1 FETCH (UID 53809 FLAGS (\Answered \Seen) BODY[HEADER.FIELDS (DATE FROM)] {80}
From: My Friend <friends4ever@gmail.com>
Date: Sun, 18 Feb 2018 10:50:34 -0500

)
* 55 EXPUNGE
* 54 EXISTS
tag13 OK Success

IMAP provides lots of knobs for allowing clients to request partial data about messages, including the size of a message, flags (like \Seen which means the message has been read, and \Answered which means the message has been replied to), individual headers, and individual MIME parts from within a MIME message, which allows clients to optimize their bandwidth usage by downloading only the data they need. For example, a client can quickly download only basic headers for all the messages in a folder in order to generate a listing of messages for a user to browse through, and then fetch the message bodies on demand.

There are just a couple more concepts we want to cover about IMAP before we wrap up.

Unsolicited responses & IDLE

You may have noticed some strange untagged responses in the previous example:

* 55 EXPUNGE
* 54 EXISTS

This is actually a notification about one message disappearing from the mailbox and another message appearing while we’ve had the mailbox open. It’s valid in IMAP for the server to send these unsolicited messages to a connected client at any time. There’s also a mode you can put an IMAP connection into, called IDLE, which is essentially only for handling notifications about changes to a mailbox, and is often used by clients to quickly sync new messages coming in.

Disconnected syncing

Disconnected syncing allows email apps to work well in the face of airplane wifi and bad cellular connections. It’s not so much a first-class feature of IMAP as a thing that you can build on top of the protocol, with significant effort, because IMAP wasn’t originally designed to support it.

The way to do a disconnected sync with IMAP4 in its original form was pretty simplistic: you’d have to take the list of UIDs you have locally for each folder and compare it to the list of UIDs currently in the folder on the server, and fetch new messages and delete old ones based on the differences. As mailboxes grew, this became more and more complex for clients to handle, and later extensions like CONDSTORE and QRESYNC improved the flow for disconnected resyncs. But even with these extensions, the process for doing a disconnected resync is complicated to implement—and even today many IMAP server implementations don’t support the newer extensions, so general-purpose clients must contain conditional logic for different servers. The general reference for how to implement disconnected syncing on top of IMAP is RFC4549.

Summary

Alright, we’ve done it! Through following along in this post, you’ve learned about the history and major components of IMAP, an open protocol for accessing and manipulating remote mailboxes. We’ve connected to an IMAP server, logged in to our email account, listed folders on the remote server, and downloaded message data. This is exactly what happens under the hood in email apps as you browse your mailbox. We also learned that because of how IMAP developed over the years, disconnected syncing is fairly complicated for clients to implement—which is why we built our delta API to simplify the process.

Next time, we’ll be diving into the other vital open protocol for email: SMTP, which is how emails are sent. We hope you’ll stick around and join in on the fun!

This post originally appeared on the Nylas Engineering blog.

Hi, I’m Christine and I started contributing to Debian when I was 15. Now I’m the CTO of Nylas, ask me anything!

Christine Spang — Wed, 18 Apr 2018 17:44:11 +0000

I started my coding career early, working on the Debian project when I turned 15. Today, I'm the CTO and co-founder of Nylas, a communications API that makes it easy for developers to build integrations with email, calendar and contacts data. To date, we have synced more than 100 terabytes of data from more than 15 billion emails.

My main goal as we scale Nylas's business is to grow a company culture we're proud of - one that is inclusive, diverse and transparent. We re-enforce our values informally every day, and more formally during our regular 6 week all-hands, no-blame retrospectives, during the hiring process, and more.

Email: Getting Back to the Basics

Christine Spang — Wed, 28 Mar 2018 18:31:34 +0000

Email — the tool that’s used by billions of people across the world to build relationships — predates the Internet itself, and its usage and user base continues to grow every day.

So, what has contributed to its ubiquity and widespread adoption?

In its simplest form, email is a global distributed system consisting of many different, interoperable client and server implementations that together allow any user to send a message to any other email user. This ability holds true regardless of which email service the user sends email from, and without each user needing to have a prior relationship with each other. The open nature of the email network means that today there are millions of email servers running around the world, and billions of email users.

Despite it’s ubiquity and foundational place in the business world, it can be difficult for modern productivity, sales and marketing applications to integrate with email. At Nylas, we’ve been building an email API that helps developers build a communication layer into their applications for over 4 years, and we’re excited to start sharing more of our email expertise with the world. This post is the first in our new “Email Experts” series, and will cover some of the basic technologies that make the email ecosystem work.

First up, some basics:

Servers

Email servers are run by service providers like Microsoft, Google, Fastmail, and your local ISP, and live in the public cloud, in business’s data centers, or even people’s homes (for those who want to host their own email). They are intended to operate in an always-on, always-reachable fashion—if they’re down, their users’ email won’t work.

Email servers are the backbone of the email network, and serve two purposes:

(1) Email servers store users’ mailboxes and mediate access to them

(2) Email servers accept outgoing mail and route the messages to other email servers

These are different enough tasks that they are typically handled by different server software—_that is, a mail provider will run _one piece of software to handle storage of users’ mailbox data and access to it, and a separate piece of software to accept outgoing messages and route them to the right place.

Clients

Email clients are the apps we all use to interact with email: mail apps on our phones and laptops, and also server-side software that needs to integrate with email mailboxes, but isn’t specifically a part of the email network itself. The Nylas API platform is an example of an email client that isn’t a user-facing app, as it is server-side software that connects to email servers and provides a simpler interface for folks who are building user-facing apps that work with email.

Protocols

The way that two different pieces of software communicate with each other is by using a standardized interface called a protocol. In the email network, email servers communicate with each other using a protocol called SMTP. All email servers communicate with each other using SMTP, because otherwise they wouldn’t be a part of the same network. But email clients are different. There are standard, open protocols for communication between email clients and servers, like IMAP, and there are also proprietary protocols that are specific to certain service providers. This was able to happen because users get the benefit of the email network regardless of which email client they’re using, because the network interoperability magic happens on the server side. Regardless of what protocol an email client uses to connect to its service provider, that email user is still able to send and receive emails from any other email user, regardless of which clients or client-server protocols the other email user uses. So email providers were free to require a proprietary protocol for clients to connect with their servers, but it was up to that provider to also create and ship mail apps for the platforms they wanted users to be able to connect from—which in practice, means that only the biggest providers attempted such a thing, while everyone else stuck to the open standards so they didn’t have to build their own clients too.

In practice, these days three families of email servers exist that every client developer needs to think about because the protocols they support for client communication are different.

Servers that implement the open standard protocols for email, IMAP and SMTP. Dovecot is a popular open source implementation.
Microsoft’s Exchange family of servers (including Office365, their hosted cloud Exchange service), which use a proprietary family of protocols.
Google’s G Suite / Gmail. Google implements the open standards of IMAP and SMTP for compatibility, but has some significant differences in its mailbox semantics which means client developers have to special case a chunk of the code for interoperating with Gmail accounts regardless of the common underlying protocol.

In the next posts, we’re going to do more of a deep dive into the open standard protocols for communicating between email clients and servers: IMAP and SMTP.

This post was originally published on the Nylas Engineering Blog.

Developer Success Engineering: Teamwork Makes the Dream Work

Christine Spang — Tue, 13 Feb 2018 22:11:52 +0000

We recently hosted a Nylas Developer Event on the topic of Developer Success Engineering. We transcribed the panel so that even if you weren't there, you wouldn't miss out on hearing everything you want to know about Developer Success Engineering from Slack, Segment.io, BugSnag and Nylas.

Panelists:

David McCreath, Head of Developer Support at Slack
Kelly Mason, Customer Engineering Team Lead here at Bugsnag
Maggie Chu, Success Engineer at Segment
Mike Pfister, Head of Solutions Engineering from Nylas

Moderator: Mary Thengvall, Developer Relations @ Nylas

Question 1: There are so many titles for the same role (developer success engineering, technical support, technical account management, solutions engineering). What do you call it at your company?

Mike: So I made my title up for myself -- I don’t know if other companies use it. To give some context on what we actually do: Nylas is currently 22 people, mostly engineers, building an API for email, calendar and contacts. Naming is hard in this general category of work [developer support engineering] - what’s more important is looking at the kind of customers and product you have to determine what your role will be.

Maggie: At Segment.io my title is Success Engineer. I was also a success engineer at my previous company which was a really small company, it was like 30 people. Right now we’re a much bigger team at Segment.io (13 people just for success engineering), so it’s a little more siloed and more focused on support. Now, I deal with code every day whereas at my last company I looked up code maybe once per month. It’s pretty different - same title, but different company and different product.

Kelly: I will echo what Mike said in that I also made up my title as a customer engineer. When I started, it was the founders doing support. Since we’re an error monitoring tool and we’re getting down in the weeds with code, it just made sense. We do support and user education on the success side (when we have time). The title is less important than what you’re doing

David: My team is actually hierarchically under customer support, but we also work with with the cross-functional platform team: DevRel, BD, developer support, product, engineering, and we all work together on the platform which is it’s own thing at Slack. My team came about because we were the agents handling the first 80 integrations we built which were super technical an buggy and required digging into logs. When we launched the platform, we ended . up being the ones that helped debug and QA the platform as it stood and we ended up just giving ourselves titles, so “developer support”.

Question 2: How do you work alongside engineering or other technical departments to make sure that you’re taking care of your customers?

David: Our support tickets are right now divided between customers who are using the integrations and developers who are building the integrations. Most of the time that we spend working cross functionally is with the platform engineering team. For a long time, we handled all of the manual QA that has now been passed off to a different team so we can focus more on the actual support of developers. We work with the product teams (a handful of product teams within platform) on making sure that they understand what the developers are looking for in a specific feature. Before a feature is released we build stuff to make sure it’s working properly.

Kelly: We sit with the customer success team and are very in line with t heir goals, but we’re actually under the engineering umbrella. Part of it is any feedback we’re getting towards career development is shifting more towards engineering. One of the cool things we get to do is build our own admin tools, so we spend a lot more time in the code with the eng team and using our own product to help our customers.

Maggie: We build out our own Zendesk tools which we rely on all the time. We display customer information, how big their account is, who is their account manager. We rely on all of these things to ensure the customer gets proper support as they’re writing in. We use many other integrations like JIRA (communicating with our engineers and product teams).

Mike: Nylas as a whole is 22 people right now. The developer success team is just myself and Lou. We work very closely with all departments at Nylas (sales, product, engineering). We’re the front line of support. Since our business is an API and our customers are mostly developers so it’s very technical. The way we assist sales is we hop on a sales call and take on what might be called a sales engineering role, giving people technical advice on how they can integrate with our API, best practices, get them onboarded quickly. We help inform product decisions because we’re communicating with customers so frequently that we h ave great insight into what they want in terms of feature requests. With engineering, we’ll escalate any bugs that we’re seeing to them. We do daily standups with the engineering team that let us surface any customer issues on a daily basis.

Question 3: How do you protect your engineering team from getting bombared with customer support tickets? How do you balance that with their day to day engineering work?

Mike: It’s important that the developer success team protect engineering time. The engineers have their own long-term projects they’re working on, and it’s difficult for them to make progress when they’re getting interrupted with customer tickets on a daily basis context switching to fix bugs. To protect our engineering team, it’s about building trust. Every time we escalate something to them that we shouldn’t’ have escalated, it’s going to be hard the next time you want them to look at a ticket. It’s our job to prioritize and take the birds eye view from all the tickets we’re looking at and surface the most important ones. The things we look at to protect engineer’s time are:

How many accounts are affected by this issue?
Are there SLAs in place with the customer that we need to honor?
Is this customer part of a deal we’re trying to close?

Maggie: For our company, we rely heavily on internal docs. So when we’re talking to our engineers, and they tell us “This is the reason why you’re seeing this error message”, we document that. We rely so heavily on this process and it’s not just our team looking at it, it’s teams from sales, customer success managers, and marketing. We all encounter the same questions across departments. We use Dropbox paper to maintain these docs.

Kelly: Give that we support pretty much every language and framework and I have only a few years of engineering experience, it’s quite often that I don’t now when I first start a problem if I can figure it out in an hour or if by the end of the day i’ll know anything more. Myself and my colleague (we’re also a two person team) run through tings in the morning and time box the ones that are configuration heavy or deep in the weeds of .net (something i just won’t know) and usually look at it for a solid 2 h ours then reassess whether to escalate or not. One person from our two engineering teams runs through everything we couldn’t resolve in the afternoon.

David: We have a similar process. Each of our engineering groups has what’s called a triage channel in Slack where support people (whether it’s us giving developer support or the customer team doing customer support) can drop in and say “I’ve got this problem and none of us can figure it out.” The engineers work in shifts on those in various departments. In platform, there’s two people and all they do is triage. But we try to protect them by understanding as much as we can about the APIs.

Question 4: How do you get metrics around what features need to be added to the product?

David: For slack, the support is part of the product. That’s the pitch we’ve made since the product came out. If you’re paying for Slack, you’re paying for support. Over the last years, we’ve rolled out an enterprise product and scaled up some of that. Now, as a team we’re figuring out how we move from 1:1 email support to broader support for the enterprise. How do we look at a 1 to many instead of 1:1 solution?

Mike: At Nylas, we have a basic support SLA that all of our customers are on automatically for free. Being as mall startup, every team is very constrained in the resources we have. Obviously we would love to have 30 minute resolution times for every ticket that comes in but that’s just not possible. So we have other tiers of SLAs that people will pay for. Generally for newer customers, we’ll priotize their ticketes as well to help them get onboarded quickly.

Maggie: At Segment we have different pricing tiers, all the way from enterprise to a free developer plan. So if you’re on the the team plan or enterprise plan, it’s a faster response time and your issues get escalated faster. Enterprise plans have a dedicated support or customer success manager.

Kelly: We rely a lot on our salesforce integration with our Zendesk. So if something is on an enterprise plan, it’ll get moved up int eh queue. But a lot of times it’s more informal in that I sit close to the sales team and the customer success team. I thought it was kind of goofy to sit with the customer success team at first, but it’s actually a huge benefit to be able to reach over to someone who has context and can help resolve something quickly.

Question 5: How do you get feature requests funneled to the right team/how do you speak on behalf of your customers?**

Kelly: We’re really interested in getting that customer feedback. It’s something we collect a lot because we value that feedback. Error monitoring is kind of new as an idea - so grouping the way you might approach a problem of fixing the bugs in your application is so unique to every customer that we feel it’s best when we h ave a critical mass of feedback and can connect closely with our engineers who are building it. Once we see critical mass, we’ll start building something.

Maggie: For any major feature request we have, we put them into JIRA reports. We have an integration with Zendesk where we can integrate Zendesk with JIRA reports. We also put the customer profile into the ticket so if they’re a big or small customer, we document everything so the product team can review it, do research, and decide which sprint to put the feedback into.

Question 6: How do you help your customers support themselves so you aren’t drowning in support tickets?**

Mike: Creating a lot of content. The most effective is a knowledge base article. i use a tool called Loom that helps you make a video and share the link with someone. We also host office hours on a weekly basis. There are 2-3 hours of weekly slots that are open, and our customers can call in and chat with a n engineer (which is basically me) every week. They love to talk about everything from their integration and how ti’s working well, to diving into a specific issue. If they don’t have an SLA, a lot of customers will use this t o get roe face time.

Maggie: Yeah, we have our public facing documents and we’re always trying to make them really beefy. Whenever a customer writes in and asks a question, they look at the docs. There’s a person on our team that focuses just on docs.

Question 7: What is the hardest part of developer success?

Maggie: A lot of our customers write in expecting us to know everything - not just form your own product, but all the integration, all of your partners, questions that you can find on StackOverflow. They expect you to be an expert in everything technical, and obviously no one is, so we have to do a lot of our own research. The hardest part is to know when to push back and say “hey, this is something you might need to research on your own, it’s not in our scope.”

Mike: The hardest part for me was being the only support person for a long time. It’s a pretty emotional job when you’re trying to make customers happy and so not having a team member like Lou was really tough. I’m super excited because we just hired another DSE and I’m continuing to hire for more technical roles for our team, and so those days are over.

David: I think for us one of the trickiest things was the speed with which we grew as a platform suddenly became very difficult to keep up with. Even though we were the ones responsible for QA, things were slipping by us and getting released without QA happening and we had to put out fires.

Kelly: I would say ours is the number of ways we get a support request. We do get people writing customer support, talking to their success manager, but we also have a very large number of open source repositories so you can always great a GitHub issue there. And that’s great because once you’ve solved something you can reference it there and everyone can look at past GitHub issues, but it also is an interesting overlap between customer engineering and out platforms team that makes those libraries. Making sure the process holds up if they’re not overwhelmed with GitHub issues and knowing where to prioritize those within the rest of our customer support issues.

Question 8: What’s your favorite part about your job?

Kelly: I think it’s pretty cool that you very rarely know what the answer to someone's question is before I start it. The amount I've learned just form our customer’s questions and about technologies i would’ve never thought to look up on my own.

Maggie: We have a very international customer base, so it’s interesting to have customers form all over the world write in from different backgrounds and languages. They may ask the question very differently.

David: For me, this role feels more like the old internet when you could just help somebody build their webpage and boom [they're just so excited]!

Mike: The kinds of people I work with are CTOs and tech leads, so my role is relationship based. It’s cool to establish that and work with someone through their entire integration. I like when I’m able to help those people and keep them happy and being able to automate processes in a way that finds elegant ways of not doing those small repeatable processes over and over again is really satisfying.

Question 9: What’s the one thing you wish someone had told you before you took on this role?

Kelly: I think the #1 thing would be everything there is to know about error handling. But given that that’s unlikely, maybe just the fact that understanding the question and figuring out what the user is getting at in their question and which part of their tech they think is going wrong. That’s the first step. Once . you can sift through that everything is easier to find.

Maggie: Knowing what the customer is actually asking. It could be so different than what you thought. I’d debug what i thought they were saying for 20 mins, and they’d come back and say “that’s not what I was asking” and I'd have to revisit the question. Always reconfirm what they’re truly asking for.

Mike: Back to my days on the graveyard shift (when I was doing support by myself), feeling the need that i Have to solve every issue as fast as possible. I wish i would’ve learned earlier that you can say no to customer request and be honest with people about expectations.

David: I think honesty is big part and not being afraid to tell the developer when “Yes there’s a problem with the API and it’s not going to be fixed today, but we’re working on it.” And then they at least know and can build around it.

Kelly: It’s kind of nice because most developers have been there before. They’re like “That’s a lot of work, I’m not going to build that”.

Question 10: Once you’ve dug into the code and know you can open a pull request for it, do you actually submit the PR or do you bring the problem to engineering?

Mike: At Nylas, yes as a developer success engineer you can get a customer support request through Zendesk, dig through Kibana, dive into logs using Honeycomb.io to figure out why things are broken, writing code, submitting to Phabricator, then actually deploying it. We have a command we’ve written at Nylas (CLI command) so you can take it from start to finish and get back to the customer and say it’s fixed. The types . of issues we fix in that way are smaller scoped. For big infra changes, we’d escalate to engineering.

David: I have commit privileges but I try not to use them. There’s enough engineers now that I’ll get it through to the PR then pass it off to the engineer to review and either commit it myself or let them do it.

Kelly: A lot of it is feeling out where that line is. Often we find that if something isn’t clear, we can make an example better. My co-worker updates a lot of our open source examples of how you might configure different parts of Bugsnag - that’s more common (random little QRs that are helpful but not specific to what our engineers are working on day to day).

Question 11: Do you have tips on stepping away from day to day work?

Maggie: For me it depend son how many people are on my ass about tickets. If i have people from many different teams saying i need to push something through right now, I get nervous about it. If it doesn’t need to be responded to asap, sometimes it can wait till tomorrow. You get the sense of urgency from the customer.

David: Letting people know you have not forgotten about them goes a long way.

Question 12: What key traits do you look for when interviewing for this role? Someone more technical or with a stronger support background?

Kelly: We look for people that are okay with being thrown in the deep-end of code. You must be comfortable starting with little information and learning as you go. Having technical background and being able to communicate well with customers is a unique trait and it’s hard to find people that can do both.

David: One of the things we screen for when hiring developer support or customer support is stamina. The people on the developer support team right now have all worked as engineers in the past and decided to move on to something else. The mindset you have to go from being heads down in your code all day to switching between other . people’s problems throughout the day is significant, so we look for people who are genuinely interested in being helpful.

Mike: We’re hiring a developer success engineering right now. They’re 60-70% communication on email, phone, etc. The other 30-40% is diving into issues. We test for their communication skills and also their technical ability - we have a little coding challenge the run through. Prior coding experience is definitely something we look for. The partner engineer role is the flip side: 20-30% communication, more time spent diving into bugs.

Maggie: For us it’s pretty similar. Having a technical background is a good foundation, but we really stress great communication skills both verbally and in writing.

Question 13: What kind of tooling do you invest in?

David: We have a pretty basic setup of Zendesk. It’s tied into Kibana and other instances. I think the code base of Slack is so unusual that we built most of our own tools around it and that’s stabilizing a little as we bring in more commercial stuff.

Kelly: We have support ticketing, and we use BugSnag to find people who have run into an error on our site. It’s very meta, but it’s really helpful when someone’s integration isn’t working. Normally that would be my least favorite thing to try to debug, but we send a custom event every time someone’s integration has failed, and I'll use Bugsnag to see if it was an incorrect password, or something bigger like the site being down. It’s funny using the tool they’re complaining about to fix the problem.

Mike: Internally we use a web app. Some of our accounts listing page - since we’re a fast growing startup, that page loads slower each day. As we hire more DSE’s, we’ll have more time to work on higher impact projects where we build out tools to support ourselves. A really awesome tool I like is called Honeycomb.io. It’s a great way to get structured log data and break down based on several fields. You can pinpoint why a particular thing is happening extremely quickly.

Question 14: How do you prevent an investigation on a ticket from becoming a rabbit hole?

David: I grab somebody else and get their eyes on it. If I realize i’m going down a rabbit hole, I’ve gone too far.

Maggie: I give myself maybe 30-40 minutes and if i haven’t figured it out, iIll ask somebody else. Sometimes I’ll ask the customer to repeat or rephrease the question or update them once I have more data,

Kelly: It’s a good point of getting more data. In that time, I’m hoping if I don’t know the answer I at least have the next question and can ask that to the user or ask the right team questions so that when I escalate it they’ve gotten a little further and I’ve saved them some time.

Mike: I live in rabbit holes on a daily basis. Working in email - that’s the value our business is providing. We abstract away all of the rabbit holes developers would have to deal with building out an email integration. I do the best that I can with the technical knowledge I have. I’ve spent a long time debugging Exchange ActiveSync issues, but at the same time I have to communicate with customers. I can’t go radio silence when I’m debugging something, so I try to document as much as possible what I find so that when I escalate the problem to engineering, they’re set up for success.

Nylas is Hiring a Partner Engineer

Christine Spang — Thu, 08 Feb 2018 01:36:25 +0000

About Us

The Nylas Cloud API makes it an order of magnitude easier for companies to add email, calendar, and contacts integrations to their applications. By being at the core of business communication, scheduling, and contacts, Nylas is shaping the future of how people work.

Our team is roughly equal by identified gender and we actively and regularly work with the entire team to shape our culture to our ideal of empowerment, transparency, and kindness.

We collaboratively wrote about our values, benefits, perks, and published them in our open-source handbook. Read it to learn more about Nylas. You can also find more details at: https://www.keyvalues.com/nylas.

About The Opportunity

We’re looking to hire our first partner engineer at Nylas. You’ll be responsible for the technical success of our customers by helping them integrate and by digging into technical issues and bugs within in the Nylas API. You’ll work directly with both our partners and Nylas’ customer success team to investigate bugs within these systems and then fix them.

Our open-source Python sync engine regularly archives terabytes of data across a massive SQL cluster, and our Flask APIs handle tens of millions of requests a day. It’s your job to quickly identify, fix, and prevent issues from happening within this architecture. You’ll have a direct impact on the reliability and scalability of our system by improving our testing infrastructure and building mechanisms to quickly identify issues before our customers do. You’ll have a direct impact on the happiness of our customers because you’ll be fixing the issues that matter to them most.

You should have a growth mindset, a track record of managing your own projects, and strong autonomy in every day work. You should also tend toward humbleness in your abilities and have an innate desire to pass knowledge onto others.

How To Apply

If this job piques your interest, don't hesitate to apply today!

Please feel free to leave questions in the comments section.

Nylas is Hiring a Developer Success Engineer

Christine Spang — Thu, 08 Feb 2018 01:30:39 +0000

About Us

Our team is roughly equal by identified gender and we actively and regularly work with the entire team to shape our culture to our ideal of empowerment, transparency, and kindness.

About The Opportunity

Nylas is looking for an intelligent, ambitious and energetic individual to join our API Developer Success team where you'll be directly responsible for ensuring the success of our partners and developers. Hundreds of businesses rely on the Nylas Cloud APIs to power their products, and your role will be critical in ensuring their sustained growth and success. You'll work with startup CEOs and top technical talent within our industry to help them build awesome applications using Nylas Cloud APIs.

This includes working alongside Nylas Engineers to help triage issues that impact reliability and speed of our product. You’ll help prioritize and recognize trends in customer issues and implement processes and automations that fix them. Your deep technical understanding of our infrastructure will allow you to onboard new customers onto our platform and help existing ones quickly.

Responsibilities

Help customers solve technical problems and answer questions about product capabilities
Lead customer conversations as a technical expert, where your knowledge and expertise of the Nylas Cloud APIs will allow you to guide implementation and onboard customers.
Manage customer support tickets and provide regular updates to open tickets
Develop customer-facing documentation and integration guides
Collaborate closely with our Development Team (provide feedback, file and fix bugs), Business Team (respond quickly to high priority issues), and Product Team (provide roadmap feedback)
Develop technical messaging and platform architecture documentation describing the Nylas Cloud APIs

What We're Looking For

You have experience communicating with users and providing product support over email, on the phone, and in person
You have experience working on technical issues, troubleshooting bugs, and communicating with engineers
You have a desire to work in a fast paced environment with lots of autonomy
You work just as well finding hidden details as you do summarizing broader trends
You have great verbal and written communication skills
You have deep empathy towards technical users and you thrive on making customers feel empowered
You’re curious and comfortable with new technologies and you love learning new tools.

How to Apply

If this job piques your interest, don't hesitate to apply today!

Please feel free to leave questions in the comments section.

Nylas is Hiring a Developer Success Engineer

Christine Spang — Thu, 08 Feb 2018 01:29:42 +0000

About Us

Our team is roughly equal by identified gender and we actively and regularly work with the entire team to shape our culture to our ideal of empowerment, transparency, and kindness.

About The Opportunity

Responsibilities

Help customers solve technical problems and answer questions about product capabilities
Lead customer conversations as a technical expert, where your knowledge and expertise of the Nylas Cloud APIs will allow you to guide implementation and onboard customers.
Manage customer support tickets and provide regular updates to open tickets
Develop customer-facing documentation and integration guides
Collaborate closely with our Development Team (provide feedback, file and fix bugs), Business Team (respond quickly to high priority issues), and Product Team (provide roadmap feedback)
Develop technical messaging and platform architecture documentation describing the Nylas Cloud APIs

What We're Looking For

You have experience communicating with users and providing product support over email, on the phone, and in person
You have experience working on technical issues, troubleshooting bugs, and communicating with engineers
You have a desire to work in a fast paced environment with lots of autonomy
You work just as well finding hidden details as you do summarizing broader trends
You have great verbal and written communication skills
You have deep empathy towards technical users and you thrive on making customers feel empowered
You’re curious and comfortable with new technologies and you love learning new tools.

Please feel free to leave questions in the comments section.

Nylas is Hiring a Senior Software Engineer

Christine Spang — Wed, 22 Nov 2017 19:30:16 +0000

About Us

Our team is roughly equal by identified gender (including engineering) and includes startup founders, Debian contributors, alums, MIT, Columbia, Yale, CMU, and Olin alums, and a professional dancer from the San Francisco Ballet. We actively and regularly work with the entire team to shape our culture to our ideal of empowerment, transparency, and kindness.

About The Opportunity

We’re hiring senior engineers to help us scale and build new products on top of our APIs. Right now, our open-source Python sync engine regularly archives terabytes of data across a massive SQL cluster, and our Flask APIs handle tens of millions of requests a day. We aim to scale that several times over in the next year.

Our senior engineers end-to-end own and ship new features, like offering Contact sync, while providing mentorship through pairing. They shard our webhook systems, then re-architect in Kafka as we scale. They solve tricky customer issues, debug slow MySQL queries, and sometimes get into the depths of MIME and other email protocols. On the side, they improve our internal developer experience, most recently with MyPy, and open-source our custom-built tools. They talk at conferences, write blog posts, and make strong showings at local events.

While our main headquarters is in the heart of San Francisco, we have recently opened an office in the Financial District of NYC. We are open to conversations about remote employees in other locations as well.

Skill & Qualifications

You should have a growth mindset, enough engineering experience to architect complex systems, a track record of managing your own projects, and a strong sense of practical shippability over engineering purity. You should also tend toward humbleness in your abilities and have an innate desire to pass knowledge onto others.

Our team collaboratively wrote about our values, benefits, perks, and published them in our open-source handbook. Read it to learn more about Nylas. You can also find more details at: https://www.keyvalues.com/nylas

How To Apply

If this job piques your interest, don't hesitate to apply today!

Please feel free to leave questions in the comments section.

Billions of Emails Synced with Python

Christine Spang — Wed, 15 Nov 2017 23:08:04 +0000

Developing an email sync engine is hard. Python makes it it easy (well, easier).

At Nylas, our love for all things Python is no secret. Our email sync engine – responsible for syncing and serving more than 10 billion emails for customers – is built entirely in Python.

Why? The answer is simple: Python is standard, reliable and (best of all for a startup like ours) boring. It’s a no-frills solution enabling us to provide a flexible and stable API for developers around the globe.

At PyBay 2017, I gave a talk why Python’s “boringness is inherently advantageous to work with. For companies building complex products, Python-driven solutions produce the most stable foundation for innovation.

Email is hard

Email protocols are notoriously fickle, requiring extensive documentation just to implement simple syncing to a SaaS app with one protocol. If a developer wants their email implementation to work with a wide variety of email standards, they have a lot of issues to consider (for example, getting Exchange, Exchange ActiveSync, and Gmail IMAP protocols to play nice together) in addition to parsing, encoding, authentication and more. The list of potential problems is extensive and there is almost never a simple solution.

These complications are simply the inevitable baggage that accompany a 50-year-old technology like email. Standards change, new and complicated technologies are introduced, but legacy protocols must always be considered.

The Challenges

When Nylas first got its start, we had to consider how we’d build a sync engine API. There were, essentially, two options: 1) Store minimal data and act as a translation layer between two platforms or 2) Mirror the contents of mailboxes and serve as many requests as we could directly.

The first option would be cheaper, but it would complicate our ability to deliver reliability to organizations that depend on uptime. So, we went with option two, even though it was more expensive. We did this by sharding databases and using a semi-monolithic architecture. This architecture positions our services across different fleets of machines, but they all share the same underlying code and models for easier integration.

This architecture positioned us to better tackle bigger challenges, like threading emails, syncing tags and folders and more. The end result: the Nylas Sync Engine. The Nylas Sync Engine is open source and provides a RESTful API that enables developers to integrate messaging into their applications. It has more than 90,000 lines of Python code, including tests and migrations, and helps to wrangle an ecosystem of protocols, protocol offshoots, parsing, encoding and much, much more.

In a Tech Stack, Boring is Good

To build out our architecture, we had to use an extensible programming language like Python and its many libraries. For example, we use flanker, a parsing library, extensively to help facilitate email deliverability. We also use Flask, Gevent, SQLAlchemy, and pytest in addition to other tools like HAproxy, nginx, gunicorn, MySQL, ProxySQL, Ansible, Redis and more. As far as stacks go, it appears as if we made some fairly bland choices. We did, and that is by design.

We chose to run with boring tools for a simple reason: we’re a small company and we can dedicate only so many resources towards driving innovative developments. In fact, part of our design philosophy is descended from Dan McKinley’s essay, “Choose Boring Technology” (if you haven’t given it a read, you should – it’s great), where he suggests each company has limited capacity for innovation before exhausting itself out of business. This concept drove our early decisions at Nylas and informs our central API philosophy, which is to enable our clients to build one integration instead of many. This pushed us towards battle-hardened technologies we know, we don’t have to worry about and which enabled us to do more elsewhere.

For example, we use MySQL to manage our database. We decided MySQL was the right choice early on because we knew it well, a great deal of potential DBAs knew it well and we needed to save that innovation energy for other issues. To be sure, relying on MySQL meant we had a lot of growing pains (see our blog post on Growing up with MySQL), but we could overcome those issues with the smart application of ProxySQL, horizontal sharding and other techniques.

Knowing our strengths in MySQL, and how we could later modify it with ProxySQL and other helped to simplify our database into smaller, more manageable pieces while enjoying almost no downtime and without having to completely redesign our database. In fact, we use MySQL to easily record and replay all the changes to a mailbox sync on our transaction tables through a little bit of magic provided by ProxySQL and SQLAlchemy. This powers how we sync, our webhooks, our streaming API and more.

With so many dependencies, we use a Python virtualenv wrapped up in a Debian package manage to deploy our application. Specifically, the tool we use is called dh-virtualenv and it enables us to use Debian’s package manager, dpkg, to deploy dependencies and vet artifacts before pushing them to Amazon S3. As McQueen notes, a simple deploy script winds up looking like this:

scp my-package.deb remote-host.example.org:
ssh remote-host.example.org

# Run the next commands on remote-host.example.org
dpkg -i my-package.deb

/usr/share/python/myproject/bin/python
>>> import myproject # it works!

With these tools in place, we enjoy a reliable tech stack that doesn’t set out to reinvent the wheel. It simply helps to sync and deliver email while meeting everyone’s expectations on uptime, scale and stability.

Syncing and Checking

We use Python on multicore machines across our fleets. This means we have to run multiple processes in order to get the most out of those multiple cores. For this, we use gevent, a coroutine library, to sync about 100 accounts on a single process. This saves us a lot on memory and OS scheduling. Here’s what a process looks like:

Each green box is a gevent greenlet. Greenlet is a powerful little Python tool that allow us to make minute control flows, such as having a greenlet managing a Gmail sync flow down to greenlets managing trash, calendar, contact syncs and more. To monitor all this, we use a special greenlet running across all flows that sends an event log if it doesn’t run in a determined amount of time. When this happens, we run a sampling profiler that samples and records the application call stack to identify where the application is getting hung up for analysis. In Python, a sampler will look a bit like this:

import collections
import signal

class Sampler(object):
   def __init__(self, interval=0.001):
     self.stack_counts = collections.defaultdict(int)
     self.interval = 0.001

    def _sample(self, signum, frame):
     stack = []
     while frame is not None:
         formatted_frame = '{}({})'.format(frame.f_code.co_name,
                                             frame.f_globals.get('__name__'))
         stack.append(formatted_frame)
         frame = frame.f_back

     formatted_stack = ';'.join(reversed(stack))
     self.stack_counts[formatted_stack] += 1
     signal.setitimer(signal.ITIMER_VIRTUAL, self.interval, 0)

    def start(self):
     signal.signal(signal.VTALRM, self._sample)
     signal.setitimer(signal.ITIMER_VIRTUAL, self.interval, 0)

These samples can be fed into a flamegraph to show us what a process is doing, how the CPU is being used and what greenlet is holding it up.

What We’re Working on Next

But as any organization knows, the fun doesn’t stop at overcoming challenges. Over the course of the next few years, we plan to invest more in the Python ecosystem. For example, we’re looking towards mypy to help alleviate management as our code grows in complexity. In particular, we’re using mypy to do type checking as a linter. It’s an incremental project, but we’re excited to have the complexity-reducing powers of mypy onboard.

We’re also going to start migrating towards Python 3. Finally, we’re looking at moving our transaction log to a Kafka event backbone. This should enable us to move towards a microservices-based architecture, giving us more flexibility by not requiring every service to talk directly to the database.

For us, Python is in nearly everything we do because of its simplicity, its diversity of libraries and its ability to work well with servers. For curious developers, there is a lot more to learn about how Python can simplify code, or form the foundation of a new application. And, thanks to its robust and productive community, learning how to execute on those possibilities can be fairly simple. That community is one of the main reasons why we use Python so extensively and why we’re looking forward to seeing what else that community can do and how we can contribute to it as well. For us, Python is standard, extensible, time-tested and, best of all for any startup – boring. We can’t wait to see where it goes from here.

Watch my full presentation here:

This article was originally published on the Nylas Engineering blog.