Forem: Troy Goode

Courier Raised a $35M Series B to Make Notifications Delightful

Troy Goode — Wed, 22 Jun 2022 16:54:57 +0000

We’re excited to announce that we’ve raised $35M to provide engineering teams with fantastic notification infrastructure! The round was led by GV with participation from our existing investors at Bessemer Venture Partners and Matrix Partners.

“Courier is building the future of notification infrastructure. The company has a fast-growing set of customers with a need to provide the best possible notification experience. GV is thrilled to lead Courier's Series B and partner with Troy Goode and the entire executive team as they deliver on an unparalleled approach to software notifications." - Sangen Zeb, GV

Courier’s vision is to be a single source of truth of all of the end-user’s notifications. We aim to bring together all notifications, whether they may be from mobile, email, Slack, or any other channel, to a single point of easy access to lift the burden of improving the notification experience from the end-user.

The next incremental step on this road is to begin to improve mobile notifications. According to a Courier survey of 157 notification recipients, respondents are more likely to engage with a smartphone notification than a text, email, push, or in-app notification. Our first goal for our Series B, therefore, is to improve the user experience of mobile notifications and we are starting down this path today by announcing new advanced mobile notification capabilities for our platform. Check out the video below to learn more about our new mobile offerings!

A new category of software: Notification Infrastructure

Every piece of software in the world sends notifications, and just about every software team has had to build the infrastructure needed to deliver those notifications in-house. This ubiquitous need for notification infrastructure demands its own category of software, which gives developers the option to use a third party API instead of building an entirely new notification system from scratch.

As a former engineer and engineering executive, I’ve experienced the pain involved with building and scaling notification infrastructure too many times. Every time I was tasked with building a new notification system for a new product, I wished there was a service I could reach for to free up my engineering team and avoid the future headache that I knew would follow.

The lack of this type of solution is what drove me to found Courier.

Courier delivers notification infrastructure to our customers as SaaS. At the center of our product is an API designed for engineering teams. This API allows developers to send notifications across any channel as well as manage user data, preferences, notification workflows, and more. In addition to the API, we provide a web Studio that allows our customers to easily design omni-channel notifications, view logs across notification channels, and providers, as well as visually build out notification workflows.

We’ve come a long way over the last three years, both as a company and a product, thanks in large part to our customers. Courier is used by hundreds of fantastic software companies such as New Relic, LaunchDarkly, Lattice, and Uservoice. These companies have partnered with Courier to deliver a better product experience to their users while dramatically reducing the amount of time and money spent on building and maintaining their own internal notification infrastructure.

“Before Courier, we were spending tons of time integrating with every delivery platform every time a customer requested a new connection from us. But now it’s as easy as making an API call. So, so excited about this launch. Huge congratulations to the Courier team!” - Eric Koslow, CEO, Lattice

What’s next: notifications that delight, not distract

Product notifications are one of the rare parts of software that affect nearly everyone on earth. If you use the internet for work, to buy flights, to network, or to do just about anything else, you receive notifications from that software.

At their worst, product notifications are annoying, distracting, and anxiety inducing. They can come through different channels from text to email to mobile notifications - or from all of them at the same time, acting as a constant disrupter in the user’s life. At their best, product notifications deliver a delightful experience and make sure that we’re aware of key events that might help catch a flight, meet a deadline at work, or connect with a loved one. In other words, they provide useful interruptions that improve the user’s life.

We will use this new capital to continue to invest in our core mission of making all product notifications delightful, starting now with mobile. What that means today is delivering an amazing toolset to the builders, product and engineering teams, who are in charge of developing product notification systems for their apps. We believe that just about everyone in these roles wants to deliver great notifications with excellent user experience, and that the better the tools and resources at their disposal are, the more able they will be to accomplish this.

Introducing Courier’s new mobile capabilities

Today, we’re announcing a new set of two tools and resources for product builders to deliver delightful notifications for their mobile applications. Mobile notifications are the most ubiquitous type of notifications and the one with most potential to annoy users. One annoying notification experience can result in a user turning off notifications altogether, which greatly reduces both the utility and engagement of any mobile application.

The first is our new mobile SDK. This SDK makes it easier for developers to manage push tokens across devices and operating systems. This gives developers better control and visibility, making it easy for them to deliver a more consistent and customized experience for their users based on the best way to reach and engage their users for their specific app. Here’s an example:

// Register multiple devices for a given user. Notifications will be delivered to each 
// device in a single send call.

{
  "tokens": [
    {
      "token": "abc123",
      "provider_key": "apn",
      "properties": {
        "test": true
      },
      "device": {
        "app_id": "com.my.app",
        "platform": "iOS",
        "model": "iPhone 13"
      }
    },
    {
      "token": "cba321",
      "provider_key": "firebase-fcm",
      "tracking": {
        "ip": "127.0.0.1",
        "lat": "0.0",
        "long": "123"
      }
    }
  ]
}

The second is our new mobile notification inbox. This inbox can be easily integrated with any mobile application, and gives the user a central place to view notifications, even if they have disabled push notifications. This gives the end user the ability to better customize their experience using an app by deciding when to access notifications, thereby reducing the number of interruptions mobile notifications cause in their daily lives while increasing engagement with the notifications developers deem necessary to use the product properly.

Visit this page to learn more about this new mobile functionality and request a demo and checkout this video of me demoing them below.

Conclusion: The work is just beginning

Our mission to make notifications delightful is a big one and there’s still a ton of work to do. We could not have made it to where we are today without all of our fantastic customers, New Relic, LaunchDarkly, Lattice, Color Health, and many others who share our belief in the value of delivering a great product notification experience. This new funding presents a great opportunity to achieve this ultimate goal. Get a demo here to learn more about Courier’s goals for mobile notifications!

If you are interested in helping to build the future of notification technology starting with mobile channels, we are hiring!

Introducing Courier Elemental

Troy Goode — Thu, 24 Feb 2022 16:54:05 +0000

Marking up notification emails with HTML out of the 1990s has always been an annoyance, but now notifications need to span across additional channels such as SMS, Slack, Microsoft Teams, mobile push, web push, etc. They also need to accommodate more dynamic use cases like magic login links, multi-language notifications, and location based alerts that have caused the content formatting challenge to become far more complicated and cumbersome.

For example, if you need to trigger a notification that includes a call-to-action for email, SMS, and Slack, the CTA needs to be an HTML button in email, a plain text link in SMS, and a Slack Block element. This task typically is approached by either creating three separate templates or coding up some if/then statements, both of which get very cumbersome very quickly.

That is why we built Courier Elemental, an omni-channel markup language for notifications. Elemental provides a powerful JSON-based syntax to describe the content of your notification for email, push, chat, or any channel you use to notify users (check out the docs). With Courier Elemental, you can customize the look, language, and structure of your notification based on locale, channel, and other more advanced custom logic. Elemental not only makes the tedious parts of notification development simple, but it also enables more dynamic, engaging notification experiences that feel like a natural exetension of the core product experience. Let’s take a look at some of the notifications that Courier Elemental enables for the Bat Signal app.

Examples

A simple email notification

{
  "message":{
    "to":{
      "email":"bruce.wayne@gmail.com"
    },
    "content":{
      "title":"Password Reset!",
      "body":"Hi Bruce, here is your temporary password: {{new_password}}"
    },
    "data":{
      "new_password":"hf73*sh1!hfshjsk"
    },
    "routing":{
      "method":"single",
      "channels":[
        "email"
      ]
    }
  }
}

A multi-channel notification with image and action

{
    "message": {
        "to": {
            "email": "bruce.wayne@gmail.com"
        },
        "content": {
            "elements": [
                {
                    "type": "meta",
                    "title": "Citizen in Need"
                },
                {
                    "type": "text",
                    "content": "Hey Bruce, {{citizen_name}} is in need of assistence!"
                },
                {
                    "type": "action",
                    "style": "button",
                    "content": "Directions",
                    "href": "{{directions_to_incident}}"
                },
                {
                    "type": "image",
                    "src": "{{citizen_image}}",
                    "href": "{{citizen_profile}}",
                    "align": "center",
                    "altText": "{{citizen_name}} Image"
                }
            ]
        },
        "data": {
            "citizen_name": "Distressed Damsel",
            "citizen_profile": "www.bat-signal.com/profile/distressed-damsel",
            "citizen_image": "www.bat-signal.com/profile/distressed-damsel/profile.png",
            "directions_to_incident": "https://www.google.com/maps/dir/ghct"
        },
        "routing": {
            "method": "all",
            "channels": [
                "email",
                "sms",
                "slack",
                "push"
            ]
        }
    }
}

An email notification with conditional content

{
  "message":{
    "to":{
      "email":"{{citizen_email}}"
    },
    "content":{
      "title":"Rescue Feedback Survey",
      "elements":[
        {
          "type":"text",
          "content":"Hi {{citizen_name}}, we're so happy that your rescue was successful! 
          Please complete this survey and tell us how we did.",
          "if":"data.rescue_success === true"
        },
        {
          "type":"text",
          "content":"Hi {{citizen_name}}, we're so sorry that your Batman was not able to provide you 
          with a successful rescue. Please complete this survey and tell us how we did.",
          "if":"data.rescue_success === false"
        },
        {
          "type":"action",
          "style":"link",
          "content":"Complete Survey",
          "href":"{{survey_link}}"
        }
      ]
    },
    "data":{
      "citizen_name":"Distressed Damsel",
      "citizen_email":"distressed.damsel@gmail.com",
      "survey_link":"https://bit.ly/3BFUief"
    },
    "routing":{
      "method":"single",
      "channels":[
        "email"
      ]
    }
  }
}

We've also put together an an example app on GitHub called the Courier Alert Center to show what notification built with Elemental look like in a real code base.

What’s next?

This is just the beginning for Courier Elemental. In the coming months, we will be releasing new elements as well as new API functionality to open up even more developer use cases. One exciting use case that Courier Elemental will be enabling soon is the ability for our customers to embed a custom notification designer in their product for their end users. Imagine a teaching platform where teachers want to create notification for students or a CRM for small businesses that wants to enable business owners to notify their customers. This is just one of many new notification use cases that will be coming to Courier in 2022.

If you want to get started building with Courier Elemental today, you only need to do two things.

You need a Courier API key which you can get for free by signing up here (every developer gets 10,000 free sends per month)
Once you have an account you need to configure one of the many notification providers we integrate with.

If you’d like to attend a more in-depth technical demo and ask any questions, please register for Developing With Elemental and join us next week!

How to Add a Direct Twilio SMS Integration With Sendgrid Emails

Troy Goode — Tue, 01 Feb 2022 18:10:06 +0000

If you have been sending emails to your users or customers, adding a new notification channel like SMS can help reach users in the ways they prefer as well as increase engagement with your site or application. At Courier, our focus is to make it simple to add new channels, but we also realize that sometimes a direct integration may be necessary. In this white paper, we share the strategies we have discovered through our work in building these types of direct integrations as we explain how to add SMS with Twilio to an existing SendGrid integration. We will help you think through the various questions involved and set up your infrastructure, with a focus on the following topics:

UX Guidelines: Best practices for adding SMS to your current routing logic for notifying your users or customers.
Templating: Using templates with your emails and SMS notifications to decouple formatting logic from API calls.
Routing Logic: How to send notifications to the right channel while always obeying user consent.
Verification: Recommended checks and fall-backs in the case of failures.
Analytics: Advice for consolidating metrics and tracking across your email and SMS channels.

UX Guidelines

To begin, consider where it will be useful to add SMS to your existing email notifications in order to create a better user experience and/or more likelihood for a conversion. It may be helpful to build a flowchart of the new notification experience so that your team can have one central point of reference for where your API logic will need to be updated.

Default to SMS

If the user has opted in to receiving SMS messages, we recommend using SMS as the default notification channel for that user. Research shows that SMS has a 98% open rate compared to email’s 20%. However, users are also likely to feel that SMS is more invasive, and therefore it should be used more sparingly than email. You should have a high quality bar for your SMS notifications and use email for lower-priority content.

Style Guideline

There are no hard rules for when to default to SMS vs. Email and your users should always have the option to select their preferred channel for each notification, but there are a few guidelines we recommend following. For notifications that involve some component to time, such as event reminders or ETAs, SMS is a generally accepted communication method. ETAs in particular are a category where most users expect SMS rather than Email.

Two categories where it may make sense to send both SMS and Email are financial transaction notifications, such as a deposit or transfer confirmation, and time-sensitive alerts where human action is required. There are many other great use cases for SMS, but these are some of the more common use cases we’ve come across.

Avoid notifications across multiple channels

While it may seem like more notifications means more chances to get the user’s attention, we strongly advise against sending both an SMS and an Email for the same event. This risks appearing spammy, and if you default to SMS, an additional email will have diminishing returns. There may be some exceptions, however; for example, you might want to let users opt in to both email and SMS confirmation about an important event on your platform (e.g. delivery of goods or services) where the risk of being perceived as spam is low.

Understand consent

While a user opt-in is required for most emails you will send, it is a hard requirement for all SMS notifications. Current UX best practices recommend a double opt-in for SMS, where the first SMS that the user receives is a request to confirm subscription to SMS notifications. Whatever your choices are in this area, we recommend that you seek guidance from your legal counsel to ensure that you are following all the necessary steps to ensure compliance with CAN-SPAM, CITA, and TCPA in the U.S., and any other rules that may apply in other countries or regions.

Templating

You most likely have at least 10–20 different email notifications that exist at various logical points in your codebase (a promotional notification, a notification when a purchase is made, etc.), each with its own template for customizing the email to the specific user or product it is targeting. To add SMS, you will need to build new templates.

One common anti-pattern is to combine templating logic with the code for making the API calls themselves. While this might be acceptable when you have only one channel, when you are integrating several channels at the same point in your code, the complexity tends to grow exponentially with each channel that you add. Therefore, as a best practice, we recommend creating flexible templates in order to make the integration cleaner.

Dynamic Templates with SendGrid

If your existing email notifications use hard-coded templates, we recommend migrating to SendGrid’s Dynamic Templates. SendGrid provides HTML templates that you can customize entirely in code or with a WYSYWIG editor, with Handlebars syntax for dynamic personalization. Dynamic templates will simplify your notification logic immensely, which is a good first step if you plan to add more channels.

SMS Templating

For SMS, there is no comparable tool for accessing and editing saved templates, so you will need to store hard-coded templates in your site’s backend. While you could use simple string concatenation or interpolation, we recommend using a text templating language like Handlebars, which makes your templates more human-readable and easier to adjust.

At this stage, you may also want to consider how you’ll support MMS elements like images. Twilio supports sending up to 10 media files (see the mediaUrl property here). There are size and type restrictions, and you can’t inline the images; they will appear either before or after the text, and that’s determined by Twilio.

Routing Logic

Once your email templating is streamlined, you should be left with much simpler API calls. Next comes the logic for the integration itself, as you add an SMS message to places in your system where you currently send an email.

Based on the UX best practices, we do not recommend sending both an email and an SMS at the same time unless requested by a user. Instead, you will want to have some if/then logic that tries one channel first, and failing that, defaults to the other. So, for each of your existing email notifications, you will want to create some logic that says something along the lines of: “if we have the user’s phone number, send them an SMS via Twilio; if not, send them an email via SendGrid”—or vice-versa.

Consent and Preferences

Each channel adds a new form of consent to be managed, so when integrating Twilio on top of SendGrid, you will probably want to take another look at how you are managing user consent internally. SendGrid’s Unsubscribe Groups are a great tool for this, but you will need to normalize the data if you want a single source of truth that can work across both email and SMS.

We recommend creating a database table that stores your users’ preferences for each notification and channel combination. While this can get combinatorially complex, you will want to make sure that users can manage the combinations independently. A user might want to opt in to SMS and email for order status updates, for example, while for marketing announcements they might want to turn SMS off and enable emails.

While revamping your settings and opt-in, don’t forget to consider your unsubscribe flow. You may want to build a webpage that lets users unsubscribe from certain categories of SMS messages, or to switch from SMS to email. Otherwise, their only option for unsubscribing will be to text STOP, meaning they will opt out of SMS notifications entirely.

Verification

To make sure your notifications successfully reach your users or customers, we recommend verifying that your messages are delivered and creating a fall-back plan if delivery fails.

Verification with SendGrid

You are probably already calling SendGrid’s Email Activity Feed API to verify that your messages have been delivered (the success code you receive when you first call their API only says that SendGrid has received your request). If you haven’t built this yet, we recommend building it as a background worker that can execute multiple times. We’ve seen delivery confirmation often take more than 10 minutes, and in some cases multiple hours. Therefore, you should consider setting a TTL and an appropriate retry delay. Also, be careful with throughput, as we’ve found it is much easier to get throttled on these calls than in the Mail Send calls. If your volume is higher than a few dozen messages per hour, consider setting up a webhook receiver from SendGrid instead of calling the API.

Verification with Twilio

A common approach here is to reuse your background worker for SendGrid verification to perform the check with Twilio as well. You’ll want to add an if-statement to your worker which says to check either the Twilio Messages API or the SendGrid Email Activity Feed API depending on the message ID returned from the original Send call. If you sent the message over both email and SMS, then you’ll need to check both APIs separately. As with SendGrid, you also have the option of setting up a webhook receiver to track delivery status.

Failure Behavior

If delivery fails in one of these channels, it is important to consider how you want your system to behave. Do you want to attempt an email message if SMS delivery fails? Would it be appropriate to send an SMS if an email failed? The answers depend on your use case, but you can create the desired behavior by adding to your routing logic. It’s usually a good idea to send Email when an SMS fails in order to ensure the notification gets delivered, but it’s not always a good idea to send an SMS when Email fails, since you want to reserve SMS for more important use cases.

In general, we recommend that you normalize the deliverability results across your channels and perform a side-by-side comparison of delivery rates across all of your channels and all of your providers beyond just SendGrid and Twilio.

Analytics

One of the most challenging aspects of integrating Twilio and SendGrid is understanding your engagement performance across channels. While SendGrid offers click-through tracking out-of-the-box, Twilio does not, so you will need to build some integration points between the two channels in order to surface cross-channel analytics.

We recommend using HTTP redirects to track the click-through performance of each of your email and SMS templates. You can create a specific URL to use for tracking each notification in each channel, and then log the source of the click while forwarding the user to the right page (usually, redirects happen so quickly that they are not noticeable).

When adding SMS notifications to your existing email pipeline, you will also want to think about the key you use to identify a user that has engaged with the content. The email address won’t be helpful for SMS, and vice-versa. Instead, you will want to create a unique per-user key that helps you aggregate metrics across channels for the same recipient. At Courier, we recommend that customers choose a recipient ID that won’t change; typically, the user ID values from your system work well as recipient IDs.

Conclusion

Adding SMS to existing Email infrastructure is a great way to start taking advantage of the multi-channel world we live in in order to increase user engagement, but many kinds of notifications are better served over other channels such as mobile and desktop push, Slack, Microsoft Teams, or other Direct Messaging platforms such as Facebook Messenger and Whatsapp. The unfortunate reality of working with these APIs is that many of the steps list above need to be replicated individually for each new channel. Every channel has its own unique formatting requirements for templates and its own API with different rules for failure. Moreover, every additional channel adds another layer of complexity to managing user preferences and to building analytics to get a complete picture of user engagement with notifications.

At scale, companies like LinkedIn, Airbnb, and Uber solve these challenges by employing teams of 20+ Full-Time software engineers dedicated to notification infrastructure. At Courier, we’re on a mission to build this as a service so that adding new channels does not introduce new complexity into your codebase, you can still take advantage of the unique features of each channel, and we build advanced functionality on top of basic notifications such as subscriber management and API-driven white-labeling.

If you’d like to learn more about how to use Courier, or just chat about notifications in general, we’d love to connect with you!

Build A Modern Product Notification System: For Engineers and Product

Troy Goode — Mon, 29 Nov 2021 19:39:23 +0000

The Developer's guide To Building Notification Systems

Part 1: User Requirements
Part 2: Scalability & Reliability
Part 3: Routing & Preferences
Part 4: Observability & Analytics

Hi, I’m Troy Goode, the founder of Courier. Today, we’re going to talk about building a modern product notification system. This talk’s really intended for engineers, product managers, anyone working for a product organization that needs to send notifications and messages to their users, and maybe is thinking about where and how they need to be building that over the next 6 months, 12 months, 18 months as their product and organization evolves. By the end of this, we’re hoping to really give you enough to get started thinking about what improvements you might want to make to your notification infrastructure or what notifications you want to send that you’re not sending today and how you should think about building and maintaining those systems moving forward. We are putting together a white paper that’ll be linked below. So take a look at that if you want to dig into even more detail than what we cover in this discussion.

So when we think about a modern product notification system, we need to think about what are the requirements, right? And what we’re going to walk you through is a few of the ways we think about it when we work with our customers and what we hear about from our customers that may have already built their own product notification system before they met Courier. Here’s the way I normally like to break it down is between requirements that are for the development team: Right? The team that’s building and maintaining these notifications and these messages, and the requirements that are really for more of the product management team, the design team, the marketing team, and support team, those that maybe aren’t directly responsible for building this infrastructure, but who really rely on this infrastructure to power a lot of the activities and objectives that they’re trying to achieve with their projects with the product.

User Requirements

When we think about the objectives and the requirements for development team, three of them we’re going to dig into deeper later on in this discussion. That would be scale and reliability, abstracting your channels and providers, thinking about how do you route between the different kinds of channel and take into account the preferences of the recipient or user? And how do you take all of the messages that are flowing through this infrastructure and put a layer of observability and analytics on top of it so you can know what is and is not working the way you would want it to?
In addition to that, though, and we won’t go into these in as quite as much detail in this discussion, but you should be thinking about what is the developer experience for other developers within the organization? Because while some companies that we work with have a dedicated, centralized notification infrastructure managed by a dedicated, centralized comms team, many, many, many more companies that we talk to don’t and this infrastructure sometimes can be centralized, sometimes not, but the team is very typically distributed and different teams will need to interact with the infrastructure that you’re building to solve different use cases.

They’re essentially a customer, they’re an internal customer of yours, so what should that experience look like? You need to be thinking about the analytics needed by dev ops and other parts of the organization, not just at the business level but also at the operational level. How do you know when the messages are going out as expected? When are they not? When are they delayed? Which of these investments are paying off and maybe you should double down on and which maybe aren’t, and maybe you should revisit and reconsider.

Last is, and this is really kind of tied to that developer experience. How do you set up good testing environments? This is actually especially challenging for messaging infrastructure. You want to be able to potentially run integration tests and you want to be able to run scale tests, but how do you do this without accidentally sending messages to real people or significantly driving up costs with your downs stream service providers? Thinking through how do you create an environment where a developer can test against their local changes? How can they test within a staging environment, pulling together a number of different possible poll requests and testing it all together to see that it’s not going to impact production negatively? Then of being able to do things like smoke tests and the actual non-testing production sends from your live environment.

Managing Volume Spikes

Whether you’re sending 100 messages a day or 100 million messages a day, you do need to think about scale and reliability. Obviously scale becomes much harder as you go to larger and larger volumes, but what we’ve found is that even for companies with really small amounts of notification volume, it’s still harder to scale than you might think. The reason why is because it tends to come in bursts. Your notification volume doesn’t really get spread out like peanut butter. If you’re sending 30,000 messages a month, that doesn’t mean you’re sending 1,000 messages a day and you wouldn’t then divide that by 24 hours and by 60 minutes. Instead, what you see is huge spikes from time to time and then long valleys.

Provider Constraints & Errors

When you’re thinking about building your infrastructure, you need to make sure that you’re accounting for what your tallest spike may be. And that’s the spike on your side but you also need to be thinking about downstream impact because whatever channel you’re using, whether it be email or mobile push or Slack or SMS, there are going to be constraints that your service provider implements as well. How many messages can you send out over how long of a period of time? You also need to be thinking about, “Okay, well, if my spike exceeds the possible spike input for that provider, I need to make sure that I’m backing up those messages and robustly being able to trickle them through the downstream service provider at the rate that that service provider allows.”

On the reliability side of things, messaging is not perfect. It’s pretty common to see issues and failures. When we’re looking at email, we have things like bounces, incorrect email addresses, also service outages for ESPs. Long delays in things like getting delivery confirmation from the various ESPs, not only at the send layer, but at the receive layer. On SMS, you see a number of issues that can vary by region where you might see temporary outages in one region of the world. While the rest of the regions are working fine.

ON things like push, it’s very common to not even be able to know, “Did my message get successfully delivered?” You might see that the Apple push notification service accepted this message. That doesn’t mean that it ever showed up on a device for the user. Across all of these different channels, each has their own unique constraints around how do you know how well things are working and under what scenarios might they fail?

Retry Infrastructure

You need to be thinking about what happens when they fail. One obvious thing to do is to make sure you have robust retry infrastructure in place so that as a message goes out, if it fails for any reason, you want to be able to requeue it and retry it. If you do this, make sure this will impact kind of the scalability requirements that you have, because if a bunch of them start to fail, let’s say it’s a general service outage of the downstream provider or let’s say imagine your API key is wrong because somebody rotated it and forgot to go update it within your environment variables. Well, now you’re going to see a ton of that volume basically get requeued and reprocessed and then fail and requeued and reprocessed. This is where things like exponential back offs come into play.

I also think you should really think about things like determining whether a failure is retry-able or not. If it’s an API key, that’s invalid, honestly, you probably shouldn’t retry it. It’s unlikely to be resolved with a subsequent request. I’ve seen a few service providers where we get intermittent API key failures and so on Courier side, we’ve had to kind of add even more intelligence but I would say that that’s an edge case. More than likely you can say if the API key is bad, you need to go that environment variable, retrying it’s not necessarily going to help.

But there are other failures that may very well be intermittent and that might be downstream at the carrier level where you are going to want to retry that. If you have these retries in place, if you have exponential back offs, you need to be thinking about the time limit on them. Retry for up to how long? Up to 72 hours? Up to 24 hours? That can also really vary by the type of content you’re trying to deliver.

Channel/Provider Failover

You probably don’t want to deliver a password reset message 72 hours later, that would be weird. Some messages, maybe it’s fine to deliver a few days even a week later. Each of the kinds of messages that you’re sending may have a different correct strategy for how long you’re willing to retry these things. Think about that and think about ways to instrument your platform to say, “Okay, for this kind of message, here’s going to be our retry policy. For this one, we’re never going to retry it. This is an important message. It’s got to go out, but if it doesn’t go out, maybe it doesn’t make sense to retry it.” I kind of think you probably wanted to retry at least a few times in a short time span, but you might not be interested in retrying for more than a few minutes. For others, you might be fine with a really long retry policy with maybe less frequent retries during that life cycle so that it can have less impact on the scalability requirements of your system.

The other thing you can consider, especially if you have multi-channel infrastructure, is when do you failover from one channel to another failure, another channel? Another thing you can consider is if you have multiple channels, when do you failover from one channel to another? Let’s say that you were going to send me a mobile push message and for whatever reason, you get a failure. You know in this case that didn’t go through. Well, you could retry it and maybe in some cases that would be something you’d want to do, but maybe in other cases, what you should do is retry that message on a different channel. This time, send it to Troy via email or SMS instead of push. That’s channel failover.
You should also consider provider failover. Let’s say you’re sending an email, maybe SendGrid is your email service provider, and for whatever reason, the SendGrid account isn’t working right now so you attempt to send the message it fails, and you can see that this is not really worth retrying with SendGrid. Again, bad API key or a service adage on SendGrid’s side. Well now, instead of failing over to a different channel, do you instead failover and call Mailgun or Postmark? A lot of times, this is something that you really want to do at scale, because you have a lot of messages going out to a lot of users and you don’t necessarily want to get in that password reset example to send this message 48 hours later. You want to make sure that it goes out now and while you probably have a service provider that’s your preferred, well, it’s good to have a backup.

Idempotency

As you think about things like retry logic, well, you also want to make sure that you’re not resending the same notification to the same user. This is unfortunately pretty easy to end up doing. One of the things that you need to bake into a system is idempotency and idempotency just means a way to track which messages have been sent to which users.

Imagine that you were sending a batch of messages and notifications to 1000 different users and you’re looping through this batch and halfway through processing it something fails. Well, you want to requeue and retry that batch. Now you make it to user number 800 and it fails again and you retry the whole thing again. But what ends up happening here is you see the same user receiving the same notification many, many times. HBO Max recently had an issue with this.

Well, what you want to do and what you want to bake into your infrastructure to prevent this is idempotency. You want to be able to key each of these notifications that you’re sending to an idempotency key similar to how Stripe does it so that you can make sure that each of those notifications can it’s processed for each user only once. Your infrastructure needs to be checking for this even if it’s processing everything in one big batch, that way you can safely retry not just individual messages, but also huge batches of messages. As you build out all of this infrastructure for retries, for failovers, for item potency, the key things you’re going to be wanting to measure are latency, and in this regard, we’re really talking about from the time that you said, “Send this message to Troy,” how long does it take for the infrastructure to attempt that send? Time to first attempt. Many things can happen after that, where maybe the provider failed but that first bit, that’s your latency and you want to keep that as small as possible. Well under a second. The gold standard here is about 200 milliseconds.

You also want to be checking things like, “Well, what is the deliverability rate I’m getting for each of these providers and for each of the channels?” Now channels tend to be more of let’s call it a performance optimization input. You can know which channels are most effective, but provider deliverability is more of a systems effectiveness measurement. You will see significant differences in deliverability between different vendors and providers that you’re using. It’s really critical to measure each of them so you can understand which one’s working well for you and which one is not. This is basically what then can drive towards the ceiling of what is possible for that channel. You don’t want to be artificially constrained by some limitations you might have placed on you by issues with the provider that you’re using.
As you pull all of this data together around things like latency and deliverability, start to come up with what you think for your own internal needs to be your SLO. What is your service level objective? What is the goal that you want to make sure that your team is consistently hitting to make sure that it’s not negatively impacting the rest of the business?

Service Level Objectives

Once you’ve identified your SLO, then you want to be thinking about, “Okay, well, what are the SLIs that help you measure that? What are the service level indicators,” maybe you’re using Datadog or some sort of other observability platforms, “That I can look at to see what is the latency? What is the request per second that we’re able to process here? What is the deliverability rate we’re getting for each channel?” Look at those SLIs, figure out a way to pull that together, and report it out to the rest of your engineering team. Maybe it’s just your team, maybe it’s the entire department. You can figure that out for what works best for your business, but you have to make sure that you’re constantly measuring this, because it will just change over time, even without you interacting with it. Measuring it at just one point in time is not sufficient. Make sure that you have processes in place to continuously measure this and compare it against your own internal objectives.

The Right Message, at The Right Time, to The Right User, through The Right Channel

A few moments ago, we were talking about failover. When do you decide to failover between one channel and another, or failover between one provider within a channel to another provider? That’s one form of routing. Routing between channels, routing between providers. But routing is actually a broader concept that you should be thinking about as you design your modern product notification system. Failover and reliability is one goal here, but also things like, well, just making sure that the message is delivered to the user using the right channel at the right time. Scheduling falls under the routing umbrella. Do you deliver that message right now or do you wait until tomorrow morning, business hours, or maybe after work? Figure out when you want to deliver this message and figure out based upon when it’s being delivered, well, which channel’s going to be most effective?

If you’re a B2B product, delivering a message in the middle of the day, perhaps Slack would be a great channel. If you’re a B2C product, delivering a message at 7:00 PM, well maybe mobile push is likely the best channel for that message. That’s kind of a naive take though because you can also take a lot of data about the user to help influence this.

For example, let’s say you have in-app notifications, a little toast that pops in from the corner of the screen or a desktop notification that appears in the corner of the operating system, or even as a PWA notification on their mobile device. Well, maybe do you only want to send those if the user is currently online? Same thing for Slack, a Slack notification. Do you want to send a message to the user in Slack if they’re not currently logged into Slack? Would you maybe instead reroute that message to email?
Take into account what you know about the user. If you don’t have their phone number, you’re not going to be sending an SMS so make sure that you are able to then reroute and redirect that notification to the channel that will be valid for them. If you have other metadata about the user, such as presence of are they online and which platform, use that to help you understand which channel’s going to be the best. Use other context around what kind of use they are, what kind of use case it is, and what time of day it is in their time zone to help determine when and via which channel you should be delivering your messages.

Dynamic Preferences

One last thing to think about when you’re thinking about building routing across channels into your notification infrastructure is well, does the user actually have a preference? As much intelligence as you can add to dynamically switch between different channels, a lot of times users might have their own opinion and being able to extend that opinion out via your app and ask them for their opinion, via for example a preferences page, is really important. Let them turn off just the notifications they don’t want to receive instead of everything. Let them say, “Hey, for this notification, I’d rather receive it via SMS instead of email.” Maybe at some point, you even consider things like letting them specify a different recipient. Maybe say, “Hey, you know what? I’m going to be on vacation. Please send this to my colleague instead of me,” or pause notifications for some time period, instead of turning it off entirely, “I’m going on PTO, don’t send any more of these notifications for the next week.”
As you think about this, there’s going to be differences between what preferences you’re willing to let users set on some channels or on some kinds of notifications versus others. Let’s say you have, password reset emails are not something you ever want somebody to unsubscribe from, whereas the weekly newsletter, you probably want them to unsubscribe from that without impacting some of your other more growth-oriented messages. Divide up the different kinds of notifications that you send. Think about how you segment them and think about the different channel options that are applicable to the user for each of those notifications.

Digests

Lastly, think about potentially what we call digests, which is pulling many, many notifications together, and instead of sending many, many notifications out to the user, do you aggregate them and send them out to the user in a batch? If you think about LinkedIn, for example. Instead of receiving an email every time somebody asks to connect with you, sometimes you’ll receive a notification from LinkedIn saying five, six, 20 people have asked to connect with you. Do you start to batch and group things together and send out digests, which will increase the value of that notification and decrease the annoyance factor?

Audit Logs

We’ve talked a little bit about observability in analytics. We talked about using SLIs to make sure that you’re meeting your SLOs. What else do you need to be pulling together? Well, the way I think about it is this. Observability is really primarily geared towards support and engineering. It’s to make sure that a system is working the way you expect it to. There are some performance indicators that would fall under this, things like those SLIs, but you also need to think about logs. It’s more challenging than I think any of us might wish it were. When we think about channels, and often this varies by both channel and provider, if we think about really mature channels, like email, if you’re a SendGrid customer, you can log into the SendGrid account and get a pretty good snapshot of what emails you’ve been sending to whom and what the impact was. Did it land? Did it bounce? Did it get opened or clicked?

But as you look at other channels, it starts to become something that you have less access to, certainly via the actual provider. Mobile push, you typically need to layer another service on top of to get access to that data. Things like Slack or WhatsApp, you have very little to no visibility into without building custom infrastructure. When you’re sending this message, how does the developer, or in some cases, support team look at what happened? What happened as soon as the developer said, “Send this notification to Troy,” what happened next? How did the system decide which channel that was going to deliver that message to Troy? What happened when it made that API call? Did it fail? Did it have to be retried? If it’s being retried, when’s it going to reattempt it and when will it be delivered?

Logging is really helpful for really looking at and debugging use cases in test and development, but also in production when you do see things go wrong. Make sure that, that data’s all being pulled together in some way, whether you’re putting it into your data warehouse and creating look at reports that give you direct logs access. Those typically are not likely real-time, which makes it hard for certain debug scenarios. Maybe instead put it in something like Datadog, which is better for the engineer debugging scenario, but maybe a little bit tougher for the support debug scenario. Regardless, make sure you’re finding a way to pull that data, those logs, together in a place that both the engineering team and the support team can access.

Engagement & UX Outcomes

Beyond the observability side of things, then we look at the analytics side of things, and this is really the business outcome side of your notifications. Is it driving the value that you’re hoping for? Is it creating engagement? Is it creating the right experience? If somebody’s resetting their password or receiving a magic login SMS or email, you want to make sure that’s coming in fast and that people are actually clicking it. You should see very few instances of people requesting password reset emails without them resetting their password. You should see very few instances of people getting a 2FA token delivered via SMS without them then logging in and inserting that 2FA token. You want to make sure that you’re measuring for each of these use cases how many of these notifications are going out, which channels are they going out via, and what was the outcome? Did it actually produce the intended effect?

You want to then look at this and you can start to see, “Hey, maybe the one provider works better than another,” or maybe, “This channel isn’t as effective as you were hoping it would be.” Pulling that data together and being able to look at it in aggregate by channel, in aggregate by notification use case, template for example, looking at it by user cohort, these are at ways you are going to want to slice and dice this data so that people in the product management team, people in the engineering team, people in the data team can look at that data and understand where does it make sense to continue to invest? Where might there be problems and there be dragons? Dig in deeper to figure out what needs to be fixed or improved.

Building a modern product notification system, honestly, it’s pretty complicated, but it’s not impossible. Most software companies have to do this at some point in their life cycle. If you’re just getting started, probably just pick a single channel and a single provider, and off you go. If it’s something like email or mobile push, which usually most companies start with one of those two, there are great platforms for both of those channels to get started with. As you expand and grow your audience and maybe start to investigate additional channels or additional use cases that are more complex, then usually need to start to think about investing in a broader ecosystem of infrastructure that can tie all of this together. At scale, you’re probably going to want multiple providers for each channel. In fact, if one of your channels is mobile push, you’re going to need that more or less from the beginning, because you’ve got the Apple ecosystem and the Android ecosystem, you’re probably going to need to service both. Start to think about, even from the very beginning, how do you abstract away these different channels, these different providers? How do you pull together all of this data, create observability, create analytics that can be consumed by your team and the rest of the organization?

Hopefully, this video gave you enough to get a feel for what parts of a modern product notification system are applicable to you today versus what might make sense down the road. For whatever parts are applicable to today, we’ve produced a white paper that digs into even more detail. We’d love to have you download it and check it out, looking for feedback, if you have any. Are there other parts of building a modern product notification system we haven’t really talked about that you’d love to hear how we do it or how we see other customers building? We also have a Discord server and would love to have you join us there. We’re all hanging out. Would love to chat with you and geek out around notifications if this is something you’re working on. Please stop by and say hi. Thank you.