Forem: SparkPost

End-To-End Email Testing With MailSlurp

Jose — Fri, 09 Feb 2018 14:01:20 +0000

How To Conduct End-To-End Testing With MailSlurp

If you send email from an application, you probably have tests that ensure your code is attempting to send that email. But do you have a right way of testing whether that email got sent? MailSlurp is an awesome free tool that helps you do that. It allows you to easily generate random inboxes to send email to and then confirm delivery through its API.

In this post, we will use the MailSlurp Javascript SDK to write an end-to-end test for a node function that sends an email. We’ll be using SparkPost to send the email. If you aren’t already sending email, here’s a good place to start (for free). For our test we’ll use Jest, but any framework should work just fine.

Our code will be using a few es2015 features, so be sure to be running a version of node that supports them (version 6.4.0 or above should be fine). We’ll be installing dependencies using npm version 5.6.0.

All the code for this demo can be found here.

Getting Set Up

Alright, let’s get started! The first step is to get set up with a MailSlurp account. It’s quick and free: Sign up for MailSlurp. Next, log in to your dashboard and grab your API key.

Once you have your API Key, we can get started with the test setup. Make sure you have these dependencies in your package.json – here’s what ours looks like, using the SparkPost client library:

"dependencies": {
    "sparkpost": "^2.1.2",
    "mailslurp-client": "^0.1.1514850454",
    "jest": "^22.1.2"
  }

And run npm install Here is our full package.json. Normally packages like Mailslurp client and jest should go under “dev-dependencies”, but we kept it simple for this demo.

Now from the top of our test file, we require Mailslurp-client and our sendEmail code:

// test file: mailslurp.test.js
const MailslurpClient = require('mailslurp-client');
const sendEmail = require('./sendEmail.js'); // our SparkPost email function

You would import whatever function you use to send email.

Right after, we initialize the MailSlurp client. This is also a good spot to store your API key in a variable.

// initialize the MailSlurp Client
const slurp = new MailslurpClient.InboxcontrollerApi()

// Sign up at MailSlurp.com to get one of these for free.
const slurpKey = process.env.MS_APIKEY;

We used environment variables to access ours; if you store credentials a different way, that’s ok too. Our slurpKey variable will be the first parameter we pass to every MailSlurp client method.

Next step is to create an inbox we can send to with the MailSlurp client’s createRandomInboxUsingPOST method:

describe('SendEmail Tests', () => {

  let address;
  let inboxId;

  beforeAll(() => {
    // Creates the inbox and set address & inboxId for use in tests
    return slurp.createRandomInboxUsingPOST(slurpKey)
      .then(({ payload }) => {
        address = payload.address; // the email address we will be sending to
        inboxId = payload.id; // UUID used to identify the inbox
      });
  });

 /* tests will go here */
});

The important things here are:

Execute this code before you run your tests
Store both address & id from the response as variables, so you can send to and access your inbox

Now for the actual test we want to send an email (sendEmail() function for us) and then use the MailSlurp client’s getEmailsForInboxUsingGET method to get the email from the inbox:

test('Email delivers', () => {
    // increase the default jest timeout to wait for delivery
    jest.setTimeout(60000);

    // pass address given to us by MailSlurp
    return sendEmail(address)
      .then(({ total_accepted_recipients }) => {
        // make sure SparkPost accepted the request
        expect(total_accepted_recipients).toEqual(1);

        // Get the list of emails from the inbox
        return slurp.getEmailsForInboxUsingGET(slurpKey, inboxId,
          {
            minCount: 1, // minimum number of emails to wait for, default wait time is 60s
          })
          .then(({ payload }) => {
            // console.log(payload) // uncomment this to see the full email
            expect(payload).toHaveLength(1);
            const [email] = payload;
            expect(email.subject).toEqual('MailSlurp Test Email');
          });
      });
  });

Email delivery can take more than a few seconds, so make sure your test waits around long enough for the email to be delivered. In the code snippet above, we change the timeout threshold to handle that. Note that we are passing address to our sendEmail function, and then passing inboxId to getEmailsForInboxUsingGET.

For this test we first asserted that MailSlurp returned an array of emails with a length of 1 (we only sent one email). Then to make sure it was the email we sent, we asserted that its subject was ‘MailSlurp Test Email’ as defined in sendEmail.js

That’s it! Run the test however you usually do. Ours is set up with npm scripts in the package.json, which runs with npm test.

Next Steps

There is plenty of room for more assertions in our example test. When using an email delivery service like SparkPost in combination with complex HTML templates, an email’s body can get pretty complicated. To make assertions about an email’s actual content, we suggest using an advanced email body parser. Once you are able to extract the HTML/text content from the body, you could easily use snapshot tests to ensure you are sending the content you want.

If this was interesting to you, you should check out the rest of MailSlurp features. You can reach us on Twitter for any questions. And you can also send 15,000 emails a month for free on SparkPost! Special thanks to Jack Mahoney for building MailSlurp and answering my questions for this post.

–Jose

The post End-To-End Email Testing With MailSlurp appeared first on SparkPost.

How To Send Faxes Via Email Using SparkPost, Twilio & Cloudinary

Avi Goldman — Fri, 12 Jan 2018 14:08:50 +0000

No Fax Machine? No Problem! SparkPost, Twilio, and Cloudinary Are Here To Save The Day!

I don’t know about you, but I don’t have a fax machine. Once in a blue moon, I have to send or receive a fax, so I was excited when Twilio shared an awesome way to receive faxes straight to your email using SparkPost. But that is only half the battle. I decided to build the ability to send faxes from an email so I would never need a fax machine again!

We’ll build a function so we can send a PDF attachment to FAX_NUMBER@YOUR_DOMAIN, which will automatically fax the PDF to the phone number before the at sign (a.k.a. the local part, for all you email geeks out there). To do this, we’ll use SparkPost’s inbound functionality, Twilio’s Fax API, and Cloudinary to glue them together. We’ll receive an email to a Twilio function, pull off the attached PDF, save it to Cloudinary, and send it as a fax.

Sign Up and Configure Your SparkPost Account

The first thing you’ll need is a SparkPost account and a domain you want to use to receive mail (a.k.a. an inbound domain). You’ll also need to create an API key with permissions to read and writeinbound domains and relay webhooks

Next, add the SparkPost MX records for your inbound domain. Once you verify that they are set up correctly, run the following cURL to add your inbound domain.

curl -XPOST \
  https://api.sparkpost.com/api/v1/inbound-domains \
  -H "Authorization: YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{ "domain": "YOUR_DOMAIN" }'

Sign Up For Cloudinary

Next, we’ll need a Cloudinary account to store the PDF we’ll fax. If you don’t know, Cloudinary is a powerful solution for storing, manipulating, and delivering all your media. Grab your cloud name, API key, and API secret and put them somewhere safe for later.

Create & Configure Twilio

We’ll use Twilio to consume the incoming emails and send the faxes. To get started, sign up for an account, and buy a phone number that can send and receive faxes.

Twilio’s serverless functions are perfect for this project. This isn’t a stateful application and it doesn’t need to be running all the time. Using one, we can quickly get set up and run our application.

We’ll need the following NPM modules for our function:

Let’s also add in our environment variables for Cloudinary and enable Twilio’s option to pass through our ACCOUNT_SID and AUTH_TOKEN so we can use their client library without worrying about our credentials:

The Code

First things first, let’s write our basic Twilio function and pull in our dependencies:

const cloudinary = require('cloudinary')
const { MailParser } = require('mailparser')
const { toArray } = require('lodash')
exports.handler = function(context, messages, callback) {
  const twilio = context.getTwilioClient()
  cloudinary.config({
    cloud_name: context.CLOUDINARY_NAME,
    api_key: context.CLOUDINARY_KEY,
    api_secret: context.CLOUDINARY_SECRET
  })
}

The messages variable is an object full of the messages SparkPost handed off to us. We’ll need to loop through the messages, pull off the attachment, save it to Cloudinary, and send off the fax.

// after the cloudinary config
const promises = toArray(messages).map((message) => {
    const toNumber = pickPhoneNumber(message)
    return pickPdfAttachment(message)
      .then((attachment) => {
        if (!attachment) { return false }
        return uploadAttachment(attachment)
      })
      .then((mediaUrl) => {
        return sendFax(twilio, toNumber, mediaUrl)
      })
  })
  Promise.all(promises)
    .then(() => callback(null, 'success'))
    .catch((error) => callback(error))

Parsing The Email

You can see an example the payload SparkPost sends in the API documentation. The important parts for us are the rcpt_toand the
content which will contain the raw email. These live inside message.msys.relay_message.

We can pull out the phone number by splitting off the local part from the rcpt_to value – everything before the @.

function pickPhoneNumber(message) {
  return message.msys.relay_message.rcpt_to.split('@')[0]
}

We’ll also need to pull the PDF off of the message. To do this, we’ll parse the RFC 822 value using the mailparser library and return the content of the first PDF attached.

function pickPdfAttachment(message) {
  return new Promise((resolve, reject) => {
    const content = message.msys.relay_message.content
    const isBase64 = content.email_rfc822_is_base64
    const body = isBase64 ? Buffer.from(content.email_rfc822, 'base64') : content.email_rfc822
    const parser = new MailParser({ streamAttachments: true })
    let attachment
    parser.on('data', (data) => {
      if(!attachment &&
          data.type === 'attachment' &&
          data.contentType === 'application/pdf') {
        attachment = data.content
      }
    })
    parser.on('error', reject)
    parser.write(body)
    parser.end(() => resolve(attachment))
  })
}

Saving The Attachment

Assuming we get an attachment, we’ll need to put it in an accessible spot for Twilio to pull it from. Enter Cloudinary. Using their Node library we can easily pipe the PDF and get the publicly accessible URL.

function uploadAttachment(attachment) {
  return new Promise((resolve, reject) => {
    const stream = cloudinary.v2.uploader.upload_stream((error, result) => {
      if (error) {
        reject(error)
      }
      else {
        resolve(result.url)
      }
    })
    attachment.pipe(stream)
  })
}

Sending The Fax

The last step for our code is to send the fax! Twilio makes this really easy. We need three values: the number we are sending to, the number we are sending from, and the media url. We’ll use the number we bought earlier to send for the “from” number. And we should have a phone number from the local part of the email and the media url from Cloudinary!

function sendFax(twilio, toNumber, mediaUrl) {
  return twilio.fax.v1.faxes.create({
      to: toNumber,
      from: YOUR_PHONE_NUMBER,
      mediaUrl: mediaUrl,
    })
    .then((res) => {
      console.log('sent', mediaUrl, 'to', toNumber)
    })
}

Create A Relay Webhook

The last piece of the puzzle it to tie our inbound domain from SparkPost to our Twilio function. Let’s create a relay webhook to pass all mail sent to our inbound domain onto our Twilio function. Copy your function path and pass it into this cURL request to create the relay webhook. Make sure that “Check for valid Twilio signature” is unchecked so that SparkPost can access it!

curl -XPOST \
  https://api.sparkpost.com/api/v1/relay-webhooks \
  -H "Authorization: YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{ "target": "YOUR_TWILIO_PATH", "match": { "domain": "YOUR_INBOUND_DOMAIN", "protocol": "SMTP" }, "name": "Email-to-Fax" }'

Send The Fax!! 🎉

Send an email over to FAX_NUMBER@YOUR_DOMAIN with your fax attached as a PDF and it should go through!

You can set up receiving faxes for a fully working email fax machine with some guidance from our friend Patrick at Twilio! Feel free to reach out if you’ve got any questions and have fun faxing!

-Avi

P.S. Before you launch this to the world, you’ll probably want to validate the phone numbers you’re sending to and add some security to who can send. I’d suggest a token in the email that you verify in the Twilio function.

The post How To Send Faxes Via Email Using SparkPost, Twilio & Cloudinary appeared first on SparkPost.

Announcing HEML: An Open Source Framework for Email

Avi Goldman — Fri, 27 Oct 2017 13:05:21 +0000

Today I am excited to announce the first release of HEML, an open source markup language for crafting clean, responsive emails.

Why

If you’re not familiar with writing HTML emails, it can be a painful process. A few months ago I started to play with some exciting interactive developments on the front end. I found the bottleneck wasn’t sending or testing the email, but simply building the email itself. At our next company hackathon, I took the opportunity to solve this problem, and thus HEML was born.

Each element in HEML renders into email-ready HTML so that you can send without worry. HEML also works to iron out CSS bugs and limitations of different email clients. An excellent example of one such bug is an obscure issue in Lotus Notes where if you use RGB decimal value, the entire style declaration will be ignored. HEML will handle that bug for you.

Our Goals

We wanted HEML to help make email more accessible to developers. The idea is for developers to jump in quickly and build their emails without wrestling with Outlook (or any other email inbox). To do this, we focused on three things.

Native feel

We wanted HEML to look and feel like HTML. As a result, it mirrors HTML as closely as possible and uses plain ol’ CSS for styling.

Forward Thinking

HEML doesn’t limit you from taking advantage of all that HTML and CSS can do. It encourages progressive enhancements. Email doesn’t have to look the same everywhere.

Extendable

HEML lets you create your custom elements, share them, and pull in other elements made by the community.

Using HEML

There are a couple of different ways to use HEML.

Get started quickly using our editor at heml.io/editor.

To use it locally, install it with:

npm install -g heml

Create your HEML email in email.heml:


<heml>
  <head>
    <subject>Email Rox!</subject>
    <style>
      body { background: SkyBlue; }
      h1 { color: DarkViolet; }
    </style>
  </head>
  <body>
    <container>
      <marquee><h1>Hello world </h1></marquee>
    </container>
  </body>
</heml>

and run:

heml develop email.heml

That starts a development server that will auto-reload your browser whenever you make a change.

Once you’re ready to send your email into the wild, run:

heml build email.heml

This generates an email.html file that is ready to be sent.

Give it a spin!

This is our take on a difficult problem. It doesn’t solve every problem presented by email, but it can help you create solutions for your unique email challenges. There has been amazing work done to simplify this challenge by MJML,Foundation for Email, and many others. We hope you find this equally as helpful!

So give it a try! Hopefully, it makes your life easier. If you have any feedback, suggestions, or bugs, let us know.

Happy Coding!

The post Announcing HEML: An Open Source Framework for Email appeared first on SparkPost.

Creating the Next Generation of Interactive Emails with Rebel Mail

Jennifer Lacey — Wed, 18 Oct 2017 13:05:32 +0000

**We are excited to announce our new partnership with Rebel. Rebel is disrupting the email marketing space and allows users to send interactive emails with no plug-ins required. This guest blog post, written by Manisha Shah highlights Rebel’s ease of use and their competitive edge in the market.

Integrate your email channel and app with in-email interactivity.

Email is one of the oldest communication protocols on the web. Unfortunately for the billions of people that use it to communicate, up until a few years ago, almost nothing has dramatically improved the user experience. After years of very little innovation, interactive email has created a flurry of exciting changes.

At Rebel, we don’t see stodgy, old email clients that are a nightmare to develop against. We see a brand new surface for your application to live on. Interactive email allows you to show a user the exact action you want them to take, without the need for a click or leave their inbox.

Taking advantage of these new opportunities for user actions is easier said than done. Coding normal emails remains a challenge for most developers due to the deep fragmentation in email clients. That’s why we believe the best way to take advantage is through our interactive email API.

It’s the only way of generating interactive emails on the fly, personalized to the user you are sending to. You don’t need to know the latest hacks to get the email rendering perfectly or the quirks of gathering data from inside an email client. All you need to know is the information you want to get from a user, and with a simple JSON request, you can get back ready-to-send interactive email HTML.

But what does that really mean?

If the above sounds fanciful and far-fetched to you, you’re not alone. However, it is that simple. All that you need to do is describe the functionality that you want to achieve in JSON and our API will convert it to HTML for you.

To prove that point we’re going to build a quick email that asks a user for some feedback on your app. It will contain stars for rating 1-5 and a text input for collecting general feedback.

The API call structure.

For this example, we’ll be looking at an abbreviated API call (I’ve taken out some of the options so that the information is clear), that will return to you HTML with two form fields, and a submit button.

As you can see below the JSON call includes an array at context.components.body.formfields, in that array there are two objects. Each describes one piece of functionality. One for the rating module and one for the text area. Here you place all the information you want to be covered displayed in the email and adjust and field-specific settings.

"context": {
    "components": [
      {
        "type": "form",
        "body": {
          "submitUrl": "http://rebelmail.com/review/",
          "id": "review-form",
          "formFields": [
            {
              "type": "rating",
              "variation": "star",
              "fluid": true,
              "layout": "stack",
              "name": "star-rating",
              "label": "How was the espresso?",
              "options": [
                {
                  "value": "1",
                  "text": "&#9733;"
                },
                {
                  "value": "2",
                  "text": "&#9733;"
                },
                {
                  "value": "3",
                  "text": "&#9733;"
                },
                {
                  "value": "4",
                  "text": "&#9733;"
                },
                {
                  "value": "5",
                  "text": "&#9733;"
                }
              ]
            },
            {
              "type": "textArea",
              "layout": "stack",
              "name": "comments",
              "label": "Anything else we should know?",
              "placeholder": "Leave your comments here...",
              "maxlength": "2000"

            }
          ],
          "submit": {
            "text": "Leave Your Review"
          }
        }
      }
    ],

The above example is abbreviated for illustrative purposes, but generally, our API allows you to set the values and styles of any element inside our interactive modules.

What you get back.

Our /campaigns/:id/html endpoint will return to you HTML, completely ready to be delivered. No need to do anything to it, just send it out. Here is the email from the full request above as seen in Apple Mail:

Interactive email analytics.

Of course what good is new technology if you can’t tell if it’s adding value to your business. That’s why we built a robust analytics platform for recording every possible action a user takes within an email. If a user clicks a radio button or star rating, we record it. If they open the email on Outlook and see a beautifully rendered static fallback message, we record it.

All that actionable data is only good if you can consume it. That’s why we created a number of ways to have our data pushed to your servers (such as a webhook).

Go forth and interact.

These kinds of forms are one excellent way to allow users to interact with your app from within an email client, and there is a myriad of ways to apply this same technology. Responding to comments, prompting inactive users to post again for the first time, or allowing users to complete portions of their onboarding while verifying their email address, are all possible.

You can do even more than forms with our flexible API, Thanks to the various modules we have created, your users can take quizzes, research or filter products, and even shop directly in the inbox. All while our robust analytics allows you to gather every action and use it for things like progressive profiling and user recommendations.

The post Creating the Next Generation of Interactive Emails with Rebel Mail appeared first on SparkPost.

Using the SparkPost Metrics API with C#

Mike Hillyer — Mon, 16 Oct 2017 13:02:48 +0000

*Today’s blog is written by Darren Cauthon, community member and original author of the SparkPost C# Library. Thanks for the post, Darren! We hope you enjoy.

How to Use the SparkPost Metrics API with C

In our first two tutorials on using SparkPost with C#, we learned how to send an email from a C# application and how to manipulate email templates using C# code. You’ll want to start there to get the SparkPost C# library installed via NuGet, it’s all good stuff.

But most of the time when you send an email from your app, you’ll also want to know what happened to that message: was it delivered? Did your user open it or even click on link in the message body? Visibility into message disposition and performance is crucial for many apps.

That’s why the SparkPost C# library includes methods for getting important metrics about the emails that you send. Today, we’re going to look at how they work.

There are two mechanisms for getting these analytics: pushing and pulling. With pushing (via a webhook), you can get your metrics streamed in near real-time as SparkPost posts the information to a web endpoint that you provide. A future tutorial will provide some help on setting up a webhook endpoint in a standard .Net MVC application.

Today, however, we’ll focus on pull-style API-based queries for your metrics via the C# wrapper. You can query the SparkPost API for just about any type of information you’d want about the emails you send within the past few weeks, and the API will return the information you requested.

This tutorial will also provide some insight into the library’s design, and how you can use it for other SparkPost API calls you’d want to make.

A SparkPost Metrics API Quick Start

When using the C# SparkPost library, the first place to check is the SparkPost API documentation. The C# library was written to mirror the SparkPost web API as closely as possible. Naturally, the API docs include specific information about metrics. Look there to see what sorts of metrics you might want to gather.

Let’s try the first method in the docs: Gathering metrics by domain. Let’s build a sample request that will return the number of accepted and bounced emails that were sent to Gmail or Yahoo Mail. Here is C# code that will return your results.

var client = new Client("YOUR API KEY");
client.CustomSettings.SendingMode = SendingModes.Sync; 

var query = new MetricsQuery {
    From = DateTime.Now.AddDays(-7),
    Metrics = new[] { "count_accepted", "count_bounce" },
    Domains = new[] { "gmail.com", "yahoo.com" },
};

var response = await client.Metrics.GetDeliverabilityByDomain(query);

Here are your results:

It’s really that easy. All of the metrics data you want, for this and any of the other API metric methods, will be returned in this form. The only real question is what data you want, so check out the SparkPost API documentation and make the query you want!

How to Keep the SparkPost C# Library Up-To-Date with API Changes

One of the difficulties in using a library for a service like SparkPost is keeping up-to-date with changes. SparkPost is always adding new features to its API, but are those new features reflected in the library? Sometimes there is a lag between when API developers add features and when library authors add support.

The C# library tries to mitigate those issues by allowing you to extend the library, yourself. The example above showed how to use the MetricsQuery, but you can pass any object to use as a query—so long as it matches the style and naming of the SparkPost service.

Here is a concrete example: SparkPost developers added a new API feature, the ability to pass a “limit to limit the size of your result. That feature was not available when the MetricsQuery class was written, so it does not have Limit available. So does that mean you can’t use Limit in C#? No, that’s still available to you – just have to pass it yourself. Like with an anonymous object:

var client = new Client("YOUR API KEY");
client.CustomSettings.SendingMode = SendingModes.Sync; 

var response = client.Metrics.GetDeliverabilityByDomain(new {
  From = DateTime.Now.AddDays(-0),
  Metrics = "count_accepted,count_bounce",
  Domains = "gmail.com,yahoo.com",
  Limit = 10
});

Another method would be to extend the MetricsQuery and add an “int Limit { get; set; } property. But the main point to notice is that so long as you stay consistent with the SparkPost documentation and naming, this library can extend itself.

The SparkPost C# Library Is Open Source

I’m the original author of the SparkPost library, but there are more than a dozen developers who have contributed code to this library, like this Metrics library, which was contributed by Aaron Sherber. Each developer on this library was either a SparkPost employee or a customer of SparkPost, all of us with an interest in making this library work well.

As the SparkPost API expands, this library will expand with it. And we’re accepting contributions from anybody who wants to help!

The post Using the SparkPost Metrics API with C# appeared first on SparkPost.

Tracking Recipient Preferences With The User Agent Header in Elixir

Ewan Dennis — Fri, 13 Oct 2017 13:01:32 +0000

Tracking Recipient Preferences With The User Agent Header in Elixir

Note: this user agent header post illustrates itself using code written in Elixir. If you prefer, you can read the PHP version.

Much has been made of the relative commercial value of particular groups of people. From super consumers to influencers, iPhone users to desktop holdouts, learning about your recipients’ preferences is clearly important. In these days of deep personalization, it’s also just nice to know a little more about your customer base. Luckily this is a pretty easy job with SparkPost message events.

In this post, I’ll review the content of the User-Agent header, then walk through the process of receiving tracking events from SparkPost’s webhooks facility, parsing your recipients’ User Agent header and using the results to build a simple but extensible report for tracking Operating System preferences. I’ll be using Elixir for the example code in this article but most of the concepts are transferrable to other languages.

SparkPost Webhook Engagement Events

SparkPost webhooks offer a low-latency way for your apps to receive detailed tracking events for your email traffic. We’ve written previously about how to use them and how they’re built so you can read some background material if you need to.

We’ll be focusing on just the click event here. Each time a recipient clicks on a tracked link in your email, SparkPost generates a click event that you can receive by webhook. You can grab a sample click event directly from the SparkPost API here. The most interesting field for our purposes is naturally msys.track_event.user_agent which contains the full User-Agent header sent by your recipient’s email client when they clicked the link.

{ "msys": { "track_event": { // ... "user_agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_3) AppleWebKit/537.36 
(KHTML, like Gecko) Chrome/41.0.2272.118 Safari/537.36" // ... } } }

Grokking The User Agent

Ok so we can almost pick out the important details from that little blob of text. For the dedicated, there’s a specification but it’s a tough read. Broadly speaking, we can extract details about the user’s browser, OS and “device from their user agent string.

For example, from my own user agent:

Mozilla/5.0 (Linux; Android 7.1.1; Nexus 6P Build/N4F26O) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.91 Mobile Safari/537.36

…you can tell I’m an Android user with a Huawei Nexus 6P device (and that it’s bang up-to-date ;).

Caveat: user agent spoofing

Some of you might be concerned about the information our user agent shares with the services we use. As is your right, you can use a browser plugin (Chrome, Firefox) or built-in browser dev tools to change your user agent string to something less revealing. Some services on the web will alter your experience based on your user agent though so it’s important to know the impact these tools might have on you.

Harvesting User Agents From SparkPost Tracking Events

Alright, enough theory. Let’s build out a little webhook service to receive, process and stash user agent details for each click tracked through our SparkPost account.

Elixir And The Web: Phoenix

The de facto standard way to build web services in Elixir is the Phoenix Framework. If you’re interested in a Phoenix getting started guide, the docs are excellent and the Up and Running guide in particular is a great place to start.

We’ll assume you already have a basic Phoenix application and focus on adding an HTTP endpoint to accept SparkPost webhook event batches.

Plug: Composable Modules For The Web

Elixir comes with a specification called â€˜Plug’ (defined here) which makes it easy to build up layers of so-called middleware on an HTTP service. The simplest form of plug is a function that accepts a connection and a set of options. We’ll use this form to build up our webhook consumer.

Handling SparkPost Webhooks Requests

Our first task is to create a “pipeline”, which is a sequence of transformations that a connection goes through. A pipeline in Phoenix is just a convenient way to compose a sequence of plugs and apply them to some group of incoming requests.

We’ll first create a “webhook pipeline and then add plugs to it to handle the various tasks in our service. All this happens in our application’s Router module:

# Create a new plug pipeline to handle SparkPost webhooks requests pipeline :webhook do
# Use the accepts plug plug :accepts, ["json"] end

You can read more about Phoenix routing and plug pipelines in the routing section of the Phoenix docs. For now, it’s important to realize that each Phoenix application includes an endpoint module which is responsible for setting up basic request processing. This includes automatic JSON parsing, which we’ll rely on here.

Unpacking SparkPost Events

Our event structure contains a certain amount of nesting which we can now strip out in preparation for consuming the tasty details inside. This is a job for our very first plug:

# Define a new plug to extract nested event fields def 
unpack_events(conn, _) do first_key_fn = &hd(Map.keys(&1))   
cleanevents = conn.params["_json"] |> Enum.map(fn evt -> 
evt["msys"] end) |> Enum.map(fn evt -> evt[first_key_fn.(evt)] 
end) assign(conn, :events, cleanevents) end # Add the new plug 
to our pipeline pipeline :webhook do plug 
:accepts, ["json"] plug :unpack_events end

There is a little magic going on here. Our endpoint applies the JSON parser plug to all requests before our pipeline starts. Our unpack_events plug can then rely upon the _json param left on the connection JSON parser.

The rest of unpack_events is just extracting the contents of the msys key on each event and the contents of the first key in that object. Finally, our unpack_events plug stored our unpacked events on a connection param for later plugs to pick up.

Filtering Events

Now lets retain just the click events (when we register our webhook with SparkPost later, we can also ask it to send only click events):

def filter_event_types(conn, types) do good_event? = 
&Enum.member?(types, &1["type"]) assign(conn, :events, 
Enum.filter(conn.assigns[:events], good_event?)) end # Apply event 
filtering after unpacking pipeline :webhook do plug :accepts, 
["json"] plug :unpack_events plug :filter_event_types ['click'] end

This plug leaves our filtered events on the :events connection param. filter_event_types accepts a list of types we care about.

There’s a lot of detail in a single event. It might be a good idea to pare things down to just the fields we care about:

def filter_event_fields(conn, kv) do pick_fields = &Map.take(&1, 
kv) assign(conn, :events, Enum.map(conn.assigns[:events], 
pick_fields)) end # JSON -> unpack -> filter on clicks -> extract 
user_agent fields pipeline :webhook do plug :accepts, ["json"]   
plug :unpack_events plug :filter_event_types, ['click'] plug 
:filter_event_fields, ['user_agent'] end

After The Plug Pipeline: The Controller

To finish up our webhook request handling, we need a controller which works after the plug pipeline to process to request and produce a response for the client. Here’s a skeleton Controller:

defmodule TrackUserAgentsWeb.ApiController do use 
TrackUserAgentsWeb, :controller def webhook(conn, _params) do     
json conn, %{"ok": true} end end

Then we can wire ApiController.webhook/2 to our router:

scope "/webhook", TrackUserAgentsWeb do pipe_through :webhook     
post "/", ApiController, :webhook end

When we register our web service with SparkPost as a webhook consumer, it’ll make HTTP requests to it containing a JSON payload of events. Now our service has a /webhook endpoint that accepts JSON, cuts our event batch down to size and responds with a happy little “ok!”.

Testing Our Progress

We can test our service by sending a test batch to it. Luckily, the SparkPost API will generate a test batch for you on request.

Grab a sample webhooks event batch from the SparkPost API: Note: this step uses cURL and jq. You can skip the jq part and remove the results key from the JSON file yourself.
curl https://api.sparkpost.com/api/v1/webhooks/events/samples | jq .results > batch.json
Start our Phoenix service:
mix phx.server
Send our test batch to the service:
curl -XPOST -H "Content-type: application/json" -d @batch.json http://localhost:4000/webhook

Parsing User-Agent

Now we’re ready to enrich our events with new information. We’ll parse the user agent string and extract the OS using the ua_inspector module. We can easily add this step to the API plug pipeline in our router:

Note: If you’re following along, remember to add ua_inspector as a dependency in mix.exs and configure it.

def os_name(ua) do if not Map.has_key?(ua, :os) do "unknown" 
else case ua.os do %UAInspector.Result.OS{} -> ua.os.name 
_ -> "unknown" end end end def parse_user_agents(conn, _) do 
events = conn.assigns[:events] |> Enum.map(&Map.put(&1, 
"user_agent", UAInspector.parse(&1["user_agent"]))) |> 
Enum.map(&Map.put(&1, "os", os_name(&1["ua"]))) assign(conn, 
:events, events) end # JSON -> unpack -> filter on clicks -> extract 
user_agent -> filter nulls -> parse pipeline :webhook do plug 
:accepts, ["json"] plug :unpack_events plug :filter_event_types, 
['click'] plug :filter_event_fields, ['user_agent'] plug 
:parse_user_agents end

Note: not all user agent strings will contain the detail we want (or even make sense at all) so we label all odd-shaped clicks with “OS: unknown”.

Alright, now we have an array of events containing only interesting fields and with an extra “os field to boot.

Generating Report-Ready Summary Data

At this point, we could just list each event and call our report done. However, we’ve come to expect some summarisation in our reports, to simplify the task of understanding. We’re interested in OS trends in our email recipients, which suggests that we should aggregate our results: collect summaries indexed by OS. Maybe we’d even use a Google Charts pie chart.

We could stop there citing “exercise for the reader but I always find that frustrating so instead, here’s a batteries-included implementation which stores click events summaries in PostgreSQL and renders a simple report using Google Charts.

An Exercise For The Reader

I know, I said I wouldn’t do this. Bear with me: if you were paying attention to the implementation steps above, you might have noticed several re-usable elements. Specifically, I drew a few filtering and reporting parameters out for re-use:

event type filters
event field filters
event “enrichment functionality

With minimal effort, you could add, filter on and group the campaign_id field to see OS preference broken down by email campaign. You could also use it as a basis for updating your own user database from bounce events with type=bounce, fields=rcpt_to,bounce_class and so on.

I hope this short walkthrough gave some practical insight on using SparkPost webhooks. With a little experimentation, the project could be made to fit into plenty of use cases and I’d be more than happy to accept contributions on that theme. If you’d like to talk more about the user agent header, your own event processing needs, SparkPost webhooks, Elixir or anything else, come find us on Slack!

The post Tracking Recipient Preferences With The User Agent Header in Elixir appeared first on SparkPost.

Community Spotlight: CodeNewbie

Jennifer Lacey — Wed, 11 Oct 2017 13:01:51 +0000

Have you ever embarked on an adventure to learn a completely new skill set? Did it make you feel overwhelmed with no idea where to start? For many of our community members, that feeling is more common than you’d think. Tons of developers or coders didn’t originally start out on that path, and today we’re chatting with someone who’s making big waves and providing a ton of resources for newbies in the coding community.

Meet Saron

Not too long ago, Saron was also new to coding. She worked for a few different start-ups, but became more and more interested in the product side of things – which required a more technical background. So Saron did what any motivated self-starter does – she vowed to teach herself. From there, she started her own bootcamp and conference to help others!

Today, Saron has built the CodeNewbie community, a growing network of over 15,000 people who join in on weekly twitter chats, listen to podcasts, attend CodeLand (check out Cole’s experience teaching a workshop) and exchange ideas with others working on projects in tech. Saron was gracious enough to share some insights on the community she’s built, her favorite part about her job, and advice for those just starting out. I hope you enjoy!

Enter CodeNewbie

What’s your background in the tech world, how did you get started down this path?

I first got interested in tech after reading the Steve Jobs book. It was the first time I saw the connection between tech and more liberal artsy topics like design and psychology. I then read everything I could find about tech and startups, and eventually worked at a few startups. I worked primarily on the business side of things, but I was always more interested in the product. Because I didn’t have a technical background of any kind, I felt that the only way I could have a say in the product was if I invested in learning how to code. So I quit my job, taught myself for a few months then enrolled in a coding bootcamp.

Why did you start CodeNewbie?

When I started my coding bootcamp, I was surprised to find that I not only learned a lot, but gained a lot of confidence in what I already knew. I realized that when I was learning on my own, I’d been internalizing the natural failure that comes with coding, and interpreting my struggles as my own stupidity. But in my class, I saw that everyone was struggling! It removed so much of the emotional burden that I could actually focus on learning. The true benefit of the bootcamp was the community. But finding that community is hard if you don’t have $11K and can’t quit your job. So I wanted to create a way for everyone to have a supportive community, and that community is #CodeNewbie.

In your opinion, how important is community in the programming world?

It’s everything. Community is how you get out of your own way. It’s how you get jobs, opportunities, make friends, build partnerships, create meaningful change. I’m here today as a result of the incredibly kind community around me, and that’s what I hope to provide for others.

Why did you start the CodeNewbie podcast?

When I decided to do the CodeNewbie Podcast, I’d been doing the #CodeNewbie Twitter chat for about a year. I loved the chat. It was such a great way to connect people and start conversations. But it’s not a great medium for diving deep into a topic or a story. Since I’d previously worked at NPR, I thought that a podcast was a great way to focus on one story, one problem, one idea, and really explore it. Audio is such a flexible medium because it’s one of the few you can consume while partaking in other activities. I felt it was a great way to provide a weekly dose of knowledge and inspiration in the lives of our community members.

Which episode is your favorite so far and why?

This is tough! I’d have to say Season 1, Episode 2: “Building community in a virtual world: Moderation tools in VR Cameron Brown. On a content level, I personally learned so much in that conversation and really appreciated the passion from Cameron. You can hear in his voice how much he loves creating a safe space for all. He talks so matter of factly about topics like diversity and inclusion, and how obviously important they are, in a way that I’ve never heard from a developer that absolutely blew me away. On a production level, I got to be more creative with the intro, so mixing that was lots of fun.

There are 150 episodes on the CodeNewbie podcast. For new listeners, where should they start?

It depends on their goal. If they’re looking to learn more about CodeNewbie in general and what we’re all about, definitely checkout the episode where my husband interviewed me. For job-hunting inspiration, I loved Paola’s incredibly powerful interview on her journey to finding a job. To hear how tech and society collide, I thought Carina’s episode on algorithms was brilliant. For an inspiring story on persistence and dedication, definitely checkout George Moore’s episode “Truck Driver.”

What are some great resources for all the CodeNewbies out there?

There are so many! The obvious ones are Codecademy, Coursera, Khan Academy. But there are also great communities like Girl Develop It, Women Who Code, RailsBridge, Hacker Hours, PyLadies.

Do you have any advice for people who want to begin their own journey in programming but don’t know how to start?

Yes! Nothing is going to make sense for awhile. It’s going to take a lot of beating your head against the wall, not really knowing how useful what you’re learning is going to be, and not building anything that seems relevant to a job before things start to click. That’s just part of the process. I think newbies obsess over starting with the right thing and seeing immediate results, and this results in people quitting much sooner than they should. So I’d say, pick a language with a large and supportive community (ruby, python, javascript are great for these reasons), pick a free resource that doesn’t require much setup (I usually recommend Codecademy), start and then *stick with it.* Over time, things will fall into place.

Questions for Saron? Join in on her weekly Twitter chats on Wednesdays at 6PM PST/9PM EST! Questions for us? Reach out on Twitter or find us in our community slack.

The post Community Spotlight: CodeNewbie appeared first on SparkPost.

How to Draw an Owl with React: A Migration Story

not shocked — Fri, 06 Oct 2017 13:05:44 +0000

A few months ago, I wrote about our adventures building a brand new project using React. Our existing SparkPost app was built many moons ago in the ancient framework known as AngularJS. In that earlier post, I said (and I quote), “So what about that 50,000-line Angular app? We won’t be migrating it to React, at least not directly.” Well, we’re migrating it to React. Directly.

It wasn’t that I lied, but let’s say I was a victim of my own lofty idealism. Shortly after that article was published, I started in on my plan to improve the Angular app. After almost two weeks of struggling to get Bower out of the project, I was immediately rethinking all of my ideals and life choices. I’d tried one thing and it was hard. It was time to leave ideals behind.

The Pivot

In all seriousness, we considered a couple of other wild ideas like writing React inside of Angular or migrating everything one page at a time. Ultimately we realized that the fastest and safest way to rebuild our entire app in a brand new framework was to do exactly what I’d publicly ridiculed: direct migration, i.e. a parallel rebuild. The big question was would SparkPost get behind this big idea?

The Pitch

Before I go on, I should explain something very clearly. The chance to completely rewrite your existing app from scratch is a rare occurrence in the world of engineering. If you come away from this article thinking, “I want to totally rewrite OUR app!”, you should know that you’re going to need a Very Good Pitch (and some flexible decision makers willing to carve you a good chunk of time).

That said, there’s tremendous value in the chance to rethink and rebuild your entire user interface as a singular, consistent unit instead of continuing to stack more cards on top of a shakier and shakier house. SparkPost got on board with the rebuild and we started in on a plan.

The Plan

So how do you rebuild an entire web application?

Step 1: Catalog and build a solid foundation of reusable components, design tools, and critical features like authentication.

Step 2: Draw the rest of the [censored] owl.

To be honest, our original plan was only slightly more fleshed out than that, and we’ve shifted and adapted it a lot along the way. Here are a few of the pieces we’ve been most happy about and that have helped us prepare for maximum owl-drawing.

Focus: Our entire application team has always been “full-stack”, bouncing from back end to front end and back. As we’ve grown we’ve recognized a need for a core group of folks who can completely focus on a project like this front-end rebuild. That kind of focus helps us maximize efficiency and create the best foundation possible. It also allows us to be a bridge to the rest of the team about how this new app is built and, eventually, how they can contribute.

Modular design : Jon Ambas, our front-end engineer and design lead (and my main partner in the early days of dreaming up this plan), has been all-in on creating a fully-featured component library to make our work in React a lot simpler and more visually consistent. We now use Matchbox throughout the app and rarely have to think about CSS for anything except custom corner cases.

Openness: Client-side code is already public when it runs in your browser. Therefore, I saw no reason to keep the code private in a repository. We don’t expect anyone will use this code but us (please don’t). It’s already been nice to be able to point to examples of things we’re doing and link directly to the code in GitHub.

Testing and Monitoring: This has been a big goal of the rebuild work we’re doing. Our Selenium-driven functional tests in Angular were taking hours to run and constantly failing until we finally turned them off completely. In the new app, we’re writing lots of tests (ðŸ‘‹ snapshots) but we decided early on that we’ll never be able to predict every bug with test cases. For that reason, we’re also investing in real-time error logging and monitoring. No more waiting for a customer to report an error introduced in a deploy that went out 8 hours ago. Fast stable tests supplemented by error monitoring is our new way forward.

“Preserve and improve” as a redesign strategy: In other words, we’ve preserved the architecture of the app. Leaving most things generally in the places you’d expect to find them. Avoid incredibly drastic changes that would require lots of approvals, user testing, etc. We’ve also taken every opportunity we’ve had to make easy, obvious improvements as we rebuild each part of the app. The result so far feels like an incredibly fresh coat of paint on a familiar base. Which will hopefully help us when we roll the new app out to our existing customers.

Rollout: Incremental rollout, to be specific, is the last big part of our very big plan and probably the least-well-defined at the moment. We know we want to incrementally roll the app out to users (randomly and/or by allowing users to opt-in) so we can test out how it works for real use-cases before we unleash it on everyone. We’re still looking at options for this, but getting the rollout right will be one of the most important parts of our plan overall.

The Point

The takeaway here is that you should be skeptical of anything I say, especially if it sounds good or idealistic. When it comes to something like application development, being practical is almost always better than being pure.

Happy owl-drawing!

The post How to Draw an Owl with React: A Migration Story appeared first on SparkPost.

Using SparkPost Templates with C#

Mike Hillyer — Mon, 02 Oct 2017 13:05:01 +0000

Email Events On Your Terms: Webhooks, Databases, AWS, and more!

Dave Gray — Fri, 29 Sep 2017 13:05:00 +0000

How can Webhooks be easier, and searching event data (AKA Message Events) maybe even greater? We’ll try to answer in this post and open source some code along the way.

Shouting “Show me the data! will earn you funny looks from most people, but not from us here at SparkPost. We are all about the data, both internally as we decide what to build, and externally when we’re delivering event data to you via Webhooks or Message Events.

Tom Cruise may actually want to see the money, but for our customers, data is king. Many of them make heavy use of our Webhooks (push model) to receive batches of event data via HTTP POST. Others prefer to use our Message Eventsendpoint, which is a pull model – you’re querying the same events, although data retention is limited to 10 days, as of this writing.

Now I don’t know about you, but whenever I hear that something is limited, the first thing I want to do is find a way around that limitation. The second thing is to show other people how I did it. In this post, I’m going to show you how to bypass our Message Events data retention limit by rolling your own low-cost queryable event database.

Building Blocks of a Service

The vision here is to ingest batches of event data, delivered by SparkPost’s Webhooks, and then be able to query that data, ideally for free. At least for cheap. Luckily, there are published best practices for doing the first part. One way to keep costs down (at least initially) is to use the AWS free tier, which is the way we’ll go in this post.

First, I’ll walk through the services I ended up using, and then briefly discuss what else I tried along the way, and why that didn’t make the cut. Almost everything in this system is defined and deployed using CloudFormation, along with pieces from the AWS Serverless Application Model (SAM). Under the hood, this uses API Gateway as an HTTP listener, and Node.js Lambda functions to “do stuff when requests are received or in response to other interesting events. More on that later.

According to the best practices linked above, we need to return 200 OK ASAP, before doing any processing of the request body, where the event data is. So we’ll run a Lambdato extract the event data and batch id from the HTTP request and save it to S3. At this point, we’re capturing the data but can’t-do a whole lot with it just yet.

Databases and Event Data

There are all sorts of options out there when it comes to databases. I chose RDS PostgreSQL since it’s a (somewhat) managed service that’s eligible for the AWS free tier. Also, I’m already familiar with it, and had some automatic partitioning code lying around that would be better as open source.

Now seems like a good time to talk about what didn’t make the cut, especially since there were so many interesting options to choose from. The first database-y thing I considered was Athena, which would let us query directly against S3. Right out of the gate, unfortunately, there’s a snag: Athena isn’t eligible for the free tier, it’s priced based on the amount of data scanned by each query. We get a raw JSON feed from the Webhook, so optimizing the storage of that data to be cost-effective to the query would be its own project.

Another database I didn’t use is Dynamo, which would have been super convenient since AWS SAM bakes in support for it. Event data in combination with the types of queries the system needed to support isn’t a great fit for Dynamo though since it doesn’t allow the number of secondary indexes we’d need in order to efficiently support the wide range of queries that Message Events provides. Dynamo would definitely have been the low-stress option. Using RDS meant I had to poke around a bit more in AWS networking land than I had planned to.

Connecting the Data Dots

Our event data is stored in S3, and we’ve chosen a database. Triggers aren’t just for databases, thankfully, and S3 lets you configure Lambda functions to run for various types of events. We’ll fire our next Lambda when a file is created in the bucket that our Webhook listener writes to. It’ll read the batch of event data, and load it into our database, which closes the loop. We’re now asynchronously loading event data sent via Webhook into our database.

The only missing piece now is a way to search for specific types of events. We can implement this using AWS SAM as well, which gives us some nice shortcuts. This last Lambda is essentially a translator between query parameters and SQL. There are quite a few options for query builders in Node, and I picked Squel.js, which was a good balance between simplicity, dependencies, and features.

This system now achieves what it set out to – we’re storing event data provided via Webhook, following best practices, and can query the data using a familiar interface. And if you need to, it’s straightforward to customize by updating the query_events Lambda to add new ways to pull out the data you need, and indexes can be added to the database to make those custom queries faster.

Why Tho, and What Next?

SparkPost sends a lot of data along with our events. For example, transmission metadata lets our customers include things like their own internal user id with each email. Event data such as opens and clicks will now include that user id, making it easier to tie things together.

Because every customer uses features like metadata differently, it’s nigh impossible for us to give everyone exactly the type of search options they’d like. Running your own event database means you’re free to implement custom search parameters. Many of our larger customers already have systems like this, whether it’s a third party tool or something they built themselves. This project aims to lower the barriers to entry, so anyone with a moderate level of familiarity with AWS and the command line can operate their own event database more easily.

There are a few things I’d like to do next, for example, setting up authentication on the various endpoints, since as things are now, they’re open to the public. I discuss a solution to this in the repo, since exposing your customer’s email addresses to the public is a no-no.

I’d also like to perform some volume testing on this system. The free tier RDS database in this setup has 20GB of storage, I’m curious to see how quickly that would fill up. It would also be nice to complete the CloudFormation conversion. Currently, the database is managed separately from the CF stack, and creating the required tables and stored procedures requires punching a hole through the firewall, er, security group. It would be nice to standardize and automate that step as well, instead of requiring mouse clicks in the AWS console.

Thanks for reading! Give us a shout on Twitter, and star, fork or submit a PR on Githubif you enjoyed the post. We’d love to hear about what you build!

The post Email Events On Your Terms: Webhooks, Databases, AWS, and more! appeared first on SparkPost.

Calm your Suppression List Anxieties with Python

SteveT — Fri, 22 Sep 2017 13:05:36 +0000

“It takes many good deeds to build a good reputation, and only one bad one to lose it.”

– Benjamin Franklin

From our handy Getting Started Guide, you know how important it is to bring your suppression list from your old provider with you. Ben Franklin was right – your email reputation will catch a nasty cold if you send to stale, unsubscribed, bounced addresses. This affects whether your messages to your real subscribers are accepted, now and in the future, so it’s best to heed the doctor’s advice.

In this article, we’ll set up an easy-to-use tool to manage your suppression lists. If you want some more background on the “what and “why of suppressions, this article is a good starting point.

Scrubbing up

The exported suppression lists we see coming from old providers are often dirty. Duplicate entries, invalid entries with more than one @ sign, invalid characters, telephone numbers instead of email addresses, you name it. We’ve seen files with weird characters in various obscure international alphabets. This might make you consider just amputating those lists, but we’ll explore techniques to preserve as much of them as we can.

Lists can be large, reaching nearly a million entries. Working with small blocks manually is going to take forever. We are going to need a robot surgeon!

Plan for treatment

Let’s set out our needs, and translate them into design goals.

Make it easy to get started and simple to use.
Make best efforts to understand your file format, even if it appears to contain weird characters.
Check and upload any size of a list without manual work.
Check the input files up front, with helpful warnings as we go.
Checks should be thorough and fast. If there are faults, we want to know exactly where they are in the file, and what’s wrong. Specifically, we need to:
- Ensure email addresses are well-formed (i.e. follow the RFCs)
- Check the other field values, such as transactional / non_transactional flags.
A “check everything but don’t change anything mode, to make it easy to find and fix faulty input data.
Allow retrieval of your whole suppression list back from SparkPost, or select time-bounded portions.
- Have time-zone awareness, while accepting times in your locale. In particular, remember that start and end times could fall on either side of a daylight savings time change.
- Keep it simple. The API supports searching by domain, source, type, description etc – however, that can be done by filtering the retrieved file afterward. If you want these features, raise an issue on the Git repository and we’ll look at it.
Work across both master account and subaccounts.
Make it easy to supply defaults for missing/optional file information.

That leads us on to making a tool with the following options:

Check the format of your files (prior to import). Always a good idea to bring your suppressions with you.
Update your suppression list in SparkPost (i.e. create if your list is currently empty).
Retrieve your suppression list from SparkPost, for example, if you want to get suppressions back into your upstream campaign management tool.
Delete your suppression list from SparkPost, i.e. clean your suppression list out. Maybe you uploaded some entries by mistake. We hope that’s a rare use-case, but it’s there for you.

Time to operate

sparkySuppress is a tool written in Python to help you manage your suppression list. The Github repo includes a comprehensive README file. There’s some help with getting Python 3 installed here if you need it.

You can configure sparkySuppress with the sparkpost.ini file, which is used to set up things you change infrequently, such as your API key, timezone, batch sizes and so on. You can leave everything except API key set to default if you like.

Email addresses from input files are checked as we go, using the excellent email_validator library. We use this to give comprehensive reporting in case of faulty addresses, for example:

Line 2 ! bad@email@address.com The email address is not valid. It must have exactly one @-sign.
Line 3 ! invalid.email@~{}gmail.com The domain name ~{}gmail.com contains invalid characters (Codepoint U+007E not allowed at position 1 in '~{}gmail.com').

The ! marks the entry as having an error. We’ll mark entries that have recoverable problems with a warning w like this:

Line 1 w need valid transactional & non_transactional flags: {'recipient': 'test.recip@gmail.com'}

An excellent character

Text files are not as simple as they appear! Unusual file character encoding can be an obstacle, particularly when you don’t have control over how the suppression list export was created in the first place.

UTF-8 is the most modern and capable encoding, but some systems may not use it. Output files exported from some older versions of Excel will be in Latin-1 for example, rather than UTF-8.

The FileCharacterEncodings setting in the sparkpost.ini provides an easy way to control how your input file will be processed. The tool reads your file using each encoding in turn, and if it finds anomalies, will try in the next encoding and so on. So if you have:

FileCharacterEncodings=utf-8,utf-16,ascii,latin-1

you will see the tool trying each encoding until it finds one that reads the whole file without error. You can select any of the standard encodings shown here.

$ ./sparkySuppress.py check klist-1.csv
Trying file klist-1.csv with encoding: utf-8
        Near line 1125 'utf-8' codec can't decode byte 0x9a in position 7198: invalid start byte
Trying file klist-1.csv with encoding: utf-16
        Near line 1 UTF-16 stream does not start with BOM
Trying file klist-1.csv with encoding: ascii
        Near line 1125 'ascii' codec can't decode byte 0x9a in position 7198: ordinal not in range(128)
Trying file klist-1.csv with encoding: latin-1
        File reads OK.

Lines in file: 8496

Your first encoding in the list is used when you’re retrieving entries back from SparkPost into a file.

A good performance

Delete is a bit special – it uses multi-threading because deletes have to be done one per call. Update and retrieve work fast when single-threaded, as each call handles a batch. You should experience good performance with the default batch size and thread settings, but you can tweak them if needed.

Practicing your medicine

In case you don’t have data from your old provider yet, here’s a tool for creating suppression lists that you can use to create a dummy file to practice on.

That’s about it! You are now a skilled suppression list surgeon. You’ll soon have your campaigns in excellent shape.

And finally…

If you are exploring this tool and want to give the author feedback, you’re welcome to visit our Community Slack channel– there’s a channel just for Python, #python. Alternatively, open a Github project issue or pull-request.

If you don’t like Python (whut?) there are some lower-level command-line SparkPost projects that provide a thin “wrapper over the API and can be used to manipulate suppression lists. Check out Node.js and Go and if you want to know more about the API and UI for suppression lists, here’s a good place to start. There’s also a node.js tool to retrieve your list back again from SparkPost for checking.

If you prefer point-and-click, the SparkPost user interface has a built-in Lists/Suppressions upload feature. This gives you a nice example template and is ideal when you have perfectly formatted files that aren’t too large, with a maximum of 10,000 recipients per file.

The post Calm your Suppression List Anxieties with Python appeared first on SparkPost.

Archiving Emails: A How-To Guide for Tracking Sent Mail

jeff-goldstein — Wed, 20 Sep 2017 13:05:36 +0000

Not every email sender needs the ability to archive their emails, but for some segments, like Financial or Health Care, tracking sent emails is often a requirement. So I decided to create a sample program that does just that. As with most of my blog posts, I have a corresponding Github repository you can take a look at if you’re interested.

You are more than welcome to use this repository as a starting point to your project or modify and update the repository for others to leverage.

There are a couple of different ways to support archiving. For this post, I decided to focus on just one method, the “Archive function within the SparkPost SMTP call. Another approach would be to store the template and the data needed to merge into that template. I may address that approach with some sample code in a future post.

But let’s get back to the approach that we’re covering today. Here is a graphic of the three distinct steps in this process; starting with the email creation down to the archiving of the email body and supporting components.

At a 32,000 foot level, those steps manifest into three development steps:

Set up an inbound relay Webhook within SparkPost. Relay Webhooks are a way to instruct SparkPost to accept inbound emails on your behalf and forward it to you over HTTP for your own consumption.
Create a tool that will receive the Relay Webhook information (the inbound email) in the form of a JSON payload.
Send an email via the SMTP protocol adding the SparkPost SMTP API header. The SMTP API header will contain the email address of that you will use for your archiving domain.

Step 1 and 2 share the “chicken or the egg syndrome. They’re so intertwined you can’t do one without the other so it may not matter which you do first, but I’m going address the setup first because you can’t do full unit testing without getting DNS and the Webhooks in place.

“Enabling Inbound Email Relaying & Relay Webhooks will walk you through the Relay Webhook setup process, but I have a few helpful hints to add. I’ll walk through the steps of the document at a high level, then add some examples that you may find useful.

Set up an Inbox Relay Webhook

The first step in the document mentioned above is to set up mx records to have the email for a given domain forwarded to SparkPost for processing. The second step in the aforementioned document has you creating an Inbound Domain using the SparkPost inbound-domain RESTful API. The body of the call is straightforward with one key/value pair, with the key named “domain”. For my example, the value matches my MX entry for the domain named geekwithapersonality.com (no judgements on the domain name, please). Thus the body of the API call is simply:

{
    "domain": "geekwithapersonality.com"
}

There is no UI functionality within SparkPost to set this up, so you must use a client like Postman or an application like CURL. SparkPost does have a Postman collection that you can leverage, which makes this step extremely easy. Next, you need to tie that inbound-domain to where your application is running. As a footnote, there can only be one relay-webhook application per inbound-domain. In this step you will call the relay-webhooks RESTful API, with a body similar to this one:

{
  "name": "Replies Webhook",
  "target": "http://www.geekwithapersonality.com/cgi-bin/inbox-relay.php",
  "auth_token": "",
  "match": {
    "protocol": "SMTP",
    "domain": "geekwithapersonality.com"
    }
}

Notice how the domain key/value pair uses the same domain value, geekwithapersonality, as the entry employed in the RESTful API inbound-domain call. This step is the one that ties the two together. So any emails that go to the domain geekwithapersonality.com will be directed to SparkPost, then translated into a JSON structure for processing. Again, there is no SparkPost UI component for this step, so you need to use something like Postman or CURL to execute this step. Unless you’re an ESP processing inbound messages for many clients, this is typically a very rare task.

Create a Collector to Process and Store the Archive Email

Even though the code is sitting in Github, I’ll describe the components and the logic for each piece. Overall, this code was fairly straightforward to write. I decide to archive the following components of each email in separate files:

All the headers used to send the email. Because the archive email went to a different location than the original email, these headers are NOT 100% equivalent to the original email, but they are close.
The full JSON structure.
The text body with the original To email address, along with the From and Subject appended to the top of the text file.
The HTML body with the original To email address along with the From and Subject appended to the top of the HTML file.
The rfc822 compliant email. This allows an application to grab the whole file and resend it if necessary.

In this sample code, I placed each group of files in their own directory and logged each Archive email processed. Beyond placing the data into a directory, I am NOT archiving these items into something more enterprise level like S3 or HP Autonomy. I’ll leave that process for another time, but there are so many CMS or archiving tools out there that creating a dozen different hooks into archiving tools would only scratch the surface. I also want the process to be as fast as possible, so I think the archiving tool should only be loosely coupled and either run off an index/queue/stack of processed items or perhaps the archiving tool will simply look for new directories and relocate them to a more permanent location.

In order to create a unique directory for each archive, I created directories with a combination of the name of the original email recipient address along with a timestamp.

The structure of the inbound webhook structure may change over time but at the moment the current structure can be found within the Relay Webhooks Example Payloads documentation of SparkPost.

Here are the json values that I grabbed in order to either store them separately or process them further:

msys/relay_message/content/to: This is the email address for the original target of the content.
msys/relay_message/rcpt_to: This is the email address used to send the archive message.
msys/relay_message/friendly_from: This is the from address presented in the original email address.
msys/relay_message/content/subject: Subject of the email.
msys/relay_message/content/headers: Once pulled, I format these into a nice html (ok, semi-nice) format for storage and reference.
msys/relay_message/content/html: The HTML body of the email.
msys/relay_message/content/text: The text body of the email.
msys/relay_message/content/email_rfc822: The full email body in rfc822 standard. It’s save as an eml file which allows most email clients to open directly.

Once I get all of this data, I process the headers into an html format. From there, I add the To, From and Subject to the HTML and Text bodies and save everything. Finally, I create a log entry for that specific entry.

Send an Archived Copy of the Email

The Archiving functionality can only be initiated by using the SMTP API (which, by the way, is a horribly named industry standard for an added header for your SMTP call). For SparkPost, the SMTP API is actually a JSON formatted header called X-MSYS-API. In that header, you are able to set items like campaign id, metadata, IP Pools, CC and BCC email addresses in addition to a list of Archiving email addresses. Here is a sample X-MSYS-API header:

X-MSYS-API:
{
  "campaign_id": "Jeffs BBQ Campaign",
  "metadata" : {
            "home_address_region": “West”,
            "smoker": "SMB"
  },
  "archive": [
             “archive@mail.geekwithapersonality.com”
  ],
"options" : {
            "open_tracking": false,
            "click_tracking": true,
            "transactional": false,
            "ip_pool": "sp_shared",
}
}

This isn’t a complete sample with all of the options available, but illustrates what a typical header looks like. Notice that the original recipient’s email address isn’t in the X-MSYS-API header, but you’ll get it in the Reply Webhook.

In the Github repository, I have added a test PHP application that calls SMTP by using the PHPMailer framework. While calling SMTP and adding in the X-MSYS-API header with an archive email address is probably the best approach for thorough unit testing, it isn’t exactly necessary. Any email sent to the domain you are targeting will get processed the same way as an archive email. So I did most of my unit testing by sending an email to my geekwithapersonality domain, either by a Postman transmission SparkPost API call or simply sending an email via my MAC email client. Neither is really an archive, but the inbox relay doesn’t care and treats all emails the same.

Voila! Archiving Emails is Simple!

Those are the three steps to set up archiving. If you are using SMTP to send emails to SparkPost, you’re probably already using the X-MSYS-API headers, so just adding the archive entry will be easy for you. Then all you need to do is point email for your archiving domain to SparkPost and create a simple app to capture the data.

I hope this was helpful – if you have any questions on archiving emails, email templates, etc, please feel free to reach out to me directly.

Happy Sending!

The post Archiving Emails: A How-To Guide for Tracking Sent Mail appeared first on SparkPost.