Forem: SteveT

Calm your Suppression List Anxieties with Python

SteveT — Fri, 22 Sep 2017 13:05:36 +0000

“It takes many good deeds to build a good reputation, and only one bad one to lose it.”

– Benjamin Franklin

From our handy Getting Started Guide, you know how important it is to bring your suppression list from your old provider with you. Ben Franklin was right – your email reputation will catch a nasty cold if you send to stale, unsubscribed, bounced addresses. This affects whether your messages to your real subscribers are accepted, now and in the future, so it’s best to heed the doctor’s advice.

In this article, we’ll set up an easy-to-use tool to manage your suppression lists. If you want some more background on the “what and “why of suppressions, this article is a good starting point.

Scrubbing up

The exported suppression lists we see coming from old providers are often dirty. Duplicate entries, invalid entries with more than one @ sign, invalid characters, telephone numbers instead of email addresses, you name it. We’ve seen files with weird characters in various obscure international alphabets. This might make you consider just amputating those lists, but we’ll explore techniques to preserve as much of them as we can.

Lists can be large, reaching nearly a million entries. Working with small blocks manually is going to take forever. We are going to need a robot surgeon!

Plan for treatment

Let’s set out our needs, and translate them into design goals.

Make it easy to get started and simple to use.
Make best efforts to understand your file format, even if it appears to contain weird characters.
Check and upload any size of a list without manual work.
Check the input files up front, with helpful warnings as we go.
Checks should be thorough and fast. If there are faults, we want to know exactly where they are in the file, and what’s wrong. Specifically, we need to:
- Ensure email addresses are well-formed (i.e. follow the RFCs)
- Check the other field values, such as transactional / non_transactional flags.
A “check everything but don’t change anything mode, to make it easy to find and fix faulty input data.
Allow retrieval of your whole suppression list back from SparkPost, or select time-bounded portions.
- Have time-zone awareness, while accepting times in your locale. In particular, remember that start and end times could fall on either side of a daylight savings time change.
- Keep it simple. The API supports searching by domain, source, type, description etc – however, that can be done by filtering the retrieved file afterward. If you want these features, raise an issue on the Git repository and we’ll look at it.
Work across both master account and subaccounts.
Make it easy to supply defaults for missing/optional file information.

That leads us on to making a tool with the following options:

Check the format of your files (prior to import). Always a good idea to bring your suppressions with you.
Update your suppression list in SparkPost (i.e. create if your list is currently empty).
Retrieve your suppression list from SparkPost, for example, if you want to get suppressions back into your upstream campaign management tool.
Delete your suppression list from SparkPost, i.e. clean your suppression list out. Maybe you uploaded some entries by mistake. We hope that’s a rare use-case, but it’s there for you.

Time to operate

sparkySuppress is a tool written in Python to help you manage your suppression list. The Github repo includes a comprehensive README file. There’s some help with getting Python 3 installed here if you need it.

You can configure sparkySuppress with the sparkpost.ini file, which is used to set up things you change infrequently, such as your API key, timezone, batch sizes and so on. You can leave everything except API key set to default if you like.

Email addresses from input files are checked as we go, using the excellent email_validator library. We use this to give comprehensive reporting in case of faulty addresses, for example:

Line 2 ! bad@email@address.com The email address is not valid. It must have exactly one @-sign.
Line 3 ! invalid.email@~{}gmail.com The domain name ~{}gmail.com contains invalid characters (Codepoint U+007E not allowed at position 1 in '~{}gmail.com').

The ! marks the entry as having an error. We’ll mark entries that have recoverable problems with a warning w like this:

Line 1 w need valid transactional & non_transactional flags: {'recipient': 'test.recip@gmail.com'}

An excellent character

Text files are not as simple as they appear! Unusual file character encoding can be an obstacle, particularly when you don’t have control over how the suppression list export was created in the first place.

UTF-8 is the most modern and capable encoding, but some systems may not use it. Output files exported from some older versions of Excel will be in Latin-1 for example, rather than UTF-8.

The FileCharacterEncodings setting in the sparkpost.ini provides an easy way to control how your input file will be processed. The tool reads your file using each encoding in turn, and if it finds anomalies, will try in the next encoding and so on. So if you have:

FileCharacterEncodings=utf-8,utf-16,ascii,latin-1

you will see the tool trying each encoding until it finds one that reads the whole file without error. You can select any of the standard encodings shown here.

$ ./sparkySuppress.py check klist-1.csv
Trying file klist-1.csv with encoding: utf-8
        Near line 1125 'utf-8' codec can't decode byte 0x9a in position 7198: invalid start byte
Trying file klist-1.csv with encoding: utf-16
        Near line 1 UTF-16 stream does not start with BOM
Trying file klist-1.csv with encoding: ascii
        Near line 1125 'ascii' codec can't decode byte 0x9a in position 7198: ordinal not in range(128)
Trying file klist-1.csv with encoding: latin-1
        File reads OK.

Lines in file: 8496

Your first encoding in the list is used when you’re retrieving entries back from SparkPost into a file.

A good performance

Delete is a bit special – it uses multi-threading because deletes have to be done one per call. Update and retrieve work fast when single-threaded, as each call handles a batch. You should experience good performance with the default batch size and thread settings, but you can tweak them if needed.

Practicing your medicine

In case you don’t have data from your old provider yet, here’s a tool for creating suppression lists that you can use to create a dummy file to practice on.

That’s about it! You are now a skilled suppression list surgeon. You’ll soon have your campaigns in excellent shape.

And finally…

If you are exploring this tool and want to give the author feedback, you’re welcome to visit our Community Slack channel– there’s a channel just for Python, #python. Alternatively, open a Github project issue or pull-request.

If you don’t like Python (whut?) there are some lower-level command-line SparkPost projects that provide a thin “wrapper over the API and can be used to manipulate suppression lists. Check out Node.js and Go and if you want to know more about the API and UI for suppression lists, here’s a good place to start. There’s also a node.js tool to retrieve your list back again from SparkPost for checking.

If you prefer point-and-click, the SparkPost user interface has a built-in Lists/Suppressions upload feature. This gives you a nice example template and is ideal when you have perfectly formatted files that aren’t too large, with a maximum of 10,000 recipients per file.

The post Calm your Suppression List Anxieties with Python appeared first on SparkPost.

Event Data Made Easy with SparkPost Message Events

SteveT — Fri, 07 Jul 2017 13:01:45 +0000

We love it when developers use SparkPost webhooks to build awesome responsive services. Webhooks are great when you need real-time feedback on what your customers are doing with their messages. They work on a “push model – you create a microservice to handle the event stream.

Did you know that SparkPost also supports a “pull model Message Events API that enables you to download your event data for up to ten days afterwards? This can be particularly useful in situations such as:

You’re finding it difficult to create and maintain a production-ready microservice. For example, your corporate IT policy might make it difficult for you to have open ports permanently listening;
You’re familiar with batch type operations and running periodic workloads, so you don’t need real-time message events;
You’re a convinced webhooks fan, but you’re investigating issues with your almost-working webhooks receiver microservice, and want a reference copy of those events to compare.

If this sounds like your situation, you’re in the right place! Now let’s walk through setting up a really simple tool to get those events.

Design goals

Let’s start by setting out the requirements for this project, then translate them into design goals for the tool:

You want it easy to customize without programming.
SparkPost events are a rich source of data, but some event-types and event properties might not be relevant to you. Being selective gives smaller output file sizes, which is a good thing, right?
Speaking of output files, you want event data in the commonly-used csv file format. While programmers love JSON, CSV is easier for non-technical users (and results in smaller files).
You want to set up your SparkPost account credentials and other basic information once and once only, without having to redo them each time it’s used. Having to remember that stuff is boring.
You need flexibility on the event date/time ranges of interest.
You want to set up your local time-zone once, and then work in that zone, not converting values manually to UTC time. Of course, if you really want to work in UTC, because your other server logs are all UTC, then “make it so.”
Provide some meaningful comfort reporting on your screen. Extracting millions of events could take some time to run. I want to know it’s working.

Events, dear programmer, events …

Firstly, you’ll need Python 3 and git installed and working on your system. For Linux, a simple procedure can be found in our previous blog post. It’s really this easy:

sudo su -
yum update -y
yum install -y python35
yum install -y wget
wget https://bootstrap.pypa.io/get-pip.py
python3 get-pip.py 
yum install -y git

For other platforms, this is a good starting point to get the latest Python download; there are many good tutorials out there on how to install.

Then get the sparkyEvents code from Github using:

$ git clone https://github.com/tuck1s/sparkyEvents.git
Initialized empty Git repository in /home/stuck/sparkyEvents/.git/
remote: Counting objects: 32, done.
remote: Compressing objects: 100% (22/22), done.
remote: Total 32 (delta 7), reused 28 (delta 5), pack-reused 0
Unpacking objects: 100% (32/32), done.

$ cd sparkyEvents

We’re the knights who say “.ini”

Set up a sparkpost.ini file as per the example in the Github README file here.

Replace <YOUR API KEY> with ~~a shrubbery~~ your specific, private API key.

Host is only needed for SparkPost Enterprise service usage; you can omit for sparkpost.com.

Events is a list, as per SparkPost Event Types; omit the line, or assign it blank, to select all event types.

Properties can be any of the SparkPost Event Properties. Definitions can split over lines using indentation, as per Python .ini file structure, which is handy as there are nearly sixty different properties. You can select just those properties you want, rather than everything; this keeps the output file to just the information you want.

Timezone can be configured to suit your locale. It’s used by SparkPost to interpret the event time range from_time and to_time that you give in command-line parameters. If you leave this blank, SparkPost will default to using UTC.

If you run the tool without any command-line parameters, it prints usage:

$ ./sparkyEvents.py 

NAME
   ./sparkyEvents.py
   Simple command-line tool to retrieve SparkPost message events into a .CSV file.

SYNOPSIS
  ./sparkyEvents.py outfile.csv from_time to_time

MANDATORY PARAMETERS
    outfile.csv output filename, must be writeable. Records included are specified in the .ini file.
    from_time
    to_time Format YYYY-MM-DDTHH:MM

from_time and to_time are inclusive, so for example if you want a full day of events, use time T00:00 to T23:59.

Here’s a typical run of the tool, extracting just over 18 million events. This run took a little over two hours to complete.

./sparkyEvents.py outfile.csv 2017-06-04T00:00 2017-06-04T23:59

SparkPost events from 2017-06-04T00:00 to 2017-06-04T23:59 America/New_York to outfile.csv
Events: <all>
Properties: ['timestamp', 'type', 'event_id', 'friendly_from', 'mailfrom', 'raw_rcpt_to', 'message_id', 'template_id', 'campaign_id', 'subaccount_id', 'subject', 'bounce_class', 'raw_reason', 'rcpt_meta', 'rcpt_tags']
Total events to fetch: 18537125
Page 1: got 10000 events in 5.958 seconds
Page 2: got 10000 events in 5.682 seconds
Page 3: got 10000 events in 5.438 seconds
Page 4: got 10000 events in 6.347 seconds
:
:

That’s it! You’re ready to use the tool now. Want to take a peek inside the code? Keep reading!

Inside the code

Getting events via the SparkPost API

The SparkPost Python library doesn’t yet have built-in support for the message-events endpoint. In practice the Python requests library is all we need. It provides inbuilt abstractions for handling JSON data, response status codes etc and is generally a thing of beauty.

One thing we need to take care of here is that the message-events endpoint is rate-limited. If we make too many requests, SparkPost replies with a 429 response code. We play nicely using the following function, which sleeps for a set time, then retries:

def getMessageEvents(uri, apiKey, params):
   try:
       path = uri + '/api/v1/message-events'
       h = {'Authorization': apiKey, 'Accept': 'application/json'}

       moreToDo = True
       while moreToDo:
           response = requests.get(path, timeout=T, headers=h, params=params)

           # Handle possible 'too many requests' error inside this module
           if response.status_code == 200:
               return response.json()
           elif response.status_code == 429:
               if response.json()['errors'][0]['message'] == 'Too many requests':
                   snooze = 120
                   print('.. pausing', snooze, 'seconds for rate-limiting')
                   time.sleep(snooze)
                   continue # try again
           else:
               print('Error:', response.status_code, ':', response.text)
               return None

   except ConnectionError as err:
       print('error code', err.status_code)
       exit(1)

Practically, when using event batches of 10000 I didn’t experience any rate-limiting responses even on a fast client. I had to deliberately set smaller batch sizes during testing, so you may not see rate-limiting occur for you in practice.

Selecting the Event Properties

SparkPost’s events have nearly sixty possible properties. Users may not want all of them, so let’s select those via the sparkpost.ini file. As with other Python projects, the excellent ConfigParser library does most of the work here. It supports a nice multi-line feature:

“Values can also span multiple lines, as long as they are indented deeper than the first line of the value.”

We can read the properties (applying a sensible default if it’s absent), remove any newline or carriage-return characters, and convert to a Python list in just three lines:

# If the fields are not specified, default to a basic few
properties = cfg.get('Properties', 'timestamp,type')
properties = properties.replace('\r', '').replace('\n', '') # Strip newline and CR
fList = properties.split(',')

Writing to file

The Python csv library enables us to create the output file, complete with the required header row field names, based on the fList we’ve just read:

fh = csv.DictWriter(outfile, fieldnames=fList, restval='', extrasaction='ignore')
fh.writeheader()

Using the DictWriter class, data is automatically matched to the field names in the output file, and written in the expected order on each line. restval=" ensures we emit blanks for absent data, since not all events have every property.
extrasaction=’ignore’ ensures that we skip extra data we don’t want.

for i in res['results']:
   fh.writerow(i) # Write out results as CSV rows in the output file

That’s pretty much everything of note. The tool is less than 150 lines of actual code.

You’re the Master of Events!

So that’s it! You can now download squillions of events from SparkPost, and can customize the output files you’re getting. You’re now the master of events!

This post was originally published on sparkpost.com

Charm your SparkPost Recipient and Suppression Lists with Python

SteveT — Thu, 08 Jun 2017 02:14:57 +0000

When developing code that sends email to your customers, it’s smart to try things out on a test list first. Using our sink-domain service helps you avoid negative impact to your reputation during tests. Next you’ll want to check that your code is working at scale on a realistic sized recipient list and flush out any code performance issues… but how?

You could use Microsoft Excel to put together .csv recipient-list files for you, but there are practical limitations, and it’s slow. You’ll be using fields like “substitution_data that require JSON encoding, and Excel doesn’t help you with those. Performance-wise, anything more than a few hundred rows in Excel is going to get cumbersome.

What we need is a tool that will generate realistic looking test lists; preferably of any size, up to hundreds of thousands of recipients, with safe sink-domain addresses. Excel would be really slow at doing that – we can do much better with some gentle programming.

The second requirement, perhaps less obvious, is testing your uploads to SparkPost’s built-in suppression-list functionality. It’s good practice to upload the suppressed addresses from your previous provider before mailing – see, for example our Migration Guides. You might need to rehearse a migration without using your real suppression addresses. Perhaps you don’t have easy access to them right now, because your old provider is playing awkward and doesn’t have a nice API. Luckily, with very little extra code, we can also make this tool generate “practice suppression lists.

You’re on my list

CSV files have a “header in line 1 of the file, giving the names of each field. Handy hint: you can get an example file, including the header line, directly from SparkPost, using the “Download a Recipient List CSV template button right here:

The SparkPost recipient-list .csv format looks like this:

email,name,return_path,metadata,substitution_data,tags
recipient@example.com,Example Recipient,reply@example.com,"{""foo"": ""bar""}","{""member"": ""Platinum"", ""region"": ""US""}","[""test"", ""example""]"
recipient2@example.com,Jake,reply@example.com,"{""foo"": ""bar""}","{""member"": ""Platinum"", ""region"": ""US""}","[""test"", ""example""]"

The metadata, substitution_data, and tags fields can carry pretty much anything you want.

SparkPost's suppression list .csv format is equally fun, and looks like this:

recipient,transactional,non_transactional,description,subaccount_id
anon11779856@demo.sink.sparkpostmail.com,true,true,Example data import,0

Let’s have an argument

Some command-line arguments would be nice, so we can change the lists we’re generating. Here’s the arguments we’ll accept, which translate nicely into design goals for this project:

A flag to say whether we’re generating a recipient list or a suppression list
How many records we want (make it optional – say 10 as a default)
A recipient domain to generate records for (optional – default as something safe, such as demo.sink.sparkpostmail.com).

Downloading and using the tool

Firstly, you’ll need python, pip, and git installed. If you don’t already have them, there are some simple instructions to in my previous blogpost. Then we use git clone to download the project. The external package names is needed, we can install that using pip3.

$ git clone https://github.com/tuck1s/gen-SparkPost-Lists-python.git
Initialized empty Git repository in /home/stuck/gen-SparkPost-Lists-python/.git/
remote: Counting objects: 32, done.
remote: Total 32 (delta 0), reused 0 (delta 0), pack-reused 32
Unpacking objects: 100% (32/32), done.

$ sudo pip3 install names
Collecting names
Installing collected packages: names
Successfully installed names-0.3.0

$ cd gen-SparkPost-Lists-python/
$ ./gen-sparkpost-lists.py recip 10

After that final command, you should see the list output to the screen. If you want to direct it into a file, you just use >, like this:

$ ./gen-sparkpost-lists.py recip 10 > mylist.csv

That’s all there is to it! If you run the tool with no arguments, it gives some guidance on usage:

$ ./gen-sparkpost-lists.py 

NAME
   ./gen-sparkpost-lists.py
   Generate a random, SparkPost-compatible Recipient- or Suppression-List for .CSV import.

SYNOPSIS
  ./gen-sparkpost-lists.py recip|supp|help [count [domain]]

OPTIONAL PARAMETERS
    count = number of records to generate (default 10)
    domain = recipient domain to generate records for (default demo.sink.sparkpostmail.com)
[stuck@ip-172-31-20-126 gen-SparkPost-Lists-python]$

Inside the code – special snowflakes

Here’s the kind of data we want to generate for our test recipient-lists.

email,name,return_path,metadata,substitution_data,tags
anon13061346@demo.sink.sparkpostmail.com,Teddy Lanier,bounce@demo.sink.sparkpostmail.com,"{""custID"": 3156295}","{""memberType"": ""bronze"", ""state"": ""KY""}","[""gwen"", ""bacon"", ""hass"", ""fuerte""]"
anon94133309@demo.sink.sparkpostmail.com,Heriberto Pennell,bounce@demo.sink.sparkpostmail.com,"{""custID"": 78804336}","{""memberType"": ""platinum"", ""state"": ""MT""}","[""bacon""]"
anon14982287@demo.sink.sparkpostmail.com,Terry Smialek,bounce@demo.sink.sparkpostmail.com,"{""custID"": 16745544}","{""memberType"": ""platinum"", ""state"": ""WA""}","[""bacon""]"

The metadata, substitution data and tags are from our example company, Avocado Industries. Let’s pick a line of that apart, and hide the double-quotes ”” so we can see it more clearly:

Metadata:

{
    "custID": 3156295
}

Substitution_data:

{
    "memberType": "bronze", 
    "state": "KY"
}

Tags (these are types of avocado, by the way!)

[
    "gwen", 
    "bacon", 
    "hass", 
    "fuerte"
]

We want each recipient email address to be unique, so that when imported into SparkPost, the list is exactly the asked-for length. Sounds easy – we can just use a random number generator to produce an ID like the ones shown above. The catch is that random functions can give the same ID during a run, and on a long run that is quite likely to happen. We need to prevent that, eliminating duplicate addresses as we go.

Python provides a nice set() datatype we can use that’s relatively efficient:

uniqFlags = set()
:
:
:
dataRow.append(randomRecip(domain, numDigits, uniqFlags))

We’ve created a global set object, uniqFlags which will acts as a scratchpad for random numbers we’ve already used – and pass it into the function randomRecip in the usual way.

# Need to treat ensureUnique only with mutating list methods such as 'add', so the updated value is returned to the calling function
def randomRecip(domain, digits, ensureUnique):
   taken = True
   while taken:
       localpartnum = random.randrange(0, 10**digits)
       taken = localpartnum in ensureUnique                    # If already had this number, then pick another one
   ensureUnique.add(localpartnum)
   return 'anon'+str(localpartnum).zfill(digits)+'@'+domain    # Pad the number out to a fixed length of digits

Python allows changes made to ensureUnique inside the function using the .add() method to show up in the global data – in other words, the parameter is called by reference.

For the other fields, picking random values from a small set of options is easy. For example:

def randomMemberType():
   tiers = ['bronze', 'silver', 'gold', 'platinum']
   return random.choice(tiers)

We can pick randomized US postal states in exactly the same way. The custID field is just a naive random number (so it might repeat). I’ve left that as an exercise for the reader to change, if you wish (hint: use another set).

For the tags field – we would like to assign somewhere between none and all of the possible Avocado varieties to each person; and for good measure we’ll randomize the order of those tags too. Here’s how we do that:

# Compose a random number of tags, in random shuffled order, from a preset short list.
# List of varieties is taken from: http://www.californiaavocado.com/how-tos/avocado-varieties.aspx
def randomTags():
   avocadoVarieties = ['bacon', 'fuerte', 'gwen', 'hass', 'lamb hass', 'pinkerton', 'reed', 'zutano']
   k = random.randrange(0, len(avocadoVarieties))
   t = avocadoVarieties[0:k]
   random.shuffle(t)
   return json.dumps(t)

What’s in a name?

SparkPost recipient-list format supports a text name field, as well as an email address. It would be nice to have realistic-looking data for that. Fortunately, someone’s already built a package that uses the 1990 US Census data, that turns out to be easy to leverage. You’ll recall we installed the names package earlier.

# Prepare a cache of actual, random names - this enables long lists to be built faster
nameList = []
for i in range(100):
   nameList.append( { 'first': names.get_first_name(), 'last': names.get_last_name() } )

The names library calls take a little while to run, which could really slow down our list creation. Rather than calling the function for every row, the above code builds a nameList of first and last names, that we can choose from later. For our purposes, it’s OK to have text names that might repeat (i.e. more than one Jane Doe) – only the email addresses need be strictly unique.

The choice of 100 in the above code is fairly arbitrary – it will give us “enough randomness when picking a random first-name and separately picking a random last-name.

Full speed ahead

A quick local test shows the tool can create a 100,000 entry recipient list – about 20MB – in seven seconds, so you shouldn’t have to wait long even for large outputs.

The output of the tool is just a text stream, so you can redirect it into a file using >, like this:

$ ./gen-sparkpost-lists.py recip 100000 >mylist.csv

You can also pipe it into other tools. CSVkit is great for this – you can choose which columns to filter on (with csvcut), display (with csvlook) etc. For example, you could easily create a file with just email, name, and substitution_data, and view it:

$ ./gen-sparkpost-lists.py recip 10 | csvcut -c1,2,5 | csvlook
|-----------------------------------------------------------------------------------------------------------|
|  email                                    | name             | substitution_data                          |
|-----------------------------------------------------------------------------------------------------------|
|  anon78856278@demo.sink.sparkpostmail.com | Linda Erdmann    | {"memberType": "gold", "state": "MN"}      |
|  anon27569456@demo.sink.sparkpostmail.com | James Glenn      | {"memberType": "platinum", "state": "PA"}  |
|  anon82026154@demo.sink.sparkpostmail.com | Mark Morris      | {"memberType": "bronze", "state": "NC"}    |
|  anon99410317@demo.sink.sparkpostmail.com | Daniel Baldwin   | {"memberType": "platinum", "state": "TX"}  |
|  anon40941199@demo.sink.sparkpostmail.com | Cammie Cornell   | {"memberType": "platinum", "state": "TX"}  |
|  anon81569289@demo.sink.sparkpostmail.com | Mary Pearce      | {"memberType": "bronze", "state": "NC"}    |
|  anon87708262@demo.sink.sparkpostmail.com | Angella Souphom  | {"memberType": "bronze", "state": "NV"}    |
|  anon74282988@demo.sink.sparkpostmail.com | Antonio Erdmann  | {"memberType": "platinum", "state": "MD"}  |
|  anon48883171@demo.sink.sparkpostmail.com | Randolph Maranto | {"memberType": "bronze", "state": "MA"}    |
|  anon17719693@demo.sink.sparkpostmail.com | Jack Hudson      | {"memberType": "silver", "state": "CA"}    |
|-----------------------------------------------------------------------------------------------------------|

And finally …

Download the code and make your own test recipient and suppression lists. Leave a comment below to let me know how you’ve used it and what other hacks you’d like to see.

This post was originally posted on SparkPost.

Sending Scheduled Mailings Simply with SparkPost

SteveT — Wed, 31 May 2017 21:48:36 +0000

Do you need to send batches of emails, synchronised to go at a set time? Are you unsure whether to develop your own campaign management tools, or buy off-the-shelf? Have you been through our Getting Started Guide, and are inspired to send your first campaign, but are feeling a bit nervous about writing your own code?

A customer of ours recently had this same need - sending out email batches of a few million emails each morning. They looked at some great fully-featured campaign management tools from SparkPost partners such as Iterable, Ongage, and Cordial that cover this need, and lots more besides. When you have many different campaign types, complex â€˜customer journey’ campaigns, and integrated WYSIWYG editors - they're a good option.

If, however, you're looking for something simple, there is another way - and you're in the right place! SparkPost’s Python library and our built-in scheduled sending feature make it easy to put something together.

We’ll put ourselves in the shoes of our friendly fictional company, Avocado Industries, and follow them through setting up a campaign. This article takes you through various features of SparkPost’s Python client library, and links to the final code here.

So what do I need?

You’re sending out a newsletter to your subscribers. You’ve created a nice looking template and uploaded it to SparkPost. You have your recipient list at hand, or can export it easily enough from your database. You want SparkPost to mail-merge the recipient personalization details in, and get your awesome send out.

These needs translate into the following design goals for the code we’re going to write:

Specify everything about your send using parameters and a small text file. You don’t want to change the code for each campaign.
Leverage the SparkPost stored template and personalization features, without doing any programming yourself.
Use local flat files for the recipient lists. The same format used by SparkPost’s stored recipient list uploads is a good fit.
Gather recipients from your list into API-call batches for efficiency - with no upper limits to the overall size of your send.
Support timezones for the scheduled send, and also support â€˜just send it now’.

Data guacamole

The SparkPost recipient-list format looks like this, with all the fields populated. Here we see Jerome’s details for the Avocado Industries loyalty scheme. We can see he’s a gold-card member, lives in Washington State, and likes several different avocado varieties.

email,name,return_path,metadata,substitution_data,tags
jerome.russell@example.com,Jerome Russell,bounce@avocado-industries.com,"{""custID"": 60525717}","{""memberType"": ""gold"", ""state"": ""WA""}","[""hass"", ""pinkerton"", ""gwen"", ""lamb hass"", ""fuerte"", ""bacon""]"

Everything apart from the email address is optional, so it would be nice to have the tool also accept just a plain old list of email addresses. It would also be nice if the tool is happy if we omit the header line. That’s easily done.

Taco me to the start

The tool is written for python3. SparkPost relies on the pip installer, and we’ll need git to obtain the tool. You can check if you already have these tools, using the following commands.

$ python3 -V
Python 3.5.1
$ pip3 -V
pip 9.0.1 from /usr/local/lib/python3.5/site-packages (python 3.5)
$git --version
git version 2.7.4

If you already have them, continue to “Add SparkPost Python Library sauce” below. Otherwise here is a simple install sequence for Amazon EC2 Linux:

sudo su -
yum update -y
yum install -y python35
yum install -y wget
wget https://bootstrap.pypa.io/get-pip.py
python3 get-pip.py 
yum install -y git

If you are using another platform, check out installation instructions for your platform here.

Add SparkPost Python Library sauce

We use pip3 to install, as follows.

$ sudo /usr/local/bin/pip3 install sparkpost
Collecting sparkpost
  Using cached sparkpost-1.3.5-py2.py3-none-any.whl
Collecting requests>=2.5.1 (from sparkpost)
  Using cached requests-2.13.0-py2.py3-none-any.whl
Installing collected packages: requests, sparkpost
Successfully installed requests-2.13.0 sparkpost-1.3.5

Get the sparkySched code from Github using:

$ git clone https://github.com/tuck1s/sparkySched.git
Cloning into 'sparkySched'...
remote: Counting objects: 55, done.
remote: Compressing objects: 100% (3/3), done.
remote: Total 55 (delta 0), reused 0 (delta 0), pack-reused 52
Unpacking objects: 100% (55/55), done.
Checking connectivity... done.

$ cd sparkySched

Gotta be .ini to win it

We now set up some attributes such as your API key, campaign, and certain substitution data in a text file, as they will be the same each time you send. An example is provided in the project called sparkpost.ini.example. Rename this to sparkpost.ini, and replace with a key you’ve created in your own SparkPost account.

[SparkPost]
Authorization = <YOUR API KEY>
#Campaign setup
Campaign = avocado-saladcopter
GlobalSub = {"subject": "Fresh avocado delivered to your door in 30 minutes by our flying saladcopter"}

And, Send!

There’s a sample file of 1000 safe test recipients included in the project that we can send to. Change the template name below from avocado-goodness to one you have in your account, and set the sending time to suit you:

$ ./sparkySched.py recips_1k_sub_n_tags.csv avocado-goodness 2017-05-08T19:10:00+01:00
Opened connection to https://api.sparkpost.com
Injecting to SparkPost:
To  1000 recips: template "avocado-goodness" start_time 2017-05-08T19:10:00+01:00: OK - in 1.62 seconds
$

If all is well, you should see the “OK” line, and your mailing is sent. That’s all you need to do. Happy sending!

Code salsa

In this section, we take a deeper look inside the code. You can skip this if you just want to use the tool instead of changing it. Here’s how we call the SparkPost API to send messages, using the SparkPost Python library:

# Inject the messages into SparkPost for a batch of recipients, using the specified transmission parameters
def sendToRecips(sp, recipBatch, sendObj):
   print('To', str(len(recipBatch)).rjust(5, ' '),'recips: template "'+sendObj['template']+'" start_time '+sendObj['start_time']+': ',end='', flush=True)

   # Compose in additional API-call parameters
   sendObj.update({
       'recipients': recipBatch,
       'track_opens':  True,
       'track_clicks': True,
       'use_draft_template': False
   })
   startT = time.time()
   try:
       res = sp.transmissions.send(**sendObj)
       endT = time.time()
       if res['total_accepted_recipients'] != len(recipBatch):
           print(res)
       else:
           print('OK - in', round(endT - startT, 3), 'seconds')
   except SparkPostAPIException as err:
       print('error code', err.status, ':', err.errors)
       exit(1)

After some helpful on-screen output about what we’re trying to send, the function composes the recipients with the other passed-in ingredients and mixes in some sensible defaults, using sendObj.update().

The SparkPost library call is wrapped in a try/except clause, as it can return errors at the application level (such as having an incorrect API key), or at the transport level (such as your Internet connection being down). This is generally good practice with any code that’s communicating with a remote service, and follows the examples packaged with our library.

We use startT, endT, and the time() function to measure how long the API call actually takes. While not strictly necessary, it’s interesting to see how performance varies with batch size, routing distance from client to server, etc.

We will now craft the code to read parameters from the .ini file and use them in the API sends. Let’s read and check the mandatory parameters:

# Get parameters from .ini file
configFile = 'sparkpost.ini'
config = configparser.ConfigParser()
config.read_file(open(configFile))
cfg = config['SparkPost']
apiKey = cfg.get('Authorization', '')           # API key is mandatory
if not apiKey:
   print('Error: missing Authorization line in ' + configFile)
   exit(1)
baseUri = 'https://' + cfg.get('Host', 'api.sparkpost.com')     # optional, default to public service

The Python library configParser does the heavy lifting. You’ve got to have an API key, so we exit if unset. baseUri defaults to the sparkpost.com API endpoint if it’s unset. The other parameters from the .ini file are read in the same way, and are described in the project README file.

There are other ways to set things up in Python, such as using environment variables. My preference is for .ini files, because the file is right there, staring at you. It’s easy to store, communicate, change and check, right there in your project.

Chickens go in, pies come out ...

Let’s look at how to read that .csv format recipient list. Python provides a nice library, csv. All the reading of double-quoted stuff that .csv files need to carry JSON objects like "{""custID"": 60525717}" is taken care of for us.

We could use csv to read the whole recipient-list into a Python array object - but that’s not a great idea, if we have squillions of addresses in our list. The client will be perfectly fast enough for our purposes, and we’ll use less client memory, if we read in just enough to give us a nice sized batch to cook each time around.

We’ll also handle line 1 of the file specially, to meet our â€˜go easy on the optional file header’ requirement. Recall that a well-formed header should look like this:

email,name,return_path,metadata,substitution_data,tags

If it’s got the single word â€˜email’ somewhere on line 1, let’s assume it really is a header, and we’ll take our field layouts from that line. The tool will be happy if you omit optional fields, or have them in a different order on the line. The only one you absolutely need is the email field.

recipBatch = []
f = csv.reader(fh_recipList)

for r in f:
   if f.line_num == 1:                         # Check if header row present
       if 'email' in r:                        # we've got an email header-row field - continue
           hdr = r
           continue
       elif '@' in r[0] and len(r) == 1:       # Also accept headerless format with just email addresses
           hdr = ['email']                     # line 1 contains data - so continue processing
       else:
           print('Invalid .csv file header - must contain "email" field')
           exit(1)

We then check if it’s really a headerless file with just a bunch of email addresses in it, by checking for a single entry with an @ sign.

In the main loop for i,h in enumerate(hdr) we use some nice Python language features to conform the data to the JSON object that SparkPost is expecting. The name field needs to be put inside the address.name JSON attribute. Return_path is added, if present. Metadata, substitution_data and tags all come in to us as JSON-formatted strings, so we unpack them using json.loads().

   # Parse values from the line of the file into a row object
   row = {}
   for i,h in enumerate(hdr):
       if r[i]:                                        # Only parse non-empty fields from this line
           if h == 'email':
               row['address'] = {h: r[i]}              # begin the address
           elif h == 'name':
               row['address'].update(name = r[i])      # add into the existing address structure
           elif h == 'return_path':
               row[h] = r[i]                           # simple string field
           elif (h == 'metadata' or h == 'substitution_data' or h == 'tags'):
               row[h] = json.loads(r[i])               # parse these fields as JSON text into dict objects
           else:
               print('Unexpected .csv file field name found: ', h)
               exit(1)

   recipBatch.append(row)
   if len(recipBatch) >= batchSize:
       sendToRecips(sp, recipBatch, txOpts)
       recipBatch = []                                 # Empty out, ready for next batch
# Handle the final batch remaining, if any
if len(recipBatch) > 0:
   sendToRecips(sp, recipBatch, txOpts)

All that’s left to do, is to chew through the list, sending each time we gather in a full sized batch. We send any final batch at the end, and we’re done.

Command-line garnish - a pinch of thyme time

The last part we need is some command-line argument parsing. The recipient-list, the template-ID, and the sending date/time are the things you might want to vary each time the tool is run. Python munges your arguments using argv[] in much the same way as other languages.

There are all kinds of nonsense possible with input date and time - such as February 30th, 24:01 and so on. Mix in timezone offsets, so the user can schedule in their local time, and no-one would seriously want to write their own time parsing code! SparkPost’s API will of course be the final arbiter on what’s good, and what’s not - but it’s better to do some initial taste-tests before we try to send.

Python’s strptime() function does mostly what we want. The format string can be made like SparkPost format, except Python has no : separator in the %z timezone. Python’s elegant negative indexing into strings (working backwards from the end of the string) makes it easy to write a small checking function.

# Validate SparkPost start_time format, slightly different to Python datetime (which has no : in timezone offset format specifier)
def isExpectedDateTimeFormat(timestamp):
   format_string = '%Y-%m-%dT%H:%M:%S%z'
   try:
       colon = timestamp[-3]
       if not colon == ':':
           raise ValueError()
       colonless_timestamp = timestamp[:-3] + timestamp[-2:]
       datetime.strptime(colonless_timestamp, format_string)
       return True
   except ValueError:
       return False

Plat du jour

If you don’t want to schedule a future start_time, you can just give today’s date and time. Times in the past are sent immediately.

Depending on where your code is running (I happen to be using an AWS virtual machine), you should see each batch get sent in a few seconds. Even though it’s single-threaded, you can schedule a million emails in around ten minutes. The actual send will proceed (at the scheduled start_time) as fast as it can.

$ ./sparkySched.py recips_100k_sub_n_tags.csv avocado-goodness 2017-04-11T23:55:00+01:00
Opened connection to https://demo.sparkpostelite.com
Injecting to SparkPost:
To 10000 recips: template "avocado-goodness" start_time 2017-04-11T23:55:00+01:00: OK - in 4.97 seconds
To 10000 recips: template "avocado-goodness" start_time 2017-04-11T23:55:00+01:00: OK - in 4.92 seconds
To 10000 recips: template "avocado-goodness" start_time 2017-04-11T23:55:00+01:00: OK - in 4.783 seconds

And that’s pretty much it. The full code, which is just over 100 actual lines, is here with easy installation instructions.

A small digestif ...

What’s the best way to test this out for real? A tool to generate dummy recipient-list with sinkhole addresses could be handy. Keep an eye out for a follow-up blogpost.. I’ve included a couple of ready-made recipes recipient files in the github project to get you started.

Is this your first dining experience with Python and SparkPost? Did I add too much seasoning? Should the author be pun-ished for these bad jokes? Let us know!

P.S. Want to talk more Python with us? Join us in our Community Slack.

This post was originally posted on the SparkPost blog.