Tracking Recipient Preferences With The User Agent Header in Elixir

Ewan Dennis — Fri, 13 Oct 2017 13:01:32 +0000

Tracking Recipient Preferences With The User Agent Header in Elixir

Note: this user agent header post illustrates itself using code written in Elixir. If you prefer, you can read the PHP version.

Much has been made of the relative commercial value of particular groups of people. From super consumers to influencers, iPhone users to desktop holdouts, learning about your recipients’ preferences is clearly important. In these days of deep personalization, it’s also just nice to know a little more about your customer base. Luckily this is a pretty easy job with SparkPost message events.

In this post, I’ll review the content of the User-Agent header, then walk through the process of receiving tracking events from SparkPost’s webhooks facility, parsing your recipients’ User Agent header and using the results to build a simple but extensible report for tracking Operating System preferences. I’ll be using Elixir for the example code in this article but most of the concepts are transferrable to other languages.

SparkPost Webhook Engagement Events

SparkPost webhooks offer a low-latency way for your apps to receive detailed tracking events for your email traffic. We’ve written previously about how to use them and how they’re built so you can read some background material if you need to.

We’ll be focusing on just the click event here. Each time a recipient clicks on a tracked link in your email, SparkPost generates a click event that you can receive by webhook. You can grab a sample click event directly from the SparkPost API here. The most interesting field for our purposes is naturally msys.track_event.user_agent which contains the full User-Agent header sent by your recipient’s email client when they clicked the link.

{ "msys": { "track_event": { // ... "user_agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_3) AppleWebKit/537.36 
(KHTML, like Gecko) Chrome/41.0.2272.118 Safari/537.36" // ... } } }

Grokking The User Agent

Ok so we can almost pick out the important details from that little blob of text. For the dedicated, there’s a specification but it’s a tough read. Broadly speaking, we can extract details about the user’s browser, OS and “device from their user agent string.

For example, from my own user agent:

Mozilla/5.0 (Linux; Android 7.1.1; Nexus 6P Build/N4F26O) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.91 Mobile Safari/537.36

…you can tell I’m an Android user with a Huawei Nexus 6P device (and that it’s bang up-to-date ;).

Caveat: user agent spoofing

Some of you might be concerned about the information our user agent shares with the services we use. As is your right, you can use a browser plugin (Chrome, Firefox) or built-in browser dev tools to change your user agent string to something less revealing. Some services on the web will alter your experience based on your user agent though so it’s important to know the impact these tools might have on you.

Harvesting User Agents From SparkPost Tracking Events

Alright, enough theory. Let’s build out a little webhook service to receive, process and stash user agent details for each click tracked through our SparkPost account.

Elixir And The Web: Phoenix

The de facto standard way to build web services in Elixir is the Phoenix Framework. If you’re interested in a Phoenix getting started guide, the docs are excellent and the Up and Running guide in particular is a great place to start.

We’ll assume you already have a basic Phoenix application and focus on adding an HTTP endpoint to accept SparkPost webhook event batches.

Plug: Composable Modules For The Web

Elixir comes with a specification called â€˜Plug’ (defined here) which makes it easy to build up layers of so-called middleware on an HTTP service. The simplest form of plug is a function that accepts a connection and a set of options. We’ll use this form to build up our webhook consumer.

Handling SparkPost Webhooks Requests

Our first task is to create a “pipeline”, which is a sequence of transformations that a connection goes through. A pipeline in Phoenix is just a convenient way to compose a sequence of plugs and apply them to some group of incoming requests.

We’ll first create a “webhook pipeline and then add plugs to it to handle the various tasks in our service. All this happens in our application’s Router module:

# Create a new plug pipeline to handle SparkPost webhooks requests pipeline :webhook do
# Use the accepts plug plug :accepts, ["json"] end

You can read more about Phoenix routing and plug pipelines in the routing section of the Phoenix docs. For now, it’s important to realize that each Phoenix application includes an endpoint module which is responsible for setting up basic request processing. This includes automatic JSON parsing, which we’ll rely on here.

Unpacking SparkPost Events

Our event structure contains a certain amount of nesting which we can now strip out in preparation for consuming the tasty details inside. This is a job for our very first plug:

# Define a new plug to extract nested event fields def 
unpack_events(conn, _) do first_key_fn = &hd(Map.keys(&1))   
cleanevents = conn.params["_json"] |> Enum.map(fn evt -> 
evt["msys"] end) |> Enum.map(fn evt -> evt[first_key_fn.(evt)] 
end) assign(conn, :events, cleanevents) end # Add the new plug 
to our pipeline pipeline :webhook do plug 
:accepts, ["json"] plug :unpack_events end

There is a little magic going on here. Our endpoint applies the JSON parser plug to all requests before our pipeline starts. Our unpack_events plug can then rely upon the _json param left on the connection JSON parser.

The rest of unpack_events is just extracting the contents of the msys key on each event and the contents of the first key in that object. Finally, our unpack_events plug stored our unpacked events on a connection param for later plugs to pick up.

Filtering Events

Now lets retain just the click events (when we register our webhook with SparkPost later, we can also ask it to send only click events):

def filter_event_types(conn, types) do good_event? = 
&Enum.member?(types, &1["type"]) assign(conn, :events, 
Enum.filter(conn.assigns[:events], good_event?)) end # Apply event 
filtering after unpacking pipeline :webhook do plug :accepts, 
["json"] plug :unpack_events plug :filter_event_types ['click'] end

This plug leaves our filtered events on the :events connection param. filter_event_types accepts a list of types we care about.

There’s a lot of detail in a single event. It might be a good idea to pare things down to just the fields we care about:

def filter_event_fields(conn, kv) do pick_fields = &Map.take(&1, 
kv) assign(conn, :events, Enum.map(conn.assigns[:events], 
pick_fields)) end # JSON -> unpack -> filter on clicks -> extract 
user_agent fields pipeline :webhook do plug :accepts, ["json"]   
plug :unpack_events plug :filter_event_types, ['click'] plug 
:filter_event_fields, ['user_agent'] end

After The Plug Pipeline: The Controller

To finish up our webhook request handling, we need a controller which works after the plug pipeline to process to request and produce a response for the client. Here’s a skeleton Controller:

defmodule TrackUserAgentsWeb.ApiController do use 
TrackUserAgentsWeb, :controller def webhook(conn, _params) do     
json conn, %{"ok": true} end end

Then we can wire ApiController.webhook/2 to our router:

scope "/webhook", TrackUserAgentsWeb do pipe_through :webhook     
post "/", ApiController, :webhook end

When we register our web service with SparkPost as a webhook consumer, it’ll make HTTP requests to it containing a JSON payload of events. Now our service has a /webhook endpoint that accepts JSON, cuts our event batch down to size and responds with a happy little “ok!”.

Testing Our Progress

We can test our service by sending a test batch to it. Luckily, the SparkPost API will generate a test batch for you on request.

Grab a sample webhooks event batch from the SparkPost API: Note: this step uses cURL and jq. You can skip the jq part and remove the results key from the JSON file yourself.
curl https://api.sparkpost.com/api/v1/webhooks/events/samples | jq .results > batch.json
Start our Phoenix service:
mix phx.server
Send our test batch to the service:
curl -XPOST -H "Content-type: application/json" -d @batch.json http://localhost:4000/webhook

Parsing User-Agent

Now we’re ready to enrich our events with new information. We’ll parse the user agent string and extract the OS using the ua_inspector module. We can easily add this step to the API plug pipeline in our router:

Note: If you’re following along, remember to add ua_inspector as a dependency in mix.exs and configure it.

def os_name(ua) do if not Map.has_key?(ua, :os) do "unknown" 
else case ua.os do %UAInspector.Result.OS{} -> ua.os.name 
_ -> "unknown" end end end def parse_user_agents(conn, _) do 
events = conn.assigns[:events] |> Enum.map(&Map.put(&1, 
"user_agent", UAInspector.parse(&1["user_agent"]))) |> 
Enum.map(&Map.put(&1, "os", os_name(&1["ua"]))) assign(conn, 
:events, events) end # JSON -> unpack -> filter on clicks -> extract 
user_agent -> filter nulls -> parse pipeline :webhook do plug 
:accepts, ["json"] plug :unpack_events plug :filter_event_types, 
['click'] plug :filter_event_fields, ['user_agent'] plug 
:parse_user_agents end

Note: not all user agent strings will contain the detail we want (or even make sense at all) so we label all odd-shaped clicks with “OS: unknown”.

Alright, now we have an array of events containing only interesting fields and with an extra “os field to boot.

Generating Report-Ready Summary Data

At this point, we could just list each event and call our report done. However, we’ve come to expect some summarisation in our reports, to simplify the task of understanding. We’re interested in OS trends in our email recipients, which suggests that we should aggregate our results: collect summaries indexed by OS. Maybe we’d even use a Google Charts pie chart.

We could stop there citing “exercise for the reader but I always find that frustrating so instead, here’s a batteries-included implementation which stores click events summaries in PostgreSQL and renders a simple report using Google Charts.

An Exercise For The Reader

I know, I said I wouldn’t do this. Bear with me: if you were paying attention to the implementation steps above, you might have noticed several re-usable elements. Specifically, I drew a few filtering and reporting parameters out for re-use:

event type filters
event field filters
event “enrichment functionality

With minimal effort, you could add, filter on and group the campaign_id field to see OS preference broken down by email campaign. You could also use it as a basis for updating your own user database from bounce events with type=bounce, fields=rcpt_to,bounce_class and so on.

I hope this short walkthrough gave some practical insight on using SparkPost webhooks. With a little experimentation, the project could be made to fit into plenty of use cases and I’d be more than happy to accept contributions on that theme. If you’d like to talk more about the user agent header, your own event processing needs, SparkPost webhooks, Elixir or anything else, come find us on Slack!

The post Tracking Recipient Preferences With The User Agent Header in Elixir appeared first on SparkPost.

Forem: Ewan Dennis

Tracking Recipient Preferences With The User Agent Header in Elixir