Forem: Patryk Kalinowski

Make your analytics work with adblockers

Patryk Kalinowski — Fri, 10 Jul 2020 10:19:43 +0000

Depending on country and industry, up to 40% of your users use adblockers. You know nothing about their activity on website, they are not on your remarketing lists and every CRM and marketing automation system reports their source as dreaded DIRECT TRAFFIC.

Of course it is possible to improve it. 100% attribution sounds enticing and is worth a lot of money, so let's try to reach it.

Disclaimer: do not implement this in any reputable company you're working for. Everything below is black hat marketing and will probably get you fired when the PR backslash hits. Also, there is no going back from black hat - consider yourself warned. Examples below intentionally lack few crucial elements you need to figure out by yourself.

I was able to put everything described here into 2kB ultimate script which you can implement with one line of code on your website, not care about anything and just capture traffic data against every adblocker, including uBlock Origin. I did this as a hobby project during several evenings - imagine what adtech companies can do.

Table of Contents:

How adblockers actually work
Make Facebook Pixel work with adblockers
Make LinkedIn Insight Tag work with adblockers
Make Google Analytics work with adblockers

How adblockers actually work

Basic adblocker monitors requests from website you are visiting: images, stylesheets and scripts. If any request URL matches one found in filter lists, it's not getting through. Analytics scripts are blocked from being loaded and data is blocked from being sent. It's actually pretty simple:

You paste analytics script on your website
Users' web browser tries to load the script - blocked
Loaded script gathers data from current pageview (user details, cookies, page visited etc.)
Script packages that data and sends to tracking server - blocked

How to avoid adblockers? Make sure none of the script URL is recognized as analytics and send data to different server not known as tracking server. It requires some reverse engineering and modifying original tracking scripts to send data to different URL (so called proxy). Good practice would be to launch your solution only when adblocker is detected to reduce server cost.

Make Facebook Pixel work with adblockers

Let's start with the code you have to paste on your website to start tracking visitors:

 <!-- Facebook Pixel Code -->
 <script>
    !function(f,b,e,v,n,t,s)
    {if(f.fbq)return;n=f.fbq=function(){n.callMethod?
    n.callMethod.apply(n,arguments):n.queue.push(arguments)};
    if(!f._fbq)f._fbq=n;n.push=n;n.loaded=!0;n.version='2.0';
    n.queue=[];t=b.createElement(e);t.async=!0;
    t.src=v;s=b.getElementsByTagName(e)[0];
    s.parentNode.insertBefore(t,s)}(window, document,'script',
    'https://connect.facebook.net/en_US/fbevents.js');
    fbq('init', '000000000000000');
    fbq('track', 'PageView');
  </script>
  <noscript><img height="1" width="1" style="display:none"
    src="https://www.facebook.com/tr?id=484906408383015&ev=PageView&noscript=1"
  /></noscript>
  <!-- End Facebook Pixel Code -->

So what this script does is:

Creates new fbq function
Creates event queue to process incoming events
Pushes two events to event queue: init and track pageview
Adds new script to be loaded on website: https://connect.facebook.net/en_US/fbevents.js

fbevents.js then loads the actual Facebook Pixel and starts processing fbq event queue. Every event is parsed and sent to Facebook tracking server. This is how tracking request looks:

https://www.facebook.com/tr/?id=0000000000000000&ev=PageView&dl=http%3A%2F%2Flocalhost%3A49925%2F&rl=&if=false&ts=1574545579948&sw=2048&sh=1152&v=2.9.13&r=stable&ec=0&o=30&it=1574545579618&coo=false&rqm=GET

Basic tracking URL is https://www.facebook.com/tr/ which receives information to which Pixel should this data be sent to (id), what kind of data is this (ev=PageView), URL to be tracked (dl) and several others.

Adblockers come in during step 4: Adds new script to be loaded on website: https://connect.facebook.net/en_US/fbevents.js by blocking the URL not letting the script to be loaded.

OK, now we know how to detect if Facebook Pixel is implemented on website: we need to check if fbq function has been created (its creation is not blocked). But how do we know if Pixel has been stopped from loading by adblocker? Let's investigate what fbevents.js does.

fbq.registerPlugin("global_config", {__fbEventsPlugin: 1, plugin: function(fbq, instance, config) { fbq.loadPlugin("opttracking");
fbq.set("experiments", [{"allocation":0.01,"code":"b","name":"batching","passRate":0.5}]);
config.set(null, "batching", {"batchWaitTimeMs":501,"maxBatchSize":10});
config.set(null, "microdata", {"waitTimeMs":500});instance.configLoaded("global_config"); }});

These are the last four lines of fbevents.js. Apart from that, it's filled with lots of minified javascript code, but we won't need it. What we see here are two additional functions within fbq object: registerPlugin and loadPlugin. These are not present in original script pasted on website, but rather created by fbevents.js. We can assume that if fbq exists but fbq.loadPlugin doesn't, fbevents.js failed to load - and adblocker did its job.

if (typeof fbq === 'function' && typeof fbq.loadPlugin === 'undefined') {
    // Facebook Pixel adblocked
}

Let's try the simplest approach - download fbevents.js file, rename it, put it on your own website and load it when original is blocked.

if (typeof fbq === 'function' && typeof fbq.loadPlugin === 'undefined') {
    // Facebook Pixel adblocked
    var s = document.createElement('script');
    s.src = "http://example.com/sneaky.js"
    document.body.appendChild(s);
}

OK, kinda worked, but not really. Renamed fbevents.js works, but it's not the actual Pixel to do the tracking. It tries to load Pixel from URL: https://connect.facebook.net/signals/config/YOUR_PIXEL_ID?v=2.9.13&r=stable. Naturally this URL is also blocked.

We could do the same thing like before: download pixel file, rename it, put in on website and inside fbevents.js replace all occurences of https://connect.facebook.net/signals/config/YOUR_PIXEL_ID to http://example.com/sneaky_YOUR_PIXEL_ID.js. I think this approach is faulty - Facebook Pixel code is probably worked on by Facebook and it will be updated regularly, making your workaround obsolete any day. I ended up with fully automated solution to make sure browser always receives current Pixel code - a proxy server.

In short, I've changed all occurences of https://connect.facebook.net/signals/config/YOUR_PIXEL_ID?v=2.9.13&r=stable to http://example.com/signals/config/YOUR_PIXEL_ID?v=2.9.13&r=stable. It keeps the same URL structure which makes it easier to debug and understand what's happening. My custom URL leads to AWS Lambda function which basically downloads current original script and returns it back to browser:

/* Download and return Facebook Pixel */

const request = require('request')

exports.handler = function (event, context, callback) {

  params = event.queryStringParameters
  params.id = event.path.slice(16, event.path.size) // extract Pixel ID from path, ex. path = '/signals/config/000000000000000'

  // Download Pixel javascript file and return it to user from custom URL
  request(`https://connect.facebook.net/signals/config/${params.id}?v=${params.v}&r=${params.r}`, function (error, response, body) {
    if (error) {
      console.log(error)
    }

    callback(null, {
      statusCode: 200,
      body: body
    });
  });
}

Great, we're one step forward. Pixel works, but tries to send data to blocked Facebook tracking URL: https://www.facebook.com/tr/. Let's create another AWS Lambda function which will become our proxy tracking server:

/* Send tracking request to Facebook */

const request = require('request')

exports.handler = function (event, context, callback) {

  callback(null, {
    statusCode: 200,
    body: ""
  });

  params = event.queryStringParameters

  // HTTP Request to Facebook tracking URL  
  request.post({ url: 'https://www.facebook.com/tr', form: params }, function (err, httpResponse, body) { 
    if (err) {
      console.log(err)
    }
  })
}

Now go to your custom fbevents.js and replace every https://www.facebook.com/tr/ with your proxy server URL, ex. http://example.com/fbtr/

From now on, all tracking requests will be sent to non-blocked URL:

http://example.com/fbtr/?id=0000000000000000&ev=PageView&dl=http%3A%2F%2Flocalhost%3A49925%2F&rl=&if=false&ts=1574545579948&sw=2048&sh=1152&v=2.9.13&r=stable&ec=0&o=30&it=1574545579618&coo=false&rqm=GET

And that's basically it! Facebook Pixel now works on every browser with every adblocker

Make LinkedIn Insight Tag work with adblockers

Here is the Insight Tag to be pasted on website:

<script type="text/javascript">
    _linkedin_partner_id = "0000000";
    window._linkedin_data_partner_ids = window._linkedin_data_partner_ids || [];
    window._linkedin_data_partner_ids.push(_linkedin_partner_id);
    </script><script type="text/javascript">
    (function(){var s = document.getElementsByTagName("script")[0];
    var b = document.createElement("script");
    b.type = "text/javascript";b.async = true;
    b.src = "https://snap.licdn.com/li.lms-analytics/insight.min.js";
    s.parentNode.insertBefore(b, s);})();
</script>
<noscript>
    <img height="1" width="1" style="display:none;" alt="" src="https://px.ads.linkedin.com/collect/?pid=0000000&fmt=gif" />
</noscript>

It's very similar to one from Facebook, so I'm going to describe it shortly to avoid repetitions.

Script creates one variable which we can use to detect if LinkedIn Tag is set up on website: _linkedin_partner_id. File blocked by adblocker is https://snap.licdn.com/li.lms-analytics/insight.min.js. Inspecting this file reveals that it creates new variable: window._already_called_lintrk=!0. We will use it to check if adblocker is active:

if (typeof _linkedin_data_partner_ids != 'undefined' && typeof _already_called_lintrk === 'undefined') {
    /* linkedin blocked */  
    var s = document.createElement('script');
    s.src = "http://example.com/sneaky.js"
    document.body.appendChild(s);
  }

If LinkedIn Insight Tag is blocked by adblocker, we load a modified version from our server. Only renaming the script will not make it work as it still tries to send tracking data to blocked URL: https://px.ads.linkedin.com/collect?. We change this URL to our own proxy server with following code (again on AWS Lambda):

/* Send tracking request to LinkedIn */

const request = require('request')

exports.handler = function (event, context, callback) {  
  callback(null, {
    statusCode: 200,
    body: ""
  });

  params = event.queryStringParameters

  // HTTP Request to LinkedIn tracking pixel  
  request.post({ url: 'https://px.ads.linkedin.com/collect?', form: params }, function (err, httpResponse, body) { 
    if (err) {
      console.log(err)
    }
  })
}

Quick and easy. LinkedIn Insight Tag now works on every browser with every adblocker.

Make Google Analytics work with adblockers

I don't know exactly why, but adblockers (especially uBlock Origin) really don't like Google Analytics. Not only URLs are blocked, but all Javascript objects are also nullified. This requires different approach and is more difficult.

We start as usual, with the code to be pasted on website. There are two versions available: current gtag.js and older analytics.js

gtag.js (basically version of Google Tag Manager):

<script async src="https://www.googletagmanager.com/gtag/js?id=UA-00000000-00"></script>
<script>
    window.dataLayer = window.dataLayer || [];
    function gtag(){dataLayer.push(arguments);}
    gtag('js', new Date());

    gtag('config', 'UA-00000000-00');
</script>

analytics.js:

 <script>
    (function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){
    (i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o),
    m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)
    })(window,document,'script','https://www.google-analytics.com/analytics.js','ga');

    ga('create', 'UA-00000000-00', 'auto');
    ga('send', 'pageview');
</script>

Google Analytics tracking is described fairly well on the internet:

You should read it to fully understand what is following.

Both code snippets above download analytics.js and create ga object holding all necessary information for analytics script to work. Important functions and variables are ga, ga.q, ga.getAll(). They are being read and modified by analytics.js as it is being loaded and starts gathering data.

I won't explain what adblockers do here in detail, but this code should be enough:

if (typeof dataLayer == "object" && dataLayer[1][0] == "config" && typeof ga == "undefined") {
    // gtag.js
}

if (typeof ga === 'function' && typeof ga.q === 'object' && typeof ga.getAll === 'undefined') {
    // analytics.js, standard adblock
}

if (typeof ga === 'function' && typeof ga.q === 'undefined' && !ga.getAll()[0]) {
    // analytics.js, ublock origin
}

You might notice that there are different rules for standard adblocker and uBlock. uBlock removes whole GA queue (ga.q is undefined) and tracker objects. Inspecting uBlock Origin source code reveals interesting thing: these variables are being replaced by empty values. Even if we manage to load analytics.js by renaming it and hiding from adblockers, it still won't work as all required data has been deleted.

One solution to this is to immediately duplicate ga object after its creation:

ga('create', 'UA-00000000-00', 'auto');
ga('send', 'pageview');

window.aa = ga;

Then, when uBlock is detected and ga removed, recreate it by assigning its values back to original variable: ga = aa

We proceed as usual: rename analytics.js, host it on different URL so it won't be detected and replace all occurences of blocked tracking URL (https://www.google-analytics.com/collect) with your own proxy server:

/* Send tracking request to Google Analytics Measurement Protocol */

const request = require('request')

exports.handler = function (event, context, callback) {

  callback(null, {
    statusCode: 200,
    body: ""
  });

  params = event.queryStringParameters

  // HTTP Request to Measurement Protocol  
  request.post({ url: 'https://www.google-analytics.com/collect', form: params }, function (err, httpResponse, body) { 
    if (err) {
      console.log(err)
    }
  })
}

This technique works for Google Analytics only partially. While Facebook Pixel and LinkedIn Insight Tag work 100% like originals, Google Analytics seems to ignore subsequent ga() invocations (ex. events triggered by user activity some time after page load). I didn't manage to find the cause for it because I got bored on the way, but if you know the reason - please let me know so I can update the post.

Why blocking 3rd party cookies won't stop adtech

Patryk Kalinowski — Fri, 10 Jul 2020 09:38:50 +0000

There are lot of misconceptions how 3rd party cookies affect website analytics and if marketing teams should be worried when they are being blocked.

I tried to explain inner workings of all this fuss as simple as possible.

Behold, 3rd party cookies for dummies:

When you visit WebsiteA.com which has Adtech.com tracking code, this code asks Adtech.com domain for cookie (VisitorID: 123). Since you are on WebsiteA.com, Adtech.com is third party domain and adtech cookies are third party cookies.

Tracking script then gathers some data on you and sends information to Adtech.com server:

event_type: pageview
url: websiteA.com/X.html
visitorID: 123

Adtech.com knows that user with VisitorID = 123 visited page X.html on WebsiteA.com

Next, you go to WebsiteB.com which is part of the same Adtech.com network. Script takes third party cookie from Adtech.com and sends back this information:

event_type: pageview
url: websiteB.com/Y.html
visitorID: 123

Adtech.com knows that user with VisitorID = 123:

visited page X.html on WebsiteA.com
visited page Y.html on WebsiteB.com

And so on, and so on...

What is Adtech.com, you ask? It's Facebook, Google, Twitter, LinkedIn and thousands of others. Tracking code is a Like button, Follow button, Retweet button, Adwords ads - basically everything which originates from those servers.

What happens when we block 3rd party cookies

You visit WebsiteA.com and tracking code asks Adtech.com for VisitorID cookie, but the request is blocked by browser. Tracking code is unable to get global unique visitor ID, so now it has to set up a first party cookie belonging to domain WebsiteA.com

This is being sent to Adtech.com:

event_type: pageview
url: websiteA.com/X.html
visitorID: AAA

Adtech.com knows that user with VisitorID = AAA visited page X.html on WebsiteA.com

Next, you go to WebsiteB.com and again, tracking code can only create VisitorID cookie specific to WebsiteB.com only: BBB on image above. What's important, that cookie will be different from the one on WebsiteA.com

event type: pageview
url: websiteB.com/Y.html
visitorID: BBB

Now, Adtech.com instead of having one user with ID = 123 visiting two websites, knows about two separate users visiting two different websites. Privacy saved once again.

Why it won't really work

Cookies are very easy and reliable solution to pass data between domains. They require almost zero computing power and are part of network protocols themselves.

Since cookies will be blocked, Adtech companies will start using different solutions to identify users. One of those is fingerprinting.

Fingerprinting uses the fact that no computer is the same. You can have 100 users on Chrome, but 30 of them are on Macbooks, 20 on Android phones and 50 on Windows.

All those people have different screens with different resolutions, different fonts installed. They are in different cities and countries, too. IP address is different for every user.

In short, every visitor is unique anyway. Now, instead of setting up a third party cookie within 1ms and zero computing power, adtech companies will deploy various fingerprinting solutions generating unique IDs anyway, but for the price of slower loading times, increased computation (which burns through your battery) and less pleasant experience in general.

Adtech is worth billions of dollars, so we can expect more and more nasty tactics.

Will it affect my Google Analytics?

No. All analytics tools use first party cookies, as there is no need to know if users from WebsiteA visit other websites. First party cookies are enough to track every activity on one website.