AB tests with Symfony 5 made easy

Thomas P — Fri, 09 Oct 2020 17:16:15 +0000

The company I work for is a data-driven company. It is necessary for us to make the best decisions and aim for our core value: being customer-centric.

Naturally, when we iterate on a feature, we study the data to define how to improve it. And because we can’t improve what we don’t measure, we measure the results of our new version with the previous version.

To do this, we almost systematically perform AB tests that we couple with our events database.

AB tests are not for experimenting things, but to measure them.

And since it's HacktoberFest, we published one of our tools in open source here: Travaux.com VariantRetriever

Next part of this article is to start your first AB test, like changing the subject of an email when a new user registers.
The experiment we are doing will be called welcome-email-experiment and it will have a control version (the actual version) and a participant (or variant) version (the new version we want to try).

When we will send this email, we’re going to require the VariantRetriever and ask it: for this experiment and this user identifier, tell me which variant it should have.

Lets run a new AB test with Symfony 5

First step is of course to install the AB test package in your project:

composer require travaux-com/variantretriever

Because it’s not a Symfony bundle, you don’t need to declare it in your AppKernel file, but you still need (recommanded in fact) to declare it as a service.

We’re going to declare it as a service with the Symfony container factory system to have our “variant retriever” with all information about our running AB tests. (welcome-email-experiment and display-cta).

services:
    App\FeatureFlag\VariantRetriever:
        factory:   ['@Travaux\VariantRetriever\Factory\VariantRetrieverFactory', 'createVariantRetriever']
        public: true
        arguments:
            - welcome-email-experiment:
                - control-email: 50
                - participant-email: 50
            - display-cta:
                - control-cta: 75
                - participant-cta: 25

The service name is actually App\FeatureFlag\VariantRetriever but I strongly encourage you to customize it.

Now you’re able to use the Symfony autowiring to retrieve the VariantRetrieverInterface and use it in your command handler.
And so on, how to use it ?

$affectedVariant = $variantRetriever
           ->getVariantForExperiment(
                  new Experiment('my-ab-test'), 
                  (string) $user->getId()
           );

VariantRetrieverInterface actually came with the getVariantForExperiment method where you will have to define the experiment you want to test (by just filling the experiment name) and the “user identifier” who just need to be a string, so it can be a UUID v4 or an auto-incremented integer.

It will return you a Variant value object that contains the name of the affected variant.

Now you can imagine select the right translation key to use depending on the affected variant for your email:

$experimentTranslationsKeys = [
       'control' => 'email.welcome.subject_control',
       'variant' => 'email.welcome.subject_variant',
];
$userVariant = $variantRetriever
           ->getVariantForExperiment(
                  new Experiment('welcome-email-experiment'), 
                  (string) $user->getId()
           );
$emailSubject = $experimentTranslationsKeys[(string) $userVariant];

This package will ensure you to always retrieve the same variant for your user, without any database or cache.

You’re now able to run your first AB in your Symfony project 🎉

Go beyond this example

As you can see, we’ve use the Symfony container factory system to instantiate our VariantRetriever, but it can also be declared with Symfony method call like this:

   Travaux\VariantRetriever\Retriever\VariantRetrieverInterface:
        class: Travaux\VariantRetriever\Retriever\VariantRetriever
        calls:
            - addExperiment:
                - !service
                    class: Travaux\VariantRetriever\ValueObject\Experiment
                    arguments:
                        - 'welcome-email-experiment'
                        - !service
                            class: Travaux\VariantRetriever\ValueObject\Variant
                            arguments: ['control', 100]
                        - !service
                            class: Travaux\VariantRetriever\ValueObject\Variant
                            arguments: ['variant', 0]

How does it work internally ?

Thanks to php crc32 function, we’re able to process a string that return everytime the same integer value. So on, even if it’s a big number, we’re able to reduce it under a range of 0 to 99 that match the rollout percentage given by one of our variant.

Also, the Travaux\VariantRetriever\Retriever\VariantRetriever class is not final to allow you to extend it. Like injecting a PsrLogger and/or the event dispatcher to keep a trace of which experiment have been affected to which user. You can also add you own method to retrieve a variant based on your User entity object like this:

public function getUserVariantForExperiment(
    Experiment $experiment,
    User $user
);

Feel Free to contribute to this project or give feedback.

Forem: Thomas P

AB tests with Symfony 5 made easy

Lets run a new AB test with Symfony 5

Go beyond this example

How does it work internally ?