Forem: Erik Booij

Lazily loading template data in PHP with Twig and Shoot

Erik Booij — Tue, 05 Mar 2019 20:06:12 +0000

Disclaimer: this is a rewrite (new and improved) of a post I've originally shared on my personal blog

If you've ever worked with a relatively large codebase with many nested templates, you have probably found yourself in a situation where you needed to either pass variables way down the template tree, pollute the global (template) scope with many variables only relevant to deeply nested templates, or find other creative solutions for injecting data into these sub templates (with Twig extensions for example).

At Coolblue we are using Twig for template rendering and we have definitely found ourselves in that position. Over the years we have tried numerous approaches for working around that problem, including but not limited to injecting fully prerendered views/templates into other templates. Nothing ever felt really clean, maintainable or futureproof, but fortunately there is now a solution that basically solves all of our problems in that area.

Shoot is a Twig extension built by Victor Welling, also a developer at Coolblue. What Shoot allows you to do is tie a presentation model to a template. The model defines and limits the variables that are in scope when rendering the template. These presentation models are pretty much plain old PHP objects extending Shoots PresentationModel, containing a protected field for every variable. Where the magic comes in though, is that these models can declare themselves as "having a presenter" by implementing the HasPresenterInterface. A presenter populates the fields in a model at runtime, by accessing its (optional) dependencies and a PSR-7 ServerRequest. Basically this means a template now fetches its own data.

Benefits

The benefits we're getting out of using Shoot over conventional handling of templates and template data:

Performance (data is only retrieved if the template is actually being rendered)
Way better maintainability
Improved testability
More isolation (no more leakage of template scope)
Increased template reuse

In the Shoot README we can find a clear overview of how the paradigm shifts from preloading all data to "lazily loading" the data, based on rendered templates:

+---------------+          +---------------+
|    Request    |          |    Request    |
+-------+-------+          +-------+-------+
        |                          |     +---------+
        |                          |     |         |
+-------v-------+          +-------v-----v-+     +-+-------------+
|   Load data   |          |  Render view  +----->   Load data   |
+-------+-------+          +-------+-------+     +---------------+
        |                          |
        |                          |
+-------v-------+          +-------v-------+
|  Render view  |          |   Response    |
+-------+-------+          +---------------+
        |
        |
+-------v-------+
|   Response    |
+---------------+

How to use Shoot

In order to actually connect your template and a presentation model, you simply add the {% model %} tag to the template pointing the fully qualified class name of your model:

{% model 'ShootDemo\\Presentation\\BlogPostModel' %}

<html lang="en">
<head>
    <title>{% if post_exists %}{{ post_title }}{% else %}404 - Not Found{% endif %}</title>
    <meta charset="UTF-8">
</head>
<body>
{% if post_exists %}
    <h1>{{ post_title }}</h1>
    {% for paragraph in post_content %}
        <p>{{ paragraph }}</p>
    {% endfor %}
{% else %}
    <h1>Post not found</h1>
    <p>
        This post could not be retrieved. Check out the other posts:
    </p>
{% endif %}

<a href="/?postId=1">Post 1</a> &nbsp; <a href="/?postId=2">Post 2</a>
</body>
</html>

As you can see we're pointing to the following model: ShootDemo\Presentation\BlogPostModel. There are three variables used in this template and these all need to be defined in the model.

class BlogPostModel extends PresentationModel
{
    /** @var string[] */
    protected $post_content = '';

    /** @var bool */
    protected $post_exists = false;

    /** @var string */
    protected $post_title = '';
}

This model now provides the variables with these values to the template, dynamically, without you needing to pass them in from the parent. This is a static model though, the post would never exist. We could step it up a notch and make it dynamic, based on the current request, by making the model implement the HasPresenterInterface:

class BlogPostModel extends PresentationModel implements HasPresenterInterface
{
    /** @var string[] */
    protected $post_content = '';

    /** @var bool */
    protected $post_exists = false;

    /** @var string */
    protected $post_title = '';

    /**
     * @return string
     */
    public function getPresenterName(): string
    {
        return BlogPostPresenter::class;
    }
}

The interface defines a single method, getPresenterName(): string. That method should return the alias by which the actual presenter can be retrieved from a PSR-11 compliant container. Since we register our presenters by their fully qualified class name, that's also what we return here. It gives the added benefit of easy traversal in your IDE. The presenter might look something like this:

class BlogPostPresenter implements PresenterInterface
{
    /** @var BlogPostRepositoryInterface */
    private $blogPostRepository;

    /**
     * @param BlogPostRepositoryInterface $blogPostRepository
     */
    public function __construct(BlogPostRepositoryInterface $blogPostRepository)
    {
        $this->blogPostRepository = $blogPostRepository;
    }

    /**
     * @param ServerRequestInterface $request
     * @param PresentationModel      $presentationModel
     *
     * @return PresentationModel
     */
    public function present(ServerRequestInterface $request, PresentationModel $presentationModel): PresentationModel
    {
        $postId = (int)($request->getQueryParams()['postId'] ?? -1);

        try {
            $blogPost = $this->blogPostRepository->fetchBlogPost($postId);
        } catch (UnableToFetchBlogPostException $ex) {
            // Unable to load blog post return presentation model, explicitly setting post_exists to false
            return $presentationModel->withVariables([
                'post_exists' => false,
            ]);
        }

        $variables = [
            'post_content' => $blogPost->content(),
            'post_exists'  => true,
            'post_title'   => $blogPost->title(),
        ];

        return $presentationModel->withVariables($variables);
    }
}

This is it, your template now automatically gets populated with the relevant values for that request. Whenever the template is rendered, it will load its model, thereby automatically calling the presenter to update the properties of the model. No more trying to determine what your template might need in your controller/request handler.

As you can see the presenter can have its own dependencies like the $blogPostRepository, by configuring it in your DI-container. It also gets access to the request in the ->present() method. The second argument it receives is the model we've just defined. You're now free to write any logic you desire in this method, as long as you return an instance of PresentationModel. A common pattern here is to use the ->withVariables() method on the model that's passed to the presenter, by passing it an associative array of values.

Keep in mind that this example is still fairly simple and passing the values down from the controller might not be hard here, but it doesn't matter where/how deep your template is located. This would work exactly the same for a template that's nested 20 layers deep or reused across different pages.

How to install Shoot

Maybe the best thing about starting to use Shoot, is that it requires very minimal changes to your setup, because you can simply continue rendering templates as you're currently doing. If you don't define a model on your template, nothing changes, it's therefore very much opt-in on a per template basis.

Installation can be done through Composer:

$ composer require shoot/shoot

After that it's a matter of instantiating the Shoot Pipeline and attaching it to Twig:

$presenterMiddleware = new PresenterMiddleware($container);
$pipeline = new Pipeline([$presenterMiddleware]);
$installer = new Installer($pipeline);

// Add Shoot to Twig
$twig = $installer->install($twig);

As you can see Shoot has its own middleware, the most important one being the PresenterMiddleware, this is the one that enables the behavior I described above. There are other Shoot middlewares that ship with Shoot like a SuppressionMiddleware which suppresses RuntimeExceptions thrown from within an {% optional %} tag in your templates, a LoggingMiddleware that logs which templates are rendering including how long that takes and an InspectorMiddleware that logs information about template rendering to your browser console. All these are optional, although you'll want to add the PresenterMiddleware to enable the lazy data loading we're talking about.

Now you'll want to add Shoots PSR-15 middleware to your applications middleware stack. This grants Shoot access to your PSR-7 request that it can use in presenters to dynamically populate presentation models. In the example below I'm using the Idealo Middleware Stack:

$middlewareStack = new Stack(
    new EmptyResponse(),
    new ShootMiddleware($pipeline),
    $requestHandler
);

Now Shoot is completely set up and it's only a matter of actually executing your middleware stack and emitting the response.

Getting started

I've made a demo project available, which you can fork/clone/download.

ErikBooij / shootdemo

A tiny demo project for shoot

Shoot Demo

This is a tiny demo implementation of Shoot and serves a reference implementation to play around with for my blog post on using Shoot for lazily loading template data.

Install

Simply clone/download, run composer install and point your webserver at public/index.php (or composer run which will start PHP's built-in web server on port 80).

View on GitHub

When you've got your local copy, it's a matter of running two commands (the second one needs elevated privileges because it binds to port 80 by default):

$ composer install
$ sudo composer run

Now you can go to http://localhost/?postId=1 to see Shoot in action.

Follow up

There is more you can do with Shoot, like passing down prepopulated models, or feeding data from your (parent) template back to a presenter. I'll cover this in a next write-up if there's any interest for it (feel free to express your interest by liking this post or following me).

If you have comments, questions or a request for some support, I hope I'll see you in the comments. Cheers!

Getting the most out of server side caching

Erik Booij — Sun, 24 Feb 2019 12:11:27 +0000

A little while back I found myself in a position where my team and I were implementing a rather heavy runtime operation on our ecommerce website. It involved the percolator functionality in Elasticsearch (I'm planning to do a write-up of how we're using that and how we optimized its performance shortly, so please click follow below if you'd like to read more about that).

Erik BooijFollow

I like tackling performance problems, experimenting with new tools and technologies and improving software a little bit every day.

When developing software, performance is or should always be a concern, but when working on a large ecommerce platform the results of performance changes become extremely visible since we can immediately correlate decreased conversion and thus lost revenue to increased page load times. Running percolate queries in Elasticsearch is a heavy and in performance terms "expensive" operation by nature, so that gave us an even bigger incentive to focus on keeping it fast.

Once we had implemented the feature and put it "live" behind a feature switch for our internal users, without any caching, we saw the feature worked as intended, and the time it took to run the query was acceptable. We did however notice it was putting strain on our Elasticsearch cluster to a certain extent. Nothing particularly worrisome, but we should be able to do better.

Optimizing our cache usage

When writing items to our cache, we usually consider how up-to-date we need them to be. For example, a product price or stock status can't be cached for more than a few minutes, but the name of a product can safely be cached for an hour or longer, since the latter is not as likely to change as the former. Realizing what type of data you're dealing with and how volatile that data is, is a good first step in determining how long you should persist it in cache.

In our case a lot of data we're caching is based on the work done by our colleagues in our commercial teams. New products are added, the navigation structure of the site is changed, content is written, etc. Within our company a workday usually doesn't start before 8am and ends before 6pm. That's valuable input for our caching strategy, because that means the result of our Elasticsearch query will not change between 6pm and 8am the next day. Or between friday 6pm and monday morning 8am even.

We started using this as input for determining our cache TTLs:

Storing the result of the query in cache on a Tuesday at 8pm? Expire it the next morning at 8am.
Storing the result of the query in cache on a Saturday at 11am? Expire it on Monday morning at 8am.

This is how what our cache TTLs look like now, the horizontal axis obviously being time, the vertical axis being number of seconds we allow the item to live in cache.

Notice the pattern? Every evening the cache TTL goes way up and every friday evening the TTL goes even further up. The flat line at the bottom is our default TTL of 3600 seconds during the workday, which is a balance between using the cache and allowing changes to pop up on the website quickly.

So what did that do to our cache hit percentage?

As you can see:

On weekday evenings, cache hit percentage increases from roughly 65% to roughly 85%
During weekends we usually see cache hit percentages between 90% and 100%.

Learnings

Initially we had missed one rather crucial point. It was easy to say, all these items expire next monday at 8am. However, if we had thought about that for a little bit longer, we would have realized this would cause a mass cache eviction at exactly that time. To prevent this and suddenly make this cache completely cold, we added some "fuzzing" (for the lack of a better word) to the expiration and started expiring within half an hour on either side of the actual threshold.

Getting started

I have created a tiny PHP library that you can use to start doing the same, although the concept is obviously not limited to PHP and porting the same functionality to any other language should be trivial. It can easily be installed through composer:

$ composer require erikbooij/cache-scheduler

The source can be checked out here:

ErikBooij / php-cache-scheduler

A tiny library to optimize cache TTLs for when your data will be inherently stale.

This Cache Scheduler is a library that allows you to vary your cache TTLs according to a self defined schedule. I've written a little blog post on why you might want to consider that.

Usage is simple:

install it with Composer:

$ composer require erikbooij/cache-scheduler

Create a schedule and scheduler:

$schedule = (new Schedule)
    ->requireUpToDateDataFrom(Schedule::MON, 8, 0)
    ->allowStaleDataFrom(Schedule::MON, 17, 30)
    ->requireUpToDateDataFrom(Schedule::TUE, 8, 0)
    ->allowStaleDataFrom(Schedule::TUE, 17, 30)
    ->requireUpToDateDataFrom(Schedule::WED, 8, 0)
    ->allowStaleDataFrom(Schedule::WED, 17, 30)
    ->requireUpToDateDataFrom(Schedule::THU, 8, 0)
    ->allowStaleDataFrom(Schedule::THU, 17, 30)
    ->requireUpToDateDataFrom(Schedule

…

View on GitHub

You can use it to create the schedule and calculate the TTL:

// Create the schedule using
//    ->requireUpToDateDataFrom(int $dayofTheWeek, int $hour, int $minute)
//    ->allowStaleDataFrom(int $dayofTheWeek, int $hour, int $minute)
$schedule = (new Schedule)
    ->requireUpToDateDataFrom(Schedule::MON, 8, 0)
    ->allowStaleDataFrom(Schedule::MON, 17, 30)
    ->requireUpToDateDataFrom(Schedule::TUE, 8, 0)
    ->allowStaleDataFrom(Schedule::TUE, 17, 30)
    ->requireUpToDateDataFrom(Schedule::WED, 8, 0)
    ->allowStaleDataFrom(Schedule::WED, 17, 30)
    ->requireUpToDateDataFrom(Schedule::THU, 8, 0)
    ->allowStaleDataFrom(Schedule::THU, 17, 30)
    ->requireUpToDateDataFrom(Schedule::FRI, 8, 0)
    ->allowStaleDataFrom(Schedule::FRI, 17, 30);

// Create the scheduler, passing it a SystemClock instance to interface with system time
$scheduler = (new Scheduler(new SystemClock))
    ->setSchedule($schedule)
    ->setExpirationSpread(ExpirationSpread::minutes(30));

// Use ->calculateTimeToLive(int $defaultTTL) to get the applicable TTL
$cache->set('cache-key', 'cache-value', $scheduler->calculateTimeToLive(3600));

The schedule and expiration spread can either be attached to the scheduler, for example when you want to provide the cache scheduler from your DI container, or passed to the ->calculateTimeToLive() method if their values depend on the execution location. The latter will override the former if both are used.

Conclusion

With little effort, it might be possible to extend the life of some of your cache entries. This depends on your completely on what data you're caching and how business processes are shaped around it. I'd love to hear your opinions on the idea and the implementation.

Keeping (large) data providers organized in PHPUnit

Erik Booij — Sun, 17 Feb 2019 11:36:29 +0000

When using data providers for you PHPUnit tests, it's easy to get carried away and add loads of different test cases with subtle differences in parameters. Let's not have a discussion on whether or not your subject under test is doing too many things if you need that many test cases, let's instead focus on how we can keep those test cases readable, understandable and manageable.

Assume say you are writing a test for the following piece of code, it sets three flags on a presentation model class depending on the current state of a session (should a primary action be shown, should a login button be shown and should an upgrade button be shown):

/**
 * @param ServerRequestInterface $request
 * @param PresentationModel      $presentationModel
 *
 * @return PresentationModel
 */
public function present(ServerRequestInterface $request, PresentationModel $presentationModel): PresentationModel
{
    // Set defaults
    $showLogin = true;
    $showUpgrade = false;
    $showPrimaryButton = true;

    $session = $this->sessionProvider->getSession();
    // Get users loggedIn status and default to false, in case the session is fresh
    $isLoggedIn = (bool)$session->get('loggedIn', false);

    if ($isLoggedIn) {
        $showLogin = false;
        $showUpgrade = true;
    }

    // Check if the user is logged in *and* a pro user
    if ($isLoggedIn && $session->get('subscriptionLevel', Subscription::LEVEL_BASIC) === Subscription::LEVEL_PRO) {
        $showUpgrade = false;
        $showPrimaryButton = false;
    }

    return $presentationModel->withVariables([
        'show_login' => $showLogin,
        'show_upgrade' => $showUpgrade,
        'show_primary_button' => $showLogin || $showPrimaryButton,
    ]);
}

We'll write three test cases for now:

It's a first visit to the website and the session is still empty
It's a visit where the logged in status is set to false and there is no subscriptionLevel set in the session
It's a visit where the logged in status is set to true and the subscriptionLevel status is PRO.

We'll be writing the data provider first:

/**
 * @return array[]
 */
public function sessionDataProvider(): array
{
    return [
        [
            [],
            true,
            false,
            true,
        ],
        [
            ['loggedIn' => false],
            true,
            false,
            true,
        ],
        [
            ['loggedIn' => true, 'subscriptionLevel' => Subscription::LEVEL_PRO],
            false,
            false,
            false,
        ],
    ];
}

Would you care to guess what those undescriptive booleans mean? You'll probably be able to figure that out, but it'll definitely take longer than necessary. How about we make it a bit more descripive? Remember that PHPUnit will simply take the values of the arrays returned by the data provider in the order in which they are defined, it doesn't care much about array keys.

/**
 * @return array[]
 */
public function sessionDataProvider(): array
{
    return [
        [
            'session' => [],
            'showLogin' => true,
            'showUpgrade' => false,
            'showPrimaryButton' => true,
        ],
        [
            'session' => ['loggedIn' => false],
            'showLogin' => true,
            'showUpgrade' => false,
            'showPrimaryButton' => true,
        ],
        [
            'session' => ['loggedIn' => true, 'subscriptionLevel' => Subscription::LEVEL_PRO],
            'showLogin' => false,
            'showUpgrade' => false,
            'showPrimaryButton' => false,
        ],
    ];
}

This way it's immediately clear what those values represent. When you're returning to this test, six months from now, you won't have to find the test implementation first to find the meaning of [true, false, true].

There is still room for improvement though. Even though it's clear what the variables mean, it's not immediately clear what we're testing. We could do better and one way of doing it would be to provide the data set with an array key too:

/**
 * @return array[]
 */
public function sessionDataProvider(): array
{
    return [
        'fresh-session' => [
            'session' => [],
            'showLogin' => true,
            'showUpgrade' => false,
            'showPrimaryButton' => true,
        ],
        'not-logged-in-subscription-unknown' => [
            'session' => ['loggedIn' => false],
            'showLogin' => true,
            'showUpgrade' => false,
            'showPrimaryButton' => true,
        ],
        'logged-in-pro-user' => [
            'session' => ['loggedIn' => true, 'subscriptionLevel' => Subscription::LEVEL_PRO],
            'showLogin' => false,
            'showUpgrade' => false,
            'showPrimaryButton' => false,
        ],
    ];
}

This again helps in the readability of your data provider. You won't ever have to think "why did I/someone add this test case, what is it even testing?".

The data provider is now pretty readable, let's quickly implement the test itself:

/**
 * @dataProvider sessionDataProvider
 *
 * @param array  $sessionData
 * @param bool   $showLogin
 * @param bool   $showUpgrade
 * @param bool   $showPrimaryButton
 *
 * @return void
 */
public function testHeaderShouldBeShownWithCorrectButtonAction(
    array $sessionData,
    bool $showLogin,
    bool $showUpgrade,
    bool $showPrimaryButton
): void {
    $this->sessionProvider->getSession()->willReturn(Session::fromData($sessionData));

    $presentationModel = $this->presenter->present($this->request, $this->presentationModel);

    $this->assertEquals($showLogin, $presentationModel->getVariable('show_login'));
    $this->assertEquals($showUpgrade, $presentationModel->getVariable('show_upgrade'));
    $this->assertEquals($showPrimaryButton, $presentationModel->getVariable('show_primary_button'));
}

Very straightforward and you'll see the name of the data set is now also available in the test output, immediately indicating which test case failed:

There were 3 failures:

1) TestClass::testHeaderShouldBeShownWithCorrectButtonAction with data set "fresh-session" (array(), true, false, true)
Failed asserting that false matches expected true.

/path/to/TestClass.php:88

2) TestClass::testHeaderShouldBeShownWithCorrectButtonAction with data set "not-logged-in-subscription-unknown" (array(false), true, false, true)
Failed asserting that false matches expected true.

/path/to/TestClass.php:88

3) TestClass::testHeaderShouldBeShownWithCorrectButtonAction with data set "logged-in-pro-user" (array(true, 'pro'), false, false, true)
Failed asserting that false matches expected true.

/path/to/TestClass.php:88

We're getting extremely close now, but we can step it up one more notch, because Failed asserting that false matches expected true is still not as helpful in debugging as it could be, we have multiple assertions that could expect true but are now getting false. Let's improve our setup again: instead of using a description of the test case as an array key in the data provider, let's make it part of the returned data set, because fortunately PHPUnit allows passing a custom error message to assertions.

/**
 * @return array[]
 */
public function sessionDataProvider(): array
{
    return [
        [
            'testCase' => 'Fresh Session',
            'session' => [],
            'showLogin' => true,
            'showUpgrade' => false,
            'showPrimaryButton' => true,
        ],
        [
            'testCase' => 'Not Logged In & Subscription Unknown',
            'session' => ['loggedIn' => false],
            'showLogin' => true,
            'showUpgrade' => false,
            'showPrimaryButton' => true,
        ],
        [
            'testCase' => 'Logged In & PRO subscription',
            'session' => ['loggedIn' => true, 'subscriptionLevel' => Subscription::LEVEL_PRO],
            'showLogin' => false,
            'showUpgrade' => false,
            'showPrimaryButton' => false,
        ],
    ];
}

/**
 * @dataProvider sessionDataProvider
 *
 * @param string $testCase
 * @param array  $sessionData
 * @param bool   $showLogin
 * @param bool   $showUpgrade
 * @param bool   $showPrimaryButton
 *
 * @return void
 */
public function testHeaderShouldBeShownWithCorrectButtonAction(
    string $testCase,
    array $sessionData,
    bool $showLogin,
    bool $showUpgrade,
    bool $showPrimaryButton
): void {
    $this->sessionProvider->getSession()->willReturn(Session::fromData($sessionData));

    $presentationModel = $this->presenter->present($this->request, $this->presentationModel);

    $this->assertEquals(
        $showLogin,
        $presentationModel->getVariable('show_login'),
        "`show_login` set incorrectly for test case \"{$testCase}\""
    );
    $this->assertEquals(
        $showUpgrade,
        $presentationModel->getVariable('show_upgrade'),
        "`show_upgrade` set incorrectly for test case \"{$testCase}\""
    );
    $this->assertEquals(
        $showPrimaryButton,
        $presentationModel->getVariable('show_primary_button'), 
        "`show_primary_button` set incorrectly for test case \"{$testCase}\""
    );
}

This would be the output if the test fails now:

There were 3 failures:

1) TestClass::testHeaderShouldBeShownWithCorrectButtonAction with data set #0 ('Empty Session', array(), true, false, true)
`show_login` set incorrectly for test case "Empty Session"
Failed asserting that false matches expected true.

/path/to/TestClass.php:88

2) TestClass::testHeaderShouldBeShownWithCorrectButtonAction with data set #1 ('Not Logged In', array(false), true, false, true)
`show_login` set incorrectly for test case "Not Logged In"
Failed asserting that false matches expected true.

/path/to/TestClass.php:88

3) TestClass::testHeaderShouldBeShownWithCorrectButtonAction with data set #2 ('Not Logged In & Basic Subscription', array(false, 'basic'), true, false, true)
`show_login` set incorrectly for test case "Not Logged In & Basic Subscription"
Failed asserting that false matches expected true.

/path/to/TestClass.php:88

This makes it very clear at first glance:

which test failed
which test case for that test failed
which assertion in that particular scenario failed

Conclusion

Describe your test cases and test parameters in order to be able to still maintain/read/debug your test somewhere in the future. Also make sure you help your future self by providing descriptive messages when your tests/assertions fail. You'll thank your past self when the time comes.