Forem: Pascal Widdershoven

Django & Webpack - without any plugins

Pascal Widdershoven — Mon, 20 Apr 2020 05:31:25 +0000

This post explores setting up Webpack in a Django project with minimal friction. The defacto solution for this, django-webpack-loader, is too heavy handed in my opinion. This post aims to provide a guide to setup Webpack in Django, without any plugins and leveraging both Webpack's and Django's strengths.

Django-webpack-loader

django-webpack-loader seems to be the defacto way of setting up Webpack in Django projects. I tried using it but didn't like that:

It requires a non-standard manifest plugin (that was also buggy for me).
It provides a set of custom Django template helpers, instead of levering the excellent Django built-in static functionality.
It required a bunch of configuration in Django, to make the custom helpers work.

It feels like a very heavy handed solution requiring both a Webpack plugin and a Django plugin.

Disclaimer: While I'm criticizing django-webpack-loader here this is not meant to knock on the hard work that the maintainer(s) have undoubtedly put into this.

Vanilla Django & Webpack

Making Django and Webpack play nice together turns out to be pretty straightforward, without needing any Webpack or Django plugins.

Django has built-in support for handling static assets. Django can serve static assets in development and compress files and hash filenames during production deployment. Webpack can do this too (and much more), but it's much more convenient to let Webpack only worry about producing your assets and Django about handling your assets.

In practise this works as follows:

1. Use Django's default Static support

Refer to the Django docs for details. Important settings are:

STATICFILES_DIRS
STATIC_URL
STATICFILES_STORAGE - Use ManifestStaticFilesStorage to let Django hash filenames for cachability.

2. Configure Webpack to write files to `STATICFILES_DIRS`

You can use any Webpack configuration you like. Only a couple of settings are important:

Have Webpack write to a directory in STATICFILES_DIRS
Configure output.publicPath to match Django's STATIC_URL
Don't let Webpack hash filenames (except for chunks, see below)
Make sure webpack-dev-server writes files to disk (so Django can serve them in development)

// Relevant parts of webpack.config.js
output: {
  path: path.resolve(__dirname, "myapp/static"), // Should be in STATICFILES_DIRS
  publicPath: "/static/", // Should match Django STATIC_URL
  filename: "[name].js", // No filename hashing, Django takes care of this
  chunkFilename: "[id]-[chunkhash].js", // DO have Webpack hash chunk filename, see below
},
devServer: {
  writeToDisk: true, // Write files to disk in dev mode, so Django can serve the assets
},

Now, during development you run both webpack-dev-server and the Django server. You can use a Procfile and a tool like Goreman to conveniently do this without having to open two terminals. Webpack writes files into STATICFILES_DIRS and Django serves the files.

To include an asset produced by Webpack you use the Django static template tag:

{% load static %}
<link rel="stylesheet" type="text/css" href="{% static 'app.css' %}">
<script type="text/javascript" src="{% static 'app.js' %}"></script>

For production deployments you first have Webpack build your assets and then use manage.py collectstatic as usual. Presto! Just plain Django and Webpack, nice and simple.

Webpack chunks (dynamic imports)

As I mentioned above it's important to do let Webpack hash chunk filenames. By default Webpack uses numeric chunk ids to refer to chunks at runtime. When Django's collectstatic runs, it hashes the filenames and then looks for unhashed references in other files to replace them.

For example if you have a stylesheet that refers to a file foo.jpg and Django renames foo.jpg to foo.695e1b313f34.jpg, it will replace the foo.jpg reference in the stylesheet with foo.695e1b313f34.jpg.

Webpack at runtime refers to chunks by their numeric chunk id by default and as such collectstatic will not correctly recognize the chunk filenames. By letting Webpack hash the chunk filenames we get properly hashed chunk filenames, so these files can be cached properly by the browser.

Example project

I've created a demo project for this setup on GitHub here so you can see it in action. It's also deployed on Heroku here.

If this was useful to you I'd love to hear from you!

Kubernetes workshop in a box

Pascal Widdershoven — Thu, 09 Apr 2020 08:26:05 +0000

We recently organised an internal Kubernetes workshop at our office. Each participant got their own 3-node Kubernetes cluster and all they needed was a laptop with an SSH client, no other software had to be installed! How? Read on to find out!

Why

I wanted participants to have a real cluster to experiment with, but didn't want to ask everyone to install Minikube. It's heavy weight, can be a bit of a hassle to setup and I just didn't want people to have to mess with stuff like this. Workshops should have a low as possible barrier to entry.

To pull this off I used Kind (Kubernetes In Docker) + a bit of scripting and Unix magic! 💪

The setup

The setup works as follows:

A single, beefy cloud server is used to host all Kubernetes clusters.
Kind creates true Kubernetes clusters inside Docker containers. This way, each cluster is fully isolated from the other clusters and clusters can run side-by-side on a single host machine. Every cluster consists of one master and two worker nodes.
The clusters are (automatically) provisioned in advance to the workshop, as creating the clusters is quite resource intensive and takes a couple of minutes per cluster.
During the workshop participants can claim their own cluster via SSH and then access their private cluster via a new SSH session.

You can see the process from a participant perspective in action below:

Hardware & costs

A Kind cluster is quite resource intensive, though in part it depends on the workloads your workshop participants are running of course. For our workshop with 15 participants we used an AWS m5.8xlarge machine. This machine type has 32 VCPU's and 128GB of RAM. It costs $1,712 per hour (in eu-west-1) and we needed it only for about 3 hours so that's about $5 total 🙂.

In hindsight, this machine was way over provisioned for this number of clusters, so you could go even cheaper 😀

You can use this too to run your own Kubernetes workshops!

We've open-sourced the tooling we built to do this here: https://github.com/kabisa/k8s-workshop-in-a-box. Feel free to use it and if you do please reach out to me I'd love to hear from you! ❤️

Localizing your app with Flutter's new gen_l10n tool

Pascal Widdershoven — Sun, 08 Mar 2020 23:00:00 +0000

I'm recently getting into Flutter so I'm still figuring out some common practises. I was surprised to find that the recommended approach for internationalization was quite cumbersome and laborious.

Fortunately, the Flutter team is working on a simplified i18n process that can already be used today and in this post I'll show you how!

Proposed new i18n process

The proposed workflow for this new i18n process is as follows:

Start with handwritten .arb files. One of these files will contain the standard per-message metadata required by the intl package.
A new tool (gen_l10n) will be used to generate a single class that contains one method per message.
Message lookup is based on the inherited locale, using the current BuildContext:

MyLocalizations.of(context).myMessage

Compared to some other solutions available on https://pub.dev this has a few advantages in my opinion:

Since this solution uses code generation, the compiler will help you use your localized messages correctly. This also means that your IDE can autocomplete translations keys.
It's based on the official intl package.
It uses a pretty versatile localization format (ARB).
It's an official solution encouraged by the Flutter team.

In the following section I'll describe how to adopt this workflow for an existing project.

Adopt gen_l10n for an existing project

To get started we need a few new dependencies.

Add gen_l10n dependencies

The code generated by gen_l10n depends on the intl and flutter_localizations package, so add these to your dependencies:

# pubspec.yaml
dependencies:
  # ... other dependencies ...
  flutter_localizations:
    sdk: flutter
  intl: ^0.16.0

The gen_l10n tool itself depends on intl_translation, so add this as a development dependency:

# pubspec.yaml
dev_dependencies:
  # ... other dev dependencies ...
  intl_translation: ^0.17.9

Define translations

Ok, so now we have all dependencies we need, let's start by defining some localized messages.

Create a directory in your project to store your localized messages. This can be any directory you like, I'm using lib/i18n:

mkdir lib/i18n

In this directory create a file called messages_en_US.arb. The name messages here can be anything you want, but the last part of the file name should refer to the locale of the messages inside.

As you can see this file is an ARB file, an Application Resource Bundle. It's a localization resource format designed by Google. You can read the specification here.

Now to define localised messages we'll start by adding the messages we want to localize to the messages_en_US.arb file we created earlier. I won't dive into everything the ARB format supports, but it's a pretty versatile format so be sure to read the specification I linked above!

{
  "greeting": "Hello, world!",
  "@greeting": {
    "type": "text",
    "description": "A friendly greeting."
  },

  "newMessages": "{newMessages, plural, =0 {No new messages} =1 {One new message} other {{newMessages} new messages}}",
  "@newMessages": {
    "type": "text",
    "description": "Number of new messages in inbox.",
    "placeholders": {
      "newMessages": {}
    }
  }
}

As you can see for every message we want to localize we add both a key and a meta key to the file. The key is how you will refer to the localized message from your code later and the meta key is that same key prefixed with @. These meta keys only need to be added in one of the locales; the locale you will use as a template.

In the example above we have both a simple message greeting and a more complex, pluralized, message that changes depending on a piece of context'; the number of new messages in this case.

Now for every other language we want to support we need to create matching files with the desired locale in the lib/i18n directory we created earlier. For example I'm going to translate these messages in Dutch so I'll add a file named messages_nl_NL.arb with the following contents:

{
  "greeting": "Hallo, wereld!",
  "newMessages": "{newMessages, plural, =0 {Geen nieuwe berichten} =1 {Één nieuw bericht} other {{newMessages} nieuwe berichten}}"
}

As you can see in this case the file only contains the keys themselves and no metadata.

Generate localization classes

With everything prepared we can now use the gen_l10n tool to generate the Dart classes we'll use to access these localized messages in our code.

As the gen_l10n is still experimental it's not yet distributed as a package on the package registry. Instead it's bundled with the Flutter SDK itself. The tool is located in the Flutter SDK at dev/tools/localization/gen_l10n.dart. In my case the Flutter SDK is installed in /Users/Pascal/.asdf/installs/flutter/1.12.13+hotfix.8-stable/ so the full path to the tool is /Users/Pascal/.asdf/installs/flutter/1.12.13+hotfix.8-stable/dev/tools/localization/gen_l10n.dart.

From your terminal, run the gen_l10n tool from the root of your Flutter project as follows:

flutter pub run "/path/to/flutter/sdk/dev/tools/localization/gen_l10n.dart" --arb-dir lib/i18n --template-arb-file=messages_en_US.arb --output-localization-file=translations.dart --output-class=Translations

(Be sure to replace /path/to/flutter/sdk above with the actual path to your Flutter SDK!)

This will generate a couple of files in lib/i18n:

lib/i18n
├── messages_all.dart
├── messages_en_US.dart
├── messages_nl_NL.dart
├── messages_en_US.arb
├── messages_nl_NL.arb
└── translations.dart

All these files should be checked into version control. The only files you manually edit, however, are the ARB files. Don't touch the generated Dart files :-)

Make localized messages available via BuildContext

To make Flutter aware of our localized messages we need to configure the localizationsDelegates and supportedLocales properties on the MaterialApp widget.

In your lib/main.dart file import the generated translations file:

import 'package:hello_world/i18n/translations.dart';

And configure the MaterialApp widget to pick up your translations:

MaterialApp(
  // ... other properties ...
  localizationsDelegates: Translations.localizationsDelegates,
  supportedLocales: Translations.supportedLocales
)

Use localized messages

With all these out of the way, we can now actually use our localized messages.

From any widget in the widget tree below MaterialApp you can now access your localized messages.

Start by importing the generated translations.dart file:

import 'package:hello_world/i18n/translations.dart';

And use the messages defined in the ARB files:

class MyLocalizedWidget extends StatelessWidget {
  @override
  Widget build(BuildContext context) {
    return Column(
      children: <Widget>[
        Text(Translations.of(context).greeting), // "greeting"
        Text(Translations.of(context).newMessages(10)) // "newMessages"
      ],
    );
  }
}

The compiler will check you're using your messages correctly and the IDE will help you with great auto completion:

Happy i18n-ing! 🎉

Extending Flutter Driver with custom commands

Pascal Widdershoven — Mon, 17 Feb 2020 00:00:00 +0000

Flutter Driver is a library to write end-to-end integration tests for Flutter apps. It’s similar to Selenium WebDriver (for web apps), Espresso (for native Android apps) and Earl Grey (for native iOS apps). It works by instrumenting the Flutter app, deploying it on a real device or emulator and then ‘driving’ the application using a suite of Dart tests.

A typical, basic Flutter Driver test looks like this:

import 'package:flutter_driver/flutter_driver.dart';
import 'package:test/test.dart';

void main() {
  group('Counter App', () {
    final counterTextFinder = find.byValueKey('counter');
    final buttonFinder = find.byValueKey('increment');

    FlutterDriver driver;

    // Connect to the Flutter driver before running any tests.
    setUpAll(() async {
      driver = await FlutterDriver.connect();
    });

    // Close the connection to the driver after the tests have completed.
    tearDownAll(() async {
      driver?.close();
    });

    test('starts at 0', () async {
      expect(await driver.getText(counterTextFinder), "0");
    });

    test('increments the counter', () async {
      await driver.tap(buttonFinder);

      expect(await driver.getText(counterTextFinder), "1");
    });
  });
}

In the setUpAll hook a connection between the test and the running application is setup via the Flutter Driver API. This works because the application is instrumented with a Flutter Driver extension; basically an API injected into your app that can receive requests from our tests to “drive” the application.

The instrumentation of the Flutter app works by wrapping your app’s main function like this:

import 'package:flutter_driver/driver_extension.dart';
import 'package:my_app/main.dart' as app;

void main() {
  enableFlutterDriverExtension(); // <-- ENABLE INSTRUMENTATION
  app.main();
}

Flutter Driver supports a handful of API’s to communicate with the running app. For example getText, tap, waitFor etc. For me, coming from Nightwatch.js, the number of things that can be done to drive the application is quite limited.

Fortunately it’s possible to extend Flutter Driver to support custom commands. These commands allow you to communicate between your tests and the application and are also the foundation for all of Flutter Driver’s own API’s like getText, tap etc.

Extending Flutter Driver

To extend Flutter Driver with a custom command we need to provide a DataHandler. As the docs say:

Optionally you can pass a [DataHandler] callback. It will be called if the
test calls [FlutterDriver.requestData].

Flutter Driver will pass whatever is sent from test with driver.requestData(...) to the DataHandler. DataHandler only supports sending and receiving Strings, so you might want to encode your messages using JSON.

To demonstrate this, let’s implement a handler to navigate back to the root route of our app. This way we can ensure that every test starts from the root page of our application.

The first step is to provide a DataHandler to Flutter Driver:

enableFlutterDriverExtension(handler: (payload) async {
  print(payload);
});

The handler will receive a String payload and can optionally return a String response.

For the sake of simplicity let’s use a String as payload for now:

enableFlutterDriverExtension(handler: (payload) async {
  if(payload == "navigate_to_root") {
    // do something smart here
  }
});

From here, we need to implement something that will allow us to navigate to the root of our app. I’m not sure if the following is necessarily the best way to do this (if you know a better way please let me know!), but it works and is relatively straightforward.

We’ll use a NavigationObserver to get a hold of the NavigatorState, which we can use to push and pop routes. We need to be able to pass in a NavigationObserver from our test entry point, so we can access it when we receive a command to navigate to root.

Change the main function of your app as follows:

void main() {
  _main(null);
}

void mainTest(NavigatorObserver navigatorObserver) {
  _main(navigatorObserver);
}

void _main(NavigatorObserver navigatorObserver) {
  runApp(MyApp(
    navigatorObserver: navigatorObserver,
  ));
}

class MyApp extends StatelessWidget {
  final NavigatorObserver navigatorObserver;

  const MyApp({Key key, this.navigatorObserver}) : super(key: key);

  @override
  Widget build(BuildContext context) {
    return MaterialApp(
      // all other MaterialApp initialisation here
      navigatorObservers: navigatorObserver == null ? [] : [navigatorObserver],
    );
  }
}

This allows us to hook up a NavigationObserver from our test wrapper like so:

import 'package:flutter/material.dart';
import 'package:flutter_driver/driver_extension.dart';
import 'package:my_app/main.dart' as app;

void main() {
  final navigationObserver = NavigatorObserver();

  enableFlutterDriverExtension(handler: (payload) async {
    if (payload == "navigate_to_root") {
      navigationObserver.navigator.popUntil(ModalRoute.withName('/'));
    }

    return null;
  });

  app.mainTest(navigationObserver);
}

Now from our tests we can send our custom command, for example in a setUp hook:

void main() {
  FlutterDriver driver;

  setUpAll(() async {
    driver = await FlutterDriver.connect();
  });

  tearDownAll(() async {
    driver?.close();
  });

  setUp(() async {
    await driver.requestData("navigate_to_root");
  });   

  /* Actual tests here */
}

This will make sure that before every test, the app navigates back to the root route no matter where we navigated to in our tests.

Of course, this is just an example of how you can implement communication between your Driver tests and your app. If you’re going to send more complex commands that require arguments you might want to send JSON data, but I’ll leave that as an exercise to you, dear reader ;-)

Generating CSS from scratch with PostCSS

Pascal Widdershoven — Mon, 06 Jan 2020 00:00:00 +0000

For a soon-to-be open-sourced project I had to generate a CSS stylesheet based on user input.

Initially I started out with good ‘ol string templating and interpolation but it soon became pretty complex, as I needed to conditionally add certain properties and declarations.

It occurred to me that what I wanted was a representation of the CSS structure in code, an Abstract Syntax Tree (AST). That would allow me to build up the CSS tree structure in code and turn it into a string later on.

I decided to see if I could use PostCSS, since I figured it must be turning CSS into an AST already.

PostCSS

PostCSS is “A tool for transforming CSS with JavaScript”. It’s widely used in the industry for things like auto-prefixing CSS.

As the tag line says, PostCSS main business is transforming CSS, not generating it from scratch as I wanted to do. Looking at the PostCSS codebase I noticed a test named it allows to build own CSS so I figured it should be possible!

Fast forward a couple of hours diving through the PostCSS codebase, reading the API docs and several PostCSS plugins, I had a working solution.

Generating CSS from scratch

To generate CSS with PostCSS you first need to build up an AST:


1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21

const postcss = require("postcss");

const fontFamily = "My Family";
const root = postcss.root();

const bodyRule = postcss.rule({ selector: "body" }).append(
  postcss.decl({
    prop: "font-family",
    value: `"${fontFamily}"`
  })
);

const fontFace = postcss.atRule({ name: "font-face" }).append([
   postcss.decl({ prop: "font-family", value: `"${fontFamily}"` }),
   postcss.decl({ prop: "src", value: "url(./fonts/myfont.woff2)" })
]);

root.append([
  fontFace,
  bodyRule
]);

The CSS code can then be generated from the CSS with root.toString(), resulting in this:


1
2
3
4
5
6
7

@font-face {
    font-family: "My Family";
    src: url(./fonts/myfont.woff2)
}
body {
    font-family: "My Family"
}

A couple of things to note:

PostCSS does not provide much help in generating proper declaration values out of the box. For example quoting a font-family value in case it contains multiple words is not handled by PostCSS automatically.
It’s easy to conditionally generate properties, rules, declarations etc. The API follows a typical builder pattern, which makes it easy to conditionally call append, or not.
The CSS output is opinionated. There are no semicolons after the last declaration and no newlines between rules.

Fortunately, PostCSS is architected quite well and allows you to provide your own “Stringifier”. I didn’t find much documentation or guidance on this though, but after a bit of code diving I settled on this:


1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27

const Stringifier = require("postcss/lib/stringifier");

class PrettyStringifier extends Stringifier {
  static new() {
    return (node, builder) => {
      let str = new this(builder);
      str.stringify(node);
    };
  }

  constructor(builder) {
    super(builder);
  }

  rule(node) {
    if (node.prev()) {
      // Add a newline after a rule, if it's preceded by another rule.
      this.builder("\n", node);
    }

    return super.rule(node);
  }

  decl(node) {
    return super.decl(node, true /* force semicolon */);
  }
}

As you can see, this is inheriting most of the default behaviour except around rules and declarations.

The custom Stringifier can be used like this:


1

root.toString(PrettyStringifier.new());

Conclusion

All in all I achieved what I had to do, but it didn’t feel like PostCSS was particularly well suited for this use-case as everything in PostCSS is geared towards transforming CSS.

Nevertheless it was a nice experiment. Having used PostCSS a lot via Autoprefixer it was interesting to dig into the internals of PostCSS to see how everything works!

If you have any tips on using PostCSS for this purpose, or perhaps other tools that might be better suited for this, please let me know!

Generating CSS from scratch with PostCSS

Pascal Widdershoven — Mon, 06 Jan 2020 00:00:00 +0000

For a soon-to-be open-sourced project I had to generate a CSS stylesheet based on user input.

Initially I started out with good ‘ol string templating and interpolation but it soon became pretty complex, as I needed to conditionally add certain properties and declarations.

I decided to see if I could use PostCSS, since I figured it must be turning CSS into an AST already.

PostCSS

PostCSS is “A tool for transforming CSS with JavaScript”. It’s widely used in the industry for things like auto-prefixing CSS.

Fast forward a couple of hours diving through the PostCSS codebase, reading the API docs and several PostCSS plugins, I had a working solution.

Generating CSS from scratch

To generate CSS with PostCSS you first need to build up an AST:

const postcss = require("postcss");

const fontFamily = "My Family";
const root = postcss.root();

const bodyRule = postcss.rule({ selector: "body" }).append(
  postcss.decl({
    prop: "font-family",
    value: `"${fontFamily}"`
  })
);

const fontFace = postcss.atRule({ name: "font-face" }).append([
   postcss.decl({ prop: "font-family", value: `"${fontFamily}"` }),
   postcss.decl({ prop: "src", value: "url(./fonts/myfont.woff2)" })
]);

root.append([
  fontFace,
  bodyRule
]);

The CSS code can then be generated from the CSS with root.toString(), resulting in this:

@font-face {
    font-family: "My Family";
    src: url(./fonts/myfont.woff2)
}
body {
    font-family: "My Family"
}

A couple of things to note:

PostCSS does not provide much help in generating proper declaration values out of the box. For example quoting a font-family value in case it contains multiple words is not handled by PostCSS automatically.
It’s easy to conditionally generate properties, rules, declarations etc. The API follows a typical builder pattern, which makes it easy to conditionally call append, or not.
The CSS output is opinionated. There are no semicolons after the last declaration and no newlines between rules.

const Stringifier = require("postcss/lib/stringifier");

class PrettyStringifier extends Stringifier {
  static new() {
    return (node, builder) => {
      let str = new this(builder);
      str.stringify(node);
    };
  }

  constructor(builder) {
    super(builder);
  }

  rule(node) {
    if (node.prev()) {
      // Add a newline after a rule, if it's preceded by another rule.
      this.builder("\n", node);
    }

    return super.rule(node);
  }

  decl(node) {
    return super.decl(node, true /* force semicolon */);
  }
}

As you can see, this is inheriting most of the default behaviour except around rules and declarations.

The custom Stringifier can be used like this:

root.toString(PrettyStringifier.new());

Conclusion

All in all I achieved what I had to do, but it didn’t feel like PostCSS was particularly well suited for this use-case as everything in PostCSS is geared towards transforming CSS.

Nevertheless it was a nice experiment. Having used PostCSS a lot via Autoprefixer it was interesting to dig into the internals of PostCSS to see how everything works!

If you have any tips on using PostCSS for this purpose, or perhaps other tools that might be better suited for this, please let me know!

HTTP Caching Gotcha: Heuristic Freshness

Pascal Widdershoven — Mon, 09 Dec 2019 14:59:45 +0000

HTTP caching gotcha: Heuristic Freshness

I recently ran into an issue where after deployment of an SPA (Single Page Application), a situation would occur where the page looked broken because CSS could not be loaded.

During analysis of the issue I ran into a thing I had never heard of: Heuristic Freshness.

Context

For context, the application is a typical SPA. The compiled application consists of a bunch of files looking something like this (simplified):

├── index.html
    └── styles.4a3f9848037579025b00.css
    └── main.31f7dadf6d2b01fc08c7.js

The browser loads index.html, which includes various CSS and JS files.

Caching related headers for the CSS and JS looked like this:

Date: Fri, 06 Dec 2019 13:09:03 GMT
Etag: "31957a05a5df3c3b315b728b40b6e10e"
Last-Modified: Mon, 02 Dec 2019 14:12:09 GMT
Expires:    Sat, 07 Dec 2019 03:09:03 GMT

index.html :

Date:   Fri, 06 Dec 2019 13:09:03 GMT
Etag: "c4238385fe77f826b5584fed1f1f1659"
Last-Modified: Tue, 03 Dec 2019 10:50:54 GMT

At first sight things looked okay, but then I noticed there weren't any Cache-Control or Expires on the index.html file.

I'm quite well aware what all these caching headers do when they are present, but I wasn't sure what would happen if they are not present 🤔.

Heuristic Freshness

This brings us to Heuristic Freshness. The HTTP specification defines that, when a server does not explicitly specify expiration times, the client (browser) can use heuristics to estimate a plausible expiration time itself.

How exactly this 'plausible' expiration time is determined is left up to the client, but it seems that in practise most browsers use the following algorithm: (now() - Last-Modified) * 0.10. This means a couple of things:

When you don't have any Cache-Control or Expires headers, the browser calculates this plausible expiry time itself.
Once your assets are cached by browsers, there's no way for you to evict them from the cache.
The files will be cached longer as time passes after deployment (assuming your Last-Modified headers reflect the time of the last deployment).

As you can see, this can result in some pretty nasty caching issues that are hard to diagnose as the duration for which files are cached will differ case by case depending on time and potentially browser used.

Cache-Control and Expires headers to the rescue!

The lesson I take away from this is that it's crucial to set either Cache-Control or Expires, to ensure you control how long files can be cached by the browser.

For single page apps like I outlined above where you have an index.html and a bunch of assets with hashed filenames, the following is a good, safe practise:

index.html:

Cache-control: private, max-age=0, no-cache

This will ensure that the browser will never use a cached copy of your index.html, without checking with the server if the cache is still valid via a conditional GET request.

For other assets with hashed filenames, you want the opposite:

Cache-Control: public, max-age=31557600

These files can be cached for a long, long time, as when they change their filenames will change as well.

With this in place the following will happen during and after a deployment:

A new index.html will be uploaded to the server.
When a user loads your application, the browser will send a conditional GET request If-Modified-Since: <previous Last-Modified value> , and the server will respond with the new version of your index.html, since the file was modified by the deployment. This happens because you've instructed the browser to always verify if the cached page can be used, using the Cache-Control header.
The browser will no longer us the old cached page.
The browser will store the new page in cache and will continue sending conditional GET requests in the future, to verify if the cached page can still be used.

Problem solved!

Resources

For more information on HTTP caching (especially about what headers do when they are present) see:

HTTP caching gotcha: Heuristic Freshness

Pascal Widdershoven — Mon, 09 Dec 2019 00:00:00 +0000

I recently ran into an issue where after deployment of an SPA (Single Page Application), a situation would occur where the page looked broken because CSS could not be loaded. During analysis of the issue I ran into a thing I had never heard of: Heuristic Freshness.

Context

For context, the application is a typical SPA. The compiled application consists of a bunch of files looking something like this (simplified):


1
2
3

├── index.html
    └── styles.4a3f9848037579025b00.css
    └── main.31f7dadf6d2b01fc08c7.js

The browser loads index.html, which includes various CSS and JS files.

Caching related headers for the CSS and JS looked like this:


1
2
3
4

Date: Fri, 06 Dec 2019 13:09:03 GMT
Etag: "31957a05a5df3c3b315b728b40b6e10e"
Last-Modified: Mon, 02 Dec 2019 14:12:09 GMT
Expires:    Sat, 07 Dec 2019 03:09:03 GMT

index.html :


1
2
3

Date:   Fri, 06 Dec 2019 13:09:03 GMT
Etag: "c4238385fe77f826b5584fed1f1f1659"
Last-Modified: Tue, 03 Dec 2019 10:50:54 GMT

At first sight things looked okay, but then I noticed there weren’t any Cache-Control or Expires on the index.html file.

I’m quite well aware what all these caching headers do when they are present, but I wasn’t sure what would happen if they are not present 🤔.

Heuristic Freshness

How exactly this ‘plausible’ expiration time is determined is left up to the client, but it seems that in practise most browsers use the following algorithm: (now() - Last-Modified) * 0.10. This means a couple of things:

When you don’t have any Cache-Control or Expires headers, the browser calculates this plausible expiry time itself.
Once your assets are cached by browsers, there’s no way for you to evict them from the cache.
The files will be cached longer as time passes after deployment (assuming your Last-Modified headers reflect the time of the last deployment).

Cache-Control and Expires headers to the rescue!

The lesson I take away from this is that it’s crucial to set either Cache-Control or Expires, to ensure you control how long files can be cached by the browser.

For single page apps like I outlined above where you have an index.html and a bunch of assets with hashed filenames, the following is a good, safe practise:

index.html:


1

Cache-control: private, max-age=0, no-cache

This will ensure that the browser will never use a cached copy of your index.html, without checking with the server if the cache is still valid via a conditional GET request.

For other assets with hashed filenames, you want the opposite:


1

Cache-Control: public, max-age=31557600

These files can be cached for a long, long time, as when they change their filenames will change as well.

With this in place the following will happen during and after a deployment:

A new index.html will be uploaded to the server.
When a user loads your application, the browser will send a conditional GET request If-Modified-Since: <previous Last-Modified value> , and the server will respond with the new version of your index.html, since the file was modified by the deployment. This happens because you’ve instructed the browser to always verify if the cached page can be used, using the Cache-Control header.
The browser will no longer us the old cached page.
The browser will store the new page in cache and will continue sending conditional GET requests in the future, to verify if the cached page can still be used.

Problem solved!

Resources

For more information on HTTP caching (especially about what headers do when they are present) see:

k3s HTTPS with Let’s Encrypt

Pascal Widdershoven — Tue, 02 Jul 2019 00:00:00 +0000

K3s is a Certified Kubernetes distribution designed for production workloads in unattended, resource-constrained, remote locations or inside IoT appliances.
It provides a ready to go Kubernetes instance packaged a single binary.

This guide will show you how to easily set up k3s with HTTPS via Let’s Encrypt.

(READMORE)

K3s comes with Traefik out of the box. Traefik itself supports automatic HTTPS via Let’s Encrypt, but setting this up with k3s turned out to be pretty cumbersome. Instead using the excellent cert-manager add-on, it's a breeze!

0: Setup k3s.

This guide assumes you have a working k3s instance and kubectl configured to talk to your k3s instance. If you haven't already just follow the docs and you'll be up and running in minutes.

1. Install Helm

Helm is a package manager for Kubernetes. It consists of a local client and a server component.
Installing Helm is quite straightfoward.

Install the client:

$ brew install kubernetes-helm

Create the service account so Helm can interact with your k3s instance:

$ kubectl create serviceaccount tiller --namespace=kube-system

Grant it admin privileges:

$ kubectl create clusterrolebinding tiller-admin --serviceaccount=kube-system:tiller --clusterrole=cluster-admin

Note: granting admin privileges to Helm might not be a good idea depending on how your cluster is setup. Read here for more details.

Install Helm on k3s:

$ helm init --service-account=tiller

2. Install cert-manager

Alright, so now we have Helm installed let's move on to installing cert-manager.
cert-manager is a Kubernetes add-on to automate the management and issuance of TLS certificates from various issuing sources, amongst which Let’s Encrypt.

Start with installing cert-managers CRDs:

$ kubectl apply -f https://raw.githubusercontent.com/jetstack/cert-manager/release-0.8/deploy/manifests/00-crds.yaml

Add the cert-manager Helm repo:

$ helm repo add jetstack https://charts.jetstack.io && helm repo update

Install cert-manager:

$ helm install \
  --name cert-manager \
  --namespace cert-manager \
  --version v0.8.1 \
  jetstack/cert-manager

This might take a minute or so. Verify the installation by checking all cert-manager pods are running:

$ kubectl get all -n cert-manager

3. Configure cert-manager

With cert-manager installed we need to configure it to use Let’s Encrypt, by creating a certificate issuer:

$ cat <<EOF > letsencrypt-prod-issuer.yaml
apiVersion: certmanager.k8s.io/v1alpha1
kind: Issuer
metadata:
 name: letsencrypt-prod
spec:
 acme:
   # The ACME server URL
   server: https://acme-v02.api.letsencrypt.org/directory
   # Email address used for ACME registration, update to your own.
   email: user@example.com
   # Name of a secret used to store the ACME account private key
   privateKeySecretRef:
     name: letsencrypt-prod
   # Enable the HTTP-01 challenge provider
   http01: {}
EOF

And apply it:

$ kubectl apply -f letsencrypt-prod-issuer.yaml

By now, cert-manager is ready to provision Let’s Encrypt certificates as we need it.

4. Deploy a service with automatic HTTPS

To try this out let's deploy a simple service. You should have a DNS name pointed to your k3s cluster for this to work. The example below uses bootcamp.k3s.example.org, be sure to update this to your own hostname!

Create the manifest:

$ cat <<EOF > k8s-bootcamp.yaml
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: k8s-bootcamp
  namespace: default
spec:
  replicas: 1
  selector:
    matchLabels:
      app: k8s-bootcamp
  template:
    metadata:
      labels:
        app: k8s-bootcamp
    spec:
      containers:
      - name: k8s-bootcamp
        image: gcr.io/google-samples/kubernetes-bootcamp:v1
---
apiVersion: v1
kind: Service
metadata:
  name: k8s-bootcamp
  namespace: default
spec:
  ports:
  - name: http
    targetPort: 8080
    port: 80
  selector:
    app: k8s-bootcamp
---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: k8s-bootcamp
  annotations:
    kubernetes.io/ingress.class: "traefik"
    certmanager.k8s.io/issuer: "letsencrypt-prod"
    certmanager.k8s.io/acme-challenge-type: http01

spec:
  tls:
  - hosts:
    # Change this to your own hostname
    - bootcamp.k3s.example.org
    secretName: bootcamp-k3s-example-org-tls
  rules:
  # Change this to your own hostname
  - host: bootcamp.k3s.example.org
    http:
      paths:
      - path: /
        backend:
          serviceName: k8s-bootcamp
          servicePort: http
EOF

And apply it:

$ kubectl apply -f k8s-bootcamp.yaml

Wait a minute or so and your endpoint should become available with a Let’s Encrypt certificate! 🎉

Caveats storing large amounts of data in Elixir Agents

Pascal Widdershoven — Thu, 25 Apr 2019 00:00:00 +0000

Recently while working on an Elixir project I ran into an interesting gotcha with Agents that caused massive amounts of resource usage. Read on to find out what happened.

What are Agents in Elixir?

Agents are a simple abstraction around state.

Often in Elixir there is a need to share or store state that must be accessed from different processes or by the same process at different points in time.

The Agent module provides a basic server implementation that allows state to be retrieved and updated via a simple API.

Elixir is an immutable language where nothing is shared by default. This has many benefits, but it also means that when you do want to share data between processes you need to do some extra work. Fortunately, Elixir provides a lot of great building blocks to achieve this like Agents, ETS and Mnesia.

So what’s the problem?

There are two ways to use the state stored in an agent:

By operating on the data form within the Agents process:

# Compute in the agent/server
def get_something(agent) do
  Agent.get(agent, fn state -> do_something_expensive(state) end)
end

By pulling the data into the client process and operating on it there:

# Compute in the client
def get_something(agent) do
  Agent.get(agent, & &1) |> do_something_expensive()
end

If you look at the code the differences are very subtle. The difference in behaviour, however, is not subtle.

In approach #1 the data will remain in the Agent process. However, if you perform expensive operations there the agent will be blocked for the entire duration of the operation, meaning no other process can access the data until the operation is finished. Using this model to respond to an HTTP request is killing for performance.

In approach #2 the Agent will not be blocked, but the data will be copied into the process that is accessing the data. When the amount of data is small this is not really a problem, but if you start storing larger amounts of data this becomes really expensive real quick.

Real life example

The impact of this can be huge as I will demonstrate in the case below.

In the project I’m working on we were storing a set of rules in an Agent. A rule is a struct with 27 fields and we were storing approximately ~ 5000 rules in the Agent. There’s an HTTP endpoint that for every request uses these rules to determine the response.

For a while this was fine, but when the load started increasing we noticed the server going out of memory. To debug this I started throwing load at the endpoint using wrk. Results below:

Running 1m test @ http://localhost:4001
  4 threads and 1000 connections
  Thread Stats Avg Stdev Max +/- Stdev
    Latency 2.23s 553.43ms 3.00s 57.78%
    Req/Sec 59.72 121.98 680.00 88.64%
  Latency Distribution
     50% 2.24s
     75% 2.67s
     90% 2.88s
     99% 3.00s
  5551 requests in 1.00m, 11.80MB read
  Socket errors: connect 0, read 5971, write 3, timeout 5506
  Non-2xx or 3xx responses: 4575
Requests/sec: 92.38
Transfer/sec: 201.05KB

As you can see 92 requests per second are handled and a lot of requests time out (take more than 3 seconds). During the test, the Elixir process consumed around 10GB of memory.

Solutions

As we’ve seen in the previous section, storing these amounts of data in an Agent requires a lot of memory and performance is frankly not great.

Looking at the code and reading the Agent documentation, I quickly realised that the root cause of this issue was the fact that all rules were copied to the process handling the HTTP request, for every request. So how can we prevent this?

‘Shared nothing’ is a very core principle of Elixir/Erlang, so the short answer is you can’t prevent the data from being copied if you want to share the data between processes. This affects all ways of storing data in memory, so not just Agents.

There are workarounds, like fast_global. Fastglobal works by dynamically compiling a module at runtime, but it’s not without drawbacks.

So the solution is to make sure the data does not have to be shared between processes. There are a variety of ways to do this. The approach I took was to create a pool of worker processes (with Poolboy) that handle executing the rules. When an HTTP request comes in, the rule matching is handled by one of the worker processes.

In code this looks roughly like this (simplified):

defmodule Worker do
  use GenServer

  def start_link(_) do
    GenServer.start_link( __MODULE__ , nil, [])
  end

  def init(_) do
    rules = State.get()
    {:ok, rules}
  end

  def handle_call({:match_rules, input}, _from, rules) do
    matches = match_rules(rules, input)
    {:reply, matches, rules}
  end
end

When a worker starts it loads (copies) the rules from the Agent (State is a module wrapping the Agent) into the worker process. Each worker process contains a copy of the rules so the memory usage is predictable.

If the rules change at runtime, the processes are simply killed and restarted so the new rules will be used automatically. Poolboy takes care of starting N workers and selecting a worker from the pool.

End result

With that in place, wrk results started looking as follows:

Running 1m test @ http://localhost:4001
  4 threads and 1000 connections
  Thread Stats Avg Stdev Max +/- Stdev
    Latency 1.04s 270.71ms 1.66s 69.89%
    Req/Sec 221.14 65.34 405.00 66.08%
  Latency Distribution
     50% 1.07s
     75% 1.25s
     90% 1.36s
     99% 1.47s
  52823 requests in 1.00m, 14.41MB read
  Socket errors: connect 0, read 1014, write 0, timeout 0
Requests/sec: 879.16
Transfer/sec: 245.55KB

As you can see the throughput increased from 92 req/sec to 879 req/sec. Average latency went down from 2.23s to 1.04s. Memory used went down from 10GB to 400MB.

Not bad!

Forem: Pascal Widdershoven

Django & Webpack - without any plugins

Django-webpack-loader

Vanilla Django & Webpack

1. Use Django's default Static support

2. Configure Webpack to write files to STATICFILES_DIRS

Webpack chunks (dynamic imports)

Example project

Kubernetes workshop in a box

Why

The setup

Hardware & costs

You can use this too to run your own Kubernetes workshops!

Localizing your app with Flutter's new gen_l10n tool

Proposed new i18n process

Adopt gen_l10n for an existing project

Add gen_l10n dependencies

Define translations

Generate localization classes

Make localized messages available via BuildContext

Use localized messages

Extending Flutter Driver with custom commands

Extending Flutter Driver

Generating CSS from scratch with PostCSS

PostCSS

Generating CSS from scratch

Conclusion

Generating CSS from scratch with PostCSS

PostCSS

Generating CSS from scratch

Conclusion

HTTP Caching Gotcha: Heuristic Freshness

HTTP caching gotcha: Heuristic Freshness

Context

Heuristic Freshness

Cache-Control and Expires headers to the rescue!

Resources

HTTP caching gotcha: Heuristic Freshness

Context

Heuristic Freshness

Cache-Control and Expires headers to the rescue!

Resources

k3s HTTPS with Let’s Encrypt

0: Setup k3s.

1. Install Helm

2. Install cert-manager

3. Configure cert-manager

4. Deploy a service with automatic HTTPS

Caveats storing large amounts of data in Elixir Agents

What are Agents in Elixir?

So what’s the problem?

Real life example

Solutions

End result

2. Configure Webpack to write files to `STATICFILES_DIRS`