Forem: Dominik Liebler

GitOps for Confluent Schema Registry

Dominik Liebler — Tue, 01 Jun 2021 12:28:42 +0000

If you are like me, you love version control. Not so much the CLI interface (git isn't very beginner-friendly after all), but the concept of having a changelog of what has been done and - if you use meaningful commit messages - also the reasons behind a change.

Another thing that I love is Apache Kafka and that it is serialization-format agnostic. However most of the time, I use Avro-serialization because having a schema does bring a lot of pros to the table: it provides a safe way to change the contracts producers and consumers share and we all know:

Change is inevitable in life. - Jack Canfield

Schema evolutions and registry

Confluent's Schema Registry provides a central storage and retrieval API for schemas to be used by Kafka consumers and procuders. It also checks compatibility for schema evolutions to enable changes be backward- and/or forward-compatible.

Schema evolutions are often performed automatically when producers change the schema they use to write. This may be fine for prototyping but can lead to serious issues in production: in case of an error in the schema a rollback might not be as easy as going back to the previous version as eg. a newly-introduced field cannot be removed as easily and might need a lot of manual intervention to get back to the previous state. Also if the schema evolution fails because of compatibility issues, it will fail only when the changes are deployed and the new code runs for the first time. This one can be mitigated by testing on another environment first (which you should do regardless of this problem), but still.

Open Source ftw!

Now I wanted to combine both technologies and open sourced schema-registry-gitops to prevent from the above issues: having a version-controlled history of schema changes and push them to the registry only when ready and reviewed in QA. It can be used in CI/CD pipelines to ensure that schema changes are compatible with previous versions and can be part of your code review process.

My team at FLYERALARM uses it in our Atlassian Bamboo Pipelines to ensure that no bad schema evolutions make it into production and we've had no issues since! It's written in Kotlin using the APIs that Confluent provides for Schema Registry and Avro-serialization.

Scoped CSS for React Components with TypeScript

Dominik Liebler — Tue, 25 May 2021 08:17:54 +0000

React applications are being composed of small components that can be used individually and in best case scenario can be reused across multiple applications. But what about the CSS that they need for layouting elements inside them?

Often you end up inventing a system for it, something like that: you group CSS rules by a class name for each component and everything that is specific to a component goes in there. It's a start but it's not perfect. Soon you start renaming stuff or you want to apply styles from a global perspective.

Coming from the Vue.js world, I especially liked the approach of Single File Components - everything that belongs to that component goes into one file, CSS, HTML and JavaScript or TypeScript and I wanted to have this in React too, so I took of into the world of CSS-in-JS.

CSS Modules

I came to React, looking for Single File Components all over the place and as it turns out, it's not that easy 😔 What I did find though, is CSS Modules 🥳
It works like this: you import the CSS as a JavaScript module , which have been mapped from your CSS class names and assign those as className properties in the JSX. I used the npm package typescript-plugin-css-modules for this. This is how a Component styled with it looks like:

import styles from "./foo.css";

const FooComponent = () => {
  return <div className={styles.myClassName}>Hello, World!</div>;
};

I still wasn't satisfied with this approach as there were still two files to edit when the component needs to be modified. Then I learned about Emotion in Jason Lengstorf's Introduction to Gatsby course on Frontend Masters and it was exactly what I was looking for. I was intruiged 😁

Emotion to the rescue

To style React components with emotion, there are several options to choose from depending on your preferences and what you want to achieve. The one I like most as a starting point, is using the css-template string helper that lets you write CSS like you would in a CSS file. This is a sample component using it to set a width, height and background-color:

/** @jsxImportSource @emotion/react */
import { css } from "@emotion/react";
import React from "react";

function Card() {
  return (
    <div
      css={css`
        width: 100px;
        height: 100px;
        background-color: red;
      `}
    >
      using the css template string helper
    </div>
  );
}

export default Card;

Really simple, huh? The first line /** @jsxImportSource @emotion/react */ tells the TypeScript compiler, how to resolve the css helper and it took me quite a while to figure it out!

But it can get quite messy when the component grows and contains more tags than just this one div. For this occasion, you can refactor the component and use styled components like this:

import styled from "@emotion/styled";

const ListItem = styled("li")`
  font-weight: bold;
`;

interface ListProps {
  items: Array<string>;
}

function List({ items }: ListProps) {
  return (
    <ul>
      {items.map((item) => (
        <ListItem key={item}>{item}</ListItem>
      ))}
    </ul>
  );
}

export default List;

As you can see, ListItem uses the styled function to create a styled component that only adds CSS to a li element and automatically renders its children in it.

Now I got to the point where I was satisfied. At least until I realized that I wanted to have kind of like a theming, where I would store colors, definitions of borders and such things that I would need over and over again in a central location. Emotion provides a Theming API, why not try that?

Theming

To start with the theme, I implemented a new class for it and created a new instance of it:

class Theme {
  readonly primaryColor: string = "green";
}

const theme = new Theme();

Theming then works like this: you provide the Theme to your components using a <ThemeProvider> and access the theme in the actual component using the useTheme hook provided by @emotion/react. Here's my App, that does exactly that:

import { ThemeProvider } from "@emotion/react";

function App() {
  return (
    <div>
      <ThemeProvider theme={theme}>
        <MyComponentWithTheme />
      </ThemeProvider>
    </div>
  );
}

And here is MyComponentWithTheme, which uses both the Theme and the css template string helper:

/** @jsxImportSource @emotion/react */
import { css } from "@emotion/react";
import { useTheme } from "@emotion/react";
import { Theme } from "../App";

function ComponentWithTheme() {
  const theme = useTheme() as Theme;

  return (
    <div
      css={css`
        width: 100px;
        height: 100px;
        background: ${theme.primaryColor};
      `}
    >
      Component using the Theme provided in App
    </div>
  );
}

export default ComponentWithTheme;

Using that, I found a way to write the CSS I need for my components directly in the components. I still need to figure out what parts of the style sheets go into a component or into a global style sheet, but it's a start.

Of course emotion does a lot more than that (like adding vendor prefixes and stuff) but I am still learning about it and find it very interesting and fun. I am looking forward for your tips and tricks around emotion and CSS-in-JS in common!

Using actors in Kotlin - a concurrent play in one act

Dominik Liebler — Mon, 16 Sep 2019 00:00:00 +0000

In february this year I already wrote about Kotlin coroutines and their usage in JavaFX applications and today I want to explore coroutines even further and talk about them in the context of actors.

Why not Threads?

A modern software system has to manage a lot of different tasks at the same time. User requests coming in via web interfaces, GUIs and other channels need to be handled while there is background work to be done. Traditionally this has been the domain of operation system threads that the application spawned and provided with work.

Photo by Héctor J. Rivas

But as you may know, threading is error-prone and not easily done right when it comes to mutating state in an complex application (if you don’t you should read Why Threads Are A Bad Idea (for most purposes)). With coroutines, there is a lightweight and simple alternative to threads in Kotlin that follows the approach of Communicating Sequential Processes. It’s the same idea that inspired Go’s goroutines:

Do not communicate by sharing memory; instead, share memory by communicating. - Effective Go

Kotlin achieves that by providing the concept of a Channel, which basically is a Queue that uses suspending functions. Using coroutines and channels, we can build a system that encapsulate mutable state in a manner that do not need any locks and synchronization and instead leverage a protocol of messages to handle concurrent updates of that state.

The actor model

Photo by Avel Chuklanov

Such a model is called an actor. It is not a new concept, instead it has been around for years and is the underlying concept of Erlang and can also be used in Java and Scala using Akka. In How does keeing state in Elixir work? I built an actor in Elixir even though the concept is not officially called an ‘actor’ in the BEAM world, but gen_server instead.

Actors come in many sizes and they can do a wide variety of tasks, but they all share the same properties:

they keep internal state
they are running concurrently
their state can be manipulated through messages only
they receive messages in a channel that is called a ‘mail box’
they process messages sequentially

Don’t try this at home!

Ok, this might be a bit exaggerated, but there is a need to warn you. Precisely to warn you about the state of implementation for the actor() function. As of writing it is a very simple API to create an actor but this is about to change in future releases of kotlinx.coroutines.

basic counter example

So what does an actor in Kotlin look like? Let’s have a look.

First of all, we need to define messages we can send to the actor. In this simple example, the actor stores a counter that can be incremented by arbitrary integer values using Message.Increment and the current value can be requested by Message.GetValue.

To send data back to the requesting code, we use a CompletableDeferred here.

sealed class Message {
    class Increment(val value: Int) : Message()
    class GetValue(val deferred: CompletableDeferred<Int>) : Message()
}

Next up, we define the actor named basicActor itself. It needs to be defined as an extension function to CoroutineScope as it will launch a coroutine to run in. It runs a simple for loop to get messages from the channel that will run until anyone closes the channel. The handling of the messages is rather straight forward.

fun CoroutineScope.basicActor() = actor<Message> {
    var counter = 0

    for (message in channel) {
        when(message) {
            is Message.Increment -> counter += message.value
            is Message.GetValue -> message.deferred.complete(counter)
        }
    }
}

That’s it! But to use it, you also have to define a client that sends messages to the actor. Here’s the code:

fun main() = runBlocking<Unit> {
    val channel = basicActor()

    channel.send(Message.Increment(1))
    channel.send(Message.Increment(2))

    val deferred = CompletableDeferred<Int>()

    channel.send(Message.GetValue(deferred))

    println(deferred.await()) // prints "3"

    channel.close()
}

In main we define a CoroutineScope by using runBlocking() which is then used to launch the actor in and send messages. channel.close() will then cause the actor to complete. Without it, the actor will keep the program running infinitely, which may be a desired state in a typically server situation.

Conclusion

Actors are an easy concept that can help to remove the hassle of concurrent computing. They are still experimental in kotlinx.coroutines but it’s a first step in the direction and I am curious of the developments and as a long-time fan of Erlang and Elixir I appreciate actors and would happily welcome them as a stable component of Kotlin coroutines.

Kotlin coroutines and JavaFX threads

Dominik Liebler — Sat, 02 Feb 2019 00:00:00 +0000

On October 29, 2018 JetBrains released a new version of Kotlin (1.3) that included a long-awaited feature of the language: coroutines. Contrary to Go’s goroutine construct, they are not implemented as a language feature. Instead they are developed as a pure library implementation written in Kotlin. But apart from that they share a lot of similiarites with goroutines.

Behind the scenes, coroutines are very lightweight because they are being multiplexed on a few threads and creating one is much faster than spawning a new thread.

You can learn more about them over at JetBrains’ tutorial: “Your first coroutine with Kotlin”. Here I want to show you the caveat that you might encounter in a JavaFX applications when using coroutines.

So this is my basic JavaFX application written in Kotlin:

class MyApp : Application() {
    private val button = Label().also { it.text = "waiting for 0s" }

    override fun start(primaryStage: Stage?) {
        val pane = Pane().also {
            it.children.add(button)
        }

        primaryStage?.scene = Scene(pane,200.0,200.0)
        primaryStage?.show()
    }
}

fun main() {
    launch(MyApp::class.java)
}

Nothing special here, just a GUI with a label that shows the text waiting for 0s. Now we want to extend our application to continuously update the label to always tell me the exact time I waited.

class MillisElapsedCounter(private val observer: Observer) {
    fun start() {
        GlobalScope.launch(Dispatchers.Default) {
            var millisElapsed = 0

            while (true) {
                delay(1000)
                millisElapsed += 1000
                observer.notify(millisElapsed)
            }
        }
    }

    interface Observer {
        fun notify(millisElapsed: Int)
    }
}

Notice that you need to add a dependency to org.jetbrains.kotlinx:kotlinx-coroutines-core:1.1.1 to be able to run this (remember: it’s just a library!)

So this is the implemention of the MillisElapsedCounter class and a corresponding Observer that is being notified every 1000ms. Clients of this class have to instantiate it providing an observer. Calling start() will then launch a new coroutine using the default dispatcher (there is a pool of dispatchers - which are threads behind the scenes - you can use) and return immediately. Note that delay(1000) is a suspend function, it will suspend execution in the thread it runs in. Running that in the application thread would freeze the UI forever and would be a bad idea.

To use the counter we also need to update MyApp to implement MillisElapsedCounter.Observer’s notify(millisElapsed: Int) and start the counter like this:

class MyApp : Application(), MillisElapsedCounter.Observer {
    private val button = Label().also { it.text = "waiting for 0s" }

    override fun start(primaryStage: Stage?) {
        MillisElapsedCounter(this).start()
        ...
    }

    override fun notify(millisElapsed: Int) {
        button.text = "waiting for ${millisElapsed / 1000}s"
    }
}

Now if you run that, it will throw a java.lang.IllegalStateException exception that is coming from JavaFX:

Exception in thread “DefaultDispatcher-worker-3” java.lang.IllegalStateException: Not on FX application thread; currentThread = DefaultDispatcher-worker-3

To prevent from concurrency issues in the state of the UI, JavaFX prevents other threads from changing it. Look at the main() function that uses launch(MyApp::class.java) to start the application. This starts a new thread called JavaFX-Launcher which in turn starts the JavaFX Application Thread which runs all the UI related stuff. This one is the only thread allowed to make changes in the UI at runtime.

This means that we need to execute the changes that are being made to button.text in this very thread. The Kotlin team implemented another library for this sole purpose: org.jetbrains.kotlinx:kotlinx-coroutines-javafx:1.1.1 provides a new Dispatcher Dispatchers.JavaFx that uses the JavaFX Application Thread (there’s also -android and -swing to be used correspondingly).

Then the solution to our problem here is to use the correct Dispatcher Dispatchers.JavaFx and to implement another interface in MyApp called CoroutineScope:

class MyApp : Application(), MillisElapsedCounter.Observer, CoroutineScope {
    override val coroutineContext: CoroutineContext
        get() = Dispatchers.JavaFx
        ...

As it is now a CoroutineScope we can launch new coroutines directly from notify() using the correct scope:

override fun notify(millisElapsed: Int) {
    launch {
        button.text = "waiting for ${millisElapsed / 1000}s"
    }
}

And that’s it! The exception will no longer occur and everything will run as expected - I waited a long time, and it still counts happily the seconds I waited :)

One thing we can improve though is that MillisElapsedCounter runs a coroutine in the GlobalScope which means that you need to keep track of it’s lifetime (if you were interested in that - eg. stopping it). More details on that in The reason to avoid GlobalScope. To fix that we could implement the interface CoroutineScope in MillisElapsedCounter as well:

class MillisElapsedCounter(private val observer: Observer) : CoroutineScope {
    override val coroutineContext: CoroutineContext
        get() = Dispatchers.Default

    fun start() {
        launch { ... }
    }
    ...

We now have two coroutines cleanly separated from each other and each running in it’s own scope. A clean and simple solution I think.

The complete source code is available on GitHub, so you can clone it and play around with it:

domnikl / kotlin-coroutines-and-javafx-threads

Sample code from my blog post

Kotlin coroutines and JavaFX threads

This is the example from my blog post with the same title.

View on GitHub

You can read more about that in Guide to UI programming with coroutines.

My four stages to learn new tech stuff

Dominik Liebler — Wed, 24 Oct 2018 00:00:00 +0000

Over the last 15 years that I have been a professional software developer now, I learned more than 16 programming languages and several frameworks, database systems and other tools that helped me in my path to master software craftsmanship.

Some of that I learned because I needed it in a project for a client but some of these technologies I was just curious about. If you are a software developer you know how fast our world moves on to new technology to solve new types of problems with it.

So you need to get going with new tech fast and know how to learn efficiently. Here’s how I learn today, a play on four stages:

Stage 1: syntax/terminology

The first thing you need to learn about a new programming language or tool is syntax and terminology. If you already know another language or tool you learned a certain way to solve problems in this language but now you have to look out how to solve similar problems in this new language and need to learn the concepts behind it.

To be honest, I learn syntax once and forget about it later anyway. I have to Google that stuff like A LOT. In my regards the best way to learn about it initially is to look out for Koans.

Koans are small exercises where you have to fix or write small parts of code and often they are backed by unit tests which you have to pass in order to move on to the next one. Have a look at Laura Diane Hamilton’s comprehensive list of Koans for that.

Most tools come with documentation and contain a simple tutorial to accompany you taking your first steps with it. Eg. if your intend to learn Docker, there is a great example in the official Docker documentation.

Third-party docs may be very useful, too. When using Go, I always have to Google stuff like how to let goroutines communicate with another using channels and I always end up using Go by example which is another great example. A great list for other languages can be found at Learn X in Y minutes.

But most of the time, I don’t set out to learn just the syntax of a new language. I wanna play around and explore. What helps me in this regard is Code Wars. By solving small problems, it forces me to learn syntax and explore idiomatic ways to solve the problem efficiently.

Stage 2: ecosystem

“If I have seen further it is by standing on the shoulders of Giants.” - Isaac Newton

So you know about syntax and terminology, but where to go next? Learning about the tooling and libraries of a new language or database system is very important (and exciting) IMO. No matter if you need logging, recording metrics or a client to connect to a specific database system there are (open source) libraries that you can build on and you need to know them if you want to build something useful with a new programming language.

Luckily there are so-called “Awesome Lists” for that on GitHub that contain everything you need for every language you can imagine.

At this stage, I try to do something useful with it. That means I can use the tool I build or at least learn something from making it. More often than not I will throw that away afterwards. Don’t ever get too sentimental with code! After all you will write lot’s of it in your career and very few of it will get to production or stay there long.

So most of the time, I will just rewrite a Unix program in my new language. I have written cat in C, Go and Kotlin - tail in Python, PHP and Java and many more. I never use them of course but they were tools I know very well and wich are documented very well so I had a clearly defined goal to go for.

Apart from that I often try to solve problems I had solved in the past with another language and try to solve it again in a new language in a more simple way, which is a compelling task sometimes.

Stage 3: best practices & experience

When it comes to adopting a new language, syntax and libraries aren’t just enough to build stable and reliable software with it that is also maintainable and can be easily extended. You also need to gain experience with that new piece of technology.

Usually that means to run it in production, get caught up in problems and get to know how to fix them. But there is another, less harmful way to get that by learning from others and by learning about best practices upfront. This stage is all about the idiomatic way to use a language or system.

There are two main paths for me here: first, there is Google. You can find a lot of content on best practices in (official) documentation or blog posts which can help you discover best practices. And it helps a lot to hear talks at conferences where people are talking about their experience in using that technology.

Stage 4: unlearn old concepts and habits

After learning a new tool or language, you shouldn’t forget to unlearn things, too. If you embrace and accept technical change (which I guess you do if you came here), you should also accept that you have to unlearn old habits, patterns and terminology at some point. After all in our industry, there is a lot of change going on all the time and at a very fast pace.

So to not get confused when learning about new tech and it’s terminology you should simply forget about old tech. An example: “table” must have had another meaning before there were relational database systems and people had to forget about that in the 1970s, when those RDBMS where new and cutting-edge technology.

That’s my four stages of learning. I don’t think this will be the last iteration of it, as I won’t stop learning and therefor evolving my learning process. I would also be very happy to hear about your learning process for inspiration.

How does keeping state in Elixir work?

Dominik Liebler — Tue, 06 Mar 2018 00:00:00 +0000

In 2011, when I was new to Erlang and functional programming, the biggest question I had was something like this:

When there is no state in functional programming, how do you store, retrieve and update state?”

I read about gen_server that really just was the point where I realised that although everyone kept saying “There is no state in functional programming” you actually do have mechanisms to store state in Erlang/Elixir for later use.

So I dived into it and learned how it’s done.

How it’s done

Actually the concept behind it is not so much rocket-science. Basically it consists of a function calling itself recursively and communicating with other processes which want to store and update the state. If the state needs to be changed, the next call will be made with the new state.

Let’s build a Stack module that holds it’s values and allows users to push and pop values from it.

We start off with a simple function that will run and keep the state by recursively calling itself and receiving basic messages to handle changes to the state:

defmodule Stack do
  def loop(state) do
    receive do
      {_from, :push, value} ->
        loop([value | state])

      {from, :pop} ->
        [h | t] = state
        send(from, {:reply, h})
        loop(t)
    end

    loop(state)
  end
end

loop/1 waits for new messages to arrive and uses receive to block until there are messages for this process which will be pattern-machted against. As you can see, there are two patterns for the two operations we support currently.

The first one will match {_from, :push, value} which handles pushes on to the stack. The sending pid _from is not needed as this is an asynchronous operation. It starts the next iteration with the new value added as the new first element of state which will again wait for new messages to arrive.

The other pattern {from, :pop} handles the pop operation. The message sender from will receive back a message with the first element of state extracted while the rest of the list is given to yet another call to loop/1 which will then wait for another message to arrive.

Test it

Let’s take our stack implementation for a spin. Start iex, paste the above module into it and we are ready to go! To start the stack, just spawn the function as a process:

iex> pid = spawn(Stack, :loop, [[]])
#PID<0.108.0>

spawn/3 expects a module (Stack), the function name as an atom (:loop) and a list of arguments, which in our case is an empty list. This makes the initial call, which is essentially Stack.loop([]). The returned pid is a process identifier to which all messages need to be sent.

To push a new value on to the stack, we need to send that pid a message like this:

iex> send(pid, {self(), :push, 1})
{#PID<0.88.0>, :push, 1}

self() returns the value of the current process (which is iex’s pid in our case) and send will return the message sent. For this operation we don’t need it, but it’s a good practice to send it anyway. Also we don’t see any change to the stack yet, but remember: the push operation is async.

Now let’s see the change in the stack by popping a value from the stack:

iex> send(pid, {self(), :pop})
{#PID<0.88.0>, :push, 1}

Here, self() is required, because we expect a message back to our iex-pid containing the value. Again, send returns the message sent and nothing else happens. That’s because we didn’t read the message we got back yet. Let’s use the iex helper flush to retrieve the contents of our main process’ mailbox:

iex> flush
{:reply, 1}
:ok

And there it is; as expected, the stack contained the value 1.

a clean client interface

As our stack is now working, we should build a nice, clean API around it. A user that would want to use the stack module should not care about which messages are required to sent to the process. Instead he/she should use only functions to place values on the stack:

Let’s extend the module and add a bunch of short functions that build up the API and a clean client user interface.

defmodule Stack do
    ...

    def start_link(state) do
        spawn_link( __MODULE__ , :loop, [init(state)])
    end

    def init(state), do: state

    def push(pid, value) do
        send(pid, {self(), :push, value})
        :ok
    end

    def pop(pid) do
        send(pid, {self(), :pop})

        receive do
            {:reply, value} -> value
        end
    end
end

If you compare this against the callbacks that you would need to implement when using GenServer, then you will see that it uses the same naming and function signatures. This is not by accident - it is to show you that GenServer does exactly the same as the above code. Except for a little more error handling and in a more generic way of course. But the concepts remain the same.

Btw. thanks to tail call optimization the call stack of the loop/1 function is constant in terms of memory usage and therefor not a problem when running forever and ever.

Here I will leave you off to some experimenting with state as messages and recursive functions. Happy Elixir’ing!

Docker - Standard_init_linux.go:187 exec user process caused no such file or directory

Dominik Liebler — Wed, 31 Jan 2018 00:00:00 +0000

I have been working a lot with Docker and Go lately and I come across a very annoying error when running tools built in Go in containers on a RaspberryPi. All of the bug reports I found had something to do with Windows-style line endings (CRLF) but I am using MacOS, so this can’t be the solution (at least) for me here.

This is the Dockerfile I have been using:

FROM alpine
WORKDIR /app
ADD washing-machine /app/washing-machine
CMD ["./washing-machine"]

Turns out building the binary on the mac doesn’t work, even using GOOS and GOARCH didn’t do the trick, so I now build the binary in the container, using the golang image from Docker hub:

FROM golang:1.9.3-alpine3.7
ADD . /app
WORKDIR /app
RUN cd /app && \
    apk update && \
    apk add git bash && \
    go get github.com/domnikl/ifttt-webhook && \
    go get github.com/domnikl/fritz-box && \
    go build -o washing-machine
CMD ["/app/washing-machine"]

Hope this will help you in getting around this annoying error message.