Forem: Gerard Klijs

Why you might eventually wish you started out using event sourcing

Gerard Klijs — Sun, 02 Jan 2022 16:21:30 +0000

Some thought on what event sourcing might be, and what it adds to the table.

Philosophical trickery

In order to be able to write this blog, but not strand to much on what event sourcing 'really' is, I will rely on some trick with words. Instead of talking about 'real' event sourcing I'll quickly go through about what event sourcing in the 'weak' sense could mean, and what event sourcing in the 'strong' sense could be. As often with this trick most systems in the wild are probably somewhere between.

It's unlikely event sourcing in the 'weak' sense will be labeled as such by people using it. Likewise, event sourcing in the 'strong' sense is so strict, it's unlikely a real system hasn't used a shortcut somewhere. It still helps for setting up the story, and the differences for when we go into the advantages of event sourcing in the 'strong' sense.

Event sourcing in the weak sense

Event sourcing in the weak sense is taking a very literal approach. Basically all we need is just some events that are stored somewhere for other systems to use. The system producing this stream might not even consume it itself, meaning inconsistencies may occur. It might also not even contain all the events since the service was started. The event also doesn't need to be a business event, it might be something technical, like the update on some row in a SQL database. The events might also not be complete, as there might be related events, but they might not as such be available in the same way.

One of the components that can be used to easily implement such a thing would be to use Debezium. With this we can set up one or multiple rows or collections, depending on the database used, and have all the changes to those available as messages in Kafka.

Event sourcing in the strong sense

The most important difference with event sourcing in the strong sense, compared to event sourcing in the weak sense, is that the events should be the single source of truth. In order to generate a new event, a 'command' is issued to the system, and depending solely on the past events, this might generate new events.

One of the ways to work in this manner in practice is by using the Axon Framework. For this to work the events need to be immutable, and we need some way to quickly get all the events related to a certain entity. Based on the past events we can than determine if the command is valid or not. There are some complexities with such a system, like when we need to coordinate between different entities. I won't go into them in this blog.

Use case, PEF, a payment provider

Let say we have a payment provider. They wanted to be quick to market, so did not spend a lot of thought on designing the architecture. Let's call it 'PEF', for pay easy and fast. In PEF most services are simple REST based services, secured, and allowing all the known CRUD operation when the one calling the service has enough rights. As always with these things any resemblance with any real company is incidental.

PEF starts using event sourcing in the weak sense

In PEF they use a microservice setup. A lot of these services need to have some access to the customer information of PEF, for example to know the bank account number of a customer. Because the customer service was swamped with REST calls as PEF was getting more customers, they decided to start using event sourcing in the weak sense for the customer information. By using Debezium some customer information thus became available in an async matter. This way another service needing to have the bank account number by a customer id, could store and update this information in its own database, preventing a lot of REST calls to the customer service.

Note that what is available is just customer updates, as the changes are made to the database. For example there is not something like a 'CustomerBlocked' event with some context. If a customer is being blocked ot might be just an update where the property 'blocked' is set to true.

Some challenges ahead

I will now get to a few cases where the differences between event sourcing in the weak sense and event sourcing in the strong sense become more evident. As PEF continues to grow, data is needed in other ways than previously was accountant for. Each of these cases provides a challenge to PEF, and depending on how things are set up, these challenges might be easy or impossible to solve.

Customer wants to know why he's blocked

After some time one of the customers is reaching out why his account is blocked. The one researching this issue doesn't have direct access to customer service database. From the messages available via Kafka he does eventually find out he was blocked three weeks ago. Unfortunately the updated record just says he was blocked from that time, but doesn't include any information regarding the reason. After reaching out to the customer service team, it turns out the reason is not stored in the database either, but it is logged at info level. Logging is searchable, but only for the past two weeks. To get the logs further back a request need to be made to the SRE team, to get the stored logs. After some back and forth they manage to get the relevant logging, and it turns out the customer was being blocked for being inactive. Happy to finally found the answer, this could be feed back to the customer, with some instructions how to reactivate his account, and what he can do to prevent the same thing from happening again.

If we use event sourcing in the strong sense, it would probably have been much easier to found out what was the cause. By the nature of event sourcing it would be easy to retrieve all the event concerning the one customer. Instead of having to search for an update where a property was set from 'false' to 'true' we likely would be able to see something like an 'CustomerBlockedEvent'. In this event we would have the relevant additional data, like the reason, and when it happened.

Data analyst want to see correlations

Via the marketing department, PEF want to build some models on the typical behavior of customers. They are aware for the time they want to run there analysis on, all the customer updates are available via Kafka. They want to compare when customers are onboarded, with which actions are executed in the web UI. Unfortunately it turns out for knowing what customer did in the web UI they can only use access logging. This is quite a struggle as the customer id's are not readable which each request made, because JWT is used. So in turn to correlate a certain call with a certain customer, they have to decode the JWT and extract the customer id. Because this is a different stream of information, a lot of the work for getting the customer information itself can't be reused.

With event sourcing we would have more uniform information to work with, and it would likely be a lot easier for the data scientist to build the model. Also since more information is available as proper events, it might be a lot easier to research similar things in the future.

Governance needs some improvements

Since PEF is a financial institution there are some strict laws that apply. One of these is that it should always be clear that if customer data is changed, it should be clear who made the change and way. For example an address might be changed because the customer has moved. This change might be done from the web UI by the customer directly, or indirectly by calling customer support. As it turned out, the current customer service wasn't compliant with these rules, and should be updated soon on the risk of losing the license.

This ment the customer service team had to go through the code base, and add relevant logging with the relevant details to be compliant. The team also added an item to the 'Definition of Done', so when in the future anything is added that updates customer information the relevant logging should also be there.

With event sourcing such nonfunctional requirements are easier to handle in a generic way. For example, we could have something like the 'CustomerContext' which is part of all commands related to the customer. In the resulting events we need to include those, to make sure they are stored with the event. It should not be that hard to add this additional information later on.

What can we learn from PEF

While event sourcing in the strong sense might seem to complex at first glance, it might actually make thing easier later on. By using certain libraries of frameworks it might not actually be much more complex than event sourcing in the weak sense. As such when designing a new architecture or service, I think it's something that should at least be considered.

I know some of the things might be controversial, please feel free to discuss.

Confluent Schema Registry and Rust

Gerard Klijs — Mon, 26 Jul 2021 17:40:27 +0000

Intro

This blog will be about the Rust library I wrote and maintain, schema_registry_converter. Since the library has little use on its own, and is about integrating data with Apache Kafka, we first need to take a few steps back.

Kafka

For me having used Apache Kafka at several clients during the years, it's sometimes hard to imagine that other developers don't know anything about Kafka. There are a lot of great articles and introductions to Kafka, like Apache Kafka 101: Introduction from Tim Burgland. Kafka is different from more traditional message queues mainly because the time messages stay available for consumers is independent on how they are consumed. Multiple apps can thus read the same messages.

For the purpose of this blog it's important to know that messages are stored on Kafka as records. Records have a value and an optional key, which are both in binary format. Another important fact is that Kafka uses topics to split the messages. One topic might exist of multiple topic-partitions which are used to make it scalable. Part of the configuration is topic specific. For example, data on certain topics can be retained longer than other ones, or can be configured as compacted such that the last message with the same key will never be deleted.

Because data is stored in a binary format, it's important for apps producing data to do so in a way apps consuming the data can make sense of those bytes. An easy way to do it is to serialise the data as JSON. Especially since it's human-readable and easy to work with in most programming languages. This does make it less transparant for the consumer, and has some other downsides like the message being relatively big.

To have more control over the data, and store it in a binary format, a schema registry can be used. One the most used registries with Kafka is the Confluent Schema Registry.

Confluent Schema Registry

Confluent is a company founded by the creators of Apache Kafka. They are providing the Confluent Platform which consists of several components,
all based on Kafka. The license for these components vary. The Schema Registry has the community-license, which basically means it's free to
use as long as you don't offer the Schema Registry itself as a SaaS solution. The source code can be found on Github.

So what is actually the Schema Registry? And how does it help to make sense of binary data? In essence Schema Registry is an application with some Rest endpoints, from which schema's can be registered and retrieved. It used to only support Apache Avro. Later support for Protobuf and JSON Schema was added. Part of the same Github project, and what makes Schema Registry easy to use, is a Collection of Java classes that are used to go from bytes to typed objects and vice versa. There are several classes that support Kafka Streams and ksqlDB next to the more low level Kafka [Producer(https://kafka.apache.org/documentation/#producerapi) and Consumer clients. There are more advanced use cases, but basically you supply the url for the Schema Registry, and the library will handle the rest. For producing data this will mean optionally register a new schema too get the correct id. The consumer will use the encoded id to fetch the schema used to produce the data. It can also be used with other frameworks like Spring Cloud Stream. For example in the Kotlin Command Handler by using the SpecificAvroSerde class. You might need to set additional properties to get this working.

All this is great when using a JVM language for your app, but might be a challenge when using another programming language. Part of the reason is that because the bytes produced by schema registry are specific to schema registry. There is always a 'magic' first byte, which allows for breaking changes at some point, and lets the clients know quickly whether the data is encoded properly. The reference to the schema that was used to serialize the data is also part of the data. This makes it impossible to use a 'standard' library. Since those bytes need to be removed for a 'standaard' library to work. This might be a valid reason to use something like Protobuf, combined with some documentation on which Protobuf schema was used for which topic. You also don't have to run a schema registry in that case, but for clients it's a bit more work to get the correct schema.

On the other, hand Schema Registry does offer a complete solution, where [Schema Compatibility (https://docs.confluent.io/platform/current/schema-registry/avro.html) can be configured. Because updates to schema's can be varified for backwards compatibility this way, consumers can use the old schema for the data. Storing the schema in a central location decouples the producers from the consumers. Making it much easier to add additional information later on without the need to immediately update the consumers once the producer start using the new schema. Another major advantage is the integration of Schema Registry with the Confluent Platform. Making it much easier to use of ksqlDB.

Rust

Rust is a pretty young language, the first stable version released in May 2015. I've been using Rust a couple of years, but in the early days it could be a lot of work just to get your code to compile again. Since the stable release, there has been no backwards incompatible changes. This has also paved the way for a lot of libraries, or crates as they are called in Rust. One of the major sources to start learning Rust is "the book", there are also books for specific subjects like WASM and async. There are also a lot of videos available on Youtube. One of those is this one, which I made specific for Java Developers.

Crates can be found on crates.io where you can be easily search for specific libraries, and all relevant information about the libraries is available. Rust is a c/c++ alternative, but it can in some cases be an alternative to Java as well. This largely depends on and what the app does, and if for the libraries used, there are Rust alternatives available.

Rust itself is open source. With the creation of the Rust foundation its future is secure. Personally I really like the tooling, like rustfmt and clippy which work as a default and easy to install formatter and linter respectively. Another nice thing is being able to write tests as documentation, with the documentation being available online like the AvroDecoder stuct from the schema_registry_converter library.

Bank demo project

What originally was created as a project used in a workshop with my Open Web colleagues has turned out to be my goto project for experimentation. The full story can be found on Dev.to. Basically it's a couple of small services that together create a virtual bank where users can log in, get an account, and transfer money. One of the iterations was used for a blog with Confluent. It's relevant for this blog as the core of the schema_registry_converter came into existence creating a Rust variant for the Command Handler part of the demo project. For that project I was using schema registry, and since I wanted to keep the rest of the system the same, I didn't want to change the binary format used with Kafka.

Like I mentioned, using schema registry with a non JVM language can be challenging. Luckily I had some previous knowledge about the internals of Schema Registry from my days at Axual. When I tried to use Rust together with the Schema Registry, Avro was the only supported format. I quickly found out there was also already a Rust library supporting Avro. So it seemed with just a couple of Rest calls to the Schema Registry server, and using the library I should be able to get it to work, which I did. The result with an early version of the library can be found in kafka_producer.rs and kafka_consumer.rs.

Schema Registry Converter

The source for the current version of the library can be found on Github. I had to increase the major version because I needed to break the API in order to support all formats supported by the current Schema Registry version. I also added the possibility to set an API key, so it can be used with Confluent Cloud, the cloud offering from Confluent. As part of the latest major refactoring it's also supporting async. This might improve performance of your app, and is also the default for the major Kafka client, more information about why you would want to use async can be found in the async book. The schemas retrieved from the Schema Registry are cached. This way the schema is only retrieved once for each id, and reused for other messages with the same id.

Next to the additional formats there was one other major change to incorporate from Schema Registry. In order to reuse registered schemas with new schemas they made it possible to have references. So when retrieving a schema, one or more pointers to other schemas might be part of the returned JSON. To make sure I got this part right in the Rust library I created a Java project which can be used from docker.

Another interesting challenge was getting the protobuf implementation correct. Contrary to Avro or JSON Schema, one proto file can describe multiple messages. In order to properly serialise the data, the kind of message used also needs to be encoded. While for Java the same part was a pretty trivial, because the Protobuf library used had an easy way to map a number to a message. I could not find something similar in Rust, so in proto_resolver.rs I used a lexer to provide the needed functionality.

What the library does is different for a producer and a consumer. For both there are action diagrams.

Producer action diagram

For the Producer it needs to encode the bytes in the proper way. This starts by enabling the feature (?MT what does feature mean here) of the correct encoder, depending on the format, and whether blocking or async is required. Then the data needs to be encoded using the encoder and one of the SubjectNameStrategies which might contain a schema. With the option of using the cache, a byte value is produced that can be used as either the key or value part of a Kafka record.

Consumer action diagram

For the consumer it's also needed to use the correct decoder, based on the expected format of the message. From the Kafka record either the key or value bytes is used. With the encoded id the matching schema will be retrieved or fetched from the cache. Depending on the decoder used a certain typed value is returned. Depending on the app this value can be used for several things, for example to write something in a database.

Because of the three formats, and the two ways of using the library, async and blocking, it would be tedious to have examples for all. To make things worse each of these six possibilities has its own encoder and decoder. Where the encoder is used for a producer and a decoder for a consumer. Both also have their own separate possibilities. For the producer it's possible to register a new schema, if the latest is not the same one, or use a schema that was already registered. For the consumer it can be a challenge to use the resulting typed struct, where it might be needed for Avro to have an Enum with all the expected possibilities. Similar to what was used for the AvroData in the demo project.

Some simple examples are available in the library itself like the async Avro Decoder:

use avro_rs::types::Value;
use schema_registry_converter::async_impl::schema_registry::SrSettings;
use schema_registry_converter::async_impl::avro::AvroDecoder;

async fn test() {
    let sr_settings = SrSettings::new(format!("http://{}", server_address()));
    let mut decoder = AvroDecoder::new(sr_settings);
    let heartbeat = decoder.decode(Some(&[0, 0, 0, 0, 1, 6])).await.unwrap().value;
    assert_eq!(heartbeat, Value::Record(vec![("beat".to_string(), Value::Long(3))]));
}

In order to prepare for a future blog with Confluent I wanted to play around with ksqlDB, which was the perfect opportunity to use the Rust library in a less trivial way. As it turned out there is even a library for communicating with ksqlDB from rust, using the Rest API. The PoC project for this contains some code to put protobuf data on a topic.

Maintaining the library

When I took the steps to turn the code I had into a library, I wanted to make sure I had decent code coverage. By leveraging codecov I now get updates on the code covered in pull requests. Not that there are many since the library does what it does, which is nicely scoped. The latest update was just updating dependencies, which might sometimes give problems especially for libraries like Avro, when the byte code used is not the same. A small update I'm thinking about adding it making it slightly easier to use protobuf when you know there is only one message in the proto schema.

Aside from the big rewrite, maintaining the library has taken very little time. From time to time there is a question about the library. It is nice to see people actively using it and hearing about how the library is used. Crates.io shows the amount of downloads over the last 90 days. What's interesting is that, from the start of this year, instead of a flat line, there are clear peeks during working days. This is just one of the signs Rust is getting more mature and used in production. I still haven't used it in a 'real' project yet, but that's just a matter of time. Recently the library was used in a Hackathon with a contribution as result which is part of the 2.1.0 release. The pull request made it possible to supply configuration for custom security of the schema registry.

Final words

Generally I enjoyed the time working on the library. Compared to Java, which has a much more mature ecosystem it's much easier to create a library which really adds value. Things like good error messages, and linter make it easier to create code I'm confident enough about to share with the community. For any questions regarding the library please use Github Discussions so that others might benefit from the answer as well.

Which talk would you want to see at Kafka Summit

Gerard Klijs — Wed, 11 Dec 2019 12:46:40 +0000

I did talk about using Kafka with GraphQL at GraphQL summit. Now I would like to go speak at Kafka Summit as well. You can help by telling which of the three proposals I had in mind, you would like to see most. Also you should submit as well.

Here they are:

Expose your Kafka using GraphQL

GraphQL is a relatively new way to describe API's and offers subscriptions. Subscriptions are a way to send a stream of updates to the client. Often they are implemented using web sockets, which makes it hard to scale.

I will explain a solution by using a demo application, which is a bank simulation. Also I will share some benchmarks to prove the scalability. This will be done mainly using Clojure, but I also implemented the GraphQL endpoint in Java, Kotlin, and some other languages.

The demo project also contains a load test, which can be used on all the implementations in order to compare them.

From Avro to GraphQL, blazingly fast

Rust is called a system programming language but it's getting more and more use-full as the ecosystem grows. As it's strongly and strictly typed it should be possible to derive both the internal needed data structured to handle binary data based on Avro schema's and to derive a GraphQL schema which can be used by frontend applications.

I will share the results of this journey, with at least some demo, and if all goes well a docker image that can be used to easily interact with Kafka using GraphQL.

Using the Confluent Schema Registry with Rust

In this talk I will go into some of the challenges I faced making a Rust library that is as much as possible similar to the Java serializers. There are a couple of challenges here.
There is no default way of adding serializers with consumers and producers. Another thing is that there is no classpath in Rust to dynamically create specific class instances. Also it turned out the Avro library for Rust didn't support all complex types well.
That being said there are some advantages to using Rust. Like the memory safety, fast startup times, and being able to build very tiny Docker images.

Experience speaking at GraphQL Summit.

Gerard Klijs — Sun, 10 Nov 2019 16:46:30 +0000

See the series for the previous part of the story, or visit the demo, github or video directly.

Preparing for GraphQL Summit

In order to prepare to speak at Graphql Summit I practiced the talk 5 times with other people present. The first time I wasn't ready and only that morning I put some slides together. It was worth the trouble through as I got a lot of feedback. With one of the most important things that there wasn't really clear what the story was.

I gradually improved the slides, making the story more concise, also because I only had 25 minutes. And while I could make some assumptions on people already knowing GraphQL, I could not do the same for Kafka. So I added a very short introduction to Kafka.

The fourth time I practiced my Presentation I got a compliment from someone that was not a developer herself, that she understood the story. I didn't change much after that.

Just before the talk

Flying on my own to the USA was quite an experience on it's own. I never flew alone before, but it's not really that different from going with someone. Just that it's just me finding out where and when to go, but it turned out fine.

After landing I wanted to use the BART to get to the hotel. The machines for the tickets are not user friendly, luckily there was a guard explaining how they worked. When I arrived at my hotel room, I wanted to charge my laptop again, that I also used in the plane. It turned out I didn't bring the correct converter with me. So after visiting some places, I finally succeeded at the Mac Store.

Going back I only had little time before getting ready to meet the other speakers while having dinner. It was really nice to meet so many interesting people.

When I came back in the Hotel I wanted to finish something I started just before leaving, trying to run the demo backend on a remote machine using docker. The big problem was that because the frontend was running on https using Netlify the backend websocket had to be wss instead of ws as wel. Eventually I worked it out. I also tried the clicker I brought with me, to not have any surprises the next day.

Day of the presentation, first day of GraphQL Summit

Conveniently the summit was held in the hotel I was staying. So I got downstairs for some breakfast. Usually at large conferences you have to eat standing, and wait for some time in the line to grab something. I was pleasantly surprised it was both a decent breakfast and that there where enough tables to sit down.

After keynotes from Matt DeBergalis, explaining mainly the current and future state of GraphQL, and the talk talk from Brie Bunge about how you can start using GraphQL in a big company, there was a break before splitting up in three different tracks.

Because my talk was the second one in the Bayview room I went there on time. There where several people present from the organization that put me on easy, and also hard me wired up before the first talk started.

I was quite nervous, but also happy once I could start to get my presentation over with. During the start of my presentation people kept walking in the room. Because I took quite some time to introduce myself and what I was going to tell, they didn't miss much and I pretty much just continued the presentation. Aside from the url for the demo not being on the slides themselves, causing it to take sometime before people joined the demo it went fine.

After the talk

It was great seeing all the other presentations while no longer having to worry about my own. Also some people had additional questions or tried to read the source code which was great. During lunch they also had this great idea of having tables where certain topics could be discussed. I would really like to see that at other conferences as well.

Just today I also finished a Java Spring boot implementation of the GraphQL endpoint. The Kotlin GraphQL server library also seems great, so I really want to give that a try. Maybe I also add both Rust and Node.js so I could compare the bunch.

For the query part I want to experiment with Hasura and Postgraphile using the current PostgreSQL Database. But I'm also interested in changing the database to either Neo4j, Dgraph or FaunaDB. It should make certain parts easier, and also give access to some advanced features like mutations and pagination.

I'm less sure after the summit subscriptions are a good fit when using Kafka and GraphQL. Especially because some tools don't work with subscriptions and they are hard to scale. To put something on Kafka it could also be enough just to know it was successfully send. Using queries for the derived view, the clients could make sure the command was properly processed.

Just before leaving for the summit I got notice of arboric, which is also something I want to get some experience with. As it seems to be able to handle a few things like authorization and authentication in a nice way. It might even bee used to make subscriptions scalable in a nice way, by directing users to the 'correct' instance based on their token.

Open bank mark goes USA

Gerard Klijs — Thu, 24 Oct 2019 20:52:56 +0000

Last few months have been really busy. I was finalising the now finally published blog postabout using Rust with Kafka. And also preparing to talk at GraphQL Summit which will be the first conference I'll be speaking.

Been a really hectic time especially since there was also some trouble at home, leaving less time as usual to do stuff. But I'm really looking forward to visit San Fransisco and learning more about GraphQL.

I'm still busy with the finishing touches to the repo that is more centred about the GraphQL part. One of the most important changes is using the GraphQL endpoint to increase the load of the system. If all goes well the presentation will be recorded.

Taking open-bank-mark to The Dutch Clojure Meetup

Gerard Klijs — Wed, 12 Jun 2019 21:35:01 +0000

I just presented open-bank-mark at the The Dutch Clojure Meetup. Focussing more on the Clojure libraries used than I did at the Kafka meetup.

In preparation I tested with setting linger.ms to 0 and 100 ms. Witch is one of the variables in Kafka that's really affecting your latency and throughput.

It was nice of Open Web to sponsor. Because it's a small meetup with only about 10 people today, there was lots of room for questions, and also sometimes for other people to answer when I didn't know. The raw data can be found here.

The human side of open-bank-mark

Gerard Klijs — Wed, 05 Jun 2019 18:28:31 +0000

TL;DR: I had a great time creating open-bank-mark even if your scared of parentheses you might want to check it out.

Recently I open sourced open-bank-mark and a week ago at the Kafka Meetup Utrecht was the fifth meetup I talked about it. During the drinks after the talk I often get questions about how it came to existence. It seemed to be like a good idea to write it down, which is what I'm doing right now.

What is open-bank-mark?

// Detect dark theme var iframe = document.getElementById('tweet-1130516421677658114-533'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=1130516421677658114&theme=dark" }

Although it nowhere explicitly mentioned in the project itself, the name of the project is a combination of 'open-bank' and benchmark. 'open-bank' was the name for the frontend part before I combined everything in one project. It's a reference to my employer Open Web but a banking application. It's also open in the sense that the code is publicly available and there is little to no security. The last part I added makes it possible to run benchmarks. For example you could switch out components for equivalent ones in another language. After running both on some environment the latency and the use of both cpu and memory of the different parts can be compared.

openweb-nl / open-bank-mark

A bank simulation application using mainly Clojure, which can be used to end-to-end test and show some graphs.

Active development has moved to kafka-graphql-examples which is more focused about graphql, and in ways is simpler than this project.

Open Bank Mark

Contents

Intro

This project is an example of an event sourcing application using Kafka The front-end can be viewed at open-bank which is for now configured to have the endpoint running on localhost In the background tab are the results of comparing 4 languages, which all ran 10 times on TravisCi, with one broker. It also contains an end-to-end test making it possible to compare different implementations or configurations. For example one could set the linger.ms setting at different values in the topology module, so everything build on Clojure will use that setting. Another option would be to drag …

View on GitHub

The passion project is kind of organically grown over the course of about one and a half years. With sometimes a month of not doing anything about it, and sometimes a month of working on it almost every evening in order to get sufficient progress for the next meetup. I used call projects like these pet projects, but never really liked how it sounded. Thanks to the article, Why every developer should have a passion project, I now have a better name for such projects.

What's next will be a roughly chronological story about the different stages of development of open-bank-mark.

Kafka platform at Rabobank

A big part of the project revolves around Kafka. It's used to decouple different components and also to make it potentially scalable. Kafka was the first of some of the tech that plays an important role in open-bank-mark I got acquainted with.

I started working with Kafka in November 2015 when I was working at the Rabobank on what would later would become the Axual platform. Previous to my start a demo was created to make people enthusiastic about Kafka. The demo showed the balances of fictional accounts, with notifications when a certain condition was met. It was kind of similar to what open-bank-mark would be. But the transactions where not executed by commands and there were a fixed amount of accounts.

I learned a great deal creating the client library part. One thing in particular I learned was that is is hard to write concurrent code with Java. The Kafka Consumer not being safe for multi-threaded access made it even harder. Which was one of the reasons that would get me interested in Clojure later on.

Clojure for the Brave and True

During the summer of 2016 I read Clojure for the Brave and True. It was fun and clear book to read. But when it came to actual programming I had a hard time to use Emacs like the book prescribed.

Only some months later I learned about Cursive which was a plugin for the IDE I was already familiar with, IntelliJ. It was a nice refreshing language, with some of the first passion projects being a snake game with the frontend with Clojurescript. It's nice to be able to use the same code for the backend and frontend. Even the game itself can be viewed in the browser or with Java using the same library.

Later on I added some rule based artificial intelligence to the snake game. I presented the snake game, together with a client for it at one of our 'Open Pizza' sessions on the fourth of May. An 'Open Pizza' session is like a regular meetup, but only for employees of Open Web.

Kafka workshop

About a year after the snake game it was time for another presentation at 'Open Pizza'. I wanted to do a Kafka workshop, to share some of my experience with it. Both event sourcing and GraphQL where the subject of 'Open Pizza' before I started preparing my talk. Both of them where demonstrated using Java, so I wanted to do something using both, with Kafka and Clojure.

And so I started with the first parts of open-bank-mark. Since I build most of it evening I quickly became annoyed with having to set the correct topics and schema's each time. So that's why I added some tooling to set those based on some edn files.

In advance I only wanted to build the backend part. But after I had that ready I couldn't resist to give it a try to create a frontend using Clojurescript. I quickly got something working, using re-graph for the GraphQL logic, and Bulma for the css. I was pretty happy with it, and since the Clojure meetup at 14 February had an open slot I did a talk showing what I had so far. It was nice to share the project and I got some valuable feedback.

Almost two months later, 5 April 2018 it was finally time to do an open pizza talk again. Turned out I focused a but too much on it being a workshop. I did have both for frontend and backend thought of things that could be improved. But I didn't have a proper introduction of the whole project to make clear what the meaning was.

I also added the Kotlin variant of the Command Handler just before the talk. I created it mostly as a non Clojure JVM example. But besides making it more complex as a whole it also caused my live demo to fail. This was because while running Clojure, some values in the database were null, and Kotlin expected them to always have values based on the data types. I did test both variants separately but I didn't test the switching before. Also an important lesson, when doing live demo's make sure you only do things you did before, and not something different.

Because of the complexity and not having a proper introduction, only a few people actually started coding. Creating a different frontend was also not without problems since there were some CORS issues when using Angular that I could not solve quickly.

Rust 2018 edition release party

Before I heard there would be a 2018 edition release party on 7 February this year, I had already done a few things with Rust. You can read more about at here. At the release party there would be a total of 6 talks and also some demo's so speaking time was limited.

I already wrote the Rust variant of the Command Handler, and in order to do that I needed something to change the bytes I received from Kafka, with the schema from the schema registry, to typed data I could use in Rust. Trying out several combinations of existing libraries I got it working. Also my experience with the Rabobank helped a lot, since I already had a good grasp on how this worked with Java. Especially since we wrote out own serialisers wrapping the Confluent Avro serialiser.

As part of learning Rust I wanted to turn the serialiser into a proper library, or a crate since that's how they are called in Rust. It's currently at 1.0.0 and can be found on crates.io. In order to make it a crate I not only put the code together, but also made it more generic. I also added some features to make it equivalent to the Java library that does the same. Another thing I did was adding unit and integration tests, and start used codecov to track code coverage.

The release party was my first Rust meetup and the first time I needed to wear a microphone so that got me a bit nervous. I think because of the nerves I went a bit too fast trough my slides. From some of the questions it wasn't all that clear what I did exactly. I also showed me bank demo, but most people were looking at the demo with web assembly witch happened at the same time, also some people where leaving since the demo's were the last part of the program. It was a nice setup with some people showing projects that where running in production, or almost there.

Kafka Meetup Utrecht

I also sometimes tweeted about open-bank-mark and got asked by a former colleague to present at the Kafka Meetup Utrecht. The talk was planned for 28 May this year. Since it was a Kafka meetup, and one of the reasons Kafka is often used is performance, I would seem look a good idea to try to compare the different implementations of the Command Handler.

In preparation for the Open Pizza the year before I already started making a test of the frontend, using etaoin with the chrome webdriver. So in order to to end-to-end performance testing I needed to complete that and eventually had a way of making transactions and validating then while measuring the time it took.

Then I created some code around it to be able to do actual testing. I first tried a library that would use the pid of the process to measure cpu usage and memory bit it was tedious to get the correct pid each time. Also I doubted the measurements. The next way to improve was to use docker to set the project up. This way I could name the containers and didn't needed the pid's anymore. I used lispyclouds/clj-docker-client to measure cpu and memory of the Docker containers, after my pull request was merged to add that feature. Using Docker has some other benefits like making it easier to switch JDK's. I ran it for some nights on my laptop, getting rid of some bugs that were sometimes break the testing program.

Now I was able to generate a lot of data, but didn't have a way to visualize it yet. I ended up using a combination of kixi.stats to preprocess the data and oz to generate html with some vega imports to show the data. The results can be viewed from the background tab on open bank. Now I really want to add routing to the frontend, but I better don't else the project keeps ongoing forever.

Just before the presentation I figured I could also use the other Rust Kafka Library. I ignored it first time cause it didn't seem well maintained. It still isn't, but people are using it and creating issues on github. Also being a native rust solution it was possible to get a tiny docker image easily.

// Detect dark theme var iframe = document.getElementById('tweet-1132220825912979457-818'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=1132220825912979457&theme=dark" }

Despite a strike of the public transport that day the meetup went through and about thirty people showed up. Since I had a lot of time to prepare for this talk and there would be around fifty people coming, I really wanted to do a proper job preparing. But I also found some small errors either in the code or the test setup. So only had my slides ready just the night before the presentation. Leaving me without time to exercise it out loud. Which caused me to sometimes search for words a little, but on overall I was quite pleased with how it went. The slides from the presentation are available.

What to do next?

I've had several positive reactions from open sourcing and sharing the project. It's also added to the lacinia docs as an example project. I don't use twitter very actively so it was quit nice to have scicloj mention me.

// Detect dark theme var iframe = document.getElementById('tweet-1133853325164711936-703'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=1133853325164711936&theme=dark" }

The last additions to open-bank-mark were some small improvements to the documentations. Also I thought it would be cleaner to leave only master and the 'one-broker' branch in the main project, and move the variants to my own fork. At the same time I added a variants section to the readme witch could host links to other variants. I don't know if I'll spend much more time on the project. I would like to give crux a try through.

xtdb / xtdb

An immutable SQL database for application development, time-travel reporting and data compliance. Developed by @juxt

XTDB is an open-source immutable database with comprehensive time-travel. XTDB has been built to simplify application development and address complex data compliance requirements. XTDB can be used via SQL and XTQL.

XTDB 2.x is in 'beta' whilst we collaborate closely with our Design Partners ahead of General Availability; if you are looking for a stable release of an immutable document database with bitemporal query capabilities, we are continuing to develop and support XTDB 1.x at https://github.com/xtdb/xtdb/tree/1.x.

Major features:

Immutable - XTDB is optimised for current-time queries, but you can audit the full history of your database at any point, without needing snapshots or accessing backups.
'Bitemporal' - all data is accurately versioned as updates are made ('system' time), but it also allows you to separately record and query when that data is, was, or will become valid in your business domain ('valid' time).
Dynamic - you don’t need…

View on GitHub

Please let me know what you think of the story and/or of the project.

Forem: Gerard Klijs

Why you might eventually wish you started out using event sourcing

Philosophical trickery

Event sourcing in the weak sense

Event sourcing in the strong sense

Use case, PEF, a payment provider

PEF starts using event sourcing in the weak sense

Some challenges ahead

Customer wants to know why he's blocked

Data analyst want to see correlations

Governance needs some improvements

What can we learn from PEF

Confluent Schema Registry and Rust

Intro

Kafka

Confluent Schema Registry

Rust

Bank demo project

Schema Registry Converter

Maintaining the library

Final words

Which talk would *you* want to see at Kafka Summit

Expose your Kafka using GraphQL

From Avro to GraphQL, blazingly fast

Using the Confluent Schema Registry with Rust

Experience speaking at GraphQL Summit.

Preparing for GraphQL Summit

Just before the talk

Day of the presentation, first day of GraphQL Summit

After the talk

Open bank mark goes USA

Taking open-bank-mark to The Dutch Clojure Meetup

The human side of open-bank-mark

What is open-bank-mark?

openweb-nl / open-bank-mark

A bank simulation application using mainly Clojure, which can be used to end-to-end test and show some graphs.

Open Bank Mark

Intro

Kafka platform at Rabobank

Clojure for the Brave and True

Kafka workshop

Rust 2018 edition release party

Kafka Meetup Utrecht

What to do next?

xtdb / xtdb

An immutable SQL database for application development, time-travel reporting and data compliance. Developed by @juxt

Which talk would you want to see at Kafka Summit