Forem: Mark A. Greenslade

Casper Kafka Event Store Pt 2

Mark A. Greenslade — Fri, 21 Oct 2022 10:58:46 +0000

Building a Kafka Cluster in a Kubernetes Cluster

Carl Sagan quoted that If you wish to make an apple pie from scratch, you must first invent the universe.

Luckily the universe has already been invented, but we still need to build our Kubernetes cluster to host our Kafka cluster.

For this POC I chose GCP. They’ve done some good work in minimising the UI. Plus the networking works seamlessly behind the scenes.

Install gcloud and kubectl and run the following command to create the cluster:

gcloud beta container \
 clusters create "[CLUSTER_NAME]" \
 --project "[PROJECT_NAME]"  \
 --zone "[ZONE]" \
 --machine-type "e2-medium" \
 --num-nodes 9 \
 --disk-size "100"

After about 5 minutes we should have a cluster.

Now we can add Zookeeper and Kafka.

Use the installation guide from this GitHub link

As we have no Producers or Consumers configured yet, Kafka will be in a waiting state. No leader has been elected yet. This all happens when topics and partitions are created. More on that later.

Things to note in this cluster

Zookeeper is a StatefulSet of three replicas. We need consistently named pods in the cluster. If we used a Deployment, every time a new pod was instantiated, it would be randomly name with a deployment hash suffix.

With StatefulSets you get consistent pod names. In this case zookeeper-0, zookeeper-1 and zookeeper-2

Likewise with the Kafka brokers.

The Kafka broker StatefulSet starts Kafka.

Lines 2–4 define the Dockerfile, source can be viewed here

I chose to roll my own Dockerfile to allow more control. There are many Kafka Docker images in the hub, but for this project I needed to be able start Kafka with overridden options.

Lines 5–8 are allocated resources per pod. I used Kenneth Owens excellent templates here to configure this POC.

Lines 12–20 are environment variables. Line 13 sets the internal IP of the newly created pod. Lines 17–20 are standard java options.

Lines 21 onward are Kafka start up options.

Line 24 starts Kafka with the default server properties files, we then override these properties in the following lines with a mix of Kubernetes and custom env vars

Line 25 sets the broker id from the pod hostname, eg pod kafka-broker-1 is assigned broker number 1 (StatefulSets give us this naming convention as discussed above)

Lines 26–29 set the listeners. The brokers need to know how to communicate with each other internally and with external clients. This is difficult to get right. These settings work and will save a lot of pain. Line 26 tells Kafka to use the default 0.0.0.0 which means listen on all interfaces. Line 27 uses the pod ip env var, KAFKA_POD_IP for the advertised listener. Confluent have a good article on listeners here.

Line 28 tells Kafka to use the text string ‘INSIDE’ as the inter broker listener name. I assumed that when it sees this string it knows that this is an internal listener.

Line 29 as this POC is an internal Kafka ensemble we can use PLAINTEXT for the security protocol. This is setting it to no encryption or authourisation. This will be changed to SSL for any production cluster.

Line 30 uses Kubernetes env vars to point to our zookeeper pods. The zookeeper deployment uses a Kubernetes service for gateway access. This is all we need to configure the broker. This gives us the option of *n *zookeepers. We’re just using 3 in this POC

After running the steps in on the GitHub page we will have a sucesfully running Kafka/Zookeeper ensemble within a Kubernetes cluster. The ensemble will be in a waiting state. Part 3 of these articles will discuss and demonstrate how to use the cluster, with Casper node events.

Casper Kafka Event Store Pt 1

Mark A. Greenslade — Thu, 20 Oct 2022 15:58:11 +0000

The Description

Casper nodes emit events on state changes. These events in the platform are categorised into three event stream types:

Deploy Events
Finality Signature Event
Main Events

Within each stream type there are multiple event types such as DeployAccepted or BlockAdded

See the Casper documentation for a full description.

The Problem

As can be seen from the diagram below, there will be n nodes emitting 3 event stream types.

This is a lot of events.

Even if the event stream is restricted to one or a group of nodes we are still seeing many events.

Casper clients use these events to build up a picture of state changes in the node and smart contracts.

These state changes can notify audit tools of any discrepancies, such as suspicious transactions, asset mutation or chain failure.

The Solution

How can we handle so many events?

To reliably consume these event streams we need a high throughput, highly available messaging service.

We need to differentiate the event types.
We need to load balance the event stream types into separate silos (Finality Signature events are the majority of events by many multiples)
We need to be able to replay the events in the case of any data loss or DR
We need confidence in our architecture

Enter Kafka, which will satisfy the concerns above.

Kafka has concepts such as Brokers, Partitions, Replicas, Topics which will all be discussed in details in later sections.

For now we just need to know that Kafka will be running n brokers acting as a cluster. The Kafka cluster will use Zookeeper to orchestrate the brokers.

Kafka uses the leader/follower pattern where it will decide amongst itself who is the leader and who follows. If the leader is lost, a new leader will be elected. An odd number is preferred as is always the case with clusters.

Like all messaging systems, we need to produce and consume messages, Kafka like RabbitMQ handily names these as Producers and Consumers. Lets look into how these fit into our architecture:

A simple and clean architecture.

The event streams are read by the producer, any transformations are made, eg text to json, and the event is added to a Kafka topic.

The consumer will then retrieve this event via its topic and process it accordingly.

Where can we host this architecture?

Kubernetes. Whatever the question in DevOps, Kubernetes is usually the answer.

For all it’s minor annoyances (looking at you YAML), what other system can give us guaranteed uptime, replicated apps, ease of deployment/updates etc etc. What else are we going to use? Docker Swarm….

So that being said, with K8s we can run multiple instances of the producer and consumer. We can also run the Kafka cluster within the K8s cluster to further guarantee its availability. Here’s what the final architecture diagram looks like:

Consumer groups are a very useful concept added to our solution. They each read a message from a topic once, so both Consumer Group A and B will both read the same message exactly once. Any other consumers can also read the message (Kafka has a time to live on messages)

As can be seen, we are writing to both an Audit and UI document store. Audit will be used to replay the event history in case of DR, UI will be a structured store which can be used by any UI or reporting tools.

Implementation details will follow in the coming articles