Forem: Emil Koutanov

Deleted

Emil Koutanov — Thu, 11 Feb 2021 07:20:12 +0000

Deleted

An Introduction to Microservices

Emil Koutanov — Thu, 11 Feb 2021 07:16:45 +0000

A Brief History

The term microservices was first coined by Dr. Peter Rodgers in 2005 and was initially known as “micro web services”. The main driver behind “micro web services” at the time was to break up single large “monolithic” designs into multiple independent components/processes, thereby making the codebase more granular and manageable.

Modular, distributed applications date for back several decades. And in this regard, microservices are not a new concept. However, what popularised microservices was the principles governing how they were designed and the way they were consumed. While conventional distributed systems of that era relied on proprietary communications protocols, microservices took advantage of open standards such as HTTP, REST, XML and JSON.

A Simple Definition

A microservice is a small, loosely coupled, distributed service. It is part of a broader microservices architecture, comprising a set of loosely coupled microservices that operate together to solve a common goal. A collection of microservices can be regarded as a system.

Read the rest of the article on Medium.

Combining strict order with massive parallelism using Kafka

Emil Koutanov — Mon, 25 Jan 2021 19:36:21 +0000

Getting the best of both worlds

Having been involved in several large-scale Kafka projects for different clients across a broad range of industries, I have heard my fair share of questions on Apache Kafka — ranging from the fundamental to the esoteric. One question that never seems to go out of fashion is: How can you maintain strict order, yet still process records in parallel?

And it's a fair question. Strict order assumes linearizability, the very notion of which seems to contradict with the objectives of parallelism.

Partial and total order

We will start by exploring the notion of order.

As expected of an event-streaming platform, Kafka preserves the order of published records, providing those records occupy the same partition. In order to understand what this means in practice, one needs to explore the architecture of Kafka topics, and the underlying sharding mechanism — partitions.

Read the rest of the article on Medium.

Contrasting NATS with Apache Kafka

Emil Koutanov — Sun, 06 Sep 2020 20:44:03 +0000

TL;DR Kafka is an Event Streaming Platform, while NATS is a closer to a conventional Message Queue. Kafka is optimised around the unique needs of emerging Event-Driven Architectures, which enrich the traditional pub-sub model with strong ordering and persistence semantics. Conversely, NATS is highly optimised around pub-sub topologies, and is an excellent platform for decoupling systems where message order and reliable delivery is a non-issue.

I’ll preface this post by pointing out that there is another product — NATS Streaming — which is a different beast and is closer to Kafka. You may want to take a detour to NATS Streaming if you after an alternative event streaming platform; otherwise, read on.

Subscriptions

At its core, NATS is about publishing and listening for messages. These depend heavily on subjects which scope messages into streams or topics. Consumers subscribe to topics either verbatim (matching the topic name precisely), or using wildcards. Below is an illustration of publisher-subject-consumer relationship in NATS.

Read the rest of the article on Medium.

Apache Kafka in a Nutshell

Emil Koutanov — Sat, 25 Apr 2020 06:15:05 +0000

So, you’ve heard of this Kafka thing that’s popping up all over the shop. From green-sprout startups to greedy multi-nationals, in all sorts of weird and wonderful contexts. It appears that someone has figured out a Swiss Army knife piece of über-tech that works for microservices, event-streaming, CQRS, and just about anything else that makes the headlines.

As someone who’s been working with Apache Kafka since 2015, helping several clients build complex event-driven and microservices-style applications, I’ve heard my fair share of questions. First, it was I who was doing most of the asking. Then I found myself on the receiving end. At the time of writing, I’m trending among the top-ten writers on Quora — a feat I owe mostly to the mistakes I’ve made along the way, which have made me a little more aware. With the shameless chest-beating out of the way, I thought I’d put together an FAQ-style post that breaks Kafka down, focusing on the aspects that matter.

Read the rest of the article on Medium

Why is Kafka so Fast

Emil Koutanov — Thu, 06 Feb 2020 22:04:35 +0000

The last few years have brought about immense changes in the software architecture landscape. The notion of a single monolithic application, or even several coarse-grained services sharing a common data store has been all but erased from the hearts and minds of software practitioners world-wide. Autonomous microservices, event-driven architecture, and CQRS are the dominant tools in the construction of contemporary business-centric applications. To top it off, the proliferation of device connectivity — IoT, mobile, wearables — are creating an upward pressure on the number of events a system must handle in near-real-time.

Let's start by acknowledging that the term 'fast' is multi-faceted, complex, and highly ambiguous. Latency, throughput, jitter, are metrics that shape and influence one's interpretation of the term. It is also inherently contextual: the industry and application domains in themselves set the norms and expectations around performance. Whether or not something is fast depends largely on one's frame of reference.

Apache Kafka is optimised for throughput at the expense of latency and jitter, while preserving other desirable qualities, such as durability, strict record order, and at-least-once delivery semantics. When someone says 'Kafka is fast', and assuming they are at least mildly competent, you can assume they are referring to Kafka's ability to safely accumulate and distribute a very high number of records in a short amount of time.

Read the rest of the article on Medium.

Getting Started with Istio

Emil Koutanov — Mon, 20 Jan 2020 07:56:52 +0000

The last few years have brought about immense changes in the software architecture landscape. A major shift that we have all witnessed is the breakdown of large monolithic and coarse-grained applications into fine-grained deployment units called microservices, communicating predominantly by way of synchronous REST and gRPC interfaces, as well as asynchronous events and message passing. The benefits of this architecture are numerous, but the drawbacks are equally evident. Aspects of software development that used to be straightforward in the 'old world', such as debugging, profiling and performance management, are now an order of magnitude more complex. Also, a microservices architecture brings its own unique challenges. Services are more fluid and elastic, and tracking of their instances, their versions and dependencies is a Herculean challenge that balloons in complexity as the service landscape evolves. To top this off, services will fail in isolation, further exacerbated by unreliable networks. Given a large enough system, parts of it may be suffering a minor outage at any given point in time, potentially impacting a subset of users, quite often without the operator's awareness. With so many 'moving parts', how does one stay on top of these challenges and ensure the system is running smoothly without impacting the customer and driving the developers out of their wits?

With the increased adoption of microservices, the industry has been steadily coming up with patterns and best-practices that have made the entire experience more palatable. Resiliency Patterns, Service Discovery, Container Orchestration, Canary Releases, Observability Patterns, BFF, API Gateway... These are some of the concepts that practitioners will employ to build more robust and sustainable distributed systems. But these concepts are just that — abstract notions and patterns — they require someone to implement them somewhere in the system. More often than not, that 'someone' is you and 'somewhere' is everywhere.

Read the rest of the article on Medium

Logging on the cheap

Emil Koutanov — Tue, 10 Dec 2019 01:26:55 +0000

To log or not to log?

Writing new code is a relatively small part of application development. A much larger part of a software engineer's life involves supporting and maintaining existing systems. First and foremost, this requires the right level of insight — understanding what exactly the system is doing, particularly when this isn't quite what you (and your customers) were expecting. Logging, tracing, remote debugging, and application performance monitoring are essential in attaining the level of observability required to comfortably operate and support a modern application suite.

When building conventional applications, the question of whether an action should be logged or not is rarely a subject of rigorous debate. (By 'conventional', we mean most e-commerce applications with fairly routine non-functional requirements concerning throughput, latency, availability, and so forth.) Occasionally, we come across a class of applications that breaks that mould. Sometimes particular modules of an otherwise conventional system might be subject to very stringent constraints around their performance. An application might perform an expensive calculation in a tight loop, or it may be required to respond (on average) within a tight time frame. Invasive instrumentation of these applications changes their timing behaviour — even simple log calls carry an overhead that might cardinally impact the customer experience. In these sorts of scenarios, the question posed at the beginning of the article — To log or not to log? — becomes a real point of debate. Do you sacrifice break-neck performance for observability, or is performance the coveted money-maker that dwarfs every other concern?

Introducing Zerolog?

This article is going to explore Zerolog (abbreviated to Zlg) — an ultra-low-overhead logging façade for performance-sensitive Java applications. The objectives of Zlg are twofold —

Ultra-low overhead for suppressed logging. In other words, the cost of calling a log method when logging for that level has been disabled is negligible. Furthermore, it is possible to invoke Zlg in such a way as to reduce these costs to absolute zero.
Uncompromised code coverage. Suppression of logging should not impact statement and branch coverage metrics. A log entry is a statement like any other — if it's worth writing, it's worth testing.

Read the rest of the article on Medium.

Official Kafdrop Docker Image

Emil Koutanov — Mon, 11 Nov 2019 21:50:57 +0000

In case you were wondering, Kafdrop has an official Docker image. Which is great; it means you don't have to clone the repo, mess around with Maven builds or even install the JVM.

Docker images are updated automatically with each release build of Kafdrop, courtesy of the public Travis build pipeline. The latest tag on DockerHub always points to the most recent stable build.

Getting started

To run Kafdrop, run the following:

docker exec -it -p 9000:9000 \
    -e KAFKA_BROKERCONNECT=host:port,host:port \
    obsidiandynamics/kafdrop:latest

Replace host:port,host:port with the bootstrap list of Kafka host/port pairs. Once the container boots up, navigate to localhost:9000 in your browser. You should see the Kafdrop landing screen:

Hey there! We hope you really like Kafdrop! Please take a moment to ⭐ the repo or Tweet about it.

Configuration

The Kafdrop Docker image is configured by simply passing in environment variables:

KAFKA_BROKERCONNECT: Bootstrap list of Kafka host/port pairs.
JVM_OPTS: JVM options.
JMX_PORT: Port to use for JMX. No default; if unspecified, JMX will not be exposed.
HOST: The hostname to report for the RMI registry (used for JMX). Default: localhost.
SERVER_PORT: The web server port to listen on. Default: 9000.
SERVER_SERVLET_CONTEXTPATH: The context path to serve requests on (must end with a /). Default: /.
KAFKA_PROPERTIES: Additional properties to configure the broker connection (base-64 encoded).
KAFKA_TRUSTSTORE: Certificate for broker authentication (base-64 encoded). Required for TLS/SSL.
KAFKA_KEYSTORE: Private key for mutual TLS authentication (base-64 encoded).

Connecting to a secure cluster

Need to connect to a secure Kafka cluster? No problem; Kafdrop supports TLS (SSL) and SASL connections for encryption and authentication. This can be configured by providing a combination of the following files:

Truststore: specifying the certificate for authenticating brokers, if TLS is enabled.
Keystore: specifying the private key to authenticate the client to the broker, if mutual TLS authentication is required.
Properties file: specifying the necessary configuration, including key/truststore passwords, cipher suites, enabled TLS protocol versions, username/password pairs, etc. When supplying the truststore and/or keystore files, the ssl.truststore.location and ssl.keystore.location properties will be assigned automatically.

These files are supplied in base-64-encoded form via environment variables:

docker exec -it -p 9000:9000 \
    -e KAFKA_BROKERCONNECT=host:port,host:port \
    -e KAFKA_PROPERTIES=$(cat kafka.properties | base64) \
    -e KAFKA_TRUSTSTORE=$(cat kafka.truststore.jks | base64) \   # optional
    -e KAFKA_KEYSTORE=$(cat kafka.keystore.jks | base64) \       # optional
    obsidiandynamics/kafdrop

Running Kafdrop in Kubernetes

You can also launch Kafdrop in Kubernetes with the aid of Helm chart.

Clone the repository (if necessary):

git clone https://github.com/obsidiandynamics/kafdrop && cd kafdrop

Install (or upgrade) the chart. Replace 3.x.x with the latest stable release.

helm upgrade -i kafdrop chart --set image.tag=3.x.x \
    --set kafka.brokerConnect=<host:port,host:port> \
    --set kafka.properties="$(cat kafka.properties | base64)" \
    --set kafka.truststore="$(cat kafka.truststore.jks | base64)" \
    --set kafka.keystore="$(cat kafka.keystore.jks | base64)"

A list of settable values can be found in values.yaml.

I'm one of the maintainers of Kafdrop and a bit of a Kafka enthusiast. I'm also an avid crusader for microservices and event-driven architecture. If you're interested in these topics, or just have a question to ask, grab me on Twitter.

Kafka Gotchas – The Must-Know

Emil Koutanov — Mon, 04 Nov 2019 03:51:02 +0000

I've assisted several large clients in building a microservices-style architecture using Kafka as a messaging backbone, having a reasonably good understanding of its abilities and the use cases that really bring them out. But I'm not a Kafka apologist by any stretch; any technology that has gone through such a rapid adoption curve is bound to polarise its audience and rub certain developers up a wrong way, and Kafka is no exception. Like anything else, you need to invest a significant amount of time in getting across Kafka and event streaming in general, before you become fully proficient and can harness its might. And be prepared to face one or two frustrations, to put it mildly, along the way.

I've compiled a list of shortcomings that may cause developer frustration, or catch out unsuspecting first-timers. In no particular order:

Too many tunable knobs

The number of configuration parameters in Kafka can be overwhelming, not just for newcomers but also seasoned pros. Possibly with the sole exception of the JVM, I cannot think of another technology that has this many configuration parameters. This isn't to say that the config options aren't necessary; but one does wonder how many of those parameters could instead be replaced by ergonomics, much like Java did with G1. So rather than specifying a plethora of individual thresholds and tolerances, let the operator set a performance target, and have the system derive an optimal set of values that best meet this target.

Read the rest of the story on Medium.

Introduction to Event Streaming with Kafka and Kafdrop

Emil Koutanov — Sun, 03 Nov 2019 08:12:10 +0000

Event sourcing, eventual consistency, microservices, CQRS... These are quickly becoming household names in mainstream application development. But do you know what makes them tick? What are the basic building blocks required to assemble complex, business-centric applications from fine-grained services without turning the lot into a big ball of mud?

This article examines a fundamental building block — event streaming. Leading the charge will be Apache Kafka — the de facto standard in event streaming platforms, which we'll observe through Kafdrop — a feature-packed web UI.

A Brief Intro

Event streaming platforms reside in the broader class of Message-oriented Middleware (MoM) and are similar to traditional message queues and topics, but offer stronger temporal guarantees and typically order-of-magnitude performance gains due to log-structured immutability. In simple terms, write operations are mostly limited to sequential appends, which make them fast. Really fast.

Read the rest of the article on Medium

Kafdrop - An Open-Source Kafka Web UI

Emil Koutanov — Tue, 22 Oct 2019 10:10:22 +0000

As a messaging platform, Kafka needs no introduction. Since its inception, it has virtually rewritten the book on event streaming and has catalysed the adoption of the now household design patterns — microservices, event-sourcing and CQRS.

Being such a godsend, it almost gets away with its notorious lack of tooling. You'd be hard-pressed to find a developer who hasn't at one time looked at the built-in CLI tools, cupped their face and uttered: "Is this it? Are you kidding me?"

What's Out There

With the popularity of Kafka, it's no surprise that several commercial vendors have jumped on the opportunity to monetise Kafka's lack of tooling by offering their own. Kafka Tool, Landoop and KaDeck are some examples, but they're all for personal use only unless you're willing to pay. (And it's not to say that you shouldn't, but that's rather beside the point.) Any non-trivial use in a commercial setting would be a violation of their licensing terms. It's one thing using them at home for tutorials or personal projects; when you're using a commercial tool without the appropriate license, you are putting your employer at risk of litigation and playing Russian Roulette with your career.

But what about open source?

When it comes to Kafka topic viewers and web UIs, the go-to open-source tool is Kafdrop. With 900K Docker pulls at the time of writing, I don't believe there are many other Kafka tools that have enjoyed this level of adoption. Kafdrop lets you view topic contents, browse consumer groups, view consumer lag, topic configuration, broker stats, and one or two other things. All in all, it does an amazing job of filling the apparent gaps in the observability tooling of Kafka, solving problems that the community has been pointing out for too long.

Kafdrop is an Apache 2.0 licensed project, like Apache Kafka itself. So it's won't cost you a penny. If you haven't used it yet, you probably ought to. So let's take a deeper look.

Getting Started

The Kafdrop web UI project is hosted on GitHub:

obsidiandynamics / kafdrop

Kafka Web UI

Kafdrop – Kafka Web UI

Kafdrop is a web UI for viewing Kafka topics and browsing consumer groups. The tool displays information such as brokers, topics, partitions, consumers, and lets you view messages.

This project is a reboot of Kafdrop 2.x, dragged kicking and screaming into the world of Java 17+, Kafka 2.x, Helm and Kubernetes. It's a lightweight application that runs on Spring Boot and is dead-easy to configure, supporting SASL and TLS-secured brokers.

Features

View Kafka brokers — topic and partition assignments, and controller status
View topics — partition count, replication status, and custom configuration
Browse messages — JSON, plain text, Avro and Protobuf encoding
View consumer groups — per-partition parked offsets, combined and per-partition lag
Create new topics
View ACLs
Support for Azure Event Hubs

Requirements

Java 17 or newer
Kafka (version 0.11.0 or newer) or Azure Event Hubs

Optional, additional integration:

Schema Registry

Getting Started

You…

View on GitHub

There's a couple of options at your disposal. You could show a little bravery by cloning the repository and building from source. It's a Java (JDK 11) SpringBoot project, and you can build it with a single Maven command, providing you have the JDK installed. If you want to go down this path, the repo's README.md file will guide you through the steps. For now, let's take the easy way — Docker. (I sure would.)

Docker images are hosted on DockerHub. Images are tagged with the Kafdrop release number. The latest tag points to the latest stable release.

To launch the container in the background, run the following command:



docker run -d --rm -p 9000:9000 \
    -e KAFKA_BROKERCONNECT=<host:port,host:port> \
    obsidiandynamics/kafdrop

The KAFKA_BROKERCONNECT environment variable must be set to the bootstrap list of brokers.

That's it. We should be up and running. Once it starts, you can launch the Kafka web UI by navigating to localhost:9000.

Note: The above example assumes an authenticated connection over a plaintext TCP socket. If your cluster is configured to use authentication and/or transport-level encryption, consult the README.md for connection options. It's actually surprisingly easy to configure Kafdrop for a SASL/SSL locked-down cluster.

Navigating the Kafka Web UI

Browsing the Cluster

The Cluster Overview screen is the landing page of the web UI.

You get to see the overall layout of the cluster — the individual brokers that make it up, their addresses and some key broker stats — whether they are a controller and the number of partitions each broker owns. The latter is quite important — as your cluster size and the number of topics (and therefore partitions) grows, you generally want to see an approximately level distribution of partitions across the cluster.

Next is the Topics List, which in most cases is what you're really here for. Any reasonably-sized microservices-based ecosystem might have hundreds, if not thousands of topics. As you'd expect, the list is searchable. The stats displayed alongside each topic are fairly ho-hum. The one worth noting is the under-replicated column. Essentially, it's telling us the number of partition replicas that have fallen behind the primary. Zero is a good figure. Anything else is indicative of either a broker or a network issue that requires immediate attention.

Note: Kafdrop is a discovery exploration tool; it is not a real-time monitoring tool. You should instrument your brokers and raise alerts when things go awry.

Listing Topics

Click on a topic in the list to get to the Topic Overview screen.

The screen is subdivided into four sections.

On the top-left, there is a summary of the topic stats — a handy view, not dissimilar to what you would have seen in the cluster overview.

On the top-right, you can view the custom configuration. In the example above, the topic runs a stock-standard config, so there's nothing to see. Had the configuration been overridden, you'd see a set of custom values like in the example below.

The bottom-left section enumerates over the partitions. The partition indexes are links — clicking through will reveal the first 100 messages in the topic.

There are several interesting parameters displayed in this section:

Parameter	Description
First offset	Also known as the low-water mark, this is the offset of the first retained message, or the high-water mark if the partition is currently void of messages. The first offset is monotonically non-decreasing with respect to any write (and by implication, read) operation that may be performed on the partition, including publishing of messages and the routine purge and compaction operations.
Last offset	The offset that will be assigned to the next message. (This is also known as the high-water mark.) Akin to the first offset, the last offset is monotonically non-decreasing.
Size	The quantity of retained messages within the partition.
Leader Node	The ID of the current leader node.
Replica Nodes	The broker IDs that have been assigned a replica of the partition. This set includes the leader node.
In-sync Replica Nodes	A subset of the replica nodes that only includes in-sync nodes.
Offline Replica Nodes	A subset of the replica nodes that only includes offline brokers. Anything other than an empty set suggests an issue with the broker nodes or the network, and requires urgent attention.
Preferred Leader	A yes/no value, indicating whether the current leader is configured as the preferred leader.
Under-replicated	A yes/no value, indicating that one or more replica nodes has fallen behind the primary. Under normal circumstances there should be no under-replicated partitions.

The consumers section on the bottom-right lists the consumer group names as well as their aggregate lag (the sum of all individual partition lags).

Viewing Consumer Groups

Clicking on the consumer group on the Topic Overview gets you into the Consumer View. This screen provides a comprehensive breakdown of a single consumer group.

The view is sectioned by topic. For each topic, a separate table lists the underlying partitions. Against each partition, we see the committed offset, which we can compare against the first and last offsets to see how our consumer is tracking. Conveniently, Kafdrop displays the computed lag for each partition, which is aggregated at the footer of each topic table.

Note: Some amount of lag is unavoidable. For every message published, there will invariably be a quantum of time between the point of publishing and the point of consumption. In Kafka, this period is usually in the order of tens or hundreds of milliseconds, depending on both the producer and consumer client options, network configuration, broker I/O capabilities, the size of the pagecache and a myriad of other factors. What you need to look out for is growing lag — suggesting that the consumer is either unable to keep up or has stalled altogether. In the latter case, you'll also notice that the lag isn't being depleted even when the producer is idling. This is when you'll need to ALT-TAB away from Kafdrop into your favourite debugger.

Viewing Messages

The Message View screen is the coveted topic viewer that has in all likelihood brought you here. You can get to the message view in one of two ways:

Click the View Messages button in the Topic Overview screen.
Click the individual partition link in the Topic Overview.

It's exactly what you'd expect — a chronologically-ordered list of messages (or records, in Kafka parlance) for a chosen partition.

Each entry conveniently displays the offset, the record key (if one is set), the timestamp of publication, and any headers that may have been appended by the producer.

There's another little trick up Kafdrop's sleeve. If the message happens to be a valid JSON document, the topic viewer can nicely format it. Click on the green arrow on the left of the message to expand it.

In Conclusion

The more you use Kafka, the more you come to discover and appreciate its true potential — not just as a versatile event streaming platform, but as a general-purpose messaging middleware that lets you assemble complex business systems from asynchronous, loosely coupled services.

You'll invariably experience the frustrations that are to be expected from a technology that has only recently entered the mainstream, by comparison with the more mature MQ brokers of yore. What's reassuring is that the Open Source community hasn't stood still, producing an evolving ecosystem, complete with the documentation and tooling necessary for us to get on with our jobs. The least we can do in return is raise a pull request once in a while or maybe answer a StackOverflow question or three.

Was this article useful to you? Take a moment to bookmark it, so others might spot it too. I'd love to hear your feedback, so don't hold back! If you are interested in Kafka or event streaming, or just have any questions, follow me on Twitter.