Kafka in a Nutshell 🌰: Events, Topics, and APIs made simple

Majd Tarbin — Fri, 08 Sep 2023 22:50:16 +0000

Intro

For many years, our default method for data storage has been databases, which have effectively served us for decades.

However, in certain cases, it might be more beneficial to focus on events rather than static data.

Events

So, what exactly is an event? An event is a timestamped indication that a specific action has occurred.

Traditionally, storing events in databases can be a tad inefficient. Instead, we use a structure known as a log — an ordered sequence of events, each accompanied by some state and a description of the event.

This structure is intuitive and scalable.

Topics

Enter Kafka — a robust system designed to manage these logs. In the Kafka ecosystem, these logs are grouped under categories called "topics".

Kafka ensures durability by writing topics across multiple disks or servers, eliminating the risks associated with a single point of failure.

The retention of topics can range from brief periods to years, or even indefinitely, and their sizes can vary from minuscule to massive.

Each event captures a specific business occurrence.

Thinking in Kafka

Kafka prompts developers to prioritize events over static entities.

Gone are the days when the norm was building a monolithic application with a singular, expansive database.

Now, whether you're working with a single service or multiple ones, they can both consume from and produce to a Kafka topic.

The Possibilities

With data residing in these topics, it becomes feasible to establish services that execute real-time analysis by consuming messages directly from topics, as opposed to relying on overnight batch processes.

Kafka Connect

Kafka offers tools, like Kafka Connect, which facilitates the capturing of database changes as events and populates them into a topic — ready for any use case you envision.

Furthermore, Kafka Connect can export events from a topic into a database or any external system.

Kafka Streams

While manipulating topics, events can be grouped based on their key, allowing for subsequent aggregation or counting before directing them to another topic.

Kafka simplifies this process through its Kafka Streams API. It's a Java-based API that encompasses all the foundational elements and infrastructure, ensuring scalability and reliability without the developer needing to build these from scratch.

Outro

Hopefully, this provides an MVP-level foundational understanding of Kafka. Your next step? Dive into the code and experiment firsthand with the Kafka quick start guide.

https://kafka.apache.org/quickstart

The Art of Doing Nothing: The Key to Doing Everything

Majd Tarbin — Wed, 22 Feb 2023 13:45:58 +0000

All around us in nature, nothing is being done, but everything gets done

Consider the lion. This magnificent beast spends a staggering 18-20 hours of its day resting, often basking under the sun's rays. And yet, despite such long hours of inactivity, it remains the undisputed king of its domain. With an entire pride to oversee, how does the lion manage to get everything done while appearing to do so little?

How can we, too, embrace this lion-like efficiency – resting more while still managing our responsibilities?

Our challenge lies in our understanding of productivity. We create extensive to-do lists, categorizing tasks based on priorities, desires, and long-term goals. And even as we start our day full of vigor and enthusiasm, we often end it feeling drained, with unchecked items lingering on our lists.

Contrary to popular belief, motivation and energy aren't magically replenished each morning like a wizard's spell slots in D&D. They are resources you expend and regenerate. Even seemingly restful activities like playing video games or endlessly scrolling through social media can deplete them, akin to how work can.

The Power of Embracing Stillness

To tap into our true potential, we must learn the art of true rest. Sit still, gaze upon the serene greenery outside, and truly disengage.

Welcome boredom. Relish it. Dive deep into its stillness.

For it's in this stillness that we find renewed energy and drive to tackle our tasks. We mustn't mistake leisure activities like memes and gaming as genuine rest. At times, these can be even more taxing than our regular work.

Structure your day strategically. Begin with the most mundane tasks and resist the urge to jump to another before completing the first. This method not only ensures efficiency but also cultivates an authentic interest in each task. Intersperse your schedule with moments of true rest, where you do absolutely nothing, or perhaps just take a gentle stroll. The key is to refrain from any activity that demands even a fraction more thought than your natural, relaxed state.

I've come across numerous studies advocating the benefits of genuine rest, and while I'm no scientist, I am an advocate of balanced productivity. This balance between activity and inactivity is what propelled me to write this piece, something I'd longed to do but lacked the energy for, until I learned to rest like a lion.

I hope you do nothing, and that you complete everything.

Forem: Majd Tarbin