Forem: Lucia Cerchie

Beyond the Basics: Windowing in Apache Kafka

Lucia Cerchie — Fri, 24 Feb 2023 20:02:41 +0000

What Is Windowing?

This post was originally published on the Confluent blog.

If you’re creating an application with Kafka Streams or ksqlDB, and that application involves aggregations, it’s likely that you’ll use windowing.

Why is that?

Well, aggregations of data accumulate over time, and without a limit, these aggregations won’t stop accumulating. Enter windowing, which defines the amount of data that can accumulate.

Note that windowing takes four forms, depending on whether the window is defined by size and period, or whether the window is event-triggered.

For example, say that you’re designing an application using moisture sensors to alert when a houseplant gets too dry. You can perform a filter for plants with low moisture readings in the past hour. If that filter returns anything, that event can trigger an aggregation over a window tumbling on every hour. The result is written to the alerts.

Let’s take a look at each of the types of windowing in finer detail.

Hopping

What is a hopping window?

A hopping window is bound by time and defined by a window size and the size of the time block at which it advances (both measured in seconds).

Consider the red window in the above diagram and think about it advancing through time: if you had a window size of 50 seconds but an advance size of 15 seconds, there would be an overlap of 35 seconds between the windows. In a hopping window, the window size is usually set to a larger amount of time than the advance size. The above diagram shows what an overlap might look like if you took the window as a rectangle and superimposed it over a wavy stream of data—the rectangles that overlap represent the overlapping window.

Note: Tumbling windows never overlap or have gaps. Hopping windows might have gaps or overlaps, or they might not.

A classic example of a hopping window implementation is a dashboard with moving averages—say, average clicks on a certain e-commerce page, like a product details page for an air fryer, for 2-minute windows in the past 24 hours.

Creating a hopping window

As stated above, there are two key pieces of information you need to configure hopping windows: window size and advance size.

When you’re working with Kafka Streams, you need to set both a windowSize and advanceSize. See the more complete example in this Kafka Streams 101 tutorial.

If you’re implementing a hopping window with ksqlDB, then you need to make sure to create a table using the WINDOW HOPPING syntax. You’ll set the two sizes with SIZE and ADVANCE BY. You can view a more complete example in this tutorial.

Tumbling

What is a tumbling window?

Similar to hopping windows, tumbling windows are also time-based. In fact, a tumbling window is a special subtype of hopping window where windowSize and advanceSize are the same. This means an individual event will only ever be present in one tumbling window––no duplicates. As you can see, this diagram differs from the former in that the rectangles do not overlap, representing the non-overlapping windows.

Tumbling windows are useful for reporting where you want events to belong to a single window, like taking the aggregate of credit card swipes in the last 55 seconds.

Creating a tumbling window

Creating a tumbling window in Kafka Streams uses the same process as a hopping window, but you need to make sure that windowSize and advanceSize are the same.

In ksqlDB, you use the WINDOW TUMBLING syntax on a TABLE and set the SIZE. The tutorials section on the Confluent website houses a full example: How to create tumbling windows.

Session

What is a session window?

A session window is triggered by events. The mechanism involves something called an “inactivity gap.” This gap is the amount of time within which the window will grow unless interrupted by a new event. When a new event flows in as input, the inactivity window resets and the window continues to grow. If your inactivity window is too short or you have lots of events, this can result in a very long window!

In the above diagram, the inactivity gaps are represented by blue rectangles, separated by an event which is represented by a star. The rectangle representing the session window spans the inactivity gaps.

Use cases involve tracking user activity if you wanted to know how many events were generated over a period of time (e.g., how long exactly was the user’s FlixMovie binge?).

Creating a session window

Creating a session window in Kafka Streams involves setting an inactivityGap. You can see a more complete description in this tutorial: Windowing.

The other option is to use ksqlDB, employing a table with a WINDOW SESSION syntax. You can view a tutorial on this here: Create session windows.

Sliding

What is a sliding window?

A sliding window is also time-based, so the size of the window is defined in seconds. In order for a sliding window to be triggered, some user events must happen within a defined window of time. For example, you might use this type of window if you wanted to calculate the data between “add-to-cart” and “purchase” events from users that were made within the defined amount of time of three hours.

In the diagram above, the defined amount of time is represented by a large light-blue rectangle. The user events are represented by two gold stars, and the purple rectangle between them represents the data window.

Creating a sliding window

When you create a sliding window in the Streams API, you need to set timeDifference. You can find a more complete example in this windowing tutorial on Confluent Developer.

Support for explicitly setting sliding windows is not yet available in ksqlDB.

A note on late events

What if your events arrive late? It could skew your analysis; therefore, you can introduce “grace periods” in tumbling, hopping, and sliding windows. It’s basically a set amount of time in which events will be caught by the window, even though the time stamps of those events are greater than the window’s end. Note that events that occur after the grace period won’t be included—those are late for good and will be missed. Session windows don’t have grace periods because they are based on user behavior only, rather than time.

Since KIP-633, there is no default grace period for Kafka Streams. However, in ksqlDB, the default is 24 hours.

Where to go from here

If you found this introduction to windowing with Apache Kafka® useful, you may also be interested in this list of resources for a deeper dive:

A Kafka Stream video from the Kafka Streams 101 course: Windowing

Kafka Streams Application tutorial

ksqlDB 101 course

Streams tutorials list

See the Event Aggregator pattern

A Framework For Knowing What You Don't Know

Lucia Cerchie — Fri, 16 Dec 2022 21:07:50 +0000

This blog post was originally posted on my website.

Over my career I've had to learn new things many, many times. I've been an elementary school teacher, a digital marketer, a software engineer, and now I'm a full-time developer advocate. Over and over again, I've had to learn completely new concepts. It's great; it keeps me humble, it keeps me happy. There's a lot of dopamine involved in learning. But one of the most difficult parts of learning is knowing what you don't know. And knowing what you don't know is important, because it keeps you from making the types of mistakes that emanate from errors in judgment.

I've learned a couple of things about defining the limits of my knowledge about a concept. I'm sharing my framework here in case it's helpful to anyone else in tech.

Stage 1: Nothing

The first stage of getting to know a new concept is the easiest to define. I'm starting from nothing!

This is great. I know exactly what I don't know. Let's say I'm completely brand-spanking new to the concept of an API. Here is a diagram of what I don't know about APIs:

It doesn't get more accurate than that. I'm not kidding. "What you don't know" will surprise you again, and again, and again. The framework I'm suggesting here will not save you from being wrong in front of people. But I think it will give you some general guidance on how to approach learning new things.

Stage 2: One Instance of a Concept

So let's consider universal concepts. Definitions that we all agree on. For example, an API according to Wikipedia is

"An application programming interface (API) is a way for two or more computer programs to communicate with each other."

Maybe that's not the most complete definition, but I think most techies would agree on what's there. Now, you can memorize this definition, but you won't have experiential knowledge of an API until you've met one out in the wild.

Let's say I'm at that stage. I'm getting used to querying a REST API, and I'm learning to build one myself with Python or Node Express or something. This definition applies to the REST API I'm building, so I'm gaining experience with one type of API. One particular instance of this universal concept. Here is another map of my brain:

So, I'm learning something about REST APIs. And that's it for now. The arrow is pointing to a boundary that I'm not aware of:

I don't know what's common to REST APIs and all APIs, and what's different.

So, I've seen one particular instance, but I've not seen another instance of an API. And this is a problem because there are different kinds of APIs underneath the universal concept of "API". Like there are different kinds, or species, of cats underneath the "cat" genus. And tigers are very different from lions.

Stage 3: Two instances, of different kinds, of a concept.

Say I start implementing not a REST API, but a GraphQL API. It's a different kind of API from a REST API, like lions are a different kind of cat from tigers. My worldview on APIs begins to break down. It's destroyed by differences between REST APIs and GraphQL APIs, like

GraphQL is an application layer, while REST is a style of API. Not every API is in the REST style!
GraphQL requests are JSONesque, while REST requests are often parameterized in URLs. Not every API uses parameters the same way! Or has the same request format.
You define GraphQL with a schema, rather than a list of endpoints. Not every API is defined the same way!

I know that diagram just got complicated, I'm sorry. But this is the most important stage to take notes at. Why? Because when you realize what assumptions you've made, you realize what questions to ask the next time you encounter a new kind of API.

These are the assumptions I've made:

All APIs are in the same style.
All API requests are formatted alike.
You define APIs in the same way.

Now that those assumptions have been broken by my experience, I know what questions to ask next time I'm learning a new kind of API.

Stage 4: The third instance of a different kind.

Ok, say I'm learning what a tRPC API is in order to implement on at work or something. Based on the kinds of assumptions I made last time, what might I ask now?

How does the style of tRPC compare to REST and GraphQL, the ones I'm familiar with?
How are tRPC requests formatted?
How do I map or define a tRPC API?

These questions will help me understand what a tRPC API is much faster than I understood what a REST or GraphQL API is.

Furthermore, I'll be understanding the general concept of an API even better, because I'll be understanding what is in common between all these kinds of APIs. I'll also be understanding GraphQL and REST APIs better at this stage, because I can then make lists of their limitations.

Generalizing this process

I think every time you learn something new, you have to go through these 4 stages and there's not really a way around that.

But if you do it consciously, it speeds up the process of learning.

So when you're beginning, acknowledging "I know nothing about this. I should ask someone who knows something about this what is the best instance to build first," can save you some pain.

Then again, once you've built your first instance, acknowledging "I might be making assumptions here that don't apply to other instances. I should ask someone who has built other kinds of instances what the differences are," will help you make decisions about whether to learn a new paradigm when you're building your next instance.

What Are Apache KafkaⓇ Consumer Group IDs?

Lucia Cerchie — Tue, 13 Dec 2022 15:32:16 +0000

This post was originally published on the Confluent blog.

Consumer Group IDs are a vital part of consumer configuration in Apache Kafka®. Setting the consumer Group ID determines what group a consumer belongs to, which has some major consequences. There are three areas in which Group IDs are particularly pertinent:

Detecting new data
Work sharing
Fault tolerance

Let’s dive in.

What is a Kafka consumer?

Kafka consumers read/consume data from Kafka producers, do the work of reading event streams. They read events, or messages, from logs called topics. Topics are further split into partitions, which are append-only logs that store the messages. This enables each topic to be hosted and replicated across a number of brokers.

As you can see in the diagram, a given consumer in a consumer group can read from multiple partitions, including multiple partitions housed in the same topic.

Using consumer Group IDs to detect new data

Group IDs are associated through the broker with bits of information called offsets, which specify the location of a given event within a partition, and as such, represent progress through the topic. Offsets in consumer groups serve the same purpose as how bookmarks or sticky tabs function in books. You can learn more about offsets in our FAQ.

Checking for new data

You can use a particular Group ID’s offset to check whether there’s been new data written to the partition. If there’s an event with a larger offset, that means there’s new data to read. If you want to know how to read the offset, here’s a command using the kafka-consumer-groups utility that will read your offsets:

kafka-consumer-groups --bootstrap-server localhost:9092 --describe --group group1 --offsets

Note that you need to provide a valid Group ID to --group if you’re trying out this command. The output will resemble the following:

`GROUP   TOPIC  PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG     OWNER
Groupname topicname     0        2               2         1       ownername

Or, if you want to learn more about how to do this with the Confluent CLI for a topic hosted in Confluent Cloud, you can check out this tutorial on reading from a specific offset and partition.

There’s more on the kafka-consumer-groups utility in our documentation, and you can always run kafka-consumer-groups—help for a full list of the options.

Consumer Group IDs in work sharing

The Group ID determines which consumers belong to which group. You can assign Group IDs via configuration when you create the consumer client. If there are four consumers with the same Group ID assigned to the same topic, they will all share the work of reading from the same topic.

If there are eight partitions, each of those four consumers will be assigned two partitions. What if there are nine partitions? That means the leftover partition will be assigned to the first consumer in the group so that one consumer reads from three partitions and the rest of the consumers read from two partitions. It’s the broker’s job to continually ensure that partitions are evenly distributed among the connected consumers.

Note: At the top, you'll see that although there are four consumers, three are idle. That's because only one consumer in the same group can read from a single partition.

This whole process is predicated on the presence of a Group ID to unify the consumers. It’s important to remember this while you’re setting up your consumers.

If you’re connecting microservices, you want to make sure that each service has its own consumer group (and hence its own Group ID). Why is that? Let’s walk through an example.

Let’s say there’s a topic “payments,” and both the “orders” microservice and the “refunds” microservice will need to read from that topic. You wouldn’t want them to share the same offsets, because if they did, the progress through the “payments” topic would be shared by “orders” and “refunds,” which would mean potential missed orders or refunds.

However, if you had a group of consumers handling “orders” by reading from partitions in the “payments” topic, then the current offset for each consumer in the group, stored in the broker, is vital to ensure continuous progress in case a consumer in the group crashes. At the same time, if consumers from another, separate group, like “refunds” are reading from the “payments” topic, they can continue their progress unaffected even if the consumers in the “orders” group are rebalancing.

The role of consumer Group IDs in fault tolerance

As the last example revealed, Group IDs also play a vital role in fault tolerance.

What happens when a consumer crashes?

Each consumer group’s broker sends “heartbeat requests” to the consumers at a set interval. If a consumer does not respond in time, a rebalance is triggered.

How does a Group ID play into rebalancing?

Well, in either case, the broker’s record of the associated offset determines where the consumer will begin reading after a rejoin. As long as the Group ID remains the same, it can pick up exactly where it left off, without any risk of data loss.

If you’re interested in learning more about rebalancing, we recommend the blog post Incremental Cooperative Rebalancing in Apache Kafka: Why Stop the World When You Can Change It?. You can also consult our FAQ.

Where to go from here

In summary, when you set a consumer Group ID in the process of creating a consumer client, that Group ID assigns the consumer to its group, which has ramifications for work sharing, detecting new data, and data recovery.

To learn more about this and other topics, check out these recommended resources:

Confluent Developer: Learn Apache Kafka through Confluent Developer tutorials, documentation, courses, blog posts, and examples.
Confluent Community: If you have a question about Apache Kafka or you’d like to meet other Kafka developers, head over to Confluent Community and introduce yourself on our Community Slack or Forum.
Streaming Audio Podcast: Listen to the Streaming Audio Podcast to hear lively conversations with Confluent users about the ins and outs of Apache Kafka. The episode Optimizing Kafka’s Internals covers consumer group internals.

Three Things I've Learned About Writing Abstracts

Lucia Cerchie — Wed, 16 Nov 2022 16:03:43 +0000

Recently I've joined the developer advocate team at Confluent, which is full of highly experienced speakers who have mentored me as I craft abstracts.

I'll be honest; when I began writing abstracts I thought, "How hard can this be? I've written plenty of blog posts, technical articles, and the occasional haiku. Abstracts will come naturally."

Reader, they did not! The art of writing and fine-tuning abstracts is challenging, but learn-able. Luckily, I've gained a lot of knowledge from my teammates and now I feel a lot more comfortable with writing abstracts. I wrote this post to hand on a few of the things I've learned.

1. Connect with your audience in the first sentence.

Let's start with this abstract that I've written up for the purpose of this blog post:

"Come to my talk about choosing React frameworks. We'll learn how and why to choose the framework that suits your web development needs. You'll learn criteria for choosing a web development framework and how to apply them. By the end of my talk, you'll know more about the React ecosystem and have the tools to get the job done."

The first sentence, "Come to my talk about choosing React frameworks," is a nice invitation but it doesn't really hook the reader. In order to connect with the audience, it's a good idea to start by mentioning their pain point. In this example, a good first few sentences might be more like the following:

"The number of React frameworks in recent years has reached an overwhelming height. Social media debates run fierce. There's only one consensus: choosing the right framework for the job is of paramount importance. But how, exactly, do we pick a framework?"

2. Position your pronouns thoughtfully.

Take a look at the abstract once more.

"The number of React frameworks in recent years has reached an overwhelming height. Social media debates run fierce. There's only one consensus: choosing the right framework for the job is of paramount importance. But how, exactly, do we pick a framework? We'll learn how and why to choose the framework that suits your web development needs. You'll learn criteria for choosing a web development framework and how to apply them. By the end of my talk, you'll know more about the React ecosystem and have the tools to get the job done."

There's "we", "you", and "my" here. Consistency is key in all writing, but for talks, you might choose "we" over other options to reflect a sense of camaraderie with the audience.

3. Let your solution for the audience's pain point be clear.

Currently, the solution that the speaker offers is vague:

"We'll learn how and why to choose the framework that suits your web development needs. We'll learn criteria for choosing a web development framework and how to apply them. By the end of the talk, we'll know more about the React ecosystem and have the tools to get the job done."

In fact, all you can really tell is that the speaker is offering some kind of solution. There are no hints as to what it might be.

Here's a better way to express the speaker's intention:

"We'll distill the criteria for selecting a React framework into three crucial questions. Then, we'll walk through a few use cases together to garner some experience making these decisions. What does the decision making process look like for building static portfolio sites, large e-commerce sites, and mobile game apps? By the end, we'll feel ready to critically appraise React frameworks, familiar or unfamiliar, for our own projects."

This gives some detail ("three crucial questions", and the use cases) without giving everything away. It also communicates the value to the audience: a confidence in their choice of framework.

Now the whole abstract reads:

"The number of React frameworks in recent years has reached an overwhelming height. Social media debates run fierce. There's only one consensus: choosing the right framework for the job is of paramount importance. But how, exactly, do we pick a framework?

We'll distill the criteria for selecting a React framework into three fundamental questions. Then, we'll walk through a few real-life use cases together to garner some experience making these decisions. What does the decision making process look like for building static portfolio sites, large e-commerce sites, and mobile game apps?

By the end, we'll feel ready to critically appraise React frameworks, familiar or unfamiliar, for our own projects."

I think this version sounds a lot more interesting and clear, don't you?

What does 'batching' mean when we're talking about Apache KafkaⓇ?

Lucia Cerchie — Fri, 28 Oct 2022 20:14:27 +0000

Today I learned that when you hear the word 'batch' in the context of Apache Kafka, it can mean one of two things:

A reference to batch-only data processing systems. Batch-only systems process data in a bounded way. That means that there's a start time and an end-time. Whether this batching is done in large or micro-batches, it is processed all at once. That's in contrast to the continuous data streaming that Apache Kafka enables, in which data is processed in event-sized pieces.
Within the data streaming context, there's something called producer batching. It's a bit of a misnomer because it's not really related to the batch-only data processing systems. A Kafka producer, the client that publishes records to the Kafka cluster, compresses messages via a process called batching to increase throughput. This batching is part of the process handling data at once and in event-sized pieces, so it doesn't mean the same thing as batch-only data processing.

In conclusion, 'batching' means, in a very general way, 'grouping stuff together'. But 'producer batching' and 'batch-only data processing systems' do not share the term in any significant sense, because they are referring to the completely different functions I described above.

Installing node-rdkafka on M1 for use with SASL

Lucia Cerchie — Fri, 14 Oct 2022 20:25:29 +0000

If you're using Kafka in a Node.js app, it's likely that you'll need node-rdkafka. This is a library that wraps the librdkafka library and makes it available in Node.js. According to the project's README, "All the complexity of balancing writes across partitions and managing (possibly ever-changing) brokers should be encapsulated in the library."

Installation steps may vary depending on the machine you're using, and whether you need SASL/SSL (Simple Authentication and Security Layer) support. You might need this, for example, if you're connecting to Confluent Cloud, as it is a required protocol to protect resources (see docs).

Here's a short, step by step guide on installing the node-rdkafka client specifically on M1 (the first chip released by Apple specifically for Mac, article here).

Currently, as of September 30 '22, you have to do it manually. You can find the reason why in the confluent-kafka Python client README.md:

NOTE: The pre-built Linux wheels do NOT contain SASL Kerberos/GSSAPI support. If you need SASL Kerberos/GSSAPI support you must install librdkafka and its dependencies using the repositories below and then build confluent-kafka using the instructions in the "Install from source" section below.

Here are the steps you'll need to take for SSL/SASL support (run in your home directory):

git clone https://github.com/edenhill/librdkafka.git
cd ./librdkafka
./configure --install-deps --source-deps-only
make
sudo make install

Then,

Use homebrew to install openssl.
Navigate out of ./librdkafka and into your Node.js project directory.

Export these variables with these commands, to make sure that the path to openssl is correct:

export CPPFLAGS=-I/opt/homebrew/opt/openssl@3/include
export LDFLAGS=-L/opt/homebrew/opt/openssl@3/lib

install openssl with npm install openssl
Next, install node-rdkafka -- now you've got SSL support for your Node.js/Kafka project!

Hopefully this helps! If you're still running into issues, you can post a request for help in Confluent Community.

Introduction to Key Apache KafkaⓇ Concepts

Lucia Cerchie — Tue, 11 Oct 2022 20:37:58 +0000

Introduction

Last week I started a brand new role: Developer Advocate at Confluent. So far I’ve been gladdened by stepping fully into an area of tech that satisfies my teaching heart. I’ve also felt supported by my new teammates. I’m new to the Kafka scene, with a background in JavaScript and GraphQL. In order to help onboard me smoothly, my teammates are meeting with me frequently, checking in on my progress (they even reviewed this article 😉), and are helping me select material from Confluent’s abundance of resources to learn Kafka.

My first step in my learning journey was this course on Kafka 101. I took notes throughout the course, which broke Kafka down to its core concepts. I'll share the synthesis of those notes below in hopes that if you're coming from a similar background, you'll find it useful!

What is Kafka?

According to its website, Kafka is a "distributed event streaming platform".

Well, what does that mean to me, a mostly-JavaScript developer with a background in web development and GraphQL? My usual approach to learning entirely new concepts is to take a back-to-square-one approach and understand some fundamental vocabulary first. Much like you might estimate the result of a math problem before tackling its execution (e.g., estimating that the sum of 219 + 38 will not be above 300), I like to situate myself with respect to the context of a new skill before executing it. The following paragraphs set the context of Kafka by defining and illustrating key concepts.

What is an Event?

So, a "distributed event streaming platform". The key piece of terminology here is 'event'. Before I took the course, I understood an event as a 'thing that happens', which still holds up within the context of Kafka. In Kafka, events are things that happen, and the representation of these things are recorded by machines.

Let’s take, for example, the event of a credit card transaction. The event is composed of a notification and the event’s state. To implement this event in Kafka, you’d need certain pieces of information, including a key, value, timestamp, and optional metadata headers. So you’d have something like:

Key: “Credit Card Payment”
Value: “Paid $12.50 for a ham sandwich”
Timestamp: “05/01/2022”

Kafka serializes these pairs into a structured format like JSON, Avro, or Protobuf. By default, event sizes in Kafka are 1MB.

What is a Topic?

At a high level, topics are ways of organizing events. They are programmatic logs of events, as opposed to application logs. You can see this article by Jay Kreps to learn more about the difference. Logs have 3 noteworthy aspects:

They are append-only. New messages are only applied to the end of a log. They can't be inserted—unlike with a different structure (like a graph).
You can't change events in a log. This was important for me to wrap my head around since I was coming from a GraphQL background, and you can certainly mutate data objects using GraphQL.
Access to logs is sequential from a given offset, so they're read by picking an arbitrary offset and scanning.

What is a Cluster?

Ok, cool, but where are these events and logs kept so I can access them? Are they hosted somewhere? Kafka's storage layer is a cluster of things called brokers. Brokers are servers that handle things like write and read requests to partitions, as well as replication. Which brings us to:

What is a Partition?

Logs are hosted and replicated across a cluster of machines, rather than clogging up one machine 🪠.
Partitions make this possible. They take a single topic log and break it up into multiple logs.

Now, how are messages written to partitions? If the messages are keyless, they'll be assigned one after the other until the last partition is reached, in which case the process starts from the top. Messages which have keys are assigned to partitions holding messages with the same keys. Kafka does this by hashing the key and using a mod function. If you want to take a deep dive into partitions, you can read this blog post by Ricardo Ferreira that examines them from the perspective of both developers and data ops.

What is a Producer?

The producer sends the key value pairs to the clusters, managing the interaction between brokers over a network. Incidentally, producers are also responsible for message assignment to partitions. You can view a producer instance created with the Confluent client in Python in the Confluent Developer website.

   # Parse the configuration.
    # See https://github.com/edenhill/librdkafka/blob/master/CONFIGURATION.md
    config_parser = ConfigParser()
    config_parser.read_file(args.config_file)
    config = dict(config_parser['default'])

    # Create Producer instance
    producer = Producer(config)

What is a Consumer?

A Kafka consumer reads the data from the stream. Many independent consumers can read from one topic. When a message is read it is not deleted.

Consumers handle scaling automatically. You can see how to instantiate a consumer instance in Node.JS on the Confluent Developer "Getting Started" tutorial.

function createConsumer(config, onData) {
  const consumer = new Kafka.KafkaConsumer(
      createConfigMap(config),
      {'auto.offset.reset': 'earliest'});

  return new Promise((resolve, reject) => {
    consumer
     .on('ready', () => resolve(consumer))
     .on('data', onData);

    consumer.connect();
  });
};

Conclusion

Let's go back to the original definition of Kafka, a "distributed event streaming platform". We now know that an event is a representation of a thing that happens expressed in a key value pair. These events are organized by topics, logs which are partitioned by producers and read by consumers. The platform is a cluster of brokers that enable replication and fault tolerance. Together, this system is Kafka.
The system arises from the viewpoint that data is not static -- that it's everywhere, flowing and interconnected. That doesn't mean you can't integrate streaming data with classic data sources like databases and REST APIs, or that you can't filter, transform, and manage it. But that's a topic for another day ;)

How To GraphQL - Tutorials to Get You Started

Lucia Cerchie — Tue, 21 Jun 2022 19:22:25 +0000

StepZen gets your GraphQL API deployed and running in minutes, with up to 5000 requests per second with low latency.

StepZen’s YouTube channel presents a wealth of short tutorials for learning how to use StepZen to build and deploy GraphQL. This helps you build projects that connect data from databases, REST APIs, external GraphQL APIs, and even SOAP APIs if that’s how you roll.

Here are some of my favorites.

Query SaaS APIs with GraphQL in a Next.js App

In this video, Roy teaches us how to query both the DEV.to REST API and the GitHub GraphQL API using StepZen GraphQL Studio. You can see how useful this can be for your development flow: Data for both APIs can be mocked before being loaded into a Next.js React application. Then, loading environment variables in your application introduce real data.

Two Ways to Query Airtable with GraphQL

Airtable is a popular platform for building collaborative apps. In this video, you’ll learn more about real-life GraphQL by watching Roy query Airtable in two different ways. You can use the prebuilt Airtable GraphQL API provided in the Studio or convert the REST API to GraphQL using ‘stepzen import curl’.

Find a GraphQL API for Airtable in StepZen GraphQL Studio here:

https://graphql.stepzen.com/airtable

Automatically Generating TypeScript Types

Don’t fear TypeScript! Roy shows us how to generate TypeScript types for our GraphQL API endpoint. Since both TypeScript and GraphQL have a type system, you can use your GraphQL schema to generate TypeScript types. This video explains how to use GraphQL Code Generator to generate those types.

Combing Data From Node.js Microservices

In this video, we learn how to combine data from different Node.js microservices with GraphQL. You can use GraphQL as a data layer for these microservices and combine the data from all these services into a single, unified API to simplify your development process. Roy introduces us to building this data layer declaratively using StepZen’s custom directives.
.

Load Testing GraphQL APIs With k6

Watch Roy teach you how to load test a GraphQL API built with StepZen. He uses the open source library k6 on a StepZen API that combines data from different sources. You can follow along with this video to test for yourself and see how a GraphQL API built and running on StepZen performs. You can also consult Roy’s blog post on the same topic.

Where to Go From Here

We've created a full How-to Playlist. Subscribe to our YouTube channel and check back often - we add new tutorials weekly. And if there’s a how-to you’d like to see, let us know.

If you have questions, feel free to slide into our Discord. We'd love to see you around! In fact, if you’re working on a GraphQL project and you’d like to join us for a community live stream or have us host your project as a YouTube video, just let us know. We’d love to host you and hear what you’re building.

To learn more about creating GraphQL APIs quickly with StepZen, see:

Consuming the Deepgram API: The GraphQL Edit

Lucia Cerchie — Fri, 10 Jun 2022 19:11:27 +0000

In this project, we’ll explore how to use Deepgram, StepZen, and Next.js to convert the Deepgram REST API into your own GraphQL API, translate audio to English and French text, and display the result in a user interface.

Deepgram’s REST API - A speech-to-text API built on a 100% deep learning platform. It offers up to 90% translation accuracy with model training with no human validation necessary. I chose to write about it because the developer experience is excellent; should you want to extend the capabilities of this API further, you’ll find exactly what you need in Deepgram’s docs and tutorials.
StepZen’s REST2GraphQL tool - A user interface designed to convert your REST APIs to GraphQL, no code necessary. StepZen introspects the endpoint, generates a schema, and deploys your GraphQL endpoint once you hit ‘Generate’. These endpoints are public and are available for you to use and share for 30 days. To take the schema you’ve created with REST2GraphQL and create a private, scalable endpoint for it, sign up for a StepZen account.
Next.js - A React framework providing features like Server Side Rendering. We’ll use it in this tutorial to fetch the data from our GraphQL endpoint. That way, you’ll learn how to safely send an API key on a server-side request.

Converting the Deepgram REST API to GraphQL

The first thing we’ll learn is how to use REST2GraphQL to convert Deepgram to a GraphQL API.

Access the tool at rest2graphql.io

Then, select ‘POST’ and enter the Deepgram ‘listen’ endpoint we’ll be using:

Next, you’ll add some query parameters.

Deepgram uses these parameters to determine the language to translate, the model to use, and whether to punctuate the text. They are sent to StepZen on ‘Generate’, and StepZen creates a GraphQL schema with query variables matching these parameters.

You’ll also need to add your Deepgram API key under the Headers tab to get access to your Deepgram endpoint. You can find the steps to create a Deepgram API key in the Deepgram docs.

Now, click Generate, and your endpoint is deployed! Query it in the browser with GraphiQL, or customize the values of your request’s variables with the ‘Query Arguments’ widget.

Consuming the GraphQL API in a Next.js App

Next, let’s look at how to consume your GraphQL endpoint in a Next.js app. I’ll show snippets to illustrate my steps, so here’s the Github repository if you’d like to clone the entire code.

I started by creating a Next.js app. Then, I created a getServerSideProps function to feed the data into the Home component, inside pages/index.js .

export async function getServerSideProps(context) {

}

export default function Home(props) {
    return (
        <div className={styles.container}>
            <Head>
                <title>Create Next App</title>
                <meta
                    name="description"

ETC...

I wrote my fetch call inside the getServerSideProps function. called the endpoint generated by REST2GraphQL, https://public6ddcc5208696d858.stepzen.net/api/deepgram/__graphql, sent the headers required by the Deepgram API, and sent my GraphQL query as the body (you can check out the generated JavaScript boilerplate to see the query that I copied).

I protected the apikey by saving it in my env. file. For information about how to use the fetch API with GraphQL, view StepZen’s documentation.

const apikey = process.env.DEEPGRAM_APIKEY

const res_in_en = await  fetch('https://public6ddcc5208696d858.stepzen.net/api/deepgram/__graphql', {
  method: 'POST',
  headers: {
    'Content-Type': 'application/json',
    Authorization: `Token ${apikey}`,
  },
  body: JSON.stringify({
    query: `
    query deepgramQuery($authorization: String!, $language: String, $model: String, $punctuate: String, $url: String) {
        deepgramQuery(
          authorization: $authorization
          language: $language
          model: $model
          punctuate: $punctuate
          url: $url
        ) {
          results {
            channels {
              alternatives {
                confidence
                transcript
                words {
                  confidence
                  end
                  punctuated_word
                  start
                  word
                }
              }
            }
          }
        }
      }
      `,
    variables: {
        url: 'https://static.deepgram.com/examples/Bueller-Life-moves-pretty-fast.wav',
        language: 'en-US',
        model: 'general',
        punctuate: 'true',
        authorization: `Token ${process.env.DEEPGRAM_APIKEY}`
    },
  }),
})

After making the call in the function, we transform the result into JSON and return it as props:

const en_data = await res_in_en.json()

    if (!en_data) {
        return {
            notFound: true,
        }
    }

    return {
        props: { en_data }, // will be passed to the page component as props
    }

Next, I surfaced the data text on the homepage by using the props passed from the getServerSideProps function to the Home component (something Next will do automatically here).

<h1>DeepGram Samples</h1>
     <h2>English Sample</h2>
         <p>
           {
     props.en_data.data.deepgramQuery.results.channels[0].alternatives[0].transcript
            }
         </p>

After that, I can see it rendered in my localhost endpoint!

French Sample

So now I’d created an English sample… how about another language? Deepgram has support for 18 languages! Let’s try French. The first thing is to repeat the 'Converting the Deepgram REST API to GraphQL' process outlined above, but with a ‘fr’ value for the language parameter rather than ‘en-US’.

I also used a French sample hosted at https://www.lightbulblanguages.co.uk/resources/audio/quelageastu.mp3. I then created a similar query and passed the result of it to props as well. You can see my final version in the Github repository, or give it a go on your own!

Where to Go From Here

In this post, you learned how to convert Deepgram’s API to GraphQL with StepZen, then consume it in a Next.js app. To learn more about these technologies:

We hope you enjoy building your own project with StepZen; please know we’d be happy to help you learn more! Jump into our Discord server to ask a question.

3 New GraphQL Developer Tools to Increase Coding Productivity

Lucia Cerchie — Thu, 02 Jun 2022 17:44:57 +0000

In our mission to make building GraphQL APIs easier for every developer, StepZen developed tools to get you started quickly and with either no- or low- code. In this post, we’ll explore some of the tools we’ve built. We’ll take a look at REST2GraphQL, an interface that enables you to convert your REST API into a GraphQL endpoint, as well as JSON2SDL, which enables you to transform JSON to Schema Definition Language. You’ll also learn how to use the StepZen cli to import a GraphQL API in a short tutorial. Let’s get started!

REST2GraphQL

With REST2GraphQL, you can enter a REST endpoint, click submit, and receive a new endpoint to access the GraphQL version of the API.

Converting REST to GraphQL is useful for many reasons. Among them is the introduction of REST to your GraphQL layer. I encourage you to give REST2GraphQL a try as a way of testing out StepZen as your GraphQL mechanism.

JSON2SDL

JSON2SDL.com is a great tool to convert JSON results into a GraphQL schema. You can use it when you only need to convert part of a REST API’s functionality to GraphQL. For example, a GET call to Github’s meta endpoint, https://api.github.com/ returns this data:

{
  "current_user_url": "https://api.github.com/user",
  "current_user_authorizations_html_url": "https://github.com/settings/connections/applications{/client_id}",
  "authorizations_url": "https://api.github.com/authorizations",
  "code_search_url": "https://api.github.com/search/code?q={query}{&page,per_page,sort,order}",
  "commit_search_url": "https://api.github.com/search/commits?q={query}{&page,per_page,sort,order}",
  "emails_url": "https://api.github.com/user/emails",
  "emojis_url": "https://api.github.com/emojis",
  "events_url": "https://api.github.com/events",
  "feeds_url": "https://api.github.com/feeds",
  "followers_url": "https://api.github.com/user/followers",
  "following_url": "https://api.github.com/user/following{/target}",
  "gists_url": "https://api.github.com/gists{/gist_id}",
  "hub_url": "https://api.github.com/hub",
  "issue_search_url": "https://api.github.com/search/issues?q={query}{&page,per_page,sort,order}",
  "issues_url": "https://api.github.com/issues",
  "keys_url": "https://api.github.com/user/keys",
  "label_search_url": "https://api.github.com/search/labels?q={query}&repository_id={repository_id}{&page,per_page}",
  "notifications_url": "https://api.github.com/notifications",
  "organization_url": "https://api.github.com/orgs/{org}",
  "organization_repositories_url": "https://api.github.com/orgs/{org}/repos{?type,page,per_page,sort}",
  "organization_teams_url": "https://api.github.com/orgs/{org}/teams",
  "public_gists_url": "https://api.github.com/gists/public",
  "rate_limit_url": "https://api.github.com/rate_limit",
  "repository_url": "https://api.github.com/repos/{owner}/{repo}",
  "repository_search_url": "https://api.github.com/search/repositories?q={query}{&page,per_page,sort,order}",
  "current_user_repositories_url": "https://api.github.com/user/repos{?type,page,per_page,sort}",
  "starred_url": "https://api.github.com/user/starred{/owner}{/repo}",
  "starred_gists_url": "https://api.github.com/gists/starred",
  "topic_search_url": "https://api.github.com/search/topics?q={query}{&page,per_page}",
  "user_url": "https://api.github.com/users/{user}",
  "user_organizations_url": "https://api.github.com/user/orgs",
  "user_repositories_url": "https://api.github.com/users/{user}/repos{?type,page,per_page,sort}",
  "user_search_url": "https://api.github.com/search/users?q={query}{&page,per_page,sort,order}"
}

Oof! Let’s say you only need the data coming back that directly relates to users. That’s:

{
  "current_user_url": "https://api.github.com/user",
  "current_user_authorizations_html_url": "https://github.com/settings/connections/applications{/client_id}",
  "user_url": "https://api.github.com/users/{user}",
  "user_organizations_url": "https://api.github.com/user/orgs",
  "user_repositories_url": "https://api.github.com/users/{user}/repos{?type,page,per_page,sort}",
  "user_search_url": "https://api.github.com/search/users?q={query}{&page,per_page,sort,order}"
}

Paste that into JSON2SDL.com, and boom! you’ve got a nicely pared-down GraphQL schema.

type Root {
  current_user_authorizations_html_url: String
  current_user_url: String
  user_organizations_url: String
  user_repositories_url: String
  user_search_url: String
  user_url: String
}

`stepzen import`

StepZen’s cli tool enables you to import all kinds of backends. Give it a try with a GraphQL API!

Run

npm install -g stepzen

stepzen login --public

mkdir gql && cd gql

stepzen import graphql

And enter

https://graphqldd.stepzen.net/api/dd1cf47f51ac830fe21dc00ec80cee65/__graphql

when you’re prompted to enter your endpoint. You can hit ‘enter’ for the other prompts since you won’t need authentication or headers to walk through this example.

Et voila! Behold, your schema is downloaded.

Now, to deploy your API, run

stepzen start

You can now query it at the localhost address provided, or consume it at the URL you’ve been provided as well.

Conclusion

We’ve covered three ways to increase your productivity when building a GraphQL API, whether you want to import an entire endpoint or only access some of its data, whether you’d like to convert a data source to GraphQL using either a graphic user interface or a command-line interface. We hope you find these ways of speeding up your development process useful!

Now, sometimes developers trade speed of development for maintainability in the long run. We’d like to note that with StepZen, you don’t have to make that tradeoff. If you sign up for an account, you’ll be able to create a fully protected backend, whether with JWT or API keys, or both. We also have a >99.99% uptime. We are happy to take care of these details so you don’t have to sacrifice your strategic solidity for tactical concerns.

If you’d like to learn more about why you’d choose StepZen, visit our Why StepZen page.

You can also

Check out StepZen docs
pre-built SaaS APIs in the GraphQL Studio.
We're always keen to see what you're building and answer any questions on Discord!

Convert REST to GraphQL: A New Low-Code Approach

Lucia Cerchie — Tue, 31 May 2022 20:37:11 +0000

Today's API and app economy is built on REST. After a couple of decades of investment in REST, it's no surprise that most of the APIs that developers consume are REST. GraphQL is the relative newcomer. Its strengths are the ability to query everything you need and only what you need, and an improved developer experience supported by capabilities like self-documentation and query hinting. This landscape leaves savvy developers looking for ways to leverage GraphQL into a world where a lot of the data they need to build today's compelling experiences resides at REST API backends. So how to transform REST endpoints to GraphQL?

Transforming REST into GraphQL doesn't have to be an arduous task. You can already do so in a few lines of code with StepZen. And we've just made it even easier with a new REST2GraphQL conversion tool!

The REST2GraphQL tool requires no CLI download and no StepZen account. It lets you create a fully functional GraphQL API from your REST endpoint. Your GraphQL API is deployed right away and available to call from your app or command line. The GraphQL API is public and is running and available for 30 days at no cost to you. Share it, query it, build against it. At any time, take what you've built, make it a permanently deployed endpoint and easily customize it, by signing up for a free StepZen account.

How to transform REST to GraphQL with one click

Paste your REST endpoint URL into the field in the REST2GraphQL web tool and click Generate. StepZen inspects your REST endpoint and generates the GraphQL API for you. Now the Endpoint field displays the GraphQL endpoint.

You can explore, test, and query the endpoint in the GraphiQL browser. Better still, you can build with your GraphQL API right away. You can even share it with your team and friends for them to build with it. You'll get code samples in a few languages that you can use to call the API. All in all you'll have a GraphQL API up and running in minutes.

Start with REST endpoint - no query params, no headers

In its simplest mode, you can generate a GraphQL API by providing just the REST endpoint value. If you start with the example provided in the tool, you'll see the REST endpoint is https://jsonplaceholder.typicode.com/todos/1. You can leave the Query Params, Header, and JSON fields blank, click Generate and see the GraphQL API generated for you, as well as cURL, JavaScript, and Python code you can use to call it.

POST JSON Data

But what if we needed to post some JSON data? For this example, we'll use the httpbin sample API: https://httpbin.org/post

First, select the POST method:

Next, paste the JSON that you'd like to send and click Generate.

Your GraphQL API is generated for you, its URL is provided in the Endpoint field: https://publicb93954a786e9c842.stepzen.net/api/httpbin/__graphql

On inspection of the JavaScript code example that we can use to query the GraphQL API, we can see that it generates a query with parameters for the JSON we need to send, key1 and key2, evident in the variables: {} field.

let headers = {};

headers["Content-Type"] = "application/json";

var graphql = JSON.stringify({
    query: "query httpbinQuery($key1: String, $key2: String){\n    httpbinQuery(key1: $key1, key2: $key2){\n        args\n        data\n        files\n        form\n        headers{\n            Accept\n            Host\n        }\n        json{\n            key1\n            key2\n        }\n        origin\n        url\n    }\n}",
    variables: {key1: String,key2: String}
})

var requestOptions = {
method: 'POST',
headers: headers,
body: graphql
};

fetch("https://publicb93954a786e9c842.stepzen.net/api/httpbin/__graphql", requestOptions)
.then(response => response.text())
.then(result => console.log(result))
.catch(error => console.log('error', error));

Add a Query Parameter

Now, say we have an endpoint, https://ghibliapi.herokuapp.com/films?limit=1, which takes the query parameter limit.

Enter the endpoint URL in the Endpoint field:

And enter the query params in the Query Params key and value fields

Click Generate and see that the query parameter is translated into your cURL request.

curl --request POST 'https://publicf1bf36ef9fe92aea.stepzen.net/api/ghibliapi/__graphql' \
--header 'Content-Type: application/json' \
--data-raw '{"query":"query ghibliapiQuery($limit: String){\n    ghibliapiQuery(limit: $limit){\n        description\n        director\n        id\n        image\n        locations\n        movie_banner\n        original_title\n        original_title_romanised\n        people\n        producer\n        release_date\n        rt_score\n        running_time\n        species\n        title\n        url\n        vehicles\n    }\n}","variables":{}}'

Add a Path Parameter

To add a path parameter to your request, enter the path and value inside the path parameter tab.

Using the https://jsonplaceholder.typicode.com/todos/1 REST endpoint as an example, the Path Params values are /todos/$todo and 1.

Examine the JavaScript, Python , and cURL examples generated for you to see how the path parameters are used.

Wrapping it up

REST2GraphQL requires no CLI download and no StepZen account. You can create a fully functional GraphQL API from your REST endpoint. It is deployed right away and available to call from your app or command line. Your public endpoint is available for 30 days for you to use in your projects and share with others. If you'd like to have a permanent endpoint, want to extend and customize the one you create here, want to combine two REST endpoints in one GraphQL API, and more, sign up for a free account.

With your StepZen account, you can generate a GraphQL API for any REST, SQL, or NoSQL endpoint in a few minutes. You can use StepZen's GraphQL directives @rest, @dbquery, @graphql, and others to customize your GraphQL. See Getting Started to launch the quickstart.

We hope you REST2GraphQL useful. We'd love your feedback and suggestions and we're always keen to hear about the things you're building and answer any questions. Hope to see you over on Discord.

A New Postman Collection of GraphQL APIs for the SaaS Services You Love

Lucia Cerchie — Mon, 16 May 2022 19:42:32 +0000

Postman’s platform includes a new collections feature to group saved API requests and share them with your colleagues or the world!

StepZen has created a unique Postman collection of GraphQL APIs for you to try out.

The collection features a wide range of APIs, including the APIs of platforms like Github, Spotify, and Twitter. It's also got a Web3 collection that features the Moralis server for Web3 apps.

Give each API a spin in the GraphQL APIs for SaaS Services collection by

selecting the API folder and then selecting ‘queries’

Hitting ‘Send’ and viewing your query response

Create Your Own GraphQL Collections By Forking Ours

One of the convenient features of Postman Collections is that you can fork a collection and edit it yourself. Here are the steps for riffing on our collections to make your own.

Select ‘create a fork’.

Label your fork and click ‘Fork Collection’.

Your fork is now available in the workspace you selected. You can now add your own queries to the API by clicking ‘Duplicate’.

You can now edit the name, endpoint, and body of your request.

Conclusion

We’ve seen some awesome projects created with these APIs, from a stock trading website to an animation app, to a serverless blog. We'd love to see what you come up with! If you’d like to see where our APIs are sourced from, visit https://graphql.stepzen.com/.

If you have questions, feel free to slide into our Discord. We'd love to see you around!

To learn more about creating GraphQL APIs quickly with StepZen, see

Forem: Lucia Cerchie

Beyond the Basics: Windowing in Apache Kafka

What Is Windowing?

Why is that?

Hopping

Creating a hopping window

Tumbling

Creating a tumbling window

Creating a session window

Sliding

Creating a sliding window

A note on late events

Where to go from here

A Framework For Knowing What You Don't Know

Stage 1: Nothing

Stage 2: One Instance of a Concept

Stage 3: Two instances, of different kinds, of a concept.

Stage 4: The third instance of a different kind.

Generalizing this process

What Are Apache KafkaⓇ Consumer Group IDs?

What is a Kafka consumer?

Using consumer Group IDs to detect new data

Checking for new data

Consumer Group IDs in work sharing

The role of consumer Group IDs in fault tolerance

What happens when a consumer crashes?

How does a Group ID play into rebalancing?

Where to go from here

Three Things I've Learned About Writing Abstracts

1. Connect with your audience in the first sentence.​

2. Position your pronouns thoughtfully.​

3. Let your solution for the audience's pain point be clear.​

What does 'batching' mean when we're talking about Apache KafkaⓇ?

Installing node-rdkafka on M1 for use with SASL

Introduction to Key Apache KafkaⓇ Concepts

Introduction

What is Kafka?

What is an Event?

What is a Topic?

What is a Cluster?

What is a Partition?

What is a Producer?

What is a Consumer?

Conclusion

How To GraphQL - Tutorials to Get You Started

Query SaaS APIs with GraphQL in a Next.js App

Two Ways to Query Airtable with GraphQL

Automatically Generating TypeScript Types

Combing Data From Node.js Microservices

Load Testing GraphQL APIs With k6

Where to Go From Here

Consuming the Deepgram API: The GraphQL Edit

Converting the Deepgram REST API to GraphQL

Consuming the GraphQL API in a Next.js App

French Sample

Where to Go From Here

3 New GraphQL Developer Tools to Increase Coding Productivity

REST2GraphQL

JSON2SDL

stepzen import

Conclusion

Convert REST to GraphQL: A New Low-Code Approach

How to transform REST to GraphQL with one click

Start with REST endpoint - no query params, no headers

POST JSON Data

Add a Query Parameter

Add a Path Parameter

Wrapping it up

A New Postman Collection of GraphQL APIs for the SaaS Services You Love

Create Your Own GraphQL Collections By Forking Ours

Conclusion

1. Connect with your audience in the first sentence.

2. Position your pronouns thoughtfully.

3. Let your solution for the audience's pain point be clear.

`stepzen import`