Forem: Peter Kraft

How To Process Events Exactly-Once

Peter Kraft — Sun, 06 Oct 2024 21:16:08 +0000

Want to process incoming events exactly-once?

Well, any distributed systems pedant will say you can't, because it's theoretically impossible. And technically, they're right: if you send a message and don't get an answer, you have no way of knowing whether the receiver is offline or just slow, so eventually you have no choice but to send the message again if you want it processed.

So if exactly-once processing is impossible, why do many systems, including DBOS, claim to provide it?

The trick is to leverage another property: idempotence. If you design a message receiver to be idempotent, then you can deliver a message to it multiple times and it will be fine because the duplicate deliveries have no effect. Thus, the combination of at-least-once delivery and idempotence is identical to exactly-once semantics in practice.

Under the hood, this is exactly how DBOS event receivers (like for Kafka) work. They generate a unique key from an event (for example, from a Kafka topic + partition + offset) and use it as an idempotency key for the event processing workflow. That way, even if an event is delivered multiple times, the workflow only processes it once.

Here's all the code you need to process Kafka messages exactly-once:

from dbos import DBOS, KafkaMessage

@DBOS.kafka_consumer(config, ["topic"])
@DBOS.workflow()
def test_kafka_workflow(msg: KafkaMessage):
    DBOS.logger.info(f"Message received: {msg.value.decode()}")

Learn more here!

Making Serverless 15x Cheaper

Peter Kraft — Thu, 13 Jun 2024 00:41:35 +0000

We’ve often heard developers working on stateful applications say that they want to adopt serverless technology to more easily deploy to the cloud, but can’t because it’s prohibitively expensive. For clarity, by “stateful applications” we mean applications that manage persistent state stored in a database (for example, a web app built on Postgres).

In this blog post, we'll show how to make serverless 15x cheaper for stateful applications. First, we'll explain why it's too expensive right now. Then, we'll show how we architected the DBOS Cloud serverless platform to make serverless affordable for stateful applications.
‍

Improving Serverless Efficiency

The first strategy DBOS Cloud uses to make serverless efficient is sharing executors across requests to improve resource utilization. The key idea is that stateful applications are typically I/O-bound: they spend most of their time interacting with remote data stores and external services and rarely do complex computation locally.

Consider an e-commerce application that shows you the status of your latest order. When you send it a request, it calls out to a database to look up what your last order was, then waits for a response, then calls out again to look up the status of that order, then waits for a response, then finally returns the status back to you. Here’s what that might look like:

Current serverless platforms execute such applications inefficiently because they launch every request into a separate execution environment. Each execution environment sends a request to the database, then is blocked doing nothing useful until it receives the database’s response. Here’s what it looks if two people concurrently look up their latest orders in AWS Lambda:

‍

For stateful applications, current serverless platforms spend most of their time doing nothing, then charge you for the idle time.

By contrast, DBOS Cloud multiplexes concurrent requests to the same execution environment, using the time one request spends waiting for a result from the database to serve another request. To make sure execution environments are never overloaded, DBOS Cloud continuously monitors their utilization and autoscales when needed, using Firecracker to rapidly instantiate new execution environments. Here’s what it looks if multiple people concurrently look up their latest orders in DBOS:

DBOS Includes Reliable Workflow Execution

The second strategy DBOS Cloud uses is leveraging DBOS Transact’s reliable workflows to efficiently orchestrate multiple functions in a single application. Many applications consist of multiple discrete steps that all need to complete for the application to succeed; for example, the checkout flow for an online store might reserve inventory, then process payment, then ship the order, then send a confirmation email:

‍
Conventional serverless platforms require an expensive external orchestrator like AWS Step Functions to coordinate these steps, executing each in sequence and retrying them when they fail. By contrast, DBOS Cloud uses the reliable workflows built into open-source DBOS Transact to guarantee transactional execution for an application at no additional cost.

DBOS Cloud vs. AWS Lambda Cost Comparison

Now that we’ve explained how DBOS Cloud achieves its cost efficiency, let’s measure it by comparing the cost to run a stateful application workload on DBOS Cloud versus AWS Lambda. We’re referencing AWS Lambda because it’s the most popular serverless platform, but the numbers are comparable for other platforms like Azure Functions or Google Cloud Functions.

Consider an application workflow with four steps. To keep the math simple, let’s say each step takes 10 ms and the application is invoked 250M times a month (~100 application invocations/second). Since the workflow takes 40 ms total, this works out to 10M execution seconds per month. Assuming a 512MB executor is used for both DBOS Cloud and Lambda, here’s how the cost compares:

DBOS Cloud Cost

We’ll assume that in DBOS, the application is implemented as four operations in a DBOS Transact reliable workflow. Stateful, reliable workflow execution is built into DBOS, so there is no need for a separate orchestrator like AWS Step Functions. The 10M execution seconds per month falls within the $40 per-month DBOS Cloud Pro pricing tier, so the total cost is $40 per month.

AWS Lambda + AWS Step Functions Cost

In AWS Lambda, this application would likely be implemented as four functions orchestrated by an AWS Step Functions workflow. AWS Lambda charges $0.20 per 1M function invocations plus $8.33 per 1M execution seconds (assuming a 512MB executor). Additionally, AWS Step Functions charges (using Express Workflows, the cheapest option) $1.00 per 1M workflow invocations plus $8.33 per 1M operation-seconds of execution. Thus the total cost for Lambda is $200 for the 1B function invocations (four per workflow) plus $83.33 for 10M execution seconds. The total cost for Step Functions is $250 for 250M workflow invocations plus $83.33 for 10M operation-seconds. The grand total is $616.66. As we said earlier in this post, that is over 15 times more expensive than DBOS Cloud. Here's a table to summarize our cost comparison:

‍

Try it out!

To get started with DBOS, follow our quickstart to download the open-source DBOS Transact framework and start running code locally.

If you have any questions about the article or about DBOS, please ask in the comments!

How to Process Events Exactly-Once with Kafka and DBOS

Peter Kraft — Fri, 03 May 2024 22:00:40 +0000

Many applications built on event streaming systems like Apache Kafka need exactly-once semantics for data integrity. In other words, they need to guarantee that events are processed exactly one time and are never under- or over-counted. However, exactly-once semantics are hard to obtain in distributed systems, so many event streaming platforms default to weaker guarantees. For example, Kafka natively supports exactly-once semantics only for stream processing, not for general applications. As a result, developers who need exactly-once semantics must invent complicated architectures to obtain it.

In this post, we’ll show you how the open-source TypeScript framework DBOS Transact simplifies makes exactly-once event processing easy. We’ll demonstrate how to write, in <15 lines of code, a Kafka consumer that processes every message sent to a topic exactly one time, regardless of interruptions, crashes, or failures.

How exactly-once Kafka message processing works with DBOS Transact

Implementing exactly-once semantics in a distributed system is challenging because it’s theoretically impossible: the sender of a message cannot always verify whether an unresponsive receiver failed to process their message or is just being slow to acknowledge receipt. To get around this problem, a common strategy is to deliver a message multiple times, but design the receiver to process each event idempotently so that duplicate message deliveries have no effect. However, developing idempotent applications is also notoriously difficult!

DBOS Transact makes exactly-once event processing easy by making idempotency easy. You can process Kafka events exactly-once with two different types of operations:

Transactions, which execute a single database transaction.
Workflows, which reliably execute multiple operations.

When your consumer receives a message from Kafka, DBOS Transact constructs an idempotency key for that message from the message’s topic, partition, and offset–a combination that is guaranteed to be unique for a Kafka cluster. What happens next depends on whether you’re executing a transaction or a workflow.

Exactly-once transactions

If you’re processing the message with a single transaction, DBOS Transact synchronously executes the transaction, records its idempotency key as part of that transaction, then commits the message’s offset to Kafka. This guarantees exactly-once processing because:

If transaction execution fails, the offset isn’t committed, so Kafka retries the message later.
If a duplicate message arrives, DBOS Transact checks the idempotency key and doesn’t re-execute the transaction (the transaction logic also handles corner cases where the message arrives twice simultaneously–a subject for a future blog post).

Exactly-once workflows

If you’re processing the message with a workflow, DBOS Transact synchronously records the idempotency key, commits the message’s offset to Kafka, then asynchronously processes the workflow. This guarantees exactly-once processing because:

If recording the idempotency key fails, the offset isn’t committed, so Kafka will send the message again later.
If the workflow fails, it will always recover and resume from where it left off because DBOS workflows are reliable.
If a duplicate message arrives, DBOS Transact checks the idempotency key and doesn’t start a new workflow.

Getting Started with Kafka and DBOS

Now, let’s see what this looks like in code. The first step to exactly-once semantics is to write your event processing code in TypeScript using DBOS Transact. You can write a single database transaction or a workflow coordinating multiple operations. For now, let’s keep it simple:

import { Workflow, WorkflowContext } from '@dbos-inc/dbos-sdk';

class KafkaExample{
  @Workflow() 
  static async kafkaWorkflow(ctxt: WorkflowContext, topic: string, partition: number, message: KafkaMessage) {      
    ctxt.logger.info(`Message received: ${message.value?.toString()}`) 
  }
}

Next, annotate your method with a @KafkaConsume decorator specifying which topic to consume events from. Additionally, annotate your class with an @Kafka decorator defining which brokers to connect to. That’s all there is to it! When you start your application, DBOS Transact will invoke your method exactly-once for each message sent to the topic.

import { KafkaConfig, KafkaMessage} from "kafkajs";
import { Workflow, WorkflowContext, Kafka, KafkaConsume } from '@dbos-inc/dbos-sdk';

@Kafka({ brokers: ['localhost:9092']})
class KafkaExample{
  @KafkaConsume("example-topic")
  @Workflow()
  static async kafkaWorkflow(ctxt: WorkflowContext, topic: string, partition: number, message: KafkaMessage) {
    ctxt.logger.info(`Message received: ${message.value?.toString()}`) 
  }
}

If you need more control, you can pass KafkaJS configuration objects to both the class and method Kafka decorators. For example, you can specify a custom consumer group ID:

@KafkaConsume("example-topic", { groupId: "custom-group-id" })
@Workflow()
static async kafkaWorkflow(ctxt: WorkflowContext, topic: string, partition: number, message: KafkaMessage) { 
  ctxt.logger.info(`Message received: ${message.value?.toString()}`)
}

To get started with DBOS Transact, check out the quickstart and docs. For more information on using Kafka with DBOS Transact, check out this docs page. To join our community:

Check out our code and contribute on GitHub.
Join us on Discord.