Forem: Taiwo Momoh

The Transactional outbox pattern

Taiwo Momoh — Sat, 21 Jun 2025 08:26:07 +0000

Problem statement

Your company runs a distributed system that handles user data. When a user is created or updated, two things are supposed to happen:

The system writes the changes to the user database (the source of truth).
It then publishes an event to Kafka, notifying downstream services so they can update their read-optimized tables.

The system successfully writes to the database. Immediately after, before it can send the Kafka event, the service crashes due to a bug, an out-of-memory error, or a sudden shutdown. 😢😢

Your write database has the correct user information but the read-optimized table has no idea this change happened.

This failure results in inconsistent data. This, right here, is what we call the Dual Write Operations Issue.

There are several ways to solve this issue, one of them is using
Transactional outbox pattern

What is transactional outbox pattern

A design pattern used in distributed systems to reliably publish events/messages to other systems only if the corresponding database transaction succeeds.

In a normal database transaction, you can’t directly publish an event to Kafka because if the transaction fails, rollback is impossible. Database transactions can be undone if something goes wrong, but Kafka messages are fire-and-forget.

So Instead of sending the event directly to Kafka, you insert the event into an outbox table in the same database transaction , This ensures that either both the user data and the event is saved, or neither is (thanks to ACID transactions).

This event is later published to Kafka by a relay process. This ensures our read-optimized tables stay in sync with the main database.

A nodejs example that insert user in the user table and inserts an event in the outbox

 try {
    // Start transaction
    await client.query('BEGIN');

    // 1. Insert user
    const userResult = await client.query(
      'INSERT INTO users (name, email) VALUES ($1, $2) RETURNING id',
      ["test", "test@email.com"]
    );
    const userId = userResult.rows[0].id;

    // 2. Insert into outbox (same transaction)
    const eventPayload = {
      userId,
      email: "test@email.com",
      timestamp: new Date().toISOString()
    };

    await client.query(
      'INSERT INTO outbox (event_type, payload) VALUES ($1, $2)',
      ['UserCreated', eventPayload]
    );

    // 3. Commit if both succeed
    await client.query('COMMIT');
    console.log('User and event saved!');

    return userId;
  } catch (error) {
    // Rollback on any error
    await client.query('ROLLBACK');
    console.error('Transaction failed:', error);
    throw error;
  }

The Relay Process

The message relay runs in the background, constantly checking the outbox table for unpublished events. When it finds this unpublished records , it sends them to Kafka and are marked as processed or deleted from the outbox.

Ways to implement the Relay Process

Polling Service is a background process that constantly checks the outbox table for new or unpublished events. While it gets the job done, it can add unnecessary load to your database, especially if you're scanning large parts of the table without proper indexing or batching, and it gets worse if it runs too frequently.

Change Data Capture (CDC) can be used in conjunction with the Transactional Outbox pattern to efficiently capture changes made to the outbox table and publish them to kafka.

Few Drawbacks

Kafka might deliver the same message multiple times. The receiving service needs to handle this gracefully by implementing idempotency, which is basically making sure that processing the same event twice doesn't break anything.
Event ordering is something you really need to watch out for too. If you're doing event sourcing, those events hitting Kafka better be in the exact same sequence as your database updates, otherwise you're going to reconstruct the wrong state.

When Should you use transactional outbox pattern

You're building an event-driven application where a database update initiates an event notification.
You want to ensure atomicity in operations that involve two services.
Mostly valid for microservice architecture, for monolith it adds unnecessary complexity and performance overhead.

Additional articles

https://docs.aws.amazon.com/prescriptive-guidance/latest/cloud-design-patterns/transactional-outbox.html

The Saga Pattern

Taiwo Momoh — Mon, 09 Jun 2025 09:18:28 +0000

This article provides an overview of what I learned about the Saga pattern.

A Saga is a sequence of local transactions. Each local transaction updates the database and publishes a message or an event (either success or failure message) to the next transaction in the Saga.

It ensures atomicity. This means the transaction either completes fully or not at all. If one of the service fails, it undoes all previous changes (e.g., an update made to the database) made by other service. This process is known as a compensating transaction.

In saga we have two primary coordination style

Orchestrator
Choreography

The Orchestrator is like a central brain. It directs each step of the Saga, telling services when to perform their operations and what to do next.

// Orchestrator
async function bookTripSaga() {
  console.log("🧠 Starting trip booking saga...");

  try {
    await reserveFlight();
    try {
      await bookHotel();
      try {
        await rentCar();
        console.log("✅ Trip booked successfully!");
      } catch (err) {
        console.error("❌ Car rental failed:", err.message);
        await cancelHotel();
        await cancelFlight();
      }
    } catch (err) {
      console.error("❌ Hotel booking failed:", err.message);
      await cancelFlight();
    }
  } catch (err) {
    console.error("❌ Flight reservation failed:", err.message);
  }

  console.log("🧠 Saga finished.");
}

This example demonstrates the basic concept of a saga orchestrator with compensating transactions for failures, and less than ideal for production use.

In the Choreography coordination style, services publish events that trigger a local transaction in other services.

Instead of a central orchestrator telling a service what to do, each service listens for events and reacts accordingly.

This Javascript example gives you a taste of the Choreography coordination style. 👇

const { EventEmitter } = require('events');

const eventBus = new EventEmitter();

// ======================
// Simulated Services
// ======================

eventBus.on('TripRequested', async () => {
  console.log("✈️  [FlightService] Reserving flight...");
  try {
    await Promise.resolve("FlightReserved");
    console.log("✅ [FlightService] Flight reserved");
    eventBus.emit('FlightReserved');
  } catch (err) {
    console.error("❌ [FlightService] Flight reservation failed:", err.message);
  }
});

eventBus.on('FlightReserved', async () => {
  console.log("🏨 [HotelService] Booking hotel...");
  try {
    await Promise.resolve("HotelBooked");
    console.log("✅ [HotelService] Hotel booked");
    eventBus.emit('HotelBooked');
  } catch (err) {
    console.error("❌ [HotelService] Hotel booking failed:", err.message);
    eventBus.emit('HotelBookingFailed');
  }
});

eventBus.on('HotelBooked', async () => {
  console.log("🚗 [CarService] Renting car...");
  try {
    await Promise.reject(new Error("Vehicle not available")); // simulate failure
    console.log("✅ [CarService] Car rented");
    eventBus.emit('TripCompleted');
  } catch (err) {
    console.error("❌ [CarService] Car rental failed:", err.message);
    eventBus.emit('CarRentalFailed');
  }
});

// Compensation handlers
eventBus.on('CarRentalFailed', async () => {
  console.log("❌ [HotelService] Canceling hotel...");
  await Promise.resolve("HotelCanceled");

  console.log("❌ [FlightService] Canceling flight...");
  await Promise.resolve("FlightCanceled");

  console.log("🧠 Trip saga compensated.");
});

eventBus.on('HotelBookingFailed', async () => {
  console.log("❌ [FlightService] Canceling flight...");
  await Promise.resolve("FlightCanceled");

  console.log("🧠 Trip saga compensated.");
});

eventBus.on('TripCompleted', () => {
  console.log("🎉 Trip booking completed successfully!");
});

// ======================
// Start Saga
// ======================
function bookTripChoreographySaga() {
  console.log("🧠 Starting trip booking (choreography saga)...");
  eventBus.emit('TripRequested');
}

bookTripChoreographySaga();

The Isolation Challenge in Sagas

In a Saga, especially a distributed saga where each service manages its own database. If multiple sagas run concurrently, they might:

Interfere with each other
See partially updated data
Trigger inconsistent business data

Sagas lacks built-in isolation. Imagine two users are trying to

Book the last room in a hotel
Both check availability → see it's free
Both try to reserve it → race condition

Without isolation, both might believe they succeeded, leading to overbooking.

So in this case developers must implement it isolation explicitly

Implementing Countermeasures 🪖🪖.

Pessimistic Concurrency (Reservation Pattern) : Temporarily “locks” a resource by marking it as reserved. So other sagas see it as unavailable.

Example

async function bookTripSaga(userId, hotelId, flightDetails ) {

  try {
    // Step 1: Reserve the hotel
    const hotelReserved = await reserveHotel(hotelId, userId);

    // If a saga is currenntly running, hotel is marked as reserved
   //  So other sagas see it as unavailable
    if (!hotelReserved) throw new Error("Hotel unavailable");

    // Step 2: Book the flight
    const flightBooked = await bookFlight(flightDetails, userId);
    if (!flightBooked) throw new Error("Flight booking failed");

    // Step 3: Mark hotel as booked
    await confirmHotel(hotelId, userId);

    return { success: true, message: "Trip booked successfully" };

  } catch (error) {
    // Compensation to marked hotel as available
    await releaseHotel(hotelId, userId);
    return { success: false, message: error.message };
  }
}

Other issues and considerations

Increased Latency :
1. If the transaction makes synchronous calls (e.g., using the Orchestrator approach), each service call waits for a response before continuing.
2. Compensatory transactions can also add latency to the overall response time. For example, if step 5 of a 6-step Saga fails, all previous steps must be rolled back, which adds additional latency.
Idempotency: Failures during a Saga (or during compensation) can be tricky to recover from. They require retry logic, and this is where idempotency becomes crucial.

👉 Read more on idempotency: Engineering Idempotency Keys

Dual writes issue: The dual-write issue occurs when two external systems has to atomically update the database and publish an event. The failure of either operation might lead to an inconsistent state. One way to solve this is to use the transactional outbox pattern.

Glossary

Transactions: In the context of databases and data storage systems, a transaction is any operation that is treated as a single unit of work, which either completes fully or does not complete at all.

How to set up path Alias in react.

Taiwo Momoh — Fri, 29 Sep 2023 23:19:09 +0000

I'm not sure if this is the right title for this, but it took me a while to find a solution for this, So i'm keeping this here.

Note: I'm assuming you are using react with vite.

Add this to the tsconfig.json file under the compilerOptions object.

"baseUrl": ".",
"paths": {
  "@/*": ["./src/*"]
}

Install @types/node to dev dependency for type safety for importing from the path module.

yarn add -d @types/node

And change the vite.config.ts to understand the @ sign as well as an alias of the "src" folder.

import path from 'path'
import react from '@vitejs/plugin-react'
import { defineConfig } from 'vite'

export default defineConfig({
  plugins: [react()],
  resolve: {
    alias: {
      '@': path.resolve(__dirname, './src'),
    },
  },
})

Solution was extracted from Borris article.

End.

Food Library- Using azure app service, NodeJs, Pug

Taiwo Momoh — Sun, 06 Mar 2022 11:57:47 +0000

Overview of My Submission

Food Library is a nodejs, expressJs out of the blue idea, the main focus of this project is making use and learning all the functions available in the app service category. like first deploying with Vscode then later implemented the CI/CD with Github, creating an alert rule and lots more.

Food Library Aims to promote your country culture through its wide variety of foods and dishes, Post your dishes and hype your country.

Submission Category:

Computing Captains Category

Link to Code on GitHub

tylerjusfly / azure-devto-hackathon

This an API though with PUG as its templating engine.

Hello World, Behold My Dev.To Azure Hackathon (runners up project)

This an API though with PUG as its templating engine.

Fuctionalities

SignUp
SignIn
Visit Dashboard
Add a dish that exist in your country to database
view dishes without creating any account
Search for dishes

Future Functions

Edit Button
Delete Dish Button
Pagination
rewriting

visit us

https://food-library-azure.herokuapp.com/

View on GitHub

The Functionalities and how to find your way around is in the README.md file.
The MIT license can be found in the LICENSE file and is part of the repository.

Forem: Taiwo Momoh

The Transactional outbox pattern

What is transactional outbox pattern

The Relay Process

Ways to implement the Relay Process

Few Drawbacks

When Should you use transactional outbox pattern

The Saga Pattern

The Isolation Challenge in Sagas

Implementing Countermeasures 🪖🪖.

Other issues and considerations

How to set up path Alias in react.

Food Library- Using azure app service, NodeJs, Pug

Overview of My Submission

Submission Category:

Link to Code on GitHub

tylerjusfly / azure-devto-hackathon

This an API though with PUG as its templating engine.

Hello World, Behold My Dev.To Azure Hackathon (runners up project)

This an API though with PUG as its templating engine.

Fuctionalities

Future Functions

visit us

Additional Resources / Info

Created my alert rule from azure app

The HomePage where a User can SignIn and Signup

My Triggered alert when i have > 2 code 200 request

View various dishes and add yours to it

There were no other developers involved in this work.