Forem: Mohamed Hassan

Simply Order (Part 9) — CQRS Pattern: Separating Reads from Writes for Better Performance

Mohamed Hassan — Mon, 08 Dec 2025 22:57:16 +0000

This is the ninth article in our series, where we design a simple order solution for a hypothetical company called Simply Order. The company expects high traffic and needs a resilient, scalable, and distributed order system.

In the previous articles:

Simply Order (Part 1) Distributed Transactions in Microservices: Why 2PC Doesn't Fit and How Sagas Help
Simply Order (Part 2) — Designing and Implementing the Saga Workflow with Temporal
Simply Order (Part 3) — Linking It All Together: Connecting Services and Watching Temporal in Action
Simply Order (Part 4) — Reliable Events with the Outbox Pattern (Concepts)
Simply Order (Part 5) — Hands-On: Building the Outbox Pattern for Reliable Events
Simply Order (Part 6) – Making APIs Idempotent: Because Users Double-Click and Networks Lie
Simply Order (Part 7) – Querying Orders with Details: API Composition Pattern
Simply Order (Part 8) — Building GraphQL API Service with spring-graphql

We built the core services with distributed transactions, reliable event handling, and idempotent operations. We also created a GraphQL API service to compose queries across multiple microservices.

However, we've encountered a hidden performance problem. Let's explore it.

The Problem

Imagine a typical order workflow in Simply Order:

A customer places an order.
The Order Service writes the order to its database.
The Payment Service processes the payment.
The Inventory Service reserves items.
The GraphQL API queries multiple services to return order details to the client.

This works fine, but consider what happens during peak traffic:

Hundreds of orders are being written simultaneously.
Simultaneously, customers are querying their orders, shipments, and payment statuses.
The database is now handling both heavy write operations (inserts, updates) and read operations that fetch complex joins across tables.

The result? Database contention. Reads are blocked by writes. Writes compete for locks. Queries become slow. Your system feels sluggish.

🤔 What if we could optimize writes and reads separately?

CQRS: Command Query Responsibility Segregation

CQRS is a pattern that separates the model for updating data (writes) from the model for reading data (reads). Instead of having one unified model that handles both operations, you maintain two separate models optimized for their specific purpose.

Core Idea

Commands: Operations that modify data (create, update, delete). They write to an optimized write model.
Queries: Operations that fetch data. They read from an optimized read model.

The two models are kept in sync through events — typically the same events we are already publishing via the Outbox Pattern.

Benefits

Independent Scalability: Scale writes and reads independently. Add more read replicas without affecting write throughput.
Performance Optimization: The write database can be optimized for transactions and consistency. The read database can be optimized for queries and reporting.
Simplified Queries: Instead of complex joins and aggregations, read models store denormalized, query-optimized data.
Event Sourcing Ready: CQRS pairs naturally with event sourcing for complete audit trails.

CQRS Approaches

There are different ways to implement CQRS, ranging from simple to complex:

1. Separate Repositories (Simple CQRS)

In this lightweight approach, you maintain two separate repositories within the same service:

Write Repository: Optimized for transactions and consistency. Used during command execution.
Read Repository: Optimized for queries. Used during read operations.

Both repositories can use the same database or different databases — the key is logical separation.

Pros:

Simple to implement.
No additional infrastructure required.
Great starting point for CQRS.
Perfect for services that don't have extreme read/write asymmetry.

Cons:

Limited separation of concerns.
Both models live in the same process.
Scaling reads independently is harder without external infrastructure.

2. Different Data Sources (Complex CQRS)

In this approach, the write and read models use completely different databases or data sources:

Write Side: Transactional database (e.g., PostgreSQL) optimized for ACID compliance.
Read Side: Search or analytics database (e.g., Elasticsearch, DynamoDB, or a data warehouse).

The read model is populated asynchronously through events published by the write side.

Pros:

Complete separation of concerns.
True independent scalability.
Write model can focus on transactions; read model can focus on performance.
Read replicas can be geo-distributed.

Cons:

Increased complexity.
Eventual consistency — reads might be slightly stale.
Requires event infrastructure to keep models in sync.
Operational overhead for multiple databases.

Implementation: Our Approach

In Simply Order, we've implemented a simple CQRS approach using separate logical repositories with same model:

The code for this project can be found in this repository:

https://github.com/hassan314159/simply-order

Since this repository is continuously updated, the code specific to this lesson can be found in the implment_cqrs_for_order_and_inventory branch. Start with:

git checkout implment_cqrs_for_order_and_inventory

We divided the Order Service into two distinct repositories:

Write Repository (Command Side)

public interface OrderRepository extends JpaRepository<OrderEntity, UUID> {
}

This repository handles:

Creating new orders
Updating order status during Saga workflow
Committing changes to the transactional database

Read Repository (Query Side)

public interface OrderSearchRepository extends JpaRepository<OrderEntity, UUID> {

    @Query("""
        select distinct o from OrderEntity o
        left join o.items i
        where (:customerId is null or o.customerId = :customerId)
          and (:status     is null or o.status = :status)
          and (:fromDate   is null or o.createdAt >= :fromDate)
          and (:toDate     is null or o.createdAt < :toDate)
          and (:sku        is null or i.sku = :sku)
        order by o.createdAt desc
        """)
    List<OrderEntity> search(
            @Param("customerId") String customerId,
            @Param("status") String status,
            @Param("fromDate") OffsetDateTime fromDate,
            @Param("toDate") OffsetDateTime toDate,
            @Param("sku") String sku
    );
}

This repository handles:

Fetching orders for a customer
Searching orders by status
Filtering by date ranges
All read operations

Service Layer

we have split the service layer logically into two

OrderCommandService
OrderQueryService

OrderCommandService.java

@Service
public class OrderCommandService {
    @Transactional
    public UUID createOrder(CreateOrderRequest request) throws JsonProcessingException {

        UUID orderId = UUID.randomUUID();
        CreateOrderCommand cmd = CreateOrderCommand.from(orderId, request);
        OrderEntity order = OrderEntity.create(cmd);
        orderRepo.save(order);

        String payload = objectMapper.writeValueAsString(order);
        outboxRepo.save(OutboxEntity.pending("OrderCreated", orderId, payload));

        return orderId;

    }
}

OrderQueryService.java

@Service
public class OrderQueryService {

    private final OrderSearchRepository orderSearchRepository;
    public final OrderMapper orderMapper;

    public OrderQueryService(OrderSearchRepository orderSearchRepository, OrderMapper orderMapper){
        this.orderSearchRepository = orderSearchRepository;
        this.orderMapper = orderMapper;
    }

    public Optional<OrderResponse> findByOrderId(UUID orderId) {
        return orderSearchRepository.findById(orderId).map(orderMapper::toDto);
    }

    public List<OrderResponse> search(String customerId, String status, OffsetDateTime fromDate, OffsetDateTime toDate, String sku) {
        return orderSearchRepository.search(customerId, status, fromDate, toDate, sku)
                .stream()
                .map(orderMapper::toDto)
                .toList();
    }

}

Same have been applied to inventory repository

Homework

Our current implementation uses separate repositories logically but still using same tables and database. The next evolution would be to:

Evolution Path

Starting Simple: Separate the Model

By separating the model — i.e., having separate tables and entities for Search, so we can gain multiple benefits:

Easier to optimize each side independently.
Better testing — you can test write and read logic separately.

Keeping Models in Sync: When an order status changes (via the Saga workflow), we can simply update both repositories within updateOrderStatus(). We can also isolate the updates of the two repositories by sending an event or using event outbox pattern and create a relay that updates OrderStatus in OrderSearchRepository

Evolving to Separate Data Sources

As your system grows, you might migrate to completely separate databases:

Extract the read model to a dedicated read database (e.g., PostgreSQL read replica or Elasticsearch).
Publish order events to Kafka when orders are created or updated.
Build a read service that consumes these events and populates the read database.

This would be a full-fledged CQRS implementation with separate data sources. Give it a try — or consider the trade-offs and decide if your system really needs that level of separation.

This gives you:

True independent scaling.
Ability to optimize read databases for your specific query patterns.
Geo-distributed read replicas.

But it also introduces complexity — you'll need to handle eventual consistency and synchronization failures.

Wrap-up

In this lesson, we introduced the CQRS pattern and showed that it doesn't have to be complex. By simply separating our repositories into write and read models, we've already gained the benefits of CQRS: independent optimization, clearer code structure, and a path forward for scaling.

CQRS pairs beautifully with the Outbox Pattern
— a pattern you've already seen in previous articles. Together, they form the backbone of scalable, reliable distributed systems.

The beauty of CQRS is that you can start simple (separate repositories) and evolve to more complex implementations (separate databases) only when you need to. Don't over-engineer from day one — understand your read/write patterns first, then evolve.

Simply Order (Part 8) – Querying Orders with Details: GraphQL in Action

Mohamed Hassan — Mon, 01 Dec 2025 18:20:23 +0000

This is the eighth article in our series, where we design a simple order solution for a hypothetical company called Simply Order. The company expects high traffic and needs a resilient, scalable, and distributed order system.

In the previous articles:

In the previous lesson, we discussed the API composition pattern and how it solves the issue of querying multiple services. In this article, we will implement a new service to act as our API composer using spring-graphql, which runs a GraphQL server exposing data from multiple microservices.

Implementation

The code for this project can be found in this repository:

https://github.com/hassan314159/simply-order

Since this repository is continuously updated, the code specific to this lesson can be found in the add_graphql_api branch. Start with:

git checkout add_graphql_api

We will use spring-graphql to build our service.

The spring-graphql dependency is the official Spring project for building GraphQL APIs in Spring Boot applications. With this dependency, we can define GraphQL schemas, implement resolvers as annotated Java methods, and expose a single /graphql endpoint for all queries and mutations. It also includes features like schema introspection and batching (to solve N+1 problems), making it simple to build flexible, type-safe APIs backed by your existing data sources.

Data Flow

Components of GraphQL

In the previous article, we saw that GraphQL consists of these components:

Language: A query language letting clients request exactly the data they need.
Schema: Defines available data types, fields, and operations (queries, mutations, subscriptions).
Server: Processes queries, validates them against the schema, and returns results.
Resolvers: Functions that fetch data for each field in the schema.
Data Sources: Databases, APIs, or services where the actual data resides.

The data flow:

Steps to Build

Server: The Execution Engine

The GraphQL server receives queries, validates them against the schema, and orchestrates data fetching through resolvers. Add the spring-graphql dependency:

<dependency>
    <groupId>org.springframework.graphql</groupId>
    <artifactId>spring-graphql</artifactId>
</dependency>

You can see it here: pom.xml

Schema: The API Contract

The schema defines all available data types, fields, and operations (queries, mutations, subscriptions) your API supports.

Schema components:

Types: Define the shape of data (e.g., User, Order, Product)
Fields: Properties of each type (e.g., User has name, email, id)
Operations:
- Query: Fetch data (read-only)
- Mutation: Modify data (create, update, delete)
- Subscription: Real-time updates via WebSocket

View the schema here: schema.graphqls

type Query {
  order(id: ID!): Order
  ordersByCustomer(customerId: ID!, limit: Int = 20): [Order!]!
  product(sku: ID!): InventoryItem
}

type Order {
  id: ID!
  customerId: ID!
  createdAt: String!
  status: OrderStatus!
  items: [OrderItem!]!
  totalAmount: Float!
}

enum OrderStatus {
  NEW
  PROCESSING
  SHIPPED
  CANCELLED
  FAILED
  COMPLETED
}

type OrderItem {
  sku: ID!
  quantity: Int!
  price: Float!
  inventory: InventoryItem
}

type InventoryItem {
  sku: ID!
  name: String!
  description: String
  availableQty: Int!
}

By defining this schema, we establish the contract and view for clients. For example, we can hide unnecessary or domain-specific fields. The InventoryItem type exposes an availableQty field while hiding implementation details like stock and reserved in the inventory service.

Also, we expose only the important statuses of OrderStatus, mapping user non-related statuses to meaningful ones (e.g., mapping OPEN & PENDING to just OPEN).

Java classes mapped to the types are here: model package

Data Sources: The Information Layer

Data sources are where data actually lives—databases, REST APIs, microservices, files, etc.

In our example, the data sources are OrderDB and InventoryDB. We don't call them directly from this microservice; instead, we use HTTP clients to query these services.

Http clients and data models are here: client package

Resolvers: The Data Fetchers

Resolvers fetch or compute data for specific fields in your schema.

We have 2 resolvers in our service:

OrderQueryController.java: Query resolvers for root-level queries, annotated with @Controller and @QueryMapping.
OrderItemFieldResolver.java: Field resolvers for nested fields, handling the N+1 problem via @BatchMapping.

Resolver code: OrderQueryController.java & OrderItemFieldResolver.java

Query Resolver Example

@QueryMapping
public Mono<Order> order(@Argument String id) {
    return orderClient.getOrder(id)
            .switchIfEmpty(Mono.error(new RuntimeException("Order not found")))
            .map(orderMapper::toModel);
}

Batch Resolver: Solving the N+1 Problem

To resolve nested fields like inventory on OrderItem, instead of querying the inventory service for each item (N queries), we use batching to fetch all required SKUs in one query:

@BatchMapping(typeName = "OrderItem", field = "inventory")
public Mono<Map<OrderItem, InventoryItem>> inventory(List<OrderItem> items) {
    List<String> skus = items.stream()
            .map(OrderItem::sku)
            .distinct()
            .toList();

    return inventoryClient.getBySkus(skus)
            .map(inventoryDtos -> {
                Map<String, InventoryItem> bySku = inventoryDtos.stream()
                        .map(inventoryMapper::toModel)
                        .collect(Collectors.toMap(InventoryItem::sku, inv -> inv));

                Map<OrderItem, InventoryItem> result = new HashMap<>();
                for (OrderItem item : items) {
                    result.put(item, bySku.get(item.sku()));
                }
                return result;
            });
}

Language: Query What You Need

GraphQL query language allows clients to specify exactly the data they want.

Sample query to fetch an order by id with its details:

{
    "query": "query { order(id: "ORD-1001") { id customerId createdAt status totalAmount items { sku quantity unitPrice } } }"
}

The outer query key contains the GraphQL operation string. Inside, the nested query specifies the fetch operation. Curly braces specify which fields to retrieve.

Homework

We have implemented the fetch operations needed for our business. However, currently, we must call the Order service directly to create orders. Can you create a mutation operation in our API service to provide a single entry point for all client interactions?

Wrap-up

In this article, the GraphQL API service was introduced as an effective API composition layer for the Simply Order system. Using spring-graphql, we defined a schema that exposes consolidated views from multiple microservices, hiding internal complexities and mapping data meaningfully for clients. We implemented query resolvers with efficient batch loading to solve common performance problems like the N+1 query issue. This design enables a scalable, flexible, and type-safe API interface that fits the needs of a high-traffic distributed order system.

Simply Order (Part 7) – Querying Orders with Details: API Composition Pattern

Mohamed Hassan — Wed, 19 Nov 2025 21:24:06 +0000

This is the seventh article in our series, where we design a simple order solution for a hypothetical company called Simply Order. The company expects high traffic and needs a resilient, scalable, and distributed order system.

In the previous articles:

In previous lessons, we built core services like Order, Payment, and Inventory, focusing on making our distributed transactions reliable using Sagas, the Outbox pattern, and Idempotency.

Now imagine the business comes with a very simple requirement:

On the Order Details page, show the order with its items, and for each item show the current stock status so customers know if they can re‑order.

The Problem

From a microservices perspective, our system looks like this:

Order Service owns orders and items by their SKU.
Inventory Service owns item details along with stock and availability per item.

Each service has its own bounded context, but users need to see their orders with the corresponding item details.

Solution 1: Direct client-to-microservice communication

This is the simplest and most naïve solution: the frontend talks directly to the services:

Call GET /orders/{id} from the Order service.
For each item in order, call GET /inventory/{sku} from the Inventory service

This leads to several problems:

The frontend suddenly “knows” about internal service boundaries.
Latency explodes; a single order fetch could trigger dozens of calls based on the number of items, known as the N+1 Problem (1 order + N items).
The frontend becomes tightly coupled with internal service APIs, so any change in service APIs requires changes in all frontend clients (webpages, mobile apps, third parties, etc.).

In short, this approach is unsuitable for a continuously evolving business and APIs.

Solution 2: API Composition

In this approach we move the complexity of aggregating responses from different services into a separate service often called the Aggregator/Composer Service:

The frontend sends requests to the Composer Service.
The Composer Service calls multiple microservices (Order, Inventory, etc.).
It collects, transforms, and merges data into a unified result, returning it to the client.

With this approach, we can overcome the problems of the naive approach:

Internal service boundaries are hidden from the client API.
The aggregator service decouples the frontend from internal APIs, so even with evolving APIs, we can preserve our contract with clients.
With a smart implementation, we can optimize the N+1 problem.

Simple custom implementation

We build a thin service called Composer or Aggregator that, for each view of our data, provides a REST endpoint which aggregates all necessary endpoints and returns the corresponding result.

Some minor drawbacks of this approach:

We have to build separate endpoints for each view of our data. For example, if some pages need different details, we may need to create different endpoints to handle those.
It can get more sophisticated with retries, parallelism, mapping, and error handling.

API Composition with GraphQL

As mentioned in GraphQL official Page

GraphQL is an open‑source query language for APIs and a server‑side runtime. It provides a strongly‑typed schema to define relationships between data, making APIs more flexible and predictable. And it isn’t tied to a specific database or storage engine — it works with your existing code and data, making it easier to evolve APIs over time.

GraphQL serves as a unified data layer across multiple services. This way you simplify API management and reduce dependencies between teams. It enables efficient data fetching while keeping the API surface flexible and maintainable.

GraphQL is built around a few core components that work together to let clients request exactly the data they need from a single endpoint. Here’s a simple introduction to each main part:

Language: A query language letting clients request exactly the data they need.
Schema: Defines available data types, fields, and operations (queries, mutations, subscriptions).
Server: Processes queries, validates them against the schema, and returns results.
Resolvers: Functions that fetch data for each field in the schema.
Data Sources: Databases, APIs, or services where the actual data lives.

Together, these components allow GraphQL to unify data from multiple sources into a single, flexible API.

Rather than custom endpoints for each composite view, GraphQL exposes a type-based schema that lets clients specify exactly what data they want.
Resolvers act as the composition code—each field in the schema calls the appropriate microservice, then GraphQL merges all results for the client.

GraphQL can be implemented with different languages and frameworks. Spring Boot provides first-class support for GraphQL, making it easy to build type-safe, efficient APIs in Java.

Warp Up

In this article we moved from naive client‑to‑microservice calls to a proper composition layer, and then saw how GraphQL naturally fits as a unified API across our Order and Inventory services. Instead of creating custom endpoints for every page, a strongly‑typed schema and resolvers let clients ask for exactly the data they need from a single endpoint, without leaking internal boundaries.

In the next article, we’ll take this one step further and implement this GraphQL composition layer using spring‑graphql on top of our existing microservices. We’ll design the schema around the “Order Details” use case, wire resolvers to Order and Inventory, and see how to handle cross‑service calls, errors, and performance concerns in a clean, type‑safe way

Simply Order (Part 6) – Making APIs Idempotent: Because Users Double-Click and Networks Lie

Mohamed Hassan — Tue, 04 Nov 2025 07:05:58 +0000

This is the fifth article in our series, where we design a simple order solution for a hypothetical company called Simply Order. The company expects high traffic and needs a resilient, scalable, and distributed order system.

In the previous articles:

We built the core services — Order, Payment, and Inventory — and discussed different approaches for handling distributed transactions across multiple services. Then, we designed and implemented the Saga workflow. Next, we introduced the problem of dual-write consistency and how the Outbox Pattern can solve it.

The Problem

In this article, we’re tackling another critical issue in the inventory service.

Imagine you place an order, and somehow the same item gets reserved twice. Could that happen?

Short answer: YES.

In distributed systems, retries are inevitable. The problem is that your service doesn’t always know whether the first request succeeded or failed. For example, our Saga workflow might retry a reserve order request due to a network delay or a temporary failure — even though the inventory service already reserved the items the first time.

Idempotency Definition

So, what exactly does idempotency mean?

An API is called idempotent when calling it multiple times with the same input produces the same result — or, more precisely, the same server-side effect.

In other words, making the same request several times leaves the system in the same state.

For example:

GET: Fetches data — the server state never changes.
PUT: Updates data — the first call changes the state, and subsequent identical calls keep the server in the same state.

Both are considered idempotent.

However, POST is not idempotent by default. Each call to a POST endpoint typically creates a new entity — so if you retry the same request, you end up with duplicates.

How to Make Our Operation Idempotent

The simplest way to make a non-idempotent API idempotent is to keep track of each request’s unique identifier — typically passed as a header like X-Idempotency-Key.

When the same key is received again, instead of re-executing the operation, we simply return the cached response stored from the first successful execution.

Implementation

The code for this project can be found in this repository:
https://github.com/hassan314159/simply-order

Since this repository is continuously updated, the code specific to this lesson can be found in the add_inventory_idempotency. branch. Start with:

git checkout add_inventory_idempotency

Order Service Changes

The Order Service, through the Saga workflow, calls the Inventory Service to reserve items.

To properly track these requests, we need to attach an idempotency key header that is unique for each order.

This way, if the same RESERVE operation is executed multiple times, the Inventory Service can detect it and avoid processing it again.

We found that the existing sagaId, which is unique for each Saga transaction, is a great candidate for our idempotency key.

Here’s how we add it inside:

dev.simplyoder.order.temporal.activities.OrderActivitiesImpl.java

headers.add("X-Idempotency-Key", sagaId + ":reserve");

The suffix :reserve helps distinguish between different operations (like reserve and release) under the same Saga context

Inventory Service Change

Idempotency Entity

To implement idempotency in the Inventory Service, we need a datastore to keep track of reserve order requests and their corresponding responses.

But what do we actually mean by response?
In an HTTP call, the response mainly consists of two parts:

The two main components for https responses are

HTTP Status – Indicates the outcome of the operation (e.g., 200 for success, 409 for conflict).
Response Payload – The body returned from the service. In our case, it contains the reservationId generated when the inventory was first reserved.

We also keep track of some important fields such as the aggregateId, which in our case represents the orderId, along with the actual idempotency key.

To make the record easily traceable and unique, we generate a deterministic UUID (a UUID derived from the idempotency key and operation).

This ensures that every combination of key and operation always produces the same UUID — which helps us quickly detect duplicate requests without reprocessing them.

Here’s the entity that represents this data:
dev.simplyoder.inventory.persistence.IdempotencyEntity

@Id
private UUID key;
private String aggregateId;
private Integer httpStatus;
@Column(length = 4000)
private String jsonResponse;

private Instant createdAt;
private Instant updatedAt;

Domain Entities

At this stage, we haven’t fully implemented the persistence layer for the Inventory Service, but we already have two main domain entities that define its core logic:

InventoryItem – Stores information about items in stock, including the item SKU, name, description, and the number of units currently available (inStockQty).
ReservationHold – Keeps track of item reservations so we can confirm or release them later, without directly modifying the main inventory table.

Here are the two entities:

InventoryItem

dev.simplyoder.inventory.persistence.InventoryItemEntity

@Id
@GeneratedValue(strategy = GenerationType.IDENTITY)
private Long id;

@Column(unique = true, length = 64, nullable = false)
privte String sku;
@Column(nullable = false, length = 200)
private String name;
@Column(columnDefinition = "text")
private String description;

@Column(nullable = false)
private int inStockQty;
@Column(nullable = false)
private int reservedQty;

@Column(nullable = false)
private Instant updatedAt = Instant.now();

ReservationHold

dev.simplyoder.inventory.persistence.ReservationHoldEntity

@Id
@GeneratedValue(strategy = GenerationType.IDENTITY)
private Long id;

@Column(nullable = false)
private UUID reservationId;

@Column(name="order_id", nullable=false)
private String orderId;

@Column(name="idempotency_key")
private String idempotencyKey;

@Column(nullable = false, length = 64)
private String sku;

@Column(nullable = false)
private int qty;

@Enumerated(EnumType.STRING)
@Column(nullable=false)
private State state; // PENDING | CONFIRMED | CANCELLED

Service Layer

dev.simplyoder.inventory.service.InventoryService

public ReservationsResponse reserve(ReservationsRequest req, String idempotencyKey) {

// create a key based on provided idempotency key from the caller and operation
final UUID rid = ReservationIds.fromKey(idempotencyKey, OP_RESERVE);

Optional<IdempotencyEntity> idempotencyRow = idempotencyRepository.findById(rid);

//1 handle idempotency  
if(idempotencyRow.isPresent()){
    return tryDeserialize(idempotencyRow.get())
            .orElseGet(() -> new ReservationsResponse(rid));
}

//2 Safe check: if domain already created reservation (previous crash)
if (!reservationHoldRepository.findByReservationId(rid).isEmpty()) {
    ReservationsResponse resp = new ReservationsResponse(rid);
    saveIdempotency(rid, req.orderId().toString(), HttpStatus.OK.value(), serializeResponse(resp));
    return resp;
}

try {
    // 3 Perform doman operation in a saperate transaction
    ReservationsResponse resp = reserveDomainTransaction(req, rid, idempotencyKey); // @Transactional
    saveIdempotency(rid, req.orderId().toString(), HttpStatus.OK.value(), serializeResponse(resp)); // REQUIRES_NEW
    return resp;
} catch (InsufficientStockException ex) {
    saveIdempotency(rid, req.orderId().toString(), HttpStatus.CONFLICT.value(), serializeResponse(Collections.EMPTY_MAP));     // REQUIRES_NEW
    throw ex;
}
}

The logic is simple:

Check Idempotency – Verify if a record already exists for the given idempotency key.
Secondary Safeguard – Handle the rare case where an idempotency record was created, but the actual item reservation never happened (for example, due to a crash right after saving the key).
Perform Reservation Logic
1. Check the available stock and deduct the requested amount.
2. Create a Reservation record so that the items can later be confirmed or released.

Note: The idempotency handling and the business operation are executed in two separate transactions, since they are logically independent.

The domain transaction logic is as follows:

@Transactional
public ReservationsResponse reserveDomainTransaction(ReservationsRequest req, UUID rid, String idempotencyKey){
    req.items().forEach(item -> {
        var itemEntity = inventoryItemRepository.findBySku(item.sku()).orElseThrow();
        int available = itemEntity.getInStockQty() - itemEntity.getReservedQty();
        if (item.qty() > available) throw new InsufficientStockException("Insufficient stock for " + item.sku());
        itemEntity.setReservedQty(itemEntity.getReservedQty() + item.qty());
        inventoryItemRepository.save(itemEntity);
    });

    // persist the holds for later release-by-id
    var rows = new ArrayList<ReservationHoldEntity>();
    for (var it : req.items()) {
        var rh = ReservationHoldEntity.reserve(rid, req.orderId().toString(), idempotencyKey, it.sku(), it.qty());
        rows.add(rh);
    }
    reservationHoldRepository.saveAll(rows);
    return new ReservationsResponse(rid);
}

Homework

For now, we’ve secured our reserve endpoint with idempotency.

However, the release operation is still not idempotent.

The same approach can be applied here — simply reuse the existing IdempotencyEntity and implement the same logic for the release endpoint to ensure it behaves consistently.

Wrap Up

We’ve now made our Inventory Service more resilient by introducing idempotency.

It’s a small change in code but a big improvement in reliability.

In distributed systems, retries aren’t optional — they’re part of survival.

By making our endpoints idempotent, we ensure that duplicated network calls or Saga retries don’t cause inconsistent states or double reservations.

Idempotency and the Outbox Pattern (from the previous part) go hand in hand — one protects against duplicated operations, the other against lost events.

Together, they make your system predictable and safe to retry, which is exactly what you want in a distributed environment.

Simply Order (Part 5) — Hands-On: Building the Outbox Pattern for Reliable Event

Mohamed Hassan — Tue, 21 Oct 2025 22:05:30 +0000

In the previous articles:

We built the core services — Order, Payment, and Inventory — and discussed different approaches for handling distributed transactions across multiple services. Then, we designed and implemented the Saga workflow. Finally, we introduced the problem of dual-write consistency and how the Outbox Pattern can solve it.

In this article, we’ll show how to implement the Outbox Pattern with a polling relay using PostgreSQL. Then, we’ll integrate this logic into our Order Service and see it in action.

The code for this project can be found in this repository:
https://github.com/hassan314159/simply-order

Since this repository is continuously updated, the code specific to this lesson can be found in the add_persistence_to_order_service. branch. Start with:

git checkout add_persistence_to_order_service

Outbox Entity Definition

Lets have a look about our OutboxEntity definition. You can find the code in OrderService under the package dev.simplyoder.order.infra.outbox

@Entity
@Table(name = "outbox")
public class OutboxEntity {
    @Id
    private UUID id;

    @Column(name = "event_type", nullable = false, length = 100)
    private String eventType;

    @Column(name = "aggregate_id", nullable = false)
    private UUID aggregateId;

    @Column(name = "payload", columnDefinition = "jsonb", nullable = false)
    @JdbcTypeCode(SqlTypes.JSON)
    private String payload;

    @Enumerated(EnumType.STRING)
    @Column(nullable = false, length = 16)
    private Status status;

    @Column(nullable = false)
    private int attempts;

    @Column(name = "available_at", nullable = false)
    private Instant availableAt;

    @Column(name = "created_at", nullable = false, updatable = false)
    private Instant createdAt;

    @Version
    private long version;

    public enum Status { PENDING, SENT, FAILED }

Where

id: the ID of the outbox record.
eventType: the type of event, e.g., OrderCreated
aggregateId: the ID of the domain object, e.g., OrderId
payload: the event payload, e.g., the created order record
status: the status of the outbox record; one of:
- PENDING: new record to be polled by the relay
- SENT: polled and executed successfully
- FAILED: polled but execution failed (after a number of attempts)
attempts: number of attempts before marking the record as FAILED
createdAt: the creation timestamp of the outbox record
availableAt: when the record becomes eligible for execution. It starts as createdAt (i.e., available immediately) and is updated on failure to support a backoff mechanism—delaying the next attempt by some time.

Outbox Relay Logic

@Scheduled(fixedDelay = 500)
public void orderWorkFlowSchedular() {
    List<OutboxEntity> items = outboxRepo.lockNextBatch(Instant.now(), batchSize); // 1
    for (OutboxEntity ob : items) { // 2
        try {
            String workflowId = "order-" + ob.getAggregateId();

            WorkflowOptions options = WorkflowOptions.newBuilder()
                    .setTaskQueue("order-task-queue")
                    .setWorkflowId(workflowId)
                    .setWorkflowIdReusePolicy(WORKFLOW_ID_REUSE_POLICY_REJECT_DUPLICATE)
                    .build();

            OrderWorkflow wf = temporalClient.newWorkflowStub(OrderWorkflow.class, options);

            wf.placeOrder(ob.getPayload());
            ob.markSent(); // 3

        } catch (WorkflowServiceException e) {
            // Workflow already exists with the same ID? Treat as success (idempotent start).
            if (isAlreadyStarted(e)) { // 4
                ob.markSent();
            } else {
                // backoff
                if (ob.getAttempts() + 1 >= maxAttempts) {
                    ob.fail();
                } else {
                    ob.reschedule(nextBackoff(ob.getAttempts()));  // 5
                }
            }
        } catch (Exception e) {
            if (ob.getAttempts() + 1 >= maxAttempts) {
                ob.fail(); // 6
            } else {
                ob.reschedule(nextBackoff(ob.getAttempts()));
            }
        }
    }
    outboxRepo.saveAll(items);
}

The Outbox Relay is implemented as a simple scheduled task in our Order Service, running every 500 ms using @Scheduled(fixedDelay = 500).
It performs the following steps:
1- Fetches and locks the next batch of outbox records using outboxRepo.lockNextBatch
2- Loops through each record and starts a new transaction or Temporal workflow using the payload and OrderId (aggregateId).
3- Updates the outbox record status to SENT in case of success — ob.markSent();.
4- If a workflow has already been started for the same OrderId, it marks the outbox record as SENT as well — if (isAlreadyStarted(e))
5- Updates the availableAt field in case of failure (but only if the number of attempts has not yet been exceeded).
6- If the number of attempts exceeds the limit, it updates the status to FAILED

lockNextBatch in OutboxRepository — fetches and locks the selected records to ensure that, if multiple service instances or relays are running, each outbox record is processed by only one relay.

To support idempotency (i.e., preventing the same workflow from running twice for the same ID), we configured our workflow with: .setWorkflowIdReusePolicy(WORKFLOW_ID_REUSE_POLICY_REJECT_DUPLICATE)

In our design, the payload contains the full Order record — including both Order and OrderItems. We chose this approach to make the relay self-contained and independent of the order business logic. In fact, the workflow and outbox code could be completely moved out of the Order Service into a standalone service, but for simplicity, we kept it within the Order Service.

Updated Order Logic

Order Entity

We use JPA to persist order entities in our PostgreSQL database.
Our repositories are implemented using Spring Data:

@Repository
public interface OrderRepository extends JpaRepository<OrderEntity, UUID> {
}

This allows us to handle all basic CRUD operations without writing boilerplate code, while still being able to define custom queries when needed.

Here is the OrderEntity class.

@Entity
@Table(name = "orders")
public class OrderEntity{

    @Id
    private UUID id;

    private UUID customerId;

    @Enumerated(EnumType.STRING)
    @Column(nullable = false, length = 40)
    private OrderStatus status = OrderStatus.OPEN;

    @Column(nullable = false, precision = 19, scale = 2)
    private BigDecimal total = BigDecimal.ZERO;

    @OneToMany(mappedBy = "order", cascade = CascadeType.ALL, orphanRemoval = true)
    @JsonManagedReference
    private List<OrderItemEntity> items = new ArrayList<OrderItemEntity>();

    @Column(name = "created_at", nullable = false, updatable = false)
    private OffsetDateTime createdAt;

    @Column(name = "updated_at", nullable = false)
    private OffsetDateTime updatedAt;

Order Service

Right now, the Order Service is much simpler and focused purely on the order domain.

@Transactional
public UUID createOrder(CreateOrderRequest request) throws JsonProcessingException {
    UUID orderId = UUID.randomUUID();
    CreateOrderCommand cmd = CreateOrderCommand.from(orderId, request);
    OrderEntity order = OrderEntity.create(cmd);
    orderRepo.save(order); //1

    String payload = objectMapper.writeValueAsString(order);
    outboxRepo.save(OutboxEntity.pending("OrderCreated", orderId, payload)); //2

    return orderId;
}

The OrderService logic consists of just two operations in a single transaction:
1- Create the Order entity
2- Create the Outboxorder

Run Application

As discussed in Lesson 3,

to run the application

cd ./deploy/local
docker compose up

please follow the same steps mentioned in Lesson 3 to create orders.

Home Work

Try stopping the Temporal service and then create a new order.
After restarting Temporal, you’ll notice that the order gets created and the distributed transaction continues automatically once Temporal becomes healthy again.

Wrapping Up

In this lesson, we implemented the Outbox Pattern with a polling relay using PostgreSQL and integrated it into our Order Service.
We saw how the relay reliably picks up pending events, triggers the corresponding Temporal workflows, and ensures eventual consistency across distributed services — even in cases of temporary failures or service downtime.

Simply Order (Part 4) — Reliable Events with the Outbox Pattern (Concepts)

Mohamed Hassan — Sun, 12 Oct 2025 08:18:45 +0000

This is the fourth article in our series, where we design a simple order solution for a hypothetical company called Simply Order. The company expects high traffic and needs a resilient, scalable, and distributed order system.

In previous lectures:

we built the core services — Order, Payment, and Inventory — and discussed the possible ways of implementing distributed transactions across multiple services. Then we designed and implemented our workflow.

In this and the next lessons, we will add data persistence to our application. We’ll start with the Order Service, using PostgreSQL to persist our orders.

But before diving into creating tables, let’s remind ourselves how the Order Service currently looks:

public UUID createOrder(CreateOrderRequest request)

public UUID createOrder(CreateOrderRequest request){
    UUID orderId = UUID.randomUUID();
    BigDecimal total = BigDecimal.ZERO;
    List<Order.Item> items = new LinkedList<>();
    for (var it : request.items()) {
        total = total.add(it.price().multiply(BigDecimal.valueOf(it.qty())));
        items.add(new Order.Item(it.sku(), it.qty(), it.price()));
    }
    Order order = new Order(orderId, request.customerId(), Order.Status.OPEN, total, items);
    orders.put(orderId, order);

    OrderWorkflow wf = client.newWorkflowStub(
            OrderWorkflow.class,
            WorkflowOptions.newBuilder()
                    .setTaskQueue("order-task-queue")
                    .setWorkflowId("order-" + orderId) // This is saga id
                    .build());

    WorkflowClient.start(wf::placeOrder, OrderWorkflow.Input.from(order, request));
    return orderId;

}

The service creates an 'Order' object and then starts the orchestrated order transaction.
We will now update the part responsible for creating the order to store it in the database instead of in memory.

⚠️ Dual-write inconsistency

Because the database and Temporal broker are independent systems, the operation is not atomic.
This can lead to inconsistent states:

DB Ok, Broker fails: We create an order in the database (status = OPEN), but no workflow transaction is started — other services remain unaware → stale order.
DB fails, Broker OK: The order is not persisted, but a workflow is started → a ghost order exists in the system.

🤔 Using a 2PC (Two-Phase Commit) transaction could solve this, but it’s slow, complex, and impractical for microservices.

Outbox Pattern

When we face such a problem, the Outbox pattern can be our savior.
It follows a very simple idea:

When we create the order in the database, we also create another record in an outbox table corresponding to that order event.
Both records are committed in one transaction — they either succeed or fail together, ensuring consistency.
A separate relay process then polls the outbox table and starts the workflow independently.

This guarantees that the distributed transaction will eventually be started, even if the broker was temporarily unavailable.

Outbox flavors

There are two main ways to implement the Outbox pattern:

Outbox with polling Relay

Flow

1- The application writes the business record (i.e., Order) and the outbox record in one transaction.
2- A poller/relay (scheduled job) scans the outbox table for new records.
3- The relay starts the workflow and marks the record as SENT.

Pros

Simple mechanism with no additional infrastructure layer.

Cons

Polling adds load to the database.
You own the relay, so you must handle retries, backoff, and failures.
The database must support transactions.

Outbox with CDC/Log Tailing

CDC Change Data Capture: A method to detect row-level changes in a database (inserts, updates, deletes) and stream them to other systems in near real time — without touching application code. Often implemented by reading the database’s transaction log.

transaction Log: An append-only record of every database change (insert, update, delete) in exact order, allowing the database to replay, undo, replicate, or stream changes.

Flow

1- Application write business record i.e Order + Outbox record in DB in one Transaction
2- A CDC tool tails the DB’s transaction log and streams outbox inserts to a stream service i.e Kafka.
3- A consumer can listen to changes and start the workflow.

Pros

No polling overhead on the DB.
Very low latency and operationally robust once set up.

Cons

Requires CDC infrastructure (Kafka Connect, Debezium, connectors, monitoring, etc.).

Our Approach

CDC is a great fit when you already have a streaming infrastructure (e.g., Kafka) or when you need near real-time event processing.
But this is not our current case.

So, we’ll implement Outbox with Polling Relay to handle this consistency problem between the database and the Temporal workflow.

Wrap-up

In this lesson, we explored one of the most important reliability patterns in distributed systems — the Outbox pattern.
We started with the dual-write problem and saw why direct DB + broker writes can lead to inconsistent states.
Then, we learned how the Outbox pattern guarantees atomicity and reliability by coupling data persistence with event publishing — either via a polling relay or CDC/log tailing approach.

In our Simply Order project, we’ll go with the polling relay implementation, since it’s simpler and requires no additional infrastructure.

➡️ In the next lesson, we’ll dive into the actual implementation:

Create the outbox table
Persist events along with orders in one transaction
Build a simple polling relay to trigger the Temporal workflow reliably

Stay tuned — that’s where we’ll make reliability real. 💪

Simply Order (Part 3) — Linking It All Together: Connecting Services and Watching Temporal in Action

Mohamed Hassan — Sun, 28 Sep 2025 17:32:04 +0000

This is the third article in our series, where we design a simple order solution for a hypothetical company called “Simply Order”. The company expects high traffic and needs a resilient, scalable, and distributed order system.

In previous lecture Designing and Implementing the Saga Workflow, we designed and implemented the Temporal workflow. In this lecture, we’ll build the skeleton for the three participating services, connect them together, and run the application using Docker Compose.

We’ll start with in-memory data only. In the next lecture, we’ll add a persistence layer, discuss the challenges that come with it, and explore possible solutions and their trade-offs. So stay tuned—let’s dive into the code for our services:

The code for this project can be found in this repository:
https://github.com/hassan314159/simply-order

Since this repository is continuously updated, the code specific to this lesson can be found in the branch connecting_and_running_app. Start with:

git checkout connecting_and_running_app

Order Service

The Order Service is our entry point—the place where we create orders and start the workflow.

Controller

@PostMapping
public ResponseEntity<CreateOrderResponse> create(@RequestBody CreateOrderRequest req) {
    UUID id = orderService.createOrder(req);
    return ResponseEntity.status(201).body(new CreateOrderResponse(id, orderService.findOrderById(id).getStatus().name()));
}

@GetMapping("/{id}")
public ResponseEntity<?> get(@PathVariable UUID id) {
    return orderService.findOrderById(id) != null ? ResponseEntity.ok(orderService.findOrderById(id)) :
           ResponseEntity.notFound().build();
}

It exposes two endpoints: one to create an order and another to retrieve an order with its status.

Service

When a new order is created, the service builds an order object with status OPEN and stores it in a Map that acts as our in-memory database. It then starts the workflow code we defined in the previous lecture, as shown below:

 public UUID createOrder(CreateOrderRequest request){
    ...

    Order order = new Order(orderId, request.customerId(), Order.Status.OPEN, total, items);
    // simple map acts as in in-memory store
    orders.put(orderId, order);

    // start oder creation saga ** WHEN INTRODUCE DB WILL BE UPDATED TO BE STARTED BY OUTBOX RELAY
    OrderWorkflow wf = client.newWorkflowStub(
            OrderWorkflow.class,
            WorkflowOptions.newBuilder()
                    .setTaskQueue("order-task-queue")
                    .setWorkflowId("order-" + orderId) // This is saga id
                    .build());

    WorkflowClient.start(wf::placeOrder, OrderWorkflow.Input.from(order, request));
    return orderId;

}

Inventory Service

The Inventory Service is very simple—it accepts a request and returns an OK response so the Saga can continue.

We added a temporary condition: if the number of items is greater than 3, the service responds with 209, indicating that the inventory does not have enough items. This allows our Saga to roll back, i.e., apply the compensation operations.

But what if the service is unreachable? Will the Saga fail on its request?
Short answer: No. This depends on the activity retry configuration we defined in the previous lecture.

We’ve also provided an endpoint that Temporal workers will call to apply the compensation. /reservations/{reservationsId}/release

@PostMapping("/reservations")
public ResponseEntity<ReservationsResponse> create(@RequestBody ReservationsRequest req) {
    if(req.items().size() < 3){
        LOG.info("Items reserved successfully");
        return ResponseEntity.ok(new ReservationsResponse(UUID.randomUUID()));
    }else {
        LOG.info("Items could not be reserved");
        return ResponseEntity.status(209).build();
    }
}

@PostMapping("/reservations/{reservationsId}/release")
public ResponseEntity<?> get(@PathVariable UUID reservationsId) {
    LOG.info("Items released for reservation: {}", reservationsId);
    return ResponseEntity.noContent().build();
}

Payment Service

Like the Inventory Service, the Payment Service authorizes requests with trivial logic: if the order cost is less than 500, it authorizes the payment; if it’s 500 or greater, it rejects it and returns 209 as a rollback signal so the Saga can apply compensation.

@PostMapping("/authorize")
public ResponseEntity<PaymentAuthorizeResponse> create(@RequestBody PaymentAuthorizeRequest req) {
    if(req.amount().compareTo(BigDecimal.valueOf(500)) < 0){
        LOG.info("Payment processed successfully");
        return ResponseEntity.ok(new PaymentAuthorizeResponse(UUID.randomUUID(), false));
    }else {
        LOG.info("Payment Failed");
        return ResponseEntity.status(209).build();
    }
}

@PostMapping("/{paymentId}/void")
public ResponseEntity<?> get(@PathVariable UUID authId) {
    LOG.info("Payment refunded: {}", authId);
    return ResponseEntity.noContent().build();
}

Run the application

We’ve provided a Docker Compose setup that builds and starts the services, Temporal Server, and Temporal UI. Temporal UI is a powerful tool that allows us to trace the progress of our workflows.

If Docker Compose isn’t installed in your environment, install it first. Then, from the repository’s root directory, navigate to the folder containing the Docker Compose file and run the services:

cd ./deploy/local
docker compose up

To check the running services, just run docker-compose ps. You should see something like:

First Order Creation

One of the best auxiliary features of Temporal is the Temporal UI — an interface that lets us trace our workflows.

Let’s open this URL in our browser: localhost:8080

For now, no workflows have been created.
Let’s start a transaction and see how we can trace it.

Let’s create our first order:

curl --location 'localhost:8081/orders' \
--header 'Content-Type: application/json' \
--data '{
    "customerId": "123e4567-e89b-12d3-a456-426614174000",
    "items": [
      {
        "sku": "ABC-001",
        "qty": 1,
        "price": 19.99
      },
      {
        "sku": "XYZ-123",
        "qty": 1,
        "price": 12.95
      }
    ]
  }'

The response should look like this:

{
    "orderId": "71ed318d-10ea-4371-af34-1581dc315dde",
    "status": "OPEN"
}

We’ve created an order, and its status is OPEN. Let’s check its status for now:

Let’s call:

curl --location 'localhost:8081/orders/71ed318d-10ea-4371-af34-1581dc315dde'

Here, 71ed318d-10ea-4371-af34-1581dc315dde is the order ID you got when creating the order. You should use the UUID generated by your own code.

{
    "id": "71ed318d-10ea-4371-af34-1581dc315dde",
    "customerId": "123e4567-e89b-12d3-a456-426614174000",
    "status": "COMPLETED",
    "total": 32.94,
    "items": [
        {
            "sku": "ABC-001",
            "qty": 1,
            "price": 19.99
        },
        {
            "sku": "XYZ-123",
            "qty": 1,
            "price": 12.95
        }
    ]
}

We can see that the status is Completed — but what happened?
This is where the Temporal UI comes in. Let’s open its URL again localhost:8080 and click on the workflow.
You can identify it as order-<UUID>

Now we can see the timeline of our order’s workflow, as shown here:

One Service is down

Let’s assume one service is down — for example, the payment service. How can Temporal handle this?

In our branch, we’ve updated the retry interval of the workflow activity to 200 seconds so we can debug the workflow while the service is down.

var retry = RetryOptions.newBuilder().setMaximumAttempts(5).setBackoffCoefficient(2.0).setInitialInterval(Duration.ofSeconds(200)).build();

To simulate a service being down, just stop the payment service.

docker compose stop payment-service

Now, let’s create another order and check its status:

curl --location 'localhost:8081/orders' \
--header 'Content-Type: application/json' \
--data '{
    "customerId": "123e4567-e89b-12d3-a456-426614174000",
    "items": [
      {
        "sku": "ABC-001",
        "qty": 1,
        "price": 19.99
      },
      {
        "sku": "XYZ-123",
        "qty": 1,
        "price": 12.95
      }
    ]
  }'

Response: 
{
    "orderId": "9cd35ac0-6973-4881-b68d-a37dbb1a2bff",
    "status": "OPEN"
}

Let’s check the status of the order right now:

curl --location 'localhost:8081/orders/9cd35ac0-6973-4881-b68d-a37dbb1a2bff'

Response
{
    "id": "9cd35ac0-6973-4881-b68d-a37dbb1a2bff",
    "customerId": "123e4567-e89b-12d3-a456-426614174000",
    "status": "INVENTORY_RESERVED",
    "total": 32.94,
    "items": [
        {
            "sku": "ABC-001",
            "qty": 1,
            "price": 19.99
        },
        {
            "sku": "XYZ-123",
            "qty": 1,
            "price": 12.95
        }
    ]

We can see that the status of the order is INVENTORY_RESERVED, which is expected. Since the payment service is down, Temporal activities are retrying calls to the payment service. According to our configuration, if the service becomes healthy again before 5 retries, our workflow can continue.

If we check the Temporal UI, we’ll see its status as Running:

Let’s start the service again:

docker compose start payment-service

Let’s check the status.

curl --location 'localhost:8081/orders/9cd35ac0-6973-4881-b68d-a37dbb1a2bff'

Response
{
    "id": "9cd35ac0-6973-4881-b68d-a37dbb1a2bff",
    "customerId": "123e4567-e89b-12d3-a456-426614174000",
    "status": "COMPLETED",
    "total": 32.94,
    "items": [
        {
            "sku": "ABC-001",
            "qty": 1,
            "price": 19.99
        },
        {
            "sku": "XYZ-123",
            "qty": 1,
            "price": 12.95
        }
    ]
}

now it is COMPLETED — whoa! 🎉

To trace the timeline of what happened, we can check the Temporal UI:

From the timeline above, we can see when the service was down (light green) and when it became healthy again and succeeded.

Home Work

Try stopping the payment service and creating an order, but this time wait longer — until the order workflow fails — and then check the status…

Wrapping Up

In this lesson, we continued our example by running the app and exploring how Temporal helps us trace and recover workflows — even when a service goes down. We saw how the Temporal UI gives us full visibility into the workflow timeline and retries.

In the next lesson, we’ll take our example one step further and add persistence, so our workflows can survive restarts and keep their state safely. 🚀

Simply Order (Part 2) — Designing and Implementing the Saga Workflow with Temporal

Mohamed Hassan — Thu, 04 Sep 2025 18:09:40 +0000

This is the second lesson in our series. In the first lesson, we defined the problem of distributed microservices in our hypothetical company Simply Order. We explored possible solutions for handling distributed transactions in microservices, and discussed why Two-Phase Commit (2PC) is not suitable for microservices and why the Saga pattern can solve these problems.

In this lesson, we move from concepts to implementation. We will design and implement the first version of our Saga workflow using Temporal as the orchestrator.

Before we code: How to think about a saga?

1) Think about the business transaction (i.e. Place Order)

The transaction will be as follows: create order → reserve inventory → authorize payment → completed

2) Draw the state machine of our transaction

To be able to implement the transaction, we need to draw the state machine — i.e. all the states that our entity will go through.

In our case, we can think of an order with these states:
OPEN-> PENDING -> INVENTORY_RESERVED -> PAYMENT_AUTHORIZED -> COMPLETED
In case of errors, the status will be one of these:
INVENTORY_FAILED, PAYMENT_FAILED, FAILED_UNKNOWN

3) Why Orchestration Saga?

⚠️ By default, we should consider Choreography because it is simpler: services just emit and listen to events. This works well for small, linear flows with few participants (see Chris Richardson, Microservices Patterns, and Sam Newman, Building Microservices).

But as soon as the flow becomes longer, more complex, or involves branching, choreography can easily turn into a spaghetti of events, making it hard to trace and evolve the process.

In the initial phase of our order service — which is relatively simple — choreography could be a good choice. But with our planned extension of the order service (as we’ll see in the next lessons), we opt for Orchestration Saga.

We could implement our own saga orchestrator, but then we would need to handle workflow state, communication between the orchestrator and microservices, reliability, non-blocking execution, and visibility into workflow progress. All of this distracts us from focusing on business logic, which is what we as developers should prioritize.

Here is where Temporal comes in as a solution…

What is Temporal

In a nutshell, Temporal is an open-source workflow orchestration platform that lets you write reliable, long-running processes in code. It handles retries, state, timeouts, and compensations automatically, so you can focus on business logic instead of building reliability infrastructure.

👉 Note: Temporal is not limited to sagas — it’s a general-purpose workflow platform. Saga orchestration is just one of many patterns you can implement with it.

In this lesson, we will focus on how to configure Temporal in our code and explore its core components.

Dive to Implmentation

The code for this project can be found in this repository:
https://github.com/hassan314159/simply-order

Since this repository is continuously updated, the code specific to this lesson can be found in the branch initial_order_saga. Start with:

git checkout initial_order_saga

Order Workflow definition

Let’s jump to the implementation of our Saga workflow in OrderWorkflowImpl.java.

    public void placeOrder(Input in) {
        String sagaId = Workflow.getInfo().getWorkflowId();
        Saga saga = new Saga(new Saga.Options.Builder()
                .setContinueWithError(true) // collect/continue if a compensation fails
                .build());
        try {
            // 1) Update status of Order to PENDING
            act.updateOrderStatus(in.orderId(), Order.Status.PENDING);

            // 2) reserve the inventory (calling inventory service)
            UUID reservationId = act.reserveInventory(in.orderId(), sagaId, in.request().items());
            saga.addCompensation(() -> act.releaseInventoryIfAny(in.orderId(), reservationId));

            // 3) Update status of Order to INVENTORY_RESERVED
            act.updateOrderStatus(in.orderId(), Order.Status.INVENTORY_RESERVED);

            // 4) authorize the payment (calling payment service)
            UUID authId = act.authorizePayment(in.orderId(), sagaId, in.total());
            saga.addCompensation(() -> act.voidPaymentIfAny(in.orderId(), authId));

            // 5) Update status of Order to PAYMENT_AUTHORIZED
            act.updateOrderStatus(in.orderId(), Order.Status.PAYMENT_AUTHORIZED);

            // 6) Complete our saga by updating Order Status to COMPLETED
            act.updateOrderStatus(in.orderId(), Order.Status.COMPLETED);
        } catch (Exception e) {
            try {
                saga.compensate();     // runs: voidPayment → releaseInventory
            } finally {
                act.updateOrderStatus(in.orderId(), classifyError(e)); // precise status
            }
            throw e;
        }
    }

One of the great features of Temporal is the separation of concerns. In this class, we define our saga flow independently of how the steps are implemented:

Update the order status to PENDING
Call the Inventory service to reserve inventory, and register a compensation step in case an error happens.
Update the order status to INVENTORY_RESERVED
Call the Payment service to authorize the payment, and again register a compensation step.
Update the order status to PAYMENT_AUTHORIZED
Finally, complete the saga by updating the order status to COMPLETED

Defining the workflow steps and compensation logic is straightforward.
We define the steps in the required business order. If any step needs compensation, we register the compensation after the step succeeds, so that if an error occurs later, compensation runs only for the steps that actually completed before the exception point.

Now let’s look at a snippet of the reserveInventory implementation from OrderActivitiesImpl.java.

var req = new Req(orderId, items.stream().map(i -> new Item(i.sku(), i.qty())).toList());
        var headers = new HttpHeaders();
        headers.add("Idempotency-Key", sagaId + ":reserve");
        var res = http.exchange(URI.create(inventoryBase + "/inventory/reservations"), HttpMethod.POST, new HttpEntity<>(req, headers), Res.class);

        if (res.getStatusCode().value() == 209) {
            throw ApplicationFailure.newNonRetryableFailure("" , "InventoryNotSufficient");
        }

The code may look like it’s blocking — just a synchronous HTTP call to the Inventory service.
However, behind the scenes, each activity (act.) is executed by a Temporal Worker, which is a process that runs in a separate worker threads. This means it’s non-blocking from the workflow’s perspective.

We can also define timeout and retry behavior, as shown in our constructor.

 var retry = RetryOptions.newBuilder().setMaximumAttempts(5).setBackoffCoefficient(2.0).setInitialInterval(Duration.ofSeconds(1)).build();
        var opts = ActivityOptions.newBuilder().setStartToCloseTimeout(Duration.ofMinutes(2)).setRetryOptions(retry).build();

Here, we defined a single config for all activities, but in practice, each activity type can have its own configuration.

For example, in this lesson we set a timeout of 2 minutes. In reality, Temporal allows activities to run indefinitely with automatic retries until they succeed — unless you explicitly configure a timeout or retry policy.

Quick Overview of Temporal Core Components

Before moving on, let’s quickly recap the core components of Temporal (we’ll dive deeper into how they interact in a following lesson):

Workflow code: Your orchestration logic (deterministic, no external I/O).
Activity code: Your side effects (call services/DBs/APIs).
Worker: The process that runs workflow code and activity code. It polls a Task Queue.
Temporal Server: Stores the event history, hands out tasks, tracks retries/timeouts.
Task Queue: A named queue that decouples the server from workers.
Client: Starts workflows, sends Signals, reads Queries.
Persistence: DB/storage where the server saves workflow history/state.

Continue Implementation...

Let’s go one step back and see how we can set up our project with Temporal and define its core components.

First, we need to import the Temporal SDK dependency:

<dependency>
    <groupId>io.temporal</groupId>
    <artifactId>temporal-sdk</artifactId>
    <version>${temporal.version}</version>
</dependency>

Then, let’s define the Client and Worker beans. You can find this in TemporalConfig.java.

 @Bean
    WorkflowServiceStubs service(@Value("${app.temporal.address}") String address) {
        return WorkflowServiceStubs.newServiceStubs(
                io.temporal.serviceclient.WorkflowServiceStubsOptions.newBuilder()
                        .setTarget(address).build());
    }

    @Bean
    WorkflowClient workflowClient(WorkflowServiceStubs service) {
        return WorkflowClient.newInstance(service);
    }

    @Bean
    WorkerFactory workerFactory(WorkflowClient client) {
        return WorkerFactory.newInstance(client);
    }

    @Bean
    Worker worker(WorkerFactory factory, OrderActivities activities) {
        Worker w = factory.newWorker("order-task-queue");
        w.registerWorkflowImplementationTypes(OrderWorkflowImpl.class);
        w.registerActivitiesImplementations(activities);
        factory.start();
        return w;
    }

We’ve seen the workflow code, but how does our app identify it as a Temporal Workflow?
Temporal provides two annotations: @WorkflowInterface and @WorkflowMethod.
We already saw the implementation of the placeOrder method.

@WorkflowInterface
public interface OrderWorkflow {
    record Input(UUID orderId, UUID customerId, BigDecimal total, CreateOrderRequest request) {
        public static Input from(Order order, CreateOrderRequest req) {
            return new Input(order.getId(), order.getCustomerId(), order.getTotal(), req);
        }
    }

    @WorkflowMethod
    void placeOrder(Input input);
}

Lastly, Temporal also needs to know the interface that defines the activities, so that:

The Worker can register those methods as activities.
The Server can schedule activity tasks and match them to the correct code running in the Worker.

and this is done simple by annotate the interface with @ActivityInterface as in OrderActivities.java

@ActivityInterface
public interface OrderActivities {
    void updateOrderStatus(UUID orderId, Order.Status status);

    UUID reserveInventory(UUID orderId, String sagaId, List<CreateOrderRequest.Item> items);

    UUID authorizePayment(UUID orderId, String sagaId, BigDecimal total);

    void voidPaymentIfAny(UUID orderId, UUID paymentId);

    void releaseInventoryIfAny(UUID orderId, UUID reservationId);
}

Wrapping up

rapping up

In this lesson, we turned the theory of Sagas into a working Order Workflow with Temporal — defining states, compensations, and error handling. This is just our first step, and there’s more to come as we add cancellation, shipment, and other real-world features.

👉 Follow along for the next parts of the series, and feel free to drop your questions or share your own Saga experiences in the comments — I’d love to hear your thoughts!

Simply Order (Part 1) Distributed Transactions in Microservices: Why 2PC Doesn’t Fit and How Sagas Help

Mohamed Hassan — Wed, 27 Aug 2025 18:40:25 +0000

This is the first article in a series that walks through a simple order solution for a hypothetical company called “Simply Order”. The company expects high traffic and needs to design a resilient, scalable, and distributed Order system.

We’ll go step by step, building the system and discussing the challenges along the way, while exploring the trade-offs between possible solutions.

In this first article, we’ll describe the problem and review different approaches — with their pros and cons — before choosing our recommended path. In the upcoming articles, we’ll implement the solution, share code examples, and highlight best practices.

The Problem

The company chose a microservice architecture to build their system. They started with an Order microservice, but soon realized it was becoming a bottleneck.

This service has to handle multiple responsibilities:
Accept orders and store them.

Communicate with the Inventory Service to check product availability.
Communicate with the Payment Service to authenticate and process user payments.
Communicate with the Fulfillment Service to start shipping and logistics.
Finally, notify users and other interested participants through different channels — e.g., messages, emails, and so on.

These multiple operations should be consistent and ideally treated as a single unit of work. But here’s the problem:

Each microservice has its own DB → can’t use a single DB transaction across them.
One service may commit successfully, while others fail. This means one service may commit successfully, while others fail.
Some services may even use NoSQL databases, which don’t support traditional transactions at all.
And to make it worse, some microservices might be unreachable at the time of the operation.
So the big question is: how do we manage these operations and keep the system consistent in the chaotic world of microservices?

Solutions

Two-Phase Commit (2PC)

A straightforward solution that often comes to mind is Two-Phase Commit (2PC).

Some relational databases and drivers support it, making it possible to coordinate a distributed transaction.

Here’s how it works:

A coordinator talks to the participants (other microservices or their databases) and asks them to prepare their transaction.
If all participants reply with yes (prepared), the coordinator sends a commit message to everyone.
If one or more reply with no (or fail to respond), the coordinator sends a rollback message to all participants.

Pros & Cons

Pros

Guarantees atomicity across multiple services.
Conceptually simple — feels like a “distributed version” of a database transaction.

Cons

Blocking: the prepare phase locks database resources until the final commit/rollback, which hurts scalability. Also, if any participant is down or unresponsive, the whole transaction may block and hold locks.
Single point of failure: if the coordinator crashes, participants may stay locked indefinitely.
Limited Support: Not all databases or drivers support XA/2PC transactions (especially many NoSQL or cloud-native DBs).

Because of these limitations, 2PC is generally not suitable for microservices in a cloud-native environment.

Saga

Saga splits one big transaction that involves multiple services into a sequence of local transactions owned by each service. If one service fails to commit its transaction, other services get informed about that failure, and instead of rollback (services do not block, they already committed their changes), they perform a compensating transaction. i.e. if the Inventory service deducted/reserved certain items, it will create another transaction to add/free the deducted items.

But how can each service be informed about other services?

There are 2 types of Saga: Choreography and Orchestration

Choreography (event-driven)

Services emit domain events, i.e. OrderCreated, InventoryReserved, PaymentFailed, etc. Other interested services subscribe to these events and can create a compensating transaction if needed. i.e. when a PaymentFailed event is published, the Inventory service creates a compensating transaction to free reserved items, the Order service updates its order status to failed, and so on.

Pros

Natural for event-driven systems.
Simple to start and each service is decoupled.

Cons

Harder to manage complex flows, timeouts, and branching.
Risk of spaghetti events: imagine if just 5 services are involved, and each service could listen to the others!

Orchestration (command-driven)

A central orchestrator sends commands to other services and waits/listens for results in a non-blocking manner.
The workflow is explicit code from domain logic, centralized to manage retries, timeouts, compensations, etc.

Pros

One source of truth, i.e. single place for flow, retries, and timeouts.
Clear visibility even with complex processes.

Cons

The orchestrator could be a single point of failure, so it is a critical component.
Slightly tighter coupling with the orchestrator API, as typically it is a 3rd-party library/component.

Saga in both modes shares some common pros and cons.

Pros

Saga fits microservices naturally as each microservice is loosely coupled from the others.
Saga is non-blocking by the nature of its communication, which is critical for the scalability of microservices.

Cons

Eventual consistency: the system is temporarily inconsistent until all steps (or compensations) finish.
Complex compensation logic: “undoing” business actions is not always straightforward.
Increased complexity: i.e. handling idempotency and compensating logic.

Wrapping up

In this article, we looked at the problem of distributed transactions, why 2PC is not suitable for microservices, and how the Saga pattern (choreography and orchestration) helps solve it.

In the next article, we’ll start implementing the solution using Temporal as an orchestrator-based Saga. We’ll walk through code examples and best practices step by step.

👉 Follow this series if you’re interested in building resilient microservices, and feel free to share your thoughts or questions in the comments — I’d love to hear your perspective.