Forem: Emily Fortuna

To Choreograph or Orchestrate your Saga, that is the question.

Emily Fortuna — Wed, 12 Jul 2023 17:46:50 +0000

The saga pattern is a distributed systems design pattern for a task that spans machine or microservice boundaries in which full execution of all steps is necessary. Partial execution is not desirable. A common life example used to explain when the saga pattern is useful is trip planning. If you’re planning on attending Replay, for example, you’d need to book a conference ticket, an airplane ticket, and a hotel. If you fail to acquire one of these things, you’ll miss out on meeting fun people in backend engineering face-to-face.

Below the surface, there are two main ways microservices can talk to one another that make your saga possible: choreography and orchestration.

Choreography

Choreography is analogous to ants in an ant colony. Like ants, each microservice has local knowledge, and shares information about state changes with other services via chemical signals called pheromones–I mean via message passing. Just as an ant trail to food emerges organically from pheromones, the overall behavior of a system as a whole that contains choreographed microservices emerges organically from each microservice’s instructions.

One tenant drilled into every software engineer’s head is the value of decoupling. Choreography embodies this idea and is straightforward to implement as a whole. Choreography can be a popular, easy choice for systems that are incrementally moving from a monolith to a microservices architecture. However, if you have any sort of ordering requirement of tasks, such as ordered steps in your saga, choreography can get unwieldy fairly quickly. Suppose we want to book the plane first so that the hotel can know your flight number and pick you up from the airport. Then we book our conference ticket (maybe there’s a discount with certain hotels). The sequence of messages that each service responded to would look like this:

However, just from looking at each microservice’s individual codebase, it’s difficult to understand the order that the system should have since that ordering is distributed throughout the code. This leads to all sorts of higher level business logic diagrams that need to be kept in sync with the code…but wouldn’t it be better if the code were just easier to read in the first place? It also can be difficult to debug the exact sequence of events that lead to a bug since control flow is not immediately clear. So, unless all of your microservices are truly independent of one another and don’t have any sort of “happens before” logic, consider using orchestration instead.

Orchestration

Orchestration, on the other hand, is like an air traffic control tower directing planes, or microservices. One service, a “super microservice” if you will, functions as the message broker sending messages directly to individual microservices telling them what to do, just like planes wait for permission to take off.

Because orchestration centralizes control flow, debugging and understanding control flow is much simpler. Additionally, since each step doesn’t need to keep track of what “happens before” messages it needs to listen to, the code for individual microservices is much simpler. Orchestration also shines in situations where many services need to interact in a single saga step. The glaring Achilles’ heel of this method is that bane of all distributed systems: the message broker is a single point of failure.

Putting it all together

So to summarize, choreography:

Is decentralized and decoupled
Is good for highly independent microservices
Is “easier” to implement, at least initially
Is an easy choice for converting established monoliths to microservices
Can make control flow unclear
Can be challenging to debug

and orchestration:

Has one service issuing “commands” to execute microservices
Makes control flow easier to understand
Easier to build with greenfield applications
Makes debugging and failure handling clearer
Is “harder” to implement initially, but pays dividends later
Has a single point of failure (the message broker)

The interesting tradeoff between these two approaches is one wants to reach for the light, agile option (choreography) in the early days and avoid over-architecting your project, but counterintuitively, orchestration is often easier to build when one uses it from the start.

So, what does Temporal do?

Temporal uses orchestration under the hood (you won’t have to implement it yourself), but also avoids that crucial drawback of a single point of failure. How is such a thing possible? Internally, Temporal records your program’s progress in a log. If that message broker were to go offline, your entire program’s history will have been saved and so that another machine can start up exactly where your program left off, as if nothing happened. This makes Temporal completely horizontally scalable.

To bring this idea back to the saga pattern, an important component of the saga pattern is driving towards completion of all the steps of the saga. The fact that Temporal ensures no progress will ever be lost means it will pick up exactly where it left off “no matter what”, including failures for an unknown length of time, completing the saga with no extra code or heavy lifting on your part.

Additionally, unlike some orchestration engines, in Temporal, the logic of your workflow is expressed entirely in code, so you don’t have to deal with json. In essence, nothing additional is needed to make a robust, failure resilient application other than the business logic of your application itself.

Conclusion

Choreography and orchestration provide different approaches to coordinating communication between microservices. Choreography is decoupled but can make debugging and control flow difficult to follow. Orchestration is centralized, but results in a single point of failure. Temporal uses orchestration under the covers, but by design safeguards against a single point of failure, allowing you to focus on writing your code with the confidence that it is failure resilient.

25 Key Terms for Speaking Distributed Systems and Temporal (an emoji-based guide)

Emily Fortuna — Thu, 29 Jun 2023 18:55:20 +0000

So you want to keep up with all the cool kids throwing around terms like “multi-cluster replication” but you don’t have time to read several textbooks. This handy quick-reference will give you the framework for following (and participating in!) conversations involving distributed systems or Temporal with ease. At the next dinner party you’ll be able to win friends and influence people with your ability to explain distributed systems succinctly in plain English… because we all know you’re not gonna do it with salad. This guide builds upon itself, with terms requiring no additional context first.

Core Distributed Systems Terms To Know

concurrency ↠

Roughly, the idea of running multiple things at once. Two people eating dinner at the same time (“in parallel”) are eating concurrently. Your operating system context switching between a web browser and IDE is also a form of concurrency.

scalability 📈

The ability for a system, such as a website, to accommodate a growing number of requests or work. You can improve scalability by finding places where work can be executed simultaneously or removing performance bottlenecks.

reliability ✅

The likelihood of a system to run without failure for a period of time. Systems can be made more reliable by reducing single points of failure, and detecting failures quickly.

eventual consistency 🐌

Let’s say you’ve replicated a database to improve reliability (and possibly scalability). Great! Eventual consistency says a change in the data in one location will eventually be updated in every location that the database lives; however, until every location is updated, a read from one of the locations may not yet have the updated value. You, the programmer, need to bear in mind you may not always have the most up-to-date data when working under this model.

strong consistency 💪

The guarantee that a data store will always provide the most up-to-date value.

CAP Theorem 🧢

The rule that you gotta pick two of the three: (strong) consistency, availability, partition tolerance. Any distributed data store can only provide at most any two of these qualities, alas. Availability is defined as every request returns a non-error response. Partition tolerance is the ability for a system to continue to operate despite requests between data store nodes being delayed or dropped. See also strong consistency and eventual consistency.

ACID 🧪

Hardcore-sounding acronym borrowed from databases that stands for atomicity, consistency, isolation, durability. See strong consistency, eventual consistency, and other deets below.

atomicity ⚛️

Executing a sequence of operations all together as if they were a single unit, or not at all.

isolation 📦

Executing a sequence of operations concurrently with another sequence has the same effect as executing each operation sequentially.

durability 🗿

Think long-lasting. Standing the test of time. Persisting—i.e. written to disk, or if you were really hardcore, etched on a stone tablet—at which point it can be looked up even in the face of system failure such as a power outage or crash.

durable execution 🔜

Similar to durability, once a program has started executing, it will continue executing to completion. Persisting every step the program takes so that execution can be continued by another process if the current process dies.

idempotent function 🥪

Scary-sounding word, less scary meaning: a function that has the same observed result when called with the same inputs, whether it is called one time or many times.

A function setting some field foo=3? Idempotent. The function foo += 3? Not idempotent, because the value of foo is dependent on the number of times your function is called. Naive implementations of functions that transfer money or send emails are also not idempotent by default.

deterministic function 🧮

Code that always has the same effect/output when given a particular input. Things that are not deterministic use some external state such as user input, a random number, or stored data. Code that reads or writes to a variable that other code can also modify simultaneously is also not deterministic.

platform 💻

Windows, iOS, Docker, and VMware are all platforms. They’re execution environments that define how programs behave inside them. Temporal is also a platform, which defines that code run with Temporal is failure and timeout resilient. You may see the term platform-level used in relation to failures. Platform-level failures are caused by low-level issues such as network errors or process crashes.

application 〉

The code you write. You may see the term application-level used in relation to failures. Application-level failures are domain-specific failures like “insufficient inventory”, or “user canceled ride request.”

event sourcing 🎤

A design pattern that creates event objects for every state change in a system, and records this sequence of events in a log (or event history). Temporal uses event sourcing “under the hood” to ensure failure resilience.

Temporal-Specific Terms

Temporal ✧

A way to run your code, a service and library that work together, that ensures your code never gets stuck in failure at the application-level. or the platform-level. While libraries like async-retry take care of retry logic for functions that fail, what happens if your code making that library call crashes? Temporal says “we gotchu.” It abstracts away complex concepts around retries, rollbacks, queues, state machines, and timers, so that no matter where the failure happens, we’ll ensure your code keeps running the way you want.

Worker 👷

The process that’s actually doing the work executing all of your Temporal code (the Workflow and Activities). Capitalized here to denote the Temporal-specific concept of a Worker, to differentiate from the generic idea of a worker process.

Workflow 📖

The high-level business logic of your program. Essentially, this is where the logic of your application begins. (Technically execution starts with the Worker, and the Worker runs the Workflow code.) All Workflow logic must be deterministic.

Activity 💾

Components of your Workflow that might fail, like network or file system calls, inventory holds, or credit card charges. The decision around how many Activities your program should have–whether you make a separate Activity for every non-deterministic call or put the entire rest of your program in an Activity (don’t do that)–is generally a function of how you’d like your program to behave when retrying a failure. For example, if a downstream instruction should always grab the very freshest data when retrying, those instructions should be grouped together in a single Activity. If you can retry with the old data, they can be in separate Activities. Since Activities can be retried, they should be idempotent.

Query 🙋

A way to inspect the state of a Workflow. The results are guaranteed to show the most recent state.

Signal 🧑‍🏫

A way to notify or send information to a Workflow. A common use case is notifying a Workflow that the user added items to their shopping cart.

retry 🔄

Generally, re-executing an Activity that has failed. Technically, Workflows can also be retried, but they are far less common, such as a developer attempting to update Workflow code running in production.

Cluster 🏘️

The collection of services and databases that make failure and timeout resilience possible. You might sometimes see this colloquially called the Temporal Server.

History 🗃️

A log of events that happened over the course of execution. This log contains attempts to run Activities, Workflow status changes (started, failed, scheduled, etc), timer events, and external information signaled to the system during the run.

In Closing

Knowing this core set of 25 terms should give you sufficient lay-of-the-land to sling references like ACID and Workflow in conversations with coworkers, friends, and family with ease! Better yet, you now know enough to dive deeper into subdomains of interest. If you’d like to try out these terms in practice, check out our getting started guides, courses, and examples in Go, Java, Python, PHP, and TypeScript.

Saga Pattern Made Easy

Emily Fortuna — Wed, 24 May 2023 18:13:00 +0000

This is part 2 of a series. To read about compensating actions, check out Compensating Actions, Part of a Complete Breakfast with Sagas.

Trip planning with sagas but without the baggage

So the last time your family went to the local park everyone was talking about the Saga Design Pattern, and now you want to know what it is, if you should make it part of your own distributed system design, and how to implement it. As we all know, software design is all about fashionable¹ trends².

The case for sagas

If you’re wondering if the saga pattern is right for your scenario, ask yourself: does your logic involve multiple steps, some of which span machines, services, shards, or databases, for which partial execution is undesirable? Turns out, this is exactly where sagas are useful. Maybe you are checking inventory, charging a user’s credit card, and then fulfilling the order. Maybe you are managing a supply chain. The saga pattern is helpful because it basically functions as a state machine storing program progress, preventing multiple credit card charges, reverting if necessary, and knowing exactly how to safely resume in a consistent state in the event of power loss.

A common life-based example used to explain the way the saga pattern compensates for failures is trip planning. Suppose you are itching to soak up the rain in Duwamish territory, Seattle. You’ll need to purchase an airplane ticket, reserve a hotel, and get a ticket for a guided backpacking experience on Mount Rainier. All three of these tasks are coupled: if you’re unable to purchase a plane ticket there’s no reason to get the rest. If you get a plane ticket but have nowhere to stay, you’re going to want to cancel that plane reservation (or retry the hotel reservation or find somewhere else to stay). Lastly if you can’t book that backpacking trip, there’s really no other reason to come to Seattle so you might as well cancel the whole thing. (Kidding!)

Above: a simplistic model of compensating in the face of trip planning failures.

There are many “do it all, or don’t bother” software applications in the real-world: if you successfully charge the user for an item but your fulfillment service reports that the item is out of stock, you’re going to have upset users if you don’t refund the charge. If you have the opposite problem and accidentally deliver items “for free,” you’ll be out of business. If the machine coordinating a machine learning data processing pipeline crashes but the follower machines carry on processing the data with nowhere to report their data to, you may have a very expensive compute resources bill on your hands³. In all of these cases having some sort of “progress tracking” and compensation code to deal with these “do-it-all-or-don’t-do-any-of-it” tasks is exactly what the saga pattern provides. In saga parlance, these sorts of “all or nothing” tasks are called long-running transactions. This doesn’t necessarily mean such actions run for a “long” time, just that they require more steps in logical time⁴ than something running locally interacting with a single database.

How do you build a saga?

A saga is composed of two parts:

Defined behavior for “going backwards” if you need to “undo” something (i.e., compensations)
Behavior for striving towards forward progress (i.e., saving state to know where to recover from in the face of failure)

The avid reader of this blog will remember I recently wrote a post about compensating actions. As you can see from above, compensations are but one half of the saga design pattern. The other half, alluded to above, is essentially state management for the whole system. The compensating actions pattern helps you know how to recover if an individual step (or in Temporal terminology, an Activity) fails. But what if the whole system goes down? Where do you start back up? Since not every step might have a compensation attached, you’d be forced to do your best guess based on stored compensations. The saga pattern keeps track of where you are currently so that you can keep driving towards forward progress.

So how do I implement sagas in my own code?

I’m so glad you asked.

leans forward

whispers in ear

This is a little bit of a trick question because by running your code with Temporal, you automatically get your state saved and retries on failure at any level. That means the saga pattern with Temporal is as simple as coding up the compensation you wish to take when a step (Activity) fails. The end.

The _why _behind this magic is Temporal, by design, automatically keeps track of the progress of your program and can pick up where it left off in the face of catastrophic failure. Additionally, Temporal will retry Activities on failure, without you needing to add any code beyond specifying a Retry Policy, e.g.,:

RetryOptions retryoptions = RetryOptions.newBuilder()
       .setInitialInterval(Duration.ofSeconds(1))
       .setMaximumInterval(Duration.ofSeconds(100))
       .setBackoffCoefficient(2)
       .setMaximumAttempts(500).build();

To learn more about how this automagic works, stay tuned for my upcoming post on choreography and orchestration, the two common ways of implementing sagas.

So to express the high-level logic of my program with both the vacation booking steps plus compensations I wish to take on failure, it would look like the following in pseudocode:

try:
   registerCompensationInCaseOfFailure(cancelHotel)
   bookHotel
   registerCompensationInCaseOfFailure(cancelFlight)
   bookFlight
   registerCompensationInCaseOfFailure(cancelExcursion)
   bookExcursion
catch:
   run all compensation activities

In Java, the Saga class keeps track of compensations for you:

@Override
public void bookVacation(BookingInfo info) {
   Saga saga = new Saga(new Saga.Options.Builder().build());
   try {
       saga.addCompensation(activities::cancelHotel, info.getClientId());
       activities.bookHotel(info);

       saga.addCompensation(activities::cancelFlight, info.getClientId());
       activities.bookFlight(info);

       saga.addCompensation(activities::cancelExcursion, 
                            info.getClientId());
       activities.bookExcursion(info);
   } catch (TemporalFailure e) {
       saga.compensate();
       throw e;
   }
}

In other language SDKs you can easily write the addCompensation and compensate functions yourself. Here's a version in Go:

func (s *Compensations) AddCompensation(activity any, parameters ...any) {
    s.compensations = append(s.compensations, activity)
    s.arguments = append(s.arguments, parameters)
}

func (s Compensations) Compensate(ctx workflow.Context, inParallel bool) {
    if !inParallel {
        // Compensate in Last-In-First-Out order, to undo in the reverse order that activies were applied.
        for i := len(s.compensations) - 1; i >= 0; i-- {
            errCompensation := workflow.ExecuteActivity(ctx, s.compensations[i], s.arguments[i]...).Get(ctx, nil)
            if errCompensation != nil {
                workflow.GetLogger(ctx).Error("Executing compensation failed", "Error", errCompensation)
            }
        }
    } else {
        selector := workflow.NewSelector(ctx)
        for i := 0; i < len(s.compensations); i++ {
            execution := workflow.ExecuteActivity(ctx, s.compensations[i], s.arguments[i]...)
            selector.AddFuture(execution, func(f workflow.Future) {
                if errCompensation := f.Get(ctx, nil); errCompensation != nil {
                    workflow.GetLogger(ctx).Error("Executing compensation failed", "Error", errCompensation)
                }
            })
        }
        for range s.compensations {
            selector.Select(ctx)
        }
    }
}

The high level Go code of steps and compensations will look very similar to the Java version:

func TripPlanningWorkflow(ctx workflow.Context, info BookingInfo) (err error) {
   options := workflow.ActivityOptions{
       StartToCloseTimeout: time.Second * 5,
       RetryPolicy:         &temporal.RetryPolicy{MaximumAttempts: 2},
   }

   ctx = workflow.WithActivityOptions(ctx, options)

   var compensations Compensations

   defer func() {
       if err != nil {
           // activity failed, and workflow context is canceled
           disconnectedCtx, _ := workflow.NewDisconnectedContext(ctx)
           compensations.Compensate(disconnectedCtx, true)
       }
   }()

   compensations.AddCompensation(CancelHotel)
   err = workflow.ExecuteActivity(ctx, BookHotel, info).Get(ctx, nil)
   if err != nil {
       return err
   }

   compensations.AddCompensation(CancelFlight)
   err = workflow.ExecuteActivity(ctx, BookFlight, info).Get(ctx, nil)
   if err != nil {
       return err
   }

   compensations.AddCompensation(CancelExcursion)
   err = workflow.ExecuteActivity(ctx, BookExcursion, info).Get(ctx, nil)
   if err != nil {
       return err
   }

   return err
}

This high-level sequence of code above is called a Temporal Workflow. And, as mentioned before, by running with Temporal, we don’t have to worry about implementing any of the bookkeeping to track our progress via event sourcing or adding retry and restart logic because that all comes for free. So when writing code that runs with Temporal, you only need to worry about writing compensations, and the rest is provided for free.

Idempotency

Well, okay, there is a second thing to “worry about.” As you may recall, sagas consist of two parts, the first part being those compensations we coded up previously. The second part, “striving towards forward progress” involves potentially retrying an activity in the face of failure. Let’s dig into one of those steps, shall we? Temporal does all the heavy lifting of retrying and keeping track of your overall progress, however because code can be retried, you, the programmer, need to make sure each Temporal Activity is idempotent. This means the observed result of bookFlight is the same, whether it is called one time or many times. To make this a little more concrete, a function that sets some field foo=3 is idempotent because afterwards foo will be 3 no matter how many times you call it. The function foo += 3 is not idempotent because the value of foo is dependent on the number of times your function is called. Non-idempotency can sometimes look more subtle: if you have a database that allows duplicate records, a function that calls INSERT INTO foo (bar) VALUES (3) will blithely create as many records in your table as times you call it and is therefore not idempotent. Naive implementations of functions that send emails or transfer money are also not idempotent by default.

If you’re backing away slowly right now because your Real World Application does a lot more complex things than set foo=3, take heart. There is a solution. You can use a distinct identifier, called an idempotency key, or sometimes called a referenceId or something similar to uniquely identify a particular transaction and ensure the hotel booking transaction occurs effectively once. The way this idempotency key may be defined based on your application needs. In the trip planning application, clientId, a field in BookingInfo is used to uniquely identify transactions.

type BookingInfo struct {
   Name     string
   ClientId string
   Address  string
   CcInfo   CreditCardInfo
   Start    date.Date
   End      date.Date
}

You also probably saw the clientId used to register the compensation in the above Java workflow code:

saga.addCompensation(activities::cancelHotel, info.getClientId());

However, using clientId as our key limits a particular person from booking more than one vacation at once. This is probably what we want. However, some business applications may choose to build an idempotency key by combining the clientId and the workflowId to allow more than one vacation at once booked per-client. If you wanted a truly unique idempotency key you could pass in a UUID to the workflow. The choice is up to you based on your application’s needs.

Many third-party APIs that handle money already accept idempotency keys for this very purpose. If you need to implement something like this yourself, use atomic writes to keep a record of the idempotency keys you’ve seen so far, and don’t perform an operation if its idempotency key is in the “already seen” set.

Benefits vs Complexity

The saga pattern does add complexity to your code, so it’s important to not implement it in your code just because you have microservices. However, if you need to complete a task (like booking a trip with an airfare and hotel) that involves multiple services and partial execution is not actually a success, then a saga will be your friend. Additionally, if you find your saga getting particularly unwieldy, it may be time to reconsider how your microservices are divided up, and roll up the ol’ sleeves to refactor. Overall, Temporal makes implementing the saga pattern in your code comparatively trivial since you only need to write the compensations needed for each step. Stay tuned for my next post, where I dig into sagas and subscription scenarios, where Temporal particularly shines in reducing complexity when working with sagas.

The full repository that uses the code mentioned in this article can be found on GitHub:

If you want to see other tutorials of sagas using Temporal, please check out the following resources:

Additionally one of my colleagues, Dominik Tornow, gave an intro to sagas on YouTube.

Learn more about Temporal in our courses, tutorials, docs, and videos.

Notes

Obviously, don’t redesign your system just because it’s the new hotness. Unless it’s a new JavaScript framework. Then npm install that new package with due haste. 😜 ↩
Don’t worry, sagas aren’t a trend; they’ve been around in databases since the 80s. You can take comfort knowing your project has a classic elegance to its design. ↩
Not that the author has absolutely any experience with this scenario whatsoever. coughs in the price of a new car 😬 ↩
Logical time is a notion in distributed computing to describe timing of events happening on different machines in distributed computing, since machines may not have a physical synchronous global clock. Logical timing is simply a causal ordering of events that occurred on these machines. In the case of long-running transactions, it basically boils down to having many “steps” that take place on different machines. ↩

Compensating Actions, Part of a Complete Breakfast with Sagas

Emily Fortuna — Thu, 04 May 2023 00:28:24 +0000

Morning! The compensating action pattern is a mouthful to say (and some call it the compensating transaction pattern), but it’s easy to digest. In distributed systems, we are often fighting against data consistency issues. Though microservices improve scalability, testability, and individual team code velocity, they introduce a new problem: basic instruction sequences now require communication between services and can leave the entire system in a bad state on failure. Much like the chaos of family members individually making cereal for breakfast, some operations that span services needed to be completed as a single unit, similarly to database transactions. Otherwise, one person may pour out cereal only to discover their kid used up the last of the milk. With multiple breakfast-makers–er, microservices–it is impossible to guarantee all steps in a cross-service multi-step operation (cereal, then milk) will run without any hiccups.

The compensating action pattern provides transaction-like guarantees for a sequence of operations amid this distributed systems chaos. The pattern simulates a transaction by either letting the sequence of operations successfully run to completion or, in the face of failure, undoing state changes that the executed operations made, as if “it never happened.” This sequence of steps of forward progress or undoing is called a “long-running transaction”. It’s important to note that undoing the state changes is different from rolling the database back to a previous snapshotted state. This is because while executing the operations of my own long-running transaction, another service or transaction could also have successfully modified the database and this state would be lost in the face of a rollback. Deciding the way to compensate a given operation depends on the specific scenario. Handling a money transfer? Probably best to put the funds back where they started. Shipping issue? Maybe just unreserve the inventory. Because the manner of compensation depends on the particular needs of the application, you, the developer, need to define it in your program.

In summary, compensating actions (sometimes called compensating transactions or compensating activities in Temporal parlance) are a design pattern for handling failure amongst distributed, but related services. If you have:

A series of operations that span more than one service and must be completed as a unit (“all or nothing”, or transaction-like semantics), and
Some operations must be undone if the entire transaction fails to complete

Then the compensating action pattern can ensure you maintain a consistent state in the event of failure!

Compensating actions are an important component of the famous saga design pattern. Compensating actions ensure that there is a way to “go backwards” (undo) and end up with a consistent state. Sagas, a design pattern for failure resilience, use compensations, but also ensure there is a way to “go forwards” (by retrying) and maintain that an entire sequence of operations acts like a single transaction.

Coding up Compensating Actions

A Quick Note on Ordering of Compensations

Suppose the steps and compensations for the cereal analogy looked like this:

getBowl()
addCereal()
addMilk()

A naive way of coding up the steps and compensations might look like this:

// DON'T DO THIS
try {
   getBowl()
   addCompensation(putBowlAway)
   addCereal()
   addCompensation(putCerealBackInBox)
   addMilk()
} catch (Exception e) {
   compensate()
   throw e
}

Suppose getBowl fails. If it fails because there are no bowls in the cabinet, the above code looks correct. However, suppose while “executing” getBowl you take a bowl out of the cabinet and then get distracted by what’s happening on Bluesky…for the entire morning. Your “breakfast workflow” times out (and therefore getBowl “fails”) because now it’s lunch. Well, now we have a problem because we jump to execute compensate and there are no compensations registered, but our bowl is out on the kitchen counter! This sort of scenario can happen if an Activity times out (in Temporal specifics, ScheduleToCloseTimeout StartToCloseTimeout and HeartbeatTimeout can set such limits), you run into a network error, or perhaps crashes after the thing that you’d want to compensate for (like acquiring the bowl) has been executed. In Temporal scenarios, this can also happen if the user cancels the Activity after the thing we want to compensate for has already executed. All these issues* can be solved by simply reordering your steps in the following way:

try {
   addCompensation(putBowlAwayIfPresent)
   getBowl()
   addCompensation(putCerealBackInBoxIfPresent)
   addCereal()
   addMilk()
} catch (Exception e) {
   compensate()
   throw e
}

You’ll note that the compensation methods changed because your compensation methods much take into account the possibility that what you want to compensate for may or may not have executed (ie the bowl may or may not be out on the counter in the event of an error and putBowlAwayIfPresent runs).

On to the Code!

If you’re writing Java, you can leverage Temporal’s Saga library to register functions you’d like to run as compensations, and Temporal will do the rest:

@Override
   public void makeBreakfast(boolean parallelCompensations) {
       // You can set parallel compensations if appropriate with the Builder
       Saga saga = new Saga(new Saga.Options.Builder()
                                    .setParallelCompensation(parallelCompensations)
                                    .build());
       try {
           saga.addCompensation(breakfastActivity::putBowlAwayIfPresent);
           breakfastActivity.getBowl();
           saga.addCompensation(breakfastActivity::putCerealBackInBoxIfPresent);
           breakfastActivity.addCereal();
           breakfastActivity.addMilk();
       } catch (ActivityFailure e) {
           saga.compensate();
           throw e;
       }

   }

In the event of a failure during addCereal() or addMilk(), the Saga class will call compensate(), and all registered compensation functions will execute (putBowlAway() and putCerealBackInBox()). The full repository is available on GitHub.

If you’re working with TypeScript, Python, or Go, you’ll need to keep track of the compensations yourself, but it’s not challenging. All you need to do is keep track of a list of compensating functions, and then execute them when there’s a failure.

Here’s the Python version:

class Compensations:
   def __init__(self, parallel_compensations=False):
       self.parallel_compensations = parallel_compensations
       self.compensations = []

   def add(self, function: typing.Callable[..., typing.Awaitable[None]]):
       self.compensations.append(function)

   def __iadd__(self, function: typing.Callable[..., typing.Awaitable[None]]):
       self.add(function)
       return self

   async def compensate(self):
       async def run_compensation(
           compensation: typing.Callable[..., typing.Awaitable[None]]
       ) -> None:
           try:
               await workflow.execute_activity(
                   compensation,
                   start_to_close_timeout=time_delta,
                   retry_policy=common_retry_policy,
               )
           except:
               workflow.logger.exception("failed to compensate")

       if self.parallel_compensations:
           all_compensations = [run_compensation(c) for c in self.compensations]
           await asyncio.gather(*all_compensations)

       else:
           for f in reversed(self.compensations):
               await run_compensation(f)

Then, the registering your custom compensations part will look very similar to the Java example:

@workflow.defn
class BreakfastWorkflow:
   @workflow.run
   async def run(self, parallel_compensations) -> None:
       compensations = Compensations(parallel_compensations=parallel_compensations)
       try:
           compensations += put_bowl_away_if_present
           await workflow.execute_activity(
               get_bowl,
               start_to_close_timeout=time_delta,
               retry_policy=common_retry_policy,
           )
           compensations += put_cereal_back_in_box_if_present
           await workflow.execute_activity(
               add_cereal,
               start_to_close_timeout=time_delta,
               retry_policy=common_retry_policy,
           )
           await workflow.execute_activity(
               add_milk,
               start_to_close_timeout=time_delta,
               retry_policy=common_retry_policy,
           )
       except Exception:
           task = asyncio.create_task(compensations.compensate())
           # Ensure the compensations run in the face of cancelation.
           await asyncio.shield(task)
           raise

The complete repository is up on GitHub.

Here’s the TypeScript version:

type Compensation = () => Promise<void>

async function compensate(compensations: Compensation[], compensateInParallel = false) {
   if (compensateInParallel) {
     compensations.map(comp => comp().catch(err => console.error(`failed to compensate: $error`)))
   }
    for (const comp of compensations) {
     try {
       await comp()
     } catch (err) {
       console.error(`failed to compensate: ${err}`)
     }
   }
 }

export async function breakfastWorkflow(compensateInParallel = false): Promise<void> {
 const compensations: Compensation[] = []
 try {
   compensations.unshift(putBowlAwayIfPresent)
   await getBowl()
   compensations.unshift(putCerealBackInBoxIfPresent)
   await addCereal()
   await addMilk()
 } catch (err) {
   await compensate(compensations, compensateInParallel)
   throw err
 }
}

Complete repository on GitHub.

In Go, we can make use of the defer keyword to check whether function execution aborted normally or whether compensations need to be run:

type Compensations []any

func (s *Compensations) AddCompensation(activity any) {
   *s = append(*s, activity)
}

func (s Compensations) Compensate(ctx workflow.Context, inParallel bool) {
   if !inParallel {
       for i := len(s) - 1; i >= 0; i-- {
           errCompensation := workflow.ExecuteActivity(ctx, s[i]).Get(ctx, nil)
           if errCompensation != nil {
               workflow.GetLogger(ctx).Error("Executing compensation failed", "Error", errCompensation)
           }
       }
   } else {
       selector := workflow.NewSelector(ctx)
       for i := 0; i < len(s); i++ {
           execution := workflow.ExecuteActivity(ctx, s[i])
           selector.AddFuture(execution, func(f workflow.Future) {
               if errCompensation := f.Get(ctx, nil); errCompensation != nil {
                   workflow.GetLogger(ctx).Error("Executing compensation failed", "Error", errCompensation)
               }
           })
       }
       for range s {
           selector.Select(ctx)
       }

   }
}

func BreakfastWorkflow(ctx workflow.Context, parallelCompensations bool) (err error) {
   // Omitted for brevity: set activity options and retry policy here.

   var compensations Compensations

   defer func() {
       // Defer is at the top so that it is executed regardless of which step might fail.
       if err != nil {
           // activity failed, and workflow context is canceled
           disconnectedCtx, _ := workflow.NewDisconnectedContext(ctx)
           compensations.Compensate(disconnectedCtx, parallelCompensations)
       }
   }()

   compensations.AddCompensation(PutBowlAwayIfPresent)
   err = workflow.ExecuteActivity(ctx, GetBowl).Get(ctx, nil)
   if err != nil {
       return err
   }

   compensations.AddCompensation(PutCerealBackInBoxIfPresent)
   err = workflow.ExecuteActivity(ctx, AddCereal).Get(ctx, nil)
   if err != nil {
       return err
   }

   err = workflow.ExecuteActivity(ctx, AddMilk).Get(ctx, nil)

   return err
}

The full repository for the Go code is available on GitHub.

*The Fine Print

There are a few Temporal-specific configurations or actions you should be aware of that might prevent your compensations from executing as you would like: if you set timeouts or retries on Workflows, your compensations might not get a chance to run before your Workflow times out (so, don’t set these limitations if you want to ensure your compensations run). Additionally, terminate and reset will not allow Workflow code to execute any finally or defer statements, so avoid these as well. If you avoid these scenarios, you can ensure your compensations properly run to completion.

Summary

Compensating actions (or compensating transactions) are a distributed systems design pattern for simulating atomic execution of operations distributed across multiple databases. If one of the distributed operations fails, their effects are undone via a compensating action. Compensations are a component of the larger saga design pattern, about which I’ll go into more detail in my next post.

The complete project with all the code mentioned in this post is available on GitHub:

To see one of my colleagues, Dominik Tornow, give an intro to sagas that builds on these ideas, please check out our YouTube video.

This is part 1 of a series. To see the second post, check out Saga Pattern Made Easy.