Forem: Loren 🤓

Time-Travel Debugging Production Code

Loren 🤓 — Tue, 08 Aug 2023 07:03:31 +0000

In this post, I’ll give an overview of time travel debugging (what it is, its history, how it’s implemented) and show how it relates to debugging your production code.

Normally, when we use debuggers, we set a breakpoint on a line of code, we run our code, execution pauses on our breakpoint, we look at values of variables and maybe the call stack, and then we manually step forward through our code's execution. In time-travel debugging, also known as reverse debugging, we can step backward as well as forward. This is powerful because debugging is an exercise in figuring out what happened: traditional debuggers are good at telling you what your program is doing right now, whereas time-travel debuggers let you see what happened. You can wind back to any line of code that executed and see the full program state at any point in your program’s history.

History and current state

It all started with Smalltalk-76, developed in 1976 at Xerox PARC. (Everything started at PARC 😄.) It had the ability to retrospectively inspect checkpointed places in execution. Around 1980, MIT added a "retrograde motion" command to its DDT debugger, which gave a limited ability to move backward through execution. In a 1995 paper, MIT researchers released ZStep 95, the first true reverse debugger, which recorded all operations as they were performed and supported stepping backward, reverting the system to the previous state. However, it was a research tool and not widely adopted outside academia.

ODB, the Omniscient Debugger, was a Java reverse debugger that was introduced in 2003, marking the first instance of time-travel debugging in a widely used programming language. GDB (perhaps the most well-known command-line debugger, used mostly with C/C++) added it in 2009.

Now, time-travel debugging is available for many languages, platforms, and IDEs, including:

Replay for JavaScript in Chrome, Firefox, and Node, and Wallaby for tests in Node
WinDbg for Windows applications
rr for C, C++, Rust, Go, and others on Linux
Undo for C, C++, Java, Kotlin, Rust, and Go on Linux
Various extensions (often rr- or Undo-based) for Visual Studio, VS Code, JetBrains IDEs, Emacs, etc.

Implementation techniques

There are three main approaches to implementing time-travel debugging:

Record & Replay: Record all non-deterministic inputs to a program during its execution. Then, during the debug phase, the program can be deterministically replayed using the recorded inputs in order to reconstruct any prior state.
Snapshotting: Periodically take snapshots of a program's entire state. During debugging, the program can be rolled back to these saved states. This method can be memory-intensive because it involves storing the entire state of the program at multiple points in time.
Instrumentation: Add extra code to the program that logs changes in its state. This extra code allows the debugger to step the program backwards by reverting changes. However, this approach can significantly slow down the program's execution.

rr uses the first (the rr name stands for Record and Replay), as does Replay. WinDbg uses the first two, and Undo uses all three (see how it differs from rr).

Time-traveling in production

Traditionally, running a debugger in prod doesn't make much sense. Sure, we could SSH into a prod machine and start the process handling requests with a debugger and a breakpoint, but once we hit the breakpoint, we're delaying responses to all current requests and unable to respond to new requests. Also, debugging non-trivial issues is an iterative process: we get a clue, we keep looking and find more clues; discovery of each clue is typically rerunning the program and reproducing the failure. So, instead of debugging in production, what we do is replicate on our dev machine whatever issue we're investigating and use a debugger locally (or, more often, add log statements 😄), and re-run as many times as required to figure it out. Replicating takes time (and in some cases a lot of time, and in some cases infinite time), so it would be really useful if we didn't have to.

While running traditional debuggers doesn't make sense, time-travel debuggers can record a process execution on one machine and replay it on another machine. So we can record (or snapshot or instrument) production and replay it on our dev machine for debugging (depending on the tool, our machine may need to have the same CPU instruction set as prod). However, the recording step generally doesn't make sense to use in prod given the high amount of overhead—if we set up recording and then have to use ten times as many servers to handle the same load, whoever pays our AWS bill will not be happy 😁.

But there are a couple scenarios in which it does make sense:

Undo only slows down execution 2–5x, so while we don't want to leave it on just in case, we can turn it on temporarily on a subset of prod processes for hard-to-repro bugs until we have captured the bug happening, and then we turn it off.
When we're already recording the execution of a program in the normal course of operation.

The rest of this post is about #2, which is a way of running programs called durable execution.

Durable execution

What's that?

First, a brief backstory. After Amazon (one of the first large adopters of microservices) decided that using message queues to communicate between services was not the way to go (hear the story first-hand here), they started using orchestration. And once they realized defining orchestration logic in YAML/JSON wasn't a good developer experience, they created AWS Simple Workfow Service to define logic in code. This technique of backing code by an orchestration engine is called durable execution, and it spread to Azure Durable Functions, Cadence (used at Uber for > 1,000 services), and Temporal (used by Stripe, Netflix, Datadog, Snap, Coinbase, and many more).

Durable execution runs code durably—recording each step in a database, so that when anything fails, it can be retried from the same step. The machine running the function can even lose power before it gets to line 10, and another process is guaranteed to pick up executing at line 10, with all variables and threads intact.¹ It does this with a form of record & replay: all input from the outside is recorded, so when the second process picks up the partially-executed function, it can replay the code (in a side-effect–free manner) with the recorded input in order to get the code into the right state by line 10.

Durable execution's flavor of record & replay doesn't use high-overhead methods like software JIT binary translation, snapshotting, or instrumentation. It also doesn't require special hardware. It does require one constraint: durable code must be deterministic (i.e., given the same input, it must take the same code path). So it can't do things that might have different results at different times, like use the network or disk. However, it can call other functions that are run normally ("volatile functions", as we like to call them 😄), and while each step of those functions isn't persisted, the functions are automatically retried on transient failures (like a service being down).

Only the steps that require interacting with the outside world (like calling a volatile function, or calling sleep('30 days'), which stores a timer in the database) are persisted. Their results are also persisted, so that when you replay the durable function that died on line 10, if it previously called the volatile function on line 5 that returned "foo", during replay, "foo" will immediately be returned (instead of the volatile function getting called again). While yes, it adds latency to be saving things to the database, Temporal supports extremely high throughput (tested up to a million recorded steps per second). And in addition to function recoverability and automatic retries, it comes with many more benefits, including extraordinary visibility into and debuggability of production.

Debugging prod

With durable execution, we can read through the steps that every single durable function took in production. We can also download the execution’s history, checkout the version of the code that's running in prod, and pass the file to a replayer (Temporal has runtimes for Go, Java, JavaScript, Python, .NET, and PHP) so we can see in a debugger exactly what the code did during that production function execution. Read this post or watch this video to see an example in VS Code.²

Being able to debug any past production code is a huge step up from the other option (finding a bug, trying to repro locally, failing, turning on Undo recording in prod until it happens again, turning it off, then debugging locally). It's also a (sometimes necessary) step up distributed tracing.

I hope you found this post interesting! If you'd like to learn more about durable execution, I recommend reading:

and watching:

Thanks to Greg Law, Jason Laster, Chad Retz, and Fitz for reviewing drafts of this post.

Technically, it doesn't have line-by-line granularity. It only records certain steps that the code takes—read on for more info ☺️. ↩
The astute reader may note that our extension uses the default VS Code debugger, which doesn’t have a back button 😄. I transitioned from talking about TTD to methods of debugging production code via recording, so while Temporal doesn’t have TTD yet, it does record all the non-deterministic inputs to the program and is able to replay execution, so it’s definitely possible to implement. Upvote this issue or comment if you have thoughts on implementation! ↩

Sync GitHub Repos to Notion

Loren 🤓 — Wed, 07 Jun 2023 00:50:43 +0000

At our last company hackathon, I built an internal tool that syncs your GitHub org's list of repositories to a Notion database. It takes this list:

and makes this database in Notion:

The database serves two purposes:

Tracking which team owns each repo, and whom to contact if you have a question. A couple of questions that come up are:
- "This sample repo is out of date—who is responsible for it?"
- "A user asked about their issue/PR on this repo—which team is responsible for responding?"

The two manually updated columns we use for this are "Owner" with team multi-select and "POC" with Person multi-select.

Tracking which GitHub teams or individuals have which roles on each repo. It's hard to view all that info on github.com, and having it all in one table makes it much easier. The columns we use for this are:
- Teams admins
- Individual admins
- Teams with role: maintainer
- Individuals with role: maintainer
- Teams with role: write
- Individuals with role: write
- Teams with role: triage
- Individuals with role: triage
- Teams with role: read
- Individuals with role: read

Since, by default, everyone in the org has write access to all repos, org members are excluded from the "Individuals with role: write" column to reduce noise.

Implementation

Sync function

The one-way sync is implemented with TypeScript code (temporalio/github-repo-notion-sync) that fetches data from the GitHub API and then updates the Notion database via Notion's client library:

sync.ts

export async function syncGithubToNotion() {
  const [repos, teammates] = await Promise.all([getRepos(), getTeammates()])
  const withContributors = await addContributors(repos)
  const completeRepos = await addCollaboratorsAndTeams(
    withContributors,
    teammates
  )
  const transformedRepos = transformRepos(completeRepos)

  await updateNotion(transformedRepos)
}

The syncGithubtoNotion function:

Fetches the org's repos (getRepos)
Fetches the org's members (getTeammates)
Fetches each repo's contributors (addContributors)
Merges and transforms the above data into the format needed
Updates the Notion db with the notion.pages.create and notion.pages.update functions (updateNotion)

Running on a schedule

syncGithubToNotion runs daily at 6am. The easiest way to set that up would have been a GitHub Action that runs on a schedule:

name: run daily at 6am

on:
  schedule:
    - cron: '0 6 * * *'

jobs:
  sync:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - env:
          NOTION_TOKEN: ${{ secrets.NOTION_TOKEN }}
          GITHUB_TOKEN: ${{ secrets.GH_TOKEN }}
        run: npm start

However, GitHub Action crons aren't very reliable:

So I used a new feature of Temporal called Schedules:

create-or-update-schedule.ts

await client.schedule.create({
  action: {
    type: 'startWorkflow',
    workflowType: syncGithubToNotion,
    taskQueue,
    workflowId: 'gh-notion-sync,
  },
  scheduleId: 'gh-notion-sync',
  spec: {
    calendars: [
      {
        comment: 'daily at 6am',
        hour: 6,
      },
    ],
  },
})

This reliably runs syncGithubToNotion daily at 6am, and since syncGithubToNotion is a durable function, if any part of it fails, that part will automatically be retried until it succeeds, and then the function will continue. It's run so reliably that even if the machine loses power on the third line of the function, the function will continue to run starting from the third line on another machine. Perhaps that's more reliability than I needed for this particular internal tool, but it was a good use case for Temporal Schedules and demo of durable functions, which you can use for your application's backend code if you want it to run extremely reliably! 😃 (For an intro and explanation of how, see Building Reliable Distributed Systems in Node.)

So that was my project 🤓. It's open source if you'd like to adapt it for your company. Let me know if you have any questions—I'd be happy to help out. I'm @lorendsr on Twitter and @loren on our community Slack 💃.

Temporal for VS Code

Loren 🤓 — Thu, 25 May 2023 19:53:02 +0000

We just published the first version of our extension for Visual Studio Code. It currently improves the debugging experience for TypeScript workflows, and in the future we'd like to add other languages, features, and editors. If you have ideas, open an issue in the GitHub repo!

When you run a workflow function, each step it takes is stored as an event in a list called the Event History. You can download a workflow's Event History from the UI or CLI in a JSON file and have an SDK replay it: the Worker calls your workflow function with the same arguments, the Worker immediately resolves activity calls with the return values recorded in Event History, and other things like timers going off and signals arriving happen in the same order they did previously. You can set breakpoints in workflow code to see what happened during the workflow's original execution. This ability to replay what happened during a function's execution, whether it's on a coworker's machine or in production, is a great help to debugging.

The Temporal VS Code extension improves the experience of replay debugging. It allows you to:

Load Event Histories by Workflow Id or JSON file
Easily start a replay
Set breakpoints on events, not just code

Usage

To learn how to use the extension, watch this video or follow these steps:

Install the extension
If you don’t already have a TypeScript workflow, create a project if you don't already have one:

  $ npx @temporalio/create@latest replay-demo --sample vscode-debugger
  $ code replay-demo

If you do have a TypeScript workflow, add a file at src/debug-replayer.ts (or can configure an alternate location):

  import { startDebugReplayer } from "@temporalio/worker"

  startDebugReplayer({
    workflowsPath: require.resolve("./workflows"),
  })

And edit the './workflows' path to match the location of your workflows file. This file is run the by the extension whenever you start a debug session. startDebugReplayer creates a replay Worker (like Worker.runReplayHistory does) and communicates with the extension.

Run Temporal: Open Panel (use Cmd/Ctrl-Shift-P to open Command Palette)

Enter a Workflow Id or choose a history JSON file (if you have the replay-demo project open, use this file)

Click Start
The Workflow Execution will start replaying and hit a breakpoint set on the first event:

Set breakpoints in code or on history events. Here's a breakpoint on the sleep line as well as the history event when the verify Signal arrives:

Hit play or step forward. In this example, after you hit play, it goes until the WorkflowExecutionSignaled event, pausing on the first line of the Signal handler:

When you hit play again, it stops on the sleep line:

You can see on the top left that the verified variable is now true.

To restart from the beginning, click the green restart icon:

If instead of restarting, you step forward or hit play again, the debugging session will end (the replay Worker gets to the end of this history file and exits). Once a debug session has ended, to restart it, go back to the MAIN tab and Start again.

The humanVerificationWorkflow workflow used above is in temporalio/samples-typescript. For a more in-depth demo, see this video.

Configuration

Server

When starting a replay by Workflow Id (and optionally Run Id), the extension downloads the history from the Temporal Server. By default, it connects to a Server running on the default localhost:7233.

To connect to a different Server:

Open the SETTINGS tab
Edit the Address field
If you're using TLS (e.g. to connect to Temporal Cloud), check the box and select your client cert and key

Entrypoint

By default, the extension will look for the file that calls startDebugReplayer at src/debug-replayer.ts. To use a different TypeScript or JavaScript file, set the temporal.replayerEntrypoint config:

Open or create .vscode/settings.json
Add the config field:

  {
    "temporal.replayerEntrypoint": "test/different-file.ts"
  }

That's the extension! If you have any problems with it, please open an issue. We also welcome contributions—please get consensus for your plan in an issue before submitting a PR ☺️. Happy debugging! 🐞💃

How Durable Execution Works

Loren 🤓 — Tue, 23 May 2023 17:58:19 +0000

This is part 2 of the series, "Building Reliable Distributed Systems in Node.js." In part 1, we went over what durable execution is, its benefits, and what a durable function looks like. In this post, we'll look at how Temporal can provide durable execution.

A function that can't fail, can last forever, and doesn't need to store data in a database? Sounds like magic. There must be a gotcha—like only a small subset of the language can be used, or it only works on specialized hardware. But in fact, it's just JavaScript—you can use the whole language, and it runs on any server that can run Node.js.

So how does this all work? You can take a look at the How Temporal Works diagram, which explains the process with Go code. In this post, we'll go through the process with the TypeScript code from the previous post in the series.

Client ↔️ Server ↔️ Worker

To start out, a Temporal application has three parts: the Client, the Server, and the Worker.

The Client and Worker both connect to the Server, which has a database and maintains state. The Client says things like "start the order() durable function," "send it a delivered Signal," and "terminate the function." The Worker is a long-running Node.js process that has our code and polls the Server for tasks. Tasks look like "run the order() durable function" or "run the normal sendPushNotification() function." After the Worker runs the code, it reports the result back to the Server.

In our delivery app, we create the Client in temporal-client.ts and use it in our Next.js serverless functions:

apps/menu/pages/api/[trpc].ts is the API for temporal.menu, and it
- Starts order Workflows.
- Queries order Workflows for their status.
apps/driver/pages/api/[trpc].ts is the API for drive.temporal.menu, and it
- Sends pickedUp and delivered Signals to order Workflows.
- Fetches a list of all order Workflows and Queries for their statuses.

We create the Worker in apps/worker/src /worker.ts:

import { NativeConnection, Worker } from '@temporalio/worker'
import * as activities from 'activities'
import { taskQueue } from 'common'
import { namespace, getConnectionOptions } from 'common/lib/temporal-connection'

async function run() {
 const connection = await NativeConnection.connect(getConnectionOptions())
 const worker = await Worker.create({
   workflowsPath: require.resolve('../../../packages/workflows/'),
   activities,
   connection,
   namespace,
   taskQueue,
 })

 await worker.run()
}

run().catch((err) => {
 console.error(err)
 process.exit(1)
})

We pass the Worker our code:

The path to our Workflows, which get bundled with Webpack and run in isolated contexts to ensure determinism.
Our imported Activities—the non-durable functions called by our order Workflow: sendPushNotification(), refundOrder(), and chargeCustomer().

And set up Render to automatically build and deploy on pushes to main:

Running each part

In production, our web apps and their serverless functions are deployed to Vercel, our long-running Worker process is deployed to Render, and they both talk to a Server instance hosted by Temporal Cloud. The Server is an open source cluster of services which work with a database (SQL or Cassandra) and ElasticSearch. You can also host all that it yourself, or you can save a lot of time and get peace of mind with higher reliability and scale by paying the experts to host it 😄.

In development, we can run all three parts locally. First, we install the Temporal CLI, which has a development version of the Server:

Homebrew: brew install temporal
cURL: curl -sSf https://temporal.download/cli.sh | sh
Manual: Download and extract the latest release and then add it to your PATH.

We start the Server with:

temporal server start-dev

In another terminal, with Node v16 or higher, run:

npx @temporalio/create@latest ./temporal-delivery --sample food-delivery
cd ./temporal-delivery
npm run dev

The dev script runs the two Next.js web apps and the Worker. The menu is running at localhost:3000, where we can click "Order", and the driver portal is at localhost:3001, where we can mark the item we ordered as picked up and delivered. Once we've done that, we can see the "Delivered" status of the order in both of the web apps. We can also see the status of the corresponding Workflow Execution in the Server's Web UI at localhost:8233:

We can see its status is Completed, and when we click on it, we see the Event History—the list of events that took place during the execution of the order() function:

The first event is always WorkflowExecutionStarted, which contains the type of Workflow being started—in this case, an order Workflow. We'll look more at events in the next section.

In the Queries tab, we can select the getStatus Query, which (assuming our Worker is still running) will send a Query to the order function, which responds that the order was delivered, the time of delivery, and which item was delivered:

Sequence of events

Now let's look at what happened behind the scenes during our order.

Start order

When we clicked the "Order" button, the API handler used a Temporal Client to send a start command to the Server:

apps/menu/pages/api/[trpc].ts

The Server saves the WorkflowExecutionStarted event to the Event History and returns. It also creates a WorkflowTaskScheduled event, which results in a "Workflow Task" (an instruction to run the order() function) getting added to the task queue, which the Worker is polling on.

The Worker receives the task, the Server adds the WorkflowTaskStarted Event, and the Worker performs the task—in this case, calling order(3). The order function runs until it hits this line:

const { chargeCustomer, refundOrder, sendPushNotification } = proxyActivities<typeof activities>({ ... })

export async function order(productId: number): Promise<void> {
 ...

 await chargeCustomer(product)

Call Activity

When the function calls the chargeCustomer Activity, the Worker tells the Server:

The Server adds the WorkflowTaskCompleted event (the Workflow Execution didn't complete—just the initial task of "run the order() function and see what happens") and an ActivityTaskScheduled event with the Activity type and arguments:

Then the Server adds an Activity Task (an instruction to run an Activity function) to the task queue, which the Worker picks up:

In development, we're only running a single Worker process, so it's getting all the tasks, but in production, we'll have enough Workers to handle our load, and any of them can pick up the Activity Task—not just the one that ran the order function.

The Worker follows the Activity Task instructions, running the chargeCustomer() function:

packages/activities/index.ts

Which calls paymentService.charge():

packages/activities/services.ts

If the function throws an error, the Worker reports it back to the Server:

And the Server schedules a retry. The default initial interval is 1 second, so in 1 second, the Activity Task will be added back to the queue for a Worker to pick up.

If the function completes successfully, the Worker reports success (and the return value, but in this case there is none) back to the Server:

The Server adds the ActivityTaskStarted and ActivityTaskCompleted events. Now that the Activity is completed, the order() function can continue executing, so the Server adds another WorkflowTaskScheduled event. It also adds a corresponding Workflow Task to the queue, which the Worker picks up (at which point the Server adds another WorkflowTaskStarted event).

Second Workflow Task

If the Worker still has the execution context of the order function, it can just resolve the chargeCustomer(product) Promise, and the function will continue executing. If the Worker doesn't have the execution context (because it was evicted from cache in order to make room for another Workflow—see WorkerOptions.maxCachedWorkflows—or the process crashed or restarted), then the Worker fetches the Event History from the Server, creates a new isolate, and calls the function again:

This time, when the function hits the await chargeCustomer(product) line, the Worker knows from event 7, ActivityTaskCompleted, that chargeCustomer has already been run, so instead of sending a "Call Activity" command to the Server, it immediately resolves the Promise. The function continues running until the next await:

packages/workflows/order.ts

const notPickedUpInTime = !(await condition(() => state === 'Picked up', '1 min'))

condition() will wait until either the state becomes Picked up or 1 minute has passed. When it's called, the Worker tells the Server to set a timer:

The Server adds the WorkflowTaskCompleted and TimerStarted events to the Event History, and sets a timer in the database. When the timer goes off, a TimerFired event will be added along with a WorkflowTaskScheduled and Workflow Task on the queue telling the Worker the 1 minute is up, at which point the Worker will know to resolve the condition() Promise.

Send Signal

But in our case, that didn't happen. Instead, before a minute was up, we clicked the "Pick up" button in the driver portal, which sent a pickedUp Signal to the Workflow:

The Server then added two events: a WorkflowExecutionSignaled event with the Signal info, and another WorkflowTaskScheduled event. Then a Workflow Task was added to the queue with the Signal info, which was picked up by the Worker:

The Worker then runs the pickedUp Signal handler:

setHandler(pickedUpSignal, () => {
 if (state === 'Paid') {
   state = 'Picked up'
 }
})

packages/workflows/order.ts

The handler changes the state to Picked up, and after Signal handlers have been called, the Worker runs all the condition() functions. Now () => state === 'Picked up' will return true, so the Worker will resolve the condition() Promise and continue executing, to see what the function does next, which will determine the next command(s) it sends to the Server.

Event History

All together, the part of the Event History we covered was:

The Event History is the core of what enables durable execution: a log of everything important that the Workflow does and gets sent. It allows us to Ctrl-C our Worker process, start it again, open up the UI, select our completed order Workflow, go to the Queries tab of the UI, and send the getStatus Query, which the Server will put on a queue for the Worker to pick up, which won't have the Workflow in cache, so it will fetch the Event History from the Server, and then call the order() function, immediately resolving any async functions with the original result from History, and then calling the getStatus handler function:

setHandler(getStatusQuery, () => {
 return { state, deliveredAt, productId }
})

packages/workflows/order.ts

Since the whole function and all the Signal handlers have been run, the { state, deliveredAt, productId } variables will all have their final values from when the function was originally executed, and they'll be returned to the Server, which returns them to the Client, which returns them to the UI to display on our screen:

Summary

We examined how durable code works under the hood—how we can:

Write functions that can't fail to complete executing (since when the Worker process dies, the next Worker to pick up the task will get the function's Event History and use the events to re-run the code until it's in the same state).
Retry the functions that might have transient failures (if the chargeCustomer Activity can't reach the payment service, the Server automatically schedules another Activity Task).

The best part is that these failures are transparent to the application developer—we just write our Workflow and Activity code, and Temporal handles reliably executing it. 💃

In the next post, you’ll learn more things you can do with durable functions. To get notified when it comes out, you can follow us on Twitter or LinkedIn. Also, check out our new Temporal 101 course!

🖖 till next time!
—Loren

Thanks to Brian Hogan, Roey Berman, Patrick Rachford, and Dail Magee Jr for reading drafts of this post.

Using the Temporal CLI

Loren 🤓 — Thu, 27 Apr 2023 18:21:14 +0000

In the Server 1.20 release we announced a new Temporal CLI, and we also released an interview and blog post with more info about it and its history.

The tl;dr is we combined our efforts on:

The next version of tctl, our old CLI
temporalite

to create a single temporal command-line tool that lets you both:

run a development version of the Server
interact with the Server

We recommend switching to temporal for development and testing, and recommend using it for production scripts once we release v1 (coming soon). We will continue supporting tctl for two more Server releases (through 1.22).

In this post, you’ll learn how to use temporal.

Contents:

Installing
Running the Server
Interacting with the Server
Environments
Aliases
Upgrading
Future

Installing

cURL

curl -sSf https://temporal.download/cli.sh | sh

Homebrew

brew install temporal

Manual

You can also manually download, extract, and add to PATH the latest binary for your CPU architecture.

Running the Server

temporal server start-dev

This:

starts the Server on localhost:7233
runs the Web UI at http://localhost:8233

By default, it doesn’t persist your data—if you start a Workflow, Ctrl-C, and run the command again, your Workflow will be gone.

If you'd like your Workflows to be saved, use the --db-filename flag:

temporal server start-dev --db-filename temporal.db

Interacting with the Server

In another terminal, you can run commands to interact with the Server. This command starts a Workflow:

$ temporal workflow start \
  --task-queue hello-world \
  --type MyWorkflow \
  --workflow-id 123 \
  --input 456

Running execution:
  WorkflowId                                   123
  RunId       357074e4-0dd8-4c44-8367-d92536dd0943
  Type        MyWorkflow
  Namespace   default
  TaskQueue   hello-world
  Args        [456]

The shorthand options are:

temporal workflow start -t hello-world --type MyWorkflow -w 123 -i 456

You can also list and describe Workflows:

$ temporal workflow list

  Status   WorkflowId     Name       StartTime
  Running         123  MyWorkflow  14 seconds ago

$ temporal workflow describe --workflow-id 123

{
  "executionConfig": {
    "taskQueue": {
      "name": "hello-world",
      "kind": "Normal"
    },
    "workflowExecutionTimeout": "0s",
    "workflowRunTimeout": "0s",
    "defaultWorkflowTaskTimeout": "10s"
  },
  "workflowExecutionInfo": {
    "execution": {
      "workflowId": "123",
      "runId": "357074e4-0dd8-4c44-8367-d92536dd0943"
    },
    "type": {
      "name": "MyWorkflow"
    },
    "startTime": "2023-04-15T06:42:31.191137Z",
    "status": "Running",
    "historyLength": "2",
    "executionTime": "2023-04-15T06:42:31.191137Z",
    "memo": {

    },
    "autoResetPoints": {

    },
    "stateTransitionCount": "1"
  },
  "pendingWorkflowTask": {
    "state": "Scheduled",
    "scheduledTime": "2023-04-15T06:42:31.191173Z",
    "originalScheduledTime": "2023-04-15T06:42:31.191173Z",
    "attempt": 1
  }
}

When listing, you can get more Workflow fields and output in JSON:

$ temporal workflow list --fields long --output json

[
  {
    "execution": {
      "workflow_id": "123",
      "run_id": "357074e4-0dd8-4c44-8367-d92536dd0943"
    },
    "type": {
      "name": "MyWorkflow"
    },
    "start_time": "2023-04-15T06:42:31.191137Z",
    "status": 1,
    "execution_time": "2023-04-15T06:42:31.191137Z",
    "memo": {},
    "task_queue": "hello-world"
  }
]

Filter out just the type with jq:

$ temporal workflow list --fields long -o json | jq '.[].type.name'

"OtherWorkflow"
"MyWorkflow"
"MyWorkflow"

And count how many Workflows of each type you have:

$ temporal workflow list --fields long -o json | jq '.[].type.name' | uniq -c

   1 "OtherWorkflow"
   2 "MyWorkflow"

For a list of common commands, see our CLI cheatsheet.

To see what else you do to Workflows, run:

$ temporal workflow 

NAME:
   temporal workflow - Operations that can be performed on Workflows.

COMMANDS:
   start        Starts a new Workflow Execution.
   execute      Start a new Workflow Execution and prints its progress.
   describe     Show information about a Workflow Execution.
   list         List Workflow Executions based on a Query.
   show         Show Event History for a Workflow Execution.
   query        Query a Workflow Execution.
   stack        Query a Workflow Execution with __stack_trace as the query type.
   signal       Signal Workflow Execution by Id or List Filter.
   count        Count Workflow Executions (requires ElasticSearch to be enabled).
   cancel       Cancel a Workflow Execution.
   terminate    Terminate Workflow Execution by Id or List Filter.
   delete       Deletes a Workflow Execution.
   reset        Resets a Workflow Execution by Event Id or reset type.
   reset-batch  Reset a batch of Workflow Executions by reset type (LastContinuedAsNew), FirstWorkflowTask), LastWorkflowTask
   trace        Trace progress of a Workflow Execution and its children.
   help, h      Shows a list of commands or help for one command

And here are all the top-level commands:

$ temporal

NAME:
   temporal - Temporal command-line interface and development server

USAGE:
   temporal [global options] command [command options] [arguments...]

VERSION:
   0.8.0 (server 1.20.1) (ui 2.13.3)

COMMANDS:
   server      Commands for managing the Temporal Server.
   workflow    Operations that can be performed on Workflows.
   activity    Operations that can be performed on Workflow Activities.
   task-queue  Operations performed on Task Queues.
   schedule    Operations performed on Schedules.
   batch       Operations performed on Batch jobs.
   operator    Operations performed on the Temporal Server.
   env         Manage environmental configurations on Temporal Client.
   completion  Output shell completion code for the specified shell (zsh, bash).
   help, h     Shows a list of commands or help for one command

You can read more about each by running temporal <command> or reading the CLI docs.

Environments

So far, the CLI has been talking to the Server at the default address, localhost:7233. To talk to another Server, like a production namespace on Temporal Cloud:

Create an environment named prod.
Pass --env prod to commands, like temporal workflow list --env prod.

To create a new environment, start setting its properties:

temporal env set prod.namespace production.f45a2
temporal env set prod.address production.f45a2.tmprl.cloud:7233
temporal env set prod.tls-cert-path /temporal/certs/prod.pem
temporal env set prod.tls-key-path /temporal/certs/prod.key

Check that you set them correctly:

$ temporal env get prod

  address        production.f45a2.tmprl.cloud:7233
  namespace      production.f45a2
  tls-cert-path  /temporal/certs/prod.pem
  tls-key-path   /temporal/certs/prod.key

If they’re correct, then this shouldn’t log a connection error:

$ temporal workflow list --env prod

For the full list of properties you can set, see the below options:

$ temporal env set -h

OPTIONS:
   Client Options:

   --address value                          The host and port (formatted as host:port) for the Temporal Frontend Service. [$TEMPORAL_CLI_ADDRESS]
   --codec-auth value                       Sets the authorization header on requests to the Codec Server. [$TEMPORAL_CLI_CODEC_AUTH]
   --codec-endpoint value                   Endpoint for a remote Codec Server. [$TEMPORAL_CLI_CODEC_ENDPOINT]
   --context-timeout value                  An optional timeout for the context of an RPC call (in seconds). (default: 5) [$TEMPORAL_CONTEXT_TIMEOUT]
   --env value                              Name of the environment to read environmental variables from. (default: "default")
   --grpc-meta value [ --grpc-meta value ]  Contains gRPC metadata to send with requests (format: key=value). Values must be in a valid JSON format.
   --namespace value, -n value              Identifies a Namespace in the Temporal Workflow. (default: "default") [$TEMPORAL_CLI_NAMESPACE]
   --tls-ca-path value                      Path to server CA certificate. [$TEMPORAL_CLI_TLS_CA]
   --tls-cert-path value                    Path to x509 certificate. [$TEMPORAL_CLI_TLS_CERT]
   --tls-disable-host-verification          Disables TLS host name verification if already enabled. (default: false) [$TEMPORAL_CLI_TLS_DISABLE_HOST_VERIFICATION]
   --tls-key-path value                     Path to private certificate key. [$TEMPORAL_CLI_TLS_KEY]
   --tls-server-name value                  Provides an override for the target TLS server name. [$TEMPORAL_CLI_TLS_SERVER_NAME]

   Display Options:

   --color value  when to use color: auto, always, never. (default: "auto")

For example, to set --codec-endpoint, you would do:

$ temporal env set prod.codec-endpoint localhost:3000

Aliases

If you like aliases, here’s what I have in my ~/.bash_profile:

alias t='temporal'
alias tw='temporal workflow'
alias ts='temporal server start-dev'
alias tsdb='temporal server start-dev --db-filename ~/temporal.db'

# send process to background so you can continue using the terminal
alias tsbg='temporal server start-dev &> /dev/null & disown'

You can also set up auto-completion for zsh or bash.

Upgrading

If you’re moving from tctl, here are the major changes to be aware of:

Global changes

New pagination mechanism with OS level tools such as less or more. Controlled by --pager or $PAGER. details
Most commands can format the output as a table or JSON using the --output option. details
You can limit the number of items when listing workflows, namespaces, etc. with, e.g. --limit 10.
DateTime formatting. details
All command and flag names now use hyphen - as delimiter instead of underscore _. details
--color option to choose when to enable/disable output coloring. Adds JSON coloring by default in TTY. details
env feature to manage CLI environments and change CLI options defaults. details
Updated the semantics of the commands and flags for consistency.
All global flags are moved under each command.

Specific changes

namespace, search-attribute, and cluster former top-level commands are now under temporal operator (these are meant for self-hosted Clusters, not Temporal Cloud).
Some commands are renamed: namespace register ➡️ namespace create, search-attribute add ➡️ search-attribute create
Added new command server start-dev!

Future

0.8.0 is the current version, and we have just a few issues left before the 1.0 release. We are following semantic versioning, so after the 1.0 release, we won’t change or remove commands or arguments until 2.0.

Here is the list of feature requests that we and the community have submitted. We would value any additions you have, and it also helps us prioritize if you give the things you want a 👍 emoji upvote! We also welcome contributions—if you’re interested in improving the CLI, comment on the relevant issue with your plans for (or questions about) a PR. Another improvement we’re discussing is being able to use our Cloud CLI commands in the main CLI so that you can do commands like temporal cloud namespace create.

Building Reliable Distributed Systems in Node

Loren 🤓 — Tue, 24 Jan 2023 05:47:18 +0000

This post introduces the concept of durable execution, which is used by Stripe, Netflix, Coinbase, Snap, and many others to solve a wide range of problems in distributed systems. Then it shows how simple it is to write durable code using our TypeScript/JavaScript SDK.

Distributed systems

When building a request-response monolith backed by a single database that supports transactions, we don’t have many distributed systems concerns. We can have simple failure modes and easily maintain accurate state:

If the client can’t reach the server, the client retries.
If the client reaches the server, but the server can’t reach the database, the server responds with an error, and the client retries.
If the server reaches the database, but the transaction fails, the server responds with an error, and the client retries.
If the transaction succeeds but the server goes down before responding to the client, the client retries until the server is back up, and the transaction fails the second time (assuming the transaction has some check–like an idempotency token–to tell whether the update has already been applied), and the server reports to the client that the action has already been performed.

As soon as we introduce a second place for state to live, whether that’s a service with its own database or an external API, handling failures and maintaining consistency (accuracy across all data stores) gets significantly more complex. For example, if our server has to charge a credit card and also update the database, we can no longer write simple code like:

function handleRequest() {
  paymentAPI.chargeCard()
  database.insertOrder()
  return 200
}

If the first step (charging the card) succeeds, but the second step (adding the order to the database) fails, then the system ends up in an inconsistent state; we charged their card, but there’s no record of it in our database. To try to maintain consistency, we might have the second step retry until we can reach the database. However, it’s also possible that the process running our code will fail, in which case we’ll have no knowledge that the first step took place. To fix this, we need to do three things:

Persist the order details
Persist which steps of the program we’ve completed
Run a worker process that checks the database for incomplete orders and continues with the next step

That, along with persisting retry state and adding timeouts for each step, is a lot of code to write, and it’s easy to miss certain edge cases or failure modes (see the full, scalable architecture). We could build things faster and more reliably if we didn’t have to write and debug all that code. And we don’t have to, because we can use durable execution.

Durable execution

Durable execution systems run our code in a way that persists each step the code takes. If the process or container running the code dies, the code automatically continues running in another process with all state intact, including call stack and local variables.

Durable execution ensures that the code is executed to completion, no matter how reliable the hardware or how long downstream services are offline. Retries and timeouts are performed automatically, and resources are freed up when the code isn’t doing anything (for example while waiting on a sleep(‘1 month’) statement).

Durable execution makes it trivial or unnecessary to implement distributed systems patterns like event-driven architecture, task queues, sagas, circuit breakers, and transactional outboxes. It’s programming on a higher level of abstraction, where you don’t have to be concerned about transient failures like server crashes or network issues. It opens up new possibilities like:

Storing state in local variables instead of a database, because local variables are automatically stored for us
Writing code that sleeps for a month, because we don’t need to be concerned about the process that started the sleep still being there next month, or resources being tied up for the duration
Functions that can run forever, and that we can interact with (send commands to or query data from)

Some examples of durable execution systems are Azure Durable Functions, Amazon SWF, Uber Cadence, Infinitic, and Temporal (where I work). At the risk of being less than perfectly objective, I think Temporal is the best of these options 😊.

Durable JavaScript

Now that we’ve gone over consistency in distributed systems and what durable execution is, let’s look at a practical example. I built this food delivery app to show what durable code looks like and what problems it solves:

temporal.menu

Don’t blame me for the logo—that’s just what Stable Diffusion gives you when you ask it for a durable delivery app logo. 🤷‍♂️😄

The app has four main pieces of functionality:

Create an order and charge the customer
Get order status
Mark an order picked up
Mark an order delivered

When we order an item from the menu, it appears in the delivery driver site (drive.temporal.menu), and the driver can mark the order as picked up, and then as delivered.

All of this functionality can be implemented in a single function of durable JavaScript or TypeScript. We’ll be using the latter—I recommend TypeScript and our library is named the TypeScript SDK, but it’s published to npm as JavaScript and can be used in any Node.js project.

Create an order

Let’s take a look at the code for this app. We’ll see a few API routes but mostly go over each piece of the single durable function named order. If you’d like to run the app or view the code on your machine, this will download and set up the project:

npx @temporalio/create@latest --sample food-delivery

When the user clicks the order button, the React frontend calls the createOrder mutation defined by the tRPC backend. The createOrder API route handler creates the order by starting a durable order function. Durable functions—called Workflows—are started using a Client instance from @temporalio/client, which has been added to the tRPC context under ctx.temporal. The route handler receives a validated input (an object with a productId number and orderId string) and it calls ctx.temporal.workflow.start to start an order Workflow, providing input.productId as an argument:

apps/menu/pages/api/[trpc].ts

import { initTRPC } from '@trpc/server'
import { z } from 'zod'
import { taskQueue } from 'common'
import { Context } from 'common/trpc-context'
import { order } from 'workflows'

const t = initTRPC.context<Context>().create()

export const appRouter = t.router({
  createOrder: t.procedure
    .input(z.object({ productId: z.number(), orderId: z.string() }))
    .mutation(async ({ input, ctx }) => {
      await ctx.temporal.workflow.start(order, {
        workflowId: input.orderId,
        args: [input.productId],
        taskQueue,
      })

      return 'Order received and persisted!'
    }),

The order function starts out validating the input, setting up the initial state, and charging the customer:

packages/workflows/order.ts

type OrderState = 'Charging card' | 'Paid' | 'Picked up' | 'Delivered' | 'Refunding'

export async function order(productId: number): Promise<void> {
  const product = getProductById(productId)
  if (!product) {
    throw ApplicationFailure.create({ message: `Product ${productId} not found` })
  }

  let state: OrderState = 'Charging card'
  let deliveredAt: Date

  try {
    await chargeCustomer(product)
  } catch (err) {
    const message = `Failed to charge customer for ${product.name}. Error: ${errorMessage(err)}`
    await sendPushNotification(message)
    throw ApplicationFailure.create({ message })
  }

  state = 'Paid'

Any functions that might fail are automatically retried. In this case, chargeCustomer and sendPushNotification both talk to services that might be down at the moment or might return transient error messages like “Temporarily unavailable.” Temporal will automatically retry running these functions (by default indefinitely with exponential backoff, but that’s configurable). The functions can also throw non-retryable errors like “Card declined,” in which case they won’t be retried. Instead, the error will be thrown out of chargeCustomer(product) and caught by the catch block; the customer receives a notification that their payment method failed, and we throw an ApplicationFailure to fail the order Workflow.

Get order status

The next bit of code requires some background: Normal functions can’t run for a long time, because they’ll take up resources while they’re waiting for things to happen, and at some point they’ll die when we deploy new code and the old containers get shut down. Durable functions can run for an arbitrary length of time for two reasons:

They don’t take up resources when they’re waiting on something.
It doesn’t matter if the process running them gets shut down, because execution will seamlessly be continued by another process.

So although some durable functions run for a short period of time—like a successful money transfer function—some run longer—like our order function, which ends when the order is delivered, and a customer function that lasts for the lifetime of the customer.

It’s useful to be able to interact with long-running functions, so Temporal provides what we call Signals for sending data into the function and Queries for getting data out of the function. The driver site shows the status of each order by sending Queries to the order functions through this API route:

apps/menu/pages/api/[trpc].ts

  getOrderStatus: t.procedure
    .input(z.string())
    .query(({ input: orderId, ctx }) => ctx.temporal.workflow.getHandle(orderId).query(getStatusQuery)),

It gets a handle to the specific instance of the order function (called a Workflow Execution), sends the getStatusQuery, and returns the result. The getStatusQuery is defined in the order file and handled in the order function:

packages/workflows/order.ts

import { defineQuery, setHandler } from '@temporalio/workflow'

export const getStatusQuery = defineQuery<OrderStatus>('getStatus')

export async function order(productId: number): Promise<void> {
  let state: OrderState = 'Charging card'
  let deliveredAt: Date

  // …

  setHandler(getStatusQuery, () => {
    return { state, deliveredAt, productId }
  })

When the order function receives the getStatusQuery, the function passed to setHandler is called, which returns the values of local variables. After the call to chargeCustomer succeeds, the state is changed to ’Paid’, and the driver site, which has been polling getStatusQuery, gets the updated state. It displays the “Pick up” button.

Picking up an order

When the driver taps the button to mark the order as picked up, the site sends a pickUp mutation to the API server, which sends a pickedUpSignal to the order function:

apps/driver/pages/api/[trpc].ts

  pickUp: t.procedure
    .input(z.string())
    .mutation(async ({ input: orderId, ctx }) => 
      ctx.temporal.workflow.getHandle(orderId).signal(pickedUpSignal)
    ),

The order function handles the Signal by updating the state:

packages/workflows/order.ts

export const pickedUpSignal = defineSignal('pickedUp')

export async function order(productId: number): Promise<void> {
  // …

  setHandler(pickedUpSignal, () => {
    if (state === 'Paid') {
      state = 'Picked up'
    }
  })

Meanwhile, further down in the function, after the customer was charged, the function has been waiting for the pickup to happen:

packages/workflows/order.ts

import { condition } from '@temporalio/workflow'

export async function order(productId: number): Promise<void> {
  // …

  try {
    await chargeCustomer(product)
  } catch (err) {
    // …
  }

  state = 'Paid'

  const notPickedUpInTime = !(await condition(() => state === 'Picked up', '1 min'))
  if (notPickedUpInTime) {
    state = 'Refunding'
    await refundAndNotify(
      product,
      '⚠️ No drivers were available to pick up your order. Your payment has been refunded.'
    )
    throw ApplicationFailure.create({ message: 'Not picked up in time' })
  }

await condition(() => state === 'Picked up', '1 min') waits for up to 1 minute for the state to change to Picked up. If a minute goes by without it changing, it returns false, and we refund the customer. (Either we have very high standards for the speed of our chefs and delivery drivers, or we want the users of a demo app to be able to see all the failure modes 😄.)

Delivery

Similarly, there’s a deliveredSignal sent by the “Deliver” button, and if the driver doesn’t complete delivery within a minute of pickup, the customer is refunded.

packages/workflows/order.ts

export const deliveredSignal = defineSignal('delivered')

export async function order(productId: number): Promise<void> {
  setHandler(deliveredSignal, () => {
    if (state === 'Picked up') {
      state = 'Delivered'
      deliveredAt = new Date()
    }
  })

  // …

  await sendPushNotification('🚗 Order picked up')

  const notDeliveredInTime = !(await condition(() => state === 'Delivered', '1 min'))
  if (notDeliveredInTime) {
    state = 'Refunding'
    await refundAndNotify(product, '⚠️ Your driver was unable to deliver your order. Your payment has been refunded.')
    throw ApplicationFailure.create({ message: 'Not delivered in time' })
  }

  await sendPushNotification('✅ Order delivered!')

If delivery was successful, the function waits for a minute for the customer to eat their meal and asks them to rate their experience.

  await sleep('1 min') // this could also be hours or even months

  await sendPushNotification(`✍️ Rate your meal. How was the ${product.name.toLowerCase()}?`)
}

After the final push notification, the order function’s execution ends, and the Workflow Execution completes successfully. Even though the function has completed, we can still send Queries, since Temporal has the final state of the function saved. And we can test that by refreshing the page a minute after an order has been delivered: the getStatusQuery still works and “Delivered” is shown as the status:

Summary

We’ve seen how a multi-step order flow can be implemented with a single durable function. The function is guaranteed to complete in the presence of failures, including:

Temporary issues with the network, data stores, or downstream services
The process running the function failing
The underlying Temporal services or database going down

This addressed a number of distributed systems concerns for us, and meant that:

We could use local variables instead of saving state to a database.
We didn’t need to set timers in a database for application logic like canceling an order that takes too long or for the built-in functionality of retrying and timing out transient functions like chargeCustomer.
We didn’t need to set up a job queue that workers polled, either for progressing to the next step or picking up unfinished tasks that were dropped by failed processes.

In the next post, we look at more of the delivery app’s code and learn how Temporal is able to provide us with durable execution.

If you have any questions, I would be happy to help! Temporal’s mission is helping developers, and I also personally find joy in it 🤗. I’m @lorendsr on Twitter, I answer (and upvote 😄) any StackOverflow questions tagged with temporal-typescript, and am @loren on the community Slack 💃.

Learn more

To learn more, I recommend these resources:

Video: Intro to Temporal and using the TypeScript SDK
Some common use cases
TypeScript SDK docs: t.mp/ts
TypeScript API reference: t.mp/ts-api
TypeScript tutorials

More blog posts about our TypeScript SDK:

Thanks to Jessica West, Brian Hogan, Amelia Mango, and Jim Walker for reading drafts of this post.