Forem: Anton Mihaylov

Implementing a Retry and DLQ Strategy in NATS JetStream

Anton Mihaylov — Sat, 29 Nov 2025 22:00:00 +0000

If you are moving from RabbitMQ or Kafka to NATS JetStream, you usually feel a sense of relief. The configuration is simpler, the binaries are smaller, and the performance is excellent.

But eventually, you hit the "Day 2" reality of event-driven architectures: Message Failures.

When a consumer fails to process a message, you can't just drop it. You need a retry policy, and eventually, a Dead Letter Queue (DLQ). While NATS provides all the primitives to build this, it does not provide a "system" out of the box. You have to wire it yourself.

This guide walks through how to architect a robust DLQ flow using native JetStream features, and how to manually handle message replays when things go wrong.

Prerequisites

NATS Server with JetStream enabled (nats-server -js)
nats CLI installed
Basic understanding of NATS streams and consumers

The Architecture of Failure

In JetStream, a DLQ isn't a special checkbox; it's just another Stream. To set this up correctly, you need a specific topology involving MaxDeliver, proper acknowledgment handling, and advisory message capture.

1. The Retry Policy

First, your consumer configuration determines how many times NATS attempts to deliver a message before giving up.

When you configure your consumer, pay attention to MaxDeliver and BackOff. A linear retry is rarely what you want. You want a geometric backoff to prevent thundering herd scenarios on your database.

nats consumer add ORDERS processing_worker \
  --max-deliver=5 \
  --backoff=1s,5s,1m,5m,10m

Important: The BackOff configuration only applies to acknowledgment timeouts, not to explicit NAK responses. When a consumer sends a Nak(), the message is redelivered immediately. To add delay on an explicit NAK, use NakWithDelay() in your client code:

// Immediate redelivery
msg.Nak()

// Delayed redelivery (recommended for transient failures)
msg.NakWithDelay(5 * time.Second)

2. The Dead Letter Stream

Once MaxDeliver is reached, the consumer stops attempting to deliver that specific message. The message is not deleted—it remains in the stream but is simply skipped by that consumer. To capture these failures, we use JetStream's advisory system.

When a message exhausts its delivery attempts, NATS publishes an advisory to:

$JS.EVENT.ADVISORY.CONSUMER.MAX_DELIVERIES.<STREAM>.<CONSUMER>

Similarly, when your application explicitly terminates a message using Term(), an advisory is published to:

$JS.EVENT.ADVISORY.CONSUMER.MSG_TERMINATED.<STREAM>.<CONSUMER>

Setting Up the DLQ Stream

Create a stream that captures these advisory messages:

nats stream add DLQ_ORDERS \
  --subjects='$JS.EVENT.ADVISORY.CONSUMER.MAX_DELIVERIES.ORDERS.*,$JS.EVENT.ADVISORY.CONSUMER.MSG_TERMINATED.ORDERS.*' \
  --storage=file \
  --retention=limits \
  --max-age=72h

The advisory payload looks like this:

{
  "type": "io.nats.jetstream.advisory.v1.max_deliver",
  "id": "EgGS1FReM5Yy1I7L9UXs4W",
  "timestamp": "2023-12-24T14:38:13.107632319Z",
  "stream": "ORDERS",
  "consumer": "processing_worker",
  "stream_seq": 6,
  "deliveries": 5
}

The stream_seq field is critical—it tells you exactly which message failed, and you can retrieve it directly:

nats stream get ORDERS 6

3. Using Term for Explicit Dead-Lettering

Instead of waiting for MaxDeliver to be exhausted, your consumer can explicitly terminate a message using Term(). This immediately stops redelivery and publishes an advisory.

Use this when your application can determine a message is permanently unprocessable:

// Message is malformed or missing required data
if !isValidPayload(msg.Data()) {
    msg.Term()  // Immediately dead-letter, don't retry
    return
}

The "Hard Way" (Manual) Replay Workflow

Let's assume you have a setup where failed messages have triggered advisories in DLQ_ORDERS.

It's 3:00 AM. An invalid payload caused 500 messages to hit the DLQ. You've pushed a hotfix to the consumer code to handle this payload. Now, you need to replay those 500 messages.

NATS does not have a "Replay" button. You have to do this via the CLI or a custom script. Here is the manual workflow for replaying messages without data loss.

Step 1: Inspect the DLQ Advisories

First, view the advisory messages to understand what failed:

nats stream view DLQ_ORDERS 10

Each advisory contains the stream_seq of the original message. To inspect the actual failed message:

nats stream get ORDERS <stream_seq>

Look at the headers and payload to understand why processing failed.

Step 2: Republishing

You need to read the failed messages and publish them back to the original subject (orders.created), not the DLQ subject.

Note: The nats stream backup and nats stream restore commands are designed for disaster recovery, not for selective message replay. They create binary snapshots and restore to the same stream, which isn't what we want here.

For a single message, you can replay it directly:

seq=6  # The stream_seq from the advisory
SOURCE_STREAM="ORDERS"
msg=$(nats stream get $SOURCE_STREAM "$seq" --json)
subject=$(echo "$msg" | jq -r '.subject')
echo "$msg" | jq -r '.data' | base64 -d | nats pub "$subject"

For batch replay of all DLQ advisories:

#!/bin/bash
# Replay all failed messages from DLQ advisories

DLQ_STREAM="DLQ_ORDERS"
SOURCE_STREAM="ORDERS"

# Get both first and last sequence numbers
info=$(nats stream info "$DLQ_STREAM" --json)
first_seq=$(echo "$info" | jq -r '.state.first_seq')
last_seq=$(echo "$info" | jq -r '.state.last_seq')

for i in $(seq "$first_seq" "$last_seq"); do
    # Get advisory and extract source stream sequence
    advisory=$(nats stream get "$DLQ_STREAM" "$i" --json 2>/dev/null) || continue
    stream_seq=$(echo "$advisory" | jq -r '.data' | base64 -d | jq -r '.stream_seq')

    # Get original message
    original=$(nats stream get "$SOURCE_STREAM" "$stream_seq" --json 2>/dev/null) || continue
    subject=$(echo "$original" | jq -r '.subject')
    data=$(echo "$original" | jq -r '.data' | base64 -d)

    # Republish
    echo "$data" | nats pub "$subject"
    echo "Replayed seq=$stream_seq to $subject"
done

The Friction Points

If you are doing this manually during an incident, you face several issues:

Deduplication edge case: If you use explicit Nats-Msg-Id headers for exactly-once publishing AND you're replaying within the dedup window (default 2 minutes), the message will be rejected. In practice, DLQ messages are usually hours old, so this rarely matters.

Ordering: If strict ordering matters, replaying from a script breaks the global order relative to new incoming messages.

Visibility: While running your script, you have no visual confirmation of the drain rate other than repeatedly running nats stream info.

The Multi-Consumer Problem

There's a subtle but serious issue with the replay workflow above: the original message is still in the source stream.

When you replay a message by publishing it to the original subject, you're creating a new message. If multiple consumers subscribe to the same stream—common in microservice architectures where different services react to the same events—this causes duplicate processing.

Consider this scenario:

ORDERS stream has two consumers: billing_worker and notification_worker
billing_worker fails to process message seq=42 (database timeout) and it hits MaxDeliver
notification_worker successfully processed seq=42 and sent a confirmation email
You replay seq=42 by publishing to orders.created
Both consumers receive the replayed message
notification_worker sends a duplicate email

This is particularly dangerous for side-effecting operations: duplicate charges, duplicate notifications, or duplicate API calls to external systems.

Workaround 1: Dedicated Replay Subject

Configure your consumers to subscribe to multiple subjects, including a replay-specific one:

# Original consumer subscribes to both subjects
nats consumer add ORDERS billing_worker \
  --filter='orders.created,orders.created.replay.billing'

When replaying, publish to the consumer-specific replay subject:

echo "$data" | nats pub "orders.created.replay.billing"

This ensures only the failed consumer receives the replayed message.

Workaround 2: Replay Headers with Consumer Filtering

Add metadata to replayed messages so consumers can detect and handle them:

# Add replay headers when republishing
echo "$data" | nats pub "$subject" \
  --header="X-Replay: true" \
  --header="X-Original-Seq: $stream_seq" \
  --header="X-Target-Consumer: billing_worker"

Consumers check the header and skip if they're not the target:

if msg.Header.Get("X-Replay") == "true" {
    target := msg.Header.Get("X-Target-Consumer")
    if target != "" && target != consumerName {
        msg.Ack()  // Not for us, acknowledge and skip
        return
    }
}

Workaround 3: Idempotent Consumers (The Right Way)

The most robust solution is designing consumers to be idempotent from the start. Store a processed message log keyed by stream_seq or a business-level identifier:

func processOrder(msg jetstream.Msg) error {
    orderID := extractOrderID(msg.Data())

    // Check if already processed
    if wasProcessed(orderID) {
        return msg.Ack()  // Idempotent: safe to skip
    }

    // Process and mark as done atomically
    if err := processAndMarkDone(orderID, msg.Data()); err != nil {
        return msg.NakWithDelay(5 * time.Second)
    }

    return msg.Ack()
}

This protects against replays, network retries, and JetStream's at-least-once delivery semantics.

Automating the Pain

The workflow above—detect, inspect, script, replay—is standard practice. It works. But it is slow, and writing one-off scripts during an outage is stress-inducing.

This is exactly why we built StreamTrace.

We realized that managing NATS JetStream shouldn't require writing bash scripts at 3 AM just to move a message from one stream to another. StreamTrace is a lightweight UI that sits on top of your NATS clusters.

It turns the "Hard Way" into a UI workflow:

Visual DLQ Management: View your DLQ stream, click the specific messages (or "Select All"), and hit Replay.
Payload Inspection: Decode JSON, Text, or Hex payloads right in the browser to debug why the failure happened before you replay.
Real-time Progress: Watch messages drain from your DLQ without repeatedly running nats stream info.

It is a self-hosted Docker container that connects directly to your NATS server. No cloud data leakage, no complex setup.

If you are tired of writing restoration scripts, check out Stream Trace.

Which is the best Speech-to-text AI in 2022?

Anton Mihaylov — Mon, 19 Dec 2022 00:00:00 +0000

As Artificial Intelligence (AI) continues to evolve and become increasingly integrated into our everyday lives, speech to text is becoming more and more prevalent. In 2022, there will be a wide array of speech to text AI’s that are available for developers to use.

In this article I’ll share my findings on choosing the most suitable AI Speech-to-text engine.

In order to do that, let me first define a couple of criteria that I find most valuable. Those of course might be different for your needs, for example you may value ease of use more than cost, which will skew the decision in favor of cloud-based solutions.

Here are my main decision drivers, ordered by importance:

Result accuracy - Very important
Cost - Very important
Ease of use - Somewhat important
Performance - Somewhat important
Extra features - speaker recognition, punctuation, profanity masking, automatic chaptering - Nice to have

And here are the contenders:

Assembly AI
Amazon Transcribe
Google Speech To Text
Vosk
OpenAI’s Whisper

Note that I’m not affiliated in any way with any of these and the information bellow is the result of my personal findings when deciding which technology to use for a project. And I’m not even close to being a machine learning expert, so this is the point of view of an web developer that wants to simply use those technologies.

I used this YouTube video, which is around 10 minutes long and contains generic dialog as well as a lot of technical terms or names.

Best Python IDE: Vim, Emacs, PyCharm, or Visual Studio Code? | Guido van Rossum and Lex Fridman

Findings

1. Assembly AI

AssemblyAI | #1 API Platform for AI Models

Assembly AI is a cloud-based solution that provides not only text-to-speech but also a variety of other NLP services.

Here’s a link to their playground that I used for testing it https://www.assemblyai.com/playground

Result accuracy - Amazing , almost no issues, even when dealing with technical terms or names. It capitalizes accurately and has proper punctuation.

Cost - it’s a cloud-based service, so it’s natural to be quite expensive. It’s comparable to other cloud based solutions at $0.015/minute , but that’s only for the core transcription service. If you need the extra Audio Intelligence features you need quite a bit more budget with the cost of $0.035/minute. But of course this could be worth it for the result accuracy and the peace of mind that comes with using a managed service
Ease of use - the plus side of cloud-based solutions is that it’s very easy to use them - you just call an API and you’re done - no infrastructure to manage.
Performance - they advertise it as around 20% of the running time, which sounds very good
Extra features - it has the fullest audio intelligence feature set of all options, if you have the budget for the more expensive plan

Verdict: Assembly AI’s quality and large feature set would be my choice if I needed a top-notch cloud-based solution and budget wasn’t a problem.

2. Amazon Transcribe

Amazon Transcribe – Speech to Text - AWS

This is AWS’s speech to text solution, which might be the obvious choice for many solutions that are already in the AWS ecosystem in one way or another.

Result accuracy - it’s a decent , way worse than Assembly AI, but workable enough for simple use cases. It has proper punctuation and mostly good capitalization. It doesn’t handle technical terms or names well

Cost - it is also a cloud-based solution, so it’s quite costly at $0.024/minute. But the cost goes down the more minutes you use per month, down to $0,0077/minute if you use more than 10 million minutes per month. Which is a good deal if you have a lot of audio to transcribe
Performance - very good - the 10-minute test video was processed for 2 minutes and 20 seconds, which is around 20% of the running time.
Extra features - you have profanity masking, speaker recognition out of the box

Verdict: I would pick Amazon Transcribe if there’s a large volume of audio to transcribe each month and it doesn’t matter that the quality is not as good as it can be.

3. Google Speech-to-Text

Speech-to-Text: Automatic Speech Recognition | Google Cloud

Google’s solution is widely known and, in general, Google has a reputation of delivering top-notch AI solutions. There’s a free demo on the page where I uploaded my video and saw the results right away.

Result accuracy - Good , not as good as AssemblyAI, but definitely better than Amazon. Handles technical terms good most of the time, but misses them here and there.

Cost - it is also a cloud-based solution, so it’s quite costly at $0.016/minute , and that is the cheaper option, which allows Google to record your sent audio and use it to “help improve the machine learning models”. It has as more expensive option at $0.024/minute , which is more private.
Ease of use - again, as a cloud-based solutions it’s very easy to use
Performance - the test I did had decent performance , considering it’s a free demo page - a 1 minute audio (that was the max demo length) was done in around 30s, with speaker recognition and punctuation turned on
Extra features - you get speaker recognition and punctuation readily available
Other pros - it’s Google, so there’s a higher chance that it will be more available and reliable as a service than other providers given their massive infrastructure and talent
Other cons - it’s Google, and a lot of people have privacy concerns with that.

Verdict: Google’s solution is a pretty stable choice if you want a cloud based solution, but you’d have to really consider if going with Assembly AI isn’t worth it, since it has better quality.

4. Vosk

VOSK Offline Speech Recognition API

This is the first open source option on the list. As such, it’s completely free to use, but you have to manage your own infrastructure to do that. This might be considered a pro to some people, because of the control and privacy, and a con to other, because of the effort needed to set it up and manage it. But it has a pretty active community, so I’m sure if you encounter any problems you will not find it hard to find solutions.

Another big plus to Vosk is that there’s a Web Assembly version, which means you can run it in the browser if you need. It’s not as fast and it required the user to download a pretty large model file, but it can be a great way to save costs, be private, and have a performant solution, by offloading the work to the user’s machine. You can test the web demo here https://ccoreilly.github.io/vosk-browser/

Result accuracy - decent , similar to Amazon Transcribe. Note that I tested larger models, the smaller 40mb models is not that good and is worse than Amazon Transcribe. There is no capitalization or punctuation, but if you want to tinker with it you can use other wasm solutions to handle that

Cost - it’s a self-hosted solution, so in general it’s less expensive than cloud-based counterparts. But of course you have to factor in the cost of managing it. You’re not really paying for a service or an application, you’d be paying for the servers than run it. Or you can run it client-side in the browser completely free
Ease of use - as self-hosted solution it’s harder to set it up and you have to manage the servers that run it. You don’t have to be a machine learning expert to use it, any mid backend engineer should be able to set it up, but you can’t compare it to just calling an external API.
Performance - I tested the WASM version, so the actual one might be a bit faster. In the browser I got a speed of around 20% of the running time on my Macbook M1 Pro, which is very good and comparable to the cloud vendors.

Verdict: it’s a decent open source solution and a great one if you need to run it in the browser. But I ’ d look more at the next option otherwise

5. Whisper by OpenAI

GitHub - openai/whisper: Robust Speech Recognition via Large-Scale Weak Supervision

Whisper is a relatively new entry, made by the OpenAI team, which are behind one of the most cutting edge technologies in the AI space right now

It’s also open source so, you can read the introduction to Vosk if you skipped it.

There’s a C++ port of Whisper, that is compilable to Web Assembly, but it’s not official and is just a hobby project for now https://github.com/ggerganov/whisper.cpp . So you can technically run it in the browser as well, but it’s questionable if that’s the right choice if you’re looking for a long-term maintainable and stable solution. Of course if needed you can also take the initiative to help the project if you have enough ML, C++ and WASM knowledge. There’s one big benefit of using that instead of the Vosk’s wasm version - that the Whisper’s models are a lot smaller, which is a very good thing, because you have to download them on every browser that uses it.

Result accuracy - amazing , comparable to Assembly AI, even with the smaller models. And you can increase the timing precision, so that, for example you get the timestamps of each word.

Cost - again, as a self-hosted solution, it’s less expensive than cloud-based counterparts. But of course you have to factor in the cost of managing it. You’re not really paying for a service or an application, you’d be paying for the servers than run it. Or you can run it client-side in the browser completely free
Ease of use - again, as self-hosted solution it’s harder to set it up and you have to manage the servers that run it
Performance - around 20% of the running time on my Macbook M1 Pro, which is very good and comparable to the cloud vendors.
Extra features - no speaker recognition, but you can use other tools to do that https://github.com/openai/whisper/discussions/264 .

Verdict: it’s an amazing open source project and that would be my go-to option if you’re looking for a self-hosted or browser-based solution.

Conclusion

In conclusion, when it comes to finding the best Speech-to-Text AI for 2022, there are a variety of options to choose from, each with it’s own use cases. Deciding which to use depends on your requirements, budget and use case.

Assembly AI is a top-notch cloud-based solution that provides an excellent result accuracy and comes with a wide range of audio intelligence features.

Amazon Transcribe is a good choice for those who need to transcribe large volumes of audio, but have a lower requirement for accuracy.

Google Speech-to-Text offers good result accuracy, as well as speaker recognition and punctuation out of the box.

Vosk is a self-hosted open-source solution that’s comparable to Amazon Transcribe in quality.

Finally, Whisper by OpenAI is another open-source solution that offers amazing accuracy and is a great choice for those looking for a self-hosted or browser-based solution. This will be my choice for the project.

Unity Tutorial: Simple Aseprite to Unity workflow using an Automated Scripted Importer

Anton Mihaylov — Thu, 30 Jul 2020 18:41:33 +0000

If you develop games with Unity and make pixel art for them with Aseprite you probably did the same annoying thing over and over again, just like me:

Create pixel art in Aseprite
Export the image to a png/pngs
Open up Unity to import them, set up animations or load them into sprites
Realize you need to adjust pixel art images
Change them up in Aseprite
Export them again
Open up Unity, but realize that you have to re-do the animation, because you added one more frame now, or see that the reference to the sprite has been broken up
Fix that
Repeat

But a lot of this can be cut down to this:

Create pixel art in Aseprite, inside the Assets folder
Save the file and have Unity load sprites and animations automatically every time you change it
..that’s it

And this all can happen with the magic of the new Unity’s Scripted Importers API. (Which is still experimental by the way, but it gets the job done)

So, anyway, let’s see how setting up a project using a custom Scripted Importer for Aseprite looks like.

Creating the Aseprite file

Our first job of course is to create the pixel art in Aseprite.

We have our green guy here, with three animations: Running, Jump Charging, and Jumping. Nothing special for now, let’s save that file to Unity and see what happens.

Importing the Aseprite file in Unity

The player asset you see highlighted there is the .ase file that we saved earlier. When expanded, you can see that all of our frames are automatically exported from Aseprite and imported into Unity. And, which is more, you have the animations on the top there, ready to be plugged in in your Animation controller. And all I did was save the Aseprite file to the Assets folder.

Let’s add an animation to an object, so we can see it in action.

Automatically importing changes

Let’s say that we now want to change up the pixel art, for example – let’s make it purple.

We hop over to Aseprite and adjust the hue a bit and we have this:

Now, if we go back to unity (I’m still in Play mode by the way), we see that it is automatically re-imported with the updated pixel art, without doing anything

Auto Importer for Aseprite

I’ve bundled up the Scripted Importer that you see in action and I made an asset on the asset store, so that not only I can benefit from it.

You can find it here:

Auto Importer for Aseprite LITE

Auto Importer for Aseprite PRO

With the only difference being that the PRO version supports animations and the lite doesn’t.

Hopefully they can be of use to you too!