Forem: Ivan Juren

Good Intentions Aren’t Enough — Measure!

Ivan Juren — Sun, 09 Nov 2025 13:44:50 +0000

Good thoughts and intuition can lead to good results, but only benchmarks lead to great results.

The Problem

Once upon a time, I had the opportunity to build a custom data structure.

We needed something like a queue to act as a cache — a rolling buffer with a constant size, always filled with the latest data as the oldest expires.

The requirements sounded simple:

Single writer, multiple concurrent readers
Readers iterate safely while data is being written
As little garbage and synchronization overhead as possible

Simple, right?

The Obvious Candidates

Naturally, the first candidates were Java’s built-in structures:

Structure	Pros	Cons
ArrayList	Simple, contiguous	O(n) to remove oldest element
LinkedList	Fast removals	Creates garbage, poor cache locality
ArrayDeque	Nearly perfect	Not thread-safe, fails under concurrent modification

Each had a deal-breaking flaw for our case.

The Custom Design

So, led by good intentions (and confidence), I decided to build my own —

the RollingBuffer.

Core idea:

Backed by a plain array → cache locality
Preallocate and reuse buckets/objects → improves memory locality and avoids garbage
A single volatile field for synchronization
Minimal API surface:

void put(long timestamp, V value);
Iterator<V> getIterator(long startTimestamp);

The iterator would snapshot the volatile write index and iterate safely up to that point.

No resizing, no garbage, no allocations during runtime.

Perfect in theory.

The First Hurdles

Which bucket to write to?

What if there’s a gap in timestamps?

Example: with 10 buckets, data for timestamps 20–30 could yield something like:

[20, 11, 22, 23, 24, …]

I decided to handle this during iteration — assuming branch prediction would make it cheap.
Synchronization when wrapping around

What if an iterator runs while the writer wraps around the array?

Solution: introduce two parameters:
- maxBuckets — total array capacity
- exposedBuckets — visible portion to readers

This created a small grace zone to prevent corruption.

First Benchmarks — Great News?

I wrote the first version and ran JMH benchmarks.

Results looked amazing:

Iteration over N elements: tiny latency
Write throughput: tens of millions ops/sec

Until I compared it against the real collections…

Reality Hits

Against the standard structures, my RollingBuffer was:

2.5× slower at iteration than ArrayDeque
~50% slower than LinkedList

Not what I expected.

I suspected the iteration-time gap check was the bottleneck — so I reworked it.

The New Approach

Instead of trying to write to the “correct” bucket (skipping gaps),

I wrote to the next available bucket every time.

That simplified iteration — no runtime checks, just a linear scan.

But now, given a startTimestamp, I needed to find the right bucket efficiently.

Enter binary search.

Even with wrap-around, modular arithmetic made it work beautifully.

Result: ~15% faster iteration.

Better, but still not stellar.

The Real Culprit

Then I found the real performance killer:

nextWriteIndex = (nextWriteIndex + 1) % totalBuckets;

and in the iterator:

nextIndex = (nextIndex + 1) % totalBuckets;

Looked harmless.

But % is slow compared to a conditional branch.

I replaced it with:

nextWriteIndex = (nextWriteIndex + 1 == totalBuckets) ? 0 : writnextWriteIndexeIndex + 1;
nextIndex = (nextIndex + 1 == totalBuckets) ? 0 : nextIndex + 1;

Result?

3× faster iteration.

Yes — just by removing a single % operator.

Final Results

Write throughput:

≈ 78 million ops/sec — about half as fast as ArrayDeque or LinkedList,

but thread-safe and similar to LMAX Disruptor performance in single writter (ended up similar in design as well).

Read throughput:

Comparison	RollingBuffer Speed
vs `ArrayList`	1.3× faster
vs `ArrayDeque`	1.4× faster
vs `LinkedList`	2.3× faster

Iteration:

Faster than both ArrayList and ArrayDeque,

fully thread-safe, and supports concurrent modification.

Not bad at all. 😎

Btw. check out this interactive graph I've made with Claude AI

The Lesson

Would I have figured this out without measuring?

No chance.

Is this level of optimization worth it for most code?

Absolutely not.

For 99.9% of use cases, built-ins are fine.

Final Thought

Intuition gets you close.

Benchmarks get you the truth.

Is this level of performance optimization really worth it?

For most people, jobs, or tasks — it’ll drown in the sea of other inefficiencies.

And not just for most — it’s like 95%+, easily.

But when it comes to the hot path of the system — the core where performance truly matters — this is where you’ll find excellence.

This process was invaluable.

I learned more about how the CPU, Java Memory Model, and JVM actually behave than any book or course could have taught me.

Github repo: https://github.com/JurenIvan/rolling-buffer

You can reason your way to correctness.

But you can only measure your way to performance.

Good Things (Compression) Take Time!

Ivan Juren — Wed, 29 Oct 2025 08:53:49 +0000

Inspired by Anton Putra style of benchmarks (if you love graphs half as much as I do, you gotta check this guy out!), I decided to dig deeper into one of Kafka producer's most underrated yet powerful settings — linger.ms.

In this post, we'll explore what linger.ms does and how it impacts end-to-end latency, CPU usage, compression efficiency, and produce/fetch rates.

🧠 What Is `linger.ms` and Why Is It Important?

Every message sent to Kafka isn't just the payload — it's wrapped with metadata (topic name, partition, key, timestamp, checksums, headers, etc.).
Sending messages one by one would therefore carry significant network and serialisation overhead.

To optimise this, Kafka producers can bundle multiple records into a single batch, reducing the per-message overhead and improving throughput.
Now, the key question becomes: how big should that batch be, and when should it be sent?
That's exactly where linger.ms comes into play.

linger.ms defines how long the Kafka producer should wait before sending a batch of records to the broker.

If linger.ms>0, the producer waits up to that duration to fill the batch — allowing more messages to accumulate and be sent together.
If linger.ms=0, records are sent immediately when the batch is full or as soon as a message arrives.

The result? Fewer network calls, better compression efficiency, and lower CPU usage — at the cost of slightly higher latency.

The Tradeoff

Lower linger.ms → lower latency, more overhead, higher CPU cost, worse compression.
Higher linger.ms → higher latency, less overhead, lower CPU cost, better compression.

It's all about balancing throughput and latency depending on your use case.

🔍 Methodology

To measure the impact, I set up a 3-broker Kafka cluster using Docker Compose with the following stack:

Kafka 4.0.0 (KRaft mode)
JMX Exporter → Prometheus → Grafana for metrics
Spring Boot producer and consumer
The producer sends JSON payloads (~400B each) to a topic (replication:3, min ISR:2, acks: all) with adjustable linger.ms.
The consumer stores messages in memory and calculates the difference in time → end-to-end latency that's reported separately.

The latency is measured using timestamps on both sides (System.currentTimeMillis()), so it reflects true client-perceived delay — not just broker-side performance.

Disclaimers: I'm doing all of this on one machine. In production, you'd want brokers on separate racks to reduce the chance of them all going down at the same time. To keep things simple, there's only one producer and one consumer. Since usual Kafka workloads have multiple consumers interested in the same stream, producer-side optimisations this way might exaggerate the impact you'd expect in production.

Stay tuned for follow-ups where I'll cover impactful consumer-side properties!

📊 Results and Analysis

1️⃣ End-to-End Latency

Higer resolution picture

Notice how decreasing linger.ms seems like a big win — the median latency drops from around 30 ms to 10 ms, perfectly
matching the 20 ms linger.ms we removed.

At first glance, that feels like a major improvement, but the long tail remains. The distribution still stretches into higher latencies, meaning not everything got faster. Usually, when deployed to the cloud with more distance between brokers, replication takes a bit longer. Since consumers are only served replicated messages, the effect on e2e latency isn't that big.

By the way, in Kafka clients 4.0, the default linger.ms is set to 5ms (from the old default of 0ms). Let's see why that might be a good idea.

2️⃣ CPU Usage

Higer resolution picture

Here's the CPU impact — and it's quite telling.

After decreasing linger.ms from 20 ms to 0, both the producer and brokers suddenly had to deal with many more, smaller batches. That means more frequent network calls, more compression operations, and generally more CPU churn.

With linger.ms=20, batching allowed Kafka to work smarter — fewer system calls, fewer packets, fewer yet better compressions per second.
At linger.ms=0, the producer becomes a bit of a machine gun, firing messages as soon as they arrive.

We could counter some of the consumer-side impact with settings like fetch.min.bytes, but let's leave that deep dive for another time. ⏳

3️⃣ Produce/Fetch Count

Higer resolution picture

Higer resolution picture

The produce rate going up is the direct consequence of reducing linger.ms. Instead of batching to amortise the overhead, almost every event is a produce request. This is why CPU churn is so much higher.

On the other side, the fetch rate also rises. Given that the consumer by default waits until there's one byte ready (the Kafka broker returns data as soon as there is an event to be returned) and given that something lands all the time instead of once every 20ms, the consumer now has a lot more work as well.

fetch.min.bytes and fetch.max.wait.ms can be real lifesavers here. (Teasing the sequel once again 😉)

4️⃣ Bytes In Total

Higer resolution picture

(Grand finale for the title metaphor 🎺🎺)

You can see an increase in bytes in across brokers after raising linger.ms.

Bigger batches mean better compression efficiency, but to have nice batches, you have to wait a little bit.
Compression algorithms work better with larger data blocks. Since data is stored and served as it arrives (check out Kafka zero copy), increasing linger.ms can reduce your storage and networking requirements. Pretty neat!

🧩 Summary

Aspect	Lower `linger.ms`	Higher `linger.ms`
Latency	✅ Lower	🚫 Higher
Broker CPU Load	🚫 Higher	✅ Lower
Client CPU Load	🚫 Higher	✅ Lower
Compression Ratio	🚫 Worse	✅ Better
Network Throughput	🚫 Larger	✅ Smaller

🔥 Turning Up the Heat

So far, we've been working with a relatively modest message rate. But what happens when we really stress the system?

I ramped up the message volume 5x and ran a progressive test: starting with linger.ms=0, then after ~5 minutes switching to linger.ms=5, another 5 minutes at linger.ms=10, and finally 5 minutes at linger.ms=20.

Higer resolution picture

Higer resolution picture

Under high load, we can observe the following CPU gains:

Producer CPU drops by ~25% as you move from linger.ms=0 to linger.ms=20
Broker CPU drops by ~40% across the same range

Given that we're pushing 5x the messages, it seems that cpu usage is a function of the number of produceRequests more than the number of messages itself.

But here's the really interesting finding: end-to-end latency doesn't improve with lower linger.ms values under high load. In fact, it gets worse.

Higer resolution picture

When linger.ms=0, the e2e latency is actually higher than when linger.ms=5 or linger.ms=10. It's as bad or even worse than linger.ms=20. Why? Because all those network calls, compression operations, and context switches create so much overhead that the system itself becomes the bottleneck. You're not getting faster responses — you're just thrashing the CPU while simultaneously degrading throughput.

This is a crucial insight: at scale, lower linger.ms doesn't mean lower latency. The overhead of frequent small batches actually increases latency perception through system congestion, while also burning CPU cycles that could be spent on actual message processing.

For completeness, check out how compression becomes better, resulting in more compact data.

Higer resolution picture

🚀 Key Takeaways

linger.ms is a powerful lever for balancing throughput vs latency.
Setting it too low wastes CPU and bandwidth on small batches.
Setting it too high hurts responsiveness — especially for latency-sensitive workloads.
Tune it alongside batch.size for your message volume and rate.
Check your setup. The old default (before Kafka clients 4.0) was linger.ms=0; the new default is 5ms. This switch alone can already have a nice impact on your Kafka bill.
If you have high load with lots of messages, adding some linger.ms might lower your e2e latency.

In other words — good things take time 😉

The Art of Knowing When to Upgrade

Ivan Juren — Fri, 24 Oct 2025 07:08:42 +0000

I recently jumped on the JSpecify hype train. It promises to finally give Java a unified approach to null-safety. The dream!
It’s been around for a while, but apart from conference mentions, I haven’t really heard companies jumping into it. I think it's a shame, given how much potential it has to transform Java (and even its reputation), but that got me thinking...

When do I decide to adopt something new in my stack?

Over time, I realized that every “should we upgrade?” question boils down to timing, risk, and expected value. Here’s my quick mental model.

Area	Decision Frequency	Production/Business Approach	Experimental/Personal Approach
1. New Java Versions	Once every 6 months	Wait for LTS releases or notable performance/features gains in non-preview/final features .	Upgrade immediately to understand new features and see whether improvements matter.
2. New Framework Versions	Once every 2-3 months	Major version upgrade after the first patch release (e.g., 4.0.0 to 4.0.1) to avoid early rough edges. Minor in the next sprint (in about a month)	Upgrade immediately to familiarize yourself.
3. Tools like JSpecify	More than once a month	Consider community momentum, prioritize and validate your gains from it. Be pragmatic!	Trust your professional instinct; adopt if you believe in the future of the project. Don't jump on every train. Let others rough out the edges.

The Risk and the Reward

JSpecify is still in its infancy, and it’s not the first attempt to solve nullability in Java.

If you want a reminder of how many such efforts have faded, just look at this StackOverflow thread — it’s a graveyard of well-intentioned efforts.

And I think it actually might!
There’s a rare consensus between the community, library authors, and toolmakers (JetBrains, looking at you 👀) to push
for a unified standard.

The Best- and Worst-Case Scenarios

🚀 Best case:
The ecosystem embraces JSpecify, tooling evolves around it, and Java becomes a much safer, more expressive language.

📉Worst case:
The initiative loses momentum — but you still end up with well-defined annotations that serve as clear hints to
developers (and maybe future tooling).

So even in the “bad” case, you gain something. In the good case, you gain a lot. 📈

For me, that’s enough.

How Deep Should You Dive In?

Closed for Modification, Open for Extension!

Recently, I wrote a post on Rethinking Optional<?> where I suggested structural changes in how we handle
nullability.

The gist: There’s no need to hide nullable fields behind Optional if we can ensure compile-time checks.

That’s a brave (maybe even reckless) stance — because if the tooling doesn’t evolve alongside the language, you could
end up with code that’s worse than before.

We’re effectively writing a subset of Java here.

If this initiative dies off and the tooling stops verifying, those “nullable” annotations won’t give the hard guarantees that my blogpost relies on.
So my final verdict; don’t throw away three decades of hard-learned safety habits just yet., but use the added safety!

My Pointers

✅ Use JSpecify -> Every compile-time warning is one less NPE in production.
⚠️ Don’t ditch proven patterns -> Keep using validation, good API design, and clear contracts. JSpecify enhances them — it doesn’t replace them (yet).
🔮 Look ahead -> Java language arhitects are already considering nullability in Java itself. It's important to say that these are very! very! early drafts mailing list, but JSpecify is a great stepping stone for your project to be ready when Valhalla and nullability in language actually drop. When that happens, tools like OpenRewrite will make migrating from annotations to native constructs a breeze. You’ll already be halfway there — and safer along the way.

Final Thoughts

Adopting new tech is about balancing enthusiasm and pragmatism.

Sometimes you wait. Sometimes you jump. But if the cost of trying is low and the upside is huge — like with JSpecify — then experimenting isn’t just reasonable, it's responsible engineering.

Experiment, but with both eyes open.

🔗 Related Reads

Would you adopt JSpecify in your production code today or would you wait for IDEs and compilers to catch up? Curious to hear your take 👇

Hey, I did a thing, you might like it! Java JSpecify Exercise

Ivan Juren — Thu, 23 Oct 2025 08:35:29 +0000

Learning by doing is the most durable kind of learning.

I’ve put together a small repo with ~20 classes, organised into easy, medium, and hard levels to introduce you to JSpecify.

So after reading the JSpecify docs, try to annotate given classes, specify the nulls, and get some real hands-on JSpecify experience.

I believe it’s really important to understand the “type” semantics shift — what @Nullable actually means. The generics really piqued my interest.

If you want a sneak peek at the kind of reasoning you’ll be doing, check out my other short read where I question one of my favourite patterns for dealing with nulls: Rethinking Optional<?>

Going from this:

To this:

More concise and safer.

🔗 Repo: https://github.com/JurenIvan/jspecify-exercise

Can you see yourself using JSpecify in your projects?

Would love to hear what you think 👇

Rethinking Optional!

Ivan Juren — Wed, 22 Oct 2025 10:14:33 +0000

Compile-Time Safety vs Runtime Safety: JSpecify in Practice

The classic defense mechanism against the billion-dollar mistake in java?
Common wisdom: use Optional, runtime checks, and Objects.requireNonNull.

But none of these eradicated the problem.

Let’s look at a simple case — a UserProfile with an optional email and last login date — and see how the code changes when using JSpecify.

Without JSpecify: Optional All the Way

public final class UserProfile {

    private final String username;
    private final String email;
    private final Instant lastLogin;

    public UserProfile(String username,
                       String email,
                       Instant lastLogin) {
        Objects.requireNonNull(username, "Username must not be null!"); // must check the value to ensure correctness
        //alternatively, check nullness on each usage of username... 
        this.username = username;
        this.email = email;
        this.lastLogin = lastLogin;
    }   // do you test this kind of code? How do you ensure invariants don't break when changes happen?

    public String getUsername() { return username; }

    public Optional<String> getEmail() {    // returns optional so caller is forced to consider Optional.empty()
        return Optional.ofNullable(email);  // exists due to developers experience, but has runtime costs(instantiation, GC) 
    }

    public Optional<Instant> getLastLogin() {
        return Optional.ofNullable(lastLogin);  
    }
    // ... rest of the impl
}

And the consumer:

class MailSenderOptional {

    public void doSomething(UserProfile userProfile) {
        if (someOptional.isPresent() && userProfile.getLastLogin().isPresent()) {
            doSomeLogic(userProfile.getUsername(),
                        userProfile.getEmail().get(),
                        userProfile.getLastLogin().get());
        }
    }

    private void doSomeLogic(String username, String email, Instant lastLogin) {
        Objects.requireNonNull(username);   // given method is private, these checks might be excessive, but had it been 
        Objects.requireNonNull(email);      // public, why/when would caller care about invariants of this class?
        Objects.requireNonNull(lastLogin);  // If this call is deep down, developer will notice only at runtime.
        System.out.println("runtime safe: " + username + " " + email + " " + lastLogin);
    }
}

Here, everything compiles fine, even if you accidentally pass null to email.

The compiler won’t warn you — only a runtime exception can catch that.

Optional usage makes it safer in runtime, but verbose, hard to chain, and doesn’t truly make your API null-safe.

Btw. Have you even run into following scenario:

 public void unlucky(UserProfile userProfile) {
    Optional<String> email = userProfile.getEmail();
    if (email.isPresent()) { // ❌ NullPointerException! variable email is null! 
        // impl...
    }
}

Optionals give you certain safety, but it's not a guarantee. To be honest I've run into this just once in my life so this is more of a theoretical issue.

With JSpecify: Compile-Time Null Awareness

Now, let’s bring in JSpecify and its @NullMarked and @Nullable annotations.

@NullMarked
public final class UserProfile {

    private final String username;
    private final @Nullable String email;
    private final @Nullable Instant lastLogin;

    public UserProfile(String username,
                       @Nullable String email,
                       @Nullable Instant lastLogin) {
        // No need for requireNonNull — compiler checks this.
        this.username = username;
        this.email = email;
        this.lastLogin = lastLogin;
    }

    public String getUsername() { return username; }

    public @Nullable String getEmail() { return email; }    // no need for Optional here, caller is forced to consider nullability

    public @Nullable Instant getLastLogin() { return lastLogin; } // if you're a fan -> these can be generated by lombok 
}

Notice that we no longer have any custom code/checks and that now this class fits into record very nicely so it looks like

@NullMarked
public record UserProfile(String username, @Nullable String email, @Nullable Instant lastLogin) {
}

And the consumer...

@NullMarked
class MailSenderJSpecify {

    public void doSomething(UserProfile userProfile) {
        // ❌ Compile error! email and lastLogin are nullable strings
        // doSomeLogic(userProfile.username(), userProfile.email(), userProfile.lastLogin());  
        // developer has to tackle it right away! 

        if (userProfile.email() != null && userProfile.lastLogin() != null) {
            // ✅ compiler knows they email and lastLogin are non-null within this block.
            doSomeLogic(userProfile.username(),
                        userProfile.email(),
                        userProfile.lastLogin());
        }
    }

    private void doSomeLogic(String username, String email, Instant lastLogin) {
        // no need for runtime checks (or writting tests for it)
        System.out.println("compile safe: " + username + " " + email + " " + lastLogin);
    }
}

...is more concise and safer than ever before. You gotta love modern Java! 🥰

The Differences

Without JSpecify	With JSpecify
Compiler is unaware of nullability	Compiler knows what can be null
Optional used to signal nullability	Plain nullable fields, checked at compile time
Safety at runtime only	Safety enforced at compile time
Verbose API (`Optional.ofNullable(...)`)	Natural Java API with null-awareness
Code compiles even if nulls passed incorrectly	Code fails to compile if nulls aren’t handled

Conclusion

Null-related bugs are common causes of production incidents and put a lot of strain in the brain-context of the developer building stuff.

JSpecify brings the type system closer to what we intuitively mean in code:

“This might be null” vs “This can never be null.”

And it integrates cleanly — no new types, no wrapper objects, just annotations and compiler support.

TL;DR

Before JSpecify: You defend against null at runtime.
With JSpecify: You prevent null at compile time.

Compile-time safety beats runtime surprises. 📈

Btw. check my repo where I've created ~20 examples (easy to hard) where you can exercise JSpecify usage and get a hang of it.

Github repo: https://github.com/JurenIvan/jspecify-exercise

High-Availability Schema Changes: Rolling Updates with Flyway and Spring Boot

Ivan Juren — Mon, 20 Oct 2025 17:11:03 +0000

Supporting multiple live instances of an application that interact with a single relational database where the schema and data must remain consistent throughout deployments is significantly more complex than migrating with just one instance, or in a simple "stop all instances, migrate, deploy" scenario.

Let's explore how to perform seamless rolling migrations with Flyway across multiple instances while ensuring high availability!

Issue at Hand

Our original deployment strategy was straightforward: anchored by Flyway for database migrations and JPA with ddl-auto=validate, we would stop all instances, run migrations on the first instance at startup, and then start the remaining instances. This “stop the world” approach worked well under previous requirements—downtime during off-peak hours was acceptable, and availability targets were met.

Over time, business requirements evolved. Clients began expecting constant access, making any downtime, even for maintenance, count against our SLA (more on it Thoughts on SLA. While not a crisis, this change highlighted that our old approach was insufficient for a high-availability environment.

In other words, we needed a strategy that allowed database migrations and application updates to occur on the fly, without interrupting service or breaking running instances. In other words, this evolution of requirements prompted us to rethink how deployments and migrations should be handled in a multi-instance setup.

How Did We Address It?

We adopted a rolling update strategy, updating application instances gradually while keeping others running. The key challenge was interoperability: old and new versions must operate concurrently with the database without causing downtime or errors.

This way, we have several instances running the old version and one instance running the updated version. This indicates that we must be able to support different versions at the same time if we would like to use this deployment strategy. Otherwise, the old or the new instances might not work as expected. In other words, interoperability is key.

Deployment Strategies breakdown on Baeldung

Since Flyway triggers our migrations, we needed to verify that it is up to the task. We addressed this with a small POC, which I covered in a previous blog post.

In short, Flyway is forward-compatible: older application versions can start even if newer migrations (based on database migration table) are detected.

Except the flyways schema/migration validation, we still have to make sure actual schema in DB is OK for queries old version will still be execution, and the JPA startup db schema validation. Simply deleting a column from DB will result with old version of service failing at query time. Also, if an old instance is restarted, JPA database schema validation will fail and the instance will not restart.

Solution

The solution is multi-step migration. An intermediary version writes "extra/legacy data" to the DB for older instances to function properly while deprecating its use of the old columns by using the new ones. This way, the both old and new version of app can work with the DB at the same time. Once all the instances are updated, we deploy yet another version that safely removes the old columns from the DB. Only then the migration is done.

Procedures to Handle the Deployment Process

For further examples, assume we have 2 instances of the app running version 1.0.0, and we want to deploy a new version.

Adding a New Field

Create a new version (v1.0.1) of the app with a migration script to add the new field.
Deploy the new version to all the instances.

(If you want to ensure non-nullability, the following steps are needed:)

Create a new version (v1.0.2) of the app with a migration script that adds the non-null constraint to the new column with a default value for all the existing rows.
Deploy the new version (v1.0.2) to at least one instance (to run the migration).

Deleting a Field

Create a new version (v1.0.1) of the app that does not use the field in the application logic but lacks the migration script that will actually delete the column from the DB.
Deploy the new version (v1.0.1) to all the instances (no DB migration is run).
Create a new version (v1.1.0) of the app with a migration script that deletes the column from the DB.
Deploy the new version (v1.1.0) to at least one instance (to run the migration).

Changing the Type of the Field or Any Other Complex Operation

Create a new version (v1.0.1) of the app with a migration script to add the new field, copy/map the data from the old column, and ensure that any new data is written to both the new column and the old column (write to both so that old instances can still work with the data, read from the new column to enable the latter step).
Deploy the new version to all the instances (at this point, the app is ready to stop using the old field).
Create a new version (v1.0.2) that deprecates the old column by writing only to the new column (final sync of the data).
Deploy the new version (v1.0.2) to all instances.
Create a new version (v1.1.0) of the app with a migration script that copies/maps the data from the old column to the new one (final sync of the data) and a migration script that deletes the old column from the DB.
Deploy the new version (v1.1.0) to at least one instance (to run the migration).

Conclusion

Ensuring interoperability between different versions of your application during rolling updates is crucial for enabling rollout when there's more than one instance interested. Flyways validation defaults play the key role in supporting this process, enabling forward-compatible database migrations that allow older versions of the application to coexist with newer ones.

Achieving this seamless transition requires careful planning and "additional steps". These extra steps, while adding complexity to the deployment process, are essential to ensure data consistency, rollback and avoiding breaking changes.

Interoperability is key. I hope this blog post provides valuable insights and practical guidance beyond the simplistic approach of merely deploying multiple instances.

Thoughts on SLA

Ivan Juren — Mon, 20 Oct 2025 16:00:51 +0000

When people talk about a “Service Level Agreement” (SLA), it rarely points to a single, clear metric. The term is often vague and open to interpretation. You’ll hear statements like “Our service guarantees 99.9% uptime”, but what does that really mean in practice? After firefighting many incidents, I have some thoughts and insights on it.

Let's put together an overview of the most common types of SLAs and share some practical examples and what it means for engineers.

What is an SLA?

An SLA is a promise from a service provider to its customers about quality and reliability.
The most common metric and the one that usually takes the spotlight is availability, often expressed as a percentage of uptime.

Here’s a reference table showing the allowed downtime for various levels of availability, assuming 24/7 operation:

Availability %	Downtime per year	Downtime per month	Downtime per day
90% ("one nine")	36.53 days	73.05 hours	2.40 hours
99% ("two nines")	3.65 days	7.31 hours	14.40 minutes
99.9% ("three nines")	8.77 hours	43.83 minutes	1.44 minutes
99.99% ("four nines")	52.60 minutes	4.38 minutes	8.64 seconds
99.999% ("five nines")	5.26 minutes	26.30 seconds	864 milliseconds

Wikipedia - High Availability

Alternative SLA Measurements

Uptime is the easiest number to quote, but it doesn’t account for when and how (badly) a system fails. Most services fail under peak load, and five minutes of downtime during that window are usually be far more damaging than an hour of scheduled maintenance at midnight. Even worse, a service might appear working (responding with 200 OK on /health), while performing terribly for users.

Common peak-load failure patterns include:

Increased latency - depending on business domain, doing things slowly can be worse than not doing it at all.
Timeouts - If system has a chain of dependencies, one slow component can snowball into increased load on all downstream services.
Partial outages - only certain users or actions fail, escaping detection, but breaking clients workflows

Most of these failures wouldn't even necessarily count up to the typical availability metric, but from single users perspective, these can have the same impact as downtime”

There are other ways to measure service quality that can sometimes provide a bit more accurate picture of clients experience:

Yield: percentage of successful transactions, providing a more accurate reflection of system performance metric for systems with variable usage. Related interesting read: Readings on Yield.
Latency thresholds: percentage of requests served within an acceptable latency range (e.g., 95% under 200 ms).
Suboptimal result rate: how often the system serves degraded or approximate responses to save resources. Useful for performance/accuracy trade-offs.

Impact of a single monthly incident on 4-nines availability

“Four nines” (99.99%) availability is often a target for high-reliability systems.
This gives downtime budget of just ~52 minutes of downtime per year, or about five minutes per month.

Consider a scenario with a single incident per month:

Incident occurs
Monitoring detects and raises an alert, ~30s
Engineer receives and answers the page, ~3 minutes
Engineer logs in and begins investigating, ~1 minute

At this point, you’ve already burned through almost your entire monthly downtime budget.
Good luck diagnosing, fixing, and recovering in the 30 seconds that remain.

To make 4-nines uptime even remotely achievable, you need early detection, self-healing mechanisms, and automated recovery long before humans get involved.

Conclusions

Traditional uptime metrics are useful, but they don't tell the whole story. True reliability means being transparent, measuring the right things, and understanding the real impact of failures on users.
SLAs are a foundational part of service reliability, and they require careful thought and design. Fixing problems in the design phase, way before they have a chance to surface takes time, but it protects your reputation.
High availability isn’t achieved through heroics! It’s engineered through prevention, visibility, and automation.

"An SLA is not just a number. It's a commitment to quality and trust."

Navigating Flyway's Unexpected Behavior in Database Evolution

Ivan Juren — Mon, 20 Oct 2025 11:03:29 +0000

As developers, we often rely on tools like Flyway to streamline the management of database migrations, ensuring that our applications evolve smoothly over time. However, there are subtleties that might be unexpected beneath the surface of this seemingly straightforward process. We came across one and would like to share it with you.

Let's examine Flyway's response to newer database versions and its implications for migration management!

Understanding Flyway's Startup Checks and the Usual Use Cases

Before diving into the peculiar behaviour that we've uncovered, let's quickly grasp Flyway's startup checks. Upon initialization, Flyway compares the checksums of migration files with those applied to the database, ensuring consistency and integrity. Any discrepancies, be they altered files or missing (executed) scripts, prompt Flyway to halt, warning us of potential issues before they escalate.

Try this at home, NOT on prod!

In some scenarios, developers may resort to manual interventions for various reasons. For instance, if there's a need to repeat the execution of the last script, they might opt to delete the corresponding migration entry in the database. This action essentially prompts Flyway to rerun the script during the next migration cycle. Similarly, if a script needs to be amended or fixed directly in the database, developers might execute the necessary changes manually, adjust the script accordingly, and adjust the stored checksum. However, such manual adjustments should be approached with caution to avoid inadvertently introducing inconsistencies or breaking the migration process.

When you start doing these things you might expect the following errors in the logs:

Checksum mismatch - when the script is altered and the checksum is not updated:

org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'flywayInitializer' defined in class path resource
[org/springframework/boot/autoconfigure/flyway/FlywayAutoConfiguration$FlywayConfiguration.class]: Validate failed: Migrations have failed validation
Migration checksum mismatch for migration version 1.00
-> Applied to database : 1993450877
-> Resolved locally    : 1305205389
Either revert the changes to the migration, or run repair to update the schema history.

Missing migration - when script is deleted from the database, or if you're using timestamps as migration versions and ignore scripts merged into the master/main in the meantime after you've created the migration file:

org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'flywayInitializer' defined in class path resource
[org/springframework/boot/autoconfigure/flyway/FlywayAutoConfiguration$FlywayConfiguration.class]: Validate failed: Migrations have failed validation
Detected resolved migration not applied to database: 1.50.
To ignore this migration, set -ignoreMigrationPatterns='*:ignored'. To allow executing this migration, set -outOfOrder=true.

An Unexpected Twist: Detecting Newer Database Versions

Here's where things get interesting. While exploring Flyway's inner workings, I stumbled upon a behaviour that defied my expectations. Imagine this scenario: our database is humming along with the latest migrations applied, proudly sporting version 2.00. However, upon inspecting our migration files, we find ourselves stuck in the past, lingering at version 1.00. One might expect Flyway to raise a red flag at this apparent mismatch, halting operations until the files catch up to the database. Instead, Flyway surprises us with a mere warning – a gentle (WARN) reminder in the log that a migration file is missing.

We can see what is happening in the logs (below).
When starting the application multiple times with different versions, we observe the following sequence of events:

(Instance 1, Version 1): The first instance of the application is started with version 1. Flyway validates and applies the migration, initializing the database schema to version 1.00.

INFO 1 --- [flyway-testing] [main] o.f.c.i.c.DbValidate: Successfully validated 1 migration (execution time 00:00.194s)
INFO 1 --- [flyway-testing] [main] o.f.c.i.c.DbMigrate : Current version of schema "public": << Empty Schema >>
INFO 1 --- [flyway-testing] [main] o.f.c.i.c.DbMigrate : Migrating schema "public" to version "1.00 - init car"
INFO 1 --- [flyway-testing] [main] o.f.c.i.c.DbMigrate : Successfully applied 1 migration to schema "public", now at version v1.00 (execution time 00:00.036s)

(Instance 2, Version 1): A second instance of the application, meant for high availability, is launched also with version 1. Flyway validates the schema, finding it already at version 1.00, thus no migration is necessary.

INFO 1 --- [flyway-testing] [main] o.f.c.i.c.DbValidate: Successfully validated 1 migration (execution time 00:00.296s)
INFO 1 --- [flyway-testing] [main] o.f.c.i.c.DbMigrate : Current version of schema "public": 1.00
INFO 1 --- [flyway-testing] [main] o.f.c.i.c.DbMigrate : Schema "public" is up to date. No migration necessary.

(Instance 2, Version 2): A new version 2 is deployed to the second instance. Flyway validates the schema, still finding it at version 1.00. However, it proceeds to migrate the schema to version 2.00, adding the necessary changes.

INFO 1 --- [flyway-testing] [main] o.f.c.i.c.DbValidate: Successfully validated 2 migrations (execution time 00:00.243s)
INFO 1 --- [flyway-testing] [main] o.f.c.i.c.DbMigrate : Current version of schema "public": 1.00
INFO 1 --- [flyway-testing] [main] o.f.c.i.c.DbMigrate : Migrating schema "public" to version "2.00 - add column"
INFO 1 --- [flyway-testing] [main] o.f.c.i.c.DbMigrate : Successfully applied 1 migration to schema "public", now at version v2.00 (execution time 00:00.034s)

(Instance 1, Version 1): The first instance is then redeployed with version 1, effectively restarting it. Flyway validates the schema and detects that it's already at version 2.00, even though the migration files only go up to version 1.00. It issues a warning about the discrepancy but proceeds without applying any migrations or startup failure.

INFO 1 --- [flyway-testing] [main] o.f.c.i.c.DbValidate: Successfully validated 2 migrations (execution time 00:00.184s)
INFO 1 --- [flyway-testing] [main] o.f.c.i.c.DbMigrate : Current version of schema "public": 2.00
WARN 1 --- [flyway-testing] [main] o.f.c.i.c.DbMigrate : Schema "public" has a version (2.00) that is newer than the latest available migration (1.00) !
INFO 1 --- [flyway-testing] [main] o.f.c.i.c.DbMigrate : Schema "public" is up to date. No migration necessary.

Implications of Flyway's Decision

This seemingly lenient approach raises questions about the trade-offs involved. While it doesn't explicitly grant us the freedom to apply migrations out of sequence, it does allow the application to start even if later migrations have been applied to the database. This can serve as an indicator of potential skipped migrations and it also enables a rolling migration approach where older instances of the application can be restarted even after the newer instance version has been started.

Conclusion

One might say that the flyway is "forward compatible" in this case. It is engineered to manage future database migrations and schema changes seamlessly, allowing ongoing functionality without needing complete future code.

Understanding Flyway's behaviour is key to successful database management. By following best practices and embracing their quirks, such as keeping migration files current and avoiding manual schema alterations, developers can navigate migrations with confidence. Hope this interesting quirk that we've found helps you!

Happy migrating!

Forem: Ivan Juren

Good Intentions Aren’t Enough — Measure!

The Problem

The Obvious Candidates

The Custom Design

The First Hurdles

First Benchmarks — Great News?

Reality Hits

The New Approach

The Real Culprit

Final Results

The Lesson

Final Thought

Good Things (Compression) Take Time!

🧠 What Is linger.ms and Why Is It Important?

The Tradeoff

🔍 Methodology

📊 Results and Analysis

1️⃣ End-to-End Latency

2️⃣ CPU Usage

3️⃣ Produce/Fetch Count

4️⃣ Bytes In Total

🧩 Summary

🔥 Turning Up the Heat

🚀 Key Takeaways

The Art of Knowing When to Upgrade

When do I decide to adopt something new in my stack?

The Risk and the Reward

The Best- and Worst-Case Scenarios

How Deep Should You Dive In?

My Pointers

Final Thoughts

🔗 Related Reads

Hey, I did a thing, you might like it! Java JSpecify Exercise

Learning by doing is the most durable kind of learning.

Rethinking Optional!

Compile-Time Safety vs Runtime Safety: JSpecify in Practice

Without JSpecify: Optional All the Way

With JSpecify: Compile-Time Null Awareness

The Differences

Conclusion

TL;DR

High-Availability Schema Changes: Rolling Updates with Flyway and Spring Boot

Issue at Hand

How Did We Address It?

Solution

Procedures to Handle the Deployment Process

Adding a New Field

Deleting a Field

Changing the Type of the Field or Any Other Complex Operation

Conclusion

Thoughts on SLA

What is an SLA?

Alternative SLA Measurements

Impact of a single monthly incident on 4-nines availability

Conclusions

Navigating Flyway's Unexpected Behavior in Database Evolution

Understanding Flyway's Startup Checks and the Usual Use Cases

Try this at home, NOT on prod!

An Unexpected Twist: Detecting Newer Database Versions

Implications of Flyway's Decision

Conclusion

🧠 What Is `linger.ms` and Why Is It Important?