Forem: madhead

When ConcurrentHashMap is not concurrent and runBlocking is not blocking

madhead — Sun, 08 Jun 2025 22:01:07 +0000

A (click)bait? Let me explain.

Take a look at this simple code. Can you predict what will be printed?

Before we dive into what’s wrong here, let’s address an obvious concern:

Does this code even make sense? It looks weird. Surely not a real-world code!

But it kind of is. Sure, it’s synthetic and simplified, but it represents a valid real-world scenario I encountered recently. The essence is straightforward: we have a ConcurrentHashMap and two concurrent computations, both trying to set the value for the same key.

In the real world, things were a bit more involved. We had to preload some data from a third-party service during the startup of a Spring Boot application. But Spring doesn’t support suspending functions in @PostConstruct, InitializingBean, or @EventListener. So we went with the simplest option: wrapping the logic in runBlocking.

To speed things up, we performed the calls in parallel (using async in the real code). While each request was loading different data, they all required service-scoped JWTs issued by another token service. These tokens are short-lived and cached using the service name as the key. And that cache (Caffeine) is, essentially, a ConcurrentHashMap.

So, if two parallel requests for the same service kick off and no token is cached yet, both will try to call the token service and cache the result. Since Java’s computeIfAbsent doesn’t accept suspend functions, we wrapped that in another runBlocking.

A code smell? Maybe. Nested runBlockings, expensive compurations inside computeIfAbsent… But hey, ConcurrentHashMap is concurrent, right? At worst, we’d waste a call or two to the token service. One thread wins and sets the value; the other one moves on.

Except… The code is completely broken:

Exception in thread "main" java.lang.IllegalStateException: Recursive update
 at java.base/java.util.concurrent.ConcurrentHashMap.computeIfAbsent(ConcurrentHashMap.java:1763)

Let’s dig in.

The exception is thrown here. So there is a variable f which is an instance of a ReservationNode. And the docs for that class immediately prove that we’re on a right track:

A place-holder node used in computeIfAbsent and compute.

Going back in the flow brings us to these lines. There’s a strange bit: it synchronizes on a local variable. Weird? Yes. But there’s a good reason: that’s how ConcurrentHashMap avoids locking the entire map. Simplifying a bit: it reserves a slot for the key and synchronizes on that specific entry. Only one thread may enter that block.

IntelliJ IDEA has a neat feature, helping quickly understand how the execution got to a particular point, called “Analyze Data Flow”.

Let’s sprinkle in some logging to observe the threads in action:

fun printlnWithThreadName(message: Any?) {
    println("[${Thread.currentThread().name}] $message")
}

And the output:

[main] Started main
[main @coroutine#1] Entered runBlocking
[main @coroutine#2] Started launch (1)
[main @coroutine#2] Computing key (1)
[main @coroutine#3] Started launch (2)
[main @coroutine#4] Computing key (1), inside runBlocking

Turns out, runBlocking… isn’t truly blocking in the way one might expect. So how is it starting multiple computations? They’re not running in parallel, after all, a single thread can’t do that. But they do appear to be running concurrently. No surprise there: that’s exactly how coroutines work. It’s cooperative multitasking: when a coroutine suspends, the thread can resume another coroutine that’s ready to continue. Multiple coroutines can take turns on a single thread.

This is when the runBlocking's documentation starts to make sense:

The default CoroutineDispatcher for this builder is an internal implementation of event loop that processes continuations in this blocked thread until the completion of this coroutine. See CoroutineDispatcher for the other implementations that are provided by kotlinx.coroutines.
When CoroutineDispatcher is explicitly specified in the context, then the new coroutine runs in the context of the specified dispatcher while the current thread is blocked. If the specified dispatcher is an event loop of another runBlocking, then this invocation uses the outer event loop.

So: runBlocking does block the calling thread until the coroutine completes. However, within that thread, it runs an event loop to dispatch and resume coroutines created inside the same runBlocking block. Since all these coroutines run on the same thread, what happens when they hit a synchronized block, like the one inside the computeIfAbsent? In other words: if a coroutine enters a synchronized block and then suspends, could another coroutine resume on the same thread while the monitor is still held? Is it just… let in?

Turns out that the answer is "yes". And that’s the problem.

ConcurrentHashMap is thread-safe, but not coroutine-safe.

Another finding: those launch blocks (or asyncs) are generally pointless. Of course IO operations like this benefit from suspension, but there’s no actual parallelism if everything is running on a single thread.

The code had been running like this for ages. What revealed the issue? I was updating our HTTP clients, introducing that exact token cache and discovered a bug in the preload logic!

Let’s give the outer runBlocking a proper CoroutineDispatcher:

Now, everything works:

[main] Started main
[DefaultDispatcher-worker-2 @coroutine#1] Entered runBlocking
[DefaultDispatcher-worker-3 @coroutine#2] Started launch (1)
[DefaultDispatcher-worker-3 @coroutine#2] Computing key (1)
[DefaultDispatcher-worker-1 @coroutine#3] Started launch (2)
[DefaultDispatcher-worker-3 @coroutine#4] Computing key (1), inside runBlocking
value1

Even limiting parallelism to one (e.g., Dispatchers.IO.limitedParallelism(1), try it!) seems to fix the issue. Is the root cause in the event-loop nature of the default dispatcher in runBlocking?

I still think, that the ConcurrentHashMap is not, generally, coroutine-safe. Even if you’re using multiple physical threads with a proper dispatcher, there’s still a chance that two computeIfAbsent operations could be scheduled on the same thread if they’re suspendable. Although this case would be very rare.

What do you think?

P.S. Another folks argued about ConcurrentHashMap's safety in coroutine world in Kotlin Slack here. Also, there is a discussion of a very similar case already in the kotlinx.coroutines GitHub: #3982.

No SNAPSHOTs

madhead — Tue, 30 Jul 2024 07:30:00 +0000

SNAPSHOTs are a confusing Maven concept. Maven is a pre-historic build tool, invented inside The Apache Software Foundation to build its (mostly Java) projects back in the 2000s. It is, thus, very opinionated, just like this post. Those opinions — including SNAPSHOTs, maven-release-plugin, POM, repository and project layouts, and many more — are still alive today and spoil modern projects.

Even ASF does not use Maven to build some of its projects anymore: Beam, Groovy, Lucene, Geode, POI, and Solr are not built with Maven. Those are not the most popular ASF projects, I know, but still, it is something.

Artifacts must be immutable, and two artifacts with the same version must be the same. That is not the case with SNAPSHOTs: they are mutable by definition and every new build could change the artifact published under the -SNAPSHOT version.

Because of that, SNAPSHOT artifacts require special treatment. They have their own updatePolicy. Do you, fellow Maven users, know off the top of your head, how often are they updated by default? Daily. SNAPSHOTs are usually uploaded to a separate repository and they are not — obviously — accepted by the Maven Central.

SNAPSHOTs are a potential source of errors. Hopefully, only during the development and testing process, because no one should use them in production.

Stick to something like Semantic Versioning, and use pre-release identifiers and unique build identifiers instead of SNAPSHOTs. BTW, "SNAPSHOT" is a valid Semver pre-release identifier, however to guarantee artifact immutability, you should include additional metadata in the version, like unique build number, commit hash, or timestamp, etc.

Abandoning SNAPSHOTs doesn’t mean that you cannot use the "latest" or "dynamic" version of the artifact in your project. Modern build tools support dynamic versions, rich versions, version ranges, constraints, and other cool stuff. I am pretty sure your build tool has it.

Just to be clear: published artifacts must be immutable, and SNAPSHOTs are not, but you could still have a dynamic version (or a dynamic, but locked/pinned, version) in your projects.

SSTables and LSM-Trees for a layperson

madhead — Tue, 05 Mar 2024 10:00:00 +0000

This article is my excerpt of Martin Kleppmann's "Designing Data-Intensive Applications" book. These data structures are explained in chapter 3, "Storage and Retrieval", pages 69 through 79. It's a really great read, I would recommend the whole book!

Everything starts with a very dumb key-value database implemented as just two Bash functions:



db_set () {
    echo "$1,$2" >> database
}

db_get () {
   grep "^$1," database | sed -e "s/^$1,//" | tail -n 1
}

The idea is to store the data in a CSV-like file:



$ source database.sh

$ db_set 1 'Anakin Skywalker'
$ db_set 2 'Luke Skywalker'
$ db_set 1 'Darth Vader'

$ cat database
1,Anakin Skywalker
2,Luke Skywalker
1,Darth Vader

$ db_get 1
Darth Vader

Note that the first value for the key 1 is overridden by the subsequent write.

This database has pretty good write performance: db_set just appends the data to a file, which is generally fast. But reads are inefficient, especially on huge data sets: db_get scans the entire file. Thus, writes are O(1) and reads are O(n).

Next, indices are introduced. An index is a data structure derived from the data itself. Maintaining an index always incurs additional costs, thus, indices always degrade write performance with the benefit of improving the reads.

One of the simplest possible indices is a hash index. This index is nothing more than a dictionary holding bytes offsets of the records in a database. Continuing previous example, assuming every char is one byte, the hash index would look like this:

Whenever you write data into the database, you also update the index. When you want to read a value for a given key, you could quickly look up an offset in the database file. Having the offset, you could use a "seek" operation to jump straight to the data location. Depending on the particular index implementation you could expect a logarithmic complexity for both reads and writes.

Next, Martin deals with the storage efficiency. Appending data to a database file exhausts disk space quickly. The fewer distinct keys you have — the more inefficient this append-only storage engine is. The solution to this problem is compaction:

When a database file grows to a certain size, you stop appending to it, create a new file (called segment) and redirect all the writes to this new file.

Segments are immutable in that sense that they are never used to append any new data. The only way to modify a segment is to write it's content into a new file, possibly with some transformations in between.

So, the compaction creates new segments containing only the most recent records for each key. Another possible enhancement at this step is merging multiple segments into a single one. Compaction and merging could be done, of course, in background. Old segments are just thrown away.

Every segment, including the one being written to, has its own index. So, when you want to find the value for a given key, you search those indices in reverse chronological order: from the most recent, to the oldest.

So far we have a data structure having these pros:

✔️ Sequential writes are generally faster than random ones

✔️ Concurrency is easy to control having a single writer process

✔️ Crash recovery is easy to implement: just read all the segments sequentially, and store the offsets in the in-memory index

✔️ Merging and compaction help to avoid data fragmentation

However, there are some limitations as well:

❗ Crash recovery could be time-consuming if segments are large and numerous

❗ Hash index must fit in memory. Implementing on-disk hash tables is much more difficult

❗ Range queries (BETWEEN) are virtually impossible

Now, with this background, let's move to the SSTables and LSM-trees. By the way, these abbreviations mean "Sorted String Tables" and "Log-Structured Merge Trees" accordingly.

SSTables are very similar to the "database" that we've seen previously. The only improvement is that we require records in segments to be sorted by key. This might seem to break the ability to use append-only writes, but that's what LSM-Trees for. We'll see in a moment!

SSTables have some advantages over those simple segments we had previously:

✔️ Merging segments is more efficient due to the records being pre-sorted. All you have to do is to compare segment "heads" on each iteration and choose the lowest one. If multiple segments contain the same key, the value from the most recent segment wins. This compact & merge process also holds the sorting of the keys.

✔️ With keys sorted, you don't need to have every single key in the index anymore. If the key B is known to be somewhere between keys A and C you could just do a scan. This also means that range queries are possible!

The final question is: how do you you get the data sorted by key?

The idea, described by Patrick O’Neil et al. in their "The Log-Structured Merge-Tree (LSM-Tree)", is simple: there are in-memory data structures, such as red-black trees or AVL-trees, that are good at sorting data. So, you split writes into two stages. First, you write the data into the in-memory balanced tree. Second, you flush that tree on the disk. Actually, there may be more than two stages, with deeper ones being bigger and "slower" then the upper (as shown in this answer).

When a write comes, you add it to the in-memory balanced tree, called memtable.
When the memtable grows big, it is flushed to the disk. It is already sorted, so it naturally creates an SSTable segment.
Meanwhile, writes are processed by a fresh memtable.
Reads are first being looked up in the memtable, then in the segments, starting from the most recent one to the oldest.
Segments are compacted and merged from time to time in background as described previously.

The scheme is not perfect, it could suffer from sudden crashes: the memtable, being an in-memory data structure, is lost. This issue could be solved by maintaining another append-only file that basically duplicates the contents of the memtable. The database only needs to read it after a crash to re-create the memtable.

And that's it! Note that all the issues of a simple append-only storage, described above, are now solved:

✔️ Now there is only one file to read in a case of a crash: the memtable backup

✔️ Indices could be sparse, thus fitting the RAM is easier

✔️ Range queries are now possible

TLDR: An SSTable is a key-sorted append-only key-value storage. An LSM-tree is a layered data structure, based on a balanced tree, that allows SSTables to exist without the controversy of being both sorted and append-only at the same time.

Congrats, you've finished this long read! If you enjoyed the explanation, make sure to check out the whole book!

On DynamoDB’s Single Table Design

madhead — Tue, 27 Feb 2024 10:00:00 +0000

What’s a Single Table Design?

The world learned about the idea of Single Table Design (STD) for DynamoDB somewhere in 2019, probably when this article came out. STD wasn’t just a weird idea by some unknown blogger like me, or an AWS DevRel, or any other kind of Internet freak, but it actually has its roots in the official Amazon DynamoDB documentation. Here:

You should maintain as few tables as possible in a DynamoDB application. Having fewer tables keeps things more scalable, requires less permissions management, and reduces overhead for your DynamoDB application. It can also help keep backup costs lower overall.

And then, a few prominent AWS folks spun this recommendation to the extreme into what we now know as STD: use a single table for all your data.

❕ Of course, no one meant literally a single table, but the suggestion
was to downsize.

At the core of STD, as we’ll discover later, is something called Index Overloading. It involves leveraging every available index to its maximum potential — thereby "overloading" — to store heterogeneous data.

❕ DynamoDB had a limit of 5 Global Secondary Indices (those were of a particular interest for the STD) back at those days, now the limit is increased to 20 GSIs per table by default.

As a reference implementation, a rather simplistic little application with just a couple of entities was presented to the public, hardly conveying the potential dangers of this approach. A bit later, another article about STD found its home in the official AWS Blog as well. Here are the links: 1 | 2 | 3.

How outraged I was!

What’s wrong with STD?

It wasn’t the STD concept itself that irked me, but the assertiveness with which it was being pushed! I remember I couldn’t hold back and even shot some DMs at our local AWS community champs, demanding answers.

My emotions rooted in the fear that someone on my project might want to adopt this approach. Ironically enough, considering I was working solo back then. With several dozen tables in our project, merging them all into one was just madness. It might have been efficient, but it would surely enrage anyone who had to work with and maintain it.

For instance, the article in the AWS Blog mentions a nifty little constraint in the domain model that limits the size of a single record.

❕ The maximum item size in DynamoDB is 400 KB, which includes both attribute names and attribute values.

It goes like this: we’re only storing 300 data points per event, so the data will fit. Isn’t that fortunate? And if it doesn’t fit, well, just split it somehow into several parts, you are on your own now! But on our project, we already had entities exceeded that limit. And we had already resorted to tricks like ZIP compression on the application side, which had already complicated maintenance: such tables are impossible to view in the console.

I wasn’t alone in my frustration. Here's an example of one of the STD discussions on Reddit. Some users found the whole idea "overwhelming", though there were just as many enthusiasts.

For me, it was obvious that the approach is at least debatable and not suitable for every situation. But being pessimistic about developers in general, I was concerned they might start applying it indiscriminately, ignoring the specific and realities of the project. The official documentation, and the hype, were just too persuasive!

Moreover, this approach is too complex if you try to implement it. And it’s not flexible. It forces you to think far ahead, really far ahead. So much far ahead, that you need to get it right for ages and from the first attempt: modifying an already created STD table, or adding a new type of query to it, is practically impossible.

Oh how the tables have turned!

Rick Houlihan, the "Inventor of @DynamoDB #SingleTableDesign", has apparently faced so much criticism over time that he felt compelled to defend himself on X. His defence, though, is more like "it was just three of us musketeers, standing agains whole Cardinal’s guard". Yesterday, he twitted:

// Detect dark theme var iframe = document.getElementById('tweet-1760469859761029228-129'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=1760469859761029228&theme=dark" }

But let’s take a closer look at what he’s saying…

I even saw @alexbdebrie apologizing for being an advocate the other day.

Oh, that’s indeed to much, no one should apologize for great experiments!

Anyone who tells you a different story than the one that follows is wrong. I led the team that invented the Single Table Design pattern. I have facts, they have at best half-informed opinions. There are 3 people in this world who could tell this story.

That’s the musketeers part…

Then follow several paragraphs explaining Index Overloading. And I think that’s exactrly what the problem with Index Overloading / STD is. It’s so complex, that requires explanation after being available for about five years already. Maybe, it was just a little bit too much for the average developer from the very beginning?

The drawback of doing this was that indexes became more and more polluted with unrelated Item types as the number of access patterns they supported increased.

Because of this it was not easy to drop and recreate indexes without table scans and batch updates which became expensive at scale.

The pattern also introduced heavy cognitive load on developers as using abstract naming for index attributes meant it was not always immediately apparent when looking at the data how the data was being indexed unless the values assigned to the generic keys were self-explanatory.

I couldn’t agree more!

All of these things were tradeoffs for applying the Index Overloading pattern, not core issues with Single Table Design itself. They were often deemed acceptable inconveniences considering the benefit of having effectively unlimited GSI’s. Most of the problems that drove the need for Index Overloading have been resolved over the years as DynamoDB has added support for 25 GSI’s, introduced on demand pricing, and eliminated the need to allocate capacity individually for each index. As a result the pattern should really be considered deprecated today.

Ah… Here we go again, aren’t we? We’ve just seen that even 5 indices are too much for the average developer, and now we have to deal with 25?

❕ I’m not sure why Rick is talking about 25 GSIs, as the official documentation still mentions 20.

Then came excuses…

Additionally, many people over the years have also taken STD to an extreme that was never intended. Mixing configuration and operational data, maintaining a single table across service boundaries, or storing unrelated data that is not accessed together in the same table. Despite the fact that there are some people out there trying very hard to rewrite history around this, none of these things were ever recommended as best practices.

Here I have to disagree. Did we see STD articles in the AWS Blog? We did! Is it the official source? It is! If the official AWS Blog is not a place for the best practices, then where is the place?

It is easy to look at the product as it exists today and criticize the design patterns of yesterday that were invented to deal with API deficiencies that no longer exist. I read some serious garbage every now and then written by people I feel should know better, Those people really had very little exposure to the process of solving the problems we faced when the patterns they criticize were introduced.

I’m sorry, to hear that, Rick, we failed your expectations. But that’s the nature of the crowd, isn’t it? Remember how developers make cults around some design patterns? Yet the wiki definition of a design pattern literally says "contextual":

In software engineering, a software design pattern is a general, reusable solution to a commonly occurring problem within a given context in software design.

Yet patterns are everywhere, often applied out of context. Same as programming languages, databases and lots of other things in the IT world.

What’s the moral of this rant?

It’s possible to brilliantly overcome certain technological limitations and propose unconventional and effective solutions to some problems. But it’s important to be cautious and thoughtful when conveying these ideas to a layperson. Architects, come down from your Ivory towers! Don’t overestimate the intellectual capabilities of developers, but also don’t overvalue your own ideas.

Every problem has its context, which is often lost. But this goes both ways! Just as developers sometimes fail to understand the applicability limits of patterns and solutions, architects sometimes neglect the context in which development occurs. You know: tight schedule, low skill…

Don’t be sorry for the great idea you gave us and don’t be angry at us for misusing it.

Declarative Gradle is a cool thing I am afraid of: Maven strikes back

madhead — Sat, 11 Nov 2023 03:25:20 +0000

Yesterday, JetBrains introduced Amper. Today, Gradle published another related article, named Declarative Gradle.

Now, I can see the direction it is leaning towards, and I am afraid of this future.

History of Java Build Tools, simplified and opinionated

NOTE: I won’t mention SBT and Leiningen here because, with all due respect, they are niche build tools. I also won’t discuss Kobalt for the same reason (besides, it’s no longer actively maintained). Additionally, I won’t touch upon Bazel and Buck in this context, mainly because I’m not very familiar with them. If you have insights or comments about these tools, please feel free to share them in the comments 👇

Let me provide a quick recall of the history of (Java) build tools.

At the beginning, there was Ant. The fact of its existence is more information than a sensible person would typically need.

Then arrived Maven, the cool kid on the block around 2005. Maven was a very specific solution to a very specific problem: it was a tool used by Apache to build its projects, mostly Java libraries, back at those days. Notably, the widely adopted src/main/java and src/test/java structure finds its roots in Apache’s practices from that era. This narrative somewhat parallels the (mostly fictitious) story linking horse’s ass to space shuttles. Following that, there was a Maven Golden Age lasting until the 2010s, largely attributed to the absence of substantial competition and the relatively simple nature of Java projects.

Yet, the landscape changed. In no particular order:

Projects became more complex.
Apps began to be packaged in Docker images.
Projects started adopting a multi-module structure, with some extending to include thousands of individual modules.
Build times increased.
The evolution of build tools expanded their role beyond traditional building functions.
Multi-language projects emerged, with a simple example being the development of a JavaScript frontend.
…and then frontends evolved, the need to build them using tools like Webpack arose, though that’s a different story.
Truly multi-platform projects unfolded. To clarify: while Java is multi-platform, developing for server, web, desktop, iOS, and Android from a single codebase became hip only recently. Yes, I am talking about Kotlin Multiplatform.
…these multi-platform projects required the use of multi-platform dependencies.
And so forth.

Maven struggled to keep pace. Its origins as a build tool for simple Java libraries within Apache left it unprepared for the evolving landscape.

Consider build caching. Maven lacks a proper mechanism for it, except a local repository, but that’s not the same. Multi-platform dependencies are another challenge. The GAV notation (group, artifact, version), Maven’s Holy Grail, proves inadequate. And it still relies on a — completely fabricated and opinionated — build lifecycle, organized into dozen-or-something phases. Why not a Directed Acyclic Graph (DAG)?

Enter Gradle, our savior. Despite being ridiculosly complex — sometimes — it was built on a simple idea — a DAG — enabling it to evolve and adapt to new challenges. That’s why…

Gradle is the most advanced build tool in the Java world!

Consider Android projects. They are arguably significantly more challenging than typical Java projects. Their builds involve tasks like processing resources and assets, supporting multiple target SDKs, managing various build flavors, and occasionally dealing with native code and dependencies targeting different CPU architectures. Yet Gradle became the default build tool for Android projects shortly after Android projects even get a build tool! If my memory serves me right, they jumped straight from Ant to Gradle, bypassing Maven, thanks to Google’s influence – a move that I find quite illustrative.

Now, let’s explore other prominent projects in the Java landscape. Kotlin, Spring, Hibernate, RxJava — you name it, they all rely on Gradle. With two possible exceptions. Oracle isn’t particularly fond of Gradle, so don’t be surprised if you discover that GraalVM is built with Maven (it actually is, but the build process is more complex than just Maven vs. Gradle). And, of course, Apache remains loyal to Maven even in big and complex projects like Hadoop and Spark. After all, it’s their child! Kafka is build with Gradle, though.

However, virtually every other major Java project uses Gradle — check your own dependencies!

In essence, Gradle is a superset of Maven. In strict mathematical terms, ponder it for a moment. Everything achievable with Maven can be replicated with Gradle, but the reverse is not true.

Gradle’s DAG is so simple and powerful that you could probably replicate any other build tool with it. The only limitation I could think of: due to the separation of configuration and execution phases, creating and injecting tasks into the DAG on the fly might not be possible. Or is it? Don’t try that at home!

If you have any counterarguments at this juncture, please share them in the comments. I’m genuinely interested in understanding any aspects I might be overlooking 👇

Why strive for declarativeness?

Alright, let’s establish a boundary here.

Gradle is one of the most advanced build tools ever, and unquestionably, it is the most advanced build tool in the Java world.

It’s based on a very simple yet scalable and extensible concept.

However, this scalability and extensibility come at the cost of complexity, and not everyone enjoys its power at this price tag.

So, a while back, a new trend emerged: declarative Gradle. I claim that this concept isn’t entirely novel, and the article I referred to at the beginning of this post essentially encapsulates what has already been circulating, rather than initiating the trend.

The Android Gradle plugin serves as a noteworthy example of declarative Gradle that has been in existence for a considerable time.

Here’s a snippet from the official documentation

plugins {
    id("com.android.application")
}

kotlin {
    jvmToolchain(11)
}

android {
    namespace = "com.example.myapp"

    compileSdk = 33

    defaultConfig {
        applicationId = "com.example.myapp"
        minSdk = 21
        targetSdk = 33
        versionCode = 1
        versionName = "1.0"
    }

    buildTypes {
        getByName("release") {
            isMinifyEnabled = true
            proguardFiles(
                getDefaultProguardFile("proguard-android.txt"),
                "proguard-rules.pro"
            )
        }
        getByName("debug") {
            …
        }
    }

    flavorDimensions += "tier"
    productFlavors {
        create("free") {
            dimension = "tier"
            applicationId = "com.example.myapp.free"
        }

        create("paid") {
            dimension = "tier"
            applicationId = "com.example.myapp.paid"
        }
    }
}

dependencies {
    implementation(project(":lib"))
    implementation("androidx.appcompat:appcompat:1.6.1")
    implementation(fileTree(mapOf("dir" to "libs", "include" to listOf("*.jar"))))
}

Is it really declarative or imperative?

Upon closer examination, you’ll notice that it is imperative! Statements like these:

flavorDimensions += "tier,
getByName("release")
create("free")

…are extremely imperative!

Or perhaps not? After all, it’s just DSL, isn’t it? And quite eloquent one, I must say. Assignments and maybe a bit of object creation and configuration are inevitable, especially when dealing with software models like this.

What’s more important, is that we don’t see any conditions and loops here. They are the true evil of imperative programming, and here, there are none!

Another example of declarative Gradle are so-called convention plugins. While the documentation about custom plugins may not explicitly state and emphasize this term (which is a problem itself!), it is used in the samples. There is also this cool guy on YouTube, advocating for these best practices as well, take a look!

The idea behind convention plugins is really interesting, and it is very simililar to the AGP example above (in fact, AGP is a convention plugin!): to conceal all the complexity behind a simple DSL. Consider, for instance, another article of mine: No-bullshit guide on publishing your Gradle projects to Maven Central. Now, instead of cluttering your build script with the cumbersome publishing block, you could create a convention plugin, resulting in a build script that looks something like this:

plugins {
    id("com.acme.publishing")
}

That’s it! You wouldn’t even have to configure the GAV, as the values could be inferred from the project itself. Moreover, this hypothetical plugin could be shared across your company, ensuring uniform publishing configurations for all company’s projects.

Thats really imperative, and this is already available in Gradle!

The problem with the "Declarative Gradle"

Kudos to the vigilant readers who spotted the inconsistency: I used the term "Declarative Gradle" with a capitalized 'D' here, but earlier, I employed the "declarative Gradle" term with a lowercase 'd'.

And there’s a reason for that!

As demonstrated in the previous section, Gradle is already declarative if you wish it to be! All the tools are at your disposal to conceal complexity behind a straightforward and declarative DSL. Granted, it requires an investment of time: reading the docs, tuning into YouTube, and the like. In larger companies, a dedicated build engineer might be a prudent choice.

However, the article I linked at the beginning actually introduces a new term: Declarative Gradle, 'D' capitalized.

This blog post explains the Gradle team’s perspective for what we call a developer-first software definition, or Declarative Gradle for short. This post outlines our plans for making the “elegant and declarative build language” part of our vision a reality.

If I understand correctly, "Declarative Gradle" is essentially the same concept as described above but perhaps more trademarked. In fact, "Declarative Gradle" seems to be a more manager-friendly term than "convention plugin," and that’s perfectly fine.

My concern lies in this:

We plan to provide a restricted DSL that separates the software definition and build logic so that the build language is fully declarative. This will effectively enforce existing best practices.

The restricted DSL will allow only a limited set of constructs, such as nesting blocks, assigning values, and selected method invocations. Generic control flow and calls to arbitrary methods will be disallowed. You will be able to write your build logic in any JVM language, such as Java, Kotlin, or Groovy, but that logic will reside in plugins (either local or published).

This is disheartening for me because it feels like a step back.

This is Maven, to be precise.

Maven is already a tool with a very restricted DSL (XML; funnily enough, XML could be Turing-complete), where the entire complexity is hidden behind plugins crafted on top of an ambiguous and controversial API. Tinkering with the Maven API is genuinely challenging, while you could kick-start your Gradle plugin right in your build script.

Flexibility is the ultimate power of Gradle, and I am afraid of losing it. I’m afraid of Gradle loosing its charm. This is the reason why it is a better build tool than Maven (and this is the first time I’m actually saying that). It’s about freedom, a quality that doesn’t exist in Maven.

Don’t get me wrong: it’s not advisable to incorporate complex, imperative build logic directly into a build script. There are effective ways to circumvent that, as outlined above and in the docs. In my nearly 10 years of using Gradle, I’ve never used a loop in a build script. I’ve used conditions a few times, probably less than 10 times overall, so less than once per year. However, I now know how to eliminate them completely. The last time I utilized allprojects or subprojects was likely 3-5 years ago, and I know how to avoid them as well.

All these things are avoidable by acquainting yourself with the latest Gradle features. Dive into the release notes! Don’t shy away from using the latest Gradle version in your project either (because if you crave for Declarative Gradle™, an upgrade is inevitable). Embrace the best practices available. If you need a refresher on Gradle, watch that guy on YouTube, he’s brilliant!

I am apprehensive that at the end of this "declarativization", we might end up with a less powerful, restricted version of Gradle. I fear that the Maven mindset — the concept of having a limited but safe tool — might prevail.

Why you shouldn’t be too worried

The existing DSL will continue to be fully supported.
It won’t be XML! 🤞
No idea or implementation triumphs without community support. Alarmists like me have a chance to resist ✊

PR preview environments with Neon, GitHub Actions, and AWS Lambda

madhead — Wed, 03 May 2023 00:02:27 +0000

You might have heard about preview environments (AKA dynamic, or ephemeral environments) in the scope of software engineering and, particularly, this term is used when speaking about the SDLC.

A preview environment — as opposed to a permanent one, like "Production", or "TST", or whatever you call them — is a short-lived environment, usually bound to a pull request. It is created on-demand when the PR is created; and destroyed when the PR is closed. It hosts a copy of the application and is used to test the changes introduced by the pull request.

If you need more information, I encourage you to jump around a few articles on the Web to grasp the idea before continuing. I won’t go into details here, but instead, I’ll be focusing on the most challenging aspect of preview environments…

The database.

The tragedy of databases in preview environments

Materials about preview environments tend to cast little light on the database, leaving you with the approximate directions, like "create a copy of the database", or "use a database per PR", but no advice on how to achieve that. And for a reason!

First of all, everyone’s application is indeed different, and there’s no one-size-fits-all solution.

But the main reason, IMO, is that a database is very different from an application and it’s hard to copy it. Databases are persistent and stateful, while applications are ephemeral and stateless. Copying apps is the same as redeploying or scaling them, maybe with a different configuration. Copying databases is always a trick.

What’s even worse, is that databases historically combine both storage and compute in them! I mean, a typical database consists of its data, represented as actual files, and a database engine, which is an application process on its own. Thus, copying a database is at least twice as hard as copying an application. Simple math!

The options here range from cloning the database using dump / restore to using no copies at all, i.e. a shared DB for all the preview environments. Let’s roughly compare them.

Here, complexity means the amount of effort required to implement the solution. Obviously, a shared environment requires no effort; migration scripts require creating a fresh database and running the migrations; and dump / restore requires creating a fresh database, creating a dump, transferring it to the fresh database, and restoring it. Don’t get me wrong, it’s not impossible, it’s just disproportionately hard and slow.

Data isolation is high for both dump / restore and migration scripts approaches because each preview environment has its own database. A shared database, on the other hand, is not isolated at all.

Data isolation is highly correlated with the ability to test and preview schema modifications. Actually, it’s the same thing, I just want to highlight it, because it’s important. If you have a shared database, you can’t easily modify the schema without breaking others. However, if you have a dedicated database, you can do whatever you want with it. The ability to test and preview schema modifications is a huge advantage of preview environments because it allows you to test the migrations before applying them to the production database.

Isolation is the opposite of data freshness. I call it the ability for the preview environment to have the latest data from the upstream database. A shared database has the freshest data because it’s the same database. Dump / restore approach could provide fresh data as well, especially if you dump and restore the database on every PR. Migration scripts are the worst here because they don’t provide any data fresh data at all. Only the data captured in the migration / seed scripts would be available.

Migration scripts approach is somewhat automated. In the best-case scenario, you would have a script that creates a fresh database (e.g. a Terraform or Helm), and a script that runs the migrations (e.g. Rails Migrations or Liquibase). Dumping and restoring is less automated because most probably, it would involve a bash scripting with, let’s say, pg_dump and pg_restore commands. No copy approach requires no automation at all.

Compromises, compromises, compromises… As you see, you trade complexity for convenience. And because of that and because deep data isolation is rarely required, and because schema modification could be arranged manually, I suppose most projects start from a shared DB approach and never move further.

Meet Neon

Now, after such a long introduction, this section would be ridiculously (in a good way) short, even in my wordy style.

Imagine there is a PostgreSQL database, that is:

Fully managed
Serverless
Autoscalable
Auto-suspendable
Separates storage and compute
Branches in a second with a single button click or API call
And, yes, provides REST API for everything
Provides generous free tier to get you started
Well-documented

It exists, and it is Neon.

Because of the unique architecture, with separated storage and compute, Neon enables instant copy-on-write branching. You could create a branch, i.e. a copy of the database, from virtually any point in time in the upstream database, and it would be created in seconds. It means, that you could branch off the production database, with the latest data, and still have full isolation. Being serverless also means that Neon is very elastic, it can scale up and down, and even suspend the compute completely when it’s not used. On top of that, Neon provides a REST API for everything, so you can automate the whole process.

It’s simple to use, it’s highly isolated, it provides branches off the latest data, and it’s easily automatable. It’s perfect for preview environments.

Drawbacks? Well, it’s in the technical preview stage yet.

I don’t see a point in repeating the docs here, so here are some rabbit holes for you:

Exploring Neon branching with GitHub Actions and AWS Lambda

Link for the impatient ones: madhead/neonbranch.

Imagine an over-engineered rock-paper-scissors game, storing game rules in a database and providing a REST API to calculate the winner. The API is of no interest today, so it’s just an AWS Lambda, implemented in Python and deployed with CDK. Meet the database:

Now, imagine we’re working on two new features. The first one expands the game with Spock and lizard, and the second one adds textual descriptions for the outcomes:

What would the database look like

We have these changes captured in Liquibase changelogs. The question is: how do we organize the CI pipelines?

Events that trigger workflows

Remember, preview environments are ephemeral and bound to the PRs. So, let’s search for the appropriate triggers in Events that trigger workflows.

Three things could happen to a PR in regard to our scenario:

It could be opened or reopened
It could be synchronized
It could be closed

When a PR is opened or reopened, we should create a new Neon branch, when a PR is closed we should delete the associated branch. When a PR is synchronized, we run the same workflow as when a PR is opened or reopened, as it would be made idempotent.

Workflows

So, we’ll need three workflows:

.github/workflows/pr-env-create.yml

name: Create PR environment

on:
  pull_request:
    types:
      - opened
      - reopened

jobs:
  pr-env-create:
    steps:
      - id: deploy
        uses: ./.github/actions/deploy
        name: Deploy
        with:
          environment: pr-${{ github.event.number }}
          neon_project: ${{ vars.NEON_PROJECT }}
          neon_token: ${{ secrets.NEON_TOKEN }}

.github/workflows/pr-env-sync.yml

name: Synchronize PR environment

on:
  pull_request:
    types:
      - synchronize

# same as above

.github/workflows/pr-env-destroy.yml

name: Destroy PR environment

on:
  pull_request:
    types:
      - closed

jobs:
  pr-env-create:
    steps:
      - uses: ./.github/actions/destroy
        name: Destroy
        with:
          environment: pr-${{ github.event.number }}
          neon_project: ${{ vars.NEON_PROJECT }}
          neon_token: ${{ secrets.NEON_TOKEN }}

I extracted the common logic into two actions, deploy and destroy:

Deploy action

.github/actions/deploy/action.yml

name: Deploy
description: Deploy the lambda

inputs:
  environment:
  neon_project:
  neon_token:

outputs:
  URL:
  db_host:
  db_user:
  db_password:

runs:
  using: composite
  steps:
    - uses: actions/setup-python@v4
      with:
        python-version: 3.9

    - run: python -m venv .venv
      working-directory: deploy
      shell: bash

    - run: |
        source .venv/bin/activate
        python -m pip install -r requirements.txt
      working-directory: deploy
      shell: bash

    - id: branch
      run: |
        source .venv/bin/activate
        python branch.py
      working-directory: deploy
      shell: bash
      env:
        NEON_TOKEN: ${{ inputs.neon_token }}
        NEON_PROJECT: ${{ inputs.neon_project }}
        NEON_BRANCH: ${{ inputs.environment }}

    - uses: actions/setup-java@v3
      with:
        distribution: "temurin"
        java-version: 17
    - run: ./gradlew update
      working-directory: migrations
      shell: bash
      env:
        NEON_HOST: ${{ steps.branch.outputs.db_host }}
        NEON_DATABASE: ${{ inputs.neon_database }}
        NEON_USER: ${{ steps.branch.outputs.db_user }}
        NEON_PASSWORD: ${{ steps.branch.outputs.db_password }}

The logic is pretty simple:

We install Python
We set up a virtual environment and install the dependencies
We run a branch.py script which we’ll see in a moment
We install Java to run Liquibase
We run Liquibase

Destroy action

.github/actions/destroy/action.yml

name: Destroy
description: Destroy the preview environment

inputs:
  environment:
  neon_project:
  neon_token:

runs:
  using: composite
  steps:
    - uses: actions/setup-python@v4
      with:
        python-version: 3.9

    - run: python -m venv .venv
      working-directory: deploy
      shell: bash

    - run: |
        source .venv/bin/activate
        python -m pip install -r requirements.txt
      working-directory: deploy
      shell: bash

    - run: |
        source .venv/bin/activate
        python unbranch.py
      working-directory: deploy
      shell: bash
      env:
        NEON_TOKEN: ${{ inputs.neon_token }}
        NEON_PROJECT: ${{ inputs.neon_project }}
        NEON_BRANCH: ${{ inputs.environment }}

We just run a different script, unbranch.py.

Automation scripts

Let’s extract Neon API calls into a separate Python module:

neon.py

import os
import time

import requests

base_url = 'https://console.neon.tech/api/v2'
headers = {
    'Accept': 'application/json',
    'Authorization': f"Bearer {os.environ['NEON_TOKEN']}"
}


def find_project(project_name: str) -> dict:
    projects = requests.get(
        url=f"{base_url}/projects",
        headers=headers,
    ).json()['projects']

    return next(project for project in projects if project['name'] == project_name)


def delete_branch(project: dict, name: str) -> dict:
    branches = requests.get(
        url=f"{base_url}/projects/{project['id']}/branches",
        headers=headers,
    ).json()['branches']
    branch = next((branch for branch in branches if branch['name'] == name))
    requests.delete(
        url=f"{base_url}/projects/{project['id']}/branches/{branch['id']}",
        headers=headers,
    )


def find_branches(project: dict, name: str) -> (dict, dict):
    branches = requests.get(
        url=f"{base_url}/projects/{project['id']}/branches",
        headers=headers,
    ).json()['branches']

    primary_branch = next(branch for branch in branches if branch['primary'])
    if name:
        branch = next((branch for branch in branches if branch['name'] == name), None)
    else:
        branch = primary_branch

    return primary_branch, branch


def get_operation_details(project: dict, operation_id: str) -> dict:
    return requests.get(
        url=f"{base_url}/projects/{project['id']}/operations/{operation_id}",
        headers=headers,
    ).json()['operation']


def create_branch(project: dict, parent: dict, name: str) -> dict:
    result = requests.post(
        url=f"{base_url}/projects/{project['id']}/branches",
        headers=headers,
        json={
            'endpoints': [
                {
                    'type': 'read_write'
                }
            ],
            'branch': {
                'parent_id': parent['id'],
                'name': name,
            }
        },
    ).json()
    operations = result['operations']

    for operation in operations:
        while True:
            operation_details = get_operation_details(project, operation['id'])
            if operation_details['status'] == 'finished':
                break
            else:
                time.sleep(5)

    return result['branch']


def find_endpoint(project: dict, branch: dict) -> dict:
    endpoints = requests.get(
        url=f"{base_url}/projects/{project['id']}/endpoints",
        headers=headers,
    ).json()['endpoints']

    return next(endpoint for endpoint in endpoints if endpoint['branch_id'] == branch['id'])


def find_role(project: dict, branch: dict) -> dict:
    roles = requests.get(
        url=f"{base_url}/projects/{project['id']}/branches/{branch['id']}/roles",
        headers=headers,
    ).json()['roles']

    return next(role for role in roles if not role['protected'])


def get_password(project: dict, branch: dict, role: dict) -> str:
    return requests.get(
        url=f"{base_url}/projects/{project['id']}/branches/{branch['id']}/roles/{role['name']}/reveal_password",
        headers=headers,
    ).json()['password']

As you see, just a bunch of functions to call Neon API. Let’s see how we use them in our scripts.

branch.py

import os

import github
import neon

if __name__ == '__main__':
    project = neon.find_project(os.environ['NEON_PROJECT'])
    primary_branch, branch = neon.find_branches(project, os.environ.get('NEON_BRANCH'))

    if not branch:
        branch = neon.create_branch(project, primary_branch, os.environ.get('NEON_BRANCH'))

    endpoint = neon.find_endpoint(project, branch)
    role = neon.find_role(project, branch)
    password = neon.get_password(project, branch, role)

    github.mask(endpoint['host'])
    github.mask(role['name'])
    github.mask(password)

    github.set_output('db_host', endpoint['host'])
    github.set_output('db_user', role['name'])
    github.set_output('db_password', password)

unbranch.py

import os

import neon

if __name__ == '__main__':
    neon.delete_branch(
        neon.find_project(os.environ['NEON_PROJECT']),
        os.environ['NEON_BRANCH'],
    )

While unbranch.py is pretty straightforward, branch.py is a bit more complicated. First, we find the project by its name. Then we find the primary branch we fork off and the branch for the preview environment. If the branch doesn’t exist, we create it. The creation of a branch is an asynchronous operation, so we have to wait for it to finish. Then we find the endpoint and the role for the new branch and get the password for the role. Finally, we mask the sensitive data and set the output variables for GitHub Actions. They are used later by other scripts, not shown here.

And that’s it! We now have a fully automated workflow that creates a new database branch for each environment and deletes it when the environment is deleted.

Explore the demo repository, madhead/neonbranch to learn more tricks, like this:

Two more things before you go.

First, there is an official Neon GitHub Action, but with a slightly different logic and use case: neondatabase/create-branch-action.

Second, if you’re using Vercel, there is an article for that: A database for every preview environment using Neon, GitHub Actions, and Vercel.

From BotFather to 'Hello World'

madhead — Thu, 27 Oct 2022 13:09:15 +0000

Great news, everybody: a few days ago, Telegram team updated their Bot API documentation! Among other things, they now have an end-to-end tutorial on the bot creation process: starting from bot registration in @BotFather, going all the way to database & deployment options.

Unfortunately, it features Java, Maven, and rubenlagus/TelegramBots library.

So, I’ve decided to copycat it, upgrading the examples to Kotlin, Gradle, and InsanusMokrassar/TelegramBotAPI! Read my version below 👇🏼

Parts of the text below are shamelessly copy-pasted from the original tutorial. But don’t be tough on me: I respect copyright and authorship and in this case, I’m appealing for the fair use doctrine, which, I believe, this article would fall into.

This guide will walk you through everything you need to know to build your first Telegram Bot.

If you already know your way around some of the basic steps, you can jump directly to the part you’re missing. Equivalent examples are available in C#, Python, Go, and TypeScript.

Introduction
Basic Tutorial
- Environment
- First Run
- Echo Bot
Advanced Tutorial
- Commands
- Navigation
- Database
- Hosting
Further Reading

Introduction

At its core, you can think of the Telegram Bot API as software that provides JSON-encoded responses to your queries.

A bot, on the other hand, is essentially a routine, software or script that queries the API by means of an HTTPS request and waits for a response. There are several types of requests you can make, as well as many different objects that you can use and receive as responses.

Since your browser is capable of sending HTTPS requests, you can use it to quickly try out the API. After obtaining your token, try pasting this string into your browser:

https://api.telegram.org/bot<YOUR_BOT_TOKEN>/getMe

In theory, you could interact with the API with basic requests like this, either via your browser or other tailor-made tools like cURL. While this can work for simple requests like the example above, it’s not practical for larger applications and doesn’t scale well.

For that reason, this guide will show you how to use libraries and frameworks, along with some basic programming skills, to build a more robust and scalable project.

If you know how to code, you’ll fly right through each step in no time — and if you’re just starting out, this guide will show you everything you need to learn.

We will use Kotlin throughout this guide as it’s one of the most popular programming languages, however, you can follow along with any language as all the steps are fundamentally the same. Since JVM is fully cross-platform, each code example will work with any operating system. If you pick another language, equivalent examples are available in C#, Python, Go and TypeScript.

Getting Ready

First, we will briefly cover how to create your first project, obtain your API token and download all necessary dependencies and libraries.

For the purposes of this guide, a similar bot from the original tutorial is also live at @TutorialBot — feel free to check it out along the way to see how your own implementation should look after each step.

Obtain Your Bot Token

In this context, a token is a string that authenticates your bot (not your account) on the bot API. Each bot has a unique token, which can also be revoked at any time via @BotFather.

Obtaining a token is as simple as contacting @BotFather, issuing the /newbot command, and following the steps until you’re given a new token. You can find a step-by-step guide here.

Your token will look something like this:

4839574812:AAFD39kkdpWt3ywyRZergyOLMaJhac60qc

Make sure to save your token in a secure place, treat it like a password, and don’t share it with anyone.

Download an IDE

To program in Kotlin, you’ll need an IDE — a special text editor that will let you write, compile and run your code. In this tutorial, we’ll use IntelliJ IDEA — there are several free, open source alternatives like Eclipse or NetBeans which work in the exact same way.

You will also need a JDK, a software kit that allows your Kotlin code to run. Most IDEs don’t include a JDK, so you should download a version compatible with your operating system separately. You can find a free, open source version here.

If you use another language, the steps are identical. You will just have to download a different IDE and software development kit.

Pick a Framework or Library

You can think of a framework as software that handles all the low-level logic for you, including the API calls, and lets you focus on your bot-specific logic.

In this tutorial, we’ll use InsanusMokrassar/TelegramBotAPI, but you can follow along with any equivalent implementation since all the underlying methods are either similar or exactly the same.

You can find many frameworks, along with code examples, in the dedicated list.

Create Your Project

In IntelliJ, go to File > New > Project.

Fill in the fields accordingly:

Name — The name of your project. For example, from-botfather-to-hello-world.
Location — Where to store your project. You can use the default value.
Language — Kotlin
Build System — The framework that will handle your dependencies. Pick Gradle.
JDK — Pick whichever version you downloaded. We’ll be using version 17.
Gradle DSL — Select Kotlin as well.
Add Sample Code — Leave this selected, it will generate some needed files for you.
Advanced Settings > GroupId — Use your reversed domain here or tutorial for the sake of this guide.
Advanced Settings > ArtifactId — You can use the default value.

After hitting Create, if you did everything correctly, your Project view in the top left should show a project structure along these lines:

from-botfather-to-hello-world
├── build.gradle.kts
├── gradle
│   └── wrapper
│       ├── gradle-wrapper.jar
│       └── gradle-wrapper.properties
├── gradle.properties
├── gradlew
├── gradlew.bat
├── settings.gradle.kts
└── src
    ├── main
    │   ├── kotlin
    │   │   └── Main.kt
    │   └── resources
    └── test
        ├── kotlin
        └── resources

Other IDEs will follow a similar pattern. Your dependency management system will have a different name (or no name at all if it’s built-in) depending on the language you chose.

If this looks scary, don’t worry. We will only be using the Main.kt file and the build.gradle.kts file. In fact, to check that everything is working so far, double-click on Main and click on the small green arrow on the left of fun main, then select the first option (Run 'MainKt'). If you followed the steps correctly, Hello World! should appear in the console below.

Add Framework Dependency

We will now instruct the IDE to download and configure everything needed to work with the API. This is very easy and happens automatically behind the scenes.

First, locate your build.gradle.kts file on the left side of the screen. Open it by double-clicking and simply add the library to the dependencies section:

dependencies {
    implementation("dev.inmo:tgbotapi:3.2.7")
}

When you’re done, your build.gradle.kts should look something like this.

Start Coding

We are ready to start coding. If you’re a beginner, consider that being familiar with your language of choice will greatly help. With this tutorial, you’ll be able to teach your bot basic behaviors, though more advanced features will require some coding experience.

Creating a Bot

There are two mutually exclusive ways of receiving updates in your bot: either by polling for them or by receiving them via a webhook.

Polling essentially means asking for the updates in an endless loop, i.e. you pull the updates from Telegram servers. With webhooks, updates are pushed into your bot by Telegram via HTTPS.

You decide which way suits you best, and the library provides you several methods to create bots depending on your choice.

We’ll start with the telegramBotWithBehaviourAndLongPolling:

suspend fun main(args: Array<String>) {                            // <1>
    val token = args[0]                                            // <2>
    val (_, job) = telegramBotWithBehaviourAndLongPolling(token) { // <3>
        this
            .allUpdatesFlow                                        // <4>
            .onEach { println(it) }                                // <5>
            .launchIn(GlobalScope)                                 // <6>
    }

    job.join()                                                     // <7>
}.

Add a suspend modifier to your main function, as the library makes heavy use of coroutines.
It’s better to pass the token as an argument for your program than hard-coding it.
telegramBotWithBehaviourAndLongPolling returns a pair of values: the bot itself and a Job. We’re not very interested in the bot value, so we do not assign it to any variable, that’s what _ means. We’ll use this job later, read below.
allUpdatesFlow is a Flow, i.e. a lazy, potentially endless sequence of Updates. Flows are used to listen to or transform the values that flow through them.
Here we just listen for the values and print them.
This listening process should occur in a CoroutineScope. For this simple example, we’ll use a GlobalScope.
Remember the job that we get at step #3? Here we wait for its completion. And this job would never actually complete on its own because the number of updates your bot potentially receives is unbound.

You might be confused by all these coroutines, jobs, scopes, flows, and, especially, by the waiting for the completion of an endless loop — it sounds like nonsense! But those are the very basic concepts and patterns of Kotlin Coroutines. After finishing this tutorial you might want to read more about them.

First Run

It’s time to run your bot for the first time. Hit the green arrow to the left of fun main and select the first option (Run 'MainKt').

And then there was nothing. Yes, a bit anticlimactic. This is because your bot has nothing to print – there are no new updates because nobody messaged it yet.

If you try messaging the bot on Telegram, you’ll then see new updates pop up in the console. At this point, you have your very own Telegram Bot – quite the achievement. Now, on to making it a bit more intelligent.

If nothing pops up, make sure you messaged the right bot and that the token you pasted in the code is correct.

Receiving Messages

Every time someone sends a private message to your bot, the onContentMessage callback will be called, and you’ll be able to handle the update (named it, if you don’t override the name of the parameter), which contains the message, along with a great deal of other info which you can see detailed here.

Let’s focus on two values for now:

The user — Who sent the message. Access it via it.from.
The message — What was sent. Access it via it.text.

Knowing this, we can make it a bit more clear in the console output.

-this.allUpdatesFlow.onEach { println(it) }.launchIn(GlobalScope)

+onContentMessage {
+    val user = it.from
+    println("${user?.firstName ?: "Unknown user"} wrote ${it.text}")
+}

This is just a basic example – you can now play around with all the methods to see everything you can pull out of these objects. You can try it.from.firstName, it.chat, and dozens more.

Knowing how to receive, process, and print incoming messages, now it’s time to learn how to answer them.

Remember to stop and re-launch your bot after each change to the code.

Sending Messages

To send a private text message, you generally need three things:

The user must have contacted your bot first. (Unless the user sent a join request to a group where your bot is an admin, but that’s a more advanced scenario).
You must have previously saved the User ID (it.from.id)
A String object containing the message text, 1–4096 characters.

With that out of the way, let’s send the first message:

suspend fun main(args: Array<String>) {
    val token = args[0]
    val bot = telegramBot(token)                                   // <1>

    bot.sendTextMessage(ChatId(1234), "Hello, World!")             // <2>
}

As long as we don’t build a complex behavior for the bot here, a simple telegramBot is enough for this example.
For this example, we’ll assume your ID is 1234. You could get your real ID from the updates received in the previous steps, or by contacting @myidbot

If you did everything correctly, your bot should text you Hello World! every time you launch your code. Sending messages to groups or channels – assuming you have the relevant permissions – is as simple as replacing 1234 with the ID of the respective chat.

Try experimenting with other types of messages, like sendPhoto, sendSticker, sendDice… A full list is available starting here.

Echo Bot

Let’s practice everything we tried so far by coding an Echo Bot. Its functionality will be rather simple: every text message it receives will be sent right back to the user.

Copying Text

The most intuitive way of coding this is just replying to any message… with itself.

In other words:

suspend fun main(args: Array<String>) {
    val token = args[0]
    val bot = telegramBot(token)

    bot.buildBehaviourWithLongPolling {                            // <1>
        onContentMessage {                                         // <2>
            bot.reply(it, it)                                      // <3>
        }
    }.join()
}

Here we’ll explore just another way to set up long polling and bot behavior: buildBehaviourWithLongPolling.
onContentMessage will be called for every message with content, e.g. text, stickers, and pictures. There are messages with no content as well, like join requests.
The reply simply replies to a message.

This tutorial assumes that updates always contain messages for the sake of simplicity. This may not always be true – be sure to implement all the proper checks in your code to handle every type of update with the appropriate methods.

Executing Commands

To learn what a command is and how it works, we recommend reading this dedicated summary. In this guide, we’ll focus on the technical side of things.

Creating Your Command

Begin by opening @BotFather. Type /mybots > Your_Bot_Name > Edit Bot > Edit Commands.

Now send a new command, followed by a brief description. For the purpose of this tutorial, we’ll implement two simple commands:

scream — Speak, I'll scream right back
whisper — Shhhhhhh

Command Logic

We want the Echo Bot to reply in uppercase when it’s in scream mode and normally otherwise.

First, let’s create a variable to store the current mode.

suspend fun main(args: Array<String>) {
    val token = args[0]
    val bot = telegramBot(token)
+   var screaming = false

    …
}

Then, let’s add a logic to switch the mode.

bot.buildBehaviourWithLongPolling {
    …

+   onCommand("scream") {
+       screaming = true
+   }
+   onCommand("whisper") {
+       screaming = false
+   }

    …
}.join()

The onCommand is called whenever the bot receives a command that is passed as a parameter to the onCommand.

Next, let’s add an extension for the CommonMessage to check if it contains any commands. We’ll use this extension as a filter in the next step.

fun CommonMessage<*>.hasCommands(): Boolean =
    (this.content as? TextContent)?.textSources?.any { it.botCommandTextSourceOrNull() != null } ?: false

fun CommonMessage<*>.hasNoCommands(): Boolean = !this.hasCommands()

Probably when you’re reading these lines, these extensions are already included in the library.

Finally, let’s tune our onContentMessage to support the modality.

onContentMessage(
    initialFilter = CommonMessage<MessageContent>::hasNoCommands
) {
    val text = it.text

    if (text != null) {
        bot.reply(it, if (screaming) text.uppercase() else text)
    } else {
        bot.reply(it, it)
    }
}

As you can see, it checks if the message is a text. If it is, the bot additionally checks if it is in a screaming mode before it replies.

And that’s it. Now the bot can execute commands and change its behavior accordingly.

Naturally, this simplified logic will change the bot’s behavior for everyone – not just the person who sent the command. This can be fun for this tutorial but won’t work in a production environment – consider using a Map, dictionary, or equivalent data structure to assign settings for individual users.

Remember to always implement a few basic global commands. You can practice by implementing simple feedback to the /start command, which we intentionally left out.

Buttons and Keyboards

To streamline and simplify user interaction with your bot, you can replace many text-based exchanges with handy buttons. These buttons can perform a wide variety of actions and can be customized for each user.

Button Types

There are two main types of buttons:

Reply Buttons — are used to provide a list of predefined text reply options.
Inline Buttons — are used to offer quick navigation, shortcuts, URLs, games and so much more.

Using these buttons is as easy as attaching a ReplyKeyboardMarkup or an InlineKeyboardMarkup to your message.

This guide will focus on inline buttons since they only require a few extra lines of code.

Creating Buttons

First of all, let’s create some buttons.

val next = CallbackDataInlineKeyboardButton(text = "Next", callbackData = "next")
val back = CallbackDataInlineKeyboardButton(text = "Back", callbackData = "back")
val url = URLInlineKeyboardButton(text = "Tutorial", url = "https://madhead.me/posts/from-botfather-to-hello-world")

Let’s go back through the fields we specified:

Text — This is what the user will see, the text that appears on the button
Callback Data — This will be sent back to the code instance as part of a new CallbackQueryUpdate, so we can quickly identify what button was clicked.
Url — A button that specifies a URL doesn’t specify callbackdata since its behavior is predefined – it will open the given link when tapped.

Creating Keyboards

The buttons we created can be assembled into two keyboards, which will then be used to navigate back and forth between two sample menus.

val firstMenuMarkup = flatInlineKeyboard { next }
val secondMenuMarkup = inlineKeyboard {
    row { back }
    row { url }
}

You can place this code wherever you prefer, the important thing is to make sure that keyboard variables are accessible from the method call that will send the new menu. If you’re confused by this concept and don’t know where to put them, just paste them above the command processing flow. Or just look at the final code here.

Sending Keyboards

Sending a keyboard only requires specifying a reply markup for the message.

bot.sendMessage(
    chat = someChat,
    text = someText,
    replyMarkup = someReplyMarkup
)

Menu Trigger

We could send a new menu for each new user, but for simplicity let’s add a new command that will spawn a menu. We can achieve this by adding a new onCommand("menu") clause to the previous behavior block.

onCommand("menu") {
    bot.sendMessage(
        chat = it.chat,
        text = "<b>Menu 1</b>",
        parseMode = HTMLParseMode,
        replyMarkup = firstMenuMarkup
    )
}

You may have noticed that we also added a new parameter, parseMode = HTMLParseMode. This is called a formatting option and will allow us to use HTML tags and add formatting to the text later on.

Try sending /menu to your bot now. If you did everything correctly, you should see a brand new menu pop up.

In a production environment, commands should be handled with an appropriate design pattern that isolates them into different executor classes – modular and separated from the main logic.

Navigation

When building complex bots, navigation is essential. Your users must be able to move seamlessly from one menu to the next.

In this example, we want the Next button to lead the user to the second menu. The Back button will send us back. To do that, we will start processing incoming MessageDataCallbackQueries, which are the results we get after the user taps on a button.

A MessageDataCallbackQuery is essentially composed of three main parameters:

id — Needed to close the query. You must always close new queries after processing them – if you don’t, a loading symbol will keep showing on the user’s side on top of each button.
from — The user who pressed the button.
data — This identifies which button was pressed.

Processing in this context just means executing the action uniquely identified by the button, then closing the query.

A very basic button handler could look something like this:

bot.buildBehaviourWithLongPolling() {
    onMessageDataCallbackQuery("next") {
        bot.edit(
            chatId = it.message.chat.id,
            messageId = it.message.messageId,
            text = secondMenu,
            parseMode = HTMLParseMode,
            replyMarkup = secondMenuMarkup,
        )
    }

    onMessageDataCallbackQuery("back") {
        bot.edit(
            message = it.message as ContentMessage<TextContent>,
            text = firstMenu,
            parseMode = HTMLParseMode,
            replyMarkup = firstMenuMarkup,
        )
    }

    onDataCallbackQuery {
        bot.answerCallbackQuery(it)
    }
}.join()

With these handlers, whenever a button is tapped, your bot will automatically navigate between inline menus. Expanding on this concept allows for endless combinations of navigable submenus, settings, and dynamic pages.

Database

Telegram does not host an update database for you – once you process and consume an update, it will no longer be available. This means that features like user lists, message lists, current user inline menus, settings, etc. have to be implemented and maintained by bot developers.

If your bot needs one of these features and you want to get started on data persistence, we recommend that you look into serialization practices and libraries for your language of choice, as well as available databases.

Implementing a database is out of scope for this guide, however, several guides are available online for simple embedded open source software solutions like SQLite, HyperSQL, Derby, and many more.

Your language of choice will also influence which databases are available and supported – the list above assumes you followed this Kotlin tutorial.

Hosting

So far, your bot has been running on your local machine – your PC. While this may be good for developing, testing, and debugging, it is not ideal for a production environment. You’ll want your bot to be available and responsive at all times, but your computer might not always be online.

This can be done in four steps:

Package your code

Making your bot easy to move and runnable outside of an IDE is essential to host it elsewhere. If you followed this tutorial, this standard guide will work for you. If you didn’t, look into export or packaging guides for your IDE and language of choice – procedures may vary but the end result is the same.
Purchase a VPS or equivalent service

A server is essentially a machine that is always online and running, without you having to worry about anything. To host your bot, you can opt for a VPS which serves this purpose and can be rented from several different providers. Another option would be to purchase a network-capable microcontroller, which come in all different specs and sizes depending on your needs.

You should ensure that all user data remains heavily encrypted at all times in your database to guarantee the privacy of your users. The same concept applies to your local instance, however, this becomes especially important once you transfer your database to a remote server.
Upload your executable/package

Once you have a working ssh connection between your machine and your new server, you should upload your executable and all associated files. We will assume the runnable jar TutorialBot.jar and its database dbase.db are currently in the /TBot folder.
```
$ scp -r /TBot/ username@server_ip:/bots/TBotRemote/
```
Run your application

Depending on which language you chose, you might have to configure your server environment differently. If you chose Kotlin, you just need to install a compatible JDK.
```
$ apt install openjdk-17-jre
$ java -version
```
If you did everything correctly, you should see a Java version as the output, along with a few other values. This means you’re ready to run your application.

Now, to run the executable:
```
$ cd /bots/TBotRemote/
$ java -jar TutorialBot.jar
```

Your bot is now online and users can interact with it at any time.

To streamline and modularize this process, you could employ a specialized docker container or equivalent service. If you followed along in one of the equivalent examples (C#, Python, Go, and TypeScript) you can find a detailed set of instructions to export and run your code here.

Answer: What is the differences between the term SSTable and LSM Tree

madhead — Thu, 29 Sep 2022 10:55:49 +0000

Feb 14 '22

Probably one of the best explanations of SSTables and LSM-Trees for mortals is given by Martin Kleppmann in his "Designing Data-Intensive Applications" book. These data structures are explained in chapter 3, "Storage and Retrieval", pages 69 through 79. It's a really great read, I would recommend the whole book!

Impatient…

Open Full Answer

Docker-based GitHub Actions in orphan branches

madhead — Mon, 11 Oct 2021 00:51:19 +0000

GitHub Actions are great for sharing & reusing logic across your CI/CD pipelines. However, sometimes sharing is not the highest priority. Sometimes you only want to hide the complexity by exctacting & encapsulating multiple build steps and replacing them with a single action.

Anyway, you have to create a GitHub action.

You could use actions stored in the same repository as the workflow. Such actions must be either written in JavaScript or be Docker actions, i.e. have a Dockerfile.

With JavaScript, at the time of the writing, actions must be written literally in JavaScript, no processing like compilation or transpilation allowed. In other words, it cannot be TypeScript: although you could use TypeScript to write an action, you’ll have to commit the dist then.

Docker actions are slow because the caching doesn’t work well at the time of the writing. Images are built from scratch every time you run a pipeline.

Finally, you may be against of polluting the main codebase with action’s code, and this is a valid argument too!

Because of this issues — polluting main codebase, forcing you to use vanilla JavaScript or being slow with other languages and runtimes — in-repository actions might not look appealing to you.

So you might want to take the hard way: to extract your action into a separate repository. But even then caching won’t work well with Docker actions and you’ll have to commit dist with JavaScript actions. However, in a separate repository you could at least have action’s own lifecycle and mitigate that with automation.

😔 That’s some extra work…

Try orphan branches instead!

The idea is to create an orphan branch in your repository. Orphan means it wont have any intersections or common commits with the main branch, a completely separate one. GitHub won’t even allow you to make a PR to the main branch from the orphan one.

Thinking about Git as a tree was a mistake, it’s actually a forest!

Now you could store your whole action, written in any language, in this branch. It would have a completely different lifecycle by having completely different workflows in this branch’s .github/workflows directory. If it’s a Docker action, you could built and publish it to the GitHub packages of the same repository and use it the main branch’s workflows without rebuilding every time.

All the issues are solved!

✅ The codebase wouldn’t have any traces of the auxiliary actions code.

✅ Actions would have a completely independent lifecycle.

✅ Docker actions would be prebuilt.

Two design patterns for Telegram Bots

madhead — Mon, 13 Sep 2021 23:08:01 +0000

I've just launched my next Telegram Bot: Tyzenhaus. It is a shared expenses tracking bot. Whether you are a group of travelers ✈️, roommates 🏠, or just friends 👫, Tyzenhaus will help you settle up your debts. Just like Splitwise, but simpler and directly in Telegram. This half-minute video describes it:

So, with a dozen bots written in Kotlin for the past two years, I want to sum up my experience and name the two most important design/architecture patterns for Telegram Bots. You'll definitely find them useful when making your own bot.

Chain of Responsibility

When writing web apps, we, the backend developers, are used to have different handlers for different URLs. Those are configured in a framework-specific manner, like annotations in Java, URLconf in Django or routing in Node.js. Let's use this name — routing — to describe this way of configuring the logic.

But this is not the case with Telegram Bot API. If you're using webhooks, all the updates will be sent to a single destination in your application. It also works the same way with long polling: you'll have a single place in your codebase, calling getUpdates in a loop. There is no routing and you have to implement it yourself.

The best way to do that is by employing a chain of responsibility pattern.

To do that, you should separate your bot logic and enclose it into small bits. Let's name them handlers. Handlers could be functions or classes, it doesn't really matter. Every handler is completely independent in a way that it solely decides whether it processes a particular incoming update or not. This is crucial and this is what makes chain of responsibility different from routing. So let me repeat: a handler decides to process or skip the update on its own, not an external router.

Finally, you combine your handlers in a list and dispatch every incoming update through it.

This is a general concept, actual implementations may vary. This chain shouldn't even be sequential: handlers could run in parallel. Although make sure that at most one handler will process an update, otherwise you may have flaky replies or replies out of order.

Finite State Machine

Another thing we're used to are smarts client for our APIs. Even the so-called thin clients are smarter than Telegram! Although we don't trust the input and always validate it on the backend (don't we?), clients may have forms grouping multiple fields and even wizard-like workflows, combining multiple forms. I mean, usually, you get a whole JSON object with multiple fields as an input. You never have to process the input one field at a time, which is exactly the case with Telegram.

Thus, you have to store all the state on a server. You may have heard about state pattern, but what I'm using is more like FSM.

FSM is just a fancy way of saying "a graph". FSM is a directed graph of states with well-known transitions between them. "Well-known" means that for every possible state you know the exact rules for transitions to other states. This predictability makes FSMs very easy to understand, maintain and extend.

A lot of processes could be described with FSMs. Look at this one, describing the expense workflow in my bot:

Green edges are "success paths", when the input is valid and the logic state could be transitioned. Red edges are unsuccessful attempts of user to provide input. E.g. passing arbitrary non-numeric strings as monetary amounts.
You've got the idea in a few seconds, don't you?

Combining chain of responsibility with FSM

And here is the beauty of those two patterns combined in three simple steps:

Process updates with a chain of responsibility. Handlers should have two inputs: incoming Telegram Bot API update and current state. It could be a state for a single user in a private chat, a group state, or even a particular user state in a particular group — it doesn't matter;
Save the resulting state;
Repeat;

You could implement literally any bot for any messenger with this simple recipe. Don't you believe me? Go and try yourself, I bet you have some great ideas!

🤑 Try Tyzenhaus out | ⭐ Star it on GitHub | 🤬 Create an issue on GitHub.

Reading Java properties in GitHub Actions

madhead — Tue, 06 Apr 2021 09:32:38 +0000

I just want to share another GitHub Action (create by me) that you may find useful in your Java projects.

Java projects often use .properties files to store configuration. Personally, I hate them in favor of YML or even XML (I beg you: use XML to configure Log4j, not .properties), but still, I have to work with them. There are existing actions in the Marketplace able to parse those, but they usually use grep and bash. Although it may work, I think it's more robust to use special facilities, like java.util.Properties to read values from .properties files.

So, I wrote a simple Kotlin program that reads Java .properties and wrapped it into a GitHub Action. This Action makes values from .properties available in your GitHub Actions workflows.

Returning a single property is as simple as:

- uses: madhead/read-java-properties@latest
  id: version
  with:
    file: gradle.properties
    property: version
    default: 0.0.1

- run: echo ${{ steps.version.outputs.value }} # Project's version from gradle.properties or 0.0.1 if it is not defined there

Alternatively, you could return all the values from the .properties file by employing the all flag:

- uses: madhead/read-java-properties@latest
  id: all
  with:
    file: gradle.properties
    all: true

- run: echo ${{ steps.all.outputs.version }} # Project's version from gradle.properties
- run: echo ${{ steps.all.outputs.groupId }} # Project's groupId from gradle.properties
  …

That's all for today, just save the link: madhead/read-java-properties, and, as always, feel free to open issues if something is not working as needed or expected.

One-stop shop for working with semantic versions in your GitHub Actions workflows

madhead — Tue, 30 Mar 2021 08:21:56 +0000

GitHub Actions are so cool!

Check out our recording about them @ Ministry of Testing Abu Dhabi if you want a quick sneak peek, but today I want to self-advertise one of the reusable Actions I made for you. And it wasn’t hard at all, thanks to the way they work.

You have heard about Semantic Versioning, haven’t you? If you are a developer you probably know a lot of libraries using Semantic Versioning. Chances are, you even use it for your projects.

Sometimes during CI/CD in such projects, you need to take simple actions with versions: compare and parse them. E.g. you may want to check that a PR to the main branch increases the version ($new_version > $old_version) or you may want to create or update a v$major.$minor / v$major tag whenever you release your project.

This is what madhead/semver-utils is for! Just drop it into your workflow and use it for various operations with versions:

- uses: madhead/semver-utils@latest
  id: version
  with:
    # A version to work with
    version: 1.2.3+42.24

    # A version to compare against
    compare-to: 2.1.0

    # A range to check agains
    satisfies: 1.x
- run: |
    echo "${{ steps.version.outputs.major }}"             # 1
    echo "${{ steps.version.outputs.minor }}"             # 2
    echo "${{ steps.version.outputs.patch }}"             # 3
    echo "${{ steps.version.outputs.build }}"             # 42.24
    echo "${{ steps.version.outputs.build-parts }}"       # 2
    echo "${{ steps.version.outputs.build-0 }}"           # 42
    echo "${{ steps.version.outputs.build-1 }}"           # 24
    echo "${{ steps.version.outputs.comparison-result }}" # <
    echo "${{ steps.version.outputs.satisfies }}"         # true
    echo "${{ steps.version.outputs.inc-major }}"         # 2.0.0
    echo "${{ steps.version.outputs.inc-premajor }}"      # 2.0.0-0
    echo "${{ steps.version.outputs.inc-minor }}"         # 1.3.0
    echo "${{ steps.version.outputs.inc-preminor }}"      # 1.3.0-0
    echo "${{ steps.version.outputs.inc-patch }}"         # 1.2.4
    echo "${{ steps.version.outputs.inc-prepatch }}"      # 1.2.4-0
    echo "${{ steps.version.outputs.inc-prerelease }}"    # 1.2.4-0

Yeah, it's basically a wapper around semver package, so the outputs may look familiar to you. But if you need more in your workflows — feel free to open an issue with a feature you’re missing.

I want this action to become a one-stop shop for working with semantic versions in your GitHub Actions workflows one day!