Forem: João Victor Martins

Scopes in Coroutines - coroutineScope x supervisorScope

João Victor Martins — Mon, 07 Oct 2024 12:45:14 +0000

As a Kotlin developer, coroutines are an essential tool for writing efficient and performant code. When you know how to work with this tool, you can enhance your app's performance with simpler code. Writing code with coroutines is less complex than using other tools for asynchronous development, but, like any technology, you need to be aware of best practices to avoid potential issues. There are several key concepts you should understand when working with coroutines, including builders, scopes, contexts, and flows. In this post, I aim to explain the differences between coroutineScope and supervisorScope, starting with an introduction to scopes in coroutines.

Scopes

According to Kotlin's documentation, CoroutineScope defines a scope for new coroutines. Every coroutine builder (such as launch, async, etc.) is an extension of CoroutineScope and inherits its coroutineContext, automatically propagating all its elements and cancellation.

In simple terms, a scope is required to create coroutines and execute tasks. You can create scopes in several ways, one of which is using the runBlocking builder.

As you can see, the IDE shows a CoroutineScope after the curly brace. Inside this scope, you can create multiple coroutines. runBlocking is the easiest way to bridge from the non-coroutine world to the coroutine world, but be cautious when using it in production. As its name suggests, it is a blocking operation, which means it will block the main thread. If you remove runBlocking, the code won't compile because the launch builder won't be within a scope.

There are two other common ways to create scopes: using supervisorScope and coroutineScope. Both are the main subjects of this post, and I will explain each one in the following sections.

coroutineScope

coroutineScope is a function, and according to Kotlin's documentation, it is designed for concurrent decomposition of work. If any child coroutine in this scope fails, the scope fails, cancelling all other children. This function returns as soon as the given block and all its child coroutines are completed. An example of using a scope looks like this:

suspend fun main() {
    call()
}

suspend fun call() {
    coroutineScope {
        launch { executeOneDay(3) }
        launch { executeOneDay(1) }
    }
}

suspend fun executeOneDay(int: Int) {
    println("entrou")
    val numberOfItems = if (int == 1) {
        delay(1000)
        error("some error")
    } else {
        delay(2000)
        int
    }
    runCatching { teste(numberOfItems) }.onFailure { println(it.message) }
}

fun teste(int: Int) {
    println("chegou no teste $int")
}

The first difference to note is the suspend keyword in the method signature. The function must be suspend to create a scope with coroutineScope. Unlike runBlocking, it does not block execution; instead, it utilizes suspend and resume activities. Let’s break down what this code does: The main() function calls the call() function. Inside call(), a suspend function, I create two coroutines using the launch builder within coroutineScope. Each coroutine calls the executeOneDay(int: Int) function with different parameters: 3 and 1.

In executeOneDay, if the parameter is 1, the code delays for 1 second and generates an error; if it’s anything other than 1, it delays for 2 seconds and returns the number. Since both coroutines run asynchronously, one will generate an error while the other will not. When you execute this code, you will see "entrou" printed twice in your terminal, followed by the error message, terminating the program. This occurs because, when one coroutine fails, coroutineScope cancels all other coroutines. This behavior is expected, but if you want different behavior, you can use supervisorScope.

supervisorScope

According to Kotlin's documentation, with supervisorScope, a failure of a child coroutine does not cause the scope to fail and does not affect other children. If we modify the previous code to use supervisorScope, the result will change:

suspend fun main() {
    call()
}

suspend fun call() {
    supervisorScope {
        launch { executeOneDay(3) }
        launch { executeOneDay(1) }
    }
}

suspend fun executeOneDay(int: Int) {
    println("entrou")
    val numberOfItems = if (int == 1) {
        delay(1000)
        error("some error")
    } else {
        delay(2000)
        int
    }
    runCatching { teste(numberOfItems) }.onFailure { println(it.message) }
}

fun teste(int: Int) {
    println("chegou no teste $int")
}

This code is identical to the previous example except for the use of supervisorScope. When you execute it, you will see the same output in the terminal: two "entrou" messages and the exception message. However, now you will also see the result from the teste(int: Int) method. This indicates that when using supervisorScope, even if one coroutine fails, the other continues to run. Changing just one function can significantly alter the behavior of your program.

Conclusion

As I mentioned at the beginning of this post, coroutines are an essential tool for Kotlin developers, but to use them effectively, it’s important to understand their nuances. Both coroutineScope and supervisorScope have their place in production code, and knowing which to use depends on your specific scenario. I hope you found this information helpful! If you'd like to discuss this topic further, feel free to reach out to me on social media.

Thank you!

References

https://kotlinlang.org/docs/coroutines-basics.html#your-first-coroutine
https://kotlinlang.org/api/kotlinx.coroutines/kotlinx-coroutines-core/kotlinx.coroutines/-coroutine-scope/

Understanding light-weight threads with Coroutines and Project Loom

João Victor Martins — Wed, 01 Nov 2023 13:14:05 +0000

Coroutines were added to Kotlin in version 1.3. Since then, people have talked a lot about light-weight threads. In September, Java 21 was released, and with it, was introduced the JEP 444 - Virtual Threads, as part of Project Loom. Both have the same goal, which is to reduce dramatically the effort of writing, maintaining, and observing high-throughput concurrent applications. There are many articles about those subjects, but sometimes, people get confused. How does it work? What is virtual thread? Coroutines and Project loom are the same? My intention with this post is to show you how light-weight threads work and some relations in both projects.

Virtual Threads

To understand how virtual threads work, we should understand the relation between OS Threads and JVM Threads. For each JVM Thread, we need an OS Thread. For instance, when our application needs to process something, it creates a JVM thread that will use an OS Thread. The problem is that sometimes these threads get in a iddle state. Suppose that our application receives a request to save a piece of information in the database. During the process of save, the thread will wait until the database returns. It does not look like a big problem, but according to our application grows, this become a problem. Threads are expensive and limited, so if the number of requests grows, it's necessary more threads. And what happens when the threads achieve the limit? It's necessary to scale the machine or increase the server instance to get more threads. Now can you understand why this is a problem? When the thread waits, we pay a lot just for it. How can we treat this? The correct answer is virtual threads. With virtual threads, we have many JVM threads to one OS thread. The good news is that the virtual thread doesn't need to wait until the finish of an external resource (DB). What occurs is that the virtual thread will be detached from an OS thread and will be in a suspended state. When the process finishes, another virtual thread can get the response. Amazing, right? Rashmin wrote in your article

Because of their lightweight nature, unlike Platform Threads, Virtual Threads are very cheap to create and can be created in very large numbers. So unlike Platform Threads, Thread Pooling is not necessary to handle multiple tasks, a separate virtual thread could be created for each task.

Now that we know how virtual threads work, let's see how we can implement them using Project Loom and Coroutines.

Coroutines

The standard library of Kotlin provides some resources to work with coroutines, but this support is designed in a minimalistic way. There are a few elements, like Continuation, suspendCoroutine, and the suspend keyword. To improve the development, it is a good idea to use kotlinx.coroutines library, which gives us a large number of elements like launch, async, and Deferred. The easiest way to work with coroutines is using runBlocking. Let's see an example

fun main() = runBlocking {
    delay(100L)
}

Inside of runBlocking, we can call suspend functions. But, be careful, according to the documentation

runBlocking runs a new coroutine and blocks the current thread until its completion. This function should not be used from a coroutine. It is designed to bridge regular blocking code to libraries that are written in suspending style, to be used in main functions and in tests.

The better way to work with coroutines is by creating a scope.

suspend fun main(): Person = coroutineScope {
     val user = async { getUserFromDB() }
     val address = async { getAddressFromAPI() }

     return Person(user.await(), address.await())
}

suspend fun getUserFromDB(): User {
     return repository.get()
}

The example above shows us that we just call a suspend function from another suspend function (or from a runBlocking function). From a suspend function I can call not suspend function, but in this way, we lose the power of coroutines. It's important to say that in our example, we needed to explicitly the corotineScope, but working with modern frameworks, like Spring, we can start from a Controller, for instance, with suspend functions, without declaring the scope.

Project Loom

Project Loom aims to bring "easy-to-use, high-throughput, lightweight concurrency" to the JRE. One feature introduced by Project Loom is virtual threads and since Java 21, thanks the JEP444, we can use this kind of code

public static void main(String... x) {
     try (var executor = Executors.newVirtualThreadPerTaskExecutor()) {
    IntStream.range(0, 10_000).forEach(i -> {
        executor.submit(() -> {
            Thread.sleep(Duration.ofSeconds(1));
            return i;
        });
    });
}

The task in this example is simple code — sleep for one second — and modern hardware can easily support 10,000 virtual threads running such code concurrently. Behind the scenes, the JDK runs the code on a small number of OS threads, perhaps as few as one.

Things would be very different if this program used an ExecutorService that creates a new platform thread for each task, such as Executors.newCachedThreadPool(). The ExecutorService would attempt to create 10,000 platform threads, and thus 10,000 OS threads, and the program might crash, depending on the machine and operating system.

Conclusion

Asynchronous programming is a great option to improve the performance of our application. Generally, people use platform threads to work with this paradigm, but as we saw previously, platform threads are very expensive. Thinking about improving that, every day new projects are emerging, like coroutines, project loom, goroutines, and so on. All of them use the concept of light-weight threads, a resource cheaper than traditional threads. In this post, we saw a way to implement this concept with coroutines and project loom. To finish, I hope that you have enjoyed the read, and any doubts, criticism, or points to consider, let me know.

References
https://openjdk.org/jeps/444
https://kotlinlang.org/docs/coroutines-basics.html#your-first-coroutine
https://rashm1n.medium.com/revolutionized-concurrency-introduction-to-java-21-virtual-threads-d563693994d9
https://kotlinlang.org/api/kotlinx.coroutines/kotlinx-coroutines-core/kotlinx.coroutines/run-blocking.html

Talking about Kotlin Coroutines

João Victor Martins — Tue, 14 Mar 2023 16:37:07 +0000

Nowadays, most web applications need to be asynchronous to attend your requests. There are many JVM libraries to work with this paradigm. The most famous is RxJava and Reactor. We can use both libraries with Kotlin, but there is an option that is better. This option is Kotlin Coroutines and we will understand how it works in this post.

Using Kotlin Coroutines

Kotlin Coroutines introduce the ability to suspend a coroutine at some point and resume it in the future. We might run our code on the Main thread and suspend it when we request data from API. When the coroutine is suspendend, the thread is not blocked and is free to go (Moskata, 2022, p. 15)

Suspension Work

Suspending functions are the essential feature of Kotlin Coroutines. When we suspend functions, it means that we can stop the process in the middle, do other things, and return to finish the previous process from the point where we stopped. How coroutines does do that? When one function is suspended, it returns an object called Continuation, which is the responsible for marking the point where the function stopped.

Built in support vc library

Coroutines consist of two components: Built-in support provided by the Kotlin Language and Kotlin coroutines library. Let's see the difference:

Built-in Support
Elements are in the kotlin.coroutines package.
Provides a few basic elements like Continuation or suspend keyword.

Coroutines Library
It is another dependency
Elements are in the kotlinx.coroutines package
Provides many elements like launch, async and so on (we will see about these elements).

Builders

One suspend function should call another suspend function that call another suspend function and so on. But when it starts? To move from normal world to coroutines world, we need a coroutines builder. We will know about 3 builders. Let's see.

launch builder: Similar to work with thread. We start a coroutine and it will run independently.

fun main() {
     GlobalScope.launch {
          delay(1000L)
          println("World")
     }
println("Hello,")
Thread.sleep(2000L)
}

Result of:
Hello,
(1 sec)
World

Without Thread.sleep this function would end immediately after launching the coroutines.

runBlocking builder: You should never block threads, just suspend them, but there are some cases that is necessary. Like the last example, we needed to block the thread, because our program could end early. In this cases, we can use runBlocking.

fun main() {
     runBlocking {
          delay(1000L)
          println("World")
     }
println("Hello,")
}

Result of:
(1 sec)
World
Hello,

async builder: The async builder is similar to launch, but there is a return.

fun main() {
     runBlocking {
          val result: Deferred<Int> = GlobalScope.async {
               delay(10000L)
               42   
          }
     val resultInt: Int = result.await()
     println(resultInt)
     }
}

Result of:
42

async is very similar to launch, so if you replace one to another, still work fine, but don't do that, because async is about producing a value. If you don't need a value, use launch.

Conclusion

As we can see, coroutines is a great option to use for asynchronous programing. My idea was to show some basic features for you start. There are many other features to cover and my intention is to write about in the next posts.

References
Moskata, M. Kotlin Coroutines Deep Dive, 2022

Using Circuit Breaker Pattern with Kotlin, Resilience4j and Retrofit

João Victor Martins — Thu, 08 Dec 2022 16:58:12 +0000

When we work with microservices, one of the main objectives is to achieve resilience. There are many ways to achieve that. Retry, bulkhead, circuit breaker, all of them improve the microservices resilience. My idea with this post is to write about circuit breaker and how it works. I will use resilience4j dependency to implement this mechanism.

Resilience4j

Resilience4j is a lightweight fault tolerance library designed for functional programming. Resilience4j provides higher-order functions (decorators) to enhance any functional interface, lambda expression or method reference with a Circuit Breaker, Rate Limiter, Retry or Bulkhead. You can stack more than one decorator on any functional interface, lambda expression or method reference. The advantage is that you have the choice to select the decorators you need and nothing else.

Circuit Breaker

The CircuitBreaker is implemented via a finite state machine with three normal states: CLOSED, OPEN and HALF_OPEN and two special states DISABLED and FORCED_OPEN.

The state of the CircuitBreaker changes from CLOSED to OPEN when the failure rate is equal or greater than a configurable threshold. For example when more than 50% of the recorded calls have failed.

The CircuitBreaker also changes from CLOSED to OPEN when the percentage of slow calls is equal or greater than a configurable threshold. For example when more than 50% of the recorded calls took longer than 5 seconds. This helps to reduce the load on an external system before it is actually unresponsive.

The failure rate and slow call rate can only be calculated, if a minimum number of calls were recorded. For example, if the minimum number of required calls is 10, then at least 10 calls must be recorded, before the failure rate can be calculated. If only 9 calls have been evaluated the CircuitBreaker will not trip open even if all 9 calls have failed.

The CircuitBreaker rejects calls with a CallNotPermittedException when it is OPEN. After a wait time duration has elapsed, the CircuitBreaker state changes from OPEN to HALF_OPEN and permits a configurable number of calls to see if the backend is still unavailable or has become available again.

If the failure rate or slow call rate is then equal or greater than the configured threshold, the state changes back to OPEN. If the failure rate and slow call rate is below the threshold, the state changes back to CLOSED.

More details in: https://resilience4j.readme.io/docs/circuitbreaker

Let's code

The first thing is to add the resilience4j-circuit-breaker on pom.xml

<!-- https://mvnrepository.com/artifact/io.github.resilience4j/resilience4j-circuitbreaker -->
        <dependency>
            <groupId>io.github.resilience4j</groupId>
            <artifactId>resilience4j-circuitbreaker</artifactId>
            <version>1.7.1</version>
        </dependency>

Now we need to configure our circuit breaker. My idea is that the circuit opens when catch 70% of errors or 70% of the requests last more than 2 nanoseconds (it's necessary to force the circuit open). My reference will be 10 requests. So, if in 10 requests, we receive 70% of errors or 70% of requests are slow, the state of circuit changes. To do that with code, let's create a Kotlin class called CircuitBreakerConfiguration.

// imports omitted 
@Configuration
class CircuitBreakerConfiguration {

    fun getConfiguration() = CircuitBreakerConfig.custom()
        .slidingWindowType(CircuitBreakerConfig.SlidingWindowType.COUNT_BASED)
        .slidingWindowSize(10)
        .slowCallRateThreshold(70.0f)
        .failureRateThreshold(70.0f)
        .waitDurationInOpenState(Duration.ofSeconds(5))
        .slowCallDurationThreshold(Duration.ofNanos(3))
        .permittedNumberOfCallsInHalfOpenState(3)
        .build()

    fun getCircuitBreaker() = CircuitBreakerRegistry.of(getConfiguration())
        .circuitBreaker("circuit-breaker-car-service")
}

As we can see, we called the custom method of CircuitBreakerConfig class and apply the configurations. We defined the type and the size of sliding window. The type was COUNT_BASED and the size was 10.

It's a good thing to understand about the sliding window. For that read the documentation.

To define the percentage of slow calls and errors that will open the circuit, we used slowCallRateThreshold and failureRateThreshold, both with 70%. There are two configurations that we need to understand better. They are waitDurationInOpenState and permmitedNumberOfCallsInHalfOpenState. The wait duration in open state is responsible to configure how many time the open circuit will pass from OPEN to HALF_OPEN. In our case, will be 5 seconds. When the circuit is in HALF_OPEN state, is permitted to request to the service (API, microservices etc...) to validate if there is an error or if the system recovered. To know how many requests is necessary to validate, we use permmitedNumberOfCallsInHalfOpenState. In our case is 3. If 70% of 3 requests fails or are more slow than 2 nanoseconds, the circuit will change FROM HALF_OPEN to OPEN and will wait more 5 seconds to change to HALF_OPEN again. The circuit won't permit requests to the service, returning CallNotPermittedException. But, if the requests don't fail, the circuit change from HALF_OPEN to CLOSED and application will work normally. Now we need to test.

Let's test

The application where we implemented the circuit breaker is simple. As we use Spring Boot, we have a RestController that calls a service. This service there are some methods and one of them calls an external service via HTTP. To do this request we use Retrofit, which a client HTTP that has good integration with resilience4j.

I won't give many details about Retrofit, for that, I recommend to read the documentation: https://square.github.io/retrofit/

The method that we need to enhance with circuit breaker is called listCarsByModel

override fun listCarsByModel(model: String) =
        carHttpService.getByModel(model)
            .execute()
            .body()
            ?.let(CarHttpToModelConverter::toModel)

There are some details about retrofit. CarHttpService is an interface that has a getByModel method. In execution time, when execute( ) method will call, retrofit will implement this interface. It is possible, because we have a configuration class fot that.

@Configuration
class CarHttpConfiguration(
    private val circuitBreakerConfiguration: CircuitBreakerConfiguration
) {

    private companion object {
        const val BASE_URL = "http://localhost:8081/cars"
    }

    private fun buildClient() = OkHttpClient.Builder().build()

    private fun buildRetrofit() = Retrofit.Builder()
        .baseUrl(BASE_URL)
        .addConverterFactory(GsonConverterFactory.create())
        .client(buildClient())
        .build()

    @Bean
    fun carHttpService(): CarHttpService = buildRetrofit().create(CarHttpService::class.java)
}

In this class we defined a buildRetrofit( ) method, that is responsible for add some configurations. We pass the url that will be call (you can change for other). We add a converter factory that will be the responsible to convert the API information to our object, and a client. After that, we defined a function called carHttpService, which is a bean. When execute( ) method is called, Retrofit uses this function to implement the interface and inject it into the service.

Until now, we have just spoken about retrofit. But what we need to know is how we can integrate resilience4j circuit breaker and retrofit.

We need to add a new dependency in our pom.xml

<!-- https://mvnrepository.com/artifact/io.github.resilience4j/resilience4j-retrofit -->
        <dependency>
            <groupId>io.github.resilience4j</groupId>
            <artifactId>resilience4j-retrofit</artifactId>
            <version>1.7.1</version>
        </dependency>

Now we just need to add one line in our retrofit configuration in builRetrofit method.

private fun buildRetrofit() = Retrofit.Builder()
//new line
        .addCallAdapterFactory(CircuitBreakerCallAdapter.of(circuitBreakerConfiguration.getCircuitBreaker()))
        .baseUrl(BASE_URL)
        .addConverterFactory(GsonConverterFactory.create())
        .client(buildClient())
        .build()

In this line, we applied an adapter, that receives a CircuitBreakerCallAdapter where we pass our circuit breaker configuration.

Without retrofit, we would have more manual work to do. This is the advantage of integrations.

The next step will be to start the application and call the endpoint that does the HTTP request more than 10 times. As we configured our slowCallDurationThreshold to 3 nanoseconds, the requests always last more time, so circuit should open. Once open, is possible to see this return

io.github.resilience4j.circuitbreaker.CallNotPermittedException: CircuitBreaker 'circuit-breaker-car-service' is OPEN and does not permit further calls

Conclusion

As we can see, circuit breaker improves the resilience of our services. Besides, it helps the external system to recover, when doesn't permit that our system does any request to it. I hope that you liked the post. Doubts, suggestions, and critics, you can comment or call me on my social media.

Testcontainers with MySQL and Redis with an Spring Boot Kotlin Application

João Victor Martins — Thu, 07 Jul 2022 13:01:32 +0000

Microservices architecture gives us some facilities, such as independently deploy, services with a unique responsibility, working with different languages, and so on. However, it also brings us some new complexities, such as duplicated code, observability tools, distributed caches tools, and logs tools. When we think about all these resources working together, we can think "How can we test these resources?". For instance, if our service uses Redis for cache, and MySQL to save and retrieve data, how can we do integration tests? Doing these tests by yourself can be difficult, but we have some manners to create that without much effort. One option is using TestContainers. To show you the tool, I will test a Spring Boot Kotlin Application as example.

TestContainers

Testcontainers is a Java library that supports JUnit tests, providing lightweight, throwaway instances of common databases, Selenium web browsers, or anything else that can run in a Docker container.

Testcontainers make the following kinds of tests easier:

Data access layer integration tests: use a containerized instance of a MySQL, PostgreSQL or Oracle database to test your data access layer code for complete compatibility, but without requiring complex setup on developers' machines and safe in the knowledge that your tests will always start with a known DB state. Any other database type that can be containerized can also be used.
Application integration tests: for running your application in a short-lived test mode with dependencies, such as databases, message queues or web servers.
UI/Acceptance tests: use containerized web browsers, compatible with Selenium, for conducting automated UI tests. Each test can get a fresh instance of the browser, with no browser state, plugin variations or automated browser upgrades to worry about. And you get a video recording of each test session, or just each session where tests failed.
Much more! Check out the various contributed modules or create your own custom container classes using GenericContainer as a base.

Let's Code

The application that will be used has a Car domain. We can perform operations of inserting new cars, searching for available cars, and changing existing cars. The system has a controller, a service, and a repository. An overview of the architecture follows.

So, when we do a POST request to an application, the data is saved on MySQL DB.

When we do a GET request, the application retrieves the data that is saved on DB and adds the information to Redis cache.

Redis is an open-source, in-memory data store used by millions of developers as a database, cache, streaming engine, and message broker. Redis' speed makes it ideal for caching database queries, complex computations, API calls, and session state.

Data in Redis

We need to guarantee that the cache was saved in Redis. For this, we can use a tool called RedisInsight to see the data. RedisInsight provides an intuitive Redis admin GUI and helps optimize your use of Redis in your applications. It supports Redis Open Source, Redis Stack, Redis Enterprise Software, Redis Enterprise Cloud, and Amazon ElastiCache. RedisInsight now incorporates a completely new tech stack based on the popular Electron and Elastic UI frameworks. And it runs cross-platform, supported on Linux, Windows, and MacOS.

As we can see, the cache's data is saved like a hash. And if we don't do POST and PUT operations to the application, this data always will be retrieved.

Testing the application

Now that we know how the application works, a controller test will be performed, where we will use MockMVC. MockMvc simulates a real request to the application, doing exactly what happens when the client calls an endpoint. This implies that at the time of running the tests, we need to have all external resources working. It is common for projects to use a real test-specific database instance to perform integration tests, however, as we saw earlier, we have a more elegant and modern way to do them, using TestContainers. As the cached service method and the repository will be called in the controller flow, it will be necessary to upload two containers, one for MySQL and the other for Redis. Let's start developing our test.

The first thing we need to do is add the TestContainers dependencies to the application's pom.xml.

<dependencyManagement>
  <dependencies>
    <dependency>
      <groupId>org.testcontainers</groupId>
      <artifactId>testcontainers-bom</artifactId>
      <version>${testcontainers.version}</version>
        <type>pom</type>
        <scope>import</scope>
    </dependency>
  </dependencies>
</dependencyManagement>

<dependencies>
  <dependency>
    <groupId>org.testcontainers</groupId>
    <artifactId>junit-jupiter</artifactId>
    <scope>test</scope>
  </dependency>
  <dependency>
    <groupId>org.testcontainers</groupId>
    <artifactId>mysql</artifactId>
    <scope>test</scope>
</dependency>

We need the TestContainers junit-jupiter dependency because it provides some resources for more efficient testing. As we are using MySQL as a database, it's necessary a dependency that provides resources to create a MySQL container for tests in a simplified way.

We need to create the configuration class of the containers that will be created at test time. It will be called DatabaseContainerConfiguration and will have the following content

@SpringBootTest(webEnvironment = SpringBootTest.WebEnvironment.RANDOM_PORT)
abstract class DatabaseContainerConfiguration {

    companion object {
        @Container
        private val mysqlContainer = MySQLContainer<Nothing>("mysql:latest").apply {
            withDatabaseName("testdb")
            withUsername("joao")
            withPassword("12345")
        }

        @Container
        private val redisContainer = GenericContainer<Nothing>("redis:latest").apply {
            withExposedPorts(6379)
        }

        @JvmStatic
        @DynamicPropertySource
        fun properties(registry: DynamicPropertyRegistry) {
            registry.add("spring.datasource.url", mysqlContainer::getJdbcUrl)
            registry.add("spring.datasource.password", mysqlContainer::getPassword)
            registry.add("spring.datasource.username", mysqlContainer::getUsername)

            registry.add("spring.redis.host", redisContainer::getContainerIpAddress)
            registry.add("spring.redis.port", redisContainer::getFirstMappedPort)
        }
    }
}

Now let's break down the meaning of each of the above features:

@Container: is used in conjunction with the @Testcontainers annotation to mark containers that should be managed by the Testcontainers extension.

We'll see about the @TestContainers annotation yet

MySQLContainer<Nothing>(...): Class that supports the creation of a MySQL container for testing. We passed a Nothing in the generics of MySQLContainer because of the class signature - public class MySQLContainer<SELF extends MySQLContainer<SELF>> extends JdbcDatabaseContainer<SELF> - as we can see, there is a recursive generic type, which with Java, we can ignore and just instantiate the class without generics. Using Kotlin, we don't have this option, but we have two ways to handle it.

Define a subclass of MySQLContainer and pass the child class to generics
Use Nothing, which by Kotlin doc's is:

Nothing has no instances. You can use Nothing to represent "a value that never exists": for example, if a function has the return type of Nothing, it means that it never returns.

In the MySQLContainer constructor, the version that we want to use from the MySQL image is informed.

withDatabaseName: Method of the MySQLContainer class that receives the name of the database as a parameter. This database will be created at runtime after the test container goes up.

withUsername: Method of the MySQLContainer class that receives the database username as a parameter.

withPassword: Method of the MySQLContainer class that receives the password from the database as a parameter.

GenericContainer<Nothing>(): As the name implies, it is a class that allows you to create a container in a generic way, which means that the methods contained in them can be reused for various resources, such as Redis, NoSQL Databases, among others. We are using it because Redis does not have a specific class like MySQL. As we are seeing in the code, to create a generic container, we just need to insert the image of the resource that we want to instantiate as a parameter.

withExposedPorts: Method of the GenericContainer class that receives the port that we will make available from the test container. This port will be used at runtime.

The properties function is used to dynamically add Redis and MySQL properties to the application. Before TestContainers uploads the two containers, the application will not be aware of them, so how would we carry out the integration tests? Using @DynamicPropertySource Kotlin understands that these properties will be dynamically set at runtime, so when uploading the MySQL and Redis container, the information that the application needs to know to reach them will be included.

Now we have everything we need for TestContainers to upload both containers. We just need to use the class where needed. As stated at the beginning of the section, we want to do an API test, where MockMVC will make a request to the controller and thus follow the flow to the database. So let's create this test.

@Testcontainers
@AutoConfigureMockMvc
@SpringBootTest(webEnvironment = SpringBootTest.WebEnvironment.RANDOM_PORT)
class CarControllerTest : DatabaseContainerConfiguration() {

    @Autowired
    private lateinit var mockMvc: MockMvc

    companion object {
        private const val URI = "/cars"
    }

    @Test
    fun `should return 200 code when call cars`() {
        mockMvc.get(URI).andExpect { status { isOk() } }
    }

As mentioned before, we can see that we have an annotation in the class declaration, which is @TestContainers. This annotation makes that, at runtime, the library identifies it and uploads the available containers through the @Container annotation, which is in our DatabaseContainerConfiguration class, which in turn was inherited by the test. This is enough for when we run the tests, the containers are instantiated.

The intention is not to talk about MockMVC, but according to the documentation, it is the main entry point for server-side Spring MVC testing support.

One last detail before running the tests is that it will be necessary to configure the migration. When the application goes up and reaches the containers, it is applied.

#src/main/resources/db/migration/V01__create_car_table_add_data.sql

create table car(
    id bigint not null auto_increment,
    name varchar(50) not null,
    model varchar(50) not null,
    primary key(id)
);

insert into car values('Golf', 'VW');

It's time to run the tests and remember, docker must be active. Running the mvn -B test command will run the tests. We can observe the containers going up at the time of the test execution, using the docker ps command in the terminal or the docker desktop.

In the test execution logs, we can see the moment when the MySQL container rises

22:17:53.206 [main] DEBUG 🐳 [mysql:latest] - Trying to create JDBC connection using com.mysql.cj.jdbc.Driver to jdbc:mysql://localhost:52550/testdb?useSSL=false&allowPublicKeyRetrieval=true with properties: {password=12345, user=joao}
22:17:53.582 [main] INFO 🐳 [mysql:latest] - Container is started (JDBC URL: jdbc:mysql://localhost:52550/testdb)
22:17:53.583 [main] INFO 🐳 [mysql:latest] - Container mysql:latest started in PT23.428294S

Just like Redis

22:17:53.583 [main] DEBUG 🐳 [redis:latest] - Starting container: redis:latest
22:17:53.583 [main] DEBUG 🐳 [redis:latest] - Trying to start container: redis:latest (attempt 1/1)
22:17:53.583 [main] DEBUG 🐳 [redis:latest] - Starting container: redis:latest
22:17:53.583 [main] INFO 🐳 [redis:latest] - Creating container for image: redis:latest

And finally the expected result

Tests Passed: 1 of 1 test - 631 ms

Conclusion

Integration tests are very important for the application, and one of the main tasks is to validate if the application's connection with external resources is being done successfully. There are N ways to do them. We can use database integration tests, in-memory databases, real instances and as presented in the post, TestContainers. TestContainers, with small settings, bring big advantages. Disposable containers, as they are only used at the time of test execution. External resource agnostic, with the GenericContainer class we can do integration testing with any resource we uploaded in docker. An excellent testing tool.

This post is in collaboration with Redis.

Application's repository: https://github.com/joao0212/car-service

References:
https://www.testcontainers.org/
https://docs.spring.io/spring-framework/docs/current/javadoc-api/org/springframework/test/web/servlet/MockMvc.html

[PT-BR] Testcontainers com MySQL e Redis

João Victor Martins — Thu, 16 Jun 2022 12:51:51 +0000

No post passado, falei sobre testes de unidade e sobre como eles auxiliam na validação das regras de negócio do sistema. Neste post minha ideia é falar sobre testes de integração, que de acordo com o blog da kenzie (2021), é quando os módulos (unidades de código) são testados em grupo para garantir que não haja nenhuma quebra naquilo que foi feito unitariamente e naquilo que está sendo integrado junto. Para exemplificá-lo, realizaremos um teste de integração em uma aplicação existente, onde utilizaremos a biblioteca TestContainers.

Testcontainers

De acordo com a documentação, Testcontainers é uma biblioteca Java que suporta testes JUnit, fornecendo instâncias leves e descartáveis de bancos de dados comuns, navegadores da Web Selenium ou qualquer outra coisa que possa ser executada em um contêiner do Docker.

Os contêineres de teste facilitam os seguintes tipos de testes:

Testes de integração da camada de acesso a dados: use uma instância conteinerizada de um banco de dados MySQL, PostgreSQL ou Oracle para testar seu código de camada de acesso a dados para compatibilidade completa.
Testes de UI/Aceitação: use navegadores da Web em contêiner, compatíveis com Selenium, para realizar testes automatizados de UI.
Muito mais: Confira os vários módulos contribuídos ou crie suas próprias classes de contêiner personalizadas usando GenericContainer como base.

Let's code

A aplicação que será utilizada, possui um domínio Car. Podemos realizar operações de inserção de novos carros, de busca de carros disponíveis e de alteração de carros existentes. O sistema possui uma controller, um service e um repository. Na camada de serviço, no método de listar carros, usamos o Redis para cache de informações e o banco de dados é um MySQL. Segue overview da arquitetura.

Será realizado um teste de controller, onde utilizaremos o MockMVC. O MockMvc simula uma requisição real à aplicação, fazendo exatamente o que ocorre quando o cliente chama um endpoint. Isso implica que no momento da execução dos testes, precisamos ter todos os recursos externos funcionando. É comum projetos utilizarem uma instância de banco de dados real, específica para testes, para realizar testes de integração, porém, como vimos anteriormente, temos uma maneira mais elegante e moderna para fazê-los, que é usando TestContainers. Como no fluxo da controller será chamado o método do service cacheado e o repositório, será preciso subir dois contêineres, sendo um pro MySQL e o outro para o Redis. Vamos iniciar o desenvolvimento do nosso teste.

A primeira ação que precisamos fazer é adicionar as dependências do TestContainers no pom.xml da aplicação.

# Resto do pom.xml omitido

<dependencyManagement>
  <dependencies>
    <dependency>
      <groupId>org.testcontainers</groupId>
      <artifactId>testcontainers-bom</artifactId>
      <version>${testcontainers.version}</version>
        <type>pom</type>
        <scope>import</scope>
    </dependency>
  </dependencies>
</dependencyManagement>

<dependencies>
  <dependency>
    <groupId>org.testcontainers</groupId>
    <artifactId>junit-jupiter</artifactId>
    <scope>test</scope>
  </dependency>
  <dependency>
    <groupId>org.testcontainers</groupId>
    <artifactId>mysql</artifactId>
    <scope>test</scope>
</dependency>

Precisamos da dependência do junit-jupiter do TestContainers, pois ela nos fornece alguns recursos para maior eficiência dos testes e por estarmos usando o MySQL como banco de dados, precisamos de uma dependência que nos forneça recursos para criar um container MySQL para testes de forma simplificada.

Precisamos criar a classe configuração dos contêineres que serão criados no momento do teste. Ela será chamada de DatabaseContainerConfiguration e terá o seguinte conteúdo

// Imports omitidos

@SpringBootTest(webEnvironment = SpringBootTest.WebEnvironment.RANDOM_PORT)
abstract class DatabaseContainerConfiguration {

    companion object {
        @Container
        private val mysqlContainer = MySQLContainer<Nothing>("mysql:latest").apply {
            withDatabaseName("testdb")
            withUsername("joao")
            withPassword("12345")
        }

        @Container
        private val redisContainer = GenericContainer<Nothing>("redis:latest").apply {
            withExposedPorts(6379)
        }

        @JvmStatic
        @DynamicPropertySource
        fun properties(registry: DynamicPropertyRegistry) {
            registry.add("spring.datasource.url", mysqlContainer::getJdbcUrl)
            registry.add("spring.datasource.password", mysqlContainer::getPassword)
            registry.add("spring.datasource.username", mysqlContainer::getUsername)

            registry.add("spring.redis.host", redisContainer::getContainerIpAddress)
            registry.add("spring.redis.port", redisContainer::getFirstMappedPort)
        }
    }
}

Agora vamos detalhar o significado de cada um dos recursos acima:

@Container: é usada em conjunto com a anotação @Testcontainers para marcar contêineres que devem ser gerenciados pela extensão Testcontainers.

Ainda vamos ver sobre a anotação @TestContainers

MySQLContainer<Nothing>(...): Classe que da suporte a criação de um contêiner MySQL para teste. Passamos um Nothing no generics de MySQLContainer e isso ocorre por causa da assinatura da classe. public class MySQLContainer<SELF extends MySQLContainer<SELF>> extends JdbcDatabaseContainer<SELF>, como podemos ver, há um tipo genérico recursivo, que com o Java, conseguimos ignorar e apenas instanciar a classe sem o generics. Usando Kotlin, não temos essa opção, mas temos duas formas de tratar.
- Definir uma subclasse de MySQLContainer e passar a classe filha para o generics
- Usar o Nothing, que pelo Kotlin doc's é:

Nothing has no instances. You can use Nothing to represent "a value that never exists": for example, if a function has the return type of Nothing, it means that it never returns*.

Como podemos ver no código, estamos usando a segunda opção. No construtor da classe, é informada a versão que queremos usar da imagem do MySQL.

withDatabaseName: Método da classe MySQLContainer que recebe o nome da database como parâmetro. Essa database será criada em tempo de execução, após o contêiner de teste subir.
withUsername: Método da classe MySQLContainer que recebe o username da database como parâmetro.
withPassword: Método da classe MySQLContainer que recebe o password da database como parâmetro.
GenericContainer<Nothing>(): Como o próprio nome já diz, é uma classe que permite criar um contêiner de forma genérica, isso significa que os métodos contidos neles, poderão ser reutilizados para vários recursos, como Redis, Bancos NoSQL, entre outros. Estamos o utilizando, pois o Redis não possui uma classe específica como o MySQL. Como estamos vendo no código, para criar um contêiner genérico, basta inserirmos como parâmetro a imagem do recurso que queremos instanciar.
withExposedPorts: Método da classe GenericContainer que recebe a porta que vamos disponibilizar do contêiner de teste. Essa porta será utilizada em tempo de execução.

A função properties serve para adicionar as propriedades do Redis e do MySQL de forma dinâmica à aplicação. Ou seja, antes do TestContainers subir os dois contêineres, a aplicação não vai ter conhecimento deles, logo como realizaríamos os testes de integração? Usando o @DynamicPropertySource o Kotlin entende que essas propriedades serão setadas de forma dinâmica, em tempo de execução, assim, ao subir o contêiner do MySQL e do Redis, as informações que a aplicação precisa conhecer para alcançá-los serão incluídas.

Agora temos tudo que precisamos para que o TestContainers suba os dois contêineres. Precisamos apenas utilizar a classe onde necessário. Como informado no começo da seção, queremos fazer um teste de API, onde o MockMVC fará uma requisição para a controller e assim seguirá o fluxo até o banco de dados. Então vamos criar esse teste.

@Testcontainers
@SpringBootTest(webEnvironment = SpringBootTest.WebEnvironment.RANDOM_PORT)
class CarControllerTest : DatabaseContainerConfiguration() {

    @Autowired
    private lateinit var webApplicationContext: WebApplicationContext

    private lateinit var mockMvc: MockMvc

    companion object {
        private const val URI = "/cars"
    }

    @BeforeEach
    fun setup() {
        mockMvc = MockMvcBuilders.webAppContextSetup(webApplicationContext)
            .apply<DefaultMockMvcBuilder?>(
                SecurityMockMvcConfigurers
                .springSecurity()).build()
    }

    @Test
    fun `should return code 200 when call cars`() {
        mockMvc.get(URI).andExpect { status { isOk() } }
    }

Como falado anteriormente, podemos perceber que temos uma anotação na declaração da classe, que é a @TestContainers. Essa anotação faz com que em tempo de execução, a biblioteca a identifique e suba os contêineres disponíveis pela a anotação @Container, que está na nossa classe DatabaseContainerConfiguration, que por sua vez foi herdada pelo teste. Isto é o suficiente para quando executarmos os testes, os contêineres sejam instanciados.

A intenção não é falar sobre o MockMVC, mas de acordo com a documentação, ele é o ponto de entrada principal para suporte de teste Spring MVC do lado do servidor.

Um último detalhe antes de executar os testes, é que será necessário configurar as migrations, para que no momento que a aplicação subir e alcançar os contêineres, as mesmas sejam aplicadas.

# src/main/resources/db/migration/V01__create_car_table_add_data.sql

create table car(
    id bigint not null auto_increment,
    name varchar(50) not null,
    model varchar(50) not null,
    primary key(id)
);

insert into car values('Golf', 'VW');

Chegou o momento de executar os testes e para isso o docker deve estar ativo. Executando o comando mvn -B test, os testes serão executados. Podemos observar os contêineres subindo no momento da execução dos testes, utilizando o comando docker ps no terminal ou o docker desktop.

Nos logs da execução dos testes, podemos ver o momento em que sobe o contêiner do MySQL

Assim como o do Redis

E finalmente o resultado esperado

Conclusão

Testes de integração possuem muita importância para aplicação, sendo uma das principais atribuições, validar se a conexão da aplicação com recursos externos estão sendo feitas de maneira bem sucedida. Existem N maneiras de fazê-los, podemos usar, por exemplo, para testes de integração com banco de dados, bancos de dados em memória, instâncias reais e como apresentado no post, TestContainers. O TestContainers, com pequenas configurações, traz grandes vantagens. Contêineres descartáveis, pois só são usados no momento da execução dos testes. Agnóstico ao recurso externo, com a classe GenericContainer, podemos fazer o teste de integração com qualquer recurso que subimos no docker. Uma excelente ferramenta para testes.

Se você sentiu que faltou alguma coisa importante para explicar, ficou com alguma dúvida ou tem alguma sugestão, não deixe de me procurar. No mais, espero que você tenha uma boa leitura. Até a próxima.

Referências
https://kenzie.com.br/blog/teste-de-integracao/
https://www.testcontainers.org/
https://docs.spring.io/spring-framework/docs/current/javadoc-api/org/springframework/test/web/servlet/MockMvc.html

[PR-BR] Testes de unidade com MockK e AssertJ

João Victor Martins — Thu, 20 Jan 2022 14:15:58 +0000

Sempre que estamos falando sobre boas práticas de desenvolvimento de software, um nome sempre aparece: testes de unidade. De acordo com Marco Tulio Valente (2020), testes de unidade são testes automatizados de pequenas unidades de código, normalmente classes, as quais são testadas de forma isolada do restante do sistema. Um teste de unidade é um programa que chama métodos de uma classe e verifica se eles retornam os resultados esperados. Esses testes podem ser feitos na maioria das linguagens de programação e cada uma delas possui bibliotecas e recursos para facilitar o desenvolvimento dos mesmos. A ideia do post é mostrar como fazer testes de unidade em kotlin usando mockK e AssertJ.

MockK

Mockk é uma biblioteca de mocking para kotlin que traz uma série de recursos como Spy, Relaxed mock, Partial mocking, Object mocks, Class mock entre outros.

AssertJ

O AssertJ fornece um rico conjunto de asserções, mensagens de erro realmente úteis, melhora a legibilidade do código de teste e foi projetado para ser super fácil de usar em sua IDE favorita.

Dependências

Para criar testes de unidade com as bibliotecas informadas, é necessário adicionar as dependências no projeto. No caso do maven basta colocar o trecho a seguir no pom.xml (verificar versões)

        <dependency>
            <groupId>io.mockk</groupId>
            <artifactId>mockk</artifactId>
            <version>1.12.1</version>
            <scope>test</scope>
        </dependency>

        <dependency>
            <groupId>org.assertj</groupId>
            <artifactId>assertj-core</artifactId>
            <version>3.21.0</version>
            <scope>test</scope>
        </dependency>

Mãos na massa

Suponha que é necessário testar o comportamento de um método que lista produtos. O método possui dois fluxos. Quando é informado o nome do produto a ser listado, deve-se retornar apenas o produto especificado. Quando o nome do produto não é informado, deve-se retornar todos os produtos.

// Resto do código omitido

fun listar(nomeProduto: String?): List<Produto> {
        return nomeProduto?.let {
            repository.findByProdutoNome(nomeProduto)
        } ?: repository.findAll()
    }

O primeiro teste será para validar o comportamento do método quando é informado o nome do produto.

import io.mockk.every
import io.mockk.mockk
import io.mockk.verify
import org.assertj.core.api.Assertions.*
// Outros imports foram omitidos

class ProdutoServiceTest {

        private val repository: ProdutoRepository = mockk()
        private val produtoService = ProdutoService(repository)

        @Test
        fun `deve listar produto por nome`() {
                every { repository.findByProdutoNome(any()) } returns listOf(Produto("Geladeira"))

                val result = produtoService.listar(any())

                verify(exactly = 0) { produtoRepository.findAll() }
                verify(exactly = 1) { produtoRepository.findByProdutoNome(any()) }

                assertThat(resultado).isNotNull
                assertThat(resultado).hasSize(1)
        }
}

Vamos entender alguns recursos apresentados. Como a ideia do teste de unidade é testar apenas o comportamento do método que contém a regra de negócio, toda a parte de repositório deverá ser mockada. Para mockar objetos com o mockK pode-se utilizar a inline function mockk(). Há duas formas de utilizá-la, da forma que foi declarada acima ou com mockk<ProdutoRepository>(). Com o repositório mockado, é necessário definir o seu comportamento. Para isto é criado o bloco every que descreve qual resposta deve ser dada quando um método do repositório for chamado. No caso acima, sempre que o método findByProdutoNome for chamado, será retornado uma lista de produtos que possui apenas um produto. O verify será o bloco responsável por verificar se o mock foi invocado como esperado. Como o método possui dois fluxos, temos que garantir que quando for informado o nome do produto, o método do repositório findAll() não seja chamado e sim o findByProdutoNome(). Qualquer comportamento diferente deste deverá fazer o teste falhar. Tudo que foi falado até aqui são recursos do mockk e já seria suficiente para fazer o teste ser executado com sucesso. Porém para garantir que o resultado retornado está correto, podemos fazer asserções e é neste momento que o AssertJ vira protagonista. Para escrever uma asserção, você sempre precisa começar passando seu objeto para o método assertThat() e então seguir com as asserções reais. No caso acima, estamos afirmando que o resultado não é nulo e a lista possui um elemento. Qualquer retorno diferente disso, significaria que o método listar de ProdutoService não está de acordo com o comportamento esperado, e mais uma vez o teste deverá falhar. Agora que entendemos todos os recursos utilizados no teste, podemos testar o outro fluxo, quando não é informado o nome do produto.

import io.mockk.every
import io.mockk.mockk
import io.mockk.verify
import org.assertj.core.api.Assertions.*
// Outros imports foram omitidos

class ProdutoServiceTest {

        private val repository: ProdutoRepository = mockk()
        private val produtoService = ProdutoService(repository)

        @Test
        fun `deve listar todos os produtos`() {
                every { repository.findAll() } returns listOf(Produto("Geladeira"), Produto("Televisao"))

                val result = produtoService.listar()

                verify(exactly = 1) { produtoRepository.findAll() }
                verify(exactly = 0) { produtoRepository.findByProdutoNome(any()) }

                assertThat(resultado).isNotNull
                assertThat(resultado).hasSize(2)
        }

Conclusão

Podemos ver que ambas as bibliotecas nos fornecem recursos poderosos para criar testes de unidades simples, elegantes e eficientes. Existem muitos outros recursos a serem explorados, o que não caberia em apenas um post. Para quem tiver interesse em continuar os estudos, abaixo, em referências, deixei alguns links que vão auxiliar. Espero que tenham gostado da leitura e qualquer dúvida, crítica ou sugestão, não deixem de comentar. Até a próxima!!

Referências
https://mockk.io/
https://joel-costigliola.github.io/assertj/
https://www.baeldung.com/kotlin/mockk
https://www.baeldung.com/introduction-to-assertj

Bibliografia
Marco Tulio Valente. Engenharia de Software Moderna: Princípios e Práticas para Desenvolvimento de Software com Produtividade, 2020.

[PT-BR] Minhas duas features preferidas do Java 17

João Victor Martins — Tue, 28 Sep 2021 13:56:06 +0000

No dia 14/09/2021, foi lançada a versão 17 do Java, sendo ela uma versão LTS (Long Term Support). Assim como nas últimas versões, a 17 trouxe algumas features bem divertidas. Minha ideia nesse post é mostrar as duas que mais me chamaram atenção, que são a 409:Sealed Classes e a 406:Pattern Matching for Switch.

Sealed Classes

Sealed classes e interfaces restringem quais outras classes ou interfaces devem estendê-las e/ou implementa-las. O objetivo é permitir que o autor da classe ou interface tenha mais controle sobre a mesma, além de prover um jeito mais declarativo de restringir o uso da super classe. Um outro objetivo é o suporte a pattern matchings.

public abstract sealed class Automovel 
    permits com.exemplo.automovel.Carro,
            com.examplo.automovel.Moto { ... }

Uma vez que eu defino quais classes irão poder estender de Automovel, qualquer outra classe diferente de Carro e Moto não deve ser permitida. É importante ressaltar, que as classes que irão estender uma sealed class, deverão estar no mesmo módulo, caso esteja usando JPMS (Java Platform Module System) ou no mesmo pacote, caso não esteja usando JPMS. Para a classes que estendem, existem algumas regras. São elas:

Uma subclasse deve ser declarada como final para manter a hierarquia.
Uma subclasse deve ser declarada como sealed e permitir a extensão de um novo conjunto de classes.
Uma subclasse deve ser declarada como non-sealed e assim permitir que qualquer classe a estenda.

Pattern Matching for Switch

Estender a correspondência de padrões para switch permite que uma expressão seja testada em relação a vários padrões, cada um com uma ação específica, de forma que consultas complexas orientadas a dados possam ser expressas de forma concisa e segura. O objetivo é melhorar a expressividade e a aplicabilidade das extensões e instruções switch, além de certificar que todas as expressões e instruções existentes continuem a compilar sem alterações.

static void printarTipoAutomovel(Automovel automovel) {
   switch(automovel) {
     case Carro -> System.out.println("É um carro");
     case Moto -> System.out.println("É uma moto");
     default -> System.out.println("Automóvel não identificado");
   }
}

É importante ressaltar que a JEP 406: Pattern Matching for switch está como preview, o que significa que ela pode ser retirada e não fazer parte das próximas versões do Java.

Concluindo

O objetivo do post era trazer as features que mais me interessaram no Java 17, sendo essa uma escolha totalmente pessoal. Existem muitas outras features legais, que podem ser vistas em: https://openjdk.java.net/projects/jdk/17/. Espero que gostem da leitura. Dúvidas, sugestões e/ou críticas, estou à disposição.

Referências
https://openjdk.java.net/jeps/409
https://openjdk.java.net/jeps/406

Partition and Clustering Key on Scylla DB

João Victor Martins — Mon, 26 Jul 2021 16:33:57 +0000

In the last 2 previous posts, we talked about Scylla DB. Today I will keep talking about the database, more specifically how works Primary Key on Scylla. We will see that the PK can be divided into two: Partition Key and Clustering Key. Let's get the difference between both and some differences in relation to a relational database.

Partition Key

Partition Key is responsible for data distribution across the nodes. It determines which node will store a given row. It can be one or more columns.

Clustering Key

Clustering Key is responsible for sorting the rows within the partition. It can be zero or more columns.

The problem of Car Table

Now that we know the difference about the Composites Keys, let's remember how we modeling our PK in Car table. In the cqlsh after selecting the Keyspace, use the command below.

DESCRIBE car

The result will be.

CREATE TABLE automobilies.car (
    id uuid PRIMARY KEY,
    brand text,
    color text,
    model text
) WITH bloom_filter_fp_chance = 0.01
    AND caching = {'keys': 'ALL', 'rows_per_partition': 'ALL'}
    AND comment = ''
    AND compaction = {'class': 'SizeTieredCompactionStrategy'}
    AND compression = {'sstable_compression': 'org.apache.cassandra.io.compress.LZ4Compressor'}
    AND crc_check_chance = 1.0
    AND dclocal_read_repair_chance = 0.1
    AND default_time_to_live = 0
    AND gc_grace_seconds = 864000
    AND max_index_interval = 2048
    AND memtable_flush_period_in_ms = 0
    AND min_index_interval = 128
    AND read_repair_chance = 0.0
    AND speculative_retry = '99.0PERCENTILE';

How we can see, the unique PK that we have is id and could be calling of Simple Primary Key. If we do a select in the database, will retrieve only one car

cqlsh:automobilies> select * from car;

 id                                   | brand | color | model
--------------------------------------+-------+-------+-------
 f714e8e5-b160-4341-807d-f4cd92b973a4 |    VW |   Red |  Golf

To improve our test, let's include more cars in the table. The same model of car will have the same id.

cqlsh:automobilies> insert into car (id, brand, color, model) values (e0625c94-e9c2-11eb-9a03-0242ac130003, 'Ford', 'Red', 'Focus');
cqlsh:automobilies> insert into car (id, brand, color, model) values (fbd04f2c-511a-43c5-b588-9e29ebcb5d7a, 'VW', 'Nardo Grey', 'Passat');

And now the select clause result in

 id                                   | brand | color      | model
--------------------------------------+-------+------------+--------
 fbd04f2c-511a-43c5-b588-9e29ebcb5d7a |    VW | Nardo Grey | Passat
 e0625c94-e9c2-11eb-9a03-0242ac130003 |  Ford |        Red |  Focus
 f714e8e5-b160-4341-807d-f4cd92b973a4 |    VW |        Red |   Golf

If we want to find just a Golf car, we can query by id

cqlsh:automobilies> select * from car where id = f714e8e5-b160-4341-807d-f4cd92b973a4;

 id                                   | brand | color | model
--------------------------------------+-------+-------+-------
 f714e8e5-b160-4341-807d-f4cd92b973a4 |    VW |   Red |  Golf

(1 rows)

But what happening if we want to query by id and color? Well, maybe just need to add the and in the query and choose color, right?

cqlsh:automobilies> select * from car where id = f714e8e5-b160-4341-807d-f4cd92b973a4 and color = 'Red';

InvalidRequest: Error from server: code=2200 [Invalid query] message="Cannot execute this query as it might involve data filtering and thus may have unpredictable performance. If you want to execute this query despite the performance unpredictability, use ALLOW FILTERING"

Like we can see, when add color, we get an error, but why?
To answer this question, we need to understand how Scylla saves data (I will use an example with another domain).

In the image above, we see that Scylla uses a PK to divide the partitions. When we use just id for query, Scylla knows exactly which node contains the data by hashing the ID (which is the partition key) and that is why we got results in the first query. In the second query, when we tried to filter for color, wasn't possible, because we not defined color as Clustering Key or as Partition Key. We would still know on which partition the data exists because we have the partition key, but we would have to do a linear scan of the entire data on that partition to find the specific row we are looking for (using ALLOW FILTERING), but performing this query would be inefficient and possibly very slow. To use ALLOW FILTERING is like this

cqlsh:automobilies> select * from car where id = f714e8e5-b160-4341-807d-f4cd92b973a4 and color = 'Red' ALLOW FILTERING;

 id                                   | brand | color | model
--------------------------------------+-------+-------+-------
 f714e8e5-b160-4341-807d-f4cd92b973a4 |    VW |   Red |  Golf

(1 rows)


cqlsh:automobilies> select * from car where id = f714e8e5-b160-4341-807d-f4cd92b973a4 and color = 'Green' ALLOW FILTERING;

---MORE---
(0 rows)

How we saw, this it's not good, so let's improve our data modeling.

Remodeling Car Table

We should know for which attributes we will want to query to define our keys. How we want a query for color and id, both will be our Pk, being id a Partition Key and color a Clustering Key.

cqlsh:automobilies> CREATE TABLE car (id uuid, brand text, color text, model text, PRIMARY KEY (id, color));

cqlsh:automobilies> DESCRIBE car;

CREATE TABLE automobilies.car (
    id uuid,
    color text,
    brand text,
    model text,
    PRIMARY KEY (id, color)
) WITH CLUSTERING ORDER BY (color ASC)
    AND bloom_filter_fp_chance = 0.01
    AND caching = {'keys': 'ALL', 'rows_per_partition': 'ALL'}
    AND comment = ''
    AND compaction = {'class': 'SizeTieredCompactionStrategy'}
    AND compression = {'sstable_compression': 'org.apache.cassandra.io.compress.LZ4Compressor'}
    AND crc_check_chance = 1.0
    AND dclocal_read_repair_chance = 0.1
    AND default_time_to_live = 0
    AND gc_grace_seconds = 864000
    AND max_index_interval = 2048
    AND memtable_flush_period_in_ms = 0
    AND min_index_interval = 128
    AND read_repair_chance = 0.0
    AND speculative_retry = '99.0PERCENTILE';

Now that we have a Clustering Key we can insert some cars with the same id, but different colors and Scylla will sort for us.

cqlsh:automobilies> insert into car (id, brand, color, model) values (f714e8e5-b160-4341-807d-f4cd92b973a4, 'VW', 'Red', 'Golf');
cqlsh:automobilies> insert into car (id, brand, color, model) values (e0625c94-e9c2-11eb-9a03-0242ac130003, 'Ford', 'Red', 'Focus');
cqlsh:automobilies> insert into car (id, brand, color, model) values (fbd04f2c-511a-43c5-b588-9e29ebcb5d7a, 'VW', 'Red', 'Passat');
cqlsh:automobilies> insert into car (id, brand, color, model) values (fbd04f2c-511a-43c5-b588-9e29ebcb5d7a, 'VW', 'Green', 'Passat');
cqlsh:automobilies> insert into car (id, brand, color, model) values (f714e8e5-b160-4341-807d-f4cd92b973a4, 'VW', 'Green', 'Golf');

Our partition will stay like this

Thus we have the partitions divided for id and rows of cars sorted by color

cqlsh:automobilies> select * from car;

 id                                   | color | brand | model
--------------------------------------+-------+-------+--------
 fbd04f2c-511a-43c5-b588-9e29ebcb5d7a | Green |    VW | Passat
 fbd04f2c-511a-43c5-b588-9e29ebcb5d7a |   Red |    VW | Passat
 e0625c94-e9c2-11eb-9a03-0242ac130003 |   Red |  Ford |  Focus
 f714e8e5-b160-4341-807d-f4cd92b973a4 | Green |    VW |   Golf
 f714e8e5-b160-4341-807d-f4cd92b973a4 |   Red |    VW |   Golf

And now we can query by id and color

cqlsh:automobilies> select * from car where id = fbd04f2c-511a-43c5-b588-9e29ebcb5d7a and color = 'Red';

 id                                   | color | brand | model
--------------------------------------+-------+-------+--------
 fbd04f2c-511a-43c5-b588-9e29ebcb5d7a |   Red |    VW | Passat

(1 rows)

Conclusion

In general, we saw how we define our PK in Scylla, can being Partition Key and Clustering Key. We need first think about our queries and after to model our tables. There are many others rules for work with these PK's, but in a unique post is not possible to cover all subject matter. I recommend that you do the courses of https://university.scylladb.com/. Well, this is what I wanted to introduce in this post, I really hope that you like it. Doubts, critics, and suggestions, I will be a disposition.

Integrating Scylla DB with Quarkus

João Victor Martins — Wed, 26 May 2021 18:18:09 +0000

Hello, everyone!! In the previous post, we talk about Scylla DB and some characteristics of the database. For those who have not seen it yet, this is the link https://dev.to/j_a_o_v_c_t_r/working-with-scylla-database-3al9. Today I intend to integrate the database with an application Java and to help us I decided to use Java + Quarkus. Quarkus is a Kubernetes Native Java stack tailored for OpenJDK HotSpot and GraalVM, crafted from the best-of-breed Java libraries and standards. It is known as Supersonic Subatomic java because it is fast and small. To those interested, the link will be at the final of the post. Ok, now that we know our goal, let's start!!

Creating the application

The first step is to create the application and for that, Quarkus gives us a simple interface that we can access at this link https://code.quarkus.io/.

On this page we put information like groupId, artifactId, build tools, and select which dependencies we will use.

Let's know about the selected dependencies

Datastack Apache Cassandra client: Driver to connect to Apache Cassandra databases. How we saw, Scylla DB is based on Cassandra, so we can use the same client.
RESTEasy JSON-B: JSON-B serialization support for RESTEasy. We will use endpoints to interact with our application
RESTEasy Mutiny: Mutiny support for RESTEasy server. If we want to call async resources (We don't use them in this example).
SmallRye Health: Monitor service health.
Micrometer Registry Prometheus: Enable Prometheus support for micrometer, so with this dependency, we can collect metrics for our application.

Ok, now we can generate our application in the Generate your application button.

Let's import the application in our IDE.

Quarkus gives us a GET resource for testing, but we are not going to use it so we can delete it.

If you want test, is enough execute mvn package quarkus:dev command, wait application is up and call http://localhost:8080/hello-reasteasy

Now we can create our entity. Entity class is a model for objects that we will work on. This object will be Car. In the name of the class will use the annotation @Entity. The name of the class should be the same as the table of database. The object will have the fields id, brand, color, and model. How we saw in the previous post, the id field is a primary key, so we will use the annotation @PartitionKey.

In the next post, I will talk about Scylla Primary Key and the differences in relation the relational databases

How we can see, the class does not compile, because we need to add some dependencies in our pom.xml. This dependencies are:

com.datastax.oss:java-driver-core: A modern, feature-rich, and highly tunable Java client library for Apache Cassandra® (2.1+) and DataStax Enterprise (4.7+), and DataStax Astra, using exclusively Cassandra's binary protocol and Cassandra Query Language (CQL) v3.
com.datastax.oss:java-driver-mapper-runtime: The mapper generates the boilerplate to execute queries and convert the results into application-level objects.

The dependencies can be downloaded in https://mvnrepository.com/

And we need to change the original maven-compiler-plugin in pom.xml for the below.

<plugin>
                <artifactId>maven-compiler-plugin</artifactId>
                <version>3.8.0</version>
                <configuration>
                    <source>${java.version}</source>
                    <target>${java.version}</target>
                    <annotationProcessorPaths>
                        <path>
                            <groupId>com.datastax.oss.quarkus</groupId>
                            <artifactId>cassandra-quarkus-mapper-processor</artifactId>
                            <version>1.0.1</version>
                        </path>
                    </annotationProcessorPaths>
                </configuration>
            </plugin>

This is necessary because Quarkus need know the annotations in compile time for injecting the resources like DAO and others. We'll see that soon.

After including the dependencies in pom.xml, we can import annotations and include constructors and getters/setters.

rest of the methods omitted!

The next step is to create the Data Access Object or more commonly DAO. Is enough for this post two methods, one select method and one insert method.

The annotation @Dao comes from the DataStax Object Mapper, which will also automatically generate an implementation of this interface for us. @Insert and @Select generate an implementation of methods. Note also the special return type of the findAll method, PagingIterable: it’s the base type of result sets returned by the driver.

Now let's create a mapper that is responsible for constructing instances of DAOs. In this case, will create an instance of CarDAO. The name will be CarMapper.

To interact with the saved information or save new information in the database, we will create a REST endpoint and for that will create a CarResource class. For our example, we will need a POST and a GET method.

In relation to the image above, let's talk about some details. The resource for access the application will be cars. We should send JSON for the application and consume JSON of the application. To save or retrieve information, we need access to the database, so for that, we inject the CarDAO in CarResource.

The last step will be to configure the information of the database in the application. For that, we will use application.properties.

And we can see the information above when we up our node Scylla.

Let's start the application using mvn package quarkus:dev command

And to use our browser to do a request for the application

Saving information

New saved information

Tested requests, our application is complete and functional. We can now explore more features.

The reason we put in our pom.xml the quarkus-smallrye-health dependency is automatically add a readiness health check to validate the connection to the Scylla cluster. We can access http://localhost:8080/q/health/ready to know if our connection is healthy or not.

Now I will pause the docker that is running ScyllaDB

And test connection again

We can see the metrics of Scylla session and about individual Scylla nodes and for this is enough add a property quarkus.cassandra.metrics.enabled=true in our application.properties what we already did. To see metrics we can access http://localhost:8080/metrics

Conclusion

My intention with this post is to show how we can integrate an application Java with Scylla DB and how easy that is using Quarkus. Some details can explorer yet, like primary key, but we can see this in the next post. I hope that you liked it and any doubt, critics, or suggestions tell me. Will be a pleasure to help you.

References
https://quarkus.io/
https://quarkus.io/guides/cassandra

Working with Scylla Database

João Victor Martins — Sat, 24 Apr 2021 13:01:41 +0000

Hello everyone! My intention today is to write about Scylla database. Scylla is a NoSQL database based on Cassandra. One difference between both is that Cassandra is developed in Java language and Scylla in C++ language. The goal of that is to avoid some Java processes that can impact the performance of the application, like garbage collector and stop the world. After this introduction, let's see how we can use Scylla and know some details about it.

Downloading image

For work with Scylla, we will use docker. The first step is to download the image. We can use the command below.

docker run --name scylla-db-test -d scylladb/scylla:4.1.0

After that, we can use some tools of Scylla and guarantee that our database is healthy.

docker exec -it scylla-db-test nodetool status

The result of executing the command above is

Datacenter: datacenter1
=======================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address     Load       Tokens       Owns    Host ID                               Rack
UN  172.17.0.2  866.47 KB  256          ?       0b8a4db2-b8ea-49cf-b618-4eaf780ba7c7  rack1

There is information about Datacenter, but this is not our focus in this post. At this moment, we should just know that our node (more information about that in the next section) is Up and Normal (UN).

Playtime

For executing commands in Scylla, there is a CLI called cqlsh.

docker exec -it scylla-db-test cqlsh

Connected to at 172.17.0.2:9042.
[cqlsh 5.0.1 | Cassandra 3.0.8 | CQL spec 3.3.1 | Native protocol v4]
Use HELP for help.
cqlsh>

The first thing that we can do is create a keyspace. Different from relational databases, Scylla doesn't have the database concept. In Scylla we work with keyspace, which is a top-level container that stores tables. Besides creating a keyspace, we need to configure other attributes like replication strategy.

CREATE KEYSPACE automobiles WITH replication = {'class': 'SimpleStrategy', 'replication_factor' : 3};

Like I said previously, it's necessary to configure replication strategy and how we can see, this is done with attribute replication. Inside brackets has two configurations. The replication_factor which means that the data inserted in a table of automobiles keyspace will be replicated in 3 nodes. Nodes are instances of Scylla DB. The other configuration is class and defines the replication strategy class to use. There are 2 options:

'SimpleStrategy': defines a replication factor for the whole cluster. The only sub-options supported is 'replication_factor' to define that replication factor and is mandatory.

'NetworkTopologyStrategy': A replication strategy that lets you set the replication factor independently for each data center. The rest of the sub-options are key-value pairs where a key is a data-center name and its value is the associated replication factor or a single key-value with Auto-Expand Replication Factor. Recommended for production. (Just for knowledge, we won't talk about this option in this post)

Now we can create our table. The name will be Car.

USE automobiles ;

CREATE TABLE car (id uuid, brand text, model text, color text, PRIMARY KEY(id)) ;

How we can see, Scylla syntax is similar to SQL syntax, and to include data, we use the insert clause.

INSERT INTO car (id, brand, color, model) VALUES (123e4567-e89b-12d3-a456-426614174000, 'VW', 'Red', 'Golf');

To see data in the table is just necessary to use the select clause.

SELECT * FROM car;

 id                                   | brand | color | model
--------------------------------------+-------+-------+-------
 123e4567-e89b-12d3-a456-426614174000 |    VW |   Red |  Golf

Conclusion

The idea of this post is to presenting Scylla DB and some operations that we can do. There are more details that I intended to present in the future, as partition key and clustering key, and how we can define our primary key mixing up of both concepts. This is crucial for better performance at the moment that we need to filter our searches. For now, that is enough. I will see you in the next post.

References
https://docs.scylladb.com/getting-started/dml/
https://university.scylladb.com/

[PT-BR] Mark and Sweep e as gerações da heap

João Victor Martins — Mon, 15 Mar 2021 20:40:54 +0000

Um dos objetivos da JVM é gerenciar a memória utilizada pelos nossos sistemas. Isto é feito através de processos que são executados ao longo do ciclo de vida da aplicação. O garbage collector é um dos mais importantes. De forma resumida, ele é o responsável por liberar espaço na memória, coletando objetos que não estão mais sendo referenciados. A ideia deste post é explicar, em detalhes, como é feita esta coleta.

Garbage Collector

Sempre que instanciamos um objeto na nossa aplicação, o mesmo é alocado em um espaço de memória chamado heap. Com o passar do tempo muitos objetos perdem suas referências e não são mais úteis para o sistema. Com um acumulo de objetos não referenciados, o garbage collector é invocado. O processo é chamado de mark and sweep e envolve dois passos. No primeiro passo é necessário descobrir quais objetos ainda possuem referência. Uma vez descobertos estes objetos serão marcados. No segundo passo, todos os outros objetos que não foram marcados estarão prontos para a coleta e é nesse momento que ocorre uma varredura para liberação do espaço de memória. Um detalhe que não foi informado é que enquanto o processo é executado, a aplicação para de rodar, com o intuito de evitar concorrência com o garbage collector. Essa parada é chamada de stop the world. Se sempre que o garbage collector for varrer a heap, os objetos que possuem referência forem marcados, elevará o tempo da parada e os usuários sentirão que a aplicação não está respondendo naquele momento, o que não é interessante. Então como fazer para melhorar o processo de coleta?

Young e Old Generation

Para evitar o problema visto anteriormente, a heap foi dividida em gerações. Essas gerações são a young generation e a old generation. A young possui um espaço menor e é dividida em 3 partes, que são o eden, s0 e s1 (survivor espaces). Todo novo objeto será criado no eden. Conforme aumenta a quantidade de objetos no eden, o garbage collector é chamado, porém o processo de marcação e varredura leva menos tempo, pois conforme informado, há menos espaço destinado para a young generation. Após a execução do gc, os objetos que ainda possuem referência, serão alocados no espaço s0. Como novos objetos estão sendo criados no eden, quando o gc é executado, ele irá olhar para os sobreviventes do eden e do s0 e os migrará para o espaço s1. Após esse processo inicial, todos os objetos criados no eden, poderão ser migrados para o s0 ou s1, sem ordem de precedência e o gc atuará em todos estes espaços. Após um número x de execuções (a depender da JVM e configurações feitas pelos usuários) os objetos sobreviventes na young generation, serão fragmentados e movidos para a old generation. Na old generation o processo de coleta será executado apenas quando necessário, ou seja, quando não possuir mais espaço e isso ocorrerá com menor frequência.

Conclusão

Percebemos que o garbage collector é um processo extremamente importante para mantermos a performance da nossa aplicação. Porém, se fosse realizado a coleta em toda a heap, por existir o stop the world, a aplicação pararia de executar, para não concorrer com o gc e isso causaria uma instisfação aos usuários do sistema, que iriam sentir que a aplicação não estaria respondendo naquele momento. Isso é evitado graças as gerações da heap. Na young generation o processo de coleta é executado com maior frequência, mas de forma muito rápida, pois é um espaço menor, fazendo com que não seja percebida pelos usuários. Já na old generation, espaço que possui os objetos sobreviventes, o processo é executado apenas quando necessário, com uma frequência bem menor, e quando ocorre, apesar de ser sentido pelos usuários, não é tão prejudicial. O assunto do post de hoje foi curto, porém complexo, então toda crítica, dúvida ou sugestão serão bem-vindas. Espero que tenham gostado. Até o próximo.