Forem: Noe Lopez

Document your API with Spring Rest Docs

Noe Lopez — Sat, 02 Dec 2023 11:50:53 +0000

Overview

Documenting you API is crucial not only for other programmers in your organization but also any clients who have to consume it. When it comes to documenting REST APIs in Spring projects we have mainly two options available:

Swagger/OpenAPI specification.
Spring REST docs.

They have a different way of producing documentation:

OpenAPI has an API first approach. The springdoc-openapi library helps to automate the generation of the documentation in JSON/YAML and HTML format APIs. It inspects the code in the Controllers to create the JSON format. It also comes with a UI to interact with the API. To customise the input/output of the operations, annotations can be declared in the source code.

Spring rest docs uses a test-driven approach. This garantees that the documentation is always up to date and accurate. By default Spring uses Asciidoctor to generate API the documentation. Asciidoctor is an open source text processor written in Ruby that can convert an Asciidoc text file to other formats like HTML, PDF, etc. The final output will be an static HTML page. Documentation is created at built-time.

We will show how you can add documentation for you APIs with Spring rest doc.

Requirements

The minimum requirements for Spring REST dosc are:

Java 17
Spring Framework 6

In our project the testing framework is JUnit 5 and the documentation will be generated by Spring MVC's test framework (aka MocMvc)

Dependencies and Configuration

The first thing to do is to set up the project build. Documentation will be generated during the project build phase.

Add the below dependency in test scope in your pom.xml. In case you are testing the web layer with WebTestClient or REST Assured then add the dependecies spring-restdocs-webtestclient or spring-restdocs-restassured respectively.

<dependency> 
    <groupId>org.springframework.restdocs</groupId>
    <artifactId>spring-restdocs-mockmvc</artifactId>
    <scope>test</scope>
</dependency>

Examining the above artifact we can see it contains the spring-restdocs-core, spring-webmvc, spring-test and the servlet-api from jakarta.

The next step is to configure the Asciidoctor plugin. With it the AsciiDoc files can be converted to HTML format.

<plugin>
    <groupId>org.asciidoctor</groupId>
    <artifactId>asciidoctor-maven-plugin</artifactId>
    <version>2.2.1</version>
    <executions>
        <execution>
        <id>generate-docs</id>
        <phase>prepare-package</phase>
        <goals>
        <goal>process-asciidoc</goal>
        </goals>
        <configuration>
            <backend>html</backend>
            <doctype>book</doctype>
        </configuration>
    </execution>
    </executions>
    <dependencies>
        <dependency>
        <groupId>org.springframework.restdocs</groupId>
        <artifactId>spring-restdocs-asciidoctor</artifactId>
    </dependency>
    </dependencies>
</plugin>

Optionally, the documentation can be packaged in the jar/war file and served as static content on your application. To do that, the resource pluging must be placed after the Asciidoc plugin as they are bound together to the prepare-package phase.

<plugin> 
    <artifactId>
</artifactId>
    <version>2.7</version>
    <executions>
    <execution>
        <id>copy-resources</id>
        <phase>prepare-package</phase>
        <goals>
            <goal>copy-resources</goal>
        </goals>
        <configuration> 
            <outputDirectory>
                    ${project.build.outputDirectory}/static/docs
        </outputDirectory>
            <resources>
            <resource>
                <directory>
              ${project.build.directory}/generated-docs
                </directory>
            </resource>
        </resources>
        </configuration>
    </execution>
    </executions>
</plugin>

In the configuration section the generated code is copied to the static/docs folder and included in the jar/war (every time mvn package is run).

We are done with the configuration. Now it is time to write an integration test for the customer controller.

JUnit 5 testing

We will start with the customers endpoint, in particular the findCustomer method which is displayed in the below lines

@RestController
@RequestMapping("api/v1/customers")
public class CustomerController {
...
    @GetMapping(path = "{customerId}", 
                produces = APPLICATION_JSON_VALUE)
    public ResponseEntity<CustomerResponse> findCustomer( 
        @PathVariable("customerId") Long id) {
        return customerService
            .findById(id)
            .map(CustomerMapper::mapToCustomerResponse)
            .map(ResponseEntity::ok)
            .orElseThrow(() -> 
                new EntityNotFoundException(id, Customer.class));
}
...

Lets write a test for findCustomer method. As we are using JUnit 5, the rest documentation extension must be used. Then, mvcmocktest annotation in order to test http semantics. This is called a slicing because it will initiate the Bean Container with just the necessary beans to load the web layer (No real http comunication happens here). Moreover, as we are only testing the Customer Controller, it can be specified in the annotation. This way it is possible narrow down the scope of the test by only loading one specific controller.

@ExtendWith({ RestDocumentationExtension.class})
@WebMvcTest(CustomerController.class)
public class CustomerControllerIntegrationTest {

    private MockMvc mockMvc;

    @BeforeEach
    void setUp(WebApplicationContext webApplicationContext, 
        RestDocumentationContextProvider restDocumentation) {
        mockMvc = MockMvcBuilders
           .webAppContextSetup(webApplicationContext)              
           .apply(documentationConfiguration(restDocumentation))
           .build();
    }
}

However, the service must be mocked as it is a dependency of the controller. The method findById in the service is invoked and will return a customer object as shown below. Then, mockMvc simulates a get request to the endpoint URL. We check that the controller returns the correct status code. Finally, andDo method is where the test method is documented and the snippets are written to a folder named customer-get-by-id.

@ExtendWith({ RestDocumentationExtension.class})
@WebMvcTest(CustomerController.class)
public class CustomerControllerIntegrationTest {
    private MockMvc mockMvc;

    @MockBean
    private CustomerService customerService;

    @BeforeEach
    void setUp(WebApplicationContext webApplicationContext, 
        RestDocumentationContextProvider restDocumentation) {
        mockMvc = MockMvcBuilders
            .webAppContextSetup(webApplicationContext)
            .apply(documentationConfiguration(restDocumentation))                        
            .build();
    }

    @Test
    @DisplayName("When getting a customer by id then return the 
        customer data.")
    void givenId_whenGetCustomerById_thenReturnCustomer() 
    throws Exception {
        final var id = 1L;

        Mockito.when(customerService.findById(id))
            .thenReturn(Optional.of(Customer.Builder
                .newCustomer()
                .id(id)
                .name("John Wick")
                .status(Customer.Status.ACTIVATED)
                .email("john.wick@gmail.com")
                .dob(LocalDate.of(1981,9,05))
                .withDetails("Info", true)
                .build()));

        mockMvc.perform(get("/api/v1/customers/{id}", id)
            .accept(MediaType.APPLICATION_JSON))
            .andExpect(status().isOk())
            .andDo(document("customer-get-by-id"));
    }
}

One important thing to mention is that the andDo method takes a ResultHanlder argument. The static method document belongs to org.springframework.restdocs.mockmvc.MockMvcRestDocumentation.

Another alternative is to load the full context with @SpringBootTest annotation. It will configure all the layers of the application. In this scenario, the data is retreived from the in-memory database H2. There is no need to have an instance of the Controller because it is managed by the Bean Container.

@ExtendWith({ RestDocumentationExtension.class})
@SpringBootTest
public class CustomerControllerIntegrationTest {
    private MockMvc mockMvc;

    @BeforeEach
    void setUp(WebApplicationContext webApplicationContext, 
        RestDocumentationContextProvider restDocumentation) {
        mockMvc = MockMvcBuilders
            .webAppContextSetup(webApplicationContext)
            .apply(documentationConfiguration(restDocumentation))
            .build();
    }

    @Test
    @DisplayName("When getting a customer by id then return the 
         customer data.")
    void givenId_whenGetCustomerById_thenReturnCustomer() 
        throws Exception {
        final var id = 1L;

        mockMvc.perform(get("/api/v1/customers/{id}", id)
                .accept(MediaType.APPLICATION_JSON))
                .andExpect(status().isOk())
                .andDo(document("customer-get-by-id"));
    }
}

Now running the command to package the application as below

mvn clean package

generates the code snippets for the test method in the folder target\generated-snippets. The sub folder customer-get-by-id contains all the asciidoc files created for the test method (Figure 1)

By default, 6 snippets are found in the directory: curl-request.adoc, http-request.adoc, http-response.adoc, httpie-request.adoc, request-body.adoc and response-body.adoc.

The contents of the file http-response.adoc are displayed next

[source,http,options="nowrap"]
----
HTTP/1.1 200 OK
Content-Type: application/json
Content-Length: 161

{"id":1,"status":"ACTIVATED","personInfo":{"name":"John Wick","email":"john.wick@gmail.com","dateOfBirth":"05/09/1981"},"detailsInfo":{"info":"Info","vip":true}}
----

Asciidoc File

Now the snippets are generated but how this can be converted to a more visually friendly format like HTML? A base asciidoc file can be added to the project referencing the snippets using the include directive. In our project the file is saved in src/docs/asciidocs/index.adoc. Its contents can be seen in the following lines

= RESTful Customer API Specification

:doctype: book

== Find a customer by id

A `GET` request is used to find a new person by id

operation::customer-get-by-id[snippets='http-request,http-response']

Asciidoc semantic is intuitive. If you are interested in it and want to look at it in more detailed, visit its website

Next, we must indicate the asciidoctor plugin:

The directory where our base/index asciidoc file is located
The output directory where the html page is written.

<configuration>                          
   <backend>html</backend>
   <doctype>book</doctype>
   <sourceDirectory>src/docs/asciidocs</sourceDirectory>
   <outputDirectory>target/generated-docs</outputDirectory>
</configuration>

Once the application is re-packaged the index.html is found in the folder target/generated-docs. It is also copied to the classes folder as well as inside the jar file in static/docs. This is done by the maven-resources-plugin that was setup in the Dependencies and Configuration section.

Next figure (Figure 2) is a screenshot of the index.html

So far, we have seen a simple example and how to put all the pieces together to create the index.html page. In the next sections, we will explore more features of Spring Doc Rest.

Path and Query parameters

Let's document the path parameters for the get customer id endpoint. The method expects the id as part of the URL. Static method pathParameters of class RequestDocumentation allows us to do so.

mockMvc.perform(get("/api/v1/customers/{id}", id)
    .accept(MediaType.APPLICATION_JSON))
    .andExpect(status().isOk())
    .andDo(document("customer-get-by-id",
             pathParameters(parameterWithName("id")
                 .description("Customer id"))))

The static document accepts a Snippet varargs as the second argument. We passed the pathParameters with a single ParameterDescriptor.

Likewise, the query parameters can be defined using the static method queryParameters. FindCustomers method in the Controller expects up to 4 optional parameters. Let's code a test for it and document it

@Test
@DisplayName("When getting customers then return list of customers.")
void givenParams_whenGetCustomers_thenReturnListOfCustomers() throws Exception {
    final var queryParams = 
        "?name=name 1&status=ACTIVATED&info=info&vip=true";

    mockMvc.perform(get("/api/v1/customers"+queryParams)
        .accept(MediaType.APPLICATION_JSON))
        .andExpect(status().isOk())
        .andDo(document("customer-search-customers",
            queryParameters(
                parameterWithName("name")
                    .description("Customer name filter"),
                parameterWithName("status")
                    .description("Customer status filter. Values  
                    ACTIVATED/DEACTIVATED/SUSPENDED"),
                parameterWithName("info")
                    .description("Customer info filter"),
                parameterWithName("vip")
                    .description("Customer vip filter. Values 
                    true/false"))));
}

Re-running the maven command to build the application will write two new adoc files. One named path-parameters.adoc in customer-get-by-id folder and the other named query-parameters.adoc in customer-search-customers folder.

Request/Response payload

By default, the body of the request and response are generated in files request-body.adoc and response-body.adoc. They contain sample data tha was used to run the test. However, more complete documentation related to the request and response can be provided.

Let us demostrate it with the tests we have coded so far. Both, findById and findCustomers response share the same json structure. The only difference is findcustomers returns an array rather than a single object. An example is presented below

{
    "id": 1,
    "status": "DEACTIVATED",
    "personInfo": {
        "name": "name 1 surname 1",
        "email": "organisation1@email.com",
        "dateOfBirth": "04/02/1971"
    },
    "detailsInfo": {
        "info": "Customer info details 1",
        "vip": false
    }
}

The response field's of the above sample are documented as follows

@Test
    @DisplayName("When getting a customer by id then return the customer data.")
    void givenValidId_whenGetCustomerById_thenReturnCustomer() throws Exception {
        final var id = 1L;

        mockMvc.perform(get("/api/v1/customers/{id}", id)
                .accept(MediaType.APPLICATION_JSON))
                .andExpect(status().isOk())
                .andDo(document("customer-get-by-id",
                        pathParameters(customerIdParam),
                        responseFields(customerResponseFields)));
}

private FieldDescriptor[] customerResponseFields = {
   fieldWithPath("id").description("Customer id"),
   fieldWithPath("status").description("Customer's status"),
   subsectionWithPath("personInfo")
      .description("The customer's personal info section"),            
   fieldWithPath("personInfo.name").description("Customer's name"),            
   fieldWithPath("personInfo.email")
      .description("Customer's email"),            
   fieldWithPath("personInfo.dateOfBirth")
      .description("Customer's Date Of Birth"),            
   subsectionWithPath("detailsInfo")
      .description("The customer's details info section"),            
   fieldWithPath("detailsInfo.info")
      .description("Customer's detailed info"),            
   fieldWithPath("detailsInfo.vip")
      .description("Customer's vip flag")};

Static method responseFields will produce a snippet describing the fields named response-fields.adoc. In our code all the fields are extracted to a member named customerResponseFields so that it can be reused and avoid repetition. Also, method subsectionWithPath will document the nested objects in the json object.

But the test for findCustomers expects an array of the same form. Well, that is easy to do as shown below

mockMvc.perform(get("/api/v1/customers"+queryParams)
    .accept(MediaType.APPLICATION_JSON))
    .andExpect(status().isOk())
    .andDo(document("customer-search-customers",
         queryParameters(customerSearchQueryParams),
         responseFields(fieldWithPath("[]")
             .description("An array of customers"))
             .andWithPrefix("[].", customerResponseFields)));

The [] represents an array in combination with the method andWithPrefix describing the content of each element in the array.

Request/Response headers

If you read all the way down to here, you are already familiar with the sprind doc principles. Documenting headers is very similar to what we have done to generate other snippets such as queryparams. The following test method verifies adding a new customer successfully. The response returns http status 201 created along with the location header to find the resouce.

@Test
@DisplayName("When adding a customer then return created with location.")
void givenCustomerRequest_whenAddCustomer_thenReturnCreated() 
    throws Exception {
    final var body = """
                {
                      "name" : "Lawrence Lesnar",
                      "email" : "lawrence.lesnar@gmail.com123",
                      "dateOfBirth" : "2001-03-28",
                      "info" : "more info here",
                      "vip" : "true"
                }
                """;

    mockMvc.perform(post("/api/v1/customers")
                       .content(body)
                       .contentType(MediaType.APPLICATION_JSON))
        .andExpect(status().isCreated())
        .andDo(document("customer-add",
               requestFields(customerRequestFields),
               responseHeaders(headerWithName("location")
          .description("The location/URL of the new customer."))));
}

This time responseHeaders method documents the location header. It also generates a snippet called response-headers.adoc with the information.

The update customer and delete customer test methods will not be presented in this article because they use all the features explained in the previos examples. They are in the repository code in case you want to take a look at them. Click here to go to code in github.

Preprocessors

Spring doc comes with some preprocessors which can be used to modify the request and response before the snippets are created. This comes in handy when you want to add or remove parts of the request/response to your documentation.

Preprocessing can be configured at test method level or for all the tests. This is setup with an OperationRequestPreprocessor or an OperationResponsePreprocessor in the document method for individual test or in the documentationConfiguration in the @Before method for all your tests.

The below code applies the prettyPrint preprocessor to any request/response documentation generate during the tests.

@BeforeEach
void setUp(WebApplicationContext webApplicationContext, 
           RestDocumentationContextProvider restDocumentation) {
    this.mockMvc = MockMvcBuilders
        .webAppContextSetup(webApplicationContext)
        .apply(documentationConfiguration(restDocumentation)
               .operationPreprocessors()
               .withRequestDefaults(prettyPrint())
               .withResponseDefaults(prettyPrint()))
        .build();
}

There are other preprocessors that allow to modify headers or URIs. Full list of preprocessors can be read in Spring doc official documentation.

Index HTML.

All the tests for our API are documented now. Let's add the generated snippets to the index.adoc asciidoc file. Then, this file will be converted to the final index.html containing all the information.

= RESTful Customer API Specification
;
:doctype: book
:icons: font
:source-highlighter: highlightjs
:toc: left
:toclevels: 2
:sectlinks:

[[Index]]
= Index

This is the index for the Customer Rest API endpoint.

[[index-find-customer]]
== Find a customer by id

A `GET` request is used to find a customer by id

operation::customer-get-by-id[snippets='http-request,path-parameters,http-response,response-fields']

[[index-find-customers]]
== Find customers

A `GET` request is used to find customers

operation::customer-search-customers[snippets='http-request,query-parameters,http-response,response-fields']

[[index-add-customer]]
== Add a customer

A `POST` request to add a new customer

operation::customer-add[snippets='http-request,request-fields,http-response,response-headers']

[[index-update-customer]]
== Update a customer

A `PUT` request to update an existing customer

operation::customer-update[snippets='http-request,path-parameters,request-fields,http-response,response-fields']

[[index-delete-customer]]
== Delete a customer by id

A `DELETE` request is used to delete an existing customer by id

operation::customer-delete[snippets='http-request,path-parameters,http-response']

And executing the maven package command again to re-build the app as well as the documentation. Html file is located in target\generated-docs as explain earlier in this article. Next figure shows a screenshot of the above adoc file.

Conclusion

We started this article looking at the different options available for documenting our APIs. Then, it was needed some setup and configuraton to have spring doc ready in our project. This part is a bit annoying but the rest is straighforward. Then, we created the test methods and added the snippets to generate the adoc files. The last step is to write an adoc file with the information contained in the snippets and conver it to html.

As usual the repository code can be found in github. Hope you liked the content of the article and subcribe!

Monitoring a Directory in Spring

Noe Lopez — Tue, 07 Nov 2023 08:56:48 +0000

Overview

The idea for this article was to add a new feature to the customer application so that it can process files copied to a directory at any time. Monitoring a directory is a common task found in many computer systems and applications. There are different options available in the java ecosystem to perform this type of job. Below are listed just a few of the possible solutions:

Java WatchService API: It was introduced in Java 7 and it is low level.
Apache commons io: The package monitor provides a component for monitoring file system events.
Spring Integration's file support: This is part of the Spring integration project which supports a wide range of Enterprise Integration Patterns.
Filewatch package from Spring Boot developer tools: It allows to watch the local filesystem for changes.

We will look at option number 4 as it is straightforward to implement plus our project is already built on top of the Spring framework.

The use case

We would like to create new customers in the app by copying a csv file to a specific location. The file will be read once it is fully transferred. Then, the csv file will be validated, processed and moved to a destination directory. Below is a sample of the csv file:

name, email, dob, info, vip
James Hart,james.hart@gmail.com,12/05/2002,Information for James,No
Will Avery,will.avery@gmail.com,23/10/1991,Information for Will,Yes
Anne Williams,anne.williams@gmail.com,12/05/1975,Information for Anne,No
Julia Norton,julia.norton@gmail.com,23/10/1984,Information for Julia,Yes

The csv file will always contain the header row with 5 columns in that particular order.

Prerequisites

This project is built with the below technologies:

Java 21
Spring Boot 3.1.5
Maven 3.9.1

You will need at least java 17+, Spring Boot 3+ and Maven 3.8+ run the source code.

Dependencies

One new dependency has to be added to the pom.xml file to be able to import the necessary classes.

<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-devtools</artifactId>
</dependency>

This is all we need to start coding in the next section.

Monitor Directories

The class to watch specific directories for file changes is FileSystemWatcher. It comes with three constructors but most likely you will use the one accepting three arguments

public FileSystemWatcher(boolean daemon,
                         Duration pollInterval,
                         Duration quietPeriod)

Let's have a look at each argument

deamon: if a deamon thread monitors the changes. Set it to true if you want the thread (the monitoring) to be killed when the jvm is stopped.
pollInterval: time to wait between checking again for changes.
quietPeriod: time to wait after a change is detected to ensure. If you transfer large files to the directory, this will have to be taken into account to avoid file corruption.

As we want all of the above argument values to be altered without affecting the source code, custom proepeties are added to the application.properties file

application.file.watch.daemon=true
application.file.watch.pollInterval=5
application.file.watch.quietPeriod=1
application.file.watch.directory=C:\\workspace\\files\\customer

The application will scan the directory for modifications every 5 mins. And the change will be trigger/propagated after 1 minute. This will suffice because the csv files are small (less than 1 MB).
And the corresponding java code to load the values in it. Spring Boot 3 supports records which keeps it brief and simple.

@ConfigurationProperties(prefix = "application.file.watch")
public record FileWatcherProperties(
    @NotBlank String directory,
    boolean daemon,
    @Positive Long pollInterval,
    @Positive Long quietPeriod
) {}

The next step is to define the configuration class with the bean type FileSystemWatch

@Configuration
@EnableConfigurationProperties(FileWatcherProperties.class)
public class CustomerFileWatcherConfig {
// ommiting class members, constructor and logger

    @Bean
    FileSystemWatcher fileSystemWatcher() {
        var fileSystemWatcher = new FileSystemWatcher(
                properties.daemon(),
                Duration.ofMinutes(properties.pollInterval()),
                Duration.ofMinutes(properties.quietPeriod()));
        fileSystemWatcher.addSourceDirectory( 
            Path.of(properties.directory()).toFile());
        fileSystemWatcher.addListener(
            new CustomerAddFileChangeListener(fileProcessor));
        fileSystemWatcher.setTriggerFilter(
            f -> f.toPath().endsWith(".csv"));        
        fileSystemWatcher.start();
        logger.info(String.format("FileSystemWatcher initialized. 
            Monitoring directory %s",properties.directory()));

        return fileSystemWatcher;
    }

    @PreDestroy
    public void onDestroy() throws Exception {
        logger.info("Shutting Down File System Watcher.");
        fileSystemWatcher().stop();
    }
}

Let's review the method fileSystemWatcher():

First an instance of fileSystemWatcher is created passing the values from the bean properties as arguments. The bean properties object is managed by the spring container and injected via constructor.
The addSourceDirectory method is invoked. It takes a File representing the directory to be monitored.
The addListener methods takes a listener for file change events. FileChangeListener is a functional interface and its method onChange will be called when files have been changed.
Optionally an FileFilter can be setup in the method setTriggerFilter to restrict the files that trigger a change. A lambda expression, evaluated to a boolean, is used to limit the files to csv.
The Last method is start to initiate monitoring the source directory for changes.

Notice that there a predestroy hookup method to shutdown gracefully the watcher when the jvm is stopped.

Adding the listener

The second step to implement a FileChangeListener where the files can be processed. This interface is functional too, hence a lambda expression or method reference can be used. In our case it will be better to place it in its own class as it will improve readability.

public class CustomerAddFileChangeListener implements 
    FileChangeListener {
    private static Logger logger = LoggerFactory.getLogger( 
        CustomerAddFileChangeListener.class);

    private CustomerCSVFileProcessor fileProcessor;

    public CustomerAddFileChangeListener(
        CustomerCSVFileProcessor fileProcessor) {
        this.fileProcessor = fileProcessor;
    }

    @Override
    public void onChange(Set<ChangedFiles> changeSet) {
        for(ChangedFiles files : changeSet)
            for(ChangedFile file: files.getFiles())
                if (file.getType().equals(ChangedFile.Type.ADD))
                    fileProcessor.process(file.getFile().toPath());
    }
}

As it has been mentioned earlier the onChange method is invoked when files have been changed. The parameter Set is a collection of files changed (since the poll interval started). Once iterating through the changeSet, the files can be obtained from each ChangedFiles object via the getFiles method. Therefore a nested loop will work to get to the individual files. The single file is of type ChangedFile which provides access to the File and Type of the change/event (This is a enum with three values ADD, DELETE and MODIFY).

Back to the code, the if statement checks the type to make sure only ADD events will handled. The others will be ingnored. The class CustomerCSVFileProcessor performs all the work. It is not in the scope of this article but it will be outlined briefly in the next section.

Processing the file

The businness logic to handle the file is found in the process method (declared in the FileProcessor interface). It needs the CustomerService to save the Customers in the database, thus it is constructor-injected. The class CustomerCSVFileProcessor is marked as a compoment because it is also injected in the Watcher.

@Component
public class CustomerCSVFileProcessor implements FileProcessor {
    public static final int NUMBER_OF_COLUMNS = 5;

    private static Logger logger = LoggerFactory.getLogger( 
        CustomerCSVFileProcessor.class);

     private CustomerService customerService;

     public CustomerCSVFileProcessor(
         CustomerService customerService) {
         this.customerService = customerService;
     }

     public void process(Path file) {
        logger.info(String.format(
            "Init processing file %s",file.getFileName()));
        var parser = CSVParser.parse(file);
        parser.getRecords().forEach(this::processRecord);
        moveFile(file);
    }

    private void processRecord(CSVRecord csvRecord) {
       if (csvRecord.size() < NUMBER_OF_COLUMNS) {
           logger.info(String.format(
              "Line %d skipped. Not enough values.", 
              csvRecord.lineNumber()));
           return;
       }

       Customer customer = customerMapper.mapToCustomer(csvRecord);
       customer.setStatus(Customer.Status.ACTIVATED);
       customerService.save(customer);
       logger.info(String.format(
           "Saved customer %s in line %d",
           customer.getName(), 
           csvRecord.lineNumber()));
    }

    private static void moveFile(Path file) {
        try {
            var destinationFolder = Path.of( 
                file.getParent().toString() + OUTPUT_FOLDER );
            Files.move(
                file, 
                destinationFolder.resolve(file.getFileName()), 
                REPLACE_EXISTING);
            logger.info(String.format(
                 "File %s has been moved to %s",file.getFileName(), 
                 destinationFolder));
        } catch (IOException e) {
            logger.error("Unable to move file "+ file, e);
        }
    }
}

The method parses the csv file with the help of a CSVParser. Then each cvs row (depicted as a CSVRecord) is mapped to a customer entity and saved it the db. Finally, the foreach loop is completed and the file is moved to a distinct location.

Testing the app

Re-running the application will start the file system watcher bean from CustomerFileWatcherConfig config class. As soon as the app is up, a csv file with a few customers is copied to the monitored directory. After 5 mins the below trace is generated

CustomerFileWatcherConfig : FileSystemWatcher initialized. Monitoring directory C:\workspace\files\customer
CustomerCSVFileProcessor   : Init processing file customerFile.csv
CustomerCSVFileProcessor   : Saved customer James Hart in line 1
CustomerCSVFileProcessor   : Saved customer Will Avery in line 2
CustomerCSVFileProcessor   : Saved customer Anne Williams in line 3
CustomerCSVFileProcessor   : Saved customer Julia Norton in line 4
CustomerCSVFileProcessor   : File customerFile.csv has been moved to C:\workspace\files\customer\output

Indeed, the data rows in the file where added as customer entities to the database and the file moved successfully.

Conclusion

In this short article we have demonstrated how to configure a Spring app to monitor local directories. This functionality comes in handy to perform automated tasks. For instance, in combination with other services as such an FTP server transferring files to a directory and then the file watcher picking it up for processing.

Hope you enjoyed the reading. As usual, source code can be found in the github project.

Spring Cache with Caffeine

Noe Lopez — Sun, 22 Oct 2023 12:14:22 +0000

Introduction

In the last article it was explained in detailed how Spring Cache works. Our customer app benefited from this by reducing the number of accesses to the service layer methods (and by extension to the repository layer). The application used the default simple implementation which is based on a ConcurrentHashMap. The main disadvantage of this option is that it does not offer a cache eviction policy. Hence, entries must be removed explicitly. That was achieved with a fixed scheduler to clean up the HashMap.

This is a rudimentary solution. That is why in this article we are going to take a look at a cache implementation supported in Spring: The Caffeine library.

Caffeine

Caffeine is a java caching library known for its efficiency. Under the hood, Caffeine employs the Window TinyLfu (build upon Bloom filter theory) policy providing high hit rate (the ratio between the number of cache hits and the total number of data accesses) and low memory footprint. This algorithm has better performance than LRU (Least Recently Used) and it is a good choice for general purpose cache.

Visit the official Caffeine git project and documentation here for more information if you are interested in the subject.

Main Features

Caffeine provides the following optional features:

Automatic loading of entries into the cache, optionally asynchronously.
Three types of eviction: size-based eviction, time-based eviction, and reference-based eviction.

Size-based eviction when a maximum is exceeded based on
frequency and recency.

Time-based expiration of entries, measured since last access or
last write.

Reference-based evicts when neither the key nor value are
strongly reachable.
Asynchronously refresh when the first stale request for an entry occurs.
Keys automatically wrapped in weak references.
Values automatically wrapped in weak or soft references.
Notification of evicted entries. Removal listeners can be setup to perform an operation when an entry is removed.
Writes propagated to an external resource.
Accumulation of cache access statistics.

Dependencies

Caffeine dependency has to be added to the pom.xml. We will download latest version 3.1.8 at the time of writing this article. This version is compatible with java 11+. If you are still running your apps in java 8 you must pick any 2.x version.

<dependency>
    <groupId>com.github.ben-manes.caffeine</groupId>
    <artifactId>caffeine</artifactId>
    <version>3.1.8</version>
</dependency>

Configuration

There are two ways to configure Caffeine in Spring. The first one is by setting the cache properties in the application configuration file. In our case, it is the application.properties file but it could be the yml too.

The below sample defines the properties for two cache regions, customer and customersSearch. Each region will have an initial capacity of 10 entries, a maximum of 100 entries and entries will be automatically removed from the cache after 1 hour since the last read or write (also called TTL time to live).

spring.cache.cache-names=customers,customersSearch
spring.cache.caffeine.spec=initialCapacity=10,maximumSize=100,expireAfterAccess=1h

The caffeine spec can be parsed from a string. The syntax is a collection of comma separated key-value pairs. Each setting corresponds to a builder method of the Caffeine class. Full list of spec props can be found here.

However, this configuration is global and has its limitations. It applies the same settings to all the caches which may not be suitable for many applications handling multiple cache regions.

Now let's have a look at the second approach. Caffeine can also be configured programatically. We start declaring a Caffeine Bean holding the spec of the Cache as shown below

Caffeine caffeineSpec() {
    return Caffeine.newBuilder()
        .initialCapacity(10)
        .maximumSize(100)
        .expireAfterAccess(1, TimeUnit.HOURS)x;
}

The next step is to create a CacheManager passing the above configuration. We can place this in a new class as demostrated here

@Configuration
@EnableCaching
public class CaffeineCacheConfig {

    @Bean
    public CacheManager cacheManager(Caffeine caffeine) {
        CaffeineCacheManager cacheManager = new 
            CaffeineCacheManager("customers", "customersSearch");
        cacheManager.setCaffeine(caffeine);
        return cacheManager;
    }

    @Bean
    Caffeine caffeineSpec() {
        return Caffeine.newBuilder()
                .initialCapacity(10)
                .maximumSize(100)
                .expireAfterAccess(1, TimeUnit.HOURS);
    }
}

As you can see Spring already comes with a specific cache implementation for Caffeine, so it makes it easier to integrate with Caffeine. The CaffeineCacheManager constructor can accept more than one cache name/region in the overloaded varargs version.

Let's test the find customer endpoint from the CLI app (check out previous articles on Spring Shell series and Spring Rest app).

The first request to the endpoint to get the customer id 2 from our cli client app.

shell(administrator1):>find-customer 2

will trigger a db call in the RestApi app

org.hibernate.SQL                        : select c1_0.id,c1_0.date_of_birth,d1_0.customer_id,d1_0.created_on,d1_0.info,d1_0.vip,c1_0.email,c1_0.name,c1_0.status from customer c1_0 join customer_details d1_0 on c1_0.id=d1_0.customer_id where c1_0.id=?
org.hibernate.orm.jdbc.bind              : binding parameter [1] as [BIGINT] - [2]
d.n.r.aspect.LoggingExecutionTimeAspect  : Method [Optional dev.noelopez.restdemo1.service.CustomerService.findById(Long)] with params [2] executed in 42 ms
d.n.r.aspect.LoggingExecutionTimeAspect  : Method [ResponseEntity dev.noelopez.restdemo1.controller.CustomerController.findCustomer(Long)] with params [2] executed in 60 ms

Then the second and subsequence calls will be loaded from the cache.

d.n.r.aspect.LoggingExecutionTimeAspect  : Method [ResponseEntity dev.noelopez.restdemo1.controller.CustomerController.findCustomer(Long)] with params [2] executed in 0 ms

Configuring Multiple Cache Regions

In the previous section the Caffeine Bean sets the spec/configuration for two regions. As each region manages two different set of data, they will probably have different needs. For example, customers region stores a single customer while customersSearch stores list of customers. Hence we may require distinct max size for the regions. Another example, suppose data A is updated less frequently than data B. In this scenario, it could make more sense to have a longer time to live set for data A. Tuning the cache properties will depend on how the data is accessed/updated/deleted/stored in you application and the constrains in resources.

Back to our application, we are interested in having particular TTL and max size for the regions. To achive it we need to make a few changes in class CaffeineCacheConfig.

@Bean
public CacheManager cacheManager() {
    CaffeineCacheManager cacheManager = new CaffeineCacheManager();
    cacheManager.registerCustomCache("customers", 
        buildCache(100,200, 2));
    cacheManager.registerCustomCache("customersSearch",
        buildCache(50,100, 1));
    return cacheManager;
}

private Cache buildCache(
    int initialCapacity, int maximumSize, int durationInHours) {
    return Caffeine.newBuilder()
        .initialCapacity(initialCapacity)
        .maximumSize(maximumSize)
        .expireAfterAccess(durationInHours, TimeUnit.HOURS)
        .build();
}

CaffeineCacheManager provides a convinient method to register a native a Caffeine Cache instance. Any number of custom caches may be registered by invoking this method as many times as needed. This allows for custom settings per cache.
The private method buildCache creates a Cache instance with some features. This is typically used with the Caffeine builder API. The build() method returns a Cache with the features set while constructing the builder.

Notification on Eviction

Caffeine provides a mechanism to notify when an entry is removed from the cache. Listeners can be added to the Cache config. There are two type of listeners:

Eviction Listener: it is triggered when an eviction happens ( meaning removal due to the policy). This operation is performed synchronously.
Removal Listener: it is triggered as a consequence of eviction or invalidation (meaning manual removal by the caller). The operation is executed asynchronously using an Executor, where the default executor is ForkJoinPool.commonPool() and can be overridden via Caffeine.executor(Executor).

Both listeners receive a RemovalListener which is a functional interface. Its abstract method signature is

void onRemoval(@Nullable K key, @Nullable V value, RemovalCause cause)

key and value are normally Object type and RemovalCause is an enum with the specific cause (EXPLICIT,REPLACED,COLLECTED,EXPIRED,SIZE).

Registering listeners in the Cache config is as simple as the next code

private Cache<Object, Object> buildCache(
    int initialCapacity, int maximumSize, int durationInHours) {
    return Caffeine.newBuilder()
        .initialCapacity(initialCapacity)
        .maximumSize(maximumSize)
        .expireAfterAccess(durationInHours, TimeUnit.MINUTES)
        .evictionListener((Object key, Object value, 
            RemovalCause cause) ->
            logger.info(String.format(
                "Key %s was evicted (%s)%n", key, cause)))
        .removalListener((Object key, Object value, 
            RemovalCause cause) ->
            logger.info(String.format(
                "Key %s was removed (%s)%n", key, cause)))
        .build();
}

Let's have a look at 3 user cases to verify the listeners.

Deleting a customer:

This operation is done using the command

shell(administrator1):>delete-customer 2
Customer 2 deleted successfully

The customer is loaded first hence it is added to the cache via @Cacheable. Then, it is deleted from the db and it is evicted manually from cache via @CacheEvict. This last action causes the listener to be executed

d.n.r.config.CaffeineCacheConfig: Key 2 was removed (EXPLICIT)

Excedding the max size in the cache:

If the cache reaches its max size, then new entries will cause old entries to be evicted.

Finding a new customer when the cachee is full

shell(administrator1):>find-customer 21

causes the removal of another entry. Note that both listeners were executed this time

d.n.r.config.CaffeineCacheConfig: Key 2 was evicted (SIZE)
d.n.r.config.CaffeineCacheConfig: Key 2 was removed (SIZE)

Expired entry:

Customer 1 is searched via the find-customer command and added to the cache with key 1. Then, there are no more accesses to key 1 and time to live expires. However, the entry is not removed until a new customer is searched. In this case customer 2 is fetched and the removal of key 1 occurs.

d.n.r.config.CaffeineCacheConfig: Key 1 was evicted (EXPIRED)
d.n.r.config.CaffeineCacheConfig: Key 1 was removed (EXPIRED)

Cleanup

By default, Caffeine does not perform cleanup and evict values "automatically" or instantly after a value expires. Instead, it carries out small pieces maintenance of work after write operations or occasionally after read operations if writes are rare.

If your cache is mainly read and seldom writes, it is possible to delegate the cleanup task to a thread. A scheduler can be specified to prompt removal of expired entries regardless of whether any cache activity is occurring at that time.

return Caffeine.newBuilder()
    .initialCapacity(initialCapacity)
    .maximumSize(maximumSize)
    .expireAfterAccess(durationInHours, TimeUnit.MINUTES)
    .evictionListener((Object key, Object value, 
        RemovalCause cause) -> logger.info(
           String.format("Key %s was evicted (%s)%n", key, cause)))
    .removalListener((Object key, Object value, 
        RemovalCause cause) -> logger.info( 
           String.format("Key %s was removed (%s)%n", key, cause)))
    .scheduler(Scheduler.systemScheduler())
.build();

After adding the systemScheduler to the cache config, the entries are automatically removed once they expire.

d.n.r.config.CaffeineCacheConfig : Key 21 was evicted (EXPIRED)
d.n.r.config.CaffeineCacheConfig : Key 21 was removed (EXPIRED)
d.n.r.config.CaffeineCacheConfig : Key 22 was evicted (EXPIRED)
d.n.r.config.CaffeineCacheConfig : Key 22 was removed (EXPIRED)

The scheduling is best-effort and does not make any hard guarantees of when an expired entry will be removed. This means, the entries may not be removed immediately at the time of expiration.

Enable Statistics

Statistics can be turned on with the recordStats in the Caffeine builder. Then, the Cache.stats() method returns a CacheStats object which provides statistics such as

hitRate(): returns the ratio of hits to requests
evictionCount(): the number of cache evictions
averageLoadPenalty(): the average time spent loading new values

For instance, we can code a Cache Controller to present the stats as well as the list of entries in the cache regions.

@RestController
@RequestMapping("admin/cache")
public class CacheController {
    private CacheManager cacheManager;

    public CacheController(CacheManager cacheManager) {
        this.cacheManager = cacheManager;
    }

    @GetMapping()
    public List<cacheInfo> getCacheInfo() {
        return cacheManager.getCacheNames()
            .stream()
            .map(this::getCacheInfo)
            .toList();
    }

    private cacheInfo getCacheInfo(String cacheName) {
        Cache<Object, Object> nativeCache = 
          (Cache)cacheManager.getCache(cacheName).getNativeCache();
        Set<Object> keys = nativeCache.asMap().keySet();
        CacheStats stats = nativeCache.stats();
        return new cacheInfo(
          cacheName, keys.size(), keys, stats.toString());
    }

    private record cacheInfo(
        String name, int size, Set<Object> keys, String stats) {}
}

Accessing the new endpoint returns valuable info that can be used to fine-tune the caches, specially in performance-critical applications.

[
    {
        "name": "customersSearch",
        "size": 5,
        "keys": [
            [null,null,null,null],
            [null,"ACTIVATED",null,null],
            [null,null,null,false],
            [null,null,null,true],
            ["name","ACTIVATED",null,null]
        ],
        "stats": "CacheStats{hitCount=0, missCount=11, loadSuccessCount=0, loadFailureCount=0, totalLoadTime=0, evictionCount=0, evictionWeight=0}"
    },
    {
        "name": "customers",
        "size": 10,
        "keys": [1,2,3,4,5,7,9,13,888,910],
        "stats": "CacheStats{hitCount=2, missCount=12, loadSuccessCount=0, loadFailureCount=0, totalLoadTime=0, evictionCount=0, evictionWeight=0}"
    }
]

Spring Actuator Cache Metrics

Spring Actuator also provides cache related information. The endpoint /caches displays a list of all configured caches. In our app, accessing the url http://localhost:8080/actuator/caches will send back the json

{
    "cacheManagers": {
        "cacheManager": {
            "caches": {
                "customersSearch": {
                    "target": "com.github.benmanes.caffeine.cache.BoundedLocalCache$BoundedLocalManualCache"
                },
                "customers": {
                    "target": "com.github.benmanes.caffeine.cache.BoundedLocalCache$BoundedLocalManualCache"
                }
            }
        }
    }
}

The metrics endpoint supports Caffeine library and provides a basic set of metrics. However, only caches that are configured on startup are bound to the registry.

The url http://localhost:8080/actuator/metrics/cache.size?tag=cache:customers displays the size of the cache region customers

{
    "name": "cache.size",
    "description": "The number of entries in this cache. This may be an approximation, depending on the type of cache.",
    "measurements": [
        {
            "statistic": "VALUE",
            "value": 10
        }
    ],
    "availableTags": [
        {
            "tag": "cache.manager",
            "values": [
                "cacheManager"
            ]
        },
        {
            "tag": "name",
            "values": [
                "customers"
            ]
        }
    ]
}

There are other metrics but they are coming empty. For this reason I would recommend to code your own controller or service to get the statistics directly from the cache provider just like it was done in section Enable Statistics.

Summary

We come to a conclusion now. We have learned how the library caffeine works internaly and its main features. Then we have seen how to configure it globlaly and with particular settings per cache region. We also explored other not so common functionality of Caffeine such as removal listeners, clean up, statistics with plenty of examples to demostrate it.

Git repo with the code is available here.

Feel free to share this post with your colleagues and subcribe for more Java and Spring related content!

Spring Cache - Speed up your App

Noe Lopez — Sun, 08 Oct 2023 14:15:49 +0000

Introduction

In modern applications it is crucial to reduce response times and optimize resources as much as possible. One of the techniques to improve application performance is caching. Caching is a hidden mechanism in which the same data can be read multiple times in a faster way. Caching also helps scalability of applications.

Spring cache is a module of the Spring framework providing support for transparently adding caching to an existing Spring application. It works by applying caching to Java methods. That means it reduces the number of executions based on the information in the cache. Whenever a cache method is invoked, Spring checks if the method was already called with the same parameters. If it was, the result from the cache is returned. Otherwise, the method is executed and the result is stored in the cache. The next time the method is called, the cached result is returned.

Be aware that is approach only works if the method ensures the same output for a given input.

Cache Providers

Spring cache is an abstraction and not an implementation. This means you can write code compatible with multiple implementations but requires the actual storage for the cache.

The main interfaces of the abstraction are org.springframework.cache.Cache and org.springframework.cache.CacheManager. Cache interface defines common cache operations like get, put, evict and clear. It is a set of key-value pairs. CacheManager allows to retreive cache regions (an instance of a Cache). it is a wrapper to obtain caches.

Spring supports the following providers:

JCache (JSR-107) (EhCache 3, Hazelcast, Infinispan, and others)
Hazelcast
Infinispan
Couchbase
Redis
Caffeine
Cache2k
Simple

If no caching library is present in the app, then the default is the simple provider. It is configured an implementation backed by a ConcurrentHashMap as the cache store.

To setup caching in an app, two actions are needed:

Configure the cache where data is stored.
Identify the methods where caching is applied.

So let's implement caching to our Customer REST Api. We will start with the simple cache in this article and then switch to other providers in future releases.

Dependencies

The first step is to add the cache dependency to the pom.xml



<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-cache</artifactId>
</dependency>

Under the hood, this starter contains the spring-context-support dependency with the spring beans, context and core artifacts.

Configuring Cache

Cache annotations are not activated by default. They must be declatively enabled so that the annotation are automatically trigger. So do so we use the @EnableCaching in any of the configuration classes of the application.



package dev.noelopez.restdemo1.config;

import org.springframework.cache.annotation.EnableCaching;
import org.springframework.context.annotation.Configuration;

@Configuration
@EnableCaching
public class CacheConfig {
}

There are some important notes to take in account with the default settings:

The default advice mode for processing caching annotations is proxy. This means local calls to the method (in the same class) are not intercepted in this mode.
When using proxy mode, annotations should only be applied to public methods.
it is recommended to only annotate concrete classes with cache annotations.

Now we are ready to mark methods with cache annotations. Let's put this into practice in the next section.

Adding entries to the cache

@Cacheable is used to designate a method as cacheable. The result is stored in the cache so that the next invocations with the same parameters are returned from the cache region avoiding to execute the method.



@Service
public class CustomerService {
...
    @Cacheable(cacheNames="customers", key="#id")
    public Optional<Customer> findById(Long id) {
        return customerRepo.findById(id);
    }
...
}

In the above snippet the method findById is marked as cacheable in the customers region. The key for cache access is the customer id. keep in mind that a Cache is basically a key-value pair store. Hence, we need a way to get the value from it. In the section section it will be explain how keys are generated in more detail.

Now let's called the method passing the argument 2. As demostrated in the below lines the methof in the service layer is triggered.



2023-09-10T19:27:21.130-03:00  INFO 15748 --- [nio-8080-exec-1]
 d.n.r.aspect.LoggingExecutionTimeAspect  : Method [Optional
 dev.noelopez.restdemo1.service.CustomerService.findById(Long)]
 with params [2] executed in 13 ms
2023-09-10T19:27:21.143-03:00  INFO 15748 --- [nio-8080-exec-1]
 d.n.r.aspect.LoggingExecutionTimeAspect  : Method [ResponseEntity
dev.noelopez.restdemo1.controller.CustomerController.findCustomers
(Long)] with params [2] executed in 34 ms

However, on the second invocation the method is not run and the value returned from the cache. As you can see only the Controller method is invoked but not the service method.



2023-09-10T19:29:18.499-03:00  INFO 15748 --- [nio-8080-exec-3] 
d.n.r.aspect.LoggingExecutionTimeAspect  : Method [ResponseEntity 
dev.noelopez.restdemo1.controller.CustomerController.findCustomers
(Long)] with params [2] executed in 1 ms

Both calls to the method return the same data to the client consuming the customer endpoint.

Key Generation

Spring uses a simple key generator that uses the method parameters provided to create a key. If a method has no parameters then an empty simple key is returned. This strategy will be ok for most use cases provided that the params implement valid equals and hascode methods.

There are situations where we want to specify how the key is generated. For instance if one of the parameters is irrelevant for the caching as it is only needed for method logic.



@Cacheable(cacheNames="customersSearch",
       key = "{#customer.name, #customer.status, 
               #customer.email, #customer.details.vip}",
public List<Customer> findByFields(Customer customer) {
    return customerRepo.findByFields(customer);
}

In the above code, findByFields method filters customers by name, status, email and vip. Hence, the key can be created only using them.

The method stores the results in a new cache region which must be declared in the cache configuration



@Configuration
@EnableCaching
public class CacheConfig {
    @Bean
    public CacheManager cacheManager() {
        return new ConcurrentMapCacheManager( 
            "customers","customersSearch");
    }
}

Conditional Caching

There are situations where we do not want a method to add an entry to the cache for every invocation. For instance, consider the following scenarios:

Entries in the cache must be only for a range of values of an argument.
The return value must meet a specific requirement.

The first case is supported by the condition parameter. It evaluates to true or false a SPEL expresion. If it resolves to true, the method call is cached. If it is false, it will not be cached. Either case, the method is always executed.

Here the findByFields will be cached if a user is searching customers whose name is less than 10 characters.



@Cacheable(cacheNames="customersSearch",
       key = "{#customer.name, #customer.status, 
               #customer.email, #customer.details.vip}",
       condition="#customer.name?.length() < 10")
public List<Customer> findByFields(Customer customer) {
    return customerRepo.findByFields(customer);
}

As name can be null, the safe navigation operator (?) is used to avoid a NullPointerException.

The second case affects the returned value rather than the input params. Spring supports this option via the unless parameter. The expression is evaluated after the method is invoked. If the expression resolves to false, then the method is cached.

Let's see this in action with a couple of examples. Suppose we are interested in caching searches returning a list of 10 customers or more.



@Cacheable(cacheNames="customersSearch",
       key = "{#customer.name, #customer.status, 
               #customer.email, #customer.details.vip}",
       condition="#customer.name?.length() < 10",
       unless ="#result.size() < 10"
)

The second case applies to the findById method introduced in previous sections. Say we only need to cached VIP customers. The next snippet shows how to implement this change in the method



@Cacheable(cacheNames="customers", key="#id", 
           unless = "not #result?.details.vip")
public Optional<Customer> findById(Long id) {
    return customerRepo.findById(id);
}

Note that #result is the actual customer (the business entity) and not the java.lang.Optional wrapper.

Updating entries to the cache

@CachePut annotation is used when the cache has to be updated and the method is always executed. Once the method invocation is completed the result is put in the cache. If the entry was already added to the cache it is updated. It requires the cache region and the key same as @Cacheable. It can be configured with conditions too.

Our application allows to update customer data. In this case, the customer cache should be updated so that the latest changes are available in the cache when the user searches for a single customer.



@Transactional
@CachePut(cacheNames="customers", key="#customer.id")
public Customer save(Customer customer) {
    return customerRepo.save(customer);
}

Removing entries from the cache

Spring cache also provides a way of removing entries from a cache. This is called cache eviction and it comes in handy for removing unused data or stale data. The annotation @CacheEvict marks methods for cache eviction so that entries can be removed. It needs one or more cache regions that will be impacted by the action.

The below code triggers cache eviction when deleting a customer from the database. Let's explain this with an example. Say customer 2 is updated hence put in the cache region customers. Then it is deleted and removed from cache, so that next time is loaded it will not be in the cache. This will force the findById method to be run again and customer 2 will not be found.



@Transactional
@CacheEvict(cacheNames="customers", key = "#id")
public void deleteById(Long id) {
    customerRepo.deleteById(id);
}

Note that cache eviction takes place after the method is invoked. If a exception is raised and the delete operation fails the eviction is not trigger. No need to remove the customer from the cache if it was not possible to delete it.

If eviction must occur before the method is invoked, you must set the attribute beforeInvocation to true (default value is false).

Another feature of @CacheEvict is allEntries attribute which permits to clear out an entire region instead of an entry based on the key.



@Transactional
@CachePut(cacheNames="customers", key="#customer.id")
@CacheEvict(cacheNames="customersSearch", allEntries=true)
public Customer save(Customer customer) {
    return customerRepo.save(customer);
}

Here the customersSearch region is totally cleaned up when a customer is updated.

Simple Cache Eviction Policy

The cache abstraction does not provide any cache eviction policy. That must be done directly through your cache provider. However, we can implement a simple cache eviction policy using a scheduler. The idea is to clean up the cache every hour so that data in it does not grow fast.

This functionality can be added to the CacheCofig class.



@Configuration
@EnableCaching
@EnableScheduling
public class CacheConfig {
    private Logger logger = 
        LoggerFactory.getLogger(CacheConfig.class);

    @Bean
    public CacheManager cacheManager() {
        return new ConcurrentMapCacheManager("customers", 
                                             "customersSearch");
    }

    @Scheduled(fixedRate = Timer.ONE_HOUR)
    @Caching(evict = {
        @CacheEvict(cacheNames="customers", allEntries=true),
        @CacheEvict(cacheNames="customersSearch", allEntries=true)
    })
    public void clearAllCaches() {
        logger.info("Clearing all caches entries");
    }
}

Let's review the above code. First thing is to enable scheduling so that Spring is aware of it. The annotation @EnableScheduling ensures that a background task executor is created. Without it, nothing gets scheduled. Then, the Scheduled annotation defines when a particular method runs.

In our code, method clearAllCaches is scheduled to be executed at one hour intervals.

Another alternative is to use the cache manager to perform the cleaning.



@Scheduled(fixedRate = Timer.ONE_HOUR)
public void clearAllCachesWithManager() {
    CacheManager manager = cacheManager();
    manager.getCacheNames().stream()
        .forEach(name -> manager.getCache(name).clear());

    logger.info("Clearing all caches entries");
}

@Caching Annotation

The annotations we have seen so far are not repeatable. Meaning they can only be used once per method. What happens if we need multiple configurations for the same @CacheEvict o @CachePut annotations? @Caching allows to set different configurations for a method. This is usedful when two or more cache regions behave differently for the same method.

Let's examine the next code



@Transactional
@Caching(evict = {
         @CacheEvict(cacheNames="customers", key = "#id"),
         @CacheEvict(cacheNames="customersSearch", allEntries=true)
})
public void deleteById(Long id) {
    customerRepo.deleteById(id);
}

Two actions are triggered after the method call is finished. The entry associated to the id is removed from the customers region if it exists. The second @CacheEvict annotation is responsible for purging the whole customersSearch region.

Class level configuration

The cache operations we have implemented are defined at method level. All of them share the same cache region customers. It can be dull and monotonous to set it for each operation. Here is where annotation @CacheConfig helps.

We can replace the cacheNames="customers" attribute from the methods by @CacheConfig as shown below



@Service
@CacheConfig(cacheNames = "customers")
public class CustomerService {
...
    @Cacheable()
    public Optional<Customer> findById(Long id) {
        return customerRepo.findById(id);
    }
...
}

We just need the @Cacheable annotation declared in the method findById. The cache entry key is the id because it is the only parameter.

Listing all keys in the Cache region

Sometimes it is needed to inspect the data stored in the cache. Neither Cache interface nor CacheManager class offer a method to get all entries or keys. That means the information has to be pulled from the cache implementation our application is using.

The full qualified cache implementation class can be obtained from the manager



System.out.println(manager.getCache(name).getClass())
System.out.println(manager.getCache(name).getNativeCache().getClass());

The above line displays the cache manage and the internal cache storage.



class org.springframework.cache.concurrent.ConcurrentMapCache
class java.util.concurrent.ConcurrentHashMap

Knowing the storage implementation class, a cast operation is all needed to access the storage implementation. As we are dealing with a map, it is straightfordward to get the keys from it as demonstrated in the next code.



private void printCachesKeys(CacheManager manager) {
    manager.getCacheNames().stream()
        .forEach(name -> printCacheKeys(manager, name));
}

private void printCacheKeys(CacheManager manager, String name) {
    ConcurrentHashMap cacheMap = (ConcurrentHashMap) 
        manager.getCache(name).getNativeCache();
    logger.info(String.format("Cache Name %s contains %s entries " 
        ,name, cacheMap.size()));
    logger.info(String.format("keys : %s", cacheMap.keySet()));
}

The method can be called before cleaning the cache or rin any other scenario. It will print the cache name and its keys.



Cache Name customersSearch contains 2 entries 

keys : [[null, null, null, true], [null, null, null, null]]

Cache Name customers contains 5 entries 

keys : [1, 2, 3, 4, 12]

Conclusion

In this article, we have learned how Spring Cache works and the different operations available like adding, updating and removing entries from the cache. We looked at some of the features such as key generation and cache class configuration. Finally, we coded some useful examples to automatically clear caches and examine its size and keys.

As usual, project code can be found in the git repository here.

Hope you enjoyed reading the article and don’t forget to subscribe to be notified for future articles on more subjects in the Spring and Java ecosystem.

Spring Shell - Availability and Customization

Noe Lopez — Sun, 27 Aug 2023 21:15:16 +0000

Introduction

This is the second part of the Spring Shell series. In the previous article a cli application was built to get customer data from an e external. Then, the data was presented in tabular format. Now we are going to extend functionality of our app to perform write operations via the customers web service.

Here is the list of items that will be covered:

Registering new commands to update/delete customers.
Capturing user input.
Validating input.
Adding Authentication.
Dynamic Command Availability.
Customization.

Registering new commands

The steps to register the new commands were already explained in the first part of these series. The new command takes one required argument, the customer id to be deleted. Code is shown here as a reference

@Command(command = "delete-customer", 
         description = "Delete Customer By Id.")
    public String deleteCustomer(
    @Option(label = "id", required = true,
    longNames = "id", shortNames = 'i') Long id) {
    customerService.deleteCustomer(id);
    return String.format("Customer %d deleted successfully", id);
}

Also it is worth printing the implementation of the delete method. Notice that the response from the web service is bodiless, hence the return type is void.

public void deleteCustomer(Long id) {
    restClient.delete()
        .uri("/{id}",id)
        .accept(MediaType.APPLICATION_JSON)
        .retrieve()
        .toBodilessEntity();
}

Let's re-run the application to verify that the command actually works as expected and customers can be removed.

Capturing user input

Next, let's implement a command to update a customer. This time, the user will be required to enter the new values of the fields. Spring shell relies on JLine (a Java library for handling console input) and many of its features are avalaible. To interact with the user and ask for input the class LineReader will be used. More information on JLine can be found in its home page.

First, we will write a class to read from the CLI which will delegate to LineReader to get user input.

public class ShellReader {
    private LineReader lineReader;

    public ShellReader(LineReader lineReader) {
        this.lineReader = lineReader;
    }

    public String readLine(String displayText) {
        return lineReader.readLine(displayText + ": ");
    }

    public String readLineRequired(String displayText) {
        String value;

        do {
            value = lineReader.readLine(displayText + ": ");

            if (value.isEmpty())
                lineReader.getTerminal().writer().println("Value 
                    is mandatory. Please enter a value.");
        } while (value.isEmpty());

        return value;
    }

    public String readLineOptions(String displayText, 
                                  List<String> options) {
        String value;

        do {
            value = lineReader.readLine(displayText +" "+ options 
                                        + ": ");

            if (!options.contains(value))
                lineReader.getTerminal().writer().println("Please 
                    select a value from "+ options);
        } while (!options.contains(value));

        return value;
    }
}

An auto-configured LineReader is injected by Spring. The ShellReader class has three methods to read input:

Method readLine simply prints a message on the console, waits for the user input and returns the value entered.
The second method readLineRequired checks that the input is not empty and loops till a non-emtpy value is typed. If the value is empty a message is printed to the consloe via getTerminal method.
The last method readLineOptions checks the input matches from a list of possible options and loops till a valid option is entered.

Next, the class will be registered as a bean so that it can be constructor-injected in the Customer Command class. The bean will be placed in the AppConfig class

@Configuration
public class AppConfig {
    // Other beans omitted

    @Bean
    public ShellReader shellReader(@Lazy LineReader lineReader) {
        return new ShellReader(lineReader);
    }
}

Finally, ShellReader is declared as a member in the command class

@Command(group = "Customer Commands")
public class CustomerCommands {
    private OuputFormatter ouputFormatter;
    private CustomerService customerService;
    private ShellReader shellReader;

    public CustomerCommands(CustomerService customerService, 
                            OuputFormatter ouputFormatter, 
                            ShellReader shellReader) {
        this.customerService = customerService;
        this.ouputFormatter = ouputFormatter;
        this.shellReader = shellReader;
    }
}

Now, we are in a position to define the command method to update a customer. Code is shown below

@Command(command = "update-customer", 
         description = "Updates Customer fields.")
public String updateCustomer(
    @Option(label = "id", required = true,
    longNames = "id", shortNames = 'i'``) Long id) {
    CustomerResponse customer = customerService.findCustomer(id);

    var name = shellReader.readLineRequired(String.format( 
        "Enter new Name (%s)",customer.personInfo().name()));
    var email = shellReader.readLineRequired(String.format( 
        "Enter new Email (%s)",customer.personInfo().email()));
    var dob = shellReader.readLineRequired(String.format( 
        "Enter new Date Of Birth (%s)", 
        customer.personInfo().dateOfBirth()));
    var info = shellReader.readLineRequired(String.format( 
        "Enter new Info (%s)",customer.detailsInfo().info()));
    var vip = shellReader.readLineOptions(String.format(
        "Is customer vip ?",customer.detailsInfo().vip()),
        List.of("Y","N"));

    var customerRequest = new CustomerRequest(
                name, 
                email, 
                DateUtils.parseDate(dob),
                info,
                vip.equals("Y"));

    customer = customerService.updateCustomer(id,customerRequest);

    return String.format("Customer %d deleted successfully. \n%s", 
           id, ouputFormatter.coverToTable(customer));
}

Let's examine the above snippet. First, the customer dto is loaded from the service. Then, each field is read from the console thanks to the ShellReader object. A customer request record is created with the data entered by the user and passed to the service for updating. The user is presented a message if the operation is completed successfully.

The following figure demonstrates the code in action when the application is run.

The update customer code of the service layer is added here just for reference

public CustomerResponse updateCustomer(Long id, 
                CustomerRequest customerRequest) {
    return restClient.put()
                .uri("/{id}",id)
                .accept(MediaType.APPLICATION_JSON)
                .body(customerRequest)
                .retrieve()
                .body(CustomerResponse.class);
}

Now that the new commands are working as expected, it is time to add new features to our application.

Validation input

Spring shell integrates with the Bean Validation API to support constrains on command parameters. Annotations on command parameters trigger a validation before the command is executed.

The following dependencie must be added to make use of Bean Validation.

<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-validation</artifactId>
</dependency>

Suppose the id to delete a customer must be a number between 1 and 9999. Adding the @Range annotation to the command param will take care of it.

public String deleteCustomer(
    @Range(min = 1, max = 9999, message = "Customer id must be 
        between {min} and {max}.")
    @Option(label = "id", required = true, longNames = "id", 
        shortNames = 'i') Long id) {
    customerService.deleteCustomer(id);
    return String.format("Customer %d deleted successfully", id);
}

Because exceptions are handled by our custom command exception resolver in the application, the thrown exception (of type ParameterValidationException) can be intercepted when the validation fails. The error messages from the constrains (in case more than one validation does not pass) can be extracted and joined together.

public class CLIExceptionResolver implements CommandExceptionResolver {
    @Override
    public CommandHandlingResult resolve(Exception ex) {
        if (ex instanceof RestClientCustomException e)
            return CommandHandlingResult.of( 
              e.getErrorDetails().message()+'\n');
        else if (ex instanceof ResourceAccessException e)
            return CommandHandlingResult.of(
              "Customer API is not available at the moment"+'\n');
        else if (ex instanceof ConversionFailedException e)
            return CommandHandlingResult.of(
              "Customer id must be a positive integer"+'\n');
        else if (ex instanceof ParameterValidationException e)
            return CommandHandlingResult.of( 
              e.getConstraintViolations()
                    .stream()
                    .map(ConstraintViolation::getMessage)
                    .collect(Collectors.joining(". "))
                    +'\n');

        return CommandHandlingResult.of(ex.getMessage()+'\n', 1);
    }
}

From the preceding code, the below output is displayed

Adding Authentication

The next feature of cli app is to create a command class to login and logout a user. The idea is to allow to access to the write commands (update and delete) based on user authentication. These commands will only be available once a user has logged in.

To keep things simple the user credentails are stored in the application.properties files. Password will be encrypted with the algorithm bcrypt. Hence, the dependency security crypto from Spring Security project can be added to the pom.xml

<dependency>
    <groupId>org.springframework.security</groupId>
    <artifactId>spring-security-crypto</artifactId>
</dependency>

The login service contains methods to login and logout. It verifies username and password are matched. If login is successful the user will be stored so that it can be later checked if the user is logged or not.

@Service
public class LoginService {
    @Value("${application.cli.username}")
    private String username;

    @Value("${application.cli.password}")
    private String hashedPassword;

    private final AtomicReference<SimpleUser> user
        = new AtomicReference<>();

    public void login(String username, String password) {
        if (!username.equals(username) || !BCrypt.checkpw(password,hashedPassword)) {
            throw new LoginException("Incorrect username/name!");
        }

        user.set(new SimpleUser(username));
    }

    public void logout() {
        user.set(null);
    }

    public boolean isLoggedIn() {
        return Objects.nonNull(user.get());
    }
}

SimpleUser is just a record with one data field, the username.

public record SimpleUser(String username) {}

Credentails are kept in the application.properties. In a real life application it may connect to a database or service to authenticate the user. We will just keep it simple in this article.

application.cli.username=administrator1
application.cli.password=$2a$12$LqTeXfZfc6sw6D86YcbLKeY4Y.Hrq0HXYQhg7vFTS0.psZoF4UdkK

The last step is to create the command class to perform the two actions. Not much going here, the new LoginService is used to authenticate the user or terminate the session by logging out.

@Command(group = "Login Commands")
public class AuthenticationCommands {
    private LoginService loginService;

    public AuthenticationCommands(LoginService loginService) {
        this.loginService = loginService;
    }

    @Command(command="login",
             description="Login to get Admin privileges.")
    public String login(
        @Size(min = 8, max = 20, message = "Username must be 
        between {min} and {max} ") 
        @Option(required = true) String username, 
        @Size(min = 10, message = "Password must be at least 
        {min} chars long")
        @Option(required = true) String password) {
        loginService.login(username,password);
        return "You are logged in now!";
    }

    @Command(command="logout",description="logout as an Admin.")
    public String logout() {
        loginService.logout();
        return "You have been logged out.";
    }
}

Now, the app can login and logout users. But they are of no use on themselves. We need a way to enable and disable commands based on them. The next section provides an answer to this.

Dynamic Command Availability.

Some registered commands require a prior action to take place before they can be invoked. For instance, the logout method will only work if the user has already logged in. Trying to call the logout method when the user is not yet authenticated, the shell will express that the method exists but it is not available at the moment.

The way of indicating command availality in Spring Shell is by annotating the method with @CommandAvailability and specifying a availability provider. The main types involved in this mecanism are Availability class and AvailabilityProvider interface. Availability class indicates whether or not a command is currently available. It is constructed with one of the two factory methods available() or unavailable() with a reason. AvailabilityProvider is a functional interface to supply an Availability instance.

Let's see how all this works. The steps to get the job done are described here:

Declare two providers as beans. One to be applied to commands with write operations like update/delete. The other to permit logging out.
Apply the annotation to the coomands with the particular provider.

The below code is added the our AppConfig class

@Bean
public AvailabilityProvider userLoggedProvider() {
    return () -> loginService.isLoggedIn()
        ? Availability.available()
        : Availability.unavailable("You are already logged in.");
}

@Bean
public AvailabilityProvider userLoggedOutProvider() {
    return  () -> !loginService.isLoggedIn()
        ? Availability.available()
        : Availability.unavailable("You are not logged in.");
}

The second step is to add the annotation on the command classes. Lets begin with AuthenticationCommands

@Command(command="login",
         description="Login to get Admin privileges.")
@CommandAvailability(provider = "userLoggedOutProvider")
public String login(...) { ... }

@Command(command="logout",description="logout as an Admin.")
@CommandAvailability(provider = "userLoggedProvider")
public String logout() { ... }

and continue with CustomerCommands

@Command(command = "update-customer", 
description = "Updates Customer fields. It requires user login.")
@CommandAvailability(provider = "userLoggedProvider")
public String updateCustomer(...) { ... }

@Command(command = "delete-customer", 
description = "Deletes Customer By Id. It requires user login.")
    @CommandAvailability(provider = "userLoggedProvider")
    public String deleteCustomer(...) { ... }

It is time to start up the app and see if all is in order. The first thing to notice is the output of the help command. It marks the unavailable commands with asterisk as demostrated in the next figure

Invoking the delete-customer command would prompt the following message on the console

The user must be logged first to be able to call the command.

Now the delete operation can be perfomed as expected. Also the help command shows that the write commands are available. Login is disabled as it does not make sense once the user is already logged.

Finally, the user can logout once it is done with the session.

Customization

Spring Shell offers a wide range of options to customize the shell. In this section we are going to look at a couple of those options.

Command Not Found message: By default a non-existing command is handled via CommandNotFoundResultHandler and outputs a simple message

Internally CommandNotFoundResultHandler is using CommandNotFoundMessageProvider which is a functional interface taking a ProviderContext and returning a text message. It is possible to define it as a bean and can be written as a lambda function.

@Bean
CommandNotFoundMessageProvider provider() {
    var message = """
               The command '%s' you entered was not found. 
               Use help to view the list of available commands 
               """;
    return ctx -> String.format(message, ctx.text());
}

As we can see the new message is returned now

Prompt Name: By default the prompt displays the value shell:>. This can be changed by implementing the PromptProvider interface. It can be either defined as a bean or a compoment class.

@Bean
PromptProvider promptProvider() {
    return () -> loginService.isLoggedIn() ?
        new AttributedString( 
            "shell("+loginService.getUser().username()+"):>") 
        : new AttributedString("shell(unknown):>");
}

Let's run the app again to verify the change and call some commands

Conclusion

In this second part, we have explored more advance features of spring shell. The project allows to build cli app easily freeing the developer to focus on core command logic. It comes with a lot of features and we have seen some of them in action. Full code can be found in the git hub repository here.
Hope you enjoyed reading the article and don’t Forget to Share This Post!

Spring Shell - Build CLI apps

Noe Lopez — Sun, 20 Aug 2023 11:25:16 +0000

What is Spring Shell?

Spring shell is a project of the Spring ecosystem that helps create shell applications (command line apps interacting via the terminal) easily.

In this article we are going to build a runnable application with some commands to communicate with the Customer RESTAPI app. We will re-use the code from the last article (New Spring RestClient).

The project

We are going to build a cli application with the following features:

Allow searching customers by different criteria and display the results in table format.
Allow finding a specific customer by id and display the information in table format.
Allow updating customer properties and display outcome of the operation. This will require authentication in the application.
Allow deleting a customer by id and display outcome of the operation. This will require authentication in the application.

In this article we will concentrate in items 1 and 2 and will leave 3 and 4 for a second part. Lets move on to the next section and start setting up the project.

Project Dependencies

This project will need two dependecies, Spring shell and Spring web. We will add the Spring validation dependency later on to validate user input.

Maven pom.xml file must contain



<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-web</artifactId>
</dependency>
<dependency>
   <groupId>org.springframework.shell</groupId>
   <artifactId>spring-shell-starter</artifactId>
</dependency>

When running the application, Spring Boot detects the shell starter dependencies in the classpath and will automatically prompt a cli as shown below

Registering Commands

We need to register 4 commands, one for each action. There are two ways to define a command: using an annotation model and using a programmatic model. We will choose the annotation model for our application. Methods and classes are marked with specific annotations to define commands.

Spring boot 3.1.x comes with new support for defining commands using annotations. The previous annotations (@shell...) are considered legacy and will be deprecated and removed. Hence, we will focus on the new annonations.

The @Command annotation is used to mark a method as candidate for command registration. It can also be placed on a class to apply settings to methods defined in the same class.

Lets create a class with a command method find a candidate by id



1. @Command(group = "Customer Commands")
2. public class CustomerCommands {
3.     private CustomerService customerService;

4.     public CustomerCommands(CustomerService customerService) {         
5.         this.customerService = customerService;
6.     }

7.     @Command(command="find-customer",
8.              description="Finds a Customer By Id")
9.     public String findCustomer(@Option(required = true, 
10.            description = "The customer id") Long id) {
11.        return customerService.findCustomer(id).toString();
12.     }
}

Let's analyse the above lines of code:

In line 1, the class is marked as a command (which means it will have command methods). The group option organises related commands that can be put together and displays them in the same group in the help option.
In line 3, our app will hand over to a service to find the customer.
In line 7, method findCustomer is annotated as a command method. The annotation accepts the command option to specify the name of the command when invoked from the shell. By default, command key follow dashed gnu-style names (For instance, method findCustomer becomes find-customer). Also, description is set to provide more info on what the command does.
In line 9, return type will be a string representing the customer. @Option marks the parameter as required command option of type Long. If it is not informed an error will be raised. It is possible to defined the short and long format of the option. That is, the prefix - or -- before the parameter value. We will see it in action when testing the code.
In line 11, The call to search the customer by id is delgated to service (Service code will be shown at a later point in the article).

But that is not enough to register a target command. It is required to use @EnableCommand and/or @CommandScan annotations.

We will register our target class CustomerCommands using @EnableCommand. It will be picked up when application is run.



@SpringBootApplication
@EnableCommand(CustomerCommands.class)
public class CustomerShellApplication {
    public static void main(String[] args) {
        SpringApplication.run( 
            CustomerShellApplication.class, args);
    }
}

Alternatively, the @CommandScan will automatically scan all command targets from all packages and classes under the RestClientShellApplication class.



@SpringBootApplication
@CommandScan
public class CustomerShellApplication {
    public static void main(String[] args) {
    SpringApplication.run(
            CustomerShellApplication.class, args);
    }
}

Now the application is ready to be tested. Before that there is one thing to mention. When the application is started the SpringBoot banner and the logs are displayed in the CLI. This will deteriorate the user experience. Both can be turned off adding the properties



logging.level.root=OFF
spring.main.banner-mode=off
spring.main.web-application-type=none

As the project contains spring mvc dependencies to make use of the restclient, the web server can be disabled.

Let's run the application and type help

As we can see in the above picture there are two command groups. The Built-In Commands provided by Spring. They are common features that can be found in most shells.
The second group is the Customer Commands group where the find-customer command is located. The command and description are shown as set in the annotation.

Searching a customer is as simple as typing any of the below lines

First line call does not specify the arg name. Second and third lines pass the short and long format respectively. In all cases, the correct customer is returned from the web service.

Service Layer

We are going to explore the Service layer more detailed in this section. As this is not directly related to Spring Shell you may skip this part and go to the Exception Handling section.

The service is a class that facilitates communication between the command class and the web service.



@Service
public class CustomerService {
    private HttpAPIHandler httpAPIHandler;

    public CustomerService(HttpAPIHandler httpAPIHandler) {
        this.httpAPIHandler = httpAPIHandler;
    }

    public CustomerResponse findCustomer(Long id) {
        return httpAPIHandler.findCustomer(id);
    }
}

It uses a handler to call the web service with the Spring RestClient. For more info on how RestClient works you can visit the previous article here



@Component
public final class HttpAPIHandler {
    private final APIProperties apiProperties;
    private final RestClient restClient;
    private final ObjectMapper objectMapper;

    public HttpAPIHandler(APIProperties apiProperties,
                          ObjectMapper objectMapper) {
        this.properties = properties;
        this.objectMapper = objectMapper;
        restClient = RestClient.builder()
            .baseUrl(properties.getUrl())
            .defaultHeader(HttpHeaders.AUTHORIZATION,
                encodeBasic(properties.getUsername(), 
                            properties.getPassword()))
            .defaultStatusHandler(
                HttpStatusCode::is4xxClientError,
                (request, response) -> {
                    var errorDetails = objectMapper.readValue( 
                        response.getBody().readAllBytes(), 
                        ErrorDetails.class);
                throw new RestClientCustomException( 
                    response.getStatusCode(), errorDetails);
               })
            .build();
    }

    public CustomerResponse findCustomer(Long id) {
        return restClient.get()
            .uri("/{id}",id)
            .accept(MediaType.APPLICATION_JSON)
            .retrieve()
            .body(CustomerResponse.class);
    }
}

The first constructor parameter injected is the APIproperties. This is a simple property configuration class which loads the url, username and password from application.properties. @Value works for a single property but we want to have them together, hence they are isolated into a separated POJO.



@Configuration
@ConfigurationProperties(prefix = "application.rest.v1.customer")
public class ClientProperties {
    String url;
    String username;
    String password;
    // setters/getter omitted
}

And the properties definitions in app



application.rest.v1.customer.url=http://localhost:8080/api/v1/customers
application.rest.v1.customer.username=user1234
application.rest.v1.customer.password=password5678

The second constructor parameter injected is an ObjectMapper. This class belongs to the jackson databind jar. It is needed to serialize the error response (json object containing status, message and timestamp) from the web service to the ErrorDetails record



public record ErrorDetails(
    int status, String message, LocalDateTime timestamp) {}

The ObjectMapper is declared in a separeted class for app configuration. To manage dates and times correctly the JavaTimeModule must be registered.



@Configuration
public class AppConfig {
    @Bean
    ObjectMapper objectMapper(){
        ObjectMapper objectMapper = new ObjectMapper();
        objectMapper.registerModule(new JavaTimeModule());
        return objectMapper;
    }
}

With that all the pieces of the service layer are connected.

Exception Handling

There are situations where the program will not complete the task to find a customer successfully. For example, what happens when the web service is down or a customer does not exists? The following figure shows the output in these scenarios.

Spring shell offers a chain of Exception Resolver implementations to resolve exceptions and can return messages to be displayed in the console. Optionally an exit code can be returned together (wrapped in the class CommandHandlingResult) however this is only for non-interactive shell.

The steps to setup a Exception resolver are described here:

Create a custom exception resolver implementing the interface CommandExceptionResolver.
Define the resolver as a bean globally (It can be defined for a particular command programmatically too. See Spring Shell documentation for more info on it).

Let's implement the Exception resolver class. Code can be seen in the next lines



public class CLIExceptionResolver implements 
    CommandExceptionResolver {
    @Override
    public CommandHandlingResult resolve(Exception ex) {
        if (ex instanceof RestClientCustomException e)
            return CommandHandlingResult.of( 
              e.getErrorDetails().message()+'\n');
        else if (ex instanceof ResourceAccessException e)
            return CommandHandlingResult.of(
              "Customer API is not available at the moment"+'\n');

        return CommandHandlingResult.of(ex.getMessage()+'\n', 1);
    }
}

CommandHandlingResult comes with an overloaded factory method of(). It takes a String representing the error message and optionally an exit code.

Next the bean must be defined. We will add it to the AppConfig class along with the ObjectMapper



@Configuration
public class AppConfig {
    @Bean
    CLIExceptionResolver customExceptionResolver() {
        return new CLIExceptionResolver();
    }
...
}

It is time to check if the exception handling works fine. Let's re-run the two tests.

Now both exceptions return the messages as per CLIExceptionResolver.

Formatting output

In this section we will be adding a new command to find customers and present the data in a table.

The new command method accepts two optional parameters. An enum representing the customer status and a Boolean indicating whether the customer is vip or not.



@Command(command = "find-customers",
         description = "Find Customers By Status and vip")
public String findCustomers(
    @Option(required = false,longNames = "status", 
            shortNames = 's') CustomerStatus status,
    @Option(required = false,longNames = "vip", 
            shortNames = 'v') Boolean vip) 
    throws JsonProcessingException {

1.    List<CustomerResponse> customers = 
        customerService.findCustomers(status, vip);
2.    return ouputFormatter.coverToTable(customers);
}

public enum CustomerStatus { ACTIVATED, DEACTIVATED, SUSPENDED } ;

In line 1, the service gets a list of customers filtering by status and/or vip. Implementation of the code in the HttpAPIHandler is shown below. The most interesting part of the code is the return statement in which the response as a string is converted to List with the help of the objectMapper. This is because body method does not accepts parameterised types (Passing List.class will return a list of maps).



public List<CustomerResponse> findCustomers(
    CustomerStatus status, Boolean vip) 
    throws JsonProcessingException {
    String response = restClient.get()
        .uri(getQueryString(status, vip))
        .accept(MediaType.APPLICATION_JSON)
        .retrieve()
        .body(String.class);
    return Arrays.stream(
       objectMapper.readValue(response,CustomerResponse[].class))
       .toList();
}

In line 2, the formatted ouput is generated with the help of the class outFormatter. This class is declared as a bean in Appconfig class where we kept all the projects beans. Then, it is constructor-injected in CustomerCommands.



@Bean
OuputFormatter ouputFormatter(){
    return new OuputFormatter();
}

Let's have a closer look this class to view how the table is created.



public final class OuputFormatter {
    public String coverToTable(List<CustomerResponse> customers) {
1.        var data = customers
            .stream()
            .map(OuputFormatter::toRow)
            .collect(Collectors.toList());
2.        data.add(0, 
                 addRow("id", "name", "email", "vip", "status"));

3.        ArrayTableModel model = new ArrayTableModel( 
              data.toArray(Object[][]::new));
4.        TableBuilder table = new TableBuilder(model);
5.        table.addHeaderAndVerticalsBorders( 
              BorderStyle.fancy_light);
6.        return table.build().render(100);
    }

    private static String[] toRow(CustomerResponse c) {
        return addRow(String.valueOf(c.id()),
                c.personInfo().name(),
                c.personInfo().email(),
                String.valueOf(c.detailsInfo().vip()),
                c.status());
    }

    private static String[] addRow(String id, String name, 
        String email, String vip, String status) {
        return new String[] {id, name, email, vip, status};
    }
}

Let's dissect the code line by line:

The list of customers is converter to a list of String[] extrating the customer response fields.
Header is added to the list head (first element of the list).
ArrayTableModel from Spring shell represents the model of the table backed by a row first array. The cosntructor takes a bidimensional array as the data (Each array inside the outer array is a row). The list is transformed to an array.
The TableBuilder configures a table from the model (object holding the table data).
Border is set to the table.
Table is built and rendered with a width of 100.

We are ready to search customers with the new command. All that is needed is to invoke the new command and inform any of the arguments.

And the styled table gives a nice touch to our shell application!

Conclusion

This article has been longer than usual but I think it was worth it. We have learned the basics of Spring Shell and some more advanced features like table formatting.

There will be a second part covering the items 3 and 4 of the Project section. We will also explore other advanced features such as Dynamic Availability.

Full code for the proejct can be found in the github repo here.

If you liked this article, do not hesitate to follow me and received new posts every month.

New RestClient in Spring 6.1

Noe Lopez — Sun, 23 Jul 2023 15:18:05 +0000

Introduction to HTTP clients in Spring

Spring framework has offered two different options to perform http requests:

RestTemplate: It was introduced in Spring 3 over a decade ago. It is an implementation of the Template pattern providing synchronous blocking communication.
WebClient: It was released in Spring 5 as part of Spring WebFlux library. It provides a fluent API and it follows a reactive model.

RestRemplate approach exposed too many HTTP features leading to a big number of overloaded methods. It employs the one thread per request paradigm from the Jakarta Servlet API.

WebClient is the replacement for RestTemplate supporting both synchronous and asynchronous calls. It is part of the Spring Web Reactive project.

Now Spring 6.1 M1 version presents RestClient. A new synchronous http client which works in a similar way to WebClient, using the same infrastructure as RestTemplate.

Setup project

We will be using Spring Boot 3.2 and the Spring web dependency. You can go to the Spring Initializr page and generate a new project selecting Spring Web dependency. With maven the pom.xml will contain

<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-web</artifactId>
</dependency>

That's it. Spring Reactive Web dependency is not needed at all.

Preparing the project

As this is a simple project to get familiar with RestClient, we are going to make http calls to the customer web service from previous articles. Also, the embedded tomcat will be disabled as there is no need to have a web container running. To do so, application.properties file will contain the property

spring.main.web-application-type=none

Then, a CommandLineRunner class will do all the job. The basic structure of the class can be seen below

@Configuration
public class Initializer implements CommandLineRunner {
    private Logger logger = 
        LoggerFactory.getLogger(Initializer.class);

    private ClientProperties properties;

    public Initializer(ClientProperties properties) {
        this.properties = properties;
    }

    public void run(String... args) throws Exception {
    }
}

In the run method is where neccessary objects are constructed in order to interact with the customer endpoint.

Creating a RestClient

To create an instance of the RestClient we have available convinient static methods:

create() method delegates in a default rest client.
create(String url) accepts a default base url.
create(RestTemplate restTemplate) initializes a new RestClient based on the configuration of the given rest template.
builder() allows to customise a RestClient with headers, error handlers, interceptors and more options.
builder(RestTemplate restTemplate) obtain a RestClient builder based on the configuration of the given RestTemplate.

Let's write a RestClient with the builder method to call the customer API.

RestClient restClient = RestClient.builder()
    .baseUrl(properties.getUrl())
    .defaultHeader(HttpHeaders.AUTHORIZATION,
        encodeBasic(properties.getUsername(), 
                    properties.getPassword())
    ).build();

Let's take a closer look at the above code:

baseUrl method is self-explanatory. it sets the base url for the client
defaultHeader allows to set an http header. There is another method named defaultHeaders which takes Consumer as argument for multiple headers. we are setting the Authorization header to pass the credentials.
properties is a simple Configuration Properties class to store the rest API data needed for the requests. And it is constructor-injected in the CommmandLineRunner class.

@Configuration
@ConfigurationProperties(prefix = "application.rest.v1.customer")
public class ClientProperties {
    String url;
    String username;
    String password;
    // getter/setter omitted
}

Three new key-values are added to the application properties file

application.rest.v1.customer.url=http://localhost:8080/api/v1/customers
application.rest.v1.customer.username=user1234
application.rest.v1.customer.password=password5678

Finally, the encodeBasic routine just for reference

private String encodeBasic(String username, String password) {
    return "Basic "+Base64
        .getEncoder()
        .encodeToString((username+":"+password).getBytes());
}

Receiving Data

Next step is to use the client to send http requests and receive the response. RestClient offers methods for each HTTP method. For instance, to search all active customers a GET request must be done. The retrieve method fetches the responce and declares how to extract it.

Let's start with a simple case by getting the full body as String.

String data = restClient.get()
    .uri("?status={STATUS}&vip={vip}","activated", true)
    .accept(MediaType.APPLICATION_JSON)
    .retrieve()
    .body(String.class);

logger.info(data);

The uri method can setup the http parameters to filter by status and vip. The first argument (a String template) is the query string appended to the base url defined in the RestClient. The second argument are uri variables (varargs) of the template.

We also specify the Media type as JSON. The output is displayed in the console:

[{"id":6,"status":"ACTIVATED","personInfo":{"name":"name 6 surname 6","email":"organisation6@email.com","dateOfBirth":"19/07/1976"},"detailsInfo":{"info":"Customer info details 6","vip":true}}]

What if we need to check the response status code or response headers? No worries, the method toEntity returns a ResponseEntity.

ResponseEntity response = restClient.get()
    .uri("?status={STATUS}&vip={vip}","activated", true)
    .accept(MediaType.APPLICATION_JSON)
    .retrieve()
    .toEntity(String.class);

logger.info("Status " + response.getStatusCode());
logger.info("Headers " + response.getHeaders());

Converting JSON

RestClient can also convert a response body in JSON format. Spring will automatically register by default MappingJackson2HttpMessageConverter or MappingJacksonHttpMessageConverter if Jackson 2 library or Jackson library are detected in the classpath. But you can register your own message converters and override the default settings.

In our case, the response can be directly converted to a record. For example, to retrieve a particular customer from the API:

CustomerResponse customer = restClient.get()
    .uri("/{id}",3)
    .accept(MediaType.APPLICATION_JSON)
    .retrieve()
    .body(CustomerResponse.class);

logger.info("Customer name: " + customer.personInfo().name());

And output to extract the customer name

Customer name: name 3 surname 3

To search for customers we just need to use the List class as demonstrated in the following code

List<CustomerResponse> customers = restClient.get()
    .uri("?status={STATUS}&vip={vip}","activated", true)    
    .accept(MediaType.APPLICATION_JSON)
    .retrieve()
    .body(List.class);

logger.info("Customers size " + customers.size());

The record classes for the customer response are shown for reference

public record CustomerResponse(
    long id, 
    String status, 
    CustomerPersonInfo personInfo, 
    CustomerDetailsInfo detailsInfo) {}

public record CustomerPersonInfo(
    String name, String email, String dateOfBirth) {}

public record CustomerDetailsInfo(String info, boolean vip) {}

Posting Data

To send a post request just call the post method. The next code snippet creates a new customer.

CustomerRequest customer = new CustomerRequest(
                "John Smith",
                "john.smith@mycompany.com",
                LocalDate.of(1998, 10, 25),
                "Customer detailed info here",
                true
);

ResponseEntity<Void> response = restClient.post()
                .accept(MediaType.APPLICATION_JSON)
                .body(customer)
                .retrieve()
                .toBodilessEntity();

if (response.getStatusCode().is2xxSuccessful()) {
    logger.info("Created " + response.getStatusCode());
    logger.info("New URL " + response.getHeaders().getLocation());
}

The response code confirms the customer was created successfully

Created 201 CREATED
New URL http://localhost:8080/api/v1/customers/11

To verify the customer was added, the above URL can be retreived via postman

{
    "id": 11,
    "status": "ACTIVATED",
    "personInfo": {
        "name": "John Smith",
        "email": "john.smith@mycompany.com",
        "dateOfBirth": "25/10/1998"
    },
    "detailsInfo": {
        "info": "Customer detailed info here",
        "vip": true
    }
}

Of course it could be fetched using the RestClient with code similar to the previous section.

The record class for the customer request is shown for reference

public record CustomerRequest(
    String name,
    String email,
    LocalDate dateOfBirth,
    String info, 
    Boolean vip) { }

Deleting Data

Making an HTTP delete request to try to remove a resource is as simple as invoking the delete method.

ResponseEntity<Void> response = restClient.delete()
    .uri("/{id}",2)
    .accept(MediaType.APPLICATION_JSON)
    .retrieve()
    .toBodilessEntity();

logger.info("Deleted with status " + response.getStatusCode());

It is worth mentioning that the response body will be empty if the operation succeeded. For this situation the method toBodilessEntity comes in handy. The customer id to be deleted is passed as uri variables.

Deleted with status 204 NO_CONTENT

Handling Errors

What happens if we try to delete or retrieve a non-existing customer? The customer endpoint will return a 404 error code along with a message details. However, RestClient will throw a subclass of RestClientException whenever a client error status (400-499) or server error status (500-599) are received.

To define our custom exception handlers there are two options that work at different levels:

In the RestClient with defaultStatusHandler method (for all http request sent with it)
For each http request with the onstatus method after the call to retreive method (this method returns a ResponseSpec interface).

The first is presented in this code snippet

RestClient restClient = RestClient.builder()
    .baseUrl(properties.getUrl())
    .defaultHeader(HttpHeaders.AUTHORIZATION,
                   encodeBasic(properties.getUsername(), 
                   properties.getPassword()))
    .defaultStatusHandler(
        HttpStatusCode::is4xxClientError,
        (request, response) -> {
             logger.error("Client Error Status " + 
             response.getStatusCode());
             logger.error("Client Error Body "+new 
                 String(response.getBody().readAllBytes()));
    })
    .build();

And the console after running the delete command line runner:

Client Error Status 404 NOT_FOUND
Client Error Body {"status":404,"message":"Entity Customer for id 2 was not found.","timestamp":"2023-07-23T09:24:55.4088208"}

The other option is implementing the onstatus method for the delete operation. It takes precedence over the RestClient defaultt handler behaviour. Hence, it is overriden as proved in the below code lines

ResponseEntity response = restClient.delete()
    .uri("/{id}",2)
    .accept(MediaType.APPLICATION_JSON)
    .retrieve()
    .onStatus(HttpStatusCode::is4xxClientError,
         (req, res) -> 
         logger.error("Couldn't delete "+res.getStatusText())
    )
    .toBodilessEntity();

    if (response.getStatusCode().is2xxSuccessful())
        logger.info("Deleted with status " + 
                     response.getStatusCode());

Now the message in the console will be

 Couldn't delete Not Found

Exchange Method

The exchange method is useful for situations where the response must be decoded differently depending on the response status. Status handlers are ignored when the exchange method is employed.

In this fictitious sample code, the response is mapped to an entity based on the status

SimpleResponse simpleResponse = restClient.get()
    .uri("/{id}",4)
    .accept(MediaType.APPLICATION_JSON)
    .exchange((req,res) -> 
        switch (res.getStatusCode().value()) {
            case 200 -> SimpleResponse.FOUND;
            case 404 -> SimpleResponse.NOT_FOUND;
            default -> SimpleResponse.ERROR;
        }
    );

Summary

In this article we have covered the main features of the new RestClient shipped with Spring 6.1 M1+. Now you should be in a position to perform the most common tasks when calling Rest APIs as well as customise error handling and use the low level http request/response with exchange.

As you can see this new API is easier to manage than the old RestTemplate. It also avoids adding the reactive web libraries in case your project only uses the WebClient API.

Thanks for reading the article and stay tuned for more articles on Java and Spring.

Spring Data JPA custom repositories

Noe Lopez — Sun, 09 Jul 2023 20:30:38 +0000

Introduction

Spring Data JPA offers the JpaRepository interface which provides CRUD/List/Paging/Sorting capabilities. Then, query methods can be defined by:

Query derived directly from method name. For example

public List<Customer> findTop5ByStatusOrderByDateOfBirthAsc( 
                         Customer.Status status);

Defining manually a query. This can be done using the @Query annotation

@Query("""
        SELECT c FROM Customer c 
        WHERE (c.status = :status or :status is null) 
          and (c.name like :name or :name is null) """)
    public List<Customer> findCustomerByStatusAndName(
            @Param("status") Customer.Status status,
            @Param("name") String name);

New Java Text Blocks feature improves readability and code looks really clean.

However, there are situations in which neither of the above options fit our needs. For instance, if the method name involves a lot of fields, the method name can become unreadable. Or if the query is built based on certain criteria, multiple method names must be used for every combination. On the other hand, the @Query annotation is not suitable for dinamic queries. We may end up with performance issues if it is needed to check for null for many fields for example.

In these cases, we need to write our custom implementation for repository methods.

Customizing Repositories

Spring allows to create repositories with custom functionality. The steps to achieve this are:

The first thing to do is to define a fragment interface with the specific methods of the custom repo.
Implement the interface by providing methods functionality.
The JPA repository interface of the Entity must extend the custom interface.

Case study:
Let's assume we have two entities related as shown below

@Entity
public class Customer {

    @Id
    @SequenceGenerator(name = "customer_id_sequence", sequenceName 
                    = "customer_id_sequence", allocationSize = 1)
    @GeneratedValue(strategy = GenerationType.SEQUENCE, generator 
                    = "customer_id_sequence")
    private Long id;
    private String name;
    private String email;
    private LocalDate dateOfBirth;

    @OneToOne(mappedBy = "customer", optional = false ,fetch = 
              FetchType.LAZY, cascade = CascadeType.ALL)
    @PrimaryKeyJoinColumn()
    private CustomerDetails details;

    public CustomerDetails getDetails() {
        return details;
    }

    public void setDetails(CustomerDetails details) {
        this.details = details;
        details.setCustomer(this);
    }
...
}

The customer class has been used in previous posts and it contains just a few members.

@Table(name = "CUSTOMER_DETAILS")
@Entity(name = "CustomerDetails")
public class CustomerDetails {
    @Id
    @Column(name = "customer_id")
    private Long id;
    @Column
    private boolean  vip;
    @Lob
    @Column
    private String info;
    @Column
    private LocalDate createdOn;
    @OneToOne(fetch = FetchType.LAZY)
    @MapsId
    @JoinColumn(name = "customer_id" )
    @JsonIgnore
    private Customer customer;
...
}

Customer details is part of a bidirectional one-to-one relationship with Customer. It adds a few more fields and separates from the parent entity. This model will scale better if Customer is subject to high write operations.

Now, let's write our customer repo so that we can have total control on the new method. Our interface will have one method to search Customers based on all fields including CustomerDetails fields.

public interface CustomizedCustomerRepository {
    public List<Customer> findByAllFields(Customer customer);
}

Next step is to implement the interface. There are two things to take into account here:

The name of the class implementing the interface must be the fragment interface name followed by Impl postfix (Postfix can be changed using the annotation @EnableJpaRepositories)
The implementation is not bound to Spring JPA. This gives the ability to use directly EntityManager or Spring JdbcTemplate. Even delegating to 3rd party library.

In our case, the will use the JPA Criteria API to build a dinamic query and filter by name, status, vip and info fields. Criteria API permits to create queries in a programmatically way.

Code is shown below

public class CustomizedCustomerRepositoryImpl implements 
    CustomizedCustomerRepository{

    @PersistenceContext
    private EntityManager entityManager;

    public List<Customer> findByAllFields(Customer customer) {
        CriteriaBuilder cb = entityManager.getCriteriaBuilder();
        CriteriaQuery<Customer> query = 
            cb.createQuery(Customer.class);
        Root<Customer> root = query.from(Customer.class);

        query.select(root).where(buildPredicates(customer,cb, 
                                                 root));

        return entityManager.createQuery(query).getResultList();
    }

    private Predicate[] buildPredicates(Customer customer, 
        CriteriaBuilder cb, Root<Customer> root) {
        List<Predicate> predicates = new ArrayList<>();
        var details = (Join<Object, Object>)root.fetch("details");

        if (Objects.nonNull(customer.getName()))
            predicates.add(cb.like(root.get("name"), 
                "%"+customer.getName()+"%"));
        if (Objects.nonNull(customer.getStatus()))
            predicates.add(cb.equal(root.get("status"), 
                customer.getStatus()));
        if (Objects.nonNull(customer.getDetails().getInfo()))
            predicates.add(cb.equal(details.get("info"), 
                customer.getDetails().getInfo()));
        if (Objects.nonNull(customer.getDetails().isVip()))
            predicates.add(cb.equal(details.get("vip"), 
                customer.getDetails().isVip()));

        return predicates.toArray(new Predicate[0]);
    }
}

Lets take a look at some of the lines is the above code. First the entity manager is injected as we will make use of the Criteria API. Then, the necessary objects to construct the query object are instanciated.

Line

Root<Customer> root = query.from(Customer.class);

creates a root of type Customer for the from clause.

Line

query.select(root).where(buildPredicates(customer,cb,root));

sets the root for the select statement and the where clause. The where clause is generated in the buildPredicates which returns a array of type Predicate. This is the input param type of the where method (varAgrs is compatible with array).

Finally, line

return entityManager.createQuery(query).getResultList();

executes the query and returns the results as a list. Note that it is needed neither casting nor mapping as JPA knows the types from the Entities.

A couple of comments about the buildPredicates method. It constructs a join fecth in line

var details = (Join<Object, Object>) root.fetch("details");

The join fetch will add the CustomerDetails fields to the select. It also prevents from the N + 1 query problem by joining the two tables (even if the one to one association is lazy).

Then, each predicate is added to a list if the corresponding field is not empty. Eventually, the list is converted to an Array.

The last step is to extend the individual repository from the customised.

public interface CustomerRepo extends 
    JpaRepository<Customer, Long> , CustomizedCustomerRepository {
...
}

All is done now. Lets call the method from the Controller layer.

Calling the method

We will invoke the method from the CustomerController. Any of the four fields can be passed into the method via GET request params. The code is displayed in the following snippet

public class CustomerController {
...
    @GetMapping
    public List<CustomerResponse> findCustomers(
        @RequestParam(name="name", required=false) String name,
        @RequestParam(name="status", required=false) 
            Customer.Status status,
        @RequestParam(name="info", required=false) String info,
        @RequestParam(name="vip", required=false) Boolean vip) {
        return customerRepo
                .findByAllFields(Customer.Builder
                        .newCustomer()
                        .name(name)
                        .status(status)
                        .withDetails(info, Boolean.valueOf(vip))
                        .build())
                .stream()
                .map(CustomerUtils::convertToCustomerResponse)
                .collect(Collectors.toList());
    }
...

All parameters are optional. If they are not informed they will not be part of the where clause. The customer object is created via Builder pattern with fluent api. Search results are converted to a DTO object CustomerResponse of type record. This way the representation of the data sent back to the client can have its own structure and not tied to the entities model.

Time to run a couple of tests. First, a GET request without params

http://localhost:8080/api/v1/customers

The generated sql code selects all columns for both entities and joins on the shared primary key. There is no where clause appended.

select c1_0.id,c1_0.date_of_birth,d1_0.customer_id, 
       d1_0.created_on,d1_0.info,d1_0.vip,c1_0.email, 
       c1_0.name,c1_0.status 
from customer c1_0 
join customer_details d1_0 on c1_0.id=d1_0.customer_id

The output returns all Customers and its Customer Details. The DTO object is just a record with two nested record components.

[
    {
        "id": 1,
        "status": "DEACTIVATED",
        "personInfo": {
            "name": "name 1 surname 1",
            "email": "organisation1@email.com",
            "dateOfBirth": "03/01/1982"
        },
        "detailsInfo": {
            "info": "Customer info details 1",
            "vip": false
        }
    },
 ...
    {
        "id": 9,
        "status": "ACTIVATED",
        "personInfo": {
            "name": "name 9 surname 9",
            "email": "organisation9@email.com",
            "dateOfBirth": "27/09/1998"
        },
        "detailsInfo": {
            "info": "Customer info details 9",
            "vip": false
        }
    }
]

The second URL will have three parameters

http://localhost:8080/api/v1/customers?vip=true&status=activated &name=%name%

SQL code contains the where clause with the expected filters for the input params. This is demostrated in the below log entries

select c1_0.id,c1_0.date_of_birth,d1_0.customer_id, 
       d1_0.created_on,d1_0.info,d1_0.vip,c1_0.email, 
       c1_0.name,c1_0.status 
from customer c1_0 
join customer_details d1_0 on c1_0.id=d1_0.customer_id 
where c1_0.name like ? escape '' and c1_0.status=? and d1_0.vip=?

binding parameter [1] as [VARCHAR] - [%name%]
binding parameter [2] as [INTEGER] - [1]
binding parameter [3] as [BOOLEAN] - [true]

The output sent back from the Controller holds the only Customer matching the filters

[
    {
        "id": 6,
        "status": "ACTIVATED",
        "personInfo": {
            "name": "name 6 surname 6",
            "email": "organisation6@email.com",
            "dateOfBirth": "18/06/1992"
        },
        "detailsInfo": {
            "info": "Customer info details 6",
            "vip": true
        }
    }
]

Conclusion

This article explained how to create a custom individual repository in Spring Data. This comes in handy when we need to enrich a repository with custom functionality and the options offered by the JpaRepository are not enough.

The code used in the article can be found in the github repository linked here

That is all for this week. New content related to Spring and Java will be publish sortly. Hope you enjoyed the read.

Spring REST working with enums

Noe Lopez — Sun, 25 Jun 2023 17:12:49 +0000

Overview

Enums are a powerful feature of the java language. They were introduced in Java 5. They are a special type that allows us to declare a set of predefined constants. They improve readability, provide compile time checks and are type safe.
In this article, we are going to extend the functionality of the customer endpoint of previous articles. What we are going to do is to add a new status field to the Customer class. Then, we will be able to persist and search based on the status.

The article is divided into two main sections:

Mapping Enums with JPA.
Request Parameters and Enums in Spring.

Without any further delay, lets move to the next section and write some code.

Mapping Enums with JPA

The simplest way to map an enum with a database column is using the @Enumerated annotation from the jakarta.persistence package. This annotation accepts two types of mapping. The default is ORDINAL which means that the enum will be stored as an integer. The other type, STRING, stores the enum as enum value/constant.

Let's see this is action. The steps are:

Create the enum. It is good practice to use capitals for the values/constants.
Add a new field to the Customer Entity.
Annotate the ne field with @Enumerated.



@Entity
public class Customer {
    public enum Status { ACTIVATED, DEACTIVATED, SUSPENDED };
...
    @Enumerated(EnumType.ORDINAL)
    private Status status;
...
}

The next picture shows records saved for the Customer Entity in the database

The ORDINAL values begin with 0. The representation of each value in the enum is then

0 -> ACTIVATED
1 -> DEACTIVATED
2 -> SUSPENDED

Lets see what happens then using the STRING type mapping in our Entity



    @Enumerated(EnumType.STRING)
    private Status status;

The data in the customer table now is presented in the following picture

Now the column is easier to interpert. On the other hand, there is duplication. This could be a problem if the volume of data is high.

These two out of the box solutions work well but they are limited. What if we want to use different values in the table or even different database types (other than INT or VARCHAR)?

Using a @Convert for custom mapping

The @Convert annotation converts an entity field to a specific value in the database. It requires a Converter class that perfoms the transformation from entity attribute state into database column representation and back again.

For our case, the values in the column will be as follows ACTIVATED -> 1, DEACTIVATED -> 2 and SUSPENDED -> 3.

The steps to accomplish this are described here:

Add code property to the Status enum. This will be the value to be in the table column.
Create a converter class to make the conversation.
Annotate the entity field.

The change in the enum is straghtforward.



public enum Status {
    ACTIVATED(1), DEACTIVATED(2), SUSPENDED(3);

    int statusId;

    private Status(int statusId) {
        this.statusId = statusId;
    }

    public int getStatusId() {
        return statusId;
    }
};

The converter class must implement the interface AttributeConverter. It is a generic interface taking two parameters and declaring two methods. The first type is the entity attribute type and the second is the type of the data to which the attribute will be converted.

Here we create the converter for the Status enum



@Converter
public class CustomerStatusConverter implements AttributeConverter<Customer.Status, Integer> {
    @Override
    public Integer convertToDatabaseColumn(Customer.Status status) 
   { 
        return status.getStatusId();
   }

    @Override
    public Customer.Status convertToEntityAttribute(Integer 
        statusId) {
        return Arrays.stream(Customer.Status.values())
                .filter(s -> s.getStatusId() == statusId)
                .findFirst()
                .orElseThrow(IllegalArgumentException::new);
    }
}

Finally, the entity attribute is annotated with @Convert indicating the converter class.



    @Convert(converter = CustomerStatusConverter.class)
    private Status status;

Optionally, the annotation @Converter can take the property autoApply. Its default value is false, hence it is disabled. When it is set to true, Spring will automatically apply the converter to the entity attribute of the type specified in the first param.

That's is all needed. To test it, we are going to persist a few Customer using a DataInitalizer.

Initializing the Customer table

We will fill up the customer table with a few random records using a CommandLineRunner. This is a functional interface declaring a run method. Spring will execute the run method after all beans are instanciated and application context is fully loaded but before application is up.



@Configuration
public class DataInitializer implements CommandLineRunner {
    private CustomerRepo customerRepo;

    public DataInitializer(CustomerRepo customerRepo) {
        this.customerRepo = customerRepo;
    }

    @Override
    public void run(String... args) {       
        IntStream.rangeClosed(1,9)
            .forEach(this::createRandomCustomer);
    }

    public void createRandomCustomer(int id) {
        Customer customer = new Customer();
        customer.setId((long)id);
        customer.setName("name "+id+" surname "+id);
        customer.setEmail("organisation"+id+"@email.com");
        customer.setDateOfBirth(LocalDate.of(1980 + 2*id,id % 12, 
            (3*id) % 28));
        customer.setStatus(switch (id % 3) {
            case 0 -> Customer.Status.ACTIVATED;
            case 1 -> Customer.Status.DEACTIVATED;
            case 2 -> Customer.Status.SUSPENDED;
            default -> throw new IllegalStateException("Unexpected 
                       value: " + id % 3);
        });

        customerRepo.save(customer);
    }
}

Now there are a few records available in the table with the values we want in the STATUS column.

Adding a find method to the Repository

The last step before moving to the web layer is to create a new method to filter by name and status in the CustomerRepo class. Spring Data JPA offers the @Query annotation to execute JPQL or native SQL. This comes in handy when Spring Data JPA query methods do not provide a solution for the search criteria.

We want to search by status and name plus ignore nulls in a single method. Then, with the help of @Query we can write the code



public interface CustomerRepo extends JpaRepository<Customer, Long> {
    @Query("""
        SELECT c FROM Customer c 
        WHERE (c.status = :status or :status is null) 
          and (c.name like :name or :name is null) """)
    public List<Customer> findCustomerByStatusAndName(
            @Param("status") Customer.Status status,
            @Param("name") String name);
}

All is set now to go to the web layer.

Request Parameters and Enums in Spring

The idea is to add filters so that clients of the API can search customers by name and status. The filters will be passed as request parameters in the GET HTTP request. Name can accept any string data. Conversely, status will be restricted to the enum predefined values.

Spring MVC supports enums as parameters as well as path variables. Simply putting the params in the method signature of the controller will be enough



@RestController
@RequestMapping("api/v1/customers")
public class CustomerController {
...
    @GetMapping
    public List<CustomerResponse> findCustomers(
        @RequestParam(name="name", required=false) String name,
        @RequestParam(name="status", required=false) 
            Customer.Status status) {
        return customerRepo
            .findCustomerByStatusAndNameNamedParams(status, name)
            .stream()
            .map(CustomerUtils::convertToCustomerResponse)
            .collect(Collectors.toList());
   }
...
}

The controller is ready to receive the parameters. Let's suppose the URL http://localhost:8080/api/v1/customers?status=ACTIVATED is invoked. Spring will attempt to convert the value coming from the http parameter status to a valid status enum.

In this case both values match, the http param and the status enum value. Hence, the response back from the API is the list of activated customers



[
    {
        "id": 3,
        "name": "name 3 surname 3",
        "email": "organisation3@email.com",
        "dateOfBirth": "09-03-1986",
        "status": "activated"
    },
    {
        "id": 6,
        "name": "name 6 surname 6",
        "email": "organisation6@email.com",
        "dateOfBirth": "18-06-1992",
        "status": "activated"
    },
    {
        "id": 9,
        "name": "name 9 surname 9",
        "email": "organisation9@email.com",
        "dateOfBirth": "27-09-1998",
        "status": "activated"
    }
]

What happens if the parameter it does not match any enum value or it is invalid? Well, if they do not match or it is invalid, Spring will throw a ConversionFailedException.

Let's demostrate it with the following calls. First, URL is http://localhost:8080/api/v1/customers?status=activated

The response is not what we could have expected



{
    "status": 400,
    "message": "There was a error converting the value activated 
                to type Status. ",
    "timestamp": "2023-06-24T19:33:26.6583667"
}

The second URL will be http://localhost:8080/api/v1/customers?status=DELETED

The response from the server could be accepted in this scenario as the status does not exist.



{
    "status": 400,
    "message": "There was a error converting the value DELETED to 
                type Status.",
    "timestamp": "2023-06-24T19:08:49.6261182"
}

The reason for this behaviour is because Spring makes use of the class StringToEnumConverterFactory. The convert method delegates to the Enum valueOf method which throws a IllegalArgumentException if no enum constant is found.



@Nullable
public T convert(String source) {
    return source.isEmpty() ? null : Enum.valueOf(this.enumType, 
       source.trim());
}

We would like to alter this behaviour and permit to filter using lower cases. Similar to what was done in the JPA section, we will have to write a custom converter.

Creating a Spring Converter

Adding a custom converter is pretty easy. Just have to implement the Converter interface and override its convert method. Convert method takes two generic types, the type of the source and the type of the outcome.



@Component
public class StringIgnoreCaseToEnumConverter implements 
   Converter<String, Customer.Status> {

    @Override
    public Customer.Status convert(String source) {
        return Customer.Status.valueOf(source.toUpperCase());        
    }
}

The above method takes the incoming String. And it is transformed to uppercase prior to return the enum constant via Enum.valueOf. Therefore, the string "activated" will become "ACTIVATED". Consequently, it will return the enum constant Status.ACTIVATED.

Let's re-test now the previous URL and check the results. Calling the URL http://localhost:8080/api/v1/customers?status=activated generates the output we wanted



[

    {

        "id": 3,

        "name": "name 3 surname 3",

        "email": "organisation3@email.com",

        "dateOfBirth": "09-03-1986",

        "status": "activated"

    },

 ...

]

Conclusion

This article has explained in detailed how to use enums in the persistence layer and the web layer. Here is a summary of the main points of it.

For the persistence layer, JPA provides the ORDINAL and STRING options out of the box. If this does not meet the requirements of the application, custom converters provide total control on how the mapping is done.

For the web layer, Spring MVC comes with enums support for request params and path variable. It converts String to enums automatically if there is exact matching. This behaviour can be changed by implementing a converter.

Code can be found in the repository by clicking in this link

Hope you enjoyed reading it. Stay tuned for more articles covering Spring and Java topics!

Spring Rest i18n Messages

Noe Lopez — Sun, 28 May 2023 17:01:23 +0000

What is internationalization?

In this article we are going to add Internationalization Support to our Spring Rest API customer endpoint. Internationalization is the process of creating an application that can be adapted to different languages and regions. What we are trying to achieve is to add support for messages in multiple languages. Therefore, clients with different language preferences will get specific messages based on the language.

Default Auto Configuration

By default, Spring Boot looks for the presence of a messages resource bundle at the root of the classpath. Autoconfiguration applies when default properties file is available (messages.properties by default). Spring boot searches for message files in the scr/main/resources folder. This can be changed with the property as show below:



spring.messages.basename=messages,config.i18n.messages

If the properties file is found, Spring Boot configures a MessageSource bean for resolving messages, with support for the parameterization and internationalization of such messages.

Spring Boot also configures a LocaleResolver implementation that looks for locales. The default implementation is AcceptHeaderLocaleResolver which inspects the Accept-Language header to match the locale for the request.

Create Messages files

Messages are stored in key-value pairs in messages_xx.properties files. The xx denotes the language code (normally a two letter code like en, fr, it...). Also the region/country (usually a two letter code) can be added for more fine grained control on locales.

For our application, two properties files will be created in the resources folder. The first one is the fallback messages.properties file with English content. It will be used when no locale is specified in the request. The second one is to support Spanish language and will be named messages_es.properties

The keys are the same in each localised file. The values or descriptions of them are particular to each language. For instance, the English content in the fallback file contians the below entries



customer.name.required=Name is required.
customer.name.size=Name must be at least 3 characters and at most 20 characters.
customer.name.invalid=Name can only contain letters and spaces.
customer.email.required=Email is required.
customer.email.invalid=Email is invalid. 
customer.dob.past=Date of Birth must be in the past.

and the Spanish language file



customer.name.required=Nombre es obligatorio.
customer.name.size=Nombre debe tener al menos 3 caracteres y 20 a lo sumo.
customer.name.invalid=Nombre solo puede estar formado por letras y espacios.
customer.email.required=Email es obligatorio.
customer.email.invalid=Email es invalido. 
customer.dob.past=Fecha de nacimiento debe estar en el pasado.

Using the keys in the code

Now the message keys are ready to be used in our application. Customer controller validates the customer request when sending a POST or PUT request. As the validations are the same for both cases, let's look at the scenario where a customer is updated



@PutMapping("{customerId}")
public ResponseEntity<Customer> updateCustomer( 
    @PathVariable("customerId") Long id, 
    @Valid @RequestBody CustomerRequest customerRequest)  {
    Customer customer = customerRepo.findById(id)
        .orElseThrow(() -> new EntityNotFoundException(id, 
         Customer.class));

    Customer updatedCustomer = CustomerUtils.convertToCustomer( 
        customerRequest);
    updatedCustomer.setId(id);
    customerRepo.save(updatedCustomer);

    return ResponseEntity.ok(updatedCustomer);
}

In the CustomerRequest class, the key is placed in the message attribute of the constrains.



public record CustomerRequest(
    @NotBlank(message = "{customer.name.required}")
    @Size(min = 3, max = 20, message = "{customer.name.size}")
    @Pattern(regexp = "[a-zA-Z\\s]+", message = " 
    {customer.name.invalid}") String name,
    @NotBlank(message = "{customer.email.required}") 
    @Email(message = "{customer.email.invalid}") String email,
    @Past(message = "{customer.dob.past}") LocalDate dateOfBirth) {}

Finally, the Adviser to handle validation exceptions



@ExceptionHandler(MethodArgumentNotValidException.class)
@ResponseStatus(HttpStatus.BAD_REQUEST)
RestErrorResponse handleException(MethodArgumentNotValidException ex) {
    String message = ex.getFieldErrors()
        .stream()
        .map(e -> e.getDefaultMessage())
        .reduce("Errors found:", String::concat);
        return new RestErrorResponse( 
        HttpStatus.BAD_REQUEST.value(), message, 
        LocalDateTime.now());
}

Now it is time to test. Our first attempt will be to make a put request without informing the Accept-Language header as shown in the next figure

It falls back to the messages.properties file, hence errors are displayed in English. The second request will send the Accept Language header with value es. The result can be viewed below

The errors are coming back in Spanish now. If more languages need to be supported by our application, then it is as simple as creating a new message file for the desired locale.

One thing that is still hardcoded in English is the "Errors found:" literal prefix as part of the error message. Let's move it to the properties files in the next section.

MessageSource class

The class MessageSource from org.springframework.context provides methods to receive translated messages. SpringBoot will create this bean for us. Then, it is injected in the classes where messages will be used. In our case, it is constructor-injected in the ControllerAdviser as below



@RestControllerAdvice
public class GlobalExceptionHandler {
    MessageSource messageSource;
    public GlobalExceptionHandler(MessageSource messageSource) {
        this.messageSource = messageSource;
    }
...
}

Next, the key-value messages will be added to the language specific property files. For the English language



errors.found=Errors found:

And for Spanish



errors.found=Errores encontrados :

Last step is to replace the hardcoded value by the key in the handler method.



@ExceptionHandler(MethodArgumentNotValidException.class)
@ResponseStatus(HttpStatus.BAD_REQUEST)
RestErrorResponse handleException(MethodArgumentNotValidException ex, Locale locale) {
    String message = ex.getFieldErrors().stream()
        .map(e -> e.getDefaultMessage())
        .reduce(messageSource.getMessage("errors.found", null, 
             locale), String::concat);
        return new RestErrorResponse( 
           HttpStatus.BAD_REQUEST.value(), message, 
           LocalDateTime.now());
}

Note that the locale can be passed as a parameter in the Exception Handler. As expected the value is correctly translated when re-running the tests. The following response is returned when informing the es value in the Accept Language header



{
    "status": 400,
    "message": "Errores encontrados : Nombre debe tener al menos 3 caracteres y 20 a lo sumo.Fecha de nacimiento debe estar en el pasado.Email es invalido. ",
    "timestamp": "2023-05-27T19:10:57.0064035"
}

Custom Exception messages

In this section, we will translate custom exception messages. In our application, EntityNotFoundException is thrown if the customer to be updated does not exist. The message is hardcoded in the Exception class. However, we want this message to be available for the supported languages.

First step is to add a new key for the error when this exception is raised.



--- english file
entity.notFound=Entity {0} for id {1} was not found.

--- spanish file
entity.notFound=Entity {0} for id {1} was not found.

The values {0} and {1} are placeholders for parameters. The first parameter will go into position {0} and the second paramerter will be applied to {1}.

Second step is to declare a new Exception handler method for it. Again, MessageSource class provides the method to get the message. Parameters are passed in the second argument of getMethodMessage as a Object array. Both values are stored in the custom Exception and are set at instatiation time.



 @ExceptionHandler(EntityNotFoundException.class)
 @ResponseStatus(HttpStatus.BAD_REQUEST)
RestErrorResponse handleException(EntityNotFoundException ex, 
    Locale locale) {
    return new RestErrorResponse(HttpStatus.BAD_REQUEST.value(),
        messageSource.getMessage(
        "entity.notFound", 
        new Object[]{ ex.getTheClassName(), ex.getId()} , locale),
        LocalDateTime.now());
    }

The custom exception code is below



public class EntityNotFoundException extends RuntimeException {
    private Long id;
    private Class theClass;
    public EntityNotFoundException(@NotNull Long id,@NotNull Class 
        theClass) {
        super(theClass.getName()+" "+id+" not found!");
        this.id = id;
        this.theClass = theClass;
    }

    public Long getId() {
        return id;
    }

    public String getTheClassName() {
        return theClass.getSimpleName();
    }
}

Time to test. Let's try to update a non-existing customer.

Conclusion

Let's review the main points of what you have learned in this article:

What is internationalization and how it is enable in SpringBoot.
What are message files.
Create multi language custom constrains error messages.
Create multi language custom exception error messages.

Now you can implement your own multi language solution in your Spring projects. Source code can be found here.

Subscribe for more content on Java and Spring!

Spring Rest Custom Data Validation

Noe Lopez — Sun, 14 May 2023 16:56:15 +0000

Introduction

This short tutorial is a continuation of the previous article Spring Rest/MVC Validation where Bean Validation was introduced.

In this part, we will explore a few more use cases which are quite common in java web apps. You will learn how to do the following validations using JSR 380 from Jakarta EE:

Validate Request Params.
Validate Request Headers.
Validate URL paths.
Create a custom constrain.

Without any further delay, let's start with the first item on the list.

Validate Request Params

We are going to add validation to the DocumentController presented in the Http Client mini series. What we are going to do is implement new functionality to search documents filtered by name. The steps are described here:

Add request param to findDocuments method
Validate min and max length of the name parameter.
Test method via http request.

Updated method code is shown here

@GetMapping
public List<DocumentResponse> findDocuments(@RequestParam
    @Min(value = 3, message = "Name must be at least 3 
                               characters")
    @Max(value = 100, message = "Name must be at most 100 
                                 characters") String name) {
...
}

Calling the endpoint http://localhost:8080/api/v1/documents?name=na will return documents successfully. Why were the validations not checked?
To enable validation at method level it is necessary to annotate the class with @Validated. This indicates Spring that the class is validated at the method level (parameters as well as return types) using AOP techniques. Also, a MethodValidationPostProcessor bean has to be configured in Spring. If you are using SpringBoot, then it is registered automatically and no extra setup is needed.

@Validated
@RestController
@RequestMapping("api/v1/documents")
public class DocumentController {

Once the annotation is applied to the Controller class, method validation will be triggered. The expected output is generated when the name parameter length is less than 3 characters.

{
    "status": 400,
    "message": "findDocuments.name: Name must be at least 3 
     characters",
    "timestamp": "2023-05-13T18:03:25.4586965"
}

JSR 380 provides @Size annotation, a convinient constrain to enforce min and max length. It can be applied to the below types:

CharSequence: length of character sequence is evaluated.
Collection: collection size is evaluated.
Map: map size is evaluated.
Array: array length is evaluated.

Replacing the two annotations with the @Size annotation, the code looks as follows

@GetMapping
public List<DocumentResponse> findDocuments(@RequestParam 
    @Size(min = 3, max = 10, message = "Name must be have at least 
          3 characters and no more than 100.") String name) 
{

Running a new test sends back the response

{
    "status": 400,
    "message": "findDocuments.name: Name must be have at least 3 
characters and no more than 100.",
    "timestamp": "2023-05-13T18:21:11.0345341"
}

Validate Request Headers

The same procedure applies for request headers validation. We can use the annotations from JSR 380. Again, method validation must be enabled so that the constrains declared before the parameter are executed. This step was done in the previous sections.
Lets look at the upload documents use case where the filename header is a mandatory piece of information and must comply with certain rules.

Must not exceed 255 characters.
Must have one dot.
Allow only alphanumeric, numbers, underscore and hyphen.
Extension is alphanumeric between 2 and 4 characters.

This is the perfect case for the @Pattern constrain which employs regular expresions.

@PostMapping()
ResponseEntity<Void> uploadDocument(@RequestBody byte[] data
    ,@RequestHeader("Content-Type") String type
    ,@RequestHeader("fileName") @Pattern(
         regexp = "^([a-zA-Z0-9_-]{2,250})\\.([a-z]{2,4})$", 
         message = "FileName is invalid.") String fileName) {
    ...
}

Sending the filename header value texttxt will cause the app to return

{
    "status": 400,
    "message": "uploadDocument.fileName: FileName is invalid.",
    "timestamp": "2023-05-13T19:18:04.5942804"
}

Validate URL paths

The setup is just as the previous two sections. Then constrains can be placed right after the @PathVariable annotation. Lets add some validations to the download document method. This method extracts the id from the URL into a Long. Ids are positive and cannot exceed a max value.

ResponseEntity<Resource> downloadDocument(@PathVariable
    @Positive(message = "Document id must be positive.")
    @Max(value = 99999999, message = "Document id cannot exceed 
        value 99999999.") Long documentId) {
...
}

Making a http request to http://localhost:8080/api/v1/documents/-1 produces

{
    "status": 400,
    "message": "downloadDocument.documentId: Document id must be 
positive.",
    "timestamp": "2023-05-13T19:44:21.8745695"
}

Creating a custom constrain

In this section, we are going to create a custom constrain which performs custom validation based on our business rules.
As an example (do not this in production), lets suppose we want to only permit to upload a specific set of extensions. We could create our own annotation @AllowedExtensions. The procedure is explained below

Create @AllowedExtensions annotation.
Create AllowedExtensionsConstrainValidator.
Setting the annotation in the parameter.

Create @AllowedExtensions annotation

An annotation is defined with the @interface keyword.

public @interface AllowedExtensions {}

Next is to setup some meta-annotations on top of it. They are needed to speficy some properties and behaviour of our annotation.

@Documented
@Target({ElementType.PARAMETER, ElementType.FIELD})
@Retention(RetentionPolicy.RUNTIME)
@Constraint( 
     validatedBy = AllowedExtensionsConstrainValidator.class)
public @interface AllowedExtensions {}

Lets see what each meta-annotation does in the above code:

@Documented ensures that our custom annotations are shown in the documentation.
@Retention annotation sets the scope of the annotation. How long the annotation will be stored. RetentionPolicy.RUNTIME will keep the annotation in the bytecode (class file) and in runtime (JVM) as the annotation is run by our application.
@Target annotation indicates where it can be applied. In our case, the annotation will be applied on parameters and fields.
@Constraint annotation marks an annotation as being a Jakarta Bean Validation constraint. It defines the validator class implementing our own custom rules to pass the validation.

Next step is to define the attributes the annotation can accept. There will be two attributes, the value containing the extensions and the message in case the validation fails.

@Documented
@Target({ElementType.METHOD, ElementType.FIELD})
@Retention(RetentionPolicy.RUNTIME)
@Constraint(validatedBy = AllowedExtensionsConstrainValidator.class)
public @interface AllowedExtensions {
    public String[] value() default {"pdf","txt","png"};

    public String message() default "File extension must be pdf, 
    txt or png";

    public Class<?>[] groups() default {};

    public Class<? extends Payload>[] payload() default {};
}

Value is a just an String array with the default extensions. Likewise, message defines the default message. Annotation default values have the default clause and can return a value of the same type.
It is also needed two more entries for groups and payload. Groups can group validation constrains together. Payload can give additional information about the validation error such as severity level or error code.

That is all for the annotation AllowedExtensions.

Create AllowedExtensionsConstrainValidator class

This class will perform the validation for our annotation. It has to implement the interface ConstraintValidator and also the isValid method. In this method our own logic for validations are defined.

ConstraintValidator defines the logic to validate a given constraint A for a given object type T. Implementations must comply to the following restriction:

T must resolve to a non parameterized type
or generic parameters of T must be unbounded wildcard types

In our case, the first type is the annotation AllowedExtensions and the second is String which is the input data to be validated (the filename).

Class code is available in the snippet below

public class AllowedExtensionsConstrainValidator implements ConstraintValidator<AllowedExtensions, String> {
    private String[] extensions;

    @Override
    public void initialize(AllowedExtensions allowedExtensions) {
        this. extensions = allowedExtensions.value();
    }

    @Override
    public boolean isValid(String fileName, 
        ConstraintValidatorContext constraintValidatorContext) {
        return Arrays.asList(extensions). 
        contains(fileName.substring(fileName.lastIndexOf(".")) 
        .toLowerCase());
    }
}

Initialize method initializes the validator in preparation for isValid. The parameter is the constrain annotation. The allowed extensions are assigned to a member variable so that they are available when the isValid method is invoked.

The isValid method verifies the file extension is on the list of valid extensions.

We are done with the constrain validator. Now it is time to use the validation.

Setting the annotation in the parameter

At this point, all that is needed is to place the annotation in the filename parameter.

@PostMapping()
ResponseEntity<Void> uploadDocument(@RequestBody byte[] data,
    @RequestHeader("Content-Type") String type,
    @RequestHeader("fileName") 
    @Pattern(regexp = "^([a-zA-Z0-9_-]\{2,200})\\.([a-z]{3,4})$", 
             message = "FileName is invalid.")
    @AllowedExtensions String fileName) {
...
}

Time to test it out sending text.doc in the header fileName of the http request.

{
    "status": 400,
    "message": "uploadDocument.fileName: File extension must be 
pdf, txt or png",
    "timestamp": "2023-05-13T22:51:13.6404863"
}

Another test, this time passing the value text.txt. We get back status 201 and the location of the newly created file.

201
http://localhost:8080/api/v1/documents/4

Summary

In this tutorial, we have learned how to validate request params, request headers, dynamic paths and create our own custom constrains. Now you have the tools to start using Bean Validation in any layer of your application and benefit from its clean and simple approach.

As usual the source code used in this article can be found in the github repo by cliking here

Spring Rest/MVC Validation

Noe Lopez — Sun, 07 May 2023 20:03:30 +0000

Introduction

Validating data is a common task that needs to be performed in almost all applications, from the presentation layer to the persistence layer. Java provides a specification (JSR 380) that makes it easy for programmers to validate application constraints. Some of the main features of this specification are:

The ability to express constraints on object models via annotations.
The ability to write custom constraints in an extensible way.
APIs to validate objects and object graphs.
APIs to validate parameters and return values of methods and constructors.
reports the set of violations (localized)
Support for Optional and container elements (generic containers, e.g. List, Map or Optional).

Bean Validation

Bean validation was defined in JSR 380 and its goal is to promote code reusability and decoupling validation logic from other layers of an application. In this tutorial, we will be using Bean Validation 3.0 with anotations. It's worth noting that there is also an XML mapping definition available, for more information visit the offical documentation). The only change in version 3.0 is the package change from javax to jakarta as it is part of Jakarta EE now.

The specification defines a small set of built-in constrains.

Version 1.1 came with : @null, @notnull, @AssertTrue, @AssertFalse, @min, @max, @DecimalMin, @DecimalMax, @Size, @Digits, @Past, @Future and @Pattern.

In version 2.0, new constrains were added: @Email, @NotEmpty, @NotBlank, @positive, @PositiveOrZero, @negative, @NegativeOrZero, @PastOrPresent and @FutureOrPresent.

These annotations can be applied to types, fields, methods, constructors, parameters, container elements or other constraint annotations, in case of composition. This is defined in the annotation class code with the @Target annotation.

@Target({METHOD, FIELD, ANNOTATION_TYPE, CONSTRUCTOR, PARAMETER})

The specification defines a metadata model and API. The reference implementation for Bean Validation is Hibernate Validator 8, which should not be confused with the Hibernate ORM project.

Adding Validation to Spring Project

Our project is built with Spring boot 3. To include Bean Validation in it, we need to add the below dependency in pom.xml file.

<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-validation</artifactId>
</dependency>

If we explore inside this starter, we can see that the implementation used by Spring is hibernate-validator.

<dependency>
    <groupId>org.hibernate.validator</groupId>
    <artifactId>hibernate-validator</artifactId>
    <version>8.0.0.Final</version>
    <scope>compile</scope>
</dependency>

Once the project depencies are downloaded, we are in a position to start validating our model objects.

Validation in Action

Let's consider adding validation to the customer/document rest api based on previous articles. For example, let's validate the input received when creating a new customer. In this scenario a POST request must be sent.

@PostMapping
public ResponseEntity<Long> addCustomer( 
   @RequestBody CustomerRequest customerRequest)  {
   Customer customer = CustomerUtils.convertToCustomer( 
                       customerRequest);
   customerRepo.save(customer);

   return ResponseEntity.created(URI.create( 
       CUSTOMER_ENDPOINT_URL+customer.getId() 
   )).build();
}

The data we want to validate comes in the request body and it is mapped to a CustomerRequest object.

public record CustomerRequest(String name, String email, 
                              LocalDate dateOfBirth) {}

Let's look at the constraints to be inforced on the DTO:

The name field is required, must have at least 3 charactes and a maximun of 20 characters. It can contain alpha and space characters.
The email field is required and should be a valid email address.
The dateOfBirth field is required and must be in the past.

Fortunately, these requirements can be met with built-in constrains provided by Bean Validation:

public record CustomerRequest(
    @NotBlank(message = "Name is required.")
    @Size(min = 3, max = 20, message = "Name must be at least 3 
       characters and at most 20 characters.")
    @Pattern(regexp = "[a-zA-Z\\s]+", message = "Name can only 
       contain letters and spaces.") String name,
   @NotBlank(message = "Name is required") @Email(message = "Email 
       is not valid.") String email,
   @Past(message = "Date of Birth must be in the past.") LocalDate 
       dateOfBirth) {
}

In the above code, the validation constraints are defined using annotations. Each constraint is applied to the corresponding field, specifying the validation rules and an error message in case of a validation failure.

There is one more thing to do. To tell Spring to validate the input Object when the method is invoked. Here is where the annotation @Valid is used.

@PostMapping
     public ResponseEntity<Long> addCustomer(@Valid @RequestBody 
         CustomerRequest customerRequest)  {

The @Valid annotation is part of JSR-303 and it is applied to the object that requires validation. It ensures that the whole object graph is validated. Nested objects marked with @Valid will be also validated when the parent object is validated. @Valid does not support group validation (more on this on a future article).

Let's run the application and test the endpoint to make it fail. The body of the POST request is described in the following lines

{
    "name": "Jo",
    "email": "myemail.com",
    "dateOfBirth": "2033-04-05"
}

Now Springboot will trigger the constrains set in the CustomerRequest class using the Hibernate Validator implementation. The response sent back from the server is not quite what we expected.

{
    "status": 400,
    "message": "Validation failed for argument [0] in public org.springframework.http.ResponseEntity<java.lang.Long>dev.noelopez.restdemo1.controller.CustomerController.addCustomer(dev.noelopez.restdemo1.dto.CustomerRequest) with 3 errors: [Field error in object 'customerRequest' on field 'name': rejected value [Jo]; codes [Size.customerRequest.name,Size.name,Size.java.lang.String,Size]; arguments [org.springframework.context.support.DefaultMessageSourceResolvable: codes [customerRequest.name,name]; arguments []; default message [name],20,3]; default message [**Name must be at least 3 characters and at most 20 characters.**]] [Field error in object 'customerRequest' on field 'dateOfBirth': rejected value [2033-04-05]; codes [Past.customerRequest.dateOfBirth,Past.dateOfBirth,Past.java.time.LocalDate,Past]; arguments [org.springframework.context.support.DefaultMessageSourceResolvable: codes [customerRequest.dateOfBirth,dateOfBirth]; arguments []; default message [dateOfBirth]]; default message [**Date of Birth must be in the past.**]] [Field error in object 'customerRequest' on field 'email': rejected value [myemail.com]; codes [Email.customerRequest.email,Email.email,Email.java.lang.String,Email]; arguments [org.springframework.context.support.DefaultMessageSourceResolvable: codes [customerRequest.email,email]; arguments []; default message [email],[Ljakarta.validation.constraints.Pattern$Flag;@3e7db681,.*]; default message [**Email is not valid.**]] ",
    "timestamp": "2023-05-06T07:48:58.2077013"
}

When the validation fails on an argument annotated with @Valid, Spring will throw a MethodArgumentNotValidException. The message in the above JSON is the message coming from this Exception. This is handled by the Controler Advice in our project.

@ExceptionHandler(Exception.class)
@ResponseStatus(HttpStatus.BAD_REQUEST)
    RestErrorResponse handleException(Exception ex) {
        return new RestErrorResponse( 
            HttpStatus.BAD_REQUEST.value(), 
            ex.getLocalizedMessage(), 
            LocalDateTime.now());
    }

What we aim to achieve is presenting the error messages defined in the annotations to the consumer in a clear and readable format. Let's proceed to the next section to understand how to accomplish this.

Handling MethodArgumentNotValidException

As mentioned earlier, the MethodArgumentNotValidException is thrown when any constraints fail during validation. This exception implements the BindingResult interface, which contains the validation results. To address this situation, we need to add a new ExceptionHandler method for handling this specific exception in the Adviser.

1. @ExceptionHandler(MethodArgumentNotValidException.class)
2. @ResponseStatus(HttpStatus.BAD_REQUEST) RestErrorResponse 
3.     handleException(MethodArgumentNotValidException ex) {

4.     String message = ex.getFieldErrors()
5.         .stream()
6.         .map(e -> " Field "+e.getField() + 
7.                   " Message " + e.getDefaultMessage())
8.         .reduce("Errors found:", String::concat);
9.    return new RestErrorResponse(HttpStatus.BAD_REQUEST.value(), 
                                   message, LocalDateTime.now());
}

Let's take a closer look at the above code snippet. In Lines 1-3 it is declared a handler method to manage any MethodArgumentNotValidException thrown. On line 4, the getFieldErrors method will return the list of FieldErrors. As you can see this method is actually delegating to the bindingResult.

public List<FieldError> getFieldErrors() {
    return this.bindingResult.getFieldErrors();
}

On line 6 the map operation receives the FieldError and returns field name and message as String. This is the field where the validation failed and the associated message that was set in the annotations.

On line 8, the terminal reduce operation is called to concatenate each String from the map operation.

Finally, on line 10 the custom error is returned.

Invoking the same POST request sent in the previous section will produce the below response:

{
    "status": 400,
    "message": "Errors found: Field dateOfBirth. Message Date of Birth must be in the past. Field email. Message Email is not valid. Field name. Message Name must be at least 3 characters and at most 20 characters.",
    "timestamp": "2023-05-07T16:09:46.8730112"
}

By implementing this exception handler, the response sent back from the server will provide clear error messages to the consumer, facilitating the identification and resolution of validation errors.

Summary

In this article, we explored JSR 380 and its usage in data validation for Java applications. We saw how to apply validation constraints using annotations, integrate Bean Validation with Spring Boot 3, and handle validation errors in a consumer-friendly manner. The provided code examples and concepts can serve as a foundation for implementing data validation in your own projects.

In future articles, we will delve into topics such as creating custom constraints and performing validations on URLs and request parameters. Stay tuned for more!

As usual code can be found in the project github repo here