Forem: Mark Dsouza

Jackson's @JsonView with SpringBoot Tutorial

Mark Dsouza — Thu, 16 Jun 2022 04:57:43 +0000

Introduction

If you are a SpringBoot dev and you work on a large project, you ought to know about the annotation provided by Jackson -@JsonView, because it is extremely powerful and elegant. @JsonView by Jackson helps selectively serialize and de-serialize objects. By doing so it reduces your code base by a ton by avoiding unnecessary classes and also those irritating getters and setters to copy the values of one object to another.

Real Time Example

Look at the below JSON Object

{
  "userId": "markbdsouza",
  "firstName": "Mark",
  "lastName": "Dsouza",
  "socialMediaAccount: {
     "twitterUrl": "https://twitter.com/MarkBDsouza",
     "githubUrl": "https://github.com/markbdsouza",
     "devUrl": "",
     "hashNodeUrl":""
  },
  "articles": [
     {
        "summary": "JS: Sort an Array of Objects on multiple columns/keys",
        "url": "https://dev.to/markbdsouza/js-sort-an-array-of-objects-on-multiple-columns-keys-2bj1"
     },
     {
        "summary": "Spring Cloud Config Server: Step by Step",
        "url": "https://dev.to/markbdsouza/spring-cloud-config-server-step-by-step-14fd"
     }
  ]
}

Here, we have a Json Object User.
We see one User has a single SocialMediaAccount Object(OneToOne Relationship) and one Customer has multiple articles (OneToMany Relationship).
In Java, for this structure, we would need 3 Classes - User, SocialMediaAccount, Article.
The 3 Classes would be as below

public class User{
  private String userId;
  private String firstName;
  private String lastName;
  private SocialMediaAccount socialMediaAccount;
  private List<Article> articles;
  // excluding getters and setters for brevity
}

public class SocialMediaAccount {
  private String twitterUrl;
  private String githubUrl;
  // excluding getters and setters for brevity
}

public class Article{
  private String summary;
  private String url;
  // excluding getters and setters for brevity
}

How JSonView Helps

Now with this one structure, we can JSON Objects containing any combination of fields to form the below JSON from endpoint 2

{
  "userId": "markbdsouza",
  "socialMediaAccount: {
     "twitterUrl": "https://twitter.com/MarkBDsouza"     
  },
  "articles": [
     {
        "summary": "JS: Sort an Array of Objects on multiple columns/keys",
        "url": "https://dev.to/markbdsouza/js-sort-an-array-of-objects-on-multiple-columns-keys-2bj1"
     },
     {
        "summary": "Spring Cloud Config Server: Step by Step",
        "url": "https://dev.to/markbdsouza/spring-cloud-config-server-step-by-step-14fd"
     }
  ]
}

OR the below structure from end point 3

{
  "userId": "markbdsouza",
  "firstName": "Mark", 
  "socialMediaAccount: {
     "twitterUrl": "https://twitter.com/MarkBDsouza",
     "githubUrl": "https://github.com/markbdsouza",
     "devUrl": "",
     "hashNodeUrl"
  }
}

OR any other structure consisting of the same variables in whatsoever permutation/combination.
The business logic for all endpoints can be exactly the same. The only difference is the annotation that needs to be provided such that when converting POJO to JSON or vice versa, Jackson will check which fields need to included when serializing/de-serializing. That way we do not have to create any new Class with the specific structure and then copy the data from the original object to the new class object for each endpoint.

Summary of Important Points regarding JsonView

The below points serve as a quick summary of everything you probably need to know about JsonView to get started with implementation.

A View in JsonView defines how you want to expose a Class when it is converted from POJO to JSON by the Jackson package. Jackson will serialize/de-serialize only the fields with the corresponding @JsonView.
Create a simple Java class(no class level annotation required) that will hold with interfaces/child classes which are our View Names. We can use inheritance so View 2 also includes everything present in View 1. This is helpful when you are building a new View that extends the fields visible in an already existing View.

public class Views{
    public interface Public{}
    public interface CustomerDetails extends Public{}
}

Once the View class is created, annotation is required at 2 places. 1) The Class itself or the instance variables of the Class that needs to be serialized/de-serialized. 2) At the endpoints in your controller so Jackson knows where to apply the View.
At the Class level(of what we are serialzing/ de-serializing), we need to annotate the required instance variable with @JsonView(Views.Public.class) where Views is the class created in the above step and Public is the interface/subclass created in it. This is what tells Jackson which instance variable of the class to serialize/de-serialize. Example :

    @JsonView(Views.Public.class)
    private String firstName;

One instance variable can have any number of JsonViews attached to it. Example:

    @JsonView({Views.Public.class, Views.UserDetails.class})
    private String email;

If we have an object of Class B as an instance variable of class A. And have attached @JsonView to the object, it will internally also serialize/de-serialize only the annotated instance variables in class B. Example:

public class A{
    @JsonView(Views.Public.class)
    private B b;
}

class B{
    private String password; // will not be included by jackson
    @JsonView(Views.Public.class)
    private String username;
}

If we do not use the @JsonView in class B, then no field will get serialized/de-serialized of B and it would be an empty object when A gets serialized/de-serialized.

Alternatively, we can assign the annotation to the entire class as well.
In the controllers, we need to annotate the endpoints accordingly so Jackson knows which endpoints have the logic while converting POJO to JSON or vice versa.
For GET endpoints, if we are returning a class and want to provide only visible fields, we annotate the end point with

    @JsonView(Views.Public.class)
    @GetMapping("/1")
    public Customer getDetails() {
            // business logic to return customer 
    }

If we are returning the same class in another endpoint, it is not necessary that we have @JsonView. If your controller does not have @JsonView, all fields of the POJO returned will be converted to JSON.
If it is a HTTP POST call and we are sending the class in our body, using @JsonView we can accept only the fields mentioned in our View. To do this

@PostMapping
public Student postLevelTwoBasic(@RequestBody @JsonView(LevelTwoViews.Public.class) Student student) {
    // some POST implementation
}

Detailed Implementation and Tutorial

In the below section, we will walk through some code and set up JsonView and see how we can use it. You can find the github link with the entire code : https://github.com/markbdsouza/jsonview-springboot-tutorial
The tutorial has been split into 3 levels. We will increasingly use more features and explore other uses of JsonView.

Level 1 : JsonView with a Single Class

To start off, we will create a Single JAVA Class and use JsonView annotation on it. We will be checking both GET as well as POST requests and see how it all works together. In future levels, we will try multiple classes and even entities.

Step 1.1 : Create JsonView Class

Create the Class containing the different views that need to be created.
Observe how we use inheritance in these interfaces. Public is kept as the base View. We have 2 different views that inherit Public - UserNameDetails and OtherDetails. And UserDetails extends UserNameDetails so it extends Public inherently.

public class LevelOneViews {
    /*
     * Any variable associated with Public will be picked up by this view
     */
    public interface Public {
    }

    /*
     * Any variable associated with UserNameDetails or Public will be picked up by this view
     */
    public interface UserNameDetails extends Public {
    }

    /*
     * Any variable associated with UserDetails, UserNameDetails or Public(since UserNameDetails internally
     *  extends Public)  will be picked up by this view
     */
    public interface UserDetails extends UserNameDetails {
    }

    /*
     * Any variable associated with OtherDetails or Public will be picked up by this view.
     * Note: Will not pick up variables associated with UserNameDetails or UserDetails
     */
    public interface OtherDetails extends Public {
    }
}

We have created 4 views here. Take a minute to see the inheritance and understand the inheritance tree.

Step 1.2 : Create Java Class with the @JsonView on required variables

Create User class with instance variables with the Views created in the above step

@Data
@ToString
public class User {
    @JsonView({LevelOneViews.Public.class})
    private int userId;
    @JsonView(LevelOneViews.UserNameDetails.class)
    private String firstName;
    @JsonView(LevelOneViews.UserNameDetails.class)
    private String lastName;
    @JsonView({LevelOneViews.UserDetails.class, LevelOneViews.OtherDetails.class})
    private String email;
    @JsonView({LevelOneViews.UserDetails.class, LevelOneViews.OtherDetails.class})
    private String organization;
    private Boolean isActive;
}

Step 1.3 : Create Controller and GET Endpoints

Create endpoints with the 4 views created. We have a private static method to create an Object of User. Note that for all methods the implementation is the same. We only call the createUser() method. The only difference in all methods is which View we are using. For each of these GET mappings using JsonView we need to annotate the entire method with @JsonView with the view. Example: @JsonView(LevelOneViews.UserNameDetails.class)

@RestController
public class UserController {

    /**
     * GET without any view
     *
     * @return created User
     */
    @GetMapping("/levelOne/default")
    public User getLevelOneUser() {
        return createUser();
    }

    /**
     * GET with Public View
     *
     * @return created User
     */
    @JsonView(LevelOneViews.Public.class)
    @GetMapping("/levelOne/public")
    public User getLevelOnePublicUser() {
        return createUser();
    }

    /**
     * GET with UserNameDetails View which extends Public
     *
     * @return created User
     */
    @JsonView(LevelOneViews.UserNameDetails.class)
    @GetMapping("/levelOne/name")
    public User getLevelOneNameUser() {
        return createUser();
    }

    /**
     * GET with UserDetails  View which extends UserNameDetails and also Public
     *
     * @return created User
     */
    @JsonView(LevelOneViews.UserDetails.class)
    @GetMapping("/levelOne/details")
    public User getLevelDetailsUser() {
        return createUser();
    }

    /**
     * GET with OtherDetails View which extends Public
     *
     * @return created User
     */
    @JsonView(LevelOneViews.OtherDetails.class)
    @GetMapping("/levelOne/other")
    public User getLevelOneOther() {
        return createUser();
    }

    /**
     * @return created user
     */
    private static User createUser() {
        User user = new User();
        user.setUserId(1);
        user.setFirstName("Mark");
        user.setLastName("Dsouza");
        user.setEmail("mark.benjamin.dsouza@google.com");
        user.setIsActive(true);
        user.setOrganization("DEV.TO");
        return user;
    }
}

There are 5 endpoints. 1 without a view and 4 with.

Step 1.4 : Test the GET Endpoints

Run the springboot application and lets test out our endpoints.

Endpoint 1: Without any View

All fields of our User Object will get deserailized
http://localhost:8080/levelOne/default

Endpoint 2: With Public View

Public View is our basic default view that has the least number of variables
http://localhost:8080/levelOne/public

Notice how only the userId column is returned even though the code for both endpoint 1 and 2 are exactly the same. This is done through JsonView.

Endpoint 3: With UserNameDetails View

http://localhost:8080/levelOne/name
UserNameDetails extends Public. So, through inheritance, it displays all fields with UserNameDetails as well as Public. So we see 3 fields in our response.

Endpoint 4: With UserDetails View

http://localhost:8080/levelOne/details
UserDetails extends UserNameDetails which through inheritance means UserDetails extends Public as well. So all fields with UserDetails,UserNameDetails and Public will be available in the response.

Endpoint 5: With OtherDetails View

http://localhost:8080/levelOne/other
OtherDetails does bit extend the above views, instead only the Public View. So Only fields with either OtherDetails or Public will be available in the response.

Step 1.5 : Add POST Endpoints

    Logger log = LoggerFactory.getLogger(UserController.class);
    /**
     * POST without any view
     *
     * @param user
     */
    @PostMapping("/levelOne/default")
    public User postLevelOneDefault(@RequestBody User user) {
        log.info(String.valueOf(user));
        return user;
    }

    /**
     * POST with Public view
     *
     * @param user
     */
    @PostMapping("/levelOne/name")
    public User postLevelOneDetails(@RequestBody @JsonView(LevelOneViews.UserNameDetails.class) User user) {
        log.info(String.valueOf(user));
        return user;
    }

Step 1.6 : Test the POST Endpoints

Endpoint 1: Without any Views

Post call to URL:http://localhost:8080/levelOne/default
Request Body:
We are sending the entire structure.
Logger: User(userId=1, firstName=Mark, lastName=Dsouza, email=mark.benjamin.dsouza@google.com, organization=DEV.TO, isActive=true)
All data gets logged.

All data gets returned.

Endpoint 2: With JsonView

Let us not try this with JsonView endpoint
Post call to URL: http://localhost:8080/levelOne/name

Logger: User(userId=1, firstName=Mark, lastName=Dsouza, email=null, organization=null, isActive=null)
Notice how even though we sent an entire object only 3 fields got de-serialized.
The response is what we saw in the logs. The controller itself did not recieve all fields, so only some fields which are linked with the corresponding view got de-serialized.

Takeaway from Level 1

We saw what classes are needed for implementing JsonView and how we can use it for HTTP GET as well as POST requests. We needed to add annotation to the Class that is being serialized/de-serialized as well as the controller endpoint/API. Also, when defining views and using them, inheritance can be used such that one field is associated with multiple views easily.

Level 2 : JsonView with multiple classes

In this level, we are going to see how things work when it isn't just a simple Java Class with just Strings and numbers. Let's see how it works when we have objects of other classes as well. These are generally OneToOne(Single Object) and OneToMany(List of Object) relationships.

Step 2.1 : Create JsonView Class

Create the Class containing the different views that need to be created.

public class LevelTwoViews {
    /*
     * Any variable associated with Public will be picked up by this view
     */
    public interface Public {
    }

    /*
     * Any variable associated with BasicStudentDetails or Public will be picked up by this view
     */
    public interface BasicStudentDetails extends Public {
    }

    /*
     * Any variable associated with BasicStudentDetails, AllStudentDetails or Public will be picked up by this view
     */
    public interface AllStudentDetails extends BasicStudentDetails {
    }
}

Here, again we have used inheritance and created 3 views of increasing access.

Step 2.2 : Create multiple Java Classes with the @JsonView on required variables

In Level2, we will be creating multiple classes to illustrate how it works

Create Student

@Data
@ToString
public class Student {
    @JsonView(LevelTwoViews.Public.class)
    private String fullName;
    @JsonView(LevelTwoViews.BasicStudentDetails.class)
    private float height;
    @JsonView(LevelTwoViews.BasicStudentDetails.class)
    private float weight;
    @JsonView(LevelTwoViews.Public.class)
    private int age;
    @JsonView(LevelTwoViews.BasicStudentDetails.class)
    private String className;
    // One to Many Relationship
    @JsonView(LevelTwoViews.AllStudentDetails.class)
    private List<SubjectScore> scores;
    // One to One Relationship
    @JsonView(LevelTwoViews.BasicStudentDetails.class)
    private School school;
}

Create School

One Student has One School - One To One Relationship

@Data
@ToString
public class School {

    @JsonView(LevelTwoViews.BasicStudentDetails.class)
    private String schoolName;
}

Create SubjectScore

One Student has multiple Subject Scores - One to Many Relationship

@Data
@ToString
public class SubjectScore {
    @JsonView(LevelTwoViews.AllStudentDetails.class)
    private String subject;
    @JsonView(LevelTwoViews.AllStudentDetails.class)
    private int score;
    private String teacher;
}

Note that teacher has no view. So irrespective of what JsonView we provide at the controller level, teacher will never get serialized/de-serialized

Step 2.3 : Create Controller and GET Endpoints

Create endpoints for the views created. We have a few private static methods to create a dummy Object. Here also for all methods the implementation is the same and the only difference is which View we are using.

@RestController
public class StudentController {

    /**
     * GET without any json view
     *
     * @return created Student
     */
    @GetMapping("/levelTwo/default")
    public Student getLevelTwoDefault() {
        return createStudent();
    }

    /**
     * GET with lowest view Public
     *
     * @return created Student
     */
    @JsonView(LevelTwoViews.Public.class)
    @GetMapping("/levelTwo/public")
    public Student getLevelTwoPublic() {
        return createStudent();
    }

    /**
     * GET with BasicStudentDetails View which extends Public
     *
     * @return created Student
     */
    @JsonView(LevelTwoViews.BasicStudentDetails.class)
    @GetMapping("/levelTwo/basic")
    public Student getLevelTwoBasicDetails() {
        return createStudent();
    }

    /**
     * GET with AllStudentDetails View which extends BasicStudentDetails,Public
     *
     * @return created Student
     */
    @JsonView(LevelTwoViews.AllStudentDetails.class)
    @GetMapping("/levelTwo/all")
    public Student getLevelTwoAllDetails() {
        return createStudent();
    }

    private static Student createStudent() {
        Student student = new Student();
        student.setAge(15);
        student.setFullName("Mark Dsouza");
        student.setWeight(49);
        student.setHeight(150);
        student.setClassName("VIII");
        student.setSchool(createSchool());
        student.setScores(createSubjectScoreList());
        return student;
    }
    private static School createSchool() {
        School school = new School();
        school.setSchoolName("Indian Public School");
        return school;
    }
    private static List<SubjectScore> createSubjectScoreList() {
        List<SubjectScore> subjectScoreList = new ArrayList<>();
        SubjectScore mathScore = new SubjectScore();
        mathScore.setScore(80);
        mathScore.setSubject("Math");
        mathScore.setTeacher("Mr. John Watts");
        SubjectScore englishScore = new SubjectScore();
        englishScore.setScore(80);
        englishScore.setSubject("English");
        englishScore.setTeacher("Mrs. Mary Johnson");
        subjectScoreList.add(mathScore);
        subjectScoreList.add(englishScore);
        return subjectScoreList;
    }

We have created 4 GET Endpoints, one with no view and 3 with the views we created.

Step 2.4 : Test the GET Endpoints

Run the SpringBoot application and let us test out our endpoints for level 2 GET methods.

Endpoint 1: Without any View

Http Method: GET
URL: http://localhost:8080/levelTwo/default
Response:

Here we get back all the data of the object and objects children as well.

Endpoint 2: With Public View

Http Method: GET
URL: http://localhost:8080/levelTwo/public
Response:

We see only data with Public View

Endpoint 3: With BasicStudentDetails

Http Method: GET
URL: http://localhost:8080/levelTwo/basic
Response:

Notice how scores isnt included in this view as we have not given the required view for that object in our main class Student.

Endpoint 4: With AllStudentDetails View

Http Method: GET
URL: http://localhost:8080/levelTwo/all
Response:

Similarly, here the teacher of class Score is not present as the view wasnt defined.

Step 2.5 : Add POST Endpoints

Now create the POST Endpoints

    Logger log = LoggerFactory.getLogger(StudentController.class);
    /**
     * POST Mapping without any views
     *
     * @param student
     * @return same student
     */
    @PostMapping("/levelTwo/default")
    public Student postLevelTwoDefault(@RequestBody Student student) {
        log.info(student.toString());
        return student;
    }

    /**
     * POST Mapping with Public View. Instance variables of Student without Public view will not be deserialized
     *
     * @param student
     * @return same student
     */
    @PostMapping("/levelTwo/public")
    public Student postLevelTwoBasic(@RequestBody @JsonView(LevelTwoViews.Public.class) Student student) {
        log.info(student.toString());
        return student;
    }

Step 2.6 : Test the POST Endpoints

Endpoint 5: POST without a view

Http Method: POST
URL: http://localhost:8080/levelTwo/default
Request Body:

Logger: Student(fullName=Mary Jane, height=137.0, weight=34.0, age=16, className=X, scores=[SubjectScore(subject=Math, score=90, teacher=Mr. John Watts), SubjectScore(subject=English, score=40, teacher=Mrs. Mary Johnson)], school=School(schoolName=New York Public School))
We have logged what is recieved by the controller and see all fields hav valid data.
Response:

We see the same data that is sent is returned completely.

Endpoint 6: POST with a view

Http Method: POST
URL: http://localhost:8080/levelTwo/public
Request Body: We use the same request as earlier
Logger: Student(fullName=Mary Jane, height=0.0, weight=0.0, age=16, className=null, scores=null, school=null)
Notice the difference here, almost EVERYTHING(except fullName and age) is defaulted even though we sent the entire data as the request body. This is where JsonView works its magic. Even though we send all the data, JsonView de-serializes ONLY the data mentioned with the appropriate view(in this case Public, which is givne only to two fields).
Response:

Since the controller itself didnt get the data, the data returned also is only what was made visible by JsonView.

Takeaway from Level 2

In Level 2, we saw how we can work with multiple Classes and how we can use views across them to achieve the data structure that we need.

Level 3: Using Entities with JsonView

In Level 3, we will use a H2 Database to pick data from instead of just some dummy data. And we will directly use the @JsonView on the Entity itself. This helps us eliminate the use of any DTO, RequestModel, ResponseModel.
Note: The Github code has included data.sql and schema.sql in the src/main/resources folder and some properties in application.properties which will automatically create the schema and insert data into the H2 DB.

Step 3.1 : Create JsonView Class

Create the Class containing the different views that need to be created.

/**
 * List of Views for LevelThree functionality.
 */
public class LevelThreeViews {
    /*
     * Any variable associated with Public will be picked up by this view
     */
    public interface Public {
    }

    /*
     * Any variable associated with CustomerDetails or Public will be picked up by this view
     */
    public interface CustomerDetails extends Public {
    }

We have created 2 views, Public and CustomerDetails.

Step 3.2 : Create Java Entity with the @JsonView on required variables

While creating our Entity, add additional @JsonView for the required instance variables.

@Entity
@Table(name = "customer")
@Data
public class Customer {
    @Id
    @GeneratedValue
    private Long id;
    @Column(name = "CUSTOMER_ID", nullable = false)
    @JsonView(LevelThreeViews.Public.class)
    private String customerId;
    @Column(name = "FIRST_NAME", nullable = false, length = 50)
    @JsonView(LevelThreeViews.CustomerDetails.class)
    private String firstName;
    @Column(name = "LAST_NAME", nullable = false, length = 50)
    @JsonView(LevelThreeViews.CustomerDetails.class)
    private String lastName;
}

The 'id' column does not have any view associated with it. We do not want to expose our DB's Primary Key to any client of our API. Instead we only expose the CUSTOMER_ID which a String value.

Step 3.3 : Create Controller and GET Endpoints

Create endpoints with the multiple views created. Here for all methods, we fetch the data from the DB. The code is exactly the same for all APIs, the only difference is which View we are using.

@RestController
public class CustomerController {
    @Autowired
    private CustomerRepository customerRepository;

    /**
     * GET Mapping without JsonView
     * @return List of Customer inserted in the DB
     */
    @GetMapping("/levelThree/default")
    public List<Customer> getLevelThree() {
        return customerRepository.findAll();
    }

    /**
     * GET Mapping with Public JsonView
     * @return List of Customer inserted in the DB
     */
    @JsonView(LevelThreeViews.Public.class)
    @GetMapping("/levelThree/public")
    public List<Customer> getLevelThreePublic() {
        return customerRepository.findAll();
    }

    /**
     * GET Mapping with CustomerDetails JsonView which extends Public as well
     * @return List of Customer inserted in the DB
     */
    @JsonView(LevelThreeViews.CustomerDetails.class)
    @GetMapping("/levelThree/details")
    public List<Customer> getLevelThreeDetails() {
        return customerRepository.findAll();
    }
}

Overall we have 3 endpoints to see the difference between NoView, PublicView and CustomerDetailsView.

Step 3.4 : Test the GET Endpoints

Run the springboot application and lets test out our endpoints.

Endpoint 1: Without any View

Http Method: GET
URL: http://localhost:8080/levelThree/default
Response:
All fields are returned since there is no view for this API

Endpoint 2: With Public View

Http Method: GET
URL: http://localhost:8080/levelThree/public
Response:
Notice that only fields with Public View are returned.

Endpoint 3: With CustomerDetailsView

Http Method: GET
URL: http://localhost:8080/levelThree/details
Response:
Notice that fields with Public and CustomerDetailsView are returned. But it does not contain the id column since we havent given the annotation and view name for the id column.

Takeaway from Level 3

Level 3 showed us how we can use entities directly and securely hide data that we dont want exposed in any APIs. This includes any sequence generated columns or audit columns or any other column as well. Similar to what we saw in Level 2, we can have related entities as well with OneToOne, OneToMany and ManyToOne Annotations.
Side Note: If you are implementing this in your code and have multiple relationships, you could run into a cyclic reference issue during deserialization. Check out JsonManagedReference and JsonBackReference on StackOverflow to solve the issue.

Conclusion

Throughout this tutorial, we have seen different instances of how we can use @JsonView for HTTP calls when it the object is part of a response or the object is part of the request body. We also saw how we can use multiple classes and entities with JsonView.
I hope you find this useful and let me know in the comments or DM me on twitter if you have any questions. Cheers and Happy Coding!

Apache Airflow for Beginners

Mark Dsouza — Thu, 28 Apr 2022 12:04:48 +0000

In this blog post, we are going to take a look at how we can setup Apache Airflow on our systems and get you as a developer, started off with just the bare minimum so you can start working on it.
For detailed documentation please always refer the Airflow official Documentation

Introduction to Airflow

Airflow is a workflow management platform for data engineering pipelines. It helps define workflows with python code and provides a rich UI to manage and monitor these workflows. Airflow supports easy integration with all popular external interfaces like DBs(SQL and MongoDB), SSH, FTP, Cloud providers etc. It sounds like it does a lot and it does, and can be intimidating, but it is really easy to get started.
The biggest plus point about Airflow is its user friendly UI.

The Most Common Terms used in Airflow

DAG (Directed acyclic graph) A DAG is a python(.py) file that defines what the steps are in our workflow. Each DAG has some common configuration and details of what task needs to be done at each step. Note: You do not need to know advanced python to start working on DAGs but should know the basics.
Tasks - Each step of our workflow is called a Task. We can provide different relationship/dependencies between tasks. Example: Task2 should run after Task1. And Task3 should run after Task2. Task4 should run if Task1, Task2 and Task3 are successful only. Task5 should run if either Task1, Task2 or Task3 fail.
Operators - Now tasks are executed using certain Airflow operators. There are around 50 Operators defined. Each operator has configuration that can be done to suit our requirement. We can even call python functions and use the output 2 or a bash operator and execute a specific command.
Connections & Hooks - There is a layer between the code and the connection details. Airflow already has integration with many external interfaces so you do not need write low level code. For example if you need a DB connection, in the Airflow UI you will create a connection with the host name, port, username and password of the connection and provide a conn_id(connection id). And in your DAG(Python code) you will ONLY use the conn_id as a reference. This segregates your code from the connection configuration and makes it reusable across environments as well.
XComs - XCom(Cross Communication) is basically the data that flows between Task1 and Task2. Note: the data passed between tasks should be very minimal and we should not pass large objects.

Installing Airflow with Docker Compose

Now that we have seen the popular concepts, let us get airflow up and running on our local machine.
You can follow the official documentation for installation

I will go through a very simple setup here.
Prerequisite: docker and docker compose should already be installed.
Download the Airflow docker compose file created by the Airflow team:
Save it on your local machine in a folder of your choice and run



docker-compose up

You should see the various images being downloaded(takes a lot of time the very first you run the command) and you will see containers starting up.
After a while all your containers will be up.
To check this open a new terminal in parallel and run



docker container ls

You should see 7 containers running

Now go to localhost:8080 and you have the below page opening up

Username: airflow
Password: airflow
This will open up your home screen

Explore features of Airflow

Let us now see the various features of Airflow.

Home Screen

Our Home Screen shows

Your home screen is auto loaded with a bunch of pre-defined DAGs. This is for your reference and you to play around with when creating your own DAG!!

DAG Views

Let's look at the first DAG - example bash operator

Tree View

Click on the DAG to bring up the details

This brings up the Tree view and shows how the steps are linked in a tree like structure.
Make sure that you Activate the DAG by clicking the switch button on the top right. This can be done for the home page as well if you noticed.
Once you click it, you can click on the Run button on the top right to execute the DAG. On clicking Run, choose 'Run DAG' option. The Run DAG w/ config option is if you have some custom configuration for the DAG which is not required in this case.

Congrats, you've run your very first DAG.

Code View

A lot of the different views are for monitoring purposes once the code is deployed. To see the code itself click on the last view the Code View.
The airflow team has given us a couple of DAGs pre built and we can now check what is written.
Below is the code for this particular DAG



#
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements.  See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership.  The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License.  You may obtain a copy of the License at
#
#   http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied.  See the License for the
# specific language governing permissions and limitations
# under the License.

"""Example DAG demonstrating the usage of the BashOperator."""

import datetime

import pendulum

from airflow import DAG
from airflow.operators.bash import BashOperator
from airflow.operators.dummy import DummyOperator

with DAG(
    dag_id='example_bash_operator',
    schedule_interval='0 0 * * *',
    start_date=pendulum.datetime(2021, 1, 1, tz="UTC"),
    catchup=False,
    dagrun_timeout=datetime.timedelta(minutes=60),
    tags=['example', 'example2'],
    params={"example_key": "example_value"},
) as dag:
    run_this_last = DummyOperator(
        task_id='run_this_last',
    )

    # [START howto_operator_bash]
    run_this = BashOperator(
        task_id='run_after_loop',
        bash_command='echo 1',
    )
    # [END howto_operator_bash]

    run_this >> run_this_last

    for i in range(3):
        task = BashOperator(
            task_id='runme_' + str(i),
            bash_command='echo "{{ task_instance_key_str }}" && sleep 1',
        )
        task >> run_this

    # [START howto_operator_bash_template]
    also_run_this = BashOperator(
        task_id='also_run_this',
        bash_command='echo "run_id={{ run_id }} | dag_run={{ dag_run }}"',
    )
    # [END howto_operator_bash_template]
    also_run_this >> run_this_last

# [START howto_operator_bash_skip]
this_will_skip = BashOperator(
    task_id='this_will_skip',
    bash_command='echo "hello world"; exit 99;',
    dag=dag,
)
# [END howto_operator_bash_skip]
this_will_skip >> run_this_last

if __name__ == "__main__":
    dag.cli()

Well you do need to know a bit of Python to understand the syntax and coding. But just the basics is more than enough to get started with DAGs.
Note: You CANNOT change the DAG in the UI directly. You need to do modify the Python file containing the DAG code and place it in the appropriate folder to do so

Graph View

Another important view is the graph view

By clicking one of the steps, you will get a popup which gives you many step level options

Probably the most important is the Log Option. This will give you all the details of the logs on the machine when the particular step was executed

Make sure you go through the other views and see what can be used for your project.

DAGS Active Filter

On navigating back to the home screen you will notice that there are 3 tabs on the top.
Always remember that even if you try and run a DAG, it won't run until it is 'Active'

Admin

Another important place to keep an eye out for is the Admin dropdown.
Lets look at 3 of these options

Variables

Variables allow you to have global level variables used across the application

Connections

Almost always you will want your airflow workflow to talk to some external service. Connections provides an interface between the actual connection string/details and the code you need to write. In the code you can refer the connection name and the details of connection can vary in each environment of airflow. This can even be a HTTP Connection for a REST API Call.

XComs

You can view the data passed between steps in your DAG.

It provides the DAG id and task id and you can easily debug runs here instead of checking the logs of multiple runs.

With these views you probably get the most important features of Airflow from a developers point of view.

Creating and Running our own DAG

Now, lets create a simple simple DAG and run it.

Create the .py file

On navigating back to your docker-compose folder. You should see 3 folders automatically created. dags, logs and plugins.
To add a DAG to the airflow instace, create a .py file ib the dags folder. Below is a sample file



from airflow import DAG
from datetime import datetime
from airflow.operators.http_operator import SimpleHttpOperator


with DAG("http_test_dag", start_date=datetime(2021, 1, 1), schedule_interval="@daily", catchup=False) as dag:
    http_call = SimpleHttpOperator(
        task_id="HttpCall",
        http_conn_id='springboot-http',
        endpoint='',
        method='GET',
        headers={"Content-Type": "application/json"},
        data=None,
        log_response=True,
        response_check=lambda response: response.ok)


http_call

This DAG uses a simple Http operator that makes a call to the connection - 'springboot-http'. This is a connection that is expected to be generated in the admin>connections.

Updating Airflow

Airflow automatically will pick up your new DAG file if placed in the folder. You do not need to do anything to deploy.
A common mistake is looking at the Code tab of a deployed DAG and refreshing and expecting to see your new code but it doesn't get updated. You might be inclined to think something is wrong with airflow and you need to stop and restart the docker containers. But the issue is with the modifications you have made in the DAG. If the code doesn't compile, the code in the code tab won't get updated.

Now you can activate the DAG and run it!!

I hope this post is helpful for anyone trying to learn about Airflow and just get quickly started so you can play around with it.

Pagination(with Hateoas), Filtering & Sorting with Spring Boot and JPA

Mark Dsouza — Thu, 07 Apr 2022 08:04:02 +0000

Introduction

In this blog post we are going to look at how we can implement pagination with hateoas while supporting filtering and sorting in a spring boot application. We will first start off implementing filtering, and then iteratively page, sorting and finally pagination wtih Hateoas.

Tech Stack

Below are the list of languages, frameworks and libraries used in this tutorial

Java 11
Maven
h2 in memory database
REST apis
SpringBoot 2.6.6
Spring Data jpa
Hateoas

Usecase for Hateoas with pagination and filterin/sorting

When fetching data from a table you probably implement your response object as a List<Object>. This works perfectly fine if it is a small data set.
When you have a data set that is 'large', you do not want to send the data to the client all at once. Imagine you have 100k rows in your table and each row has 10 columns. If the user is only vieweing maybe 50 rows in the UI(our client) at a specific time, why send 100k rows?
Unless it a specific export functionality, you shouldn't send back all the data to the user. Instead you should send a subset of the data only.
At the same time, we do not want to restrict the user from seeing all the data. If the user wants to add a filter or sort some data out of these 100k rows, he should be able to do it.
From a front end perspective, if the user clicks on the next page, they will do another call to the backend to get the next set of data.
This is where Pagination + Filtering + Sorting becomes very very helpful.
Hateoas also provides links to other pages which is extremely helpful as we will see.

What will our end result look like

Before we get into the details, take a look at our request URL and the Response format we finally end up with. We will create a long url because we are allowing a high level of customization.

Request URL: http://localhost:8080/api/v4/customers?firstNameFilter=R&lastNameFilter=S&page=0&size=10&sortList=firstName&sortOrder=ASC
Response:



{
   "_embedded": {
      "customerModelList": [
         {
         "id": 971,
         "customerId": "de6b8664-ba90-41fc-a9f4-da7d0b89c106",
         "firstName": "Rabi",
         "lastName": "Dufour"
         },
         {
         "id": 339,
         "customerId": "44b5c01d-c379-4f66-b8ed-0fda4837db4e",
         "firstName": "Rachelle",
         "lastName": "Fleischer"
         },
         {
         "id": 838,
         "customerId": "443b06fd-7160-4234-9102-93afb0f6d9ad",
         "firstName": "Rafaelia",
         "lastName": "Bladen"
         }
      ]
   },
   "_links": {
      "first": {
         "href": "http://localhost:8080/api/v4/customers?firstNameFilter=R&sortList=firstName&sortOrder=ASC&page=0&size=3&sort=firstName,asc"
      },
      "self": {
         "href": "http://localhost:8080/api/v4/customers?firstNameFilter=R&sortList=firstName&sortOrder=ASC&page=0&size=3&sort=firstName,asc"
      },
      "next": {
        "href": "http://localhost:8080/api/v4/customers?firstNameFilter=R&sortList=firstName&sortOrder=ASC&page=1&size=3&sort=firstName,asc"
      },
      "last": {
        "href": "http://localhost:8080/api/v4/customers?firstNameFilter=R&sortList=firstName&sortOrder=ASC&page=19&size=3&sort=firstName,asc"
      }
   },
   "page": {
      "size": 3,
      "totalElements": 60,
      "totalPages": 20,
      "number": 0
   }
}

As you can see in this case, the endpoint provides you links on how you can access the other data. It also tells you how much data is there, how many pages and the elements per page and more. This is extremely helpful for any client accessing our API to traverse through the data.

Github link

The entire code discussed below is available on Github
https://github.com/markbdsouza/hateoas-with-pagination

Create a Base spring boot app

First before we get to the actual code, let's walk through our base code and set up a simple controller, service, entity and repository. Something you'd do for any spring boot application. Then let's also set up the Database and insert some data.
NOTE: Skip this section if you just need to see how the implementation is done. This is only for anyone following along.

Create base app

You can create the base application by using the start.spring.io website.
Enter your app details and add the below dependencies

Spring Web
Spring Data JPA
H2 Database
Spring HATEOAS This will auto create your pom.xml and project folders

Create Entity and Repository

We will be creating 1 Entity-customer with 4 fields - id(sequence generated), customerId(unique string), first name and last name. We will also keep getters and setters for encapsulation



@Entity
@Table(name="customer")
public class Customer {
    @Id
    @GeneratedValue
    private Long id;
    @Column(nullable = false, unique = true)
    private String customerId;
    @Column(nullable = false, length = 50)
    private String firstName;
    @Column(nullable = false, length = 50)
    private String lastName;

    public Long getId() {
        return id;
    }
    public void setId(Long id) {
        this.id = id;
    }
    public String getCustomerId() {
        return customerId;
    }
    public void setCustomerId(String customerId) {
        this.customerId = customerId;
    }
    public String getFirstName() {
        return firstName;
    }
    public void setFirstName(String firstName) {
        this.firstName = firstName;
    }
    public String getLastName() {
        return lastName;
    }
    public void setLastName(String lastName) {
        this.lastName = lastName;
    }
}

Create a repoistory interface for the entity. We will update this later.



@Repository
public interface CustomerRepository extends JpaRepository<Customer, Long> {}

Create Service

Now create a simple service file to fetch the data



@Service
public class CustomerService {
    @Autowired
    private CustomerRepository customerRepository;

    public List<Customer> fetchCustomerDataAsList() {
        // Fetch all customers using findAll
        return customerRepository.findAll();
    }
}

Create Controller

Now lets create a basic controller with a GET request that calls the service method



@RestController
public class CustomerController {
    @Autowired
    private CustomerService customerService;
    /**
     * @return List of all customers
     */
    @GetMapping("/api/v0/customers")
    public List<Customer> fetchCustomersAsList() {
        return customerService.fetchCustomerDataAsList();
    }
}

Now we have all our base code setup. Let's move on to configuring our Database and getting some data inserted.

Setup our Database

Update the application.properties to map our H2 DB



spring.h2.console.enabled=true
spring.h2.console.settings.web-allow-others=true
spring.datasource.username=sa
spring.datasource.password=
spring.datasource.url=jdbc:h2:mem:testdb
spring.datasource.driverClassName=org.h2.Driver
spring.datasource.initialization-mode=always
spring.datasource.initialize=true
spring.jpa.database-platform=org.hibernate.dialect.H2Dialect
spring.jpa.hibernate.ddl-auto=update

Now with our DB setup, we can use the springboot auto configuration to run scripts on our DB when the application is started.
Create a schema.sql file



CREATE TABLE customer (
   id INT NOT NULL,
   first_name VARCHAR(100) NOT NULL,
   last_name VARCHAR(100) NOT NULL,
   customer_id VARCHAR(100) NOT NULL
);

And create a data.sql file



insert into customer values(1,'Dorine','McGrouther','6252fcab-a17e-4af4-aa70-0fda826e67cf');
insert into customer values(2,'Gerianna','Capstack','f787ce02-06b7-4fc6-be83-408c652e924b');
insert into customer values(3,'Rosemarie','Salerno','4ac787e6-2534-43ea-a86e-16957b7410a2');

The script in the Github repository has around 1000 customers in the file.
Each time the application runs, since we are using a h2 database, it will run the schema.sql script and then the data.sql script which will set up our database environment.

Test our base app

To test it out launch the spring boot application with your IDE and use your browser and enter the URL
http://localhost:8080/api/v0/customers

If you've used the insert script in the repository, you will get 1000 records in this JSON output.

Implementing Filtering

Now let us first implement basic filtering in our code. We will allow the user to filter on 2 fields - firstName and/or lastName. For this we will create a new endpoint and pass in these 2 variables as Request parameters with a default value of ""

Update Repository

Add the below query to our CustomerRepository Interface



     String FILTER_CUSTOMERS_ON_FIRST_NAME_AND_LAST_NAME_QUERY = "select b from Customer b where UPPER(b.firstName) like CONCAT('%',UPPER(?1),'%') and UPPER(b.lastName) like CONCAT('%',UPPER(?2),'%')";

    @Query(FILTER_CUSTOMERS_ON_FIRST_NAME_AND_LAST_NAME_QUERY)
    List<Customer> findByFirstNameLikeAndLastNameLike(String firstNameFilter, String lastNameFilter);

We have written a custom query that will filter based on first name and last name.
Now all we need to do is call it from our service.

Update Service

Add a new method to our service



    public List<Customer> fetchFilteredCustomerDataAsList(String firstNameFilter, String lastNameFilter) {
        // Apply the filter for firstName and lastName
        return customerRepository.findByFirstNameLikeAndLastNameLike(firstNameFilter, lastNameFilter);
    }

Notice that we are still returning a List<Customer>. It is a filtered list though.

Update Controller

Add a new endpoint that accepts 2 fields as filter conditions



    /**
     * @param firstNameFilter Filter for the first Name if required
     * @param lastNameFilter  Filter for the last Name if required
     * @return List of filtered customers
     */
    @GetMapping("/api/v1/customers")
    public List<Customer> fetchCustomersAsFilteredList(@RequestParam(defaultValue = "") String firstNameFilter,
                                                       @RequestParam(defaultValue = "") String lastNameFilter) {
        return customerService.fetchFilteredCustomerDataAsList(firstNameFilter, lastNameFilter);
    }

Test it out

Run the application and this hit the v1 URL which we created
http://localhost:8080/api/v1/customers?firstNameFilter=ur&lastNameFilter=as
Try this out with different filters. Instead of all 1000 customers we will get only the filtered list of customers

In this case there are only 2 customers with first name containing 'ur' and last name containing 'as'.

Implementing Page with Filtering

Page is a common interface used to fetch data in a page format from our database. Though the format doesnt necessarily contain pertinent information, it still is limited data set and we can apply our filtering logic to it.

Update Repository

Add the below query to our CustomerRepository Interface



    @Query(FILTER_CUSTOMERS_ON_FIRST_NAME_AND_LAST_NAME_QUERY)
    Page<Customer> findByFirstNameLikeAndLastNameLike(String firstNameFilter, String lastNameFilter, Pageable pageable);

The only difference is that we are also passing a pageable object with the filters and returning a Page<Customer> instead of List<Customer>

Update Service

Add a new method to our service



    public Page<Customer> fetchCustomerDataAsPageWithFiltering(String firstNameFilter, String lastNameFilter, int page, int size) {
        // create Pageable object using the page and size
        Pageable pageable = PageRequest.of(page, size);
        // fetch the page object by additionally passing pageable with the filters
        return customerRepository.findByFirstNameLikeAndLastNameLike(firstNameFilter, lastNameFilter, pageable);
    }

Here we need 2 additional inputs- page and size. Page indicates which page we want to fetch from the DB. And size referes to the number of objects we want on each page.
Note: Page starts at 0 so be careful.
So if there are 1000 objects and we give a page of 2 and size of 50. We will get the third set of 101-150 objects.

Update Controller

Add a new endpoint that accepts 2 fields as filter conditions and 2 fields for page functionality



    /**
     * @param firstNameFilter Filter for the first Name if required
     * @param lastNameFilter  Filter for the last Name if required
     * @param page            number of the page returned
     * @param size            number of entries in each page
     * @return Page object with customers after filtering
     */
    @GetMapping("/api/v2/customers")
    public Page<Customer> fetchCustomersWithPageInterface(@RequestParam(defaultValue = "") String firstNameFilter,
                                                          @RequestParam(defaultValue = "") String lastNameFilter,
                                                          @RequestParam(defaultValue = "0") int page,
                                                          @RequestParam(defaultValue = "30") int size) {
        return customerService.fetchCustomerDataAsPageWithFiltering(firstNameFilter, lastNameFilter, page, size);
    }

Notice that we have kept the default page as 0 and size as 30.

Test it out

Run the application and this hit the v2 URL which we created with additional parameters
http://localhost:8080/api/v2/customers?firstNameFilter=Ar&lastNameFilter=sp&page=0&size=30

We can see more information being displayed but it doesn't seem to be too relevant. If required, you could do some custom coding to convert the format.

Implementing Page with Filtering and Sorting

Now we will implement sorting to the above logic.

Update Repository

No update in repository is required. We will reuse the previous method that returns a Page<Customer>

Update Service

Add a new method to our service



    public Page<Customer> fetchCustomerDataAsPageWithFilteringAndSorting(String firstNameFilter, String lastNameFilter, int page, int size, List<String> sortList, String sortOrder) {
        // create Pageable object using the page, size and sort details
        Pageable pageable = PageRequest.of(page, size, Sort.by(createSortOrder(sortList, sortOrder)));
        // fetch the page object by additionally passing pageable with the filters
        return customerRepository.findByFirstNameLikeAndLastNameLike(firstNameFilter, lastNameFilter, pageable);
    }

    private List<Sort.Order> createSortOrder(List<String> sortList, String sortDirection) {
        List<Sort.Order> sorts = new ArrayList<>();
        Sort.Direction direction;
        for (String sort : sortList) {
            if (sortDirection != null) {
                direction = Sort.Direction.fromString(sortDirection);
            } else {
                direction = Sort.Direction.DESC;
            }
            sorts.add(new Sort.Order(direction, sort));
        }
        return sorts;
    }

We have 2 methods. 1 public method that we will call from our controller which has 2 additional inputs - sortList and sortOrder.
SortList takes in the list of string on which we need to sort it and sortOrder takes either ASC or DESC.
The second private method creates the Sort.Order list object which is used in creating the Page object.

Update Controller

Add a new endpoint that accepts 2 fields as filter conditions, 2 fields for page functionality and 2 for sort functionality



        /**
     * @param firstNameFilter Filter for the first Name if required
     * @param lastNameFilter  Filter for the last Name if required
     * @param page            number of the page returned
     * @param size            number of entries in each page
     * @param sortList        list of columns to sort on
     * @param sortOrder       sort order. Can be ASC or DESC
     * @return Page object with customers after filtering and sorting
     */
    @GetMapping("/api/v3/customers")
    public Page<Customer> fetchCustomersWithPageInterfaceAndSorted(@RequestParam(defaultValue = "") String firstNameFilter,
                                                                   @RequestParam(defaultValue = "") String lastNameFilter,
                                                                   @RequestParam(defaultValue = "0") int page,
                                                                   @RequestParam(defaultValue = "30") int size,
                                                                   @RequestParam(defaultValue = "") List<String> sortList,
                                                                   @RequestParam(defaultValue = "DESC") Sort.Direction sortOrder) {
        return customerService.fetchCustomerDataAsPageWithFilteringAndSorting(firstNameFilter, lastNameFilter, page, size, sortList, sortOrder.toString());
    }

Test it out

Run the application and this hit the v3 URL which we created with additional parameters
http://localhost:8080/api/v3/customers?firstNameFilter=Ria&lastNameFilter=ss&page=0&size=5&sortList=firstName&sortOrder=ASC

The only advantage here compared to our last version is that we can sort the data now.

Implementing Pagination with HATEOAS

Coming to our final step - implementing hateoas with filtering and sorting.

Update Repository

No update in repository is required. We will reuse the already created method returning Page<Customer>

Update Service

No update is required. We will reuse the method returning Page<Customer>

Create files to support Hateoas

Now to use hateoas we convert the page format to a hateoas PagedModel format. To do so we need 2 additional files

Customer Model (extends RepresentationModel<>)
Customer Model Assembler(extends RepresentationModelAssemblerSupport<,>)

Create CustomerModel

First off we need to create Customer Model that extends the hateoas RepresentationModel which is part of the body of the response we send. We also create getters and setters for the object variables.



/**
 * The CustomerModel class extends the Hateoas Representation Model and is required if we want to convert the Customer
 * Entity to a pagination format
 */
public class CustomerModel extends RepresentationModel<CustomerModel> {
    private Long id;
    private String customerId;
    private String firstName;
    private String lastName;

    public Long getId() {
        return id;
    }
    public void setId(Long id) {
        this.id = id;
    }
    public String getCustomerId() {
        return customerId;
    }
    public void setCustomerId(String customerId) {
        this.customerId = customerId;
    }
    public String getFirstName() {
        return firstName;
    }
    public void setFirstName(String firstName) {
        this.firstName = firstName;
    }
    public String getLastName() {
        return lastName;
    }
    public void setLastName(String lastName) {
        this.lastName = lastName;
    }
}

If we dont want the sequence generated id to be returned we can remove it from this object itself.

Create CustomerModelAssembler

Now we need the assembler which extends RepresentationModelAssemblerSupport and we pass in the from object which is our entity and to object which is our newly created Model. We are forced to override the toModel() method in which we copy the values of the entity to the model.



/**
 * This class extends RepresentationModelAssemblerSupport which is required for Pagination.
 * It converts the Customer Entity to the Customer Model and has the code for it
 */
@Component
public class CustomerModelAssembler extends RepresentationModelAssemblerSupport<Customer, CustomerModel> {
    public CustomerModelAssembler() {
        super(CustomerController.class, CustomerModel.class);
    }

    @Override
    public CustomerModel toModel(Customer entity) {
        CustomerModel model = new CustomerModel();
        // Both CustomerModel and Customer have the same property names. So copy the values from the Entity to the Model
        BeanUtils.copyProperties(entity, model);
        return model;
    }
}

Update Controller

Autowire 2 components required



    @Autowired
    private CustomerModelAssembler customerModelAssembler;

    @Autowired
    private PagedResourcesAssembler<Customer> pagedResourcesAssembler;

and add a new endpoint that will call the Page



    /**
     * @param firstNameFilter Filter for the first Name if required
     * @param lastNameFilter  Filter for the last Name if required
     * @param page            number of the page returned
     * @param size            number of entries in each page
     * @param sortList        list of columns to sort on
     * @param sortOrder       sort order. Can be ASC or DESC
     * @return PagedModel object in Hateoas with customers after filtering and sorting
     */
    @GetMapping("/api/v4/customers")
    public PagedModel<CustomerModel> fetchCustomersWithPagination(
            @RequestParam(defaultValue = "") String firstNameFilter,
            @RequestParam(defaultValue = "") String lastNameFilter,
            @RequestParam(defaultValue = "0") int page,
            @RequestParam(defaultValue = "30") int size,
            @RequestParam(defaultValue = "") List<String> sortList,
            @RequestParam(defaultValue = "DESC") Sort.Direction sortOrder) {
        Page<Customer> customerPage = customerService.fetchCustomerDataAsPageWithFilteringAndSorting(firstNameFilter, lastNameFilter, page, size, sortList, sortOrder.toString());
        // Use the pagedResourcesAssembler and customerModelAssembler to convert data to PagedModel format
        return pagedResourcesAssembler.toModel(customerPage, customerModelAssembler);
    }

Notice how we are using the autowired components. Using the assembler we are converting the Page<Customer> to PagedModel<Customer> by calling the toModel() method

Test it out

Run the application and this hit the v4 URL which we created with additional parameters
http://localhost:8080/api/v4/customers?firstNameFilter=R&lastNameFilter=S&page=0&size=4&sortList=firstName&sortOrder=ASC

Taking a closer look at our output we notice some really beneficial information.

"_links" - this key contains an object with links to the first,self,next and last page. These links can be used from the frontend team to use pagination in the UI.
"page" - this tells us how many elements, totaly elements after filtering and sorting, total pages and current page number.
"_embedded" - contains the list of CustomerModels. The actual data that is converted from the entity to the model.

Conclusion

I spent a lot of time figuring this out when i needed to implement it at a work project and I hope this helps someone out there needing to do the same.
Check out the Github link and play around with it and see if it works in your project as well.
https://github.com/markbdsouza/hateoas-with-pagination
Feel free to leave a comment, if you have any questions or need any help.

Polymorphism in Java - A Quiz on things you probably didn't know

Mark Dsouza — Wed, 26 Jan 2022 12:29:43 +0000

Well if you're reading this, you probably know Java and have worked with a Java Framework. The below post is for all levels of Java 8 developers, anyone looking to improve. I'll try and cover some more advanced concepts of how polymorphism works and some gotchas that exist through some questions and answers!

Questions

See if you can answer the below questions. They are in increasing order of difficulty. Take your time and answer them. Some will have compilation errors/run time errors.
Write down your answer on a notepad and check them all at once. The answers and their explanations are in the second section of this post.

Level 1

Concepts: Polymorphism
Question: Which lines when commented will the code successfully compile?

package com.markbdsouza.polymorphism;

interface Movable{ void move();}
interface Eats{ void eat();}
interface Vertebrae extends Movable, Eats{}
class Dog implements Vertebrae {
    public void move() {
        System.out.println("Dog moves on 4 legs");
    }
    public void eat() {
        System.out.println("Dog eats food");
    }
}
public class Level1 {
    public static void main(String[] args) {
        // which lines when commented will the code successfully compile ?
        Movable a = new Dog(); // line 17
        a.move(); // line 18
        a.eat(); // line 19
        Eats b = new Dog(); // line 20
        b.move();// line 21
        b.eat();// line 22
        Vertebrae c = new Dog(); // line 23
        c.move();// line 24
        c.eat();// line 25
    }
}

Level 2

Concepts: Polymorphism
Question: What is the output of the below program?

package com.markbdsouza.polymorphism;

interface Movable {
   void move();
}
class Fish implements Movable {
   void move(){System.out.println("swims");}
}
public class Level2 {
    public static void main(String[] args) {
        Movable fish = new Fish();
        fish.move();
    }
}

Level 3

Concepts : Overriding + Polymorphism
Question: Given that FileNotFoundException and EOFException are child exceptions of IOException, will the below code compile?

package com.markbdsouza.polymorphism;

import java.io.EOFException;
import java.io.FileNotFoundException;
import java.io.IOException;

class Animal{
    public void move() throws FileNotFoundException { System.out.println("moves in some way");}
}
class Fish extends Animal{
    public void move() throws FileNotFoundException, EOFException {System.out.println("swims");}
}
public class Level3 {
    public static void main(String[] args) throws IOException {
        Animal z = new Fish(); z.move();
    }
}

Level 4

Concept: Polymorphism + Overloading
Question: What is the output of the below program?

package com.markbdsouza.polymorphism;

class Car {}
class Audi extends Car {}
public class Level4 {
    public void drive(Car car) {
        System.out.println("Using a Car");
    }
    public void drive(Audi audi) {
        System.out.println("Using an Audi");
    }
    public static void main(String[] args) {
        Car car = new Car();
        Audi audi = new Audi();
        Car audiCar = new Audi();
        Level4 myTestClass = new Level4();
        myTestClass.drive(car);
        myTestClass.drive(audi);
        myTestClass.drive(audiCar);
    }
}

Level 5

Concepts: Overloading + Overriding + Polymorphism
Question: What is the output of the below code?

package com.markbdsouza.polymorphism;

class Car {
    public int getSpeed() {return 100;}
}
class Audi extends Car {
    // overridden method as it already exists in the Car class
    public int getSpeed() {return 500;}
}
public class Level5 {
    public void drive(Car car) {
        System.out.println("Using a Car and driving at " + car.getSpeed());
    }
    // overloaded version of drive method
    public void drive(Audi audi) {
        System.out.println("Using an Audi and driving at " + audi.getSpeed());
    }
    public static void main(String[] args) {
        // Create 3 objects
        Car car = new Car();
        Audi audi = new Audi();
        Car audiCar = new Audi();
        // Testing overloading and overriding with polymorphism
        Level5 myTestClass = new Level5();
        myTestClass.drive(car);
        myTestClass.drive(audi);
        myTestClass.drive(audiCar);
    }
}

Answers to the Quiz

Level 1

This was simple and straight forward - Line 19 and Line 21 have issues.
Since Dog, in its hierarchy chain implements Vertebrae, Eats and Movable, any of them can be used a reference variable and be assigned to an instance of Dog. But that does not mean they can access all Dog methods.

        Movable a = new Dog(); 
        a.move(); 
        // a.eat(); // Compile Error - Moveable interface has only a move method
        Eats b = new Dog(); 
        // b.move(); // Compile Error -  Eats interface has only the eat method
        b.eat();
        Vertebrae c = new Dog(); 
        c.move(); // Has access to move method because Vertebrae exteds Movable 
        c.eat();  // Has access to eat method because Vertebrae exteds Eats

Go back to Question-1

Level 2

This was a trick question! If you guessed swims you're mistaken. You actually get a compiler error.

move() in com.markbdsouza.polymorphism.Fish cannot implement move() in com.markbdsouza.polymorphism.Movable
  attempting to assign weaker access privileges; was public

In an interface all methods are public and abstract by default. When a class tries to implement the method in the interface, it HAS to be PUBLIC as well. When overridding a method, we cannot make the method more restrictive. So the class implementation of the method has to be public. There are no 2 ways about it.

Go back to Question-2

Level 3

The code does not compile. We get the error

move() in com.markbdsouza.polymorphism.Fish cannot override move() in com.markbdsouza.polymorphism.Animal
  overridden method does not throw java.io.EOFException

We can NOT throw checked exceptions(non Runtime exceptions) that are NEW or BROADER than those declared in the original overridden method. In this case the overridden method had a FileNotFoundException and the overriding method additionally has an EOFException. This is not allowed.
The overriding method can also throw narrower or fewer exceptions just not more! Note that the overriding method can however throw any unchecked (Runtime exception) that it wants.
Go back to Question-3

Level 4

You might be surprised to see that the output is

Using a Car
Using an Audi
Using a Car

The choice of which overloaded that should be called is decided at compile time itself. The reference type (Type of the reference variable and not the actual type of the object it is pointing to) decides which overloaded method is invoked.
When we use a reference type of Car, during compilation it is decided which overloaded method is invoked. This is why Using a Car is picked up
Go back to Question-4

Level 5

Similar to Level 4 with a slight twist!

Using a Car and driving at 100
Using an Audi and driving at 500
Using a Car and driving at 500

If you don't believe me run the code yourself and check !
Well you probably guessed line 1 and 2 correctly. In Line 3, what happens is even though the actual Object at runtime is an Audi Object and not just a Car object, the choice of which overloaded method that should be called is decided at compile time itself and not runtime. The reference type (Type of the reference variable and not the actual type of the object it is pointing to) decides which overloaded method is invoked. - All similar to question 4
So now we know that Car audiCar = new Audi(); will invoke the below method

public void drive(Car car) {
        System.out.println("Using a Car and driving at " + car.getSpeed());
    }

We see that car.getSpeed() is called in the method and the compiler will throw no error during compilation because this method does in fact exist in the Car object.
But now during run time, the method that gets executed is the object type (type of the object that is used and not type of the reference variable). For our Car audiCar = new Audi(); the type of the object is Audi. So during runtime the code that gets invoked is the below method

public int getSpeed() {return 500;}

Merging these 2 concepts together of overloading and overriding we get the 3rd line as Using a Car and driving at 500 !! Hope that all made sense
Go back to Question-5

You will probably not use this sort of overloading + overriding with polymorphism in real life, so don't kill yourself if you didn't know the answer to the questions

I hope you learnt something from this article!! If you know any java devs, ask them these questions and see if they get all of them right!!

Spring Cloud Config Server - Encryption and Decryption

Mark Dsouza — Tue, 07 Sep 2021 18:01:21 +0000

While using Spring Cloud Config Server, we also have the feature to encrypt sensitive information that is otherwise stored as plain text in our external git repository. This prevents anyone who has read access to the external repository from accessing confidential information and is highly highly recommended if you are using a Spring Cloud Config Server.
Note: If you haven't set up Spring Cloud Config Server yet check out my other article on how to set up a spring cloud config server.

There are 2 types of encryption provided

Symmetric - Quicker to setup and easier to use. Uses a unique alphanumeric set of characters to both encrypt and decrypt the properties.
Asymmetric - Higher security and harder to set up. It requires you to use java command line utility called keytool which creates a key store file (containing private and public keys). Encryption is done with the public key and the private key is used to decrypt the data.

Pre-requisite - JCE

If you use OpenJDK11 you can skip this step as JCE comes along with it.
If you are on Java 8, Go ahead and download JCE which we will need to encrypt data : https://www.oracle.com/in/java/technologies/javase-jce8-downloads.html
This includes 2 jars, which need to be added to our runtime environment : local_policy.jar and US_export_policy.jar
You can read the README and proceed with install JCE. All you basically need to do is navigate to your JDK installation path and place these 2 jars under lib/security. Once done, make sure you close all java apps and reopen them. If you still see errors while performing the below steps, try and restart your machine once.

1) Using Symmetric Encryption

I have set up my config server, so I can do a GET request to : http://localhost:8888/all-ms/default
where all-ms is an micro service name and default is the spring profile to get the configured values in my external git repository

1.1) Enable Asymmetric Encryption

To enable encryption all you need to do is add a property to application.properties of your config server.



encrypt.key=APODNU3093r2rbjzxcn09u213asdhy08WRFH

Your key can be any long random alphanumeric string.

1.2) End points for encryption/decryption

Now let us encrypt your data first. Launch your config server spring boot application and send a HTTP POST request to your server with endpoint /encrypt and in the body send the data that needs to be encrypted.
localhost:8888/encrypt
The response will contain the encrypted value in the body.

You can also check the decrpytion by sending a HTTP post request to localhost:8888/decrypt , this time the body should have the encrypted value.

To use this encryption in your application, check the secret data that you have in your remote git repository. This could be a data base password/a token.
e.g. spring.datasource.password=DatabasePassword
I will use the above endpoint to encrypt this.

1.3) Configure your repository

Now in my external properties file, I will replace



 spring.datasource.password=DatabasePassword

with



spring.datasource.password={cipher}4ae0e7516ba2688519d46a09ec147a96badd9c9e34ffa5c778c26a9d608ad58d2cbfba7a38c87934692ceff4f2ca4bfc

Note the prefix of {cipher} which tells your server that decryption needs to be done for this particular property. In my repo, I am using a JWT token. I encrypted the token and set the value in my repo to



token.secret={cipher}a0098f22f7fe1d0ca4dc0e03a95c9a721173b4486e9e51c9180c5afb3ff1a773dcba1fc3f4fcbc7060e7d93a8d312d1e

1.4) Test it out

When I test out my config by accessing my properties, it automatically decrypts my token given the encryption key in the config server even though in my external repository I have stored the encrypted value.

2) Asymmetric Encryption

Firstly we need to generate a keystore file which contains the 2 keys. The public key is used to encrypt information and the private key is used to decrypt information.

2.1 Creating the key store file

In your command prompt, type out



keytool -genkeypair -alias myApiTutorialKey -keyalg RSA -dname "CN=Mark Dsouza,OU=MyApiDev,O=markbdsouza.com,L=Bangalore,S=KA,C=IN" -keypass a1b2c3d4 -keystore myApiTutorialKey.jks -storepass a1b2c3d4

Here a1b2c3d4 is the password and myApiTutorialKey.jks is the file created.
If you can’t find keytool on executing, go to the bin folder of your JDK and you should see it there.
If you are on windows, make sure you run command prompt in admin mode if you face any issues.
Please check online for more details on configuring this
https://docs.oracle.com/cd/E19683-01/806-4078/6jd6cjru7/index.html
This will create a myApiTutorialKey.jks with the password a1b2c3d4.

2.2 Add the file to your project

I now place this file in my resources folder to easily access this file in my spring boot application.

2.3 Enable Asymmetric Encryption

Now for the configuration in application.properties of your spring cloud config server add the below properties.



encrypt.keystore.location=classpath:key/myApiTutorialKey.jks
encrypt.keystore.password=a1b2c3d4
encrypt.keystore.alias=myApiTutorialKey

Make sure you use the same password and alias you have given when creating the keystore file.

2.4 End points for encryption/decryption

Launch your app and similar to the symmetric way of doing things, you can send a POST request to localhost:8888/encrypt with the body being the value you want to encrypt.

Similarly we can decrypt using the decrypt endpoint.

2.5 Configure your repository

To encrypt data in our remote git repository, add {cipher} as a prefix for the property so that the server knows that it needs to be decrypted when being fetched.

2.6 Test it out

When I test out my config by accessing my properties, it automatically decrypts my property with the key store file.

And that's about it. It honestly is a very short process to encrpyt your data and I hope this tutorial helped you on your spring cloud journey!

Getting started with Apache NiFi

Mark Dsouza — Mon, 16 Aug 2021 18:19:58 +0000

If you're looking to get a quick understanding about Apache NiFi, how you can get started, some key concepts, tips and tricks, and other resources on the open source tool, you've come to the right place

What is Apache NiFi and why should I use it?

"An easy to use, powerful, and reliable system to process and distribute data." - NiFi official website
NiFi is a great open source tool. With NiFi you can build a data flow from anywhere to anywhere - local file system, cloud, HDFS, nosql, rdbms, kafka. Litereally Anywhere to Anywhere.
If you have a fixed flow of steps with no computation in between or complex business logic that you need to do - NiFi makes a lot of sense.
A simple use case: You want to transport your data at the end of each day from a DB to say HDFS - you can easily do this with NiFi. This is a fixed set of steps that need to be done.

NiFi is EASY to use!

You do not need to be a technical person to use NiFi. It has a very robust interactive web browser UI and all you need to do is drag and drop in processors (actions to be done using data) and link them to each other. All you need to know is the configuration of that processor.
For instance, if you are connecting to the Database, you need to know the details of how you can connect to your Database.
For certain use cases you might need to know a little bit of the NiFi Expression Language. After building a few flows this becomes pretty easy to understand.
Docs link : https://nifi.apache.org/docs/nifi-docs/html/expression-language-guide.html

Getting NiFi running on your machine

Where do I download NiFi?

The team releases updates every few months. So get the latest version here.
Make sure to download the binary : https://nifi.apache.org/download.html
There is even a docker image if you have docker installed on your machine
https://hub.docker.com/r/apache/nifi/

Launching NiFi

For windows users navigate to bin and run the run-nifi.bat file. You need to have java installed on your machine and added to the PATH variable.
For linux or mac run the command - ```bin/nifi.sh

run

This will launch your NiFi server on https://localhost:8443/nifi/
![Login Screen](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/f4vr2iyq9c713rqm7gj3.PNG)To get your credentials, go to logs/nifi-app.txt and search for your username and password.
![Login credentials in the log file](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/3080ch387gtqeaa0i58m.PNG)After a successful log in, you will see your home screen where you can start building
![Nifi Home Screen](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/z0njbx7w88xqbfipohey.PNG)For any issues : https://nifi.apache.org/docs/nifi-docs/html/getting-started.html

# Key Features of NiFi
## Flow Files
The data itself is referred to as a Flow File. It has 2 key parts. The content (raw data of the file) and Attributes which would contain some metadata about the flow file.
For example when you send a HTTP Request, the body of the HTTP request would be the Flow File Content and the headers would be the attributes.
The flow data does not get copied each step. It is temporarily stored on the NiFi server and an auto clean up happens once the flow file is finished processing. The flow file works sort of like a pointer as the file flows through your entire flow. 
## Processors
A processor is the task that you want to execute on the data/flow file.
Do you want to Read from HDFS Or read a file from a location on your machine? Do a HTTP GET Request? Generate a File? Insert into a NoSQL or RDBMS Database? Each of this is can be done with a processor. Overall there are 288 processors as of today that you can use. ![All Processors](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/f6ai94p0qxx3hkss0j5g.PNG)And if you don't find the EXACT processor you need? You can write your own with custom code as well. The NiFi teams keeps adding new processors as well.
Once you drag in a processor, it will need to be configured. For example If you are generating a file - you can provide the data in the file. ![Configure Generate Flow File](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/o3juunj1x29r1ds8kryw.PNG) Not all configuration is mandatory. 
If you miss a mandatory configuration you will see a warning across the processor with descriptions of what is missing
![image](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/jcnsb4097gvrsb5x6hxe.png)
Note: Any processor can use any flow file which is what makes NiFi so powerful. 
## Linking processors with connectors
We can connect processors to each other using a connector. Depending on the processor, you may have multiple outcomes of the processor doing the intended action.
![image](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/ism1w8539c1taomrub8y.png) In the example, I have a consume Kafka processor and there can be 2 outcomes - parse.failure or success. You can easily create different flow File paths for either outcome. Maybe you want to log the failure and not proceed further and if it is a success you might have 2 3 additional steps. 
![image](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/d3w3tspn6xzpqrzub4m4.png) If a processor is the end of a flow, we need to explicitly tell NiFi that it is the end. Otherwise NiFi expects us to keep going and will throw a warning message. 
![image](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/6hqd3nn13y0cfh5rn8kh.png)Moment you link 2 processors, there is a Queue created. If your 2nd processor is paused or has an error in the configuration, the flow Data will enter the queue in this connector. 
TIP: If you are linking 2 different relationships between processorA and processorB draw 2 different lines creating 2 different connectors. This specially helps in debugging.

## Controller Services
Controller Services in NiFi are centrally stored. If you have 10 different calls for various CRUD Operations depending on some business logic, you do not need to provide 10 different connections to the DB. All you need to do is create the Service once and then reference the service for each of your processors
![image](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/76zprv2siehrqxkixmp4.png) You will also need to provide a jar file for a driver (if required) so NiFi can make the appropriate connection 

## Other concepts
**Process Groups** - You can group a part of a flow helping you segregate different parts of your flow logically
**Input Port/Output Port** - An input to your Process Group is in the form of an input port (not the server port). This is how data enters the Process group. Similarly, in the group, there can be multiple path terminating in multiple output ports. One group can have multiple input ports.
**Templates** - You can create a template containing the entire or part of your flow. This can then be shared with other developers. You can save/import/delete templates.
**Funnel** - You can combine data from different sources with funnel
**Label** - For better visualization. Has no impact on the data.

## Conditional Statements in Nifi
### If Else Logic based on attributes/content
The most useful processor to implement dynamic behavior is the RouteOnAttribute/RouteOnContent processor. This basically helps you implement IF ELSE statements while the data flows. Say NiFi is reading a HTTP request, you can do some routing based on the type of request (a different route for POST and DELETE to the same HTTP endpoint). 

### For Loops in Nifi
NiFi isn't great for working with loops. But there are ways to do it. You can split one file into many files based on some condition. Say you have a list of fileNames as an attribute. FileA,FileB,FileC. Now you want to create 3 files and place them in a target folder. You can split your first FlowFile and create 3 flow files(or X Flow files) with each flow file with an attribute of the individual file name. 
But things that might be just 2 3 lines in JS, Java or any other language, might take a little thinking to implement using NiFi Processors. Example: I had an array of strings. and wanted to remove 1 string based on some business logic - this was extremely hard to do and took way too many steps to implement which is otherwise 1 array filter function in any programming language. (You can opt to go for custom coding in NiFi as well which is ideal for these more complicated scenarios) 
Note: you can remerge your flow files after splitting them up as well.

## Scheduling 
There are 2 ways of scheduling - Time Driven or CRON driven 
![image](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/zgfeqhewdhj19elzs7uv.png)

## Data Provenance
Data provenance is the documentation of where a piece of data comes from and the processes and methodology by which it was produced. NiFi lets you easily track data at the application level
![image](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/34eqogg9y8lxf7e2zzwp.png)![image](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/7znqn8tl4qq8dzv1u1ex.png) 
## Tips on debugging
Let’s say you have a simple flow but you need to see the data that is coming to create your flow easily. You can create your connections and stop the processor you haven’t configured yet![Queue](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/1qwspvm353rvz1viybll.PNG)I have a simple Generate log file connected to a Log Attribute
As you can see, the connector has a Queue count displayed.
![Queue Options](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/v8c0d8s0puvjstd05kho.png)Right click to view the Queue
![List Queue](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/3s3zmt8a1zc1x0obdfiy.PNG)Now you can see all flow files that are currently in this queue. You can view the attributes and metadata of the file and even see the raw content.    
![Queue View](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/i8qyfc8l50yefqrd1piz.PNG)
![Queue Content](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/1molym42y6t3bwvjuajk.PNG)This helps you code and see the data as it progresses through the entire flow. 

## When not to use NiFi
NiFi isn’t great for computation. Basic transformation? That’s fine. But if you want to do something like take a SUM of a column or Aggregations - NiFi isn’t meant for that.

## Other Resources
### Where can I get help?
I have to point you to the official docs first
https://nifi.apache.org/docs.html
https://nifi.apache.org/docs/nifi-docs/html/getting-started.html
https://nifi.apache.org/developer-guide.html
https://cwiki.apache.org/confluence/display/NIFI/FAQs
For a good quick read
https://www.guru99.com/apache-nifi-tutorial.html
Though these docs are pretty great, there aren't any real examples of how to use a processor. The 2 great forums you'll find your answers are Cloudera and Stack overflow. I found a lot of useful use cases in the Cloudera forums. So don't overlook those results in your google searches

### Are there any good videos on NiFi?
When it comes to tools, I find it a lot easier to grasp things by watching unlike code which can be read since there is UI interaction and not static code. 
A set of well curated videos touching many essential features of NiFi with plenty of hands on: https://www.youtube.com/watch?v=VVnFt54jUQ8&list=PL55symSEWBbMBSnNW_Aboh2TpYkNIFMgb
A long (bits can be skipped) overview with hands on: 
https://www.youtube.com/watch?v=fblkgr1PJ0o

If you have any other suggestions/content for NiFi beginners, please share them in the comments section.
Happy Learning !!

Write better Java Code

Mark Dsouza — Sat, 22 May 2021 13:56:17 +0000

Clean Code is under rated. If you've ever got frustrated with the way some code was written or spent way too much trying to understand what some function is doing, it probably means that code could be written better.

You can surely build great things without following the principles of clean code. But it becomes a LOT LOT easier for you and your team to maintain the same code if it is easily understandable and written well. This is a continuous process that should be taken into account with each line of code you write and is not a retrospective action you work on at the end of the week/sprint.

Java projects especially can get really big and complex, and clean code helps your team build, enhance and fix bugs faster and more efficiently. And if you're a beginner, this can be a lot to take in but it surely helps to get started off on the right foot.

In the below sections, I'll go over points/best practices to keep in mind while writing java code.

All about naming

Thumb rule - always be as specific as possible with all names.

Naming Classes

Class Names should ideally be nouns or can be abstract and are written in PascalCase. Example: AudioTransmitter, SalaryCalculator.
As much as possible try and make it specific to what your use case is. If you have a FileReader class to read xml files, a better way of naming it would be XmlFileReader. That way if there is an enhancement later to have a JSON or CSV FileReader, you can have separate classes for it instead of having all the logic in one class.

Naming Variables

Written in camelCase, variable names should be descriptive yet concise.
Avoid naming variables with alphabets such as a,b,x,y,z unless it is a temp variable like an index. *When using booleans start it with 'is'. Example: isValid instead of valid.
For Constants use ALL_CAPS_WITH_UNDERSCORES. Example: MAX_CUSTOMER_COUNT Basically, if we see the variable in a random piece of code, we should be able to understand what type of data it holds. Example: customerAccountDetails

Naming Methods

The name of the method should describe 'what' is done by the method (it doesn't matter how). eg: generateSalaryReport(). The method tells us it is creating the salary report. We don't care how it is being done internally.
A method should ideally have only 1 purpose. If the method does more than 1 thing, you should ideally split it up such that each method does exactly what it says.

All about Methods

Avoid writing huge methods. Each method should be specific and do just one thing. Keep in mind if multiple methods have the same code in them, you are repeating yourself. Instead create a small private method in the class and call the method wherever the duplication is observed.
When returning an object, avoid returning null. When we do so we are assuming that all clients of the method expect null and handle it. If they are aware they need to add an extra try catch every single time your method is called. This is just repetitive code. Instead return an empty Collection or throw an error if appropriate.
Avoid having too many arguments in a method. Too many arguments also increase complexity and might show that you need to split your method into multiple parts each doing a small part of the logic. Or are a few of those arguments very cohesive and do they actually need a class of their own? If yes, it might mean more code and an extra class but it's a lot easier and more scale-able.
Avoid flag arguments which are boolean in nature. This points to the fact that you have too much logic for the function again. Split the true and false part into two separate methods and call them accordingly.
Fail Fast & Return Early. Check for invalid arguments right at the start and throw an error instead of executing code that will ultimately fail for the same reason. This will save on execution time and make it easier to debug. Return early is the way of writing methods so that the expected positive result is returned at the end of the function and the rest of the code terminates the execution (by returning or throwing an exception) when conditions are not met.

Constructors

If you have a Class that has many variables that need to be assigned and multiple combinations of these variables therefore needing many constructors, a good way to organize your code is to use the Builder Design Pattern. This is a Creational design pattern and you dynamically assign variables when creating the object.
Another way to avoid duplication of code in constructors is using Constructor chaining, where you call one constructor from inside the other.

All about classes

Try and follow the Single Responsibility Principle (from SOLID principles) - the class should have only one reason to change. This results in smaller well defined classes.
It should be cohesive - where fields and methods are highly related to each other. Higher the cohesion, the better. This will also help you avoid classes with way too many variables.
Follow Loose coupling - a change in one class requires minimum/0 changes in another class. This reduces the inter dependency between classes. Most frameworks follow loose coupling for this reason.
Use Dependency Injection if possible.
Uses interfaces as much as possible. If you ever want to change the implementation, it becomes a very quick and easy change. Allows you to be very flexible in the future. Example use List<String> customerNames = ArrayList<String>(); instead of ArrayList<String> customerNames = ArrayList<String>();
Encapsulation - keep variables private and any methods that need not be public, private. Even for getters and setters, only create them if it needs to be accessed from outside the class.

Exception Handling

Avoid catching Throwable or Exception Classes. As the programmer, you need to know when Errors(Example: OutOfMemory) happen or when Runtime Exceptions that you would want to know when it happens.
Do not catch and handle Null pointer exceptions in the catch block, instead write code to prevent the null from happening itself(if possible) and if not just do a if(obj != null).
Exceptions when caught should be specific. Only if you expect an error to happen, you should try and catch it. Also Java 7 allows you to catch multiple exceptions in one block with catch( IOException | SQLException e) and handle all exceptions the same way.
When catching an exception, make sure you log the details(with a logging framework) of the exception and surely do not keep it empty.
When catching an exception make sure you mention why the exception happened. Example if it is an Illegal Argument Exception, you can mention what type of value the argument had for the exception to happen. This helps with debugging.
Use try with resources(if using java 7+) wherever possible to avoid additional code.
Do not catch and handle exceptions you do not expect to occur. You should add catch blocks only for those exceptions that you are aware could happen for a reason. Anything else is extra avoidable code.

Comments

Base rule: Never comment code out.

A comment should only increase our understanding of code.
Avoid stating obvious points in comments that you can easily understand by reading the code.
Keep note that that comments are rarely updated when code is updated. This can cause issues. As a team, always make sure any updates to code, is also reflected on comments or remove the comment altogether in such cases.
Writing comments in the form of JavaDocs for public methods that will be used by clients of that class is extremely valuable.

Styling and Formatting

Most Java IDEs have an inbuilt code formatter which helps you do this with a shortcut.
The Google Java Style Guide and Java Style Guide by Twitter, both have some best practices when it comes to writing java code.
Make sure you use line breaks and brackets wisely, making the code readable. Less lines of code doesn't make it more readable.

How to check code quality?

There are plugins that help you check if code is being reused, written as well as they can be. SonarLint their very tag line is - Fix issues before they exist. It is available on all popular Java IDEs and you can do a full project scan to see how the code can be improved.
SonarQube is an open source tool that can be added to your CI/CD pipeline and is extremely popular to check the entire project's code quality.

At the end of the day, as a team, decide on what standards you would like to adhere to and stick to it. If each member follows his own best practices, it helps no one. It often helps to keep a document for each project with the coding standards followed by all and is shared with any new members joining the team.
Make sure when you are reviewing code with Pull Requests, your teammates are adhering to the rules agreed upon (and they make sure you do as well) so your overall code is consistent throughout the application.

If you have any additional pointers to add that you feel have helped you and your team out, leave them in the comments down below.

JS: Sort an Array of Objects on multiple columns/keys

Mark Dsouza — Thu, 22 Apr 2021 06:03:50 +0000

Let's see how you can completely configure the sorting of an Array of objects. Let's say we have the below data set for our entire example.



let objs = [
  { name: 'Mark',
    age: 30,
    RollNo: 'R01'
  },
  { name: 'Anne',
    age: 20,
    RollNo: 'R02'
  },
  { name: 'James',
    age: 40,
    RollNo: 'R03'
  },
  { name: 'Jerry',
    age: 30,
    RollNo: 'R04'
  },
  { name: 'Lucy',
    age: 30,
    RollNo: 'R05'
  },
  { name: 'Mark',
    age: 30,
    RollNo: 'R06'
  },
]

Looking at the raw data we have with a console.table(objs)

Single Column Sort

Now say we want to sort this data across one column. The best way to do this is the sort() method. Check out the documentation.
An example from there down below on a simple array of Strings



const months = ['March', 'Jan', 'Feb', 'Dec'];
months.sort();
console.log(months);

The output is ["Dec", "Feb", "Jan", "March"]
This automatically sorts the original array in alphabetical order and returns the original array as well on calling sort().