Forem: Hoai Nam NGUYEN

Integration tests and microservices applications

Hoai Nam NGUYEN — Fri, 08 Nov 2019 15:45:55 +0000

Test is the norm

Testing is the de facto standard in software development. Just take a look at different popular projects on GitHub, and you may see that they all have tests.

Or search books about testing, and we can find many of them.

In most of the job interviews, people always ask me about whether I test my code. There is no doubt that good developers must be very skilled in testing code.

Sad reality

Testing is hard. Because writing correct code is hard, and writing code that is easy to test is even harder. That's why even most developers I've talked to say that testing is essential, most of them do not test enough!

Junior developers made another extreme mistake when they learned about testing, and they want to prove their ability to test their code. As a result, they added tests everywhere. The real problem is that they test too much unimportant code and let the most valuable, sensible, and sophisticated code untested. Let's take a look at the example below.

def main():
    args = parse_args()
    result = compute(args)
    return result

I see a senior developer added ten tests for the function parse_args and zero for the function compute. If he's not a junior developer, then he's must not be an honest person. He wants to show that he has tests, even it just tests 5% of the logic.

In one company, I've even seen developers test code during the demo! Let's say it is about a Spark streaming application that takes some messages from Kafka and send it to a database. During the demo, the developer said: "Hey, you see, I've just created a message and send it to the Kafka broker, and you can see that in the logs and now when I go to the database, I can see the value persisted". This developer did not write any tests for this feature. The feature is only valid during the demo, and any change after that can make it broken.

Legacy code

When we talk about legacy code, we think mostly about projects over ten years old, with developers come and go. It's hard to change anything because there are not enough tests, and it works in production.

But I've seen a project that is not yet used in production and contains lots of legacy code with no tests.

How can we refactor legacy code without breaking anything, when there are almost no tests in place? Then the integration test is the solution.

Integration tests

The integration test is different than the unit test by these aspects:

Unit test is for a function. The integration test is for the whole application
Unit test uses mock to fake the dependencies (database connection, web request, etc.), the integration test uses real components.
Unit test is very cheap and quick. The integration test is expensive and slow

Unit tests and integration tests are complimentary.
In the integration test, the hard work is to create a script that does these things:

Preparing of the input data.
Preparing of the external component like database or message broker (must be in the clean state).
Preparing of the configuration, usually different from the one in production.
Launching of the jobs.
A system that waits for the job to complete then show any errors occurred.
Process that cleans the data generated by the test, if it's successful.

In a microservices architecture, there are many applications working together in the data pipeline. The integration test should launch one after another and verify all the intermediate results.

The most challenging feature of the integration test is the independence of the executions. Developers and build machines should be able to run tests at the same time. Therefore, the folder that contains all files needed during the test like binary executor file, configuration, and data should have a random name. In my project, I chose UUID to generate a random name for each execution.

Combination of integration tests and regression tests

Until now, the tests I've described above only prove that these applications finished successfully. But what about the values it produces?

We need a script that checks whether these values are corrected. For example, the length of input and output must be the same. But as I mentioned at the beginning of this article, we often deal with projects with not enough tests, and we have to start to write some tests before refactoring. Most of the time, the product owner does not know all the specs, and the original developers of the project are not available. So how do we check the output values of the tests?

It turns out that it is not hard at all. We can take the idea from the regression tests that I described in an old post. The idea is simple: we run the tests for the first time, and copy all the output values into another folder. Then we write the script that compares the result of the new tests to the reference values. These tests make sure that we don't change any values during refactoring. When we have new features that modify the values, then we can safely copy these new values into the references. Each time we do that, we have to make sure the PO is ok with these new values.

Conclusion

Most developers agree that tests are essential. But in many projects with even experienced developers, I don't see enough tests. If there are not enough tests, the new developers are afraid of making a change to the code base. The solution is to create global integration tests that run all the applications one after another, then compare the output values to some reference values that we saved earlier. The challenge of these tests is the ability to proceed independently and concurrently.

I hope this article helps you to start writing more tests in your project.

Create a Validation DSL from scratch with scala

Hoai Nam NGUYEN — Sat, 23 Mar 2019 18:11:08 +0000

Introduction

If I have to pick any subject to talk about functional programming, I will choose the validation. Any developers who do not need to validate some input data and provide all the errors associated with them?

It's a common mistake by junior developers to ignore the error handlings process. They take the correct input, test the program and are very happy when they see it works, then push to production.

The better way to handle the errors is to catch all possible errors in the main program. It's just not ideal because if you have a big configuration file with hundreds of parameters, then you have to correct one error after another. We want to see all the errors at once. Let's explore how functional programming techniques can solve this problem nicely.

Simple implementation

In functional programming, we have to think about the type that describes best our problem. The Try or Either data type is not the best candidate because it can only carry one error value. So let's write our Validation data type.

sealed trait Validated[E, A]
case class Invalid(errors: List[E]) extends Validated[E, A]
case class valid(result: A) extends Validated[E, A]

We defined the data type Validated with two type parameters E and A. If it can be an invalid result that carries a list of errors of type E, and if it's a valid result, it carries the result of type A.

Wait a minute; there could be a problem here. We know that a List could be Empty or having something. In this case, there is a possibility that this value exists: Invalid(Nil) and it's very wrong. How on earth an invalid value that could not provide any error values? If there are no errors, then it must be a Valid value.

Let's quote from Yaron Minsky

Making illegal states unrepresentable

Let's make a data type called NonEmptyList that can never be empty.

case class NonEmptyList[A](first: A, rest: List[A])

Another improvement to be considered is that we have to make the data type covariant by both E and A to accept any subtype of E and A

Here's the complete definition of our data type

case class NonEmptyList[+A](first: A, rest: List[A])

sealed trait Validated[+E, +A]
case class Invalid(errors: NonEmptyList[E]) extends Validated[E, A]
case class Valid(result: A) extends Validated[E, A]

Let's make an example to be easier to understand. Imagine we have an input in the form of Map[String, String] and we want to validate and convert it into a Persontype defined as follow:

case class Person(name: String, age: Int)

We have two fields to parse from the Map. The first field is the name of type String. We have to define our exception data type. For the sake of example, we can specify only one error data type.

case class ErrorParsing(message: String)

type Validation[A] = Validated[ErrorParsing, A]

Let's write our code to parse the fieldName String value from the input

def validateString(input: Map[String, String], fieldName: String): Validation[String] = {
    input.get(fieldName) match {
      case Some(v) => Valid(v)
      case None => Invalid(NonEmptyList(ErrorParsing(s"Field $fieldName does not exist"), Nil))
    }
  }

And another function to parse the Intvalue from the input

 def validateInt(input: Map[String, String], fieldName: String): Validation[Int] = {
    input.get(fieldName) match {
      case Some(v) => Try(v.toInt) match {
        case Success(int) => Valid(int)
        case Failure(exception) => Invalid(NonEmptyList(ErrorParsing(s"Cannot convert $v to String"), Nil))
      }
      case None => Invalid(NonEmptyList(ErrorParsing(s"field $fieldName does not exist"), Nil))
    }
  }

Now we have a value of Validation[String] and a value of Validation[Int]

def validatePerson(input: Map[String, String]): Validation[Person] = {
    val validatedName = validateString(input, "name")
    val validatedAge = validateInt(input, "age")

    (validatedName, validatedAge) match {
      case (Valid(name), Valid(age)) => Valid(Person(name, age))
      case (Invalid(nameError), Valid(_)) => Invalid(nameError)
      case (Valid(_), Invalid(ageError)) => Invalid(ageError)
      case (Invalid(errorName), Invalid(errorAge)) => Invalid(
        NonEmptyList(errorName.first, errorAge.first :: errorName.rest)
      )
    }
  }

This code does the job. But somehow we can see that this code is not reusable. If we have another case class called Position(x: Int, y: Int), we have to rewrite the same code again and again.

This leads us to write a more generic function called validate2 that takes 2 validations with different types and a constructor for a third value.

def validate2[A, B, C](first: Validated[ValidationError, A], second: Validated[ValidationError, B])(f: (A, B) => C): Validated[ValidationError, C] = {
    (first, second) match {
      case (Valid(a), Valid(b)) => Valid(f(a, b))
      case (Invalid(firstError), Valid(_)) => Invalid(firstError)
      case (Valid(_), Invalid(secondError)) => Invalid(secondError)
      case (Invalid(firstError), Invalid(secondError)) => Invalid(
        NonEmptyList(firstError.first, secondError.first :: firstError.rest)
      )
    }
  }

Introduction to cats

The function validate2 is very similar to the function productdefined in an applicative functor.

trait Applicative[F[_]] extends Functor[F] {
    def product[A, B](fa: F[A], fb: F[B]): F[(A, B)]
}

And this is way more generic than our code that only works with the type Validated[ValidationError, A]. This code can work with a lot of higher kind type that we already know (Option, Either, Future, List). Using a library like cats that defined all these functions is very convenient. Using the cats library, we can have automatically map2, map3 and even mapN

Take a look at validation in cats and see the very convenient syntax to define a validation:

def validateForm(username: String, password: String, firstName: String, lastName: String, age: Int): ValidationResult[RegistrationData] = {
    (validateUserName(username),
    validatePassword(password),
    validateFirstName(firstName),
    validateLastName(lastName),
    validateAge(age)).mapN(RegistrationData)
  }

Conclusion

In this article, I've introduced the needs and a simple implementation of the validation. We quickly arrive at the point that we need a more powerful abstraction to reuse code. This is how a generic library like cats or scalaz can shine.

Designing a comparator library for regression test

Hoai Nam NGUYEN — Mon, 22 Oct 2018 14:43:04 +0000

Regression test

Let's quote the definition of regression testing from Wikipedia:

Regression testing is a type of software testing which verifies that software which was previously developed and tested still performs the same way after it was changed or interfaced with other software. Changes may include software enhancements, patches, configuration changes.

This kind of test is prevalent in the numerical library. In such projects, the developers want to make sure that the output of the program (which are numerical values) does not change. The benefits are:

Safe refactoring: Developers don't be scared of moving code around, renaming things as long as the tests passed
Controle: Developers can control changes they made to the library. They expect to see only the differences in some part of the library

Regression test workflow

We can see our software as a function that takes an input and produce an output. The regression test does not guarantee the correctness of the output, and it only tells if the output was modified since the last run.

This common implementation of regression test consists of:

A collection of input of the program. It's usually under the form of text file.
A collection of output where output = f(input). The output is persisted somewhere
When the test run, it goes through all the input in the collection, apply the function and compare the result with the output stored in the file system.

If the test pass, then congratulation, there is nothing to do. However, what if it fails? In this case, you should analyze the differences between the reference value and the new value. The quality of the comparator is essential to speed up this process. Let's have a look at an example:

Example 1:

You have 2 objects that are a list of thousands of elements. If you use the standard scala ==, you only have the True or False as a result. However, there are 1000 elements, and you want to see which elements are modified. You want to have a message like this instead: 512 / 1.2 != 1.3 where 512 is the position of the element

Example 2:

You have 2 objects of type Toto defined by:

case class Toto(x: X, y: Y, z: Z)

The comparator should tell which field contains the differences between 2 objects

x / left != right

The ability to compare 2 case class is particularly helpful. If the calculation represents significant steps, we want to output all the intermediate steps so we can control when the bug is introduced into the system.

When comparing two doubles, the comparator should take into account an acceptable error because we always have the numerical error in the numerical calculation: Only a little rearrangement of the function can result in some 1e-12 difference.

Now there are 2 possibilities when the regression test fails: You either see what you have done is wrong and corrected it or you think the new value is the better value. For the latter situation, you have to update the references values to the latest version.

Updating the reference values are risky, and you should be sure about the new values. A CVS should track the reference values.

Designing the comparator

The comparator is the center part of the integration tests and the purpose of the library called comparator.

The API

In this section, I describe step by step the way I've followed in implementing this library.

The very first question to ask is of course: What if the API? We want to compare 2 values, and the output is the list of differences between these 2 values

Comparator.compare[A](left: A, right: A): List[Diff]

The function signature tells us that it takes two values of the same type and return a list of differences between these values.

The Diff must provide 2 facts: The position of the difference and the difference. For example, when comparing two lists, it should specify the position of these different elements

case class Diff(paths: List[String], diff: ComparatorDiff)

This class can output something like: data / 0 / x: 5 is not equal to 6

The ComparatorDiff can be a String but it will be difficult to test... That's why I've chosen to use some type system.

sealed trait Side
case object Left extends Side
case object Right extends Side


sealed trait ComparatorDiff

case class SizeDiff(left: Int, right: Int) extends ComparatorDiff
case class DoubleDiff(left: Double, right: Double) extends ComparatorDiff
case class KeyNotExist(key: String, side: Side) extends ComparatorDiff
case object TypeDiff extends ComparatorDiff

Since we have to specify the path, we should have a method that adds new path into the child path

def compareWithPath(path: String, left: A, right: A)(implicit err: AcceptanceError): List[Diff] = {
    val temp = compare(left, right)
    temp.map { diff => diff.copy(paths = path :: diff.paths)
    }
  }

Since we choose List to store paths, the prepend operator is O(1)

Comparator of some useful types

The library provides some essential type classes

implicit def doubleComparator(implicit err: AcceptanceError): Comparator[Double] = (left: Double, right: Double) => {
    val abs = Math.abs(left - right)
    val diff = Diff(Nil, DoubleDiff(left, right)) :: Nil
    if (left == 0.0 || right == 0.0)
      if (abs > err.absolute) diff else Nil
    else {
      val rel = abs / left
      if (rel > err.relative) diff else Nil
    }
  }

Notice that the doubleComparator is not a value but an implicit function because It requires an AcceptenceError

Other example is the comparator of List

implicit def listComparator[A](implicit c: Comparator[A]): Comparator[List[A]] = (left: List[A], right: List[A]) => {
    if (left.lengthCompare(right.size) != 0)
      Diff(Nil, SizeDiff(left.size, right.size)) :: Nil
    else {
      left.zip(right).zipWithIndex.flatMap {
        case ((l, r), index) =>
          c.compareWithPath(index.toString, l, r)
      }
    }
  }

The signature of the implicit function listComparator tells that if we can provide a comparator of type A, then the compiler can create an instance of type Comparator[List[A]] automatically. The implicit system is beneficial here to reduce much boilerplate code

The comparator instances I suggest you take a look in the code source are Option and Map.

Using shapeless power

In the last part, we've defined some useful type classes that can be composed automatically by the compiler in many ways. However, we still have lots of boilerplate code to write. For example what if we want to compare tuple? There are no ways to tell the compiler to construct the comparator of tuple2, tuple3 automatically, without using the macro. Fortunately, this is the purpose of the powerful library called shapeless.

Explaining the library shapeless is out of the article's scope. However, I show how to use shapeless to create derive automatically type class instances

I strongly recommend the free guide to fully understand shapeless.

The ideal behind shapeless is that It can convert any object (case class) into a generic representation. The generic representation is used to compose the child type classes to create the parent type class. The users only need to write the composition function and let the implicit mechanism works with the compiler to create automatically the required type class.

Shapeless provides 2 intermediate representation.

The first is HList that stands for a heterogenous list which represents a case class as a tuple. In this representation, we don't have the information about the field. The name of the field is needed because the comparator should show the position of the difference

The second is LablledGeneric is similar to HList but with the information of the name of the field. For example, the generic representation of case class Person(name: String, age: Int) is

String with KeyTag[Symbol with Tagged["name"], String] ::
Int with KeyTag[Symbol with Tagged["numCherries"], Int] ::
Boolean with KeyTag[Symbol with Tagged["inCone"], Boolean] ::
HNil

The entry point

The entry point of the implicit search:

implicit def genericComparator[A, H](implicit
    generic: LabelledGeneric.Aux[A, H],
    hEncoder: Lazy[Comparator[H]]): Comparator[A] =
    (left: A, right: A) => {
      hEncoder.value.compare(generic.to(left), generic.to(right))
    }

The signature of the function tells that if we can provide a way to convert an object of type A to the intermediate representation of type H and the comparator of type H, then it can create a comparator of type A

To create the comparator of type A. We have to write a function that takes 2 elements of type A and return their difference. Since we have the comparator of the representation of A, we can convert both elements to H and use this comparator instead

Comparator of `HList`

To construct the instance of Comparator of any case class including Tuple. We have to define 2 functions

The first one is the comparator of HNil:

implicit val hNilComparator: Comparator[HNil] = (left: HNil, right: HNil) => Nil

The second one is:

 implicit def hlistFieldComparator[K <: Symbol, H, T <: HList](implicit
    witness: Witness.Aux[K],
    hEncoder: Lazy[Comparator[H]],
    tEncoder: Comparator[T]): Comparator[FieldType[K, H] :: T]) = {
    val fieldName: String = witness.value.name

    }

As always, the signature of the function tells us that if we have the Comparator of the head, the tail and way to access the name of the head, then we can create a comparator of the combination head :: tail.
We can use the name of the field provided by the implicit value witness.
The complete function is:

 implicit def hlistFieldComparator[K <: Symbol, H, T <: HList](implicit
    witness: Witness.Aux[K],
    hEncoder: Lazy[Comparator[H]],
    tEncoder: Comparator[T]): Comparator[FieldType[K, H] :: T] = {
    val fieldName: String = witness.value.name
    (left: FieldType[K, H] :: T, right: FieldType[K, H] :: T) => {
      val headDiffs = hEncoder.value.compare(left.head, right.head)
      val tailDiffs = tEncoder.compare(left.tail, right.tail)
      val headWithPath = headDiffs.map(d => d.copy(paths = fieldName :: d.paths))
      headWithPath ++ tailDiffs
    }
  }

The logic is the same: We have the comparator of the head, so we use it to compute the differences of the head. Since we have the name of the head's field, we need to prepend the field name to the result. We have the comparator of the tail, we use it the compute the difference of the tail. Finally we append the two lists together to form the final result

Comparator of `Product`

In the same fashion, shapeless let us construct any type class of a trait if we can provide all the type class of the concrete case class.

The pattern is the same as HList, shapeless can construct an intermediate representation of the trait called CoProduct. For example

sealed trait People

case class Man(name: String) extends People
case class Woman(name: String) extends People

The intermediate representation of People is Man :+: Woman :+: CNil which is very similar to the standard Either:
Either[Man, Either[Woman, CNil]]

If we admit the power of shapeless that can create automatically this intermediate type for us, then we can define 2 functions (or values)

The first one is CNil. We never have to evaluate a type class of CNil so we can throw exceptions or return a Nil

 implicit val cNilComparator: Comparator[CNil] = (left: CNil, right: CNil) => Nil

The second function is the one that combines the Comparator[Head] and the Comparator[Tail]

implicit def coProductComparator[H, T <: Coproduct](implicit hComparator: Lazy[Comparator[H]], tComparator: Comparator[T]): Comparator[H :+: T] =
    (left: :+:[H, T], right: :+:[H, T]) => {
      (left, right) match {
        case (Inl(h1), Inl(h2)) => hComparator.value.compare(h1, h2)
        case (Inr(h1), Inr(h2)) => tComparator.compare(h1, h2)
        case (x, y)             => Diff(Nil, TypeDiff(x.toString, y.toString)) :: Nil
      }
    }

The function is relatively simple. It says that if 2 elements are the left type, use the comparator of the head. If the 2 elements are of the right type, use the comparator of the tail. Otherwise, this is an error because the 2 elements are not the same type

Examples

This example uses every feature offered by shapeless

sealed trait Tree 
case class Branch(left: Tree, right: Tree) extends Tree 
case class Leaf(value: Option[Double]) extends Tree

In order to use the comparator

Comparator.compare[Tree](left, right)

This example uses all the power offered by shapeless

The LabelledGeneric, the HList to create type class of any case class with a field name
The CoProduct to create type class of any trait
Use Lazy type class helps to work with recursive implicit search

# Conclusion

shapeless is a perfect tool for the job, and the final product helps to boost the performance of developers. They can see precisely what changed in the input and be able to correct bugs quickly

Designing a json migration tool

Hoai Nam NGUYEN — Mon, 22 Oct 2018 14:41:10 +0000

The problems

Json is probably the most popular data format today. It's used both for data exchange (between front-end and backend for example). It's also used to store inside database like MongoDB, even postgresql supports json data type.

Working with json means defining 2 functions:

A Writer that converts an object into json format
A Reader that reads a json and produces an object

An example of this use case is:

case class Model(x: String, y: String)

In play-json, we can define a Format and uses the macro helper to automatically define a Reader and a Writer of Model

implicit val modelFormat: Format[Model] = Json.format[Model]

This is an example value:

{ “x”: “toto”, “y”: “tata” }

Everything works perfectly and now the json data is persisted in database and test folder.

One day the client want to add one more field in the model, the new model should be:

case class Model(x: String, y: String, z: Int)

We don't need to modify the Format because the macro takes care of it automatically.

But what about the old json format? The current format cannot read the old json data because it lacks z

Now we should find a solution in order to be able to read the old json format by setting the z to a default value

First solution

We've found a quick solution to the problem. We can for example define a Reader that can read both current and old version.

The first thing we should do is to split the Format into Reader and Writer:

implicit val modelReads: Reads[Model] = (
  (JsPath \ "x").read[String] and
  (JsPath \ "y").read[String]
)(Model.apply _)

implicit val modelWrites: Writes[Model] = (
  (JsPath \ "x").write[Double] and
  (JsPath \ "y").write[Double]
)(unlift(Model.unapply))

Then we define two versions of the Reader:

val modelReadsV0: Reads[Model] = (
  (JsPath \ "x").read[String] and
  (JsPath \ "y").read[String] and
  Reads.pure(0)
)(Model.apply _)

implicit val modelReadsV1: Reads[Model] = (
  (JsPath \ "x").read[String] and
  (JsPath \ "y").read[String] and
  (JsPath \ "z").read[Int]
)(Model.apply _) orElse modelReadsV0

The system works like that: the implicit readers is always in the last version. If the ReaderV1 can not read the json, its used the ReaderV0 which defines a default value for the field z

But the problems of this design are:

If we add or remove a field, we need to update all the readers
Do not work in more complicated cases: move a field to a nother place, rename a field, change value of a field
Splitting Format to Reader and Writer can cause a Reader/Writer mismatch. We need to add a lot of tests

Second solution

Imperative vs Functional

The solution is to clone the SQL migration design: Each migration is defined by a script.

The question is how we define this migration script. The JsValue is immutable and creating new value from the old one is tedious. I am a fan of functional programming but this is where the imperative way is much more easy. For example

Imperative way	Functional way
`x.y.z += 1`	`x.copy(y = x.y.copy(z = x.y.z + 1))`
`value[“key25”] = {“key251”: value[“key2”][“key21”]}`	`(__ \ 'key25 \ 'key251).json.copyFrom( (__ \ 'key2 \ 'key21).json.pick )`

I write the library for a team of both functional and less functional programer in the team. It should be easy to write migration script and the functional way is very hard for this task. Moreover, for the advanced task it's near impossible to do. For example, let's say we want to modify all the JsObject that has the field toto and change the field value to 0. An example of this json value is:

{
“X”:  { “toto”: 0 },
“Y”:  [{“toto”: 1}, {“tata”: 2} ],
“Z” :  { “zz”: {“toto”: 2}}
}

The value that we want to modify can be inside a field, nested in 2 levels fields or even inside an array. Updating these values in a functional way is hard and the only way I've found is using advanced concept like the zipper

All the difficulties lead to the obvious solution: convert temporary immutable value into mutable version, applying changes and convert back to the original value

Wrapper for mutable version

Let's define a trait for this new data struture:

sealed trait JsValueWrapper
case class JsObjectWrapper(value: collection.mutable.Map[String, JsValueWrapper]) extends JsValueWrapper
case class JsStringWrapper(value: String) extends JsValueWrapper
case class JsArrayWrapper(value: ArrayBuffer[JsValueWrapper]) extends JsValueWrapper
case class JsBooleanWrapper(value: Boolean) extends JsValueWrapper
case class JsNumberWrapper(value: BigDecimal) extends JsValueWrapper
case object JsNUllWrapper extends JsValueWrapper

This build an equivalent of all the possbile values of JsValue. The only difference is the JsObjectWrapper and JsArrayWrapper use mutable collection internally

Conversion functions

Let's define 2 methods to convert between JsValue and JsValueWrapper. Of course there are some recursivities when dealing with JsObject and JsArray

implicit def create(input: JsValue): JsValueWrapper =
   input match {
     case x: JsObject    => JsObjectWrapper(collection.mutable.Map(x.value.mapValues(create).toSeq: _*))
     case x: JsArray     => JsArrayWrapper(ArrayBuffer(x.value: _*).map(create))
     case x: JsString    => JsStringWrapper(x.value)
     case x: JsBoolean   => JsBooleanWrapper(x.value)
     case x: JsNumber    => JsNumberWrapper(x.value)
     case JsNull         => JsNUllWrapper
   }



implicit def toJson(input: JsValueWrapper): JsValue = {
input match {
    case x: JsObjectWrapper    => JsObject(x.value.map { case (name, value) => (name, toJson(value)) }.toSeq)
    case x: JsArrayWrapper     => JsArray(x.value.map(toJson))
    case x: JsStringWrapper    => JsString(x.value)
    case x: JsBooleanWrapper   => JsBoolean(x.value)
    case x: JsNumberWrapper    => JsNumber(x.value)
    case JsNUllWrapper         => JsNull
}
}

Script

Let's define a trait that defines a method migrate that make in place modification

trait JsonMigrator {
 def migrate(input: JsValueWrapper): Unit
 def transform(input: JsValueWrapper): JsValueWrapper = {
   migrate(input)
   input
 }
}

We should define a way to combine multiple scripts to form a global script. This operation is called append and if we can define a neutral script (a script that does nothing), then we can define a Monoid instance

implicit val monoid: Monoid[JsonMigrator] = new Monoid[JsonMigrator] {
   def zero: JsonMigrator = (_: JsValueWrapper) => ()
   def append(f1: JsonMigrator, f2: => JsonMigrator): JsonMigrator = {
     (input: JsValueWrapper) => {
       f1.migrate(input)
       f2.migrate(input)
     }
   }
 }

Some helpers

We should create some conversion implicit to help our users write more concise migration code. But it's not a safe operation because the asInstanceOf can throw exceptions

implicit class JsObjectWrapperConverter(input: JsValueWrapper) {
   def apply(field: String): JsValueWrapper = input.asInstanceOf[JsObjectWrapper].value(field)
   def number: BigDecimal = input.asInstanceOf[JsNumberWrapper].value
   def map: mutable.Map[String, JsValueWrapper] = input.asInstanceOf[JsObjectWrapper].value
   def setDefault(field: String, value: JsValueWrapper): Unit = {
     if (!has(field))
       input.asInstanceOf[JsObjectWrapper].map.update(field, value)
   }
   def remove(field: String): Option[JsValueWrapper] = {
     input.asInstanceOf[JsObjectWrapper].map.remove(field)
   }

Examples

Let's say we have 3 migrations:

private val migrator1 = new JsonMigrator() {
    def migrate(x: JsValueWrapper): Unit = {
      x("field1").map.remove("field11")
      ()
    }
  }
private val migrator2 = new JsonMigrator { // add new field field1/field12
def migrate(input: JsValueWrapper): Unit =
  input("field1").map.update("field12", "myNewField")

private val migrator3 = new JsonMigrator { //  Change all sFields to "hahaha"
def migrate(input: JsValueWrapper): Unit =
  PathResolver.migrate(input, List(RecurFieldCond(HasField("sField")))) { w =>
    w.map.update("sField","hahaha")
  }
}

We can create a global migrator because we've already defined the Monoid for it

import scalaz.syntax.foldable._
import scalaz.std.list._
val globalMigrator = List(migrator1, migrator2, migrator3).suml

Now with the global script, we convert the immutable json value into the mutable version, transform it and convert back to the immutable version

val x = JsValueWrapper.create(json) // first we need to create a mutable version of the original json
allMigrator.migrate(x) // then mutate it by applying the global migration
val result = JsValueWrapper.toJson(x)

Integration into deployment system

This is just an example how we can automatically migrate a database with a lot of json: The database store the current version somewhere. When executing a migration, the proram fetches the current version and find all the migration scripts, combine them to make a global script, iterate all the json value in database, convert to mutable version, transform it with the global script and finally convert it back to the immutable version

The users have to maintain a file that defines the mapping between version and script:

val l = List(
  (0, migration1),
  (1, migration2),
  (2, migration3)
)

If 2 users modify the file at the same time, they have to resolve conflict during the merge

Conclusion

The library is successful because my team uses it every day. There is some very complex script that may take more than 100 lines of code. One of the big avantage is that we can use the Format type class instead of Reader/Writer thus there is no mismatch problem between the reader and the writer

A json migration tool

Hoai Nam NGUYEN — Mon, 22 Oct 2018 14:27:42 +0000

json-migration

This project tries to help building a json migration framework in the same way of a database migration tool.

The json libary used in this example is play-json

The aim of this project is to make the migration script easy to write by people who does not have much experience in
functional programming. Even experienced programmers can stuck with the Coast to coast design

This comes with a cost because it's not type safe: if the user wants to update a field which is a string but it's an object
in realty, then an Exception is throw. Remember, exception can be throwed anywhere inside the migration script.

In a real world project, the users should backup their databases before applying any migrations

For the moment, I did not make it available to import from sbt. If you want to integrate into your project, just copy
the files and add the dependencies.


libraryDependencies += "org.scalaz" %% "scalaz-core" % "7.2.17"
libraryDependencies += "com.typesafe.play" %% "play-json" % "2.6.7"
libraryDependencies += "org.scalatest" %% "scalatest" % "3.0.4" % "test"

How to use

Suppose we want have a json value:


    {"field1" : {
         "field11": 100
        },
        "field2": [
         {
           "sField": "Do it"
         },
         {
           "sField": "good",
           "s1": "fine"
         }
        ],
        "field3": {
         "sField": "nice"
        }
      }

We want to apply a list of transformations to this json value:

Remove the field1 / field11
Add the field field1 / 'field12' with the value myNewField
For every field sField, replace all the values by hahaha. The field sField is in multiple place: inside an array of field2 and inside field3

The result we want to see is:

{
  "field1" : {
   "field12": "myNewField"
  },
  "field2": [
   {
     "sField": "hahaha"
   },
   {
     "sField": "hahaha",
     "s1": "fine"
   }
  ],
  "field3": {
   "sField": "hahaha"
  }
}

Create a list of migrators

The library provides a nice way to describe easily:

Extends the trait JsValueWrapperImplicits to have automatic conversion between Json and the mutable wrapper

Create a new JsonMigrator that defines a function migrate. This function takes a JsValueWrapper which is
a mutable value

first migration

private val migrator1 = new JsonMigrator() {
    def migrate(x: JsValueWrapper): Unit = {
      x("field1").map.remove("field11")
      ()
    }
  }

x("field1") means I know x is a JsObject and I access the field field1
y.map.remove("field11") means I know y is a JsObject and I remove the field field11

second migration

  // add new field field1/field12
private val migrator2 = new JsonMigrator {
def migrate(input: JsValueWrapper): Unit =
  input("field1").map.update("field12", "myNewField")
}

third migration

//  Change all sFields to "hahaha"
private val migrator3 = new JsonMigrator {
def migrate(input: JsValueWrapper): Unit =
  PathResolver.migrate(input, List(RecurFieldCond(HasField("sField")))) { w =>
    w.map.update("sField", JsStringWrapper("hahaha"))
  }
}

This migration is a little more tricky. Inside the function migrate we use a helper PathResolver.migrate that
takes the first input as JsValueWrapper and a list of FieldCond. RecurFieldCond(HasField("sField")) means find all
the Jsvalue that is an JsObject and has the field sField. The second argument is a function that makes the
change in that value. Note that the w object is the found object and not the the value of the field sField

Once we have the list of migrators, we can combine these to make a global json migrator

import scalaz.syntax.foldable._
import scalaz.std.list._
val globalMigrator = List(migrator1, migrator2, migrator3).suml

Note that you need to import scalaz syntax in order to use the suml utilities

Migration

Now when the global migrator is created, the only thing to do is to migrate the original json:

val x = JsValueWrapper.create(json) // first we need to create a mutable version of the original json
allMigrator.migrate(x) // then mutate it by applying the global migration
JsValueWrapper.toJson(x) shouldBe jsonResult // the result must be identical to the desired result

Compare it to JsonTransformer

I will compare the JsonTransformer in the [tutorial([https://www.playframework.com/documentation/2.6.x/ScalaJsonTransformers) with this mutable version

The origin json is

val gizmo = Json.obj(
  "name" -> "gizmo",
  "description" -> Json.obj(
    "features" -> Json.arr( "hairy", "cute", "gentle"),
    "size" -> 10,
    "sex" -> "undefined",
    "life_expectancy" -> "very old",
    "danger" -> Json.obj(
      "wet" -> "multiplies",
      "feed after midnight" -> "becomes gremlin"
    )
  ),
  "loves" -> "all"
)

The transformed json is

val gremlin = Json.obj(
  "name" -> "gremlin",
  "description" -> Json.obj(
    "features" -> Json.arr("skinny", "ugly", "evil"),
    "size" -> 30,
    "sex" -> "undefined",
    "life_expectancy" -> "very old",
    "danger" -> "always"
  ),
  "hates" -> "all"
)

The pure transformer solution:

import play.api.libs.json._
import play.api.libs.json.Reads._
import play.api.libs.functional.syntax._

val gizmo2gremlin = (
  (__ \ 'name).json.put(JsString("gremlin")) and
  (__ \ 'description).json.pickBranch(
    (__ \ 'size).json.update( of[JsNumber].map{ case JsNumber(size) => JsNumber(size * 3) } ) and
    (__ \ 'features).json.put( Json.arr("skinny", "ugly", "evil") ) and
    (__ \ 'danger).json.put(JsString("always"))
    reduce
  ) and
  (__ \ 'hates).json.copyFrom( (__ \ 'loves).json.pick )
) reduce

Wow, it's scary! Can you explain it to a data scientist developer ?

And the solution with this library is:

val migrator = new JsonMigrator {
  def migrate(input: JsValueWrapper): Unit =
    input.map.update("name", "gremlin")
    val description = input.map("description").map
    description.update("features", JsArrayWrapper("skinny", "ugly", "evil"))
    val currentSize = description("size").number
    description.update("size", currentSize * 3) 
    description.update("danger", "always")
    input.map.remove("danger")
    input.map.update("hates", "all")
  }
}

I feel this approach is easier than the purely functional style.

The tutorial from Play's website does not include any recursive transformations. Correct me if I'm wrong, but I think it's impossible to do it
without using a zipper datastructure

Going further

This code is extracted in a real world project. If you want to integrate the migration into your workflow, you have to do more things:

Create a list of transformation:

val l = List(
  (0, migration1),
  (1, migration2),
  (2, migration3)
)

If your json lives inside a database, then you have to store the version somewhere. To build a global transformer you have to know the start version and create
the global transformation from the start version to the lastest version.

That's it

If you've found it useful, please let me know

Forem: Hoai Nam NGUYEN

Integration tests and microservices applications

Test is the norm

Sad reality

Legacy code

Integration tests

Combination of integration tests and regression tests

Conclusion

Create a Validation DSL from scratch with scala

Introduction

Simple implementation

Introduction to cats

Conclusion

Designing a comparator library for regression test

Regression test

Regression test workflow

Example 1:

Example 2:

Designing the comparator

The API

Comparator of some useful types

Using shapeless power

The entry point

Comparator of HList

Comparator of Product

Examples

Designing a json migration tool

The problems

First solution

Second solution

Imperative vs Functional

Wrapper for mutable version

Conversion functions

Script

Some helpers

Examples

Integration into deployment system

Conclusion

A json migration tool

json-migration

How to use

Create a list of migrators

first migration

second migration

third migration

Migration

Compare it to JsonTransformer

Going further

Comparator of `HList`

Comparator of `Product`