Forem: Pedro F Marquez

Running Go code inside a NodeJS app with WASM (Part 1/2, 2023)

Pedro F Marquez — Mon, 13 Feb 2023 18:01:00 +0000

NodeJS is an excellent tool to quickly set up small services like HTTP servers thanks to its ease of integration with other libraries like SSR for web frameworks and NoSQL databases like MongoDB.

However, due to its interpreted nature, JavaScript is not the right tool for memory-intensive tasks. Other compiled languages like Golang get the upper hand for some tasks, as they are optimized for them. It would be awesome if we could get the best of both worlds, isn't it? Enter WebAssembly.

Brief disclaimer

The example explained in this post -as most WebAssmebly-specific libraries are at this point in time- is based on some experiment libraries, which are subject to changes in the future. This doesn't mean you cannot use them in production, but you should be aware that the APIs of these libraries are not guaranteed to be backward compatible in future releases and extra refactoring work may be needed when updating versions of Go or NodeJS.

Some background

JavaScript is an interpreted language. This means that, due to its dynamic typing nature, the application cannot make assumptions about what type of data will be used at run time: A variable passed to a function could be a string, an object or even a function.

While dynamic typing provides a lot of flexibility to developers, the interpreter executing the code cannot optimize the code ahead of time. This is why JavaScript engines like those used in browsers perform Just-in-time (JIT) optimization.

JIT essentially is a caching strategy over the code and the specific types used in it. As code is interpreted and executed, the JIT compiler will keep track of the types used in the code, convert it to bytecode (low-level instructions) and use this optimized version of the code the next time this code is called.

While JIT performs many other specific improvements, in general, this is its goal: Compile the JavaScript code into optimized instructions. Thanks to JIT compilation, engines like browsers can execute interpreted code at almost native code speeds (the keyword being almost). And here lies the advantage of compiled languages like Go: Their strongly typed nature (among other language-specific characteristics) allows compilers to perform all optimizations ahead of time, resulting in applications that only need to execute bytecode.

WebAssembly

WebAssembly (WASM) is a specification that allows us to write code that can be fully compiled and optimized ahead of time, and use the compiled code in environments where typically interpreted languages are used, like browsers or in server-side interpreted engines.

WASM features

Fast. WASM-compiled code is almost as fast as natively compiled applications. This allows us to run memory-intensive applications that typically would be impossible to run in low-resources clients like browsers or IoT devices.
Secure. WASM modules only have access to a linear set of memory, sandboxed from the rest of the application. This reduces the risk of having malicious code accessing data or resources they shouldn't.
Portable. WASM works as a container (or VM), which allows it to compile bytecode that can be run in multiple architectures: Browsers, mobile devices, and backend servers; as long as it supports the WASM specification, it can execute WASM code.
Flexible. WASM itself is a compilation target. This means that, potentially, you can use any programming language to write code, and produce bytecode that can run almost everywhere.

WASM limitations

The same isolation and flexibility that provides most of WASM's strengths are the cause of most of its limitations:

WASM doesn't provide a default memory management (e.g. Garbage Collection) mechanism. This means that, currently, each WASM module needs to be shipped with its memory management code. (This is one of the reasons why Rust shines for WASM modules, as its memory management it's baked into the language)
Communication between the WASM module and the rest of the application needs to be done in very simple types (bytes, ints and floats). No complex types are supported yet. This is why most WASM compilers also provide some glue-code to map between complex types like strings or arrays. The Web Assembly System Interface (WAS) is an on-progress standard aimed to solve this last limitation; once it's mature it will allow easy interoperation with almost every environment. WASI is already available in some WSAM compilers and runtimes.

The use case: Log tailing

Most WASM examples are simple "Hello World". For this post, I thought it was useful to see a more specific use case, a problem that I've found in the past.

Imagine you have a deployed application that is logging output into a text file. We want to parse each line in this file in real-time, as it is written to it, and send it somewhere to be parsed, displayed and/or stored. Existing tools like Logstash already provide this functionality, but we want to write something ourselves that give us control over the parsing process.

First, let's create a simple NodeJS script named logger.js to simulate the creation of this log file:

const fs = require('fs')
const util = require('util')

var logFile = fs.createWriteStream('test.log', { flags: 'a' })

function log() {
  // console.log(...arguments);
  logFile.write(util.format.apply(null, arguments) + '\n')
}

function randomElement(arr) {
  return arr[Math.floor(Math.random() * arr.length)]
}

function randomTime() {
  const time = [1000, 100, 3000, 0]
  return randomElement(time)
}

function randomLevel() {
  const levels = ['WARN', 'INFO', 'ERROR']
  return randomElement(levels)
}

let i = 0
function startLogger() {
  log(`${randomLevel()} Log number ${i++}`)

  setTimeout(() => {
    startLogger()
  }, randomTime())
}

startLogger()

First, we create a NodeJS ReadStream using fs.createWriteStream to open the test.log file in "append" mode. Then we start recursively looping using setTimeout. This simulates an application that logs text in the file at a variable rate. At each iteration:

We print some simple text with a random log level from {'WARN', 'INFO', 'ERROR'} to the log file.
We randomly pick the number of milliseconds from {1000, 100, 3000, 0} to wait for the next iteration.

Reading the log file in Go

We will write our module in Go. Go offers native WASM compilation without the need of installing other libraries or compilers.

Enabling the output of complex types

As I mentioned before, WASM modules cannot output complex types, so if we want to be able to send out something other than numbers or bytes, we need some "glue" code that takes care of mapping types like strings and slices into bytes. The Go repository offers the very convenient wasm_exec.js, which will allow our WASM module to communicate with the outside world.

The wasm_exec.js script as it is has some limitations:

It was created to be used on browsers. We can work around this fact by making some small changes to the script to make it compatible with NodeJS. We will define these changes in the next post.
It doesn't support custom data structures. This makes sense, as our WASM module would have to export some sort of schema so whoever consumes the module knows how to parse the output. However, wasm_exect.js does support slices and maps, which allows us to marshal our Go-defined structures into objects like map[string]interface{}.

Our current example differs from other simpler examples in the fact that we will not just output a single result from the WASM module. We want to be able to send parsed logs back to NodeJS as they are appended to the log file.

We could use polling to repeatedly call the WASM module to retrieve the latest logs, but this is not an optimal approach, as it requires fine-tuning to get the logs in real-time without excessive memory consumption.

A better approach is to use a callback: We define a function in NodeJS that can be called from the WASM module every time new logs are parsed and ready to be consumed.

The Go code

First, we create a new Go module:

go mod init wasm-test.com

In the same folder where the go.mod file was created, we create wasm.go.

First, we will define the types of our output. For convenience, we will create a new type called ParsedLogs, which is just an alias for map[string]interface{}.

type ParsedLogs = map[string]interface{}

Now, let's define a function that has only one responsibility: Sending the output back to NodeJS. However, since we want to test that everything works without having to compile and deploy the module, for now, our callback will just print the parsed logs into the console:

func ModuleOutput(parsedLogs ParsedLogs) {
    fmt.Println(parsedLogs)
}
type OutputCallback = func(parsedLogs ParsedLogs)

The OutputCallback is another alias we create just for convenience, so we don't have to use the verbose func(parsedLogs ParsedLogs) when we pass our function as a parameter to the main process.

Now, we define two functions: main, which is the entry point to the module, and Execute, which will contain the actual log parsing logic:

func Execute(callbackFn OutputCallback) {
    //... call callbackFn to send parsed logs
}

func main() {
    Execute(ModuleOutput)
}

Reading the log file

It's time to extend the Execute function to read our log file. For this example, we will hard-code the path to the log file, but you could pass this as a parameter, or retrieve it from a global function defined in NodeJS.

func Execute(callbackFn OutputCallback) {
    file, err := os.Open("./test.log")
    if err != nil {
        log.Fatal(err)
    }
    defer file.Close()

    acc := NewAccumulator()
    defer close(acc.out)

    go func() {
        for {
            select {
            case str := <-acc.out:
                l := parse(str)
                callbackFn(l.ToMap())
            case <-time.After(2 * time.Second):
                acc.Flush()
            }
        }
    }()

    reader := bufio.NewReader(file)

    for {
        line, err := reader.ReadString('\n')
        if err != nil {
            if err == io.EOF {
                time.Sleep(500 * time.Millisecond)
                continue
            }
            log.Fatal(err)
        }
        acc.Append(line)
    }
}

There are a lot of things going on in here, so let's break them down.

First, we open our log file, making sure we defer its close function:

file, err := os.Open("./test.log")
if err != nil {
    log.Fatal(err)
}
defer file.Close()

Then, we create an instance of Accumulator -a structure we haven't defined yet, but will take care of aggregating logs as they are printed-:

acc := NewAccumulator()
defer close(acc.out)

Now, the rest of the code does two things in parallel:

In a loop, it reads the log file, one line at a time, and sends it to the Accumulator.
In another loop, it checks the contents of Accumulator, and if there are processed logs ready to be sent out, it calls the output callback passing with the parsed logs as a parameter.

Counterintuitively, the first loop can be seen at the end of the function:

reader := bufio.NewReader(file)

for {
    line, err := reader.ReadString('\n')
    if err != nil {
        if err == io.EOF {
            time.Sleep(500 * time.Millisecond)
            continue
        }
        log.Fatal(err)
    }
    acc.Append(line)
}

We create a file reader, and inside the infinite loop, we read the file one line at a time. If the file reader reaches the end of the file (which happens when ReadString returns io.EOF as an error), we just pause the process for half a second before checking again if the file has new log entries. This call to time.Sleep is critical, otherwise, the loop would be executed at full-speed, consuming more memory than needed.

When the reader outputs a new line, we just add it to the Accumulator.

We execute the second loop inside a goroutine, which will allow us to have another infinite loop running in parallel.

go func() {
    for {
        select {
        case str := <-acc.out:
            l := parse(str)
            callbackFn(l.ToMap())
        case <-time.After(2 * time.Second):
            acc.Flush()
        }
    }
}()

Inside this loop, we check the Accumulator's attribute out, which is a go channel. If the channel has a new log entry, we process it using the parse function -which we will define next-, and we will send the processed result to the output callback.

We will explain what the second case inside the select block does in the following section.

The accumulator

One thing to notice is that some log entries don't fit in a single line. If we tried to parse each line in the text file in isolation, we wouldn't be able to parse some of them as they are a continuation of the previous line. To solve this challenge, we will accumulate each line into an instance of strings.Builder.

We see that independent log entries contain a log level like WARN or ERROR. This helps us to identify if a line is the beginning of a new log entry or part of the previous line.

The Accumulator struct is basically a wrapper around strings.Builder:


var r, _ = regexp.Compile("(WARN|ERROR|INFO) (.+)")

type Accumulator struct {
    sb  strings.Builder
    out chan string
}

func NewAccumulator() Accumulator {
    return Accumulator{sb: strings.Builder{}, out: make(chan string, 10)}
}

func (a *Accumulator) Append(str string) {
    if r.MatchString(str) {
        a.Flush()
    }
    a.sb.WriteString(str)
}

func (a *Accumulator) Flush() {
    if a.sb.Len() > 0 {
        res := a.sb.String()
        go func(res string) {
            a.out <- res
        }(res)
        a.sb = strings.Builder{}
    }
}

We use the Append function to add a new line to the Accumulator. Inside this function, we check if the log contains the words ERROR, WARN or INFO; if they don't match, we append them to the string builder. However, if they match, we call the Flush function.

The Flush function checks if the string builder is empty. If it's not, it will convert it to a string, and send it to the out go channel. It then clears the string builder by creating a new instance, making it ready to start receiving new lines from the log file.

Now, remember we were also calling Accumulator.Flush in the infinite loop that reads the output of the out channel:

select {
case str := <-acc.out:
    l := parse(str)
    callbackFn(l.ToMap())
case <-time.After(2 * time.Second):
    acc.Flush()
}

This is because the accumulator itself only calls Flush when it receives a new line of text. If we only called Flush inside Append, the Accumulator would wait to output the latest log entry until the log file has new content, which would result in always missing the last entry from the output.

For instance, if our log file only contained the following line:

WARN - Hello

And then a second line is added:

WARN - Hello
ERROR - Something awful happened!

The first line will be sent to NodeJS when the second one is appended. But the second line itself would remain inside the Accumulator. If the process stops sending logs completely, the NodeJS code would never see the second line.

However, since our code is calling Flush every 2 seconds after not receiving new processed logs, the second entry will be processed and sent back to NodeJS.

Parsing the logs

The code used to parse the logs is pretty straightforward. It addresses the simple formatting of the log entries in our example, but you can easily extend it to parse more complex patterns:

We create a simple Log structure, which has a function ToMap, which returns an instance of ParsedLogs -the alias we previously defined for map[string]interface{}-.

type Log struct {
    Level string
    Msg   string
}

func (l Log) ToMap() ParsedLogs {
    m := make(map[string]interface{})
    m["level"] = l.Level
    m["msg"] = l.Msg
    return m
}

Then, we finally define the parse function which we called in our goroutine. This function uses Regex to parse the log level from the entry and the text that follows it. It creates an instance of Log and fills its attributes with the values parsed from the Regex.

func parse(str string) Log {
    log := Log{}
    if !r.MatchString(str) {
        log.Msg = str
        return log
    }
    groups := r.FindStringSubmatch(str)

    log.Level = groups[1]
    log.Msg = groups[2]
    return log
}

Technically we didn't need to define a new structure, as we can easily output a map[string]interface{} directly. However, the extra abstraction gives us a placeholder if we needed to perform extra parsing logic.

Now, we can test our code works by running logger.js and wasm.go in parallel:

Now it's time to update our Go code to be compiled as a WASM module.

The syscall/js module

We will import the go module "syscall/js", which is defined in its docs as follows:

Package js gives access to the WebAssembly host environment when using the js/wasm architecture. Its API is based on JavaScript semantics.
This package is EXPERIMENTAL. Its current scope is only to allow tests to run, but not yet to provide a comprehensive API for users. It is exempt from the Go compatibility promise.

In essence, this library allows us to communicate with the NodeJS code that will execute our WASM module.

From syscall/js we can get references to global variables inside NodeJS. Then, we can define a function in NodeJS like this:

globalThis.logCallback = () => {
  /* ... */
}

And then, inside our Go code, we can get a reference to that function as follows:

func ModuleOutput(parsedLogs ParsedLogs) {
    logCallback := js.Global().Get("logCallback")
    logCallback.Invoke(parsedLogs)
}

One thing you will notice is that syscall/js is only available for the WASM architecture. If we now try to run our Go module with go run, we will see an error message like the following:

package command-line-arguments
        imports syscall/js: build constraints exclude all Go files in /usr/local/go/src/syscall/js

As soon as we import syscall/js, the compiler constraints the type of architectures this module can be built fo.

This is the reason why we didn't call js.Global() directly inside Execute; it decouples most of the code from WASM-specific dependencies, which makes them easier to reuse in non-WASM applications.

If we want to execute our code now, we will have to compile it into WebAssmebly bytecode; this can be done with the following command:

GOOS=js GOARCH=wasm go build -o app.wasm

By setting the GOARCH environment variable to wasm, the module now will compile successfully. In addition, the GOOS=js environment variable lets the compiler know we will use this WASM module in JavaScript.

If the go build command completes successfully, you should see a new file named app.wasm in the root folder. This is the WASM module we will import in NodeJS.

In the next blog post we will explore the second part of this process: Importing the compiled WASM module inside a NodeJS app.

Conclusion

With the current capabilities of NodeJS and Golang, it is now possible to use WebAssembly as a way to create code modules that perform almost as well as native code does, and enable functionality that historically has been prohibitive in terms of memory and execution time in interpreted languages like JavaScript.

We have seen that Golang offers compilation support for WASM out of the box. In this example we used native Go features like Go channels and goroutines, making the point that those can operate well in the context of WebAssembly.

One important thing to notice before we end is that executing WASM in NodeJS -instead of a browser like more common examples do- gives us access to host-specific libraries like fs to read files and to a memory runtime. All these things are not directly available in the browser's JavaScript engines, which means the WASM module we created in this example would not run there.

However, there are other, more fleshed-out, libraries like wasmer-go that provides a runtime and help us navigate around these limitations. The wasmer-go documentation provides a good summary of these challenges:

The major problem is that, whilst the Go compiler supports WebAssembly, it does not support WASI (WebAssembly System Interface). It generates an ABI that is deeply tied to JavaScript, and one needs to use the wasm_exec.js file provided by the Go toolchain, which doesn't work outside a JavaScript host.

Then, it is important to point out that our example only works in NodeJS. The compiled WASM module cannot be used in other WASM-enabled environments. Regardless, we can take the same code and use tools like wasmer-go or tinygo and this limitation can be worked around.

WebAssembly has a promising future as a standard to modularize interoperable code. We can expect that future improvements around the specification (especially with WASI) will open the door to using WASM for creating embeddable containers of functionality that can be imported in almost any environment from the browser and backend servers to edge-computing devices like those in IoT.

Software Design Patterns: The builder pattern

Pedro F Marquez — Wed, 16 Nov 2022 17:06:25 +0000

Design patterns in software development exist to make our lives easier. Patterns are not "how things are supposed to be", but well-defined recipes that solve common challenges while writing software. Today, we will explore one common design pattern: The builder pattern.

Challenge: Too many parameters or attributes

When we build production software, it is common to reach a point where our code is becoming really verbose. For instance, imagine we want to create a Java class that contains a lot of information in it:

class Product {
  private String id;
  private String name;
  private String shortDescription;
  private String longDescription;
  private float price;
  private List<String> specifications;
  // ...
}

Now, many if not all of these attributes are required, so we find ourselves writing a huge constructor function for the class:

public Product(
    String id,
    String name,
    String shortDescription,
    String longDescription,
    float price,
    List<String> specifications
    ) {
      // set the attributes
}

Now, imagine we have three types of products:

Products that only have a short description
Products that only have a long description
Products that have both

Let us assume that our app requires us to handle all three cases separately. This will lead us to call this constructor in three different ways:

Product product = new Product("p1", "Some product", "Short description", null, price, specs);
//...
Product product = new Product("p1", "Some product", null, "Long description", price, specs);
//...
Product product = new Product("p1", "Some product", "Short description",
                                "Long description", price, specs);
//...

Notice how we have to fill those unnecessary values with null to make up for the missing values.

Of course, we can create a new constructor for each case:

class Product {
  // both short and long descriptions
  public Product(
    String id,
    String name,
    String shortDescription,
    String longDescription,
    float price,
    List<String> specifications
    ) {
      // set the attributes
  }

  // only long description
  public Product(
    String id,
    String name,
    String longDescription,
    float price,
    List<String> specifications
    ) {
      // set the attributes
  }

  // only a short description
  public Product(
    String id,
    String name,
    String shortDescription,
    float price,
    List<String> specifications
    ) {
      // set the attributes
  }

}

That's a lot of new lines! And, the more variations we get, the worst it gets. Every time we find a new variation or combination of required and missing fields, we would have to write a new constructor.

All this extra code is boilerplate code: It has no value for the business case and no custom logic.

Note: Languages like Python work around this issue by allowing us to make parameters optional using named parameters. Something like Product(id, name, shortDescription="something", price) using only one constructor is perfectly legal! Java also has recently introduced the concept of records, which considerably reduces the amount of boilerplate code needed for POJOs

Wouldn't it be nice if we could just set the parameters that we actually need?

Why not just use setters?

Of course, we could add setter methods for each attribute and only make required fields shared by all use cases (e.g. id and name) part of the constructor. This would allow us to just set the fields we need.

The problem here is that there is no guarantee that nothing is preventing us from creating objects missing required fields. If the expectation is that at least one of shortDescription or longDescription is present, there is no way to validate that one of those fields is always present in this class alone. We would depend on whichever class using Product to validate that all required fields are present.

So, it would be nice if we could set only the attributes we need and validate that all required fields are present. Enter the builder pattern.

ProductBuilder

The builder pattern is an Object-Oriented pattern that separates the construction of an object from its representation. It is implemented by creating a Builder class (notice builder is not a special class or interface we must extend) that has two main properties:

It has methods to take each attribute independently.
- Optionally, each method returns this, to allow for chaining setter methods.
It has a single method to build an instance of the class we want to create.

Let us look at an example of a builder for the Product class:

class ProductBuilder {
  String id,
  String name,
  String shortDescription,
  String longDescription,
  float price,
  List<String> specifications

  // required fields in the constructor
  public ProductBuilder(String id, String name) {
    this.id = id;
    this.name = name;
  }

  public ProductBuilder withShortDescription(String shortDescription) {
    this.shortDescription = shortDescription;
    return this;
  }

  public ProductBuilder withLongDescription(String longDescription) {
    this.longDescription = longDescription;
    return this;
  }

  public ProductBuilder withPrice(float price) {
    this.price = price;
    return this;
  }

  public ProductBuilder withSpecifications(List<String> specifications) {
    this.specifications = specifications;
    return this;
  }

  public void validateAllRequiredFieldsArePresent() {
    // check if all fields are present.
    // e.g. one of shortDescription or longDescription should be present
    // if the validation fails, throw an exception.
  }

  public Product build() {
    this.validateAllRequiredFieldsArePresent();
    return new Product(
      this.id,
      this.name,
      this.shortDescription,
      this.longDescription,
      this.price,
      this.specifications
    )
  }
}

It's easy to miss how creating a whole new class is better than having multiple constructors, at least in terms of the number of lines. It will be clear soon what the advantages are.

Rhe following code snippet is an example of how to create a Product instance for each of our three use cases:

Product p1 = new ProductBuilder("p1", "Some product")
  .withLongDescription("Long description")
  .withPrice(12.99)
  .withSpecifications(specs)
  .build()

Product p1 = new ProductBuilder("p1", "Some product")
  .withShortDescription("Short description")
  .withPrice(12.99)
  .withSpecifications(specs)
  .build()

Product p1 = new ProductBuilder("p1", "Some product")
  .withShortDescription("Short description")
  .withLongDescription("Long description")
  .withPrice(12.99)
  .withSpecifications(specs)
  .build()

Now, the builder pattern has given us a lot of benefits:

We only set the attributes we need.
We can set the attributes in any order. We can use withPrice before withShortDescription, and things still work.
We don't need to write new code when we define new combinations of attributes. We just set the needed attributes in the builder.

The key to this magic lies in the way we create our setter methods:

public ProductBuilder withLongDescription(String longDescription) {
  this.longDescription = longDescription;
  return this;
}

Notice how the method signature returns an instance of ProductBuilder; the method itself returns this. This simple pattern allows us to chain multiple setters like we saw in the example:

// chained:
new ProductBuilder("p1", "Some product")
  .withShortDescription("Short description")
  .withPrice(12.99);

// line by line:
ProductBuilder builder = new ProductBuilder("p1", "Some product");
builder = builder.withShortDescription("Short description");
builder = builder.withPrice(12.99);

Chaining setter methods give our builder a higher level of expressiveness: It is easy to read and understand what attributes we're setting, and we do so in a very compact way.

The builder pattern also allows us to add expressiveness to more complex configurations of data. For instance, we can create setter methods for specifications that better capture cases like when a product has no specifications:

public ProductBuilder withEmptySpecifications() {
  this.specifications = new ArrayList<String>();
  return this;
}

new ProductBuilder("p1", "Some product")
  .withEmptySpecifications()
  .build();

Using withEmptySpecifications() is more developer-friendly than having to do:

withSpecifications(new ArrayList<String>())

The added benefit of the builder pattern over adding required fields through setters likes in the build method. Here, we can do all sorts of validations and transformations on the added attributes, right before returning a built instance of Product.

The method validateAllRequiredFieldsArePresent can check before creating an instance of Product that the product has at least one of the description fields, or throw an exception if not.

The builder pattern helps us guarantee that each instance of Product will be valid, and still get the benefits of simple getters.

More benefits: Stubbing

When we write unit tests, we build stub objects that emulate the behavior of real services. The builder pattern is extremely useful here, as it allows us to create builders for configuring a stub object.

Imagine we have a service that saves instances of a class Record into a database. The real service implements an interface called DatabaseService. For our unit tests, we create a class called StubPrintService which implements the same interface.

In our example, StubPrintService.record() will always return success.

interface DatabaseService {
  CompletableFuture<boolean> saveRecord(Record record);
}

class StubPrintService implements DatabaseService{
  @override
  CompletableFuture<boolean> saveRecord(Record record) {
    return CompletableFuture.supplyAsync(
      () -> {
        return true;
      }
    );
  }
}

Now, we want to test the case where the saveRecord service returns false. We could pass the expected value through a constructor to the stub service:

class StubPrintService implements DatabaseService{
  private boolean expectedResult;
  public StubPrintService(boolean expectedResult) {
    this.expectedResult = expectedResult;
  }

  @override
  CompletableFuture<boolean> saveRecord(Record record) {
    return CompletableFuture.supplyAsync(
      () -> {
        return this.expectedResult;
      }
    );
  }
}

But, just as in our previous example, as we need to add more variants to our test, adding more constructors becomes verbose. This can easily be fixed with creating a builder for defining test cases:

class StubPrintServiceBuilder() {
  boolean expectedResult;

  public StubPrintServiceBuilder withSuccess() {
    this.expectedResult = true;
    return this;
  }

  public StubPrintServiceBuilder withFailure() {
    this.expectedResult = false;
    return this;
  }

  public void PrintService build() {
    return new StubPrintService(this.expectedResult);
  }
}

Now, each test case we write will be easy to understand:

PrintService stubPrintService = new StubPrintServiceBuilder()
                                .withSuccess()
                                .build();

PrintService stubPrintService = new StubPrintServiceBuilder()
                                .withFailure()
                                .build();

We can look at each test and know what exactly we're testing. Additionally, if the PrintService depends on other classes internally, StubPrintServiceBuilder can take care of creating the right instances for the test case.

In a way, the builder defines a configuration: The developer writes what the object should be or how it should behave, and the builder takes care of how to put things together to achieve the expected behavior.

Conclusion

When we deal with classes with tons of attributes, the builder pattern reduces the amount of boilerplate code we must write to create objects that have different combinations of those attributes.

The builder separates the logic necessary to create an object from the object itself, allowing the original class to be compact and only contain code that provides "business" value for the application.

The builder pattern gives us flexibility on what attributes and in which order we must set to create an object, without losing any assurance that the created objects are valid and contain all required fields.

The only trade-off we could find is that creating builders for simple POJO classes may be overkill. As with any useful tool, use it when it is actually needed. Don't use an industrial hammer to hang a picture in your wall.

Building a performant heap (priority queue): A deep dive

Pedro F Marquez — Fri, 04 Nov 2022 12:03:00 +0000

Priority queues are one of the data structures that have intrigued (and confused) me the most. Its implementation is different from many tree-like data structures, and its inner processes seem to be different from other structures like balanced binary trees. This is why I've put some time into building one myself and getting familiar with the inner workings of this popular data structure.

According to the great Wikipedia, a priority queue (or heap) is defined as follows:

"A priority queue is an abstract data type similar to a regular queue or stack data structure in which each element additionally has a priority associated with it. In a priority queue, an element with high priority is served before an element with low priority."

Basically, in a priority queue, you insert elements and when you pull out those elements, the ones with the highest priority will exit first.

Priority queues are also called heaps, and they come in two flavors: min heaps and max heaps. As their name indicates, the priority in which elements are extracted is their value: For min heaps, the smallest elements come out first, whereas, for max heaps, it is the largest elements the ones that come out first.

So, when we talk about "higher priority", for min heaps it means the elements are smaller, and for the max heap, it means that the elements a larger.

For instance, if you insert the following numbers -in that order-: 3,5,6,2,9,1 into a min heap, if you pull them out one by one, you will get them in ascendent order: 1,2,3,5,6,9. The order would be reversed if this was a max heap.

Priority queues have multiple applications: finding the largest/smallest N elements in a list in almost linear time, keeping track of processes with high priority in a multi-process application, or even sorting a list of elements in O(nlogn) time.

While the generalized advice is to always use data structures that are well-tested and already included in most coding languages, learning the inner workings of heaps helps us decide whether this data structure is the correct one for our use case. In short, there is rarely a case where you want to implement your own data structures when you are already getting them for free.

Having said that, before jumping into the implementation details, to understand how priority queues work we need to underline some specific facts about them.

Properties

Internally, priority queues have two key properties:

Each element in the heap has two or fewer children, and the children must be of a smaller or equal priority than their parent.
There is only one single root element: The one with the highest priority.

As long as our heap maintains those properties, we're free to use the implementation details we wish (though, as we will see, the most common implementation tends to be optimal).

Another property can be seen, though there are implementations of heaps that don't necessarily follow this property:

When represented as a binary tree, the heap is a complete tree.

Priority queues are commonly stored in arrays, but they are structured like trees

If you look at any implementation of a heap, you will find that it is structured like a binary tree: It has "nodes" and each node has two "children" that have strictly less or the same priority as the parent node.

In practice, the heap is stored in a list or array. Each element in the list is a node, and to find each node's children, we employ a clever trick: The left child is given by solving the formula l = i*2 + 1, where i is the index for the parent node, and the right child is given by r = i*2 + 1.

The following two images show how elements 0 and 1 find their children in the list. Element 0 of the array finds their children in indices 1 and 2

And element 1 of the array finds their children in indices 3 and 5

When calculating the index for the children for a given node, it's critical to remember that the children may not exist in the list (think of the case where the heap has only one element), and we must check the list's length before retrieving the nodes to avoid getting an index-out-of-bounds error.

Why not just use a tree instead? Trees have the added memory overhead of having to keep pointers or references from one node to its children. And, since the "tree" will always be complete, an array or list will be the most efficient way to store the elements in the priority queue.

Priority queues don't keep a sorted list of elements

Since we can sort a list by adding all their elements into a heap and pulling them out one by one, it is easy to think that those elements are internally ordered. This is not true.

All that heaps care about is keeping track of the largest or smallest element at any time. When we add or remove elements from the heap, it will internally reorganize its contents to make sure that the new largest/smallest element is now at the top of the heap.

However, heaps do keep a specific order in their elements. When seen as a tree, for each node in the heap, all its children will be of a lower or equal priority than the node's value. This ordering brings a good property: The first -smallest/largest- element will always be the first element in the heap's internal array.

Finding the top element in the heap is O(1) fast, but adding or removing elements is less fast -O(logn)-

Since we know the heap will always keep the largest element in the first element of its internal array, finding the element with the highest priority is as easy as reading the first element.

But this guarantee -the largest/smallest element being at the top- comes from the fact that the queue keeps moving its elements around when we add or remove new elements.

For instance, if a min heap had only two elements, [2,3] finding the smallest is fast. In constant time -or O(1)- we just poll the element at index 0. However, if we insert another number, there is no guarantee that the first element will be the smallest anymore: If we insert a 1, the heap will re-order its contents to something like [1,3,2]. As we can see, the heap keeps its properties (the first element is the smallest, and its children are larger than it), but this re-ordering comes at a price.

A performant heap will re-order its elements on insertions and deletions in O(logn) time (more details on why below). This is why sorting elements using a heap (also called "heap-sort") takes O(nlogn) time: We insert n elements, and each insertion takes O(logn) time to re-order the heap to keep its properties.

In practice, heaps can store any comparable value

While most examples of heaps deal with numbers, in real-life applications heaps can hold references to complex data structures like objects. The only requirement is that these data structures are comparable: We can take two instances of these data structures and quickly (in constant time, preferably, or else the heap loses all its performance properties) determine which one has a higher priority. For numbers it's easy (compare their values), but other, more complex structures, need to provide a way to compare two elements: Comparing one of its attributes, like a "priority" or "order" value, or any similar method.

Now that we understand the key facts of priority queues, let's look at their inner workings.

The API

For the sake of simplicity, we will only allow numbers in our heap. If we wanted to allow any kind of object, we would have to modify the parts of the code where we compare elements to use a comparator function, provided by the consumer of the heap class. We also assume that the heap cannot store null values, as they cannot be compared (we will use null as a special value to move elements around during each operation; more details to follow).

We will define a JavaScript class with the expected interface for a heap:

class Heap {
  /**
   * Add an element to the heap
   * @param {number} el - the element to add
   */
  push(el) {}

  /**
   * Read the element with the highest priority
   * without removing it from the heap.
   *
   * @returns the value of the element with the
   * highest priority
   */
  peek() {}

  /**
   * Read the element with the highest priority
   * and remove it from the heap.
   *
   * @returns the element with the highest priority
   */
  poll() {}
}

As the comments at the top of each element, the heap will have three basic operations:

Push. Add elements to the heap
Peek. Reading the top element, without removing it from the heap
Poll. Reading the top element, removing it from the heap

Peek

We begin with the simple case of all: peek.

We define an array for the contents of the heap, and when we peek at an element, we will return the first element of such array:

class Heap {
    content = []

    peek() {
        if (this.content.length === 0) {
         return null
      }
      return this.content[0]
    }

If the content is empty, we return either a null value. We may throw an exception too, but I think it's more user-friendly to not force consumers to handle exceptions for an operation that does not break anything.

Time complexity

This operation is the fastest of all. It takes constant time -O(1)-, since we don't need to search for the top-priority element in the heap. It's always at the top.

Push

For the push operation we will do a couple of things:

We add the element at the end of the content array.
Iteratively, we swap the new element with its parent until we find its correct place in the heap.

/**
* Add an element to the heap
* @param {number} el - the element to add
*/
push(el) {
    this.content.push(el)
    this.size += 1
    this._reorderElements(this.content.length - 1)
}

The implementation of this._reorderElements requires us to find the parent of a given node. To do so, we will use the inverse of the formula we previously defined for finding a node's children:

If the index is an even number (e.g. 2), we find the parent at elementIndex / 2.
If the index is an odd number (e.g. 3), we find the parent element at (elementIndex - 1) / 2.

You can run examples on these formulas and you will find that they indeed return their correct parent.

The following function will allow us to find the parent for a given node:

/**
* Find the parent element of a given node
*
* @param {number} elementIndex the index of the child node
* @returns the index for the parent
*/
getParent: function (elementIndex) {
    if (elementIndex == 0) {
        return null
    }
    if (elementIndex % 2 > 0) {
        return Math.floor(elementIndex / 2)
    }
    return Math.floor((elementIndex - 1) / 2)
},

Notice that, if the element index is already 0, we return null, as the first element has no parent.

Going back to the push method: We use JavaScript's Array method push to add to the end of the array, then we call the sub-method this._reorderElements on the last element index (this.content.length - 1).

/**
* Reorder the heap contents to make sure the element at
* elementIndex is in the right place.
*
* @param {number} elementIndex
*/
_reorderElements(elementIndex) {
    let elementVal = this.content[elementIndex]
    let parentIndex = HeapUtils.getParent(elementIndex)

    while (parentIndex !== null) {
        if (this._shouldSwap(this.content[parentIndex], elementVal)) {
        HeapUtils.swap(this.content, parentIndex, elementIndex)

        elementIndex = parentIndex
        parentIndex = HeapUtils.getParent(elementIndex)
        } else {
            break
        }
    }
}

/**
* Compare elements and return if we should swap elements
*
* @param {number} parentValue
* @param {number} elementValue
* @returns true if the elements should be swapped
*/
_shouldSwap(parentValue, elementValue) {
    return this.isMaxHeap
        ? parentValue < elementValue
        : parentValue > elementValue
}

Let's unpack what the _reorderElements function is doing:

We get the value of the element at elementIndex.
We find its parent's index and its value in content.
Iteratively, we perform the following steps:
- If elementIndex is the first element, getParent will return null, so we don't enter the loop.
- We compare the parent's value with the element's value. If they are in the right order, we just stop the loop.
- If they are not in the right order, we swap their values and we update elementIndex to point to its new location (where its parent used to be); we then find its new parent.

What this function achieves is to move up the new element until it's located in its correct place. At each step, we don't need to check the parent's current children, as we know for sure that the parent will be larger or equal in priority.

Time complexity

How many iterations does this function take? At most, we do as many swaps as the tree's height. Since the content is structured as a balanced binary tree (a property that comes from being a complete tree), we know that the tree's height is logn where n is the number of elements in the tree.

So, it follows that the time complexity for the push operation is O(logn).

Poll

The poll operation is slightly more complicated. We will perform the following steps:

Get the value of the first element in content.
Mark the first element for deletion. In this implementation, we do so by setting its value to null.
Move the null element down the heap, swapping it with one of its children (the one with the highest priority).
Stop until the null element is at the end of the array.

/**
* Read the element with the highest priority
* and remove it from the heap.
*
* @returns the element with the highest priority
*/
poll() {
    if (this.content.length === 0) {
        // optionally, throw and exception
        return null
    }
    const result = this.content[0]

    this.content[0] = null

    HeapUtils.heapifyOne(this.content, 0, this.isMaxHeap)
    if (this.content[this.content.length - 1] === null) {
        this.content.pop()
    }
    this.size -= 1
    return result
}

The first lines take care of retrieving the first element from contents. In this case, we might want to throw an exception, as this operation is expected to update the contents of the heap.

We set the first element's value to null and then we start to re-order the heap with the function heapifyOne:

/**
* Heapify on just one element at startIndex
*
* @param {array} content
* @param {number | null} startIndex
* @returns
*/
heapifyOne: function (content, startIndex, isMaxHeap) {
    if (startIndex === null || startIndex >= content.length) {
        return
    }

    const [left, right] = HeapUtils.getChildren(content, startIndex)

    // If both are null, no need to swap elements
    if (left === null && right === null) {
        return
    }

    // If one of the children is null, swap the parent with
    // the one that is not null
    if (left === null || right === null) {
        HeapUtils.swapAndHeapify(content, startIndex, left || right, isMaxHeap)
        return
    }

    function getElementToSwap() {
        if (isMaxHeap) {
        return content[right] > content[left] ? right : left
        }
        return content[right] < content[left] ? right : left
    }

    function isLeaf(index) {
        const children = HeapUtils.getChildren(content, index)
        return children[0] === null && children[1] === null
    }

    let d = getElementToSwap()

    if (d === left) {
        const leftc = isLeaf(left)
        const rightc = isLeaf(right)

        if (leftc && rightc) {
        HeapUtils.swap(content, left, right)
        d = right
        }
    }

    HeapUtils.swapAndHeapify(content, startIndex, d, isMaxHeap)
    return
},
}

This function makes use of several helper functions, which are defined below:

/**
* Find the indices of the child elements for a given parent.
* The indices can be null if:
* - The index is out of bounds
* - The value of content at the index is null
*
* @param {array} content the list of elements
* @param {number} parentIndex the index of the parent element
* @returns [number | null, number | null], for the indices of both children.
*/
getChildren: function (content, parentIndex) {
    const left = parentIndex * 2 + 1
    const right = parentIndex * 2 + 2

    function getValidChild(index) {
        if (index < content.length && content[index] !== null) {
        return index
        }
        return null
    }
    return [getValidChild(left), getValidChild(right)]
},

/**
* Swap the values of two indices
*
* @param {array} content - the list of elements
* @param {number} index1 - element 1
* @param {number} index2 - element 2
*/
swap: function (content, index1, index2) {
    const t = content[index2]
    content[index2] = content[index1]
    content[index1] = t
},

/**
* Swap two elements at startIndex and otherIndex,
* then heapify on otherIndex
*
* @param {array} content
* @param {number} startIndex
* @param {number} otherIndex
*/
swapAndHeapify: function (content, startIndex, otherIndex, isMaxHeap) {
    HeapUtils.swap(content, startIndex, otherIndex)
    HeapUtils.heapifyOne(content, otherIndex, isMaxHeap)
},

Let's break down the heapifyOne function.

This function is called heapifyOne because it will only re-order the elements in the tree that are descendants of startIndex. It's important to make this distinction because you will find that some implementations of heapify will check all the elements in the heap to make sure they are in their right place.

This distinction is what makes our heap more performant: We assume that at any given moment the heap has the right order. It's only during push or poll that we need to change the element's order, but in those operations, we can focus only on the ascendants (parents) or descendants (children) of the elements being added or removed. We don't need to check the rest of the elements because we know they will be in the right place.

Note: The common implementation of heapify is useful to build priority queues out of unordered arrays in place. In our implementation, we can do the same thing by inserting one element at a time out of the unordered array. The downside of our implementation is that it requires us to copy the array into the heap. If you need to performantly convert existing arrays into heaps, it's better to use a regular implementation of heapify.

The first step in heapifyOne is to check if startIndex is valid (non-null and within the range of content). If it isn't, we stop.

Then we find the children elements of startIndex. Here, there are a couple of cases we need to address:

The element has no children. In this case, we don't need to swap any elements, as we reached the bottom of the array.
The element has only one child. This happens if there are only two elements in the heap, or if the element is almost at the bottom of the tree/array. In this case, we swap the element with the existing child; we don't need to compare values as we know that the parent is null and needs to be moved to the bottom.
The element has two children. In this case, we swap the null element with the child with the highest priority, guaranteeing that the heap keeps its properties.
After swapping the elements, we recursively apply heapifyOne to the new position of the null element. This recursion will stop once we reach the bottom of the heap.

Let's give some more thought to the case where the null element has two children. Why don't we need to re-order this whole sub-tree?

When we swap the null element with the child with the highest priority, the whole sub-three -except the null value- still retains its properties. We know that each child has a lower or equal priority than its parent, so moving up one of the children -let's say the left one- will not affect the order of the other children. The following image shows this process:

Notice a small contradiction here. In the image above, the generated tree is not strictly complete. The deepest sub-tree has a null value in its left child and a valid element in the right child. We cannot just swap the null child with the other non-null child, because the non-null child may have children on their own.

These null "holes" happen in a very specific condition:

The null node has two children.
The two children are not null.
The chosen children to be swapped with null is the left node.
Both children are "leaf" nodes: They have no children on their own.

The following code from heapifyOne takes care of amending these "holes":

function isLeaf(index) {
  const children = HeapUtils.getChildren(content, index)
  return children[0] === null && children[1] === null
}

let d = getElementToSwap()

if (d === left) {
  const leftc = isLeaf(left)
  const rightc = isLeaf(right)

  if (leftc && rightc) {
    HeapUtils.swap(content, left, right)
    d = right
  }
}

HeapUtils.swapAndHeapify(content, startIndex, d, isMaxHeap)

If we find that the chosen node to swap with null is the left child, we check if both are "leaf" nodes. If they are, we can safely swap the values of both children and swap the null value with the right child (which now has the value of the left child).

This simple process guarantees that the null value will be placed at the right child of the bottom sub-tree, keeping the properties of the heap without having to re-order the whole tree.

The heap would still work without amending these "holes", but the inner content array could grow more than needed, keeping a lot of null values between valid elements.

Once heapifyOne completes its recursive execution, the null element will be at the bottom of the heap, and we can safely remove it from content with content.pop().

Time complexity

The number of swaps we need to perform to move the deleted node from the heap is at most, again, the height of the tree, or O(logn).

Time complexity: Heap sort

As we mentioned before, we can sort a list of numbers by adding each number to the heap and then pulling them out. We can break down this process of "Heap-sort" into parts:

Number of push operations: n, where n is the number of elements in the list.
Number of poll operations: n.

Since each push and poll operation takes O(logn), the whole sorting operation takes O(nlogn) + O(nlogn), which can be reduced to O(logn), the same time complexity of other stable sorting methods like merge sort.

Challenge: Finding the K largest elements

To complete this post, let us look at a common use for priority queues: Finding the K largest elements in a list.

We can break this problem down into simple steps:

We iterate through each element in the list called numbers.
Using a min heap (more details on this counter-intuitive choice to follow), we keep track of the largest elements in the list.
If the size of the heap is smaller than K (the number of largest elements to find), we just push the number to the heap.
If the size of the heap is equal to or larger than K (meaning we already have K numbers in the heap) and the new number to add is larger than the element at the top of the heap, we remove the smallest element in the heap and push the new number. This is why we chose a min heap in this problem, to quickly locate the smallest value and replace it with the new, larger number.

function nLargestNumbers(numbers, K) {
  const heap = new Heap()
  for (const num of numbers) {
    if (heap.size < K) {
      heap.push(num)
    } else {
      if (num > heap.peek()) {
        heap.poll()
        heap.push(num)
      }
    }
  }
  return heap.content
}

console.log(nLargestNumbers([1, 4, 2, 4, 5, 6, 3, 9, 1], 3))
// result: [5, 6, 9]

Time complexity for "Finding the N largest elements" using a min heap

In the worst case scenario where the numbers are sorted in ascendent order, the overall time complexity will be O(log(n)), where n is the number of elements in the list numbers. This is because at every iteration we will have to push a new element into the heap, which takes O(logn). As you can see, this is no better than just sorting the numbers and getting the last (or first) K elements.

However, in average, using a heap will perform fewer operations than any sorting algorithm: If the top K elements are near the beginning of the array, once the heap contains them it will not have to do any more push or poll iterations. In the best-case scenario where the first K elements are at the beginning of the array, the overall time complexity using a priority queue will be O(KlogK) + O(n-K), which approximates O(n) as K gets smaller.

Another scenario for this problem where priority queues will perform consistently better is where the list of numbers is not given at once. If the method is expected to receive one element at a time, using something like merge-sort will require n sort operations, or n*O(logn), whereas the priority queue will take O(nlogn) consistently.

Visualizing a priority heap, step by step

As a bonus, here is a step-by-step animation of the priority queue we just implemented. Feel free to play with it, changing configurations and playing back and forth the animation using the slider.

Conclusion

Priority queues are simpler than they seem. As long as we maintain their main properties (each child of a given element is of a lower or equal priority as the element, and there is only one single root element, and the push and poll operations can be done in O(log) time), any method that we use to update the priority queue will work.

The implementation we saw here does minimal work to fulfill the definition of a priority queue. Other implementations may include keeping the order of insertion when elements have the same priority or keeping track of the top K elements in constant time by guaranteeing that the direct children of each node are ordered from left to right.

I hope this post makes things clearer for you next time you think about using priority queues. Understanding a tool a little better makes us more comfortable using it.

Create a ML classification model with PyTorch

Pedro F Marquez — Mon, 03 Oct 2022 23:09:08 +0000

Machine Learning (ML) is one of the topics in software development that is surrounded by a mystic aura: There is a certain feeling of magic around a piece of code that seems to be able to predict events or recognize complex patterns. This is a reason why ML keeps being confused with Artificial Intelligence.

One of the most common tasks in ML is classification: Creating a model that, after being trained with a dataset, it can label specific examples of data into one or more categories.

In this post, we will use PyTorch -one of the most popular ML tools- to create and train a simple classification model using neural networks (NN).

A warning about PyTorch and Neural networks

For all their power, tools used to create NNs like PyTorch and Tensorflow are not always the best choice for some ML tasks.

Development teams often choose these tools because they are popular and well-known. However, as more experienced ML practitioners will tell you, tasks like classification can more easily be achieved in Python by tools like scikit-learn using models like logistic regression.

Neural networks have a higher complexity than models like linear or logistic regression: They require more fine-tuning of parameters, and often also require more data, training time and computation power.

Having said that, let's review how a simple NN can be constructed for classification; we will revisit this topic in another post using scikit-learn.

A quick primer on Neural networks

Before reading further in this section let me give you a warning: If this is the first time you're reading about neural networks, this part will be complex and hard to fully grasp at first. It is OK if you don't understand what each term in this section means. Feel free to skim through this section and move forward, as the code examples may make things clearer.

The input for a neural network (at both training and prediction time) is a matrix of data of size (r,c), where each row r is a data example and each column c is an attribute. The output depends on the layer structure of the NN, but typically is another matrix of size (r, t), where r is the same number of examples as the input, and t is the number of columns used to represent the result of our model.

Neural Networks are composed of layers: Each layer transforms the data (e.g. applying a linear transformation by multiplying the matrix of data with a vector of weights that are updated at each step of the training process) and passes it to the next layer for further transformations. The last layer in the network outputs the expected output (in our case, the classification result) for the model.

Individual layers are composed of neurons, where the number of neurons corresponds to the input size of each layer. In the following image we see a network where the first layer (the input layer) has three neurons; so the size of the input layer would be (r=Examples,c=3).

At each training step, we measure the error for the model: How many times the prediction was wrong when compared with a labeled set of test data. The model then uses the error's value and updates the weights to minimize the expected error.

Each layer's weights are updated using a process called backpropagation: The training algorithm calculates the partial derivates of each weight to find the direction in the linear space (remember, weights are vectors) in which we need to update the weight's value for minimizing the expected error in the predictions.

For instance, if the weight needs to increase its value, its partial derivate will point in the opposite direction than the case in which we would need to decrease its value.

For a full in-depth explanation of how Neural Networks and backpropagation work, take a look at the free, online book "Neural Networks and Deep Learning". The following video also provides a great explanation on backpropagation:

The task

For this example, we will address the popular practice problem "Titanic - Machine Learning from disaster" from Kaggle. For those who don't know Kaggle, it is a portal where tons of datasets and data problems can be found. Companies sometimes post their datasets and create contests for people to try to solve specific data problems.

The problem to solve for this dataset is to -given a dataset containing examples of passengers who traveled in the famed ship Titanic, along with an attribute specifying whether they survived the sinking or not- predict whether a specific example of a passenger would have survived.

The model building process

The steps to construct ML models may vary depending on the task at hand. However, most models follow the same pattern:

Find the data that will use for training
Load and clean the data
Encode the data
Split data into training, test and validation sets
Define and train the model
Validate the accuracy of the model

Find the data that will use for training

The data needs to be labeled: A set of multiple examples of the data we plan to use for our predictions, along the label, which in our case is "Survived" -with a value of 1 if the passenger survived, and a value of 0 otherwise. Each column in the data is an attribute, while each row is an instance or example of the data.

Load and clean the data

This is where most of the work of a data engineer is spent. The data will contain multiple undesirable characteristics that would prevent our model from successfully training and predicting:

Empty attributes. Most of the time, we fill empty attributes with the mean or the mode of the other attributes that do have value. Other times, we might just drop the column or the row.
Redundant data. For instance, some columns are a direct function of others and don't provide any extra information to the model. We can safely drop these columns.

Encode the data

PyTorch models only accept numeric values as input. We must encode string or categorical columns into numeric values. For this, we have a few options:

One-hot encoding: Columns that contain categories (e.g. "Sex" contains values like "Male" or "Female") can be converted into numeric values by adding a column for each category, and setting a value of 1 for those columns in that category (and 0 for those that don't).
Label encoding: Assign direct numeric values to each category (e.g. 1 to "Female" and 2 to "Male"), and replace the string values with their numeric representation. The downside of this approach is that it gives an implicit numeric relationship to the categories: The model will assume that the categories may be ordered when there is no real ordinal relationship in attributes like "Sex".

Split data into training and test sets

Before we start training our model, we must split all our data into three parts: A training set, a test set and a validation set. The majority of the data (e.g. 80%) goes into the training set, and a smaller percentage of the data (let's say 10% and 10%) goes into the test and validation sets. Let me elaborate on why.

At each step of the training process, we want to evaluate our model to see how well the training process is going and make the necessary adjustments. For this, we try to predict a few examples and measure their error.

To avoid overfitting, we take these evaluation examples from a set of data that wasn't used for training (otherwise our model would be "cheating", as it would know the "answer" for those examples). For that, we reserve a part of our dataset and leave it out of the training dataset. To this held out data, we call it test set and the process of using this set during training is called cross-validation.

_Note: To keep things simple we will not use cross-validation for our example, so we will only split our data into training and validation (90%-10%).

Then, once we complete the training of our model, we need to do the last evaluation of the model's performance. For this, we will use the validation set. Keeping the validations set out of the training process will provide a more accurate measure of the model's performance, for the same reason we keep the test set out of the training data.

A great explanation of the importance and process of splitting datasets can be in this post by Cassie Kozyrkov.

Define and train the model

Defining the model is the second most important task in this process.

In the case of NNs, we must define the number and size of each layer in the network, activation layers, and the hyperparameters for the training (batch size, learning rates, number of training iterations and so on). This configuration is what makes NNs more complex to define and train than simpler models like logistic regression.

Validate the accuracy of the model

Finally, we use our trained model to make predictions on our validation data set. We compare the answers given by the model with the actual labels of the validation set, and we calculate the statistical information of how many times the model did a bad prediction. Some measures of this performance are accuracy, precision, recall and F1. Here's a post about each of these measurements and how they reflect different parts of a model's performance.

The code

You can find all the code used in this example in this Google Collab notebook. This whole example is written in Python, so we will use Python libraries for ML.

In this example, we will use:

Pandas to load and manipulate our dataset.
numpy for matrix and vector operations.
scikit-learn for splitting our dataset
PyTorch for building the NN.

import numpy as np
import pandas as pd
from sklearn.model_selection import train_test_split

Read the data from the CVS files

We download the train.csv dataset from Kaggle and put it in the same directory as our Python code.

The following is a table with the description of each attribute in the data:

Variable	Definition	Key
survival	Survival	0 = No1 = Yes
pclass	Ticket class	1 = 1st2 = 2nd3 = 3rd
sex	Sex
Age	Age in years
sibsp	# of siblings / spouses aboard the Titanic
parch	# of parents / children aboard the Titanic
ticket	Ticket number
fare	Passenger fare
cabin	Cabin number
embarked	Port of Embarkation	C = CherbourgQ = QueenstownS = Southampton

We then load our data into memory using Pandas:

train = pd.read_csv('train.csv')

We can see some examples of the dataset using Panda's head method:

train.head()

PassengerId	Survived	Pclass	Name	Sex	Age	SibSp	Parch	Fare	Cabin	Embarked	Title	FamilySize
0	1	0	3	Braund, Mr. Owen Harris	male	22	1	A/5 21171	7.25	nan	S	Mr	Couple
1	2	1	1	Cumings, Mrs. John Bradley (Florence Briggs Thayer)	female	38	1	PC 17599	71.2833	C85	C	Mrs	Couple
2	3	1	3	Heikkinen, Miss. Laina	female	26	0	STON/O2. 3101282	7.925	nan	S	Miss	Single
3	4	1	1	Futrelle, Mrs. Jacques Heath (Lily May Peel)	female	35	1	113803	53.1	C123	S	Mrs	Couple
4	5	0	3	Allen, Mr. William Henry	male	35	0	373450	8.05	nan	S	Mr	Single

Clean the data

A few transformations are needed:

Drop columns that don't provide information for the classification. For instance, the ticket number ticket provides little to no information to predict if a passenger survived that cannot be found in other attributes in the data.
Fill empty values (e.g. fill empty age values with the median of existing age records)
Group columns into more meaningful labels:
- E.g. add the values SibSp and Parch into a single category FamilySize.
- Transform FamilySize into a category (e.g. single, couple, large).
Drop redundant columns (e.g. SibSp and Parch, as they have been merged into FamilySize)

If you imported more than one dataset (like an extra test dataset, or unlabeled data to predict), this cleanup needs to be done for all of them.

Note: Data cleanup is specific to the dataset. You need to understand what your dataset is trying to achieve, which columns have a direct relation with the prediction result, which columns are unnecessary and how to better fill empty values and group categories.

def clean_data(dataset):
  dataset_title = [i.split(',')[1].split('.')[0].strip() for i in dataset['Name']]
  dataset['Title'] = pd.Series(dataset_title)
  dataset['Title'].value_counts()
  dataset['Title'] = dataset['Title'].replace(['Lady', 'the Countess', 'Countess', 'Capt', 'Col', 'Don', 'Dr', 'Major', 'Rev', 'Sir', 'Jonkheer', 'Dona', 'Ms', 'Mme', 'Mlle'], 'Rare')

  dataset['FamilySize'] = dataset['SibSp'] + dataset['Parch'] + 1

  def count_family(x):
    if x < 2:
        return 'Single'
    elif x == 2:
        return 'Couple'
    elif x <= 4:
        return 'InterM'
    else:
        return 'Large'

  dataset['FamilySize'] = dataset['FamilySize'].apply(count_family)
  dataset['Embarked'].fillna(dataset['Embarked'].mode()[0], inplace=True)
  dataset['Age'].fillna(dataset['Age'].median(), inplace=True)
  dataset = dataset.drop(['PassengerId', 'Cabin', 'Name', 'SibSp', 'Parch', 'Ticket'], axis=1)
  return dataset


train_clean = clean_data(train)

print(train_clean)

X = train_clean.iloc[:, 1:]
y = train_clean.iloc[:, 0]

X

We can find some examples of the result of this data cleanup.

Pclass	Sex	Age	Fare	Embarked	Title	FamilySize
0	3	male	22.0	7.2500	S	Mr	Couple
1	1	female	38.0	71.2833	C	Mrs	Couple
2	3	female	26.0	7.9250	S	Miss	Single
3	1	female	35.0	53.1000	S	Mrs	Couple
4	3	male	35.0	8.0500	S	Mr	Single
...	...	...	...	...	...	...	...
886	2	male	27.0	13.0000	S	Rare	Single
887	1	female	19.0	30.0000	S	Miss	Single
888	3	female	28.0	23.4500	S	Miss	InterM
889	1	male	26.0	30.0000	C	Mr	Single
890	3	male	32.0	7.7500	Q	Mr	Single

Encode data

PyTorch only accepts numbers for attributes, so we convert all categorical/string data into numbers.

pd.get_dummies gets the values in categorical_columns and creates a new numeric column for each category in those columns (e.g. theSex:{male, female} column is split into two columns: SexMale: 0 and SexFemale: {1, 0} )

Just as with data cleanup, this also needs to be done for unlabeled data for which we will try to get actual predictions out of the trained model

def encode_data(X):
  categorical_columns = ['Pclass','Sex', 'FamilySize', 'Embarked', 'Title']

  X_enc = pd.get_dummies(X, prefix=categorical_columns, columns = categorical_columns, drop_first=True)
  return X_enc

X_enc = encode_data(X)
X_enc.head()

Age	Fare	Pclass_2	Sex_male	FamilySize_InterM	Embarked_Q	Title_Miss	Title_Mr	Title_Mrs	Title_Rare
0	22.0	7.2500	1	1	0	1	0	1	0
1	38.0	71.2833	0	0	0	0	0	0	1
2	26.0	7.9250	1	0	1	1	1	0	0
3	35.0	53.1000	0	0	0	1	0	0	1
4	35.0	8.0500	1	1	1	1	0	1	0

Split test data into train and validation datasets

We split our training data into train and val. During training, the model will learn using train and we then will use val to measure the number of times our model predicted correctly or not.

We split 90% for training and 10% of data for validation.

Don't forget to also split the labels y, so they are assigned to the correct sub-datasets

# split training data into training and test
x_train, x_val, y_train, y_val = train_test_split(X_enc, y, test_size = 0.1)

We confirm that the data has been split in the expected percentages

print(X_enc.shape, x_train.shape, x_val.shape)

(891, 14) (801, 14) (90, 14)

The PyTorch model

This PyTorch model defines a neural network with the following layers:

Linear transformation with an input of 14 columns (one for each of our data columns, after cleanup and encoding), and output of size 270
A dropout layer with a chance of %1 of dropping a specific row (for reducing overfitting).
A ReLu activation layer
Another linear transformation with an input of 270 (the output of the first linear transformation) and an output of 2 (the probabilities of surviving/not surviving)

Here are some links for resources, if you want to gain more understanding of each of these components:

# Pythorch model:

import torch
import torch.nn as nn
import torch.nn.functional as F

class Model(nn.Module):

    def __init__(self):
        super(Net, self).__init__()
        self.fc1 = nn.Linear(14, 270)
        self.fc2 = nn.Linear(270, 2)

    def forward(self, x):
        x = self.fc1(x)
        x = F.dropout(x, p=0.1)
        x = F.relu(x)
        x = self.fc2(x)
        x = torch.sigmoid(x)

        return x

model = Model()

Training parameters

The following are the hyperparameters we will set into our model:

We train the model in batches of 50 samples to avoid over-fitting.
We train the model for 50 epochs/iterations, to iteratively train the model.
We train the model with a learning rate of 0.01
To calculate the error during training (which will allow the model to update weights through backpropagation), we use the Cross entropy loss function (commonly used for binary classification).
We use the Adam optimizer for finding the gradients.

Some more information on cross-entropy loss can be found in this article.

# Model params:
batch_size = 50
num_epochs = 50
learning_rate = 0.01
batch_no = len(x_train) // batch_size

criterion = nn.CrossEntropyLoss()
optimizer = torch.optim.Adam(net.parameters(), lr=learning_rate)

Train the model

We train the network model. More details in the code comments.

from sklearn.utils import shuffle
from torch.autograd import Variable

# Iterate for the number of epochs
for epoch in range(num_epochs):
    # Print the epoch number every 5 epochs
    if epoch % 5 == 0:
        print('Epoch {}'.format(epoch+1))

    # Shufle the datasets to randomize the data rows that
    # will be added to the batch and avoid training over the same 50 rows
    # at each epoch
    x_train, y_train = shuffle(x_train, y_train)
    # Mini batch learning
    for i in range(batch_no):
        start = i * batch_size
        end = start + batch_size

        # Convert the Pandas dataset into PyTorch variables of the size
        # of the batch
        x_var = Variable(torch.FloatTensor(x_train.values[start:end]))
        y_var = Variable(torch.LongTensor(y_train.values[start:end]))

        # Restart the gradients
        optimizer.zero_grad()

        # Run a training step: Pass the training data to
        # the neural network layers
        ypred_var = model(x_var)

        # Calculate the training loss
        loss =criterion(ypred_var, y_var)

        # update the gradients based on the training loss for the batch
        loss.backward()
        optimizer.step()

Epoch 1
Epoch 6
Epoch 11
Epoch 16
Epoch 21
Epoch 26
Epoch 31
Epoch 36
Epoch 41
Epoch 46

Measure the model's performance

We now predict the labels for our validation set x_val.

The results of the prediction in result = model(test_var) will be a matrix of where the rows are the predicted values for each validation row, and the columns are:

The percentage of probability that the passenger survived
The percentage of probability that the passenger didn't survive

Both columns should add up to 1 for a given row, and the closer to 1 for each of those columns, the most certain the model is about the result

## convert the Pandas dataframe for the validation data to a PyTorch variable
validation_data = Variable(torch.FloatTensor(x_val.values), requires_grad=True)

## Use "no_grad" to not update the model's gradients, as we are not training
## this model anymore
with torch.no_grad():
    ## get the predicted values
    result = model(validation_data)

## Sample 5 results
result[0:5, :]

tensor([[1.0000e+00, 1.0036e-13],
        [9.9993e-01, 5.8530e-14],
        [1.0000e+00, 1.0000e+00],   
        [1.0000e+00, 1.0000e+00],
        [9.9999e-01, 5.0890e-12]])

Since we want a binary classification (survived/didn't survive), we get the result with the highest percentage of confidence (1=survived, otherwise 0) using PyTorch's max function.

values, labels = torch.max(result, 1)
## sample the first 5 results
labels[0:5]

tensor([0, 0, 0, 0, 0])

For measuring the model's performance, we calculate a simple percentage of accuracy:

num_right = the number of rows where the prediction matched the actual value in the validation set
all_rows = the total number of rows
accuracy = num_right / all_rows

num_right = np.sum(labels.data.numpy() == y_val)
all_rows = len(y_val)
print('Accuracy {:.2f}'.format(num_right / all_rows))

Accuracy 0.78

As the number of correctly predicted examples increases, the accuracy will also increase. In theory, a perfect model would achieve an accuracy of 1.0, and the worst classifier -which is smart enough to beat the probability of getting a result right by accident, and always chooses the wrong prediction- would achieve an accuracy of 0.0.

Performance is an extremely subjective concept. An accuracy of 0.78 is really good for this problem, but when we are dealing with problems that have a critical impact on people's lives (medical and finance-related problems), 0.78 may not be good enough for considering a model as performant.

An observation about the accuracy results

The accuracy is based only on the existing examples in the dataset. Once we use it in data outside of this dataset, accuracy may be lower as new data may contain attributes that don't exist in this dataset or may have attributes that indicate a different correlation to the classification result. This is why, the more training data we have, the more accurate our performance measurement will be.

The accuracy will also vary between model training executions, as the training data is split randomly between the training and validation datasets.

Notice that accuracy is a kind-of misleading metric: If the model predicts all data samples as "survived" (which is indicative of an incorrect model), you might still get relatively high accuracy. This is why we use other metrics like the F1 score, and do further analysis on the results with analysis tools like confussion matrices and visual representations of the data.

Another downside of just looking at accuracy leaves out a lot of information that may be critical for evaluating a model. Going back to the example of medical applications: For a model that predicts whether a patient has some sort of disease based on their medical history, the impact of a false negative -predicting false when the patient does have the disease- is considerably higher than those of false positives -predicting a patient may have the disease when they don't-.

A false positive can be corrected by doctors performing more studies on the patient, while false negatives may leave a patient without the treatment they need. Precision and recall are good metrics to know -in addition to accuracy- in this case.

Conclusion

In this post we created and trained a neural network for classification in PyTorch.

The work for building Machine Learning models is 80% data analysis and cleanup, and 20% model configuration and coding. The model's performance will depend completely on the data we use to train it; If the data is not clean (it has multiple empty values or redundant information), the model's performance will suffer.

Neural Networks are a powerful tool for ML tasks, but they also have added complexity that other, more focused, tools. This example could have been solved with logistic regression, with similar performance. The added complexity of NNs can lead to over-complicated models that perform worst than simpler models.

NNs also require some understanding of more complex topics like backpropagation, which can be a roadblock for people starting to work in ML.

Having said that, NNs have the advantage of capturing hidden signals in the dataset that would require extra effort in data engineering to get a good performance in simpler models.

Extra material

Common Rust problems for beginners: Ownership & mutability

Pedro F Marquez — Mon, 26 Sep 2022 15:34:50 +0000

Learning how to code in Rust can feel like a daunting task. While it is a very powerful language it is not precisely known for being a simple language.

The same concepts that make Rust such a great language are the same that may cause problems for newcomers: ownership, lifetimes, among others.

It is not that these concepts are inherently difficult, it's just that they provide safety checks that other languages don't; these safety checks can become an obstacle to people who is familiar with other, more forgiving, languages.

This is why, when someone is writing their first lines of Rust and tries to do simple tasks like passing an object to a function, the compiler will stop them with errors.

Rust compiler relies heavily on static code analysis to find memory errors. While other languages like C or Java allow developers to write code that may result in undesired states (e.g. NullPointerException and stack overflows) -which then need to be handled during runtime with error handling and exceptions-, Rust will not allow such code to compile.

In this post, we will focus on common problems that any beginner will find when they start their journey into the world of Rust.

Trying to modify immutable variables

In the following example, we create an instance of a struct called Book, and we try to assign a different value to its title property:

struct Book {
    title: String
}

fn main() {
    let book = Book {
        title: String::from("The Rust Programming Language")
    };

    book.title = String::from("Another book")
}

If we try to compile this code, we will get the following error:

error[E0594]: cannot assign to `book.title`, as `book` is not declared as mutable

Solution

This first problem is easy to spot. By default, all Rust variables are declared as immutable (read-only/constant). The goal of this decision is for developers to be explicit when they expect a variable to change values.

If we want to make book mutable, we must declare it as let mut book:

struct Book {
    title: String
}

fn main() {
    let mut book = Book {
        title: String::from("The Rust Programming Language")
    };

    book.title = String::from("Another book")
}

Another flavor of this same error can be found in the following example:

let x = 1;
x = 2;

error[E0384]: cannot assign twice to immutable variable `x`

The same solution applies: Use let mut x = 1 to allow future assignments to x.

Trying to use a "moved" variable

Following the previous example, we now define two functions that receive an instance of Book and try to do something with it (e.g. printing its title):

fn print_book(book: Book) {
    println!("Book: {}", book.title);
}

fn do_something_else(book: Book) {
    println!("Again: {}", book.title);
}

fn main() {
    let book = Book {
        title: String::from("The Rust Programming Language")
    };

    print_book(book);
    do_something_else(book);
}

If we try to compile this code, we will find the following error:

error[E0382]: use of moved value: `book`
  --> src/main.rs:19:23
   |
14 |     let book = Book {
   |         ---- move occurs because `book` has type `Book`,
   |               which does not implement the `Copy` trait
...
18 |     print_book(book);
   |                ---- value moved here
19 |     do_something_else(book);
   |                       ^^^^ value used here after move

Solutions

This problem is caused by Rust's ownership rules around data references.

Rust has no garbage collector like Java, nor it requires developers to manually "destroy" variable references to free the allocated memory. Instead, it uses ownership to define when to clear the memory referred by a variable: When execution reaches the end of a function, all variables owned by that function will go out of scope and their memory will be released.

When the program starts, the main function has ownership of the book variable. Then, when we call print_book(book), the function print_book now owns book. When execution reaches the end of print_book, book will be out of scope and its data is cleared.

There are multiple solutions to this:

Change print_book to return ownership of book to main.
Change print_book to receive a borrowed reference of book.

Solution 1: Returning ownership

The following code will compile correctly:

fn print_book(book: Book) -> Book {
    println!("Book: {}", book.title);
    book
}

fn do_something_else(book: Book) {
    println!("Again: {}", book.title);
}

fn main() {
    let mut book = Book {
        title: String::from("The Rust Programming Language")
    };

    book = print_book(book);
    do_something_else(book);
}

Notice that we changed book's declaration to make it mutable, so we can re-assign its value. Also, we updated print_book to return Book once it's done with it.

I'm not a big fan of this approach, as I find it verbose and difficult to deal with if print_book is expected to return something else.

Solution 2: Use a borrowed reference

The following code will compile correctly:

fn print_book(book: &Book) {
    println!("Book: {}", book.title);
}

fn do_something_else(book: Book) {
    println!("Again: {}", book.title);
}

fn main() {
    let book = Book {
        title: String::from("The Rust Programming Language")
    };

    print_book(&book);
    do_something_else(book);
}

Notice we updated print_book to receive an instance of &Book, which is an immutable reference to an instance of Book. This is called borrowing. We also changed print_book(book) to print_book(&book), to send a reference of book to the function.

Now, print_book never gets ownership of book: It will receive the read-only reference. When execution reaches the end of print_book, book will not go out of scope, as main still owns it.

Just keep in mind that, in this example, do_something_else does get ownership of book, which is perfectly legal.

Another solution is to follow the compiler's advice and implement the Copy trait for Book, which then would create copies of book when passed to other functions. But then we would be using different instances of book, which has some downsides:

Extra memory allocation. If book is large and we copy it every time we pass it to a function, we are needlessly consuming memory.
Doesn't play well with mutations. If one of the functions is expected to change book's contents, then we must return the copy so the next functions can have access to book's updated values. This results in the same flow as returning ownership, but with extra steps.

Trying to borrow as both mutable and immutable in the same scope

Now, we will modify the book's title using a new function, and print it with another one. The following code will compile correctly:

fn rename_book(book: &mut Book) {
    book.title = String::from("Something else");
}

fn print_title(book: &Book) {
    println!("Book: {}", book.title);
}

fn main() {
    let mut book = Book {
        title: String::from("The Rust Programming Language")
    };

    rename_book(&mut book);
    print_title(&book);
}

However, you will find that doing the following -which seems like it's practically doing the same thing- does not compile:

fn rename_book(book: &mut Book) {
    book.title = String::from("Something else");
}

fn print_title(book: &Book) {
    println!("Book: {}", book.title);
}

fn main() {
    let mut book = Book {
        title: String::from("The Rust Programming Language")
    };

    let mut book1 = &mut book;
    let book2 = &book;

    rename_book(&mut book1);
    print_title(&book2);
}

error[E0502]: cannot borrow `book` as immutable because it is also borrowed as mutable
  --> src/main.rs:20:17
   |
19 |     let mut book1 = &mut book;
   |                     --------- mutable borrow occurs here
20 |     let book2 = &book;
   |                 ^^^^^ immutable borrow occurs here
21 |
22 |     rename_book(&mut book1);
   |                 ---------- mutable borrow later used here

Aren't we also borrowing as both mutable and immutable in the first example? Not really.

Solution

Rust will not allow us to create mutable and immutable references of an object in the same scope.

In the first example, the actual borrow happens once execution reaches each function rename_book and print_title, so there is only one borrowed reference at a time.

While the most obvious solution to this problem would be to use the first example where everything compiles correctly, let us explore a more contrived solution, which will shed some light on the ownership rules:

fn main() {
    let mut book = Book {
        title: String::from("The Rust Programming Language")
    };

    {
        let mut book1 = &mut book;
        rename_book(&mut book1);
    }

    let book2 = &book;
    print_title(&book2);
}

This little workaround makes our code compilable again. When we wrap the borrowed instance of book1 and rename_book with a block {}, we are creating a new scope for them. Any variable declared inside the block will go out of scope once execution reaches the block's end. Then, by the time execution reaches let book2, book1 will not exist anymore and we will only have a single borrow.

Trying to keep a reference to a struct value

Coming from other languages, it's not uncommon for us to try to keep a reference for an object's attribute:

fn main() {
    let mut book = Book {
        title: String::from("The Rust Programming Language")
    };

    let title = book.title;

    rename_book(&mut book);

    println!("Title: {}", title);
}

While this is correct in other languages, Rust will complain:

error[E0382]: borrow of partially moved value: `book`
  --> src/main.rs:20:17
   |
18 |     let title = book.title;
   |                 ---------- value partially moved here
19 |
20 |     rename_book(&mut book);
   |                 ^^^^^^^^^ value borrowed here after partial move

Solution

When we do let title = book.title, we are passing ownership of the title attribute from book to the main function, which means we cannot use book in rename_book. We could give title's ownership back to book, but that would prevent us from using the title variable by itself.

To fix this problem, we could clone the title as let title = book.title.clone(). However, this approach only works if one, title implements the Clone trait (which strings do by default), and two, whatever we use title for doesn't expect to get the actual value of book.title:

fn rename_book(book: &mut Book) {
    book.title = String::from("Something else");
}

fn print_title(book: &Book) {
    println!("Title: {}", book.title);
}

fn main() {
    let mut book = Book {
        title: String::from("The Rust Programming Language")
    };

    let title = book.title.clone();

    rename_book(&mut book);

    println!("Title: {}", title);
    print_title(&book);
}

Title: The Rust Programming Language
Title: Something else

If we wanted to keep a real reference to book.title, we must borrow a reference to it, being careful of not creating immutable and mutable borrows in the same scope:

fn main() {
    let mut book = Book {
        title: String::from("The Rust Programming Language")
    };

    {
        rename_book(&mut book);
    }

    let title = &book.title;

    println!("Title: {}", title);
    print_title(&book);
}

Title: Something else
Title: Something else

Of course, this is a very self-contained example that will not work in other cases. For instance, what if we want to keep a reference to book.title -independent from book- and return it from a function?

fn create_book() -> (Book, &String) {
    let book = Book {
        title: String::from("The Rust Programming Language")
    };

    return (book, &book.title);
}

fn main() {
    let (mut book, title) = create_book();
    rename_book(&mut book);
    println!("{}", &title);
}

The first problem we will find is the following:

error[E0106]: missing lifetime specifier
  --> src/main.rs:16:28
   |
16 | fn create_book() -> (Book, &String) {
   |                            ^ expected named lifetime parameter
   |
   = help: this function's return type contains a borrowed value,
    but there is no value for it to be borrowed from
help: consider using the `'static` lifetime

We will not go in-depth on the topic of lifetimes, but what Rust is saying is that, since we're returning a borrowed reference to a String, the developer needs to add a lifetime to the return type declaration to let Rust know how long is that reference expected to live.

We can try -spoiler-alert, unsuccessfully- to follow Rust's compiler advice and annotate &String with a 'static lifetime:

fn create_book() -> (Book, &'static String) {
    let book = Book {
        title: String::from("The Rust Programming Language")
    };

    return (book, &book.title);
}

error[E0382]: borrow of moved value: `book`
  --> src/main.rs:21:19
   |
17 |     let book = Book {
   |         ---- move occurs because `book` has type `Book`, which does not implement the `Copy` trait
...
21 |     return (book, &book.title);
   |             ----  ^^^^^^^^^^^ value borrowed here after move
   |             |
   |             value moved here

By the end of the create_book function, book is moved out of the function, and ownership is passed to the caller (main()); because of this, we cannot borrow book.title (this is the same problem we found in the first example of this post).

Even if we try to store the book title's borrowed reference, we cannot just return a borrowed reference to book.title:

fn create_book() -> (Book, &'static String) {
    let book = Book {
        title: String::from("The Rust Programming Language")
    };

    let title = &book.title;

    return (book, title);
}

error[E0515]: cannot return value referencing local data `book.title`
  --> src/main.rs:23:12
   |
21 |     let title = &book.title;
   |                 ----------- `book.title` is borrowed here
22 |
23 |     return (book, title);
   |            ^^^^^^^^^^^^^ returns a value referencing data owned by the current function

Even if it looks like we're passing the ownership of title to main(), in reality, we are trying to return a reference to a variable owned by create_book; such variable will also go out of scope at the end of create_book.

What options do we have here?

Using Rc to create immutable references

Basic ownership rules work well in simpler applications. However, larger, more complex code will inevitably require getting multiple references to a single object and passing it across functions. For those cases, std::rc::Rc -Reference Counter- is a handy tool.

Reference counters allow us to create multiple immutable references to a single object. These references can be safely passed from one function to another:

use std::rc::Rc;

struct Book {
    title: Rc<String>
}

fn create_book() -> (Book, Rc<String>) {
    let book = Book {
        title: Rc::new(String::from("The Rust Programming Language"))
    };

    let title = Rc::clone(&book.title);

    return (book, title);
}

fn main() {
    let (book, title) = create_book();

    println!("{}", book.title);
    println!("{}", title);
}

The Rust Programming Language
The Rust Programming Language

Under the hood, Rc keeps a count of the number of references that have been created. Once all references go out of scope, the object they were pointing at will be removed from memory. In a way, Rc works as a very simplified Garbage Collector.

This example is a bit misleading, as it looks like we're cloning book.title's value, when we're cloning a reference to it. This difference is clearer if we update book.title's value.

Using Rc and RefCell to create mutable references

Reference counting by default is immutable. If we want to update book.title we must introduce RefCell to make title a mutable reference:

use std::rc::Rc;
use std::cell::RefCell;

struct Book {
    title: Rc<RefCell<String>>
}

fn create_book() -> (Book, Rc<RefCell<String>>) {
    let book = Book {
        title: Rc::new(RefCell::new(String::from("The Rust Programming Language")))
    };

    let title = Rc::clone(&book.title);

    return (book, title);
}

fn rename_book(book: &Book) {
    book.title.replace(String::from("Something else"));
}

fn main() {
    let (book, title) = create_book();

    rename_book(&book);
    println!("{}", book.title.borrow());
    println!("{}", title.borrow());
}

This code will print:

Something else
Something else

With this example, we demonstrate two points:

The Rc::clone(&book.title) returns a clone of the reference to book.title, not a clone of book.title's value. Otherwise, println!("{}", title.borrow()) would print the old value "The Rust Programming Language".
Using an Rc<RefCell<String>> variable type for book.title allows us to create copies of its reference and mutate its value.

A warning on Rc and RefCell

While Rc and RefCell allow us to create a more complex workflow, they also increase the complexity within your application. Just from the ergonomics point of view, having to pass around variables with <Rc<RefCell<T>> gets very verbose quickly.

Also, using reference counters when it's not needed increases the risk of memory leaks: If we keep a reference copy for too long, the underlying memory it points to may not be released. In cases where we implement things like HTTP servers using Rust, we might accumulate reference copies to objects that are not used, increasing the amount of consumed memory. This is a common problem in languages like Java, where keeping references to unused variables also lead to memory leaks that the Garbage Collector cannot fix.

When possible, we must avoid overusing these tools and try to work with the basic rules of ownership.

With this last example, we reached the end of this post. However, in future posts we will explore more potential difficulties that beginners find when learning Rust.

Conclusion

Rust ownership and mutability rules are completely different from the way other languages enforce memory safety. While these tools can make it difficult for newcomers to ramp up in their learning journey, they are the reason why Rust is so safe and lightweight.

Not only Ownership allows Rust to operate without tools like Garbage Collectors, but it also forces developers to think deeper into how memory is allocated, used and released in their applications. Languages like C and C++ offer this level of control too, but without the safeguards that Rust's compiler enforces.

While it seems like the Rust compiler throws too many errors, it is for our own good. It's better to catch these problems during compilation than to try to debug them on execution time, many times in production environments.

Source code gen in Flutter & Dart: How to create a builder to generate SQL code

Pedro F Marquez — Wed, 21 Sep 2022 06:42:18 +0000

In the first part of this series of posts, we discussed how Flutter and -Dart in general- makes use of builders to generate source code during the build process.

We saw the example of json-serializable, a builder that "decorates" Dart classes with methods to serialize and deserialize its contents to -and from- JSON.

In this post, we will describe how to create a custom builder that will generate both SQL and Dart code out of a Dart class. This builder will follow a pattern similar to json-serializable: We will define a set of annotations that we will use to annotate classes for which we want to generate SQL code.

Some inspiration

You may be familiar with the Hibernate ORM, an implementation of the Java Persistence API (JPA) specification.

In JPA, we can annotate Java classes and their attributes with @Table, @Entity, @Id and @Column -among others- to create a mapping between a SQL table and a Java class.

For instance, if we wanted to map a class Employee to a SQL table called EMPLOYEE, we would annotate the class as follows:

import javax.persistence.*;

@Entity
@Table(name = "EMPLOYEE")
public class Employee {
   @Id @GeneratedValue
   @Column(name = "id")
   private int id;

   @Column(name = "first_name")
   private String firstName;

   @Column(name = "last_name")
   private String lastName;
   //...

This mapping lets the application generate SQL code to create, query and update the SQL table.

Drawing inspiration from JPA, we will create a builder to generate the SQL for creating a SQL table out of annotated Dart classes.

A note about database and generators: The example in this post is only for educative purposes. If you want something more production-ready that performs the kind of SQL manipulation our example generator does, take a look at Dirft or Floot.

Plan

From a high level, here is what we will do:

Define a project structure
Define a set of annotations
Create methods to build SQL
Define the builder
Using and executing the builder

Step 1: Define our project structure

For reasons that we will define in a couple of sections, we will create two different projects: db_builder and db_builder_generator.

Other builders follow a similar pattern:

injectable, which is split between the packages injectable and injectable_generator.
json-serializable, which is split between json_annotation and json_serializable.

The two Dart projects will contain the following structure:

db_builder
├── build.yaml
├── lib
│   ├── annotations.dart
│   └── ...
└── pubspec.yaml

db_builder_generator
├── build.yaml
├── lib
│   └── builder
│       ├── builder.dart
│       ├── builders
│           └── ...
├── pubspec.yaml

Step 2: Define a set of annotations

In Dart, there is no special syntax to create annotations. A regular Dart class can be used as an annotation.

Inside the db_builder project, we will create a lib/annotations.dart file and we will start defining classes for our annotations:

// Will be used as: @DatabaseTable("table_name")
class DatabaseTable {
  final String name;
  const DatabaseTable(this.name);
}

// Will be used as: @DatabaseColumn("col_name", "TEXT", isRequired: true)
class DatabaseColumn {
  final String name;
  final String type;
  final bool isRequired;
  final bool isNotNull;
  final bool isUnique;
  final String? defaultValue;

  const DatabaseColumn(this.name, this.type,
    {this.isRequired = false,
    this.isNotNull = false,
    this.isUnique = false,
    this.defaultValue});
}

These annotations will be added to classes used to hold database record data:

@DatabaseTable("test_table_2")
class TestEntity2 {
  int id;
  String name;
  @DatabaseColumn("file_path", "TEXT")
  String filePath;

  TestEntity2(this.id, this.name, {this.filePath = ""});
}

The annotations as used in the previous code snippet work as regular class constructors:

DatabaseTable("test_table_2")
DatabaseColumn("file_path", "TEXT")

Step 3: Create methods to build SQL

The next step is to create methods that will allow us to serialize the contents of our annotations into SQL statements. We will generate table definition statements like the following:

CREATE TABLE test_table_2 (file_path TEXT)

There are many ways to approach the serialization process, but the simplest one is to override the annotation's toString method:

@JsonSerializable()
class DatabaseColumn {
  // ...

  @override
  String toString() {
    var params = ['$name ${type}'];
    if (isRequired) {
      params.add('REQUIRED');
    }
    if (isNotNull) {
      params.add('NOT NULL');
    }
    if (isUnique) {
      params.add("UNIQUE");
    }
    if (defaultValue != null) {
      params.add("DEFAULT '$defaultValue'");
    }

    return params.join(" ");
  }

  // ...

Step 4: Define the builder

Now we get to the core of the post. To define a builder, we must create a build.yaml file inside the directory db_builder_generator -which we defined in previous steps- and create the Dart source code for the builder.

Our build process will consist of two sub-steps:

Create intermediate files .part.sql for each Dart file that contains annotated classes.
Collet all .part.sql and merge them into a single /sql/create.sql file.

Defining the code for the builder

A simple way to create the code for the builder is to extend the class Builder from the package build/src/builder/builder.dart.

Note: Since we are generating SQL files first, extending Builder is a good choice. However, once we start generating Dart source code, a better choice will be to extend GeneratorForAnnotation<T> from the sourcegen/sourcegen.dart package. We will explore Dart-code generation in the next post of this series

class SQLBuilder extends Builder {
  final BuilderOptions options;

   SQLBuilder({
    required this.options,
  })

  // ...
}

Notice that the builder can receive options to customize the building process.

Now, we must override two key methods:

@override
FutureOr<void> build(BuildStep buildStep)

@override
Map<String, List<String>> get buildExtensions => const { //... }

The buildExtensions getter returns a map of the file extensions that will be used for the generated files:

@override
Map<String, List<String>> get buildExtensions => const {
  r'.dart': ['.part.sql'],
};

This mapping describes that we will take as input .dart files, and we will generate files with the extension .part.sql.

The build method defines the actual process of creating the SQL files. First, we need to collect all Dart classes that contain the @DatabaseTable annotation:

TypeChecker typeChecker = TypeChecker.fromRuntime(DatabaseTable);

Future<Iterable<AnnotatedElement>> getClassesAnnotatedWithTable(
      BuildStep buildStep) async {
    final lib = await buildStep.resolver
        .libraryFor(buildStep.inputId, allowSyntaxErrors: true);
    final libraryReader = LibraryReader(lib);
    return libraryReader.annotatedWith(typeChecker);
  }

We use the utility class LibraryReader from source_gen/src/library.dart to gain access to the collection of files in the project.

Then, we use the method annotatedWith to retrieve all the AnnotatedElement classes (e.g. TestEntity2 for our example) that contain the annotation type defined by TypeChecker.fromRuntime(DatabaseTable).

Then, we iterate through the list of AnnotatedElements, and we extract the attributes for each class:

var annotated = await getClassesAnnotatedWithTable(buildStep);

var allFields = annotated.map((field) {
  if (field.element is! ClassElement) {
    return null;
  }
  ClassElement element = field.element as ClassElement;
  var fieldList = getAllFields(element);
  // ...
});

Since our classes could be extending other classes, and getting extra column attributes through inheritance, we must search attributes at two levels:

Attributes in the annotated class (through SimpleElementVisitor).
Attributes in the parent class(es) (through a InheritanceManager3).

import 'package:analyzer/src/dart/element/inheritance_manager3.dart' // ignore: implementation_imports
    show
        InheritanceManager3;

List<FieldElement> getAllFields(ClassElement element) {
    var manager = InheritanceManager3();
    var visitor = ModelVisitor();

    final inheritedFields = <String, FieldElement>{};
    for (var v in manager.getInheritedConcreteMap2(element).values) {
      if (v is PropertyAccessorElement && v.isGetter) {
        assert(v.variable is FieldElement);
        final variable = v.variable as FieldElement;
        assert(!inheritedFields.containsKey(variable.name));
        inheritedFields[variable.name] = variable;
      }
    }
    element.visitChildren(visitor);
    return [...visitor.elements, ...inheritedFields.values];
  }

Getting inherited attributes

We use the method manager.getInheritedConcreteMap2(element) in the class InheritanceManager3 to retrieve the list of classes in the inheritance chain for the annotated class.

We must iterate through each inherited attribute, filtering out attributes that are not property getters:

if (v is PropertyAccessorElement && v.isGetter) {
  assert(v.variable is FieldElement);
  final variable = v.variable as FieldElement;
  inheritedFields[variable.name] = variable;
}

Getting the annotated class' attributes

We create a ModelVisitor class extending SimpleElementVisitor from the package package:analyzer/dart/element/visitor.dart to extract the attributes for the annotated class:

var visitor = ModelVisitor();
element.visitChildren(visitor);

// ...

class ModelVisitor extends SimpleElementVisitor {
  late DartType className;
  Map<String, DartType> fields = {};
  Map<String, List<ElementAnnotation>> metaData = {};
  List<FieldElement> elements = [];

  @override
  visitConstructorElement(ConstructorElement element) {
    className = element.type.returnType;
  }

  @override
  visitFieldElement(FieldElement element) {
    fields[element.name] = element.type;
    metaData[element.name] = element.metadata;
    elements.add(element);
  }

  @override
  visitTopLevelVariableElement(TopLevelVariableElement element) {
    fields[element.name] = element.type;
    metaData[element.name] = element.metadata;
  }

  // ...
}

Having collected all attributes in the class -both direct and inherited-, we must filter out those that are not annotated with @DatabaseColumn:

List<DatabaseColumn> getDatabaseColumns(List<FieldElement> fieldList) {
    return fieldList
        .where((element) => columnTypeChecker.hasAnnotationOf(element))
        .map((element) => columnTypeChecker.firstAnnotationOfExact(element))
        .map((element) => DatabaseColumn.fromDartObj(element!))
        .toList();
  }

At the same time, we create a method DatabaseColumn.fromDartObj to parse the contents of the annotation instance:

factory DatabaseColumn.fromDartObj(DartObject obj) {
    var reader = ConstantReader(obj);

    var all = DatabaseColumn("", "")
        .toJson()
        .keys
        .map((key) => Tuple2(key, reader.read(key).literalValue));

    return DatabaseColumn.fromJson(
        Map.fromIterable(all, key: (e) => e.item1, value: (e) => e.item2));
}

Instead of writing code to pass each attribute by name in DatabaseColumn, we use json_serializable to convert the annotation attributes to JSON and iterate through each of its attributes. In this way, as we add more attributes to DatabaseColumn, we don't need to update the builder to reflect those changes.

This is the reason why we created two packages db_builder and db_builder_generator: json_serializable is a builder by itself. When I tried to keep both the annotations and the builder code in the same package, the build process would result in conflicts between json_serializable and db_builder; the builder would delete the partial files for the JSON serialization code, causing compilation problems in the build process for db_builder.

Of course, you could use the classes in the analyzer package to extract the attribute classes, and that may allow you to keep the annotations and the builder in the same package. However, this example shows that keeping two separate packages gives you more flexibility to run other builders in your annotation classes.

Having retrieved, filtered and converted each annotated attribute into instances of DatabaseColumn, all we have left to do is serialize the contents of each annotation into a SQL CREATE statement:

var allFields = annotated.map((field) {
  if (field.element is! ClassElement) {
    return null;
  }
  ClassElement element = field.element as ClassElement;
  var fieldList = getAllFields(element);
  var fields = getDatabaseColumns(fieldList);
  return buildCreateStatement(field, fields);
}).where((element) => element != null);


// ...
final db = sql.sqlite3.openInMemory();

String buildCreateStatement(
      AnnotatedElement field, List<DatabaseColumn> fields) {
  var query =
      "CREATE TABLE ${field.annotation.peek("name")?.objectValue.toStringValue()} (${fields.join(",\n")})";

  try {
    db.prepare(query);
  } on SqliteException catch (ex, stack) {
    log.warning("The generated SQL has errors: $query", [ex, stack]);
  }
  return query;
}

Notice an extra thing here: Since I want to generate valid SQL statements, I'm using the sqlite3 to create an in-memory database and try to dry-run the statement with db.prepare(query).

This extra level of validation may not be necessary if you're completely sure the SQL code you're generating is valid, but I think it's nice to get that extra sanity check, especially if you plan to create more complex SQL code like views or JOIN queries.

Once we have a list of SQL statements, we write them to their corresponding files using buildStep.writeAsString:

String _generatedExtension = '.part.sql'

AssetId createAssetId(BuildStep buildStep) {
  var filePAth = buildStep.inputId.path;
  var newfile = path.join(path.dirname(filePAth),
      path.basenameWithoutExtension(filePAth) + _generatedExtension);
  return AssetId(buildStep.inputId.package, newfile);
}

// ...

buildStep.writeAsString(
        createAssetId(buildStep), allFields.join("\n"))

Notice that we get a list of CREATE statements for a single file, as Dart files can contain multiple class definitions. If your code is organized to have only one class definition per file, this list will always contain one single CREATE statement.

The writeAsString method requires us to pass an AssetId, which is the complete definition for the file to be generated. In this case, we create a file with the same path as the original Dart file, but using the extension .part.sql.

Now, putting it all together, we complete our implementation of the build method:

@override
FutureOr<void> build(BuildStep buildStep) async {
  log.fine("Build SQL");

  try {
    var annotated = await getClassesAnnotatedWithTable(buildStep);

    var allFields = annotated.map((field) {
      if (field.element is! ClassElement) {
        return null;
      }
      ClassElement element = field.element as ClassElement;
      var fieldList = getAllFields(element);
      var fields = getDatabaseColumns(fieldList);
      return buildCreateStatement(field, fields);
    }).where((element) => element != null);

    if (allFields.isEmpty) {
      return;
    }

    await buildStep.writeAsString(
        createAssetId(buildStep), allFields.join("\n"));
  } catch (err, stack) {
    log.severe("Failed to generate SQL", err, stack);
  }
}

To create a cleaner API, we can create a single db_builder_generator/builder.dart file that exports the builders you want to use:

import 'builders/SqlBuilder.dart';

Builder sqlBuilder(BuilderOptions options) => SQLBuilder(options: options);

Defining the builder in build.yaml

Builders are defined by the builders attribute in build.yaml:

builders:
  sql_builder:
    target: ':sql_builder'
    import: 'package:db_builder_generator/builder/builder.dart'
    builder_factories: ['sqlBuilder']
    build_extensions: { '.dart': ['.part.sql'] }
    auto_apply: dependents
    build_to: cache

We can find all the attributes for build.yaml in the documentation for build_config. The following are extracts of the attributes we're defining above:

import: Required. The import uri that should be used to import the library containing the Builder class. This should always be a package: uri.
builder_factories: A List which contains the names of the top-level methods in the imported library which are a function fitting the typedef Builder factoryName(BuilderOptions options).
build_extensions: Required. A map from input extension to the list of output extensions that may be created for that input. This must match the merged buildExtensions maps from each Builder in builder_factories.
auto_apply: Optional. The packages which should have this builder automatically applied. Defaults to 'none' The possibilities are:
- "none": Never apply this Builder unless it is manually configured
- "dependents": Apply this Builder to the package with a direct dependency on the package exposing the builder.
build_to: Optional. The location that generated assets should be output to. The possibilities are:
- "source": Outputs go to the source tree next to their primary inputs.
- "cache": Outputs go to a hidden build cache and won't be published. The default is "cache". If a Builder specifies that it outputs to "source" it will never run on any package other than the root.

So, at a high level, we're passing the location of our builder, the map of output extensions, we only apply it to packages that depend on our builder, and we build to the cache.

The cache's location can be found at the project's root in .dart_tool/build/generated/.

We chose to build to cache because we don't want to generate a SQL file for each file containing annotated classes. We want to create one single /lib/sql/create.sql file containing all create statements in our code base. To get to this result, we must merge all generated files.

Second builder: "combineSql"

Builders can depend on the results of other builders. We will use this fact to create a second builder that will take all .part.sql files and merge them into a single file:

class SqlCombinerBuilder implements Builder {
  @override
  final buildExtensions = const {
    r'$lib$': ['sql/create.sql']
  };

  @override
  Future<void> build(BuildStep buildStep) async {
    log.fine("Merge all");
    try {
      final exports = buildStep.findAssets(Glob('**/*.part.sql'));
      final content = [
        await for (var exportLibrary in exports)
          '-- from: \'${exportLibrary.changeExtension('.dart').uri}\' \n'
              '${await buildStep.readAsString(exportLibrary)};',
      ];
      if (content.isNotEmpty) {
        buildStep.writeAsString(
            AssetId(buildStep.inputId.package, 'lib/sql/create.sql'),
            content.join('\n'));
      }
    } catch (err, stack) {
      log.severe("Failed to merge generated SQL", err, stack);
    }
  }
}

This second builder follows the same pattern as the first one: Extend Builder and override the build and buildExtensions method/attribute.

However, in this builder, we collect all files that have the extension .part.sql as follows:

final exports = buildStep.findAssets(Glob('**/*.part.sql'));

Then, we merge them all together in a single string:

final content = [
  await for (var exportLibrary in exports)
    '-- from: \'${exportLibrary.changeExtension('.dart').uri}\' \n'
        '${await buildStep.readAsString(exportLibrary)};',
];

The -- characters indicate a SQL comment, which is useful to know which Dart file was used to generate the current CREATE statement.

Finally, if the result string is not empty, we write it to lib/sql/create.sql:

buildStep.writeAsString(
  AssetId(buildStep.inputId.package, 'lib/sql/create.sql'),
  content.join('\n'));

Finally, we expose this builder in builder.dart:

Builder combineSql(BuilderOptions options) => SqlCombinerBuilder();

build.yaml definition for "combineSql"

Just as we did with the first builder, we must define an entry in the build section of build.yaml:

builders:
  sql_builder:
    # ...
  combine_sql:
    target: ':sql_builder'
    import: 'package:db_builder_generator/builder/builder.dart'
    builder_factories: ['combineSql']
    build_extensions: { $lib$': ['sql/create.sql'] }
    auto_apply: dependents
    build_to: source
    required_inputs: ['.part.sql']
    applies_builders: ['db_builder_generator:sql_builder']

We define two extra attributes in this configuration:

required_inputs: Optional, list of extensions, defaults to empty list. If a Builder must see every input with one or more file extensions they can be specified here and it will be guaranteed to run after any Builder which might produce an output of that type.
applies_builders: Optional, list of Builder keys. Specifies that other builders should be run on any target which will run this Builder.

Deleting intermediate files

Since we don't need all the .part.sql files anymore, we can clean them up to avoid taking up disk space unnecessarily.

We can use FileDeletingBuilder from the builder package to clean up files as a post-process builder. We expose the builder in builders.dart:

PostProcessBuilder partCleanup(BuilderOptions) =>
    FileDeletingBuilder(BuilderOptions);

And we create one last configuration in build.yaml, but this time in the post_process_builders attribute, instead of the builders section:

builders:
  sql_builder:
    # ...
  combine_sql:
    # ...
post_process_builders:
  part_cleanup:
    import: 'package:db_builder_generator/builder/builder.dart'
    builder_factory: 'partCleanup'

Step 5: Using and executing the builder

Having imported our packages into a project -e.g. mypackage--, we can configure them in the dependent project's own mypackage/build.yaml:

targets:
  $default:
    builders:
      json_serializable:
        # ...
      db_builder_generator|sql_builder:
        enabled: true
        generate_for:
          include:
            - lib/models/**.dart

This configuration will scope the code generation process to only find annotated classes in the mypackage/lib/models/**.dart path. In that directory, we have a single file TestTable2.dart with the example we described before:

@DatabaseTable("test_table_2")
class TestEntity2 {
  int id;
  String name;
  @DatabaseColumn("file_path", "TEXT")
  String filePath;

  TestEntity2(this.id, this.name, {this.filePath = ""});
}

Finally, we execute all builders with the following command:

dart run build_runner watch --delete-conflicting-outputs

If everything runs without errors, you should now find a file in the dependent project's path mypackage/lib/sql called mypackage/lib/sql/create.sql:

-- from: 'package:mypackage/models/TestTable2.dart'
CREATE TABLE test_table_2 (file_path TEXT)

We have successfully generated SQL code out of annotated classes! The dart run build_runner watch command will spawn a process that will watch for changes to files in the source code, and regenerate the SQL files appropriately.

Next steps

In the next post of this series, we will explore how to apply these same concepts to generate Dart source code. While the basic elements are the same, packages like source_gen provide extra utilities that make the process smoother.

Conclusion

Creating builders involves low-level manipulation of Dart classes and type definitions. However, developers who are familiar with reflection concepts in other languages will find that packages like analyzer follow similar patterns.

Source code generation is a powerful tool to generate code that would be time-consuming to write manually and/or requires some level of introspection into existing source code. This fits the definition of "boilerplate" code.

I find that builders are especially useful in cases where we need to write repetitive code for each attribute in multiple class definitions: Object serialization and deserialization, attribute decoration, translating data in objects to specific formats, and in general, any use case that would be simplified if we could just iterate through a list of classes or attributes.

Source code generation is not exclusive to Dart. Java relies heavily on the same patterns for libraries like Project Lombok, which reduces the amount of boilerplate code (getters, setters, and so on) developers need to write, and increases developer productivity.

Converting and customizing Markdown files to HTML with Unified, remark & rehype

Pedro F Marquez — Mon, 12 Sep 2022 01:47:25 +0000

Hosting your own blog is the kind of thing that every software developer wants to do. It's usually cheaper, you have control over your data, and you have the option to monetize it.

Whether you're a web developer or not, writing HTML can be cumbersome if you want to post fairly often. Most of the time you will want to focus on the content of your post, not on the code quality of the web page you're building to host it.

Here, Markdown is a great tool to speed up your blog posting capabilities.

Markdown

Markdown defines a set of rules that indicate what kind of formatting text should have. The following are some of the most basic rules:

Element	Markdown Syntax	Example
Heading	# H1 ## H2 ### H3	H1 H2 H3
Bold	bold text	bold text
Italic	italicized text	italicized text
Blockquote	> blockquote	(Adds a quote container)
Ordered List	1. First item 2. Second item 3. Third item	First item Second item Third item
Unordered List	- First item - Second item - Third item	First item Second item Third item
Code	`code\`	code
Horizontal Rule	---
Link	[title](https://www.example.com)	title
Image	![alt text](image.jpg)	(displays image.jpg)

Markdown makes it easy to write formatted text. For instance, the following snippet of Markdown:

# Welcome to my blog!

Start with the [first article](/blog/2022/9/markdown-to-html)

Gets formatted like follows:

Welcome to my blog!

Start with the first article

Many tools exist to build blogs from Markdown. Jekyll is a popular choice to build static websites out of Markdown.

However, when you are writing a technical blog, you will start noticing the limitations of working with out-of-the-box solutions. The point I hit these limits was with code formatting.

Limitation: Code formatting

Markdown has two ways of highlighting code. The block element:

    ```
    public class App {
      public static void main(String[] args) { ... }
    }
    ```

And the inline element

`System.out.println("Hello")`

PrismJS and Highlight.js are two JavaScript libraries to highlight code keywords for multiple programming languages.

With PrismJS, setting a language for formatting is as easy as introducing the library (along with the CSS for a given theme) and then adding the language name to the code block declaration. For the example above, we get the following result:

    ```java
    public class App {
      public static void main(String[] args) { ... }
    }
    ```

Is rendered like:

public class App {
  public static void main(String[] args) { ... }
}

But, what happens with the inline code? Markdown rules won't allow us to do something like

`System.out.println("Hello")`java

If we wanted to add custom CSS classes to Markdown-generated components, we find that Markdown doesn't specify the syntax to do so. Tools like Jekyll do allow for use of HTML elements inside Markdown, but it starts defeating a bit the purpose of using Markdown to speed-up writing.

If we want to extend Markdown syntax, we must modify the way Markdown is converted to HTML.

Getting our hands dirty: Introducing Unified

Unified is a framework to process Markdown. It's a plugin-based tool that allows you to inspect and modify the way Markdown is converted to other formats like HTML.

Its general structure looks like this:

import { unified } from 'unified'

const result = await unified()
  .use(...) // add a plug-in
  .process(await read('./example.md'))

Unified relies on two toolsets remark and rehype:

remark: "remark is a tool that transforms markdown with plugins. These plugins can inspect and change your markup. You can use remark on the server, the client, CLIs, deno, etc."
rehype: "rehype is a tool that transforms HTML with plugins. These plugins can inspect and change the HTML. You can use rehype on the server, the client, CLIs, deno, etc."

With a combination of plugins from these two toolsets, we can fully customize the way we parse Markdown into HTML.

Some common plugins are:

remark-parse: remark plugin to add support for parsing markdown input.
remark-rehype: remark plugin that turns markdown into HTML to support rehype.
rehype-document: rehype plugin to wrap a fragment in a document.
rehype-format: rehype plugin to format (pretty-print) an HTML document.
rehype-stringify: rehype plugin to add support for serializing HTML (to string).

Putting all these plugins together, the example looks like this:

import { unified } from 'unified'
import remarkParse from 'remark-parse'
import remarkRehype from 'remark-rehype'
import rehypeDocument from 'rehype-document'
import rehypeFormat from 'rehype-format'
import rehypeStringify from 'rehype-stringify'
import { read } from 'to-vfile'

const result = await unified()
  .use(remarkParse)
  .use(remarkRehype)
  .use(rehypeDocument)
  .use(rehypeFormat)
  .use(rehypeStringify)
  .process(await read('./test/example.md'))

Where result is a VFile -a virtual representation of the HTML file- that can be easily saved as an HTML file.

Analyzing the transformation process

The following image represents the process the Markdown file follows to become HTML:

The following is a step-by-step description of the process:

The Markdown file contents are read and placed into a VFile.
remark parses and process the Markdown text, and converts it into an abstract syntax tree (AST). This tree is composed of unist nodes, which follow the Markdown Abstract Syntax Tree (mdast) structure.
remark-rehype translates the mdast structure into a Hypertext Abstract Syntax Tree (hast) structure, also composed of unist nodes.
rehype process and transforms the hast structure into HTML.

Unified allows us to build plugins and process both the mdast and hast structures.

mdast and hast

Before we try to customize the transformation process, let us look at how mdast and hast are represented.

In mdast, each node represents an abstract element. For instance, the paragraph or p element is represented as follows:

{
  "type": "paragraph",
  "children": [{ "type": "text", "value": "Alpha bravo charlie." }]
}

That same node has the following representation in hast as follows:

{
  "type": "element",
  "tagName": "p",
  "properties": {},
  "children": [{ "type": "text", "value": "Alpha bravo charlie." }]
}

Notice that mdast has a different level of abstraction than hast: hast represents HTML elements, where "paragraph" is defined with the tag name p.

We can look at the same difference for code. The following markdown...

    ```
    foo()
    ```

...becomes the following mdast:

{
  "type": "code",
  "lang": null,
  "meta": null,
  "value": "foo()"
}

...and the following hast:

{
  "type": "element",
  "tagName": "pre",
  "properties": {},
  "children": [
    {
      "type": "element",
      "tagName": "code",
      "properties": {},
      "children": [
        {
          "type": "text",
          "value": "foo()\n"
        }
      ]
    }
  ]
}

And the inline version bar()\ becomes the mdast:

{
  "type": "inlineCode",
  "value": "bar()"
}

...and the hast:

{
  "type": "element",
  "tagName": "code",
  "properties": {},
  "children": [
    {
      "type": "text",
      "value": "bar()"
    }
  ]
}

Notice that the block version of code wraps the <code/> HTML element in <pre/>, to preserve alignment and whitespaces. The inline version, however, just creates a <code/> element.

The block version allows us to add a value to the lang property. The following Mardown:

    ```js
    foo();
    ```

...gets parsed as follows:

{
  "type": "code",
  "lang": "js",
  "meta": null,
  "value": "foo();"
}

and:

{
  "type": "element",
  "tagName": "pre",
  "properties": {},
  "children": [
    {
      "type": "element",
      "tagName": "code",
      "properties": {
        "className": ["language-js"]
      },
      "children": [
        {
          "type": "text",
          "value": "foo();\n"
        }
      ]
    }
  ]
}

The Markdown parser follows the HTML5 specification for <code/> to define the language used in that block: Using the lang attribute with a value of language-[LANG].

As we can see, we can use CSS to find <code/> blocks with class names for a given language (e.g. language-js) and process them accordingly.

In fact, PrismJS looks for <code class="language-[LANG]></code/> blocks to tokenize the text content and highlight keywords.

While the HTML5 specification allows for the use of lang in <code/> elements not wrapped in <pre/> elements, there is no defined way in the Markdown syntax to set that attribute.

Adding "lang" to inline code

We will extend the Rehype process to interpret inline code blocks like follows:

`highlighted^const inline = "code";`

We will split the contents of the code block in two: Text before a separator ^ and the text after that:

const [part1, part2] = ['markdown', 'const inline = "code";']

Then, we will use the first part as the code's class name, and the second part as its contents.

The following code snippet is an extract of the Rehype plugin rehype-inline-code-classname I published to NPM to allow setting a class name to inline code blocks:

import { Element, Root, Text } from 'hast'
import { Transformer } from 'unified'
import { visit } from 'unist-util-visit'

export function rehypeInlineCodeClassNamePlugin(
  options?: Options
): void | Transformer<Root, Root> {
  const separator = options?.separator ?? '^'
  const trailing = options?.trailing ?? false

  return function (root) {
    function extract(value: String) {
      const [_, part1, part2] =
        value.match(new RegExp(`(.+)\\${separator}(.+)`)) ?? []

      return trailing
        ? { className: part2, content: part1 }
        : { className: part1, content: part2 }
    }

    visit(
      root,
      'element',
      function visitor(node: Element, i: number | null, parent: any) {
        if (node.tagName !== 'code') return
        if (!parent || parent.tagName === 'pre') return

        const [{ value, ...rest }] = node.children as Text[]
        if (value.includes(separator) && value.length > 2) {
          const { className, content } = extract(value)
          node.children = [{ value: content, ...rest }] as Text[]
          node.properties = {
            className: className,
          }
        }
      }
    )

    return root
  }
}
export type Options =
  | {
      separator: string
      trailing: boolean
    }
  | void
  | undefined

Working with AST: visit

In this plugin, we use the function visit from unist-util-visit to navigate the hast tree and find all nodes:

visit(root, 'element',
      function visitor(node: Element, i: number | null, parent: any) {
        //..
      }

Notice that visit receives three parameters:

The root node of the hast.
The type of node we want to visit. In our example, it will iterate through all nodes of type element.
A function that will receive references to the node, and index i and the node's parent.

From here, we can modify the hast in any way we need to: Remove, add or update nodes.

Processing all "code" elements

We filter out all the nodes that don't have a tagName of code, and all tagName:code nodes that have a tagName:pre node as a parent, as those are block-type code elements:

if (node.tagName !== 'code') return
if (!parent || parent.tagName === 'pre') return

We get the node's value attribute and check if it contains the expected separator ^ (and the code element actually has enough characters for having left and right substrings):

const [{ value, ...rest }] = node.children as Text[]
if (value.includes(separator) && value.length > 2) {
  //...
}

If the value can be parsed with the separator, then we extract both substrings using a RegExp:

const [_, part1, part2] =
  value.match(new RegExp(`(.+)\\${separator}(.+)`)) ?? []

The plugin allows a configuration object to define a custom separator and whether we want to set the class name at the end of the block like follows:

`const inline = "code";^highlighted`

Depending on the configuration, we set className and content with part1 and part2; then we update the hast node as follows:

const { className, content } = extract(value)
node.children = [{ value: content, ...rest }] as Text[]
node.properties = {
  className: className,
}

We reset the node's children attribute with the actual content of the code block, and we set the className property to the substring we extracted using the separator.

Using the plugin

To include our plugin, we just need to add it to the chain of plugins after remark-rehype, as we're working with a hast:

import { myRehypePlugin } from '../src/plugin/index.ts'

const result = await unified()
  .use(remarkParse)
  .use(remarkRehype)
  .use(myRehypePlugin) // our rehype plugin
  .use(rehypeDocument)
  .use(rehypeFormat)
  .use(rehypeStringify)
  .process(await read('./test/example.md'))

We now can set any class name we need for inline code blocks:

`language-java^foo()`

will become:

<code class="markdown">^</code>

Revisiting code formatting

Since we now can set class names to our inline code blocks, we can use CSS and JavaScript to further format and style these elements:

code[class*='language-'] {
  background-color: #f2f5ff;
  padding: 0.1em;
  border-radius: 0.3em;
  white-space: normal;
}

Unfortunately, the changes we have done are not enough to operate with plugins like rehype-prism, as those assume <code/> blocks are wrapped by <pre/> elements. To highlight keywords in our code, we must use a tool to tokenize and highlight keywords, which is exactly what PrismJS does.

In a future blog entry, we will extend this example to use refractor to create a custom rehype plugin which extends rehype-prism's capabilities, and allows us to highlight keywords in inline code blocks.

Conclusion

Unified offers an ecosystem of plugins that allow us to extend and customize the way Markdown is converted to HTML.

Creating a rehype or remark plugin requires us to handle abstract syntax trees, but unist tools like unist-util-visit offer helper functions that reduce the level of complexity of these tasks.

Frameworks like NextJS integrate nicely with Unified/remark plugins, allowing to convert Markdown files to HTML which can be easily rendered in the application. They even offer integration with MDX, a superset of Markdown that allows the use of React components inside Markdown files.

However, Unified can be used to generate static HTML pages directly, allowing us to keep updated our self-hosted blogs.

On hosting

Free hosting for self-managed blogs exist out there. A good example is Github Pages, which integrates pretty nicely with tools like Jekyll. But we can just use HTML files created through Unified.

New book: "Backend Developer in 30 Days"

Pedro F Marquez — Sun, 28 Aug 2022 18:42:00 +0000

"Backend Developer in 30 Days" is now available on Amazon and bpbonline.com!!

In this book, I tried to capture what I would have loved to know a decade ago when I was a junior developer: A holistic view of the most critical areas of back-end development, from defining and building APIs to continuously releasing an application.

I wrote this book in the middle of a pandemic and a master's degree; so I'm delighted to release this book after all that effort.

A big thank you to BPB for their patience, trust, and support while I wrote this book.

The following is the book description:

This book connects ideas on API design, data modeling, practical testing, application security, and system design, among other critical areas of backend development.

Build stronger support system to power your enterprise applications

Key Features

Figure out the most important elements of backend and application development.
Know how to construct large-scale, distributed applications using industry best practices and software architecture principles.
Provides a career map for becoming a successful backend developer, including advice on skills and tools.

Description

In today's world, becoming an experienced backend developer is a difficult job that requires a lot of work. There are an excessive number of stacks and technologies to master, and new ones often gain popularity. Nonetheless, they share the same fundamental concepts: data storage, security, performance, testing, etc.

This book aims to teach and train you to become a successful backend developer with a solid skill set in developing and implementing the real engine of a successful enterprise application.

Implementation topics like setting up a web server, designing and developing APIs, creating and running automated tests, and working with various types of databases are all addressed in detail.

The book prepares developers to maintain the safety and security of their applications through the use of procedures that avoid application and data breaches. While you learn about every part of backend development, the book teaches you how to deal with errors and find and fix problems.

By the end of the book, you will have a firm grasp on the fundamental principles that underpin backend programming, including application architectures, design patterns, essential development activities, and help for debugging.

What you will learn

Gain knowledge to build the momentum of a solid backend developer and choose areas to investigate in depth.
Learn to integrate backend development as the top tech businesses do.
Comprehend the distinction between SQL and NoSQL, containers, APIs, and web servers.
Design large-scale systems step-by-step.
Grow from junior backend developer to senior developer, including the required skills and responsibilities.

Who this book is for

This book would greatly benefit readers who are new to backend operations for web and mobile apps, such as junior software developers, web developers, application developers, and frontend and backend coders. Basic programming skills will help you practice this book's learnings.

Building Multi-User Apps
The Client-Server Architecture
Designing APIs
End-to-end Data Management
Automating Application Testing
Securing Applications
Handling Errors
Adopting Frameworks
Deploying Applications
Creating High-performance Apps
Designing a System
Bootstrap Your Career Path

If you have any feedback about the book, don't hesitate to send me an email to pedro.f.marquez.soto@gmail.com

Note: If you're buying from Mexico, it's easier to order the book from BP online (the link is below), as they don't charge for delivery.

Source code generation in Flutter & Dart (Part 1): Reflection and code gen

Pedro F Marquez — Sun, 28 Aug 2022 18:15:30 +0000

If you have worked on Flutter projects (or you're a Java developer), you may be familiar with the Mockito library.

Mockito allows developers to create mock objects from existing classes, stub their methods and verify or assert their behavior during tests.

If you have used Mockito for Flutter or Dart projects, you will find the following in their documentation:

To use Mockito's generated mock classes, add a build_runner dependency in your package's pubspec.yaml file, under dev_dependencies; something like build_runner: ^1.11.0.

Some of the code examples will show how to mock an existing class like Cat as follows:

@GenerateMocks([ Cat ])

Then it expects you to run the following command:

flutter pub run build_runner build

The build_runner build command creates a file called *.mocks.dart. What exactly is happening here?

Reflection (and the lack of it)

Flutter's FAQ contains the following question:

***Does Flutter come with a reflection / mirrors system?*** No. Dart includes dart:mirrors, which provides type reflection. But since Flutter apps are pre-compiled for production, and binary size is always a concern with mobile apps, this library is unavailable for Flutter apps. Using static analysis we can strip out anything that isn’t used (“tree shaking”)... This guarantee is only secure if Dart can identify the code path at compile time. To date, we’ve found other approaches for specific needs that offer a better trade-off, such as code generation.

According to Wikipedia, Reflection is:

In computer science, reflective programming or reflection is the ability of a process to examine, introspect, and modify its own structure and behavior

In other languages like Java, libraries like Mockito use reflection to get information at run-time about the classes they are modifying. For instance, a Java function can get all the available methods for a class with the following code:

Class c = Class.forName("java.lang.String");
Method m[] = c.getDeclaredMethods();

In this example, the function gets an array containing all the functions that are part of the java.lang.String class.

We can see how this could be practical for libraries like Mockito: It can find all the methods of a given class and provide stub implementations based on their return and parameter types. All in runtime.

A common approach is to mark classes with annotations, then find those classes through reflection and do something with them:

Create mock implementations.
Instantiate singletons for the class.
Create and inject instances of other classes into the constructor (like the Spring framework does).
Decorate classes to provide behavior like JSON serialization or String representations, and so on.

A good example on how reflection is leveraged is the Java snippet we included in the previous post "Easy parallelism and multi-threading with Java's CompletableFuture" (updated to remove generic classes for clarity):

private static <Book> List<Book> parseJSON(String textResponse) {
    final ObjectMapper objectMapper = new ObjectMapper();
    List<T> objects = new ArrayList<>();
    try {
      objects =
        objectMapper.readValue(textResponse, new TypeReference<List<Book>>() {});
    } catch (JsonMappingException e) {
      // TODO: Do something with the error
    } catch (JsonProcessingException e) {
      // TODO: Do something with the error
      e.printStackTrace();
    }
    return objects;
  }

In this code example, the Jackson mapper uses reflection to find out the type of class it should use to de-serialize the JSON string:

List<T> objects = objectMapper.readValue(textResponse, new TypeReference<List<Book>>() {});

The parameter new TypeReference<List<Book>>() {} creates a class reference that lets Jackson'sObjectMapperknow that it should parse the JSON and create an instance ofList<Book>.

However, we cannot do this kind of JSON de-serialization in Flutter. Without reflection, we need to consider other approaches for these same tasks. For instance, the package json_serializable uses source code generation for decorating classes with methods to serialize and de-serialize JSON strings.

Source code generation

When we don't have access to metadata about classes in run-time, one alternative is to do this introspection at compile time.

While Flutter limits reflection at run-time, tools like source_get rely on two low-level Dart packages:

build: "Defines the basic pieces of how a build happens and how they interact."
analyzer: "This package provides a library that performs static analysis of Dart code. It is useful for tool integration and embedding."

Build provides an interface to implement builders: classes that provide an entry point to the compilation process, allowing us to create files as a result of our source code.

The analyzer package provides tools to retrieve information about our classes: The methods and variables they contain and the annotations they use, among others.

The tool build_runner (which is mentioned in Mockito's documentation) uses these two low-level packages to generate source code.

High-level process: json_serializable

At a high level, the json_serializable package performs the following steps:

Have developers annotate classes with @JsonSerializable. The annotated class must conform to a set of guidelines for the process to work correctly.
Have developers run the build_runner tool to generate the code that allows these classes to serialize and de-serialize JSON strings.

At a lower level, what the source code generation process is doing is the following:

Developers execute the build_runner tool and, in turn, build_runner executes json_serializable builder classes.
- The builder uses the build to define what files it will read from the source code, and what files it will generate.
- The builder uses analyzer to find all classes annotated with @JsonSerializable.
- The builder uses analyzer to find all the attributes and constructors for the serializable classes.
- The builder creates Dart classes for serializing and deserializing JSON strings into the attributes it found with analyzer.
- The builder cleans and puts these new classes in the same package as the annotated classes as .g.dart files.
- These .g.dart files are parts, which allow you to split a class into multiple files.

Following the example from json_serializable's docs, we annotate the following class contained in example.dart:

@JsonSerializable()
class Person {
  /// The generated code assumes these values exist in JSON.
  final String firstName, lastName;

  /// The generated code below handles if the corresponding JSON value doesn't
  /// exist or is empty.
  final DateTime? dateOfBirth;

  Person({required this.firstName, required this.lastName, this.dateOfBirth});

  /// Connect the generated [_$PersonFromJson] function to the `fromJson`
  /// factory.
  factory Person.fromJson(Map<String, dynamic> json) => _$PersonFromJson(json);

  /// Connect the generated [_$PersonToJson] function to the `toJson` method.
  Map<String, dynamic> toJson() => _$PersonToJson(this);
}

After running build_runner, the following file example.g.dart is created:

part of 'example.dart';

Person _$PersonFromJson(Map<String, dynamic> json) => Person(
      firstName: json['firstName'] as String,
      lastName: json['lastName'] as String,
      dateOfBirth: json['dateOfBirth'] == null
          ? null
          : DateTime.parse(json['dateOfBirth'] as String),
    );

Map<String, dynamic> _$PersonToJson(Person instance) => <String, dynamic>{
      'firstName': instance.firstName,
      'lastName': instance.lastName,
      'dateOfBirth': instance.dateOfBirth?.toIso8601String(),
    };

All we have to do is import example.g.dart into example.dart and we will have access to the private functions _$PersonFromJson and _$PersonToJson.

Advantages and disadvantages

The lack of reflection forces developers to be explicit about the types and use cases they need to support. However, we can still rely on generics to create reusable code.

Generating source code reduces the number of manual work developers need to do for repetitive tasks. Without source_gen and json_serializable, we would have to manually create toJson and fromJson functions listing every attribute for each class. There are alternatives for this too, using generics and inheritence. However, it still requires a good amount of manual work.

While generating source code reduces this manual work, it introduces a layer of complexity to our project: Now we have a big chunk of source code that will not be available for inspection until we run the build tool. We can always check-in the generated code into version control, helpings better track changes done by re-running the build tool.

But the main disadvantage is the added complexity. Source code generators depend on the code having very specific conditions (e.g. having public constructors, follow name conventions) which may cause the build process to fail when not met. It's easy to find ourselves spending hours of our day just to figure out that the builder wasn't configured correctly.

What's next

In the next blog post, we will create a simple plugin to build source code for Flutter projects. We will start deep-diving into the build process and how to create reusable modules that can be imported by other projects, just like json_serializable or mockito do.

Easy parallelism and multi-threading with Java’s CompletableFuture

Pedro F Marquez — Wed, 17 Aug 2022 20:53:08 +0000

Modern applications (especially those based on microservices architectures) heavily rely on fetching data from
remote servers. However, if not done correctly, the overhead added by these requests can seriously
reduce your application's performance.

In this blog post, we will explore ways to parallelize these requests using
Java's CompletableFuture.

Background

Long lost are the old days when Java developers had to use the Thread or Runnable classes
for simple tasks like making requests to remote servers.

In Java 8, the Concurrency API was updated to include CompletableFuture, a class that does a great
job abstracting out the complexity of running tasks in multiple Threads.
This interface, in addition to lambda functions,
makes up for a very intuitive way of running tasks in parallel in separate threads.

For instance, imagine we want to make two requests to a REST API hosted on a remote server:
One to fetch a list of books and one to fetch a list of authors:

.../api/authors
# Returns:
[
  {
    "firstName": "Pedro",
    "lastName": "Marquez-Soto"
  }
]
.../api/books
# Returns:
[
  {
    "type": "Book",
    "name": "Amazing book"
  }
]

Unfortunately, these endpoints are extremely slow: The books endpoint takes 2.5 seconds to complete
and the authors endpoint takes 1.5 seconds.

Making HTTP requests in Java

A simple way to make HTTP requests to remote servers is through the java.net.http.HttpClient class:

The following code snippet displays an example of making a GET request to the address stored in the string variable url:

var client = HttpClient.newHttpClient();

// create a request
var request = HttpRequest
    .newBuilder(URI.create(url))
    .header("accept", "application/json")
    .build();

// use the client to send the request
final var response = client.send(request, BodyHandlers.ofString());

The variable response will contain the string representation of the server response.

Now, we can parse the JSON string into a class object using Jackson's ObjectMapper as follows:

private static <T> List<T> parseJSON(String textResponse) {
    final ObjectMapper objectMapper = new ObjectMapper();
    List<T> objects = new ArrayList<>();
    try {
      objects =
        objectMapper.readValue(textResponse, new TypeReference<List<T>>() {});
    } catch (JsonMappingException e) {
      // TODO: Do something with the error
    } catch (JsonProcessingException e) {
      // TODO: Do something with the error
      e.printStackTrace();
    }
    return objects;
  }

Notice that this function is using a generic type, so we can reuse this function to parse JSON strings
into many different classes:

List<Book> books = App.<Book>parseJSON();
List<Author> books = App.<Author>parseJSON();

The classes Book and Author are plain Java objects with empty constructors, getters, and setters:

public class Author {

  private String firstName;
  private String lastName;

  public Author() {}

  public String getFirstName() {
    return firstName;
  }
  // the rest of getters and setters
}

Putting it all together, we can create a function to request the data from each endpoint:

private static <T> List<T> fetchSync(String url) {
    var client = HttpClient.newHttpClient();

    var request = HttpRequest
        .newBuilder(URI.create(url))
        .header("accept", "application/json")
        .build();

    // you may need to handle the checked exception in "send"
    final var response = client.send(request, BodyHandlers.ofString());
    return App.<T>parseJSON(response.body());
}

And we can use this function to make the two requests we need. Also, we will measure the time
it takes for both requests to complete:

long start = System.nanoTime();
// Fetch the data:
List<Book> bookList = fetchSync("https://[your-api-here]/api/books");
List<Book> authorList = fetchSync("https://[your-api-here]/api/authors");

System.out.println(
    "Sync calls processed in " +
    Duration.ofNanos(System.nanoTime() - start).toSeconds() +
    " sec\n\n"
    );

If we execute this code (and the REST APIs are also up and running), we would see a message like follows:

Sync calls processed in 4 sec

The time both requests took to complete is consistent with the knowledge we have about the endpoints' performance:

[Books_API_request] = 2.5 sec
[Authors_API_request] = 1.5 sec
[Books_API_request] + [Authors_API_request] = 4 sec

Here, we are making sequential and blocking, requests. The second request for
authorList will not start until the first request for bookList completes. However, both requests are not dependent on each other:
We could do both simultaneously and still receive the same results.

Multi-threading with CompletableFuture

The java.net.http.HttpClient has a non-blocking version of its send method: sendAsync.

public abstract <T> CompletableFuture<HttpResponse<T>> sendAsync(
  HttpRequest request,
  HttpResponse.BodyHandler<T> responseBodyHandler)

Instead of returning a string response directly, this method returns a CompletableFuture.
The CompletableFuture class is a
wrapper for the result of an operation that may or may not have finished yet.

When using the synchronous method HttpResponse.send, once execution reaches client.send(...), the main application thread will pause and block every other action until the server returns a response (or the request fails). However, HttpResponse.sendAsync will not block execution; it will immediately return the CompletableFuture instance and continue execution to the next line.

The request may or may not have been completed, soCompletableFuture could be in a loading state. If we want to do something with the result once the request completes, we can register a listener function to the Future with Future.thenApply:

CompletableFuture<String> response = client
  .sendAsync(request, BodyHandlers.ofString())
  .thenApply((httpResponse) -> httpResponse.body());

// of using the static method representation:

CompletableFuture<String> response = client
  .sendAsync(request, BodyHandlers.ofString())
  .thenApply(HttpResponse::body);

The method thenApply works similarly to Java Stream's map: Chain a function that will be executed with the result of the previous operation as a parameter; that function then transforms the data and returns something else (in our example, the function transforms the HttpResponse to a String).

While this may be confusing, let us continue with our example to make things more transparent.

The following function is an asynchronous version of the fetchSync function we wrote before:

private static <T> CompletableFuture<List<T>> fetchAsync(String url) {
    // create a client
    var client = HttpClient.newHttpClient();

    // create a request
    var request = HttpRequest
      .newBuilder(URI.create(url))
      .header("accept", "application/json")
      .build();

    // use the client to send the request
    final var response = client
      .sendAsync(request, BodyHandlers.ofString())
      .thenApply(HttpResponse::body)
      .thenApply(App::<T>parseJSON);

    // the response:
    return response;
  }

This function then can be called as follows:

CompletableFuture<List<Book>> booksFuture = fetchAsync(
  "https://[your-api-here]/api/books"
);
CompletableFuture<List<Author>> authorsFuture = fetchAsync(
  "https://[your-api-here]/api/authors"
);

Here, Java automatically detects the types of the booksFuture and authorsFuture variables and passes the correct class type to the generic function.

If we were to execute these functions as they are, we would see no results at all. This is because the application won't wait for the server responses to complete. To block execution and wait for a CompletableFuture to finish, we can use the following code:

CompletableFuture<List<Book>> booksFuture = fetchAsync(
  "https://[your-api-here]/api/books"
);
CompletableFuture<List<Author>> authorsFuture = fetchAsync(
  "https://[your-api-here]/api/authors"
);
booksFuture.get();
authorsFuture.get();

We now measure the time this code snippet takes. The following is the measurement result:

Async calls processed in 2 sec

Both requests took 2 seconds. That is half the time it took to make both requests with blocking requests!

Let us convert the time to milliseconds to better assess what is happening here:

# Making just the book request:
Calls processed in 2503 sec
# Making just the author request:
Calls processed in 1505 sec

# Both as synchronous calls:
Calls processed in 4343 sec

# Both as asynchronous calls:
Calls processed in 2504 sec

When we run both requests in parallel, we can see that the total time it takes to complete both tasks is equal to the time it takes to complete the longest of them (in this case, the books request, which takes 2.5 seconds).

This is a significant improvement over the synchronous/blocking calls, whose total time equals the sum of the time it takes to make each request one after the other.

Long operations

Java's CompletableFuture is not just a good way to handle network requests. Any function that may take a long time is a good candidate for being handled through Futures: Reading and writing files, and memory-intensive operations.

For instance, we can simulate a long-running operation with the following function:

static CompletableFuture<String> longRunningOperation() {
    return CompletableFuture.supplyAsync(
      () -> {
        sleep(5000);
        return "Success";
      }
    );
  }

  static void sleep(long milliseconds) {
    try {
      Thread.sleep(milliseconds);
    } catch (InterruptedException e) {
      // TODO Auto-generated catch block
      e.printStackTrace();
    }
  }

  // ....

  var futureResult = longRunningOperation();
  System.out.println("Getting result of long running operation:");
  System.out.println(futureResult.get());

  /*
  > "Getting result of long running operation:"
  > "Success"
  */

The method CompletableFuture.supplyAsync gets a thread from ForkJoinPool#commonPool(), and executes the lambda function in that thread.

Conclusion

CompletableFuture is a robust tool that should be in any Java developer's toolset.

It provides a native way to schedule work in parallel threads, which is particularly useful for making network requests to remote servers. Its API provides many practical methods to orchestrate asynchronous calls better, giving developers control to improve their application's performance.

Other languages have interfaces similar to CompletableFuture. For instance, JavaScript has Promises, which is a concept almost identical to Java's Futures.

If we don't need to share resources between threads, Java Futures and the Concurrency API can be a simpler alternative to classic classes and interfaces like Thread and Runnable. However, For more low-level tasks requiring synchronization and memory handling across all threads, Thread is still a powerful tool.

Building a dark-mode theme with CSS variables

Pedro F Marquez — Fri, 05 Aug 2022 15:50:00 +0000

In this post, we will explore CSS variables and the prefers-color-scheme media query, and we will build a small example of how to implement dark-mode in a website without using any external libraries and with little to no use of JavaScript.

Overview

In recent years, “dark mode” has become widely popular. Ever since the beginning of the Internet,
web developers have looked for ways to customize their users' experiences with different color palettes;
most of the time leading to custom solutions that relied heavily on JavaScript and working around CSS’s rules.

Modern CSS, however, now includes features that allow developers to override easily override styles
across a website or web application.

CSS Variables

With many years in the making, browser support for CSS variables expanded around 2016.
Now, newer versions of most modern browsers support them.

It’s important to notice that these variables are supported by vanilla CSS; there is no need to
use a pre-processor like Sass or Less.

As their name indicates, CSS variables allow us to store values once and use them across our stylesheet.

The following example creates two variables in a body block and one in a h1: body-color,
font-color and h1-color.

body {
  --body-color: black;
  --font-color: white;
  background-color: var(--body-color);
  color: var(--font-color);
}

h1 {
  --h1-color: gold;
  color: var(--h1-color);
}

Notice that variables are prefixed with --. That is how CSS recognizes that they are variables.

Then, within those same blocks, we apply CSS rules like background-color and color, and we get
the variables’ values by passing them to the function var().

While this example may seem over-simplistic, you can see the potential: We can take those variables
and put them in a global scope:

:root {
  --body-color: black;
  --font-color: white;
  --h1-color: gold;
}

body {
  background-color: var(--body-color);
  color: var(--font-color);
}

h1 {
  color: var(--h1-color);
}

Notice that we moved the variable definitions to a block named :root. This pseudo-class matches the <html> tag.
This means that, from here, all CSS rules on the page will have access to these variables.

Some benefits

CSS variables allow us to centralize and standardize values for CSS rules across our web page.
You can encapsulate specific values in reusable variables.

Most websites use a palette of 3 or 4 colors: primary, secondary, and accent colors.
We can create a variable for each, and refer to those variables directly in every other CSS rule.
If you want to change the palette, just change the variable definitions; no need to update each rule in
every stylesheet.

Overriding CSS variables.

CSS variables follow the same rules as the rest of CSS; thus, variables can be overridden:

<h2>This is gold</h2>
<h2 class="page1">This is green</h2>

:root {
  --h1-color: gold;
}

h2.page1 {
  --h1-color: green;
}

h2 {
  color: var(--h1-color);
}

Since we can override CSS rules by adding specificity to the selector, we can also override the
variable definitions in :root:

:root {
  --body-color: white;
  --font-color: black;
  --h1-color: gold;
}

// we only override the variables we need:
[data-dark-mode] {
  --body-color: black;
  --font-color: white;
}

Now the :root block contains the variable definitions for “light-mode”, and the
block [data-dark-mode] overrides the body-color and font-color variables.

This means that, if the element <html> has the attribute data-dark-mode, the page
will have a black background and white font, instead of the default white background and black font:

<!-- Light mode: -->
<html>
  <!-- ... -->

  <!-- Dark mode: -->
  <html data-dark-mode></html>
</html>

The attribute data-dark-mode is no reserved word. This attribute can be
anything like data-dark-theme or data-cool-dark-mode-on.

Just remember to prefix the attribute with data-, as this is a custom attribute.

Switching between dark and light modes

If we want to allow users to switch between light and dark modes, we will need a couple of
JavaScript lines. First, create a button:

<button id="dark-mode-toggle">Toggle dark mode</button>

Then, add an event listener for the “click” event of that button. Inside that event listener,
get a reference to the html element, and use the toggleAttribute method:

document
  .querySelector("#dark-mode-toggle")
  .addEventListener("click", function () {
    document.querySelector("html").toggleAttribute("data-dark-mode");
  });

And that’s it! When the user clicks the button, the attribute data-dark-mode will be toggled
on and off the <html> element, overriding the styles.

The “prefers-color-scheme” media query

Users now can configure their operative systems (Android, iOS, MacOS, and so on) to use
dark-mode everywhere, when possible. Browsers can detect this setting through media queries,
specifically, the prefers-color-scheme media query:

@media (prefers-color-scheme: light) {
  // detect OS-configured light-mode and apply rules
}

@media (prefers-color-scheme: dark) {
  // detect OS-configured dark-mode and apply rules
}

Using these media queries, we can set the page to always be in dark-mode if the user configured
this option in their device:

:root {
  --body-color: white;
  --font-color: black;
  --h1-color: gold;
}

[data-dark-mode] {
  --body-color: black;
  --font-color: white;
}

@media (prefers-color-scheme: dark) {
  :root {
    --body-color: black;
    --font-color: white;
  }
}

Optionally, we can remove the dark-mode toggle button if the option is already set at the OS level:

// ...
@media (prefers-color-scheme: dark) {
  :root {
    --body-color: black;
    --font-color: white;
  }
  #dark-mode-toggle {
    display: none;
  }
}

The best part of this media query is that it removes the need of using JavaScript to apply dark-mode to the
page. But, as always, make sure that the browsers your expect to target support the media query.

Codepen

Conclusion

CSS variables are extremely useful to standardize our stylesheets and make them configurable.

In this post we implemented dark-mode in a very simple web page, but the benefits of CSS variables and
the prefers-color-scheme media query are more apparent once we apply them in a real application with tons
of CSS rules.

Originally posted to https://pedromarquez.dev/blog/2022/7/dark-mode-css

Managing state in Flutter apps with data streaming

Pedro F Marquez — Wed, 20 Jul 2022 00:17:00 +0000

Managing state in a client app is always a tricky (and opinionated) topic.
Web developers are familiar with tools like Redux or MobX. These tools follow very specific patterns that are not always easy to learn or understand by new developers.

These same tools exist in the Flutter world, but recently I've been exploring an alternative that may seem consistently easier. We can manage app state with rxdart and get_it.

Some background

At any given moment, an application will have state. If we think of an app like Instagram, the state can be
the list of posts and stories the application renders. State can also be your app preferences, the contents of the
story you're about to post, the list of notifications and the number of un-read messages.

In general, state is data.

Let's think of notifications. When a user sends you a message, many components in the application need to reflect this change of state: The little red dot that tells you have un-red messages must be rendered, the list of messages must be updated, and so on.

At a lower level, state is a collection of variables. In Flutter, we extend the StatefulWidget class when we want to create a widget that will contain state:

import 'package:flutter/material.dart';

class Update {
  Update({required this.description});

  String description;
}

class MyStatefulWidget extends StatefulWidget {
  @override
  State<StatefulWidget> createState() => MyStatefulWidgetState();
}

class MyStatefulWidgetState extends State<MyStatefulWidget> {
  List<Update> updates = [];

  @override
  Widget build(BuildContext context) {
    print("Updates: $updates");
    return Column(
      children: [
        ElevatedButton(
          onPressed: () {
            fetchUpdates().then((newUpdates) {
              setState(() {
                updates = newUpdates;
              });
            });
          },
          child: const Text("Fetch notifications"),
        ),
        Flexible(
          child: ListView.builder(
            itemCount: updates.length,
            itemBuilder: (BuildContext context, int index) {
              return ListTile(
                title: Text(updates[index].description),
              );
            },
          ),
        )
      ],
    );
  }
}

The MyStatefulWidgetState widget renders the following view:

In this example, when the user clicks the Fetch notifications button, the function fetchUpdates() is called. This function returns a Future object, and once the request completes successfully, we set the list of updates in the variable updates.

The variable updates is state within this widget.

Calling setState when we set the new value of the variable updates will re-render this widget only. If this widget contains other widgets (e.g. the ElevatedButton and the ListView), those widgets will be re-rendered as well.

Now, we want to add an icon to the top bar to indicate the user that new notifications have been fetched. The following widget will render that notification icon:

import 'package:flutter/material.dart';

class NotificationIcon extends StatelessWidget {
  @override
  Widget build(BuildContext context) {
    var notificationsCount = 0;
    return Stack(
        children: <Widget>[
        const Icon(Icons.notifications),
        Positioned(
            right: 0,
            child: Container(
            padding: EdgeInsets.all(1),
            decoration: BoxDecoration(
                color: Colors.red,
                borderRadius: BorderRadius.circular(6),
            ),
            constraints: const BoxConstraints(
                minWidth: 12,
                minHeight: 12,
            ),
            child: Text(
                '$notificationsCount',
                style: const TextStyle(
                color: Colors.white,
                fontSize: 8,
                ),
                textAlign: TextAlign.center,
            ),
            ),
        )
        ],
    );
  }
}

How can we notify the NotificationIcon widget that MyStatefulWidget has fetched a new list of notifications?

Some options...

We could re-render our component to include NotificationIcon in MyStatefulWidget. The problem with this approach is that we may need to also include other, unrelated widgets.

Another approach is to use Inherited Widgets. However, this approach may require us to modify other widgets than these two, which is not always possible.

How can we notify to independent widgets of state changes?

Enter state management.

State management through streams

Data streams are analog to queues: One part of the application will publish data to a stream and other parts of the application can subscribe that stream and consume the data it produces.

We can rely on streams to communicate state changes across independent components.

A popular implementation of streams for Flutter apps is rxdart.

We can add them to our project with the following command:

flutter pub add rxdart

Creating a stream

Then, we can create a service to create a stream:

import 'package:rxdart/rxdart.dart';

import 'MyStatefulWidget.dart';

class NotificationsService {
  final BehaviorSubject<List<Update>> notificationSubject = BehaviorSubject.seeded([]);

  ValueStream get stream$ => notificationSubject.stream;

  updateNotifications(List<Update> updates) {
    notificationSubject.add(updates);
  }
}

Before we can use this service, we need a way for widgets to access a shared instance of the stream. For this, we will use get_it:

flutter pub add get_it

Registering a service

This tools is a service locator: It allows us to create singleton instances of objects and share it across the application without having to pass them as paramaters.

We can create an instance of NotificationService when the app starts as follows:

void main() {
  GetIt getIt = GetIt.I;
  getIt.registerSingleton<NotificationsService>(NotificationsService());
  runApp(const MyApp());
}

Now, anywhere in our app, we can get this singleton instance with the following code:

GetIt getIt = GetIt.I;
var service = getIt.get<NotificationsService>();

We update MyStatefulWidgetState to get the service, and send the new updates to the stream:

fetchUpdates().then((newUpdates) {
    GetIt getIt = GetIt.I;
    var service = getIt.get<NotificationsService>();
    service.updateNotifications(newUpdates);
    ...

Subscribing to the stream

Now, we will update NotificationIcon to subscribe to the stream and listen for new updates.

For this, we will use StreamBuilder, which accepts a stream and re-renders its contents when the stream produces new results.

The following code shows the updated version of NotificationIcon:

import 'package:flutter/material.dart';
import 'package:get_it/get_it.dart';

import 'MyStatefulWidget.dart';
import 'NotificationsService.dart';

class NotificationIcon extends StatelessWidget {
  late NotificationsService service;

  NotificationIcon() {
    GetIt getIt = GetIt.I;
    service = getIt.get<NotificationsService>();
  }

  @override
  Widget build(BuildContext context) {
    return StreamBuilder(
      stream: service.stream$,
      builder: (BuildContext context, AsyncSnapshot<dynamic> snapshot) {
        if (!snapshot.hasData || snapshot.data == null) {
          return const CircularProgressIndicator();
        }

        List<Update> updates = snapshot.data;

        if (updates.isEmpty) {
          return Container();
        }
        var notificationsCount = updates.length;

        return Stack(
          children: <Widget>[
            const Icon(Icons.notifications),
            Positioned(
              right: 0,
              child: Container(
                padding: EdgeInsets.all(1),
                decoration: BoxDecoration(
                  color: Colors.red,
                  borderRadius: BorderRadius.circular(6),
                ),
                constraints: const BoxConstraints(
                  minWidth: 12,
                  minHeight: 12,
                ),
                child: Text(
                  '$notificationsCount',
                  style: const TextStyle(
                    color: Colors.white,
                    fontSize: 8,
                  ),
                  textAlign: TextAlign.center,
                ),
              ),
            )
          ],
        );
      },
    );
  }
}

Focus on the important part here:

 return StreamBuilder(
      stream: service.stream$,
      builder: (BuildContext context, AsyncSnapshot<dynamic> snapshot) {
        if (!snapshot.hasData || snapshot.data == null) {
          return const CircularProgressIndicator();
        }

        List<Update> updates = snapshot.data;

        if (updates.isEmpty) {
          return Container();
        }
        var notificationsCount = updates.length;
        // ... render the notification icon

Since the stream may be empty, we need to handle the case where the widget has not received any updates. The class AsyncSnapshot has multiple methods that allow us to check the stream state and act accordingly.

Now, when the user clicks in the button, the new updates will be sent to the stream and all widgets listening to it will re-render:

We could take this example even further and make MyStatefulWidgetState a stateless widget, getting its state directly from the same stream. Feel free to take this extra step and practice this state management approach.

Pros and cons

Some of the advantages of this state management approach are:

It is simple. Multiple, independent widgets can be notified of state changes without changing their parent widgets, and without passing any extra props.
It allows us to use more stateless widgets. Stateless widgets have a more predictable behavior, and require less boilerplate code.
It reduces unnecessary re-rendering. If we were to pass app state as properties through the widget tree, all intermediate widgets will be re-rendered when state changes. With this approach, only the widgets that need to re-render do so.
Less conventions. There is no defined API to follow. No reducers or mappers. While conventions are useful for large projects, smaller apps can benefit of the flexibility this approach provides.

On the other side, the disadvantages are:

It may be too flexible. This approach opens the door to a lack of structure in the project. More structure is better for large teams and large code bases.
If not used responsibly, get_it can lead to a bunch of global state. Developers can abuse the magic of this service locator and add state that doesn't need to be stored.

Conclusion

State management is hard. Because of these, developers have created many tools to store state and communicate state changes. Some of those tools will fit better to your specific project needs.

Using streams along with a service locator is a simple yet effective way to handle global state.

(originally posted in my personal blog)