Forem: Zean Qin

Extract handwritten data from paper forms using Azure Form Recognizer with supervised learning

Zean Qin — Mon, 26 Oct 2020 11:05:07 +0000

Background

At the shift changeover in most hospitals, a paper form is completed to summarise the last shift and highlight key aspects of the upcoming shift. As an example, a completed form could look like below.

These completed forms are gathered together at a later point, and a person will go through each and manually enter the data on a computer. The recorded data can then be fed into an analytical system to generate insights.

The goal of the project

The main goal of this project is to streamline the data collection process by building a tool that,

the users can use to take a picture of the completed forms using their smart phones,
the tool then automatically extracts the data and pre-fills the form, and
the user can review and save the data.

The user can choose to either take the picture and upload it immediately after completing the form, or take pictures of multiple completed forms at a later point.

A few other features include,

providing a digital version of the form that the user can fill in directly on their phone/tablet,
saving the original image along with the extracted data,
linking data e.g. Unit, Users etc. to existing entities in the system.

Demo

At Riberry, we build a platform for collecting data and generating insights within hospital environments, as well as delivering clinical improvement programs. This article only focuses on extracting data from paper forms using OCR - a feature on the data collection side of things.

And below is a demo of how data is extracted from a paper form and entered into the system.

Architecture overview

We're using the Form Recognizer service from Microsoft to extract data from the forms. More specifically, we've trained a custom model, instead of using the pre-built models, to better handle the structure of the forms that our clients use.

At the very high level, we will train a model first. The users can then upload photos of the filled forms via the frontend (a.k.a Riberry) to the API called Rumble. Rumble then uses the trained model to extract data from the form and transforms and cleans up the extracted data before sending it back to the user. The image below has a more detailed explanation of this process.

Train the model

Note: the focus of this article is to showcase the final product and to reflect on the lessons learnt in adding the OCR support, rather than a step-by-step guide on how to use the Azure Form Recognizer service. There is no point in writing a guide either; you can't really beat their documentation and examples.

If you're looking at learning how to use the Form Recognizer service, I highly recommend reading through their documentation - it's really the most effective way.

At the core of the Form Recognizer service is a set of REST APIs that allow you to train a model by using supervised/unsupervised machine learning, manage (e.g. list, delete, copy) models, and extract data using custom/pre-built models.

In addition, it also offers an open source tool (OCR-Form-Tools) that interacts with these REST APIs and provides an intuitive UI to label your data, train a model and validate the model by predicting a test document.

Tip: In their guide on setting up the sample labelling tool, they talk about running the tool via Docker.

Instead of using Docker, I find it's much easer to just check out a copy of the tool from GitHub and build/run it from source.

After having the tool up and running, I was able to label our sample data, train a model and test the model with ease.

Label the training data

Train a model

Validate the trained model

Miscellaneous Reflections

Now that the project has been deployed to production, it's a good time to look it retrospectively and reflect on various things/lessons that I learnt from doing it.

1. The training process

There should be 3 main stages in predicting the contents of a form using the supervised learning approach,

detect the text elements (printed or handwritten) on the form and calculate their expected sizes and positions,
extract the actual text content from each element,
classify the elements, using their size and position info obtained from step 1, into the different labels specified.

Tip: With the supervised learning, we're fitting a model for classifying elements on the page into different labels based on the position info of these elements on the page.

And we're NOT training any NLP models to extract texts from the elements identified on the page. This part is handled internally by the Form Recognizer service.

The position and size of each text element is expressed by its bounding box which consists of 8 numbers. And these 8 numbers should be the main features for each training instance i.e. every text element.

There are also likely a lot of data argumentation and feature manipulation techniques applied on the training data to expand the dataset artificially, in order to improve the accuracy of the models. For example, they'll likely use some label-preserving transformation techniques such as denoising/sharpening, elastic distortion, affine transformation, dimensionality reduction etc.

But overall, the cost function should be relatively simple with just a few variables. And I'm not surprised that only 5 documents are required to get started with.

2. Adding support for checkbox/radio question

The Form Recognizer service doesn't officially support checkboxes or radio buttons, but I've been able to get around it easily by,

applying a label for each option in a checkbox or radio question, and
at prediction time, treating those labels with a value present as the options that the user has selected.

As an example, the form that the hospitals used contains a checkbox question with two options - AM or PM. During the training process, I'd set up the labels _Shift.9:00 and _Shift.17:00 for the AM and the PM option respectively (see below).

Hypothetically, if we fed the form above to the trained model, a string 'x' will be extracted for both _Shift.9:00 and _Shift.17:00. I'd consider both options checked if this was a checkbox question, and I'd consider the first option is checked if this was a radio question.

3. Extracting the data asynchronously and notifying the users when the result is ready

After POSTing a form to their endpoint to extract data from it, you get back an Operation-Location, which is a URL containing the resultId used to track the progress and obtain the result of the analyse operation. And we're meant to repeatedly check this URL for the result data.

Tip: I set the frequency of checking for the result to once per second. This seems to be the same frequency that their OCR tool uses for checking for the result.

Tip: If your users have to wait for the result before taking the next action, compress the image before POSTing it for analysis to shorten wait time.

The images taken on smartphones are fairly large these days, and I compress them down to <1MB before sending them for analysis.

If your users don't have to wait for the analyse result, I'd highly recommend uploading the files into a blob storage and running a background task to analyse these files asynchronously. Users can be notified after the result is ready, if needed.

We couldn't do it because one of the requirements is that users need to take a photo of the form, extract data from it in real-time and save it to generate insights. In practice, the process usually takes around 5 seconds.

4. Don't over-automate the data binding process

4.1 Background

The fields on the paper form don't match up exactly with those on the digital version - some data on the paper form is ignored while there are a couple of extra fields on the web version. In addition, while the paper form rarely changes, we keep updating the digital form slightly either by adding/removing questions or moving around the existing questions.

4.2 How we designed it

The questions and their layouts on the web form are defined using a JSON file. Without getting into the details, in the screenshot below, the JSON structure (right) defines the Team section (left) of the form.

However, we also use the names of the questions from the JSON file to automatically construct the label (used in OCR) for that question. For example, in the screenshot below, the label - Team..NursesShort - for the "Nurses Short" question is created by joining the names of all its ancestors and itself with a dot.

4.3 Problem with the design

This means that whenever we change the digital version of the form (e.g. adding/removing questions or re-arranging the fields on the form), a new set of labels will be generated. And we'd have to re-name the labels used in Form Recognizer, re-train the model, copy it to production etc. This is a really time consuming process.

4.4 What we should have done instead

We should have de-coupled the label generation from the JSON definition of a form.

Given that the paper form rarely changes, we should have just hardcoded a set of labels to be used in the Form Recognizer service. We could then expand the JSON definition to include a label property that specifies the label used for each question, if applicable.

This not only simplifies the code for binding the data (i.e. converting the extracted data into domain objects), but also means that we can freely re-arrange the questions on the form without having to re-train the model in Form Recognizer.

References

Feedbacks

That's it! Let me know in the comment section below if you have any feedbacks.

Typography basics and best practices for software developers

Zean Qin — Mon, 21 Sep 2020 11:57:12 +0000

Typography is about shaping and laying out the texts on your website to create a pleasant user experience. Web design is 95% typography. And it's essential for website developers to know the basics and some common practices.

There's a lot more jargon and subjectivity in typography than in other areas. This article is intended as a concise and practical guide for software developers on choosing/using custom fonts and laying out the texts nicely and comfortably. It covers the basics to get you most of the way there.

Basics

Let's start with some basic terms and the common categories of fonts.

1. Typeface vs. Font

"Typeface" and "Font" mean different things,

typeface is the design (e.g. shape) of a collection of letters, numbers and symbols (also called glyphs), whereas
font is a specific size, weight, or style (e.g. regular, bold, italic) of a typeface.

Most of the time people use them interchangeably. But hey, we're trying to look professional here ;-).

2. Common Font Categories

Typefaces come in all different shapes, and there is no single classification system. Below are a few of the most commonly seen categories.

2.1 Serif

A “serif” is a small stroke attached to the ends of letters, giving them a traditional feel. The "serif" category includes a few sub-categories such as Old Style, Classical, Neo-Classical, Transitional, Clarendon, etc. These typefaces are mostly used in books and newspapers.

Below is what the "Caslon" font (in the Old Style subcategory) looks like. Notice the small feet present at the tops and bottoms of each letter.

You can see the top 10 most popular serif fonts here.

2.2 Sans Serif

"Sans" comes from the French "without" and as the name states - sans-serif typefaces don’t have these extra strokes, giving them a smooth, modern feel. It includes humanist, geometric and grotesk etc. as sub-categories.

Below is the font called "Futura" in the Sans Serif category. Again, notice the clean and straight lines, compared to the serif fonts.

You can see the top 10 most popular sans serif fonts here.

2.3 Monospace

Monospaced typefaces are non-proportional - every glyph takes up the exact same space. It's possible to arrange them into visual structures. And your code editor probably uses a monospace font.

Below is the "Courier" font which is a monospace font.

You can see the top 10 most popular monospace fonts here.

3. "Web safe" Fonts vs Custom Fonts

A font is "web safe" if most computers have it installed already, and these computers don't have to download it separately when visiting your site. Examples include Arial, Times New Roman, Courier, Georgia, Verdana and more.

A website can declare a custom font as a resource, just like CSS, images or JavaScripts. And the visiting browser will download the font and apply it to the texts on the website.

Using fonts

You can apply a font by using the "font-family" attribute with the following syntax in CSS.

body {
  font-family: 'custom-font', fallback1, fallback2;
}

Where is the custom-font coming from? Well, it could be from either your OS if you have it installed (e.g. Segoe UI on Windows, or Roboto on Android), or a third party such as Google Fonts, Adobe Fonts etc. In the later case, you most likely need to tell the browser to download it by including the <link> tag in the head of your page, like below.

<html>
  <head>
    <link href="https://fonts.googleapis.com/css?family=Roboto" rel="stylesheet" />
  </head>
</html>

Flash of Unstyled Text

In the case that a custom font used by a website is not available on the visiting computer, it needs to be transferred across the wire. This takes time! The browser will use a fallback font in the font stack until the custom font is loaded. This can cause the flash of unstyled text (FOUT).

As an example, suppose you had the font stack below,

body {
  font-family: Merriweather, Georgia;
}

and the font "Merriweather" needs to be downloaded by the browser, you could have the texts to flash like below after the page is loaded.

Some common approaches to mitigate FOUT include,

using a third party (e.g. Google Fonts, Adobe Fonts etc.) to optimise the font files and deliver them via a CDN,
transforming your font files using tools like Transfonter,
picking a fallback font that's similar to the custom font you want to use with tools like Font style matcher.

Native font stack

A “native font stack” allows for optimum text rendering on every device and OS with zero latency. For example, Bootstrap 4 uses the following native font stack by default,

$font-family-sans-serif:
  // Safari for OS X and iOS (San Francisco)
  -apple-system,
  // Chrome < 56 for OS X (San Francisco)
  BlinkMacSystemFont,
  // Windows
  "Segoe UI",
  // Android
  "Roboto",
  // Basic web fallback
  "Helvetica Neue", Arial, sans-serif,
  // Emoji fonts
  "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Symbol" !default;

And you can find more system font stacks here.

Best Practices

I've listed some of the practices that I follow in choosing font size, heading size, line height, letter spacing and how many characters should be included in one line.

1. Font sizing

Sizing in CSS can be specified in either absolute units (think px) or relative units (e.g. em). The size of an element specified in absolute units doesn't change, while the size of an element specified in relative units depends on the size of other elements on the page.

You should avoid using absolute units like px, and instead use relative units like em as much as possible.

The purpose of the px unit should be to serve as the foundation of a type system based on relative units. In other words, it’s an absolute value that a relative unit can point to in order to define its own size relative to that value.

Fun fact, the px unit doesn't actually have anything to do with screen pixels - it's just a poorly chosen name. It's actually an non-linear angular measurement. And this is why you can actually specify pixels in decimals such as 12.4px.

As an example, most desktop browsers usually sets the texts inside the body tag to 16px by default. You can use 2em on an element if you want it to be twice as big as the body text.

When setting the size of some texts, try to think in relative units - "I want element A to be twice as big as element B". Instead of thinking "I want element A to be 20px".

2. Heading sizing

The size of headings are usually expressed in relative units. And it's common to define 6 levels of headings.

Bootstrap 4 defines the size of headings as below,

Another popular approach is to define a modular scale to define the heading sizes. Basically, it means specifying a root number and a ratio, and ratios are multiplied by the base to produce a scale of numbers that is proportionally related. For example, the following scale uses base 1em and the ratio 1.5. You can use this tool to create your custom scale.

3. Line height

Line height can be specified in CSS using the line-height property like below,

p {
  line-height: 1.5; # Note - better to not use any unit here.
}

When you don't put a unit, it's relative to the computed size of the text. For example, if the text is 12px (or 0.75em in a browser with default font size of 16px) in size, the line height will be 18px.

The browser sets the line height to 1.2 by default, but it's usually too tight. And a line height of 1.5 makes the viewing experience a lot better. See image below, the line height on the left is 1.2 while the line height on the right is 1.5.

4. Letter spacing

This is used a lot less compared to line-height or font-size. In general, we only need to specify custom spacing for texts that are either too big or too small. And you can do it in CSS using the letter-spacing property. Similar to font-size, it's best to be specified in relative units such as em.

p {
  letter-spacing: 1em;
}

5. Length of line

The general consensus is that lines should contain 60 to 70 characters for best reading experience, especially for text heavy pages.

And you can specify the line length in CSS using the following property,

p {
  width: 50ch;
}

The unit ch represents the width, or more precisely the advance measure, of the glyph "0". And 50ch generally results in line width to be 60 to 70 characters in length. A lot of text heavy sites, such as Medium.com, or even the Google search result page, put a similar limit on the number of characters in a line.

That's all. Let me know in the comments section if you have any feedbacks!

What are registers in Vim, how to use them and the best practices.

Zean Qin — Thu, 10 Sep 2020 13:42:36 +0000

Overview

A register in Vim is basically a slot in memory that you can put a piece of text in and read it out later. Each register is like the clipboard provided by the OS (e.g. Windows, MacOS etc.), except that Vim has a LOT of them instead of just one.

It takes a while to learn how to use them and some slow repetitions in the beginning to develop the muscle memory, but you'll never go back to the simple yank/paste again after getting comfortable with it.

Basic usage

You can list all registers and their contents by using the :reg command, or using the :reg <register-a> <register-b> ... to filter to the specific registers and their contents.

Each register is accessed following the pattern "<register-name><command>. Note it's got 3 parts,

the " prefix,
the <register-name>, and
the <command> to perform on that register.

For example, "ay yanks the current selected text and stores it in the register "a, and "ap pastes the current content of register "a.

Registers

There're 9 different types of registers provided by Vim. I've listed them all below, but only included detailed explanation (with some common practices) for the ones I've used the most. For the advanced ones, I've included a link to the official documentation.

1. Unnamed (or default) register

The " register (accessed via "") is the default register.

Its content will be updated whenever you delete (using the d, c, s or x commands) or yank (using the y command) some text.

You can read the content of this register by using "". For example, you can do ""p to paste the value store in it.

Vim reads the content of the unnamed register for any put command (p or P) which does not specify a register. So, p is the same as ""p

2. Numbered registers

These are the registers "0, "1, "2, "3, "4, "5, "6, "7, "8, "9.

"0 will be set by the most recent yank command, therefore it always has the content of the latest yank.

If you yanked some text, you can always refer to it using "0p.

"1 will be set by the most recent delete or change command only if the deleted or changed text is longer than one line (also called big delete).

"2, "3, "4, "5, "6, "7, "8, "9 stores the other 8 latest big deletion or change. With each successive big deletion or change command, Vim shifts the previous contents of "1 into "2, "2 into "3, and so forth, losing the previous contents of "9.

3. Small delete register

The register - is set by a delete command only if the deleted text is smaller than one line (also called small delete).

Tip: The latest yanked texts will always appear in "0. The latest small deletion will appear in "- while the latest big deletion will appear in "0. The other latest 8 big deletion will appear in "2, "3, "4, "5, "6, "7, "8, "9.

4. Named registers

These are the registers "a to "z, or "A to "Z.

These registers are only used when you say so. Specify them as lowercase letters to replace their previous contents or as uppercase letters to append to their previous contents.

Tip: I only use a couple of these as more permanent storage for code snippets such as console.log(` `), as compared to the temporary storage provided by the numbered registers and the small delete register.

I find myself can survive with just the basic registers mentioned so far, and rarely use the ones below. But if you're interested, keep reading.

5. Read-only registers

These are %, #, : and ..

You can use them only with the p, P, and :put commands and with CTRL-R.

". contains the last inserted text (the same as what is inserted with the insert mode commands CTRL-A and CTRL-@).

Note: this doesn't work with CTRL-R on the command-line. It works a bit differently, like inserting the text instead of putting it (textwidth and other options affect what is inserted).

"% contains the name of the current file.

"# contains the name of the alternate file.

": contains the most recent executed command-line. Example: Use @: to repeat the previous command-line command.

The command-line is only stored in this register when at least one character of it was typed. Thus it remains unchanged if the command was completely from a mapping.

6. Expression register

The expression register is the "=. According to the documentation,

This is not really a register that stores text, but is a way to use an expression in commands which use a register. The expression register is read-only; you cannot put text into it. After the '=', the cursor moves to the command-line, where you can enter any expression. All normal command-line editing commands are available, including a special history for expressions. When you end the command-line by typing <CR\>, Vim computes the result of the expression. If you end it with <Esc\>, Vim abandons the expression. If you do not enter an expression, Vim uses the previous expression (like with the "/" command).
The expression must evaluate to a String. A Number is always automatically converted to a String. For the p and :put command, if the result is a Float it's converted into a String. If the result is a List each element is turned into a String and used as a line. A Dictionary or FuncRef results in an error message.
If the "= register is used for the p command, the String is split up at <NL\> characters. If the String ends in a <NL\>, it is regarded as a linewise register.

Please see the documentation for details

7. Selection and drop registers

These are the "*, "+, and "~.

They're used for storing and retrieving the selected text for the GUI. As I've never used this, I won't write down all the details for it. If you're interested, you could read the documentation for details.

8. Black hole register

The "_ is the black hole register. As the name implies, nothing will happen if you write to it, and nothing will be returned from it if you try to read its content.

Tip: You can direct your deleted text to the black hole register to avoid affecting other normal registers.

9. Last search pattern register

This is the "/ register, and it stores the most recent search pattern.

According to the documentation,

This is used for n and hlsearch. It is writable with :let, you can change it to have hlsearch highlight other matches without actually searching. You can't yank or delete into this register. The search direction is available in v:searchforward. Note that the valued is restored when returning from a function function-search-undo.

References

Common Git commands and configurations used in a day-to-day workflow

Zean Qin — Thu, 10 Sep 2020 06:31:57 +0000

I'm a command line lover and use Git from the terminal all the time.

If you're like me, you might start getting annoyed by some little things at a certain point. For example, it starts to get a bit annoying to always type two separate commands - git add <file> then git commit -m 'Your commit message' - to commit your changes. Or maybe you want to have a better looking git history when you type git log. Or you want your local branch to be automatically pruned when the remote branch has been deleted. Little things like these - you get the idea.

Over time, I have built up a curated list of commands, aliases and configrations that I use on a daily basis that makes my workflow more efficient and pleasant. And I'd like to share them with you below.

List your local branches and their remote tracking branches

In addition, this command also shows the hash code and the commit message of the latest commit. It also tells you if the remote branch has been deleted.

git branch -vv

For example, running the command produces the following output on my machine,

Checkout a remote branch

With Git versions ≥ 1.6.6 and with only one remote, you can just do:

git fetch
git checkout <branch_name>

git checkout <branch_name> will NOT work in modern Git if you have multiple remotes. In this case use

git checkout -b <branch_name> <remote_name>/<branch_name>

or the shorthand

git checkout -t <remote_name>/<branch_name>

Add and commit in one command

Add either one of the following aliases to your global Git config file (usually at ~/.gitconfig on a Linux/Mac OS system). I prefer the second one because it saves a few more keystrokes.

# add a `add-commit` alias, or
git config --global alias.add-commit '!git add -A && git commit'

# add a `ac` alias to save a few more keystrokes
git config --global alias.ac '!git add -A && git commit'

And use it with

git add-commit -m 'My commit message' # or
git ac -m 'My commit message'

Delete a branch both locally and remotely

When you're done with a branch, you can delete it from both the remote and your local machine using the commands below.

# delete a remote branch
git push -d <remote_name> <branch_name> # or
git push -D <remote_name> <branch_name>

# delete a local branch
git branch -d <branch_name> # or
git branch -D <branch_name>

Note that in most cases the name is origin.

Note: The -d option is an alias for --delete, which only deletes the branch if it has already been fully merged in its upstream branch. You could also use -D, which is an alias for --delete --force, which deletes the branch "irrespective of its merged status".

Delete all branches that have been merged in remote

Assume you have a long running development branch, and you branch off it to create different feature branches e.g. feature/A, feature/B.

After your peers have reviewed your pull requests for both features, merged them back to development and deleted them from remote. You can delete feature/A and feature/B from your local by running,

# switch to the development branch first
git checkout development

# delete local branches whose remote tracking branches have been merged back to development
git delete-merged

You probably have noticed that delete-merged is not a Git command - it's actually an alias we set up. The parameters used in the actual command is different depending on your setup. But you can follow the following steps to construct a command that suits your needs.

Step 1: Check out the development branch.

Step 2: List all branches that have been merged into it in remote.

git branch --merged

Step 3: You might see a few branches that you don't want to remove e.g. master, release etc. And you can filter down the list by excluding those branches by

git branch --merged | egrep -v "(^\*|master|development|skip_branch_name)"

The regular expression used by the egrep command basically means "all branches whose name starts with master, development or skip_branch_name will not be deleted".

You can modify the branches above or add your own branches that you don't want to delete.

Step 4: Delete all local branches that are already merged into the currently checked out branch

git branch --merged | egrep -v "(^\*|master|development|skip_branch_name)" | xargs git branch -d

Step 5: Set a global alias deleted-merged for the command

git config --global alias.delete-merged 'git branch --merged | egrep -v "(^\*|master|development|skip_branch_name)" | xargs git branch -d'

Discard unstaged files in current working directory

To discard all unstaged files in current working directory,

git checkout -- .

For a specific file, use

git checkout -- path/to/file/to/revert

The -- is to remove argument ambiguation.

Rename a local and remote branch

Step 1: Rename your local branch

If you are on the branch you want to rename:

git branch -m <new-name>

If you are on a different branch:

git branch -m <old-name> <new-name>

Step 2: Delete the <old-name> remote branch and push the <new-name> local branch

git push origin :<old-name> <new-name>

Step 3: Reset the upstream branch for the <new-name> local branch

Switch to the branch and then:

git push origin -u <new-name>

Prettify the Git log

You can format your Git log to look like below and set an alias for it.

Step 1: Set up the following alias in your global Git config file

git config --global alias.lg "log --graph --pretty=format:'%Cred%h%Creset -%C(yellow)%d%Creset %s %Cgreen(%cr) %C(bold blue)<%an>%Creset' --abbrev-commit --date=relative"

Step 2: Append additional flags if needed

# filter by start date
--after="2016-01-31"
--since="2016-01-31"

# filter by end date
--before="2017-03-10"
--until="2017-03-10"

# filter by author
--author="Zean Qin"

My most commonly used command is

git lg --after="yesterday" --author="Zean"

Prettify Git diff

I use diff-so-fancy to make the diffs more readable. Follow the official setup steps and then just run

git diff

Prune remote-tracking branches automatically for a branch on the other side that has already been deleted

Without git fetch --prune, remote-tracking branches for a branch
the other side already has removed will stay forever.

To always --prune for git fetch and git pull in all your Git repositories:

git config --global fetch.prune true

To always --prune but from one single repository,

git config remote.origin.prune true
                 #^^^^^^
                 #replace with your repo name

Make Git to use Vim as editor for writing commit messages

If you want to set the editor only for Git, do either (you don’t need both):

Set core.editor in your Git config: git config --global core.editor "vim"
Set the GIT_EDITOR environment variable: export GIT_EDITOR=vim

If you want to set the editor for Git and also other programs, set the standardized VISUAL and EDITOR environment variables:

Setting both is not necessarily needed, but some programs may not use the more-correct VISUAL.

export VISUAL=vim
export EDITOR="$VISUAL"