Forem: Yehonathan Sharvit

100 things I learned writing my first technical book

Yehonathan Sharvit — Sun, 26 Dec 2021 05:58:28 +0000

I just completed the manuscript of Data-Oriented Programming and I thought it was a good opportunity to reflect on what I learned from this journey.

Here are a hundred things I learned writing my first technical book:

Writing a technical book is much harder than writing blog posts.
Writing a blog post is like running a sprint while writing a book is like running a marathon.
Writing my first technical book without a publisher would have been a MISSION: IMPOSSIBLE!
Each piece of the book content must be clear and interesting. Each part, each chapter, each section, each paragraph, each sentence.
"Clear" is more important that "interesting". If something is not clear to your reader, it cannot be interesting for for them.
A possible way to make things clear is go from concrete to abstract.
A possible way to make things interesting is to teach the material as a story with fiction characters and a bit of drama.
The "why" is more important than the "what".
The "what" is more important than the "how".
An average writer makes the reader think the author is smart. A good writer makes the reader think the reader is smart.
A technical book is written for MQRs (Minimal Qualified Readers).
Figuring out what are the qualifications of your MQRs (Minimal Qualified Readers) is important as it allows you to assume what knowledge your readers already have.
It's hard to figure out what are the qualifications of your MQRs (Minimal Qualified Readers).
Checking book sales could be addictive.
Making a good Table of Contents is crucial as it is the first part of the book potential readers will encounter.
Making a good Table of Contents is hard as you need to figure out what you really want to talk about.
The Table of Contents might evolve a bit as you write your book.
You should resist the temptation to write the first chapter before the Table of Contents is ready.
It's not necessary to write chapters in order. But it's easier.
Never assume that your readers will read the next chapter only because they have enjoyed the previous chapter.
You should always convince your readers why what you are teaching is important and relevant for them.
Before writing a chapter, you should formulate to yourself what is the main objective of the chapter.
If a chapter has two main objectives, it's a sign that you should split it into two chapters.
A chapter should be treated like a piece of software. You should resist the temptation of writing the chapter contents without a plan.
A possible way to make things interesting is to use concrete examples.
A possible way to make things clear inside a chapter is to start with the easy stuff and increase the level of difficulty as the chapter goes.
Do not hesitate to highlight sentences that convey an important message.
It's OK to engage in writing a technical book without mastering every topic you want to cover in your book.
Writing technical book involves a descent amount of research even if you consider yourself as an expert in the field.
Finding attractive but accurate titles to book chapters is an art.
You can learn a lot from a failed attempt to write a book, provided that you put your ago aside.
If you try to write a Wikipedia article about the topic of your book before it is mentioned by other sources, it will be rejected.
It's possible to write a technical book while keeping your day job as a programmer, provided that you are willing to wake up early or sleep late.
Writing a technical book takes between a year and two.
Don't rush! Enjoy the journey...
It makes lot of sense to use a source control software for your manuscript.
AsciiDoc rocks!
PlantUML rocks!
NeoVim rocks!
Using a tool - like PlantUML - that generates diagrams from text makes it easy to refactor multiple diagrams at once (e.g rename a label, change a color).
People on Reddit could feel hurt by opinions that take them out of their comfort zone.
On Reddit, when people feel hurt, they could become violent.
Being mentored by an experienced writer is a blessing.
If you are lucky enough to be mentored by an experienced writer, ask them to be hard with you. That's how you are going to improve your book!
A good technical reviewer is a representative of your MQRs (Minimal Qualified Readers). They can tell you upfront is something is going to be unclear to your readers.
You should make sure your readers will never frown while reading your book.
A project manager that pays attention to the details is important.
Your publisher is your partner.
You could make more dollars per copy by self-publishing but you'd probably sell much less copies.
Asking early feedback from external reviewers is a great source of improvement.
Releasing an early version of the book (approx. when the first third is ready) allows you to find out if the topic of your book is interesting.
Finding a good book title is hard.
Finding a good book subtitle is even harder.
You need to be very careful not to hurt the sensitivity of any of your readers.
Having your book featured on HackerNews home page do not mean selling lots of copies.
Twitter is a great medium to share ideas from your book.
Writing a book could sometimes take you to flow.
My real motivation for writing a book was neither to be famous nor to be rich. It only wanted to accomplish a child's dream.
It's hard to find your voice.
Once you have found the your voice, the writing flows much better.
Usually readers stop reading after reading the middle of the book. If you want them to read the second half of your book, you need to find a way to hook them.
A possible way to hook your readers is to tell a story.
Inspiration is not linear. It's OK to stop writing for a couple of hours.
Motivation is not linear. It's OK to stop writing for a couple of weeks.
Be open to critics - even when they hurt your ego.
The more you write, the more you like it.
It's safe to assume that every developer can read JavaScript.
It's a great feeling to mention the work of other authors.
You should make sure that each and every code snippet - that appears in your book - runs as expected.
Invoking "it's so obvious I don't need to explain it" is not an acceptable argument.
Writing your teaching materials as a dialogue between an imaginary expert and a imaginary novice is a very useful process in order to figure out what questions your materials might raise in your reader's mind.
Sometimes the questions that an imaginary novice would ask about the stuff you teach would be tough. Don't ignore them. It's an opportunity to make your book better.
Rewriting a chapter from scratch because you forgot to save your work could be a blessing as writing from scratch might lead to a material of higher quality.
Writing in a coffee shop makes me feel like a famous author, but in fact I am much more productive at home.
Writing a preface - after the whole manuscript is ready - is really a pleasure!
You should think about the way your contents is going to appear on the paper. Use headlines, highlights, call outs and diagrams to make sure it doesn't look boring.
Resist the temptation to impress your readers with "cool stuff" if you think it might confuse them.
Working on your book is a good reason to wake up early. Sometimes, before sunrise (even in summer!).
Include at least 2 or 3 diagrams in every chapter. It makes the material fun to read and easier to grasp.
Draw your diagrams on a sheet of paper before using a drawing software.
It's OK to use colors in diagrams for the online version of the book. But remember that the print version of the book will be not be in color.
Mind maps are a great visualization tool. Use them smartly.
When a section is more difficult to read that the others, let your readers know about it.
When a section is more difficult to read that the others, make it skippable.
It's OK - from times to times - to copy-paste a diagram in order to save from your readers the need to flip back.
Asking a friend or a colleague to read your work in progress is not a productive idea. The best feedback come from people you don't know.
Brainstorming with a friend or a colleague about a difficulty you encounter might be a productive idea.
Throwing away some (good) ideas is sometimes necessary. Not easy but necessary.
When you are blocked in the middle of a chapter, it might be a sign that you need to rethink the chapter.
When you are blocked in the middle of a chapter, it might be a sign that you need to rest and come back later.
Adapting parts of your book to blog posts could be a good idea. But you need to resist the temptation of copy-pasting verbatim as the blog posts will be without the context of the book.
If feels great when someone with lots of followers tweets about the fun they had reading your book.
Don't worry if your English is not perfect. Your manuscript will be proofread later.
"Not being an native English speaker" is not an excuse for your lack of clarity.
Writing an appendix is much easier than writing a chapter.
Using humour in a technical book is possible. Hopefully, it's well appreciated.
You should write the chapter introduction after all the other parts of the chapter are written.
Getting positive feedback - even from people who are easily enthusiastic - feels good.
Front matter is the last part an author writes.
Writing a hundred things you learned from writing a technical book is not as difficult as it may seem.

That's it! If you find some of these lessons interesting you might want to write a book of your own or to take a look a the one I wrote: Data-Oriented Programming.

Data-Oriented Programming: A link in the chain of programming paradigms

Yehonathan Sharvit — Fri, 24 Dec 2021 04:17:45 +0000

Data-Oriented Programming is not an invention. It has its origins in the 1950s and the invention of LISP and is based on a set of best practices that can be found in both Functional Programming and Object-Oriented Programming. However, this paradigm has only been applicable in production systems at scale since the 2010s and the implementation of efficient persistent data structures.

This article traces the major ideas and discoveries which, over the years, have allowed the emergence of DOP.

Timeline

1958: LISP

In LISP, John McCarthy has the ingenious idea to represent data as generic immutable lists and to invent a language that makes it very natural to create lists and to access any part of a list. That's the reason why LISP stands for LISt Processing.

In as sense, LISP lists are the ancestors of JavaScript object literals. The idea that it makes sense to represent data with generic data structures (DOP Principle #2) definitely comes from LISP.

The main limitation of LISP lists is that when we update a list, we need to create a new version of it by cloning it and it has a negative impact on performances both in terms of CPU and memory.

1981: Values and Objects

In a beautiful, short and easy-to-read paper named Values and Objects in Programming Languages, Bruce MacLennan clarifies the distinction between values and objects. In a nutshell:

Values are timeless abstractions for which the concepts of updating, sharing and instantiation have no meaning. For instance, numbers are values.
Objects exist in time and hence can be created, destroyed, copied, shared and updated. For instance, an employee in a human resource software system is an object.

The meaning of the term object in this paper is not exactly the same as in the context of Object-Oriented Programming.

The author explains why it's much simpler to write code that deals with values than code that deals with objects.

This paper has been a source of inspiration for Data-Oriented Programming as it encourages us to implement our systems in such a way that most of our code deals with values.

2000: Ideal Hash Trees

Phil Bagwell invented a data structure called Hash Array Mapped Trie (HAMT). In his paper Ideal Hash trees, he used HAMT to implement hash maps with nearly ideal characteristics both in terms of computation and memory usage.

HAMT and Ideal hash trees are the foundation of efficient persistent data structures.

2006: Out of the Tar Pit

In Out of the Tar Pit, Ben Moseley and Peter Marks claim that complexity is the single major difficulty in the development of large-scale software systems. In the context of their paper, complexity means what make a system hard to understand.

The main insight of the authors is that most of the complexity of software systems in not essential but accidental: the complexity doesn't come from the problem we have to solve but from the software constructs we use to solve the problem. They suggest various ways to reduce complexity of software systems.

In a sense, Data-Oriented Programming is a way to get us out of the tar pit.

2007: Clojure

Rich Hickey, an Object-Oriented Programming expert, invented Clojure to make it easier to develop information systems at scale. Rich Hickey likes to summarize Clojure core value with the phrase: "Just use maps!". By maps, he means immutable maps to be manipulated efficiently by generic functions. Those maps were implemented using the data structures presented by Phil Bagwell in "Ideal Hash Trees".

Clojure has been the main source of inspiration for Data-Oriented Programming. In a sense, Data-Oriented Programming is a formalization of the underlying principles of Clojure and how to apply them in other programming languages.

2009: Immutability for all

Clojure's efficient implementation of persistent data structures has been attractive for developers from other programming langauges. In 2009, there were ported to Scala. Over the years, they have been ported to other programming languages either by organizations (like Facebook for Immutable.js) or by individual contributors (like Glen Peterson for Paguro in Java).

Nowadays, DOP is applicable in virtually any programming language!

DOP principles as best practices

The principles of Data-Oriented Programming are not new. They come from best practices that are well-known among software developers from various programming languages. The innovation of Data-Oriented programming is the combination of those principles into a cohesive whole.

In this section, we put each one of the 4 DOP principles into its broader scope.

Principle #1: Separate code from data

Separating code from data used to be the main point of tension between Object-Oriented Programming (OOP) and Functional Programming (FP). Traditionally, in OOP we encapsulate data together with code in stateful objects, while in FP we write stateless functions that receive data they manipulate as an explicit argument.

This tension has been reduced over the years as it is possible in FP to write stateful functions with data encapsulated in their lexical scope. Moreover, OOP languages like Java and C# have added support for anonymous functions (lambdas).

Principle #2: Represent data with generic data structures

One of the main innovation of JavaScript when it was released in December 1995 was the easiness to create and manipulate hash maps via object literals. The increasing popularity of JavaScript over the years as a language used everywhere (frontend, backend, desktop) has influenced the developer community to represent data with hash maps when possible. It feels more natural in dynamically-typed programming languages, it is applicable also in statically-typed programming languages.

Principle #3: Data is immutable

Data immutability is considered as a best practice as it makes the behaviour of our program more predictable. For instance, in Effective Java, Joshua Block mentions "Minimize mutability" as one of Java best practices.

There is a famous quote from Alan Kay - who is considered by many as the inventor of Object-Oriented Programming - about the value of immutability:

The last thing you wanted any programmer to do is mess with internal state even if presented figuratively. Instead, the objects should be presented as site of higher level behaviors more appropriate for use as dynamic components. (...) It is unfortunate that much of what is called "object-oriented programming" today is simply old style programming with fancier constructs. Many programs are loaded with "assignment-style" operations now done by more expensive attached procedures.

Unfortunately, until 2007 and the implementation of efficient persistent data structures in Clojure, immutability was not applicable for production applications at scale.

Nowadays, efficient persistent data structures are available in most programming languages.

Language	Library
Java	Paguro
C#	Provided by the language
JavaScript	Immutable.js
Python	Pyrsistent
Ruby	Hamster

In addition to that, many languages provide support for read-only objects natively. Java added record classes in Java 14. C# introduced a record type in C# 9. There is a ECMAScript proposal for supporting immutable records and tuples in JavaScript . Python 3.7 introduced Immutable data classes.

Principle #4: Separate data schema from data representation

One of the more virulent critics against dynamically-typed programming languages used to be related to the lack of data validation. The answer that dynamically-typed languages used to give to this critics was that you trade data safety for data flexibility.

Since the development of data schema languages like JSON schema it is natural to validate data even when data is represented as hash maps.

Wrapping up

In this article, we have explored the ideas that inspired Data-Oriented Programming and the discoveries that made it applicable in production systems at scale in most programming languages.

A simple way to reduce complexity of Information Systems

Yehonathan Sharvit — Wed, 19 May 2021 11:12:01 +0000

When it comes to information systems, things can get pretty complex, to say the least. A typical information system like a web service, at the most basic level, is just one process in a massive, integrated data pipeline. It deals mostly with data processing: fetching data, transforming it and passing it on to another system. But as other systems pile up on top of it, the complexity builds up quickly. Managing and mitigating that complexity then becomes a major challenge for developer teams.

Traditionally, information systems have been implemented using software programming paradigms like Object-Oriented Programming, based on the concept of “objects”, which can contain data and code. Information systems that follow Object-Oriented Programming with no constraints tend to be complex, in the sense that they are hard to understand and hard to maintain.

The increase of system complexity tends to reduce the velocity of the development team as it takes more time to add new features to the system. Hard-to-diagnose issues occur more frequently in production. Issues that cause either user frustration when the system doesn’t behave as expected or even worse, system down time.

Three aspects of Object-Oriented programming are a source of complexity:

Data encapsulation in objects
Non-flexible data layout in classes
State mutation

Data encapsulation inside objects is beneficial in many cases. However, in the context of modern information systems, data encapsulation tends to create complex class hierarchies where objects are involved in many relations with other objects.

Over the years, this complexity has been alleviated by the invention of advanced design patterns and software frameworks. But information systems built with Object-Oriented programming still tend to be complex.

Representing every piece of data through a class is helpful for tooling (e.g. autocompletion in the editor) and errors like accessing non-existing fields are detected at compile time. However, the rigidity of class layout makes data access not flexible. In the context of information systems, it’s painful: Each and every variation of data is represented by a different class. For instance, in a system that deals with customers, there is a class that represents a customer as seen by the database and a different class that represents a customer as seen by the data manipulation logic. Similar data with different field names, but the proliferation of classes is unavoidable. The reason is that data is “locked” in classes.

In multi-threaded information systems, the fact that the state of the object’s is allowed to be mutated is another source of complexity. The introduction of various lock mechanisms in order to prevent data from being modified concurrently and to ensure the state of our objects remain valid makes the code harder to write and to maintain. Sometimes, before passing data to a method form third-party libraries, we use a defensive copy strategy to make sure our data is not modified. The addition of lock mechanisms or defensive copy strategy makes our code more complex and less performant.
Data-Oriented Programming (DOP) is a set of best practices that have been followed by developers in order to reduce complexity of information systems.

The idea behind DOP is to simplify the design and implementation of information systems by treating data as a “first-class citizen”. Instead of designing information systems around objects that combine data and code, DOP guides us to separate code from data and to represent data with immutable generic data structures. As a consequence, in DOP developers manipulate data with the same flexibility and serenity as they manipulate numbers or strings in any program.
DOP reduces system complexity by following three core principles:

Separating code from data
Representing data with generic data structures
Keeping data immutable

One possible way to adhere to DOP in an Object-Oriented programming language is to write code in static class methods that receive data they manipulate as an explicting argument.
The separation of concerns achieved by separating code from data tends to make the class hierarchy less complex: instead of designing a system with a class diagram made of entities involved in many relationships, the system is made of two disjoint simpler subsystems: a code subsystem and a data subsystem.

When we represent data with generic data structures (like hash maps and lists), data access is flexible and it tends to reduce the number of classes in our system.
Keeping data immutable brings serenity to the developer when they need to write a piece of code in a multi-threaded environment. Data validity is ensured without the need to protect the code with lock mechanisms or defensive copy.
DOP principles are applicable both to Object-Oriented and to functional programming languages. However, for Object-Oriented developers, the transition to DOP might require more of a mind shift than for functional programming developers, as DOP guides us to get rid of the habit of encapsulating data in stateful classes.

Yehonathan Sharvit has been working as a software engineer since 2000, programming with C++, Java, Ruby, JavaScript, Clojure and ClojureScript. He currently works as a software architect at CyCognito, building software infrastructures for high scale data pipelines. He shares insights about software at his tech blog. Yehonathan recently published the book Data-Oriented Programming available from Manning.

Two approaches of sameness in programming: OOP vs FP

Yehonathan Sharvit — Thu, 22 Apr 2021 12:22:33 +0000

We often talk about the importance of writing our code in terms of pure functions which, like math functions, have no side effects and return the same values for the same arguments. In this article, I propose to explore another fundamental aspect of the difference between functional programming and object-oriented programming: the notion of sameness, that is to say: how do we define that two "things" are the same?

Ship of Theseus

Do you know the story about the ship of Theseus? This Greek hero whose ship, according to legend, was preserved by the Athenians for centuries: they removed the worn planks from the ship and replaced them until no original planks remained.

The question then arose as to whether it was still the same ship or whether the maintenance had made it a different ship.

What do you think? And most importantly what does that have to do with programming?

Sameness in math

Let's start by exploring the concept of sameness in math and ask ourselves a question similar to the ship of Theseus in the realm of sets.

Consider a set with three elements: the numbers 1, 2 and 3.

What happens when we replace a number from this set, for example: the number 3 by the number 4?

Obviously, we get a set than is not the same as the original set!

Now imagine that we have a set with the numbers 1, 2 and 10. What happens when we replace 10 with 3? Well, we get a set with the numbers 1, 2 and 3.

A set with the numbers 1, 2 and 3 like the one in the previous example? Is it the same set or another set with the same elements? Funny question, isn't it?

For mathematics, the answer is formal: two sets having the same elements are the same! This is one of the axioms of axiomatic set theory. This axiom even has a pretty name: it is called the axiom of extensionality¹.

According to mathematics, therefore, the ship of Theseus is no longer the same as soon as one of its planks is replaced

Sameness in programming

Let's move on to programming now, if you don't mind. Imagine a product sold on an e-commerce site, a pretty coffee mug, for instance. Let's simplify it and say that a mug has only two attributes: a description and a price. What happens when you drop the price of a cup? Is this the same cup or is it a different cup?

Obviously, this is the same cup! In programming, the identity of an object is more than the values of its attributes.

According to programming, therefore, the ship of Theseus remains the same, although all its planks have been replaced.

Functional programming

We are at the core of an obvious contradiction between the world of programming and the world of mathematics. As Bruce MacLennan so beautifully wrote in his beautiful article "Values and Objects in Programming Languages", in 1982²:

Math is value oriented programming.

Programming is object-oriented mathematics.

What he means by an object is an entity having a set of attributes at a given time. Whereas a value is, by definition, an entity that never changes.

In a way, it can be said that the fundamental divergences between functional programming and object-oriented programming revolve around the manipulation of the object/value cursor. Functional programming encourages developers to activate the cursor towards values while object-oriented programming encourages developers to activate the cursor towards objects.

One of the reasons why a program written in functional programming is less complex than an object-oriented program is because, as we have just seen, it is more complex to define the sameness of objects than the sameness of values.

Back to Theseus

Writing a program these days is sometimes as much of a challenge as finding a way out of Daedalus' Labyrinth. According to legend, Theseus managed to find his way out of the Labyrinth with the help of Ariadne's thread.
I am going to let you meditate on the following question: Could we consider functional programming as Ariadne's thread that will allow us to find our way out of the maze of complexity created by our programs?

This article was first published on my tech blog.

https://en.wikipedia.org/wiki/Axiom_of_extensionality ↩
MacLennan, Bruce. (1982). Values and Objects in Programming Languages. ↩

Data-Oriented programming simplicity illustrated by coding challenges

Yehonathan Sharvit — Thu, 01 Apr 2021 21:17:33 +0000

According to Data-Oriented programming, the best way to reduce complexity of information systems is to follow three basic principles:

Separate code from data
Keep data immutable
Represent data with generic data structures

Here are a series of 6 short programming challenges and their solutions written in JavaScript according to the principles of Data-Oriented programming. The purpose is to illustrate the simplicity of Data-Oriented programming.

If you agree with DOP principles, please implement your solutions in the programming language of your choice, according to DOP principles. I'm sure you'll enjoy!

If you disagree with one or more DOP principles, feel free to implement the solutions by breaking one or more principles, and explain why you think that your solution is simpler than the ones that I wrote.

Rules

You are allowed to choose any programming language
You are allowed to use any third-party library
You are allowed to use reflection
In the context of the challenges, simplicity is more important than performances.
Submit your code snippets as a pull request to the official book source code Github repository, under the challenges folder
In case you disagree with DOP, please add a few words as comments in your code that explain why you think that your solution is simpler than the ones that I wrote.
Four copies of Data-Oriented programming will be given away among the folks that submit a correct solution to at least 4 of the challenges.

Data model

Here is the library data model in an informal way:

More formally, here is a UML diagram that describes the relationships between the data entities of the library:

Here is an example of library data in JavaScript that we are going to use through this article:

var libraryData = {
    "name": "The smallest library on earth",
    "address": "Here and now",
    "catalog": {
        "booksByIsbn": {
            "978-1779501127": {
                "isbn": "978-1779501127",
                "title": "Watchmen",
                "publicationYear": 1987,
                "authorIds": ["alan-moore",
                              "dave-gibbons"],
                "bookItems": [
                    {
                        "id": "book-item-1",
                        "rackId": "rack-17",
                    },
                    {
                        "id": "book-item-2",
                        "rackId": "rack-17",
                    }
                ]
            }
        },
        "authorsById": {
            "alan-moore": {
                "name": "Alan Moore",
                "bookIsbns": ["978-1779501127"]
            },
            "dave-gibbons": {
                "name": "Dave Gibbons",
                "bookIsbns": ["978-1779501127"]
            }
        }
    },
  "userManagement": {
    "librarians": {
      "franck@gmail.com" : {
        "email": "franck@gmail.com",
        "encryptedPassword": "bXlwYXNzd29yZA=="
      }
    },
    "members": {
      "samantha@gmail.com": {
        "email": "samantha@gmail.com",
        "encryptedPassword": "c2VjcmV0",
        "isBlocked": false,
      }
    }
  }
};
~~~{% endraw %}

# Warm up

What's the title of the book whose ISBN is "978-1779501127" in upper case?

In this article, I am using [Lodash FP](https://github.com/lodash/lodash/wiki/FP-Guide) configured so that it never mutates data in place. Instead of mutating data in place, functions like {% raw %}`_.set()` create a new version.

~~~javascript
var fp = _.noConflict();
var _ = fp.convert({
  // Specify capping iteratee arguments.
  'cap': false,
  // Specify currying.
  'curry': false,
  // Specify fixed arity.
  'fixed': false,
  // Specify immutable operations.
  'immutable': true,
  // Specify rearranging arguments.
  'rearg': false
});
~~~

~~~javascript
var informationPath = ["catalog", "booksByIsbn", "978-1779501127", "title"]; 
_.get(libraryData, informationPath).toUpperCase();
~~~

In Data-Oriented programming, each piece of information has an information path. As you'll see through the upcoming challenges, this unusual approach has many benefits.


# Challenge #1: Retrieve a piece of information

**Challenge ‍**: Write a function named {% raw %}`getBookProperty`{% endraw %} that receives library data and ISBN and a field name and returns the value of the field for the book with the given ISBN{% raw %}

~~~javascript
function getBookProperty(libraryData, isbn, fieldName) {
  var informationPath = ["catalog", "booksByIsbn", isbn, fieldName]; 
  return _.get(libraryData, informationPath);
}
~~~{% endraw %}

Example of usage:{% raw %}

~~~javascript
getBookProperty(libraryData, "978-1779501127", "title");
~~~{% endraw %}

In Data-Oriented programming, data fields are first-class citizens. We are free to create and combine field names dynamically in our program.

# Challenge #2: Search information

**Challenge ‍**: Write a function named {% raw %}`bookInfo`{% endraw %} that receives library data and a string and returns a JSON string that contains book information about the books whose title contains the given string, in a case insensitive way. Book information is made of: title, isbn, author full names.

**Remark**: You are not allowed to extract author names from author ids. Assume that author ids are opaque strings.{% raw %}

~~~javascript
function authorNames(catalogData, book) {
  return _.map(_.get(book, "authorIds"),
               function(authorId) {
    return _.get(catalogData, ["authorsById", authorId, "name"]);
    });
}

function bookInfo(catalogData, book) {
  return  {
    "title": _.get(book, "title"),
    "isbn": _.get(book, "isbn"),
    "authorNames": authorNames(catalogData, book)
  };
}

function searchBooksByTitle(libraryData, query) {
  var catalogData = _.get(libraryData, "catalog");
  var allBooks = _.get(catalogData, "booksByIsbn");
  var matchingBooks = _.filter(allBooks, function(book) { 
    return _.get(book, "title").toLowerCase()
      .includes(query.toLowerCase());
  });
  return JSON.stringify(_.map(matchingBooks, function(book) {
    return bookInfo(catalogData, book);
  }));
}
~~~{% endraw %}

Example of usage:{% raw %}

~~~javascript
searchBooksByTitle(libraryData, "watCH");
~~~{% endraw %}

Here, the main benefits are the power of expression of {% raw %}`map`{% endraw %} and {% raw %}`reduce`{% endraw %} combined with the freedom of creating on the fly a book info structure and serialize it for free.

# Challenge #3: Add a piece of information

**Challenge**: Write a function named {% raw %}`blockMember`{% endraw %} that receives library data and an email address and returns a new version of library data **without altering the original version**, where the user with the given email is blocked.

Remember that I am using a version of Lodash that, instead of mutating data in place, creates a new version.{% raw %}


~~~javascript
function blockMember(libraryData, email) {
  var informationPath = ["userManagement", "members", email, "isBlocked"]; 
  return _.set(libraryData, informationPath, true);
}
~~~{% endraw %}

Example of usage:{% raw %}

~~~javascript
blockMember(libraryData, "samantha@gmail.com");
~~~{% endraw %}

In Data-Oriented programming, data is immutable. Functions like {% raw %}`_.set()_`{% endraw %} make it efficient (both in terms of memory and computation) to create modified versions of data.

# Challenge #4: Rename keys in a data entity

**Challenge**: Write a function named {% raw %}`renameKeys`{% endraw %} that receives a data entity and a key mappings and returns a new data entity, without altering the original entity, where the fields are renamed according to the key mappings{% raw %}

~~~javascript
function renameKeys(map, keyMap) {
  return _.reduce(keyMap,
                  function(res, newKey, oldKey) {
                    var value = _.get(map, oldKey);
                    var resWithNewKey = _.set(res, newKey, value);
                    var resWithoutOldKey = _.omit(resWithNewKey, oldKey);
                    return resWithoutOldKey;
                  },
                  map);
}
~~~{% endraw %}

{% raw %}`renameKeys`{% endraw %} works with author entities:{% raw %}

~~~javascript
var alanMoore = {
  "name": "Alan Moore",
  "bookIsbns": ["978-1779501127"]
};
renameKeys(alanMoore, {"bookIsbns": "books"}); 
~~~{% endraw %}

{% raw %}`renameKeys`{% endraw %} works also with book item entities:{% raw %}

~~~javascript
var bookItem = {
  "id": "book-item-1",
  "rackId": "rack-17",
  "isLent": true
};

renameKeys(bookItem, {"rackId": "id",
                     "id": "bookItemId"}); 
~~~{% endraw %}

In Data-Oriented programming, data entities are represented with generic data structures that can be manipulated with generic functions that work with any data entity.

# Challenge #5: Merge pieces of information

**Challenge**: Write a function named {% raw %}`mergeAndSerialize`{% endraw %} that receives two pieces of book information, one from the database and one from an external service like [Open Library Books API](https://openlibrary.org/dev/docs/api/books) and returns a JSON string with information from both sources.

~~~javascript
var watchmenFromDB = {
        "isbn": "978-1779501127",
        "title": "Watchmen",
        "publicationYear": 1987,
        "authorIds": ["alan-moore",
                      "dave-gibbons"],
        "bookItems": [
          {
            "id": "book-item-1",
            "rackId": "rack-17",
            "isLent": true
          },
          {
            "id": "book-item-2",
            "rackId": "rack-17",
            "isLent": false
          }
        ]
      };

var watchmenFromOpenLib = {
  "publishers": [
    "DC Comics"
  ],
  "number_of_pages": 334,
  "weight": "1.4 pounds",
  "physical_format": "Paperback",
  "subjects": [
    "Graphic Novels",
    "Comics & Graphic Novels",
    "Fiction",
    "Fantastic fiction"
  ],
  "isbn_13": [
    "9780930289232"
  ],
  "title": "Watchmen",
  "isbn_10": [
    "0930289234"
  ],
  "publish_date": "April 1, 1995",
  "physical_dimensions": "10.1 x 6.6 x 0.8 inches"
}
~~~{% endraw %}

We simply merge the two maps:{% raw %}

~~~javascript
_.merge(watchmenFromDB, watchmenFromOpenLib);
~~~{% endraw %}

And we JSON serialize the result:{% raw %}

~~~javascript
function mergeAndSerialize(a, b) {
  return JSON.stringify(_.merge(a, b));
}
~~~{% endraw %}

When we represent data with generic data structures, we benefit from many well defined functions like {% raw %}`merge`{% endraw %}, implemented either in the programming language itself or in third-party libraries like {% raw %}`Lodash.js.


# Challenge #6: Compare versions of data

**Challenge**: Write a function named `{% endraw %}diff{% raw %}` that receives two versions of library data and returns an object that contains the diff between the two versions, in the format of your choice.

~~~javascript
function diffObjects(data1, data2) {
  var emptyObject = _.isArray(data1) ? [] : {};
  if(data1 == data2) {
    return emptyObject;
  }
  var keys = _.union(_.keys(data1), _.keys(data2));
  return _.reduce(keys,
                  function (acc, k) {
    var res = diff(_.get(data1, k),
                   _.get(data2, k));
    if((_.isObject(res) && _.isEmpty(res)) ||
       (res == "data-diff:no-diff")) {
      return acc;
    }
    return _.set(acc, [k], res);
  },
                  emptyObject);
}

function diff(data1, data2) {
  if(_.isObject(data1) && _.isObject(data2)) {
    return diffObjects(data1, data2);
  }
  if(data1 !== data2) {
    return data2;
  }
  return "data-diff:no-diff";
}

diff(libraryData, updatedLibraryData);

diff(libraryData, libraryData);

When every piece of data in the system is represented with generic data structures, it is quite easy to compare recursively different data versions.

Conclusion

Do you like 😃 DOP or do you hate 😡 it?

Anyway, it's time to show off your coding skills 🧑‍💻!

Submit your code snippets as a pull request to the official book source code Github repository, under the challenges folder.
In case you disagree with DOP, please add a few words as comments in your code that explain why you think that your solution is simpler than the ones that I wrote.

This article was first published on my blog.

How to access data dynamically in Java without losing type safety

Yehonathan Sharvit — Sun, 28 Mar 2021 21:24:14 +0000

Liquid syntax error: 'raw' tag was never closed

The concepts behind Data-Oriented programming

Yehonathan Sharvit — Wed, 17 Mar 2021 13:51:58 +0000

Liquid syntax error: 'raw' tag was never closed

Data-Oriented programming in Java

Yehonathan Sharvit — Fri, 05 Mar 2021 12:09:03 +0000

Liquid syntax error: 'raw' tag was never closed

Structural sharing with 7 lines of JavaScript.

Yehonathan Sharvit — Fri, 26 Feb 2021 08:12:25 +0000

This article was first published on my blog.

Motivation

When we build an application that embraces data immutability, we handle changes in data by creating a new version of data instead of mutating data in place, without compromising on performance. There are various techniques to achieve data immutability in an efficient manner.

JavaScript data manipulation libraries like Ramda and Lodash FP provide an interesting implementation of structural sharing via a technique called path copying. The cool thing with path copying is that it works with native JavaScript objects.

After reading this article, you will:

😄 Be part of the few lucky JavaScript developers that clearly understand what is structural sharing
💻 Implement structural sharing in a few lines of JavaScript code
⚠️ Be aware of the limitation of path copying
🤗 Be motivated to read my book about Data-Oriented programming

Remark: This article assumes that you are already aware of the benefits of data immutability.

What is structural sharing?

Structural sharing provides an efficient way to share data between multiple versions of it, instead of copying the whole data.

It's kind of similar to the way git manages multiple versions of your source code: git doesn't copy all the files on each commit. Instead, the files that are not changed by a commit are shared with previous commits.

The same strategy could be applied with data. Let's take as an example a map that represents the data of a library in a library management system. In this map, we have 4 fields:

name: a string for the name of the library
address: a string for the address of the library
catalog: a map with the book information
users: a map for the user information

Here is an example of a tiny library, with two users and a single book:

var libraryData = {
  "name": "The smallest library on earth",
  "address": "Here and now",
  "users": [
    {"username": "user-1",
     "password": "pass-1"}, 
    {"username": "user-2",
     "password": "pass-2"}
  ],
  "catalog": {
    "books": [
      {
        "title": "Watchmen",
        "publicationYear": 1986,
        "authors": [
          {
            "firstName": "Alan",
            "lastName": "Moore"
          },
          {
            "firstName": "Dave",
            "lastName": "Gibbons"
          }
        ]
      }
    ]
  }
}
~~~{% endraw %}


Suppose we want to create a version of the library where the {% raw %}`address`{% endraw %} field is modified. We can achieve that by **shallow copying** the original library and modify the {% raw %}`address`{% endraw %} field, using {% raw %}`Object.assign()`{% endraw %}.{% raw %}

~~~js
function set(obj, key, val) {
  var copy = Object.assign({}, obj);
  copy[key] = val;
  return copy;
}
~~~{% endraw %}

For fields whose values are strings, it works fine because **strings are immutable** in JavaScript. But what about fields whose values are maps? We don't want changes made on one of version of the map to be reflected on the other versions of the map!

Like in git, we don't want changes in a commit to affect files in a previous commit!

We could solve this challenge in a very naive way by **deep copying** the whole map when creating a new version of it. But of course, it would have a negative impact on the performance, both in terms of memory and CPU.

Another approach is to **prevent data from being changed**. When data is immutable, we don't need to protect ourselves from the threat we just mentioned. It is safe to do a shallow copy of the data because immutable data never changes.


Data immutability could be guaranteed either by convention (and code reviews) or by the program itself. For example, JavaScript provides a way to prevent data from being changed, via {% raw %}`Object.freeze()`. Here is an implementation of a deep freeze, from [MDN](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Object/freeze):

~~~js
function deepFreeze(object) {
  const propNames = Object.getOwnPropertyNames(object);
  // Freeze properties before freezing self
  for (const name of propNames) {
    const value = object[name];
    if (value && typeof value === "object") {
      deepFreeze(value);
    }
  }
  return Object.freeze(object);
}
~~~

That's the essence of structural sharing:

1. Make sure data is **immutable**
1. Create new version of data via **shallow copying**

Here is a code snippet for a function called {% raw %}`set()`{% endraw %} that implements structural sharing when the change occurs at the **root of the map**:{% raw %}

~~~js
function shallowCopy(o) {
  if(Array.isArray(o)) {
    return Object.assign([], o);
  }
  return Object.assign({}, o);
}

function set(o, k, v) {
  var copy = shallowCopy(o);
  copy[k] = v;
  return copy;
}
~~~{% endraw %}

Creating a new version of data with a change at the root of a map is easy. Now, we are going to show how to handle changes at any nesting level in a map.

# Implementation of path copying in JavaScript

Suppose we want to update the password of a user and see how to apply **recursively** our structural sharing strategy:

1. Shallow copy {% raw %}`name`{% endraw %}, {% raw %}`address`{% endraw %} and {% raw %}`catalog`{% endraw %}.
1. Use a modified version of {% raw %}`users`{% endraw %}:
   1. Shallow copy all users except {% raw %}`user-1`{% endraw %}
   1. Use a modified version of {% raw %}`user-1`{% endraw %}:
      1.  Shallow copy all the fields except {% raw %}`password`{% endraw %}
      1.  Modify {% raw %}`password`{% endraw %}


![change-password](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/fb4zadg38l01xrju5ojc.png)

The code for it would look like this:{% raw %}

~~~js
var nextUser = Object.assign({}, libraryData.users[1]);
nextUser.password = "new-pass-2";
var nextUsers = Object.assign([], libraryData.users);
nextUsers[1] = nextUser;
var nextLibraryData = Object.assign({}, libraryData);
nextLibraryData.users = nextUsers;
~~~{% endraw %}


And now, let's generalize this approach with a recursive functions. As we promised in the article title, here is an implementation of structural sharing in **7 lines of JavaScript code**:

~~~js
function setIn(m, [k, ...restOfPath], v) {
  var modifiedNode = v;
  if (restOfPath.length > 0) {
    modifiedNode = setIn(m[k], restOfPath, v);
  }
  return set(m, k, modifiedNode);
}
~~~

Here is how we use `setIn` to modify the password of a user:

~~~js
var libraryDataV1 = setIn(libraryData, ["users", 1, "password"], "new-pass-2");
libraryDataV1.users[1].password // "new-pass-2"
~~~

Of course, the previous version is left unchanged:

~~~js
libraryData.users[1].password // "pass-2"
~~~


# Efficiency of structural sharing

Path copying is **usually efficient** -- both in terms of memory and computation -- because most of the nodes in a nested map are copied by reference (shallow copy).

For example, the catalog map (that could be a huge object) is shared between the original library data and the new version of library data. They both use the same reference.

~~~js
libraryDataV1.catalog === libraryData.catalog
~~~

Path copying works fine with deeply nested data where at each nesting level we don't have too many elements. When we have **many elements at some level**, shallow copying might be an issue. Suppose we have a million user in our system, copying a million references each time we update the password of a user is not acceptable. 

The same issue occurs with git if you have a folder with too many files. 

In my book about [Data-Oriented programming](https://www.manning.com/books/data-oriented-programming?utm_source=viebel&utm_medium=affiliate&utm_campaign=book_sharvit2_data_1_29_21&a_aid=viebel&a_bid=d5b546b7), I discuss techniques to overcome this limitation.

The book illustrate in details the benefits of building a software system based upon data immutability.

3 programming paradigms that every Object-Oriented developer should know

Yehonathan Sharvit — Sun, 21 Feb 2021 07:55:02 +0000

Data-oriented programming is not a new concept. It is a paradigm that is applied by Object-Oriented developers in order to reduce the complexity of the systems they build.

The purpose of my book Data-oriented programming is to unveil the principles underlying this paradigm and to illustrate their benefits in the context of a software system written in an Object-Oriented language like Java, C# or C++.

The present article describes the distinction between Data-oriented programming and two other programming paradigms whose name contain the term data: Data-oriented design and Data-driven programming.

Each paradigm has a its own objective and pursues it by focusing on a different aspect of data.

There are only two hard things in Computer Science: cache invalidation and naming things. (Phil Karlton)

Data-oriented design

Data-oriented design is a program optimization approach motivated by efficient usage of the CPU cache, used mostly in video game development.

The approach is to focus on the data layout, separating and sorting fields according to when they are needed, and to think about transformations of data.

In this context, what's important is how the data resides in memory.

The objective of this paradigm is to improve the performance of the system.

Data-driven programming

Data-driven programming is the idea that you create domain specific languages (DSLs) which are made out of descriptive data. It is a branch of declarative programming.

In this context, what's important is to describe the behaviour of a program in terms of data.

The objective of this paradigm is to increase code clarity and to reduce the risk of bugs related to mistakes in the implementation of the expected behaviour of the program.

Data-oriented programming

Data-oriented programming is a paradigm that treats data of the system as a first-class citizen. Data is represented by generic immutable data structures (like maps and vectors) that are manipulated by general purpose functions (like map, filter, select, group, sort ...).

In this context, what's important is the representation of data by the program.

The objective of this paradigm is to reduce the complexity of the system.

Summary

Before we conclude, I'd like to mention that there is a draft Wikipedia article about Data-Oriented programming. You are welcome to contribute to the article.

We saw that the three data-related paradigms have different objectives and each of them pursues its objective by focusing on a different aspect of data inside a program.

Paradigm	Objective	Data
Data-oriented design	Increase performance	Data layout
Data-driven programming	Increase clarity	Behaviour described by data
Data-oriented programming	Reduce complexity	Data representation

3 data-related programming paradigms that every C++ developer should know

Yehonathan Sharvit — Sun, 21 Feb 2021 07:44:55 +0000

Data-oriented programming is not a new concept. It is a paradigm that is applied by C++ developers in order to reduce the complexity of the systems they build.

The purpose of my book Data-oriented programming is to unveil the principles underlying this paradigm and to illustrate their benefits in the context of a software system written in C++.

Each paradigm has a its own objective and pursues it by focusing on a different aspect of data.

There are only two hard things in Computer Science: cache invalidation and naming things. (Phil Karlton)

Data-oriented design

Data-oriented design is a program optimization approach motivated by efficient usage of the CPU cache, used mostly in video game development.

The approach is to focus on the data layout, separating and sorting fields according to when they are needed, and to think about transformations of data.

In this context, what's important is how the data resides in memory.

The objective of this paradigm is to improve the performance of the system.

Data-driven programming

Data-driven programming is the idea that you create domain specific languages (DSLs) which are made out of descriptive data. It is a branch of declarative programming.

In this context, what's important is to describe the behaviour of a program in terms of data.

The objective of this paradigm is to increase code clarity and to reduce the risk of bugs related to mistakes in the implementation of the expected behaviour of the program.

Data-oriented programming

In this context, what's important is the representation of data by the program.

The objective of this paradigm is to reduce the complexity of the system.

Summary

Before we conclude, I'd like to mention that there is a draft Wikipedia article about Data-Oriented programming. You are welcome to contribute to the article.

We saw that the three data-related paradigms have different objectives and each of them pursues its objective by focusing on a different aspect of data inside a program.

Paradigm	Objective	Data
Data-oriented design	Increase performance	Data layout
Data-driven programming	Increase clarity	Behaviour described by data
Data-oriented programming	Reduce complexity	Data representation

3 data-related programming paradigms that every C# developer should know

Yehonathan Sharvit — Sun, 21 Feb 2021 07:43:20 +0000

Data-oriented programming is not a new concept. It is a paradigm that is applied by C# developers in order to reduce the complexity of the systems they build.

The purpose of my book Data-oriented programming is to unveil the principles underlying this paradigm and to illustrate their benefits in the context of a software system written in C#.

Each paradigm has a its own objective and pursues it by focusing on a different aspect of data.

There are only two hard things in Computer Science: cache invalidation and naming things. (Phil Karlton)

Data-oriented design

Data-oriented design is a program optimization approach motivated by efficient usage of the CPU cache, used mostly in video game development.

The approach is to focus on the data layout, separating and sorting fields according to when they are needed, and to think about transformations of data.

In this context, what's important is how the data resides in memory.

The objective of this paradigm is to improve the performance of the system.

Data-driven programming

Data-driven programming is the idea that you create domain specific languages (DSLs) which are made out of descriptive data. It is a branch of declarative programming.

In this context, what's important is to describe the behaviour of a program in terms of data.

The objective of this paradigm is to increase code clarity and to reduce the risk of bugs related to mistakes in the implementation of the expected behaviour of the program.

Data-oriented programming

In this context, what's important is the representation of data by the program.

The objective of this paradigm is to reduce the complexity of the system.

Summary

Before we conclude, I'd like to mention that there is a draft Wikipedia article about Data-Oriented programming. You are welcome to contribute to the article.

We saw that the three data-related paradigms have different objectives and each of them pursues its objective by focusing on a different aspect of data inside a program.

Paradigm	Objective	Data
Data-oriented design	Increase performance	Data layout
Data-driven programming	Increase clarity	Behaviour described by data
Data-oriented programming	Reduce complexity	Data representation