Forem: Eric Normand

But the World is Mutable

Eric Normand — Mon, 25 Sep 2017 15:47:07 +0000

Immutability is a hard topic to breach. As a programmer used to modeling the world, you might object to immutable data structures. How do you model a changing world? Why would you choose to use immutable data structures when everything in the world is changeable?

Let's do a little thought experiment. Let's look at a nice mutable system: paper and pencil. You can write, erase, and write again. It's very convenient. It lets you correct mistakes. And when you don't need something anymore, you can easily erase it.

Now answer this: would you trust a bank that used pencils to record transactions? It would be easy: whenever you would withdraw money, they would erase the old balance and write the new balance. And if you transferred money from one account to another, they'd erase two balances and write the new ones in. It may sound great, but there's a reason banks don't use pencils: they want to be sure nothing has changed. That sounds like immutability.

This is a bank ledger. Each transaction gets its own line. Always done in pen. It's an example of an append-only data structure. You can answer questions about the past like "How much money was in the account at the close of last Tuesday?" by going up lines until you find the last entry for Tuesday. And you can do that because you never modify existing entries. You only add new entries on blank lines.

This is another example of an append-only data structure in the real world: medical records. Each patient gets a file that everything is added to. You never modify old records. That way, everything is recorded, even the wrong diagnoses (mistakes) of the doctor.

It turns out that traditional information systems that need a high degree reliability create immutable records out of mutable paper. Even though you could in theory scratch out some data and write it again, or white it out, or find some other way to mutate the document, a mark of professionalism in the job is to discipline yourself to adhere to strict append-only behaviors.

Wouldn't it be nice if the machine took care of the discipline for us? Even though RAM and disk are mutable like paper and pen, we can impose a discipline inside of our program. We could rely on the programmer to never accidentally overwrite existing data. But that's just shifting the burden. Instead, we can build in immutability into our data structures and make a paper that cannot be overwritten.

That's how immutable data structures work. All new pieces of information are written to new locations in memory. Only when it is proven that a location is never going to be used again is it reused.

Reliable paper-based systems use immutable data. There was a time when computer memory was expensive and we had to reuse storage, so we couldn't make immutable systems. But RAM is cheap now! We should be using immutable data, just as banks have done for hundreds of years. Ready to join the 13th century?¹

If you're interested in a language with a very cool set of powerful immutable data structures, probably the most cutting edge immutable data structures in any language, you're in luck! Check out LispCast Introduction to Clojure at PurelyFunctional.tv. It's a video course with animations, exercises, and screencasts that teaches you Clojure.

Photo credits: Ledger and Medical Records

The Double-entry method of accounting can trace its history back to 13th century Florence. ↩

Global Mutable State

Eric Normand — Mon, 18 Sep 2017 14:42:42 +0000

One of the biggest problems in software is global mutable state. It makes your code difficult to work with, and once you go down the road, it keeps getting worse. Reducing the amount of global mutable state in your program is one of the best ways to improve the quality of your code, regardless of whether it's procedural or functional.

Definition

Global mutable state has three words, and each is important:

Global means that it's accessible from any other point in your code. This ties all of your code together. You have to reason about the whole program instead of reasoning about a small part, because any other part can touch it.

Mutable means that it can be changed. You'll usually see that anyone who can read the value can also change it. Two reads right next to each other in the code might return different values. Or, worse, the data structures they return themselves are changing, even after a read.

State is harder to define. But it basically means that the value depends on the history of the program. How far into the history? Well, in the worst case (namely, global mutable state), it means the entire history. You have to know everything about how the program was executed, including how threads were interleaved.

When you combine global, mutable, and state, you get a big mess. When people say "it's hard to reason about", what they really mean is "it's got bugs and you can't tell by reading the code".

The nice thing is that you can systematically remove those same three aspects. And you can remove them more or less separately. I like to say that it's possible to program functionally in any language, even the most procedural languages out there. One way to do that is to reduce the amount of global mutable state as close to zero as you can.

Identifying Global Mutable State

Some telltale signs: multiple variables in the global scope (in Clojure: multiple atoms in the toplevel of a namespace), reading and writing to the globals with no clear patterns (or reading from the globals multiple times in a small piece of code). The variable could have changed values between reads.

Cleaning up

It's actually hard to get rid of global mutable state once it's in there. Its usage will spread if it's not tied down. Global mutable state is so useful that it can actually be used for many different purposes. After a while, it's hard to see what the usage patterns are and how you would go about replacing them. But we can tackle each of the naughty aspects in turn.

1) Does the variable need to be global?

Maybe you can rework the code so that an object is passed into
functions instead of being a global variable. That would mean you can create a new instance each time you run the code, which at least guarantees that it is starting from a known value each time and that you are encapsulating the mutation in different executions.

In other words, turn global variables into local variables. The best is local to the function doing the mutation (or smaller scope, if possible). Next best is an instance variable on a local object.

It's very tempting to use globals because they're an easy way for different parts of the code to work together. Here's an example:

var file;                            // the dreaded global variables
var recordCount;

function readFile() {
  file = openFile("input.txt");      // global mutation here
}

function countRecords() {
  recordCount = 0;
  for(var c in file.lines()) {       // global read
    recordCount++;                   // global mutation here
  }
}

function generateOutput() {
  for(var c in file.lines()) {       
    print(c + "," + recordCount);
  }
}

function processFile() {
  readFile();                        // these lines have to be in this order
  countRecords();
  generateOutput();
}

Let's try to make the variables less global using the technique above.

// got rid of the globals
function readFile(state) {                // functions now take the state
  state.file = openFile("input.txt");
}

function countRecords(state) {            // see, the state is now an argument
  var x = 0;                              // use a local here, instead of storing
  for(var c in state.file.lines()) {      //   intermediate values in the global
    x++;
  }
  state.recordCount = x;                  // then assign the state once
}

function generateOutput(state) {          // state as argument, again
  for(var c in state.file.lines()) {
    print(c + "," + state.recordCount);   
  }
}

function processFile() {
  var state = {};                         // the state is now local (still mutable)
  readFile(state);                       
  countRecords(state);                   
  generateOutput(state);
}

The biggest transformation we do is to pass a state object to each of the methods. It is no longer global. Each time we run processFile we will generate a new instance. We start from a known initial state and we know we won't have any contention for that object.

The other transformation we did was to rely more on local variables for accumulating intermediate values. This may seem trivial, but it means that at no point does our state object contain inconsistent data. It either does not contain the data or it's correct.

2) Does it need to be mutable?

Are there functions that read from but don't write to the variable? They could be changed to take the current value as an
argument. Reducing the amount of code that relies on those particular variables is a good thing.

In other words, do as much work as possible using only the arguments and return values of your functions. Isolate the mutation of the variable to a small portion of your code.

Let's apply this technique to code we just modified.

function readFile() {
  return openFile("input.txt");     // instead of mutating state,
}                                   //    just return the value

function countRecords(file) {       // take just the state you need as arguments
  var x = 0;
  for(var c in file.lines()) {
    x++;
  }
  return x;                         // return the value you calculate
}

function generateOutput(file, recordCount) { // take the two values you need
  for(var c in file.lines()) {               //     as arguments
    print(c + "," + recordCount);
  }
}

function processFile() {
  var file = readFile();     // then use local variables
                             //    (initialized but never mutated)
  var recordCount = countRecords(file);
  generateOutput(file, recordCount);
}

We've translated code that wrote to a mutable argument into code that merely returns the value it calculates. Then we use local variables to hold the return values for later. Notice how readFile is doing so little work now (it's just a function call) that maybe we will want to remove it and just call the openFile directly. That is up to you to decide, but it's one of the things I notice a lot when removing mutation: functions become trivial to read and write, and often they are so trivial you will want to inline them.

function countRecords(file) {
  var x = 0;
  for(var c in file.lines()) {
    x++;
  }
  return x;
}

function generateOutput(file, recordCount) {
  for(var c in file.lines()) {
    print(c + "," + recordCount);
  }
}

function processFile() {
  var file = openFile("input.txt"); // we can just inline this one-liner
  var recordCount = countRecords(file);
  generateOutput(file, recordCount);
}

3) Does it need to be state?

Can the algorithms be reworked so that their natural input and outputs (arguments and return values) are used instead of writing to a location? For instance, maybe you're using the variable to count stuff. Instead of the function adding to a variable, maybe it could just return the total count instead.

Programs need state. But do we need to rely on the state to get the right answer? And does our state need to depend on the whole history of the program?

Let's go through step by step in our code, removing state.

function countRecords(file) {
  var x = 0;                    // here's our state
  for(var c in file.lines()) {
    x++;                        // it changes each time through the loop
  }
  return x;
}

The variable x is state. Its value depends on how many times the loop body has executed. Usually, this kind of counting loop is unnecessary because the standard library can already count a
collection.

function countRecords(file) {
  return file.lines().length();  // we prefer not having to deal with the state
}

Wow! There's no state, now. And in fact, it's so short we can just inline it. It's called once in processFile. Let's inline it there.

function processFile() {
  var file = openFile("input.txt");
  var recordCount = file.lines().length(); // inline the one-liner (optional)
  generateOutput(file, recordCount);
}

That's better. But we still have state. It's not terribly much, but let's continue with the exercise. Notice how we rely on the state of recordCount to pass to generateOutput. What's to guarantee that the count we provide isn't different from the count of file? One possible direction to go is to move the recordCount calculation into generateOutput. Why should generateOutput trust someone else when it could just calculate it itself?

function generateOutput(file) { // eliminate an argument that needed to be kept in sync
  var recordCount = file.lines().length(); // calculate it ourselves
  for(var c in file.lines()) {
    print(c + "," + recordCount);
  }
}

function processFile() {  // now our process is two steps
  var file = openFile("input.txt");
  generateOutput(file);
}

And now we don't need that little local variable called file.

function processFile() {
  generateOutput(openFile("input.txt")); // it can be written as one step
}

Conclusion

I've taken this simple example to an extreme. And, yes, this was a trivial example. But my experience with real world code tells me that you see the same kind of improvements when you remove global mutable state in real systems. The code becomes easier to reason about (because you're reasoning locally). It becomes easier to refactor. It becomes easier to eliminate code.

Reducing global mutable state is one of the hallmarks of functional programming. But it's also just good coding. You can (and should) do this kind of refactoring in any programming language or paradigm. If you're interested in going deeper with functional programming, I recommend the PurelyFunctional.tv Newsletter. It's a weekly email about Functional Programming, with a focus on Clojure. I'll also send you some great information about learning Clojure.

Programming Paradigms and the Procedural Paradox

Eric Normand — Wed, 30 Aug 2017 19:08:14 +0000

I'm a collector of perspectives. I think each perspective we have within reach is another option we have to solve problems. We should all learn as many as possible. Each one increases the number and quality of solutions we can create.

Programming paradigms are different perspectives on solving a problem with software. Each of the paradigms is valuable. But they seem so hard to define. People will discuss endlessly what each paradigm means, trying to be inclusive of what they consider important and what they don't. To take an example, we get definitions of functional programming which are satisfying to the definer but not to everyone. And we get people pointing fingers, saying "that's not real object-oriented programming". These discussions are unsatisfying because they rehash the same tired ideas and never reach any firm conclusions.

I'd like to take a broader perspective and try to shed some light on why they are so hard to define. That might help me understand how to define them well.¹

What is a programing paradigm?

paradigm

a philosophical and theoretical framework of a scientific school or discipline within which theories, laws, and generalizations and the experiments performed in support of them are formulated

Merriam-Webster

paradigm

a) a framework containing the basic assumptions, ways of thinking, and methodology that are commonly accepted by members of a scientific community.
b) such a cognitive framework shared by members of any discipline or group: the company’s business paradigm.

Dictionary.com

Most of the time, programming paradigms are described in terms of their features or constraints. I think this is a useful perspective. Languages associated with a paradigm often share many features. For instance, functional languages typically have first-class functions.

But there is a much better way to think of the paradigms that doesn't reduce them to lists of features. Each of the major paradigms is a wholistic approach to solving problems with code. The paradigms are frameworks containing basic assumptions, ways of thinking, and methodology. To define a paradigm in terms of the features of a language (OO is encapsulation, FP is no state) is to ignore the definition of paradigm as a way of thinking. It is a totally mental thing, which is why you can program any paradigm in any language.

I'd like to go through the three predominant paradigms (procedural, object-oriented, and functional) and describe both their features and their wholistic methodologies. Along the way, we'll explore why the definitions of the paradigms are so hard to define.

Procedural Programming

Procedural Programming is a very solid programming paradigm that has proven its viability in industry. It doesn't get as much credit as it deserves these days.

The features

Procedural Programming is characterized by statements which each have an effect. For instance, the effect could be setting the value of a variable or it could be printing a line to the terminal. You often see procedural languages have subroutines (sometimes called functions) which contain other statements. Subroutines allow you to build your own effects from other effects, give them names, and call them like any another statement. You get reuse and abstraction.

The methodology

The Procedural Programming approach to solving problems with software is to treat any solution like a series of steps to be performed. Each step could actually be a complex subtask that includes many smaller steps. I really like this approach to solving problems. It corresponds so well to a very common way of describing a solution: a list of steps to take to arrive at the solution.

It's very much like the grade school exercises of describing in detail how to make a sandwich. It's a very complex procedure if you try to write it out. Let's assume you are only allowed to give instructions in terms of gross body movements like "pick up", "grasp", and "turn your arm". You could painstakingly describe every action that you needed to perform to make a sandwich. In order to make it more tractable, you'd probably want to break down common actions as subroutines and name them appropriately. It's the only way to manage the number of steps.

Many everyday solutions are given in terms of steps to accomplish. Recipes in cookbooks. Furniture assembly instructions. Directions to the library. Any How-To material. It's something we all are very familiar with.

A lot of people associate procedural with global variables and unrestrained mutable state. However, I don't think that's central to the paradigm. It's simply that most procedural languages don't provide any other good ways of managing state. The paradigm itself makes perfect sense and combined with powerful tools, it could be great. A quick example: Communicating Sequential Processes. Adding a channel abstraction to procedural (sequential) code makes it work pretty well for concurrency.

Some good examples of languages that are clear examples of being really well-suited to procedural programming are Python and Basic.

The paradox

Another reason I like Procedural is that the language features map so well to the paradigm. Statements in series (on subsequent lines) mean sequential steps. Subtasks are defined and named with subroutines. Repeating steps are done with loops. And that's basically it. Any other features of the language are just niceties on top.

And this close match between language features and metaphor is actually another feature. Tasks and subtasks match so well with statements and subroutines. You can teach the features and the paradigm in one go!

There is such a correspondence between them that it can feel like there is no paradigm at all. And this is what I'll call the Procedural Paradox. When your language is so well-suited to the paradigm that you can't distinguish between them, you're winning, yet the paradigm is invisible. The Procedural Paradox gets us used to talking about features, not metaphors and thought processes. It becomes difficult to talk about paradigms that aren't as well-matched to the language.

Object Oriented Programming

I really like Object Oriented Programming. I studied it a lot at University and I tinkered with different ways of setting up solutions to problems. I explored Design Patterns.

After I got out of college, though, I started to read more about it. I came across the work of Alan Kay, the inventor of Object Oriented Programming. He talked about Object Oriented Programming in a way that none of my professors had. And I learned that Alan Kay does not consider Java to be Object Oriented. It was the language I used the most in my OO Design classes. That caused a deep rift in my soul that I've tried to mend since. If Java isn't OO, then what is?

Of course the answer is Smalltalk. Apparently Smalltalk was so glamorous that other languages took on some of the features of Smalltalk to call themselves OO. Those languages became what people learned instead of Smalltalk. So now we refer to Java and C++ and others as Object Oriented, even though they're not.

I've watched a lot of talks by Alan Kay. Some several times. I recommend them, though he's a deep thinker and doesn't lay it all out for you. I had to log many hours of Kay's talks to really start to understand what he was talking about. Well worth it, though. I'll make reference to his work because I think he's good at distinguishing the thought process from the features. If you want to get into Alan Kay talks, start with The computer revolution hasn't happened yet. That's the gateway talk.

The features

Many people have tried to decipher what Alan Kay actually meant by the term OOP. Luckily, he pops up on the internet every now and then to answer questions like these. There's an email where he lists the features of OO as he sees it:

Message passing
Encapsulation of local state
Late binding

Late binding is easy to explain. A common compiler trick is to figure out exactly what instructions need to be called so you can optimize. So you do lookups as soon as possible, sometimes as soon as compile time. You might see a compiler building a table of all functions and their names. Then it might even inline those functions if it guesses it's faster code. For example, Java will inline method calls. Late binding means you can't do that. You have to wait till the message is received to look up the method definition, because it could change its meaning at any point.

Encapsulation is the idea that an object can maintain its own state and keep it internally consistent. The only way to read or change the state is by sending a message. Java actually does this as a feature.

Message passing is interesting. A message sent must be received and decoded on the other side. I believe this features is absent in Java and many other OO languages. Methods in Java are little more than functions executed inside of the object's scope. There is no receipt of a message which must be interpreted. That interpretation step is very important because it allows different objects to interpret the message in different ways--not just the standard way objects typically interpret messages by looking them up in a vtable. For instance, it could have a method missing method that does something else. The message has to exist for this to work.

So those are the features. What's the approach to solving problems?

The methodology

The approach to solving problems is a little harder to decipher. We have some clues, like that Java is not it. And I have watched a lot of talks and read everything I could from Kay. It turns out that the email I mentioned before has a lot of clues.

1. Cells/network of computers

When Alan Kay was trying to create a new programming system, he was inspired by cells in a body. The number of cells is huge. The number of faults is huge (cells die or misbehave all the time). Yet the organism is surprisingly resilient (though also not perfect). Cells in our body are self-contained and send chemical signals to each other. It's an interesting metaphor for both the size of the object (cells are tiny) and the scale we can achieve (trillions of cells). Likewise, another metaphor he brings up often is individual computers on a network. Each computer sends and receives messages.

So we're looking at an approach to solving a problem which breaks the problem down into objects that send each other messages. If you wanted to find an analogue in the real world, you'd probably have to look to org charts or other diagrams of communication. For instance, the restaurant guest tells the waiter what she wants to eat. The waiter tells the cook. Then the cook tells the waiter when the food is ready, etc. Each of those steps is a message passed.

From The Early History of Smalltalk:

The last thing you wanted any programmer to do is mess with internal state even if presented figuratively. Instead, the objects should be presented as sites of higher level behaviors more appropriate for use as dynamic components.

2. Getting rid of data

At the time Smalltalk was conceived, much of Computer Science and programming had to do with data structures. That basically meant algorithms for walking pointers and keeping things organized in memory. For instance, you would write a loop that traversed a linked list by following the next pointer. He saw the object as a way to avoid having to organize things so intricately. Objects managed their own, small states and knew how to access them to provide a high-level interface.

However, doing encapsulation right is a commitment not just to abstraction of state, but to eliminate state oriented metaphors from programming.>
-- The Early History of Smalltalk

This idea of avoiding state-oriented metaphors is very reminiscent of the Tell-Don't-Ask principle that we hear about a lot in the Ruby community.

3. Algebras

Alan Kay had a degree in Mathematics. Very briefly, an algebra is a set of elements and the operations on those elements. So an object could belong to many such sets, and so belong to many algebras. Just as a simple example, a String could belong to the algebra of text and also the algebra of lists (of characters).

Kay was trying to provide a means of expressing generic behaviors that objects could participate in. We see an attempt at this with Java-style interfaces. But a more sophisticated version is Ruby's pervasive Duck Typing.

The most obvious example of this kind of OO programming in a language is Smalltalk. But Erlang might be a better fit since it's more cleanly about message passing. It doesn't have the classes, methods, and inheritance--features which people often conflate with OOP.

The problem

I should say I really like the paradigm as espoused in Smalltalk and Alan Kay's work. The torch is carried on, at least partially, by the Ruby community. But even there, it is touch-and-go. Why is that?

I think the reason is the Procedural Paradox. The supporting features of the paradigm do not map well with the assumptions and thought processes of the paradigm. Sure, the features support that kind of thinking, but they don't correspond one-to-one. In procedural programming, the methodology of "procedural abstraction" (breaking a task into subtasks) corresponds so well with "use more subroutines". However, no imperative in the OO methodology corresponds to a simple imperative to use the features. I can't say "use more late binding" and expect more algebras to develop. In order to learn "OO Design", we need books to describe Tell-Don't-Ask, Command-Query-Separation, and many of the Design Patterns which explain how they map down to features of the paradigm.

Alan Kay talks about some of these failings of OO, which were evident way back in 1993, in The Early History of Smalltalk. What was clear was that just borrowing the features without the methodology was not going to get you Object Oriented Programming, though many people still call it that.

Four techniques used together—persistent state, polymorphism, instantiation, and methods-as-goals for the object—account for much of the power. None of these require an "object-oriented language" to be employed—ALGOL 68 can almost be turned to this style—an OOPL merely focuses the designer's mind in a particular fruitful direction.

-- The Early History of Smalltalk

Perhaps aligning the features with the techniques will help programmers do "real OOP" more often and we can dispense with the arguments about OO Design. For instance, maybe we could call methods "goals" and interfaces "algebras".

Sandi Metz, a veteran Smalltalker, is one person carrying the "message passing" torch in the Ruby community. In Nothing is Something, she shows Rubyists how pervasive message passing is in Smalltalk: even conditionals were done with message passing. She's a great teacher and highly respected. However, this talk shows how far non-Smalltalk programmers stray from the paradigm. We'll see that functional does not fare much better.

Not real OOP

Alan Kay himself said that when he used the term Object-Oriented, he didn't have Java or C++ in mind. So how is Java not object-oriented? Obviously, since you could use the message-passing paradigm in any language, that applies to Java, too. But what often happens in Java is that classes are used to merely bundle state and operations on that state into one place. Great! Except then all of the state is exposed through property accessors (also known as getters and setters). You've basically recreated structs with functions. There's no abstraction into algebras.

One of the problems with OOP, in my opinion, is how tenuous this line is between "abstract computer" that I can send messages to and "smart structs" (merely bundling functions with data). On one side of the line, we see the beautiful flourishing of high-productivity, low line-count, highly abstracted systems. On the other, a proliferation of giant classes, all tightly coupled. Sandi Metz presents some simple rules that seem to help in this talk. Fred George talks about this also in The Secret Assumption of Agile.

So what is the paradigm of Java? I don't know if it has one clear, essential paradigm. If I had to pick one, I'd say that it's a top-down modeling paradigm. Remember, we're talking about an approach to solving problems. And my overwhelming experience is that in Java, you are encouraged to model the entities in a problem using classes and their behaviors using methods. I've given a talk about the problems with OOP education called Lies my OO Teacher Told me, and in it I discuss the dangers of trying to model the world this way. The talk goes into more depth, but just briefly: when you're programming a school registration system, do you really want a Student class and a Course class? Do they have a register method? If you take a step back, what you're really simulating is the book the school uses to keep track of which students are in which courses. And that book has clear properties (for instance, it probably has a first-come-first-served discipline for registering for courses).

Inheritance

A lot of people associate OOP with inheritance, a class hierarchy, and is-a relationships. That's probably due to how OOP is taught in school. I don't think this class hierarchy can actually get you any computation without the message passing. It's essentially a static notion, not one that can result in computation. Smalltalk had inheritance, but it was debated within the Smalltalk group itself and was not considered central to the language. I think the excessive focus on a class hierarchy is a legacy of trying to marry the static type system with inheritance.

Functional Programming

Functional Programming is old but it's finally getting a lot of attention in industry. People are really curious about it, there's plenty of hype, and there's plenty of confusion as people try to pin down just what exactly FP is.

The features

The features of Functional Programming are easily apparent:

Immutable data
First-class functions
Lexical closures
Pure functions
...

Not all functional languages have all of these features, just like not all OOP languages have classes. They're just more commonly used in functional languages.

The methodology

The approach to problem solving is less mapped out. There have been many attempts and I will try my hand here. Whereas procedural programming expresses a solution as a series of steps, and OOP expresses a solution as communicating objects, Functional Programming expresses solutions as data, calculations, and effects.

1. Data

Data goes back to the very beginnings of writing where people would mark down how many cattle they traded. They would draw a picture or tie a knot for each head. The number was permanently recorded in the medium--hence we want data to be immutable.

Data plays a similar role in computer programs. You record something, such as user input, or you fetch a stored record, say from the database, and it is passed around and used in calculations. Most programs use data in some way, but it is not explicitly called out in the paradigm like it is in Functional Programming.

2. Calculation

Functional Programming makes a distinction between effectful operations and pure calculation. Calculations take data as input and return data as output. I like to think of them as "thinking" separated from "acting". You can think about what you need in the store as you are shopping. Or you can take some time to calculate what you need before you go to the store.

3. Effects

Effects are the reasons we run programs. We run programs for their effects, generally. That means we want to see something on the screen or send an email or do something in the world.

By separating out these three ideas, we decompose a problem. For instance, serving up a web page may look something like this:

Query DB => Data => Render HTML => HTML => Send response
  Effect    Data    Calculation    Data      Effect

Why separate things along these boundaries? Well, it's because of how they compose. For instance, two effects can be composed either in series (one after the other) or in parallel (same time), but the composed thing is a new effect. Let's write this very roughly as Effect + Effect => Effect.

Two calculations can be composed in a similar way. I can take two functions and chain them in series (function (x) { return f(g(x)); }). Or I can run them both and return both answers (function (x) { return [f(x), g(x)]; }). In either case, I get a new calculation. Calculation + Calculation => Calculation.

And, just to be complete, there are many ways to combine data (hashmaps, lists, tuples, etc.), but I always get data out. Data + Data => Data.

The interesting thing is when we combine two things from different groups. I can compose a calculation, like uppercasing a string, with an effect, like printing. function (s) { println(uppercase(s)); }. In the FP paradigm, that would be seen as a new effect. So Calculation + Effect => Effect.

Similarly, I could combine data with a calculation. This makes what's called a closure, which is just another kind of calculation. Data + Calculation => Calculation.

If let's say I make a new thing that always prints (Effect) "Hello" (Data) (e.g., function() { println("Hello"); }), that new thing would be an Effect, Data + Effect => Effect.

These are just some examples of how things can compose.

But then it really gets fun ... because you can make everything first class. You can have effects in your data, calculations as arguments to other calculations, etc. A function is just data until you call it. And applying a function to complex data is akin to interpreting code in a language. Remember these ideas are all in your mind.

And of course, if you can compose, you can decompose. Making code more functional usually involves separating out parts of your code along these three boundaries. Take an Effect and pull out some Calculation from it. Pull out some Data from your Calculation, etc.

State (a thing that varies with time) may deserve a place among those three. But as some people have said, you can always create State from Effects, so we'll just bundle it in there for now.

The problems

One of the problems is that many functional languages lack a strict correspondence between language features and these three key concepts. For instance, in Clojure there really is no distinction between Calculations and Effects. We use fns for both. Haskell does it the best with its IO type which marks a clear and useful line around what it considers an Effect. However, the distinction between calculation and effect is made for you arbitrarily. I do believe that the correspondence is slightly better than in OOP. In general, making something "more functional" means moving more of your code from effects into calculations and data.

Another problem is that I can't find anyone else who breaks things down this way. People often talk about a lack of state or a lack of side-effects, but this is clearly not what defines FP. Functional programs are run for their side effects just like programs from other paradigms. And the fact that these things "problematic" features are called out goes to show how important they are to the approach. Some people talk about programming with first-class functions, which hints at the idea of combining calculations, but that leaves out the other two ideas.

Notice also that the FP perspective fruitfully distinguishes calculations from effects, but that doesn't mean that distinction isn't also fruitful in the other paradigms. It's more that it's not part of the definition. Applying the FP analysis to the other paradigms can definitely be valuable and another example of the value of multiple perspectives.

Discussion

It may be a fool's errand to even attempt to classify the paradigms. Do they even exist? I think they exist in our minds, which is where much of our work happens. We need these kinds of classification structures to help us do our work and talk about it.

The nice thing about these paradigms is that if you notice, none of them are pure. Sure, computers sending messages to each other is a great metaphor, but you still have to program each of those computers somehow. In Erlang, that programming tends to be done in a functional way, but in Ruby, it's procedural. And when programming our effects in FP, they tend to be done in a procedural way (in Haskell using the do notation or in Clojure just by sequencing effects). This is definitely not an either-or thing. It's more like complementary approaches. You can see them happening at different levels of architecture.

However, having said that, there is the conflict between Alan Kay's goal of "getting rid of data" and FP's enthusiastic use of data. I have to say this is a big discussion in itself. To touch on it briefly, look at Smalltalk's insistence that "everything is an object". Even numbers are message receivers. Contrast this with Erlang which distinguishes data (numbers, arrays, tuples, hashmaps, etc) and processes (which receive messages).

In a way, they're both getting rid of some of the worst parts of data--that is, the arbitrary use of blocks of bytes to represent data structures and the corresponding code to walk those data structures. Erlang made a practical decision to not allow new types of data and so could guarantee that they could all be efficiently handled by the VM and you don't get the chance to make arbitrary data structures in memory. On the other hand, Smalltalk's goal of "getting rid of data" (data being inert encodings of facts that need to be interpreted by the receiver) was never realized and is still an ongoing thought experiment in the mind of Alan Kay.

Conclusions

We need to distinguish between the features and the methodologies. When we argue, we need to be aware that it doesn't make much sense to argue about which features define a paradigm. The paradigm is mental and often hard to express, which is why we talk about "real OOP" and doing "FP in my language".

There is perhaps hope to break the curse of the Procedural Paradox. Can we make the languages a better expression of the paradigm? Or, more generally, what does starting with the paradigm tell us about how to design languages? There is no doubt that all of these paradigms are useful and that they are not mutually exclusive. They are simply different perspectives on the same thing--solving problems with code. A language and its features should be more clearly framed as support for expressing our thinking, regardless of paradigm.

If you're into exploring this kind of thing, you should subscribe to the free PurelyFunctional.tv Newsletter. It's ten links to deep stuff that I'm reading and watching, mostly about Clojure and Functional Programming.

For another perspective on the paradigms themselves, please read Programming Paradigms and Expressive Power. ↩

Reasoning About Code

Eric Normand — Mon, 21 Aug 2017 16:06:50 +0000

A lot of people talk about "reasoning about code". I certainly do. It's something that I don't think I ever heard when I was an OO programmer. What does it mean?

It's a tough question, because it's not one of those technical terms with a technical definition. It's just something people say when they are talking about the benefits of functional programming. But since I say it, too, I might as well give my understanding of the term.

To me, "reasoning about code" is all about the limitations of the human mind. If we were hyper geniuses, we could read any amount of code and just understand it. But we're not and we can't. As programs get bigger, we can't help but lose track of what's happening in a program.

People are used to interacting with the real world, where effects tend to be local. For instance, if I'm locked in my house, a person ten miles away cannot attack me. When we walk down the street at night, we know there are people who might hurt us somewhere in the world. But what we're concerned about is if they are close by. Instead of starting with the locations of all attackers and calculating the probability of each of them being able to harm us, we look around where we are and we evaluate the people we see. Humans think locally because that's where the action is.

It's this sense of locality that we find in functional programming languages. When you look at functional code, several things help you localize yourself:

Scopes are localizing. Definitions cannot "leak in" from elsewhere.
Pure functions are localizing. Pure functions will act the same regardless of when they are called or how many times they are called.
Immutable values are localizing. No need to worry about other parts of the code modifying it.
Isolated and consolidated side-effects. At least they're happening in the same place.

Since everything is relatively local, you have less to read and understand to be able to reason about what it's going to do. There are some common things that are non-localizing in programming languages.

Global mutable state. Anything, in any part of the code, can change it at any time. The opposite of local.
Scope leak. Variables can often be modified outside of their scope, or mutations in one scope are visible outside the scope.
Mutable objects. You are referencing an object that is changing out from under you.
Side effects. Well, once you send a message across the wire, or receive on, it's not so local.

So what should we do? Push state down from global to local. Make it as local as possible. Factor out pure functions from your code, which should isolate state change. Use immutable values whenever possible. Isolate and consolidate your side effects (so at least you know where they are happening).

Conclusions

People talk about "reasoning about code" a lot and it's not clear that it's meaningful. But I do use the term and when I do I mean "things are more local so I can keep them in my head". It's a notion that serves me well. For instance, it acts like a code smell. If I'm not able to keep something in my head, it's time to make it more functional.

If you like this topic, or you'd like to get into functional programming, you might like to try the PurelyFunctional.tv Newsletter. It's a weekly email about Clojure, Functional Programming, and how they relate to the history of technology.

Immutable Paper

Eric Normand — Mon, 14 Aug 2017 16:56:40 +0000

I've said before that immutable data is easier to reason about because it's more like the real world. That seems counterintuitive because the world is mutable. I can move things around, erase a chalkboard, and even change my name. How, then, is immutable data more like the real world than mutable data?

Here are two stories that will explain my point. Imagine I take a clean sheet of looseleaf paper, the kind with blue lines. I take out a nice, fat marker, and write the word banana in big letters. Then I fold it up and hand it to Susan. "This is very important", I say, "so listen carefully. Put this in your pocket and don't let anyone touch it until you get home. Then you can take it out."

Susan doesn't understand why, but she puts it in her pocket. She makes sure nobody goes in her pocket all day. When she gets home, she finds the paper folded there like she remembers it. She opens it up and reads it.

What will be written on it? (This is not a trick question.)

Everyone will know the answer. It's the word banana. But how did you know? And how are you so sure? In fact, the world fulfills our expectations so many times, all day long, and we don't even think about it. We've learned these tendencies of the universe since we were babies. This kind of reasoning may even be baked into the structure of our nervous systems. Our understanding of the world is intuitive. You might say "we can reason about it".

Now, let me tell you another story. Let's say I construct an object P (for Paper), and set a property on it to the string "banana". I pass the object to a method on a different object S (for Susan), which stores that object in a private property. The object S keeps that property protected, never letting anything write to it, and it makes sure that no method is ever called that can write to it. After a while, object S prints out the contents of P.

What do you expect to be stored in P? (This also is not a trick question.)

The answer is "I don't know". It could be "banana". It could be something else if the object P was modified by something else that had a reference to it. We can't know what it will say because it's mutable and that lets in magical effects like "action at a distance". We can no longer reason about it.

Many objects in our universe are constantly changing on their own. The locations of moving cars, the contents of a box of uranium (through decay), the brightness of a candle. Each time we look at it, we expect it to be different. But so, too, are many objects basically static unless acted upon by something else. A house, a glass of water, a piece of paper. The object persists virtually unchanged in any way I would care about. And this is the part of the world that immutable objects model better than mutable objects.

The thing is, once you share two references to the same mutable object, you're way outside the world we live in and know. Two references to the same object would be like two people having the same paper in their pockets at the same time. You could do shared-nothing like Erlang does (everything is copied as it passes between actors), but Erlang also chooses to make things immutable. You could also always copy objects before you store them. This practice requires tough discipline and also undoes any efficiency advantage you gain by using mutable data in the first place.

Mutable objects are important. They can model very common and useful things. For instance, you might model a filing cabinet as a mutable object because you can take stuff out and put stuff in. But you will want to model the papers inside as immutable because they can't change while they're inside. Having both is what makes it work.

Clojure makes it very easy to separate the two. Data structures like lists, vectors, hash maps, and sets are all immutable in Clojure. And Clojure has a toolbox of mutable things, too: atoms, refs, agents, and vars. These are good for modeling the current state of something shared and changing.

Well, I hope these stories explain why immutable objects do act like real world objects in many useful cases, and how this is a key part of why people claim that Clojure code is easier to reason about. If you like this idea and want to explore it more, check out The PurelyFunctional.tv Newsletter. It's a weekly newsletter for

The Joy of Programming to Learn

Eric Normand — Fri, 11 Aug 2017 17:54:22 +0000

What is your main motivation for programming?

Back in the 90s I was reading something online (I don't remember where) that was saying the consensus was that it's the joy of absolute control of the machine. Being an impressionable youth, I felt alienated. This was certainly not my main motivation.

Now, don't get me wrong, I can enjoy control over an inanimate object like anybody. Programming in C gives me that feeling. But there is something else that drives me to program. It wasn't until about nine years ago that I started finding others I could resonate with and further understand my own joy in programming.

I love ideas. I love the history of ideas. I love how ideas are shaped by your environment. I love how fun it is to explore bad ideas. I love how ideas can be communicated to another person through time and space. And I love that some ideas have power.

It's one of the things I do in my newsletter and on my blog -- I develop ideas. I've developed a lot of wrong ideas and a few right ones. But it's through the process of developing them that their contours and characteristics are known. Far from being an expression of my beliefs, the essays are an expression of that process and the love of ideas in themselves.

Programming, to me, is a way to develop ideas. You set up a system with code and you can then empirically determine its properties. The thrill of programming, for me, is found more in the exploration of ideas than in the joy of controlling machines. I have written neural networks to learn about neuroscience, genetic algorithms to learn about evolution, and other simulations to learn about the thing I'm simulating.

About ten years ago I discovered Alan Kay, specifically The Computer Revolution Hasn't Happened Yet and his work at VPRI. I found someone who had explored programming as a medium of ideas. He was interested in understanding how computers could be made easier so that anybody, including children, could develop, explore, and communicate ideas -- perhaps ideas that would simply be impossible to explore otherwise.

I read Seymour Pappert and his ideas of the computer as a tool to explore difficult ideas. Really, go read Mindstorms. Nicholas Negroponte paraphrased Pappert's vision best: "Programming is like thinking about thinking. And debugging is a close approximation of learning about learning." When you program, you translate your thoughts into executable form. Debugging your program is close to debugging your thoughts.

The person carrying this torch now is Bret Victor. He's obviously going deep on these ideas and seems to be dedicating his life to them. What's more amazing than that is his ability to communicate the ideas. Put Media for Thinking the Unthinkable on your must-watch list. If you don't feel a little sad for what might have been, then watch The Future of Programming. He takes Alan Kay's ideas and makes them super-accessible.

I don't doubt that there are more people like them. They are using the computer as a medium in the Marshall McLuhan sense. And while computers are pervading our environment more and more, the power for them to enable new kinds of learning and thinking goes largely untapped.

Lisps and other highly expressive languages help a lot. They let you express the problem much faster -- without dealing with much of the minutiae of controlling the computer. I've always been on the lookout for a system I could use to quickly implement the ideas in papers. It's crazy how long it can take, even when the algorithm is written right there for you! The best languages, Clojure included, are still nowhere near where we could imagine them being.

I'm sending this out, like a beacon, to attract others who enjoy programming for the immense potential it has for helping us learn. Are you one of them? Why do you program? Comment below and let me know :)

BTW, if you're interested in the big ideas behind why we program, you should check out the PurelyFunctional.tv Newsletter. It's a weekly romp through the history, present, and future of Clojure and functional programming.

What do I have to learn to be hirable in Clojure?

Eric Normand — Thu, 10 Aug 2017 17:39:55 +0000

From a reader:

I'm currently working as a software developer in Ruby but I would love to work in Clojure, though I'm still a beginner. Unfortunately, I don't think my company would switch to Clojure any time soon. And I also don't have much free time to work on Clojure. What do I have to learn to be hirable in Clojure?

This is kind of a trick question. The reason it's a trick question is technically you need zero Clojure knowledge to get a job in Clojure. I know this because I just met someone who was hired right out of a Rails internship into an existing Clojure codebase.¹ He had absolutely no Clojure experience.

I talk to a lot of people who work in Clojure and are hiring for Clojure. There's two broad categories of experience requirements for those jobs.

1. They want experienced Clojure programmers.

You'll find jobs like this for a couple of reasons. Sometimes they want to replace an experienced programmer who is leaving. And sometimes they're just starting up and want to hit the ground running. I have also seen it where the CTO likes Clojure but wants someone better than them because Clojure is new ground.

For these jobs, they're really looking for someone with a solid Clojure skills. But there's a secret I'll reveal at the end of this article.

2. There is no Clojure experience required.

This category is more interesting. You'll often hear them say they're "looking for good people". They know it will be hard to find experienced Clojure programmers.

For these jobs they are willing to invest time and money making you a Clojure programmer. You will be paid to learn Clojure, often working with mentors.

Clojure experience will help, of course. I can only imagine the second type gets way more applications, so you'll want to stand out. But still, you don't need Clojure experience. Here are some things that will make you stand out. These are arranged roughly in order from the most powerful to the least.

1. Clojure experience.

"Finished"² projects on Github.

These don't have to take long. They don't need to be complicated. Just show that you can make things happen with Clojure.

Bonus points for

2. Other Lisps

At the very least, this shows that you know what you're getting into. Even a college course or some completed exercises in SICP would help.

Bonus points for

Macros
Solving interesting problems (with code on GitHub!)

3. Other Functional Programming experience

Haskell, Erlang, Scala -- heck, even functional JavaScript or Java 8 streams will be impressive.

Bonus points for

Knowing your 3 functional tools (map, filter, reduce)
Using immutable data structures to model a problem

4. JVM experience

Clojure runs on the JVM, and there's just a lot of little details about how objects work, how the JVM is deployed, and all of that stuff that comes in handy when programming Clojure. It could help you stand out a little.

Bonus points for

Deploying JVM applications
Using JVM languages besides Java
Programming with Java 8 streams
Configuring the JVM

5. JavaScript experience

If you're going to use ClojureScript at this job, knowledge of the browser, the DOM API, and all of the issues with JavaScript deployment will be helpful.

Bonus points for

Using other compile-to-js languages
Building React applications (especialy with Redux)
Using Immutable.js or Mori

6. Multiple languages

If you're going to be learning a new language on the job, show that you can actually do that. The best candidates will have at least 4 languages under their belts with a variety of paradigms.

Bonus points for

Two or more paradigms (OOP, FP, procedural, logic)
Two or more syntax families (C vs Python vs Ruby vs Haskell)

After all this, the number 1 reason people don't get hired is they don't apply to enough jobs. Get your resume together, trudge through your work/school/side project memories and find things from the list above to highlight. Then apply to lots of jobs, bring those things up in the cover letter and interview.

The little secret that I hinted at before is that even jobs that are looking for experienced Clojure programmers often will hire smart people who are excited to learn. Requirements are just a suggestion. So go apply!

Here are good sources of Clojure jobs.

The only other thing is that getting jobs is also still about who you know. There's a lot to say on that, and you've got more of an advantage than you probably know.

Seriously, I never would have thought there would be multiple companies hiring for Clojure in New Orleans. They all happened entirely without me. ↩
If you're like me, projects are never finished. But a "finished" project accomplishes some goal. It's useful and it works. ↩

Can I do FP in my language?

Eric Normand — Mon, 07 Aug 2017 19:40:21 +0000

A very common question I get from experienced programmers is "What does an FP language give me that I don't already have? Can't I just do FP in my language?" This is a great question and I hope I have an equally great answer.

I really sympathize with this question. JavaScript has map, filter, and reduce. Java 8 has the stream library. It seems like every language is adding more and more functional features, including immutable data structures, first-class functions, function composition, destructuring, and sophisticated type systems.

And I agree: You can do Functional Programming in any language.

Functional Programming is just another paradigm. So just like you can do Procedural Programming in any language, or you can do Object Oriented Programming in any language, or Logic Programming in any language, you can do Functional Programming in any language.

But wait. That answers the question, but does it answer the deeper assumptions behind the question?

So let me ask you this: can I do OOP in C? According to my statement above, the answer is "yes". In fact, I've built small object systems in C before. So why would I need an OO language if I can do it in C? I feel like my object system gave me all of the advantages I needed to do OOP. But here's the thing: could I have programmed OOP in C without already knowing OOP? I don't think so.

These paradigms are called paradigms because they are wholistic perspectives about how to approach problems. OOP breaks problems into objects that communicate with messages. FP breaks problems down into data that represents the state of the process being modeled. Procedural breaks a problem into a series of steps that can solve the problem. Each paradigm has a different approach to solving problems. By "doing FP in Java" without already mastering FP, you're implying that FP isn't a paradigm, it's just a set of features (immutable data, pure functions, function composition, etc).

I've seen it many times. People say they're doing FP in JavaScript. What they mean is they sprinkle reduce and map around and they use some pure functions. There's a lot of value in that. But otherwise their code is procedural. They have not learned the paradigm, only the features.

The value of a language that strongly enforces a paradigm is that it immerses you in that approach to solving problems. The more it enforces the paradigm, the more you are forced to attempt solutions using that paradigm. The more functional your language, the more training you get in approaching problems in a functional way. The same goes for OOP and Procedural. In essence, you get more practice because you can't rely on the other paradigms you already know.

So let me say this, more controversial statement: outside of languages labeled functional (Clojure, Haskell, Elm, Scala, Erlang, etc.), I don't see much depth in the functional programming being done. Unless they're already familiar with functional programming by working for a long time in a functional language, programmers are borrowing only little bits of the FP paradigm and missing the best part, which is the change in perspective on approaching problems.

That's not to say it's not possible to learn real FP in JavaScript. But it will take lots of discipline and help. The language won't guide you. You'll need to guide yourself or find someone who will guide you.

Conclusions

All of the paradigms are great because a "point of view is worth 80 IQ points". You can do any paradigm in any language. But it is very hard to learn a paradigm without immersing yourself. And it's hard to immerse yourself in a paradigm without using a language that strongly encourages the paradigm. Until you do that, you might be borrowing features of the paradigm, but you're not approaching the problem in a new way. In short, Functional Programming is not a set of features. If you want to learn it, jump in the deep end and be immersed.

If you're ready to take a dive, please check out PurelyFunctional.tv where I teach Clojure and functional programming.

Rock on!
Eric

When in doubt, refactor at the bottom

Eric Normand — Mon, 10 Jul 2017 17:12:53 +0000

I was recently in a discussion on JavaScript Jabber. Aimee Knight had been reading my articles about abstraction and asked a really poignant question, which I will paraphrase:

When do you think it's okay to extract out an abstraction?

I wasn't really prepared to answer this question. I was expecting to talk about ClojureScript. I gave the question a shot, but of course, after the recording I had a better idea of what to say.

One way to refactor code is to look for repetition. You look for blocks of code that are similar, then extract the common parts into a function. People justify this type of refactoring as supporting the principle of Don't Repeat Yourself. They call it DRYing up your code.

DRYing up your code has become so common that now there is advice against doing it too much. Sandi Metz's advice is very popular:

Duplication is far cheaper than the wrong abstraction.

Her article explains the problems very well. Go read it.

The situation goes like this: you see ten lines of code that are almost the same as ten lines of code somewhere else. They're doing something similar and you think you could extract out the common parts into a function and make the uncommon parts into parameters to the function. You do so, your code gets shorter, and you're happy.

Metz goes on to talk about the follow-on effects, which are that someone wants to reuse that abstraction but needs those ten lines to be different in just one more way, so they add another parameter. Parameters aggregate over time until the code is a mess.

People have different ways to mitigate this. Some people will say "wait until the code is repeated three times before you extract it". It is much more likely to be used a fourth time, and you're more likely to capture all of the parameters the first time.

This method does count as an abstraction since you are naming a function. The name, of course, is of vital importance. Many people say that if you can't think of a good name for your abstraction, it's probably not a thing. I'm a fan of that, but naming is hard. Sometimes I can't think of a good name when it really is a thing.

But there's a deeper problem. I actually think ten lines of code is too big for an abstraction (Metz agrees). Ten lines of code that happen to be repeated are so unlikely to have any interesting properties, including being bug-free, reusable, and composable, that you should be very skeptical of being able to factor the whole thing out as common. However, don't despair! You can more often than not find lots of one-, two-, or three-line bits of code that are 1.) easy to name and 2.) reusable. If you break enough of those out, you may start to see some underlying structure to those repeated ten lines as they become replaced by your new, small abstractions.

When people ask me "how do I refactor this?", that's my first response. Start naming the small bits, even if they're not repeated. Names give meaning to your code. It organizes your code and your thinking. And even if the named bit is never reused, it's still valuable to have a name attached. If you get some reuse out of it, that's gravy.

This process is a purely mechanical, almost statistical way of finding reusability. You're betting that smaller abstractions are more likely to be reusable than big ones. It's a good bet to make,. However, refactoring cannot fix code that's broken. It can only improve the quality of the code, not its underlying meaning.

Sometimes that's all you want to do. You've got some existing code. It works. You just want to clean it up. That's fine. It's like making your desk neater. You can stack up the papers more neatly, arrange the stapler and pencil holder. Throw away the old coke cans. You can do all of that without really knowing what the work is that is done at the desk.

But sometimes you need to look at the meaning of the code, and really do some work at that level. If you want to file those papers away, you have to know where to file them. You have to know what the paper is about and why you have it. Do we need this stapler? Maybe it's not used in this line of work? Maybe it needs to be a longer stapler. We need to know much more about the meaning of that stapler in the work being done.

At this point, simple metrics about how many times something is repeated or how many lines of code are extracted won't cut it. In fact, simply extracting existing code won't cut it at all. We need to go deeper into what we're trying to do and find the structure there--not in our code itself. When you're at that point, there isn't much advice. Here is what I've found/created.

Domain-Driven Design is a methodology for developing code alongside domain experts to really get into the "what and why" of your abstractions.
Denotational Design is a method for thinking deeply about abstractions using Category Theory.
Building Composable Abstractions is my attempt to elaborate a process for designing abstractions that doesn't require complex math. It relies heavily on your intuition of physical systems and delays writing code as long as possible.

I appreciate these methods because they force us to revisit the domain in deep ways.

Conclusions

Repetition in our code can be a sign that there is an abstraction lurking in our code, ready to be extracted. Although the sign may be clear, we can't always be sure how best to extract it. If you're unsure, don't extract it. The longer the abstraction is, the more likely it won't be reusable. But every ten-line bit of repeated code has nine two-line bits and eight three-line bits. There's probably something there to extract. Start there, with smaller abstractions. Start refactoring at the bottom! Finally, there are limits to refactoring. Sometimes you need to revisit the meaning of your abstractions, not just the current code.

If you're interested in the big ideas in Computer Science and the philosophical questions of programming, you should check out the PurelyFunctional.tv Newsletter. It's a weekly romp through the theory and practice of Clojure and functional programming.

What is an abstraction?

Eric Normand — Tue, 04 Jul 2017 13:38:54 +0000

For a term we use so much in our field, there are very few definitions of abstraction. And when I gave my talk called Building Composable Abstractions, that was a persistent question: what did I mean by abstraction? I'm going to be talking about abstractions a lot, both on this site, and in verbal discussions. I'd like to know what I'm talking about. Instead of searching for a precise definition, I'd like to expand on the background ideas that make it so difficult to define.

There are two uses of abstraction that I think will serve well to begin this discussion. The first is from one of my favorite programming books, Structure and Interpretation of Computer Programs by Abelson and Sussman (Section 1.1).

means of abstraction, by which compound elements can be named and manipulated as units.

In their "definition", abstraction is about naming. Naming is a funny thing. It's used for identity--as in the name of a person--and also to impart a meaning--as in to name an idea. We have a tendency as humans to come up with new terms. They are new names for perhaps new meanings. It is part of our natural linguistic abilities. When we program, we do this all the time. Whenever we create a variable or name a function, we're inventing a new term and assigning it a meaning.

Its meaning, in our program, is its behavior. What does this thing do? When should I use it? How do I use it? A function calculates a return value based on its arguments. You use the function by calling it or passing it as an argument to something that will call it. In Clojure, you call a function by putting its name in the first position in parens. In Clojure, functions can also have effects.

Programming language theorists call enumerating all of this meaning the semantics of the language. But it's just another word for meaning. Giving a descriptive name to a thing is all about giving it a clear meaning.

How are Clojure functions implemented by the compiler? We mostly don't care. We write functions and call them without much regard for the things they compile to. Insofar as we can ignore those implementation details, we call the abstraction robust. When that implementation becomes important and we can't trust the abstraction to work as it is intended, we call the abstraction leaky. A robust abstraction is able to hide the details from us and give us a new basis on which to base other abstractions.

That brings me to the second "definition", this one by Edsger Dijkstra.

The purpose of abstraction is not to be vague, but to create a new semantic level in which one can be absolutely precise.

Dijkstra's quote goes right to the matter of its purpose, which we have yet to go into. We want to create a new semantic level (meaning again) where we can be precise. My reading of the word precise is that we need to be able to say exactly what we mean and no more. We want our compound elements to be exactly suited to their purpose.

I believe this is the hard part of programming that we always talk about. We are simultaneously inventing a new purpose and the thing which is supposed to be suited to it. Then we have to name it to make that purpose clear. Three things we're doing simultaneously. That's a lot of degrees of freedom that can lead us to an imprecise abstraction.

There is a lot of distrust of abstraction in our industry, and I think rightly so. We have been burned time and again by abstraction for abstraction's sake and abstractions that hide problems. These are abstractions that don't fulfill their purpose and should be distrusted. But we should not distrust all abstractions. Part of our job is to learn what constitutes a good one.

Here's a great example of the distrust of abstractions. Joel Spolsky, internet entrepreneur, coined an aphorism he called "The Law of Leaky Abstractions":

All non-trivial abstractions, to some degree, are leaky.

He gives the example of TCP: the abstraction is "make a TCP connection and send data reliably, despite lost packets and other vagaries of networks". It's a great abstraction except it has a leak: what if you pull the network cable? Nothing will get through! According to the law, the mechanism will always "leak" through eventually. It reminds me that metaphors can only be stretched so far and that there will be some true things we cannot prove.

I think the law has a lot of truth in it, but his examples are a bit of a strawman. Why? Well, sockets handle a lot for you, and they also have well-defined errors. If the connection is severed, your socket will raise an error. So the socket abstraction is not actually hiding that network cable problem from you. On the contrary, it's building it into its fabric. Perhaps the real problem is that errors can so easily be ignored in our programming language. So I like to think about what I call an "Inverse Spolksy's Law":

All too-trivial abstractions, to some degree, are leaky.

The idea is that the abstractions that are leaky are the ones that are not precise in the way Dijkstra specified. They're hiding something they really can't hide. To make it more precise, you need to hoist more into the higher abstraction level than you would ideally want to. And this is the number one sin I find in abstractions: they hide too much.

Abstraction is something we also see in Algebra. We give a value a name, though we don't know what that value might be. We call these variables in Algebra. We can manipulate these names like values and arrive at sensible answers. This shows a very beautiful relationship between mechanical symbolic manipulation, meaning representation, and mechanical calculation. Abstractions have laws in themselves. Hence I can manipulate a program algebraically and talk about its properties, all without running it. The abstraction becomes a thing to talk about.

Something in there points to the roots of intelligence, and many books have been written about this. The reason it is so hard to talk about is that we don't have very good introspection into how we think. We're still understanding it after trying for thousands of years. We're better at doing it than knowing what we're doing.

The thing I love about programming is that we deal so directly in meaning, like an artist or a philosopher does. We know that, in the end, we are building a mechanism to control electron flow in our computers. Yet we work in the world of ideas. My brain wants nothing more than to escape from the mundanity of logic gates and build sculptures of thought.

Conclusions

We deal in abstractions every day as programmers. We're either using them, creating them, or debugging them. Abstractions are a natural extension of our linguistic abilities. They let us name concepts so that we can use them to form bigger ideas. Much of language design and industry programming books are about how exactly to make these abstractions. How do we go about making these things? Are there better and worse ways? These are some of the things we must understand better as software takes over more and more of our lives.

The Bootstrapping Mindset

Eric Normand — Thu, 16 Mar 2017 22:25:14 +0000

I'm on a campaign to find a solution to the web:

// Detect dark theme var iframe = document.getElementById('tweet-824717450428157953-529'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=824717450428157953&theme=dark" }

One of the great things about very robust and powerful abstractions is that they can give you tremendous leverage. The leverage can be so great that you can build something much greater than the sum of its parts. However, I worry that the web is an end to this kind of abstraction. I'd like to explore why that is and what we can do about it.

Bootstrapping

In the 19th Century, Charles Sanders Peirce, the father of Pragmatism and Semiotics, was struggling with some very deep questions in the philosophy of mind. Two of those questions were: How can a flawed/limited mind make itself smarter? And how can a spoken language be developed that refers to abstract concepts when the language doesn't have words for it yet??

Peirce's answer to both of these questions was that we can bootstrap. For example, if we have basic literacy, we can read books just beyond our current level and pick up new vocabulary and concepts. We use those new vocabulary to read more books, ad infinitum. Similarly, given enough concrete icons and indices (which are easily learned), we can define abstract concepts with them and assign arbitrary symbols. Thus we can invent new words, explain them to others, and use them to speak abstractly.

We start with a limited set of capabilities and, building with those, construct an expanded set of capabilities. I call this the Bootstrapping Mindset. One of the reasons I like Lisp is that it is a manifestation of the Bootstrapping Mindset. It's not unique to Lisp, but that's where I first saw its application in programming.

To summarize the last linked article briefly, Lisp starts with a few primitives easily defined in machine code. You construct an interpreter using those primitives. And now you have an executable lambda calculus. The leverage from that process is amazing.

Let me put it another way. Let's say you could write some algorithm in 10,000 lines of machine code. Great! You could also write a small Lisp implementation in 1,000 lines of machine code. Then you can write your algorithm in 10 lines of Lisp. Given a big enough project, you should build your own tools. Investment in tools will pay for itself many times over.

You also see this phenomenon in FORTH (build a small set of words and a way to define new words) and Smalltalk (build a small set of message interpreters and a way to define new interpreters).

In all of these examples, we have a "vocabulary" that is expanded by using the vocabulary itself. A great example and explanation is Growing a Language. Guy Steele explores the idea of how we can make a language that is not limited by our current understanding. The future potential of the language is limitless.

To my mind, Turing Machines are bootstrapping engines. Once you've got Turing-completeness, you can interpret any language. The differentiator is expressivity. Which bootstrapping steps get you to where your problem is trivial?

In all of these examples, we find a huge leap in expressiveness. We want to build something powerful enough and robust enough to very rarely (if ever) deal with the lower levels. I can write Clojure code all day and never think about the JVM bytecode, let alone the machine code that is actually running. When was the last time you debugged your code's NAND gates?

Here are my two requirements for bootstrapping. I call them:

Leverage: Quantum leap in expressivity.
Robustness: Rarely deal with the lower level.

If you don't have these two things, it might be very useful, but it's not bootstrapping.

// Detect dark theme var iframe = document.getElementById('tweet-543424657139851264-677'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=543424657139851264&theme=dark" }

The Web

So here's my original concern: can we bootstrap on top of the web? I believe we desperately need it. We are spending billions of dollars each year on the silliest problems of our own invention. And I'm tired of the tedium of working at such a low level. Yes, the web is a great economic force for good in the world, but the amount of crap work it generates is staggering.

Let me give an example of the "crap work" I'm talking about. We have at our disposal very powerful graphic design tools like Adobe Illustrator and Sketch. We can draw freely and make things as beautiful as humans can. Even if we limit our work to only what is possible on the web, we then manually compile that design into angle brackets. It is a specialized skill, with conferences, books, trade magazines, and training, to be able to turn a design into something that your browser can use. I respect the people who do it, because it's very hard. But I cannot respect the need for people to waste their time learning the magic incantations that make your design work in browsers. The fact that we do not see a great tool that outputs HTML + CSS directly is a bad sign.

Let me dig into this even more, because I don't want to give the wrong impression. Making stuff work in browsers is important. But actually getting it to work is way more work than discovering how it should work, which is the actual work of design. We can't focus on what's important: turning pixels into an expression of our humanity. Why not? Because we can't get it to render correctly. We're talking about unimportant minutiae like the best way to select an element in CSS. Should you add a new class? Or use nested selectors? There is no good answer because they all suck, and so we fight over it and call it "philosophy". Or we wait for committees to add yet more stuff to the standards which still doesn't solve the problem.

Let me put it another way: people have written custom layout engines in fewer lines of code than your average PHP page. We lob wrapper divs and CSS rules in our fight against the browser.

I'm not complaining that writing HTML, CSS, and JavaScript is a drag. Machine code can be a drag, too. But I can escape machine code by bootstrapping. The problem I'm talking about is: I don't believe we know how to bootstrap out of the web. If we could, we would make it more alive and infused with our values.

Now that I'm totally bummed out and have sufficiently defined the problem and what I'm looking for, I think it would be useful to likeminded searchers if I go over what I've found in my search.

Obvious non-solutions

These projects are great, they're just not what I'm looking for.

Flexbox

Flexbox is great, but it's not the last thing I need to finally stop writing CSS because I have abstracted over it. Instead, it just makes living with CSS more comfortable.

Bootstrap

Yes, what a fitting name! Bootstrap promises something: use their HTML and the CSS just works. I'll also throw in all of the similar frameworks under this heading since the analysis is the same.

It's not quite the total solution I'm looking for. It's a bit like giving someone a tent to solve their home mortgage problem. You're still dealing with HTML, URLs, and all the rest. For quick web projects, I still turn to it, though.

Ruby on Rails

I remember when I first saw that build a blog in 15 minutes video. Wow! I was so excited. Back then, 15 minutes was unheard of.

But 10 years later, I overhear what Rails programmers complain about: how to write their templates, how to write their CSS, how to set up their routes, etc. In other words, they're using powerful tools to make the web more comfortable. They have not escaped it. Throw in some other well-known frameworks here, including Pedestal, Luminus, and Yada.

Garden

Several people have responded to my search with tweets about Garden, a library for generating CSS with Clojure. While it does give you the ability to use powerful logic to output CSS (high leverage), I don't see it giving you any ability to bootstrap out of CSS (low robustness). For example, could you give me a general solution to centering a div using Garden? You're still dealing with the difficult semantics of CSS.

Hopeful partial solutions

Hoplon

Hoplon is a ClojureScript web framework. It does several interesting things.

It makes DOM elements into functions so they can be used in code like regular functions.
It makes your elements reactive to changes like spreadsheets.
It makes your client-server communication look like function calls.

It turns out that these three features together help a lot. And it gives a good foundation to build on. Let's see where this project goes.

Hoplon UI

I'm keeping my eye on this project. The premise is simple: if you just wrap all of your divs in two more divs (which you're probably doing a lot of anyway with wrapper and container and inner divs), you can "solve" layout with a bit of JavaScript. And those wrapper divs can be made automatically with a few more lines of code. It turns out that this is fewer divs than you would write if you did it by hand! I don't quite understand it yet but it's kind of exciting. I'm proud that this comes right out of the ClojureScript community.

Grid Style Sheets

Solve layout with a Cassowary constraint solver. It's a custom layout engine that is more expressive than CSS for layout. And it's got a declarative language for building the constraints. It definitely has the Bootstrap Mindset. You're still living in HTML + CSS, but it acts much more like you think it should.

Pollen

I am awestruck by this project. It produced Practical Typography, a beautiful online book. The pages look like hand-tuned HTML by a seasoned professional web designer. The author wrote his own authoring format using Racket (a Lisp) that outputs to HTML + CSS. It definitely has the Bootstrap Mindset. It highlights the value of writing your own tools and that maybe the problem is that we haven't peeled back enough layers of the stack.

React

I mention React here for its philosophy and its potential. It managed to find a high-leverage abstraction to bootstrap out of error-prone DOM manipulations. That was a serious pain point of the web. And the same abstraction is being used on "native" platforms like phones and desktops. With React on the web, you're still living in HTML and CSS land, but it's way better than setting innerHTML by hand.

From REST to CQRS with Clojure, Kafka, & Datomic

This is just a talk at Clojure/conj in 2015. He's talking about the stack he's using, but there's an important bootstrap embedded in it. He's bootstrapping on the web as a communication protocol. He defines a single endpoint that accepts POSTs and a Server Sent Events endpoint. The POST endpoint is for sending commands (capturing the intent of the user) and the Server Sent Events endpoint is for updates from the server. That's it. No more fiddling around with URLs and "routes". This is a true bootstrap. High-leverage and high-robustness.

Conclusions, a glimmer of hope, and a plea

Bootstrapping is a beautiful phenomenon. We can transcend our current limitations using nothing but our currently limited abilities. It is a wonderful expression of our humanity. However, are we investing our web apps with humanity? Or are we toiling away in the bracket mines, fixing our CSS, choosing how to name our wrapper divs, and solving problems, yet again, that we created?

The purpose of bootstrapping is to solve that problem once and for all, so that we can focus on what's important. We can take a long walk, or we can "fill [our creations] with ideas, and emotions, and humor, and warmth." (quoted from Reversing the Tide of Declining Expectations, Matthew Butterick, TYPO Berlin 2012.) We need to do that more than ever, for the world and for ourselves.

We live in an interesting time. I've been waiting for a long time to see how Clojure (or any Lisp, really) is going to transcend the web. It has so many great tools! And the community understands the Bootstrapping Mindset. Yet I haven't seen the bootstrapping happen. Can we make it happen? Can we peel back the layers and build an engine for expressing our humanity? What kind of expressivity do we want? Our imagination is the only limitation.

If you're interested in the Bootstrapping Mindset, the big ideas in Computer Science, or Lisp, you should check out the PurelyFunctional.tv Newsletter. It's a weekly romp through the history, present, and future of Lisp and functional programming.

The Idea of Lisp

Eric Normand — Tue, 13 Dec 2016 21:35:15 +0000

LISP. It conjures up visions of a bygone age of computers the size of refrigerators, ALL CAPS CODE, and parentheses. Oh! so many parentheses! So why is Object-Oriented Programming's creator so enamored with the idea of Lisp? And what can he mean by a programming language being an idea anyway? Should I blame my Computer Science education for not teaching it to me?

Lisp was first introduced to the world in a paper called Recursive Functions of Symbolic Expressions and Their Interpretation by Machines, Part I, written by John McCarthy. In it, McCarthy introduces many new ideas to programming. Among them are conditional expressions (that's right, if/then/else) and using more than one letter--sometimes even words and phrases--for variables (like they still do in math). Your favorite programming language owes those two features to John McCarthy. But there is an even deeper idea lurking in the definition of Lisp itself.

He defines 5 primitive operations (atom, eq, cons, car, and cdr) along with a conditional expression. It also assumes the ability to define functions. And then he uses those to define an entire programming language, defined in itself. Let me say that again: John McCarthy wrote 6 easy things in machine code, then combined them to make a programming language. Before that, the only higher-level programming language was Fortran, which took 18 man-years to develop. Fortran was a big achievement, but Lisp was a big idea.

Let's unpack this tremendous idea a bit:

The bootstrapping material was very small.

These 6 things give you lists of symbols which can be interpreted. They define a very small "kernel" which can be easily ported to other systems. It is a small "fixed point" of agreement. All other meaning can be defined in terms of them.

The language was defined in terms of itself as an interpreter.

This is a proof by construction that the language is universal. The idea of Turing Completeness actually has two parts. The first is that you can compute anything computable. That one is satisfied by just about all programming languages (Lisp included since it has recursive functions). However, the second part is much more special. Turing Machines can be seen as universal when they can interpret any other Turing Machine. Well, Lisp is defined as an interpreter in terms of itself from the get-go, just like a Universal Turing Machine. Lisp is a universal language because it can interpret its own code. While you can certainly write a JavaScript interpreter in JavaScript, none of the work is done for you.

The meaning of expressions in the language is defined by the interpreter.

You could write your own interpreter that assigned different meanings to the expressions. This is Alan Kay's notion of "late binding". Since we don't know much about how to program well, it would be a mistake to build in too many assumptions into the base of the language. So we want a system that will allow us to swap out the assumptions as we learn more without having to throw it all away.

The expressions are written in data structures usable in the language.

The expressions are written as recursive linked lists. The 5 primitives are all you need to walk these data structures and interpret them.

Lispers have enjoyed working with this highly flexible "kernel", though most Lisp systems make practical compromises, like compiling the expressions to machine code instead of interpreting it each time. Here are some of the features of Lisp that follow directly from the Idea of Lisp.

Macros

Macros are functions that take code and return code. They are code transformers. They extend the expressiveness of the language and allow you to do computation at compile time.

Data-Driven Programming

Lispers often write their own interpreters in Lisp for new languages they create. They re-enact the bootstrapping of Lisp for their own languages. These can be seen as Domain-Specific Languages.

Programming language experimentation

Because it was made to write its own interpreter, Lisp is great for experimenting with alternative semantics for languages.

We're all too forgetful of the history of our field. The most significant languages in our industry are changed every so often. The new hot language will be gone in 15 years. But Lisp, the idea, endures. Its most promising incarnation at the moment is Clojure, which runs on the JVM and in JavaScript.

However, the ideas of Lisp actually live on in your favorite language:

The REPL

REPL stands for Read-Eval-Print-Loop, which are the names of the four Lisp constructs that defined it. If your language has an interactive prompt where you can type in code and see it run, that comes from Lisp.

Recursive functions

Computer Scientists in 1960 knew that recursive functions were possible, but they thought they would be too expensive. Fortran, the other major language at the time, did not have recursive functions back then. Now recursive functions are table stakes.

Garbage collection

Lisp was the first language with garbage collection, mainly because the language created a lot of temporary objects and it ran for a long time.

Conditional expressions

Yes, John McCarthy invented the conditional expression. He lobbied the Algol committee to add them to Algol, from which most languages got them today.

Multi-character variable names

I mentioned this before, but it bears repeating: programmers were following math's lead and using one-letter variable names before McCarthy came along.

Literal data structures

Can you write arrays and maps directly with syntax in your language? Well, Lisp did that first using parens ().

Here's the full quote from Alan Kay:

Most people who graduate with CS degrees donâ€™t understand the significance of Lisp. Lisp is the most important idea in computer science.

I didn't graduate with that significance. It took a lot of reading and exploration after university to feel like I had a proper education in Computer Science. I try to share what I'm learning in my newsletter. The more I read and learn, the more fascinated I am by the depth and breadth of what was done over forty years ago.

If you're interested in the big ideas in Computer Science, the history of programming, or Lisp, you should check out the PurelyFunctional.tv Newsletter. It's a weekly romp through the history, present, and future of Lisp and functional programming.