Why can’t developers KISS?

Onurbon — Wed, 30 Oct 2019 15:01:24 +0000

Software development has become so complex that it’s almost funny.

I’ve heard many developers praise the “Keep It Simple, Stupid” principle in public, but I’m yet to meet one describing the code they work with as “simple”.

One of my favourite posts of all time, the insightful What’s Awful about Building Software, has already covered nicely what complex software feels like and how it affects our lives. So I’m going to assume that you’ve either read it or experienced some of the pains yourself. But having now spent 10 years building software, including the last 3 years building developer-facing products (to try and simplify programming), I wanted to reflect on the reasons why it is so hard in practice to change our code, our tooling, our culture and the status quo.

Below are 10 reasons that made it to my list, in no particular order:

we have an ego problem — developers can have strong opinions about the right way of doing things, and the right way is their way, regardless of how complex others find it
we suffer from impostor syndrome — some blame themselves for not being smart enough when they should in fact blame the unnecessary complexity that crept into the code
we can be pessimist control freaks — instead of embracing high-level, declarative and automated frameworks, we often prefer the much more complex low-level alternative to remain in full control, just in case we might need this control down the line
we’re particularly snobbish about low-code — those “Enterprise low-code platform” are designed for the “suits” and not for smart people, right? Granted, the vendor lock-in of traditional low-code platforms can be a non-starter, but why isn’t anyone building an open-source platform?
complex tools make us feel smart — think about complex tools such as Kubernetes or Redux with their ~50k stars on GitHub; those are great for large-scale software projects, but you’re not Google, and some of us only use them out of vanity (to feel smart or boost our CVs)
we still confuse code with Maths — yes, good abstractions can be extremely helpful (relational algebra gave us SQL), but there’s nothing simple about functions calling functions calling functions calling functions…
we get very attached to our code — we might have learned about all the problems that come with “code ownership”, but deep down, we’re still territorial and tend to push back when anyone suggests to refactor our own code just to make it simpler
radical innovation is hard and risky — how many technical entrepreneurs will dare building disruptive tools to “simplify software” when Microsoft is worth $1T and controls a big chunk of the developer tools industry, including GitHub?
developers care, but no one else will — it could really help if non-technical colleagues were contributing to the effort of keeping code simple, but that’s hard in practice. How could a PM or a designer tell the difference between simple code or complex code?
it’s all because of Trump and Brexit — just kidding here, but doesn’t everything boil down to politics at the end? It might help if society started to care a bit more about software transparency. Perhaps the complexity of software should even be regulated one day.

Despite all of the above, I personally remain very optimistic that things will eventually change for the better, simply because software has to become simpler to write and maintain. The need for software and the shortage of developers have never been so acute, the low-code space is booming, and VCs have a renewed appetite for dev tools, thus creating new opportunities.

More importantly, and despite all the aforementioned hurdles, there are still many passionate developers out there shipping new open-source solutions to try and fix the complex mess we’re in. If you’d prefer to look at one of those new solutions (instead of just reading me rant about old problems), please consider giving Prodo a spin and discussing it on Slack!

Combining React, Firebase and TypeScript with zero boilerplate.

Onurbon — Thu, 17 Oct 2019 22:57:40 +0000

We’ve spent the last couple of years building a variety of apps with React, Redux, TypeScript and Firebase. We love each of those technologies, but combining everything together in a nice way proved very difficult, and we continue to hear other teams struggling with similar issues. This post summarises our take on the core problems and provides some hints on the solutions that we’re now building.

The problem: mixing apples with oranges

Building full-stack applications with JavaScript in 2019 often requires integrating at least 3 or 4 pieces:

a framework to manage your visual components, e.g. React;
a framework to manage your application’s state, e.g. Redux;
a backend solution for your data and authentication, e.g. Firebase; and
if you like catching bugs at compile time, a type-system like TypeScript

A few solutions have been proposed to manage most of the above combination, such as the react-redux-firebase library on GitHub. But after 700+ commits from 80+ contributors (and even the much welcomed introduction of React Hooks), this library has not yet managed to provide a truly simple and intuitive developer experience, and we don’t think it can.

The core of the problem, in our opinion, is that a React/Redux/Firebase stack will always end up mixing very different patterns and computational paradigms for manipulating data:

🍏 React is processing data with synchronous functions , which are typically evaluated multiple times as the data is loaded and modified,
🍊Redux doesn’t let you just call functions, but instead forces you to dispatch actions which modify data using reducers that are typically evaluated a single time.
🍏 Because redux itself doesn’t have built-in support for asynchronous effects, you’ll probably introduce the concept of thunks (with redux-thunks) or sagas (with redux-sagas)
🍊Or you might have replaced redux by an alternative, thus introducing yet more concepts such as observables (e.g. MobX) or streams (e.g. Rx)
🍏 The interaction with Firebase relies on asynchronous functions , and the real-time data from Firestore is updated multiple times using subscriptions
🍊React Hooks may then be used to try and simplify a lot of the above, but they rely on yet another paradigm. Hooks are intuitively a form of dependency injection , and they typically handle the asynchronous nature of side effects using callback functions.

This is a huge technical mess, and is a very typical example of “framework fatigue”, something that is only getting worse in the JavaScript world. It is not an intuitive way to build applications, and forces beginners to deal with a steep learning curve. And it’s also a real-world problem for experts; writing code takes longer because of the boilerplate, reviewing code takes longer because of the general awkwardness and inconsistency, and writing tests is very cumbersome — meaning that you probably won’t write that many. And on top of all that, getting useful feedback from TypeScript is hopeless because most of the data coming from Firestore won’t be typed.

The solution: rethinking state management

We believe the problem of inconsistent patterns and paradigms can only be tackled by changing the layer that ties every thing together: the state management layer. A lot of simplicity can be achieved by reinventing state management with the right priorities in mind. In particular, we think that a good solution should:

Provide (most of) the same benefits as Redux, without the boilerplate and unnecessary abstractions (we’ll typically rely on proxies and/or transpilation to achieve this).
Drastically simplify the computational model to keep it in line with React (based on synchronous functions, evaluated multiple times).
Allow users to integrate backend technologies like Firebase by adding a “plugin” (in contrast to Redux’s middleware system, which is very low-level).
Enable maximal TypeScript inference by design (meaning that you define your schema once, and then TypeScript infers all your types, without requiring manual annotations).

Replacing Redux is quite an ambitious task — we’re talking about a library with 50k+ stars on GitHub, and more than 3M weekly downloads on npm. And we’re also not the first developers interested in replacing Redux (we found about ~100 related libraries on GitHub). But we haven’t found any attempt that followed the same priorities.

You might also have encountered dozens of “Redux is dead” blog posts on Medium in the last few months, often making the argument that GraphQL and/or React Hooks can help you get rid of Redux. But we found no real evidence of this outside of toy examples of apps. Hooks only provide a more convenient syntax to deal with React state or React context, but they’ve not drastically changed state management. Neither Hooks nor GraphQL have truly replaced Redux — they only let you reduce the amount of Redux needed in your application. That’s why we believe that state management remains a problem worth digging into.

Show me some code!

DISCLAIMER: We’re still experimenting with different options and invite you to check our repo to see what our framework looks like today. But we would also love your feedback (on Slack or in the comments below this article) on the syntax and architecture that we’re envisioning for the long term, which is outlined below.

Step 1: Define your data models

Prodo will give you a function dataModel parameterised by a type definition and a list of plugins. You’ll then be able to export variables such as state (the redux-state of your app), auth (the authentication data, automatically pulled from Firebase) and db (automatically synced with Firestore) which can be used directly in your action definitions. You’ll also be able to export a hook called useData to use (and watch changes in) the data in your React components.

Note in the above that we’re not exporting any type definition, because we won’t need any in the other files. That’s because we’ll be able to infer everything from the types of db, state, auth and useData.

Step 2: Define your actions

Prodo actions will be defined as plain old JavaScript functions and those function will be able to use mutation operations to modify data in your state and your database, as if they consisted of in-memory JSON objects. But this is obviously an abstraction.

Under the hood, we’ll use immer.js to implement copy-on-write semantics and keep track of data changes in a non-destructive way (thus making time travel debugging possible). We’ll also rely on dynamic binding to map state and db to the current execution context of each action. Intuitively, this means that an expression such as state.roomId will in fact evaluate to window.currentProdoContext.state.foo. This global variable currentProdoContext will then need to be to swapped when different actions start and terminate, by we’ll be able to do this in a reliable way as long as long as actions remain synchronous.

Note however that keeping actions syncrhonous doesn’t prevent us from using asynchronous effects, although this will also require special care. In particular, if the newId function in the above snippet was fetching some remote data, it would first need to throw an error saying “I’m not ready yet!”, and the action will need to be restarted from the top when the data does come in.

Those are only some of the technical details that will need to be addressed, but this was hopefully enough to give you an idea of how we want to bring the computational model closer to React’s model, using only synchronous functions, and allowing to re-evaluate these functions multiple times until all the data is ready and the computed patched can be applied.

Step 3: Using your data and actions from React

After defining our data models and our actions, we will then use them directly in React with the (typed) hooks that we have exported:

Here is what a Message component would look like:

Here is what a RoomSelector component (with a controlled input) would look like:

Here is what a PostMessage component would look like:

Finally, here is what our chat application will look like at the end, using a query function from Prodo’s plugin to pull and watch collections from Firestore:

A fair amount of non-trivial heuristics will be needed to ensure that the above components are re-rendered (efficiently) when the data is updated, but the experiments we’ve conducted so far suggest that we should be able to match Redux’s and MobX’s performances. You may also note that the above example — unlike most frameworks — are not requiring you to wrap every single component inside a “connect” functions. This is again something we’ve been able to achieve in our experiments (by dynamically redefining React’s createElement function to auto-connect all the relevant components), but the performance implications there will also require proper testing. And these are only some examples of the exciting challenges which we’re tackling now to achieve a truly simple, boilerplate-free experience for our users.

What now?

If you’ve liked what you’ve read so far, or just want to see where this is going, please consider starring our repository on GitHub, to let us know that you care. You can also stay up-to-date with upcoming features, and more importantly, join the discussion by joining our Slack Community.

If you want to jump right in, you can try Prodo today — our docs are online at https://docs.prodo.dev and you can quickly build your own example by following the tutorial here.