Forem: Chris James

HTMX is the future

Chris James — Fri, 05 May 2023 14:31:26 +0000

The current state of web application development

User expectations of the web are now that you have this super-smooth no-reload experience. Unfortunately, it's an expectation that is usually delivered with single-page applications (SPAs) that rely on libraries and frameworks like React and Angular, which are very specialised tools that can be complicated to work with.

A new approach is to put the ability to deliver this UX back into the hands of engineers that built websites before the SPA-craze, leveraging their existing toolsets and knowledge, and HTMX is the best example I've used so far.

The costs of SPA

SPAs have allowed engineers to create some great web applications, but they come with a cost:

Hugely increased complexity both in terms of architecture and developer experience. You have to spend considerable time learning about frameworks.
- Tooling is an ever-shifting landscape in terms of building and packaging code.
- Managing state on both the client and server
- Frameworks, on top of libraries, on top of other libraries, on top of polyfills. React even recommend using a framework on top of their tech:

React is a library. It lets you put components together, but it doesn’t prescribe how to do routing and data fetching. To build an entire app with React, we recommend a full-stack React framework.

By their nature, a fat client requires the client to execute a lot of JavaScript. If you have modern hardware, this is fine, but these applications will be unusable & slow for those on older hardware or in locations with slow and unreliable internet connections.
- It is very easy to make an SPA incorrectly, where you need to use the right approach with hooks to avoid ending up with abysmal client-side performance.
Some SPA implementations of SPA throw away progressive enhancement (a notable and noble exception is Remix). Therefore, you must have JavaScript turned on for most SPAs.
If you wish to use something other than JavaScript or TypeScript, you must traverse the treacherous road of transpilation.
It has created backend and frontend silos in many companies, carrying high coordination costs.

Before SPAs, you'd choose your preferred language and deliver HTML to a user's browser in response to HTTP requests. This is fine, but it offers little interactivity and, in some cases, could make an annoying-to-use UI, especially regarding having the page fully reload on every interaction. To get around this, you'd typically sprinkle varying amounts of JS to grease the UX wheels.

Whilst this approach can feel old-fashioned to some, this approach is what inspired the original paper of REST, especially concerning hypermedia. The hypermedia approach of building websites led to the world-wide-web being an incredible success.

Hypermedia?

The following is a response from a data API, not hypermedia.



{
  "sort": "12-34-56",
  "number": "87654321",
  "balance": "123.45"
}

To make this data useful in an SPA, the code must understand the structure and decide what to render and what controls to make available.

REST describes the use of hypermedia. Hypermedia is where your responses are not just raw data but are instead a payload describing the media (think HTML tags like <p>, headers, etc.) and how to manipulate it (like form, input).

A server returning HTML describing a bank account, with some form of controls to work with the resource, is an example of hypermedia. The server is now responsible for deciding how to render the data (with the slight caveat of CSS) and what controls should be displayed.



<dl>
  <dt>Sort</dt><dd>12-34-56</dd>
  <dt>Number</dt><dd>87654321</dd>
  <dt>Balance</dt><dd>£123.45</dd>
</dl>
<form method="POST" action="/transfer-funds">
  <label>Amount <input type="text" /></label>
  <!-- etc -->
  <input type="submit" value="Do transfer" />
</form>

The approach means you have one universal client, the web browser; it understands how to display the hypermedia responses and lets the user work with the "controls" to do whatever they need.

Carson Gross on The Go Time podcast

...when browsers first came out, this idea of one universal network client that could talk to any application over this crazy hypermedia technology was really, really novel. And it still is.

If you told someone in 1980, “You know what - you’re gonna be using the same piece of software to access your news, your bank, your calendar, this stuff called email, and all this stuff”, they would have looked at you cross-eyed, they wouldn't know what you were talking about, unless they happened to be in one of the small research groups that was looking into this sort of stuff.

Whilst ostensibly, people building SPAs talk about using "RESTful" APIs to provide data exchange to their client-side code, the approach is not RESTful in the purist sense because it does not use hypermedia.

Instead of one universal client, scores of developers create bespoke clients, which have to understand the raw data they fetch from web servers and then render controls according to the data. With this approach, the browser is more of a JavaScript, HTML and CSS runtime.

By definition, a fatter client will carry more effort and cost than a thin one. However, the "original" hypermedia approach arguably is not good enough for all of today's needs; the controls that the browser can work with and the way it requires a full page refresh to use them mean the user experience isn't good enough for many types of web-app we need to make.

HTMX and hypermedia

Unlike SPAs, HTMX doesn't throw away the architectural approach of REST; it augments the browser, improving its hypermedia capabilities and making it simpler to deliver a rich client experience without having to write much JavaScript if any at all.

You can use whatever programming language you like to deliver HTML, just like we used to. This means you can use battle-tested, mature tooling, using a "true RESTful" approach, resulting in a far more straightforward development approach with less accidental complexity.

HTMX allows you to design pages that fetch fragments of HTML from your server to update the user's page as needed without the annoying full-page load refresh.

We'll now see this in practice with the classic TODO-list application.

Clojure HTMX TODO

First-of-all, please don't get overly concerned with this being written in Clojure. I did it in Clojure for fun, but the beauty of this approach is that you can use whatever language you like, so long as it responds to HTTP requests.

Nothing special here, but it does feel like a SPA. There are no full-page reloads; it's buttery smooth, just like all the other SPA demos you would've seen.

The difference here is:

I did not write any JavaScript.
I also didn't cheat by transpiling Clojure into JavaScript. (see ClojureScript)

I made a web server that responds to HTTP requests with hypermedia.

HTMX adds the ability to define richer hypermedia by letting you annotate any HTML element to ask the browser to make HTTP requests to fetch fragments of HTML to put on the page.

The edit control

The most exciting and impressive part of this demo is the edit action. The way an input box instantly appears for you to edit and then quickly update it again feels like it would require either a lot of vanilla JS writing or a React-esque approach to achieve, but what you'll see is it's absurdly simple.

Let's start by looking at the markup for a TODO item. I have clipped the non-edit markup for clarity.



<li hx-target="closest li">
  <form action="/todos/2a5e549c-c07e-4ed5-b7d4-731318987e05" method="GET">
      <button hx-get="/todos/2a5e549c-c07e-4ed5-b7d4-731318987e05" hx-swap="outerHTML">📝</button>
  </form>
</li>

It maybe looks a lot, but the main things to focus on for understanding how the edit functionality works:

On the <li>, an attribute hx-target tells the browser, "When you get a fragment to render, this is the element I want you to replace". The children inherit this attribute, so for any HTMX actions inside this <li>, the HTML returned will replace the contents of the <li>.
hx-get on the edit button means when you click it, HTMX with tell the browser to do an HTTP GET to the URL and fetch some new markup to render to the <li> in place of what's there.
The form is not essential for the example, but it allows us to support the functionality for non-JavaScript users, which will be covered later.

When you start working with HTMX, an easy way to understand what's going on is to look at the network in the browser's developer tools.

When a user clicks the edit button, the browser does an HTTP GET to the specific todo resource. The server returns a hypermedia response, which is a representation of that resource with some hypermedia controls.



<form action="/todos/45850279-bf54-4e2e-a95c-c8c25866a744/edit"
      hx-patch="/todos/45850279-bf54-4e2e-a95c-c8c25866a744" hx-swap="outerHTML" method="POST">
  <input name="done" type="hidden" value="false"/>
  <input name="name" type="text" value="Learn Rust"/>
  <input type="submit"/>
</form>

HTMX then takes that HTML and replaces whatever we defined as the hx-target. So the user now sees these hypermedia controls for them to manipulate the resource, instead of the row pictured before.

You'll notice the form has a hx-patch attribute, which means when it is submitted, the browser will send a PATCH with the data to update the resource. The server then responds with the updated item to render.

Embracing the web

There's more to HTMX, but this is the crux of the approach, which is the same as the approach that most websites were made before SPAs became popular.

The user goes to a URL
The server returns hypermedia (HTML), which is content with controls.
Browser renders hypermedia
Users can use the controls to do work, which results in an HTTP request sent from the browser to the server.
The server does business logic, and then returns new hypermedia for the user to work with

All HTMX does, is make the browser better at hypermedia by giving us more options regarding what can trigger an HTTP request and allowing us to update a part of the page rather than a full page reload.

By embracing the hypermedia and not viewing the browser as merely a JavaScript runtime, we get a lot of simplicity benefits:

We can use any programming language.
We don't need lots of libraries and other cruft to maintain what were basic benefits of web development.
- Caching
- SEO-friendliness
- The back button working as you'd expect
- etc.
It is very easy to support users who do not wish to, or cannot use JavaScript

This final point is crucial to me and to my current employer. I work for a company that works on products used worldwide, and our content and tools must be as usable by as many people as possible. It is unacceptable for us to exclude people through poor technical choices.

This is why we adopt the approach of progressive enhancement.

Progressive enhancement is a design philosophy that provides a baseline of essential content and functionality to as many users as possible, while delivering the best possible experience only to users of the most modern browsers that can run all the required code.

All the features in the TODO app (search, adding, editing, deleting, marking as complete) all work with JavaScript turned off. HTMX doesn't do this for "free", it still requires engineering effort, but because of the approach, it is inherently simpler to achieve. It took me around an hour's effort and did not require significant changes.

How it supports non-JavaScript

When the browser sends a request that was prompted by HTMX, it adds a header HX-Request: true , which means on the server, we can send different responses accordingly, very much like content negotiation.

The rule of thumb for a handler is roughly:



parseAndValidateRequest()
myBusinessLogic()

if request is htmx then
    return hypermedia fragment
else
    return a full page
end

Here's a concrete example of the HTTP handler for dealing with a new TODO:



(defn handle-new-todo [get-todos, add-todo]
  (fn [req] (let [new-todo (-> req :params :todo-name)]
              (add-todo new-todo)
              (htmx-or-vanilla req
                               (view/todos-fragment (get-todos))
                               (redirect "/todos")))))

The third line is our "business logic", calling a function to add a new TODO to our list.

The fourth line is some code to determine what kind of request we're dealing with, and the subsequent lines either render a fragment to return or redirect to the page.

So far, this seems a recurring theme when I've been developing hypermedia applications with HTMX. By the very architectural nature, if you can support updating part of a page, return a fragment; otherwise, the browser needs to do a full page reload, so either redirect or just return the entire HTML.

HTML templating on the server is in an incredibly mature state. There are many options and excellent guides on how to structure and add automated tests for them. Importantly, they'll all offer some composition capabilities, so the effort to return a fragment or a whole page is extremely simple.

Why is it The Future ?

Obviously, I cannot predict the future, but I do believe HTMX (or something like it) will become an increasingly popular approach for making web applications in the following years.

Recently, HTMX was announced as one of 20 projects in the GitHub Accelerator

It makes "the frontend" more accessible.

Learning React is an industry in itself. It moves quickly and changes, and there are tons to learn. I sympathise with developers who used to make fully-fledged applications being put off by modern frontend development and instead were happy to be pigeonholed into being a "backend" dev.

I've made reasonably complex systems in React, and whilst some of it was pretty fun, the amount you have to learn to be effective is unreasonable for most applications. React has its place, but it's overkill for many web applications.

The hypermedia approach with HTMX is not hard to grasp, especially if you have some REST fundamentals (which many "backend" devs should have). It opens up making rich websites to a broader group of people who don't want to learn how to use a framework and then keep up with its constantly shifting landscape.

Less churn

Even after over 10 years of React being around, it still doesn't feel settled and mature. A few years ago, hooks were the new-fangled thing that everyone had to learn and re-write all their components with. In the last six months, my Twitter feed has been awash with debates and tutorials about this new-fangled "RSC" - react server components. Joy emoji.

Working with HTMX has allowed me to leverage things I learned 15-20 years ago that still work, like my website. The approach is also well-understood and documented, and the best practices are independent of programming languages and frameworks.

I have made the example app in Go and Clojure with no trouble at all, and I am a complete Clojure novice. Once you've figured out the basic syntax of a language and learned how to respond to HTTP requests, you have enough to get going; and you can re-use the architectural and design best practices without having to learn a new approach over and over again.

How much of your skills would be transferable from React if you had to work with Angular? Is it easy to switch from one react framework to another? How did you feel when class components became "bad", and everyone wanted you to use hooks instead?

Cheaper

It's just less effort!

Hotwire is a library with similar goals to HTMX, driven by the Ruby on Rails world. DHH tweeted the following.

Hotwiring Rails expresses the desire to gift a lone full-stack developer all the tools they need to build the next Basecamp, GitHub, or Shopify. Not what a team of dozens or hundreds can do if they have millions in VC to buy specialists. Renaissance tech for renaissance people.

That's why it's so depressing to hear the term "full stack" be used as a derogative. Or an impossible mission. That we HAVE to be a scattered band of frontend vs backend vs services vs whatever group of specialists to do cool shit. Absolutely fucking not.

Without the cognitive overload of understanding a vast framework from the SPA world and the inherent complexities of making a fat client, you can realistically create rich web applications with far fewer engineers.

More resilient

As described earlier, using the hypermedia approach, making a web application that works without JavaScript is relatively simple.

It's also important to remember that the browser is an untrusted environment, so when you build a SPA, you have to work extremely defensively. You have to implement lots of business logic client side; but because of the architecture, this same logic needs to be replicated on the server too.

For instance, let's say we wanted a rule saying you cannot edit a to-do if it is marked as done. In an SPA world, I'd get raw JSON, and I'd have to have business logic to determine whether to render the edit button on the client code somewhere. However, if we wanted to ensure a user couldn't circumvent this, I'd have to have this same protection on the server. This sounds low-stakes and simple, but this complexity adds up, and the chance of misalignment increases.

With a hypermedia approach, the browser is "dumb" and doesn't need to worry about this. As a developer, I can capture this rule in one place, the server.

Reduced coordination complexity

The complexity of SPAs has created a shift into backend and frontend silos, which carries a cost.

The typical backend/frontend team divide causes a lot of inefficiencies in terms of teamwork, with hand-offs and miscommunication, and makes getting stuff done harder. Many people mistake individual efficiencies as the most critical metric and use that as justification for these silos. They see lots of PRs being merged, and lots of heat being generated, but ignoring the coordination costs.

For example, let's assume you want to add a new piece of data to a page or add a new button. For many teams, that'll involve meetings between teams to discuss and agree on the new API, creating fakes for the frontend team to use and finally coordinating releases.

In the hypermedia approach, you don't have this complexity at all. If you wish to add a button to the page, you can add it, and you don't need to coordinate efforts. You don't have to worry so much about API design. You are free to change the markup and content as you please.

Teams exchanging data via JSON can be extremely brittle without care and always carries a coordination cost. Tools like consumer-driven contracts can help, but this is just another tool, another thing to understand and another thing that goes wrong.

This is not to say there is no room for specialisation. I've worked on teams where the engineers built the web application "end to end", but we had people who were experts on semantic, accessible markup who helped us make sure the work we did was of good quality. It is incredibly freeing not to have to negotiate APIs and hand off work to one another to build a website.

More options

Rendering HTML on the server is a very well-trodden road. Many battle-tested and mature tools and libraries are available to generate HTML from the server in every mainstream programming language and most of the more niche ones.

Wrapping up

I encourage developers looking to reduce the costs and complexities of web application development to check out HTMX. If you've been reluctant to build websites due to the fair assessment that front-end development is difficult, HTMX can be a great option.

I'm not trying to claim that SPAs are now redundant; there will still be a real need for them when you need very sophisticated and fast interactions where a roundtrip to the server to get some markup won't be good enough.

In 2018 I asserted that a considerable number of web applications could be written with a far simpler technological approach than SPAs. Now with the likes of HTMX, this assertion carries even more weight. The frontend landscape is dominated by waiting for a new framework to relieve the problems of the previous framework you happened to be using. The SPA approach is inherently more complicated than a hypermedia approach, and piling on more tech might not be the answer, give hypermedia a go instead.

Check out some of the links below to learn more.

Reduce WIP by practicing trunk-based development, rather than pull requests

Chris James — Mon, 26 Jul 2021 10:14:58 +0000

Poorly managed work in progress (WIP) is a common cause for low performance in development teams. When you have a team juggling too much work, it engages in frequent, expensive context switching, which reduces quality and slows delivery. Your team is working hard, but not delivering.

The reduction in quality has a compounding effect on WIP as you will get unexpected bugs, re-work, and outages; which in itself causes more WIP. Unplanned work is poison for productivity and predictability.

Managing WIP involves looking at how work flows through your system, trying to make it go quickly, smoothly and predictably. Identifying bottlenecks & constraints allows you to manage them more effectively and focus your efforts on improving them and making sure only the highest priority work goes to them. Any efforts to improve efficiency outside these constraints is waste because it’s the bottlenecks in your system that dictate pace. With a focus on quality you can reduce the amount of unplanned work which helps you manage WIP.

There is no one true way of working, the best method depends on your team and the context they’re in. Managing WIP, and having a smooth flow of work, is not a negotiation if you are serious about managing an effective team.

Pretend you are the lead of a 4-person development team. How your team approaches creating value obviously has a big impact on its ability to deliver it, yet many teams copy other team’s rigid models, rather than thinking about the context they are in and tailoring the process to it.

The open source, pull-request model

This is an often-cited, popular way for teams to work.

Each developer picks up a ticket/story/task
Developer writes code on a branch. This allows them to work isolated, focused, and not affect other team members with their work which may not be completed.
When the developer believes the work is finished, they raise a pull request. When another developer is free, they can take a look at the work and then initiate a feedback cycle with the person who raised the PR until it is of sufficient quality.
The code gets merged and deployed to customers

Often the reasons people adopt this model is it allows a degree of gatekeeping and socialising knowledge. It’s argued that it can help prevent mistakes from being shipped and maintain code quality.

High WIP

For a team of 4 developers, having at least 4 tickets in play at a time feels like high WIP. Chasing 100% “resource” allocation is not a desirable management goal because all systems need a level of slack to cope with unexpected work. If a team is running at full-capacity it is inflexible, and inflexible things tend to break under pressure.

If you raise a PR and none of your colleagues are free, you may be tempted to pick up another ticket. The WIP of the team has now increased to 5. You can easily imagine, and have probably experienced, scenarios where this is a slippery slope to climbing WIP.

How many times have you heard:

It’s done, it just needs to be reviewed

In software development, the word “just” is often doing a lot of heavy lifting. The work is not “done” until it is actually delivering value.

The feedback cycle in the PR can take some time with asynchronous back-and-forth, so you may end up having to switch context between multiple tickets at a time. In lean terms, the process has caused a bottleneck.

Even with a team of only 4 you can easily end up in a situation where they’re context switching between the story they’re assigned to, but also reviewing lots of other developers of code. The personal WIP for each team member can climb steeply depending on how back-and-forth the review process is. As the WIP increases for each person, the ability to get code reviewed diminishes.

PRs are remote-friendly due to their asynchronous nature. This is true, but you have to accept then you will have a lot of unfinished work in play, which will raise your team’s WIP.

If I were to be managing a team working like this, I think it would be prudent to ensure that you only have 3 developers working on stories at a time, giving your system some slack. The other developer can help reviews happen quicker, find ways to improve flow, increase quality in other ways (improve tests, logging, metrics, e.t.c.), and you’ll have more slack to deal with unplanned work.

Slow integration

The longer your code lives on a branch, the less integrated you are with the rest of your team. Whilst the 4 of you are described as a team, you are all working on different versions of the code, as individuals.

If you find it hard to get your colleagues to review your changes, this slow integration compounds, and teams that adopt this system often have to deal with challenging merge conflicts. In the previous example this way of working can lead to multiple PRs that need merging which can result in multiple merge conflicts having to be resolved. This is waste.

Unpredictable flow

The review back-and-forth can last minutes, hours, days, or even weeks. In the worst cases, it involves a lot of re-work. Once the developer falls in to the trap of picking up another story whilst their other is in review, WIP can quickly spiral out of control.

Plenty of successful teams ship software like this, it’s not unworkable, but I’m not convinced it’s simple or efficient. I think even when it is managed extremely well, it still causes a lot of waste.

Trunk-based development (TBD) and pair programming

What is the most obvious way to reduce WIP?

Do less work concurrently
Concentrate on finishing work
Increase quality to reduce unexpected re-work

So instead of having 4 developers work on 4 tickets, let's instead have 2 developers pairing per ticket.

By having 2 heads work on a problem, it liberates us from having to gate-keep the main branch. We should trust that 2 developers on our team can take ownership of a problem from idea to production. Therefore, they can commit their code straight to main. This simplifies the flow of work. We no-longer have a wasteful bottleneck to shipping value.

No, I can’t trust 2 developers to take an idea from kick-off to production.

How many then? 3, 4? Or worse, can it only be put into the main branch if it is blessed by the lead? If you’re a lead making yourself a bottleneck, you’re not doing a good job. You’re not spreading your knowledge, you’re not coaching well enough, you’re increasing bus factor, and you’re hurting the productivity of your team.

Improved flow, higher quality and less re-work

By integrating frequently through the day, the chance of merge conflict is low and even when it happens it is typically trivial to fix. Far easier than having to merge a week’s worth of changes back into main.

This is true continuous integration, the team can consistently see what everyone is working on, which helps collaboration. No longer will you hear excuses like:

I’d like to refactor the blah blah, but I can’t because I know Alice is working on a PR related to it.

By removing the barriers to changing code and having a tight integration with your team (rather than disparate branches) you can now fearlessly refactor.

It’s an assumption, but not an unfair one, that the quality of code should be higher if 2 people are working on a change. High internal-quality makes changing our system simpler and should reduce the chance of unexpected re-work.

If you practice continuous deployment, it means you’ll release smaller changes frequently throughout the day. This reduces risk and makes recovery easier compared to diagnosing a pull request which may have been worked on for a number of days (or weeks!) rather than hours.

The idea of code-review and socialising ideas is now constant and easily available. Developers don’t have to write large amounts of code and then inconvenience another developer, forcing them to context switch to give feedback on your ideas. Instead, you have a colleague that you can easily bounce ideas with.

To make sure the socialising of code is spread amongst all developers, we ensure that we rotate the pairs frequently. We’ll also do a “code review” session every week for a couple of hours when we can talk about our code in broader strokes, in a more holistic way; rather than focusing on an isolated change in a PR.

Risk

If the idea of pushing directly to main is uncomfortable for you, it’s likely you’re working on an unsafe system. For TBD to work, you need to engineer your system so that it is safe to work on. You’ll need things like:

Good alerting
Easy or automatic rollbacks
An excellent test-suite
Observability
“Test in prod”

If you aren't testing in prod you aren't testing in reality -- just a weak dime store knockoff. Then you light the fuse and walk away with your fingers crossed. Test in prod. Please.

— Charity Majors

Gatekeeping around PRs and releasing is often reliability-theatre. Nothing can save you from mistakes, they’re inevitable. What you need to focus on is your mean time to recovery.

To commit to main directly, safely & frequently you need an excellent, fast test-suite and a well configured CI-pipeline. Developers need to be disciplined and only commit code they’re happy to put on the main branch.

Every developer is touching mainline, so all features grow in the mainline… which acts as a communication point. With CI, the mainline must always be healthy, so in theory (and often in practice) you can safely release after any commit.

— Martin Fowler

To live with the constraints of wanting to continuously integrate but not make “bad” commits, you need to find ways to break your work down into small, achievable pieces. This discipline makes work more achievable and will reduce risk. By working like this, I’d expect our 2 pairs of engineers to push several times per day.

Trying TBD

If you do decide to try TBD, be prepared for some friction and growing-pains if you have a team unfamiliar working in this way. A few years back, I wrote about how I introduced TBD to a team that was previously familiar with a PR-based approach.It will be a journey of uncovering issues in quality, bottlenecks and collaborating to find ways to improve them.

TBD may expose some uncomfortable shortcomings in your system, and the quality of your code and architecture. I’ve noticed that often a PR flow masks problems like:

Developers finding it hard to break problems down in to smaller, less risky changes. They have not learned how to work in small batches.
Low internal quality. If your system suffers from a lot of inappropriate tight coupling, you’ll tend to trip over each other’s toes when working together on the main branch.
Poor/inadequate tests. Lots of teams use the “CI server” as a crutch for poorly written tests that they run slowly and infrequently. Every so often, the system is designed in such a way that important tests are impossible to run locally.
- If you’re pushing every hour, you need to be able to test locally and be confident your change is shippable and won’t affect your colleagues negatively.

This is a journey that many factories (real factories!) have gone through. To work leanly, with excellent flow, managing WIP requires you to look at how you do work and commit to increase quality from start to finish as a team.

Which is best for your team?

TBD is not niche or unusual. Plenty of big businesses, in well-regulated industries, work with pairs of developers committing many times a day to do the main branch. Chromium releases every day and does not use branches.

In this short video, Dave Farley shows a fin-tech company in a well-regulated space working with TBD

Usually, these high-performing teams ship these changes extremely frequently to customers, so they can get real feedback quickly and build superior products.

Whilst pairing may seem like an expensive frivolity, the workflow is simpler, more focused and of higher quality. You’ll find that this approach can out-perform the same team working as individuals on branches. There is plenty of evidence to back up this assertion.

Five years' worth of academically defensible State of DevOps research that shows causality of feature branching with poor performance.

— Daniel Terhorst-North

PRs optimise for individual resource allocation, but introduce bottlenecks in your system which drive up WIP, and make the flow of work less smooth than it could be. High WIP diminishes a team’s productivity and is a management dysfunction, making work harder than it needs to be.

To those unfamiliar with TBD, it may seem risky and even unprofessional, but it forces you to work with more discipline, adopt best practices which will make your system more resilient and the work simpler.

(Mainline development) is an extremely effective way of developing, and the only one which enables you to perform continuous integration.

— Continuous Delivery: Jez Humble & David Farley

Many arguments that dismiss TBD as impractical for their team often are centred around fear, and a lack of trust. If you’re a leader of a team, don’t dodge these issues with walls and process. Deal with it.

Pull-requests work excellently in the open-source, where you can't trust external contributors. For your closed-source, day job though?

Despite abundant evidence that trunk-based development practices contribute to better software delivery performance, some developers who are used to the GitHub-recommended workflow remain skeptical

— 2017 State of DevOps Report

Think about your context, examine how work flows through your system, set a WIP limit and stick to it. Find ways to improve the flow and your team’s performance will improve.

The TDD Thinking Hats

Chris James — Fri, 25 Jun 2021 09:02:19 +0000

When working with people who are learning test-driven development (TDD), I have a habit of talking about “the hat we’re wearing”.

This comes from reading the book Six Thinking Hats.

The premise of the method is that the human brain thinks in a number of distinct ways which can be deliberately challenged, and hence planned for use in a structured way allowing one to develop tactics for thinking about particular issues.

When a group wears a particular hat, they engage and discuss an idea in a deliberate way. For instance, if the group wears the yellow hat, they should think positively about an idea, even if that feels unnatural to them. This means everyone explores positives in something, even if individuals within the group have some biases against it.

I’ve seen developers who mix up the steps in TDD, where they are refactoring whilst the tests are failing, and they are trying to introduce new behaviour at the same time. These same developers frequently skip the explicit refactoring step too, so they make the whole process more difficult and still end up with code that requires a lot of improvement.

I find trying to have a more focused, deliberate mindset during the TDD steps helps me practice it more effectively, simply, and avoid some common TDD pitfalls.

This posts attempts to describe the hats you should wear during the TDD steps.

TDD feels right when you’re not wearing any single hat for a long time. If you’re switching through them frequently, that’s telling you that you’re iteratively making positive progress on the system.

When writing a new test: The green, ambitious hat

Write the test you want to see

In this situation, we should be consumer focused, and optimistic about the kind of code we’re going to write.

It can be helpful to ask yourself “if this code already existed in the open-source world, how would I want to use it, and how should it behave?” Try to be ambitious, you get to design the API by describing how it would be used by a consumer, in this case your test.

Take pride in the test you write, keep asking yourself whether the test describes the what and why effectively. Have empathy for your colleagues who’ll be reading and maintaining these tests.

At the same time, think about how you can cut scope sensibly, so you are confident that you’ll be able to make the test pass and have this positive change committed to source control in a reasonable amount of time. Have an eye on the end-state, but keep thinking about each small step toward that goal.

Aim to wear this hat frequently, but for short periods, so that you’re working iteratively with fast feedback loops on your decisions.

But, I don’t feel confident

This is normal! The first step, is often the hardest. There can be numerous reasons why writing a test is difficult:

The existing test suite is a hot mess

Adding a new test to an existing suite can be difficult for various reasons. Perhaps the existing tests are poorly written. Often they are a reflection of poor design, such as lots of complicated mock setup.

This might be time to reflect on the design of the code, and it may be prudent to pick up the blue hat to update the design of your system to facilitate the next change.

I don’t really know what I’m trying to make

Talk to your colleagues and stakeholders. Ask them what they think might be the first valuable step. If you don’t understand the problem you’re trying to solve, it’s going to be impossible to write a test for it.

You may wish to invest some time moving up a few abstraction levels to create an acceptance test to give you a more clearly defined north-star.

When the tests are failing: The red, uncomfortable to wear hat

When practicing TDD effectively, you’ll wear this hat frequently, but it should feel deliberate when you do, and you should never be wearing it for very long.

When the tests are failing, you should feel uncomfortable. Your software is not working correctly.

Endeavour to get the tests passing (green) as quickly as possible.

You’re not safe to make changes that are not directly related to passing the test, as you’ve lost your safety net. Any further changes not dedicated to making the tests pass can make the situation worse by adding confusion and complexity.

Make the test work quickly, committing whatever sins necessary in the process.

-- Kent Beck

It’s important to be thinking about good source control practice. Once you get the tests passing again, commit your changes. If you ended up wearing this hat during refactoring, it’s often better not to be precious about the code you wrote, just revert it.

Stop wearing multiple hats at once

In my experience, many developers wear too many other hats whilst wearing this one in particular, and can make life difficult for themselves because by introducing more changes tends to make the failing test situation worse.

Too often, developers worry too much about writing "beautiful code" and get distracted away from the goal, which is to make the software work. We’ll have time to make the code great once we’ve made the test pass.

When you’re refactoring: The blue, confident hat

We are free to be creative! Maybe you were thinking about a design with the Red hat or the Green hat on - now is the time you can try it out.

We feel safe to make improvements because we get fast and reliable feedback that whatever changes we’re doing are valid. This allows us to aggressively refactor and take pride in our work.

Make a positive change
Verify the change is valid by running your unit tests.
Commit the change to source control
Go to 1

All our efforts are backed by tests against real, desired behaviour, so design decisions we make will be “real”, rather than for imaginary concerns.

When wearing this hat we get to improve the code, experiment and learn what works. Be brave and try different approaches to improve the code. If it doesn’t work out, it’s not a problem, as you’ll always be able to get back to working code with a git checkout src/whatever.

Wearing this hat frequently through your day will steadily make you a better developer because you get more chances to experiment and learn.

If you have an idea for an improvement that you think will take many hours, pause for a moment. It might be worth going through a few more TDD cycles before making this commitment. By adding more tests, and therefore more behaviour; you are giving yourself more concrete information to make better design decisions.

Like the other hats, don’t let your head get hot wearing it for many hours in a row, as you might end up in a refactoring rabbit-hole. Little and often is the key.

Wearing the hats while pairing/mobbing

Six Thinking Hats says that everyone should be wearing the same hat at the same time, so that the group remains focused on the particular task, and you don’t end up moving in different directions and getting distracted.

For instance, the group could be wearing the red hat, but you can see a way the code could be improved. Don’t break the group’s flow, just take a note and bring it up when you’re all wearing the blue hat. Remember, if you’re practicing TDD well, you’ll be switching hats frequently through the day.

If you’re a more experienced member of the group, perhaps you can try using this thinking tool as a way of moderating how your team works through a problem more deliberately, and with better focus.

Do you wear these hats?

To those who feel confident and proficient with TDD, do these feelings and behaviours resonate with your experience? Do you disagree or do you have anything that’s missing here? Please reach out to me on Twitter.

The Why of TDD

Chris James — Fri, 28 May 2021 14:42:37 +0000

Many people have false starts with test-driven development (TDD). When I discuss it with people, many have some misunderstandings around the method, in particular the motivations behind it.

I believe if developers have a better understanding of why to use it, they’ll have a better chance of practicing TDD more effectively and get more value out of it.

This posts explores the reasons for using TDD and offers some guidance as to how to practice it more effectively.

I was invited to speak about this topic at the GoSG, a meet-up for the Go programming enthusiasts in Singapore. You can find the recording on YouTube.

This post quotes Growing Object-Oriented Software Guided by Tests (GOOS) (written by Steve Freeman and Nat Pryce) extensively as it aligns very closely with how I was taught TDD. I strongly suggest grabbing a copy and giving it a read. Ignore that the examples are in Java, the main lessons from the book are broadly language-agnostic.

Introduction

If you’re new to software development, there’s several knowledge hurdles you need to leap over. The one you’ll be judged on the most is your ability to use the tools in front of you to get stuff working and get it to the customers. How well you do this and how delightful the result can be referred to as external quality.

Once you get proficient at this, the main challenges facing mid-level and above are to do with internal quality. This focuses on your ability to work on and design systems so that the cost of change is reasonable. This is important because if a system is useful, it will have to change over its lifetime.

When talking to non-technical folk about this, you can use an analogy of the quality of a factory, and its equipment. If parts of your machinery are unreliable or hard to deal with, the factory will begin to grind to a halt. This is also true of software. If internal quality declines, the ability to affect the external quality of the software will decline too.

There are a lot of software developers out there who can bash the code out enough to make useful software. A problem in our industry is we’re often incentivised and praised by our ability to work through “tickets” quickly and trying to hit (often arbitrary) deadlines. Frequently though, the internal quality gets neglected, and we hear these stories over and over again:

Why is the estimate for changing the system 3 weeks? It should be 3 days!

We need to spend a month re-writing the Foobar module.

I can’t stand to work on this legacy system again. Please let me move to a different project, or I’ll find somewhere else to work.

There are fewer developers who can write software that can live for many years without painful re-writes and staggering maintenance costs. The reason simply is, it’s really hard. It requires effort and study. I’ve been writing software for over 15 years and I still find it hard and feel the need to keep studying and reflecting.

There are developers and stakeholders who have advanced skills in justifying reasons for not caring about internal quality. They’ll say things like:

The only thing that matters is working software

What about time to market?

All worthwhile systems have to be changed (see Lehman’s laws of software evolution). “All that matters is working software” can’t be used as an excuse for poor internal quality today when you will have to change the external quality tomorrow. People tend to under-estimate the costs of poor internal quality and over-estimate the cost of keeping internal quality high.

You need to be thinking about the design of your system because the problem you are solving is different to everyone else’s. Domain knowledge (or lack-of) has a giant impact on what designs are suitable, so no single blog post can tell you how to design your system. It’s your responsibility as an engineer.

Many developers work in environments which have a real fear around speed of delivery, and you can get tempted in to short-term promises of productivity. A new programming language! A shiny new framework. Even folder structures in the Go community seem to be endlessly discussed as some kind of silver-bullet to good software. Chasing these things is a distraction and is not equipping you to strengthen your skills at writing software with high internal quality, you’re just learning another thing that will be unfashionable in a few years.

This is why I think TDD is such a valuable method. It gives you a structured way to confront and think about the internal quality of your system. TDD is a skill which can work in any context, even on Mars. It is an excellent tool for training yourself and your team to design better software.

TDD can be seen as a developer frivolity that slows teams down. This is true when you are first learning it (like any skill) but with practice and training, you’ll find yourself working at a sustainable pace and consistently delivering value. High internal quality will facilitate improvements to external quality.

Adam Luzsi points out:

… most people who start to learn TDD, are also in the process of learning software design implicitly through TDD. And this causes the feeling of "this slows me down".

High internal quality is not exclusive to practitioners of TDD but if you’re on the fence about TDD I hope this post encourages you to take another look. Maybe at least make you reflect on your own methods for writing great software.

What is TDD?

Test-driven development is conceptually simple to follow:

Write a test for a small, desired behaviour
See it fail
Write just enough code to make it pass
Refactor
Go to 1 for next increment of behaviour

Yet, there appears to be many developers diverging a lot from what was described by Kent Beck and others.

This inconsistency makes discussing the merits of TDD difficult because every so often what someone believed to be a typical TDD process seemed to be following a TDD-like process but getting key parts of it wrong.

Why does this happen? I think tech in general suffers a problem of being extremely faddy and people not really understanding why they are adopting a particular framework or technique. If you don’t understand what TDD is for, you’re unlikely to practice it effectively.

So, why test-driven development?

Here are the typical answers you’ll get:

Prevent defects
Good test coverage
Confidence to refactor

Rightly some will argue:

These benefits are not unique to writing tests first

Why write tests first?

It’s worth exploring the why of TDD in detail as I believe it helps clarify how to use the method effectively. Many people have false starts with TDD.

GOOS picks up on this:

“We’ve seen teams that write tests and code at about the same time where the code is a mess and the tests just raise the cost of maintenance. They’d made a start but hadn’t yet learned that the trick … to let the tests guide development.”

You have to know what behaviour you’re trying to build to write a test

Developers go too fast, diving in to code and letting their imaginations run wild.

Jo Crossick on Twitter

When I write tests first, it's because I know that I'm itching to write over-complicated, over-ambitious code that is destined to become a hot mess, and tests is the only thing that can hold me back from creating instant technical debt

A lack of method to the design process can lead to software designed against loose, ill-defined requirements. This design can quickly become a drag on the team’s productivity. It can confuse developers, baking-in to your system assumptions that are untrue and behaviours that aren’t actually needed.

Writing a test first demands you precisely define, and focus on what you’re trying to achieve; this is a key part of the design process

Writing a test first means you have to:

remove any ambiguity, meaning you have to really understand the behaviour you need
cut down the scope to a small piece of useful functionality, liberating you from having to design the whole system

Rather than having to be somehow clever enough to know all the requirements the system would ever need and committing to a design, instead you design code for behaviour you have precisely defined. You design, for one thing at a time, and you do it properly.

It offers a fast and continuous feedback loop on your design

Writing a test gives you a sense of the design of a particular unit at the beginning. Many developers will dive in, creating types and abstractions for hours on end and only when they plug it all together and run their application do they begin to see the flaws in their design. Until code is executed and the behaviours observed, the design has had no real validation.

With TDD you get feedback at the start and as you iterate through more behaviour you will keep getting guidance from your tests. There is no faster way of getting feedback on your design than by declaring what success looks like at the start.

You will have a sense of progress as you take small, validated steps toward your goal. If practiced well, you won’t end up in situations of having to throw hours or days worth of code away.

Whilst not being unique to test-first, tests reflect your design back at you. If a test is difficult to read or write, it is telling you something. A key skill for practicing TDD well is to listen to your tests.

It gives you momentum and by focusing on internal quality will help you continue to change your software cheaply.

Many developers “spin many plates”, darting from one part of the code to the other, changing different bits of code and in general working in a fairly chaotic way. This way of working makes life difficult and more prone to error and re-work.

Writing a test first acts as a stabiliser. This sets you a clear, short-term objective and lets you focus on doing one thing well, rather than worrying about competing concerns.

The refactoring step improves your chances of maintaining good internal quality which is essential for a system to be malleable. We’ve all worked on systems where the internal quality is terrible and how frustrating it is how difficult it can be to make a seemingly simple change.

The iterative nature of TDD means you keep working on achieving one small improvement at a time.

It does not slow you down

https://www.geepawhill.org/2018/04/14/tdd-the-lump-of-coding-fallacy/

GeePaw has an excellent video which dispels the idea that TDD slows you down. In fact, it's the opposite when you forget about the silly idea that software development's bottleneck is simply typing code. Thinking is the bottleneck, we work in a knowledge-based field and TDD offers us a way to think with clarity.

It’s true that when you first practice TDD you will go slower than before. In cases where maybe your design skills aren’t as strong as you think, TDD may feel frustrating, but you can use it as a tool to improve your ability to write modular code.

It’s a skill that has to be learned and practiced but with time you’ll find yourself working quicker, confidently and with more focus.

The qualities required to make code unit-testable are good qualities in themselves

GOOS:

“…the effort of writing a test first also gives us rapid feedback about the quality of our design ideas—that making code accessible for testing often drives it towards being cleaner and more modular.”

Modular code is:

Easy to repurpose and re-use within different contexts
Focused
Simple to test
Small, with clear responsibility
Easy to understand

Many developers have a healthy skepticism to premature abstraction, but it can go too far.

It is not good to have that what and the how of code mixed up within a function featuring 100s of lines of code, with mixed up and complected concerns. Rich Hickey covers the topic of easiness vs simplicity in his incredible talk Simple Made Easy. Having all your code in one method is “easy”, but it is not simple.

Systems built up of modular, cohesive units are simpler to understand and easier to change.

It’s easier to write good tests beforehand

Writing tests after the code is written is challenging to get right. You can end up with tests that may not fail when you want them to, have cryptic error messages or simply not have good coverage. You’ll also fall in to the trap of testing your design, rather than the behaviour (more on this later).

Writing tests first and seeing them fail helps you validate their usefulness and helps you focus on testing from the consumer’s point of view, rather than the person who wrote it.

It requires you to constantly refactor

A key, but often skipped step by novices is the refactoring step. It is far easier and more sustainable to do frequent, small refactors around the code you’re actually working on throughout the day, rather than leaving it for “a refactoring/technical-debt sprint”.

Ill-factored code makes the system harder to re-design. Often the act of refactoring reveals designs. As you consolidate code and DRY things up you start to “see” abstractions that can simplify matters.

Digression: Refactoring sprints are an absurd idea and point to some real team disfunction

You want the benefits of increased internal quality now, not in 3 weeks time.
Everyone knows they are hard to do and frequently fail to give the pay-off you’d hope for.
It’s your job to maintain internal quality because that’s what helps you deliver external quality at a reasonable cost. Don’t ask for permission to do your job.

How to use TDD

Work iteratively, in small positive steps

Working on a single, small, well-defined thing is simple, and you’re more likely to succeed. You’ll get fast feedback on whether you’re going in the right direction.
Working on many, large, loosely defined things at once is difficult, and you’re more likely to make mistakes. The feedback loop is slow.

When practicing TDD you should always be thinking about how to reach the vague end-state with specific, achievable steps. If you’re writing a unit test that you know will take hours of effort to make it pass, you need to rethink your approach.

It can be challenging to break work down into small steps, and it will also challenge your design skills but working this way means you are more likely to succeed, and you’ll reduce waste.

When TDD feels right

Constant, small progress throughout the day. Never feels slow, feels deliberate.
Frequent commits to source control per hour.
Work feels “safe”, less chaotic than maybe you’re used to.
Mistakes are rarely large. Happy to revert changes if you realise you’ve gone down a wrong path.

Writing the test first

There’s a difference between knowing to write a test first and doing it in a way that helps you design your software.

GOOS:

“We want each test to be as clear as possible an expression of the behaviour to be performed by the system or object. While writing the test, we ignore the fact that the test won’t run, or even compile, and just concentrate on its text”

Rather than worrying about how the code will work at this point, concentrate on the behaviour, particularly from the user’s point of view. It’s essential you block out implementation detail from your head at this point in the process because you’re just trying to nail down exactly what you want to achieve.

The what and the how are two separate design activities which are too often mixed up, which makes coherent decision-making more difficult than it needs to be.

Focus on being precise and limited in scope, this will help you uncover assumptions and help you keep focused.

TDD is an iterative approach to development. When you are writing your tests you should be mindful of how to keep the work moving with fast feedback loops and small positive steps.

Start from the top down for new features

TDD is focused on letting you design for the behaviour you precisely need, iteratively. When starting on a new area you need to identify a key, important behaviour and then aggressively cut scope.

From there you want to take a “top down” approach, starting with an acceptance test (AT) that exercises the behaviour from the outside. This will act as a north-star for your efforts. All you should be focused on is making that test pass. This test will likely be failing for a while whilst you develop enough code to make it pass.

Once you have your AT set up you can then break into the TDD process to drive out enough units to make the AT pass. The trick is to not worry too much about design at this point, just get enough code to make the AT pass because at this point you’re still learning and exploring the problem.

Taking this first step is often bigger than you think, setting up web-servers, routing, configuration e.t.c, which is why keeping the scope of the work small is important. We want to make that first positive step on our blank canvas, have it backed by a passing AT so that we can then continue to iterate quickly and safely.

As you develop, listen to your tests, and they should give you signals to help you push your design in a better direction but again, anchored to the behaviour rather than our imagination.

Typically, your first “unit” that does the hard work to make the AT pass will grow too big to be comfortable, even for this small amount of behaviour. This is when you can start thinking about how to break the problem down and introduce new collaborators.

This is where test-doubles are very useful because most of the complexity that lives internally within software doesn’t usually reside in implementation detail, but “between” the units and how they interact with each other.

/*
The design "outside the braces" is what often has 
the biggest impact on internal quality
*/

func NewStockCheckHandler(
    stockChecker StockChecker, 
    productStore ProductStore,
) http.Handler {

/* 
implementation detail inside the braces is usually 
low thrills and simple to test
*/

}

We can use test-doubles to explore the interactions between our collaborators with a much tighter feedback loop than if we tried to design and develop units isolated and then integrate them later.

For further iterations we may not need to write a new AT, we can simply add new behaviours to our existing units at a faster pace than the first one. We’ve made our first step, and we’re confident that we’re not developing them blindly because the units are integrated with the rest of the system.

On “TDD-ing a design”

Too often people assert that TDD is only compatible with a “bottom-up” approach where you imagine a bunch of abstractions up-front and then you “TDD them”.

This is called “TDD-ing a design”. You’re not using TDD as a design method, merely a way of writing tests around some code. This often results in classes or functions being overly developed in isolation for extended periods and then only when you try to integrate them into the system and use them you realise the design is wrong.

This is the point where people declare TDD to be a poor design tool when in practice, they already had a design in their head and just added a very long feedback loop to it, exhaustively testing code and various edge-cases before using it in their actual system.

Remember that TDD demands we start with a test for a desired behaviour, not a design. This is very much “top down” development. When done correctly you stop “over-designing” and you write and design the code you actually need.

By TDD-ing a design you can fall into the trap of tests being tightly coupled to implementation rather than behaviour. This becomes extremely problematic when you want to change your implementation detail and your tests will become a burden. I talk about this in detail in “The Tests Talk”.

Listen to your tests

If you follow the TDD process well you should end up with modular, loosely coupled, cohesive units within your system. This journey is rarely smooth. As requirements and knowledge changes your design will have to evolve with it.

Tests can offer you a focused lens into areas in your system where you can view the impact of your design isolated, which makes it far easier to appraise. Tests reflect your design back at you.

Here are some common test smells to look out for. Commit them to memory, and you’ll be able to raise some interesting questions about your design to answer.

High number of test-doubles (mocks), complicated setup

This is telling you that the unit under test has to collaborate with many things, this pain in your test world will be multiplied in all the cases you want to use this unit.

This pain can sometimes feel hidden in the “real” usages but comes apparent when you start to change things and feel the pain of the tight and inappropriate coupling. You should be asking yourself why this unit has to collaborate with so many other things.

It points to leaky abstractions which could be remedied by:

Look at the test doubles. Are you having to mock out lots of methods? Perhaps you can consolidate this interaction into another more cohesive unit
- Maybe the unit itself has too many, unrelated behaviours. See if you can come up with a more cohesive design.

Unclear intent

Does your test really explain the code’s behaviour and why?

GOOS:

A test called testBidAccepted() tells us what it does, but not what it’s for.

If the test is unclear it could mean the responsibilities of the unit being tested is unclear and warrants a redesign.

Wide scope

This points to a lack of focus. Perhaps you need to break the problem down in to smaller parts.

Value the quality of your tests

Start being more attentive about the quality of your test code. This will result in you writing tests that are cheaper to maintain and will challenge you to write more focused, modular code resulting a better designed system.

Writing high-quality tests is cheaper than maintaining poorly written ones

In a high-performing team where it must respond to client needs quickly and cheaply, good tests will enable them to have a productive experience. The opposite is also true.

Write the tests you want to see

The easiest, most generic feedback I give to my colleagues is asking them to explain their tests out-loud. Often it’s harder than they imagine, even when the tests appear terse; and then an interesting conversation about the design almost always follows.

Reflect on the tests you have now. Do they express the intent of the behaviour well? Do they describe the why clearly? If not, what would it take? Is it a simple refactoring of your tests or does it require some redesign?

Be attentive when you watch the test fail

Be sure it fails the way you expect and that it fails with a clear message. If you skip this step you may be glossing over design decisions, or perhaps you don’t understand the requirements clearly enough.

Usually making the test fails “nicely” is a multi-step process, but it’s important to go through the steps until you get a satisfactory failure message due to your code not exhibiting the failure you need.

View being in “the red” as something to get out of as soon as possible

You should be running your unit tests extremely frequently, which is why it’s important they run quickly and don’t depend on slow things like the network or file-system.

If at any point your tests are failing (so you’re in the red), you should be thinking about the shortest path to get back to “green”.

Often the safest way to do this is to revert your changes. By working in small increments this should never feel like a big deal.

It is forbidden to refactor or redesign code in the red state because your feedback loop as to whether the changes are valid is broken.

Commit your work frequently

To be able to use your tests to help you guide your design they need to be green, but it’s inevitable that sometimes you’ll go down the wrong path and want to get back to safety. It’s therefore critical that you practice good source control practices and commit your work frequently.

I usually commit my work once I’ve made the test pass and that means if I get into a bad state during refactoring I can easily get back to working software again.

So much of applying TDD is working in small, safe steps, so your mind is freer to think about the design of your system rather than juggling multiple concerns at once.

Refactor aggressively when in the green

Most of the time your system should be in a state when you can refactor freely.

Ill-factored code is problematic because it becomes hard to understand, which makes it harder to re-design. The process of refactoring in general will reveal improvements in the design for you.

Read Martin Fowler’s refactoring book and get in to the habit of applying common refactors. A lot of code I observe is not well-factored, but the good news is it’s a relatively easy skill to learn and once you’re proficient takes a matter of seconds or minutes to perform; in particular with help from your IDE.

Wrapping up

If you don’t practice TDD, what is your method for designing software? You do you! Can you describe your method to someone else, so they can follow a similar approach? It’s healthy to retrospect on how you work, to see if there are ways to improve.

No matter your design method, internal quality is essential for the productivity of your team. So many systems suffer from poor productivity and high costs because some developers and teams will neglect internal quality. It’s a false-economy to ignore internal quality and the pace at which you will be able to affect the external quality of the system will decrease quickly if you do.

It is of course possible to have software with high internal quality without practicing TDD, but I am not especially clever and want an easy life.

When I design up front:

it’s often wrong
I have to think about too many things at once
I design for things I don’t need

TDD offers a methodical, incremental way of designing code which feels like a simpler, more evidence-based way of creating useful, malleable software.

Stick to the simple TDD process, listen to your tests and practice. This should result in you designing software, verified against concrete desired behaviours; rather than your imagination.

How my website works

Chris James — Thu, 27 Aug 2020 14:47:52 +0000

I recently redesigned my website. It’s not exactly a technical marvel, but I feel that is its strength, and I wish more projects I observed prioritised delivering useful content rather than expensive to implement designs. Most people just want to read some text and look at some pictures and it’s extremely easy to do this if you don’t tie yourself up in flavour-of-the-month technologies.

On the way I got a chance to brush up on some of the more modern CSS techniques which was a lot of fun. People who complain about modern-day web development don’t know how lucky they’ve got it.

There’s so many fantastic techniques and tools built into web browsers compared to 10 years ago that don’t require a lick of JavaScript.

The approach

Under-the-hood not much has changed.

The server has a posts folder which contains markdown files for each post.
The web server is written in Go. When it starts up it opens up every markdown file, converts it into HTML and puts it in an in-memory map.
When it gets requests, the server simply looks at the path and matches it to a key in the map of posts, returning the pre-rendered post.
The frontend is plain HTML and CSS because it’s the best, simplest, most performant way of delivering textual content to users.
It is deployed to Heroku and sits behind CloudFlare CDN.

I’ll use the simplest tools I can to deliver a good experience. This will sometimes rely on using CSS techniques that won’t work on older browsers. This is fine so long as my content is still accessible and looks ok. This is called "graceful degradation".

Graceful degradation is a design philosophy that centers around trying to build a modern web site/application that will work in the newest browsers, but falls back to an experience that while not as good still delivers essential content and functionality in older browsers.

New things I learned

CSS Variables

On the face of it they seem like a nice way to DRY up some code and give some semantics to things like colours.

:root {
    --nav-colour: hsla(354, 100%, 71%, 1);
    --link-colour: hsl(354, 100%, 71%);
    --bg-colour: hsla(30, 100%, 99%, 1);
    --code-colour: hsla(327, 24%, 35%, 1);
    --text-colour: hsla(280, 12%, 20%, 1);
    --header-colour: hsla(280, 12%, 20%, 1);

    --font-size: 2.2rem;
    --line-height-multiplier: 1.5;
}

time {
    color: var(--code-colour);
}

I think they’re an interesting way of identifying the complexity of your design. If you give yourself the constraint that the only place you’re allowed to declare a colour value is within your variables then you’ll have a well-organised list of colours on your website. If that list gets long, or it becomes hard to name your variables ask yourself

Do I really need all these colours for my website?

You can take the same line of thinking in respect to the variables you use with your typography and your general spacing. By using variables it forces you to think and will help give your design a more consistent and less cluttered feel.

Media query for dark mode

Dark-mode feels pretty faddy to me but I am pleased there is a nice progressive way to change the style of your website without relying on any wonky JavaScript.

If supported, my website will automatically change into dark-mode with no prompting from the user. In OS X you can set dark mode to automatic which will go from light mode to dark through the day and my website will respect the user’s wish.

This feels much better than expecting users to have to update some kind of setting on every website they visit.

This works especially well with CSS variables. To enable dark-mode all I had to do was change the value of my variables.

@media (prefers-color-scheme: dark) {
    :root {
        --nav-colour: #15B097;
        --link-colour: #3282b8;
        --bg-colour: #051923;
        --code-colour: #15B097;
        --text-colour: #EDF6F9;
        --header-colour: #1C77C3;
    }

    img {
        filter: brightness(.8) contrast(1.2);
    }

    body {
        font-weight: 350;
    }
}

Easy!

Fallback for older browsers

Without any effort, my website is still useable for users of Internet Explorer (and probably Netscape Navigator).

Web browsers are very forgiving and will do their best effort to render even if they see syntax they don’t understand. As I am relying on robust, battle-tested technology (HTML & CSS) it "works" - the content is still accessible.

If I had done some react-whizz-bang framework single page app I would have to jump through a lot more hoops to reliably deliver some text.

Fallback techniques

Including this in the head of the document means this CSS file will only be loaded by users of IE9 and older.

<!--[if lte 9]>
    <link rel="stylesheet" type="text/css" href="/static/fallback.css">
<![endif]-->

The following media query lets me style for IE10 and IE11

@media all and (-ms-high-contrast: none), (-ms-high-contrast: active) {
    body {
        /* IE 10 & 11 specific style declarations */
    }
}

I don’t anticipate many users of IE11 or under being especially interested in a programmer’s blog, but it can’t hurt to help them out.

It's not pixel-perfect, but it doesn't have to be. I have seen teams spend days fixing minor pedantic styling issues with Internet Explorer and I can't imagine that's the best way to spend expensive developer time. Are you building websites to satisfy your users or designers?

Performance

All the content delivered and styled in 40ms with under 5kb of payload
(excluding post-specific images, which I try to minimise)

All these fancy computers we use have an impact on the environment. By taking a minimalist approach my website's carbon footprint is cleaner than 99% of the websites tested on websitecarbon.com

Big deal Chris, it’s only HTML and CSS, it’s not a real website

Yeah, it’s only HTML and CSS, brilliant isn’t it. Easy to make, accessible and performant. I made my first website around 21 years ago, and I find it quite comforting the same technology I worked with then works now.

I wish more websites were written like this rather than clogging my network and computer with megabytes of useless JavaScript written by developers running transpilers over transpilers on frameworks built on top of other frameworks and for what?

Fancy transitions?
The client not doing a full page refresh going between pages?

Do your users actually care about this? Click between links on my archive page and tell me it’s too slow and has a bad user experience.

There are projects where SPAs are a good fit, but many developers are poor at making this decision and add a huge amount of complexity for no real reason.

Wrapping up

Making a static website does not require a fancy blog generator built on top of a gig of node_modules. You do not need special tools to make a great website.

Even if you’re not a confident “backend” developer you don't have to write a web server like I did; you can still… create websites with plain old HTML files, written by hand!

This is a totally legitimate way of making many kinds of websites and will most likely result in the best user experience for the least effort.

Start naming your test doubles correctly

Chris James — Thu, 30 Jul 2020 10:07:02 +0000

A lot of developers are very vague when they name their test doubles which makes tests needlessly more difficult to understand.

In conversation there’s also often ambiguities which cause problems

I overheard on a podcast

Have you ever used a mock where you make assertions on the call?

This post will hopefully illuminate why that sentence is a bit silly!

By making a little more effort to be precise with your naming, you can communicate the intent of your tests and the design of your system more effectively

Quick primer on dependencies

To separate concerns, we often make objects that will “depend” on other objects.

We “inject” or “pass in” these dependencies when we construct them, so they can do a job.

This is called Dependency Injection.

Example

A ShoppingCart sends an email when its sendOrder method is called

We don’t want ShoppingCart to have to know how to send emails as well as everything else so we give it an Emailer when we create it.

new ShoppingCart(new MailgunEmailer())

It can then use the emailer without having to know how emails are sent, and you have your separation of concerns.

sendOrder() {
    // interesting domain logic
    this.emailer.send(confirmation)
}

This approach has implications for our tests.

With tests, we prefer not to use real objects for dependencies because we want to be able to control them, so we use something different

It would not be desirable to send a real email every time a test is run.
We sometimes depend on things that are slow, but we like our tests to be fast because feedback loops are extremely important
It would be hard for us to exercise failure scenarios using real dependencies. How do you get a 3rd party email service to deliberately fail?

Use “Test Doubles”

Not specifically mocks

A test double is what you inject into an "object under test" (OUT) to control a dependency.

Like a stunt double!

Using the precise words for test doubles helps reveal intent and documents your design more effectively

It’s not about being big and clever, it helps you communicate unambiguously. Software development is a team-sport so clear communication will help your team work better.

The different kinds of test doubles

Dummies

Sometimes you have arguments to functions that you don’t care about. Dummies are the arguments you pass in.

const dummyLogger = jest.fn()

Stubs

Some objects will query dependencies for data.

Stub the dependency

const stubUserService = jest.fn(() => 'Miss Jones')

Spies

Sometimes you want to call a method (or more abstractly send a command) to a dependency.

It can be hard to tell in a test this happened from the outside as it’s an internal detail.

If you want to verify these commands, use a spy.

const spyEmailSender = jest.fn()
// do some stuff
expect(spyEmailSender.mock.calls.length).toBe(1)

Fakes

Usually a “real” implementation but constrained.

Like an in-memory database.

Not as popular anymore due to things like Docker which let you spin up local versions of databases without the risk of there being differences between a fake and a real implementation.

Mocks

Mocks are a kind of test double!

Mocks are precise and encapsulate the detail of your interactions more explicitly.

You describe upfront what calls you expect on your test double

Order
Arguments
Specific return values

If the object under test doesn’t call things as expected the mock will error.

So returning to

Have you ever used a mock where you make assertions on the call?

What is being described is just a mock. If you're not worried about the calls, you're talking about a stub.

Why “mocking too much is bad”

The risk is your tests become needlessly precise and coupled to implementation detail.

This is also true of spies as they have a similar nature.

If you're overly specific with your mocks, when you want to refactor things you may find tests failing for annoying reasons that have nothing to do with behaviour.

Why “having too many test doubles is bad”

Listen to your tests if they cause you pain

If you have many test doubles in a test that means your OUT has many dependencies.

Does that sound right to you? It shouldn’t and it means you probably need to take a look at your design.

An anecdote

Yesterday I reviewed a test which had a double named

getterFn

I had to spend some time reading the test to understand what was going on, it turns out it should've been called:

spyCSVFetcher

Once this was ascertained it turned out the writer had actually neglected to spy on the number of calls, which was actually a core part of what was supposed to be tested.

If the double had been named correctly in the first place (which is very little effort) I would know to look for assertions on the way it was called to help me understand what the intent of the code is. Instead, it took some investigation.

Summary

Consistently using the recognised names for test doubles will help you be precise when communicating with other developers. It's a low-effort way to improve the way you work.

For instance if you name something a dummy rather than a mock the readers of your tests can forget about that particular test double and focus on the important collaborators.

Tests should act as documentation so describing your test doubles clearly will help reveal the design of the test for maintainers.

Quiz

What does a test with lots of doubles tell you?
How do you improve the design of a test with many test doubles?
Name all the kinds of test doubles
How is a mock different from a spy?

Work 2019: Building a brand new team

Chris James — Sat, 28 Dec 2019 09:31:36 +0000

This post is a summary of work stuff over the past year which covers my first adventures into

Technical leadership and line-management
Hiring and building a brand new team

I think I've helped build a pretty rad team, so if you're interested in that kind of thing, read on.

Rough start

After working in one of the most productive, interesting and fun teams I've ever had the pleasure to be a part of, the team has to be disbanded due to complicated (and boring) politics.

We were given freedom to move in to almost any team within the organisation but I really didn't take our team being disbanded well at all so I think in my eyes every opportunity looked boring and unfulfilling. It was never going to work out.

New job

Happily a new opportunity came about with my previous employer. I'd worked there for just under 2 years; it was really enjoyable, I liked the environment and I'm proud of the work we did and how we did it. It felt like we stood behind the good DevOps principles and it worked out well for us.

This would be a different role though, I would a technical lead. I have been a technical lead before but in a different environment and without the line management responsibility needed for this role.

I was a little tentative about this at first. I hadn't been a line manager before and everyone I've spoken to about it to implies it's horrible. My future boss spun it pretty well though:

I'd get to hire my own team
I'd get to run it more-or-less how I like (so long as we deliver, obviously).
- He knows I have strong opinions about how teams should be run and is happy to give me the autonomy to try to make it happen.

I deliberated on this and thought about leads who had worked well with me and those who hadn't. A couple of people who've had the dubious pleasure of managing me have had a real positive effect on my career (and life, I suppose). They pushed me into things I never saw myself doing and gave me confidence to stand behind my convictions.

When I put all that together with someone giving me this rare opportunity to pay it forward and to help some people find the joy I get out of software-engineering; I had to give it a go. I may never get this kind of opportunity again!

My personal goal for this job

Learn how to create great teams who not only deliver good software with minimal stress, and actually enjoy it.
Grow some engineers

Hiring

The initial suggested makeup for my team was:

A senior
A mid-level
An associate (junior)

Now where I work isn't as glamorous as one of these fancy fin-tech startup whatevers, but it is a good place to work. The environment is great, we're empowered and work in a pretty cutting edge way. We go to DevOps conferences and hear about best practices and we can honestly say we do a lot of them. We deploy many many times per day, deployment is a non-event for us.

Getting that across to the highly competitive London job market I guess is hard because I had to do a lot of recruitment effort to eventually get the right people.

Diversity and all that

I was determined not to hire 3-4 clones of myself. I am infuriating to work with. Studies show diverse teams have better results. Hiring diverse teams it's the right thing to do too.

We approached Makers Academy to help us hire as their raison d'etre is bringing people from different backgrounds into software development. I've had a very positive experience working with Makers graduates over the past 5 years, they've always made the teams and people around them better so I wanted more of that.

It was a fun and intense experience of pitches, chatting to lots of people and interviewing a number of great candidates. If it were up to me we would've hired more but in the end we hired 3 excellent developers who are thankfully nothing like me.

The team

The stars aligned quite nicely in that after months of recruitment going nowhere my team of 3 associates and a senior more or less joined at the same time. It was super exciting to have a new team suddenly appear in the office.

Having a brand new team, and role suddenly land on my lap all at once was pretty intimidating at first but the people I hired made it a lot easier because they're all excellent people and I had good support from other people in the department.

How I tried to make it work

Before leaving Springer Nature I had a meeting with one of the managers which was more impactful than he probably realises. He said

You need to take learning management as seriously as you do learning software development. In the end you have a direct effect on people's lives

OK, it's a bit dramatic but it is true. I reflected on managers who hadn't worked for me and how frustrated and annoyed I was about the situation even outside of work.

It's definitely a problem that a lot of people seem to think that management comes naturally to certain people and is not a skill to learn and practice.

So I picked up some books and watched some talks in the hope I could actually achieve my goals and not let my team down.

The Phoenix Project & The Machine That Changed The World

These are two books that talk about the problems you have when trying to build big complicated things with lots of people. They're very different books,

The Phoenix Project is described as a "DevOps Novel" which sounds lame as hell but is actually really enjoyable.

The Machine That Changed The World talks about the approaches to manufacturing cars. From artisan bespoke cars, to mass production and then lean manufacturing. It sounds far away from software development but as you read it you find yourself nodding along, drawing parallels with our industry.

There's loads to take from them and here are the main bits I got:

Command and control definitely does not work. You need to empower the people who actually create things to make decisions as they're the people with the most up-to-date, relevant knowledge.
You must respect work in progress (WIP) limits. It's way better for a team to properly finish tasks and only work on a small number of things than hurriedly work through dozens of tasks. Disrespect of WIP will slow you down.
A better understanding of "lean". Once you read these two books you have a very clear idea of where agile gets a lot of its ideas from.

I summarise these ideas in more detail in The ghost of Henry Ford is ruining your development team

Resilient management

This is a very practical book on technical leadership, it's quite short and I have found it invaluable. It has given me concrete things to do to help build my team.

Growing people

It has some fascinating insights between the terms "coaching" and "mentoring" which I share with basically everyone because it really has changed the way I try to help people.

Mentoring Giving advice and perspective to help someone solve a problem. Quite a direct way of giving information. Good for overcoming roadblocks. Focused on the problem and solution.
Coaching Asking open questions to explore a topic more. I like to see it as trying to help someone "join the dots" of an idea. This style will have a longer term impact on people and will help both parties explore ideas more.

Having these ideas explained has made me more conscious about how I try to help people solve problems. I find myself looking for coaching opportunities more when working with people and have tried to take that into our workshops by encouraging coaches to draw on real-life analogies to help explain technical ideas (David Wickes is extremely good at this).

Self reflection

The book discusses how when people go from engineering to leadership need to understand their "default work mode" and how it might not be compatible with different goals as a lead. I think my default work mode is "just fucking do it" and "ask for forgiveness, not permission"; this has broadly worked out quite well for me as an engineer in teams. However as a lead I need to check myself.

Sometimes I just want to open my IDE and refactor some code I dont like and git commit -am "refactor". On a high level that is the right thing to do but how does that help my teammates in the long-run? I should see these things as a coaching opportunity and instead find time to bring it up with people and explore the space with them so that in future everyone else can see the problems in our code better.

The book asks you what you think your management style is so you can reflect on it. Just asking that question of myself was weird and difficult, this is what I came up with:

No micromanaging. Feel strongly about empowerment, ownership and trust. I have no interest in denying clever people the opportunity to use their skills to achieve our goals.
Reliant on honest feedback, sometimes I don’t observe myself not being great
Hope to focus more on growing people than “getting shit done” but I will need to check myself because that is my “default work mode”.
Want to act as a sponsor, finding opportunities for people to grow

I like to think that I've stuck to my guns with this style and it seems to be working out OK.

Setting direction

The book also instructed me to write a vision (or North Star) for the team

Be an amazing team that is continuously improving that ships well-engineered software with minimal hassle and stress by relying on craft and taking advantage of our autonomy to apply industry best practices.

We will take pride in our actual agility to confidently change and evolve our software as and when we need to.

If we get good at creating great systems, our stakeholders will always be happy with us and will continue to give us autonomy.

And a mission, which is a more grounded version of it

Openly and honestly communicate with each other to guide ourselves so we can grow
A culture of learning and sharing knowledge
Confidence to suggest changes to improve our ways of working and our code
Continuously tightening feedback loops to guide improvement

And a strategy to get there

Finds ways to break problems down into smaller chunks so we’re never working on something that feels too intimidating. Perfect is the enemy of good, better to ship 80% of the feature quickly than 90% in 3 months time.
Low WIP (work in progress) so we’re not spinning too many plates and that allows us to focus on small tasks that we can execute well.
Culture of feedback. Practice saying something nice every day! Remember that giving someone “negative” feedback is an act of kindness (so long as it’s done kindly).
Practice continuous integration. Both in respect to code and knowledge.
Pairing, mobbing. Even if you work on a task by yourself it should be considered very unusual not to seek feedback on your work as you do it.
Give ourselves time to research, in particular pre standup research times
Share interesting things you learn.
Ensure everyone has time to work on all aspects of our code and development practice, actively fight any silos of knowledge forming.
Everyone on the team contributes in forming how we work.
We should take pride in our work. If you’re not proud of something, we discuss how to make it better.
Identify any single points of failure we may have in our ways of working and work to spread the knowledge & responsibility

If nothing else it felt cathartic to write this down and I've encouraged the team to contribute to the document.

I sometimes feel my role, if anything is to keep pushing the team in the direction this describes so that we do become that team that I hope we will be.

A talk from Makers

A few of us went to a Leading a Diverse Team workshop at Makers. It was interesting and one talk resonated with me a lot. It discussed how often as a leader we project our own ideas as to how to accomplish a goal based on our own experiences. This is fine and well intentioned but it's from a position of bias and you may be unintentionally shutting out different ideas.

A better, more empowering way of trying to achieve what I'm trying to achieve is setting a vision but then encouraging the individuals in my team to think about the unique ways they can contribute toward that vision that I may not be thinking of.

My way of achieving goals is one means to that end but it doesn't have to be the only one; and there's probably better ones that my team can think of based on their own background, skills and experience. It's my job to encourage people and build a great environment so they can explore their own ideas.

Lessons learned, things to improve

Introvert Chris

I am one of those introvert-ish people who can seem relatively outgoing most of the time but I definitely suffer from running out of social energy quite quickly and just wanting to get a break from people (no offence, team).

This role definitely has taken its toll on introvert Chris at times and I need to find ways of managing that better but I dont have many good ideas yet. I think perhaps now my team is settled more I shall try and delegate a few more meetings to people as it will save me some social energy and is an opportunity for others to contribute in those areas.

You cant just mob

I hear a lot of great things about mobbing and I am totally on-board but for whatever reason when we tried it did not go well. I suspect it was a function of the team being very unfamiliar with each other and the nature of the work (it was very new-project, infrastructure work).

Still, like leadership it's probably something you have to study and not just an innate skill. I hope to read more about this and try it again next year.

General time management

I'm not happy with the amount of time I've got to pair with everyone. I'd say I do it quite often but not enough. The times where I paired I've been able to give much more useful feedback in our one-to-ones.

When the role was first offered to me it was proposed I would manage 3 people. That quickly turned to 5 and at the start of 2020 7. To serve all these people well I am definitely going to have to think carefully about what I am doing every day and try to find opportunities to delegate useful tasks that will help people grow and free my time up at the same time.

Study more

The few books I got through gave me some excellent ideas and set me up well. I want to continue reading this subject to get better. I'm currently reading An Elegant Puzzle which is very interesting but seems maybe better suited to a head of engineering role rather than the one I'm in right now.

The next books will be:

Final reflections

I'm glad I didn't try and "wing-it" with management. I'm still learning though and there's lots of room for improvement but I think the upfront study I did helped grease the wheels for my team and get it off to a good start.

During hiring I was very upfront about what my expectations and aspirations for the team were which resulted in a group of people joining who seem to be bought into the idea and are contributing toward it.

I'm really pleased and proud of how things are going. I shouldn't take for granted that we managed to turn a group of strangers with various backgrounds and experiences into a great team. The culture is fun, open, curious and we take pride in our work; which is honestly exactly what I wanted.

I feel everyone has become better engineers since joining and I am confident we'll do great next year so long as we keep pushing ourselves to get better and better. (that felt like a bit of a lame manager-esque rallying call, sorry)

I think the most gratifying moment for me was a small one. A few weeks after the team formed I went on holiday. When I came back one of my team mates told me they had changed the process. I replied:

Good

It's the team's process, not mine. I want them to be comfortable making decisions about how we work and voicing strong opinions about the way we do things, and this demonstrated we are heading in the right direction.

Gamifying Continuous Integration

Chris James — Thu, 28 Nov 2019 14:54:32 +0000

If you read the internet, you'll soon realise that a large number of software developers don't understand what CI is and why it's important.

It's important to state that continuous integration is not a server.

Like the word DevOps, people seem to think these important principals of software engineering are about tools and products. Perhaps misinformed managers think if they buy the right magic box they can finally deliver their projects on time.

Continuous integration is continuous integration

Read the words.

It's about making sure that when you are working on code is the most up to date version of the code as a team.

In an ideal world if you could somehow all work on same computer at the same time comfortably, that would be pure CI.

If you've ever used VS-code's Live Share plugin, that's as pure a CI experience you can get in a remote environment. If you've not used it, it allows you to connect to a colleague's VS code and edit the code together as if you're working on the same computer. The changes you make are reflected (or integrated) to everyone else in the session - the feedback is immediate.

A much simpler experience of CI is mobbing. Everyone is working from the same version of the code together.

How do you know if you're not practicing CI well?

One obvious sign is developers wasting lots of time resolving large merge conflicts. Merge conflicts are a direct result of you working on a version of the code that isn't up to date. Your team are not integrating their changes continuously and therefore as they make changes the risk of making invalid changes increases.

The time wasted on merge conflicts has compounding effects as it increases the delays of integrating your own changes to the system.

One might argue you can resolve this by structuring work differently to avoid conflicts, even going as far as to re-architect your system to avoid merge conflicts. An often touted reason to go for microservices is it allows teams to work independently.

Reader, I have worked on a team with 20 odd developers, shipping frequently on a monolith and very rarely did we have merge conflicts. We also didn't do pull requests, we committed to master.

If you don't practice CI you are not working with your team as well as you could

Changes to the system are invisibly stored on your colleague's computers where they are of no use to you.

You could be imagining some great abstraction given the code in front of you but unfortunately the code in front of you is not the source of truth because unknown to you another developer merges a gigantic change after spending a week on a feature.

People become reluctant to refactor parts of the code because someone is working on a branch that is vaguely near the code you want to change so you wait a few days and then forget about it.

How do you CI well?

The simplest way is for everyone to commit to master, frequently. It really is that obvious!

To be good at continuous integration, you should continuously integrate your changes to master. That way everyone will be working with the same version of code for the most part.

It's important that you respect your colleagues and run your tests before pushing. If the build is broken the team should stop committing unless it's to directly fix the problem. If people keep committing whilst the build is broken it will become very difficult to diagnose what the problem is and the new commits may compound the issue.

The process, in full

git pull -r
Make a positive change to the system
git commit -am "added blockchain terraform machine-learning module 2.0"
git pull -r
./build.sh && git push

This practice is simple to follow, doesn't require elaborate review systems, branch strategies etc, and forces good habits.

In order to push frequently, you need to pull changes frequently
You can only push frequently if you work on small tasks, iteratively. There are many, many reasons why this is a wonderful, less risky and easier way of working. (see TDD, XP, any decent agile book, etc etc)

It wont entirely stop merge conflicts occurring, but they should be very infrequent and because you're doing small changes they're typically trivial to resolve.

But what about code review?

The industry seems to have conflated reviewing code with a very specific, relatively new process of looking at code when a pull request is submitted; usually too late once all the code is written.

Did you know that you can talk about code, at any time during the day?

Did you also know that the correct abstractions and patterns may not be apparent when the code is "done" and you'd be better off encouraging an environment where people are free to refactor code when they want to, rather than having to go through a laborious process every time they want to change the code. I've worked on and observed projects that practice code review very strictly and yet the code is still not great.

Relying on a process of checking code before it is merged is not going to result in a healthy codebase alone.

With this way of working I strongly encourage pair programming because people rightly assert that as a lone developer it is very easy to do something wrong if you don't gather feedback. Pair programming facilitates a constant feedback loop on what you are writing.

What if someone commits something wrong/broken?

First of all, no process in the world can prevent this. Stuff does go wrong, and you would be better asking yourself:

How can we detect and recover from problems easier?

Secondly, this approach does require a healthy and collaborative team that talk to each other regularly about what they are working on. They should be talking to each other when they start a bit of work about what they are going to do, how they will release it safely, what kind of tests, etc. Couple that with working on small things iteratively the risk of pushing something catastrophic are very low.

If someone does push some code that maybe works (so the tests pass, monitoring is all good, etc) but is actually "bad". Well so what? We can refactor it if we don't like it.

What if I, a tech lead want to check every change before merging

I don't want to work with gatekeepers.

I want to work on a team that trusts one-another.

Perfect is the enemy of good.

So is a system of pull requests bad?

Not at all, they're perfect for open source projects where you want to welcome contributions from other people but dont have implicit trust. In that case you will want to review changes that go into your precious project.

For a team where you trust the developers all it adds is overhead.

Liberation

We should be removing the barriers to changing software, not adding walls and walls of bureaucracy.

In this kind of environment people fix code when they see it needs fixing.

This way of working tightens feedback loops and increases actual meaningful agility.

CI is not just about the code

It's important to integrate knowledge of domain, work etc. (so important not to have silos of work with just the BAs/UXD etc)
Who is working on what (standups, kick-offs, etc)

Gaming it

I feel very strongly about working this way over a pull request approach when you are working on a team you trust.

Most of my team until recently were very unfamiliar with this way of working so I made a silly dashboard on one of the TVs which showed who committed to master most over a given week. It penalises you points if you push a change which breaks the build.

After just a few weeks this slight nudging has improved the team's approach to CI, we've had significantly less merge conflicts, people are refactoring more often and the number of failed builds has also dropped.

The code for it is on github

And if you have docker, you can run it with

$ docker run -p 8000:8000 quii/ci-league

If you want it to look at a private repo you'll need to supply a github personal access token as an environment variable

$ docker run -p 8000:8000 -e GITHUB_TOKEN=supersecret quii/ci-league

It doesn't do any fancy auto-refreshing but most browsers have extensions to auto refresh a tab, we set it to update every 10 minutes here.

Wrapping up

As usual in software, we seem to suffer as an industry of remembering the why of all of these fancy terms.

If you appreciate the why of CI you may not fall in to the trap of blinding applying best practices that work in a different context to the one you are working in. As I said pull-requests in an open source project make total sense and work great. In a different context it adds overhead.

You should instead focus on what your environment your team is working in, the constraints you have and use principles like CI to guide what process best fits.

The carbon footprint of your website

Chris James — Wed, 13 Nov 2019 07:29:03 +0000

Stumbled across this website https://www.websitecarbon.com/ which measures the carbon footprint of your website.

I've always believed in lightweight, efficient websites to deliver content. I firmly believe in not using client-side JS unless absolutely necessary because not everyone has an Internet Telephone Professional Maximum on a 4g connection.

I ranted about this and riled up the community about this subject here

The Web I Want

Chris James ・ Aug 20 '18 ・ 6 min read

#web #html #performance #a11y

Here's how my website scores

What's your score?

Given the huge amount of evidence pointing to how website performance effects users and the environment, what are you going to do about it?

For more on this check out the slides of this excellent talk https://twitter.com/hanopcan/status/1187675166236643329

The ghost of Henry Ford is ruining your development team

Chris James — Thu, 31 Oct 2019 15:05:52 +0000

People talk a lot about "lean". Maybe they've read The Machine That Changed The World and were inspired by how lean manufacturing transformed the motor industry. They read how Japanese companies became incredibly efficient and blew the competition out of the water and then they take that to their software development team.

This sounds good but too often I feel like people miss the real genius of lean manufacturing and continue to run their organisation with a top-heavy, command and control structure; the only difference is a constant message of

Be more efficient! Stop being wasteful! LEEEEEAN

Agile, Devops and other buzzwords clearly get a lot of their inspiration from lean manufacturing and these ways of working gain their efficiencies (and other, bigger benefits) broadly by empowering development teams and giving them responsibility.

A primer on mass production

Before Henry Ford’s assembly line, cars were made to order by craftsmen. It took a long time to make them and were reserved for the wealthy.

The assembly line made it possible to make cars cheaply which made them accessible to a far broader market.

How?

It’s a big subject, but here are some pithy bullet points

In order to reduce cost he needed to produce cars at high volume with low-skilled workers, not craftsmen.
The assembly line was a collection of expensive, inflexible machinery that scaled to volume and could be operated by cheap labour doing very simple tasks.
Emphasis on standardisation, being able to replace one worker with another cheaply and easily.

Problems

The assembly line was very intolerant to disruption

Had to add lots of buffers such as storing supplies, space and additional workers to smooth production
Better to accept defects and fix them after the car was made than to stop the line

The work was very dispiriting. Ford even passed on some of the savings and profits to his workers by heavily increasing their salaries but people just didn’t want to do the same boring task 100s of times a day.

The assembly line being heavily optimised for volume with expensive, specific machines meant it is very inflexible so creating different cars was a very difficult and costly endeavour.

When designing new cars it took what we would recognise as a very waterfall approach, of a project being passed around many departments and problems being unsurfaced way too late due to poor feedback loops and not a good structure of ownership.

Lean manufacturing

Japanese car makers studied the production line and tried a different way. When you hear about kanban and kaizen in agile circles, well it all came from here.

From Wikipedia

For many, lean is the set of "tools" that assist in the identification and steady elimination of waste. As waste is eliminated quality improves while production time and cost are reduced.

There isn't a canonical "lean" way of working, but the one that resonates strongest with me is "The Toyota Way"

The Toyota Way, in which the focus is upon improving the "flow" or smoothness of work, thereby steadily eliminating mura ("unevenness") through the system and not upon 'waste reduction' per se... The implementation of smooth flow exposes quality problems that already existed, and thus waste reduction naturally happens as a consequence. The advantage claimed for this approach is that it naturally takes a system-wide perspective, whereas a waste focus sometimes wrongly assumes this perspective

How does lean try to reduce waste?

Another big subject and there are good resources on it with a google search. The main drive of it from my perspective is actively trying to identify waste and then fix it. Lean describes 3 types

Mura: Waste due to changes in demand. Lean advocates a "pull" system rather than "push". So you only make a car if there is demand for it rather than trying to forecast demand which is often difficult and makes supply inconsistent. In software this would be akin to perhaps making a cheap prototype to prove an idea before committing to a bigger endeavour.
Muri: Waste by trying to do too much stuff at once. This is why if you're doing Kanban and not observing work-in-progress limits you're not doing Kanban. It can sometimes feel counter-intuitive that you can go faster by doing less work but you've probably worked in a situation where a team is working really hard and everything feels chaotic and unreliable. This is also why smaller teams are generally preferable.
Muda: Non-value adding work. A classic example in software would be developers constantly struggling with flaky builds and not addressing the underlying causes.

Lean emphasises the idea of "flow", optimising the processes for delivery and reducing interruptions in the production process. (think about trunk based development vs pull requests here)

Quality

For quality you adopt Kaizen, striving for perfection and hunting down the root causes of issues and fixing them. You adopt a culture of quality across the value stream, not after the product is "finished".

Command and control vs autonomy and empowerment

What I mainly took from the book "The Machine That Changed The World" was how it contrasted mass manufacturing and lean manufacturing in respect to management.

Lean manufacturing acknowledges that it's impossible to heavily micro-manage the production of a complicated product like a car with thousands of people working on it.

The best way for Kaizen to work is to coach and empower the do-ers to find the best ways to deliver.

Signs you're not in a lean company

People with job titles of "manager" or "leader" but are nothing more than coordinators

When designing new cars there is the idea of the Shusa (not sushi!) who in short was actually in charge of things. They could bring people to help and make decisions and were not worried about being overruled by the whims of upper management; they were empowered to make the project work. People would say that these people actually made their mark on a project "oh you can tell that's a Dan car because of X Y and Z".

Contrast that to places I've worked where people have had very lofty job titles but no actual empowerment and were nothing more than a communication funnel between the people who actually make things and the people who could actually make decisions.

How common is it in our industry for tech leads to be miserable simply because they're promoted into what they'll actually call a "shit shield". Seriously, can we stop saying that as if it's a good thing. It's a problem.

A leader should be able to lead

The development team has no say on priorities

Lean famously relies on Kaizen; the concept of continually striving for perfection. In lean plants workers can stop the plant if they find a problem and they work hard to make sure it doesn't happen again.

If you see something that is technically really wrong and is slowing your team down, how empowered are you to fix it? Do you work with a scrum master who will humour you but probably thinks "technical stories" are "waste" ?

Being lean and efficient is not simply done by only working on the most important feature at a given time. It's about the builders working hard to create a system and environment that can be worked on efficiently. That takes the form of the usual best practices; tightening and improving feedback loops, continuous delivery, easy to setup, simple delivery pipeline, excellent code, etc.

The lean plants worked hard to reduce problems and defects and to give themselves flexibility. This only happened because the workers were given the autonomy to make it happen; not micromanagement from above.

How does your team compare? Are you able to fix problems when you see them (which is the best time) or do you have to wait for the "technical debt sprint"?

All teams have a standard, immutable way of working

Toyota didn't just make up lean and be successful, they had to practice and iterate at it to figure out how to be good. Your IT teams need to practice and work at being "lean"

You also have to understand that we dont have teams of resources, they are individuals working on different problems. This means if you want to get the most out of people you need to give them autonomy to figure out how they can work together most effectively.

How does a team get better at working together if they cant gather feedback by retrospecting and then improving the way they work?

Larger organisations often talk about standardisation to allow developers to move between teams. That in itself isn't a bad thing but it's not important as autonomy.

If you want teams to be empowered enough to practice Kaizen and let them find how to work efficiently you have to accept that teams will work differently.

This is fine and in my experience moving between teams is always a challenge even if everyone works the same way, because teams are more complicated than a set of tools and processes.

The people who write the software are not responsible for QA or running it in production

In mass manufacturing the assembly line optimised itself for volume and would defer any kind of quality problems to the end of the line where another team would pick things up and fix them. The problem of this is the feedback loop just didn't exist and systemic problems would never get addressed, which brings waste.

I consider a QA column in a kanban wall as a smell. QA in lean worlds is a part of the entire development process, not something tacked on at the end. QA is a responsibility and an inherent quality of a good team.

From the continuous delivery website

It’s much cheaper to fix problems and defects if we find them immediately—ideally before they are ever checked into version control, by running automated tests locally. Finding defects downstream through inspection (such as manual testing) is time-consuming, requiring significant triage. Then we must fix the defect, trying to recall what we were thinking when we introduced the problem days or perhaps even weeks ago.

If you're throwing your software over a wall to an ops team then how do you expect to write a good system?

Boredom

If you're trying to turn your development team into a homogeneous assembly lines pumping out JIRA tickets at volume you might start to bore some people (and probably build some crappy products).

You probably pay your engineers a lot of money because they're presumably quite clever, so why are you denying them the space to think and take responsibility?

I've worked on projects which should be to be honest quite boring but turned out to be great fun because I worked in an engaged, empowered and fun team who took pride in doing things the "right" way and practiced Kaizen.

Not only was it great for us but we delivered the project which was initially seen as very risky and tricky with very little stress. Not only that but what we learned in respect to shipping software was adopted by other teams in the organisation.

Kaizen is fun, rewarding and will improve your outcomes.

Too scared to release and get feedback

In the book Lean Thinking: Banish Waste and Create Wealth in Your Corporation it says

Identify value from the customer's perspective

Doing this with software products is notoriously difficult but it's made even more difficult if you dont have the courage to release your software to the world and gather feedback.

Too often software is managed as a project which "ends" (this is nonsense, obviously) and no one takes any lessons from the customers. This is the ultimate form of waste that lean strives to fight against.

The transformational lean technologies and methodologies

I think it’s interesting to think about the transformative technologies over the past few years and how they relate to lean. I think a lot of things appear big and transformative but are they solving real problems? Are they helping you be lean?

Docker

Docker is an empowering technology allowing developers to use technology far more freely. Rather than asking for permission to use a technology and have it installed in production etc we can use whatever we like because we just ship containers

The main benefit for me though is it has made the difference between a local computer and production much smaller, which allows developers to take more responsibility for the quality of their software.

Synonymous with moving away from highly specialised, inflexible tooling in the mass manufacturing world

Devops

Devops is lean. It is about tightening feedback loops, improving the quality of our tooling and processes and taking responsibility of our software. Read The Phoenix Project. Just do it.

Cloud computing (e.g AWS)

Again this is another empowering development which lets the shop floor workers easily adapt the technology landscape to solve problems. Synonymous with having more flexible machinery in the lean car factories as opposed to highly specialised ones from mass manufacturing.

Wrapping up - we're not building cars

Agile and DevOps have clearly taken a lot of lessons from the lean manufacturing of cars. All of them are about building complicated things with lots of people which requires a different way of thinking vs mass manufacturing if you wish to do it efficiently with a happy workforce.

But when you're reading about lean it's important to remember that we software developers have different goals

We're not trying to build software at volume
We're trying to deliver the promise of software, products that are malleable and can change to users needs quickly and cheaply.

A big thing about lean is trying to identify value from a customer's perspective and eliminating anything that doesn't contribute to it. People take this lesson in software to aggressively deprioritise anything that isn't obviously a feature.

The thing is, this is just a very immature way of looking at what the stakeholders of software are. The "real" customer does benefit from:

Making it more operable so it has less downtime
Improving the test suite so the software can be changed easily to meet customer needs quickly
etc etc

I think examining the "value stream" for waste is a great thing, there's nothing more dispiriting than working on something for months with no real evidence it'll bring any benefit to anyone.

Too often though this is led by "product" people with little collaboration from the technical side. The team is unable to practice Kaizen and instead wastes its time working around technical debt, slow builds, poor feedback loops, manual testing etc. This results in a product which eventually becomes too hard to change and cant react to what the user wants in the long-run. A far cry from the flexible, malleable and efficient development system we're all supposedly striving for.

This comes back to what I wrote earlier, is your tech lead empowered to lead? Or is she just making sure you put estimates on JIRA tickets? Is she treated as an equal when deciding priorities for work? Does she have to argue constantly for scraps of time for the team to practice Kaizen whilst passive-aggressively being accused of being wasteful for even suggesting non-feature work?

Final final thought

From Wikipedia again

The cultural and managerial aspects of lean are arguably more important than the actual tools or methodologies of production itself. There are many examples of lean tool implementation without sustained benefit, and these are often blamed on weak understanding of lean throughout the whole organization.

Replace the word lean with "agile" and I think you'll have a lot of software developers nodding with knowing looks on their faces.

Go back to basics with MVC

Chris James — Thu, 25 Jul 2019 17:11:26 +0000

This is another follow up post to an episode of Go time I listened to the other day which seems to get my creative juices flowing.

It talked about web development in Go and a few things stuck out to me

A cautionary tale of generating HTML

Early on in the episode it was described how the built-in template library wasn't very expressive and described that the Go Buffalo framework uses Plush

Powerful, flexible, and extendable, Plush is there to make writing your templates that much easier.

Enter Jade

8ish years ago I worked on a pretty major project which was written in Scala. We decided to use the Jade template engine to generate our HTML which, like Plush allows you to do a lot of clever coding inside your templates.

This felt amazing at first but quickly became a nightmare. With the flexibility Jade gave us we made a bit of a mess. Our templates became ugly, hard to understand and difficult to test as more and more business logic leaked into our templates

In software I often see flexibility as rope that a team can hang itself with and I tend to prefer very opinionated and constrained things (like Go!).

What did we learn?

Software engineers are doomed to relearn things. We had forgotten about separation of concerns, in particular the guidance the Model-View-Controller design pattern prescribes.

MVC

It's quite surprising how badly people misunderstand MVC. If you go on Reddit or Twitter you will hear people telling other developers that "MVC isn't suited to Go".

Often people seem to think MVC is about folder structures, class names, or other fairly superficial concerns. So when developers from say a Rails background show their Go code with an array of folders they get derided and thrown to the lions.

MVC does not prescribe folder structure and I worry a lot of people seem to miss the point of the principle.

Model–View–Controller (usually known as MVC) is an architectural pattern commonly used for developing user interfaces that divides an application into three interconnected parts.

Controller

Intercepts "requests" (such as HTTP), parses them and then calls the appropriate "Model", getting some kind of data as a result and then sends it to a "View". In the Go world, that's usually a http.Handler

Model

Loosely, it's where your domain logic lives. I personally think it's quite poorly named but the important thing is the separation of concern. It must be decoupled from the Controller and View, so if you see anything HTTP related passed through from the controller it's likely you're violating that decoupling. They should also know nothing about views.

View

It doesn't know anything about controllers or models. It should probably get passed to it some kind of ViewModel which is just a collection of data it needs to render a view. A well designed view is simple and doesnt have domain logic; because templates are very hard to test cheaply.

What do these principles buy you?

Your domain code is cleanly separated from the rest of the system so it is easy to test and can be "plugged in" to different uses beyond your web server; for example a CLI tool, or just as a package for others to use.

It is trivial to create and edit views as they are merely mappings from a bundle of generic data into HTML (or whatever). It also makes them more accessible for frontend developers et al.

Your controller has clear concerns and are also easy to test.

Circling back to the start, I would be very cautious about something that allows me to write very expressive code for my view. These kind of tools trick developers with promises of convenience and power but often lead to leaky abstractions unless you are very disciplined.

Now have a think about all the tutorials you may have read around HTTP servers and Go. The good ones will recommended all of these principles without perhaps just calling them out. Rather than the community dancing around the idea of MVC, just embrace it. It's a tried and tested pattern and most people advocate it anyway, just not explicitly. If we were explicit then maybe there wouldn't be a new post every week about how to structure a HTTP server!

How to structure your web app

Understand the idea of MVC.
Your http.Handlers are your controllers, make sure they only do things controllers do according to above.
All your other business logic (Model) lives elsewhere, in packages centered around real things. If it's a bank i'd maybe have packages for Account, Currency, Customer e.t.c. Your controller will call things in your packages to do useful stuff.
Depending on what you're building you can just use the encoding packages to spit out XML or JSON, or if it's HTML just use template/html.

Learn Go with tests - Roman Numerals and Property Based Tests

Chris James — Wed, 10 Jul 2019 12:28:40 +0000

This is a post taken from a project called Learn Go with tests the aim of which is to get a familiarity with Go and learn techniques around TDD

You can find all the code for this chapter here

Some companies will ask you to do the Roman Numeral Kata as part of the interview process. This chapter will show how you can tackle it with TDD.

We are going to write a function which converts an Arabic number (numbers 0 to 9) to a Roman Numeral.

If you haven't heard of Roman Numerals they are how the Romans wrote down numbers.

You build them by sticking symbols together and those symbols represent numbers

So I is "one". III is three.

Seems easy but there's a few interesting rules. V means five, but IV is 4 (not IIII).

MCMLXXXIV is 1984. That looks complicated and it's hard to imagine how we can write code to figure this out right from the start.

As this book stresses, a key skill for software developers is to try and identify "thin vertical slices" of useful functionality and then iterating. The TDD workflow helps facilitate iterative development.

So rather than 1984, let's start with 1.

Write the test first

func TestRomanNumerals(t *testing.T) {
    got := ConvertToRoman(1)
    want := "I"

    if got != want {
        t.Errorf("got '%s', want '%s'", got, want)
    }
}

If you've got this far in the book this is hopefully feeling very boring and routine to you. That's a good thing.

Try to run the test

./numeral_test.go:6:9: undefined: ConvertToRoman

Let the compiler guide the way

Write the minimal amount of code for the test to run and check the failing test output

Create our function but don't make the test pass yet, always make sure the tests fails how you expect

func ConvertToRoman(arabic int) string {
    return ""
}

It should run now

=== RUN   TestRomanNumerals
-------- FAIL: TestRomanNumerals (0.00s)
    numeral_test.go:10: got '', want 'I'
FAIL

Write enough code to make it pass

func ConvertToRoman(arabic int) string {
    return "I"
}

Refactor

Not much to refactor yet.

I know it feels weird just to hard-code the result but with TDD we want to stay out of "red" for as long as possible. It may feel like we haven't accomplished much but we've defined our API and got a test capturing one of our rules; even if the "real" code is pretty dumb.

Now use that uneasy feeling to write a new test to force us to write slightly less dumb code.

Write the test first

We can use subtests to nicely group our tests

func TestRomanNumerals(t *testing.T) {
    t.Run("1 gets converted to I", func(t *testing.T) {
        got := ConvertToRoman(1)
        want := "I"

        if got != want {
            t.Errorf("got '%s', want '%s'", got, want)
        }   
    })

    t.Run("2 gets converted to II", func(t *testing.T) {
        got := ConvertToRoman(2)
        want := "II"

        if got != want {
            t.Errorf("got '%s', want '%s'", got, want)
        }
    })
}

Try to run the test

=== RUN   TestRomanNumerals/2_gets_converted_to_II
    --- FAIL: TestRomanNumerals/2_gets_converted_to_II (0.00s)
        numeral_test.go:20: got 'I', want 'II'

Not much surprise there

Write enough code to make it pass

func ConvertToRoman(arabic int) string {
    if arabic == 2 {
        return "II"
    }
    return "I"
}

Yup, it still feels like we're not actually tackling the problem. So we need to write more tests to drive us forward.

Refactor

We have some repetition in our tests. When you're testing something which feels like it's a matter of "given input X, we expect Y" you should probably use table based tests.

func TestRomanNumerals(t *testing.T) {
    cases := []struct {
        Description string
        Arabic      int
        Want        string
    }{
        {"1 gets converted to I", 1, "I"},
        {"2 gets converted to II", 2, "II"},
    }

    for _, test := range cases {
        t.Run(test.Description, func(t *testing.T) {
            got := ConvertToRoman(test.Arabic)
            if got != test.Want {
                t.Errorf("got '%s', want '%s'", got, test.Want)
            }
        })
    }
}

We can now easily add more cases without having to write any more test boilerplate.

Let's push on and go for 3

Write the test first

Add the following to our cases

{"3 gets converted to III", 3, "III"},

Try to run the test

=== RUN   TestRomanNumerals/3_gets_converted_to_III
    --- FAIL: TestRomanNumerals/3_gets_converted_to_III (0.00s)
        numeral_test.go:20: got 'I', want 'III'

Write enough code to make it pass

func ConvertToRoman(arabic int) string {
    if arabic == 3 {
        return "III"
    }
    if arabic == 2 {
        return "II"
    }
    return "I"
}

Refactor

OK so I'm starting to not enjoy these if statements and if you look at the code hard enough you can see that we're building a string of I based on the size of arabic.

We "know" that for more complicated numbers we will be doing some kind of arithmetic and string concatenation.

Let's try a refactor with these thoughts in mind, it might not be suitable for the end solution but that's OK. We can always throw our code away and start afresh with the tests we have to guide us.

func ConvertToRoman(arabic int) string {

    var result strings.Builder

    for i:=0; i<arabic; i++ {
        result.WriteString("I")
    }

    return result.String()
}

You may not have used strings.Builder before

A Builder is used to efficiently build a string using Write methods. It minimizes memory copying.

Normally I wouldn't bother with such optimisations until I have an actual performance problem but the amount of code is not much larger than a "manual" appending on a string so we may as well use the faster approach.

The code looks better to me and describes the domain as we know it right now.

The Romans were into DRY too...

Things start getting more complicated now. The Romans in their wisdom thought repeating characters would become hard to read and count. So a rule with Roman Numerals is you cant have the same character repeated more than 3 times in a row.

Instead you take the next highest symbol and then "subtract" by putting a symbol to the left of it. Not all symbols can be used as subtractors; only (1), X (10), C (100) and M (1,000).

For example 5 in Roman Numerals is V. To create 4 you do not do IIII, instead you do IV.

Write the test first

{"4 gets converted to IV (cant repeat more than 3 times)", 4, "IV"},

Try to run the test

=== RUN   TestRomanNumerals/4_gets_converted_to_IV_(cant_repeat_more_than_3_times)
    --- FAIL: TestRomanNumerals/4_gets_converted_to_IV_(cant_repeat_more_than_3_times) (0.00s)
        numeral_test.go:24: got 'IIII', want 'IV'

Write enough code to make it pass

func ConvertToRoman(arabic int) string {

    if arabic == 4 {
        return "IV"
    }

    var result strings.Builder

    for i:=0; i<arabic; i++ {
        result.WriteString("I")
    }

    return result.String()
}

Refactor

I dont "like" that we have broken our string building pattern and I want to carry on with it.

func ConvertToRoman(arabic int) string {

    var result strings.Builder

    for i := arabic; i > 0; i-- {
        if i == 4 {
            result.WriteString("IV")
            break
        }
        result.WriteString("I")
    }

    return result.String()
}

In order for 4 to "fit" with my current thinking I now count down from the Arabic number, adding symbols to our string as we progress. Not sure if this will work in the long run but let's see!

Let's make 5 work

Write the test first

{"5 gets converted to V", 5, "V"},

Try to run the test

=== RUN   TestRomanNumerals/5_gets_converted_to_V
    --- FAIL: TestRomanNumerals/5_gets_converted_to_V (0.00s)
        numeral_test.go:25: got 'IIV', want 'V'

Write enough code to make it pass

Just copy the approach we did for 4

func ConvertToRoman(arabic int) string {

    var result strings.Builder

    for i := arabic; i > 0; i-- {
        if i == 5 {
            result.WriteString("V")
            break
        }
        if i == 4 {
            result.WriteString("IV")
            break
        }
        result.WriteString("I")
    }

    return result.String()
}

Refactor

Repetition in loops like this are usually a sign of an abstraction waiting to be called out. Short-circuiting loops can be an effective tool for reabability but it could also be telling you something else.

We are looping over our Arabic number and if we hit certain symbols we are calling break but what we are really doing is subtracting over i in a ham-fisted manner.

func ConvertToRoman(arabic int) string {

    var result strings.Builder

    for arabic > 0 {
        switch {
        case arabic > 4:
            result.WriteString("V")
            arabic -= 5
        case arabic > 3:
            result.WriteString("IV")
            arabic -= 4
        default:
            result.WriteString("I")
            arabic--
        }
    }

    return result.String()
}

Given the signals I'm reading from our code, driven from our tests of some very basic scenarios I can see that to build a Roman Numeral I need to subtract from arabic as I apply symbols
The for loop no longer relies on an i and instead we will keep building our string until we have subtracted enough symbols away from arabic.

I'm pretty sure this approach will be valid for 6 (VI), 7 (VII) and 8 (VIII) too. Nonetheless add the cases in to our test suite and check (I wont include the code for brevity, check the github for samples if you're unsure).

9 follows the same rule as 4 in that we should subtract I from the representation of the following number. 10 is represented in Roman Numerals with X; so therefore 9 should be IX.

Write the test first

{"9 gets converted to IX", 9, "IX"}

Try to run the test

=== RUN   TestRomanNumerals/9_gets_converted_to_IX
    --- FAIL: TestRomanNumerals/9_gets_converted_to_IX (0.00s)
        numeral_test.go:29: got 'VIV', want 'IX'

Write enough code to make it pass

We should be able to adopt the same approach as before

case arabic > 8:
    result.WriteString("IX")
    arabic -= 9

Refactor

It feels like the code is still telling us there's a refactor somewhere but it's not totally obvious to me, so let's keep going.

I'll skip the code for this too, but add to your test cases a test for 10 which should be X and make it pass before reading on.

Here are a few tests I added as I'm confident up to 39 our code should work

{"10 gets converted to X", 10, "X"},
{"14 gets converted to XIV", 14, "XIV"},
{"18 gets converted to XVIII", 18, "XVIII"},
{"20 gets converted to XX", 20, "XX"},
{"39 gets converted to XXXIX", 39, "XXXIX"},

If you've ever done OO programming, you'll know that you should view switch statements with a bit of suspicion. Usually you are capturing a concept or data inside some imperative code when in fact it could be captured in a class structure instead.

Go isn't strictly OO but that doesn't mean we ignore the lessons OO offers entirely (as much as some would like to tell you).

Our switch statement is describing some truths about Roman Numerals along with behaviour.

We can refactor this by decoupling the data from the behaviour.

type RomanNumeral struct {
    Value  int
    Symbol string
}

var RomanNumerals = []RomanNumeral {
    {10, "X"},
    {9, "IX"},
    {5, "V"},
    {4, "IV"},
    {1, "I"},
}

func ConvertToRoman(arabic int) string {

    var result strings.Builder

    for _, numeral := range RomanNumerals {
        for arabic >= numeral.Value {
            result.WriteString(numeral.Symbol)
            arabic -= numeral.Value
        }
    }

    return result.String()
}

This feels much better. We've declared some rules around the numerals as data rather than hidden in an algorithm and we can see how we just work through the Arabic number, trying to add symbols to our result if they fit.

Does this abstraction work for bigger numbers? Extend the test suite so it works for the Roman number for 50 which is L.

Here are some test cases, try and make them pass.

{"40 gets converted to XL", 40, "XL"},
{"47 gets converted to XLVII", 47, "XLVII"},
{"49 gets converted to XLIX", 49, "XLIX"},
{"50 gets converted to L", 50, "L"},

If you're a cheater, all you needed to add to the RomanNumerals array is

{50, "L"},
{40, "XL"},

And the rest!

Here are the remaining symbols

Arabic	Roman
100	C
500	D
1000	M

Take the same approach for the remaining symbols, it should just be a matter of adding data to both the tests and our array of symbols.

Does your code work for 1984: MCMLXXXIV ?

Here is my final test suite

func TestRomanNumerals(t *testing.T) {
    cases := []struct {
        Arabic int
        Roman  string
    }{
        {Arabic: 1, Roman: "I"},
        {Arabic: 2, Roman: "II"},
        {Arabic: 3, Roman: "III"},
        {Arabic: 4, Roman: "IV"},
        {Arabic: 5, Roman: "V"},
        {Arabic: 6, Roman: "VI"},
        {Arabic: 7, Roman: "VII"},
        {Arabic: 8, Roman: "VIII"},
        {Arabic: 9, Roman: "IX"},
        {Arabic: 10, Roman: "X"},
        {Arabic: 14, Roman: "XIV"},
        {Arabic: 18, Roman: "XVIII"},
        {Arabic: 20, Roman: "XX"},
        {Arabic: 39, Roman: "XXXIX"},
        {Arabic: 40, Roman: "XL"},
        {Arabic: 47, Roman: "XLVII"},
        {Arabic: 49, Roman: "XLIX"},
        {Arabic: 50, Roman: "L"},
        {Arabic: 100, Roman: "C"},
        {Arabic: 90, Roman: "XC"},
        {Arabic: 400, Roman: "CD"},
        {Arabic: 500, Roman: "D"},
        {Arabic: 900, Roman: "CM"},
        {Arabic: 1000, Roman: "M"},
        {Arabic: 1984, Roman: "MCMLXXXIV"},
        {Arabic: 3999, Roman: "MMMCMXCIX"},
        {Arabic: 2014, Roman: "MMXIV"},
        {Arabic: 1006, Roman: "MVI"},
        {Arabic: 798, Roman: "DCCXCVIII"},
    }
    for _, test := range cases {
        t.Run(fmt.Sprintf("%d gets converted to '%s", test.Arabic, test.Roman), func(t *testing.T) {
            got := ConvertToRoman(test.Arabic)
            if got != test.Roman {
                t.Errorf("got '%s', want '%s'", got, test.Roman)
            }
        })
    }
}

I removed description as I felt the data described enough of the information.
I added a few other edge cases I found just to give me a little more confidence. With table based tests this is very cheap to do.

I didn't change the algorithm, all I had to do was update the RomanNumerals array.

var RomanNumerals = []RomanNumeral{
    {1000, "M"},
    {900, "CM"},
    {500, "D"},
    {400, "CD"},
    {100, "C"},
    {90, "XC"},
    {50, "L"},
    {40, "XL"},
    {10, "X"},
    {9, "IX"},
    {5, "V"},
    {4, "IV"},
    {1, "I"},
}

Parsing Roman Numerals

We're not done yet. Next we're going to write a function that converts from a Roman Numeral to an int

Write the test first

We can re-use our test cases here with a little refactoring

Move the cases variable outside of the test as a package variable in a var block.

func TestConvertingToArabic(t *testing.T) {
    for _, test := range cases[:1] {
        t.Run(fmt.Sprintf("'%s' gets converted to %d", test.Roman, test.Arabic), func(t *testing.T) {
            got := ConvertToArabic(test.Roman)
            if got != test.Arabic {
                t.Errorf("got %d, want %d", got, test.Arabic)
            }
        })
    }
}

Notice I am using the slice functionality to just run one of the tests for now (cases[:1]) as trying to make all of those tests pass all at once is too big a leap

Try to run the test

./numeral_test.go:60:11: undefined: ConvertToArabic

Write the minimal amount of code for the test to run and check the failing test output

Add our new function definition

func ConvertToArabic(roman string) int {
    return 0
}

The test should now run and fail

-------- FAIL: TestConvertingToArabic (0.00s)
    --- FAIL: TestConvertingToArabic/'I'_gets_converted_to_1 (0.00s)
        numeral_test.go:62: got 0, want 1

Write enough code to make it pass

You know what to do

func ConvertToArabic(roman string) int {
    return 1
}

Next, change the slice index in our test to move to the next test case (e.g. cases[:2]). Make it pass yourself with the dumbest code you can think of, continue writing dumb code (best book ever right?) for the third case too. Here's my dumb code.

func ConvertToArabic(roman string) int {
    if roman == "III" {
        return 3
    }
    if roman == "II" {
        return 2
    }
    return 1
}

Through the dumbness of real code that works we can start to see a pattern like before. We need to iterate through the input and build something, in this case a total.

func ConvertToArabic(roman string) int {
    total := 0
    for range roman {
        total++
    }
    return total
}

Write the test first

Next we move to cases[:4] (IV) which now fails because it gets 2 back as that's the length of the string.

Write enough code to make it pass

// earlier..
type RomanNumerals []RomanNumeral

func (r RomanNumerals) ValueOf(symbol string) int {
    for _, s := range r {
        if s.Symbol == symbol {
            return s.Value
        }
    }

    return 0
}

// later..
func ConvertToArabic(roman string) int {
    total := 0

    for i := 0; i < len(roman); i++ {
        symbol := roman[i]

        // look ahead to next symbol if we can and, the current symbol is base 10 (only valid subtractors)
        if i+1 < len(roman) && symbol == 'I' {
            nextSymbol := roman[i+1]

            // build the two character string
            potentialNumber := string([]byte{symbol, nextSymbol})

            // get the value of the two character string
            value := romanNumerals.ValueOf(potentialNumber)

            if value != 0 {
                total += value
                i++ // move past this character too for the next loop
            } else {
                total++
            }
        } else {
            total++
        }
    }
    return total
}

This is horrible but it does work. It's so bad I felt the need to add comments.

I wanted to be able to look up an integer value for a given roman numeral so I made a type from our array of RomanNumerals and then added a method to it, ValueOf
Next in our loop we need to look ahead if the string is big enough and the current symbol is a valid subtractor. At the moment it's just I (1) but can also be X (10) or C (100).
- If it satisfies both of these conditions we need to lookup the value and add it to the total if it is one of the special subtractors, otherwise ignore it
- Then we need to further increment i so we dont count this symbol twice

Refactor

I'm not entirely convinced this will be the long-term approach and there's potentially some interesting refactors we could do, but I'll resist that in case our approach is totally wrong. I'd rather make a few more tests pass first and see. For the meantime I made the first if statement slightly less horrible.

func ConvertToArabic(roman string) int {
    total := 0

    for i := 0; i < len(roman); i++ {
        symbol := roman[i]

        if couldBeSubtractive(i, symbol, roman) {
            nextSymbol := roman[i+1]

            // build the two character string
            potentialNumber := string([]byte{symbol, nextSymbol})

            // get the value of the two character string
            value := romanNumerals.ValueOf(potentialNumber)

            if value != 0 {
                total += value
                i++ // move past this character too for the next loop
            } else {
                total++
            }
        } else {
            total++
        }
    }
    return total
}

func couldBeSubtractive(index int, currentSymbol uint8, roman string) bool {
    return index+1 < len(roman) && currentSymbol == 'I'
}

Write the test first

Let's move on to cases[:5]

=== RUN   TestConvertingToArabic/'V'_gets_converted_to_5
    --- FAIL: TestConvertingToArabic/'V'_gets_converted_to_5 (0.00s)
        numeral_test.go:62: got 1, want 5

Write enough code to make it pass

Apart from when it is subtractive our code assumes that every character is a I which is why the value is 1. We should be able to re-use our ValueOf method to fix this.

func ConvertToArabic(roman string) int {
    total := 0

    for i := 0; i < len(roman); i++ {
        symbol := roman[i]

        // look ahead to next symbol if we can and, the current symbol is base 10 (only valid subtractors)
        if couldBeSubtractive(i, symbol, roman) {
            nextSymbol := roman[i+1]

            // build the two character string
            potentialNumber := string([]byte{symbol, nextSymbol})

            if value := romanNumerals.ValueOf(potentialNumber); value != 0 {
                total += value
                i++ // move past this character too for the next loop
            } else {
                total++ // this is fishy...
            }
        } else {
            total+=romanNumerals.ValueOf(string([]byte{symbol}))
        }
    }
    return total
}

Refactor

When you index strings in Go, you get a byte. This is why when we build up the string again we have to do stuff like string([]byte{symbol}). It's repeated a couple of times, let's just move that functionality so that ValueOf takes some bytes instead.

func (r RomanNumerals) ValueOf(symbols ...byte) int {
    symbol := string(symbols)
    for _, s := range r {
        if s.Symbol == symbol {
            return s.Value
        }
    }

    return 0
}

Then we can just pass in the bytes as is, to our function

func ConvertToArabic(roman string) int {
    total := 0

    for i := 0; i < len(roman); i++ {
        symbol := roman[i]

        if couldBeSubtractive(i, symbol, roman) {
            if value := romanNumerals.ValueOf(symbol, roman[i+1]); value != 0 {
                total += value
                i++ // move past this character too for the next loop
            } else {
                total++ // this is fishy...
            }
        } else {
            total+=romanNumerals.ValueOf(symbol)
        }
    }
    return total
}

It's still pretty nasty, but it's getting there.

If you start moving our cases[:xx] number through you'll see that quite a few are passing now. Remove the slice operator entirely and see which ones fail, here's some examples from my suite

=== RUN   TestConvertingToArabic/'XL'_gets_converted_to_40
    --- FAIL: TestConvertingToArabic/'XL'_gets_converted_to_40 (0.00s)
        numeral_test.go:62: got 60, want 40
=== RUN   TestConvertingToArabic/'XLVII'_gets_converted_to_47
    --- FAIL: TestConvertingToArabic/'XLVII'_gets_converted_to_47 (0.00s)
        numeral_test.go:62: got 67, want 47
=== RUN   TestConvertingToArabic/'XLIX'_gets_converted_to_49
    --- FAIL: TestConvertingToArabic/'XLIX'_gets_converted_to_49 (0.00s)
        numeral_test.go:62: got 69, want 49

I think all we're missing is an update to couldBeSubtractive so that it accounts for the other kinds of subtractive symbols

func couldBeSubtractive(index int, currentSymbol uint8, roman string) bool {
    isSubtractiveSymbol := currentSymbol == 'I' || currentSymbol == 'X' || currentSymbol =='C'
    return index+1 < len(roman) && isSubtractiveSymbol
}

Try again, they still fail. However we left a comment earlier...

total++ // this is fishy...

We should never be just increment total as that implies every symbol is a I. Replace it with

total += romanNumerals.ValueOf(symbol)

And all the tests pass! Now that we have fully working software we can indulge ourselves in some refactoring, with confidence

Refactor

Here is all the code I finished up with. I had a few failed attempts but as I keep emphasising, that's fine and the tests help me play around with the code freely.

import "strings"

func ConvertToArabic(roman string) (total int) {
    for _, symbols := range windowedRoman(roman).Symbols() {
        total += allRomanNumerals.ValueOf(symbols...)
    }
    return
}

func ConvertToRoman(arabic int) string {
    var result strings.Builder

    for _, numeral := range allRomanNumerals {
        for arabic >= numeral.Value {
            result.WriteString(numeral.Symbol)
            arabic -= numeral.Value
        }
    }

    return result.String()
}

type romanNumeral struct {
    Value  int
    Symbol string
}

type romanNumerals []romanNumeral

func (r romanNumerals) ValueOf(symbols ...byte) int {
    symbol := string(symbols)
    for _, s := range r {
        if s.Symbol == symbol {
            return s.Value
        }
    }

    return 0
}

func (r romanNumerals) Exists(symbols ...byte) bool {
    symbol := string(symbols)
    for _, s := range r {
        if s.Symbol == symbol {
            return true
        }
    }
    return false
}

var allRomanNumerals = romanNumerals{
    {1000, "M"},
    {900, "CM"},
    {500, "D"},
    {400, "CD"},
    {100, "C"},
    {90, "XC"},
    {50, "L"},
    {40, "XL"},
    {10, "X"},
    {9, "IX"},
    {5, "V"},
    {4, "IV"},
    {1, "I"},
}

type windowedRoman string

func (w windowedRoman) Symbols() (symbols [][]byte) {
    for i := 0; i < len(w); i++ {
        symbol := w[i]
        notAtEnd := i+1 < len(w)

        if notAtEnd && isSubtractive(symbol) && allRomanNumerals.Exists(symbol, w[i+1]) {
            symbols = append(symbols, []byte{byte(symbol), byte(w[i+1])})
            i++
        } else {
            symbols = append(symbols, []byte{byte(symbol)})
        }
    }
    return
}

func isSubtractive(symbol uint8) bool {
    return symbol == 'I' || symbol == 'X' || symbol == 'C'
}

My main problem with the previous code is similar to our refactor from earlier. We had too many concerns coupled together. We wrote an algorithm which was trying to extract Roman Numerals from a string and then find their values.

So I created a new type windowedRoman which took care of extracting the numerals, offering a Symbols method to retrieve them as a slice. This meant out ConvertToArabic function could simply iterate over the symbols and total them.

I broke the code down a bit by extracting some functions, especially around the wonky if statement to figure out if the symbol we are currently dealing with is a two character subtractive symbol.

There's probably a more elegant way but I'm not going to sweat it. The code is there and it works and it is tested. If I (or anyone else) finds a better way they can safely change it - the hard work is done.

An intro to property based tests

There have been a few rules in the domain of Roman Numerals that we have worked with in this chapter

Cant have more than 3 consecutive symbols
Only (1), X (10), C (100) and M (1,000) can be "subtractors"
Taking the result of ConvertToRoman(N) and passing it to ConvertToArabic should return us N

The tests we have written so far can be described as "example" based tests where we provide the tooling some examples around our code to verify.

What if we could take these rules that we know about our domain and somehow exercise them against our code?

Property based tests help you do this by throwing random data at your code and verifying the rules you describe always hold true. A lot of people think property based tests are mainly about random data but they would be mistaken. The real challenge about property based tests is having a good understanding of your domain so you can write these properties.

Enough words, let's see some code

func TestPropertiesOfConversion(t *testing.T) {
    assertion := func(arabic int) bool {
        roman := ConvertToRoman(arabic)
        fromRoman := ConvertToArabic(roman)
        return fromRoman == arabic
    }

    if err := quick.Check(assertion, nil); err != nil {
        t.Error("failed checks", err)
    }
}

Rationale of property

Our first test will check that if we transform a number into Roman, when we use our other function to convert it back to a number that we get what we originally had.

Given random number (e.g 4).
Call ConvertToRoman with random number (should return IV if 4).
Take the result of above and pass it to ConvertToArabic.
The above should give us our original input (4).

This feels like a good test to build us confidence because it should break if there's a bug in either. The only way it could pass is if they have the same kind of bug; which isn't impossible but feels unlikely.

Technical explanation

We're using the testing/quick package from the standard library

Reading from the bottom, we provide quick.Check a function that it will run against a number of random inputs, if the function returns false it will be seen as failing the check.

Our assertion function above takes a random number and runs our functions to test the property.

### Run our test

Try running it; your computer may hang for a while, so kill it when you're bored :)

What's going on? Try adding the following to the assertion code.

assertion := func(arabic int) bool {
    if arabic <0 || arabic > 3999 {
        log.Println(arabic)
        return true
    }
    roman := ConvertToRoman(arabic)
    fromRoman := ConvertToArabic(roman)
    return fromRoman == arabic
}

You should see something like this:

=== RUN   TestPropertiesOfConversion
2019/07/09 14:41:27 6849766357708982977
2019/07/09 14:41:27 -7028152357875163913
2019/07/09 14:41:27 -6752532134903680693
2019/07/09 14:41:27 4051793897228170080
2019/07/09 14:41:27 -1111868396280600429
2019/07/09 14:41:27 8851967058300421387
2019/07/09 14:41:27 562755830018219185

Just running this very simple property has exposed a flaw in our implementation. We used int as our input but:

You cant do negative numbers with Roman Numerals
Given our rule of a max of 3 consecutive symbols we cant represent a value greater than 3999 (well, kinda) and int has a much higher maximum value than 3999.

This is great! We've been forced to think more deeply about our domain which is a real strength of property based tests.

Clearly int is not a great type. What if we tried something a little more appropriate?

`uint16`

Go has types for unsigned integers, which means they cannot be negative; so that rules out one class of bug in our code immediately. By adding 16, it means it is a 16 bit integer which can store a max of 65535, which is still too big but gets us closer to what we need.

Try updating the code to use uint16 rather than int. I updated assertion in the test to give a bit more visibility.

assertion := func(arabic uint16) bool {
    if arabic > 3999 {
        return true
    }
    t.Log("testing", arabic)
    roman := ConvertToRoman(arabic)
    fromRoman := ConvertToArabic(roman)
    return fromRoman == arabic
}

If you run the test they now actually run and you can see what is being tested. You can run multiple times to see our code stands up well to the various values! This gives me a lot of confidence that our code is working how we want.

The default number of runs quick.Check performs is 100 but you can change that with a config.

if err := quick.Check(assertion, &quick.Config{
    MaxCount:1000,
}); err != nil {
    t.Error("failed checks", err)
}

Further work

Can you write property tests that check the other properties we described?
Can you think of a way of making it so it's impossible for someone to call our code with a number greater than 3999?
- You could return an error
- Or create a new type that cannot represent > 3999
  - What do you think is best?

Wrapping up

More TDD practice with iterative development

Did the thought of writing code that converts 1984 into MCMLXXXIV feel intimidating to you at first? It did to me and I've been writing software for quite a long time.

The trick, as always, is to get started with something simple and take small steps.

At no point in this process did we make any large leaps, do any huge refactorings, or get in a mess.

I can hear someone cynically saying "this is just a kata". I cant argue with that, but I still take this same approach for every project I work on. I never ship a big distributed system in my first step, I find the simplest thing the team could ship (usually a "Hello world" website) and then iterate on small bits of functionality in manageable chunks, just like how we did here.

The skill is knowing how to split work up, and that comes with practice and with some lovely TDD to help you on your way.

Property based tests

Built into the standard library
If you can think of ways to describe your domain rules in code, they are an excellent tool for giving you more confidence
Force you to think about your domain deeply
Potentially a nice complement to your test suite

Forem: Chris James

HTMX is the future

The current state of web application development

The costs of SPA

Hypermedia?

HTMX and hypermedia

Clojure HTMX TODO

The edit control

Embracing the web

How it supports non-JavaScript

Why is it The Future ?

It makes "the frontend" more accessible.

Less churn

Cheaper

More resilient

Reduced coordination complexity

More options

Wrapping up

Further reading and listening

Reduce WIP by practicing trunk-based development, rather than pull requests

The open source, pull-request model

High WIP

Slow integration

Unpredictable flow

Trunk-based development (TBD) and pair programming

No, I can’t trust 2 developers to take an idea from kick-off to production.

Improved flow, higher quality and less re-work

Risk

Trying TBD

Which is best for your team?

The TDD Thinking Hats

When writing a new test: The green, ambitious hat

But, I don’t feel confident

The existing test suite is a hot mess

I don’t really know what I’m trying to make

When the tests are failing: The red, uncomfortable to wear hat

Stop wearing multiple hats at once

When you’re refactoring: The blue, confident hat

Wearing the hats while pairing/mobbing

Do you wear these hats?

The Why of TDD

Introduction

What is TDD?

So, why test-driven development?

Why write tests first?

You have to know what behaviour you’re trying to build to write a test

It offers a fast and continuous feedback loop on your design

It gives you momentum and by focusing on internal quality will help you continue to change your software cheaply.

It does not slow you down

The qualities required to make code unit-testable are good qualities in themselves

It’s easier to write good tests beforehand

It requires you to constantly refactor

Digression: Refactoring sprints are an absurd idea and point to some real team disfunction

How to use TDD

Work iteratively, in small positive steps

When TDD feels right

Writing the test first

Start from the top down for new features

On “TDD-ing a design”

Listen to your tests

High number of test-doubles (mocks), complicated setup

Unclear intent

Wide scope

Value the quality of your tests

Be attentive when you watch the test fail

View being in “the red” as something to get out of as soon as possible

Commit your work frequently

Refactor aggressively when in the green

Wrapping up

How my website works

The approach

New things I learned

CSS Variables

Media query for dark mode

Fallback for older browsers

Fallback techniques

Performance

Wrapping up

Start naming your test doubles correctly

Quick primer on dependencies