Forem: Matthew Revell

[Boost]

Matthew Revell — Wed, 18 Feb 2026 14:35:34 +0000

The Future of Software Has a Lot More Builders. They’re Going to Need a Home.

Jon Gottfried for Major League Hacking (MLH) ・ Feb 18

Tales of regret from developer onboarding

Matthew Revell — Tue, 01 Feb 2022 14:01:00 +0000

Whether you're just starting out as a developer or you've got years under your belt, learning a new codebase is a defining experience.

Onboarding shapes not only your enjoyment of that project but can influence what sort of work you go on to do next. Either you're running screaming in the opposite direction, or you've found something you want to do again.

For every codebase onboarding that feels like an effortless walk in a sunny meadow, there are plenty more that evoke a 3am hike up an unlit rocky path during a hurricane.

Below is an inexhaustive collection of postcard-sized onboarding stories from real developers. They illustrate some of the specific ways in which onboarding can go wrong, and, at the end, there's a hint at one way we might make it better for the developers who come after us. Names have been changed.

I've suffered, so shall you

If a problem shared really is a problem halved, then perhaps that explains what happened to Mark.

The first day on the job, the incumbent developer asked a question, "How do you feel about spaghetti code?" Not leaving any time to answer, he followed up with, "Because what we have here is code soup."

The product should have been straightforward. It was a web front end to research data held in a PostgreSQL database. However, Mark was the fourth developer to work on the system in a short period of time. It was almost as though each prior developer had gone out of their way to avoid any consistency with their predecessor. Instead, they picked the tools and conventions they preferred. That led to fun situations such as Hibernate taking care of user and project records, while JDBC was used for general querying. As Mark says, "Why have one data abstraction layer when you can have two?"

Sure, the codebase itself was a mess, but the onboarding was essentially a surrender. "I know this is terrible, but I've suffered, so that means you should too," probably isn't the ideal onboarding maxim.

Mismatched expectations

Starting a new job is somewhat stressful, even in ideal circumstances. There's the pressure to show your best in those 30-60-90 reviews and to win round your new colleagues.

But for Gavin, a Python developer, starting one job, in particular, was especially fraught. "They knew I wasn't a Ruby developer but somehow also expected that I did know Ruby."

On top of learning the new codebase, Gavin also had to learn a new language. Not an impossible task but also one that no one at the company had made allowances for. "They pointed me at rubykoans.com and wished me well. The Koans are good, but they don't cover de facto essential stuff like Rails," Gavin said. After years in the Python ecosystem, he also found that the difference in approach between Python and Ruby docs took some getting used to.

For Gavin, it was the shock of the mismatched expectations, combined with his own desire to show his best work, that made this particular onboarding hard. "One person's experience of learning new tech can be worlds apart from your own: I was joining a company of Ruby enthusiasts who probably found it hard to put themselves into my shoes. I had 20 other things competing for my time and attention, as well as that huge pressure to be productive. Having to also learn Ruby, Rails, and the special way in which Rails was used at that company, before being able to be productive was very stressful. I came close to quitting or being fired."

Why can't you just understand?!

There are situations where a problem space is just plain hard. Some things take time to understand fully. But other times, it's not the problem that's hard to understand, it's more that the person getting you up to speed lacks the necessary empathy.

One developer, Gráinne, found that the person onboarding her was perhaps too close to the code to see it from a newcomer's point of view.

"On my first day, I showed up, got access to the code, and then had a two-hour sit down with the CTO. He was also the architect of the product and the lead developer. The trouble was that he talked very fast and got quite exasperated when you didn't immediately understand what he was saying."

For Gráinne that meant making whatever notes she could during the onboarding/interrogation and then trying to fill in the gaps by reading the code.

But even past the onboarding itself, a Monday morning could put things back to square one, "The CTO would often, over a weekend, massively change huge chunks of the code, breaking things in horrible ways, and then would shout about it until we fixed the mess."

We are unreliable narrators

Perhaps there's a theme that unites all of these stories: bad developer onboarding happens when there's no process or a process that fails to take account of people's different expectations and approaches.

Jeroen, a developer for a large multinational, put it this way, "Much of what we do in our business is communication. When we communicate, we're always referring to knowledge in our heads that we hope or assume is also present in the other party's head."

The problem is that it's hard for us to understand where that shared frame of reference ends and where we need to teach something completely new. Gráinne's CTO became frustrated when other people didn't understand his explanations because he couldn't imagine that other people had different experiences to his own.

Yes, we can grow as teachers and communicators. But there's a deeper issue. Onboarding shouldn't be an oral tradition.

Codebase onboarding as oral tradition

Let's take a moment to imagine something different.

You're sat around a fire with your family and close friends. The day has gone well. There is food, you have shelter, and yet there's a threat.

Some of the younger children have been wandering into the woods and eating mushrooms. Most have been fine, but one or two have become seriously ill.

You want to protect the children, but they don't understand why some mushrooms are bad, and others are just really tasty. You don't even know yourself. So, you tell a story to put it in a context you can all understand. You think on your feet. One of the elder members of your village can tell good mushrooms from bad. So, you say that all mushrooms have an evil spirit that only old people can cast out.

The story works. Over the years, it gets retold and embellished. Eventually, there's so much mythos around mushrooms that the original intent of the stories becomes forgotten. Even so, somehow, it continues to protect people, despite there now being a lot of unnecessary ritual associated with preparing mushroom-based meals.

Stories passed by word of mouth work act as vehicles for important knowledge. And we still use stories in our work as software developers.

Onboarding is a documentation problem

When everything is urgent, it's hard to prioritize work that will help you in the future. So it's somewhat understandable why onboarding a new colleague can be haphazard, clumsy, and inconsistent.

However, a bad onboarding is often actually a symptom of other problems. One particular issue is simply that many codebases lack good documentation. While individuals methods might have excellent comments, perhaps understanding of the whole system is less well documented. For example, there are teams where no two members would draw the same architecture diagram despite working on the same product.

At CodeSee, they've built tools that make it easier for teams to onboard new developers. And that's not by doing special extra work that just gets used in onboarding. It's by creating a shared understanding that benefits everyone who works on a codebase from onboarding and for the long term.

Regretful man photo by Nipan Chawcharernpon
Spaghetti by Maurijn Pach
Pomegranate seeds by by Tima Miroshnichenko
Angry cat by Pixabay
Scrabble tiles by Kathy Jones
Campfire by Jens Mahnke

Improve Your Developer Onboarding Path With the 10-3-10 Test

Matthew Revell — Tue, 18 Jan 2022 12:19:07 +0000

Getting to the point is essential for a smooth developer onboarding. Within seconds, you need to answer your product or project's what, why, and how.

But it can be hard to put ourselves in the shoes of a developer who knows nothing of what we do. That's why it's essential that we test and improve the onboarding path we give developers.

One way to do that is the 10-3-10 test. It's quick to perform, requires no special tools, and pretty much anyone can do it. Here's how.

From zero to Hello World in ten minutes

The idea behind the 10-3-10 test is pretty simple. A developer should land on your site without any context and achieve the Hello World state within 10 minutes.

Before a developer can get to that point, they need to know what the product does and then register to get a developer account.

Within all three of those milestones -- understanding, registration, and first product use -- lurks danger. At each stage, you're asking the developer to give you something on trust. At first, it's their time, then a small amount of information about themselves, and finally, it's both their time and their mental energy. It might sound over the top, but you need to reward the developer for each of those. Get the reward wrong, and the danger becomes apparent: the developer will lose trust and look elsewhere.

Drop-off is real

Work by Microsoft Research shows that typical web users decide whether to stay or go within 10 seconds of arriving at a web page. There is even an old study that suggests web page visitors form an opinion within 50 milliseconds of landing.

Keeping someone on the page long enough to explain the offering is just the start, though. Research on conversion rates by product analytics firm Heap suggests that 63% of SaaS users fail to complete product sign-up.

We are operating in a world of spam, clickbait, unregulated advertising, and other underhanded attempts to grab our attention. People have trained themselves to apply harsh relevance criteria to the information they find on the web.

Arguably, developers are not typical web users. Whether that makes them more or less patient will depend on their level of motivation, among other factors. What is clear, though, is that you have very little time in which to capture a developer's attention and then turn that into an interest in your product.

Three questions

There are multiple pressures on your developer onboarding flow. Product marketing might have a particular wording they want you to use. Someone in the growth team wants to try an experiment they read about recently. The VP of Sales is pushing for more data collection to assign new developer sign-ups to the appropriate salesperson.

Whether good or bad, the reality of these demands is that they can lead to a loss of focus. And in most companies, the demand for such change never really stops.

The 10-3-10 test is simple, repeatable, and quick, helping you measure whether you're continuing to serve developers. It asks three questions:

10 (understanding): Can a developer get a solid idea of what your product does within ten seconds of arriving on a developer-targeted landing page?
3 (registration): Can that person register for a developer account within three minutes of landing?
10 (first use): Can they get to Hello World ten minutes after first landing on the site? Whether you perform each test informally yourself or set up an ethnographic research session, it's worth diving into some of the factors that influence each stage first.

Understanding

How often have you been to a product website and come away none the wiser about what it does?

Competing demands too often lead to product pages that say a lot but communicate nothing. A common scenario is messaging that attempts to address too many different audiences at once. If a vendor compromises its message to serve both developers and, say, procurement managers, it'll serve neither well.

Sometimes, it's just that companies overthink how they talk about their products. As someone writing about a product, it can be hard to extract yourself from the context you have. It's tempting to build every comment, every meeting about the Chief Marketing Officer's vocab preferences, every barb thrown by the competition into that one very clever headline. But does that serve someone who just wants to know what problem your product solves and how?

The first step to passing the understanding test is to make sure you have a dedicated space to speak primarily to developers.

Next, stay specific and tell developers what your product does for them. Consider CodeSee's own home page. The headline and sub-head get straight to the action:

Visualize codebases faster

Map an entire codebase in just a few clicks.

The short follow-up paragraph then expands on that in a way that sticks entirely to the facts of how the product benefits developers.

Avoid the temptation to over-egg how you describe your offering. "SMS API," rather than "customer engagement solution." That kind of messaging has its place for certain audiences, and it makes product marketers feel good, but for developers, it's just another hurdle in the way of understanding whether that thing solves a problem that they have.

To pass the first test, a typical developer should understand the practical purpose of your product within ten seconds of landing on your developer home page.

Registration

Registration is a chasm in your developer onboarding. This is where developers must put the most trust in you by providing personal information without a clear idea of the pay-off. Provide them with guide ropes, way finding, and a hint of the reward that is to come.

First, how do you get a developer from the landing page to sign-up? The button or link you provide needs to be immediately obvious, not so wordy that people are tempted to skip it, and yet still communicate what's to come. Oh, and the copy must avoid scaring people off.

"Get started" and "Start now" are common choices. However, at least one study has shown that "Get started" attracts clicks, but not all are from people who are ready to try, download, or buy your product.

The Miro API's "Start building" is clearer. Click that, and you know you're about to get hands-on with the API.

CodeSee's "Try Maps now" communicates a lot in three words. "Try" tells you that you're going to be hands-on with the product while making it feel casual rather than intimidating. Naming the product enforces what's coming next, and "now" adds a sense that clicking will lead to a fast pay-off.

That's a lot for a button.

The registration process itself must stick to the principle of "minimum viable information". Sales and marketing colleagues might push to capture more and more information during registration, but your primary aim here is not to add another lead to the CRM. Instead, registration is about enabling access and then allowing the rest of the experience and product to win over the developer.

That doesn't mean you can't ask for more than just an email address and password. Most people have some tolerance for providing extra information. However, you should aim to frame at least some of those additional questions in terms of how they'll benefit the developer. Twilio's registration process is great for this. It asks some questions that might help their sales funnel but also collects information, such as your preferred programming language, that makes the developer dashboard more helpful.

To pass the second test, a typical developer should be able to get a developer account within three minutes of landing on your developer home page.

Hello World

Providing a smooth path through registration is good, but it's only a staging post. The developer's goal is to see for themselves whether your product solves their problem and whether they like the idea of working with your company.

There is enormous scope for ways to improve the path to Hello World, so let's focus on one thing. When the developer first logs into your dashboard, take them by the hand through the steps they need to achieve something worthwhile.

Such a first use experience is more than flashing up a quick-start guide or a click-through signposting of the dashboard's UI. It is a highly focused path that takes the developer from no knowledge of your product to that Hello World interaction. It builds on the supplemental information you've gathered during sign-up. Are they a Ruby developer? Great, get them to download your SDK's gem. They told you they work for a large company? Suggest they link their new account with Okta or other SSO providers to simplify integration with corporate sign-on.

That highly focused first use experience should contain everything the developer needs to get a feel for your solution and what it's like to develop with, in as efficient a way as possible.

If a typical developer can get to Hello World within ten minutes of the first landing on your developer pages, you've passed the 10-3-10 test.

And there's more

The 3-10-3 test is a blunt instrument, but it's useful as a way to keep us honest about whether our onboarding serves developers. It works well for commercial products, and there's plenty of scope to adapt it to onboarding a developer onto a team building a product or a developer joining an open source project.

Microscope picture by Public Domain Pictures

Building an app business using Heroku and Salesforce

Matthew Revell — Tue, 01 Dec 2020 16:22:07 +0000

The past decade was the era of the mobile app store.

Brilliant ideas, well executed, made fortunes for indie developers who were able to reach audiences of billions from their spare rooms, garages, and co-working spaces.

Thanks to the Google Play Store and Apple’s App Store, the best apps could reach users on merit rather than marketing budget.

But those days are gone.

Today, mobile app stores are dominated by big name vendors, while everyone else fights for a handful of ad dollars among millions of competitors. And mobile users are fickle. Most apps are uninstalled within a week of being used.

There is good news, though. We are now in the era of the business to business app marketplace, such as the Salesforce AppExchange. Motivated customers with long-term needs and realistic budgets are actively looking to buy solutions. So, what is it like for a solo developer or small team to develop such a business?

The size of the opportunity

Before we dive into the specifics of developing for an ecosystem such as Salesforce, there’s the question of whether it’s going to be worthwhile.

The short answer is: yes, absolutely.

Analyst firm IDC forecasts that, by 2024, the Salesforce ecosystem will create 4.2 million jobs and contribute $1.2 trillion in new business revenues to local economies.

Crucially, the majority of the value created by Salesforce is going to partners, such as companies making apps for the Salesforce AppExchange. In fact, for every dollar that Salesforce generates, its partners earn six dollars.

Okay, big numbers aren’t always relatable so let’s look at the opportunity in terms of relative growth. Spending on cloud software is predicted to grow 19% each year over that period. For non-cloud, it’s just 3%. If you’re creating something new, it stands a better chance of success if you deliver it as SaaS. And a marketplace will help you reach willing customers.

Look, I've got my stack and I like it

Even if you haven’t used Salesforce, then you’re almost certainly aware of its reach. All those thousands of organizations using Salesforce to run their day to day business offer a potential audience for app developers.

So, the opportunity looks good but how do you actually build an app for one of these marketplaces? Perhaps you’re thinking that it means getting used to a bunch of exotic languages and frameworks.

In the case of Salesforce AppExchange, you use technologies that you’d find anywhere else. Let’s start with the back-end. If you usually work with Ruby, Python, Go, Java, or pretty much any other open source language, then you can use that to do most of your back-end’s heavy lifting on Heroku. Add in Heroku Postgres, Heroku Redis, and tooling such as Apache Kafka on Heroku, and you have a standard, modern web stack.

So, where does Salesforce come in? As Heroku is part of the Salesforce platform, you can easily sync data between a Heroku Postgres database and one or more Salesforce orgs. In effect, your app’s back-end can live mostly on Heroku and synchronize with Salesforce when the need arises.

When it comes to the front-end, you’ll need to work more closely with Salesforce itself as your users will interact with your app from within Salesforce’s UI. Even so, the tools you’ll be using will be immediately familiar. Lightning Web Components is a JavaScript front-end framework much like ReactJS, with ready made elements for common scenarios and the flexibility to create your own when you need to.

In fact, this is another place where mobile development offers a good analogy. If you’re building a mobile app you’ll need to use specific APIs for the front-end, while the back-end can be designed however you like.

How SharinPix built their business with Salesforce

SharinPix is an image management app built specifically for the Salesforce AppExchange. It began life when its founder, Jean-Michel Mougeolle, needed to process photos taken by field agents working for his then company. The agents would upload the images into Salesforce but the company needed to resize them and add supplemental data such as automatic annotation and watermarks.

The challenge for Jean-Michel was that the core Salesforce platform didn’t lend itself to general development tasks, such as image manipulation. However, if he could use a standard web development framework, such as Ruby on Rails, then he could plug into standard image manipulation libraries quite easily.

Already familiar with Heroku, Jean-Michel built a proof of concept using Heroku Connect, which synchronizes data between Heroku Postgres and Salesforce. On Heroku, he had a Rails app that would take images from Salesforce, manipulate them, and then sync the processed images and metadata back into Salesforce. Crucially for Jean-Michel, no one who tried the demo app had any clue that Heroku was involved; as far as they were concerned, it was a standard Salesforce application.

Shortly after, Jean-Michel spun SharinPix out of his original company and today has a team of developers serving 20,000 users. And while the development side has given SharinPix the best of both worlds -- standard web dev on Heroku, with easy integration into Salesforce’s UI -- the company’s success has been as much about the route to market that the AppExchange provides.

“AppExchange is fantastic,” says Jean-Michel. “Customers know that our app is secure and trustworthy because it has the Salesforce stamp of approval. Discovery is also very good. People can search for image manipulation then find our app page, where they can watch videos, read about the app, and start a trial. We don’t have to play the SEO game or spend on ads, because we have a ready made audience.”

Getting your start in the Salesforce ecosystem

The next big opportunities for developer entrepreneurs might not be in the most obvious places. B2B marketplaces are growing, have audiences that have proven their willingness to spend money, and provide an ecosystem of tooling, marketing support, and more.

So, where do you go from here? If you’re interested in building for the Salesforce ecosystem, then there’s a wealth of learning materials on the Trailhead learning site, as well as a friendly and helpful community of fellow developers.

Mug image by Danielle MacInnes

Safari icon photo by Brett Jordan

How to help Hacktoberfest contributors become long term community members

Matthew Revell — Tue, 29 Sep 2020 16:55:36 +0000

Hacktoberfest is just around the corner and soon thousands of people will make their first contributions to open source. For many projects taking part that’ll mean a flurry of pull requests and then a return to normal. In some cases, though, Hacktoberfest contributions have led to ongoing involvement with an open source community.

That chimes with one of Hacktoberfest’s values: Short-term action, long-term impact. And, as Josh Simmons (Salesforce’s Senior Open Source Strategist, as well as President of the Open Source Initiative) puts it:

“Drive through contributions are underrated, but of course we should strive to retain contributors. Reward exploration and make contribution opportunities plain to see! Sometimes it's just not clear how to answer the question: what next?”

So, if you run an open source project, how can you help Hacktoberfest contributors take that next step in your community?

Why do people contribute to open source?

Perhaps the first question to ask is, “Why do people contribute to open source?”

We can find one take from the roughly 16,000 people who answered Slashdata’s survey question on that very topic.

Twenty nine per cent of respondents say they contribute to open source in order to improve their coding skills. The second biggest motivator was a belief in the principles of open source. Third, representing 22% of respondents, was simply, “It’s fun”.

Slashdata’s findings chime with academic research into the motivations of open source contributors. Eighteen years ago the authors of the paper Working for Free? Motivations for Participating in Open-Source Projects identified similar, although not identical, motivations:

Intrinsic motivation: a sense of personal fulfilment deriving from contributing, self-improvement, etc
Altruism: the desire to do good in the world
Community identification: finding satisfaction in being part of something
Future rewards: improved employability, peer recognition, revenue from associated services
Personal needs: perhaps the ancestor of open source motivation, i.e. “scratching your own itch”.

Broadly speaking, people take part in open source for social, employment, education, and ethical reasons. And, perhaps it goes without saying, each of these fits within Daniel Pink’s autonomy, mastery, purpose framework of motivation.

However, Hacktoberfest is different compared to open source as a whole because it encourages contributions that wouldn’t happen otherwise. So, we should also look to understand what motivates Hacktoberfest participants specifically.

Why do people take part in Hacktoberfest?

According to the Hacktoberfest organisers’ own research, 46% of participants in 2019’s event were first time open source contributors.

Despite that, their motivations are similar to those of the broader open source contributor community. More than three quarters of Hacktoberfest participants said that they took part in order to learn. Almost half said they were motivated by a desire to have fun.

Why do people quit open source?

Understanding why people contribute to open source is only half the story. No matter how well you meet people’s initial motivations, you’ll unnecessarily lose community members if you fail to understand the frustrations and frictions that build up over time.

According to Digital Ocean’s Currents research series, the top three reasons for reducing involvement in open source are:

“It’s too difficult to contribute”
“It takes too much time to contribute”
“I’m burnt out from contributing”.

Arguably, the second and third reasons are just different ways of saying, “It’s too difficult to contribute”.

And while you should minimise the friction that contributors face, it’s worth accepting that some people simply lose interest. Rather than fight against natural attrition, you should develop a graceful way for people to leave the project. Acknowledging that people move on, and that is okay, will make people feel good about handing over their responsibilities in a smooth way.

Answer the call

So, then, how does understanding people’s open source motivators and demotivators help create a path from Hacktoberfest contributor to long term community member? Broadly speaking, the research suggests that Hacktoberfest participants want to enjoy themselves while learning new skills and doing something worthwhile.

If you want your project to provide a longer term home for those Hacktoberfest contributors then you need to create an environment where people can meet those needs repeatedly and outside the framework of Hacktoberfest.

But if that sounds like extra work then don’t worry. The things you need in place for a good Hacktoberfest experience are the foundations of a healthy open source contributor experience:

reduced friction when contributing to your project
meeting contributors at their level of ability
helping people feel a sense of accomplishment
engendering a sense of belonging.

What does that look like in practice?

Reducing friction

As Hacktoberfest happens mostly on GitHub, your contributor experience begins with your README. Vonage’s Lorna Mitchell describes your README as your documentation landing page and that’s right for users of your project. For contributors, you should provide a very visible link from the README to your CONTRIBUTING or CONTRIBUTORS file.

CONTRIBUTORS is an opportunity to lay out the norms and expectations of your project in a way that sets newcomers up for success. You don’t want contributors to discover your copyright assignment rule only after they’ve submitted a pull request, for example.

However, CONTRIBUTORS has an equally important role when it comes to helping people find a longer term home in your community. This is your opportunity to show the humans within the community. If participants can feel a connection to other people, rather than simply writing some lines of code, then your project will have a greater impact on them. Use the CONTRIBUTORS file to link to your community spaces -- whether that’s a forum, real time chat, or something else -- and encourage newcomers to meet your community there.

Meeting people at their level

If almost half of Hacktoberfest participants are new to open source contribution, that means that more than half have at least some experience. That raises a challenge: how do you provide a great first experience for people of different levels of experience?

The team at Sendgrid asked themselves that question when they participated in Hacktoberfest 2017. Their solution was to calculate story points for each issue and then apply an easy, medium, or _hard _tag. That’s a fairly low cost way of meeting contributors at their level.

An additional approach would be to sketch out possible solutions within the issues, to give contributors pointers to help get them started.

Recognise and reward

Hacktoberfest itself is about creating an encouraging and rewarding environment for contributors, but you can make your project more memorable by offering your own rewards?

Remember Daniel Pink’s framework of motivation? In his book Drive, Pink writes about the importance of autonomy, mastery, and purpose as intrinsic rewards. Sure, people will respond to free swag in the short term but what really drives them is the impact their contribution has on their sense of self and their place in the world.

So, yes, provide swag that’s exclusive to the people who contribute to your project. But remember that’s a short term win, when it comes to motivation. If someone contributed in order to have fun, or learn, or do good, then that’s where you should focus. A free t-shirt can help with a sense of tribal belonging but it can’t help people find a long term home in your community.

For the longer term, you should create a process that recognises and rewards people’s contributions, that makes clear to them what impact they’ve had. Build relationships, not swag budgets.

Open source isn’t just about code

Code has a privileged position in open source. But code isn’t the whole story. Arguably, code is just the end product of a larger process.

It’s a bit like how the singer in a band often gets greater recognition than their bandmates. Even if the drummer wrote all the lyrics and the bass player wrote the music, it’s the singer who takes pride of place on posters and gets interviewed about their thoughts on the world. It’s not a perfect analogy but, in a similar way, writing code is a highly visible and valuable task yet it stands on the work of others.

User research, UX design, community management, technical writing, visual design, accessibility enablement, marketing, localisation, and many other roles play their part in a healthy open source project. Ubuntu did an excellent job early on of encouraging diversity of roles within their community, enabling the project to have a greater impact both for its commercial sponsor and for the people who took part.

Hacktoberfest is an event that focuses on code but bear in mind that people taking their first steps into open source may not want code to be their long term contribution. Organise your project in such a way that role diversity is celebrated and enabled.

Good luck!

Remember, Hacktoberfest is primarily about fun and learning. If your project benefits from getting some issues fixed, then that’s great! If you find that a small number of Hacktoberfest contributors become long term members of your community, then that’s a nice bonus.

From an organisational perspective, Hacktoberfest is about more than helping people contribute to your open source project. Companies are increasingly taking the opportunity of Hacktoberfest to introduce an open source culture. The open source programmes office at Indeed, for example, has used Hacktoberfest in recent years to encourage open source contribution from their own engineering team. Salesforce are doing something similar. Back to Josh Simmons:

“At Salesforce, we're using Hacktoberfest as a way to activate new contributors internally, but that's just the first step–we want to build a culture of open source contribution.”

But why? Clearly there are many reasons why companies large and small benefit from contributing to open source. One strong reason, as identified in Slashdata’s research, is that almost half of developers expect companies to contribute to open source. If nothing else, having a healthy open source programmes office can act to attract developers to your team.

Whether you’re running a project, contributing, or encouraging others to contribute, Hacktoberfest can be an intense few weeks. So, get some rest because October will be busy.

Phone photo by Paweł Czerwiński

What is a commit log and why should you care?

Matthew Revell — Tue, 08 Sep 2020 14:46:00 +0000

Logs are everywhere in software development. Without them there’d be no relational databases, git version control, or most analytics platforms.

For many developers that’s the end of our relationship with logs. We use tools that rely on logs. We turn to logs to understand why something went wrong. And up until fairly recently, you could probably go an entire career without thinking much more deeply about them than that.

Commit logs, in particular, have come out of the wings and into the spotlight. At the heart of tools such as Apache Kafka, commit logs are essential to handling increased volumes of data, bringing coherence to distributed systems, and providing a common source of truth for microservices architectures.

Understanding how commit logs work will help you to build more resilient systems using tools such as Kafka.

It’s a log and that’s about it

The good news is that commit logs are about as straightforward as it gets. They are a sequence of records, each with its own unique identifier. Want to add a record? No problem, it gets appended at the end of the log. Want to change an existing record? That you can’t do; once written, records are immutable.

What about reading? That always happens from left to right. While there’s no query, as such, you can use offsets to specify the start and end points of your read.

And that’s it. We can all go home. Well, perhaps not just yet. As often, the raw “what” is far less interesting than the “why”.

Logs are there even when everyone else has left you

Commit logs have been around for a long while and that’s because they solve a core problem in software development. They provide a source of truth for what has happened in a system and in what order.

Let’s think about a relational database such as PostgreSQL. In Postgres, commit logs are called write ahead logs. Each write to a Postgres database must first be recorded in the write ahead log before the data is changed in either a table or an index. The first benefit is that it speeds up database writes.

Writing to a commit log is relatively fast, even on disk. Writing to more complex data structures, such as a relational table, is necessarily slower. So long as the transaction is recorded in the write ahead log, the changes to indexes and tables can happen in memory, with a page to disk later on. That way, the slow part of writing to the database happens asynchronously. Even if the plug is pulled from the server before the table and index changes are written to disk, the database can be recreated by following the story of what happened and in what order as recorded in the write ahead log.

The bigger benefit, perhaps, is that any database can be recreated from scratch simply by following the write ahead log. Whether that’s as part of disaster recovery or for the purpose of streaming live changes to a read-only replica.

Now, what if that commit log could serve the same purpose and not just for a database but, instead, an entire software architecture?

Logs can handle volume

Let’s say you’re building an ecommerce application. You want to better understand how customers make their way around the site and so you record every click, every search term, every page view.

Regardless of how you later process that data, you first need to capture it. Perhaps the quickest solution to develop would be to store each event in your operational database. But that puts you on the wrong side of a trade-off. The operational database -- most likely a relational database -- has a relatively slow write time. That’s fine because it gives you benefits such as rich query, transactionality, and mutability. But right now all you want to do is capture the data and worry about what to do with it later.

One option would be to put Redis in front of your operational database, so that it can soak up the volume and release slowly at a rate suited to the main database. However, that’s just offsetting the problem and leaves your relatively expensive operational database full of semi-structured data. The visitor data is coming through in huge volumes and there’s no point in increasing the cost and complexity of your operational database in order to handle “nice to have” data such as this.

A commit log is ideal, though. The data you need to store is made up of discrete, ordered events. And the simplicity of commit logs means that they can easily handle far larger volumes of data than a typical relational database.

In fact, this is a typical use case for Apache Kafka, which puts a commit log at the heart of its data model.

Logs are a source of truth

Logs are fast, they are simple, they handle large volumes of data, and they are a source of truth. This makes them ideal for situations where multiple autonomous components form a single system. That could be a full blown microservices architecture or a monolith where you’ve hived off one or two processes.

In these situations, a log does two jobs. One is that it provides a single place for all parts of the system to find changes. The other is that it imposes an order on those changes simply by the fact of how it works. Let’s go back to ecommerce to explore what this means.

Say we have a simple ecommerce application made up of five distinct services: catalog, payments, orders, customers, and shipping. In this situation, a customer order might take this path:

The order system writes the customer’s order to the log.
The catalog system reads the order from the log and checks that there is sufficient stock to fulfill it. It finds there is and so temporarily reduces the stock count accordingly and writes a new record to the log indicating that the order can be fulfilled.
The payment system reads the record showing that the order can be fulfilled and attempts to take payment from the customer’s card. The payment is successful and the payment system writes back to the log that the order can progress.
The shipping system reads the log and prepares the order for dispatch.
The customer system reads the same entry and updates the customer’s record with details of their new order.
A dispatcher in the warehouse marks the package as sent in the shipping system. The shipping system writes to the log that the package has shipped.
The customer system sees that status change from the shipping system and updates the customer’s tracking information accordingly.
The catalog confirms the earlier temporary reduction in stock level.

In reality, there would be many more steps. For example, there might be an entire microservice that kicks into life when a stock reduction message appears in the log and reduces the stock level in the operational database by the right amount. The point is, though, that each of these systems never interacts directly. Each one writes to and reads from the log. That has several benefits.

With the log as the source of truth, there’s no danger that one service might accidentally pre-empt or overrule another because everything happens in order. Another benefit is that, so long as the log is available, the system can theoretically continue even when some components are offline.

Committing to logs

You could build your own log but you’re likely to get a better return on your time using something like Apache Kafka.

Kafka gives you the central benefit of a commit log -- the immutable, ordered record of events -- with the benefits of integrating with common data sources, writing data out to other systems such as Postgres, and the ability to act on and transform the data it processes. There are SDKs for most common languages and if you use a hosted service you don’t need to worry about administering your own Kafka cluster. And, believe me, that's no small task; take a look at how Heroku manages their huge fleet of Kafka instances.

Logs photo by Oliver Paaske
Ringbinders photo by Viktor Talashuk
Truth photo by Magda Ehlers

How to run multi-tenant Kafka

Matthew Revell — Mon, 22 Jun 2020 14:33:47 +0000

Apache Kafka is a beast. Forget for a moment what it actually does. I’m talking about running it in production. Even experienced teams find that getting the most out of Apache Kafka can become a serious time sink.

At Heroku, their DevOps team looks after Kafka on behalf of thousands of developers through the Apache Kafka on Heroku service. Not everyone can justify the expense of their own dedicated Kafka cluster, though. To make Kafka available for testing, development, and smaller production use cases, Heroku offers access to multi-tenant clusters.

If running just one Kafka instance is a full-time job, though, what does it take to operate multi-tenant Kafka clusters at Heroku’s scale?

Making the right trade-offs

Running anything multi-tenant is about making the right trade-offs. But some compromises are off the table. Security, as you’d expect, isn’t up for discussion.

Performance, too, must be good enough that someone could use a multi-tenant Apache Kafka on Heroku cluster as part of their production stack.

So, what about functionality? Perhaps Heroku could reduce the number of brokers or Zoopkeeper instances. But that, too, is a compromise too far. There’s little use in a testing or development instance that behaves differently from a production cluster.

So, what’s left? How do you run secure, performant, fully functional multi-tenant Kafka at a fraction of the price of a dedicated Kafka cluster? By using Kafka’s own functionality, plus some of Heroku’s, to securely divide a single cluster between multiple customers.

Oh, and lots of planning, testing, and automation.

Security through isolation

Security in multi-tenant environments usually boils down to, “Can other people see my stuff?”

The Heroku Kafka team handles this problem in two main ways. The first solution is to use Kafka’s access control lists (ACLs). They’re enforced at the cluster-level and specify which users can perform which actions on which topics.

Second, Heroku uses namespaces to separate each tenant’s resources. Let’s say you add multi-tenant Kafka to your Heroku account. The Heroku provisioning system automatically generates a name -- something like wabash-58799 -- and associates it with your account when it creates the Kafka resource. After that, Heroku verifies that your account is associated with the right resource name each time you perform an action on Kafka. That way, only your account can access any activity on that resource, providing another level of security that is unique to Heroku.

Staying on top of noisy neighbors

Just as one tenant must not be able to access another’s data, all tenants of a cluster must have fair access. So, even if another customer is processing huge numbers of events, it should not disturb your usage.

Heroku uses Kafka’s built-in support for quotas on producers and consumers, meaning that there is a fixed limit on the number of bytes each tenant can read or write per second. That way, every user gets their fair share of the computing resources available.

Maintaining availability

Noisy neighbours are a solvable problem. However, some multi-tenant services make it almost impossible to avoid them.

Think about traditional shared hosting offerings, where they promise the Earth for $3 a month. Much of the time, they’re overprovisioning. Squeezing a thousand customers each expecting 100 GB of disk space onto a machine with a 1 TB hard drive works only if most customers use only a fraction of their full allocation.

Heroku’s multi-tenant Kafka immediately provisions the full set of resources purchased. So, you don’t end up with a hundred people all trying to use the same gigabyte of disk space. And even if a customer does go beyond their disk quota, for example, Heroku will automatically expand their limit while emailing a notification that they need to upgrade their service.

Availability is basically about setting sane defaults, like this. Have the system behave in the way that maximizes its usefulness. Often that means provisioning more than the Kafka defaults. For example, higher partition settings (from one to eight, to maximize throughput), additional replicas (from one to three, to ensure data is not lost), and more in-sync replicas (from one to two, to truly confirm that a replica received a write).

Testing for the real world

The saying goes that being prepared is half the battle. Knowing what could go wrong enables you to avoid those problems before they happen.

The Heroku team have run extensive tests on their multi-tenant Kafka offering to simulate real-world usage, failure scenarios, and extreme workloads. For example, hammering a cluster with a million messages, then taking one of the brokers offline to see what happens. Or operating a cluster normally then stopping and restarting a server to check that failover works.

Those one-off tests have developed into a test suite that creates an empty cluster then generates fifty users. Those users attach the Kafka add-on to their application and then create several producers and consumers each. From there, realistic usage profiles are assigned, such as having 10% of the test users generate very small amounts of traffic, while 20% send very large messages at slow speeds, and so on. Then, the tests gradually increase the number of users to determine a multi-tenant cluster’s operational limits.

Through that testing, the Heroku team identified issues before they became a problem for real users. There’s more detail in a talk called “Running Hundreds of Kafka Clusters with Five People.”

Kafka in more places

For most development teams, getting the benefits of Kafka without having to actually run the Kafka cluster is the ideal situation. You’re free to focus on building your product without worrying about learning the ins and outs of yet another platform. Multi-tenant Kafka takes that a step further by making it affordable for situations where a dedicated cluster is overkill and yet where Kafka can have a benefit.

There’s more about what the Heroku team have learned from working with Kafka over on the Heroku blog.

Cover photo by Sophie Dale
Stormtrooper photo by Liam Tucker

Three ways to do good with a side project

Matthew Revell — Wed, 22 Apr 2020 12:32:08 +0000

Side projects make the world go round. Think about it. You can learn new skills, try out ideas, and take paths you never normally would. When you’re ready, you can apply what worked to your day job.

Or, maybe, the side project will become the day job. Slack, Twitter, and Docker each began as tools that were essentially side projects.

Of course, not every side project ends up changing the world quite like that. But side projects can change lives. Developers around the world are using their side project time to donate code and mentorship to good causes. If that sounds interesting, then here are some ways that you can find projects in need of your skills.

The United Nations

Since 2000, the United Nations has been one of the pioneers in matching online volunteers with projects around the world.

The UN Volunteers program provides an easily searchable database of volunteering opportunities with projects focused on peace and development goals. Whether it’s creating an online forum for young people in north west Africa, developing a schools management system for a non-profit in Burundi, or PHP development for an NGO in Latin America, the tech opportunities through UN Volunteers each relate to focused projects with a clear end goal and estimated project length.

The program is about more than only matching volunteers to projects, though. They provide tools to help manage each collaboration and guidance to ensure a good experience for both sides. With 12,000 volunteers each year, the program offers a good first step for anyone looking to volunteer their tech skills for the first time.

Social Coder

While code-based opportunities are just part of the UN Volunteers program, there are organizations that focus purely on volunteering for software developers.

Social Coder connects developers with charities who have specific projects. Charities shares details of their projects with Social Coder, who then share them with developers who have registered their interest.

Sometimes it can be hard for non-developers to understand the effort and value involved in creating software. Social Coder aim to address that by assigning a market value to each project they administer. No money changes hands but Social Coder say that it’s important that the beneficiaries of the volunteer work get an idea of the scope of work donated.
Past Social Coder projects have helped Medecins Sans Frontieres, the Homeless World Cup, and Street Doctors.

Covid-19

If you’re looking for a way to help specifically with the current Covid-19 situation, then you’ll find opportunities with the Code Against Covid-19 project. Organized by Codementor, the project aims to connect developers with programs combating the virus.

Projects currently using the platform include Operation Masks, which is working to deliver 500,000 masks each day to front line medics in the US. There’s also Covid Watch. That’s a project organized in collaboration with Stanford University and the University of Waterloo. It uses a smartphone app to track who has been in contact with whom by monitoring their Bluetooth signals. Crucially, it does this anonymously. If someone tells the app they have Covid-19, it can then alert any other app users who have been in contact with them.

Developers can also register with the platform to let Covid-19 projects know that they’re available for free or at reduced rates.

Unleash some happy accidents

A lot of the commentary around software development, especially amongst start-ups, is about achieving outsized financial gains. Some would argue that the best way to help good causes is to get rich and then give away your money.

Let’s face it, while many people in tech make a good living most of us won’t be setting up a foundation any time soon. If you have some spare time and you want to use it for good then volunteering your skills can make a difference to people in need. And, who knows, maybe you’ll stumble across a happy accident that results in even more good. I'd love to hear of other ways that people have found to use their side project time to contribute something to those in need.

Mug photo by Nathan Lemon

Coding Against Covid

Matthew Revell — Tue, 21 Apr 2020 14:19:10 +0000

Somehow, staying at home doesn’t feel like it’s enough. It’s essential but there must be more.

Across the world, people have picked up the tools most familiar to them. It’s a natural instinct. We want to make a difference at a time of crisis.

Musicians have live streamed performances, comedians have self-filmed sketches, and developers have written code to help make sense of our situation.

People have put together tools to help improve our understanding of what has happened. Others have launched projects to fix a problem in our response to the pandemic. Then, there are crowd-sourced guides for those of us directly affected either by the virus or the fallout from lockdown.

Heroku’s role has been to carry on as normal, keeping apps running and data flowing. Some people who have built projects in response to Covid-19 have chosen Heroku to deploy their software. Here are three such projects. As you might expect, this isn’t an endorsement of any of them.

Measuring the impact of social distancing

Sat at home, it can be hard to see the impact of social distancing. A group from Element AI have put together a tool that helps put it in context.

The Corona Calculator visualizes recent infection and related data from countries across the world. Select your country from the drop-down and you can see what has happened so far.

Most interestingly, though, is the ability to see how social distancing might change future infection rates.

Using two sliders, you can adjust the number of people that the infected person might come into contact with each day. The predicted number of cases in your country changes accordingly. Putting it into unambiguously practical terms, the resultant peak number of hospitalizations is shown alongside the number of hospital beds in your country.

Getting PPE to health workers

Lack of personal protective equipment for health workers and others on the coronavirus frontline has been a problem in a number of countries. A group of volunteers came together in March to create a global effort to get face masks, and other protective equipment, to healthcare workers and first responders.

Masks for Heroes connects people who need face masks with people who are able to donate it. Requests for masks are tracked as dropped pins on a Google map, enabling donors to quickly find need in their local area.

How to live in a pandemic

The Coronavirus Tech Handbook is a crowd-sourced collection of resources for navigating life during the pandemic, originated by the London College of Political Technologists.

Each section of the guide focuses on the needs of a particular group, such as parents and guardians or software developers, offering a mix of original content and links to resources elsewhere on the web.

Anyone can get involved with editing the handbook, with each section having its own WhatsApp group to help coordinate work.

Making a contribution through code

How each of us reacts to this pandemic is, of course, personal. For some, it’s a time of reflection. For many it’s a period where life itself is uncertain. For those of us able to stay home and who are well, it’s a time where we might have newly free time along with an urge to do something.

Building software that helps make sense of the pandemic can be a constructive way to channel our focus and feel that we’ve contributed. I’d love to hear from other people who are building projects, big or small, related to the current situation.

Don't complain about pager alerts. Change something.

Matthew Revell — Tue, 14 Apr 2020 10:34:54 +0000

The cliche goes that artists must suffer for their art. But as a developer, should you also expect to suffer?

Y’know, there’s something to be said about the motivating experience of getting woken up by a page in the early hours. You’ll never let that happen again. But unless you’re building specifically to minimise your exposure then all you can apply are sticking plasters.

So, what does a framework for avoiding those dreaded 2AM page alerts look like?

There’s a lot of advice out there and the best of it falls into three groups:

Be intentional
Be honest about risk
Buy, don’t build.

Even then, most of that comes down to one key take away: the resilience of your platform in getting value to users is the only thing that matters.

There are no prizes for writing your own framework in an exotic language or tuning every last aspect of your server OS. Shipped code is everything.
Let’s unpack each one.

Be intentional

Starting a new project is like opening a toy box. There’s a world of possibilities. New technologies jostle for attention alongside old favorites. You get to learn by doing.

Always wanted to try a NoSQL database? Great! Now, you can. Bored of React? Try VueJS, instead!

But before you do, ask yourself a question: are you building a potential business or are you playing? Either answer is fine but the outcome is different.

If learning something new is really what you want to do then create a hobby project. Playing with novel tech is great. But love it for what it is. It’s a chance to learn, not a business in the making.

If delivering value to users is your main goal, then you need to be sure of how that impacts your technology choices. If you were an architect building an extension to your own house, you might experiment with unusual materials and techniques. If you’re building a new apartment block for clients, then you’d stick with tried and true methods that meet the brief, the budget, and the deadline.

So, when you come to open that toy box, be intentional in making choices that simplify delivering value to your users.

Be honest about risk

Whether you're bootstrapped or funded, building a business is risky enough. Choosing a battle-tested tech stack is just one part of reducing risk.

When we’re excited about a project, especially one that we’re building into a business, it’s easy to misjudge risk. To some extent, that’s healthy. Focus only on what could go wrong and you’d never open your laptop.

I may be misremembering it, but there’s a saying that goes something like, “forewarned is good for your forearms”. Either way, knowing up front what might go wrong gives you a fighting chance when it does go wrong.

Mapping out your risks doesn’t have to take long. Get a whiteboard, take each major part of your stack in turn, and write out anything that’s a realistic risk. Even thinking about the obvious stuff will help you come up with solutions.

Buy, don’t build

The last part is the easiest because it requires you to do less. Think back to why you’re doing this. There’s some unique value that you can bring to the world. Your one job is to deliver that value as efficiently as possible.

If building your dream means working evenings and weekends, why would you want to spend them reinventing the wheel? And this isn’t just the obvious stuff. You probably haven’t considered writing your own frontend framework and no one is writing server side code in Assembly language. But building your own server infrastructure and spending a chunk of your time on devops has become normalised in our culture.

Outsource. Buy, don’t build. Find the fastest route from successful pull request to seeing that code in production. Deploy your product on a platform like Heroku, Google Cloud Run, or AWS Elastic Beanstalk. The point is to find the right tool for the job and then pay someone else to look after it, rather than deploy your code to a stock VM image that you need to manage into the wee hours.

As the solo dev on a project, it’s your responsibility to stand on the shoulders of all those developers who went before you. Take advantage of other people’s expertise. If, later on, you need to swap a component out for something you’ve written yourself then take that decision when the need arises. It’s better to launch with off-the-shelf components that work rather than get stuck never finishing a home-grown solution.

Sleep well, most of the time

Bugs happen. Third-party systems go down. Corner cases become startlingly apparent at the moment you realise you hadn’t considered them.

None of the advice here is to say that you can avoid the unknown. Instead, it’s about reducing your exposure to risk. Choosing Rails, Spring, or Django gives you the backing of a worldwide cohort of developers who have solved the same problems you’re working on. Mapping your risks gives you time to address them before they become a problem. Buying-in tooling and platforms means you can trust specialists to look after the commodity parts you need but that don’t make a material difference to your end users.

There’s a shared duty in a small development team. So, there will be times when you get paged out of hours. However, the tech stack you choose has a significant impact on the reliability of your product. Boring tech makes the world go round. Learn the weird stuff through side projects. Choose proven tech for work and you’re more likely to get a better night’s sleep.

Pink clock photo Mpho Mojapelo
Toys photo Vanessa Bucceri
Whiteboarding photo This is Engineering RAEng

Two million ways to use Postgres

Matthew Revell — Tue, 31 Mar 2020 13:09:56 +0000

You need a database. Maybe you need more than one.

Fifteen years ago you’d choose MySQL if you were from the open source camp, SQL Server if you were on the Microsoft stack, and Oracle if you were operating in a large enterprise.

Today, we’re spoilt for choice when it comes to storing, querying, and manipulating data. But despite all the noise around NoSQL, Postgres has quietly come to be the default data store for many developers.

So, what is it about Postgres? Why has it become the preferred choice for much of the Rails community? What led Microsoft to buy a major Postgres vendor and Amazon to migrate from Oracle to largely Postgres-based data stores?

When I was looking for an answer, I have to be honest and say that I decided to be lazy. Rather than hunt around the web, I decided to look at one place where there were lots of Postgres installs. With more than two million managed Postgres databases, Heroku seemed like the perfect place to satisfy both my curiosity and my laziness.

As your main datastore

Let’s start at the beginning. What do you want from a database?

Priority number one for many people is that the data store should get on and quietly do its job. Unless you’re particularly into databases, you probably want one you don’t have to think about. You want to write data and you want to retrieve data.

Postgres ticks pretty much every box when it comes to being boring in a good way:

Straightforward licensing: Postgres itself has a permissive open source licence and is developed by a global community rather than a single vendor. That reduces the risk of future bills and changes in product availability.
Standards compliant: Postgres meets 150 of 164 of the requirements of the SQL:2003 standard; no database management system meets all 164. That makes it easier to swap out, should you want to.
More than relational: support for object inheritance, plus key-value and JSON data types, mean Postgres can reduce your ops overhead in that you might not need to spin-up a NoSQL data store.

If you’re not worrying about your database, then you can focus on building something great.

Let’s say you’re growing a side-project into a business. You need to invest all those evenings and weekends in your product, not into worrying about the specifics of your data store. For Sugarmate, an app that helps people manage diabetes, Heroku Postgres has enabled its founder Josh to consider the data layer a solved problem, while he grows the app into a business with users worldwide.

When you need to scale

Postgres scales. Vertically and horizontally. From a few hundred records to a multi-terabyte monster. Just how you scale with Postgres depends on your data model and data access needs.

In many scenarios, increasing memory and CPU gives just the right capacity boost. What happens, though, when you need more? Before you reach for the nearest NoSQL enthusiast to tell you why their preferred solution is better than Postgres, consider that:

Postgres scales out: for read-heavy workloads, PostgreSQL’s built-in support for read-only replicas simplifies spreading the load across multiple Postgres nodes.
Partitioning isn’t a dirty word: in write-heavy situations, splitting your data set into distinct databases is a perfectly acceptable way of gaining capacity.
Offloading helps: using Redis alongside Postgres can lighten the load, whether through caching or by shifting predictable data requests completely to Redis.

Sure, you’ll need to do some prep work to scale Postgres but there are trade-offs with any kind of data scaling.

When you need copies of your data elsewhere

Your production database is sacrosanct. But you still need real data for development, testing, and analysis.

Postgres has two ways to help:

Forking: create a new independent database from a snapshot of another database.
Following: live stream changes from one database –– say, your production instance –– to another, such as that used by an analytics team.

Take the example of Bonobos. They’re a fashion retailer with several teams across the company, such as data science, accounts, and warehouse, who need access to production data. Rather than put the burden on their main database, the Bonobos engineering team created read-only follower instances that share the data safely across the company.

An added bonus of Postgres being open source is that there are no additional licensing costs for those secondary instances.

As a data warehouse

Data warehouses have traditionally been big, complex beasts.

But what do you really want from a data warehouse? A single place to store a lot of data and a flexible way to query it. Both of which are Postgres specialities.

Thanks to the follower database concept, streaming data from other Postgres instances into a Postgres-based data warehouse is simple. Rhino, a provider of an insurance backed alternative to property rental deposits, takes it a step further by streaming data from multiple non-Postgres sources into their Postgres data warehouse. Using homegrown tooling or ETL such as Mulesoft, Postgres can ingest data from pretty much any source and then allow you to query it with good old SQL.

For querying Salesforce data

Part of the joy of the Postgres ecosystem lies in the ways that companies have integrated with and extended the core database.

There are extensions that add geographic and time series support, those that make it easier to migrate from other data stores, and yet others that help with partitioning, ACLs, indexing, and more.

One integration, Heroku Connect, takes care of bidirectional sync between a Postgres instance and data held in Salesforce. That opens the possibility of writing apps that work with Salesforce data and also of writing tools that make it easier to query and analyse Salesforce data.

Similar to Rhino’s use of Postgres as a data warehouse, syncing Salesforce data into Postgres unlocks that data for deeper query and analysis. Need to dive deep into, say, the relative order value between two different regions? Postgres can help you to pull out subtleties from the data that wouldn’t be possible otherwise. In fact, Salesforce’s own go to market planning team uses Heroku Connect as a core part of their annual planning process.

Postgres isn’t jealous

Postgres is an excellent foundation precisely because it’s so good in a bunch of different situations.

It might be the only database you ever need. Or perhaps you’ll find some parts of your dataset are better off elsewhere. The social network Dubsmash switched recording likes of their user videos to a simple key-value store instead, while holding onto Postgres for everything else.

The past decade has seen massive upheaval in the world of databases. Through it all, though, Postgres has been there chugging away serving queries, taking in data, and adding new features to its roster. I’m curious to hear from other people in the comments below about how you’re using Postgres, especially if it’s in an unusual or unexpected way.

Anyway, here’s to Postgres, whether it’s self-hosted in a data center, running on a Raspberry Pi cluster, running in a managed PostgreSQL offering, or however you’re using it.

Matrix laptop photo by Markus Spiske
Sequoia tree photo by Nina Luong
Storage units photo by Joshua Coleman

Preparing your Postgres data for scale-out

Matthew Revell — Mon, 03 Feb 2020 09:16:10 +0000

App success is a double-edged sword. You’re getting the uptake you wanted but now you have new problems.

Those hacky bits of code you thought you could get away with suddenly become bottlenecks. But that’s nothing compared to what’s happening in your Postgres instance.

There comes a point, though, when you’ve optimized everything and yet you just can’t squeeze any more performance out of your database layer. That’s when it’s time to look at how to scale Postgres.

What type of scale?

There are two ways to scale a database:

Scale-up: you have one database instance but give it more memory, CPU, disk
Scale-out: you add more database instances.

Scaling up –– or vertical scaling –– is relatively easy. Add more CPU and, broadly speaking, Postgres can handle more concurrent connections. Add RAM and more queries will run in memory rather than paging out to disk. Add more disk and you have more capacity both for the data itself and indexes to speed-up querying.

That approach can only go so far. Not only are there limits to how large the database server can grow but you have an increasingly unwieldy single point of failure.

Scaling out –– otherwise known as horizontal scaling –– adds more database nodes that each take part of the workload. This takes more work than scaling up but means that you’re no longer piling more and more work onto a single database instance.

	Scale-up	Scale-out
Ease	Can be very easy, especially if using a hosted Postgres service	Takes up-front planning and ongoing maintenance
Effectiveness	Delivers a quick performance boost but with diminishing returns	Offers a long-term solution to scaling issues
Unique benefits	The entire database scales for both reads and writes	Parts, and maybe all, of your database can stay online even if an individual node goes offline
Gotchas	The database is a single point of failure	It’s easier to scale out for reads than writes The data model might need to change

Read-only or read-write

If you choose to scale-out, then there’s another decision you need to make straight away: are you scaling reads or both reads and writes?

A standard Postgres database can have just one primary node. That’s the node that accepts writes. However, it can have many secondary read-only nodes. If your app’s data pattern is read-heavy, that’s often an relatively easy model for scaling-out. Let’s say your app provides train times. Most database activity will be time table look-ups. In that case, secondary read-only nodes would be an ideal way to scale capacity.

If your app is write heavy, then a single writable node quickly becomes a bottleneck. In that case, you need to think about splitting the write-heavy parts of your database into separate partitions or “shards”.

Sharding data

The idea behind sharding data is pretty simple: you effectively split your database into multiple separate databases. And with multiple databases come multiple writable primary nodes.

Sharding isn’t unique to scale-out; it’s common to shard data even in scale-up scenarios, as splitting data can help reduce index growth and reduce write locking on a single instance. However, for most Postgres users, sharding is the only way to achieve a writeable scale-out database.

Either way, splitting your database into different partitions impacts how you think about your data. The good news is that there are common sharding patterns you can choose from. Which you select depends on the shape of your data and how you need to query it.

Four common sharding patterns

When you come to shard your data, you should start by looking at whether a commonly used sharding scheme suits your app. That way, you’ll find it easier to get help from peers and online.

The most common sharding schemes are:

Range
Vertical
Key-based
Directory-based.

Let’s look at each in turn.

Range-based partitioning

Let’s say you want to scale -out a database of customers. One simple way to shard the database would be by customer name. People whose name begins with A-F might be in partition number 1, G-L in partition 2, M-R in partition 3, and S-Z in the fourth partition.

This is easy to think about. You can take a look at your data and quickly come up with the ranges that make sense. However, it’s easy to get into difficulties. Naive ranges –– such as alphabetizing surnames –– can easily lead to one shard being much busier than others. For example, according to the US census, there were 2.37 million Smiths in the United States in the year 2000, 1.85 million Joneses, and 1.5 million people with the last name Williams. In our simple example above, that fourth partition would be home to two of the top three most popular names.

Looking more closely at the data we’re working with can help us to make nuanced decisions in setting up a range-based partitioning scheme. We might decide that S gets its own partition, for example, in order to handle all those Smiths.

However, the risk with range-based partitioning is that the scheme isn’t necessarily drawn from the shape of the data but instead from something that makes sense to us as humans.

Vertical partitioning

Take the example of an ecommerce site. There are natural distinctions between different types of data that only rarely need to come together. For example, product descriptions and warehouse employee data are distinct.

Vertical partitioning is where you take the natural divisions in your data as the lead for your sharding scheme. So, in your ecommerce example, we might have a shard for product information, one for shipping, another for competitor pricing, and so on.

As this scheme is based on the data itself, then sizing each shard is more obvious. Employee timesheets might need a far smaller node than, say, customer account data.

The main disadvantage, though, is that you will eventually get to a point where you’ve partitioned the data as much as makes sense. Let’s say your ecommerce site were particularly successful. You might have a shard dedicated only to customer records. How then should you further partition that data if you need to scale more?

Hash-based partitioning

The previous two schemes were based on things that make sense to humans. Hash based partitioning is effectively random. Rather than partitioning on some aspect of the data itself, you create a hash from the key of what you’re storing and that determines which partition to use.

In our ecommerce example, the key for a customer record might be their email address. Let’s say we have five servers. We perform a hash function on the email address that gives us the number 137692 and we find the partition where it should live by performing a modulo operation using the number of servers as the divisor. In this case, the modulo gives us 2, so that customer’s record should live in partition 2.

In effect, hash-based partitioning is random and so it’s less likely for hotspots to appear. Smiths, for example, would find themselves distributed across the cluster. Hash-based sharding can also be a solution to the problem of how to further scale a vertically sharded database.

The downside is that adding more servers to the cluster would require recalculating where each record should live, because it will change the divisor in the modulo operation.

Directory-based partitioning

This is the most complex scheme of the four as it requires an intermediary layer between your app code and the database. With directory-based sharding, that intermediary layer is a look-up service that implements another sharding scheme.

The advantage here is that you can change the underlying partitioning scheme without having to change your application code. It can also act as a bridge between a previous scheme and its replacement. In our hash-based example above, a look-up service could translate hashes created for a five node cluster into hashes that work in a newly enlarged ten server cluster.

One obvious disadvantage is that now you’re running not only a partitioned database cluster but also another service. That increases maintenance and adds a step on the path between your app and the data you need.

Sharding helps but it’s not all good

Sharding offers a way to scale-out a write-heavy Postgres database but there are disadvantages. Joins across shards become expensive and might not even be possible at all. You might even find you need to compromise one of the key advantages of a relational database and maintain multiple copies of the same data in different places.

There are other options. Redis can help offset the burden on your Postgres instance, for both reads and writes. And once you’ve taken that first step outside of Postgres, you might find that some aspects of your data are better suited to other tools, such as Elastic or Kafka.

Either way, scaling beyond a single Postgres instance takes preparation and a thorough understanding of how your app accesses your data.

Jukebox letters photo by Diomari Madula
Candy photo by vaun0815
Numbers photo by Nick Hillier
Drawers photo by Kolar.io