Forem: Evgenii Ponomarev

When to Use Nested Functions

Evgenii Ponomarev — Sun, 14 May 2023 08:54:02 +0000

Warning: the word function appears more often than you are probably used to. Sorry.

Alright, let's begin: nested functions are functions defined inside other functions (promised and delivered). You saw them in many forms, like this one:

Here, the function multiply is defined inside its parent, pow, although it does not have to be this way.

Or does it? If nothing else uses multiply, are there any reasons to move it out?

Tough choice

This topic regularly ignites lengthy bike-shedding conversations regardless of the programming language (stack overflow about python, reddit about swift), so I want to jump in and list some more pros and cons which you can apply for your needs.

Why you should not use nested function

Nested functions reduce readability

The bigger a function is, the harder it is to read it.

Look at this example:

By defining three additional functions inside showNews, you make the reader's job harder: they have to scan all of them before reaching the part with actual logic.

It might be much shorter:

The reader still can jump to any functions to see how they work, but now it's optional.

Nested functions influence future growth

One may argue that not all nested functions affect readability.

In the previous example, you had three big helpers — pretty bad. What if there was only one?

Does having loadNews inside the primary function cause any harm? After all, it is just several extra lines.

The danger of this approach shows itself when someone needs to add more functionality. You should not be surprised when your colleagues follow the same style and add other nested functions next to the existing one. After all, they just follow the rules you defined as an author.

All it takes to degrade a good codebase is one example of a poor pattern and people's intent to maintain the code style.

Check the article about "Reckless Reuse" for more examples of how code becomes unmaintainable without anyone noticing.

When nested functions are good

Now you may have an idea that all nested functions are evil and should be avoided by any means.

This is not true.

In some cases, they bring some benefits that can outweigh the disadvantages.

Sometimes you do not want others to use your functions

Yes, functions are meant to be reused. But you may not want it, e.g. if a function handles your particular case but is not suitable for everyone:

There are many cases in which isEqual returns an unexpected result:

But it may be the exact behaviour you need.

In this case, it will be safer to define this function inside saveNews so that nobody has to deal with its peculiarities.

Sometimes you need access to an outer scope

Many languages give nested functions access to the variables from the parent scope and even allow to modify them. You may need to explicitly declare such variables (Python) or capture them by reference (C++), but the benefits are the same:

In this case, the alternative would be passing all arguments to reportError every time we call it, using Function.prototype.bind or wrapping it into a factory. While I prefer different things depending on the situation, I see why somebody may use nested functions here.

Some implementation details are helpful

From the reader's point of view, there are few things worse than jumping between functions to understand how the whole thing works.

Try to understand this piece:

The functions onSuccess and onError do not only hide implementation details but also obscure their purpose. You know something happens, but you don't know what exactly.

In this case, it would be better to define onSuccess and onError within the primary function so that the reader knows what is happening without leaving it:

Two things to note here:

Implementation details are still hidden: you know what is happening but not how
Try to avoid situations like this by better naming and defining better APIs. The downsides of nested functions are still there.

So, what to do?

I usually try to avoid nested functions because they make their parents bigger and, as a result, harder to read. Even if a nested function is concise and the impact is negligible, it can potentially become a slippery slope and serve as a bad example for adding similar ones.

But, as you can see, there are some situations when going this path does worth the downsides.

I hope that now you can defend any side in this holy war and justify the style you prefer. Happy coding!

As always, you can subscribe to Resilient Systems and receive new articles by email if you haven't done it yet.

You also can find me on Twitter or somewhere else – I am always happy to chat :-)

The power of "I don't know"

Evgenii Ponomarev — Wed, 10 May 2023 08:41:31 +0000

Imagine you have a colleague named Boris. And Boris has a problem: he's been working on a new feature for five days, but now it's all falling apart. The application starts up, runs for a few minutes, and then shuts down with some weird error message and a huge stack trace.

Boris tries googling the error, but no luck. It looks like it's related to one of the microservices the application uses, and it's too specific for anyone outside the company to know. So he copies the error and posts it in a team chat, asking if anyone knows what it means.

You see the question, but you have no idea.

What would you do?

While you're thinking, let me tell you what most people would do: nothing. And that's totally reasonable: they saw the question, they don't know the answer, and they don't want to spam the chat, so they just carry on with what they were doing before. One person does it, then another, and then a few more. Soon, everyone has seen the question and hasn't said anything.

But Boris is completely unaware of this. Maybe people are just busy and he needs to wait a little longer?

So he waits, until someone posts something else in the chat - another unrelated question – and gets a few responses. Now Boris has to wonder if people would see his question when they open the chat.

The best thing to do in this situation would be to say "I don't know" or react with a "🤷‍♀️" to keep the chat from getting too noisy.

This lets Boris (and others) know that they don't need to wait for an answer from you, and it can help bring more attention to the problem, potentially leading to a quicker resolution. Or at least, Boris won't have to wait unnecessarily.

Even though this isn't the biggest problem you may have (it’s not like Boris is going to wait for an answer for eight months while the company goes bankrupt because he can't complete the feature), being blocked is annoying. "I don't know" may not solve the problem, but it's a step in the right direction.

The more influential you are, the more valuable your "I don't know" is.

Boris might think "Even the tech lead who wrote a half of all code doesn't know this error," and feel a little better about being stuck.

A recently hired junior might think "Wow, it's okay to admit you don't have an answer," with the sense of uplift.

And someone who was hesitant to speak up without a definite answer, might suggest a couple of things to try since nobody else has any ideas.

Keeping meetings on track

Evgenii Ponomarev — Wed, 03 Aug 2022 21:39:38 +0000

You organise a meeting to agree on use-cases for a new feature. It is going well, and then a colleague asks which database you plan to use.

You’re preparing a response (because you haven’t decided yet), but someone suggests MongoDB.

“But why not PostgreSQL?” another person counters. “We have a similar service, and it only needs one instance of PostgreSQL – I wonder why the same configuration wouldn’t work for you”.

It surprises a few people. ”One instance?! What is your load?”

“It’s not high at the moment. We may need to consider adding more capacity as we grow, but I think three or four instances will be enough”.

“Have you ever questioned why we have so many databases in the same company?”

And now you understand that the meeting went off-track.

How do you prevent it?

Should you ignore the question about databases? Or stop on the MongoDB suggestion? Or cut off the PostgreSQL-guy? And if yes, why him?

And what if their points are not as irrelevant as you think?

Here is a list of tips to help you keep meetings on track.

At the beginning of the meeting...

... explicitly state the goals.

Sometimes, people are unaware of the goals – an additional reminder will not harm. Plus, it will help you return the meeting on track later; you’ll need to remind everyone what you’re trying to achieve.

... make people aware of time constraints.

We have 60 minutes, and I need about 40 for the presentation – the rest is for the discussion and Q&A – something like that is enough to make everyone conscious of time.

... share what exactly you plan to cover.

When people know the order of topics, they won’t interrupt with questions you plan to handle later.

During the meeting...

... watch out for side conversations.

Irrelevant questions are usually not a problem because you choose how much time to spend answering them. Irrelevant comments are fine either – you can acknowledge it and follow up later.

Most of the time waste is caused by side conversations – when several people are engaged in a dialogue about something that does not contribute to the goal. But yes, they usually start from irrelevant questions or irrelevant comments.

... timebox side conversations.

It is hard to assess whether a conversation goes off-track or someone just needs more time to make a point. Decide how much time you should wait before stopping it.

... treat irrelevant conversations equally no matter how they start.

It is common to say something like “it is probably irrelevant but …” before making a point that is indeed irrelevant. If you have limited time and something does not contribute to the discussion – stop it.

After the meeting...

... follow up on what was left undiscussed.

Irrelevant ≠ not valuable. You may need to stop some conversations during the meeting, but people still have ideas worth discussing. Reach out to the people whose comments or questions didn’t get proper attention for a follow-up conversation.

As always, you can subscribe to Resilient Systems and receive new articles by email if you haven't done it yet.

You also can find me on Twitter or somewhere else – I am always happy to chat :-)

What to do if you're a bottleneck

Evgenii Ponomarev — Fri, 19 Nov 2021 11:12:58 +0000

Jackson is a go-to person when it comes to MongoDB.

You have a problem — you go to Jackson. He’s an expert. Of course, you can go to anyone else – it’s a free world. You can go to Molly if you want. But you know what Molly would recommend, don’t you? She’ll tell you to ask Jackson, that overworked guy with a giant todo list.

Let’s talk about what to do if you ended up being a Jackson and need a way out.

What kind of a bottleneck are you?

There are two different kinds of overworked Jacksons: knowledge bottlenecks and expertise bottlenecks.

Knowledge bottlenecks are the ones who answer questions:

Who owns Payment API?
Which database has client emails?
When is our next release?

If you're one of those, write documentation. Sorry for such trivial advice, but boring problems need boring solutions.

It's worth writing docs even if people don't read them. Sounds stupid, but it will serve as your second brain from where you can quickly copy answers.

Watch out for an urge to share your screen and demonstrate something. It's always better to record a video (or a gif) and share it instead – this way, you will be able to reuse it later.

That was all about knowledge bottlenecks.

Expertise bottlenecks are trickier. They don't just answer questions — they solve problems:

This program crashes for Netherlands users.
My DB query is slow.
Our photo gallery page leaks memory.

This situation is much more complicated because you need more experts to share the load. But unfortunately, there are no books or tutorials you could give to somebody and turn them into an expert – they need to build the expertise themselves. And it takes time.

But how to accelerate the process?

Limited opportunities

If someone wants to learn how to play the guitar, they need to practice playing the guitar. Likewise, if one wants to learn how to fix memory leaks, they need to fix memory leaks.

The tricky part is there are much more opportunities to play the guitar than memory leaks to fix. They don't happen every day.

Zoom out now and look where you ended up: people call you whenever a challenging problem arises because nobody else has a similar experience. And nobody else has a similar experience because you solve all the challenging problems. A chicken and egg situation.

Growing new experts

The naive advice would be to stop doing what you're doing so that others can practice and build necessary expertise in time. But, unfortunately, it's overly utopian – a few companies have the luxury of accumulating unsolved problems while their best specialist sits idle and waits for new experts to emerge.

But I'm not saying it's impossible – there is a way to grow new experts without slowing down the process. Here it is:

Pick one successor to mentor. Learning opportunities are scarce resource, so resist the temptation to pick more than one.
Redirect all requests to this person. Need help? Ping Nelson, he knows this stuff.
Be available for this person. Answer all their questions as soon as possible, pair program or pair debug when necessary.

It works because:

The company will not slow down in critical situations – if the time is tight, the mentoree can ask you for help.
You can tune your involvement depending on the circumstances to manage risks and guide the learning.
This person starts earning credibility from day one, solving real problems with your invisible help.

In the beginning, be prepared to play broken telephone answering proxied questions:

It may look like a waste of time, but it's far from it. Yes, the future expert acts as a proxy, but they learn about the domain. Even if it's a proxy, it's a caching proxy:

As their experience grows, they will start asking more complex questions:

This is the point where you can return to your expert's duties, which you now can split with another person:

This is a cross-post from Resilient Systems – a newsletter about strategic software engineering.

The Story About a Smart Boss

Evgenii Ponomarev — Thu, 12 Aug 2021 19:47:33 +0000

Molly had a very smart boss.

Once, they had a system outage in the middle of the night, which nobody could solve until they called Molly. She logged in, scanned through all the changes from the day before, read the logs, did her own tests, and found the culprit – some expired certificates on one of their production clusters.

When the boss saw it the following day, he gathered everyone in one place and praised her in front of the others.

"Thanks to Molly, the outage persisted for only 45 minutes. She was quick, accurate, and she did the exact things I would do in this situation.

I had numerous incidents like this in my life, you know. And now it almost feels easy — I see the symptoms, and the root cause becomes apparent. But even with all this experience, I'm not sure I could have done it faster than Molly."

And everyone clapped him because he was very smart and humble, and they knew he would have found these expired certificates much quicker.

Another time, the whole team were brainstorming how to reduce the latency of their API.

Molly knew one or two things about it and proposed to cache frequently used data in Redis.

"I think it can be a huge success," the boss commented. "It's great that you mentioned that because I've been analyzing Redis myself and was just about to create the task to begin using it. Do you guys all agree about giving it a try?"

And everyone nodded because he was very smart and always had excellent and well-thought-out ideas.

"Actually," the boss pointed to Molly, "considering you also proposed Redis, why wouldn't you create tasks yourself?"

And everyone found it to be a good idea because wise leaders delegate work to their people.

Later that day, Molly called her old friend. He was a manager in another company, and she knew they were hiring.

"Is it what, my birthday?" her friend laughed. "But how is your team gonna take it?"

She paused for a second. "I'll miss them... but they don't need me much anyway. I don't think anything will change".

Molly was right – nothing has changed with her departure.

She had a chance to see this for herself a year later when she met her former boss.

"We just talked about you yesterday! There was a huge outage because of some expired certificates — put us down for 12 hours! I remember you were good at fixing those..."

Molly was startled but chose to switch to something more positive. "Those certificates are always tricky, don't even remind me… So how's the cool stuff, huh? API?"

"A bit slow as always, you know. But we plan to fix it soon — I think if we can implement some sort of a cache, or perhaps find an open-source thing."

"Like Redis," Molly suggested.

"Exactly! You know, I've been looking at it myself recently — my favourite option so far!"

As always, you can subscribe to Resilient Systems and receive new articles by email if you haven't done it yet.

You also can find me on Twitter or somewhere else – I am always happy to chat :-)

Challenges with driving technical initiatives

Evgenii Ponomarev — Mon, 05 Apr 2021 17:32:19 +0000

You work on a Big Improvement: rewrite backend on Go, move services to AWS, change a task tracker — something that affects multiple people's day-to-day work.

And then the initiative faces a pushback.

How much it hurts depends on when you encounter it: rejected proposals feel more bearable than having to throw away the nearly finished work you've been doing for months.

How to prevent it?

I have repeatedly found myself on both sides of the fence — as a proponent of innovations and as a member of the sceptical camp. Now, I am pretty sure that Big Improvement's future entirely depends on its author: how they prioritise different kinds of work, address concerns and adapt to feedback.

Let's figure out how to deal with yours

For the purpose of this article, I assume that:

1) you did your homework and know that your company will benefit from this initiative

2) you are open to feedback and ready to adjust if needed

Why technical initiatives fail

Ask yourself: why would anybody reject a good initiative? Assuming nobody is malicious, the answer is obvious – they do not think it is good. But why?

There are many possible reasons:

Potential reasons for a pushback

Some of them are more frequent than others, and I want to highlight these three as the most common:

People do not believe you can reach promised goals
People tried what you built and did not like it
People disagree that the problem is worth solving

Let's see how you can prepare to face each of them.

Reason 1: People do not believe you can reach promised goals

Imagine a medieval kingdom that has been terrorised by a gigantic dragon for centuries. The dragon mercilessly burns the fields with the best crops, devours cattle, and kidnaps the most beautiful (by dragon standards) princesses.

You, a foreign knight, come in armour and on a white horse and ask for thirty armed men and two bags of gold to finish off the dragon forever – something that many have tried to do, but no one has yet succeeded.

What do you expect to hear back?

They look sceptical

How it may sound to you

Why do you think it would be more performant and easier to maintain? Are you sure? I'm not sure...

– Carina

What the root cause is

The longer a problem exists, the harder it is to believe that somebody can solve it.

Especially if there were many unsuccessful attempts in the past.

Psychologists call this effect "Learned helplessness" and suggest that it can spread like a virus, infecting individuals, teams and organisations. Collectives that catch it become inert and pessimistic — it requires enormous effort to overcome the dominant belief that nothing can be done.

How to deal with it

Scepticism is contagious. Therefore, it is easier to prevent it than change many people's mind. But you cannot do it with bare arguments.

Never ending arguments

Instead, you can benefit from applying some principles of the hypothetico-deductive method:

First of all, you agree on a definition of the problem with those who already tried to tackle it
Then, you propose your idea – something that should solve the problem
Together, you deduce the list of consequences: what would the solved problem look like. The list may be pretty long, so it is worth focusing on the riskiest cases: if the hypothesis is incorrect, you want to know it as soon as possible
And finally, you agree on the list of experiments that may help you to disprove the consequences and fail fast

The shorter version of the method

If it looks too abstract, let me bring a more concrete example:

How to structure a refactoring

In the example above, we propose using Rust to solve existing performance problems.

Instead of reasoning whether it would work or not, we identify parts of the system with higher chances of performance problems, agree on how failure would look like and design the experiments that may reveal the shortcomings.

Even though it may sound straightforward, it is common to see two other approaches which do not end up well:

a proposal faces reasonable scepticism and does not get through not being backed by solid evidence
engineers either do not identify or do not prioritise risky cases and then have to throw away a large amount of work

But anyway, it was a pretty easy one – time for the more interesting case!

Reason 2: People tried what you built and did not like it

This happens a little bit later, when you think you built something good enough for others to use.

You prepare a polished demo that fulfils all expectations and makes everyone's hands hurt because of incessant applause:

A well-rehearsed first demo

But when others start using it... well, this happens:

This guy does it all wrong!

How it may sound to you

I like the old way better.

Yes, I needed to change the code in five places.

Yes, the code was hard to test.

Yes, it was less performant.

But at least I could get it working because I knew what to do.

Now, I have no idea what's going on, and it all keeps breaking!

Can you de-engineer it back?

— Marcus, the early adopter

What the root cause is

There are several possible causes, and in most cases, they occur together:

Not enough documentation Writing docs is never the top priority during a turbulent time of building something new
Unexpected bugs After the flawless announcement, people may develop high expectations and forget that this is only the first version
Unfinished implementation It may be frustrating for early adopters to start working with the new thing and then stumble upon a blocker
Rosy retrospection In psychology, rosy retrospection is the tendency to recall the past more positively than the present (even when the facts suggest the opposite)

How to deal with it

Ensure that people have the best possible experience working with a new system from day one.

It would be ideal if you provide:

Documentation with exhaustive examples
A list of known bugs and the current state of the roadmap to manage expectations before people start using it
Support during development so that you can answer questions as they arise and unblock people

If it is impossible to do all of this because you don't have many resources, I would recommend keeping other people away from this version until it meets all criteria. The quality of the first impression means much more than time.

You may have valid reasons not to wait, e.g. when other engineers want to start building something else on top of your system while it is in development. All situations are different, but my suggestion for this case would be to pair with them until they have necessary context.

Reason 3: People disagree that the problem is worth solving

Imagine yourself in a group of bright scientists trying to invent powerful yet affordable quantum computers.

At some point, your experiments pay off, and you find the way to manufacture inexpensive quantum chips –– with them, even smartphones would vastly outperform existing supercomputers from the IBM labs.

However, there is one minor drawback. Its architecture is absolutely incompatible with what we have nowadays, making most of our computer science knowledge redundant. Needless to say, some developers may find having to learn all from the beginning a little unfortunate.

To say the least.

How it may sound to you

What do you mean by 'automated releases'? All you need to do is to log in to the production machine, download new code and run three commands – I can tell you which ones.

You call it a problem just because you were not here 12 years ago!

– Dr Nicolas Peter, PhD

What the root cause is

To discuss the root cause, we need to consider two separate cohorts:

Those who are used to the current way of doing things
Those who somehow benefit from it

The first group's reasoning is pretty obvious (just make the transition smooth for them); therefore, I will focus on the second one.

Nobody introduces inefficient systems on purpose. At least, these situations are pretty rare.

Quite the opposite – people usually have good intentions:

They see a big problem (same as you!)
They implement a solution (similar to your Big Improvement!)
They become experts in it.

In time, the solution loses its appeal: it does not satisfy new requirements, there are open source analogues which do the same, and so on. It is time to replace it with something better, but it will imply throwing away all hard-earned knowledge, taking off an expert hat and having to learn something new.

Besides, the experts know the current version like the back of their hands and see all arguments as exaggerated.

How to deal with it

The traditional way of constructing an argument almost guarantees resistance. This is what I call a traditional way:

You state the problem
You demonstrate evidence of why this problem is vast and needs solving
You describe how bright the future would be if you solved this problem
You explain how you are going to do it

For all its factuality, this approach is devoid of any compassion: you go to the people who built the thing, to the people who use the thing, and to the people who have become experts in the thing and tell them not only that the thing is terrible, but that it has become a huge problem and needs to be replaced immediately.

The alternative approach acknowledges the history of an existing system and focuses on what changed since it was initially created. This can help build a foundation for productive collaboration with the current group of experts and get them on board.

This list of questions can help have a constructive conversation with the experts:

What were the initial goals of the system?
Does the system still satisfy all its initial requirements?
Which parts of the system are you (the expert) not happy with?
Which parts of the system have degraded in time due to technical debt and negligence?
Have we received new requirements that the system does not meet?
How can we make it right together?

This approach allows you to join forces with knowledgeable people to find out much more about current and potential use cases, identify hidden problems, and not repeat old mistakes. Experts, in turn, can retain and update their experience after replacing the existing system. Win-win.

One final tip

Rejection can be painful. Fear of rejection is real.

You may want to avoid this by not letting anyone into your plans until everything is perfect.

Do not fall into this trap because building what nobody needs is much worse than facing a pushback.

Instead, be wise in driving your initiative because its future depends on you: how you identify stakeholders, prioritise different kinds of work and adapt to feedback.

Liked this article?

Subscribe for more and receive new ones by email.

You can also find me on Twitter – I am always happy to chat :-)

Four streams of defects for prioritising bugfixes

Evgenii Ponomarev — Tue, 08 Dec 2020 23:13:15 +0000

This is the adapted transcript of the talk I gave to my cat because all conferences had been cancelled, and my internet was too slow for online events.

How urgent is it to fix bugs?

The question may sound ridiculous, but the answer isn't straightforward. Well, it depends on a bug, isn't it?

When it's not released yet, prioritisation is simple: we have to fix it before the release. Of course, there may be some exceptions, but it sounds like a reasonable rule for most of the situations.

What's about bugs which are already in production? What does worth fixing?

It's all about impact

Do you need to fix a leaking tap on the kitchen?

Not everybody is annoyed by leaking taps [source]

Usually, it depends on how lazy you are, but I'll make the decision a little bit easier by giving you more data:

Your partner likes to study there, and the sound of dripping water makes it hard to concentrate
The water bill was $40 higher this month
Last time you left some mugs in the sink, and they blocked the drain — you were just lucky to notice it and prevent the flood

I think this information will help you make the right decision without hesitation.

Same with bugs.

As soon as you know the impact, prioritisation becomes much easier.

Four streams of defects

To prioritise effectively, you need to answer many questions:

How many clients are affected?
Which piece of functionality does work incorrectly?
Are there other bugs we need to prioritise?
When did it start happening?
and so on

The way organisations deal with those questions depends on how they detect bugs. There are four big channels you can use to discover defects in your systems:

Customer feedback
Error reports
Exploratory testing
Stats anomalies

Most of the time, companies rely on only one or a couple of channels and neglect the rest – I'll show the consequences of this approach later. For now, let's talk about each of the four streams of defects and how we can make the most of it.

Customer feedback

That's a simple and the most ubiquitous way of finding bugs – to delegate it to your clients. They don't like something, and they complain – you just have to listen to it and schedule the work.

Some unhappy people [source]

Unfortunately, a lot of companies stop there and use only this channel.

There is what it may lead to :

Not all features are covered. Your clients aren't going to complain if you stop showing ads by mistake. They won't notice if you break behavioural tracking either, which is the same for all non-customer-facing features of your product.
Number of complains as a prioritisation factor. Some clients naturally report more issues than others, but does it make their reports more important? What about the people who gave up and stopped using your product?
Technical support becomes a bottleneck. Triaging incoming requests is tedious work that requires time and coordination. If all you have is plain text messages from users – it's going to be hard. Just be prepared.
Hard to reason when it started happening. The fact that the bug is reported today doesn't mean it's a recent one – it may have been there for years until it bothered this particular user.

How to use this channel effectively:

Make sure clients can always send their feedback. Offer multiple ways to reach out to you, especially, if you use third-party widgets for collecting feedback (such as Userback). Advertise your twitter account, email-support, anything that can be a backup communication channel.
Allow users to send you attachments. Don't expect clients to send you proper bug reports using internal jargon. You may know that "☑️" is called a checkbox, but for them, it's going to be that thing or that thing that doesn't work. Allow users to send you screenshots, gifs and videos in any format – it'll save lots of time for both sides.
Collect as much metadata as possible. Operating system version, browser type, screen resolution, locale, local time – everything that can help you reproduce the problem and is allowed by your privacy policy. This becomes even more important when clients report accessibility issues – they may have a non-trivial setup, but that doesn't make these errors any less important.

Error reports

Time for the next channel – error reports. You detect an exception in your code and report it to Sentry or something similar.

780 users are affected [source]

It compliments client feedback by addressing its shortcomings:

You have data for all features regardless of how important they are for users
You can see the overall impact based on the number of exceptions, not the number of complaints
These questions can be combined together to simplify technical support
Some of the errors would correlate with releases and could be mitigated by rolling them back

As it may look like a perfect solution, it still has some downsides :

Not every bug causes an exception. Misaligned elements on your webpage may block important pieces of your workflow, but probably won't trigger an exception
Number of exceptions as a prioritisation factor. Errors aren't equal, therefore prioritising bugs by a number of exceptions has the same issues as relying on the number of complaints.
Hard to reproduce. Having an exact stack trace doesn't mean that it's going to be easy to fix, especially when an exception isn't originated in your business logic. Mysterious cannot read property 'data' of undefined can appear out of nowhere and frighten all the stakeholders by how many of them you receive

How to use this channel effectively:

Make sure you can easily locate all the recent changes. It's important to react quickly when you start receiving a new type of reports. Usually, it means that you've recently released a bug — but when and where? The more moving parts you have, the more important it is to journal all the changes and be able to correlate them with first occurrences of a new type of exception
Setup SLAs for dealing with reports. Data is useless if nobody looks at it proactively. Setup the procedures for categorising new errors so you won't miss anything important
Invest in traceability. Where this incorrect request came from? How come the system reached this state? The more subsystems are involved in use cases, the harder it may be to answer those questions if your code isn't instrumented properly.

Exploratory testing

Does your company have people who continuously work to break your systems?

These folks are testing something, not sure what exactly [source]

I won't be surprised if it doesn't. Unfortunately, many businesses cut costs on manual testing as they see it outmoded and redundant. Such a view is gaining popularity because manual testing is often perceived as repetitive execution of pre-defined scenarios. This type of work indeed has to be automated.

Exploratory testing, in contrary, is a process of breaking the system without predefined scenarios to find situations that developers didn't think about. When organised properly, it can cover the blind spots of Customer feedback and Error reports:

Testers can find bugs in all types of features, even if they don't generate exceptions. If the banners on your website aren't visible because of wrong CSS-styles, testers will catch it
Testers can find reproduction steps for mysterious exceptions. If you vaguely know where the exception comes from, your QA team will stress-test that piece of the system in order to cause the same crash and find a way to do it reliably
Testers can test new features before you enable real clients. The biggest downside of other streams of defect is that they only catch bugs which are already impacting clients. Regardless of how good your auto-tests are, there are still use-cases the engineers haven't thought of. QA-engineers with the right tooling and good experience can make new releases smoother

How to use this channel effectively:

Invest in tools for testing. Do your QA engineers have different environments for testing or they use the same setup for everything? Try to provide them with all the right tools: virtual machines, mobile devices, accessibility software
Automate as much as possible. Obviously, it's a huge waste if smart testers have to manually do something that can be done by machine. If they need to perform the same actions every test cycle — automate it
Make sure they know how your system should behave. In other words, document your features. Many teams rely on common sense and expect that testers will a bug when they see it. Unfortunately, it's not always the case: if a tester asks colleagues if something is a bug — it's a sign of unclear behaviour that should be documented.
Involve testers in prioritisation. QA engineers should not only find a bug and create a ticket — they should have all the information to assess the damage and suggest a priority. Make sure they can detect if something is a new regression, know the relative impact of different features and have all the necessary information about past and planned releases.

Stats anomalies

Do you collect statistics? Page views, registrations, orders, clicks on advertisement? If yes, this channel can be the most important one for your business because it shows how bugs affect your key metrics.

Something doesn't feel right looking at this stats... [source]

You released a new version and number of registrations suddenly fell by 15% — having this data makes prioritisation much more straightforward.

Although, it may sound like a panacea, using only this channel can be not enough because:

Anomalies don't necessarily correlate with your releases. There are many factor which can affect your system, and you may need to dig deeper to understand how to fix it.
Stats may be impacted by various factors, not only bugs. Anomalies can be caused by genuine reasons: weather, public holidays, major events — not all of them are bad, even if your metrics go down.
Changes aren't always rapid. It's simple when you update business logic that lives on the server-side, preferably on one instance. You introduce a bug and the metrics immediately go down — easy to catch. When it comes to front-end, mobile applications or other systems with the longer rollout, the change isn't that sharp.

How to use this channel effectively:

Make your dashboards as close to real-time as possible. Data is much less useful when it comes with a big lag, especially if you have to wait for a Hadoop job to generate you a daily report (yes, that's a very frequent situation!)
Collect data about user environment. Most of the major systems for collecting metrics allow sending metadata along with stats. It can give your QA engineers enough clues to reproduce the issue or can help you to figure out a genuine reason for the anomaly.
Setup alerts. Use all capabilities of your system — many of them allow you to setup thresholds where something is considered an anomaly, so you don't need to monitor graphs yourself.

What is your score?

How many of those streams do you currently support? Do you rely on one of them or have multiple?

All four streams can give you a very good understanding of what's going on:

Customer feedback — what happens?
Error reports — how many users are affected? when did it start?
Exploratory testing — how to reproduce?
Stats anomalies — how does it impact business?

Then, prioritisation becomes much easier, as for the leaking tap from our previous example:

Situation 1:

All new clients are affected since we've released "XYZ" — nobody can register on our website.

It decreased the number of orders by 20%.

Solution: revert "XYZ"

Situation 2:

We had two complaints from the customers, but only people with IE11 are affected.

It started 3 weeks ago and haven't visibly impacted any of the key metrics yet.

Solution: won't fix, show a notification that we don't support IE 11.

As always, if you have any questions, feedback, or just want to talk about something, feel free to ping me on [Twitter], and subscribe to "Resilient Systems" if you haven't already.

Not as Productive as Others?

Evgenii Ponomarev — Tue, 24 Nov 2020 20:30:00 +0000

This is another article about pacers – people whom we use as an example to adjust our behaviour, mostly unconsciously.

Surrounded by faster pacers

In this part, we will talk about problems that may occur when you feel that a lot of people in your team are much more productive than you.

In many cases, it doesn't lead to any problems – people happily work together and achieve great results. But sometimes their mutual influence can be destructive, and people would feel demotivated and discouraged.

Several symptoms are indicating that you're affected by faster pacers:

You feel guilty seeing that colleagues have achieved more than you
You are afraid of taking challenging tasks which can make you stuck for a while
You feel like you have to work longer hours to keep up with others

Daily standups or any other form of sync with colleagues is the perfect time to diagnose this problem:

Are you nervous about sharing your progress?
Do those meetings make you feel like you have not done enough?
Do you find it challenging to describe your results and plans?

If any of these describes you, a closer look is needed.

Five actions to solve the problem

The most popular advice I heard in this situation is to take it easy. It is normal to have somebody more productive than you, left alone the fact that nobody can be a top performer in all situations.

Even though I fully understand the reasoning behind this suggestion, I do not think it is helpful. If somebody is nervous about attending standups, you cannot fix it by suggesting not to worry.

Instead, you can face this problem, find out the reasons and prepare an improvement plan, even if the problem exists only in your imagination. Then, it's going to be up to you whether to follow this plan or not, but at least you will take matters into your own hands – that alone can be sufficient to address most of the symptoms.

Let's talk about the five things which you can do to get out of this unpleasant situation.

Action one: demystify top performers

no two writers are the same, like snowflakes and fingerprints. No one will ever write in just the way that you do, or in just the way that anyone else does. Because of this fact, there is no real competition between writers. <...> Writing is a matter strictly of developing oneself. You compete only with yourself. You develop yourself by writing.

– John McPhee

In time, everyone develops their areas of expertise. No matter how broad or narrow they are, one is the same — people are much more productive when their job overlaps with those areas.

Before bringing up your own expertise, I would recommend you you to think about your more productive colleagues. Every time they do something great, ask yourself what helped them to achieve those results.

Something that could sound self-deprecating at first:

Mark finished this giant feature for a week. That one would take more than a month for me!

Can be rephrased and cleansed of magic:

Mark finished this giant feature in a week because he's been building similar ones for the past three years.

He did more than ten last year – now he's extremely good at it.

Now you can see that Mark's fast pace didn't appear overnight – it required years of deliberate practice. Moreover, you know that doing more things in this area can help you to close this gap.

That is not the only way of reframing achievements. It could be something like this:

Mark finished it in a week because he wrote the whole system since the beginning and he knows every line of code by heart.

or even this:

... because he did not have the internet at home and spent all his free time working in the office.

No matter the situation, your goal should be to stop seeing people as just productive and start noticed the reasons behind their results. Most of the time, those reasons are ordinary and achievable by anybody.

Action two: find your comfort zone

Simply put, there are three types of activities:

Something you enjoy the most
Something you are good at
Most important things for the company at the moment

Three types of activities at work

The intersection of all three circles is your niche in the team, but finding and expanding your niche deserves a separate article. For this topic, I will focus only on the comfort zone (highlighted in green).

Everybody has ups and downs, and you will have many periods of not being at your best.

One of the smartest things you can do is to prepare something that can give you a little boost when needed — a type of work you like to be doing or an area where you can be very productive.

If you do not have a comfort zone right now — build one:

Familiarize yourself with plans of your team.
Pick an area which will require work in the future.
Start building expertise there to capitalize on it when the time comes.

Action three: define expectations and track achievements

The easiest way to not meet expectations is is to have no expectations at all. No matter what you achieve, you can always find a room for improvement.

The simplest way of coping with that is to formulate your goals before you start working towards them. I find daily plans most precise and helpful for this purpose; here is one of my recent ones:

The expectations were clear and realistic, but I ended up not finishing half of what I planned to do.

Was it a problem? No, because I knew that I had done more important things instead, and there was absolutely no reason to feel sad about my initial plan.

Action four: keep expectations reasonable

Sometimes results are noticeably small in comparison to goals:

planned: 7; finished: 1

While this is a call to think about what can be improved, I would recommend checking your expectations for feasibility first.

There is a simple exercise to figure out if you are unrealistic in your estimations:

Take some tasks your team plans to work on and imagine how much time each of them would take for you. Can you complete a particular task one day? In a week? Or maybe you can do five of those in an hour?
Write it somewhere.
When other people complete those tasks, check how their results match your estimations. Were they working within their niches or tried something new? How did it impact them?

This exercise is also helpful when your plans are unambitious and require correction:

100% done!

Action five: analyse and improve

Detailed plans can be beneficial even when everything looks good:

Perfect plan and execution

Even though the result matched your expectation, a picky perfectionist can find some food for reflection:

Digging deeper

I rarely use this method – maybe once every two or three months, but it always helps to take control and find something I can change.

Prove it works

At the beginning of this article, I mentioned that the best way of noticing this problem is to pay attention to your behaviour when you need to share your progress and describe the plans.

Feeling that you have not done enough can contribute to the demotivation, which will prevent you from working at your best, which will cause dissatisfaction of not achieving enough, which will incur even more demotivation, which will further decrease your productivity, which will make it even harder to get out of this loop.

The good news is that the actions above can help you to

Understand if your expectations are realistic and how to adjust them if they are not
Find the quick way to get back on track after a period of dissatisfaction
Figure out what is the exact reason for dissatisfaction and how to improve it

If anything, it doesn't leave too many reasons to worry.

As always, you can subscribe to Resilient Systems and receive new articles by email if you haven't done it yet :-)

Who Influences Your Working Pace?

Evgenii Ponomarev — Tue, 15 Sep 2020 19:22:27 +0000

Who has the most significant impact on your working pace? Does it help you to grow or slows you down?

In this article, I'll list several types of people we may unknowingly use as pacers to adjust own behaviour. If you know them, you can control this effect and even change how you affect others.

This is the second article about pacers – people, who can impact the performance of teams and companies by just speeding up or slowing down.

The previous one is here.

Three big categories.

Pacers in your team.

It's reasonable to assume that your teammates have many opportunities to impact your pace, but the level of this impact varies from person to person.

Most likely, colleagues with whom you interact more frequently will have more influence on your attitude and productivity than others.

If it doesn't surprise you that students living together often achieve similar results, it should also sound normal when something like this happens in the office. People, who have adjacent desks, frequently synchronise schedules, including lunches, coffee-breaks and how much of productive time they have.

People with similar backgrounds is a similar story. Graduated from one university? Hired in the same year? Same age as you? The more traits people have in common, the more likely they would influence each other.

If you have any grading system at work, colleagues with adjacent ranks can impact you a lot too.

What are people in the next grade suppose to do? May you be up for a promotion? Do you bring more value than people on a preceding level?

The last two types of teammates who can influence your pace are the most and least productive people. One shows the team what is possible, while another — what is acceptable.

There is one caveat, though — since it's difficult to measure actual productivity, you will have to deal with a perceived one. Make sure you see the entire iceberg, not the tip of it, before your perception impacts the pace.

The last type definitely needs more attention, and I'll come back to it in one of the next articles. For now, let's move on to the next category.

2. Pacers outside of your team.

Not anybody can find a pacer within the team — what if you're already a most prolific worker and now you need somebody to push you harder? Luckily, the world doesn’t end outside your team, so there are several options.

Company-wide stars is the option available for middle-size and large companies.

Joined Google because one famous developer works there? Now you can have a chance to have more insights on how they work and pick their pace if you like to.

Family and friends is probably the most obvious example of pacers you can have outside the team.

Your colleagues work extra time, but you leave at 6 because this is how the pace of your partner looks like? A hard-working and extremely productive best-friend can be better than all motivational programs in revealing what you re capable of.

3. Imaginary pacers.

Who said pacers have to be real? Some of them may exist only in your head, but it doesn't always mean that you're getting crazy.

Memories of yourself from the past can be a genuine imaginary pacer.

What did you achieve last year? Are you improving or falling behind?

Although it may look like a good idea, it can become dangerous when those memories fade. We don't have complete control over what we remember, so our perception of how much we did and how hard it was can be very different from reality.

Now I want to return for a second to the most and least productive people because they're able to form long-lived imaginary pacers for the whole team.

Have you met someone who was considered extremely productive because they were like that in the past? Even if they no longer produce the same results, others may still try to keep up with the pace that the original pacer could not maintain. The same can go with low performers, even after they bounced back.

The last type of pacers I want to mention here is famous people — actors, athletes, entrepreneurs, writers and even prominent open-source contributors. Not all of them show the shortcomings and failures of the public, which makes it easy to build overly idealised and hard to follow role models.

Putting all together.

This is the whole list of most common types of pacers which you might pick for yourself:

In your team
- Close colleagues
- People with similar backgrounds
- Adjacent ranks
- The most and least productive
Outside your team
- Company-wide stars
- Family and friends
Imaginary pacers
- Memories of yourself
- Memories high and low performers
- Famous people  ## What is next?

Try to identify your pacers and think about the implications:

Who has the biggest impact on your pace at work?
Is this impact positive or negative for you?
Is it possible for you to keep up with this pace in the long run?

Give it a little time to check how your behaviour may affect others:

Who could pick you as their pacer based on those categories?
If all your colleagues copied your working pace, how would it affect the team?

The next article will be about demotivation and how to stay sane when your pacers are much more productive than you. Subscribe to "Resilient Systems" to get it as soon as it's ready.

As always, if you like the topic and want to talk about it, ping me on Twitter or here, see you soon!

Pacers – driving force of teams

Evgenii Ponomarev — Tue, 08 Sep 2020 19:29:49 +0000

— Why are some teams more productive than others?

— Why do some people regularly overwork and eventually burn out?

— How to create an environment where people feel safe to have ups and downs while striving for perfection?

During the next few weeks, I’ll be writing about pacers (= pacemakers) — people, who can impact the performance of teams and companies by just speeding up or slowing down. This article is the first one that briefly touches the concept — the more practical stuff will follow soon after.

Let’s jump straight to the point and start with a short Q&A:

It starts with running

Have you met Eddy, my imaginary friend, who is about to run his first marathon?

As pretty much any work, it’s about knowing what you’re capable of, preparing a plan and following it until job is done.

It looks like Eddy can do all of it, can’t he? Let me ask him again short after the race:

Following the plan can be challenging when the environment doesn’t support you. It’s hard to maintain your own pace when yet another grandma outruns you with no visible effort.

You know that you have to follow your pace, but it's easier said than done.

Luckily, Eddy has one more chance in several weeks — would he learn anything from the recent failure?

That’s quite an improvement — instead of chasing people whose abilities and strategies are unknown, he will run together with his friends.

They will play the role of his pacers — people whose job is to be a benchmark for others.

Let’s find out if it was better this time.

Eddy managed to slow down and had enough energy to finish the race, but the pacers weren’t running fast enough for him to show the best performance.

What’s about your best performance?

Look at your team

Both issues frequently happen around us:

People pick up a pace they cannot maintain because their colleagues appear as more productive. With time, it leads to ~~overtraining and injuries~~ stress and burnout.
People do less than they can because their colleagues aren’t doing much either. Eventually, the whole team slows down to the point of being completely unproductive, and everybody’s skills degrade with not enough practice.

Found yourself in one of those situations? Feel that somebody from your team can experience it too?

You can improve it.

Pacing issues may sound too hard to address, especially when you’re an engineer and who seem to have no power over the situation. But there are still several important things you can do:

Help those who need a hand to increase the pace of the whole team.
Control your own behaviour and be a healthy example.
Talk openly about all the challenges you face to make attitude and process as visible as the final result.
Never hesitate to ask for help and encourage others to do the same.

All of these will require better coverage, and I’ll do it in the following parts. Here is the rough plan for the next articles:

Who are the pacers in your team and how we choose them
Two types of pacers: attitude and contribution
How to improve team pace
Why it’s much harder with remote work
Dangerous pacemakers and false pacemakers: how to identify and deal with

As always, if you like the topic and want to talk about it, ping me on Twitter or by email.

And subscribe to "Resilient Systems" to get new posts sooner than anywhere else.

How to Choose Side Projects That Won't Burn You Out

Evgenii Ponomarev — Fri, 17 Apr 2020 16:09:41 +0000

Do you have side projects? If not, you probably should think about it – they can be helpful in the long term.

But I'd suggest being mindful when you're starting one – otherwise, it can cause unpleasant and unnecessary headache, as it did for me many times. But how can we avoid it? What defines sustainable projects? Let's think about it together.

"Unhealthy" side projects

It is not always smooth. I had plenty of side-projects I wish I had never taken.

Sleepless nights, strict deadlines, demanding users – all of these made me want to finish it sooner and come back to the safe old world, where I had only a full-time job and nothing else.

I know at least one person who experienced the same:

James Corden (Francis Henshall) – ONE MAN, TWO GUVNORS. Credit: Johan Persson

Francis Henshall is the main character of the comedy "One Man, Two Guvnors". This is a setup from Wikipedia:

<...> Francis Henshall becomes separately employed by two men <...>.

Francis tries to keep the two from meeting, in order to avoid each of them learning that Francis is also working for someone else.

It goes normal for a minute or two until two bosses start giving him urgent orders at the same time. As a bonus, the main character holds his own personal interest, which sometimes conflicts with the goals of his employers. What can go wrong?

This is the best illustration of how unhealthy side projects look like.

Honestly speaking, those are not really side projects. The word "side" implies that you have something "main", like a full-time job, family, friends and other areas of your life you aren't ready to sacrifice.

If a side project cannibalises your time and attention from the "main" activities, you're dealing with either of two options:

The side project is unsustainable
It is not a side project, and you need to be explicit about your priorities

Which projects are "healthy"?

I've formulated these criteria for myself:

I should know an exhaustive list of goals I'm trying to achieve
If I fail a project, only those goals should be affected
Those goals should not be time-bounded

Why do I emphasise the part about goals so much? Because I need to estimate a blast radius in case the project fails, which frequently happens with side projects.

– Would I let someone down?

– Would it affect my reputation?

– Would I be affected financially?

– Would it require unexpected extra work?

I want to be able to answer "no" to all of these questions.

How to stress-test your projects

As always, I need someone to demonstrate an example. Today it will be an extraverted Jane.

Imagine that a friend is asking her to build a website. What could be Jane's motivation?

These goals will not affect other duties – she will be fine even if she fails them. But what's about the friend? How urgent is it for him?

That's the best possible response – if Jane doesn't have enough time, she will ask him to find somebody else.

If the time boundaries were stricter, "no" would be a preferable answer. After all, she can learn Typescript in a much more relaxed environment.

Time for the last check.

Is there any chance that this project will become urgent?

At first glance, no. There is plenty of time, why would it become urgent?

But it's not that simple. Imagine them in two months, after Jane built the website:

Then it quickly becomes unsustainable and causes all the pain I've mentioned before.

What if the goals are ambitious?

Does it really mean that you should deny everything if it can become urgent? In the end, all ambitious projects imply potential success and consequential urgency.

This would be quite a thoughtless strategy.

Luckily, there is a way out.

You need to remove yourself from a critical path

If you're building something with high potential – try to spread the load. Build a community, even if it's just a community of two. There will be much less chance that unexpected urgency will catch you off guard.

Being extra explicit also serves the same purpose:

The same approach works with pretty much any system: you can either strictly define the SLA (= agree on expectations) or add more resources (= build a community).

After all, you won't do any good if you're burned out.

Take care*!

* and subscribe to "Resilient Systems" if you haven't already.

You can also do it on Facebook or follow me on Twitter, where I share all the links.

How Bob used implementation details in tests

Evgenii Ponomarev — Mon, 23 Mar 2020 20:10:14 +0000

Have you met Bob? Probably yes, if you read What Real Madrid can teach us about leadership or my recent article about using "silent grouping" in estimation.

If you haven't – don't worry, I'll introduce him now.

Here is Bob:

One interesting fact about Bob is that he is really into testing:

But something really odd happened to him recently. He wrote a new feature, covered its logic with tests and sent it to code review.

Can you imagine how dumbfounded he was when Jane, his colleague from the picture above, rejected the pull request with this comment:

And it was just the beginning of a long conversation.

Let's challenge best practices

Bob quickly responded to Jane's comment and started refreshing the page, waiting for an answer.

"You know that GitHub will notify you about all new comments, right?" Jane saw what he's doing and laughed at that.

"Do you have time now?"

Bob nodded: it was his last task, and he had all the time in the world.

Jane moved her chair closer to Bob's desk and looked at the screen once again. "Any real reason..." she said thoughtfully. "I don't know, it's just good practice, isn't it? People wouldn't do it if it was bad."

"People recommended bloodletting not too long ago," Bob replied with the smile. "And CSS-in-JS".

Jane laughed, "CSS-in-JS is cool, come on. But you have a point, let's find out a reason".

She took the keyboard, opened Google and started searching.

Implementation details in tests

She pressed enter and opened several links from the results.

The majority of the pages had the same advice – to not touch anything private in the tests. But as Bob supposed, they could not find any clear rationale for not doing it.

"Well...", Jane started. "These people all ask the same question – what is the point of testing private stuff if you can test public interface? Public methods will call private ones anyway, and eventually, everything will be tested".

"The reason is the same as for writing any unit-tests – to test small pieces of code in isolation. If a public method calls five private functions, I'd like to have separate tests for those functions, to see which of them are failing."

"But why?"

"Because otherwise, I gonna have a red test for the public method, with no clue what went wrong. And I'll need to debug it to locate the exact part which caused the problem. But we can have more granular data if we have tests for private functions. If they fail, you'll see where it was broken".

"You're probably right..." said Jane, her mouth tightening.

What Bob said made sense, but she wasn't entirely convinced. "I will approve your pull request".

Her colleague saw the scepticism in her eyes, which wasn't the desired outcome he had hoped for. "Do you still disagree?"

"Not really... your point sounds legit, but why all these people keep advocating against using private methods in tests? I mean, someone like you could just come and tell everybody why we need it... Unless they have some strong arguments".

The first reaction that Bob had was to shrug his shoulders and merge the pull request. But then a good idea came to his mind. "Do you want to ask Lucas?"

"Oh, yes!" she exclaimed. "Weird that we didn't think about it earlier".

And they went to talk to their team lead.

Let's ask the team lead

"We just want to know what you think about using private stuff in tests," Jane started when three of them entered the meeting room.

"What is private stuff?" Lucas didn't seem to understand the question.

"Private is something which is not public," Bob declared with a philosophical expression.

Jane continued, "Like private methods, for example. Many people warn against testing them, but Bob has a point why it can be useful".

"But how would you do it, when these methods are private, and you don't have access to them?"

"Well..." Jane tried to explain. "There are still many ways, right?

Some languages have workarounds to access private elements, for example, @VisibleForTesting in Java. Languages like Python and Javascript may not have real privacy at all".

"Privacy by convention," Bob added.

"Okay, I got it," said Lucas with the look as if he got it. "The short answer – it depends".

Bob rolled his eyes. He recently noticed a correlation – the more experienced people are, the more frequently they use the phrase 'it depends'. He didn't like it, but at least he knew how to look more knowledgeable.

"Depends on what?" he asked.

"On the system you're building," Lucas said and took the marker. "Unit tests are just code, and every code has its cost."

He came to the whiteboard and started writing:

"Cow?" Bob laughed and put two fingers to the head, imitating horns.

Lucas gave him a disappointed look and rewrote the formula:

"It's the cost of writing. If you develop a prototype which is going to be thrown away as soon as it's demonstrated, you don't really care about future maintenance, you aim to produce correct code as fast as possible.

On the contrary, if you're working on something with a potentially long lifespan, you may want to invest more time on writing code, if it can save future costs".

Jane and Bob nodded synchronously.

Lucas continued, "Your decision whether to use private implementation details in tests or not depends on what you're trying to optimize for. Why do we need unit-tests at all?"

Bob felt that it was a rhetorical question, but Lucas seemed to wait for an answer. Jade started first.

"To not break anything if we change code in the future," she replied.

"One," Lucas said and underlined the phrase 'Cost of Maintenance' in his formula. "What else?"

"I do it to not test my changes manually all the time," Bob decided to try it too.

"And here is your cow, man," Lucas winked and drew the line under 'Cost of Writing'.

"If we don't need to run it manually after every change, writing code will take much less time," he continued. "There are many other reasons for writing tests, but let's not overcomplicate for now."

"So, how is it related to private methods in tests?" Jane asked, confused.

Lucas didn't answer but started drawing something else on the board:

"The problem is not only about methods," he finally said. "There are many ways how you can use implementation details in tests, and we can group them into two large categories. Does it make sense?"

Jane and Bob looked at each other. "Hm... sort of..." Jane said.

"Okay, okay," Lucas realized that he might have made it too complicated. He put back the marker and opened the laptop. "It's gonna be better now – I have a perfect example".

Let's find some old code

The "perfect example", as Lucas put it, was written by him three years ago. It took a while to find it in the repository's history because the code had been changed quite a bit since then.

"Don't laugh, it's quite old," he hid unrelated functions from the file and turned the screen towards his colleagues.

https://gist.github.com/elergy/...

"What was it for?" Jane asked.

"It was supposed to keep track on available books in the library," Lucas answered. "Physical books, not for Kindle. A library keeps information inside the internal property _books and gives two methods to the public: addBook and takeBook. Makes sense so far?"

Jane and Bob didn't have anything to ask.

Lucas continued "Time to test it. Which tests we would need to write to cover the method takeBook? I see at least two: the one when we return null, and the one when we find a book and return it".

He picked the marker again and wrote these two cases:

"Sounds valid?" he asked.

"Kinda..." Bob replied uncertainly and examined the code again:

gist.github.com/elergy/...

"We can make it more specific," he said. "We return null if the book does not exist at all, right? At the same time, we return null if we request a valid book, but there are no available copies in the library."

"Oh, yes!" Lucas agreed. "We need many cases! In fact, we do have many cases, check it out."

He opened a large file with tests and randomly picked one test-case in the middle:

gist.github.com/elergy/...

"Here, we test that we cannot take a book if it's already taken.

Firstly, we add some books into the library: two books written by Stephen King, one – by George Orwell.

Then we take one book from the library and check that we can't do it once again. Sounds reasonable?"

"Yep," Bob said impatiently. He was used to watching YouTube on the 2x speed, and now he regretted that he can't speed up Lucas.

Lucas went on talking, "But I didn't really like the way we were writing it. The preparation phase looked cumbersome: for this case, it takes five steps, for others, it might be even longer. So I decided to try another approach".

He took a red marker and circled the word "Mocking" on his first drawing:

"Instead of calling several methods to achieve the state we needed, we decided to mock the final state – just override the property _books."

He returned to the laptop and opened a newer revision in the repository. The test looked slightly different:

gist.github.com/elergy/...

"Now, the preparation phase has only two steps instead of five: create a new library and change library._books to whatever we want. And it's always gonna be two steps, no matter which situation we need to replicate. We wrote many tests following the same approach until we saw that it wasn't as good as we thought..."

Bob finally looked incredulous. The test seemed to be very decent, "Why not good? What do you mean?"

Lucas spread his hands, "They quickly became unreliable. The tests were easy to write, even for some very complex examples.

But we had changed the code as well, and eventually, these mocks had gone out of sync. This is how the function addBook looked in some time, do you see the difference?"

gist.github.com/elergy/...

Jane found it first, "Yes, the fields author and title aren't on the top-level anymore – now they're inside info".

"Which makes our old mock invalid," Lucas confirmed.

He opened the mocked state again:

gist.github.com/elergy/...

Bob laughed, "So, you're saying that the test would still pass because library.takeBook cannot find the requested book and returns null?".

Lucas smiled, "Exactly! And we have a weird situation – the test is green, but it doesn't check the scenario it was supposed to verify. And if we break the code, the tests will not warn us".

He returned to the code and removed the check for !book.available:

gist.github.com/elergy/...

"This code will return any known book, regardless of whether it's available or not. We can take the same book one million times, and our tests will be silent.

But look at the first version of the tests again:"

gist.github.com/elergy/...

"It only uses the publicly available methods.

If we change private implementation details, this test will continue working just as before, giving us enough confidence".

Let's talk about static typing

"Wait a minute," Bob understood the problem but didn't fully agree with the conclusion. "Isn't it a problem of Javascript? If you used Java or even Typescript, you wouldn't miss this problem – the language itself would flag that the mock has an incorrect type."

"Actually, that's a good point," Lucas agreed. "How could Typescript protect us?"

"It just wouldn't allow us to run these tests until we fix outdated mocks."

Lucas skimmed through the file with tests, "We have sixty-two test cases. You're saying that we need to update mocks in all of them, right?"

"Of course!" Bob exclaimed. "How else?"

Lucas came to the board and highlighted Cost of Maintenance:

"There is no doubt that this situation is much better than having unreliable tests. But let me say it aloud, so you can hear how ridiculous it sounds – we changed two lines of private code, and now we need to fix sixty-two test cases.

It would never happen if we used only public API in tests, without mocking private details."

Jane sighed deeply, "You know, I was exactly in this situation one week ago. I slightly changed the shape of the Redux State, and then spent the whole day updating tests, which had outdated mocks..."

Lucas looked interested, "Oh, Redux is a huge topic! Maybe you can check our code and think about how to avoid this problem in the future? You can even write a blog post about it later."

Let's take a break and summarize

"That was about mocking," Lucas concluded and pointed out on the board where it still was highlighted.

"You see what happens when you mock private implementation details in tests, right?"

"Yep," Bob said.

"When we change code, even slightly , we may have to spend too much time fixing tests which didn't have to be broken."

"Only if you are lucky," Lucas reminded.

" These tests are much less reliable , because we may not notice that mocks were out of sync. You think that everything is tested, while, in fact, it's not."

"What's about asserting?" Jane asked. "Is it okay to call private methods to verify that they work correctly?"

Lucas checked his watches, "Have to run to another meeting, can we continue later?"

"Can you just tell us if it's okay or not?"

Lucas looked at the board as if trying to collect his thoughts and come up with a brilliant answer. Then he shifted his gaze to Bob and sighed wearily.

Subscribe to "Resilient Systems" to get all the latest posts delivered straight to your inbox

Credits for the camera on the cover picture goes to Benedikt Geyer on Unsplash