Forem: Klarna

Architecture Decision Studio

Thibault Jan Beyer — Wed, 30 Sep 2020 10:34:04 +0000

Fast actionable collaborative decisions

Is your team forming knowledge silos rather than working together? Do you have difficulties getting actionable outcomes from your meetings? Would you like to hold a fun and useful remote session to find solutions to problems together as a team but don't know how to start? We've designed a "remote first" method for this. But before I tell you more about it, let me quickly explain how we got there.

Can’t wait? Get the template and jump reading further below!

A lil’ Backstory

“At Klarna things change rapidly as we move very fast”

They said. How true. Within the first month of joining, we shifted focus 3 times. Handing over some services we created to other teams, taking over their services. While it was exciting to gain insight in a lot of areas from different perspectives, that also meant we got to handle a lot of tasks simultaneously. Naturally, individual knowledge silos formed around each team member. We ended up in a situation where each one of us was working on a separate feature/project. We identified the problem and were trying to re-focus, to get the whole team back working together on one single project at a time again. I had an MVP running for a product that was meant to become one of our main focuses.

My colleagues would avoid taking tickets regarding the next iteration of the project. I felt trapped, explaining the same thing over and over again. Wasn’t able to delegate tasks to other team members because of a lack of knowledge or context required.

It was time to break the silos, regroup, and take common decisions!

Usually, that is when we would lock us up in the “war room” (meeting room next door) and iterate over the project setup. Wait, a tiny room full of sweaty engineers during a pandemic? What a bad idea 🦠🦠🦠

No whiteboard, no in-person conversations. How can you run an architecture session to align everyone and brainstorm, while making sure everyone’s included? That’s where we set sails on a mission to try out collaborative techniques online using tools like Jamboard (because it’s part of the G Suite and sounds cool).

Trying the Architecture Golf

I started by just trying to squeeze the Architecture Golf, also referred to as “Group Whiteboard Sketching” into an online session without any big planning or preparation. Just to see how it goes.

Here is how it went: I took half the time just drawing out the MVP, I had some architecture diagram and explained it. By drawing out the steps and explaining it as we go we find team members are all onboarded around the problem space and can break down issues in approaches more rapidly. It also keeps it easy for everyone to follow and gives a common understanding of the problem space, which we found makes it easier for teammates to share and challenge ideas later on. Unfortunately, digitally that took way longer than it would have on a whiteboard.
Then I went ahead talking about the use cases that we want to solve and the rest of the time we tried to find a solution to the use-cases by an open discussion around the diagram. It was good as people were very happy to finally, after several months of isolation, kind of work together again. It also helped to explain the service in detail once and have everyone understand the architecture and vaguely the challenges ahead. But it wasn’t great either. We didn’t start with the most important use-case. The one we picked was not very clear, which led to a big discussion taking up the whole remaining time.

Example of an Architecture Golf session.

Trying the Lightning Decision Jam

So with these thoughts in mind, I set up a follow-up session trying another technique. Some sort of free interpretation of the Lightning Decision Jam. This time, focusing on only the most urgent use-case.

Overall I enjoyed the jam. It led to a decision. However, it kinda confused some participants. The structure was not clearly defined and poorly prepared from my side. I asked participants to draw their idea on a piece of paper first and when their turn comes to re-draw it digitally. A bad approach as it wasted time. Because of that, we required yet another session to refine the ideas presented, even fewer people attended.

Example of a Lightning Decision Jam Step 1–3.

Feedback & Iterate

In order to be able to iterate fast & improve I sent out a simple Google form survey asking for feedback.

Example of a Google form survey.

Knowing that my team is awesome, I was very keen on their actionable feedback.
And they sure did not let me down:

Architecture Golf: great to share knowledge, understanding, - and ownership of the project
Lightning Decision Jam: good to find an actual actionable solution to one problem
Generally: The sessions were missing structure, did not match time-lines and were not focused enough
When stuck on a topic, park it (write it down) and move forward to another topic
Quiet team members, even though they might have fabulous ideas, didn’t speak. Becoming a discussion between the loudest team members and the rest just watching silently.

Time to iterate. Read up more on lean sessions and find ideas to conduct them remotely. Plus, I remembered something similar to the lightning jam that an awesome project manager held back at edenspiekermann, it was called The Design Studio Method.

Soon enough we had another use-case to tackle. So I sharpened my keyboard and tried to funnel the learnings into a new grandiose session.

As it is an improved combination of Architecture Golf, Lightning Decision Jam, and Design Studio, let’s just name it:

The Architecture Decision Studio

Helps you find actionable solutions for your problems. We designed in a “remote first” fashion. It is a great tool to find solutions to problems within existing systems. Get everyone's input and produce actionable output. The first half is about finding a decision, the second half is about the architecture.
2x 1h meetings worked great for us.

Can’t wait? Get the template!

Guidelines when conducting the studio

Before the session

Research: how are other companies/teams solving this problem?
Define a list of topics you need to discuss and limit yourself to these
Keep the number of participants small: the more, the longer it will take. 3-5 is perfect
Split the sessions into multiple meetings
Consider when best to schedule meetings to get the best out of the attendees
Be very clear what needs to be discussed and what the supposed outcome of the meeting is
Have a narrow and well-defined scope
Write a pre-read, explaining the problem and get attendees to think about the agenda items beforehand

During the session

Create a safe space! Don’t openly challenge people's opinions. Don’t always give feedback, even if you don’t agree with an idea, let it live. Give only constructive feedback. If someone does not fully understand, follow, or does not feel well, they shouldn’t be forced to participate actively and feel free to just observe the session to learn. If none of your team feels comfortable, be honest about it, then this might not be the right format for your team.
Have an agenda
Start with introducing the agenda, what is getting addressed, what will happen, which decisions are going to be made and what are the next steps
Timebox discussions & timebox each agenda item
If it is looking like it will run over time, take it offline or bump it to another meeting
If an agenda item is becoming blocked, feel free to park it and move onto an item where consensus is likely

Afterward

Cut the meeting 5 minutes before time is up to summarize it again and the next steps
Ask for “anonymous” feedback on the session (survey)
Create actionable next steps
Ideally, send out a summary

Summary of the Architecture Decision Studio. Get the Jamboard template!

Stepping in the Studio

Step 0 — Pre-read

Share a pre-read in the invite where people can read up details on the problem/use-case you’re trying to tackle and any information you think is useful.

Step 1 — Introduction — 8min

Set up a presentation where you take about 5-10 minutes to quickly share the knowledge you have and introduce the problem/feature. Inspiration:
What is the task (why this meeting)
What else might come our way in the future?
Summary of previous discussions
How is it done/can this be done currently?

Afterwards, ask for questions. Make sure that everything is clear to all attendees.

Step 2 — Warm-up — 2min

Warm-up your audience and put them in the mood to participate. Ensure that there is a safe space for discussion. Make everyone feel comfortable. I.e. Brainstorming is a good idea (feel free to do something else instead): Start the timer for 2-3mins and have everyone write stickers on a board. After the timer is over, briefly go over the stickers, organize & group them, clarify uncertainties. Force yourself and others to talk about positive things. Whether it is “Pros and Cons of the current solution” or “What do we have already and what is missing” or “your favourite animals”. Use it to give a bit of context. Make it generic so that everyone can participate.

Example of a pros and cons warm-up

Step 3 — Ideas — 8min

This is where the magic happens. As an introvert, I love this part. It’s “working alone together”. In contrast to an offline meeting, online it is very important that you prepare this slide. You know who is attending so prepare 1 slide per participant (the Design Studio method would suggest that you split each slide into 6 squares, but that is up to you). Each participant can then within the 5-10mins draw as many solutions as they can come up with on their slide. The more the merrier.

This is not a drawing competition. It doesn't matter whether someone draws a sketch, an abstract idea or even just writes a phrase. As long as they can explain it later on, it’s perfect.

Step 4 — Presentation — 2min/pp

One by one, go through each slide and give each participant the opportunity to present their idea(s). 2-3mins per person. No feedback. No comments. No “I like/dislike this idea… It’s possible/impossible…”. This is solely about presenting the idea. Questions that help to understand the idea are allowed. By not allowing ideas to be questioned or shut down you create a safe place and open up lines of thinking that may have been quickly disregarded previously.

Make sure to give each idea a “friendly name” (2-3 words that summarize the idea). Will be useful later on. Put it on a post-it and go on to the next slide. Next person. Rinse. Repeat.

Step 5 — Voting — 1.5min/pp

Put all idea summaries together. From now on the ideas belong to the group, not to individuals. Make sure not to add names to the ideas. Just place the ideas up for a vote. They should stand for themselves. If you already have repetitive ideas you can group them together as one idea.

Now is the time to vote. Everyone has 1-2mins to give, one by one, friendly feedback focusing on the problem. Take each idea and say what problems the idea solves or not. If you think the idea is a potential solution candidate, give it a thumbs up (vote). There is no limit of votes.

Once everyone has voted you should have a winning idea.

Draw?

If there is a draw for the first place, you have various options:

Go back to step 3 and have another round of idea drawing where everyone can improve ideas or combine ideas to form another better idea. (this is the suggested way)
You can let people discuss why they voted for one or the other idea and then re-vote
The team-lead can decide for a solution in order to unblock the situation.

Example of a winning idea

We have a winner!

You should now have a 1st place, 2nd place, and 3rd place idea(s). Usually, the first place is the solution we want to implement. Feel free to put 2nd and 3rd in your backlog if they address other important issues but focus on the 1st idea within this studio.

Break

Perfect time to take a break. You may split the meeting into several meetings and stop here. Otherwise, this is a good time for a 5-15min break.

If you had a long break, start the next step by recapitulating in 3mins what was achieved in the last session, which decisions were taken, and what will be tried to achieve now.

Step 6 — Refinement — 45min

Refine the idea and figure out the steps necessary to implement it.

Have the existing website/app layout/system pre-sketched
Take turns drawing/writing what is necessary to be changed to adapt the solution idea
Each participant has 1-2minutes to:
- Change the existing design where necessary, OR
- Add to the design (or sketches done by the previous participant(s)), OR
- Pass their turn if they don’t have anything to add Ideally, whenever the design changes, take a screenshot to document the evolution.

Naturally, discussions will evolve as people are sketching. That’s ok, go with the flow and don’t try to force it. However, try to keep the time schedule and turn-based system. It helps to give everyone an equal amount of speak/sketch/write time, makes sure that the discussion is not one-sided, gives quiet people the opportunity to come to word, and prevents the discussion from getting out of hand.

If you still feel that a topic still gets out of hand, park it, and continue with something else.
Limit the refinement part to somewhere around 40-50mins depending on the complexity.

Step 7 — Actionables — 10min

Write down summaries of all the changes necessary (from the refinement). Write down the open questions. Write down the parked topics. These are your actionables. Ideally, they can be translated into tickets. If there is not enough time left-over to do it together, feel free to take it offline after the session.

Closure

Summarize what was achieved and make sure to honestly thank your participants because they’re the best!

Template

Jamboard template

What now?

We tried the Decision Studio several times already. It was fun and effective. Putting us on the same page, fostering collaboration, and producing actionable output.

It’s not done yet, we’ll continue to use the Decision Studio Process and encourage other teams to try it. Continuously improving the format. I’m planning a central gateway for it to make updates/refinements easier.

However, I fear that by standardizing it, it might become boring. To fight boredom, there has to be variety! So I strongly encourage you to make tiny changes in each session and tailor the Studio exactly to your current needs to really make it your own!
(and please share your outcome with us)

That’s it, keep it smoooth!

Stepping Down as a Dictator: Giving Great Teams Permission to Make Awesome Decisions.

Phil Bennett — Wed, 17 Jun 2020 07:23:31 +0000

At Klarna, we hire amazing people. This statement isn't hyperbole. I've not met one person in the year that I've been at Klarna who didn't impress me in some way.

But one thing that I've learned this last year is that if you put a group of amazing people in a room, they struggle to make decisions. The deeper a team thinks about its problem space, the more it generates strong, convincing ideas and solutions. And therefore, the harder it is for the team to decide how to move forward.

In this article, I will explain how we're using structured, collaborative, and fun decision making processes to help drive our team forward and have maximum impact on our problem space.

Collective Analysis Paralysis

Analysis Paralysis is the concept that an individual can over-analyze a problem and block themselves from moving forward with a solution. The paralysis problem is exacerbated when a group of highly analytical individuals come together as a team.

Klarna hires against a set of leadership principles. These principles define how we go about delivering products. One of those principles is 'Detailed Thinkers'.

Leaders know that to be a disruptor in a competitive industry requires radical, detailed thinking. They are ambitious and communicate a daring direction that inspires results, but they also love details and have a contagious desire for knowledge. They leave no stone unturned when it comes to finding Smoooth solutions. Nothing is impossible. They are a unique combination of ambitious, free-thinking with a meticulous eye for detail.

As a result, we end up with teams with strong analytical skills.This is what happened in my team. The result is impressive people who struggle to make collective decisions. We are great at generating concepts, ideas, and potential solutions, but agreeing on the right outcome has always been a challenge.

The Problem with Deadlock Breaking.

As a manager, or lead within a team, it’s your responsibility to break these kinds of stalemate situations. And there’s an easy solution to this. As the lead, you get a single overriding vote to unblock the discussion and move the team forward.

This single vote is traditionally what managers do. They use their wealth of experience and knowledge to choose the right option. However, this veto vote has two significant challenges associated with it; bias, and ownership.

Challenge 1: Bias

As a manager with 20 years of industry experience, I have biases. I actively try to keep them in check but the situations that I have experienced over my career have built up a set of subconscious patterns in my decision making. If I make the final call, it will be based more on my knowledge and understanding than the collective consciousness of the team.

Challenge 2: Ownership

The second challenge is that, as the lead, I now own that decision. It’s mine and it doesn’t fully belong to the team. If we had multiple conflicting ideas to start with, I have dismissed the ideas of a proportion of the group.

This is manageable with good leadership skills, but it would have been a whole lot better if we had collaboratively made that choice.

Unblocking Collaboratively in 5 Easy Steps.

As a team, we use several tools to help us make significant decisions collaboratively. Two of these tools are Design Sprints for larger problems and Lightning Decision Jams for smaller ones.

The Design Sprint is a week-long five-phase process that helps teams use design thinking to come up with awesome product ideas.

The Lightning Decision Jam is a similar process that focuses on a similar set of phases into the time slot of a single meeting.
What is shared with both of these processes is that they follow a tight timeline that allows you to use the best elements of your team to come to fantastic decisions.

1. Understand the Problem.

The first step in any decision-making process is fully understanding the problem you’re trying to solve. It would be best if you had a detailed outline of what you want to achieve as a team before doing any problem-solving. If you’re all trying to solve different problems, you’re already setting off on the wrong foot.

When pushed for time, you can use your team to perform this step. However, when you have the luxury of time, it’s great to get domain expert views on the problem. The Design Sprint sets aside a time that allows you to invite experts into your session and interview them to gain a better understanding of the problem area.

2. Explosive Creativity.

Never discredit the hidden skills in your team. In talented, deep-thinking teams anyone can come up with the best solution for your customer, not just the designers and product managers.

Get everyone in on the solutions part of the process, give them all time to draft, sketch and propose ideas. Give them permission to be creative.

Both of these processes use the concept of ‘working alone together’. They let each teammate work on the solutions process independently but in the same physical space.

The outcome of this step is a ton of fantastic ideas.

3. Collect Solutions.

Now that you have at least as many great ideas as you have great minds in your team, it’s time to collect and present the ideas.
These ideas should be presented with minimal discussion. The concept is that the ideas should be clear and stand up on their own, without spending time digging into the details.

If the idea is great, then you should need no more than a few sketches and three minutes of your time to describe it. If the concept can fit on one post-it note, all the better.

4. Vote.

Now to make the decision. In all of these processes, the voting step is simplest; everyone gets some voting points; they spread the voting point between the ideas that they think are best.
The best idea wins.

It’s that simple; immediately you’ve got down from several awesome ideas to the one the team collectively thinks is the best.

Time to deliver!

5. Own it.

What is so amazing about this process is that involving the whole team in every step of the process breeds a fantastic level of ownership. These solutions belong to the group, not the manager, designer, or product owner.

We are all building the thing that we all chose to create, and we’re going to smash it as a team!

Next Steps.

To run a similar process, you only really need one thing, a facilitator. That person can be anyone in the team, so it could easily be you.

Getting an agreement for a team to spend a whole week on a design sprint can be challenging, the first time. What I would recommend is next time you have a problem, and you can’t collectively agree on the best way forward, use the Lightning Decision Jam process or define something very similar.

Once you have built confidence in this process, you will find it easier to convince people that the investment of a Design Sprint is worthwhile.

I can promise you that once you’ve run one Design Sprint, you’ll have plenty of evidence to convince people to do more.

6 Lessons learned from optimizing the performance of a Node.js service

benzaita — Tue, 14 Jan 2020 08:05:45 +0000

Here at Klarna, we put a lot of effort into empowering our developers to deliver high-quality and secure services. One of the services we provide our developers with is a platform for running A/B tests. A critical component of this platform is a fleet of processes that for every incoming request, makes the decision: which flavor of the test (A or B) to expose the request to. That, in turn, determines what color to render a button, what layout to show the user, or even which 3rd party backend to use. These decisions have a direct impact on user experience.

The performance of each process in this fleet is critical since it is used synchronously in the critical decision paths in the Klarna ecosystem. A typical requirement in such flows is to decide within a single-digit latency for 99.9% of the requests. To be confident that we keep adhering to these requirements, we developed a performance testing pipeline to load test this service.

Takeaway #1: performance testing can give us confidence that we are not degrading performance with each release.

Although we barely saw any performance issues in two years that this platform is in production, the tests were unambiguously showing some issues. Several minutes into the test, at a moderate and stable request rate, the request duration spikes up from its normal range to several seconds:

We decided that although this did not happen in production yet, it was just a matter of time until the real-life load “catches up” with the synthesized load, and therefore, this is something worth investigating.

Takeaway #2: by “cranking up” the load we can expose problems before they ever reach production.

Another thing to note is that it takes around two or three minutes for the problems to appear. In the first iterations, we ran this test for only two minutes. Only after extending the duration of the test to ten minutes, we discovered this problem.

Takeaway #3: long load tests can surface different kinds of problems. If everything looks OK, try extending the duration of the test.

We normally monitor services using the following metrics: number of incoming requests per second, duration of incoming requests, and the error rate. These give a pretty good indication of whether the service is experiencing problems or not.

But these metrics do not offer any insights when the service misbehaves. When things go wrong, you need to know where the bottleneck is. For that, you need to monitor the resources that the Node.js runtime uses. The obvious ones are CPU and memory utilization. But sometimes these are not the actual bottlenecks. In our case, the CPU utilization was low, and the memory utilization was low as well.

Another resource that Node.js uses is the event loop. In the same way we need to know how many megabytes of memory the process is using, we also need to know how many “tasks” the event loop needs to handle. The event loop is implemented in a C++ library called “libuv” (here is a great talk about the event loop by Kenneth Gibson). The term it uses for these “tasks” is Active Requests. Another important metric to follow is the number of Active Handles, which is the number of open file handles or sockets that the Node.js process holds (for a complete list of the kinds of handles, see the libuv documentation). So if the test is using 30 connections, it would make sense to see around 30 Active Handles. Active Requests is the number of operations pending on these Handles. Which operations? The full list is available in the libuv documentation, but these can be read/write operations, for example.

Looking at the metrics reported by the service, there was something wrong. While the number of active handles is what we would expect (around 30 in this test), the number of active requests was disproportionately large — several tens of thousands:

We still didn’t know which types of requests were in the queue, though. After breaking down the number of active requests by their type, the picture was clearer. One type of request stood out in the reported metrics: UV_GETADDRINFO. This type of request is generated when Node.js is attempting to resolve a DNS name.

But why would it generate so many DNS resolution requests? Turns out that the StatsD client we are using attempts to resolve the hostname for each outgoing message. To be fair, it does offer an option to cache the DNS results, but that option does not respect the TTL of that DNS record — it caches the results indefinitely. So if that record is updated after the client already resolved it, the client will never be aware of it. Since the StatsD load balancer might be redeployed with a different IP, and we cannot force a restart of our service to update the DNS cache, this approach of indefinitely caching the results was not an option for us.

The solution we came up with was to add proper DNS caching outside of the client. It’s not hard to do by monkey patching the “DNS” module. And the results were better:

Takeaway #4: don’t forget to consider DNS resolution when thinking about outgoing requests. And don’t ignore the record’s TTL — it can literally break your app.

After solving this problem, we re-enabled some more features in the service and tested again. Specifically, we enabled a piece of logic that produces a message to a Kafka topic for every incoming request. The tests revealed, again, significant spikes in response time (seconds) for significant periods:

Looking at the metrics from the service showed an obvious problem in that very feature we just enabled — the latency of producing messages to Kafka was extremely high:

We decided to try a trivial improvement — queuing the outgoing messages in memory and flushing them in a batch every second. Running the test again, we saw a clear improvement in the response times of the service:

Takeaway #5: batch I/O operations! Even when async, I/O is expensive.

Final note: running the tests mentioned above would have been impossible without a way to run tests with reproducible and consistent results. The first iterations of our performance testing pipeline did not provide us confidence in their results since they were not consistent. Investing in a proper testing pipeline allowed us to try out things, experiment with fixes, and mostly be confident that the numbers we are looking at are not coincidental.

Takeaway #6: Before attempting any improvements, you should have a test that you trust its results.

Frequently Asked Questions (FAQ)

I’ve received some questions about which tools were used to perform the tests here. There are a couple of tools used here:
The load is generated by an internal tool that simplified running Locust in distributed mode. Basically we just need to run a single command and that tool will spin up the load generators, provide them with the test script, and collect the results to a dashboard in Grafana. These are the black screenshots in the article. This is the perspective of the (client) in the tests.
The service under test is reporting metrics to Datadog. These are the white screenshots in the article.